| *** kfox1111 has joined #openstack-infra | 00:00 | |
| kfox1111 | how do I determine the log dir for https://review.openstack.org/#/c/449826/ while its still running? | 00:01 |
|---|---|---|
| kfox1111 | you use to be able to link off of a currently failed job in the zuul page. it now shows the telnet url even for finished jobs now. | 00:01 |
| kfox1111 | I want to look at the failed jobs before the other jobs are done to help me debug faster. | 00:02 |
| *** claudiub has quit IRC | 00:03 | |
| *** rtjure has joined #openstack-infra | 00:03 | |
| jeblair | kfox1111: i see logs.o.o urls for all the jobs that are finished | 00:05 |
| jeblair | i see telnet only for the centos-binary-3-ceph-multi job which is still running | 00:05 |
| *** slaweq has joined #openstack-infra | 00:07 | |
| kfox1111 | weird... | 00:08 |
| *** dave-mccowan has joined #openstack-infra | 00:11 | |
| *** slaweq has quit IRC | 00:12 | |
| *** Swami has quit IRC | 00:14 | |
| ianw | tonyb: yeah, i think the question was https://review.openstack.org/#/c/504964 first or EOL branches first. i think if we remove the branches, then the regexes disappear and everyone's happy | 00:16 |
| ianw | pabelanger: you aware of issues with gate-dsvm-nodepool-ubuntu-src ? | 00:16 |
| *** hongbin has quit IRC | 00:18 | |
| *** pvaneck has quit IRC | 00:19 | |
| * fungi guesses, based on e-mail thread about preinstalling packages, that dmsimard was not yet around back when we have devstack-specific node images | 00:19 | |
| *** pvaneck has joined #openstack-infra | 00:19 | |
| fungi | s/have/had/ | 00:19 |
| tonyb | ianw: except some projects really want to keep a mitaka branch. I don't think they have periodic jobs though | 00:20 |
| *** dhajare has joined #openstack-infra | 00:20 | |
| tonyb | ianw: Oh wait I mis read what that was doing. | 00:21 |
| *** pvaneck_ has joined #openstack-infra | 00:21 | |
| *** bobh has joined #openstack-infra | 00:21 | |
| *** Apoorva_ has joined #openstack-infra | 00:22 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 00:23 | |
| *** pvaneck has quit IRC | 00:24 | |
| * tonyb doesn't really know what the right way forward is :( | 00:24 | |
| *** pvaneck_ has quit IRC | 00:25 | |
| *** Apoorva has quit IRC | 00:26 | |
| *** Apoorva_ has quit IRC | 00:26 | |
| ianw | tonyb: i feel like we'd be best to remove EOL branches, then remove the regexes, to minimise possibility of breakage | 00:27 |
| ianw | i think the issue is that if we can clear this out it's easier for zuulv3 deployment | 00:28 |
| tonyb | ianw: But what about the 2 projects that really want a stable/mitaka branch? | 00:28 |
| tonyb | wont removing the regexes break them? | 00:28 |
| tonyb | not that they have much in the way of CI anyway | 00:28 |
| ianw | yeah, and easier to just special-case two things than worry about generic solutions | 00:29 |
| *** ijw has joined #openstack-infra | 00:29 | |
| tonyb | ianw: okay Fine by me | 00:29 |
| tonyb | ianw: Shall I update the list and send it to y'all? | 00:29 |
| *** bcat_ has quit IRC | 00:29 | |
| jeblair | clarkb, dmsimard: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_uuid:0a235d3cda2c45a1b0b1fb9d644b451f that works now, so i think we can promote the logstash stuff from base-test to base | 00:31 |
| ianw | tonyb: yeah, i think so, i've done a couple of EOL's and can work up a plan | 00:32 |
| *** ijw has quit IRC | 00:32 | |
| *** ijw has joined #openstack-infra | 00:32 | |
| tonyb | ianw: jhesketh has a nice script in openstack-infra/relaese-tools, I generate a script to call that script | 00:33 |
| tonyb | :) | 00:33 |
| *** thorst has joined #openstack-infra | 00:34 | |
| ianw | yep, cool, i think the only magic required is adding yourself to the right group temporarily | 00:34 |
| *** mriedem has quit IRC | 00:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Promote logstash jobs from base-test to base https://review.openstack.org/505451 | 00:40 |
| jeblair | dmsimard, clarkb, mordred: ^ i think that's gtg | 00:41 |
| * jhesketh reads scrollback | 00:41 | |
| jhesketh | ianw, tonyb: I'm happy to help with EOL'ing again if needed | 00:41 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add netaddr requirements for running ipv4|ipv6 filters https://review.openstack.org/504522 | 00:42 |
| ianw | jhesketh: all the better :) if we can clear this it's one less thing for zuulv3 to worry about | 00:43 |
| *** maciejjo1 has joined #openstack-infra | 00:43 | |
| *** bcat_ has joined #openstack-infra | 00:43 | |
| *** maciejjozefczyk has quit IRC | 00:45 | |
| *** dave-mcc_ has joined #openstack-infra | 00:45 | |
| jhesketh | ianw, tonyb: so none of those projects need liberty or mitaka branches any more? Or as ianw mentioned are they going to be special cased in v3, in which case I'm not sure we can simply eol the branch | 00:47 |
| *** dave-mccowan has quit IRC | 00:48 | |
| tonyb | jhesketh: as I understand it we're going to EOL all the branches older than mitaka | 00:48 |
| tonyb | jhesketh: and EOL the mitaka branches for everything except 2 projects | 00:48 |
| ianw | jhesketh: i think we want to avoid the regexes in https://review.openstack.org/#/c/504964 being applied by the migration script unnecessarily | 00:49 |
| tonyb | jhesketh: and thos 2 projects will get the special case | 00:49 |
| *** ijw has quit IRC | 00:49 | |
| tonyb | jhesketh: and then those regexes can be removed 'globaly' | 00:49 |
| jhesketh | tonyb: so the 2 exceptions don't need those regexes? | 00:49 |
| ianw | what exactly are the two exceptions? do they even match jobs in that regex? | 00:50 |
| *** dpawlik has joined #openstack-infra | 00:50 | |
| tonyb | ianw: Actually it's 5 repos from 2 projects (I had misremembered the thread) | 00:51 |
| *** danpawlik has quit IRC | 00:51 | |
| *** mat128 has joined #openstack-infra | 00:51 | |
| tonyb | project['name'] in ['openstaack/group-based-policy', 'openstaack/group-based-policy-automation', 'openstaack/group-based-policy-ui', 'openstaack/python-group-based-policy-client', 'openstaack/networking-bigswitch']: | 00:51 |
| *** maciejjo1 has quit IRC | 00:51 | |
| tonyb | that's the list I haven't verified that have jobs that match the regexes | 00:51 |
| *** maciejjozefczyk has joined #openstack-infra | 00:52 | |
| tonyb | Only without the 'aa' :( | 00:53 |
| jhesketh | they have python jobs so it's quite likely they're using the regex | 00:54 |
| *** dave-mcc_ has quit IRC | 00:54 | |
| *** zhurong has joined #openstack-infra | 00:54 | |
| *** nicolasbock has quit IRC | 00:55 | |
| fungi | openståkk | 00:55 |
| fungi | or is it øpenståkk? | 00:56 |
| tonyb | :) | 00:56 |
| ianw | yeah, the have -trusty jobs | 00:56 |
| *** LindaWang has joined #openstack-infra | 00:56 | |
| ianw | it seems we really only need two special cases then | 00:57 |
| fungi | note that we've said as soon as infra no longer runs trusty jobs we're axing those images | 00:57 |
| ianw | with (group-based-policy-.*) and networking-bigswitch | 00:57 |
| jhesketh | so assuming we have to keep their branches and their trusty/34 jobs, perhaps the migration should be hard coded to skip over those regexes so zuul2.5 still knows about them and then we can manually special case them in v3 | 00:57 |
| ianw | jhesketh: i was thinking add two matches per ^^, EOL everything else, remove global regex ... profit? | 00:58 |
| jhesketh | oh, I see... special case them in v25 too | 00:59 |
| jhesketh | yeah that's probably a good idea | 00:59 |
| *** ijw has joined #openstack-infra | 00:59 | |
| *** pahuang has quit IRC | 01:00 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /tenants route https://review.openstack.org/503268 | 01:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/status route https://review.openstack.org/503269 | 01:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs route https://review.openstack.org/503270 | 01:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route https://review.openstack.org/466561 | 01:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 01:02 |
| ianw | ok, i'll see if i can come up with something in a little bit | 01:02 |
| jhesketh | cool :-) | 01:03 |
| jhesketh | tonyb: if you want to send through an eol list I can process it | 01:03 |
| tonyb | jhesketh: it is in transit | 01:03 |
| *** bobh has quit IRC | 01:03 | |
| *** aeng has quit IRC | 01:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 01:04 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 01:04 |
| *** ijw has quit IRC | 01:04 | |
| tonyb | jhesketh: http://lists.openstack.org/pipermail/openstack-dev/2017-September/122311.html | 01:05 |
| tonyb | jhesketh: if the format isn't useful I can present it another way just tell me how you'd like it | 01:06 |
| jhesketh | tonyb: ah cool, I must have missed that in my email sort :-( | 01:06 |
| jhesketh | thanks! | 01:07 |
| *** tpsilva has joined #openstack-infra | 01:07 | |
| jhesketh | tonyb: nah that format is heaps easier to use than your previous one, thanks! | 01:07 |
| pabelanger | ianw: I am not, however I haven't looked at it in a while | 01:07 |
| tonyb | jhesketh: I'll update my other tool to use this format | 01:07 |
| *** dangers_away has quit IRC | 01:08 | |
| jhesketh | tonyb: did you need/want to wait for feedback about eol'ing any of those repos (it's likely you already have but I've not been paying very close attention :-s) | 01:08 |
| jhesketh | I have to leave for an errand, but will look at it this afternoon | 01:08 |
| *** slaweq has joined #openstack-infra | 01:08 | |
| tonyb | jhesketh: Cool. Interested parties have had 2+weeks to complain (and hence the 5 excpetions) so I think you're good to go when you get back | 01:09 |
| *** bobh has joined #openstack-infra | 01:11 | |
| *** slaweq has quit IRC | 01:14 | |
| *** liusheng has quit IRC | 01:15 | |
| *** eumel8 has quit IRC | 01:15 | |
| *** liujiong has joined #openstack-infra | 01:17 | |
| *** pahuang has joined #openstack-infra | 01:18 | |
| *** cuongnv has joined #openstack-infra | 01:19 | |
| *** baoli has joined #openstack-infra | 01:20 | |
| *** cshastri has joined #openstack-infra | 01:20 | |
| *** bobh has quit IRC | 01:25 | |
| *** rwsu has quit IRC | 01:25 | |
| *** slaweq has joined #openstack-infra | 01:30 | |
| *** bobh has joined #openstack-infra | 01:31 | |
| *** Kevin_Zheng has joined #openstack-infra | 01:33 | |
| dmsimard | ianw: o/ are you able to re-run a single post job that failed due to the ongoing gerrit shenanigans ? | 01:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add mitaka/trusty filters for projects wanting that branch https://review.openstack.org/505462 | 01:34 |
| ianw | dmsimard: umm, yes i think | 01:34 |
| ianw | jhesketh / tonyb: ^^ as discussed | 01:35 |
| dmsimard | http://lists.openstack.org/pipermail/openstack-dev/2017-September/122300.html -- seems like only ara-tarball-signing failed | 01:35 |
| dmsimard | huh, wait, that was actually an error on missing zuul cloner, wth ? | 01:35 |
| *** slaweq has quit IRC | 01:36 | |
| ianw | dmsimard: istr something about that in scrollback ... | 01:37 |
| *** bobh has quit IRC | 01:37 | |
| dmsimard | jeblair: re: logstash, the errors in the linters job from https://review.openstack.org/#/c/502515/ are weird ? It's complaining about missing roles all of a sudden | 01:38 |
| *** dhajare has quit IRC | 01:38 | |
| dmsimard | looks totally unrelated to the change itself and the logs are indeed in logstash, just confusing | 01:38 |
| ianw | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-09-19.log.html#t2017-09-19T19:43:59 | 01:40 |
| ianw | dmsimard: ^ so the conclusion seems to be it happened in the middle of a puppet run, maybe | 01:40 |
| dmsimard | ianw: oh, I must have missed that ping | 01:40 |
| ianw | i think i can re-trigger it | 01:40 |
| *** bobh has joined #openstack-infra | 01:42 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Add an experimental multinode job https://review.openstack.org/505464 | 01:42 |
| dmsimard | ianw: I'll rebase the multinode stack take the opportunity to squash the rdo repository fix in it, since I'm fixing the ip6tables thing | 01:43 |
| *** yamahata has quit IRC | 01:46 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up firewalls https://review.openstack.org/504553 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up hosts file https://review.openstack.org/504552 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up firewalls https://review.openstack.org/504553 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 01:49 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 01:49 |
| *** dargains has quit IRC | 01:52 | |
| *** dargains has joined #openstack-infra | 01:53 | |
| dmsimard | jeblair: yeah.. there's definitely something going on, tox-linters is failing on totally unrelated patches with missing roles | 01:53 |
| dmsimard | wait, hang on, this is a warning from Ansible 2.4 ?? | 01:54 |
| dmsimard | 2017-09-20 01:51:45.771927 | ubuntu-xenial | [DEPRECATION WARNING]: 'include' for playbook includes. You should use | 01:54 |
| *** slaweq has joined #openstack-infra | 01:54 | |
| dmsimard | 2017-09-20 01:51:45.772055 | ubuntu-xenial | 'import_playbook' instead. This feature will be removed in version 2.8. | 01:54 |
| dmsimard | infra-root: ^ did we unknowingly upgrade ansible on the zuul node s? | 01:54 |
| *** hongbin has joined #openstack-infra | 01:55 | |
| dmsimard | https://github.com/openstack-infra/zuul/commit/4203ff46aaecc49ea575887b55a65377b5b31ecc#diff-b4ef698db8ca845e5845c4618278f29a was meant to prevent that and I just tested it locally, ansible 2.3.2.0 gets installed with that | 01:56 |
| *** dargains has quit IRC | 01:57 | |
| openstackgerrit | zhangyangyang proposed openstack-dev/pbr master: Remove py26 support https://review.openstack.org/504843 | 01:58 |
| *** slaweq has quit IRC | 01:58 | |
| *** kukacz has quit IRC | 02:00 | |
| ianw | dmsimard: wait, is it 0.14.3 or 0.14.3rc1 ? | 02:01 |
| dmsimard | ianw: 0.14.3rc1 passed, it's 0.14.3 that failed | 02:01 |
| *** kukacz_ has joined #openstack-infra | 02:01 | |
| dmsimard | infra-root: ok the ansible 2.4 is (sort of) a false alarm, it looks like ansible-lint is picking up 2.4 for us and therefore exploding the job. I'll send a patch. | 02:02 |
| ianw | oh, it would help if i typed 0.14.3 instead of 0.4.3 | 02:03 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: networking-midonet: Add an experimental multinode job https://review.openstack.org/505464 | 02:04 |
| *** bobh has quit IRC | 02:04 | |
| dmsimard | ianw: sorry, the next 1.0.0 version will be easier to type :p | 02:05 |
| ianw | dmsimard: ok, so it's signed now | 02:06 |
| ianw | http://logs.openstack.org/89/89289c511c8c48c8409dd1fb997fc24e64d1dae6/release/ara-tarball-signing/f812ce9/console.html | 02:06 |
| ianw | for my own future reference, the incantation for this was : sudo /opt/zuul/tools/trigger-job.py --url=http://zm08.openstack.org/p --job ara-tarball-signing --project openstack/ara --pipeline release --newrev 89289c511c8c48c8409dd1fb997fc24e64d1dae6 --refname 0.14.3 --logpath /89/89289c511c8c48c8409dd1fb997fc24e64d1dae6/release/ara-tarball-signing/f812ce9/ | 02:07 |
| dmsimard | \o/ | 02:07 |
| ianw | then spin three times, throw salt over your shoulder and sacrifice a chicken | 02:07 |
| dmsimard | quite the incantation inded | 02:08 |
| dmsimard | part of the complexity was that you had to link it to the parent job I guess ? | 02:08 |
| ianw | dmsimard: no, it's just wgetting the file and signing it | 02:09 |
| *** liusheng has joined #openstack-infra | 02:09 | |
| ianw | which was already there since the prior job passed | 02:09 |
| *** tinwood has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-infra | 02:12 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Pin ansible<2.4 in test-requirements for ansible-lint https://review.openstack.org/505467 | 02:12 |
| *** pahuang has quit IRC | 02:15 | |
| *** slaweq has joined #openstack-infra | 02:15 | |
| dmsimard | infra-root ^ fix for broken tox-linters gate job | 02:15 |
| * jhesketh returns | 02:19 | |
| *** dave-mccowan has joined #openstack-infra | 02:19 | |
| dmsimard | ianw: gerrit still feels a little bit sluggish :( | 02:19 |
| ianw | dmsimard: nothing looks terribly out of spec on the memory/cpu front | 02:21 |
| *** dave-mcc_ has joined #openstack-infra | 02:21 | |
| *** slaweq has quit IRC | 02:21 | |
| dmsimard | submitting an online GUI quick edit took a long time with the yellow "working" at the top, it never really finished and I eventually got the yellow notification at the bottom right to say I had done an update on the review | 02:21 |
| clarkb | ya thats a known issue | 02:21 |
| dmsimard | several long seconds later there was the openstackgerrit notification on the channel | 02:21 |
| clarkb | affects gertty in a weird way too | 02:21 |
| clarkb | fungi put it on etherpad iirc | 02:22 |
| dmsimard | clarkb: oh let me re-read through the pad, I'm surely out of date | 02:22 |
| mordred | look it's a jhesketh ! | 02:22 |
| jhesketh | o/ | 02:22 |
| jhesketh | ianw: I think we can EOL the branches before your patch (looking at the comment you left on mordred's) | 02:23 |
| *** dave-mccowan has quit IRC | 02:23 | |
| jhesketh | ianw, tonyb: in that ^ case, any objections to me processing the EOL list now? | 02:23 |
| ianw | jhesketh: actually, yeah, since the regex is still there, so ++ from me fwiw | 02:23 |
| *** Sukhdev has quit IRC | 02:24 | |
| jhesketh | ianw: comment on your patch fyi | 02:26 |
| jhesketh | I'll go ahead with the EOL now then | 02:27 |
| jhesketh | fungi, clarkb: ^ | 02:27 |
| dmsimard | mordred: is it safe to start testing deployment projects with the migration script as of https://review.openstack.org/#/c/505380/ ? | 02:27 |
| *** lbragstad has joined #openstack-infra | 02:27 | |
| dmsimard | (looks like the tip of that tree is failing) | 02:27 |
| *** pahuang has joined #openstack-infra | 02:28 | |
| jhesketh | tonyb: you've got some different eol tags, I can't see how that wouldn't be deliberate so I'm going to use those, but I'm curious why? | 02:28 |
| *** bobh has joined #openstack-infra | 02:30 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Pin ansible<2.4 in test-requirements for ansible-lint https://review.openstack.org/505467 | 02:31 |
| dmsimard | zuul-sphinx zuul.d fix has two +2's, looking for someone else to kick the can :) https://review.openstack.org/#/c/504797/ | 02:37 |
| *** dave-mcc_ has quit IRC | 02:38 | |
| *** csomerville has quit IRC | 02:46 | |
| openstackgerrit | zhangbailin proposed openstack-infra/tripleo-ci master: Update document url and fix a spelling error https://review.openstack.org/505475 | 02:47 |
| *** jascott1 has quit IRC | 02:48 | |
| *** jascott1 has joined #openstack-infra | 02:49 | |
| *** jascott1 has quit IRC | 02:53 | |
| *** slaweq has joined #openstack-infra | 02:53 | |
| *** aeng has joined #openstack-infra | 02:56 | |
| *** slaweq has quit IRC | 02:59 | |
| *** namnh has joined #openstack-infra | 03:11 | |
| *** sshnaidm has joined #openstack-infra | 03:13 | |
| prometheanfire | things just slow? | 03:14 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Support zuul.d directories https://review.openstack.org/504797 | 03:14 |
| clarkb | prometheanfire: can you be more specific? | 03:15 |
| *** rwsu has joined #openstack-infra | 03:16 | |
| prometheanfire | ya, that wasn't that helpful, just seems like gate is backed up, I should probably just look at the status page... | 03:18 |
| *** cody-somerville has joined #openstack-infra | 03:28 | |
| *** cody-somerville has joined #openstack-infra | 03:28 | |
| clarkb | there was a backlog after the zuul restart to fix post jobs | 03:28 |
| clarkb | but we should mostly be through that now | 03:28 |
| prometheanfire | ya, I imagine :D | 03:28 |
| *** csomerville has joined #openstack-infra | 03:30 | |
| *** thorst has quit IRC | 03:30 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 03:31 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 03:31 |
| *** pahuang has quit IRC | 03:32 | |
| *** baoli has quit IRC | 03:32 | |
| *** cody-somerville has quit IRC | 03:33 | |
| *** baoli has joined #openstack-infra | 03:33 | |
| *** yolanda has quit IRC | 03:34 | |
| *** yolanda has joined #openstack-infra | 03:34 | |
| *** slaweq has joined #openstack-infra | 03:37 | |
| *** dangers_away has joined #openstack-infra | 03:39 | |
| *** slaweq has quit IRC | 03:42 | |
| *** pahuang has joined #openstack-infra | 03:46 | |
| *** sshnaidm has quit IRC | 03:46 | |
| *** bobh has quit IRC | 03:47 | |
| *** slaweq has joined #openstack-infra | 03:50 | |
| *** tpsilva has quit IRC | 03:53 | |
| *** coolsvap has joined #openstack-infra | 03:53 | |
| *** slaweq has quit IRC | 03:54 | |
| *** ykarel has joined #openstack-infra | 03:57 | |
| *** Sree has joined #openstack-infra | 04:00 | |
| *** ramishra has quit IRC | 04:01 | |
| *** slaweq has joined #openstack-infra | 04:02 | |
| *** hongbin has quit IRC | 04:04 | |
| *** gildub_ has joined #openstack-infra | 04:05 | |
| *** slaweq has quit IRC | 04:07 | |
| *** ykarel_ has joined #openstack-infra | 04:08 | |
| *** baoli has quit IRC | 04:08 | |
| *** ykarel has quit IRC | 04:10 | |
| *** slaweq has joined #openstack-infra | 04:15 | |
| *** slaweq has quit IRC | 04:21 | |
| *** gouthamr has quit IRC | 04:21 | |
| *** markvoelker has quit IRC | 04:22 | |
| *** markvoelker has joined #openstack-infra | 04:22 | |
| *** lbragstad has quit IRC | 04:23 | |
| *** markvoelker has quit IRC | 04:27 | |
| *** thorst has joined #openstack-infra | 04:31 | |
| *** thorst has quit IRC | 04:35 | |
| *** lbragstad has joined #openstack-infra | 04:39 | |
| *** links has joined #openstack-infra | 04:39 | |
| *** pvaneck has joined #openstack-infra | 04:40 | |
| *** ramishra has joined #openstack-infra | 04:40 | |
| *** links has quit IRC | 04:41 | |
| *** xarses has quit IRC | 04:43 | |
| *** xarses has joined #openstack-infra | 04:44 | |
| *** pvaneck has quit IRC | 04:44 | |
| *** xarses has quit IRC | 04:45 | |
| *** sbezverk has joined #openstack-infra | 04:45 | |
| *** xarses has joined #openstack-infra | 04:45 | |
| *** stakeda has joined #openstack-infra | 04:47 | |
| *** bcat_ has quit IRC | 04:48 | |
| *** xarses has quit IRC | 04:50 | |
| *** xarses has joined #openstack-infra | 04:51 | |
| *** zhurong has quit IRC | 04:53 | |
| *** links has joined #openstack-infra | 04:59 | |
| *** links has quit IRC | 05:00 | |
| *** jaosorior has joined #openstack-infra | 05:01 | |
| *** jtomasek has joined #openstack-infra | 05:09 | |
| *** csomerville has quit IRC | 05:11 | |
| *** jtomasek has quit IRC | 05:12 | |
| *** lbragstad has quit IRC | 05:13 | |
| *** links has joined #openstack-infra | 05:16 | |
| *** links has quit IRC | 05:17 | |
| *** rossella_s has quit IRC | 05:21 | |
| *** rossella_s has joined #openstack-infra | 05:23 | |
| *** udesale has joined #openstack-infra | 05:23 | |
| *** aeng has quit IRC | 05:30 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Only inject cloud-init in CentOS 7.3 https://review.openstack.org/504850 | 05:30 |
| *** aeng has joined #openstack-infra | 05:30 | |
| *** thorst has joined #openstack-infra | 05:31 | |
| *** pahuang has quit IRC | 05:32 | |
| *** gcb has joined #openstack-infra | 05:33 | |
| *** claudiub has joined #openstack-infra | 05:35 | |
| *** thorst has quit IRC | 05:36 | |
| *** cody-somerville has joined #openstack-infra | 05:36 | |
| *** cody-somerville has joined #openstack-infra | 05:36 | |
| *** psachin has joined #openstack-infra | 05:38 | |
| *** mat128 has quit IRC | 05:42 | |
| *** ykarel_ has quit IRC | 05:44 | |
| *** ykarel has joined #openstack-infra | 05:45 | |
| *** nmathew has joined #openstack-infra | 05:45 | |
| *** pahuang has joined #openstack-infra | 05:46 | |
| *** gildub_ has quit IRC | 05:51 | |
| *** jascott1 has joined #openstack-infra | 05:54 | |
| *** slaweq has joined #openstack-infra | 05:58 | |
| *** slaweq has quit IRC | 06:04 | |
| *** ykarel_ has joined #openstack-infra | 06:04 | |
| *** ykarel has quit IRC | 06:07 | |
| *** udesale has quit IRC | 06:16 | |
| *** slaweq has joined #openstack-infra | 06:20 | |
| *** pvaneck has joined #openstack-infra | 06:22 | |
| *** markvoelker has joined #openstack-infra | 06:23 | |
| *** e0ne has joined #openstack-infra | 06:23 | |
| *** pvaneck has quit IRC | 06:23 | |
| *** slaweq has quit IRC | 06:25 | |
| *** rlandy|bbl is now known as rlandy | 06:26 | |
| *** makowals has joined #openstack-infra | 06:26 | |
| *** yolanda has quit IRC | 06:27 | |
| *** yolanda has joined #openstack-infra | 06:29 | |
| *** slaweq has joined #openstack-infra | 06:30 | |
| *** thorst has joined #openstack-infra | 06:32 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config master: Skip bifrost tinyipa jobs on xenial newton https://review.openstack.org/500773 | 06:34 |
| *** slaweq_ has joined #openstack-infra | 06:34 | |
| *** cody-somerville has quit IRC | 06:35 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Add mitaka/trusty filters for projects wanting that branch https://review.openstack.org/505462 | 06:35 |
| *** thorst has quit IRC | 06:36 | |
| *** slaweq has quit IRC | 06:37 | |
| *** rcernin has joined #openstack-infra | 06:43 | |
| yolanda | AJaeger, do your custom dashboard work for you after the gerrit upgrade? | 06:44 |
| *** florianf has joined #openstack-infra | 06:45 | |
| *** pahuang has quit IRC | 06:45 | |
| *** zhurong has joined #openstack-infra | 06:45 | |
| *** andreas_s has joined #openstack-infra | 06:48 | |
| openstackgerrit | Merged openstack-infra/project-config master: add whereto to gerritbot for openstack-doc https://review.openstack.org/505314 | 06:49 |
| *** slaweq has joined #openstack-infra | 06:52 | |
| *** pahuang has joined #openstack-infra | 06:53 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add devstack job for Cyborg https://review.openstack.org/504141 | 06:54 |
| *** cody-somerville has joined #openstack-infra | 06:55 | |
| *** udesale has joined #openstack-infra | 06:56 | |
| *** ykarel__ has joined #openstack-infra | 06:57 | |
| *** e0ne has quit IRC | 06:57 | |
| *** markvoelker has quit IRC | 06:57 | |
| AJaeger | yolanda: check http://git.openstack.org/cgit/openstack/gerrit-dash-creator/ - sdague merged a few changes to fix dashboards. Mine is simply enough and didn't need changes | 06:57 |
| *** slaweq has quit IRC | 06:57 | |
| *** ykarel_ has quit IRC | 06:59 | |
| *** cody-somerville has quit IRC | 07:02 | |
| *** pcaruana has joined #openstack-infra | 07:02 | |
| *** shardy has joined #openstack-infra | 07:02 | |
| *** openstackgerrit has quit IRC | 07:02 | |
| *** slaweq has joined #openstack-infra | 07:03 | |
| *** dhajare has joined #openstack-infra | 07:03 | |
| *** tesseract has joined #openstack-infra | 07:05 | |
| *** slaweq has quit IRC | 07:09 | |
| *** armax has quit IRC | 07:15 | |
| *** dulek has joined #openstack-infra | 07:15 | |
| *** pgadiya has joined #openstack-infra | 07:18 | |
| AJaeger | jhesketh: thanks for EOLing those branches. Do you want to followup to tonyb 's mail with "DONE"? | 07:18 |
| jhesketh | AJaeger: yep, I am still going at the moment though | 07:21 |
| jhesketh | and need to collate any problems etc | 07:21 |
| AJaeger | jhesketh: I see... | 07:21 |
| *** ykarel__ is now known as ykarel | 07:25 | |
| *** yamahata has joined #openstack-infra | 07:25 | |
| *** ccamacho has joined #openstack-infra | 07:27 | |
| *** ralonsoh has joined #openstack-infra | 07:28 | |
| *** openstackgerrit has joined #openstack-infra | 07:29 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Branch cleanup https://review.openstack.org/505516 | 07:29 |
| *** threestrands has quit IRC | 07:29 | |
| *** thorst has joined #openstack-infra | 07:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Branch cleanup (1/n) https://review.openstack.org/505516 | 07:33 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Branch cleanup (2/n): pbr https://review.openstack.org/505518 | 07:33 |
| AJaeger | jhesketh, ianw, some additional branch cleanups ^ | 07:33 |
| AJaeger | there's more to do if anybody wants - will allow us to convert less jobs over... | 07:34 |
| odyssey4me | :( it appears that gerrit is dead? | 07:34 |
| AJaeger | odyssey4me: seems to work here | 07:35 |
| odyssey4me | hmm, let me try resetting my connection | 07:35 |
| yolanda | mm AJaeger , i thought i was using same as you | 07:35 |
| AJaeger | Ah, that one - I'm not really using that currently ;( | 07:36 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: WIP: Branch cleanup (3/n) https://review.openstack.org/505520 | 07:39 |
| *** thorst has quit IRC | 07:39 | |
| *** hashar has joined #openstack-infra | 07:40 | |
| *** jascott1 has quit IRC | 07:42 | |
| *** jpena|off has joined #openstack-infra | 07:45 | |
| *** jpena|off is now known as jpena | 07:45 | |
| *** amoralej has joined #openstack-infra | 07:46 | |
| *** zhurong has quit IRC | 07:48 | |
| openstackgerrit | lu.li proposed openstack/os-testr master: Update "indentify" to "identify" in comments https://review.openstack.org/505524 | 07:49 |
| *** martinkopec has joined #openstack-infra | 07:52 | |
| *** markvoelker has joined #openstack-infra | 07:54 | |
| *** jaosorior has quit IRC | 07:55 | |
| *** jpich has joined #openstack-infra | 07:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openSUSE Tumbleweed support to the unbound element https://review.openstack.org/504168 | 08:00 |
| *** alexchadin has joined #openstack-infra | 08:02 | |
| *** jtomasek has joined #openstack-infra | 08:05 | |
| *** yamahata has quit IRC | 08:06 | |
| *** slaweq has joined #openstack-infra | 08:09 | |
| *** acoles has joined #openstack-infra | 08:11 | |
| *** slaweq has quit IRC | 08:13 | |
| *** dizquierdo has joined #openstack-infra | 08:16 | |
| *** dizquierdo has quit IRC | 08:17 | |
| *** bexelbie has left #openstack-infra | 08:17 | |
| *** alexchadin has quit IRC | 08:18 | |
| *** jaosorior has joined #openstack-infra | 08:19 | |
| *** slaweq has joined #openstack-infra | 08:19 | |
| *** alexchadin has joined #openstack-infra | 08:23 | |
| *** slaweq has quit IRC | 08:24 | |
| amoralej | ianw, we need https://review.openstack.org/#/c/500212/ for centos-7.4, could we get a new tagged release that includes it? | 08:24 |
| *** acoles has left #openstack-infra | 08:25 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Switch chair for RPM packaging https://review.openstack.org/504865 | 08:27 |
| *** markvoelker has quit IRC | 08:28 | |
| *** bogdando has joined #openstack-infra | 08:32 | |
| bogdando | o/ Can we merge https://review.openstack.org/#/c/498766 ? | 08:32 |
| *** bogdando has left #openstack-infra | 08:32 | |
| *** efoley has joined #openstack-infra | 08:33 | |
| *** yamamoto_ has quit IRC | 08:33 | |
| *** thorst has joined #openstack-infra | 08:35 | |
| *** egonzalez has joined #openstack-infra | 08:35 | |
| *** yamamoto has joined #openstack-infra | 08:39 | |
| *** slaweq has joined #openstack-infra | 08:39 | |
| *** liujiong_lj has joined #openstack-infra | 08:41 | |
| *** liujiong has quit IRC | 08:43 | |
| *** zhurong has joined #openstack-infra | 08:44 | |
| *** slaweq has quit IRC | 08:45 | |
| *** liujiong_lj is now known as liujiong | 08:47 | |
| *** derekh has joined #openstack-infra | 08:48 | |
| *** yamamoto has quit IRC | 08:52 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 08:52 | |
| *** yamamoto has joined #openstack-infra | 08:54 | |
| *** pbourke has quit IRC | 08:54 | |
| *** pbourke has joined #openstack-infra | 08:55 | |
| *** slaweq_ has quit IRC | 09:00 | |
| *** martinkopec has quit IRC | 09:00 | |
| *** slaweq has joined #openstack-infra | 09:01 | |
| ianw | amoralej: umm, sure, tomorrow my time though? | 09:01 |
| amoralej | ianw, ok | 09:01 |
| amoralej | thanks | 09:01 |
| *** slaweq_ has joined #openstack-infra | 09:02 | |
| *** slaweq has quit IRC | 09:02 | |
| ianw | or if yolanda has time, that's cool too | 09:02 |
| *** slaweq has joined #openstack-infra | 09:02 | |
| *** thorst has quit IRC | 09:02 | |
| *** slaweq has quit IRC | 09:06 | |
| *** dargains has joined #openstack-infra | 09:08 | |
| *** slaweq has joined #openstack-infra | 09:12 | |
| amoralej | are jobs in post pipelines working now? | 09:15 |
| amoralej | or still working on it? | 09:15 |
| amoralej | ianw, yolanda ^ | 09:16 |
| yolanda | sorry, quite busy during the morning, let's see if i can have some time in the afternoon, or if not, i'll report to ianw to do it tomorrow | 09:17 |
| *** slaweq has quit IRC | 09:18 | |
| *** tosky has joined #openstack-infra | 09:21 | |
| *** markvoelker has joined #openstack-infra | 09:25 | |
| ianw | amoralej: should be wokring | 09:26 |
| amoralej | thx | 09:27 |
| *** makowals has quit IRC | 09:27 | |
| *** sambetts|afk is now known as sambetts | 09:28 | |
| *** makowals has joined #openstack-infra | 09:28 | |
| *** efoley has quit IRC | 09:30 | |
| *** slaweq has joined #openstack-infra | 09:34 | |
| *** e0ne has joined #openstack-infra | 09:34 | |
| *** electrofelix has joined #openstack-infra | 09:34 | |
| *** stakeda has quit IRC | 09:37 | |
| *** slaweq has quit IRC | 09:39 | |
| openstackgerrit | Marios Andreou proposed openstack-infra/elastic-recheck master: Add check for failing tripleo post deploy validation (pingtest) https://review.openstack.org/505574 | 09:42 |
| *** jascott1 has joined #openstack-infra | 09:43 | |
| *** slaweq has joined #openstack-infra | 09:44 | |
| *** jascott1 has quit IRC | 09:47 | |
| *** slaweq has quit IRC | 09:50 | |
| *** dtantsur|afk is now known as dtantsur | 09:50 | |
| *** hashar is now known as hasharAway | 09:52 | |
| *** gongysh has joined #openstack-infra | 09:53 | |
| gongysh | is review.openstack.org down? | 09:53 |
| frickler | gongysh: not for me. it seems there were some issues seen when using gertty, though | 09:57 |
| *** markvoelker has quit IRC | 09:57 | |
| *** udesale has quit IRC | 10:01 | |
| *** claudiub|2 has joined #openstack-infra | 10:01 | |
| *** toabctl has quit IRC | 10:01 | |
| *** claudiub has quit IRC | 10:04 | |
| *** dargains has quit IRC | 10:04 | |
| *** alexchadin has quit IRC | 10:06 | |
| *** sdague has joined #openstack-infra | 10:14 | |
| *** namnh has quit IRC | 10:16 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/505148 | 10:17 |
| *** LindaWang has quit IRC | 10:18 | |
| *** cuongnv has quit IRC | 10:22 | |
| *** zhurong has quit IRC | 10:22 | |
| *** Sree has quit IRC | 10:25 | |
| *** nicolasbock has joined #openstack-infra | 10:42 | |
| *** jkilpatr has quit IRC | 10:44 | |
| andreaf | gmann: did you see my reply on your comment on https://review.openstack.org/#/c/498886? what do you think? | 10:44 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Initial skeleton for devstack-tempest base job https://review.openstack.org/504246 | 10:45 |
| *** rossella_s has quit IRC | 10:45 | |
| *** yamamoto has quit IRC | 10:46 | |
| *** nicolasbock has quit IRC | 10:46 | |
| *** rossella_s has joined #openstack-infra | 10:47 | |
| *** slaweq has joined #openstack-infra | 10:48 | |
| *** ykarel is now known as ykarel|afk | 10:49 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Branch cleanup (3/n) https://review.openstack.org/505520 | 10:50 |
| *** yamamoto has joined #openstack-infra | 10:51 | |
| frickler | hmm, openstackrecheck also seems to be lagging far behind, is this related to the email issue? like https://review.openstack.org/504630 failed in gate at 03:09, but that was reported in #os-qa at 10:05, long after a new version had been merged | 10:54 |
| *** slaweq has quit IRC | 10:55 | |
| *** markvoelker has joined #openstack-infra | 10:55 | |
| *** martinkopec has joined #openstack-infra | 10:55 | |
| *** gongysh has quit IRC | 10:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Branch cleanup (3/n) https://review.openstack.org/505520 | 10:57 |
| *** cshastri has quit IRC | 10:58 | |
| *** thorst has joined #openstack-infra | 10:59 | |
| *** nicolasbock has joined #openstack-infra | 10:59 | |
| *** LindaWang has joined #openstack-infra | 11:00 | |
| *** efoley has joined #openstack-infra | 11:04 | |
| *** thorst has quit IRC | 11:04 | |
| AJaeger | team, mordred, regarding branch cleanup, please review https://review.openstack.org/505462 , https://review.openstack.org/505516 https://review.openstack.org/505518 https://review.openstack.org/#/c/505520 | 11:04 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/release-tools master: Fix options to allow --warn-exit in eol-branch.sh https://review.openstack.org/505593 | 11:05 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/release-tools master: Set up gerrit remote manually in eol_branch.sh https://review.openstack.org/505594 | 11:05 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/release-tools master: Check the gerrit remote is as expected, eol_branch https://review.openstack.org/505595 | 11:05 |
| *** liujiong has quit IRC | 11:06 | |
| *** martinkopec has quit IRC | 11:06 | |
| *** inc0 has quit IRC | 11:11 | |
| *** rossella_s has quit IRC | 11:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Skip bifrost tinyipa jobs on xenial newton https://review.openstack.org/500773 | 11:12 |
| andreaf | jeblair, mordred, ianw, yolanda, AJaeger: I'm looking for a simple way to reproduce the environment setup by native zuul v3 nodes - either holding a node from CI or something like running a local zuulv3 with vagrant to provision the node | 11:12 |
| *** rossella_s has joined #openstack-infra | 11:13 | |
| andreaf | I need to refine the pre/post bits of the devstack-tempest job (pulling the right logs, fixing subunit bits etc) and testing that via CI changes only is just to slow :) | 11:14 |
| andreaf | I guess I could use vagrant + ansible and write a play that would invoke what zuul would but I'd rather not re-invent the wheel :) | 11:15 |
| *** dave-mccowan has joined #openstack-infra | 11:16 | |
| *** jkilpatr has joined #openstack-infra | 11:16 | |
| *** toabctl has joined #openstack-infra | 11:17 | |
| AJaeger | andreaf: sorry, can't help here myself. You always can build the nodepool images locally from project-config/tools/build-image.sh - but I don't know about specific zuul v3 setup | 11:17 |
| *** yolanda has quit IRC | 11:18 | |
| *** dizquierdo has joined #openstack-infra | 11:19 | |
| *** martinkopec has joined #openstack-infra | 11:19 | |
| andreaf | AJaeger: thanks - having the exact same image is certainly a plus, but basically I'm looking for a way to run a job as defined in .zuul.yaml without having to re-invent what zuul already does | 11:19 |
| andreaf | I'm looking at zuul code for inspiration :) | 11:19 |
| *** slaweq has joined #openstack-infra | 11:20 | |
| *** yolanda has joined #openstack-infra | 11:24 | |
| *** slaweq has quit IRC | 11:26 | |
| *** markvoelker has quit IRC | 11:28 | |
| *** nmathew has quit IRC | 11:34 | |
| *** kong has left #openstack-infra | 11:36 | |
| *** hasharAway is now known as hashar | 11:39 | |
| *** slaweq has joined #openstack-infra | 11:43 | |
| *** rossella_s has quit IRC | 11:43 | |
| *** tosky_ has joined #openstack-infra | 11:44 | |
| *** tosky has quit IRC | 11:45 | |
| *** rossella_s has joined #openstack-infra | 11:47 | |
| *** alexchadin has joined #openstack-infra | 11:47 | |
| *** slaweq has quit IRC | 11:47 | |
| *** rossella_s has quit IRC | 11:51 | |
| *** bobh has joined #openstack-infra | 11:52 | |
| *** Sree has joined #openstack-infra | 11:52 | |
| *** slaweq has joined #openstack-infra | 11:53 | |
| *** rossella_s has joined #openstack-infra | 11:53 | |
| *** Sree has quit IRC | 11:56 | |
| *** thorst has joined #openstack-infra | 11:57 | |
| *** Sree has joined #openstack-infra | 11:58 | |
| *** slaweq has quit IRC | 11:59 | |
| *** rossella_s has quit IRC | 12:01 | |
| *** rossella_s has joined #openstack-infra | 12:03 | |
| *** jpena is now known as jpena|lunch | 12:04 | |
| *** ykarel|afk is now known as ykarel | 12:06 | |
| *** alexchadin has quit IRC | 12:06 | |
| *** alexchadin has joined #openstack-infra | 12:07 | |
| *** inc0 has joined #openstack-infra | 12:07 | |
| *** bobh has quit IRC | 12:08 | |
| *** rhallisey has joined #openstack-infra | 12:09 | |
| *** dprince has joined #openstack-infra | 12:09 | |
| *** dargains has joined #openstack-infra | 12:10 | |
| *** rossella_s has quit IRC | 12:11 | |
| *** wolverineav has joined #openstack-infra | 12:14 | |
| *** rossella_s has joined #openstack-infra | 12:14 | |
| *** dargains has quit IRC | 12:14 | |
| *** trown|outtypewww is now known as trown | 12:16 | |
| *** tosky_ is now known as tosky | 12:16 | |
| *** markvoelker has joined #openstack-infra | 12:17 | |
| *** markvoelker has quit IRC | 12:21 | |
| *** Goneri has joined #openstack-infra | 12:21 | |
| *** tpsilva has joined #openstack-infra | 12:22 | |
| *** markvoelker has joined #openstack-infra | 12:22 | |
| *** slaweq has joined #openstack-infra | 12:27 | |
| *** dargains has joined #openstack-infra | 12:27 | |
| *** dargains_ has joined #openstack-infra | 12:28 | |
| *** dargains has quit IRC | 12:31 | |
| *** slaweq has quit IRC | 12:32 | |
| *** rlandy has joined #openstack-infra | 12:35 | |
| *** amoralej is now known as amoralej|lunch | 12:36 | |
| *** slaweq has joined #openstack-infra | 12:37 | |
| *** efoley has quit IRC | 12:38 | |
| *** udesale has joined #openstack-infra | 12:39 | |
| *** aarefiev has quit IRC | 12:40 | |
| *** slaweq has quit IRC | 12:42 | |
| *** jcoufal has joined #openstack-infra | 12:47 | |
| *** slaweq has joined #openstack-infra | 12:47 | |
| *** slaweq has quit IRC | 12:52 | |
| panda | pabelanger: is the container proxy up and running ? can we use it, can you tell me the address ? | 12:55 |
| *** mriedem has joined #openstack-infra | 12:56 | |
| *** slaweq has joined #openstack-infra | 12:58 | |
| *** bobh has joined #openstack-infra | 12:59 | |
| *** felipemonteiro has joined #openstack-infra | 13:01 | |
| *** ykarel is now known as ykarel|afk | 13:02 | |
| *** felipemonteiro__ has joined #openstack-infra | 13:02 | |
| *** slaweq has quit IRC | 13:04 | |
| *** jpena|lunch is now known as jpena | 13:04 | |
| *** felipemonteiro has quit IRC | 13:06 | |
| *** ykarel|afk has quit IRC | 13:06 | |
| *** gouthamr has joined #openstack-infra | 13:07 | |
| pabelanger | panda: you can source /etc/ci/mirror_info.sh on the images, here is a list of variables you get: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n84 | 13:07 |
| *** ociuhandu has joined #openstack-infra | 13:08 | |
| *** xyang1 has joined #openstack-infra | 13:08 | |
| *** esberglu has joined #openstack-infra | 13:09 | |
| *** efoley has joined #openstack-infra | 13:09 | |
| panda | pabelanger: ok so I'll have to export http_proxy=NODEPOOL_LXC_IMAGE_PROXY before attempting any pulls. And it's ready to be used ? | 13:11 |
| *** baoli has joined #openstack-infra | 13:15 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Branch cleanup (3/n) https://review.openstack.org/505520 | 13:17 |
| *** efoley has quit IRC | 13:19 | |
| *** slaweq has joined #openstack-infra | 13:19 | |
| *** alexchadin has quit IRC | 13:21 | |
| *** hashar is now known as hasharAway | 13:22 | |
| *** slaweq has quit IRC | 13:24 | |
| *** kgiusti has joined #openstack-infra | 13:25 | |
| *** gongysh has joined #openstack-infra | 13:28 | |
| *** gongysh has quit IRC | 13:28 | |
| *** efoley has joined #openstack-infra | 13:28 | |
| *** slaweq has joined #openstack-infra | 13:30 | |
| *** camunoz has joined #openstack-infra | 13:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle first play in playbook matching no nodes https://review.openstack.org/505626 | 13:31 |
| *** jheroux has joined #openstack-infra | 13:32 | |
| *** ykarel|afk has joined #openstack-infra | 13:32 | |
| *** efoley has quit IRC | 13:33 | |
| pabelanger | panda: no, this is a reverse proxy cache not http proxy. You can see how the openstack-ansible team setup LXC image proxy | 13:35 |
| *** slaweq has quit IRC | 13:36 | |
| kazsh | Ajaeger: could you kindly check my change again ?, I amended it based upon your comment. https://review.openstack.org/#/c/504104/ https://review.openstack.org/#/c/504166/ | 13:37 |
| Zara | mtreinish: I've just rechecked the os-testr patch since it looked like the latest incarnation just had timeouts during tests. | 13:37 |
| *** martinkopec has quit IRC | 13:39 | |
| *** camunoz has quit IRC | 13:39 | |
| *** camunoz has joined #openstack-infra | 13:39 | |
| *** chlong has joined #openstack-infra | 13:40 | |
| *** amoralej|lunch is now known as amoralej | 13:40 | |
| *** efoley has joined #openstack-infra | 13:41 | |
| *** pgadiya has quit IRC | 13:41 | |
| mtreinish | Zara: hmm on py35 it looks like it timed out, but on py27 I'm still chasing down relative import paths: http://logs.openstack.org/25/504525/4/check/gate-storyboard-python27-ubuntu-xenial/8091008/console.html#_2017-09-18_01_16_13_720737 | 13:44 |
| *** jascott1 has joined #openstack-infra | 13:45 | |
| *** alexchadin has joined #openstack-infra | 13:46 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 13:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 13:46 |
| mnaser | i'm doing some maint work and i just found an instance which was created 2 days ago for openstack ci | 13:46 |
| mnaser | does that make sense? | 13:47 |
| mnaser | that it's been created for 2 days (so i assume its stuck or idle) | 13:47 |
| pabelanger | mnaser: possible, we have some vms on nl01 / nl02 (zuulv3) that are hanging around. Once we land a new nodepool.yaml file we'll likely manually clean them up | 13:48 |
| mnaser | okay no worries! | 13:49 |
| *** jascott1 has quit IRC | 13:49 | |
| AJaeger | kazsh: will review at another time - right now we cannot merge any new project creation changes, we first need to fix a few things. | 13:52 |
| AJaeger | kazsh: I will also not +2 unless there's a +1 from the heat PTL | 13:52 |
| *** mat128 has joined #openstack-infra | 13:53 | |
| AJaeger | pabelanger, regarding branch cleanup, could you review https://review.openstack.org/505462 , https://review.openstack.org/505516 https://review.openstack.org/505518 https://review.openstack.org/#/c/505520 , please? | 13:53 |
| kazsh | AJaeger: Thanks for your response, ok will ask him to review it for now :) | 13:53 |
| fungi | mnaser: it's also possible to have a handful of representatives of infrequently-used node types hang around longer since nodepool pre-builds a few to have them on hand for when jobs request them | 13:54 |
| pabelanger | infra-root: https://review.openstack.org/504500/ and https://review.openstack.org/504495/ our the changes needed to have per-hostname nodepool.yaml configuration for each nodepool-launcher. Thoughts? | 13:54 |
| pabelanger | AJaeger: looking | 13:54 |
| *** ykarel|afk is now known as ykarel| | 13:55 | |
| AJaeger | thanks, pabelanger. jhesketh retired all the branches, so we can remove jobs and don't need to convert them ;) | 13:55 |
| *** zhurong has joined #openstack-infra | 13:55 | |
| Zara | mtreinish: oh, sorry, I looked at the py27 logs but I must've searched the wrong thing (I normally search 'FAIL') since I saw a failure but no obvious reason for it. :/ | 13:57 |
| mnaser | fungi its a trusty node so i think that might be why its not so frequently used | 13:57 |
| *** felipemonteiro__ has quit IRC | 13:58 | |
| Zara | actually I have no idea what I did since now I see it | 13:58 |
| *** udesale has quit IRC | 13:59 | |
| Zara | maybe I started searching from the bottom of the page :S | 13:59 |
| mtreinish | Zara: heh, no worries. I'm curious about the py35 job too, so we'll see if that fails again | 14:01 |
| *** efoley has quit IRC | 14:02 | |
| *** efoley has joined #openstack-infra | 14:02 | |
| Zara | cool, I hope it does something really interesting so I can feel better about making gerrit exercise | 14:02 |
| *** slaweq has joined #openstack-infra | 14:04 | |
| *** efoley has quit IRC | 14:04 | |
| *** efoley_ has joined #openstack-infra | 14:04 | |
| *** Sree has quit IRC | 14:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add mitaka/trusty filters for projects wanting that branch https://review.openstack.org/505462 | 14:04 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings master: Remove channel checking. https://review.openstack.org/505656 | 14:07 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings master: JJB meetings are being held in dedicated channel https://review.openstack.org/496314 | 14:07 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 14:07 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 14:07 |
| *** ijw has joined #openstack-infra | 14:08 | |
| *** slaweq has quit IRC | 14:09 | |
| *** armax has joined #openstack-infra | 14:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Branch cleanup (1/n) https://review.openstack.org/505516 | 14:13 |
| *** rbrndt has joined #openstack-infra | 14:14 | |
| *** slaweq has joined #openstack-infra | 14:14 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/bindep master: SUSE: Update openSUSE Tumbleweed references for lsb-release-3.X.X https://review.openstack.org/505657 | 14:14 |
| *** ijw has quit IRC | 14:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Branch cleanup (2/n): pbr https://review.openstack.org/505518 | 14:16 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 14:18 |
| *** slaweq has quit IRC | 14:19 | |
| jeblair | frickler: there are no gerrit issues specific to gertty. the only issue seen in gertty is in editing patchsets, and that is also seen when using the web ui. | 14:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Deal with link-logs macro https://review.openstack.org/505387 | 14:24 |
| tonyb | Now that 485117 has merged to allow projects to host meetings in $project channels what's the best way to ensure that meetbot permissions are correct | 14:24 |
| *** slaweq has joined #openstack-infra | 14:24 | |
| *** pblaho has quit IRC | 14:25 | |
| jeblair | tonyb: change the accessbot config | 14:27 |
| *** apetrich has quit IRC | 14:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove liberty/mitaka job regexes https://review.openstack.org/504964 | 14:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove unmatched single quotes from jenkins jobs https://review.openstack.org/504965 | 14:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Have publiccloud-wg gate on itself, not api-wg https://review.openstack.org/504967 | 14:27 |
| *** apetrich has joined #openstack-infra | 14:27 | |
| mordred | AJaeger, pabelanger, jeblair: ^^ I pulled those three out of the larger stack and rebased on current master - they are the updates/changes to the existing jobs | 14:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mapping file setting to skip jobs from share queues https://review.openstack.org/504966 | 14:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make yaml2ical publication job https://review.openstack.org/504968 | 14:29 |
| mordred | and there's the other stack fixed up to not have the v2 fixes in it | 14:29 |
| *** dangers_away has quit IRC | 14:29 | |
| *** slaweq has quit IRC | 14:29 | |
| jeblair | tonyb: move meetbots to global and remove from the individual channels? | 14:29 |
| tonyb | jeblair: 'cause I don't really grok how that works ... do you mean add the 2 lines @ http://git.openstack.org/cgit/openstack-infra/project-config/tree/accessbot/channels.yaml#n158 to every channel listed there? | 14:29 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 14:30 |
| *** andreas_s has quit IRC | 14:30 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 14:30 |
| tonyb | jeblair: okay let me try that | 14:30 |
| AJaeger | mordred: thanks, I'll review those now - could you review https://review.openstack.org/505520 , please? I'm cleaning up old jobs... | 14:31 |
| jeblair | mordred, AJaeger: wasn't there something about 5 repos still having mitaka branches/jobs? does that affect 504964? | 14:31 |
| pabelanger | +2 on stack | 14:33 |
| mordred | jeblair: that has been taken care of ... | 14:33 |
| tonyb | jeblair: Yeah there are 5 repos that still want mitaka branches | 14:33 |
| AJaeger | jeblair: we merged https://review.openstack.org/505462 | 14:33 |
| mordred | yah. that ^^ | 14:33 |
| AJaeger | jeblair: that takes care of those 5 ^ | 14:33 |
| *** aarefiev has joined #openstack-infra | 14:34 | |
| * tonyb sees a bunch of stable/* branches going away and thanks jhesketh | 14:34 | |
| jeblair | oh gotcha, we have specifc filters now, so we can remove the high-level ones | 14:34 |
| AJaeger | tonyb: did you hear from fuel guys on 7.0 and 8.0 branches? If we could remove those as well, we have even less special cases... | 14:34 |
| mordred | jeblair: yah - and for those projects, expanding the python-jobs project-template so that we can add the variants is, I believe, perfectly fair | 14:34 |
| *** slaweq has joined #openstack-infra | 14:34 | |
| tonyb | AJaeger: My first round only looked for branches that matched release series I didn't look at numeric ones | 14:34 |
| tonyb | AJaeger: that's next week I guess | 14:34 |
| *** dangers has joined #openstack-infra | 14:35 | |
| tonyb | AJaeger: I couldn't see the wood for the trees so picked the easy things first | 14:35 |
| AJaeger | tonyb: ;) | 14:35 |
| mordred | jeblair, pabelanger, AJaeger: if you're still in the reviewing mood, the stack ending with https://review.openstack.org/#/c/504968 would also be nice to land | 14:35 |
| tonyb | AJaeger: once my mirror finishes I'll see what's left but I probably wont action it until next week | 14:36 |
| AJaeger | tonyb: if we can EOL some larger blocks before the Zuul migration, we don't need to move those jobs over. But no worries... | 14:36 |
| AJaeger | tonyb: ok | 14:36 |
| tonyb | AJaeger: Oh rats I think we missed the boat there | 14:36 |
| AJaeger | tonyb: so be it... | 14:37 |
| pabelanger | mordred: I have to run an errand now, but will get back to 504968 when I return | 14:37 |
| tonyb | AJaeger: even if I generated the list today I don't think that it's enough notice | 14:37 |
| mordred | pabelanger: woot thanks! | 14:37 |
| clarkb | jeblair: there is a response to your gerrit bug | 14:37 |
| AJaeger | tonyb: today's branch removals have already a lot ;) | 14:37 |
| clarkb | seems you were likely on the right track? | 14:37 |
| jeblair | clarkb: yah, i guess we need to set our account cache to 32,000 entries? | 14:38 |
| fungi | or clean up the account list somehow? | 14:38 |
| mordred | AJaeger, tonyb: it's ok - we can migrate them and then delete them :) | 14:38 |
| clarkb | seems silly but that is apparently the recommendation | 14:38 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: [accessbot] Default to running a meetbot in all channels https://review.openstack.org/505669 | 14:38 |
| fungi | i think we still have a ton of accounts "synced" from launchpad back in the old openstack-cla group days which never actually used gerrit for anything | 14:39 |
| tonyb | jeblair: like that ^^ ? | 14:39 |
| clarkb | ya I wonder if it ignores inactive accounts | 14:39 |
| jeblair | i mean, if we set it to the number of accounts, it's not so much a cache as an in-memory copy of the database. | 14:39 |
| clarkb | ya thats the silly part, seems like they are saying that is how you have to run if you want email | 14:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 14:40 |
| jeblair | tonyb: almost | 14:40 |
| *** slaweq has quit IRC | 14:40 | |
| *** ihrachys has joined #openstack-infra | 14:41 | |
| * tonyb awaits pointers | 14:42 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route https://review.openstack.org/466561 | 14:42 |
| jeblair | tonyb: left comment | 14:42 |
| fungi | ignoring accounts marked inactive='Y' only cuts the total by 122 anyway | 14:43 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: [accessbot] Default to running a meetbot in all channels https://review.openstack.org/505669 | 14:43 |
| tonyb | jeblair: Thanks, fixed, sorry for the really obvious error | 14:44 |
| *** efoley_ has quit IRC | 14:44 | |
| fungi | if we marked all accounts lacking an openid in account_external_ids we might be able to shave off another couple hundred tops | 14:45 |
| clarkb | fungi: ya we'd have to go through and mark a bunch of them that way. the other stackalytics account, old third party ci, etc | 14:45 |
| clarkb | still relatively small compared to 26k | 14:46 |
| *** jpena is now known as jpena|brb | 14:46 | |
| fungi | if only gerrit logged the date someone most recently authenticated | 14:46 |
| fungi | maybe it does and i just haven't found it | 14:46 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/storyboard master: Bump minimum os-testr requirement to 1.0.0 https://review.openstack.org/504525 | 14:46 |
| mordred | I honestly don't understand why gerrit needs an in-memory cache of that at all - like, what query are they doing to be able to send an email that takes more than 0.001 seconds? | 14:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 14:47 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 14:47 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings master: JJB meetings are being held in dedicated channel https://review.openstack.org/496314 | 14:47 |
| mordred | it's a 32k row table - it's not _large_ | 14:47 |
| jeblair | mordred: nothing complicated, they're just doing it 100,000 times | 14:47 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings master: Remove channel checking. https://review.openstack.org/505656 | 14:48 |
| mordred | they're making 100k queries to the db for each email? | 14:48 |
| jeblair | mordred: yep | 14:48 |
| mordred | jeblair: so you're saying that they don't know how to write software | 14:48 |
| jeblair | mordred: joins are 'foreign' | 14:48 |
| jeblair | i made a sql joke | 14:48 |
| mordred | :) | 14:48 |
| fungi | joins in software ;) | 14:48 |
| mordred | why do I have the feeling this is likely something to do with h2 support or something | 14:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Branch cleanup (4/n): Remove mitaka from regexes https://review.openstack.org/505676 | 14:49 |
| clarkb | it sounds like at least some devs think it is a bug at least | 14:49 |
| mordred | oh that's good | 14:49 |
| fungi | lowest-common-denominator model preparing to put everything in a key-value store? | 14:49 |
| clarkb | so maybe we can treat edwins suggestion more like a worl around for now | 14:49 |
| clarkb | fungi: considering google is already in basically that that is probably the real cause | 14:50 |
| *** efoley_ has joined #openstack-infra | 14:50 | |
| fungi | but yeah, all things considered, with a 48gib heap, a 32k entry cache for accounts is... well... inconsequential? | 14:50 |
| jeblair | fungi: hopefully? i don't think we know the size of the entries | 14:51 |
| fungi | fair point. could be lots of columns with large-allocation data types | 14:51 |
| fungi | 10 columns with an average 256-byte type would be something like 78mb cache at 32k entries | 14:52 |
| fungi | so not tiny, but still only ~0.1% of the heap | 14:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Bump gerrit account caches to 32768 https://review.openstack.org/505677 | 14:53 |
| jeblair | clarkb, fungi, mordred: ^ <shrug> | 14:53 |
| clarkb | https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#cache.name.memoryLimit | 14:54 |
| clarkb | there is memory limit | 14:54 |
| *** pblaho has joined #openstack-infra | 14:54 | |
| *** xarses_ has joined #openstack-infra | 14:54 | |
| jeblair | clarkb: memoryLimit means number of entries for these caches | 14:55 |
| jeblair | (because they are fixed size) | 14:55 |
| jeblair | (this is the value we're changing) | 14:55 |
| clarkb | ah | 14:56 |
| clarkb | in any case we give it a go and have more data to report back to gerrit? | 14:56 |
| *** slaweq has joined #openstack-infra | 14:56 | |
| *** hongbin has joined #openstack-infra | 14:56 | |
| jeblair | may as well :) | 14:56 |
| fungi | trying to infer the difference between memorylimit and disklimit... does it split less-frequently-accessed entries into a disk cache? | 14:57 |
| fungi | the documentation there isn't entirely clear on that point | 14:57 |
| *** efoley_ has quit IRC | 14:57 | |
| *** sbezverk has quit IRC | 14:57 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 14:58 |
| jeblair | fungi: i want to say show-caches didn't have any entries in the disk column for these caches | 14:58 |
| jeblair | (checking now, but it's slow) | 14:58 |
| fungi | okay, so either we haven't enabled disk-backed caching for them or it wouldn't use it regardless | 14:58 |
| jeblair | fungi: ya. even if it were applicable, i assume every email would still need to page in every cache entry | 14:59 |
| jeblair | so maybe faster than the network round-trip to the database? but still redonculous | 14:59 |
| jeblair | (confirmed: there are no 'disk' entries for these caches in the show-caches table) | 15:00 |
| clarkb | re new project creation we just need gerritlib release now? I can work on that once actually awake | 15:01 |
| *** slaweq has quit IRC | 15:01 | |
| *** efoley_ has joined #openstack-infra | 15:02 | |
| *** alexchadin has quit IRC | 15:02 | |
| jeblair | mordred: 504609 +2 with comment | 15:02 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 15:04 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 15:04 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: DNM: Test Ansible 2.4 https://review.openstack.org/505354 | 15:04 |
| *** efoley_ has quit IRC | 15:05 | |
| *** efoley_ has joined #openstack-infra | 15:05 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Be more descriptive when publishing service-types https://review.openstack.org/505679 | 15:05 |
| mordred | jeblair: how's about that ^^?? | 15:05 |
| jeblair | mordred: in 504506 what's the afs secret used for? | 15:05 |
| *** zhurong has quit IRC | 15:05 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Disable action and lookup plugins from 2.4 https://review.openstack.org/505419 | 15:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Port in changes from ansible 2.4 command module https://review.openstack.org/505430 | 15:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Handle first play in playbook matching no nodes https://review.openstack.org/505626 | 15:06 |
| mordred | jeblair: nothing! good catch | 15:06 |
| *** Nil_ has joined #openstack-infra | 15:07 | |
| *** slaweq has joined #openstack-infra | 15:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add specs publication jobs https://review.openstack.org/504506 | 15:08 |
| *** dargains_ has quit IRC | 15:08 | |
| mordred | jeblair: fixed | 15:08 |
| *** dizquierdo has quit IRC | 15:08 | |
| *** slaweq has quit IRC | 15:13 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 15:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove liberty/mitaka job regexes https://review.openstack.org/504964 | 15:15 |
| kfox1111 | is pycrypto depricated now? | 15:17 |
| *** jpena|brb is now known as jpena | 15:18 | |
| *** dave-mcc_ has joined #openstack-infra | 15:18 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove unmatched single quotes from jenkins jobs https://review.openstack.org/504965 | 15:18 |
| clarkb | kfox1111: yes I think cryptography is current focus of python crypto development | 15:19 |
| *** dave-mccowan has quit IRC | 15:19 | |
| *** LindaWang has quit IRC | 15:20 | |
| *** links has joined #openstack-infra | 15:21 | |
| kfox1111 | ok. thanks. we had two ps's going. one that put pycrypto dep back, and one that removed it. so I'll focus on the one that works with it being removed. | 15:21 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 15:21 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 15:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Have publiccloud-wg gate on itself, not api-wg https://review.openstack.org/504967 | 15:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Make a publish-docs-draft base job https://review.openstack.org/504622 | 15:23 |
| openstackgerrit | zhangyangyang proposed openstack-infra/bindep master: Cleanup test-requirements https://review.openstack.org/505688 | 15:23 |
| *** links has quit IRC | 15:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Rework openstack doc publishing to not be overridable https://review.openstack.org/504233 | 15:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Add tag-releases to the mapping https://review.openstack.org/504505 | 15:25 |
| fungi | infra-root: heads up about http://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-9798.html | 15:25 |
| *** yamahata has joined #openstack-infra | 15:25 | |
| fungi | i've confirmed files01 and static have apache2 2.4.7-1ubuntu4.1804.1 installed as of today | 15:26 |
| fungi | which are the only places i can think of people might be able to exploit that via malicious .htaccess files | 15:26 |
| fungi | but keep an eye out | 15:26 |
| jeblair | fungi: thanks (and agreed) | 15:27 |
| electrofelix | anyone to review https://review.openstack.org/475474 for requests-mock, I'd like to use this library but current behaviour is broken converting python-jenkins to requests https://review.openstack.org/#/c/205637/ | 15:27 |
| clarkb | fungi: should be easy to patch at least once packages are available? | 15:27 |
| fungi | clarkb: that's what i meant by 2.4.7-1ubuntu4.1804.1 | 15:28 |
| clarkb | ah | 15:28 |
| fungi | the patched version is installed on them | 15:28 |
| dmsimard | fungi: iirc it's only exploitable under certain conditions | 15:28 |
| clarkb | do we need to restart apache as well? | 15:28 |
| fungi | dmsimard: right, in particular unusual configuration which someone _could_ set in a .htaccess file | 15:28 |
| fungi | clarkb: dpkg took care of it when upgrading | 15:28 |
| *** slaweq_ has quit IRC | 15:29 | |
| clarkb | perfect | 15:29 |
| dmsimard | Yup, not saying this isn't the case for us but just pointing out we're likely unaffected | 15:29 |
| dmsimard | Updating all the time is important anyway :) | 15:29 |
| clarkb | electrofelix: I think you want jamielennox for that who is possibly bavk on an au timezone? | 15:29 |
| fungi | dmsimard: sure, however we let people upload .htaccess files to files01 and static | 15:29 |
| fungi | dmsimard: which is why i mentioned those servers in particular | 15:30 |
| dmsimard | Oh, makes sense | 15:30 |
| electrofelix | clarkb: he's in Oz, ok well guess I've been pinging him at the wrong time for the last few months so | 15:31 |
| *** dhajare has quit IRC | 15:31 | |
| fungi | pretty sure jamielennox never sleeps anyway | 15:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 15:32 |
| *** ijw has joined #openstack-infra | 15:34 | |
| clarkb | we seem yo have a gerrit spike around 1400 today and yesterday though far more pronounced yesterday | 15:35 |
| clarkb | according to melody | 15:35 |
| *** chenying_ has quit IRC | 15:35 | |
| *** jtomasek has quit IRC | 15:36 | |
| *** chenying_ has joined #openstack-infra | 15:36 | |
| AJaeger | team, could I get one review for https://review.openstack.org/#/c/505520 and two for https://review.openstack.org/#/c/505676/ , please? Those are cleanups now that most <= mitaka branches are EOLed | 15:37 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 15:38 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 15:38 |
| *** slaweq has joined #openstack-infra | 15:39 | |
| *** vhosakot has joined #openstack-infra | 15:39 | |
| *** dargains has joined #openstack-infra | 15:42 | |
| *** slaweq has quit IRC | 15:45 | |
| *** leifmadsen has quit IRC | 15:46 | |
| *** leifmadsen has joined #openstack-infra | 15:47 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 15:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 15:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 15:55 |
| clarkb | http://logs.openstack.org/77/505677/1/check/gate-openstackci-beaker-ubuntu-trusty/97b2199/console.html#_2017-09-20_15_36_52_799009 thats a new one | 15:55 |
| *** jascott1 has joined #openstack-infra | 15:55 | |
| clarkb | jeblair: ^ related to your change. I figure recheck and see if consistent? | 15:55 |
| clarkb | AJaeger: ya I'll review shortly | 15:56 |
| AJaeger | thanks, clarkb | 15:56 |
| jeblair | clarkb: i wonder if something changed in the trusty puppet repos? | 15:56 |
| *** Sukhdev_ has joined #openstack-infra | 15:59 | |
| pabelanger | and back | 15:59 |
| anticw | clarkb: thanks! | 15:59 |
| *** jascott1 has quit IRC | 15:59 | |
| *** Jeffrey4l_ has joined #openstack-infra | 16:01 | |
| pabelanger | jeblair: mordred: did we want to push forward with nodepool.yaml specific config per nodepool-launcher? 504500 and 504495 | 16:02 |
| *** Douhet has quit IRC | 16:03 | |
| jeblair | pabelanger: +2 thx | 16:04 |
| *** Rockyg has joined #openstack-infra | 16:04 | |
| *** Douhet has joined #openstack-infra | 16:04 | |
| *** Jeffrey4l has quit IRC | 16:06 | |
| pabelanger | jeblair: Shrews: do we have a proceedure to shutdown a nodepool-launcher? I stopped nl02.o.o last night, but I must not have done it properly, as we have a locked node currently. 0000028986 | 16:06 |
| *** Jeffrey4l has joined #openstack-infra | 16:07 | |
| jeblair | pabelanger: normal init script shutdown should work | 16:07 |
| pabelanger | jeblair: okay cool, that's what I used | 16:07 |
| *** jascott1 has joined #openstack-infra | 16:08 | |
| *** Jeffrey4l_ has quit IRC | 16:08 | |
| *** florianf has quit IRC | 16:08 | |
| *** efoley_ has quit IRC | 16:08 | |
| Shrews | pabelanger: what state? zuul can lock nodes, too | 16:08 |
| pabelanger | Shrews: ready | 16:09 |
| *** jascott1 has quit IRC | 16:09 | |
| *** jascott1 has joined #openstack-infra | 16:09 | |
| jeblair | if zl02 is shut down, it's not possible for it to hold the lock on that node any more | 16:09 |
| Shrews | zl? | 16:09 |
| pabelanger | okay, good to know | 16:10 |
| jeblair | nl | 16:10 |
| *** jascott1 has quit IRC | 16:10 | |
| jeblair | nl01 holds the lock on that node | 16:10 |
| *** jascott1 has joined #openstack-infra | 16:10 | |
| pabelanger | we actually have 8 ready nodes in locked state right now | 16:10 |
| pabelanger | I am also wondering if we should also stop nl01.o.o, release nodes back to clouds then land new configuration for nodepool-launcher | 16:11 |
| jeblair | launcher-debug.log.2017-09-18_21:2017-09-18 22:42:31,971 DEBUG nodepool.driver.openstack.OpenStackNodeRequestHandler: Locked existing node 0000028986 for request 100-0000036366 | 16:12 |
| *** caphrim007 has joined #openstack-infra | 16:12 | |
| Shrews | pabelanger: those locked ready nodes all have the allocated_to field populated | 16:13 |
| Shrews | so they are locked for a particular request | 16:13 |
| jeblair | 100-0000036366 is in a retry loop | 16:13 |
| Shrews | pabelanger: allocation now shows up in 'nodepool list --detail' output | 16:13 |
| jeblair | i'm going to guess that those 2 requests are for 5 node sets | 16:14 |
| pabelanger | Yup, some of the ready nodes that are locked are for 36366 loop | 16:14 |
| jeblair | and we're in the same situation we were in the other day -- they can't be satisfied because those providers have a node sitting in ready or error | 16:15 |
| jeblair | yeah 366 and 377 are both for 5 node sets | 16:15 |
| Shrews | 100-0000036366 is on vexxhost-ca-ymq-1 | 16:15 |
| Shrews | max-servers 5 | 16:16 |
| Shrews | so, yeah | 16:16 |
| pabelanger | oh, interesting | 16:16 |
| Shrews | that pool already has 5 | 16:16 |
| *** dulek has left #openstack-infra | 16:16 | |
| jeblair | the 5th node is a ready opensuse node, it's unusable | 16:16 |
| pabelanger | so, if opensuse node was used and delete, it would launch the 5th for 36366? | 16:17 |
| Shrews | that's an interesting situation. something has to use that opensuse node before the other request can proceed | 16:17 |
| Shrews | pabelanger: yeah | 16:17 |
| jeblair | pabelanger: yep. feel free to delete it now and you will unstick it | 16:17 |
| pabelanger | okay, good to know | 16:17 |
| *** rcernin has quit IRC | 16:17 | |
| pabelanger | deleting now to confirm that fixes it | 16:17 |
| jeblair | so anyway -- the only thing nodepool could do differently here is to delete that node itself. what *we* could do differently is not have our request size and quota size be the same :) | 16:18 |
| Shrews | ++ | 16:18 |
| jeblair | both would probably be good fixes :) | 16:18 |
| pabelanger | Ya, I think we could bump vexxhost up to 10 | 16:18 |
| jeblair | Shrews: i think maybe once the algorithm hits the provider-is-paused stage, it could go looking for ready nodes to kill? | 16:18 |
| pabelanger | there we go | 16:19 |
| pabelanger | unlocked now | 16:19 |
| *** efoley_ has joined #openstack-infra | 16:19 | |
| pabelanger | will do the same with citycloud-kna1 | 16:19 |
| jeblair | pabelanger: thx | 16:19 |
| Shrews | jeblair: yeah. not sure how that would play out with static nodes, but in the current form, should be doable | 16:19 |
| pabelanger | thank you! Now I better understand what is happening :) | 16:20 |
| Shrews | pabelanger: np. the spec explains it all, but takes a while for it all to sink in | 16:20 |
| Shrews | a bit complex | 16:20 |
| jeblair | also, "nodepool request-list" is a new command and really handy :) | 16:20 |
| Shrews | you're welcome :-P | 16:21 |
| pabelanger | Oh right, I forgot about that | 16:21 |
| *** Sree has joined #openstack-infra | 16:22 | |
| pabelanger | Shrews: hows your puppet? Care to review 504495 and 504500, that should be per-launcher nodepool.yaml files | 16:23 |
| pabelanger | well, 1 in puppet, other is nodepool.yaml syntax | 16:23 |
| pabelanger | is* | 16:23 |
| Shrews | jeblair: that's probably an important fix to add to nodepool, now that I think about it. since a pool will be paused, it will never handle another request to get that unusable node out of the picture | 16:24 |
| Shrews | pabelanger: i don't really know puppet, except for the ones on Sesame Street | 16:24 |
| smcginnis | Is it a know issue that gerrit emails are substantially delayed? | 16:24 |
| pabelanger | smcginnis: ya, jeblair has a bug report upstream about it | 16:24 |
| smcginnis | pabelanger: ack, thanks! | 16:24 |
| pabelanger | smcginnis: https://bugs.chromium.org/p/gerrit/issues/detail?id=7261 | 16:25 |
| * smcginnis will watch and keep patient | 16:26 | |
| clarkb | jeblair: re puppet repos changing, maybe? we definitely haven't built new images in a while though so woul dhave to be upstream of our image builds I think | 16:26 |
| pabelanger | seems the current feedback is bump our account cache for gerrit | 16:27 |
| jeblair | clarkb: it hit the issue again | 16:27 |
| clarkb | interesting | 16:28 |
| clarkb | pabelanger: I noticed that we have pinned paramiko <2.0.0 in gerritlib as well due to the compilation requirements. With cryptography providing wheels now I think those are no longer a concern. We might want to do a followup release to unpin that | 16:29 |
| clarkb | infra-root I'm going to push a 0.7.0 tag for gerritlib nowish | 16:29 |
| pabelanger | clarkb: ya, seems like a good idea | 16:29 |
| clarkb | this should fix create-project calls to gerrit and hopefully allow us to create new projects again | 16:29 |
| fungi | clarkb: thanks! | 16:30 |
| jeblair | clarkb, fungi: how does this response look? http://paste.openstack.org/show/621552/ | 16:31 |
| jeblair | fungi: also, it looks like your estimation of size seems correct. | 16:33 |
| *** dizquierdo has joined #openstack-infra | 16:34 | |
| clarkb | tag pushed | 16:34 |
| clarkb | jeblair: ++ | 16:35 |
| clarkb | if a tunable parameter defaults to unusable values and there is basically one valid value it should be as you say | 16:35 |
| *** ociuhandu has quit IRC | 16:36 | |
| *** rlandy is now known as rlandy|brb | 16:38 | |
| *** nicolasbock has quit IRC | 16:39 | |
| *** sbezverk has joined #openstack-infra | 16:40 | |
| pabelanger | +1 | 16:40 |
| clarkb | ok I guess the next thing to look at is why trusty jobs started failing? | 16:41 |
| *** e0ne has quit IRC | 16:42 | |
| clarkb | uhm | 16:42 |
| jeblair | clarkb: ya; i set an auto-hold for the job | 16:42 |
| clarkb | gerritlib has a release announce job? | 16:42 |
| clarkb | did I do an end around around release management? I didn't realize any of our projects were release managed that way? | 16:42 |
| jeblair | clarkb: i think everything does now, but they noop if they aren't configured | 16:42 |
| clarkb | ah ok | 16:42 |
| openstackgerrit | Merged openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 16:42 |
| clarkb | https://pypi.python.org/pypi/gerritlib/ new release is there, puppet should apply to review.o.o on next run and then we can manually run manage-projects again | 16:43 |
| pabelanger | clarkb: when you have a moment, https://review.openstack.org/504495/ and https://review.openstack.org/504500/ nodepoolv3 changes needed for per-launcher configuration | 16:44 |
| *** Sree has quit IRC | 16:44 | |
| *** sambetts is now known as sambetts|afk | 16:44 | |
| *** Sree has joined #openstack-infra | 16:45 | |
| *** shardy has quit IRC | 16:48 | |
| *** felipemonteiro has joined #openstack-infra | 16:48 | |
| *** ijw has quit IRC | 16:48 | |
| clarkb | pabelanger: ya, can you review https://review.openstack.org/#/c/505676/1 for AJaeger? you reviewed the other one in his list | 16:49 |
| *** Sree has quit IRC | 16:50 | |
| *** jpich has quit IRC | 16:50 | |
| *** felipemonteiro__ has joined #openstack-infra | 16:51 | |
| pabelanger | +3 | 16:52 |
| *** Sukhdev_ has quit IRC | 16:52 | |
| *** psachin has quit IRC | 16:53 | |
| dhellmann | clarkb : before I start approving releases, I want to double check that everything is ok with the jobs in those queues. I think I saw that we were all clear, but can you confirm? | 16:53 |
| jeblair | andreaf: another thing you might consider when working on post-playbooks is to temporarily turn the job into a no-op (or just a shell task that touches a log file or something). that can speed things up a lot. | 16:53 |
| clarkb | pabelanger: I've approved the nl changes too | 16:53 |
| clarkb | dhellmann: yes as far as I know zuul is properly handling gerrit event stream data for all queues now. I even just made a gerritlib release and it seemed to work as expected | 16:54 |
| fungi | jeblair: response lgtm, sorry for the delay. as for my size estimate, that was sheer luck trying to figure within a rough order of magnitude | 16:54 |
| clarkb | dhellmann: maybe start with a single one and double check first? | 16:54 |
| dhellmann | clarkb : good idea | 16:54 |
| SamYaple | clarkb: re: trigger dockerhub build via POST. That plan still works, *however* to setup the build in the first place i must be a member of the organization. So to make this work I would need to be a read-only member of github.com/openstack or have another member create the builds | 16:54 |
| SamYaple | so im kind of stuck | 16:54 |
| *** felipemonteiro has quit IRC | 16:54 | |
| clarkb | SamYaple: ugh, that sounds like a dockerhub design flaw | 16:55 |
| jeblair | clarkb: http://logs.openstack.org/77/505677/1/check/gate-openstackci-beaker-ubuntu-trusty/ae47400/console.html had the hiera error; node should be held | 16:55 |
| SamYaple | well its not a design flaw as much as a delibereate decission | 16:55 |
| jeblair | clarkb: 158.69.89.160 | 16:55 |
| clarkb | jeblair: thanks | 16:55 |
| *** slaweq has joined #openstack-infra | 16:56 | |
| SamYaple | whether i agree with it or not, its the way theyve decided to allow automated builds | 16:56 |
| clarkb | mordred: pabelanger you are probable interested in SamYaple's thing above and I think likely have better background on it than I do | 16:56 |
| pabelanger | reading | 16:56 |
| *** hasharAway has quit IRC | 16:57 | |
| *** hashar has joined #openstack-infra | 16:57 | |
| jeblair | clarkb: we don't have the puppetlabs repos on those nodes? | 16:57 |
| clarkb | jeblair: possibly not because we stopped using puppet in dib? | 16:58 |
| SamYaple | quick recap: I want automated builds on dockerhub. i need to either be a member of the github organization (openstack) or have a member of the organziation setup the auotmated builds (initially). I can trigger them from the POST job | 16:58 |
| clarkb | jeblair: I wonder if upstream ubuntu pulled hiera from their repo? | 16:59 |
| *** Apoorva has joined #openstack-infra | 16:59 | |
| pabelanger | clarkb: SamYaple: mordred: reading https://docs.docker.com/docker-hub/builds/ it does seem to indicate we need to have a tigher integration with github then we'd likely want. Meaning, doesn't look like there is anonymous build support | 16:59 |
| SamYaple | pabelanger: we dont need to install the trigger, we can setup a trigger via a POST job | 17:00 |
| pabelanger | clarkb: SamYaple mordred: unless we want to think about creating adding openstack-zuul or another bot to the account | 17:00 |
| SamYaple | but it wont let the automated build work without at least being a member of openstack org | 17:00 |
| *** Apoorva has quit IRC | 17:00 | |
| *** slaweq has quit IRC | 17:00 | |
| pabelanger | right | 17:00 |
| *** srobert has joined #openstack-infra | 17:00 | |
| clarkb | meaning the thing triggering the build must be a member of that org in github? | 17:00 |
| SamYaple | clarkb: actually just setting up the automated build initially | 17:00 |
| *** egonzalez has quit IRC | 17:01 | |
| pabelanger | ya, this looks like opt-in to github things, then you can start builds | 17:01 |
| SamYaple | well you dont even have to opt-in (i mean you can for full integration) | 17:01 |
| clarkb | but the build runs on the dockerhub side? I'm confused as to what exactly needs to happen in github | 17:01 |
| pabelanger | To use automated builds you must have an account on Docker Hub and on the hosted repository provider (GitHub or Bitbucket) | 17:01 |
| SamYaple | but if i was a read-only member of openstack right now, this would all work | 17:01 |
| openstackgerrit | Merged openstack-infra/project-config master: Branch cleanup (3/n) https://review.openstack.org/505520 | 17:01 |
| *** srobert has quit IRC | 17:02 | |
| pabelanger | ya, I don't think we want to add people to openstack project on github | 17:02 |
| pabelanger | all of our tools currently are using APIs | 17:02 |
| clarkb | jeblair: puppet-prepare-node jjb macro is what runs that | 17:02 |
| clarkb | jeblair: I think that is a hold over from when we were preinstalling puppet on the nodes | 17:03 |
| pabelanger | https://docs.docker.com/docker-hub/builds/#link-to-a-hosted-repository-service screenshot is also a little scary. "Administer your repositories" | 17:03 |
| clarkb | jeblair: my hunch is that we can drop that purging entirely now | 17:03 |
| SamYaple | without having at least one read-only member of the openstack org on github, i don't see how this could ever work | 17:03 |
| SamYaple | pabelanger: no, thats not what im talking about. we dont need all that | 17:03 |
| SamYaple | from the github side at least | 17:03 |
| pabelanger | SamYaple: is there another doc I can look at? | 17:03 |
| *** derekh has quit IRC | 17:03 | |
| fungi | SamYaple: i expect this becomes easier when we finish executing the plan to flatten namespaces and give replication control to others outside infra (presumably multiple teams with differing org needs?) | 17:04 |
| SamYaple | pabelanger: that doc has all the information, but we dont need to do all of the bits it says to | 17:04 |
| clarkb | jeblair: if mnaser is ok with it (because puppet-openstack uses it too) I think we should just rm both the yum and apt-get puppet/hiera purge lines | 17:04 |
| pabelanger | SamYaple: we also talked about publising pipeline at PTG with mordred, and think we might have something for zuulv3 in a short a mount of time (pending zuulv3 roll out). Then we might not need a hosted service for builds | 17:04 |
| SamYaple | pabelanger: specifically anything on github, we dont *need* to do | 17:04 |
| *** rlandy|brb is now known as rlandy | 17:04 | |
| *** srobert has joined #openstack-infra | 17:04 | |
| fungi | SamYaple: in the interim, you could probably pilot something similar by setting up a daemon somewhere which pulls from gerrit and pushes into github.org/loci or whatever, and then set up dockerhub builds that way | 17:04 |
| andreaf | jeblair: sure that's a good idea | 17:05 |
| andreaf | jeblair: still I wouldn't mind having a local environment where to test things | 17:05 |
| clarkb | jeblair: let me go ahead and push that change so we have something conrete to look at | 17:05 |
| *** srobert has quit IRC | 17:05 | |
| * mnaser reads buffer | 17:05 | |
| pabelanger | fungi: ya, good idea | 17:05 |
| SamYaple | fungi: yea ive already got that going :( its just such a pain and not something i can give my other team members access to to manage and setup | 17:05 |
| *** srobert has joined #openstack-infra | 17:05 | |
| *** slaweq has joined #openstack-infra | 17:06 | |
| andreaf | jeblair: isn't there a way to make a job defined for .zuul.yaml into something that plain ansible can run? then I could just use vagrant and pass the root play as a provisioning script | 17:06 |
| *** supertakumi86 has joined #openstack-infra | 17:06 | |
| jeblair | andreaf: well, the playbooks are plain ansible playbooks -- you should just be able to make an inventory file and run them (you can copy the generated inventory file from a job and change the host info) | 17:08 |
| SamYaple | if there is no chance of getting even a single read-only member in the openstack org (and nothing beyond that, so no triggers of any kind are needed), then there is nothing that I can do with dockerhub | 17:08 |
| jeblair | andreaf: just skip the base job playbooks (setting up git repos and copying log files) | 17:09 |
| andreaf | jeblair: ok.. but what about running the pre jobs and the post jobs in reverse order? I thought that bit was something zuul would take care to build? | 17:09 |
| mnaser | clarkb just trying to understand, is there an issue with purging things? | 17:09 |
| jeblair | andreaf: yeah, it would be a series of ansible-playbook invocations you'd need to do | 17:09 |
| fungi | SamYaple: longer term, i expect we want to mostly empty or even completely retire github.com/openstack in favor of lots of different orgs under control of various parties, so i'm hesitant to suggest any solution built on the assumption that we mirror our repos there | 17:10 |
| *** slaweq has quit IRC | 17:10 | |
| *** dave-mcc_ has quit IRC | 17:10 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Only remove puppet in jobs if present https://review.openstack.org/505751 | 17:10 |
| clarkb | mnaser: ya see http://logs.openstack.org/77/505677/1/check/gate-openstackci-beaker-ubuntu-trusty/97b2199/console.html#_2017-09-20_15_36_52_799009 my change above is the proposal for working around this problem | 17:11 |
| fungi | SamYaple: but if _you_ mirror your repos to that service, then _you_ can make whatever guarantees you like about it | 17:11 |
| andreaf | jeblair: right I though maybe there is some python API or so in zuul that builds that series of calls, and we could use it to create a root playbook from a job definition in .zuul.yaml | 17:11 |
| jeblair | SamYaple: it may be useful to revisit this in a week or two when mordred is less focused on the zuul v3 migration. | 17:11 |
| *** dave-mccowan has joined #openstack-infra | 17:11 | |
| clarkb | mnaser: there is a possibility that my change somehow breaks all the puppet jobs (in which case I guess we revert and try again) | 17:11 |
| andreaf | jeblair: it would replace the current reproduce.sh :) | 17:11 |
| clarkb | mnaser: monstly want you to be aware so you can provide input too | 17:11 |
| *** trown is now known as trown|lunch | 17:11 | |
| mnaser | clarkb i wonder if that has to do with the fact trusty probably doesnt ship with hiera | 17:11 |
| clarkb | mnaser: the odd thing is that this job has been running for years I thought | 17:12 |
| *** ykarel| has quit IRC | 17:12 | |
| mnaser | i remember making a change | 17:12 |
| mnaser | where if that process failed | 17:12 |
| mnaser | it would pass anyways | 17:12 |
| mnaser | clarkb https://github.com/openstack-infra/project-config/commit/dd6b03b65c7e1e6414a5463185aeb677a7610dde#diff-03d414c17dcd54548b8810c4a442b655 | 17:12 |
| mnaser | thats why | 17:12 |
| jeblair | andreaf: there are a number of complications with that. we'd like to have tools like that, but they haven't been written yet and aren't trival (though they are [mostly] possible) | 17:13 |
| SamYaple | fungi: so there is going to be an github.com/nova/ and a github.com/neutron/ etc? | 17:13 |
| clarkb | mnaser: ah ok, considering that change do you think my proposed change is a good way to address it? | 17:13 |
| clarkb | mnaser: basically check if puppet is there and only uninstall it then | 17:13 |
| mnaser | clarkb: sure, or maybe add a '|| true' to the purge line? i cant imagine an purge/remove operation failing for any other reason that the package missing | 17:14 |
| andreaf | jeblair: I guess an alternative could be to run a local zuul - is it possible in zuul to have a node provider other than nodepool? or to configure nodepool to use a local VM as a fixed node? | 17:14 |
| clarkb | mnaser: I like the if because it makes the rm on ubuntu explicit too (though with -f I think file not existing returns 0) | 17:15 |
| clarkb | infra-root can has review on https://review.openstack.org/#/c/505751/ in an effort to get gerrit cache size increase in? | 17:16 |
| fungi | SamYaple: the most attractive model i've seen for official openstack deliverables which make up part of the openstack "product" is to have openstack-apis and openstack-libs and openstack-tools and so on, and then teams which build things that aren't part of the "product" can mirror into team-specific orgs if they like | 17:16 |
| *** slaweq has joined #openstack-infra | 17:16 | |
| mnaser | clarkb realistically speaking, if we can verify/check that the images will never have puppet in them, can we just drop that macro? | 17:16 |
| pabelanger | Hmm | 17:16 |
| pabelanger | http://logs.openstack.org/00/504500/1/gate/gate-openstackci-beaker-ubuntu-trusty/f949786/console.html#_2017-09-20_17_08_56_570642 | 17:16 |
| mnaser | or the purge part of it at least | 17:17 |
| fungi | SamYaple: and people hosting unofficial repos on our infrastructure can run some of the same mirroring tools somwhere if they like | 17:17 |
| pabelanger | anybody know why we pin pip <8 there^ | 17:17 |
| pabelanger | same with virtualenv | 17:17 |
| fungi | pabelanger: odds are that was a temporary workaround which never got undone. check git history for that line | 17:17 |
| clarkb | mnaser: our image builds will no longer put puppet in them and that should be constant for us at this point. The only reason I kept it around was for potential third party CIers still putting puppet in their images | 17:17 |
| clarkb | mnaser: the chances of that are slim but was being cautious | 17:17 |
| pabelanger | fungi: yup, doing so now | 17:17 |
| pabelanger | also looks like clarkb is working on beaker trusty failures now | 17:18 |
| clarkb | ya fails becaues hiera does not exist | 17:18 |
| *** ianychoi_ has joined #openstack-infra | 17:18 | |
| clarkb | 505751 for the fix | 17:18 |
| mnaser | i mean afaik the only third party ci that i know of for puppet-openstack which is functioning is rdo | 17:18 |
| SamYaple | fungi: man that all sounds far more complicated (and would take alot longer) than adding a hidden read-only user to an existing org | 17:19 |
| mnaser | but i don think they depend on project-config | 17:19 |
| jeblair | andreaf: no, and even if it were, it would be super-complicated; one of the main ideas of zuulv3 is that playbooks are just playbooks, so any tooling we do should be about making it easy to just run them using regular ansible. that's pretty close -- you just need an inventory file, the git repos set up, and maybe an ansible config file to set role paths. | 17:19 |
| SamYaple | fungi: what you described sounds at least 2 cycles away | 17:19 |
| pabelanger | clarkb: +2 | 17:19 |
| *** ociuhandu has joined #openstack-infra | 17:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Branch cleanup (4/n): Remove mitaka from regexes https://review.openstack.org/505676 | 17:19 |
| fungi | SamYaple: yeah, point being many (most) of the infra team don't want responsibility for the official orgs on github and view them as marketing/social media outlets, but while we're still being held responsible for them it's hard to justify granting people special access for unrelated purposes and we certainly wouldn't want to give anyone the impression they'll stick around or stay up to date | 17:21 |
| andreaf | jeblair: ok got it | 17:21 |
| *** ianychoi has quit IRC | 17:22 | |
| *** rbrndt has quit IRC | 17:22 | |
| *** slaweq has quit IRC | 17:22 | |
| clarkb | is there a way to make github membership open to anyone for ready only membership? (I have no idea if that is sane) | 17:22 |
| mnaser | clarkb id really like to just drop the remove.. but i guess you're saying it is possible that there is third party CI | 17:22 |
| mnaser | afaik fuel ci reports failures all the time | 17:22 |
| mnaser | and rdo ci doesnt use this afaik | 17:22 |
| clarkb | mnaser: ya I'm not aware of anyone actually using it, if you prefer removing it entirely I can push a new patchset | 17:23 |
| * clarkb does this | 17:23 | |
| fungi | SamYaple: and quite likely the eventual solution you'll need (which sounds like what you're already doing) is to run a mirroring service yourself to a github org under your control | 17:23 |
| mnaser | clarkb yeah lets do that, it has to be done at some point anyways :> | 17:23 |
| *** ramishra has quit IRC | 17:24 | |
| SamYaple | fungi: i get what you are saying, but surely there is a balance? read-only user with no triggers or anything related rather than having to run a daemon on my infrastructure to keep an unrelated thing up to date | 17:24 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Don't remove puppet in puppet prep macro. https://review.openstack.org/505751 | 17:24 |
| clarkb | mnaser: pabelanger ^ | 17:24 |
| mnaser | i just confirmed that facter/puppet are not included in the latest puppet jobs | 17:25 |
| mnaser | (yum not matching) | 17:25 |
| fungi | SamYaple: really, just don't rely on our github mirroring | 17:25 |
| pabelanger | clarkb: ya, that's sounds right +2 | 17:26 |
| fungi | if you want to rely on a github mirror, i suggest running one yourself | 17:26 |
| mnaser | clarkb lgtm. i'll check p-o-i jobs are okay after | 17:26 |
| mnaser | and if they're not we'll fix them :D | 17:26 |
| pabelanger | clarkb: mnaser: infact, most of that logic can be bindep now. But zuulv3 will also change that too | 17:26 |
| pabelanger | not bindep, but the JJB logic | 17:27 |
| mnaser | we'll see how bad it breaks p-o-i :D | 17:28 |
| *** Swami has joined #openstack-infra | 17:31 | |
| *** jpena is now known as jpena|off | 17:32 | |
| clarkb | anticw: sorry missed your ping earlier, I take it that means the functionality you wanted has returned in a useable way? | 17:32 |
| anticw | almost yes thanks | 17:33 |
| anticw | the gitweb defaults mean i don't get a single -u diff for some reason as we did before, i'm not sure if that's a local setting i can change | 17:34 |
| clarkb | anticw: I think that is because we switched from gitweb to cgit as the backend. This was an item we had intended on changing a long time ago and this situation finally pushed us to do it | 17:34 |
| anticw | yeah, arguably a lot of people prefer the current defaults and it's prettier. of course they are 10% wrong but likely there are more of them than me :) | 17:35 |
| clarkb | anticw: in the top right should be a few drop downs one is called mode, you can change that to unified | 17:35 |
| *** tosky has quit IRC | 17:35 | |
| *** dtantsur is now known as dtantsur|afk | 17:35 | |
| clarkb | I don't know if there is a good way to persist that, but it is an option | 17:36 |
| anticw | clarkb: yeah, i can and do ... more clicks ... i'm just extremely opinionated | 17:36 |
| *** efoley_ has quit IRC | 17:36 | |
| mordred | SamYaple: heya! sorry - I was afk - reading dockerhub scrollback now | 17:36 |
| clarkb | anticw: also gerrit has unified diffs built in but doens't have all diffs on a single page until we get to the next release | 17:37 |
| jeblair | dmsimard: reviewed the multinode stack; mostly cosmetic things. | 17:37 |
| dmsimard | jeblair: ok I'll take a look shortly. Thanks! | 17:38 |
| *** slaweq has joined #openstack-infra | 17:38 | |
| anticw | clarkb: yeah, i know a lot of people really like the way gerrit does things, i'm almost certainly in a minority in thinking a single -u diff is most useful | 17:38 |
| jeblair | anticw: gertty does have both, in case you like terminals | 17:38 |
| anticw | i like it because i can scroll up/down and see things without having to touch mouse | 17:38 |
| mordred | SamYaple: ah - so what I _REALLY_ want to do once we roll out v3 is to provide the ability (in the form of some base jobs you can use without having to talk to us) for you to push images that have been built in infra to dockerhub, rather than having dockerhub build them for us | 17:38 |
| anticw | jeblair: what do you mean? | 17:38 |
| clarkb | btw gerrit 2.14 does have single page with all the diffs if you use polygerrit ui, finally ~5 years after I rmembre we first asked for it | 17:39 |
| jeblair | anticw: http://git.openstack.org/cgit/openstack/gertty/tree/README.rst | 17:39 |
| mordred | SamYaple: is there a specific benefit to having dockerhub build something I may not be aware of? I think today doing that would be mostly a functionof "there is no other/better way to accomplish publishing images from openstack repos to dockerhub" - but I may be missing something | 17:39 |
| clarkb | jeblair: can you review 505751? | 17:39 |
| clarkb | jeblair: should be fix for getting your cache update in | 17:39 |
| *** sdague has quit IRC | 17:40 | |
| jeblair | clarkb: +3 | 17:40 |
| anticw | jeblair: thanks, i'll take a look at some point | 17:40 |
| clarkb | ok I don't think we pip install -U gerritlib on review.o.o | 17:40 |
| clarkb | puppet has run since the package update was pushed and pip still shows 0.6.0 | 17:40 |
| jeblair | anticw: i'm pretty sure fungi uses it in the way you describe -- unified diff (and all in one page). and you definitely don't have to use the mouse. (but you can if you want) | 17:41 |
| clarkb | ah we pip install -U jeepyb but jeepyb didn't update in this case | 17:41 |
| *** e0ne has joined #openstack-infra | 17:41 | |
| clarkb | I need breakfast but then I will likely just upgrade gerritlib by hand (feel free to veto me on that, open to other ideas but running out of patience) | 17:42 |
| jeblair | clarkb: ++ | 17:42 |
| SamYaple | mordred: images built in infra are not suitable for production in my opinion. this is primarily due to the fact that they would be built from package mirrors that are not signed | 17:42 |
| *** electrofelix has quit IRC | 17:42 | |
| clarkb | SamYaple: nothing prevents you from changing that | 17:42 |
| *** slaweq has quit IRC | 17:42 | |
| clarkb | SamYaple: you have root (as evidenced by the ability to docker) | 17:43 |
| mordred | SamYaple: gotcha. so - thta's an excellent point - and yah, what clarkb said, we can totally set up publication jobs that don't use the mirrors | 17:43 |
| fungi | jeblair: anticw: indeed i do use gertty in unified diff mode | 17:43 |
| fungi | and on a tiny terminal, too! | 17:43 |
| clarkb | the images we boot do not use those mirrors either | 17:43 |
| SamYaple | ok, i thought that was undesired | 17:43 |
| clarkb | er the image builds we boot off of don't | 17:43 |
| mordred | SamYaple: so I'd love to work with you to see if we can get a thing going in v3 that directly meets your needs | 17:43 |
| mordred | SamYaple: well - it's only undesired because of internet reliability and the gate in general | 17:44 |
| clarkb | SamYaple: well the tradeoff is reliability | 17:44 |
| SamYaple | fair enough | 17:44 |
| clarkb | but dockerhub likely to be similarly reliable assuming they use the same upstream mirrors | 17:44 |
| mordred | SamYaple: I have dockerhub related todo items for next week ... how about I ping you when I start working on them and we can tackle them together and solve all of your problems at once :) | 17:44 |
| pabelanger | we also have unsigned mirrors because we've never been able to crack the mirror indexes issues on ubuntu. However, I think a few people have ideas how to fix that | 17:45 |
| SamYaple | pabelanger: what mirror index issue? | 17:45 |
| mordred | yah. I think we were also going to investigate whether our apache caching proxy would work for apt repos | 17:46 |
| mordred | SamYaple: we curently mirror apt repos by using reprepro on an AFS volume to fetch the packages - which results in us building our own indexes from scratch | 17:46 |
| pabelanger | SamYaple: we regenerate indexes because we don't want to delete existing packages right away, otherwise current nodes break if they apt-get again | 17:46 |
| SamYaple | mordred: im using apt-cacher-ng right now, but ive used apache before for this purpose. so it *can* work | 17:46 |
| mordred | SamYaple: the plus side to this is that reprepro allows us to track state and only update incrementally - as well as to verify that our repo is in a consistent state before we publish | 17:47 |
| SamYaple | ah yea. i know that issue | 17:47 |
| fungi | pabelanger: SamYaple: it's less mirroring the signed indices that's the problem, more adding tools to validate the indices don't refer to packages missing from our mirror due to ordering issues or broken upstream mirrors | 17:47 |
| mordred | yah. that | 17:47 |
| pabelanger | clarkb: didn't your brother also write something for apt mirroring? | 17:47 |
| mordred | which is also a TDL for yum/dnf | 17:47 |
| pabelanger | fungi: ya | 17:47 |
| mordred | in any case - there are two perfectly valid issues we should fix here ... | 17:48 |
| fungi | pabelanger: SamYaple: if we generate the indices ourselves we can ignore the consistency issue, but then that means we don't have access to upstream signing | 17:48 |
| fungi | so it was a compromise until the validity checker got implemented | 17:48 |
| *** slaweq has joined #openstack-infra | 17:48 | |
| clarkb | calebb: ^ | 17:48 |
| pabelanger | ya, I think we could look at upstream signing issue again, when time is there. I know it is not the first time somebody has asked for gpg signed packaging on ubuntu | 17:49 |
| mordred | one is that I don't think any of us are particularly happy bout hte unsigned repos thing | 17:49 |
| calebb | pabelanger: yeah i did, runs mirrors.cat.pdx.edu/ubuntu | 17:49 |
| dmsimard | jeblair: ok, I'll address your comments. Did you also have a look at the integration tests in openstack-zuul for multinode things ? | 17:49 |
| mordred | the other one is that we want there to be a usable story for publishing docker images to docker hub (or s/docker images/things/ ; s/dockerhub/places/ ) | 17:49 |
| *** supertakumi86 has quit IRC | 17:50 | |
| jeblair | dmsimard: no i'll do that now, thanks | 17:50 |
| SamYaple | fair enough. i started on a project to do mirroring directly into s3/swift a while ago. it was to address the package expiry issue with object expires-at. i should look at it again | 17:50 |
| mordred | SamYaple: but - I think we can solve your thing next week or week after at the lastest without solving the whole thing | 17:50 |
| pabelanger | mordred: +1 unsigned repos | 17:50 |
| *** Apoorva has joined #openstack-infra | 17:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Branch cleanup (5/n) https://review.openstack.org/505762 | 17:50 |
| pabelanger | ya, once zuulv3 is out the door, I'm happy to piviot to docker things | 17:50 |
| calebb | pabelanger: https://github.com/squidboylan/apt-package-mirror is where it is | 17:51 |
| pabelanger | calebb: nice, thanks! I'll look at it shortly | 17:51 |
| *** Apoorva has quit IRC | 17:51 | |
| SamYaple | here is the think i started on... https://github.com/samyaple/apt-reflect . repo --> s3 with no local commit | 17:52 |
| *** yamahata has quit IRC | 17:52 | |
| dmsimard | jeblair: btw I'll improve the comment but for reference, the 'JSON' issue is what we discovered last week about the static include_role tasks https://github.com/ansible/ansible/issues/30385 | 17:52 |
| dmsimard | jeblair: It's fixed in ARA but I've been struggling to fix it in zuul ( https://review.openstack.org/#/c/504238/ ) without access to executor logs. | 17:53 |
| dmsimard | oh, that reminds me that the non-fatal 404's patch in ara was released yesterday. | 17:54 |
| jeblair | dmsimard: ah that | 17:54 |
| dmsimard | jeblair: yes, the fingers :D | 17:54 |
| dmsimard | truncated jsons* | 17:54 |
| *** slaweq has quit IRC | 17:54 | |
| pabelanger | calebb: looks like mirrors everything by default? | 17:54 |
| dmsimard | jeblair: I'd likely need to pair with an infra-root for a little while to see what's going on in the logs when we hit that bug.. bit of trial and error involved and for some reason I hadn't been able to reproduce locally with mordred's test-logs.sh | 17:55 |
| calebb | pabelanger: correct, i dont have a way currently to selectively mirror | 17:55 |
| pabelanger | calebb: k, good to know. | 17:56 |
| calebb | pabelanger: i believe i looked into it and found that with how the ubuntu mirrors are laid it outs very hard to do so | 17:56 |
| pabelanger | ya | 17:57 |
| clarkb | I am updating gerritlib on review.o.o now | 17:57 |
| pabelanger | clarkb: ack | 17:57 |
| jeblair | dmsimard: ack, i can help with that after lunch if you haven't found a buddy by then | 17:58 |
| dmsimard | sure | 17:58 |
| clarkb | and now I'm going t ofigure out how to run manage-projects by hand too | 17:59 |
| jeblair | dmsimard: one thing on your first multinode integration patch; the rest looks good | 17:59 |
| *** tesseract has quit IRC | 17:59 | |
| fungi | clarkb: sudo manage-projects | 17:59 |
| dmsimard | sweet | 17:59 |
| calebb | pabelanger: and fwiw our problems with referencing missing files on the mirrors since we put that in prod are mostly if not entirely gone | 17:59 |
| fungi | clarkb: optionally pass command-line arguments listing the specific repos you want processed if you don't want it to do all | 18:00 |
| mordred | mtreinish: is there a way to get stestr to not print test lines as it goes? if not, would you accept such a thing as a feature request? (I don't necessarily want it to be config in the repo itself - I just don't personally want to see thatoutput so would love to be able to set a $something somewhere to silence successful output | 18:00 |
| *** amoralej is now known as amoralej|off | 18:00 | |
| jeblair | clarkb, fungi: can you push through https://review.openstack.org/505451 please? | 18:00 |
| clarkb | fungi: thanks, I'm running the command that puppet runs and now http://git.openstack.org/cgit/openstack/networking-lagopus lives | 18:01 |
| clarkb | I think new images build sshould work at this point | 18:01 |
| pabelanger | calebb: ya, it is also possible we'd just setup a reverse proxy cache for ubuntu mirrors now per mordred suggestion, but would need to run it past clarkb too. Then we don't need to mirror 1TB of data | 18:01 |
| fungi | jeblair: on it now | 18:01 |
| mordred | pabelanger: WELL - actually, I forgot when I asked that that there is another issue our curent mirror design fixes | 18:02 |
| clarkb | manage-projects is done, I will keep an eye on nodepool image builds now | 18:02 |
| calebb | pabelanger: yeah, that makes sense | 18:02 |
| jeblair | dmsimard, pabelanger: is there a fix for the tox-linters pulling in ansible 2.4 issue? | 18:02 |
| dmsimard | jeblair: I believe that merged yesterday | 18:02 |
| fungi | jeblair: the example dnm change was a huge help, thanks! | 18:02 |
| dmsimard | I sent a fix | 18:02 |
| jeblair | dmsimard: cool thx | 18:02 |
| pabelanger | jeblair: I haven't looked sorry | 18:02 |
| mordred | pabelanger: which is that upsream mirrors purge old packages some times, which means that sometimes in the gate someone will have done an apt-get update and have a local cache of the index then tryto install something and it's nt there | 18:02 |
| dmsimard | jeblair: https://review.openstack.org/#/c/505467/ | 18:02 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove unused infra-manual trusty job https://review.openstack.org/505765 | 18:03 |
| mordred | so we do the delayed purge on our side to prevent that from breaking people | 18:03 |
| pabelanger | mordred: ya that, so I would think our proxy cache would still have the packages, assuming it was hit before the indexes refresh right? | 18:03 |
| mordred | pabelanger: yah- but if it wasn't it might not have them | 18:03 |
| pabelanger | right | 18:03 |
| mordred | pabelanger: in any case, it's a thing we'll need to carefully test | 18:03 |
| pabelanger | agree | 18:03 |
| clarkb | AJaeger: infra-root I think we can remove the freeze on new projects now unless you'd like toget other gerrit related things sorted out first | 18:04 |
| mordred | pabelanger: it's possible that the thing you mention (packages still in cache) would be good enough | 18:04 |
| fungi | clarkb: seems fine to me now | 18:04 |
| mordred | clarkb: I'm fine with it | 18:04 |
| mordred | clarkb, fungi: ALTHOUGH ... | 18:04 |
| mordred | if you wanted to review those jeepyb patches real quick | 18:04 |
| pabelanger | mordred: ya, hopefully. I don't know how often upstream ubuntu does new indexes | 18:04 |
| calebb | pabelanger: mordred my tool keeps things around for extra time, doesnt just delete them when they are deleted upstream | 18:04 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron https://review.openstack.org/436798 | 18:04 |
| pabelanger | calebb: ++ | 18:04 |
| mordred | then I could pay attention to th enext project adds and make sure manage-projects is still good after them | 18:04 |
| mordred | calebb: ++ | 18:05 |
| pabelanger | clarkb: I also got the [GitHub] Subscribed to 2 openstack repositories emails for the new projects | 18:05 |
| fungi | as did i | 18:05 |
| fungi | mordred: 467235 and the 3 others depending on that? | 18:06 |
| clarkb | I'm going to mark the gerrit memory use item on the new gerrit issues etherpad as "fixed". It needs more memory than older gerrit seemed to be it isn't growing indefinitely so meh | 18:07 |
| clarkb | jeblair: were you the person that added the ? behavior in new gerrit opens reverse search item to the etherpad? | 18:09 |
| clarkb | jeblair: I still get a help menu so that is weird | 18:09 |
| jeblair | clarkb: etherpad link? | 18:10 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Branch cleanup (6/n) https://review.openstack.org/505766 | 18:10 |
| mordred | fungi: yah | 18:10 |
| clarkb | jeblair: https://etherpad.openstack.org/p/gerrit-2.13-issues | 18:10 |
| AJaeger | clarkb: shall I merge a new project creation change? We have one ready... | 18:10 |
| *** slaweq has joined #openstack-infra | 18:10 | |
| jeblair | clarkb: not i. | 18:10 |
| clarkb | AJaeger: sounds like mordred has some jeepyb changes he wants in first | 18:10 |
| clarkb | AJaeger: why don't we review those then see | 18:11 |
| mordred | they aren't URGENt - but we tend to forget about jeepyb until there's an event - so I figure taking care of them now while we're paying attention would be nice | 18:11 |
| dmsimard | jeblair: for devstack-gate's multi-node networking, do we really not want to inherit from multinode ? clarkb and I discussed that the new v3 devstack job could re-use the multi-node-bridge role to set up bridges as required, not unlike what the legacy jobs do | 18:11 |
| mordred | clarkb, AJaeger: https://review.openstack.org/#/q/status:open+project:openstack-infra/jeepyb+branch:master+topic:fix-manage-projects | 18:12 |
| dmsimard | the new bridge role is set up to default to "br-infra" but was designed to allow for custom parameters like 'br-ex', etc. | 18:12 |
| jeblair | dmsimard: i can't fully parse that, sorry. | 18:12 |
| jeblair | dmsimard: i don't believe the legacy jobs will use the multinode job | 18:12 |
| jeblair | dmsimard: only the v3 native jobs can/will | 18:12 |
| *** ijw has joined #openstack-infra | 18:13 | |
| dmsimard | jeblair: right, hang on trying to find the comment I'm referring to | 18:13 |
| *** tosky has joined #openstack-infra | 18:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Make retired acl check less greedy https://review.openstack.org/505768 | 18:14 |
| mordred | fungi: ^^ followup to fix the thing you mentioned | 18:14 |
| dmsimard | jeblair: ah, nevermind we're on the same wavelength -- I misread https://review.openstack.org/#/c/504787/ | 18:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Promote logstash jobs from base-test to base https://review.openstack.org/505451 | 18:14 |
| *** ralonsoh has quit IRC | 18:14 | |
| fungi | thanks mordred! | 18:14 |
| *** slaweq has quit IRC | 18:15 | |
| jeblair | dmsimard: groovy! | 18:15 |
| dmsimard | jeblair: how would you like the nodesets to be set up for those tests btw ? | 18:15 |
| dmsimard | jeblair: what is an "anonymous" node set ? basically no nodeset and just declare the nodes directly in the job ? | 18:15 |
| jeblair | dmsimard: maybe just add them to the jobs themselves (since it1s 1:1) | 18:15 |
| AJaeger | mordred: looks fine, thanks - but need second +2 ;) | 18:15 |
| jeblair | dmsimard: yeah | 18:15 |
| dmsimard | jeblair: right, okay | 18:15 |
| dmsimard | sure. | 18:15 |
| clarkb | mordred: reviewing those changes I remember what my concern was re cache out of sync or not | 18:15 |
| jeblair | i'm going afk till after lunch | 18:16 |
| clarkb | mordred: should pushed_to_gerrit be False even though project is in Gerrit we will overwrite the project | 18:16 |
| *** Apoorva has joined #openstack-infra | 18:16 | |
| mordred | clarkb: ah - lemme look at that real quick | 18:17 |
| clarkb | mordred: might be nice ot have a belt and suspender around that where if you have commits to error and force a human to sort it out | 18:17 |
| AJaeger | clarkb: https://review.openstack.org/#/c/503829 is the project creation change that I'll +2 now - feel free to +A whenever mordred's changes landed. I'll remove my WIP there now | 18:17 |
| *** yamamoto has quit IRC | 18:17 | |
| clarkb | *if you have commits in gerrit | 18:17 |
| clarkb | AJaeger: ok | 18:17 |
| *** trown|lunch is now known as trown | 18:17 | |
| openstackgerrit | Merged openstack-infra/jeepyb master: Don't mark as created in github until we're done https://review.openstack.org/467235 | 18:17 |
| mordred | clarkb: actually - it's fine | 18:19 |
| mordred | clarkb: we don't force-push | 18:19 |
| mordred | clarkb: so if pushed-to-gerrit is erroneously False but we have pushed to gerrit and new commits have landed, the push will fail because git | 18:20 |
| *** slaweq has joined #openstack-infra | 18:20 | |
| *** Sukhdev_ has joined #openstack-infra | 18:20 | |
| mtreinish | mordred: there is a flag for that on stestr run, --no-subunit-trace (or something like that) | 18:21 |
| clarkb | mordred: gotcha I see that in make_local_copy now | 18:21 |
| mtreinish | mordred: http://stestr.readthedocs.io/en/latest/MANUAL.html#adjusting-test-run-output | 18:21 |
| mordred | clarkb: there is still a potential issue there though - which is that if you get into that scenario, I don't think the cache will ever get set porperly so you'll always think you need to push, and will try and fail each time it runs | 18:21 |
| clarkb | it will either noop bceause all commits there are fail because no | 18:21 |
| mordred | yah | 18:22 |
| clarkb | that makes me feel better about it | 18:22 |
| mtreinish | mordred: if you think we need more flexibility or different output options I'd gladly except a pull request for adding that | 18:22 |
| mordred | if we want to improve that, we should detect the difference between "I can't push because commit conflict" and "I can't push for other reasons" - and set pushed-to-gerrit if the first thing happens | 18:22 |
| AJaeger | clarkb: https://github.com/openstack/neutron-tempest-plugin is still empty - but imported in git.openstack.org . Will mordred's updates fix this automatically or are manual steps needed? | 18:23 |
| clarkb | AJaeger: when I ran manage-projects I did so without project limitations so I expect that will need manual help | 18:24 |
| *** rbrndt has joined #openstack-infra | 18:25 | |
| clarkb | possibly permissions update, I will take a look after reviewing these changes | 18:25 |
| mordred | I actually believe the patch we just landed will help this situation self-heal | 18:25 |
| clarkb | mordred: should I let oyu run it then and check? | 18:26 |
| clarkb | (rather than intervene now) | 18:26 |
| *** felipemonteiro__ has quit IRC | 18:26 | |
| *** slaweq has quit IRC | 18:26 | |
| *** felipemonteiro__ has joined #openstack-infra | 18:26 | |
| mordred | yah - one sec | 18:26 |
| clarkb | ok the jeepyb stack is approved | 18:27 |
| clarkb | I'll wait | 18:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Don't remove puppet in puppet prep macro. https://review.openstack.org/505751 | 18:27 |
| mordred | k. it would have self-corrected | 18:27 |
| AJaeger | great, mordred ! | 18:28 |
| mordred | it was just a gerrit replication thing and the current logic re-did the things it needed to | 18:28 |
| openstackgerrit | Merged openstack-infra/jeepyb master: Add description to needs_update calculations https://review.openstack.org/478939 | 18:28 |
| mordred | AJaeger: do we have any additional new projects coming? | 18:28 |
| clarkb | mordred: https://review.openstack.org/#/c/503829 is the one linked a little earlier | 18:28 |
| *** felipemonteiro has joined #openstack-infra | 18:28 | |
| clarkb | mordred: you'll want to make sure that jeepyb gets updated in all the places before running that | 18:28 |
| clarkb | not sure if puppet will update jeepyb before running manage-projects | 18:28 |
| clarkb | (seems like a bug if not but annoying if you want to watch it if so) | 18:29 |
| mordred | ok. I'll land that when jeepyb is in the places | 18:29 |
| openstackgerrit | Merged openstack-infra/jeepyb master: Stop showing retired repos in cgit https://review.openstack.org/478992 | 18:30 |
| *** eeiden has quit IRC | 18:30 | |
| AJaeger | clark, mordred, 503829 is the only other new repo that is ready to review - the others miss PTL+1 or have a -1... | 18:30 |
| *** felipemonteiro__ has quit IRC | 18:31 | |
| openstackgerrit | Merged openstack-infra/jeepyb master: Don't include retired projects for codesearch https://review.openstack.org/498919 | 18:31 |
| mordred | AJaeger: sweet - well - one is all we need :) | 18:31 |
| AJaeger | ;) | 18:31 |
| openstackgerrit | Matthew Treinish proposed openstack-dev/cookiecutter master: Switch to use stestr as the default test runner https://review.openstack.org/505772 | 18:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up hosts file https://review.openstack.org/504552 | 18:33 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up firewalls https://review.openstack.org/504553 | 18:33 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 18:33 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 18:33 |
| *** Sukhdev_ has quit IRC | 18:33 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 18:33 |
| *** askb has joined #openstack-infra | 18:33 | |
| clarkb | ubuntu-xenial-0000001077 | ubuntu-xenial | nb04 | qcow2,vhd,raw | ready | 00:00:01:48 | | 18:34 |
| clarkb | success! | 18:34 |
| pabelanger | yay | 18:34 |
| clarkb | I've marked off image build problems on the etherpad now | 18:35 |
| *** felipemonteiro has quit IRC | 18:35 | |
| clarkb | once we can get the cache change in and restart gerrit (hopefully for the last time) I will send a status update to the mailing list | 18:35 |
| *** felipemonteiro has joined #openstack-infra | 18:35 | |
| clarkb | mordred: I think I just maybe noticed a flaw in your cgit change to jeepyb | 18:36 |
| clarkb | oh actually nevermind | 18:36 |
| AJaeger | team, FYI with the three changes in my stack starting at https://review.openstack.org/#/c/505762, I've done all the branch retirement things that I could find. | 18:36 |
| clarkb | was worried that image builds would fail to clone them but they don't clone through cgit and instead use smart http cgi thing | 18:36 |
| AJaeger | clarkb: nodepool images don't include retired repos | 18:37 |
| fungi | clarkb: further, i think we have similar logic in our cache-repos element to skip the same repos | 18:37 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Publish static constraints files from the periodic pipeline https://review.openstack.org/505774 | 18:37 |
| fungi | yeah, what AJaeger just said | 18:37 |
| clarkb | perfect | 18:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/505148 | 18:38 |
| *** dangers is now known as dangers_away | 18:38 | |
| clarkb | not sure if anyone noticed but we are pretty full steam ahead today. Good exercise of new gerrit | 18:40 |
| pabelanger | Wow | 18:40 |
| pabelanger | Sep 20 14:58:34 elasticsearch03 puppet-user[77130]: Finished catalog run in 741.66 seconds | 18:40 |
| pabelanger | that is a long time to be running puppet | 18:40 |
| clarkb | there shouldn't be much in there either unless it had to redownloda the packaging for some reason | 18:40 |
| pabelanger | looks like it has been like that for some time | 18:41 |
| AJaeger | clarkb: I noticed - takes ages to get project-config changes through check queue ;( | 18:41 |
| *** efoley has joined #openstack-infra | 18:42 | |
| *** slaweq has joined #openstack-infra | 18:42 | |
| clarkb | AJaeger: ya I think our use of trusty slows us down because there aren't many trusty ready nodes | 18:43 |
| clarkb | pabelanger: Sep 20 18:37:59 elasticsearch03 puppet-user[111476]: (/Stage[main]/Elasticsearch/File[/var/lib/elasticsearch]) Failed to generate additional resources using 'eval_generate': No such file or directory - /var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2017.09.20/4/index/_5ed4_Lucene41_0.pos that looks suspicious | 18:44 |
| clarkb | it shouldn't be doing anything with the es indexes | 18:44 |
| pabelanger | clarkb: ya, that is what I am looking at now | 18:44 |
| clarkb | ok I'll leave you to it, let me know if yo uhave any questions I can help with | 18:45 |
| *** Sree has joined #openstack-infra | 18:46 | |
| *** askb has quit IRC | 18:46 | |
| *** slaweq has quit IRC | 18:47 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-elasticsearch master: Stop managing file permissions under /var/lib/elasticsearch https://review.openstack.org/505778 | 18:48 |
| pabelanger | clarkb: ^first optimization | 18:48 |
| pabelanger | shouldn't need to chmod ever file under /var/lib/elasticsearch | 18:49 |
| *** Sree has quit IRC | 18:50 | |
| clarkb | nice | 18:50 |
| *** camunoz has quit IRC | 18:51 | |
| *** slaweq has joined #openstack-infra | 18:52 | |
| pabelanger | that about the only thing I see that might slow down puppet | 18:53 |
| pabelanger | everything else seems right | 18:53 |
| mtreinish | mordred: if those options weren't enough I just remembered subunit-trace had a sort of middle ground option too: https://github.com/mtreinish/stestr/pull/98 | 18:53 |
| openstackgerrit | Merged openstack-infra/jeepyb master: Make retired acl check less greedy https://review.openstack.org/505768 | 18:56 |
| mordred | mtreinish: iI think the option is fine - I think what I want is the ability to set it via envvar or config file in ~/.config | 18:58 |
| mordred | mtreinish: I'll poke around a bit and see if I can come up with a way to do that that isn't insane | 18:58 |
| mordred | mtreinish: (I don't want to change how tox invokes it for other people - just for me) | 18:59 |
| *** slaweq has quit IRC | 18:59 | |
| mtreinish | mordred: you can always do tox -- --no-subunit-trace but that's a lot of extra typing | 19:00 |
| *** felipemonteiro__ has joined #openstack-infra | 19:00 | |
| *** felipemonteiro has quit IRC | 19:04 | |
| jeblair | that command has a nearly 25% dash ratio! :) | 19:04 |
| *** camunoz has joined #openstack-infra | 19:05 | |
| * jeblair goes back to lunching | 19:05 | |
| mtreinish | jeblair: heh, yeah I think it could use some more dashes | 19:05 |
| clarkb | mordred puppet failed on git0X trying to deal with six for jeepyb | 19:07 |
| clarkb | mordred: sudo journalctl | grep puppet-user to see it | 19:08 |
| clarkb | I think this may be a yum vs pip problem | 19:08 |
| *** yamahata has joined #openstack-infra | 19:09 | |
| *** coolsvap has quit IRC | 19:14 | |
| *** yamamoto has joined #openstack-infra | 19:17 | |
| *** camunoz has quit IRC | 19:23 | |
| jdandrea | fungi Is there a way to tell if openstackinfra is the founder of #openstack-valet? I'm trying to confirm if I need to do anything else to that channel in terms of transferring any founder status/etc. | 19:24 |
| mordred | clarkb: awesome. thanks - looking | 19:25 |
| *** yamamoto has quit IRC | 19:26 | |
| mordred | clarkb: SIGH | 19:26 |
| mordred | clarkb: cloud-init | 19:26 |
| pabelanger | Hmm, nodepool seems unhappy | 19:28 |
| pabelanger | looking | 19:28 |
| pabelanger | possible issue in ovh | 19:29 |
| clarkb | pabelanger: woo guessing full steam ahead made things fall over | 19:29 |
| pabelanger | clarkb: ya, I am wondering if related to new images | 19:30 |
| clarkb | jdandrea: you can ask chanserv for the channel access list founder is +F or something | 19:30 |
| clarkb | pabelanger: oh maybe | 19:30 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-ovh | 19:30 |
| fungi | jdandrea: i checked it before when you were asking and looks already set up, but /msg chanserv access #openstack-valet list | 19:30 |
| jdandrea | clarkb *nod* I was trying but failing in my efforts. Looking for the correct cmd again... | 19:30 |
| pabelanger | clarkb: vexxhost failures spiking too | 19:30 |
| clarkb | pabelanger: could just be caching on nova nodes being slow | 19:30 |
| jdandrea | fungi Oh thank you. That's what I thought but wasn't fully recalling. | 19:30 |
| mordred | infra-root: I'd like to suggest we remove python-six on centos - it'll cause cloud-init and abrt to be uninstalled - but we do not need or use those | 19:31 |
| pabelanger | clarkb: possible | 19:31 |
| mordred | the other option is that we can rework puppet-jeepyb to install jeepyb into a virtualenv | 19:31 |
| mordred | (this, btw, would have happened with any update to jeepyb and is not related to these patches) | 19:31 |
| fungi | mordred: i thought we still relied on cloud-init on our control-plane servers since we use images which don't include glean | 19:32 |
| jdandrea | fungi So in this list I see gomarivera, who originally set it up and is no longer involved. I'm taking his place. Do I need to get on that list instead of him? https://paste.ofcode.org/GM4hGYH475yrqk5BhWpKEE | 19:32 |
| pabelanger | clarkb: a lot of Connection timed out errors | 19:32 |
| *** ijw has quit IRC | 19:33 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 19:34 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 19:34 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 19:34 |
| mordred | fungi: we still *have* cloud-init on our control-plane servers, but once they boot and get our ssh key on them cloud-init no longer does anything for us | 19:34 |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config master: Increase timeouts for Manila DSVM jobs https://review.openstack.org/493092 | 19:34 |
| mordred | fungi: since on non-rax we have dhcp and on rax there is the nova-agent xenserver thing that pulls in the network config | 19:34 |
| *** slaweq has joined #openstack-infra | 19:34 | |
| clarkb | pabelanger: from server show? | 19:34 |
| fungi | jdandrea: it's easy enough for one of us to add you... the challenge is finding a good way to confirm whoever else is in that channel is in favor of that since it's not an official project and we don't have clear-cut rules of succession for unofficial projects | 19:34 |
| fungi | jdandrea: i'll /join and ask | 19:34 |
| clarkb | mordred: maybe try it on one git server first before doing that? | 19:34 |
| jdandrea | fungi Ahh. ok. | 19:34 |
| mordred | clarkb: I shall do that | 19:35 |
| clarkb | mordred: just to make sure cloud init isnt doing something we actually rely on somehow | 19:35 |
| jdandrea | fungi We are in contact with him. Maybe we can get him to join too and give the all clear. Would that do the trick? | 19:35 |
| clarkb | but I agree I think after initial bootstrap it can go away | 19:35 |
| mordred | clarkb: yah - I've deleted it from git01 - I'm going to reboot it now to make sure it comes back | 19:35 |
| pabelanger | clarkb: just in laucher-debug.log | 19:35 |
| *** askb has joined #openstack-infra | 19:35 | |
| pabelanger | looking at openstack server list now | 19:35 |
| mordred | clarkb: (that should be safe given load balancer, right?) | 19:36 |
| pabelanger | Wow | 19:36 |
| pabelanger | kernel panic | 19:36 |
| mordred | pabelanger: that's an ssh test error | 19:37 |
| mordred | pabelanger: ooh! where? | 19:37 |
| pabelanger | http://paste.openstack.org/show/621562/ | 19:37 |
| clarkb | mordred: you may need to tell haproxy to gracfully stop it but it does do keep alive checks on the backends too | 19:37 |
| pabelanger | that is in ovh | 19:37 |
| clarkb | mordred: minimal impact if you just reboot it | 19:37 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Publish static constraints files from the periodic pipeline https://review.openstack.org/505774 | 19:37 |
| pabelanger | clarkb: mordred: possible there was a new release of DIB? | 19:37 |
| fungi | jdandrea: you could just get gomarivera to connect and /msg chanserv access #openstack-valet add jdandrea +Aefortv | 19:38 |
| clarkb | pabelanger: eell we didnt have image builds working for days | 19:38 |
| clarkb | and fixing networking lagopus resulted in new images | 19:38 |
| mordred | pabelanger: so that's what the console log showed you - not a local kernel panic for you caused by openstack client :) | 19:38 |
| jdandrea | fungi That's what I'll try, thank you. Tin (who you addressed in the other room) may be afk. I'll check both. | 19:38 |
| jdandrea | Appreciate the help! | 19:38 |
| mordred | pabelanger: ah "[ 1.171846] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)" | 19:39 |
| pabelanger | mordred: ya, sorry. Kernel panic on latest xenial | 19:39 |
| fungi | jdandrea: my pleasure | 19:39 |
| pabelanger | mordred: yup, see that oo | 19:39 |
| pabelanger | too* | 19:39 |
| clarkb | any oyher images doingit? | 19:39 |
| pabelanger | checking | 19:39 |
| clarkb | if so problem is likelu dib and not the distro | 19:39 |
| *** Rockyg has quit IRC | 19:40 | |
| mordred | clarkb: k. I rebooted post cloud-init removal and git01 is fine | 19:40 |
| clarkb | cool | 19:40 |
| jdandrea | fungi To confirm, can gomarivera issue that cmd with +Aefortv flags? | 19:40 |
| clarkb | mordred: maybe we only do it on centos7 since elsewhere is fine? | 19:41 |
| fungi | jdandrea: yes, should work just fine | 19:41 |
| jdandrea | fungi ok thx | 19:41 |
| fungi | jdandrea: that would then grant you the same permissions they have | 19:41 |
| pabelanger | clarkb: only xenial is uploaded atm | 19:41 |
| clarkb | ah | 19:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-jeepyb master: Remove python-six on centos https://review.openstack.org/505785 | 19:42 |
| mordred | clarkb: ^^ | 19:42 |
| pabelanger | centos-7 is next online | 19:42 |
| *** slaweq has quit IRC | 19:42 | |
| clarkb | mordred: does yum transitively remove cloud init or just fail? with pyyaml it looked like it would just fail | 19:43 |
| pabelanger | okay 1 centos-7 in inap-mtl01, checking now | 19:43 |
| pabelanger | ianw: see ^ when you are online. Wondering if something in DIB changed | 19:45 |
| pabelanger | trying to see if centos-7 is affected | 19:45 |
| *** slaweq has joined #openstack-infra | 19:45 | |
| mordred | clarkb: it transitively removed cloud-init when I ran it by hand | 19:46 |
| *** felipemonteiro__ has quit IRC | 19:47 | |
| *** cody-somerville has joined #openstack-infra | 19:47 | |
| *** cody-somerville has quit IRC | 19:47 | |
| *** cody-somerville has joined #openstack-infra | 19:47 | |
| *** liusheng has quit IRC | 19:48 | |
| *** felipemonteiro has joined #openstack-infra | 19:48 | |
| *** liusheng has joined #openstack-infra | 19:48 | |
| clarkb | huh neat | 19:48 |
| clarkb | mordred: actually now we are getting spammed | 19:53 |
| clarkb | the machine id went away and cgit really wants that | 19:53 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Publish static constraints files from the periodic pipeline https://review.openstack.org/505774 | 19:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove openstack-doc-build-to-move https://review.openstack.org/504785 | 19:53 |
| clarkb | that is aggravating | 19:53 |
| clarkb | review updated with ^ info | 19:55 |
| jdandrea | @fungi Ok, we've reached out to gomarivera and I'm PM'ed lamt. Crossing fingers. :) | 19:57 |
| mordred | clarkb: AWESOME | 19:57 |
| mordred | clarkb: any indication in the email as to where I could see a log file on the server? | 19:58 |
| jdandrea | fungi - lamt: "I will ping fungi whenever I can hop onto IRC. My IRC connection at work today seems to be broken." Womp womp. | 19:58 |
| clarkb | journaltctl is best guess | 19:59 |
| clarkb | cgi-bin/apache is ehat is crashing | 19:59 |
| clarkb | its a segfault with core dump too | 20:00 |
| clarkb | how to make apache core dump? uninstall cloud init | 20:00 |
| mordred | clarkb: wow - there is nothing in the apache error log about it | 20:00 |
| pabelanger | 2017-09-20 18:13:52,828 INFO nodepool.image.build.ubuntu-xenial: /tmp/in_target.d/finalise.d/50-bootloader: line 160: type: grub2-mkconfig: not found | 20:01 |
| pabelanger | that doesn't look right | 20:01 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add multinode integration jobs and integration tests for known_hosts https://review.openstack.org/504787 | 20:01 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 20:01 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 20:01 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-hosts-file https://review.openstack.org/505789 | 20:01 |
| pabelanger | oh, maybe it is fine | 20:01 |
| mordred | clarkb: sorry to keep bothering you - but couldyou maybe orward me a mail? I don't seem to be getting the emails nor can I find info on the disk right now | 20:01 |
| clarkb | ya | 20:02 |
| dmsimard | jeblair: I believe I've addressed all your concerns about the multi-node stack. Turns out there wasn't any tests for the hosts file role so I added that in as well. | 20:03 |
| clarkb | mordred: sent | 20:03 |
| mordred | clarkb: thank you | 20:04 |
| clarkb | pabelanger: did centos 7 boot? | 20:05 |
| pabelanger | clarkb: okay, debian-jessie just came online with latest images | 20:05 |
| pabelanger | clarkb: still waiting | 20:05 |
| clarkb | ah | 20:05 |
| clarkb | http://logs.openstack.org/77/505677/1/check/gate-openstackci-beaker-ubuntu-trusty/e04a328/console.html#_2017-09-20_19_56_43_411177 when it rains it pours | 20:07 |
| clarkb | I'm going to push a patch to pin that next | 20:07 |
| *** gouthamr has quit IRC | 20:09 | |
| clarkb | how was that even ever working before | 20:09 |
| mordred | AJaeger: if you're still around, could you look at https://review.openstack.org/#/c/504506 ? | 20:10 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_infra_spec_helper master: Pin metadata-json-lint to < 2.0.0 https://review.openstack.org/505792 | 20:12 |
| clarkb | I think ^ will address the gems problem | 20:12 |
| pabelanger | mordred: when you have spare time, openstack server list on inap seems slow. I'd be interested to hear why that would be | 20:14 |
| mgagne | pabelanger: me? | 20:14 |
| pabelanger | mgagne: maybe? I admit I'm not sure where to look for slowness | 20:15 |
| mgagne | pabelanger: me neither, we have a similar slowness perception in our private cloud but didn't bother so far to look into it | 20:16 |
| pabelanger | clarkb: centos-7 is good | 20:16 |
| pabelanger | clarkb: so far, panic is limited to xenial | 20:16 |
| mgagne | pabelanger: so I've created a task in our backlog to look into it | 20:16 |
| clarkb | pabelanger: interesting | 20:16 |
| pabelanger | clarkb: let me see if I can get a trusty online | 20:17 |
| *** spzala has joined #openstack-infra | 20:17 | |
| pabelanger | clarkb: for now, maybe we roll back xenial to keep gate moving? | 20:17 |
| mordred | clarkb: if you have a sec, I could use a second set of eyes ... | 20:17 |
| mgagne | currently running kilo with cellsv1 if that changes anything, we plan on upgrade to mitaka very soon | 20:17 |
| clarkb | pabelanger: we'll have to pause the builds and then delete the most recent image yes | 20:17 |
| clarkb | mordred: sure | 20:17 |
| pabelanger | clarkb: ya, was going to manually apply pause on builders, so emergency file for them | 20:18 |
| mordred | pabelanger: unfortunately I cna't help much with 'openstack server list' - it should be a single command from osc perspective | 20:18 |
| mordred | clarkb: so - here's the deal | 20:18 |
| pabelanger | clarkb: then push up puppet change | 20:18 |
| clarkb | pabelanger: ok, go for it | 20:18 |
| inc0 | mordred: so, about registry, do we want backend on disks or swift? | 20:18 |
| mordred | clarkb: if you do jounalctl | grep machine-id on git01.o.o then skip down to the dracut line | 20:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Support Gerrit 2.13 ref-updated events https://review.openstack.org/505795 | 20:19 |
| mordred | inc0: that's an excellent question - debugging a issue on one of our servers atm ... just a sec | 20:20 |
| mgagne | pabelanger: I found the openstack server list command to be much slower than nova list, 2x | 20:20 |
| mordred | clarkb: oh- nevermind, that line is a red-herring | 20:20 |
| mgagne | but I think we should compare pure API calls, not libraries... | 20:20 |
| clarkb | mordred: ok I was looking at it and wondering. I do notice that file is 444 so should be readable by anyone | 20:20 |
| clarkb | so now am extra confused why apache was complaining | 20:20 |
| mordred | clarkb: I thought there was an issue with the git daemon deciding to ignore /var/lib/git prefix ... | 20:21 |
| clarkb | mordred: can you hit http://git01.openstack.org:8080/cgit | 20:21 |
| mordred | clarkb: yah - I can't find anything broken | 20:21 |
| mordred | clarkb: yup | 20:21 |
| mordred | clarkb: worked fine | 20:21 |
| clarkb | so can I | 20:21 |
| *** pabelanger has quit IRC | 20:21 | |
| *** pabelanger has joined #openstack-infra | 20:21 | |
| pabelanger | #status log nb03.o.o / nb04.o.o added to emergency file | 20:21 |
| openstackstatus | pabelanger: finished logging | 20:22 |
| dansmith | I haven't gotten any emails from gerrit since 3am this morning and I know I've generated a few.. is that known or concerning? | 20:22 |
| mordred | clarkb: I think that's a VERY old log | 20:22 |
| pabelanger | dansmith: known issue, we are working to resolve it | 20:22 |
| mordred | clarkb: :Sep 30 03:21:00 git01 systemd: Starting Session 49771 of user cgit. | 20:22 |
| dansmith | pabelanger: rock on thanks | 20:22 |
| mordred | clarkb: note "Sep 30" | 20:22 |
| mordred | clarkb: yah: Date: Mon, 11 Apr 2016 10:58:31 +0000 | 20:22 |
| mordred | clarkb: so - one of the things that uninstalling cloud-init did was $something related to abrt which is the bug reporting thing | 20:23 |
| clarkb | mordred: huh and it only just sent out now? weird | 20:23 |
| pabelanger | okay, bad xenial DIB is deleted | 20:23 |
| mordred | clarkb: so I hav ea hunch that doing thta may have just unstuck a pile of old reports perhaps? | 20:23 |
| clarkb | mordred: ya could be | 20:23 |
| clarkb | I'll update my vote | 20:23 |
| mordred | clarkb: woot! (and *phew*) | 20:23 |
| *** armax has quit IRC | 20:23 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: DNM: Test Ansible 2.4 https://review.openstack.org/505354 | 20:24 |
| fungi | clarkb: mordred: it only just dawned on me that what you were troubleshooting was related to the abrt e-mails i received... one was from as far back as 2015 | 20:25 |
| clarkb | dansmith: tl;dr is gerrit now asynchronously sends email and is doing like 100k db queries per email so it is slow. Upstream has told us to make our caches big enough for all accounts to void that so we are trying to get the change for that merged but its like 3 deep in the list of fixes now because image builds are broken and gems can't install on trusty, etc | 20:25 |
| fungi | which i found odd, and should have brought up i guess | 20:25 |
| mordred | dansmith: tl;dr - it's BOTH known AND concerning :) | 20:26 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Pause ubuntu-xenial DIBs https://review.openstack.org/505798 | 20:26 |
| pabelanger | mordred: clarkb: ^ mind a +3 to pause xenial | 20:27 |
| clarkb | dansmith: oh ya also xenial is kernel panicing | 20:27 |
| mordred | fungi: could I cajole you into a review on https://review.openstack.org/#/c/504506/ ? | 20:27 |
| jeblair | zuulv3 is /win 21 | 20:28 |
| jeblair | ga | 20:28 |
| pabelanger | okay, xenial nodes coming back online | 20:28 |
| pabelanger | still waiting to see if trusty is an issue | 20:28 |
| jeblair | zuulv3 is wedged; i have tests that reproduce the error, so i'm going to restart it to break the loop | 20:28 |
| *** srobert has quit IRC | 20:29 | |
| jeblair | oh, i'm also going to delete the debug log because it's 30G | 20:29 |
| pabelanger | +1 | 20:29 |
| clarkb | nice, should be smaller now that we aren't logging that one lib right? | 20:29 |
| clarkb | caching thing | 20:29 |
| jeblair | clarkb: well, it was mostly because it was stuck in an infinite loop | 20:30 |
| clarkb | ah | 20:30 |
| dansmith | clarkb: mordred: wow. :/ | 20:30 |
| jeblair | clarkb: which i have successfully reproduced in a test! | 20:30 |
| jeblair | so now my tests are stuck in an infinite loop | 20:30 |
| clarkb | jeblair: btw in the saga of getting gerrit cache numbers bumped up https://review.openstack.org/#/c/505792/ is a thing as is getting puppet to work on the git backends again | 20:30 |
| clarkb | so there have now been a total of 3 broken things preventing that change from going in :/ | 20:30 |
| jeblair | clarkb: i +3d that | 20:31 |
| clarkb | thanks | 20:31 |
| *** dargains has quit IRC | 20:31 | |
| clarkb | gah there is another lib that can't old ruby | 20:32 |
| clarkb | http://logs.openstack.org/92/505792/1/check/gate-puppet-openstack-infra-spec-helper-unit-ubuntu-trusty/32a87ef/console.html#_2017-09-20_20_16_55_765816 | 20:32 |
| clarkb | were we somehow installing newer ruby on trusty? | 20:32 |
| *** dargains has joined #openstack-infra | 20:32 | |
| * clarkb takes a look at ubuntu trusty packages | 20:32 | |
| pabelanger | no, I think we are ruby1.9 | 20:33 |
| pabelanger | so, newer gems getting pulled in needing >2.0 | 20:33 |
| *** ijw has joined #openstack-infra | 20:33 | |
| clarkb | trusty-updates has ruby2.0 in it | 20:33 |
| clarkb | pabelanger: ya but why did that all of a sudden change? makes me wonder if we stopped installing something like ruby2.0 | 20:33 |
| clarkb | mnaser: ^ maybe you know? | 20:34 |
| pabelanger | clarkb: http://logs.openstack.org/92/505792/1/check/gate-puppet-openstack-infra-spec-helper-unit-ubuntu-trusty/32a87ef/console.html#_2017-09-20_20_15_02_992910 | 20:34 |
| mnaser | hi | 20:34 |
| pabelanger | seems like only 1.9 is getting installed | 20:34 |
| * mnaser looks | 20:34 | |
| pabelanger | clarkb: ya, not sure. maybe just new releases? | 20:34 |
| mnaser | why dont you just pin webmock in puppet-openstack_infra_spec_helper ? | 20:35 |
| clarkb | mnaser: well I just pinned another lib so have to hunt them all down but yes | 20:35 |
| mnaser | fun time :D | 20:35 |
| clarkb | mostly just curious why its broken now | 20:35 |
| clarkb | and thinking we must've had ruby 2.0 somehow? | 20:35 |
| *** ijw has quit IRC | 20:35 | |
| mordred | pabelanger, fungi, jeblair: mind +3ing https://review.openstack.org/#/c/505785/ to unbreak jeepyb on the git servers (no rush - just don't want it to get lost in the excitement) | 20:35 |
| mnaser | hmm | 20:36 |
| *** ijw has joined #openstack-infra | 20:36 | |
| pabelanger | clarkb: maybe we should just spend the time on upgrading everything to xenial :) | 20:36 |
| *** dargains has quit IRC | 20:36 | |
| clarkb | pabelanger: problem is I really want to restart gerrit today with the new cache limits | 20:36 |
| clarkb | :/ | 20:36 |
| mnaser | clarkb i wouldnt know honestly, we dont have anything testing on trusty i think :( | 20:37 |
| pabelanger | ya, in that case pinning is right | 20:37 |
| pabelanger | mordred: +3 | 20:37 |
| mordred | pabelanger: thank you! | 20:37 |
| pabelanger | oh, fungi too :) | 20:37 |
| mordred | fungi: thank you! | 20:38 |
| mnaser | i dont see webmock usage :> | 20:38 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_infra_spec_helper master: Pin metadata-json-lint and webmock https://review.openstack.org/505792 | 20:39 |
| clarkb | mnaser: interesting, in any case I hope ^ gets us moving again | 20:39 |
| clarkb | jeblair: ^ new patchset pins additional package | 20:39 |
| mnaser | im searching | 20:39 |
| mnaser | it looks like webmock was added when the repo was created based off of puppet-openstack spec helper | 20:39 |
| mnaser | so maybe itw as just copy-pasta'd | 20:39 |
| clarkb | could be | 20:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mapping entries for proposal jobs https://review.openstack.org/504608 | 20:40 |
| mnaser | looks like we still have it in puppet-openstack-integration | 20:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add publish-service-types-authority job and mapping https://review.openstack.org/504609 | 20:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add xstatic-check-version and openstack-tox-pypy https://review.openstack.org/504610 | 20:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add mapping file setting to skip jobs from share queues https://review.openstack.org/504966 | 20:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make yaml2ical publication job https://review.openstack.org/504968 | 20:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Be more descriptive when publishing service-types https://review.openstack.org/505679 | 20:40 |
| clarkb | I'll make a note that that is a possible cleanup but focus right now is getting gerrit running with different cache settings | 20:40 |
| mnaser | ..yet we don't sue it either heh | 20:40 |
| kfox1111 | our gates just broke on being able to install docker. | 20:40 |
| kfox1111 | do we cache centos-extra's? | 20:41 |
| *** dprince has quit IRC | 20:41 | |
| pabelanger | yes | 20:41 |
| clarkb | kfox1111: we mirror them http://mirror.dfw.rax.openstack.org/centos/7/extras/ | 20:41 |
| kfox1111 | getting this in the logs: | 20:42 |
| kfox1111 | 2017-09-20 20:23:45.557883 | No package docker-1.12.6-16.el7.centos available. | 20:42 |
| kfox1111 | looks like the mirror rolled but the cache is new enough it still thinks its valid? | 20:42 |
| kfox1111 | maybe I should 'yum clean all' in the job? | 20:42 |
| *** dargains has joined #openstack-infra | 20:42 | |
| pabelanger | kfox1111: mind posting a log? | 20:43 |
| clarkb | the important distinction there is we don't proxy cache centos mirror we actually mirror it | 20:43 |
| clarkb | so in theory the mirror should always be valid (though sometimes hwne upstream breaks that behavior breaks) | 20:43 |
| pabelanger | ya, looks like 2017-09-07 is when docker package updated | 20:43 |
| clarkb | but ya a log would be helpful, we don't even know if it used the mirror | 20:43 |
| kfox1111 | jobs are still running, but: telnet://104.239.174.198:19885 | 20:43 |
| kfox1111 | if the mirror went from 7.3 -> 7.4 without keeping the 7.3 stuff around, it might explain it. | 20:44 |
| pabelanger | right, because we have centos/7 for mirror, not point releases | 20:44 |
| pabelanger | so possible 7.4 has broken you some how | 20:45 |
| kfox1111 | yeah. I think the vm is 7.3 with a recent up to date but pulling from 7.4 repo now. | 20:45 |
| kfox1111 | I just added a yum clean all at the begining. I think that should fix it. will give it a shot. | 20:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Add specs publication jobs https://review.openstack.org/504506 | 20:45 |
| clarkb | our vm should be 7.4 | 20:45 |
| pabelanger | ya, was about to say | 20:45 |
| clarkb | since that broke tripleo | 20:45 |
| pabelanger | 2017-09-20 20:18:44.089830 | 2017-09-20 18:09 | 20:45 |
| kfox1111 | hmm... don't know then. :/ | 20:46 |
| pabelanger | that is images from today actually | 20:46 |
| pabelanger | kfox1111: how long have you seen the issue? | 20:46 |
| kfox1111 | last few hours. | 20:46 |
| *** efried has joined #openstack-infra | 20:46 | |
| *** Sree has joined #openstack-infra | 20:46 | |
| openstackgerrit | Merged openstack-dev/pbr master: Remove py26 support https://review.openstack.org/504843 | 20:46 |
| kfox1111 | while fixing another breakage (pycrypto) | 20:46 |
| mordred | wow. it really does pour when it rains | 20:48 |
| pabelanger | kfox1111: I'm missing where docker is installed in telnet link | 20:48 |
| pabelanger | kfox1111: have a timestamp? | 20:48 |
| efried | harlowja Seen this with Taskflow lately? AttributeError: 'DiGraph' object has no attribute 'nodes_iter' | 20:48 |
| harlowja | have not seen | 20:48 |
| harlowja | but perhaps a new networkx release did this | 20:49 |
| efried | That's what my guy thought. | 20:49 |
| *** spzala has quit IRC | 20:49 | |
| *** rwsu has quit IRC | 20:50 | |
| *** Sree has quit IRC | 20:51 | |
| clarkb | mordred: I'm || that close to just applying the cache changes by hand and restarting... I know this is the wrong way to do it but running out of hours in the day and the stack of fixes is only getting deeper | 20:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix infinite loop on reconfiguration exception https://review.openstack.org/505807 | 20:51 |
| clarkb | spec helper change did finally pass on trusty though | 20:51 |
| kfox1111 | pabelanger: 2017-09-20 20:23:45.557883 | No package docker-1.12.6-16.el7.centos available. | 20:51 |
| efried | harlowja So what does one do about such a thing? | 20:51 |
| clarkb | so once that merges should be able to rerun jobs on cache update and hopefully pass and merge then restart gerrit (assuming git backends are also puppeting by thne) | 20:52 |
| *** jistr has quit IRC | 20:52 | |
| harlowja | prob restrict the requirements contraint and / or then adjust taskflow | 20:52 |
| clarkb | you should adjust taskflow first to make sure it fixes the problem | 20:52 |
| clarkb | then update requirements if so | 20:52 |
| pabelanger | kfox1111: I wonder if the google yum repos is making that a hard dependency | 20:53 |
| kfox1111 | hmm... | 20:53 |
| kfox1111 | that could be. | 20:54 |
| kfox1111 | I have a review with the yum clean all going. lets see what happens. | 20:54 |
| mordred | clarkb: I will not judge you if you apply by hand and restart | 20:55 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Publish static constraints files from the periodic pipeline https://review.openstack.org/505774 | 20:55 |
| mordred | clarkb: but also it seems like maybe we're close to success anyway | 20:56 |
| clarkb | ya I think we should be able to get it in properly soon | 20:56 |
| *** darvon has quit IRC | 20:57 | |
| *** mat128 has quit IRC | 20:58 | |
| efried | Downgrading networkx in my venv fixes the problem for me. | 20:58 |
| openstackgerrit | Merged openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/503611 | 20:58 |
| efried | I'm seeing it in pypowervm, which doesn't currently have a direct requirement for networkx | 20:58 |
| *** jascott1 has quit IRC | 20:59 | |
| *** pcaruana has quit IRC | 20:59 | |
| efried | so it would be a little weird to put it into requirements.txt there just to restrict it. | 20:59 |
| *** jascott1 has joined #openstack-infra | 20:59 | |
| *** acoles has joined #openstack-infra | 20:59 | |
| mordred | tonyb, dhellmann: ^^ the "Publish static constraints" job above is the type of job that can't be auto-migrated and we'll need to write a v3 version of it | 21:00 |
| *** jascott1 has quit IRC | 21:00 | |
| *** jheroux has quit IRC | 21:00 | |
| tonyb | mordred: Sure Once I get a +1 from Jenkins I'll -W. | 21:01 |
| mordred | cool | 21:01 |
| mordred | I mostly don't want it to fall into the cracks | 21:01 |
| *** dargains has quit IRC | 21:01 | |
| *** acoles has left #openstack-infra | 21:01 | |
| tonyb | mordred: Thanks. It isn't really menat for merging rigth now but I figured I'd stop discussin abstracts and publish soem code | 21:01 |
| mordred | tonyb: ++ | 21:02 |
| kfox1111 | hmm.... with the yum clean all in there, yeah... same issue... | 21:02 |
| tonyb | mordred: truns out I'm just doing it badly ;P | 21:02 |
| *** trown is now known as trown|outtypewww | 21:02 | |
| mordred | tonyb: well - publishing stuff is hard ;) | 21:02 |
| tonyb | :) | 21:02 |
| mordred | infra-root, AJaeger: ^^ general heads up - any new jobs that involve new publishers - or publishing new sorts of things need to be put into a freeze | 21:02 |
| *** darvon has joined #openstack-infra | 21:02 | |
| *** felipemonteiro__ has joined #openstack-infra | 21:02 | |
| kfox1111 | nm..... looks like it is pinned in the gate scripts. | 21:02 |
| kfox1111 | sorry for the noise. | 21:02 |
| mordred | infra-root, AJaeger: they're typically the types of jobs that need a manual translation - and I'm worried that jobs will fall through the cracks if we land new ones now | 21:03 |
| *** rwsu has joined #openstack-infra | 21:03 | |
| *** e0ne has quit IRC | 21:03 | |
| pabelanger | +1 | 21:03 |
| *** thorst has quit IRC | 21:03 | |
| fungi | yeah, i've starred that change for later review | 21:04 |
| *** e0ne has joined #openstack-infra | 21:04 | |
| *** e0ne has quit IRC | 21:05 | |
| *** jkilpatr has quit IRC | 21:05 | |
| *** e0ne has joined #openstack-infra | 21:05 | |
| clarkb | woo spec helper change is in the gate now | 21:05 |
| clarkb | then I get to recheck jeblair's change after that merges | 21:05 |
| *** e0ne has quit IRC | 21:05 | |
| *** felipemonteiro has quit IRC | 21:06 | |
| *** e0ne has joined #openstack-infra | 21:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add mapping entries for proposal jobs https://review.openstack.org/504608 | 21:06 |
| *** e0ne has quit IRC | 21:06 | |
| *** e0ne has joined #openstack-infra | 21:07 | |
| *** e0ne has quit IRC | 21:07 | |
| *** esberglu has quit IRC | 21:07 | |
| *** thorst has joined #openstack-infra | 21:07 | |
| *** e0ne has joined #openstack-infra | 21:07 | |
| *** e0ne has quit IRC | 21:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-service-types-authority job and mapping https://review.openstack.org/504609 | 21:08 |
| *** jascott1 has joined #openstack-infra | 21:08 | |
| clarkb | pabelanger: we probably want to scp the image of nb0x and mount it locally/boot it locally to see what is going on with it | 21:09 |
| clarkb | (I feel like debugging that is just enough distraction that I don't ant to until gerrit is restarted, so I'll hold off on that until later) | 21:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Add xstatic-check-version and openstack-tox-pypy https://review.openstack.org/504610 | 21:10 |
| *** iyamahat has joined #openstack-infra | 21:11 | |
| *** dprince has joined #openstack-infra | 21:11 | |
| *** xyang1 has quit IRC | 21:11 | |
| *** thorst has quit IRC | 21:12 | |
| *** jascott1 has quit IRC | 21:12 | |
| *** slaweq has quit IRC | 21:14 | |
| *** jascott1 has joined #openstack-infra | 21:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add mapping file setting to skip jobs from share queues https://review.openstack.org/504966 | 21:15 |
| openstackgerrit | Merged openstack-infra/project-config master: Pause ubuntu-xenial DIBs https://review.openstack.org/505798 | 21:16 |
| *** slaweq has joined #openstack-infra | 21:16 | |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config master: Stop running redundant manila lvm jobs https://review.openstack.org/505818 | 21:17 |
| *** rhallisey has quit IRC | 21:18 | |
| openstackgerrit | Merged openstack-infra/puppet-openstack_infra_spec_helper master: Pin metadata-json-lint and webmock https://review.openstack.org/505792 | 21:18 |
| *** jascott1 has quit IRC | 21:19 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add xstatic-check-version and xstatic-publish-to-pypi https://review.openstack.org/503866 | 21:19 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add pypy project-templates https://review.openstack.org/504116 | 21:19 |
| mordred | fungi, tonyb: it might actually be a great change (since it's not terribly complex on the zuul side) to use as a learning example once v3 is live | 21:19 |
| tonyb | mordred: +1 | 21:19 |
| mordred | speaking of ... | 21:20 |
| tonyb | mordred: I'll be back in Australia next week so less overlap but I'm keen to start learning | 21:20 |
| clarkb | 505677 rechecked for hopefully the last time | 21:21 |
| *** esberglu has joined #openstack-infra | 21:21 | |
| mordred | tonyb, fungi: https://review.openstack.org/#/c/504968 adds a job that publishes a new type of thing to a new location | 21:22 |
| mordred | tonyb, fungi: it's slighlymore complex than what the constraints job needs to do since it's adding a reference to a whole new server | 21:22 |
| mordred | but the pieces are there | 21:22 |
| mordred | oh. actually | 21:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Be more descriptive when publishing service-types https://review.openstack.org/505679 | 21:24 |
| *** esberglu_ has joined #openstack-infra | 21:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make yaml2ical publication job https://review.openstack.org/504968 | 21:24 |
| mordred | I left something off before | 21:24 |
| *** esberglu has quit IRC | 21:24 | |
| *** jcoufal has quit IRC | 21:24 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add multinode integration jobs and integration tests for known_hosts https://review.openstack.org/504787 | 21:24 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-hosts-file https://review.openstack.org/505789 | 21:24 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-firewall https://review.openstack.org/504788 | 21:24 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Add integration tests for multi-node-bridge role https://review.openstack.org/504789 | 21:24 |
| dmsimard | mordred: ^ now without lookup shenanigans | 21:24 |
| *** jkilpatr has joined #openstack-infra | 21:25 | |
| dmsimard | mordred, jeblair: oops, syntax error. Where would I put groups for in-job nodesets ? | 21:26 |
| *** chlong has quit IRC | 21:26 | |
| jeblair | 505795,1 in zuulv3 gate is waiting on a node that has been building for 15m | 21:26 |
| dmsimard | https://review.openstack.org/#/c/504787/ doesn't work. | 21:26 |
| tonyb | mordred: Thanks. As someone very new to ansible it even makes sense ;P | 21:26 |
| mordred | tonyb: yay! | 21:26 |
| jeblair | dmsimard: under groups | 21:26 |
| jeblair | err | 21:26 |
| *** rlandy has quit IRC | 21:26 | |
| clarkb | mordred: I've also rechecked your puppet-jeepyb change as it appears to have hit the same trusty issues as jeblairs | 21:26 |
| jeblair | dmsimard: under nodes | 21:26 |
| dmsimard | jeblair: oh, oops. Let's try that. | 21:26 |
| mordred | clarkb: thanks! | 21:27 |
| jeblair | hrm, that doesn't look quite right | 21:27 |
| *** jascott1 has joined #openstack-infra | 21:27 | |
| dmsimard | jeblair: what, the groups ? | 21:27 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/bindep master: SUSE: Update openSUSE Tumbleweed references for lsb-release-3.X.X https://review.openstack.org/505657 | 21:27 |
| mordred | tonyb: oh - that reminds me - I wanted to find some good examples to point to to show "look,you can also just do shell snippets if you wnat" | 21:27 |
| jeblair | dmsimard, mordred: gah! job nodes doesn't support groups | 21:27 |
| dmsimard | jeblair: when declaring a nodeset, nodes and groups are at the same level -- thus I did the same thing at the job level. | 21:28 |
| dmsimard | doh | 21:28 |
| mordred | jeblair: yup. I was just about to say the same thing | 21:28 |
| jeblair | i think that's a bug; they were intended to be identical | 21:28 |
| mordred | jeblair: it's becaue the node code in the job definition looks for string vs. dict | 21:28 |
| mordred | or string, vs. dict vs. list iirc ... | 21:28 |
| *** armax has joined #openstack-infra | 21:28 | |
| jeblair | well, they don't share an implementation | 21:28 |
| jeblair | so the groups support needs to be reimplemented for job | 21:29 |
| mordred | I thought they did. oh! ok | 21:29 |
| dmsimard | jeblair: should I revert to non-anonymous nodesets or you feel that's a quick fix ? | 21:29 |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Publish static constraints files from the periodic pipeline https://review.openstack.org/505774 | 21:29 |
| *** jascott1 has quit IRC | 21:29 | |
| tonyb | this would be much less public if I just ran the unit tests in a local VM :( | 21:30 |
| jeblair | dmsimard: i think it's important to fix and would prefer to stick with anonymous nodesets | 21:30 |
| dmsimard | sure. Sorry about finding all these bugs left and right :P | 21:30 |
| jeblair | dmsimard: yeah, cut it out already. ;) | 21:30 |
| jeblair | mordred: you want to make the groups change or you want me to? | 21:31 |
| mordred | jeblair: so - looking at the configparser code for this - adding groups should be easy | 21:31 |
| mordred | jeblair: HOWEVER | 21:31 |
| mordred | jeblair: deciding _how_ to add it ... a little trickier | 21:31 |
| jeblair | mordred: yeah. | 21:32 |
| jeblair | https://etherpad.openstack.org/p/mtfUJTlQhx | 21:32 |
| dmsimard | mordred, jeblair: it makes the syntax kinda weird too ? It would look like this? http://paste.openstack.org/show/621571/ | 21:32 |
| *** dave-mccowan has quit IRC | 21:32 | |
| ianw | pabelanger: catching up ... yolanda may have done a dib release, not sure yet | 21:32 |
| *** jascott1 has joined #openstack-infra | 21:33 | |
| dmsimard | jeblair: yeah, having groups at the same level as the nodes "replicates" the same syntax as within the nodeset | 21:33 |
| *** kong_ has joined #openstack-infra | 21:33 | |
| *** kong_ is now known as kong | 21:33 | |
| *** Sukhdev has joined #openstack-infra | 21:34 | |
| jlvillal | Any idea why a job today the releasenotes job link I get a 404 error when I click it? https://review.openstack.org/#/c/493527/ | 21:35 |
| *** jascott1 has quit IRC | 21:35 | |
| jlvillal | Hopefully a random failure. But I thought I would mention it. | 21:35 |
| dirk | pabelanger: could you review https://review.openstack.org/#/c/502316/ ? | 21:36 |
| ianw | jlvillal: that's quite odd -- the rsync failed according to http://logs.openstack.org/27/493527/5/check/gate-ironic-releasenotes/2678b38/_zuul_ansible/ansible_log.txt | 21:38 |
| ianw | but the job didn't fail | 21:38 |
| jlvillal | ianw: Yeah. It is odd. | 21:38 |
| *** jascott1 has joined #openstack-infra | 21:39 | |
| pabelanger | ianw: clarkb: mordred: ya, looks like just limited to xenial. So possible regression some place I think | 21:41 |
| pabelanger | trusty is good | 21:41 |
| ianw | pabelanger: this is ubuntu-minimal not finding root? | 21:42 |
| ianw | there hasn't been a dib release | 21:43 |
| *** jascott1 has quit IRC | 21:43 | |
| dmsimard | Gerrit is still very much behind in emails, is that sorted out yet ? Receiving emails from yesterday. | 21:43 |
| jeblair | dmsimard: it's basically what clarkb, pabelanger, mnaser, and i have been working on all day :( | 21:43 |
| dmsimard | Doh, sorry, been half afk most of the day | 21:44 |
| *** wolverineav has quit IRC | 21:44 | |
| pabelanger | ianw: specifically, xenial. Trusty appears to boot fine | 21:44 |
| pabelanger | [ 1.131253] VFS: Cannot open root device "LABEL=cloudimg-rootfs" or unknown-block(0,0): error -6 | 21:44 |
| pabelanger | 2017-09-12 18:16:17,764 INFO nodepool.image.build.ubuntu-xenial: /tmp/in_target.d/finalise.d/50-bootloader: line 150: [: =~: binary operator expected | 21:47 |
| pabelanger | ianw: ^ seems wrong | 21:47 |
| ianw | pabelanger: yeah, ok, that's what yolanda mentioned. apparently it works if that is switched to /dev/sda1 , but the label is there | 21:47 |
| ianw | pabelanger: ahh i think that's fixed but not released, that was a ppc match iirc | 21:48 |
| *** efoley has quit IRC | 21:48 | |
| pabelanger | ianw: ah, so yolanda had the issue too? | 21:48 |
| jeblair | mordred, dmsimard: i think that's 3 votes for #4? | 21:48 |
| mordred | yah | 21:48 |
| mordred | jeblair: I think you'll likely be quicker implementing #4 and I'm getting close to EOD here | 21:49 |
| jeblair | mordred: okay, i'll take it. | 21:49 |
| mordred | jeblair: although if you have not implemented it in the morning, I can jump on it first thing | 21:49 |
| ianw | pabelanger: yeah, and it would explain the nodepool job failure. i'm just going to context switch into all this and see what's up locally | 21:49 |
| *** hashar has quit IRC | 21:50 | |
| ianw | since it seems to happen on master and the prior release, i'm suspecting it's upstream that's changed | 21:50 |
| pabelanger | ianw: ah, cool. Ya, it would be good to see what that started failing | 21:50 |
| *** jascott1 has joined #openstack-infra | 21:50 | |
| pabelanger | ianw: yup, I think that make sense | 21:50 |
| *** jascott1 has quit IRC | 21:55 | |
| clarkb | puppety things are in the gate now \o/ just need to be patient | 21:56 |
| *** kzaitsev_pi has quit IRC | 21:57 | |
| ianw | pabelanger: excellent it reproduces in rdocloud, so that's a start | 22:00 |
| jeblair | mordred and i were just chatting in #zuul about some things | 22:02 |
| jeblair | and realized now might be a really good time to answer that question we had about whether docs-draft was still necessary as a separate partition | 22:02 |
| *** dprince has quit IRC | 22:02 | |
| *** kzaitsev_pi has joined #openstack-infra | 22:02 | |
| jeblair | it looks like logs.o.o rotation is currently 30 days? | 22:02 |
| jeblair | docs-draft rotation is 21 | 22:02 |
| clarkb | that sounds right | 22:03 |
| jeblair | if we increased docs-draft rotation to 30, we would be using 427G instead of 299G | 22:03 |
| *** slaweq has quit IRC | 22:03 | |
| jeblair | we have 3.0T available on logs, so we can accomodate that. we can fold docs-draft into logs and gain another 1.5T. | 22:04 |
| *** efoley has joined #openstack-infra | 22:04 | |
| jeblair | so if we just stopped using docs-draft and served it out of logs.o.o, we'd have a net gain of about 1TB of space. | 22:04 |
| mordred | also, if we ditch docs-draft and setting success-url to a subdir of log_url like we are with the v3 tox-docs job currently ... then you can navigate up a url dir to get to your build logs if you need them that way | 22:04 |
| mordred | s/and setting/and started setting/ | 22:04 |
| clarkb | that to me is probably the biggest feature | 22:05 |
| jeblair | yeah, so it's a bit of a ux improvement :) | 22:05 |
| mordred | yah. | 22:05 |
| clarkb | since otherwise you have to magically know its a domain name change | 22:05 |
| mordred | yup | 22:05 |
| mordred | jeblair, clarkb: I think given the UX improvment and the fact that it's actually a net gain of space ... | 22:05 |
| mordred | I'm very in favor of making the change | 22:05 |
| *** thorst has joined #openstack-infra | 22:06 | |
| *** baoli has quit IRC | 22:06 | |
| mordred | I can work up the patches in the morning to unwind the docs-draft addition to v3 and update the migration script to dtrt | 22:06 |
| clarkb | I am in favor too | 22:06 |
| clarkb | mordred: python-six change on centos has mreged | 22:07 |
| jeblair | yeah. so maybe let's send an email to the infra list. then other than that, the implementation is simple: 1) let the v3 transition happen without docs-draft 2) wait 3 weeks. 3) lvextend && resize2fs | 22:07 |
| jeblair | clarkb, mordred: i can send that email if you want | 22:08 |
| clarkb | so that will apply on the next puppet run when we should also apply the next gerrit cache settings | 22:08 |
| clarkb | jeblair: that would be great | 22:08 |
| mordred | clarkb: woot | 22:09 |
| mordred | jeblair: ++ | 22:09 |
| mordred | jeblair: k. I'll send email to the list in the morning when I write the patches | 22:09 |
| jeblair | mordred: oh, i think i'll have it covered in the mail i'm writing now; but feel free if you think more needs to be said. | 22:10 |
| *** thorst has quit IRC | 22:10 | |
| *** Apoorva_ has joined #openstack-infra | 22:11 | |
| mordred | jeblair: cool | 22:11 |
| *** tobiash has quit IRC | 22:11 | |
| *** slaweq has joined #openstack-infra | 22:13 | |
| mnaser | does infra have any suggestions regarding our external dependency to puppet modules hosted at github.com | 22:13 |
| mnaser | we cant use the forge because not all modules are there | 22:13 |
| *** Apoorva has quit IRC | 22:13 | |
| pabelanger | Ya, I'd like to see what on the forge would work, if we setup a reverse proxy cache. Then maybe see why the left over modules aren't published to it | 22:15 |
| clarkb | it also came up at the PTG and general thought seemed to be lets address it on a case by csae basis because otherwise we are caching/mirroring all of github. But also zuulv3 keeps local copies of all repos so it should make it a little better | 22:15 |
| *** bobh has quit IRC | 22:16 | |
| *** bobh has joined #openstack-infra | 22:17 | |
| *** xarses_ has quit IRC | 22:17 | |
| *** slaweq has quit IRC | 22:18 | |
| *** lvdombrk has quit IRC | 22:19 | |
| *** lvdombrk has joined #openstack-infra | 22:19 | |
| *** akscram1 has quit IRC | 22:21 | |
| *** bobh has quit IRC | 22:21 | |
| *** akscram1 has joined #openstack-infra | 22:22 | |
| jeblair | clarkb: all *configured* repos, and we still have to figure out the opt-in process for github repos (and maybe we have a level of configuration where the repo is there but ignored for zuul configuration and reporting). but yeah, that's a potential aid. | 22:22 |
| mordred | jeblair: we should maybe work up an email to the dev list and/or tc about that at some point | 22:23 |
| *** slaweq has joined #openstack-infra | 22:23 | |
| *** Goneri has quit IRC | 22:26 | |
| mnaser | well the reason not all modules are not published on the forge isnt within our control | 22:26 |
| mnaser | the forge doesnt automatically poll for tags in github repos and create releases, the publisher must do that | 22:26 |
| *** esberglu_ has quit IRC | 22:27 | |
| jeblair | clarkb: oh, though, zuul may well still be unable to check out a github project if github is down (it does have checks to make sure its cache is up to date). granted, maybe we can make those non-fatal in some circumstances, so there may be a workable solution there. | 22:27 |
| *** slaweq has quit IRC | 22:27 | |
| *** yamamoto has joined #openstack-infra | 22:28 | |
| clarkb | jeblair: ya I also think having a local clone in one place that is updated then pushed is likely to be more reliable than git cloning from scratch everywhere | 22:28 |
| *** felipemonteiro__ has quit IRC | 22:29 | |
| jeblair | clarkb: yep | 22:29 |
| jeblair | clarkb: i think this can be really useful, but we're probably a few steps away from where we need to be before that's fully robust. | 22:29 |
| clarkb | ya | 22:29 |
| clarkb | also I think that the gerrit settings update may timeout on the centos 7 job in the gate | 22:30 |
| clarkb | I'm going to be bad and reboot the node | 22:30 |
| clarkb | so that we don't have to go through everything again | 22:31 |
| jeblair | i thought you were going to bed which does sound like a good idea to be honest | 22:31 |
| jeblair | it's kind of one of those days | 22:31 |
| clarkb | and speaking of the devil its hung on github puppet things | 22:31 |
| clarkb | hrm zuul doesn't seem to have noticed (eg job isn't restarted, guessing ssh has to time out?) | 22:32 |
| clarkb | maybe there was a better way to do that... | 22:32 |
| clarkb | jeblair: yes one of those weeks | 22:33 |
| *** thorst has joined #openstack-infra | 22:33 | |
| *** thorst has quit IRC | 22:38 | |
| *** thorst has joined #openstack-infra | 22:39 | |
| clarkb | clearly what I thought was a good idea was not as that job is still stuck there | 22:39 |
| clarkb | jeblair: can we kick the process on the zuul launcher side? | 22:39 |
| *** efoley has quit IRC | 22:39 | |
| *** bcat_ has joined #openstack-infra | 22:40 | |
| *** thorst has quit IRC | 22:40 | |
| jeblair | clarkb: should be able to kill the associated ansible proc | 22:42 |
| clarkb | ok /me will go looking | 22:42 |
| clarkb | will that cause the job to restart or to fail? (at this point fail would be good) | 22:43 |
| ianw | pabelanger: ok ... so it comes down to initramfs not being created. kernel can't mount root by LABEL=, that's initramfs's job, hence the issue. will dig a bit more to see what changed, but i think we got it | 22:43 |
| ianw | yolanda: ^ | 22:43 |
| *** tpsilva has quit IRC | 22:44 | |
| mnaser | yeah we've been seeing puppet jobs fail due to github a bunch of times today | 22:44 |
| mnaser | jeblair clarkb i think as an interesting solution is that we always checkout tags, zuul can check if it has a local copy of the requested tag and not even try to contact github in the case | 22:45 |
| mnaser | because we never pull master from github, only tagged commits | 22:45 |
| clarkb | oh sweet job is rerunning | 22:46 |
| *** Sree has joined #openstack-infra | 22:47 | |
| *** ijw has quit IRC | 22:47 | |
| *** ijw has joined #openstack-infra | 22:48 | |
| clarkb | mordred: git02-08 failed this last run but 01 did not, and I think the puppet-jeepyb change should've been in | 22:50 |
| openstackgerrit | Merged openstack-infra/system-config master: Bump gerrit account caches to 32768 https://review.openstack.org/505677 | 22:51 |
| clarkb | mordred: I think it is complaining about uninstalling python-six because things need it | 22:51 |
| clarkb | \o/ finally that gets in | 22:51 |
| *** Sree has quit IRC | 22:52 | |
| clarkb | mordred: hack fix is to manually remove it from each of those nodes and then let puppet run next time happily | 22:52 |
| clarkb | mordred: anymore I'm willing to consider that just to get everything moving | 22:52 |
| *** ianychoi_ is now known as ianychoi | 22:52 | |
| *** jistr has joined #openstack-infra | 22:53 | |
| *** cody-somerville has quit IRC | 22:56 | |
| clarkb | I'm going to send a followup update on gerrit things to mailing list. Running out of steam today, and thinking might be best to just get everyone up to date then pick back up in the morning | 22:58 |
| *** rbrndt has quit IRC | 22:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job.nodeset parameter to supercede job.nodes https://review.openstack.org/505845 | 23:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove Job.nodes https://review.openstack.org/505846 | 23:01 |
| *** jistr has quit IRC | 23:02 | |
| *** threestrands has joined #openstack-infra | 23:02 | |
| *** jistr has joined #openstack-infra | 23:04 | |
| *** xarses_ has joined #openstack-infra | 23:04 | |
| *** claudiub|2 has quit IRC | 23:10 | |
| *** gouthamr has joined #openstack-infra | 23:11 | |
| clarkb | #status log Please don't merge any new project creation changes until mordred gives the go ahead. We have new puppet problems on the git backends and there are staged jeepyb changes we want to watch before opening the flood gates | 23:17 |
| openstackstatus | clarkb: finished logging | 23:17 |
| clarkb | ok just sent a novel update on gerrit situation | 23:18 |
| clarkb | infra-root is anyone left today? I think I would like to go ahead and manually apply jeblair's cache numbers change and restart gerrit | 23:20 |
| clarkb | otherwise it will bug me | 23:20 |
| ianw | clarkb: i'm here | 23:20 |
| ianw | but am a bit behind on what's happening | 23:21 |
| clarkb | ianw: basically upstream said you need to have all your accounts in the accounts cache so we need to increase the size of our accounts cache | 23:21 |
| clarkb | unfortunately while the change to puppet hsa merged for that puppet won't run on review.o.o until the git backend hosts are puppeting again (currently not due to reasons) | 23:22 |
| ianw | so it's no so much a cache but an index? | 23:22 |
| clarkb | rather than wait on that I'd like to get gerrit running with the new config as email is important to some users | 23:22 |
| clarkb | ianw: ya... | 23:22 |
| clarkb | ianw: the config calls it a cache but if you don't have it a major feature of gerrit apparently can't work | 23:22 |
| clarkb | and the default is set to 1k | 23:22 |
| clarkb | and we have 26k accounts | 23:22 |
| ianw | so it's just a stop, backup index (for sanity), kick puppet, restart? | 23:23 |
| clarkb | so I will be applying https://review.openstack.org/#/c/505677 by hand and restarting gerrit | 23:23 |
| clarkb | in this case I don't think I would even backup the index | 23:23 |
| clarkb | since it is just a config change (and we've now shown multiple times that puppet applies cleanly when it runs) | 23:23 |
| clarkb | and no puppet kick | 23:23 |
| *** bobh has joined #openstack-infra | 23:23 | |
| clarkb | I'd just edit gerrit.config and stop start | 23:24 |
| ianw | oh right, not puppeting currently | 23:24 |
| clarkb | jeblair: its your change you ok with ^ | 23:25 |
| *** bcat_ has quit IRC | 23:25 | |
| *** ijw has quit IRC | 23:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job.nodeset parameter to supercede job.nodes https://review.openstack.org/505845 | 23:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove Job.nodes https://review.openstack.org/505846 | 23:27 |
| jeblair | clarkb: o/ | 23:27 |
| clarkb | jeblair: tldr is I want to manually edit gerrit.config with your change and restart gerrit nowish | 23:27 |
| clarkb | that way we can get that done and watch it tomorrow while fixing git backed puppeting | 23:27 |
| *** gouthamr has quit IRC | 23:27 | |
| jeblair | clarkb: ya, i'm at that point as well | 23:28 |
| clarkb | ianw: jeblair ok I will go ahead and do that now then | 23:28 |
| jeblair | clarkb: and am focused on this now if you need help | 23:28 |
| clarkb | ok | 23:28 |
| prometheanfire | what's the mail queue at if you don't mind my asking? | 23:28 |
| clarkb | 8500 or so | 23:28 |
| jeblair | and we will lose it on the restart | 23:29 |
| prometheanfire | nice | 23:29 |
| prometheanfire | lol | 23:29 |
| prometheanfire | that's only 6 days at an email a min | 23:29 |
| jeblair | (and we've already lost thousands of others) | 23:29 |
| ianw | clarkb: excellent ... i'll just be on standby if i can help, thanks for the updates and email too :) | 23:29 |
| *** dizquierdo has quit IRC | 23:30 | |
| clarkb | ok config is edited restarting now | 23:30 |
| clarkb | stop and start are so slow now | 23:32 |
| clarkb | still waiting on start | 23:32 |
| clarkb | nothing in the log yet either, I wonder if it is prepopulating that entire cache on start | 23:32 |
| clarkb | service gerrit start says it failed | 23:33 |
| clarkb | but there is a gerrit process running and that process logged it is ready... | 23:33 |
| jeblair | clarkb: it *is* querying all accounts | 23:33 |
| clarkb | I think it must have a timeout in the init script | 23:33 |
| jeblair | up to 12k | 23:33 |
| clarkb | so I think we are fine, but just to know that that may happen | 23:33 |
| jeblair | of course, there may just be an email in the queue | 23:34 |
| *** xarses_ has quit IRC | 23:34 | |
| jeblair | it's also querying a lot of changes | 23:34 |
| clarkb | it is up and serving things to me | 23:34 |
| jeblair | changes may be from normal traffic | 23:34 |
| clarkb | so I think I am going to ignore the fact that the init script thought it failed | 23:34 |
| jeblair | ++ | 23:34 |
| clarkb | there are now 15k total tasks but the vast majority are replication | 23:35 |
| jeblair | and yeah, we have an email queue of 14 now | 23:35 |
| jeblair | we'll see what happens once it hits the top of the account list | 23:35 |
| jeblair | which i think just happened | 23:36 |
| jeblair | (almost 27k accounts, btw) | 23:36 |
| jeblair | email queue is 0 now | 23:36 |
| jeblair | so *now* the weird queries i'm seeing ale selecitng the same change over and over again | 23:37 |
| jeblair | changes | | | | | 23:37 |
| jeblair | that's the change cache row | 23:38 |
| clarkb | meaning it isn't caching any changes yet? | 23:38 |
| jeblair | it certainly looks like there isn't a change cache | 23:38 |
| clarkb | navigating to and from a couple changes doesn't feel particularly slow | 23:39 |
| * mnaser would be more than happy to help fix puppet things | 23:39 | |
| clarkb | did we have entries in that cache before? | 23:39 |
| clarkb | I don't have the old cache pastes handy | 23:39 |
| jeblair | good question | 23:39 |
| jeblair | i'll dig up irc logs | 23:39 |
| *** ijw has joined #openstack-infra | 23:39 | |
| clarkb | 14k total tasks now and email is sitting at 0 | 23:39 |
| jeblair | oh i think i pasted to #gerrit which hasn't said anything since yesterday | 23:39 |
| jeblair | http://paste.openstack.org/show/621488/ | 23:40 |
| clarkb | it is empty there too, odd but we didn't change that behavior | 23:40 |
| jeblair | i tcpdumped for, idunno, maybe 30 seconds | 23:40 |
| clarkb | and email seems to work so thats a plus | 23:40 |
| jeblair | it queried change 505845 1190 times during that window | 23:40 |
| jeblair | it's not 100,000 queries | 23:41 |
| jeblair | but it's still bonkers | 23:41 |
| *** thorst has joined #openstack-infra | 23:41 | |
| clarkb | ya | 23:41 |
| *** vhosakot has quit IRC | 23:42 | |
| jeblair | and the exim log shows we're sending email at a good clip now | 23:42 |
| *** hongbin has quit IRC | 23:44 | |
| * clarkb makes a note to look at gerrit init script to see why it thinks it failed | 23:44 | |
| jeblair | i asked in #gerrit whether there's actually a change cache | 23:44 |
| *** caphrim007 has quit IRC | 23:44 | |
| clarkb | jeblair: thanks. also yay for working email | 23:45 |
| jeblair | haha | 23:45 |
| jeblair | i was just trying to track down MERGER_FAILURES in zuul v3 | 23:46 |
| jeblair | right hand, meet left. | 23:46 |
| jeblair | (gerrit was restarted) | 23:46 |
| clarkb | :) | 23:46 |
| *** thorst has quit IRC | 23:46 | |
| clarkb | I'm going to cross email off our list | 23:46 |
| jeblair | \o/ | 23:47 |
| smcginnis | Nice work all. | 23:48 |
| rm_work | does the new gerrit take a lot longer to push things around in git now? | 23:49 |
| rm_work | trying to do a cherry-pick for a patch i just pushed up, getting "Couldn't find remote ref" for it | 23:50 |
| rm_work | noticed the same thing a few times today and yesterday | 23:50 |
| rm_work | it takes a *while* | 23:50 |
| clarkb | rm_work: that is beacuse we just restarted gerrit and so it replicates everything | 23:50 |
| rm_work | ah k | 23:50 |
| clarkb | typically we don't restart gerrit often so you don't notice | 23:50 |
| rm_work | alright, so for now, I just need to wait for a bit | 23:50 |
| clarkb | but we have the last couple days working out some of these last prolems post upgrade | 23:50 |
| clarkb | ya | 23:50 |
| rm_work | kk | 23:51 |
| clarkb | and my internet connection just started having high packet loss | 23:51 |
| clarkb | I think that is a sign it is time to stop | 23:51 |
| *** bobh has quit IRC | 23:52 | |
| smcginnis | It's trying to tell you to take a well earned break. | 23:53 |
| clarkb | rm_work: after a restart we ened up with ~16k total tasks for replication, currnetly down to ~9k | 23:53 |
| clarkb | rm_work: so in another half hour or so we should be caught up | 23:53 |
| *** tosky has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!