| *** kgiusti has left #openstack-infra | 00:01 | |
| *** slaweq has joined #openstack-infra | 00:02 | |
| kfox1111 | is something up with zuul? | 00:02 |
|---|---|---|
| jlk | kfox1111: a gerrit restart happened | 00:02 |
| kfox1111 | I submitted an update to #505761 a while ago and it sin't showing up on the board. | 00:03 |
| kfox1111 | should I submit another revision then? | 00:03 |
| rm_work | kfox1111: i think just wait? if it doesn't show up in ~30m then try a recheck or something | 00:06 |
| clarkb | I would recheck | 00:06 |
| *** bobh has joined #openstack-infra | 00:06 | |
| clarkb | you probably raced zuul reconnecting to get the event? | 00:06 |
| *** slaweq has quit IRC | 00:06 | |
| *** askb has quit IRC | 00:06 | |
| kfox1111 | k. thanks. | 00:08 |
| *** Apoorva_ has quit IRC | 00:08 | |
| *** Apoorva has joined #openstack-infra | 00:08 | |
| *** ociuhandu_ has joined #openstack-infra | 00:12 | |
| *** ociuhandu has quit IRC | 00:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: test nodeset error https://review.openstack.org/505856 | 00:15 |
| *** thorst has joined #openstack-infra | 00:16 | |
| *** olaph1 has joined #openstack-infra | 00:17 | |
| *** olaph has quit IRC | 00:18 | |
| jeblair | clarkb: changes cache has to be explicitly enabled | 00:19 |
| jeblair | clarkb: (because it doesn't work in a multi-master setup) | 00:19 |
| clarkb | huh more googley things leaking through (multi master isn't a thing anywhere else aiui) | 00:20 |
| clarkb | ? | 00:20 |
| mnaser | i know everyone has had a long day but looks like zuul is having problems | 00:21 |
| mnaser | http://logs.openstack.org/09/503909/2/check/gate-puppet-openstack-integration-4-scenario002-tempest-centos-7/ec14759/console.html | 00:21 |
| mnaser | "fatal: read error: Connection reset by peer" | 00:21 |
| mnaser | happening for all jobs so seems like an issue with git.openstack.org | 00:22 |
| clarkb | ugh | 00:22 |
| clarkb | the thing to do here is manually hit each of all the git.o.o backends to try and find the problem | 00:23 |
| *** ijw has quit IRC | 00:23 | |
| clarkb | git://git0X.openstack.org:29418/ is base url for X = 1-8 | 00:23 |
| * clarkb does this | 00:23 | |
| *** ijw has joined #openstack-infra | 00:23 | |
| kfox1111 | yeah. I'm seeing that too. | 00:24 |
| clarkb | ok they all work from here implying it isn't catastrophic failure of the backends themselves | 00:25 |
| clarkb | git.openstack.org frontend works as well... | 00:25 |
| *** sbezverk has quit IRC | 00:25 | |
| mnaser | eh | 00:25 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=862&rra_id=all shows we aren't using a ton of bw (so rax likely isn't rate limiting us) | 00:26 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add initramfs-tools for ubuntu-minimal https://review.openstack.org/505857 | 00:26 |
| *** sbezverk has joined #openstack-infra | 00:26 | |
| mnaser | i wonder if it magically fixed itself | 00:26 |
| mnaser | it seems to clone fine now at least locally | 00:27 |
| ianw | is it worth looking at haproxy logs? | 00:27 |
| clarkb | ianw: ya probably | 00:27 |
| clarkb | ianw: are you able to do that? I'd have to get on other machine with ssh keys to do so | 00:27 |
| ianw | seeing now ... | 00:28 |
| ianw | ok, 158.69.78.193 was the host in question | 00:28 |
| ianw | hmm, nothing in the haproxy logs for that ip today | 00:29 |
| *** Apoorva_ has joined #openstack-infra | 00:30 | |
| ianw | today being utc which ticked over 1/2 hour ago | 00:30 |
| ianw | but anyway, that job failed at 2017-09-21 00:16:27.494415 | 00:30 |
| ianw | ego i don't think it made it to git.o.o | 00:31 |
| ianw | ergo | 00:31 |
| *** zhurong has joined #openstack-infra | 00:31 | |
| *** slaweq has joined #openstack-infra | 00:32 | |
| *** Apoorva has quit IRC | 00:34 | |
| *** Apoorva_ has quit IRC | 00:34 | |
| ianw | hmm, logstash shows a few issues | 00:35 |
| ianw | http://logs.openstack.org/1d/1de54e20f63e429aa0c811b61a03ee42e39311d8/post/nova-branch-tarball/403c96c/console.html <- inap, so different region | 00:36 |
| *** baoli has joined #openstack-infra | 00:36 | |
| *** baoli has quit IRC | 00:36 | |
| *** slaweq has quit IRC | 00:36 | |
| jeblair | gerritbot is awol | 00:37 |
| ianw | again that didn't hit haproxy | 00:37 |
| clarkb | jeblair: I wonder if we lost it in the restart | 00:38 |
| ianw | i wonder if git.o.o had a connection blip? | 00:38 |
| jeblair | looking | 00:38 |
| clarkb | ianw: considering it is multiple regions external to the datacenter it lives in that seems likely | 00:38 |
| jeblair | gerritbot getting events; it may just be split | 00:38 |
| clarkb | ah | 00:38 |
| *** openstackgerrit has quit IRC | 00:41 | |
| *** openstackgerrit has joined #openstack-infra | 00:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 00:41 |
| jeblair | openstackgerrit: welcome back | 00:42 |
| *** slaweq has joined #openstack-infra | 00:42 | |
| jeblair | i restarted it | 00:42 |
| *** Swami has quit IRC | 00:45 | |
| *** slaweq has quit IRC | 00:46 | |
| *** Sree has joined #openstack-infra | 00:48 | |
| *** cuongnv has joined #openstack-infra | 00:50 | |
| *** hichihara has joined #openstack-infra | 00:50 | |
| *** slaweq has joined #openstack-infra | 00:51 | |
| *** LindaWang has joined #openstack-infra | 00:52 | |
| *** Sree has quit IRC | 00:52 | |
| ianw | clarkb / mnaser : here's the hits on haproxy per-second from 00:00 utc to now -> https://imgur.com/a/nCsjq | 00:54 |
| ianw | it seems like we had a blip to me | 00:54 |
| *** thorst has quit IRC | 00:54 | |
| *** slaweq has quit IRC | 00:56 | |
| mnaser | ianw yeah a few big dropoffs.. weird | 00:59 |
| *** slaweq has joined #openstack-infra | 01:01 | |
| *** slaweq has quit IRC | 01:08 | |
| rm_work | does openstackgerrit take a while to start doing stuff again in all channels? | 01:17 |
| rm_work | i see you restarted em | 01:17 |
| clarkb | it has more channels than the freenode limit so does a least recently used rotation of channel joins | 01:18 |
| clarkb | it should join and report when it has events to report then may go away if your channel is least recently used and some other channels event comes in | 01:19 |
| *** hichihara has quit IRC | 01:20 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 01:20 |
| *** dave-mccowan has joined #openstack-infra | 01:21 | |
| rm_work | hmm | 01:22 |
| rm_work | it appears to have been in the channel the whole time | 01:22 |
| rm_work | but didn't comment | 01:22 |
| rm_work | might have just raced the restart | 01:22 |
| rm_work | hmmm no because i see the restart part/join there | 01:22 |
| rm_work | and then e said something, and then ... missed a commit | 01:23 |
| rm_work | weird | 01:23 |
| rm_work | ah well | 01:23 |
| *** cshastri has joined #openstack-infra | 01:25 | |
| *** namnh has joined #openstack-infra | 01:27 | |
| rm_work | ping again for my semi-troll but also actually IMO valid https://review.openstack.org/#/c/430393/ :) | 01:28 |
| *** jdandrea_ has joined #openstack-infra | 01:28 | |
| clarkb | ya I looked at that myself and a lot of the time is spent in osc | 01:30 |
| clarkb | I think there was diacussion of not using entrypoints for built in commands to speed those up at least | 01:30 |
| clarkb | anwyays already has my +2 | 01:30 |
| rm_work | yeah | 01:31 |
| *** dave-mcc_ has joined #openstack-infra | 01:33 | |
| ianw | ahh damit networkx 2.0 must have released | 01:34 |
| ianw | and totally broken all their apis | 01:34 |
| harlowja | looks like it, ha | 01:35 |
| harlowja | uploaded on '2017-09-20' (from pypi) | 01:35 |
| harlowja | i'd just ummm do <2.0 | 01:35 |
| harlowja | taskflow i can fix pretty easily | 01:35 |
| harlowja | at some point, lol | 01:35 |
| *** dave-mccowan has quit IRC | 01:35 | |
| ianw | harlowja: yeah, i think this might be a global pin, because it's all different | 01:35 |
| harlowja | ya, looks like they dropped all the old way of doing things | 01:36 |
| ianw | yep it wasn't too hard to update, just a pain | 01:37 |
| openstackgerrit | Michael Johnson proposed openstack-infra/project-config master: Remove Octavia dashboard projects integration jobs https://review.openstack.org/505904 | 01:38 |
| harlowja | damn people in https://github.com/networkx/networkx/blob/master/CONTRIBUTORS.rst lol | 01:38 |
| * harlowja makes sure my name isn't in there, lol | 01:38 | |
| jeblair | i restarted zuulv3 to pick up the, erm, don't infinite loop fix. | 01:38 |
| *** Sukhdev has quit IRC | 01:39 | |
| ianw | harlowja: https://review.openstack.org/#/c/505905/ | 01:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Report friendly errors when nodeset/secrets missing https://review.openstack.org/505856 | 01:40 |
| *** bobh has quit IRC | 01:43 | |
| *** yamahata has quit IRC | 01:46 | |
| *** iyamahat has quit IRC | 01:46 | |
| *** mriedem has quit IRC | 01:51 | |
| *** thorst has joined #openstack-infra | 01:53 | |
| *** thorst has quit IRC | 01:53 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 01:53 | |
| *** amotoki has quit IRC | 01:55 | |
| *** amotoki has joined #openstack-infra | 01:56 | |
| *** mrunge has quit IRC | 01:56 | |
| *** strigazi has quit IRC | 01:56 | |
| *** hamzy has quit IRC | 01:57 | |
| *** kazsh has quit IRC | 01:57 | |
| *** dougshelley66 has quit IRC | 01:57 | |
| *** hamzy has joined #openstack-infra | 01:57 | |
| *** ijw has quit IRC | 01:58 | |
| *** strigazi has joined #openstack-infra | 01:58 | |
| *** kazsh has joined #openstack-infra | 01:59 | |
| *** mrunge has joined #openstack-infra | 01:59 | |
| *** ijw has joined #openstack-infra | 02:01 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add initramfs-tools for ubuntu-minimal https://review.openstack.org/505857 | 02:01 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Avoid networkx 2.0 release https://review.openstack.org/505908 | 02:01 |
| *** slaweq has joined #openstack-infra | 02:03 | |
| *** hongbin has joined #openstack-infra | 02:04 | |
| *** ijw has quit IRC | 02:05 | |
| *** slaweq has quit IRC | 02:08 | |
| *** sshnaidm has joined #openstack-infra | 02:13 | |
| *** bobh has joined #openstack-infra | 02:13 | |
| *** cody-somerville has joined #openstack-infra | 02:15 | |
| *** gouthamr has joined #openstack-infra | 02:19 | |
| *** dave-mcc_ has quit IRC | 02:23 | |
| ianw | Unable to connect to mirror.bhs1.ovh.openstack.org | 02:23 |
| ianw | hmmm | 02:23 |
| ianw | it's one of those sort of days | 02:24 |
| *** bobh has quit IRC | 02:25 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Zuul v3 migration test drive: puppet-openstack https://review.openstack.org/505921 | 02:29 |
| dmsimard | mordred, jeblair, clarkb: ^ first stab at a deployment project test drive | 02:29 |
| mordred | clarkb: I'll cleanup the git0* stuff in the morning - agree about the just manually removing for now | 02:30 |
| johnsom | Yeah, just saw a job bomb with ovh... | 02:31 |
| ianw | mordred: do you know how to log into the ovh web site? "console url ..." doesn't seem to work | 02:31 |
| mordred | ianw: I do not :( | 02:31 |
| ianw | well then i can't really see what's on the console of the ovh mirror, if anything | 02:31 |
| ianw | so is the region dead, or just the host? | 02:31 |
| *** efried has quit IRC | 02:31 | |
| ianw | guesses welcome :) | 02:31 |
| johnsom | Looking | 02:32 |
| *** efried has joined #openstack-infra | 02:32 | |
| ianw | openstack console url show "mirror-host" returns a 500 | 02:32 |
| ianw | (Request-ID: req-5da4cba2-efe8-4dfb-a8a7-faf490075c89) if that helps | 02:32 |
| johnsom | ubuntu-xenial-ovh-bhs1-11044167 could not get to the mirror mirror.bhs1.ovh.openstack.org | 02:32 |
| johnsom | Same with ubuntu-xenial-ovh-bhs1-11044166 | 02:33 |
| clarkb | we can set min ready to 0 there until its sorted out | 02:33 |
| clarkb | er max servers | 02:34 |
| *** yolanda has quit IRC | 02:34 | |
| *** yolanda has joined #openstack-infra | 02:35 | |
| ianw | clarkb: yep, unless you've got other ideas on how i can debug the mirror? putting hte obvious things in the boxes on the ovh website doesn't let me in | 02:35 |
| johnsom | ubuntu-xenial-ovh-bhs1-11044169 too, so pretty safe to say if the job uses the mirror it's dead... | 02:36 |
| *** kuromagi^ has quit IRC | 02:36 | |
| ianw | you really don't need fancy monitoring systems. just give infra some nodes and an irc regex and we'll do it all for you ... | 02:37 |
| clarkb | I've got nothibg unless it waskilled like before | 02:37 |
| clarkb | could try to boot a new one | 02:38 |
| johnsom | ianw lol. We considered a canary as are mascot as we always seem to find the issues.... | 02:38 |
| ianw | clarkb: says it's active. i'll try a reboot, if nothing happens i'll put np in emergency and hand-update out ovh-bhs1 | 02:39 |
| clarkb | ok | 02:39 |
| *** kuromagi has joined #openstack-infra | 02:40 | |
| ianw | hard reboot applied ... give it a minute | 02:40 |
| ianw | it's status is still HARD_REBOOT | 02:42 |
| ianw | still. ok, i'm going to call it stuck. will reduce region and send an email; after ovh have diagnosed it if we need to rebuild we can | 02:44 |
| clarkb | ++ | 02:44 |
| *** ramishra has joined #openstack-infra | 02:46 | |
| ianw | #status log OVH-BHS1 mirror has disappeared unexpectedly. did not respond to hard reboot. nodepool.o.o in emergency file and region max-servers set to 0 | 02:50 |
| openstackstatus | ianw: finished logging | 02:50 |
| *** thorst has joined #openstack-infra | 02:53 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Zuul v3 migration test drive: puppet-openstack https://review.openstack.org/505921 | 02:54 |
| dmsimard | ianw: I see "emergency file" being mentioned a lot. What is it ? | 02:55 |
| ianw | dmsimard: just the file that stops puppet running on the host | 02:55 |
| dmsimard | Also, what is the wiki status page again ? | 02:55 |
| ianw | https://wiki.openstack.org/wiki/Infrastructure_Status | 02:55 |
| clarkb | its an ansible grouo we prevent ansible from running against | 02:55 |
| dmsimard | ianw: thanks :D | 02:55 |
| dmsimard | there's no v3 coverage on project-config? :( | 02:57 |
| *** thorst has quit IRC | 02:59 | |
| *** kzaitsev_pi has quit IRC | 03:00 | |
| *** lbragstad has joined #openstack-infra | 03:00 | |
| *** kzaitsev1pi has joined #openstack-infra | 03:00 | |
| *** baoli has joined #openstack-infra | 03:01 | |
| *** slaweq has joined #openstack-infra | 03:02 | |
| *** slaweq has quit IRC | 03:07 | |
| *** tobiash has joined #openstack-infra | 03:10 | |
| dmsimard | mordred: omg the generated bash is not bashate compliant | 03:12 |
| dmsimard | how dare you | 03:12 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 03:13 |
| *** rwsu has quit IRC | 03:14 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Exclude playbooks/legacy from bashate linter https://review.openstack.org/505926 | 03:19 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Zuul v3 migration test drive: puppet-openstack https://review.openstack.org/505921 | 03:19 |
| *** lbragstad has quit IRC | 03:21 | |
| ianw | if it's indents, the latest version of bashate does a much better job of that | 03:21 |
| *** armax has quit IRC | 03:22 | |
| dmsimard | ianw: we can't really vouch that the bash things migrated from JJB builders will be bashate compliant and I don't think we'll want to go through each JJB builder to make sure they are :) | 03:22 |
| ianw | oh no, definitely not | 03:22 |
| dmsimard | thus, my patch ^ | 03:23 |
| *** baoli has quit IRC | 03:23 | |
| dmsimard | ianw: btw this is ready for another go after a round of jeblair's comments https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul-jobs+branch:master+topic:zuulv3-multinode | 03:24 |
| *** Sukhdev has joined #openstack-infra | 03:24 | |
| dmsimard | The integration tests for these ( https://review.openstack.org/#/q/status:open+project:openstack-infra/openstack-zuul-jobs+branch:master+topic:zuulv3-multinode ) are currently blocked on https://review.openstack.org/#/c/505845/ | 03:25 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 03:29 |
| *** rwsu has joined #openstack-infra | 03:31 | |
| *** jdandrea_ has quit IRC | 03:35 | |
| *** coolsvap has joined #openstack-infra | 03:37 | |
| *** udesale has joined #openstack-infra | 03:41 | |
| *** slaweq has joined #openstack-infra | 03:42 | |
| *** Sree has joined #openstack-infra | 03:46 | |
| *** slaweq has quit IRC | 03:47 | |
| *** yamahata has joined #openstack-infra | 03:50 | |
| *** rwsu has quit IRC | 03:50 | |
| *** slaweq has joined #openstack-infra | 03:52 | |
| *** thorst has joined #openstack-infra | 03:55 | |
| *** slaweq has quit IRC | 03:56 | |
| *** thorst has quit IRC | 03:59 | |
| *** iyamahat has joined #openstack-infra | 04:00 | |
| *** erlon has quit IRC | 04:03 | |
| *** hongbin has quit IRC | 04:06 | |
| *** rwsu has joined #openstack-infra | 04:09 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Support networkx 2.0 https://review.openstack.org/505963 | 04:09 |
| *** cody-somerville has quit IRC | 04:11 | |
| *** cody-somerville has joined #openstack-infra | 04:12 | |
| *** cody-somerville has joined #openstack-infra | 04:12 | |
| *** gouthamr has quit IRC | 04:12 | |
| *** rwsu has quit IRC | 04:21 | |
| *** cody-somerville has quit IRC | 04:26 | |
| *** jrist has joined #openstack-infra | 04:31 | |
| *** ykarel| has joined #openstack-infra | 04:37 | |
| *** stakeda has joined #openstack-infra | 04:37 | |
| *** claudiub|2 has joined #openstack-infra | 04:40 | |
| *** psachin has joined #openstack-infra | 04:41 | |
| *** rwsu has joined #openstack-infra | 04:41 | |
| *** links has joined #openstack-infra | 04:43 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add initramfs-tools for ubuntu-minimal https://review.openstack.org/505857 | 04:43 |
| *** Rockyg has joined #openstack-infra | 04:46 | |
| *** ykarel has joined #openstack-infra | 04:46 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Don't try to update cloud-init anymore https://review.openstack.org/504850 | 04:47 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add initramfs-tools for ubuntu-minimal/pin networkx <2.0 https://review.openstack.org/505857 | 04:49 |
| *** ykarel| has quit IRC | 04:49 | |
| *** thorst has joined #openstack-infra | 04:55 | |
| *** thorst has quit IRC | 05:00 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Use upper-constraints in tox tests https://review.openstack.org/505975 | 05:00 |
| *** pcaruana has joined #openstack-infra | 05:09 | |
| *** e0ne has joined #openstack-infra | 05:16 | |
| *** markvoelker has quit IRC | 05:16 | |
| *** markvoelker has joined #openstack-infra | 05:17 | |
| *** rwsu has quit IRC | 05:18 | |
| *** e0ne has quit IRC | 05:20 | |
| *** markvoelker has quit IRC | 05:22 | |
| *** Sukhdev has quit IRC | 05:23 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Update roadmap in the README https://review.openstack.org/502188 | 05:26 |
| *** jtomasek has joined #openstack-infra | 05:30 | |
| *** rwsu has joined #openstack-infra | 05:33 | |
| *** udesale__ has joined #openstack-infra | 05:35 | |
| *** udesale has quit IRC | 05:38 | |
| *** rwsu has quit IRC | 05:39 | |
| *** iyamahat has quit IRC | 05:45 | |
| *** udesale has joined #openstack-infra | 05:50 | |
| *** slaweq has joined #openstack-infra | 05:51 | |
| *** udesale__ has quit IRC | 05:52 | |
| *** slaweq has quit IRC | 05:55 | |
| *** thorst has joined #openstack-infra | 05:56 | |
| openstackgerrit | zhangyangyang proposed openstack-dev/cookiecutter master: Cleanup test-requirements https://review.openstack.org/505987 | 05:58 |
| *** pgadiya has joined #openstack-infra | 05:59 | |
| *** aeng has quit IRC | 06:01 | |
| *** thorst has quit IRC | 06:01 | |
| *** Rockyg has quit IRC | 06:03 | |
| *** Douhet has quit IRC | 06:06 | |
| *** Douhet has joined #openstack-infra | 06:07 | |
| AJaeger | ianw, yolanda , could you review the stack starting at https://review.openstack.org/#/c/505762/ , please? That's the final mitaka branch cleanup | 06:08 |
| *** claudiub has joined #openstack-infra | 06:10 | |
| *** matbu has joined #openstack-infra | 06:11 | |
| matbu | morning | 06:11 |
| *** claudiub|2 has quit IRC | 06:12 | |
| *** sshnaidm has quit IRC | 06:14 | |
| *** iyamahat has joined #openstack-infra | 06:19 | |
| *** florianf has joined #openstack-infra | 06:19 | |
| *** andreas_s has joined #openstack-infra | 06:21 | |
| *** thorre has quit IRC | 06:22 | |
| *** rcernin has joined #openstack-infra | 06:28 | |
| *** udesale has quit IRC | 06:29 | |
| *** dhajare has joined #openstack-infra | 06:30 | |
| *** udesale has joined #openstack-infra | 06:30 | |
| *** martinkopec has joined #openstack-infra | 06:30 | |
| *** ykarel_ has joined #openstack-infra | 06:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Be more descriptive when publishing service-types https://review.openstack.org/505679 | 06:36 |
| *** ykarel has quit IRC | 06:36 | |
| *** thorre has joined #openstack-infra | 06:36 | |
| *** dhajare has quit IRC | 06:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: [accessbot] Default to running a meetbot in all channels https://review.openstack.org/505669 | 06:41 |
| *** ralonsoh has joined #openstack-infra | 06:45 | |
| *** markmcd has quit IRC | 06:50 | |
| *** markmcd1 has joined #openstack-infra | 06:50 | |
| *** dhajare has joined #openstack-infra | 06:50 | |
| openstackgerrit | Merged openstack-infra/puppet-project_config master: Update nodepool_config_file_zuulv3 to use hostname https://review.openstack.org/504500 | 06:51 |
| AJaeger | jhesketh: there are still a few open stable/liberty and stable/mitaka changes - do you want to abandon all of them? https://review.openstack.org/#/q/branch:stable/liberty+status:open and https://review.openstack.org/#/q/branch:stable/mitaka+status:open - except group-based-policy and friends | 06:56 |
| *** thorst has joined #openstack-infra | 06:57 | |
| *** xinliang has quit IRC | 06:57 | |
| *** jpich has joined #openstack-infra | 06:58 | |
| *** upama has joined #openstack-infra | 07:01 | |
| *** thorst has quit IRC | 07:02 | |
| openstackgerrit | zhangyangyang proposed openstack-infra/gear master: Cleanup test-requirements https://review.openstack.org/506017 | 07:04 |
| *** pcaruana has quit IRC | 07:05 | |
| *** dhajare has quit IRC | 07:08 | |
| yamamoto | is there a common practice how to maintain info like "needs backport to xxx branch" in storyboard? | 07:09 |
| *** xinliang has joined #openstack-infra | 07:10 | |
| *** xinliang has quit IRC | 07:10 | |
| *** xinliang has joined #openstack-infra | 07:10 | |
| *** ccamacho has quit IRC | 07:13 | |
| openstackgerrit | zhangyangyang proposed openstack-infra/gerritlib master: Cleanup test-requirements https://review.openstack.org/506021 | 07:15 |
| *** ChanServ sets mode: +o openstack | 07:17 | |
| *** markvoelker has joined #openstack-infra | 07:18 | |
| *** tesseract has joined #openstack-infra | 07:19 | |
| *** jtomasek has quit IRC | 07:23 | |
| *** iyamahat has quit IRC | 07:23 | |
| *** pcaruana has joined #openstack-infra | 07:23 | |
| *** jtomasek has joined #openstack-infra | 07:25 | |
| *** hashar has joined #openstack-infra | 07:29 | |
| *** alexchadin has joined #openstack-infra | 07:29 | |
| *** slaweq has joined #openstack-infra | 07:30 | |
| ianw | amoralej|off: sorry dib release didn't happen today, as gate is broken. see https://review.openstack.org/505857. we're working on it | 07:30 |
| *** kzaitsev_pi has joined #openstack-infra | 07:31 | |
| *** kzaitsev1pi has quit IRC | 07:32 | |
| *** slaweq has quit IRC | 07:34 | |
| *** zhurong has quit IRC | 07:35 | |
| *** dhajare has joined #openstack-infra | 07:38 | |
| *** slaweq has joined #openstack-infra | 07:40 | |
| *** jpena|off is now known as jpena | 07:41 | |
| *** slaweq has quit IRC | 07:44 | |
| *** slaweq has joined #openstack-infra | 07:50 | |
| *** egonzalez has joined #openstack-infra | 07:51 | |
| *** markvoelker has quit IRC | 07:52 | |
| *** mwarad has joined #openstack-infra | 07:53 | |
| *** slaweq has quit IRC | 07:54 | |
| *** zhurong has joined #openstack-infra | 07:55 | |
| openstackgerrit | zhangyangyang proposed openstack-dev/hacking master: Cleanup test-requirements https://review.openstack.org/506032 | 07:56 |
| *** thorst has joined #openstack-infra | 07:58 | |
| *** liujiong has joined #openstack-infra | 08:00 | |
| *** Sree has quit IRC | 08:02 | |
| *** Sree has joined #openstack-infra | 08:02 | |
| *** thorst has quit IRC | 08:02 | |
| *** ykarel__ has joined #openstack-infra | 08:03 | |
| openstackgerrit | zhangyangyang proposed openstack-infra/grafyaml master: Cleanup test-requirements https://review.openstack.org/506035 | 08:04 |
| openstackgerrit | zhangyangyang proposed openstack-infra/glean master: Cleanup test-requirements https://review.openstack.org/506036 | 08:04 |
| *** ykarel_ has quit IRC | 08:06 | |
| *** rwsu has joined #openstack-infra | 08:07 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add initramfs-tools for ubuntu-minimal https://review.openstack.org/505857 | 08:08 |
| jhesketh | AJaeger: huh, that's odd.. I wonder why my script didn't pick them up... I might have to take a look at that tomorrow | 08:09 |
| AJaeger | jhesketh: not critical for tonight - thanks | 08:11 |
| *** dizquierdo has joined #openstack-infra | 08:15 | |
| *** slaweq has joined #openstack-infra | 08:19 | |
| *** slaweq has quit IRC | 08:24 | |
| *** nmathew has joined #openstack-infra | 08:24 | |
| *** electrofelix has joined #openstack-infra | 08:24 | |
| *** amoralej|off is now known as amoralej | 08:29 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add initramfs-tools for ubuntu-minimal https://review.openstack.org/505857 | 08:29 |
| *** shardy has joined #openstack-infra | 08:30 | |
| *** lin_yang has quit IRC | 08:31 | |
| *** ykarel__ is now known as ykarel|lunch | 08:31 | |
| *** kzaitsev1pi has joined #openstack-infra | 08:32 | |
| *** kzaitsev_pi has quit IRC | 08:33 | |
| andreykurilin | hi folks! Does mirroring with github still work? | 08:35 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/irc-meetings master: Remove networking-midonet meeting https://review.openstack.org/506059 | 08:35 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/irc-meetings master: Remove networking-midonet meeting https://review.openstack.org/506059 | 08:37 |
| *** hashar has quit IRC | 08:38 | |
| *** hashar has joined #openstack-infra | 08:39 | |
| *** threestrands has quit IRC | 08:40 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add initramfs-tools for ubuntu-minimal https://review.openstack.org/505857 | 08:42 |
| *** derekh has joined #openstack-infra | 08:43 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add initramfs-tools for ubuntu-minimal https://review.openstack.org/505857 | 08:44 |
| *** dizquierdo has quit IRC | 08:45 | |
| *** markvoelker has joined #openstack-infra | 08:48 | |
| *** dizquierdo has joined #openstack-infra | 08:49 | |
| *** pgadiya has quit IRC | 08:53 | |
| *** pbourke has quit IRC | 08:55 | |
| *** thorst has joined #openstack-infra | 08:59 | |
| andreykurilin | Also, I could pull the latest merged patch via gerrit remote. | 09:01 |
| andreykurilin | *could not | 09:02 |
| *** thorst has quit IRC | 09:03 | |
| *** shardy is now known as shardy_afk | 09:05 | |
| *** pbourke has joined #openstack-infra | 09:20 | |
| *** markvoelker has quit IRC | 09:21 | |
| *** yamamoto has quit IRC | 09:21 | |
| *** yamamoto has joined #openstack-infra | 09:22 | |
| *** tosky has joined #openstack-infra | 09:22 | |
| *** ccamacho has joined #openstack-infra | 09:25 | |
| *** mwarad has quit IRC | 09:27 | |
| *** ykarel|lunch is now known as ykarel | 09:27 | |
| *** alexchadin has quit IRC | 09:34 | |
| *** alexchadin has joined #openstack-infra | 09:35 | |
| *** sambetts|afk is now known as sambetts | 09:36 | |
| *** slaweq has joined #openstack-infra | 09:37 | |
| *** alexchadin has quit IRC | 09:38 | |
| *** tesseract has quit IRC | 09:41 | |
| *** tesseract has joined #openstack-infra | 09:41 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Support networkx 2.0 https://review.openstack.org/505963 | 09:43 |
| openstackgerrit | Merged openstack/diskimage-builder master: Support networkx 2.0 https://review.openstack.org/505963 | 09:43 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: debootstrap: Call update-initramfs explicitly https://review.openstack.org/504588 | 09:44 |
| *** slaweq_ has joined #openstack-infra | 09:48 | |
| *** threestrands has joined #openstack-infra | 09:52 | |
| *** slaweq_ has quit IRC | 09:53 | |
| *** rwsu has quit IRC | 09:54 | |
| *** Sree has quit IRC | 09:54 | |
| *** zhurong has quit IRC | 09:55 | |
| *** dizquierdo has quit IRC | 09:59 | |
| *** cshastri has quit IRC | 09:59 | |
| *** thorst has joined #openstack-infra | 10:00 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 10:01 | |
| *** cuongnv has quit IRC | 10:03 | |
| *** thorst has quit IRC | 10:04 | |
| *** rwsu has joined #openstack-infra | 10:07 | |
| *** alexchadin has joined #openstack-infra | 10:09 | |
| *** alexchadin has quit IRC | 10:10 | |
| *** threestrands has quit IRC | 10:11 | |
| *** zhurong has joined #openstack-infra | 10:11 | |
| *** rwsu has quit IRC | 10:13 | |
| *** e0ne has joined #openstack-infra | 10:13 | |
| *** udesale has quit IRC | 10:16 | |
| *** hashar has quit IRC | 10:16 | |
| *** markvoelker has joined #openstack-infra | 10:19 | |
| *** yee37927 has joined #openstack-infra | 10:24 | |
| *** yee379 has quit IRC | 10:24 | |
| frickler | andreykurilin: replication to github seems to be lagging behind by a couple of hours | 10:29 |
| *** stakeda has quit IRC | 10:30 | |
| andreykurilin | frickler: yup ;( will wait for a fix. thanks | 10:31 |
| odyssey4me | Where can I find the image build definitions for the images built in openstack-infra? | 10:31 |
| odyssey4me | specifically the nodepool images | 10:31 |
| *** rwsu has joined #openstack-infra | 10:32 | |
| *** jesusaur has quit IRC | 10:36 | |
| *** inc0 has quit IRC | 10:36 | |
| *** jkilpatr has quit IRC | 10:38 | |
| *** jesusaur has joined #openstack-infra | 10:39 | |
| *** inc0 has joined #openstack-infra | 10:40 | |
| *** rhallisey has joined #openstack-infra | 10:47 | |
| *** ijw has joined #openstack-infra | 10:50 | |
| cmurphy | odyssey4me: they are defined here http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n1042 and then the custom elements that references are in here http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements | 10:50 |
| *** ccamacho has quit IRC | 10:51 | |
| *** ccamacho has joined #openstack-infra | 10:51 | |
| *** markvoelker has quit IRC | 10:52 | |
| *** ijw has quit IRC | 10:54 | |
| odyssey4me | ah, awesome thanks cmurphy :) | 10:55 |
| *** liujiong has quit IRC | 10:58 | |
| *** ccamacho has quit IRC | 10:58 | |
| *** ccamacho has joined #openstack-infra | 10:59 | |
| *** Sree has joined #openstack-infra | 10:59 | |
| *** chenying_ has quit IRC | 10:59 | |
| *** mat128 has joined #openstack-infra | 11:00 | |
| *** chenying_ has joined #openstack-infra | 11:00 | |
| *** jpena is now known as jpena|lunch | 11:00 | |
| *** thorst has joined #openstack-infra | 11:00 | |
| *** namnh has quit IRC | 11:01 | |
| *** dtantsur|afk is now known as dtantsur | 11:02 | |
| *** amoralej is now known as amoralej|lunch | 11:03 | |
| *** Sree has quit IRC | 11:03 | |
| *** claudiub|2 has joined #openstack-infra | 11:05 | |
| *** thorst has quit IRC | 11:05 | |
| *** claudiub has quit IRC | 11:06 | |
| *** jkilpatr has joined #openstack-infra | 11:10 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Revert "Support networkx 2.0" https://review.openstack.org/506118 | 11:10 |
| openstackgerrit | zhangyangyang proposed openstack-infra/mqtt_statsd master: Cleanup test-requirements https://review.openstack.org/506120 | 11:11 |
| *** Sree has joined #openstack-infra | 11:12 | |
| *** ociuhandu_ has quit IRC | 11:15 | |
| *** rwsu has quit IRC | 11:21 | |
| *** dizquierdo has joined #openstack-infra | 11:23 | |
| *** mwarad has joined #openstack-infra | 11:24 | |
| openstackgerrit | Tom Barron proposed openstack-infra/project-config master: Stop running redundant manila lvm jobs https://review.openstack.org/505818 | 11:25 |
| *** dprince has joined #openstack-infra | 11:30 | |
| *** nmathew has quit IRC | 11:32 | |
| *** ociuhandu has joined #openstack-infra | 11:33 | |
| *** rwsu has joined #openstack-infra | 11:34 | |
| *** shardy_afk is now known as shardy | 11:35 | |
| *** mwarad has quit IRC | 11:36 | |
| *** sdague has joined #openstack-infra | 11:38 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Initial skeleton for devstack-tempest base job https://review.openstack.org/504246 | 11:38 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check check log files https://review.openstack.org/505840 | 11:38 |
| *** calbers has joined #openstack-infra | 11:43 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Improve test case node_assignment_at_quota https://review.openstack.org/506134 | 11:46 |
| *** Sree has quit IRC | 11:46 | |
| *** ldnunes has joined #openstack-infra | 11:48 | |
| *** tpsilva has joined #openstack-infra | 11:49 | |
| *** markvoelker has joined #openstack-infra | 11:49 | |
| *** zhurong has quit IRC | 11:51 | |
| *** chlong has joined #openstack-infra | 11:53 | |
| *** wolverineav has joined #openstack-infra | 11:53 | |
| *** markvoelker has quit IRC | 11:54 | |
| *** mwarad has joined #openstack-infra | 11:55 | |
| *** markvoelker has joined #openstack-infra | 11:56 | |
| *** lastmikoi has left #openstack-infra | 11:59 | |
| *** thorst has joined #openstack-infra | 11:59 | |
| *** udesale has joined #openstack-infra | 12:02 | |
| *** udesale has quit IRC | 12:04 | |
| *** udesale has joined #openstack-infra | 12:04 | |
| *** psachin has quit IRC | 12:06 | |
| tosky | did anyone notice issues with the mirrors? | 12:06 |
| tosky | I saw " W: Failed to fetch http://mirror.bhs1.ovh.openstack.org/ubuntu/dists/xenial-updates/InRelease Unable to connect to " | 12:06 |
| tosky | at least https://review.openstack.org/#/c/505885/ | 12:06 |
| mnaser | tosky i noticed that in a puppet job too | 12:07 |
| mnaser | which failed at around the same time | 12:07 |
| tosky | uhm uhm | 12:07 |
| tosky | I wonder if I should just recheck or wait | 12:07 |
| mnaser | i havent noticed any other failures so | 12:07 |
| mnaser | i am rechecking my job and if it happens again ill report | 12:07 |
| *** jpena|lunch is now known as jpena | 12:08 | |
| tosky | ok, let's recheck while people from west side of Atlantic are still not totally awake and the gates are not full! | 12:08 |
| mnaser | yep, up early gets you a clean queue :p | 12:08 |
| *** dave-mccowan has joined #openstack-infra | 12:09 | |
| *** amoralej|lunch is now known as amoralej | 12:10 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Remove unreachable code https://review.openstack.org/506137 | 12:11 |
| *** yamahata has quit IRC | 12:12 | |
| *** mwarad has quit IRC | 12:18 | |
| *** HeOS has joined #openstack-infra | 12:23 | |
| *** trown|outtypewww is now known as trown | 12:24 | |
| *** ociuhandu has quit IRC | 12:25 | |
| *** bobh has joined #openstack-infra | 12:26 | |
| *** tikitavi has joined #openstack-infra | 12:29 | |
| *** jaypipes has joined #openstack-infra | 12:33 | |
| *** erlon has joined #openstack-infra | 12:34 | |
| *** udesale has quit IRC | 12:39 | |
| *** claudiub has joined #openstack-infra | 12:42 | |
| *** claudiub|2 has quit IRC | 12:43 | |
| *** mriedem has joined #openstack-infra | 12:46 | |
| *** Sree has joined #openstack-infra | 12:46 | |
| *** LindaWang has quit IRC | 12:47 | |
| dmsimard | What's the launchpad for openstack.org website bugs again ? | 12:48 |
| *** ociuhandu has joined #openstack-infra | 12:48 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove networking-midonet meeting https://review.openstack.org/506059 | 12:50 |
| *** xyang1 has joined #openstack-infra | 12:51 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Zuul v3 migration test drive: puppet-openstack https://review.openstack.org/505921 | 12:52 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 12:53 |
| *** ociuhandu has quit IRC | 12:54 | |
| dmsimard | clarkb, jeblair: btw, just in case it wasn't obvious, we (RDO) as third party CI with zuul, hit very much the same problems as you did upstream | 12:57 |
| *** bh526r has joined #openstack-infra | 12:57 | |
| dmsimard | for the gerrit upgrade, refs things and such | 12:57 |
| dmsimard | it might warrant a tad of a wider communication for 3rd party CIs using zuul against the upstream gerrit | 12:58 |
| dmsimard | We've had to backport https://review.openstack.org/#/c/505290/ but we forgot about https://review.openstack.org/#/c/505795/ which we're doing now | 12:58 |
| dmsimard | (I don't think there's another one) | 12:58 |
| *** mat128 has quit IRC | 13:00 | |
| *** esberglu has joined #openstack-infra | 13:01 | |
| *** d0ugal has quit IRC | 13:02 | |
| *** d0ugal has joined #openstack-infra | 13:03 | |
| *** d0ugal has quit IRC | 13:03 | |
| *** d0ugal has joined #openstack-infra | 13:03 | |
| *** jcoufal has joined #openstack-infra | 13:09 | |
| *** kgiusti has joined #openstack-infra | 13:12 | |
| *** LindaWang has joined #openstack-infra | 13:12 | |
| *** Goneri has joined #openstack-infra | 13:13 | |
| *** ociuhandu has joined #openstack-infra | 13:14 | |
| *** claudiub has quit IRC | 13:14 | |
| *** andreas_s has quit IRC | 13:15 | |
| *** claudiub has joined #openstack-infra | 13:15 | |
| fungi | tosky: mnaser: there was some sort of hang with mirror.bhs1.ovh.o.o between ~01:15-03:45z so if the failure was in that timeframe then it's a known issue | 13:21 |
| tosky | fungi: ack, thanks; in fact the recheck passed, so I thought it was a temporary glitch | 13:21 |
| mnaser | i hate to ask, but what would be the "next step" in dealing with a third party CI that is failing all the time .. it's not voting but its adding a lot of noise in terms of emails/comments | 13:23 |
| fungi | tosky: https://wiki.openstack.org/wiki/Infrastructure_Status | 13:23 |
| tosky | fungi: uh, thanks, I didn't know that page | 13:23 |
| *** mwarad has joined #openstack-infra | 13:23 | |
| mnaser | fuel ci is commenting on puppet changes with failures all the time, contacted #fuel-infra, #fuel, etc.. unable to reach anyone (not surprisingly) | 13:24 |
| mnaser | would it be possible to disable the account? | 13:24 |
| fungi | mnaser: if you've already tried reaching out to the ci operator on record, let us know an identifier for it and we set it inactive | 13:25 |
| *** gouthamr has joined #openstack-infra | 13:25 | |
| fungi | dmsimard: https://bugs.launchpad.net/openstack-org/+filebug is what you were looking for | 13:25 |
| mnaser | fungi http://irclog.perlgeek.de/fuel/2017-09-05 .. channel been silent since | 13:26 |
| fungi | mnaser: yeah, i can set it inactive and send something to the third-party-ci operators mailing list | 13:27 |
| mnaser | not sure how i can get the identifier, it comments as "Fuel CI" | 13:27 |
| fungi | that's enough of an identifier | 13:27 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/506166 | 13:27 |
| fungi | mnaser: i'm guessing it's this one: https://wiki.openstack.org/wiki/ThirdPartySystems/Fuel_Packaging_CI | 13:28 |
| mnaser | yup | 13:28 |
| fungi | and they do indeed list #fuel-infra as a primary point of contact | 13:29 |
| mnaser | ooh actually i think not fungi | 13:29 |
| mnaser | https://wiki.openstack.org/wiki/ThirdPartySystems/Fuel_CI | 13:29 |
| mnaser | this one | 13:29 |
| *** weshay is now known as weshay_bbiab | 13:30 | |
| fungi | ahh, well both also mention the infra liaisons at https://wiki.openstack.org/wiki/Fuel#People | 13:30 |
| fungi | neither of whom (iremizov nor igorbelikov) seem to be in this channel now | 13:31 |
| fungi | they also both claim to use the same gerrit username, so those may be redundant entries, or they may simply have multiple ci systems sharing one gerrit account | 13:32 |
| fungi | regardless, i'll look it up, set it inactive and disconnect any open connections for it | 13:32 |
| mnaser | fungi thank you | 13:34 |
| *** threestrands has joined #openstack-infra | 13:36 | |
| cmurphy | the proposal bot seems to be freaking out, it's rebasing all its patches every few minutes since yesterday https://review.openstack.org/#/c/502776/24 | 13:37 |
| fungi | #status log Gerrit account 8971 for "Fuel CI" has been disabled due to excessive failure comments | 13:38 |
| openstackstatus | fungi: finished logging | 13:39 |
| sdague | cmurphy: I think it seems to be regenerating whether or not it changed things | 13:39 |
| *** coolsvap has quit IRC | 13:39 | |
| *** martinkopec has quit IRC | 13:39 | |
| fungi | cmurphy: i'll take a look at one of its job logs in a few if nobody beats me to it | 13:39 |
| sdague | there were a ton of requirements changes landed yesterday, but I did think the bot didn't propose over the same change | 13:39 |
| sdague | maybe old gerrit prevented that | 13:39 |
| mordred | oh joy | 13:40 |
| cmurphy | sdague: yeah it's reproposing even though there are no differences | 13:40 |
| cmurphy | fungi: awesome thanks | 13:40 |
| *** baoli has joined #openstack-infra | 13:43 | |
| *** threestrands has quit IRC | 13:45 | |
| AJaeger | cmurphy: was a change merged between the repropose? Then it would be a rebase and fine - otherwise definitely not | 13:45 |
| *** mat128 has joined #openstack-infra | 13:46 | |
| AJaeger | cmurphy: looking at that cahgne it says "rebased" but nothing merged since 30th of July. That is indeed completely broken ;( | 13:49 |
| openstackgerrit | zhangyangyang proposed openstack-infra/nodepool master: Cleanup test-requirements https://review.openstack.org/506181 | 13:49 |
| AJaeger | mordred: could you check https://review.openstack.org/505774 - I gave a WIP since it's a new publish job. Do you agree? | 13:50 |
| fungi | cmurphy: sdague: AJaeger: http://logs.openstack.org/a9/a98e6f0cce6e97d964ccb8d9909fd3380d7c8cfb/post/propose-requirements-updates/18a8976/console.html is the log from the most recent run | 13:51 |
| mordred | AJaeger: yes please | 13:51 |
| *** mdbooth has joined #openstack-infra | 13:51 | |
| AJaeger | mordred: reading tonyb's comment - it's up for discussion anyhow, so WIP is fine ;) | 13:52 |
| *** Sree has quit IRC | 13:52 | |
| fungi | it thinks it altered 2 lines between 502776,23 and 502776,24 http://logs.openstack.org/a9/a98e6f0cce6e97d964ccb8d9909fd3380d7c8cfb/post/propose-requirements-updates/18a8976/console.html#_2017-09-21_13_30_24_190993 | 13:52 |
| fungi | in test-requirements.txt | 13:53 |
| AJaeger | fungi, ' kind: NO_CHANGE' at http://logs.openstack.org/a9/a98e6f0cce6e97d964ccb8d9909fd3380d7c8cfb/post/propose-requirements-updates/18a8976/console.html#_2017-09-21_13_30_22_754316 | 13:53 |
| *** Sree has joined #openstack-infra | 13:53 | |
| fungi | but yeah, gerrit seems to have emitted a warning? "remote: (W) No changes between prior commit 26e4103 and new commit f030db7" | 13:54 |
| AJaeger | http://logs.openstack.org/a9/a98e6f0cce6e97d964ccb8d9909fd3380d7c8cfb/post/propose-requirements-updates/18a8976/console.html#_2017-09-21_13_30_25_037951 '(W) No changes between prior commit 26e4103 and new commit f030db7 ' | 13:54 |
| AJaeger | fungi: yep | 13:54 |
| tonyb | AJaeger: Yeah WiP is totally appropriate for that change. | 13:54 |
| tonyb | AJaeger: It will probably get moved to post rather than periodic as I seem to be in the minority | 13:54 |
| AJaeger | tonyb: ok, good | 13:55 |
| fungi | so this suggests that gerrit has ceased rejecting new patchsets with no change from previous | 13:55 |
| *** slaweq has quit IRC | 13:55 | |
| AJaeger | ;( | 13:56 |
| mordred | boo | 13:56 |
| AJaeger | mordred, fungi, pabelanger, if either of you has some review love for this small branch cleanup stack, it would be appreciated: https://review.openstack.org/#/c/505762/ Thanks | 13:57 |
| mordred | WELL - we can rework the script to test for changes before submitting - I think we have examples of that in other places right? | 13:57 |
| *** Sree has quit IRC | 13:57 | |
| mordred | AJaeger: +3 from me | 13:59 |
| mordred | AJaeger: python-jobs-trusty and python34-jobs ... do we expect to keep them or for them to go away completely? | 13:59 |
| mordred | AJaeger: I ask because we have jobs and project-templates for them in the openstack-zuul-jobs - so if we are going to get rid of them from the config completely we shouldn't forget those there | 14:00 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Initial skeleton for devstack-tempest base job https://review.openstack.org/504246 | 14:00 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check check log files https://review.openstack.org/505840 | 14:00 |
| AJaeger | mordred: we have one or two users for python-jobs-trusty and python34-jobs. | 14:01 |
| mordred | fungi, AJaeger, sdague: also - if we can live with the churn | 14:01 |
| mordred | gah | 14:01 |
| mordred | AJaeger: ok | 14:01 |
| fungi | i wonder if this became a configuration option? | 14:01 |
| mordred | fungi, AJaeger, sdague: also - if we can live with the churn (since we have to do new images for changes to the propse script anyway) ... | 14:01 |
| sdague | mordred: the churn is a big waste | 14:01 |
| sdague | basically you approve requirements patches | 14:02 |
| sdague | they get into the gate | 14:02 |
| sdague | then gate reset | 14:02 |
| sdague | so requirements patches end up chewing up a huge amount of CI | 14:02 |
| fungi | a wishlist optimization on that job was to check whether the change is approved already and create a new change instead of a new patchset in that situation | 14:03 |
| mordred | oh - totally. so - what I was getting at is that as of monday changes to that script will be self-testable - so we could put a halt on requirements patch approvals until monday if the self-testing update will make fixing it easier | 14:03 |
| sdague | sure, but even just on the propose side | 14:03 |
| AJaeger | mordred: group-based-policy projects still use python-jobs-trusty (they left mitaka branch) | 14:03 |
| sdague | a requirements change lands and you trigger patches in 40 projects | 14:03 |
| mordred | AJaeger: gotcha. cool | 14:04 |
| AJaeger | mordred: fuel-agent and fuel-mirror are the only projects with python34-jobs. I could propose to remove these as well if that helps | 14:04 |
| AJaeger | mordred: we just need to make a decision that python34 is not supported on trusty anymore ;) | 14:05 |
| pabelanger | http://mirror01.bhs1.ovh.openstack.org/ looks to be back online | 14:05 |
| *** gomarivera has joined #openstack-infra | 14:05 | |
| fungi | pabelanger: yeah, it rebooted (according to its lastlog) at 03:45z | 14:06 |
| pabelanger | fungi: okay, I'm going to see about removing nodepool.o.o from emergency file to get more nodes online | 14:06 |
| AJaeger | fungi, that optimization is done for translation jobs - if anybody wants to "steal" the code | 14:07 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 14:08 |
| pabelanger | #status log nodepool.o.o removed from emergency file, ovh-bhs1 came back online at 03:45z. | 14:08 |
| openstackstatus | pabelanger: finished logging | 14:08 |
| fungi | in the meantime, i'm scouring the gerrit 2.13 docs looking to see if rejecting no-op patchsets is a config option now | 14:08 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Protect against p_opt not having prompt attribute https://review.openstack.org/506194 | 14:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{source}/{project}.pem route https://review.openstack.org/502530 | 14:09 |
| pabelanger | fungi: clarkb: is this new? https://review.openstack.org/#/admin/groups/1837,members OpenStack Project Creator is a member of the networking-lagopus-core group | 14:10 |
| pabelanger | I don't think I seen that before | 14:10 |
| *** camunoz has joined #openstack-infra | 14:10 | |
| fungi | pabelanger: possible new gerrit behavior is to add the group creator as an owner even when called from the api | 14:11 |
| fungi | (it already did that when you created groups from the webui) | 14:11 |
| pabelanger | fungi: okay, I'm going to add it to our etherpad to track | 14:12 |
| pabelanger | I also think we might have a stuck event in gerrit show-queue | 14:12 |
| openstackgerrit | Merged openstack-infra/project-config master: Branch cleanup (5/n) https://review.openstack.org/505762 | 14:13 |
| dmsimard | fungi: thanks | 14:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove unused infra-manual trusty job https://review.openstack.org/505765 | 14:14 |
| *** gomarivera has quit IRC | 14:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: Branch cleanup (6/n) https://review.openstack.org/505766 | 14:14 |
| *** armax has joined #openstack-infra | 14:14 | |
| *** gomarivera has joined #openstack-infra | 14:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create per hostname nodepool-launcher configuration https://review.openstack.org/504495 | 14:15 |
| openstackgerrit | Merged openstack-infra/project-config master: networking-midonet: Add an experimental multinode job https://review.openstack.org/505464 | 14:15 |
| fungi | unrelated, bit looks like they already added an access control for their "hashtags" functionality... https://review.openstack.org/Documentation/access-control.html#category_edit_hashtags | 14:18 |
| *** Swami has joined #openstack-infra | 14:19 | |
| *** gomarivera has quit IRC | 14:19 | |
| mordred | what's a hashtag? | 14:19 |
| fungi | extensible change tagging metadata | 14:19 |
| *** dtantsur is now known as dtantsur|brb | 14:20 | |
| fungi | i thought we needed to be on the notedb backend for that to work though | 14:20 |
| fungi | which supposedly isn't fully implemented until 2.14 | 14:20 |
| mordred | gotcha | 14:20 |
| mordred | seems likea cool thing once it exists | 14:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Make yaml2ical publication job https://review.openstack.org/504968 | 14:21 |
| fungi | we've had lots of people asking for it, and waiting for the implementation to become available (particularly for group change prioritization/coordination) | 14:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Octavia dashboard projects integration jobs https://review.openstack.org/505904 | 14:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Exclude playbooks/legacy from bashate linter https://review.openstack.org/505926 | 14:25 |
| *** bobh has quit IRC | 14:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Modify the check-uc jobs to only trigger on upper-constratints changes https://review.openstack.org/504501 | 14:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Increase timeouts for Manila DSVM jobs https://review.openstack.org/493092 | 14:26 |
| *** hashar has joined #openstack-infra | 14:26 | |
| fungi | not finding any way to configure that behavior. the warning is unfortunately the same as it used to be, except it was previously followed by a "remote rejected the change ..." failure | 14:26 |
| *** ldnunes has quit IRC | 14:27 | |
| *** ykarel has quit IRC | 14:29 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 14:31 |
| *** rbrndt has joined #openstack-infra | 14:32 | |
| *** openstackgerrit has quit IRC | 14:33 | |
| clarkb | looks like create group takes an initial member arg now, I bet default is self if not provided | 14:33 |
| clarkb | we can probably have project creator remove itself after the create group call | 14:34 |
| clarkb | or set some info somewhere that tells it what a better initial member is | 14:34 |
| *** felipemonteiro has joined #openstack-infra | 14:38 | |
| stephenfin | dhellmann: Have you noticed that the 'NOT label:Code-Review>=2' query in Gerrit not longer works? | 14:38 |
| stephenfin | i.e. to find patches that have not been reviewed | 14:38 |
| *** camunoz has quit IRC | 14:38 | |
| *** openstackgerrit has joined #openstack-infra | 14:39 | |
| fungi | stephenfin: sdague wrote at length about that on the -dev ml | 14:39 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Initial skeleton for devstack-tempest base job https://review.openstack.org/504246 | 14:39 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check check log files https://review.openstack.org/505840 | 14:39 |
| *** camunoz has joined #openstack-infra | 14:39 | |
| stephenfin | fungi: Thank you, once again. I'll go find that | 14:39 |
| *** ldnunes has joined #openstack-infra | 14:39 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:39 | |
| clarkb | pabelanger: any more info on possible stuck gerrit task? | 14:39 |
| fungi | stephenfin: there is a new query option specifically for that now (forget the name off the top of my head) rather than relying on a side effect of matching on all code-review vote values | 14:40 |
| dims | fungi : you will get a kick out of this :) https://github.com/golang/go/issues/21956 | 14:40 |
| pabelanger | clarkb: I haven't looked into it yet, but looks like we are failing to replicate to github again. | 14:40 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-exim master: Process mailman addresses last https://review.openstack.org/484989 | 14:40 |
| pabelanger | 4179f970 waiting .... 06:12:48.232 (retry 311) [648b1da9] push git@github.c | 14:40 |
| fungi | infra-root: confirmed behavior ^ | 14:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Include -attic in retired calculations https://review.openstack.org/506219 | 14:40 |
| clarkb | pabelanger: if you run with -w it will show you the full repo | 14:41 |
| jeblair | fungi: is that nova-specs? | 14:41 |
| fungi | infra-root: latest patchset on https://review.openstack.org/484989 was the result of `git review -d 484989;git commit --amend [not changing anything in the commit message];git review` | 14:41 |
| clarkb | which can be useful | 14:41 |
| pabelanger | clarkb: thanks | 14:42 |
| pabelanger | 4179f970 waiting .... 06:12:48.232 (retry 311) [648b1da9] push git@github.com:openstack/nova-specs.git | 14:42 |
| fungi | jeblair: what pabelanger's looking at? i dunno. i'm still looking at the no-op changes being allowed | 14:42 |
| jeblair | fungi: yes. your arrow pointed at pabelanger's line on my display. :) | 14:42 |
| fungi | we raced | 14:42 |
| pabelanger | Oh, known issue with nova-specs? | 14:42 |
| *** felipemonteiro has quit IRC | 14:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Include -attic in retired calculations https://review.openstack.org/506219 | 14:43 |
| fungi | so anyway, gerrit 2.13 definitely no longer rejects new patchsets with no changes in them from the previous patchset: http://paste.openstack.org/show/621632/ | 14:43 |
| *** hongbin has joined #openstack-infra | 14:43 | |
| fungi | doesn't seem to be configurable behavior that i can find, so time to go digging in release notes/changelogs i guess | 14:44 |
| mordred | clarkb, fungi: I fixed the jeepyb issue on the git0* - then ran create_cgitrepos by hand on git02 - I found one more place in which it can improve which is https://review.openstack.org/506219 | 14:44 |
| mordred | clarkb, fungi: I have run that version as well on git02 - and you can see the results: http://git02.openstack.org:8080/cgit | 14:44 |
| clarkb | fungi: at least new behavior is more user friendly. We just have to be smarter on reqs updates | 14:44 |
| mordred | (I also refactored a little while I was in there) | 14:44 |
| fungi | clarkb: well, maybe more user friendly... i wonder how it does with patch series where some were not amended at all (testing that next) | 14:45 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-askbot master: Don't run syncdb or migrate until after install https://review.openstack.org/416658 | 14:47 |
| fungi | okay, that ^ seemed to dtrt | 14:47 |
| fungi | so i agree new behavior is fine as long as we fix any job(s) which relied on the old behavior | 14:47 |
| *** dizquierdo has quit IRC | 14:49 | |
| pabelanger | dims: fungi: interesting debate: issues/21956 | 14:49 |
| dims | pabelanger : y, hard to explain the infra needed for large efforts - way beyond github can handle | 14:50 |
| fungi | review of the release notes indicates we may want to consider turning on the "Reject Implicit Merges" option, which is the fix for the long-standing bug/undesired behavior where people propose changes to the wrong branch and unintentionally fast-forward it when merged | 14:50 |
| fungi | pabelanger: no idea what issues/21956 refers to | 14:51 |
| pabelanger | fungi: no rush, it was dims comment in backscroll. You're likely focused on gerrit efforts :) | 14:52 |
| dims | fungi : will leave a link on -dev channel for later | 14:52 |
| *** andreas_s has joined #openstack-infra | 14:52 | |
| pabelanger | okay, ovh-bhs1 max-servers bumped, should start running jobs again shortly | 14:53 |
| openstackgerrit | Joe D'Andrea proposed openstack-dev/pbr master: Remove win32/nt checks for wrapper script gen https://review.openstack.org/502524 | 14:54 |
| clarkb | pabelanger: I really like the comment that said "I avoided it for years and then tried and it took less than 15 minutes and worked great" | 14:54 |
| clarkb | because I think most people want to complain and arent really trying because it is different | 14:54 |
| *** xarses_ has joined #openstack-infra | 14:55 | |
| pabelanger | clarkb: agree! | 14:55 |
| andreas_s | Hi, is there any known issue with the diskimage builder and ubuntu-minimal? Since yesterday I'm facing a python error "KeyError: 'blockdev'" - using the latest 2.8 release with the elements "ubuntu-minimal" and "vm" | 14:56 |
| pabelanger | Hmm, so we have hostname issues again on nl01.openstack.org (hostname nl01.openstack.org) and nl02.openstack.org (hostname nl02). I cannot remember what we said was the correct syntax, and our fix. I do remember cloud-init is involved | 14:56 |
| pabelanger | right, /etc/cloud/cloud-init.disabled | 14:58 |
| *** felipemonteiro_ has quit IRC | 14:59 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:59 | |
| andreas_s | Opened a bug for the diskimage-builder thing: https://bugs.launchpad.net/diskimage-builder/+bug/1718702 | 14:59 |
| openstack | Launchpad bug 1718702 in diskimage-builder "ubuntu minimal fails with: KeyError: 'blockdev'" [Undecided,New] | 14:59 |
| openstackgerrit | Tony Breeds proposed openstack-infra/release-tools master: Set up gerrit remote manually in eol_branch.sh https://review.openstack.org/505594 | 14:59 |
| openstackgerrit | Tony Breeds proposed openstack-infra/release-tools master: Check the gerrit remote is as expected, eol_branch https://review.openstack.org/505595 | 14:59 |
| *** supertakumi86 has joined #openstack-infra | 15:03 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Disable cloud-init, if installed https://review.openstack.org/506228 | 15:04 |
| pabelanger | clarkb: mordred:^ should disable cloud-init | 15:04 |
| pabelanger | to fix our hostname from changing on boot | 15:05 |
| mordred | pabelanger, clarkb: so - how about we go one step further ... | 15:07 |
| mordred | pabelanger, clarkb: (although I totally agree with that patch) | 15:07 |
| *** links has quit IRC | 15:07 | |
| mordred | pabelanger, clarkb: we just uninstalled python-six on git0* which caused cloud-init to get uninstalled | 15:07 |
| pabelanger | mordred: sure, we can do that on ubuntu too | 15:08 |
| mordred | pabelanger, clarkb: how about we go ahead and just uninstall it - and do an apt-get autoremove after we do to uninstall the various python libs that accidentally got installed by it | 15:08 |
| mordred | OR | 15:08 |
| mordred | be more specific about it and just do "uninstall python-six python-requests python-paramiko" and ignore the fact that that will uninstall cloud-init | 15:09 |
| mordred | or both | 15:09 |
| *** trown is now known as trown|brb | 15:10 | |
| clarkb | my concern is that it is used in some clouds and generally we dont have problems. Another fix for pabelanger's thing is to use $fqdn? | 15:10 |
| pabelanger | ya, ::fqdn works | 15:11 |
| *** felipemonteiro has joined #openstack-infra | 15:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Just delete cloud-init https://review.openstack.org/506235 | 15:11 |
| mordred | clarkb: what do you mean? | 15:11 |
| *** ociuhandu has quit IRC | 15:12 | |
| clarkb | mordred: on not rax cloud init actually does things | 15:12 |
| clarkb | and removing it is potentially not as noopy there | 15:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-project_config master: Use ::fqdn for nodepoolv3 yaml file https://review.openstack.org/506239 | 15:13 |
| pabelanger | clarkb: mordred ^ | 15:13 |
| mordred | clarkb: it should not do things anywhere after first boot | 15:13 |
| mordred | clarkb: all the rest of our clouds dhcp | 15:13 |
| *** felipemonteiro_ has quit IRC | 15:13 | |
| *** vhosakot has joined #openstack-infra | 15:14 | |
| clarkb | one place it can/could is when dealing with snapshots. | 15:14 |
| mordred | the only other things cloud-init would do for us after first boot are things like setting hostnames (which it does wrong per pabelanger patch) and potentially mounting config-drive for us on reboot- but we don't use config-drive after reboot anyway) | 15:14 |
| mordred | clarkb: yah- but we don't do snapshots | 15:14 |
| mordred | because they're unsafe | 15:14 |
| *** trown|brb is now known as trown | 15:14 | |
| clarkb | we do in some cases like testing lists in place upgrade | 15:15 |
| openstackgerrit | zhangyangyang proposed openstack-infra/python-storyboardclient master: Cleanup test-requirements https://review.openstack.org/506243 | 15:15 |
| mordred | clarkb: ok. fair. BUT - we could a) replace cloud-init with glean when we uninstall it b) install cloud-init or glean as a step before doing a snapshot c) I could shut up :) | 15:16 |
| clarkb | its a potentially large change one I think should be tested and thought about before applying globally. Especially since there is a simple workaround in pabelanger's case for now. | 15:16 |
| clarkb | And probably the best way to do that is switch to self built images with glean | 15:16 |
| clarkb | (and avoid all the baggage in $cloud image) | 15:17 |
| pabelanger | Ya, I'm thinking once zuulv3 is done, I might tackle DIB images for control plane. | 15:19 |
| *** gomarivera has joined #openstack-infra | 15:23 | |
| clarkb | hrm is nova-specs preventing other github syncing from happening? | 15:24 |
| *** felipemonteiro_ has joined #openstack-infra | 15:24 | |
| *** iyamahat has joined #openstack-infra | 15:24 | |
| clarkb | I'm going to try and stop the nova specs task and see if the others go through | 15:24 |
| pabelanger | ya, I believe so | 15:25 |
| *** andreas_s has quit IRC | 15:25 | |
| *** bobh has joined #openstack-infra | 15:26 | |
| clarkb | actuall 051d6d4e 06:13:26.703 [2518b15d] push git@github.com:openstack/ironic.git may be the thing stopping the other github syncs | 15:27 |
| clarkb | infra-root I am going to kill that ^ task as it has had more than 9 hours to do whatever it needs to do | 15:27 |
| *** felipemonteiro has quit IRC | 15:27 | |
| clarkb | ya github is moving now | 15:28 |
| *** iyamahat has quit IRC | 15:28 | |
| clarkb | fungi: AJaeger is anyone working on updating the reqs scripts to check differences more accurately yet? | 15:30 |
| *** bobh has quit IRC | 15:30 | |
| clarkb | I've added ^ to the etherpad | 15:30 |
| *** dhajare has quit IRC | 15:31 | |
| *** felipemonteiro_ has quit IRC | 15:32 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:33 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Zuul v3 migration test drive: puppet-openstack https://review.openstack.org/505921 | 15:33 |
| dmsimard | mordred, jeblair: ^ restored patchset with required-projects | 15:33 |
| *** felipemonteiro has joined #openstack-infra | 15:34 | |
| AJaeger | clarkb: not me, I'll be travelling next week and need to prepare a bit... | 15:34 |
| fungi | clarkb: i haven't started yet, but can get to it after lunch (it's wasteful but not super disruptive except when reqs team are approving changes) | 15:35 |
| clarkb | gotcha | 15:35 |
| clarkb | ya I guess the problem is mostly in new patches kicking out things in the gate | 15:35 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{source}/{project}.pem route https://review.openstack.org/502530 | 15:36 |
| *** bobh has joined #openstack-infra | 15:36 | |
| pabelanger | clarkb: mordred: mind a review on https://review.openstack.org/506239/ currently blocking nodepool-launcher things for zuulv3 | 15:37 |
| *** felipemonteiro_ has quit IRC | 15:37 | |
| clarkb | mordred: jeepyb change lgtm | 15:39 |
| clarkb | pabelanger: looking | 15:39 |
| openstackgerrit | zhangyangyang proposed openstack/python-jenkins master: Cleanup test-requirements https://review.openstack.org/506262 | 15:39 |
| *** e0ne has quit IRC | 15:40 | |
| pabelanger | clarkb: mirror.regionone.tripleo-test-cloud-rh1.openstack.org is out of HDD space | 15:40 |
| pabelanger | trying to see what can be cleaned up manually | 15:40 |
| clarkb | is its cache cleaner not keeping up | 15:41 |
| *** jdandrea has left #openstack-infra | 15:41 | |
| pabelanger | we have 100GB total | 15:41 |
| pabelanger | clarkb: checking | 15:41 |
| *** slaweq has joined #openstack-infra | 15:41 | |
| pabelanger | no, htcacheclean looks good | 15:42 |
| pabelanger | currently at 48GB | 15:42 |
| pabelanger | I think we also say 50GB for AFS too? | 15:42 |
| pabelanger | ya | 15:42 |
| *** iyamahat has joined #openstack-infra | 15:43 | |
| pabelanger | so, we might have to decrease something by 10GB | 15:43 |
| pabelanger | until tripleo-test-cloud-rh1 is taken offline | 15:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Reduce apache reverse proxy cache to 40GB https://review.openstack.org/506270 | 15:46 |
| pabelanger | clarkb: ^ | 15:46 |
| mordred | pabelanger, fungi, jeblair: could I sell any of you on +3ing https://review.openstack.org/#/c/506219/ so that I can clear that off my stack and go back to not caring about it ? | 15:47 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: encrypt_secret: remove the trailing '/' when building url https://review.openstack.org/506271 | 15:48 |
| mordred | pabelanger, fungi, jeblair: if you didn't catch from scrollback, it's been run on 02 and the results can be seen at http://git02.openstack.org:8080/cgit | 15:48 |
| pabelanger | +2 so others can review | 15:49 |
| clarkb | already had my +2 | 15:49 |
| *** felipemonteiro has quit IRC | 15:49 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:49 | |
| pabelanger | mordred: also https://review.openstack.org/506239/ if you don't mind | 15:49 |
| mordred | pabelanger: done | 15:50 |
| *** trown is now known as trown|brb | 15:53 | |
| *** felipemonteiro has joined #openstack-infra | 15:53 | |
| clarkb | no more github replication tasks, I am going to manully trigger an ironic replication now to see if it goes through | 15:54 |
| *** sambetts is now known as sambetts|afk | 15:54 | |
| clarkb | seemed to work, guessing there was a glitch in the github matrix earlier today (utc) that made the last one block | 15:55 |
| fungi | if it was a task for nova-specs, we know that's broken replicating due to a corrupt/missing object | 15:56 |
| *** felipemonteiro_ has quit IRC | 15:56 | |
| clarkb | ya I killed that one too since it has several hundred retries but openstack/ironic was blocking all github replication since ~0600UTC | 15:57 |
| clarkb | so I killed that task and everything else cleared through and manual ironic replication cleared as well | 15:57 |
| fungi | ahh | 15:58 |
| fungi | yeah that sounds more like a github hiccup | 15:58 |
| *** sbezverk has quit IRC | 15:59 | |
| jeblair | mordred: +3 | 16:00 |
| *** caphrim007 has joined #openstack-infra | 16:00 | |
| *** trown|brb is now known as trown | 16:01 | |
| *** sbezverk has joined #openstack-infra | 16:01 | |
| *** ralonsoh has quit IRC | 16:02 | |
| *** upama has quit IRC | 16:02 | |
| *** ralonsoh has joined #openstack-infra | 16:02 | |
| *** ralonsoh has quit IRC | 16:02 | |
| openstackgerrit | Merged openstack-infra/jeepyb master: Include -attic in retired calculations https://review.openstack.org/506219 | 16:03 |
| mordred | jeblair: thanks! | 16:04 |
| *** camunoz has quit IRC | 16:04 | |
| *** slaweq has quit IRC | 16:05 | |
| *** bh526r has quit IRC | 16:06 | |
| *** slaweq has joined #openstack-infra | 16:06 | |
| *** yamahata has joined #openstack-infra | 16:07 | |
| *** weshay_bbiab is now known as weshay | 16:07 | |
| clarkb | and looks like ovh-bhs1 is back in the lineup | 16:08 |
| pabelanger | clarkb: ya, but seeing some node launch failures atm | 16:08 |
| *** jascott1 has joined #openstack-infra | 16:09 | |
| clarkb | pabelanger: looks like it is slowly falling off? I wonder if that is images need to be synced behavior | 16:12 |
| *** slaweq has quit IRC | 16:12 | |
| *** florianf has quit IRC | 16:13 | |
| clarkb | oh looks like fewer attempts too... | 16:13 |
| pabelanger | clarkb: ya, that's what I am waiting to see too but not sure yet | 16:13 |
| *** egonzalez has quit IRC | 16:13 | |
| pabelanger | I am wondering if gearman / nodepool are out of sync. Because we have jobs waiting for 1+hr, but nodepool doesn't seem to be building a large demand | 16:13 |
| clarkb | that cna happen if we have leaked in nodepool too | 16:14 |
| clarkb | (so nodepool thinks it is done) | 16:14 |
| pabelanger | current ready nodes look okay | 16:15 |
| pabelanger | clarkb: have we restarted nodepool recently? | 16:16 |
| pabelanger | I know zuul has been a few times | 16:16 |
| clarkb | I don't think so | 16:16 |
| clarkb | I had to clean up leaked used nodes after the restart of zuul last time | 16:17 |
| clarkb | but I don't think it has been restarted since | 16:17 |
| pabelanger | should we consider doing one? | 16:17 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Removed unused 'status: ' string from log line https://review.openstack.org/505378 | 16:18 |
| *** LindaWang has quit IRC | 16:18 | |
| tonyb | Is there an old 2.12 gerrit anywhere? I'm trying to work out why the election tools don't work anymore | 16:18 |
| *** hashar has quit IRC | 16:18 | |
| mordred | infra-root: ok - jeepyb updates are live and good - http://git.openstack.org/cgit/ now no-longer shows retired repos | 16:18 |
| *** Apoorva has joined #openstack-infra | 16:18 | |
| clarkb | tonyb: no, also we were on 2.11 before. Do you have tracebacks or similar to look at? | 16:19 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add job.nodeset parameter to supercede job.nodes https://review.openstack.org/505845 | 16:19 |
| rodrigods | hey clarkb, another look at https://review.openstack.org/#/c/492223/ ? :) | 16:19 |
| pabelanger | clarkb: there we go, nodepool just decided to build 200 nodes | 16:19 |
| clarkb | tonyb: it is relatively easy to spin up a dev instance locally. It will use an h2 database and all that. Definitely not production quality but should be useable for testing things | 16:19 |
| tonyb | clarkb: not a traceback but i *think* the API is returning different stuff | 16:19 |
| *** camunoz has joined #openstack-infra | 16:19 | |
| clarkb | tonyb: you basically grab the war file, and run the init script and it will interactively have you set it up real quick and default is to use google openid iirc | 16:20 |
| *** mwarad has quit IRC | 16:21 | |
| tonyb | clarkb: okay. I'll give that a try I guess there's a puppet module to help me grok any non-default settings from review.openstack.org | 16:21 |
| clarkb | there is | 16:22 |
| clarkb | though little of that should affect api respones I think | 16:22 |
| clarkb | its mostly auth and gitweb and so on | 16:22 |
| *** jascott1 has quit IRC | 16:23 | |
| *** jascott1 has joined #openstack-infra | 16:23 | |
| *** jascott1 has quit IRC | 16:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/jeepyb master: Remove extra writing of description https://review.openstack.org/506277 | 16:24 |
| pabelanger | clarkb: you're still good with https://review.openstack.org/506228/ | 16:24 |
| *** jascott1 has joined #openstack-infra | 16:26 | |
| tonyb | Yeah they changed the API :( | 16:27 |
| tosky | question: so far I was told that the instances used by zuul have the same features (8 GiB of RAM, etc); does it match what is written here, or there could be different flavors? https://docs.openstack.org/infra/system-config/infra-cloud.html#requirements | 16:28 |
| jeblair | tonyb: what changed? | 16:28 |
| tonyb | compare https://gerrit-documentation.storage.googleapis.com/Documentation/2.11.10/rest-api-accounts.html#suggest-account and https://gerrit-review.googlesource.com/Documentation/rest-api-accounts.html#query-account | 16:28 |
| * tonyb looks to see if there is an option I can use the get the old output if not I'll just chnage endpoints | 16:28 | |
| pabelanger | tosky: vcpu could be less then 8, depending on provider | 16:29 |
| clarkb | though right now they are all the same iirc | 16:29 |
| pabelanger | tonyb: but all should have 8GB of ram, and 30GB root | 16:29 |
| tosky | ack, thanks | 16:29 |
| tonyb | Yeah I can just add o=DETAILS to get the old output | 16:29 |
| fungi | tosky: yes, though a better reference as a consumer is probably https://docs.openstack.org/infra/manual/testing.html | 16:30 |
| pabelanger | clarkb: mordred: fungi: I'd like to see about landing https://review.openstack.org/506270/ to help address 100% disk in tripleo-test-cloud-rh1 | 16:30 |
| tosky | fungi: uhm, interesting, thanks | 16:32 |
| fungi | tosky: the infra-cloud requirements document was more of a spec for what we wanted out of a cloud we were building, so not completely applicable (though fairly similar) to what we get from other providers | 16:34 |
| clarkb | tosky: is there something in particular you are looking for? | 16:34 |
| fungi | tosky: we do also have a requirements document from providers contributing server resources to us: https://docs.openstack.org/infra/system-config/contribute-cloud.html | 16:35 |
| fungi | s/from/for/ | 16:35 |
| fungi | but the test environment chapter i linked from the infra-manual is probably what you're looking for | 16:35 |
| clarkb | infra-root my parents just flew into town so I will likely be in and out over the next couple days. Will try to keep on top of gerrit and zuul related items | 16:35 |
| tosky | clarkb: properly planning the resources for few future sahara scenario multinode jobs, where the compute node should run more heavy instances | 16:35 |
| fungi | thanks for the heads up clarkb! | 16:35 |
| pabelanger | clarkb: enjoy! | 16:35 |
| tosky | so if I know that root is 30GiB, and I have to use it to run 3 instances which requires a disk of 10GiB each, I need to be careful | 16:36 |
| tosky | or tune devstack to use the additional partition for nova | 16:36 |
| tosky | or use a 3 node multinode, or... I don't know | 16:36 |
| fungi | tosky: or put your storage in /opt | 16:37 |
| pabelanger | Ya, /opt should have more space | 16:37 |
| tosky | fungi: that's part of "tune devstack" | 16:37 |
| fungi | (optionally formatting and mounting it if you're not using devstack-gate) | 16:37 |
| clarkb | I think we may already do that for devstack via devstack gate? | 16:37 |
| clarkb | something to check on at least | 16:38 |
| tosky | is there something easy to specify the disk layout? Or will it be as part of the default set of roles provided by zuul v3? | 16:38 |
| fungi | tosky: part of the work around zuul v3 is an effort to decompose devstack-gate into multiple ansible roles, and i expect there will be a role for the disk layout stuff (if it hasn't already been written that is) | 16:39 |
| *** slaweq has joined #openstack-infra | 16:39 | |
| fungi | which should allow you to just include that in a job even if you're not using the rest of devstack-gate | 16:40 |
| tosky | good to know - so I take that if I'm using devstack-gate, there are already some parameters to tune it? I never tried to change the disk layout so fa | 16:41 |
| tosky | far* | 16:41 |
| clarkb | yes devstack-gate should in theory take care of all that for you already | 16:41 |
| clarkb | I don't know if nova puts instance root disks in /opt though that would be something to check on | 16:41 |
| fungi | tosky: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/playbooks/roles/fix_disk_layout/tasks/main.yaml looks like it's already been decomposed | 16:42 |
| *** kiennt has joined #openstack-infra | 16:43 | |
| tosky | clarkb: if the disk can be 30GiB in some providers, yeah, it would make sense to have /opt for instance root disks | 16:43 |
| *** tikitavi has quit IRC | 16:43 | |
| jeblair | infra-root: a patch landed to zuul that i think needs to be removed from the feature/zuulv3 branch, and reverting is insufficient. i can correct this by performing a force-push, or by recreating the branch at the parent commit (and then, i guess, re-proposing patches that landed afterwords). | 16:45 |
| jeblair | does anyone have thoughts about the procedure here? | 16:45 |
| *** dtantsur|brb is now known as dtantsur | 16:46 | |
| jeblair | i think if anyone else asked this, i'd probably say "a) don't do that" followed by "b) i guess we can recreate the feature branch as long as we're just pointing to a commit already in the tree and the ptl is okay with it". | 16:46 |
| clarkb | deleting the branch and recreating it is probaly the more "normal" user workflow | 16:47 |
| fungi | got it, so based on discussion in #zuul its because there's a semver-indicating footer in the commit message and pbr will see it even if you revert? | 16:47 |
| jeblair | fungi: yes, or at least that i don't know what pbr will do if i revert and don't trust that behavior will always be consistent or that i can predict what will happen as things move around branches in the future. | 16:48 |
| clarkb | maybe pbr should be taught to back out those version bumps on revert? as another option | 16:48 |
| clarkb | that would only apply to newer pbr thoug | 16:48 |
| pabelanger | branch delete / recreate also keeps servers happy running from feature/zuulv3? | 16:48 |
| fungi | i lean toward recreating the branch at the parent commit, but don't feel strongly either way | 16:48 |
| *** slaweq has quit IRC | 16:48 | |
| fungi | both options are unfortunately messy | 16:48 |
| jeblair | clarkb: maybe it does? i don't actually know. but i don't ever want to activate those code paths. | 16:48 |
| fungi | and i need to skip out to lunch. people just got here. back as soon as i can be | 16:49 |
| *** slaweq has joined #openstack-infra | 16:49 | |
| *** camunoz has quit IRC | 16:49 | |
| *** jtomasek has quit IRC | 16:49 | |
| *** jtomasek has joined #openstack-infra | 16:50 | |
| mordred | clarkb: I think we should DEFINITELY teach pbr how to deal with a revert of a sem-ver footer - but probably not today | 16:50 |
| *** tesseract has quit IRC | 16:50 | |
| jeblair | so how about i recreate the branch at parent and send an email to openstack-infra letting folks know it happened? | 16:50 |
| jeblair | clarkb, mordred: ^ that sound okay? | 16:51 |
| *** gomarivera has quit IRC | 16:51 | |
| *** dizquierdo has joined #openstack-infra | 16:51 | |
| clarkb | jeblair: yes, the other question I had is we havne't merged back to master since it happened right? (I don't think we have at all) | 16:51 |
| clarkb | since that could potentially confuse merging things if it had already been merged but pretty sure they are completely divergent | 16:51 |
| jeblair | clarkb: correct, no branch merges have happened since | 16:51 |
| *** iyamahat has quit IRC | 16:52 | |
| *** yamahata has quit IRC | 16:52 | |
| mordred | jeblair: ++ | 16:52 |
| *** rcernin has quit IRC | 16:52 | |
| *** Swami has quit IRC | 16:53 | |
| openstackgerrit | Merged openstack-infra/puppet-project_config master: Use ::fqdn for nodepoolv3 yaml file https://review.openstack.org/506239 | 16:53 |
| pabelanger | wfm | 16:54 |
| clarkb | ya I think that works | 16:54 |
| pabelanger | can also help with zuulv3 servers if needed for git things | 16:54 |
| sdague | 500 errors on cloning - http://logs.openstack.org/27/506227/1/gate/gate-nova-api-ref/ad1c8cf/console.html#_2017-09-21_16_39_43_685428 ? | 16:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop using docs-draft.openstack.org for doc drafts https://review.openstack.org/506305 | 16:54 |
| *** dave-mcc_ has joined #openstack-infra | 16:55 | |
| clarkb | I wish it logged the url, but looks like zm03 | 16:55 |
| * clarkb will look at apache logs | 16:55 | |
| pabelanger | I see End of script output before headers: git-http-backend around that time in error logs for apache | 16:56 |
| *** dave-mccowan has quit IRC | 16:58 | |
| clarkb | [Thu Sep 21 16:39:43.588195 2017] [cgid:error] [pid 13382:tid 140414879225600] [client 91.106.198.37:48364] End of script output before headers: git-http-backend | 16:58 |
| clarkb | that ^ | 16:58 |
| pabelanger | yup | 16:58 |
| pabelanger | looks to have just started up | 16:58 |
| clarkb | I think that means git-http-backend "crashed" before writing all the data | 16:58 |
| *** jascott1 has quit IRC | 16:58 | |
| pabelanger | first instance seems to be on Mon Sep 18 22:55:04.513483 2017 | 16:58 |
| *** jascott1 has joined #openstack-infra | 16:59 | |
| kiennt | Hi all, sorry for interrupted. I'm Kien Nguyen from Fujitsu. In email [1], Jeremy said he've marked the older account (22406) and 've left the new one active. Unfortunately, the older is the right account. Please deactive the 26112 and reactive 22406 | 16:59 |
| pabelanger | okay, maybe not just started | 16:59 |
| kiennt | [1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122441.html | 16:59 |
| *** electrofelix has quit IRC | 16:59 | |
| kiennt | thank you so much | 16:59 |
| clarkb | kiennt: we might want to wait for fungi just to make sure there wasn't any database surgery performed too | 16:59 |
| clarkb | kiennt: but fungi should be able to sort it out | 16:59 |
| *** jascott1 has quit IRC | 17:00 | |
| *** jascott1 has joined #openstack-infra | 17:00 | |
| *** baoli has quit IRC | 17:00 | |
| *** slaweq has quit IRC | 17:00 | |
| *** slaweq has joined #openstack-infra | 17:01 | |
| *** camunoz has joined #openstack-infra | 17:01 | |
| kiennt | clarkb: thank you :) | 17:01 |
| *** baoli has joined #openstack-infra | 17:01 | |
| *** jpich has quit IRC | 17:01 | |
| *** derekh has quit IRC | 17:02 | |
| *** ramishra has quit IRC | 17:03 | |
| *** supertakumi86 has quit IRC | 17:03 | |
| jeblair | clarkb: hrm, i can't delete/recreate zuulv3 with open changes | 17:03 |
| clarkb | ya that is longstanding gerrit behavior iirc | 17:04 |
| clarkb | I think you may be able to abandon them then restore them after though | 17:04 |
| jeblair | clarkb: should i abandon all changes (and then restore?), or should we switch tacks and force-push? | 17:04 |
| clarkb | maybe test the abandon restore behavior on review-dev first? | 17:04 |
| *** jascott1 has quit IRC | 17:04 | |
| clarkb | but ya force push is other way around that | 17:04 |
| clarkb | and maybe simpler in the end | 17:05 |
| clarkb | (with roughly thesame end result for gerrit users) | 17:05 |
| *** jpena is now known as jpena|off | 17:05 | |
| dmsimard | infra-root: would appreciate reviews on https://review.openstack.org/#/c/505921/ to test-drive puppet-openstack with zuulv3 migrated content, to give me enough time to dig out any issues I might find, especially if we're still considering a monday cutover | 17:05 |
| clarkb | rodrigods: done | 17:07 |
| *** trown is now known as trown|lunch | 17:07 | |
| pabelanger | dmsimard: sorry, I haven't been following, why not just wait until we merger zuul-migrate script on monday? As I understand it, that's when we'll all be starting to track down check job issues | 17:08 |
| dmsimard | pabelanger: because I'm already finding issues with just putting the review up and want to see ahead of time if there are any other issues ? | 17:09 |
| *** Swami has joined #openstack-infra | 17:09 | |
| dmsimard | we're going to eventually revert that patch, it's just a way of uncovering potential problems that we have no faced with (relatively) simple projects such as devstack | 17:10 |
| dmsimard | I think it was already agreed that I was going to test something other than devstack jobs ahead of time to test the result of the migration script | 17:10 |
| dmsimard | FWIW the migrated content in that review is verbatim from the tip of the zuulv3-migration topic | 17:11 |
| dmsimard | minus the non-voting and the required-projects which is the issue we realized | 17:11 |
| dmsimard | https://review.openstack.org/#/c/505926/ was another (minor) issue | 17:12 |
| *** tosky has quit IRC | 17:12 | |
| jeblair | clarkb: yeah, abandon/restore seems to work | 17:12 |
| dmsimard | the range of possible issues when testing hundreds of jobs is pretty crazy, by testing a small subset I think we can have a good sample without losing our sanity | 17:13 |
| pabelanger | dmsimard: okay, I admit, I haven't been following the last few days, so will defer to others | 17:15 |
| pabelanger | I'd likely switch out OVB jobs, as they are pinned to tripleo-test-cloud-rh1 and moving downstream to RDO | 17:16 |
| *** chenying_ has quit IRC | 17:17 | |
| *** chenying_ has joined #openstack-infra | 17:18 | |
| openstackgerrit | Merged openstack-infra/system-config master: Reduce apache reverse proxy cache to 40GB https://review.openstack.org/506270 | 17:20 |
| *** rbrndt has quit IRC | 17:20 | |
| *** kiennt has quit IRC | 17:21 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:22 | |
| *** felipemonteiro has quit IRC | 17:22 | |
| *** felipemonteiro has joined #openstack-infra | 17:25 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Migrate docs-draft jobs to emit to logs/html https://review.openstack.org/506316 | 17:27 |
| *** felipemonteiro_ has quit IRC | 17:28 | |
| mordred | jeblair, clarkb, AJaeger: ^^ please check me on the logic in the very-long comment I left in that, as well as my rsync syntax (I believe I have trailing slash vs. non-trailing-slash correct - but I've been wrong before) | 17:28 |
| mordred | pabelanger: ^^ | 17:28 |
| *** mat128 has quit IRC | 17:32 | |
| openstackgerrit | German Eichberger proposed openstack-infra/project-config master: [WIP] Generate Octavia Amphora image artefact https://review.openstack.org/501890 | 17:33 |
| AJaeger | mordred: comment looks fine | 17:34 |
| mordred | AJaeger: awesome. thanks! | 17:34 |
| AJaeger | mordred: no time to check rsync / ... | 17:36 |
| *** iyamahat has joined #openstack-infra | 17:36 | |
| *** camunoz has quit IRC | 17:37 | |
| *** erlon has quit IRC | 17:37 | |
| *** mat128 has joined #openstack-infra | 17:42 | |
| *** lastmikoi has joined #openstack-infra | 17:44 | |
| *** dhajare has joined #openstack-infra | 17:46 | |
| *** mat128 has quit IRC | 17:47 | |
| *** jascott1 has joined #openstack-infra | 17:49 | |
| *** mat128 has joined #openstack-infra | 17:49 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Create build-openstack-sphinx-docs job https://review.openstack.org/504785 | 17:51 |
| *** srobert has joined #openstack-infra | 17:51 | |
| *** srobert has quit IRC | 17:52 | |
| *** srobert has joined #openstack-infra | 17:52 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Bump nodepoolv3 max-server to 10 https://review.openstack.org/506324 | 17:53 |
| *** jascott1 has quit IRC | 17:53 | |
| pabelanger | infra-root: ^ would be helpful to land, so we don't wedge nodepool-launcher again. We need to have larger max-servers per provider then our largest nodeset, which seems to be 5 today. | 17:54 |
| pabelanger | #status log nl01.o.o and nl02.o.o are both back online with site-specific nodepool.yaml files. | 17:54 |
| openstackstatus | pabelanger: finished logging | 17:54 |
| *** srobert_ has joined #openstack-infra | 17:55 | |
| *** ijw has joined #openstack-infra | 17:56 | |
| *** ianychoi has quit IRC | 17:57 | |
| *** Sukhdev has joined #openstack-infra | 17:57 | |
| *** gomarivera has joined #openstack-infra | 17:58 | |
| *** ianychoi has joined #openstack-infra | 17:58 | |
| *** srobert has quit IRC | 17:58 | |
| *** mat128 has quit IRC | 18:02 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:02 | |
| *** Sukhdev has quit IRC | 18:03 | |
| *** tosky has joined #openstack-infra | 18:06 | |
| *** esberglu has quit IRC | 18:07 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings master: Update channel list from project-config https://review.openstack.org/505656 | 18:07 |
| *** esberglu has joined #openstack-infra | 18:08 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings master: JJB meetings are being held in dedicated channel https://review.openstack.org/496314 | 18:08 |
| *** dtantsur is now known as dtantsur|afk | 18:08 | |
| *** esberglu has quit IRC | 18:12 | |
| fungi | okay, back and catching up | 18:12 |
| fungi | i guess kiennt has parted again | 18:13 |
| fungi | i'll fix | 18:13 |
| mordred | pabelanger: I took over your remove-openstack-doc-buld-to-move patch as part of the rework of docs-draft | 18:13 |
| *** dizquierdo has quit IRC | 18:14 | |
| mordred | pabelanger: also, +2 on that patch | 18:14 |
| fungi | infra-root: oh crap, i just overrode everyone's lp openids. working to restore from backup copy of those columns now but it will take me a few minutes to build up a mapping :/ | 18:15 |
| mordred | fungi: whoops | 18:16 |
| jeblair | fungi: should we stop gerrit in the interim? | 18:16 |
| fungi | #status notice Gerrit OpenIDs have been accidentally overwritten and are in the process of being restored | 18:16 |
| openstackstatus | fungi: sending notice | 18:16 |
| fungi | jeblair: maybe... it will primarily impact anyone logging in between now and when i get it reset | 18:16 |
| *** mat128 has joined #openstack-infra | 18:16 | |
| fungi | i missed a constraint on account_id when updating the table | 18:16 |
| jeblair | i think we should, and i will do it | 18:16 |
| fungi | thanks | 18:16 |
| fungi | i'm in the process of extracting the previous table state now | 18:17 |
| fungi | and sorry :/ | 18:17 |
| -openstackstatus- NOTICE: Gerrit OpenIDs have been accidentally overwritten and are in the process of being restored | 18:17 | |
| jeblair | it is one of those weeks | 18:17 |
| fungi | ugh | 18:17 |
| *** esberglu has joined #openstack-infra | 18:18 | |
| jlvillal | What does that mean, for those of us who aren't familiar with OpenIDs? | 18:18 |
| jlvillal | We can't log in? | 18:18 |
| fungi | you'd end up logged into the wrong account | 18:18 |
| fungi | when logging out and back in | 18:18 |
| jeblair | i wanted to stop gerrit because i didn't want to worry about people logging in to the wrong account :) | 18:18 |
| fungi | but we're temporarily stopping gerrit | 18:19 |
| jlvillal | Sweet, I'll take fungi's account ;) | 18:19 |
| openstackstatus | fungi: finished sending notice | 18:19 |
| fungi | #status alert The Gerrit service on review.openstack.org is being taken offline briefly to perform database repair work but should be back up shortly | 18:19 |
| openstackstatus | fungi: sending alert | 18:19 |
| AJaeger | jlvillal: coffee break time ;) | 18:20 |
| jlvillal | AJaeger: That is what it means! :) | 18:21 |
| jlvillal | I'll make it lunch time. | 18:21 |
| fungi | i think the easiest solution will be for me to backup the current state of the reviewdb database, reapply the overnight database backup of just reviewdb.external_ids, and then check for any new entries between the two and attempt manual fix-up | 18:21 |
| mordred | fungi: ++ | 18:22 |
| clarkb | I'm hanging out with parents if there is something specific I can do to help let me know | 18:22 |
| clarkb | also that plan sgtm | 18:22 |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being taken offline briefly to perform database repair work but should be back up shortly | 18:22 | |
| *** ChanServ changes topic to "The Gerrit service on review.openstack.org is being taken offline briefly to perform database repair work but should be back up shortly" | 18:22 | |
| fungi | any rows where the external_id field is not like "https://login.ubuntu.com/%" were untouched | 18:22 |
| clarkb | ok so its fairly limited to openids | 18:23 |
| clarkb | unless there was other content in same row like api password data | 18:23 |
| fungi | correct | 18:23 |
| *** pcrews has quit IRC | 18:23 | |
| *** kambiz has joined #openstack-infra | 18:24 | |
| fungi | myswldump is currently writing a backup to /home/fungi/pre-restore.sql.gz | 18:24 |
| dansmith | hey guys, I suddenly found I had amazing powers in gerrit and have deleted a few projects, hope that's okay, kthx | 18:24 |
| *** ijw has quit IRC | 18:25 | |
| openstackstatus | fungi: finished sending alert | 18:25 |
| fungi | dansmith: the account you would have inadvertently been shuffled into had basically no privileges at all, so that would be surprising ;) | 18:25 |
| dansmith | fungi: heh, dangit | 18:25 |
| jeblair | dansmith: also if you can tell us how to delete projects that would be awesome :) | 18:26 |
| mordred | ++ | 18:27 |
| fungi | bonus points if you can delete them cleanly ;) | 18:27 |
| jeblair | yeah, this is one of those faery wish questions; must include all the qualifiers | 18:27 |
| dansmith | lol | 18:27 |
| fungi | once this emergency backup completes, i'll be working off /home/gerrit2/mysql_backups/gerrit.sql.gz.1 which, despite it's name, is the most recent backup courtesy of logrotate creating empty files | 18:27 |
| dansmith | this joke has gone terribly | 18:27 |
| fungi | mordred: jeblair: cheat-sheet notes on how to limit a source command to just a single table? | 18:28 |
| fungi | or do i need to edit the dump? | 18:28 |
| mordred | fungi: editing the dump is gonna be the easiest | 18:29 |
| mnaser | ^ | 18:29 |
| *** ijw has joined #openstack-infra | 18:29 | |
| SamYaple | im going to create a new user called 'sam2; DROP TABLES projects' to see what happens | 18:29 |
| mordred | fungi: you could alternately restore to a temp schema, then dump just that table | 18:29 |
| mordred | fungi: but I think that'll take longer given data size | 18:30 |
| jeblair | there's a sed command in https://stackoverflow.com/questions/1013852/can-i-restore-a-single-table-from-a-full-mysql-mysqldump-file which looks plausible | 18:30 |
| *** jdandrea has joined #openstack-infra | 18:30 | |
| fungi | cool, i'll be working on an uncompressed copy of the nightly backup at gerrit-to-restore.sql | 18:30 |
| fungi | /home/fungi/gerrit-to-restore.sql specifically | 18:31 |
| fungi | oh, except not there, no space left on device | 18:31 |
| *** pcrews has joined #openstack-infra | 18:31 | |
| fungi | /home/gerrit2/gerrit-to-restore.sql instead | 18:31 |
| clarkb | put it in /home/gerrit2 ya that has space or should | 18:31 |
| jeblair | (/opt also looks promising) | 18:32 |
| *** rbrndt has joined #openstack-infra | 18:33 | |
| fungi | creating /home/gerrit2/gerrit-to-restore-filtered.sql via sed -n -e to filter between account_external_ids and the subsequent table | 18:34 |
| fungi | it's definitely way tinier than /home/gerrit2/gerrit-to-restore.sql | 18:35 |
| *** zoli is now known as zoli|gone | 18:35 | |
| *** zoli|gone is now known as zoli | 18:35 | |
| fungi | can other infra-root peeps double-check review.o.o:/home/gerrit2/gerrit-to-restore-filtered.sql and make sure that looks like just the account_external_ids table | 18:35 |
| fungi | ? | 18:36 |
| *** yamahata has joined #openstack-infra | 18:36 | |
| *** trown|lunch is now known as trown | 18:36 | |
| fungi | looks like i need to remove a line at the very end | 18:37 |
| fungi | CREATE TABLE `account_group_by_id` ( | 18:37 |
| mordred | fungi: looking | 18:38 |
| mnaser | yeah you want to have the DROP TABLE, CREATE TABLE and the INSERT's (ideally) | 18:38 |
| fungi | i've removed the stray line at the end | 18:39 |
| mnaser | but /me can't look :) | 18:39 |
| mordred | fungi: the last 7 lines seem bogus | 18:39 |
| fungi | i can manually DROP TABLE `account_external_ids` before i source that file too | 18:39 |
| mordred | fungi: I think sourcing it is fine- and the file looks good - but all the bits at the end related to account_group_by_id need to go away | 18:40 |
| fungi | mordred: thanks, those are presumably cruft leading up to the next table... checking | 18:40 |
| jeblair | agreed, other than that lgtm. | 18:41 |
| fungi | trimming them now | 18:41 |
| fungi | editing a 4gb file with veeeeery log lines is slow | 18:42 |
| jeblair | fungi: would 'head -25' do it? | 18:42 |
| *** armax has quit IRC | 18:44 | |
| *** armax has joined #openstack-infra | 18:44 | |
| fungi | well, i also noticed my pre-restore dump ran out of space in my homedir too, so rerunning that to /home/gerrit2/pre-restore.sql.gz at the same time | 18:45 |
| fungi | does the edited version look sane? i added the drop at the beginning too | 18:46 |
| anticw | fungi: if i have to edit dumps i --skip-extended-insert | 18:46 |
| mordred | fungi: yes | 18:47 |
| jeblair | fungi: yes | 18:47 |
| fungi | anticw: thanks for the tip | 18:47 |
| fungi | okay, as soon as the retry on the pre-restore dump completes, i'll source /home/gerrit2/gerrit-to-restore-filtered.sql | 18:47 |
| pabelanger | Wow, that took way longer then expected | 18:48 |
| fungi | pabelanger: you're back just in time for the fun i've created for everyone | 18:49 |
| pabelanger | logfile for unbound on centos doesn't have right selinux settings. We could either do 1) move logfile to /var/lib/unbound/unbound.log or 2) generate custom selinux module which we compile and install at DIB image build time, keeping /var/log/unbound.log | 18:50 |
| fungi | pabelanger: is there a 3) ? | 18:50 |
| mnaser | setenforce 0? | 18:50 |
| fungi | though moving the logfile seems sane enough | 18:50 |
| * mnaser giggles and walks away | 18:50 | |
| fungi | okay, dump completed this time. restoring account_external_ids now | 18:51 |
| jeblair | pabelanger: 1 wfm | 18:51 |
| clarkb | we could also drop logfile as it ended up being not useful | 18:51 |
| pabelanger | well, I'm actually trying to debug some dns failures on centos, maybe we should just dump to stdout and hope systemd catches it? | 18:52 |
| fungi | http://paste.openstack.org/show/621652 is the output from the source command | 18:52 |
| *** ijw has quit IRC | 18:52 | |
| fungi | curious about the ERROR 1231 there | 18:52 |
| fungi | however, previous openids seem to be back to their prior distinctive glory | 18:53 |
| mnaser | do you have a SET character_set_client in the dump somewhere maybe? | 18:53 |
| jeblair | that probably got filtered out | 18:53 |
| fungi | anybody want to poke around the current (restored) state of reviewdb.account_external_ids? | 18:54 |
| jeblair | the default client char set on that machine is utf8 anyway | 18:55 |
| jeblair | so was probably a noop | 18:55 |
| mnaser | https://stackoverflow.com/a/12689443 | 18:55 |
| mnaser | its probably because we trimmed the dump | 18:55 |
| jeblair | yes, the original dump had a line to save the client charset | 18:56 |
| fungi | ahh | 18:56 |
| *** mriedem1 has joined #openstack-infra | 18:56 | |
| jeblair | so the filtered dump was referencing an undefined var | 18:56 |
| mnaser | don't think it's much of an issue anyways | 18:56 |
| mnaser | (but good to know) | 18:56 |
| mordred | agree | 18:56 |
| *** mriedem has quit IRC | 18:56 | |
| jeblair | i typed 'use reviewdb;' and now it's sitting there and i have no prompt. | 18:56 |
| fungi | i can include that and rerun the source if anyone is concerned | 18:56 |
| fungi | jeblair: maybe i need to unlock tables? | 18:57 |
| fungi | not sure what would cause a use to hang | 18:57 |
| mnaser | ^ thats it | 18:57 |
| *** bandini has joined #openstack-infra | 18:57 | |
| mordred | building completion hash | 18:57 |
| jeblair | | 59637 | gerrit2 | 10.223.160.141:44548 | reviewdb | Field List | 114 | Waiting for table | | | 18:57 |
| jeblair | it finished | 18:57 |
| fungi | i have terminated my mysqlclient session in case that was locking | 18:57 |
| jeblair | that was *very* slow | 18:57 |
| mnaser | there was a mysql client option to avoid it | 18:57 |
| mnaser | fast startup, or something. | 18:58 |
| mordred | it was the table lock | 18:58 |
| fungi | sorry about that | 18:58 |
| mordred | which building the completion hash was waiting for | 18:58 |
| jeblair | looking at the table, monty and i have different account external ids. | 18:58 |
| jeblair | that's a good sign. | 18:59 |
| fungi | that seems... normal | 18:59 |
| mnaser | could do a select count(account_field) from table; and select count(distinct(account_field)) from table; and compare | 19:00 |
| mnaser | but that's a tad of an overkill probably | 19:00 |
| jeblair | the max account id in external_ids is 26947 and in accounts it's 26951. so we should have 4 accounts with no external ids. | 19:01 |
| *** Apoorva_ has joined #openstack-infra | 19:01 | |
| fungi | yeah, i'll find them in a comparative dump of the two tables. i expect to also find some preexisting accounts who changed external id data today | 19:01 |
| fungi | and will get them mapped back | 19:02 |
| fungi | luckily all i botched was overwriting the account_id on rows with openid urls, so filtering the rest for differences should be trivial | 19:03 |
| jeblair | fungi: table looks good to me | 19:04 |
| *** Apoorva has quit IRC | 19:04 | |
| fungi | okay, anyone opposed to going ahead and starting gerrit again? the remaining cleanup should be doable with the service running (worst case i'll need to flush caches after the inserts/updates) | 19:04 |
| jeblair | fungi: wfm | 19:05 |
| AJaeger | fungi: start it but don't tell the world yet - let's first test a bit | 19:05 |
| fungi | agreed | 19:05 |
| fungi | it's starting now | 19:05 |
| fungi | curious to see if the initscript times out and reports failure again | 19:06 |
| fungi | since that was noted on the last restart | 19:06 |
| fungi | Starting Gerrit Code Review: OK | 19:06 |
| fungi | huh, so not this time anyway | 19:06 |
| * AJaeger is logged in again as Andreas Jaeger | 19:07 | |
| fungi | webui is up and responding again | 19:07 |
| AJaeger | (was not logged out) | 19:07 |
| mordred | woot! | 19:07 |
| jeblair | i logged out and in as corvus | 19:07 |
| mnaser | i am already logged in and was able to login again in an incognito window :> | 19:07 |
| *** mriedem1 is now known as mriedem | 19:07 | |
| smcginnis | Separate thing, but anyone else notice the search textbox on the top is extra long requiring scrolling to the right since the upgrade? | 19:08 |
| clarkb | smcginnis: yes its likely due to our new logo css | 19:08 |
| mordred | smcginnis: yah- it was a little bad before the upgrade but is definitely worse now | 19:08 |
| clarkb | mnaser: ^ you might be unterested in that | 19:08 |
| mnaser | or the resolution you run? i dont see anything | 19:08 |
| mordred | oh - it's super bad for me | 19:08 |
| smcginnis | Is there another option under the My heading too? That seems to push things over as well. | 19:09 |
| mnaser | mordred mind sharing a screenshot? cause i dont see it here in chrome unless i resize the window down | 19:09 |
| clarkb | smcginnis: yes | 19:09 |
| mnaser | but my screen is pretty big so that could be related | 19:09 |
| fungi | looks like i was able to log out and back in as me | 19:09 |
| mordred | mnaser: https://imgur.com/a/KiyEd | 19:09 |
| AJaeger | started new browser , logged in - looks fine | 19:09 |
| mnaser | ok i certainly dont have as much dashboards as everyone here so | 19:10 |
| mnaser | that's probably why i dont see it :p | 19:10 |
| fungi | mnaser: it also depends on which mode you have the webui in as that impacts the menuish options along the header of the page | 19:10 |
| mordred | mnaser: https://imgur.com/a/KhBzo is another one that kills me | 19:10 |
| mnaser | im not sure what would be the best solutionin this case | 19:10 |
| mordred | mnaser: me neither | 19:10 |
| fungi | i wonder whether we could rescale the search box | 19:10 |
| mnaser | we could make teh search box smaller | 19:11 |
| smcginnis | mordred: You probably need a wider screen for all those tabs any way. | 19:11 |
| mnaser | and that can alliveate some of that issue | 19:11 |
| fungi | okay, before i move on to the next stage of cleanup, any objections to lifting #status alert? | 19:11 |
| fungi | things seem to be back to normal for me | 19:11 |
| mordred | mnaser: what if we made it vertically smaller - so that it could fit above the line that divides the top menu from the bottom menu? | 19:11 |
| *** lukebrowning has joined #openstack-infra | 19:11 | |
| mnaser | the design seems to be a table split into 3 columns | 19:12 |
| mordred | and if you switched to a list of dashboards they would slide under the box instead of pushing it to the right? | 19:12 |
| mordred | ah | 19:12 |
| mordred | well, so mcuh for that :) | 19:12 |
| AJaeger | fungi, did you see http://lists.openstack.org/pipermail/openstack-dev/2017-September/122445.html - Kiên Nguyễn Tuấn said that you retired the wrong openid | 19:12 |
| mnaser | yeah that would have been a nice cleanup | 19:12 |
| fungi | AJaeger: yep, that's what started this whole mess :( | 19:12 |
| mordred | we could float the search box and tie it to the upper right | 19:12 |
| AJaeger | fungi: Ah ;( | 19:13 |
| fungi | AJaeger: in attempting to undo that i reset the account_id column for all openids to his | 19:13 |
| AJaeger | Is there a way to get rid of those duplicate openids for good? | 19:13 |
| mnaser | mordred let me do a first fix by making the search smaller as a step one | 19:13 |
| fungi | AJaeger: switch to an openid provider we control | 19:14 |
| fungi | AJaeger: and then do <something> to catch that situation i suppose | 19:14 |
| jeblair | fungi: status ok is okay by me | 19:14 |
| mnaser | mordred if you have a minute, do you mind right clicking on the search textbar, hit "inspect", on the right side under element.style, add "width: 150px" and let me know if that at least solves the change page? | 19:15 |
| mnaser | until we figure out what to do for the other cases | 19:15 |
| fungi | AJaeger: i think the problem occurs when users sign into the ubuntu one sso with a different e-mail address, and that ends up creating a new openid (which lp has secret sauce to figure out how to combine into the same lp account apparently) | 19:15 |
| fungi | AJaeger: if they then log into gerrit, it gets a different openid which gerrit thinks means needs a new account, even if it has the same e-mail address as an existing account with a different openid | 19:16 |
| mnaser | ooo | 19:17 |
| mnaser | looks like we had it resized at 250px before | 19:17 |
| mnaser | but afaik gerrit is undergoing gwt removal | 19:17 |
| mnaser | so the css doesn't match | 19:17 |
| AJaeger | fungi ;( | 19:17 |
| mnaser | let me fix it to restore things | 19:17 |
| fungi | #status ok OpenIDs for the Gerrit service have been restored from a recent backup and the service is running again; before/after table states are being analyzed now to identify any remaining cleanup needed for changes made to accounts today | 19:17 |
| openstackstatus | fungi: sending ok | 19:17 |
| *** ijw has joined #openstack-infra | 19:17 | |
| jeblair | fungi: thanks for testing the restore procedure! :) | 19:17 |
| fungi | jeblair: heh. i use slip-ups like this as a teaching tool, to help new sysadmins know mistakes happen to everyone | 19:18 |
| jeblair | i'm going to go forage for food now | 19:18 |
| fungi | thanks for the help jeblair, mordred, mnaser! | 19:18 |
| fungi | i'm getting to work on the table comparisons to find the stragglers now | 19:19 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/system-config master: Fix Gerrit search bar width https://review.openstack.org/506329 | 19:19 |
| mnaser | mordred clarkb smcginnis ^ :) | 19:19 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 19:20 | |
| mnaser | at least it'll make things back to the way they were | 19:20 |
| -openstackstatus- NOTICE: OpenIDs for the Gerrit service have been restored from a recent backup and the service is running again; before/after table states are being analyzed now to identify any remaining cleanup needed for changes made to accounts today | 19:20 | |
| *** e0ne has joined #openstack-infra | 19:21 | |
| *** ijw has quit IRC | 19:22 | |
| pabelanger | jeblair: mind a +3 on https://review.openstack.org/506324/ | 19:23 |
| openstackstatus | fungi: finished sending ok | 19:23 |
| pabelanger | that should be last thing for nodepool-launchers for zuulv3 currently | 19:23 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Update logfile for unbound on centos/fedora https://review.openstack.org/506332 | 19:23 |
| mordred | AJaeger: 'gate-{name}-pep8-ubuntu-trusty' | 19:24 |
| mordred | AJaeger: does that still need to be a thing? (It's not a problem - but I just happened to notice it) | 19:24 |
| AJaeger | mordred: fuel and group-based-policy -and cinder branch driverfixes/mitaka use that (but feel free to double check) - part of python-jobs-trusty | 19:25 |
| *** xyang1 has quit IRC | 19:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Extract required-projects from job content https://review.openstack.org/506335 | 19:26 |
| mordred | AJaeger: gotcha. theother things make sense- a trusty version of a pep8 job makes less sense to me ... | 19:29 |
| AJaeger | mordred: yeah - that was part of xenial move - trusty pep8 runs on mitaka and older... | 19:30 |
| fungi | pep8 jobs from the mitaka era may need to install dependencies that don't install cleanly on xenial, i suppose | 19:31 |
| dirk | infra-root Issues on job setup: http://logs.openstack.org/45/505545/1/gate/gate-cross-glance-python35/8c4f374/console.html | 19:31 |
| *** e0ne has quit IRC | 19:31 | |
| dirk | Happens only occasionally though | 19:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Allow mapping to handle pep8 for trusty https://review.openstack.org/506336 | 19:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove comment about unused required-projects https://review.openstack.org/506337 | 19:32 |
| *** jaypipes has quit IRC | 19:35 | |
| pabelanger | ovh-bhs1 finally looks happy again | 19:36 |
| pabelanger | dirk: ya, we've been seeing random zuul-merger failures over the last few days | 19:36 |
| pabelanger | I think ianw looked into it a bit yesterday | 19:36 |
| fungi | dirk: that looks like a node in infracloud saw connectivity issues to a zuul merger (in rackspace) | 19:36 |
| *** xyang1 has joined #openstack-infra | 19:37 | |
| fungi | so could be on eth infracloud/hpe side, on the rackspace side, or anywhere on the internet in between | 19:37 |
| fungi | one more thing zuul v3 will make better | 19:38 |
| * dirk crosses fingers | 19:39 | |
| *** pcaruana has quit IRC | 19:41 | |
| fungi | (zuul v3 pushes changes onto nodes instead of telling them where to pull from, so would presumably catch any connectivity issues at that stage and rerun the jobs automatically) | 19:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Readd bigswitch trusty jobs https://review.openstack.org/506341 | 19:43 |
| AJaeger | and that's my bug fix of the day ^ | 19:43 |
| *** dave-mcc_ is now known as dave-mccowan | 19:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Bump nodepoolv3 max-server to 10 https://review.openstack.org/506324 | 19:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow dict-based new jobs to have templated names https://review.openstack.org/506344 | 19:48 |
| *** weshay is now known as weshay_bbiab | 19:50 | |
| *** olaph1 has quit IRC | 19:51 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Bump min-ready to 10 for ubuntu-xenial https://review.openstack.org/506346 | 19:52 |
| *** mat128 has quit IRC | 19:54 | |
| AJaeger | team, could you review https://review.openstack.org/506341 ,please? I removed one job too many in my branch cleanup | 19:58 |
| mordred | jeblair: I just found an obscure and strange edgecase that's mildly annoying but not actually *problematic* | 19:58 |
| *** thorst has quit IRC | 19:58 | |
| mordred | AJaeger: +2 | 19:58 |
| AJaeger | thanks, mordred | 19:59 |
| mordred | jeblair: we have a job in project-config (in v2) called "gate-infra-docs-index" - which runs tox with the infra-docs environment | 19:59 |
| mordred | jeblair: (as well as a bunch of other ones, like gate-project-config-gerrit and gate-project-config-irc-access - with tox env gerrit and irc respectively) | 20:00 |
| mordred | jeblair: but we can't define jobs in project-config that depend on jobs in zuul-jobs, right? so we'll need to wind up defining those jobs in openstack-zuul-jobs and then adding them to the project-pipeline in project-config | 20:02 |
| mordred | jeblair: (I mean, at least until your updates to job order sequencing) | 20:02 |
| *** thorst has joined #openstack-infra | 20:03 | |
| mordred | jeblair: anyway - it's not an issue-issue - just a thing weird niggly I noticed while going through migration content | 20:03 |
| jeblair | mordred: could we define those in project-config? | 20:05 |
| *** ijw has joined #openstack-infra | 20:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add ubuntu-trusty to mapped jobs appropriately https://review.openstack.org/506348 | 20:07 |
| *** thorst has quit IRC | 20:07 | |
| mordred | jeblair: they need to have parent: on a job defined in zuul-jobs | 20:07 |
| jeblair | mordred: ah, then yes. awkward. definitely one of the reasons to rework that. | 20:08 |
| mordred | yah | 20:08 |
| AJaeger | mordred: commented on 506348 | 20:10 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use publish-docs-draft base job for docs-draft publishers https://review.openstack.org/504624 | 20:11 |
| *** ijw has quit IRC | 20:11 | |
| *** xarses_ has quit IRC | 20:14 | |
| *** armax has quit IRC | 20:18 | |
| jeblair | i'm beginning the work to rewind feature/zuulv3 | 20:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Strip devstack-legacy down to base minimum https://review.openstack.org/506352 | 20:21 |
| *** annegentle has joined #openstack-infra | 20:22 | |
| *** ijw has joined #openstack-infra | 20:27 | |
| *** gouthamr has quit IRC | 20:28 | |
| *** kgiusti has left #openstack-infra | 20:35 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Initial skeleton for devstack-tempest base job https://review.openstack.org/504246 | 20:36 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check check log files https://review.openstack.org/505840 | 20:36 |
| fungi | okay, so it seems of the three new gerrit accounts created between the nightly backup and restore, one (26949) was all null in the accounts table no clue how that happens, and the other two (26950 and 26951) had only e-mail addresses and openids in the account_external_ids table, which i've now restored | 20:36 |
| fungi | we also have a couple of existing accounts who set rest api passwords during that timeframe, so i'm putting those back into place now | 20:36 |
| *** xarses_ has joined #openstack-infra | 20:37 | |
| *** ijw has quit IRC | 20:38 | |
| *** jascott1 has joined #openstack-infra | 20:40 | |
| *** thorst has joined #openstack-infra | 20:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Removed unused 'status: ' string from log line https://review.openstack.org/506356 | 20:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job.nodeset parameter to supercede job.nodes https://review.openstack.org/506357 | 20:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use publish-docs-draft base job for docs-draft publishers https://review.openstack.org/506358 | 20:42 |
| jeblair | that ^ completes the zuulv3 rewind | 20:43 |
| *** annegentle has quit IRC | 20:43 | |
| jeblair | fungi: how did you restore the openids? | 20:43 |
| fungi | jeblair: matching on the email_address column | 20:44 |
| jeblair | fungi: but didn't you clear out the openid values? | 20:44 |
| fungi | no, i overwrote the account_id values so the external_id column was still there | 20:44 |
| fungi | we simply lost the mapping of what the correct account_ids were for them | 20:45 |
| jeblair | ohhhhhh gotcha | 20:45 |
| jeblair | that's convenient | 20:45 |
| jeblair | fungi: you make good errors | 20:45 |
| fungi | i suppose :/ | 20:45 |
| *** jkilpatr has quit IRC | 20:47 | |
| anticw | re: gerrit, you could search before with "-foo" to exclude items that had foo in the subject; for example to find some repos and ignore anything WIP i could do: https://review.openstack.org/#/q/projects:openstack/openstack-helm+status:open+-wip | 20:48 |
| anticw | that no longer works "-foo" doesn't exclude food but i guess does inclusive AND on "-foo" ? | 20:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/506166 | 20:49 |
| fungi | infra-root: account_external_ids manual repair has been completed to re-add entries as they should have appeared between the nightly backup and my screwup... now i probably need to flush caches but since the largest caches we have are the accounts caches i'm thinking i may as well just do --all instead of specifying those individually; opinions? | 20:53 |
| jeblair | fungi: sounds reasonable | 20:55 |
| *** Apoorva_ has quit IRC | 20:55 | |
| *** lukebrowning has quit IRC | 20:55 | |
| *** Apoorva has joined #openstack-infra | 20:55 | |
| mordred | fungi: ++ | 20:56 |
| fungi | flushing now | 20:57 |
| * fungi listens for the loud sucking sound | 20:58 | |
| * fungi imagines it's like that scene from yellow submarine | 20:58 | |
| *** trown is now known as trown|outtypewww | 20:58 | |
| *** Apoorva_ has joined #openstack-infra | 21:01 | |
| *** hamzy has quit IRC | 21:03 | |
| fungi | the flush seems to have completed fairly quickly | 21:03 |
| *** Apoorva has quit IRC | 21:04 | |
| *** jkilpatr has joined #openstack-infra | 21:04 | |
| *** jcoufal has quit IRC | 21:06 | |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder master: Revert "Support networkx 2.0" https://review.openstack.org/506376 | 21:11 |
| *** ijw has joined #openstack-infra | 21:15 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 21:15 |
| *** felipemonteiro has quit IRC | 21:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/devstack-gate master: Remove devstack-legacy base job https://review.openstack.org/506352 | 21:17 |
| *** slaweq has quit IRC | 21:17 | |
| *** felipemonteiro_ has joined #openstack-infra | 21:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Emit shell instead of script tasks https://review.openstack.org/505379 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Omit some jobs from shared queue calculation https://review.openstack.org/505380 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Deal with link-logs macro https://review.openstack.org/505387 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Migrate docs-draft jobs to emit to logs/html https://review.openstack.org/506316 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Extract required-projects from job content https://review.openstack.org/506335 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow dict-based new jobs to have templated names https://review.openstack.org/506344 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Extract required projects from embedded clonemaps https://review.openstack.org/506377 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Support zuul-cloner command line arguments https://review.openstack.org/506378 | 21:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use legacy-dsvm-base as base job for dsvm jobs https://review.openstack.org/506379 | 21:19 |
| *** felipemonteiro__ has joined #openstack-infra | 21:19 | |
| *** ijw has quit IRC | 21:19 | |
| *** ldnunes has quit IRC | 21:20 | |
| mordred | jeblair, fungi, clarkb, pabelanger, jlk, dmsimard: ^^ nice big stack for ya there - I believe that covers all of the cases I can come up with for extracting required projects | 21:21 |
| mordred | mnaser: do all of your puppet-openstack integration jobs that use the openstack/puppet-openstack-integration repo have puppet-openstack-integration in their names? | 21:22 |
| jlk | oh damn, they have a new parent since I last looked, and it's a big one. | 21:23 |
| mnaser | mordred nope, there are cases where we have our beaker jobs that depend on puppet-openstack-integration | 21:23 |
| *** felipemonteiro_ has quit IRC | 21:23 | |
| *** baoli has quit IRC | 21:23 | |
| mnaser | so beaker job clones puppet-openstack-integration and uses the classes in it to test a module individually (ex: deploy keystone+swift only for puppet-swift) | 21:23 |
| mordred | piddle | 21:24 |
| clarkb | mordred: while I'm back to computer for a minute, it looks like puppet is running against git backends and review.o.o successfully now \o/ | 21:25 |
| clarkb | mordred: I guess you just manually removed the packages we didn't want to get that going again? | 21:25 |
| clarkb | I've just approved mnaser's fix for the gerrit css. Puppet should apply that then on the next restart of gerrit (there have been a lot of those) it will get applied | 21:26 |
| *** ihrachys has quit IRC | 21:26 | |
| mordred | clarkb: yah - I fixed all the git backend stuff- you'll also note that there are less projects displayed in cgit now :) | 21:27 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Fixed OAUTH2 client config https://review.openstack.org/506384 | 21:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add a base job for puppet-openstack-integration https://review.openstack.org/506385 | 21:27 |
| mordred | mnaser: ^^ | 21:27 |
| mnaser | clarkb last time i checked, gerrit applied css without a restart! | 21:28 |
| *** armax has joined #openstack-infra | 21:28 | |
| * mnaser looks at mordred change | 21:28 | |
| *** Sukhdev has joined #openstack-infra | 21:28 | |
| clarkb | mnaser: oh maybe it will just work then | 21:29 |
| mnaser | mordred i *think* you're missing puppet- in there | 21:30 |
| mordred | mnaser: oh! those are all supposed to be puppet-{foo} ? | 21:32 |
| mnaser | mordred i believe so? we don't clone the project repos (source code), but we clone the puppet modules for the projects | 21:33 |
| mordred | kk | 21:33 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: make diskimage-builder use upper-constraints https://review.openstack.org/506388 | 21:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add a base job for puppet-openstack-integration https://review.openstack.org/506385 | 21:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements https://review.openstack.org/506166 | 21:34 |
| *** Apoorva_ has quit IRC | 21:34 | |
| *** Apoorva has joined #openstack-infra | 21:35 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Use upper-constraints in tox tests https://review.openstack.org/505975 | 21:35 |
| *** Goneri has quit IRC | 21:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for puppet-openstack-integration based jobs https://review.openstack.org/506389 | 21:36 |
| mordred | mnaser: ok - updated in https://review.openstack.org/506385 and then added support for emitting it in https://review.openstack.org/506389 | 21:37 |
| *** jascott1 has quit IRC | 21:37 | |
| *** dhajare has quit IRC | 21:38 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix Gerrit search bar width https://review.openstack.org/506329 | 21:39 |
| mnaser | mordred do we want to add the beaker jobs in there (they install all modules as well) and do we want to suffix legacy-puppet-openstack-integration with base for consistency with dsvm? (the former more of a nit) | 21:39 |
| mordred | mnaser: they're taken care of | 21:39 |
| mnaser | oooh | 21:40 |
| mordred | mnaser: the migration script already detects that they need puppet-openstack-integration in their required projects because of the jjb macro that clones it - so I added a condition to the migration script that if one of the already detected required projects is puppet-openstack-integration, then we should map it to that base job | 21:40 |
| mnaser | gotcha | 21:41 |
| mnaser | lgtm then.. from what i understand so far :D | 21:41 |
| mordred | mnaser: when that change runs its tests properly (https://review.openstack.org/#/c/506389) there will be a zuul-migrate job that will contain the converted job output | 21:42 |
| mordred | I need to fix a pep8 and re-push in just a sec | 21:42 |
| *** dprince has quit IRC | 21:42 | |
| mnaser | yes it will be good to have a look at that | 21:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 21:43 |
| *** makowals_ has joined #openstack-infra | 21:44 | |
| *** zhenguo has quit IRC | 21:45 | |
| *** bauzas has quit IRC | 21:45 | |
| *** fdegir has quit IRC | 21:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Create build-openstack-sphinx-docs job https://review.openstack.org/504785 | 21:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add legacy-dsvm-base job for converted devstack jobs https://review.openstack.org/506373 | 21:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add a base job for puppet-openstack-integration https://review.openstack.org/506385 | 21:45 |
| *** adisky has quit IRC | 21:46 | |
| *** edwarnicke has quit IRC | 21:46 | |
| *** fdegir has joined #openstack-infra | 21:46 | |
| *** thorst has quit IRC | 21:46 | |
| *** coreycb has quit IRC | 21:46 | |
| *** simondodsley has quit IRC | 21:46 | |
| *** kmARC has quit IRC | 21:46 | |
| *** jbryce has quit IRC | 21:46 | |
| *** gmann has quit IRC | 21:47 | |
| *** beisner has quit IRC | 21:47 | |
| *** mrhillsman has quit IRC | 21:47 | |
| *** efried has quit IRC | 21:47 | |
| *** makowals has quit IRC | 21:47 | |
| *** evrardjp has quit IRC | 21:47 | |
| *** vkmc has quit IRC | 21:47 | |
| *** rm_work has quit IRC | 21:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Update mapping file to use nodeset instead of node https://review.openstack.org/506390 | 21:47 |
| *** dasanind has quit IRC | 21:48 | |
| *** sweston has quit IRC | 21:48 | |
| *** kmARC has joined #openstack-infra | 21:48 | |
| *** adisky has joined #openstack-infra | 21:48 | |
| *** simondodsley has joined #openstack-infra | 21:48 | |
| *** coreycb has joined #openstack-infra | 21:48 | |
| *** amotoki has quit IRC | 21:48 | |
| *** beisner has joined #openstack-infra | 21:50 | |
| *** mrhillsman has joined #openstack-infra | 21:50 | |
| *** sweston has joined #openstack-infra | 21:50 | |
| *** amotoki has joined #openstack-infra | 21:51 | |
| *** tpsilva has quit IRC | 21:51 | |
| *** evrardjp has joined #openstack-infra | 21:52 | |
| *** dasanind has joined #openstack-infra | 21:52 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Extract required-projects from job content https://review.openstack.org/506335 | 21:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow dict-based new jobs to have templated names https://review.openstack.org/506344 | 21:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Extract required projects from embedded clonemaps https://review.openstack.org/506377 | 21:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Support zuul-cloner command line arguments https://review.openstack.org/506378 | 21:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use legacy-dsvm-base as base job for dsvm jobs https://review.openstack.org/506379 | 21:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add support for puppet-openstack-integration based jobs https://review.openstack.org/506389 | 21:52 |
| *** bauzas has joined #openstack-infra | 21:52 | |
| *** vkmc has joined #openstack-infra | 21:53 | |
| *** jascott1 has joined #openstack-infra | 21:54 | |
| *** lin_yang has joined #openstack-infra | 21:56 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Switch job.nodes to nodeset https://review.openstack.org/506394 | 21:57 |
| andreaf | mordred, jeblair: I'm tyring to run some zuul tests on my d-g patch by it keeps disappearing from the check queue (2nd time now) - are you resetting the queue by any chance? | 21:58 |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Add Constraints support https://review.openstack.org/412747 | 21:59 |
| jeblair | andreaf: yes, sorry, i had to restart zuul a couple times today; i think i'm okay for a couple hours if you want to recheck now. | 21:59 |
| *** felipemonteiro__ has quit IRC | 21:59 | |
| andreaf | ok thanks | 22:00 |
| *** felipemonteiro__ has joined #openstack-infra | 22:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Switch job.nodes to nodeset https://review.openstack.org/506396 | 22:00 |
| *** zhenguo has joined #openstack-infra | 22:00 | |
| *** edwarnicke has joined #openstack-infra | 22:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Add a base job for puppet-openstack-integration https://review.openstack.org/506385 | 22:02 |
| *** iyamahat_ has joined #openstack-infra | 22:02 | |
| *** iyamahat has quit IRC | 22:02 | |
| *** rm_work has joined #openstack-infra | 22:02 | |
| *** rm_work has quit IRC | 22:02 | |
| *** rm_work has joined #openstack-infra | 22:02 | |
| *** xarses_ has quit IRC | 22:03 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Revert "Support networkx 2.0" https://review.openstack.org/506398 | 22:03 |
| *** efried has joined #openstack-infra | 22:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Switch job.nodes to nodeset https://review.openstack.org/506399 | 22:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Add legacy-dsvm-base job for converted devstack jobs https://review.openstack.org/506373 | 22:05 |
| *** felipemonteiro__ has quit IRC | 22:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove tox_upper_constraints_file https://review.openstack.org/506401 | 22:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Re-remove openstack-doc-build from project-config https://review.openstack.org/506402 | 22:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Rename tox_upper_constraints_file to tox_constraints_file https://review.openstack.org/506403 | 22:11 |
| clarkb | pabelanger: since you debugged the xenial images yesterday. ianw says next release of dib (2.9) should have the fix we need | 22:12 |
| *** wolverineav has quit IRC | 22:12 | |
| *** ijw has joined #openstack-infra | 22:13 | |
| ianw | clarkb / pabelanger: yep ... it turned out the kernel dropped requirements on initrd tools to help make smaller container images | 22:13 |
| ianw | because we boot with root=LABEL=... , no initrd meant we couldn't find the root partition | 22:14 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Create build-openstack-sphinx-docs job https://review.openstack.org/504785 | 22:14 |
| ianw | (kernel doesn't support LABEL=, the initrd probes for it and does a switch root) | 22:14 |
| openstackgerrit | Tony Breeds proposed openstack/diskimage-builder master: Add Constraints support https://review.openstack.org/412747 | 22:15 |
| *** srobert_ has quit IRC | 22:16 | |
| *** slaweq has joined #openstack-infra | 22:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Stop using docs-draft.openstack.org for doc drafts https://review.openstack.org/506305 | 22:18 |
| *** yolanda has quit IRC | 22:18 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: make diskimage-builder use upper-constraints https://review.openstack.org/506388 | 22:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Allow mapping to handle pep8 for trusty https://review.openstack.org/506336 | 22:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove comment about unused required-projects https://review.openstack.org/506337 | 22:20 |
| *** yolanda has joined #openstack-infra | 22:20 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Migrate node information to nodeset instead of nodes https://review.openstack.org/506405 | 22:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fixed a few earlier review nits https://review.openstack.org/506406 | 22:21 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Setting LOGFILE https://review.openstack.org/506407 | 22:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch job.nodes to nodeset https://review.openstack.org/506394 | 22:22 |
| *** slaweq has quit IRC | 22:22 | |
| mordred | ianw: WOW FUN | 22:23 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Switch job.nodes to nodeset https://review.openstack.org/506396 | 22:23 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Switch job.nodes to nodeset https://review.openstack.org/506399 | 22:23 |
| *** bobh has quit IRC | 22:23 | |
| *** iyamahat_ has quit IRC | 22:23 | |
| jlk | ugh, that's.... neat... | 22:23 |
| openstackgerrit | Merged openstack-infra/openstackid master: Fixed OAUTH2 client config https://review.openstack.org/506384 | 22:23 |
| mordred | ianw: so you're saying that in addition to our server operating systems having design choices dictated by the needs of desktop systems - we now ALSO have server OS needs being dictated by the needs of folks doing tiny lightweight containers? that's eversonice | 22:24 |
| *** iyamahat has joined #openstack-infra | 22:24 | |
| clarkb | mordred: on stable long term support releases no less | 22:24 |
| jlk | mordred: well... you kind of lost that game at systemd | 22:24 |
| mordred | jlk: do you ever love how the place in which linux has been categorically DOMINANT in basically every conceivable manner seems to be perpetually an annoyance or afterthought to everyone in the business? | 22:24 |
| ianw | i did think it was a bit of a surprising thing to pop in a xenial kernel | 22:24 |
| mordred | jlk: yes. that was "needs of desktop" | 22:25 |
| *** xarses_ has joined #openstack-infra | 22:25 | |
| jlk | for some reason, we make the people who work on the server side of things use the OS as a desktop | 22:25 |
| jlk | so it's only natural those people focus on what's in front of them | 22:25 |
| jlk | ianw: I'm sure somebody had fun adding to a text matrix | 22:26 |
| jlk | test | 22:26 |
| pabelanger | clarkb: ianw: okay, cool. Is that something we'll be tagging tonight? | 22:26 |
| ianw | mordred: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1700972 | 22:26 |
| openstack | Launchpad bug 1700972 in linux-azure (Ubuntu Xenial) "Please only recommend or suggest initramfs-tools | linux-initramfs-tool for kernels able to boot without initramfs" [Low,Fix committed] - Assigned to Marcelo Cerri (mhcerri) | 22:26 |
| ianw | i have no idea what the -azure means, and i don't want to know :) | 22:26 |
| ianw | pabelanger: yep, as soon as it merges i hope | 22:26 |
| prometheanfire | heh, didn't know zuul-git-prep-upper-constraints existed | 22:26 |
| jlk | ignore it for now | 22:26 |
| mordred | jlk: I dunno - I use linux on the desktop by choice and all of the changes to "make the desktop better" all seem to be driven by people who hate linux and unix in general | 22:26 |
| jlk | if you can :) | 22:26 |
| clarkb | mordred: and amazingly its still a better desktop than anything else I've tried >_> | 22:27 |
| jlk | mordred: they just don't want the same Linux users, they want new and different Linux users! | 22:27 |
| prometheanfire | ianw: what does zuul-git-prep-upper-constraints do? | 22:27 |
| mordred | instead turning it in to a thing that's STILL not appealing to people who prefer osx or windows, and now ALSO pisses of people who actually like linux on the desktop | 22:27 |
| Sukhdev | AJaeger : Hi Andreas any update on the infra updates? - I am still waiting on this approval - https://review.openstack.org/#/c/503829/ | 22:27 |
| pabelanger | ianw: amazing | 22:27 |
| mordred | clarkb: right? | 22:27 |
| mordred | jlk: yah. I'm not sure which users they're actually going to gain - but whatev ;) | 22:27 |
| jlk | prometheanfire: it copies the upper-constraints.txt file from requires repo into your repo's checkout, for use with tox | 22:27 |
| pabelanger | ianw: OH, and because we don't install recommends / suggests it broke us | 22:28 |
| mordred | Sukhdev: oh! sorry - that's my bad | 22:28 |
| clarkb | fwiw all of the non crazy desktop stuff is still out there | 22:28 |
| pabelanger | neat | 22:28 |
| jlk | prometheanfire: but we're doing something different for zuul v3 | 22:28 |
| * clarkb runs xmonad on xfce happily | 22:28 | |
| clarkb | and the experience is basically the same as it was 10 years ago | 22:28 |
| mordred | Sukhdev: I was asking folks to hold off on those while I got some manage-projects changes landed | 22:28 |
| mordred | Sukhdev: I have just +3d that | 22:28 |
| prometheanfire | jlk: ok, that's part of what we want | 22:28 |
| prometheanfire | awesome, just awesome :D | 22:29 |
| jlk | prometheanfire: yeah, in v3 base jobs, we're automatically doing it | 22:29 |
| mordred | prometheanfire: yah - EVERYTHING about that is different in v3 :) | 22:29 |
| jlk | for the openstack v3 base jobs | 22:29 |
| prometheanfire | jlk: what path? | 22:29 |
| mordred | yup | 22:29 |
| prometheanfire | mordred: I meant I use awesomewm :P | 22:29 |
| jlk | prometheanfire: we make the requires repo part of what zuul checks out for the job, and then just directly points at the checked out upper-constraints, unless your project has it's own, or you define one in a job | 22:30 |
| mordred | hahaha | 22:30 |
| fungi | clarkb: yeah, ratpoison hasn't gotten any "prettier" either (how can it with no window decorations?), which suits me just fine ;) | 22:30 |
| mordred | jlk: ++ | 22:30 |
| jlk | prometheanfire: tox can be pointed to such a file outside of the directory path it's working in | 22:30 |
| clarkb | also network manager has greatly improved so in that way its gotten better | 22:31 |
| clarkb | (though my desktop runs wicd) | 22:31 |
| ianw | pabelanger / clarkb: where did we end up with that ovh mirror? | 22:32 |
| jeblair | my desktop has a monitor on it | 22:32 |
| mordred | my desktop has a monitor on it - and is in the other room | 22:32 |
| mordred | becaue the couch is more pleasant | 22:32 |
| clarkb | ianw: it came back and pabelanger put the region back in srevice | 22:32 |
| pabelanger | yup | 22:32 |
| jeblair | mordred: couch in the office ftw | 22:33 |
| mordred | wow. the v3 status page almost looks like a normal v2 status page with all the activity right now :) | 22:33 |
| mnaser | "Unknown project openstack/puppet-aodh" | 22:33 |
| prometheanfire | ya, using networkmanager here | 22:33 |
| mnaser | i guess that has to be somehow added as well? | 22:33 |
| prometheanfire | wicd dies if exposed to too many wifi networks at once | 22:33 |
| mordred | mnaser: yah - the puppet projects aren't in v3 yet | 22:33 |
| ianw | ¯\_(ツ)_/¯ | 22:33 |
| mordred | mnaser: so we'll land that patch after we add things | 22:33 |
| pabelanger | https://review.openstack.org/506346/ will give a few more min-ready for xenial | 22:33 |
| mordred | pabelanger: I was about to say - I thought I had +2d that already | 22:34 |
| mnaser | mordred ack | 22:34 |
| mordred | maybe clarkb will nudge it across | 22:34 |
| *** sdague has quit IRC | 22:34 | |
| mordred | mnaser: BUT - long story short - I think your jobs should at least not immediately start failing post transition | 22:34 |
| * mnaser kinda wants to start writing the jobs over the weekend | 22:35 | |
| fungi | prometheanfire: my opinion of networkmangler changed completely when i discovered nmcli | 22:35 |
| clarkb | mordred: which thing? | 22:35 |
| prometheanfire | lol | 22:35 |
| mordred | odyssey4me, dmsimard, pabelanger: ^^ we should probably take a quick gander at tripleo and openstack-ansible and put in something similar like we did for puppet and devstack | 22:35 |
| mordred | clarkb: https://review.openstack.org/506346/ | 22:35 |
| Sukhdev | mordred : Thanks a bunch | 22:36 |
| clarkb | that is an aggregate min ready of 20? | 22:36 |
| clarkb | do we also have to reduce max servers on prod nodepool? | 22:37 |
| clarkb | pabelanger: ^ | 22:37 |
| clarkb | oh no min ready is negotiated via zk so should be shared | 22:37 |
| *** mriedem has quit IRC | 22:37 | |
| pabelanger | ya, prod nodepool should be okay | 22:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Adding new project networking-gluon https://review.openstack.org/503829 | 22:37 |
| clarkb | oh hey css fix did apply without a restart | 22:37 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add legacy-dsvm-base job for converted devstack jobs https://review.openstack.org/506373 | 22:38 |
| clarkb | smcginnis: can you do a hard refresh of gerrit and check if the css looks right to you now? | 22:38 |
| clarkb | or not hte css but how it renders | 22:38 |
| *** xyang1 has quit IRC | 22:38 | |
| pabelanger | Hmm | 22:39 |
| *** slaweq has joined #openstack-infra | 22:39 | |
| pabelanger | 52d3fe16 22:39:19.476 22:38:19.476 (retry 138) [7ed73b8c] push git@github.com:openstack/nova-specs.git | 22:39 |
| pabelanger | seems stuck again | 22:39 |
| mnaser | github has been very unreliable lately (i noticed r10k timeouts a yesterday) | 22:39 |
| ianw | pabelanger: ohh, did we fix that? | 22:39 |
| pabelanger | I think somebody said nova-specs had an issue? | 22:39 |
| pabelanger | ianw: I'm not sure what the original issue is | 22:40 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/devstack-gate master: Copy over docker logs if any https://review.openstack.org/506411 | 22:40 |
| mordred | mnaser: converted things http://logs.openstack.org/89/506389/2/check/zuul-migrate/8aeb5ea/ can now be looked upon | 22:40 |
| ianw | pabelanger: the git repo is corrupt, i have a suggested fix but never found a good time to try it, let me find the email | 22:40 |
| ianw | pabelanger: http://lists.openstack.org/pipermail/openstack-dev/2017-June/119166.html | 22:41 |
| ianw | i think, if we just copy the repo from ~gerrit2/ ... somewhere, then run the commands against it ... we should be ok? i don't think gerrit needs to be shut down? | 22:41 |
| mordred | mnaser: so if you look in http://logs.openstack.org/89/506389/2/check/zuul-migrate/8aeb5ea/zuul.d/99converted-jobs.yaml - you can see the job definitions and project-templates | 22:41 |
| ianw | clarkb: ^^ ? will gerrit care if we fiddle with a repo under it if it's likely not in use? | 22:42 |
| mordred | mnaser: http://logs.openstack.org/89/506389/2/check/zuul-migrate/8aeb5ea/zuul.d/99converted-projects.yaml has the project pipeline definitions | 22:42 |
| mordred | mnaser: and the content itself is http://logs.openstack.org/89/506389/2/check/zuul-migrate/8aeb5ea/playbooks/legacy/ | 22:42 |
| mnaser | mordred having a look now | 22:42 |
| fungi | ianw: i have always assumed it does care if you fiddle with git repos while it's running (and in fact may require a reindex after starting it back up again) | 22:43 |
| ianw | fungi: ok ... well that complicates things | 22:43 |
| fungi | well, online reindex is likely fine for that part | 22:43 |
| ianw | in my defense, i figured there was enough going on with the updates to not do this while gerrit was stopped last few days :) | 22:44 |
| fungi | yup | 22:45 |
| *** slaweq has quit IRC | 22:45 | |
| *** xarses_ has quit IRC | 22:46 | |
| mnaser | mordred i thiiink i found a bug but please be patient as it might just be me missing things | 22:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Don't emit net-info macro https://review.openstack.org/506412 | 22:46 |
| mordred | mnaser: nah- you probably found a bug :) | 22:46 |
| mnaser | zuul/layout.yaml 'gate-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet' has a bunch of skip-ifs | 22:47 |
| mordred | mnaser: (eyeballs on the output is very welcome) | 22:47 |
| mnaser | http://logs.openstack.org/89/506389/2/check/zuul-migrate/8aeb5ea/zuul.d/99converted-projects.yaml | 22:47 |
| mnaser | look at openstack/puppet-aodh | 22:47 |
| mnaser | only the legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet job has the branch and irrelevant files stuff | 22:47 |
| mnaser | legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-puppet has no branch *but* has skip-if.. which are not listed | 22:47 |
| mnaser | https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1696-L1705 | 22:47 |
| openstackgerrit | Merged openstack-infra/project-config master: Bump min-ready to 10 for ubuntu-xenial https://review.openstack.org/506346 | 22:49 |
| mnaser | (and i have one more thing which is more of a nit (because of the amount of duplication it creates) | 22:50 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Setting LOGFILE https://review.openstack.org/506407 | 22:51 |
| mordred | mnaser: fascinating on the 0001 ... because they're applied for openstack/puppet-tripleo | 22:52 |
| mnaser | mordred the other thing (which may be related? or not?) -- the scenario jobs are not listed as templates but listed as joke | 22:52 |
| mnaser | s/joke/job (lol) | 22:52 |
| mnaser | i.e. puppet-desigate has "puppet-openstack-integration-jobs-scenario003" as template, and no defined jobs, in the converted projects.yaml, it has the jobs listed there which creates a ton of duplication | 22:53 |
| *** Sree has joined #openstack-infra | 22:54 | |
| smcginnis | clarkb: Search box looks much better! | 22:54 |
| mnaser | mordred ^ oh yeah, search box is back to regular size so hopefully that makes your experience better | 22:55 |
| mordred | mnaser: yes! it's much better | 22:55 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Emit shell instead of script tasks https://review.openstack.org/505379 | 22:55 |
| mordred | mnaser: the 003 thing is a by-product of needing to expand templates to apply the per-project filters and whatnot - but I'm confused why puppet-designate is expanded like that | 22:55 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Omit some jobs from shared queue calculation https://review.openstack.org/505380 | 22:56 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Deal with link-logs macro https://review.openstack.org/505387 | 22:56 |
| *** Sukhdev has quit IRC | 22:57 | |
| mordred | mnaser: we should get Shrews to look at that conversion tomorrow (hoping he's back online | 22:57 |
| mordred | mnaser: because there are times when we have to expand the template ... but puppet-openstack-integration-jobs-scenario003 does't look like one of them | 22:58 |
| *** Sree has quit IRC | 22:59 | |
| clarkb | mordred: re jeepyb and new projects, did anyone get around to removing project creator from the initial group creation? | 22:59 |
| mnaser | mordred i wonder if its the little detection code you wrote which is manipulating it, it might be productive to see a patch that runs the convert without that addition.. but whatever you think moves us forward | 23:00 |
| clarkb | I'll have to take a look at it if not | 23:00 |
| mordred | mnaser: if you look further up in that stack - http://logs.openstack.org/79/506379/2/check/zuul-migrate/a2e5557/ for instance | 23:00 |
| mordred | before that - you'll can look at the previous migrate run | 23:01 |
| *** xarses_ has joined #openstack-infra | 23:01 | |
| mordred | mnaser: allso - if you get bored and want to play locally - the migration script is in the feature/zuulv3 branch of the zuul repo - there is a utility script called tools/run-migration.sh that will run it - and it expects project-config repo to be adjacent to the zuul repo | 23:02 |
| *** iyamahat has quit IRC | 23:03 | |
| *** iyamahat_ has joined #openstack-infra | 23:03 | |
| mnaser | mordred yeah same thing even for other ones, i might have a look later | 23:03 |
| *** slaweq has joined #openstack-infra | 23:03 | |
| mordred | mnaser: also - if you look at puppet-aodh again - the exclusions are duplicated | 23:04 |
| clarkb | fungi: ianw re nova-specs is there any easy way to disable replication for that one project? | 23:04 |
| *** andreww has joined #openstack-infra | 23:04 | |
| fungi | clarkb: i think you can configure replication with a regular expression match on project name | 23:05 |
| clarkb | ooh there is replication maxRetries we can configure | 23:05 |
| fungi | oh nice | 23:05 |
| clarkb | maybe we want to set that to something like 10 | 23:05 |
| *** xarses_ has quit IRC | 23:05 | |
| clarkb | nova-specs is currently at 160 something | 23:05 |
| mordred | mnaser: and it looks like we think we applied the exclusions to puppet-aodh: http://logs.openstack.org/89/506389/2/check/zuul-migrate/8aeb5ea/job-output.txt.gz#_2017-09-21_22_29_31_760768 | 23:06 |
| fungi | clarkb: nice thing about it _not_ giving up is that it stays in our face as a repo in need of fixing | 23:07 |
| clarkb | ya | 23:07 |
| *** rbrndt has quit IRC | 23:07 | |
| fungi | if we'd already had maxretries=10 we likely would never have found out | 23:07 |
| *** slaweq has quit IRC | 23:08 | |
| mordred | mnaser: I may have found the bug | 23:10 |
| mnaser | mordred im trying to understand migrate.py and im in scanForProjectMatchers which i think is calling findReferencedTemplateNames which then gets it in `templates_to_expand` and expands it | 23:11 |
| mordred | mnaser: so - first of all MASSIVE APOLOGIES for the code in that file - it's... not pretty | 23:12 |
| mnaser | aha, no worries, it's just something to get a task done once :P | 23:13 |
| mordred | mnaser: yup | 23:14 |
| mordred | mnaser: in any case - I'm currently looking at processPipeline in applyProjectMatchers | 23:15 |
| *** hamzy has joined #openstack-infra | 23:15 | |
| mordred | mnaser: in the first part of the loop, it assigns to job but I believein that case job was a string so that assignment is getting lost | 23:15 |
| mnaser | ok so scanForProjectMatchers should return [] because nothing matches it, which means exp_template_names should be [], which means if template['name'] in exp_template_names will always evaluate to false and it shouldn't be expanded | 23:15 |
| * mnaser looks | 23:16 | |
| mnaser | (oh i was talking about the curious case of the scenarios not being templated, i think you're mentioning the missing skip-if stuff | 23:16 |
| mordred | yes - sorry - I'm looking at missing skip-if | 23:17 |
| *** claudiub has quit IRC | 23:17 | |
| mordred | (and also why is the existing skip-if duplicated - which I thin may be the same thing) | 23:17 |
| mordred | HAH - I made an amazing new bug ... | 23:18 |
| *** ijw has quit IRC | 23:18 | |
| mnaser | but isn't job being overwritten by a dict | 23:19 |
| * mnaser has never been a fan of passing things around by reference | 23:20 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Initial skeleton for devstack-tempest base job https://review.openstack.org/504246 | 23:21 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check check log files https://review.openstack.org/505840 | 23:21 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Setting LOGFILE https://review.openstack.org/506407 | 23:22 |
| mordred | I like doing it in C++ where there's a & staring at you | 23:22 |
| mnaser | yeah, in this its not very obvious | 23:22 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove Job.nodes https://review.openstack.org/505846 | 23:22 |
| mnaser | im biting in and cloning zuul to help out and figure out the template thing | 23:22 |
| *** hongbin has quit IRC | 23:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Report friendly errors when nodeset/secrets missing https://review.openstack.org/505856 | 23:23 |
| *** bobh has joined #openstack-infra | 23:24 | |
| *** bobh has quit IRC | 23:24 | |
| mordred | mnaser: ok. I think I fixed the first thing | 23:24 |
| *** slaweq has joined #openstack-infra | 23:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Correctly apply irrelevant-files to simple job names https://review.openstack.org/506415 | 23:27 |
| mordred | mnaser: ^^ | 23:27 |
| *** gildub_ has joined #openstack-infra | 23:27 | |
| mordred | mnaser: the fix to that makes the other bug make sense btw | 23:28 |
| mordred | mnaser: it's expanded from the template becaues there are file exclusions added to it | 23:28 |
| mnaser | mordred checking and im trying my hand at fixing the other one :p | 23:28 |
| mordred | mnaser: but - they weren't getting added becaue of the previous bug | 23:28 |
| mordred | so the logic was correctly expanding them from the template, but it was incorrectly applying the irrelevant-files | 23:28 |
| *** gmann has joined #openstack-infra | 23:29 | |
| mordred | mnaser: HOWEVER - if you want to keep poking ... I think there is a possible optimization that could be made if your brain doesn't melt first ... | 23:29 |
| mnaser | i see what you mean | 23:29 |
| mnaser | they're there because they should be expanded | 23:29 |
| mnaser | but because they're not they are out of place | 23:29 |
| mordred | yah | 23:29 |
| mnaser | where's the zuul v3 status page (i'll ask once i promise) | 23:29 |
| mordred | mnaser: http://zuulv3.openstack.org/ | 23:30 |
| mnaser | i tried every iteration except that one | 23:30 |
| mordred | mnaser: although once we migrate it'll be back at status.o.o/zuul | 23:30 |
| mordred | :) | 23:30 |
| mordred | mnaser: if we could determine that a given set of exclusions apply to all instances of a job, we could apply them to the base job definition itself and not need to expand the project-template | 23:30 |
| mordred | mnaser: unfortunately the "emit job" and "emit project pipeline" are two pretty unconnected loops so that won't be super straightforward | 23:31 |
| *** slaweq has quit IRC | 23:31 | |
| mordred | mnaser: but for some of these puppet-integration jobs it seems like the irrelevant-files could easily go on the job itself and remove the need for template expansion | 23:32 |
| Shrews | mordred: mnaser: uh oh | 23:32 |
| mordred | mnaser: otoh - it might be easier to just deal with that post-transition | 23:32 |
| mordred | mnaser: with a new set of shiny jobs | 23:32 |
| Shrews | much apologies for having to look at any of that code | 23:32 |
| mordred | Shrews: it actually wasn't bad! | 23:32 |
| mnaser | look at all of the irrelevant files | 23:32 |
| mnaser | the file just got so much bigger | 23:32 |
| mnaser | lol | 23:32 |
| mnaser | from 1.2M to 5.2M | 23:33 |
| mordred | yah | 23:33 |
| mordred | Shrews: https://review.openstack.org/506415 fwiw | 23:33 |
| mnaser | uhoh | 23:33 |
| mnaser | something messed up | 23:33 |
| mnaser | i curl'd the file to make sense out of it - http://logs.openstack.org/15/506415/1/check/zuul-migrate/c6d33cd/zuul.d/99converted-projects.yaml | 23:34 |
| mnaser | (dont open in a browse r and save yourself the trouble) | 23:34 |
| mnaser | and now the irrelevant-files are repeated | 23:34 |
| mnaser | (should they be a set instead of a list?) | 23:34 |
| mordred | oh! I see that - I saw that in a different place before and fixed it with one set transition | 23:34 |
| mnaser | oh dang | 23:35 |
| mnaser | if you pipe output from curl to less | 23:35 |
| mnaser | it doesnt download everything | 23:35 |
| mnaser | and downloads as you scroll through .. that's kinda cool. | 23:35 |
| mordred | mnaser: running locally with one more list(set()) added | 23:36 |
| mnaser | im just not sure if this is doing what it ssupposed to do.. like why "legacy-tempest-dsvm-swift-zaqar" job has '^designate/locale/.*$' in its irrelevant-files | 23:37 |
| mordred | yah. that seems to have done it locally | 23:37 |
| mordred | mnaser: well ... it turns out the global regexes were really hard to understand :) | 23:37 |
| mordred | mnaser: so it's entirely possible that thereis an overly broad regex somewhere that's applying to more than it should | 23:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Correctly apply irrelevant-files to simple job names https://review.openstack.org/506415 | 23:38 |
| mnaser | mordred https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1227 | 23:38 |
| mordred | mnaser: ^^ that shoudl fix the repeitition | 23:38 |
| mnaser | it looks like it may be dragging aaaaall of those rather than the project specific one | 23:38 |
| mnaser | so its not taking into question the project: inside the matcher but matching all of it | 23:38 |
| Shrews | mnaser: that might be possible. that code was created under duress | 23:39 |
| mnaser | im running through it now | 23:40 |
| *** gouthamr has joined #openstack-infra | 23:40 | |
| * mordred needs to go make the dinner - will resume with mnaser on this tomorrow - thanks for the great eyes mnaser ! | 23:40 | |
| *** markvoelker has quit IRC | 23:40 | |
| mnaser | mordred np, ill have a look and maybe try to fix this matcher thing, seems straight forward, bon appetit | 23:41 |
| *** jbryce has joined #openstack-infra | 23:45 | |
| Shrews | mnaser: if you can't figure it out, let me know (and provide an example i can look at) and i'll take a peek in the morning, too. will hopefully feel better then | 23:45 |
| mnaser | Shrews np, i'll try and will do. feel better! | 23:45 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove Job.nodes https://review.openstack.org/505846 | 23:45 |
| *** thorst has joined #openstack-infra | 23:46 | |
| *** Swami has quit IRC | 23:46 | |
| *** slaweq has joined #openstack-infra | 23:48 | |
| *** thorst has quit IRC | 23:50 | |
| *** thorst has joined #openstack-infra | 23:50 | |
| *** andreww has quit IRC | 23:52 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Report friendly errors when nodeset/secrets missing https://review.openstack.org/505856 | 23:53 |
| *** thorst has quit IRC | 23:53 | |
| *** slaweq has quit IRC | 23:54 | |
| *** michaelxin has quit IRC | 23:54 | |
| *** tosky has quit IRC | 23:54 | |
| mnaser | Shrews mordred ok looking at the latest output it seems to have fixed it | 23:54 |
| *** Goneri has joined #openstack-infra | 23:55 | |
| mordred | mnaser: the whole thing? | 23:56 |
| mnaser | mordred yeah.. i dont know why, but it seems to be clean, unduplicated. http://logs.openstack.org/15/506415/2/check/zuul-migrate/e36c200/zuul.d/99converted-jobs.yaml | 23:56 |
| mordred | oh - yah - so it does | 23:56 |
| mnaser | and seems to have the right info | 23:56 |
| mordred | I do not understand why the most recent thing fixed it | 23:56 |
| Shrews | awesome | 23:56 |
| Shrews | mordred: because magic | 23:57 |
| mordred | but le'ts just back away slowly and not ask further questions | 23:57 |
| *** vhosakot has quit IRC | 23:57 | |
| mnaser | exactly how i feel | 23:57 |
| jeblair | andreaf: i need to restart zuul again, sorry. i'll recheck the 2 changes running now. | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!