| jeblair | okay i think i grok and i think we don't want to land 519 as-is | 00:00 |
|---|---|---|
| clarkb | https://review.openstack.org/#/c/508520/4/zuul.d/zuul-legacy-jobs.yaml ya no branch there | 00:00 |
| clarkb | jeblair: if I add the branch would you expect it to help? | 00:00 |
| jeblair | the nnet-newton job is designed to have an internal branch override. so it's a job that's meant to run newton versions of software. | 00:00 |
| mordred | legacy-tempest-dsvm-nnet-newton has a branch matcher in the project definition in project-config | 00:00 |
| * clarkb is confused | 00:01 | |
| superdan | note the job doesn't actually say -newton on it | 00:01 |
| jeblair | that's distinct from branch matching, which is selecting when a job should run | 00:01 |
| clarkb | oh! | 00:01 |
| clarkb | got it | 00:01 |
| clarkb | so in this case fixing the zuul bug is what we want | 00:01 |
| jeblair | looking for mordred's thing, gimme a sec | 00:01 |
| jeblair | mordred: i think the use in project config is just on devstack-gate and tempest, yeah? the use on nova is only through the project template in ozj? | 00:03 |
| mordred | jeblair: yes - I agree with this | 00:03 |
| jeblair | okay, i don't have an opinion on whether integrated-gate-nova-net should run dsvm-nnet or dsvm-nnet-newton. but whatever job it runs, if we only want that job to run on changes to stable/newton branch, we should wait for my fix. | 00:04 |
| mordred | jeblair: ++ | 00:05 |
| jeblair | er... actually, if we want it only to run on stable/newton, we can leave the branch matcher in there as it is now, that should work. | 00:06 |
| jeblair | so, if dsvm-nnet is running on the right changes, but running with the wrong branches of software, we can change the job that's run from dsvm-nnet to dsvm-nnet-newton, but leave the branch matcher. | 00:07 |
| mordred | jeblair: uhoh. the pipeline patch we landed broke zuul unittests - I'll work on getting that fixed | 00:07 |
| mordred | jeblair: at least we're consistent with breaking thing when we do that | 00:07 |
| jeblair | mordred: :) | 00:07 |
| mordred | jeblair: actually - real quick ... | 00:08 |
| mordred | jeblair: http://logs.openstack.org/29/508629/2/check/tox-py35/a357136/testr_results.html.gz | 00:08 |
| mordred | jeblair: on the first test: AttributeError: 'NoneType' object has no attribute 'item' | 00:08 |
| jeblair | superdan, mriedem: can you clarify which thing you want to fix in 508519? job running *on* wrong branch, or job running *with* wrong branch checked out? | 00:08 |
| superdan | jeblair: running on the wrong changes I think | 00:09 |
| mordred | jeblair: is requestNodes(None) a valid thing to do? or is that just that wayfor tests? | 00:09 |
| superdan | jeblair: we should only be testing nnet on <= newton | 00:09 |
| jeblair | superdan: so that ran on master? | 00:09 |
| superdan | jeblair: it did | 00:09 |
| jeblair | superdan: can you point me at a run? | 00:09 |
| superdan | jeblair: http://logs.openstack.org/18/505418/13/check/legacy-tempest-dsvm-nnet/ac47376/ | 00:10 |
| superdan | jeblair: that change is in check right now, but that job is done | 00:10 |
| jeblair | perfect | 00:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle build_set being None for priority https://review.openstack.org/508634 | 00:13 |
| jeblair | superdan: okay, that looks related to the bug i'm fixing. it's new unexpected behavior though. it does indeed look like the branch matcher there is being ignored. so yeah, i think i need to fix that bug before we can fix this. | 00:13 |
| mordred | jeblair, clarkb: ^^ that should, I believe, fix unittests | 00:13 |
| superdan | jeblair: okay thanks.. it's been blocking nova for two days :/ | 00:13 |
| *** harlowja has quit IRC | 00:14 | |
| superdan | jeblair: if there are any other creative workarounds that are more likely to be landable in short order I think it's worth considering them | 00:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle build_set being None for priority https://review.openstack.org/508634 | 00:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Protect against builds dict changing while we iterate https://review.openstack.org/508629 | 00:14 |
| mnaser | i guess zuul is doing its reconfigure dance right now | 00:14 |
| jeblair | mnaser: just finished yes | 00:14 |
| jeblair | and started again | 00:15 |
| mnaser | ..every single time i say something | 00:15 |
| mordred | superdan: we could turn the job to non-voting or comment it out and just don't land any nova newton changes until the bug is fixed | 00:15 |
| superdan | mordred: that would be completely fine with me | 00:16 |
| mordred | superdan: I normally don't like suggesting such a thing - but newton should be fairly low traffic so a couple of days is unlikely to introduce a breakage | 00:16 |
| superdan | I don't even like newton. never did. | 00:16 |
| superdan | yeah | 00:16 |
| jeblair | mordred: *probably* just tests. | 00:17 |
| clarkb | also its nova net | 00:17 |
| superdan | I dunno how to do that thing, especially in the new world order | 00:17 |
| *** SumitNaiksatam has quit IRC | 00:17 | |
| jeblair | superdan: you're gonna like it | 00:18 |
| superdan | jeblair: I just yell at something? oh boy! | 00:18 |
| *** tosky has quit IRC | 00:19 | |
| * superdan looks for the voice command button on zuulv3.o.o | 00:19 | |
| *** bobh has joined #openstack-infra | 00:19 | |
| *** slaweq has quit IRC | 00:20 | |
| * mordred wants to make a video for a summit that's a parody of alexa commercials but with superdan saying "zuul, merge my change" | 00:20 | |
| SamYaple | superdan: ive found if you yell loud enough mordred will show up | 00:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Temporarily make nnet-newton non-voting https://review.openstack.org/508636 | 00:20 |
| SamYaple | but not to help, to yell with you | 00:20 |
| superdan | mordred: you and me. let's make it happen. | 00:20 |
| jeblair | superdan, mordred: ^ that's the non-voting version | 00:21 |
| superdan | SamYaple: heh yeah | 00:21 |
| superdan | jeblair: I also like people doing things for me :) | 00:21 |
| jeblair | we could also just not run on nova; that's pretty easy too | 00:21 |
| superdan | would that generate less load? | 00:22 |
| superdan | because now it's running on things it shouldn't | 00:22 |
| superdan | including high-traffic things like master | 00:22 |
| jeblair | yes | 00:22 |
| clarkb | I rechecked 508302 to test multinode d-g things | 00:22 |
| mordred | jeblair: I can make that one real quick | 00:22 |
| jeblair | mordred: k | 00:23 |
| clarkb | since fix for subnodes group merged above | 00:23 |
| *** slaweq has joined #openstack-infra | 00:23 | |
| mnaser | on a more positive note, p-o-i in-repo jobs are in the gate :D | 00:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Stop running integrated-gate-nova-net while it's broken https://review.openstack.org/508637 | 00:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Revert "Stop running integrated-gate-nova-net while it's broken" https://review.openstack.org/508638 | 00:24 |
| mordred | mnaser: WOOT! | 00:24 |
| mordred | jeblair, clarkb, superdan: there's turning them off - and the revert patch to turn them back on when we're ready | 00:24 |
| *** jtomasek has joined #openstack-infra | 00:25 | |
| superdan | mordred: jeblair: I'm crafting a mail to the -dev list right now to warn about newton approvals | 00:25 |
| clarkb | we should proably craft a zuulv3 update too | 00:25 |
| clarkb | mordred: ^ you had been doing those | 00:25 |
| *** thorst has quit IRC | 00:25 | |
| *** lukebrowning has quit IRC | 00:25 | |
| *** thorst has joined #openstack-infra | 00:25 | |
| mordred | clarkb: if I do it it'll likely be tomorrow or later when I'm drunk - I'm a little past EOD atm | 00:25 |
| mordred | clarkb: but I can totally do one in the morning | 00:26 |
| clarkb | ok | 00:26 |
| *** thorst has quit IRC | 00:27 | |
| mordred | also - maybe this time infra-manual will publish itself | 00:27 |
| clarkb | mordred: were fixes for docs publishing made? I havne't managed to track that | 00:27 |
| superdan | mordred: I'm going to send the email to the dev list assuming we're merging your version of the workaround, mmkay? | 00:28 |
| * clarkb reviews mordreds change to make that reality | 00:28 | |
| clarkb | +2'd anyone else able to review 508637? | 00:29 |
| *** jtomasek has quit IRC | 00:29 | |
| fungi | sure | 00:29 |
| *** baoli_ has quit IRC | 00:30 | |
| superdan | clarkb: fungi mordred: thanks, I'll send this email, go get dinner, and then return to stare at the gate with hopeful eyes | 00:30 |
| *** yamamoto has joined #openstack-infra | 00:31 | |
| superdan | I hear staring helps.. Something about a watched pot. | 00:31 |
| *** markvoelker has quit IRC | 00:31 | |
| mnaser | superdan: tbh i cant be the only one who glares at zuul status page way too often | 00:31 |
| *** lukebrowning has joined #openstack-infra | 00:32 | |
| superdan | mnaser: I have 1/4 of a monitor above my head dedicated to it 24x7 | 00:32 |
| superdan | mnaser: can you beat that? | 00:32 |
| *** vhosakot has quit IRC | 00:32 | |
| *** hemna_ has joined #openstack-infra | 00:32 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix some publishing jobs https://review.openstack.org/508562 | 00:32 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Add mention of legacy nodesets to migration instructions https://review.openstack.org/508512 | 00:32 |
| fungi | i used to, until my browser started complaining that it thought jquery was taking too long | 00:32 |
| superdan | fungi: dash.py ftw :) | 00:32 |
| fungi | heh | 00:32 |
| mnaser | superdan i have 12 tabs of it open because i lose track of it which tab i have it open in... and so i open another tab of it :-P | 00:32 |
| superdan | mnaser: that's spendy :) | 00:32 |
| * superdan -> food | 00:33 | |
| SamYaple | superdan: i have a little python script that pulls just my changes from status.json and it sits on a monitor in the corner. kinda the same thing :) | 00:33 |
| SamYaple | well i guess it pulls all changes, but it just displays mine :) | 00:33 |
| *** esberglu has joined #openstack-infra | 00:33 | |
| *** yamamoto has quit IRC | 00:36 | |
| superdan | SamYaple: https://github.com/kk7ds/openstack-gerrit-dashboard | 00:36 |
| *** lukebrowning has quit IRC | 00:36 | |
| clarkb | anyone know what git is complaining about at the bottom of http://logs.openstack.org/02/508302/3/check/legacy-tempest-dsvm-neutron-multinode-full/89fa828/logs/devstack-gate-setup-workspace-new.txt ? | 00:37 |
| SamYaple | superdan: oh would you look at that. too bad NIH | 00:37 |
| superdan | SamYaple: https://imgur.com/a/TjsMu | 00:37 |
| superdan | SamYaple: hah, indeed | 00:37 |
| clarkb | I think that may be a race in our git copying stuff? | 00:37 |
| SamYaple | im definitely going to give that a go. thanks! | 00:37 |
| *** esberglu has quit IRC | 00:38 | |
| *** lukebrowning has joined #openstack-infra | 00:38 | |
| clarkb | we rsync the dir over | 00:38 |
| clarkb | and that must include the index.lock | 00:38 |
| kfox1111 | how about this one: http://logs.openstack.org/65/508565/1/check/legacy-kolla-kubernetes-deploy-centos-binary-2-external-ovs/6f4a95a/job-output.txt.gz#_2017-09-30_00_23_30_328671 | 00:39 |
| kfox1111 | sudo yum isn't working on a centos box. | 00:39 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add missing horizon project for octavia-dashboard https://review.openstack.org/508579 | 00:39 |
| clarkb | kfox1111: its a ubuntu box | 00:40 |
| SamYaple | superdan: ouch. python2. i dont have python2. guess ill make it python3 | 00:40 |
| clarkb | kfox1111: see http://logs.openstack.org/65/508565/1/check/legacy-kolla-kubernetes-deploy-centos-binary-2-external-ovs/6f4a95a/zuul-info/host-info.primary.yaml | 00:40 |
| clarkb | kfox1111: so the node assignment isn't working | 00:40 |
| kfox1111 | ah. | 00:40 |
| kfox1111 | hmm... those come from lagacy-base nodeset now? | 00:42 |
| kfox1111 | which looks like its hardcoded to ubuntu-xenial. :/ | 00:42 |
| kfox1111 | can you override nodeset in a child job? | 00:42 |
| *** lukebrowning has quit IRC | 00:43 | |
| clarkb | yes | 00:43 |
| kfox1111 | ok. thanks. I'll try that next. | 00:43 |
| kfox1111 | oh. gotta run. thanks for the help. | 00:43 |
| clarkb | if you specify somethibg more specific it will use that instead | 00:43 |
| *** lukebrowning has joined #openstack-infra | 00:44 | |
| *** thorst has joined #openstack-infra | 00:47 | |
| *** gouthamr_ has joined #openstack-infra | 00:48 | |
| *** lukebrowning has quit IRC | 00:49 | |
| mnaser | gah of course | 00:49 |
| mnaser | http://mirror.regionone.infracloud-vanilla.openstack.org:8080/buildlogs.centos/centos/7/cloud/x86_64/openstack-pike/repodata/repomd.xml: [Errno 12] Timeout on http://mirror.regionone.infracloud-vanilla.openstack.org:8080/buildlogs.centos/centos/7/cloud/x86_64/openstack-pike/repodata/repomd.xml: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds') | 00:49 |
| mnaser | anything known about that ^ ? | 00:49 |
| mordred | mnaser: mostly thatsometimes infracloud is slow | 00:50 |
| mnaser | poop | 00:50 |
| mordred | yah | 00:50 |
| *** lukebrowning has joined #openstack-infra | 00:50 | |
| *** LindaWang has joined #openstack-infra | 00:50 | |
| mordred | I was really hoping your jobs were all going to pass! | 00:50 |
| mnaser | well i guess we'll have undergo gates again for puppet | 00:50 |
| mnaser | (which kinda sucks because there are like twice as much at this current state) | 00:51 |
| mnaser | so it makes it terrifying to hope all 20 or so pass lol | 00:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Stop running integrated-gate-nova-net while it's broken https://review.openstack.org/508637 | 00:51 |
| *** thorst has quit IRC | 00:52 | |
| mnaser | ok | 00:52 |
| mnaser | 2017-09-30 00:33:35.024003 | centos-7 | http://mirror.dfw.rax.openstack.org:8080/buildlogs.centos/centos/7/cloud/x86_64/openstack-pike/repodata/repomd.xml: [Errno 12] Timeout on http://mirror.dfw.rax.openstack.org:8080/buildlogs.centos/centos/7/cloud/x86_64/openstack-pike/repodata/repomd.xml: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds') | 00:52 |
| mnaser | something is up | 00:52 |
| mnaser | i wonder if buildlogs.centos is maybe broken | 00:53 |
| mnaser | nope its not | 00:53 |
| * mnaser shrugs | 00:53 | |
| mordred | mnaser: oh - well - that's a whole other thing - I certainly don't expect dfw to be slow | 00:53 |
| mordred | mnaser: when it rains it pours, right? | 00:54 |
| mnaser | mordred haha, i was just about to say that | 00:54 |
| *** lbragstad has joined #openstack-infra | 00:54 | |
| *** boris_42_ has quit IRC | 00:55 | |
| *** lukebrowning has quit IRC | 00:55 | |
| *** slaweq has quit IRC | 00:57 | |
| *** lukebrowning has joined #openstack-infra | 00:57 | |
| *** lukebrowning has quit IRC | 01:01 | |
| *** lukebrowning has joined #openstack-infra | 01:03 | |
| *** jascott1 has quit IRC | 01:03 | |
| *** jascott1 has joined #openstack-infra | 01:04 | |
| *** larainema has quit IRC | 01:06 | |
| *** slaweq has joined #openstack-infra | 01:07 | |
| *** lukebrowning has quit IRC | 01:08 | |
| *** jascott1 has quit IRC | 01:08 | |
| *** huanxie has joined #openstack-infra | 01:08 | |
| *** ijw has quit IRC | 01:10 | |
| *** leakypipes has quit IRC | 01:10 | |
| *** ijw has joined #openstack-infra | 01:10 | |
| *** baoli has joined #openstack-infra | 01:11 | |
| *** rossella_s has quit IRC | 01:13 | |
| *** e0ne has joined #openstack-infra | 01:16 | |
| *** mriedem has quit IRC | 01:16 | |
| *** lukebrowning has joined #openstack-infra | 01:16 | |
| *** rossella_s has joined #openstack-infra | 01:17 | |
| *** kjackal_ has joined #openstack-infra | 01:19 | |
| jeblair | especially in dallas | 01:19 |
| *** e0ne has quit IRC | 01:20 | |
| *** lukebrowning has quit IRC | 01:20 | |
| *** lukebrowning has joined #openstack-infra | 01:22 | |
| *** bobh has quit IRC | 01:22 | |
| *** gouthamr_ has quit IRC | 01:22 | |
| *** e0ne has joined #openstack-infra | 01:25 | |
| *** jtomasek has joined #openstack-infra | 01:25 | |
| *** lukebrowning has quit IRC | 01:26 | |
| *** esberglu has joined #openstack-infra | 01:27 | |
| *** lbragstad has quit IRC | 01:28 | |
| *** lukebrowning has joined #openstack-infra | 01:28 | |
| *** e0ne has quit IRC | 01:30 | |
| *** jtomasek has quit IRC | 01:30 | |
| SamYaple | if anyone is around to merge this, i would appreciate it. I am not fixing the legacy jobs and without this patch my gates are blocked https://review.openstack.org/#/c/508552/ | 01:31 |
| *** kiennt26 has joined #openstack-infra | 01:32 | |
| *** esberglu has quit IRC | 01:32 | |
| *** yamamoto has joined #openstack-infra | 01:32 | |
| *** lukebrowning has quit IRC | 01:33 | |
| *** lukebrowning has joined #openstack-infra | 01:34 | |
| *** bobh has joined #openstack-infra | 01:37 | |
| *** yamamoto has quit IRC | 01:38 | |
| *** e0ne has joined #openstack-infra | 01:40 | |
| *** zhurong has quit IRC | 01:40 | |
| *** lukebrowning has quit IRC | 01:41 | |
| *** srobert has joined #openstack-infra | 01:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove loci-jobs from project-config https://review.openstack.org/508552 | 01:43 |
| fungi | SamYaple: ^ | 01:43 |
| *** lukebrowning has joined #openstack-infra | 01:45 | |
| *** e0ne has quit IRC | 01:45 | |
| *** srobert has quit IRC | 01:48 | |
| *** srobert has joined #openstack-infra | 01:49 | |
| *** lukebrowning has quit IRC | 01:49 | |
| *** zhurong has joined #openstack-infra | 01:51 | |
| *** srobert has quit IRC | 01:51 | |
| *** e0ne has joined #openstack-infra | 01:51 | |
| *** lukebrowning has joined #openstack-infra | 01:51 | |
| *** srobert has joined #openstack-infra | 01:51 | |
| *** srobert has quit IRC | 01:53 | |
| *** liujiong has joined #openstack-infra | 01:53 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/zuul feature/zuulv3: Fix pep8 error https://review.openstack.org/508643 | 01:55 |
| *** lukebrowning has quit IRC | 01:56 | |
| jlvillal | fungi, ^^^ and anyone else around | 01:56 |
| *** e0ne has quit IRC | 01:56 | |
| fungi | thanks! | 01:56 |
| *** lukebrowning has joined #openstack-infra | 01:57 | |
| jlvillal | You're welcome. Thanks for all your (and others) hard work! | 01:58 |
| SamYaple | fungi: thanks so much! life saver you are | 01:58 |
| SamYaple | fungi: i wonder if you can tell me, how does one run a task as root or with sudo? | 01:59 |
| SamYaple | a simple 'become: true"? | 01:59 |
| *** e0ne has joined #openstack-infra | 01:59 | |
| *** dave-mccowan has joined #openstack-infra | 02:00 | |
| ianw | SamYaple: probably, but at some points sudo has been removed, such as running unit tests, to avoid unwanted behaviour | 02:01 |
| SamYaple | ianw: this would be a pre-run from the job to setup docker among other things | 02:01 |
| *** lukebrowning has quit IRC | 02:01 | |
| SamYaple | so sudo should be availble to those tasks, yes? | 02:01 |
| ianw | yep | 02:02 |
| SamYaple | awesome. thanks for saving me a patchset :) | 02:02 |
| fungi | yeah, we seem to use become elsewhere in native v3 jobs at any rate | 02:03 |
| fungi | or sudo in the shell | 02:03 |
| *** lukebrowning has joined #openstack-infra | 02:03 | |
| fungi | either way | 02:03 |
| SamYaple | sure but im going pure ansible where possible | 02:03 |
| fungi | there's a revoke-sudo role you can add which will remove the ability of the zuul user to elevate privileges thereafter | 02:04 |
| fungi | should you need such an assurance | 02:04 |
| *** e0ne has quit IRC | 02:04 | |
| SamYaple | clever! thanks | 02:04 |
| fungi | basically just a translation of our old revoke-sudo jjb macro | 02:04 |
| *** SumitNaiksatam has joined #openstack-infra | 02:05 | |
| fungi | we rely on it in more basic jobs to make sure that things like unit tests trying to run things as root don't sneak in when they're unwanted | 02:05 |
| SamYaple | i think i can safely revoke-sudo after setup-gate, and i will attempt to do just that to enforce some structure on my gates | 02:06 |
| SamYaple | im really digging zuulv3. you all did a great job on it and so far its been worth the wait | 02:07 |
| *** lukebrowning has quit IRC | 02:08 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/openstack-zuul-jobs master: Remove legacy loci jobs https://review.openstack.org/508556 | 02:10 |
| *** dave-mccowan has quit IRC | 02:12 | |
| *** e0ne has joined #openstack-infra | 02:12 | |
| *** slaweq has quit IRC | 02:12 | |
| *** lukebrowning has joined #openstack-infra | 02:16 | |
| *** e0ne has quit IRC | 02:17 | |
| *** slaweq has joined #openstack-infra | 02:17 | |
| *** caphrim007 has joined #openstack-infra | 02:20 | |
| *** e0ne has joined #openstack-infra | 02:20 | |
| *** lukebrowning has quit IRC | 02:21 | |
| *** esberglu has joined #openstack-infra | 02:21 | |
| *** lukebrowning has joined #openstack-infra | 02:22 | |
| SamYaple | so is merge-check goign away? | 02:23 |
| fungi | yeah, that predates similar functionality in newer gerrit releases, and it's also a lot of extra strain on the mergers (especially when we're not running as many as we normally would) | 02:24 |
| *** e0ne has quit IRC | 02:25 | |
| *** caphrim007 has quit IRC | 02:25 | |
| SamYaple | cool. just making sure | 02:25 |
| SamYaple | heard some chatter about it | 02:25 |
| *** bobh has quit IRC | 02:25 | |
| *** esberglu has quit IRC | 02:26 | |
| SamYaple | do you know what variables are available to ansible? I am trying to get $NODEPOOL_DOCKER_REGISTRY_PROXY in an ansible template | 02:26 |
| *** lukebrowning has quit IRC | 02:26 | |
| *** lukebrowning has joined #openstack-infra | 02:28 | |
| SamYaple | also, im using the group "jenkins" for some permission things.... is that going away? | 02:28 |
| fungi | reconsuming envvars in ansible itself is a bit beyond me at the moment, i've only seen us access them from other shell tasks, but someone with deeper knowledge can likely say | 02:29 |
| fungi | as for the "jenkins" unix group, i expect that will be going but there's probably a "zuul" group replacing it | 02:30 |
| SamYaple | but "zuul" doesn't exist just yet, correct? | 02:31 |
| *** markvoelker has joined #openstack-infra | 02:32 | |
| fungi | it should, but lemme check our dib element that creates the user | 02:32 |
| SamYaple | im writing fresh gates, so anything i can do now to be zuulv3ish i would like to | 02:33 |
| *** thorst has joined #openstack-infra | 02:33 | |
| *** lukebrowning has quit IRC | 02:33 | |
| fungi | jobs, but yeah | 02:34 |
| fungi | understood | 02:34 |
| *** yamamoto has joined #openstack-infra | 02:34 | |
| SamYaple | jobs right | 02:35 |
| *** lukebrowning has joined #openstack-infra | 02:35 | |
| fungi | SamYaple: https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/zuul-worker/install.d/60-zuul-worker#n12 | 02:35 |
| fungi | groupadd zuul | 02:35 |
| fungi | it's run as part of our image build process on all distros we run | 02:35 |
| SamYaple | awesome sauce. so i can switch to that instead of jenkins | 02:35 |
| *** e0ne has joined #openstack-infra | 02:36 | |
| SamYaple | mmmmm its been a few months since ive ansible'd anything. all saltstack recently. ansible definetely has some nice stuff | 02:37 |
| *** harlowja has joined #openstack-infra | 02:39 | |
| *** lukebrowning has quit IRC | 02:39 | |
| *** yamamoto has quit IRC | 02:39 | |
| *** e0ne has quit IRC | 02:40 | |
| *** thorst has quit IRC | 02:40 | |
| *** thorst has joined #openstack-infra | 02:40 | |
| *** thorst has quit IRC | 02:40 | |
| *** lukebrowning has joined #openstack-infra | 02:41 | |
| *** bobh has joined #openstack-infra | 02:42 | |
| SamYaple | fungi: http://logs.openstack.org/25/508625/9/check/loci-cinder/abc9539/job-output.txt.gz#_2017-09-30_02_41_52_648471 | 02:43 |
| SamYaple | from that start point to were it starts installing docker are all the vars and envs stuff you can access | 02:43 |
| *** lukebrowning has quit IRC | 02:45 | |
| SamYaple | {{ nodepool }} is a dict with enough info for me to put together NODEPOOL_DOCKER_REGISTRY_PROXY it looks like | 02:46 |
| *** bobh has quit IRC | 02:47 | |
| Jeffrey4l | compare these two files. | 02:47 |
| Jeffrey4l | http://logs.openstack.org/76/508376/8/check/legacy-kolla-dsvm-build-centos-binary-centos-7/871628e/zuul-info/inventory.yaml | 02:47 |
| Jeffrey4l | http://logs.openstack.org/76/508376/8/check/legacy-kolla-dsvm-build-ubuntu-binary/5961d75/zuul-info/inventory.yaml | 02:47 |
| *** e0ne has joined #openstack-infra | 02:47 | |
| Jeffrey4l | why one of the filename is centos-7, another one is primary? | 02:47 |
| SamYaple | Jeffrey4l: it now defaults to ubuntu-xenial | 02:48 |
| SamYaple | if you want a different distro you apend the distro you want (centos-7 in this case) | 02:48 |
| SamYaple | its pulled from the name of hte job | 02:49 |
| Jeffrey4l | SamYaple, but base on these node. the /etc/nodepool/primary_node_private will be not exist. https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/playbooks/legacy/pre.yaml#L20 | 02:49 |
| Jeffrey4l | SamYaple, have any idea to fix this? | 02:50 |
| *** sree has joined #openstack-infra | 02:50 | |
| SamYaple | Jeffrey4l: im pretty sure multinode stuff is still borked for zuulv3, and thats part of multinode | 02:50 |
| SamYaple | not positive though, so i open up the question to the channel | 02:50 |
| *** slaweq has quit IRC | 02:51 | |
| SamYaple | Jeffrey4l: ohhh nvm. i understand your question sorry | 02:51 |
| *** lukebrowning has joined #openstack-infra | 02:51 | |
| SamYaple | Jeffrey4l: im not sure as to the answer | 02:51 |
| *** e0ne has quit IRC | 02:52 | |
| *** sree has quit IRC | 02:52 | |
| Jeffrey4l | aha, thanks. | 02:53 |
| clarkb | the node names are decouples from the distro | 02:53 |
| clarkb | the inventory file will tell you ehat they are though | 02:53 |
| Jeffrey4l | clarkb, how i can get the inventory file path directly? in my script? | 02:54 |
| clarkb | that I dont know. It is available though | 02:54 |
| clarkb | I just dont know details | 02:54 |
| Jeffrey4l | got. thanks. | 02:55 |
| *** harlowja has quit IRC | 02:55 | |
| *** lukebrowning has quit IRC | 02:55 | |
| *** e0ne has joined #openstack-infra | 02:57 | |
| *** lukebrowning has joined #openstack-infra | 02:57 | |
| jeblair | SamYaple, Jeffrey4l: you both may be interested in https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html | 02:59 |
| jeblair | though it looks like nodepool vars aren't document there yet | 02:59 |
| jeblair | Jeffrey4l: you shouldn't be able to directly access the ansible inventory *file* in your scripts, but all the variables in the inventory should be there, so you can use them in the normal manner in ansible | 03:00 |
| *** slaweq has joined #openstack-infra | 03:00 | |
| Jeffrey4l | yes. i found the inventory is copied to current user home folder. i can use it to get any variable i want | 03:02 |
| jeblair | (the file is outside of the work/ directory (see that doc link above), and so isn't accessible to normal jobs, but we have a trusted playbook which runs at the end of every job which copies the inventory file to the log server) | 03:02 |
| SamYaple | jeblair: yea i just dumped all the vars in ansible in a debug statement to find nodepool stuff. would it be possible to get the contents of https://github.com/openstack-infra/project-config/blob/master/nodepool/scripts/configure_mirror.sh in ansible variable form? | 03:02 |
| *** lukebrowning has quit IRC | 03:02 | |
| SamYaple | ive assembled what i need by hand for now | 03:02 |
| Jeffrey4l | https://github.com/openstack-infra/zuul/blob/68d3ba566ca622a3ea98531d1d635c802ed6d5e7/playbooks/zuul-stream/pre.yaml#L19 | 03:02 |
| *** e0ne has quit IRC | 03:02 | |
| jeblair | Jeffrey4l: oh, that's a playbook that used in zuul tests, it's not normally run. | 03:03 |
| jeblair | Jeffrey4l: you'll just access inventory variables using " {{ }} " syntax like normal | 03:03 |
| *** lukebrowning has joined #openstack-infra | 03:04 | |
| Jeffrey4l | jeblair, the script in project is render by using ansible template module? | 03:04 |
| jeblair | Jeffrey4l: oh, i see what you're saying -- you're copying it *out* of the log root in your job | 03:04 |
| jeblair | that's... unexpected. | 03:04 |
| Jeffrey4l | what i want now it get the primary node private ipv4 address. | 03:05 |
| jeblair | Jeffrey4l: i would not rely on that as an interface. i would not recommend doing that; i'm not sure that's an interface that we expect people to use. | 03:05 |
| SamYaple | Jeffrey4l: im pretty sure about what youre trying to do, just use the hostvars to get the ip address | 03:06 |
| jeblair | Jeffrey4l: just use " {{ host_vars['primary'].nodepool.private_ipv4 }} " or something like that. | 03:06 |
| jeblair | ya | 03:06 |
| *** markvoelker has quit IRC | 03:06 | |
| Jeffrey4l | SamYaple, can i use {{ hostvars }} in one of kolla's script? | 03:06 |
| Jeffrey4l | let me try. | 03:06 |
| Jeffrey4l | thanks. | 03:06 |
| SamYaple | Jeffrey4l: no, but you could write a pre-run that uses hosts vars to write out the file like you expect for your legacy script | 03:07 |
| jeblair | SamYaple: ++ | 03:07 |
| SamYaple | jeblair: im getting the hang of this! | 03:07 |
| jeblair | that's a much more stable interface | 03:07 |
| SamYaple | its like breathing | 03:07 |
| jeblair | yay! | 03:07 |
| superdan | SamYaple: (re py3) yeah, it's old from before I knew better, I will gladly merge PRs for py3 compat though :) | 03:08 |
| Jeffrey4l | SamYaple, where the pre-run should i add? | 03:08 |
| *** lukebrowning has quit IRC | 03:08 | |
| SamYaple | Jeffrey4l: check this patchset https://review.openstack.org/#/c/508625/ | 03:08 |
| Jeffrey4l | not faimilar the zuulv2 ;( | 03:08 |
| Jeffrey4l | thanks | 03:08 |
| SamYaple | were doing alot of the same stuff in LOCI | 03:08 |
| *** e0ne has joined #openstack-infra | 03:09 | |
| Jeffrey4l | so the .zuul.yaml definition will be ran automatically? cool | 03:09 |
| SamYaple | superdan: already have one ready. since requests pulls in six, i just six'd up the place | 03:09 |
| superdan | cool | 03:10 |
| *** lukebrowning has joined #openstack-infra | 03:10 | |
| jeblair | Jeffrey4l: https://docs.openstack.org/infra/manual/zuulv3.html#ansible-playbooks and https://docs.openstack.org/infra/manual/zuulv3.html#job-inheritance may help | 03:10 |
| Jeffrey4l | reading. thanks. | 03:10 |
| SamYaple | Jeffrey4l: right. youll have to play with it a bit to get the feel, and i might suggest just scrapping large parts of your legacy gate for this. knowing whats in those gate jobs, you can get to native zuulv3 pretty quickly | 03:11 |
| SamYaple | even if its just wrapping the existing scripts | 03:11 |
| Jeffrey4l | got. let me dig these more. | 03:12 |
| SamYaple | Jeffrey4l: inc0 seemed ok with the idea of turning off the gate jobs and just going straight native too, so sync up with him tomorrow :) | 03:13 |
| SamYaple | turning off the legacy jobs*** | 03:13 |
| Jeffrey4l | i will. | 03:14 |
| *** e0ne has quit IRC | 03:14 | |
| *** lukebrowning has quit IRC | 03:14 | |
| *** esberglu has joined #openstack-infra | 03:16 | |
| *** e0ne has joined #openstack-infra | 03:16 | |
| *** lukebrowning has joined #openstack-infra | 03:16 | |
| *** baoli has quit IRC | 03:17 | |
| *** esberglu has quit IRC | 03:20 | |
| *** zhurong has quit IRC | 03:21 | |
| *** e0ne has quit IRC | 03:21 | |
| *** lukebrowning has quit IRC | 03:21 | |
| *** caphrim007 has joined #openstack-infra | 03:21 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Add a concurrency limit to zuul-executor https://review.openstack.org/508649 | 03:21 |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-infra/project-config master: End gate of openstack/pylockfile https://review.openstack.org/508650 | 03:21 |
| *** lukebrowning has joined #openstack-infra | 03:22 | |
| *** Sukhdev has joined #openstack-infra | 03:23 | |
| superdan | so um | 03:24 |
| superdan | I do not hate the new live console thing for running jobs | 03:25 |
| superdan | that is all. | 03:25 |
| SamYaple | superdan: +1 to that | 03:25 |
| *** caphrim007 has quit IRC | 03:26 | |
| jeblair | Shrews, tobiash: ^ you have fans | 03:26 |
| jeblair | we have so much more planned for it too -- like the ability to stream specific service logs while a job is running | 03:27 |
| *** lukebrowning has quit IRC | 03:27 | |
| jeblair | (also, it's all backed by a simple tcp protocol; once we expose that, you can have other tools use it. it's compatible with finger.) | 03:28 |
| *** lukebrowning has joined #openstack-infra | 03:29 | |
| jeblair | superdan: so it should be a cinch to have your status console do something with the raw stream as well | 03:29 |
| superdan | jeblair: orly, that would be interesting | 03:29 |
| *** e0ne has joined #openstack-infra | 03:30 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Do not use shebang in rally legacy jobs https://review.openstack.org/508622 | 03:30 |
| *** zhurong has joined #openstack-infra | 03:32 | |
| *** slaweq has quit IRC | 03:32 | |
| *** baoli has joined #openstack-infra | 03:32 | |
| jeblair | the tcp stream is plumbed through to the executor, but we still need to multiplex it from the scheduler host | 03:33 |
| jeblair | so for example "finger c7bbb35c608541aab0faf70ac7d9edce@ze06.openstack.org" works now | 03:33 |
| *** thorst has joined #openstack-infra | 03:33 | |
| jeblair | but it's not a stable interface yet, so i'd be wary coding to it | 03:33 |
| *** lukebrowning has quit IRC | 03:33 | |
| *** e0ne has quit IRC | 03:34 | |
| *** lukebrowning has joined #openstack-infra | 03:35 | |
| jeblair | (which is the same as 'echo "c7bbb35c608541aab0faf70ac7d9edce" | nc -q -1 ze06.openstack.org 79' fwiw) | 03:36 |
| *** yamamoto has joined #openstack-infra | 03:36 | |
| *** kiennt26 has quit IRC | 03:37 | |
| *** thorst has quit IRC | 03:38 | |
| *** slaweq has joined #openstack-infra | 03:38 | |
| *** lukebrowning has quit IRC | 03:39 | |
| *** yamamoto has quit IRC | 03:41 | |
| *** e0ne has joined #openstack-infra | 03:44 | |
| *** lukebrowning has joined #openstack-infra | 03:46 | |
| *** e0ne has quit IRC | 03:49 | |
| *** kiennt26 has joined #openstack-infra | 03:49 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-infra/project-config master: Remove project pylockfile from infrastructure systems https://review.openstack.org/508656 | 03:50 |
| *** lukebrowning has quit IRC | 03:51 | |
| *** lukebrowning has joined #openstack-infra | 03:52 | |
| *** thorst has joined #openstack-infra | 03:54 | |
| *** lukebrowning has quit IRC | 03:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Do not add implied branch matchers in project-templates https://review.openstack.org/508658 | 03:58 |
| *** lukebrowning has joined #openstack-infra | 03:58 | |
| *** thorst has quit IRC | 03:58 | |
| *** ekcs has quit IRC | 03:59 | |
| *** lbragstad has joined #openstack-infra | 03:59 | |
| *** e0ne has joined #openstack-infra | 03:59 | |
| *** lukebrowning has quit IRC | 04:03 | |
| *** markvoelker has joined #openstack-infra | 04:03 | |
| *** e0ne has quit IRC | 04:04 | |
| *** lukebrowning has joined #openstack-infra | 04:05 | |
| jlvillal | jeblair, Or anyone else. So unit tests seem to be failing for zuul: http://logs.openstack.org/43/508643/1/check/tox-py35/c5b45dc/job-output.txt.gz#_2017-09-30_02_59_06_443746 | 04:06 |
| * jlvillal doesn't know how to fix it | 04:06 | |
| clarkb | jlvillal: oh the fix for that change you asked about all failing shouldve merged I think | 04:08 |
| clarkb | the log copying thing | 04:08 |
| jlvillal | clarkb, Uh, it didn't. Because pep8 is failing for zuul | 04:09 |
| jlvillal | clarkb, I proposed a pep8 fix, but now the unit tests are failing for zuul. | 04:09 |
| jlvillal | clarkb, https://review.openstack.org/#/c/508643/ | 04:09 |
| jlvillal | clarkb, But it is approved to be merged :) | 04:09 |
| *** lukebrowning has quit IRC | 04:09 | |
| *** esberglu has joined #openstack-infra | 04:10 | |
| clarkb | wel it could be failing for other reasons | 04:10 |
| clarkb | but that specific log issue should be fixes | 04:10 |
| jlvillal | clarkb, I mean zuul failed pep8, so the fix couldn't merge | 04:10 |
| jlvillal | clarkb, I proposed a pep8 fix to zuul. But now zuul itself is failing unit tests | 04:11 |
| *** lukebrowning has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:11 | |
| jlvillal | clarkb, Unless I'm misunderstanding you | 04:11 |
| clarkb | ya but the unittests themselves are failing its not the job config | 04:11 |
| *** e0ne has joined #openstack-infra | 04:12 | |
| clarkb | the errors you asked about earlier today were the job failing to copy logs in some cases | 04:12 |
| clarkb | a fix for that jas merged | 04:12 |
| jlvillal | The log file fix, I think is this: https://review.openstack.org/508629 | 04:12 |
| jlvillal | clarkb, Oh so it wasn't this patch then: https://review.openstack.org/508629 | 04:12 |
| SamYaple | when checking out repos (with required-projects/zuul-cloner) would it be possible to modify the behaviour to create a branch rather than exist on detached FETCH_HEAD? if not, where would I make changes to allow that to happen? | 04:12 |
| jlvillal | clarkb, Thanks! | 04:12 |
| jlvillal | I had thought it was the other patch | 04:12 |
| SamYaple | I need a branch to exist because I am using a local cache to feed docker builds and those want to see a branch name | 04:13 |
| clarkb | SamYaple: I think if you use zuulv3 native repo setup its checked out on the branch the change is proposed to | 04:13 |
| clarkb | so say master for changes to master | 04:14 |
| clarkb | but the master you get is the proposed state | 04:14 |
| *** esberglu has quit IRC | 04:14 | |
| SamYaple | right, so i have https://github.com/openstack/loci/blob/master/scripts/gate-setup.sh#L12-15 | 04:15 |
| SamYaple | i wouldn't need to do anything *if* it got a branch name | 04:15 |
| Jeffrey4l | is that possible to use in-project-jobs to override or inherit the legacy jobs? | 04:15 |
| *** lukebrowning has quit IRC | 04:15 | |
| SamYaple | Jeffrey4l: you probably want parent/child stuffhttps://docs.openstack.org/infra/manual/zuulv3.html#job-inheritance m | 04:15 |
| Jeffrey4l | then we can migrate the jobs little by little. | 04:16 |
| Jeffrey4l | then parent jobs will not run? | 04:16 |
| *** huanxie has quit IRC | 04:16 | |
| SamYaple | no to stop running jobs you would have to change openstack-zuul-jobs, and you can do that bit by bit | 04:16 |
| *** e0ne has quit IRC | 04:16 | |
| *** slaweq has joined #openstack-infra | 04:16 | |
| SamYaple | but my guess is you want to just *tweak* the job, and i think you can accomplish what you want with inheritance | 04:16 |
| *** ykarel has joined #openstack-infra | 04:17 | |
| *** zhurong has quit IRC | 04:17 | |
| Jeffrey4l | got thanks. | 04:17 |
| SamYaple | or rather, you would have to change project-config/zuul.d but you can tweak openstack-zuul-jobs as needed too | 04:17 |
| clarkb | SamYaple: you'd stop using zuul cloner entirely | 04:17 |
| *** lukebrowning has joined #openstack-infra | 04:17 | |
| clarkb | SamYaple: and use the repo as given | 04:17 |
| clarkb | zuulv3 pushes the repos into the test env rather than the pull of v2 | 04:18 |
| SamYaple | clarkb: no i get that. im using required-projects already, but it still leaves the branch in a deatched state so i have to entry the folder and set it as a branch | 04:18 |
| clarkb | SamYaple: I thought it wasnt auppoed to do that | 04:18 |
| SamYaple | oh well let me double check the behaviour im seeing | 04:18 |
| clarkb | and you be on a branch called master with your change as HEAD | 04:18 |
| *** armax has joined #openstack-infra | 04:18 | |
| *** baoli has quit IRC | 04:21 | |
| *** lukebrowning has quit IRC | 04:22 | |
| *** lukebrowning has joined #openstack-infra | 04:25 | |
| *** e0ne has joined #openstack-infra | 04:26 | |
| *** lukebrowning has quit IRC | 04:30 | |
| *** e0ne has quit IRC | 04:30 | |
| *** lbragstad has quit IRC | 04:30 | |
| *** lukebrowning has joined #openstack-infra | 04:31 | |
| *** e0ne has joined #openstack-infra | 04:33 | |
| *** lukebrowning has quit IRC | 04:36 | |
| *** markvoelker has quit IRC | 04:36 | |
| *** lukebrowning has joined #openstack-infra | 04:38 | |
| *** e0ne has quit IRC | 04:38 | |
| *** lukebrowning has quit IRC | 04:42 | |
| *** lukebrowning has joined #openstack-infra | 04:44 | |
| *** e0ne has joined #openstack-infra | 04:45 | |
| *** lukebrowning has quit IRC | 04:49 | |
| SamYaple | clarkb: awesome. thanks! | 04:49 |
| SamYaple | working as you described | 04:49 |
| *** slaweq has quit IRC | 04:50 | |
| *** e0ne has quit IRC | 04:50 | |
| *** yamamoto has joined #openstack-infra | 04:51 | |
| *** e0ne has joined #openstack-infra | 04:52 | |
| *** lukebrowning has joined #openstack-infra | 04:52 | |
| *** thorst has joined #openstack-infra | 04:55 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test CI job https://review.openstack.org/508660 | 04:56 |
| *** e0ne has quit IRC | 04:57 | |
| *** lukebrowning has quit IRC | 04:57 | |
| *** slaweq has joined #openstack-infra | 04:57 | |
| *** Sukhdev has quit IRC | 04:58 | |
| *** kjackal_ has quit IRC | 04:58 | |
| *** armax has quit IRC | 04:58 | |
| *** lukebrowning has joined #openstack-infra | 04:59 | |
| *** e0ne has joined #openstack-infra | 05:02 | |
| *** yamamoto has quit IRC | 05:02 | |
| *** lukebrowning has quit IRC | 05:03 | |
| cloudnull | looks like repos that come into /home/zuul/src/git.openstack.org/openstack/ have no "origin" remote, is that expected? | 05:03 |
| *** lukebrowning has joined #openstack-infra | 05:05 | |
| SamYaple | awesome! i have completely zuulv3'd my setup-gate stuff. pure ansible, no shell | 05:05 |
| SamYaple | thanks for everyones help | 05:05 |
| *** e0ne has quit IRC | 05:07 | |
| *** lukebrowning has quit IRC | 05:09 | |
| *** lukebrowning has joined #openstack-infra | 05:11 | |
| *** e0ne has joined #openstack-infra | 05:11 | |
| *** kiennt26 has quit IRC | 05:11 | |
| AJaeger_ | infra-root, infra-manual is still not publishing - I do not see any log files ;( | 05:12 |
| gcb | AJaeger_: about https://review.openstack.org/#/c/508650 , should I just touch files in zuul.d and don't touch file in zuul ? | 05:13 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Removed confusing comments https://review.openstack.org/508601 | 05:13 |
| SamYaple | gcb: yes, just touch zuul.d for now | 05:14 |
| SamYaple | the files in that patch are not currently used | 05:14 |
| gcb | SamYaple: thanks | 05:15 |
| SamYaple | gcb: youll also want to get a change ready for openstack-zuul-jobs and specify Needed-By | 05:15 |
| SamYaple | gcb: see https://review.openstack.org/#/c/508552/ | 05:15 |
| *** e0ne has quit IRC | 05:16 | |
| *** lukebrowning has quit IRC | 05:16 | |
| *** gouthamr has joined #openstack-infra | 05:18 | |
| *** e0ne has joined #openstack-infra | 05:19 | |
| gcb | SamYaple: thanks | 05:20 |
| *** caphrim007 has joined #openstack-infra | 05:22 | |
| *** e0ne has quit IRC | 05:24 | |
| *** caphrim007 has quit IRC | 05:27 | |
| *** slaweq has quit IRC | 05:28 | |
| *** lukebrowning has joined #openstack-infra | 05:28 | |
| *** jtomasek has joined #openstack-infra | 05:28 | |
| *** e0ne has joined #openstack-infra | 05:29 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test CI job https://review.openstack.org/508660 | 05:31 |
| *** slaweq has joined #openstack-infra | 05:31 | |
| *** gouthamr has quit IRC | 05:32 | |
| *** lukebrowning has quit IRC | 05:32 | |
| *** markvoelker has joined #openstack-infra | 05:33 | |
| *** jtomasek has quit IRC | 05:33 | |
| *** e0ne has quit IRC | 05:33 | |
| *** lukebrowning has joined #openstack-infra | 05:34 | |
| gcb | SamYaple: I didn't find pylockfile related stuff in openstack-infra/openstack-zuul-jobs, so don't need post patch ? | 05:35 |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-infra/project-config master: End gate of openstack/pylockfile https://review.openstack.org/508650 | 05:36 |
| SamYaple | gcb: dont touch zuul/layout | 05:38 |
| SamYaple | got to zuul.d/projects.yaml | 05:38 |
| SamYaple | thats where you want to change things | 05:39 |
| *** lukebrowning has quit IRC | 05:39 | |
| *** e0ne has joined #openstack-infra | 05:40 | |
| *** lukebrowning has joined #openstack-infra | 05:41 | |
| *** e0ne has quit IRC | 05:45 | |
| sshnaidm|off | pabelanger, seems like pypi mirror is out of sync, it fails all ovb jobs: http://logs.openstack.org/60/508660/2/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq-ocata/1fcb63d/job-output.txt.gz#_2017-09-30_05_38_50_235073 | 05:45 |
| *** lukebrowning has quit IRC | 05:45 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-infra/project-config master: End gate of openstack/pylockfile https://review.openstack.org/508650 | 05:47 |
| *** lukebrowning has joined #openstack-infra | 05:47 | |
| *** lukebrowning has quit IRC | 05:51 | |
| *** lukebrowning has joined #openstack-infra | 05:52 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack-infra/project-config master: Remove project pylockfile from infrastructure systems https://review.openstack.org/508656 | 05:53 |
| *** lukebrowning has quit IRC | 05:56 | |
| *** lukebrowning has joined #openstack-infra | 05:58 | |
| *** esberglu has joined #openstack-infra | 05:58 | |
| *** esberglu has quit IRC | 06:02 | |
| *** yamamoto has joined #openstack-infra | 06:03 | |
| *** lukebrowning has quit IRC | 06:03 | |
| *** slaweq has quit IRC | 06:05 | |
| *** markvoelker has quit IRC | 06:06 | |
| *** slaweq has joined #openstack-infra | 06:07 | |
| *** yamamoto has quit IRC | 06:07 | |
| *** thorst has quit IRC | 06:08 | |
| *** lukebrowning has joined #openstack-infra | 06:09 | |
| *** e0ne has joined #openstack-infra | 06:11 | |
| *** lukebrowning has quit IRC | 06:14 | |
| *** e0ne has quit IRC | 06:15 | |
| *** lukebrowning has joined #openstack-infra | 06:15 | |
| sshnaidm|off | if /etc/nodepool/provider gone, how do I know which cloud the job is running on? | 06:16 |
| frickler | cloudnull: yes, see https://review.openstack.org/508344 | 06:20 |
| *** lukebrowning has quit IRC | 06:20 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test CI job https://review.openstack.org/508660 | 06:20 |
| *** lukebrowning has joined #openstack-infra | 06:21 | |
| *** Douhet has quit IRC | 06:23 | |
| *** e0ne has joined #openstack-infra | 06:26 | |
| *** lukebrowning has quit IRC | 06:26 | |
| *** lukebrowning has joined #openstack-infra | 06:28 | |
| *** jtomasek has joined #openstack-infra | 06:30 | |
| *** e0ne has quit IRC | 06:31 | |
| *** lukebrowning has quit IRC | 06:32 | |
| *** lukebrowning has joined #openstack-infra | 06:33 | |
| *** kaisers has quit IRC | 06:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add missing projects for infra puppet jobs https://review.openstack.org/508526 | 06:34 |
| *** kaisers has joined #openstack-infra | 06:34 | |
| *** jtomasek has quit IRC | 06:34 | |
| *** Douhet has joined #openstack-infra | 06:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add missing projects for infra puppet jobs https://review.openstack.org/508526 | 06:36 |
| *** lukebrowning has quit IRC | 06:37 | |
| frickler | AJaeger_: api-ref is also missing projects, if you want to take a look, otherwise I'll try to do it later today http://logs.openstack.org/47/508647/1/check/legacy-api-ref/92e3398/job-output.txt.gz#_2017-09-30_06_04_46_425016 | 06:38 |
| *** lukebrowning has joined #openstack-infra | 06:39 | |
| *** slaweq has quit IRC | 06:40 | |
| *** e0ne has joined #openstack-infra | 06:40 | |
| *** jtomasek has joined #openstack-infra | 06:41 | |
| *** lukebrowning has quit IRC | 06:44 | |
| *** e0ne has quit IRC | 06:45 | |
| *** slaweq has joined #openstack-infra | 06:47 | |
| *** larainema has joined #openstack-infra | 06:49 | |
| *** esberglu has joined #openstack-infra | 06:52 | |
| *** lukebrowning has joined #openstack-infra | 06:53 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-tempest-dsvm-nnet >newton jobs https://review.openstack.org/508520 | 06:54 |
| *** e0ne has joined #openstack-infra | 06:55 | |
| *** esberglu has quit IRC | 06:57 | |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/openstack-zuul-jobs master: Add kolla-ansible dependency for kolla project https://review.openstack.org/508664 | 06:57 |
| openstackgerrit | Numan Siddique proposed openstack-infra/project-config master: ovn: Make scenario007-multinode-oooq-container voting https://review.openstack.org/502899 | 06:57 |
| *** lukebrowning has quit IRC | 06:58 | |
| *** lukebrowning has joined #openstack-infra | 07:00 | |
| *** e0ne has quit IRC | 07:00 | |
| *** yamamoto has joined #openstack-infra | 07:04 | |
| *** lukebrowning has quit IRC | 07:04 | |
| *** thorst has joined #openstack-infra | 07:05 | |
| *** yamamoto has quit IRC | 07:08 | |
| *** e0ne has joined #openstack-infra | 07:11 | |
| *** esberglu has joined #openstack-infra | 07:13 | |
| *** lukebrowning has joined #openstack-infra | 07:13 | |
| *** e0ne has quit IRC | 07:17 | |
| *** esberglu has quit IRC | 07:17 | |
| *** lukebrowning has quit IRC | 07:18 | |
| *** slaweq has quit IRC | 07:19 | |
| *** lukebrowning has joined #openstack-infra | 07:20 | |
| *** e0ne has joined #openstack-infra | 07:22 | |
| *** kzaitsev1pi has joined #openstack-infra | 07:22 | |
| *** e0ne has quit IRC | 07:22 | |
| *** caphrim007 has joined #openstack-infra | 07:23 | |
| *** lukebrowning has quit IRC | 07:24 | |
| *** kzaitsev_pi has quit IRC | 07:24 | |
| *** slaweq has joined #openstack-infra | 07:27 | |
| *** caphrim007 has quit IRC | 07:27 | |
| *** pcaruana has joined #openstack-infra | 07:30 | |
| *** lukebrowning has joined #openstack-infra | 07:31 | |
| *** armax has joined #openstack-infra | 07:32 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add missing projects to telemetry jobs https://review.openstack.org/508448 | 07:34 |
| *** lukebrowning has quit IRC | 07:35 | |
| *** yamamoto has joined #openstack-infra | 07:35 | |
| *** lukebrowning has joined #openstack-infra | 07:37 | |
| *** liujiong has quit IRC | 07:40 | |
| *** lukebrowning has quit IRC | 07:41 | |
| *** ykarel_ has joined #openstack-infra | 07:43 | |
| *** lukebrowning has joined #openstack-infra | 07:43 | |
| *** ykarel has quit IRC | 07:45 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check subunit file processing https://review.openstack.org/508171 | 07:45 |
| *** xinliang has quit IRC | 07:46 | |
| *** lukebrowning has quit IRC | 07:48 | |
| *** esberglu has joined #openstack-infra | 07:48 | |
| *** armax has quit IRC | 07:49 | |
| *** yamamoto has quit IRC | 07:50 | |
| *** esberglu has quit IRC | 07:53 | |
| *** lukebrowning has joined #openstack-infra | 07:55 | |
| *** xinliang has joined #openstack-infra | 07:59 | |
| *** lukebrowning has quit IRC | 07:59 | |
| *** slaweq has quit IRC | 08:00 | |
| *** lukebrowning has joined #openstack-infra | 08:01 | |
| *** markvoelker has joined #openstack-infra | 08:03 | |
| *** thorst has quit IRC | 08:03 | |
| *** lukebrowning has quit IRC | 08:05 | |
| *** lukebrowning has joined #openstack-infra | 08:06 | |
| *** thorst has joined #openstack-infra | 08:08 | |
| *** slaweq has joined #openstack-infra | 08:10 | |
| *** lukebrowning has quit IRC | 08:11 | |
| *** shardy has quit IRC | 08:11 | |
| *** lukebrowning has joined #openstack-infra | 08:12 | |
| *** thorst has quit IRC | 08:13 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add requirements to bifrost jobs https://review.openstack.org/508452 | 08:15 |
| *** lukebrowning has quit IRC | 08:17 | |
| *** lukebrowning has joined #openstack-infra | 08:19 | |
| *** jascott1 has joined #openstack-infra | 08:21 | |
| *** rtjure has quit IRC | 08:22 | |
| *** lukebrowning has quit IRC | 08:23 | |
| *** lukebrowning has joined #openstack-infra | 08:25 | |
| *** jascott1 has quit IRC | 08:25 | |
| *** lukebrowning has quit IRC | 08:29 | |
| *** lukebrowning has joined #openstack-infra | 08:31 | |
| *** lukebrowning has quit IRC | 08:36 | |
| *** markvoelker has quit IRC | 08:36 | |
| *** gcb has quit IRC | 08:37 | |
| *** lukebrowning has joined #openstack-infra | 08:38 | |
| *** d0ugal has joined #openstack-infra | 08:41 | |
| *** lukebrowning has quit IRC | 08:42 | |
| *** esberglu has joined #openstack-infra | 08:42 | |
| *** slaweq has quit IRC | 08:43 | |
| *** lukebrowning has joined #openstack-infra | 08:44 | |
| *** esberglu has quit IRC | 08:47 | |
| *** lukebrowning has quit IRC | 08:48 | |
| *** yamamoto has joined #openstack-infra | 08:50 | |
| *** slaweq has joined #openstack-infra | 08:53 | |
| *** lukebrowning has joined #openstack-infra | 08:55 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** lukebrowning has quit IRC | 08:59 | |
| *** lukebrowning has joined #openstack-infra | 09:01 | |
| *** lukebrowning has quit IRC | 09:06 | |
| *** apuimedo has quit IRC | 09:07 | |
| *** lukebrowning has joined #openstack-infra | 09:07 | |
| *** thorst has joined #openstack-infra | 09:09 | |
| *** bradm_ is now known as bradm | 09:12 | |
| *** lukebrowning has quit IRC | 09:12 | |
| *** lukebrowning has joined #openstack-infra | 09:14 | |
| *** psachin has joined #openstack-infra | 09:17 | |
| *** tosky has joined #openstack-infra | 09:17 | |
| *** lukebrowning has quit IRC | 09:18 | |
| *** lukebrowning has joined #openstack-infra | 09:20 | |
| *** lukebrowning has quit IRC | 09:24 | |
| *** caphrim007 has joined #openstack-infra | 09:24 | |
| *** slaweq has quit IRC | 09:25 | |
| *** lukebrowning has joined #openstack-infra | 09:26 | |
| *** caphrim007 has quit IRC | 09:29 | |
| *** lukebrowning has quit IRC | 09:31 | |
| *** bhavik1 has joined #openstack-infra | 09:31 | |
| *** lukebrowning has joined #openstack-infra | 09:32 | |
| *** d0ugal has quit IRC | 09:33 | |
| *** slaweq has joined #openstack-infra | 09:33 | |
| *** markvoelker has joined #openstack-infra | 09:34 | |
| *** bhavik1 has quit IRC | 09:35 | |
| *** xinliang has quit IRC | 09:35 | |
| *** esberglu has joined #openstack-infra | 09:37 | |
| *** lukebrowning has quit IRC | 09:37 | |
| *** lukebrowning has joined #openstack-infra | 09:39 | |
| *** esberglu has quit IRC | 09:41 | |
| *** lukebrowning has quit IRC | 09:43 | |
| *** lukebrowning has joined #openstack-infra | 09:45 | |
| *** lukebrowning has quit IRC | 09:50 | |
| *** yamamoto has joined #openstack-infra | 09:52 | |
| *** lukebrowning has joined #openstack-infra | 09:55 | |
| *** yamamoto has quit IRC | 09:57 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add python-tempestconf project https://review.openstack.org/508502 | 10:00 |
| *** lukebrowning has quit IRC | 10:00 | |
| *** lukebrowning has joined #openstack-infra | 10:01 | |
| *** thorre_se has joined #openstack-infra | 10:03 | |
| *** rtjure has joined #openstack-infra | 10:04 | |
| *** lukebrowning has quit IRC | 10:05 | |
| *** slaweq has quit IRC | 10:07 | |
| *** thorre has quit IRC | 10:07 | |
| *** thorre_se is now known as thorre | 10:07 | |
| *** markvoelker has quit IRC | 10:07 | |
| *** lukebrowning has joined #openstack-infra | 10:07 | |
| *** thorst has quit IRC | 10:08 | |
| *** lukebrowning has quit IRC | 10:12 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add python-tempestconf project https://review.openstack.org/508502 | 10:12 |
| *** slaweq has joined #openstack-infra | 10:13 | |
| *** lukebrowning has joined #openstack-infra | 10:14 | |
| *** lukebrowning has quit IRC | 10:18 | |
| *** lukebrowning has joined #openstack-infra | 10:20 | |
| *** lukebrowning has quit IRC | 10:25 | |
| *** lukebrowning has joined #openstack-infra | 10:26 | |
| *** slaweq has quit IRC | 10:26 | |
| *** slaweq has joined #openstack-infra | 10:26 | |
| *** yamamoto has joined #openstack-infra | 10:28 | |
| *** lukebrowning has quit IRC | 10:30 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Add missing project to legacy-api-ref job https://review.openstack.org/508673 | 10:31 |
| *** lukebrowning has joined #openstack-infra | 10:33 | |
| *** LindaWang has quit IRC | 10:35 | |
| *** lukebrowning has quit IRC | 10:37 | |
| *** lukebrowning has joined #openstack-infra | 10:37 | |
| *** pcaruana has quit IRC | 10:41 | |
| *** lukebrowning has quit IRC | 10:42 | |
| frickler | so there is still some issue with devstack multinode jobs, there is too little action on the subnode, causing grenade to fail for lack of a second compute node | 10:44 |
| frickler | http://logs.openstack.org/66/508366/2/check/legacy-grenade-dsvm-neutron-multinode/5817e99/zuul-info/inventory.yaml | 10:44 |
| *** lukebrowning has joined #openstack-infra | 10:44 | |
| frickler | "subnodes: hosts: secondary: null" at the bottom there looks suspicious | 10:44 |
| *** yamamoto has quit IRC | 10:45 | |
| *** pbourke has quit IRC | 10:46 | |
| *** pbourke has joined #openstack-infra | 10:48 | |
| *** lukebrowning has quit IRC | 10:48 | |
| *** lukebrowning has joined #openstack-infra | 10:50 | |
| *** yamamoto has joined #openstack-infra | 10:53 | |
| *** lukebrowning has quit IRC | 10:54 | |
| *** lukebrowning has joined #openstack-infra | 10:56 | |
| *** LindaWang has joined #openstack-infra | 10:58 | |
| *** lukebrowning has quit IRC | 11:01 | |
| *** lukebrowning has joined #openstack-infra | 11:02 | |
| *** thorst has joined #openstack-infra | 11:05 | |
| *** lukebrowning_ has joined #openstack-infra | 11:06 | |
| *** lukebrowning has quit IRC | 11:07 | |
| *** lukebrowning has joined #openstack-infra | 11:10 | |
| *** lukebrowning_ has quit IRC | 11:10 | |
| *** chenying_ has quit IRC | 11:11 | |
| *** liusheng has quit IRC | 11:11 | |
| *** lukebrowning has quit IRC | 11:15 | |
| *** yamamoto has quit IRC | 11:20 | |
| *** yamamoto has joined #openstack-infra | 11:20 | |
| *** lukebrowning has joined #openstack-infra | 11:21 | |
| *** esberglu has joined #openstack-infra | 11:25 | |
| *** thorst has quit IRC | 11:25 | |
| *** caphrim007 has joined #openstack-infra | 11:26 | |
| *** thorst has joined #openstack-infra | 11:26 | |
| *** lukebrowning has quit IRC | 11:26 | |
| *** lukebrowning has joined #openstack-infra | 11:28 | |
| *** kjackal_ has joined #openstack-infra | 11:28 | |
| *** esberglu has quit IRC | 11:29 | |
| *** slaweq has quit IRC | 11:30 | |
| *** caphrim007 has quit IRC | 11:30 | |
| *** slaweq has joined #openstack-infra | 11:30 | |
| *** slaweq has quit IRC | 11:31 | |
| *** thorst has quit IRC | 11:31 | |
| *** lukebrowning has quit IRC | 11:32 | |
| *** lukebrowning has joined #openstack-infra | 11:34 | |
| *** psachin has quit IRC | 11:37 | |
| *** lukebrowning has quit IRC | 11:38 | |
| *** lukebrowning has joined #openstack-infra | 11:40 | |
| *** lukebrowning has quit IRC | 11:44 | |
| *** lukebrowning has joined #openstack-infra | 11:46 | |
| *** lukebrowning has quit IRC | 11:51 | |
| *** thorst has joined #openstack-infra | 11:52 | |
| *** sree has joined #openstack-infra | 11:53 | |
| *** lukebrowning has joined #openstack-infra | 11:53 | |
| *** thorst has quit IRC | 11:57 | |
| *** sree has quit IRC | 11:57 | |
| *** lukebrowning has quit IRC | 11:58 | |
| *** lukebrowning has joined #openstack-infra | 11:59 | |
| *** kiennt26 has joined #openstack-infra | 12:01 | |
| *** lukebrowning_ has joined #openstack-infra | 12:02 | |
| *** lukebrowning has quit IRC | 12:04 | |
| *** markvoelker has joined #openstack-infra | 12:04 | |
| *** lukebrowning_ has quit IRC | 12:07 | |
| *** kiennt26 has quit IRC | 12:08 | |
| *** lukebrowning has joined #openstack-infra | 12:08 | |
| *** kiennt26 has joined #openstack-infra | 12:09 | |
| *** lukebrowning_ has joined #openstack-infra | 12:10 | |
| *** lukebrowning has quit IRC | 12:13 | |
| *** lukebrowning has joined #openstack-infra | 12:13 | |
| *** thorst has joined #openstack-infra | 12:13 | |
| *** lukebrowning_ has quit IRC | 12:15 | |
| *** lukebrowning has quit IRC | 12:18 | |
| *** thorst has quit IRC | 12:18 | |
| *** esberglu has joined #openstack-infra | 12:19 | |
| *** lukebrowning has joined #openstack-infra | 12:19 | |
| *** esberglu has quit IRC | 12:23 | |
| *** lukebrowning has quit IRC | 12:24 | |
| *** lukebrowning has joined #openstack-infra | 12:26 | |
| *** lukebrowning has quit IRC | 12:30 | |
| *** lukebrowning has joined #openstack-infra | 12:31 | |
| *** lukebrowning has quit IRC | 12:35 | |
| *** lukebrowning has joined #openstack-infra | 12:37 | |
| *** markvoelker has quit IRC | 12:38 | |
| *** ykarel has joined #openstack-infra | 12:38 | |
| *** ykarel_ has quit IRC | 12:39 | |
| *** lukebrowning has quit IRC | 12:41 | |
| *** kiennt26 has quit IRC | 12:45 | |
| *** lukebrowning has joined #openstack-infra | 12:47 | |
| *** baoli has joined #openstack-infra | 12:48 | |
| *** lukebrowning has quit IRC | 12:52 | |
| *** lukebrowning has joined #openstack-infra | 12:55 | |
| *** lukebrowning_ has joined #openstack-infra | 12:58 | |
| *** lukebrowning has quit IRC | 12:59 | |
| *** lukebrowning has joined #openstack-infra | 13:01 | |
| *** sree has joined #openstack-infra | 13:01 | |
| *** lukebrowning_ has quit IRC | 13:02 | |
| *** lukebrowning has quit IRC | 13:05 | |
| mnaser | sshnaidm|off why do you need to know which cloud it works on? | 13:06 |
| *** lukebrowning has joined #openstack-infra | 13:12 | |
| *** Kevin_Zheng has quit IRC | 13:13 | |
| *** esberglu has joined #openstack-infra | 13:13 | |
| *** thorst has joined #openstack-infra | 13:15 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/openstack-zuul-jobs master: Add shade dependency to bifrost jobs https://review.openstack.org/508679 | 13:15 |
| *** LindaWang has quit IRC | 13:15 | |
| *** jpena|off has quit IRC | 13:16 | |
| *** ltomasbo has quit IRC | 13:16 | |
| *** amoralej has quit IRC | 13:16 | |
| *** lukebrowning has quit IRC | 13:16 | |
| *** esberglu has quit IRC | 13:17 | |
| *** slaweq has joined #openstack-infra | 13:17 | |
| *** lukebrowning has joined #openstack-infra | 13:18 | |
| clarkb | frickler: at least in the normal devstack neutron multinode job looks like devstack is not creating the glance cache dir on the subnode (/var/cache/glance) | 13:18 |
| *** LindaWang has joined #openstack-infra | 13:18 | |
| clarkb | frickler: my plan was to track normal mulyinode down first as its lesa complicated that grenade | 13:19 |
| clarkb | also I think ^ is a bug complwtely indepedent of thia transitio | 13:19 |
| *** slaweq has quit IRC | 13:19 | |
| *** slaweq has joined #openstack-infra | 13:20 | |
| *** rossella_s has quit IRC | 13:21 | |
| *** lukebrowning has quit IRC | 13:23 | |
| *** LindaWang has quit IRC | 13:24 | |
| *** rossella_s has joined #openstack-infra | 13:24 | |
| clarkb | sshnaidm|off: that package isnt listed on the index, I wonder how pip is finding it http://mirror.regionone.tripleo-test-cloud-rh1.openstack.org/pypi/simple/xstatic-jquery/ | 13:24 |
| *** slaweq has quit IRC | 13:24 | |
| *** lukebrowning has joined #openstack-infra | 13:24 | |
| *** caphrim007 has joined #openstack-infra | 13:26 | |
| *** wolverineav has joined #openstack-infra | 13:27 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add diskimage-builder/sahara requirements in updown jobs https://review.openstack.org/508396 | 13:28 |
| *** lukebrowning has quit IRC | 13:29 | |
| *** edmondsw has joined #openstack-infra | 13:29 | |
| *** rossella_s has quit IRC | 13:30 | |
| *** lukebrowning has joined #openstack-infra | 13:31 | |
| *** caphrim007 has quit IRC | 13:31 | |
| *** lukebrowning_ has joined #openstack-infra | 13:33 | |
| *** rossella_s has joined #openstack-infra | 13:33 | |
| *** edmondsw has quit IRC | 13:33 | |
| *** sree has quit IRC | 13:34 | |
| *** markvoelker has joined #openstack-infra | 13:35 | |
| *** lukebrowning has quit IRC | 13:35 | |
| *** sree has joined #openstack-infra | 13:37 | |
| *** lukebrowning_ has quit IRC | 13:38 | |
| *** yamamoto has quit IRC | 13:38 | |
| *** lukebrowning has joined #openstack-infra | 13:40 | |
| *** rossella_s has quit IRC | 13:41 | |
| *** LindaWang has joined #openstack-infra | 13:42 | |
| *** lukebrowning_ has joined #openstack-infra | 13:42 | |
| *** rossella_s has joined #openstack-infra | 13:43 | |
| *** lukebrowning has quit IRC | 13:44 | |
| *** anthonyper has quit IRC | 13:45 | |
| *** anthonyper has joined #openstack-infra | 13:45 | |
| *** lukebrowning has joined #openstack-infra | 13:46 | |
| *** lukebrowning_ has quit IRC | 13:47 | |
| *** lukebrowning has quit IRC | 13:50 | |
| *** lukebrowning has joined #openstack-infra | 13:51 | |
| *** lukebrowning_ has joined #openstack-infra | 13:54 | |
| *** lukebrowning has quit IRC | 13:56 | |
| frickler | clarkb: iiuc the /var/cache/ dirs are no longer needed with fernet keys, also glance might not be running on the subnode | 13:57 |
| *** rossella_s has quit IRC | 13:57 | |
| *** rossella_s has joined #openstack-infra | 13:58 | |
| *** lukebrowning has joined #openstack-infra | 13:58 | |
| *** lukebrowning_ has quit IRC | 13:59 | |
| *** ltomasbo has joined #openstack-infra | 14:00 | |
| frickler | clarkb: if I interpret the logs correctly, the results are in /home/zuul/workspace/api-ref instead /home/zuul/api-ref: http://logs.openstack.org/47/508647/2/check/legacy-api-ref/55f4403/job-output.txt.gz#_2017-09-30_10_39_50_631490 | 14:01 |
| openstackgerrit | Jeffrey Zhang proposed openstack-infra/openstack-zuul-jobs master: Add kolla-ansible dependency for kolla project https://review.openstack.org/508664 | 14:02 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add shade dependency to bifrost jobs https://review.openstack.org/508679 | 14:02 |
| *** rossella_s has quit IRC | 14:03 | |
| *** lukebrowning has quit IRC | 14:03 | |
| *** lukebrowning has joined #openstack-infra | 14:03 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Fix legacy-api-ref job https://review.openstack.org/508673 | 14:04 |
| fungi | cloudnull: the repos provided by zuul v3 are pushed onto the nodes, and so have no origin, correct. this is expected (though the side effects like pip not indicating git in the freeze output were unanticipated) | 14:04 |
| kfox1111 | whats the difference between nodeset legacy-centos and centos? | 14:05 |
| *** rossella_s has joined #openstack-infra | 14:05 | |
| *** thorst has quit IRC | 14:05 | |
| fungi | sshnaidm|off: the provider should be in an ansible variable now so you can access it that way if you need it for some reason; there's a nodepool subtree for each host in the inventory, including keys for cloud and region | 14:07 |
| fungi | sshnaidm|off: for example http://logs.openstack.org/18/508518/2/gate/build-openstack-sphinx-docs/b224292/zuul-info/inventory.yaml | 14:08 |
| openstackgerrit | Kevin Fox proposed openstack-infra/openstack-zuul-jobs master: Fix kolla-kubernetes nodesets https://review.openstack.org/508681 | 14:08 |
| *** lukebrowning has quit IRC | 14:08 | |
| *** esberglu has joined #openstack-infra | 14:08 | |
| *** markvoelker has quit IRC | 14:08 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Add warning about Zuul v2 examples https://review.openstack.org/508518 | 14:09 |
| *** lukebrowning has joined #openstack-infra | 14:10 | |
| fungi | now i'm seeing what jeblair meant about the swap size breaking cacti... we have no graphs for anything. i'll see if i can find a fix | 14:11 |
| *** esberglu has quit IRC | 14:12 | |
| *** lukebrowning has quit IRC | 14:15 | |
| *** rossella_s has quit IRC | 14:15 | |
| frickler | clarkb: also, I was focusing on the grenade job as it is the single check blocking devstack currently | 14:16 |
| fungi | frickler: oh, so the other multinode jobs for devstack are in good shape now? | 14:16 |
| *** lukebrowning has joined #openstack-infra | 14:16 | |
| frickler | fungi: no, but non-voting ;) | 14:16 |
| fungi | ahh | 14:16 |
| fungi | okay, so i've caught an infra-manual post job running and am streaming the console | 14:17 |
| fungi | waiting to see where/how it asplodes | 14:17 |
| mordred | fungi: I am doing the exact same thing | 14:18 |
| fungi | hah | 14:18 |
| mordred | fungi: maybe between the two of us we can figure out how to publish some html ;) | 14:19 |
| *** rossella_s has joined #openstack-infra | 14:19 | |
| fungi | i wouldn't count on it | 14:19 |
| mordred | me either | 14:19 |
| mordred | fungi: btw - I'm very much looking forward to landing and deploying tristanC's dashboard so that looking at historical things is easier | 14:19 |
| * fungi is not actually here, just procrastinating at mowing the lawn | 14:19 | |
| fungi | mordred: agreed! | 14:19 |
| * mordred is computering while doin gmorning coffee | 14:19 | |
| *** sree has quit IRC | 14:20 | |
| fungi | yeah, i'm still coffeeing as well, that's my excuse anyway | 14:20 |
| *** sree has joined #openstack-infra | 14:20 | |
| *** lukebrowning has quit IRC | 14:21 | |
| *** jascott1 has joined #openstack-infra | 14:21 | |
| *** lukebrowning has joined #openstack-infra | 14:22 | |
| fungi | it finished, and so far i'm not seeing what broke | 14:24 |
| fungi | also seems to have successfully logged to http://logs.openstack.org/6bc86e72b37c7147f6840f28c97d455bf7f3b2/fe6bc86e72b37c7147f6840f28c97d455bf7f3b2/post/publish-openstack-python-docs-infra/0944e91/ | 14:24 |
| fungi | that log url looks a bit hefty | 14:25 |
| *** rossella_s has quit IRC | 14:25 | |
| fungi | oh! looks like we're publishing post logs to uuid[2:]/uuid/ rather than uuid[:2]/uuid/ | 14:26 |
| *** jascott1 has quit IRC | 14:26 | |
| fungi | another place where one colon makes a difference | 14:26 |
| kfox1111 | https://review.openstack.org/#/c/508681/ is ready for reviews please. | 14:26 |
| fungi | pretty sure i approved the change which switched it to that from uuid[-2:]/uuid/ too and didn't catch it | 14:26 |
| fungi | finding now | 14:26 |
| *** lukebrowning has quit IRC | 14:26 | |
| kfox1111 | should be a quick one. | 14:27 |
| *** rossella_s has joined #openstack-infra | 14:27 | |
| fungi | indeed, 99% of the work is finding where that is | 14:28 |
| mordred | fungi: hah! indeed! | 14:28 |
| fungi | https://review.openstack.org/508346 | 14:28 |
| fungi | i'll follow up to that one | 14:28 |
| *** lukebrowning has joined #openstack-infra | 14:28 | |
| *** yamahata has joined #openstack-infra | 14:29 | |
| *** slaweq has joined #openstack-infra | 14:29 | |
| mordred | fungi: BWAHAHAHAHAHAHAHAHA | 14:30 |
| mordred | fungi: I have solved the puzzle of why it's "not publishing" | 14:30 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-jobs master: Yet still more fix post log location https://review.openstack.org/508684 | 14:31 |
| fungi | there we go | 14:31 |
| mordred | fungi: we're publishing to /afs/.openstack.org/docs/infra/infra-manual (because we made the infradocs published publish to docs/infra/{{ short_project_name }} | 14:31 |
| fungi | mordred: ooh, excellent find! | 14:31 |
| mordred | fungi: https://docs.openstack.org/infra/infra-manual/zuulv3 works just fine | 14:31 |
| *** lrossetti has joined #openstack-infra | 14:31 | |
| fungi | indeed it does | 14:32 |
| * mordred makes a special job for infra-manual real quick | 14:32 | |
| *** lukebrowning has quit IRC | 14:33 | |
| *** zhenguo has quit IRC | 14:34 | |
| *** ykarel has quit IRC | 14:34 | |
| *** lukebrowning has joined #openstack-infra | 14:35 | |
| *** ykarel has joined #openstack-infra | 14:35 | |
| *** slaweq has quit IRC | 14:39 | |
| *** sree has quit IRC | 14:39 | |
| *** yamamoto has joined #openstack-infra | 14:39 | |
| *** lukebrowning has quit IRC | 14:39 | |
| *** lukebrowning_ has joined #openstack-infra | 14:39 | |
| *** mat128 has joined #openstack-infra | 14:40 | |
| *** jtomasek has quit IRC | 14:40 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make publish-infra-manual job https://review.openstack.org/508685 | 14:40 |
| *** jtomasek has joined #openstack-infra | 14:41 | |
| fungi | The project template "publish-infra-manual" was not found. | 14:42 |
| fungi | i love zuul's parser reporting | 14:42 |
| *** lukebrowning_ has quit IRC | 14:44 | |
| *** yamamoto has quit IRC | 14:45 | |
| *** lukebrowning has joined #openstack-infra | 14:46 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Fix grenade multinode job https://review.openstack.org/508473 | 14:47 |
| *** bobh has joined #openstack-infra | 14:48 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Create openstack-sphinx named publish jobs https://review.openstack.org/508687 | 14:49 |
| *** lukebrowning_ has joined #openstack-infra | 14:49 | |
| *** lukebrowning has quit IRC | 14:50 | |
| *** mat128 has quit IRC | 14:50 | |
| clarkb | fungi: :2 failed a test too | 14:51 |
| *** jamesdenton has quit IRC | 14:51 | |
| *** lukebrowning has joined #openstack-infra | 14:51 | |
| fungi | nice | 14:51 |
| fungi | clarkb: looks like that's the pipeline precedence issue | 14:52 |
| *** lukebrowning_ has quit IRC | 14:53 | |
| fungi | we may need to squash the zuul unit test fix for that with the zuul pep8 fix to get things merging there again | 14:53 |
| *** lukebrowning_ has joined #openstack-infra | 14:55 | |
| fungi | mordred: what's the deal again with legacy vs non-legacy nodesets? | 14:55 |
| *** lukebrowning has quit IRC | 14:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Create openstack-sphinx named publish jobs https://review.openstack.org/508687 | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Consume publish-openstack-sphinx-docs https://review.openstack.org/508688 | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove old publish-openstack-python-docs jobs https://review.openstack.org/508689 | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Make publish-openstack-sphinx-docs project templates https://review.openstack.org/508690 | 14:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Remove old publish-openstack-python-docs templates https://review.openstack.org/508691 | 14:56 |
| fungi | 508681 seems to be mixing them in legacy jobs and i'm not sure if there might be a reason | 14:56 |
| mordred | there's a 5 node dance ot rename the publish docs jobs to match the build docs jobs | 14:56 |
| AJaeger_ | fungi, clarkb, mordred: Quick review request for https://review.openstack.org/#/c/508673/ , please- I'll fix some more of these afterwards (need single +2A) | 14:56 |
| mordred | fungi: legacy- has the node named "primary" - non-legacy names the main node after the node label | 14:56 |
| mordred | fungi: we made legacy versions of the multi-node nodesets just for ease of global search and replace | 14:57 |
| *** lukebrowning has joined #openstack-infra | 14:57 | |
| fungi | mordred: okay, so the legacy single-node sets just name the node to primary as well? | 14:58 |
| mordred | yah- because the devstack-gate jobs depended on that | 14:58 |
| fungi | i think kfox1111 was asking the same question earlier as well, presumably because of that change | 14:58 |
| mordred | one of the steps for folks to migrate their legacy jobs to non-legacy is to stop using the legacy nodesets | 14:58 |
| mordred | it's just not a thing we were in a position to do smartly at that point | 14:59 |
| *** lukebrow_ has joined #openstack-infra | 14:59 | |
| *** lukebrowning_ has quit IRC | 14:59 | |
| *** sree has joined #openstack-infra | 15:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Make publish-infra-manual job https://review.openstack.org/508685 | 15:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Create openstack-sphinx named publish jobs https://review.openstack.org/508687 | 15:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Consume publish-openstack-sphinx-docs https://review.openstack.org/508688 | 15:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Remove old publish-openstack-python-docs jobs https://review.openstack.org/508689 | 15:01 |
| mordred | fungi: so I think that patch looks fine | 15:01 |
| mordred | AJaeger_: https://review.openstack.org/#/c/508685/ should fix infra-manual publication | 15:01 |
| *** esberglu has joined #openstack-infra | 15:01 | |
| fungi | mordred: i think we need to squash your 508634 with 508643 (or force merge the latter) to unblock zuul changes | 15:02 |
| *** lukebrowning has quit IRC | 15:02 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add missing projects for infra puppet jobs https://review.openstack.org/508526 | 15:02 |
| fungi | my post log patch is also blocked on that | 15:02 |
| *** thorst has joined #openstack-infra | 15:02 | |
| AJaeger_ | mordred: thanks a lot! | 15:03 |
| AJaeger_ | questoin on 508681 - why is that using for nodeset sometimes legacy in name, sometimes not. That seems very unuasal. I see the comments above but don't get it... | 15:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle build_set being None for priority https://review.openstack.org/508634 | 15:04 |
| *** lukebrow_ has quit IRC | 15:04 | |
| mordred | fungi, clarkb: ^^ that should unbreak zuul which should unbreak zuul-jobs | 15:04 |
| *** sree has quit IRC | 15:04 | |
| mordred | fungi: oh - hah. didn't realize there was a second patch to squash :) | 15:04 |
| *** markvoelker has joined #openstack-infra | 15:05 | |
| mordred | fungi: I'm gonna depends-on your patch on the zuul patch real quick | 15:05 |
| *** lukebrowning has joined #openstack-infra | 15:05 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Yet still more fix post log location https://review.openstack.org/508684 | 15:06 |
| *** esberglu has quit IRC | 15:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Protect against builds dict changing while we iterate https://review.openstack.org/508629 | 15:06 |
| mordred | AJaeger_: I don't think there is a good reason | 15:07 |
| AJaeger_ | mordred: should we ask for consistency in that change? | 15:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/openstack-zuul-jobs master: Fix kolla-kubernetes nodesets https://review.openstack.org/508681 | 15:08 |
| mordred | AJaeger_, fungi: ^^ I just fixed it :) | 15:08 |
| AJaeger_ | thanks, mordred | 15:09 |
| mordred | AJaeger_: it's easier to work on these things on a saturday morning when the gate load is lower :) | 15:09 |
| *** lukebrowning has quit IRC | 15:10 | |
| *** thorst has quit IRC | 15:10 | |
| *** thorst has joined #openstack-infra | 15:10 | |
| AJaeger_ | definitely ;) We should block all others for next week - sent them on vacation ;) | 15:11 |
| AJaeger_ | I see that no new translations have been proposed the last two days, so thta one looks broken... | 15:11 |
| AJaeger_ | mordred, fungi, will either of you rm -rf the infra-manual directory on afs, please? Let's clean up... | 15:12 |
| *** lukebrowning has joined #openstack-infra | 15:12 | |
| fungi | AJaeger_: i should be able to do that, just a sec | 15:12 |
| fungi | done | 15:13 |
| fungi | i haven't had time yet to track down the image generation failure for cacti, but i've run out of opportunity to procrastinate on mowing the lawn. i'll try to take a look later if nobody beats me to it | 15:14 |
| *** lbragstad has joined #openstack-infra | 15:15 | |
| fungi | in the meantime, zuulv3 memory utilization is only about 1/3 ram, suggesting that whatever leak there may be is probably dependent on activity levels | 15:15 |
| *** lukebrowning has quit IRC | 15:16 | |
| *** kjackal_ has quit IRC | 15:17 | |
| *** edmondsw has joined #openstack-infra | 15:17 | |
| *** lukebrowning has joined #openstack-infra | 15:18 | |
| *** mriedem has joined #openstack-infra | 15:19 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Fix src location https://review.openstack.org/508695 | 15:20 |
| *** LindaWang has quit IRC | 15:20 | |
| AJaeger_ | this fixes - like https://review.openstack.org/#/c/508673/ - src locations, I reviewed all of them ^ | 15:20 |
| *** LindaWang has joined #openstack-infra | 15:21 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-tempest-dsvm-nnet https://review.openstack.org/508519 | 15:21 |
| *** edmondsw has quit IRC | 15:22 | |
| mordred | AJaeger_: cool. that reminds me - I need to finish work on the new releasenotes job | 15:22 |
| AJaeger_ | mordred, fungi, https://review.openstack.org/508473 looks sane to me, do you agree? | 15:23 |
| *** lukebrowning has quit IRC | 15:23 | |
| AJaeger_ | mordred: Ah, good. I noticed the duplicates and wondered already... | 15:23 |
| mordred | AJaeger_: also - we erroneously mapped both gate-{name}-releasenotes and {name}-releasenotes to legacy-{name}-releasenotes ... so we'll need to fix that one sooner rather than later | 15:23 |
| *** thorst has quit IRC | 15:23 | |
| AJaeger_ | any ideas why https://review.openstack.org/#/c/508563/ has RETRY_LIMIT errors | 15:23 |
| mordred | AJaeger_: yes on 508473 | 15:24 |
| *** lukebrowning has joined #openstack-infra | 15:24 | |
| mnaser | AJaeger_ log_file = '{log_dir}/{envlist}-siblings.txt'.format(envlist=envlist) | 15:24 |
| mnaser | KeyError: 'log_dir' | 15:24 |
| mordred | AJaeger_: looks like a bug in siblings: http://logs.openstack.org/63/508563/3/check/build-openstack-sphinx-docs/e2f6413/job-output.txt.gz#_2017-09-30_06_49_35_029142 | 15:24 |
| mnaser | in ansible_module_tox_install_sibling_packages.py | 15:24 |
| mordred | mnaser: oh - nice catch | 15:25 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix legacy-api-ref job https://review.openstack.org/508673 | 15:25 |
| AJaeger_ | mnaser: thanks | 15:26 |
| mordred | I feel like we pushed upa fix for thatalready | 15:26 |
| *** lukebrowning_ has joined #openstack-infra | 15:26 | |
| mnaser | i think thats a bug in the actual fix | 15:27 |
| mordred | oh! | 15:27 |
| mnaser | yes it is | 15:27 |
| mnaser | left a comment | 15:27 |
| mnaser | :p | 15:27 |
| mnaser | AJaeger_ ara is really good at catching things when you see a MODULE FAILURE only in my experience, it has much more info | 15:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make fetch-tox-output more resilient https://review.openstack.org/508563 | 15:28 |
| AJaeger_ | mnaser: ok | 15:28 |
| *** lukebrowning has quit IRC | 15:29 | |
| *** srobert has joined #openstack-infra | 15:31 | |
| *** lukebrowning_ has quit IRC | 15:31 | |
| *** srobert has quit IRC | 15:31 | |
| *** srobert has joined #openstack-infra | 15:32 | |
| mordred | AJaeger_, fungi, clarkb: these: https://review.openstack.org/#/q/status:open+topic:unbreak-zuulv3 are reasonably high-priority | 15:33 |
| *** lukebrowning has joined #openstack-infra | 15:33 | |
| mnaser | mordred what is the likeliness that we can get a trusted repo that we control (i.e. puppet-openstack-integration becomes a trusted repo or we create puppet-project-config) so we can define all of our projects in there rather than having a .zuul.yaml in every single project (though, i have some ideas to work around it with project-templates) .. but trying to think how we can avoid duplication | 15:34 |
| AJaeger_ | mordred: these looks fine | 15:35 |
| mnaser | it feels *slightly* confusing that puppet-nova will have puppet-module-jobs as a template and there is no obvious reference as to where that comes from | 15:35 |
| mordred | mnaser: that won't be possible in the short term - so you'll need to do the project-templates hack you came up with ... because at the moment a trusted repo gets jobs run in trusted context - which means those jobs can define content that bypasses some ofthe security restrictions we have on the executor | 15:35 |
| mnaser | mordred ok no worries, i guess we can add a puppet-openstack-projects.yaml and split things off to.. in order to clean it up | 15:36 |
| mordred | mnaser: we've discussed adding a third type of repo - that would allow what you're talking about without granting the ability to run jobs without the security restrictions | 15:36 |
| *** srobert has quit IRC | 15:36 | |
| mordred | mnaser: since your use case is a totally reasonable and valid one | 15:36 |
| *** lukebrowning_ has joined #openstack-infra | 15:36 | |
| mnaser | i think we'll just add .zuul.yaml files to every project referencing the template (though i am slightly bothered already because of the amount of CI churn that'll generate unfortunately :() -- but not so bad, just $time | 15:37 |
| *** lukebrowning has quit IRC | 15:37 | |
| mnaser | best time to do it is on the weekend with an empty gate, baha | 15:37 |
| *** markvoelker has quit IRC | 15:39 | |
| AJaeger_ | empty gates - but still takes 10+ minutes from approving a job to starting to run in gate which I consider long | 15:39 |
| *** lukebrowning has joined #openstack-infra | 15:40 | |
| mordred | AJaeger_: yah - that's related to the inefficient reconfiguration issue | 15:40 |
| mordred | AJaeger_: that's very high on jeblair's list of things to fix | 15:40 |
| mnaser | AJaeger_ yeah that is slightly frustrating but yep i'm hopeful that will get addressed | 15:41 |
| *** lukebrowning_ has quit IRC | 15:41 | |
| mnaser | i'd be concerned if we did a flawless zuulv3 cutover :-P | 15:41 |
| *** yamamoto has joined #openstack-infra | 15:41 | |
| *** lukebrowning_ has joined #openstack-infra | 15:41 | |
| AJaeger_ | yes, this is now running on scale for the first time... | 15:42 |
| *** lukebrowning has quit IRC | 15:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix typo in comment https://review.openstack.org/508697 | 15:44 |
| AJaeger_ | mordred: trivial typo on your stack - not worth to respin the patch for that ^ | 15:44 |
| mnaser | btw is the cirros image no longer cached in the images? | 15:45 |
| mnaser | or am i using the wrong nodeset | 15:45 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix grenade multinode job https://review.openstack.org/508473 | 15:46 |
| *** lukebrowning_ has quit IRC | 15:46 | |
| AJaeger_ | fungi, clarkb: What about https://review.openstack.org/#/c/508687/ and https://review.openstack.org/#/c/508685/ ? Getting those in will mordred with the rest of the stack - could you +2A, please? | 15:46 |
| mordred | mnaser: the nodesets shouldn't affect image content - so I'd expect thecirros image to be there | 15:46 |
| *** yamamoto has quit IRC | 15:47 | |
| mnaser | https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L131-L143 | 15:47 |
| mnaser | i guess we're looking for it in /home/zuul/cache/files | 15:47 |
| *** lukebrowning has joined #openstack-infra | 15:48 | |
| clarkb | mordred: zuul fix still failing unittests | 15:48 |
| mordred | GAH | 15:48 |
| clarkb | AJaeger_: looking | 15:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/gear master: Add a send lock to the base Connection class https://review.openstack.org/508698 | 15:48 |
| mordred | clarkb: that's actually a race-condition test | 15:49 |
| mordred | I'll recheck | 15:49 |
| *** srobert has joined #openstack-infra | 15:49 | |
| *** lukebrowning_ has joined #openstack-infra | 15:50 | |
| clarkb | ok | 15:50 |
| kfox1111 | https://review.openstack.org/#/c/508681/1 just needs a wf please. | 15:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Fix src location https://review.openstack.org/508695 | 15:51 |
| mordred | kfox1111: done | 15:52 |
| AJaeger_ | bbl | 15:52 |
| *** lukebrowning has quit IRC | 15:52 | |
| *** lukebrowning_ has quit IRC | 15:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Make fetch-tox-output more resilient https://review.openstack.org/508563 | 15:55 |
| mordred | clarkb, AJaeger_: ^^ added depends-on on the zuul patch that fixes the unittests | 15:55 |
| *** esberglu has joined #openstack-infra | 15:56 | |
| jeblair | i am bad at colons | 15:59 |
| *** esberglu has quit IRC | 16:00 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix kolla-kubernetes nodesets https://review.openstack.org/508681 | 16:00 |
| *** lukebrowning has joined #openstack-infra | 16:02 | |
| fungi | jeblair: i hesitate to think what term we might end up calling a review scoped to colons | 16:03 |
| fungi | hopefully the pattern ceases and we never have to find out | 16:04 |
| * fungi goes back to not procrastinating at yardwork | 16:04 | |
| clarkb | jeblair: https://review.openstack.org/#/c/508634/3 is an important one | 16:06 |
| jeblair | reboot system boot 3.13.0-132-gener Sat Sep 30 09:33 - 16:07 (06:34) | 16:07 |
| jeblair | fungi: ^ cacti | 16:07 |
| *** lukebrowning has quit IRC | 16:07 | |
| jeblair | i can not find the rrd files | 16:07 |
| jeblair | Sep 30 09:20:22 cacti01 snmpd[10472]: Connection from UDP/IPv6: [2001:4800:7818:103:be76:4eff:fe04:7ed0]:43875 | 16:08 |
| jeblair | Sep 30 09:33:01 cacti01 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="1108" x-info="http://www.rsyslog.com"] start | 16:08 |
| *** LindaWang has quit IRC | 16:08 | |
| jeblair | no indication why | 16:08 |
| clarkb | weird | 16:08 |
| clarkb | https://review.openstack.org/#/c/508695/2 updates things for docs publishing | 16:08 |
| *** lukebrowning has joined #openstack-infra | 16:09 | |
| *** srobert has quit IRC | 16:09 | |
| jeblair | i think we keep the rra on a volume but maybe it's not in fstab | 16:10 |
| clarkb | mordred: comment on https://review.openstack.org/#/c/508685/2 is that worth a respin? | 16:12 |
| *** superdan is now known as dansmith | 16:12 | |
| jeblair | okay, rra remounted | 16:12 |
| jeblair | graphs are back, though obviously missing a few hours of data | 16:13 |
| *** lukebrowning has quit IRC | 16:14 | |
| fungi | jeblair: thanks!!! i totally had tunnel vision thinking this was fallout from your mention of the larger swap value breaking collection | 16:14 |
| mordred | clarkb: let's just do a followup ... | 16:14 |
| clarkb | mordred: ok I'll approve | 16:14 |
| fungi | not realizing this was yet still more new and unexpected/unrelated broken | 16:14 |
| jeblair | fungi: me too... and then i spent a few minutes wondering how raising a max value could delete all the rrd files | 16:15 |
| fungi | heh | 16:15 |
| fungi | i totally didn't even spot that the rrd files were missing, so... :/ | 16:15 |
| jeblair | i clicked on the wrench icon and ran the command locally and got a file not found | 16:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Update publish-infra-manual description https://review.openstack.org/508701 | 16:15 |
| fungi | i was still pouring over unhelpful cacti/poller/rrd log files before i got pulled away | 16:16 |
| *** lukebrowning has joined #openstack-infra | 16:17 | |
| jeblair | clarkb: thanks; all reviewed | 16:17 |
| openstackgerrit | James E. Blair proposed openstack-infra/gear master: Add a send lock to the base Connection class https://review.openstack.org/508698 | 16:18 |
| clarkb | jeblair: mordred yesterday I noticed some weirdness with the git repos. There was an index lock file rsynced by devstack-gate. Are the git repos shared on the executor when they are copied? If copied using something like rsync that could be causing problems (because the index file would be copied over) | 16:19 |
| clarkb | er lock file not index file | 16:19 |
| *** lukebrowning_ has joined #openstack-infra | 16:20 | |
| jeblair | clarkb: hrm... nothing should be touching the repos when they're copied over, except the copying itself; was it a multinode job? i wonder if we're copying them in parallel, and that's an artifact of that? | 16:21 |
| clarkb | jeblair: yes it was a multnode job | 16:21 |
| jeblair | clarkb: however, we are no longer using rsync, we're using git pushes | 16:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Make publish-infra-manual job https://review.openstack.org/508685 | 16:21 |
| clarkb | ya the rsync I saw was on the devstack-gate side | 16:21 |
| clarkb | and it managed to copy the index lock file from the source | 16:21 |
| clarkb | (perhaps we need to update d-g to not use rsync?) | 16:22 |
| clarkb | mordred: https://review.openstack.org/#/c/508687/3 is confusing me, a new job is created then the post queue is updated to use a completely different job? | 16:22 |
| *** lukebrowning has quit IRC | 16:22 | |
| jeblair | clarkb: hrm, i would expect the git pushes to leave pristine files on the worker, so i don't know where the index lock would have come from... | 16:22 |
| clarkb | jeblair: ya me too | 16:22 |
| clarkb | jeblair: let me try and find the log again | 16:23 |
| jeblair | clarkb: was it fatal? | 16:23 |
| clarkb | http://logs.openstack.org/02/508302/3/check/legacy-tempest-dsvm-neutron-multinode-full/89fa828/logs/devstack-gate-setup-workspace-new.txt yes the job failed | 16:23 |
| clarkb | pretty sure as a result of ^ | 16:23 |
| *** lukebrowning has joined #openstack-infra | 16:24 | |
| *** thorst has joined #openstack-infra | 16:24 | |
| *** lukebrowning_ has quit IRC | 16:24 | |
| mordred | clarkb: the new job created is for the followup patches - 508687 also renames the infra-index job - which is what the post update is about | 16:25 |
| jeblair | clarkb: i'm stumped | 16:26 |
| clarkb | jeblair: it doesn't happen 100% of the time either implying it is a race of some sort, probably something to watch out for and investigate more if we can get more data | 16:27 |
| *** lukebrowning has quit IRC | 16:28 | |
| jeblair | ya | 16:28 |
| clarkb | mordred: jeblair also mriedem debugging newton things found http://logs.openstack.org/16/507816/1/check/legacy-novaclient-dsvm-functional/806ee46/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-09-30_15_50_47_806 I think d-g branch checkout logic is broken | 16:29 |
| clarkb | it falls bcak to master there but should've checked out newton | 16:30 |
| *** lukebrowning has joined #openstack-infra | 16:30 | |
| clarkb | looks like because we rely on origin in d-g /me looks to fix | 16:30 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix src location https://review.openstack.org/508695 | 16:32 |
| *** lukebrowning_ has joined #openstack-infra | 16:33 | |
| *** d0ugal has joined #openstack-infra | 16:35 | |
| *** lukebrowning has quit IRC | 16:35 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update publish-infra-manual description https://review.openstack.org/508701 | 16:35 |
| mnaser | hmm | 16:36 |
| *** markvoelker has joined #openstack-infra | 16:36 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Check for branches in zuulv3 git repos properly https://review.openstack.org/508702 | 16:36 |
| clarkb | mriedem: mordred jeblair ^ I think that is the fix for devstack gate checkout behavior | 16:37 |
| clarkb | mriedem: maybe you want to depends on ^ and see if that makes it happy? | 16:37 |
| mnaser | i removed the legacy jobs in this change - https://review.openstack.org/#/c/508388 - i did a dnm test with a depends-on in here - https://review.openstack.org/#/c/508696/ - but the zuul job queue shows it running both jobs including legacy ones.. any clues (i assumed in that DNM check, only new jobs would run and the legacy wouldn't) | 16:37 |
| *** lukebrowning_ has quit IRC | 16:38 | |
| clarkb | mnaser: because 8388 is in project-config which is a trusted repo its contents are not executed speculatively | 16:38 |
| mnaser | aaaah okay | 16:38 |
| clarkb | mnaser: so you'll get the msater contents until your change merges | 16:38 |
| mnaser | ill abandon that change to avoid using up resources then | 16:38 |
| mnaser | gotcha | 16:38 |
| mriedem | clarkb: will do, thanks | 16:39 |
| *** lukebrowning has joined #openstack-infra | 16:39 | |
| mnaser | also what are the odds that my job is progressing if the console is stuck | 16:40 |
| mnaser | (it seems to be a console issue though, not stuck in execution cause its blocked mid-message) | 16:40 |
| clarkb | mnaser: that I don't know, does refreshing browser page loda it up? or even using nc/finger directly? | 16:40 |
| mnaser | i dont know how to use nc/finger directly (point towards docs? :D) and i find refreshing many times usually to get it to work.. but this time it didnt regardless of the # of attempts :< | 16:41 |
| mnaser | oh well it just completed succesfully | 16:41 |
| mnaser | for one of them at least | 16:41 |
| *** lukebrowning_ has joined #openstack-infra | 16:41 | |
| mnaser | http://zuulv3.openstack.org/static/stream.html?uuid=d313401db03c44789cbb69f4b426e163&logfile=console.log | 16:41 |
| clarkb | mnaser: `finger $jobuuid@zuulv3.openstack.org` iirc | 16:41 |
| mnaser | this one is always stuck at the openstack-keystone line | 16:41 |
| fungi | clarkb: we have no multiplexer yet, so you have to know the executor | 16:42 |
| clarkb | ah | 16:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle build_set being None for priority https://review.openstack.org/508634 | 16:42 |
| mnaser | well, it completed. looks like ansible doesnt like a lot of output being emitted | 16:42 |
| fungi | well, multiplexer/router (i'm not sure which is the better term for the mechanism we're imaging) | 16:43 |
| mnaser | hopefully down the line ansible-izing things will prevent this | 16:43 |
| *** yamamoto has joined #openstack-infra | 16:43 | |
| fungi | mnaser: yeah, also there's an interpreter layer in between the json ansible is emitting to convey the stdout stream, and what's getting displayed. possible the problem is an interpreter fail there, or buffering or who knows | 16:44 |
| mnaser | its a feature to push users to rewrite their jobs in ansible | 16:44 |
| mnaser | :D | 16:44 |
| *** lukebrowning has quit IRC | 16:44 | |
| fungi | we include both the interpreted txt stream and the ansible emitted json in the job results | 16:44 |
| fungi | just in case | 16:44 |
| clarkb | the two changes that dep'd on 8634 have been reapproved to enqueue them in the gate | 16:44 |
| fungi | ptg bookended by hurricanes. plowing through three weeks of growth with a reel mower is _slow_ going | 16:45 |
| clarkb | I hvaen't had to mow in like a month | 16:46 |
| fungi | (hence my regular breaks to procrastinate^Wrest) | 16:46 |
| clarkb | but rain is back so will hva eto soon | 16:46 |
| *** lukebrowning_ has quit IRC | 16:46 | |
| mnaser | https://review.openstack.org/#/c/508388/ -- wouldn't mind a few +2s so that we can merge the removal of legacy jobs once the change for new jobs in-repo merges (it's in gate now) | 16:46 |
| fungi | even if it never rained here, the grass would just absorb it from the 100%rh air | 16:46 |
| fungi | my lawn gets watered by the fog | 16:47 |
| mnaser | tesla's talking about autopilot and all but the real automation we need is lawn mowers :( | 16:48 |
| fungi | mnaser: they make them. only a few hundred us$ | 16:48 |
| *** yamamoto has quit IRC | 16:48 | |
| clarkb | mnaser: you don't need mowers on mars or the moon though | 16:48 |
| fungi | but christine won't let me buy one | 16:48 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Limit concurrency in zuul-executor under load https://review.openstack.org/508649 | 16:48 |
| clarkb | gotta think about the long game if you are elon | 16:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Do not add implied branch matchers in project-templates https://review.openstack.org/508658 | 16:48 |
| *** lukebrowning has joined #openstack-infra | 16:48 | |
| fungi | plus, while tiresome, this is a free workout i guess ;) | 16:49 |
| mnaser | wow they do TIL | 16:49 |
| fungi | there were a couple in my old neighborhood before i moved to the beach | 16:49 |
| fungi | basically an oversized roomba with blades | 16:50 |
| *** lukebrowning_ has joined #openstack-infra | 16:51 | |
| *** jtomasek has quit IRC | 16:51 | |
| SpamapS | mow? | 16:51 |
| SpamapS | I don't think I should mow my front yard | 16:52 |
| fungi | mnaser: the robotic lawn mowers aren't really that exciting. you should check out the robotic pool cleaners and robotic gutter cleaners | 16:52 |
| SpamapS | the palm trees, rose bushes, and rosemary plants might take a while to mow | 16:52 |
| fungi | SpamapS: cactus doesn't take well to mowing? | 16:52 |
| * mnaser is sticking to the condo life for as long as possible | 16:52 | |
| fungi | yeah, that's the "lawn" i _want_ | 16:52 |
| SpamapS | I haven't xerascaped to that level yet. Still pretending we have water for shrubs. ;) | 16:52 |
| SpamapS | I do have a "lawn" of moss. | 16:53 |
| * jeblair sends SpamapS some water | 16:53 | |
| SpamapS | jeblair: MORE | 16:53 |
| *** lukebrowning has quit IRC | 16:53 | |
| SpamapS | MORE | 16:53 |
| fungi | jeblair: you actually are too! | 16:53 |
| SpamapS | but more | 16:53 |
| SpamapS | Actually we don't take water from the San Joaquin | 16:53 |
| fungi | the city of very dry angels | 16:53 |
| *** hemna_ has quit IRC | 16:54 | |
| SpamapS | Ours is mostly from the Owens valley. | 16:54 |
| *** bhavik1 has joined #openstack-infra | 16:54 | |
| SpamapS | The eastern side of the snow melt that jeblair and co were never going to get anyway. | 16:54 |
| *** eharney has quit IRC | 16:54 | |
| *** lukebrowning has joined #openstack-infra | 16:54 | |
| fungi | heh | 16:54 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Yet still more fix post log location https://review.openstack.org/508684 | 16:54 |
| SpamapS | It was just going to sit in the ground acquifers and make the Owens valley beautiful. | 16:54 |
| SpamapS | Muir shmuir. | 16:54 |
| jeblair | and habitable | 16:54 |
| fungi | given i live on a sandbar. i don't know why people here pretend it's the burbs and plant lawns and non-native shrubs/trees | 16:55 |
| jeblair | that whole thing is replaying in nevada right now; lv wants to drain the aquifers in the great basin | 16:55 |
| *** lukebrowning_ has quit IRC | 16:55 | |
| SpamapS | DWP owns 90% of all the land in the Owens valley. They can make any part you want to rent habitable. | 16:55 |
| fungi | SpamapS: and as soon as they don't like you any longer, desert for you! | 16:56 |
| *** bhavik1 has quit IRC | 16:56 | |
| *** lukebrowning_ has joined #openstack-infra | 16:56 | |
| *** bhavik1 has joined #openstack-infra | 16:56 | |
| jeblair | mordred, clarkb, fungi: https://review.openstack.org/508658 should fix the last known project-template related errors (the 'stuff not running on stable branch' bug) | 16:57 |
| jeblair | i have to go now | 16:57 |
| mordred | jeblair: woot! | 16:57 |
| SpamapS | fungi: everybody likes me! | 16:57 |
| mordred | anybody want to merge 508687 ? | 16:57 |
| fungi | thanks jeblair! that's a huge fix | 16:57 |
| *** bhavik1 has quit IRC | 16:58 | |
| * fungi thinks monday is looking brighter and brighter for zuul v3 | 16:58 | |
| mordred | jeblair: nice job condensing the explanation into something understandable in the docs | 16:58 |
| *** bhavik1 has joined #openstack-infra | 16:58 | |
| *** lukebrowning has quit IRC | 16:59 | |
| clarkb | I was hoping for another small change :) | 16:59 |
| fungi | clarkb: you should know better | 16:59 |
| fungi | the commit message is an essay unto itself. very helpful | 17:00 |
| *** bhavik1 has quit IRC | 17:00 | |
| * fungi is going to digest the commit message and doc bullet points for that one while mowing and waiting for tests to report back | 17:01 | |
| *** lukebrowning_ has quit IRC | 17:01 | |
| *** thorst has quit IRC | 17:05 | |
| *** ykarel_ has joined #openstack-infra | 17:05 | |
| *** edmondsw has joined #openstack-infra | 17:05 | |
| clarkb | dansmith: https://review.openstack.org/508658 fix for nnet problem | 17:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Update project creators guide with zuul v3 information https://review.openstack.org/508596 | 17:07 |
| clarkb | jeblair's patch failed pep8, probably worth someone pushing a fix while zuul is quiet | 17:07 |
| SpamapS | woot tests pass on load average governor | 17:07 |
| clarkb | I'll work on that now | 17:07 |
| SpamapS | I should write actual tests for IT | 17:07 |
| *** lukebrowning has joined #openstack-infra | 17:07 | |
| *** ykarel has quit IRC | 17:08 | |
| *** markvoelker has quit IRC | 17:09 | |
| AJaeger_ | mordred, I addressed clarkb's comments on 508596 so that we have some content to publish again ;) | 17:09 |
| AJaeger_ | reviews welcome | 17:09 |
| *** edmondsw has quit IRC | 17:10 | |
| *** lukebrowning has quit IRC | 17:12 | |
| clarkb | mriedem: looks like novaclient change still failing but now on libs from git check which I think ianw has a fix for in devstack | 17:12 |
| AJaeger_ | would be nice to have +2A on https://review.openstack.org/#/c/508687/ as well... | 17:12 |
| mordred | AJaeger_: lgtm! | 17:13 |
| *** lukebrowning has joined #openstack-infra | 17:14 | |
| mriedem | clarkb: this? https://review.openstack.org/#/q/topic:libs_from_git+status:open | 17:14 |
| mordred | clarkb: https://review.openstack.org/#/c/508344/ and https://review.openstack.org/#/c/508366/ | 17:14 |
| mordred | looks lke it's having issues with grenade multinode | 17:14 |
| AJaeger_ | mordred: we merged earlier a change for that | 17:15 |
| fungi | i think frickler was working on getting grenade multinode unblocked | 17:15 |
| fungi | yeah, maybe that's in now? | 17:15 |
| AJaeger_ | mordred: I mean https://review.openstack.org/508473 fixed one grenade issue | 17:16 |
| fungi | i recall reviewing it at least | 17:16 |
| AJaeger_ | merged in the last hour | 17:16 |
| * fungi heads back out to the battle of the grasslands | 17:16 | |
| clarkb | mriedem: ya I htink so | 17:16 |
| clarkb | grenade multinode might be fixed by my d-g change as it should make branches work properly | 17:17 |
| clarkb | mordred: https://review.openstack.org/508702 is hopefully the last piece to working multinode | 17:17 |
| AJaeger_ | I rechecked the two changes that mordred pointed out | 17:17 |
| clarkb | er multinode grenade | 17:17 |
| * AJaeger_ heads down for dinner | 17:18 | |
| *** thorst has joined #openstack-infra | 17:18 | |
| * SpamapS detaches as well | 17:18 | |
| *** lukebrowning_ has joined #openstack-infra | 17:18 | |
| *** lukebrowning has quit IRC | 17:18 | |
| clarkb | I'll have a new ps for jeblairs change in a moment, just running pep8 locally | 17:18 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul feature/zuulv3: Do not add implied branch matchers in project-templates https://review.openstack.org/508658 | 17:19 |
| clarkb | there | 17:19 |
| dmsimard | clarkb: spending some time on the ara firefox issue now. Dunno if I'm going to find it but I'm trying :D | 17:20 |
| dmsimard | (╯°□°）╯︵ ┻━┻ | 17:20 |
| dmsimard | javascript | 17:20 |
| clarkb | I have approved 8687 | 17:20 |
| clarkb | AJaeger_: mordred ^ | 17:20 |
| *** lukebrowning_ has quit IRC | 17:22 | |
| *** thorst has quit IRC | 17:22 | |
| *** lukebrowning has joined #openstack-infra | 17:23 | |
| dmsimard | clarkb: this is the particular function that doesn't work on firefox: https://github.com/openstack/ara/blob/master/ara/templates/report.html#L420-L423 | 17:27 |
| dmsimard | but there's no error or warnings in the developer console | 17:27 |
| clarkb | mnaser may know, he is secretly a web wizard | 17:28 |
| mnaser | hi | 17:28 |
| *** lukebrowning has quit IRC | 17:28 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create openstack-sphinx named publish jobs https://review.openstack.org/508687 | 17:28 |
| *** caphrim007 has joined #openstack-infra | 17:28 | |
| * mnaser installs firefox | 17:28 | |
| dmsimard | eh I'll just try random things like checking if updating jquery fixes it | 17:29 |
| mnaser | got a sample page that's failing? (i know zuulv3 that generates a ton :p) | 17:29 |
| dmsimard | mnaser: you know the permalink icons in ara modals ? they don't work in ff | 17:29 |
| *** lukebrowning has joined #openstack-infra | 17:29 | |
| dmsimard | I'm probably largely due for a patternfly/bootstrap/jquery update dance | 17:31 |
| dmsimard | maybe for 1.0 | 17:31 |
| mnaser | i dont think its a js issue | 17:31 |
| mnaser | i think its a rendering issue | 17:31 |
| mnaser | inspecting the code it looks like it actually contains the href="blah" | 17:31 |
| dmsimard | the bower.json for patternfly requests jquery ~ 2.1.4 but the xstatic jquery is 1.10.something | 17:32 |
| mnaser | i suspect a button tag cannot contain a link (or at least something along that line in firefox) | 17:32 |
| mnaser | "The HMTL syntax does not allow an a element within a button element, so it is not surprising that it does not work across browsers." | 17:32 |
| *** lukebrowning_ has joined #openstack-infra | 17:33 | |
| mnaser | i think you can drop the button and add the classes to the link directly | 17:33 |
| mnaser | one moment let me try it | 17:33 |
| *** caphrim007 has quit IRC | 17:33 | |
| dmsimard | I *think* the button part was important for inheriting the right patternfly classes but I forget | 17:33 |
| dmsimard | you're probably on to something though | 17:34 |
| mnaser | got it | 17:34 |
| mnaser | let me push something fo ryou | 17:34 |
| mnaser | damn | 17:34 |
| mnaser | git pull on my local copy of ara => 180 files changed, 7517 insertions(+), 3188 deletions(-) | 17:34 |
| mnaser | :P | 17:34 |
| dmsimard | that's master | 17:35 |
| dmsimard | you'd be surprised if you pulled feature/1.0 :p | 17:35 |
| *** lukebrowning has quit IRC | 17:35 | |
| dmsimard | oh I didn't check if ara gate was broken by v3 yet | 17:35 |
| kfox1111 | so... how about this one: | 17:35 |
| dmsimard | I guess we'll find out | 17:35 |
| kfox1111 | cat: /etc/nodepool/primary_node_private: No such file or directory | 17:35 |
| dmsimard | kfox1111: wrong nodeset for a job | 17:35 |
| kfox1111 | how does that work now? | 17:35 |
| dmsimard | kfox1111: likely a multinode job that has a single node nodeset | 17:35 |
| kfox1111 | I think that file always existed. | 17:35 |
| *** lukebrowning has joined #openstack-infra | 17:36 | |
| kfox1111 | now it only exists if its a multinode job? | 17:36 |
| dmsimard | not sure if it existed on single node jobs but I can tell you that right now I still have a v2 setup | 17:36 |
| *** lukebrowning_ has quit IRC | 17:37 | |
| dmsimard | yup, that exists for single node jobs | 17:37 |
| dmsimard | not sure if it does in v3 | 17:37 |
| kfox1111 | yeah. single node jobs are failing now. | 17:37 |
| dmsimard | kfox1111: you have something in your job that relies on that file ? | 17:37 |
| mordred | - name: Create nodepool primary file | 17:38 |
| mordred | copy: | 17:38 |
| mordred | dest: /etc/nodepool/primary_node_private | 17:38 |
| mordred | content: "{{ hostvars['primary']['nodepool']['private_ipv4'] }}" | 17:38 |
| mordred | when: hostvars['primary'] is defined | 17:38 |
| dmsimard | mordred: so only if there is a host named primary | 17:38 |
| mordred | so - the key for that to work is you need a 'primary' node defined - which is in the legacy- versions of the nodesets | 17:38 |
| mordred | even for the single-node jobs | 17:38 |
| dmsimard | kfox1111: what job is it ? | 17:38 |
| mnaser | clarkb dmsimard https://review.openstack.org/508704 :) | 17:39 |
| * dmsimard looks | 17:39 | |
| kfox1111 | dmsimard: yeah. the code setting up kubernetes was always setting up the primary node the same way if it was single or multi. | 17:39 |
| kfox1111 | mordred: so I can add that role to my jobs? | 17:39 |
| dmsimard | kfox1111: these files are no longer provisioned on native zuul v3, we are "manually" adding them on legacy jobs | 17:40 |
| *** lukebrowning has quit IRC | 17:40 | |
| dmsimard | kfox1111: as to not break backwards compatibility | 17:40 |
| dmsimard | what you're looking for is now in hostvars['nodepool']['private_ipv4'] | 17:40 |
| kfox1111 | we will need them for multinode jobs at minimum. are those considered legacy too? | 17:41 |
| kfox1111 | oh... so we need to pass those hostvars thorugh env vars or something? | 17:41 |
| dmsimard | kfox1111: legacy jobs have a legacy prefix :) | 17:41 |
| dmsimard | kfox1111: what job is it that yo have is failing ? | 17:41 |
| kfox1111 | all of them. sec... | 17:41 |
| kfox1111 | https://review.openstack.org/#/c/508682/ | 17:42 |
| *** jtomasek has joined #openstack-infra | 17:42 | |
| mordred | yah - you just need to switch from non-legacy nodesets to legacy nodesets for now | 17:43 |
| kfox1111 | hmm... ok. | 17:43 |
| dmsimard | mnaser: looks like it works now but there are other permalink icons | 17:43 |
| mordred | so - this: https://review.openstack.org/#/c/508681/ - just prefix all of them with legacy- ... sorry, I didn't figure you'd have a problem or I would have said something during review | 17:44 |
| dmsimard | mnaser: that's the one for playbooks, there's also a few for modals | 17:44 |
| kfox1111 | the single node lagacy nodeset will create the files too? | 17:44 |
| mnaser | dmsimard let me look for them | 17:44 |
| dmsimard | mnaser: like https://github.com/openstack/ara/blob/master/ara/templates/report.html#L310 | 17:44 |
| mordred | kfox1111: the single node legacy nodesets name their node 'primary' - so the logic in the pre playbook that is in the legacy base job will write out the file | 17:44 |
| *** yamamoto has joined #openstack-infra | 17:44 | |
| dmsimard | mnaser: just search for 'link-modal' | 17:45 |
| kfox1111 | k. | 17:45 |
| * mordred will update the docs with a discussion of this - turns out it's important as part of the transition | 17:45 | |
| *** Goneri has joined #openstack-infra | 17:45 | |
| dmsimard | mordred: didn't we fix non-legacy to legacy nodesets yesterday ? | 17:45 |
| dmsimard | I guess we missed a couple ? | 17:45 |
| *** AJaeger_ is now known as AJaeger | 17:46 | |
| kfox1111 | dmsimard: I updated my nodesets this morning as it never migrated the distro's used properly. | 17:46 |
| kfox1111 | I used the non legacy version though since I didn't know the difference. | 17:46 |
| mnaser | there, pushed a fix for the others too dmsimard :) | 17:47 |
| mordred | yah - which is fair - it's not documented :) | 17:47 |
| *** lukebrowning has joined #openstack-infra | 17:48 | |
| dmsimard | clarkb: mnaser fixed the issue, he's our hero | 17:48 |
| SamYaple | are we using async tasks anywhere in infra? i have 4 tasks that i want to run per item. those 4 tasks need to run serially. I want to async the group of tasks though, so i can run all the with_items in parallel | 17:48 |
| openstackgerrit | Kevin Fox proposed openstack-infra/openstack-zuul-jobs master: Fix kolla-kubernetes to use legacy types https://review.openstack.org/508705 | 17:49 |
| SamYaple | if there is an example of that i would love to see it | 17:49 |
| dmsimard | mnaser: I'll bring a batch of stickers to openstack canada :p | 17:49 |
| kfox1111 | ok. lets see if this will work. | 17:49 |
| *** kjackal_ has joined #openstack-infra | 17:49 | |
| dmsimard | SamYaple: async tasks are hard but I posted something about them a while back https://dmsimard.com/2017/05/10/limiting-the-amount-of-concurrent-asynchronous-tasks-in-ansible/ | 17:49 |
| dmsimard | I don't believe anything we have uses async right now | 17:49 |
| mordred | SamYaple: we USED to use async tasks for everything in zuul 2.5 so we could deal with timing things out | 17:50 |
| *** yamamoto has quit IRC | 17:50 | |
| mordred | SamYaple: and they're wonky enough that we stopped using them - even after fixing 4 or 5 bugs in them upstream | 17:50 |
| mordred | SamYaple: which is to say ... have fun :) | 17:51 |
| mnaser | :D | 17:51 |
| dmsimard | mordred: it took me like a day to come up with that blog post, fun was not had | 17:51 |
| SamYaple | well it isn't too bad, i can do singular tasks async, i was just trying to do a group of them async-like | 17:51 |
| dmsimard | harder than it needs to be | 17:51 |
| clarkb | mordred: mriedem frickler I think we are in a bad state for grenade multinode. My fix for branches failed on glance api but my fix for glance api in multi ide is likely to fail grenade due to branch derpage | 17:52 |
| SamYaple | dmsimard: did you copy the docs or did the docs copy you on that blog post? | 17:52 |
| SamYaple | its a good read | 17:52 |
| dmsimard | SamYaple: I submitted the docs :P | 17:52 |
| clarkb | so we may need to force merge one of those to get multinode grenade working | 17:52 |
| SamYaple | dmsimard: nice | 17:52 |
| *** lukebrowning has quit IRC | 17:52 | |
| dmsimard | SamYaple: https://github.com/ansible/ansible/pull/24457/commits | 17:52 |
| SamYaple | haha i believe you :P | 17:52 |
| mnaser | another magical case of nothing but -- END OF STREAM -- in a build http://zuulv3.openstack.org/static/stream.html?uuid=4ed130489df44817ac521a3d9465c98a&logfile=console.log | 17:53 |
| mnaser | tho i suspect it's actually running (i hope so) | 17:53 |
| clarkb | mnaser: I'm suspecting its related to multinode jobs | 17:54 |
| mordred | clarkb: I think force-merging the branch derpage one is the thing that makes the most sense to force | 17:54 |
| clarkb | as my multinide debug runs all had that | 17:54 |
| clarkb | mordred: ya | 17:54 |
| mnaser | puppet integration jobs are single node though (but i guess if log collection has a bug it might be related) | 17:54 |
| mordred | clarkb, mnaser: we have tested/watched multi-node log streaming before, so the streamer does support it - which is not to say that something about multi-node is borking it | 17:55 |
| SamYaple | i get "END OF STREAM" if i open the console.log to quickly, if i wait a minute or two before opening the log ive never gotten END OF STREAM" | 17:55 |
| dmsimard | if an infra-root has nothing else better to do, we still need to figure out what else is failing from https://review.openstack.org/#/c/504238/ but I need access to executor logs | 17:55 |
| mordred | yah- that's probably a thing we should look in to | 17:55 |
| mnaser | yeah but this is a job that's been running for close to an hour soooo | 17:55 |
| mnaser | last time it happened the job ended up completing | 17:55 |
| mordred | there's almost certainly a bug in there somewhere - we need to refactor zuul_stream (that's high on my TDL once the dust settles) | 17:56 |
| mnaser | but we're 1 job away from landing zuulv3 native jobs.. so i really hope it's working or we'll have to undergo the whole fun of the check/gate again | 17:56 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Delete IncludeRole object from result object for include_role tasks https://review.openstack.org/504238 | 17:57 |
| SamYaple | mnaser: let me rephrase, if i click on console log and get END OF STREAM, it never changes even if i refresh. if i wait a few minutes before the first time i click on it ive never seen END OF STREAM, but that has just been my experince | 17:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Update retire instructions for Zuul 3 https://review.openstack.org/508706 | 17:58 |
| mnaser | SamYaple that might be useful info for mordred's further debugging :> | 17:58 |
| SamYaple | well then mordred see above | 17:59 |
| AJaeger | infra-manual is still not published correctly. I do not see it in zuulv3.o.o's post queue, the change is merged but no logs exist at http://logs.openstack.org/fe/fe6bc86e72b37c7147f6840f28c97d455bf7f3b2 and content is not up. | 18:00 |
| mnaser | are all zuul executors run the "finger" console stuff? | 18:00 |
| mordred | AJaeger: there is a bug (that was fixed but maybe race condition) that cause the url to be wrong: | 18:00 |
| mordred | AJaeger: http://logs.openstack.org/6bc86e72b37c7147f6840f28c97d455bf7f3b2/fe6bc86e72b37c7147f6840f28c97d455bf7f3b2/ | 18:00 |
| mnaser | ze02.openstack.org and ze04.openstack.org are returning conenction refused | 18:00 |
| mnaser | so that could be the reason we are unable to get logs if jobs land there? | 18:01 |
| *** lukebrowning has joined #openstack-infra | 18:01 | |
| * mordred has to afk - will help look in to the finger issue upon return ... the finger thing should be started by the zuul_console task in the base pre-playbook fwiw | 18:01 | |
| AJaeger | mordred, mnaser I was confused - https://review.openstack.org/#/c/508596/ has not merged yet | 18:02 |
| AJaeger | I thought it had - the change I referenced indeed failed... | 18:02 |
| AJaeger | So, once 508596 is merged, I will check again | 18:02 |
| mnaser | i think it might have been something odd that happened where it never queued | 18:03 |
| fungi | AJaeger: 508684 is what should fix the post log locations, and only merged about an hour ago, so maybe hadn't reconfigured yet? | 18:03 |
| fungi | (as far as no logs) | 18:03 |
| *** peterlisak has quit IRC | 18:04 | |
| *** onovy has quit IRC | 18:04 | |
| mordred | fungi: it seems to have no jobs triggered :( | 18:05 |
| AJaeger | fungi: I looked at the wrong job ;( | 18:05 |
| AJaeger | fungi, want to +2A next change for mordred : https://review.openstack.org/#/c/508690/, please? | 18:05 |
| mordred | 2017-09-30 18:03:44,444 DEBUG zuul.IndependentPipelineManager: Project openstack-infra/infra-manual not in pipeline <Pipeline check> for change <Change 0x7f2a96b7f860 508706,1> | 18:06 |
| *** ykarel_ has quit IRC | 18:06 | |
| *** markvoelker has joined #openstack-infra | 18:06 | |
| mordred | did I break it earlier? | 18:06 |
| *** lukebrowning has quit IRC | 18:06 | |
| mordred | oh. HAHAHAHAHAHAHAH | 18:06 |
| mordred | yes | 18:06 |
| mordred | sorry | 18:06 |
| mordred | TOTALLY my fault | 18:06 |
| AJaeger | mordred: yeah, was just going to ask what's going on there... | 18:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add check and gate jobs for infra-manual https://review.openstack.org/508707 | 18:07 |
| *** lukebrowning has joined #openstack-infra | 18:07 | |
| AJaeger | oh... | 18:07 |
| mordred | sorry - when we made the new publish-infra-manual job - I removed the template, which left us with nothing in check and gate :) | 18:07 |
| AJaeger | ;) | 18:07 |
| mordred | ok. I now must AFK - back ina bit | 18:07 |
| *** mat128 has joined #openstack-infra | 18:10 | |
| *** ykarel_ has joined #openstack-infra | 18:10 | |
| *** GregHouse has quit IRC | 18:10 | |
| kfox1111 | https://review.openstack.org/#/c/508705/ should be ready | 18:10 |
| AJaeger | kfox1111: on it... | 18:10 |
| kfox1111 | AJaeger: thanks. | 18:11 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Do not add implied branch matchers in project-templates https://review.openstack.org/508658 | 18:11 |
| *** lukebrowning_ has joined #openstack-infra | 18:12 | |
| *** lukebrowning has quit IRC | 18:12 | |
| dmsimard | clarkb: can you confirm that permalinks are working for you in http://logs.openstack.org/04/508704/2/check/legacy-ara-integration-py35-latest-fedora-26/32f66eb/logs/build/reports/049ed69f-d899-42e5-b686-1d7c4de26bcb.html ? | 18:16 |
| *** lukebrowning_ has quit IRC | 18:16 | |
| clarkb | dmsimard: I've got weekend things now but will check after | 18:16 |
| *** thorst has joined #openstack-infra | 18:18 | |
| *** bobh has quit IRC | 18:19 | |
| *** slaweq has joined #openstack-infra | 18:20 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add legacy puppet check jobs https://review.openstack.org/508374 | 18:20 |
| ykarel_ | I too faced "END OF STREAM" on one of the job | 18:22 |
| *** ykarel_ is now known as ykarel | 18:22 | |
| *** GregHouse has joined #openstack-infra | 18:22 | |
| jlvillal | Any chance this patch can be +A? It has three +2, but was rebased: https://review.openstack.org/#/c/508629/ | 18:22 |
| *** lukebrowning has joined #openstack-infra | 18:24 | |
| ykarel | "END OF STREAM" is here http://zuulv3.openstack.org/static/stream.html?uuid=ebc3dbb462ec4f9a9ad2b1ef1ce03b8a&logfile=console.log | 18:24 |
| mnaser | sigh | 18:25 |
| ykarel | can i see other logs apart from console.log as well when job is running | 18:25 |
| mnaser | it timed out :( | 18:25 |
| mnaser | 40 jobs later, and we have to run another 40 again. | 18:25 |
| *** lukebrowning_ has joined #openstack-infra | 18:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add check and gate jobs for infra-manual https://review.openstack.org/508707 | 18:27 |
| *** lukebrowning has quit IRC | 18:28 | |
| *** lukebrowning has joined #openstack-infra | 18:29 | |
| *** lukebrowning_ has quit IRC | 18:30 | |
| *** lukebrowning_ has joined #openstack-infra | 18:32 | |
| *** kjackal_ has quit IRC | 18:33 | |
| *** lukebrowning has quit IRC | 18:33 | |
| *** slaweq has quit IRC | 18:35 | |
| *** lukebrowning has joined #openstack-infra | 18:35 | |
| *** slaweq has joined #openstack-infra | 18:36 | |
| *** slaweq has quit IRC | 18:36 | |
| *** slaweq has joined #openstack-infra | 18:36 | |
| *** slaweq has quit IRC | 18:36 | |
| *** lukebrowning_ has quit IRC | 18:37 | |
| *** esberglu has joined #openstack-infra | 18:37 | |
| *** lukebrowning_ has joined #openstack-infra | 18:37 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix kolla-kubernetes to use legacy types https://review.openstack.org/508705 | 18:37 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Make publish-openstack-sphinx-docs project templates https://review.openstack.org/508690 | 18:37 |
| *** markvoelker has quit IRC | 18:39 | |
| *** lukebrowning has quit IRC | 18:40 | |
| *** peterlisak has joined #openstack-infra | 18:40 | |
| *** lukebrowning has joined #openstack-infra | 18:41 | |
| *** onovy has joined #openstack-infra | 18:42 | |
| *** esberglu has quit IRC | 18:42 | |
| *** lukebrowning_ has quit IRC | 18:42 | |
| *** ihrachys has quit IRC | 18:42 | |
| *** lrossetti_ has joined #openstack-infra | 18:42 | |
| *** lrossetti has quit IRC | 18:45 | |
| *** lukebrowning has quit IRC | 18:45 | |
| *** yamamoto has joined #openstack-infra | 18:46 | |
| *** yamamoto has quit IRC | 18:50 | |
| *** lukebrowning has joined #openstack-infra | 18:51 | |
| AJaeger | Now we need a +2A on https://review.openstack.org/#/c/508688 - with these depends-on, it takes a bit of time to get mordred's stack in... | 18:51 |
| *** aviau has quit IRC | 18:51 | |
| *** aviau has joined #openstack-infra | 18:52 | |
| *** lukebrowning has quit IRC | 18:55 | |
| *** lukebrowning_ has joined #openstack-infra | 18:55 | |
| *** lukebrowning has joined #openstack-infra | 18:58 | |
| *** lukebrowning_ has quit IRC | 19:00 | |
| *** lukebrowning_ has joined #openstack-infra | 19:00 | |
| *** lukebrowning has quit IRC | 19:03 | |
| *** lukebrowning_ has quit IRC | 19:04 | |
| *** eroux has joined #openstack-infra | 19:05 | |
| *** bobh has joined #openstack-infra | 19:06 | |
| ykarel | Hi folks can some when check if i am going with Zuulv3 migration correctly in: https://review.openstack.org/#/c/508676/, i have one query as well in comments | 19:06 |
| *** dave-mccowan has joined #openstack-infra | 19:06 | |
| *** lukebrowning has joined #openstack-infra | 19:06 | |
| *** slaweq has joined #openstack-infra | 19:07 | |
| *** bobh has quit IRC | 19:10 | |
| *** yamahata has quit IRC | 19:10 | |
| *** lukebrowning has quit IRC | 19:10 | |
| *** iyamahat has joined #openstack-infra | 19:11 | |
| *** lukebrowning has joined #openstack-infra | 19:12 | |
| *** iyamahat has quit IRC | 19:13 | |
| *** iyamahat has joined #openstack-infra | 19:14 | |
| *** bobh has joined #openstack-infra | 19:14 | |
| *** dave-mccowan has quit IRC | 19:16 | |
| *** lukebrowning has quit IRC | 19:16 | |
| *** lukebrowning has joined #openstack-infra | 19:17 | |
| *** lukebrowning has quit IRC | 19:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use default releasenotes jobs https://review.openstack.org/508708 | 19:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete releasenotes jobs https://review.openstack.org/508709 | 19:22 |
| AJaeger | mordred: those two are just cleanups ^ | 19:22 |
| *** lukebrowning has joined #openstack-infra | 19:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Update retire instructions for Zuul 3 https://review.openstack.org/508706 | 19:24 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update project creators guide with zuul v3 information https://review.openstack.org/508596 | 19:25 |
| *** lukebrowning_ has joined #openstack-infra | 19:26 | |
| *** iyamahat_ has joined #openstack-infra | 19:27 | |
| *** iyamahat has quit IRC | 19:27 | |
| *** lukebrowning has quit IRC | 19:27 | |
| *** lukebrowning has joined #openstack-infra | 19:28 | |
| AJaeger | why is openstack/openstack showing up in the post queue on statusv3.o.o? | 19:29 |
| *** caphrim007 has joined #openstack-infra | 19:29 | |
| AJaeger | Also, I see the infra-manual change in the post queue - but without any post job available ;( | 19:29 |
| AJaeger | Ah, now it shows... | 19:30 |
| *** iyamahat__ has joined #openstack-infra | 19:30 | |
| *** lukebrowning_ has quit IRC | 19:31 | |
| *** esberglu has joined #openstack-infra | 19:32 | |
| AJaeger | and openstack/openstack had no jobs associaed, so was quickly removed from post again | 19:32 |
| *** slaweq has quit IRC | 19:32 | |
| *** iyamahat has joined #openstack-infra | 19:33 | |
| *** lukebrowning has quit IRC | 19:33 | |
| *** iyamahat_ has quit IRC | 19:33 | |
| *** iyamahat has quit IRC | 19:34 | |
| *** caphrim007 has quit IRC | 19:34 | |
| *** iyamahat has joined #openstack-infra | 19:34 | |
| ykarel | AJaeger, something similar i saw for magnum experimental: in experimental and experimental-tripleo, after few seconds experimental-tripleo was quickly removed | 19:35 |
| *** iyamahat__ has quit IRC | 19:35 | |
| clarkb | its because zuul now has to check if there are any speculative jobs to run | 19:36 |
| jeblair | dmsimard: permalinks in ff work for me on that change | 19:36 |
| *** esberglu has quit IRC | 19:36 | |
| *** markvoelker has joined #openstack-infra | 19:36 | |
| ykarel | clarkb, ack | 19:37 |
| *** lukebrowning has joined #openstack-infra | 19:37 | |
| ykarel | Hi can some when check if i am going with Zuulv3 migration correctly in: https://review.openstack.org/#/c/508676/, i have one query as well in comments | 19:38 |
| ykarel | clarkb, AJaeger ^^ | 19:38 |
| *** iyamahat has quit IRC | 19:39 | |
| *** lukebrowning has quit IRC | 19:42 | |
| clarkb | dmsimard: yup works for me now | 19:43 |
| clarkb | ykarel: I think you want to use just base probably rather than legacy base | 19:44 |
| clarkb | ykarel: the idea is to get away from legacy-* entirely over time | 19:44 |
| clarkb | ykarel: look in zuul-jobs | 19:44 |
| clarkb | or the non legacy jobs in openstack-zuul-jobs | 19:44 |
| jeblair | ykarel: and you just omit the 'parent: base' line since that's the default. | 19:45 |
| AJaeger | yeah, we have log files for post-job of infra-manual at http://logs.openstack.org/01/0185170542b0dcb09c3efab75101f1c5f9e1e989/ - but failed to obtain token for afs, see http://logs.openstack.org/01/0185170542b0dcb09c3efab75101f1c5f9e1e989/post/publish-infra-manual/88d771c/ara/ | 19:45 |
| *** lukebrowning has joined #openstack-infra | 19:45 | |
| AJaeger | end result: https://docs.openstack.org/infra/manual/creators.html still not updated | 19:45 |
| AJaeger | mordred, fungi, so, two steps further - still at least one missing ^ | 19:45 |
| jeblair | that may mean afs kernel module wasn't installed | 19:45 |
| ykarel | clarkb, i can't find similar job for legacy-dsvm-base | 19:46 |
| jeblair | AJaeger: indeed, ze03 has no afs | 19:46 |
| AJaeger | argh ;( | 19:47 |
| jeblair | 03 and 04 have no afs, others are okay | 19:47 |
| * AJaeger calls it a day now... | 19:47 | |
| *** lukebrowning_ has joined #openstack-infra | 19:47 | |
| AJaeger | would love to see a +2A on https://review.openstack.org/#/c/508688 to move those depends-ons forward - if anybody has some time later... | 19:48 |
| *** dave-mccowan has joined #openstack-infra | 19:48 | |
| jeblair | okay afs is running on 03 and 04 now (i just stopped/started the client). it may have been in install/reboot race from when we created the servers | 19:48 |
| *** edmondsw has joined #openstack-infra | 19:49 | |
| *** lukebrow_ has joined #openstack-infra | 19:49 | |
| *** lukebrowning has quit IRC | 19:49 | |
| clarkb | ykarel: looks like most non legacy jobs dep on 'base' which is the default parent and is in project-config | 19:52 |
| *** lukebrowning_ has quit IRC | 19:52 | |
| *** edmondsw has quit IRC | 19:53 | |
| *** lukebrow_ has quit IRC | 19:53 | |
| ykarel | clarkb, so what you suggest to use in place of legacy-dsvm-base, base? | 19:58 |
| *** lukebrowning has joined #openstack-infra | 19:59 | |
| clarkb | ykarel: yes | 20:00 |
| ykarel | clarkb, most of the projects are using legacy-dsvm-base(devstack-gate setup), if we use base, all projects have to bring the logic to in-tree, shouldn't common jobs reside in openstack-zuul-jobs? | 20:00 |
| clarkb | ykarel: it depends on what your job is trying to accomplish, devstack-gate has the beginnings of devstack-gate related logic jobs in that repo | 20:01 |
| clarkb | mordred: I think the other problem with multinode is we don't have the allow all ip traffic between all nodes iptables rules in place? | 20:03 |
| clarkb | dmsimard: ^ you had to add that for the zuulv3 native jobs right? is/was that added to the legacy jobs? | 20:04 |
| *** lukebrowning has quit IRC | 20:04 | |
| dmsimard | topic:zuulv3-multinode | 20:04 |
| dmsimard | Probably needs another rebase now... | 20:04 |
| clarkb | ya I see the change, but thats not for legacy jobs right? | 20:04 |
| *** lukebrowning has joined #openstack-infra | 20:05 | |
| dmsimard | There's devstack-gate, that's it ? | 20:05 |
| clarkb | I guess legacy jobs could include that role | 20:05 |
| clarkb | dmsimard: its all the legacy grenade multinode and neutron multinode and dvr multinode jobs | 20:05 |
| dmsimard | So that should be handled by legacy devstack gate roles | 20:06 |
| ykarel | clarkb, yes we rely on devstack-gate, i found devstack-legacy job here: https://github.com/openstack-infra/devstack-gate/commit/8551487bd61526ce68fb92030c840b57c77d0807, is this can be used | 20:06 |
| clarkb | dmsimard: yes, I just don't think it currently is because it was handled in nodepool ready script before (so it wasn't something that was auto ported) | 20:07 |
| clarkb | mordred: ^ you probably know best if that is true | 20:07 |
| clarkb | ykarel: the legacy config is all supposed to go away | 20:07 |
| clarkb | ykarel: I think if you are rewriting jobs it is best to not depend on the legacy stuff | 20:07 |
| *** dave-mcc_ has joined #openstack-infra | 20:07 | |
| dmsimard | Yeah sorry on my phone right now so hard to check | 20:08 |
| clarkb | ykarel: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/.zuul.yaml#n7 that might be better place to start | 20:08 |
| *** lukebrowning has quit IRC | 20:09 | |
| ykarel | clarkb, Ok will look at it, or will bring all stuff in magnum tree then, Thanks | 20:09 |
| *** dave-mccowan has quit IRC | 20:10 | |
| *** markvoelker has quit IRC | 20:10 | |
| *** jtomasek has quit IRC | 20:13 | |
| clarkb | dmsimard: did you want to address jlk's comments on https://review.openstack.org/#/c/504552/4/playbooks/multinode/pre.yaml and on the firewall change? | 20:14 |
| clarkb | I approved the firewall change as that one is mostly a noop comment but comment at ^ seems more significant | 20:14 |
| dmsimard | That particular comment is related to the TODO | 20:14 |
| dmsimard | Which is the zuul stream truncated json issue | 20:15 |
| dmsimard | Include role allows to group the conditionals at the playbook level rather than inside each role | 20:15 |
| *** lukebrowning has joined #openstack-infra | 20:16 | |
| clarkb | dmsimard: but we don't have any conditionals yet? | 20:17 |
| dmsimard | Due to the nature of includes, you need to set 'static: no' if you have conditionals that need to be evaluated before doing the include | 20:18 |
| dmsimard | And adding static: no triggers the bug | 20:18 |
| clarkb | but we don't have any conditions unless I'm missing something | 20:19 |
| dmsimard | Relevant links explaining the issue are in the todo | 20:19 |
| dmsimard | No, we took them out temporarily in order to be able to land them, hence the todo | 20:20 |
| clarkb | I guess to me it makes more sense to worry about them if/when those bugs ever get fixed and we update with a conditional | 20:20 |
| *** lukebrowning has quit IRC | 20:20 | |
| clarkb | but anyways its fine, I understand now | 20:21 |
| dmsimard | ¯\_(ツ)_/¯ | 20:21 |
| dmsimard | I don't have a strong opinion | 20:21 |
| dmsimard | I personally prefer include_role to the roles directive | 20:22 |
| *** hongbin has joined #openstack-infra | 20:22 | |
| *** larainema has quit IRC | 20:22 | |
| dmsimard | afk for a while | 20:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Consume publish-openstack-sphinx-docs https://review.openstack.org/508688 | 20:23 |
| *** sree has joined #openstack-infra | 20:26 | |
| *** esberglu has joined #openstack-infra | 20:26 | |
| *** lukebrowning has joined #openstack-infra | 20:28 | |
| *** sree has quit IRC | 20:30 | |
| *** esberglu has quit IRC | 20:30 | |
| *** dave-mcc_ has quit IRC | 20:32 | |
| *** lukebrowning has quit IRC | 20:33 | |
| *** lukebrowning has joined #openstack-infra | 20:34 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Set up firewall for legacy multinode jobs https://review.openstack.org/508710 | 20:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Check for branches in zuulv3 git repos properly https://review.openstack.org/508702 | 20:38 |
| *** lukebrowning has quit IRC | 20:38 | |
| clarkb | lets see if ^ does what I think it will do | 20:38 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Protect against builds dict changing while we iterate https://review.openstack.org/508629 | 20:39 |
| *** lukebrowning has joined #openstack-infra | 20:40 | |
| *** wolverineav has quit IRC | 20:42 | |
| clarkb | mordred: when you get back to a computer it would be great if you could look over ^ and dmsimard's multinode changes as we are piggy backing off of some of that in legacy here | 20:42 |
| *** lukebrowning_ has joined #openstack-infra | 20:43 | |
| mordred | clarkb: lgtm | 20:44 |
| *** lukebrowning has quit IRC | 20:44 | |
| mordred | clarkb: out of curiosity - any reason to not just change the git_has_branch function? | 20:45 |
| clarkb | mordred: because third party ci mostly | 20:45 |
| mordred | gotcha | 20:45 |
| clarkb | mordred: I didn't want to worry about backward compat for people using d-g with zuulv2 | 20:45 |
| mordred | and good point | 20:45 |
| *** yamamoto has joined #openstack-infra | 20:47 | |
| *** edmondsw has joined #openstack-infra | 20:47 | |
| clarkb | as an aside it is always fun to rediscover how much more complicated multinode testing is for all the reasons (and reaffirms my belief that start with 2 node jobs before N node jobs is the way to go) | 20:47 |
| *** lukebrowning_ has quit IRC | 20:47 | |
| clarkb | mordred: also is http://logs.openstack.org/53/504553/7/gate/tox-linters/710372a/ just derp ssh? | 20:48 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Use sidebar from openstackdocstheme 1.17 https://review.openstack.org/503091 | 20:50 |
| *** edmondsw has quit IRC | 20:51 | |
| *** jascott1 has joined #openstack-infra | 20:51 | |
| *** yamamoto has quit IRC | 20:52 | |
| *** lukebrowning has joined #openstack-infra | 20:52 | |
| clarkb | feature request, have the job name in the console log url | 20:53 |
| clarkb | even if it is as an ignored parameter on the backend | 20:53 |
| clarkb | maybe changeid too | 20:53 |
| clarkb | dmsimard: for when you return http://zuulv3.openstack.org/static/stream.html?uuid=2774519e506c44e98e4035ce6fc1578d&logfile=console.log the iptables there didn't do the public ip addr in both directions for both nodes | 20:56 |
| clarkb | it did the private ipv4 and public ipv6 for both nodes in both directions but not the public ipv4s at least not based on what was logged | 20:56 |
| clarkb | (I think that is fine for legacy jobs though) | 20:56 |
| *** lukebrowning has quit IRC | 20:56 | |
| clarkb | http://logs.openstack.org/02/508702/2/check/legacy-grenade-dsvm-neutron-multinode/2774519/ is the log dir | 20:57 |
| *** yamahata has joined #openstack-infra | 20:57 | |
| mordred | clarkb: yah - this is a thing that has happened a couple of times (the ssh derp) and we haven't been able to figure it out yet | 20:58 |
| clarkb | http://logs.openstack.org/10/508710/1/check/tox-linters/f007f47/job-output.txt.gz#_2017-09-30_20_43_05_124278 bug in linter job? its not finding the role in zuul-jobs (and not looking in that repo at all) | 20:59 |
| clarkb | but zuul itself seems fine with it (eg change didn't get thrown out by zuuls own parsing of it) | 20:59 |
| *** rtjure has quit IRC | 20:59 | |
| clarkb | hrm ssh failure happened in the change ahead of that one too | 21:00 |
| *** lukebrowning has joined #openstack-infra | 21:00 | |
| mordred | clarkb: it's a depends-on yeah? that's interesting - I don't see anyhting about those patches that would make non-speculative rules apply | 21:01 |
| SamYaple | re: git hook for dockerhub. I believe we landed on infra would evenetually have teh ability to mirror from git.o.o to an organization in my control the same way it mirrors to github. was that part waiting on zuulv3 | 21:02 |
| mordred | clarkb: http://logs.openstack.org/10/508710/1/check/tox-linters/f007f47/zuul-info/inventory.yaml shows the inheritance path is righ | 21:02 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results https://review.openstack.org/507980 | 21:02 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Throwaway patch to check subunit file processing https://review.openstack.org/508171 | 21:02 |
| mordred | SamYaple: yes it was - but I'd also like to fix the things in zuul as they are now that make you desire that git hook | 21:03 |
| mordred | SamYaple: like - the need for that indicates a systemic failure somewhere else, and while we can always do things to facililate forward progress, I wanna make sure we improve whatever needs to be improved so that such a thing is not needed | 21:04 |
| *** lukebrowning has quit IRC | 21:05 | |
| *** rtjure has joined #openstack-infra | 21:05 | |
| SamYaple | ah well thats going to be a hard mark to hit, if not impossible right. because we use internal mirrors and what not (as we should) but i would like to publish images built to dockerhub built in dockerhub rather than a seperate ci system | 21:05 |
| clarkb | mordred: ya and it definitely ran that role in the d-g change job too | 21:06 |
| clarkb | mordred: so zuul itself is working, I think just a bug in the linters job | 21:06 |
| mordred | SamYaple: oh - right - it was the signed packages thing | 21:06 |
| SamYaple | mordred: not just, we also (for loci) pass in a local git mirror so the container installs with zuul-cloner built repos for depends on changes | 21:07 |
| *** lukebrowning has joined #openstack-infra | 21:07 | |
| mordred | SamYaple: that's a thing we should fix in any case, so that the mirrors carry the upstream signatures and stuff | 21:07 |
| mordred | well - that should also change | 21:07 |
| mordred | because zuul-cloner is deprecated / going away | 21:08 |
| SamYaple | eh yea ive already switched to required-projects | 21:08 |
| SamYaple | but im talking about `git clone` in the container, we pass that path as a variable to the local git branch that required-projects dropped in | 21:08 |
| mordred | yah | 21:09 |
| SamYaple | point being, building in infra is different than building in dockerhub no matter how you slice, at least for loci | 21:09 |
| SamYaple | not a bad thing though | 21:09 |
| mordred | right. as it should be | 21:09 |
| mordred | what I want to get to is the place where we're able to publish the actual container built for the gate job - so that the thing you test in gate jobs is the actual binary artifact published to wherever it's published | 21:10 |
| mordred | we're still not quite there yet - and there are a few steps to go | 21:10 |
| mordred | but otherwise there is a hole in the testing chain | 21:10 |
| mordred | as you're testing a container built at one moment in time, landing the change and then publishing a container built in a completely different place | 21:10 |
| mordred | so the testing isn't testing the thing that people will consume | 21:10 |
| mordred | it's testing something similar - but not the actual thing | 21:10 |
| mordred | whether or not you *use* that or not is a different question ... | 21:11 |
| *** lukebrowning has quit IRC | 21:12 | |
| SamYaple | well thats just it, for the most part i dont want to publish the image we are currently building in the check and gate jobs. or rather i would like the check jobs to use the internal mirrors and the gate job to *not* use the internal mirrors and publish that way | 21:12 |
| SamYaple | that would work for me | 21:12 |
| mordred | but I want to get to the point where we can reasonably, consistently and correctly do that- and choices about whether or not a particular thing does so is a question of desire or policy rather than a technical question | 21:12 |
| mordred | why would you care about the internal mirrors if we update the mirrors to be legit upstream signed mirrors rather than rebuilt reprepro locally signed mirrors? | 21:12 |
| *** lukebrowning has joined #openstack-infra | 21:13 | |
| mordred | (curious) | 21:13 |
| mordred | I *totally* understand why you would want to not use the mirrors in their current signing state :) | 21:13 |
| SamYaple | mordred: because to enforce those mirrors means we would have to differ from what someone else would build | 21:13 |
| SamYaple | im trying to get the images we publish to match what *anyone* can make | 21:14 |
| clarkb | SamYaple: but it wouldb't be different if using upstream signed packages | 21:14 |
| mordred | it shouldn't be - it should be the exact same - the packages would be the same packages | 21:14 |
| clarkb | (signatures wouldn't line up othrewise) | 21:14 |
| mordred | yah. we're currently mirroring apt ... weirdly | 21:14 |
| SamYaple | again, and i know this is specific to loci, but its not just apt mirrors. signing would remove that sure, but then we have a git server on the host building the containers so we dont clone from git.o.o in the container and can take advantage of mirrors and caches | 21:15 |
| SamYaple | but i dont want to publish a binary artifact based on cached bits | 21:16 |
| SamYaple | specifically with git | 21:16 |
| clarkb | git also (via hashes) can guaruntee you arne't getting anything special | 21:16 |
| clarkb | so you use thecache, update to whatever is on git.o.o and go from there | 21:16 |
| *** lukebrowning_ has joined #openstack-infra | 21:17 | |
| *** lukebrowning has quit IRC | 21:17 | |
| mordred | SamYaple: in any case - there's some work coming up to change how we mirror things related to github - it won't be in the next couple of weeks or anything - but it's definitely on the radar | 21:17 |
| SamYaple | scenario: i build nova from a commit that they later tag 17.0.0. in the container if you were to inspect, you wouldn't see 17.0.0 installed, you would see whatever pbr got at the time | 21:18 |
| clarkb | SamYaple: right so you wouldn't only use the git cache (in fact we almost never only use the git cache0 | 21:18 |
| mordred | SamYaple: right. because that was the state of nova at the time- the same is true of the image if it was build on dockerhub | 21:18 |
| clarkb | the git cache is there to minimize the delta between HEAD and when we cached | 21:18 |
| mordred | SamYaple: that has nothing to do with caching -that has to do with the point in time when the container was built | 21:18 |
| clarkb | we always update to reflect current git | 21:18 |
| SamYaple | mordred: but i can retrigger the dockerhub build based on new tags (automatically) | 21:19 |
| mordred | SamYaple: you can retrigger builds in zuul as well | 21:19 |
| clarkb | as you can with zuul... | 21:19 |
| SamYaple | i can trigger loci to rebuild nova when nova tags something? | 21:19 |
| clarkb | s/loci/loci jobs/ yes | 21:19 |
| mordred | SamYaple: yah | 21:19 |
| SamYaple | oh. hm. i seem to have missed that bit. does nova need to be involved in this process? or is this something i can trigger entirely in loci repo? | 21:20 |
| mordred | SamYaple: it'll be a patch to project-config zuul.d/projects.yaml | 21:20 |
| mordred | SamYaple: because it's a cross-project thing - but it's straightforward and easy | 21:21 |
| *** lukebrowning_ has quit IRC | 21:21 | |
| clarkb | and we do similar for things like oslo library releases | 21:21 |
| SamYaple | i mean if you are telling me this is possible and no one has nay reason to block it, this was my major concern | 21:21 |
| clarkb | make oslo release that pings requirements which then pings all the projects | 21:21 |
| pabelanger | o/ | 21:22 |
| pabelanger | I have internets again! | 21:22 |
| SamYaple | if we get signed repos and the process you both just told me about then yes mordred and clarkb, infra will do exactly what i want | 21:22 |
| pabelanger | I haven't read backscroll yet, I plan to do that this evening | 21:22 |
| SamYaple | o/ pabelanger | 21:22 |
| pabelanger | but ready to get back to work for Monday | 21:22 |
| mordred | SamYaple: yah - in fact, I'm quite motivated to help do thethings to make sure you are able to do the things you need | 21:22 |
| mordred | pabelanger: welcome back! | 21:22 |
| clarkb | pabelanger: tldr jeblair fixed some fun bugs in zuul config loading. We will need a restart for that at some point. But not yet as we are gathering memory consumption data over the weekend because we OOM'd zuul on friday | 21:23 |
| clarkb | also fixes for all the things continue to happen | 21:23 |
| mordred | we're not spiking with the memory as much | 21:24 |
| mordred | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all | 21:24 |
| clarkb | mordred: are we swapping at all? | 21:25 |
| SamYaple | mordred: this information tilts the 1-job-per-project-all-distros vs 1-job-per-distro-all-projects heavily in the per-project jobs thing | 21:25 |
| mordred | doens't look like it | 21:25 |
| SamYaple | because then i can just trigger nova rebuild/publish job when nova tags | 21:25 |
| mordred | SamYaple: yah - from a conceptual point of view job-per-project is the thing that makes the most sense - I wold mostly see job-per-distro as a short-term workaround if resourcing was inapporpriate for job-per-project | 21:25 |
| SamYaple | not everyone | 21:25 |
| *** lukebrowning has joined #openstack-infra | 21:26 | |
| SamYaple | right | 21:26 |
| mordred | SamYaple: exactly - it lets you line up build events with source code events | 21:26 |
| SamYaple | luckily loci is generally low-entropy. so even if every commit to loci (that wasn't docs) built everything, that wouldn't happen often | 21:26 |
| SamYaple | thats exciting | 21:26 |
| clarkb | pabelanger: dmsimard I expect that the multinode tripleo jobs will need the multinode firwall fix like what I've got for devstack jobs too, fyi. But maybe lets prove it out with regular dsvm jobs first | 21:27 |
| clarkb | since there is at least one problem with the linters job | 21:27 |
| clarkb | mordred: oh! you know what? we removed the implied requires repos from depends on didn't we? | 21:28 |
| mordred | SamYaple: \o/ I'm excited | 21:28 |
| clarkb | mordred: I bet that is why the linters job fails | 21:28 |
| SamYaple | nice. well with the secret management, i can go ahead and start publishing to dockerhub (even if it is with the unsigned repos) | 21:28 |
| *** chem has quit IRC | 21:28 | |
| clarkb | (maybe that is a reason to have it again?) | 21:28 |
| mordred | clarkb: wait - what do you mean? | 21:28 |
| clarkb | mordred: didn't jeblair make a patch to remove depends-on projects from required-projects list implicitly? | 21:28 |
| clarkb | eg you have to explicitly set up that relationship? I may be misremembering though | 21:28 |
| mordred | clarkb: no - we only did thatfor tox-siblings ... so that it's only attempting to install projects in required-projects if they're there | 21:28 |
| clarkb | aha | 21:29 |
| clarkb | and tox-linters wouldn't use tox-siblings? I imagine it does? | 21:29 |
| *** yamamoto has joined #openstack-infra | 21:29 | |
| mordred | not to get zuul ansible roles | 21:29 |
| clarkb | in any case something to debug once I prove the fixes work | 21:29 |
| mordred | those are set up on the executor | 21:29 |
| clarkb | gotcha | 21:29 |
| mordred | what *might* be happening here ... | 21:29 |
| mordred | is that the only place in which roles: is listing openstack-infra/zuul-jobs is transitively via the base job (there's nothing else in the chain using it) | 21:30 |
| *** lukebrowning has quit IRC | 21:30 | |
| mordred | so when you get to the legacy-base job, it doesn't bring in a speculative copyof zuul-jobs- only the non-speculative | 21:30 |
| mordred | try adding a roles: openstack-infra/zuul-jobs to legacy-base and see if it picks up the depends-on version of zuul-jobs' roles for that depennds-on | 21:31 |
| clarkb | will do (but not yet because new patchset will kick all my tests out of queues) | 21:31 |
| mordred | ++ | 21:32 |
| *** yamahata has quit IRC | 21:34 | |
| *** lukebrowning has joined #openstack-infra | 21:37 | |
| *** lukebrowning_ has joined #openstack-infra | 21:40 | |
| * SpamapS has also struggled a bit with the inheritance path for some bits | 21:41 | |
| *** lukebrowning has quit IRC | 21:41 | |
| *** yamamoto has quit IRC | 21:41 | |
| SpamapS | but I'm guilty of not paying super close attention to the docs :-P | 21:41 |
| clarkb | in this case it is all working except for the linting job | 21:42 |
| clarkb | so in that regard its pretty nifty | 21:42 |
| SpamapS | Ideally most end users don't need to think too much about it. | 21:42 |
| SpamapS | parent to something useful, add in your roles... enjoy | 21:42 |
| clarkb | mordred: how possible would it be to silence the ara file copies? | 21:44 |
| clarkb | mordred: its a ton of vertical space in devstack jobs | 21:44 |
| *** lukebrowning_ has quit IRC | 21:45 | |
| SpamapS | clarkb: no_log: true in the task | 21:45 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Multi-node: Set up hosts file https://review.openstack.org/504552 | 21:45 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Multi-node: Set up firewalls https://review.openstack.org/504553 | 21:45 |
| clarkb | SpamapS: ya Iguess the question is whether or not we worry about needing those logs | 21:45 |
| clarkb | I think for general log file copies we want them | 21:45 |
| clarkb | but ara is a ton of small files that are mostly noise unless debugging ara | 21:46 |
| *** lukebrowning has joined #openstack-infra | 21:46 | |
| SpamapS | agreed | 21:46 |
| mordred | clarkb: very - I think we left them in for debugging - but I agree - it's noisy | 21:46 |
| SpamapS | you could log that listing into a separate file | 21:46 |
| mordred | clarkb: oh - wait - are you talking zuul's ara or devstack-gate's ara | 21:48 |
| SpamapS | I've been wondering about ways to influence the job running .. one interesting thing might be some way to set something liuke "ignore_no_log" to get more output when debugging | 21:48 |
| mordred | clarkb: because those are not rationalized into a single thing in legacy jobs | 21:48 |
| clarkb | mordred: both | 21:48 |
| clarkb | its not an urgent problem just lots of scrolling debugging the multinode failures | 21:49 |
| mordred | SpamapS: so the thing I've got on my plate is to make a thing that looks like job-output.txt but is actually javascript rendered in the broswer from job-output.json | 21:49 |
| mordred | SpamapS: so that you can show or hide as much or as little data as you want | 21:49 |
| clarkb | also ^F error works great with ansible output | 21:49 |
| clarkb | this has helped with the scrolling | 21:49 |
| mordred | as well as having pre and post tasks collapsed by default unless they have errors | 21:49 |
| mordred | clarkb: ++ | 21:49 |
| fungi | mordred: fold per task? that would be nifty... and then auto-fold the pre and post tasks maybe? | 21:50 |
| fungi | heh | 21:50 |
| mordred | fungi: yup. that too | 21:50 |
| fungi | yeah, tyhat | 21:50 |
| *** lukebrowning has quit IRC | 21:50 | |
| mordred | fungi: so - fold per phase - pre, run, post - inside that fold per job in the inheritance hierarchy, then fold per task | 21:50 |
| fungi | sgtm | 21:50 |
| mordred | it's about three down on the TODO list for post-dust settling - but it's a thing I want sooner rather than later - because *oy* | 21:52 |
| *** bobh has quit IRC | 21:54 | |
| clarkb | also I miss a clear success/faill message in the output | 21:55 |
| clarkb | and I think ^ may confuse logstash/e-r possibly | 21:55 |
| *** bobh has joined #openstack-infra | 21:55 | |
| *** lukebrowning has joined #openstack-infra | 21:57 | |
| clarkb | mordred: also the grenade job in one of my changes hit the index lock error again. We should consider using git rather than rsync | 21:58 |
| *** ijw has joined #openstack-infra | 21:58 | |
| mordred | clarkb: well - I think we should figure out why that's happening - because the thing that's being rsynced should be specific to the job in question | 22:00 |
| *** bobh has quit IRC | 22:00 | |
| mordred | clarkb: and there shouldn't be any git operations happening once we get to job content that's doing the rsync | 22:00 |
| *** yamamoto has joined #openstack-infra | 22:00 | |
| *** ykarel has quit IRC | 22:01 | |
| *** bobh has joined #openstack-infra | 22:01 | |
| mordred | clarkb: so - you might be right - but I'd lke to see if we can figure out why it's happening, bcause my (possibly incorrect) understanding of the world tells me it's a thing thatshouldn't be able to happen - and thus may point to a larger bug | 22:01 |
| *** lukebrowning has quit IRC | 22:01 | |
| clarkb | ya I wonder if it is related to something like what jeblair initially mentioned about multinode | 22:01 |
| clarkb | but really have no idea at this point | 22:02 |
| *** yamamoto has quit IRC | 22:02 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate master: Basic processing of test results https://review.openstack.org/507980 | 22:04 |
| *** lukebrowning has joined #openstack-infra | 22:06 | |
| clarkb | http://logs.openstack.org/03/508703/2/check/legacy-grenade-dsvm-neutron-multinode/9147b69/job-output.txt.gz is super close to passing | 22:06 |
| clarkb | grenade did its whole thing successfully from what I can tell | 22:06 |
| *** markvoelker has joined #openstack-infra | 22:07 | |
| clarkb | looks like its running devstack-gate on the subnode too? | 22:07 |
| clarkb | mordred: wsn't there a change to stop running multinode d-g against all but to run only on primary? | 22:09 |
| clarkb | https://review.openstack.org/#/c/508473/ it was incomplete | 22:09 |
| * clarkb makes another patch | 22:09 | |
| *** lukebrowning has quit IRC | 22:10 | |
| mordred | clarkb: yah. that was probably something we could have predicted | 22:11 |
| mordred | clarkb: honestly - we could likely just do a global search and replace in playbooks/legacy and set hosts: primary on all of them ... | 22:11 |
| mordred | clarkb: since there are no v2 jobs that can actually deal with multi-node v3 style | 22:11 |
| clarkb | ya | 22:12 |
| clarkb | my one concern with that is if we merged something in that needs to run everywhere as part of migration | 22:12 |
| clarkb | let me start just with the mulitnode jobs that don't work | 22:12 |
| * clarkb is writing a patch for that now | 22:12 | |
| mordred | clarkb: ++ | 22:12 |
| mordred | clarkb: in general the playbooks in playbooks/legacy should be safe - we shouldn't have put any content that needs to do things on all nodes anywhere else | 22:13 |
| clarkb | but I think this may be the last change needed to make it work | 22:13 |
| mordred | clarkb: well - we'd want to exclude playbooks/legacy/pre.yaml :) | 22:13 |
| mordred | clarkb: \o/ | 22:13 |
| *** esberglu has joined #openstack-infra | 22:14 | |
| *** lukebrowning has joined #openstack-infra | 22:17 | |
| *** esberglu has quit IRC | 22:19 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Set up firewall for legacy multinode jobs https://review.openstack.org/508710 | 22:21 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Don't run d-g against all hosts in multinode jobs https://review.openstack.org/508711 | 22:21 |
| clarkb | ok that includes the roles: fix and the primary node scoping | 22:21 |
| clarkb | will recheck changes as soon as they kick out | 22:21 |
| *** lukebrowning has quit IRC | 22:22 | |
| *** lukebrowning has joined #openstack-infra | 22:24 | |
| *** lukebrowning has quit IRC | 22:28 | |
| *** lukebrowning has joined #openstack-infra | 22:30 | |
| *** lukebrowning has quit IRC | 22:34 | |
| clarkb | mordred: the roles fix fixed linters | 22:35 |
| *** lukebrowning has joined #openstack-infra | 22:36 | |
| *** jamesdenton has joined #openstack-infra | 22:38 | |
| *** markvoelker has quit IRC | 22:40 | |
| clarkb | so for tripleo jobs likely they need these same fixes | 22:40 |
| *** lukebrowning has quit IRC | 22:40 | |
| *** lukebrowning has joined #openstack-infra | 22:42 | |
| *** lukebrowning has quit IRC | 22:47 | |
| clarkb | oh you know what? I'm wondering if the g-api fail snuck in as part of the primary vs all node selection | 22:48 |
| clarkb | will be intreesting to see if my devstack chnage is needed at all | 22:48 |
| *** lukebrowning has joined #openstack-infra | 22:49 | |
| *** lukebrowning has quit IRC | 22:53 | |
| *** lukebrowning has joined #openstack-infra | 22:55 | |
| *** lukebrowning_ has joined #openstack-infra | 22:58 | |
| *** lukebrowning has quit IRC | 22:59 | |
| *** lukebrowning has joined #openstack-infra | 23:00 | |
| *** lukebrowning_ has quit IRC | 23:02 | |
| *** eroux has quit IRC | 23:03 | |
| *** yamamoto has joined #openstack-infra | 23:03 | |
| *** lukebrowning has quit IRC | 23:05 | |
| *** yamamoto has quit IRC | 23:08 | |
| *** tosky has quit IRC | 23:08 | |
| *** esberglu has joined #openstack-infra | 23:08 | |
| *** lukebrowning has joined #openstack-infra | 23:09 | |
| *** esberglu has quit IRC | 23:13 | |
| *** lukebrowning has quit IRC | 23:13 | |
| *** lukebrowning has joined #openstack-infra | 23:15 | |
| *** mriedem has quit IRC | 23:16 | |
| *** hongbin has quit IRC | 23:18 | |
| *** lukebrowning_ has joined #openstack-infra | 23:18 | |
| *** lukebrowning has quit IRC | 23:20 | |
| *** ijw has quit IRC | 23:21 | |
| *** lukebrowning_ has quit IRC | 23:22 | |
| mordred | clarkb: oh - yah - good point | 23:23 |
| *** lukebrowning has joined #openstack-infra | 23:24 | |
| *** thorst has quit IRC | 23:25 | |
| *** thorst has joined #openstack-infra | 23:25 | |
| *** thorst has quit IRC | 23:26 | |
| *** lukebrowning has quit IRC | 23:29 | |
| *** lukebrowning has joined #openstack-infra | 23:31 | |
| *** caphrim007 has joined #openstack-infra | 23:31 | |
| *** bobh has quit IRC | 23:32 | |
| *** bobh has joined #openstack-infra | 23:33 | |
| clarkb | SUCCESS! | 23:34 |
| *** lukebrowning_ has joined #openstack-infra | 23:34 | |
| clarkb | SHIP IT | 23:34 |
| *** lukebrowning has quit IRC | 23:35 | |
| *** caphrim007 has quit IRC | 23:36 | |
| *** markvoelker has joined #openstack-infra | 23:37 | |
| *** bobh has quit IRC | 23:38 | |
| *** lukebrowning_ has quit IRC | 23:38 | |
| *** lukebrowning has joined #openstack-infra | 23:40 | |
| clarkb | infra-root please review topic:fix-multinode this should fix a whole host of multinode related problems and get them working again | 23:41 |
| clarkb | with those fixed then ianw's fixes for libs from git can get in and from there all the things | 23:42 |
| mordred | clarkb: \o/ | 23:44 |
| clarkb | at the very least it fixes the grenade multinode job which is the voting one | 23:44 |
| clarkb | if there are other problems we can continue to iterate | 23:44 |
| *** lukebrowning has quit IRC | 23:45 | |
| mordred | clarkb: all +2 from me | 23:45 |
| mordred | clarkb: and yah - as long as grenade multinode is working, and we got rid of the weird nnet job for nova - I think that's the bulk of it - other than the memory thing | 23:45 |
| *** lukebrowning has joined #openstack-infra | 23:45 | |
| clarkb | and weird nnet job should be fixed by jeblairs configuration fix in zuul | 23:46 |
| mordred | clarkb: we seem to have levelled off with memory too | 23:46 |
| mordred | clarkb: although obviously we don't have as much activity | 23:46 |
| clarkb | ya though if you look at it it makes some large jumps | 23:46 |
| *** thorst has joined #openstack-infra | 23:46 | |
| clarkb | wondering if there is some behavior that creates a leak | 23:46 |
| mordred | but it doesn't even look like we're creeping - I wonder if it's a total concurrency thing | 23:46 |
| clarkb | ya could be | 23:47 |
| mordred | like - it expands memory as more things are going at once | 23:47 |
| mordred | and doens't contract - because python | 23:47 |
| *** lukebrowning_ has joined #openstack-infra | 23:48 | |
| *** lukebrowning has quit IRC | 23:50 | |
| *** thorst has quit IRC | 23:51 | |
| *** lukebrowning_ has quit IRC | 23:52 | |
| SamYaple | making alot of use of atexit tends to do that, unless you also litter in copious amounts of 'del's | 23:52 |
| clarkb | its also just python because of how lists and dicts and everythin work | 23:54 |
| clarkb | they all expand but don't contract (even if you del) | 23:54 |
| SamYaple | del'ing an element in a dict doesn't free the memory? | 23:55 |
| clarkb | it does for the element itself but not hte dict overhead | 23:55 |
| *** lukebrowning has joined #openstack-infra | 23:55 | |
| clarkb | (assuming that was the lsat ref to the element) | 23:55 |
| clarkb | python data structures don't shrink | 23:55 |
| *** nunchuck has joined #openstack-infra | 23:56 | |
| SamYaple | oh i see. | 23:56 |
| *** deduped has joined #openstack-infra | 23:56 | |
| SamYaple | ill have to test that to get a better idea of how it works, i hadnt considered the dict structure itself | 23:56 |
| clarkb | lists grow in powers of 2 iirc | 23:56 |
| *** LindaWang has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!