| dmsimard | the network overlay config patch is passing, so yay ? https://review.openstack.org/#/c/435933/ | 00:06 |
|---|---|---|
| dmsimard | I haven't done any other change other than rebasing and addressing comments, I'd get involved in the playbook/role syntax but meh | 00:07 |
| dmsimard | I don't like the mixed style of yaml/ansible where sometimes things are proper yaml dicts and sometimes "=" are used across the same set of things | 00:07 |
| fungi | yamsible | 01:05 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul-jobs master: Role to copy the build ssh key to other users https://review.openstack.org/496413 | 01:23 |
| dmsimard | fungi: hah, yamsible. That should be a thing. | 01:50 |
| clarkb | dmsimard: I think the dvr job which failed is the one that exercises that code the most but I also think its expected to fail for reasons unrelated to devstack-gate : | 02:07 |
| clarkb | er :/ | 02:07 |
| *** weshay is now known as weshay_PTO_AM | 02:37 | |
| *** xinliang has joined #zuul | 02:44 | |
| *** xinliang has quit IRC | 02:44 | |
| *** xinliang has joined #zuul | 02:44 | |
| *** bhavik1 has joined #zuul | 03:56 | |
| *** bhavik1 has quit IRC | 04:39 | |
| *** mordred has quit IRC | 05:02 | |
| *** adam_g has quit IRC | 05:02 | |
| *** jtanner has quit IRC | 05:02 | |
| *** mattclay has quit IRC | 05:02 | |
| *** mordred has joined #zuul | 05:03 | |
| *** mattclay has joined #zuul | 05:03 | |
| *** adam_g has joined #zuul | 05:04 | |
| *** jtanner has joined #zuul | 05:31 | |
| *** isaacb has joined #zuul | 06:19 | |
| *** isaacb has quit IRC | 07:13 | |
| *** isaacb has joined #zuul | 07:22 | |
| *** amoralej|off is now known as amoralej | 07:46 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** SotK has quit IRC | 08:48 | |
| *** SotK has joined #zuul | 08:49 | |
| *** electrofelix has joined #zuul | 09:00 | |
| *** jkilpatr has quit IRC | 10:42 | |
| *** isaacb has quit IRC | 10:44 | |
| *** jkilpatr has joined #zuul | 11:19 | |
| *** openstackgerrit has joined #zuul | 11:58 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Install build private key too https://review.openstack.org/494302 | 11:58 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add known_hosts generation for multiple nodes https://review.openstack.org/494700 | 12:00 |
| mordred | Shrews: what theheck is up with that py35 test? | 12:08 |
| Shrews | mordred: no clue. timeout somewhere | 12:08 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Re-enable test_delayed_repo_init https://review.openstack.org/493659 | 12:10 |
| Shrews | mordred: i've been seeing random timeouts in other places. maybe a slow vm? | 12:11 |
| Shrews | jeblair's devstack-gate changes look good except for https://review.openstack.org/496422 | 12:13 |
| *** bhavik1 has joined #zuul | 12:26 | |
| Shrews | jeblair: i just left an additional comment on https://review.openstack.org/496422 about the var there. Based on my experience with the shade test ansible roles, I don't think you need to define the devstack_base_dir again. Then again, you're placing it under 'defaults' and shade uses 'vars', so that *might* make a difference as far as scope. | 12:26 |
| Shrews | ansible var scoping is quite confusing :/ | 12:27 |
| pabelanger | morning! | 12:29 |
| mordred | Shrews: yah - in this case defaults is more like python keyword arguments | 12:29 |
| pabelanger | Shrews: ya, not a fan of it myself. I think puppet does a better job in that case | 12:30 |
| mordred | Shrews: it defines the variable for the role if nothing else defines it | 12:30 |
| mordred | Shrews: so if someone uses that role without passing a parameter and without first using setup-devstack-log-dir, the value in defaults will hold | 12:30 |
| mordred | Shrews: (we should probably switch to using defaults in our shade test roles fwiw) | 12:31 |
| Shrews | mordred: right. but since those roles are contained to devstack-gate, would we ever have a need to call the fetch role w/o the setup role? | 12:31 |
| Shrews | pre.yaml is always calling setup | 12:31 |
| Shrews | and post.yaml is always calling fetch | 12:31 |
| Shrews | i mean, it doesn't hurt to leave it. just wanted to mention it in case jeblair thought it might be worth simplifying it a bit. :) | 12:32 |
| mordred | doubtful - I think defining it in defaults is a belt/suspenders/style rather than stricly necessary in this case | 12:32 |
| mordred | ++ | 12:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/496690 | 12:35 |
| mordred | jeblair, pabelanger: ^^ that's a merge up with master - I null-merged one change but mentioned that in the commit message | 12:36 |
| dmsimard | mordred: had you seen my comment in https://review.openstack.org/#/c/494700/ ? | 12:38 |
| pabelanger | mordred: Nice | 12:39 |
| mordred | dmsimard: ah - I missed it, sorry - but the reason we're doing the thing with the module is that we already have the hostkeys and we know that they are valid/accurate | 12:39 |
| mordred | dmsimard: doing keyscan would mean we'd have to just trust what we get over the wire at that moment which we're actually in a position to avoid needing to do in our case | 12:40 |
| pabelanger | mordred: left comment | 12:40 |
| dmsimard | mordred: ok, it seemed to me the keys were retrieved by Ansible facts and so wasn't any more reliable | 12:41 |
| mordred | dmsimard: right - but those are retrieved over an ssh connection that we've already validated since we store hostkeys from nodepool into the executors known_hosts - then the executor ssh's into the nodes for gather_facts and collects the host_key over that connection - rather than hitting the ip and fetching the hostkey from that interaction | 12:42 |
| dmsimard | Ok, and facts will *always* be gathered ? If so, that's ok :) | 12:43 |
| mordred | they will in our case - we have fact gathering set up to run once at the top and cache | 12:44 |
| dmsimard | Ack | 12:44 |
| mordred | pabelanger: yah - we've been waiting to delete that file until we freeze master to keep it easy to merge | 12:44 |
| mordred | pabelanger: but I'd love to delete that file because it shows up in my git grep for things :) | 12:44 |
| Shrews | mordred: pabelanger: the last 3 timeout test failures (for nodepool) that I've looked at have all been on rax nodes. any known issues there? | 12:46 |
| mordred | Shrews: not that I'm aware of, but I'm not aware of things | 12:46 |
| Shrews | oh, just hit something on chocolate now. hrm, lots of instability | 12:47 |
| pabelanger | Shrews: got log? | 12:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove zuul 2.5 specific files https://review.openstack.org/496693 | 12:47 |
| pabelanger | Shrews: we are having some networking issues there | 12:47 |
| mordred | pabelanger: but there you go as a followup ^^ | 12:47 |
| Shrews | pabelanger: recent chocolate: http://logs.openstack.org/51/496251/1/check/nodepool-coverage-ubuntu-xenial/ba35e9a/ | 12:48 |
| pabelanger | mordred: +2 to both | 12:48 |
| Shrews | pabelanger: rax: http://logs.openstack.org/51/496251/1/check/gate-nodepool-python35/734cde5/console.html | 12:48 |
| pabelanger | Shrews: rax timed out. looks like testr died or was just running slow | 12:49 |
| pabelanger | same with infracloud, but zookeeper maybe died? | 12:49 |
| * Shrews blames post-eclipse sun particles | 12:52 | |
| *** bhavik1 has quit IRC | 13:09 | |
| Shrews | ugh, the py35 job is going to timeout _again_ | 13:09 |
| mordred | Shrews: jeez | 13:11 |
| pabelanger | mordred: okay, so hoping to finish up twine testing this morning. Just waiting for jeblair to review: https://review.openstack.org/496450/ Once landed and testing works, I'll move on to testing with gpg key | 13:22 |
| pabelanger | mordred: this afternoon, I guess I'll then move into devstack-gate things | 13:22 |
| *** amoralej is now known as amoralej|lunch | 13:27 | |
| *** amoralej|lunch is now known as amoralej | 14:12 | |
| dmsimard | mordred: I fixed up https://review.openstack.org/#/c/495929 and https://review.openstack.org/#/c/495928 -- they should be good to go | 14:25 |
| dmsimard | clarkb: you're right about that dvr job, it seems like it's just failing all the time and not just on the overlay patch | 14:26 |
| dmsimard | I'll try and rope in an expert | 14:26 |
| jeblair | pabelanger: -1 on 496450 | 14:42 |
| jeblair | mordred: are you working on the merge resolution to 254957? | 14:45 |
| jeblair | pabelanger: why is it okay to merge 496690 without a resolution to 254957? | 14:46 |
| dmsimard | clarkb: how would you feel about taking out that dvr job (which is >90% failures http://i.imgur.com/fvTwrfG.png ) and adding "gate-tempest-dsvm-neutron-dvr-ubuntu-xenial" which is voting in tempest ? | 14:48 |
| dmsimard | The tempest job isn't HA so it's not as good as an exercise but at least it can be relied on | 14:49 |
| pabelanger | jeblair: re: 496450. Okay, I understand your comment, I've had that issue in the pass and I'll refactor common bits into roles. Would you also like a more discriptive playbook name? I've just been defaulting to pre.yaml / run.yaml / post.yaml for filename | 14:50 |
| jeblair | pabelanger: i think naming the playbook for the job is fine if we use it that way. if we started reusing them like roles, we'd have to make them more descriptive, like roles i think. but my preference for this would be the role refactor and simple playbooks. | 14:52 |
| pabelanger | jeblair: okay, maybe at PTG we can talk about about reusing playbooks across jobs also. I'll make the changes now to continue moving forward | 14:54 |
| jeblair | pabelanger: i'm not completely -2 on playbook reuse, but i think we have to have good answers to questions like why it makes things easier to use/understand, as compared to roles. the thing i'm most worried about is ending up with spaghetti. | 14:58 |
| jeblair | pabelanger: so yeah, good topic of conversation. :) | 14:58 |
| clarkb | dmsimard: we need a dvr + multinode job to test the overlay properly. Is there a non ha dvr multinode job we can use? | 14:58 |
| dmsimard | clarkb: hm, dunno /me looks | 14:58 |
| clarkb | dmsimard: the reason for that is with dvr each instance terminates l3 so we actually get traffic over our overlay | 14:59 |
| clarkb | without dvr neutron just backhauls everything on the controller gor itself | 14:59 |
| dmsimard | clarkb: yeah, makes sense | 15:01 |
| dmsimard | clarkb: only other dvr multinode job I see is gate-grenade-dsvm-neutron-dvr-multinode-ubuntu-xenial | 15:01 |
| dmsimard | it's voting in neutron | 15:01 |
| clarkb | if the tempest tests grenade runs ssh into instances I think that job is sufficient | 15:02 |
| clarkb | mtreinish would probably know | 15:02 |
| dmsimard | clarkb: yeah it does, it runs tempest too http://logs.openstack.org/38/496438/2/check/gate-grenade-dsvm-neutron-dvr-multinode-ubuntu-xenial/afee022/logs/stackviz/#/stdin/timeline | 15:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add .zuul.yaml https://review.openstack.org/496788 | 15:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add fetch-python-sdist-output role https://review.openstack.org/496789 | 15:05 |
| dmsimard | clarkb: I'll go ahead and propose a patch to substitute those two jobs | 15:08 |
| clarkb | dmsimard: maybe move the ha job to experimental | 15:09 |
| dmsimard | clarkb: sure | 15:09 |
| dmsimard | clarkb: oh would you look at that | 15:09 |
| dmsimard | clarkb: there's already gate-tempest-dsvm-neutron-dvr-multinode-full-ubuntu-xenial-nv in experimental | 15:09 |
| pabelanger | jeblair: okay, 496450 updated per your comments. Do you mind confirming | 15:10 |
| jeblair | pabelanger: lgtm | 15:33 |
| jeblair | mordred: more missing output: http://logs.openstack.org/42/496442/3/check/devstack-lxc/2533fb0/job-output.txt.gz#_2017-08-23_15_36_37_076438 those two tasks emit output (shows up in json, not text log) | 15:42 |
| mordred | jeblair: that's really weird content to miss | 15:46 |
| jeblair | pabelanger: want to +3 496788? | 15:48 |
| jeblair | (it's a master change) | 15:49 |
| pabelanger | jeblair: done! | 15:49 |
| pabelanger | ya, too me a second to see that | 15:49 |
| openstackgerrit | Merged openstack-infra/zuul master: Add .zuul.yaml https://review.openstack.org/496788 | 15:54 |
| SpamapS | ooooooooooo | 15:56 |
| SpamapS | neat | 15:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only depend-on open changes https://review.openstack.org/496808 | 15:58 |
| mordred | jeblair: ^^ I think that's it - I thought it was going to be worse than that | 15:58 |
| jeblair | mordred: cool | 15:59 |
| jeblair | mordred: any chance Failure using method (v2_playbook_on_task_start) in callback plugin (<ansible.plugins.callback.zuul_stream.CallbackModule object at 0x7f883755cc50>): 'CallbackModule' object has no attribute 'play'" is related? | 15:59 |
| pabelanger | https://review.openstack.org/496789/ could use a +3 Adds fetch-python-sdist-output role. | 16:00 |
| mordred | jeblair: yah - maybe so | 16:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: The play object is at self._play not self.play https://review.openstack.org/496810 | 16:03 |
| mordred | jeblair: ^^ | 16:03 |
| mordred | jeblair: thank you - that would be the issue | 16:03 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add fetch-python-sdist-output role https://review.openstack.org/496789 | 16:03 |
| jeblair | mordred: cool, i'm working on a simple test, i'll stack it on that change to verify the fix | 16:06 |
| mordred | jeblair: cool | 16:07 |
| *** jkilpatr has quit IRC | 16:08 | |
| SpamapS | wow.. the OpenSSH auth keys file format is... | 16:20 |
| SpamapS | diabolically weird | 16:20 |
| SpamapS | "The options field is optional; its presence is determined by whether the line starts with a number or not (the options field never starts with a number)" | 16:20 |
| SpamapS | But neither do any of the other fields so..... ? | 16:20 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: The play object is at self._play not self.play https://review.openstack.org/496810 | 16:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add a test to verify basic console output https://review.openstack.org/496824 | 16:23 |
| jeblair | mordred: that did not seem to fix it ^ :( | 16:24 |
| *** weshay_PTO_AM is now known as weshay | 16:25 | |
| jeblair | mordred: (it did fix the warning; but it did not fix the lack of output) | 16:31 |
| mordred | jeblair: ok. I'm about to lunch, I'll dig in to try to find/fix that after lunch | 16:31 |
| jeblair | mordred: cool thx | 16:32 |
| jeblair | mordred: oh, hrm, i wonder if the test is invalid due to lack of zuul_console | 16:33 |
| pabelanger | jeblair: mordred: interested with the following: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.yaml#n173 playbook/publish/pypi ran first, then python-tarball/post: http://logs.openstack.org/42/91893a0d643e02704ad8c8c6ec86aacfd03bad42/release/release-openstack-python/fbab54c/job-output.txt.gz | 16:37 |
| pabelanger | I would have expected the other way around | 16:37 |
| jeblair | pabelanger: i agree | 16:38 |
| pabelanger | jeblair: do you have an idea where I should look to fix? | 16:39 |
| pabelanger | I can reverse the playbooks for testing too | 16:39 |
| jeblair | pabelanger: don't reverse the playbooks, fix zuul | 16:40 |
| jeblair | pabelanger: that'll be in configloader | 16:40 |
| pabelanger | k, looking now | 16:40 |
| *** jkilpatr has joined #zuul | 16:53 | |
| *** dkranz has joined #zuul | 17:01 | |
| * SpamapS may have found a doc bug in sshd's AUTHORIZED KEY FILE FORMAT section .... | 17:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add a test to verify basic console output https://review.openstack.org/496824 | 17:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add a test to verify basic console output https://review.openstack.org/496824 | 17:15 |
| jeblair | mordred, Shrews: ^ okay, that doesn't do anything to verify streaming output, but it does verify output over localhost. it also, i think, fixes a bug which was causing us not to see any output from localhost. | 17:16 |
| * SpamapS finds the bottom of the rathole | 17:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Ensure post-run playbooks are ordered correctly https://review.openstack.org/496833 | 17:16 |
| jeblair | i can't imagine the bottom of a rathole is pleasant | 17:16 |
| pabelanger | jeblair: ^ I believe that fixes our issue. Review most welcome | 17:17 |
| jeblair | pabelanger: yeah, that looks right to me, assuming the tests agree :) | 17:18 |
| jeblair | SpamapS: ^ agree with pabelanger's change? | 17:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Only depend-on open changes https://review.openstack.org/496808 | 17:22 |
| mordred | jeblair: that looks great | 17:26 |
| mordred | jeblair: (the log one) | 17:26 |
| jeblair | mordred: cool. i think we could also check the ansible connection method instead of defaulting that to localhost. <shrug> | 17:27 |
| mordred | jeblair: also, ugh. thank you for the comma | 17:27 |
| jeblair | np. easier than typing that out :) | 17:27 |
| SpamapS | https://github.com/ansible/ansible/pull/28449#pullrequestreview-58152542 <-- the bottom of the ssh rathole btw | 17:27 |
| SpamapS | jeblair: looking | 17:27 |
| jeblair | SpamapS: wow yeah that looks like it was "fun" | 17:29 |
| * mordred hands SpamapS a bundt cake | 17:29 | |
| SpamapS | a cake, with a hole in it! | 17:29 |
| SpamapS | so yes I agree with the way the tests turn out in 496833 ... | 17:30 |
| SpamapS | I'm not entirely sure I grasp why reversed() is needed. | 17:30 |
| SpamapS | but the test shows the result I would expect. | 17:30 |
| SpamapS | shall I +A? | 17:30 |
| mordred | I +Ad | 17:30 |
| SpamapS | yay | 17:30 |
| SpamapS | the old +6+A | 17:31 |
| jeblair | SpamapS: yeah, i mostly wanted to make sure that matched other folks expectations | 17:31 |
| mordred | jeblair: ok - the backport patch broke all the scheduler tests - I'm gonna say that means there's something wrong with it | 17:33 |
| jeblair | mordred: pyflakes says there's still some problems | 17:34 |
| jeblair | but yeah, it's starting to look like there's more to it after that | 17:35 |
| mordred | jeblair: I thikn I see it | 17:36 |
| mordred | jeblair: boolean swap when I ported the change | 17:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only depend-on open changes https://review.openstack.org/496808 | 17:37 |
| mordred | jeblair: it was suposed to go from not dep.is_merged to dep.open - but it went to not dep.open instead :) | 17:38 |
| jeblair | mordred: aha, of course | 17:38 |
| jeblair | mordred: still need to fix the pyflakes stuff | 17:38 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Ensure post-run playbooks are ordered correctly https://review.openstack.org/496833 | 17:38 |
| *** electrofelix has quit IRC | 17:40 | |
| mordred | jeblair: fixed | 17:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only depend-on open changes https://review.openstack.org/496808 | 17:42 |
| jeblair | mordred: lgtm; assuming that passes i'll add 4 points to my review of the merge commit :) | 17:43 |
| mordred | jeblair: \o/ | 17:43 |
| pabelanger | jeblair: mordred SpamapS: We'll need restarts for 496833. Do we need anything else? | 17:47 |
| mordred | pabelanger: jeblair's test for logging | 17:49 |
| jeblair | yeah, since it became more than just a test but also a fix | 17:49 |
| mordred | pabelanger: 496824 which has also afunctional fix | 17:49 |
| SpamapS | FYI, update on the python bug, doko rejected my upload to add another SRU patch in another upload. The SRU team should still get to it in the next few days. | 17:51 |
| mordred | SpamapS: thanks | 17:51 |
| pabelanger | mordred: lgtm +3 | 17:51 |
| pabelanger | I wasn't smart enough to figure out attachements for testr | 17:51 |
| pabelanger | it would be cool if we had the job-output.txt there | 17:52 |
| mordred | jeblair: https://review.openstack.org/#/c/496808 came back green | 17:56 |
| jeblair | pabelanger: BaseTestCase.attachLogs has the magic. but maybe let's spruce that up after ptg. | 17:57 |
| pabelanger | jeblair: ++ | 17:58 |
| jeblair | mordred: i +3d the whole stack | 17:58 |
| mordred | jeblair: woot | 17:58 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add a test to verify basic console output https://review.openstack.org/496824 | 17:59 |
| mordred | pabelanger: ^^ k. I think we're good to restart things now | 18:03 |
| pabelanger | k, let me confirm puppet updated things | 18:04 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/496690 | 18:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove zuul 2.5 specific files https://review.openstack.org/496693 | 18:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Only depend-on open changes https://review.openstack.org/496808 | 18:07 |
| pabelanger | mordred: should we do that for nodepool too?^ | 18:08 |
| jeblair | pabelanger: how about i manually update and restart? | 18:08 |
| pabelanger | jeblair: yes please | 18:08 |
| pabelanger | jeblair: mordred: another thing related to nodepool. Will we need to bring more nodepool-launchers online for PTG? | 18:09 |
| jeblair | ze01 restarted | 18:09 |
| dmsimard | do we want to ansibilize fix_disk_layout ? https://github.com/openstack-infra/devstack-gate/blob/a33308eb3b11a0121f69ca4db11e692d999c60fa/functions.sh#L306 | 18:11 |
| jeblair | pabelanger: yes, we will want to have the full nodepool system up and running for ptg | 18:11 |
| jeblair | dmsimard: yes -- in fact, that's the next thing we'll need. i think the way we may want to do that is start by just putting it in a role as a verbatim shell script. then it can evolve into being more ansible-ish as warranted (though that, or at least large parts of it, may benefit from remaining as shell -- i'm not sure ansible would make that clearer) | 18:14 |
| dmsimard | jeblair: if no one's working on that, I can take a stab at it -- it's in the way of splitting more stuff out ( setup_workspace https://github.com/openstack-infra/devstack-gate/blob/a33308eb3b11a0121f69ca4db11e692d999c60fa/functions.sh#L485 ) | 18:15 |
| jeblair | dmsimard: i think that would be swell. as soon as you do that, we can add it to the nascent devstack job, since that needs to happen basically right at the start | 18:16 |
| dmsimard | okay, on it. | 18:16 |
| pabelanger | jeblair: okay, I've made note of it | 18:17 |
| pabelanger | jeblair: do we need to restart scheduler for 496833? | 18:26 |
| mordred | pabelanger: yah | 18:27 |
| pabelanger | okay | 18:28 |
| pabelanger | okay, zuulv3.o.o restarted | 18:34 |
| mordred | \o/ | 18:34 |
| pabelanger | mordred: I see this warning for callback pluging: http://paste.openstack.org/show/619210/ | 18:39 |
| dmsimard | that reminds me, it really sucks to troubleshoot callbacks right now -- apparently they've made it better in 2.4 so the actual trace/exception bubbles up | 18:41 |
| pabelanger | mordred: also, do you think it make sense to move pip install twine outside of upload-pypi? I have a path issue, and figured do the same logic like the bindep and tox roles, where we expect the executable to be setup before hand | 18:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create pypi_twine_executable https://review.openstack.org/496843 | 18:44 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_merge_conflict_reports https://review.openstack.org/468670 | 19:00 |
| *** tobiash has quit IRC | 19:04 | |
| *** tobiash has joined #zuul | 19:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create pypi_twine_executable https://review.openstack.org/496843 | 19:22 |
| pabelanger | jeblair: mordred: mind looking at https://review.openstack.org/496843/ and https://review.openstack.org/496844/ I am blocked on testing pypi uploads | 19:22 |
| *** olaph1 has joined #zuul | 19:41 | |
| *** olaph has quit IRC | 19:41 | |
| jeblair | pabelanger: why didn't the old twine thing work? | 19:53 |
| pabelanger | jeblair: path issue finding twine | 19:53 |
| pabelanger | we'd likely need to add ~/.local/bin into PATH variable | 19:54 |
| jeblair | pabelanger: because pip install --user didn't put it into $PATH gotach | 19:54 |
| jeblair | gotcha | 19:54 |
| pabelanger | ya, that is what I am thinking | 19:54 |
| jeblair | okay. i wonder what --user is for then. the manpage is no help on the manner "Install using the user scheme". | 19:55 |
| pabelanger | you mean pip install --user? | 19:57 |
| jeblair | pabelanger: yes | 19:57 |
| jeblair | pabelanger: -1 on 496844 +2 on the other | 19:57 |
| pabelanger | jeblair: I've always thought --user just installs things into the ~/.local path | 19:57 |
| jlk | good morning again. | 19:59 |
| jlk | what's the haps? | 19:59 |
| pabelanger | jeblair: replied with question | 20:00 |
| jeblair | jlk: o/ | 20:00 |
| jeblair | pabelanger: i like the install if not installed option. that lets us easily add it to the executors later for efficiency. | 20:01 |
| jeblair | pabelanger: if that's too hard, always install is fine for now | 20:01 |
| jeblair | (i just thought maybe we had examples we could copypasta) | 20:02 |
| pabelanger | let me look quick, I don't think we do | 20:02 |
| jeblair | pabelanger: tox? | 20:02 |
| jlk | any high priority changes needing reviews? | 20:03 |
| pabelanger | jeblair: no, it has the same bug too | 20:03 |
| pabelanger | so, we'd need to fix that | 20:03 |
| pabelanger | let me take a stab | 20:04 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Role to copy the build ssh key to other users https://review.openstack.org/496413 | 20:06 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add support for gzipping static ARA reports https://review.openstack.org/495551 | 20:07 |
| mordred | pabelanger, jeblair: so the --user thing putting into ~/.local is bong because we don't have ~/.local/bin in PATH? | 20:11 |
| Shrews | mordred: so that random nodepool gate-nodepool-python35 failure... turns out when i run 'tox -epy35' locally, it's actually using 3.6 (i don't have 3.5). I wonder if python version matters here with the failure.... | 20:15 |
| mordred | Shrews: oh. well ... | 20:15 |
| clarkb | tox should actually fail in that case | 20:15 |
| mordred | Shrews: that would be "fun" | 20:15 |
| clarkb | and complain that you don't hae the right interpreter available | 20:16 |
| mordred | clarkb: we have "basepython=python3" in the tox.ini testenv ... | 20:16 |
| clarkb | oh uh why? | 20:16 |
| mordred | Shrews, clarkb: maybe we should remove that and put it in specifically for the pep8 job | 20:16 |
| clarkb | I would put it in eg pep8 but not in base | 20:16 |
| mordred | clarkb: so that pep8 also goes under py3 | 20:16 |
| Shrews | also need it for coverage, but can duplicate it there | 20:16 |
| clarkb | that way the env specific targets all work as expected | 20:16 |
| clarkb | or you can spell out all the env specific ones and override I guess | 20:17 |
| mordred | I think there might be a way to tell a tox env to be based on another one ... looking real quick | 20:17 |
| Shrews | oh, well neat. i just got it to hang locally... and zookeeper is definitely running, so can rule that out. i think we have something wonkers in our tests | 20:18 |
| mordred | Shrews: woot | 20:19 |
| pabelanger | mordred: ya, see remote: https://review.openstack.org/496844 Set pypi_twine_executable for upload-pypi role now | 20:20 |
| pabelanger | I think that is more complex, but gives use ability to add twine to executors later | 20:21 |
| mordred | cool | 20:21 |
| pabelanger | hopefully that is all we need to test the upload | 20:22 |
| *** jkilpatr has quit IRC | 20:22 | |
| pabelanger | everything else looked good | 20:22 |
| mordred | ++ | 20:22 |
| mordred | pabelanger: out of curiosity - when you do pre_tasks/post_tasks in project-config playbooks, is that just expediency for doing a thing right now and we can do a role refactor later? | 20:24 |
| mordred | pabelanger: (mostly asking because I'm happy to follow up with some role patches as we go, but don't want to be counter productive) | 20:24 |
| jlk | I think it also causes handlers to trigger | 20:24 |
| mordred | jlk: using pre-tasks vs. roles does? | 20:24 |
| pabelanger | mordred: ya, mostly expediency for the moment | 20:25 |
| mordred | pabelanger: kk - wfm | 20:25 |
| pabelanger | but can turn that into a role | 20:25 |
| jlk | I'm looking to verify, but I think pre-tasks end is a handler barrier, as is roles, and post-tasks. | 20:25 |
| mordred | jlk: do handlers in a rle wait til the end of a listof roles to fire? | 20:25 |
| jlk | "rle" ? | 20:26 |
| jlk | oh sorry | 20:26 |
| pabelanger | ya, I also use pre_tasks for things that need to be done before a role | 20:26 |
| mordred | jlk: I would have expected a role handler to fire at the end of the role - but I could also understand it being at the end of the roles list | 20:26 |
| Shrews | mordred: oh, this is a race between the image upload threads and shutting down the main thread. \o/ | 20:26 |
| mordred | Shrews: yay! it's an actual issue! | 20:26 |
| jlk | honestly I have to look this up in my book. | 20:26 |
| mordred | jlk: I'm disappointed you don't have the book memorized | 20:26 |
| pabelanger | cause, I don't know the ordering of role and task. is it task then role or role then task? | 20:26 |
| Shrews | mordred: also... are you still out of breath? :) | 20:26 |
| jlk | LOL | 20:26 |
| mordred | Shrews: :) | 20:26 |
| mordred | pabelanger: oh - I think pre_tasks + role is much better than mixing role and task | 20:27 |
| jlk | handlers notified within pre_tasks, tasks, and post_tasks sections are automatically flushed in the end of section where they were notified; | 20:27 |
| jlk | handlers notified within roles section are automatically flushed in the end of tasks section, but before any tasks handlers. | 20:27 |
| jlk | (found it on ansible's website) | 20:27 |
| * Shrews just witnessed the fastest talk by mordred EVER. and it was still long | 20:27 | |
| pabelanger | mordred: ya | 20:27 |
| mordred | pabelanger: I was mostly wanting to check in if there was a reason for not-role | 20:27 |
| jlk | There _IS_ an order for tasks vs roles | 20:27 |
| pabelanger | mordred: no reason | 20:28 |
| pabelanger | mordred: I can make it a role now, and also update bindep and tox to do this | 20:29 |
| pabelanger | but figured we want twine uploads working first | 20:29 |
| jlk | FYI | 20:29 |
| jlk | the order is | 20:29 |
| jlk | pre_tasks (handlers), roles, tasks, (role handlers), (task handlers), post_tasks (handlers) | 20:30 |
| pabelanger | ya, that lines up with what I thought | 20:30 |
| *** dkranz has quit IRC | 20:30 | |
| pabelanger | zuul gave a -2 for some reason on 496843 | 20:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add ensure-twine role that upload-pypi depends on https://review.openstack.org/496872 | 20:31 |
| mordred | jlk, pabelanger: what about that ^^? | 20:31 |
| * jlk looks | 20:31 | |
| pabelanger | role dependenies, never used them before | 20:33 |
| jlk | when roles start depending on more generic roles, it starts to feel a little puppetish. But ¯\_(ツ)_/¯ | 20:33 |
| jlk | Ursula used them A LOT | 20:33 |
| pabelanger | how did it turn out? | 20:33 |
| mordred | jlk: do you think it's a good or bad idea? we can also kill the depend and list both roles in the calling playbook | 20:33 |
| pabelanger | my gut is saying, it might turn into a dependency nightmark | 20:34 |
| dmsimard | I find that meta dependencies are sort of confusing | 20:34 |
| pabelanger | but, that is from puppet dependency days | 20:34 |
| mordred | not sure which thing 'feels' better- so definitely interested in past experiences | 20:34 |
| dmsimard | It's like an implicit dep, you're going to run a role but end up running two (or more) | 20:34 |
| mordred | sounds like in general we're not so crazy about the dependency approach? | 20:34 |
| dmsimard | It also prevents you from using the super powerful include_role task | 20:34 |
| dmsimard | <3 include_role | 20:35 |
| pabelanger | mordred: I think we should maybe try at PTG, but I'd prefer not to use them atm | 20:35 |
| pabelanger | to much of a mental map already | 20:35 |
| mordred | kk | 20:35 |
| dmsimard | For example, you can use include_role inside a block and then have a rescue to collect logs if the role fails | 20:35 |
| pabelanger | dmsimard: I too like include_role | 20:35 |
| jlk | role deps kind of go against "least surprise" | 20:35 |
| dmsimard | You can't do that with the regular role directive, if something fails in a role, you're done | 20:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add ensure-twine role https://review.openstack.org/496872 | 20:36 |
| mordred | remote: https://review.openstack.org/496873 Use ensure-twine role to ensure twine exists | 20:36 |
| pabelanger | +2 | 20:36 |
| jeblair | pabelanger: did you look at why it -2d? | 20:36 |
| jlk | well, you could put the recover IN the role | 20:36 |
| jlk | but yeah | 20:36 |
| pabelanger | jeblair: sorry, where? | 20:36 |
| dmsimard | jlk: but then you need to put everything in the role in blocks | 20:37 |
| dmsimard | jlk: instead of wrapping it completely | 20:37 |
| pabelanger | jeblair: oh, zuul | 20:37 |
| pabelanger | jeblair: ya, still looking | 20:37 |
| pabelanger | but POST_FAILURE | 20:37 |
| pabelanger | not much in logs | 20:37 |
| jlk | dmsimard: sure. But it's an interesting thought. Should the recover for the role live WITH the role, or outside of it? | 20:37 |
| dmsimard | jlk: they're not mutually exclusive | 20:38 |
| dmsimard | jlk: you can wrap all your role includes into a rescue that wraps up the job (i.e, log collection) so that uncaught failures/exceptions are still caught | 20:38 |
| dmsimard | and then individually rescue inside your roles on a need basis | 20:38 |
| jlk | nod | 20:39 |
| pabelanger | jlk: jeblair: I am going to recheck 496843 and try finger job. debug.log doesn't have anything helpful | 20:39 |
| dmsimard | a failure that is rescued inside a role won't trigger the rescue from the outer block | 20:39 |
| jlk | I think I've shied away from include_role because it's so new, and it was pretty broken last time I looked at it | 20:39 |
| dmsimard | jlk: there was an issue with handlers was the last thing I'm aware of, but the fix landed in 2.3.2 iirc | 20:40 |
| pabelanger | okay, I am seeing POST_FAILURE in check too | 20:40 |
| pabelanger | some something is up | 20:40 |
| dmsimard | jeblair: so I wanted to do something nice for fix_disk_layout but from quick prototyping it looks like it will make it harder to review since the steps are more verbose/broken down, I guess I'll go with the vertabim approach but makes me :( | 20:41 |
| dmsimard | I'm confident I can get 100% the same result with pure ansible but meh | 20:42 |
| pabelanger | looks like emit-ara-html | 20:42 |
| jeblair | pabelanger: i just approved the gzip ara change | 20:42 |
| dmsimard | pabelanger: the html gzip merged yeah | 20:42 |
| pabelanger | http://paste.openstack.org/show/619231/ | 20:42 |
| dmsimard | huh /me looks | 20:42 |
| jeblair | dmsimard: re disk layout -- yeah, at least getting it in there in a self-contained role will let us easily iterate | 20:43 |
| Shrews | mordred: hrm, i think i was wrong on the race. digger deeper... | 20:43 |
| Shrews | digging, even | 20:43 |
| dmsimard | pabelanger, jeblair: okay, that's no good -- ansible wants to create a .gz archive of the ara folder and that's not the intended goal, submitting a fix now | 20:44 |
| pabelanger | k | 20:45 |
| pabelanger | I think we'll need to force merge the fix | 20:46 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Don't create a gzipped folder: recursively gzip ARA contents https://review.openstack.org/496916 | 20:48 |
| dmsimard | pabelanger, jeblair ^ | 20:48 |
| dmsimard | "gzip --recursive --best {{ zuul.executor.log_root }}/ara" is what devstack-gate uses right now. | 20:49 |
| pabelanger | dmsimard: why didn't you just add dest? | 20:50 |
| dmsimard | pabelanger: it wants to create an ara.gz folder | 20:50 |
| dmsimard | that's not going to fly for serving it over the web | 20:50 |
| dmsimard | we want to recursively gzip every html/css/js files in the ara folder | 20:50 |
| pabelanger | oh, you want to compress all files inside the directory | 20:51 |
| dmsimard | right | 20:51 |
| pabelanger | you could use find / archive tasks. Do we to do this the ansible way? | 20:53 |
| pabelanger | jlk: jeblair: mordred: ^ do you have a preference | 20:54 |
| jeblair | pabelanger: no pref | 20:55 |
| dmsimard | pabelanger: don't want to use find | 20:58 |
| dmsimard | there's potentially thousands of files in there | 20:58 |
| dmsimard | so you'll end up doing 1000's of archive tasks | 20:59 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Don't create a gzipped folder: recursively gzip ARA contents https://review.openstack.org/496916 | 20:59 |
| dmsimard | which is overkill | 20:59 |
| pabelanger | dmsimard: true | 20:59 |
| dmsimard | I guess we need a test job for zuul-jobs :/ | 21:03 |
| dmsimard | sorry about that, I'll go ahead and put something together to try and avoid that from re-ocurring | 21:04 |
| pabelanger | jeblair: mordred: did we just want to squash 496873 and 496844 otherwise 496844 still needs a +2 /+3 | 21:05 |
| jeblair | pabelanger: lets just merge them all | 21:08 |
| pabelanger | jeblair: thanks | 21:09 |
| mordred | dmsimard, pabelanger: yes - we need a test job for at least the base logging things - I'm going to get annoyed enough any day now and write one | 21:09 |
| dmsimard | mordred: I'll get something up by EOD | 21:10 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create pypi_twine_executable https://review.openstack.org/496843 | 21:11 |
| pabelanger | mordred: I've started on the ansible-lint across the 4 projects jobs | 21:13 |
| Shrews | someone please remind me of the magic incantation to set concurrency=1 with tox.... | 21:18 |
| jeblair | Shrews: heh "ttrun" ? :) | 21:19 |
| jeblair | (that's all i know how to use anymore) | 21:19 |
| Shrews | jeblair: yeah, running that now. but wanted to see if testr/subunit related since i see that process hanging | 21:20 |
| Shrews | if *this is* | 21:21 |
| jeblair | ah | 21:21 |
| Shrews | not finding ANY output that is helpful so far | 21:21 |
| dmsimard | oh man, fix_disk_layout is one hell of a rabbit hole | 21:22 |
| jeblair | Shrews: if you just do "tox -e py35 --concurrency=1" does that work? | 21:22 |
| Shrews | jeblair: no | 21:23 |
| Shrews | hrm, maybe: tox -epy36 -- --concurrency=1 | 21:23 |
| Shrews | yup | 21:24 |
| dmsimard | going through fix_disk_layout <- setup_workspace -> setup_project -> ['git_clone_and_cd', 'git_remote_set_url', 'git_remote_update', 'git_prune', 'git_fetch_at_ref', 'git_checkout'] <- git_timed | 21:24 |
| * Shrews now waits 4x as long for a failure | 21:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Document list of configuration items for include/exclude https://review.openstack.org/496928 | 21:25 |
| mordred | jeblair: ^^ let's see how that feels | 21:25 |
| jeblair | dmsimard: wait, i thought you were just working on fix_disk_layout? | 21:26 |
| mordred | dmsimard: I believe you can assume in fix_disk_layout that setup_workspace and git cloning will all have been done | 21:26 |
| jeblair | well, moreover, i don't see anything in there that requires that they be done | 21:26 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add ensure-twine role https://review.openstack.org/496872 | 21:27 |
| dmsimard | jeblair: yeah, I was looking at where fix_disk_layout was used which is from setup_workspace -- do we need to move setup_workspace too ? | 21:28 |
| jeblair | dmsimard: large parts of setup_workspace are irrelevant, we need to consider it piecemeal | 21:29 |
| dmsimard | ack | 21:29 |
| *** jkilpatr has joined #zuul | 21:29 | |
| Shrews | jeblair: mordred: seems we began seeing the random nodepool py35 job hangs with this change https://review.openstack.org/492219 which seems like a good candidate for causing that kind of problem. Will either find it, or revert the change. | 21:33 |
| mordred | Shrews: yes- I agree, that definitely seems like a good candidate for causing that kind of problem | 21:34 |
| Shrews | which saddens me, b/c i liked that change | 21:34 |
| mordred | yah | 21:34 |
| mordred | Shrews: well, maybe there's an 'easy' fix | 21:35 |
| mordred | Shrews: at least you probably found the issue | 21:35 |
| dmsimard | hey, with in-repo .zuul.yaml's, can you inherit from a parent job that lives in another project (if the project is in "required-projects") ? | 21:44 |
| jeblair | dmsimard: you can inherit from any job that has already been defined (project order in tenant configuration matters) regardless of required-projects | 21:45 |
| jlk | I.... I think so? I forget what all the rules are for inheritance. What kind of project can be a parent. | 21:45 |
| dmsimard | jeblair: what is this magic | 21:45 |
| mordred | dmsimard: there is so much magic in v3 you're going to love | 21:46 |
| jeblair | (i'd like to change that to support inheriting from projects after the current one too... but... well, sometime after 13 days from now :) | 21:46 |
| mordred | ++ | 21:46 |
| dmsimard | So what prevents a project from using a name that was defined elsewhere ? | 21:46 |
| dmsimard | Wouldn't that mess up things ? | 21:46 |
| jeblair | dmsimard: we ask nicely | 21:46 |
| jlk | I believe you get config errors | 21:46 |
| dmsimard | ok :) | 21:46 |
| jlk | right? | 21:47 |
| jeblair | dmsimard: first name wins. "important" projects come early so they usually win. and yes, if you try to do that, config errors. | 21:47 |
| clarkb | jeblair: thats yaml behavior too right? (first name wins) | 21:47 |
| dmsimard | jeblair: ok so for example your devstack-gate .zuul.yaml has "base" for a parent, but doesn't necessarily declare where base is from | 21:47 |
| dmsimard | (where is base from anyway?) | 21:47 |
| jeblair | clarkb: well, we avoid dict name collisions | 21:47 |
| mordred | dmsimard: openstack-infra/project-config | 21:48 |
| dmsimard | well there ya go, thanks | 21:48 |
| dmsimard | black magic I tell you | 21:48 |
| mordred | dmsimard: the main base things in are openstack-infra/project-config:zuul.yaml - and other standard shared things that don't need to do trusted behaviors are in openstack-infra/zuul-jobs | 21:48 |
| dmsimard | (the nice kind of black magic) | 21:48 |
| jeblair | (modulo "shadow" -- we allow zuul-jobs to shadow project-config, so jobs defined in zuul-jobs with names that duplicate project-config won't have a config error, and the definitions in project-config still win) | 21:48 |
| mordred | ++ | 21:48 |
| mordred | yah- project-config and zuul-jobs are special | 21:49 |
| mordred | but also mostly define things that should just work for you | 21:49 |
| mordred | jeblair: looking at the d-g .zuul.yaml - do we want a multinode-base job that does the extra bits to set up multi-node things and then have devstack inherit from that even for single-node jobs? | 21:50 |
| jeblair | dmsimard: the "base" in that job definition is actually a noop -- that's the default -- we could omit it. i mostly put it there to have the TODO to change it to multinode when the multinode parent job exists | 21:50 |
| dmsimard | mordred: so wait, we could just yank devstack-lxc into zuul-jobs to get some minimum amount of coverage ? | 21:50 |
| mordred | dmsimard: coverage from? | 21:51 |
| dmsimard | mordred: from random things bricking the gate | 21:51 |
| jeblair | mordred: that's what that TODO means (that's detailed in the etherpad) | 21:51 |
| mordred | oh - no - well - so we also put devstack-gate high in the list of projects - so it's going to come first and win most of the time | 21:51 |
| mordred | jeblair: cool | 21:51 |
| dmsimard | mordred: I wouldn't redefine the job, but add it to the check gate of zuul-jobs | 21:51 |
| dmsimard | here, let me try | 21:52 |
| mordred | dmsimard: that's more complex actually | 21:52 |
| dmsimard | mordred: oh ? | 21:52 |
| mordred | dmsimard: the things that would cause us problems currently are in trusted parts of the job config which means they don't update job config speculatively | 21:52 |
| jeblair | dmsimard: zuul-jobs runs the zuul py35 unit tests for that reason; i'm not sure adding devstack-gate to it would cover anything not already covered by that or devstack-gate jobs | 21:52 |
| mordred | dmsimard: so landing changes that affect how the base job does logging don't get self-tested like other changes, because that would open things to attacks exposing secrets and stuff | 21:53 |
| dmsimard | mordred: makes sense | 21:53 |
| clarkb | out of curiousity what is devstack-lxc? a job that installs openstack with lxc compute driver? | 21:53 |
| jeblair | in fact, ara generation is probably the first non-trivial trusted pre/post role that actually *could* be tested on a test node | 21:54 |
| dmsimard | clarkb: jeblair's ongoing work, like https://review.openstack.org/#/c/496864/ | 21:54 |
| jeblair | clarkb: yeah, i just picked it as a simple job that used the devstack local_conf macro | 21:54 |
| clarkb | gotcha was wondering where lxc came from that makes sense (re local_conf) | 21:54 |
| dmsimard | jeblair: yeah, I was interested in something that exercised the base/pre/post at a minimum inside zuul-jobs -- right now we just do tox-linters. That's why I thought adding devstack-lxc could have made sense but I'll make something up. | 21:55 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Refactor ensure-twine https://review.openstack.org/496945 | 21:56 |
| pabelanger | mordred: ^ updates jlk suggested. Also, it didn't work as expected. So now, we condition just on rc value for both | 21:56 |
| dmsimard | jeblair: I think ideally each role would be tested individually somehow ? Otherwise we need to run all of them or something | 21:56 |
| jeblair | dmsimard: we can make a job for roles that need to be specially tested and run them only on changes to that role (file: roles/ara-generation) | 21:58 |
| pabelanger | dmsimard: this is how I tests roles today: http://git.openstack.org/cgit/openstack/ansible-role-bindep/tree/tests/test.yaml we could do the same thing in zuulv3 for some of the more simply roles | 21:58 |
| pabelanger | I like the file: roles/foobar idea too | 21:59 |
| dmsimard | jeblair: oh, from filtering in the layout ? | 22:00 |
| dmsimard | jeblair: that's a good idea too. | 22:00 |
| mordred | pabelanger, jeblair: zuul-jobs has playbooks/base/pre.yaml I don't think that's used for anyhting | 22:00 |
| mordred | and a post-ssh.yaml | 22:01 |
| mordred | am I missing anything? | 22:01 |
| mordred | I thnk those are leftovers | 22:01 |
| jeblair | mordred: they are the "sample" base job | 22:02 |
| mordred | jeblair: but there is no sample base job anymore | 22:03 |
| jeblair | well there you go | 22:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add multinode base job https://review.openstack.org/496948 | 22:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove base playbooks https://review.openstack.org/496949 | 22:03 |
| mordred | remote: https://review.openstack.org/496951 Use multinode base job for devstack base job | 22:05 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a no-op job to test that the 'base' parent job works https://review.openstack.org/496952 | 22:08 |
| dmsimard | should stuff from pre be shown in the console ? | 22:10 |
| dmsimard | cause it's not | 22:10 |
| dmsimard | from "pre-run" | 22:10 |
| jeblair | dmsimard: unless there's a bug, there is no way you can cause a change to a role in zuul-jobs used by a playbook in project-config to be used before it merges. | 22:10 |
| pabelanger | jeblair: mordred: do you also mind reviewing: https://review.openstack.org/496945/ update to ensure-twine | 22:11 |
| dmsimard | jeblair: ah, bummer, my brain now assumes there's more black magic than there is :) | 22:11 |
| dmsimard | jeblair: ok, let's test roles individually then. | 22:12 |
| dmsimard | but hang on, let's take the opportunity to test if that's really true | 22:12 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a no-op job to test that the 'base' parent job works https://review.openstack.org/496952 | 22:13 |
| jeblair | dmsimard: that spell requires divining the intent of the author, which i haven't mastered yet. :) when used by a config project, those roles run in the trusted execution context which means they can cause damage to the executor. so zuul will use the branch tip to check out those roles in that case. when the same role is used in the untrusted execution context, it uses the proposed change. so the python unit test roles are self-testing, but ... | 22:14 |
| jeblair | ... log upload roles aren't. | 22:14 |
| dmsimard | mordred: so, the stuff from base's 'pre-run' isn't displaying in the console, is that expected ? It goes from ara's DB initialization to the child job's 'run' | 22:15 |
| mordred | dmsimard: no- I would expect to see logging of pre-run tasks | 22:15 |
| dmsimard | jeblair: I confirm that there's no bug, I re-introduced the archive task bug from earlier and that didn't break it | 22:16 |
| dmsimard | mordred: http://paste.openstack.org/show/619245/ | 22:16 |
| dmsimard | mordred: so it kind of "hangs" for a few seconds after ARA's db initialization but I'd assume that base's pre-run is being ran during that time | 22:16 |
| mordred | dmsimard: I am unpleased about that | 22:16 |
| mordred | dmsimard: I see pre-run on this one: http://logs.openstack.org/51/496951/1/check/devstack-lxc/f10e28b/job-output.txt.gz | 22:17 |
| pabelanger | its been some time since I've seen prepare_workspace in job-output.txt | 22:17 |
| mordred | ok. we need to dig in to that - we should see all the pre-run tasks | 22:17 |
| pabelanger | Ya, I think ARA database is doing something | 22:18 |
| mordred | I'm close to EOD - but I can dig in to that tomorrow if nobody else has | 22:18 |
| dmsimard | mordred: yeah it's missing a bunch of stuff, like mirror config and stuff | 22:18 |
| mordred | I think there's a logging interaction between we need to sort out - I started a patch | 22:18 |
| pabelanger | mordred: please https://review.openstack.org/496945/ so I can finish up testing :) | 22:18 |
| dmsimard | mordred: want me to open up a bug or something ? | 22:18 |
| mordred | dmsimard: not unless you really want to | 22:20 |
| dmsimard | ok | 22:20 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add multinode base job https://review.openstack.org/496948 | 22:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Squelch ara initializing message from log https://review.openstack.org/496955 | 22:20 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove base playbooks https://review.openstack.org/496949 | 22:20 |
| mordred | I maybe never pushed that one up | 22:20 |
| pabelanger | pretty sure it is alembic causing issues with base/pre.yaml | 22:22 |
| pabelanger | 2017-08-23 22:22:00,684 DEBUG zuul.AnsibleJob: [build: 073cb03f59ba4b079ecdf8b2565a4560] Ansible output: b'INFO [alembic.runtime.migration] Context impl SQLiteImpl.' | 22:22 |
| pabelanger | I see that in debug.log | 22:22 |
| dmsimard | pabelanger: that's from ara | 22:23 |
| mordred | yah | 22:23 |
| dmsimard | pabelanger: ara's logging (especially with alembic) is.. deficient | 22:23 |
| mordred | my working hunch is that we have fighting logging configs | 22:23 |
| pabelanger | mordred: yes, I agree | 22:23 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Refactor ensure-twine https://review.openstack.org/496945 | 22:24 |
| pabelanger | jeblair: should I still be seeing data from gearman in debug.log? | 22:24 |
| pabelanger | 'sensative data' | 22:24 |
| mordred | I think we need to add an optional way to set a logging config file (via env var or whatnot) to both ara and zuul_stream and stop using basicConfig | 22:24 |
| jeblair | pabelanger: i don't think so | 22:24 |
| pabelanger | okay, I still do :( | 22:25 |
| mordred | dmsimard: I'll send you some ara patches tomorrow and also ones for zuul_stream | 22:25 |
| mordred | and with that - my laptop is dying and I need to join th ehumans | 22:25 |
| pabelanger | Ah, I think I see the issue | 22:26 |
| jeblair | pabelanger: i'll fix it | 22:26 |
| pabelanger | jeblair: okay, thanks | 22:26 |
| dmsimard | jeblair: is there an example of file filtering for controlling job execution ? I'd assume it's not the same as in v2 and I can't find an example, the spec turns up empty on keywords | 22:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove excess debugging from playbook prep https://review.openstack.org/496956 | 22:29 |
| jeblair | pabelanger: ^ | 22:29 |
| jeblair | dmsimard: not yet, but here are the docs: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#attr-job.files | 22:29 |
| dmsimard | jeblair: ah that's a thing, thanks -- bookmarked | 22:30 |
| * dmsimard was actually poking at configloader right now | 22:30 | |
| jeblair | dmsimard: "git grep files: tests" may get you some examples too | 22:31 |
| jeblair | but it's pretty straightforward | 22:31 |
| pabelanger | WOOT | 22:38 |
| pabelanger | https://testpypi.python.org/pypi/sandbox | 22:38 |
| pabelanger | lucky 0.0.13 | 22:39 |
| pabelanger | http://logs.openstack.org/2a/2fc84b6b8a50a1589b1e3a63e6eac3520f03b42a/release/release-openstack-python/5b7544d/ | 22:39 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Squelch ara initializing message from log https://review.openstack.org/496955 | 22:39 |
| pabelanger | jeblair: mordred: moving onwards to gpg siging next^ | 22:40 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove excess debugging from playbook prep https://review.openstack.org/496956 | 22:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a job to test emit-ara-html https://review.openstack.org/496952 | 22:49 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add a job to test emit-ara-html https://review.openstack.org/496952 | 22:53 |
| dmsimard | ^ is still a WIP, I'll finish later gotta get off for now | 22:53 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Add a job to test emit-ara-html https://review.openstack.org/496952 | 22:59 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Add a job to test emit-ara-html https://review.openstack.org/496952 | 23:01 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Add a job to test emit-ara-html https://review.openstack.org/496952 | 23:24 |
| jeblair | dmsimard: feedback on 496935 | 23:29 |
| dmsimard | Ack | 23:30 |
| jeblair | dmsimard: note that's an example of why, when changing the current v2 devstack job, we have to wait for all the jobs to complete and carefully read the test output. it's very easy to introduce an error that still causes the tests to pass in some cases. | 23:30 |
| *** lennyb has quit IRC | 23:31 | |
| dmsimard | BTW I have no clue why 496952 is failing. I'm assuming the default role path isn't picking up the roles from the roles dir (Ansible defaults to ”{{playbook_dir}}/roles”) but there's nothing in the console showing any hint of an error | 23:31 |
| jeblair | dmsimard: a project's own roles/ directory is automatically added to the role path | 23:33 |
| jlk | mordred: fyi in our modules, we don't have to call subprocess directly, there is an Ansible module util for that. rc, out, err = module.run_command(blah, check_rc=BOOL) | 23:34 |
| jeblair | dmsimard: why did you use irrelevant-files? | 23:35 |
| dmsimard | jeblair: so that updating readme.rst wouldn't trigger the integration tests ? | 23:36 |
| jeblair | dmsimard: that actually means "don't run this test if the *only* change is to README" | 23:36 |
| jeblair | dmsimard: chances of that are slim | 23:36 |
| dmsimard | Oh, huh. Okay. | 23:37 |
| jeblair | dmsimard: let's go ahead and say that the default policy of any project in openstack-infra is not to use "irrelevant-files". it's confusing, nobody understands it, and none of us like it. :) | 23:37 |
| dmsimard | Deal | 23:37 |
| jeblair | dmsimard: *files* on the other hand, is okay. so "files: ^roles/emit-ara-html/.*' would be fine. at any rate, that can wait for later. | 23:38 |
| dmsimard | Yeah the job does trigger so I think that part is fine | 23:39 |
| jeblair | dmsimard: in the ara output for that, i don't see the main playbook run at all. | 23:39 |
| jeblair | dmsimard: i think it's the same error as ansible-lint | 23:41 |
| jeblair | 2017-08-23 23:03:07,214 DEBUG zuul.AnsibleJob: [build: 3d0a331f621c4806a684fe6fd4f0d6c7] Ansible output: b"ERROR! 'role' is undefined" | 23:41 |
| *** lennyb has joined #zuul | 23:41 | |
| jeblair | mordred: what's the word on getting ansible parse/syntax errors into the build output? eg http://paste.openstack.org/show/619246/ | 23:42 |
| dmsimard | jeblair: right, I caught that when testing manually and that's why I added "skip_ansible_lint" tags to the tasks but apparently it doesn't work | 23:43 |
| dmsimard | jeblair: I guess I could try removing the role var and hardcode it for the test of isolating that's where the issue is | 23:44 |
| jeblair | dmsimard: ansible is emitting that error | 23:44 |
| jeblair | dmsimard: so i think ansible-lint caught an actual error. probably shouldn't ignore it? | 23:44 |
| dmsimard | jeblair: the linters don't have the variable set, the var is set inside the integration job | 23:45 |
| jeblair | to be clear, the line i pasted above is output from ansible-playbook logged on the executor | 23:45 |
| dmsimard | I meant to share a single playbook for multiple roles | 23:45 |
| dmsimard | jeblair: is that in the *linters* job? or the integration job ? | 23:45 |
| pabelanger | dmsimard: https://review.openstack.org/495463/ is realivent to unset variables for linting | 23:45 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: WIP: Add a job to test emit-ara-html https://review.openstack.org/496952 | 23:45 |
| pabelanger | now I solved it | 23:45 |
| pabelanger | how* | 23:45 |
| dmsimard | need to afk a bit for food before I digest myself | 23:46 |
| dmsimard | brb | 23:46 |
| jeblair | dmsimard: it is the ansible-playbook output from the integration job. the ansible-lint job would not produce output like that from ansible-playbook, its output appears in the build log. | 23:46 |
| pabelanger | are we having executor run role integration, or have executor run ansible-playbook (on node) to then run role | 23:49 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
