| *** SpamapS has joined #zuul | 00:00 | |
| *** ajafo has joined #zuul | 00:00 | |
| *** rcarrillocruz has joined #zuul | 00:00 | |
| *** jesusaur has joined #zuul | 00:00 | |
| *** pabelanger has joined #zuul | 00:00 | |
| *** cinerama` has joined #zuul | 00:00 | |
| *** yolanda has joined #zuul | 00:00 | |
| *** openstackgerrit has joined #zuul | 00:00 | |
| *** dmsimard|off has joined #zuul | 00:00 | |
| *** kmalloc has joined #zuul | 00:00 | |
| *** mordred has joined #zuul | 00:00 | |
| *** jkilpatr has joined #zuul | 00:00 | |
| *** lennyb has joined #zuul | 00:00 | |
| *** dkranz has joined #zuul | 00:00 | |
| *** amoralej|off has joined #zuul | 00:00 | |
| *** tflink has joined #zuul | 00:00 | |
| *** TheJulia has joined #zuul | 00:00 | |
| *** maxamillion has joined #zuul | 00:00 | |
| *** jtanner has joined #zuul | 00:00 | |
| *** Diabelko has joined #zuul | 00:00 | |
| *** ianw has joined #zuul | 00:00 | |
| *** rbergeron has joined #zuul | 00:00 | |
| *** fbouliane has joined #zuul | 00:00 | |
| *** mnaser has joined #zuul | 00:00 | |
| *** olaph has joined #zuul | 00:00 | |
| *** robled has joined #zuul | 00:00 | |
| *** bstinson has joined #zuul | 00:00 | |
| *** adam_g has joined #zuul | 00:00 | |
| *** robcresswell has joined #zuul | 00:00 | |
| *** pbrobinson has joined #zuul | 00:00 | |
| *** clarkb has joined #zuul | 00:00 | |
| *** kklimonda has joined #zuul | 00:00 | |
| *** jamielennox has joined #zuul | 00:00 | |
| *** tristanC has joined #zuul | 00:00 | |
| *** Shrews has joined #zuul | 00:00 | |
| *** toabctl has joined #zuul | 00:00 | |
| *** mattclay has joined #zuul | 00:00 | |
| *** patrickeast has joined #zuul | 00:00 | |
| *** SotK has joined #zuul | 00:00 | |
| *** zaro has joined #zuul | 00:00 | |
| *** jeblair has joined #zuul | 00:00 | |
| *** rfolco has joined #zuul | 00:00 | |
| *** eventingmonkey has joined #zuul | 00:00 | |
| *** jlk has joined #zuul | 00:00 | |
| *** mmedvede has joined #zuul | 00:00 | |
| *** mgagne has joined #zuul | 00:00 | |
| *** EmilienM has joined #zuul | 00:00 | |
| *** _ari_ has joined #zuul | 00:00 | |
| *** gothicmindfood has joined #zuul | 00:00 | |
| *** zigo has joined #zuul | 00:00 | |
| *** fungi has joined #zuul | 00:00 | |
| *** leifmadsen has joined #zuul | 00:00 | |
| *** ChanServ has joined #zuul | 00:00 | |
| *** tepper.freenode.net sets mode: +o ChanServ | 00:00 | |
| *** kmalloc is now known as Guest11213 | 00:03 | |
| *** smyers has joined #zuul | 00:04 | |
| *** dkranz has quit IRC | 00:28 | |
| *** isaacb has joined #zuul | 03:51 | |
| *** jamielennox has quit IRC | 04:07 | |
| *** jamielennox has joined #zuul | 04:17 | |
| *** isaacb has quit IRC | 05:37 | |
| *** leifmadsen has quit IRC | 05:57 | |
| *** leifmadsen has joined #zuul | 05:58 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_delayed_repo_init https://review.openstack.org/493659 | 06:38 |
|---|---|---|
| *** isaacb has joined #zuul | 07:26 | |
| *** isaacb has quit IRC | 07:39 | |
| *** isaacb has joined #zuul | 07:51 | |
| *** isaacb has quit IRC | 08:13 | |
| *** isaacb has joined #zuul | 08:22 | |
| *** electrofelix has joined #zuul | 08:28 | |
| *** jkilpatr has quit IRC | 10:34 | |
| *** jkilpatr has joined #zuul | 10:52 | |
| *** clarkb has quit IRC | 11:52 | |
| *** clarkb has joined #zuul | 12:19 | |
| *** dkranz has joined #zuul | 12:27 | |
| *** persia has joined #zuul | 13:17 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add some examples to user guide Concepts section https://review.openstack.org/493873 | 13:30 |
| Shrews | jeblair: ^^^ I'm fairly pleased with the clarity that change adds. It seemed to make more sense in the Concepts doc, rather than the config doc. | 13:31 |
| *** dkranz has quit IRC | 14:06 | |
| *** dkranz has joined #zuul | 15:16 | |
| *** Guest11213 is now known as kmalloc | 15:19 | |
| *** kmalloc has joined #zuul | 15:19 | |
| *** isaacb has quit IRC | 15:35 | |
| jeblair | mordred, pabelanger, clarkb, SpamapS: who's around for devstack etherpadding? | 16:00 |
| mordred | jeblair: morning! tying up one loose end real quick - be there in just a sec | 16:01 |
| pabelanger | here too | 16:02 |
| jeblair | https://etherpad.openstack.org/p/AIFz4wRKQm | 16:03 |
| clarkb | o/ | 16:03 |
| clarkb | with toast and tea | 16:03 |
| jeblair | clarkb: to share? | 16:04 |
| clarkb | I don't think tcp works that way | 16:04 |
| jeblair | i added our previous roles refactoring list to the etherpad | 16:06 |
| jeblair | pabelanger: what was involved with 'ssh keys'? | 16:07 |
| pabelanger | jeblair: writing ansile role to distrubute them to nodes | 16:08 |
| pabelanger | but with 1 time keys, I think that has changed | 16:08 |
| jeblair | pabelanger: yeah, we now have a public key on every node (as the zuul user at least -- do we use a different user in devstack-gate?) so i think the only thing missing there would be to copy the private key as well. we can do that. | 16:09 |
| clarkb | nova live migration requires root be able to ssh everwhere too | 16:09 |
| clarkb | we add that in d-g for multinode jobs | 16:10 |
| clarkb | should be easy enough to set a username on the playbook and do it for zuul/jenkins and root though | 16:10 |
| jeblair | ya | 16:10 |
| jeblair | as i think about the ssh keys thing -- what's the applicability of the two "need to..." items that i put in there? should installing the private key to ~zuul on all nodes be part of the base job add-build-sshkey role? what about installing keys as root? should that be a new role? | 16:16 |
| jeblair | clarkb, mordred, pabelanger: ping | 16:18 |
| clarkb | jeblair: I think the base job should handle base connectivity, thats a nice way to make multinode jobs easier for people writing them | 16:19 |
| clarkb | jeblair: but then we probably want to expose that same role for people that need to set up other users like root for libvirt live migrations | 16:19 |
| clarkb | and that can be added to specific jobs that need it | 16:19 |
| clarkb | just thinking that not every job is going to be native ansible on day one so preserving old multinode connectivity method of ssh between nodes will be useful | 16:19 |
| pabelanger | we could try using add-build-sshkey first, sure | 16:20 |
| jeblair | mordred, clarkb, pabelanger: are we all here in irc? | 16:21 |
| clarkb | I am | 16:21 |
| pabelanger | I am | 16:21 |
| mordred | clarkb, jeblair: I kinda think it would be nice if add-build-sshkey in the base job would install keys everywhere so that all nodes can talk to each other | 16:21 |
| jeblair | sorry, a couple of quick meta questions: | 16:21 |
| jeblair | 1 do we want to continue to chat in irc, or only on etherpad? | 16:22 |
| clarkb | I think we should record decisions/details on the etherpad because it keeps that important info next to the todo list | 16:22 |
| clarkb | but irc may be better for having conversation and reaching those decisions or finding out those details | 16:22 |
| jeblair | 2 do we want to add a little bit of structure to this? a minute ago there were chats going on in areas of the etherpad too far apart to keep on my screen | 16:22 |
| jeblair | less structure and chat in etherpad worked pretty well last time, but the problem space wasn't as big | 16:23 |
| mordred | I can now do both IRC and etherpad- so whichever | 16:23 |
| jeblair | okay, mordred started some big picture items, let's focus on those for a minute, then get back to the minutia of roles | 16:23 |
| jeblair | pabelanger: what's the network sanity check role from our original list? | 16:41 |
| jeblair | is that covered by what we do in the base job, or is that related to network overlay? | 16:41 |
| clarkb | jeblair: it pings the mirror and does a dns lookup iirc | 16:41 |
| pabelanger | jeblair: we just ping our mirrors | 16:41 |
| pabelanger | ya | 16:42 |
| jeblair | we're doing that in base now iirc? | 16:42 |
| clarkb | jeblair: its behaviopr that devstack-gate had for a long time that got ansiblified basically to shrot circuit if networking si really broken | 16:42 |
| mordred | we are doing that in base now | 16:42 |
| pabelanger | yes, in base | 16:42 |
| jeblair | okay, i'll strike it | 16:42 |
| jeblair | can we chat about the localrc stuff for a bit? | 16:43 |
| jeblair | this is one of those things where we have lots of jobs which add little bits of localrc config | 16:43 |
| mordred | yup. I believe clarkb and I were talking about different poritions of this | 16:43 |
| jeblair | it would be good if we can have an interface where folks can just inherit from devstack and add their bit of localrc | 16:44 |
| pabelanger | will defer to mordred and clarkb for localrc. need to afk for a moment | 16:44 |
| jeblair | all in zuul.yaml, i think | 16:44 |
| clarkb | the way it works today is you describe what you want in the localrc in your JJB job then devstack-gate comes along and uses devstack-utils to merge that into the localrc that will be used | 16:45 |
| jeblair | let's find a good example of that real quick | 16:45 |
| SpamapS | jeblair: doohhhhh I did the timezone math wrong and had devstack etherpadding for 15 minutes from now | 16:45 |
| SpamapS | apologies! | 16:45 |
| jeblair | SpamapS: we have a ways to go yet :) | 16:45 |
| SpamapS | Good good... | 16:46 |
| SpamapS | (sad thing is I actually have UTC on my google calendar and just forgot to look at it) | 16:46 |
| clarkb | jeblair: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/neutron.yaml#n124 thats one I found thinking neutron would have an example | 16:46 |
| * SpamapS joins the fray | 16:46 | |
| clarkb | I think its a reasonable example since its more than one var being modified | 16:46 |
| jeblair | clarkb: cool | 16:46 |
| jeblair | so we have a neat thing in zuul -- | 16:47 |
| jeblair | normally when a job inherits from another job, attributes are overridden. but the 'vars' attribute (which is job variables that are exported to ansible) is a little special | 16:47 |
| jeblair | it attempts to merge child and parent | 16:48 |
| jeblair | vars is a dictionary. if there are any key collisions, the child wins, but any keys defined by the parent not defined by the child stick around | 16:48 |
| jeblair | that behavior is recursive | 16:48 |
| jeblair | so if a value in vars is a dictionary, it too is merged | 16:49 |
| jeblair | i think we might be able to do something with that; lemme try in etherpad | 16:49 |
| mordred | yah - I think that behavior will be useful - if we make the equiv of the local_conf jjb macro a job input var ... | 16:49 |
| mordred | I think we also need to ensure devstack-utils is installed in a pre-run playbook ... | 16:50 |
| clarkb | and the merge step is called somewhere | 16:50 |
| jeblair | clarkb: what does "[[local|localrc]]" mean? | 16:50 |
| clarkb | jeblair: that is the local.conf python ini config section | 16:51 |
| clarkb | jeblair: basically its the localrc portion of the ini local.conf file | 16:51 |
| jeblair | the section is literally called "loca|localrc" ? | 16:51 |
| clarkb | ya | 16:51 |
| clarkb | let me find an example file | 16:51 |
| SpamapS | I thought that meant it can be either one. | 16:51 |
| mordred | and I think we should make a module that exposes dsconf in ansible so that people can use that in their jobs if they need to conditionally set or not set config values ... | 16:51 |
| clarkb | http://logs.openstack.org/73/493673/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/91e7093/logs/local.conf.txt.gz | 16:51 |
| mordred | such as: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/shade.yaml#n46 | 16:52 |
| pabelanger | ya, I like the example jeblair is mocking up now for localrc vars | 16:52 |
| pabelanger | we do the same with environment variables in tox jobs | 16:52 |
| pabelanger | seems to work well | 16:52 |
| clarkb | and then we can define the base localrc in the base job and we'll merge this on top? that should work well | 16:53 |
| clarkb | in that case we may not need devstack-utils since its job is to merge those items but if zuul/ansible already merge it all for us that dep goes away maybe | 16:54 |
| jeblair | clarkb: right... is there some example for that? i can put it in the etherpad | 16:54 |
| pabelanger | ya | 16:54 |
| clarkb | jeblair: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n589 | 16:55 |
| clarkb | jeblair: basically its a command that you run that merges two ini files together | 16:55 |
| jeblair | mordred: it looks like that shade conditional is conditional on what specific job the template is being used for; but if you think of the job as parent and child, then you may not need conditionals like that | 16:55 |
| mordred | yes - absolutely! | 16:56 |
| clarkb | I think I would do vars:\n devstack_local.conf:\n 'local|localrc':\n Foo: Bar | 16:57 |
| mordred | jeblair: mostly just wanted to point out that since there is a python util called "dsconf" that's used by devstack and others to manipulate files later and that we need installed for d-g anyway- we might as well expose it in a module in case there are times when it actually is needed | 16:57 |
| mordred | it can go in the post-ptg stack though | 16:57 |
| jeblair | mordred: *nod* i think clarkb is thinking that its behavior is close enough to zuul's internal variable merge that it may not be necessary? | 16:57 |
| clarkb | then you can have arbitrary ini headers. Then somewheredown the line have something that can write that out in the ini format the devstack wants. | 16:57 |
| clarkb | jeblair: ya | 16:57 |
| mordred | cool | 16:58 |
| mordred | emitting that should be fairly easy | 16:59 |
| clarkb | I put an example of what the behavior is today as I understand it | 17:00 |
| clarkb | and mocked out what I'd expect zuul/ansible to do which i think is equivalent | 17:00 |
| jeblair | clarkb: that looks right, and ii updated my examples to your syntax | 17:00 |
| jeblair | so when the devstack-neutron-whatever job runs, it will will have all those variables, along with VIRT_DRIVER=fake which is set by the devstack job. | 17:01 |
| clarkb | then the base job would have a base config in the same way devstack-gate has a base config. It sets error on clone, lists a safe IP addr range, sets password values etc | 17:01 |
| clarkb | jeblair: feel free to delete my examples if you think we are done with that | 17:02 |
| jeblair | and the "write localrc" pre-playbook role in the base devstack job will receive all of those variables, so it will write the correct thing | 17:02 |
| clarkb | yup | 17:02 |
| mordred | if we want to support setting abitrary filenames, I think we need a top-level container - but is doing that at the vars level actually valuable given local.conf itself has internal support for that? | 17:02 |
| clarkb | mordred: I don't think the local.conf is an arbitrary filename | 17:03 |
| mordred | right | 17:03 |
| mordred | so the 'devstack_local.conf' variable name isn't meant to imply there could be a different variable name that does something similar yeah? | 17:04 |
| clarkb | correct, its mostly there to make it a unique top level key so that you could have another key that said use this ip range for network overlay | 17:04 |
| clarkb | or use this usename for execution or whatever | 17:04 |
| mordred | (the dot in the name makes it seem like it's going to do something magic with varname -> filename) | 17:04 |
| jeblair | maybe we should call it 'dsconf' or something? | 17:05 |
| mordred | gotcha. I'd suggest devstack_local_conf then just for my eyeballs - I think the dot will get confusing | 17:05 |
| clarkb | wfm | 17:05 |
| jeblair | mordred: that's a good compromise | 17:05 |
| mordred | woot. I think this is gonna be nice | 17:05 |
| jeblair | it still clearly maps to the filename so people can make that connection, but doesn't look as magic | 17:05 |
| clarkb | now one potential gotcha here is that order matters iirc | 17:06 |
| clarkb | does zuul/ansible preserve dict key ordering? I think pyyaml might actually | 17:06 |
| jeblair | clarkb: order in an ini file matters? | 17:06 |
| clarkb | jeblair: ya because its actually reading that section in as if it were bash :/ | 17:07 |
| jeblair | clarkb: by default, pyyaml does not, but we can make it do so. | 17:07 |
| clarkb | we can have dtroyer or sdague confirm that behavior too before we attempt to accomdate it | 17:07 |
| mordred | yah- the code is even already in the zuul repo iirc | 17:07 |
| mordred | I believe later values override earlier values ... | 17:07 |
| clarkb | mordred: ya the overriding of keys doesn't bother me | 17:08 |
| mordred | but maybe that behavior is less important if we have key merging? | 17:08 |
| clarkb | but in our example we use $HOST_IP so that has to be set before hand in the ini file | 17:08 |
| mordred | no- I mean, I think order matters because they have a latest-overrides-earlier behavior in ini reading | 17:08 |
| mordred | which is useful if you just want to append things, yeah? | 17:08 |
| mordred | ahhhhhhh | 17:08 |
| mordred | gotcha | 17:08 |
| clarkb | mordred: yup I'm saying that portion of it is fine as is | 17:08 |
| *** jkilpatr has quit IRC | 17:08 | |
| clarkb | but we need to handle the set HOST_IP first then use $HOST_IP later | 17:08 |
| mordred | so if these vars can refer to other previous vars, then yeah, we'll need to do the ordereddict trick | 17:09 |
| jeblair | okay. that's not ideal (it adds a behavior that is not guaranteed by yaml), but i think we can make that happen. | 17:10 |
| clarkb | it should be backward compatible for anyone that doesn't care about ordering | 17:10 |
| clarkb | but yes does add a behavior change | 17:10 |
| jeblair | well, i'm less worried about backwards compat, and more worried about (and give me a little rope here) language purity | 17:12 |
| jeblair | i think we can make this work in zuul, i just worry that it might impact our ability to do something else later, and maybe makes the problem domain for tools that work with the config more fragile... | 17:12 |
| jeblair | oh i have an idea | 17:12 |
| clarkb | an alternative is to pass literal block strings around as in jjb and let devstack utils or ds tools whatever it is called do the work | 17:12 |
| * mordred waits for jeblair's idea | 17:13 | |
| jeblair | we have an ansible role that actually writes this out. it could actually resolve variable dependencies and write them out in the right order, even with unordered input | 17:13 |
| jeblair | (yes, that means "parsing" bash, but a very limited subset of bash) | 17:13 |
| mordred | jeblair: how would the ansible role get the depends? wouldn't the zuul var merging/override get there first? | 17:14 |
| jeblair | mordred: it would see that "CLIENT_ADDRESS=$HOST_IP" uses $HOST_IP so it makes sure to write out HOST_IP first | 17:14 |
| mordred | jeblair: OH | 17:14 |
| mordred | jeblair: I understand what you are saying now | 17:15 |
| mordred | jeblair: yah - I dont' think that's too difficult | 17:15 |
| clarkb | I think that works in the simple case but you'll need to be careful if any of fancy bash string manipulation features are used | 17:15 |
| clarkb | (and maybe just say no you don't get those) | 17:15 |
| jeblair | clarkb: yeah | 17:15 |
| mordred | let's start with jeblairs idea (it also means no changes to the zuul config loading and can just be done in the local.conf emitter) | 17:16 |
| clarkb | would be good to run that by sdague but I think he is on vacation for some large chunk of time | 17:16 |
| mordred | we can always revisit if there's fancy bash that's essential and have ordereddict in our back pocket if we get in a hole | 17:16 |
| jeblair | ya, if we have to throw in ordereddict in a pinch, we can do that. but this feels a little more conservative so it'd be nice to start there. | 17:17 |
| mordred | ++ | 17:17 |
| mordred | also - I think it's TOTALLY doable to auto-translate local_conf jjb macro invocations to a devstack_local_conf variable setting | 17:18 |
| jeblair | mordred: woot | 17:18 |
| mordred | so that even one-for-one jobs can get the local_conf stuff nicely | 17:18 |
| jeblair | i think all the stuff in https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n258 is going to be complicated | 17:20 |
| jeblair | a lot of that can go into the devstack base job as simple variables | 17:20 |
| jeblair | but some of it is more conditional | 17:20 |
| clarkb | the python3 thing for example would be the python3 d-g job inheriting form base then overriding the python version var | 17:20 |
| clarkb | but yes there is a lot of logic in there that will have to be untablged to end up with ^ | 17:20 |
| jeblair | so do we start making a job hierarchy (like devstack-ironic which is the base job and virt-driver=ironic and all the related values already set) | 17:21 |
| jeblair | clarkb: hey i think we just said similar things :) | 17:21 |
| clarkb | where it will get really tricky is where existing jobs have many of those vars overlapping | 17:21 |
| jeblair | that might be the way to start. but again, we do have a 'devstack_localrc' role. so if we need to keep some of that conditional logic, we can still do it there. | 17:21 |
| clarkb | neutron + dvr + python 3 + multinode (I don't know that that job exists today but illustrates the trickyness) | 17:22 |
| * SpamapS is kind of lost | 17:22 | |
| jeblair | SpamapS: we're focusing on the part of the devstack-gate job which writes the devstack localrc | 17:23 |
| jeblair | SpamapS: see lines 52, 67, and >76 in the etherpad for examples of how we think we can implement that with job variables | 17:24 |
| SpamapS | been waiting for a chance to contribute but I really don't know if I can help | 17:24 |
| jeblair | SpamapS: stick around -- we may have some tractable tasks soon | 17:24 |
| jeblair | mordred, clarkb: is that really all we need with the roles? is it true that with all the stuff that's gone into the base roles, all we have left are the additional ssh work, network overlay, setup swap, and devstack localrc? | 17:26 |
| mordred | SpamapS: :Q | 17:26 |
| mordred | gah | 17:26 |
| jeblair | no :Qing! | 17:27 |
| clarkb | jeblair: and log hanlding. I think that captures it just going through a mental play by play of what d-g does | 17:27 |
| clarkb | we may need something to deal with d-g hooks | 17:27 |
| mordred | jeblair: devstack-gate-vm-wrap.sh itself - which I was just scanning through - and hooks | 17:27 |
| mordred | clarkb: jinx | 17:27 |
| clarkb | oh and executing tempest/grenade | 17:27 |
| jeblair | let's tackle logs for a sec | 17:27 |
| mordred | doesn't devstack itself have a log collection thing now? like devstack tells us what logs should be collected - or puts them into logs/ itself? | 17:28 |
| clarkb | mordred: sort of, it dumps all openstack services into journald with a devstack@ prefix | 17:29 |
| clarkb | mordred: so we do a for service enabled journald dump devstack@$service | 17:29 |
| clarkb | however thats only on pike/master at this point | 17:29 |
| mordred | clarkb: right - so if we put that logic into a devstack script somewhere | 17:29 |
| mordred | clarkb: in devstack itself | 17:29 |
| clarkb | we could execute that script yes | 17:30 |
| mordred | then have a log collection role that runs it if it exists, and then grabs ~stack/logs ? | 17:30 |
| clarkb | (don't forget system services like mysql and libvirt and rabbit) | 17:30 |
| clarkb | ya something like that should work. We'll probably want compression to happen too | 17:30 |
| mordred | I guess what I mean is that in general if we can make devstack the one responsible for putting whatever it wants collected into ~stack/logs | 17:30 |
| clarkb | and maybe don't rely on underlying script to alawys be nice and compress (but noop if it has) | 17:30 |
| mordred | then we can do a verison of our existing log collection roles that grabs from ~stack/logs intead of ~zuul/logs? | 17:31 |
| mordred | is that reasonable? or should we have that be in the devstack-gate-collect-logs role? | 17:32 |
| jeblair | mordred: i think we abandoned that approach to log collection, and now rely on the job doing appropriate log collection in its post playbook | 17:32 |
| mordred | ok. | 17:32 |
| jeblair | mordred: so, eg, tox post playbook copies node:/.tox/whatever to executor:logs | 17:32 |
| mordred | right | 17:32 |
| mordred | sorry - I've been unclear - lemme try again | 17:32 |
| jeblair | so i guess the corresponding thing is our base devstack job should copy all:~stack/logs to executor:logs ? | 17:33 |
| mordred | devstack runs- there are logs on the machine, but they are in various places depending on devstack config | 17:33 |
| mordred | $something runs to get those collected into all:~stack/logs | 17:33 |
| mordred | devstack-gate post-playbook copys all:~stack/logs to executor:logs | 17:33 |
| SpamapS | are we staging them on the executor so we can have untrusted do the staging, and then trusted do the insertion into log storage? | 17:34 |
| mordred | question is- do we have the post playbook itself be the thing that has the logic for step 2? or do we make that a devstack responsibility? | 17:34 |
| jeblair | SpamapS: yes; the base job handles the last bit | 17:34 |
| SpamapS | Just making sure I understand the flow. | 17:35 |
| jeblair | mordred: thinking | 17:36 |
| SpamapS | From a devstack-agnostic point of view, what I'd like is for something to tell my job where to stage its logs to.. and then anything I stage there gets pushed up into logs. | 17:36 |
| SpamapS | so considering devstack, I want that logic in the d-g roles. | 17:37 |
| jeblair | mordred: blue sky -- i think it'd be nice to have the devstack post playbook copy things from wherever they are to the executor. ie, skip ~stack/logs. | 17:37 |
| jeblair | mordred: and if child jobs want to copy additional things, have their own post playbooks | 17:37 |
| mordred | jeblair: wfm | 17:38 |
| jeblair | let me revise/clarify that: | 17:38 |
| jeblair | i think it'd be nice to have the devstack post playbook copy the core things it knows about from wherever they are to the executor. ie, skip ~stack/logs. and if child jobs want to copy additional things, have their own post playbooks. | 17:38 |
| jeblair | (see the 'core things it knows about' insertion) | 17:38 |
| SpamapS | Yeah that's how I see it too. [pre: make a staging place]-path_to_staging_place->[job1][child-of-job1][child-of-job1]->[post: scoop them up and publish] | 17:38 |
| mordred | SpamapS: yah | 17:39 |
| jeblair | SpamapS: yeah, though the staging place i'm arguing for is the one that zuul provides on the executor, so we don't need to do anything in pre | 17:39 |
| SpamapS | Oh I thought base did that. Kk.. even simpler. :) | 17:39 |
| jeblair | yeah, $jobroot/logs is part of the "api" because that's where the console log goes | 17:40 |
| mordred | jeblair: and I think we already have a "copy logs to executor" role that takes arguments? or if we don't we should :) | 17:40 |
| jeblair | mordred: unsure. it'd be a one-liner either way :) | 17:40 |
| jeblair | so having said that -- here's something else to consider | 17:41 |
| jeblair | earlier we made it easy to change devstack's configuration without writing new playbooks -- would this plan for logs cause folks to have to write post-playbooks? or is it the case that because we're proposing the log collection happen in devstack itself, anything we can configure devstack to do with just localrc variables will automatically have its logs collected? | 17:42 |
| clarkb | I think the vast majority of stuff will be caught in the journald devstack@ dumping | 17:43 |
| clarkb | basically any service started by devstack's run_process function | 17:44 |
| mordred | jeblair: well, you sold me on not having devstack collect them since that would involve copying to ~stack/logs I think - maybe if we make a genearl "copy-logs-to-executor" role that takes an arg of what to copy, we could have the devstack-gate base job take a variable which is "Additional logs to copy"? | 17:44 |
| *** jkilpatr has joined #zuul | 17:44 | |
| clarkb | items I expect may end up being additional post playbooks would be for things like docker/k8s in kolla land, zookeeper in nodepool jobs | 17:44 |
| clarkb | should be one or two per jobs and not in every job | 17:45 |
| mordred | yah- where there's legit additional things that need to happen for reasons that arent' just log collection | 17:45 |
| jeblair | mordred: yeah, i think that's workable -- but is it necessary? it looks to me like anything that would need additional logs collected beyond what devstack does would already have local playbooks for the job itself, in which case, a post-playbook for log doesn't seem like a huge cost | 17:45 |
| mordred | yah | 17:46 |
| jeblair | so maybe we don't add that right now, but it's something we can throw in easily if needed | 17:47 |
| mordred | ++ | 17:47 |
| jeblair | okay, next topic: devstack-gate-vm-wrap.sh itself - and hooks | 17:47 |
| clarkb | oh also older devstack will have the screen-* files in the logs dir so we check that too | 17:47 |
| jeblair | clarkb: ++ | 17:47 |
| jeblair | actually, let's stay on logs a bit more | 17:48 |
| jeblair | what do we need to do to get from where we are today to what we're talking about | 17:49 |
| clarkb | a role that knows to look for old logs dir location or generates logs using journalctl otherwise. Also knows to grab mysql, libvirt, etc logs | 17:50 |
| jeblair | okay, i'll try to pit this around line 31+ in etherpad | 17:51 |
| mordred | the cleanup_host function | 17:51 |
| mordred | in functions.sh | 17:51 |
| mordred | is the thing that has the logic today | 17:52 |
| jeblair | okay that makes sense | 17:52 |
| jeblair | line 31 look right? | 17:53 |
| clarkb | jeblair: yes | 17:54 |
| jeblair | okay, devstack-gate-vm-wrap and hooks | 17:54 |
| jeblair | it seems that hooks generally should just be replaced by playbooks | 17:54 |
| clarkb | ++ | 17:54 |
| mordred | yup - ALTHOUGH - there's a catch | 17:55 |
| jeblair | so the devstack job should have a playbook which runs devstack. and if you override the job, you can replace that. | 17:55 |
| jeblair | (and if you need to add pre/post, of course you can do that) | 17:55 |
| mordred | which is that currently the hook scripts are run in a context with a bunch of d-g env vars set | 17:55 |
| clarkb | mordred: ya and many of them go on to run our default hook at the end of their execution | 17:55 |
| mordred | the 'easy' path for folks would be a post-playbook that just does shell: existing/hook/script.sh | 17:55 |
| mordred | but that would be missing the d-g vars | 17:56 |
| mordred | so ... what if we did the following: | 17:56 |
| clarkb | (so we won't be able to delete devstack-vm-gate.sh right away) | 17:56 |
| jeblair | mordred: post playbook? they want to run stuff after devstack and tempest runs? | 17:56 |
| * jeblair waits for mordred | 17:57 | |
| mordred | - split devstack-vm-gate-wrap into two bits - the first part that does all the env var stuff it does today, then outputs all of those vars into a sourceable file - run that as a d-g pre playbook | 17:58 |
| mordred | have a d-g pre and d-g post playbook that take a path to a hook script, and if it gets one, runs that with the d-g vars sourced | 17:59 |
| mordred | turn the second half of devstack-vm-gate-wrap into pre run and post playbooks (instead of command-line ansible) | 17:59 |
| jeblair | mordred: i like the general direction, but i'd like to make sure we're able to deprecate all the hook stuff eventually. so maybe we should avoid doing any of that in the base jobs, but have the conversion script do that expilicitly for jobs with hooks. | 18:03 |
| jeblair | so, if you have some jjb that defines a pre_hook, you get an auto-converted job with a 'legacy-devstack-gate-pre-hook' pre playbook? | 18:04 |
| mordred | jeblair: yah - that works- mostly don't want to make people rewrite their existing shell scripts in order to adopt the new base jobs | 18:04 |
| jeblair | cool | 18:04 |
| mordred | jeblair: yah | 18:04 |
| jeblair | i just rejiggered the jobs at the bottom of the etherpad | 18:05 |
| clarkb | I've got to pop out now to prep for meeting | 18:05 |
| jeblair | clarkb: thanks! | 18:06 |
| *** electrofelix has quit IRC | 18:06 | |
| jeblair | i added 'devstack' as a pre-run item on the devstack job, thinking that lets anyone make a job that inherits from devstack so they have devstack all set up | 18:06 |
| jeblair | like the tempest job below it (which runs tempest in the main playbook) | 18:06 |
| jeblair | but i wonder if that's incompatible with devstack-gate pre hooks | 18:06 |
| jeblair | where to the pre-test hooks run | 18:07 |
| mordred | well- based on what you said earlier- let's think of it as two stacks - a legacy-d-g stack and a new-d-g stack | 18:07 |
| mordred | so the legacy one does the hooks thing - but the non-legacy one does not and just uses pre/post as v3 normally does | 18:08 |
| jeblair | it's pre_test_hook (normally empty); gate_hook (normally devstack and tempest, though tempest running is controlled by variable); then post_test_hook(normally empty) | 18:08 |
| jeblair | mordred: okay, so we may need a legacy base devstack job to satisfy the conversion script and usage of hooks | 18:09 |
| mordred | well- we can have a run-devstack role and a run-tempest role, and the run playbook for the devstack job only run the devstack role but the run playbook of the tempest job run both | 18:09 |
| mordred | jeblair: yah | 18:09 |
| jeblair | oh that's a good idea too | 18:10 |
| jeblair | i guess the devstack-in-pre job idea is mostly useful for openstack consumer jobs, like nodepool. not so much openstack service jobs. | 18:10 |
| jeblair | maybe we should do both things. | 18:11 |
| mordred | yah - and having it as a separate role lets us totally do that | 18:11 |
| mordred | like that maybe | 18:12 |
| jeblair | mordred: and devstack-consumer is what a nodepool-like job would inherit from? | 18:12 |
| mordred | yah | 18:12 |
| jeblair | cool, that makes sense | 18:12 |
| mordred | jeblair: we said earlier we can do the roles in openstack-zuul-jobs/zuul-jobs, but I think some of the above playbook composition does want to happen in the d-g repo - and should be fairly easy (easier than role decomp) to do in place | 18:15 |
| mordred | especially since a bunch of it is running ansible already anyway | 18:15 |
| jeblair | mordred: agreed | 18:15 |
| jeblair | mordred: i've moved our role list down to below line 37 | 18:16 |
| jeblair | i'm going to annotate it with repos | 18:16 |
| pabelanger | yes, that is helpful | 18:16 |
| mordred | jeblair: should the devstack and tempest roles eventually live in devstack and tempest rather than d-g? | 18:18 |
| jeblair | mordred: i would like that | 18:18 |
| jeblair | mordred: maybe that's a chat to have at the ptg? | 18:18 |
| mordred | ++ | 18:18 |
| mordred | for now, let's name them something with d-g inthe title so that if/when we move them to devstack and tempest we can do that without d-g inthe title and migrate with less rename steps | 18:19 |
| jeblair | okay, i think that captures the roles and jobs we need to write | 18:19 |
| jeblair | why the -gate suffix? | 18:19 |
| jeblair | oh i missed that line ^ | 18:19 |
| mordred | (from the above statement- future-proof for intent to move thenm) | 18:19 |
| mordred | we can also skip it though | 18:20 |
| mordred | ot | 18:20 |
| jeblair | we can try it that way | 18:21 |
| jeblair | do we want to stop now, or do folks want to start claiming todo items? | 18:23 |
| pabelanger | the first 2 steps in TODO, we can start now. The next also require adding d-g to zuulv3.o.o, is that something we want to do sooner then later? | 18:23 |
| jeblair | pabelanger: we can go ahead and add it i think | 18:24 |
| mordred | oh - I wrote that change last night actually... | 18:24 |
| mordred | https://review.openstack.org/493974 | 18:25 |
| mordred | oh 0 wait - didn't we say we wanted to make "install ssh keys on all hosts" its own role? | 18:26 |
| jeblair | how about this -- we all have other stuff on our plate, so rather than dividing all of that up right now, we leave the devstack-gate etherpad first-come fist-served. as you find time to work on the devstack-gate stuff, go to the etherpad and claim a thing. | 18:26 |
| mordred | jeblair: ++ | 18:26 |
| jeblair | mordred: see lines 42 and 45 | 18:26 |
| jeblair | mordred: i was thinking we add the private key to the existing role, but make root its own new role | 18:27 |
| mordred | jeblair: I'm going to add the above text about splitting devstack-vm-gate-wrap into some pieces | 18:27 |
| mordred | jeblair: ahhhh. I can't read | 18:27 |
| pabelanger | Ya, I want to get some eyes on 492671 first, that is are test secret for zuulv3-dev | 18:27 |
| jeblair | or i can't type :) | 18:27 |
| jeblair | mordred: fyi, i have 2 things that have pushed onto the top of my stack: 1) find out why zuulv3 gate pipeline doesn't work. 2) add unit tests for leaking secrets. | 18:28 |
| jeblair | i'm going to run out and grab some lunch before the next irc meeting. :) | 18:29 |
| mordred | jeblair: yes. my stack is to finish the docs/publish tasks that are on my plate and work on migration script | 18:30 |
| *** jkilpatr has quit IRC | 19:03 | |
| *** dkranz has quit IRC | 19:03 | |
| *** leifmadsen has quit IRC | 19:03 | |
| *** jkilpatr has joined #zuul | 19:03 | |
| *** dkranz has joined #zuul | 19:03 | |
| *** leifmadsen has joined #zuul | 19:04 | |
| fungi | i guess https://etherpad.openstack.org/p/AIFz4wRKQm is the thing i said i'd read through when i got home, judging from scrollback | 20:30 |
| fungi | oof, that's some content! looking forward to this | 20:31 |
| *** dkranz has quit IRC | 20:48 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Test that secrets don't leak into logs https://review.openstack.org/494015 | 20:54 |
| jeblair | pabelanger, mordred, clarkb, fungi: is that the sort of thing we're looking for? | 20:54 |
| pabelanger | jeblair: Wow, ya. I think that is what we want to do. Walk logs directory | 20:59 |
| clarkb | jeblair: based on the commit message yes | 20:59 |
| mordred | jeblair: that looks good - one quick suggestion - add a positive test to the same setup that is a playbook that writes the secret content to a flag file- so you can verify that 'test-password' is actually the content of the secret (and it's not just passing the test because of a name mismatch) | 21:08 |
| mordred | jeblair: I'm pretty sure that's not the case right now- but in case someone makes a change in the future | 21:08 |
| mordred | oh. wait | 21:08 |
| mordred | nevermind | 21:08 |
| mordred | I just read the place where you do that already | 21:09 |
| * mordred goes back into his hole | 21:09 | |
| mordred | also, your implementation of that idea is better than mine | 21:09 |
| jeblair | mordred: oh you had an implementation of that? | 21:10 |
| mordred | jeblair: no - my suggestion aboveof what to do - your test is better than my words | 21:10 |
| jeblair | oh ok cool | 21:10 |
| jeblair | and yeah, i think the test covers your suggestion (which is good) | 21:11 |
| mordred | jeblair: we still have to restart the scheduler on updates to main.yaml yeah? | 21:12 |
| jeblair | basically, the test verifies that 'test-password' shows up in exactly two files: the file the job is writing it out to, and the file that holds the decrypted variable for ansible | 21:12 |
| jeblair | mordred: yes, or issue a sighup to reconfigure | 21:12 |
| mordred | jeblair: I am saying things to you privately and I do not thik you are receiving them | 21:14 |
| jeblair | mordred: that is fascinating! i am indeed receiving nothing | 21:14 |
| mordred | jeblair: I continue to be able to see what you are typing | 21:16 |
| *** mordred has quit IRC | 21:27 | |
| *** mordred has joined #zuul | 21:27 | |
| fungi | jeblair: 492287 is great--thanks for putting that together? | 21:27 |
| fungi | also, i'm not caught up on scrollback in here... are the excess of RETRY_LIMIT events a known issue (hopefully)? | 21:27 |
| fungi | er, thanks for putting that together! (proper use of punctuation eludes me sometimes) | 21:27 |
| fungi | 489691 got me thinking... what is the current behavior if someone does attempt to (either accidentally or intentionally) define a job under a name which is already in use by another repo? does zuul refuse to test that change? fail? undefined behavior still? | 22:21 |
| fungi | okay, this is not so good... 494015 seems to have fallen into a check loop | 22:23 |
| fungi | zuul keeps leaving a +1 on it (instead of +2) and reenqueuing into the gate over and over | 22:24 |
| fungi | so not check loop i guess, but gate loop | 22:24 |
| fungi | i wonder if we just merged something which broke the gate pipeline config? | 22:25 |
| fungi | 494006 maybe? | 22:26 |
| fungi | last reconfig event for the scheduler on zuulv3.o.o was at 20:42:01 utc according to the debug log | 22:28 |
| fungi | 2017-08-15 20:42:01,696 INFO zuul.DependentPipelineManager: On success: [<zuul.driver.sql.sqlreporter.SQLReporter object at 0x7f68f98d6e10>, <zuul.driver.gerrit.gerritreporter.GerritReporter object at 0x7f68f98d6be0>] | 22:34 |
| fungi | 2017-08-15 22:21:04,937 DEBUG zuul.GerritReporter: Report change <Change 0x7f68f98f6908 494015,1>, params {'Verified': 2, 'submit': True}, message: Build succeeded (gate pipeline). | 22:34 |
| fungi | so why is gerrit recording it as a verified +1 and no submit? | 22:34 |
| * fungi checks zuul account perms | 22:35 | |
| fungi | it looks like the zuul account in gerrit is in the "continuous integration tools development" group which the all-projects acl only grants verify -1..+1 | 22:44 |
| fungi | and not in the "continuous integration tools" group which is granted -2..+2 (that currently contains only the jenkins account) | 22:44 |
| pabelanger | Ya, I don't think we ever changed that | 22:44 |
| fungi | i'll un-approve 494015 for now so that it hopefully ceases looping | 22:44 |
| pabelanger | IIRC, we did that first to avoid zuul +2 things behind the back of jenkins | 22:44 |
| fungi | makes sense. is there an acl change proposed for the repos we want zuul able to do this for? | 22:44 |
| pabelanger | I have not, but sounds like a great idea to dp | 22:44 |
| pabelanger | do* | 22:44 |
| fungi | we can always allow it on a per-repo basis for the moment (grant Verify -2..+2 and Submit to the Continuous Integration Tools Development group on the refs/heads/* ref block) | 22:44 |
| clarkb | does it make sense to just update the users global group membership at this point? | 22:44 |
| fungi | that's more or less what i'm asking | 22:44 |
| fungi | simple change is to add zuul to the Continuous Integration Tools group in gerrit (and optionally remove it from Continuous Integration Tools Development) | 22:44 |
| clarkb | I think if we are happy to have it gating and +2'ing and submitting some projects than we should be fine with it for any project | 22:44 |
| fungi | i'm cool with that, just looking to others for consensus | 22:44 |
| fungi | and also glad that what i found was not actually a bug in zuulv3, and also that there was sufficient detail in the logs for me to piece together the actual issue | 22:44 |
| mordred | fungi, clarkb: yay you found the gate issue! | 22:44 |
| fungi | heh | 22:44 |
| mordred | and yes, adding zuul to the group is the right choice | 22:44 |
| mordred | fungi: I'm thrilled you were able to diagnose that from logs! | 22:44 |
| fungi | i would have gotten there sooner, but have been so disconnected i didn't realize zuulv3.o.o hadn't _actually_ +2'd and submitted a change yet | 22:45 |
| fungi | for some reason i thought that was already happening as of late last week | 22:46 |
| fungi | so most of the confusion is me being late to the party (as usual) | 22:46 |
| clarkb | well I think its setup to work so unexpected that it didn't/doesnt | 22:48 |
| fungi | right, more that i assumed someone had approved something to those repos already since the switch ;) | 22:49 |
| mordred | clarkb, fungi: y'all ok with me going ahead and moving zuul into the CI TOols group? | 23:10 |
| clarkb | wfm | 23:10 |
| mordred | done | 23:10 |
| mordred | I have re-approved that change | 23:10 |
| mordred | and it is in gate ... | 23:11 |
| jeblair | begin 8 minute drumroll | 23:12 |
| SpamapS | https://www.youtube.com/watch?v=ian6NyXpszw | 23:19 |
| SpamapS | "drumroll please..." | 23:19 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Test that secrets don't leak into logs https://review.openstack.org/494015 | 23:20 |
| jeblair | \o/ | 23:20 |
| mordred | "Change has been successfully merged into the git repository by Zuul" \o/ | 23:21 |
| pabelanger | Yay | 23:43 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!