| *** JasonCL has joined #zuul | 00:12 | |
| *** JasonCL has quit IRC | 00:19 | |
| SpamapS | clarkb: correct | 00:27 |
|---|---|---|
| clarkb | SpamapS: ok just wanted to confirm as there are setups that would result in more than one handler thread that would need syncing between but in that setup you should only get the one handler if I've read the code correctly | 00:27 |
| clarkb | so much simpler to solve for this constrained case (in theory) | 00:27 |
| SpamapS | If splitting into 3 pools would solve my problem I can do that. :) | 00:28 |
| clarkb | I don't think it will | 00:28 |
| SpamapS | yeah I don't think so either | 00:28 |
| SpamapS | I am looking at reducing from 5 to 4 nodes | 00:28 |
| clarkb | in that case you'd have three handlers "fighting" to grab requests and fulfill them including for min ready | 00:28 |
| SpamapS | unfortunately we baked some assumptions in deep | 00:29 |
| clarkb | I expect the end results would be similar to what you currently experience now with the random assignment | 00:29 |
| mordred | it's definitely an interesting case | 00:30 |
| mordred | the ratio of quota to nodeset size and the azs existing is fun | 00:31 |
| clarkb | a hack you could do is similar to the one we are using with our multiple launchers. Basically configure one launcher with one az and have it with a >0 min ready | 00:32 |
| clarkb | then have a second (and possibly third) launcher for the remaining azs and no min ready configured in them | 00:32 |
| clarkb | you'd be constrained to servicing all min ready from a single az but that should make it consistent at least | 00:33 |
| SpamapS | we'll see how it goes with a patch to turn off AZ stickyness. | 00:33 |
| SpamapS | I think that will for the most part solve the lag issues I have. | 00:34 |
| *** JasonCL has joined #zuul | 00:36 | |
| *** JasonCL has quit IRC | 00:40 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add opensuse-tumbleweed to nodepool dsvm testing https://review.openstack.org/545183 | 01:05 |
| *** rlandy|rover is now known as rlandy|rover|bbl | 01:23 | |
| *** JasonCL has joined #zuul | 01:54 | |
| *** JasonCL has quit IRC | 02:00 | |
| corvus | mordred: the zuul_stream callback plugin handles both log lines from the remote host, and internally generated log lines (ie, announcing play start/stop). if we move the remote log receiver from zuul_stream into an executor subprocess, we'd be writing to the log file from two processes. since we could have multiple tasks starting and stopping simultaneously, we probably can't count on that being clean. | 02:02 |
| corvus | does that mean the component in the executor should be more like a proxy? it receives log lines from remote hosts, and zuul_stream fetches them from it and logs them? | 02:02 |
| corvus | mordred: or is there some other way to set that up....? | 02:02 |
| corvus | mordred: oh... zuul_stream could *send* it's internally generated log lines to the executor component. then the executor component is the only one writing them... | 02:04 |
| *** toabctl has quit IRC | 02:05 | |
| corvus | that seems obvious now that i think about it. then again, everything about the logging system seems obvious in retrospect. nothing seems obvious in prospect. :) | 02:06 |
| *** harlowja has quit IRC | 02:52 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 03:10 | |
| dmsimard | What's this "Open CI" thing I seem to keep hearing about ? Is that openlab ? | 04:01 |
| dmsimard | mrhillsman: ^ | 04:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Support ubuntu-bionic (18.04) with nodepool dsvm https://review.openstack.org/544574 | 04:12 |
| dmsimard | corvus: Just looked at the logs for that particular error. It has nothing to do with the zuul_json callback truncating the json file, unfortunately.. but does highlight another case of Ansible breaking the callback contract by passing a string instead of a bool. I'll try to figure out if it's been fixed in Ansible but in any case I'll have to work around it in ARA. | 04:14 |
| *** rlandy|rover has quit IRC | 04:24 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Support ubuntu-bionic (18.04) with nodepool dsvm https://review.openstack.org/544574 | 04:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Source /etc/ci/mirror_info.sh for mirrors https://review.openstack.org/546032 | 04:37 |
| SpamapS | corvus: it's just software. | 04:45 |
| *** harlowja has joined #zuul | 04:54 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul master: Make test-logs.sh more convenient to use https://review.openstack.org/546037 | 04:55 |
| dmsimard | Shrews: this one might interest you: http://paste.openstack.org/show/678121/ | 04:55 |
| dmsimard | ERROR zuul.Scheduler: Unable to process autohold for None | 04:55 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Support ubuntu-bionic (18.04) with nodepool dsvm https://review.openstack.org/544574 | 05:20 |
| *** bhavik1 has joined #zuul | 05:20 | |
| *** bhavik1 has quit IRC | 05:31 | |
| *** toabctl has joined #zuul | 06:08 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use a status code to detect unknown vs. missing tenant https://review.openstack.org/545879 | 06:39 |
| openstackgerrit | Merged openstack-infra/nodepool master: Hack for test_delete_now rare failures https://review.openstack.org/545982 | 06:43 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove Paste from the dependencies https://review.openstack.org/546000 | 06:47 |
| *** chrnils has joined #zuul | 06:47 | |
| *** threestrands has quit IRC | 07:10 | |
| *** harlowja has quit IRC | 07:10 | |
| *** JasonCL has joined #zuul | 07:22 | |
| *** JasonCL has quit IRC | 07:27 | |
| AJaeger | mordred, infra-root: http://zuul.openstack.org/jobs.html and http://zuul.openstack.org/builds.html are broken completely right now. I get a json output, not a nicely rendered page. | 07:38 |
| tobiash | feels like matrix | 08:34 |
| *** swest has quit IRC | 08:44 | |
| *** swest has joined #zuul | 08:45 | |
| *** jpena|off is now known as jpena | 08:46 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add regex support to project stanzas https://review.openstack.org/535713 | 08:49 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add winrm certificate handling https://review.openstack.org/535717 | 08:57 |
| AJaeger | tobiash, ianw and myself fixed the Zuul/nodepool integration tests, could you review the stacks starting at https://review.openstack.org/#/c/545163/ and https://review.openstack.org/#/c/545158/ , please? | 09:06 |
| tobiash | AJaeger: yes, can do that in a few minutes | 09:06 |
| AJaeger | thanks! | 09:08 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix builds queued forever after failure to get node request https://review.openstack.org/537335 | 09:08 |
| tobiash | AJaeger: +2 with comment on https://review.openstack.org/#/c/545158/14 | 09:17 |
| tobiash | AJaeger: stacks lgtm | 09:20 |
| *** hashar has joined #zuul | 09:22 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add regex support to project stanzas https://review.openstack.org/535713 | 09:26 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix nodepool-zuul-functional https://review.openstack.org/545163 | 09:30 |
| *** jimi_|ansible has joined #zuul | 09:34 | |
| *** jimi_|ansible has joined #zuul | 09:34 | |
| *** jimi|ansible has quit IRC | 09:35 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix nodepool-zuul-functional https://review.openstack.org/545158 | 09:51 |
| AJaeger | tobiash: thanks! | 09:58 |
| AJaeger | tobiash: put it on my todo list for cleanup later | 09:59 |
| tobiash | AJaeger: just a nit ;) | 10:00 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove install-distro step for nodepool integration https://review.openstack.org/545221 | 10:13 |
| *** elyezer has quit IRC | 10:18 | |
| *** tosky has joined #zuul | 10:19 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul master: Add nodepool-zuul-functional as non-voting check job https://review.openstack.org/545203 | 10:19 |
| openstackgerrit | Merged openstack-infra/nodepool master: Remove copying of legacy install-distro script https://review.openstack.org/545222 | 10:24 |
| openstackgerrit | Merged openstack-infra/nodepool master: Avoid tools/nodepool-integration-setup.sh from zuul https://review.openstack.org/545228 | 10:24 |
| *** elyezer has joined #zuul | 10:27 | |
| AJaeger | tobiash: could you readd your +1 to https://review.openstack.org/545203 , please? I rebased since I misread gerrit's note | 10:37 |
| AJaeger | +A I mean | 10:37 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove tools/nodepool-integration-setup.sh https://review.openstack.org/545229 | 10:53 |
| *** JasonCL has joined #zuul | 11:09 | |
| *** JasonCL has quit IRC | 11:32 | |
| *** JasonCL has joined #zuul | 11:33 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 11:37 |
| *** JasonCL has quit IRC | 11:38 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow operator to specify nodes TTL https://review.openstack.org/543403 | 11:44 |
| tobiash | AJaeger: done | 12:02 |
| *** elyezer has quit IRC | 12:15 | |
| *** elyezer has joined #zuul | 12:19 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add nodepool-zuul-functional as non-voting check job https://review.openstack.org/545203 | 12:22 |
| mordred | AJaeger: looking in to the builds/jobs dashboards | 12:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Extract an abstract base Parser class https://review.openstack.org/545610 | 12:33 |
| mordred | AJaeger: fixed in prod - patch coming | 12:36 |
| *** jpena is now known as jpena|lunch | 12:44 | |
| AJaeger | thanks, mordred and tobiash ! | 12:46 |
| *** toabctl has quit IRC | 13:02 | |
| *** dkranz has joined #zuul | 13:10 | |
| *** toabctl has joined #zuul | 13:23 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Only DeletedNodeWorker should delete nodes https://review.openstack.org/546146 | 13:32 |
| *** rlandy has joined #zuul | 13:33 | |
| *** rlandy is now known as rlandy|rover | 13:34 | |
| *** JasonCL has joined #zuul | 13:35 | |
| dmsimard | mordred: sorry if this is obvious but why are we messing with all these rewrite rules instead of having zuul-web as a proper wsgi app with URL routing ? | 13:37 |
| dmsimard | it sounds like we're putting a lot of logic into apache that should be built in the app | 13:38 |
| dmsimard | I'm not familiar with aiohttp but it does seem like there's the concept of URL routing available | 13:39 |
| *** JasonCL has quit IRC | 13:39 | |
| Shrews | mhu: 546146 is going to affect your held node cleanup code. Sorry about that | 13:40 |
| dmsimard | Shrews: that reminds me, did you see my message last night ? | 13:41 |
| mhu | Shrews, thanks for the heads-up, I'll look into it | 13:41 |
| Shrews | dmsimard: no | 13:41 |
| dmsimard | Shrews: came across this error all over zuul.log on the scheduler yesterday: http://paste.openstack.org/show/678121/ | 13:41 |
| dmsimard | thought you might know what was going on | 13:42 |
| Shrews | dmsimard: hrm, never seen that. not sure what's happening there | 13:42 |
| dmsimard | Shrews: it's hard to tell what exactly is going on, yeah -- I might submit a first patch to give us some insight on the exception because the logs are interleaved so it's a bit hard to tell what the exception relates to | 13:43 |
| *** jpena|lunch is now known as jpena | 13:44 | |
| dmsimard | corvus: in case you were curious, tristanC found my url_pattern/failure-pattern/success-pattern issue I asked about last week :/ https://github.com/softwarefactory-project/zuul-distgit/blob/master/0001-model-keep-jenkins-url-as-is.patch | 13:46 |
| Shrews | corvus: mordred: I'm sort of thinking we should remove the --now option to 'nodepool delete'. Because the delete thread runs really quickly, there's not any need that I can see to say | 13:47 |
| Shrews | "do it now" | 13:47 |
| Shrews | If we remove it, then we totally eliminate that test_delete_now race. | 13:48 |
| Shrews | current delete thread interval is 5 seconds | 13:49 |
| dmsimard | Shrews: never realized we got rid of --now in v3 | 13:49 |
| Shrews | dmsimard: we didn't, which is my point | 13:49 |
| Shrews | i think we should get rid of it | 13:49 |
| *** JasonCL has joined #zuul | 13:50 | |
| *** JasonCL has quit IRC | 13:51 | |
| mordred | dmsimard: we're working towards less rewrite rules - there's a few things to untangle, but hopefully this should all be sane by the end of this week | 13:53 |
| *** JasonCL has joined #zuul | 13:53 | |
| dmsimard | mordred: ok, ideally the webserver should be as dumb as possible :) | 13:53 |
| dmsimard | facilitates implementation with other webservers/proxies if people don't like apache or whatnot | 13:54 |
| mordred | yah - the one thing we'll have rewrite rules for (which will be optional/an optimization) is offloading serving the static html/javascript we produce from webpack | 13:54 |
| mordred | so there should be like, one rewrite rule | 13:54 |
| dmsimard | are you planning on packaging the js/css stuff like horizon ? with xstatic ? | 13:55 |
| dmsimard | or will there always be an expectation that there's a puppet or ansible to mash the two together ? | 13:55 |
| dmsimard | (I guess the RPM packaging from software factory makes this kind of redundant) | 13:56 |
| mordred | dmsimard: but yes - one of the reasons this javascript stack has been slow going in is that we want to make sure that everything works if you don't have any apache, if you want to off-load static assets to apache but proxy dynamic api calls to zuul-web, if you want to do the second thing but whitelabel a specific tenant - and if you want to put the static web assets somewhere else (like swift) | 13:56 |
| mordred | dmsimard: HELL NO (to xstatic) | 13:56 |
| mordred | dmsimard: xstatic is a plague that I find very offensive (there is no reason to use python tools to package javascript when the javascript ecosystem has their own set of tools) | 13:57 |
| mordred | dmsimard: BUT - totally will package the html/javascript into an installable bundle | 13:57 |
| *** electrofelix has quit IRC | 13:57 | |
| dmsimard | mordred: yeah, I don't think it's pretty either but doing "pip install" and getting everything installed without having to use npm or whatever is pretty neat | 13:57 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Remove --now option to 'delete' CLI command https://review.openstack.org/546152 | 13:58 |
| Shrews | ^^^ WIP for discussion | 13:58 |
| *** JasonCL has quit IRC | 13:58 | |
| mordred | dmsimard: yes. pip install zuul will get you the web content | 13:58 |
| *** JasonCL has joined #zuul | 13:58 | |
| dmsimard | mordred: how? my understanding is that currently we need shenanigans like https://github.com/openstack-infra/puppet-zuul/blob/master/manifests/web.pp#L164 | 13:59 |
| mordred | dmsimard: none of the javascript tooling stack is landed yet - that's this week :) | 13:59 |
| dmsimard | mordred: oh, so I'm just looking at old stuff, okay :p | 13:59 |
| mordred | dmsimard: so- the javascript stack will use npm/webpack to generate the html bundles which it writes to zuul/web/static ... that way if we run the js build before running setup.py sdist then the bult js files will be inside the python package and thus pip install will work | 14:00 |
| mordred | dmsimard: however, if we wanted to deploy the static assets independently, we can grab the js bundle produced by https://review.openstack.org/#/c/538131/9 | 14:01 |
| mordred | and just untar it in an apache folder (like we do for storyboard-webclient)- or even publish it to swift if we got really crazy :) | 14:01 |
| mordred | dmsimard: tl;dr - COMPLETELY agree with you on the rewrite rules :) | 14:01 |
| *** sshnaidm|off is now known as sshnaidm | 14:05 | |
| tobiash | mordred: is your pending dashboard fix easy? | 14:09 |
| tobiash | mordred: I'm currently rebasing my deployment branch | 14:09 |
| tobiash | should I base from before the web patches from yesterday or just wait for your fix? | 14:10 |
| mordred | tobiash: which pending dashboard fix? | 14:11 |
| tobiash | mordred: "<mordred> AJaeger: fixed in prod - patch coming" | 14:11 |
| tobiash | from today | 14:11 |
| mordred | tobiash: oh! yah - that was just a fix to our puppet ... one sec | 14:11 |
| mordred | tobiash: https://review.openstack.org/546134 was the issue | 14:12 |
| tobiash | ah, so no zuul patch | 14:12 |
| mordred | tobiash: nope - just got rewrite rule order wrong | 14:12 |
| tobiash | that's why I didn't see it :) | 14:12 |
| mordred | :) | 14:12 |
| tobiash | ok my rewrite rules are simpler so this shouldn't be an issue for me :) | 14:13 |
| mordred | dmsimard: thanks for reminding me - the javascript stack doesn't actually update the tarball build/publication job to include the javascript - I should fix that | 14:13 |
| *** JasonCL has quit IRC | 14:28 | |
| *** JasonCL has joined #zuul | 14:29 | |
| rcarrillocruz | \o/ for getting static stuff with pip install | 14:49 |
| rcarrillocruz | do I read right that by default aiohttp will serve static ? last time i checked, i couldn't figure out how to change the root folder in aiohttp/zuul-web, like it was hardcoded and thus relative to the zuul package install path | 14:52 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Remove --now option to 'delete' CLI command https://review.openstack.org/546152 | 14:54 |
| mordred | rcarrillocruz: yah - once the javascript stack is in, you should not have to mess with the root folder location | 14:57 |
| rcarrillocruz | sweet | 15:00 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 15:07 |
| *** JasonCL has quit IRC | 15:08 | |
| *** JasonCL has joined #zuul | 15:08 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: WIP: allow disabling sticky AZs https://review.openstack.org/546175 | 15:10 |
| *** JasonCL has quit IRC | 15:11 | |
| *** JasonCL has joined #zuul | 15:11 | |
| SpamapS | ^^ no idea if it works | 15:12 |
| SpamapS | will be playing with it soon | 15:12 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 15:14 |
| dmsimard | mordred: yay | 15:16 |
| tobiash | SpamapS: it will work partly | 15:18 |
| tobiash | SpamapS: with this change it will take any ready nodes, but will still stick to a randomly self chosen az for nodes it creates | 15:20 |
| tobiash | (see review) | 15:20 |
| tobiash | hrm, this will interfere with the azs defined by a pool if any | 15:22 |
| tobiash | SpamapS: maybe it's better to name it disable-az-handling or something like that and then just bypass any az handling? | 15:22 |
| SpamapS | Well I don't mind if it creates the new ones all in a single AZ. | 15:50 |
| SpamapS | I just want it to take whatever's ready. | 15:50 |
| SpamapS | But I can see where it might be less confusing if it just stuck to random allocation all the time. | 15:51 |
| *** hashar has quit IRC | 16:44 | |
| *** chrnils has quit IRC | 16:59 | |
| corvus | mordred: i'll sign up for doing the executor log receiver thing. i'll make it a followup patch to your wip stack, so i shouldn't interfere with that if you update it. | 17:04 |
| *** dkranz has quit IRC | 17:07 | |
| mordred | corvus: cool - let me know if the code there doesn't make any sense... but I *think* it should be fairly easy for your brain | 17:08 |
| *** harlowja has joined #zuul | 17:23 | |
| pabelanger | Hmm, should nodepool-dsvm test jobs be using upper-constraints for diskimage-builder? First time I've noticed that | 17:24 |
| pabelanger | http://logs.openstack.org/74/544574/11/check/nodepool-functional-py35/1041e03/job-output.txt.gz#_2018-02-20_15_33_39_671125 | 17:25 |
| pabelanger | also, we appear to be using python2.7 pip | 17:25 |
| clarkb | pabelanger: devstack's pip install routines only use constraints | 17:27 |
| clarkb | as for python2 vs python3 you need to set the use python3 flag | 17:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Install nodepool with python3 for dvsm job https://review.openstack.org/546254 | 17:29 |
| pabelanger | clarkb: does that work?^ | 17:29 |
| clarkb | pabelanger: I would just set it for the job as a whole | 17:29 |
| clarkb | openstack passes tempest under python3, just diasble swift | 17:30 |
| pabelanger | k | 17:30 |
| clarkb | and I'm not sure if that will work. depends if the python3 setup stuff happens on the fly when necessary or if it needs to do it early on | 17:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Use python3 for devstack testing https://review.openstack.org/546254 | 17:38 |
| pabelanger | k, I think that should work | 17:38 |
| pabelanger | I'll see what is needed to bump DIB in upper-requirements | 17:38 |
| clarkb | pabelanger: it doesnt need to be in upper reuirements does it? | 17:39 |
| pabelanger | clarkb: not sure, its there today | 17:39 |
| pabelanger | I don't know why | 17:39 |
| clarkb | oh I see | 17:39 |
| pabelanger | not sure the process to remove something from upper-constraints | 17:39 |
| clarkb | check with ianw I seem to recall that maube that was done for dib testing of some sort | 17:40 |
| pabelanger | kk | 17:43 |
| corvus | mordred: what do you think about having only one log receiver per job, and having an extra argument to the command module to tell it what 'host' it's on, then adding the host as extra data to the log line, then having the receiver pull that out when doing the string formatting (a la the timestamp)? | 17:50 |
| corvus | mordred: (as opposed to more or less the current structure, which is one receiver per inventory host, and the receiver internally knows which host its for) | 17:50 |
| corvus | mordred: that means we don't need to try to map inventory hosts to log receiver socket paths, etc... | 17:52 |
| *** dkranz has joined #zuul | 17:54 | |
| mordred | corvus: yes. I think that is exactly right | 17:59 |
| corvus | kk | 17:59 |
| *** openstackgerrit has quit IRC | 18:03 | |
| *** jpena is now known as jpena|off | 18:18 | |
| *** harlowja has quit IRC | 18:23 | |
| *** tosky has quit IRC | 18:24 | |
| SpamapS | tobiash: I'm curious if you find yourself holding nodes a lot for users? | 18:32 |
| SpamapS | I've been noodling on making a hold-manager.. | 18:32 |
| SpamapS | Let users do their own holds. | 18:32 |
| SpamapS | and clean them up more easily | 18:33 |
| SpamapS | Because it turns out it's pretty useful to have an automatic dev environment maker. | 18:33 |
| SpamapS | That makes VMs that are identical to your eventual CI tests. | 18:33 |
| SpamapS | But it's a little janky.. | 18:33 |
| SpamapS | set autohold... bounce ssh.. clean up nodes manually... | 18:34 |
| clarkb | SpamapS: one idea I've had with that is resurrecting plans to make our images more public. You might consider having nodepool makr your images as public in your cloud then users can boot on demand? | 18:34 |
| SpamapS | ehh | 18:34 |
| SpamapS | The zuul variable is just as important | 18:34 |
| SpamapS | and the trees in the state | 18:34 |
| SpamapS | clarkb: that said.. | 18:34 |
| SpamapS | If I could have a 'zuul-executor --local --change XX' ..... | 18:34 |
| SpamapS | that would be incredibly helpful | 18:34 |
| clarkb | yes, being able to build the tree locally would be useful | 18:35 |
| SpamapS | Like, basically let users make their own VMs and inventory, and then let zuul-executor operate on them. | 18:35 |
| SpamapS | but I keep coming around to the fact that it's not really about who owns the VMs in OpenStack.. I just need to be able to SSH to them, and delete them when I'm done. | 18:36 |
| clarkb | one thing we've found though is that a lot of the problems with people understanding test failures have less to do with the git tree and more to do with bad assumptions around how clouds or cloud instances work | 18:36 |
| SpamapS | Oh, that's not my case.. :) | 18:36 |
| clarkb | ah ok | 18:36 |
| clarkb | for us that tends to be the case | 18:36 |
| SpamapS | We are missing stuff in the output that helps you debug why a job failed.. | 18:36 |
| SpamapS | It gets better and better. | 18:36 |
| clarkb | "Oh! this instance has two network interfaces and neither are called eth0!" | 18:36 |
| SpamapS | But sometimes I just need to log in and mess with stuff. | 18:36 |
| SpamapS | I have as many fails because of misunderstandings of how Ansible works than anything. :) | 18:37 |
| SpamapS | But really, a lot of times we just hold nodes to debug and inspect. | 18:37 |
| clarkb | ara has been extremely useful for us in debugging the ansible aspects | 18:38 |
| SpamapS | "WTF? Why?" --> hold.... play... fix... push | 18:38 |
| clarkb | thank you dmsimard! | 18:38 |
| SpamapS | Definitely. harlowja just got it working in our CI. | 18:38 |
| tobiash | SpamapS: I didn't hold any node yet for my users | 18:38 |
| SpamapS | one problem we're having is we're using ARA in prod and it exposes passwords. :-/ | 18:38 |
| SpamapS | Unless you use the giant no_log hammer. | 18:38 |
| tobiash | SpamapS: but we're currently in process of ramping up the number of users ov zuulv3 ;) | 18:38 |
| SpamapS | One reason I'm holding nodes too, is I'm not developing like, microservices. | 18:39 |
| SpamapS | We're developing kolla-ansible automation glue. | 18:39 |
| SpamapS | So we are doing a lot of like, why can't this node talk to that node. | 18:39 |
| tobiash | but holding nodes as a service usable by the normal users will save me a lot of time in the future probably | 18:40 |
| SpamapS | anyway... I think we can make holds better.. but I like the idea of having a way to point a local zuul executable at a repo and have it roughly duplicate what happens inside zuul. | 18:40 |
| SpamapS | Another idea I have is something like Depends-On, but Hold-key: {{ Put an SSH Key Here }} | 18:40 |
| SpamapS | or in a comment | 18:41 |
| SpamapS | like recheck-with-hold {{ key }} | 18:41 |
| tobiash | lol, that would at least be easy to use :) | 18:41 |
| SpamapS | and then have zuul like, automatically clean the held nodes up if the PR merges. | 18:41 |
| SpamapS | but that does start to get scary... quotas and used nodes.. | 18:42 |
| SpamapS | making it too easy would be bad. ;) | 18:42 |
| SpamapS | so yeah.. stuff to think about | 18:42 |
| pabelanger | clarkb: Shrews: https://review.openstack.org/546254/ is an easy review for python3 nodepool. Confirmed to be working now in devstack logs | 18:48 |
| dmsimard | SpamapS: passwords where ? how ? | 18:51 |
| dmsimard | clarkb: yay ara \o/ | 18:51 |
| pabelanger | tobiash: danke | 18:52 |
| *** harlowja has joined #zuul | 18:55 | |
| *** openstackgerrit has joined #zuul | 18:57 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/nodepool master: Refactor playbooks/nodepool-zuul-functional/pre.yaml https://review.openstack.org/546272 | 18:57 |
| AJaeger | tobiash: is that what you suggested? ^ | 18:57 |
| *** dkranz has quit IRC | 18:58 | |
| *** harlowja_ has joined #zuul | 18:59 | |
| *** harlowja has quit IRC | 18:59 | |
| tobiash | AJaeger: yes, a role for a single task normally seems a bit odd but that makes ordering clear on the first sight :) | 19:02 |
| openstackgerrit | Merged openstack-infra/nodepool master: Use python3 for devstack testing https://review.openstack.org/546254 | 19:03 |
| corvus | SpamapS: https://storyboard.openstack.org/#!/story/2000801 | 19:04 |
| dmsimard | SpamapS: I need to step away for a bit but I'm curious to dig into your ARA password thing, let's chat later. ARA picks up whatever Ansible sends to it which should not be very much different than what would be displayed in the console. | 19:05 |
| pabelanger | easy review on zuul to clean up bindep.txt https://review.openstack.org/544569/ | 19:41 |
| *** tobiash has quit IRC | 19:50 | |
| EmilienM | it would be awesome to have branch filtering support on http://zuul.openstack.org/builds.html | 19:53 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul web: add admin endpoint, enqueue & autohold commands https://review.openstack.org/539004 | 19:54 |
| corvus | EmilienM: should not be difficult to add | 19:55 |
| *** tobiash has joined #zuul | 19:55 | |
| *** JasonCL has quit IRC | 20:03 | |
| *** JasonCL has joined #zuul | 20:04 | |
| *** JasonCL_ has joined #zuul | 20:07 | |
| *** JasonCL has quit IRC | 20:09 | |
| *** JasonCL_ has quit IRC | 20:11 | |
| openstackgerrit | Merged openstack-infra/zuul master: Ensure only python3 is installed with bindep.txt https://review.openstack.org/544569 | 20:19 |
| *** dmellado has quit IRC | 20:58 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add additional builder debug logging https://review.openstack.org/546303 | 21:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Start log receiver in executor https://review.openstack.org/546304 | 21:09 |
| corvus | mordred: ^ untested, but i think that's the bulk of the effort | 21:10 |
| mordred | corvus: woot | 21:20 |
| corvus | mordred: i'm assuming you'll squash the series at some point | 21:28 |
| SpamapS | dmsimard: items with passwords show in ARA output. | 21:29 |
| dmsimard | SpamapS: so they'd show in the console output too, right ? What ends up showing depends on the module, really.. what module have you seen this happen with ? | 21:30 |
| SpamapS | dmsimard: kolla_docker | 21:33 |
| SpamapS | dmsimard: also we see them in the invocation | 21:33 |
| SpamapS | with verbose==2, which I'd expect not to include invocation | 21:34 |
| dmsimard | SpamapS: I skimmed https://github.com/openstack/kolla-ansible/blob/master/ansible/library/kolla_docker.py real quick and didn't see any special treatment regarding passwords.. they should address that to at the very least leave the sensitive information out | 21:37 |
| SpamapS | dmsimard: yeah I'm not necessarily blaming ARA :) | 21:44 |
| SpamapS | it's just the thing that I caught the leaks with | 21:44 |
| dmsimard | SpamapS: I wonder if no_log could be improved to just filter out a list of fields (in upstream ansible) | 21:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Start log receiver in executor https://review.openstack.org/546304 | 21:51 |
| dmsimard | SpamapS: the thing that sucks is that, ultimately, the "respect" of no_log is left up to the callbacks | 21:53 |
| dmsimard | SpamapS: the data from the module is sent to the callbacks unfiltered and then the "burden" is on the callback to filter the data out | 21:53 |
| dmsimard | This lead to some interesting discussions, there was a CVE for it but nothing really came of it https://bugzilla.redhat.com/show_bug.cgi?id=1440912 | 21:54 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Correct hyphenation in host/group vars https://review.openstack.org/546317 | 21:54 |
| clarkb | ya ansible pushes a lot of this top level logic down into the actual module implementations | 21:55 |
| clarkb | retry for sync was broken due to this | 21:55 |
| corvus | tobiash, pabelanger: ^ can we merge https://review.openstack.org/546317 asap? that slipped through review | 21:55 |
| pabelanger | +2 | 21:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove support for erroneous host_vars and group_vars https://review.openstack.org/546318 | 21:57 |
| pabelanger | corvus: test fixtures also need updating, but assume that will happen once we remove support for host_vars / group-vars | 21:57 |
| corvus | pabelanger: yeah i did it in that second patch ^ | 21:58 |
| pabelanger | wfm | 21:58 |
| corvus | clarkb: it might be late for tobiash; can you +3 https://review.openstack.org/546317 asap? that slipped through review | 21:58 |
| corvus | 21:57 < pabelanger> +2 | 21:58 |
| corvus | gah | 21:58 |
| corvus | clarkb: it might be late for tobiash can you +3 https://review.openstack.org/546317 ? | 21:58 |
| dmsimard | clarkb: it's kind of a weird approach.. best practices (e.g, https://cwe.mitre.org/data/definitions/212.html ) basically say that sensitive data should not be sent in the first place | 21:58 |
| clarkb | corvus: yup done | 21:58 |
| dmsimard | clarkb: modules are executed remotely and callbacks are executed locally so it's quite a large surface area -- the problem is (and the nature of that CVE) that callbacks are not necessarily trusted.. an unprivileged user could drop a callback in a path loaded by Ansible but by then I guess you could assume it's a local exploit and if you have local exploit you can do worse things.. | 22:00 |
| corvus | clarkb, pabelanger: it looks like nothing using that has landed yet, so how about we go ahead and merge the second patch | 22:04 |
| clarkb | ah ok so no jobs at all merged yet? | 22:04 |
| pabelanger | +1, I won't be able to help with restart. Getting called away from computer | 22:05 |
| *** AJaeger has quit IRC | 22:06 | |
| *** dmellado has joined #zuul | 22:18 | |
| *** AJaeger has joined #zuul | 22:19 | |
| openstackgerrit | Merged openstack-infra/zuul master: Correct hyphenation in host/group vars https://review.openstack.org/546317 | 22:19 |
| *** threestrands has joined #zuul | 22:23 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove support for erroneous host_vars and group_vars https://review.openstack.org/546318 | 22:39 |
| clarkb | corvus: ^ we wait for puppet to apply that then are ready? | 22:40 |
| clarkb | also does that need to have executors restarted? | 22:40 |
| corvus | clarkb: just scheduler | 22:40 |
| *** openstackgerrit has quit IRC | 23:04 | |
| *** openstackgerrit has joined #zuul | 23:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't store references to secret objects from jobs https://review.openstack.org/546428 | 23:10 |
| *** JasonCL has joined #zuul | 23:10 | |
| *** JasonCL has quit IRC | 23:19 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!