| *** JasonCL has joined #zuul | 00:05 | |
| *** JasonCL has quit IRC | 00:12 | |
| *** JasonCL has joined #zuul | 00:20 | |
| *** JasonCL has quit IRC | 00:36 | |
| *** JasonCL has joined #zuul | 00:37 | |
| *** JasonCL has quit IRC | 01:29 | |
| *** robled has quit IRC | 02:58 | |
| *** robled has joined #zuul | 02:59 | |
| *** robled has quit IRC | 02:59 | |
| *** robled has joined #zuul | 02:59 | |
| *** bhavik has joined #zuul | 06:02 | |
| *** threestrands_ has joined #zuul | 06:26 | |
| *** threestrands has quit IRC | 06:34 | |
| *** yolanda has quit IRC | 06:34 | |
| *** xinliang has quit IRC | 06:34 | |
| *** dmsimard has quit IRC | 06:34 | |
| *** xinliang has joined #zuul | 06:40 | |
| *** yolanda has joined #zuul | 06:41 | |
| *** dmsimard has joined #zuul | 06:42 | |
| *** bhavik has quit IRC | 06:49 | |
| *** sshnaidm has joined #zuul | 07:11 | |
| *** threestrands_ has quit IRC | 07:13 | |
| *** jpena|off is now known as jpena | 08:14 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add new tox-remote job https://review.openstack.org/551301 | 08:22 |
|---|---|---|
| tobiash | clarkb, corvus, mordred: that should be ready now ^ | 08:23 |
| *** electrofelix has joined #zuul | 08:36 | |
| *** JasonCL has joined #zuul | 09:32 | |
| *** JasonCL has quit IRC | 09:38 | |
| *** JasonCL has joined #zuul | 09:49 | |
| *** hashar has joined #zuul | 09:51 | |
| *** JasonCL has quit IRC | 09:58 | |
| *** JasonCL has joined #zuul | 10:22 | |
| *** JasonCL has quit IRC | 10:54 | |
| *** JasonCL has joined #zuul | 10:58 | |
| *** JasonCL has quit IRC | 11:02 | |
| tobiash | oh, just had an idea to improve that a bit | 11:25 |
| *** JasonCL has joined #zuul | 11:34 | |
| *** JasonCL has quit IRC | 11:41 | |
| hughsaunders | electrofelix: pong | 11:42 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add new tox-remote job https://review.openstack.org/551301 | 11:53 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add new tox-remote job https://review.openstack.org/551301 | 11:54 |
| tobiash | clarkb, corvus, mordred: that injects a fixture dir into the bwrap context which we can use for the forbidden source tests ^ | 11:55 |
| electrofelix | hughsaunders: hiya, just wanted to sync a bit on the nodepool plugin | 11:59 |
| hughsaunders | electrofelix: ok :) | 11:59 |
| hughsaunders | Been making some progress... currently faffing with JDK installation. | 11:59 |
| electrofelix | turns out from my initial look at a zuul trigger plugin, that really need make the nodepool a dependency of it, as it becomes overly complex for no added gain trying to make zuul <--> jenkins work together without nodepool | 12:00 |
| hughsaunders | I'll push what I've got, its still super rough. | 12:01 |
| electrofelix | that's ok, just looking to understand where I can help | 12:03 |
| electrofelix | trying to build a plan for time to be dedicated here to work on the plugin to deliver nodepoolv3 <--> jenkins as a first step | 12:04 |
| hughsaunders | electrofelix: https://github.com/rcbops/nodepool-plugin/pull/7 | 12:06 |
| hughsaunders | Thats the current latest, have a poke at the head of the branch. | 12:07 |
| hughsaunders | I'd be happy to move it over into the openstack namespace, get more eyes and use gerrit etc. | 12:07 |
| hughsaunders | But its not in a useable form yet | 12:07 |
| electrofelix | thanks, looking over it now | 12:12 |
| *** myoung has joined #zuul | 12:17 | |
| hughsaunders | electrofelix: the docker-compose file in the docker dir is a good way to experiment, that combine with mvn hpi:run. | 12:18 |
| *** myoung is now known as myoung|ruck | 12:18 | |
| hughsaunders | but see the readme in the docker/ dir | 12:18 |
| *** dkranz has joined #zuul | 12:20 | |
| electrofelix | will look to use that, might have to make a few tweaks to use it for some static slaves as that will be our starting point ;-) | 12:22 |
| hughsaunders | :) | 12:26 |
| *** rlandy has joined #zuul | 12:31 | |
| *** JasonCL has joined #zuul | 12:35 | |
| *** sshnaidm has quit IRC | 12:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade to webpack 4 https://review.openstack.org/551987 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Express the bootstrap css depend in css https://review.openstack.org/551988 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Split status and stream into typescript modules https://review.openstack.org/551989 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add typing to getSourceUrl https://review.openstack.org/551990 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break build list out into its own module https://review.openstack.org/551991 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break job out into its own module https://review.openstack.org/551993 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break job list out into its own module https://review.openstack.org/551994 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break tenant list out into its own module https://review.openstack.org/551995 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break project detail and list out into their own module https://review.openstack.org/551996 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Move webpack html template to web/config https://review.openstack.org/551997 | 12:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate webpack config to typescript https://review.openstack.org/551998 | 12:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Rename javascript package to zuul-dashboard https://review.openstack.org/551999 | 12:40 |
| * mordred is not actually here today - but had some fun on the airplane yesterday I thought I'd push up | 12:44 | |
| *** sshnaidm has joined #zuul | 12:45 | |
| tobiash | must have been a long flight | 12:58 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/projects routes https://review.openstack.org/550979 | 13:00 |
| mordred | tobiash: :) | 13:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Rename javascript package to zuul-dashboard https://review.openstack.org/551999 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page https://review.openstack.org/537870 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift html templates into components https://review.openstack.org/551327 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use arrow functions for http callbacks https://review.openstack.org/551399 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade to webpack 4 https://review.openstack.org/551987 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Express the bootstrap css depend in css https://review.openstack.org/551988 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Split status and stream into typescript modules https://review.openstack.org/551989 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add typing to getSourceUrl https://review.openstack.org/551990 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break build list out into its own module https://review.openstack.org/551991 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break job out into its own module https://review.openstack.org/551993 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break job list out into its own module https://review.openstack.org/551994 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break tenant list out into its own module https://review.openstack.org/551995 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break project detail and list out into their own module https://review.openstack.org/551996 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Move webpack html template to web/config https://review.openstack.org/551997 | 13:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate webpack config to typescript https://review.openstack.org/551998 | 13:07 |
| mordred | ok. that's it for me for the day | 13:07 |
| tobiash | mordred: have fun with whatever you'll be doing | 13:07 |
| tristanC | could we make the zuul_stream refactor a post-3.0 thing and release the current master head soon? | 13:08 |
| tristanC | it seems like we lost mordred to a javascript frenzy :-) | 13:11 |
| tobiash | jlk: around? | 13:23 |
| tobiash | I think github3.py broke us | 13:23 |
| tobiash | http://paste.openstack.org/show/698721/ | 13:23 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add new tox-remote job https://review.openstack.org/551301 | 13:48 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Temporarily fork github3.py https://review.openstack.org/552028 | 14:13 |
| tobiash | jlk, corvus: do we have a better strategy than that ^ ? | 14:13 |
| tobiash | until there is a github3.py release | 14:13 |
| *** jpena is now known as jpena|lunch | 14:14 | |
| tristanC | tobiash: another strategy would be to clone a working version into a zuul "vendor" directory | 14:16 |
| clarkb | or just pin it in requirements | 14:17 |
| tobiash | I vote against the vendor directory | 14:17 |
| tobiash | that's 74M extra | 14:18 |
| clarkb | (you can put a hash in the url used in requirements I think and then specify it to be before the breakage) | 14:18 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Pin github3.py revision https://review.openstack.org/552032 | 14:20 |
| tobiash | clarkb: like that ^ ? | 14:20 |
| hughsaunders | electrofelix: pushed another commit to that PR, then merged it, so master is now where its at. | 14:20 |
| clarkb | tobiash: ya | 14:22 |
| corvus | tristanC: the main thing about zuul_stream that i was concerned about releasing with was the lack of testing -- if someone finds a bug (and folks have) we can't land a fix because there are no tests (every time we try, we break something else). however, the new tox-remote job could probably be used to get the extra testing we need there, so if we can make some tests around that in the next couple days, i | 14:26 |
| corvus | think we could release with that and do the refactor later (with benefit of tests!) | 14:26 |
| corvus | tobiash, clarkb: i think we were agreed that if github3.py isn't released before we're ready, we would vendor it | 14:27 |
| tobiash | corvus: means vendor it pulling it into zuul or mirroring that in openstack? | 14:28 |
| corvus | tobiash, clarkb: i guess we could pin for now and still hope it's released before we're ready | 14:28 |
| tobiash | this lib is huge | 14:28 |
| corvus | tobiash: pulling it into the zuul repo. that way we don't have to release a fork | 14:28 |
| corvus | tobiash: maybe we don't pull it all in :) | 14:28 |
| tobiash | corvus: ok, without tests and git it's 928K | 14:30 |
| corvus | tobiash: 73M is tests | 14:30 |
| corvus | ya | 14:30 |
| tobiash | with that I can live | 14:30 |
| corvus | tobiash: is there a pr/issue against github3.py for the breakage? | 14:31 |
| tobiash | not yet | 14:31 |
| tobiash | I just had the time to confirm that the last merged pr broke us | 14:31 |
| tobiash | it's seems to be the huge refactoring jlk was helping with | 14:32 |
| tobiash | corvus, clarkb: I think the tox-remote job should be ready now | 14:33 |
| electrofelix | hughsaunders: great, it'll probably be a couple more weeks before I can be actively involved aiming for starting in 3-4 weeks | 14:33 |
| hughsaunders | electrofelix: cool, hopefully we'll have the basic functionality going by then | 14:33 |
| hughsaunders | electrofelix: I just had the first job execution :D | 14:33 |
| hughsaunders | electrofelix: but still lots to do around node cleanup | 14:34 |
| electrofelix | every step brings us closer :D | 14:35 |
| corvus | tobiash, clarkb: i think there's a misunderstanding about the jobdir cleanup | 14:38 |
| corvus | tobiash, clarkb: see my comment on https://review.openstack.org/551301 | 14:38 |
| tobiash | corvus: responded to the first point | 14:41 |
| corvus | tobiash: ok. i think that's fine for now. the main downside is it could be fragile and need updating if bwrap is refactored | 14:42 |
| tobiash | corvus: shall I stick it with a todo? | 14:42 |
| tobiash | maybe we anyway want to be able to mount stuff into different paths in the future | 14:43 |
| corvus | tobiash: nah, i don't know what we'd do. i'd just leave it | 14:43 |
| tobiash | k | 14:43 |
| corvus | tobiash: i don't think so -- that would mean we'd depend on bwrap functionality, could hurt our ability to use other systems. at least, i wouldn't want to do it without careful thought. | 14:43 |
| tobiash | ah right, that needs bind | 14:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add new tox-remote job https://review.openstack.org/551301 | 14:45 |
| tobiash | the gh3 pin is green, shall we go with this and then vendor it or directly vendor it? | 14:48 |
| corvus | tobiash: let's go with the pin for now and continue to push off vendoring until the last minute | 14:49 |
| tobiash | :) | 14:49 |
| corvus | i just approved the pin | 14:49 |
| tobiash | thanks :) | 14:49 |
| *** jpena|lunch is now known as jpena | 15:09 | |
| openstackgerrit | Merged openstack-infra/zuul master: Pin github3.py revision https://review.openstack.org/552032 | 15:10 |
| kklimonda | is it technically possible to gate with 2 zuul instances running different configuration and jobs, sharing only gerrit (and projects)? | 15:14 |
| clarkb | not to gate | 15:15 |
| kklimonda | I'd prefer if an answer to that was no | 15:15 |
| clarkb | but you can have multiple zuuls providing infromation to gerrit. But only one of them can gate | 15:15 |
| kklimonda | only one can do the final submit, right? | 15:15 |
| tobiash | yes | 15:15 |
| kklimonda | and it is possible to require +2 votes from two different gerrit users, just a matter of configuration of the "primary" zuul | 15:16 |
| clarkb | in that setup you'd have race issues if primary finished first and couldn't submit | 15:17 |
| clarkb | its probably hackable but would feel clunky at times | 15:17 |
| kklimonda | would there be any risk of two zuul schedulers building a different job graph from the same event stream? | 15:19 |
| hughsaunders | does nodepool have a hook for running a script just before marking a request as fulfilled? | 15:50 |
| clarkb | hughsaunders: not anymore, idea is to hvae tools like zuul do that checking instead | 15:51 |
| clarkb | since they tend to have a richer representation of how to do those things | 15:51 |
| hughsaunders | OK. I've been having issues with Jenkins' auto JDK installation and wanted to try installing it from apt before the node is handed to Jenkins. | 15:52 |
| hughsaunders | I guess I'll have to do that from within the Jenkins Nodepool plugin if nodepool doesn't have a suitable hook. | 15:53 |
| corvus | hughsaunders: or you could make a custom image with that installed | 15:53 |
| hughsaunders | Maybe an AptJDKInstallationStrategy could work. | 15:53 |
| hughsaunders | corvus: yeah, but custom images are sooo much slower on $cloud | 15:53 |
| corvus | clarkb: i think https://review.openstack.org/551301 is ready if you have a sec to re-review | 16:12 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix safe path checks https://review.openstack.org/552074 | 16:12 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add further test cases to tox-remote https://review.openstack.org/552075 | 16:12 |
| clarkb | looking now | 16:12 |
| tobiash | corvus, clarkb ^ | 16:13 |
| *** electrofelix has quit IRC | 16:18 | |
| clarkb | grr gerrit fighting me (working on it though) | 16:22 |
| clarkb | is 552075 not loading in the web ui for anyone else? | 16:26 |
| clarkb | it seems to affect me once I'm logged in but not being logged in it is fine? | 16:26 |
| tobiash | clarkb: just looked at it | 16:27 |
| clarkb | maybe it is something with my account? this is really weird | 16:27 |
| tobiash | clarkb: 552075 breaks the py35 tests :( | 16:28 |
| tobiash | checking | 16:28 |
| tobiash | but tox-remote is green | 16:28 |
| tobiash | I think I spotted the problem | 16:31 |
| clarkb | I'm wondering if this is a local networking problem :( | 16:32 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add further test cases to tox-remote https://review.openstack.org/552075 | 16:33 |
| clarkb | I'm not seeing packet loss but other http is also slow | 16:33 |
| tobiash | that should hopefully fix the tests ^ | 16:34 |
| clarkb | I restarted local wifi connection on laptop and it seems happier now | 16:34 |
| clarkb | but its sad again. I wonder if my ap is unhappy | 16:39 |
| clarkb | ok used my phone to approve the base of the stack :) | 16:41 |
| *** jpena is now known as jpena|brb | 16:42 | |
| tobiash | phone is a good fallback :) | 16:43 |
| kklimonda | hmm,can roles be added to the zuul.d/ directory so they don't live in the project root? | 16:56 |
| openstackgerrit | Merged openstack-infra/zuul master: Add new tox-remote job https://review.openstack.org/551301 | 17:00 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix safe path checks https://review.openstack.org/552074 | 17:04 |
| clarkb | corvus: tobiash ^ | 17:06 |
| tobiash | \o/ | 17:06 |
| tobiash | 552075 is also green now :_ | 17:06 |
| tobiash | :) | 17:06 |
| corvus | tobiash: do you want to draft an email to the zuul-announce list describing the vulnerability, or should i? | 17:11 |
| tobiash | I can do after I had something to eat | 17:11 |
| corvus | tobiash: ok, thanks :) | 17:12 |
| corvus | i think we only need to restart executors for this... does that sound right? | 17:13 |
| corvus | kklimonda: roles can live alongside playbooks if you don't want the project itself to generally provide a role | 17:14 |
| kklimonda | @corvus but not if that role is used in an external playbook | 17:14 |
| kklimonda | (well, that's a question really :)) | 17:15 |
| corvus | kklimonda: right, in that case, it needs to be at the root. the thinking is that if a role is "exported" so to speak, it should not be in a zuul-specific way. it should be easy to use by ansible in general. | 17:16 |
| kklimonda | well, those are zuul-specific roles | 17:16 |
| kklimonda | so, to explain what I'm looking into - I have three projects: build-container, deploy-containers, integration-tests | 17:17 |
| kklimonda | I was thinking about putting roles into their corresponding projects | 17:17 |
| kklimonda | so a role used to build containers (used for deployment) would be in the build-container project etc. | 17:18 |
| kklimonda | this way, if the project changes something about containers (for example renames them) it could fix its job at the same time | 17:19 |
| corvus | those may not be as zuul-specific as they sound: "an ansible role to build the containers" | 17:19 |
| kklimonda | but then I'd have to fix a role used to deploy containers at the same time, and that introduces a dependency cycle | 17:19 |
| kklimonda | build-container has to test if deploy-container still works, but it requires a change to deploy-container role | 17:20 |
| kklimonda | so perhaps keeping those roles in a common repository, and accepting that we have to add transitions every time we break something, is the way to go afterwards. | 17:21 |
| corvus | put another way, consider whether these can be general purpose ansible roles that you happen to, at the moment, only be running inside of zuul. | 17:21 |
| kklimonda | sigh | 17:21 |
| kklimonda | the code for building containers is custom-tailored | 17:22 |
| kklimonda | so we just launch an internal script now really | 17:22 |
| tobiash | corvus: yes, executor should be enough | 17:22 |
| kklimonda | but some changes "break job API" so to speak - container names change, or tagging scheme, and that requires us to update the job that tests that they can still be deployed | 17:23 |
| *** jpena|brb is now known as jpena | 17:24 | |
| corvus | kklimonda: i think deciding whether you want to force continuity or have an external check on api breakage is a big part of determining where things like that live; you can go either way depending on what workflow you want. | 17:25 |
| *** rlandy is now known as rlandy|afk | 17:29 | |
| kklimonda | corvus: btw, if I want to discuss a larger topic of zuul/nodepool (for example windows support) is coming to vancouver beneficial? What's the format of the summit, is it only presentations all day long, or is there a time for discussions? | 17:31 |
| corvus | kklimonda: there's going to be a concurrent conference called 'opendev' during the summit focused on ci/cd topics, and summit attendees with an interest in that area will be encouraged to attend. it will have talks, working sessions, and hands-on sessions, so we'll have more opportunities for collaboration. | 17:34 |
| corvus | kklimonda: but it's not going to be like a ptg... so a discussion like that might be more of a hallway track item... | 17:34 |
| corvus | kklimonda: also, tobiash should be part of that discussion, and he may not be in vancouver... | 17:35 |
| corvus | kklimonda: tobiash has done quite a bit to implement windows support, and it's accepted as a feature we want to support. | 17:36 |
| kklimonda | well, I'd still like to meet the people, and it's being paid by my employee, so I'm mostly curious how much will I actually will have to work ;) | 17:36 |
| tobiash | corvus: maybe I get the chance for vancouver | 17:37 |
| corvus | kklimonda: oh yes, several of us will be there and would be very happy to meet you. your attendance at opendev would be great regardless :) | 17:37 |
| corvus | kklimonda, tobiash: ^ all the better then :) | 17:38 |
| tobiash | just got wife-approval and unofficial management approval | 17:38 |
| tobiash | so if it makes sense I try to get there | 17:38 |
| kklimonda | corvus: it shares venue and pass with summit? | 17:38 |
| corvus | kklimonda: yep: http://2018.opendevconf.com/ | 17:38 |
| corvus | kklimonda: folks can register for just opendev, but also summit attendees can attend. | 17:38 |
| kklimonda | corvus: the other request I'm hearing from our windows developers is adding support in nodepool for creating separated networks - they are working on SDN and want to isolate tests from each other. Has that been considered in the past? | 17:40 |
| rcarrillocruz | wellll | 17:40 |
| rcarrillocruz | i'd be more than interestad in that one | 17:40 |
| rcarrillocruz | like | 17:40 |
| rcarrillocruz | per-label networks ? | 17:40 |
| * rcarrillocruz works on ansible networking btw | 17:40 | |
| corvus | i read it as more like "per-request" network | 17:40 |
| corvus | ie, "give me 5 nodes and a brand new network" | 17:41 |
| rcarrillocruz | you can today attach nods to multiple networks | 17:41 |
| rcarrillocruz | but all nodes get attached to them | 17:41 |
| rcarrillocruz | that's what i use today | 17:41 |
| rcarrillocruz | corvus: yeah, that would be awesome for my use cases too | 17:41 |
| rcarrillocruz | cos i have folks asking me how we could test topologies with nodepool | 17:41 |
| rcarrillocruz | per-job/request networks would be awesome | 17:42 |
| kklimonda | yeah, also not having to attach every network to every node would be great | 17:42 |
| rcarrillocruz | i honestly don't think it would be much work , just adding networks to per-label and process them in the create_server call | 17:42 |
| rcarrillocruz | kklimonda: if you could open a story, would be happy to take a look | 17:42 |
| rcarrillocruz | next week i'll be in a team meeting, and will discuss things we may need from nodepool/zuul | 17:43 |
| corvus | we've talked a bit about expanding nodepool to be able to provide more kinds of resources. i think we'll probably end up doing that. we didn't want to try to do anything like that before the 3.0 release. but i think we'll probably need a design document for it. | 17:43 |
| rcarrillocruz | ++ | 17:43 |
| kklimonda | sure | 17:43 |
| rcarrillocruz | corvus: happy to look , get assigned | 17:43 |
| corvus | ('cause it's not just networks, same thing could be useful for storage, etc) | 17:44 |
| rcarrillocruz | i think i could justify work hours from my team as that is a use case we may need implemented | 17:44 |
| rcarrillocruz | right, in networking world is common to have networking images require certain HDDs attached as volumes | 17:44 |
| rcarrillocruz | otherwise they won't boot | 17:44 |
| rcarrillocruz | i can circumvent that by tweaking the booting image and adding partitions/filesystems with guestfs | 17:45 |
| rcarrillocruz | but would be awesome to have volumes and other resources as first class citizens | 17:45 |
| corvus | i feel like "how to obtain resources needed for tests" in general is a good topic for opendev. | 17:45 |
| * rcarrillocruz sighs, checks again opendev dates but thinking i'll be parenting newborn | 17:46 | |
| corvus | as far as implementation specifics for zuul -- we can talk about it in the hallway in vancouver, or on the mailing list. and when we have an idea of how it should work, write up a specification. | 17:46 |
| rcarrillocruz | ++ | 17:46 |
| tobiash | corvus: https://etherpad.openstack.org/p/CcPPe1NOFa | 17:50 |
| tobiash | probably needs some polishing by a native speaker ;) | 17:50 |
| corvus | tobiash: let's drop 'cve' since we didn't request one | 17:51 |
| tobiash | k | 17:51 |
| Shrews | nodepool needs are beginning to sound like a cloud for clouds | 17:52 |
| Shrews | perhaps we should rename to "cloudpool" for 4.0 :) | 17:52 |
| corvus | Shrews: yeah, it's the universal cloud api for all instances of all clouds. but that's what we need. :) | 17:52 |
| Shrews | meh, we're not asking for much. just to support all things in all the various clouds. lol | 17:53 |
| Shrews | 4.0 will be fun | 17:54 |
| kklimonda | is 4.0 when we can run multiple schedulers for HA/rolling restarts? ;) | 17:55 |
| tobiash | kklimonda: that's on the roadmap | 17:56 |
| corvus | tobiash: i'd like to emphasize the role of bubblewrap here a bit more -- does line 13 look ok? can we maybe insert that as the second sentence and drop the last sentence from line 3? | 17:58 |
| tobiash | that sounds great | 17:58 |
| corvus | tobiash: basically, i want folks to know that if they're running bubblewrap, they're probably okay, and if they aren't, they probably aren't. | 17:58 |
| corvus | okay, i'll move it in | 17:58 |
| rcarrillocruz | Shrews: MOAR WORK | 17:59 |
| corvus | fungi, clarkb: can you take a look at https://etherpad.openstack.org/p/CcPPe1NOFa ? | 18:00 |
| fungi | yep | 18:01 |
| *** myoung|ruck is now known as myoung|afk | 18:01 | |
| *** myoung|afk is now known as myoung|biab | 18:01 | |
| clarkb | lgtm | 18:01 |
| corvus | did we by any chance change how hostnames are reported in metrics? | 18:02 |
| tobiash | corvus: yes there was a change to replace . and : by _ | 18:02 |
| corvus | tobiash: that matches what i'm seeing in tcpdump. :) | 18:03 |
| fungi | corvus: that looks fine. if we're interested in adopting some of the more formal process used by the openstack vmt, there are some useful cut-n-paste templates with recommended wording at https://security.openstack.org/vmt-process.html#templates | 18:03 |
| tobiash | corvus: change 547309 | 18:03 |
| fungi | since this is a vulnerability in an unreleased codebase though, a lot of the wording there is probably irrelevant | 18:04 |
| corvus | fungi: oh awesome. i think we should do that for next time.... i'll let tobiash decide if he wants to reword or just go with what we have so far. | 18:04 |
| fungi | i'd recommend at least including information on the reporter of the vulnerability (if they're okay with that) since it's good form to give credit in advisories | 18:05 |
| tobiash | fungi: the reporter was me ;) | 18:05 |
| fungi | yeah, totally up to you tobiash! | 18:06 |
| fungi | if you don't want credit for reporting it, then there's no need to include a sentence saying who reported it | 18:06 |
| tobiash | fungi: you mean this template https://security.openstack.org/vmt-process.html#openstack-security-advisories-ossa ? | 18:07 |
| fungi | in openstackland we tend to get a fair number of vulnerabilities reported by security researchers who like to see themselves or their employers credited for doing that work | 18:07 |
| fungi | tobiash: well, the ossa template is actually yaml input to a sphinx extension we use to transform it into restructuredtext (which is then used as a basis for a published html rendering, but also intended to be suitable for e-mailing to mailing lists) | 18:08 |
| tobiash | ah, ok | 18:08 |
| fungi | i mostly just meant wording we have in the impact description | 18:09 |
| tobiash | ah, ok | 18:09 |
| tobiash | looking | 18:09 |
| fungi | the main points of communicating a vulnerability are to describe the impact (so phrasing it relative to a potential exploit scenario can sometimes be helpful) and letting users know where to get the fix | 18:10 |
| fungi | a common trap advisory authors fall into is focusing on implementation details of the vulnerable code or the patch, so we use that impact description template as a reminder to ourselves to stick to (or at least start out with) the most important information | 18:11 |
| fungi | the degree of technical detail worth embedding in an advisory text is just enough to disambiguate it from past or likely future similar vulnerabilities | 18:12 |
| *** sshnaidm is now known as sshnaidm|afk | 18:13 | |
| fungi | ideally people interested in the deeper details will look at any linked bug reports or commit messages and code for the fixes you've linked in the advisory anyway so there's not a lot of need to duplicate information which can be found there | 18:13 |
| corvus | okay, so we found a problem with the security patch | 18:20 |
| corvus | http://logs.openstack.org/22/551822/1/gate/openstack-tox-py27/623f25c/ara/file/71b64ee4-f5b4-49ef-aed3-419a39127f07/#line-4 | 18:20 |
| -openstackstatus- NOTICE: Most jobs in zuul are currently failing due to a recent change to zuul; we are evaluating the issue and will follow up with a recommendation shortly. For the moment, please do not recheck. | 18:21 | |
| *** ChanServ changes topic to "Most jobs in zuul are currently failing due to a recent change to zuul; we are evaluating the issue and will follow up with a recommendation shortly. For the moment, please do not recheck." | 18:21 | |
| corvus | http://logs.openstack.org/22/551822/1/gate/openstack-tox-py27/623f25c/ | 18:21 |
| tobiash | corvus: I think I've spotted the problem | 18:21 |
| tobiash | in the playbook dirs we link to the according repos containing the ansible stuff | 18:21 |
| tobiash | this is probably a symlink and gets resolved before the path check | 18:22 |
| tobiash | maybe we should change that to use hard linking? | 18:22 |
| corvus | tobiash: oh interesting idea | 18:22 |
| corvus | http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py#n1159 | 18:23 |
| corvus | that's the code which does the symlinking | 18:23 |
| tobiash | I looked at the comment describing the dir structure http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py#n299 | 18:23 |
| tobiash | which looked pretty similar to the rejected path | 18:23 |
| dmsimard | I've been out all morning so I have little to no context to work with, I'll catch up with the backlog and see if I can help. | 18:24 |
| fungi | availability of hardlinking tends to be filesystem-specific so may not be a good choice | 18:24 |
| fungi | unless we already make an assumption elsewhere that hardlinks are an acceptable expectation | 18:25 |
| tobiash | fungi: we use hard linking for repos already | 18:25 |
| tobiash | afaik | 18:26 |
| fungi | okay, then i retract my concern ;) | 18:26 |
| dmsimard | fungi: if we see a filesystem constraint, it should be documented -- I think it's reasonable to expect people to deploy on "non-exotic" filesystems | 18:26 |
| tobiash | but have to double heck | 18:26 |
| tobiash | check | 18:26 |
| dmsimard | fungi: and if someone wants to deploy on an exotic filesystem, they can send patches to support it :D | 18:26 |
| tobiash | fungi: at least the disk accountant argues with hard links: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py#n91 | 18:27 |
| dmsimard | corvus: did you send that email from the etherpad already ? I spotted a typo. | 18:28 |
| fungi | i expect the announcement isn't going out until we can safely use the patch linked from it | 18:28 |
| dmsimard | ok /me fixes typo | 18:29 |
| corvus | dmsimard, fungi: yeah, i'd say that's on hold until we figure out next step | 18:29 |
| * dmsimard continues catching up with backscroll | 18:29 | |
| *** dtruong has joined #zuul | 18:30 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Use hard linking for playbook environments https://review.openstack.org/552110 | 18:31 |
| corvus | when i wrote the symlink code, i was (a) not aware that a role would need to access files inside its own directories, and so was not aware of the mismatch between the bwrap security model and the path-checking model. so i didn't think about hard links vs symlinks in that context. | 18:31 |
| tobiash | corvus: how about that? ^ | 18:31 |
| tobiash | corvus: and maybe we also need to whitelist the playbook dir as I think might not be inside the work dir | 18:32 |
| tobiash | or do we bind that into the work dir during run? | 18:32 |
| corvus | and (b) i was thinking that symlinks provided more debugability than hard links (you could see that a given playbook was either trusted or untrusted by looking at the symlink). i don't think that's a reason to stick with symlinks. | 18:33 |
| clarkb | 18:33 | |
| corvus | tobiash: we don't want to whitelist it because we don't want a job to be able to change the playbook it's running | 18:33 |
| clarkb | sorry | 18:33 |
| tobiash | corvus: hrm, but we need to allow it as input sources | 18:34 |
| corvus | i wonder if another approach would be to relax the path check on src files... | 18:34 |
| corvus | maybe work/ and trusted/ are okay for src= | 18:34 |
| corvus | the nice thing about that is that it would fix dmsimard's problem -- you could use {{ role_path }} explicitly | 18:35 |
| dmsimard | I'm struggling to understand how hard links would be different than symlinks in our context -- they end up pointing to the same "content", a symlink is more or less a redirection but a hard link is a direct reference to the inode so there's no such redirection. Is the problem the content the symlinks can point to ? Or rather the fact that we don't control where they point ? | 18:35 |
| dmsimard | (note I'm still catching up with a LOT of backscroll) | 18:35 |
| tobiash | what I don't understand is why does the tox-remote test work (and most other tasks)? | 18:35 |
| tobiash | so this is probably already in the work dir within the bwrap context | 18:36 |
| corvus | tobiash: we may not have a test which has a trusted role used by an untrusted playbook | 18:36 |
| tobiash | corvus: so then I think the symlink may be the only problem | 18:36 |
| dmsimard | corvus: the reason why we don't allow access to /trusted is because it'd allow someone to modify the content there before running them I guess ? | 18:36 |
| corvus | dmsimard: yep | 18:37 |
| fungi | dmsimard: if the inode is accessible to the process in question but there is a pattern match being performed on the dereferenced link target which determines that it doesn't follow an acceptable pattern, a hardlink to the same inode would work around that | 18:37 |
| dmsimard | fungi: but are we trying to prevent people from working around that ? | 18:37 |
| corvus | dmsimard: people, yes. zuul, no. | 18:38 |
| corvus | where is the fetch-testr role? | 18:39 |
| fungi | my understanding is we're talking about how to allow zuul to poke holes in the trusted veil without allowing untrusted jobs to do the same | 18:39 |
| dmsimard | corvus: so this might be a stupid idea but what if we just "mounted" the trusted content in read only instead ? | 18:39 |
| tobiash | corvus: zuul-jobs | 18:39 |
| corvus | sorry, fetch-subunit-output | 18:39 |
| corvus | fungi: yep, that's how i'd characterize the hard-link option. | 18:39 |
| corvus | hrm, why is it in tursted/ ? | 18:40 |
| dmsimard | Could the executor prepare the content outside the bwrap and then mount it read only in the bwrap ? I'm mostly just thinking out loud. | 18:40 |
| tobiash | corvus: maybe because there is no speculative version of zuul-jobs involved | 18:40 |
| corvus | dmsimard: it is mounted read-only in the bwrap. | 18:40 |
| tobiash | and the base job uses zuul-jobs | 18:40 |
| corvus | dmsimard: brwap is our second level defence. we're trying to fix the first level, which is patch checking. | 18:40 |
| dmsimard | corvus: hmm, in the expectation that people can /kind of/ run this without bwrap ? | 18:41 |
| corvus | dmsimard: currently, because of the vulnerability in patch checking, we are relying *only* on the second level defence. | 18:41 |
| fungi | fell off the tightrope onto the safety net, but need to get back on the tightrope ;) | 18:41 |
| corvus | fungi: yep | 18:41 |
| dmsimard | well, I was mostly wondering how people could end up modifying trusted content (re: my earlier question) if the trusted content was mounted read only | 18:42 |
| corvus | dmsimard: i'm not going to recommend anyone do that, though some have expressed an interest. for me, this is a defence-in-depth exercise. | 18:42 |
| dmsimard | corvus: +1 | 18:42 |
| kklimonda | does the new Depends-On syntax means we can do cross-connection dependencies? | 18:42 |
| dmsimard | kklimonda: yes, it's been tested before | 18:42 |
| dmsimard | between shade and ansible iirc ? | 18:43 |
| corvus | dmsimard: the answer is "with bwrap, they can't", but we need to ignore that in order to fix the path check so it stands alone as an effective defense. | 18:43 |
| kklimonda | dmsimard: do I have to connect zuul to each source, or can it just use the url to fetch change and its dependencies? | 18:43 |
| fungi | kklimonda: the former for now. there's been discussion of potentially supporting some semblance of the latter in the future i think | 18:43 |
| dmsimard | kklimonda: I believe projects need to be in Zuul's main configuration file (where the included projects are listed), hopefully someone can correct me if I'm wrong | 18:43 |
| *** ChanServ changes topic to "Discussion of the project gating system Zuul | Docs: http://docs.openstack.org/infra/zuul/ | Source: https://git.openstack.org/cgit/openstack-infra/zuul/ | Roadmap: https://storyboard.openstack.org/#!/board/53 | Channel logs: http://eavesdrop.openstack.org/irclogs/%23zuul/" | 18:43 | |
| -openstackstatus- NOTICE: Zuul has been restarted without the breaking change; please recheck any changes which failed tests with the error "Accessing files from outside the working dir ... is prohibited." | 18:43 | |
| corvus | tobiash: oh, gotcha. the base job pulled in zuul-jobs, and since there's no change to that, zuul didn't bother to create a non-trusted checkout of it. | 18:44 |
| kklimonda | thanks | 18:44 |
| corvus | okay, back to solutions -- have we poked any holes in the hard-link idea, or is it sound? | 18:46 |
| dmsimard | kklimonda: btw to your earlier question, despite not having two "gating" zuuls, it's possible to require a +verified vote from two different zuul instances | 18:47 |
| tobiash | regarding as write destination I don't think there is a real difference | 18:47 |
| corvus | tobiash: not sure i understand that | 18:47 |
| dmsimard | kklimonda: you need to look at the "require" stanza http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml#n70 | 18:48 |
| tobiash | maybe we should make the path checks for targets to not rely on bwrap ro-mount | 18:48 |
| dmsimard | kklimonda: (the trigger stanza as well, depends on how you set things up) | 18:48 |
| corvus | tobiash: i still don't understand | 18:48 |
| tobiash | corvus: just thought if there is a different 'overwrite playbooks' risk with hard links than with symlinks | 18:49 |
| kklimonda | dmsimard: so the idea is that I guard submit action somehow, probably by creating a new vote type so that submit action is done only after both instances voted? | 18:50 |
| *** harlowja has joined #zuul | 18:50 | |
| dmsimard | kklimonda: in gerrit "submit" is a privilege that you can give to users (or not) -- in OpenStack's gerrit, no users have submit privileges but infra-root can "escalate" their own privileges to have submit privileges | 18:51 |
| dmsimard | kklimonda: so you "guard" against submit through the gerrit ACL | 18:51 |
| kklimonda | dmsimard: right, but if I have two zuul instances doing gating, I can't have "primary" zuul submit patch to gerrit until "secondary" zuul voted +2 | 18:52 |
| dmsimard | kklimonda: what you do at the pipeline level is to require a +verified vote from multiple users to enter gate | 18:52 |
| fungi | kklimonda: right, what dmsimard said. and we create a workflow vote which tells our ci system we're ready for a final pass at testing before it should attempt to submit the change for merge | 18:52 |
| fungi | kklimonda: hrm, i don't know if we've tried having two zuul deployments relying on each other for decision making | 18:53 |
| kklimonda | yes, but this still sounds like a single zuul doing an actual gating, requiring multiple votes to start the gate pipeline - or do I misunderstand something? | 18:53 |
| fungi | kklimonda: correct | 18:53 |
| kklimonda | I'd prefer to hear that gating with two zuul is a horrible idea, and I'm a bad person for even considering that | 18:54 |
| dmsimard | kklimonda: there can only ever be one "system" doing the gating, otherwise it's error prone | 18:54 |
| dmsimard | kklimonda: so what you do is to require the two zuuls to return +verified in the "check" pipeline so that a change can enter the "gate" pipeline (on one Zuul) | 18:54 |
| tobiash | kklimonda: it is a horrible idea | 18:54 |
| kklimonda | tobiash: thanks ;) | 18:55 |
| tobiash | gating is a sequential process and you simply cannot synchronize two zuuls such that they have the same changes in gate and press the submit button together | 18:55 |
| dmsimard | kklimonda, tobiash: well, hang on -- it's not horrible and there's a use case for it.. For example, if you have third party CI that you want to make "voting". | 18:55 |
| fungi | kklimonda: you might be able to have verify +1, +2 and +3 and then have zuul #1 perform its final testing when asked and have it leave a verify +2, then have zuul #2 trigger on a verify +2 comment and leave a verify +3 | 18:55 |
| dmsimard | But it'll be "voting" for the "check" pipeline, not the gate pipeline. | 18:55 |
| fungi | it would serialize the final testing between the two of course | 18:55 |
| kklimonda | I'll just go with "there can be only one gate keeper" | 18:55 |
| corvus | okay, if anyone would like to discuss the security issue, please follow me to #openstack-infra-incident | 18:55 |
| tobiash | dmsimard: but that's not two zuuls gating together | 18:55 |
| kklimonda | dmsimard: right, we actually run our zuul in this configuration today | 18:56 |
| kklimonda | we have two different CI systems - zuulv3 for linux, and zuulv2 + jenkins for windows | 18:56 |
| tobiash | the gate is the last validation of the final future state which cannot be done by two independent systems | 18:56 |
| kklimonda | our (noop) gate requires both to vote +1 | 18:56 |
| tobiash | having check in several systems is ok though | 18:57 |
| kklimonda | tobiash: yes, my gut feeling was that requiring votes from multiple systems to enter gating is fine, but there can be only one system with the final say in merge. | 18:58 |
| dmsimard | kklimonda: we're on the same wavelength. | 18:59 |
| tobiash | yes | 19:00 |
| kklimonda | It would be easier if the concept of gating was actually understood better - I have a lot of discussions where people conflate gating and testing | 19:00 |
| fungi | we keep trying to correct people when they misuse the terminology | 19:01 |
| kklimonda | so now I start every discussion explaining what gating is | 19:01 |
| fungi | but it never seems to sink in | 19:01 |
| *** rlandy|afk is now known as rlandy | 19:01 | |
| fungi | for that matter, people tend to conflate jobs with tests, and good luck getting them to be clear when they're talking about one or the other | 19:01 |
| *** openstackgerrit has quit IRC | 19:04 | |
| mrhillsman | can i use zuul.yaml to define services to deploy via devstack? | 19:07 |
| mrhillsman | nvm | 19:09 |
| *** jpena is now known as jpena|off | 19:12 | |
| *** myoung|biab is now known as myoung|ruck | 19:13 | |
| *** openstackgerrit has joined #zuul | 19:31 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Allow trusted for find_needle https://review.openstack.org/552127 | 19:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Allow trusted for find_needle https://review.openstack.org/552127 | 20:16 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Allow trusted for find_needle https://review.openstack.org/552127 | 20:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Allow trusted for find_needle https://review.openstack.org/552127 | 20:23 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Allow trusted for find_needle https://review.openstack.org/552127 | 20:26 |
| jlk | toabctl: I'm here now | 20:27 |
| SpamapS | corvus: FYI, I ended up going rogue and re-did all my slides for the SCALE talk. http://fewbar.com/zuul-ci-crossing-streams/ | 20:27 |
| jlk | tobiash: er I'm here now. | 20:27 |
| SpamapS | My favorite slide: | 20:28 |
| SpamapS | Speculative merging for high velocity merging | 20:28 |
| SpamapS | [ switch to other presentation ] | 20:28 |
| SpamapS | I really would love to have the zuul simulation in some form that could just be iframed or something. | 20:28 |
| SpamapS | jlk: o/ | 20:28 |
| tobiash | jlk: o/ | 20:29 |
| dmsimard | SpamapS: someone could do one of those fancy animated gifs that illustrates that in some way | 20:29 |
| dmsimard | SpamapS: or asciinema | 20:29 |
| jlk | my brain tries to read that as "ascii enema" | 20:29 |
| dmsimard | jlk: guinness-- | 20:30 |
| tobiash | jlk: so it looks like the latest change to github3.py broke zuul for us (luckily not production ;) ) | 20:30 |
| *** dkranz has quit IRC | 20:30 | |
| jlk | oh noes, did he merge that big change? I bet he did. | 20:31 |
| jlk | I've been out of it for a couple weeks | 20:31 |
| tobiash | it looks like it crashes when querying repos without an attached licence? | 20:31 |
| tobiash | yah he merged it a day ago | 20:31 |
| tobiash | we've merged a pin to requirements.txt before that merge | 20:31 |
| tobiash | as a quick fix | 20:32 |
| jlk | cool. Is there an open issue somewhere that indicates the breakage? | 20:32 |
| tobiash | not yet | 20:32 |
| tobiash | I was knee deep in the security issues and had just time to do the pin to unbreak us | 20:32 |
| jlk | okay. I'm happy to help work on it, or shepherd a patch through. I'm getting caught back up at work though and won't be able to spend a lot of time immediately on diagnosis. | 20:33 |
| tobiash | I'll be stuck on security fixes and windows integration in nodepool next 1-2 weeks | 20:34 |
| jlk | cool cool. I'll add it to my list to look into it I suppose. Where was it breaking? handling events? | 20:35 |
| tobiash | I think during getBranches in the gh driver | 20:35 |
| tobiash | at least after prime_installation map | 20:36 |
| tobiash | at least my integration zuul refused to start ;) | 20:37 |
| jlk | oh that's pretty fast then. That should be fairly easy to sort out. | 20:37 |
| jlk | I just have to resurrect my set up to launch a pile of containers for this. | 20:37 |
| tobiash | jlk: but maybe it can be even triggered easier by requesting the repo object for a repo without license | 20:39 |
| tobiash | just as a quick guess without having to resurrect zuul and dealing first with getting it running again ;) | 20:39 |
| jlk | okay, thanks! | 20:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Allow trusted for find_needle https://review.openstack.org/552127 | 20:59 |
| tobiash | corvus: removed debug print and added a comment ^ | 21:00 |
| Shrews | gah, are we back to meeting 1 hour from now? stupid DST | 21:01 |
| corvus | Shrews: first topic on agenda! | 21:02 |
| corvus | tobiash: i ran the remote test locally -- it fails the patch test | 21:02 |
| corvus | tobiash: hah | 21:02 |
| Shrews | oh, i certainly ++ any earlier times | 21:02 |
| corvus | tobiash: i think that may just be me :) | 21:02 |
| * corvus installs patch on test node | 21:03 | |
| tobiash | missing patch? | 21:03 |
| tobiash | :) | 21:03 |
| corvus | yep, passes now :) | 21:03 |
| corvus | i'll run the normal test suite locally too | 21:03 |
| corvus | the previous version of the patch passed all the other jobs | 21:03 |
| tobiash | k | 21:04 |
| corvus | clarkb: can you review https://review.openstack.org/552075 ? | 21:07 |
| corvus | i'd like to land that before we land https://review.openstack.org/552127 | 21:08 |
| clarkb | ya Im grabbing lunch then will be back to computer | 21:09 |
| corvus | once we land those, i'd like to restart openstack's executors again, and send out the announcement: https://etherpad.openstack.org/p/CcPPe1NOFa | 21:11 |
| corvus | it's been updated to reference the new patch | 21:11 |
| corvus | also, in working on this issue, tobias has found another related issue, which we've discussed privately | 21:12 |
| corvus | the announcement mentions that as well | 21:12 |
| corvus | the gist is that it looks like there's another way one might bypass these path checks. again, i think bwrap prevents real damage from occuring. | 21:13 |
| corvus | i think under the circumstances, the most responsible thing to do is to just let folks know there will be at least one more security update related to this, and that using bwrap should mitigate the issue | 21:14 |
| tobiash | yay, 552127 passed zuul-tox-remote :) | 21:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix error in test_jobs_executed https://review.openstack.org/552142 | 21:24 |
| corvus | tobiash: it also passed my local py35 tests | 21:24 |
| corvus | (except i learned that 552142 is needed) | 21:25 |
| corvus | but i'm confident enough to request clarkb review both 552127 and 552075 now when he's back from lunch :) | 21:25 |
| tobiash | +2 on 552142 | 21:26 |
| corvus | tobiash: and if you need to sign off for the evening, i think it's probably okay to send the email to zuul-announce whenever you're ready | 21:27 |
| *** hashar has quit IRC | 21:27 | |
| tobiash | I'll probably awake today until the first topic in the meeting ;) | 21:28 |
| tobiash | ok, mail is prepared, just have to press the send button when the time has come | 21:31 |
| fungi | the wording is still what's in https://etherpad.openstack.org/p/CcPPe1NOFa i guess | 21:33 |
| tobiash | yah, I still can change things there | 21:34 |
| fungi | i've made a handful of minor (mostly grammatical) edits | 21:38 |
| tobiash | thx | 21:38 |
| dmsimard | I think the emphasis on Bubblewrap is definitely appropriate | 21:40 |
| dmsimard | I mean, we're basically allowing arbitrary users run arbitrary code on the Ansible control node :) | 21:41 |
| SpamapS | I keep forgetting to mention the bubblewrap integration | 21:42 |
| SpamapS | which is really cool | 21:42 |
| fungi | granted that's not the intent, and fixing these sorts of bugs is still important | 21:42 |
| SpamapS | although I also kind of see it as this like, just good due dilligence we did for when people ask how we protect the executor. | 21:42 |
| fungi | ideally zuul will appropriately constrain arbitrary code supplied by arbitrary users, but for now that's not as solid as we want it to be | 21:42 |
| corvus | yes. though i will say that i was inclined to treat bwrap as required before, and this has only strengthened my conviction on that... | 21:44 |
| SpamapS | This is why we have two layers of security. :) | 21:44 |
| dmsimard | I discussed a bit with upstream in #ansible-devel (and off the record) about file/path filtering and access in general and it made me think about other means of "restricting" access to things that have nothing to do with python or Ansible -- namely chroots and plain unix permissions (or even filesytem ACLs if we want to get fancy) | 21:44 |
| SpamapS | corvus: agreed. | 21:44 |
| corvus | i wonder if there would be any objection at this point to removing the ability to use nullwrap...? | 21:44 |
| * clarkb is pulling up changes to revie wnow | 21:44 | |
| fungi | right now we're finding holes in our assumptions/implementation where constraining ansible is concerned, but who's to say next time it won't be holes in bubblewrap we're finding and then thankful we have robust constraints in place for ansible? | 21:44 |
| clarkb | I had bun for lunch | 21:44 |
| SpamapS | The ability to run with it turned off is entirely to allow weird things that nobody should do without patching. | 21:44 |
| dmsimard | grabbing some food, be back in a while | 21:44 |
| corvus | fungi: indeed | 21:44 |
| dmsimard | fungi: +1 | 21:45 |
| SpamapS | Ideally we'd have 1 more layer, which is a MAC of some sort. | 21:45 |
| fungi | do we know of any deployments which aren't using bubblewrap? | 21:45 |
| SpamapS | not I | 21:45 |
| clarkb | corvus: ok so you want 552075 in first? | 21:46 |
| * clarkb starts there | 21:46 | |
| jlk | Is meeting happening in 15 minutes, or did DST throw things around? | 21:46 |
| fungi | i recall some of the reason for teh nullwrap was so people could run it on platforms which don't (yet?) support or provide bwrap | 21:46 |
| corvus | jlk: 15m | 21:46 |
| corvus | clarkb: ideally yes | 21:46 |
| fungi | but no idea whether those users were pure speculation | 21:46 |
| corvus | clarkb: i tested the other change with that in place, so i know it's good. if there's a problem with 552075 we can land the other first though. | 21:46 |
| SpamapS | fungi: one of those platforms is unprivileged docker containers. | 21:47 |
| SpamapS | though I believe that is solved now | 21:47 |
| tobiash | the nullwrap driver doesn't support secrets so the only full featured execution wrapper is brwapp anyway | 21:47 |
| SpamapS | and bwrap with USER things in the kernel works unprivileged | 21:47 |
| fungi | cool, so yes i have no objection to further crippling/removing the ability to run without bubblewrap for now | 21:48 |
| SpamapS | I'm still of the opinion that it's better to give people a "turn off the security" option than to require them to patch the code to do so. They'll end up running an old patched version of zuul instead of new ones, because they have to keep porting forward their "turn off the security" patch. :-/ | 21:48 |
| *** _ari_ has quit IRC | 21:49 | |
| SpamapS | Like unix, I think we should give people enough rope to hang themselves, and then a little more. | 21:49 |
| *** fbo has quit IRC | 21:50 | |
| fungi | hrm, yeah i suppose _if_ there are users who want to do that (but are there?) then it's either using outdated insecure zuul or running up to date zuul configured to be insecure. with appropriate flashing warnings around it, i guess the latter is at least no worse than the former | 21:50 |
| corvus | the danger is if it gets used because of some solveable problem -- like "i didn't know how to install bwrap so i turned it off" | 21:50 |
| *** _ari_ has joined #zuul | 21:51 | |
| SpamapS | Same things happens on Linux. "I didn't know how to use SELinux, so I turned it off. I didn't know how to run the daemon as not-root, so I ran it as root." | 21:52 |
| *** fbo has joined #zuul | 21:52 | |
| corvus | yeah. i'd like to avoid any suggestion that that's okay. :) | 21:52 |
| fungi | "a common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools" (douglas adams) | 21:52 |
| SpamapS | Also one thing I've never looped back to is some integration tests where we try to run evil playbooks with nullwrap to see if they succeed. | 21:53 |
| corvus | basically, i'd feel better if our security announcements said "we had a problem but bubblewrap protected us" rather than "we had a problem, and as long as you didn't disable bubblewrap you're okay" :) | 21:53 |
| corvus | SpamapS: we have a test framework now for doing that, though it's using bubblewrap, not nullwrap. so we're running evil playbooks and expecting failure. | 21:54 |
| corvus | SpamapS: https://review.openstack.org/552075 expands on that and adds more tests | 21:54 |
| SpamapS | I suppose you can still detect whether bubblewrap or ansible wrappers saved you | 21:54 |
| corvus | SpamapS: oh... to test bwrap. hrm. yeah, the tests we have now all hit the ansible checks, since that's layer 1... i don't think we have a way of testing the *wrap layer without the ansible checks. | 21:55 |
| SpamapS | Anyway I'm still fine with security messages saying "... as long as you didn't disable bubblewrap" which to me is the same thing as "as long as you didn't aim the gun at your foot and pull the trigger, you should be ok." | 21:56 |
| SpamapS | what if we rename nullwrap to footgun ? | 21:56 |
| clarkb | corvus: tobiash can you see the comment on 552075? Other than that I think the change is fine | 21:56 |
| clarkb | want to make sure I'm not missing anything there | 21:57 |
| corvus | SpamapS: when we created openstack package repos that said "testing-only-do-not-use-in-production" they got used in production. | 21:57 |
| tobiash | looking | 21:57 |
| corvus | SpamapS: this, in part, informs my position on the footgun-availability subject :| | 21:57 |
| SpamapS | footgun-this-time-we-mean-it ;) | 21:57 |
| SpamapS | Another though.. maybe disable it in master, and wait for screams? | 21:58 |
| SpamapS | thought | 21:58 |
| SpamapS | LIke, I know I'm not dependent on it at all. | 21:58 |
| SpamapS | Sounds like none of us need it. | 21:58 |
| tobiash | clarkb: oh yes, that was from the pre-mount-stuff-into-bwrap phase | 21:58 |
| SpamapS | so, yeah, chuck it, and if somebody comes out of the wood work.. | 21:58 |
| corvus | yeah, we can go ahead and put it out there and see | 21:58 |
| tobiash | clarkb: would you be ok with a followup which removes hte regexp? | 21:58 |
| clarkb | tobiash: sure, I'll go ahead and approve the existing change then | 21:59 |
| tobiash | or shall I update this now? | 21:59 |
| clarkb | tobiash: its fine, I've approved it, will review your followup change as soon as it is up | 21:59 |
| corvus | meeting time in #openstack-meeting-alt | 22:00 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove superfluous regexp matcher from assemble test https://review.openstack.org/552154 | 22:01 |
| tobiash | clarkb: ^ | 22:01 |
| clarkb | corvus: tobiash I am approvint the allow trusted paths in is safe path change now | 22:05 |
| corvus | clarkb: thx! | 22:05 |
| tobiash | :) | 22:06 |
| openstackgerrit | Merged openstack-infra/zuul master: Add further test cases to tox-remote https://review.openstack.org/552075 | 22:13 |
| *** myoung|ruck is now known as myoung|bbl | 22:16 | |
| openstackgerrit | Merged openstack-infra/zuul master: Allow trusted for find_needle https://review.openstack.org/552127 | 22:24 |
| tobiash | corvus: that merged ^ so I can press 'send' now? | 22:25 |
| corvus | tobiash: i think so! i'll have to restart our executors later.. either after the meeting or maybe tomorrow. best not wait for that. | 22:28 |
| tobiash | ok | 22:28 |
| clarkb | corvus: when meeting is done https://review.openstack.org/#/c/552154/ is a quick followup from the earlier patch set | 22:57 |
| corvus | done! | 22:58 |
| clarkb | corvus: following up from the meeting does zuul error by default if you don't have bwrap installed or will it fall back gracefully? | 23:04 |
| corvus | clarkb: it'll error; you have to change the config if you want to use nullwrap | 23:06 |
| clarkb | in that case I think we probably wouldn't have needed a cve even if there had been a zuul release | 23:07 |
| clarkb | since its very explicitly this is how it should work and if you change it we are telling you it is insecure | 23:07 |
| corvus | good | 23:09 |
| corvus | i feel like we're skating by though, and i'd like to confront the issue head-on, so i think i'll still start a discussion about dropping nullwrap | 23:10 |
| corvus | i created a zuul-security team in storyboard | 23:11 |
| corvus | i wonder if there's any way for a user to determine whether they are on a team? | 23:11 |
| *** rlandy is now known as rlandy|bbl | 23:12 | |
| clarkb | ++ to dropping nullwrap. This case seems to be a good example of why we bwrap | 23:14 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove superfluous regexp matcher from assemble test https://review.openstack.org/552154 | 23:16 |
| pabelanger | looks like I missed some excitement today, and meeting | 23:33 |
| pabelanger | Back online in the morning, but https://review.openstack.org/551015/ is an interesting patch for zuul-fingergw I'd love some feedback on, specifically why that is the fix for OVH. Can't figure out why. | 23:34 |
| clarkb | pabelanger: I'd guess we don't get working ipv6 on ovh so that doesn't work reliably? | 23:35 |
| clarkb | pabelanger: maybe check sysctl -a output for ipv6 things? | 23:36 |
| pabelanger | clarkb: yah, maybe. For some reason, it seems limited there | 23:36 |
| pabelanger | sure | 23:36 |
| clarkb | pabelanger: but ya I would look at ifconfig/ip addr output and sysctl -a | grep ipv6 | 23:39 |
| pabelanger | clarkb: k, trying to collect that now | 23:40 |
| clarkb | also is it a specific distro behavior? | 23:40 |
| clarkb | possible that one distro changed how they set up ipv6? | 23:40 |
| pabelanger | clarkb: no, seems to fail on fedora-27, ubuntu-xenial and ubuntu-bionic | 23:40 |
| abelur | I am trying to setup zuul locally ... | 23:45 |
| abelur | any head start pointers on setting up zuul locally and getting started with the dev process? | 23:45 |
| clarkb | abelur: https://docs.openstack.org/infra/zuul/admin/zuul-from-scratch.html is the deployment guide | 23:46 |
| abelur | clarkb: thanks ... is it possible to setup the env with docker? | 23:48 |
| clarkb | abelur: tobias is running it under k8s/openshift, so yes, but I'm not sure there is a documented method for that yet | 23:49 |
| pabelanger | clarkb: http://paste.openstack.org/show/698790/ | 23:52 |
| pabelanger | node is also held | 23:52 |
| clarkb | that looks fine | 23:53 |
| clarkb | net.ipv6.conf.all.disable_ipv6 = 0 in particular | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!