| *** jkilpatr has quit IRC | 00:08 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Wrap handle_keys with debug statement https://review.openstack.org/489777 | 00:13 |
|---|---|---|
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix webapp path parsing https://review.openstack.org/489779 | 00:13 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Simplify github status descriptions https://review.openstack.org/489767 | 00:42 |
| *** jkilpatr has joined #zuul | 01:00 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Support fedora-26 for nodepool dsvm job https://review.openstack.org/482942 | 01:13 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make github ssl verification configurable https://review.openstack.org/489573 | 01:14 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add max-nodes-per-job tenant setting https://review.openstack.org/489481 | 01:14 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use zuul attributes in nodeset section https://review.openstack.org/489506 | 01:14 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add a short name to the project in the inventory https://review.openstack.org/488877 | 01:15 |
| *** jkilpatr has quit IRC | 01:17 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use project object in _uploadPack in gerrit driver https://review.openstack.org/488560 | 01:27 |
| *** https_GK1wmSU has joined #zuul | 01:50 | |
| *** https_GK1wmSU has left #zuul | 01:51 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Support fedora-26 for nodepool dsvm job https://review.openstack.org/482942 | 04:30 |
| tobiash | jlk: sorry, that was too late for me (utc+2) | 05:12 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Don't ignore inexistent jobs in config https://review.openstack.org/488758 | 05:18 |
| tobiash | jeblair: fixed the broken test case ^^^ | 05:18 |
| tobiash | jlk: I like your change about the status descriptions :) | 05:22 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Simplify github status descriptions https://review.openstack.org/489767 | 05:31 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Simplify run tox task https://review.openstack.org/487551 | 05:44 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Rename Node.hold_reason to 'comment' https://review.openstack.org/489717 | 06:33 |
| *** hashar has joined #zuul | 07:35 | |
| *** amoralej|off is now known as amoralej | 07:44 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix github dependent pipeline with merge https://review.openstack.org/489925 | 08:30 |
| tobiash | jlk: that fixes broken github gating with merge for me ^^^ | 08:31 |
| *** openstackgerrit has quit IRC | 08:33 | |
| *** _ari_|gone is now known as _ari_ | 11:23 | |
| *** jkilpatr has joined #zuul | 11:24 | |
| *** dkranz_ has quit IRC | 12:10 | |
| *** amoralej is now known as amoralej|lunch | 12:18 | |
| *** electrofelix has joined #zuul | 12:26 | |
| electrofelix | for zuulv3 is it intended to allow multiple sources (gerrit, github, etc) into the same pipeline? or will pipelines from different sources always need to be separate? | 12:26 |
| electrofelix | based on what I thought was going to be supported for cross-project-dependencies I was assuming it was planned to be the former at some point? | 12:27 |
| electrofelix | but I can't find anything definitive | 12:27 |
| tobiash | electrofelix: zuulv3 already supports multi-source pipelines | 12:28 |
| *** amoralej|lunch is now known as amoralej | 12:29 | |
| tobiash | electrofelix: but cross project dependencies are currently limited to a single source, but cross source dependencies will probably be supported with the final release of zuulv3 | 12:29 |
| electrofelix | tobiash: oh, that's very nice to know. So a single check pipeline can be used for patches from either gerrit or github? instead of needing two different ones? | 12:30 |
| tobiash | electrofelix: yes | 12:30 |
| tobiash | electrofelix: https://github.com/openstack-infra/project-config/blob/master/zuul.yaml | 12:31 |
| tobiash | electrofelix: this is already an example of a multi-source check pipeline | 12:31 |
| tobiash | electrofelix: you just have to specify the triggers, requirements, reporters for each source in the pipeline | 12:32 |
| electrofelix | thanks for the update, that example will help immensely | 12:33 |
| electrofelix | is there currently any reason not to run zuul v3 besides that it's assumed not 100% stable? any features that are missing that were in v2? | 12:34 |
| mordred | electrofelix: mostly documentation and we still might break things on you | 12:37 |
| mordred | electrofelix: it's working great though | 12:39 |
| mordred | electrofelix: also, as tobiash said, cross-source depends are definitely on the roadmap | 12:39 |
| electrofelix | as we use a docker image from a specific SHA1 that would only happen when we re-spun to a newer SHA1 | 12:39 |
| tobiash | electrofelix: so you should not run a productive zuulv3 environment yet ;) | 12:40 |
| electrofelix | not needing to maintain a separate set of pipelines for Gerrit and GitHub changes would tidy things up immensely for us | 12:40 |
| electrofelix | in the mean time I might look at some local tweaks to make it easier to deal with the consolidation in the future | 12:41 |
| *** openstackgerrit has joined #zuul | 12:42 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches https://review.openstack.org/490009 | 12:42 |
| tobiash | jeblair, jlk: what do you think about ^^^? | 12:42 |
| tobiash | jeblair, jlk: that would fix my problem with a 'branch and pull' model on a shared repository | 12:43 |
| tobiash | jeblair, jlk: I think this is more generic and more secure in comparison with a manually managed list of branch filters in the tenant config (which might still be a thing we want, e.g for supporting experimental branches) | 12:44 |
| *** amoralej is now known as amoralej|lunch | 12:47 | |
| mordred | tobiash: oh, I like that | 12:51 |
| tobiash | :) | 12:51 |
| mordred | tobiash: I'm pondering the config option for things like openstack-zuul and thinking about us potentially wanting to be able to override that setting on a per-repo basis | 12:52 |
| tobiash | mordred: override per repo might be a good idea, that would probably go into tenants.yaml | 12:54 |
| mordred | tobiash: like - if openstack wanted to run some tests on PRs to github.com/pypa/pip but that wasn't set up with branch protection, but we still wanted to go generally with the 'only protected branches' approach | 12:54 |
| mordred | tobiash: ++ | 12:54 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches https://review.openstack.org/490009 | 12:57 |
| *** mnaser has left #zuul | 12:59 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add --detail option to nodepool list command https://review.openstack.org/489720 | 13:17 |
| Shrews | tobiash: is that what you were looking for in the test? ^^^ | 13:18 |
| tobiash | mordred: shitty, if I want to override that in tenant config I'd have to pass the tenant through several layers of the configloader and driver to getprojectbranches | 13:18 |
| tobiash | Shrews: almost, it would be cool, if there also would be an assert for the detailed with 11 columns | 13:20 |
| Shrews | tobiash: not sure i understand. do you want to check that there are exactly 11 columns? | 13:20 |
| tobiash | Shrews: so a test without details which asserts on 5 columns and a test which asserts the detailed list with 11 columns as it was before | 13:21 |
| *** dkranz_ has joined #zuul | 13:22 | |
| Shrews | tobiash: so, a column count? self.assertEquals(num_columns, 5) or self.assertEquals(num_columns, 11) ? | 13:23 |
| tobiash | Shrews: so if I understood correctly this 5 is a column count right? | 13:24 |
| Shrews | tobiash: err, i guess those numbers are off, but yes, column counts | 13:25 |
| Shrews | 8 and 15, i think | 13:25 |
| tobiash | Shrews: https://review.openstack.org/#/c/489720/4/nodepool/tests/test_commands.py@66 | 13:25 |
| tobiash | Shrews: or is this 5 something different? | 13:26 |
| Shrews | tobiash: that's the column containing the value we are verifying. the example you pasted in the comment doesn't really make sense in that respect | 13:26 |
| Shrews | so 5 == status | 13:27 |
| Shrews | we don't want to change that | 13:27 |
| tobiash | Shrews: ah that explains my confusion | 13:27 |
| Shrews | ok, phew. i was getting really confused too :) | 13:28 |
| tobiash | Shrews: ok, then I'm suggesting checking additionally the number of columns for each detailed and non-detailed | 13:28 |
| tobiash | Shrews: that would at least prove that that --detail is different than not supplying --detail | 13:29 |
| Shrews | tobiash: we don't really do that with any other command, and would require changing assert_listed() somehow (which is used in lots of places) | 13:30 |
| Shrews | trying to think of a way to do that that doesn't impact lots of tests | 13:31 |
| *** xinliang has quit IRC | 13:32 | |
| tobiash | Shrews: possibly just a new assert, but if that's too much overhead I'm also ok with not doing this (I'll be testing this automatically from time to time while watching nodepool@work) | 13:33 |
| *** amoralej|lunch is now known as amoralej | 13:37 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add --detail option to nodepool list command https://review.openstack.org/489720 | 13:43 |
| Shrews | tobiash: ^^^ that's the only way i can think of to do what you want there. if that doesn't cut it, i think we should not worry about it. | 13:44 |
| *** xinliang has joined #zuul | 13:44 | |
| tobiash | Shrews: :) | 13:46 |
| tobiash | +2 | 13:46 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add hold job to nodepool list output https://review.openstack.org/489721 | 13:48 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix github dependent pipeline with merge https://review.openstack.org/489925 | 14:32 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Remove getProjectBranches from FakeGithubConnection https://review.openstack.org/490043 | 14:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update tox-tarball to use tox role https://review.openstack.org/489705 | 14:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Log file stats for tox tarball / wheels https://review.openstack.org/490045 | 14:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Use py35 tox envlist for documentation https://review.openstack.org/489707 | 14:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Log file stats for tox tarball / wheels https://review.openstack.org/490045 | 14:51 |
| pabelanger | jeblair: mordred: any issue restarting zuulv3 to pick up public key fixes from yesterday? | 15:12 |
| *** electrofelix has quit IRC | 15:23 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add hold job to nodepool list output https://review.openstack.org/489721 | 15:24 |
| jeblair | pabelanger: go for it | 15:27 |
| jeblair | tobiash: i like the protected branches thing (though i still think we'll probably need a specific branch list at some point). but i think it should probably go into the tenant config. i think that's a setting that is likely to vary from tenant to tenant and even project to project. | 15:29 |
| tobiash | jeblair: yes, currently working on it | 15:31 |
| tobiash | jeblair: so you want it *only* in tenant config? | 15:31 |
| jlk | tobiash: will be reading those changes this morning | 15:33 |
| tobiash | ok | 15:33 |
| jeblair | tobiash: yes i think so. connection doesn't seem like the right level. | 15:37 |
| jlk | hrm. | 15:39 |
| jlk | so putting it in tenant config means that the repo owner can't make that decision and set it for themselves. | 15:40 |
| jlk | it'll have to go through whomever owns the tenant config repo. | 15:40 |
| jlk | tobiash: I'm still really weirded out that 140+ chars overflows 1024 bytes. Doesn't seem right at all, but ¯\_(ツ)_/¯ | 15:41 |
| tobiash | jlk: this is probably a bug in the docs or in our github enterprise version | 15:43 |
| jlk | Fair enough. A pipeline name of over 140 chars would be pretty silly anyway | 15:43 |
| jlk | and it'd make the github UI look weird when displaying the description | 15:43 |
| pabelanger | jeblair: mordred: yay: http://zuulv3.openstack.org/keys/gerrit/openstack-dev/sandbox.pub | 15:46 |
| pabelanger | thanks for the help | 15:46 |
| mordred | pabelanger: woot! | 15:49 |
| pabelanger | mordred: jeblair: I'm having some design troubles with tarball publisher, think we could jump into pbx.o.o so I can bounce some ideas off you? | 16:17 |
| jeblair | pabelanger: wfm | 16:17 |
| mordred | sure | 16:17 |
| pabelanger | k, give me a minute to get the room | 16:18 |
| pabelanger | sip:6000@pbx.openstack.org | 16:21 |
| pabelanger | https://wiki.openstack.org/wiki/Infrastructure/Conferencing for DIDs | 16:21 |
| pabelanger | jeblair: both mordred and I are in the conference room | 16:23 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add --detail option to nodepool list command https://review.openstack.org/489720 | 16:39 |
| tobiash | jeblair, jlk: regarding exclude unprotected branches in tenant config: https://etherpad.openstack.org/p/vQIkn0Y7ya | 16:40 |
| tobiash | jeblair, jlk: would that be ok? | 16:40 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add hold job to nodepool list output https://review.openstack.org/489721 | 16:47 |
| jlk | tobiash: I don't see anything necessarily wrong with that. I can't think of a good way to let a project directly set it, outside of tenant config. | 16:49 |
| tobiash | ok, then I'll go this approach | 16:50 |
| jlk | I also defer to jeblair on it :D | 16:50 |
| Shrews | aaahhh, 'nodepool list' output is so much nicer now | 17:00 |
| mordred | pabelanger, jeblair: https://pypi.python.org/pypi/shade/1.22.2 | 17:04 |
| jeblair | fungi: are there gpg signatures on pypi? | 17:04 |
| jeblair | clarkb: ^? | 17:04 |
| clarkb | ya you can pushed signed packages but I don't think pip verifies them on installation | 17:05 |
| jeblair | clarkb: is there a way to get the signature back if you upload it to pypi? | 17:05 |
| pabelanger | https://pypi.python.org/pypi/pip | 17:05 |
| jeblair | apparently so ^ :) | 17:06 |
| fungi | jeblair: cheeseshop pypi "supports" uploading them but the maintainers have declared it a dead feature they won't support in "warehouse" pypi | 17:07 |
| fungi | so i never bothered to add uploading them there and we only upload them to tarballs.o.o as a result | 17:07 |
| * jeblair head explodes | 17:07 | |
| *** hashar has quit IRC | 17:08 | |
| clarkb | ya I didn't think we were uploading them but didn't realize that was why | 17:09 |
| clarkb | fungi: we relying on on md5 instead? | 17:09 |
| fungi | there have been extensive threads, mostly it boils down to "most users of pypi won't know how to tell if the signature is from someone authorized to provide it" and the lack of interest in implementing additional access controls to prevent uploads for unsigned releases or releases signed by "wrong" keys | 17:09 |
| clarkb | thats a reasonable concern, I just don't know how you do anything better for verifying when people want to do so | 17:10 |
| fungi | there's a separate pep which has been in draft state for years about implementing tuf (the update framework) | 17:10 |
| fungi | https://www.python.org/dev/peps/pep-0458/ | 17:10 |
| fungi | i guess there's also https://www.python.org/dev/peps/pep-0480/ now | 17:11 |
| fungi | one of those the-perfect-is-the-enemy-of-the-good situations | 17:11 |
| fungi | spend years debating the total solution when a partial solution could have at least been put in place in the interim | 17:12 |
| *** harlowja has joined #zuul | 17:13 | |
| fungi | also part of the issue is, i think, that dstufft and maybe others involved in pypa infrastructure harbor a general dislike of openpgp and would rather see it die quietly | 17:14 |
| fungi | but i gather there's a fear that if they implement signature uploading in warehouse, they'll be expected to maintain backward compatibility for that no matter what "better" solution they eventually agree on | 17:15 |
| jeblair | fungi: do you have a few minutes to hop on pbx with us? | 17:15 |
| fungi | sure | 17:15 |
| fungi | 6000? | 17:15 |
| jeblair | (also, anyone else is welcome to join) | 17:15 |
| jeblair | fungi: yup | 17:15 |
| jeblair | the topic is release job logistics and is moving towards signing | 17:16 |
| fungi | https://docs.openstack.org/infra/system-config/signing.html#generation | 17:26 |
| fungi | --export-options export-clean,export-minimal | 17:34 |
| mordred | -rw-rw-r-- 1 mordred mordred 4089 Aug 2 12:35 temp.gnupg | 17:35 |
| *** pbrobinson has quit IRC | 17:39 | |
| *** pbrobinson has joined #zuul | 17:40 | |
| jeblair | https://davesteele.github.io/gpg/2014/09/20/anatomy-of-a-gpg-key/ | 17:42 |
| *** _ari_ is now known as alivigni | 17:48 | |
| *** alivigni is now known as _ari_ | 17:49 | |
| *** amoralej is now known as amoralej|off | 18:08 | |
| fungi | pabelanger: are you going to write up a call summary? | 18:10 |
| pabelanger | fungi: I could | 18:11 |
| fungi | might be a good topic to brain-dump to the ml | 18:11 |
| pabelanger | sure | 18:11 |
| fungi | thanks! | 18:11 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches https://review.openstack.org/490134 | 18:35 |
| tobiash | jeblair, jlk: second try for protected branches ^^^ | 18:36 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches https://review.openstack.org/490134 | 18:45 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches https://review.openstack.org/490134 | 18:47 |
| pabelanger | fungi: summary send to openstack-infra ML | 18:53 |
| pabelanger | sent* | 18:53 |
| fungi | thanks! i readily await finding time to read it ;) | 18:54 |
| mordred | jeblair: are node-less jobs possible? | 19:05 |
| pabelanger | mordred: Ya, found that out yesterday. Job passes by default, but nothing ran because it was missing in inventory | 19:09 |
| jeblair | mordred: yep. intentionally so. any issues should be treated as bugs. | 19:11 |
| pabelanger | https://review.openstack.org/#/c/489689/ ps4 is zuul comment | 19:11 |
| mordred | jeblair: so just nodes: [] ? | 19:14 |
| jeblair | mordred: yep, that's the idea. | 19:21 |
| mordred | kk | 19:26 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add zuul.executor.work_root to inventory https://review.openstack.org/490173 | 19:40 |
| pabelanger | jeblair: mordred: ^would appreciate feedback | 19:41 |
| jeblair | pabelanger: this is so you can make new staging dirs off of work root? | 19:42 |
| jeblair | (just want to make sure there isn't a more specific thing that we should expose) | 19:43 |
| pabelanger | jeblair: ya, was thinking of having work_dir/tarballs on executor | 19:43 |
| jeblair | gotcha. wfm. | 19:43 |
| pabelanger | I thought it was dirty writing into src_root | 19:43 |
| jeblair | yepu | 19:49 |
| mordred | ++ wfm | 19:54 |
| pabelanger | jeblair: mordred: So, I think we might have a few issues running commands on ze01.o.o inside bwrap. We have a very limited /etc folder atm. | 20:03 |
| pabelanger | just testing pip command with bwrap-zuul, and already failing to find lsb_release | 20:03 |
| pabelanger | Ya, starting to think going to be a little harder to run things in bubblewrap as we have it ATM. | 20:16 |
| jeblair | pabelanger: maybe we could make a twine venv and bind-mount it in? | 20:17 |
| pabelanger | I figured I could just bindmount /etc/lsb-release in zuul-bwrap, that will work on ze01.o.o (ubuntu-xenial), but /etc/lsb-release doesn't exist on fedora | 20:17 |
| jeblair | (the twine venv doesn't help us put something in zuul-jobs though) | 20:18 |
| pabelanger | jeblair: what about setting up a static node to do this from? We still plan to have them right? | 20:19 |
| pabelanger | we could then manage that via puppet or ansible-playbook | 20:19 |
| jeblair | pabelanger: if we fail to do this with v3 and dynamic nodes, then we've failed one of the principal goals of v3, so i'd rather not. :) | 20:20 |
| jeblair | pabelanger: it would be better to have chained jobs as we do today, but just with 2 dynamic nodes instead of 1 dynamic+1 static. | 20:21 |
| jeblair | pabelanger: but still, i'd rather try to use the executor and 1 node | 20:21 |
| pabelanger | jeblair: right, that was in fact my first design attempt. 2 nodes to handle it | 20:21 |
| jeblair | pabelanger: want to see what you have to add to bwrap to make pip work? | 20:24 |
| pabelanger | jeblair: I think I can get it working on ubuntu easy, but supporting fedora / centos / other going to get complicated fast | 20:25 |
| jeblair | pabelanger: i think it's worth having the list of those requirements for ubuntu at least. then we can think about whether we want to build this into zuul (in which case it needs to be platform aware) or just leave this as a local config option (site admin bind-mounts whatever files they need) | 20:25 |
| pabelanger | I almost want to say, we should have bubblwrap build up from a ubuntu-minimal DIB or something... but that is more work. | 20:25 |
| pabelanger | jeblair: Ya, some sort of bwrap config file might work also, much more flexable. I'll try to get pip working first | 20:28 |
| jeblair | pabelanger: right. we do already support local ro/rw bind options | 20:28 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add README for tox role https://review.openstack.org/489750 | 20:31 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Update tox-tarball to use tox role https://review.openstack.org/489705 | 20:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use undefined values instead of 40 zeroes https://review.openstack.org/486225 | 20:35 |
| pabelanger | k, got it working on fedora | 20:36 |
| pabelanger | let me test on ubuntu | 20:37 |
| *** hashar has joined #zuul | 20:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Bindmount /etc/lsb-release into bubblewrap https://review.openstack.org/490200 | 20:50 |
| *** https_GK1wmSU has joined #zuul | 20:53 | |
| *** https_GK1wmSU has left #zuul | 20:54 | |
| *** dkranz_ has quit IRC | 20:56 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Log file stats for tox tarball / wheels https://review.openstack.org/490045 | 20:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets https://review.openstack.org/487243 | 20:58 |
| jeblair | pabelanger: why depends-on instead of git parent? | 20:59 |
| jeblair | pabelanger: ignore me :) | 20:59 |
| pabelanger | jeblair: it was likely possible I did it wrong to :) I have to many terminals open ATM | 21:01 |
| jeblair | nope, i just confused some changes | 21:01 |
| pabelanger | jeblair: 490200 will likely break tristanC and softwarefactory too | 21:03 |
| pabelanger | /etc/lsb-release doesn't exist on centos | 21:03 |
| jeblair | 20:28 < jeblair> pabelanger: right. we do already support local ro/rw bind options | 21:04 |
| jeblair | pabelanger: https://docs.openstack.org/infra/zuul/feature/zuulv3/admin/components.html#id7 | 21:05 |
| jeblair | pabelanger: search for ro_paths | 21:05 |
| pabelanger | jeblair: Ah, I see. So, is 490200 needed or should I update our ro_paths in hiera? | 21:06 |
| jeblair | pabelanger: we can make it work now easily by doing the ro_paths. we should check in with monty though because doing too much of this locally may affect our ability to have an effective twine job in zuul-jobs. | 21:07 |
| jeblair | and by monty, i mean mordred. :) | 21:07 |
| pabelanger | ok | 21:08 |
| jeblair | pabelanger: i think regardless it should be fine to add that to ro_paths now to continue working | 21:11 |
| jeblair | (to our local hiera i mean) | 21:12 |
| jeblair | just note that we want to check back in on that as part of the refactor into zuul-jobs | 21:12 |
| pabelanger | jeblair: wfm | 21:12 |
| pabelanger | jeblair: trusted_ro_dirs is the old version right? | 21:15 |
| *** hashar has quit IRC | 21:15 | |
| jeblair | pabelanger: yeah, thought we changed it? | 21:16 |
| pabelanger | ya, still have old way in puppet | 21:16 |
| pabelanger | I can fix | 21:16 |
| pabelanger | ah | 21:17 |
| pabelanger | https://review.openstack.org/#/c/488862/ | 21:17 |
| pabelanger | +3 | 21:17 |
| mordred | what did I do? | 21:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Make timer tests less racy https://review.openstack.org/490208 | 21:19 |
| mordred | pabelanger, jeblair: I'm not sure what I shoudl be commenting on ... | 21:19 |
| jeblair | mordred: in order to pip install twine in bwrap, we need more things. looks like lsb_release is all we need for ubuntu, but different things for different oses. | 21:20 |
| mordred | oh goodie. dare I ask why we need - oh - we need those to be able to pip install ANYTHING don't we? | 21:20 |
| jeblair | mordred: so should we try to handle that automagically in zuul internals, or expect users to add that to their trusted_ro_paths config setting | 21:20 |
| jeblair | apparently | 21:20 |
| mordred | jeblair: I think we should handle it automagically in zuul - having a functional pip in our bubblewrap seems like a thing that is legit to come up for folks | 21:21 |
| mordred | and the solution will be obtuse | 21:21 |
| mordred | "wy can't I pip install" "you are on centos so need to add XXX to trusted and untrusted ro_paths in your config" | 21:21 |
| jeblair | mordred: i agree. my concern with that though is how far we go with that. and whether installing software on the executor is something jobs should be doing. | 21:22 |
| mordred | I think we should keep the bar high - but I think pip is a reasonable thing in such a heavily python system | 21:22 |
| pabelanger | mordred: for that to work, I think we need to pick just 1 operating system inside bwrap | 21:22 |
| pabelanger | otherwise, it will be a lot of cfgmgmt in zuul.bwrap driver | 21:23 |
| jeblair | all considerations of other users aside, bind-mounting a twine venv into openstack's executors seems ideal. that's almost exactly the opposite of what we want from a user-friendly system though. | 21:23 |
| mordred | jeblair: in general I tihnk jobs should not be doing that - but this is a thing we know we need to do for a very normal and basic job in zuul-jobs | 21:23 |
| jeblair | mordred: i agree. i'm worried about what we need to do to distribute a go package. or docker image. | 21:23 |
| jeblair | all of those have equal legitimacy in zuul-jobs, and almost certainly come with their own additional requirements on being able to be done on the executor. | 21:24 |
| mordred | that is an excellent point | 21:24 |
| mordred | WELL ... if we go the other direction ... | 21:24 |
| jeblair | i may be about to walk back from suggesting we do this on the executor. :) | 21:24 |
| pabelanger | jeblair: right, I think to do it write, means we pull in a lot more stuff atm | 21:25 |
| mordred | say for sake of argument that we make a requirements.txt for zuul-jobs into which we put things that need to be installed on the executor for zuul-jobs content to work | 21:25 |
| pabelanger | right* | 21:25 |
| mordred | if twine is installed globally on the executor, then we don't need to bind-mount anything | 21:25 |
| mordred | same with putting gnupg into a bindep.txt for zuul-jobs | 21:26 |
| mordred | it may not be perfect- but it would let us communicate sanely that there are some things that need to be installed on the executor. we could also just add twine and gnupg to zuul's requirements.txt and bindep.txt respectively - but you don't actually need them to run zuul, so that seems weird | 21:27 |
| mordred | OH | 21:27 |
| mordred | we could put them into zuul in an [executor] extra and with an executor tag in bindep | 21:27 |
| pabelanger | but we don't run bindep on ze01.o.o right now | 21:27 |
| mordred | "these are things you need to install on executors in order to fully use zuul-jobs" | 21:28 |
| mordred | pabelanger: right. I mean into a bindep.txt file that is in a repo so that we can communicate to deployers depdendencies they need to install | 21:28 |
| pabelanger | I see | 21:28 |
| jeblair | mordred: it doesn't seem right that zuul-jobs should have any requirements on the executor. it has so many implications for deployment. | 21:28 |
| pabelanger | Hmm, kinda of feels odd to run ansible / puppet to setup executor, do then run ansible to run against the thing you just installed | 21:29 |
| jeblair | i should phrase that as "t doesn't seem right that zuul-jobs should place any requirements on the executor" | 21:29 |
| pabelanger | jeblair: agree | 21:29 |
| jeblair | i mean that in the english sense of the word requirements. | 21:29 |
| jeblair | not as a technical term :) | 21:30 |
| jeblair | (even though they are 1:1 here) | 21:30 |
| mordred | that would potentially mean not doing signing/publication on the executor itself yeah? | 21:30 |
| jeblair | mordred: yeah, i have nearly talked myself out of that | 21:30 |
| pabelanger | or wait until we have something, something docker executor? | 21:30 |
| mordred | pabelanger: I do not want to depend on anything anything docker | 21:30 |
| mordred | especially not if the thing it's solving is 'security' | 21:31 |
| pabelanger | right | 21:31 |
| mordred | ;) | 21:31 |
| pabelanger | but there is discussion about putting executor inside k8s, other tech | 21:31 |
| jeblair | pabelanger: if your real desire is a lighter-weight execution environment than vms, in which case, yes, this would be an appropriate use of that. | 21:31 |
| pabelanger | but, I'm side tracking | 21:31 |
| pabelanger | so, if not twine from ze01.o.o, then multi-node job, is left right? | 21:32 |
| jeblair | pabelanger: i'm not sure what multinode gets us. but two chained jobs like we have now, but with two nodepool nodes rather than static nodes would work. | 21:33 |
| jeblair | (rather than static+nodepool i mean) | 21:33 |
| pabelanger | jeblair: okay, yes. I was thinking that | 21:33 |
| pabelanger | I say, lets start with that | 21:34 |
| pabelanger | see how it looks | 21:34 |
| mordred | jeblair: so an upload to tarballs / download and upload dance? | 21:34 |
| jeblair | mordred: that's still not a great fit for zuul-jobs though, since it requires a tarballs intermediary. | 21:34 |
| mordred | yah | 21:34 |
| mordred | this, I think, requires some amount of thought | 21:34 |
| jeblair | the *real* problem here is that publishing to pypi is so complicated. :) | 21:34 |
| pabelanger | mordred: maybe this is the time we look at better artifact handling then | 21:34 |
| jeblair | pabelanger: i think if we want to open that can now, we should cancel the ptg rollout | 21:35 |
| pabelanger | jeblair: Hmm.. that bad? | 21:36 |
| mordred | yah. I think we can do tarball dance for now - it's openstack specific, but that's fine, they're customer 0 | 21:36 |
| jeblair | pabelanger: designing universal artifact handling for zuul is not something i want to do under deadline. | 21:36 |
| pabelanger | jeblair: no, understood | 21:36 |
| pabelanger | okay, tarball dance once again :) | 21:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow use of the uri module from localhost https://review.openstack.org/490215 | 21:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 21:37 |
| mordred | jeblair: there's both enablement of uri and then fixing that hole ^^ | 21:37 |
| pabelanger | mordred: jeblair: https://review.openstack.org/490132/ gets us uploads to tarballs.o.o and depends-on | 21:38 |
| pabelanger | we'll need to wait until zuul patch lands and reset to test | 21:38 |
| jeblair | mordred: fix looks good but we should have a test for that | 21:41 |
| mordred | jeblair: totally agree - that's next | 21:41 |
| mordred | pabelanger: first patch +A - reading second patch now | 21:41 |
| pabelanger | mordred: thanks | 21:45 |
| mordred | pabelanger: something about the name publish-tox-openstack-tarballs is weird to me ... but I feel like I'm bikeshedding in my head ... so bear with me here for just a sec | 21:46 |
| pabelanger | mordred: please do! | 21:48 |
| mordred | pabelanger: the underlying job is "tox-tarball" - so I kinda think it should maybe be publish-tox-tarball - but then I'm not sure tox-tarball belongs in zuul-jobs as opposed to openstack-zuul-jobs since using a tox venv to make a tarball is fairly openstack specific action | 21:48 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul.executor.work_root to inventory https://review.openstack.org/490173 | 21:49 |
| mordred | but then that makes me wonder what our 'prefix-with-openstack' rules are ... we don'tneed them for everything, really only things that we expect a zuul-jobs version of a thing, yeah? | 21:49 |
| pabelanger | mordred: right, we only need the add_hosts part really in project-config. The rest could be zuul-jobs, however we cannot persist add_hosts across repo | 21:51 |
| pabelanger | add_hosts also needs to be trusted | 21:51 |
| mordred | SOOOOO - if I removed the tox thing from the scenario, I'd say that the general thing a job in zuul-jobs would do is "publish-python-sdist" or "publish-to-pypi" - and publishing a tarball to tarballs.o.o is "publish-openstack-tarball" perhaps ... | 21:51 |
| mordred | pabelanger: well, "tox -evenv -- python setup.py sdist" is totally openstack specific | 21:51 |
| pabelanger | mordred: Right | 21:51 |
| mordred | pabelanger: (also, I'm rambling beecause I don't actually have a full thought ;) ) | 21:52 |
| jeblair | publish-openstack-tarball sounds nice, but what's the name of the job to publish a go tarball to tarballs.o.o? | 21:52 |
| mordred | pabelanger: so - I think maybe just 'publish-openstack-tarball' | 21:52 |
| pabelanger | mordred: that would mean that tox-tarball today is openstack specific, which lives in zuul-jobs | 21:52 |
| jeblair | pabelanger: yeah, mordred's going to try to fix that with a PTI change :) | 21:52 |
| pabelanger | ok | 21:53 |
| pabelanger | so, to recap. Just rename job to publish-openstack-tarball? | 21:53 |
| pabelanger | and keep tox-tarball for now? | 21:53 |
| mordred | pabelanger: yes - I think the other stuff is not necessary to do anything about now | 21:53 |
| jeblair | wait | 21:53 |
| mordred | well - actually | 21:53 |
| mordred | yah | 21:54 |
| jeblair | 21:52 < jeblair> publish-openstack-tarball sounds nice, but what's the name of the job to publish a go tarball to tarballs.o.o? | 21:54 |
| mordred | jeblair had a very good questoin | 21:54 |
| jeblair | someone needs to answer that first :) | 21:54 |
| pabelanger | oh, I missed that | 21:54 |
| mordred | should it be publish-openstack-python-tarball? | 21:54 |
| jeblair | that wfm | 21:55 |
| pabelanger | ok | 21:55 |
| pabelanger | it's the tox part we have an issue with | 21:55 |
| mordred | yah. I don't think tox has much to do with the intent of the job for me at the moment | 21:56 |
| pabelanger | k | 21:56 |
| mordred | the openstack part captures that some of the details about the job are openstack-specific | 21:56 |
| mordred | otherwise patch looks great | 21:57 |
| pabelanger | remote: https://review.openstack.org/490132 Create publish-openstack-python-tarball trusted job | 21:58 |
| mordred | pabelanger: ++ | 21:59 |
| pabelanger | neat! | 21:59 |
| *** jkilpatr has quit IRC | 22:01 | |
| mordred | jeblair: do you remember off the top of your head where the test was you wrote related to testing the action plugins blocked things? | 22:07 |
| jeblair | mordred: https://review.openstack.org/454396 | 22:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add some lookup plugin tests https://review.openstack.org/454396 | 22:08 |
| jeblair | mordred: pushed up a rebased patchset since its test logs have expired anyway ^ | 22:08 |
| mordred | cool | 22:08 |
| jeblair | we'll see where it's failing | 22:08 |
| pabelanger | okay, going to step out for a bit. I'll look at restarting zuulv3.o.o then and continue testing uploads with openstack-dev/sandbox project. | 22:10 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Docs: use zuul:attr in project definition https://review.openstack.org/489774 | 22:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets https://review.openstack.org/487243 | 22:16 |
| jeblair | clarkb: can you give https://review.openstack.org/486225 a quick look? | 22:17 |
| jeblair | clarkb: at least a conceptual review (it has sufficient code review i think) | 22:18 |
| clarkb | looking now | 22:19 |
| clarkb | and done. Straightforawrd so I went ahead and apporved it | 22:20 |
| clarkb | (we need a zuul env shim thing anyways since we remove all the env vars so rewriting that as '0' * 40 if necessary won't be terrible | 22:21 |
| jeblair | clarkb: ++ | 22:21 |
| jeblair | i think very few things use that too | 22:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: use zuul:attr in secrets https://review.openstack.org/490236 | 22:23 |
| jeblair | that ^ is all of config.rst using the new directives | 22:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use undefined values instead of 40 zeroes https://review.openstack.org/486225 | 22:27 |
| *** https_GK1wmSU has joined #zuul | 22:37 | |
| *** jkilpatr has joined #zuul | 22:38 | |
| *** https_GK1wmSU has left #zuul | 22:40 | |
| *** jkilpatr has quit IRC | 22:50 | |
| *** jkilpatr has joined #zuul | 22:50 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Docs: use zuul:attr in secrets https://review.openstack.org/490236 | 22:52 |
| jlk | mordred: jeblair: are you two around? I want to talk about this change allowing URI locally | 23:01 |
| jeblair | jlk: o/ | 23:02 |
| jeblair | jlk: we were thinking it would make the job 'hit the rtfd webhook url' (which is just a get request) easy, and didn't see a way it could reduce security. but maybe we missed something. :) | 23:03 |
| jlk | I looked at the module, it does have a method where it downloads $SOMETHING and writes it to a local file | 23:03 |
| mordred | jlk: oh piddle - I missed that | 23:04 |
| jlk | which I think means it needs to go through the safe paths path | 23:04 |
| jeblair | ah yup | 23:04 |
| mordred | ++ | 23:05 |
| clarkb | could also just shell: curl | 23:05 |
| clarkb | I guess you need to update hte module either way though | 23:05 |
| jlk | do we allow shell locally? | 23:06 |
| mordred | no | 23:06 |
| jlk | right. | 23:06 |
| clarkb | no it would have to remote | 23:06 |
| mordred | so we'd have to special case such a thing | 23:06 |
| mordred | right- the goal here is that for some things, specifically hook-rtfd is the working example - it's actually not necessary to have a remote node at all | 23:07 |
| jlk | With the POST and body argument, you could do a lookup on files, but I think we protect the lookup filter, right? | 23:07 |
| mordred | yes, we protect against lookup with lookup directly | 23:07 |
| jlk | there is a small DoS potential, if they point at a ginormous file, or a purposefully slow link. One ties up bandwidth, the other ties up the worker slot. | 23:08 |
| jlk | though on the latter, there's a global job max run time, right? | 23:08 |
| mordred | yah - but there's WAY better ways to dos a zuul install | 23:08 |
| mordred | yup | 23:08 |
| jlk | and there's a filesystem overflow protection? | 23:08 |
| mordred | jlk: SpamapS just wrote that and I think it landed | 23:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 23:09 |
| mordred | gah. updated the wrong patch in the stack | 23:09 |
| jlk | yup, thought so. | 23:09 |
| jlk | Any way they could force a download to /dev/null ? | 23:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 23:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow use of the uri module from localhost https://review.openstack.org/490215 | 23:09 |
| mordred | jlk: not with the dest path protection | 23:10 |
| jlk | alrighty | 23:10 |
| mordred | jlk: thanks for catching dest! | 23:11 |
| jlk | yay I'm helping! | 23:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add zuul:var directive and role https://review.openstack.org/490246 | 23:13 |
| jeblair | mordred: i think that's where the 'foo[]' thing is really going to come in handy ^ | 23:13 |
| mordred | jeblair: ++ | 23:15 |
| pabelanger | and back, objections for a zuulv3.o.o restart once current jobs finish running? | 23:15 |
| jlk | mordred: so sorry, but uri also accepts all of the file module arguments, so we need to safe path more things. | 23:15 |
| mordred | jeblair: is the eventual goal to merge the zuul-sphinx stuff in zuul into the zuul-sphinx repo? | 23:16 |
| mordred | jlk: it does? | 23:16 |
| jlk | http://docs.ansible.com/ansible/latest/uri_module.html see "other" | 23:16 |
| jlk | "others" | 23:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Ensure ref-updated jobs run with their ref https://review.openstack.org/488216 | 23:16 |
| mordred | jlk: *headdesk* | 23:17 |
| mordred | jlk: although - that now makes me realize that we should ALSO block file:// urls | 23:18 |
| jlk | oh... yeah good thought | 23:18 |
| jeblair | mordred: right now, there's no overlap with what's used in job docs vs zuul docs. that might change after i go back and make zuul job docs nicer (i think some of the things we do in zuul docs would be nice for job/role variables). | 23:18 |
| jlk | er | 23:18 |
| jlk | wait. | 23:18 |
| jeblair | mordred: but i think we'll want to document zuul's own jobs with zuul-sphinx, so first step is to add zuul-sphinx to zuul's requirements and then add to the zuul domain in zuul, instead of colliding with it as we do now. | 23:19 |
| jlk | we want to prevent users from reading content from the filesystem they have access to? | 23:19 |
| jeblair | mordred: if we think that merging the two would be useful, we can port things into zuul-sphinx and out of zuul itself | 23:19 |
| jeblair | but that's easy to sequence like that | 23:19 |
| jeblair | jlk: wow that's subtle. :) | 23:20 |
| jeblair | (others) | 23:20 |
| mordred | jlk: we want to make sure that they don't provide a file url that points to files outside of the 'safe' path | 23:21 |
| mordred | jlk: but there's really no good reason to use the uri module for file locations - so just blocking scheme == 'file' if it's on localhost seems somewhat decent,yeah? | 23:22 |
| mordred | jlk: I mean - the uri module docs say: HTTP or HTTPS URL in the form (http|https)://host.domain[:port]/path | 23:22 |
| mordred | jlk: so it's possible it's already covered in the module | 23:22 |
| * jlk reads the module | 23:22 | |
| jlk | a quick scan does not show a limitation of sheme | 23:25 |
| jlk | what happens if the input a http url to a website they control, which has a redirect to a file:// location? Is that even a thing? | 23:27 |
| mordred | jlk: nice brain | 23:28 |
| mordred | jlk: I have no idea if that's possible or not | 23:29 |
| jlk | It doesn't seem that you're blocked from switching schema in htaccess, I jsut don't know where the "file" bit would be processed. | 23:30 |
| mordred | RedirectHandlerFactory in lib/ansible/module_utils/urls.py | 23:31 |
| pabelanger | okay, since nobody objected going to preform the restart of zuulv3.o.o | 23:32 |
| mordred | jlk: so - the RedirectHandlerFactory has a "safe" redirects setting where it will only do redirects if the method is GET or HEAD | 23:33 |
| mordred | that doesn't full answer your question though | 23:33 |
| jlk | sure, but a GET method, redirected to a file:// path, maybe could read a local file | 23:34 |
| mordred | well - fwiw, it uses urllib2 - so I think I feel a test case coming up i my future | 23:34 |
| mordred | cause I'm honestly fascinated to know what will happen | 23:34 |
| mordred | but I'm going to stop computering for today | 23:35 |
| pabelanger | okay, restart | 23:35 |
| jlk | according to the RFC, a 302 (redirect) sends back a new URI, and the client is supposed to re-do the request using the new URI location. So if it got a file:/// it'd try to do it locally | 23:36 |
| jlk | yeah I'm really curious now too! | 23:36 |
| clarkb | if only little python clients respected things like CORS | 23:42 |
| pabelanger | Hmm, I don't think gearman is responding on zuulv3.o.o | 23:44 |
| pabelanger | I didn't see scheduler pick up configuration | 23:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Use py35 tox envlist for documentation https://review.openstack.org/489707 | 23:56 |
| jlk | I'm not the only one to have thought of this: https://curl.haxx.se/docs/adv_20090303.html | 23:58 |
| pabelanger | okay, odd. got it working. I don't if I didn't shutdown properly or not. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!