| jeblair | pabelanger, mordred: i need to add one more thing to the secrets change (and also fix pep8 -- apparently tests pass though), so i've marked it wip; i'll fix it up tomorrow. | 00:19 |
|---|---|---|
| jeblair | pabelanger, mordred: i think we need to expose the job.untrusted_secrets attribute so that we can specify that, for example, the tarball upload job must not be run in check (even though it now *could* because it only has trusted secrets). of course, defining it in openstack-zuul-jobs would achieve the same effect, but that would only be as a side effect -- we shouldn't have to make that choice because of that. | 00:21 |
| jeblair | (otherwise you could write a check job to upload something to tarballs.o.o which is bonkers) | 00:22 |
| jeblair | anyway, that's a very small change, so if you wanted to go ahead and become familiar with the change, that's okay. just wanted to let you know there's one more revision coming, but it'll be small. | 00:23 |
| *** xinliang has joined #zuul | 00:45 | |
| *** xinliang has quit IRC | 00:45 | |
| *** xinliang has joined #zuul | 00:45 | |
| *** jkilpatr has quit IRC | 01:04 | |
| tobiash | jeblair: feel free to do anything you need with these | 04:38 |
| *** amoralej|off is now known as amoralej | 07:23 | |
| *** xinliang has quit IRC | 07:48 | |
| *** xinliang has joined #zuul | 08:00 | |
| *** xinliang has quit IRC | 08:00 | |
| *** xinliang has joined #zuul | 08:00 | |
| *** electrofelix has joined #zuul | 08:19 | |
| *** xinliang has quit IRC | 08:23 | |
| *** xinliang has joined #zuul | 08:39 | |
| *** xinliang has quit IRC | 08:39 | |
| *** xinliang has joined #zuul | 08:39 | |
| *** xinliang has quit IRC | 08:52 | |
| *** xinliang has joined #zuul | 09:06 | |
| *** xinliang has quit IRC | 09:06 | |
| *** xinliang has joined #zuul | 09:06 | |
| *** xinliang has quit IRC | 09:31 | |
| *** xinliang has joined #zuul | 09:44 | |
| *** xinliang has quit IRC | 09:44 | |
| *** xinliang has joined #zuul | 09:44 | |
| *** jkilpatr has joined #zuul | 10:59 | |
| *** xinliang has quit IRC | 12:20 | |
| *** xinliang has joined #zuul | 12:33 | |
| *** Diabelko has joined #zuul | 12:41 | |
| Diabelko | hello \o | 12:41 |
| *** amoralej is now known as amoralej|lunch | 13:31 | |
| *** dkranz_ has joined #zuul | 13:31 | |
| mordred | jlk: ping whence you awaken | 14:14 |
| *** jkilpatr has quit IRC | 14:20 | |
| *** jkilpatr has joined #zuul | 14:20 | |
| Diabelko | when I go to zuul:8001/status/change/1337,1 during the job I can see json with status just fine | 14:21 |
| Diabelko | however, when the job finishes I only see [], not the last status | 14:22 |
| Diabelko | is this expected behavior? | 14:22 |
| Diabelko | and, is there a way to change it? | 14:22 |
| *** amoralej|lunch is now known as amoralej | 14:24 | |
| mordred | Diabelko: it is the expected behavior - the status is only the current live status. however, there is a patch up for review from tristanC to add some endpoints and a dashboard to see historical data | 14:44 |
| mordred | Diabelko: so once that's landed it should be easy to accomplish the thing you're trying to do | 14:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add sphinx-autodoc-typehits sphinx extension https://review.openstack.org/492557 | 14:46 |
| mordred | jeblair: ^^ I came across that while looking at something else. we can't land it until there's a release containing a PR I submitted - so I added the PR link to the depends-on | 14:46 |
| jeblair | mordred: cool; should we wip that for now? | 14:48 |
| Diabelko | mordred: okay, because what I'm trying to do is to get that nice fancy table with build status review.openstack.org has | 14:53 |
| Diabelko | and I'm running into a lot of issues there | 14:53 |
| Diabelko | but good to know this behavior is expected ;) | 14:53 |
| jlk | mordred: pong, but just barely | 15:35 |
| clarkb | new gerrit supports CI system comments natively now so in theory building that table becomes much easier once on newer gerrit (doesn't have to rely on magic strings that we just assume to be in place but may still need hacky js? not sure how much of the hacks have to stay around) | 15:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Bind secrets to their playbooks https://review.openstack.org/492307 | 16:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Expose final job attribute https://review.openstack.org/479382 | 16:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove 'auth' dict from jobs https://review.openstack.org/492309 | 16:14 |
| jeblair | pabelanger, mordred, clarkb, SpamapS, jlk: ^ those should be ready to go now; i'd appreciate careful review of that since it alters secrets substantially. | 16:14 |
| jeblair | mordred, pabelanger: any job stuff i can help move along while i await reviews? | 16:15 |
| pabelanger | sure, will look shortly | 16:17 |
| mordred | jeblair: what's the deal with the --dir removal in the bubblewrap invocation? | 16:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename allow-secrets to allow-untrusted-secrets https://review.openstack.org/492614 | 16:23 |
| jeblair | mordred: gimme a sec to come up with a satisfactory answer :) | 16:25 |
| jeblair | mordred: i was looking at the directories we bind in, and that seemed redundant. it means "create a directory at <foo>" but right after it, we bind in an already existing directory. | 16:27 |
| jeblair | mordred: strictly speaking, it's probably an unrelated change and maybe could have been its own change. | 16:27 |
| jeblair | i just noticed it since i was reworking the directory structure a bit | 16:28 |
| mordred | jeblair: not a big deal - just curious (it didn't seem related, but then I was wondering if I was missing something) | 16:29 |
| jeblair | i'll leave a note | 16:30 |
| jeblair | also worth noting there: i am changing the cwd from jobdir/ to jobdir/work/. i don't think that will affect anything, but worth considering. | 16:30 |
| jeblair | (it seems that ansible uses the playbook directory as cwd anyway; that's why i don't think it will affect much) | 16:31 |
| pabelanger | +2 on the stack, excited to try them out now | 16:32 |
| mordred | jeblair: bikeshed - how about "allow-secrets-in-untrusted" instead? allow-untrusted-secrets reads weird enough that it hurts my head a smidge | 16:35 |
| mordred | or even allow-secrets-in-untrusted-context if we don't mind the super-long name | 16:35 |
| jeblair | mordred: agree with the head hurting. | 16:36 |
| jeblair | mordred: i wonder if we can come up with a phrasing that we can use on the job as well. because basically, it's: pipeline.allow-untrusted-secrets means it can run jobs with job.untrusted-secrets. so while the phrase 'untrusted secrets' kinda needs to be unpacked a bit, at least the logic there seems pretty clear. | 16:39 |
| jeblair | mordred: i don't mind the superlongname -- pipeline.allow-secrets-in-untrusted-context alone works for me, but i worry it increases the distance from job.untrusted-secrets. is that okay? or should we do job.secrets-in-untrusted-context or something? | 16:40 |
| jeblair | (i was also thinking maybe we could use new terminology, like 'restricted secrets' or something; not sure if that helps or not) | 16:41 |
| mordred | jeblair: yah - maybe that - because it makes both things extra clear on a topic where we likely want to minimize confusion | 16:41 |
| mordred | (job.secrets-in-untrusted-context and pipeline.allow-secrets-in-untrusted-context) | 16:41 |
| mordred | we need more english words | 16:42 |
| jeblair | or maybe we should just switch to emoji | 16:42 |
| mordred | jeblair: I'm 100% certain that would fix everything | 16:46 |
| *** bhavik1 has joined #zuul | 16:51 | |
| *** electrofelix has quit IRC | 17:08 | |
| jeblair | pabelanger: were you going to remove the openstack dir from 491093? | 17:10 |
| pabelanger | jeblair: yes, I'll push up a change in a few minutes | 17:13 |
| *** bhavik1 has quit IRC | 17:20 | |
| pabelanger | jeblair: updated | 17:20 |
| jeblair | pabelanger, mordred: ^ +2 | 17:21 |
| pabelanger | jeblair: so, more about openstack-publish-tarball job. Once we approve your secrets change, you would like to move the GPG into a secret, correct? | 17:21 |
| jeblair | pabelanger: i think the gpg key, ssh key, and pypi creds can all be secrets at that point | 17:24 |
| pabelanger | jeblair: okay, so with the openstack-publish-tarball jobs, we don't want that to run in check / gate right? And assume it is okay in experimental / post / release / perodic, etc | 17:25 |
| pabelanger | basicaly trying to establish with pipelines have secrets | 17:26 |
| pabelanger | which* | 17:26 |
| jeblair | pabelanger: with the new changes, all pipelines can use secrets in the trusted context | 17:26 |
| jeblair | pabelanger: so we can use secrets for the log server too | 17:26 |
| pabelanger | okay, that's what I wanted to confirm. I thought that was the case | 17:27 |
| mordred | pabelanger: oh - so - re: yesterday's discussion about the python to extract the name ... | 17:28 |
| jeblair | pabelanger: when the new change lands, we should set untrusted-secrets (or whatever we end up calling it) on publish-openstack-tarball to ensure it doesn't run in check | 17:28 |
| mordred | pabelanger: I think I was wrong and I think you don't need it | 17:28 |
| pabelanger | mordred: okay, cool. | 17:28 |
| pabelanger | jeblair: ack | 17:28 |
| jeblair | pabelanger, mordred: with new secrets stuff, is it possible to have a single job do tarball build, sign, and upload to tarballs and pypi? | 17:29 |
| dmsimard | jeblair, mordred: so that weird issue from #openstack-infra was about one of the zuul launchers being out of date -- do you think this is something we could detect pre-emptively? The launchers register themselves to zuul, right ? Is there some sort of sanity check before jobs are sent to it ? | 17:30 |
| dmsimard | But, as usual, you can reply that this isn't in an issue with zuul v3 and that's cool too | 17:30 |
| jeblair | dmsimard: i'm going to do that ^ :) | 17:31 |
| dmsimard | sweet | 17:31 |
| mordred | dmsimard: yah - it's fixed in v3 :) | 17:31 |
| jeblair | dmsimard: (and it's happened once in 1.5 years of zuul v2.5) | 17:31 |
| pabelanger | jeblair: I think so, we should attempt it for sure | 17:31 |
| dmsimard | you know you guys are really hyping zuul v3, it's like everything is fixed, all rainbows and unicorns | 17:32 |
| dmsimard | there is no bugs, only zuul v3 | 17:32 |
| jeblair | dmsimard: we try to learn as we go :) but zuulv3 will introduce entirely new, exciting bugs | 17:33 |
| jeblair | pabelanger: if so, that job will need to be a trusted job as well -- adding inventory hosts and running gpg on the executor are both things that need to happen in config repos | 17:34 |
| dmsimard | jeblair: oh, sure -- it'll solve issues and create new ones | 17:34 |
| dmsimard | jeblair: I was promised that containers would solve all my problems | 17:34 |
| dmsimard | only they didn't tell me all about the "other" problems | 17:35 |
| mordred | jeblair: re: build/sign/upload - yeah, I think so? because I think we can define the bits that do the signing and uploading in project-config and have them grab secrets - and then it should be safe for those jobs to be used by people in their untrusted repos | 17:35 |
| jeblair | pabelanger: why don't you etherpad a sketch of the job real quick so you, mordred, and i can all get on the same page before starting on that. | 17:36 |
| pabelanger | jeblair: I still think we have some issues running stuff on the executor today inside bubblewrap. That was the main reason we did the 2 stage tarball | 17:36 |
| mordred | oh. right. software install needs | 17:36 |
| jeblair | pabelanger: oh -- is it that we can't run gpg in bwrap? | 17:36 |
| mordred | it's getting the dependency installed in the first place I believe | 17:37 |
| pabelanger | I haven't tested gpg, but we cannot run even run apt-get install foo today | 17:37 |
| jeblair | (when we make these jobs, let's leave comments about this) | 17:37 |
| jeblair | twine! | 17:37 |
| mordred | it's not bwrap itself that's the problem - it's job content needing software to be installed on the executor | 17:37 |
| jeblair | that was the thing wasn't it | 17:37 |
| mordred | jeblair: well, twine can be installed locally - pip install --user twine should be fine | 17:37 |
| pabelanger | right, we needed to pip install twine, and it failed on lsb_release | 17:37 |
| mordred | ah. lsb_release | 17:37 |
| mordred | and that was becuase lsb_release wasn't installed or because it was missing files? | 17:38 |
| pabelanger | so, I think we should try and do executor, but that is more yak shaving and not something we want to do for PTG? | 17:38 |
| pabelanger | mordred: bwrap didn't bindmount in /etc/lsb-release files | 17:38 |
| jeblair | pabelanger: did you look into what would be needed on both fedora and ubuntu? | 17:39 |
| jeblair | to make pip install work? | 17:39 |
| mordred | gotcha. so we should be able to put those into the optional bind-mount-if-there list, right? I mean, seriously,lsb_release not working is a fundamental break from my pov | 17:39 |
| pabelanger | jeblair: /etc/lsb-release for debuntu and /etc/lsb-release.d;/etc/distro-release (IIRC) for redhat | 17:39 |
| pabelanger | mordred: ya, I was looking at zuul.conf changes, we be talked out selfs out of that approach for now | 17:40 |
| jeblair | pabelanger: okay, so now that we know both of those, yes, i think we can jush hard-code those (conditionally) in zuul | 17:40 |
| mordred | yah - I agree with hard-coding those - they're fundamental parts of a working system | 17:40 |
| *** amoralej is now known as amoralej|off | 17:41 | |
| pabelanger | k, I had a patch up, but abandoned. Let me find it again | 17:41 |
| mordred | cool | 17:41 |
| mordred | pabelanger, jeblair: I could be wrong, but I think continuing to yak-shave on this isn't a bad thing - it's one of the new things we want to be able to do well, and if we can have even just one fully-plumbed-through example that would be great, yeah? | 17:42 |
| pabelanger | https://review.openstack.org/490200 was debuntu only, will need to update for redhat | 17:42 |
| jeblair | mordred: yes i think this is important. | 17:43 |
| pabelanger | mordred: well, I was thinking another way would have been to have bwrap maybe just bindmount a ubuntu-minimal (xenial) DIB tarball and we didn't have to worry about picking the individual files for each specific operating system. We'd basically say, zuul-executor bubblewrap was always operating system X | 17:44 |
| jeblair | pabelanger: we explicitly decided not to do that because it's exceedingly complicated. i don't think anything has changed with regard to that. | 17:45 |
| jeblair | that was something we discussed at length in the bubblewrap spec | 17:45 |
| pabelanger | Right, it became an issue distributing things | 17:45 |
| jeblair | pabelanger, mordred: can we come up with a plan for the tarball and release job(s)? | 17:48 |
| jeblair | so that we know what jobs we're writing, what they are going to do, and what features/bugs are blocking them? | 17:48 |
| mordred | jeblair: ok. I need to put the migration script aside and switch my brain to something else for a bit - so I'm going to context swtich back to job hacking (as much as I love being a relational join optimizer) | 17:49 |
| mordred | jeblair: yes - I think that's a great idea | 17:49 |
| pabelanger | sure, I was going to look at moving ssh key into secrets first | 17:50 |
| jeblair | pabelanger, mordred: https://etherpad.openstack.org/p/mVSVwG4xos | 17:51 |
| jeblair | pabelanger: that should all probably be a role, right? | 17:52 |
| pabelanger | yes, if we expect another playbook not to inherit the publish-openstack-playbook. But still needs to live in project-config I think | 17:53 |
| *** jtanner has joined #zuul | 17:54 | |
| *** openstackgerrit has quit IRC | 19:03 | |
| jeblair | pabelanger, mordred: okay, i'll start on my items after lunch | 19:04 |
| mordred | jeblair, pabelanger: ++ - and thanks, that was fun | 19:05 |
| jeblair | w00t! | 19:05 |
| jeblair | i'm glad we did that; it's complicated :) | 19:05 |
| pabelanger | ITS HAPPENING | 19:07 |
| *** openstackgerrit has joined #zuul | 19:26 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Bind secrets to their playbooks https://review.openstack.org/492307 | 19:26 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove 'auth' dict from jobs https://review.openstack.org/492309 | 19:28 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Expose final job attribute https://review.openstack.org/479382 | 19:28 |
| jlk | mordred: what did you want? | 19:52 |
| *** jkilpatr has quit IRC | 20:11 | |
| jeblair | mordred, pabelanger: two more bikeshed colors to consider on https://review.openstack.org/492614 | 20:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Bindmount /etc/lsb-release into bubblewrap https://review.openstack.org/490200 | 20:28 |
| *** dkranz_ has quit IRC | 20:37 | |
| clarkb | jeblair: lsb-release isn't something set by the container itself if eg you run a fedora container on ubuntu? | 20:44 |
| jeblair | clarkb: bubblewrap is pretty light-weight containment; it doesn't use an image, it just sets up namespaces. so there's no reason for it to supply a file like that on its own. | 20:47 |
| clarkb | oh right its just bind mounting in /bin and /usr/bin and such | 20:48 |
| jeblair | yep, so these amount to some missing files that some already-installed programs expect | 20:48 |
| pabelanger | jeblair: sorry, left a -1 on release files | 20:49 |
| pabelanger | /etc/distro-release should have been "/etc/distrib-release" | 20:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Bindmount /etc/lsb-release into bubblewrap https://review.openstack.org/490200 | 20:55 |
| jeblair | pabelanger: ^ how's that approach? | 20:55 |
| pabelanger | +2 | 20:59 |
| *** jkilpatr has joined #zuul | 21:00 | |
| mordred | jeblair: lookg great - aslo, your other two paint colors look great too | 21:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add Zuul to gate pipeline https://review.openstack.org/492689 | 21:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add zuul-jobs to gate pipeline https://review.openstack.org/492691 | 21:14 |
| mordred | jeblair: https://review.openstack.org/#/c/492689 seems to be unhappy with syntax and depends-on - but I think that's expected, yeah? | 21:17 |
| jeblair | mordred: right -- project-config doesn't get speculative execution, so that change has to land first | 21:18 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Bindmount /etc/lsb-release into bubblewrap https://review.openstack.org/490200 | 22:13 |
| *** openstackgerrit has quit IRC | 22:18 | |
| mordred | jeblair: I rechecked https://review.openstack.org/#/c/492689/ and it's now green | 22:38 |
| jeblair | mordred: i saw, and 91 too | 22:38 |
| jeblair | mordred: i just rechecked 92 | 22:38 |
| mordred | jeblair: we just simul-rechecked 92 | 22:39 |
| jeblair | mordred: i reckon those are ready for +3 now | 22:40 |
| mordred | yup | 22:40 |
| mordred | I have +2'd 89 and 91 | 22:40 |
| mordred | jeblair: maybe these are fine for just the two of us to nudge in | 22:40 |
| jeblair | i think so, they seem pretty procedural | 22:41 |
| jeblair | mordred: 92 is green | 22:42 |
| mordred | +A | 22:45 |
| *** openstackgerrit has joined #zuul | 22:57 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add zuul-jobs to gate pipeline https://review.openstack.org/492691 | 22:57 |
| pabelanger | \o/ | 22:58 |
| pabelanger | awesomeness | 22:58 |
| mordred | pabelanger: IKR??? | 23:01 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add Zuul to gate pipeline https://review.openstack.org/492689 | 23:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Bindmount /etc/lsb-release into bubblewrap https://review.openstack.org/490200 | 23:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 23:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Rename tox_command_line in docs to tox_extra_args https://review.openstack.org/489758 | 23:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: WIP job for non-OpenStack sphinx build https://review.openstack.org/492709 | 23:46 |
| mordred | jeblair: ^^ rebased that stack which had gone into conflict with the gate addition | 23:46 |
| jeblair | mordred: cool, i'd love to have that | 23:47 |
| *** pabelanger has quit IRC | 23:47 | |
| *** pabelanger has joined #zuul | 23:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Use shell for apt-get update https://review.openstack.org/492716 | 23:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Use shell for apt-get update https://review.openstack.org/492716 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!