| *** dangtrinhnt has joined #zuul | 00:41 | |
| *** armstrongs has joined #zuul | 00:56 | |
| *** swest has quit IRC | 01:37 | |
| *** Goneri has quit IRC | 01:48 | |
| *** rfolco has quit IRC | 01:48 | |
| *** swest has joined #zuul | 01:52 | |
| *** dangtrinhnt has quit IRC | 01:55 | |
| *** dangtrinhnt has joined #zuul | 01:56 | |
| *** dangtrinhnt has quit IRC | 01:57 | |
| *** dangtrinhnt has joined #zuul | 02:04 | |
| *** dangtrinhnt has quit IRC | 02:07 | |
| *** dangtrinhnt_ has joined #zuul | 02:07 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: test-upload-logs-swift: revert download script https://review.opendev.org/715755 | 02:11 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: bulk-download : role with script to download all log files https://review.opendev.org/715756 | 02:11 |
| *** bhavikdbavishi has joined #zuul | 03:25 | |
| *** bhavikdbavishi1 has joined #zuul | 03:30 | |
| *** bhavikdbavishi has quit IRC | 03:31 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:31 | |
| *** dangtrinhnt_ has quit IRC | 03:37 | |
| *** dangtrinhnt has joined #zuul | 03:50 | |
| *** evrardjp has quit IRC | 04:03 | |
| *** dangtrinhnt has quit IRC | 04:08 | |
| *** evrardjp has joined #zuul | 04:10 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 04:13 |
| *** dangtrinhnt has joined #zuul | 04:24 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-sphinx master: Ignore __pycache__ directory when looking at roles https://review.opendev.org/715764 | 04:31 |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #zuul | 04:36 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 04:44 |
| *** saneax has joined #zuul | 04:44 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 04:51 |
| *** dangtrinhnt has quit IRC | 04:58 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 05:00 |
| *** bhavikdbavishi has quit IRC | 05:06 | |
| *** swest has quit IRC | 05:12 | |
| *** swest has joined #zuul | 05:13 | |
| *** bhavikdbavishi has joined #zuul | 05:22 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 05:31 |
| AJaeger | mnaser: the problem was openstack-zuul-jobs, so test that one as well, please | 05:36 |
| *** bhavikdbavishi has quit IRC | 05:44 | |
| *** bhavikdbavishi has joined #zuul | 05:45 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 05:49 |
| *** bhavikdbavishi1 has joined #zuul | 05:56 | |
| *** bhavikdbavishi has quit IRC | 05:57 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 05:57 | |
| *** bhavikdbavishi has quit IRC | 06:01 | |
| *** bhavikdbavishi has joined #zuul | 06:02 | |
| *** dpawlik has joined #zuul | 06:23 | |
| *** bhavikdbavishi has quit IRC | 06:31 | |
| *** bhavikdbavishi has joined #zuul | 06:32 | |
| *** bhavikdbavishi1 has joined #zuul | 06:37 | |
| *** bhavikdbavishi has quit IRC | 06:39 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:39 | |
| *** bhavikdbavishi has quit IRC | 06:51 | |
| *** bhavikdbavishi has joined #zuul | 06:52 | |
| *** jcapitao has joined #zuul | 07:17 | |
| *** tosky has joined #zuul | 07:25 | |
| *** ysandeep|rover is now known as ysandeep|rover|l | 07:25 | |
| *** jcapitao has quit IRC | 07:29 | |
| *** jcapitao has joined #zuul | 07:31 | |
| *** bhavikdbavishi1 has joined #zuul | 07:34 | |
| *** arxcruz|off is now known as arxcruz | 07:35 | |
| *** bhavikdbavishi has quit IRC | 07:35 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 07:35 | |
| *** armstrongs has quit IRC | 07:39 | |
| *** avass has joined #zuul | 07:41 | |
| tobiash | infra-root: when was your last scheduler restart? We've been hit by a significant memleak after our last update on friday (oom within less than a day). | 07:45 |
| tobiash | looks like opendev is not (yet) affected and I'd like to strip down the patches that might have introduced this | 07:46 |
| *** rishabhhpe has joined #zuul | 07:49 | |
| *** jpena|off is now known as jpena | 07:53 | |
| *** bhavikdbavishi has quit IRC | 08:08 | |
| tobiash | right now I suspect https://review.opendev.org/714852 as the source as this is the only real thing we introduced on friday to our zuul | 08:28 |
| *** bhavikdbavishi has joined #zuul | 08:30 | |
| *** harrymichal has joined #zuul | 08:33 | |
| *** bhavikdbavishi has quit IRC | 08:34 | |
| *** ysandeep|rover|l is now known as ysandeep|rover | 08:34 | |
| *** bhavikdbavishi has joined #zuul | 08:36 | |
| *** harrymichal has quit IRC | 08:57 | |
| *** rishabhhpe has quit IRC | 09:13 | |
| *** rishabhhpe has joined #zuul | 09:16 | |
| rishabhhpe | Hello All , Facing issue while fetching and cherry-picking the clone from cinder repo for stable/train release branch .. is there any one else facing the same issue .. i can see last check in done two days back .. and my CI is also failing with same issue from past two days only | 09:20 |
| *** rishabhhpe has quit IRC | 09:25 | |
| *** rishabhhpe has joined #zuul | 09:25 | |
| AJaeger | tobiash: I think we log this via status log, let me check... | 09:26 |
| tobiash | AJaeger: thanks, I think I found the reason (not yet merged change above) | 09:27 |
| AJaeger | tobiash: https://wiki.openstack.org/wiki/Infrastructure_Status has an entry on the 28th of February | 09:27 |
| tobiash | cool, thx | 09:28 |
| AJaeger | rishabhhpe: you asked on #openstack-qa and I think you got the answer there: If you have local content in your checkout copy, it would explain the problem. So, get rid of that | 09:30 |
| rishabhhpe | Ajaeger: thats a fresh VM and it is taking all the stuff while installing the devstack apart from that none other checking i am doing in that repo .. also when same code i am checking out from u release and doing the same no error is coming | 09:32 |
| rishabhhpe | so it seems some wrong check in done for train release .. which is causing the problem .. that is why i posted the same here with new reframed sentence .. so that other can also comment if they are facing the same problem | 09:33 |
| *** sgw has quit IRC | 09:39 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow configure-mirrors to enable extra repos https://review.opendev.org/693887 | 09:40 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow configure-mirrors to enable extra repos https://review.opendev.org/693887 | 09:41 |
| *** harrymichal has joined #zuul | 09:43 | |
| AJaeger | rishabhhpe: Please try reproducing this on a clean tree locally | 09:45 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Improve job and node information banner https://review.opendev.org/677971 | 09:47 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Avoid confusing rsync errors when source folders are missing https://review.opendev.org/670044 | 09:52 |
| *** bhavikdbavishi has quit IRC | 09:53 | |
| *** bhavikdbavishi has joined #zuul | 09:53 | |
| zbr | morning everyone! i wonder if i can get some help with some (simple) reviews, the pile is growing and I already had to drop valid ones | 10:01 |
| zbr | like https://review.opendev.org/#/c/703425/ | 10:01 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: For pre blocks to wrap text https://review.opendev.org/681532 | 10:08 |
| *** dmellado has quit IRC | 10:50 | |
| *** jcapitao is now known as jcapitao_lunch | 11:02 | |
| *** rfolco has joined #zuul | 11:24 | |
| *** jpena is now known as jpena|lunch | 11:34 | |
| *** ysandeep|rover is now known as ysandeep|rover|b | 11:51 | |
| *** rlandy has joined #zuul | 12:01 | |
| *** dmellado has joined #zuul | 12:06 | |
| *** rishabhhpe has quit IRC | 12:09 | |
| *** ysandeep|rover|b is now known as ysandeep|rover | 12:13 | |
| mnaser | AJaeger: https://review.opendev.org/715928 :) | 12:14 |
| AJaeger | mnaser: thanks! I suggest to make it fail, so that we see the reporting as well ;) But let's try first without failure. | 12:18 |
| *** jcapitao_lunch is now known as jcapitao | 12:24 | |
| *** jpena|lunch is now known as jpena | 12:30 | |
| mnaser | AJaeger: i got a failure in openstacksdk, but sure, i can make a failure case in there too | 12:31 |
| AJaeger | mnaser: then we're good | 12:32 |
| mnaser | AJaeger: you can checkout the topic i used for the change :) | 12:33 |
| AJaeger | mnaser: will do later, thanks | 12:33 |
| mordred | mnaser: the sdk patch is so cool | 12:35 |
| jkt | how do I run zuul's own test suite? I started by creating a venv, then running `pip install -e .`, and then `tox -l`, but tox fails with "ERROR: unknown environment 'py3-docker'" | 12:55 |
| jkt | so I tried `pip install docker`, but that doesn't appear to help | 12:55 |
| jkt | do I need docker locally? | 12:55 |
| jkt | ah, well, here's TESTING.rst, never mind | 12:55 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 13:03 |
| *** harrymichal has quit IRC | 13:05 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: install-docker: allow removal of conflicting packages https://review.opendev.org/702304 | 13:05 |
| *** irclogbot_1 has quit IRC | 13:13 | |
| mordred | zbr: in that install-docker patch - docker.io is the debuntu package name - could you add that to the defaults for those? | 13:15 |
| *** guilhermesp has quit IRC | 13:15 | |
| *** jtanner has quit IRC | 13:15 | |
| *** gundalow has quit IRC | 13:15 | |
| *** rlandy has quit IRC | 13:15 | |
| *** rlandy has joined #zuul | 13:16 | |
| *** jtanner has joined #zuul | 13:16 | |
| fungi | tobiash: `stat /var/run/zuul/scheduler.pid` says 2020-03-03 15:24:04.487207572 | 13:19 |
| *** irclogbot_1 has joined #zuul | 13:19 | |
| fungi | doesn't appear we logged the git commit id for that restart at https://wiki.openstack.org/wiki/Infrastructure_Status | 13:21 |
| tobiash | fungi: thanks, on deeper look I saw that we did only incluse the node leak mentioned above into our deployment so that seems to be the source of the memleak (I now also can spot that in the change, but cannot explain why that's leaking 1gb/2h in our deployment) | 13:21 |
| fungi | it's worth noting that our status log there indicates we restarted the scheduler on 2020-02-27 due to "memory pressure" | 13:23 |
| tobiash | I've commented on https://review.opendev.org/714852 with the suspected source of the memleak | 13:23 |
| tobiash | it's like inception, a node leak fix introduces a mem leak... | 13:24 |
| fungi | tobiash: okay, so you've had 714852 applied in production, the leak may not be on master yet? | 13:33 |
| zbr | mordred: sure, let me look again at it. | 13:33 |
| *** bhavikdbavishi has quit IRC | 13:35 | |
| mordred | tobiash: I agree - that patch does seem likely - but also, 1gb/2h seems like a large leak | 13:35 |
| tobiash | fungi: correct, it's not merged yet | 13:36 |
| fungi | thanks for confirming | 13:36 |
| *** Goneri has joined #zuul | 13:40 | |
| *** harrymichal has joined #zuul | 13:41 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: install-docker: allow removal of conflicting packages https://review.opendev.org/702304 | 13:45 |
| *** bhavikdbavishi has joined #zuul | 13:50 | |
| *** lseki has joined #zuul | 13:57 | |
| *** sgw has joined #zuul | 13:57 | |
| *** dmellado has quit IRC | 14:04 | |
| *** bhavikdbavishi has quit IRC | 14:05 | |
| *** dmellado has joined #zuul | 14:05 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 14:11 |
| *** Goneri has quit IRC | 14:11 | |
| *** ysandeep|rover is now known as ysandeep|away | 14:12 | |
| *** Goneri has joined #zuul | 14:14 | |
| AJaeger | mnaser: looks great! | 14:21 |
| AJaeger | team, do we want to merge https://review.opendev.org/#/c/715727/ again? That's the pep8 inline comments feature. | 14:22 |
| tobiash | mordred: hrm, I wonder if the node-request references the buildset or layout, that would explain such a large leak | 14:30 |
| mordred | tobiash: oh yeah | 14:31 |
| tobiash | mordred: my theory is currently imploding :/ | 14:35 |
| mordred | oh no | 14:35 |
| tobiash | according to https://opendev.org/zuul/zuul/src/branch/master/zuul/nodepool.py#L74 the leaked node requests would be reported in the stats | 14:35 |
| tobiash | which is not the case | 14:35 |
| mordred | tobiash: I agree. if your theory was solid, you should see that rising | 14:36 |
| *** avass has quit IRC | 14:36 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow configure-mirrors to enable extra repos https://review.opendev.org/693887 | 14:37 |
| tobiash | mordred: it crashed on friday with a queue of 42 | 14:38 |
| mordred | ugh :( | 14:38 |
| mordred | zbr: there's actually a spec about improvements to the mirror construct - let me see if I can find it | 14:40 |
| fungi | mordred: it's basically in the zuul docw | 14:40 |
| fungi | docs | 14:40 |
| *** rishabhhpe has joined #zuul | 14:40 | |
| corvus | mordred, fungi, zbr: https://zuul-ci.org/docs/zuul-jobs/mirror.html | 14:40 |
| corvus | mordred, fungi, zbr: a change working on implementing it for os mirrors: https://review.opendev.org/677578 | 14:41 |
| fungi | er, yeah, zuul-docs. just found it | 14:41 |
| mordred | corvus: thanks! that's what I was looking for :) | 14:41 |
| fungi | gah, zuul-jobs docs i mean | 14:41 |
| zbr | fungi: mordred: the linked change is 6mo old, and very red. | 14:50 |
| mordred | yup. it still needs work | 14:50 |
| mordred | but the overall design has been agreed to across the zuul folks - so it's just a matter of pushing the implementation forward. doing so would greatly benefit everybody | 14:51 |
| zbr | i would say a minor fix today is much better that a big refactoring that never ships :D | 14:51 |
| mordred | well, refactorings never ship if we only ever work on minor fixes today ;) | 14:52 |
| tobiash | mordred: at least the requests do have a ref to the buildset | 14:52 |
| tobiash | so maybe a few leaked requests are already enough to leak a lot | 14:52 |
| mordred | tobiash: yeah | 14:52 |
| *** bhavikdbavishi has joined #zuul | 14:55 | |
| tobiash | at least yappy tells me that there are around twice as many tenant objects in memory than there are configured tenants | 15:00 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: install-docker: allow removal of conflicting packages https://review.opendev.org/702304 | 15:00 |
| *** bhavikdbavishi has quit IRC | 15:01 | |
| *** yoctozepto has quit IRC | 15:07 | |
| *** yoctozepto has joined #zuul | 15:08 | |
| *** toabctl has quit IRC | 15:08 | |
| zbr | corvus: i do not think is ok to prevent changes today just because 6mo ago there was an attempt to rewrite a role, clearly if nobody touched in 6mo is very unlikely that it will be addressed in the next weeks. | 15:08 |
| zbr | i am not against the rewrite, but looking for more pragmatic approaches, like what can we deliver in resonable amount of time | 15:09 |
| *** Miouge has quit IRC | 15:09 | |
| *** arxcruz has quit IRC | 15:09 | |
| *** EmilienM has quit IRC | 15:10 | |
| *** arxcruz has joined #zuul | 15:10 | |
| *** Miouge has joined #zuul | 15:11 | |
| fungi | zbr: nobody has touched it in that timespan because what we have now is working reasonably well, but reviewing new changes to it is a pain and will only get worse as we continue to tack more things onto it. consider this as like what teams in openstack complain about when vendors just want to cram their new features into the software and not help with generalized improvements | 15:11 |
| *** EmilienM has joined #zuul | 15:11 | |
| zbr | which are these "generalized improvements"? asking because afaik, just ubuntu is tested on all roles, all the other platforms have partial coverage. | 15:13 |
| zbr | fungi: to be clear, that open particular change is not very important | 15:18 |
| zbr | i still have a long list of of generalized improvements, like https://review.opendev.org/#/c/690057/ -- is this ok? | 15:20 |
| fungi | generalized improvements meaning a logical framework to handle communicating mirror/proxy resources for zuul environments in a consistent manner, rather than what we have now which is basically just a forklift of what congealed in opendev over the years | 15:20 |
| zbr | that is really a hard goal | 15:22 |
| fungi | we've met harder goals in the past | 15:24 |
| fungi | most of the challenge is coming up with a schema/protocol for encoding mirror details. the implementation flows from that | 15:28 |
| *** dmellado has quit IRC | 15:32 | |
| *** rishabhhpe has quit IRC | 15:33 | |
| zbr | how about https://review.opendev.org/#/c/706262 ? | 15:37 |
| *** andreykurilin has quit IRC | 15:39 | |
| *** dmellado has joined #zuul | 15:39 | |
| fungi | zbr: how about it in what sense? that doesn't seem related to mirror configuration | 15:44 |
| *** tdasilva has quit IRC | 15:45 | |
| *** evgenyl has quit IRC | 15:45 | |
| *** donnyd has quit IRC | 15:45 | |
| *** Shrews has quit IRC | 15:45 | |
| *** stevthedev has quit IRC | 15:45 | |
| *** tributarian has quit IRC | 15:45 | |
| *** ttx has quit IRC | 15:45 | |
| *** masterpe has quit IRC | 15:45 | |
| *** tdasilva has joined #zuul | 15:45 | |
| *** evgenyl has joined #zuul | 15:45 | |
| *** donnyd has joined #zuul | 15:45 | |
| *** Shrews has joined #zuul | 15:45 | |
| *** stevthedev has joined #zuul | 15:45 | |
| *** tributarian has joined #zuul | 15:45 | |
| *** ttx has joined #zuul | 15:45 | |
| *** masterpe has joined #zuul | 15:45 | |
| *** andreykurilin has joined #zuul | 15:46 | |
| *** masterpe has quit IRC | 15:48 | |
| *** hashar has joined #zuul | 15:51 | |
| zbr | fungi: they are unrelated. amount mirrors, I got the message. if suppose that if I can find some time I may rework the refactoring patch in the future. | 15:57 |
| zbr | fungi: i tried to find some references regarding filenames put under sudoers.d/ directory but I was not able to find anything concrete. | 16:11 |
| zbr | my impression was that others used "usernames" not service names (likely the same), but i have no proof. | 16:12 |
| zbr | clearly documentation does not make any recomandation | 16:12 |
| fungi | yeah, i'm mostly curious, do you know why it would be assumed that /etc/sudoers.d/zuul would only contain authorization for a user account named "zuul" (or for that matter, that it would make sense for the file in that directory to be named the same as whatever username ansible uses)? | 16:12 |
| fungi | sudo doesn't actually care what the file is named | 16:13 |
| fungi | it's an arbitrary filename | 16:13 |
| fungi | makes me wonder if what people really want is for that filename to be configurable via a variable | 16:14 |
| fungi | rather than tying the filename unnaturally to the username | 16:14 |
| zbr | i am inclined to believe it makes a little bit more sense to use usernames, mainly because you can run multiple copies of the same service | 16:14 |
| clarkb | I would use a filename that ties it back to the source | 16:15 |
| zbr | in the end the file is expected to set priviledges for the user not for the service | 16:15 |
| clarkb | zuul-jobs-manage-sudo or whatever | 16:15 |
| fungi | 690057 would be encoding the expectation that whatever username ansible is authenticating with, there will be a file under /etc/sudoers.d with the same name | 16:15 |
| clarkb | it doesn't need to be configurable as long as we avoid collisions (and tieing it to the source is a good way of doing that) | 16:15 |
| fungi | if ansible logs in with a username of "admin" or "ubuntu" or whatever, you can put a line for that in /etc/sudoers.d/zuul (or multiple entries if you like, or authorize %groups instead of usernames in it, even) | 16:16 |
| fungi | clarkb: yeah, i'm trying to understand the actual concern. if the concern is that people want the filename to be something other than zuul then that could be made configurable i suppose | 16:18 |
| clarkb | fungi: I'm also not sure such a concern is valid? | 16:18 |
| clarkb | it says zuul in the name because it originated from zuul | 16:18 |
| clarkb | that is a good thing :) | 16:18 |
| fungi | right, it's the only explanation i can come up with for people wanting that file to not be named "zuul" | 16:18 |
| zbr | TBH, i would not have cared about that change if I was not suggested to rebased another on on it. I guess that some zuul deployments do create a sudoers file that is using a custom username filename instead of the service name. | 16:23 |
| corvus | tobiash: yes, i think the noderequest does have a path to the layout: NodeRequest -> BuildSet -> Item -> Layout [also Item -> Pipeline -> Tenant] | 16:27 |
| clarkb | zbr: fungi ah is the use case to clean up an existing file? | 16:28 |
| clarkb | I could see making the cleanup side configurable if that is the use case | 16:28 |
| tobiash | corvus: yeah, found it and it explains the severity of the leak | 16:28 |
| tobiash | just did a restart 90 minutes ago and the revert seems to flatten the curve | 16:29 |
| corvus | tobiash: cool, i left 2 comments on the change. the question about the alternate approach is mostly just exploration; i'm not suggesting that's a good idea, just sounds worth looking into | 16:30 |
| zbr | clarkb: based on who raised the CR, i expect Volvo uses a custom setup. I wonder if tristanC can give some insights regarding rdo/sf usage. | 16:30 |
| zbr | if we have more than one deployment that assumed that that correct location is sudoers.d/$username instead of sudoers.d/$service maybe it make sense to do it. | 16:31 |
| tobiash | corvus: thanks, responded | 16:32 |
| tobiash | I'll think I'll try that as your suggestion seems to be more memleak proof for the future | 16:32 |
| tobiash | especially given that if we have a memleak there it's quite severe | 16:33 |
| tobiash | your suggestion would also not require test code changes as well | 16:35 |
| *** evrardjp has quit IRC | 16:36 | |
| *** evrardjp has joined #zuul | 16:36 | |
| *** jcapitao has quit IRC | 16:40 | |
| corvus | mordred, tobiash: i'm worried i'm getting confused on https://review.opendev.org/714508 can you double check my comment? :) | 16:44 |
| *** masterpe has joined #zuul | 16:47 | |
| mordred | corvus: you're right - we're missing a not. will update in just a minute | 16:47 |
| tobiash | corvus: you're totally right | 16:48 |
| tobiash | that's why we require two reviews :) | 16:48 |
| *** avass has joined #zuul | 16:50 | |
| *** zxiiro has joined #zuul | 17:09 | |
| *** jpena is now known as jpena|off | 17:22 | |
| *** bhavikdbavishi has joined #zuul | 17:42 | |
| *** bhavikdbavishi1 has joined #zuul | 17:44 | |
| *** bhavikdbavishi has quit IRC | 17:46 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 17:46 | |
| mnaser | i know this might not be the best place, but know there's quite a few gertty users in here | 18:03 |
| mnaser | is there a shortcut of a "go to change by id" ? | 18:03 |
| fungi | ctrl-o is the default keybinding for a gerrit search | 18:04 |
| fungi | if you just stick the change number in there, you'll be taken to it | 18:05 |
| mnaser | let me try that | 18:05 |
| fungi | there is also a cli command for that | 18:05 |
| fungi | `gertty --open ...` from a shell will instruct the already running gertty process to open that | 18:06 |
| fungi | so you could for example have your browser call that on specific urls | 18:06 |
| mnaser | hmm, ctrl-o seems unresponsive, but i wonder if that's just because tmux is capturing that (or not sending it) | 18:06 |
| fungi | (though i think to use --open you may have to phrase your change identifier in the form of a url) | 18:07 |
| fungi | the ? key should give you interactive, context-sensitive help | 18:08 |
| fungi | under Global Keys my help screen says "CTRL-o Search for changes" | 18:08 |
| fungi | maybe you've remapped it | 18:08 |
| mnaser | hmm, ctrl+r works (i see the sync number going up), but ctrl+o does not do anything (and i did not remap it, and the contextual help shows it is still ctrl+o) | 18:08 |
| fungi | or yeah, maybe your terminal or something is eating ctrl-o, in which case you could rebind it in your .gertty.yaml | 18:08 |
| mnaser | even ctrl+q works too, so it must be something with ctrl+o | 18:09 |
| mnaser | https://apple.stackexchange.com/a/3255 | 18:10 |
| mnaser | "⌃+O is, for some reason (probably having to do with it being used for flow control on some kinds of serial connections) set to be discarded by the terminal driver (i.e. not Terminal.app, but the part of the OS between it and the shell). You can get rid of this with the command stty discard undef. To make this change permanent, add this command to your .bash_profile and .bashrc files." | 18:10 |
| fungi | neat! | 18:11 |
| fungi | ftr, xorg->xterm->bash->mosh->tmux->gertty is handling ctrl-o just fine for me | 18:12 |
| fungi | i guess that's really xorg->ratpoison->xterm->bash->mosh->tmux->bash->gertty | 18:12 |
| openstackgerrit | Merged zuul/zuul-jobs master: test-upload-logs-swift: revert download script https://review.opendev.org/715755 | 18:14 |
| mnaser | fungi: that did the trick | 18:18 |
| mnaser | fungi: next on my list is figuring out how to make clicking those in weechat launch in gertty, but that's for another time | 18:18 |
| avass | regarding: https://review.opendev.org/#/c/706262/ | 18:18 |
| avass | I couldn't remember why we wanted this and I've been looking around and I can't actually find a usecase for it on our end | 18:19 |
| fungi | mnaser: yeah, and that's what you're going to want to use `gertty --open ...` for | 18:21 |
| *** rlandy is now known as rlandy|brb | 18:29 | |
| avass | But I'll have to make sure with some other teams so I'm not just missing something | 18:29 |
| fungi | avass: cool, i'm not adamantly opposed to the idea, just would want to be sure we understand the use case because there might be better options to address it | 18:32 |
| fungi | having a role magically change its behavior depending on the username seems likely to lead to more confusion | 18:33 |
| avass | fungi: it could be that we had some strange setup somewhere that has been removed since we're always changing so many things | 18:41 |
| avass | fungi: to be honest, it would make more sense to have that file configurable, preferably in nodepool somehow since in aws we're using the 90-cloud-init-users file | 18:42 |
| *** y2kenny has joined #zuul | 18:43 | |
| *** bhavikdbavishi has quit IRC | 18:48 | |
| corvus | mnaser: https://opendev.org/ttygroup/gertty#user-content-macos and https://opendev.org/ttygroup/gertty#user-content-terminal-integration may be useful | 18:52 |
| mnaser | corvus: oh maybe i can update the docs to provide an alternative | 18:53 |
| *** rlandy|brb is now known as rlandy | 18:54 | |
| corvus | mnaser: thanks! | 18:55 |
| y2kenny | Since namespace and pod created by the k8s driver is ephemeral, is there a recommended way to share persistent resources (something like a build cache or repo cache?) | 18:58 |
| *** y2kenny28 has joined #zuul | 19:02 | |
| *** y2kenny28 has left #zuul | 19:02 | |
| *** y2kenny has quit IRC | 19:03 | |
| *** y2kenny has joined #zuul | 19:05 | |
| fungi | y2kenny: what we do is have jobs which prebuild artifacts like that and then configure them as variables which our jobs can refer to | 19:08 |
| y2kenny | do you mean you pass in the prebuild artifacts via the executor? | 19:09 |
| fungi | host the prebuilt artifacts somewhere reachable from the job nodes | 19:09 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Strip by default in tools/encrypt_secret https://review.opendev.org/714508 | 19:10 |
| mordred | corvus, tobiash: I think that got it right now? ^^ | 19:10 |
| fungi | y2kenny: an example in opendev's deployment is python wheels, because we deal with a lot of python-based projects. normally when you install a source distribution package for a python module the package management tooling performs just-in-time compilation of that package into a binary "wheel" | 19:10 |
| fungi | y2kenny: so we have jobs which periodically build those for all our test platforms and then add them to a central repository reachable from the job nodes | 19:11 |
| y2kenny | fungi: um... I suppose this is not using k8s or openshift objects? I am wondering because there is really no way to share stuff like persistence volume since things are in different namespaces | 19:12 |
| fungi | y2kenny: right, we use publication channels | 19:12 |
| fungi | have the executor fetch the artifacts from the build nodes and write them into a central repository (an afs volume in this case) | 19:13 |
| fungi | and then we front that afs volume with a webserver and the build nodes can fetch prebuilt things from it over https | 19:14 |
| y2kenny | fungi: ok... I will need to rethink the architecture a bit. | 19:15 |
| fungi | in the general case of stuff like ccache, i don't know how you'd go about having untrusted jobs contribute to a build cache, taking problems like write credentials and cache poisoning into account | 19:15 |
| fungi | it would definitely be an interesting space to explore though | 19:15 |
| mordred | that might be less of a concern in a more trusted environment | 19:16 |
| y2kenny | yea... I haven't got to that level of security at this point. | 19:16 |
| mordred | but ccache operates as a network service - so if environmentally it was ok to use a shared one amongst the users, you coudl imagine plumbing the config env vars through | 19:17 |
| mordred | we dont' have good examples of that from opendev - becuase of the public nature :) | 19:17 |
| fungi | right, in which case you'd maybe just fetch the cache from each job node during log collection and have the executor merge them into some central cache which the job nodes know to pull from (or which you have an early role in pre sync a copy of onto the job node) | 19:17 |
| mordred | fungi: right - but I mean - ccache does that by itself without needing to sync | 19:17 |
| fungi | or even embed your ccache credentials on the job nodes if that's safe for you | 19:17 |
| mordred | assuming it's a thing that's ok to use based on mutual trust | 19:17 |
| mordred | yeah | 19:18 |
| fungi | right, maybe put a ccache server somewhere your job nodes have a route to in that case, and just configure your jobs to know the dns name or address of the ccache server you want them to all push into and pull from? | 19:19 |
| fungi | the solution will likely depend a lot on the exact nature of the data you want to cache and your risk model for builds | 19:20 |
| mordred | yeah | 19:20 |
| mordred | we pondered (briefly so far) a similar thing related to the bazel build farm service google runs when thinking about builds in gerrit's new zuul | 19:21 |
| mordred | I don't have a solution yet - because it's a bit complex in that case due to the public nature again | 19:21 |
| fungi | as mordred notes, most of the standard library in zuul was built with the idea that you may be running an entirely publicly accessible zuul deployment which takes untrusted changes from anyone on the internet and runs jobs against that, so is constrained by the requisite levels of paranoia | 19:21 |
| y2kenny | right. more thinking for me to do. I will probably go without cache for now and optimize for performance later. | 19:21 |
| mordred | yeah. it means you can be sure it's all paranoid for security ... but might be overkill in some more trusted envs | 19:22 |
| mordred | y2kenny: ++ | 19:22 |
| mordred | I'm a fan of correctness first, optimize second :) | 19:22 |
| y2kenny | it's good that you guys are thinking about the security stuff from the start. I don't think these type of things can be bolted on later. | 19:22 |
| y2kenny | but it does adds to the learning curve. I am just reading up on the job secret stuff right now. Need to convince myself storing encrypted secrets in git is ok. | 19:24 |
| y2kenny | (I am sure it is... but putting things in git just feel so... forever :)) | 19:25 |
| *** saneax has quit IRC | 19:27 | |
| fungi | credentials are only forever if you can't revoke/replace them! ;) | 19:37 |
| fungi | in my opinion, if you have credentials which are irreplaceable/irrevokeable, then you should probably not put them in git (or use them at all really) | 19:38 |
| fungi | i'm honestly more concerned about leaking plaintext secrets in job logs than someone working out how to decrypt an asymmetrically-encrypted secret in a git repository | 19:41 |
| y2kenny | fungi: make sense. (My credentials are definitely revocable) | 19:50 |
| *** mugsie has quit IRC | 19:51 | |
| *** harrymichal has quit IRC | 19:52 | |
| *** mugsie has joined #zuul | 19:54 | |
| fungi | but yeah, if there's anything i've learned from my many years focusing on information security, it's that you need to plan for when, not if, you lose control of your secrets, and design your policies and processes to limit the impact of those incidents, rather than having highly-guarded secrets and getting by on the hope that they're never leaked | 19:57 |
| *** dpawlik has quit IRC | 19:59 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:07 | |
| *** dpawlik has joined #zuul | 20:08 | |
| *** dpawlik has quit IRC | 20:13 | |
| y2kenny | does the addition of secrets make a pre-review job into a post-review job? | 20:42 |
| y2kenny | (got an "unable to freeze job graph" after I associated a secret to a job...) | 20:44 |
| fungi | y2kenny: no, basically you can't include a secret in a job which is being run in a pre-review pipeline, nor can you add a job with secrets to a pre-review pipeline | 20:45 |
| y2kenny | o... what is the reason behind that? | 20:47 |
| fungi | because an untrusted change could `echo $secret` in the console log, while code reviewers are expected to keep an eye out for such shenanigans | 20:47 |
| y2kenny | oh right... of course.... | 20:48 |
| fungi | but if you don't have that risk, you could just set all your pipelines as "post-review" | 20:48 |
| y2kenny | silly me :) | 20:49 |
| *** hashar has quit IRC | 20:49 | |
| y2kenny | I will do that for now only because I am still experimenting | 20:49 |
| fungi | as you pointed out, we do try to design for such situations | 20:49 |
| y2kenny | I am trying to figure out how to pass secrets into k8s for image creation | 20:50 |
| y2kenny | and I hope to do the trying in pre-review without submiting the experimental patches | 20:50 |
| fungi | right, ultimately you would probably do that in a pipeline triggered periodically off an already merged repository state, or in a pipeline where approved changes go to be merged | 20:51 |
| fungi | or possibly triggered from events of the branches getting updated when changes merge, or something along those lines | 20:51 |
| fungi | zuul's security model assumes the people approving changes for a repository are already trusted with access to secrets stored in that repository | 20:52 |
| y2kenny | understood. | 20:52 |
| corvus | y2kenny: here's a job that uses secrets to deploy an app (zuul itself) to k8s: https://gerrit.googlesource.com/zuul/ops/+/refs/heads/master/.zuul.yaml | 20:52 |
| y2kenny | corvus: great, thanks! | 20:53 |
| corvus | y2kenny: that's in a config-project repo, so the entire repo is trusted and never speculative; you'll note there's no pre-review check job (but there could and should be -- we have the ability to spin up a k8s and perform a test deployment; it's just new and ENOTIME) | 20:53 |
| *** tjgresha_ has quit IRC | 20:57 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Add ZooKeeper TLS support https://review.opendev.org/712733 | 20:59 |
| openstackgerrit | Merged zuul/zuul-jobs master: Improve job and node information banner https://review.opendev.org/677971 | 21:37 |
| *** y2kenny has left #zuul | 22:00 | |
| *** tosky has quit IRC | 22:20 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: zuul-restart: change service order to prevent tenant loading failure https://review.opendev.org/715424 | 22:28 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Set default secret mode to 0400 https://review.opendev.org/714501 | 22:29 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add integration test playbook https://review.opendev.org/714165 | 22:30 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add nodepool launcher service initial deployment https://review.opendev.org/715310 | 22:31 |
| tristanC | corvus: mnaser: it seems like i found a simple solution with 715424 to fix automatic zuul restart issues. | 22:42 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add nodepool external config https://review.opendev.org/715311 | 22:50 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Update dib dep to 2.35.0 https://review.opendev.org/716104 | 22:50 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Adapt the integration playbook to be usable locally https://review.opendev.org/714163 | 22:53 |
| *** saneax has joined #zuul | 22:53 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add nodepool labels to integration test https://review.opendev.org/715316 | 22:53 |
| *** rfolco has quit IRC | 22:57 | |
| *** rfolco has joined #zuul | 22:58 | |
| *** rfolco has quit IRC | 22:59 | |
| clarkb | there is a question about why we use merge if necessary and not rebase if necessary (or cherry pick) with zuul and gerrit on the mailing list | 23:08 |
| clarkb | I would respond but my memory is really fuzzy on this. I think it comes down to gerrit rewrites your commit messages and not just the commits themsevles when it does that and it creates problems? | 23:08 |
| mordred | clarkb: it's dev confustion | 23:10 |
| fungi | heh, i was about to say the same thing ;) | 23:10 |
| mordred | want me to answer? | 23:10 |
| clarkb | mordred: ya I think someone that has a clearer memory should respond | 23:11 |
| mordred | which ML? | 23:11 |
| fungi | the commit id (git sha) in your local copy may never appear in gerrit's repo because of it rewriting the commit to rebase or cherry-pick | 23:11 |
| fungi | mordred: zuul-discuss | 23:12 |
| mordred | ahh | 23:12 |
| fungi | though a more modern reason to avoid those is that it's not compatible with signed commits | 23:13 |
| fungi | gerrit can't recreate your signing key/signature | 23:13 |
| fungi | so you may push a change with a signed commit, but gerrit rebases or cherry-picks that and recreates the commit with no signature | 23:14 |
| fungi | because it has no other option | 23:14 |
| fungi | also there's the point that if you really don't want to see a bunch of messy merge commits in your git log, it's just a --no-merges away | 23:15 |
| *** sanjayu_ has joined #zuul | 23:23 | |
| *** saneax has quit IRC | 23:26 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add nodepool kubernetes pod label to integration test https://review.opendev.org/715316 | 23:26 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add nodepool kubernetes pod label to integration test https://review.opendev.org/715316 | 23:28 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!