| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 00:02 |
|---|---|---|
| *** ajitha has joined #zuul | 00:02 | |
| fungi | corvus: looks like flake8 complained about insufficient spacing between some class definitions | 00:06 |
| fungi | unit test should hopefully report shortly | 00:06 |
| *** tosky has quit IRC | 00:09 | |
| *** sshnaidm is now known as sshnaidm|off | 00:12 | |
| fungi | yeah, tests are good with it | 00:30 |
| *** hamalq has quit IRC | 00:56 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 00:58 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Fix trigger event forwarding bug https://review.opendev.org/c/zuul/zuul/+/782335 | 01:09 |
| fungi | corvus: ^ minor whitespace appeasement | 01:10 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: WIP: Try to repro recheck failure https://review.opendev.org/c/zuul/zuul/+/782336 | 01:11 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 01:23 |
| corvus | fungi: oh thanks! | 01:35 |
| fungi | i just wanted to be able to register my +2 before i lose consciousness, in case apac/emea folks are keen to approve it | 01:38 |
| *** evrardjp has quit IRC | 03:33 | |
| *** evrardjp has joined #zuul | 03:33 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 03:37 |
| *** vishalmanchanda has joined #zuul | 04:10 | |
| *** ykarel has joined #zuul | 04:28 | |
| *** EmilienM has quit IRC | 05:13 | |
| *** dmsimard has quit IRC | 05:14 | |
| *** dmsimard has joined #zuul | 05:14 | |
| *** jfoufas1 has joined #zuul | 05:22 | |
| *** ykarel has quit IRC | 05:23 | |
| *** ykarel has joined #zuul | 05:25 | |
| *** ajitha has quit IRC | 07:31 | |
| *** jcapitao has joined #zuul | 07:46 | |
| *** rpittau|afk is now known as rpittau | 08:13 | |
| *** sshnaidm|off has quit IRC | 08:34 | |
| *** hashar has joined #zuul | 08:41 | |
| *** jpenag is now known as jpena | 08:57 | |
| *** ykarel is now known as ykarel|lunch | 08:57 | |
| *** nils has joined #zuul | 09:02 | |
| *** tosky has joined #zuul | 09:04 | |
| *** saneax has quit IRC | 09:04 | |
| avass | corvus: I can do another test today and check if paste.openstack truncated the logs. but no I'm not running anything custom (except for the digital ocean patch in nodepool) | 09:08 |
| *** holser has joined #zuul | 09:14 | |
| *** harrymichal has joined #zuul | 09:21 | |
| swest | avass: I can reproduce the issue | 09:27 |
| swest | will follow up with a fix and more details later | 09:28 |
| *** vishalmanchanda has quit IRC | 09:28 | |
| avass | swest: nice :) | 09:43 |
| *** ykarel|lunch is now known as ykarel | 10:01 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Fix bug w/ None event in Gitub event forwarder https://review.opendev.org/c/zuul/zuul/+/782395 | 10:10 |
| swest | avass: corvus: fungi: tobiash: ^ fix for the missing trigger for github events | 10:10 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Fix bug w/ None event in Github event forwarder https://review.opendev.org/c/zuul/zuul/+/782395 | 10:12 |
| avass | swest: thanks! I'll do a check with that when I'm off work | 10:12 |
| *** sshnaidm has joined #zuul | 10:13 | |
| *** sshnaidm is now known as sshnaidm|off | 10:13 | |
| *** tosky has quit IRC | 10:13 | |
| *** tosky has joined #zuul | 10:14 | |
| *** harrymichal has quit IRC | 10:57 | |
| *** harrymichal has joined #zuul | 10:58 | |
| *** jcapitao is now known as jcapitao_lunch | 11:00 | |
| *** yoctozepto has quit IRC | 11:02 | |
| *** yoctozepto has joined #zuul | 11:03 | |
| *** vishalmanchanda has joined #zuul | 11:11 | |
| *** rlandy has joined #zuul | 11:35 | |
| *** EmilienM has joined #zuul | 12:05 | |
| *** hashar is now known as hasharLunch | 12:09 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** ykarel has quit IRC | 12:38 | |
| *** ykarel has joined #zuul | 12:38 | |
| *** ykarel_ has joined #zuul | 12:44 | |
| *** ykarel has quit IRC | 12:47 | |
| *** jcapitao_lunch is now known as jcapitao | 12:50 | |
| *** mgoddard has quit IRC | 12:53 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Ensure Gentoo Repo directory https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 12:58 |
| *** piotrowskim has joined #zuul | 13:02 | |
| *** hasharLunch is now known as hashar | 13:17 | |
| *** ykarel_ is now known as ykarel | 13:17 | |
| *** ykarel_ has joined #zuul | 13:22 | |
| *** ykarel__ has joined #zuul | 13:23 | |
| *** ykarel has quit IRC | 13:24 | |
| *** ykarel_ has quit IRC | 13:26 | |
| *** ykarel has joined #zuul | 13:26 | |
| *** ykarel__ has quit IRC | 13:28 | |
| *** ykarel has quit IRC | 13:32 | |
| *** ykarel has joined #zuul | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| corvus | swest, avass: lgtm +3 thx | 13:47 |
| *** dmsimard has quit IRC | 13:53 | |
| *** dmsimard has joined #zuul | 13:54 | |
| *** mgoddard has joined #zuul | 13:57 | |
| *** GomathiselviS has joined #zuul | 14:16 | |
| GomathiselviS | Hi Requesting a merge https://review.opendev.org/c/zuul/zuul-jobs/+/773474, if the base-test has passed with this change. | 14:18 |
| *** jfoufas1 has quit IRC | 14:19 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: WIP: Set Gentoo profile in configure-mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/782339 | 14:30 |
| *** ajitha has joined #zuul | 14:37 | |
| avass | GomathiselviS: I'm not sure if anyone did a base-test loop? | 15:06 |
| avass | pabelanger: ^ ? | 15:06 |
| mordred | corvus: https://review.opendev.org/c/zuul/zuul/+/782335 has 2x+2 - left off +A in case you wanted anyone else to look at it | 15:17 |
| corvus | mordred: +w, i think we should get it in. tobiash might want to retro-review | 15:19 |
| mordred | swest: github patch failed in gate. looks like unrelated issue | 15:19 |
| corvus | i'll just toggle my +w | 15:20 |
| tobiash | corvus: I was just about to hit that button but then got distracted | 15:21 |
| *** frickler has quit IRC | 15:37 | |
| *** Eighth_Doctor has quit IRC | 15:37 | |
| *** mordred has quit IRC | 15:37 | |
| *** irclogbot_0 has quit IRC | 15:37 | |
| *** frickler has joined #zuul | 15:37 | |
| *** masterpe has quit IRC | 15:38 | |
| *** irclogbot_1 has joined #zuul | 15:40 | |
| *** y2kenny has joined #zuul | 15:44 | |
| *** irclogbot_1 has quit IRC | 15:49 | |
| *** irclogbot_3 has joined #zuul | 15:54 | |
| y2kenny | Hi, I understand for untrusted project, delegation of ansible task to run on the executor is limited to a whitelist of module. Is it possible to make that whitelist configurable/override-able? I am trying to run ipmitool from the executor to capture machine log via serial-over-lan. | 15:54 |
| fungi | y2kenny: it's implemented as a symlink farm in the ansible deployment zuul manages on executors, so you could alter that | 15:55 |
| y2kenny | fungi: do you have the specific coordinates? | 15:57 |
| y2kenny | (I am using the executor image from DockerHub) | 15:57 |
| fungi | yeah, i'm gettig the url but i'm also on a videoconference call and my workstation is struggling | 15:57 |
| y2kenny | oh thanks. no rush, take your time. | 15:57 |
| fungi | y2kenny: lookup modules example for the base ansible version is https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/lookup | 15:59 |
| fungi | y2kenny: we also deploy from zuul container images, but use zuul's ansible management to create that at /var/lib/zuul/ansible/2.9/zuul/ansible/lookup/ | 16:01 |
| *** hashar has quit IRC | 16:01 | |
| *** hashar has joined #zuul | 16:03 | |
| *** y2kenny has quit IRC | 16:06 | |
| *** y2kenny has joined #zuul | 16:07 | |
| tobiash | corvus: do you know if the official nodepool images are using jemalloc? | 16:08 |
| y2kenny | fungi: um... I don't think I understand. So if I want to use a shell task to run ipmitool, do I create ipmitool.py under lookup that looks like fileglob.py? | 16:08 |
| tobiash | we switched the launchers from alpine to a python image and now they show memory leak behavior (without changing nodepool itself) | 16:08 |
| pabelanger | avass: we did in zuul.a.c | 16:09 |
| pabelanger | but not opendev | 16:09 |
| pabelanger | we've fork the role into our trusted context for testing | 16:09 |
| pabelanger | https://github.com/ansible/project-config/tree/master/roles/add-build-sshkey-fork | 16:09 |
| *** mordred has joined #zuul | 16:10 | |
| fungi | y2kenny: oh, you're wanting to allow shell tasks? that does get a bit more hairy. the safer approach would probably be to create an ansible module for ipmitool and use that, yeah | 16:10 |
| *** ykarel has quit IRC | 16:15 | |
| y2kenny | fungi: so are the files under lookup full ansible module? (I am not that familiar with the Ansible ecosystem.) | 16:16 |
| y2kenny | fungi: do I need to restart the executor if I modify the lookup directory? | 16:18 |
| corvus | y2kenny: a module won't help here, that's restricted in the same way; instead run ipmitool from a trusted playbook, or run it from a host other than the executor | 16:18 |
| *** Eighth_Doctor has joined #zuul | 16:19 | |
| corvus | y2kenny: you can compose jobs with playbooks from multiple repos, so you can have a post-run playbook in a config project that runs ipmitool | 16:19 |
| corvus | there is no way to work around zuul's security posture here that we can guarantee will work in the future | 16:20 |
| y2kenny | corvus: post or pre? I want to start capturing the log, run the test job while the capturing is still going | 16:20 |
| y2kenny | I think I tried to do it in pre but the capturing stopped... not sure if I missed something | 16:21 |
| corvus | y2kenny: pre works too, i just guessed and guessed wrong :) but does it require the process to keep running the whole time? | 16:21 |
| corvus | because if so, then that won't work because zuul is really good about killing processes at the end of a playbook run. | 16:21 |
| pabelanger | GomathiselviS: https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L5 explains how to do a base-test, you'd need to fork the role into that repo first | 16:21 |
| *** masterpe has joined #zuul | 16:21 | |
| pabelanger | then update a job to use 'base-test' as the parent | 16:21 |
| y2kenny | corvus: it does... and I think that's what I ran into | 16:21 |
| pabelanger | which would then confirm the role doesn't break | 16:21 |
| tobiash | corvus: this is memory consumption of one launcher with previous alpine, then switched to python:3.9-slim: https://paste.pics/e5037e9d70416e9e7b989a070476a3ba | 16:22 |
| corvus | y2kenny: options are getting slim then; i think it's down to run it in a single trusted playbook, or get an extra observer node (or container or something) to run it :/ | 16:22 |
| y2kenny | corvus: I thought I read some doc about starting local docker registry for a job but I don't think that's between pre and run | 16:23 |
| corvus | y2kenny: it is, but the registry is on another node (not the executor) so that's like my "observer node" suggestion | 16:23 |
| openstackgerrit | Merged zuul/zuul master: Fix trigger event forwarding bug https://review.opendev.org/c/zuul/zuul/+/782335 | 16:23 |
| corvus | tobiash: i can't find any mention of jemalloc in zuul, nodepool, or opendev/system-config which holds the base images | 16:24 |
| y2kenny | corvus: yea... and I think last time I look into that route, I ran into the problem of scheduling from multiple nodepool label | 16:24 |
| tobiash | corvus: yeah, checked that as well | 16:25 |
| corvus | tobiash: i thought you had a change to switch to that though? | 16:25 |
| corvus | tobiash: https://review.opendev.org/c/opendev/system-config/+/725431 | 16:26 |
| tobiash | just found that as well | 16:26 |
| tobiash | so no idea why our launcher leaks on 3.9-slim | 16:27 |
| y2kenny | corvus: the job currently run on scarce baremetal node so ideally the observer would not be running on another baremetal. But afaik, we currently cannot schedule multiple node across multiple node/nodepool provider. | 16:30 |
| corvus | tobiash: so basically jemalloc leaks with some versions and non-jemalloc leaks with others? | 16:30 |
| tobiash | corvus: that would be worst case | 16:31 |
| tobiash | but that is my impression atm | 16:31 |
| tobiash | next try will be py39-slim with jemalloc and see if that works | 16:31 |
| tobiash | unfortunately we need to rollout every iteration to production to see an effect | 16:31 |
| tobiash | but at least so far nothing broke | 16:32 |
| *** y2kenny has quit IRC | 16:32 | |
| tobiash | but that could be even workload related, jemalloc has been reverted in python-builder due to zuul-scheduler leaking | 16:33 |
| corvus | y2kenny: for this, i think you have 2 good options (other than "get a cloud or a k8s") to get 'lightweight' observer nodes -- you can register the same real node multiple times with different users, or you can register the same node multiple times with a max-parallel-jobs setting, but make sure you update your base job pre-playbook not to run any repo setup on that node since multiple jobs will conflict with | 16:33 |
| corvus | each other. | 16:33 |
| tobiash | corvus: our zuul is on python:3.8-slim without jemalloc and works | 16:34 |
| tobiash | very weird | 16:34 |
| corvus | tobiash: let me know if you find a better way to test this :) does your nodepool load generator show it? | 16:35 |
| tobiash | good idea, need to try that | 16:35 |
| openstackgerrit | Merged zuul/zuul master: Fix bug w/ None event in Github event forwarder https://review.opendev.org/c/zuul/zuul/+/782395 | 16:35 |
| avass | the javascript tarball promote job just failed for that ^ | 16:38 |
| corvus | avass: i think it's in permafail state and maybe time for us to get rid of it? | 16:38 |
| avass | I'm not actually sure what it does, just noted it | 16:39 |
| avass | but yeah that looks like taht always fails | 16:40 |
| fungi | we were talking for a while about dropping that, though i think the docs still mention it as one way to deploy the web content? | 16:41 |
| *** y2kenny has joined #zuul | 16:44 | |
| avass | oh it fails because the dashboard isn't built in gate | 16:45 |
| openstackgerrit | Albin Vass proposed zuul/zuul master: Only promote zuul-build-dashboard when it's built https://review.opendev.org/c/zuul/zuul/+/782559 | 16:51 |
| avass | how about that ^? | 16:52 |
| *** cloudnull has quit IRC | 16:54 | |
| *** cloudnull has joined #zuul | 16:54 | |
| avass | swest, corvus, fungi, tobiash: looks like the github issue is fixed, thanks! | 16:57 |
| tobiash | cool :) | 16:57 |
| corvus | cool, i'll plan on restarting opendev in a little bit | 16:59 |
| *** GomathiselviS has quit IRC | 17:00 | |
| y2kenny | corvus: I do have a k8s, but it is on a separate provider/driver. The situation is basically trying to schedule a node from OpenStack Driver and a node from Azure driver for the same job. The registering the resource multiple time is an interesting strategy I haven't thought about before. I don't think it will work though since my use case | 17:01 |
| y2kenny | for these baremetal nodes from pxe boot and up. The purpose of the ipmitool is to have a different way to monitor the baremetal node via BMC even if the system crash catastrophically | 17:01 |
| *** iurygregory has quit IRC | 17:02 | |
| y2kenny | corvus: sounds like I might have to have a persistent service to do log capturing outside of Zuul workflow instead of launching things within Zuul | 17:02 |
| avass | I suppose it wouldn't make sense for nodepool to do that? | 17:06 |
| *** jangutter_ has joined #zuul | 17:08 | |
| corvus | yeah, we've talked about having nodepool schedule a single request cross-provider. it's something we'd like to support, but needs dev work. | 17:08 |
| corvus | y2kenny: can you run the whole thing as a trusted playbook? or do you need speculative execution of job changes? | 17:09 |
| *** jangutter has quit IRC | 17:12 | |
| corvus | y2kenny: if you can do it as a trusted playbook, then you can run that on the executor as you discussed earlier; you would just lose the speculative execution of changes to the job itself. that may be a reasonable compromise in this situation? | 17:17 |
| y2kenny | corvus: I think I need speculative execution because the job is testing hw device driver. I already put some portion of the job in trusted playbook (like booting of the baremetal node with the test driver and I am able to capture the boot log there.) | 17:24 |
| y2kenny | actually wait... may be I am mixing up some concept | 17:25 |
| y2kenny | corvus: can I define the whole playbook as trusted but still have a speculative trigger? I think I can right? | 17:26 |
| *** rpittau is now known as rpittau|afk | 17:26 | |
| avass | y2kenny: the test can still be speculative but not the zuul job config | 17:26 |
| y2kenny | ok... may be there's something I can play with | 17:28 |
| *** iurygregory has joined #zuul | 17:29 | |
| corvus | y2kenny, avass: yep | 17:42 |
| *** jcapitao has quit IRC | 17:46 | |
| *** hashar is now known as hasharAway | 17:48 | |
| *** jpena is now known as jpena|off | 18:05 | |
| *** vishalmanchanda has quit IRC | 18:17 | |
| *** hamalq has joined #zuul | 18:18 | |
| *** GomathiselviS has joined #zuul | 18:23 | |
| *** y2kenny has quit IRC | 18:32 | |
| *** ajitha has quit IRC | 19:16 | |
| avass | there's no way to OR pipeline requirements right? | 19:38 |
| avass | I wanna be able to self approve in github but for some reason the bots approval shows up as 'read' permission. I figured a workaround could be to allow 'write' OR username=<bot-name> | 19:40 |
| *** GomathiselviS has quit IRC | 19:43 | |
| *** hashar_ has joined #zuul | 20:00 | |
| *** hasharAway has quit IRC | 20:01 | |
| *** nils has quit IRC | 20:03 | |
| *** hashar__ has joined #zuul | 20:03 | |
| *** hashar_ has quit IRC | 20:06 | |
| *** sean-k-mooney has quit IRC | 20:24 | |
| *** hashar__ has quit IRC | 20:35 | |
| *** jangutter has joined #zuul | 21:22 | |
| *** jangutter_ has quit IRC | 21:26 | |
| *** harrymichal has quit IRC | 23:15 | |
| *** rlandy has quit IRC | 23:53 | |
| *** hamalq has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!