| *** hamalq has joined #zuul | 00:56 | |
| *** hamalq has quit IRC | 01:01 | |
| *** sshnaidm|afk has quit IRC | 02:06 | |
| *** evrardjp has quit IRC | 02:33 | |
| *** evrardjp has joined #zuul | 02:33 | |
| *** hamalq has joined #zuul | 02:57 | |
| *** hamalq has quit IRC | 03:01 | |
| *** ykarel__ has joined #zuul | 03:40 | |
| *** ykarel__ is now known as ykarel | 04:16 | |
| *** paladox has quit IRC | 04:20 | |
| *** ricolin has quit IRC | 04:35 | |
| *** ricolin has joined #zuul | 04:48 | |
| *** hamalq has joined #zuul | 04:58 | |
| *** hamalq has quit IRC | 05:03 | |
| *** jangutter has quit IRC | 06:04 | |
| *** jangutter has joined #zuul | 06:04 | |
| *** jcapitao has joined #zuul | 06:17 | |
| *** hamalq has joined #zuul | 06:59 | |
| *** hamalq has quit IRC | 07:03 | |
| *** hashar has joined #zuul | 07:03 | |
| *** rpittau|afk is now known as rpittau | 07:15 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Fix race condition related to out-of-band acks https://review.opendev.org/c/zuul/zuul/+/783607 | 07:16 |
|---|---|---|
| *** mhu has joined #zuul | 07:28 | |
| *** tosky has joined #zuul | 08:28 | |
| *** saneax has joined #zuul | 08:33 | |
| *** ykarel is now known as ykarel|lunch | 08:53 | |
| *** hamalq has joined #zuul | 09:00 | |
| *** hamalq has quit IRC | 09:04 | |
| *** ajitha has joined #zuul | 09:05 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Move project secrets key loading to key storage https://review.opendev.org/c/zuul/zuul/+/758939 | 09:16 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Store secrets keys and SSH keys in Zookeeper https://review.opendev.org/c/zuul/zuul/+/758940 | 09:16 |
| *** nils has joined #zuul | 09:17 | |
| *** holser has quit IRC | 09:25 | |
| *** ykarel|lunch is now known as ykarel | 09:27 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Store secrets keys and SSH keys in Zookeeper https://review.opendev.org/c/zuul/zuul/+/758940 | 09:27 |
| *** holser has joined #zuul | 09:27 | |
| *** hashar is now known as hasharLunch | 09:36 | |
| *** sshnaidm|afk has joined #zuul | 10:18 | |
| *** jangutter has quit IRC | 10:24 | |
| *** jangutter has joined #zuul | 10:25 | |
| *** jangutter_ has joined #zuul | 10:47 | |
| *** jangutter_ has joined #zuul | 10:47 | |
| *** jangutter has quit IRC | 10:50 | |
| *** hamalq has joined #zuul | 11:00 | |
| *** hamalq has quit IRC | 11:05 | |
| *** jcapitao is now known as jcapitao_lunch | 11:07 | |
| *** shanemcd has quit IRC | 11:19 | |
| *** shanemcd has joined #zuul | 11:19 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 11:45 | |
| *** sduthil has joined #zuul | 12:19 | |
| *** jcapitao_lunch is now known as jcapitao | 12:25 | |
| *** paladox has joined #zuul | 12:32 | |
| *** cloudnull has joined #zuul | 12:41 | |
| *** hamalq has joined #zuul | 13:01 | |
| *** hamalq has quit IRC | 13:06 | |
| *** hasharLunch is now known as hashar | 13:17 | |
| *** GomathiselviS has joined #zuul | 13:36 | |
| *** ykarel_ has joined #zuul | 13:39 | |
| *** ykarel has quit IRC | 13:42 | |
| GomathiselviS | corvus fungi : looking for merge today - https://review.opendev.org/c/zuul/zuul-jobs/+/773474 | 14:01 |
| *** harrymichal has joined #zuul | 14:13 | |
| corvus | tristanC: fyi https://zuul.opendev.org/api/tenant/zuul/pipeline/check/project/zuul/zuul/branch/master/freeze-job/zuul-build-image | 14:21 |
| corvus | jhesketh: ^ | 14:22 |
| tristanC | corvus: nice thanks a lot, i'll give it a try with the zuul-runner cli! | 14:24 |
| avass | corvus: oh nice | 14:24 |
| *** jangutter has joined #zuul | 14:39 | |
| *** jangutter_ has quit IRC | 14:40 | |
| *** jangutter_ has joined #zuul | 14:40 | |
| *** jangutter has quit IRC | 14:44 | |
| corvus | i'm looking at https://grafana.opendev.org/d/5Imot6EMk/zuul-status?orgId=1&from=now-7d&to=now and trying to figure out if the ~100 node requests with available capacity represents some kind of new lag related to zk, or if this is what a monday ramp-up looks like | 14:44 |
| corvus | last week if we had > 100 node requests for more than an hour we were at over 725 nodes in use | 14:47 |
| corvus | now we're over 100 requests for several hours with 625 in use | 14:47 |
| clarkb | corvus: cross check against errors booting instances to make sure there isn't a new launch failure? | 14:49 |
| avass | the event processing time doesn't seem to match the node request ramp up | 14:49 |
| corvus | clarkb: yeah, simplest explanation is probably a cloud thing | 14:49 |
| corvus | avass: agree; i'm not seeing any zk metrics correlating | 14:49 |
| avass | I suppose the spike around 06:00 does match, I suppose that's a periodic pipeline triggering? | 14:55 |
| *** ykarel_ is now known as ykarel | 14:55 | |
| avass | oh it is :) | 14:57 |
| corvus | okay we need to install 'less' on our nodepool images :) | 14:58 |
| avass | number of znodes/ephemeral nodes/watches and data size does seem to increase over time | 14:59 |
| corvus | that correlates well with nodes in use | 14:59 |
| tobiash | do you have throughput numbers like jobs/h? | 15:02 |
| *** hamalq has joined #zuul | 15:02 | |
| tobiash | we found those useful to see if there is unusual behavior (aka at quota throughput is comparable) | 15:03 |
| tobiash | we often spotted issues in the past when we saw unusual high or low throughput than usual when the system is under load | 15:04 |
| corvus | there's launched-per-hour; i don't know if we have completed-per-hour handy, but i'm sure we could get it | 15:06 |
| *** hamalq has quit IRC | 15:07 | |
| tobiash | I guess launched-per-hour is similar enough to completed-per-hour | 15:07 |
| corvus | they should at least correlate | 15:07 |
| corvus | i see a lot of arm64 requests outstanding | 15:09 |
| corvus | 121 | 15:09 |
| avass | yeah nothing seem to be starting in check-arm64 | 15:10 |
| corvus | our arm64 cloud situation is not as robust; i would not be surprised if there's an operational issue there | 15:10 |
| clarkb | 10 days ago it had trouble with finding hypervisors to place the VMs on | 15:10 |
| corvus | yep, the cloud's ssl cert has expired | 15:11 |
| corvus | so this is a false alarm for #zuul; lookes like the behavior change is an #opendev ops issue and not related to zk work | 15:11 |
| fungi | kevinz fixed the expired cert there for us last time, but i guess it has expired again now | 15:13 |
| fungi | probably three months ago ;) | 15:13 |
| tristanC | corvus: with trigger events being stored in zk, shouldn't the ZNodes values be higher than last week? | 15:30 |
| *** ykarel is now known as ykarel|away | 15:33 | |
| *** hashar has quit IRC | 15:39 | |
| *** ykarel|away has quit IRC | 15:42 | |
| fungi | GomathiselviS: corvus: i approved https://review.opendev.org/773474 just now, and will keep tabs on any impact in opendev's deployment | 15:52 |
| fungi | the copy in opendev's build-test was demonstrated properly defaulting to rsa keys and able to run the rest of jobs normally | 15:54 |
| corvus | tristanC: the event queue is in zk, and ideally the queue length should stay near zero, so we shouldn't see an increase in storage size or znode count (unless something goes wrong). we might see it spike up 100 or so on reconfigurations or similar situations where we stop processing event queues. | 15:54 |
| corvus | fungi, GomathiselviS, thanks :) | 15:55 |
| fungi | s/build-test/base-test/ | 15:55 |
| corvus | fungi: i knew you meant testing thingamajig | 15:55 |
| GomathiselviS | fungi corvus pabelanger : Thanks for the help ! | 15:56 |
| fungi | GomathiselviS: thanks for your patience with the complexity of testing lower-level roles like that one | 15:58 |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| *** saneax has quit IRC | 16:08 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Create a template for ssh-key and size https://review.opendev.org/c/zuul/zuul-jobs/+/773474 | 16:10 |
| *** hamalq has joined #zuul | 16:15 | |
| *** nils has quit IRC | 16:25 | |
| *** hamalq has quit IRC | 16:31 | |
| *** hamalq has joined #zuul | 16:31 | |
| *** hamalq has quit IRC | 16:33 | |
| *** hamalq has joined #zuul | 16:33 | |
| *** jcapitao has quit IRC | 16:43 | |
| openstackgerrit | Shturm Svetlana proposed zuul/zuul-jobs master: Fix undefined error for zuul_ssh_key_algorithm https://review.opendev.org/c/zuul/zuul-jobs/+/783717 | 16:54 |
| clarkb | fungi: ^ is that related to the change you helped land? | 16:57 |
| tristanC | clarkb: it seems like it yes, in https://review.opendev.org/c/zuul/zuul-jobs/+/773474 , the remove-build-ssh-key is now using a variable that was not added to the defaults | 17:00 |
| clarkb | tristanC: but it is added as a role var? | 17:01 |
| clarkb | are role vars only accessible within a role? | 17:02 |
| fungi | clarkb: GomathiselviS: corvus: it hasn't been breaking jobs in opendev as far as i can tell | 17:02 |
| fungi | i wonder how it's getting used in the breaking environment | 17:03 |
| tristanC | fungi: i guess this happen when using the remove-build-ssh-key role directly | 17:04 |
| fungi | and the vars defined in the role aren't getting used? | 17:05 |
| clarkb | oh if only that role is used and not the add-build-sshkey role? | 17:05 |
| fungi | wouldn't using the role also instantiate everything from roles/add-build-sshkey/vars/main.yaml ? | 17:05 |
| clarkb | that could be | 17:05 |
| fungi | aha, i see what you're saying | 17:05 |
| fungi | i missed it was for a different role | 17:07 |
| fungi | i suppose caching could influence that behavior | 17:08 |
| *** jangutter has joined #zuul | 17:09 | |
| *** jangutter_ has quit IRC | 17:12 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix undefined error for zuul_ssh_key_algorithm https://review.opendev.org/c/zuul/zuul-jobs/+/783717 | 17:18 |
| corvus | i'm glad we didn't merge that on friday afternoon | 17:29 |
| *** harrymichal has quit IRC | 17:30 | |
| *** harrymichal has joined #zuul | 17:30 | |
| fungi | yep! | 17:50 |
| *** harrymichal has left #zuul | 17:57 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix ZK-related race condition in github driver https://review.opendev.org/c/zuul/zuul/+/783726 | 17:58 |
| corvus | swest, tobiash: ^ that's an alternate fix for the race | 18:02 |
| *** cloudnull has quit IRC | 18:43 | |
| *** Goneri has joined #zuul | 18:58 | |
| *** Goneri has quit IRC | 18:59 | |
| *** cloudnull has joined #zuul | 19:21 | |
| *** harrymichal has joined #zuul | 19:30 | |
| *** ajitha has quit IRC | 19:48 | |
| *** jangutter_ has joined #zuul | 19:49 | |
| *** jangutter has quit IRC | 19:52 | |
| *** nhicher has quit IRC | 20:56 | |
| *** fbo has joined #zuul | 20:59 | |
| *** nhicher has joined #zuul | 21:00 | |
| *** harrymichal has quit IRC | 21:34 | |
| *** GomathiselviS has quit IRC | 22:05 | |
| *** cloudnull has quit IRC | 22:09 | |
| *** y2kenny has joined #zuul | 22:18 | |
| y2kenny | Hi, I am seeing a lot of "Waiting on logger" in my logs (job-output.txt) even though the command output shows up in job-output.json, what could be the cause of this? (I already have the 'start-zuul-console' role at the beginning of the play) | 22:23 |
| clarkb | y2kenny: its basically the time between the job starting to do stuff and start-zuul-console successfully starting the remote logger process | 22:23 |
| clarkb | y2kenny: there was a semi recent change made to reduce the amount of that output as it was fairly verbose previously (I think now its cut by 1/10th) | 22:24 |
| y2kenny | clarkb: ok... so sounds like the start-zuul-console role never suceeded on my baremetal node. What is the requirement for the start-zuul-console? Are there any good way to debug it? | 22:25 |
| y2kenny | I see this: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/start-zuul-console/tasks/main.yaml but not too sure where to go from there. | 22:27 |
| clarkb | off the top of my head I think it may only run on linux? thought some of the windows users can conform or deny that. It starts a python process that listens on port 19885 which zuul connects to to fetch the data over | 22:28 |
| clarkb | zuul/ansible/base/library/zuul_console.py is the code that runs to do this | 22:29 |
| y2kenny | ok... I am using linux on the baremetal node but I wonder if fedora's firewall rules blocks it by default | 22:29 |
| fungi | y2kenny: they might, we bake an exception for that into our node images: https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/nodepool-base/install.d/20-iptables#L60 | 22:43 |
| corvus | tobiash: a couple of questions on https://review.opendev.org/663413 | 22:44 |
| y2kenny | fungi: thanks! | 22:44 |
| *** cloudnull has joined #zuul | 22:52 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!