| *** tosky has quit IRC | 00:11 | |
| *** yoctozepto has quit IRC | 00:13 | |
| *** yoctozepto has joined #zuul | 00:13 | |
| *** hamalq has quit IRC | 00:26 | |
| *** ajitha has joined #zuul | 00:51 | |
| corvus | tobiash: based on #opendev, it looks like the executor's merger's multiprocessing pool may be leaking GitPython git.Repo objects -- clarkb found that a paused executor had 139 "git cat-file" processing running. GitPython starts those when creating a git.Repo object, and keeps them running as long as the object isn't deleted. | 01:06 |
|---|---|---|
| corvus | tobiash: unfortunately, all of the debugging tools we have for memory leaks can't cross the multiprocessing barrier, so i'm currently at a loss for how to debug that. | 01:07 |
| *** Eighth_Doctor has quit IRC | 01:45 | |
| *** jamesmcarthur has quit IRC | 02:12 | |
| *** jamesmcarthur has joined #zuul | 02:13 | |
| *** jamesmcarthur has quit IRC | 02:13 | |
| *** jamesmcarthur has joined #zuul | 02:13 | |
| *** Eighth_Doctor has joined #zuul | 02:14 | |
| *** zenkuro has quit IRC | 02:23 | |
| *** rlandy|bbl is now known as rlandy | 02:31 | |
| *** jamesmcarthur has quit IRC | 02:41 | |
| *** jamesmcarthur has joined #zuul | 02:41 | |
| *** irclogbot_2 has quit IRC | 02:52 | |
| *** irclogbot_3 has joined #zuul | 02:56 | |
| *** rlandy has quit IRC | 03:02 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Azure: switch to Azul https://review.opendev.org/c/zuul/nodepool/+/778814 | 03:09 |
| *** bhavikdbavishi has joined #zuul | 03:29 | |
| *** bhavikdbavishi1 has joined #zuul | 03:44 | |
| *** bhavikdbavishi has quit IRC | 03:45 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:45 | |
| *** saneax has joined #zuul | 03:57 | |
| *** bhavikdbavishi has quit IRC | 04:01 | |
| *** bhavikdbavishi has joined #zuul | 04:03 | |
| *** jamesmcarthur has quit IRC | 04:27 | |
| *** jamesmcarthur has joined #zuul | 04:27 | |
| *** bhavikdbavishi has quit IRC | 04:30 | |
| *** bhavikdbavishi has joined #zuul | 04:32 | |
| *** jamesmcarthur has quit IRC | 04:32 | |
| *** ykarel has joined #zuul | 04:38 | |
| *** jamesmcarthur has joined #zuul | 04:51 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #zuul | 05:33 | |
| *** jfoufas1 has joined #zuul | 05:35 | |
| *** jamesmcarthur has quit IRC | 05:36 | |
| *** ykarel has quit IRC | 05:50 | |
| *** ykarel has joined #zuul | 05:53 | |
| *** ykarel_ has joined #zuul | 06:08 | |
| *** ykarel has quit IRC | 06:10 | |
| *** ykarel_ is now known as ykarel | 06:10 | |
| openstackgerrit | Daniel Blixt proposed zuul/nodepool master: Add shell-type to more providers https://review.opendev.org/c/zuul/nodepool/+/778580 | 06:19 |
| *** vishalmanchanda has joined #zuul | 06:25 | |
| *** bhavikdbavishi has quit IRC | 06:40 | |
| *** piotrowskim has joined #zuul | 07:09 | |
| *** bhavikdbavishi has joined #zuul | 07:34 | |
| *** jamesmcarthur has joined #zuul | 07:36 | |
| *** bhavikdbavishi1 has joined #zuul | 07:37 | |
| *** bhavikdbavishi has quit IRC | 07:39 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 07:39 | |
| *** jamesmcarthur has quit IRC | 07:40 | |
| *** wuchunyang has joined #zuul | 07:47 | |
| openstackgerrit | Daniel Blixt proposed zuul/nodepool master: WIP: Add shell-type config https://review.opendev.org/c/zuul/nodepool/+/775371 | 07:50 |
| openstackgerrit | Daniel Blixt proposed zuul/nodepool master: Add shell-type to more providers https://review.opendev.org/c/zuul/nodepool/+/778580 | 07:51 |
| *** rpittau|afk is now known as rpittau | 08:21 | |
| *** bhavikdbavishi has quit IRC | 08:29 | |
| *** bhavikdbavishi has joined #zuul | 08:30 | |
| openstackgerrit | Daniel Blixt proposed zuul/nodepool master: Add shell-type to more providers https://review.opendev.org/c/zuul/nodepool/+/778580 | 08:37 |
| *** hashar has joined #zuul | 08:50 | |
| *** jpena|off is now known as jpena | 08:54 | |
| tobiash | corvus: what we might be able to do is to inject a 'repl-server-task' into the processor via repl | 08:56 |
| tobiash | like inception | 08:56 |
| *** dry has joined #zuul | 08:57 | |
| *** msuszko has quit IRC | 08:59 | |
| *** nils has joined #zuul | 09:00 | |
| tobiash | corvus: I don't see leaked git processes in our deployment. Do you think those leaked processes are a reason or a symptom of the oom kills? | 09:02 |
| avass | I don't think we're leaking git processes either | 09:10 |
| *** tosky has joined #zuul | 09:23 | |
| *** harrymichal has joined #zuul | 09:29 | |
| *** bhavikdbavishi1 has joined #zuul | 09:45 | |
| *** bhavikdbavishi has quit IRC | 09:47 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 09:47 | |
| *** wuchunyang has quit IRC | 10:16 | |
| *** holser has quit IRC | 10:21 | |
| openstackgerrit | Daniel Blixt proposed zuul/nodepool master: Add shell-type to more providers https://review.opendev.org/c/zuul/nodepool/+/778580 | 10:25 |
| *** saneax has quit IRC | 10:36 | |
| *** jangutter has joined #zuul | 10:41 | |
| *** jangutter has quit IRC | 10:43 | |
| *** jangutter has joined #zuul | 10:43 | |
| *** jangutter_ has quit IRC | 10:45 | |
| *** hashar has quit IRC | 11:03 | |
| *** bhavikdbavishi has quit IRC | 11:04 | |
| *** hashar has joined #zuul | 11:04 | |
| *** bhavikdbavishi has joined #zuul | 11:06 | |
| *** bhavikdbavishi has quit IRC | 11:18 | |
| *** bhavikdbavishi has joined #zuul | 11:18 | |
| *** ykarel_ has joined #zuul | 11:19 | |
| *** bhavikdbavishi1 has joined #zuul | 11:22 | |
| *** ykarel has quit IRC | 11:22 | |
| *** bhavikdbavishi has quit IRC | 11:23 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 11:23 | |
| *** ykarel_ is now known as ykarel | 11:23 | |
| *** jangutter has quit IRC | 11:29 | |
| *** jangutter has joined #zuul | 11:30 | |
| *** holser has joined #zuul | 11:35 | |
| *** holser has quit IRC | 11:37 | |
| *** holser has joined #zuul | 11:38 | |
| *** holser has quit IRC | 11:57 | |
| *** holser has joined #zuul | 12:01 | |
| *** ykarel_ has joined #zuul | 12:07 | |
| avass | does zuul not report anything back to gerrit when it fails to submit a change? | 12:08 |
| *** ykarel has quit IRC | 12:09 | |
| avass | I guess merge-failure refers to the 'MERGE_FAILURE' state. maybe there should also be a 'submit-failure' reporter here: https://zuul-ci.org/docs/zuul/reference/pipeline_def.html#attr-pipeline.merge-failure ? | 12:11 |
| *** zenkuro has joined #zuul | 12:24 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Make ZooKeeper mandatory for Scheduler https://review.opendev.org/c/zuul/zuul/+/756716 | 12:27 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Move setupZK() helper function to BaseTestCase class https://review.opendev.org/c/zuul/zuul/+/776639 | 12:27 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Initialize ZooKeeper connection in server rather than in cmd classes https://review.opendev.org/c/zuul/zuul/+/776640 | 12:27 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Make ConnectionRegistry mandatory for Scheduler https://review.opendev.org/c/zuul/zuul/+/757095 | 12:27 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Instantiate executor client, merger, nodepool and app within Scheduler https://review.opendev.org/c/zuul/zuul/+/757149 | 12:27 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: DNM: Reduce number of jobs for SOS development https://review.opendev.org/c/zuul/zuul/+/775081 | 12:27 |
| *** rlandy has joined #zuul | 12:33 | |
| *** jpena is now known as jpena|lunch | 12:34 | |
| *** hashar is now known as hasharLunch | 12:39 | |
| *** zenkuro has quit IRC | 12:49 | |
| *** zenkuro has joined #zuul | 12:49 | |
| swest | corvus: I got rid of the typing patches. first change in the sos stack is now https://review.opendev.org/c/zuul/zuul/+/756716/ | 12:52 |
| *** bhavikdbavishi has quit IRC | 13:04 | |
| *** hasharLunch is now known as hashar | 13:14 | |
| openstackgerrit | Jonas Sticha proposed zuul/zuul master: Make repo state buildset global https://review.opendev.org/c/zuul/zuul/+/738603 | 13:14 |
| *** bhavikdbavishi has joined #zuul | 13:21 | |
| *** jpena|lunch is now known as jpena | 13:28 | |
| *** jangutter has quit IRC | 13:33 | |
| *** jangutter has joined #zuul | 13:34 | |
| *** bhavikdbavishi has quit IRC | 13:42 | |
| *** jangutter has quit IRC | 13:43 | |
| *** jangutter has joined #zuul | 13:44 | |
| openstackgerrit | Daniel Blixt proposed zuul/nodepool master: Add shell-type to more providers https://review.opendev.org/c/zuul/nodepool/+/778580 | 13:46 |
| *** zenkuro has quit IRC | 13:56 | |
| *** zenkuro has joined #zuul | 13:57 | |
| openstackgerrit | Daniel Blixt proposed zuul/nodepool master: Add shell-type to more providers https://review.opendev.org/c/zuul/nodepool/+/778580 | 14:10 |
| openstackgerrit | Daniel Blixt proposed zuul/nodepool master: Add shell-type config https://review.opendev.org/c/zuul/nodepool/+/775371 | 14:14 |
| *** jangutter has quit IRC | 14:16 | |
| *** jangutter has joined #zuul | 14:17 | |
| *** ykarel_ is now known as ykarel | 14:32 | |
| *** zenkuro has quit IRC | 14:35 | |
| *** zenkuro has joined #zuul | 14:36 | |
| openstackgerrit | Daniel Blixt proposed zuul/zuul master: Use shell-type config from nodepool https://review.opendev.org/c/zuul/zuul/+/775382 | 14:41 |
| *** Phoenikzz has joined #zuul | 14:51 | |
| avass | corvus: wanna take another look at 775382 and 775371 if you get time over? | 14:52 |
| *** rpittau is now known as rpittau|afk | 15:03 | |
| corvus | avass: what happens in k8s with that? | 15:05 |
| avass | corvus: that's just configuring ansible so it knows how it should format it's commands I believe | 15:11 |
| avass | corvus: I can fire up a test just in case | 15:11 |
| corvus | avass: yeah, maybe it's used in the same way with the shell/command module in a kubectl connection type? that makes sense, but i just wanted to make sure. i'm guessing no effect for the k8s namespace case. | 15:15 |
| corvus | avass: left 2 more questions on https://review.opendev.org/775371 #2 might be a -1 unless i'm missing something cause i just woke up :) | 15:22 |
| avass | corvus: can probably check the azure part on monday, trying to fire up a k8s pod right now and namespaces shouldn't be affected :) | 15:25 |
| corvus | avass: cool, thanks! i just pushed up some major azure rework yesterday, but i will be happy to rebase that change on the shell-type change: https://review.opendev.org/778814 | 15:26 |
| avass | corvus: oh, nice I'll take a look at that later | 15:27 |
| avass | corvus: yeah running a containg using fish-shell needs ansible_shell_type: fish | 15:28 |
| avass | container* | 15:28 |
| corvus | aquarium? ;) | 15:28 |
| avass | heh :) | 15:29 |
| corvus | avass: cool, that makes sense. i just wanted to make sure i understood and that the k8s environment wasn't going to give us any surprises there | 15:29 |
| *** ykarel has quit IRC | 15:37 | |
| Phoenikzz | corvus: posted answers to questions | 15:40 |
| corvus | Phoenikzz: thanks! replied and +2 | 15:45 |
| *** jfoufas1 has quit IRC | 16:06 | |
| *** hashar is now known as hasharAway | 16:27 | |
| *** ikhan has quit IRC | 16:47 | |
| *** ikhan has joined #zuul | 17:33 | |
| *** jamesmcarthur has joined #zuul | 17:53 | |
| openstackgerrit | Vitaliy Lotorev proposed zuul/zuul master: Fix zuul.artifacts example in documentation https://review.opendev.org/c/zuul/zuul/+/778238 | 17:54 |
| fungi | if anyone is interested in seeing live stats from a fairly busy zuul deployment, opendev's is under what will probably be its peak load for the first half of this year, and seems to be performing quite well: https://grafana.opendev.org/d/5Imot6EMk/zuul-status | 17:55 |
| *** jpena is now known as jpena|off | 17:58 | |
| avass | fungi: nice! | 18:02 |
| *** irclogbot_3 has quit IRC | 18:03 | |
| *** irclogbot_1 has joined #zuul | 18:06 | |
| fungi | openstack is preparing for its next semi-annual release, and next week is the deadline for merging feature changes, so what you see there is the product of extreme procrastination on the part of some ~1k developers ;) | 18:08 |
| avass | yeah we get that too. everyone wants to merge at the same time :) | 18:10 |
| *** irclogbot_1 has quit IRC | 18:24 | |
| *** hasharAway has quit IRC | 18:27 | |
| *** irclogbot_3 has joined #zuul | 18:29 | |
| *** jamesmcarthur_ has joined #zuul | 18:34 | |
| *** jamesmcarthur has quit IRC | 18:38 | |
| *** holser has quit IRC | 18:42 | |
| *** holser has joined #zuul | 18:42 | |
| *** ikhan has quit IRC | 19:00 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 19:12 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 19:22 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 19:24 |
| openstackgerrit | Merged zuul/zuul master: Report executor stats per zone https://review.opendev.org/c/zuul/zuul/+/740448 | 19:35 |
| openstackgerrit | Merged zuul/zuul master: Update zoned executor stats https://review.opendev.org/c/zuul/zuul/+/777852 | 19:36 |
| openstackgerrit | Merged zuul/zuul master: Optionally allow zoned executors to process unzoned jobs https://review.opendev.org/c/zuul/zuul/+/673840 | 19:38 |
| *** jamesmcarthur_ has quit IRC | 19:52 | |
| *** jamesmcarthur has joined #zuul | 19:53 | |
| *** jamesmcarthur has quit IRC | 19:58 | |
| openstackgerrit | Gomathi Selvi Srinivasan proposed zuul/zuul-jobs master: Create a template for ssh-key and size https://review.opendev.org/c/zuul/zuul-jobs/+/773474 | 20:09 |
| tobiash | corvus: is there a reason to change from id to hostname in the azure change? | 20:59 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 21:03 |
| avass | something really really strange is happening on that ^ change. If I no_log aws_secret_key, aws_access_key also gets no_logged and ansible breaks the generated urls since it censors the aws_access_key value from them... | 21:05 |
| tobiash | corvus: commented on 773540 (and it's marked as wip which prevented it from entering the gate( | 21:05 |
| avass | oh... it no logs the value since the secret_key and the access_key is the same value | 21:10 |
| *** hamalq has joined #zuul | 21:14 | |
| *** ajitha has quit IRC | 21:18 | |
| corvus | tobiash: re azure -- it looks like id and hostname are equally unique (you can't have a duplicate hostname in the same resource group). so effectively the id becomes a bunch of stuff that doesn't matter followed by the hostname. let me get an example | 21:25 |
| corvus | tobiash: /subscriptions/........-....-....-....-............/resourceGroups/nodepool/providers/Microsoft.Compute/virtualMachines/bionic-azure-0000000001 | 21:26 |
| corvus | that's an id | 21:26 |
| corvus | so the real answer: that makes the external id column in "nodepool list" really long. :) | 21:26 |
| tobiash | corvus: k, hostname makes sense then :) | 21:26 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Format multi-line log entries https://review.opendev.org/c/zuul/nodepool/+/773540 | 21:34 |
| corvus | tobiash: re 773540 thanks... :) and, um, i totally definitely knew it was marked WIP and my secret plan i didn't tell anyone was to use it to test that the wip detection in zuul was working! | 21:36 |
| openstackgerrit | Merged zuul/zuul master: Fix zuul.artifacts example in documentation https://review.opendev.org/c/zuul/zuul/+/778238 | 21:36 |
| corvus | totally, unrelated: i think maybe i should have gertty highlight WIP in red. | 21:36 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 22:01 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 22:08 |
| openstackgerrit | Merged zuul/nodepool master: Add shell-type config https://review.opendev.org/c/zuul/nodepool/+/775371 | 22:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 22:17 |
| *** rlandy has quit IRC | 22:19 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 22:25 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 22:34 |
| *** nils has quit IRC | 22:38 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 22:46 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 22:56 |
| *** iurygregory has quit IRC | 22:57 | |
| *** sduthil has quit IRC | 23:01 | |
| *** ikhan has joined #zuul | 23:11 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: Update tests/base.py to use proper git data https://review.opendev.org/c/zuul/zuul/+/742746 | 23:14 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: Fix gerrit merge commit change with zuul configuration https://review.opendev.org/c/zuul/zuul/+/762886 | 23:14 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: Fix zuul-client enqueue-ref when oldrev/newrev aren't provided https://review.opendev.org/c/zuul/zuul/+/765767 | 23:14 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: Improve merger getFilesChanges, Fix edge cases https://review.opendev.org/c/zuul/zuul/+/762887 | 23:14 |
| *** ikhan has quit IRC | 23:22 | |
| *** iurygregory has joined #zuul | 23:22 | |
| *** harrymichal has quit IRC | 23:39 | |
| *** hamalq has quit IRC | 23:44 | |
| openstackgerrit | Merged zuul/nodepool master: Format multi-line log entries https://review.opendev.org/c/zuul/nodepool/+/773540 | 23:45 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: Improve merger getFilesChanges, Fix edge cases https://review.opendev.org/c/zuul/zuul/+/762887 | 23:47 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!