Wednesday, 2021-03-10

corvustristanC: we should probably merge https://review.opendev.org/779089 since the dep landed00:03
*** vishalmanchanda has quit IRC00:05
openstackgerritJames E. Blair proposed zuul/zuul master: Instantiate executor client, merger, nodepool and app within Scheduler  https://review.opendev.org/c/zuul/zuul/+/77908700:06
*** jamesmcarthur has quit IRC00:21
*** jamesmcarthur has joined #zuul00:22
openstackgerritMerged zuul/zuul-client master: Switch to TLS ZK based functional job  https://review.opendev.org/c/zuul/zuul-client/+/77908900:22
*** jamesmcarthur has quit IRC00:30
*** jamesmcarthur has joined #zuul00:35
*** jamesmcarthur has quit IRC00:35
*** tosky has quit IRC00:36
*** piotrowskim has quit IRC00:36
*** avass has quit IRC00:41
*** yourname has joined #zuul00:42
*** jamesmcarthur has joined #zuul01:08
*** jamesmcarthur has quit IRC01:17
*** jamesmcarthur has joined #zuul01:19
*** jamesmcarthur has quit IRC01:23
*** sassyn has quit IRC01:23
*** jamesmcarthur has joined #zuul01:23
openstackgerritJames E. Blair proposed zuul/nodepool master: Support threadless deletes  https://review.opendev.org/c/zuul/nodepool/+/77963701:25
corvusi think that's the last big missing piece for the state machine work; i think it's ready for polishing and then maybe porting some more drivers over01:25
*** jamesmcarthur has quit IRC01:26
corvustobiash, mordred: ^ fyi01:26
*** jamesmcarthur has joined #zuul01:27
*** iurygregory_ has joined #zuul01:30
*** jamesmcarthur has quit IRC01:32
*** jamesmcarthur has joined #zuul01:33
*** imtiazc has joined #zuul01:36
*** iurygregory has quit IRC01:38
*** mordred has quit IRC01:38
*** ttx has quit IRC01:38
*** jamesmcarthur has quit IRC01:39
*** jamesmcarthur has joined #zuul01:39
*** ttx has joined #zuul01:47
*** jamesmcarthur has quit IRC01:47
*** mordred has joined #zuul01:54
*** imtiazc has left #zuul01:56
*** imtiazc has joined #zuul02:06
*** hamalq has quit IRC02:14
*** jamesmcarthur has joined #zuul03:12
*** jamesmcarthur has quit IRC03:13
*** jamesmcarthur has joined #zuul03:13
*** jamesmcarthur has quit IRC03:13
*** jamesmcarthur has joined #zuul03:27
*** jamesmcarthur has quit IRC03:33
*** jamesmcarthur has joined #zuul03:34
*** jamesmcarthur has quit IRC03:38
*** jamesmcarthur has joined #zuul03:52
*** jamesmcarthur has quit IRC03:57
*** jamesmcarthur has joined #zuul03:58
*** jamesmcarthur has quit IRC04:05
*** jamesmcarthur has joined #zuul04:07
*** ykarel has joined #zuul04:16
*** bhavikdbavishi has joined #zuul04:32
*** bhavikdbavishi1 has joined #zuul04:34
*** bhavikdbavishi has quit IRC04:36
*** bhavikdbavishi1 is now known as bhavikdbavishi04:36
*** jamesmcarthur has quit IRC04:44
*** vishalmanchanda has joined #zuul04:59
*** ajitha has joined #zuul05:11
*** jamesmcarthur has joined #zuul05:27
*** jamesmcarthur has quit IRC05:31
*** evrardjp has quit IRC05:33
*** evrardjp has joined #zuul05:33
*** saneax has joined #zuul06:09
*** jfoufas1 has joined #zuul06:09
*** bhavikdbavishi has quit IRC06:27
*** jamesmcarthur has joined #zuul06:49
*** jamesmcarthur has quit IRC06:50
*** saneax has quit IRC07:08
*** wuchunyang has joined #zuul07:15
*** ajitha has quit IRC07:21
*** piotrowskim has joined #zuul07:40
*** bhavikdbavishi has joined #zuul07:43
*** bhavikdbavishi has quit IRC07:47
*** yourname is now known as avass07:48
*** saneax has joined #zuul07:52
avassmordred: wait til you learn about vi{"ay with vi{"ap  to copy one ansible jinja expression and replace another with it. ;)08:01
*** jcapitao has joined #zuul08:04
*** hashar has joined #zuul08:15
*** arxcruz|off is now known as arxcruz08:24
*** rpittau|afk is now known as rpittau08:27
openstackgerritTobias Henkel proposed zuul/zuul master: ci: use bionic for functional tests  https://review.opendev.org/c/zuul/zuul/+/70882008:37
*** tosky has joined #zuul08:46
*** jamesmcarthur has joined #zuul08:50
*** jamesmcarthur has quit IRC08:55
*** hashar has quit IRC08:55
*** jpena|off is now known as jpena08:58
*** hashar has joined #zuul09:04
*** saneax has quit IRC09:09
openstackgerritBenjamin Schanzel proposed zuul/zuul-jobs master: ensure-helm: skip helm init for helm v3 and newer  https://review.opendev.org/c/zuul/zuul-jobs/+/77969809:39
*** jangutter_ has joined #zuul09:55
*** wuchunyang has quit IRC09:56
*** jangutter has quit IRC09:59
*** ykarel is now known as ykarel|lunch10:00
lyrHi. I'm testing a zuul migration. I created /configured a new zuul instance, forked one of our repository, and I'm ending up with "Unable to freeze job graph: Decryption failed.". I guess that's a missing secret issue, but I don't get it, this repository got no secret10:02
openstackgerritBenjamin Schanzel proposed zuul/zuul-jobs master: ensure-helm: skip helm init for helm v3 and newer  https://review.opendev.org/c/zuul/zuul-jobs/+/77969810:03
*** harrymichal has joined #zuul10:13
*** sduthil has quit IRC10:29
*** sduthil has joined #zuul10:29
*** dry has quit IRC10:36
*** Eighth_Doctor has quit IRC10:36
*** mordred has quit IRC10:36
*** dry has joined #zuul10:38
*** jangutter has joined #zuul10:42
*** jangutter_ has quit IRC10:45
*** masterpe has joined #zuul10:46
*** hashar has quit IRC10:53
*** Eighth_Doctor has joined #zuul11:04
*** mordred has joined #zuul11:04
*** jangutter has quit IRC11:07
*** jangutter has joined #zuul11:07
*** decimuscorvinus has quit IRC11:07
*** decimuscorvinus has joined #zuul11:08
*** jangutter_ has joined #zuul11:18
*** ykarel|lunch is now known as ykarel11:18
*** jangutter has quit IRC11:22
openstackgerritDong Zhang proposed zuul/zuul master: Display branch of queue in status page  https://review.opendev.org/c/zuul/zuul/+/77761311:31
*** iurygregory_ is now known as iurygregory11:33
*** harrymichal has left #zuul11:38
*** saneax has joined #zuul11:41
*** jangutter has joined #zuul11:43
*** jangutter_ has quit IRC11:46
*** jcapitao is now known as jcapitao_lunch11:49
*** ykarel_ has joined #zuul12:17
*** ykarel has quit IRC12:20
*** jpena is now known as jpena|lunch12:34
zbravass: what should I do about unstable roles in zuul-roles? I am trying to finish the linter upgrade to v5 but I keep encountering broken stuff, unrelated to my changes.13:05
zbri do not have time or energy to fix everything...13:06
*** jcapitao_lunch is now known as jcapitao13:06
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Bits to keep ansible-lint happy  https://review.opendev.org/c/zuul/zuul-jobs/+/77927613:11
openstackgerritDong Zhang proposed zuul/zuul master: Display branch of queue in status page  https://review.opendev.org/c/zuul/zuul/+/77761313:12
avasszbr: are tests failing or is it mostly the linter upgrade that doesn't like parts of zuul-jobs?13:13
zbravass: yeah, sorted, false alarm. I was confused by the history run which showed m 80% failure in the last 20 days.13:13
*** ykarel_ is now known as ykarel13:15
*** ajitha has joined #zuul13:24
*** jangutter_ has joined #zuul13:25
*** jangutter has quit IRC13:30
avasszbr: ah :)13:37
zbravass: i am now working to port the no-owner rule to the linter, i need to implement new feature: rule disabled by default.13:37
*** jpena|lunch is now known as jpena13:40
openstackgerritSimon Westphahl proposed zuul/zuul master: Component Registry in ZooKeeper  https://review.opendev.org/c/zuul/zuul/+/75918713:41
openstackgerritSimon Westphahl proposed zuul/zuul master: Move management and result events to model  https://review.opendev.org/c/zuul/zuul/+/76116313:41
openstackgerritSimon Westphahl proposed zuul/zuul master: Allow (de-)serialization of management events  https://review.opendev.org/c/zuul/zuul/+/76116413:41
openstackgerritSimon Westphahl proposed zuul/zuul master: Allow (de-)serialization of result events  https://review.opendev.org/c/zuul/zuul/+/76116513:41
openstackgerritSimon Westphahl proposed zuul/zuul master: Add and fix fields in driver trigger event models  https://review.opendev.org/c/zuul/zuul/+/76116613:41
openstackgerritSimon Westphahl proposed zuul/zuul master: Allow (de-)serialization of trigger events  https://review.opendev.org/c/zuul/zuul/+/76116713:41
openstackgerritSimon Westphahl proposed zuul/zuul master: DNM: Reduce number of jobs for SOS development  https://review.opendev.org/c/zuul/zuul/+/77508113:41
*** jangutter_ has quit IRC13:44
*** jangutter has joined #zuul13:45
*** nils has joined #zuul13:51
openstackgerritBenedikt Löffler proposed zuul/nodepool master: Support openstack server groups  https://review.opendev.org/c/zuul/nodepool/+/76354013:57
*** GomathiselviS has joined #zuul14:34
tristanCwe are trying to measure the resources used by a tenant, and it seems like the `zuul.nodepool.resources.tenant.name.cores' or '.ram' gauge are not working very well (the value somehow increase and never goes back to zero)14:37
GomathiselviScorvus - hi, Can you please review https://review.opendev.org/c/zuul/zuul-jobs/+/773474, when you get the time?14:38
tristanCis there existing documentation or tool that can be used to measure tenant usage?14:39
fungitristanC: sounds like a counter instead of a gauge? but yeah when clarkb's around he's been doing a lot of per-project usage measurements, that probably generalizes to per-tenant fairly well (though may be an unnecessarily complicated way to approach it, i'm not sure)14:42
corvustristanC, fungi: yeah, there's a counter and a gauge: https://zuul-ci.org/docs/zuul/reference/monitoring.html#stat-zuul.nodepool.requests.resources14:43
corvusso the gauge should go up and down with current usage, and the counter should always increase.  it represents, for example cpu-seconds.14:44
tristanCfungi: corvus: we are using the gauge, and it seems to work fine, up to a point where the values get stuck, as if a single `subtract_resources` failed14:44
corvusoh hrm14:45
tristanCand then it get stuck again to a higher minimum14:45
tristanCor perhaps this is related to our influxdb backend. Would it be possible to graph those in opendev?14:46
corvushttps://graphite.opendev.org/?width=586&height=308&target=stats.gauges.zuul.nodepool.resources.tenant.zuul.cores14:46
corvusthat seems to be moving up and down over the past 24h14:47
tristanCcorvus: thanks, that looks correct14:48
corvusthough our minimum is 8 on that graph14:49
corvusi wonder if that's a bug, or a held node....14:49
clarkbtristanC: corvus  I think tobiash has been using those values14:50
clarkbmay have insight14:51
tristanCcorvus: here is our: https://softwarefactory-project.io/grafana/d/e78QjA8Gk/fedora-tenant?viewPanel=3&from=now-30d&to=now&orgId=114:51
tristanCoh, could it be that held node are not removed from the gauge when deleted?14:51
tobiashwe use that metric14:52
corvuscould be; i did hold at least one node for a job in the zuul tenant since the restart; i can't remember if i held more14:52
tobiash(via statsd-exporter)14:52
tristanCtobiash: and are you having this issue where the gauge nevers goes back to 0?14:53
tobiashour load never goes to 0 so cannot tell exactly14:53
tobiashtristanC: maybe you've hit a leak there14:54
tobiashat least it should go back to near 0 :D14:54
corvustristanC, tobiash: i think it's holds14:55
tristanCit does seems like zuul calls subtract_resources for held node14:55
corvusoh?14:55
tristanCdoes not*14:55
corvusah yes14:55
corvusthat's what i was seeing :)14:55
corvusit looks like it does not subtract for held nodes; it only does it in returnNodeSet, but that isn't called when we call holdNodeSet14:56
tobiashso held nodes make the metric leak?14:56
corvusso we should probably duplicate the subtract calls in holdNodeSet14:56
corvusor if we're pretty sure no one is deleting held nodes except via "zuul autohold-delete" (or api equivalent) then we could do the subtract there14:57
corvusthat would be slightly more accurate, but only if people don't "nodepool delete" the held nodes14:58
corvustristanC: nice catch :)14:58
tristanCthank you for the prompt feedback, it's good to know that can be used for tenant usage report14:59
*** jamesmcarthur has joined #zuul15:00
tristanCso would it be ok to also substract the resource for held node, i think it would be better to track those held usage in another metric/dashboard15:02
openstackgerritJonas Sticha proposed zuul/nodepool master: aws: add support for uploading diskimages  https://review.opendev.org/c/zuul/nodepool/+/73521715:37
avasscorvus: where are those logged? we don't seem to be getting those stats at all so I guess they're limited to a specific provider or something?15:45
openstackgerritJonas Sticha proposed zuul/nodepool master: aws: add image upload test  https://review.opendev.org/c/zuul/nodepool/+/77584415:49
avasswe are getting other zuul and nodepool stats so I think statsd is working otherwise15:49
openstackgerritJonas Sticha proposed zuul/nodepool master: aws: add image upload test  https://review.opendev.org/c/zuul/nodepool/+/77584415:51
*** jangutter_ has joined #zuul15:51
pabelangerthanks for ^, would use15:52
*** dmsimard8 has joined #zuul15:52
*** dmsimard has quit IRC15:54
*** dmsimard8 is now known as dmsimard15:54
avasspabelanger: yeah I'm really happy someone is working on that :)15:54
*** jangutter has quit IRC15:55
*** harrymichal has joined #zuul15:56
tobiashavass: the resources are dependent on resource information in the node zk data structure which is not filled by every nodepool driver16:03
avasstobiash: that's what I thought. maybe it's worth updating the aws driver for that16:05
tobiash++16:06
zbravass: adoption of last custom rule from zuul: https://github.com/ansible-community/ansible-lint/pull/145016:16
avasszbr: nice16:18
*** jangutter has joined #zuul16:35
*** jangutter_ has quit IRC16:39
jangutterSo... super newbie question, but when a failure occurs in the pre-run portion of a parent job, I see zuul retrying before eventually failing. Is there a way in the job I can fail fast (i.e. tell zuul not to bother retrying)?16:40
corvusjangutter: yep see https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.attempts16:44
corvusoh, *in the job*?  like, in the playbook itself, not the job definition?  in that case -- no i don't think so16:44
jangutteryeah - totally esoteric, for example if pre-run-a fails, don't retry, otherwise, retry 3 times.16:46
jangutterI was wondering if you could do that with zuul_return for example.16:46
corvusavass: i don't think those starts are logged, i think they're only reported via statsd16:47
corvusjangutter: yeah, i don't think that exists, but i do think it's feasible to add it as you describe16:47
janguttercorvus: thanks, it's not a high priority, I was just wondering if I was missing something obvious. Global retries is not so bad.16:49
*** hashar has joined #zuul16:54
clarkbyou might be able to move that into a run playbook (as those don't get retried unless it is detected as an ansible networking failure)16:57
avasscorvus: sorry that's what I meant17:03
corvusavass: so you're not seeing that metric being reported?17:03
avasscorvus: no but as tobiash explained they're not reported by all nodepool drivers17:03
corvusavass: ah yep, that seems likely.  the azure driver will have it when i'm done.17:05
*** rpittau is now known as rpittau|afk17:05
avasscorvus: nice! :)17:05
*** jcapitao has quit IRC17:08
avasscorvus: I've been bogged down with other tasks but I think I got time to take a look at the azure stack tomorrow17:15
tobiashavass: this is where the openstack driver does this: https://opendev.org/zuul/nodepool/src/branch/master/nodepool/driver/openstack/handler.py#L15717:17
corvusavass: cool, only the first change is ready for review yet, the changes after that are very much WIP (failing tests, pep8 issues, etc).  but the stack does work against actual azure :)17:19
*** jfoufas1 has quit IRC17:26
*** jangutter_ has joined #zuul17:26
*** jangutter has quit IRC17:31
openstackgerritMerged zuul/zuul master: Reduce amount of 'Waiting on logger' messages sent  https://review.opendev.org/c/zuul/zuul/+/77788717:36
*** akrpan-pure has joined #zuul17:40
avasstobiash: yeah I remember seeing that when working on the digitalocean driver but never realized it was reporting it to statsd17:49
*** ykarel has quit IRC17:50
*** saneax has quit IRC17:53
*** jpena is now known as jpena|off17:58
*** piotrowskim has quit IRC18:00
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Bits to keep ansible-lint happy  https://review.opendev.org/c/zuul/zuul-jobs/+/77927618:04
*** akrpan-pure has quit IRC18:22
clarkbtobiash: you mentioned that you alredy start up new launchers for the same provider side by side. Do you think you can review https://review.opendev.org/c/zuul/nodepool/+/779616 which will make that process a bit more robust18:30
openstackgerritTristan Cacqueray proposed zuul/zuul master: WIP: replace console-stream websocket with event stream  https://review.opendev.org/c/zuul/zuul/+/77958118:36
openstackgerritAlbin Vass proposed zuul/zuul master: Add messages to make the job setup more transparent  https://review.opendev.org/c/zuul/zuul/+/77788518:42
*** reiterative has quit IRC18:45
*** reiterative has joined #zuul18:46
*** jamesmcarthur has quit IRC18:50
*** jamesmcarthur has joined #zuul18:50
*** jamesmcarthur has quit IRC18:51
*** jamesmcarthur has joined #zuul18:52
*** jamesmcarthur has quit IRC18:52
*** jamesmcarthur has joined #zuul18:52
*** jamesmcarthur has quit IRC18:57
fungisince zuul is our main consumer of gear, at least for the present, i'd appreciate some zuul-focused eyes on changes 714709 and 747119 (particularly the second) to help confirm they're not going to present a problem19:16
*** GomathiselviS has quit IRC19:16
fungithanks tristanC, tobiash and hashar! i'm more comfortable approving those changes knowing that you support them19:25
*** dpawlik6 has joined #zuul19:34
*** Tahvok_ has joined #zuul19:35
*** mugsie__ has joined #zuul19:37
*** tosky_ has joined #zuul19:37
*** smyers_ has joined #zuul19:37
*** paulalbertella has joined #zuul19:38
*** paladox_ has joined #zuul19:41
*** reiterative has quit IRC19:42
*** dmsimard has quit IRC19:42
*** tosky has quit IRC19:42
*** Tahvok has quit IRC19:42
*** dpawlik has quit IRC19:42
*** paladox has quit IRC19:42
*** mugsie has quit IRC19:42
*** smyers has quit IRC19:42
*** dpawlik6 is now known as dpawlik19:42
*** Tahvok_ is now known as Tahvok19:42
*** smyers_ is now known as smyers19:42
*** dmsimard has joined #zuul19:43
*** irclogbot_2 has quit IRC19:44
*** paladox_ is now known as paladox19:45
*** mugsie__ is now known as mugsie19:47
*** ajitha has quit IRC19:54
tobiashclarkb: lgtm20:02
clarkbthanks20:02
tobiashclarkb: could you put https://review.opendev.org/c/zuul/zuul-client/+/775694 to your review list? It improves help messages of zuul-client.20:03
clarkbtobiash: yup20:03
tobiashthanks :)20:03
*** harrymichal has quit IRC20:04
*** harrymichal has joined #zuul20:04
clarkbcorvus: tobiash  ^ two comments on there, one is a functional problem but the other I'm fine with either way, more a matter of preference20:09
tobiashclarkb: good catch20:12
tobiashclarkb: regarding the raise, I don't seem to get a traceback in local testing20:13
clarkbthe nodepool functional k8s and openshift jobs failed on the launcher id change, I'm looking into it20:15
*** tosky_ is now known as tosky20:16
tobiashclarkb: I'd bet it's the rate limit20:18
clarkbit looks like timeouts launching k8s labels https://zuul.opendev.org/t/zuul/build/a066e70f3c6942e4a2d2593b8cc11b88/log/job-output.txt#187920:19
clarkbthe minikube log seems to indicate it successfully made a k8s20:20
tobiashthe openshift one timeouted during openshift installation20:20
clarkbslow nodes maybe?20:21
tobiashmaybe20:23
tobiashthe k8s one also has plenty of error messages in minikube20:23
tobiashI'd recheck20:23
clarkbthe same jobs passed in check so I'm reasonably confident it isn't related to the cahnge20:23
clarkbok rechecking20:23
*** tosky has quit IRC20:49
*** tosky has joined #zuul20:49
*** hashar has quit IRC20:52
openstackgerritMerged zuul/zuul-client master: Add builds subcommand  https://review.opendev.org/c/zuul/zuul-client/+/75099920:56
*** nils has quit IRC21:01
*** hashar has joined #zuul21:14
openstackgerritTobias Henkel proposed zuul/zuul-client master: Output subcommand help on argument errors  https://review.opendev.org/c/zuul/zuul-client/+/77569421:27
avasstobiash: should the ram usage be reported in Gb or Mb?21:27
tobiashavass: I think it's gb21:28
avasstobiash: then I'll have to convert it from Mb to Gb :)21:30
openstackgerritAlbin Vass proposed zuul/nodepool master: Add quota information to aws driver  https://review.opendev.org/c/zuul/nodepool/+/77986821:32
avasssomething like that ^, not tested yet but I can see if I get time over tomorrow. It's a bit weird since ram is only reported by describe-instance-types while cores actually used is reported by the created instance (but not used ram)21:34
tobiashavass: I think I was wrong, it's Mb21:35
openstackgerritAlbin Vass proposed zuul/nodepool master: Add quota information to aws driver  https://review.opendev.org/c/zuul/nodepool/+/77986821:35
fungiMb or MB (or MiB)? they're not the same ;)21:35
fungithat is, megabits, megabytes, or mebibytes?21:36
tobiashfungi: don't care actually, it's just forwarding the data it gets from the cloud ;)21:36
tobiashmost probably mebibytes to be exact21:36
avassfungi: don't forget about windows MB ;)21:38
avasswindows reports mebibytes but abbreviates it as MB21:40
*** saneax has joined #zuul21:40
corvuswindows probably classifies pluto is a planet too21:44
fungiooh, burn21:44
*** jamesmcarthur has joined #zuul21:44
*** saneax has quit IRC21:46
*** johnsom has quit IRC21:46
*** saneax has joined #zuul21:46
*** johnsom has joined #zuul21:46
openstackgerritMerged zuul/nodepool master: Uniquely identify launchers  https://review.opendev.org/c/zuul/nodepool/+/77961622:01
*** vishalmanchanda has quit IRC22:08
*** saneax has quit IRC22:17
openstackgerritMerged zuul/zuul-client master: Output subcommand help on argument errors  https://review.opendev.org/c/zuul/zuul-client/+/77569422:32
*** maxamillion has quit IRC22:34
*** maxamillion has joined #zuul22:35
*** harrymichal has quit IRC22:40
*** hashar has quit IRC22:45
*** ericsysmin has quit IRC23:08
*** ericsysmin has joined #zuul23:08
openstackgerritJames E. Blair proposed zuul/nodepool master: Add a state machine driver framework  https://review.opendev.org/c/zuul/nodepool/+/77907123:23
openstackgerritJames E. Blair proposed zuul/nodepool master: Add an example state machine driver  https://review.opendev.org/c/zuul/nodepool/+/77907223:23
openstackgerritJames E. Blair proposed zuul/nodepool master: Improve state machine keyscan handling  https://review.opendev.org/c/zuul/nodepool/+/77937923:23
openstackgerritJames E. Blair proposed zuul/nodepool master: Support threadless deletes  https://review.opendev.org/c/zuul/nodepool/+/77963723:23
openstackgerritJames E. Blair proposed zuul/nodepool master: WIP: add azure state machine driver  https://review.opendev.org/c/zuul/nodepool/+/77942023:23
*** rpittau|afk has quit IRC23:37
*** rpittau|afk has joined #zuul23:37
*** jamesmcarthur has quit IRC23:43
*** jamesmcarthur has joined #zuul23:43
*** jamesmcarthur has quit IRC23:47
*** jamesmcarthur has joined #zuul23:47
*** jamesmcarthur has quit IRC23:52
*** jamesmcarthur has joined #zuul23:52

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!