Thursday, 2023-02-02

@clarkb:matrix.orgI believe ttx has a couple of talks there00:09
@chgans:matrix.orgHi there, what are you using to connect Gerrit/Zull with matrix/element? Some custom code or is there any official plugins? (that i couldn't find)00:15
@clarkb:matrix.org> <@chgans:matrix.org> Hi there, what are you using to connect Gerrit/Zull with matrix/element? Some custom code or is there any official plugins? (that i couldn't find)00:16
Its the gerritbox-matrix bot that tristanC wrote. Let me see if I can find a link to it
@clarkb:matrix.orghttps://softwarefactory-project.io/r/plugins/gitiles/software-factory/gerritbot-matrix.git/ I'm not sure that is the best link but should be cloneable from there00:17
@chgans:matrix.orgawesome, i'll have a look, thanks!00:18
@chgans:matrix.orgwow, haskell 🤔 the documentation is sparse... :P00:20
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 872482: Fix race condition in pipeline change list init https://review.opendev.org/c/zuul/zuul/+/87248200:33
@clarkb:matrix.org> <@chgans:matrix.org> wow, haskell 🤔 the documentation is sparse... :P00:35
This is opendev's ansible role to manage our deployment of it (the one running for this room) that might help as far as knowing what config to set etc https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/matrix-gerritbot
-@gerrit:opendev.org- Ian Wienand proposed:00:54
- [zuul/zuul-jobs] 872365: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365
- [zuul/zuul-jobs] 872258: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258
- [zuul/zuul-jobs] 872375: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 872485: Simplufy shouldRefresh* checks https://review.opendev.org/c/zuul/zuul/+/87248501:37
@jim:acmegating.comianw: is that stack ready?  (i've been waiting for it to settle down before taking a look)01:39
@iwienand:matrix.orgcorvus: haha it was, but ansible-lint has released and broken the gate.  i'm having a look now01:39
@jim:acmegating.comok.  i'll leave you to it.  consider me interested when it's ready :)01:40
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 871106: Require latest layout for processing mgmt events https://review.opendev.org/c/zuul/zuul/+/87110601:45
-@gerrit:opendev.org- Ian Wienand proposed:04:08
- [zuul/zuul-jobs] 872371: linters-requirements : update Ansible to 2.12 https://review.opendev.org/c/zuul/zuul-jobs/+/872371
- [zuul/zuul-jobs] 872365: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365
- [zuul/zuul-jobs] 872258: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258
- [zuul/zuul-jobs] 872375: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375
- [zuul/zuul-jobs] 872489: ansible-lint: Ignore some command-instead-of-module warnings https://review.opendev.org/c/zuul/zuul-jobs/+/872489
- [zuul/zuul-jobs] 872490: ansible-lint: fix a bunch of command-instead-of-shell errors https://review.opendev.org/c/zuul/zuul-jobs/+/872490
- [zuul/zuul-jobs] 872491: ansible-lint: add names to blocks/includes, etc. https://review.opendev.org/c/zuul/zuul-jobs/+/872491
- [zuul/zuul-jobs] 872492: ansible-lint: ignore use of mkdir https://review.opendev.org/c/zuul/zuul-jobs/+/872492
- [zuul/zuul-jobs] 872493: ansible-lint: use pipefail https://review.opendev.org/c/zuul/zuul-jobs/+/872493
- [zuul/zuul-jobs] 872494: ansible-lint: ignore latest git pull https://review.opendev.org/c/zuul/zuul-jobs/+/872494
- [zuul/zuul-jobs] 872495: ansible-lint: uncap https://review.opendev.org/c/zuul/zuul-jobs/+/872495
-@gerrit:opendev.org- Ian Wienand proposed:04:09
- [zuul/zuul-jobs] 872371: linters-requirements : update Ansible to 2.12, ansible-lint <6.12.0 https://review.opendev.org/c/zuul/zuul-jobs/+/872371
- [zuul/zuul-jobs] 872489: ansible-lint: Ignore some command-instead-of-module warnings https://review.opendev.org/c/zuul/zuul-jobs/+/872489
- [zuul/zuul-jobs] 872490: ansible-lint: fix a bunch of command-instead-of-shell errors https://review.opendev.org/c/zuul/zuul-jobs/+/872490
- [zuul/zuul-jobs] 872491: ansible-lint: add names to blocks/includes, etc. https://review.opendev.org/c/zuul/zuul-jobs/+/872491
- [zuul/zuul-jobs] 872492: ansible-lint: ignore use of mkdir https://review.opendev.org/c/zuul/zuul-jobs/+/872492
- [zuul/zuul-jobs] 872493: ansible-lint: use pipefail https://review.opendev.org/c/zuul/zuul-jobs/+/872493
- [zuul/zuul-jobs] 872494: ansible-lint: ignore latest git pull https://review.opendev.org/c/zuul/zuul-jobs/+/872494
- [zuul/zuul-jobs] 872495: ansible-lint: uncap https://review.opendev.org/c/zuul/zuul-jobs/+/872495
- [zuul/zuul-jobs] 872365: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365
- [zuul/zuul-jobs] 872258: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258
- [zuul/zuul-jobs] 872375: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375
@iwienand:matrix.orgof course i'm sure that will touch a bunch of jobs that have now bitrotted ... anyway, fingers crossed04:10
@avass:vassast.orgApparently curl has stopped using zuul for their ci now: https://daniel.haxx.se/blog/2023/02/01/curls-use-of-many-ci-services/07:14
@q:fricklercloud.de> <@avass:vassast.org> Apparently curl has stopped using zuul for their ci now: https://daniel.haxx.se/blog/2023/02/01/curls-use-of-many-ci-services/09:11
and with good reasoning https://mastodon.social/@bagder/109782783782628905
@avass:vassast.orgYep. I'm also not a huge fan of the GitHub interface for zuul (like he says all jobs show up as the same job in GitHub). And i tried to help out when they had issues but i didn't have access to the system they were using so i couldn't actually debug or fix anything.09:15
@avass:vassast.orgThey also weren't really using any zuul features since they didn't want to be locked in to the system 09:26
@mhuin:matrix.org> <@jim:acmegating.com> ttx mhu sometimes go; pinging them for input; i won't be able to attend this year :(09:29
Sorry, not this year ... But I do make sure to advertise the project as much as possible: https://i.postimg.cc/zfZ8wBV7/20230202-102430.jpg
@mhuin:matrix.org> <@chgans:matrix.org> wow, haskell 🤔 the documentation is sparse... :P09:32
it should be relatively straightforward to use but if you need some help you can ask here https://matrix.to/#/#softwarefactory-project:matrix.org
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 872519: Fix serialization of ProjectContext objects https://review.opendev.org/c/zuul/zuul/+/87251909:50
@tcarrez:matrix.orgI'll be at FOSDEM... but not doing a Zuul talk. I haven't seen any Zuul content10:15
@mhuin:matrix.orgThe CI track is not as loaded as in the previous years10:25
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 810955: GUI: Add tenant dropdown to top menu https://review.opendev.org/c/zuul/zuul/+/81095510:48
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 794854: Test zuul-client console-stream https://review.opendev.org/c/zuul/zuul/+/79485410:49
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 771962: zuul-client: test "show running-jobs" command https://review.opendev.org/c/zuul/zuul/+/77196210:50
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 872552: Keycloak tutorial: update to keycloak 20, add CLI example https://review.opendev.org/c/zuul/zuul/+/87255212:21
@westphahl:matrix.orgClark: replied to your comment on https://review.opendev.org/c/zuul/zuul/+/871108/13:23
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 868131: Track object versions in the Buildset object https://review.opendev.org/c/zuul/zuul/+/86813114:06
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 810955: GUI: Add tenant dropdown to top menu https://review.opendev.org/c/zuul/zuul/+/81095514:41
@jim:acmegating.comIt looks like curl was using a very old zuul run by a commercial service.  it seems a little disingenuous to say that the "zuul team" didn't respond to their issues in that situation.  but i certainly understand their decision.14:45
@buktop:matrix.orgHi all,14:49
We have some proposed changes for extending the openshift nodepool drivers and we would appreciate if someone can review them.
- https://review.opendev.org/c/zuul/nodepool/+/871060
- https://review.opendev.org/c/zuul/nodepool/+/867971
- https://review.opendev.org/c/zuul/nodepool/+/872318
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 842290: Remove zuul CLI, REST ops https://review.opendev.org/c/zuul/zuul/+/84229015:23
@clarkb:matrix.orgswest: corvus the comments on those pipeline event cleanup changes make sense to me. Feel free to approve. Or I'll hit the +A button in a few minutes after distro updates reboots and logging back into gerrit16:14
@mhuin:matrix.orgdunno if it's just me being unlucky but it seems like the zuul-build-image has been failing for the last 4 hours, RETRY_LIMITS hit for 3 very different patches (one on doc, one on GUI, one on client)16:39
@jim:acmegating.commhu if it's failing on different jobs, then yeah, it's likely unluck.  possibly hitting docker rate limits on re-used ip addresses.16:40
@mhuin:matrix.orglast error is "AppArmor enabled on system but the docker-default profile could not be loaded" - is that new?16:40
@jim:acmegating.comoh that's different16:40
@jim:acmegating.commhu: there's some talk in #opendev about that16:40
@clarkb:matrix.orgyes, docker-ce 23.0.0 released a few hours ago and depends on the apparmor package but does not install it16:40
@clarkb:matrix.orgI think we can/should update the role installing docker-ce to also explicitly install apparmor and that should hopefully fix things16:41
@jim:acmegating.comso the randomness might be due to different package indexes16:41
@mhuin:matrix.orgthere you go - thanks corvus and Clark 16:41
@jim:acmegating.comClark: ++16:41
@clarkb:matrix.orgmhu: are you interested in writing that change? If not can you link to one of the job logs so that we can be sure we update the correct location16:42
@mhuin:matrix.orgClark, if you know which role to update, go for it, as it would take me some time to pinpoint it16:43
@clarkb:matrix.orgI think it is he ensure-docker role. Let me take a look16:44
@mhuin:matrix.orgClark: these are the RETRY_LIMITs I was impacted with https://zuul.opendev.org/t/zuul/builds?job_name=zuul-build-image&project=zuul%2Fzuul&result=RETRY_LIMIT&skip=0&limit=316:44
@mhuin:matrix.org"Task 16:45
Start the buildset registry
failed running on host
ubuntu-jammy"
@mhuin:matrix.orgis that a haiku?16:45
@mhuin:matrix.org(I just copy/pasted the error from one of the failed builds' status page)16:45
@fungicide:matrix.orga terrifying haiku16:47
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 872569: Install apparmor when installing upstream docker-ce https://review.opendev.org/c/zuul/zuul-jobs/+/87256916:48
@mhuin:matrix.orgClark: ++16:48
@clarkb:matrix.orgThat may do it. I put a note in the commit message that opensuse may need a similar fix too and CI should tell us if we run ensure-docker on opensuse16:48
@avass:vassast.org> <@jim:acmegating.com> It looks like curl was using a very old zuul run by a commercial service.  it seems a little disingenuous to say that the "zuul team" didn't respond to their issues in that situation.  but i certainly understand their decision.16:55
Yeah vexxhost was hosting it for them
@clarkb:matrix.orgLooks like `docker buildx create --name foo` is no longer valid under docker 23.0.0. I'm not yet sure what the fix for that is. The docs haven't been udpated for new docker yet...17:04
@jim:acmegating.comClark: is that causing that change to fail?  should we be pinning instead?17:07
@clarkb:matrix.orgcorvus: yes, in the test job that tests multiarch builds (these depend on buildx to set up the emulated build env)17:10
@clarkb:matrix.orgI put a hold on the zuul-jobs-test-registry-docker job so that I can interact with it directly a bit more17:16
@clarkb:matrix.orgI'm beginning to suspect that 23 doesn't include buildx in the packages like 20 did. Trying to confirm this now17:22
@clarkb:matrix.orgya there is a new separate package17:22
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 872569: Fix ensure-docker for docker-ce 23.0.0 upstream packaging https://review.opendev.org/c/zuul/zuul-jobs/+/87256917:27
@clarkb:matrix.orgOk this should hopefully get us closer. THe registry build test jobs use ensure-docker out of a trusted context which means we may need to force merge this...17:28
@clarkb:matrix.orgyup first multiarch job has passed. I need to eat breakfast, but other zuulians (and opendev admins) should think about how we want ot handle the base-job usage of the roll. I think whether we pin or roll forward a force merge is likely17:35
@jim:acmegating.comClark: thanks; i think force-merge roll-forward sounds good -- especially since we're already broken.17:38
@fungicide:matrix.orgi've +2'd the new revision, but didn't approve it because i didn't want to cancel the check builds and possibly slow discovery of yet another problem with it17:38
@jim:acmegating.comif we force-merge, it might be good to push up a change that runs all the registry test jobs afterwords to double check it worked as expected.17:39
@clarkb:matrix.orgya a noop change to ensure-docker would do it I think17:44
@clarkb:matrix.orgzuul-jobs-test-ensure-kubernetes-crio-ubuntu-bionic is also failing. I don't really grasp the error message17:45
@clarkb:matrix.orgif related I don't think that one will be any worse off with the force merge though17:46
@jim:acmegating.comClark: it started failing ~1 week ago17:49
@clarkb:matrix.orgalso the linters job fails but ianw has a stack to fix that and I believe it to be independent too17:49
@jim:acmegating.comso likely unrelated, but also likely another real problem.  i believe ianw felt the crio path for that had a limited lifespan anyway.  it's conceivable that the fix for that is to remove the job.17:50
@jim:acmegating.com * the crio error is likely unrelated, but also likely another real problem.  i believe ianw felt the crio path for that had a limited lifespan anyway.  it's conceivable that the fix for that is to remove the job.17:50
@clarkb:matrix.orgcool. In that case I'm not seeing anything yet that says "force merging this is a terrible idea17:51
@jim:acmegating.comClark: yeah i think maybe we've seen all the green we need/expect to see at this point17:51
@clarkb:matrix.orgcorvus: tox-py38 is the last one I expect to be green. Maybe wait on it?17:51
@jim:acmegating.comClark: yep.  i'm just assuming that the one in the middle isn't going to fail for a legit reason.  hopefully that doesn't jinx it.17:55
@jim:acmegating.com(also nothing in this change should touch that)17:55
@clarkb:matrix.orgya good point. Looks like it is finishing up and should be successful though. Who wants to push the buttons? SHould I?17:55
@jim:acmegating.comClark: i vote you :)17:56
@clarkb:matrix.orgok working on it. Will take a few minutes while I sort out gerrit perms and stuff17:56
@fungicide:matrix.orgi support bypassing zuul to merge this change17:56
@fungicide:matrix.orgthanks!17:56
-@gerrit:opendev.org- cboylan.admin merged on behalf of Clark Boylan: [zuul/zuul-jobs] 872569: Fix ensure-docker for docker-ce 23.0.0 upstream packaging https://review.opendev.org/c/zuul/zuul-jobs/+/87256918:00
@clarkb:matrix.orgDone18:00
@clarkb:matrix.orgI'll push up a noop chagne to ensure-docker to check that the buildset registry jobs are happy now18:01
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 872594: DNM double checking docker 23 fixes https://review.opendev.org/c/zuul/zuul-jobs/+/87259418:02
@clarkb:matrix.orgthe promote jobs for the force merged change failed but that is due to no artifact being available. I believe this is expected18:04
@clarkb:matrix.orghttps://github.com/moby/moby/issues/44900 upstream is aware of the apparmor issue18:05
@clarkb:matrix.orgwe might be able to rollback apparmor installs, but also I think its fine if we don't18:05
@clarkb:matrix.org * we might be able to eventually rollback apparmor installs, but also I think its fine if we don't18:06
@clarkb:matrix.orgOnce I'm happy with the DNM sanity check results I guess I should review ianw's linterfixes18:09
@clarkb:matrix.orgthe buildset registry job is paused now. It wasn't able to get that far before18:12
@clarkb:matrix.orgok all jobs that failed due to base-jobs use of ensure-docker have passed now. mhu you should be good to recheck things18:26
@clarkb:matrix.orgianw: I've reviewed your stack. One change converts shell to command when using pipes so I -1'd it and then another updates the docker image tag value and I had a questions about the pipeline version of that. And I rechecked a few changes to get results back after fixing docker 23 stuff18:44
@clarkb:matrix.orgianw: hrm it looks like we're hitting the unsupported mime type issue in some of those jobs. I'm guessing we need to install an up to date skopeo in those contexts too (I think because we're emulating the executor in those tests to shuffle data around?)19:15
@iwienand:matrix.orgClark: thanks, i'll loop back on it soon20:11
-@gerrit:opendev.org- Ian Wienand proposed:20:31
- [zuul/zuul-jobs] 872490: ansible-lint: fix a bunch of command-instead-of-shell errors https://review.opendev.org/c/zuul/zuul-jobs/+/872490
- [zuul/zuul-jobs] 872491: ansible-lint: add names to blocks/includes, etc. https://review.opendev.org/c/zuul/zuul-jobs/+/872491
- [zuul/zuul-jobs] 872492: ansible-lint: ignore use of mkdir https://review.opendev.org/c/zuul/zuul-jobs/+/872492
- [zuul/zuul-jobs] 872493: ansible-lint: use pipefail https://review.opendev.org/c/zuul/zuul-jobs/+/872493
- [zuul/zuul-jobs] 872494: ansible-lint: ignore latest git pull https://review.opendev.org/c/zuul/zuul-jobs/+/872494
- [zuul/zuul-jobs] 872495: ansible-lint: uncap https://review.opendev.org/c/zuul/zuul-jobs/+/872495
- [zuul/zuul-jobs] 872365: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365
- [zuul/zuul-jobs] 872258: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258
- [zuul/zuul-jobs] 872375: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375
@jkt_:matrix.orgso I just got biten in my rear by a typo in my commit message, I said `Depend-on` instead of `Depends-on`. What would be the easiest way of writing a super-simple job which lints the commit message? I guess I still need a VM for that, and that I cannot "just" run something on the executor, right?22:27
@jkt_:matrix.orgI don't feel like writing a gerrit plugin for this22:30
@clarkb:matrix.orgJan Kundrát: it should be doable on the executor22:33
@clarkb:matrix.organd the origin remote should be configured to show you the merged state and the normal local branches are the speculative state so you can look at commit messages between those two points22:33
@clarkb:matrix.orgOne thing to keep in mind generally when linting commit messages is that many users don't run linters after commiting. They run them first, things go green then commit. In this case its probably fine though since its about establishing relationships between changes22:34
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 872482: Fix race condition in pipeline change list init https://review.opendev.org/c/zuul/zuul/+/87248222:35
@clarkb:matrix.orgfungi: (or anyone else) I'm looking at startinga patch to kazoo to fix this SSLv23 deprecation warning. It looks like the things we are supposed to use is PROTOCOL_TLS_CLIENT but that enables hostname checking. We connect to our servers via IP addrs and I don't think our certs have IP addrs as altnames. Does this imply that python is basically saying you should not do ssl to something with ip addrs. Or I guess we must stick IP addrs in the altnames?22:53
@clarkb:matrix.orgIn particular kazoo is a library and if opendev's zuul has these concerns I want to be extra careful and not suddenly require everyone to change their certificate setups22:53
@clarkb:matrix.orgInstead of writing a small patch I might need to start with an issue to begin a conversation with upstream22:55
@clarkb:matrix.orgThe other zuul warning appears to come from the garbage collector warning about network connections not being explicitly closed. I'm not sure what to do about those yet as I think our manual running of the garbage collector to look for leaked resources trips this ahead of when we might close things? This one is definitely going to be a puzzle.22:56
@clarkb:matrix.orghrm one option for the TLS thing may be to explicitly disable hostname checking22:59
@clarkb:matrix.organd make a note that maybe one day this should change22:59
@clarkb:matrix.orghrm any idea why our CI jobs get those extra warnings but when I run tests locally I don't get them?23:24
@clarkb:matrix.orgmaking it hard to determine if the patch I've written to kazoo helps or not23:24
@jim:acmegating.comClark: old local venv?23:27
@clarkb:matrix.orgno I ran with `nox -N`23:27
@clarkb:matrix.orgI'm wondering if it has to do with how python is configured on the systems23:28
@clarkb:matrix.orgschool run time. I think this shifted from fixing kazoo to understanding why we get warnings in CI better23:29
@clarkb:matrix.orgwill have to dig more later23:29
@jim:acmegating.comClark: you talking about `DeprecationWarning: ssl.PROTOCOL_TLS is deprecated`?23:29
@clarkb:matrix.orgyes23:30
@jim:acmegating.com(if so, i get those locally on ubuntu)23:30
@clarkb:matrix.orghttps://paste.opendev.org/show/b9u5dSEucSKT5ef5Ngpc/ is the patch I'm working with against kazoo 2.9.023:30
@clarkb:matrix.orgya so maybe its a suse python vs ubuntu python thing23:30
@iwienand:matrix.orgcorvus: would you have a sec for the bottom 5 changes of https://review.opendev.org/q/topic:ansible-lint-req-2.12 ... the rest are going to need some squashing and reordering to deal with the registry/docker/to be debugged changes on top, but the bottom ones are ok i think so it would help my rebasing sanity23:37
@jim:acmegating.comClark: that patch appears to resolve the problem in my environment23:38
@jim:acmegating.comianw: does "bottom" mean "first to land"?23:39
@iwienand:matrix.orgcorvus: ahh yep; https://review.opendev.org/c/zuul/zuul-jobs/+/872371 23:39
@iwienand:matrix.orgup, or down, until https://review.opendev.org/c/zuul/zuul-jobs/+/872492/323:41
@iwienand:matrix.orgi guess depending if you're antipodean or not :)23:41
@jim:acmegating.comianw: i knew there was a joke like that in there somewhere. ;)  i just opened the stack in gertty and it puts first-to-land at the top and i had a moment of ...dizzyness?  :)23:43

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!