| frickler | infra-root: nb04 shouldn't have been affected by the new nb stuff, or should it? it did send a cron mail with "docker-compose: not found" | 07:56 |
|---|---|---|
| frickler | nb05 is also not completely happy: time="2025-03-18T09:19:02Z" level=warning msg="/etc/nodepool-builder-compose/docker-compose.yaml: `version` is obsolete" | 09:24 |
| frickler | I still cannot see any indication of branch creation events having been missed at https://review.opendev.org/q/topic:%22create-2025.1%22+status:open . I wonder whether the issue we had was more about branch deletions. I guess we'll find out in a couple of weeks when we EOL 2023.2 ;) | 10:07 |
| Ajay | hi guys | 12:13 |
| Ajay | is this channel active ? | 12:14 |
| frickler | well not as active as some people would expect it to be, seemingly ... | 12:33 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Use explicit docker-compose path in cronjobs https://review.opendev.org/c/opendev/system-config/+/944867 | 13:21 |
| fungi | frickler: ^ that should address the cron email | 13:22 |
| zigo | I've been looking for hours on how to set vendor metadata to have VMs of our public cloud to use our Debian mirror rather than the cloudflare CDN, and I wonder if you guys know how ... (I know this is not a support channel here, just asking an advice as a friend... ) | 13:26 |
| opendevreview | Karolina Kula proposed opendev/glean master: WIP: Add support for CentOS 10 keyfiles https://review.opendev.org/c/opendev/glean/+/941672 | 13:27 |
| frickler | zigo: I think usually you would just bake the mirror selection into custom images that you offer your customers. I don't know how you would achieve this with generic upstream images | 13:28 |
| frickler | (that's kind of what we do for our CI) | 13:28 |
| zigo | frickler: I'm quite sure it's possible to do that through the cloud-init APT plugin and vendor metadata. | 13:28 |
| zigo | I've found already what to put in the metadata. | 13:29 |
| zigo | I just didn't find out how to actually *SET* the vendor metadata ... | 13:29 |
| zigo | The doc tells about a .json file, however, cloud-init takes a yaml as input. | 13:29 |
| fungi | fwiw, we do just bake a preferred sources list into our own images | 13:30 |
| fungi | but obviously we're not using distro-supplied images | 13:31 |
| fungi | because we want to include git repository caches and other things in ours to speed up jobs | 13:31 |
| frickler | zigo: setting the vendordata is a deployer thing, which we are not. maybe amorin or mnaser know more. or people in the nova channel | 13:33 |
| fungi | https://docs.cloud-init.io/en/latest/reference/yaml_examples/apt.html does talk about how to set it with yaml, e.g. with a userdata boot option | 13:33 |
| zigo | Thanks guys. | 13:34 |
| zigo | I'll ask Arnaud or Mohamed. | 13:34 |
| fungi | looks like it talks about how amazon is setting it in ec2, could possibly reverse-engineer from an ec2 metadata dump | 13:34 |
| amorin | I dont think we provide any vendor data for this in our situation. And I don't think we change the image neither, so I believe it goes to the official debian cdn just like you zigo | 13:43 |
| zigo | Oh ok. | 13:44 |
| amorin | IIRC the only thing we do is that we install qemu-guest-agent, and we put that inside the image with a manifest in /etc/cloud to let the customer know about this | 13:46 |
| zigo | We found that qemu-guest-agent is too intrusive for linux, and decided to push it only in our Windows image since we build these, when we prefer to keep Linux images pristine. Maybe installing qemu-guest-agent at boot time with the vendor metadata would also be the way. | 13:48 |
| jrosser | i have some distant memory that ubuntu using nova.clouds.archive.ubuntu.com in its cloud images was a mechanism to somehow allow the operator to make that be a local mirror without messing with the image | 13:49 |
| jrosser | but i cannot find any reference to that just now | 13:50 |
| jrosser | i have also some memory that the first part of that is the name of the openstack compute az | 13:51 |
| Clark[m] | fungi: frickler since docker-compose is going away (it's just a shim we add to pass to docker compose on noble) I don't think we need to worry about relying on PATH in the cron job. There isn't a future where docker-compose is a thing we will get from the distro. | 13:58 |
| fungi | k | 13:59 |
| Clark[m] | Instead, we're more likely to switch to docker compose which isn't in /usr/local/bin and a non issue for cron paths | 13:59 |
| mnaser | zigo: https://bugs.launchpad.net/cloud-init/+bug/1837530 might give you some context. I believe you need a json with cloud-init key defined. And I think it needs to be set on all computes too if you want it to work with config drive | 14:23 |
| opendevreview | Karolina Kula proposed opendev/glean master: WIP: Add support for CentOS 10 keyfiles https://review.opendev.org/c/opendev/glean/+/941672 | 14:23 |
| clarkb | looking at ubuntu noble images in nodepool I think nodepool did clean up the records for the old image even though it was owned by either nb01 or nb02 | 14:34 |
| clarkb | as mentioend that won't clean things up on disk on those servers since the builder process isn't running but that is ok as we'll clean up nb01 and nb02 compeltely at some point in the near future | 14:35 |
| clarkb | zigo: it isn't clear if the problem is having VMs respect the vendor data or if you are having trouble configuring nova to set the vendor data in the first place. In case the problem is having nova provide the vendor data I think https://docs.openstack.org/nova/latest/admin/vendordata.html has some pointers | 14:39 |
| JayF | clarkb: fungi: https://www.youtube.com/watch?v=n32_afoiBJA your episode of GROSS-OUT just went live | 14:40 |
| clarkb | exciting | 14:41 |
| fungi | thanks! it was fun, looking forward to doing it again | 14:47 |
| opendevreview | Clark Boylan proposed opendev/lodgeit master: Run lodgeit with granian instead of uwsgi https://review.opendev.org/c/opendev/lodgeit/+/944805 | 14:50 |
| clarkb | oh I just discovered granian has a factory flag that should simplify all of this | 14:56 |
| opendevreview | Clark Boylan proposed opendev/lodgeit master: Run lodgeit with granian instead of uwsgi https://review.opendev.org/c/opendev/lodgeit/+/944805 | 15:01 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: mirror-container-images: use skopeo to mirror multiarch images https://review.opendev.org/c/zuul/zuul-jobs/+/944878 | 15:13 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: mirror-container-images: use skopeo to mirror multiarch images https://review.opendev.org/c/zuul/zuul-jobs/+/944878 | 15:22 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run lodgeit with granian https://review.opendev.org/c/opendev/system-config/+/944806 | 15:26 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: mirror-container-images: use skopeo to mirror multiarch images https://review.opendev.org/c/zuul/zuul-jobs/+/944878 | 15:37 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: mirror-container-images: use skopeo to mirror multiarch images https://review.opendev.org/c/zuul/zuul-jobs/+/944878 | 15:56 |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: mirror-container-images: use skopeo to mirror multiarch images https://review.opendev.org/c/zuul/zuul-jobs/+/944878 | 16:03 |
| hspease | Hi everyone! The OpenInfra Foundation would like to highlight OpenDev for the March newsletter. I could use your help to develop the content for this. What I need from you all is 1-2 paragraphs for the newsletter blurb. If you would like to do something a bit longer form, we can do a full article that the newsletter links out to. I have put together an etherpad to start drafting in | 17:02 |
| hspease | https://etherpad.opendev.org/p/opendev_newsletter | 17:02 |
| hspease | The newsletter will go out on March 31, so if we can finalize the draft in the etherpad by March 26 that would be wonderful! | 17:02 |
| fungi | thanks hspease! it'll be fun to talk about some of the things we've been doing recently | 17:02 |
| hspease | Yay! | 17:03 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: docs: Switch a mailing list to default moderation https://review.opendev.org/c/opendev/system-config/+/944893 | 17:08 |
| fungi | JayF: just a heads up, i've gone ahead and done that ^ on openstack-discuss due to the recent spate of spam subscribers since the weekend; we can see if it's manageable, or try other things too if needed | 17:12 |
| fungi | i'm going to send a quick mail to the list about the change, though it should only impact us (the moderators) and any new subscribers | 17:13 |
| JayF | so basically new subs need their first post approved? | 17:13 |
| fungi | dansmith: ^ related to earlier conversation in #openstack-tc | 17:13 |
| dansmith | fungi: ++ | 17:14 |
| fungi | JayF: basically, yes. more like approve their first post and also switch the moderation setting for them, i don't think it can be done all at once in a single action, but will know more as we try it out | 17:14 |
| *** dmellado0755393733 is now known as dmellado075539373 | 17:14 | |
| dansmith | fungi: it can | 17:15 |
| fungi | oh, even better | 17:15 |
| dansmith | fungi: in the moderation thing if you click on the message itself, it will show you the text, and let you approve/reject and there's also a drop-down to "change this user's moderation flag to..." | 17:15 |
| dansmith | at least in my version of hyperkitty/postorious | 17:15 |
| fungi | perfect, i had overlooked that drop-down, but otherwise that's my basic workflow already | 17:16 |
| fungi | and we're on the latest versions, so presumably works like yours | 17:16 |
| dansmith | yeah, I wish I could constrain the box to only have a few options because there are things in there I would never want to do (like mark a user as "skips all other checks and allow immediately" etc | 17:16 |
| dansmith | but it's better than a multi-step | 17:17 |
| fungi | yes, i included a comment about the "allow (skip other processing)" option in the above docs addition, since it seeems like a bit of a foot gun | 17:17 |
| dansmith | fo'shizzle | 17:17 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: docs: Switch a mailing list to default moderation https://review.opendev.org/c/opendev/system-config/+/944893 | 17:23 |
| dansmith | fungi: btw, are you as disappointed with mm3 as I am? | 17:24 |
| fungi | not terribly. i like it better than sticking with mm2 anyway | 17:24 |
| dansmith | I nursed my mm2 along for way too long and was expecting/hoping to gain a lot of features in mm3 that were always lacking, yet it seems less feature-rich and more like a bag of parts | 17:25 |
| tonyb | hspease: cool beans I'll take a stab too! | 17:25 |
| dansmith | sticking with mm2 isn't really an option AFAICT, but I was just super disappointed | 17:25 |
| fungi | yes, it does seem like the different components were designed by different groups of people | 17:25 |
| fungi | and i'm not a huge django fan | 17:25 |
| dansmith | yeah | 17:25 |
| dansmith | mine (the web side) also goes off the rails every once in a while and sends loadavg to 24 until I kill it | 17:26 |
| fungi | and too much of what i would think of as application admin functionality is still punted down a layer to django (though it's getting incrementally better) | 17:26 |
| dansmith | resource controls on docker and a healthcheck has automated that, but it's pretty stupid | 17:26 |
| dansmith | yeah, that's insanity.. "oh you want to admin something? here's the python api" | 17:26 |
| dansmith | like, it's nice that it's there but... no knobs for some basic stuff is pretty crazy | 17:27 |
| fungi | i don't even entirely mind the python repl like mailman client interactive shell, so much as having to do some things through django itself (for example, accounts are handled as django native constructs and only partly exposed by the mailman services) | 17:28 |
| dansmith | hyperkitty also doesn't (apparently) show web-posters that their message is moderated, so I get a lot of someone posting the same thing eight times overnight until they realize maybe it's something else, so I have to approve one and delete 7 each time | 17:28 |
| fungi | granted, i see people do th same thing when posting by e-mail, they miss or ignore the moderation hold notification and just send the same thing again, and again... | 17:29 |
| dansmith | the repl shell just seems obscure and foot-gun-esque to me.. it has been okay, and I like that I can script it when I need, but it's not very "make a quick change from my phone while traveling" friendly | 17:29 |
| clarkb | its definitely obscure | 17:29 |
| dansmith | sure, it just seems way worse if the person is using the web forum view and it just never shows up at all | 17:29 |
| clarkb | they don't even really show you how to invoke it iirc. Its just heres a thing | 17:29 |
| fungi | agreed, it does feel more like that's a backstop for features that haven't been added in the ui yet | 17:30 |
| dansmith | clarkb: yeah, it's pretty disappointing | 17:30 |
| fungi | but, again, mm2 was also that way to a great extent | 17:30 |
| dansmith | I keep thinking "surely there's some way to do this" or even "there must be a better example of how to do this" | 17:30 |
| fungi | there was the cli for some things, but anything complicated meant writing a python script (and dealing with pickle files) | 17:31 |
| dansmith | fungi: to some extent yes, but not as much and the CLI tools at least had better help IME | 17:31 |
| fungi | so it's more like that simply hasn't improved in mm3 | 17:31 |
| dansmith | well, I never had to do those complicated things with mm2, | 17:31 |
| dansmith | but I have in mm3 already which means it's a regression | 17:31 |
| dansmith | anyway, sounds like I'm not missing some major UI improvement that everyone else has, which is comforting at least :) | 17:32 |
| clarkb | fungi: I posted a question on your change | 17:33 |
| fungi | dansmith: not really, though i've found the mailman-users mailing list extremely helpful, and the maintainers generally willing to add small improvements on request | 17:34 |
| dansmith | well, that's good.. I usually just resort to swearing instead of productive, constructive activities like that :) | 17:44 |
| fungi | the next version is going to have some improvements i requested, like ways to not include the bodies of messages in moderator notifications and verp probes, because when people try to post spam or mailservers bounce back posts as suspected spam sending new copies just makes matters worse | 17:46 |
| dansmith | ah yeah, I get that sometimes for sure, although I do a lot of moderation by seeing those posts send to me | 17:47 |
| dansmith | maybe they should render the message to a PNG and email that :P | 17:47 |
| dansmith | (email that to the admin I mean) | 17:49 |
| fungi | right, it's a catch-22. knowing that there's a legitimate post waiting does sometimes prompt me to visit the moderator panel earlier than i otherwise would, but we have a lot of lists whose moderators are using e.g. gmail accounts and it ends up causing gmail to consider our list server a spam source when we're constantly sending them copies of the spam caught in moderation | 17:52 |
| dansmith | yeah | 17:55 |
| dansmith | I'm telling you.. render to PNG.. :) | 17:55 |
| dansmith | I guess maybe gmail would OCR them.. sounds like something they'd do | 17:55 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: docs: Switch a mailing list to default moderation https://review.opendev.org/c/opendev/system-config/+/944893 | 17:56 |
| fungi | clarkb: ^ amended the text to clarify based on your question | 17:57 |
| fungi | see if that helps | 17:57 |
| clarkb | fungi: yes thanks | 17:57 |
| fungi | dansmith: could have a feature to encrypt the messages with e.g. openpgp or s/mime | 17:59 |
| dansmith | fungi: now you're talkin' | 18:02 |
| clarkb | fungi: I'm wondering if we shoudl go ahead and approve your doc update and the cronjob fix for nodepool builders | 18:02 |
| opendevreview | Sylvain Bauza proposed opendev/irc-meetings master: Add René as nova meeting chair https://review.opendev.org/c/opendev/irc-meetings/+/944924 | 18:03 |
| fungi | clarkb: i mean, they're my changes, so obviously i'm a fan of having them merged, no argument from me ;) | 18:04 |
| frickler | didn't we have a size limit on ML messages? https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/HXDJYFT4HEXHNQQIKL4BP7KPDRZYZOFR/attachment/2/l3-agent.log-20250315 is almost 1M encoded | 18:05 |
| opendevreview | Karolina Kula proposed opendev/glean master: WIP: Add support for CentOS 10 keyfiles https://review.opendev.org/c/opendev/glean/+/941672 | 18:05 |
| fungi | frickler: yes, that message was held in moderation, i approved it hesitantly rather than try to get the sender to shrink their attached log | 18:06 |
| clarkb | If you look at the message headers I believe that mailman records when moderators approve things | 18:07 |
| fungi | though if they make a habit of it, i probably would | 18:07 |
| fungi | yes, there are special headers indicating it was held for moderation and approved | 18:07 |
| frickler | ah, yes, "X-Mailman-Rule-Hits: max-size" also | 18:08 |
| fungi | cool, that's also useful to be able to see in there | 18:09 |
| opendevreview | Merged opendev/irc-meetings master: Add René as nova meeting chair https://review.opendev.org/c/opendev/irc-meetings/+/944924 | 18:17 |
| clarkb | I think the new builders have built eveything but bookworm, centos-9-stream, gentoo, and openeuler. Gentoo and openeuler are not building anyway due to problems | 18:51 |
| clarkb | I'm going to manually trigger builds for bookworm and centos-9-stream so that we can get feedback on whether those work | 18:51 |
| clarkb | and maybe that is a good indicator we should consider removing those images until interested parties can resurrect them? | 18:54 |
| cardoe | So I'm trying to make heads or tails of the OpenStack Helm image building which is in the loci repo. It's using the buildset-registry (the only project to do so I think?) but then it seems to run its own registry again for creating a container of requirements wheels and pushing it there. Is the buildset-registry not shared between nodes? | 19:27 |
| cardoe | I'm also looking to change maybe to a mount=cache instead of having a container with wheels running nginx being necessary. Is mount=cache supported in container building on zuul? | 19:28 |
| clarkb | cardoe: the buildset registry is intended to be shared between all running builds within the buildset | 19:28 |
| tonyb | cardoe: We (OpenDev sysadmins) use the buildset-registry in a lot of jobs. You should trivially be able to share it between nodes in a given job/build | 19:30 |
| tonyb | cardoe: I don't know about mount=cache or where you'd like to use it but a node deployed as part of a job should be able to run podman/docker with any args you like | 19:31 |
| cardoe | got an example somewhere? I just want something else to compare against | 19:32 |
| clarkb | look int system-config/zuul.d/system-config-run.yaml | 19:32 |
| clarkb | re mount=cache you may also want to look at system-config/docker/python-builder and system-config/docker/python-base | 19:32 |
| cardoe | opendev.org/opendev/system-config ? | 19:32 |
| clarkb | we use multistage builds with a builder to produce all the wheels for a thing then those get copied to python-base based images and installed when actually installing the thing | 19:32 |
| clarkb | avoids having a bunch of extra dependencies and build artifacts laying around | 19:33 |
| clarkb | cardoe: yes | 19:33 |
| cardoe | Yeah that's how I build mine. | 19:33 |
| cardoe | I'm just trying to negotiate upstreaming what I've got. | 19:33 |
| cardoe | I've noticed that even vexxhost builds it themselves like https://github.com/vexxhost/atmosphere/blob/main/images/ironic/Dockerfile | 19:34 |
| cardoe | All part of my crazy dream of uniting downstreams into upstream. | 19:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Rebuild our base python container images https://review.opendev.org/c/opendev/system-config/+/944789 | 20:00 |
| clarkb | now with the uwsgi pin | 20:01 |
| clarkb | cardoe: ^ that change is actually going to exercise a lot of what you are talking about | 20:06 |
| cardoe | perfect thank you | 20:06 |
| clarkb | cardoe: you'll notice there is a buildset registry job running. Then we'll build python-base and python-builder images and upload them to that registry. Then the uwsgi-base jobs will start and fetch the python-base image from the buildset registry as uwsgi-base is built on python-base | 20:06 |
| clarkb | when the jobs run in the gate they will also upload to docker hub. Then when the change merges jobs will run to promote those images by retagging them in docker hub to go from change specific tags to their production tag values | 20:07 |
| clarkb | cardoe: https://review.opendev.org/c/opendev/system-config/+/944806 and its depends on https://review.opendev.org/c/opendev/lodgeit/+/944805 may be a better example | 20:11 |
| clarkb | as that also shows using containers across buildsets using the intermediate registry | 20:11 |
| clarkb | centos 9 stream and bookworm images have successfully built on the new builders | 20:13 |
| clarkb | I also approved the cronjob fix | 20:13 |
| opendevreview | Merged opendev/system-config master: Use explicit docker-compose path in cronjobs https://review.opendev.org/c/opendev/system-config/+/944867 | 20:21 |
| clarkb | that changes appears to have applied on nb05 at least | 20:39 |
| clarkb | there is a gap in the weather. I'm going to pop out and get a bike ride in while I can | 20:39 |
| clarkb | I should be back well before gerrit meets | 20:39 |
| fungi | have fun! | 20:44 |
| corvus | cardoe: there's documentation (with pictures!) about what all the different registries and jobs and roles do here: https://docs.opendev.org/opendev/base-jobs/latest/docker-image.html | 21:00 |
| corvus | cardoe: technically, that's an implementation of the system described here: https://zuul-ci.org/docs/zuul-jobs/latest/docker-image.html (this is a more general documentation; the other one is opendev-specific and should be preferred since it's more relevant) | 21:01 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add upload-image-s3 role https://review.opendev.org/c/zuul/zuul-jobs/+/944813 | 22:28 |
| clarkb | https://zuul.opendev.org/t/openstack/build/f932b64a9e18445d8faebd1aae53f31c/log/job-output.txt#1933-1966 we are still failing the uwsgi build this time in amd64 too | 22:43 |
| clarkb | oh maybe only amd64 the other build seems to be cancelled when this one fails | 22:43 |
| clarkb | https://github.com/unbit/uwsgi/issues/2630 I think this may be the issue | 22:46 |
| clarkb | there is a 2.0.25.1 that apparently should address this | 22:47 |
| clarkb | I'm going to see about adding pcre support but that doesn't seem critical | 22:52 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Rebuild our base python container images https://review.opendev.org/c/opendev/system-config/+/944789 | 22:57 |
| clarkb | something like that maybe | 22:57 |
| clarkb | nb06's cronjob will run at 00:01 I think | 23:10 |
| clarkb | though I may be migrated to the couch to watch gerrit meets by then | 23:10 |
| clarkb | the debian libpcre2 packages are weird | 23:28 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Rebuild our base python container images https://review.opendev.org/c/opendev/system-config/+/944789 | 23:28 |
| clarkb | they split out 8bit, 16 bit and 32 bit libraries | 23:28 |
| clarkb | fungi: is the openstack-discuss email about FIP mac addrs being incomplete something that went through the new moderation setup? | 23:37 |
| clarkb | it says rule miss: approved so not sure if that maens you have to approve it manually or not | 23:38 |
| cardoe | Does that uWSGI include some of the latest commits? | 23:40 |
| clarkb | cardoe: no we are pinning to an older version because the latest versions don't build on aarch64 | 23:41 |
| clarkb | and it installs from the sdists on pypi. No special patching on our end | 23:41 |
| clarkb | hrm 2.0.25.1 segfaults. I can try 2.0.25 with the pcre libs in place and maybe that will work? | 23:52 |
| clarkb | still wow | 23:52 |
| cardoe | I gotta find the patch but I linked it on the ML about OpenStack-helm. They aren’t gracefully stopping so it goes south on k8s. | 23:53 |
| cardoe | I’m just wanting to simplify how OpenStack services are containerized at the end of the day. | 23:54 |
| clarkb | in this case we're using uwsgi with paste/lodeit. And I've got chagnes up to switch that over to granian | 23:55 |
| clarkb | but I want to get these new images built first if I can then we can do the move away from uwsgi more carefully rather than rushing | 23:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Rebuild our base python container images https://review.opendev.org/c/opendev/system-config/+/944789 | 23:57 |
| clarkb | if that doesn't work I'll try 2.0.24 | 23:57 |
| cardoe | I don’t even know granian. I wanna move from uWSGI too. | 23:57 |
| cardoe | Projects are still moving to it which is surprising. | 23:57 |
| clarkb | granian is a rust server that supports asgi, wsgi, and their own thing called rsgi | 23:58 |
| clarkb | I brought it up in #openstack-tc earlier and one of the concerns is it basically has one primary developer | 23:58 |
| clarkb | and it isn't pacakged for distros | 23:58 |
| clarkb | but the upside for me is they publish wheels for x86_64 and aarch64 so we don't have to build anything when pip installing | 23:58 |
| clarkb | and it does both asgi and wsgi so in theory you don't need multiple tools as things become more async aware | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!