Wednesday, 2021-03-24

« Tuesday, 2021-03-23 Index Thursday, 2021-03-25 »
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134000:39
*** zhanglong has joined #openstack-kolla00:41
*** schwicht has joined #openstack-kolla01:08
*** rouk has quit IRC01:23
*** LinPeiWen has joined #openstack-kolla02:06
*** ivan_lin has quit IRC02:09
*** LinPeiWen2 has joined #openstack-kolla02:19
*** zhanglong has quit IRC02:48
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134003:06
*** zhanglong has joined #openstack-kolla03:20
*** zhanglong has quit IRC03:46
*** skramaja has joined #openstack-kolla03:55
*** vishalmanchanda has joined #openstack-kolla04:41
*** zhanglong has joined #openstack-kolla05:36
*** wuchunyang has joined #openstack-kolla05:36
*** Luzi has joined #openstack-kolla05:40
*** iniazi_ has joined #openstack-kolla06:25
*** iniazi has quit IRC06:27
mnasiadkamorning07:21
*** shyamb has joined #openstack-kolla07:22
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Bump max Ansible version to 2.10  https://review.opendev.org/c/openstack/kayobe/+/78089807:25
*** shyamb has quit IRC07:28
*** shyamb has joined #openstack-kolla07:29
*** parallax has quit IRC07:29
*** ohorecny2 has joined #openstack-kolla07:31
*** zhanglong has quit IRC07:32
*** schwicht has quit IRC07:39
*** shyam89 has joined #openstack-kolla07:44
*** ysirndjuro has quit IRC07:46
*** shyamb has quit IRC07:47
*** shyam89 has quit IRC07:49
*** luksky has joined #openstack-kolla07:55
hrwmorning07:59
hrw ERROR nova.compute.manager [instance: 1529def5-4708-4436-b3e8-f2059e5ad435] TypeError: can't concat str to bytes08:00
hrwgetting closer to working openstack08:00
*** bengates has joined #openstack-kolla08:02
*** bengates has quit IRC08:06
*** bengates has joined #openstack-kolla08:06
*** bengates has quit IRC08:07
*** bengates has joined #openstack-kolla08:07
mnasiadkahrw: it's a never ending story ;)08:09
*** amoralej|off is now known as amoralej08:11
*** andrewbonney has joined #openstack-kolla08:12
*** bengates_ has joined #openstack-kolla08:15
*** bengates has quit IRC08:18
*** rpittau|afk is now known as rpittau08:19
*** ricolin has quit IRC08:27
hrwyeah08:27
hrwtoo bad that I found time for it so late in cycle08:28
*** ricolin has joined #openstack-kolla08:32
*** ChenSa has joined #openstack-kolla08:38
openstackgerritMark Goddard proposed openstack/kayobe master: CI: use retries for control host bootstrap in seed VM jobs  https://review.opendev.org/c/openstack/kayobe/+/78266808:50
openstackgerritMerged openstack/kayobe master: CI: Fix IP address detection in baremetal compute test  https://review.opendev.org/c/openstack/kayobe/+/78232408:54
hrwcan someone remind me how to switch install to devmode?08:56
hrwok. found08:58
*** vishalmanchanda has quit IRC09:01
mgoddardmorning09:03
hrwmgoddard: https://review.opendev.org/c/openstack/kolla/+/782619 https://review.opendev.org/c/openstack/kolla/+/782247 https://review.opendev.org/c/openstack/kolla/+/782606 are vital to get Debian running09:03
hrwmgoddard: and nova change we discuss on nova channel as cherry on top09:05
*** brinzhang has quit IRC09:07
*** gfidente has joined #openstack-kolla09:08
*** e0ne has joined #openstack-kolla09:19
*** e0ne has quit IRC09:22
*** e0ne has joined #openstack-kolla09:24
*** e0ne has quit IRC09:24
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: rabbitmq: bump Erlang requirements to v23 in Debian  https://review.opendev.org/c/openstack/kolla/+/78224709:28
hrwmgoddard: commented on other ones09:28
*** brinzhang has joined #openstack-kolla09:30
*** zhanglong has joined #openstack-kolla09:30
mnasiadkamaybe it's a stupid idea - but shouldn't we have debian-source jobs as voting?09:34
hrwmnasiadka: +2 in Xena09:39
hrwnow it is at: 2021-03-24 10:36:23.905 7 ERROR nova.compute.manager [instance: ef71de37-1a2b-4f60-9fb4-d5942ffe7fc4] libvirt.libvirtError: unable to open '/sys/fs/cgroup/machine/qemu-1-instance-00000002.libvirt-qemu/': No such file or directory09:40
*** shyamb has joined #openstack-kolla09:41
*** vishalmanchanda has joined #openstack-kolla09:42
*** parallax has joined #openstack-kolla09:52
*** zhanglong has quit IRC09:53
*** e0ne has joined #openstack-kolla09:56
*** strigazi has joined #openstack-kolla09:58
*** brinzhang_ has joined #openstack-kolla10:03
*** brinzhang has quit IRC10:06
*** e0ne has quit IRC10:09
ChenSahi, did anyone try enabling ovs-dpdk on the latest release of kolla?10:10
openstackgerritMark Goddard proposed openstack/kolla-ansible master: magnum: Add CA certificate configuration for internal TLS  https://review.opendev.org/c/openstack/kolla-ansible/+/78106210:13
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: CI: Enable TLS in all jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/78238710:17
openstackgerritMerged openstack/kolla-ansible master: prometheus: Collect metrics from rabbitmq  https://review.opendev.org/c/openstack/kolla-ansible/+/62833710:23
openstackgerritMark Goddard proposed openstack/kayobe master: Ubuntu: AppArmor config for seed VM provisioning  https://review.opendev.org/c/openstack/kayobe/+/78227210:41
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227310:42
*** shyamb has quit IRC10:46
*** shyamb has joined #openstack-kolla10:55
openstackgerritMark Goddard proposed openstack/kayobe master: DNM: Test IPA build with Ansible 2.10  https://review.opendev.org/c/openstack/kayobe/+/78268910:58
*** wuchunyang has quit IRC10:59
*** brinzhang0 has joined #openstack-kolla11:08
*** brinzhang_ has quit IRC11:12
openstackgerritMerged openstack/kolla-ansible stable/victoria: nova-cell: Stop printing ceph keys in output  https://review.opendev.org/c/openstack/kolla-ansible/+/78211811:19
*** ohorecny2 has quit IRC11:20
openstackgerritMerged openstack/kolla-ansible stable/victoria: don't use the same CIDR in octavia_amp_network_cidr and init-run-once  https://review.opendev.org/c/openstack/kolla-ansible/+/78119911:25
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227311:33
ChenSahi, did anyone try enabling ovs-dpdk on the latest release of kolla?11:36
mgoddardChenSa: I think wuchunyang uses it, but they are not here right now11:36
*** shyamb has quit IRC11:37
*** shyamb has joined #openstack-kolla12:23
openstackgerritMark Goddard proposed openstack/kayobe stable/victoria: Support configuring docker_registry network_mode  https://review.opendev.org/c/openstack/kayobe/+/78270312:26
*** sean-k-mooney has joined #openstack-kolla12:43
*** skramaja has quit IRC12:59
*** LinPeiWen2 has quit IRC12:59
*** shyamb has quit IRC13:02
*** ChenSa has quit IRC13:07
*** bengates_ has quit IRC13:09
*** brinzhang0 has quit IRC13:09
*** bengates has joined #openstack-kolla13:10
*** amoralej is now known as amoralej|lunch13:12
*** bengates has quit IRC13:13
*** bengates has joined #openstack-kolla13:14
*** e0ne has joined #openstack-kolla13:18
*** devfaz has quit IRC13:21
*** bengates has quit IRC13:22
*** bengates has joined #openstack-kolla13:23
*** devfaz has joined #openstack-kolla13:24
*** ysirndjuro has joined #openstack-kolla13:37
yoctozeptowth with these rc -13 in CI13:48
yoctozeptomnasiadka: cephadm failing on ubuntu often13:49
yoctozepto2021-03-24 09:46:14.086790 | primary | FileExistsError: [Errno 17] File exists: '/var/log/ceph/42e24910-8c85-11eb-8add-1bdc6ebc3286'13:49
kevkomgoddard: do you have a time for quick look mariadb review ?14:01
mgoddardkevko: if using haproxy, are multiple shards supported, and if so, what happens with shards other than the default?14:03
openstackgerritMark Goddard proposed openstack/kayobe master: Import merge_configs and merge_yaml from Kolla Ansible  https://review.opendev.org/c/openstack/kayobe/+/77899414:04
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: Use merge_configs and merge_yaml to generate custom config  https://review.opendev.org/c/openstack/kayobe/+/78274914:04
kevkomgoddard: yes, they are supported, but other shards has no haproxy VIP:3006, only deployed14:04
mgoddardkevko: ok, thanks for clarifying behaviour14:04
kevkomgoddard: I think this is how it should work ...14:04
kevkomgoddard: if user want to proxy all shards, he just turn-on proxysql ..14:05
mgoddardkevko: is there any special meaning to mariadb_default_database_shard_id when proxysql is usd?14:05
kevkomgoddard: yes, from ID number , internal groups of proxysql is generated14:06
kevkomgoddard: can't be undefined14:06
mgoddardkevko: right, but the 'default' shard behaves the same as others?14:06
kevkomgoddard: writer,backup_writer,reader,offline =  id*10, id*10+1,etc..14:06
mgoddardi.e., it is only really 'special' when using haproxy?14:06
kevkomgoddard: sorry, don't understand the question14:07
mgoddardkevko: ok, in haproxy mode, the default shard is the one that gets load balanced14:07
kevkomgoddard: yes14:08
mgoddardin proxysql mode, it is just the same as any other shard, right?14:08
kevkomgoddard: yes14:08
mgoddardok14:08
*** suff has joined #openstack-kolla14:08
kevkomgoddard: well, I've implemented your request to not create root_shard_{{ default_shard_id }} ..14:09
mgoddardkevko: so I think that mariadb_default_database_shard_id is conflating two things14:09
mgoddard1. which type of load balancer am I using14:09
kevkomgoddard: so you can login to mariadb default shard via proxysql by root user14:09
mgoddard2. which shard will be load balanced when using haproxy14:09
mgoddardkevko: ok, all that sounds like more of an accident than a useful feature? :)14:10
mgoddards/all/although/14:10
kevkomgoddard: what ? why ?14:10
mgoddardmaybe I'm wrong. Why is it useful?14:10
kevkomgoddard: what do you mean specifically ?14:11
kevkomgoddard: like feature - that it is able to have several clusters behind proxysql ?14:11
mgoddardsorry, I mean that root user can access the default shard14:11
*** amoralej|lunch is now known as amoralej14:12
kevkomgoddard: well, it's like it is now ...you *are* able to login via mysql -h VIP -u root -pSECRET also via haproxy14:12
kevko*via haproxy14:12
mgoddardkevko: WDYT to this: http://paste.openstack.org/show/803866/14:13
kevkobut this is about proxysql templating only .. in mariadb refactor it is only about one thing -> If creating root_shard_ID or only root14:13
kevkomgoddard: not needed I think -> you can set mariadb_default_database_shard in globals to 0 or to 1 or whatewer ... and in inventory just set this also to hosts ..14:15
kevkomgoddard: and you have what you wanted ..14:15
mgoddardkevko: perhaps this makes the intention clearer: http://paste.openstack.org/show/803867/14:17
hrwelo14:18
kevkomgoddard: well, could be .. why not .. but i think it's quite cosmetic14:20
mgoddardkevko: having only a single root user is not cosmetic14:21
kevkomgoddard: ah, now I understand maybe ...14:21
kevkomgoddard: on new deployments where you have proxysql turned off, shards can be deployed ..but will be deployed only with root user14:21
mgoddardright14:21
kevkomgoddard: if you switch to proxysql and reconfigure .. proxysql is configured first (order in site.yml) and in mariadb role which is running right after ..users are created extra14:22
mgoddardthere are also a few places where the code tries to infer whether it is using haproxy or proxysql. Having mariadb_loadbalancer can make those conditions nice and explicit14:22
kevko^^ am i right ?14:23
kevkothis was your point ..14:23
mgoddardkevko: I don't think that was my point :)14:24
mgoddardkevko: my point was that if we have haproxy with multiple shards, all shards should use the root user14:25
kevkomgoddard: yes14:25
kevkomgoddard: but if you reconfigure ..you will have root_shard_ID14:26
mgoddardbut this is what the code looks like now:14:26
mgoddardmariadb_shard_database_user: "{% if mariadb_shard_id == mariadb_default_database_shard_id %}{{ database_user }}{% else %}root_{{ mariadb_shard_name }}{% endif %}"14:26
kevkoyeah, you are right ..14:26
mgoddardso only the default shard uses root14:26
kevkonow it creates root for default ..and others with root_shard_ID14:27
mgoddardright14:27
kevkomgoddard: understand,14:27
kevkomgoddard: ok, that's small change ..can edit14:27
kevkomgoddard: something else ? :P14:29
mgoddardjust looking at mariadb_default_database_shard_hosts14:30
kevkovariable where hosts from default shard is stored ..14:31
kevkoit is used in internal_haproxy_members14:31
mgoddardthat part always seems tricky14:31
mgoddarddoes it need to be in group_vars/all?14:32
kevkoyes, unfortunatelly yes14:32
kevkoit's because you can run kolla-ansible ..... -t mariadb14:32
kevkoor -t loadbalancer14:32
mgoddardshouldn't be an issue14:33
kevkoreally ?14:33
mgoddardsite.yml includes mariadb role, loadbalancer.yml14:33
mgoddardso it gets the mariadb defaults14:33
mgoddardthen loadbalancer.yml imports haproxy-config14:33
mgoddardthe fiddly part is that hostvars does not contain role default variables, unless they have been overridden by inventory or globals.yml14:34
kevkomgoddard: i've tested and i really had to place it in globals (before i was doing dirty set_fact if you remember ..this is nicer)14:36
*** suff has quit IRC14:36
mgoddardkevko: ok14:39
*** Luzi has quit IRC14:43
*** bengates has quit IRC14:46
*** bengates has joined #openstack-kolla14:47
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt14:51
mgoddard^ meeting in 1014:51
yoctozeptook14:51
*** rafaelweingartne has joined #openstack-kolla14:51
hrway14:57
kevko\o/14:59
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Mar 24 15:00:19 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
*** rafaelweingartne has quit IRC15:00
hrw]°[15:00
osmanlicilegio/15:00
headphoneJameso/15:01
mgoddard\o\ |o| /o/15:01
parallax\o15:01
yoctozeptoo_15:01
mgoddard#topic agenda15:02
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:02
mgoddard* Roll-call15:03
mgoddard* Announcements15:03
mgoddard** PTG 19th - 23rd April, registration open | https://april2021-ptg.eventbrite.com | https://www.openstack.org/ptg/15:03
mgoddard* Review action items from the last meeting15:03
mgoddard* CI status15:03
mgoddard* Review requests15:03
mgoddard* PTG team signup http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html15:03
mgoddard* Quay.io15:03
mgoddard* Wallaby release planning15:03
mgoddard#topic announcements15:03
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:03
mgoddard#info PTG 19th - 23rd April, registration open15:03
mgoddard#link https://april2021-ptg.eventbrite.com15:03
mgoddard#link https://www.openstack.org/ptg/15:03
mgoddard#link https://etherpad.opendev.org/p/kolla-xena-ptg15:04
mgoddardPlease add your name to ^ if you plan to attend15:04
mgoddardWe will discuss PTG more later15:04
*** jovial[m] has joined #openstack-kolla15:04
mgoddard#info Kolla feature freeze next week15:05
mgoddardAny other announcements?15:05
mgoddard#topic Review action items from the last meeting15:06
*** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)"15:06
mgoddardyoctozepto try out quay.io15:06
mgoddardhe most certainly did15:06
mgoddardwill discuss later15:06
mgoddard#topic CI status15:06
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:06
* hrw 15:07
mgoddardlots of nice notes on debian issues, thanks hrw15:07
hrwand we need all of them merged15:07
*** wuchunyang has joined #openstack-kolla15:08
mgoddardthis one makes x86 zuul pass: https://review.opendev.org/c/openstack/kolla/+/78260615:09
hrwthen would love to get some help on checking do things work15:09
yoctozeptoI've added a note on rc -1315:09
hrwmgoddard: so s/-1/+2/ ;D15:10
yoctozepto+2315:11
yoctozeptooops15:11
hrwI would need to check Debian in victoria, ussuri, train15:11
mgoddardyoctozepto: how often is ubuntu cephadm failing15:12
yoctozeptoquite often but not 100% I think15:12
yoctozeptolet's see the recent runs15:12
yoctozeptohttps://zuul.openstack.org/builds?job_name=kolla-ansible-ubuntu-source-cephadm&branch=master15:13
yoctozeptoseems not horribly often15:13
yoctozeptoor perhaps it was not always ubuntu15:13
yoctozeptolet me see15:13
yoctozeptoyup, bingo15:14
yoctozeptoit just I recognised ubuntu as happening more often15:14
yoctozeptocould be15:14
yoctozeptobut it's on both15:14
mgoddardok, sounds like more investigation needed15:15
yoctozeptoI wonder if it's not rc -13 in disguise15:15
yoctozeptoperhaps the block storage is unreliable or something15:15
yoctozeptoI think it would make sense to decrease the amount of retries15:16
openstackgerritPierre Riteau proposed openstack/kayobe master: [DNM] Set environment for testing  https://review.opendev.org/c/openstack/kayobe/+/77341115:16
yoctozeptoespecially for non-voting jobs15:16
yoctozeptoas we sometimes wait for several runs15:16
yoctozeptowonder what happens there15:16
mgoddardI don't think we're going to solve this here. Let's move on15:16
mgoddard#topic Review requests15:17
*** openstack changes topic to "Review requests (Meeting topic: kolla)"15:17
mgoddardYou know the drill. One review per person15:17
yoctozeptoI will be back on masakari15:17
yoctozeptonothing new today :-(15:17
hrwI do not have one. Debian needs 3 ;D15:18
hrwfound one: https://review.opendev.org/c/openstack/kolla/+/782386 - ussuri backport15:18
mgoddardI will choose https://review.opendev.org/c/openstack/kolla-ansible/+/78106215:19
hrwas we still use Ussuri and want upgrade to newer qemu to test SVE guests15:19
mnasiadkayoctozepto: https://review.opendev.org/c/openstack/kolla-ansible/+/761872 - that one is for you ;)15:19
headphoneJamesI have some Questions about test case requirements for Let's Encrypt15:19
headphoneJameshttps://review.opendev.org/c/openstack/kolla-ansible/+/74134015:20
mgoddardheadphoneJames: go for it15:20
headphoneJamesCurrently, I use "pebble" to create the TLS certificate. Then I distribute that TLS certificate to all haproxy15:21
yoctozeptomnasiadka: why me?15:21
mnasiadkayoctozepto: well, find another active core in k-a that does not work in StackHPC :)15:22
yoctozeptomnasiadka: wuchunyang15:22
mnasiadkais he on the meeting? nope15:22
headphoneJamesThe certificate is not valid, because pebble is a testing product. Therefore, all subsequent requests to the OpenStack deployment would need to ignore the insecure certificate.15:23
yoctozeptomnasiadka: ok15:23
mgoddardheadphoneJames: why isn't it valid? is there no CA certificate available for it?15:23
headphoneJamesSince I don't have access to the certificate authority for the certificate generated by pebble, I added a boolean to allow for insecure curl method executions to get around this for now.15:23
headphoneJamesThe valid CA cert is generated by pebble -I have not determined a way to pull that certificate out of pebble / docker volume distribute it to the executor that's running the test15:25
headphoneJamesNote, when I run this with let's encrypt proper (not functional test with pebble), the certificate generated is valid and trusted15:26
*** rafaelweingartne has joined #openstack-kolla15:27
headphoneJamesMy first question is just validating the logs for certbot (That a certificate was properly generated) and that the certificate is distributed to all HAProxy enough for a test case?15:28
yoctozeptowe should be able to get the CA cert from pebble15:28
yoctozeptomnasiadka: y no healthchecks?15:28
wuchunyangyes15:28
mnasiadkayoctozepto: where?15:28
yoctozeptomnasiadka: left a comment15:29
mgoddardheadphoneJames: there's not much point in running a full test suite with insecure mode. We need to either do a more targeted test of the letsencrypt code, or somehow get hold of the CA cert15:29
yoctozeptomnasiadka: on ovn-octavia15:29
*** stand has joined #openstack-kolla15:29
mgoddardheadphoneJames: e.g. using something like the openssl suite to grab the cert from haproxy and check that it has come from pebble15:30
mnasiadkayoctozepto: actually we can disable distributed FIP, but where do we communicate what CI does nowadays? :D15:30
yoctozeptomnasiadka: what about the commit message for the starters? :P15:30
mnasiadkacommit message sounds good :)15:31
mnasiadkawill update15:31
yoctozeptolet's do the distr fip15:31
headphoneJamesmgoddard: the openssl suite approach would be doable if that feels acceptable15:31
yoctozeptosounds fancier15:31
yoctozeptoand get those healthchecks in15:31
mgoddardheadphoneJames: that would be better than nothing15:32
mgoddardand probably necessary in any case to verify that the cert has been rotated15:33
mgoddardI think we've derailed a bit15:33
mgoddardLet's move on15:33
mgoddard#topic PTG team signup15:33
*** openstack changes topic to "PTG team signup (Meeting topic: kolla)"15:33
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html15:34
mgoddardTomorrow is the deadline to choose time slots15:34
mgoddardI didn't get any responses regarding using earlier slots15:34
mgoddardso I propose we stick to the usual plan15:34
mgoddard13:00-17:00 UTC on Monday and Tuesday for Kolla and Kolla Ansible15:35
mgoddard13:00-15:00 on Wednesday for Kayobe15:35
mgoddard#vote15:35
hrw+115:35
wuchunyang13:00-17:00 UTC is good for me15:35
yoctozepto+115:36
mnasiadka+115:36
mgoddarddone15:37
mgoddardplease add your names to https://etherpad.opendev.org/p/kolla-xena-ptg15:37
mgoddardplease also add discussion topics!15:38
hrw* deprecate 'base' image15:38
hrwops, wrong window15:38
yoctozeptolet's deprecate something :-)15:39
mgoddarddeprecate yoctozepto15:40
yoctozepto:-(15:40
hrwmgoddard: you want to be PTL for rest of your life?15:41
mgoddardof course15:41
hrw+115:41
yoctozepto<315:41
hrwuf. nova change which makes Debian work just got +w ;D15:41
hrwor rather s/work/fail in known place/15:42
yoctozeptogood enuff15:42
mgoddard#topic Quay.io15:42
*** openstack changes topic to "Quay.io (Meeting topic: kolla)"15:42
mgoddardyoctozepto has started a nice PoC of using Quay.io in CI15:43
mgoddard#link https://review.opendev.org/c/openstack/kolla/+/78113015:43
mgoddard#link https://review.opendev.org/c/openstack/kolla-ansible/+/78154615:43
yoctozeptothanks, I've pushed all master and victoria images there15:43
mgoddard#link https://review.opendev.org/c/openstack/kolla/+/78189915:43
yoctozeptoexcept for centos binary which was failing at the time15:43
yoctozeptobut it's no biggie15:44
mgoddardso I think we have two things to discuss15:44
mgoddard1. any concerns15:44
mgoddard2. plan15:44
mgoddardyoctozepto: any concerns?15:44
yoctozepto~> https://review.opendev.org/q/topic:%22quay.io%22+projects:openstack/kolla15:44
yoctozeptothere is one limitation15:44
yoctozeptoin that new repositories get pushed as private15:45
yoctozeptoquay.io is "actively investigating" how to improve this15:45
yoctozeptoI have a script that fixes it for all repos15:45
yoctozeptobut it has to be run with human user credentials15:45
yoctozeptootoh, we don't create new images that often15:46
yoctozeptoand quay.io might fix it sooner or later15:46
yoctozeptoother than that, I am quite happy with it15:46
yoctozepto(not to mention having total control over it now)15:46
yoctozepto(though I can share)15:46
yoctozeptoas for the plan15:47
yoctozeptoI would consider adding daily quay.io publishing jobs15:47
yoctozeptoleaving dockerhub ones in place to run their weekly sacred dance15:47
yoctozeptothen switching kolla-ansible to test from quay.io15:47
yoctozeptowe can run back to dockerhub if it proves worse15:48
yoctozepto;p15:48
mgoddardso we would keep publishing to dockerhub for the time being15:48
mgoddardit probably makes sense15:48
mgoddardless disruption to users15:49
mgoddardno need to clean up15:49
mgoddardalthough we would have no way to test the images15:49
mgoddardperhaps a weekly test pipeline15:49
hrwmgoddard: we build weekly and publish to dockerhub and quay.io in same job?15:49
mgoddardyoctozepto suggests publishing to quay.io daily15:50
yoctozeptoyes, quay.io more often15:50
yoctozeptoto get on track like we had it before ;p15:50
hrwdaily job on mon-sat to quay, weekly on sun to quay,docker?15:50
mgoddardit would probably be simpler to just have separate publishing jobs15:51
hrwsure15:51
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: [CI] Drop the workaround in Masakari client calls  https://review.opendev.org/c/openstack/kolla-ansible/+/77718215:51
mgoddardalthough potentially we could more easily test and promote to dockerhub15:51
yoctozeptowell, we can publish to dockerhub daily15:52
yoctozeptoit was not publishing that was broken15:52
yoctozeptoit was pulls15:52
yoctozepto(and is)15:52
mgoddardI think weekly is fine15:52
yoctozepto"is enough"15:52
hrwis both15:52
mgoddardand doesn't double our CI load15:53
*** markmcclain has joined #openstack-kolla15:53
yoctozeptowell, we can publish from the same jobs15:53
hrwwe may write in docs "please use quay" and keep dockerhub as source for those who still use it15:53
yoctozeptobut I guess we could timeout15:53
yoctozeptoand not know what to blame15:53
openstackgerritPierre Riteau proposed openstack/kayobe master: [DNM] Set environment for testing  https://review.opendev.org/c/openstack/kayobe/+/77341115:53
yoctozeptohrw: yes15:53
yoctozeptofor the time being15:53
yoctozeptolet's see15:53
mgoddardso we have a rough plan15:53
yoctozeptoI will enact it15:54
yoctozeptowill make me happy15:54
hrwwe can also deprecate dockerhub in Xena and do quay only in Yeti15:54
mgoddardwonderful15:54
mgoddardwhat about account credentials for quay.io15:54
mgoddardcurrently I don't think any of us have credentials for dockerub15:54
mgoddardwhich you might argue is a security feature15:55
yoctozeptoI can give you admin access15:55
yoctozeptoas you are PTL15:55
mgoddardthat's the opposite of what I'm suggesting :)15:55
yoctozeptoand I'm just a humble CI servant :D15:55
yoctozeptooh, someone has to have them15:55
yoctozeptosomeone from the previous team generated them for dockerhub15:56
mgoddardif any person has credentials, it would allow them to compromise the images15:56
yoctozeptothey did not appear magically15:56
yoctozeptomind you15:56
yoctozeptoit all boils down to trust15:57
mgoddardindeed15:57
yoctozeptoI can't give you a better answer15:57
yoctozeptoI trust myself15:57
yoctozeptoI trust the PTL15:57
mgoddardbut quite a lot of effort goes into trust in zuul15:57
mgoddardthis could effectively side step that15:57
yoctozeptowe can perhaps write something down who is moderating the images15:57
yoctozeptozuul has it encrypted15:58
yoctozeptoas it always had15:58
hrwI think that opendev infra should have it somewhere15:58
yoctozeptoand we trust it a lot15:58
yoctozeptoas well as all its admins15:58
hrwin case of bus incident happening with yoctozepto and mgoddard15:58
yoctozeptoyeah, they can decryp the secrets15:58
mgoddardpotentially, a zuul job could rotate the password, encrypt it, and print the encrypted result15:58
mgoddardfor example15:58
mgoddardthen no human would have the password15:59
hrwI know that I do not want to know it15:59
mgoddardbut infra could access it15:59
yoctozeptoI would worry about that thing getting broken in the middle15:59
yoctozeptonew password and it being nowhere15:59
mgoddardit's possible16:00
mgoddardwe'd probably need a safety access account until we know it works16:00
mgoddardanyway, we're out of time16:00
* hrw out16:00
yoctozeptothanks mgoddard16:00
mgoddardI think this is a concern though, and we shouldn't assume anything about what happened with dockerhub16:00
mgoddardperhaps we should put something on openstack-dicuss16:00
yoctozeptoyes, please do; sometimes we get really nice insight16:01
mgoddard#action mgoddard email openstack-discuss about quay.io credentials16:01
mgoddard#endmeeting16:01
*** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:01
openstackMeeting ended Wed Mar 24 16:01:44 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-24-15.00.html16:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-24-15.00.txt16:01
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-24-15.00.log.html16:01
rafaelweingartneHey guys, I was waiting for the open questions/discussions moment :)16:02
rafaelweingartnecan I ask you guys something?16:02
rafaelweingartneWe have been experiencing some slowness when executing/using Kolla-ansible. We notice that the "Gather facts" tasks is running slow due to the high number of interfaces in the compute nodes16:02
rafaelweingartnehave you guys experienced something similar?16:03
rafaelweingartneInternally, we added "filter: "ansible_[!qt]*"" here: https://github.com/openstack/kolla-ansible/blob/cdffc4845f8b4e642f3e896870fbef338310efa4/ansible/gather-facts.yml#L3416:03
rafaelweingartneAnd we would like to check if you guys are also experiencing the same, before opening the patch upstream16:04
mgoddardrafaelweingartne: sounds like something outside of kolla ansible16:04
wuchunyangyes, we have the similar issue.16:04
wuchunyanghttps://etherpad.opendev.org/p/kolla-wallaby-ptg  line: 55416:05
rafaelweingartnewuchunyang: I see16:05
rafaelweingartneexactly that16:05
wuchunyangactually, filter key  just filters the results. ansible still gather the facts.16:06
*** bengates has quit IRC16:06
rafaelweingartneyes, but then it is just once, and the huge number of entries there do not slow the rest of the process16:06
*** bengates has joined #openstack-kolla16:07
wuchunyanghave you try to use ansible cache ?16:07
kevkowhat about fact caching ? it should fix (except first run)16:07
rafaelweingartneby just applying the filter, the run time changed from  0:18:23.854 to 0:02:27.42916:07
rafaelweingartnekevko: we have not tried to use caching.16:08
rafaelweingartneI will check this as well then, before creating a patch16:08
kevkoi'm also curious ..because when we are upgrading through kolla ..we are waiting also 30 minutes for facts :(16:08
rafaelweingartneyes, and that was starting to bother us, specially now with bigger and bigger setups16:09
rafaelweingartnethanks for the hint guys16:09
rafaelweingartneI will check the caching, and then I will propose something for Kolla-ansible16:10
wuchunyanghope you can fix this .16:11
*** rafaelweingartne has quit IRC16:12
*** wuchunyang has left #openstack-kolla16:12
mnasiadkaI saw that some time ago: https://github.com/ansible/ansible/issues/7365416:14
priteauI also saw slowdowns due to large number of interfaces on Neutron nodes with many networks/routers. Fact caching helps16:30
mnasiadkayoctozepto: about healthcheck, it seems octavia driver agent communicates with octavia-api using a socket, we have healthcheck_socket but it requires lsof - which is not installed - are we happy with installing lsof in base? :D16:31
yoctozeptomnasiadka: makes sense16:32
mnasiadkayoctozepto: so I'll followup with another two changes for lsof in Kolla and octavia-driver-agent healthcheck - does that sound ok? (I'll only update the commit message in this change)16:33
yoctozeptomnasiadka: fine by me since you may want to backport this one, no?16:34
mnasiadkayoctozepto: well, it would be nice, but all in all it's a feature :)16:34
mnasiadka(although it was a bug that we didn't have octavia-driver-agent) :)16:34
yoctozeptomnasiadka: yeah, I would go with backporting16:35
yoctozeptoanyhow, I've added octavia to ovn in https://etherpad.opendev.org/p/KollaAnsibleScenarios16:36
mnasiadkathanks16:37
mnasiadkaupdated commit message16:37
openstackgerritPierre Riteau proposed openstack/kayobe master: [DNM] Set environment for testing  https://review.opendev.org/c/openstack/kayobe/+/77341116:37
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add missing octavia-driver-agent  https://review.opendev.org/c/openstack/kolla-ansible/+/76187216:38
openstackgerritMichal Nasiadka proposed openstack/kolla master: base: Add lsof for healthcheck_socket  https://review.opendev.org/c/openstack/kolla/+/78281516:40
openstackgerritPierre Riteau proposed openstack/kayobe master: [DNM] Set environment for testing  https://review.opendev.org/c/openstack/kayobe/+/77341116:56
*** samcat116 has joined #openstack-kolla17:11
*** rpittau is now known as rpittau|afk17:24
openstackgerritMerged openstack/kayobe master: Add Ironic enabled_bios_interfaces and default_bios_interface settings  https://review.opendev.org/c/openstack/kayobe/+/78162017:25
dardeleanHello. Are there plans to support CentOS Stream? Or what is the approach for the future of CentOS in Kolla, if discussed. Thanks17:29
*** suff has joined #openstack-kolla17:35
*** jonaspaulo has joined #openstack-kolla17:36
dardeleanor rocky linux17:40
openstackgerritPierre Riteau proposed openstack/kayobe master: [DNM] Set environment for testing  https://review.opendev.org/c/openstack/kayobe/+/77341117:42
*** gfidente is now known as gfidente|af17:46
*** gfidente|af is now known as gfidente|afk17:46
*** ysirndjuro has quit IRC17:49
openstackgerritPierre Riteau proposed openstack/kayobe master: [DNM] Set environment for testing  https://review.opendev.org/c/openstack/kayobe/+/77341118:01
*** bengates_ has joined #openstack-kolla18:08
*** amoralej is now known as amoralej|off18:10
mgoddarddardelean: Wallaby will be based on CentOS stream18:11
*** bengates has quit IRC18:11
*** andrewbonney has quit IRC18:12
mgoddarddardelean: rocky linux tbd18:12
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134018:13
openstackgerritMerged openstack/kolla-ansible master: Remove Monasca Log Transformer  https://review.opendev.org/c/openstack/kolla-ansible/+/76990018:21
kevkomgoddard: still here ?18:25
kevkomgoddard: i can't add conditional what you advised ..18:25
kevkomgoddard: in mariadb role review i don't have variable enable_proxysql yes18:25
kevkoyet18:25
yoctozeptokevko: how so? it should be global18:26
kevkoyoctozepto: well, yes, but i have it in follow-up patch18:26
yoctozeptouh-oh18:27
yoctozeptothat explains it18:27
yoctozeptoneed mgoddard around18:27
kevkoyoctozepto: what i can do is to add this conditional to follow-up patch for proxysql ..18:27
yoctozeptosounds sensible18:28
openstackgerritMerged openstack/kolla-ansible master: Disable Monasca Log Metrics service by default  https://review.opendev.org/c/openstack/kolla-ansible/+/76990118:32
openstackgerritRadosław Piliszek proposed openstack/kolla master: Deprecate chrony  https://review.opendev.org/c/openstack/kolla/+/78284018:36
openstackgerritMerged openstack/kolla-ansible master: Support disabling Monasca alerting pipeline  https://review.opendev.org/c/openstack/kolla-ansible/+/76990219:03
openstackgerritMerged openstack/kolla-ansible master: Support bypassing Monasca Log API for control plane logs  https://review.opendev.org/c/openstack/kolla-ansible/+/77621919:03
openstackgerritMerged openstack/kolla-ansible master: Follow up fixes for Monasca docs and formatting  https://review.opendev.org/c/openstack/kolla-ansible/+/77819119:04
*** dking has joined #openstack-kolla19:07
*** suff has quit IRC19:10
openstackgerritMerged openstack/kolla-ansible master: Add missing octavia-driver-agent  https://review.opendev.org/c/openstack/kolla-ansible/+/76187219:12
sean-k-mooneyyoctozepto: fyi i was trying to do stable victoria all in one yesterady and deploying without haproxy is broken again19:14
yoctozeptosean-k-mooney: ooh, could be as we are not testing this particular scenario19:15
yoctozeptosean-k-mooney: care to report? (and maybe fix too? :-) )19:15
sean-k-mooneyit keeps trying to connect to the vip address19:15
sean-k-mooneywhich you know wont exist19:15
sean-k-mooneyfor example https://github.com/openstack/kolla-ansible/blob/stable/victoria/ansible/roles/mariadb/tasks/wait_for_loadbalancer.yml19:16
sean-k-mooneyis always run https://github.com/openstack/kolla-ansible/blob/5e638b757bdda9fbddf0fe0be5d76caa3419af74/ansible/roles/mariadb/tasks/register.yml#L219:17
yoctozeptooh, that would be my bad19:17
sean-k-mooneyhttps://github.com/openstack/kolla-ansible/commit/03cd7eb3563788ca1ffd7b19130f18614887a54619:18
sean-k-mooneythat added it19:18
yoctozeptowe should probably add, at least non-voting, this very minimal deployment19:18
yoctozeptoooh, glad not me19:18
sean-k-mooneyi tought we used to use this in the gate at one point19:18
sean-k-mooneythere are other issues19:18
sean-k-mooneyi added a when enable_haprozy | bool19:18
yoctozeptoyeah, but then haproxy was more common still19:19
sean-k-mooneyto skpi it but it will fail a little latere then19:19
sean-k-mooneyhere https://github.com/openstack/kolla-ansible/blob/5e638b757bdda9fbddf0fe0be5d76caa3419af74/ansible/roles/mariadb/tasks/check.yml#L219:20
sean-k-mooneywhich is invoked here https://github.com/openstack/kolla-ansible/blob/3411b9e4201e619613cef427d228cf00f310a144/ansible/roles/mariadb/tasks/deploy.yml#L13-L1419:21
yoctozeptoI am dropping this duplication at some point19:24
yoctozeptoI remember that part19:24
yoctozeptoprobably for wallaby though19:24
yoctozeptoor not merged yet19:24
yoctozeptobut yeah, the point is this is not haproxy-less-friendly19:24
yoctozeptoso to speak19:25
yoctozeptothat said, any particular reason you don't want haproxy?19:25
* yoctozepto selling the working version19:25
sean-k-mooneyoh this was a test deployment on openstack and i just didint feel like creating a neutron port to resreve an ip for it so i disabled it as a workaround last night19:26
sean-k-mooneyand then found it was broken again19:26
sean-k-mooneythis is about the 5 time its been broken it happens every cople of releases19:27
sean-k-mooneyso im use to just using ha proxy19:27
sean-k-mooneywe still document haproxy less deployment for ip constratined envrioments so either we shoudl fix it or drop support offcally and update the doc19:28
sean-k-mooneyhttps://github.com/openstack/kolla-ansible/blob/3411b9e4201e619613cef427d228cf00f310a144/doc/source/admin/advanced-configuration.rst#ip-address-constrained-environments19:29
sean-k-mooneyyoctozepto: this is just the env i was setting out to test if i could expose the service with different dns subdomains instead of ports19:30
yoctozeptosure thing!19:30
yoctozeptomakes sense you would want it like that19:31
yoctozeptoit could also prove more beneficial for those who develop the services using kolla19:31
sean-k-mooneywell i normally use ha proxy even for singel node but it was 3 am  so thought this would be quicker19:31
sean-k-mooneytruned out not to be in the end19:31
yoctozeptoyeah, but in some envs it really does not make sense to find that vip address19:32
sean-k-mooneyyep19:32
yoctozeptowell, you can always spin it up on some loopback19:32
yoctozeptobut still19:32
yoctozepto:D19:32
yoctozeptoI guess we add one job for that19:32
yoctozeptono need to test all distros19:32
sean-k-mooneyya i could have chossen a 127.0.x.y address19:32
sean-k-mooneyi would proably jsut add it to one of the existing jobs19:33
*** zijlboot has quit IRC19:33
sean-k-mooneyor make it a supper shrot one19:33
sean-k-mooneythat just install the db19:33
*** zijlboot has joined #openstack-kolla19:33
yoctozeptoand keystone ;d19:34
*** samcat116 has quit IRC19:37
sean-k-mooneyi mean you praobly want to enable the core services really19:39
sean-k-mooneythat not much longer but ya one job should be more then enough19:40
sean-k-mooneydocumenting the use fo a 127 address could be valid for singel node too as an alterinive to disabling haproxy19:40
sean-k-mooneypart of the issue is proably  cause by https://github.com/openstack/kolla-ansible/commit/026f5cc48a37422cc9ba4b1a277fd8aea73a3e03#diff-792522c538794367129103be0ba66a30d71e4f609ea24bd1d47e7bc9add7ae0019:41
sean-k-mooneyyoctozepto: have ye drop support for deploying mariadb without galara by the way19:42
sean-k-mooneythat used to be one of the big thing that change when you did single node in the past19:42
*** livelace has joined #openstack-kolla19:43
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: Refactor mariadb to support shards  https://review.opendev.org/c/openstack/kolla-ansible/+/76695219:43
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: Rename role haproxy to loadbalancer  https://review.opendev.org/c/openstack/kolla-ansible/+/77061819:44
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: Add proxysql support for database  https://review.opendev.org/c/openstack/kolla-ansible/+/77021519:44
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: Edit services roles to support database sharding  https://review.opendev.org/c/openstack/kolla-ansible/+/77021619:44
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: [CI] Test ProxySQL with shards in the nova cells scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/77062119:44
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: Add mariadb arbitrator to mariadb role  https://review.opendev.org/c/openstack/kolla-ansible/+/78081119:44
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: [CI] Test Mariadb-Arbitrator with shards in the nova cells scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/78097019:44
sean-k-mooneyah i see this is been reqorked a lot a the moment19:44
*** e0ne has quit IRC19:51
sean-k-mooneythat might be the issue alhtough i dont have time to confirm not but if we are configuring glarea even for single node perhaps it never fully starts19:55
sean-k-mooneyoh chrony is being remvoed? is that not the replacemnt for ntpd?19:58
sean-k-mooneyhas it been replaced again19:58
sean-k-mooneypersonlly i would be sad to see it go as i prefer havign kolla manage that in a contaienr instead of using the distos default way but i guess i can see why you might remvoe it20:01
sean-k-mooneysame is true for things like multipathd20:02
sean-k-mooneyi much prefer having as little as possible running on the host out side of a container beyond docker20:02
yoctozeptosean-k-mooney: one-node galera is fine, should not require haproxy20:05
yoctozeptoyeah, I like this approach too20:05
yoctozeptobut ntp is too basic nowadays, like ssh20:05
sean-k-mooneywell isnt ntp dperecated20:06
yoctozeptoand we are fighting distros ways20:06
yoctozeptoI mean ntp as any provider20:06
sean-k-mooneyat least on the rhel side they wanted to move it to chorny at openpoint20:06
yoctozeptonot the ntpd20:06
sean-k-mooneyand likely systmed will take it over20:06
yoctozeptoyeah, that's what is happening on debuntu now20:07
sean-k-mooneyya i think ntpd was depcreated in rhel7 and replace with chrony in rhel 8 and i assume systemd will just do it in rhle920:07
sean-k-mooneyso not really that concerend with removing it20:07
yoctozeptoindeed20:07
sean-k-mooneyjust like to install as little as posibel on the hosts20:07
yoctozeptome too20:08
*** also_stingrayza has joined #openstack-kolla20:10
sean-k-mooneyi havent really been following kolla dev fro a while has anything related to podman come up lately20:10
*** stingrayza has quit IRC20:10
*** bengates_ has quit IRC20:11
sean-k-mooneypersonally im fine using docker but on the topic of less stuff running on the host not haveing a deamon has some advantages althgouh i still prefer docker since it just works20:11
sean-k-mooneygetting podmand to work on anything that is not fedroa based and with out sudo is a pain.20:12
*** samcat116 has joined #openstack-kolla20:33
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134020:50
yoctozeptosean-k-mooney: no progress that I know of; pity that Canonical is kinda entrenched with Snapd and Red Hat analogously with Podman; and suse leaving openstack and going 100% kubernetes21:02
*** jonaspaulo has quit IRC21:02
*** livelace has quit IRC21:03
sean-k-mooneywell snapd and podman do very different things21:07
sean-k-mooneysnapd is more like flatpack21:07
sean-k-mooneybut with a deamon21:07
sean-k-mooneycanonoical provide docker via snapd as a way to install it now21:07
sean-k-mooneyadn for ubunut core i think that is the only way to install it21:08
sean-k-mooneysicne that has an imuntable root file system21:08
sean-k-mooneyi only use podman if its preinstalled21:08
yoctozeptosean-k-mooney: yup, they are two quite different things (but one could obviously do openstack in either) but I mean the adoption model - it does not want to go beyond the parent company distro21:17
yoctozeptoanyhow, time to bed now for me21:18
yoctozeptotake care21:18
*** samcat116 has quit IRC21:34
*** kevko has quit IRC21:37
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134022:12
*** kevko has joined #openstack-kolla22:36
*** vishalmanchanda has quit IRC22:41
*** kevko has quit IRC23:00
*** stand has quit IRC23:20
*** Underknowledge has quit IRC23:40
*** Underknowledge has joined #openstack-kolla23:51
*** Underknowledge has quit IRC23:54
*** Underknowledge has joined #openstack-kolla23:54
*** Underknowledge has quit IRC23:59
*** Underknowledge has joined #openstack-kolla23:59
« Tuesday, 2021-03-23 Index Thursday, 2021-03-25 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!