| *** dviroel is now known as dviroel|out | 00:10 | |
| *** raukadah is now known as chandankumar | 04:43 | |
| *** ysandeep|out is now known as ysandeep | 04:50 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Remove testing on debian https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/820663 | 07:16 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Remove support for Ubuntu Bionic https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/820669 | 07:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Remove testing on Centos-8 https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/820679 | 07:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_zun master: Remove testing on Centos-8 https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/820679 | 07:21 |
| *** ysandeep is now known as ysandeep|lunch | 07:23 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove support for Debian Buster https://review.opendev.org/c/openstack/openstack-ansible/+/820664 | 07:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove support for Ubuntu Bionic https://review.opendev.org/c/openstack/openstack-ansible/+/820671 | 07:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove centos-8 support https://review.opendev.org/c/openstack/openstack-ansible/+/820854 | 07:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove CI jobs for centos-8 https://review.opendev.org/c/openstack/openstack-ansible/+/820982 | 07:43 |
| noonedeadpunk | with UCA we should be careful as it will make major upgrade of os version with any minor one... | 08:11 |
| noonedeadpunk | *of openstack version | 08:12 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/820689 | 08:12 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/820690 | 08:12 |
| noonedeadpunk | was doing right the same lol | 08:12 |
| jrosser | do we see this before where openstack-ansible-tests fails getting the new branch global pins from openstack-ansible repo | 08:14 |
| jrosser | understandable as that's not branched yet | 08:14 |
| jrosser | but it blocks glance and keystone | 08:14 |
| noonedeadpunk | I bet we saw... | 08:14 |
| jrosser | this is job filters not stopping the functional tests I think | 08:15 |
| noonedeadpunk | um, for functional test tox change should trigger checks I believe? | 08:15 |
| noonedeadpunk | as they're run with tox eventually | 08:16 |
| jrosser | yeah, sounds reasonable | 08:16 |
| jrosser | I remove them for keystone :) https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820678 | 08:17 |
| opendevreview | James Gibson proposed openstack/ansible-role-uwsgi master: Add support for TLS to UWSGI https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/820532 | 08:29 |
| *** ysandeep|lunch is now known as ysandeep | 08:35 | |
| noonedeadpunk | ah, yes, make sense. For some reason I thought it does test federation somehow and not just apache deployment.... | 08:50 |
| jrosser | I had a look into the test yesterday and it's really not doing anything smart at all | 08:52 |
| noonedeadpunk | well, let me look into glance test actually... | 08:52 |
| jrosser | yeah, I was thinking that it maybe not really difficult to put an nfs server on the AIO and get rid of that one too | 08:53 |
| jrosser | then perhaps we are very very close to not needing the tests repo any more | 08:54 |
| noonedeadpunk | I think things like systemd_* and config_tempalte still use functonal tests | 08:55 |
| noonedeadpunk | but most likely they should be just local ones and don't need anything from outside | 08:55 |
| noonedeadpunk | at least in theory | 08:55 |
| noonedeadpunk | and we also have sync jobs there which should be moved somewhere | 08:55 |
| noonedeadpunk | I wonder if we even need nfs test as we can pretty easily test ceph/swift scenarios | 08:56 |
| jrosser | I did wonder if some sort of common role with tasks_from would remove the need to sync all these tasks to many repos | 08:57 |
| noonedeadpunk | oh! | 08:57 |
| noonedeadpunk | we can ship these things as collection for example | 08:58 |
| jrosser | then like only one place ever needed to fix things if they break, and much less circular dependancy problems | 08:58 |
| noonedeadpunk | I mean - simple role with jsut main.yml | 08:58 |
| jrosser | well yes, could be a role per set of common tasks | 08:58 |
| jrosser | or we could have a role as a kind of bucket to keep them all in | 08:59 |
| jrosser | I like the idea generally as it seems we can remove a bunch of complication / CI overhead it it works out nicely | 09:00 |
| noonedeadpunk | I'm trying to recall if in any role it matters when service_setup is executed | 09:00 |
| noonedeadpunk | and can't think of any | 09:00 |
| noonedeadpunk | the amount of data that will be passed will be huge though | 09:01 |
| noonedeadpunk | to that role I mean | 09:01 |
| noonedeadpunk | But I agree about idea - it would simplify things a lot | 09:02 |
| jrosser | for some of these common tasks it's already like that - https://github.com/openstack/openstack-ansible-os_glance/blob/master/tasks/main.yml#L60-L76 | 09:03 |
| jrosser | we pretty much treat these task includes like they are roles anyway | 09:04 |
| noonedeadpunk | I mean - if it would be single role that called once - we would need to sum all _service, _oslomsg and _oslodb vars in one pass | 09:05 |
| jrosser | oh, well I kind of meant | 09:05 |
| noonedeadpunk | to use tasks_from, I got it | 09:05 |
| jrosser | yeah | 09:05 |
| noonedeadpunk | I just moved forward a bit and thought that eventually we can do that in one include... | 09:06 |
| noonedeadpunk | as these 3 things we always run against single host only and we run them all at the beginning | 09:06 |
| noonedeadpunk | It doesn't scale though | 09:06 |
| noonedeadpunk | so would be hard to add different loginc | 09:07 |
| noonedeadpunk | *logic | 09:07 |
| noonedeadpunk | I just can't explain why, but for some reason I'm squeamish when it comes to tasks_from.... | 09:08 |
| jrosser | I get that, yes | 09:08 |
| jrosser | because in this case it feels like abusing a role, as that's the only thing we can put in a collection, when really what we want is shared tasks | 09:09 |
| jrosser | but perhaps that's an argument for making a collection with a whole set of single purpose small roles in it | 09:09 |
| noonedeadpunk | So like 3 roles with jsut main.yml in 1 collection sound even better to me ヽ(。_°)ノ | 09:09 |
| jrosser | like mq_setup... blah blah | 09:09 |
| jrosser | yes | 09:09 |
| noonedeadpunk | and this collection might be the same one that we'll use for our plugins? | 09:10 |
| jrosser | previously the overhead of making many zuul repos would make it pretty heavy just to do anything | 09:10 |
| noonedeadpunk | or different, whatever | 09:10 |
| jrosser | but one collection with several roles -> win | 09:10 |
| noonedeadpunk | (but I still not sure if this concept would be good for service roles) | 09:11 |
| noonedeadpunk | however I'm was thinking about moving to the same concept things that don't really change like systemd_* stuff into 1 collection | 09:11 |
| noonedeadpunk | but not sure | 09:12 |
| noonedeadpunk | anyway, that's quite different topic I guess | 09:22 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820777 | 09:42 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/820689 | 09:55 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/820690 | 10:00 |
| *** ysandeep is now known as ysandeep|afk | 10:11 | |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add documentation of security improvements made to Openstack Ansible https://review.opendev.org/c/openstack/openstack-ansible/+/820370 | 10:24 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Update release name for Xena https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/820944 | 10:35 |
| *** dviroel|out is now known as dviroel | 10:38 | |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Update Calico Felix version https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/820654 | 10:41 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add openstack-ansible-utils collection https://review.opendev.org/c/openstack/openstack-ansible/+/820998 | 10:41 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Use common service setup tasks from a collection rather than in-role https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820999 | 10:41 |
| jrosser | noonedeadpunk: ^ these seem to work locally, so gives an idea about how we could use a common utils collection | 10:43 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Remove uw_apache functional jobs. https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820678 | 10:43 |
| noonedeadpunk | JamesGibo: regarding https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/820857. It feels like we use intermediate everywhere for _ssl_ca_cert? Like rabbit https://opendev.org/openstack/openstack-ansible-rabbitmq_server/src/branch/master/defaults/main.yml#L178 or haproxy | 10:43 |
| noonedeadpunk | But I wonder if these all really intend to be intermediates and not roots? | 10:44 |
| noonedeadpunk | jrosser: yeah. that exactly what I thought tbh | 10:44 |
| noonedeadpunk | it's more a question where to base that and how to name it | 10:45 |
| jrosser | indeed, name is tricky | 10:46 |
| jrosser | perhaps it's a way to migrate things out of the plugins repo and stop using that | 10:46 |
| jrosser | which is why I landed on 'utils' in the end | 10:46 |
| noonedeadpunk | yes, indeed, I was thinking to merge that with our plugins as well - might make sense | 10:47 |
| noonedeadpunk | or you meant oposite? :D | 10:47 |
| noonedeadpunk | *opposite | 10:47 |
| jrosser | plugins is kind of a odd name now that we have collections | 10:48 |
| jrosser | so the choice is to either somehow evolve the plugins repo into a collection | 10:48 |
| jrosser | or to make a new collection and copy what we want out of plugins into it | 10:48 |
| noonedeadpunk | I don't think we can name it as openstack-ansible-collections | 10:49 |
| jrosser | I was thinking the second choice was cleaner | 10:49 |
| jrosser | yes | 10:49 |
| noonedeadpunk | will be toooooo confusing with http://opendev.org/openstack/ansible-collections-openstack | 10:49 |
| jrosser | that's why I did not yet think of anything better than openstack-ansible-utils | 10:49 |
| jrosser | as it's kind of internal to OSA things | 10:49 |
| noonedeadpunk | while I don't have better idea, utils vs ops is a bit... | 10:50 |
| jrosser | openstack-ansible-common thats somehow not better either | 10:51 |
| noonedeadpunk | yeah | 10:51 |
| jrosser | anyway - seems the code is easy part :) | 10:51 |
| noonedeadpunk | naming is always the most hard one hehe | 10:52 |
| jrosser | I ran an AIO through to keystone with it, and I think it's a really big improvement over syncing the files | 10:52 |
| noonedeadpunk | tbh now I'm close to the first option - to evolve plugins to collection | 10:52 |
| noonedeadpunk | first of all I won't have to create new repo :D | 10:53 |
| jrosser | haha | 10:53 |
| noonedeadpunk | but eventually all names are confusing in some sort of | 10:53 |
| noonedeadpunk | perfect would be openstack-ansible-collection, but since we already have ansible-collections-openstack... | 10:53 |
| noonedeadpunk | openstack-ansible-deliverables ?:) | 10:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/xena: Update release name for Xena https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/820887 | 10:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove retrievement of config_tempalte as a module https://review.opendev.org/c/openstack/openstack-ansible/+/821001 | 11:01 |
| noonedeadpunk | I just realized big isssue with collections - we can't have depends-on them I believe? | 11:02 |
| noonedeadpunk | as I'm not sure we can feed local path for ansible-galaxy? | 11:02 |
| noonedeadpunk | having that said, we can't really test them as well against aio? | 11:03 |
| noonedeadpunk | so we should have some local tests for them I believe... | 11:04 |
| noonedeadpunk | also I'm not really sure what to do with https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/820857 and if we actually made things correctly in other repos where provided jsut intermediate and not chain or root? | 11:06 |
| noonedeadpunk | as ca-certificate | 11:06 |
| jrosser | -EMEETING, bbl | 11:06 |
| *** ysandeep|afk is now known as ysandeep | 11:16 | |
| JamesGibo | In my opinion there are a number of roles that deploy a file called *-ca.pem, but the contents of the file is the intermediate ca, i think this should be changed to be the Root CA. It will work because the when a client checks a server cert, the client will trust the server because it trust the intermdiate and stop evaluating the trust chain there and not use the root ca, but really it should be trusting the root ca as this has a longer | 11:40 |
| JamesGibo | The common practice should be the server presents the required chain to a client so it can verify trust using a Root CA | 11:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add galaxy metadata and roles to use as a collection https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/821009 | 11:52 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add openstack-ansible-plugins as a collection https://review.opendev.org/c/openstack/openstack-ansible/+/820998 | 11:53 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add openstack-ansible-plugins as a collection https://review.opendev.org/c/openstack/openstack-ansible/+/820998 | 11:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Use common service setup tasks from a collection rather than in-role https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820999 | 11:56 |
| jrosser | we are certainly needing a way to use zuul sources for collections | 11:57 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_glance master: Add support for TLS to Glance https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/821011 | 12:01 |
| noonedeadpunk | Well I actually thought that chains are the thing to be used mainly | 12:05 |
| noonedeadpunk | as eventually certs are signed with intermediate? | 12:05 |
| noonedeadpunk | so itermediate should be also provided to check for the relation | 12:05 |
| noonedeadpunk | but again, I'm not that much of expert in whole ssl topic | 12:05 |
| noonedeadpunk | So it was more like an impression | 12:06 |
| noonedeadpunk | and to mention - mysqlclient is not happy with just intermediate | 12:06 |
| jrosser | imho there is a whole bunch of variation in implementation | 12:06 |
| jrosser | if the program understands the system CA store you can give it the intermediate, and it can validate it | 12:07 |
| jrosser | but if it doesn't you somehow need to supply the whole chain | 12:07 |
| jrosser | this is kind of messy and will be different for each thing we try | 12:07 |
| noonedeadpunk | but I'd say that chain is better then jsut root? | 12:10 |
| noonedeadpunk | or | 12:10 |
| noonedeadpunk | ? | 12:10 |
| jrosser | well we need to be specific | 12:10 |
| jrosser | do we talk about the intermediate+cert that a server must present | 12:10 |
| jrosser | or a client cert in mTLS | 12:11 |
| jrosser | or the server side of mTLS which must validate the client cert | 12:11 |
| noonedeadpunk | talking about stuff that is produced by https://opendev.org/openstack/ansible-role-pki/src/branch/master/tasks/standalone/create_ca.yml#L135-L144 | 12:11 |
| noonedeadpunk | I guess it's root+intermediate? | 12:12 |
| jrosser | well yes and that's because of oddness in libvirtd | 12:12 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_glance master: Add support for TLS to Glance https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/821011 | 12:12 |
| noonedeadpunk | well, galera also asks for this _or_ root | 12:13 |
| jrosser | this is likely one of the cases where it doesn't know about the system trust store | 12:13 |
| noonedeadpunk | not galera, but mysql client | 12:13 |
| noonedeadpunk | ie https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/820857/2/defaults/main.yml | 12:13 |
| noonedeadpunk | So James suggested to use Root here instead, so I'm kind of trying to iterate | 12:14 |
| noonedeadpunk | as passing root sounds like variables needs to be renamed | 12:14 |
| noonedeadpunk | and chain souds kind of fair alternative I guess? | 12:14 |
| noonedeadpunk | but as I really don't understand some things there, I'd rely on your expertise | 12:15 |
| jrosser | I think I'm missing something tbh | 12:16 |
| jrosser | but it's possible that we have a mistake, for example the server cert should be the chain | 12:16 |
| noonedeadpunk | ah, well... | 12:17 |
| jrosser | can inspect this with openssl tools I think | 12:17 |
| noonedeadpunk | I can play in aio with galera | 12:17 |
| noonedeadpunk | I haven't tried chain as cert though | 12:18 |
| jrosser | sort of `openssl s_client -showcerts -connect gnupg.org:443` | 12:18 |
| jrosser | need to start with the basics | 12:18 |
| jrosser | does the Galera server give the server cert and the intermediate on port 3306? | 12:19 |
| jrosser | if not we need to fix that rather than hack around | 12:19 |
| noonedeadpunk | nah | 12:19 |
| noonedeadpunk | `139816755000640:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:331:` | 12:20 |
| jrosser | JamesGibo: ^ interested in your opinion here are you're much more across this currently that I am | 12:20 |
| noonedeadpunk | The reason why I thought that mysql allows unsecure is if telnet, you will jsut see `5.5.5-10.6.5-MariaDB-1:10.6.5+maria~focal-logM@PNw.|�J9"9:X.x49^Rmysql_native_password^]` | 12:21 |
| noonedeadpunk | which sounds like smtp way - you need to start tls command or smth like that | 12:21 |
| noonedeadpunk | yeah requires `-starttls mysql | 12:22 |
| noonedeadpunk | https://paste.opendev.org/show/811541/ | 12:23 |
| jrosser | ok so that looks good - the server provides it's cert and the intermediate | 12:24 |
| noonedeadpunk | but it's with this change applied already... | 12:24 |
| jrosser | isn't this what makes that happen https://github.com/openstack/openstack-ansible-galera_server/blob/master/defaults/main.yml#L238 | 12:27 |
| noonedeadpunk | ah, yes, indeed | 12:28 |
| noonedeadpunk | I just rollbacked the thing to verify | 12:29 |
| jrosser | there's two chains | 12:29 |
| jrosser | there's server+intermediate for "normal" things | 12:30 |
| jrosser | then we had to make intermediate+root for some unusual behaviour in libvirtd | 12:30 |
| noonedeadpunk | and mysqlclient seems to be same here | 12:30 |
| jrosser | right | 12:30 |
| jrosser | so it's possible we need to install another file | 12:31 |
| noonedeadpunk | https://paste.opendev.org/show/811542/ | 12:31 |
| jrosser | rather than change the ones we have for the server | 12:31 |
| jrosser | oh well look https://github.com/openstack/openstack-ansible-galera_server/blob/master/defaults/main.yml#L238 | 12:32 |
| jrosser | the "CA" (!) is the intermediate | 12:32 |
| noonedeadpunk | yes and we have that everywhere | 12:33 |
| noonedeadpunk | which I found a bit weird | 12:33 |
| noonedeadpunk | ie https://opendev.org/openstack/openstack-ansible-rabbitmq_server/src/branch/master/defaults/main.yml#L178 | 12:33 |
| noonedeadpunk | (was following that for galera) | 12:33 |
| jrosser | in the case of rabbitmq I think it finds the actual root in the system trust store to complete the trust-chain | 12:34 |
| noonedeadpunk | ok | 12:34 |
| jrosser | but the error from mysql client is that it doesn't know what the thing is that has signed the intermediate | 12:34 |
| noonedeadpunk | so eventually we should try having different `ssl-ca` - for client and for server, right? | 12:35 |
| jrosser | another factor here is that 99% of all examples never use an intermediate | 12:36 |
| noonedeadpunk | openssl output from localhost:3306 is just the same in botyh cases | 12:36 |
| jrosser | but that's just never really true for a real company CA | 12:36 |
| jrosser | you could be right on needing different ssl-ca for client and server | 12:39 |
| jrosser | it's unfortunate that it doesn't use the system trust store | 12:39 |
| jrosser | because each time we embed the root in a chain like this we set a massive trap for when it comes time to rotate it | 12:39 |
| jrosser | wtf https://bugs.mysql.com/bug.php?id=54158 | 12:41 |
| jrosser | this seems to contain some helpful tips https://smallstep.com/hello-mtls/doc/client/mysql-client | 12:42 |
| jrosser | appears to confirm that the client needs a root+intermediate bundle | 12:43 |
| *** ysandeep is now known as ysandeep|brb | 12:49 | |
| jrosser | noonedeadpunk: I think we also need to make the generation of the different certs conditional | 12:50 |
| jrosser | I was thinking about this after what spates was trying to do with mtls for neutron/ovn | 12:50 |
| jrosser | some places you have just a server, some places just a client, some places both | 12:50 |
| jrosser | but it's all the same role, with one set of definitions | 12:51 |
| jrosser | that is kind of also the case with the galera role, because in utility we are just a client | 12:51 |
| noonedeadpunk | sorry was in a meeting | 12:51 |
| jrosser | np | 12:52 |
| noonedeadpunk | the problem with galera s that client and server are kind of aligned... | 12:52 |
| jrosser | aligned = same place? | 12:52 |
| noonedeadpunk | well yes, because debia-start is used for server startup and utilize client part | 12:53 |
| noonedeadpunk | different configs though | 12:53 |
| noonedeadpunk | but well, it's not pki fault I guess, but more matter of _pki_install_certificates definition | 12:54 |
| jrosser | right, so if in the list of certificates to generate/install, we have something like condition: "{{ is foo in groups['bar'] }}' | 12:54 |
| jrosser | then in the ski role we have when: {{ item.condition | default(True) }} | 12:54 |
| jrosser | *pki | 12:54 |
| noonedeadpunk | ah, yes, that will be usefull for sure | 12:54 |
| jrosser | we can control easily what gets generated/installed on different groups | 12:54 |
| noonedeadpunk | agree | 12:55 |
| jrosser | this didn't really come up yet as we're only just starting to think about mTLS | 12:55 |
| jrosser | anyway, looks like there is a bug in mysql client from my link earlier | 12:58 |
| noonedeadpunk | well, it's been quite a while hehe | 12:58 |
| noonedeadpunk | also - I'm not using mTLS atm | 12:59 |
| noonedeadpunk | but yeah. patch looks valid according to stepca docs | 13:00 |
| noonedeadpunk | is it ?:) | 13:00 |
| jrosser | ah well that's the thing | 13:00 |
| jrosser | do we want to create a root+intermediate CA bundle for the client | 13:00 |
| jrosser | create/copy | 13:00 |
| jrosser | and leave the server configured more normally | 13:00 |
| noonedeadpunk | from what I saw from openssl output - it doesn't matter | 13:02 |
| jrosser | it would be better if there server was not configured with the root | 13:02 |
| noonedeadpunk | But I agree here | 13:02 |
| noonedeadpunk | let's try to quickly implement conditional installs then | 13:02 |
| jrosser | sure | 13:02 |
| jrosser | two ways, either with conditional, or inline jinja in the role like we have in some places | 13:03 |
| jrosser | second is pretty unreadable though | 13:03 |
| *** ysandeep|brb is now known as ysandeep | 13:07 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-pki master: Allow to provide conditions for pki_install_certificates https://review.opendev.org/c/openstack/ansible-role-pki/+/821023 | 13:09 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add galaxy metadata and roles to use as a collection https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/821009 | 13:28 |
| noonedeadpunk | jrosser: btw, funny thing. If you don't provide ssl-ca for client - it's just satisfied | 13:28 |
| jrosser | is that the same as "don't validate the server certificate" | 13:29 |
| noonedeadpunk | it's different | 13:30 |
| noonedeadpunk | as cert is never issued for localhost | 13:30 |
| noonedeadpunk | and the error would be `ERROR 2026 (HY000): SSL connection error: Validation of SSL server certificate failed` | 13:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Do not verify certificate for local connects https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/820857 | 13:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove retrievement of config_tempalte as a module https://review.opendev.org/c/openstack/openstack-ansible/+/821001 | 13:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: [DNM] Test TLS with infra test https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/820861 | 13:43 |
| *** ysandeep is now known as ysandeep|dinner | 13:49 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/820223 | 13:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone stable/xena: Remove uw_apache functional jobs. https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/821033 | 13:56 |
| jrosser | ^ I think I did that in the branching patches, which won't merge otherwise | 13:56 |
| noonedeadpunk | yeah, indeed | 13:56 |
| noonedeadpunk | was trying to catch what needs to be packported | 14:01 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/xena: Update Calico Felix version https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/821034 | 14:01 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Fix UCA enablement for Focal https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/820942 | 14:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/xena: Fix UCA enablement for Focal https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/821035 | 14:34 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/820777 | 14:38 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts stable/xena: Update release name for Xena https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/820887 | 14:43 |
| *** dviroel is now known as dviroel|lunch | 14:56 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/wallaby: Fix UCA enablement for Focal https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/821084 | 15:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/victoria: Fix UCA enablement for Focal https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/821085 | 15:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/victoria: Fix UCA enablement for Focal https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/821085 | 15:19 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: [WIP] Add support for TLS to Glance backends https://review.opendev.org/c/openstack/openstack-ansible/+/821090 | 15:23 |
| noonedeadpunk | so https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/820857/3 is super simple now.... | 15:30 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use local repositories for ansible collections in CI https://review.opendev.org/c/openstack/openstack-ansible/+/821093 | 15:34 |
| jrosser | noonedeadpunk: right, you just tell it to connect as ssl, no more complex than that | 15:35 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use local repositories for ansible collections in CI https://review.opendev.org/c/openstack/openstack-ansible/+/821093 | 15:36 |
| noonedeadpunk | is it theoretically working? ^ | 15:40 |
| noonedeadpunk | I'll rollback this change then https://review.opendev.org/c/openstack/openstack-ansible/+/821001/2/zuul.d/jobs.yaml | 15:40 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove retrievement of config_tempalte as a module https://review.opendev.org/c/openstack/openstack-ansible/+/821001 | 15:42 |
| jrosser | the CI/collections? as far as I can test it, I cloned a collection to /home/zuul/.... and set ZUUL_SRC_PATH in a AIO and it seemed to do the right thing | 15:43 |
| jrosser | it's so nearly similar to what we do with roles, yet completely differnt | 15:43 |
| jrosser | I need to go sit in a quiet corner for a while now to recover :) | 15:43 |
| noonedeadpunk | hehe :) | 15:44 |
| noonedeadpunk | well, what we do for roles is indeed.... | 15:44 |
| noonedeadpunk | could be harldy read | 15:44 |
| *** ysandeep|dinner is now known as ysandeep | 15:45 | |
| noonedeadpunk | I just was not sure if ansible-galaxy will be happy with git+file:// and without version | 15:45 |
| noonedeadpunk | and what it will actually install | 15:45 |
| noonedeadpunk | as it relies on galaxy.yml as for version | 15:46 |
| jrosser | it copies version from the requirements file | 15:46 |
| jrosser | and this is kind of tricky | 15:46 |
| jrosser | actually the patch is wrong | 15:46 |
| jrosser | it would need to be 'master', -ish | 15:47 |
| jrosser | well specifically HEAD, I guess | 15:47 |
| jrosser | and that gives another problem, in CI we would test always head-of-branch rather than the released version number | 15:48 |
| noonedeadpunk | I'd say HEAD might be fine.... As we want to test incoming changes and use depends-on as well | 15:49 |
| jrosser | but then we don't test what people deploy | 15:50 |
| * jrosser looks for the quiet corner again | 15:51 | |
| noonedeadpunk | we don't do that for roles either | 16:00 |
| noonedeadpunk | so collections is smaller concern here... | 16:00 |
| jrosser | if we move those common tasks into a collection like I tried out today | 16:01 |
| noonedeadpunk | probably we should have some periodic jobs or smth, that would explicitly checkout latest tag and do deployment properly | 16:01 |
| jrosser | then we want the CI to work pretty much like it does for roles today | 16:01 |
| noonedeadpunk | yep, which means we don't test what exactly people deploy... | 16:02 |
| jrosser | the collection release process is a bit of a distraction for things that are integral in openstack-ansible, seems more appropriate to stick to master there | 16:02 |
| jrosser | well, stick to master / bump she on stable | 16:02 |
| jrosser | *sha | 16:02 |
| jrosser | external collections is different as we don't expect to depends-on those | 16:03 |
| jrosser | *except* the openstack modules, which sits right in the middle | 16:03 |
| jrosser | so maybe different collections get treated differently, we stick with released versions for upstream stuff | 16:04 |
| jrosser | and we use a git sha for our own internal collections | 16:04 |
| noonedeadpunk | if we don't want to spend more time on maintaining them - we should leave them to be like external one | 16:04 |
| noonedeadpunk | otherwise we will have to debug possible issues on master | 16:04 |
| jrosser | oh sure yes, stable branches are different | 16:05 |
| jrosser | but we need patches to openstack-ansible-plugins to actually test that patch | 16:05 |
| noonedeadpunk | s/them/ansible-collections-openstack/ | 16:05 |
| jrosser | rather than install some random version of itself ;) | 16:05 |
| * noonedeadpunk doesn't have any quiet corner | 16:06 | |
| jrosser | git+file might be wrong anyway, because that clones the repo | 16:07 |
| noonedeadpunk | so for plugins we can make symplinks to old locations like we do with config_template to keep compatability | 16:07 |
| jrosser | there's a /path/to/collection with type: dir as well which might be better | 16:07 |
| noonedeadpunk | oh, I didn't know it's supported | 16:07 |
| noonedeadpunk | it's smth new that came with 2.11? | 16:08 |
| jrosser | https://docs.ansible.com/ansible/latest/user_guide/collections_using.html#installing-a-collection-from-source-files | 16:09 |
| noonedeadpunk | great version picker... available options - latest,devel,5,2.9 :( | 16:11 |
| jrosser | yeah, I was sad about that | 16:11 |
| noonedeadpunk | sorry, 5 is not even an option... | 16:11 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts stable/xena: Fix UCA enablement for Focal https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/821035 | 16:12 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron stable/xena: Update Calico Felix version https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/821034 | 16:13 |
| *** dviroel|lunch is now known as dviroel | 16:13 | |
| noonedeadpunk | oh, so we don't test a shit now for X | 16:13 |
| jrosser | hmm? because we don't branch the main repo yet so it's testing SHA from before the branch? | 16:15 |
| noonedeadpunk | it's not testing a thing | 16:16 |
| noonedeadpunk | because job is not present for that branch | 16:16 |
| noonedeadpunk | (I guess) | 16:16 |
| jrosser | ah well the tests aren't defined | 16:16 |
| jrosser | whats left over in the roles to merge? | 16:16 |
| noonedeadpunk | Well I wanted to backport SSL for galera and purge config_template module | 16:16 |
| noonedeadpunk | but probably it's worth to branch now | 16:17 |
| noonedeadpunk | and make rc2 later or smth | 16:17 |
| jrosser | glance too | 16:17 |
| noonedeadpunk | yeah | 16:17 |
| jrosser | but we did merge the .gitreview so maybe that's ok | 16:17 |
| noonedeadpunk | I was working on nfs for integrated before all meetings that followed :( | 16:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-pki master: Allow to provide conditions for certificates https://review.opendev.org/c/openstack/ansible-role-pki/+/821023 | 16:27 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Add option to force encryption of all health checks over SSL https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/820572 | 16:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add nfs deployment scenario https://review.opendev.org/c/openstack/openstack-ansible/+/821096 | 16:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance master: Replace NFS test with integrated one https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/821097 | 16:44 |
| *** ysandeep is now known as ysandeep|out | 17:10 | |
| spatel | what is the deal here? - https://paste.opendev.org/show/811548/ | 17:23 |
| spatel | I used github repo that works! and now i git this error - https://paste.opendev.org/show/811554/ | 17:24 |
| spatel | hit* | 17:24 |
| jrosser | did you apt-update .... to get the new CA package? | 17:25 |
| spatel | running - apt-get install ca-certificates | 17:27 |
| spatel | we should add this in official doc or osa/bootstrap | 17:28 |
| spatel | I meant somewhere here - https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/deploymenthost.html | 17:28 |
| spatel | That works! | 17:29 |
| opendevreview | Merged openstack/openstack-ansible-os_mistral master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/820435 | 18:09 |
| noonedeadpunk | jrosser: I wonder _if_ we can manage colletions clone process just with a-r-r? That would reduce some complexity around CI logic at least. And then they can be just installed with dir path? | 18:48 |
| noonedeadpunk | at least internal ones | 18:48 |
| noonedeadpunk | as we know exactly where they will be placed. | 18:49 |
| noonedeadpunk | and they will be just bumped by defult as any other role | 18:50 |
| noonedeadpunk | I'm not sure how good this approach is, just came to mind and decided to share :) | 18:50 |
| noonedeadpunk | it would be chicken-egg though if we ever decide to move parallel git clone module to collection.... | 18:51 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_neutron master: Add Support for DPDK Bonding https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/814825 | 19:08 |
| jamesdenton | spatel ^^^ if you could kick the tires (tyres) on that, i would appreciate it | 19:09 |
| spatel | jamesdenton ?? | 19:10 |
| spatel | oh! - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/814825 | 19:11 |
| jamesdenton | you may recall i mentioned moving the bond config to the provider network library vs having a related block of configuration for each bond defined somewhere else | 19:11 |
| jamesdenton | this is that. | 19:11 |
| mgariepy | err anyone having issue with ceph client not being run Train and up ? | 19:12 |
| mgariepy | with train** | 19:12 |
| jamesdenton | spatel https://paste.opendev.org/show/811560/ | 19:13 |
| spatel | checking my patch.. i am little lost here but let me understand and read your comments. | 19:17 |
| spatel | jamesdenton you are saying we shouldn't use this stanza right? instead use this method - https://paste.opendev.org/show/811560/ | 19:22 |
| jamesdenton | the pastebin, i was simply demonstrating how to leverage the existing provider bridge definitions to define a set of interfaces used to construct the respective bond. and some extra parameters | 19:23 |
| jamesdenton | since we have to define the provider bridges anyway, and we already have precedent for adding a single interface to an ovs bridge, it made sense to me to do the same thing for bonds | 19:24 |
| jamesdenton | so, you just add multiple interfaces that get used to construct a bond port | 19:24 |
| jamesdenton | ovs_dpdk_pci_addresses will still be a list of all possible dpdk interfaces, since it is used to populate /etc/dpdk/interfaces and apply the dpdk driver | 19:26 |
| spatel | i was trying to keep it separate because adding config in openstack_user_config will apply to all nodes (nework + compute ) that may create issue or what about mix environment where we have come DPDK node but others non-DPDK or without bonding | 19:27 |
| jamesdenton | yes, it does get tricky with a heterogenous environment. but that can always be handled with host vars. but even the earlier patches would have applied to all | 19:29 |
| spatel | my problem is i don't have my dpdk bonding lab where i can test my code.. :( | 19:31 |
| spatel | jamesdenton do we need this stanza - https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/814825/6..7/tasks/providers/setup_ovs_dpdk.yml#129 | 19:32 |
| jamesdenton | Come to think of it, there is already a way to configure separate provider network configurations based on group membership, so network hosts vs compute hosts, or whatever. https://docs.openstack.org/openstack-ansible/latest/user/prod/provnet_groups.html | 19:34 |
| jamesdenton | spatel that's showing me the diff. | 19:34 |
| spatel | i would say lets not add all possible scenario right now and make it complicated. lets see how we can simple add bonding on specified dpdk nic | 19:38 |
| spatel | what else we can do in my existing patch to get it going so we can merge.. | 19:39 |
| spatel | later we can come back and decided how to make more flexible. | 19:40 |
| spatel | next week i am planning to setup new DPDK based lab with bonding hope i can re-test my patch there | 19:40 |
| jamesdenton | once a feature or workflow is in, it's in, and difficult to change later since it affects production configurations. So there will likely need to be greater consensus among the group on how to best move forward from here. IMO, the two patches are different sides of the same coin. i don't think there's a rush here, so hopefully the others can chime in | 19:44 |
| spatel | jamesdenton agreed about there is no rush here we can take our time to test this patch and push it for merge | 19:45 |
| jamesdenton | :thumbsup: | 19:46 |
| spatel | give me little time to setup my lab to test in environment, because we don't have CI job also to validate this patch | 19:46 |
| spatel | we need real environment to test it | 19:46 |
| jamesdenton | agreed. I've tested it on this AIO w/ 2x 10G broadcom, but greater testing is appreciated | 19:47 |
| spatel | did you test bonding? | 19:48 |
| jamesdenton | yes | 19:48 |
| spatel | with my patch? | 19:48 |
| spatel | or your patch? | 19:48 |
| jamesdenton | i did not test your patch, as i would rather see anything network-related moved to the provider_network library with the other network-plugging things. | 19:49 |
| jamesdenton | which is what i was trying to say in my comments | 19:49 |
| jamesdenton | and what i tried to demonstrate with my patch. that's all. | 19:49 |
| spatel | you did test with your way and if works then why don't we go with that path? | 19:50 |
| jamesdenton | perhaps? we'll see what comes out of the review | 19:50 |
| spatel | I can delete my patch and you can submit your | 19:51 |
| spatel | i would love to test that one | 19:51 |
| jamesdenton | no, that's not necessary. i simply pushed a patchset over your existing patch w/ the difference. | 19:51 |
| jamesdenton | i.e. built on what you had there already | 19:52 |
| spatel | perfect! go ahead.. and take control of it | 19:53 |
| spatel | i will also test next week.. | 19:53 |
| jamesdenton | teamwork. | 19:54 |
| spatel | dreamwork | 19:55 |
| spatel | jamesdenton what is the status of your DPDK loadtesting? | 19:55 |
| spatel | i would love to see your outcome | 19:56 |
| jamesdenton | 2022? | 19:59 |
| jamesdenton | seriously, though, lacking bandwidth on that at the moment | 19:59 |
| jamesdenton | my main goal with that is just to see if Mellanox ASAP can offload firewall rules (sec grps) | 20:00 |
| spatel | damn! Mellanox | 20:02 |
| spatel | i thought you are doing simple packet rate testing but sounds like advance level of stuff | 20:02 |
| jamesdenton | just doing this: https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch-asap.html | 20:03 |
| spatel | I asked my manager to provide some hardware where i can do some DPDK testing so lets see but again same issue (because of holiday not getting enough time) | 20:03 |
| jamesdenton | the packet rate testing would be with trex or pktgen, and maybe dpdk within the DUT (using ASAP on compute) | 20:04 |
| jamesdenton | my dpdk numbers always look crappy, so i'm sure there's some tuning i'm missing and a general lack of understanding :D | 20:05 |
| spatel | very interesting.. doc | 20:06 |
| spatel | that is where i am struggling.. my DPDK number also looking crappy even after all possible tuning.. | 20:07 |
| spatel | last thing i would like to to is compile dpdk+ovs instead of using distro package | 20:08 |
| spatel | which model of Mallanox you have? | 20:08 |
| spatel | i may ask my company to buy couple of card for lab | 20:08 |
| jamesdenton | I have a couple of CX-4 Lx and some CX-6 dX | 20:10 |
| spatel | i would like to buy some.. | 20:11 |
| admin1 | what does " Future support is not guaranteed" mean there in the context ? | 20:11 |
| spatel | soon we are building new openstack for NVDIA / GPU and may need these kind of tech | 20:12 |
| spatel | jamesdenton does it offload packet processing on NIC correct? what about vRouter etc? | 20:12 |
| spatel | assuming it will handle by OpenFlow | 20:13 |
| jamesdenton | right, not offloaded AFAIK | 20:17 |
| spatel | hmm! sounds good | 20:20 |
| spatel | jamesdenton do you have 100G nic or 10g | 20:23 |
| spatel | looking at price its showing $1000 around for CX-6 DX | 20:23 |
| jamesdenton | in my lab it's mainly 10G w/ some 25G. i had 100G for a brief moment | 20:24 |
| jamesdenton | i think the ones i have are 25G | 20:24 |
| spatel | https://www.nvidia.com/en-us/networking/ethernet/connectx-6-lx/ | 20:25 |
| jamesdenton | yeah lemme double check | 20:25 |
| vakuznet | need help with rdo deps issue https://paste.openstack.org/show/811562/ | 20:26 |
| jamesdenton | spatel https://store.nvidia.com/en-us/networking/store/product/MCX621102AC-ADAT/nvidiamcx621102ac-adatconnectx-6dxenadaptercard25gbecryptoenabled/ | 20:26 |
| spatel | Oops!! still costly but let me see if i get approval to buy 2 card | 20:27 |
| spatel | vakuznet what centos and openstack version are you running? | 20:28 |
| vakuznet | centos8-ussuri | 20:28 |
| spatel | that is old.. | 20:29 |
| vakuznet | https://trunk.rdoproject.org/centos8-ussuri/deps/latest/noarch/ | 20:29 |
| vakuznet | yeah, trying to upgrade :) | 20:30 |
| jamesdenton | i seem to recall something here. | 20:32 |
| spatel | This is all centos8 mess | 20:32 |
| jamesdenton | you might need to add the centos-release-nfv-openvswitch repo? | 20:32 |
| spatel | yes | 20:33 |
| jamesdenton | thats where the ovs stuff lives now IIRC | 20:33 |
| spatel | vakuznet something like this - https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/vars/redhat.yml#L16 | 20:33 |
| spatel | ussuri is old code so that patch didn't back-ported yet | 20:34 |
| jamesdenton | yeah, but the task to bring that var in doesn't exist in ussuri | 20:34 |
| jamesdenton | https://opendev.org/openstack/openstack-ansible-os_neutron/commit/c3cb8525f43e7b43f428a36fc8be11d6ac21ad4f | 20:34 |
| spatel | I agreed its not going to be simple.. you need some hacks | 20:34 |
| jamesdenton | or just yum it up by hand | 20:35 |
| spatel | by hand is much easier and cheaper solution :) | 20:35 |
| vakuznet | so idea is to replace rdo-deps repo with centos-release-nfv-openvswitch repo? | 20:37 |
| jamesdenton | in addition to | 20:41 |
| vakuznet | ok, thank you. | 20:44 |
| mgariepy | tl;dr; ceph nautilus + python3 == no-go | 20:57 |
| jamesdenton | oh? | 21:05 |
| mgariepy | yeo | 21:09 |
| mgariepy | also this is broken ;D https://github.com/openstack/openstack-ansible-ceph_client/blob/stable/train/vars/debian.yml#L43-L46 | 21:09 |
| jrosser | mgariepy: you can fix it :) | 21:46 |
| mgariepy | i will | 21:46 |
| mgariepy | haha | 21:46 |
| mgariepy | i just need 1 day .. | 21:47 |
| mgariepy | fixing stuff between meetings first then fixing upstream :D | 21:47 |
| mgariepy | the hard thing is ceph upstream repo do not have python3-ceph. | 21:49 |
| mgariepy | only ubuntu uca have.. | 21:49 |
| mgariepy | which fails on depends.. | 21:49 |
| mgariepy | tl;dr; it's somewhat a mess. | 21:49 |
| *** ysandeep|out is now known as ysandeep | 23:51 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!