| opendevreview | Merged openstack/openstack-ansible stable/victoria: Bump OpenStack-Ansible Victoria https://review.opendev.org/c/openstack/openstack-ansible/+/815336 | 00:00 |
|---|---|---|
| kleini | I notice growing queues in vhosts /nova and /neutron named notifications_designate.info and notifications_designate.error. How do I find the consumers of these queues? | 07:33 |
| *** sshnaidm is now known as sshnaidm|afk | 07:45 | |
| noonedeadpunk | kleini: if you don't have designate deloyed - there's no consumers for these | 07:45 |
| noonedeadpunk | in prior releases we had a bug that designate notifications were pushed even when designate was not deployed | 07:46 |
| noonedeadpunk | maybe you're hitting that | 07:46 |
| noonedeadpunk | oh, well, for neutron this patch is present for ages actually | 07:48 |
| noonedeadpunk | but anyway, the idea is that if designate is defined in inventory, (nova|neutron)_designate_enabled are renderred as true, and then in nova/neutron.conf under oslo_messaging_notifications topic is set to notifications_designate | 07:50 |
| noonedeadpunk | this is needed for designate to create records to get resolvable hostnames or be able to provide records for floating ips iirc | 07:51 |
| kleini | I have designate deployed. So it is not consuming those queues. | 07:52 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add playbook to generate any user defined certificates https://review.opendev.org/c/openstack/openstack-ansible/+/816522 | 08:23 |
| *** sshnaidm|afk is now known as sshnaidm | 09:40 | |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova master: Enable TLS for live migrations https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/815224 | 09:41 |
| noonedeadpunk | kleini: oh, yeah, I guess we missed defining that https://opendev.org/openstack/openstack-ansible-os_designate/src/branch/master/templates/designate.conf.j2#L409 | 10:51 |
| noonedeadpunk | :( | 10:51 |
| kleini | Hmm, same for Nova. Will write the issue down in my TODO list and provide a patch, once it works for me. | 11:03 |
| kleini | noonedeadpunk: thanks for the pointer | 11:03 |
| noonedeadpunk | I guess the tricky thing there is that you might require domain_id | 11:05 |
| kleini | https://docs.openstack.org/designate/victoria/admin/samples/config.html designate configuration reference tells me, there should be zone_ids and not domain_ids. this makes more sense for me. | 11:28 |
| noonedeadpunk | yeah. I think we jsut haven't updated it for ages since it's commented out and has no effect | 11:46 |
| noonedeadpunk | and everybody just uses overrides there | 11:46 |
| noonedeadpunk | but I believe we should be smarter there | 11:46 |
| noonedeadpunk | I just don't have designate in prod now so I wasn't digging into it lately | 11:46 |
| kleini | https://paste.opendev.org/show/810867/ <- this is, what I have configured now, but designate-sink is still not reading those queues | 11:53 |
| noonedeadpunk | kleini: I think also vhost should be provided at least? | 12:17 |
| noonedeadpunk | because nova/neutron puts notification inside their vhosts | 12:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove note about metal/horizon compatability https://review.opendev.org/c/openstack/openstack-ansible/+/771573 | 12:23 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove note about metal/horizon compatability https://review.opendev.org/c/openstack/openstack-ansible/+/771573 | 12:23 |
| kleini | noonedeadpunk: How do I provide the vhost? | 12:30 |
| noonedeadpunk | that is damn good question.... | 12:37 |
| noonedeadpunk | so eventually that's the code https://opendev.org/openstack/designate/src/branch/master/designate/notification_handler/neutron.py but it does not have any connection details | 12:40 |
| noonedeadpunk | I wonder how designate folks see this working... | 12:41 |
| noonedeadpunk | As I actually thought it to be like implemented in ceilometer | 12:41 |
| noonedeadpunk | when ceilometer connects to other services with explicitly provided details | 12:42 |
| noonedeadpunk | And how designate aims to work if, let's say, neutron uses standalone rabbitmq cluster... | 12:42 |
| noonedeadpunk | so feels like these are things to ask a designate team... not sure how active they are though | 12:43 |
| noonedeadpunk | hm, so eventually, for notification for designate we need to provide quite different connection. ie set `nova_oslomsg_notify_vhost: /designate` | 12:47 |
| noonedeadpunk | kleini: oh, I guess this addresses the thing https://specs.openstack.org/openstack/oslo-specs/specs/victoria/support-transports-per-oslo-notifications.html | 12:49 |
| kleini | So, this needs to be changed on the Nova and Neutron side? | 12:50 |
| noonedeadpunk | yeah | 12:50 |
| noonedeadpunk | and that is not _that_ straightforward tbh | 12:51 |
| kleini | As I don't really need these notifications. I will disable them in Nova and Neutron for now. | 12:56 |
| opendevreview | Merged openstack/openstack-ansible-os_tempest stable/ussuri: Pin neutron-tempest-plugin to v1.6.0 https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/815631 | 13:22 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build stable/train: Set centos-7 jobs to non voting https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817219 | 14:08 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build stable/train: Workaround distro provided pip having old CA certs on centos-7 https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/816473 | 14:09 |
| johnsom | noonedeadpunk kleini Designate doesn’t used rabbit for integration with nova/neutron anymore. Neutron now talks directly to the designate API. | 14:10 |
| noonedeadpunk | oh.... | 14:11 |
| noonedeadpunk | then we need to do some clean-up | 14:12 |
| noonedeadpunk | didn't know that | 14:12 |
| johnsom | designate-sink is there for backward compatibility for one services using it | 14:12 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build stable/train: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817221 | 14:15 |
| opendevreview | Jonathan Rosser proposed openstack/ansible-role-python_venv_build stable/train: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817221 | 14:15 |
| noonedeadpunk | johnsom: and how nova is acknoledged that designate is here? As I can't find any config option for nova. I see extension for neutron though | 14:25 |
| noonedeadpunk | or it just auto discovers from catalog? | 14:26 |
| noonedeadpunk | as I'd expect to have some section with auth to it... | 14:27 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova master: Enable TLS for VNC from novncproxy to compute hosts https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/817222 | 14:39 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_nova master: Enable TLS for VNC from novncproxy to compute hosts https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/817222 | 14:42 |
| opendevreview | Aleksandr proposed openstack/openstack-ansible master: Avoiding of setup of Vault on containers hosts (Vault role support) https://review.opendev.org/c/openstack/openstack-ansible/+/800787 | 14:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add Vault role support https://review.opendev.org/c/openstack/openstack-ansible/+/800787 | 14:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-vault master: Initial commit to Vault role https://review.opendev.org/c/openstack/ansible-role-vault/+/800792 | 14:58 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Nov 9 15:00:28 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| admin1 | o/ | 15:00 |
| jrosser_ | o/ hello | 15:01 |
| damiandabrowski[m] | hey! | 15:01 |
| noonedeadpunk | I almost failed again with sumer time hehe | 15:01 |
| mgariepy | hoo. early ! | 15:03 |
| noonedeadpunk | #topic office hours | 15:04 |
| noonedeadpunk | I don't think we have any recent bugs, so jumping directly here | 15:04 |
| noonedeadpunk | I saw really great work regarding tls encryption for nova | 15:04 |
| noonedeadpunk | and even for VNC encryption - that's awesome | 15:05 |
| jrosser_ | not related directly to OSA but we are seeing leaking fd in nova-compute after wallaby upgrade | 15:05 |
| noonedeadpunk | I fully failed my part due to internal stuff that I couldn't put away... | 15:05 |
| noonedeadpunk | oh | 15:05 |
| noonedeadpunk | wow, you already upgraded! | 15:06 |
| jrosser_ | in case anyone else finds the same thing it looks like this https://bugs.launchpad.net/oslo.messaging/+bug/1949964 | 15:06 |
| * noonedeadpunk subscribed | 15:07 | |
| noonedeadpunk | interesting | 15:07 |
| jrosser_ | and yes for nova TLS i think the patches are really close | 15:07 |
| jrosser_ | we have it deployed in multinode lab | 15:07 |
| noonedeadpunk | may I dare to ask if you tested live migrations? | 15:07 |
| jrosser_ | it would be great to get more testing of this, in particular as it takes ansible hostname and nodename facts and uses them in the certificate | 15:08 |
| jrosser_ | and i think there are differences in the way people name their hosts, like fqdn or not | 15:08 |
| noonedeadpunk | Yeah, we have ppl who can test this I believe | 15:08 |
| jrosser_ | and this may interact with DNS blah blah and cause the cert verification to fail | 15:08 |
| noonedeadpunk | At least I got several requests to notify about having first beta of X for test | 15:08 |
| jrosser_ | we are currently testing internal VIP = https with cert from PKI role, and nova TLS | 15:09 |
| noonedeadpunk | and, we have mixed scenario of naming hosts in our sandbox :D | 15:09 |
| jrosser_ | migration looks good i think, was just trying to get james here in IRC | 15:09 |
| noonedeadpunk | so that might be interesting | 15:09 |
| jrosser_ | we were discussing what needs doing next here earlier today | 15:10 |
| jrosser_ | and there is cleanup of the nova SSH keys | 15:10 |
| noonedeadpunk | oh, so you also implemented tls ssh auth there? | 15:10 |
| jrosser_ | and also that we decide (?) that TLS is the only supported nova migration from now? | 15:10 |
| jrosser_ | well, i think now that there is no need for those keys at all | 15:11 |
| jrosser_ | unless i misunderstand how its working | 15:11 |
| noonedeadpunk | actually it might be that, yes | 15:11 |
| noonedeadpunk | since we set client certs there anyway | 15:11 |
| * jrosser_ waves to JamesGibo | 15:11 | |
| jrosser_ | james has done all this excellent work on nova TLS | 15:12 |
| noonedeadpunk | answering on if it's the only supported migration - I think yes | 15:12 |
| noonedeadpunk | At least I don't see other reall non-deprecated options | 15:12 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/ussuri: Set centos-7 jobs to non voting https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/816316 | 15:13 |
| noonedeadpunk | We might keep it for another release just in case though | 15:13 |
| jrosser_ | removing all the code for nova SSH keys would be great, but we can leave that for Y if we want to | 15:13 |
| noonedeadpunk | I think we should default to TLS now, with easy option for fallback | 15:14 |
| jrosser_ | it does kind of have to be all one way or the other, i think | 15:14 |
| jrosser_ | you can't have a mixed config across the compute nodes | 15:14 |
| noonedeadpunk | yeah, it can't be both | 15:14 |
| jrosser_ | is there anything else that we want to complete TLS/PKI wise for the X release? | 15:15 |
| noonedeadpunk | Well we talked about memcached | 15:16 |
| noonedeadpunk | But it's not a requirement for sure | 15:16 |
| jrosser_ | we did discuss a little about how to start transitioning the backend services to https, but that looks really quite "interesting" problem | 15:16 |
| jrosser_ | like how to do it without a huge downtime | 15:16 |
| noonedeadpunk | Oh, yes, that's interesting... I guess you can't have mix of backends? | 15:17 |
| noonedeadpunk | in terms of http/https | 15:18 |
| mgariepy | if the services listen both http and https, can we configure backend with both scheme in haproxy and confirm it work on https ? | 15:18 |
| mgariepy | or we need to move 1 backend at the time in haproxy over https.. | 15:18 |
| jrosser_ | it's not clear really | 15:19 |
| jrosser_ | as the haproxy play runs kind of first for all the services | 15:19 |
| JamesGibo | Hi, just caught up with meeting via irclogs! | 15:19 |
| mgariepy | why couldn't we have multiple time the backends in the haproxy backend list ? | 15:19 |
| noonedeadpunk | same host/same port? | 15:19 |
| mgariepy | it will be on a different port anyway. | 15:19 |
| noonedeadpunk | but port for services is the same | 15:20 |
| mgariepy | ho.. yep .. :/ | 15:20 |
| noonedeadpunk | I _think_ we can manage backends during runtime | 15:20 |
| noonedeadpunk | the same way we put them to maint? | 15:20 |
| jrosser_ | right so anyway - reason i bring this up is its quite a complex problem | 15:21 |
| noonedeadpunk | nah, module supports only enable/disable/drain I guess | 15:21 |
| jrosser_ | and we need to start thinking about it even if theres no answer right now | 15:21 |
| spatel | jrosser_ that rabbitMQ bug is interesting.. i haven't seen any behavior yet. | 15:21 |
| mgariepy | can we have 2 pools of backends ? | 15:22 |
| noonedeadpunk | Well, at least I'm think that you can pass to haproxy socket to drop a specific backend or add another one | 15:23 |
| noonedeadpunk | So even if there's no ansible module ready for that - that is not impossible | 15:23 |
| jrosser_ | the awkward part is that when all this is completed you want the https backend to be on the same well known port numbers as the http one used to be | 15:24 |
| noonedeadpunk | And we can write config after all services are reloaded. or actually trigger haproxy role for migration in each playbook, which would be pretty nasty I guess. | 15:24 |
| jrosser_ | so it feels like there are multiple phases involved | 15:25 |
| noonedeadpunk | yeah | 15:25 |
| jrosser_ | ok well maybe most important is testing the nova TLS stuff | 15:26 |
| noonedeadpunk | I suggest merge it to be able to test easily | 15:26 |
| noonedeadpunk | we can patch if afterwards anytime | 15:26 |
| jrosser_ | JamesGibo: we've tested migration is working? :) | 15:27 |
| JamesGibo | Yeah, it is working for us | 15:27 |
| JamesGibo | Using the HAproxy API to manage the backends is an intressting idea, i will have a think about that | 15:29 |
| jrosser_ | noonedeadpunk: how much of the hashicorp vault stuff would you like to get done for X? | 15:30 |
| jrosser_ | maybe we should start an etherpad with todo/patch links | 15:31 |
| noonedeadpunk | personally I don't care _that_ much:) | 15:31 |
| noonedeadpunk | as I don't have time for it at all. But folks eager to push stuff | 15:32 |
| noonedeadpunk | it's tough given their awareness of the project overall | 15:32 |
| noonedeadpunk | I think we can merge vault role and hopefully agree on "concept" patch | 15:33 |
| noonedeadpunk | and we can iterate on the vault to add internal storage and safe tokens storage then we have now | 15:34 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/ussuri: Workaround distro provided pip having old CA certs on centos-7 https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/816317 | 15:34 |
| noonedeadpunk | but at least not to get overwhelmed with depends-on there | 15:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/ussuri: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817250 | 15:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/ussuri: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817250 | 15:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build stable/ussuri: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817250 | 15:36 |
| noonedeadpunk | I think more important is to fix zun | 15:36 |
| * jrosser_ looks for andrewbonney | 15:36 | |
| noonedeadpunk | as I haven't return to it. And the question is in libcyrur that's they don't build for deb | 15:37 |
| andrewbonney | Didn't realise it was broken. I'll take a look tomorrow | 15:37 |
| noonedeadpunk | so options were either snap (fewwww) or build from source | 15:37 |
| jrosser_ | this is no deb for focal, or just new version theres no deb at all? | 15:37 |
| noonedeadpunk | no versions for deb at all | 15:37 |
| noonedeadpunk | *new | 15:37 |
| noonedeadpunk | well eventually we relied opensuse repo for deb which was a bit naive but worked | 15:38 |
| noonedeadpunk | oh, wait, I guess it was not kuryr but kata... | 15:39 |
| noonedeadpunk | yes, it was kata... | 15:40 |
| noonedeadpunk | so whole 2.0 version is not available for deb | 15:40 |
| noonedeadpunk | and to be specific - way was broken for debian only | 15:42 |
| noonedeadpunk | so we might set it as nv now I guess | 15:43 |
| andrewbonney | kata isn't required for zun to work, so in the worst case it could be disabled, but it would be nice to fix it | 15:43 |
| noonedeadpunk | but we still need to find the way to move forward | 15:43 |
| noonedeadpunk | Also, I asked damiandabrowski[m] to be another pair of eyes for https://etherpad.opendev.org/p/db_pool_calculations and help out with landing patches | 15:45 |
| noonedeadpunk | Would be great if we can get new numbers soon | 15:46 |
| noonedeadpunk | ok, awesome, thanks everyone for joining! | 15:47 |
| johnsom | noonedeadpunk I didn't want to bug you during the meeting, but DNS records are really only tied to neutron ports/fips/etc. so neutron handles interacting with designate on behalf of nova during the port plugs. There is no longer a direct link from nova to designate. | 15:56 |
| noonedeadpunk | aha, fair enough | 15:58 |
| noonedeadpunk | I just had some recallings that nova-metadata was pushing for sink or smth like that | 15:59 |
| noonedeadpunk | but yes, I agree that makes sense | 15:59 |
| noonedeadpunk | #endmeeting | 15:59 |
| opendevmeet | Meeting ended Tue Nov 9 15:59:49 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-11-09-15.00.html | 15:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-11-09-15.00.txt | 15:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-11-09-15.00.log.html | 15:59 |
| damiandabrowski[m] | yeah, i will look at db_pool_calculations hopefully this week | 16:00 |
| mgariepy | the number of worker per service seems to be on the low side for some services like neutron | 16:05 |
| mgariepy | keystone also. | 16:09 |
| mgariepy | on a smallish cloud with ~100 compute nodes i'm at ~900 connections | 16:16 |
| noonedeadpunk | well yes | 16:16 |
| mgariepy | and it's with minimal services | 16:16 |
| noonedeadpunk | and 90% of them are just sleeping | 16:17 |
| noonedeadpunk | and never used | 16:17 |
| mgariepy | i would say 99.5 but yes. | 16:17 |
| mgariepy | so we should lower the thread per service then ? | 16:17 |
| noonedeadpunk | The idea was to low max_pool_size | 16:18 |
| noonedeadpunk | and instead increase db_max_overflow | 16:18 |
| noonedeadpunk | because overflow steps in when pool_size is not enough | 16:19 |
| noonedeadpunk | and pool_size always fully used | 16:19 |
| noonedeadpunk | ie https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/786592 | 16:20 |
| mgariepy | ha ok. | 16:21 |
| spatel | noonedeadpunk around? | 16:57 |
| noonedeadpunk | yep | 16:59 |
| opendevreview | Merged openstack/openstack-ansible stable/ussuri: Fetch upper constraints file with curl rather than allow pip to download it https://review.opendev.org/c/openstack/openstack-ansible/+/815632 | 18:10 |
| spatel | noonedeadpunk :) are you around | 20:02 |
| spatel | if you don't tag me i won't able to see notification.. :( | 20:02 |
| spatel | anyway its late for you so will talk tomorrow | 20:02 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!