| *** arxcruz is now known as arxcruz|rover | 08:52 | |
| noonedeadpunk | can I get another review on https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 ? | 09:25 |
|---|---|---|
| noonedeadpunk | I guess that's the last sufficient change we want to bring to X | 09:25 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_cinder master: WIP: test fix for zun volume tempest failures https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/819141 | 09:44 |
| jrosser | noonedeadpunk: i think that this breaks things https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/816739/1/tasks/main.yml#b58 | 09:57 |
| jrosser | oh, the whole change actually not that specific line | 09:58 |
| jrosser | like with /var/lock/cinder vs. /var/lock/cinder-volume getting mixed up | 09:58 |
| jrosser | -EMEETINGS for a while - interested to know what you think | 09:59 |
| noonedeadpunk | jrosser: oh, well, it does. but I'm not sure what stands behind sharing lock/run dir for services? I mean - shouldn't it be cleaner when each service has it's own lock dir and not interferring? As before that cinder-api, cinder-scheduler, cinder-backup and cinder-volume used same lock/run directories | 10:30 |
| noonedeadpunk | So I guess I aimed to add more isolation and simplify logic a bit, but I'm not sure how it actually broke things? | 10:31 |
| noonedeadpunk | Also I didn't want to patch all roles to update directory path back then... | 10:32 |
| noonedeadpunk | But we can revert this as might be my assumptions why we did the way we did were wrong | 10:33 |
| noonedeadpunk | I don't see smth obvious, right? | 10:41 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Add [nova] section to ironic.conf https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/818115 | 10:42 |
| noonedeadpunk | oh, we need to set this up for slices.... | 10:44 |
| noonedeadpunk | or? | 10:56 |
| noonedeadpunk | hm.... | 11:07 |
| noonedeadpunk | ok, so it depends on setting of oslo_concurrency in each service | 11:08 |
| noonedeadpunk | which yeah, breaks things currently :( | 11:09 |
| noonedeadpunk | so we have for example `cinder_lock_path` but it's not used for `systemd_lock_path` https://opendev.org/openstack/openstack-ansible-os_cinder/src/branch/master/tasks/cinder_install.yml#L51 | 11:12 |
| noonedeadpunk | so yeah, I missed oslo part in the patch ( | 11:15 |
| noonedeadpunk | damn... | 11:21 |
| noonedeadpunk | if we just revert, this assumption is just wrong https://opendev.org/openstack/ansible-role-systemd_service/src/branch/stable/wallaby/templates/systemd-tmpfiles.j2#L5 | 11:23 |
| noonedeadpunk | as currently lock path should be /run/lock/ and with that replace it would be /run/run ? | 11:24 |
| noonedeadpunk | I'd say we need somehow to setup lock/run based on service slice name. The problem is that we need to have lock path defined inside the roles while not making weird assumptions in systemd-service role... | 11:33 |
| noonedeadpunk | We for sure need to revert patch you mentioned. The question is though if we should also revert https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/816735/2 | 11:33 |
| noonedeadpunk | as it does not make much sense | 11:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Use slice name for lock/run by default https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/819298 | 11:55 |
| opendevreview | Merged openstack/openstack-ansible master: Deprecate OVN-related haproxy configuration https://review.opendev.org/c/openstack/openstack-ansible/+/813858 | 11:59 |
| opendevreview | Merged openstack/openstack-ansible stable/victoria: Update functional test requirements url https://review.opendev.org/c/openstack/openstack-ansible/+/819015 | 11:59 |
| opendevreview | Merged openstack/openstack-ansible stable/victoria: Set default for octavia_barbican_enabled https://review.opendev.org/c/openstack/openstack-ansible/+/769179 | 11:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Globally define systemd_lock_dir https://review.opendev.org/c/openstack/openstack-ansible/+/819300 | 11:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/819304 | 12:09 |
| noonedeadpunk | jrosser: does this make any sense https://review.opendev.org/q/topic:%22systemd_run_dir%22+(status:open%20OR%20status:merged) ? | 12:10 |
| noonedeadpunk | as I tbh not sure how to act better here... | 12:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Globally define systemd_lock_dir https://review.opendev.org/c/openstack/openstack-ansible/+/819300 | 12:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/819304 | 12:13 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts stable/ussuri: Revert "Add CentOS 8.4 support" https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818485 | 13:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Drop designate notifications topic https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/819314 | 13:50 |
| damiandabrowski[m] | hi guys, today I'm working on SQLAlchemy/oslo.db pooling and I started to wonder for how long we should keep inactive mysql connections(a.k.a wait_timeout/connection_recycle_time).... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/TVlhlMHfTrObOzynMQruMQab) | 14:08 |
| noonedeadpunk | Hm, is there any reason for 1m sleeping connection at all then? | 14:12 |
| damiandabrowski[m] | with pooling, we are able to re-use these connections. It's about performance | 14:19 |
| damiandabrowski[m] | but if connection wasn't re-used within 1 minute, it shouldn't be a problem to drop it and create a new one when needed IMO | 14:19 |
| noonedeadpunk | well connections that are alive for 1m are unlikely to have a chance to be reused? | 14:21 |
| noonedeadpunk | So in most cases new ones will be spawned anyway? | 14:21 |
| noonedeadpunk | jrosser: mgariepy wdyt ^ | 14:21 |
| noonedeadpunk | I'd say it's worth to reduce to smth like 10m | 14:22 |
| jrosser | feels like we are always going to need 100% headroom in max_connections for when an haproxy failover happens? | 14:25 |
| damiandabrowski[m] | i think they will be re-used at some point, but keeping the connection open for let's say 30min only because it may be re-used probably doesn't make any sense and it's better to close it and create a new one when needed | 14:25 |
| noonedeadpunk | well, re-using opened connection is the way faster | 14:26 |
| noonedeadpunk | it's the reason why they exist at the first place | 14:26 |
| noonedeadpunk | re-creating them too fast I think creates other overhead | 14:27 |
| noonedeadpunk | but I'm not huge expert tbh | 14:27 |
| noonedeadpunk | jrosser: I tend to agree here | 14:27 |
| noonedeadpunk | But I believe damiandabrowski[m] was considering other changes to pooling as well :p | 14:28 |
| jrosser | yeah | 14:28 |
| jrosser | does anything actively make new connections? like threads starting/stopping dynamically? | 14:28 |
| noonedeadpunk | But it's probably more if you do haproxy restart 3 times in an hour you would need 300% or smth like that? | 14:29 |
| jrosser | i think a restart would drop the haproxy<>galera connections | 14:29 |
| damiandabrowski[m] | jrosser: it's about how many keepalived failovers may occur within wait_timeout. But keeping a 100% headroom for max_connections with a relatively small wait_timeout should be fine | 14:30 |
| damiandabrowski[m] | graceful restart will drop these connections, but for ex. powercut won't | 14:30 |
| jrosser | right | 14:30 |
| damiandabrowski[m] | regarding to Your question about making new connections: oslo.db is responsible for that, these 3 variables are the most important:... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/ABOyPztlSvmsBDQeYBEKypFk) | 14:32 |
| noonedeadpunk | So imo we shouldn't set this lower then 10 mins | 14:33 |
| noonedeadpunk | but would be great to hear other opinions:) | 14:41 |
| jrosser | feel like i'm not really understanding much of this, isnt wait_timeout a setting on the db rather than on the client | 14:45 |
| jrosser | and ultimately we need a consistent / sensible set of things on mariadb/haproxy/oslo.db together | 14:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Globally define systemd_lock_dir https://review.opendev.org/c/openstack/openstack-ansible/+/819300 | 14:51 |
| damiandabrowski[m] | i just realized that re-using connections is not only about saving time for establishing a new connection - it's also about persisting buffers(like read_buffer_size) | 14:55 |
| damiandabrowski[m] | so yeah, 10 minutes looks reasonable | 14:55 |
| damiandabrowski[m] | i already spent some time on galera/oslo.db connection pooling & limits and i have some idea in my mind, I'll write a change draft soon | 14:57 |
| noonedeadpunk | awesome! | 15:01 |
| jrosser | andrew wrote an etherpad about this too, which it could be quite useful to turn into documentation, given just how many moving parts there are here | 15:09 |
| damiandabrowski[m] | that's right, i have it https://etherpad.opendev.org/p/db_pool_calculations | 15:34 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_rally master: Install PyMySQL as rally commands may not work without it https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/819348 | 15:51 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts stable/victoria: Revert "Add CentOS 8.4 support" https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818486 | 16:16 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!