| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Add draining step for mysql https://review.opendev.org/c/openstack/kolla-ansible/+/951474 | 07:21 |
|---|---|---|
| opendevreview | Seunghun Lee proposed openstack/kayobe master: Make custom Gather facts run with default python for kayobe-target-venv https://review.opendev.org/c/openstack/kayobe/+/951425 | 08:03 |
| opendevreview | Christian Berendt proposed openstack/kolla stable/2024.2: [mariadb] Fix incremental backup using correct base dir https://review.opendev.org/c/openstack/kolla/+/951500 | 08:25 |
| opendevreview | Christian Berendt proposed openstack/kolla stable/2025.1: [mariadb] Fix incremental backup using correct base dir https://review.opendev.org/c/openstack/kolla/+/951501 | 08:25 |
| mnasiadka | Morning | 10:04 |
| mnasiadka | Might be a bit absent this week - have a week long company workshops | 10:05 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Add draining step for mysql https://review.opendev.org/c/openstack/kolla-ansible/+/951474 | 10:07 |
| mnasiadka | frickler, kevko: should I cancel the Weds meeting? | 10:13 |
| frickler | mnasiadka: yes, that would be good, I have a conflict, too | 13:07 |
| kevko | it's ok | 13:19 |
| kevko | frickler: can u hold one job for me ? | 13:19 |
| kevko | so i can connect inside ? | 13:19 |
| kevko | please | 13:19 |
| frickler | kevko: sure, which one? | 13:21 |
| frickler | kevko: also did you already look into backporting https://review.opendev.org/c/openstack/kolla/+/950948 ? or should I do it myself? | 13:22 |
| kevko | frickler: i gave +2 | 13:24 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [DNM] Add draining step for mysql https://review.opendev.org/c/openstack/kolla-ansible/+/951474 | 13:25 |
| kevko | frickler: ^^ this one ... I really need to check what's going on with that mariadb instance ..as proxysql really can't connect .. I wrote also "draining" the node ..turned off mysql check as it's actually not needed for one node .... but proxysql can't connext after container restart ...and I really dont know why | 13:26 |
| frickler | kevko: oh, I missed that christian already proposed these, have been mostly offline today. thx anyway | 13:26 |
| frickler | kevko: so that's the patch, but which job, *-mariadb? note that I can only hold jobs that are failing | 13:27 |
| kevko | frickler: kolla-ansible-debian-upgrade of course :) | 13:27 |
| kevko | frickler: that one | 13:27 |
| frickler | ah, ok, will do | 13:28 |
| kevko | frickler: thank you very much | 13:28 |
| frickler | kevko: can you remind me of your ssh key, please? | 14:21 |
| opendevreview | Seunghun Lee proposed openstack/kayobe master: Make custom Gather facts run with default python for kayobe-target-venv https://review.opendev.org/c/openstack/kayobe/+/951425 | 14:34 |
| kevko | frickler: yep | 14:37 |
| kevko | frickler: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKWnPCazqD5k1tJObnp7vpNCih9csNoVoGQg0t3wzQnb7ClHiWwiQ2La/UbI3VUuvxRIb+cAznXO9ZziM3QCtMfH+Ac2Z9Tl6s1AhGDuuRsveUvN5YIPqCNFE8QDbi9M2u46a1Lkwog/kTdPkZ2qh4z1jugFXQ3DDn/P7ORCiyYCEaU6HwoEzPWF0W8f1sAVcCXqJukrwa79QQJ4LkH//L/0vKV3s81+UC7L0LGoTiaHWdgFNDZDaGSZK5848zVReI47n1AlU7Rte52stfsSHBNFYTL2j2ljRdQe7NSUjYLUnp | 14:37 |
| kevko | 8iTa6xSUZ8TQajoqCwRIUvS8qjrzR0ro/x0Pn1n7 michal@michal-desktop | 14:37 |
| kevko | frickler: it will fail in a while | 14:56 |
| kevko | frickler: I hope this time it won't be a success :D | 15:03 |
| kevko | frickler: it's failing ..currectly logs are uploading ...so ? can u give me an ip ? | 15:17 |
| kevko | ok, i probably have | 15:38 |
| kevko | frickler: hmm, this is not about proxysql fail ...there is mariadb container down ...without reason :/ | 15:46 |
| kevko | root@primary:~# docker ps -a | grep maria | 15:46 |
| kevko | 85472f9a9e92 primary:4000/lokolla/mariadb-server:change_951474 "dumb-init -- kolla_…" About an hour ago Exited (0) 32 minutes ago mariadb | 15:46 |
| mnasiadka | Something surely stopped it | 16:01 |
| kevko | mnasiadka: what ? | 16:02 |
| kevko | who ? | 16:02 |
| kevko | mnasiadka: i am logged into primary ... i started mariadb and restart proxysql ( proxysql has a watchdog timer ...if there is reaaallly long time non-responsive thread ... it's restarted ..but as it is in container ..it's dead) | 16:04 |
| kevko | and working | 16:04 |
| kevko | mnasiadka: okay, debian upgrade job is just out of memory ..no bug in kolla-ansible .... | 16:06 |
| kevko | mnasiadka frickler: 2025-06-02T15:12:24.501268+00:00 debian kernel: [ 5520.861831] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-85472f9a9e92bea414084ca4679e4dcc5b209a1120fdc7ccc2b2223190f3a7e9.scope,task=mariadbd,pid=219306,uid=42434 | 16:06 |
| mnasiadka | Well, it wasn’t before - what changed? | 16:07 |
| mnasiadka | And why mariadb is taking that all memory? | 16:07 |
| kevko | well, I think it's just small testing server | 16:08 |
| kevko | mnasiadka: as we discussed before ... | 16:09 |
| kevko | I can see 8cpus and 8gb ram ... | 16:09 |
| kevko | which is not enough I would say | 16:09 |
| mnasiadka | It was enough for long time | 16:09 |
| mnasiadka | So stop complaining :) | 16:09 |
| kevko | mnasiadka: I started mariadb and proxysql ...everything is running | 16:09 |
| kevko | mnasiadka: 7.01 from 7.75 is taken | 16:10 |
| mnasiadka | Well, as I said - something changed | 16:16 |
| mnasiadka | Maybe proxysql is taking too much memory? | 16:16 |
| kevko | https://paste.openstack.org/show/bryRjp7mtIEjaPDivuXk/ | 16:20 |
| kevko | mnasiadka: it's not ...it takes 15 mb :D | 16:39 |
| mnasiadka | kevko: I have a feeling dockerd is taking too much memory ;) | 16:44 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Override MariaDB memory usage for CI https://review.opendev.org/c/openstack/kolla-ansible/+/951575 | 16:47 |
| kevko | mnasiadka: nope, it's mariadb ... lets see ^^ | 17:15 |
| frickler | oh, oom, nice | 17:32 |
| opendevreview | Merged openstack/kayobe master: Make custom Gather facts run with default python for kayobe-target-venv https://review.opendev.org/c/openstack/kayobe/+/951425 | 18:49 |
| opendevreview | Hao Wang proposed openstack/kolla master: Support building zaqar container image in Kolla https://review.opendev.org/c/openstack/kolla/+/887560 | 19:15 |
| kevko | frickler, mnasiadka: So what, can we merge override just for CI to lower innodb buffer pool size ... I think it's not clever to set it to half of host .... This is failed job in another review | 19:45 |
| kevko | https://00b9edcb114e0ac8e05a-b611493cf8fd4459149d00d14c03b361.ssl.cf5.rackcdn.com/openstack/bb2da432ccc4410dbe916aa989c060bc/primary/logs/kolla_configs/mariadb/galera.cnf ...innodb_buffer_pool_size = 3176M while whole host has 8000 gb without swap | 19:45 |
| mnasiadka | Fine by me - if something really small works - then why not | 19:46 |
| kevko | mnasiadka: What I mean is that innodb_buffer_pool_size is simply memory allocated at MariaDB startup, and it doesn’t make sense to have it set by default to half of the entire RAM in our CI. Maybe we can open a discussion about changing the default... or maybe just write in the documentation that users should set it themselves after running MySQL | 19:52 |
| kevko | Tuner (actually, this value can already be overridden via a global variable even now ). | 19:52 |
| kevko | mnasiadka: Considering the CI environment — where everything runs on AIO, including libvirt — I think it makes sense to tweak this setting for tests. So I’ll merge it, OK? Can you at least give me a +2 ? It will unblock CI regarding this job .... | 19:52 |
| opendevreview | Merged openstack/kolla stable/2025.1: [mariadb] Fix incremental backup using correct base dir https://review.opendev.org/c/openstack/kolla/+/951501 | 20:22 |
| mnasiadka | kevko: just override that in tests/templates/globals-default.j2 for now | 20:34 |
| kevko | mnasiadka: hmm , but i also lower other stuff ... | 20:35 |
| opendevreview | Merged openstack/kolla stable/2024.2: [mariadb] Fix incremental backup using correct base dir https://review.opendev.org/c/openstack/kolla/+/951500 | 20:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 20:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 20:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 20:48 |
| opendevreview | Hao Wang proposed openstack/kolla master: Support building zaqar container image in Kolla https://review.opendev.org/c/openstack/kolla/+/887560 | 20:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 20:55 |
| opendevreview | Hao Wang proposed openstack/kolla-ansible master: Support deploy Zaqar with kolla-ansible https://review.opendev.org/c/openstack/kolla-ansible/+/892615 | 21:09 |
| kevko | mnasiadka: okay, it's not configurable ...as it's set in template ... | 21:26 |
| kevko | mnasiadka: https://paste.openstack.org/show/b4lMVKtTWVJIYWwn1mIS/ | 21:26 |
| mnasiadka | well, then it makes sense to be configurable, or just supply a galera.cnf in CI templates | 21:27 |
| kevko | mnasiadka: that what I've done :D ... https://review.opendev.org/c/openstack/kolla-ansible/+/951575 << | 21:28 |
| kevko | mnasiadka: add galera.cnf config override for CI | 21:28 |
| kevko | mnasiadka: it's configurable already ...via config override ...no need to add ansible variable i thiknk... | 21:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 22:17 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 22:25 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Set previous_release to 2025.1 https://review.opendev.org/c/openstack/kolla-ansible/+/950668 | 22:26 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Enable virtualenv testing for Debian/Ubuntu https://review.opendev.org/c/openstack/kolla-ansible/+/950306 | 22:26 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add bifrost to baremetal https://review.opendev.org/c/openstack/kolla-ansible/+/950743 | 22:26 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Check drain while upgrading rabbitmq https://review.opendev.org/c/openstack/kolla-ansible/+/951133 | 22:26 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add default_hostgroup for users in proyxsql https://review.opendev.org/c/openstack/kolla-ansible/+/951599 | 22:26 |
| opendevreview | Merged openstack/kolla master: rabbitmq: Update to 4.1 https://review.opendev.org/c/openstack/kolla/+/950404 | 23:16 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!