| *** bbezak is now known as Guest17269 | 00:00 | |
| opendevreview | Verification of a change to openstack/kayobe stable/2025.1 failed: Make custom Gather facts run with default python for kayobe-target-venv https://review.opendev.org/c/openstack/kayobe/+/951720 | 02:46 |
|---|---|---|
| opendevreview | Dai Dang Van proposed openstack/kolla-ansible master: Enable unified alerting for Grafana https://review.opendev.org/c/openstack/kolla-ansible/+/949114 | 03:24 |
| opendevreview | Dai Dang Van proposed openstack/kolla-ansible master: Add grafana native metrics target https://review.opendev.org/c/openstack/kolla-ansible/+/949024 | 03:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 05:45 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26 https://review.opendev.org/c/openstack/kolla-ansible/+/937242 | 05:54 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26 https://review.opendev.org/c/openstack/kolla-ansible/+/937242 | 05:54 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ansible-lint: Make roles passable with shared profile https://review.opendev.org/c/openstack/kolla-ansible/+/920159 | 05:54 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26 https://review.opendev.org/c/openstack/kolla-ansible/+/937242 | 05:55 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26 https://review.opendev.org/c/openstack/kolla-ansible/+/937242 | 05:55 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/946014 | 05:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: octavia: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/946010 | 06:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add ceilometer scenario https://review.opendev.org/c/openstack/kolla-ansible/+/944276 | 06:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add ceilometer scenario https://review.opendev.org/c/openstack/kolla-ansible/+/944276 | 06:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: gnocchi: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/940101 | 06:04 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: gnocchi: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/940101 | 06:04 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: gnocchi: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/940101 | 06:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add ceilometer scenario https://review.opendev.org/c/openstack/kolla-ansible/+/944276 | 06:06 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: gnocchi: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/940101 | 06:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: cli: Allow for missing globals.yml when globals.d exists https://review.opendev.org/c/openstack/kolla-ansible/+/940388 | 06:12 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simply globals-default by moving to globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/940320 | 06:12 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simply globals-default by moving to globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/940320 | 06:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simply globals-default by moving to globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/940320 | 06:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/940320 | 06:14 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/940320 | 06:14 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/940320 | 06:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/940320 | 06:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/940320 | 06:16 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/940320 | 06:16 |
| mnasiadka | oh boy | 06:18 |
| kevko | :D | 06:25 |
| kevko | mnasiadka: Testing backports script or what :D ? | 06:26 |
| mnasiadka | naah, tried to be lazy and click my way in Gerrit | 06:27 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: masakari: Add support for using uWSGI https://review.opendev.org/c/openstack/kolla-ansible/+/951115 | 06:34 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26 https://review.opendev.org/c/openstack/kolla-ansible/+/937242 | 06:38 |
| kevko | :D | 06:38 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: Use latest cirros image https://review.opendev.org/c/openstack/kayobe/+/931685 | 07:28 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Fix sanity check in proxysql https://review.opendev.org/c/openstack/kolla/+/951741 | 07:43 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard https://review.opendev.org/c/openstack/kolla-ansible/+/951599 | 07:47 |
| osmanlicilegi | morning | 07:55 |
| opendevreview | Dai Dang Van proposed openstack/kolla-ansible master: Add grafana native metrics target https://review.opendev.org/c/openstack/kolla-ansible/+/949024 | 08:17 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard https://review.opendev.org/c/openstack/kolla-ansible/+/951599 | 08:17 |
| kevko | frickler: I've addressed your comments ...please can u check if you are OK now ? | 08:17 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Fix sanity check in proxysql https://review.opendev.org/c/openstack/kolla/+/951741 | 08:21 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Fix ProxySQL rule sanity check to support username-based routing https://review.opendev.org/c/openstack/kolla/+/951741 | 08:26 |
| frickler | kevko: thx, I'll have another look later, I'd have given you a +2 this morning already if it hadn't been about your comment that you wanted to do further improvements | 08:28 |
| opendevreview | Dai Dang Van proposed openstack/kolla-ansible master: Enable unified alerting for Grafana https://review.opendev.org/c/openstack/kolla-ansible/+/949114 | 08:31 |
| kevko | frickler: thank you, well, another improvent was add another rule for username based routing (not only default_hostgroup for mysql_users, but also based on user, schemaname in mysql_query_rules ) | 08:44 |
| kevko | frickler: thank you, well, another improvent was add another rule for username based routing (not only default_hostgroup for mysql_users, but also based on user, schemaname in mysql_query_rules ) ...paste here https://paste.openstack.org/show/bbwhMsoetPpH92zCWfl7/ | 08:50 |
| frickler | kevko: yeah, thx, this sure does look reasonable, one day I need to dig into how proxysql really works, for now I'll just trust your testing | 08:59 |
| mnasiadka | frickler: wondering - we stopped using kolla-ansible-core Gerrit group - but it still has members - should we clean it up? | 08:59 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Replace outdated core team list by Gerrit links https://review.opendev.org/c/openstack/kolla-ansible/+/951015 | 09:00 |
| frickler | mnasiadka: good point, iirc there was also a way to retire these by either renaming or hiding them in gerrit, need to check with fungi for details | 09:04 |
| mnasiadka | frickler: thanks :) | 09:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Introduce docker_image_url variable https://review.opendev.org/c/openstack/kolla-ansible/+/951747 | 09:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ansible-lint: Fix yaml[octal-values] https://review.opendev.org/c/openstack/kolla-ansible/+/951748 | 09:34 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26 https://review.opendev.org/c/openstack/kolla-ansible/+/937242 | 09:34 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ansible-lint: Fix yaml[octal-values] https://review.opendev.org/c/openstack/kolla-ansible/+/951748 | 09:35 |
| *** mikal4 is now known as mikal | 09:42 | |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Add centos10 job https://review.opendev.org/c/openstack/kolla-ansible/+/951751 | 09:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 09:49 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard https://review.opendev.org/c/openstack/kolla-ansible/+/951599 | 09:51 |
| kevko | mnasiadka frickler: As I'm digging through the Kolla-Ansible code and working on stabilizing the upgrade process for RabbitMQ and MariaDB, and also messing around with ProxySQL while reading up a lot on these topics... I think we have a misconfigured oslo.db connection pooling. believe this because, historically, there was an issue with skipping | 09:59 |
| kevko | the VIP, which left pre-established connection pools dead... and as a result, pooling was effectively disabled in Kolla and a recycle of 10 seconds was configured. But this is fundamentally wrong — OpenStack makes a ton of small queries like SELECT * FROM something LIMIT 1; and similar... In a large stack, every single connection is quite | 09:59 |
| kevko | expensive, especially since each one initiates a new TCP connection unnecessarily. | 09:59 |
| mnasiadka | it was some oslo.db bug most probably, check the historical commits? they should have some information | 09:59 |
| kevko | mnasiadka frickler: problem with connections I solved historically with https://review.opendev.org/c/openstack/kolla-ansible/+/777772 << | 10:00 |
| kevko | mnasiadka frickler: https://bugs.launchpad.net/kolla-ansible/+bug/1917068 | 10:00 |
| kevko | mnasiadka frickler: you thought it was ...but actually It was just workarounded by https://review.opendev.org/q/Ib6a62d4428db9b95569314084090472870417f3d | 10:01 |
| mnasiadka | well, in mariadb directly via haproxy probably it makes sense | 10:02 |
| mnasiadka | wondering if we should not think about getting rid of not using proxysql | 10:02 |
| kevko | mnasiadka: First of all, I think this was just a small band-aid on a bigger wound — because what used to be a major freeze turned into a smaller one that now takes up to 10 seconds to resolve... And secondly, we had to set, for example, max_overflow, which creates connection pools beyond the limits configured in oslo.db... | 10:02 |
| kevko | mnasiadka: proxysql IS already default :) | 10:03 |
| mnasiadka | is default, but we still sort of support the old mode | 10:03 |
| kevko | mnasiadka: but this is not about proxysql ...it was about VIP switch ... | 10:03 |
| mnasiadka | if somebody is persistent | 10:03 |
| mnasiadka | although we don't test it | 10:03 |
| kevko | mnasiadka: as I said ... I've seen this while haproxy was used | 10:04 |
| kevko | mnasiadka: i also have a video :D ..wait a minute | 10:04 |
| kevko | maybe ... | 10:04 |
| kevko | mnasiadka: check https://bugs.launchpad.net/kolla-ansible/+bug/1917068 << my comments and also confirmation for example from chalansonnet (schalans) wrote on 2021-03-01: << | 10:08 |
| kevko | mnasiadka: and I am 100% pretty sure this can be set again to use pooling ...for both proxysql/haproxy usage | 10:11 |
| opendevreview | Verification of a change to openstack/kayobe stable/2025.1 failed: Make custom Gather facts run with default python for kayobe-target-venv https://review.opendev.org/c/openstack/kayobe/+/951720 | 10:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Add centos10 job https://review.opendev.org/c/openstack/kolla-ansible/+/951751 | 10:51 |
| kevko | haha, that 8gb node for AIO set on primary is not enough i would say :D now OOM killed VM :D | 11:42 |
| kevko | 2025-06-04T11:39:04.159421+00:00 debian kernel: [ 7108.969825] Out of memory: Killed process 160427 (qemu-system-x86) total-vm:3542952kB, anon-rss:664600kB, file-rss:0kB, shmem-rss:0kB, UID:42436 pgtables:1772kB oom_score_adj:0 | 11:42 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 11:44 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Use smaller flavor for CI https://review.opendev.org/c/openstack/kolla-ansible/+/951758 | 11:58 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard https://review.opendev.org/c/openstack/kolla-ansible/+/951599 | 11:59 |
| fungi | frickler: osuosl got sufficient funding, but can always use more: https://lists.osuosl.org/pipermail/hosting/2025-May/000644.html | 12:29 |
| fungi | oops, sorry, that was meant for another channel | 12:30 |
| fungi | frickler: mnasiadka: yeah, a gerrit admin can mark groups manually as hidden from th group list | 12:32 |
| fungi | i usually also empty them and rename them just to be thorough | 12:32 |
| mnasiadka | fungi: can you or frickler just do that so people don't get confused? :) | 12:47 |
| fungi | sure | 13:41 |
| fungi | mnasiadka: frickler: done | 13:45 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard https://review.opendev.org/c/openstack/kolla-ansible/+/951599 | 13:53 |
| *** Guest17269 is now known as bbezak | 13:54 | |
| kevko | Hi bbezak, review this ? https://review.opendev.org/c/openstack/kolla/+/951741 ? | 13:55 |
| kevko | can u please ? | 13:55 |
| mnasiadka | fungi: thank you | 13:59 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard https://review.opendev.org/c/openstack/kolla-ansible/+/951599 | 14:04 |
| bbezak | kevko: you've got +8 now | 14:05 |
| kevko | bbezak thanks :) | 14:14 |
| kevko | mnasiadka also thanks 👍 | 14:16 |
| mnasiadka | No problem ;) | 14:16 |
| kevko | mnasiadka: btw, i will have set of important patches for mariadb and rabbitmq ...especially while upgrading .... | 14:29 |
| kevko | mnasiadka: https://review.opendev.org/c/openstack/kolla-ansible/+/951133 | 14:29 |
| kevko | mnasiadka: and https://review.opendev.org/c/openstack/kolla-ansible/+/951474 | 14:29 |
| kevko | and depends on | 14:30 |
| kevko | it's working already ..but i need to amend scripts and make it bulletproof | 14:30 |
| kevko | especially rabbitmq is needed as community drain just turn on node to maintenance ...and don't wait for leader move :D :D :D | 14:30 |
| kevko | and in a sec we are maintenance second node and then third :D | 14:31 |
| kevko | which is bad of course :D | 14:31 |
| opendevreview | Merged openstack/kolla master: Fix ProxySQL rule sanity check to support username-based routing https://review.opendev.org/c/openstack/kolla/+/951741 | 15:50 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 16:20 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: ansible-lint: Fix jinja[spacing] https://review.opendev.org/c/openstack/kayobe/+/951790 | 16:38 |
| kevko | mnasiadka: frickler: we need to bump AIO jobs to something more than 8cpu 8gb ram I would say ..it's unstable ..sometimes it works sometimes it's not :/ | 17:18 |
| opendevreview | Verification of a change to openstack/kayobe stable/2025.1 failed: Make custom Gather facts run with default python for kayobe-target-venv https://review.opendev.org/c/openstack/kayobe/+/951720 | 17:18 |
| mnasiadka | kevko: there is nothing bigger than 8gb | 17:19 |
| frickler | well there is, but not in big quantities. also things will get better in the future with the nodepool-in-zuul feature, but no reliable timeframe for that yet | 17:26 |
| frickler | so for now, better look into reducing job size or making them multinode if needed? | 17:27 |
| mnasiadka | I still would like to understand what is the problem and why it doesn’t happen on all distros | 17:37 |
| kevko | mnasiadka: well, ubuntu, rocky can have different kernel tunning, differnet malloc, glibc... whatever ... | 20:41 |
| kevko | mnasiadka: i will try to run shell script with docker stats + top + some ps aux --sort=-rss or whatever in background and log into /tmp/logs/whatever/ram_consuption ...every 10 seconds for example ...or what ... | 20:42 |
| kevko | btw ...if some kolla change is merged ...but kolla-ansible job has still depends-on ...but kolla change is merged in master ...is image built before run or not ...because it looks like it's not :) | 20:43 |
| *** mikal8 is now known as mikal | 21:10 | |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 22:49 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!