| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: octavia: Fix hm-interface after bumping openstack collection https://review.opendev.org/c/openstack/kolla-ansible/+/878209 | 07:03 |
|---|---|---|
| mnasiadka | frickler: should work now | 07:03 |
| frickler | mnasiadka: hmm, did they drop os_security_group_info from the collection again? this was working earlier, wasn't it? | 07:10 |
| mnasiadka | frickler: have no clue, but if os_security_group works now, no need to change - I'll make it better in a followup | 07:11 |
| frickler | ack | 07:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/880809 | 08:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/880809 | 08:05 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/zed: Update notes about CentOS support https://review.opendev.org/c/openstack/kolla-ansible/+/880839 | 08:25 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/zed: Add precheck to fail if RabbitMQ HA needs configuring https://review.opendev.org/c/openstack/kolla-ansible/+/880840 | 08:26 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/zed: Add precheck to fail if RabbitMQ HA needs configuring https://review.opendev.org/c/openstack/kolla-ansible/+/880840 | 08:27 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/yoga: Add precheck to fail if RabbitMQ HA needs configuring https://review.opendev.org/c/openstack/kolla-ansible/+/880841 | 08:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Bump up etcd, lego and prometheus components versions to latest https://review.opendev.org/c/openstack/kolla/+/880920 | 08:27 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/yoga: Add precheck to fail if RabbitMQ HA needs configuring https://review.opendev.org/c/openstack/kolla-ansible/+/880841 | 08:28 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible stable/xena: Add precheck to fail if RabbitMQ HA needs configuring https://review.opendev.org/c/openstack/kolla-ansible/+/880842 | 08:28 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: cli: fix find globals.d https://review.opendev.org/c/openstack/kolla-ansible/+/880886 | 08:50 |
| opendevreview | Merged openstack/kayobe master: Allow to use own repos.y[a]ml file to build containers https://review.opendev.org/c/openstack/kayobe/+/845338 | 08:53 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/zed: Allow to use own repos.y[a]ml file to build containers https://review.opendev.org/c/openstack/kayobe/+/880923 | 08:53 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/yoga: Allow to use own repos.y[a]ml file to build containers https://review.opendev.org/c/openstack/kayobe/+/880924 | 08:53 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe stable/xena: Allow to use own repos.y[a]ml file to build containers https://review.opendev.org/c/openstack/kayobe/+/880925 | 08:54 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible master: Fix: apply int filter to rabbitmq_queue_expiry_ms https://review.opendev.org/c/openstack/kolla-ansible/+/880927 | 08:55 |
| opendevreview | Merged openstack/kolla-ansible master: Configure coordination in default for masakari-api https://review.opendev.org/c/openstack/kolla-ansible/+/874729 | 09:11 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Update notes about CentOS support https://review.opendev.org/c/openstack/kolla-ansible/+/880839 | 09:11 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/880809 | 09:13 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Pass through kolla inventories as is https://review.opendev.org/c/openstack/kayobe/+/802864 | 09:19 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Use merge_configs and merge_yaml to generate Kolla custom config https://review.opendev.org/c/openstack/kayobe/+/782749 | 09:19 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Add cached plugin https://review.opendev.org/c/openstack/kayobe/+/803064 | 09:19 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Kayobe environment dependencies https://review.opendev.org/c/openstack/kayobe/+/802865 | 09:19 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Kayobe environment dependencies https://review.opendev.org/c/openstack/kayobe/+/802865 | 09:22 |
| frickler | seems we have some CI issue since yesterday, lots of n-v jobs seem affected, e.g. https://zuul.opendev.org/t/openstack/builds?job_name=kolla-ansible-ubuntu-cephadm&project=openstack/kolla-ansible | 09:23 |
| frickler | might be https://review.opendev.org/c/openstack/kolla-ansible/+/877697 as I already mentioned on that patch, seems just nobody cared :( https://review.opendev.org/c/openstack/kolla-ansible/+/877697 | 09:25 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Add retries to get_url and uri tasks https://review.opendev.org/c/openstack/kayobe/+/880931 | 09:25 |
| opendevreview | Dr. Jens Harbott proposed openstack/kolla-ansible master: Revert "ansible: bump min to 2.13 and max to 2.14" https://review.opendev.org/c/openstack/kolla-ansible/+/880843 | 09:26 |
| frickler | ^^ testing with a revert to check my guess | 09:27 |
| SvenKieske | mhm, why don't we have a policy that all comments should be addressed before a merge? seems pretty basic, no? | 09:28 |
| frickler | that certainly would be helpful I agree | 09:30 |
| SvenKieske | that's the default for most git forges today. it's a different discussion who can mark comments as resolved, but at least they should be resolved imho. | 09:31 |
| mnasiadka | frickler: that's not ansible version related, I've seen these in the past, for some reason on Ubuntu mariadb is unavailable when do some first db operations in keystone role | 09:32 |
| mnasiadka | frickler: and it doesn't show up every time, but maybe after the upgrade it is ;-) | 09:32 |
| frickler | so a race condition that gets triggered 100% by new ansible because it is either faster or slower? | 09:33 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/880809 | 09:34 |
| frickler | but maybe also a reason to reconsider n-v status for jobs, because mostly that means they just get ignored | 09:35 |
| mnasiadka | well, we could do that after we branch antelope, now it would mean a lot of firefighting :) | 09:36 |
| kevko_ | what is n-v | 09:39 |
| frickler | non-voting | 09:39 |
| kevko_ | aaaa :d thanks | 09:43 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Bump up etcd, lego and prometheus components versions to latest https://review.opendev.org/c/openstack/kolla/+/880920 | 09:44 |
| SvenKieske | regarding the mariadb non availabilty: does the CI make a test first if the db is ready before doing db operations via keystone? db setup takes some time and I have seen failures related to it in other CIs also. | 09:48 |
| kevko_ | SvenKieske: there is 10 wait_for mariadb is up and it's not | 09:53 |
| frickler | SvenKieske: yes, it does a pretty complicated dance to set up replication and is failing consistently halfway through, have a look at the deploy log | 09:54 |
| frickler | mnasiadka: all the jobs that failed on 877697 have 100% failure after it merged, all had at least partial success before, even openeuler. so I reject your claim that the failure is unrelated | 09:55 |
| SvenKieske | mhm is that the standard mariadb/galera thing? some mariadb/galera versions crash straight up, maybe we hit something there | 09:55 |
| SvenKieske | checking the logs.. | 09:56 |
| kevko_ | https://zuul.opendev.org/t/openstack/build/ab327bfc1b22436a810740af0af7d243/log/primary/logs/kolla/mariadb/mariadb.txt | 09:56 |
| SvenKieske | kevko: thanks, a link is always appreciated; I still struggle with the zuul gui sometimes. | 09:59 |
| kevko_ | mee too :D .. | 09:59 |
| SvenKieske | :D | 10:00 |
| kevko_ | i was checking diff for wait_for.py in ansible github project ..and there are not big changes on first seen | 10:00 |
| mnasiadka | well, speaking about mariadb - 10.11 is the next long term release which we should probably move to, question if now, or in B | 10:00 |
| frickler | since more ubuntu jobs are failing and we are testing ansible_max on buntu and ansible_min on others, likely ansible==7.x might be the culprit. we could crosscheck by swapping this between ubuntu and rocky | 10:00 |
| SvenKieske | well afaik that mariadb log from kevko shows replication not working, that is, quorum not forming consensus fast enough? IIRC mariadb patchlevel is never fixed in the default install. I remember vivdly having to fix it to a certain release in the past. might be worthwhile to check the mariadb changelog. | 10:04 |
| SvenKieske | it's curious though that this happens after that ansible bump | 10:04 |
| mnasiadka | has it passed successfully on the revert? | 10:04 |
| SvenKieske | there's no output yet posted to gerrit from zuul, can I search zuul by gerrit review number or something? I only see job filters so far. | 10:06 |
| SvenKieske | ah got it | 10:06 |
| SvenKieske | failure: https://zuul.opendev.org/t/openstack/build/1441deb8681c498ba34171b009117ae3 | 10:07 |
| SvenKieske | mariadb clustercheck does not look good to me: https://zuul.opendev.org/t/openstack/build/1441deb8681c498ba34171b009117ae3/log/primary/logs/kolla/mariadb/mariadb-clustercheck.txt | 10:08 |
| SvenKieske | this really reminds me of those old mariadb bugs, because they would also occur only from time to time, not always.. | 10:09 |
| mnasiadka | clustercheck looks perfectly similar on successful jobs | 10:09 |
| mnasiadka | it's just socat log, which is a bit irritating | 10:10 |
| kevko_ | drop clustercheck and switch to proxysql ( :) ) | 10:11 |
| mnasiadka | kevko_: fl1nt had some security concerns ;-) | 10:13 |
| SvenKieske | ah okay, yeah I was wondering..? no successful connection logged? I like proxysql, but I guess it's not the answer to HA sql server - in this case. | 10:13 |
| mnasiadka | we probably could stop logging socat nonsense and make clustercheck log directly to a file | 10:13 |
| mnasiadka | xinetd was a bit better with logging | 10:13 |
| kevko_ | mnasiadka: really ? what concerns :D ... so haproxy user without password in clustercheck is better ? :D even if no real permissions in mariadb for that user ? :D | 10:21 |
| kevko_ | (for haproxy) | 10:22 |
| mnasiadka | kevko_: I'm not getting into this discussion again | 10:29 |
| mnasiadka | I'm just comparing the tool used to launch clustercheck service - with xinetd I could read the logs | 10:29 |
| mnasiadka | With socat enforced by RHEL 9, I can't ;-) | 10:29 |
| mmalchuk | lets add logs support for socat) | 10:32 |
| mmalchuk | mnasiadka please merge last patches: https://review.opendev.org/q/topic:dib-yaml-width in k-a and kayobe | 10:34 |
| mmalchuk | thanks | 10:34 |
| kevko_ | mnasiadka: no problem ...but anyone with access to internal network can run something like mysql -h 192.168.205.10 -u haproxy information_schema -e "select * from $i" -B and run it in concurrent processes .. haha .. i will try sometimes in a future to check if i can kill mariadb instance (probably not ..but i will try :)) | 10:38 |
| mnasiadka | kevko_: then we should at least allow that only from the haproxy hosts ;-) | 10:39 |
| mmalchuk | kevko_ hacker) | 10:39 |
| kevko_ | sorry for i in $(mysql -h 192.168.205.254 -u haproxy information_schema -e 'show tables' -B ); do $(mysql -h 192.168.205.254 -u haproxy information_schema -e "select * from $i" -B &);done | 10:39 |
| kevko_ | this is also not discussion for me ... i am turning on proxysql and disabling clustercheck on all deployments since stein and no problem :) | 10:40 |
| mmalchuk | interesting... stable? ubuntu2004 and xena for example? | 10:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Improve clustercheck logging https://review.opendev.org/c/openstack/kolla-ansible/+/880939 | 10:49 |
| mmalchuk | mnasiadka -d -d -d : Prints fatal, error, warning, notice, and info messages. | 10:53 |
| mmalchuk | for socat | 10:54 |
| mnasiadka | mmalchuk: more spam? no thanks | 10:54 |
| mnasiadka | I'm rather interested in clustercheck script output, not the ip stack messages | 10:54 |
| mmalchuk | mnasiadka -v show only transfered data | 10:54 |
| mmalchuk | no info | 10:54 |
| mmalchuk | reasonable, but info not only ip messages | 10:55 |
| mnasiadka | so the manual lies, ehh | 10:56 |
| mmalchuk | everybody lies (c) Dr.House | 10:57 |
| mmalchuk | mnasiadka please merge another Kayobe patches: https://review.opendev.org/q/topic:fix-acl-forgotten | 10:58 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for multiple ceph files https://review.opendev.org/c/openstack/kolla-ansible/+/877413 | 11:11 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for multiple ceph files https://review.opendev.org/c/openstack/kolla-ansible/+/877413 | 11:14 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Improve clustercheck logging https://review.opendev.org/c/openstack/kolla-ansible/+/880939 | 11:14 |
| opendevreview | Merged openstack/kolla-ansible master: Trivial fix - add int filter for rabbitmq definitions https://review.opendev.org/c/openstack/kolla-ansible/+/880887 | 11:15 |
| mnasiadka | -d -d should be enough | 11:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Improve clustercheck logging https://review.opendev.org/c/openstack/kolla-ansible/+/880939 | 11:18 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/880809 | 11:24 |
| mnasiadka | frickler: seems it is Ansible, let's merge the revert, and then work on revert of the revert ;-) | 11:26 |
| opendevreview | Magnus Lööf proposed openstack/kolla-ansible master: Fix faulty precheck for RabbitMQ https://review.opendev.org/c/openstack/kolla-ansible/+/866938 | 11:30 |
| opendevreview | Magnus Lööf proposed openstack/kolla-ansible master: Enable TLS backend for designate https://review.opendev.org/c/openstack/kolla-ansible/+/866524 | 11:32 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: Fix maximum width of the DIB Multiline-YAML https://review.opendev.org/c/openstack/kolla-ansible/+/880757 | 11:52 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Fix maximum width of the DIB Multiline-YAML https://review.opendev.org/c/openstack/kolla-ansible/+/880758 | 11:52 |
| opendevreview | Merged openstack/kayobe stable/zed: Fix maximum width of the DIB Multiline-YAML https://review.opendev.org/c/openstack/kayobe/+/880875 | 11:53 |
| frickler | mnasiadka: ack, updated the patch accordingly | 12:00 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/880809 | 12:18 |
| opendevreview | Merged openstack/kayobe stable/yoga: Fix maximum width of the DIB Multiline-YAML https://review.opendev.org/c/openstack/kayobe/+/880876 | 12:20 |
| opendevreview | Merged openstack/kayobe stable/xena: Fix maximum width of the DIB Multiline-YAML https://review.opendev.org/c/openstack/kayobe/+/880877 | 12:38 |
| opendevreview | Merged openstack/kolla-ansible master: Revert "ansible: bump min to 2.13 and max to 2.14" https://review.opendev.org/c/openstack/kolla-ansible/+/880843 | 13:20 |
| opendevreview | Merged openstack/kayobe master: Revert back to MichaelRigart.interfaces https://review.opendev.org/c/openstack/kayobe/+/880742 | 13:27 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 14:25 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/zed: Revert back to MichaelRigart.interfaces https://review.opendev.org/c/openstack/kayobe/+/881015 | 15:12 |
| opendevreview | Mark Goddard proposed openstack/kayobe stable/yoga: Revert back to MichaelRigart.interfaces https://review.opendev.org/c/openstack/kayobe/+/881016 | 15:12 |
| mmalchuk_ | mnasiadka review and merge please https://review.opendev.org/q/topic:fix-acl-forgotten | 15:48 |
| *** kevko8 is now known as kevko | 15:58 | |
| kevko | test | 15:59 |
| mmalchuk_ | passed | 16:00 |
| kevko | cool | 16:02 |
| kevko | i've setup thelounge as irc to be logged in always ..and it is very cool and very nice :P | 16:03 |
| mmalchuk_ | https://thelounge.chat/ ? | 16:05 |
| kevko | yep | 16:07 |
| kevko | kevko: t | 16:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Revert "Revert "ansible: bump min to 2.13 and max to 2.14"" https://review.opendev.org/c/openstack/kolla-ansible/+/881018 | 16:50 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: TLS certificates management sync with Kolla-Ansible https://review.opendev.org/c/openstack/kayobe/+/793697 | 23:06 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!