Thursday, 2023-04-20

opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: octavia: Fix hm-interface after bumping openstack collection  https://review.opendev.org/c/openstack/kolla-ansible/+/87820907:03
mnasiadkafrickler: should work now07:03
fricklermnasiadka: hmm, did they drop os_security_group_info from the collection again? this was working earlier, wasn't it?07:10
mnasiadkafrickler: have no clue, but if os_security_group works now, no need to change - I'll make it better in a followup07:11
fricklerack07:13
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos  https://review.opendev.org/c/openstack/kolla/+/88080908:03
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos  https://review.opendev.org/c/openstack/kolla/+/88080908:05
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/zed: Update notes about CentOS support  https://review.opendev.org/c/openstack/kolla-ansible/+/88083908:25
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/zed: Add precheck to fail if RabbitMQ HA needs configuring  https://review.opendev.org/c/openstack/kolla-ansible/+/88084008:26
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/zed: Add precheck to fail if RabbitMQ HA needs configuring  https://review.opendev.org/c/openstack/kolla-ansible/+/88084008:27
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/yoga: Add precheck to fail if RabbitMQ HA needs configuring  https://review.opendev.org/c/openstack/kolla-ansible/+/88084108:27
opendevreviewMichal Nasiadka proposed openstack/kolla master: Bump up etcd, lego and prometheus components versions to latest  https://review.opendev.org/c/openstack/kolla/+/88092008:27
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/yoga: Add precheck to fail if RabbitMQ HA needs configuring  https://review.opendev.org/c/openstack/kolla-ansible/+/88084108:28
opendevreviewMark Goddard proposed openstack/kolla-ansible stable/xena: Add precheck to fail if RabbitMQ HA needs configuring  https://review.opendev.org/c/openstack/kolla-ansible/+/88084208:28
opendevreviewMerged openstack/kolla-ansible stable/yoga: cli: fix find globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/88088608:50
opendevreviewMerged openstack/kayobe master: Allow to use own repos.y[a]ml file to build containers  https://review.opendev.org/c/openstack/kayobe/+/84533808:53
opendevreviewMaksim Malchuk proposed openstack/kayobe stable/zed: Allow to use own repos.y[a]ml file to build containers  https://review.opendev.org/c/openstack/kayobe/+/88092308:53
opendevreviewMaksim Malchuk proposed openstack/kayobe stable/yoga: Allow to use own repos.y[a]ml file to build containers  https://review.opendev.org/c/openstack/kayobe/+/88092408:53
opendevreviewMaksim Malchuk proposed openstack/kayobe stable/xena: Allow to use own repos.y[a]ml file to build containers  https://review.opendev.org/c/openstack/kayobe/+/88092508:54
opendevreviewMatt Crees proposed openstack/kolla-ansible master: Fix: apply int filter to rabbitmq_queue_expiry_ms  https://review.opendev.org/c/openstack/kolla-ansible/+/88092708:55
opendevreviewMerged openstack/kolla-ansible master: Configure coordination in default for masakari-api  https://review.opendev.org/c/openstack/kolla-ansible/+/87472909:11
opendevreviewMerged openstack/kolla-ansible stable/zed: Update notes about CentOS support  https://review.opendev.org/c/openstack/kolla-ansible/+/88083909:11
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos  https://review.opendev.org/c/openstack/kolla/+/88080909:13
opendevreviewMark Goddard proposed openstack/kayobe master: Pass through kolla inventories as is  https://review.opendev.org/c/openstack/kayobe/+/80286409:19
opendevreviewMark Goddard proposed openstack/kayobe master: Use merge_configs and merge_yaml to generate Kolla custom config  https://review.opendev.org/c/openstack/kayobe/+/78274909:19
opendevreviewMark Goddard proposed openstack/kayobe master: Add cached plugin  https://review.opendev.org/c/openstack/kayobe/+/80306409:19
opendevreviewMark Goddard proposed openstack/kayobe master: Kayobe environment dependencies  https://review.opendev.org/c/openstack/kayobe/+/80286509:19
opendevreviewMark Goddard proposed openstack/kayobe master: Kayobe environment dependencies  https://review.opendev.org/c/openstack/kayobe/+/80286509:22
fricklerseems we have some CI issue since yesterday, lots of n-v jobs seem affected, e.g. https://zuul.opendev.org/t/openstack/builds?job_name=kolla-ansible-ubuntu-cephadm&project=openstack/kolla-ansible09:23
fricklermight be https://review.opendev.org/c/openstack/kolla-ansible/+/877697 as I already mentioned on that patch, seems just nobody cared :( https://review.opendev.org/c/openstack/kolla-ansible/+/87769709:25
opendevreviewMark Goddard proposed openstack/kayobe master: Add retries to get_url and uri tasks  https://review.opendev.org/c/openstack/kayobe/+/88093109:25
opendevreviewDr. Jens Harbott proposed openstack/kolla-ansible master: Revert "ansible: bump min to 2.13 and max to 2.14"  https://review.opendev.org/c/openstack/kolla-ansible/+/88084309:26
frickler^^ testing with a revert to check my guess09:27
SvenKieskemhm, why don't we have a policy that all comments should be addressed before a merge? seems pretty basic, no?09:28
fricklerthat certainly would be helpful I agree09:30
SvenKieskethat's the default for most git forges today. it's a different discussion who can mark comments as resolved, but at least they should be resolved imho.09:31
mnasiadkafrickler: that's not ansible version related, I've seen these in the past, for some reason on Ubuntu mariadb is unavailable when do some first db operations in keystone role09:32
mnasiadkafrickler: and it doesn't show up every time, but maybe after the upgrade it is ;-)09:32
fricklerso a race condition that gets triggered 100% by new ansible because it is either faster or slower?09:33
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos  https://review.opendev.org/c/openstack/kolla/+/88080909:34
fricklerbut maybe also a reason to reconsider n-v status for jobs, because mostly that means they just get ignored09:35
mnasiadkawell, we could do that after we branch antelope, now it would mean a lot of firefighting :)09:36
kevko_what is n-v09:39
fricklernon-voting09:39
kevko_aaaa :d thanks 09:43
opendevreviewMichal Nasiadka proposed openstack/kolla master: Bump up etcd, lego and prometheus components versions to latest  https://review.opendev.org/c/openstack/kolla/+/88092009:44
SvenKieskeregarding the mariadb non availabilty: does the CI make a test first if the db is ready before doing db operations via keystone? db setup takes some time and I have seen failures related to it in other CIs also.09:48
kevko_SvenKieske: there is 10 wait_for mariadb is up and it's not 09:53
fricklerSvenKieske: yes, it does a pretty complicated dance to set up replication and is failing consistently halfway through, have a look at the deploy log09:54
fricklermnasiadka: all the jobs that failed on 877697 have 100% failure after it merged, all had at least partial success before, even openeuler. so I reject your claim that the failure is unrelated09:55
SvenKieskemhm is that the standard mariadb/galera thing? some mariadb/galera versions crash straight up, maybe we hit something there09:55
SvenKieskechecking the logs..09:56
kevko_https://zuul.opendev.org/t/openstack/build/ab327bfc1b22436a810740af0af7d243/log/primary/logs/kolla/mariadb/mariadb.txt09:56
SvenKieskekevko: thanks, a link is always appreciated; I still struggle with the zuul gui sometimes.09:59
kevko_mee too :D ..09:59
SvenKieske:D10:00
kevko_i was checking diff for wait_for.py in ansible github project ..and there are not big changes on first seen 10:00
mnasiadkawell, speaking about mariadb - 10.11 is the next long term release which we should probably move to, question if now, or in B10:00
fricklersince more ubuntu jobs are failing and we are testing ansible_max on buntu and ansible_min on others, likely ansible==7.x might be the culprit. we could crosscheck by swapping this between ubuntu and rocky10:00
SvenKieskewell afaik that mariadb log from kevko shows replication not working, that is, quorum not forming consensus fast enough? IIRC mariadb patchlevel is never fixed in the default install. I remember vivdly having to fix it to a certain release in the past. might be worthwhile to check the mariadb changelog.10:04
SvenKieskeit's curious though that this happens after that ansible bump10:04
mnasiadkahas it passed successfully on the revert?10:04
SvenKieskethere's no output yet posted to gerrit from zuul, can I search zuul by gerrit review number or something? I only see job filters so far.10:06
SvenKieskeah got it10:06
SvenKieskefailure: https://zuul.opendev.org/t/openstack/build/1441deb8681c498ba34171b009117ae310:07
SvenKieskemariadb clustercheck does not look good to me: https://zuul.opendev.org/t/openstack/build/1441deb8681c498ba34171b009117ae3/log/primary/logs/kolla/mariadb/mariadb-clustercheck.txt10:08
SvenKieskethis really reminds me of those old mariadb bugs, because they would also occur only from time to time, not always..10:09
mnasiadkaclustercheck looks perfectly similar on successful jobs10:09
mnasiadkait's just socat log, which is a bit irritating10:10
kevko_drop clustercheck and switch to proxysql ( :) )10:11
mnasiadkakevko_: fl1nt had some security concerns ;-)10:13
SvenKieskeah okay, yeah I was wondering..? no successful connection logged? I like proxysql, but I guess it's not the answer to HA sql server - in this case.10:13
mnasiadkawe probably could stop logging socat nonsense and make clustercheck log directly to a file10:13
mnasiadkaxinetd was a bit better with logging10:13
kevko_mnasiadka: really ? what concerns :D ... so haproxy user without password in clustercheck is better ? :D even if no real permissions  in mariadb for that user ? :D 10:21
kevko_(for haproxy)10:22
mnasiadkakevko_: I'm not getting into this discussion again10:29
mnasiadkaI'm just comparing the tool used to launch clustercheck service - with xinetd I could read the logs10:29
mnasiadkaWith socat enforced by RHEL 9, I can't ;-)10:29
mmalchuklets add logs support for socat)10:32
mmalchukmnasiadka please merge last patches: https://review.opendev.org/q/topic:dib-yaml-width in k-a and kayobe10:34
mmalchukthanks10:34
kevko_mnasiadka: no problem ...but anyone with access to internal network can run something like mysql -h 192.168.205.10 -u haproxy information_schema -e "select * from $i" -B   and run it in concurrent processes .. haha .. i will try sometimes in a future to check if i can kill mariadb instance (probably not ..but i will try :))10:38
mnasiadkakevko_: then we should at least allow that only from the haproxy hosts ;-)10:39
mmalchukkevko_ hacker)10:39
kevko_sorry for i in $(mysql -h 192.168.205.254 -u haproxy information_schema -e 'show tables' -B ); do $(mysql -h 192.168.205.254 -u haproxy information_schema -e "select * from $i" -B &);done10:39
kevko_this is also not discussion for me ... i am turning on proxysql and disabling clustercheck on all deployments since stein and no problem :)10:40
mmalchukinteresting... stable? ubuntu2004 and xena for example?10:48
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Improve clustercheck logging  https://review.opendev.org/c/openstack/kolla-ansible/+/88093910:49
mmalchukmnasiadka -d -d -d : Prints fatal, error, warning, notice, and info messages.10:53
mmalchukfor socat10:54
mnasiadkammalchuk: more spam? no thanks10:54
mnasiadkaI'm rather interested in clustercheck script output, not the ip stack messages10:54
mmalchukmnasiadka -v show only transfered data10:54
mmalchukno info10:54
mmalchukreasonable, but info not only ip messages10:55
mnasiadkaso the manual lies, ehh10:56
mmalchukeverybody lies (c) Dr.House10:57
mmalchukmnasiadka please merge another Kayobe patches: https://review.opendev.org/q/topic:fix-acl-forgotten10:58
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add support for multiple ceph files  https://review.opendev.org/c/openstack/kolla-ansible/+/87741311:11
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add support for multiple ceph files  https://review.opendev.org/c/openstack/kolla-ansible/+/87741311:14
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Improve clustercheck logging  https://review.opendev.org/c/openstack/kolla-ansible/+/88093911:14
opendevreviewMerged openstack/kolla-ansible master: Trivial fix - add int filter for rabbitmq definitions  https://review.opendev.org/c/openstack/kolla-ansible/+/88088711:15
mnasiadka-d -d should be enough11:15
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Improve clustercheck logging  https://review.opendev.org/c/openstack/kolla-ansible/+/88093911:18
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos  https://review.opendev.org/c/openstack/kolla/+/88080911:24
mnasiadkafrickler: seems it is Ansible, let's merge the revert, and then work on revert of the revert ;-)11:26
opendevreviewMagnus Lööf proposed openstack/kolla-ansible master: Fix faulty precheck for RabbitMQ  https://review.opendev.org/c/openstack/kolla-ansible/+/86693811:30
opendevreviewMagnus Lööf proposed openstack/kolla-ansible master: Enable TLS backend for designate  https://review.opendev.org/c/openstack/kolla-ansible/+/86652411:32
opendevreviewMerged openstack/kolla-ansible stable/yoga: Fix maximum width of the DIB Multiline-YAML  https://review.opendev.org/c/openstack/kolla-ansible/+/88075711:52
opendevreviewMerged openstack/kolla-ansible stable/xena: Fix maximum width of the DIB Multiline-YAML  https://review.opendev.org/c/openstack/kolla-ansible/+/88075811:52
opendevreviewMerged openstack/kayobe stable/zed: Fix maximum width of the DIB Multiline-YAML  https://review.opendev.org/c/openstack/kayobe/+/88087511:53
fricklermnasiadka: ack, updated the patch accordingly12:00
opendevreviewMichal Nasiadka proposed openstack/kolla master: opensearch: move to yum/apt repos  https://review.opendev.org/c/openstack/kolla/+/88080912:18
opendevreviewMerged openstack/kayobe stable/yoga: Fix maximum width of the DIB Multiline-YAML  https://review.opendev.org/c/openstack/kayobe/+/88087612:20
opendevreviewMerged openstack/kayobe stable/xena: Fix maximum width of the DIB Multiline-YAML  https://review.opendev.org/c/openstack/kayobe/+/88087712:38
opendevreviewMerged openstack/kolla-ansible master: Revert "ansible: bump min to 2.13 and max to 2.14"  https://review.opendev.org/c/openstack/kolla-ansible/+/88084313:20
opendevreviewMerged openstack/kayobe master: Revert back to MichaelRigart.interfaces  https://review.opendev.org/c/openstack/kayobe/+/88074213:27
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add support of podman deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/79922914:25
opendevreviewMark Goddard proposed openstack/kayobe stable/zed: Revert back to MichaelRigart.interfaces  https://review.opendev.org/c/openstack/kayobe/+/88101515:12
opendevreviewMark Goddard proposed openstack/kayobe stable/yoga: Revert back to MichaelRigart.interfaces  https://review.opendev.org/c/openstack/kayobe/+/88101615:12
mmalchuk_mnasiadka review and merge please https://review.opendev.org/q/topic:fix-acl-forgotten15:48
*** kevko8 is now known as kevko15:58
kevkotest15:59
mmalchuk_passed16:00
kevkocool16:02
kevkoi've setup thelounge as irc to be logged in always ..and it is very cool and very nice :P 16:03
mmalchuk_https://thelounge.chat/ ?16:05
kevkoyep16:07
kevkokevko: t16:13
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Revert "Revert "ansible: bump min to 2.13 and max to 2.14""  https://review.opendev.org/c/openstack/kolla-ansible/+/88101816:50
opendevreviewMaksim Malchuk proposed openstack/kayobe master: TLS certificates management sync with Kolla-Ansible  https://review.opendev.org/c/openstack/kayobe/+/79369723:06

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!