Wednesday, 2022-08-31

opendevreviewMerged openstack/kayobe stable/xena: Remove unused parameter from seed container example  https://review.opendev.org/c/openstack/kayobe/+/85480000:01
usr2038Hi, Can i use kolla-ansible in ubuntu 18?06:26
congntadmin1: You can only install openstack version victoria on Ubuntu 18 (https://github.com/openstack/kolla-ansible/blob/stable/victoria/ansible/roles/prechecks/vars/main.yml)06:33
congntusr2038: You can only install openstack version victoria on Ubuntu 18 (https://github.com/openstack/kolla-ansible/blob/stable/victoria/ansible/roles/prechecks/vars/main.yml)06:33
fricklernote that both stable/victoria and Ubuntu 18.04 LTS (there is no "ubuntu 18") are close to the end of their life, so it makes no sense at all to start a new deployment on this06:45
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [CI] Run the cells scenario on Ubuntu  https://review.opendev.org/c/openstack/kolla-ansible/+/84848606:56
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [CI] Run the cells scenario on Ubuntu  https://review.opendev.org/c/openstack/kolla-ansible/+/84848606:56
opendevreviewMichal Nasiadka proposed openstack/kolla stable/xena: mariadb: Fix RPM repository URLs  https://review.opendev.org/c/openstack/kolla/+/85533207:04
opendevreviewMichal Nasiadka proposed openstack/kolla stable/xena: mariadb: Fix RPM repository URLs  https://review.opendev.org/c/openstack/kolla/+/85533207:06
mnasiadkahrw, yoctozepto, mgoddard: Basically Rocky9 multinode CI fails on MariaDB (with c8s images), I did set up some environment, and it seems sst transfer is failing - so I don't think we should merge that yet.07:12
yoctozeptomnasiadka: whaaa; if the images have not changed, what could be affecting the sst transfer?07:15
yoctozeptomaybe ulimits again?07:16
opendevreviewMerged openstack/kolla master: Bump prometheus services to latest version  https://review.opendev.org/c/openstack/kolla/+/85434707:18
yoctozeptobtw, should we backport the bump? ^07:19
mnasiadkawe should07:20
mnasiadkayoctozepto: maybe ulimits again, let me try to limit them :D07:20
yoctozeptomnasiadka: :D07:20
yoctozeptomnasiadka: if so, it might be a crazy ride, where we need to limit them also for services like redis and etcd as they all do seriosuly customised memory management; similarly for elastic I guess07:22
yoctozeptooh the joys of global changes07:22
mnasiadkamaybe we need a global ulimit for each container :D07:22
yoctozeptoyeah, that could be what it will boil down to07:23
yoctozeptolet's see07:23
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: [CI] Stop testing upgrades  https://review.opendev.org/c/openstack/kolla-ansible/+/85531607:28
yoctozepto^ ussuri kolla-ansible gate unblocker07:33
opendevreviewRadosław Piliszek proposed openstack/kolla stable/ussuri: [CI] Stop testing upgrades  https://review.opendev.org/c/openstack/kolla/+/85531707:35
yoctozeptoand the same for kolla ^07:35
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Fix haproxy prechecks after deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/85524907:36
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: keystone: Remove redundant 'when'  https://review.opendev.org/c/openstack/kolla-ansible/+/85524407:36
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: [security] Make Ironic tftpd run as nobody  https://review.opendev.org/c/openstack/kolla-ansible/+/85523907:36
opendevreviewMerged openstack/kolla-ansible master: [CI] Run the cells scenario on Ubuntu  https://review.opendev.org/c/openstack/kolla-ansible/+/84848608:07
*** jopdorp_ is now known as jopdorp08:07
opendevreviewMichal Nasiadka proposed openstack/kayobe master: nclu-switch: Fix order in template  https://review.opendev.org/c/openstack/kayobe/+/85531808:10
clayjarI've reached the pinnacle of tech success: Fallen onto a new IRC client and managed to get connected.08:48
yoctozeptoclayjar: congrats! and welcome, good humour is welcome as well08:52
clayjarMotivated by a question of course: On a brand new deployment, neutron-openvswitch-agent is restarting. Can't connect to MySQL server on '10.4.77.1' in logs. That's the internal keepalived VIP.08:55
yoctozeptothe obvious counter-question is whether you can actually reach that VIP from that node (as the containers simply utilise the host networking)08:56
clayjarYes indeed, and no I can't. However I can't reach the equivalent IP on another cluster, and neutron is happy :/ But if that's a red herring, I'll focus on this cluster.08:57
clayjar`ss -tlnp | grep :3306` shows haproxy listening on that IP:port, on both hosts running haproxy containers. But I can't see where the IP would be assigned - possibly my keepalived ignorance.09:00
yoctozeptoodd, it's required to be reachable as the services talk to the database via vip unless haproxy is disabled09:00
clayjarYeah, so I've got somethingw rong.09:00
yoctozeptodoes `ip -4 a` show this address anywhere?09:00
clayjarSure enough, a reconfigure run fails the 'wait for MariaDB to be available via HAProxy' task.09:00
clayjarNo09:01
yoctozeptothen need to see what keepalived is doing09:01
yoctozeptoit should be activating this address on one of the haproxy nodes09:01
clayjarkeepalived container is running on the same 2 hosts as haproxy09:02
clayjar`docker exec -u 0 -it keepalived bash -c 'kill -USR2 $(cat /var/run/keepalived.pid) && cat /tmp/keepalived.stats'` shows the second one has `Became master: 1`, Advertisements: .. Sent: 177199, all else 0. Received: 177195 on the other one, all else 0.09:06
yoctozeptohmm, you can try restarting both keepaliveds to see if they get to set this address anywhere09:07
yoctozeptoalso then check their logs09:07
yoctozeptothey should negotiate who the master is and do all the necessary dance09:07
clayjarscratching my head looking for logs. /var/log/kolla/ has no file or content saying keepalive09:11
yoctozeptodocker logs09:11
yoctozepto(I agree that might not have been obvious enough)09:12
yoctozepto`docker logs keepalived`09:12
clayjaroh dear, sorry :s09:12
clayjarSure enough 'VRRP_Instance(kolla_internal_vip_51) Entering MASTER STATE'09:14
yoctozeptoand the address got assigned finally?09:17
clayjarno unfortunately09:17
yoctozeptoand no other messages that could indicate a failure?09:18
clayjarWARNING - default user 'keepalived_script' for script execution does not exist - please create.09:21
clayjarsame message on working cluster's keepalived logs09:22
yoctozeptoit's harmless09:22
clayjarfwiw I've stopped keepalived & haproxy containers, running reconfigure09:25
clayjarIP exists!09:25
yoctozeptoawesome!09:27
clayjarI wouldn't mind knowing how and why it didn't get one earlier - or in fact would have had and lost it, otherwise that check task would have failed on previous runs - but I don't know what I don't know at this stage.09:28
yoctozeptono idea here either, looks like something got stuck in the networking stack, might be a finicky bug09:29
clayjaryeah. I'll get this (basically eol) train job done, then hurry up with upgrades before worrying about finicky things09:29
clayjarokay COB is upon me. Thank you for the support09:30
yoctozeptoyw09:32
clayjarjob for another day: local irc jumphost to run weechat off. For now, farewell09:33
opendevreviewMark Goddard proposed openstack/kolla stable/xena: mariadb: Fix RPM repository URLs  https://review.opendev.org/c/openstack/kolla/+/85533210:48
opendevreviewDoug Szumski proposed openstack/kolla-ansible stable/ussuri: Update ironic iPXE configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/85533811:01
opendevreviewVerification of a change to openstack/kayobe stable/yoga failed: Remove unused parameter from seed container example  https://review.opendev.org/c/openstack/kayobe/+/85479911:19
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Update ironic iPXE configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/85533811:56
usr2038congnt: thanks congnt. i have missed to strain kolla-ansible version. now its ok, but i encountered new things. first bootstrap-servers complained about that it couldn't remove chrony apparmor profile(because chrony is not installed). i bypassed it by installing before bootstrap. than precheck complained about that there is no docker package. virtual environment have that package but it checks system. is there a place that i inform kolla that i11:57
opendevreviewMerged openstack/kolla master: Make keystone_bootstrap accept no ADMIN_URL too  https://review.opendev.org/c/openstack/kolla/+/85483712:15
opendevreviewMerged openstack/kolla stable/ussuri: [CI] Stop testing upgrades  https://review.opendev.org/c/openstack/kolla/+/85531712:15
opendevreviewMerged openstack/kolla-ansible stable/ussuri: [CI] Stop testing upgrades  https://review.opendev.org/c/openstack/kolla-ansible/+/85531612:15
opendevreviewMerged openstack/kolla-ansible stable/yoga: Fix haproxy prechecks after deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/85524512:35
opendevreviewVerification of a change to openstack/kolla-ansible stable/xena failed: Fix haproxy prechecks after deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/85524612:35
opendevreviewMerged openstack/kolla-ansible stable/ussuri: Fix haproxy prechecks after deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/85524912:37
opendevreviewVerification of a change to openstack/kolla-ansible stable/wallaby failed: Fix haproxy prechecks after deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/85524712:38
opendevreviewRadosław Piliszek proposed openstack/kolla stable/xena: mariadb: Fix RPM repository URLs  https://review.opendev.org/c/openstack/kolla/+/85533212:42
opendevreviewRadosław Piliszek proposed openstack/kolla stable/wallaby: mariadb: Fix RPM repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534112:44
opendevreviewRadosław Piliszek proposed openstack/kolla stable/wallaby: mariadb: Fix repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534112:49
opendevreviewRadosław Piliszek proposed openstack/kolla stable/victoria: mariadb: Fix repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534212:49
opendevreviewRadosław Piliszek proposed openstack/kolla stable/wallaby: mariadb: Fix repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534112:51
opendevreviewRadosław Piliszek proposed openstack/kolla stable/victoria: mariadb: Fix repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534212:52
opendevreviewRadosław Piliszek proposed openstack/kolla stable/ussuri: mariadb: Fix repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534312:52
opendevreviewRadosław Piliszek proposed openstack/kolla stable/ussuri: mariadb: Fix repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534312:56
opendevreviewVerification of a change to openstack/kolla-ansible stable/victoria failed: Fix haproxy prechecks after deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/85524812:57
yoctozeptolet's first patch kolla to avoid these surprises, eh12:57
opendevreviewMerged openstack/kolla-ansible stable/ussuri: keystone: Remove redundant 'when'  https://review.opendev.org/c/openstack/kolla-ansible/+/85524412:59
opendevreviewMerged openstack/kayobe stable/xena: Fix IP allocation limit in overcloud inventory discover  https://review.opendev.org/c/openstack/kayobe/+/85514413:09
opendevreviewMerged openstack/kayobe stable/yoga: Fix IP allocation limit in overcloud inventory discover  https://review.opendev.org/c/openstack/kayobe/+/85514313:09
opendevreviewMerged openstack/kayobe stable/wallaby: Fix IP allocation limit in overcloud inventory discover  https://review.opendev.org/c/openstack/kayobe/+/85514513:09
opendevreviewMerged openstack/kayobe stable/xena: Fix use of become_user by installing acl package  https://review.opendev.org/c/openstack/kayobe/+/85514813:09
opendevreviewMerged openstack/kayobe stable/wallaby: Fix use of become_user by installing acl package  https://review.opendev.org/c/openstack/kayobe/+/85514913:09
opendevreviewMerged openstack/kolla-ansible stable/yoga: keystone: Remove redundant 'when'  https://review.opendev.org/c/openstack/kolla-ansible/+/85524013:18
opendevreviewVerification of a change to openstack/kolla-ansible stable/xena failed: keystone: Remove redundant 'when'  https://review.opendev.org/c/openstack/kolla-ansible/+/85524113:18
opendevreviewVerification of a change to openstack/kolla-ansible stable/wallaby failed: keystone: Remove redundant 'when'  https://review.opendev.org/c/openstack/kolla-ansible/+/85524213:26
opendevreviewMerged openstack/kolla-ansible stable/victoria: keystone: Remove redundant 'when'  https://review.opendev.org/c/openstack/kolla-ansible/+/85524313:30
opendevreviewMerged openstack/kolla-ansible stable/yoga: [security] Make Ironic tftpd run as nobody  https://review.opendev.org/c/openstack/kolla-ansible/+/85523513:40
opendevreviewMerged openstack/kolla-ansible stable/xena: [security] Make Ironic tftpd run as nobody  https://review.opendev.org/c/openstack/kolla-ansible/+/85523613:40
opendevreviewMerged openstack/kolla-ansible stable/wallaby: [security] Make Ironic tftpd run as nobody  https://review.opendev.org/c/openstack/kolla-ansible/+/85523713:40
opendevreviewMerged openstack/kolla-ansible stable/victoria: [security] Make Ironic tftpd run as nobody  https://review.opendev.org/c/openstack/kolla-ansible/+/85523813:40
opendevreviewMerged openstack/kolla-ansible stable/ussuri: [security] Make Ironic tftpd run as nobody  https://review.opendev.org/c/openstack/kolla-ansible/+/85523913:40
opendevreviewMerged openstack/kolla master: Revert "Workaround setuptools==60.0.0 breakages"  https://review.opendev.org/c/openstack/kolla/+/82224813:40
opendevreviewMerged openstack/kolla master: Drop centos8s jobs from experimental pipeline  https://review.opendev.org/c/openstack/kolla/+/85512813:40
mnasiadkamgoddard mnasiadka hrw yoctozepto bbezak parallax Fl1nt frickler adrian-a - meeting in 1013:51
mnasiadka#startmeeting kolla14:00
opendevmeetMeeting started Wed Aug 31 14:00:01 2022 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'kolla'14:00
mnasiadka#topic rollcall14:00
mnasiadkao/14:00
TimBeermann[m]o&14:01
TimBeermann[m]*o/14:01
mgoddard\o14:02
fricklero/14:03
yoctozeptoo/14:03
mnasiadka#topic agenda14:04
mnasiadka* CI status14:04
mnasiadka* Release tasks14:04
mnasiadka* Regular stable releases (first meeting in a month)14:04
mnasiadka* Current cycle planning14:04
mnasiadka* Additional agenda (from whiteboard)14:04
mnasiadka* Open discussion14:04
mnasiadka#topic CI Status14:04
mnasiadkaSo how is CI? All branches broken by MariaDB repo change and docker-py 6?14:05
fricklerat least prometheus seems to be fixed14:07
yoctozeptoyeah, merging url and docker fixes14:08
mnasiadkaOk then, seems we have everything covered in CI14:08
mnasiadka#topic Release tasks14:08
yoctozeptomonasca-grafana fails in victoria centos814:09
mnasiadkaI think we need to do a stable release dance after we merge docker-py 6 fix14:09
yoctozeptoblocks the other patch14:09
yoctozeptoaye, agreed14:09
mnasiadkayoctozepto: we could change it to non-voting or just remove it - it's victoria-em...14:09
yoctozeptoremoving sounds good to me14:10
mnasiadkaanybody opposes?14:10
mnasiadka(I hope not)14:10
mnasiadkaok, done deal14:11
mnasiadkayoctozepto: I guess you'll do it?14:11
yoctozeptoyeah14:12
fricklerdo we have plans to EOL u+v?14:13
opendevreviewRadosław Piliszek proposed openstack/kolla stable/victoria: mariadb: Fix repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534214:13
mnasiadkaWe EOLed Train I think, that's probably enough14:14
yoctozeptoyeah, they mostly fare well14:14
mnasiadkaSo - cycle highlights14:14
mnasiadkaAnybody willing to raise a patch for them or should I?14:14
yoctozeptono bandwidth for it left14:15
mnasiadkaOk, that's on me14:17
mnasiadka#todo mnasiadka raise a patch for cycle highlights14:17
mnasiadkaReminder - feature freeze is R-2 - so it's basically 3 weeks time14:17
mnasiadka#topic Current cycle planning14:18
mnasiadkaSo, I'm still fighting a bit with Rocky Linux 9, probably MariaDB is suffering from raised ulimit in EL9 (and god knows what other issues lie ahead).14:18
mnasiadkaMight be merging it will drag a bit...14:19
mnasiadkaAny other features that need attention?14:19
yoctozeptosurely the podman stuff14:20
opendevreviewMerged openstack/kolla-ansible stable/ussuri: Update ironic iPXE configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/85533814:20
mnasiadkaYes, that would be nice - I guess systemd patch should go first?14:22
yoctozeptoI guess so too14:22
mnasiadkaSo, if cores have capacity - it would be nice to get reviewed https://review.opendev.org/c/openstack/kolla-ansible/+/816724 until next meeting14:23
mnasiadkaok, no additional agenda (apart cycle highlights)14:23
mnasiadka#topic Open Discussion14:23
mnasiadkaAnybody? Anything?14:24
headphoneJamesLet's encrypt patch would be nice to have some eyes on14:24
mnasiadkaheadphoneJames: yes, that's true14:24
headphoneJamesIt should be ready to go14:24
hrwhow we are with CS9? I see that zuul gave +1 to both kolla and k-a14:25
mnasiadkaYes, but multinode on RL9 is failing14:25
mnasiadkain k-a14:25
mnasiadkaI'm working on this14:25
hrwthx14:25
mnasiadkacentos-release-* packages are already in RL9, so once k-a part is working, I'll also upload a patch for Rocky images on top of CS9 change14:26
hrwcool14:27
mnasiadkabtw, it seems this year I need to talk with ptgbot to schedule Kolla session - I'm planning to schedule the same slots as last PTG14:28
mnasiadkaI hope that's ok14:28
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: Move to CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/83666414:28
hrwrebased to get rid of conflicts14:29
mnasiadkaok14:29
mnasiadkaLet's finish for today - thanks for coming!14:29
mnasiadka#endmeeting14:29
opendevmeetMeeting ended Wed Aug 31 14:29:30 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:29
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-08-31-14.00.html14:29
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-08-31-14.00.txt14:29
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-08-31-14.00.log.html14:29
hrwmnasiadka: ptg? fine for me - we are good at handling 'o, someone is not present' situation14:29
* hrw out 14:30
yoctozeptothanks mnasiadka 14:32
opendevreviewMerged openstack/kolla stable/ussuri: mariadb: Fix repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534315:21
opendevreviewMerged openstack/kolla stable/xena: mariadb: Fix RPM repository URLs  https://review.opendev.org/c/openstack/kolla/+/85533215:31
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add systemd container control  https://review.opendev.org/c/openstack/kolla-ansible/+/81672415:40
opendevreviewFranco Mariotti proposed openstack/kolla-ansible master: Allow exposing OpenStack exporter via HAProxy  https://review.opendev.org/c/openstack/kolla-ansible/+/85155316:28
opendevreviewFranco Mariotti proposed openstack/kolla-ansible master: Allow exposing OpenStack exporter via HAProxy  https://review.opendev.org/c/openstack/kolla-ansible/+/85155316:29
opendevreviewMerged openstack/kolla stable/victoria: mariadb: Fix repository URLs  https://review.opendev.org/c/openstack/kolla/+/85534216:52
opendevreviewMerged openstack/kayobe stable/yoga: Fix use of become_user by installing acl package  https://review.opendev.org/c/openstack/kayobe/+/85514717:46
opendevreviewMerged openstack/kolla-ansible master: [TrivialFix] Fix task name in Ironic config  https://review.opendev.org/c/openstack/kolla-ansible/+/85476318:07
opendevreviewMerged openstack/kayobe stable/yoga: Remove unused parameter from seed container example  https://review.opendev.org/c/openstack/kayobe/+/85479919:18
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/yoga: [TrivialFix] Fix task name in Ironic config  https://review.opendev.org/c/openstack/kolla-ansible/+/85534520:01
underknowledge[m]test for client20:15
opendevreviewMerged openstack/kolla-ansible stable/wallaby: keystone: Remove redundant 'when'  https://review.opendev.org/c/openstack/kolla-ansible/+/85524222:32
opendevreviewMerged openstack/kolla-ansible stable/victoria: Fix haproxy prechecks after deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/85524822:56
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Fix haproxy prechecks after deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/85524722:56
opendevreviewMerged openstack/kolla-ansible stable/xena: keystone: Remove redundant 'when'  https://review.opendev.org/c/openstack/kolla-ansible/+/85524123:10
opendevreviewMerged openstack/kolla-ansible stable/yoga: [TrivialFix] Fix task name in Ironic config  https://review.opendev.org/c/openstack/kolla-ansible/+/85534523:11
opendevreviewMerged openstack/kolla-ansible stable/xena: Fix haproxy prechecks after deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/85524623:11

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!