Wednesday, 2021-02-24

*** e0ne has quit IRC00:01
*** jamesbenson has joined #openstack-kolla00:03
*** jamesbenson has quit IRC00:08
*** jamesbenson has joined #openstack-kolla00:21
*** stingrayza has quit IRC00:29
*** jamesbenson has quit IRC00:39
*** brinzhang has joined #openstack-kolla00:40
*** LinPeiWen has joined #openstack-kolla00:40
*** stingrayza has joined #openstack-kolla00:41
*** jamesbenson has joined #openstack-kolla00:49
*** mchlumsky1 has joined #openstack-kolla00:49
*** dmsimard0 has joined #openstack-kolla00:50
*** dmsimard has quit IRC00:51
*** dmsimard0 is now known as dmsimard00:51
*** mchlumsky has quit IRC00:51
*** mchlumsky1 is now known as mchlumsky00:51
*** jamesbenson has quit IRC00:54
*** xinliang has joined #openstack-kolla01:11
*** mchlumsky has quit IRC01:11
*** mchlumsky has joined #openstack-kolla01:12
*** brinzhang has quit IRC01:13
*** brinzhang has joined #openstack-kolla01:15
*** zzzeek has quit IRC01:15
*** brinzhang has quit IRC01:16
*** zzzeek has joined #openstack-kolla01:17
*** ricolin_ has joined #openstack-kolla01:21
*** maharg101 has joined #openstack-kolla01:22
*** maharg101 has quit IRC01:26
*** ivan_lin has joined #openstack-kolla01:27
*** cz3 is now known as Halabala01:34
*** Halabala is now known as cz301:34
*** jamesbenson has joined #openstack-kolla02:00
*** jamesbenson has quit IRC02:04
*** zul has quit IRC02:08
*** ricolin_ has quit IRC02:11
*** ricolin_ has joined #openstack-kolla02:11
*** jamesbenson has joined #openstack-kolla02:15
*** jamesbenson has quit IRC02:20
*** calbers has quit IRC02:37
*** calbers has joined #openstack-kolla02:37
*** jamesbenson has joined #openstack-kolla02:59
*** jamesbenson has quit IRC03:04
*** brinzhang has joined #openstack-kolla03:17
*** maharg101 has joined #openstack-kolla03:22
*** maharg101 has quit IRC03:27
*** jamesbenson has joined #openstack-kolla03:51
*** portdirect has joined #openstack-kolla03:52
*** jamesbenson has quit IRC03:56
*** jamesbenson has joined #openstack-kolla03:59
*** xinliang has quit IRC04:01
*** jamesbenson has quit IRC04:04
*** jamesbenson has joined #openstack-kolla04:53
*** jamesbenson has quit IRC04:57
*** cah_link has joined #openstack-kolla05:03
*** cah_link has quit IRC05:12
oklhostmorning05:20
*** maharg101 has joined #openstack-kolla05:23
*** maharg101 has quit IRC05:28
*** jamesbenson has joined #openstack-kolla05:54
*** jamesbenson has quit IRC05:58
*** jamesbenson has joined #openstack-kolla05:59
*** ricolin has quit IRC06:00
*** jamesbenson has quit IRC06:03
*** LinPeiWen has quit IRC06:16
*** LinPeiWen has joined #openstack-kolla06:17
*** LinPeiWen79 has joined #openstack-kolla06:24
*** LinPeiWen has quit IRC06:25
*** ricolin_ has quit IRC06:27
*** ricolin has joined #openstack-kolla06:28
*** jamesbenson has joined #openstack-kolla06:33
*** cah_link has joined #openstack-kolla06:43
*** jbadiapa has quit IRC06:45
*** jamesbenson has quit IRC06:51
*** LinPeiWen79 has quit IRC06:58
*** jamesbenson has joined #openstack-kolla06:59
*** jamesbenson has quit IRC07:04
*** LinPeiWen has joined #openstack-kolla07:05
yoctozeptomoorning07:14
*** e0ne has joined #openstack-kolla07:38
*** e0ne has quit IRC07:39
*** jamesbenson has joined #openstack-kolla07:46
*** jamesbenson has quit IRC07:51
*** luksky has joined #openstack-kolla07:54
*** jamesbenson has joined #openstack-kolla07:59
*** amoralej|off is now known as amoralej08:00
*** jamesbenson has quit IRC08:04
*** rpittau|afk is now known as rpittau08:11
*** LinPeiWen has quit IRC08:13
*** maharg101 has joined #openstack-kolla08:20
*** bengates has joined #openstack-kolla08:23
*** andrewbonney has joined #openstack-kolla08:27
*** jamesbenson has joined #openstack-kolla08:34
*** jamesbenson has quit IRC08:39
openstackgerritMark Goddard proposed openstack/kayobe-config-dev stable/victoria: Reduce neutron RPC workers to 1  https://review.opendev.org/c/openstack/kayobe-config-dev/+/77713008:41
openstackgerritMark Goddard proposed openstack/kayobe-config-dev stable/ussuri: Reduce neutron RPC workers to 1  https://review.opendev.org/c/openstack/kayobe-config-dev/+/77713108:41
openstackgerritMark Goddard proposed openstack/kayobe-config-dev stable/train: Reduce neutron RPC workers to 1  https://review.opendev.org/c/openstack/kayobe-config-dev/+/77713208:41
mgoddardmorning08:47
mnasiadkamorning08:56
*** jamesbenson has joined #openstack-kolla08:59
*** jamesbenson has quit IRC09:04
*** dougsz has joined #openstack-kolla09:17
mnasiadkaI have a feeling that ubuntu nodes in the CI are cursed09:18
mnasiadkafirst fail - timedatectl error (connection timed out), second try - mariadb is unreachable09:18
openstackgerritAna Peric proposed openstack/kolla-ansible master: Avoid errors in apparmor chrony and libvirt profile removal in Debian  https://review.opendev.org/c/openstack/kolla-ansible/+/77625209:20
*** LinPeiWen has joined #openstack-kolla09:21
*** e0ne has joined #openstack-kolla09:21
openstackgerritMark Goddard proposed openstack/kayobe master: Ubuntu: document overcloud development environment  https://review.opendev.org/c/openstack/kayobe/+/76979109:21
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Add output printing to host-command-run  https://review.opendev.org/c/openstack/kayobe/+/77647109:22
*** dougsz has quit IRC09:23
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/74239909:24
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add missing octavia-driver-agent  https://review.opendev.org/c/openstack/kolla-ansible/+/76187209:26
*** bengates has quit IRC09:28
*** bengates has joined #openstack-kolla09:29
*** gfidente has joined #openstack-kolla09:30
*** bengates has quit IRC09:32
*** bengates has joined #openstack-kolla09:32
*** gfidente is now known as gfidente|afk09:39
*** wuchunyang has joined #openstack-kolla09:44
*** jamesbenson has joined #openstack-kolla09:45
*** jamesbenson has quit IRC09:50
openstackgerritMerged openstack/kolla-ansible master: Improve trove  releasenote  https://review.opendev.org/c/openstack/kolla-ansible/+/77711309:57
*** LinPeiWen has quit IRC09:57
openstackgerritMark Goddard proposed openstack/kayobe master: Ubuntu: container image build  https://review.opendev.org/c/openstack/kayobe/+/77512909:58
*** jamesbenson has joined #openstack-kolla09:59
*** jamesbenson has quit IRC10:04
*** brinzhang has quit IRC10:04
*** brinzhang has joined #openstack-kolla10:05
*** dougsz has joined #openstack-kolla10:15
*** wuchunyang has quit IRC10:16
*** Fl1nt has joined #openstack-kolla10:26
Fl1ntGood morning everyone!10:27
oklhostmorning Fl1nt10:29
Fl1nto/ oklhost Did you fixed your problem with cURL?10:30
oklhostFl1nt: Yes, with -L the build went well10:30
*** zijlboot has quit IRC10:34
*** zijlboot has joined #openstack-kolla10:34
*** jamesbenson has joined #openstack-kolla10:35
oklhostFl1nt: you mentioned the project needs more debian/ubuntu reviewers, maybe I can do that? is there some guidance for reviewers?10:36
oklhostwe're running ubuntu in production10:37
Fl1ntah sweeet!10:38
*** jamesbenson has quit IRC10:39
Fl1ntI would say the contribut guide is a first good read and then, ask complementary questions in here :D10:39
Fl1nthttps://docs.openstack.org/kolla-ansible/latest/contributor/index.html10:39
*** dougsz has quit IRC10:39
openstackgerritMark Goddard proposed openstack/kayobe master: Ubuntu: update Apt cache before package installation  https://review.opendev.org/c/openstack/kayobe/+/77513010:40
mgoddardoklhost: reviews always appreciated. There's some info here: https://docs.openstack.org/kolla/latest/contributor/code-reviews.html10:41
oklhostThanks! Well, I will try to understand everything... which I didn't a few month ago. Thats the reason why I never submitted any blueprints of my code to deploy adjutant w/ kolla/kolla-ansible ;) gerrit etc. is pretty new to me and sounds a bit complicated at the first view, for a newbie like me. I do not want to annoy anyone with stupid questions ;)10:47
mgoddardoklhost: no question too stupid :)10:48
*** k_mouza has joined #openstack-kolla10:48
oklhostmgoddard: nice to hear. I'll give it a try than :) It would be an honor to help.10:49
hrwmorning10:52
openstackgerritMark Goddard proposed openstack/kolla-ansible master: docs: add code reviews to contributor guide  https://review.opendev.org/c/openstack/kolla-ansible/+/77735610:52
Fl1ntoklhost, regarding gerrit, feel free to ask, it's a bit different from what we're all used to with classic github/git process, but once you get the minimal amount of knowledge it's pretty streamlined.10:54
*** LinPeiWen has joined #openstack-kolla10:54
oklhostFl1nt: thanks! :)10:55
*** k_mouza has quit IRC10:55
*** k_mouza has joined #openstack-kolla10:56
*** jamesbenson has joined #openstack-kolla10:59
*** eliaswimmer has joined #openstack-kolla11:01
*** k_mouza_ has joined #openstack-kolla11:03
*** jamesbenson has quit IRC11:04
*** k_mouza has quit IRC11:06
openstackgerritPierre Riteau proposed openstack/kayobe master: Report available entropy  https://review.opendev.org/c/openstack/kayobe/+/77736511:15
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Fix copying Swift ring files  https://review.opendev.org/c/openstack/kayobe/+/77390011:17
openstackgerritMark Goddard proposed openstack/kayobe stable/train: DNM: Test Swift  https://review.opendev.org/c/openstack/kayobe/+/77390311:18
*** e0ne has quit IRC11:20
*** e0ne has joined #openstack-kolla11:22
*** bengates has quit IRC11:23
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add missing octavia-driver-agent  https://review.opendev.org/c/openstack/kolla-ansible/+/76187211:23
*** bengates has joined #openstack-kolla11:23
*** kevko has joined #openstack-kolla11:24
*** kevko_ has joined #openstack-kolla11:26
*** kevko_ has quit IRC11:27
*** kevko_ has joined #openstack-kolla11:28
*** kevko has quit IRC11:28
openstackgerritMark Goddard proposed openstack/kolla-ansible master: docs: add code reviews to contributor guide  https://review.opendev.org/c/openstack/kolla-ansible/+/77735611:32
*** k_mouza_ has quit IRC11:45
*** k_mouza has joined #openstack-kolla11:46
*** jamesbenson has joined #openstack-kolla11:50
*** jamesbenson has quit IRC11:54
openstackgerritMark Goddard proposed openstack/kayobe master: CI: display VM console log on ping or SSH failure  https://review.opendev.org/c/openstack/kayobe/+/77737211:55
*** jamesbenson has joined #openstack-kolla11:59
*** jamesbenson has quit IRC12:04
openstackgerritMark Goddard proposed openstack/kayobe stable/train: CI: use overlay2 storage driver for overcloud deploy  https://review.opendev.org/c/openstack/kayobe/+/77737312:07
*** k_mouza has quit IRC12:08
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Fix copying Swift ring files  https://review.opendev.org/c/openstack/kayobe/+/77390012:09
openstackgerritMark Goddard proposed openstack/kayobe stable/train: DNM: Test Swift  https://review.opendev.org/c/openstack/kayobe/+/77390312:10
*** jamesbenson has joined #openstack-kolla12:18
*** e0ne has quit IRC12:19
*** k_mouza has joined #openstack-kolla12:21
openstackgerritMark Goddard proposed openstack/kayobe master: Refactor network filter plugins to a Python module  https://review.opendev.org/c/openstack/kayobe/+/77717512:31
openstackgerritMark Goddard proposed openstack/kayobe master: Refactor veth configuration role variable generation  https://review.opendev.org/c/openstack/kayobe/+/77717612:31
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: Ubuntu: support systemd-networkd  https://review.opendev.org/c/openstack/kayobe/+/77650012:31
openstackgerritMark Goddard proposed openstack/kayobe stable/train: CI: use overlay2 storage driver for overcloud deploy  https://review.opendev.org/c/openstack/kayobe/+/77737312:31
*** gfidente|afk is now known as gfidente12:37
*** vishalmanchanda has joined #openstack-kolla12:46
openstackgerritcyrille babon proposed openstack/kolla master: add lsscsi due to change in os-brick  https://review.opendev.org/c/openstack/kolla/+/77714212:57
*** amoralej is now known as amoralej|lunch12:59
*** jamesbenson1 has joined #openstack-kolla12:59
*** jamesbenson has quit IRC13:02
openstackgerritcyrille babon proposed openstack/kolla master: add lsscsi due to change in os-brick  https://review.opendev.org/c/openstack/kolla/+/77714213:02
*** dougsz has joined #openstack-kolla13:15
*** LinPeiWen has quit IRC13:17
*** jamesbenson1 has quit IRC13:26
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Fix Cinder log parsing  https://review.opendev.org/c/openstack/kolla-ansible/+/77738113:27
*** dougsz has quit IRC13:31
*** k_mouza has quit IRC13:31
*** k_mouza has joined #openstack-kolla13:32
*** dougsz has joined #openstack-kolla13:38
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: Ubuntu: support systemd-networkd  https://review.opendev.org/c/openstack/kayobe/+/77650013:39
*** zul has joined #openstack-kolla13:41
*** dougsz has quit IRC13:43
*** Luzi has joined #openstack-kolla13:47
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Move from ceph-ansible to cephadm  https://review.opendev.org/c/openstack/kolla-ansible/+/77141813:48
*** e0ne has joined #openstack-kolla13:50
*** jamesbenson has joined #openstack-kolla13:52
*** jamesbenson has quit IRC13:57
*** dciabrin has quit IRC13:58
*** amoralej|lunch is now known as amoralej13:58
*** dciabrin has joined #openstack-kolla13:58
openstackgerritMark Goddard proposed openstack/kayobe master: Ubuntu: update Apt cache before package installation  https://review.opendev.org/c/openstack/kayobe/+/77513013:59
*** dciabrin has quit IRC14:03
openstackgerritPierre Riteau proposed openstack/kayobe master: Report available entropy  https://review.opendev.org/c/openstack/kayobe/+/77736514:04
*** rpittau is now known as rpittau|afk14:04
*** dciabrin has joined #openstack-kolla14:09
*** LinPeiWen has joined #openstack-kolla14:15
*** ivan_lin has quit IRC14:18
*** Luzi has quit IRC14:28
*** luksky has quit IRC14:36
*** jamesbenson has joined #openstack-kolla14:37
*** Luzi has joined #openstack-kolla14:43
*** dciabrin has quit IRC14:46
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt14:51
mgoddard^ meeting in 1014:51
*** jamesbenson has quit IRC14:56
*** jamesbenson has joined #openstack-kolla14:59
*** wuchunyang has joined #openstack-kolla14:59
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Feb 24 15:00:12 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
mgoddard\o15:00
headphoneJameso/15:00
wuchunyang\o15:00
yoctozepto\o/15:03
hrw]o[15:03
mgoddard#topic agenda15:03
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:03
kevko_hi :P15:03
mgoddard* Roll-call15:04
mgoddard* Announcements15:04
mgoddard* Review action items from the last meeting15:04
mgoddard* CI status15:04
mgoddard* Review requests15:04
mgoddard* yoctozepto: Stop caring about NTP/chrony15:04
mgoddard* Wallaby release planning15:04
mgoddard#topic announcements15:04
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:04
mgoddardNone from me15:04
mgoddardAnyone else?15:04
wuchunyangno15:04
mgoddard#topic Review action items from the last meeting15:05
*** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)"15:05
mgoddardThere were none15:05
mgoddard#topic CI status15:05
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:05
mgoddardKolla Train still broken15:07
mgoddardjust rechecked https://review.opendev.org/c/openstack/kolla/+/77460215:07
hrwuf. it was just pull limit15:08
*** ohorecny2 has joined #openstack-kolla15:08
mgoddardyeah15:09
mnasiadkaI've seen multiple occurences of jobs where one of the hosts fail on "Failed to connect to the host via ssh", but that's independent from us...15:09
mgoddardI started this to track mounting kayobe CI failures: https://etherpad.opendev.org/p/kayobe-ci-failures15:10
mnasiadkado we have ssh retries configured in Ansible?15:10
*** jamesbenson has quit IRC15:10
mnasiadkaSeems not really, I'll submit a patch15:10
mgoddardmnasiadka: is it zuul's ansible or ours?15:11
ohorecny2Hi all, I would like to ask if there are some plans to support Podman instead of Docker in case of kolla and kolla-ansible. Do community want to support it in future?15:12
*** dciabrin has joined #openstack-kolla15:12
mgoddardohorecny2: hi, we are currently in a meeting15:14
mnasiadkamgoddard: it's ours, during deploy15:14
mgoddardohorecny2: there is interest in the community for podman, but no one has started work on it yet. You might want to talk to Fl1nt about it15:15
mgoddardohorecny2: I also made a small PoC: https://github.com/stackhpc/kolla-ansible/commit/e44d4b028e3aa24955dd12271783287ae43a560315:15
mgoddardmnasiadka: we could try adding retries15:15
mnasiadkamgoddard: that's what I said - will raise a patch :)15:15
mgoddardmnasiadka: I was agreeing15:16
mnasiadkamgoddard: In darkest corners of my brain I couldn't see a situation somebody would disagree :)15:16
mnasiadkaBut thank you for agreeing :)15:16
mgoddardit will only help if the network issues are temporary15:17
mgoddard#topic Review requests15:19
*** openstack changes topic to "Review requests (Meeting topic: kolla)"15:19
mgoddardDoes anyone have a patch they would like to be reviewer15:19
mgoddard*reviewed15:19
*** wuchunyang has quit IRC15:19
headphoneJameshttps://review.opendev.org/c/openstack/kolla-ansible/+/74134015:19
*** wuchunyang has joined #openstack-kolla15:19
yoctozeptoall WIP15:19
headphoneJamesThe Lets Encrypt patch is ready, still putting the test together15:20
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add ssh retries and disable group names validation  https://review.opendev.org/c/openstack/kolla-ansible/+/77740215:20
wuchunyangwhat 's the plan  of the octavia management network patch ? i have raised it to email . but no help for me.15:21
ohorecny2@mgoddard: thank you for reply, I will contact @Fl1nt15:22
mgoddardwuchunyang: bbezak has been investigating15:22
mgoddardwuchunyang: will see if he's available to discuss15:22
mgoddardheadphoneJames: I'll add RP+115:22
*** jamesbenson has joined #openstack-kolla15:22
mgoddardseems like bbezak is busy15:23
kevko_mgoddard: yes, i want to review , but you know :) ..just joking15:23
wuchunyangok, BTW, who is bbezak . sorry  i don't know him.15:24
mgoddardwuchunyang: bbezak found that qemu-kvm package is the cause.15:24
mnasiadkaI've been investigating with him15:24
mgoddardok, mnasiadka can update15:24
wuchunyangnice. that's enough to me.. haha15:24
mgoddardwuchunyang: bbezak is my colleague, Bartosz Bezak15:24
wuchunyangthanks very much.15:24
mnasiadkaqemu-kvm 5.1 brings the issue, but it seems it's only problematic if the VM is Ubuntu and the flavor is like 1vcpu 2-4G RAM15:24
mnasiadkaon octavia CentOS 8 test image it works properly15:25
mnasiadkaon bigger flavors it also works with Ubuntu15:25
mnasiadkaand the issue is crashing the VM while adding second port (for vrrp)15:25
mnasiadkabbezak will raise a bug in Nova/qemu, and do a followup on the mailing list15:26
mnasiadkain the meantime, we could pin qemu-kvm to 4.2 in CentOS images15:26
mnasiadkawuchunyang/mgoddard: do we have the same problem on Ubuntu hypervisors?15:26
wuchunyangno15:27
wuchunyangjust centos 815:27
bbezakSorry I'afk. Will update this change and reply to openstack discuss group15:27
bbezakI'm afk15:27
mnasiadkabbezak: that's what I wanted to write, that you'll also update the change :)15:27
wuchunyangyou said if i use centos image. it should work around this problem. right ?15:28
mgoddardyes15:28
wuchunyangnice. Let's work around for CI first.15:29
*** muhaha has joined #openstack-kolla15:29
mgoddardok15:30
wuchunyangthanks for mgoddard mgoddard mgoddard15:30
mgoddardmgoddard^315:30
mgoddard#topic yoctozepto: Stop caring about NTP/chrony15:30
*** openstack changes topic to "yoctozepto: Stop caring about NTP/chrony (Meeting topic: kolla)"15:30
mgoddardyoctozepto stopped caring about time15:31
yoctozeptoyup15:31
yoctozeptothat's the plan15:31
yoctozepto8-)15:31
mnasiadkayou mean finally deprecate chrony container? :)15:31
yoctozeptojoking aside, this is what we have planned15:31
yoctozeptoyes, mnasiadka :P15:32
yoctozeptowe are moving this cycle-cycle15:32
yoctozeptotime to enact :D15:32
mnasiadkathank to gods of total time chaos15:32
yoctozeptoit is time to drop time15:33
mgoddardseriously though, what is the plan?15:33
*** jbadiapa has joined #openstack-kolla15:35
mgoddardyoctozepto?15:38
mgoddardanyone?15:38
wuchunyang?15:39
mgoddardseems like we have lost the leader of this discussion. Let's come back to it later15:40
mgoddard#topic Wallaby release planning15:40
*** openstack changes topic to "Wallaby release planning (Meeting topic: kolla)"15:40
mgoddardFriendly reminder: Kolla feature freeze: Mar 29 - Apr 0215:40
yoctozeptomgoddard: sorry, yes, funky connection15:41
yoctozeptoyou can't see me gone because of my bouncer15:41
yoctozeptoanyhow15:41
mgoddardok15:41
mgoddard#undo15:41
openstackRemoving item from minutes: #topic Wallaby release planning15:41
yoctozeptothanks15:42
yoctozeptoso15:42
yoctozeptoafair, kayobe handles ntp on the host side15:42
yoctozeptoand any default install configures ntp nowadays15:42
yoctozeptofor a myriad of reasons15:42
yoctozeptopoor man's option is to recommend an ansible module to manage ntp15:43
yoctozepto(I guess kayobe uses one already, can go with it)15:43
yoctozeptoluxurious option would be to integrate it with kolla-ansible15:43
yoctozeptobut it is blurry15:43
yoctozeptoand I think it belongs to the level of tools like kayobe15:44
mgoddardkayobe actually removed support for NTP config since CentOS 815:45
yoctozeptowe have timesync prechecks in already15:45
mgoddardand relies on Kolla Ansible :)15:45
yoctozeptomgoddard: oh gosh15:45
mgoddardhowever, we'll probably change change approach15:45
mgoddardand configure chrony on CentOS, possibly systemd-resolved or chrony on Ubuntu15:46
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: switch to CentOS 8 Stream  https://review.opendev.org/c/openstack/kolla/+/77284115:46
hrwwhy not systemd-resolved on all distros?15:46
mgoddardso I don't think it affects this discussion15:46
mgoddardhrw: chrony is standard out of the box on centos15:46
mgoddardhold on15:47
mgoddardsorry, systemd-timesyncd15:47
*** admin0 has joined #openstack-kolla15:47
Fl1ntI'm in sorry ^^15:47
mgoddardtoo many systemd-*d in ubuntu :)15:47
yoctozeptoyup15:47
yoctozeptoall right then15:48
yoctozeptoso we have the timesync prechecks since victoria15:48
yoctozeptohttps://docs.openstack.org/releasenotes/kolla-ansible/victoria.html15:48
yoctozeptoas I mentioned15:48
yoctozeptoso the sanity checks are there in case users want to go rogue 8-)15:48
mgoddardI suppose my only concern is for non-kayobe users who want more than the out of the box NTP config15:48
yoctozeptowe can adapt docs15:48
yoctozeptoslap renos15:48
yoctozeptogo ml15:48
yoctozeptoyeah, we would point them to ansible roles for that15:49
yoctozeptogalaxy should have something nice15:49
yoctozeptofwiw, I have this part in puppet15:49
yoctozepto<no judging>15:49
Fl1ntsystemd-*d are not too many as it is a way to address concerns to people that can't stop complaining that systemd is a monolithic monster that isn't respecting unix principles :p15:50
yoctozeptowe have *not* officially deprecated chrony15:50
yoctozeptoso we should do a deprec this cycle15:50
Fl1ntwhich alternative would you like to use?15:50
yoctozeptodeprec on both kolla and k-a15:51
yoctozeptoand with recommendations15:51
yoctozeptoFl1nt: give recommendations and leave it to the users15:51
yoctozeptontp is nothing extraordinary nowadays15:52
Fl1ntyou mean, we let it free for user as we're now relying on hosts right?15:52
Fl1ntmgoddard, I synced with ohorecny2 about podman btw15:53
mgoddardFl1nt: ok, we can discuss in a min if you want15:54
mgoddarddoes anyone have opinions on the proposed approach to NTP?15:54
Fl1ntTBN: as we talk about wallaby, in order for me to work on SAML2 keystone SSO, I need the OIDC patch to be merged on master if everything is fine as I'll base my patch on it.15:54
yoctozeptoFl1nt: what hosts?15:54
yoctozeptoFl1nt: we are talking time synchronization15:54
Fl1ntyes15:54
mgoddardFl1nt: it's merged15:54
Fl1ntmgoddard, oooh sweet, perfect!15:55
yoctozeptomgoddard: do you have any opinions?15:55
Fl1ntyoctozepto, HW host that provide time to containers services?15:55
mgoddardyoctozepto: I gave one15:56
mgoddardyoctozepto: I think we should solicit feedback from the community15:56
Fl1ntpersonally I can use chrony or systemd-timesyncd no issue as long as it works ^^15:56
yoctozeptomgoddard: ack15:56
yoctozeptoFl1nt: leaving ntp to users15:57
Fl1ntyes15:57
mgoddardan NTP client isn't difficult to setup, but it might be a bit annoying for users currently relying on k-a for it15:57
kevko_can I just ask why you want to remove chrony/ntp from kolla ?15:58
mgoddardbut if there are no strong objections from the community then I'm fine with it15:58
mgoddardgood question kevko_ :)15:58
kevko_:D15:58
kevko_so, what is the answer ? :)15:59
Fl1nthonestly, I don't have strong opinion on this one, leave it or remove it I don't really care as long as services are able to get time, personnally all my hosts are ansible bootstraped using chrony for now but don't really care.15:59
Fl1ntto switch to something else16:00
yoctozeptokevko_: mostly because we are forced to handle whatever the host throws at us16:00
yoctozeptoand it's extra code to maintain16:00
yoctozeptofor no real benefit16:00
yoctozeptoit's not like kolla-ansible brings new ntp features to the table :-)16:01
Fl1ntgood reason, if it doesn't add any feature or advantage, just remove it.16:01
mgoddardwell hold on16:01
mgoddardkolla-ansible is about automation, not augmenting network time protocols16:01
Fl1ntyes16:02
kevko_well, I don't thing that especially some small chrony image (and k-a part) is difficult to maintain16:02
rockeynot to throw a log on the fire, but doesn't k-a still need to do routine checks on ntpd's on system before relying on the hosts either way? k-a is dependant on synced time, laying this out on the enduser could backfire, right?16:02
Fl1ntbut the point is, time is already automated pretty much everywhere in enterprise by either the provisioning service or another bootstrap/base ansible internal playbook or puppet or whatever16:02
mgoddardso if installing and configuring an NTP client makes users lives easier then that is arguably beneficial16:02
*** ivan_lin has joined #openstack-kolla16:03
mgoddardrockey: we did add a check that time is synced16:03
Fl1ntare you talking about the chrony container or the host bootstrap configuration task ?16:03
mgoddardansible/roles/prechecks/tasks/timesync_checks.yml16:03
*** dougsz has joined #openstack-kolla16:03
mgoddardFl1nt: probably both16:04
yoctozeptoyeah, the point is that most hosts come with some ntp daemon running16:04
mgoddardI think for various reasons, we can probably agree that running chrony in a container isn't a great idea16:04
yoctozeptoand we spend time fixing bugs on detecting those or adapting security profiles16:04
mgoddardbut mainly what because yoctozepto just said16:04
yoctozeptoand it's a very basic service nowadays16:04
yoctozeptoyeah, mgoddard16:04
kevko_agree, I also added some patch ..16:05
mgoddardso I'm definitely +1 to removing the container16:05
kevko_so, remember16:05
Fl1ntcontainer is probably too much, automate the ntp configuration at task level can still be let as is using chrony or up to the user as long as the appropriate binding is made within services containers.16:05
*** LinPeiWen has quit IRC16:05
mgoddardthe question for me is whether it is useful to manage a service on the host16:05
mgoddardthe main use case being to use a custom NTP server16:05
yoctozeptomgoddard: yup, that's what the recommendations would be for16:05
yoctozeptodo note k-a never configured a server16:06
yoctozeptoso users had to do it elsewhere16:06
mgoddardI mean, configure the client to use a custom NTP server16:06
mgoddardthat was possible with the chrony container16:07
Fl1ntthat only make sense as you pointed out, if kolla/k-a is providing with a ntp server for the cluster, but it's highly unlikely that anyone on enterprise side of things will base their cluster on it as most of ITs or new installation do it on auxilliary/commander/bastion/whatever its named server.16:07
yoctozeptoyes, my point is that such users are already knowledgeable enough to do it with any ntp role :-)16:07
*** LinPeiWen has joined #openstack-kolla16:07
rockeymgoddard: well then, as a happy user, i'd +1 the removal too, mostly due to "that most hosts come with some ntp daemon running" statement from yoctozepto16:08
yoctozeptoayeee, we are past time16:08
mgoddardindeed16:08
yoctozeptoaction this on me16:08
yoctozeptoand let's close16:08
yoctozeptosorry for taking so much time16:08
mgoddard#action yoctozepto to ask openstack-discuss about NTP16:08
mgoddardThanks all16:08
mgoddard#endmeeting16:08
*** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:08
openstackMeeting ended Wed Feb 24 16:08:45 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:08
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-02-24-15.00.html16:08
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-02-24-15.00.txt16:08
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-02-24-15.00.log.html16:08
Fl1ntthanks ^^16:09
*** ivan_lin has quit IRC16:10
openstackgerritMark Goddard proposed openstack/kayobe master: Refactor veth configuration role variable generation  https://review.opendev.org/c/openstack/kayobe/+/77717616:10
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: Ubuntu: support systemd-networkd  https://review.opendev.org/c/openstack/kayobe/+/77650016:11
*** LinPeiWen has quit IRC16:11
*** LinPeiWen has joined #openstack-kolla16:11
*** k_mouza_ has joined #openstack-kolla16:14
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer  https://review.opendev.org/c/openstack/kolla-ansible/+/75428516:17
*** k_mouza has quit IRC16:17
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer  https://review.opendev.org/c/openstack/kolla-ansible/+/75428516:18
*** wuchunyang has quit IRC16:20
*** dougsz has quit IRC16:25
kevko_if noone is going to talk ..16:26
kevko_can I discuss my findings with keystone weird approach (was discussing these days here ..)16:26
*** vishalmanchanda has quit IRC16:36
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add ssh retries  https://review.opendev.org/c/openstack/kolla-ansible/+/77740216:42
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Add output printing to host-command-run  https://review.opendev.org/c/openstack/kayobe/+/77647116:53
Fl1ntkevko_, go ahead16:55
*** jamesbenson has quit IRC17:01
mnasiadkayoctozepto: where did we end up with Cinder Active/Active? Some rabbit hole? :)17:01
*** luksky has joined #openstack-kolla17:02
*** jamesbenson has joined #openstack-kolla17:02
kevko_Well, I found that if I restart keepalived, and VIP is switched to another controller WSGI Workers are busy and trying to connect to DB ..17:07
kevko_BUT keepalived sent GARP so ConnectionPool on host where keystone is invalid and I can see in strace for wsgi process that it is trying to 'select 1'17:08
kevko_if i kill wsgi process or restart apache ..everything is OK ..does anyone now how to fix it ?17:09
kevko_maybe it can be done by some kernel tuning or wsgidaemonprocess setting in apache ?17:09
kevko_I think this is problem not only for keystone ..but for OS services globally17:09
kevko_strace -> http://paste.openstack.org/show/802969/17:10
kevko_Fl1nt: ^^^17:12
*** k_mouza_ has quit IRC17:14
*** k_mouza has joined #openstack-kolla17:14
*** amoralej is now known as amoralej|off17:14
Fl1ntkevko_, interesting, how many clients and nodes on your cluster?17:14
kevko_Fl1nt: well, now I can reproduce it with bash which is sending token issue to keystone :D17:15
kevko_Fl1nt: did you see my debug video I post here few days ago ?17:15
Fl1ntnope ^^17:16
kevko_Fl1nt: let me send you17:16
kevko_Fl1nt: https://download.kevko.ultimum.cloud/video_debug.mp417:17
kevko_Fl1nt: on left I have controller0 on right controller1 ..in center windows i am running ip a (254 is vip) ..and when I switch VIP ..you can see keystone where VIP was before is stucked17:18
kevko_Did you catch the point ?17:19
mnasiadkakevko_: you mean that the connection to MariaDB through the VIP is not closed on keepalived switch?17:19
Fl1ntkevko_, I think your situation will (is?) fixed by the patch that lower the connection pool of services to 017:20
kevko_mnasiadka: well, yes ..but it's not that easy (i think)17:21
kevko_Fl1nt: nope, I've lowered pool to 0 and it is same17:22
Fl1nthum... I'll have a look on tomorrow ;-)17:23
Fl1ntas it's pretty late in here ^^17:23
kevko_mnasiadka: point is that if worker start to process request and VIP is switched in the middle..it is broken17:23
kevko_late ?17:24
kevko_where are u from :P17:24
mnasiadkakevko_: that we know for a long time :)17:24
kevko_mnasiadka: well, but I think it should be fixed :D ... I know that if i restart haproxy ..it will be ok ..but it is just workaround17:25
kevko_(or keystone itself)17:25
Fl1ntkevko_, late for work stuff :p17:26
kevko_but, i think that there has to be some solution17:26
Fl1ntI need to relax.17:26
kevko_Fl1nt: ah, ok .. This is my relax :)17:26
kevko_I was just wondering there should be some kernel tuning or maybe wsgi option to fix this behaviour17:26
openstackgerritMark Goddard proposed openstack/kayobe master: Ubuntu: update Apt cache before package installation  https://review.opendev.org/c/openstack/kayobe/+/77513017:33
*** bengates has quit IRC17:34
*** gfidente is now known as gfidente|afk17:40
*** jonaspaulo has joined #openstack-kolla17:42
*** jamesbenson has quit IRC17:47
*** jamesbenson has joined #openstack-kolla17:50
*** jamesbenson has quit IRC17:52
*** jamesbenson has joined #openstack-kolla17:52
*** Luzi has quit IRC17:56
*** bengates has joined #openstack-kolla17:57
*** jamesbenson has quit IRC17:58
*** maharg101 has quit IRC18:00
*** bengates has quit IRC18:01
*** bkslash has joined #openstack-kolla18:03
*** bkslash has quit IRC18:04
*** bkslash has joined #openstack-kolla18:13
*** jonaspaulo has quit IRC18:15
*** bkslash has quit IRC18:15
*** muhaha has quit IRC18:21
*** Fl1nt has quit IRC18:30
*** andrewbonney has quit IRC18:42
*** maharg101 has joined #openstack-kolla18:53
*** k_mouza has quit IRC19:00
*** bengates has joined #openstack-kolla19:13
*** cah_link has quit IRC19:17
*** eliaswimmer has quit IRC19:18
*** bengates has quit IRC19:20
*** eliaswimmer has joined #openstack-kolla19:51
*** bengates has joined #openstack-kolla19:57
*** bengates has quit IRC20:01
*** k_mouza has joined #openstack-kolla20:14
*** k_mouza has quit IRC20:19
*** zzzeek has quit IRC20:34
*** zzzeek has joined #openstack-kolla20:34
openstackgerritPierre Riteau proposed openstack/kayobe stable/train: CI: use overlay2 storage driver for overcloud deploy  https://review.opendev.org/c/openstack/kayobe/+/77737320:43
*** maharg101 has quit IRC20:54
*** eliaswimmer has quit IRC20:57
*** eliaswimmer has joined #openstack-kolla21:08
*** LaurentDumont has quit IRC21:17
*** LaurentDumont has joined #openstack-kolla21:19
*** kevko_ has quit IRC21:24
*** zzzeek has quit IRC21:26
*** zzzeek has joined #openstack-kolla21:28
*** zzzeek has quit IRC21:49
*** zzzeek has joined #openstack-kolla21:53
*** eliaswimmer has quit IRC21:58
*** ohorecny2 has quit IRC22:12
*** bengates has joined #openstack-kolla22:38
*** luksky has quit IRC22:39
*** bengates has quit IRC22:49
*** maharg101 has joined #openstack-kolla22:51
*** maharg101 has quit IRC22:56
*** zzzeek has quit IRC23:14
*** zzzeek has joined #openstack-kolla23:15
*** e0ne has quit IRC23:17
*** ivan_lin has joined #openstack-kolla23:22
*** LinPeiWen has quit IRC23:25

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!