Wednesday, 2021-03-03

*** Underknowledge has quit IRC01:08
*** Underknowledge has joined #openstack-kolla01:08
*** Underknowledge2 has joined #openstack-kolla01:51
*** Underknowledge has quit IRC01:54
*** Underknowledge2 is now known as Underknowledge01:54
*** admin0 has quit IRC02:04
openstackgerritwu.chunyang proposed openstack/kolla master: DNM: test1  https://review.opendev.org/c/openstack/kolla/+/77834802:04
openstackgerritwu.chunyang proposed openstack/kolla master: DNM: test2  https://review.opendev.org/c/openstack/kolla/+/77834902:04
*** r3ap3r has quit IRC02:29
*** r3ap3r has joined #openstack-kolla02:33
*** Underknowledge has quit IRC02:38
*** Xuchu_ has joined #openstack-kolla02:43
*** Xuchu has quit IRC02:46
*** iniazi has joined #openstack-kolla03:09
*** LinPeiWen has quit IRC03:21
*** dasp has quit IRC03:34
*** suff has joined #openstack-kolla03:44
*** dasp has joined #openstack-kolla03:45
*** k_mouza has joined #openstack-kolla04:00
*** k_mouza has quit IRC04:05
*** shyamb has joined #openstack-kolla04:16
*** zzzeek has quit IRC04:17
*** zzzeek has joined #openstack-kolla04:17
*** zzzeek has quit IRC04:22
*** zzzeek has joined #openstack-kolla04:23
*** LinPeiWen has joined #openstack-kolla05:02
*** vishalmanchanda has joined #openstack-kolla05:27
*** cah_link has joined #openstack-kolla05:36
*** brinzhang has joined #openstack-kolla05:41
oklhostmorning05:50
*** zzzeek has quit IRC06:06
*** wuchunyang has joined #openstack-kolla06:10
*** Xuchu_ has quit IRC06:22
*** Xuchu has joined #openstack-kolla06:22
*** shyamb has quit IRC06:24
*** mchlumsky has quit IRC06:28
*** mchlumsky has joined #openstack-kolla06:28
*** wuchunyang has quit IRC06:36
*** zzzeek has joined #openstack-kolla06:41
*** mrunge has quit IRC06:56
*** mrunge has joined #openstack-kolla06:57
openstackgerritLin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for redis services  https://review.opendev.org/c/openstack/kolla-ansible/+/77837107:07
*** luksky has joined #openstack-kolla07:19
*** waxfire0 has joined #openstack-kolla07:35
*** waxfire has quit IRC07:35
*** waxfire0 is now known as waxfire07:35
*** rpittau|afk is now known as rpittau07:52
*** bengates has joined #openstack-kolla08:03
*** bengates has quit IRC08:03
*** bengates has joined #openstack-kolla08:04
*** andrewbonney has joined #openstack-kolla08:14
*** wuchunyang has joined #openstack-kolla08:20
*** amoralej|off is now known as amoralej08:22
*** wuchunyang has quit IRC08:24
*** amoralej has joined #openstack-kolla08:25
*** wuchunyang has joined #openstack-kolla08:31
*** alexandreperreau has quit IRC08:34
hrwmorning08:35
*** dougsz has joined #openstack-kolla08:59
*** bengates_ has joined #openstack-kolla09:05
wuchunyangmorning09:05
*** bengates has quit IRC09:09
*** Underknowledge has joined #openstack-kolla09:11
*** shyamb has joined #openstack-kolla09:16
*** shyam89 has joined #openstack-kolla09:16
LinPeiWenmorning09:21
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: octavia: support tenant management network  https://review.opendev.org/c/openstack/kolla-ansible/+/75558909:24
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: CI: octavia: remove octavia from magnum scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/75428509:24
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer  https://review.opendev.org/c/openstack/kolla-ansible/+/77839009:24
*** Xuchu has quit IRC09:43
openstackgerritLin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for skydive services  https://review.opendev.org/c/openstack/kolla-ansible/+/77839309:46
*** shyam89 has quit IRC10:11
*** shyamb has quit IRC10:11
mgoddardmorning10:21
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: Use Docker healthchecks for watcher services  https://review.opendev.org/c/openstack/kolla-ansible/+/77840310:23
hrwmgoddard: I declare train to be dead by pull limits10:23
hrwhttps://review.opendev.org/c/openstack/kolla/+/774602 is unable to pass10:24
mgoddardhrw: pull limits are temporary10:24
hrw8 failures in a row10:24
mgoddardapart from the one that passed?10:25
hrwyesterday I did recheck at 23:22 when we had no CI jobs10:25
hrwfailed.10:25
mgoddardupgrade jobs use stein images which are less likely to be cached10:25
mgoddardwe are not the only users of the registry mirror10:25
mgoddardalthough I do have my suspicions about the caching of images in the limestone region - every pull limit failure I've seen in kayobe has been in limestone10:26
*** wuchunyang has quit IRC10:28
*** k_mouza has joined #openstack-kolla10:36
openstackgerritLin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for octavia services  https://review.opendev.org/c/openstack/kolla-ansible/+/77818010:36
*** shyam89 has joined #openstack-kolla10:37
*** shyamb has joined #openstack-kolla10:37
hrwok10:50
*** luksky has quit IRC11:07
*** luksky has joined #openstack-kolla11:08
*** luksky has quit IRC11:09
openstackgerritLin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for octavia services  https://review.opendev.org/c/openstack/kolla-ansible/+/77818011:09
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use Docker healthchecks for cyborg services  https://review.opendev.org/c/openstack/kolla-ansible/+/77821211:10
*** kevko has joined #openstack-kolla11:14
*** kevko_ has joined #openstack-kolla11:16
*** kevko has quit IRC11:16
openstackgerritLin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for senlin services  https://review.opendev.org/c/openstack/kolla-ansible/+/77841311:23
openstackgerritMerged openstack/kayobe master: Report available entropy  https://review.opendev.org/c/openstack/kayobe/+/77736511:25
*** luksky has joined #openstack-kolla11:25
*** kevko_ has quit IRC11:31
*** kevko has joined #openstack-kolla11:31
kevkoyoctozepto: sorry, you asked how many seconds it is before kernel kill non-active tcp connections when set 3,4,5 (keepalive thing)11:32
kevkoyoctozepto: here it is -> https://pracucci.com/linux-tcp-rto-min-max-and-tcp-retries2.html11:32
kevko3 = cca 1.4s , 4 = 3sec , 5 = 6.2 sec11:33
openstackgerritPierre Riteau proposed openstack/kayobe master: Fix documentation of control host bootstrap  https://review.opendev.org/c/openstack/kayobe/+/77844211:55
*** shyam89 has quit IRC12:00
*** shyamb has quit IRC12:00
openstackgerritMerged openstack/kayobe master: Fix documentation of control host bootstrap  https://review.opendev.org/c/openstack/kayobe/+/77844212:31
*** amoralej is now known as amoralej|lunch13:15
kevkoyoctozepto: here ?13:21
kevkoyoctozepto: i would like to take this -> https://review.opendev.org/c/openstack/kolla-ansible/+/728796/ and rework little bit (not use new group ..but I think it's more clear to add flag if arbiter or not)13:22
kevkoas you said in last commend ..13:22
yoctozeptokevko: you are very welcome to13:35
*** wuchunya_ has joined #openstack-kolla13:35
kevkoyoctozepto: should I take and update review or create new one ?13:44
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: Use Docker healthchecks for watcher services  https://review.opendev.org/c/openstack/kolla-ansible/+/77840313:45
*** e0ne has joined #openstack-kolla13:45
kevkoyoctozepto: because i'm going to commit on top of my mariadb role refactor ..13:45
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer  https://review.opendev.org/c/openstack/kolla-ansible/+/77839013:51
kevkoyoctozepto: + i've already replied to your question from yesterday above ^^13:52
kevkoyoctozepto: regarding to that ..what do you mean ..which value should be middle gold ?13:53
*** amoralej|lunch is now known as amoralej13:56
openstackgerritPierre Riteau proposed openstack/kayobe stable/victoria: Update IPA docs and test build with extra-hardware  https://review.opendev.org/c/openstack/kayobe/+/77694714:09
openstackgerritLin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for gnocchi services  https://review.opendev.org/c/openstack/kolla-ansible/+/77846714:12
yoctozeptokevko: for any interactive service 3-4 sounds like a reasonable default14:13
kevkoyoctozepto: i think 3 is good ..it is abourt 1.5 sec which is quite big window also on "small" networks ..14:13
kevkoyoctozepto: i've sent an article where you can find a table with times ...14:14
yoctozeptoairship went with 514:14
yoctozeptohonestly, beyond 5 it is very likely to hit other timeouts14:14
kevkoyoctozepto: like what timeouts ?14:16
kevkoyoctozepto: these option is for packets which were not acknowledged ..14:16
kevkoand trying to retransmit14:17
yoctozeptokevko: application-level timeouts14:40
yoctozeptoI mean in general14:40
kevkoyoctozepto: well problem is that it was not hit by app-level14:43
kevkoyoctozepto: anyway, i think 3 is OK + add option to configure yourself14:44
openstackgerritDoug Szumski proposed openstack/kolla master: Upgrade from ELK6 to ELK7 FOSS release  https://review.opendev.org/c/openstack/kolla/+/73856714:47
mgoddardmeeting in 1014:51
mgoddard^ mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt14:51
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Upgrade service configuration for ELK 7  https://review.opendev.org/c/openstack/kolla-ansible/+/74098614:59
mgoddard#startmeeting kolla15:01
openstackMeeting started Wed Mar  3 15:01:56 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
*** openstack changes topic to " (Meeting topic: kolla)"15:01
openstackThe meeting name has been set to 'kolla'15:02
mgoddard#topic rollcall15:02
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:02
mgoddard\o15:02
hrwo][o15:02
parallaxo/15:02
wuchunya_o15:02
dougsz{o15:02
headphoneJameso/15:02
kplanto715:02
priteau|o|15:03
mgoddard|-o-|15:03
mgoddard#topic agenda15:04
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:04
mgoddard* Roll-call15:04
mgoddard* Announcements15:04
mgoddard** Combined TC/PTL nomination open http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020811.html15:04
mgoddard** OpenStack feature freeze next week15:04
mgoddard* Review action items from the last meeting15:04
mgoddard* CI status15:04
mgoddard* Review requests15:04
mgoddard* PoC: image build & test pipeline (https://review.opendev.org/c/openstack/kolla/+/777796 and https://review.opendev.org/c/openstack/kolla-ansible/+/777946)15:04
mgoddard* Wallaby release planning15:04
mgoddard#topic Announcements15:04
*** openstack changes topic to "Announcements (Meeting topic: kolla)"15:04
mgoddard#info Combined TC/PTL nomination open15:04
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020811.html15:04
mgoddardAnyone is welcome to run for Kolla PTL15:05
mgoddard#info OpenStack feature freeze next week15:06
*** LinPeiWen has quit IRC15:06
mgoddard#link https://releases.openstack.org/wallaby/schedule.html15:06
mgoddard#info Kolla feature freeze will be Mar 29 - Apr 0215:06
mgoddardAny other announcements?15:07
mgoddard#topic  Review action items from the last meeting15:07
*** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)"15:07
mgoddardyoctozepto to ask openstack-discuss about NTP15:08
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020707.html15:08
mgoddardThanks yoctozepto15:08
mgoddard#topic CI status15:08
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:08
mgoddardKolla train still broken15:09
mgoddardThe fix keeps getting hit by dockerhub pull limits15:09
mgoddardhttps://review.opendev.org/c/openstack/kolla/+/77460215:09
mgoddardwe had an issue with neutron-server builds on master, but it was fixed by https://review.opendev.org/c/openstack/kolla/+/77799215:10
mgoddardunclear whether it affects other branches15:11
mgoddardkolla-ansible NFV CI job seems to be failing on master15:11
mgoddardhttps://ac90fbbc9cd1b2f919e7-c0288c15cf27fe5a39c9948ecafb7329.ssl.cf1.rackcdn.com/778179/1/check/kolla-ansible-centos8-source-scenario-nfv/e3b78d7/secondary1/logs/docker_logs/tacker_conductor.txt15:11
mgoddardtacker processes fail to import toscaparser15:11
wuchunya_i think this is a tacker bug15:12
mgoddardis it not a missing package in our image?15:12
wuchunya_i will add the package requirement to the  tacker project.15:13
yoctozepto\o/15:13
wuchunya_this package should be in tacker requirements.15:13
yoctozeptomgoddard: it is but tacker should be listing it15:13
*** happyhemant has joined #openstack-kolla15:13
mgoddardhttps://pypi.org/project/tosca-parser/15:13
yoctozeptoyeah, just what wuchunya_ is saying15:13
mgoddardprobably, unless it is an optional dep15:14
*** wuchunya_ is now known as wuchunyang15:14
yoctozeptoit's not the first time tacker is clumsy15:14
yoctozeptoso that's a good question15:15
mgoddard#action wuchunyang to propose toscaparser in tacker requirements to fix NFV job15:15
wuchunyangok , no problem15:15
mgoddardthanks15:16
mgoddardanyone want to review https://review.opendev.org/c/openstack/kolla-ansible/+/761519/ to enable rabbitmq TLS in CI?15:17
yoctozeptodone15:18
yoctozeptoclever plug ;p15:19
mgoddardtacker plot thickens: requirements.txt has tosca-parser>=1.6.0 # Apache-2.015:19
*** luksky has quit IRC15:19
*** luksky has joined #openstack-kolla15:19
yoctozeptohmm, that's intriguing15:19
*** luksky has quit IRC15:19
yoctozeptoperhaps it's run in a different env?15:19
wuchunyangthe package name is nfv-toscaparser15:19
mgoddardthere are two packages15:20
yoctozeptothis should be the one though:15:20
yoctozeptohttps://github.com/openstack/tosca-parser15:20
*** muhaha has joined #openstack-kolla15:20
yoctozeptohmm15:20
mgoddardnfv-toscaparser looks old, last release 1.1.115:20
mgoddardhttps://pypi.org/project/tosca-parser/15:21
mgoddardanyway, we don't all need to solve it15:21
mgoddardKayobe CI had a couple of improvements this week15:21
mgoddardbare metal testing reliability should be improved15:22
mgoddardwe still hit pull limits15:22
mgoddardmost often in the limestone region15:22
mgoddard#topic Review requests15:23
*** openstack changes topic to "Review requests (Meeting topic: kolla)"15:23
mgoddardDoes anyone have a patch they would like to be reviewed this week?15:23
parallaxPossibly this small one: https://review.opendev.org/c/openstack/kolla-ansible/+/77422215:23
mgoddardadded RP+115:24
yoctozeptohttps://review.opendev.org/c/openstack/kolla-ansible/+/76795015:24
yoctozepto'tis still rotting ;p15:24
dougszELK7 upgrade patches should be fairly easy15:25
mgoddardRP+1 all round15:27
mgoddardit's cheating a little bit, but I did some reviews of the healthcheck patches today, so I'll plug those https://review.opendev.org/q/topic:%22container-health-check%22+(status:open%20OR%20status:merged)15:28
mgoddardwould be nice to finish that one off15:28
mgoddard#topic PoC: image build & test pipeline15:28
*** openstack changes topic to "PoC: image build & test pipeline (Meeting topic: kolla)"15:28
mgoddardThis is mine15:29
mgoddardAfter fixing some kayobe CI issues last week, the next biggest obstacle to stability is dockerhub pull limits15:29
mgoddardWe've made changes that make CI usable, but it is still annoying when it fails from time to time15:30
mgoddardSo I thought I would put some effort into working out how to use the opendev container registry15:30
mgoddardGiven that we may see this as a potential fix for our dockerhub woes15:31
mgoddard#link https://review.opendev.org/c/openstack/kolla/+/77779615:31
mgoddard#link https://review.opendev.org/c/openstack/kolla-ansible/+/77794615:31
mgoddardthe commit message tries to give a high level overview15:32
mgoddardit's based on this setup:15:32
mgoddard#link https://docs.opendev.org/opendev/base-jobs/latest/docker-image.html#a-repository-with-producers-and-consumers15:33
mgoddardand allows different jobs to produce and consume container images15:33
mgoddardthe PoC has one job that builds images, then pushes them to a registry15:33
mgoddardand another job that pulls images from the registry and tests them15:33
mgoddarda key part here being that dockerhub is not involved (much)15:34
openstackgerritPierre Riteau proposed openstack/kayobe master: Change docker_registry network_mode to host  https://review.opendev.org/c/openstack/kayobe/+/76037115:34
mgoddardis anyone listening?15:34
hrwsounds good15:35
dougszACK15:35
priteauEveryone is looking at the changes :)15:35
wuchunyangyes15:35
mgoddardI'll give you a few minutes15:36
*** luksky has joined #openstack-kolla15:36
dougszI'm sure this has been asked before, but it wasn't possible to request unlimited pulls from Docker hub?15:38
yoctozeptodougsz: we have to pay with blood15:38
hrwmore or less we are doomed15:38
yoctozeptoor soul15:38
dougszcool, I see :)15:38
hrwI have read description of kolla patch and we may end with pushing GBs of data between CI nodes15:39
hrwlife sucks15:39
yoctozeptoshucks15:39
mgoddardright15:40
mgoddardthat is one of my main concerns15:40
mgoddardthere are two tiers of registry involved15:40
mgoddardthe buildset registry, a temporary node running in another job. I believe this should be on the same cloud (but not certain)15:41
mgoddardthe intermediate registry15:41
mgoddard^ there is only one of these, and it lives in rackspace15:41
mgoddardfor $reasons, images generally get pushed to both registries15:41
hrwor each k-a job does: start registry, build images and push to local, do own job, destroy15:41
hrwthat way no data send but all jobs take longer15:42
mgoddardno, build and deploy are in separate jobs15:42
mgoddardwell, what you describe is what we have already15:42
hrw+ caching registry in each opendev cloud to not fetch debian/centos/ubuntu base image15:43
hrwthis way we touch docker hub only in publish jobs15:43
mgoddardthere are quite a few options for how it would work15:45
mgoddardI suppose we ought to try to list them, and work out which ones fit with the changes we want to make15:46
hrwand I assume that opendev already asked dockerhub to get 'unlimited pull' and got rejected15:46
mgoddardit's possible we could just publish to and pull from the infra registry as well as dockerhub, and keep everything else the same15:46
mgoddardsee earlier dicussion about soul and blood15:47
mgoddardif we think this option looks good, then we probably need to have a conversation with opendev infra team15:48
mgoddardbut while poking around in the opendev config, I found option B15:48
mgoddardthe registry mirrors in opendev are not the official docker registry, just an apache caching proxy15:49
yoctozeptooh, that's bad15:50
mgoddard#link https://opendev.org/opendev/system-config/src/commit/4310315afe27c040b239a72a1c248ddabf7fdfa5/playbooks/roles/mirror/templates/mirror.vhost.j2#L45315:50
mgoddardwhich means that they are able to support quay.io15:51
mgoddardthe lack of a registry mirror was one of my main concerns about switching to quay.io15:51
yoctozeptobut also might be the reason they let us hit the limits so often15:51
mgoddardit could be15:52
yoctozeptoyes, that is true15:52
yoctozeptoso we could reconsider quay.io15:52
mgoddardindeed15:52
yoctozeptoand tell docker goodbye15:52
yoctozeptowell, dockerhub*15:52
yoctozeptoare the bases present in quay?15:52
mgoddardhopefully they have centos, ubuntu and debian15:54
mgoddardthere doesn't seem to be the same 'official' set of images in quay.io though15:54
hrwcentos 8 stream is official on quay15:55
hrwand only there15:55
mgoddardit shouldn't really matter where the base lives15:55
hrwyep15:56
mgoddard#action mgoddard to write up options for CI registry15:57
mgoddardI'll try to present some options next week, hopefully we can make a decision15:57
yoctozeptogreat15:57
mgoddard2 minutes for open discussion15:58
mgoddard#topic open discussion15:58
*** openstack changes topic to "open discussion (Meeting topic: kolla)"15:58
kevkotoo short window for open discussion :P16:00
yoctozeptoclosed discussion then16:01
kevko:D16:01
mgoddardThanks all16:01
mgoddard#endmeeting16:01
yoctozeptothanks mgoddard16:01
*** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:01
openstackMeeting ended Wed Mar  3 16:01:42 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:01
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-03-15.01.html16:01
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-03-15.01.txt16:01
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-03-15.01.log.html16:01
mgoddardkevko: did you have a topic for open discussion?16:01
* hrw off16:02
yoctozeptomgoddard, wuchunyang: why are we removing octavia from magnum scenario?16:02
kevkomgoddard: Well, everything what i am proposing i am also discussing (most not in meeting)16:02
kevkomgoddard: but yeah, i wanted to discuss value of net.ipv4.tcp_retries216:03
kevko*default value16:03
wuchunyangyoctozepto  the nodes load is too high16:05
wuchunyangand octavia failed to create a member vm.16:05
kevkoalso, there is also another review for openstack-ansible https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/778028 where they are doing the same16:06
wuchunyangwhen we disable magnum,trove,designate , octavia works.16:06
mgoddardkevko: sorry, ran out of time. Looks like they are going for 8 retries16:08
mgoddardkevko: 3 does seem low16:08
*** wuchunyang has quit IRC16:10
*** ysirndjuro has joined #openstack-kolla16:11
*** muhaha27 has joined #openstack-kolla16:15
*** muhaha27 has quit IRC16:16
yoctozeptowell, oracle recommends 3 for their HA16:18
yoctozeptoit would probably be safer if we used pacemaker for keeping the vip16:19
yoctozeptoto reduce the flaps16:19
*** e0ne has quit IRC16:37
openstackgerritMerged openstack/kolla-ansible master: Revert "CI: Temporarily disable rabbitmq internal tls"  https://review.opendev.org/c/openstack/kolla-ansible/+/76151916:38
mgoddardkevko: btw, anyone is welcome to put topics on the agenda: https://wiki.openstack.org/wiki/Meetings/Kolla16:43
mgoddardkevko: makes it a bit easier to plan around16:43
openstackgerritMerged openstack/kolla-ansible master: [CI] Cinder upgrade testing  https://review.opendev.org/c/openstack/kolla-ansible/+/76795016:47
kevkomgoddard: 3 is 1.5 second which is too enough i think16:56
*** luksky has quit IRC16:59
kevkomgoddard: 8 = 51 seconds ...that's too much for VIP failover16:59
*** luksky has joined #openstack-kolla17:00
*** stand has quit IRC17:01
*** etp has quit IRC17:01
*** imcsk8 has quit IRC17:01
*** mgoddard has quit IRC17:01
*** _Cyclone_ has quit IRC17:01
*** timss has quit IRC17:01
*** openstackgerrit has quit IRC17:02
kevkoyoctozepto: rfc saying at least 317:03
kevkoyoctozepto: https://tools.ietf.org/html/rfc1122#page-100  paragraph (d)17:03
*** stand has joined #openstack-kolla17:05
*** etp has joined #openstack-kolla17:07
*** imcsk8 has joined #openstack-kolla17:07
*** mgoddard has joined #openstack-kolla17:07
*** _Cyclone_ has joined #openstack-kolla17:07
*** timss has joined #openstack-kolla17:07
yoctozeptokevko: "The value of R2 SHOULD correspond to at least 100 seconds."17:12
*** e0ne has joined #openstack-kolla17:13
kevkoyoctozepto: should, no need to, or must :) ...for example for kubernetes 100 seconds is too much and starting to recluster :(17:15
kevkobut let me edit my patch right now to make it optional for user17:16
dougszSimple one to help with slow API headaches: https://review.opendev.org/c/openstack/kolla-ansible/+/77850717:17
yoctozeptokevko: hah, if it was that simple17:20
yoctozeptotbh, the equipment is much faster nowadays17:20
yoctozeptobut as I said17:21
yoctozeptoI would feel more comfortable if we did not rely on keepalived but pacemaker17:21
kevkoyoctozepto: really ? why ?17:21
yoctozeptothat would avoid flaps from networking equipment wreaking havoc17:21
yoctozeptosee, keepalived is great when you expect the node to fail more often than network paths17:21
yoctozeptoas it is quite quick to ensure *at least one* node has the vip17:22
yoctozeptothe case is it can happen on both sides of network fragmentation17:22
kevkoyoctozepto: whatever you use, you should tune retries2 parameter17:23
yoctozeptowith pacemaker you ensure there is *exactly one*17:23
yoctozeptowell, tbh, the services themselves should be more resilient17:23
yoctozeptothe proper app-level timeouts should kick in17:23
yoctozeptoas it happens with 99% other services17:24
yoctozeptojust not with openstack components to mariadb :D17:24
yoctozeptoI think we should document it17:24
yoctozeptoand allow operators to test this workaround17:24
kevkohaha, services themselves should be more resilient ... in openstack ? nice dream :)17:25
yoctozepto:-)17:25
*** e0ne has quit IRC17:26
*** bengates_ has quit IRC17:26
*** bengates has joined #openstack-kolla17:27
mgoddardyoctozepto: oh dear, don't tell me you're drinking the pacemaker kool aid now?17:28
dougszsdake on Pacemaker back in 2016! http://104.130.124.113/irclogs/%23kolla/%23kolla.2016-03-23.log.html17:29
kevkomgoddard: +1 :D :D17:31
*** bengates has quit IRC17:32
*** dougsz has quit IRC17:34
yoctozeptomgoddard: I don't want pacemaker for everything; just saying how keepalived can betray you17:35
mgoddardkeepalived isn't ideal, for sure17:35
*** gfidente is now known as gfidente|afk17:41
*** bengates has joined #openstack-kolla17:42
*** e0ne has joined #openstack-kolla17:43
*** e0ne has quit IRC17:43
*** jonaspaulo has joined #openstack-kolla17:49
kevkoyoctozepto: default : 3 , optional -> https://review.opendev.org/c/openstack/kolla-ansible/+/77777217:50
kevkoyoctozepto: Radoslaw Piliszek 13 Feb 2020 -> For where keepalived is used, pacemaker is actually an overkill. Though it might be required for masakari stuff. :D17:58
*** e0ne has joined #openstack-kolla17:59
*** amoralej is now known as amoralej|off17:59
*** jonaspaulo has quit IRC17:59
yoctozeptokevko: yes, I know, I wrote that one myself18:01
kevko:D18:01
mgoddardkevko: what I don't understand - why it only happens on the previous VIP primary node18:04
*** bengates_ has joined #openstack-kolla18:04
*** admin0 has joined #openstack-kolla18:08
*** bengates has quit IRC18:08
*** rpittau is now known as rpittau|afk18:11
*** andrewbonney has quit IRC18:23
kevkomgoddard: Actually, we do not know the exact reason. Our theory is that it is somewhat connected to incorrect handling of gratuitous arp that is send by keepalive - while other nodes accept this arp packet, updates its arp cache and marks tcp connection somehow as invalid, the node that was master before updates arp cache late and most importantly does not mark connections as stale or invalid. This could be some kernel issue, however, we did not18:26
kevkoinvestigate the problem deep in kernel due to enormous complexity of kernel network stack.18:26
kevkomgoddard: check also this https://bugzilla.redhat.com/show_bug.cgi?id=134892918:31
openstackbugzilla.redhat.com bug 1348929 in keepalived "keepalived VIP becomes unreachable after a switch and a following higher prio advert" [High,Closed: wontfix] - Assigned to rohara18:31
*** alexandreperreau has joined #openstack-kolla18:32
yoctozeptokevko, mgoddard: most likely because the assumption is that the host should die, not keepalived18:33
yoctozeptojust don't eat me please, but I wonder if pacemaker's VIP resource behaves any better :P18:34
yoctozeptoand now I pumice to not mention it more today18:34
*** kevko has quit IRC18:36
*** muhaha52 has joined #openstack-kolla19:07
*** muhaha52 has quit IRC19:08
*** kevko has joined #openstack-kolla19:08
*** kevko has quit IRC19:14
iniazianyone using ussuri w/octopus ceph (cephadm deployed) & erasure coded pools (i.e. use replication for metadata, and then specify ec data pool) of course external ceph.  Trying to figure out how to specify data pools... in ceph.conf that cephadm maintains generates is minimal... in olden days, ceph.conf would include rgw gateway info etc.  or do I just generate the ceph.conf with the right fsid but add the other stuff for kolla to deploy?19:20
*** suff has quit IRC19:23
yoctozeptoiniazi: ceph.conf only really needs to hold the fsid and monitors addresses19:52
yoctozeptothe rest is in services config19:52
*** alexandreperreau has quit IRC19:55
*** happyhemant has quit IRC20:13
*** k_mouza has quit IRC20:49
iniaziyoctozepto, i'm not sure what you mean services config... i.e. for kolla-ansible config dir, where I specify a ceph.conf file?  or is there something in globals.yml i can set to override data pool for specific services?  sorry for newbie questions21:08
*** muhaha has quit IRC21:12
*** e0ne has quit IRC21:23
*** gary_perkins has joined #openstack-kolla21:27
iniazisorry, my example is from https://themeanti.me/technology/2018/08/23/ceph_erasure_openstack.html, how he says to configure the data pool via ceph.conf21:28
iniazior maybe you are talking about the same thing I'm talking about... do I just generate a ceph.conf for the services config dir (i.e. config/cinder/ceph.conf and then specify the data pool there (and so on for each of the services)?21:31
*** irclogbot_3 has quit IRC21:33
*** sean-k-mooney has quit IRC21:33
*** Wasaac has quit IRC21:33
*** ianw has quit IRC21:33
*** Reepicheep has quit IRC21:33
*** ozzzo has quit IRC21:33
*** atmark has quit IRC21:33
*** Mr_Freezeex has quit IRC21:33
*** wathoom has quit IRC21:33
*** irclogbot_3 has joined #openstack-kolla21:34
*** sean-k-mooney has joined #openstack-kolla21:34
*** Wasaac has joined #openstack-kolla21:34
*** ianw has joined #openstack-kolla21:34
*** Reepicheep has joined #openstack-kolla21:34
*** atmark has joined #openstack-kolla21:34
*** ozzzo has joined #openstack-kolla21:34
*** Mr_Freezeex has joined #openstack-kolla21:34
*** wathoom has joined #openstack-kolla21:34
*** e0ne has joined #openstack-kolla21:38
*** zzzeek has quit IRC21:44
*** zzzeek has joined #openstack-kolla21:45
*** cah_link has quit IRC22:09
*** e0ne has quit IRC22:42
*** bengates_ has quit IRC22:47
*** bengates has joined #openstack-kolla22:48
*** k_mouza has joined #openstack-kolla22:49
*** bengates has quit IRC22:52
*** k_mouza has quit IRC22:54
*** stingrayza has quit IRC23:11
*** stingrayza has joined #openstack-kolla23:14
*** k_mouza has joined #openstack-kolla23:17
*** ysirndjuro has left #openstack-kolla23:17
*** k_mouza has quit IRC23:22
*** k_mouza has joined #openstack-kolla23:27
*** k_mouza has quit IRC23:31
*** k_mouza has joined #openstack-kolla23:36
*** luksky has quit IRC23:39
*** zzzeek has quit IRC23:40
*** k_mouza has quit IRC23:41
*** zzzeek has joined #openstack-kolla23:42
*** vishalmanchanda has quit IRC23:44

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!