| *** Underknowledge has quit IRC | 01:08 | |
| *** Underknowledge has joined #openstack-kolla | 01:08 | |
| *** Underknowledge2 has joined #openstack-kolla | 01:51 | |
| *** Underknowledge has quit IRC | 01:54 | |
| *** Underknowledge2 is now known as Underknowledge | 01:54 | |
| *** admin0 has quit IRC | 02:04 | |
| openstackgerrit | wu.chunyang proposed openstack/kolla master: DNM: test1 https://review.opendev.org/c/openstack/kolla/+/778348 | 02:04 |
|---|---|---|
| openstackgerrit | wu.chunyang proposed openstack/kolla master: DNM: test2 https://review.opendev.org/c/openstack/kolla/+/778349 | 02:04 |
| *** r3ap3r has quit IRC | 02:29 | |
| *** r3ap3r has joined #openstack-kolla | 02:33 | |
| *** Underknowledge has quit IRC | 02:38 | |
| *** Xuchu_ has joined #openstack-kolla | 02:43 | |
| *** Xuchu has quit IRC | 02:46 | |
| *** iniazi has joined #openstack-kolla | 03:09 | |
| *** LinPeiWen has quit IRC | 03:21 | |
| *** dasp has quit IRC | 03:34 | |
| *** suff has joined #openstack-kolla | 03:44 | |
| *** dasp has joined #openstack-kolla | 03:45 | |
| *** k_mouza has joined #openstack-kolla | 04:00 | |
| *** k_mouza has quit IRC | 04:05 | |
| *** shyamb has joined #openstack-kolla | 04:16 | |
| *** zzzeek has quit IRC | 04:17 | |
| *** zzzeek has joined #openstack-kolla | 04:17 | |
| *** zzzeek has quit IRC | 04:22 | |
| *** zzzeek has joined #openstack-kolla | 04:23 | |
| *** LinPeiWen has joined #openstack-kolla | 05:02 | |
| *** vishalmanchanda has joined #openstack-kolla | 05:27 | |
| *** cah_link has joined #openstack-kolla | 05:36 | |
| *** brinzhang has joined #openstack-kolla | 05:41 | |
| oklhost | morning | 05:50 |
| *** zzzeek has quit IRC | 06:06 | |
| *** wuchunyang has joined #openstack-kolla | 06:10 | |
| *** Xuchu_ has quit IRC | 06:22 | |
| *** Xuchu has joined #openstack-kolla | 06:22 | |
| *** shyamb has quit IRC | 06:24 | |
| *** mchlumsky has quit IRC | 06:28 | |
| *** mchlumsky has joined #openstack-kolla | 06:28 | |
| *** wuchunyang has quit IRC | 06:36 | |
| *** zzzeek has joined #openstack-kolla | 06:41 | |
| *** mrunge has quit IRC | 06:56 | |
| *** mrunge has joined #openstack-kolla | 06:57 | |
| openstackgerrit | Lin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for redis services https://review.opendev.org/c/openstack/kolla-ansible/+/778371 | 07:07 |
| *** luksky has joined #openstack-kolla | 07:19 | |
| *** waxfire0 has joined #openstack-kolla | 07:35 | |
| *** waxfire has quit IRC | 07:35 | |
| *** waxfire0 is now known as waxfire | 07:35 | |
| *** rpittau|afk is now known as rpittau | 07:52 | |
| *** bengates has joined #openstack-kolla | 08:03 | |
| *** bengates has quit IRC | 08:03 | |
| *** bengates has joined #openstack-kolla | 08:04 | |
| *** andrewbonney has joined #openstack-kolla | 08:14 | |
| *** wuchunyang has joined #openstack-kolla | 08:20 | |
| *** amoralej|off is now known as amoralej | 08:22 | |
| *** wuchunyang has quit IRC | 08:24 | |
| *** amoralej has joined #openstack-kolla | 08:25 | |
| *** wuchunyang has joined #openstack-kolla | 08:31 | |
| *** alexandreperreau has quit IRC | 08:34 | |
| hrw | morning | 08:35 |
| *** dougsz has joined #openstack-kolla | 08:59 | |
| *** bengates_ has joined #openstack-kolla | 09:05 | |
| wuchunyang | morning | 09:05 |
| *** bengates has quit IRC | 09:09 | |
| *** Underknowledge has joined #openstack-kolla | 09:11 | |
| *** shyamb has joined #openstack-kolla | 09:16 | |
| *** shyam89 has joined #openstack-kolla | 09:16 | |
| LinPeiWen | morning | 09:21 |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: octavia: support tenant management network https://review.opendev.org/c/openstack/kolla-ansible/+/755589 | 09:24 |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: CI: octavia: remove octavia from magnum scenario https://review.opendev.org/c/openstack/kolla-ansible/+/754285 | 09:24 |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer https://review.opendev.org/c/openstack/kolla-ansible/+/778390 | 09:24 |
| *** Xuchu has quit IRC | 09:43 | |
| openstackgerrit | Lin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for skydive services https://review.opendev.org/c/openstack/kolla-ansible/+/778393 | 09:46 |
| *** shyam89 has quit IRC | 10:11 | |
| *** shyamb has quit IRC | 10:11 | |
| mgoddard | morning | 10:21 |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: Use Docker healthchecks for watcher services https://review.opendev.org/c/openstack/kolla-ansible/+/778403 | 10:23 |
| hrw | mgoddard: I declare train to be dead by pull limits | 10:23 |
| hrw | https://review.opendev.org/c/openstack/kolla/+/774602 is unable to pass | 10:24 |
| mgoddard | hrw: pull limits are temporary | 10:24 |
| hrw | 8 failures in a row | 10:24 |
| mgoddard | apart from the one that passed? | 10:25 |
| hrw | yesterday I did recheck at 23:22 when we had no CI jobs | 10:25 |
| hrw | failed. | 10:25 |
| mgoddard | upgrade jobs use stein images which are less likely to be cached | 10:25 |
| mgoddard | we are not the only users of the registry mirror | 10:25 |
| mgoddard | although I do have my suspicions about the caching of images in the limestone region - every pull limit failure I've seen in kayobe has been in limestone | 10:26 |
| *** wuchunyang has quit IRC | 10:28 | |
| *** k_mouza has joined #openstack-kolla | 10:36 | |
| openstackgerrit | Lin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for octavia services https://review.opendev.org/c/openstack/kolla-ansible/+/778180 | 10:36 |
| *** shyam89 has joined #openstack-kolla | 10:37 | |
| *** shyamb has joined #openstack-kolla | 10:37 | |
| hrw | ok | 10:50 |
| *** luksky has quit IRC | 11:07 | |
| *** luksky has joined #openstack-kolla | 11:08 | |
| *** luksky has quit IRC | 11:09 | |
| openstackgerrit | Lin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for octavia services https://review.opendev.org/c/openstack/kolla-ansible/+/778180 | 11:09 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Use Docker healthchecks for cyborg services https://review.opendev.org/c/openstack/kolla-ansible/+/778212 | 11:10 |
| *** kevko has joined #openstack-kolla | 11:14 | |
| *** kevko_ has joined #openstack-kolla | 11:16 | |
| *** kevko has quit IRC | 11:16 | |
| openstackgerrit | Lin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for senlin services https://review.opendev.org/c/openstack/kolla-ansible/+/778413 | 11:23 |
| openstackgerrit | Merged openstack/kayobe master: Report available entropy https://review.opendev.org/c/openstack/kayobe/+/777365 | 11:25 |
| *** luksky has joined #openstack-kolla | 11:25 | |
| *** kevko_ has quit IRC | 11:31 | |
| *** kevko has joined #openstack-kolla | 11:31 | |
| kevko | yoctozepto: sorry, you asked how many seconds it is before kernel kill non-active tcp connections when set 3,4,5 (keepalive thing) | 11:32 |
| kevko | yoctozepto: here it is -> https://pracucci.com/linux-tcp-rto-min-max-and-tcp-retries2.html | 11:32 |
| kevko | 3 = cca 1.4s , 4 = 3sec , 5 = 6.2 sec | 11:33 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Fix documentation of control host bootstrap https://review.opendev.org/c/openstack/kayobe/+/778442 | 11:55 |
| *** shyam89 has quit IRC | 12:00 | |
| *** shyamb has quit IRC | 12:00 | |
| openstackgerrit | Merged openstack/kayobe master: Fix documentation of control host bootstrap https://review.opendev.org/c/openstack/kayobe/+/778442 | 12:31 |
| *** amoralej is now known as amoralej|lunch | 13:15 | |
| kevko | yoctozepto: here ? | 13:21 |
| kevko | yoctozepto: i would like to take this -> https://review.opendev.org/c/openstack/kolla-ansible/+/728796/ and rework little bit (not use new group ..but I think it's more clear to add flag if arbiter or not) | 13:22 |
| kevko | as you said in last commend .. | 13:22 |
| yoctozepto | kevko: you are very welcome to | 13:35 |
| *** wuchunya_ has joined #openstack-kolla | 13:35 | |
| kevko | yoctozepto: should I take and update review or create new one ? | 13:44 |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: Use Docker healthchecks for watcher services https://review.opendev.org/c/openstack/kolla-ansible/+/778403 | 13:45 |
| *** e0ne has joined #openstack-kolla | 13:45 | |
| kevko | yoctozepto: because i'm going to commit on top of my mariadb role refactor .. | 13:45 |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer https://review.opendev.org/c/openstack/kolla-ansible/+/778390 | 13:51 |
| kevko | yoctozepto: + i've already replied to your question from yesterday above ^^ | 13:52 |
| kevko | yoctozepto: regarding to that ..what do you mean ..which value should be middle gold ? | 13:53 |
| *** amoralej|lunch is now known as amoralej | 13:56 | |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/victoria: Update IPA docs and test build with extra-hardware https://review.opendev.org/c/openstack/kayobe/+/776947 | 14:09 |
| openstackgerrit | Lin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for gnocchi services https://review.opendev.org/c/openstack/kolla-ansible/+/778467 | 14:12 |
| yoctozepto | kevko: for any interactive service 3-4 sounds like a reasonable default | 14:13 |
| kevko | yoctozepto: i think 3 is good ..it is abourt 1.5 sec which is quite big window also on "small" networks .. | 14:13 |
| kevko | yoctozepto: i've sent an article where you can find a table with times ... | 14:14 |
| yoctozepto | airship went with 5 | 14:14 |
| yoctozepto | honestly, beyond 5 it is very likely to hit other timeouts | 14:14 |
| kevko | yoctozepto: like what timeouts ? | 14:16 |
| kevko | yoctozepto: these option is for packets which were not acknowledged .. | 14:16 |
| kevko | and trying to retransmit | 14:17 |
| yoctozepto | kevko: application-level timeouts | 14:40 |
| yoctozepto | I mean in general | 14:40 |
| kevko | yoctozepto: well problem is that it was not hit by app-level | 14:43 |
| kevko | yoctozepto: anyway, i think 3 is OK + add option to configure yourself | 14:44 |
| openstackgerrit | Doug Szumski proposed openstack/kolla master: Upgrade from ELK6 to ELK7 FOSS release https://review.opendev.org/c/openstack/kolla/+/738567 | 14:47 |
| mgoddard | meeting in 10 | 14:51 |
| mgoddard | ^ mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt | 14:51 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Upgrade service configuration for ELK 7 https://review.opendev.org/c/openstack/kolla-ansible/+/740986 | 14:59 |
| mgoddard | #startmeeting kolla | 15:01 |
| openstack | Meeting started Wed Mar 3 15:01:56 2021 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| *** openstack changes topic to " (Meeting topic: kolla)" | 15:01 | |
| openstack | The meeting name has been set to 'kolla' | 15:02 |
| mgoddard | #topic rollcall | 15:02 |
| *** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:02 | |
| mgoddard | \o | 15:02 |
| hrw | o][o | 15:02 |
| parallax | o/ | 15:02 |
| wuchunya_ | o | 15:02 |
| dougsz | {o | 15:02 |
| headphoneJames | o/ | 15:02 |
| kplant | o7 | 15:02 |
| priteau | |o| | 15:03 |
| mgoddard | |-o-| | 15:03 |
| mgoddard | #topic agenda | 15:04 |
| *** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:04 | |
| mgoddard | * Roll-call | 15:04 |
| mgoddard | * Announcements | 15:04 |
| mgoddard | ** Combined TC/PTL nomination open http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020811.html | 15:04 |
| mgoddard | ** OpenStack feature freeze next week | 15:04 |
| mgoddard | * Review action items from the last meeting | 15:04 |
| mgoddard | * CI status | 15:04 |
| mgoddard | * Review requests | 15:04 |
| mgoddard | * PoC: image build & test pipeline (https://review.opendev.org/c/openstack/kolla/+/777796 and https://review.opendev.org/c/openstack/kolla-ansible/+/777946) | 15:04 |
| mgoddard | * Wallaby release planning | 15:04 |
| mgoddard | #topic Announcements | 15:04 |
| *** openstack changes topic to "Announcements (Meeting topic: kolla)" | 15:04 | |
| mgoddard | #info Combined TC/PTL nomination open | 15:04 |
| mgoddard | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020811.html | 15:04 |
| mgoddard | Anyone is welcome to run for Kolla PTL | 15:05 |
| mgoddard | #info OpenStack feature freeze next week | 15:06 |
| *** LinPeiWen has quit IRC | 15:06 | |
| mgoddard | #link https://releases.openstack.org/wallaby/schedule.html | 15:06 |
| mgoddard | #info Kolla feature freeze will be Mar 29 - Apr 02 | 15:06 |
| mgoddard | Any other announcements? | 15:07 |
| mgoddard | #topic Review action items from the last meeting | 15:07 |
| *** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)" | 15:07 | |
| mgoddard | yoctozepto to ask openstack-discuss about NTP | 15:08 |
| mgoddard | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020707.html | 15:08 |
| mgoddard | Thanks yoctozepto | 15:08 |
| mgoddard | #topic CI status | 15:08 |
| *** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:08 | |
| mgoddard | Kolla train still broken | 15:09 |
| mgoddard | The fix keeps getting hit by dockerhub pull limits | 15:09 |
| mgoddard | https://review.opendev.org/c/openstack/kolla/+/774602 | 15:09 |
| mgoddard | we had an issue with neutron-server builds on master, but it was fixed by https://review.opendev.org/c/openstack/kolla/+/777992 | 15:10 |
| mgoddard | unclear whether it affects other branches | 15:11 |
| mgoddard | kolla-ansible NFV CI job seems to be failing on master | 15:11 |
| mgoddard | https://ac90fbbc9cd1b2f919e7-c0288c15cf27fe5a39c9948ecafb7329.ssl.cf1.rackcdn.com/778179/1/check/kolla-ansible-centos8-source-scenario-nfv/e3b78d7/secondary1/logs/docker_logs/tacker_conductor.txt | 15:11 |
| mgoddard | tacker processes fail to import toscaparser | 15:11 |
| wuchunya_ | i think this is a tacker bug | 15:12 |
| mgoddard | is it not a missing package in our image? | 15:12 |
| wuchunya_ | i will add the package requirement to the tacker project. | 15:13 |
| yoctozepto | \o/ | 15:13 |
| wuchunya_ | this package should be in tacker requirements. | 15:13 |
| yoctozepto | mgoddard: it is but tacker should be listing it | 15:13 |
| *** happyhemant has joined #openstack-kolla | 15:13 | |
| mgoddard | https://pypi.org/project/tosca-parser/ | 15:13 |
| yoctozepto | yeah, just what wuchunya_ is saying | 15:13 |
| mgoddard | probably, unless it is an optional dep | 15:14 |
| *** wuchunya_ is now known as wuchunyang | 15:14 | |
| yoctozepto | it's not the first time tacker is clumsy | 15:14 |
| yoctozepto | so that's a good question | 15:15 |
| mgoddard | #action wuchunyang to propose toscaparser in tacker requirements to fix NFV job | 15:15 |
| wuchunyang | ok , no problem | 15:15 |
| mgoddard | thanks | 15:16 |
| mgoddard | anyone want to review https://review.opendev.org/c/openstack/kolla-ansible/+/761519/ to enable rabbitmq TLS in CI? | 15:17 |
| yoctozepto | done | 15:18 |
| yoctozepto | clever plug ;p | 15:19 |
| mgoddard | tacker plot thickens: requirements.txt has tosca-parser>=1.6.0 # Apache-2.0 | 15:19 |
| *** luksky has quit IRC | 15:19 | |
| *** luksky has joined #openstack-kolla | 15:19 | |
| yoctozepto | hmm, that's intriguing | 15:19 |
| *** luksky has quit IRC | 15:19 | |
| yoctozepto | perhaps it's run in a different env? | 15:19 |
| wuchunyang | the package name is nfv-toscaparser | 15:19 |
| mgoddard | there are two packages | 15:20 |
| yoctozepto | this should be the one though: | 15:20 |
| yoctozepto | https://github.com/openstack/tosca-parser | 15:20 |
| *** muhaha has joined #openstack-kolla | 15:20 | |
| yoctozepto | hmm | 15:20 |
| mgoddard | nfv-toscaparser looks old, last release 1.1.1 | 15:20 |
| mgoddard | https://pypi.org/project/tosca-parser/ | 15:21 |
| mgoddard | anyway, we don't all need to solve it | 15:21 |
| mgoddard | Kayobe CI had a couple of improvements this week | 15:21 |
| mgoddard | bare metal testing reliability should be improved | 15:22 |
| mgoddard | we still hit pull limits | 15:22 |
| mgoddard | most often in the limestone region | 15:22 |
| mgoddard | #topic Review requests | 15:23 |
| *** openstack changes topic to "Review requests (Meeting topic: kolla)" | 15:23 | |
| mgoddard | Does anyone have a patch they would like to be reviewed this week? | 15:23 |
| parallax | Possibly this small one: https://review.opendev.org/c/openstack/kolla-ansible/+/774222 | 15:23 |
| mgoddard | added RP+1 | 15:24 |
| yoctozepto | https://review.opendev.org/c/openstack/kolla-ansible/+/767950 | 15:24 |
| yoctozepto | 'tis still rotting ;p | 15:24 |
| dougsz | ELK7 upgrade patches should be fairly easy | 15:25 |
| mgoddard | RP+1 all round | 15:27 |
| mgoddard | it's cheating a little bit, but I did some reviews of the healthcheck patches today, so I'll plug those https://review.opendev.org/q/topic:%22container-health-check%22+(status:open%20OR%20status:merged) | 15:28 |
| mgoddard | would be nice to finish that one off | 15:28 |
| mgoddard | #topic PoC: image build & test pipeline | 15:28 |
| *** openstack changes topic to "PoC: image build & test pipeline (Meeting topic: kolla)" | 15:28 | |
| mgoddard | This is mine | 15:29 |
| mgoddard | After fixing some kayobe CI issues last week, the next biggest obstacle to stability is dockerhub pull limits | 15:29 |
| mgoddard | We've made changes that make CI usable, but it is still annoying when it fails from time to time | 15:30 |
| mgoddard | So I thought I would put some effort into working out how to use the opendev container registry | 15:30 |
| mgoddard | Given that we may see this as a potential fix for our dockerhub woes | 15:31 |
| mgoddard | #link https://review.opendev.org/c/openstack/kolla/+/777796 | 15:31 |
| mgoddard | #link https://review.opendev.org/c/openstack/kolla-ansible/+/777946 | 15:31 |
| mgoddard | the commit message tries to give a high level overview | 15:32 |
| mgoddard | it's based on this setup: | 15:32 |
| mgoddard | #link https://docs.opendev.org/opendev/base-jobs/latest/docker-image.html#a-repository-with-producers-and-consumers | 15:33 |
| mgoddard | and allows different jobs to produce and consume container images | 15:33 |
| mgoddard | the PoC has one job that builds images, then pushes them to a registry | 15:33 |
| mgoddard | and another job that pulls images from the registry and tests them | 15:33 |
| mgoddard | a key part here being that dockerhub is not involved (much) | 15:34 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Change docker_registry network_mode to host https://review.opendev.org/c/openstack/kayobe/+/760371 | 15:34 |
| mgoddard | is anyone listening? | 15:34 |
| hrw | sounds good | 15:35 |
| dougsz | ACK | 15:35 |
| priteau | Everyone is looking at the changes :) | 15:35 |
| wuchunyang | yes | 15:35 |
| mgoddard | I'll give you a few minutes | 15:36 |
| *** luksky has joined #openstack-kolla | 15:36 | |
| dougsz | I'm sure this has been asked before, but it wasn't possible to request unlimited pulls from Docker hub? | 15:38 |
| yoctozepto | dougsz: we have to pay with blood | 15:38 |
| hrw | more or less we are doomed | 15:38 |
| yoctozepto | or soul | 15:38 |
| dougsz | cool, I see :) | 15:38 |
| hrw | I have read description of kolla patch and we may end with pushing GBs of data between CI nodes | 15:39 |
| hrw | life sucks | 15:39 |
| yoctozepto | shucks | 15:39 |
| mgoddard | right | 15:40 |
| mgoddard | that is one of my main concerns | 15:40 |
| mgoddard | there are two tiers of registry involved | 15:40 |
| mgoddard | the buildset registry, a temporary node running in another job. I believe this should be on the same cloud (but not certain) | 15:41 |
| mgoddard | the intermediate registry | 15:41 |
| mgoddard | ^ there is only one of these, and it lives in rackspace | 15:41 |
| mgoddard | for $reasons, images generally get pushed to both registries | 15:41 |
| hrw | or each k-a job does: start registry, build images and push to local, do own job, destroy | 15:41 |
| hrw | that way no data send but all jobs take longer | 15:42 |
| mgoddard | no, build and deploy are in separate jobs | 15:42 |
| mgoddard | well, what you describe is what we have already | 15:42 |
| hrw | + caching registry in each opendev cloud to not fetch debian/centos/ubuntu base image | 15:43 |
| hrw | this way we touch docker hub only in publish jobs | 15:43 |
| mgoddard | there are quite a few options for how it would work | 15:45 |
| mgoddard | I suppose we ought to try to list them, and work out which ones fit with the changes we want to make | 15:46 |
| hrw | and I assume that opendev already asked dockerhub to get 'unlimited pull' and got rejected | 15:46 |
| mgoddard | it's possible we could just publish to and pull from the infra registry as well as dockerhub, and keep everything else the same | 15:46 |
| mgoddard | see earlier dicussion about soul and blood | 15:47 |
| mgoddard | if we think this option looks good, then we probably need to have a conversation with opendev infra team | 15:48 |
| mgoddard | but while poking around in the opendev config, I found option B | 15:48 |
| mgoddard | the registry mirrors in opendev are not the official docker registry, just an apache caching proxy | 15:49 |
| yoctozepto | oh, that's bad | 15:50 |
| mgoddard | #link https://opendev.org/opendev/system-config/src/commit/4310315afe27c040b239a72a1c248ddabf7fdfa5/playbooks/roles/mirror/templates/mirror.vhost.j2#L453 | 15:50 |
| mgoddard | which means that they are able to support quay.io | 15:51 |
| mgoddard | the lack of a registry mirror was one of my main concerns about switching to quay.io | 15:51 |
| yoctozepto | but also might be the reason they let us hit the limits so often | 15:51 |
| mgoddard | it could be | 15:52 |
| yoctozepto | yes, that is true | 15:52 |
| yoctozepto | so we could reconsider quay.io | 15:52 |
| mgoddard | indeed | 15:52 |
| yoctozepto | and tell docker goodbye | 15:52 |
| yoctozepto | well, dockerhub* | 15:52 |
| yoctozepto | are the bases present in quay? | 15:52 |
| mgoddard | hopefully they have centos, ubuntu and debian | 15:54 |
| mgoddard | there doesn't seem to be the same 'official' set of images in quay.io though | 15:54 |
| hrw | centos 8 stream is official on quay | 15:55 |
| hrw | and only there | 15:55 |
| mgoddard | it shouldn't really matter where the base lives | 15:55 |
| hrw | yep | 15:56 |
| mgoddard | #action mgoddard to write up options for CI registry | 15:57 |
| mgoddard | I'll try to present some options next week, hopefully we can make a decision | 15:57 |
| yoctozepto | great | 15:57 |
| mgoddard | 2 minutes for open discussion | 15:58 |
| mgoddard | #topic open discussion | 15:58 |
| *** openstack changes topic to "open discussion (Meeting topic: kolla)" | 15:58 | |
| kevko | too short window for open discussion :P | 16:00 |
| yoctozepto | closed discussion then | 16:01 |
| kevko | :D | 16:01 |
| mgoddard | Thanks all | 16:01 |
| mgoddard | #endmeeting | 16:01 |
| yoctozepto | thanks mgoddard | 16:01 |
| *** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:01 | |
| openstack | Meeting ended Wed Mar 3 16:01:42 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-03-15.01.html | 16:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-03-15.01.txt | 16:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-03-15.01.log.html | 16:01 |
| mgoddard | kevko: did you have a topic for open discussion? | 16:01 |
| * hrw off | 16:02 | |
| yoctozepto | mgoddard, wuchunyang: why are we removing octavia from magnum scenario? | 16:02 |
| kevko | mgoddard: Well, everything what i am proposing i am also discussing (most not in meeting) | 16:02 |
| kevko | mgoddard: but yeah, i wanted to discuss value of net.ipv4.tcp_retries2 | 16:03 |
| kevko | *default value | 16:03 |
| wuchunyang | yoctozepto the nodes load is too high | 16:05 |
| wuchunyang | and octavia failed to create a member vm. | 16:05 |
| kevko | also, there is also another review for openstack-ansible https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/778028 where they are doing the same | 16:06 |
| wuchunyang | when we disable magnum,trove,designate , octavia works. | 16:06 |
| mgoddard | kevko: sorry, ran out of time. Looks like they are going for 8 retries | 16:08 |
| mgoddard | kevko: 3 does seem low | 16:08 |
| *** wuchunyang has quit IRC | 16:10 | |
| *** ysirndjuro has joined #openstack-kolla | 16:11 | |
| *** muhaha27 has joined #openstack-kolla | 16:15 | |
| *** muhaha27 has quit IRC | 16:16 | |
| yoctozepto | well, oracle recommends 3 for their HA | 16:18 |
| yoctozepto | it would probably be safer if we used pacemaker for keeping the vip | 16:19 |
| yoctozepto | to reduce the flaps | 16:19 |
| *** e0ne has quit IRC | 16:37 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Revert "CI: Temporarily disable rabbitmq internal tls" https://review.opendev.org/c/openstack/kolla-ansible/+/761519 | 16:38 |
| mgoddard | kevko: btw, anyone is welcome to put topics on the agenda: https://wiki.openstack.org/wiki/Meetings/Kolla | 16:43 |
| mgoddard | kevko: makes it a bit easier to plan around | 16:43 |
| openstackgerrit | Merged openstack/kolla-ansible master: [CI] Cinder upgrade testing https://review.opendev.org/c/openstack/kolla-ansible/+/767950 | 16:47 |
| kevko | mgoddard: 3 is 1.5 second which is too enough i think | 16:56 |
| *** luksky has quit IRC | 16:59 | |
| kevko | mgoddard: 8 = 51 seconds ...that's too much for VIP failover | 16:59 |
| *** luksky has joined #openstack-kolla | 17:00 | |
| *** stand has quit IRC | 17:01 | |
| *** etp has quit IRC | 17:01 | |
| *** imcsk8 has quit IRC | 17:01 | |
| *** mgoddard has quit IRC | 17:01 | |
| *** _Cyclone_ has quit IRC | 17:01 | |
| *** timss has quit IRC | 17:01 | |
| *** openstackgerrit has quit IRC | 17:02 | |
| kevko | yoctozepto: rfc saying at least 3 | 17:03 |
| kevko | yoctozepto: https://tools.ietf.org/html/rfc1122#page-100 paragraph (d) | 17:03 |
| *** stand has joined #openstack-kolla | 17:05 | |
| *** etp has joined #openstack-kolla | 17:07 | |
| *** imcsk8 has joined #openstack-kolla | 17:07 | |
| *** mgoddard has joined #openstack-kolla | 17:07 | |
| *** _Cyclone_ has joined #openstack-kolla | 17:07 | |
| *** timss has joined #openstack-kolla | 17:07 | |
| yoctozepto | kevko: "The value of R2 SHOULD correspond to at least 100 seconds." | 17:12 |
| *** e0ne has joined #openstack-kolla | 17:13 | |
| kevko | yoctozepto: should, no need to, or must :) ...for example for kubernetes 100 seconds is too much and starting to recluster :( | 17:15 |
| kevko | but let me edit my patch right now to make it optional for user | 17:16 |
| dougsz | Simple one to help with slow API headaches: https://review.opendev.org/c/openstack/kolla-ansible/+/778507 | 17:17 |
| yoctozepto | kevko: hah, if it was that simple | 17:20 |
| yoctozepto | tbh, the equipment is much faster nowadays | 17:20 |
| yoctozepto | but as I said | 17:21 |
| yoctozepto | I would feel more comfortable if we did not rely on keepalived but pacemaker | 17:21 |
| kevko | yoctozepto: really ? why ? | 17:21 |
| yoctozepto | that would avoid flaps from networking equipment wreaking havoc | 17:21 |
| yoctozepto | see, keepalived is great when you expect the node to fail more often than network paths | 17:21 |
| yoctozepto | as it is quite quick to ensure *at least one* node has the vip | 17:22 |
| yoctozepto | the case is it can happen on both sides of network fragmentation | 17:22 |
| kevko | yoctozepto: whatever you use, you should tune retries2 parameter | 17:23 |
| yoctozepto | with pacemaker you ensure there is *exactly one* | 17:23 |
| yoctozepto | well, tbh, the services themselves should be more resilient | 17:23 |
| yoctozepto | the proper app-level timeouts should kick in | 17:23 |
| yoctozepto | as it happens with 99% other services | 17:24 |
| yoctozepto | just not with openstack components to mariadb :D | 17:24 |
| yoctozepto | I think we should document it | 17:24 |
| yoctozepto | and allow operators to test this workaround | 17:24 |
| kevko | haha, services themselves should be more resilient ... in openstack ? nice dream :) | 17:25 |
| yoctozepto | :-) | 17:25 |
| *** e0ne has quit IRC | 17:26 | |
| *** bengates_ has quit IRC | 17:26 | |
| *** bengates has joined #openstack-kolla | 17:27 | |
| mgoddard | yoctozepto: oh dear, don't tell me you're drinking the pacemaker kool aid now? | 17:28 |
| dougsz | sdake on Pacemaker back in 2016! http://104.130.124.113/irclogs/%23kolla/%23kolla.2016-03-23.log.html | 17:29 |
| kevko | mgoddard: +1 :D :D | 17:31 |
| *** bengates has quit IRC | 17:32 | |
| *** dougsz has quit IRC | 17:34 | |
| yoctozepto | mgoddard: I don't want pacemaker for everything; just saying how keepalived can betray you | 17:35 |
| mgoddard | keepalived isn't ideal, for sure | 17:35 |
| *** gfidente is now known as gfidente|afk | 17:41 | |
| *** bengates has joined #openstack-kolla | 17:42 | |
| *** e0ne has joined #openstack-kolla | 17:43 | |
| *** e0ne has quit IRC | 17:43 | |
| *** jonaspaulo has joined #openstack-kolla | 17:49 | |
| kevko | yoctozepto: default : 3 , optional -> https://review.opendev.org/c/openstack/kolla-ansible/+/777772 | 17:50 |
| kevko | yoctozepto: Radoslaw Piliszek 13 Feb 2020 -> For where keepalived is used, pacemaker is actually an overkill. Though it might be required for masakari stuff. :D | 17:58 |
| *** e0ne has joined #openstack-kolla | 17:59 | |
| *** amoralej is now known as amoralej|off | 17:59 | |
| *** jonaspaulo has quit IRC | 17:59 | |
| yoctozepto | kevko: yes, I know, I wrote that one myself | 18:01 |
| kevko | :D | 18:01 |
| mgoddard | kevko: what I don't understand - why it only happens on the previous VIP primary node | 18:04 |
| *** bengates_ has joined #openstack-kolla | 18:04 | |
| *** admin0 has joined #openstack-kolla | 18:08 | |
| *** bengates has quit IRC | 18:08 | |
| *** rpittau is now known as rpittau|afk | 18:11 | |
| *** andrewbonney has quit IRC | 18:23 | |
| kevko | mgoddard: Actually, we do not know the exact reason. Our theory is that it is somewhat connected to incorrect handling of gratuitous arp that is send by keepalive - while other nodes accept this arp packet, updates its arp cache and marks tcp connection somehow as invalid, the node that was master before updates arp cache late and most importantly does not mark connections as stale or invalid. This could be some kernel issue, however, we did not | 18:26 |
| kevko | investigate the problem deep in kernel due to enormous complexity of kernel network stack. | 18:26 |
| kevko | mgoddard: check also this https://bugzilla.redhat.com/show_bug.cgi?id=1348929 | 18:31 |
| openstack | bugzilla.redhat.com bug 1348929 in keepalived "keepalived VIP becomes unreachable after a switch and a following higher prio advert" [High,Closed: wontfix] - Assigned to rohara | 18:31 |
| *** alexandreperreau has joined #openstack-kolla | 18:32 | |
| yoctozepto | kevko, mgoddard: most likely because the assumption is that the host should die, not keepalived | 18:33 |
| yoctozepto | just don't eat me please, but I wonder if pacemaker's VIP resource behaves any better :P | 18:34 |
| yoctozepto | and now I pumice to not mention it more today | 18:34 |
| *** kevko has quit IRC | 18:36 | |
| *** muhaha52 has joined #openstack-kolla | 19:07 | |
| *** muhaha52 has quit IRC | 19:08 | |
| *** kevko has joined #openstack-kolla | 19:08 | |
| *** kevko has quit IRC | 19:14 | |
| iniazi | anyone using ussuri w/octopus ceph (cephadm deployed) & erasure coded pools (i.e. use replication for metadata, and then specify ec data pool) of course external ceph. Trying to figure out how to specify data pools... in ceph.conf that cephadm maintains generates is minimal... in olden days, ceph.conf would include rgw gateway info etc. or do I just generate the ceph.conf with the right fsid but add the other stuff for kolla to deploy? | 19:20 |
| *** suff has quit IRC | 19:23 | |
| yoctozepto | iniazi: ceph.conf only really needs to hold the fsid and monitors addresses | 19:52 |
| yoctozepto | the rest is in services config | 19:52 |
| *** alexandreperreau has quit IRC | 19:55 | |
| *** happyhemant has quit IRC | 20:13 | |
| *** k_mouza has quit IRC | 20:49 | |
| iniazi | yoctozepto, i'm not sure what you mean services config... i.e. for kolla-ansible config dir, where I specify a ceph.conf file? or is there something in globals.yml i can set to override data pool for specific services? sorry for newbie questions | 21:08 |
| *** muhaha has quit IRC | 21:12 | |
| *** e0ne has quit IRC | 21:23 | |
| *** gary_perkins has joined #openstack-kolla | 21:27 | |
| iniazi | sorry, my example is from https://themeanti.me/technology/2018/08/23/ceph_erasure_openstack.html, how he says to configure the data pool via ceph.conf | 21:28 |
| iniazi | or maybe you are talking about the same thing I'm talking about... do I just generate a ceph.conf for the services config dir (i.e. config/cinder/ceph.conf and then specify the data pool there (and so on for each of the services)? | 21:31 |
| *** irclogbot_3 has quit IRC | 21:33 | |
| *** sean-k-mooney has quit IRC | 21:33 | |
| *** Wasaac has quit IRC | 21:33 | |
| *** ianw has quit IRC | 21:33 | |
| *** Reepicheep has quit IRC | 21:33 | |
| *** ozzzo has quit IRC | 21:33 | |
| *** atmark has quit IRC | 21:33 | |
| *** Mr_Freezeex has quit IRC | 21:33 | |
| *** wathoom has quit IRC | 21:33 | |
| *** irclogbot_3 has joined #openstack-kolla | 21:34 | |
| *** sean-k-mooney has joined #openstack-kolla | 21:34 | |
| *** Wasaac has joined #openstack-kolla | 21:34 | |
| *** ianw has joined #openstack-kolla | 21:34 | |
| *** Reepicheep has joined #openstack-kolla | 21:34 | |
| *** atmark has joined #openstack-kolla | 21:34 | |
| *** ozzzo has joined #openstack-kolla | 21:34 | |
| *** Mr_Freezeex has joined #openstack-kolla | 21:34 | |
| *** wathoom has joined #openstack-kolla | 21:34 | |
| *** e0ne has joined #openstack-kolla | 21:38 | |
| *** zzzeek has quit IRC | 21:44 | |
| *** zzzeek has joined #openstack-kolla | 21:45 | |
| *** cah_link has quit IRC | 22:09 | |
| *** e0ne has quit IRC | 22:42 | |
| *** bengates_ has quit IRC | 22:47 | |
| *** bengates has joined #openstack-kolla | 22:48 | |
| *** k_mouza has joined #openstack-kolla | 22:49 | |
| *** bengates has quit IRC | 22:52 | |
| *** k_mouza has quit IRC | 22:54 | |
| *** stingrayza has quit IRC | 23:11 | |
| *** stingrayza has joined #openstack-kolla | 23:14 | |
| *** k_mouza has joined #openstack-kolla | 23:17 | |
| *** ysirndjuro has left #openstack-kolla | 23:17 | |
| *** k_mouza has quit IRC | 23:22 | |
| *** k_mouza has joined #openstack-kolla | 23:27 | |
| *** k_mouza has quit IRC | 23:31 | |
| *** k_mouza has joined #openstack-kolla | 23:36 | |
| *** luksky has quit IRC | 23:39 | |
| *** zzzeek has quit IRC | 23:40 | |
| *** k_mouza has quit IRC | 23:41 | |
| *** zzzeek has joined #openstack-kolla | 23:42 | |
| *** vishalmanchanda has quit IRC | 23:44 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!