| *** pescobar has quit IRC | 00:07 | |
| *** pescobar has joined #openstack-kolla | 00:08 | |
| *** rouk has quit IRC | 00:35 | |
| *** vishalmanchanda has quit IRC | 00:41 | |
| *** Arador has quit IRC | 00:46 | |
| *** oyrogerg has quit IRC | 00:48 | |
| *** k_mouza has joined #openstack-kolla | 01:08 | |
| *** k_mouza has quit IRC | 01:20 | |
| *** k_mouza has joined #openstack-kolla | 01:20 | |
| *** k_mouza has quit IRC | 01:25 | |
| *** skramaja has joined #openstack-kolla | 03:02 | |
| *** brinzhang has joined #openstack-kolla | 03:03 | |
| headphoneJames | I have updated the Let's Encrypt spec: https://etherpad.opendev.org/p/kolla-ansible-letsencrypt-https | 03:38 |
|---|---|---|
| *** k_mouza has joined #openstack-kolla | 04:21 | |
| *** zzzeek has quit IRC | 04:24 | |
| *** zzzeek has joined #openstack-kolla | 04:25 | |
| *** k_mouza has quit IRC | 04:26 | |
| *** zzzeek has quit IRC | 04:46 | |
| *** zzzeek has joined #openstack-kolla | 04:49 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-kolla | 05:33 | |
| *** zzzeek has quit IRC | 05:47 | |
| *** zzzeek has joined #openstack-kolla | 05:48 | |
| *** vishalmanchanda has joined #openstack-kolla | 06:00 | |
| oklhost | Morning | 06:37 |
| *** rpittau|afk is now known as rpittau | 06:49 | |
| *** jbadiapa has quit IRC | 06:53 | |
| *** nikparasyr has joined #openstack-kolla | 06:54 | |
| *** abdysn has joined #openstack-kolla | 08:05 | |
| *** wuchunyang has joined #openstack-kolla | 08:07 | |
| openstackgerrit | lixuehai proposed openstack/kolla-ansible stable/train: Use hostPID for ceph-mgr deployment https://review.opendev.org/763134 | 08:16 |
| *** k_mouza has joined #openstack-kolla | 08:22 | |
| openstackgerrit | lixuehai proposed openstack/kolla-ansible stable/rocky: Use hostPID for ceph-mgr deployment https://review.opendev.org/763135 | 08:24 |
| *** k_mouza has quit IRC | 08:26 | |
| *** bengates has joined #openstack-kolla | 08:27 | |
| openstackgerrit | Victor Chembaev proposed openstack/kolla-ansible master: 1904702: rabbitmq handler refactored to restart services in serial https://review.opendev.org/763137 | 08:56 |
| mnasiadka | morning | 08:57 |
| mnasiadka | yoctozepto: so we're dead in the water with docker hub? ;) | 08:57 |
| yoctozepto | mnasiadka: pretty much, though I doubt it's water any longer! | 08:59 |
| mnasiadka | well, I don't see good alternatives | 08:59 |
| mnasiadka | quay.io has the same problem, but in smaller extent, github packages has some transfer limits I think | 09:00 |
| mnasiadka | ideally infra could host a registry, it would even limit the amount of traffic we are doing in CI ;) | 09:00 |
| openstackgerrit | Victor Chembaev proposed openstack/kolla-ansible master: RabbitMQ handler refactored to restart services in serial https://review.opendev.org/763137 | 09:03 |
| openstackgerrit | Victor Chembaev proposed openstack/kolla-ansible master: RabbitMQ handler refactored to restart services in serial https://review.opendev.org/763137 | 09:05 |
| *** mgoddard has joined #openstack-kolla | 09:07 | |
| *** parallax has quit IRC | 09:09 | |
| mnasiadka | yoctozepto: hmm, github packages is free for public packages - maybe this is the option we want to investigate? | 09:09 |
| *** parallax has joined #openstack-kolla | 09:09 | |
| mgoddard | headphoneJames: hi | 09:10 |
| *** shyamb has joined #openstack-kolla | 09:13 | |
| *** Abdallahyas has joined #openstack-kolla | 09:21 | |
| *** abdysn has quit IRC | 09:23 | |
| mgoddard | mnasiadka: infra does host a registry, but it's only supposed to be used for CI jobs | 09:24 |
| *** yankcrime has quit IRC | 09:24 | |
| mnasiadka | mgoddard: well then as I said, maybe it's time to evaluate Github? | 09:24 |
| mnasiadka | we're not really happy with signing a pact with the devil I guess (regarding open source agreement with Docker) :) | 09:25 |
| hrw | morning | 09:25 |
| mgoddard | well, we need to switch to the infra registry at some point | 09:25 |
| hrw | what about move from daily to weekly publish? | 09:25 |
| mgoddard | hrw: then jobs only fail 1 day per week? :) | 09:26 |
| hrw | mgoddard: and stronger accent put on building images by users | 09:26 |
| mnasiadka | I think there's a lot of users, which don't build images by themselves, and it was convenient for them they could just download them off docker hub | 09:27 |
| hrw | so weekly images for them | 09:27 |
| mnasiadka | we need to be better in pushing and failing the publish job | 09:28 |
| mnasiadka | now it doesn't fail when there's a failed push for one of the images | 09:28 |
| hrw | and if we fail then unpublish just published ones? | 09:29 |
| hrw | as we publish :master each time this ends with lack of complete sets | 09:30 |
| *** shyamb has quit IRC | 09:30 | |
| *** kukacz has quit IRC | 09:31 | |
| mnasiadka | hrw: I was rather thinking about retries at least, not unpublishing :) | 09:31 |
| *** abdysn has joined #openstack-kolla | 09:31 | |
| hrw | push, ok, push, ok, push, 'sorry you reached a limit', retry push, 'sorry', retry push, 'sorry' you mean? | 09:32 |
| *** Abdallahyas has quit IRC | 09:33 | |
| hrw | and then TIMEOUT | 09:34 |
| wuchunyang | I think there's a lot of users, which don't build images by themselves, and it was convenient for them they could just download them off docker hub +1 | 09:35 |
| wuchunyang | in china, many repos can not access . so it always build failed | 09:36 |
| hrw | and we publish ~1500 images. | 09:37 |
| hrw | 6 sets for x86-64, 1 for aarch64 | 09:37 |
| hrw | one set is 200-250 images. if we lucky then most of x86-64 are on separate CI systems | 09:38 |
| *** kukacz has joined #openstack-kolla | 09:38 | |
| wuchunyang | hrw does it build all images in ci ? | 09:38 |
| *** e0ne has joined #openstack-kolla | 09:38 | |
| hrw | wuchunyang: all 'buildable' ones | 09:38 |
| *** Fl1nt has joined #openstack-kolla | 09:39 | |
| hrw | btw - yesterday was 3y since I became kolla core ;d | 09:39 |
| Fl1nt | good morning! | 09:39 |
| Fl1nt | happy bday core contributor :p | 09:39 |
| hrw | https://marcin.juszkiewicz.com.pl/2017/11/17/i-am-now-core-reviewer-in-kolla/ | 09:39 |
| wuchunyang | hrw thanks for your nice work. | 09:39 |
| hrw | thx | 09:40 |
| wuchunyang | we use kolla in produciton, and it works happily now | 09:41 |
| hrw | wuchunyang: x86-64? | 09:42 |
| wuchunyang | the best paint is monitoring in our env . | 09:42 |
| wuchunyang | hrw yes | 09:42 |
| wuchunyang | do you use arm server ? | 09:43 |
| hrw | wuchunyang: then mostly someone's else work than mine ;D | 09:43 |
| Fl1nt | team, if I correctly tracked down the change, this is the commit that introduced support for backend traffic encryption right? https://opendev.org/openstack/kolla-ansible/commit/b475643c112f49701db2fb3d1493987888c97c8a | 09:43 |
| hrw | wuchunyang: aarch64 only | 09:43 |
| Fl1nt | for haproxy role of course | 09:43 |
| wuchunyang | hrw cool. we do not use aarch64 now .. many we use it in future . | 09:44 |
| Fl1nt | hrw, do you think Apple entering the ARM (not really ARM but yet based on ARM RISC arch) market and kind of proving that you can get better performance than traditional x86_64 CPUs with lower amount of cores etc is a good thing for the whole ARM Servers world? | 09:45 |
| *** yoctozepto has quit IRC | 09:50 | |
| *** yoctozepto has joined #openstack-kolla | 09:51 | |
| *** brinzhang_ has joined #openstack-kolla | 09:51 | |
| *** brinzhang has quit IRC | 09:54 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: CI: Improve publish playbook https://review.opendev.org/763146 | 09:54 |
| yoctozepto | mgoddard, mnasiadka: https://docs.github.com/en/free-pro-team@latest/packages/getting-started-with-github-container-registry/migrating-to-github-container-registry-for-docker-images | 09:55 |
| yoctozepto | they have container registry in beta | 09:55 |
| yoctozepto | essentially "packages" look deprecated | 09:56 |
| *** zzzeek has quit IRC | 09:56 | |
| hrw | Fl1nt: M1 is Arm. | 09:56 |
| yoctozepto | seemingly public repos are void of any limits | 09:56 |
| yoctozepto | well, microsoft might be able to execute that | 09:56 |
| hrw | Fl1nt: Apple is Arm licensee with license to make own cpu cores. Each such design has to pass testsuite to be called Arm cpu. | 09:57 |
| Fl1nt | hrw, that was my thought but I read somewhere that it kind of slightly slip from what ARM licence is and so can't be named ARM strictly but yes. | 09:57 |
| mnasiadka | yoctozepto: yeah, just a question which account are we going to use - create a kolla one, or ask infra to create us a token under openstack namespace | 09:57 |
| hrw | Fl1nt: Apple calls it 'Apple Silicon' to bump name ;D | 09:57 |
| Fl1nt | yeah marketing stunt :p | 09:58 |
| yoctozepto | mnasiadka: well, considering that each team goes their way, I would not want to stress infra before we settle on a final solution | 09:58 |
| yoctozepto | and then it's probably not required either | 09:58 |
| Fl1nt | I'm really astonished to the power gain over my classical intel based mac so far | 09:58 |
| yoctozepto | I see this as an extra for users anyhow | 09:58 |
| mnasiadka | yeah, and if we can keep kolla/ namespace in the same way, it might be easier | 09:58 |
| hrw | Fl1nt: and yes, M1 will improve Arm space. More people will use Arm cpus so things like support for aarch64 in languages/libraries will need to improve. | 09:59 |
| *** zzzeek has joined #openstack-kolla | 09:59 | |
| hrw | Fl1nt: also containers/microservices/etc will get more and more often built for both aarch64 and x86-64 | 09:59 |
| mnasiadka | yoctozepto: well, probably we can't keep the same namespace, there's an individual under github.com/kolla ;-) | 09:59 |
| yoctozepto | mnasiadka: obviously, Koll Andreas | 10:00 |
| yoctozepto | :D | 10:00 |
| yoctozepto | that's it for our luck | 10:00 |
| yoctozepto | anyhow, kolla does not mean much outside the openstack community | 10:00 |
| yoctozepto | so we could be openstack-kolla | 10:00 |
| mnasiadka | yup | 10:01 |
| mnasiadka | oh boy, changing the publish playbook spawns all kolla-build and k-a jobs | 10:03 |
| yoctozepto | we don't have to go day-1 official source of images, remember that | 10:03 |
| *** shyamb has joined #openstack-kolla | 10:03 | |
| yoctozepto | also, for CI we might still be better off using infra's container registry | 10:03 |
| yoctozepto | and only then replicate to the "proper" one | 10:04 |
| hrw | so one job to publish to infra registry, one to publish outside and then k-a use infra registry on runs? | 10:06 |
| *** k_mouza has joined #openstack-kolla | 10:14 | |
| *** yankcrime has joined #openstack-kolla | 10:16 | |
| mnasiadka | mgoddard, yoctozepto: so seems we have a statement around Cinder | 10:20 |
| mgoddard | mnasiadka: looks like it | 10:24 |
| mgoddard | gorka's blog post from 5 years ago: https://gorka.eguileor.com/a-cinder-road-to-activeactive-ha/ | 10:24 |
| Fl1nt | hrw, still working on my "lab" project on my own, I've made it available to a few contacts that are really happy for now, as this step is validated, I'll move on with a real ARM server platform, but it's really hard to find what I'm looking for so far. | 10:25 |
| mgoddard | we still have a bit of a mess to figure out, given we have potential ussuri deployments without backend_host | 10:25 |
| mgoddard | we'll need to make sure we have a path that works for everyone | 10:25 |
| *** kevinz has quit IRC | 10:26 | |
| Fl1nt | mgoddard, I took time to look at this cinder patch, it's utterly non sense, well, for RBD I mean. It trigger CEPH works where it's not needed, but you where right on point, the patch is actually doing kind of a A/A as it create a failover operation on failure detection. | 10:28 |
| Fl1nt | on failure detection of a cinder-volume agent. | 10:28 |
| yoctozepto | mgoddard: yeah, we need to send our sorries in release notes | 10:28 |
| hrw | Fl1nt: and what you look for? | 10:29 |
| *** Abdallahyas has joined #openstack-kolla | 10:29 | |
| Fl1nt | I'm looking for a SoC tiny enough to not requiring any active thermal dissipation system, ideally with a minimum of 8 core at 2.5Ghz but it seems like everything close to those specs are custom made one for big company. | 10:31 |
| *** abdysn has quit IRC | 10:33 | |
| *** Abdallahyas has quit IRC | 10:35 | |
| mnasiadka | mgoddard: so let's add back backend_host, as part of the bugfix (so we don't do mayhem now), and then work on the upgrade path to proper A/A | 10:35 |
| yoctozepto | mgoddard: 'tis better https://gorka.eguileor.com/simpler-road-to-cinder-active-active/ | 10:36 |
| mnasiadka | I don't think we'll be able to move to proper A/A without proper coordination backend testing and doing additional upgrade steps | 10:36 |
| yoctozepto | mnasiadka: the issue is we need to do more | 10:36 |
| yoctozepto | simply adding backend_host will not make current ussuri deployments happy | 10:36 |
| yoctozepto | it can break some | 10:36 |
| yoctozepto | we need to provide extra guidance | 10:36 |
| mnasiadka | it's still less time to deliver than doing proper A/A | 10:37 |
| mnasiadka | or maybe the same, if we want to setup a test environment and move from no backend_host to backend_host | 10:38 |
| mnasiadka | ;) | 10:38 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: fluentd: Remove conditions from handler https://review.opendev.org/763155 | 10:40 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: fluentd: Remove conditions from handler https://review.opendev.org/763155 | 10:44 |
| *** dougsz has joined #openstack-kolla | 10:46 | |
| hrw | Fl1nt: check SolidRun HoneyComb. | 10:46 |
| hrw | Fl1nt: mini-itx, 16 cores, up to 64GB ram, nvme, pcie | 10:47 |
| *** wuchunyang has quit IRC | 10:55 | |
| *** dougsz has quit IRC | 11:03 | |
| *** abdysn has joined #openstack-kolla | 11:09 | |
| yoctozepto | mnasiadka: well, we know what will break | 11:13 |
| yoctozepto | mnasiadka: volumes associated to the now-missing backend_host | 11:14 |
| mgoddard | kolla in the top500: https://www.stackhpc.com/sc20-top500.html | 11:39 |
| *** e0ne has quit IRC | 11:44 | |
| openstackgerrit | likui proposed openstack/kolla-ansible master: Update docs for Ubuntu 20.04 https://review.opendev.org/763167 | 11:44 |
| openstackgerrit | Victor Chembaev proposed openstack/kolla-ansible master: RabbitMQ handler refactored to restart services in serial https://review.opendev.org/763137 | 11:46 |
| *** wuchunyang has joined #openstack-kolla | 11:50 | |
| *** wuchunyang has quit IRC | 12:03 | |
| *** shyamb has quit IRC | 12:05 | |
| *** zzzeek has quit IRC | 12:13 | |
| *** zzzeek has joined #openstack-kolla | 12:16 | |
| *** e0ne has joined #openstack-kolla | 12:18 | |
| *** zzzeek has quit IRC | 12:27 | |
| *** zzzeek has joined #openstack-kolla | 12:28 | |
| *** stingrayza has joined #openstack-kolla | 12:33 | |
| *** Abdallahyas has joined #openstack-kolla | 12:37 | |
| *** abdysn has quit IRC | 12:41 | |
| *** abdysn has joined #openstack-kolla | 12:41 | |
| *** Abdallahyas has quit IRC | 12:42 | |
| *** wuchunyang has joined #openstack-kolla | 12:46 | |
| *** jonaspaulo has joined #openstack-kolla | 12:52 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: [baremetal]: Use $releasever in docker-ce repo https://review.opendev.org/762979 | 12:55 |
| *** jonaspaulo has quit IRC | 12:55 | |
| *** Abdallahyas has joined #openstack-kolla | 12:59 | |
| yoctozepto | mgoddard: woop woop | 12:59 |
| *** abdysn has quit IRC | 13:00 | |
| *** abdysn has joined #openstack-kolla | 13:00 | |
| *** Abdallahyas has quit IRC | 13:04 | |
| *** abdysn has quit IRC | 13:10 | |
| *** wuchunyang has quit IRC | 13:16 | |
| yoctozepto | mgoddard, mnasiadka: centos also moving to quay https://pagure.io/centos-infra/issue/77 | 13:20 |
| yoctozepto | dockerhub will be limiting them | 13:20 |
| yoctozepto | what a joke | 13:21 |
| yoctozepto | well, I guess I'm just biased and sad | 13:21 |
| yoctozepto | but surely everyone benefits from open source so why make it sooo complicated | 13:21 |
| *** wuchunyang has joined #openstack-kolla | 13:27 | |
| *** kevko has joined #openstack-kolla | 13:28 | |
| kevko | hi guys, could someone advice my how can i get list of subgroups of some group in ansible :/ ? | 13:28 |
| Fl1nt | hrw, thanks a lot, gonna check that out. | 13:29 |
| openstackgerrit | Bartosz Bezak proposed openstack/kayobe master: Support for firewalld ZONE option in network interfaces configuration https://review.opendev.org/763180 | 13:31 |
| Fl1nt | oooh so cool, they even offers customization and redesign, that's awesome hrw !!! <3 | 13:32 |
| Fl1nt | yoctozepto, well, I would say because money? | 13:34 |
| Fl1nt | mgoddard, really interesting share about HPC | 13:35 |
| hrw | Fl1nt: if you go for their services then mention me ;D | 13:36 |
| Fl1nt | with your real name or your irc one? | 13:36 |
| hrw | Fl1nt: not that I will have something from it ;D | 13:36 |
| hrw | Fl1nt: both would be best | 13:36 |
| Fl1nt | yeah but it's still really important for both of you, them from a business standpoint and you from a relationship/trust standpoint so I use their services for sure I'll do. | 13:37 |
| Fl1nt | s/so I use/so if I use/ | 13:37 |
| mgoddard | yoctozepto: I don't think centos is moving to quay, just that they have images on quay and dockerhub | 13:38 |
| hrw | Fl1nt: thanks | 13:39 |
| *** stingrayza has quit IRC | 13:39 | |
| *** stingrayza has joined #openstack-kolla | 13:40 | |
| mgoddard | stein fix we should merge soon: https://review.opendev.org/#/c/763155/ | 13:40 |
| patchbot | patch 763155 - kolla-ansible (stable/stein) - fluentd: Remove conditions from handler - 2 patch sets | 13:40 |
| hrw | mgoddard: +2 with comment | 13:44 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: fluentd: Remove conditions from handler https://review.opendev.org/763155 | 13:48 |
| mgoddard | hrw: fair point | 13:48 |
| *** e0ne has quit IRC | 13:51 | |
| hrw | +2 again | 13:51 |
| yoctozepto | mgoddard: https://review.opendev.org/763155 | 13:58 |
| patchbot | patch 763155 - kolla-ansible (stable/stein) - fluentd: Remove conditions from handler - 3 patch sets | 13:58 |
| yoctozepto | mgoddard: yeah, the 'move' was an overstatement but they are willing to revive their quay images because of dockerhub limits applying to them | 14:00 |
| *** e0ne has joined #openstack-kolla | 14:01 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: fluentd: Remove conditions from handler https://review.opendev.org/763155 | 14:02 |
| mgoddard | yoctozepto: updated | 14:02 |
| yoctozepto | mgoddard: thx, super clear now | 14:07 |
| *** jaicaa has quit IRC | 14:14 | |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: Use trove.conf for all trove services https://review.opendev.org/763191 | 14:19 |
| *** TrevorV has joined #openstack-kolla | 14:32 | |
| mgoddard | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak | 14:50 |
| mgoddard | ^ meeting in 10 | 14:50 |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: [CI] Add trove smoke test CI https://review.opendev.org/763197 | 14:54 |
| *** skramaja has quit IRC | 14:54 | |
| wuchunyang | https://review.opendev.org/#/c/763155/ this change , i think we should +w now ? | 15:00 |
| patchbot | patch 763155 - kolla-ansible (stable/stein) - fluentd: Remove conditions from handler - 4 patch sets | 15:00 |
| mgoddard | #startmeeting kolla | 15:01 |
| openstack | Meeting started Wed Nov 18 15:01:23 2020 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| *** openstack changes topic to " (Meeting topic: kolla)" | 15:01 | |
| openstack | The meeting name has been set to 'kolla' | 15:01 |
| mgoddard | #topic rollcall | 15:01 |
| *** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:01 | |
| *** imtiazc has joined #openstack-kolla | 15:01 | |
| yoctozepto | o/ | 15:02 |
| mgoddard | \o | 15:02 |
| wuchunyang | o/ | 15:02 |
| headphoneJames | o/ | 15:02 |
| mgoddard | #topic agenda | 15:04 |
| *** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:04 | |
| mgoddard | * Roll-call | 15:04 |
| mgoddard | * Announcements | 15:04 |
| mgoddard | ** Vote for Kolla Wallaby priorities https://etherpad.opendev.org/p/kolla-wallaby-priorities | 15:04 |
| mgoddard | * Review action items from the last meeting | 15:04 |
| mgoddard | * CI status | 15:04 |
| mgoddard | * Victoria release planning | 15:04 |
| mgoddard | * Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits | 15:04 |
| mgoddard | * Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062 | 15:04 |
| openstack | Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka) | 15:04 |
| mgoddard | * Wallaby PTG actions | 15:04 |
| mgoddard | * Review new retirements (Wallaby) | 15:04 |
| mgoddard | * Cinder v2 to be dropped in Wallaby http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018697.html | 15:04 |
| mgoddard | #topic announcements | 15:04 |
| *** openstack changes topic to "announcements (Meeting topic: kolla)" | 15:04 | |
| mgoddard | #info Vote for Kolla Wallaby priorities | 15:04 |
| mgoddard | #link https://etherpad.opendev.org/p/kolla-wallaby-priorities | 15:04 |
| mgoddard | I'll close the poll at the end of the week | 15:05 |
| yoctozepto | ++ | 15:05 |
| mgoddard | #action mgoddard to email openstack-discuss about final reminder for wallaby priority voting | 15:05 |
| mgoddard | Any other announcements? | 15:06 |
| wuchunyang | no... | 15:07 |
| mgoddard | #topic Review action items from the last meeting | 15:07 |
| *** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)" | 15:07 | |
| mgoddard | mgoddard check if escurator broken and mark unbuildable if so | 15:07 |
| mgoddard | I think this was resolved | 15:07 |
| mgoddard | #topic CI status | 15:07 |
| *** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:07 | |
| mnasiadka | resolved with a workaround I think, we are still waiting for escurator to make a new release and go back to normal :) | 15:08 |
| mgoddard | ok | 15:08 |
| mgoddard | [minor] NFV job broken due to lack of Aodh for Tacker/Heat | 15:09 |
| mgoddard | I guess that affects master & victoria? | 15:09 |
| mgoddard | master/victoria: 2020-10-23 08:02:41.182 6 CRITICAL nova [req-24a27c19-7657-43dc-bb1b-b93223fd8b29 - - - - -] Unhandled error: TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs' - https://bugs.launchpad.net/nova/+bug/1902696 | 15:12 |
| openstack | Launchpad bug 1902696 in oslo.messaging "nova-compute fails with Unhandled error: TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs'" [Undecided,New] | 15:12 |
| imtiazc | Is there any plan to support Selinux with Kolla? | 15:13 |
| mgoddard | imtiazc: we're in a meeting right nw | 15:13 |
| mnasiadka | mgoddard: nfv and nova rabbitmq tls bug are two different things? | 15:14 |
| mgoddard | yes | 15:14 |
| mgoddard | I got no response so moved on... | 15:15 |
| yoctozepto | i will take a look at that nfv later this week | 15:15 |
| yoctozepto | just hoped someone else would do but it seems no real interest in nfv nowadays | 15:16 |
| *** Arador has joined #openstack-kolla | 15:17 | |
| wuchunyang | i have never used nfv before.. | 15:18 |
| mgoddard | I just commented on https://review.opendev.org/#/c/761194/. I'm not convinced we need to wait for a kombu release | 15:19 |
| patchbot | patch 761194 - requirements - Pin kombu and amqp requirements due to incompatibi... - 2 patch sets | 15:19 |
| mnasiadka | mgoddard: I think it needs some analysis ;) | 15:19 |
| mgoddard | maybe. I'm no requirements expert | 15:20 |
| mnasiadka | or maybe it's already fixed, there was a revert somewhere - let's just recheck it and analyse if it fails ;) | 15:20 |
| yoctozepto | wuchunyang: me neither | 15:20 |
| mnasiadka | yoctozepto: nobody has :) | 15:20 |
| mgoddard | https://review.opendev.org/#/c/761519/ | 15:20 |
| patchbot | patch 761519 - kolla-ansible - Revert "CI: Temporarily disable rabbitmq internal ... - 1 patch set | 15:20 |
| yoctozepto | mnasiadka: :D | 15:21 |
| mgoddard | ok, move on? | 15:21 |
| mnasiadka | mgoddard: sent for recheck, will monitor that | 15:21 |
| mnasiadka | yep, move on! | 15:21 |
| mgoddard | #topic Victoria release planning | 15:21 |
| *** openstack changes topic to "Victoria release planning (Meeting topic: kolla)" | 15:21 | |
| yoctozepto | mgoddard: you can action me on nfv | 15:22 |
| mgoddard | #action yoctozepto fix NFV | 15:22 |
| mgoddard | I think we are at the point where we should list the blockers for the Victoria release | 15:22 |
| mgoddard | #link https://etherpad.opendev.org/p/KollaWhiteBoard | 15:23 |
| mgoddard | L144 | 15:23 |
| mgoddard | hit me | 15:23 |
| mgoddard | RMQ TLS? | 15:23 |
| mnasiadka | for sure | 15:24 |
| mgoddard | probably not a real blocker | 15:24 |
| mgoddard | but would be nice to release the feature working | 15:24 |
| mnasiadka | well, we can post a release note that it currently doesn't work due to this bug, and then post a reno when it's fixed | 15:24 |
| mgoddard | Any other blockers? | 15:26 |
| mgoddard | Cinder active/active | 15:26 |
| mnasiadka | no critical bugs in launchpad | 15:27 |
| mnasiadka | well, cinder active/active with current config just enforces users to rewrite volumes. to another cinder-volume agent ;) | 15:27 |
| mnasiadka | but we have that somewhere later in agenda I think | 15:27 |
| mgoddard | yes | 15:28 |
| mgoddard | I don't think we should release without a long-term config though | 15:28 |
| mgoddard | https://review.opendev.org/#/c/760308/ | 15:28 |
| patchbot | patch 760308 - kolla-ansible - kibana: Remove 6.x migration from upgrade - 1 patch set | 15:28 |
| yoctozepto | mgoddard: long-term | 15:30 |
| yoctozepto | ? | 15:30 |
| yoctozepto | mgoddard: you mean to push a-a? | 15:30 |
| yoctozepto | or a/a | 15:30 |
| mnasiadka | so should we discuss it now, if it's a blocker and how are we planning to make it work? | 15:30 |
| mgoddard | I mean let's not release with one config, then recommend changing it once we've worked out how to do it properly | 15:31 |
| mgoddard | anyway, it's a later topic | 15:31 |
| mgoddard | any other release blockers? | 15:32 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Revert "CI: Temporarily disable rabbitmq internal tls" https://review.opendev.org/761519 | 15:33 |
| mgoddard | I proposed an RC1 release for kayobe, still waiting on it | 15:33 |
| mgoddard | https://review.opendev.org/#/c/763022/ | 15:34 |
| patchbot | patch 763022 - releases - Release Kayobe 9.0.0.0rc1 and branch for Victoria - 1 patch set | 15:34 |
| mgoddard | #topic Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits | 15:34 |
| *** openstack changes topic to "Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits (Meeting topic: kolla)" | 15:35 | |
| yoctozepto | what should we do | 15:35 |
| yoctozepto | oh what should we doooo | 15:35 |
| yoctozepto | :D | 15:35 |
| mnasiadka | remove kolla in Docker Hub and deprecate ;) | 15:35 |
| * mgoddard makes notes in https://etherpad.opendev.org/p/docker-pull-limits | 15:36 | |
| * yoctozepto does too | 15:37 | |
| yoctozepto | let's look there mnasiadka | 15:38 |
| mnasiadka | looking | 15:38 |
| mgoddard | yoctozepto, mnasiadka, hrw | 15:45 |
| mgoddard | are those the options? | 15:45 |
| mgoddard | are there more? | 15:46 |
| *** nikparasyr has left #openstack-kolla | 15:46 | |
| mnasiadka | there was a proposal to push weekly and encourage users to build their own images, but it doesn't matter if we fail daily or weekly (well matters if we wait another week to push something, and we fail again) | 15:46 |
| yoctozepto | mgoddard: 3 is likely extra to me, not a user-faced option | 15:47 |
| yoctozepto | mnasiadka: well, limits are mostly on pulls | 15:47 |
| mnasiadka | yeah, that's true | 15:47 |
| yoctozepto | still, the limits are per ip address | 15:47 |
| mgoddard | well if we do 3 then we don't use dockerhub in CI | 15:47 |
| yoctozepto | so unless one user downloads more than 100 images at once | 15:47 |
| mgoddard | so no limits | 15:47 |
| yoctozepto | they should be pretty happy | 15:47 |
| yoctozepto | mgoddard: yeah, that's what I started to explore in my mind | 15:48 |
| yoctozepto | mgoddard: the general - of 3 is that we don't yet know how it is supposed to work | 15:49 |
| mgoddard | unclear if the registry is designed to hold images between jobs? | 15:49 |
| mgoddard | yoctozepto: I had a look at it, I have more of an understanding now | 15:49 |
| mgoddard | I think we need some more info from infra about the intermediate registry | 15:51 |
| yoctozepto | mgoddard: I looked at it in the past, though I had lots of other stuff on my mind so forgot | 15:52 |
| yoctozepto | mgoddard: one is definitely temporary | 15:52 |
| yoctozepto | mgoddard: so it would not do | 15:52 |
| mgoddard | buildset is temporary | 15:52 |
| yoctozepto | mgoddard: not remember any permanent | 15:52 |
| mgoddard | intermediate is permanent but not guaranteed to be reliable | 15:52 |
| yoctozepto | mgoddard: hah, could you link to the docs? | 15:54 |
| mgoddard | #link https://zuul-ci.org/docs/zuul-jobs/docker-image.html | 15:55 |
| mgoddard | #link https://docs.opendev.org/opendev/base-jobs/latest/docker-image.html | 15:55 |
| mgoddard | I don't think we have an answer, but we do have some options listed | 15:56 |
| mgoddard | let's move on | 15:56 |
| yoctozepto | mgoddard: but this intermediate is infra-level | 15:56 |
| mgoddard | yes | 15:56 |
| mgoddard | #topic Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062 | 15:56 |
| *** openstack changes topic to "Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062 (Meeting topic: kolla)" | 15:56 | |
| openstack | Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka) | 15:56 |
| mgoddard | What cases to we need to consider? | 15:57 |
| mgoddard | Train with backend_host set | 15:58 |
| yoctozepto | it's probably best to go real a/a already | 15:58 |
| yoctozepto | less playing around | 15:58 |
| mgoddard | Ussuri with backend_host set | 15:58 |
| yoctozepto | yes, train is supposedly fine | 15:58 |
| mgoddard | Ussuri with backend_host not set | 15:58 |
| yoctozepto | yup | 15:58 |
| yoctozepto | we might want to craft help instructions for currently affected | 15:58 |
| mgoddard | So we only need to change ussuri (and beyond) | 15:58 |
| yoctozepto | so train->ussuri broken upgraders | 15:59 |
| mgoddard | and ensure that an upgrade from train with backend_host set works | 15:59 |
| mgoddard | and bear in mind that users may have custom config with backend_host in :) | 15:59 |
| yoctozepto | yeah, lovely case | 15:59 |
| yoctozepto | we need to use all our channels to make this a known issue | 16:00 |
| yoctozepto | which means | 16:00 |
| yoctozepto | release notes | 16:00 |
| yoctozepto | and mailing list | 16:00 |
| openstackgerrit | Victor Chembaev proposed openstack/kolla-ansible master: RabbitMQ handler refactored to restart services in serial https://review.opendev.org/763137 | 16:00 |
| yoctozepto | and potentially personal emails | 16:00 |
| mgoddard | kolla klub | 16:00 |
| yoctozepto | yup | 16:00 |
| mgoddard | so I guess first step is to decide on the end state | 16:01 |
| mgoddard | cluster=ceph, coordinator enabled | 16:01 |
| mgoddard | precheck for a coordinator? | 16:01 |
| mnasiadka | yoctozepto: well, we had backend_host in docs only for external ceph, if someone had template overrides for cinder in Train, it should work in Ussuri I think? | 16:02 |
| mgoddard | mnasiadka: yes. it should work as well as it ever did | 16:03 |
| mgoddard | the problem comes if you try to remove the custom config and use what's in kolla | 16:03 |
| mnasiadka | so, we should have a precheck for coordinator backend - and use redis or etcd when they are enabled | 16:04 |
| mnasiadka | should we favor one over the other, if both are enabled? | 16:04 |
| mgoddard | we do use them if enabled | 16:04 |
| mnasiadka | so we just need precheck, if external ceph and cinder-volume group has more than one host | 16:05 |
| mnasiadka | or even only if cinder-volume > 1 | 16:05 |
| mgoddard | yes | 16:05 |
| yoctozepto | though we need to make redis and/or etcd robust in ha | 16:05 |
| mnasiadka | so we need to test both | 16:06 |
| yoctozepto | otherwise it might be better to stick to bad and fugly active-active in active-passive | 16:06 |
| mgoddard | ? | 16:06 |
| mnasiadka | do we have any CI with coordination backend for any service? | 16:06 |
| mgoddard | I don't believe so | 16:07 |
| mnasiadka | well, anyway I need to look into why ceph-ansible jobs are failing so often, so we just may enable it there\ | 16:07 |
| mnasiadka | I remember we had some issues with etcd3/etcd3gw, so maybe this is the backend we should look into ;) | 16:07 |
| yoctozepto | mnasiadka: etcd3gw is the way to go | 16:09 |
| mgoddard | let's wrap up, we're over time | 16:09 |
| yoctozepto | the other one breaks w/ eventlet | 16:09 |
| yoctozepto | let's continue after | 16:09 |
| yoctozepto | thx mgoddard | 16:09 |
| mgoddard | #endmeeting | 16:09 |
| *** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:09 | |
| openstack | Meeting ended Wed Nov 18 16:09:58 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:10 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-11-18-15.01.html | 16:10 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-11-18-15.01.txt | 16:10 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-11-18-15.01.log.html | 16:10 |
| openstackgerrit | zengchen proposed openstack/kolla-ansible master: Fix kolla-ansible to work with pyenv-virtualenv. https://review.opendev.org/762365 | 16:11 |
| *** wuchunyang has quit IRC | 16:15 | |
| *** bsanjeewa has joined #openstack-kolla | 16:16 | |
| bsanjeewa | Hi, I'm trying to deploy 2 node overcloud with "kayobe overcloud provision", both nodes reboot after image deployment, with active Provisioning State. But they do not boot into the disk. UEFI doesn't even list hard disk as a boot option. What to do? | 16:19 |
| yoctozepto | mgoddard, mnasiadka: still some time for cinder a/a discussion? | 16:21 |
| mgoddard | hi bsanjeewa, you need to modify the image for UEFI | 16:22 |
| mgoddard | ipa_build_dib_packages: | 16:22 |
| mgoddard | - "gdisk" | 16:22 |
| mgoddard | - "efibootmgr" | 16:22 |
| mgoddard | - "efivar" | 16:22 |
| mgoddard | kolla_bifrost_dib_elements: | 16:23 |
| mgoddard | - "block-device-efi" | 16:23 |
| mgoddard | - "grub2" | 16:23 |
| mgoddard | - ... | 16:23 |
| mgoddard | bsanjeewa: http://paste.openstack.org/show/800169/ | 16:23 |
| mgoddard | yoctozepto: yes, in 5 mins | 16:24 |
| bsanjeewa | mgoddard: thanks. will try it and update | 16:24 |
| mgoddard | back | 16:29 |
| *** bsanjeewa has quit IRC | 16:30 | |
| openstackgerrit | Merged openstack/kolla-ansible master: kibana: Remove 6.x migration from upgrade https://review.opendev.org/760308 | 16:33 |
| *** yan0s has joined #openstack-kolla | 16:37 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/victoria: kibana: Remove 6.x migration from upgrade https://review.opendev.org/763214 | 16:37 |
| yoctozepto | mgoddard: ok | 16:45 |
| yoctozepto | and mnasiadka?\ | 16:45 |
| yoctozepto | seems not | 16:51 |
| yoctozepto | a'ight | 16:51 |
| yoctozepto | shall we make another etherpad for cinder? | 16:51 |
| yoctozepto | https://etherpad.opendev.org/p/kolla-cinder-ceph-active-active | 16:52 |
| *** bengates has quit IRC | 16:56 | |
| mgoddard | yoctozepto: Kolla Ussuri dropped internal Ceph | 16:57 |
| mgoddard | it's not related | 16:57 |
| *** bengates has joined #openstack-kolla | 16:57 | |
| mgoddard | yoctozepto: in Train we documented config for external ceph (inc backend_host) | 16:58 |
| mgoddard | in Ussuri we automated it, without backend_host | 16:58 |
| mgoddard | unclear why it was dropped | 16:58 |
| mgoddard | possibly in some gerrit comment somewhere | 16:58 |
| mnasiadka | back | 16:58 |
| mnasiadka | I think I dropped it, because it was meant for active/passive - maybe just didn't thoroughly think it through - that now we have non-ha cinder-volume | 16:59 |
| mnasiadka | personally I think it's not that bad, but we should add a note how to move volumes from one cinder-volume to another with cinder-manage | 17:01 |
| *** bengates has quit IRC | 17:02 | |
| yoctozepto | per what was said in reply to us | 17:03 |
| yoctozepto | for those happy with no `backend_host`, we can just add `cluster` and configure coordination properly | 17:03 |
| yoctozepto | or even configure `cluster` and have slightly better behaviour than backend_host | 17:03 |
| yoctozepto | still with no coordination | 17:03 |
| openstackgerrit | OpenStack Release Bot proposed openstack/kayobe stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/763219 | 17:17 |
| openstackgerrit | OpenStack Release Bot proposed openstack/kayobe stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/763220 | 17:17 |
| openstackgerrit | OpenStack Release Bot proposed openstack/kayobe master: Update master for stable/victoria https://review.opendev.org/763221 | 17:17 |
| openstackgerrit | OpenStack Release Bot proposed openstack/kayobe master: Add Python3 wallaby unit tests https://review.opendev.org/763222 | 17:17 |
| openstackgerrit | OpenStack Release Bot proposed openstack/kayobe-config stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/763223 | 17:17 |
| openstackgerrit | OpenStack Release Bot proposed openstack/kayobe-config stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/763224 | 17:17 |
| openstackgerrit | OpenStack Release Bot proposed openstack/kayobe-config-dev stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/763225 | 17:18 |
| openstackgerrit | OpenStack Release Bot proposed openstack/kayobe-config-dev stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/763226 | 17:18 |
| *** e0ne has quit IRC | 17:21 | |
| *** e0ne has joined #openstack-kolla | 17:22 | |
| *** mgoddard has quit IRC | 17:33 | |
| *** e0ne has quit IRC | 17:33 | |
| mnasiadka | yoctozepto: yeah, I'll look into that tomorrow | 17:34 |
| *** jaicaa has joined #openstack-kolla | 17:43 | |
| *** bsanjeewa has joined #openstack-kolla | 17:49 | |
| *** e0ne has joined #openstack-kolla | 17:57 | |
| *** Fl1nt has quit IRC | 17:59 | |
| *** k_mouza has quit IRC | 18:05 | |
| *** k_mouza has joined #openstack-kolla | 18:06 | |
| *** k_mouza has quit IRC | 18:10 | |
| *** hjensas__ is now known as hjensas | 18:22 | |
| *** LinPeiWen67 has quit IRC | 18:26 | |
| -openstackstatus- NOTICE: The Gerrit service at review.opendev.org is being restarted quickly as a pre-upgrade sanity check, estimated downtime is less than 5 minutes. | 18:36 | |
| *** e0ne has quit IRC | 18:36 | |
| *** rpittau is now known as rpittau|afk | 18:39 | |
| *** bsanjeewa has quit IRC | 18:40 | |
| *** gfidente is now known as gfidente|afk | 18:45 | |
| *** k_mouza has joined #openstack-kolla | 19:00 | |
| *** kevko has quit IRC | 19:01 | |
| *** openstackgerrit has quit IRC | 19:02 | |
| *** k_mouza has quit IRC | 19:04 | |
| *** kevko has joined #openstack-kolla | 19:25 | |
| *** sluna has quit IRC | 19:27 | |
| *** sluna has joined #openstack-kolla | 19:27 | |
| yoctozepto | btw, remember about gerrit downtime Fri-Sun | 19:36 |
| yoctozepto | (due to upgrade) | 19:36 |
| *** vishalmanchanda has quit IRC | 19:49 | |
| *** openstackgerrit has joined #openstack-kolla | 20:38 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix kolla-ansible to work with pyenv-virtualenv https://review.opendev.org/762365 | 20:38 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: fluentd: Remove conditions from handler https://review.opendev.org/763155 | 20:39 |
| *** ricolin has quit IRC | 20:40 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix kolla-ansible to work with pyenv-virtualenv https://review.opendev.org/762365 | 20:41 |
| *** mgoddard has joined #openstack-kolla | 20:50 | |
| *** TrevorV has quit IRC | 20:54 | |
| *** kevko has quit IRC | 21:43 | |
| *** mgoddard has quit IRC | 21:44 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Add support for GlusterFS NFS Manila backend https://review.opendev.org/746384 | 21:44 |
| *** kevko has joined #openstack-kolla | 22:19 | |
| *** vkmc has quit IRC | 22:26 | |
| *** vkmc has joined #openstack-kolla | 22:26 | |
| *** johnsom has quit IRC | 22:27 | |
| *** johnsom has joined #openstack-kolla | 22:29 | |
| *** yan0s has quit IRC | 22:45 | |
| *** Arador has quit IRC | 23:22 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!