| *** vishalmanchanda has quit IRC | 00:05 | |
| *** k_mouza has joined #openstack-kolla | 00:05 | |
| *** k_mouza has quit IRC | 00:10 | |
| *** brinzhang has joined #openstack-kolla | 00:22 | |
| *** sean-k-mooney1 has joined #openstack-kolla | 00:40 | |
| *** sean-k-mooney has quit IRC | 00:41 | |
| Aethylred | Resolved by "no space" issue | 00:51 |
|---|---|---|
| Aethylred | but now it fails on "Creating haproxy mysql user" https://github.com/openstack/kolla-ansible/blob/5e638b757bdda9fbddf0fe0be5d76caa3419af74/ansible/roles/mariadb/handlers/main.yml#L51 | 00:52 |
| Aethylred | which is a pain, because the check for this role _only_ happens during the mariadb bootstrap, and is not checked on repeating deploy | 00:52 |
| Aethylred | if fails with: FAILED! => {"changed": false, "msg": "kolla_toolbox container is not running."} | 00:58 |
| Aethylred | This is on a first deploy using Kolla-Ansible 11.0.0 on CentOS8 with ussari | 00:59 |
| Aethylred | repeating the deploy progresses to "[mariadb : wait for MariaDB to be available via HAProxy]" which then fails, presumably because the haproxy mysql user is missing. | 00:59 |
| Aethylred | So the only way to repeat the original HAproxy mysql user issue is to destroy it all. | 01:01 |
| Aethylred | In addition, this error occurs regardless of if enable_haproxy is true or false | 01:03 |
| Aethylred | recover_mariadb hangs at "[mariadb : wait for MariaDB to be available via HAProxy]" as well | 01:04 |
| *** imtiazc has joined #openstack-kolla | 01:31 | |
| *** imtiazc_ has joined #openstack-kolla | 01:53 | |
| *** imtiazc has left #openstack-kolla | 01:57 | |
| *** ivan_lin has quit IRC | 02:50 | |
| *** ivan_lin has joined #openstack-kolla | 02:51 | |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: Remove cinder v2 api https://review.opendev.org/c/openstack/kolla-ansible/+/779455 | 03:25 |
| openstackgerrit | Merged openstack/kolla-ansible master: Use Docker healthchecks for ironic services https://review.opendev.org/c/openstack/kolla-ansible/+/779078 | 03:42 |
| *** k_mouza has joined #openstack-kolla | 03:56 | |
| *** k_mouza has quit IRC | 04:00 | |
| *** Aethylred has quit IRC | 04:32 | |
| *** vishalmanchanda has joined #openstack-kolla | 04:59 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 05:15 |
| *** imtiazc has joined #openstack-kolla | 05:15 | |
| *** wuchunyang has joined #openstack-kolla | 05:50 | |
| *** shyamb has joined #openstack-kolla | 05:58 | |
| *** e0ne has joined #openstack-kolla | 06:31 | |
| *** e0ne has quit IRC | 06:31 | |
| *** wuchunyang has quit IRC | 06:59 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/victoria: Rename eswitchd.conf -> eswitchd.conf.j2 https://review.opendev.org/c/openstack/kolla-ansible/+/779462 | 07:00 |
| *** cah_link has joined #openstack-kolla | 07:10 | |
| *** LinPeiWen has quit IRC | 07:13 | |
| *** wuchunyang has joined #openstack-kolla | 07:15 | |
| *** e0ne has joined #openstack-kolla | 07:22 | |
| *** jbadiapa has joined #openstack-kolla | 07:23 | |
| *** e0ne has quit IRC | 07:27 | |
| *** LinPeiWen has joined #openstack-kolla | 07:32 | |
| *** Fl1nt has joined #openstack-kolla | 07:52 | |
| Fl1nt | Howdy team! | 07:52 |
| *** amoralej|off is now known as amoralej | 08:01 | |
| *** bengates has joined #openstack-kolla | 08:17 | |
| *** bengates has quit IRC | 08:18 | |
| *** bengates has joined #openstack-kolla | 08:19 | |
| *** brinzhang_ has joined #openstack-kolla | 08:20 | |
| *** brinzhang has quit IRC | 08:23 | |
| *** bengates has quit IRC | 08:24 | |
| *** bengates has joined #openstack-kolla | 08:25 | |
| *** rpittau|afk is now known as rpittau | 08:27 | |
| parallax | Morning | 08:32 |
| *** gfidente|afk is now known as gfidente | 08:53 | |
| openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible stable/ussuri: Synchronise kolla-ansible installation with other branches https://review.opendev.org/c/openstack/kolla-ansible/+/779510 | 08:54 |
| *** andrewbonney has joined #openstack-kolla | 09:03 | |
| mgoddard | Morning | 09:22 |
| *** shyamb has quit IRC | 09:22 | |
| *** LinPeiWen has quit IRC | 09:26 | |
| mnasiadka | morning | 09:30 |
| wuchunyang | morning | 09:32 |
| *** shyamb has joined #openstack-kolla | 09:34 | |
| oklhost | morning | 09:37 |
| mnasiadka | mgoddard: I see tripleo is marching towards centos stream - should we start doing something in that direction? | 09:52 |
| *** wuchunyang has quit IRC | 09:56 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Use Docker healthchecks for octavia services https://review.opendev.org/c/openstack/kolla-ansible/+/778180 | 10:05 |
| openstackgerrit | Merged openstack/kolla-ansible master: Use Docker healthchecks for trove services https://review.opendev.org/c/openstack/kolla-ansible/+/778170 | 10:05 |
| dardelean | Hello. I am planning to update a kolla train env to the latest train images. I am planning to rerun deploy but with the latest images tag. For stability and uptime reasons, I would like to use "--limit" to update the controllers one by one, I have 3, and then the computes together. Is this ok? Would the DB be impacted in a negative way? Or if you have any other recomandations. Thanks. | 10:07 |
| mgoddard | mnasiadka: hrw already started | 10:11 |
| hrw | elo | 10:12 |
| hrw | mnasiadka: tell docker hub to hold with pull limits and we may even pass zuul with this patch | 10:12 |
| hrw | mnasiadka: https://review.opendev.org/c/openstack/kolla/+/772841 | 10:13 |
| hrw | added reno | 10:15 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: switch to CentOS 8 Stream https://review.opendev.org/c/openstack/kolla/+/772841 | 10:15 |
| mnasiadka | hrw: ok then, we are switching - RDO will only now support Stream? | 10:22 |
| mnasiadka | just thinking about people that don't want to go Stream for next half a year or something | 10:23 |
| *** shyamb has quit IRC | 10:25 | |
| hrw | mnasiadka: they are free to not go stream on host | 10:28 |
| mnasiadka | and that won't cause any problems? | 10:29 |
| hrw | why would it? | 10:29 |
| hrw | did we required to have 8.3 on host with 8.3 containers? | 10:30 |
| mnasiadka | well, not really, just thinking out loud if it caused any problems for anybody | 10:30 |
| mnasiadka | but I guess not | 10:30 |
| *** e0ne has joined #openstack-kolla | 10:35 | |
| *** shyamb has joined #openstack-kolla | 10:40 | |
| *** LinPeiWen has joined #openstack-kolla | 10:44 | |
| *** k_mouza has joined #openstack-kolla | 10:49 | |
| *** eliaswimmer has joined #openstack-kolla | 10:51 | |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: CI: Use cached cirros image for seed vm job https://review.opendev.org/c/openstack/kayobe/+/773163 | 11:04 |
| *** shyam89 has joined #openstack-kolla | 11:09 | |
| *** shyamb has quit IRC | 11:11 | |
| *** k_mouza has quit IRC | 11:15 | |
| *** k_mouza has joined #openstack-kolla | 11:15 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/ussuri: Synchronise kolla-ansible installation with other branches https://review.opendev.org/c/openstack/kolla-ansible/+/779510 | 11:33 |
| *** shyam89 has quit IRC | 11:52 | |
| *** shyamb has joined #openstack-kolla | 11:53 | |
| hrw | wow! my stream change managed to pass zuul | 11:58 |
| *** kevko has joined #openstack-kolla | 12:06 | |
| *** shyamb has quit IRC | 12:07 | |
| *** sean-k-mooney1 is now known as sean-k-mooney | 12:16 | |
| *** Mareo has joined #openstack-kolla | 12:20 | |
| kevko | mgoddard: Hi, i just replied to all comments for mariadb refactor ... If you have some time ..we can discuss quickly .. | 12:44 |
| openstackgerrit | Merged openstack/kayobe master: Add support for custom shm_size for seed container(s) https://review.opendev.org/c/openstack/kayobe/+/777698 | 12:44 |
| *** wuchunyang has joined #openstack-kolla | 12:56 | |
| openstackgerrit | Maksim Malchuk proposed openstack/kolla-ansible master: Correctly configure S3 Token Middleware for Swift https://review.opendev.org/c/openstack/kolla-ansible/+/774791 | 13:03 |
| *** amoralej is now known as amoralej|lunch | 13:04 | |
| mgoddard | kevko: thanks, just reading | 13:16 |
| kevko | I didn't correct my english grammar ..and I wrote fast ..so ..sorry, If you will have some questions ..ask please .. | 13:23 |
| *** brinzhang_ has quit IRC | 13:23 | |
| *** brinzhang_ has joined #openstack-kolla | 13:24 | |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: Fix wrong configurations https://review.opendev.org/c/openstack/kolla-ansible/+/779764 | 13:31 |
| *** skramaja has quit IRC | 13:32 | |
| openstackgerrit | Merged openstack/kayobe stable/ussuri: Fix IPA build on stable/ussuri https://review.opendev.org/c/openstack/kayobe/+/775936 | 13:34 |
| openstackgerrit | Merged openstack/kayobe stable/train: Fix building CentOS 8 IPA images on stable/train https://review.opendev.org/c/openstack/kayobe/+/773944 | 13:34 |
| openstackgerrit | Merged openstack/kayobe stable/train: Test building seed deployment images in the seed job (CentOS 8 only) https://review.opendev.org/c/openstack/kayobe/+/773853 | 13:36 |
| mgoddard | kevko: my confusion about the haproxy user is because you have mariadb_monitor_user/password variables defined but not really used in the patch | 13:36 |
| mgoddard | kevko: replied | 13:43 |
| openstackgerrit | Merged openstack/kayobe stable/train: Fix CentOS 7 IPA build and test in CI https://review.opendev.org/c/openstack/kayobe/+/776595 | 13:46 |
| kevko | mgoddard: agree that *maybe* i have this variables from other patch in chain .. i can rebase .. | 13:47 |
| mgoddard | kevko: maybe yes? https://review.opendev.org/c/openstack/kolla-ansible/+/766952/40/ansible/group_vars/all.yml :) | 13:48 |
| kevko | mgoddard: will check and replace hardcoded user with variable | 13:50 |
| mgoddard | ok | 13:50 |
| kevko | mgoddard: now i am doing arbitrator .. when i will have it completed ..will add to chain .. | 13:51 |
| mgoddard | kevko: I don't mind if the variable is used or not in the patch, just needs to be consistent | 13:51 |
| kevko | mgoddard: and apply comments | 13:51 |
| kevko | agree | 13:51 |
| *** eliaswimmer has quit IRC | 13:57 | |
| *** amoralej|lunch is now known as amoralej | 13:59 | |
| openstackgerrit | Merged openstack/kayobe master: CI: display VM console log on ping or SSH failure https://review.opendev.org/c/openstack/kayobe/+/777372 | 14:08 |
| openstackgerrit | Merged openstack/kayobe master: Bump Ansible version range to 2.9 - 2.9 https://review.opendev.org/c/openstack/kayobe/+/754054 | 14:09 |
| *** jrollen is now known as jroll | 14:13 | |
| kevko | mgoddard: if i summarize comments .. it's not in bad shape, right ? It is just ..cosmetic things... | 14:21 |
| kevko | mostl | 14:22 |
| kevko | *mostly | 14:22 |
| mgoddard | kevko: yes, I think it's mostly there | 14:23 |
| mgoddard | Meeting in 7 minutes | 14:53 |
| mgoddard | ^ mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt | 14:53 |
| hrw | aaaaahhh! | 14:54 |
| hrw | run! run! | 14:54 |
| hrw | ;P | 14:54 |
| * yoctozepto is hiding | 14:55 | |
| mgoddard | no running or hiding please | 14:55 |
| mgoddard | I will come and find you if I have to | 14:56 |
| hrw | and then we go for a beer/favourite drink/ | 14:56 |
| yoctozepto | is blood a type of drink? | 15:00 |
| mgoddard | #startmeeting kolla | 15:00 |
| openstack | Meeting started Wed Mar 10 15:00:49 2021 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: kolla)" | 15:00 | |
| openstack | The meeting name has been set to 'kolla' | 15:00 |
| mgoddard | #topic rollcall | 15:00 |
| *** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:01 | |
| yoctozepto | \o | 15:01 |
| mgoddard | \-------o | 15:01 |
| yoctozepto | o_ | 15:01 |
| priteau | \o/ | 15:02 |
| hrw | -°- | 15:02 |
| hrw | yoctozepto: only while it is fresh. | 15:03 |
| mgoddard | #topic agenda | 15:03 |
| *** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:03 | |
| mgoddard | * Roll-call | 15:03 |
| mgoddard | * Announcements | 15:03 |
| mgoddard | ** PTG 19th - 23rd April, registration open | https://april2021-ptg.eventbrite.com | https://www.openstack.org/ptg/ | 15:03 |
| mgoddard | ** OpenStack feature freeze this week | 15:03 |
| mgoddard | * Review action items from the last meeting | 15:03 |
| mgoddard | * CI status | 15:03 |
| mgoddard | * Review requests | 15:03 |
| mgoddard | * PTG team signup http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html | 15:03 |
| mgoddard | * Glance OSSN-0088 http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020947.html | 15:03 |
| mgoddard | * PoC: image build & test pipeline (https://review.opendev.org/c/openstack/kolla/+/777796 and https://review.opendev.org/c/openstack/kolla-ansible/+/777946) | 15:03 |
| mgoddard | * Wallaby release planning | 15:03 |
| mgoddard | #topic announcements | 15:04 |
| *** openstack changes topic to "announcements (Meeting topic: kolla)" | 15:04 | |
| Fl1nt | o/ | 15:04 |
| mgoddard | #info Project Teams Gathering (PTG) registration open | 15:04 |
| mgoddard | #link https://april2021-ptg.eventbrite.com | 15:04 |
| mgoddard | #link https://www.openstack.org/ptg/ | 15:04 |
| mgoddard | It will run from 19th-23rd April | 15:05 |
| mgoddard | We can discuss it more later | 15:05 |
| *** zzzeek has quit IRC | 15:05 | |
| mgoddard | #info OpenStack feature freeze this week | 15:05 |
| mgoddard | Which can only mean we are not far from Kolla feature freeze, so let's all get reviewing | 15:06 |
| yoctozepto | yes, this is super painful | 15:06 |
| yoctozepto | and sad | 15:06 |
| *** zzzeek has joined #openstack-kolla | 15:06 | |
| mgoddard | Any other announcements? | 15:06 |
| mgoddard | #topic Review action items from the last meeting | 15:07 |
| *** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)" | 15:07 | |
| mgoddard | wuchunyang to propose toscaparser in tacker requirements to fix NFV job | 15:08 |
| mgoddard | mgoddard to write up options for CI registry | 15:08 |
| mgoddard | I think the tacker issue turned out to be more complicated | 15:08 |
| wuchunyang | tacker guys will fix this . | 15:08 |
| mgoddard | #link https://bugs.launchpad.net/bugs/1918339 | 15:08 |
| openstack | Launchpad bug 1918339 in kolla "kolla-ansible-centos8-source-scenario-nfv Zuul test fails due to import error in tacker " [Undecided,New] | 15:08 |
| mgoddard | I completed my action, we can discuss later | 15:09 |
| mgoddard | #topic CI status | 15:09 |
| *** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:09 | |
| *** baha has joined #openstack-kolla | 15:09 | |
| mgoddard | observed on ussuri, victoria and master at least | 15:09 |
| mgoddard | centos8-ceph-upgrade jobs seem to be retried 3 times only to fail in some weird way | 15:09 |
| mgoddard | similarly other multinode centos8 jobs (like cells) fail: 'Ensuring config directories exist' | 15:09 |
| mgoddard | yoctozepto suspects centos8.3 | 15:09 |
| mgoddard | Do we still see that, or should we scrub from the whiteboard? | 15:10 |
| yoctozepto | I've seen that recently yes | 15:10 |
| yoctozepto | but it's not that often | 15:10 |
| mgoddard | ok | 15:10 |
| yoctozepto | we can scrub it and remember it has happened previously | 15:10 |
| yoctozepto | could be some cloud unhappy with centos | 15:11 |
| yoctozepto | we know what the DISK_FULL are | 15:11 |
| yoctozepto | it was due to swift spaces | 15:11 |
| yoctozepto | for logs | 15:11 |
| yoctozepto | but it was not close to 6.3GB | 15:11 |
| yoctozepto | more like 630MB | 15:12 |
| yoctozepto | still dumb to save and transfer | 15:12 |
| yoctozepto | the responsible was libvirt debug | 15:12 |
| yoctozepto | going to review now | 15:12 |
| yoctozepto | mgoddard: https://review.opendev.org/c/openstack/kolla-ansible/+/779251 | 15:13 |
| yoctozepto | if you agree, I will just edit this | 15:13 |
| yoctozepto | (or you can, whatever) | 15:14 |
| mgoddard | yoctozepto: agree | 15:14 |
| yoctozepto | ok | 15:15 |
| yoctozepto | editing | 15:15 |
| mgoddard | cool | 15:15 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Introduce nova_libvirt_logging_debug https://review.opendev.org/c/openstack/kolla-ansible/+/779251 | 15:15 |
| yoctozepto | mgoddard: approve ^ | 15:16 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Reduce number of logs and disable ara HTML report https://review.opendev.org/c/openstack/kolla-ansible/+/777647 | 15:16 |
| mgoddard | yessir | 15:16 |
| yoctozepto | :D | 15:17 |
| mgoddard | ok, I think we're good for CI | 15:18 |
| yoctozepto | agreed | 15:18 |
| mgoddard | I did some 'tidying' of the whiteboard, it had developed some preamble | 15:18 |
| mgoddard | (the CI section) | 15:18 |
| mgoddard | #topic Review requests | 15:19 |
| *** openstack changes topic to "Review requests (Meeting topic: kolla)" | 15:19 | |
| openstackgerrit | Merged openstack/kayobe stable/ussuri: Test building seed deployment images in the seed job https://review.opendev.org/c/openstack/kayobe/+/774055 | 15:19 |
| mgoddard | Hit me with your code review requests (1 per person) | 15:19 |
| openstackgerrit | Merged openstack/kayobe master: Wait for overcloud manage and provide state transitions https://review.opendev.org/c/openstack/kayobe/+/775135 | 15:19 |
| openstackgerrit | Merged openstack/kayobe master: CI: Use cached cirros image for seed vm job https://review.opendev.org/c/openstack/kayobe/+/773163 | 15:19 |
| priteau | CI in Kayobe is better (pull rate limit issues seem gone?) but not perfect (seeing occasional HTTP 520 from Galaxy) | 15:19 |
| yoctozepto | I am pushing things with masakari atm so nothing specific from me | 15:19 |
| priteau | (sorry, missed the topic change) | 15:19 |
| * yoctozepto slaps mgoddard around a bit with a large trout | 15:20 | |
| mgoddard | priteau: +1, although I think pull limits are just luck, I have seen them | 15:20 |
| mgoddard | yoctozepto: that is not a review request | 15:20 |
| yoctozepto | mgoddard: y, looks valid :D | 15:21 |
| mgoddard | Nobody wants a review this week? | 15:23 |
| kevko | do you think we will process mariadb,proxysql ... this week ? | 15:24 |
| hrw | stream passed zuul so I assume that it will see revuiews | 15:24 |
| mgoddard | kevko: probably not the whole patch chain | 15:24 |
| mgoddard | kevko: but one can hope | 15:24 |
| kevko | so, wallaby will be withou :/ ? | 15:24 |
| yoctozepto | we are not freezing yet kevko | 15:25 |
| kevko | but it is behind the door | 15:25 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/ussuri: Update IPA docs and test build with extra-hardware https://review.opendev.org/c/openstack/kayobe/+/779811 | 15:25 |
| yoctozepto | worry not | 15:25 |
| mgoddard | I would suggest that we aim to merge 2 of the proxysql patches per week | 15:26 |
| mgoddard | hopefully that should get us there by feature freeze | 15:26 |
| yoctozepto | starting from the bottom | 15:26 |
| mgoddard | yes | 15:26 |
| mgoddard | mariadb sharding patch looks close | 15:26 |
| mgoddard | https://review.opendev.org/c/openstack/kolla-ansible/+/766952 | 15:27 |
| kevko | in few hours i will get into and check again | 15:27 |
| mgoddard | the next one is https://review.opendev.org/c/openstack/kolla-ansible/+/770618/18 | 15:27 |
| kevko | btw, i have some small patches i want to review ..let me check it | 15:28 |
| kevko | https://review.opendev.org/c/openstack/kolla-ansible/+/775627 << | 15:28 |
| headphoneJames | Working on the let's encrypt functional test, getting closer | 15:29 |
| mgoddard | sounds good headphoneJames | 15:30 |
| mgoddard | Let's move on | 15:30 |
| mgoddard | #topic PTG team signup | 15:30 |
| *** openstack changes topic to "PTG team signup (Meeting topic: kolla)" | 15:30 | |
| mgoddard | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html | 15:30 |
| mgoddard | We need to book some time for the PTG | 15:31 |
| mgoddard | Our 'usual' approach is 4 hours on Monday, 4 hours on Tuesday, and 2 hours on Wednesday for Kayobe | 15:31 |
| mgoddard | We usually take the 13:00+ UTC slot | 15:32 |
| yoctozepto | I won't be able to dedicate that much time this time | 15:32 |
| yoctozepto | perhaps we could do 2 hours per deliverable | 15:32 |
| mgoddard | I was thinking, should we try a slot in the 4:00 - 8:00 UTC window? | 15:33 |
| hrw | any fits me | 15:34 |
| mgoddard | Not because I like waking up early, but it might attract some people in Asian timezones | 15:34 |
| mgoddard | wuchunyang: around? | 15:34 |
| priteau | Keep in mind that clocks will have changed by then | 15:35 |
| mgoddard | good point | 15:36 |
| priteau | So 4-8 UTC will be 5-9 UK time and 6-10 CEST | 15:36 |
| yoctozepto | yes | 15:37 |
| mgoddard | so how do we decide? Should I send an email to openstack-discuss to gauge interest? | 15:37 |
| yoctozepto | ++ | 15:37 |
| mgoddard | #action mgoddard send an email to openstack-discuss to gauge interest in the 'early' PTG slot | 15:37 |
| mgoddard | #topic Glance OSSN-0088 | 15:38 |
| *** openstack changes topic to "Glance OSSN-0088 (Meeting topic: kolla)" | 15:38 | |
| mgoddard | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020947.html | 15:38 |
| mgoddard | who dis? | 15:39 |
| mgoddard | Is there anything for us to do here? | 15:40 |
| priteau | Ship secure policies by default? | 15:40 |
| mgoddard | I would probably prefer glance to do that | 15:41 |
| wuchunyang | hi | 15:41 |
| yoctozepto | it was me | 15:41 |
| yoctozepto | yes, I agree | 15:41 |
| yoctozepto | but it was thrown at the deployment projects | 15:41 |
| mgoddard | hi wuchunyang | 15:41 |
| yoctozepto | how should we approach this issue? | 15:41 |
| yoctozepto | hi wuchunyang | 15:41 |
| wuchunyang | which issue ? i was out just now | 15:42 |
| hrw | wuchunyang: PTG in Asian time | 15:43 |
| mgoddard | wuchunyang: I think I pinged you earlier about the PTG. I'll send an email to openstack-discuss, you can reply there | 15:43 |
| yoctozepto | wuchunyang: the issue question was not targeted at you ;-) | 15:43 |
| wuchunyang | ok.. i will read the mail.. | 15:44 |
| mgoddard | So far we have avoided modifying default policies | 15:44 |
| mgoddard | The email is targeted at operators rather than deployment tools | 15:44 |
| priteau | Maybe this is a topic to explore more generally | 15:45 |
| mgoddard | Should we make an exception for this specific issue? | 15:45 |
| priteau | Hardened Kolla-Ansible | 15:45 |
| yoctozepto | well, it looks crappy to let it have this hole by default | 15:45 |
| Fl1nt | I suggest we don't change the default policy as if it really is a security issue upstream services will do it, and if not it means that an optional part of the service and should be let up to the operators | 15:45 |
| priteau | Could be a global variable that would change some settings to be more secure, with the downside of breaking some features | 15:46 |
| yoctozepto | all right, we have got the first topic for ptg | 15:46 |
| priteau | yoctozepto: It's a hole that some deployments may rely on | 15:46 |
| Fl1nt | we shouldn't, it will have impact on upgrades/updates as some side effects will arise. | 15:46 |
| yoctozepto | priteau: yes, unfortunately | 15:46 |
| yoctozepto | yeah, stable/upgrades is something I would prefer not to touch indeed | 15:47 |
| mgoddard | I think if it were as simple as just changing the policy, glance would do it | 15:47 |
| mgoddard | they are just as capable as us of making the change | 15:47 |
| yoctozepto | yes, it would have to be behind a flag like priteau said | 15:47 |
| Fl1nt | but are they calling metadef APIs ? | 15:48 |
| Fl1nt | ^what are | 15:48 |
| mgoddard | I suppose I'm not against having it behind a flag | 15:48 |
| yoctozepto | Fl1nt: https://docs.openstack.org/api-ref/image/v2/metadefs-index.html | 15:49 |
| Fl1nt | yeah something like INVALID_GLANCE_OSSN0088 | 15:49 |
| mgoddard | What I didn't mention earlier in the PTG section is that I created an etherpad | 15:49 |
| * hrw out | 15:49 | |
| mgoddard | #link https://etherpad.opendev.org/p/kolla-xena-ptg | 15:49 |
| Fl1nt | aaaaah metadefinitions, got it | 15:49 |
| mgoddard | does someone want to add this topic? | 15:49 |
| priteau | Fl1nt: https://docs.openstack.org/glance/latest/#metadata-definitions | 15:49 |
| yoctozepto | Fl1nt: yeas | 15:50 |
| yoctozepto | someone added it | 15:50 |
| yoctozepto | (not me) | 15:50 |
| mgoddard | me | 15:50 |
| Fl1nt | what I don't understand, is that from the mailing list responses, it seems to only be a default policy shape issue on glance metadef api, so glance should just add the usual is_admin filter, but did they responded to the mail? | 15:52 |
| *** dmsimard8 has joined #openstack-kolla | 15:52 | |
| mgoddard | I don't see any responses | 15:52 |
| mgoddard | We're running out of time | 15:53 |
| mgoddard | Let's discuss at the PTG | 15:53 |
| Fl1nt | yep | 15:53 |
| mgoddard | #topic PoC: image build & test pipeline | 15:53 |
| *** openstack changes topic to "PoC: image build & test pipeline (Meeting topic: kolla)" | 15:53 | |
| mgoddard | #link https://etherpad.opendev.org/p/docker-pull-limits | 15:53 |
| mgoddard | I wrote up some options in there | 15:53 |
| mgoddard | See 3a. and 3b. | 15:53 |
| *** dmsimard has quit IRC | 15:54 | |
| *** dmsimard8 is now known as dmsimard | 15:54 | |
| mgoddard | I think we don't have time to discuss this topic properly today | 15:55 |
| mgoddard | But please read the etherpad, and we can discuss next week | 15:55 |
| mgoddard | #topic Open discussion | 15:56 |
| *** openstack changes topic to "Open discussion (Meeting topic: kolla)" | 15:56 | |
| priteau | I would like to know what is our plan for Docker iptables management in Xena | 15:57 |
| priteau | It was announced to be disabled in Victoria I think? | 15:57 |
| Fl1nt | woot? | 15:57 |
| priteau | But it wasn't | 15:57 |
| priteau | https://review.opendev.org/c/openstack/kolla-ansible/+/751795 | 15:57 |
| Fl1nt | ah you're talking about the infamous ebtable thingy ? | 15:57 |
| priteau | and https://review.opendev.org/c/openstack/kolla-ansible/+/689870 | 15:58 |
| priteau | No, I am talking about Docker manipulating iptables rules on its own | 15:58 |
| yoctozepto | yeah, we are late | 15:58 |
| priteau | Particularly, changing the default FORWARD chain policy | 15:58 |
| priteau | Which can cause issues in some environments | 15:58 |
| Fl1nt | Don't use use host networking model ? | 15:59 |
| Fl1nt | s/use/you/ | 15:59 |
| priteau | Of course, but it still has an impact | 15:59 |
| * yoctozepto went away | 16:00 | |
| *** LinPeiWen has quit IRC | 16:00 | |
| mgoddard | Good question priteau | 16:00 |
| mgoddard | "As Michal pointed out, this breaks kolla-build unless you set network_mode = host in kolla-build.conf." | 16:00 |
| Fl1nt | yes, I do on my own. | 16:01 |
| mgoddard | I think I had problems when I tested it here: https://review.opendev.org/c/openstack/kolla-ansible/+/751982 | 16:01 |
| mgoddard | I can try it again | 16:02 |
| mgoddard | we do now have network_mode = host in kolla-build.conf | 16:03 |
| mgoddard | anyway, time's up | 16:03 |
| mgoddard | Thanks all | 16:03 |
| mgoddard | #endmeeting | 16:03 |
| *** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:03 | |
| openstack | Meeting ended Wed Mar 10 16:03:21 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:03 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-10-15.00.html | 16:03 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-10-15.00.txt | 16:03 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-10-15.00.log.html | 16:03 |
| wuchunyang | I found a new bug that ansible os_nova_flavor module does not satisfy idempotency,it will recreate the flavor when swap is set to 0. this cause the octavia containers to restart | 16:03 |
| Fl1nt | got to go, see ya folks! | 16:04 |
| wuchunyang | ansible try to query the flavor , when swap is set to 0, nova will return empty string instead. | 16:04 |
| mgoddard | wuchunyang: what if we pass an empty string? | 16:05 |
| wuchunyang | the filed must int. | 16:05 |
| mgoddard | wuchunyang: it probably needs to be fixed in the collection | 16:05 |
| wuchunyang | yes, should we add a workaround now ? | 16:06 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: [docker] Added a new flag to disable default network https://review.opendev.org/c/openstack/kolla-ansible/+/689870 | 16:07 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Disable Docker iptables and bridge networking by default https://review.opendev.org/c/openstack/kolla-ansible/+/751795 | 16:07 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: test kolla build without iptables or bridge https://review.opendev.org/c/openstack/kolla-ansible/+/751982 | 16:07 |
| openstackgerrit | Merged openstack/kayobe master: CI: Add Ubuntu seed job https://review.opendev.org/c/openstack/kayobe/+/775267 | 16:08 |
| *** wuchunyang has quit IRC | 16:08 | |
| mgoddard | wuchunyang: swap: "{{ octavia_amp_flavor.swap | default(omit, true) }}" | 16:09 |
| *** Fl1nt has quit IRC | 16:09 | |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: Use Docker healthchecks for zun services https://review.opendev.org/c/openstack/kolla-ansible/+/779826 | 16:12 |
| openstackgerrit | Merged openstack/kolla-ansible master: Correctly configure S3 Token Middleware for Swift https://review.opendev.org/c/openstack/kolla-ansible/+/774791 | 16:35 |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 16:46 |
| *** rpittau is now known as rpittau|afk | 17:05 | |
| yoctozepto | mgoddard, mnasiadka: cephadm failure https://zuul.opendev.org/t/openstack/build/5bed2d2292164ee7b6b7f006197ecbe9 | 17:05 |
| yoctozepto | and it looks similar to the one you mentioned today from the whiteboard, mgoddard | 17:06 |
| yoctozepto | but here on ubuntu | 17:06 |
| openstackgerrit | Michal Arbet proposed openstack/kolla-ansible master: Set changed_when to false for group_by tasks https://review.opendev.org/c/openstack/kolla-ansible/+/779836 | 17:08 |
| openstackgerrit | Merged openstack/kayobe stable/train: CI: use overlay2 storage driver for overcloud deploy https://review.opendev.org/c/openstack/kayobe/+/777373 | 17:24 |
| ozzzo | I built a train test cluster on a heat stack. When I try to build a VM it fails with "No valid host was found" | 17:24 |
| ozzzo | so I looked at my hypervisors, and they are using more memory than they have: | 17:24 |
| ozzzo | | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.230 | up | 2 | 8 | 32768 | 30719 | | 17:24 |
| ozzzo | what is using 32G of RAM on an empty HV? | 17:25 |
| kevko | ps faux ? :D | 17:28 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/train: Update IPA docs and test build with extra-hardware https://review.opendev.org/c/openstack/kayobe/+/779839 | 17:28 |
| ozzzo | it's not actually used on the compute node; just in the hypervisor list | 17:29 |
| ozzzo | I have 22G free | 17:30 |
| ozzzo | Mem: 30157 1412 22899 17 5845 28323 | 17:30 |
| *** amoralej is now known as amoralej|off | 17:36 | |
| kevko | well, it's reporting your node available resources, isn't it ? | 17:36 |
| ozzzo | I think the hypervisor list is pulling from placement | 17:36 |
| ozzzo | I'm not exactly sure how that works | 17:36 |
| ozzzo | but somehow openstack thinks I've used up 32G out of 30G available, before building any VMs | 17:37 |
| kevko | what is the command ? | 17:38 |
| kevko | hypervisor list << if i run this ..i don't have info about ram | 17:39 |
| ozzzo | os hypervisor list --long | 17:41 |
| kevko | weird | 17:42 |
| ozzzo | Isn't there some kind of memory reservation setting in nova.conf? I wonder if I have that set to 32G | 17:47 |
| ozzzo | ahah that's it: reserved_host_memory_mb | 17:48 |
| ozzzo | what's a reasonable value for a compute on a VM, maybe 2G? | 17:48 |
| kevko | don't understand | 17:49 |
| ozzzo | in nova.conf, I have reserved_host_memory_mb = 32768 | 17:49 |
| ozzzo | reserving 32G for the hypervisor os | 17:49 |
| ozzzo | which is reasonable in prod where the boxes have tons of memory | 17:50 |
| ozzzo | but on a 30G VM, not so much | 17:50 |
| yoctozepto | ozzzo: 1G is enough if you run a very basic config | 17:50 |
| kevko | ah, you are asking how much you should decrease reserved host memory ? | 17:50 |
| ozzzo | righton, ty! | 17:50 |
| ozzzo | trying 1G | 17:50 |
| yoctozepto | you might also want to configure the nodes in placement regarding the overcommit | 17:50 |
| yoctozepto | to avoid surpises :D | 17:50 |
| ozzzo | I'm not sure I follow | 17:51 |
| yoctozepto | by default there is certain overcommit allowed | 17:51 |
| yoctozepto | so that you can run vms with more memory than you have on the host | 17:52 |
| yoctozepto | this part is configurable via the placement client | 17:52 |
| ozzzo | with resource provider set? | 17:53 |
| yoctozepto | yes | 17:56 |
| ozzzo | ok I'll experiment with that, ty | 17:56 |
| *** bengates has quit IRC | 17:56 | |
| *** bengates has joined #openstack-kolla | 17:57 | |
| *** bengates has quit IRC | 17:57 | |
| *** bengates has joined #openstack-kolla | 17:58 | |
| *** bengates has quit IRC | 18:03 | |
| *** jonaspaulo has joined #openstack-kolla | 18:10 | |
| *** dcapone2004 has joined #openstack-kolla | 18:12 | |
| *** gfidente is now known as gfidente|afk | 18:16 | |
| *** andrewbonney has quit IRC | 18:19 | |
| *** tbarron has joined #openstack-kolla | 18:37 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 18:43 |
| *** jonaspaulo has quit IRC | 18:43 | |
| *** e0ne has quit IRC | 18:48 | |
| *** zul has quit IRC | 18:54 | |
| *** cah_link has quit IRC | 18:55 | |
| *** cah_link has joined #openstack-kolla | 19:12 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Introduce nova_libvirt_logging_debug https://review.opendev.org/c/openstack/kolla-ansible/+/779251 | 19:22 |
| openstackgerrit | Merged openstack/kolla-ansible master: Reduce number of logs and disable ara HTML report https://review.opendev.org/c/openstack/kolla-ansible/+/777647 | 19:23 |
| *** bengates has joined #openstack-kolla | 19:24 | |
| *** bengates has quit IRC | 19:30 | |
| *** bengates has joined #openstack-kolla | 19:31 | |
| *** brinzhang0 has joined #openstack-kolla | 19:33 | |
| *** iniazi has joined #openstack-kolla | 19:34 | |
| *** dciabrin__ has joined #openstack-kolla | 19:34 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/victoria: Introduce nova_libvirt_logging_debug https://review.opendev.org/c/openstack/kolla-ansible/+/779785 | 19:34 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/victoria: Reduce number of logs and disable ara HTML report https://review.opendev.org/c/openstack/kolla-ansible/+/779786 | 19:34 |
| *** dcapone2004 has quit IRC | 19:34 | |
| *** also_stingrayza has joined #openstack-kolla | 19:34 | |
| *** bengates has quit IRC | 19:35 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Introduce nova_libvirt_logging_debug https://review.opendev.org/c/openstack/kolla-ansible/+/779787 | 19:36 |
| *** benj_- has joined #openstack-kolla | 19:37 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Reduce number of logs and disable ara HTML report https://review.opendev.org/c/openstack/kolla-ansible/+/779788 | 19:37 |
| *** dosaboy_ has joined #openstack-kolla | 19:38 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Reduce number of logs and disable ara HTML report https://review.opendev.org/c/openstack/kolla-ansible/+/779788 | 19:39 |
| *** dmsimard has quit IRC | 19:42 | |
| *** brinzhang_ has quit IRC | 19:42 | |
| *** mjturek has quit IRC | 19:42 | |
| *** baha has quit IRC | 19:42 | |
| *** iniazi_ has quit IRC | 19:42 | |
| *** benj_ has quit IRC | 19:42 | |
| *** oyrogerg has quit IRC | 19:42 | |
| *** dosaboy has quit IRC | 19:42 | |
| *** hjensas has quit IRC | 19:42 | |
| *** dciabrin_ has quit IRC | 19:42 | |
| *** Underknowledge has quit IRC | 19:42 | |
| *** stingrayza has quit IRC | 19:42 | |
| *** dmsimard has joined #openstack-kolla | 19:43 | |
| *** Underknowledge has joined #openstack-kolla | 19:43 | |
| *** irclogbot_3 has quit IRC | 19:44 | |
| *** hjensas has joined #openstack-kolla | 19:48 | |
| *** ysirndjuro has joined #openstack-kolla | 19:49 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Reduce number of logs and disable ara HTML report https://review.opendev.org/c/openstack/kolla-ansible/+/779788 | 20:04 |
| *** bsanjeewa has joined #openstack-kolla | 20:09 | |
| *** bsanjeewa has quit IRC | 20:10 | |
| *** eliaswimmer has joined #openstack-kolla | 20:11 | |
| *** bsanjeewa has joined #openstack-kolla | 20:13 | |
| eliaswimmer | It's only open for one day, but just in case someone is looking for a job building a new multi-datacenter openstack cluster for our university in Vienna, please apply: https://jobs.tuwien.ac.at/Job/145680 (only CV needed) | 20:14 |
| mnasiadka | yoctozepto: that’s probably some cephadm bug, will look into that tomorrow. | 20:27 |
| *** gfidente|afk is now known as gfidente | 20:30 | |
| *** Aethylred has joined #openstack-kolla | 20:38 | |
| *** rouk has joined #openstack-kolla | 20:44 | |
| rouk | how do you make migration_interface_address not brick cold migrations? | 20:45 |
| rouk | live migrations work right, but cold goes back to the api address, which isnt listening anymore with nova_ssh | 20:46 |
| *** gfidente is now known as gfidente|afk | 20:50 | |
| *** gfidente|afk has quit IRC | 20:57 | |
| *** k_mouza has quit IRC | 20:57 | |
| *** rouk has quit IRC | 21:27 | |
| openstackgerrit | Maksim Malchuk proposed openstack/kolla-ansible stable/ussuri: Correctly configure S3 Token Middleware for Swift https://review.opendev.org/c/openstack/kolla-ansible/+/779870 | 21:41 |
| openstackgerrit | Maksim Malchuk proposed openstack/kolla-ansible stable/victoria: Correctly configure S3 Token Middleware for Swift https://review.opendev.org/c/openstack/kolla-ansible/+/779871 | 21:42 |
| *** johnsom has quit IRC | 21:46 | |
| *** johnsom has joined #openstack-kolla | 21:46 | |
| *** bsanjeewa has quit IRC | 22:04 | |
| *** bsanjeewa_ has joined #openstack-kolla | 22:05 | |
| *** vishalmanchanda has quit IRC | 22:08 | |
| *** eliaswimmer has quit IRC | 22:09 | |
| *** zzzeek has quit IRC | 22:16 | |
| *** zzzeek has joined #openstack-kolla | 22:18 | |
| headphoneJames | is a comment with "Depends-On:" enough to trigger zuul to build the kolla containers in the Depends-Ons to use in the current test run? | 22:33 |
| headphoneJames | I'm trying to determine if https://review.opendev.org/c/openstack/kolla-ansible/+/741340 is building https://review.opendev.org/#/c/741339 and using the new let's encrypt containers | 22:34 |
| *** k_mouza has joined #openstack-kolla | 22:42 | |
| *** cah_link has quit IRC | 22:43 | |
| *** k_mouza has quit IRC | 22:43 | |
| *** k_mouza has joined #openstack-kolla | 22:43 | |
| *** k_mouza has quit IRC | 22:47 | |
| *** k_mouza has joined #openstack-kolla | 22:53 | |
| *** k_mouza has quit IRC | 22:57 | |
| *** ysirndjuro has left #openstack-kolla | 23:20 | |
| *** k_mouza has joined #openstack-kolla | 23:28 | |
| *** rpittau|afk has quit IRC | 23:37 | |
| *** rpittau|afk has joined #openstack-kolla | 23:37 | |
| *** k_mouza has quit IRC | 23:55 | |
| *** k_mouza has joined #openstack-kolla | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!