Wednesday, 2021-03-10

*** vishalmanchanda has quit IRC00:05
*** k_mouza has joined #openstack-kolla00:05
*** k_mouza has quit IRC00:10
*** brinzhang has joined #openstack-kolla00:22
*** sean-k-mooney1 has joined #openstack-kolla00:40
*** sean-k-mooney has quit IRC00:41
AethylredResolved by "no space" issue00:51
Aethylredbut now it fails on "Creating haproxy mysql user" https://github.com/openstack/kolla-ansible/blob/5e638b757bdda9fbddf0fe0be5d76caa3419af74/ansible/roles/mariadb/handlers/main.yml#L5100:52
Aethylredwhich is a pain, because the check for this role _only_ happens during the mariadb bootstrap, and is not checked on repeating deploy00:52
Aethylredif fails with: FAILED! => {"changed": false, "msg": "kolla_toolbox container is not running."}00:58
AethylredThis is on a first deploy using Kolla-Ansible 11.0.0 on CentOS8 with ussari00:59
Aethylredrepeating the deploy progresses to "[mariadb : wait for MariaDB to be available via HAProxy]" which then fails, presumably because the haproxy mysql user is missing.00:59
AethylredSo the only way to repeat the original HAproxy mysql user issue is to destroy it all.01:01
AethylredIn addition, this error occurs regardless of if enable_haproxy is true or false01:03
Aethylredrecover_mariadb hangs at "[mariadb : wait for MariaDB to be available via HAProxy]" as well01:04
*** imtiazc has joined #openstack-kolla01:31
*** imtiazc_ has joined #openstack-kolla01:53
*** imtiazc has left #openstack-kolla01:57
*** ivan_lin has quit IRC02:50
*** ivan_lin has joined #openstack-kolla02:51
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: Remove cinder v2 api  https://review.opendev.org/c/openstack/kolla-ansible/+/77945503:25
openstackgerritMerged openstack/kolla-ansible master: Use Docker healthchecks for ironic services  https://review.opendev.org/c/openstack/kolla-ansible/+/77907803:42
*** k_mouza has joined #openstack-kolla03:56
*** k_mouza has quit IRC04:00
*** Aethylred has quit IRC04:32
*** vishalmanchanda has joined #openstack-kolla04:59
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134005:15
*** imtiazc has joined #openstack-kolla05:15
*** wuchunyang has joined #openstack-kolla05:50
*** shyamb has joined #openstack-kolla05:58
*** e0ne has joined #openstack-kolla06:31
*** e0ne has quit IRC06:31
*** wuchunyang has quit IRC06:59
openstackgerritMerged openstack/kolla-ansible stable/victoria: Rename eswitchd.conf -> eswitchd.conf.j2  https://review.opendev.org/c/openstack/kolla-ansible/+/77946207:00
*** cah_link has joined #openstack-kolla07:10
*** LinPeiWen has quit IRC07:13
*** wuchunyang has joined #openstack-kolla07:15
*** e0ne has joined #openstack-kolla07:22
*** jbadiapa has joined #openstack-kolla07:23
*** e0ne has quit IRC07:27
*** LinPeiWen has joined #openstack-kolla07:32
*** Fl1nt has joined #openstack-kolla07:52
Fl1ntHowdy team!07:52
*** amoralej|off is now known as amoralej08:01
*** bengates has joined #openstack-kolla08:17
*** bengates has quit IRC08:18
*** bengates has joined #openstack-kolla08:19
*** brinzhang_ has joined #openstack-kolla08:20
*** brinzhang has quit IRC08:23
*** bengates has quit IRC08:24
*** bengates has joined #openstack-kolla08:25
*** rpittau|afk is now known as rpittau08:27
parallaxMorning08:32
*** gfidente|afk is now known as gfidente08:53
openstackgerritPierre Riteau proposed openstack/kolla-ansible stable/ussuri: Synchronise kolla-ansible installation with other branches  https://review.opendev.org/c/openstack/kolla-ansible/+/77951008:54
*** andrewbonney has joined #openstack-kolla09:03
mgoddardMorning09:22
*** shyamb has quit IRC09:22
*** LinPeiWen has quit IRC09:26
mnasiadkamorning09:30
wuchunyangmorning09:32
*** shyamb has joined #openstack-kolla09:34
oklhostmorning09:37
mnasiadkamgoddard: I see tripleo is marching towards centos stream - should we start doing something in that direction?09:52
*** wuchunyang has quit IRC09:56
openstackgerritMerged openstack/kolla-ansible master: Use Docker healthchecks for octavia services  https://review.opendev.org/c/openstack/kolla-ansible/+/77818010:05
openstackgerritMerged openstack/kolla-ansible master: Use Docker healthchecks for trove services  https://review.opendev.org/c/openstack/kolla-ansible/+/77817010:05
dardeleanHello. I am planning to update a kolla train env to the latest train images. I am planning to rerun deploy but with the latest images tag. For stability and uptime reasons, I would like to use "--limit" to update the controllers one by one, I have 3, and then the computes together. Is this ok? Would the DB be impacted in a negative way? Or if you have any other recomandations. Thanks.10:07
mgoddardmnasiadka: hrw already started10:11
hrwelo10:12
hrwmnasiadka: tell docker hub to hold with pull limits and we may even pass zuul with this patch10:12
hrwmnasiadka: https://review.opendev.org/c/openstack/kolla/+/77284110:13
hrwadded reno10:15
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: switch to CentOS 8 Stream  https://review.opendev.org/c/openstack/kolla/+/77284110:15
mnasiadkahrw: ok then, we are switching - RDO will only now support Stream?10:22
mnasiadkajust thinking about people that don't want to go Stream for next half a year or something10:23
*** shyamb has quit IRC10:25
hrwmnasiadka: they are free to not go stream on host10:28
mnasiadkaand that won't cause any problems?10:29
hrwwhy would it?10:29
hrwdid we required to have 8.3 on host with 8.3 containers?10:30
mnasiadkawell, not really, just thinking out loud if it caused any problems for anybody10:30
mnasiadkabut I guess not10:30
*** e0ne has joined #openstack-kolla10:35
*** shyamb has joined #openstack-kolla10:40
*** LinPeiWen has joined #openstack-kolla10:44
*** k_mouza has joined #openstack-kolla10:49
*** eliaswimmer has joined #openstack-kolla10:51
openstackgerritPierre Riteau proposed openstack/kayobe master: CI: Use cached cirros image for seed vm job  https://review.opendev.org/c/openstack/kayobe/+/77316311:04
*** shyam89 has joined #openstack-kolla11:09
*** shyamb has quit IRC11:11
*** k_mouza has quit IRC11:15
*** k_mouza has joined #openstack-kolla11:15
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/ussuri: Synchronise kolla-ansible installation with other branches  https://review.opendev.org/c/openstack/kolla-ansible/+/77951011:33
*** shyam89 has quit IRC11:52
*** shyamb has joined #openstack-kolla11:53
hrwwow! my stream change managed to pass zuul11:58
*** kevko has joined #openstack-kolla12:06
*** shyamb has quit IRC12:07
*** sean-k-mooney1 is now known as sean-k-mooney12:16
*** Mareo has joined #openstack-kolla12:20
kevkomgoddard: Hi, i just replied to all comments for mariadb refactor ... If you have some time ..we can discuss quickly ..12:44
openstackgerritMerged openstack/kayobe master: Add support for custom shm_size for seed container(s)  https://review.opendev.org/c/openstack/kayobe/+/77769812:44
*** wuchunyang has joined #openstack-kolla12:56
openstackgerritMaksim Malchuk proposed openstack/kolla-ansible master: Correctly configure S3 Token Middleware for Swift  https://review.opendev.org/c/openstack/kolla-ansible/+/77479113:03
*** amoralej is now known as amoralej|lunch13:04
mgoddardkevko: thanks, just reading13:16
kevkoI didn't correct my english grammar ..and I wrote fast ..so ..sorry, If you will have some questions ..ask please ..13:23
*** brinzhang_ has quit IRC13:23
*** brinzhang_ has joined #openstack-kolla13:24
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: Fix wrong configurations  https://review.opendev.org/c/openstack/kolla-ansible/+/77976413:31
*** skramaja has quit IRC13:32
openstackgerritMerged openstack/kayobe stable/ussuri: Fix IPA build on stable/ussuri  https://review.opendev.org/c/openstack/kayobe/+/77593613:34
openstackgerritMerged openstack/kayobe stable/train: Fix building CentOS 8 IPA images on stable/train  https://review.opendev.org/c/openstack/kayobe/+/77394413:34
openstackgerritMerged openstack/kayobe stable/train: Test building seed deployment images in the seed job (CentOS 8 only)  https://review.opendev.org/c/openstack/kayobe/+/77385313:36
mgoddardkevko: my confusion about the haproxy user is because you have mariadb_monitor_user/password variables defined but not really used in the patch13:36
mgoddardkevko: replied13:43
openstackgerritMerged openstack/kayobe stable/train: Fix CentOS 7 IPA build and test in CI  https://review.opendev.org/c/openstack/kayobe/+/77659513:46
kevkomgoddard: agree that *maybe* i have this variables from other patch in chain .. i can rebase ..13:47
mgoddardkevko: maybe yes? https://review.opendev.org/c/openstack/kolla-ansible/+/766952/40/ansible/group_vars/all.yml :)13:48
kevkomgoddard: will check and replace hardcoded user with variable13:50
mgoddardok13:50
kevkomgoddard: now i am doing arbitrator .. when i will have it completed ..will add to chain ..13:51
mgoddardkevko: I don't mind if the variable is used or not in the patch, just needs to be consistent13:51
kevkomgoddard: and apply comments13:51
kevkoagree13:51
*** eliaswimmer has quit IRC13:57
*** amoralej|lunch is now known as amoralej13:59
openstackgerritMerged openstack/kayobe master: CI: display VM console log on ping or SSH failure  https://review.opendev.org/c/openstack/kayobe/+/77737214:08
openstackgerritMerged openstack/kayobe master: Bump Ansible version range to 2.9 - 2.9  https://review.opendev.org/c/openstack/kayobe/+/75405414:09
*** jrollen is now known as jroll14:13
kevkomgoddard: if i summarize comments .. it's not in bad shape, right ? It is just ..cosmetic things...14:21
kevkomostl14:22
kevko*mostly14:22
mgoddardkevko: yes, I think it's mostly there14:23
mgoddardMeeting in 7 minutes14:53
mgoddard^ mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt14:53
hrwaaaaahhh!14:54
hrwrun! run!14:54
hrw;P14:54
* yoctozepto is hiding14:55
mgoddardno running or hiding please14:55
mgoddardI will come and find you if I have to14:56
hrwand then we go for a beer/favourite drink/14:56
yoctozeptois blood a type of drink?15:00
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Mar 10 15:00:49 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:01
yoctozepto\o15:01
mgoddard\-------o15:01
yoctozeptoo_15:01
priteau\o/15:02
hrw-°-15:02
hrwyoctozepto: only while it is fresh.15:03
mgoddard#topic agenda15:03
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:03
mgoddard* Roll-call15:03
mgoddard* Announcements15:03
mgoddard** PTG 19th - 23rd April, registration open | https://april2021-ptg.eventbrite.com | https://www.openstack.org/ptg/15:03
mgoddard** OpenStack feature freeze this week15:03
mgoddard* Review action items from the last meeting15:03
mgoddard* CI status15:03
mgoddard* Review requests15:03
mgoddard* PTG team signup http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html15:03
mgoddard* Glance OSSN-0088 http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020947.html15:03
mgoddard* PoC: image build & test pipeline (https://review.opendev.org/c/openstack/kolla/+/777796 and https://review.opendev.org/c/openstack/kolla-ansible/+/777946)15:03
mgoddard* Wallaby release planning15:03
mgoddard#topic announcements15:04
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:04
Fl1nto/15:04
mgoddard#info Project Teams Gathering (PTG) registration open15:04
mgoddard#link https://april2021-ptg.eventbrite.com15:04
mgoddard#link https://www.openstack.org/ptg/15:04
mgoddardIt will run from 19th-23rd April15:05
mgoddardWe can discuss it more later15:05
*** zzzeek has quit IRC15:05
mgoddard#info OpenStack feature freeze this week15:05
mgoddardWhich can only mean we are not far from Kolla feature freeze, so let's all get reviewing15:06
yoctozeptoyes, this is super painful15:06
yoctozeptoand sad15:06
*** zzzeek has joined #openstack-kolla15:06
mgoddardAny other announcements?15:06
mgoddard#topic Review action items from the last meeting15:07
*** openstack changes topic to "Review action items from the last meeting (Meeting topic: kolla)"15:07
mgoddardwuchunyang to propose toscaparser in tacker requirements to fix NFV job15:08
mgoddardmgoddard to write up options for CI registry15:08
mgoddardI think the tacker issue turned out to be more complicated15:08
wuchunyangtacker guys will fix this .15:08
mgoddard#link https://bugs.launchpad.net/bugs/191833915:08
openstackLaunchpad bug 1918339 in kolla "kolla-ansible-centos8-source-scenario-nfv Zuul test fails due to import error in tacker " [Undecided,New]15:08
mgoddardI completed my action, we can discuss later15:09
mgoddard#topic CI status15:09
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:09
*** baha has joined #openstack-kolla15:09
mgoddardobserved on ussuri, victoria and master at least15:09
mgoddardcentos8-ceph-upgrade jobs seem to be retried 3 times only to fail in some weird way15:09
mgoddardsimilarly other multinode centos8 jobs (like cells) fail: 'Ensuring config directories exist'15:09
mgoddardyoctozepto suspects centos8.315:09
mgoddardDo we still see that, or should we scrub from the whiteboard?15:10
yoctozeptoI've seen that recently yes15:10
yoctozeptobut it's not that often15:10
mgoddardok15:10
yoctozeptowe can scrub it and remember it has happened previously15:10
yoctozeptocould be some cloud unhappy with centos15:11
yoctozeptowe know what the DISK_FULL are15:11
yoctozeptoit was due to swift spaces15:11
yoctozeptofor logs15:11
yoctozeptobut it was not close to 6.3GB15:11
yoctozeptomore like 630MB15:12
yoctozeptostill dumb to save and transfer15:12
yoctozeptothe responsible was libvirt debug15:12
yoctozeptogoing to review now15:12
yoctozeptomgoddard: https://review.opendev.org/c/openstack/kolla-ansible/+/77925115:13
yoctozeptoif you agree, I will just edit this15:13
yoctozepto(or you can, whatever)15:14
mgoddardyoctozepto: agree15:14
yoctozeptook15:15
yoctozeptoediting15:15
mgoddardcool15:15
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Introduce nova_libvirt_logging_debug  https://review.opendev.org/c/openstack/kolla-ansible/+/77925115:15
yoctozeptomgoddard: approve ^15:16
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Reduce number of logs and disable ara HTML report  https://review.opendev.org/c/openstack/kolla-ansible/+/77764715:16
mgoddardyessir15:16
yoctozepto:D15:17
mgoddardok, I think we're good for CI15:18
yoctozeptoagreed15:18
mgoddardI did some 'tidying' of the whiteboard, it had developed some preamble15:18
mgoddard(the CI section)15:18
mgoddard#topic Review requests15:19
*** openstack changes topic to "Review requests (Meeting topic: kolla)"15:19
openstackgerritMerged openstack/kayobe stable/ussuri: Test building seed deployment images in the seed job  https://review.opendev.org/c/openstack/kayobe/+/77405515:19
mgoddardHit me with your code review requests (1 per person)15:19
openstackgerritMerged openstack/kayobe master: Wait for overcloud manage and provide state transitions  https://review.opendev.org/c/openstack/kayobe/+/77513515:19
openstackgerritMerged openstack/kayobe master: CI: Use cached cirros image for seed vm job  https://review.opendev.org/c/openstack/kayobe/+/77316315:19
priteauCI in Kayobe is better (pull rate limit issues seem gone?) but not perfect (seeing occasional HTTP 520 from Galaxy)15:19
yoctozeptoI am pushing things with masakari atm so nothing specific from me15:19
priteau(sorry, missed the topic change)15:19
* yoctozepto slaps mgoddard around a bit with a large trout15:20
mgoddardpriteau: +1, although I think pull limits are just luck, I have seen them15:20
mgoddardyoctozepto: that is not a review request15:20
yoctozeptomgoddard: y, looks valid :D15:21
mgoddardNobody wants a review this week?15:23
kevkodo you think we will process mariadb,proxysql ... this week ?15:24
hrwstream passed zuul so I assume that it will see revuiews15:24
mgoddardkevko: probably not the whole patch chain15:24
mgoddardkevko: but one can hope15:24
kevkoso, wallaby will be withou :/ ?15:24
yoctozeptowe are not freezing yet kevko15:25
kevkobut it is behind the door15:25
openstackgerritPierre Riteau proposed openstack/kayobe stable/ussuri: Update IPA docs and test build with extra-hardware  https://review.opendev.org/c/openstack/kayobe/+/77981115:25
yoctozeptoworry not15:25
mgoddardI would suggest that we aim to merge 2 of the proxysql patches per week15:26
mgoddardhopefully that should get us there by feature freeze15:26
yoctozeptostarting from the bottom15:26
mgoddardyes15:26
mgoddardmariadb sharding patch looks close15:26
mgoddardhttps://review.opendev.org/c/openstack/kolla-ansible/+/76695215:27
kevkoin few hours i will get into and check again15:27
mgoddardthe next one is https://review.opendev.org/c/openstack/kolla-ansible/+/770618/1815:27
kevkobtw, i have some small patches i want to review ..let me check it15:28
kevkohttps://review.opendev.org/c/openstack/kolla-ansible/+/775627 <<15:28
headphoneJamesWorking on the let's encrypt functional test, getting closer15:29
mgoddardsounds good headphoneJames15:30
mgoddardLet's move on15:30
mgoddard#topic PTG team signup15:30
*** openstack changes topic to "PTG team signup (Meeting topic: kolla)"15:30
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html15:30
mgoddardWe need to book some time for the PTG15:31
mgoddardOur 'usual' approach is 4 hours on Monday, 4 hours on Tuesday, and 2 hours on Wednesday for Kayobe15:31
mgoddardWe usually take the 13:00+ UTC slot15:32
yoctozeptoI won't be able to dedicate that much time this time15:32
yoctozeptoperhaps we could do 2 hours per deliverable15:32
mgoddardI was thinking, should we try a slot in the 4:00 - 8:00 UTC window?15:33
hrwany fits me15:34
mgoddardNot because I like waking up early, but it might attract some people in Asian timezones15:34
mgoddardwuchunyang: around?15:34
priteauKeep in mind that clocks will have changed by then15:35
mgoddardgood point15:36
priteauSo 4-8 UTC will be 5-9 UK time and 6-10 CEST15:36
yoctozeptoyes15:37
mgoddardso how do we decide? Should I send an email to openstack-discuss to gauge interest?15:37
yoctozepto++15:37
mgoddard#action mgoddard send an email to openstack-discuss to gauge interest in the 'early' PTG slot15:37
mgoddard#topic Glance OSSN-008815:38
*** openstack changes topic to "Glance OSSN-0088 (Meeting topic: kolla)"15:38
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020947.html15:38
mgoddardwho dis?15:39
mgoddardIs there anything for us to do here?15:40
priteauShip secure policies by default?15:40
mgoddardI would probably prefer glance to do that15:41
wuchunyanghi15:41
yoctozeptoit was me15:41
yoctozeptoyes, I agree15:41
yoctozeptobut it was thrown at the deployment projects15:41
mgoddardhi wuchunyang15:41
yoctozeptohow should we approach this issue?15:41
yoctozeptohi wuchunyang15:41
wuchunyangwhich issue ? i was out just now15:42
hrwwuchunyang: PTG in Asian time15:43
mgoddardwuchunyang: I think I pinged you earlier about the PTG. I'll send an email to openstack-discuss, you can reply there15:43
yoctozeptowuchunyang: the issue question was not targeted at you ;-)15:43
wuchunyangok.. i will read the mail..15:44
mgoddardSo far we have avoided modifying default policies15:44
mgoddardThe email is targeted at operators rather than deployment tools15:44
priteauMaybe this is a topic to explore more generally15:45
mgoddardShould we make an exception for this specific issue?15:45
priteauHardened Kolla-Ansible15:45
yoctozeptowell, it looks crappy to let it have this hole by default15:45
Fl1ntI suggest we don't change the default policy as if it really is a security issue upstream services will do it, and if not it means that an optional part of the service and should be let up to the operators15:45
priteauCould be a global variable that would change some settings to be more secure, with the downside of breaking some features15:46
yoctozeptoall right, we have got the first topic for ptg15:46
priteauyoctozepto: It's a hole that some deployments may rely on15:46
Fl1ntwe shouldn't, it will have impact on upgrades/updates as some side effects will arise.15:46
yoctozeptopriteau: yes, unfortunately15:46
yoctozeptoyeah, stable/upgrades is something I would prefer not to touch indeed15:47
mgoddardI think if it were as simple as just changing the policy, glance would do it15:47
mgoddardthey are just as capable as us of making the change15:47
yoctozeptoyes, it would have to be behind a flag like priteau said15:47
Fl1ntbut are they calling metadef APIs ?15:48
Fl1nt^what are15:48
mgoddardI suppose I'm not against having it behind a flag15:48
yoctozeptoFl1nt: https://docs.openstack.org/api-ref/image/v2/metadefs-index.html15:49
Fl1ntyeah something like INVALID_GLANCE_OSSN008815:49
mgoddardWhat I didn't mention earlier in the PTG section is that I created an etherpad15:49
* hrw out15:49
mgoddard#link  https://etherpad.opendev.org/p/kolla-xena-ptg15:49
Fl1ntaaaaah metadefinitions, got it15:49
mgoddarddoes someone want to add this topic?15:49
priteauFl1nt: https://docs.openstack.org/glance/latest/#metadata-definitions15:49
yoctozeptoFl1nt: yeas15:50
yoctozeptosomeone added it15:50
yoctozepto(not me)15:50
mgoddardme15:50
Fl1ntwhat I don't understand, is that from the mailing list responses, it seems to only be a default policy shape issue on glance metadef api, so glance should just add the usual is_admin filter, but did they responded to the mail?15:52
*** dmsimard8 has joined #openstack-kolla15:52
mgoddardI don't see any responses15:52
mgoddardWe're running out of time15:53
mgoddardLet's discuss at the PTG15:53
Fl1ntyep15:53
mgoddard#topic PoC: image build & test pipeline15:53
*** openstack changes topic to "PoC: image build & test pipeline (Meeting topic: kolla)"15:53
mgoddard#link https://etherpad.opendev.org/p/docker-pull-limits15:53
mgoddardI wrote up some options in there15:53
mgoddardSee 3a. and 3b.15:53
*** dmsimard has quit IRC15:54
*** dmsimard8 is now known as dmsimard15:54
mgoddardI think we don't have time to discuss this topic properly today15:55
mgoddardBut please read the etherpad, and we can discuss next week15:55
mgoddard#topic Open discussion15:56
*** openstack changes topic to "Open discussion (Meeting topic: kolla)"15:56
priteauI would like to know what is our plan for Docker iptables management in Xena15:57
priteauIt was announced to be disabled in Victoria I think?15:57
Fl1ntwoot?15:57
priteauBut it wasn't15:57
priteauhttps://review.opendev.org/c/openstack/kolla-ansible/+/75179515:57
Fl1ntah you're talking about the infamous ebtable thingy ?15:57
priteauand https://review.opendev.org/c/openstack/kolla-ansible/+/68987015:58
priteauNo, I am talking about Docker manipulating iptables rules on its own15:58
yoctozeptoyeah, we are late15:58
priteauParticularly, changing the default FORWARD chain policy15:58
priteauWhich can cause issues in some environments15:58
Fl1ntDon't use use host networking model ?15:59
Fl1nts/use/you/15:59
priteauOf course, but it still has an impact15:59
* yoctozepto went away16:00
*** LinPeiWen has quit IRC16:00
mgoddardGood question priteau16:00
mgoddard"As Michal pointed out, this breaks kolla-build unless you set  network_mode = host in kolla-build.conf."16:00
Fl1ntyes, I do on my own.16:01
mgoddardI think I had problems when I tested it here: https://review.opendev.org/c/openstack/kolla-ansible/+/75198216:01
mgoddardI can try it again16:02
mgoddardwe do now have network_mode = host in kolla-build.conf16:03
mgoddardanyway, time's up16:03
mgoddardThanks all16:03
mgoddard#endmeeting16:03
*** openstack changes topic to "IRC meetings on Wednesdays @ 15:00 UTC - agenda @ https://goo.gl/OXB0DL | Whiteboard: https://bit.ly/2MM7mWF | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:03
openstackMeeting ended Wed Mar 10 16:03:21 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:03
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-10-15.00.html16:03
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-10-15.00.txt16:03
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2021/kolla.2021-03-10-15.00.log.html16:03
wuchunyangI found a new bug that ansible os_nova_flavor module does not satisfy idempotency,it will recreate the flavor when swap is set to 0. this cause the octavia containers to restart16:03
Fl1ntgot to go, see ya folks!16:04
wuchunyangansible try to query the flavor , when swap is set to 0, nova will return empty string instead.16:04
mgoddardwuchunyang: what if we pass an empty string?16:05
wuchunyangthe filed must int.16:05
mgoddardwuchunyang: it probably needs to be fixed in the collection16:05
wuchunyangyes, should we add a workaround now ?16:06
openstackgerritMark Goddard proposed openstack/kolla-ansible master: [docker] Added a new flag to disable default network  https://review.opendev.org/c/openstack/kolla-ansible/+/68987016:07
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Disable Docker iptables and bridge networking by default  https://review.opendev.org/c/openstack/kolla-ansible/+/75179516:07
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: test kolla build without iptables or bridge  https://review.opendev.org/c/openstack/kolla-ansible/+/75198216:07
openstackgerritMerged openstack/kayobe master: CI: Add Ubuntu seed job  https://review.opendev.org/c/openstack/kayobe/+/77526716:08
*** wuchunyang has quit IRC16:08
mgoddardwuchunyang: swap: "{{ octavia_amp_flavor.swap | default(omit, true) }}"16:09
*** Fl1nt has quit IRC16:09
openstackgerritwu.chunyang proposed openstack/kolla-ansible master: Use Docker healthchecks for zun services  https://review.opendev.org/c/openstack/kolla-ansible/+/77982616:12
openstackgerritMerged openstack/kolla-ansible master: Correctly configure S3 Token Middleware for Swift  https://review.opendev.org/c/openstack/kolla-ansible/+/77479116:35
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134016:46
*** rpittau is now known as rpittau|afk17:05
yoctozeptomgoddard, mnasiadka: cephadm failure https://zuul.opendev.org/t/openstack/build/5bed2d2292164ee7b6b7f006197ecbe917:05
yoctozeptoand it looks similar to the one you mentioned today from the whiteboard, mgoddard17:06
yoctozeptobut here on ubuntu17:06
openstackgerritMichal Arbet proposed openstack/kolla-ansible master: Set changed_when to false for group_by tasks  https://review.opendev.org/c/openstack/kolla-ansible/+/77983617:08
openstackgerritMerged openstack/kayobe stable/train: CI: use overlay2 storage driver for overcloud deploy  https://review.opendev.org/c/openstack/kayobe/+/77737317:24
ozzzoI built a train test cluster on a heat stack. When I try to build a VM it fails with "No valid host was found"17:24
ozzzoso I looked at my hypervisors, and they are using more memory than they have:17:24
ozzzo|  3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU            | 172.16.2.230 | up    |          2 |     8 |          32768 |     30719 |17:24
ozzzowhat is using 32G of RAM on an empty HV?17:25
kevkops faux ? :D17:28
openstackgerritPierre Riteau proposed openstack/kayobe stable/train: Update IPA docs and test build with extra-hardware  https://review.opendev.org/c/openstack/kayobe/+/77983917:28
ozzzoit's not actually used on the compute node; just in the hypervisor list17:29
ozzzoI have 22G free17:30
ozzzoMem:          30157        1412       22899          17        5845       2832317:30
*** amoralej is now known as amoralej|off17:36
kevkowell, it's reporting your node available resources, isn't it ?17:36
ozzzoI think the hypervisor list is pulling from placement17:36
ozzzoI'm not exactly sure how that works17:36
ozzzobut somehow openstack thinks I've used up 32G out of 30G available, before building any VMs17:37
kevkowhat is the command ?17:38
kevkohypervisor list << if i run this ..i don't have info about ram17:39
ozzzoos hypervisor list --long17:41
kevkoweird17:42
ozzzoIsn't there some kind of memory reservation setting in nova.conf? I wonder if I have that set to 32G17:47
ozzzoahah that's it: reserved_host_memory_mb17:48
ozzzowhat's a reasonable value for a compute on a VM, maybe 2G?17:48
kevkodon't understand17:49
ozzzoin nova.conf, I have reserved_host_memory_mb = 3276817:49
ozzzoreserving 32G for the hypervisor os17:49
ozzzowhich is reasonable in prod where the boxes have tons of memory17:50
ozzzobut on a 30G VM, not so much17:50
yoctozeptoozzzo: 1G is enough if you run a very basic config17:50
kevkoah, you are asking how much you should decrease reserved host memory ?17:50
ozzzorighton, ty!17:50
ozzzotrying 1G17:50
yoctozeptoyou might also want to configure the nodes in placement regarding the overcommit17:50
yoctozeptoto avoid surpises :D17:50
ozzzoI'm not sure I follow17:51
yoctozeptoby default there is certain overcommit allowed17:51
yoctozeptoso that you can run vms with more memory than you have on the host17:52
yoctozeptothis part is configurable via the placement client17:52
ozzzowith resource provider set?17:53
yoctozeptoyes17:56
ozzzook I'll experiment with that, ty17:56
*** bengates has quit IRC17:56
*** bengates has joined #openstack-kolla17:57
*** bengates has quit IRC17:57
*** bengates has joined #openstack-kolla17:58
*** bengates has quit IRC18:03
*** jonaspaulo has joined #openstack-kolla18:10
*** dcapone2004 has joined #openstack-kolla18:12
*** gfidente is now known as gfidente|afk18:16
*** andrewbonney has quit IRC18:19
*** tbarron has joined #openstack-kolla18:37
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134018:43
*** jonaspaulo has quit IRC18:43
*** e0ne has quit IRC18:48
*** zul has quit IRC18:54
*** cah_link has quit IRC18:55
*** cah_link has joined #openstack-kolla19:12
openstackgerritMerged openstack/kolla-ansible master: Introduce nova_libvirt_logging_debug  https://review.opendev.org/c/openstack/kolla-ansible/+/77925119:22
openstackgerritMerged openstack/kolla-ansible master: Reduce number of logs and disable ara HTML report  https://review.opendev.org/c/openstack/kolla-ansible/+/77764719:23
*** bengates has joined #openstack-kolla19:24
*** bengates has quit IRC19:30
*** bengates has joined #openstack-kolla19:31
*** brinzhang0 has joined #openstack-kolla19:33
*** iniazi has joined #openstack-kolla19:34
*** dciabrin__ has joined #openstack-kolla19:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/victoria: Introduce nova_libvirt_logging_debug  https://review.opendev.org/c/openstack/kolla-ansible/+/77978519:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/victoria: Reduce number of logs and disable ara HTML report  https://review.opendev.org/c/openstack/kolla-ansible/+/77978619:34
*** dcapone2004 has quit IRC19:34
*** also_stingrayza has joined #openstack-kolla19:34
*** bengates has quit IRC19:35
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Introduce nova_libvirt_logging_debug  https://review.opendev.org/c/openstack/kolla-ansible/+/77978719:36
*** benj_- has joined #openstack-kolla19:37
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Reduce number of logs and disable ara HTML report  https://review.opendev.org/c/openstack/kolla-ansible/+/77978819:37
*** dosaboy_ has joined #openstack-kolla19:38
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Reduce number of logs and disable ara HTML report  https://review.opendev.org/c/openstack/kolla-ansible/+/77978819:39
*** dmsimard has quit IRC19:42
*** brinzhang_ has quit IRC19:42
*** mjturek has quit IRC19:42
*** baha has quit IRC19:42
*** iniazi_ has quit IRC19:42
*** benj_ has quit IRC19:42
*** oyrogerg has quit IRC19:42
*** dosaboy has quit IRC19:42
*** hjensas has quit IRC19:42
*** dciabrin_ has quit IRC19:42
*** Underknowledge has quit IRC19:42
*** stingrayza has quit IRC19:42
*** dmsimard has joined #openstack-kolla19:43
*** Underknowledge has joined #openstack-kolla19:43
*** irclogbot_3 has quit IRC19:44
*** hjensas has joined #openstack-kolla19:48
*** ysirndjuro has joined #openstack-kolla19:49
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Reduce number of logs and disable ara HTML report  https://review.opendev.org/c/openstack/kolla-ansible/+/77978820:04
*** bsanjeewa has joined #openstack-kolla20:09
*** bsanjeewa has quit IRC20:10
*** eliaswimmer has joined #openstack-kolla20:11
*** bsanjeewa has joined #openstack-kolla20:13
eliaswimmerIt's only open for one day, but just in case someone is looking for a job building a new multi-datacenter openstack cluster for our university in Vienna, please apply: https://jobs.tuwien.ac.at/Job/145680 (only CV needed)20:14
mnasiadkayoctozepto: that’s probably some cephadm bug, will look into that tomorrow.20:27
*** gfidente|afk is now known as gfidente20:30
*** Aethylred has joined #openstack-kolla20:38
*** rouk has joined #openstack-kolla20:44
roukhow do you make migration_interface_address not brick cold migrations?20:45
rouklive migrations work right, but cold goes back to the api address, which isnt listening anymore with nova_ssh20:46
*** gfidente is now known as gfidente|afk20:50
*** gfidente|afk has quit IRC20:57
*** k_mouza has quit IRC20:57
*** rouk has quit IRC21:27
openstackgerritMaksim Malchuk proposed openstack/kolla-ansible stable/ussuri: Correctly configure S3 Token Middleware for Swift  https://review.opendev.org/c/openstack/kolla-ansible/+/77987021:41
openstackgerritMaksim Malchuk proposed openstack/kolla-ansible stable/victoria: Correctly configure S3 Token Middleware for Swift  https://review.opendev.org/c/openstack/kolla-ansible/+/77987121:42
*** johnsom has quit IRC21:46
*** johnsom has joined #openstack-kolla21:46
*** bsanjeewa has quit IRC22:04
*** bsanjeewa_ has joined #openstack-kolla22:05
*** vishalmanchanda has quit IRC22:08
*** eliaswimmer has quit IRC22:09
*** zzzeek has quit IRC22:16
*** zzzeek has joined #openstack-kolla22:18
headphoneJamesis a comment with "Depends-On:" enough to trigger zuul to build the kolla containers in the Depends-Ons to use in the current test run?22:33
headphoneJamesI'm trying to determine if https://review.opendev.org/c/openstack/kolla-ansible/+/741340 is building https://review.opendev.org/#/c/741339 and using the new let's encrypt containers22:34
*** k_mouza has joined #openstack-kolla22:42
*** cah_link has quit IRC22:43
*** k_mouza has quit IRC22:43
*** k_mouza has joined #openstack-kolla22:43
*** k_mouza has quit IRC22:47
*** k_mouza has joined #openstack-kolla22:53
*** k_mouza has quit IRC22:57
*** ysirndjuro has left #openstack-kolla23:20
*** k_mouza has joined #openstack-kolla23:28
*** rpittau|afk has quit IRC23:37
*** rpittau|afk has joined #openstack-kolla23:37
*** k_mouza has quit IRC23:55
*** k_mouza has joined #openstack-kolla23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!