Wednesday, 2019-07-03

*** calbers has quit IRC00:04
openstackgerritGaëtan Trellu proposed openstack/kolla master: WIP: Add HAcluster containers  https://review.opendev.org/66876500:08
*** calbers has joined #openstack-kolla00:13
*** calbers has quit IRC00:26
*** calbers has joined #openstack-kolla00:27
*** cgrosjean has quit IRC00:36
*** cgrosjean has joined #openstack-kolla00:37
*** cgrosjean has quit IRC00:41
*** schwicht has quit IRC00:41
*** Sravan has joined #openstack-kolla01:24
*** Sravan has quit IRC01:28
*** kplant has quit IRC01:41
*** iniazi_ has joined #openstack-kolla01:41
*** iniazi has quit IRC01:43
*** iniazi has joined #openstack-kolla01:45
*** iniazi_ has quit IRC01:46
*** iniazi_ has joined #openstack-kolla01:49
*** iniazi has quit IRC01:50
*** unicell has joined #openstack-kolla01:56
*** cah_link has quit IRC02:00
*** unicell has quit IRC02:02
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed  https://review.opendev.org/66877302:08
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed  https://review.opendev.org/66877302:29
*** unicell has joined #openstack-kolla02:29
*** unicell has quit IRC02:34
openstackgerritgugug proposed openstack/kolla-ansible master: Update the UPPER_CONSTRAINTS_FILE to releases.openstack.org  https://review.opendev.org/66877702:38
openstackgerritgugug proposed openstack/kolla-ansible master: Update the UPPER_CONSTRAINTS_FILE to releases.openstack.org  https://review.opendev.org/66877702:43
openstackgerritGaëtan Trellu proposed openstack/kolla master: WIP: Add HAcluster containers  https://review.opendev.org/66876502:44
*** dasp has quit IRC02:46
*** skramaja has joined #openstack-kolla02:55
*** unicell has joined #openstack-kolla02:59
*** dasp has joined #openstack-kolla03:03
*** unicell has quit IRC03:09
*** unicell has joined #openstack-kolla03:17
*** unicell has quit IRC03:22
openstackgerrithuang liujie proposed openstack/kolla master: Blacklist sphinx 2.1.0 (autodoc bug)  https://review.opendev.org/66877903:30
*** Sravan has joined #openstack-kolla03:31
openstackgerrithuang liujie proposed openstack/kolla master: Blacklist sphinx 2.1.0 (autodoc bug)  https://review.opendev.org/66877903:36
*** Sravan has quit IRC03:59
*** Sravan has joined #openstack-kolla04:03
*** Sravan has quit IRC04:12
*** unicell has joined #openstack-kolla04:18
*** unicell has quit IRC04:22
*** unicell has joined #openstack-kolla04:30
*** Sravan has joined #openstack-kolla04:33
*** whoami-rajat has joined #openstack-kolla04:34
*** unicell has quit IRC04:35
*** Sravan has quit IRC04:37
*** unicell has joined #openstack-kolla04:42
*** unicell has quit IRC04:47
*** unicell has joined #openstack-kolla04:55
*** unicell has quit IRC05:00
*** pcaruana has joined #openstack-kolla05:06
*** unicell has joined #openstack-kolla05:07
*** unicell has quit IRC05:12
*** unicell has joined #openstack-kolla05:14
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the build-openstack-deploy-guide faild  https://review.opendev.org/66878605:25
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the build-openstack-deploy-guide failed  https://review.opendev.org/66878605:34
*** Sravan has joined #openstack-kolla05:35
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the build-openstack-deploy-guide failed  https://review.opendev.org/66878605:38
*** Sravan has quit IRC05:40
*** igordc has quit IRC05:41
*** Luzi has joined #openstack-kolla05:54
yoctozeptomnasiadka: reviewing hard06:21
yoctozeptomorning06:21
mnasiadkamorning06:21
mnasiadkayoctozepto: well, finally I have some time06:21
mnasiadkaThat I don't need to sacrifice for Jenkins and other crap ;)06:22
yoctozeptomnasiadka: ;D06:24
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add ceph-mds/rgw/nfs to gate  https://review.opendev.org/66241206:25
*** luksky has joined #openstack-kolla06:27
*** Sravan has joined #openstack-kolla06:27
*** Sravan has quit IRC06:31
*** lemko has joined #openstack-kolla06:35
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the build-openstack-deploy-guide failed  https://review.opendev.org/66878606:44
*** luksky has quit IRC06:48
*** unicell has quit IRC06:58
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix deploy guide build (missing kolla project reference)  https://review.opendev.org/66880107:04
*** luksky has joined #openstack-kolla07:06
*** rpittau|afk is now known as rpittau07:06
openstackgerritMerged openstack/kolla-ansible stable/stein: Restart all nova services after upgrade  https://review.opendev.org/66793407:30
*** cgrosjean has joined #openstack-kolla07:37
*** cgrosjean has quit IRC07:37
openstackgerritMerged openstack/kolla master: ubuntu: use UCA/train  https://review.opendev.org/66590807:37
openstackgerritMerged openstack/kolla stable/pike: Fix bandit failure B105 in keystone fernet  https://review.opendev.org/66862507:37
openstackgerritMerged openstack/kolla stable/stein: Update versions: ./tools/version-check.py  https://review.opendev.org/66849107:37
openstackgerritMerged openstack/kolla stable/rocky: Fix deployment of ceph-nfs container failing due to wrong USER  https://review.opendev.org/66782507:37
openstackgerritMerged openstack/kolla master: Fixes ceph-osd deploys with loop devices.  https://review.opendev.org/66822207:37
openstackgerritMerged openstack/kolla master: Extract CI vars for use in kolla-ansible jobs  https://review.opendev.org/66822607:37
openstackgerritMerged openstack/kolla master: Add more entries into SKIPPED_IMAGES  https://review.opendev.org/66779707:37
openstackgerritMerged openstack/kolla master: Enable CI jobs for ceph upgrade, NFV, cinder LVM, ironic  https://review.opendev.org/66606807:38
openstackgerritMerged openstack/kolla master: build: add timestamps to start/end of build  https://review.opendev.org/66758807:38
openstackgerritMerged openstack/kolla master: base: do not put grafana repo twice  https://review.opendev.org/66758707:38
*** cgrosjean has joined #openstack-kolla07:38
*** luksky11 has joined #openstack-kolla07:43
mgoddardmorning07:43
yoctozeptomorning mgoddard07:43
*** luksky has quit IRC07:45
*** Sravan has joined #openstack-kolla07:50
openstackgerritRadosław Piliszek proposed openstack/kolla stable/stein: Extract CI vars for use in kolla-ansible jobs  https://review.opendev.org/66881507:51
*** Sravan has quit IRC07:55
*** luksky11 has quit IRC07:55
*** rgogunskiy has quit IRC08:01
*** dougsz has joined #openstack-kolla08:08
*** luksky11 has joined #openstack-kolla08:08
*** luksky11 has quit IRC08:11
*** shyamb has joined #openstack-kolla08:21
*** priteau has joined #openstack-kolla08:26
*** luksky11 has joined #openstack-kolla08:27
*** shyamb has quit IRC08:28
*** luksky11 has quit IRC08:33
*** ivve has joined #openstack-kolla08:34
*** zzzeek has quit IRC08:48
*** zzzeek has joined #openstack-kolla08:49
*** shyamb has joined #openstack-kolla08:53
*** luksky11 has joined #openstack-kolla09:12
openstackgerritMerged openstack/kolla-ansible stable/rocky: (Rocky and Queens only) CI: Fix ceph jobs for kolla  https://review.opendev.org/66833509:13
*** Sravan has joined #openstack-kolla09:35
*** Sravan has quit IRC09:39
yoctozeptomirror.iad.rax down again today in case you want to test something in CI09:44
*** guozijn has joined #openstack-kolla09:50
*** k_mouza has joined #openstack-kolla09:51
*** Wasaac has joined #openstack-kolla09:57
*** lei-zh has joined #openstack-kolla09:59
*** skramaja has quit IRC10:01
*** lei-zh has quit IRC10:01
*** skramaja has joined #openstack-kolla10:01
*** lei-zh has joined #openstack-kolla10:02
*** guozijn has quit IRC10:14
*** shyamb has quit IRC10:20
*** lei-zh has quit IRC10:36
*** Sravan has joined #openstack-kolla10:37
mnasiadkawho would want to test anything, if rackspace is down :)10:39
*** shyamb has joined #openstack-kolla10:40
*** Sravan has quit IRC10:41
mnasiadkagood all those merges went through10:45
mnasiadkamgoddard: I might not make it to the meeting today, but I'll try at least to follow on a phone app ;)10:46
mgoddardmnasiadka: no problem. Any updates?10:47
mnasiadkamgoddard: not really, had a lot of $job stuff going on, but now they will leave me alone for a month :)10:47
mgoddardmnasiadka: lucky you!10:47
mnasiadkamgoddard: let's see how lucky I will be from 1st of August :)10:48
mgoddardmnasiadka: I think very lucky10:48
mnasiadkamgoddard: usually if you're on leave period - you're like radioactive chernobyl artifacts :)10:48
mnasiadkaso that's why a month of peace :)10:49
mnasiadkamgoddard: is there anything I should focus on now? some Stein release dependencies?10:51
mgoddardmnasiadka: this is my biggest stein issue (after galera) right now: https://review.opendev.org/66766510:51
mgoddardmnasiadka: stein bug fixing always useful10:53
mgoddardmnasiadka: some good stuff in https://etherpad.openstack.org/p/kolla-ci-errors to try to get on top of10:53
mnasiadkamgoddard: well, I think nobody ever thought of bifrost as something that needs persisting :) so that's why upgrade was not covered properly :)10:53
mnasiadkamgoddard: I don't see any LP bugs targeted to 8.0.0, or am I blind? :)10:54
mnasiadkamgoddard: can we mark this BP as done, or do we need anything else? https://blueprints.launchpad.net/kolla/+spec/centos-ceph-nautilus10:55
yoctozeptomnasiadka: https://launchpad.net/kolla-ansible/+milestone/8.0.0 ?10:56
mnasiadkayoctozepto: yeah, I looked at kolla, my bad :)10:56
yoctozepto;-)10:56
*** guozijn has joined #openstack-kolla10:58
yoctozeptomnasiadka: re: bp - you can slap the 'back to centos ceph repo' changes there for consistency11:00
mnasiadkayoctozepto: done11:03
yoctozeptomnasiadka: thanks11:06
*** guozijn has quit IRC11:10
*** ohwhyosa[m] has joined #openstack-kolla11:14
*** Sravan has joined #openstack-kolla11:19
*** k_mouza has quit IRC11:21
yoctozeptomgoddard: want me to handle nova discovery issue with ironic further?11:21
*** kplant has joined #openstack-kolla11:22
*** Sravan has quit IRC11:24
*** priteau has quit IRC11:32
*** Wasaac has quit IRC11:32
*** Wasaac has joined #openstack-kolla11:33
ohwhyosa[m]Morning everyone!11:34
ohwhyosa[m]https://pastebin.com/PGAABX9h u.u been like that since yesterday, tried with kolla-ansible destroy (which fails when I use --include-images, and if I don't use it, it hangs forever on one of the hosts at "Remove kolla configuration files"11:35
ohwhyosa[m]For now I'm just gonna disable ceilometer (though yesterday it failled on other services and stuff)11:36
ohwhyosa[m]Still loving it, though, I get it is my fault for doing something the wrong way11:37
*** Wasaac has quit IRC11:38
yoctozeptoohwhyosa[m]: morning, are you doing an upgrade or fresh install, which release (of openstack and kolla-ansible)?11:38
*** schwicht has joined #openstack-kolla11:42
*** k_mouza has joined #openstack-kolla11:52
ohwhyosa[m]rocky, fresh install11:52
ohwhyosa[m]yoctozepto: well, "fresh install" after kolla destroy11:52
*** schwicht has quit IRC11:53
ohwhyosa[m]kolla-ansible -i multinode destroy --yes-i-really-really-mean-it11:53
ohwhyosa[m]But it hangs on two hosts (like forever, and -vvvv just says escalation succeeded)11:55
ohwhyosa[m]also, the proxy var for the registry should be put on the container as -e whatever or before running the registry, right? Else it wouldn't work, at least for me11:55
ohwhyosa[m]want me to add it to the docs?11:56
ohwhyosa[m]oh! yoctozepto you need globals.yml and multinode inv right?12:07
*** Wasaac has joined #openstack-kolla12:11
*** schwicht has joined #openstack-kolla12:15
*** shyamb has quit IRC12:16
*** priteau has joined #openstack-kolla12:17
*** Wasaac has quit IRC12:18
goldyfruito/12:23
*** Wasaac has joined #openstack-kolla12:23
openstackgerritMerged openstack/kolla-ansible master: Fix deploy guide build (missing kolla project reference)  https://review.opendev.org/66880112:27
*** shyamb has joined #openstack-kolla12:27
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Add a spec for Nova cells  https://review.opendev.org/61664512:30
*** cgrosjean has quit IRC12:32
*** cgrosjean has joined #openstack-kolla12:35
*** cah_link has joined #openstack-kolla12:37
ohwhyosa[m]yoctozepto: globals.yml https://pastebin.com/NxzQuqkj12:37
ohwhyosa[m]and  multinode https://pastebin.com/1UW7v1uE12:40
*** cah_link has quit IRC12:44
yoctozeptoaah rocky, will take a look, can you post a bug on launchpad so that I don't forget?12:45
ohwhyosa[m]Yeah, sure, what would you need me to put in there?12:46
ohwhyosa[m]I could try fresh deploying stein12:46
*** happyhemant has joined #openstack-kolla12:46
ohwhyosa[m]the inventory has the vars removed for ehhhm reasons, really12:46
*** cah_link has joined #openstack-kolla12:47
*** schwicht has quit IRC12:50
*** schwicht has joined #openstack-kolla13:09
*** owalsh has quit IRC13:13
*** cah_link has quit IRC13:13
*** owalsh has joined #openstack-kolla13:14
*** owalsh has quit IRC13:15
*** priteau has quit IRC13:15
*** owalsh has joined #openstack-kolla13:15
*** dougsz has quit IRC13:18
*** priteau has joined #openstack-kolla13:18
*** dougsz has joined #openstack-kolla13:18
*** owalsh_ has joined #openstack-kolla13:20
*** BjoernT has joined #openstack-kolla13:20
*** dougsz has quit IRC13:22
*** owalsh has quit IRC13:23
*** shyamb has quit IRC13:23
*** owalsh_ is now known as owalsh13:23
mnasiadkaohwhyosa[m]: the problem with Ansible hanging during become sounds like some dns problems, or something similar...13:24
*** skramaja has quit IRC13:30
priteauWould it make sense to configure HAProxy to use the health check middleware, for those services that support it? See https://docs.openstack.org/oslo.middleware/latest/reference/healthcheck_plugins.html13:34
mnasiadkapriteau: most probably it makes sense, I thought we are doing some kind of healthcheck already in HAProxy, but if it doesn't use oslo healthcheck - might make sense to think about implementing it :)13:41
priteauFrom what I can see it uses `check inter 2000 rise 2 fall 5`13:41
priteauI assume it hits / and that's it13:42
*** Wasaac has quit IRC13:43
*** Wasaac_ has joined #openstack-kolla13:43
*** zzzeek has quit IRC13:49
*** zzzeek has joined #openstack-kolla13:51
*** zzzeek has quit IRC13:53
*** zzzeek has joined #openstack-kolla13:54
*** zzzeek has quit IRC13:55
*** zzzeek has joined #openstack-kolla13:56
*** spiette has quit IRC13:57
*** Wasaac_ has quit IRC13:58
*** BjoernT_ has joined #openstack-kolla13:59
*** spiette has joined #openstack-kolla14:01
*** BjoernT has quit IRC14:02
*** k_mouza has quit IRC14:06
*** BjoernT_ is now known as BjoernT14:08
*** cgrosjean has quit IRC14:12
*** ivve has quit IRC14:14
*** munimeha1 has joined #openstack-kolla14:18
*** cgrosjean has joined #openstack-kolla14:23
*** Sravan has joined #openstack-kolla14:26
*** Sravan has quit IRC14:31
*** k_mouza has joined #openstack-kolla14:33
ohwhyosa[m]<mnasiadka "ohwhyosa: the problem with Ansib"> Hmmm I'm using ip directly, and it does connect to them in other steps, how would it be dns mnasiadka ?14:35
mnasiadkaohwhyosa[m]: and resolv.conf is empty?14:35
mnasiadka(On the problematic nodes, not on the deployer node)14:36
mnasiadkapriteau: so then there is room for improvement :)14:36
ohwhyosa[m]on the problematic ones, resolv.conf has 127.0.0.53 (so, the netplan stub, our actual netplan addresses are google and bind9) and options edns014:40
*** Wasaac has joined #openstack-kolla14:46
*** NOXOiD has joined #openstack-kolla14:46
*** generalfuzz has joined #openstack-kolla14:47
*** igordc has joined #openstack-kolla14:50
*** factor has quit IRC14:51
*** factor has joined #openstack-kolla14:52
yoctozeptomnasiadka, priteau: the biggest check joke is the kolla-ansible check action running "post-deployment smoke tests" which usually checks only whether mariadb says "MariaDB" on its VIP address :-)14:52
*** Luzi has quit IRC14:58
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Clean up Zuul repo mirrors for CentOS  https://review.opendev.org/66415114:59
*** icarusfactor has joined #openstack-kolla15:00
*** factor has quit IRC15:00
yoctozeptomeeting, meeting?15:01
mgoddard#startmeeting kolla15:02
openstackMeeting started Wed Jul  3 15:02:09 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:02
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:02
*** openstack changes topic to " (Meeting topic: kolla)"15:02
openstackThe meeting name has been set to 'kolla'15:02
mgoddard#topic rollcall15:02
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:02
mgoddard\o15:02
yoctozeptoo/15:03
mgoddardmnasiadka: around?15:03
mgoddardegonzalez: ping15:03
mnasiadkaYup15:03
mnasiadkao/15:03
mgoddardcloudnull: \o15:04
cloudnullo/15:04
mgoddard#topic agenda15:04
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:04
mgoddard* Roll-call15:04
mgoddard* Announcements15:04
mgoddard** Kolla Train priorities results are in https://etherpad.openstack.org/p/kolla-train-priorities15:04
mgoddard** Vote for kayobe becoming (or not) a kolla deliverable http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007250.html15:04
mgoddard** Vote for yoctozepto becoming core: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007428.html15:04
mgoddard** Kevin Carter (cloudnull) is now Kolla liaison for TripleO15:04
mgoddard* Review action items from last meeting15:04
mgoddard* Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:05
mgoddard* Stein release status15:05
mgoddard* IPv6 community goal15:05
mgoddard#topic announcements15:05
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:05
mgoddard#info Kolla Train priorities results are in15:05
mgoddard#link https://etherpad.openstack.org/p/kolla-train-priorities15:05
mgoddardI updated the whiteboard with these work items15:05
mgoddard#info Vote for kayobe becoming (or not) a kolla deliverable15:05
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007250.html15:05
mgoddard#info Vote for yoctozepto becoming core15:06
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007428.html15:06
mgoddard#info Kevin Carter (cloudnull) is now Kolla liaison for TripleO15:06
cloudnullo/15:06
mgoddardIs that a reasonable way to describe your role cloudnull?15:06
cloudnullcorrect15:07
yoctozeptocloudnull: or would you prefer TripleO agent? ;D15:07
mgoddardgreat, welcome15:07
mgoddardany other announcements?15:07
mgoddardNo action items last time15:08
cloudnullI'm still ramping up on the current state of Kolla, but I will be running point on Kolla espionage for TripleO :)15:08
yoctozeptocloudnull: that's the spirit!15:08
mgoddardcloudnull: we'll soon have you working as a double agent :p15:08
cloudnull:P15:08
mgoddard#topic Kolla whiteboard15:08
*** openstack changes topic to "Kolla whiteboard (Meeting topic: kolla)"15:09
mgoddard#link https://etherpad.openstack.org/p/KollaWhiteBoard15:09
yoctozeptowhiteboard pretty colorful nowadays15:09
mgoddardchecking CI15:09
yoctozeptoCI unblocked15:09
mgoddardas long as the colour is GREEN I'm happy15:09
yoctozeptoinfra issues handled as soon as they appear15:09
yoctozeptosorry for my bg color15:10
mgoddardCI mostly looking green15:11
yoctozeptomgoddard: Bifrost FAILING on OVH cloud due to weird hostname issues, passing elsewhere.15:11
mgoddardthanks for some great work on CI fixes yoctozepto15:11
yoctozeptowas to ask you about this point15:11
mgoddardgo ahead15:11
yoctozeptomgoddard: you're welcome15:11
yoctozeptomgoddard: I think I have some lag with IRC15:12
yoctozeptosorry if I seem delayed15:12
yoctozeptoasking if there is an example of this failure15:12
yoctozeptoso I could take a look and take it off the whiteboard ;D15:12
mgoddardyoctozepto: it's fixed in master because bifrost removed rabbitmq15:13
mgoddard'fixed'15:13
mgoddardso might not be worth your time15:14
*** priteau has quit IRC15:14
yoctozeptomgoddard: k, thanks for explaining15:14
yoctozeptocould you put that on whiteboard?15:14
yoctozeptothis explanation15:14
mgoddarddone15:14
yoctozeptothanks15:15
yoctozeptomaybe a quick peek at CI issues15:15
mgoddardyou mean https://etherpad.openstack.org/p/kolla-ci-errors?15:15
mgoddard#link https://etherpad.openstack.org/p/kolla-ci-errors15:16
yoctozeptoyup15:16
yoctozeptothere is this special category at the bottom15:16
mgoddardsure, lead the way15:16
yoctozeptofor odd Zuul behavior15:16
yoctozeptothe thing is to notify infra and leave a link there15:16
yoctozeptomaybe they will find the root cause15:16
mgoddardwhat is the symptom?15:16
mgoddardcould you add that to the page?15:17
yoctozeptomgoddard: discrepancy between txt log/json log/ara/primary ara/primary log, oddities like files existing before being created15:17
mgoddardmessy15:17
yoctozeptoyup, Zuul must run the same instance twice or more15:18
yoctozeptorare event I think but still15:18
yoctozeptobest if they track it down15:18
yoctozeptoonto the 'normal' issues15:18
yoctozepto"    maybe we should consider retrying all such tasks like 3 times to ensure it is not a "once in a thousand" issue"15:19
yoctozeptothis is for trying to start a container once when docker registry has its break15:19
yoctozeptoworth considering I think15:20
mgoddard+115:20
yoctozeptothen the next issue15:20
mgoddardwe have seen a few issues recently where retries would have helped15:20
yoctozepto500 Internal Server Error creating services and endpoints15:20
yoctozeptono idea what's the cause15:20
yoctozeptohelp appreciated15:21
mgoddardin that keystone bootstrap issue it looks like keystone just hadn't come up properly yet15:21
yoctozeptohmm, yeah, leaving note to check keystone logs15:22
yoctozeptolet's not lag the meeting15:22
mgoddardsure. Good info though, keep it up (and others too please help populate it)15:22
yoctozeptoothers random15:22
yoctozeptothis I find bizarre: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'ansible_memtotal_mb'15:22
yoctozeptothis must be cosmic rays effects15:23
mgoddardyeah, weird15:23
yoctozeptolet's go on with the meeting, please, mgoddard15:23
mgoddardok15:23
mgoddardjust to finish off the whiteboard15:23
mgoddardif you are working on a feature or larger piece of work, please keep progress up to date on the whiteboard15:24
mgoddardparticularly for Train priorities15:24
*** henriqueof has joined #openstack-kolla15:24
mgoddard#topic Stein release status15:24
*** openstack changes topic to "Stein release status (Meeting topic: kolla)"15:24
mgoddardI think we're getting close15:24
mgoddardbeen a bit busy the last couple of days, but I think I'm converging on a solution for the galera issues15:25
yoctozeptogreat!15:25
*** dking_desktop has joined #openstack-kolla15:25
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: DNM: try waiting for mariadb to become operational  https://review.opendev.org/66851415:25
mgoddard^ there it is... git review was patiently waiting for a yes/no15:25
yoctozeptomgoddard: loyal servant15:26
mgoddardif that works I'll squash it into the main bugfix patch15:26
mgoddardthe other main issue that concerns me is bifrost upgrades sometimes fail, fixed by https://review.opendev.org/66766515:26
mgoddardalthough we merged a workaround in kayobe so we're at least unblocked there15:27
* yoctozepto yoctozepto concerned about nova discovery too15:27
mgoddardthat's true15:27
* yoctozepto asking whether you prefer him to finish the patch or leave it entirely to you15:28
*** priteau has joined #openstack-kolla15:28
mgoddardI haven't looked at it today, do you think it needs much more work?15:28
yoctozeptonope, not much, straigthen the naming15:29
yoctozeptoand fix ironic15:29
*** Sravan has joined #openstack-kolla15:29
yoctozeptoprobably only regex replace is wrong there15:29
yoctozeptoI linked a test for you15:29
yoctozepto#link https://review.opendev.org/66874215:29
henriqueofWhat's the problem iwth Ironic? Can someone point me the issue link?15:30
henriqueofOh, nevermind, thanks!15:30
openstackgerritgugug proposed openstack/kolla-ansible master: Fix the genconfig subcommand failed  https://review.opendev.org/66877315:30
openstackgerritgugug proposed openstack/kolla-ansible master: Update the UPPER_CONSTRAINTS_FILE to releases.openstack.org  https://review.opendev.org/66877715:30
mgoddardhenriqueof: I think yoctozepto is referring to a problem with ironic in https://review.opendev.org/#/c/66862315:31
yoctozeptomgoddard, henriqueof: exactly15:31
mgoddardstein seems to go on and on15:31
henriqueofmgoddard: Thanks!15:31
mgoddardOTOH I think it will be our best tested release15:31
yoctozeptomgoddard: better that than stagnation15:31
henriqueofI am really looking forward to a Stein deployment in the nex weeks.15:32
mgoddardhenriqueof: great, do feed back any issues15:32
mgoddard#topic IPv6 community goal15:32
*** openstack changes topic to "IPv6 community goal (Meeting topic: kolla)"15:32
mgoddardI added this because yoctozepto offered to pick up the IPv6 work15:33
henriqueofWell, it'll be a production so I need to know what problems I am expected to find,15:33
henriqueofBefore deploying.15:33
yoctozeptomgoddard: I did15:33
mgoddard#link https://blueprints.launchpad.net/kolla-ansible/+spec/ipv6-control-plane15:33
*** brtknr has joined #openstack-kolla15:33
henriqueofAlso, I think the best way is using the source from head branch, right?15:33
*** icarusfactor has quit IRC15:34
yoctozeptonot yet tried coordinating with the other projects15:34
*** factor__ has joined #openstack-kolla15:34
mgoddardhenriqueof: for the stein release use stable/stein15:34
*** Sravan has quit IRC15:34
yoctozeptoas for us we rely a lot on ipv4 ansible vars15:34
mgoddardyoctozepto: I think this will be key15:34
yoctozeptoso that's the main issue to overcome15:34
mgoddardalso with Tripleo and OSA15:34
yoctozeptothere was a patch for fallback15:34
yoctozeptocloudnull: TripleO ^15:34
mgoddardcloudnull is well placed in both projects15:34
* cloudnull reading back 15:35
yoctozepto#link https://review.opendev.org/57599415:35
mgoddardcloudnull: IPv6 cross community15:35
henriqueofI wanted to try TripleO but I don't have enough resources now to keep a undercloud running.15:35
yoctozeptomgoddard: no OSA agents around?15:35
mgoddardyoctozepto: cloudnull knows a thing or two about OSA15:36
* cloudnull former agent of OSA 15:36
yoctozeptogeez, you are a double agent already15:36
yoctozeptowe are making you triple now15:36
cloudnulltripleo agent :P15:36
yoctozepto;D15:36
yoctozeptohahahaha15:36
yoctozeptogood one15:36
*** k_mouza has quit IRC15:36
mgoddardI think a mailing list post will be a good idea, perhaps once we know what questions to ask15:37
yoctozeptosure, I can handle the communication15:37
mgoddardcloudnull: do you know what the current state of IPv6 is in Tripleo?15:37
mgoddardor OSA for that matter15:37
cloudnullI do not know what the current state is for TripleO15:38
henriqueofyoctozepto: double agent? That was for who? lol15:38
yoctozeptoguys, I must be going now15:38
*** factor has joined #openstack-kolla15:38
yoctozeptohenriqueof: to cloudnull15:39
*** factor__ has quit IRC15:39
cloudnullOSA supports ipv6 in instances and has for a very long time, IPv6 infra works too, though only for physical host and lxc based deployments15:39
yoctozeptosorry for leaving15:39
yoctozeptowill read backlog15:39
yoctozeptosee ya15:39
mgoddardbye yoctozepto, thanks for attending15:39
cloudnullthe general rule in OSA was to do a dual stack network, I suspect that works for most deployments regardless of tech / deployment15:40
mgoddardI've seen kolla-based deployments with IPv6 for instances, but currently we're tied to v4 for infra15:40
*** k_mouza has joined #openstack-kolla15:40
mgoddardcloudnull: what would you put in the endpoint catalogue? (assuming no DNS)15:41
cloudnullwe'd assume DNS, especially when wanting to use ipv615:41
mgoddardok15:41
cloudnullotherwise [xx:yy:zz] should work ?15:42
cloudnullthough Ive never tried.15:42
mgoddardjust thinking at that point you'd need to choose between v4 or v615:42
henriqueofOkay, so no known critical problems for current stable/stein release, right? I can use for a production deployment, iright?15:43
mgoddardgiven that yoctozepto has left let's move on from IPv615:43
mgoddard#topic open discussion15:43
*** openstack changes topic to "open discussion (Meeting topic: kolla)"15:43
mgoddardlet's start with henriqueof's last question15:43
mgoddard> Okay, so no known critical problems for current stable/stein release, right? I can use for a production deployment, iright?15:44
mgoddardhenriqueof: strictly, kolla hasn't released stein yet15:44
mgoddardhenriqueof: it has a couple of issues we discussed earlier in the meeting15:45
mgoddardthe main one affecting upgrades15:45
mgoddardanyone else have anything to discuss?15:45
mgoddardthanks for attending al15:46
henriqueofmgoddard: Thanks very much, so no official kolla release therefore, unmet depencency for kolla-ansible and that makes a no production ready version.15:46
mgoddardhenriqueof: I expect it to land in the next week, assuming nothing else goes wrong :)15:47
*** michaelbarkdoll has quit IRC15:47
henriqueofOh great! I'll wait then.15:47
mgoddard#emdmeeting15:47
ohwhyosa[m]typo15:50
ohwhyosa[m]mgoddard:15:50
ohwhyosa[m]Didn't want to bother during the meeting, can we comment during them (related stuff, I mean)15:51
mgoddardohwhyosa[m]: if its related, please feel free to get involved15:58
mgoddardohwhyosa[m]: if it's unrelated please either ask in 'open discussion' at the end or wait until afterwards15:58
ohwhyosa[m]Yes! That I meant15:59
ohwhyosa[m]After reading you I believe some of the issues I've had with multinode might be solved with retries16:00
ohwhyosa[m]The deployment attempts mainly failled at bootstraping something16:00
ohwhyosa[m]oh mgoddard I don't know if it may be on purpose, but you wrote #emdmeeting instead of #endmeeting16:01
openstackgerritMerged openstack/kolla master: Blacklist sphinx 2.1.0 (autodoc bug)  https://review.opendev.org/66877916:01
*** k_mouza has quit IRC16:09
*** rpittau is now known as rpittau|afk16:14
*** Wasaac has quit IRC16:17
*** cgrosjean has quit IRC16:17
priteau#endmeeting16:19
*** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:19
openstackMeeting ended Wed Jul  3 16:19:56 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:19
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-07-03-15.02.html16:19
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-07-03-15.02.txt16:20
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-07-03-15.02.log.html16:20
priteauThanks ohwhyosa[m] :-)16:20
*** Wasaac has joined #openstack-kolla16:21
*** Wasaac has quit IRC16:25
mgoddardoops!16:27
*** priteau has quit IRC16:28
*** munimeha1 has quit IRC16:31
* yoctozepto is back, reading log16:35
yoctozeptonot missed much, good16:36
*** Sravan has joined #openstack-kolla16:37
*** nde has joined #openstack-kolla16:39
*** unicell has joined #openstack-kolla16:42
*** unicell has quit IRC16:42
*** henriqueof has quit IRC16:43
yoctozeptomgoddard: checked keystone logs for those CI failed jobs16:51
yoctozeptotwo variants of wsrep failure16:51
yoctozeptoand one general db-related16:51
yoctozeptoadded that next to the links16:52
mgoddardnice, thanks yoctozepto16:54
*** jonaspaulo has joined #openstack-kolla16:54
yoctozeptomgoddard: none are upgrade jobs, so there is a mariadb/galera problem with regular deploy too16:55
mgoddardyoctozepto: boo16:57
yoctozeptoyeah, I know17:00
yoctozeptomgoddard: are you waiting for something particular with mariadb now or just some amount of time?17:00
yoctozeptoI mean - in the cod17:00
*** factor has quit IRC17:03
*** openstackgerrit has quit IRC17:04
ohwhyosa[m]hey yoctozepto what info do you want me to put on launchpad? I asked before but forgot to mention you17:05
yoctozeptoohwhyosa[m]: no problem, just the regular stuff: error message, your configs, maybe the relevant logs (docker, service)17:06
yoctozeptoconfig is globals.yml, the inventory and any custom things that are probably related17:06
ohwhyosa[m]good, I'm first trying to redeploy after clean installing ubuntu server on the 4 nodes, but i believe I have the logs config and info on pastebin still17:08
yoctozeptothe more you are sure of the bug, the better17:08
ohwhyosa[m]Is it overkill to install ara on the deployment node?17:08
ohwhyosa[m]Which also we were using as compute, control and network, which might be overkill17:08
ohwhyosa[m]I'm not sure it was a bug as much as a problem with my config, to be fair17:09
ohwhyosa[m]though it seemed to crash always on bootstraping stages, which might be due to the retrying thingie talked about during the meeting17:09
yoctozeptoohwhyosa[m]: we don't test deployment of all the services though17:11
yoctozeptoso for non-main ones the chance of bugs lurking is higher17:11
yoctozeptoceilometer/gnocchi is definitely not main17:12
ohwhyosa[m]I went overboard with the services, put in pretty much everything that seemed interesting (mainly logging/monitoring stuff)17:12
ohwhyosa[m]plus the opendaylight and some stuff17:12
ohwhyosa[m]By the way, in the docs, should I bootstrap or do as indicated in the multinode specific page?17:13
ohwhyosa[m]I tried doing both and conflicts between the systemd file and the daemon.json17:13
ohwhyosa[m]Docker doesn't like being told what to do twice xD17:14
yoctozeptoohwhyosa[m]: bootstrap mostly ensures that docker is installed and running as we would like it to17:14
yoctozeptoohwhyosa[m]: share that as a bug too17:14
yoctozeptoif you are not doing something utterly odd17:14
ohwhyosa[m]perfect!17:14
yoctozeptoand k-a fails17:14
yoctozeptothen report17:15
yoctozeptowe can sometimes do something ;-)17:15
ohwhyosa[m]so when bootstraping I don't need to actually install docker myself, right?17:15
*** munimeha1 has joined #openstack-kolla17:15
ohwhyosa[m]I do have to install some python reqs but not the docker stuff?17:15
ohwhyosa[m]yoctozepto: so far you guys have gone well beyond the call of duty17:16
ohwhyosa[m]Well, I say guys, but guys and gals, so you people17:17
yoctozeptoohwhyosa[m]: exactly, docker is handled for you17:28
yoctozeptoyeah, python and ansible on deployer17:28
*** k_mouza has joined #openstack-kolla17:35
*** k_mouza_ has joined #openstack-kolla17:36
*** k_mouza has quit IRC17:36
*** k_mouza_ has quit IRC17:44
*** igordc has quit IRC17:45
*** luksky11 has quit IRC17:46
*** Sravan has quit IRC17:56
ohwhyosa[m]and on the hosts, asumming it is an ubutnu 18 system17:58
ohwhyosa[m]that or ansible_python_interpreter17:58
ohwhyosa[m]in the inv17:58
*** openstackgerrit has joined #openstack-kolla17:59
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66896717:59
*** BjoernT has quit IRC17:59
*** Sravan has joined #openstack-kolla18:01
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66896718:04
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66897118:20
*** Sravan has quit IRC18:22
*** BjoernT has joined #openstack-kolla18:22
yoctozepto^ if you ever notice cr+2 v+1 w+1 change that is not gated, do a recheck; merging the above commits should prevent such situation18:23
*** Sravan has joined #openstack-kolla18:23
*** kplant has quit IRC18:26
*** lemko has quit IRC18:40
*** luksky11 has joined #openstack-kolla18:42
*** KeithMnemonic has joined #openstack-kolla18:51
*** igordc has joined #openstack-kolla19:05
*** whoami-rajat has quit IRC19:12
*** Sravan has quit IRC19:20
openstackgerritMerged openstack/kolla-ansible master: CI: Use template-overrides.j2 from kolla  https://review.opendev.org/66797119:22
openstackgerritMerged openstack/kolla-ansible master: Update the UPPER_CONSTRAINTS_FILE to releases.openstack.org  https://review.opendev.org/66877719:28
*** whoami-rajat has joined #openstack-kolla19:29
*** igordc has quit IRC19:31
*** henriqueof has joined #openstack-kolla19:33
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: CI: Use template-overrides.j2 from kolla  https://review.opendev.org/66898319:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: CI: Use template-overrides.j2 from kolla  https://review.opendev.org/66898319:35
*** kplant has joined #openstack-kolla19:44
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Collect docker and systemd configs  https://review.opendev.org/66844519:44
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Collect docker and systemd configs  https://review.opendev.org/66844519:45
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Wait for all compute services before cell discovery  https://review.opendev.org/66862319:46
*** hamzaachi has joined #openstack-kolla19:48
*** Sravan has joined #openstack-kolla19:50
*** lemko has joined #openstack-kolla19:52
*** dciabrin has quit IRC19:53
*** Sravan has quit IRC19:54
*** Sravan has joined #openstack-kolla20:16
*** Sravan has quit IRC20:20
*** goldyfruit has quit IRC20:24
*** goldyfruit has joined #openstack-kolla20:24
*** munimeha1 has quit IRC20:27
*** Sravan has joined #openstack-kolla20:30
*** nde has quit IRC20:39
*** goldyfruit has quit IRC20:47
*** goldyfruit has joined #openstack-kolla21:02
*** igordc has joined #openstack-kolla21:14
*** pcaruana has quit IRC21:16
openstackgerritMerged openstack/kolla-ansible master: CI: set the same gate queue for kolla and kolla-ansible  https://review.opendev.org/66896721:21
*** schwicht has quit IRC21:31
*** Sravan has quit IRC21:35
*** Sravan has joined #openstack-kolla21:37
*** Sravan has quit IRC21:39
*** Sravan has joined #openstack-kolla21:39
*** henriqueof has quit IRC21:43
*** Sravan has quit IRC21:58
*** Sravan has joined #openstack-kolla21:59
*** schwicht has joined #openstack-kolla21:59
*** lemko has quit IRC22:00
*** schwicht has quit IRC22:12
*** schwicht has joined #openstack-kolla22:14
*** Sravan has quit IRC22:16
*** cgrosjean has joined #openstack-kolla22:18
*** jonaspaulo has quit IRC22:21
*** Sravan has joined #openstack-kolla22:25
*** Sravan has quit IRC22:29
*** whoami-rajat has quit IRC22:42
*** hamzaachi has quit IRC22:48
*** luksky11 has quit IRC22:48
*** schwicht has quit IRC22:54
*** owalsh_ has joined #openstack-kolla23:10
*** owalsh has quit IRC23:10

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!