Wednesday, 2019-06-19

*** gfhellma_ has joined #openstack-kolla00:07
*** gfhellma_ has quit IRC00:09
*** zhanglong has quit IRC00:17
*** gregoryo has joined #openstack-kolla00:25
*** spiette has quit IRC00:29
*** spiette has joined #openstack-kolla00:31
*** zhanglong has joined #openstack-kolla00:40
*** kplant has quit IRC00:50
*** igordc has quit IRC01:16
*** spiette has quit IRC01:21
*** spiette has joined #openstack-kolla01:23
*** zhanglong has quit IRC01:30
*** zhanglong has joined #openstack-kolla01:34
*** guozijn has joined #openstack-kolla01:53
*** zhanglong has quit IRC01:54
*** hongbin has joined #openstack-kolla01:58
openstackgerritZijian Guo proposed openstack/kolla-ansible stable/rocky: Update service_provider for FWaaS v2  https://review.opendev.org/66620002:01
*** whoami-rajat has joined #openstack-kolla02:08
*** Sravan has joined #openstack-kolla02:36
*** Sravan has quit IRC02:40
*** guozijn has quit IRC03:18
*** guozijn has joined #openstack-kolla04:09
*** Sravan has joined #openstack-kolla04:21
*** guozijn has quit IRC04:22
*** hongbin has quit IRC04:25
*** Sravan has quit IRC04:31
openstackgerritTatsuma Matsuki proposed openstack/kolla-ansible master: Add logrotate configuration for horizon  https://review.opendev.org/66621404:33
openstackgerritMerged openstack/kolla-ansible master: Remove heat_watch_server_url configuration option  https://review.opendev.org/66518504:39
*** harsha has quit IRC04:42
*** Sravan has joined #openstack-kolla04:44
*** Sravan has quit IRC04:50
*** Sravan has joined #openstack-kolla05:07
*** spiette has quit IRC05:27
*** yankcrime has quit IRC05:27
*** timss has quit IRC05:28
*** spiette_ has joined #openstack-kolla05:28
*** timss has joined #openstack-kolla05:28
*** _Cyclone_ has quit IRC05:30
*** irclogbot_3 has quit IRC05:30
*** irclogbot_3 has joined #openstack-kolla05:31
*** mgoddard has quit IRC05:31
*** _Cyclone_ has joined #openstack-kolla05:33
*** mgoddard has joined #openstack-kolla05:33
*** skramaja has joined #openstack-kolla05:42
*** Sravan has quit IRC05:47
*** guozijn has joined #openstack-kolla05:50
*** Luzi has joined #openstack-kolla05:50
gregoryoI wonder why, but neither of my two keepalived containers had MASTER state. Perhaps because an even number is a bad idea?05:59
gregoryoManually changing one to MASTER and restarted the service in that container made the VIP start responding.05:59
*** guozijn has quit IRC06:03
*** cah_link has joined #openstack-kolla06:04
*** zhangfei has joined #openstack-kolla06:04
*** spsurya has joined #openstack-kolla06:16
yoctozeptomorning06:16
yoctozeptogregoryo: nope, keepalived is fine with even numbers06:16
gregoryoOdd then that neither got MASTER from a deployment.06:17
yoctozeptoodd indeed06:17
gregoryoExcuse the pun.06:17
yoctozeptoI have seen master-master due to firewall issues06:18
yoctozeptobut not the other way06:18
gregoryoThankfully a reconfigure run (to enable LDAP - that was so easy!) didn't break it.06:18
yoctozeptothe pun was fine06:18
yoctozeptoyou enabled ldap in a separate domain I presume?06:18
gregoryoThis was BACKUP in both.06:18
yoctozeptoyeah, that's the other state06:19
yoctozeptoI think there is also an INIT state and a FAILURE state06:19
gregoryoYeah in /etc/keystone/domains/keystone.ourdomain.conf06:19
yoctozeptobut these are rare06:19
yoctozeptonext time check logs, maybe there was a third one :P06:20
yoctozeptokeepalived is eager to become MASTER06:20
gregoryoI love the Kolla’s Deployment Philosophy page. Markedly different - and better - compared to our previous deployments with Fuel and Puppet OpenStack.06:20
yoctozeptooh, yeah, exact reason why I am not doing tripleo, osa or whatever06:21
gregoryoWould you be able to help me understand (and then articulate to colleagues!) how OSA is different in this regard?06:22
*** dpawlik has joined #openstack-kolla06:22
*** quoth_ has quit IRC06:23
yoctozeptowell, sure06:24
yoctozeptolast time I checked they went the road of templating all the stuff06:24
yoctozeptoso they curate their own way of configuring ansible roles for deploying openstack06:25
yoctozeptoalso, they didn't use docker at the time06:25
gregoryoRight, a lot like Puppet OpenStack then.06:25
yoctozeptobut LXC I presume?06:25
gregoryoAh so they've switched to Docker? Yes, LXC was what I read.06:25
yoctozeptoor baremetal was an option too06:25
yoctozeptoI don't know now06:25
yoctozeptonot following06:25
yoctozeptoyeah, it was probably meant to be06:26
yoctozeptopuppet replacement06:26
yoctozeptoas ansible is pushed to replace puppet more and more06:26
yoctozeptowhich I find good because it is actually saner (though I still like puppet in general)06:26
gregoryoI started reading https://www.packtpub.com/au/virtualization-and-cloud/openstack-cloud-computing-cookbook-fourth-edition and it says LXC - published Jan 2018.06:27
*** threestrands has joined #openstack-kolla06:27
yoctozeptoI would go read OSA's latest docs to confirm, seen many books claim false but this is probably true06:28
gregoryoI think Puppet is probably better for managing pets, Ansible a bit more cattle-focused. Lots of overlap of course, but higher overhead to start using Puppet.06:29
yoctozeptohttps://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/overview.html06:29
yoctozeptolxc06:29
gregoryoAnyway, the templating angle is important. It's a benefit I hadn't even realised we would get out of Kolla.06:30
yoctozeptogregoryo: yeah, you might be right but the model in general is reversed as puppet is usually left to run in background periodically checking the inventory06:30
yoctozeptotrue, this way kolla is  simpler too06:30
yoctozeptonot fixing every little config issue06:31
yoctozepto:D06:31
gregoryoCuration and scaling win. Let upstream defaults reign, change what you need to in only one place. Keep the deployment tool out of the way where possible, and make less work for its maintainers.06:31
yoctozeptoas if I was hearing myself!06:33
gregoryoHmmm, https://docs.openstack.org/openstack-ansible/latest/ says Stein is in pre-release. If true, that is a telling example of the extra curation work at each release.06:33
yoctozeptoheh, we are no better here06:33
yoctozeptoours is too06:33
gregoryoHaha fair enough.06:33
yoctozeptothough it's fairing completion06:33
yoctozeptowe still can't release as soon as other projects do06:34
yoctozeptoas we depend on them06:34
yoctozeptoto stabilize first06:34
gregoryoWe haven't got to rocky yet ):06:34
yoctozeptoand the team is actually small as you can see06:34
yoctozeptobut I have Stein working in evaluation here locally06:34
yoctozeptovia kolla06:34
yoctozeptocore services deploy fine06:35
gregoryoHow about heat and sahara?06:35
yoctozeptoI think heat works but have not used sahara06:35
gregoryoOkay thanks. I'll stay tuned.06:36
yoctozeptowell, you can try yourself :D06:36
yoctozeptoI have sahara on my deploy plans too06:36
gregoryoI have too much to try already just in getting a Kolla deploy to look like our production environment!06:37
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Remove heat_watch_server_url configuration option  https://review.opendev.org/66623206:37
yoctozeptogregoryo: I don't have real production yet so it's one less thing to worry about 8-)06:38
gregoryoWe're over 2 years in prod now, Newton through to Queens.06:42
gregoryoHad some issues in January which confirmed wanting to get off our snowflake Fuely Puppety arrangement.06:43
yoctozeptoI see, but is the project still developed?06:45
yoctozeptoI mean fuel06:45
gregoryoNope.06:46
gregoryoI mean, maybe, but not so much that we notice. We turned off our Fuel controller about a year ago and moved to Puppet OpenStack. That went well initially but that project feels like it's fading too.06:47
yoctozeptono wonder, as the whole puppet scene is06:49
gregoryoEmilien Macchi I think was developing it and was a great help, but RedHat bought Ansible, and apparently are retooling TripleO from Puppet to Ansible, so that momentum will die or has died.06:50
*** guozijn has joined #openstack-kolla06:50
yoctozeptoyeah, they are doing this now06:51
yoctozeptothough they are also relaying on custom config and heat06:51
yoctozeptorelying* which I find the wrong approach to config06:54
*** rpittau|afk is now known as rpittau07:02
*** igordc has joined #openstack-kolla07:05
*** luksky has joined #openstack-kolla07:06
*** ivve has joined #openstack-kolla07:07
*** igordc has quit IRC07:10
*** luksky has quit IRC07:11
mnasiadkamorning07:15
mnasiadkayoctozepto: do you ever sleep? :)07:15
yoctozeptomnasiadka: nope, sleep is for the weak :D07:16
yoctozeptomorning ;-)07:16
mnasiadkayoctozepto: then I'm weak :)07:16
yoctozepto:P07:17
yoctozeptoI sleep 6-8 hours a day07:17
yoctozeptotoday patching our servers against SACK vuln07:19
yoctozeptoturned out you can use SACK cap to bring down servers to knees07:19
mnasiadkamy blessing that I work in R&D...07:21
*** luksky has joined #openstack-kolla07:23
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Test blazar in nfv scenario  https://review.opendev.org/65146307:26
yoctozeptomnasiadka: https://review.opendev.org/66592707:36
yoctozeptoMark left it for you to decide07:36
yoctozeptolet's unbreak this task today ;D07:37
yoctozeptoneeds backport to stein07:37
openstackgerritRadosław Piliszek proposed openstack/kolla master: Fix deployment of ceph-nfs container failing due to wrong USER  https://review.opendev.org/66623907:48
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Test ceph-nfs as well when testing ceph  https://review.opendev.org/66624007:51
*** ktibi has joined #openstack-kolla07:58
*** threestrands has quit IRC07:59
*** gkadam has joined #openstack-kolla08:00
*** dougsz has joined #openstack-kolla08:01
mnasiadkamgoddard: https://review.opendev.org/#/c/665927 - check my comment and let's agree on a way forward :)08:02
mgoddardmnasiadka: you may be right about the script, but it does not take too long to do manually, and I think it's good to inspect patches first08:04
mgoddardmnasiadka: about stable policy, IMO we should backport bug fixes to all active stable branches (Q,R,S) where it makes sense08:05
mgoddardmnasiadka: I have been hit many times downstream by a too relaxed backport policy08:05
mgoddardmnasiadka: it's very annoying to hit a bug that has been fixed but not backported08:05
mgoddardthere may be some grey lines around the word 'bug' here, those can be discussed per-patch08:06
*** nathharp has joined #openstack-kolla08:06
openstackgerritDoug Szumski proposed openstack/kolla-ansible stable/stein: Remove obsolete roles middleware  https://review.opendev.org/66624208:08
openstackgerritDoug Szumski proposed openstack/kolla-ansible stable/rocky: Remove obsolete roles middleware  https://review.opendev.org/66624308:09
mgoddardmnasiadka: replied on https://review.opendev.org/#/c/66592708:09
*** Wasaac has joined #openstack-kolla08:15
*** Wasaac has quit IRC08:15
*** Wasaac has joined #openstack-kolla08:16
*** k_mouza has joined #openstack-kolla08:16
*** Wasaac has quit IRC08:17
*** Wasaac has joined #openstack-kolla08:17
*** gregoryo has quit IRC08:19
*** Wasaac has quit IRC08:20
*** Wasaac has joined #openstack-kolla08:21
*** pcaruana has quit IRC08:27
*** zhanglong has joined #openstack-kolla08:35
yoctozeptomgoddard, mnasiadka: re:backport-policy I consider stein to be active so everything not related to train specifically or implementing completely new features should go there still; older branches should be fix only08:44
*** pcaruana has joined #openstack-kolla08:45
mnasiadkayoctozepto: so, approved the ceph crap - do you want to work on the final solution, or should I add this to my pile of stuff?:)08:45
yoctozeptomnasiadka: can we coordinate this somehow? I can have it on me, sounds good for flexing ansible muscles08:46
mnasiadkayoctozepto: sure, take this - if you have any questions just ask :)08:46
yoctozeptomgoddard: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2019-06-18.log.html#t2019-06-18T18:40:54 <- horizon, can't see the guy's report on launchpad, he indeed minded that08:46
mgoddardneed to head out, will be back in 2 hours08:47
yoctozeptomnasiadka: I get it you want to do the other side limit too after checking the clients, though this probably conflicts with doing an update on a running system08:48
yoctozeptoas VMs will be old clients08:48
yoctozeptoso while it will work in CI, we end up with bad experience for users08:49
yoctozeptoand the other fix will result in better error message than EPERM in case we miss to upgrade something :-)08:49
yoctozeptoso this part is +1 from me08:49
mnasiadkayoctozepto: old clients means libvirtd normally, we should check if after libvirtd upgrade the ceph connection uses old client proto or new one08:49
yoctozeptothe other is -108:49
yoctozeptomnasiadka: qemu stays anyway08:50
yoctozeptowill ceph like this limit then?08:50
mnasiadkayoctozepto: if old one - we should document what to do after upgrade08:50
yoctozeptoyup, docs +108:50
yoctozeptoI think we should only improve the check for servers and add to docs about clients08:51
mnasiadkaYeah, let’s start with that08:51
yoctozepto+108:51
yoctozeptomnasiadka: I left a note of our discussion: https://review.opendev.org/66592708:55
*** Dubla1 has joined #openstack-kolla08:56
*** k_mouza has quit IRC09:01
*** k_mouza has joined #openstack-kolla09:01
*** lemko has joined #openstack-kolla09:02
*** luksky has quit IRC09:24
*** guozijn has quit IRC09:32
*** gfidente has joined #openstack-kolla09:33
*** k_mouza_ has joined #openstack-kolla09:33
yoctozeptolooks like nfs ganesha is broken anyway in Ubuntu http://logs.openstack.org/30/666030/1/check/kolla-ansible-ubuntu-source-ceph/b705211/primary/logs/kolla/ceph/ceph-nfs.txt.gz09:34
*** k_mouza has quit IRC09:37
*** igordc has joined #openstack-kolla09:37
yoctozeptomgoddard, mnasiadka: is it on purpose that not every service has its entry in fluentd? seems documented otherwise (as if fluentd had all the logs)09:42
*** janki has joined #openstack-kolla09:43
*** k_mouza_ has quit IRC09:49
*** k_mouza has joined #openstack-kolla09:50
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix placement log perms in config.json  https://review.opendev.org/66614609:56
openstackgerritMerged openstack/kolla-ansible master: Make Ceph upgrade check Ceph release to avoid EPERM  https://review.opendev.org/66592709:56
openstackgerritMerged openstack/kolla-ansible stable/queens: Add blazar to fluentd aggregation  https://review.opendev.org/66613909:56
*** k_mouza has quit IRC10:02
*** luksky has joined #openstack-kolla10:07
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix issue finding custom, host specific plugins  https://review.opendev.org/66614210:10
*** k_mouza has joined #openstack-kolla10:17
*** k_mouza has quit IRC10:17
*** k_mouza has joined #openstack-kolla10:18
*** sshnaidm is now known as sshnaidm|afk10:24
*** JqckB has joined #openstack-kolla10:25
*** ktibi has quit IRC10:26
*** k_mouza has quit IRC10:27
*** k_mouza has joined #openstack-kolla10:28
*** dave-mccowan has joined #openstack-kolla10:32
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Make Ceph upgrade check Ceph release to avoid EPERM  https://review.opendev.org/66626010:33
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/stein: Make Ceph upgrade check Ceph release to avoid EPERM  https://review.opendev.org/66626010:34
*** k_mouza has quit IRC10:36
*** k_mouza has joined #openstack-kolla10:36
*** k_mouza has quit IRC10:39
*** Wasaac has quit IRC10:45
*** priteau has joined #openstack-kolla10:46
*** zhanglong has quit IRC10:47
*** Wasaac has joined #openstack-kolla10:47
*** spiette_ has quit IRC10:50
mgoddardyoctozepto: I'm guessing from the py3 comment igordc is using ubuntu. hrw recently made some changes for it recently, including apache config. I wonder if those were in the tested code11:04
mgoddardyoctozepto: testing horizon would definitely be a good idea. I wonder if a simple curl would have caught this11:04
igordcmgoddard, for the case when the horizon container doesn't have the expected python27 script path a simple curl would catch easily (just gives a 404)11:06
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Use become for all docker tasks  https://review.opendev.org/66614011:07
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Check for 'become' in tasks that use Docker in pep8  https://review.opendev.org/66614111:07
openstackgerritDoug Szumski proposed openstack/kolla master: Bump Monasca Fluentd output plugin  https://review.opendev.org/66626511:08
*** kplant has joined #openstack-kolla11:08
igordcthe issues I faced were on sunday11:08
*** ktibi has joined #openstack-kolla11:10
*** zhangfei has quit IRC11:13
yoctozeptomgoddard, igordc: https://review.opendev.org/64237511:15
yoctozeptothough there seem to be no apache-related changes11:15
*** VickyG has quit IRC11:17
igordcyoctozepto, https://review.opendev.org/#/c/642375/28/docker/horizon/extend_start.sh11:20
igordcyoctozepto, this is the path that was pointing to non-existent python2.7: SITE_PACKAGES="/usr/lib/python${KOLLA_DISTRO_PYTHON_VERSION}/site-packages"11:21
*** rgogunskiy has joined #openstack-kolla11:21
igordcyoctozepto, my mistake; it's this one:     SITE_PACKAGES="/var/lib/kolla/venv/lib/python${KOLLA_DISTRO_PYTHON_VERSION}/site-packages"11:22
igordccopied the wrong line11:22
mgoddardigordc: can you confirm you're using ubuntu bionic?11:22
yoctozeptoeh, nice one that11:22
yoctozeptohrw patched binary path11:22
igordcmgoddard, yeah the host is bionic11:23
yoctozeptowhen he wanted the source path11:23
yoctozeptomgoddard: the issue is it's in the wrong branch of src/bin if11:23
yoctozeptoI'm inspecting the patch to see if it's the only place11:24
mgoddardyoctozepto: why is it the wrong branch?11:25
yoctozeptomgoddard: "move to Python 3 for source images"11:25
yoctozeptoif [[ ${KOLLA_INSTALL_TYPE} == "binary" ]]; then11:26
yoctozeptoseems wrong to modify this one instead of the other?11:26
mgoddardoh I see. That's true. The patch was previously a mix of source + binary, I split out binary into a separate patch, and maybe that was left behind11:28
mgoddardalso seems like it was wrong11:28
mgoddardigordc: can you check your images to see which of those paths is correct for binary images?11:28
igordcmgoddard, let me check my notes11:29
yoctozeptoit's the only place in the change that has this oddity11:29
yoctozeptowonder why CI did not catch this11:30
yoctozeptobecause it blocks apache11:31
igordcmgoddard, I didn't save the path for the binary case but I'm pretty sure I saw what's in the patch: /usr/lib/python3/dist-packages11:31
mgoddardigordc: you can just pull down the image11:31
igordcmgoddard, for the source case I was seeing /var/lib/kolla/venv/lib/python2.7/site-packages (horizon.conf) and the same, but python3.6, in the container11:32
igordcneed a few minutes to turn on the machine11:33
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Make fluentd-elasticsearch configuration more robust  https://review.opendev.org/66174711:41
*** ohwhyosa has joined #openstack-kolla11:43
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Enable buffering to file for Monasca logs  https://review.opendev.org/66627111:44
igordcpython: /usr/bin/python2.7 /usr/bin/python /usr/bin/python3.6 /usr/bin/python3.6m /usr/lib/python3.6 /usr/lib/python2.7 /usr/lib/python3.7 /etc/python2.7 /etc/python /etc/python3.6 /usr/local/lib/python2.7 /usr/local/lib/python3.6 /usr/share/python11:50
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: Use td-agent-gem to install fluentd monasca plugin on ubuntu  https://review.opendev.org/66607511:52
igordcmgoddard, /usr/lib/python3.6/site-packages does not exist in my 3day old ubuntu-binary-horizon:master11:53
igordcbut /usr/lib/python3/dist-packages does11:54
mgoddardigordc: ok, but the script checks for SITE_PACKAGES="/usr/lib/python3/dist-packages" on ubuntu11:54
mgoddardok11:54
mgoddardso that change seems sound11:54
igordcand there's python 2.7,3, 3.6, 3.7 dirs11:54
mgoddardigordc: looks like the problem is in kolla ansible11:55
mgoddardgrep -rn 2.7 ansible/roles/horizon/11:55
*** Wasaac has quit IRC11:55
mgoddardigordc: I think we need a distro_python_version variable in kolla ansible11:56
mgoddardlots of references to 2.7 in ansible/11:56
igordcmgoddard, yeah.. that first else is definitely problematic for the source image11:57
igordcmgoddard, I just noticed that all my source:stein images are apparently 2 months old (from my previous attempt)11:58
mgoddardigordc: source:stein is still 2.711:58
igordcmgoddard, I had assumed the docker images would get updated when re-running kolla-ansible11:59
mgoddardbinary:stein is 3.611:59
mgoddardigordc: not if the tag doesn't change. You need to pull new images explicitly in that case11:59
igordcmgoddard, but the 404 failure was in source:master11:59
mgoddardigordc: yeah, probably due to references to 2.7 in k-a12:00
igordcmgoddard, lol I'm surprised my source:stein deployment went so well this time, even internet for vms works this time12:00
igordcmgoddard, is it safe to upgrade all my source:stein images/containers without breaking my current openstack deployment? offline time is fine12:02
mgoddardigordc: it should be, yes. Always worth backing up before an upgrade though12:04
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: Add custom filters for checking services  https://review.opendev.org/66592212:08
openstackgerritMerged openstack/kolla-ansible master: Add logrotate configuration for horizon  https://review.opendev.org/66621412:08
igordcmgoddard, will 'kolla-ansible upgrade'12:11
igordc do the trick after pulling all images?12:11
mgoddardigordc: yes12:11
igordcawesome12:12
*** goldyfruit has quit IRC12:25
yoctozeptomgoddard, mnasiadka: do you know whether Zuul runs kolla-ansible with pipelining or without?12:26
mgoddardyoctozepto: I think without12:27
*** priteau has quit IRC12:27
mgoddardI don't think we touch ansible.cfg12:27
yoctozeptoyeah, we are not12:27
yoctozeptoI was wondering whether Zuul installs some default12:28
yoctozeptoput it on my todo to check12:28
*** sshnaidm|afk is now known as sshnaidm12:29
*** Wasaac has joined #openstack-kolla12:31
openstackgerritMerged openstack/kolla-ansible stable/rocky: Fix mongo command for checking replication status  https://review.opendev.org/66614512:34
openstackgerritMerged openstack/kolla-ansible stable/stein: Elevate privileges for copying gnocchi policy.json  https://review.opendev.org/66611912:34
openstackgerritMerged openstack/kolla-ansible stable/rocky: Elevate privileges for copying gnocchi policy.json  https://review.opendev.org/66612012:34
*** Wasaac has quit IRC12:37
*** Wasaac has joined #openstack-kolla12:48
*** dciabrin has quit IRC12:49
*** rgogunskiy has quit IRC12:49
*** Wasaac has quit IRC12:53
*** rgogunskiy has joined #openstack-kolla12:55
*** rgogunskiy has quit IRC13:03
*** zhanglong has joined #openstack-kolla13:04
*** Wasaac has joined #openstack-kolla13:05
*** Wasaac has quit IRC13:10
*** skramaja has quit IRC13:12
*** Wasaac has joined #openstack-kolla13:20
*** Wasaac has quit IRC13:24
*** spiette has joined #openstack-kolla13:24
*** schwicht has joined #openstack-kolla13:25
*** goldyfruit has joined #openstack-kolla13:26
*** Wasaac has joined #openstack-kolla13:27
ohwhyosaHey there! Kolla ansible is failling in the "Checking if kolla_internal_vip_address_is in the same network as api on all nodes"13:28
ohwhyosaIt is an all-in-one deployment13:29
ohwhyosaAnd indeed it should be because its value is {{network_interface}}13:29
ohwhyosaNevermind, think i got it13:30
*** lemko has quit IRC13:32
ohwhyosabut now it does fail with the ping to itself13:32
ohwhyosawhich it does perfectly13:33
*** zhanglong has quit IRC13:34
*** root____ has joined #openstack-kolla13:34
*** root____ has left #openstack-kolla13:34
*** cah_link has quit IRC13:40
*** k_mouza has joined #openstack-kolla13:41
*** scottsol has joined #openstack-kolla13:49
*** scottsol has left #openstack-kolla13:54
*** mchlumsky has joined #openstack-kolla13:54
*** janki has quit IRC13:57
*** rouk has joined #openstack-kolla13:58
mgoddardohwhyosa: did you fix the ping issue?13:59
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix Blazar Nova aggregate in multi-region setup  https://review.opendev.org/66613113:59
openstackgerritMerged openstack/kolla-ansible stable/stein: Remove heat_watch_server_url configuration option  https://review.opendev.org/66623213:59
openstackgerritMerged openstack/kolla-ansible master: Fix default deployment of freezer, use mariadb.  https://review.opendev.org/66148513:59
openstackgerritMerged openstack/kolla-ansible stable/rocky: Remove obsolete roles middleware  https://review.opendev.org/66624313:59
ohwhyosamgoddard, nope! (i did kinda recklessly jump into deply, though, and now it has trouble restarting mariadb master, failling on the wait for it handler)14:01
mgoddardohwhyosa: if you want to start again at any point, use kolla-ansible destroy14:01
mgoddardohwhyosa: one big benefit of containers - you can clean up and start again :)14:01
mgoddard(although don't use that in production :) )14:02
*** gardenshed has joined #openstack-kolla14:04
*** dciabrin has joined #openstack-kolla14:05
*** dpawlik has quit IRC14:11
ohwhyosamgoddard, Nice! I'll take that into account, only use in production if you hate your clients and have a passport at hand hahahaha14:14
mgoddardohwhyosa: true :)14:14
ohwhyosa--yes-i-really-really-mean-it had me rofl14:15
ohwhyosaalso because i skipped the second really the first time14:15
ohwhyosaso i had to really really mean it14:15
ohwhyosamgoddard, it should all be executed as sudo, with the aio right?14:17
ohwhyosaElse it throws a bunch of permission denieds14:17
mgoddardohwhyosa: it should work without sudo14:18
ohwhyosaAAAAAnd now the bootstrap fails also u.u14:18
ohwhyosaThis one might have been due to the sudo14:19
mgoddardohwhyosa: ubuntu bionic? which kolla-ansible branch?14:19
ohwhyosaHmmm so -K14:19
ohwhyosaubuntu bionic, installed with pip14:19
ohwhyosaas in sudo pip install kolla-ansible14:20
kplantwhich openstack release are you trying to install?14:21
ohwhyosanow i miserably failed because of another permission denied (etc/hosts) in bootstrap servers! No -K option? (tried and it seems not to recognize it)14:21
ohwhyosastein14:21
kplantiirc kolla-ansible 7.1 is available in pypi14:22
kplantand that supports up to rocky14:22
kplantunless you did ==8.0.0rc114:22
ohwhyosaI didn't, lemme check the version14:22
kplantit's probably 7.1.114:22
*** ivve has quit IRC14:23
ohwhyosa--version doesn't seem to be a valid param either14:23
ohwhyosaI'll reinstall with 8 anycase14:23
ohwhyosathank you!14:23
kplantpip freeze | grep kolla-ansible14:24
*** scottsol has joined #openstack-kolla14:25
ohwhyosakplant, s*** i knew that, thanks14:25
kplantif you want 8.x from pypi make sure you also install kolla==8.0.0.0rc114:25
kplantnot just kolla-ansible==8.0.0.0rc114:25
ohwhyosaHmmm it seems the playbooks may have disabled/changed my eno1 ip?14:25
ohwhyosasolved now, but i found it funny, after "some" of the executions it seems to have left a single address hahahah14:26
*** JamesBenson has joined #openstack-kolla14:26
ohwhyosakplant, thanks! I'll upgrade both14:26
kplantmgoddard: hey speaking of 8.x .... any guestimate on the next rc?14:28
*** Wasaac has quit IRC14:29
ohwhyosastill permission denied with the hosts14:30
ohwhyosaI'll just add an ansible_became=true to the inventory14:30
ohwhyosaI have it passwordless so it should work14:30
*** JamesBen_ has joined #openstack-kolla14:33
*** Wasaac has joined #openstack-kolla14:33
*** Luzi has quit IRC14:36
*** JamesBenson has quit IRC14:36
mgoddardkplant: still plagued by ceph issues :( I think we're getting close though14:42
*** gfhellma has joined #openstack-kolla14:45
kplanti see you at least have the nautilus ceph clients working now14:46
*** dking_desktop has quit IRC14:49
*** luksky has quit IRC14:50
mgoddardkplant: yes, we switched to nautilus using upstream ceph packages, but this caused issues for tripleo so now switching back to centos ceph, but blocked on them making a release...14:52
mgoddard== Meeting in 10 minutes ==14:52
mgoddard^ mgoddard mnasiadka hrw egonzalez14:52
kplantmgoddard: booo14:59
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Jun 19 15:00:30 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
mgoddard\o15:00
scottsolo/15:00
JamesBen_\o15:01
gardenshedo/15:01
mloza\o/15:01
mgoddardmnasiadka: around?15:02
mnasiadkamgoddard: sort of :)15:03
mnasiadkao/15:03
mgoddard:)15:03
stackedsaxI thought I was an hour late -- just starting?15:03
mgoddardhi stackedsax yeah15:03
mgoddard#topic agenda15:03
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:03
mgoddard* Roll-call15:03
mgoddard* Announcements15:03
mgoddard** Last call for priorities https://etherpad.openstack.org/p/kolla-train-priorities15:03
mgoddard* Review action items from last meeting15:03
mgoddard* Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:03
mgoddard* Stein release status15:03
mgoddard* Kolla Ansible performance https://etherpad.openstack.org/p/kolla-ansible-speed-ups15:03
mgoddard* (vishwanathj): Elasticsearch15:04
mgoddard* Octavia usability improvements15:04
mgoddard* Vote: Kayobe to become a kolla deliverable?15:04
mgoddard#topic announcements15:04
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:04
mgoddard#info Last call for kolla train priorities15:04
mgoddard#info Last call for kolla train priorities15:04
mgoddard#link https://etherpad.openstack.org/p/kolla-train-priorities15:04
mgoddardplease cast your votes - no need to be core15:04
mgoddardAny other announcements?15:04
*** ktibi has quit IRC15:04
mgoddard#topic Review action items from last meeting15:05
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:05
mgoddardmgoddard to try again again to disable ocata publishing job15:05
mgoddardI finally did it!15:05
mgoddardTurns out we needed to remove the job definitions on the stable/ocata branch, rather than using a branch filter on master15:06
mgoddard#link https://review.opendev.org/66574215:06
mgoddardNeeds reviews15:06
mgoddardAnd depends on https://review.opendev.org/66593515:06
mgoddard#topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:06
*** openstack changes topic to "Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard (Meeting topic: kolla)"15:06
mgoddardChecking CI status15:07
mgoddardMostly looks happy15:08
mgoddardI went through and backported all bug fixes in the last month yesterday15:10
mgoddardlots to review15:10
mgoddardAnyone have anything else from the whiteboard?15:10
mgoddardAny other features we should be trackng on there?15:10
mgoddardAdded CentOS/RHEL815:11
mgoddard#topic Stein release status15:12
*** openstack changes topic to "Stein release status (Meeting topic: kolla)"15:12
mgoddardWe switched to Ceph nautilus using upstream ceph packages, but this caused issues for tripleo so now we are switching back to CentOS ceph packages15:12
mgoddardHowever this is blocked on an update to those packages15:12
mgoddardThis is the one to watch:15:13
mgoddard#link https://review.opendev.org/#/c/666015/15:13
mgoddardAny questions on Stein?15:14
mgoddard#topic Kolla Ansible performance https://etherpad.openstack.org/p/kolla-ansible-speed-ups15:14
*** openstack changes topic to "Kolla Ansible performance https://etherpad.openstack.org/p/kolla-ansible-speed-ups (Meeting topic: kolla)"15:14
mgoddard#link https://etherpad.openstack.org/p/kolla-ansible-speed-ups15:14
mgoddardThis was added by klindgren who has been experiencing performance issues at a scale of around 400 hypervisors15:15
mgoddardWho is interested in kolla-ansible at large scale?15:15
mnasiadkaMe15:15
mnasiadkaDo we have any ARA recordings of the run with 400 hypervisors?15:16
gardenshedme15:16
mgoddardmnasiadka: I'm not aware of any. I don't know if they'd be happy to share but we could ask15:17
mgoddardI'd like to see results from running with the profile_tasks callback plugin15:17
mgoddardI see a bunch of people on the Etherpad15:18
*** gfidente has quit IRC15:18
mgoddardDoes anyone have experience with running kolla ansible at scale? Or even just ansible?15:18
*** gkadam has quit IRC15:19
mnasiadkaAnsible yes, but no crazy huge roles15:19
mgoddardIf you have some good tips or ideas, please add to the etherpad15:20
mnasiadkaFirst we need to understand what actions are problematic (templating?)15:20
mgoddardYes, I think templating is hitting us hard15:21
mgoddardThere are certain patterns that kolla-ansible uses, and we should try to profile those in isolation15:21
mgoddardI'd like to try to get interested parties to join a call at some point to discuss this15:22
mgoddardscottsol, stackedsax - interested?15:22
scottsolyes I can join that15:22
stackedsaxI'd like to focus on the TLS stuff for now15:22
mgoddardstackedsax: sure15:22
scottsolI can look over what we have done already and update the etherpad15:23
scottsolits not "huge" scale yet but we already started to see some bottlenecks even with 20-30 nodes15:23
scottsoljust some minor changes have already started to cut it down15:24
mgoddardthat would be helpful15:26
mgoddardlet's keep it to the etherpad for now15:26
mgoddardany other thoughts?15:26
mgoddard#topic (vishwanathj): Elasticsearch15:28
*** openstack changes topic to "(vishwanathj): Elasticsearch (Meeting topic: kolla)"15:28
* jroll rolls in late, sorry15:28
mgoddardhi jroll15:28
mgoddardvishwanathj who proposed this topic is not here15:29
mgoddard#topic Octavia usability improvements15:29
*** openstack changes topic to "Octavia usability improvements (Meeting topic: kolla)"15:29
mgoddardrm_work: around?15:29
jrollI won't promise he's not, but he usually isn't this time of day :)15:30
mgoddardThis was something that came up at the Denver summit kolla feedback session - Octavia is quite hard to deploy in kolla and requires some manual effort15:30
mgoddard#link https://etherpad.openstack.org/p/DEN-train-kolla-feedback15:30
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Enable InfluxDB TSI by default  https://review.opendev.org/66631815:31
mgoddardFrom that pad:15:31
mgoddardgenerate certificates is really manual15:31
mgoddarddoesn't support seperate CAs for internal / external15:31
mgoddardSounds useful to have15:31
mgoddardOSA has some interesting bits15:31
mgoddardFor context, rm_work is Octavia PTL and also has kolla experience, so could be in a good position to help improve this situation15:31
stackedsaxthanks for the context on where that came from.15:32
mgoddardscottsol and co have deployed octavia, and have some notes they could share15:32
stackedsaxrm_dev and I chatted because you told us to, mgoddard15:32
mgoddard:)15:32
stackedsaxbut now that I know the context, we might chat again15:32
mgoddardsorry, thought I'd explained but maybe not well enough15:33
mgoddardso what's the best path forward here?15:33
stackedsaxno worries, we had a nice convo, anway.  Just, maybe, not so useful :D15:33
*** gfidente has joined #openstack-kolla15:34
mgoddardrm_work suggested he might not have lots of time immediately, but I'm sure would be able to provide guidance15:34
stackedsaxyeah, that's the sense I got, as well15:34
scottsolwe could start by listing what has to be done manually? maybe another etherpad...?15:34
stackedsaxcertainly, from that pad it seems like it's something that should be addressed within the scope of the TLS everywhere conversation15:35
scottsolwe have some changes that we can push back to fix some bugs / add minor improvments15:35
stackedsaxscottsol: that would be good15:35
stackedsaxmaybe just on the TLS everywhere pad15:35
stackedsaxhttps://etherpad.openstack.org/p/kolla-internal-tls15:35
mgoddardhow about a ML discussion on openstack-discuss? starting with where we're at now, and then looking at gaps15:35
scottsolsure15:35
mgoddardwe could always link to a pad from there, but the ML might be a good way to strike up a cross-project discussion15:36
mgoddardI can kick that off. Interested parties please look out for it and reply15:37
mgoddard#action mgoddard to post to ML about octavia in kolla15:37
mgoddardother thoughts on this?15:38
mgoddard#topic Vote: Kayobe to become a kolla deliverable?15:39
*** openstack changes topic to "Vote: Kayobe to become a kolla deliverable? (Meeting topic: kolla)"15:39
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006901.html15:39
mgoddard^ That post started the discussion about Kayobe becoming an official project15:39
mgoddardThere are two options on the table15:39
mgoddard1. become a deliverable of the Kolla project15:39
mgoddard2. become an official top level OpenStack project15:40
*** k_mouza_ has joined #openstack-kolla15:40
*** gfhellma has quit IRC15:40
mgoddardthe only feedback it received from the kolla team was from hrw who was happy with either approach15:40
mgoddardI think the best path forward is to take a vote. I would not participate15:41
mgoddardDoes that sound reasonable?15:41
scottsolyep :)15:41
chasonagree :)15:42
mgoddardI think the electorate would be the kolla core team, but would certainly be interested to hear opinions from others in the community15:42
mgoddardmnasiadka?15:42
yoctozeptoo/15:42
yoctozeptodid not know you have a meeting today15:42
yoctozeptonot core but opting for 115:43
*** Wasaac has quit IRC15:43
*** k_mouza has quit IRC15:43
yoctozeptoto me it's an extra layer of glue (kolla)15:43
scottsolI think they would fit well together, and it would make it easier for keeping them sync'd. If not they could always be playing cat and mouse to stay aligned.15:44
mnasiadkamgoddard: agree to vote, don’t have an opinion on kayobe atm :)15:44
mgoddardok. I will send an email to openstack-discuss15:44
*** k_mouza_ has quit IRC15:45
mgoddard#action mgoddard to ask openstack-discuss to vote on making kayobe a kolla deliverable15:45
*** k_mouza has joined #openstack-kolla15:45
mgoddardmnasiadka: now is the time to form one then :)15:45
mgoddardAny other thoughts on this?15:46
mgoddard#topic Open discussion15:46
*** openstack changes topic to "Open discussion (Meeting topic: kolla)"15:46
mgoddardAnyone have anything else to discuss today?15:47
*** Wasaac has joined #openstack-kolla15:47
yoctozeptomgoddard: I have one thing - CI errors15:47
mgoddardyoctozepto: go ahead15:47
yoctozeptoif everybody was kind enough to report them to launchpad :-)15:47
yoctozeptosome are random due to Zuul and infra15:48
yoctozeptobut some are random but real15:48
yoctozeptoso I think it's worth the effort15:48
mgoddardyoctozepto: good point15:48
yoctozeptovide nova15:48
mgoddardwe are probably a bit too 'recheck happy'15:48
yoctozeptoyeah, figured that out15:49
mgoddardI have thought about this in the past15:49
*** vishwanathj has joined #openstack-kolla15:49
mgoddardwhat about an etherpad with a taxonomy of CI errors and a count of how many times each has been hit?15:50
yoctozeptoalso in this regard - scope of kolla-ansible testing - it does not try to deploty everything and surely it should not15:50
openstackgerritMerged openstack/kolla-ansible stable/rocky: Fix issue finding custom, host specific plugins  https://review.opendev.org/66614315:50
*** vishwanathj has quit IRC15:50
yoctozepto> <mgoddard> what about an etherpad with a taxonomy of CI errors and a count of how many times each has been hit?15:50
yoctozeptogood idea15:50
mgoddardyoctozepto: want to create one?15:50
yoctozeptothough if it looks legit then straight to launchpad15:50
mgoddardand link to it from the whiteboard?15:50
mgoddard+115:50
yoctozeptomgoddard: sure, need to learn how to :D15:51
* yoctozepto writes it down in todo15:51
yoctozepto> <yoctozepto> also in this regard - scope of kolla-ansible testing - it does not try to deploty everything and surely it should not15:51
mgoddardjust make up a name and append it to https://etherpad.openstack.org/p/15:51
yoctozeptomgoddard: ok, I'll prepare it15:51
mgoddard#action yoctozepto to create an etherpad with a taxonomy of CI errors and a count of how many times each has been hit15:51
mgoddardwe have a CI status section in the whiteboard which would be a good place for this link15:52
yoctozepto(for the third time) <yoctozepto> also in this regard - scope of kolla-ansible testing - it does not try to deploty everything and surely it should not15:52
yoctozeptobut maybe it's time to rethink what should be15:52
yoctozeptomaybe it has already been done :P15:52
mgoddardwe have ongoing work to improve CI coverage15:53
mgoddardnormally as additional jobs15:53
mgoddardis that what you mean?15:54
yoctozeptoyup15:54
mgoddardjust added the WIP patches to the whiteboard (at the bottom)15:54
yoctozeptook, cool15:55
mgoddardI'd like to see the ironic and mariadb patches get some more attention15:56
mgoddardas they're ready to go15:56
mgoddard#link https://review.opendev.org/#/c/568829/15:56
mgoddard#link https://review.opendev.org/65566315:56
mgoddardthere is also more coverage we could get within our existing jobs, e.g. horizon as discussed earlier15:57
yoctozeptoyup15:57
yoctozeptoI also saw something ceph-related today/yesterday?15:58
yoctozeptowhere I suggested glance15:58
yoctozeptohttps://review.opendev.org/#/c/666068/15:58
yoctozepto'tis the change15:58
mgoddardthat's true. It was adding the ceph-upgrade job to kolla15:58
yoctozeptoyou did -1 on your workflow15:58
mgoddardyoctozepto: yeah, I had some dummy changes to trigger the jobs. Need to remove now I know they work15:59
mgoddardglance is an interesting idea. We should probably add to all ceph jobs for both kolla and kolla-ansible15:59
mgoddardWe're at time.16:00
mgoddardThanks for attending, everyone16:00
mgoddard#endmeeting16:00
*** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:00
openstackMeeting ended Wed Jun 19 16:00:29 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.html16:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.txt16:00
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.log.html16:00
yoctozeptomgoddard: I thought it is already for kolla-ansible, will recheck16:00
yoctozeptoyup, it's using base, so glance triggers it too16:01
yoctozeptoeverything triggers it in fact16:01
mgoddardyeah16:02
openstackgerritMark Goddard proposed openstack/kolla master: Enable CI jobs for ceph upgrade, NFV, cinder LVM  https://review.opendev.org/66606816:05
mgoddardyoctozepto: ^16:05
yoctozeptomgoddard, on it!16:06
yoctozeptomgoddard: glance and cinder-lvm?16:07
yoctozeptonot related imho16:08
yoctozepto(no that I mind doing extra tests)16:08
*** rpittau is now known as rpittau|afk16:10
openstackgerritMark Goddard proposed openstack/kolla master: Enable CI jobs for ceph upgrade, NFV, cinder LVM  https://review.opendev.org/66606816:10
rm_workheh sorry yeah, was asleep, woke up just in time to start the octavia meeting (which is now)16:11
yoctozeptoso now we have zun and kuryr tested with ubuntu16:12
yoctozeptoand bifrost + all the NFV stuff with centos16:12
yoctozeptogood, uniformity is good16:14
*** nathharp_ has joined #openstack-kolla16:14
*** spsurya has quit IRC16:18
*** nathharp has quit IRC16:18
*** nathharp_ has quit IRC16:19
*** gardenshed has left #openstack-kolla16:21
*** Wasaac has quit IRC16:24
*** JqckB has quit IRC16:25
mgoddardno worries rm_work16:27
*** scottsol has quit IRC16:28
*** rgogunskiy has joined #openstack-kolla16:34
openstackgerritMerged openstack/kolla-ansible stable/stein: Remove obsolete roles middleware  https://review.opendev.org/66624216:35
*** rgogunskiy has quit IRC16:38
*** gfhellma has joined #openstack-kolla16:44
*** k_mouza_ has joined #openstack-kolla16:45
*** k_mouza has quit IRC16:49
*** k_mouza_ has quit IRC16:49
*** gfhellma_ has joined #openstack-kolla16:50
*** gfidente has quit IRC16:51
openstackgerritMerged openstack/kolla-ansible stable/stein: Add blazar to fluentd aggregation  https://review.opendev.org/66613716:52
yoctozeptoCI errors dashboard done16:53
yoctozeptoand linked16:53
*** gfhellma has quit IRC16:53
*** Sravan has joined #openstack-kolla17:22
openstackgerritMerged openstack/kolla-ansible stable/rocky: Support multi-region discovery of Nova cells  https://review.opendev.org/66612617:23
openstackgerritMerged openstack/kolla-ansible stable/stein: Support multi-region discovery of Nova cells  https://review.opendev.org/66612517:23
*** mgoddard has quit IRC17:25
*** mgoddard has joined #openstack-kolla17:27
openstackgerritRadosław Piliszek proposed openstack/kolla master: Make SKIPPED_IMAGES more flexible  https://review.opendev.org/66511617:33
*** JqckB has joined #openstack-kolla17:34
openstackgerritMerged openstack/kolla-ansible stable/rocky: Add blazar to fluentd aggregation  https://review.opendev.org/66613817:40
*** JqckB has quit IRC17:42
*** k_mouza has joined #openstack-kolla17:46
*** factor__ has joined #openstack-kolla17:49
*** icarusfactor has quit IRC17:49
*** k_mouza has quit IRC17:51
*** dougsz has quit IRC17:51
openstackgerritRadosław Piliszek proposed openstack/kolla master: Make SKIPPED_IMAGES more flexible  https://review.opendev.org/66511617:52
*** Sravan has quit IRC17:54
*** Sravan has joined #openstack-kolla17:54
yoctozeptomgoddard, mnasiadka: https://review.opendev.org/665116 let's merge before we remove more images ;-)17:55
yoctozeptoor add new for that matter17:59
*** gfhellma__ has joined #openstack-kolla18:03
*** gfhellma_ has quit IRC18:06
*** pcaruana has quit IRC18:11
yoctozeptohttp://logs.openstack.org/68/666068/3/check/tripleo-build-containers-centos-7/7bab8bd/logs/build.log.txt.gz18:11
yoctozeptonow tripleo killed our CI18:11
yoctozeptomwhahaha: ^18:12
mwhahaha2019-06-19 17:04:01 | INFO:kolla.common.utils.base:Cannot find a valid baseurl for repo: centos-nfs-ganesha28/7/x86_6418:12
mwhahahasounds like a bad repo config18:12
yoctozeptoonly in tripleo18:12
yoctozeptounfortunately we have it voting :P18:13
mwhahahak let me look18:13
mwhahahanot sure where that's coming from18:13
mwhahahait's coming from the centos storage sig18:14
mwhahahahttp://logs.openstack.org/68/666068/3/check/tripleo-build-containers-centos-7/7bab8bd/logs/build.log.txt.gz#_2019-06-19_17_03_5518:14
yoctozeptogeez18:14
mwhahahacentos-release-ceph-nautilus18:14
yoctozeptoso they did not mirror it then?18:14
mwhahahalooks like a bad repo file18:14
mwhahahago ahead and propose to non-vote the tripleo job18:17
mwhahahawill have to tack this down18:17
yoctozeptoit's ok, nothing super hectic, I attacked #centos-devel channel too18:18
mwhahahaits coming in via tripleo because we configure ceph repos by default18:19
yoctozeptoyeah, I know18:19
mwhahahai know there was a patch to move kolla back to the storage sig stuff18:19
yoctozeptoyup18:19
yoctozeptoand we wanted ganesha18:19
mwhahahabut i thought i recalled a comment about it not being ok until friday18:19
yoctozeptoso they delivered18:19
mwhahaha"delivered"18:19
yoctozeptobut apparently something went awry18:19
yoctozeptoyes, exactly18:20
yoctozeptoFriday was the day18:20
yoctozeptonot Wednesday18:20
yoctozeptoso we broken till Friday ;D18:20
mwhahahasounds about right18:21
*** luksky has joined #openstack-kolla18:30
mnasiadkaTalk to you next week guys, take care :)18:35
*** ivve has joined #openstack-kolla18:50
*** Wasaac has joined #openstack-kolla19:09
*** JqckB has joined #openstack-kolla19:13
*** Wasaac has quit IRC19:14
*** ivve has quit IRC19:16
openstackgerritRadosław Piliszek proposed openstack/kolla master: Clean up test process  https://review.opendev.org/66458019:34
openstackgerritMichael Bayer proposed openstack/kolla master: Add python-sqlalchemy-collectd client  https://review.opendev.org/65967819:44
*** Sravan has quit IRC19:46
*** kplant has quit IRC19:47
*** Sravan has joined #openstack-kolla19:59
*** Wasaac has joined #openstack-kolla19:59
*** hamzaachi has joined #openstack-kolla20:03
*** Sravan has quit IRC20:05
*** gfhellma__ has quit IRC20:07
*** schwicht has quit IRC20:08
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Fix Qinling deployment and reconfigure  https://review.opendev.org/66643720:19
*** hamzaachi has quit IRC20:20
*** hamzaachi has joined #openstack-kolla20:21
*** gfhellma has joined #openstack-kolla20:22
*** cah_link has joined #openstack-kolla20:22
*** gfhellma_ has joined #openstack-kolla20:24
*** gfhellma has quit IRC20:28
*** schwicht has joined #openstack-kolla20:30
*** cah_link has quit IRC20:31
*** cah_link has joined #openstack-kolla20:34
*** cah_link has quit IRC20:39
*** gfhellma_ has quit IRC20:39
*** Wasaac has quit IRC20:42
*** cah_link has joined #openstack-kolla20:45
*** schwicht has quit IRC20:46
*** whoami-rajat has quit IRC20:47
openstackgerritMerged openstack/kolla-ansible stable/queens: Hide logs when looping over passwords  https://review.opendev.org/66613020:58
*** hamzaachi has quit IRC21:05
*** schwicht has joined #openstack-kolla21:08
openstackgerritMerged openstack/kolla-ansible stable/queens: Support multi-region discovery of Nova cells  https://review.opendev.org/66612721:11
goldyfruitIs there a way to keep using python 2.7 ?21:14
goldyfruitLike having a variable to export ?21:14
*** kplant has joined #openstack-kolla21:18
*** hamzaachi has joined #openstack-kolla21:20
*** cah_link has quit IRC21:23
*** Sravan has joined #openstack-kolla21:24
goldyfruitI don't understand why some images are build in python 2.7 and others in python 3.x21:28
goldyfruitI'm using master, keystone is under 2.7 and qinling under 3.621:28
goldyfruitBase Ubuntu 18.0421:29
goldyfruitInstall type source21:29
goldyfruitThe only solution I found so far is to force 2.7 here https://github.com/openstack/kolla/blob/d436001b43f0a7e3c161ac4a2c5d09d36c6f1015/kolla/image/build.py#L71721:38
*** cah_link has joined #openstack-kolla21:39
*** gfhellma has joined #openstack-kolla21:42
goldyfruitWhich force KOLLA_DISTRO_PYTHON_VERSION=2.7 variable21:44
openstackgerritMerged openstack/kolla stable/queens: Add sha256 in ceph fetch keys  https://review.opendev.org/66608421:51
openstackgerritMerged openstack/kolla stable/rocky: [neutron] Do not migrate all subprojects  https://review.opendev.org/66607621:51
*** goldyfruit has quit IRC21:54
*** JamesBen_ has quit IRC21:54
*** cah_link has quit IRC21:58
mlozahello, I have 3 controllers, I'm getting tons of RMQ broken pipe in one controller, how do I repair this?22:17
*** zbr|ruck has quit IRC22:17
openstackgerritMerged openstack/kolla stable/queens: Remove HTTP configuration shipped in openstack-nova-placement-api RPM  https://review.opendev.org/66600222:21
*** JqckB has quit IRC22:22
*** hamzaachi has quit IRC22:30
*** luksky has quit IRC22:39
*** Wasaac has joined #openstack-kolla22:43
*** pomac has quit IRC22:43
*** Wasaac has quit IRC22:47
*** pomac has joined #openstack-kolla22:50
*** gfhellma_ has joined #openstack-kolla22:52
openstackgerritMerged openstack/kolla-ansible stable/rocky: Fix Blazar Nova aggregate in multi-region setup  https://review.opendev.org/66613222:53
openstackgerritMerged openstack/kolla-ansible stable/queens: Fix Blazar Nova aggregate in multi-region setup  https://review.opendev.org/66613322:53
openstackgerritMerged openstack/kolla-ansible stable/rocky: Hide logs when looping over passwords  https://review.opendev.org/66612922:53
openstackgerritMerged openstack/kolla-ansible stable/rocky: Fix monasca grafana organisation check  https://review.opendev.org/66612422:53
*** gfhellma has quit IRC22:55
openstackgerritMerged openstack/kolla-ansible stable/stein: Hide logs when looping over passwords  https://review.opendev.org/66612822:58
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix monasca grafana organisation check  https://review.opendev.org/66612322:58
openstackgerritMerged openstack/kolla-ansible stable/rocky: Fixes VMTP deploys.  https://review.opendev.org/66612222:58
openstackgerritMerged openstack/kolla-ansible stable/stein: Fixes VMTP deploys.  https://review.opendev.org/66612122:58
openstackgerritMerged openstack/kolla-ansible stable/rocky: Add ansible_nodename (system hostname) to /etc/hosts  https://review.opendev.org/66609823:01
openstackgerritMerged openstack/kolla-ansible stable/queens: Add ansible_nodename (system hostname) to /etc/hosts  https://review.opendev.org/66609923:01
openstackgerritMerged openstack/kolla-ansible stable/rocky: [heat] Multi-region support for bootstrap  https://review.opendev.org/66613523:01
openstackgerritMerged openstack/kolla-ansible stable/stein: Update service_provider for FWaaS v2  https://review.opendev.org/66611823:10
openstackgerritMerged openstack/kolla-ansible stable/queens: [heat] Multi-region support for bootstrap  https://review.opendev.org/66613623:10
*** Sravan has quit IRC23:11
*** JamesBenson has joined #openstack-kolla23:12
*** JamesBenson has quit IRC23:17
*** Sravan has joined #openstack-kolla23:20
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix mongo command for supporting other port  https://review.opendev.org/66614823:22
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix mongo command for checking replication status  https://review.opendev.org/66614423:22
openstackgerritMerged openstack/kolla-ansible stable/rocky: Fix mongo command for supporting other port  https://review.opendev.org/66614923:22
openstackgerritMerged openstack/kolla-ansible stable/rocky: Update service_provider for FWaaS v2  https://review.opendev.org/66620023:22
openstackgerritMerged openstack/kolla-ansible stable/queens: nova: Fix DBNotAllowed during compute startup  https://review.opendev.org/66609623:22
openstackgerritMerged openstack/kolla-ansible stable/stein: Do not create Keystone domain-specific config directory if useless  https://review.opendev.org/66605923:22
*** gfhellma_ has quit IRC23:25
*** gfhellma_ has joined #openstack-kolla23:25
*** gfhellma__ has joined #openstack-kolla23:30
*** gfhellma_ has quit IRC23:31
*** gfhellma_ has joined #openstack-kolla23:43
*** gfhellma__ has quit IRC23:47
*** gfhellma_ has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!