| *** gfhellma_ has joined #openstack-kolla | 00:07 | |
| *** gfhellma_ has quit IRC | 00:09 | |
| *** zhanglong has quit IRC | 00:17 | |
| *** gregoryo has joined #openstack-kolla | 00:25 | |
| *** spiette has quit IRC | 00:29 | |
| *** spiette has joined #openstack-kolla | 00:31 | |
| *** zhanglong has joined #openstack-kolla | 00:40 | |
| *** kplant has quit IRC | 00:50 | |
| *** igordc has quit IRC | 01:16 | |
| *** spiette has quit IRC | 01:21 | |
| *** spiette has joined #openstack-kolla | 01:23 | |
| *** zhanglong has quit IRC | 01:30 | |
| *** zhanglong has joined #openstack-kolla | 01:34 | |
| *** guozijn has joined #openstack-kolla | 01:53 | |
| *** zhanglong has quit IRC | 01:54 | |
| *** hongbin has joined #openstack-kolla | 01:58 | |
| openstackgerrit | Zijian Guo proposed openstack/kolla-ansible stable/rocky: Update service_provider for FWaaS v2 https://review.opendev.org/666200 | 02:01 |
|---|---|---|
| *** whoami-rajat has joined #openstack-kolla | 02:08 | |
| *** Sravan has joined #openstack-kolla | 02:36 | |
| *** Sravan has quit IRC | 02:40 | |
| *** guozijn has quit IRC | 03:18 | |
| *** guozijn has joined #openstack-kolla | 04:09 | |
| *** Sravan has joined #openstack-kolla | 04:21 | |
| *** guozijn has quit IRC | 04:22 | |
| *** hongbin has quit IRC | 04:25 | |
| *** Sravan has quit IRC | 04:31 | |
| openstackgerrit | Tatsuma Matsuki proposed openstack/kolla-ansible master: Add logrotate configuration for horizon https://review.opendev.org/666214 | 04:33 |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove heat_watch_server_url configuration option https://review.opendev.org/665185 | 04:39 |
| *** harsha has quit IRC | 04:42 | |
| *** Sravan has joined #openstack-kolla | 04:44 | |
| *** Sravan has quit IRC | 04:50 | |
| *** Sravan has joined #openstack-kolla | 05:07 | |
| *** spiette has quit IRC | 05:27 | |
| *** yankcrime has quit IRC | 05:27 | |
| *** timss has quit IRC | 05:28 | |
| *** spiette_ has joined #openstack-kolla | 05:28 | |
| *** timss has joined #openstack-kolla | 05:28 | |
| *** _Cyclone_ has quit IRC | 05:30 | |
| *** irclogbot_3 has quit IRC | 05:30 | |
| *** irclogbot_3 has joined #openstack-kolla | 05:31 | |
| *** mgoddard has quit IRC | 05:31 | |
| *** _Cyclone_ has joined #openstack-kolla | 05:33 | |
| *** mgoddard has joined #openstack-kolla | 05:33 | |
| *** skramaja has joined #openstack-kolla | 05:42 | |
| *** Sravan has quit IRC | 05:47 | |
| *** guozijn has joined #openstack-kolla | 05:50 | |
| *** Luzi has joined #openstack-kolla | 05:50 | |
| gregoryo | I wonder why, but neither of my two keepalived containers had MASTER state. Perhaps because an even number is a bad idea? | 05:59 |
| gregoryo | Manually changing one to MASTER and restarted the service in that container made the VIP start responding. | 05:59 |
| *** guozijn has quit IRC | 06:03 | |
| *** cah_link has joined #openstack-kolla | 06:04 | |
| *** zhangfei has joined #openstack-kolla | 06:04 | |
| *** spsurya has joined #openstack-kolla | 06:16 | |
| yoctozepto | morning | 06:16 |
| yoctozepto | gregoryo: nope, keepalived is fine with even numbers | 06:16 |
| gregoryo | Odd then that neither got MASTER from a deployment. | 06:17 |
| yoctozepto | odd indeed | 06:17 |
| gregoryo | Excuse the pun. | 06:17 |
| yoctozepto | I have seen master-master due to firewall issues | 06:18 |
| yoctozepto | but not the other way | 06:18 |
| gregoryo | Thankfully a reconfigure run (to enable LDAP - that was so easy!) didn't break it. | 06:18 |
| yoctozepto | the pun was fine | 06:18 |
| yoctozepto | you enabled ldap in a separate domain I presume? | 06:18 |
| gregoryo | This was BACKUP in both. | 06:18 |
| yoctozepto | yeah, that's the other state | 06:19 |
| yoctozepto | I think there is also an INIT state and a FAILURE state | 06:19 |
| gregoryo | Yeah in /etc/keystone/domains/keystone.ourdomain.conf | 06:19 |
| yoctozepto | but these are rare | 06:19 |
| yoctozepto | next time check logs, maybe there was a third one :P | 06:20 |
| yoctozepto | keepalived is eager to become MASTER | 06:20 |
| gregoryo | I love the Kolla’s Deployment Philosophy page. Markedly different - and better - compared to our previous deployments with Fuel and Puppet OpenStack. | 06:20 |
| yoctozepto | oh, yeah, exact reason why I am not doing tripleo, osa or whatever | 06:21 |
| gregoryo | Would you be able to help me understand (and then articulate to colleagues!) how OSA is different in this regard? | 06:22 |
| *** dpawlik has joined #openstack-kolla | 06:22 | |
| *** quoth_ has quit IRC | 06:23 | |
| yoctozepto | well, sure | 06:24 |
| yoctozepto | last time I checked they went the road of templating all the stuff | 06:24 |
| yoctozepto | so they curate their own way of configuring ansible roles for deploying openstack | 06:25 |
| yoctozepto | also, they didn't use docker at the time | 06:25 |
| gregoryo | Right, a lot like Puppet OpenStack then. | 06:25 |
| yoctozepto | but LXC I presume? | 06:25 |
| gregoryo | Ah so they've switched to Docker? Yes, LXC was what I read. | 06:25 |
| yoctozepto | or baremetal was an option too | 06:25 |
| yoctozepto | I don't know now | 06:25 |
| yoctozepto | not following | 06:25 |
| yoctozepto | yeah, it was probably meant to be | 06:26 |
| yoctozepto | puppet replacement | 06:26 |
| yoctozepto | as ansible is pushed to replace puppet more and more | 06:26 |
| yoctozepto | which I find good because it is actually saner (though I still like puppet in general) | 06:26 |
| gregoryo | I started reading https://www.packtpub.com/au/virtualization-and-cloud/openstack-cloud-computing-cookbook-fourth-edition and it says LXC - published Jan 2018. | 06:27 |
| *** threestrands has joined #openstack-kolla | 06:27 | |
| yoctozepto | I would go read OSA's latest docs to confirm, seen many books claim false but this is probably true | 06:28 |
| gregoryo | I think Puppet is probably better for managing pets, Ansible a bit more cattle-focused. Lots of overlap of course, but higher overhead to start using Puppet. | 06:29 |
| yoctozepto | https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/overview.html | 06:29 |
| yoctozepto | lxc | 06:29 |
| gregoryo | Anyway, the templating angle is important. It's a benefit I hadn't even realised we would get out of Kolla. | 06:30 |
| yoctozepto | gregoryo: yeah, you might be right but the model in general is reversed as puppet is usually left to run in background periodically checking the inventory | 06:30 |
| yoctozepto | true, this way kolla is simpler too | 06:30 |
| yoctozepto | not fixing every little config issue | 06:31 |
| yoctozepto | :D | 06:31 |
| gregoryo | Curation and scaling win. Let upstream defaults reign, change what you need to in only one place. Keep the deployment tool out of the way where possible, and make less work for its maintainers. | 06:31 |
| yoctozepto | as if I was hearing myself! | 06:33 |
| gregoryo | Hmmm, https://docs.openstack.org/openstack-ansible/latest/ says Stein is in pre-release. If true, that is a telling example of the extra curation work at each release. | 06:33 |
| yoctozepto | heh, we are no better here | 06:33 |
| yoctozepto | ours is too | 06:33 |
| gregoryo | Haha fair enough. | 06:33 |
| yoctozepto | though it's fairing completion | 06:33 |
| yoctozepto | we still can't release as soon as other projects do | 06:34 |
| yoctozepto | as we depend on them | 06:34 |
| yoctozepto | to stabilize first | 06:34 |
| gregoryo | We haven't got to rocky yet ): | 06:34 |
| yoctozepto | and the team is actually small as you can see | 06:34 |
| yoctozepto | but I have Stein working in evaluation here locally | 06:34 |
| yoctozepto | via kolla | 06:34 |
| yoctozepto | core services deploy fine | 06:35 |
| gregoryo | How about heat and sahara? | 06:35 |
| yoctozepto | I think heat works but have not used sahara | 06:35 |
| gregoryo | Okay thanks. I'll stay tuned. | 06:36 |
| yoctozepto | well, you can try yourself :D | 06:36 |
| yoctozepto | I have sahara on my deploy plans too | 06:36 |
| gregoryo | I have too much to try already just in getting a Kolla deploy to look like our production environment! | 06:37 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Remove heat_watch_server_url configuration option https://review.opendev.org/666232 | 06:37 |
| yoctozepto | gregoryo: I don't have real production yet so it's one less thing to worry about 8-) | 06:38 |
| gregoryo | We're over 2 years in prod now, Newton through to Queens. | 06:42 |
| gregoryo | Had some issues in January which confirmed wanting to get off our snowflake Fuely Puppety arrangement. | 06:43 |
| yoctozepto | I see, but is the project still developed? | 06:45 |
| yoctozepto | I mean fuel | 06:45 |
| gregoryo | Nope. | 06:46 |
| gregoryo | I mean, maybe, but not so much that we notice. We turned off our Fuel controller about a year ago and moved to Puppet OpenStack. That went well initially but that project feels like it's fading too. | 06:47 |
| yoctozepto | no wonder, as the whole puppet scene is | 06:49 |
| gregoryo | Emilien Macchi I think was developing it and was a great help, but RedHat bought Ansible, and apparently are retooling TripleO from Puppet to Ansible, so that momentum will die or has died. | 06:50 |
| *** guozijn has joined #openstack-kolla | 06:50 | |
| yoctozepto | yeah, they are doing this now | 06:51 |
| yoctozepto | though they are also relaying on custom config and heat | 06:51 |
| yoctozepto | relying* which I find the wrong approach to config | 06:54 |
| *** rpittau|afk is now known as rpittau | 07:02 | |
| *** igordc has joined #openstack-kolla | 07:05 | |
| *** luksky has joined #openstack-kolla | 07:06 | |
| *** ivve has joined #openstack-kolla | 07:07 | |
| *** igordc has quit IRC | 07:10 | |
| *** luksky has quit IRC | 07:11 | |
| mnasiadka | morning | 07:15 |
| mnasiadka | yoctozepto: do you ever sleep? :) | 07:15 |
| yoctozepto | mnasiadka: nope, sleep is for the weak :D | 07:16 |
| yoctozepto | morning ;-) | 07:16 |
| mnasiadka | yoctozepto: then I'm weak :) | 07:16 |
| yoctozepto | :P | 07:17 |
| yoctozepto | I sleep 6-8 hours a day | 07:17 |
| yoctozepto | today patching our servers against SACK vuln | 07:19 |
| yoctozepto | turned out you can use SACK cap to bring down servers to knees | 07:19 |
| mnasiadka | my blessing that I work in R&D... | 07:21 |
| *** luksky has joined #openstack-kolla | 07:23 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Test blazar in nfv scenario https://review.opendev.org/651463 | 07:26 |
| yoctozepto | mnasiadka: https://review.opendev.org/665927 | 07:36 |
| yoctozepto | Mark left it for you to decide | 07:36 |
| yoctozepto | let's unbreak this task today ;D | 07:37 |
| yoctozepto | needs backport to stein | 07:37 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Fix deployment of ceph-nfs container failing due to wrong USER https://review.opendev.org/666239 | 07:48 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Test ceph-nfs as well when testing ceph https://review.opendev.org/666240 | 07:51 |
| *** ktibi has joined #openstack-kolla | 07:58 | |
| *** threestrands has quit IRC | 07:59 | |
| *** gkadam has joined #openstack-kolla | 08:00 | |
| *** dougsz has joined #openstack-kolla | 08:01 | |
| mnasiadka | mgoddard: https://review.opendev.org/#/c/665927 - check my comment and let's agree on a way forward :) | 08:02 |
| mgoddard | mnasiadka: you may be right about the script, but it does not take too long to do manually, and I think it's good to inspect patches first | 08:04 |
| mgoddard | mnasiadka: about stable policy, IMO we should backport bug fixes to all active stable branches (Q,R,S) where it makes sense | 08:05 |
| mgoddard | mnasiadka: I have been hit many times downstream by a too relaxed backport policy | 08:05 |
| mgoddard | mnasiadka: it's very annoying to hit a bug that has been fixed but not backported | 08:05 |
| mgoddard | there may be some grey lines around the word 'bug' here, those can be discussed per-patch | 08:06 |
| *** nathharp has joined #openstack-kolla | 08:06 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible stable/stein: Remove obsolete roles middleware https://review.opendev.org/666242 | 08:08 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible stable/rocky: Remove obsolete roles middleware https://review.opendev.org/666243 | 08:09 |
| mgoddard | mnasiadka: replied on https://review.opendev.org/#/c/665927 | 08:09 |
| *** Wasaac has joined #openstack-kolla | 08:15 | |
| *** Wasaac has quit IRC | 08:15 | |
| *** Wasaac has joined #openstack-kolla | 08:16 | |
| *** k_mouza has joined #openstack-kolla | 08:16 | |
| *** Wasaac has quit IRC | 08:17 | |
| *** Wasaac has joined #openstack-kolla | 08:17 | |
| *** gregoryo has quit IRC | 08:19 | |
| *** Wasaac has quit IRC | 08:20 | |
| *** Wasaac has joined #openstack-kolla | 08:21 | |
| *** pcaruana has quit IRC | 08:27 | |
| *** zhanglong has joined #openstack-kolla | 08:35 | |
| yoctozepto | mgoddard, mnasiadka: re:backport-policy I consider stein to be active so everything not related to train specifically or implementing completely new features should go there still; older branches should be fix only | 08:44 |
| *** pcaruana has joined #openstack-kolla | 08:45 | |
| mnasiadka | yoctozepto: so, approved the ceph crap - do you want to work on the final solution, or should I add this to my pile of stuff?:) | 08:45 |
| yoctozepto | mnasiadka: can we coordinate this somehow? I can have it on me, sounds good for flexing ansible muscles | 08:46 |
| mnasiadka | yoctozepto: sure, take this - if you have any questions just ask :) | 08:46 |
| yoctozepto | mgoddard: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2019-06-18.log.html#t2019-06-18T18:40:54 <- horizon, can't see the guy's report on launchpad, he indeed minded that | 08:46 |
| mgoddard | need to head out, will be back in 2 hours | 08:47 |
| yoctozepto | mnasiadka: I get it you want to do the other side limit too after checking the clients, though this probably conflicts with doing an update on a running system | 08:48 |
| yoctozepto | as VMs will be old clients | 08:48 |
| yoctozepto | so while it will work in CI, we end up with bad experience for users | 08:49 |
| yoctozepto | and the other fix will result in better error message than EPERM in case we miss to upgrade something :-) | 08:49 |
| yoctozepto | so this part is +1 from me | 08:49 |
| mnasiadka | yoctozepto: old clients means libvirtd normally, we should check if after libvirtd upgrade the ceph connection uses old client proto or new one | 08:49 |
| yoctozepto | the other is -1 | 08:49 |
| yoctozepto | mnasiadka: qemu stays anyway | 08:50 |
| yoctozepto | will ceph like this limit then? | 08:50 |
| mnasiadka | yoctozepto: if old one - we should document what to do after upgrade | 08:50 |
| yoctozepto | yup, docs +1 | 08:50 |
| yoctozepto | I think we should only improve the check for servers and add to docs about clients | 08:51 |
| mnasiadka | Yeah, let’s start with that | 08:51 |
| yoctozepto | +1 | 08:51 |
| yoctozepto | mnasiadka: I left a note of our discussion: https://review.opendev.org/665927 | 08:55 |
| *** Dubla1 has joined #openstack-kolla | 08:56 | |
| *** k_mouza has quit IRC | 09:01 | |
| *** k_mouza has joined #openstack-kolla | 09:01 | |
| *** lemko has joined #openstack-kolla | 09:02 | |
| *** luksky has quit IRC | 09:24 | |
| *** guozijn has quit IRC | 09:32 | |
| *** gfidente has joined #openstack-kolla | 09:33 | |
| *** k_mouza_ has joined #openstack-kolla | 09:33 | |
| yoctozepto | looks like nfs ganesha is broken anyway in Ubuntu http://logs.openstack.org/30/666030/1/check/kolla-ansible-ubuntu-source-ceph/b705211/primary/logs/kolla/ceph/ceph-nfs.txt.gz | 09:34 |
| *** k_mouza has quit IRC | 09:37 | |
| *** igordc has joined #openstack-kolla | 09:37 | |
| yoctozepto | mgoddard, mnasiadka: is it on purpose that not every service has its entry in fluentd? seems documented otherwise (as if fluentd had all the logs) | 09:42 |
| *** janki has joined #openstack-kolla | 09:43 | |
| *** k_mouza_ has quit IRC | 09:49 | |
| *** k_mouza has joined #openstack-kolla | 09:50 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix placement log perms in config.json https://review.opendev.org/666146 | 09:56 |
| openstackgerrit | Merged openstack/kolla-ansible master: Make Ceph upgrade check Ceph release to avoid EPERM https://review.opendev.org/665927 | 09:56 |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: Add blazar to fluentd aggregation https://review.opendev.org/666139 | 09:56 |
| *** k_mouza has quit IRC | 10:02 | |
| *** luksky has joined #openstack-kolla | 10:07 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix issue finding custom, host specific plugins https://review.opendev.org/666142 | 10:10 |
| *** k_mouza has joined #openstack-kolla | 10:17 | |
| *** k_mouza has quit IRC | 10:17 | |
| *** k_mouza has joined #openstack-kolla | 10:18 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:24 | |
| *** JqckB has joined #openstack-kolla | 10:25 | |
| *** ktibi has quit IRC | 10:26 | |
| *** k_mouza has quit IRC | 10:27 | |
| *** k_mouza has joined #openstack-kolla | 10:28 | |
| *** dave-mccowan has joined #openstack-kolla | 10:32 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Make Ceph upgrade check Ceph release to avoid EPERM https://review.opendev.org/666260 | 10:33 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Make Ceph upgrade check Ceph release to avoid EPERM https://review.opendev.org/666260 | 10:34 |
| *** k_mouza has quit IRC | 10:36 | |
| *** k_mouza has joined #openstack-kolla | 10:36 | |
| *** k_mouza has quit IRC | 10:39 | |
| *** Wasaac has quit IRC | 10:45 | |
| *** priteau has joined #openstack-kolla | 10:46 | |
| *** zhanglong has quit IRC | 10:47 | |
| *** Wasaac has joined #openstack-kolla | 10:47 | |
| *** spiette_ has quit IRC | 10:50 | |
| mgoddard | yoctozepto: I'm guessing from the py3 comment igordc is using ubuntu. hrw recently made some changes for it recently, including apache config. I wonder if those were in the tested code | 11:04 |
| mgoddard | yoctozepto: testing horizon would definitely be a good idea. I wonder if a simple curl would have caught this | 11:04 |
| igordc | mgoddard, for the case when the horizon container doesn't have the expected python27 script path a simple curl would catch easily (just gives a 404) | 11:06 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Use become for all docker tasks https://review.opendev.org/666140 | 11:07 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Check for 'become' in tasks that use Docker in pep8 https://review.opendev.org/666141 | 11:07 |
| openstackgerrit | Doug Szumski proposed openstack/kolla master: Bump Monasca Fluentd output plugin https://review.opendev.org/666265 | 11:08 |
| *** kplant has joined #openstack-kolla | 11:08 | |
| igordc | the issues I faced were on sunday | 11:08 |
| *** ktibi has joined #openstack-kolla | 11:10 | |
| *** zhangfei has quit IRC | 11:13 | |
| yoctozepto | mgoddard, igordc: https://review.opendev.org/642375 | 11:15 |
| yoctozepto | though there seem to be no apache-related changes | 11:15 |
| *** VickyG has quit IRC | 11:17 | |
| igordc | yoctozepto, https://review.opendev.org/#/c/642375/28/docker/horizon/extend_start.sh | 11:20 |
| igordc | yoctozepto, this is the path that was pointing to non-existent python2.7: SITE_PACKAGES="/usr/lib/python${KOLLA_DISTRO_PYTHON_VERSION}/site-packages" | 11:21 |
| *** rgogunskiy has joined #openstack-kolla | 11:21 | |
| igordc | yoctozepto, my mistake; it's this one: SITE_PACKAGES="/var/lib/kolla/venv/lib/python${KOLLA_DISTRO_PYTHON_VERSION}/site-packages" | 11:22 |
| igordc | copied the wrong line | 11:22 |
| mgoddard | igordc: can you confirm you're using ubuntu bionic? | 11:22 |
| yoctozepto | eh, nice one that | 11:22 |
| yoctozepto | hrw patched binary path | 11:22 |
| igordc | mgoddard, yeah the host is bionic | 11:23 |
| yoctozepto | when he wanted the source path | 11:23 |
| yoctozepto | mgoddard: the issue is it's in the wrong branch of src/bin if | 11:23 |
| yoctozepto | I'm inspecting the patch to see if it's the only place | 11:24 |
| mgoddard | yoctozepto: why is it the wrong branch? | 11:25 |
| yoctozepto | mgoddard: "move to Python 3 for source images" | 11:25 |
| yoctozepto | if [[ ${KOLLA_INSTALL_TYPE} == "binary" ]]; then | 11:26 |
| yoctozepto | seems wrong to modify this one instead of the other? | 11:26 |
| mgoddard | oh I see. That's true. The patch was previously a mix of source + binary, I split out binary into a separate patch, and maybe that was left behind | 11:28 |
| mgoddard | also seems like it was wrong | 11:28 |
| mgoddard | igordc: can you check your images to see which of those paths is correct for binary images? | 11:28 |
| igordc | mgoddard, let me check my notes | 11:29 |
| yoctozepto | it's the only place in the change that has this oddity | 11:29 |
| yoctozepto | wonder why CI did not catch this | 11:30 |
| yoctozepto | because it blocks apache | 11:31 |
| igordc | mgoddard, I didn't save the path for the binary case but I'm pretty sure I saw what's in the patch: /usr/lib/python3/dist-packages | 11:31 |
| mgoddard | igordc: you can just pull down the image | 11:31 |
| igordc | mgoddard, for the source case I was seeing /var/lib/kolla/venv/lib/python2.7/site-packages (horizon.conf) and the same, but python3.6, in the container | 11:32 |
| igordc | need a few minutes to turn on the machine | 11:33 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Make fluentd-elasticsearch configuration more robust https://review.opendev.org/661747 | 11:41 |
| *** ohwhyosa has joined #openstack-kolla | 11:43 | |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Enable buffering to file for Monasca logs https://review.opendev.org/666271 | 11:44 |
| igordc | python: /usr/bin/python2.7 /usr/bin/python /usr/bin/python3.6 /usr/bin/python3.6m /usr/lib/python3.6 /usr/lib/python2.7 /usr/lib/python3.7 /etc/python2.7 /etc/python /etc/python3.6 /usr/local/lib/python2.7 /usr/local/lib/python3.6 /usr/share/python | 11:50 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/rocky: Use td-agent-gem to install fluentd monasca plugin on ubuntu https://review.opendev.org/666075 | 11:52 |
| igordc | mgoddard, /usr/lib/python3.6/site-packages does not exist in my 3day old ubuntu-binary-horizon:master | 11:53 |
| igordc | but /usr/lib/python3/dist-packages does | 11:54 |
| mgoddard | igordc: ok, but the script checks for SITE_PACKAGES="/usr/lib/python3/dist-packages" on ubuntu | 11:54 |
| mgoddard | ok | 11:54 |
| mgoddard | so that change seems sound | 11:54 |
| igordc | and there's python 2.7,3, 3.6, 3.7 dirs | 11:54 |
| mgoddard | igordc: looks like the problem is in kolla ansible | 11:55 |
| mgoddard | grep -rn 2.7 ansible/roles/horizon/ | 11:55 |
| *** Wasaac has quit IRC | 11:55 | |
| mgoddard | igordc: I think we need a distro_python_version variable in kolla ansible | 11:56 |
| mgoddard | lots of references to 2.7 in ansible/ | 11:56 |
| igordc | mgoddard, yeah.. that first else is definitely problematic for the source image | 11:57 |
| igordc | mgoddard, I just noticed that all my source:stein images are apparently 2 months old (from my previous attempt) | 11:58 |
| mgoddard | igordc: source:stein is still 2.7 | 11:58 |
| igordc | mgoddard, I had assumed the docker images would get updated when re-running kolla-ansible | 11:59 |
| mgoddard | binary:stein is 3.6 | 11:59 |
| mgoddard | igordc: not if the tag doesn't change. You need to pull new images explicitly in that case | 11:59 |
| igordc | mgoddard, but the 404 failure was in source:master | 11:59 |
| mgoddard | igordc: yeah, probably due to references to 2.7 in k-a | 12:00 |
| igordc | mgoddard, lol I'm surprised my source:stein deployment went so well this time, even internet for vms works this time | 12:00 |
| igordc | mgoddard, is it safe to upgrade all my source:stein images/containers without breaking my current openstack deployment? offline time is fine | 12:02 |
| mgoddard | igordc: it should be, yes. Always worth backing up before an upgrade though | 12:04 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: Add custom filters for checking services https://review.opendev.org/665922 | 12:08 |
| openstackgerrit | Merged openstack/kolla-ansible master: Add logrotate configuration for horizon https://review.opendev.org/666214 | 12:08 |
| igordc | mgoddard, will 'kolla-ansible upgrade' | 12:11 |
| igordc | do the trick after pulling all images? | 12:11 |
| mgoddard | igordc: yes | 12:11 |
| igordc | awesome | 12:12 |
| *** goldyfruit has quit IRC | 12:25 | |
| yoctozepto | mgoddard, mnasiadka: do you know whether Zuul runs kolla-ansible with pipelining or without? | 12:26 |
| mgoddard | yoctozepto: I think without | 12:27 |
| *** priteau has quit IRC | 12:27 | |
| mgoddard | I don't think we touch ansible.cfg | 12:27 |
| yoctozepto | yeah, we are not | 12:27 |
| yoctozepto | I was wondering whether Zuul installs some default | 12:28 |
| yoctozepto | put it on my todo to check | 12:28 |
| *** sshnaidm|afk is now known as sshnaidm | 12:29 | |
| *** Wasaac has joined #openstack-kolla | 12:31 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Fix mongo command for checking replication status https://review.opendev.org/666145 | 12:34 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Elevate privileges for copying gnocchi policy.json https://review.opendev.org/666119 | 12:34 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Elevate privileges for copying gnocchi policy.json https://review.opendev.org/666120 | 12:34 |
| *** Wasaac has quit IRC | 12:37 | |
| *** Wasaac has joined #openstack-kolla | 12:48 | |
| *** dciabrin has quit IRC | 12:49 | |
| *** rgogunskiy has quit IRC | 12:49 | |
| *** Wasaac has quit IRC | 12:53 | |
| *** rgogunskiy has joined #openstack-kolla | 12:55 | |
| *** rgogunskiy has quit IRC | 13:03 | |
| *** zhanglong has joined #openstack-kolla | 13:04 | |
| *** Wasaac has joined #openstack-kolla | 13:05 | |
| *** Wasaac has quit IRC | 13:10 | |
| *** skramaja has quit IRC | 13:12 | |
| *** Wasaac has joined #openstack-kolla | 13:20 | |
| *** Wasaac has quit IRC | 13:24 | |
| *** spiette has joined #openstack-kolla | 13:24 | |
| *** schwicht has joined #openstack-kolla | 13:25 | |
| *** goldyfruit has joined #openstack-kolla | 13:26 | |
| *** Wasaac has joined #openstack-kolla | 13:27 | |
| ohwhyosa | Hey there! Kolla ansible is failling in the "Checking if kolla_internal_vip_address_is in the same network as api on all nodes" | 13:28 |
| ohwhyosa | It is an all-in-one deployment | 13:29 |
| ohwhyosa | And indeed it should be because its value is {{network_interface}} | 13:29 |
| ohwhyosa | Nevermind, think i got it | 13:30 |
| *** lemko has quit IRC | 13:32 | |
| ohwhyosa | but now it does fail with the ping to itself | 13:32 |
| ohwhyosa | which it does perfectly | 13:33 |
| *** zhanglong has quit IRC | 13:34 | |
| *** root____ has joined #openstack-kolla | 13:34 | |
| *** root____ has left #openstack-kolla | 13:34 | |
| *** cah_link has quit IRC | 13:40 | |
| *** k_mouza has joined #openstack-kolla | 13:41 | |
| *** scottsol has joined #openstack-kolla | 13:49 | |
| *** scottsol has left #openstack-kolla | 13:54 | |
| *** mchlumsky has joined #openstack-kolla | 13:54 | |
| *** janki has quit IRC | 13:57 | |
| *** rouk has joined #openstack-kolla | 13:58 | |
| mgoddard | ohwhyosa: did you fix the ping issue? | 13:59 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix Blazar Nova aggregate in multi-region setup https://review.opendev.org/666131 | 13:59 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Remove heat_watch_server_url configuration option https://review.opendev.org/666232 | 13:59 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix default deployment of freezer, use mariadb. https://review.opendev.org/661485 | 13:59 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Remove obsolete roles middleware https://review.opendev.org/666243 | 13:59 |
| ohwhyosa | mgoddard, nope! (i did kinda recklessly jump into deply, though, and now it has trouble restarting mariadb master, failling on the wait for it handler) | 14:01 |
| mgoddard | ohwhyosa: if you want to start again at any point, use kolla-ansible destroy | 14:01 |
| mgoddard | ohwhyosa: one big benefit of containers - you can clean up and start again :) | 14:01 |
| mgoddard | (although don't use that in production :) ) | 14:02 |
| *** gardenshed has joined #openstack-kolla | 14:04 | |
| *** dciabrin has joined #openstack-kolla | 14:05 | |
| *** dpawlik has quit IRC | 14:11 | |
| ohwhyosa | mgoddard, Nice! I'll take that into account, only use in production if you hate your clients and have a passport at hand hahahaha | 14:14 |
| mgoddard | ohwhyosa: true :) | 14:14 |
| ohwhyosa | --yes-i-really-really-mean-it had me rofl | 14:15 |
| ohwhyosa | also because i skipped the second really the first time | 14:15 |
| ohwhyosa | so i had to really really mean it | 14:15 |
| ohwhyosa | mgoddard, it should all be executed as sudo, with the aio right? | 14:17 |
| ohwhyosa | Else it throws a bunch of permission denieds | 14:17 |
| mgoddard | ohwhyosa: it should work without sudo | 14:18 |
| ohwhyosa | AAAAAnd now the bootstrap fails also u.u | 14:18 |
| ohwhyosa | This one might have been due to the sudo | 14:19 |
| mgoddard | ohwhyosa: ubuntu bionic? which kolla-ansible branch? | 14:19 |
| ohwhyosa | Hmmm so -K | 14:19 |
| ohwhyosa | ubuntu bionic, installed with pip | 14:19 |
| ohwhyosa | as in sudo pip install kolla-ansible | 14:20 |
| kplant | which openstack release are you trying to install? | 14:21 |
| ohwhyosa | now i miserably failed because of another permission denied (etc/hosts) in bootstrap servers! No -K option? (tried and it seems not to recognize it) | 14:21 |
| ohwhyosa | stein | 14:21 |
| kplant | iirc kolla-ansible 7.1 is available in pypi | 14:22 |
| kplant | and that supports up to rocky | 14:22 |
| kplant | unless you did ==8.0.0rc1 | 14:22 |
| ohwhyosa | I didn't, lemme check the version | 14:22 |
| kplant | it's probably 7.1.1 | 14:22 |
| *** ivve has quit IRC | 14:23 | |
| ohwhyosa | --version doesn't seem to be a valid param either | 14:23 |
| ohwhyosa | I'll reinstall with 8 anycase | 14:23 |
| ohwhyosa | thank you! | 14:23 |
| kplant | pip freeze | grep kolla-ansible | 14:24 |
| *** scottsol has joined #openstack-kolla | 14:25 | |
| ohwhyosa | kplant, s*** i knew that, thanks | 14:25 |
| kplant | if you want 8.x from pypi make sure you also install kolla==8.0.0.0rc1 | 14:25 |
| kplant | not just kolla-ansible==8.0.0.0rc1 | 14:25 |
| ohwhyosa | Hmmm it seems the playbooks may have disabled/changed my eno1 ip? | 14:25 |
| ohwhyosa | solved now, but i found it funny, after "some" of the executions it seems to have left a single address hahahah | 14:26 |
| *** JamesBenson has joined #openstack-kolla | 14:26 | |
| ohwhyosa | kplant, thanks! I'll upgrade both | 14:26 |
| kplant | mgoddard: hey speaking of 8.x .... any guestimate on the next rc? | 14:28 |
| *** Wasaac has quit IRC | 14:29 | |
| ohwhyosa | still permission denied with the hosts | 14:30 |
| ohwhyosa | I'll just add an ansible_became=true to the inventory | 14:30 |
| ohwhyosa | I have it passwordless so it should work | 14:30 |
| *** JamesBen_ has joined #openstack-kolla | 14:33 | |
| *** Wasaac has joined #openstack-kolla | 14:33 | |
| *** Luzi has quit IRC | 14:36 | |
| *** JamesBenson has quit IRC | 14:36 | |
| mgoddard | kplant: still plagued by ceph issues :( I think we're getting close though | 14:42 |
| *** gfhellma has joined #openstack-kolla | 14:45 | |
| kplant | i see you at least have the nautilus ceph clients working now | 14:46 |
| *** dking_desktop has quit IRC | 14:49 | |
| *** luksky has quit IRC | 14:50 | |
| mgoddard | kplant: yes, we switched to nautilus using upstream ceph packages, but this caused issues for tripleo so now switching back to centos ceph, but blocked on them making a release... | 14:52 |
| mgoddard | == Meeting in 10 minutes == | 14:52 |
| mgoddard | ^ mgoddard mnasiadka hrw egonzalez | 14:52 |
| kplant | mgoddard: booo | 14:59 |
| mgoddard | #startmeeting kolla | 15:00 |
| openstack | Meeting started Wed Jun 19 15:00:30 2019 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: kolla)" | 15:00 | |
| openstack | The meeting name has been set to 'kolla' | 15:00 |
| mgoddard | #topic rollcall | 15:00 |
| *** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:00 | |
| mgoddard | \o | 15:00 |
| scottsol | o/ | 15:00 |
| JamesBen_ | \o | 15:01 |
| gardenshed | o/ | 15:01 |
| mloza | \o/ | 15:01 |
| mgoddard | mnasiadka: around? | 15:02 |
| mnasiadka | mgoddard: sort of :) | 15:03 |
| mnasiadka | o/ | 15:03 |
| mgoddard | :) | 15:03 |
| stackedsax | I thought I was an hour late -- just starting? | 15:03 |
| mgoddard | hi stackedsax yeah | 15:03 |
| mgoddard | #topic agenda | 15:03 |
| *** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:03 | |
| mgoddard | * Roll-call | 15:03 |
| mgoddard | * Announcements | 15:03 |
| mgoddard | ** Last call for priorities https://etherpad.openstack.org/p/kolla-train-priorities | 15:03 |
| mgoddard | * Review action items from last meeting | 15:03 |
| mgoddard | * Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard | 15:03 |
| mgoddard | * Stein release status | 15:03 |
| mgoddard | * Kolla Ansible performance https://etherpad.openstack.org/p/kolla-ansible-speed-ups | 15:03 |
| mgoddard | * (vishwanathj): Elasticsearch | 15:04 |
| mgoddard | * Octavia usability improvements | 15:04 |
| mgoddard | * Vote: Kayobe to become a kolla deliverable? | 15:04 |
| mgoddard | #topic announcements | 15:04 |
| *** openstack changes topic to "announcements (Meeting topic: kolla)" | 15:04 | |
| mgoddard | #info Last call for kolla train priorities | 15:04 |
| mgoddard | #info Last call for kolla train priorities | 15:04 |
| mgoddard | #link https://etherpad.openstack.org/p/kolla-train-priorities | 15:04 |
| mgoddard | please cast your votes - no need to be core | 15:04 |
| mgoddard | Any other announcements? | 15:04 |
| *** ktibi has quit IRC | 15:04 | |
| mgoddard | #topic Review action items from last meeting | 15:05 |
| *** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)" | 15:05 | |
| mgoddard | mgoddard to try again again to disable ocata publishing job | 15:05 |
| mgoddard | I finally did it! | 15:05 |
| mgoddard | Turns out we needed to remove the job definitions on the stable/ocata branch, rather than using a branch filter on master | 15:06 |
| mgoddard | #link https://review.opendev.org/665742 | 15:06 |
| mgoddard | Needs reviews | 15:06 |
| mgoddard | And depends on https://review.opendev.org/665935 | 15:06 |
| mgoddard | #topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard | 15:06 |
| *** openstack changes topic to "Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard (Meeting topic: kolla)" | 15:06 | |
| mgoddard | Checking CI status | 15:07 |
| mgoddard | Mostly looks happy | 15:08 |
| mgoddard | I went through and backported all bug fixes in the last month yesterday | 15:10 |
| mgoddard | lots to review | 15:10 |
| mgoddard | Anyone have anything else from the whiteboard? | 15:10 |
| mgoddard | Any other features we should be trackng on there? | 15:10 |
| mgoddard | Added CentOS/RHEL8 | 15:11 |
| mgoddard | #topic Stein release status | 15:12 |
| *** openstack changes topic to "Stein release status (Meeting topic: kolla)" | 15:12 | |
| mgoddard | We switched to Ceph nautilus using upstream ceph packages, but this caused issues for tripleo so now we are switching back to CentOS ceph packages | 15:12 |
| mgoddard | However this is blocked on an update to those packages | 15:12 |
| mgoddard | This is the one to watch: | 15:13 |
| mgoddard | #link https://review.opendev.org/#/c/666015/ | 15:13 |
| mgoddard | Any questions on Stein? | 15:14 |
| mgoddard | #topic Kolla Ansible performance https://etherpad.openstack.org/p/kolla-ansible-speed-ups | 15:14 |
| *** openstack changes topic to "Kolla Ansible performance https://etherpad.openstack.org/p/kolla-ansible-speed-ups (Meeting topic: kolla)" | 15:14 | |
| mgoddard | #link https://etherpad.openstack.org/p/kolla-ansible-speed-ups | 15:14 |
| mgoddard | This was added by klindgren who has been experiencing performance issues at a scale of around 400 hypervisors | 15:15 |
| mgoddard | Who is interested in kolla-ansible at large scale? | 15:15 |
| mnasiadka | Me | 15:15 |
| mnasiadka | Do we have any ARA recordings of the run with 400 hypervisors? | 15:16 |
| gardenshed | me | 15:16 |
| mgoddard | mnasiadka: I'm not aware of any. I don't know if they'd be happy to share but we could ask | 15:17 |
| mgoddard | I'd like to see results from running with the profile_tasks callback plugin | 15:17 |
| mgoddard | I see a bunch of people on the Etherpad | 15:18 |
| *** gfidente has quit IRC | 15:18 | |
| mgoddard | Does anyone have experience with running kolla ansible at scale? Or even just ansible? | 15:18 |
| *** gkadam has quit IRC | 15:19 | |
| mnasiadka | Ansible yes, but no crazy huge roles | 15:19 |
| mgoddard | If you have some good tips or ideas, please add to the etherpad | 15:20 |
| mnasiadka | First we need to understand what actions are problematic (templating?) | 15:20 |
| mgoddard | Yes, I think templating is hitting us hard | 15:21 |
| mgoddard | There are certain patterns that kolla-ansible uses, and we should try to profile those in isolation | 15:21 |
| mgoddard | I'd like to try to get interested parties to join a call at some point to discuss this | 15:22 |
| mgoddard | scottsol, stackedsax - interested? | 15:22 |
| scottsol | yes I can join that | 15:22 |
| stackedsax | I'd like to focus on the TLS stuff for now | 15:22 |
| mgoddard | stackedsax: sure | 15:22 |
| scottsol | I can look over what we have done already and update the etherpad | 15:23 |
| scottsol | its not "huge" scale yet but we already started to see some bottlenecks even with 20-30 nodes | 15:23 |
| scottsol | just some minor changes have already started to cut it down | 15:24 |
| mgoddard | that would be helpful | 15:26 |
| mgoddard | let's keep it to the etherpad for now | 15:26 |
| mgoddard | any other thoughts? | 15:26 |
| mgoddard | #topic (vishwanathj): Elasticsearch | 15:28 |
| *** openstack changes topic to "(vishwanathj): Elasticsearch (Meeting topic: kolla)" | 15:28 | |
| * jroll rolls in late, sorry | 15:28 | |
| mgoddard | hi jroll | 15:28 |
| mgoddard | vishwanathj who proposed this topic is not here | 15:29 |
| mgoddard | #topic Octavia usability improvements | 15:29 |
| *** openstack changes topic to "Octavia usability improvements (Meeting topic: kolla)" | 15:29 | |
| mgoddard | rm_work: around? | 15:29 |
| jroll | I won't promise he's not, but he usually isn't this time of day :) | 15:30 |
| mgoddard | This was something that came up at the Denver summit kolla feedback session - Octavia is quite hard to deploy in kolla and requires some manual effort | 15:30 |
| mgoddard | #link https://etherpad.openstack.org/p/DEN-train-kolla-feedback | 15:30 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Enable InfluxDB TSI by default https://review.opendev.org/666318 | 15:31 |
| mgoddard | From that pad: | 15:31 |
| mgoddard | generate certificates is really manual | 15:31 |
| mgoddard | doesn't support seperate CAs for internal / external | 15:31 |
| mgoddard | Sounds useful to have | 15:31 |
| mgoddard | OSA has some interesting bits | 15:31 |
| mgoddard | For context, rm_work is Octavia PTL and also has kolla experience, so could be in a good position to help improve this situation | 15:31 |
| stackedsax | thanks for the context on where that came from. | 15:32 |
| mgoddard | scottsol and co have deployed octavia, and have some notes they could share | 15:32 |
| stackedsax | rm_dev and I chatted because you told us to, mgoddard | 15:32 |
| mgoddard | :) | 15:32 |
| stackedsax | but now that I know the context, we might chat again | 15:32 |
| mgoddard | sorry, thought I'd explained but maybe not well enough | 15:33 |
| mgoddard | so what's the best path forward here? | 15:33 |
| stackedsax | no worries, we had a nice convo, anway. Just, maybe, not so useful :D | 15:33 |
| *** gfidente has joined #openstack-kolla | 15:34 | |
| mgoddard | rm_work suggested he might not have lots of time immediately, but I'm sure would be able to provide guidance | 15:34 |
| stackedsax | yeah, that's the sense I got, as well | 15:34 |
| scottsol | we could start by listing what has to be done manually? maybe another etherpad...? | 15:34 |
| stackedsax | certainly, from that pad it seems like it's something that should be addressed within the scope of the TLS everywhere conversation | 15:35 |
| scottsol | we have some changes that we can push back to fix some bugs / add minor improvments | 15:35 |
| stackedsax | scottsol: that would be good | 15:35 |
| stackedsax | maybe just on the TLS everywhere pad | 15:35 |
| stackedsax | https://etherpad.openstack.org/p/kolla-internal-tls | 15:35 |
| mgoddard | how about a ML discussion on openstack-discuss? starting with where we're at now, and then looking at gaps | 15:35 |
| scottsol | sure | 15:35 |
| mgoddard | we could always link to a pad from there, but the ML might be a good way to strike up a cross-project discussion | 15:36 |
| mgoddard | I can kick that off. Interested parties please look out for it and reply | 15:37 |
| mgoddard | #action mgoddard to post to ML about octavia in kolla | 15:37 |
| mgoddard | other thoughts on this? | 15:38 |
| mgoddard | #topic Vote: Kayobe to become a kolla deliverable? | 15:39 |
| *** openstack changes topic to "Vote: Kayobe to become a kolla deliverable? (Meeting topic: kolla)" | 15:39 | |
| mgoddard | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006901.html | 15:39 |
| mgoddard | ^ That post started the discussion about Kayobe becoming an official project | 15:39 |
| mgoddard | There are two options on the table | 15:39 |
| mgoddard | 1. become a deliverable of the Kolla project | 15:39 |
| mgoddard | 2. become an official top level OpenStack project | 15:40 |
| *** k_mouza_ has joined #openstack-kolla | 15:40 | |
| *** gfhellma has quit IRC | 15:40 | |
| mgoddard | the only feedback it received from the kolla team was from hrw who was happy with either approach | 15:40 |
| mgoddard | I think the best path forward is to take a vote. I would not participate | 15:41 |
| mgoddard | Does that sound reasonable? | 15:41 |
| scottsol | yep :) | 15:41 |
| chason | agree :) | 15:42 |
| mgoddard | I think the electorate would be the kolla core team, but would certainly be interested to hear opinions from others in the community | 15:42 |
| mgoddard | mnasiadka? | 15:42 |
| yoctozepto | o/ | 15:42 |
| yoctozepto | did not know you have a meeting today | 15:42 |
| yoctozepto | not core but opting for 1 | 15:43 |
| *** Wasaac has quit IRC | 15:43 | |
| *** k_mouza has quit IRC | 15:43 | |
| yoctozepto | to me it's an extra layer of glue (kolla) | 15:43 |
| scottsol | I think they would fit well together, and it would make it easier for keeping them sync'd. If not they could always be playing cat and mouse to stay aligned. | 15:44 |
| mnasiadka | mgoddard: agree to vote, don’t have an opinion on kayobe atm :) | 15:44 |
| mgoddard | ok. I will send an email to openstack-discuss | 15:44 |
| *** k_mouza_ has quit IRC | 15:45 | |
| mgoddard | #action mgoddard to ask openstack-discuss to vote on making kayobe a kolla deliverable | 15:45 |
| *** k_mouza has joined #openstack-kolla | 15:45 | |
| mgoddard | mnasiadka: now is the time to form one then :) | 15:45 |
| mgoddard | Any other thoughts on this? | 15:46 |
| mgoddard | #topic Open discussion | 15:46 |
| *** openstack changes topic to "Open discussion (Meeting topic: kolla)" | 15:46 | |
| mgoddard | Anyone have anything else to discuss today? | 15:47 |
| *** Wasaac has joined #openstack-kolla | 15:47 | |
| yoctozepto | mgoddard: I have one thing - CI errors | 15:47 |
| mgoddard | yoctozepto: go ahead | 15:47 |
| yoctozepto | if everybody was kind enough to report them to launchpad :-) | 15:47 |
| yoctozepto | some are random due to Zuul and infra | 15:48 |
| yoctozepto | but some are random but real | 15:48 |
| yoctozepto | so I think it's worth the effort | 15:48 |
| mgoddard | yoctozepto: good point | 15:48 |
| yoctozepto | vide nova | 15:48 |
| mgoddard | we are probably a bit too 'recheck happy' | 15:48 |
| yoctozepto | yeah, figured that out | 15:49 |
| mgoddard | I have thought about this in the past | 15:49 |
| *** vishwanathj has joined #openstack-kolla | 15:49 | |
| mgoddard | what about an etherpad with a taxonomy of CI errors and a count of how many times each has been hit? | 15:50 |
| yoctozepto | also in this regard - scope of kolla-ansible testing - it does not try to deploty everything and surely it should not | 15:50 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Fix issue finding custom, host specific plugins https://review.opendev.org/666143 | 15:50 |
| *** vishwanathj has quit IRC | 15:50 | |
| yoctozepto | > <mgoddard> what about an etherpad with a taxonomy of CI errors and a count of how many times each has been hit? | 15:50 |
| yoctozepto | good idea | 15:50 |
| mgoddard | yoctozepto: want to create one? | 15:50 |
| yoctozepto | though if it looks legit then straight to launchpad | 15:50 |
| mgoddard | and link to it from the whiteboard? | 15:50 |
| mgoddard | +1 | 15:50 |
| yoctozepto | mgoddard: sure, need to learn how to :D | 15:51 |
| * yoctozepto writes it down in todo | 15:51 | |
| yoctozepto | > <yoctozepto> also in this regard - scope of kolla-ansible testing - it does not try to deploty everything and surely it should not | 15:51 |
| mgoddard | just make up a name and append it to https://etherpad.openstack.org/p/ | 15:51 |
| yoctozepto | mgoddard: ok, I'll prepare it | 15:51 |
| mgoddard | #action yoctozepto to create an etherpad with a taxonomy of CI errors and a count of how many times each has been hit | 15:51 |
| mgoddard | we have a CI status section in the whiteboard which would be a good place for this link | 15:52 |
| yoctozepto | (for the third time) <yoctozepto> also in this regard - scope of kolla-ansible testing - it does not try to deploty everything and surely it should not | 15:52 |
| yoctozepto | but maybe it's time to rethink what should be | 15:52 |
| yoctozepto | maybe it has already been done :P | 15:52 |
| mgoddard | we have ongoing work to improve CI coverage | 15:53 |
| mgoddard | normally as additional jobs | 15:53 |
| mgoddard | is that what you mean? | 15:54 |
| yoctozepto | yup | 15:54 |
| mgoddard | just added the WIP patches to the whiteboard (at the bottom) | 15:54 |
| yoctozepto | ok, cool | 15:55 |
| mgoddard | I'd like to see the ironic and mariadb patches get some more attention | 15:56 |
| mgoddard | as they're ready to go | 15:56 |
| mgoddard | #link https://review.opendev.org/#/c/568829/ | 15:56 |
| mgoddard | #link https://review.opendev.org/655663 | 15:56 |
| mgoddard | there is also more coverage we could get within our existing jobs, e.g. horizon as discussed earlier | 15:57 |
| yoctozepto | yup | 15:57 |
| yoctozepto | I also saw something ceph-related today/yesterday? | 15:58 |
| yoctozepto | where I suggested glance | 15:58 |
| yoctozepto | https://review.opendev.org/#/c/666068/ | 15:58 |
| yoctozepto | 'tis the change | 15:58 |
| mgoddard | that's true. It was adding the ceph-upgrade job to kolla | 15:58 |
| yoctozepto | you did -1 on your workflow | 15:58 |
| mgoddard | yoctozepto: yeah, I had some dummy changes to trigger the jobs. Need to remove now I know they work | 15:59 |
| mgoddard | glance is an interesting idea. We should probably add to all ceph jobs for both kolla and kolla-ansible | 15:59 |
| mgoddard | We're at time. | 16:00 |
| mgoddard | Thanks for attending, everyone | 16:00 |
| mgoddard | #endmeeting | 16:00 |
| *** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:00 | |
| openstack | Meeting ended Wed Jun 19 16:00:29 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.html | 16:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.txt | 16:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-06-19-15.00.log.html | 16:00 |
| yoctozepto | mgoddard: I thought it is already for kolla-ansible, will recheck | 16:00 |
| yoctozepto | yup, it's using base, so glance triggers it too | 16:01 |
| yoctozepto | everything triggers it in fact | 16:01 |
| mgoddard | yeah | 16:02 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Enable CI jobs for ceph upgrade, NFV, cinder LVM https://review.opendev.org/666068 | 16:05 |
| mgoddard | yoctozepto: ^ | 16:05 |
| yoctozepto | mgoddard, on it! | 16:06 |
| yoctozepto | mgoddard: glance and cinder-lvm? | 16:07 |
| yoctozepto | not related imho | 16:08 |
| yoctozepto | (no that I mind doing extra tests) | 16:08 |
| *** rpittau is now known as rpittau|afk | 16:10 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Enable CI jobs for ceph upgrade, NFV, cinder LVM https://review.opendev.org/666068 | 16:10 |
| rm_work | heh sorry yeah, was asleep, woke up just in time to start the octavia meeting (which is now) | 16:11 |
| yoctozepto | so now we have zun and kuryr tested with ubuntu | 16:12 |
| yoctozepto | and bifrost + all the NFV stuff with centos | 16:12 |
| yoctozepto | good, uniformity is good | 16:14 |
| *** nathharp_ has joined #openstack-kolla | 16:14 | |
| *** spsurya has quit IRC | 16:18 | |
| *** nathharp has quit IRC | 16:18 | |
| *** nathharp_ has quit IRC | 16:19 | |
| *** gardenshed has left #openstack-kolla | 16:21 | |
| *** Wasaac has quit IRC | 16:24 | |
| *** JqckB has quit IRC | 16:25 | |
| mgoddard | no worries rm_work | 16:27 |
| *** scottsol has quit IRC | 16:28 | |
| *** rgogunskiy has joined #openstack-kolla | 16:34 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Remove obsolete roles middleware https://review.opendev.org/666242 | 16:35 |
| *** rgogunskiy has quit IRC | 16:38 | |
| *** gfhellma has joined #openstack-kolla | 16:44 | |
| *** k_mouza_ has joined #openstack-kolla | 16:45 | |
| *** k_mouza has quit IRC | 16:49 | |
| *** k_mouza_ has quit IRC | 16:49 | |
| *** gfhellma_ has joined #openstack-kolla | 16:50 | |
| *** gfidente has quit IRC | 16:51 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Add blazar to fluentd aggregation https://review.opendev.org/666137 | 16:52 |
| yoctozepto | CI errors dashboard done | 16:53 |
| yoctozepto | and linked | 16:53 |
| *** gfhellma has quit IRC | 16:53 | |
| *** Sravan has joined #openstack-kolla | 17:22 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Support multi-region discovery of Nova cells https://review.opendev.org/666126 | 17:23 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Support multi-region discovery of Nova cells https://review.opendev.org/666125 | 17:23 |
| *** mgoddard has quit IRC | 17:25 | |
| *** mgoddard has joined #openstack-kolla | 17:27 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Make SKIPPED_IMAGES more flexible https://review.opendev.org/665116 | 17:33 |
| *** JqckB has joined #openstack-kolla | 17:34 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Add blazar to fluentd aggregation https://review.opendev.org/666138 | 17:40 |
| *** JqckB has quit IRC | 17:42 | |
| *** k_mouza has joined #openstack-kolla | 17:46 | |
| *** factor__ has joined #openstack-kolla | 17:49 | |
| *** icarusfactor has quit IRC | 17:49 | |
| *** k_mouza has quit IRC | 17:51 | |
| *** dougsz has quit IRC | 17:51 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Make SKIPPED_IMAGES more flexible https://review.opendev.org/665116 | 17:52 |
| *** Sravan has quit IRC | 17:54 | |
| *** Sravan has joined #openstack-kolla | 17:54 | |
| yoctozepto | mgoddard, mnasiadka: https://review.opendev.org/665116 let's merge before we remove more images ;-) | 17:55 |
| yoctozepto | or add new for that matter | 17:59 |
| *** gfhellma__ has joined #openstack-kolla | 18:03 | |
| *** gfhellma_ has quit IRC | 18:06 | |
| *** pcaruana has quit IRC | 18:11 | |
| yoctozepto | http://logs.openstack.org/68/666068/3/check/tripleo-build-containers-centos-7/7bab8bd/logs/build.log.txt.gz | 18:11 |
| yoctozepto | now tripleo killed our CI | 18:11 |
| yoctozepto | mwhahaha: ^ | 18:12 |
| mwhahaha | 2019-06-19 17:04:01 | INFO:kolla.common.utils.base:Cannot find a valid baseurl for repo: centos-nfs-ganesha28/7/x86_64 | 18:12 |
| mwhahaha | sounds like a bad repo config | 18:12 |
| yoctozepto | only in tripleo | 18:12 |
| yoctozepto | unfortunately we have it voting :P | 18:13 |
| mwhahaha | k let me look | 18:13 |
| mwhahaha | not sure where that's coming from | 18:13 |
| mwhahaha | it's coming from the centos storage sig | 18:14 |
| mwhahaha | http://logs.openstack.org/68/666068/3/check/tripleo-build-containers-centos-7/7bab8bd/logs/build.log.txt.gz#_2019-06-19_17_03_55 | 18:14 |
| yoctozepto | geez | 18:14 |
| mwhahaha | centos-release-ceph-nautilus | 18:14 |
| yoctozepto | so they did not mirror it then? | 18:14 |
| mwhahaha | looks like a bad repo file | 18:14 |
| mwhahaha | go ahead and propose to non-vote the tripleo job | 18:17 |
| mwhahaha | will have to tack this down | 18:17 |
| yoctozepto | it's ok, nothing super hectic, I attacked #centos-devel channel too | 18:18 |
| mwhahaha | its coming in via tripleo because we configure ceph repos by default | 18:19 |
| yoctozepto | yeah, I know | 18:19 |
| mwhahaha | i know there was a patch to move kolla back to the storage sig stuff | 18:19 |
| yoctozepto | yup | 18:19 |
| yoctozepto | and we wanted ganesha | 18:19 |
| mwhahaha | but i thought i recalled a comment about it not being ok until friday | 18:19 |
| yoctozepto | so they delivered | 18:19 |
| mwhahaha | "delivered" | 18:19 |
| yoctozepto | but apparently something went awry | 18:19 |
| yoctozepto | yes, exactly | 18:20 |
| yoctozepto | Friday was the day | 18:20 |
| yoctozepto | not Wednesday | 18:20 |
| yoctozepto | so we broken till Friday ;D | 18:20 |
| mwhahaha | sounds about right | 18:21 |
| *** luksky has joined #openstack-kolla | 18:30 | |
| mnasiadka | Talk to you next week guys, take care :) | 18:35 |
| *** ivve has joined #openstack-kolla | 18:50 | |
| *** Wasaac has joined #openstack-kolla | 19:09 | |
| *** JqckB has joined #openstack-kolla | 19:13 | |
| *** Wasaac has quit IRC | 19:14 | |
| *** ivve has quit IRC | 19:16 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Clean up test process https://review.opendev.org/664580 | 19:34 |
| openstackgerrit | Michael Bayer proposed openstack/kolla master: Add python-sqlalchemy-collectd client https://review.opendev.org/659678 | 19:44 |
| *** Sravan has quit IRC | 19:46 | |
| *** kplant has quit IRC | 19:47 | |
| *** Sravan has joined #openstack-kolla | 19:59 | |
| *** Wasaac has joined #openstack-kolla | 19:59 | |
| *** hamzaachi has joined #openstack-kolla | 20:03 | |
| *** Sravan has quit IRC | 20:05 | |
| *** gfhellma__ has quit IRC | 20:07 | |
| *** schwicht has quit IRC | 20:08 | |
| openstackgerrit | Gaëtan Trellu proposed openstack/kolla-ansible master: Fix Qinling deployment and reconfigure https://review.opendev.org/666437 | 20:19 |
| *** hamzaachi has quit IRC | 20:20 | |
| *** hamzaachi has joined #openstack-kolla | 20:21 | |
| *** gfhellma has joined #openstack-kolla | 20:22 | |
| *** cah_link has joined #openstack-kolla | 20:22 | |
| *** gfhellma_ has joined #openstack-kolla | 20:24 | |
| *** gfhellma has quit IRC | 20:28 | |
| *** schwicht has joined #openstack-kolla | 20:30 | |
| *** cah_link has quit IRC | 20:31 | |
| *** cah_link has joined #openstack-kolla | 20:34 | |
| *** cah_link has quit IRC | 20:39 | |
| *** gfhellma_ has quit IRC | 20:39 | |
| *** Wasaac has quit IRC | 20:42 | |
| *** cah_link has joined #openstack-kolla | 20:45 | |
| *** schwicht has quit IRC | 20:46 | |
| *** whoami-rajat has quit IRC | 20:47 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: Hide logs when looping over passwords https://review.opendev.org/666130 | 20:58 |
| *** hamzaachi has quit IRC | 21:05 | |
| *** schwicht has joined #openstack-kolla | 21:08 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: Support multi-region discovery of Nova cells https://review.opendev.org/666127 | 21:11 |
| goldyfruit | Is there a way to keep using python 2.7 ? | 21:14 |
| goldyfruit | Like having a variable to export ? | 21:14 |
| *** kplant has joined #openstack-kolla | 21:18 | |
| *** hamzaachi has joined #openstack-kolla | 21:20 | |
| *** cah_link has quit IRC | 21:23 | |
| *** Sravan has joined #openstack-kolla | 21:24 | |
| goldyfruit | I don't understand why some images are build in python 2.7 and others in python 3.x | 21:28 |
| goldyfruit | I'm using master, keystone is under 2.7 and qinling under 3.6 | 21:28 |
| goldyfruit | Base Ubuntu 18.04 | 21:29 |
| goldyfruit | Install type source | 21:29 |
| goldyfruit | The only solution I found so far is to force 2.7 here https://github.com/openstack/kolla/blob/d436001b43f0a7e3c161ac4a2c5d09d36c6f1015/kolla/image/build.py#L717 | 21:38 |
| *** cah_link has joined #openstack-kolla | 21:39 | |
| *** gfhellma has joined #openstack-kolla | 21:42 | |
| goldyfruit | Which force KOLLA_DISTRO_PYTHON_VERSION=2.7 variable | 21:44 |
| openstackgerrit | Merged openstack/kolla stable/queens: Add sha256 in ceph fetch keys https://review.opendev.org/666084 | 21:51 |
| openstackgerrit | Merged openstack/kolla stable/rocky: [neutron] Do not migrate all subprojects https://review.opendev.org/666076 | 21:51 |
| *** goldyfruit has quit IRC | 21:54 | |
| *** JamesBen_ has quit IRC | 21:54 | |
| *** cah_link has quit IRC | 21:58 | |
| mloza | hello, I have 3 controllers, I'm getting tons of RMQ broken pipe in one controller, how do I repair this? | 22:17 |
| *** zbr|ruck has quit IRC | 22:17 | |
| openstackgerrit | Merged openstack/kolla stable/queens: Remove HTTP configuration shipped in openstack-nova-placement-api RPM https://review.opendev.org/666002 | 22:21 |
| *** JqckB has quit IRC | 22:22 | |
| *** hamzaachi has quit IRC | 22:30 | |
| *** luksky has quit IRC | 22:39 | |
| *** Wasaac has joined #openstack-kolla | 22:43 | |
| *** pomac has quit IRC | 22:43 | |
| *** Wasaac has quit IRC | 22:47 | |
| *** pomac has joined #openstack-kolla | 22:50 | |
| *** gfhellma_ has joined #openstack-kolla | 22:52 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Fix Blazar Nova aggregate in multi-region setup https://review.opendev.org/666132 | 22:53 |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: Fix Blazar Nova aggregate in multi-region setup https://review.opendev.org/666133 | 22:53 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Hide logs when looping over passwords https://review.opendev.org/666129 | 22:53 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Fix monasca grafana organisation check https://review.opendev.org/666124 | 22:53 |
| *** gfhellma has quit IRC | 22:55 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Hide logs when looping over passwords https://review.opendev.org/666128 | 22:58 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix monasca grafana organisation check https://review.opendev.org/666123 | 22:58 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Fixes VMTP deploys. https://review.opendev.org/666122 | 22:58 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fixes VMTP deploys. https://review.opendev.org/666121 | 22:58 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Add ansible_nodename (system hostname) to /etc/hosts https://review.opendev.org/666098 | 23:01 |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: Add ansible_nodename (system hostname) to /etc/hosts https://review.opendev.org/666099 | 23:01 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: [heat] Multi-region support for bootstrap https://review.opendev.org/666135 | 23:01 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Update service_provider for FWaaS v2 https://review.opendev.org/666118 | 23:10 |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: [heat] Multi-region support for bootstrap https://review.opendev.org/666136 | 23:10 |
| *** Sravan has quit IRC | 23:11 | |
| *** JamesBenson has joined #openstack-kolla | 23:12 | |
| *** JamesBenson has quit IRC | 23:17 | |
| *** Sravan has joined #openstack-kolla | 23:20 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix mongo command for supporting other port https://review.opendev.org/666148 | 23:22 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix mongo command for checking replication status https://review.opendev.org/666144 | 23:22 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Fix mongo command for supporting other port https://review.opendev.org/666149 | 23:22 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Update service_provider for FWaaS v2 https://review.opendev.org/666200 | 23:22 |
| openstackgerrit | Merged openstack/kolla-ansible stable/queens: nova: Fix DBNotAllowed during compute startup https://review.opendev.org/666096 | 23:22 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Do not create Keystone domain-specific config directory if useless https://review.opendev.org/666059 | 23:22 |
| *** gfhellma_ has quit IRC | 23:25 | |
| *** gfhellma_ has joined #openstack-kolla | 23:25 | |
| *** gfhellma__ has joined #openstack-kolla | 23:30 | |
| *** gfhellma_ has quit IRC | 23:31 | |
| *** gfhellma_ has joined #openstack-kolla | 23:43 | |
| *** gfhellma__ has quit IRC | 23:47 | |
| *** gfhellma_ has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!