Wednesday, 2019-08-07

*** kevinz has joined #openstack-kolla00:02
*** logan- has quit IRC00:12
*** logan- has joined #openstack-kolla00:13
*** ivve has quit IRC00:21
*** hongbin has joined #openstack-kolla00:34
*** gregoryo has joined #openstack-kolla00:41
*** kevinz has quit IRC01:04
*** igordc has quit IRC01:13
*** igordc has joined #openstack-kolla01:15
*** gregoryo has quit IRC01:22
*** igordc has quit IRC01:23
*** pcaruana has quit IRC01:26
*** gregoryo has joined #openstack-kolla01:50
*** kevinz has joined #openstack-kolla02:04
*** kevinz has quit IRC02:38
*** BjoernT has quit IRC02:43
*** BjoernT_ has joined #openstack-kolla02:43
*** BjoernT_ has quit IRC02:47
*** igordc has joined #openstack-kolla02:49
*** BjoernT has joined #openstack-kolla02:52
*** whoami-rajat has joined #openstack-kolla03:19
*** pcaruana has joined #openstack-kolla03:44
*** hongbin has quit IRC03:49
*** BjoernT has quit IRC04:12
*** skramaja has joined #openstack-kolla04:27
*** iclon__ has joined #openstack-kolla04:41
*** iclon_ has quit IRC04:43
*** shyamb has joined #openstack-kolla04:55
*** icarusfactor has joined #openstack-kolla05:08
*** factor has quit IRC05:08
*** Luzi has joined #openstack-kolla05:19
*** igordc has quit IRC05:31
shyambDo we have container for pacemaker?05:37
shyambor we run pacemaker as service?05:37
shyambdirectly on host05:38
*** cah_link has joined #openstack-kolla05:43
*** shyamb has quit IRC05:44
*** shyamb has joined #openstack-kolla05:53
*** icarusfactor has quit IRC06:01
*** factor has joined #openstack-kolla06:02
*** shyamb has quit IRC06:05
*** shyam89 has joined #openstack-kolla06:05
*** gregoryo has quit IRC06:06
*** dr_feelgood has joined #openstack-kolla06:28
*** dpawlik has joined #openstack-kolla06:46
mnasiadkawe don't run pacemaker06:52
mnasiadkatripleo does06:52
*** rpittau|afk is now known as rpittau07:07
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: Add 'allow *' to getting ceph mds keyring  https://review.opendev.org/67504007:13
*** shyam89 has quit IRC07:14
openstackgerritRadosław Piliszek proposed openstack/kolla stable/rocky: ceph mds cap 'allow *' in extend_start  https://review.opendev.org/67504207:17
yoctozeptomnasiadka: for rocky centos requires rpcbind for ganesha too ;-(07:17
yoctozeptoadditionally one has to start it because just installing will leave it stopped07:18
yoctozeptoargh07:18
yoctozepto;/07:19
openstackgerritRadosław Piliszek proposed openstack/kolla stable/queens: ceph mds cap 'allow *' in extend_start  https://review.opendev.org/67504307:22
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/queens: Add 'allow *' to getting ceph mds keyring  https://review.opendev.org/67504507:25
hrwmorning07:27
*** dr_feelgood has quit IRC07:31
*** shyam89 has joined #openstack-kolla07:33
*** dr_feelgood has joined #openstack-kolla07:33
mnasiadkayoctozepto: in rocky probably we have luminous, maybe attached nfs-ganesha version is also oldish :)07:33
*** ivve has joined #openstack-kolla07:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: Rocky-only: add rpcbind to CentOS as well  https://review.opendev.org/67504807:36
yoctozeptomnasiadka: yeah, figured that much07:36
stingrayzamorning o/07:38
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: Add ceph-mds/rgw/nfs to gate  https://review.opendev.org/67447907:38
*** kevinz has joined #openstack-kolla07:39
*** jistr is now known as jistr|afk07:42
yoctozeptomgoddard, mnasiadka, hrw: https://review.opendev.org/673302 and then we have green master07:42
hrw+207:43
yoctozeptoand rocky has like 5 patches07:43
yoctozepto4 k-a, 1 k07:43
yoctozeptowith them stein should be green again07:43
yoctozeptoand rocky obviously too07:44
*** spsurya has joined #openstack-kolla07:50
mgoddardmorning07:52
spsuryamorning mgoddard07:53
mgoddardhi spsurya07:53
spsuryaindeed evening for07:53
*** shyam89 has quit IRC07:54
spsuryaschedule is out  https://www.openstack.org/summit/shanghai-2019/summit-schedule/#day=2019-11-0507:55
hrwhi mgoddard spsurya07:59
mgoddardspsurya: no search results for 'kolla'. I checked submissions and there were a lot07:59
spsuryahi hrw07:59
hrwopenstack summit is one of those events where I do not even look at schedule08:00
* spsurya mgoddard: yes, no kolla keyword this08:00
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: Rocky-only: add rpcbind to CentOS as well  https://review.opendev.org/67504808:00
yoctozeptohi mgoddard spsurya08:04
yoctozeptoso where did all the kolla submissions go? :-(08:04
spsuryayoctozepto: might be in queue for next summit.08:06
spsurya:(08:06
spsuryamay be this time  didn't get slot08:07
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Enable iscsid on cinder-backup hosts  https://review.opendev.org/67505208:09
*** Wasaac has joined #openstack-kolla08:11
yoctozeptospsurya: well, that's sad :-(08:12
spsuryayoctozepto: yeah08:13
yoctozeptomgoddard: https://bugs.launchpad.net/kolla-ansible/+bug/1838624 shouldn't we be targetting things at train-1 ?08:14
openstackLaunchpad bug 1838624 in kolla-ansible train "Cinder-backup iscsi lvm backend" [Medium,In progress] - Assigned to Mark Goddard (mgoddard)08:14
mgoddardyoctozepto: maybe. I don't think we generally bother with milestones in this project though08:15
yoctozeptomgoddard: ok ;-)08:16
mgoddardyoctozepto: plus, both train-1 and train-2 have been and gone :D https://releases.openstack.org/train/schedule.html08:16
mgoddardI might just delete them from LP actually08:17
*** dougsz has joined #openstack-kolla08:21
*** shyam89 has joined #openstack-kolla08:22
*** dr_feelgood has quit IRC08:23
yoctozeptomgoddard: yeah, go ahead08:25
yoctozeptoyou just reminded me how far behind we are ;p08:25
*** dr_feelgood has joined #openstack-kolla08:26
mgoddarddone08:26
mgoddardyoctozepto: yeah, this is the last development milestone for train08:26
mgoddardwe're normally a little behind, but...08:27
mgoddardyoctozepto: probably a good point to reprioritise and make sure we're reviewing the patches we want to land08:27
yoctozeptomgoddard: good point for today's meeting ;-)08:31
*** shyamb has joined #openstack-kolla08:32
*** shyam89 has quit IRC08:34
yoctozeptomgoddard, hrw: re: docker-ce - loose note: during today's upgrade to my Cisco VIRL I noticed they migrated from distro-provided docker to docker-ce08:36
yoctozeptofact is they use ubuntu 16.04 still08:36
hrwyoctozepto: 16.04? we do not use it anymore08:36
yoctozeptoyeah, the point is thought that enterprises are not afraid of docker-ce - hence we probably should not be either08:37
yoctozepto;-)08:38
hrw;d08:38
*** dr_feelgood has quit IRC08:39
hrwwill look later08:40
hrwtoday I have CI morning.08:41
hrwjust not openstack CI :D08:41
*** lemko has joined #openstack-kolla08:43
yoctozeptohrw: pity!08:44
hrwyoctozepto: removing old jobs is a pure pleasure08:44
*** kevinz has quit IRC08:48
*** kevinz has joined #openstack-kolla08:51
openstackgerritRadosław Piliszek proposed openstack/kolla stable/stein: Stein-only: update packages  https://review.opendev.org/67506208:53
yoctozepto^ we should probably have some proposal bot for that stuff08:53
mnasiadkayoctozepto: we have, it's called tools/version-check.py ;)09:02
mnasiadkaI'm in the middle of making it better09:02
yoctozeptomnasiadka: I used that09:03
yoctozeptothe thing is I had to use it to learn there are pending versions09:03
shyambHi09:03
yoctozepto;-)09:03
shyambgetting "RemoteError: Remote error: IncompatibleObjectVersion Version 1.1 of CellMappingList is not supported"09:03
shyambin one of my custom container,09:03
shyambUsed rocky base image09:03
shyambrunning this container on queens09:03
openstackgerritMerged openstack/kolla-ansible stable/stein: Add ceph-mds/rgw/nfs to gate  https://review.opendev.org/67330209:10
*** shyamb has quit IRC09:10
*** dr_feelgood has joined #openstack-kolla09:15
*** dr_feelgood has quit IRC09:21
*** dr_feelgood has joined #openstack-kolla09:22
*** shyamb has joined #openstack-kolla09:24
*** gfidente has joined #openstack-kolla09:36
*** dr_feelgood has quit IRC09:39
*** jistr|afk is now known as jistr09:54
*** shyamb has quit IRC09:57
*** aissat-mus has joined #openstack-kolla10:00
aissat-musHi, is there any tips to upgrade from Rocky to Stein?10:07
aissat-musI notice that Docker version has changed. do I need to bootstrap my servers?10:08
*** aissat-mus has quit IRC10:20
*** aissat-mus has joined #openstack-kolla10:26
*** Wasaac has quit IRC10:34
*** Wasaac has joined #openstack-kolla10:38
mgoddardaissat-mus: if you want to upgrade docker, you can run bootstrap-servers again. Please take note of the notes here: https://docs.openstack.org/kolla-ansible/latest/reference/deployment-and-bootstrapping/bootstrap-servers.html#subsequent-bootstrap-considerations10:39
*** shyamb has joined #openstack-kolla10:50
*** sean-k-mooney has quit IRC10:52
*** aissat-mus43 has joined #openstack-kolla10:59
aissat-mus43mgoddard many thanks11:01
*** aissat-mus has quit IRC11:02
*** sean-k-mooney has joined #openstack-kolla11:09
hrwmgoddard: https://review.opendev.org/667904 ("Fixes for MariaDB bootstrap and recovery") is master, rocky, queens. we do not need that in stein?11:18
mgoddardhrw: no, that patch was adapted from the mariadb stein upgrade fix patch11:19
hrw2ok11:19
*** kplant has joined #openstack-kolla11:19
*** lemko has quit IRC11:23
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: repair ceph_nfs container start failed  https://review.opendev.org/67509611:29
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: Add ceph-mds/rgw/nfs to gate  https://review.opendev.org/67447911:31
yoctozeptomonkey-patching rocky11:33
yoctozeptothis only shows it is not worth adding anything without creating tests for it and running them on quasi-regular basis11:33
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add support for sha256 in ceph key distribution  https://review.opendev.org/65584511:40
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add support for sha256 in ceph key distribution  https://review.opendev.org/65584511:40
*** shyamb has quit IRC11:40
*** shyamb has joined #openstack-kolla11:41
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Stop using MountFlags=shared in Docker configuration  https://review.opendev.org/67458711:51
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270011:51
*** Wasaac has quit IRC11:53
*** Wasaac_ has joined #openstack-kolla11:53
*** Wasaac_ has quit IRC11:54
*** Wasaac has joined #openstack-kolla11:55
hrwmgoddard, yoctozepto: I think that your comments got included in current ver. still needs work11:56
*** shyam89 has joined #openstack-kolla11:56
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add support for sha256 in ceph key distribution  https://review.opendev.org/65584511:57
*** shyamb has quit IRC12:01
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270012:11
hrwdocker_runtime_directory handled12:11
hrw'docker_custom_option' is something I would rather deprecate as 'docker_config' is now variable so can be populated with whatever12:12
yoctozeptohrw: will see your change later12:14
hrwyoctozepto: thanks12:15
yoctozeptoI did a few rebuilds from kolla recently and reported now: https://bugs.launchpad.net/kolla/+bug/183931912:15
openstackLaunchpad bug 1839319 in kolla "Inefficient usage of cache due to "ADD plugins-archive /"" [Undecided,New]12:15
yoctozeptoit bugged me that it seems to sometimes do too much12:16
yoctozeptoand found the reason ;D12:16
yoctozeptonow somebody to fix it though12:16
hrwyoctozepto: looks painful in situations with lot of images on host.12:18
* hrw digs for docker_custom_option and R@#%@#$ 12:19
hrwdocker_custom_option: "--insecure-registry {{ api_interface_address }}:4000"12:19
openstackgerritMerged openstack/kolla-ansible stable/stein: Add mon address to ceph release version check  https://review.opendev.org/67489112:19
openstackgerritMerged openstack/kolla-ansible stable/rocky: Add 'allow *' to getting ceph mds keyring  https://review.opendev.org/67504012:19
hrwshouldn't it be instead: docker_registry: "{{ api_interface_address }}:4000"12:19
hrw?12:19
yoctozeptoskip_existing also seems broken and cannot help me, meh12:20
hrwskip_existing is evil.12:20
yoctozeptopain points with motivation to fix12:21
hrwas it does 'docker images' every time. on arm64 it means +40s per check12:21
yoctozeptohrw: insecure is orthogonal to setting the registry to use12:21
yoctozeptooh my12:21
yoctozeptoI'm actually running stein now12:22
yoctozeptoso even without yout latest fixes12:22
yoctozeptoah, meh12:22
yoctozeptoalso, regexp matching is bad, try to build keystone12:22
yoctozeptoit builds barbican as well12:22
yoctozeptobecause one of its parts matches12:23
hrwyoctozepto: if we have 'docker_registry' set then we pass it as 'insecure-registries' anyway12:23
*** aissat-mus43 has quit IRC12:23
hrwyoctozepto: tell me more... ^grafana is what I have in 'linaro' profile12:23
yoctozeptohrw: yes, that\s the default12:23
yoctozeptoyeah, though it looks weird when defined in profile12:24
hrwyoctozepto: so it is same way on CI12:24
yoctozeptobecause it's a comma-delimited list12:24
hrwas tests/templates/globals-default.j2 is CI one12:24
yoctozeptothere is an issue in that CI12:24
yoctozeptoI tried to fix12:24
yoctozeptomgoddard reversed it12:25
yoctozeptobecause it actually depended on order12:25
yoctozeptoI forgot the details12:25
openstackgerritRafael Weingärtner proposed openstack/kolla-ansible master: Standardize the configuration of "oslo_messaging" section  https://review.opendev.org/67062612:25
yoctozeptoyou can find it in log12:25
hrwdocker--12:25
yoctozeptoi'm looking forward to buildah12:25
yoctozeptothough nobody seems to have taken the ball12:26
yoctozeptopicked up* the ball12:26
hrw-docker_custom_option: " -H unix:///var/run/docker.sock -H tcp://{{ api_interface_address }}:2375 --cluster-store=etcd://{{ api_interface_address }}:2379 --cluster-advertise={{ api_interface_address }}:2375"12:27
hrw+docker_config: [ "host": "unix:///var/run/docker.sock", "host": "tcp://{{ api_interface_address }}:2375", "cluster-store": "etcd://{{ api_interface_address }}:2379", "cluster-advertise={{ api_interface_address }}:2375"]12:27
hrwlet's see how CI will react12:27
hrwafter one more change12:27
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270012:28
openstackgerritMerged openstack/kolla-ansible stable/rocky: ceph: fixes to deployment and upgrade  https://review.opendev.org/67488912:29
openstackgerritMerged openstack/kolla-ansible stable/queens: Fixes for MariaDB bootstrap and recovery  https://review.opendev.org/66970312:29
openstackgerritMerged openstack/kolla stable/rocky: ceph mds cap 'allow *' in extend_start  https://review.opendev.org/67504212:29
openstackgerritMerged openstack/kolla stable/stein: Stein-only: update packages  https://review.opendev.org/67506212:29
*** rafaelweingartne has joined #openstack-kolla12:32
*** jistr is now known as jistr|call12:37
*** shyam89 has quit IRC12:39
*** zbr has quit IRC12:46
mnasiadkayoctozepto: you're waiting for buildah in kolla CI?12:48
*** lemko has joined #openstack-kolla12:48
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use default BGP speaker driver in bgp_dragent.ini  https://review.opendev.org/67511112:52
*** shyamb has joined #openstack-kolla12:55
goldyfruit_o/13:02
*** zbr has joined #openstack-kolla13:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270013:05
hrwone day I will write good ansible code on second attempt13:06
mnasiadkaI always end up on 7th attempt13:20
*** shyamb has quit IRC13:23
hrwmnasiadka: 7th as '7th is perfect' or 'abandon instead of doing 8th'?13:26
kplantdefinitely the 'if they don't like it, they can fix it'13:26
mnasiadkahrw: rather I submit 7th, want to abandon but somebody gives +2 faster :)13:27
*** dr_feelgood has joined #openstack-kolla13:27
yoctozeptomnasiadka: buildah support in kolla in general13:29
mnasiadkayoctozepto: do you think it will build faster, or what?:)13:31
yoctozeptomnasiadka: worth checking out, at least it's daemonless,  having to have docker daemon running for building is so-so ;p13:33
hrwkolla is not multicore friendly13:33
hrwthere are not so many images not based on openstack-base one. and this one takes far too long time13:34
*** kevinz has quit IRC13:35
*** BjoernT_ has joined #openstack-kolla13:36
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use default BGP speaker driver in bgp_dragent.ini  https://review.opendev.org/67511113:37
yoctozeptohrw: you would probably need to run some clustering algorithm on deps of different projects to get a good optimization13:38
yoctozeptostill, it could be quite flaky13:39
hrwyoctozepto: or scrap pypi.org and do it from scratch13:39
hrwas now it just gets what authors send13:40
hrwinstead of taking sources and doing binary builds on their infra13:40
hrwso adding new arch would be just adding machines to infra13:40
yoctozeptoI don't follow, suddenly you want to rebuild everything? ;D13:42
hrwyoctozepto: python module author usually has x86 machine. do source tarball, wheel and upload to pypi. so when you 'pip install somemodule' you get wheel and it is done in 2s. then on !x86 I have to install python-dev and bunch of other ones and wait 20 minutes for 'pip install somemodule'13:44
*** factor has quit IRC13:44
hrwyoctozepto: when I start 'kolla build everything' then build queue is nearly empty until openstack-base finish. once it is done then real building starts13:45
*** factor has joined #openstack-kolla13:45
hrwINFO:kolla.common.utils.openstack-base:Successfully tagged 10.101.16.1:5000/haerwu/ubuntu-source-openstack-base:8.1.013:45
hrwINFO:kolla.common.utils.openstack-base:Built at 2019-08-02 12:22:44.144467 (took 0:16:17.522189)13:45
hrwcompare with CI:13:46
hrwINFO:kolla.common.utils.openstack-base:Successfully tagged kolla/ubuntu-source-openstack-base:8.1.013:46
hrwINFO:kolla.common.utils.openstack-base:Built at 2019-08-07 10:17:24.513995 (took 0:04:59.628040)13:46
*** dcapone2004 has joined #openstack-kolla13:48
dcapone2004we are getting some horizon permission errors on a fresh AIO install of stein using centos and source builds using kolla ansible to deploy...I was wondering if anyone else was experiencing this?13:50
hrwdcapone2004: let me check where my daugther keeps glass ball...13:51
dcapone2004lol....is that asking me to post specifics?  because there are a lot and when exploring the filesystem it appears that a lot of the stuff is owned by root and I can move past each error by changing ownership to horizon13:52
yoctozeptohrw: ah, now I get you13:54
yoctozeptodcapone2004: well, at least the original error messages would be nice13:55
hrwdcapone2004: yes, it is sarcastic way of saying 'show logs'13:55
hrwdcapone2004: but my daughter has real glass ball too ;D13:55
yoctozeptohrw: frozen style? ;p13:55
hrwyoctozepto: bought before Frozen the movie13:56
dcapone2004yeah, I got the reference and enjoyed it, but they seem to be neverending...but I will give a short history to not overly bore you....I deployed using the AIO file, some customizations to globals.yml to enable/disable certain things13:56
dcapone2004after deployment everything was working when using the command line, but horizon wasn't working, I couldn't create networks, if I used the run once script to setup the demo stuff, I couldn't see network detail (I could see the network, but if I selected the network in horizon and clicked the subnets tab, nothing happened)13:57
yoctozeptodcapone2004: as for stein+centos+source - this is what I am running and horizon works fine the for things like nova, glance or cinder at least13:57
*** whoami-rajat has quit IRC13:57
yoctozeptoand neutron13:57
dcapone2004so I wiped the demo stuff, and recreated images, networks, routers, volumes, vms from the command line and everything worked but still horizon was not picking much of it up, so I entered the horizon container and updated local_settings and turned debugging on13:58
mnasiadkayoctozepto: that shouldn't be complicated, python-podman and go :)13:59
dcapone2004when I did I got this error14:00
dcapone2004OfflineGenerationError at /project/14:00
dcapone2004You have offline compression enabled but key "3dcdc14a426856978c36be7fea160fa32caa3b3167f519795a0c3e31bbe119f8" is missing from offline manifest. You may need to run "python manage.py compress". Here is the original content:14:00
dcapone2004which I googled and there was a really old bug in Horizon that had caused this in the past....https://bugs.launchpad.net/horizon/+bug/115576514:00
openstackLaunchpad bug 1155765 in OpenStack Dashboard (Horizon) "Offline compression enabled but key is missing from offline manifest" [Medium,Fix released] - Assigned to Yves-Gwenael Bourhis (yves-gwenael-bourhis)14:00
dcapone2004I also tried to force the offline compression and that is throwing an error14:01
kplantcan we see your globals.yml? i'm constantly deploying kolla aios with centos+stein+source with zero problems14:02
yoctozeptomnasiadka: yeah, but this is lo-prio14:02
hrwansible... true/yes/on/file_not_found14:05
hrw    "docker_registry_insecure": "yes"14:05
hrwfatal: [c4n1.mlab.devplanet.cf]: FAILED! => {"msg": "The conditional check 'docker_registry_insecure' failed. The error was: error while evaluating conditional (docker_registry_insecure): 'yes' is undefined\n\nThe error appears to have been in '/home/hrw/devel/kolla/venv3/share/kolla-ansible/ansible/roles/baremetal/tasks/post-install.yml': line 78, column 3, but may\nbe elsewhere in the file14:05
hrwdepending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: setup docker registries\n  ^ here\n"}14:05
hrwafwiefw14:05
dcapone2004kplant: yes working on a pastebin now14:06
yoctozeptohrw: where is that to be seen in code14:07
hrwyoctozepto: sorting out now. https://review.opendev.org/672700 + all.yml + post-install.yml14:09
yoctozeptohrw: commented14:10
*** kevinz has joined #openstack-kolla14:10
hrws/yes/True in all.yml works14:10
dcapone2004https://pastebin.com/6S05jRyK14:10
hrwyoctozepto: thanks. that works too14:11
*** dpawlik has quit IRC14:15
*** kevinz has quit IRC14:15
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use default BGP speaker driver in bgp_dragent.ini  https://review.opendev.org/67511114:19
dcapone2004kplant: anything?  I feel like it is a pretty stock globals.yml file14:19
kplantdcapone2004: sorry, didn't see your paste14:19
*** kevinz has joined #openstack-kolla14:20
yoctozeptodcapone2004: ah, you enabled some more network-related plugins in horizon14:22
kplanti was just looking at vpnaas, yeah14:22
yoctozeptomay you post the raw horizon logs14:22
kplanthaven't played with that myself14:22
dcapone2004yes, and oddly enough, those pages work in horizon lol14:22
yoctozeptoso which do not?14:22
dcapone2004almost all of the core pages14:23
yoctozeptoI got an impression you had no working horizon14:23
yoctozeptoah14:23
dcapone2004networks, routers, images14:23
yoctozeptothat would be odd if enabling plugins disabled all the rest14:23
hrwdocker--14:24
yoctozeptobut you only enabled vpnaas14:24
dcapone2004let me redeploy without vpnaas and see if everything else starts working again, I do not really need vpnaas for this poc, I am just looking to learn kolla ansible because I am trying to transition from RDO packstack14:24
yoctozeptook, welcome aboard :D14:24
yoctozeptogood choice14:24
hrwansible has 'to_(nice_)json' filter. which gives json which docker choke on14:25
stingrayza+1 k-a has been the best thing in my openstack life since, well, openstack itself ;)14:25
dcapone2004yoctozepto: not sure what you mean on I only enabled vpnaas, I thought the core horizon stuff was all enabled as well?14:25
dcapone2004I keep reading how great it is.....first go around hasn't quite impressed me lol, so I figure I have some learning to do14:26
dcapone2004doesn't this #enable_horizon: "{{ enable_openstack_core | bool }}" default enable all of the core horizon functionality14:27
dcapone2004like the networks, routers, images, etc14:27
yoctozeptodcapone2004: no, I was about those uncommented horizon lines14:28
yoctozeptoand then only vpnaas was really enabled14:28
yoctozeptocore horizon is enabled14:29
yoctozeptocould be something wrong with vpnaas plugins breaking core14:29
yoctozeptobut need you to post logs14:29
dcapone2004yeah fwaas as a service didn't deploy properly....another problem I encountered yesterday....that was more fatal in that neutron wouldn't stay alive as the fwaas plugin parameters were not set correctly in the container14:29
yoctozeptonot only story ;-)14:29
*** Luzi has quit IRC14:30
openstackgerritRafael Weingärtner proposed openstack/kolla-ansible master: Standardize the configuration of "oslo_messaging" section  https://review.opendev.org/67062614:30
dcapone2004no I disabled fwaas but I thought the "or" logic of the config file meant I didn't have to specifically comment back out the horizon lines as it would take it as disabled since the projects themselves were disabled14:30
dcapone2004I'm getting a lot of these instead of horizon when debugging is enabled:14:31
dcapone2004OSError at /project/networks/60366de5-d553-4210-9847-a211bd331231/detail14:31
dcapone2004[Errno 13] Permission denied: '/var/lib/kolla/venv/lib/python2.7/site-packages/static/dashboard/js/1ba732d085e1.js'14:31
dcapone2004Request Method:GET14:31
dcapone2004Request URL:http://173.237.205.46/project/networks/60366de5-d553-4210-9847-a211bd331231/detail14:31
dcapone2004Django Version:1.11.2014:31
dcapone2004Exception Type:OSError14:31
dcapone2004Exception Value:14:31
dcapone2004[Errno 13] Permission denied: '/var/lib/kolla/venv/lib/python2.7/site-packages/static/dashboard/js/1ba732d085e1.js'14:31
dcapone2004Exception Location:/var/lib/kolla/venv/lib/python2.7/site-packages/django/core/files/storage.py in _save, line 34714:31
dcapone2004Python Executable:/var/lib/kolla/venv/bin/python14:31
dcapone2004inside of horizon*14:31
yoctozeptodcapone2004: yeah, the magic works14:32
dcapone2004docker ps14:33
dcapone2004redeploying without vpnaas and seeing what happens14:34
*** jistr|call is now known as jistr14:36
hrwok. current version passes bootstrap14:36
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270014:36
*** Wasaac has quit IRC14:38
*** dr_feelgood has quit IRC14:38
*** Wasaac has joined #openstack-kolla14:39
*** dr_feelgood has joined #openstack-kolla14:40
*** chason has joined #openstack-kolla14:42
hrwwho will chair today? I have other meeting at same time  ;(14:42
yoctozeptomgoddard as always?14:43
mgoddardyeah, I'm back full time now so I'll drive14:43
*** altlogbot_3 has quit IRC14:46
*** altlogbot_0 has joined #openstack-kolla14:47
*** cah_link has quit IRC14:51
mgoddardmeeting in 10 minutes14:51
mgoddardping mgoddard mnasiadka hrw egonzalez yoctozepto ^14:51
dcapone2004so before your meeting which is way more important than me....wanted to give an update that disable vpnaas fixed everything with Horizon14:52
hrwdcapone2004: :)14:52
dcapone2004so I'm unsure of the status of the fwaas and vpnaas projects as they both seem to go to depracation and them come out of depracation on the regular14:52
yoctozeptopong mgoddard14:53
hrwdcapone2004: if you enable neutron-vpnaas-plugin then it will enable horizon plugin automatically iirc14:53
mgoddardwell played yoctozepto14:53
dcapone2004so I don't know if my issues with both of these project deployments should be reported as bugs with kolla ansible or if they are simply not supported14:53
yoctozeptodcapone2004: report bug14:53
mnasiadkavpnaas... that's a very... unfortunate project14:54
dcapone2004hrw: not sure if you were here for my other messages, but when I enabled vpnass in the kolla's globals.yml file, everything else in horizon broke14:54
hrwok14:54
dcapone2004that is why I am asking if it should be reported as a bug, or if support is basically dropped14:54
dcapone2004I also had problems with fwaas14:55
goldyfruit_dcapone2004, vpnaas has a long story14:55
yoctozeptodcapone2004: I think octavia took over some neutron plugins14:55
goldyfruit_been a L3 agent then dropped from neutron support then added back as a neutron extension14:55
yoctozeptoah, it was lbaas14:56
mnasiadkayoctozepto: not vpnaas, it's in an semi-orphan state14:56
yoctozeptomnasiadka: yeah, reading now14:56
mnasiadkayoctozepto: neutron team did some fixes to make it work,  but they don't really care about it14:56
yoctozeptoanyways, it should not break whole horizon14:56
dcapone2004it doesn't in packstack :-)14:56
mnasiadkabut the permission denied looks interesting14:56
yoctozeptolooks like deployment issue14:57
mnasiadkaand why the heck is it trying to write in site-packages :D14:57
yoctozeptoprobably fixable on our side then14:57
mnasiadkamaybe some config14:57
yoctozeptomnasiadka: django caching its scripts14:57
*** factor has quit IRC14:57
dcapone2004don't waste many resources on it given the projects status14:57
mnasiadkayoctozepto: every time I deployed fresh kolla release - horizon was broken, so no worries :)14:57
dcapone2004I just enabled it to have something to change in the globals.yml to learn kolla ansible14:58
dcapone2004fwaas I am more interested in because I bleive fwaas2 is still ana ctive project14:58
yoctozeptomnasiadka: lolz, in fact k-a gave m first working horizon14:58
*** factor has joined #openstack-kolla14:58
NOXOiDfor what its worth i have vpnaas working fine with k-a centos source14:58
dcapone2004and that completely didn't work when enabled (although I haven't tried without vpnass enabled as well which will be my next test)14:59
dcapone2004noxoid: stein?14:59
NOXOiDyes14:59
*** nafiux has joined #openstack-kolla14:59
dcapone2004hmmm, how long along did you deploy?14:59
dcapone2004ago*14:59
NOXOiDmonth or so14:59
NOXOiDusing stein rc214:59
mnasiadkayoctozepto: I started deploying with k-a since Liberty or Mitaka I think :)14:59
dcapone2004so it could have broke recently15:00
dcapone2004I will attempt to reenable and deploy again15:00
NOXOiDpossibly, i have since "upgraded" to the official k-a stein release15:00
*** cah_link has joined #openstack-kolla15:00
dcapone2004just hit go button on redeploy with vpnaas enabled again and will see what happens15:01
hrw 8 files changed, 61 insertions(+), 30 deletions(-)15:01
mgoddardok, meeting time now - please continue this discussion afterwards15:01
mgoddard#startmeeting kolla15:01
openstackMeeting started Wed Aug  7 15:01:34 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
*** openstack changes topic to " (Meeting topic: kolla)"15:01
openstackThe meeting name has been set to 'kolla'15:01
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible master: Configure docker without guessing how it starts  https://review.opendev.org/67270015:01
mgoddard#topic rollcall15:01
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:01
mgoddard\o15:02
mnasiadkao/15:02
chasono/15:02
hrwo/15:02
Wasaaco/15:02
jovial[m]0/15:02
nafiuxo/15:02
yoctozeptoo/15:02
rafaelweingartnesorry, it is my first meeting. What is a rollcall?!15:02
WasaacIt's where everyone waves15:02
kplanto/15:03
yoctozeptoexactly!15:03
mgoddardrafaelweingartne: thanks for joining. Raise your hand: o/15:03
rafaelweingartneo/15:03
yoctozeptolike in a classroom15:03
nafiuxThis is my first meeting as well :)15:03
*** factor has quit IRC15:03
mgoddardit just records who is present15:03
mgoddardnice turnout today15:03
mgoddard#topic agenda15:03
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:03
yoctozeptomgoddard: everyone waiting for you probably15:03
mgoddard* Roll-call15:04
mgoddard* Announcements15:04
mgoddard** 99cloud working on kolla-cli - it lives on15:04
mgoddard** PTL (mgoddard) is back15:04
mgoddard** Added review priority for kayobe in gerrit15:04
mgoddard** Kayobe now covered in kolla meetings15:04
mgoddard* Review action items from last meeting15:04
mgoddard* Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:04
mgoddard* Kayobe Stein release status15:04
mgoddard* Train release planning15:04
mgoddard* Ocata EOL15:04
mgoddard* Shanghai summit15:04
mgoddard* Ceph ansible migration15:04
mgoddard#topic announcements15:04
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:04
mgoddard#info 99cloud working on kolla-cli - it lives on15:04
yoctozeptohurray15:04
mgoddardJeffrey4l_ is the only one I'm aware of who uses IRC15:05
mgoddardthanks for picking it up15:05
mgoddardI added a deliverable file to the releases repo for the Train cycle, so hopefully it will get its first release then15:05
mgoddard#info PTL (mgoddard) is back15:05
yoctozeptowelcome back15:06
yoctozeptoeveryone missed you15:06
goldyfruit_<315:06
mgoddardFamily all doing well, trying to get used to working on less sleep15:06
mgoddardaw thanks :)15:06
mgoddard#info Added review priority for kayobe in gerrit15:06
hrwmgoddard: sleep when baby sleeps15:07
*** Wasaac has quit IRC15:07
mgoddardThis puts kayobe in sync with other kolla projects - you can now set the review priority field in gerrit to highlight 'important' reviews15:07
mgoddard+1 for important, +2 for critical15:07
hrwmgoddard: and no longer x/kayobe but kolla/kayobe?15:07
yoctozeptoopenstack/15:08
*** Wasaac has joined #openstack-kolla15:08
mgoddardhrw: not yet, still waiting on gerrit restart :(15:08
hrwor. just no x/15:08
yoctozepto:-(15:08
mgoddardall patches in review and ready to go15:08
mgoddardmaybe I should nudge them15:08
mgoddard#action mgoddard to ask infra about restarting gerrit15:08
mgoddard#info Kayobe now covered in kolla meetings15:08
yoctozeptomgoddard: delivery of kayobe for train?15:08
mgoddardyoctozepto: yes - patch is approved, but wanted to wait for the rename15:09
mgoddardWe didn't do a proper introduction as I was away, but please welcome the kayobe team to our meeting15:10
mgoddardcurrently we have jovial[m] in attendance from the core team, and Wasaac is also a contributor15:10
jovial[m]hi - I've moved over from kayobe land15:10
hrwwelcome!15:10
yoctozeptowelcome jovial[m]15:10
mgoddarddougsz has a clash with monasca meeting and priteau is with a customer but should join next time15:11
yoctozeptomgoddard; I meant first official release :-)15:11
mgoddard(both cores)15:11
mgoddardyoctozepto: yeah15:11
*** cah_link has quit IRC15:11
yoctozeptogreat!15:11
yoctozeptois there a merger of cores?15:11
mgoddardall have contributed to kolla too, so I'm sure you recognise each other15:11
yoctozeptoyeah, sure - wondered if we end up in one team or still two separate?15:13
mgoddardI don't want to do an automatic merge of teams - let's keep it based on performance. Anyone who contributes & provides useful reviews on a regular basis can be considered core for that team15:13
mgoddardwe have several groups in gerrit - kolla-core, kolla-ansible-core, kolla-cli-core, kayobe-core15:13
hrwmgoddard: fully agree15:13
yoctozeptomgoddard: +115:14
mgoddardgreat15:14
mgoddardlots of announcements today :)15:14
hrwyoctozepto: you got all kolla at once as it was normal way but not everyone went that way15:14
mgoddardif anyone has more questions about the new setup, please raise in open discussion15:15
* yoctozepto feeling special ;D15:15
mgoddard#topic Review action items from last meeting15:15
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:15
mgoddardno action items15:16
*** cah_link has joined #openstack-kolla15:16
mgoddard#topic Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard15:16
*** openstack changes topic to "Kolla whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard (Meeting topic: kolla)"15:16
mgoddardI merged the kayobe whiteboard into ours15:16
mgoddardI think the main CI issues recently are around Ceph, and we have fixes in place or in review15:17
mgoddardright yoctozepto?15:17
yoctozeptoright mgoddard15:17
yoctozeptorocky waiting for reviews15:17
yoctozeptonever again15:17
mgoddardI guess we should add kolla-cli CI to the list15:18
mnasiadkayoctozepto: don't worry, you fixed a deprecated feature :)15:18
hrwcan kayobe guys clean line 135+ in whiteboard?15:18
hrwnow as you have review priority15:18
mgoddardhrw: sure15:20
mgoddarddone15:20
hrwthx15:20
mgoddardneed to do another round of stable backporting15:21
mgoddardlast checked about 6 weeks ago15:21
mgoddard#action mgoddard or someone else to check stable backports15:21
mgoddard#topic Kayobe Stein release status15:22
*** openstack changes topic to "Kayobe Stein release status (Meeting topic: kolla)"15:22
mgoddardNow that kolla stein has been released we have no excuses in kayobe, and should also release15:22
hrw;P15:23
*** dr_feelgood has quit IRC15:23
mgoddarddougsz, jovial[m], priteau: I marked a few patches as RP+1, I'd like to try getting these into master before we branch15:23
mgoddardhttps://tiny.cc/kayobe-review-dash15:23
dougszwill take a look, thanks15:24
jovial[m]I'll take a look as soon as possible15:24
mgoddardthanks15:24
mgoddard#topic Train release planning15:25
*** openstack changes topic to "Train release planning (Meeting topic: kolla)"15:25
mgoddardyoctozepto and I were discussing earlier that Train is fast approaching15:25
mgoddardfeature freeze is september 13th, we normally freeze a few weeks after - so end of september15:26
mgoddard#link https://releases.openstack.org/train/schedule.html15:26
mgoddardit seems like a good time to reflect on our priority features15:27
mgoddardL133 in https://etherpad.openstack.org/p/KollaWhiteBoard15:27
mgoddardAnyone know the status of CentOS 8?15:28
mgoddardhttps://wiki.centos.org/About/Building_815:28
mgoddardlooks like they are quite far through, but I don't know how long they normally tak15:28
mgoddardwe are dependent on it for python3 on centos15:29
mgoddardI expect we'll have a last minute push15:29
*** factor has joined #openstack-kolla15:30
mnasiadkamost likely, and then RDO will send another pack of nails for our coffin :)15:30
mgoddard;)15:30
jovial[m]also interesting was RHEL 7.7 with inplace kernel upgrades15:31
mgoddardnot much we can do there, unless we try using fedora15:31
mgoddardjovial[m]: i.e. no reboot?15:31
mnasiadkamgoddard: there is some f28 change by someone from RedHat, but I don't know the status15:31
jovial[m]mgoddard: yeah - I believe that to the case15:31
mgoddardmnasiadka: yeah, for kolla. Would be nice to have a kolla-ansible run with py315:32
mgoddardOk, I guess we just wait on this one15:32
mnasiadkamgoddard: ah, that one - shouldn't be hard to do a preliminary test15:32
mnasiadkamgoddard: but who knows what won't work on CentOS 8 :)15:33
mgoddardDefine support matrix has not been started AFAIK15:33
mgoddardanyone planning on picking it up?15:33
yoctozeptosupport matrix is important but I'm too young core to handle this15:33
yoctozepto;p15:34
mgoddardI vaguely remember someone saying they were interested, but they didn't get added as an owner15:34
yoctozeptonot me15:34
mnasiadkayoctozepto: since you're the youngest core - you can create a framework, and then we can populate :)15:34
goldyfruit_mgoddard, kplant no ?15:34
mgoddardI think a lot of it will need collaboration, but as mnasiadka says someone needs to drive15:34
mgoddardpart of it is agreeing on vocabulary15:35
mgoddardgoldyfruit_: I added kplant as an owner, we can remove if he disagrees :)15:35
mgoddardNova cells is underway by dougsz15:36
kplantit wasn't me :-)15:36
mgoddardok, removing kplant :)15:37
goldyfruit_hum, my bad then´15:37
mgoddardSupport newer ansible versions: mnasiadka working on it15:37
mnasiadkayup15:38
mgoddardanything to say on it mnasiadka?15:38
mnasiadkawell, there are two changes - one for bumping up Ansible in kolla-toolbox15:38
mnasiadkabut in order to properly constrain pip packages versions in subsequent release branches - we need to use upper-constraints.txt15:39
yoctozeptoquick reject that was15:39
mnasiadkawhich is not available in binary15:39
mnasiadkaso I need to rework some of the kolla/common/config.py stuff - should be ready this week15:39
mnasiadkathen another piece of work is migrating from kolla_* Ansible modules to upstream os_* - where it makes sense15:40
mnasiadkabut it's like an extra thing to do15:40
mgoddardis this to include get upper-constraints for binary?15:40
mgoddardhow would you do it? curl the official URL?15:41
mgoddarde.g. https://releases.openstack.org/constraints/upper/master15:41
mnasiadkamgoddard: yeah, that was the idea - but then we need to have a variable that points to current openstack release :-)15:41
mgoddardright15:41
mgoddardok, thanks15:41
goldyfruit_(mgoddard, it was ohwhyosa)15:41
mgoddardaha, nice one goldyfruit_15:42
mgoddardDebian/Ubuntu Python 3 support is mostly done, but there is a bug in kolla-ansible that shows we need some changes there too15:43
mgoddardhrw: https://review.opendev.org/#/c/674241/15:43
mgoddardpython path stuff15:43
hrwwill look15:44
mgoddardthanks15:44
mgoddardTLS everywhere has a number of patches in flight, kklimonda working on it15:44
mgoddardwhat's the status kklimonda?15:44
yoctozeptohe was not around15:44
mgoddardpatches need updating IIRC15:45
mgoddardlibvirt TLS needs another review15:46
mgoddardand merge conflicts fixing15:46
mgoddard[kolla] Health checks15:47
mgoddardmnasiadka: started looking at it?15:47
mnasiadkamgoddard: yes, had some preliminary code somewhere, will look into that after Ansible bump up15:47
mgoddardk15:48
mgoddard[kolla ansible] Improving test coverage15:48
mgoddardlots of good stuff done here already15:48
mgoddarda few more in progress15:48
mgoddarddougsz: would be nice to get an intial monasca test merged that we can iterate on15:49
dougszah yes, sorry, I need to debug why it is failing15:49
mnasiadkamgoddard: I was thinking the same about prometheus and all those exporters that we have...15:49
mnasiadkaI mean if somebody would like to do it :)15:49
mgoddardyeah, would be nice to have that covered15:50
*** icarusfactor has joined #openstack-kolla15:50
mgoddardSimplifying handler conditionals - done15:50
mgoddard[kolla] Podman/buildah support - no owner yet15:50
mgoddard[kolla ansible] IPv6 - yoctozepto have you started it?15:50
hrwI am afraid that podman thing will slip into Ucycle15:51
yoctozeptomgoddard: yeah, sketching15:51
mgoddardhrw: I expect so. It's low on priorities so that's ok15:51
hrwBaptisteGer is working on ipv6 deployment in Linaro. no idea how much time during week he has for it15:51
mgoddardyoctozepto: cool15:51
mgoddardhrw: using kolla ansible?15:51
hrwmgoddard: yes15:51
mgoddardcool yoctozepto & BaptisteGer should talk :)15:52
yoctozeptoyeah, he sent me his diff15:52
mgoddardnice15:52
*** factor has quit IRC15:52
mgoddardhrw: can I add BaptisteGer as another owner of that feature or is it just downstream?15:52
yoctozeptoobviously it drops ipv4 support the way he did it :-)15:52
mgoddardlol15:52
yoctozeptoipv6 only man15:53
mgoddardgot to start somewhere15:53
yoctozeptodone right15:53
hrwmgoddard: add as interested rather than owner15:53
yoctozeptoI will bother him to test my proposal when ready15:53
mgoddardhrw: done15:53
hrwyoctozepto: he may base on some old patch which did that way15:53
hrwand it is easier to get ipv6 machines than ipv4 ones15:53
yoctozeptothe diff is good for scoping, no problem15:54
mgoddard[kolla ansible] Upgrade checkers - no progress since PoC15:54
mgoddard[kolla] Define source image config via JSON - I think mnasiadka picked this up15:55
*** icarusfactor has quit IRC15:55
*** whoami-rajat has joined #openstack-kolla15:55
mgoddardmnasiadka: if it has a bug/blueprint, please link it on the whiteboard15:55
mgoddard[kolla ansible] SELinux - needs an owner15:55
mnasiadkamgoddard: yes, as I wandered into the abyss called common/config.py - but I will most probably first fix the Ansible bump up to make it fast, and then continue with the JSON stuff later on - if that's ok15:56
mgoddardsure15:56
mgoddardok let's move on in the last few minutes15:56
yoctozeptoI have working selinux experience but not willing to be blamed for failing the delivery :-)15:56
mnasiadkaif SELinux doesn't have an owner now - it will be hard to get anywhere before end of Sep15:56
mgoddardhopefully that was useful though15:56
yoctozeptoadd as potential owner or sth15:56
mgoddardwell sdake claimed it used to 'just work' :)15:56
yoctozeptocould be15:57
mnasiadkamgoddard: it did, but neutron-ovs-agent stopped working long time ago15:57
mgoddardtripleo does it with kolla images15:57
yoctozeptodocker is free of selinux checks most likely15:57
mnasiadkamgoddard: yes, but they don't use libvirt and ovs in container :)15:57
mgoddard#topic: Ocata EOL15:57
*** openstack changes topic to ": Ocata EOL (Meeting topic: kolla)"15:57
mgoddardWho wants to EOL ocata?15:57
mgoddardWe have waited the required 6 months with failing CI15:58
mnasiadkaI think the question should be who wants to keep it alive :-)15:58
mgoddardI guess so15:58
mgoddardanyone want to keep it alive?15:58
yoctozeptoI want to EOL pike15:59
yoctozeptodon't look at me15:59
mgoddardcan't EOL pike until CI has been failing for 6 months :)15:59
mnasiadkayeah, I wanted to point out that pike is EM (or should be)15:59
mgoddardofficially15:59
hrwocata is dead.15:59
mgoddardok I think we agree on EOL ocata16:00
hrwfor me <queens is dead. queens is dead. rocky is what we still have in use16:00
*** jbadiapa has joined #openstack-kolla16:00
mgoddardwe still need to keep queens alive16:00
yoctozeptoqueens is dead, long live rocky?16:00
mgoddarduntil EM16:00
mgoddard#topic Shanghai summit16:00
*** openstack changes topic to "Shanghai summit (Meeting topic: kolla)"16:00
mgoddardAnyone going?16:01
goldyfruit_I'll try16:01
* hrw skips16:01
mgoddardgoldyfruit_: would you be available to help with kolla forum sessions e.g. answering questions etc.?16:02
mgoddardspsurya has agreed to handle the project update and project onboarding as I won't be going16:02
mgoddardwith help from Jeffrey4l_ and colleagues16:02
goldyfruit_mgoddard, If I'm going yes16:02
*** henriqueof has quit IRC16:02
nafiuxI want to get more involved on kolla project in general, will start reading the https://docs.openstack.org/kolla/latest/contributor/index.html before asking questions on how everything is getting organized :)16:03
hrwnafiux: wait for open discussion part16:03
nafiuxSure!16:03
mgoddardwe're over time on the meeting, does anyone need to go or can we finish off?16:04
yoctozeptomgoddard: rafaelweingartne16:04
rafaelweingartneis it the open discussions moment?16:04
rafaelweingartneI was waiting for it16:05
mgoddardLet's discuss Ceph ansible migration next time (or after meeting in channel)16:05
yoctozeptoyeah, the meeting was a bit longer than usual16:05
mgoddard#topic open discussion16:05
*** openstack changes topic to "open discussion (Meeting topic: kolla)"16:05
mgoddardrafaelweingartne: you're up16:05
rafaelweingartneok, so I have been working on this PR (https://review.opendev.org/#/c/670626/) ``Standardize the configuration of "oslo_messaging" section``16:06
rafaelweingartneand we have had some debate regarding the use of those "oslo_messaging" sections16:06
rafaelweingartneour team here, when we first used Kolla-ansible, we expected it to deploy a minimum viable openstack setup, which in our perspective should include event messaging16:07
*** igordc has joined #openstack-kolla16:07
mgoddardI guess 'minimum viable' is open to interpretation :)16:08
rafaelweingartnejust when we were debugging an bug that is caused by Ceilometer and barbican keystone listener, when they are using the same topic that we found out what was happening16:08
rafaelweingartnemgoddard: yes16:08
hrwrafaelweingartne: those notifications goes to ceilometer, right?16:08
rafaelweingartneyes and no16:08
rafaelweingartnethere are some misconfigurations16:08
hrwI mean 'notifications require ceilometer'16:09
rafaelweingartnefor instance, barbican cannot connect to the same topic. Otherwise, it will get events that were not destinated to it16:09
yoctozeptohrw: no16:09
yoctozeptohrw: ceilometer is just the main consumer16:09
hrwok16:09
rafaelweingartnehrw: yes, that is our interpretation, but there is nothing hardcoded, all of it can be customized, and some projects do that16:09
hrwrafaelweingartne: ok16:10
openstackgerritgugug proposed openstack/kolla-cli master: Move kolla prechecks action into kolla_actiom model  https://review.opendev.org/67514016:10
rafaelweingartneso, after discussing with Radek, and later with Mark, we proposed to apply the same method that was being used in Nova role in all of the other projects that had the oslo messaging section declared, but not configured16:10
rafaelweingartnejust to give you guys a quick highlight of that idea, in Nova we can decide if oslo_messaging is going to be configured if there are at least of topic enabled16:11
rafaelweingartneand then, the topics are enabled/disabled based on components that are enabled/disabled (e.g. ceilometer)16:11
rafaelweingartneWhat I think we need to discuss here is: "do we have an idea what are the projects that need notifications so the deployment can be considered an MVP?"16:12
rafaelweingartneby asking that, I am assuming that the method/standard applied in Nova role is widely accepted by the community, and therefore, we just need to discuss where to apply it16:13
yoctozeptowell, it seems ceilometer can be configured to listen for any project's notifications16:13
rafaelweingartneyes16:13
yoctozeptoso we can coordinate that as well16:13
*** skramaja has quit IRC16:14
yoctozeptoand then configure any service that:16:14
yoctozepto1) exists?16:14
yoctozepto2) we know produces notifications?16:14
yoctozepto^ choose one16:14
mgoddardI would say 216:14
rafaelweingartnewell, I was not aware of all of the projects that produced notifications16:14
rafaelweingartnetherefore, I just went to the roles, and looked for the ones that had the "oslo_messaging" section declared16:15
rafaelweingartneand worked to configure them using the approach I described16:15
rafaelweingartnebut sure, we can extend this to all service that push some notification16:15
mgoddardI would more say "restrict" this to services that push notifications16:16
*** chason has quit IRC16:16
yoctozepto+1 to mgoddard way16:16
rafaelweingartneall of them then?16:17
yoctozeptoreview the modified services16:17
yoctozeptoit should be easier16:17
yoctozeptoand drop the notifications for those that do not produce them for sure16:17
mgoddardfor example, I don't believe mistral produces notifications, so we should not add variables for it16:17
rafaelweingartnebecause right now, we seem to be enabling notifications in just a few (more than a few actually) of them16:17
mgoddard(I am open to being proven wrong)16:17
*** lemko has quit IRC16:17
rafaelweingartneand regarding the drop of the oslo messaging sections16:18
yoctozeptorafaelweingartne: the thing is none of current cores was responsible for notifications integration in k-a16:18
rafaelweingartnehmm16:18
rafaelweingartnegot it16:18
yoctozeptoso we cannot say why it was done this way16:18
yoctozeptomight be copy-paste syndrome16:18
rafaelweingartneyes, this caused a bit of confusion for us16:18
*** nafiux has quit IRC16:19
rafaelweingartneit actually looked quite a lot of copy/paste indeed16:19
rafaelweingartneand that is why we decided to fix it by apply the same standard that we have in some well-known project roles such as the nova one16:19
rm_workdropping in randomly, but as an example, Octavia should have an oslo_messaging configuration section, but we don't produce notifications (we just READ from a queue -- which I think is the same for a lot of projects?)16:20
mgoddardI think we agree that is a good approach, we just need to review where we apply it16:20
rm_workapologies if I missed some context16:20
rafaelweingartneyes, but this is a problem16:20
rafaelweingartnefor instance for the projects that had the section declared, but did not use it16:21
mgoddardrm_work: that is very good context, and one reason why I was concerned about removing these sections16:21
rafaelweingartneshould I do the removal in the same PR?16:21
rafaelweingartneor should I wait and do that in a different one?16:21
mgoddardrafaelweingartne: based on rm_work's comment, please do not remove any oslo_messaging_notification sections16:21
rafaelweingartneI would rather do in a second PR just with the removals16:21
yoctozeptoI thought the section configured producers, not consumers16:21
rm_workahh hold on16:21
rm_workI just read this:16:22
rm_work[09:15:04] rafaelweingartne:therefore, I just went to the roles, and looked for the ones that had the "oslo_messaging" section declared16:22
rafaelweingartneme too16:22
rm_workif it's [oslo_messaging_notifications], then that's maybe fine, we don't declare that16:22
mgoddardrm_work: ok, thanks16:22
rm_workbut i am not sure if it's relevant for readers-only16:22
yoctozeptothanks rm_work16:22
yoctozeptolemme check then, but I'm 99% sure it is for producers only16:22
mgoddardI'm still a little wary of removing them16:22
*** jbadiapa has quit IRC16:23
mgoddardif we can verify it's only used for producers I would be less wary16:23
rafaelweingartneyoctozepto: I checked, and it was for producers only16:23
*** ivve has quit IRC16:23
rafaelweingartnebut I am quite new to openstack, so I might be mistaken16:23
mgoddardeither way, separate PR please16:23
yoctozeptohttps://docs.openstack.org/oslo.messaging/stein/configuration/opts.html#oslo-messaging-notifications16:23
yoctozepto*sending*16:24
mgoddardok we're approaching 30 minutes over time in the meeting. The discussion can continue in here if people are free16:24
mgoddardThanks everyone16:24
rm_workcool, just got concerned due to the abbreviation :D16:24
mgoddard#endmeeting16:24
*** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:24
openstackMeeting ended Wed Aug  7 16:24:18 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:24
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-08-07-15.01.html16:24
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-08-07-15.01.txt16:24
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-08-07-15.01.log.html16:24
rafaelweingartneok, then, for the first PR, I will apply the same method for all projects that have the "oslo messaging" sections16:24
yoctozeptomgoddard, rafaelweingartne, rm_work: let's continue though16:24
rafaelweingartnewhich is what I have already done16:24
rafaelweingartnethen, after the merge, I will investigate which one of those projects are not using the sections, and then I will create a PR to remove it16:25
mgoddardrafaelweingartne: yoctozepto and I both said earlier we want to only do it when we know that services produce notifications16:25
rafaelweingartneok16:25
yoctozepto<mgoddard> either way, separate PR please16:25
yoctozeptoso what in the first16:25
yoctozeptowhat in the second16:25
mgoddardI think we are just confusing the approach - I would rather not add and then remove variables16:25
yoctozeptoconfused16:26
yoctozeptome neither16:26
rafaelweingartneI am confused as well16:26
mgoddardfirst PR - add new variables for services that produce notifications16:26
mgoddardsecond PR - remove [oslo_messaging_notifications] sections for services that do not produce notifications16:26
mgoddardmake sense?16:26
rafaelweingartneok, then I will need to go through the list, and check each one of them16:26
yoctozeptorafaelweingartne: unfortunately yeah16:27
mgoddardso there may be some files that need to be removed from the first PR16:27
yoctozeptobut glory awaits :D16:27
rafaelweingartnethen  for the ones that do not produce, I will not do anything with respect to those configs16:27
rafaelweingartnethen I only remove them in a second PR16:27
mgoddardright16:27
rafaelweingartnenow I got it16:27
yoctozeptomgoddard: what about some note in contributor guide16:27
mgoddardwhat sort of note?16:27
yoctozepto"don't add notifications section if you don't support notifications"16:28
rafaelweingartne+116:28
yoctozeptothat sort16:28
yoctozeptoput in better word16:28
mgoddardlol16:28
mgoddardpossibly16:28
mgoddardcould add in the role writing guide16:28
mgoddardpeople will still copy & paste :)16:28
mgoddardanyway, I think we have agreement16:29
rafaelweingartnewell, at least now we have this discussion logged16:29
mgoddardrafaelweingartne: are you in a rush to get the keystone, barbican & nova fixes merged?16:29
rafaelweingartnetherefore, we can now review new roles, and ask for people not to use the notification section, when it is not needed16:29
rafaelweingartnemgoddard: internally we already did16:30
rafaelweingartnebut sure, I would love to close this before moving on with new things16:30
rafaelweingartneI already have the list of services, therefore, I will start to review and check which one of them are not using that config16:30
rafaelweingartneit is not going to be that hard16:31
mgoddardIt just feels like those 3 bug fixes are separate and could be in a different commit which we could approve today/tomorrow16:31
rafaelweingartnehmm16:31
mgoddardbut don't worry if it's too much work to extract16:31
yoctozeptomgoddard: today approve ceph if you want to approve something :-)16:31
rafaelweingartneI think we can move on with that single fix16:31
rafaelweingartneeven if it takes a bit more work from my side16:33
rafaelweingartneit is more or less done anyways16:33
mgoddardok16:33
mgoddardin future though please use separate commits for separate bug fixes16:33
rafaelweingartnesure16:33
rafaelweingartnewill do that16:33
rafaelweingartnethanks for the time16:34
mgoddardit makes reviewing easier, and we can manage backports to stable branches16:34
rafaelweingartneyes, I do agree16:34
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible stable/rocky: Add ceph-mds/rgw/nfs to gate  https://review.opendev.org/67447916:34
mgoddardeasy to miss details in big patches16:34
mgoddardok, I think we have agreement. Thanks for fixing this16:34
rafaelweingartneyes, I think we do16:35
rafaelweingartnewelcome16:35
*** factor has joined #openstack-kolla16:35
yoctozeptorafaelweingartne: thanks from me too16:35
yoctozeptohappy to have contributions16:36
*** nafiux has joined #openstack-kolla16:37
yoctozeptoguys, 4 ceph-related proposals pending from me: 2 rocky, 2 queens16:37
yoctozeptorocky:16:37
yoctozeptohttps://review.opendev.org/67447916:38
yoctozeptohttps://review.opendev.org/67504816:38
yoctozeptoqueens:16:38
yoctozeptohttps://review.opendev.org/67504516:38
yoctozeptohttps://review.opendev.org/67504316:38
mnasiadkaLet’s see what Zuul has to say :)16:38
*** gfidente has quit IRC16:38
yoctozeptoZuul is working only on 1 of 4 of them16:38
hrwyoctozepto: queens fail16:38
yoctozeptohrw: thanks, missed that16:39
hrwhttps://review.opendev.org/#/c/672700/ (my docker change) goes into green.16:39
yoctozeptosummary tables are not that clear16:39
hrwwith releasenotes taken care of and doc fixes16:39
yoctozeptoseems queens source fails all around16:39
* hrw off16:42
goldyfruit_o/16:43
yoctozeptoall right, rechecks issued16:43
mgoddardheading off, will look at reviews tomorrow16:44
*** nafiux has quit IRC16:44
atmarkhello, I'm getting tons of 'Either [None] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/' in the logs. I have a stable/stein deployment16:46
*** igordc has quit IRC16:51
openstackgerritMerged x/kayobe stable/queens: Generate bifrost hostvars prior to provisioning  https://review.opendev.org/67450616:55
*** rpittau is now known as rpittau|afk16:56
*** dougsz has quit IRC16:57
*** jonaspaulo has joined #openstack-kolla17:00
*** rafaelweingartne has quit IRC17:01
*** igordc has joined #openstack-kolla17:10
*** altlogbot_0 has quit IRC17:17
*** altlogbot_1 has joined #openstack-kolla17:23
*** nafiux has joined #openstack-kolla17:25
*** altlogbot_1 has quit IRC17:31
*** henriqueof has joined #openstack-kolla17:32
*** altlogbot_0 has joined #openstack-kolla17:35
dcapone2004atmark:  do you have a fresh kolla ansible install and the errors are appearing horizon?17:37
openstackgerritMerged x/kayobe master: Use new kolla-ansible upgrade-bifrost command  https://review.opendev.org/66763217:48
*** dpawlik has joined #openstack-kolla17:50
openstackgerritMerged x/kayobe master: Revert "Workaround: stop bifrost services before seed upgrade"  https://review.opendev.org/66888817:52
*** nafiux has quit IRC17:54
*** nafiux has joined #openstack-kolla17:57
*** spsurya has quit IRC18:22
*** Wasaac has quit IRC18:29
*** Wasaac has joined #openstack-kolla18:30
*** k_mouza has joined #openstack-kolla18:37
*** BjoernT has joined #openstack-kolla18:48
*** BjoernT_ has quit IRC18:50
*** Wasaac has quit IRC18:52
*** Wasaac has joined #openstack-kolla19:00
*** Wasaac has quit IRC19:01
*** k_mouza has quit IRC19:02
*** nafiux has quit IRC19:03
openstackgerritMerged openstack/kolla master: Change monasca-grafana source branch to use grafana v4.2.0 with keystone changes  https://review.opendev.org/67271719:12
*** BjoernT_ has joined #openstack-kolla19:15
*** BjoernT has quit IRC19:17
*** rouk has joined #openstack-kolla19:22
*** dcapone2004 has quit IRC19:23
roukhey all, looking for the solution to credential-key setup/sync/rotation in kolla, but it seems like theres only stuff for fernet, but my logs are very unhappy with using null key for credential encryption.19:29
*** Wasaac has joined #openstack-kolla19:29
roukwhats the recommended way to get credential-keys set and synced in kolla? or do i have to bake them into images?19:30
*** dave-mccowan has joined #openstack-kolla19:33
*** vesper11 has quit IRC19:37
*** Wasaac has quit IRC19:38
*** nafiux has joined #openstack-kolla19:42
*** kplant has quit IRC19:43
*** Wasaac has joined #openstack-kolla19:44
*** dpawlik has quit IRC19:53
*** factor has quit IRC20:03
*** factor has joined #openstack-kolla20:03
*** factor has quit IRC20:05
*** factor has joined #openstack-kolla20:05
*** dr_feelgood has joined #openstack-kolla20:10
*** ivve has joined #openstack-kolla20:15
*** whoami-rajat has quit IRC20:17
*** nafiux has quit IRC20:17
openstackgerritRafael Weingärtner proposed openstack/kolla-ansible master: Standardize the configuration of "oslo_messaging" section  https://review.opendev.org/67062620:21
*** factor has quit IRC20:32
*** factor has joined #openstack-kolla20:32
*** factor has quit IRC20:35
*** factor has joined #openstack-kolla20:36
*** aissat-mus has joined #openstack-kolla21:00
*** dr_feelgood has quit IRC21:00
*** BjoernT_ has quit IRC21:08
*** aissat-mus has quit IRC21:10
*** Wasaac has quit IRC21:10
*** Wasaac has joined #openstack-kolla21:15
*** vesper11 has joined #openstack-kolla21:25
*** henriqueof has quit IRC21:33
*** factor has quit IRC21:35
*** factor has joined #openstack-kolla21:35
*** cah_link has quit IRC21:39
*** Wasaac has quit IRC21:43
*** Wasaac has joined #openstack-kolla21:43
*** mchlumsky has quit IRC21:44
*** nafiux has joined #openstack-kolla21:54
*** jonaspaulo has quit IRC21:58
*** nafiux has quit IRC22:38
*** ivve has quit IRC22:45
*** nafiux has joined #openstack-kolla22:46
*** Wasaac has quit IRC23:19
*** igordc has quit IRC23:35

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!