Wednesday, 2019-10-16

*** calbers has joined #openstack-kolla00:04
*** goldyfruit_ has joined #openstack-kolla00:05
*** schwicht has quit IRC00:25
*** calbers has quit IRC00:45
*** vmixor has quit IRC00:46
*** calbers has joined #openstack-kolla00:47
*** schwicht has joined #openstack-kolla00:50
*** k_mouza has joined #openstack-kolla01:29
*** k_mouza has quit IRC01:33
*** igordc has quit IRC01:47
*** spsurya has joined #openstack-kolla01:59
*** ileixe has quit IRC02:03
openstackgerritZijian Guo proposed openstack/kolla-ansible master: Add haproxy stats listening on external network  https://review.opendev.org/68884402:23
*** k_mouza has joined #openstack-kolla02:30
*** k_mouza has quit IRC02:34
*** dave-mccowan has quit IRC02:56
*** factor has joined #openstack-kolla03:04
*** rohit02 has joined #openstack-kolla04:02
rohit02hello team04:03
rohit02does kolla stein support the multidomain functionality04:03
*** jdwidari has quit IRC04:10
*** skramaja has joined #openstack-kolla04:36
*** pcaruana has joined #openstack-kolla04:37
*** rohit02 has quit IRC04:40
*** rohit02 has joined #openstack-kolla04:41
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: handle push error properly  https://review.opendev.org/68839005:02
*** pcaruana has quit IRC05:14
*** jcmdln has quit IRC05:14
*** Luzi has joined #openstack-kolla05:18
openstackgerritjacky06 proposed openstack/kolla-cli master: Typo fixed: Update Upgrade to Check  https://review.opendev.org/68885005:43
openstackgerritya.wang proposed openstack/kolla-ansible master: Add 'physical_networks' into horizon config template  https://review.opendev.org/68885206:03
openstackgerritMatthias Runge proposed openstack/kolla stable/queens: Service assurance plugins and write_prometheus  https://review.opendev.org/68885506:10
*** pcaruana has joined #openstack-kolla06:13
*** hjensas has quit IRC06:15
yoctozeptorohit02: keystone multidomain? for sure06:26
yoctozeptomorning channel06:30
yoctozeptohrw: docker registry down again :/06:30
hrwgeiptyh23894t397!@#06:30
openstackgerritMatthias Runge proposed openstack/kolla stable/queens: Service assurance plugins and write_prometheus  https://review.opendev.org/68885506:36
*** dpawlik has joined #openstack-kolla06:39
yoctozeptoand it seems to be working again06:41
mnasiadkamorning06:53
*** dpawlik has quit IRC06:53
*** dpawlik has joined #openstack-kolla06:54
yoctozeptoubuntu binary does not build trove and rally06:57
mnasiadkayeah, trove - I have a bug raised in UCA07:00
mnasiadkathey are fixing irt07:00
yoctozeptohttps://bugs.launchpad.net/kolla/+bug/184829507:00
openstackLaunchpad bug 1848295 in kolla train "trove and rally unbuildable for ubuntu binary" [Critical,Triaged]07:00
yoctozeptomnasiadka: comment on it07:00
mnasiadkadone07:01
mnasiadkaI haven't investigated rally07:01
mnasiadkaat least RDO is on a good path07:02
mnasiadkaas soon as tripleo releases RC07:02
yoctozeptomnasiadka: looks like ubuntu patched only eoan07:02
yoctozeptowhat about bionic ;p07:02
mnasiadkaask them in the ticket :D07:02
mnasiadkaI raised that ticket yesterday, so that was ekhm... extremely fast07:02
yoctozeptoyeah indeed07:03
yoctozeptomnasiadka: correct me if i'm wrong but rally just looks as it is gone?07:04
mnasiadkayoctozepto: yeah, it's missing from Packages.gz in train... just raise a bug07:07
mnasiadkaor go ask in #ubuntu-server :)07:08
yoctozeptomnasiadka: raised it07:08
*** dpawlik has quit IRC07:12
hrwyoctozepto: those tests follow version without _error in name07:15
yoctozeptohrw: ah07:17
*** ivve has joined #openstack-kolla07:19
mnasiadkathat's fantastic, we have a neutron-server-ovn, that differs from neutron-server by one package? delightful...07:22
hrwuf. first uwsgi/django app deployed on server07:23
yoctozeptomnasiadka: that's cheap to push though07:23
mnasiadkayoctozepto: maybe, but that's idiocy :)07:23
yoctozeptomnasiadka: well, probably07:23
hrwmnasiadka: #action mnasiadka merge neutron-server-ovn to neutron-server in Ussuri cycle?07:23
yoctozeptothough the package might be changing some behavior07:23
mnasiadkahrw: looks like I'll be adding ovn to k-a if things go gladly during a PoC for a customer - swill do that07:24
mnasiadkayoctozepto: it's a python library...07:24
yoctozeptomnasiadka: then unlikely07:24
hrwyoctozepto: replied07:29
*** priteau has quit IRC07:29
*** priteau has joined #openstack-kolla07:30
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Increase job run attempts to 5  https://review.opendev.org/68886607:33
*** hjensas has joined #openstack-kolla07:34
yoctozeptohrw: replied too07:35
yoctozeptoI guess we can push this07:35
yoctozeptoand then look at making the code nicer07:35
hrwyoctozepto: so +2 it07:35
yoctozeptohrw: yes, sir07:36
hrwmnasiadka: https://review.opendev.org/#/c/68839007:41
*** cah_link has joined #openstack-kolla07:42
mnasiadkafine by me07:43
mnasiadkanow we could add support for just pushing the failed ones :)07:43
hrwmnasiadka: by 'failed' you mean 'not present in registry'?07:44
mnasiadkahrw: for example, or the ones that are push_failed07:44
hrwmnasiadka: push_error ones are "build session specific"07:45
hrwmnasiadka: next run you do not know which they are07:45
hrwmnasiadka: and in normal build you can do "--retries X" to have X attempts to push07:45
mnasiadkahrw: well, if the user can supply the names of those - then it's easy to push them once again07:45
mnasiadkahrw: if you get out of storage space on the registry, then retry won't help :D07:46
*** devep has joined #openstack-kolla07:46
hrwmnasiadka: kolla-build --push --registry a.b.c.d:xy "image1 image2 image4"07:46
mnasiadkatrue07:46
hrwmnasiadka: heh. I got out of space recently ;d on registry box ;D07:46
osmanlicilegimorning07:49
yoctozeptohrw: though that will rebuild by default07:51
yoctozeptoif they are built then best is to just use docker then07:51
hrwthat's too07:51
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: WIP: CI: rework check-logs  https://review.opendev.org/68887007:54
*** rgogunskiy has joined #openstack-kolla07:55
*** gfidente has joined #openstack-kolla08:00
* hrw out08:05
openstackgerritMark Goddard proposed openstack/kolla stable/stein: CI: pep8: Fix yamllint error on .yamllint  https://review.opendev.org/68887108:05
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: CI: pep8: Fix yamllint error on .yamllint  https://review.opendev.org/68887208:06
*** rpittau|afk is now known as rpittau08:12
*** dougsz has joined #openstack-kolla08:14
*** k_mouza has joined #openstack-kolla08:16
*** dpawlik has joined #openstack-kolla08:17
*** k_mouza has quit IRC08:19
*** k_mouza has joined #openstack-kolla08:19
mgoddardmorning08:20
*** dpawlik has quit IRC08:21
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157308:44
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Use versioned python binary with fetch ceph keyrings  https://review.opendev.org/68839208:50
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Increase timeout for upgrade jobs by 30 minutes  https://review.opendev.org/68887908:57
mgoddardyoctozepto: back to ipv609:01
yoctozeptomnasiadka, hrw: https://review.opendev.org/68887909:04
yoctozeptomgoddard: +309:04
mgoddardyoctozepto: considered trying a DNM run all the tests with IPv6 patch?09:22
mgoddard(like we did for tls09:22
yoctozeptomgoddard: already done before asking for reviews09:25
yoctozeptobut yeah, easy to do :-)09:25
mgoddardyoctozepto: and they pass?09:25
yoctozeptomgoddard: yeah09:27
mgoddardb00m09:27
yoctozeptonow ceph will fail without mnasiadka patch merged09:27
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: CI: Run all IPv6  https://review.opendev.org/68888709:27
yoctozeptobut whatever09:27
yoctozeptoalso, centos will obviously fail09:27
yoctozeptobut all ubuntu debian do pass09:27
mnasiadkawhat is this? fail day? :)09:28
yoctozeptomnasiadka: fail festival09:29
yoctozeptomerge my CI changes09:29
yoctozeptoand we are stabler :-)09:29
mnasiadkayeah, let's merge everything and worry later :P09:30
yoctozeptomnasiadka: clearly you haven't looked at them09:32
yoctozepto#%^@#%^09:34
yoctozeptoadding bindep increased centos chance of failure09:34
yoctozeptoit has troubles with ipv609:34
yoctozeptoand now it fails very early09:34
yoctozeptodue to bindep09:35
yoctozepto(and I mean ipv6 in ci in general, not my ipv6)09:35
yoctozepto%$#^%^09:35
yoctozeptoanyways09:36
yoctozeptoneed to wait with rechecking kolla jobs before the timeout increase merges09:36
yoctozeptoas it just kills gate09:36
*** dpawlik has joined #openstack-kolla09:47
yoctozeptomnasiadka: https://review.opendev.org/688866 also09:48
*** dpawlik has quit IRC09:52
openstackgerritMerged openstack/kolla-ansible stable/stein: HAProxy backend connection limits  https://review.opendev.org/68864909:53
openstackgerritMerged openstack/kolla-ansible master: Fixes glance image cache deployment.  https://review.opendev.org/68865309:54
openstackgerritDincer Celik proposed openstack/kolla-ansible stable/stein: Fixes glance image cache deployment.  https://review.opendev.org/68889109:57
egonzalezHi, are we aware of this https://linux.slashdot.org/story/19/10/14/2231235/flaw-in-sudo-enables-non-privileged-users-to-run-commands-as-root ?10:01
egonzalezhttps://github.com/openstack/kolla/blob/master/docker/neutron/neutron-base/neutron_sudoers10:01
yoctozeptoegonzalez: but it makes it possible to run as root and we specifically allow root10:03
yoctozeptoso it's irrelevant10:03
mrungeI would expect distributions to patch sudo anyways10:04
mrunge... that should not be a kolla issue10:04
*** dpawlik has joined #openstack-kolla10:06
openstackgerritMerged openstack/kolla master: Add disk dev name check function  https://review.opendev.org/68832810:11
*** hamdyk has joined #openstack-kolla10:20
mnasiadkaoh crap, I made double base64 with the patch, need to look into it10:25
openstackgerritMerged openstack/kolla master: handle push error properly  https://review.opendev.org/68839010:27
openstackgerritMerged openstack/kolla stable/queens: Service assurance plugins and write_prometheus  https://review.opendev.org/68885510:27
*** dpawlik has quit IRC10:28
hamdykHi everyone10:32
hamdykWe are working on adding infiniband feature for tripleo10:32
hamdykin the process we need to install the package: python-networking-mlnx on the following containers: neutron-server, neutron-dhcp and neutron-l3-agent10:33
hamdykI'm thinking that we could add the package to container image "neutron-base"10:33
hamdykhere: https://github.com/openstack/kolla/blob/master/docker/neutron/neutron-base/Dockerfile.j2#L3810:33
hamdykis that OK ?10:33
yoctozeptomgoddard: quick response10:35
openstackgerritRadosław Piliszek proposed openstack/kolla stable/stein: handle push error properly  https://review.opendev.org/68889710:38
openstackgerritRadosław Piliszek proposed openstack/kolla stable/rocky: handle push error properly  https://review.opendev.org/68889810:38
*** skramaja has quit IRC10:46
osmanlicilegihamdyk: do you use rdma?10:56
*** dougsz has quit IRC11:01
*** dpawlik has joined #openstack-kolla11:01
yoctozeptoosmanlicilegi: I believe mellanox ppl use rdma :-)11:18
openstackgerritDincer Celik proposed openstack/kolla-ansible master: Fixes missing boolean for Neutron FWaaS  https://review.opendev.org/68890611:24
*** kplant has joined #openstack-kolla11:24
*** dpawlik has quit IRC11:28
osmanlicilegiyoctozepto: well I'm not familiar with mellanox and infiniband stuff. here at my new job, there are plans to switch to mellanox so I was planning to work on neutron-mlx too.11:29
*** dpawlik has joined #openstack-kolla11:31
yoctozeptoosmanlicilegi: I have plenty of mellanox but we use them mostly to run storage11:31
yoctozeptosometimes some hpc mpi11:31
osmanlicilegiyoctozepto: have you experienced ceph over rdma?11:32
yoctozeptoosmanlicilegi: it's b0rken on my connectx-311:32
yoctozeptoI tried everything11:33
yoctozeptoI have a bug report open somewhere for that11:33
yoctozeptothe point is the speed of storage drives is much below the limits of ipoib so we are fine either way11:33
kplant^11:34
kplantmaybe something like nvme/nvdimm + cpu + 100GbE on te same numa node can change that11:35
kplantbut that's a little expensive11:35
osmanlicilegikplant: if you wanna talk about nvme, 1st you should talk about money :]11:36
osmanlicilegiso I'm happy with all-flash farms11:36
kplantwe use nvme, but not the config i listed above11:37
kplanti'll probably be testing that soon though11:37
kplantmaybe nvdimm for WAL11:37
kplantset some money on fire11:37
yoctozeptoset some datacenter on fire...11:39
yoctozeptooh wait11:39
kplantburning money to create the initial dc and money to replace everything... burning (money)^211:40
kplanti respect it11:40
yoctozeptoxD11:40
kplantactually it's 2*money but "money squared" sounds cooler11:40
osmanlicilegiabba is right, money is funny11:41
kplantit must be :-)11:42
*** schwicht has quit IRC11:49
hamdykosmanlicilegi: for infiniband no11:56
yoctozeptohamdyk: what does that do then?11:58
yoctozeptois it mellanox ethernet switch configuration then?11:58
hamdykwe are supporting infiniband for tenant physical network12:01
hamdykso users can create instances with direct port over IB fabric12:01
yoctozeptoI see12:02
*** ivve has quit IRC12:03
*** schwicht has joined #openstack-kolla12:06
openstackgerritMerged openstack/kolla-ansible master: CI: Increase job run attempts to 5  https://review.opendev.org/68886612:06
*** Luzi has quit IRC12:11
*** schwicht has quit IRC12:17
*** ivve has joined #openstack-kolla12:18
*** schwicht has joined #openstack-kolla12:18
openstackgerritMerged openstack/kolla master: Neutron: add support to use legacy iptables  https://review.opendev.org/68367912:22
*** priteau has quit IRC12:22
*** Luzi has joined #openstack-kolla12:27
*** dougsz has joined #openstack-kolla12:40
openstackgerritEddie Yen proposed openstack/kolla stable/rocky: Add disk dev name check function  https://review.opendev.org/68891812:42
*** schwicht has quit IRC12:42
*** hjensas has quit IRC12:47
*** Luzi has quit IRC13:02
openstackgerritMerged openstack/kolla-ansible master: CI: Increase timeout for upgrade jobs by 30 minutes  https://review.opendev.org/68887913:11
*** dpawlik has quit IRC13:15
*** hjensas has joined #openstack-kolla13:17
*** dougsz has quit IRC13:19
openstackgerritEddie Yen proposed openstack/kolla stable/stein: Add disk dev name check function  https://review.opendev.org/68892613:23
*** lemko has joined #openstack-kolla13:25
mgoddardyoctozepto: do I need to make any more changes to the cells patch?13:30
*** dpawlik has joined #openstack-kolla13:36
openstackgerritMark Goddard proposed openstack/kayobe master: Build pdf doc  https://review.opendev.org/68261613:36
*** dougsz has joined #openstack-kolla13:41
*** rohit02 has quit IRC13:49
*** vmixor has joined #openstack-kolla13:50
*** chason has joined #openstack-kolla14:03
*** dpawlik has quit IRC14:05
yoctozeptomgoddard: unit tests if you want now, but follow-up would be ok as well14:05
openstackgerritMerged openstack/kolla-ansible stable/stein: Fixes glance image cache deployment.  https://review.opendev.org/68889114:07
openstackgerritwangfaxin proposed openstack/kolla master: Fix source link.  https://review.opendev.org/68893814:08
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Neutron: add support to use legacy iptables  https://review.opendev.org/68596714:08
mgoddardyoctozepto: you have my +214:11
yoctozeptomgoddard: you have mine as well14:14
yoctozeptobut one of us has to rebase anyway14:15
yoctozeptoI can rebase yours too if we decide to merge ipv6 first14:15
yoctozeptoI know what to look out for14:15
mgoddardit's a race14:15
openstackgerritRadosław Piliszek proposed openstack/kolla master: Fix fetch_ceph_keys to run in python3  https://review.opendev.org/68863614:16
mgoddardhrw, mnasiadka: ^14:17
yoctozeptoI think cells do less tests14:19
yoctozeptoso it's more beneficial to merge ipv6 not to stress CI with retries14:19
yoctozeptoeither way is fine when CI is ignored14:19
*** mgoddard has quit IRC14:28
*** mgoddard has joined #openstack-kolla14:30
*** jdwidari has joined #openstack-kolla14:33
*** hamdyk has quit IRC14:36
*** primeministerp has quit IRC14:44
*** priteau has joined #openstack-kolla14:50
mgoddardmeeting in 10 minutes14:52
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne ^14:53
mnasiadkayoctozepto: thanks for following up on my ceph patch, I'm busy^2 today...14:53
yoctozeptomnasiadka: no problem, need to get this sorted out, I just applied first thought, hope it gets it done14:55
mnasiadkayoctozepto: it should14:55
*** munimeha1 has joined #openstack-kolla14:59
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Oct 16 15:00:48 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
mgoddard\o15:00
jovial[m]0/15:01
yoctozeptoo/15:01
generalfuzzo/15:01
priteauo/15:01
osmanlicilegio/15:02
mgoddard#topic agenda15:02
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:02
chasono/15:02
mgoddard* Roll-call15:02
mgoddard* Announcements15:02
mgoddard** OpenStack Train final release this week15:02
mgoddard** OpenStack Queens becomes EM next week on Friday15:02
mgoddard* Review action items from last meeting15:02
mgoddard* CI status15:03
mgoddard* Train release planning15:03
mgoddard* Review priorities15:03
mgoddard* Which images should we mark as maintained in the support matrix?15:03
mgoddard* Reducing load of build & publishing jobs on CI & Dockerhub (continued) https://etherpad.openstack.org/p/kolla-train-image-evaluation15:03
mgoddard#topic announcements15:03
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:03
mgoddard#info OpenStack Train final release this week15:03
mgoddardI think it should happen today15:03
mgoddard#info OpenStack Train final release this week15:03
mgoddard#info OpenStack Queens becomes EM next week on Friday15:03
yoctozeptomgoddard: it happened15:03
mgoddardfinal release?15:04
yoctozepto#link http://lists.openstack.org/pipermail/openstack-announce/2019-October/002024.html15:04
mgoddardoh yeah15:04
mgoddardthanks15:04
yoctozeptonp15:04
*** ivve has quit IRC15:05
mgoddardI think we agreed to stop publishing images in extended maintenance (EM), so we should disable queens soon15:05
yoctozeptonext Friday15:05
yoctozeptoor later if they postpone15:05
yoctozeptobut yeah, we should do this as soon15:05
mgoddard#topic Review action items from last meeting15:05
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:05
mgoddardyoctozepto to investigate tacker + glance store15:05
mgoddardmgoddard to EOL pike15:05
mgoddardmgoddard to backport publishing job timeout increase15:05
mgoddardmnasiadka to look at not building or publishing deprecated images15:06
mgoddardwe had a few15:06
mgoddardyoctozepto: I saw your email15:06
mgoddardso we need shared storage now?15:06
yoctozeptomgoddard: indeed :<15:07
yoctozeptoit does not look like tacker is too organized now15:07
openstackgerritMark Goddard proposed openstack/kolla stable/stein: CI: increase both limits to 3h  https://review.opendev.org/68895415:08
yoctozeptothey are not too responsive either15:08
mgoddardshould we do the same as glance and enable only on one host?15:08
mnasiadkaand mark that limitation in docs?15:08
mnasiadka(or a red note)15:08
mnasiadkarel note :)15:09
mgoddardif only we had some location independent way of storing images :/15:09
mgoddardsounds fair15:09
mgoddardyoctozepto: are you going to pick it up?15:09
yoctozeptomgoddard: one instance of tacker?15:11
mgoddardyes15:11
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: CI: increase both limits to 3h  https://review.opendev.org/68895615:11
yoctozeptook, let me see to that15:11
mgoddardthanks15:11
yoctozeptoI don't know what tacker does really15:12
yoctozeptowill need to read up ;-)15:12
mgoddard#action yoctozepto to fix tacker & NFV scenario job15:12
mgoddardsomething NFV something15:12
yoctozeptothat's what I know :-)15:13
mgoddardI proposed a patch for EOL pike, still waiting for +A15:13
mgoddardjust now I did the other action item :)15:13
mgoddardmnasiadka: I don't think you did yours did you?15:14
*** scottsol has joined #openstack-kolla15:14
mgoddard#action mnasiadka to look at not building or publishing deprecated images15:14
mnasiadkamgoddard: nope, I started locally, it wasn't a two-liner - and got a lot of work on the OVN PoC this week, but hopefully I can look into that tomorrow15:14
mgoddardk15:15
mgoddard#topic CI status15:15
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:15
mgoddardNFV still borked15:15
mgoddardwe have some issues on ubuntu train in kolla?15:15
yoctozeptoyes15:15
mgoddard#link https://bugs.launchpad.net/kolla/+bug/184829515:15
openstackLaunchpad bug 1848295 in kolla train "trove and rally unbuildable for ubuntu binary" [Critical,Triaged]15:15
yoctozeptoubuntu ceph scenario is broken atm15:15
mgoddardah, that too15:15
yoctozeptoand this15:16
yoctozeptoyeah, both15:16
mgoddardis there a bug for the ceph issue?15:16
mgoddardlooks like no15:16
yoctozeptonope15:16
mnasiadkaI don't think so15:16
mnasiadkalet me raise it15:16
yoctozeptobut proposal is there15:16
yoctozepto:D15:16
mgoddardthanks15:17
mgoddardotherwise I think we're ok on the kolla side?15:17
mgoddardjovial[m] is looking at baremetal boot issues in kayobe CI15:18
yoctozeptomnasiadka: fails bslurp now https://1ef7fd5154b0aa7f60f5-cfb50cc5198678afd23d9756bd7ac094.ssl.cf5.rackcdn.com/688392/6/check/kolla-ansible-ubuntu-source-ceph/f4a235d/primary/logs/ansible/deploy15:18
yoctozeptomgoddard: kolla yup15:19
jovial[m]yeah, not much luck so far. I've got a wild hunch that it may be because we are using `cpu-model` and this is broken with haswell cpus.15:19
mnasiadkayoctozepto: it was the same with previous version of the patch - I'll look into it tomorrow (or you can)15:19
jovial[m]`host-model`15:19
yoctozeptomnasiadka: let it be on you then15:20
mgoddardjovial[m]: certain it's not a networking issue?15:20
mgoddardreproduced it locally?15:20
jovial[m]pretty sure - unless the kernel is getting corrupted on transit15:21
mgoddardwould be good if you could pass a checksum argument15:21
mgoddardwe could try switching from pxe to ipxe?15:22
mgoddardudp -> tcp15:22
mgoddardanyways, better move on15:23
mgoddard#topic Train release planning15:23
*** openstack changes topic to "Train release planning (Meeting topic: kolla)"15:23
mgoddardThe train approaches15:23
mgoddardand the opportunity for puns is running out15:23
jovial[m]yeah, using IPXE seems like a good idea.15:23
mgoddardwe have IPv6 and cells with FFEs, both have a +2 but last to merge will need a rebase15:24
mgoddardgeneralfuzz asked about FFE for TLS CA patches15:25
mgoddard#link https://review.opendev.org/#/c/686024/15:25
mgoddardI've had my head down on cells, and am away tomorrow and friday15:26
generalfuzzI sure did15:26
yoctozeptotrain releases just today15:26
yoctozeptoin two weeks we could get it in15:27
mnasiadkaor three months15:27
yoctozeptoffe +1 from me ;-)15:27
yoctozeptomnasiadka: then not15:27
*** dpawlik has joined #openstack-kolla15:27
yoctozeptobut yeah15:27
yoctozeptowe have 3 months formally ;p15:27
mnasiadkayes15:27
mgoddardwell we did agree on this friday as a deadline for features15:27
yoctozeptohmm, indeed15:27
yoctozeptoregarding order15:28
yoctozeptoI am in favor of ipv6 first15:28
yoctozeptothen I can rebase cells on that15:28
yoctozeptosince mgoddard is gone15:28
yoctozeptoCI and dougsz verify that15:28
yoctozeptoand mnasiadka + hrw make it w+115:28
mgoddardif we have at least two cores sign up to review TLS and its dependency I'd accept an FFE15:28
mnasiadkaI can sign up to review it - it would be a waste to merge the installation method in base container, but not the deployment part.15:29
yoctozeptome too15:30
mgoddardok, thanks15:30
generalfuzz+1 thanks15:31
yoctozeptomnasiadka: opv6 -> https://review.opendev.org/68157315:31
yoctozeptoipv6*15:31
*** dpawlik has quit IRC15:31
yoctozeptoor we can accept hrw's +215:31
mgoddard#topic Review priorities15:32
*** openstack changes topic to "Review priorities (Meeting topic: kolla)"15:32
mgoddardWe have a few feature patches with RP+115:32
mgoddardI just added it to the TLS CA patch too15:33
mnasiadkayoctozepto: just to be sure - are we merging ipv6 first and rebasing cells afterwards, or the other way around? :)15:33
mgoddardI'd say it would be easier to merge cells first, as ipv6 is more search + replace friendly15:34
mgoddardbut whatever works15:34
yoctozeptomnasiadka: ipv6 first for less CI work15:35
*** neolee has joined #openstack-kolla15:35
yoctozeptomgoddard: you do realize it's more or less the same stuff either way? ;p15:35
yoctozeptoyou either get it incoming or existing15:35
neoleehello guys15:36
neolee{"msg": "The conditional check 'easy_install_available' failed. The error was: template error while templating string: no test named 'version'. String: {{ not (ansible_distribution == 'Ubuntu' and\n        ansible_distribution_major_version is version(18, 'ge'))\n   and\n   not (ansible_distribution == 'Debian' and\n15:36
neoleeansible_distribution_major_version is version(10, 'ge')) }}\n\nThe error appears to have been in '/usr/share/kolla-ansible/ansible/roles/baremetal/tasks/install.yml': line 104, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Install pip\n  ^ here\n"}15:36
yoctozeptooh my15:36
mgoddardyoctozepto: sort of15:36
mgoddardipv6 can lean on grep to verify correctness, cells cannot15:36
mgoddardanyways15:37
mgoddardneolee: we are currently in a meeting15:37
yoctozeptomgoddard: what diffs are for15:37
yoctozeptoI guess you just have a different approach at things ;p15:38
neoleeok15:38
openstackgerritWill Szumski proposed openstack/kayobe-config-dev master: Switch to using IPXE  https://review.opendev.org/68896215:38
yoctozeptoanyways, let's continue15:38
neoleesorry15:38
mgoddardfor priorities, there will be some release patches coming. I'll RP+1 those15:38
mgoddardThen let's focus on bugs15:38
mgoddardPlease check launchpad bug lists, so we can merge any fixes we have proposed15:39
mgoddardAlso gerrit review dashboards15:39
mgoddard#link https://bugs.launchpad.net/kolla/train/+bugs15:40
mgoddard#link https://bugs.launchpad.net/kolla-ansible/train/+bugs15:40
mgoddardand beware any bugs not yet tagged as train15:40
mgoddardAnything else we need to prioritise?15:41
yoctozepto*chuckles* nfv *chuckles*15:41
yoctozeptobut it's on me15:41
mgoddard#topic Which images should we mark as maintained in the support matrix?15:41
*** openstack changes topic to "Which images should we mark as maintained in the support matrix? (Meeting topic: kolla)"15:41
mgoddard#link https://docs.openstack.org/kolla/latest/support_matrix.html15:42
yoctozeptothat is the question15:43
*** pbing19 has joined #openstack-kolla15:43
mgoddard#link https://etherpad.openstack.org/p/kolla-train-image-evaluation15:43
mgoddardI made a new section15:43
*** k_mouza has quit IRC15:43
mgoddardImages to mark as maintained15:43
mgoddardL5815:43
mgoddardthrow up your suggestions, comment on others15:43
*** k_mouza has joined #openstack-kolla15:43
mgoddardThe definition of 'maintained' in this case is 'kolla core team is maintaining versions'15:44
mgoddardbut not tested in CI15:44
pbing19I am building the container using local horizon source code. but somehow my customization changes are not coming into the horizon container15:45
yoctozeptopbing19: meeting ongoing15:45
yoctozeptomgoddard: images we care about I guess15:45
yoctozeptowe can repeat those T15:45
mgoddardI guess anything with T & C, we bump C to M15:46
pbing19yoctozepto:ok15:46
*** dpawlik has joined #openstack-kolla15:47
yoctozeptomgoddard: yup15:48
mgoddardanyone seeing the list in that etherpad?15:48
mnasiadkayup15:49
mgoddardare there others we should bump to M?15:50
*** scottsol has quit IRC15:51
mgoddarddoes 'maintaining versions' have meaning for binary images?15:51
mgoddardand for source images can we apply M to anything managed by version-check.py? i.e. most official openstack15:52
*** scottsol has joined #openstack-kolla15:52
yoctozeptomgoddard: not sure, I think M needs some knowledge from us15:52
yoctozeptoso yeah15:52
yoctozeptoat least source15:52
yoctozeptobinary excluded15:52
*** scottsol has quit IRC15:52
yoctozeptoand those we have experience with15:52
yoctozepto(or at least pretend)15:53
mgoddardwhat do you mean by knowledge?15:53
yoctozeptomgoddard: have you ever deployed dragonflow15:54
mgoddardyoctozepto: no, and it is deprecated :)15:54
yoctozeptobut surely you could find more :-)15:54
mgoddardmaybe we need a better definition of M15:54
mgoddardwe're not saying it's tested and working, that's T15:55
mgoddardwe're just saying we bump versions from time to time right?15:55
mnasiadkayes15:56
yoctozeptomgoddard: hmm, I think that's too lazy15:56
mgoddardcompare with elasticsearch, it will only be updated if someone proposes it15:56
yoctozeptoI thought it would be something like we don't test but we generally know how to troubleshoot15:56
yoctozeptolet's see what the list looks like when your idea is applied15:56
mgoddardaodh though will be updated each release, and point releases within it15:56
yoctozeptoand then excluded exotic things15:57
yoctozeptoshall we15:57
mgoddardok15:57
hrwmorning15:59
*** rgogunskiy has quit IRC16:00
*** scottsol has joined #openstack-kolla16:00
mgoddardnew list up16:02
mgoddardI think considering our job as maintainers here might help16:03
mgoddardwe don't promise to fix all bugs16:03
mgoddardbut part of our job for the images involves keeping the versions of deployed software up to date16:03
*** jdwidari has quit IRC16:04
mgoddardM could be interpreted as those things we regularly keep updated16:04
mgoddardC is more community updated16:04
yoctozeptohmm, ok16:04
mgoddardif you have a better deinition I'm open to it16:05
mgoddardanyway, we're out of time16:06
mgoddardthanks all16:06
mgoddard#endmeeting16:06
*** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:06
openstackMeeting ended Wed Oct 16 16:06:34 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:06
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-10-16-15.00.html16:06
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-10-16-15.00.txt16:06
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-10-16-15.00.log.html16:06
*** gfidente has quit IRC16:08
neoleeyay time for questions16:09
osmanlicilegicore guys, you're doing great. the community appreciates your work and effort.16:10
neoleehi guys, i got this bug with centos , https://bugs.launchpad.net/kolla-ansible/+bug/181349216:10
openstackLaunchpad bug 1813492 in kolla-ansible stein "easy_install doesn't exist in 18.04" [High,Fix released] - Assigned to Mark Goddard (mgoddard)16:10
neoleeany idea?16:10
mgoddardthanks osmanlicilegi :)16:10
yoctozeptoosmanlicilegi: thanks :-)16:11
kplantagreed, you guys are killing it16:11
mgoddardosmanlicilegi, kplant: if you have a little time for some reviews it would be appreciated16:12
osmanlicilegimgoddard: for me the roadmap seems fine16:13
*** cah_link has quit IRC16:14
hrwyoctozepto: so which patch goes first? ipv6 or cells?16:16
hrwmgoddard: stable/* patches fail on pep8 due to no yamlint patch which is in gates16:17
mgoddardhrw: ah, true16:18
hrw;D16:18
openstackgerritMark Goddard proposed openstack/kolla master: Tidy up release notes for Train  https://review.opendev.org/68896816:19
*** scottsol has quit IRC16:23
*** neolee has quit IRC16:24
*** jbadiapa has quit IRC16:27
yoctozeptohttp://www.linux-magazine.com/Online/News/Red-Hat-Announces-CentOS-Stream?utm_source=Linux+Update&utm_campaign=Linux_Update_196_Configuring_Bluetooth_2019-16-10&utm_medium=email16:27
yoctozeptohrw: ipv6 :D16:27
hrwyoctozepto: want +W now or plan to update it?16:29
yoctozeptohrw: now, this very moment ;D16:29
hrwdone16:30
yoctozeptothen can already start rebasing cells16:30
hrwyoctozepto: after rebasing cells I expect you to follow with whatever patches were suggested in comments ;D16:30
yoctozeptohrw: you mean ipv6 follow-ups or cells?16:32
hrwipv616:32
hrwcells are mgoddard's16:32
yoctozeptoyeah, I've got docs16:32
yoctozeptoand will probably raise bugs for lacking features16:32
yoctozeptoin case someone wishes to pick them up16:33
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Openvswitch: some ovs tools require ovs daemons pidfiles  https://review.opendev.org/68596416:34
mgoddardyoctozepto, hrw, mnasiadka: I don't think I will be able to attend Bialystok16:35
*** jbadiapa has joined #openstack-kolla16:35
mgoddardbut if you want to meet there in person I would happily dial in16:35
hrw:(16:35
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Neutron: add support to use legacy iptables  https://review.opendev.org/68596716:36
yoctozeptomgoddard: :<16:37
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Tidy up release notes for Train  https://review.opendev.org/68897316:39
*** k_mouza_ has joined #openstack-kolla16:42
*** k_mouza has quit IRC16:45
*** k_mouza_ has quit IRC16:47
*** chason has quit IRC16:48
openstackgerritMerged openstack/kolla stable/stein: CI: pep8: Fix yamllint error on .yamllint  https://review.opendev.org/68887116:49
openstackgerritMerged openstack/kolla stable/rocky: CI: pep8: Fix yamllint error on .yamllint  https://review.opendev.org/68887216:49
openstackgerritMerged openstack/kolla master: Fix source link.  https://review.opendev.org/68893816:49
*** rpittau is now known as rpittau|afk16:50
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Support multiple nova cells  https://review.opendev.org/67565916:53
*** devep has quit IRC16:55
yoctozeptomgoddard: ^ I marked one non-standard place with a comment16:55
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: CI: test cells with IPv6  https://review.opendev.org/68897816:58
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Openvswitch: some ovs tools require ovs daemons pidfiles  https://review.opendev.org/68596417:00
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Openvswitch: some ovs tools require ovs daemons pidfiles  https://review.opendev.org/68596417:03
*** goldyfruit has joined #openstack-kolla17:03
yoctozeptomnasiadka, hrw: you can re-review cells btw17:04
mnasiadkayoctozepto: will do in a minute17:04
mnasiadkaI'm tired of closing one bug in ceph-ansible and finding another17:05
yoctozepto:/17:05
yoctozeptofeeling you17:05
kplantyeah... idk what happened between 3.2 and 4.017:05
*** goldyfruit_ has quit IRC17:05
mnasiadkaI have no clue what their CI does, but it doesn't do what it should.17:05
yoctozeptokplant, mnasiadka: create a group of unhappy downstreams? :D17:05
mnasiadkawell, in Octopus with DeepSea Orchestrator it might change.17:06
mnasiadkaAnd we can end up with spawning one ceph-mgr container and sending a ReST API call17:06
kplantcheck if the yaml syntax is correct and say it's green!?17:06
yoctozeptokplant: could be17:07
mnasiadka(I doubt it will work from the beginning though)17:07
mnasiadkakplant: well, at least it checks yaml syntax :D17:07
yoctozeptomnasiadka: it never does :D17:07
kplantso how are you getting fsid+keys from ceph-ansible if you're using containerized and there's no more fetch dir?17:08
kplantare you just picking what you want from ceph-ansible and using it as a role?17:08
mnasiadkawell, for now I just use a crafted site-container.yml.sample17:14
mnasiadkato make it more streamlined for kolla vars and inventory17:14
mnasiadkakeys are outputted to /etc/ceph/*17:14
mnasiadkaso I can live with that17:15
mnasiadkawell, the fsid... for now I don't need that, but it would be good to save it somewhere17:15
* hrw off17:16
openstackgerritMerged openstack/kayobe master: Try running `docker info` again if it fails  https://review.opendev.org/68847417:17
kplanti think the only reason i get it is to know which fetch directory is valid -_-17:18
kplantsince it's fetch/<fsid>/17:18
*** munimeha1 has quit IRC17:18
*** neolee has joined #openstack-kolla17:25
*** priteau has quit IRC17:27
*** k_mouza has joined #openstack-kolla17:28
mnasiadkakplant: it's good to have when recovering mons17:32
mnasiadka:)17:32
*** k_mouza has quit IRC17:32
* mnasiadka off too17:32
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Support multiple nova cells  https://review.opendev.org/67565917:42
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: CI: test cells with IPv6  https://review.opendev.org/68897817:43
*** dougsz has quit IRC17:44
openstackgerritWill Szumski proposed openstack/kayobe master: CI: Don't set cpu mode  https://review.opendev.org/68899017:47
*** lemko has quit IRC17:55
*** spsurya has quit IRC17:58
openstackgerritMerged openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157318:22
yoctozeptoyahoo :D18:23
osmanlicilegiyoctozepto: congrats :]18:27
yoctozeptoosmanlicilegi: thanks18:28
yoctozeptocells on ipv6 looks passing as well18:28
yoctozeptohopefully to end18:28
*** dpawlik has quit IRC18:30
*** dpawlik has joined #openstack-kolla18:32
*** dpawlik has quit IRC18:39
pbing19i have deployed the container using kolla-ansible. i did modify in my horizon source code before build and check into the container. i could not see the changes19:04
pbing19inside the container19:04
pbing19expected to my changes should come inside /var/lib/kolla/venv/local/lib/python2.7/site-package/horizon19:07
pbing19of container.. am i missing anything?19:08
yoctozeptopbing19: not sure what exactly you did there19:09
yoctozeptodid you rebuild and distribute the modified container image?19:09
yoctozeptohrw, mnasiadka: since mgoddard cannot travel to Bialystok, the details depend mostly on you19:10
pbing19while doing a kolla build i have mention a source and path of my horizon source code into the kolla-build.conf19:11
pbing19and created image then i have pushed my image into local registry19:12
pbing19and then i ran kolla ansible for deployment19:12
pbing19horizon container is up and inside the container i can see the horizon folder as well19:12
pbing19but whatever changes i have done inside the horzon folder its not reftected in my ui19:13
yoctozeptopbing19: did you configure kolla-ansible to use the same registry that you pushed to?19:13
yoctozepto(and the same labels blah blah)19:14
pbing19yes yoctozepto19:14
yoctozeptothen did pull and deploy?19:14
pbing19yes19:14
yoctozeptothen no idea :-)19:14
pbing19just i want to know that if i am using my registry then horizon code should copy to the path /var/lib/kolla/venv/local/lib/python2.7/site-package/horizo19:15
pbing19all py files19:15
pbing19?19:15
pbing19its taking from there only right19:16
yoctozeptoyou should configure kolla build to use your sources wherever they are19:31
yoctozeptothen they will get built into the image19:31
yoctozeptoand installed into venv19:31
yoctozeptofirst inspect the image locally because you might have just built the original image19:32
*** weaselkeeper has joined #openstack-kolla19:37
pbing19i have added the type=local and path of my source code and start build.. do i have any other change to install into venv19:40
weaselkeepertrying to do an update on one of my openstack clusters, can someone tell me what happened to the kolla/ubuntu-source-nova-placement-api container? it doesn't seem to be there any more, did it get replaced by something else?19:43
yoctozeptopbing19: nope19:49
yoctozeptoweaselkeeper: it's gone now19:49
yoctozeptoreplaced by placement-api19:50
yoctozeptoupgrades do handle this19:50
yoctozeptomake sure to use proper version of kolla-ansible and proper release of images19:50
weaselkeeperstill on queens for that cluster :(19:50
yoctozepto(as configured in globals.yml)19:50
yoctozeptoqueens->rocky are supported19:51
yoctozeptoare you trying something else there?19:51
*** kplant has quit IRC19:51
weaselkeeperwell, newbie to K-A so may well be breaking something, but I did a designate deploy and now need to update the rest of the cluster to get neutron to play nice19:51
yoctozeptomake sure you used kolla-ansible for queens19:52
yoctozeptoand configured it to deploy queens19:52
yoctozeptoand you should be just fine19:52
weaselkeeperyah, that's the case.19:52
yoctozeptouse pull and deploy19:52
pbing19yoctozepto: I have checked build log openstack base image its taking from my registry.. and perform the steps as per mention in Dockerfile.j219:52
yoctozeptoit's late here so I am leaving but you can post your versions and links to configs19:52
yoctozeptowill take a look at it right tomorrow19:52
weaselkeeperpull broke on no nova-placement-api, whic is what lead me here :P19:52
yoctozeptoweaselkeeper: and full error message19:53
yoctozeptowill see to making it work19:53
yoctozeptopbing19: run the image locally and inspect its contents19:53
yoctozeptonightie night19:53
*** neolee has quit IRC19:53
pbing19yoctozepto: thanks for your support will do that19:54
pbing19bye19:54
weaselkeeperI wonder if the problem is that our inventory has a "placement-api" group?19:54
weaselkeeper\me waves19:54
* weaselkeeper waves19:54
weaselkeeperfrick I am rusty on irc19:54
weaselkeepernm, I think I screwed something up, kolla/ubuntu-source-nova-placement-api:queens exists, but was failing to pull on some of my nodes.19:58
*** igordc has joined #openstack-kolla20:15
*** pcaruana has quit IRC20:28
*** cally725 has joined #openstack-kolla21:04
cally725just restarted my two controllers and my mariadb is not comming up please help21:05
cally725even after doing kolla-ansible -i INVENTORY mariadb_recovery21:06
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Neutron: Added loading ip6_tables kernel module  https://review.opendev.org/68596521:11
_Cyclone_getting a deploy fatal: "'placement_database_password' is undefined".  Seeing this error deploying Stein.  Did a pull and tried again, same message and stop point (TASK [placement : Copying over placement.conf]).  Not sure if it is related, however confirmed an entry in /etc/kolla/passwords.yml for placement_keystone_password is populated.  Is there something else I can check/confirm?21:17
pbing19cally725: what is the error it shows21:18
cally725pbing19 the mariadb contaner on one controller always shut down after 5 second21:19
*** igordc has quit IRC21:19
pbing19 cally725: while deploying are you facing any error21:20
cally725pbing19 here is the erreor in the fluentd mariadb log on the faulty container [ERROR] WSREP: gcs/src/gcs_group.cpp:group_p21:20
pbing19cally725: not sure21:23
*** igordc has joined #openstack-kolla21:29
*** k_mouza has joined #openstack-kolla22:00
*** k_mouza has quit IRC22:04
*** pbing19 has quit IRC22:53
*** cally725 has quit IRC23:13
*** verdurin has quit IRC23:40
openstackgerritViktor Michalek proposed openstack/kolla-ansible master: Neutron: Added loading ip6_tables kernel module  https://review.opendev.org/68596523:44
*** verdurin has joined #openstack-kolla23:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!