Wednesday, 2019-10-23

*** schwicht has quit IRC00:29
*** hamzy__ has joined #openstack-kolla00:51
*** hamzy_ has quit IRC00:53
*** schwicht has joined #openstack-kolla01:19
*** k_mouza has joined #openstack-kolla01:30
*** k_mouza has quit IRC01:36
*** goldyfruit has joined #openstack-kolla02:25
*** spsurya has joined #openstack-kolla02:52
openstackgerritMerged openstack/kolla-ansible master: Add prelude for Train release  https://review.opendev.org/68975003:04
*** radeks_ has joined #openstack-kolla03:36
*** radeks_ has quit IRC03:41
*** radeks has joined #openstack-kolla03:42
openstackgerritMerged openstack/kolla stable/stein: Fix python3 compatibility for barbican  https://review.opendev.org/69000303:47
*** skramaja has joined #openstack-kolla05:52
*** cah_link has joined #openstack-kolla05:53
*** igordc has joined #openstack-kolla05:58
*** gfidente has joined #openstack-kolla06:02
*** igordc has quit IRC06:03
*** dpawlik has joined #openstack-kolla06:05
*** cah_link has quit IRC06:06
*** shyamb has joined #openstack-kolla06:28
openstackgerritDat Le proposed openstack/kolla master: Support Shibboleth service provider for keystone to keystone.  https://review.opendev.org/69047806:54
yoctozeptomorning07:09
BlinkizHello. I would like to understand a thing. I can see that ubuntu containers are using ubuntu-cloud.archive.canonical.com to get newer versions of cloud packages.07:11
yoctozeptoindeed07:11
BlinkizIf I in a container like glance-api run "apt update" I get newer version available to install. Like for example upgrade of cephs librbd1 from 12.2.12 into 13.2.6.07:12
BlinkizHow come I do not have 13.2.6 to begin with here?07:12
Blinkizoh, I see the weight is -1 on packages from that repository07:13
BlinkizOkay. Then I have answered my own question :)07:14
yoctozeptoLaurentDumont: hmm, so you are having an issue already with one external network as well - the problem might be vlan, the physnet1 is configured to be a flat network by default - you have to change that07:14
yoctozeptoBlinkiz: yeah, there can be preferences set07:15
BlinkizI would like to have librbd 13.2.6 in all containers that are using ceph. Is that possible for me to configure?07:15
BlinkizAm thinking about mounting volumes but maybe there is an easier way?07:15
*** shyamb has quit IRC07:15
*** jbadiapa has joined #openstack-kolla07:16
openstackgerritDat Le proposed openstack/kolla master: Support Shibboleth service provider for keystone to keystone.  https://review.opendev.org/69047807:21
yoctozeptoBlinkiz: rebuild the packages without the preferences07:21
BlinkizI did not have compatibility problems between my external v14 ceph and kolla when I was using stein centos containers. Do centos have newer version by default than ubuntu?07:23
BlinkizBecause kolla-ansible's new thing is external ceph this needs to have handled somehow. "ceph-ansible" suggest to use "stable-4.0" branch which means mostly ceph v1407:24
osmanlicilegimorning07:32
hrwkawusia++07:33
*** cah_link has joined #openstack-kolla07:36
*** ivve has joined #openstack-kolla07:39
*** shyamb has joined #openstack-kolla07:39
openstackgerritpengyuesheng proposed openstack/kolla-cli master: Switch to Ussuri jobs  https://review.opendev.org/69048607:40
*** cah_link has quit IRC07:41
*** cah_link has joined #openstack-kolla07:45
yoctozeptohrw: sipping my cappuccino now07:49
hrwblack ft07:49
hrww07:49
mnasiadkaBlinkiz: yes, ubuntu has mimic version of libraries and centos has nautilus08:03
mnasiadkamorning08:03
mnasiadkaBlinkiz: it will work though, unless your cluster does not allow pre-nautilus clients.08:04
Blinkizmnasiadka, yeah it seems to work alright.08:04
*** mgoddard has joined #openstack-kolla08:05
Blinkizmnasiadka, So will Kolla's release of train also use ceph luminous client libraries?08:05
mnasiadkaBlinkiz: we are depending on Ubuntu Cloud Archive for those libraries08:06
mnasiadkaso we'll install whatever they deliver in train08:06
*** rpittau|afk is now known as rpittau08:07
Blinkizmnasiadka, Ubuntu cloud archive has Ceph mimic. But I can see the apt priority on these packages is -108:07
mnasiadkawell, UCA for Train has Nautilus08:07
mnasiadkaI'm not a Ubuntu expert, I have no clue then from where we do install ceph packages08:08
BlinkizCeph is installed from http://mirror.sjc1.vexxhost.openstack.org/ubuntu with apt priority 500. http://mirror.sjc1.vexxhost.openstack.org/ubuntu-cloud-archive is also available but with -1 priority.08:09
hrwwe limited version08:09
BlinkizAm guessing. I do not really know.08:09
hrwtoo late to drop limit in train, have to drop that in Ussuri08:09
BlinkizHow can I find out which version of ceph is installed in the container kolla/centos-source-ceph-base?08:11
mnasiadkarpm -vqa | grep ceph ?08:11
BlinkizI do not have the container locally. Maybe I should just get it and find out.. hmm08:11
hrwmoment08:12
hrw14.2.1 for centos08:12
Blinkizoh08:12
hrw14.2.2 for ubuntu08:12
mnasiadkayeah, nautilus in stein, and nautilus in train08:12
hrw12.2.11 for Debian08:12
Blinkizand 12.2.12 in kolla/ubuntu-source-ceph-base08:12
Blinkizanyway in my stein it is like that08:13
*** dougsz has joined #openstack-kolla08:13
BlinkizHow come you say "14.2.2 for ubuntu"?08:14
hrwmaster08:14
Blinkizalright..08:14
hrwlooked in logs/ubuntu-binary/ceph-base.log08:14
*** k_mouza has joined #openstack-kolla08:14
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637608:15
BlinkizLogs.. Is that something I also can do when am looking for information?08:15
mnasiadkaas I said - it's nautilus in train :)08:15
BlinkizThat is very nice. nautilus in train :)08:15
hrwcentos-binary-test1021/ceph-base.log:INFO:kolla.common.utils.ceph-base:---> Package ceph-osd.x86_64 2:14.2.1-0.el7 will be installed08:15
hrwdebian-binary-test1021/ceph-base.log:INFO:kolla.common.utils.ceph-base:Unpacking ceph-osd (12.2.11+dfsg1-2.1+b1) ...08:15
hrwdebian-binary-test1021/ceph-base.log:INFO:kolla.common.utils.ceph-base:Preparing to unpack .../00-ceph-osd_12.2.11+dfsg1-2.1+b1_amd64.deb ...08:15
hrwubuntu-binary-test1021/ceph-base.log:INFO:kolla.common.utils.ceph-base:Unpacking ceph-osd (14.2.2-0ubuntu3~cloud0) ...08:15
Blinkizhrw, Is these logs public? Can I also look into them somehow?08:16
hrwBlinkiz: those are builds from my local build08:16
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: base: drop not used apt preferences for Ubuntu  https://review.opendev.org/69050708:17
Blinkizhrw, Okay. Well. Am happy that you are saying that ceph libs will be updated to nautilus in kolla-ansible 9.x08:17
hrwmnasiadka: BP+1 ^^08:18
hrws/BP/RP/08:18
Blinkiz#690507 ! Awesome. That was creating questions for me this morning.08:19
mnasiadkaBlinkiz: you can always look at periodic jobs logs for installed package versions: http://zuul.openstack.org/builds?project=openstack%2Fkolla&pipeline=periodic08:21
Blinkizmnasiadka, Thank you :)08:22
*** dswebb has joined #openstack-kolla08:25
*** shyamb has quit IRC08:45
*** shyamb has joined #openstack-kolla08:46
*** pcaruana has joined #openstack-kolla09:02
openstackgerritDincer Celik proposed openstack/kolla-ansible master: [docker] Added a new flag docker_disable_default_network  https://review.opendev.org/68987009:13
mnasiadkayoctozepto: did you raise that change to add rally to unbuildable on ubuntu?09:20
yoctozeptomnasiadka: nah but can do (thought you are going to and then forgot)09:22
mnasiadkaI'll do it :)09:23
openstackgerritRadosław Piliszek proposed openstack/kolla master: Do not build rally image for Ubuntu binary  https://review.opendev.org/69052509:26
yoctozeptomnasiadka: did not see your reply, sorry09:26
mnasiadkano problem :)09:26
mnasiadkadidn't even start09:27
mnasiadkastill it will fail on trove09:27
yoctozeptomnasiadka: but trove was fixed09:27
yoctozeptoin UCA09:27
yoctozeptoonly rally remains b0rken and we get rid of it09:28
yoctozeptoso it should pass as long as there are no new issues09:28
yoctozeptoI doubt there are09:28
mnasiadkaah ok09:28
yoctozeptoI forgot one thing09:29
yoctozeptosupport matrix09:29
yoctozeptohow are we going to handle this now?09:29
yoctozeptochange together09:29
yoctozeptochange separately09:29
yoctozepto?09:29
mnasiadkaI would say change together09:29
mnasiadkamaybe somebody will have cycles in U to make it automatic :)09:30
yoctozeptoah, that would be nice but I doubt it's close to other priorities ;-)09:30
*** shyam89 has joined #openstack-kolla09:30
openstackgerritRadosław Piliszek proposed openstack/kolla master: Do not build rally image for Ubuntu binary  https://review.opendev.org/69052509:32
*** shyam89 has quit IRC09:34
*** shyamb has quit IRC09:35
openstackgerritDoug Szumski proposed openstack/kolla-ansible stable/stein: Perform database migration in monasca upgrade  https://review.opendev.org/69052909:45
openstackgerritDoug Szumski proposed openstack/kolla-ansible stable/rocky: Perform database migration in monasca upgrade  https://review.opendev.org/69053009:45
openstackgerritMark Goddard proposed openstack/kolla stable/stein: CI: Increase job run attempts to 5  https://review.opendev.org/69053109:51
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: CI: Increase job run attempts to 5  https://review.opendev.org/69053209:55
*** hjensas is now known as hjensas|afk09:55
openstackgerritMark Goddard proposed openstack/kolla stable/stein: Revert to uwsgi-plugin-python for barbican source images  https://review.opendev.org/69053309:59
openstackgerritMark Goddard proposed openstack/kolla master: Add example of setting OPENSTACK_RELEASE to release docs  https://review.opendev.org/68973210:01
*** hamzaachi has joined #openstack-kolla10:02
*** shyamb has joined #openstack-kolla10:03
openstackgerritMerged openstack/kolla-ansible master: Extract cephx keys from vault encrypted files  https://review.opendev.org/68975310:04
openstackgerritMerged openstack/kolla master: Add prelude for Train 9.0.0 release  https://review.opendev.org/68974710:04
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637610:07
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Docs: Add IPv6 control plane (address families)  https://review.opendev.org/68969310:09
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Docs: Add IPv6 control plane (address families)  https://review.opendev.org/68969310:10
*** shyam89 has joined #openstack-kolla10:17
*** shyamb has quit IRC10:20
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: run check-logs in post too  https://review.opendev.org/68887010:28
yoctozeptomgoddard, mnasiadka: ^10:29
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Zun: test cinder volume attachment in gate  https://review.opendev.org/68966010:31
*** pcaruana has quit IRC10:41
*** gedcib has joined #openstack-kolla10:45
*** shyam89 has quit IRC10:50
*** shyamb has joined #openstack-kolla10:52
gedcibhi, does someone have domains panel working in horizon? im trying to enable it on rocky.10:54
*** dougsz has quit IRC11:00
*** k_mouza has quit IRC11:00
*** k_mouza has joined #openstack-kolla11:00
*** shyam89 has joined #openstack-kolla11:03
*** skramaja has quit IRC11:04
*** shyamb has quit IRC11:07
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637611:18
*** kplant has joined #openstack-kolla11:19
openstackgerritMerged openstack/kolla-ansible master: Swift: compute the list of containers dynamically in the reconfigure task  https://review.opendev.org/68988211:22
openstackgerritMerged openstack/kolla master: Add example of setting OPENSTACK_RELEASE to release docs  https://review.opendev.org/68973211:22
*** jdwidari has joined #openstack-kolla11:26
*** schwicht has quit IRC11:36
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Tidy up release notes for Train  https://review.opendev.org/68897311:40
*** k_mouza has quit IRC11:50
*** shyam89 has quit IRC11:55
BlinkizOkay. I just installed OpenStack train with todays master branch for kolla-ansible. No hiccups at all. I must have done something wrong :)12:03
*** schwicht has joined #openstack-kolla12:06
hrwBlinkiz: run "kolla-ansible destroy --I-really-really-mean-it" before anyone notice!12:10
kplanti think the openstack thought police have already zeroed in on his position12:11
kplant'fraid it's too late12:11
Blinkizhrw, :)12:12
BlinkizIt is called agile development!12:14
*** goldyfruit has quit IRC12:14
mnasiadkastop the agile marketing please :)12:17
kplantplease12:17
kplantthat cancer has made its way into my org12:18
kplantSAFe!12:18
mnasiadkaoh hell, I ran before my previous employer implemented that12:22
mnasiadkaall those agile release trains, PI planning where nobody knows what to do, etc :)12:23
kplantman... PI planning is nothing but people trying to flex in front of everyone else12:25
kplantand complain12:25
kplantand it's 2 full days of herding cats12:25
mnasiadkayep12:28
mnasiadkaimagine this done remotely with x teams trying to scream at the same time12:28
kplantoh no - we're forced to travel12:28
mnasiadkaand my team was spread between San Jose and India12:28
mnasiadka:)12:28
mnasiadkaso essentially it was more than 8 hours :)12:28
kplantand we also have: required standups, required backlog grooming, required queue replenishments and required retrospectives12:29
kplantwe're of the mindset "we're going to keep having meetings until we find out why no work is being done"12:29
mnasiadkado you have a clone that does that, while you work? :)12:29
mnasiadkahaha, good mindset12:29
kplantno :-(12:29
kplanti usually end up working 10-12 hours day12:30
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove python clients from test-requirements.txt  https://review.opendev.org/69055612:30
*** hamzy_ has joined #openstack-kolla12:30
mgoddardeveryone loves a meeting12:32
*** hamzy__ has quit IRC12:33
mnasiadkamgoddard: you're referring to today's Kolla meeting? :)12:33
mgoddardoh yeah12:33
mgoddardmy meeting brings all the koalas to the channel12:34
mgoddard"my meeting" :p12:34
*** hamzy has joined #openstack-kolla12:35
mnasiadkayour PI planning :D12:36
kplantweekly queue replenishments12:36
kplantya scrum master, you12:36
*** cah_link has quit IRC12:36
*** cah_link has joined #openstack-kolla12:37
openstackgerritPierre Riteau proposed openstack/kayobe master: Add new variables for customising the deployment image in Bifrost  https://review.opendev.org/69055912:37
*** hamzy_ has quit IRC12:37
*** schwicht has quit IRC12:39
*** dougsz has joined #openstack-kolla12:40
*** amoralej is now known as amoralej|lunch12:44
yoctozeptoOUR meetings are productive12:51
*** pcaruana has joined #openstack-kolla12:53
*** dpawlik has quit IRC12:57
*** dave-mccowan has joined #openstack-kolla12:57
hrwI skip today's one12:57
mgoddardkplant: I will add an item to our agenda for today: SAFe kolla12:57
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: run check-logs in post too  https://review.opendev.org/68887012:58
kplant(-:13:01
yoctozeptocrazy, crazy, crazy is my scrum master :D13:01
*** hamzy has quit IRC13:03
*** yan0s has joined #openstack-kolla13:07
*** schwicht has joined #openstack-kolla13:08
*** dpawlik has joined #openstack-kolla13:09
*** radeks has quit IRC13:16
*** dave-mccowan has quit IRC13:17
*** k_mouza has joined #openstack-kolla13:17
*** radeks has joined #openstack-kolla13:18
*** amoralej|lunch is now known as amoralej13:23
*** goldyfruit has joined #openstack-kolla13:25
hrwmnasiadka: what your boss told for a trip to Białystok?13:39
mnasiadkahrw: oh hell, forgot to ask - will do tomorrow13:39
openstackgerritMerged openstack/kayobe master: Use releases.o.o for upper constraints  https://review.opendev.org/68261713:39
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Provide sensible default cell group names  https://review.opendev.org/69003913:47
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637613:48
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Provide sensible default cell group names  https://review.opendev.org/69003913:51
*** hamzy has joined #openstack-kolla13:53
openstackgerritMerged openstack/kolla-ansible master: Perform database migration in monasca upgrade  https://review.opendev.org/68982613:58
*** ivve has quit IRC14:05
*** dpawlik has quit IRC14:11
openstackgerritDoug Szumski proposed openstack/kolla-ansible master: Provide sensible default cell group names  https://review.opendev.org/69003914:12
*** dave-mccowan has joined #openstack-kolla14:16
*** priteau has joined #openstack-kolla14:21
*** X-dark has joined #openstack-kolla14:34
openstackgerritMerged openstack/kolla stable/rocky: CI: Increase job run attempts to 5  https://review.opendev.org/69053214:36
*** dave-mccowan has quit IRC14:37
X-darkHi, quick question. I cannot find in the doc what networks are needed to be reachable from the provisioning node. More specifically, does kolla needs to reach the internal api network or SSH access to the nodes is enough?14:39
mgoddardhi X-dark - SSH access should be sufficient14:41
*** dr_feelgood has joined #openstack-kolla14:43
X-darkmgoddard: ok, thx14:43
* hrw out14:44
openstackgerritMark Goddard proposed openstack/kolla master: Fix mariabackup arguments  https://review.opendev.org/69058914:44
kplanti think i'm going to miss the meeting today, i was going to float the idea of bumping rabbitmq to 3.8 in U for the built-in prometheus and grafana support14:45
kplantno need for an external exporter :-)14:45
mlozahi, it possible to have kolla-ansible install designate with powerdns as backend?14:49
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Zun: test cinder volume attachment in gate  https://review.opendev.org/68966014:52
mgoddardmeeting in 10 minutes14:52
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne ^14:52
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use mariabackup for database backups  https://review.opendev.org/69059814:53
openstackgerritMerged openstack/kolla stable/stein: CI: Increase job run attempts to 5  https://review.opendev.org/69053114:53
noxoidmloza, not natively but i have it configured to do just that14:56
*** pcaruana has quit IRC14:57
mlozanoxoid: did you had to set designate_backend: "bind9" to complete the deployment and then switch the config to powerdns?14:58
noxoidi commented it out14:58
noxoidand provided my own pools.yml14:58
noxoidthere arent many modifications needed to get it to work14:58
noxoidjust a few to prevent kolla from installing bind where it is not needed14:58
noxoidpm if you want more info, these guys are going to have their meeting14:59
mgoddard#startmeeting kolla15:01
openstackMeeting started Wed Oct 23 15:01:20 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
*** openstack changes topic to " (Meeting topic: kolla)"15:01
mgoddard#topic rollcall15:01
openstackThe meeting name has been set to 'kolla'15:01
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:01
mgoddard\o15:01
yoctozeptoo/15:01
generalfuzz o/15:01
yoctozeptonoxoid, mloza: contribute to docs :-)15:02
yoctozeptomnasiadka: ?15:02
mlozayoctozepto: ok once I have successful setup15:03
mnasiadkao/15:03
mgoddard#topic agenda15:03
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:03
mgoddard* Roll-call15:03
mgoddard* Announcements15:03
mgoddard** OpenStack Queens becomes extended maintenance (EM) on Friday15:03
mgoddard* Review action items from last meeting15:04
mgoddard* CI status15:04
mgoddard* Train release planning15:04
mgoddard* Review priorities15:04
mgoddard* Idea: community meetings15:04
mgoddard* Which images should we mark as maintained in the support matrix?15:04
mgoddard* Reducing load of build & publishing jobs on CI & Dockerhub (continued) https://etherpad.openstack.org/p/kolla-train-image-evaluation15:04
mgoddard#topic announcements15:04
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:04
mgoddard#info OpenStack Queens becomes extended maintenance (EM) on Friday15:04
mgoddardWe will then disable our Queens publishing jobs15:04
mgoddardAny others?15:04
yoctozeptona-ah15:05
mgoddard#topic Review action items from last meeting15:05
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:05
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use mariabackup for database backups  https://review.opendev.org/69059815:05
mgoddardyoctozepto to fix tacker & NFV scenario job15:06
mgoddardmnasiadka to look at not building or publishing deprecated images15:06
mgoddardyoctozepto: done, right?15:06
mgoddardmnasiadka: not done?15:06
yoctozeptofixed, though ugly, I want to check if this glance store works like the glance api's store15:06
yoctozeptothen we could just apply analogous config15:06
mnasiadkaHaven’t looked, focused on fixing RDO and UCA ;)15:06
yoctozeptoand be done with ceph backend15:06
*** ni291187 has joined #openstack-kolla15:07
yoctozeptoi'll treat it as a bug fix if you don't mind :-)15:07
yoctozeptoprobably be done before final release15:07
yoctozeptoso not that breaking15:07
mgoddardok15:08
mgoddard#topic CI status15:08
yoctozeptoparaphrasing, yoctozepto looking at keeping ha for tacker-conductor15:08
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:08
yoctozeptogreen, fixing ubuntu binary just today15:08
mgoddardnice15:09
mgoddardkayobe is currently broken due to the inventory change for cells15:09
mgoddardpatch in review15:09
yoctozeptoI noticed kayobe still has some struggle?15:09
yoctozeptoah15:09
mgoddarddo we still have issues with ceph keys in CI?15:10
yoctozeptomnasiadka: ^15:11
yoctozeptoI don't think so15:11
*** dswebb has quit IRC15:11
mnasiadkaWere there any?15:11
mnasiadkaAh, ubuntu python315:12
mgoddardI kept seeing some a while ago15:12
mnasiadkaNo, runs smoothly now15:12
yoctozeptomnasiadka: https://review.opendev.org/688870 - while you around15:12
yoctozepto(shameless plug)15:12
mgoddarddoesn't seem to be an issue15:12
yoctozeptolet's keep going15:12
mgoddardok cool15:12
mgoddard#topic Train release planning15:12
*** openstack changes topic to "Train release planning (Meeting topic: kolla)"15:12
* yoctozepto approving mgoddard's patches15:13
mgoddardI pushed up a bunch of patches on Monday15:13
*** ni291187 has quit IRC15:13
yoctozeptooh, he did15:13
mgoddardstill have https://review.opendev.org/68897315:13
mgoddardstill waiting on RDO15:14
yoctozeptomnasiadka: ^^15:14
mgoddardhttps://review.opendev.org/#/c/689706/15:14
mgoddardwell, I think we're waiting on a tripleoclient release15:15
mnasiadkaYup15:15
yoctozeptoINFO:kolla.common.utils.tripleoclient:No package openstack-tripleo-validations available.15:15
mgoddardafter that we can branch15:15
mnasiadkaToday it will be there15:16
mgoddardok15:16
yoctozeptogreat15:16
mgoddardLet's try to work out which bug fixes we want in the release15:16
mgoddard#link https://launchpad.net/kolla/+milestone/9.0.015:16
mgoddardstarting with kolla15:16
mgoddardtop 3 in progress15:17
mgoddardyoctozepto: is rally/trove complete?15:17
*** chasonc has joined #openstack-kolla15:18
mnasiadkayes it is, trove fixed in UCA, we disabled building rally because it's not in UCA15:18
mnasiadkaah not yet15:18
mnasiadkachange is at the gates, rechecked :)15:18
mgoddardcool15:19
mgoddardI have patches for mariabackup15:19
mnasiadkait has Closes-Bug, so will close it.15:19
mgoddardI will revive that mariadb CI job for moar testing15:19
yoctozeptocomplete15:19
yoctozeptomoar CI failures15:19
mgoddardthen we're down to 'apt_sources_list option does not work', which I think we can take or leave15:20
yoctozeptodrop that15:20
yoctozeptothat's 4 years lol15:20
yoctozepto(almost)15:20
mgoddardnow onto our more difficult child15:20
mgoddard#link https://launchpad.net/kolla-ansible/+milestone/9.0.015:20
mnasiadkamgoddard: I think hrw fixed the apt_sources thing?15:20
mgoddardmnasiadka: there is a patch, I had a comment15:20
mgoddardif it is updated & merged, it gets in15:21
mgoddardstarting with 'in progress' bugs15:21
mgoddardmariabackup, ditto above15:21
mgoddardquite a few of these have been open for a while15:22
mgoddard#link https://bugs.launchpad.net/kolla-ansible/+bug/183072415:22
openstackLaunchpad bug 1830724 in kolla-ansible train "fluentd not reconnecting to ES on failures" [Medium,In progress] - Assigned to Krzysztof Klimonda (kklimonda)15:22
*** scottsol has joined #openstack-kolla15:23
mgoddardHas not been updated in some time15:24
mgoddardI'd guess it's not going to land15:24
mgoddardI've commented but will move it out15:24
mnasiadkaI think there's a change a bit abandoned, and I don't know how it relates to current state after moving to td-agent15:24
mgoddardit seems like a useful change though15:25
mgoddardI certainly know some people who need it or something like it15:25
*** cah_link has quit IRC15:25
mgoddardlet's see if they reply15:25
mgoddard#link https://bugs.launchpad.net/kolla-ansible/+bug/183306415:25
openstackLaunchpad bug 1833064 in kolla-ansible train "nova-status upgrade check should be run after DB sync and online migrations" [Medium,In progress] - Assigned to Mark Goddard (mgoddard)15:25
*** k_mouza_ has joined #openstack-kolla15:26
mgoddardthis was a little controversial15:26
*** jovial[m] has joined #openstack-kolla15:26
mgoddardeasy to change, but does make the check less useful15:26
*** k_mouza_ has quit IRC15:26
mgoddard#link https://bugs.launchpad.net/bugs/184562915:27
openstackLaunchpad bug 1845629 in kolla-ansible train "Fluentd config incorrectly parses MariaDB logs" [Medium,In progress] - Assigned to Isaac Prior (wasaac)15:27
mnasiadkamgoddard: we can always run it before and after15:27
mnasiadka(if it makes any sense)15:27
mgoddardI'd like to get that one in, I marked it RP+115:27
mnasiadkayoctozepto: can you look at this change? :)15:28
mgoddardwe have a patch for https://bugs.launchpad.net/bugs/184646715:28
openstackLaunchpad bug 1846467 in kolla-ansible "RabbitMQ container - high CPU load on multicore systems" [Medium,In progress] - Assigned to Jan Vondra (janvondra)15:28
mgoddarddo we want it?15:29
mgoddard(in train)15:29
*** k_mouza has quit IRC15:29
mgoddardit's kind of a feature, but could be used to resolve that bug15:29
yoctozeptomgoddard: I can handle rabbitmq15:29
yoctozeptomnasiadka: look at which one15:29
mnasiadkayoctozepto: fluentd mariadb15:30
mgoddardyoctozepto: thanks15:30
mnasiadkamgoddard: rabbitmq needs some release notes and maybe some docs - after that I think the code is fine.15:30
mnasiadkaah, yoctozepto took this, good15:30
mnasiadkayoctozepto: https://review.opendev.org/#/c/686428/15:30
*** yan0s has quit IRC15:31
mgoddarddougsz was also going to look at https://bugs.launchpad.net/bugs/1845629 (https://review.opendev.org/686428)15:31
openstackLaunchpad bug 1845629 in kolla-ansible train "Fluentd config incorrectly parses MariaDB logs" [Medium,In progress] - Assigned to Isaac Prior (wasaac)15:31
mgoddardas he is fluent in fluentd15:31
mnasiadkagood15:31
mnasiadkamgoddard: https://bugs.launchpad.net/kolla-ansible/+bug/1673944 - approval to move milestone to 10.0 as this is a Wishlist?15:31
openstackLaunchpad bug 1673944 in kolla-ansible "external-ceph task for nova assumes nova/cinder ceph user" [Wishlist,Triaged] - Assigned to Michal Nasiadka (mnasiadka)15:31
mgoddardmnasiadka: +115:32
mgoddardonto triaged bugs15:33
mgoddard#link https://bugs.launchpad.net/kolla-ansible/+bug/184682015:33
openstackLaunchpad bug 1846820 in kolla-ansible "nova-conductor may crash during deploy due to haproxy-keystone 504" [High,Triaged]15:33
yoctozeptomnasiadka: ok, I see, where is your review?15:33
mnasiadkayoctozepto: in progress :)15:33
mgoddardwe have marked it as high, nova as low :)15:34
mgoddarddo we really think it's high?15:34
mnasiadkamgoddard: I had a couple of occurrences of keystone 504 when running ceph-ansible CI... I can propose a patch to improve the logging of mod_wsgi, but there won't be magic - it's hard to find the root cause15:34
mnasiadkamgoddard: it's the worst type of bug, happens sometimes :)15:35
mgoddardwe have seen intermittent issues with keystone in CI since enabling vxlan15:35
mnasiadkamaybe it's a bit vxlan related, but as always it's hard to tell15:36
mgoddardit might just have exposed an issue15:36
yoctozeptomgoddard: since running haproxy15:36
mnasiadkawe could improve vxlan testing - like do ping test with near-mtu size15:36
yoctozeptoI tell you15:36
yoctozeptothis is timeouts on haproxy15:36
mnasiadkaand maybe improve haproxy logging to try to isolate the issue15:36
yoctozeptowe can raise them and see what happens15:37
yoctozeptogeez, I analyzed that, haproxy timeouts waiting15:37
yoctozeptoand closes the connection to the backend15:37
yoctozeptoI just did not get to raising timeout values15:37
yoctozeptobe my guest :D15:37
mgoddardhow high are the timeouts currently?15:38
mnasiadkayoctozepto: so you raise timeouts, I'll improve logging here and there and let's see :)15:38
mgoddard10s for some, 1m for others15:39
mgoddardsounds like mnasiadka and yoctozepto will investigate15:39
yoctozeptomgoddard: where did you get that? ;p15:39
mnasiadkayoctozepto: you wanted to raise haproxy timeouts, we saw that in your eyes :D15:40
mgoddardyou seem to have it in hand :)15:40
yoctozeptoI meant the timeouts, guys15:40
mnasiadkayes, raise the bloody timeouts, I'll make sure if it happens again - we have proper logs.15:40
mnasiadkaagreed?15:40
mgoddardI think the relevant timeout is 1m15:41
yoctozeptoah, geez, I read 1h15:41
yoctozeptoand was like wtf15:41
yoctozeptomnasiadka: more than agreed15:41
mgoddardthat sounds like it should be sufficient. maybe it is blocked in the backend15:41
yoctozeptomgoddard: the relevant is 10s15:42
yoctozeptoat least that what it looked like15:42
yoctozeptowhen I checked15:42
mnasiadkaconnect and check timeout, yeah15:42
mgoddardsure? 504 should be server timeout15:42
mgoddardanyway, needs investigation15:42
yoctozeptoyeah, needs, not remember15:43
mgoddardwhat about this one: https://bugs.launchpad.net/kolla-ansible/+bug/184524415:43
openstackLaunchpad bug 1845244 in kolla-ansible "Nova scheduler is stopped after each reboot" [High,New]15:43
yoctozeptobut it's a timeout, no other relevant errors in there15:43
mgoddardnot targeted for train currently, but it is 'high'15:43
yoctozeptomgoddard: yeah, it's sad15:43
yoctozeptoi'm stuck with it15:43
yoctozeptoit just drops out immediately15:43
mgoddardI'll target to train to get it on our radar15:43
yoctozeptosure, be my guest15:43
yoctozeptobut we need someone to figure this one out15:44
yoctozeptoI am clearly missing something "obvious"15:44
mnasiadkait happens only in stein?15:45
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Zun: fix Cinder (volume) iSCSI support  https://review.opendev.org/69061415:45
yoctozeptomnasiadka: looks like since stein15:45
yoctozeptobut not checked with current train15:45
mnasiadkawell, out of curiosity I can look into that15:45
yoctozeptonor rocky for the record :D15:46
yoctozeptook15:46
mgoddardI have a VM up, I'll give it a poke15:46
*** goldyfruit has quit IRC15:46
mnasiadkayoctozepto: can you put in docker engine version in the bug?15:47
mgoddardare there other k-a bugs we should be focussing on for train?15:47
yoctozepto19.03.215:48
mgoddardor any other train planning things we should discuss?15:49
generalfuzzperhaps TLS?15:49
mgoddardah, hi generalfuzz15:49
mgoddardI think you did not get the promised reviews15:50
*** scottsol has quit IRC15:50
mgoddardstrictly, we did say no more features after last friday15:50
mnasiadkayeah, just got a heart attack because I forgot those15:50
generalfuzzyikes - lower the priority then15:51
*** igordc has joined #openstack-kolla15:51
mnasiadkawe're talking about this change https://review.opendev.org/#/c/686024/ ?15:51
generalfuzzyes15:51
mnasiadkaor there's another one?15:51
mnasiadkait's in merge conflict15:51
mnasiadkaso what's there to review?15:52
mgoddardsure, it needs an update. probably due to cells & ipv615:52
generalfuzzI'll update asap15:52
mnasiadkaI think it had the same problem end of last week15:52
yoctozeptocells more likely15:52
mnasiadkabut maybe I'm wrong15:53
yoctozeptoipv6 unlikely to conflict15:53
mgoddardI think we'll have to push this one out to Ussuri generalfuzz, sorry15:53
generalfuzzok, so it goes15:53
mgoddardwe need to be putting review time into bugs right now15:53
generalfuzzunderstand15:53
mgoddardspeak up once things have calmed down and we'll try to merge it early on in Ussuri15:54
*** scottsol has joined #openstack-kolla15:54
mgoddard6 minutes left, let's move on15:55
mgoddard#topic Idea: community meetings15:55
*** openstack changes topic to "Idea: community meetings (Meeting topic: kolla)"15:55
mgoddardI was speaking to osmanlicilegi who was interested in attending the kolla PTG15:55
mgoddardbut also suggested having some community meetings15:55
mgoddardI think it's a nice idea15:55
mgoddardless development focussed, more about (human) networking15:56
yoctozepto+115:56
mgoddardit could be a good way to onboard new contributors15:56
yoctozepto+115:56
mgoddardI also thought it might be a good idea if people are interested to do a presentation of the onboarding slides I gave in Denver, but via video conference15:57
mgoddard#link https://docs.google.com/presentation/d/11gGW93Xu7DQo_G1LiRDm6thfB5gNLm39SHuKcgSW8FQ/edit15:57
mgoddardcould be a separate thing15:57
mgoddardbut it has information on how to get more involved15:57
mgoddardthoughts?15:57
mgoddardhow often should such a meeting be held?15:57
mgoddardwho would join?15:58
scottsola couple of us would :)15:58
mgoddardgreat15:59
mgoddardI assume osmanlicilegi would15:59
mnasiadkamgoddard: maybe let's try to have it as part of the again virtual PTG?15:59
yoctozeptonice presentation15:59
yoctozeptokayobe needs updating :D15:59
mgoddardyeah15:59
yoctozeptomnasiadka: or semivirtual15:59
yoctozeptoif you come16:00
mnasiadkayeah, we'll see - I need to talk to the boss16:00
mgoddardI was thinking an hour, once a month16:00
mgoddardtoo much? too little?16:00
mnasiadkamgoddard: let's start with having an hour as part of kolla ptg, and then try to schedule another one - month later, and see what is the attendance?16:01
yoctozeptomgoddard: we are always in hurry with an hour16:01
mnasiadkaor something like that16:01
yoctozeptosounds good16:02
mgoddardok, time at the bar16:02
mgoddardthanks all16:02
mgoddard#endmeeting16:02
*** openstack changes topic to "Topic for #openstack-kolla is: support: ask.openstack.org | New to Kolla: docs @ https://docs.openstack.org/kolla/latest/ | | Kolla IRC meetngs on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:02
openstackMeeting ended Wed Oct 23 16:02:48 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:02
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-10-23-15.01.html16:02
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-10-23-15.01.txt16:02
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2019/kolla.2019-10-23-15.01.log.html16:02
yoctozeptomgoddard, mnasiadka: ah, found the culprit16:03
yoctozeptoetcd was destabilizing cinder part in zun16:03
yoctozeptoshould we make coordination default to no?16:04
mnasiadkahow can a key value store destabilize something?16:04
yoctozeptoit's our CI16:04
yoctozepto:-)16:04
yoctozeptotimeouts all around16:04
yoctozeptohaving coordination in cinder does not seem necessary16:05
yoctozeptoin most cases that is16:05
mgoddardexcept for LVM backend?16:05
noxoidyoctozepto, ive heard its necessary when you have multiple cinder-volume instances running16:07
mnasiadkayoctozepto: zun in CI is multinode?16:07
stackedsaxare people headed to Shanghai at all?16:09
yoctozeptowe only enable it with etcd and/or redis16:09
yoctozeptomnasiadka: yes16:09
mnasiadkayoctozepto: then you need coordination backend, unless you just want to spawn single cinder-volume :)16:09
openstackgerritMerged openstack/kolla-ansible master: tox: Keeping going with docs  https://review.opendev.org/68837116:10
openstackgerritMerged openstack/kolla stable/stein: Revert to uwsgi-plugin-python for barbican source images  https://review.opendev.org/69053316:10
mnasiadkastackedsax: good question - I don't think any kolla active core is going16:10
stackedsaxmnasiadka: yeah, we don't have plans to go at the moment.  A couple of us were at Denver, just wondering if it would be useful to have an emissary in Shanghai16:12
mnasiadkawell, maybe in Vancouver next year we could have a physical PTG ;)16:13
yoctozeptomnasiadka: but we have multinode in cinder-lvm16:13
yoctozeptowithout coordination16:13
yoctozeptoand it works16:13
yoctozeptothere is nothing in cinder docs that suggests using redis/etcd as coordinators16:13
mnasiadkayoctozepto: and you use lvm in zun?16:13
yoctozeptomnasiadka: yeah16:14
mnasiadkayeah, then it might work16:14
mnasiadkacinder uses too for coordinated locking, when you use one shared filesystem I guess16:15
yoctozeptomgoddard, mnasiadka, noxoid: https://docs.openstack.org/cinder/train/contributor/high_availability.html16:15
mnasiadkatooz16:15
yoctozeptofound some blurry notes about coordination16:15
yoctozeptolooks like it just defaults to active-passive16:15
noxoidTo support Active-Active deployments a specific driver will need to be configured using the backend_url configuration option in the coordination section.16:15
noxoidthats the main bit16:15
yoctozeptoso we keep ha16:15
yoctozeptoyea16:15
yoctozeptoso active-active destabilizes ci16:16
noxoidi enabled it on my k-a deployment and its been working fine16:16
noxoidin active active16:16
yoctozeptoyeah, on mine with ceph too16:16
yoctozeptobut I disabled to keep things simple anyway16:16
noxoidduring my research i came across a random mailing list post saying it may not be needed when using a ceph backend16:16
yoctozeptoyet it seems to be an extra burden for our CI16:16
*** amoralej is now known as amoralej|off16:16
noxoidbut it seems better to have it than not16:16
yoctozeptomaybe raising those timeouts could help us16:16
yoctozeptowell, with ceph cinder volume is like a messenger who could just passes on the path to the volume16:17
mnasiadkayoctozepto: destabilizes? maybe you just need to increase timeouts :)16:18
yoctozeptomnasiadka: wrote that 2 lines above you16:18
mnasiadkayoctozepto: with ceph you can just have cinder-volume on the controller :)16:18
yoctozeptomnasiadka: exactly16:18
mnasiadkaok, enough16:19
mnasiadkatalk to you tomorrow :)16:19
yoctozeptobuh-bye16:19
*** gedcib has quit IRC16:23
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Add mariadb test  https://review.opendev.org/65566316:27
*** jdwidari has quit IRC16:28
openstackgerritMerged openstack/kolla master: Do not build rally image for Ubuntu binary  https://review.opendev.org/69052516:32
*** scottsol has quit IRC16:32
*** k_mouza has joined #openstack-kolla16:48
openstackgerritMerged openstack/kolla master: tox: Keeping going with docs  https://review.opendev.org/68837316:49
*** igordc has quit IRC16:50
*** rpittau is now known as rpittau|afk16:50
*** hamzaachi has quit IRC16:51
*** k_mouza has quit IRC16:53
*** dougsz has quit IRC16:56
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Zun: test cinder volume attachment in gate  https://review.opendev.org/68966016:57
*** hjensas|afk is now known as hjensas17:00
openstackgerritMerged openstack/kolla-ansible master: CI: run check-logs in post too  https://review.opendev.org/68887017:02
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637617:27
openstackgerritMerged openstack/kolla master: base: drop not used apt preferences for Ubuntu  https://review.opendev.org/69050717:34
*** chasonc has quit IRC17:40
*** amoralej|off is now known as amoralej17:48
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: haproxy timeout - raise value for server  https://review.opendev.org/69070217:58
*** spsurya has quit IRC17:59
*** jdwidari has joined #openstack-kolla18:01
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: DNM: haproxy timeout - raise value for server  https://review.opendev.org/69070218:08
*** hamzaachi has joined #openstack-kolla18:10
*** jdwidari has quit IRC18:10
yoctozeptomgoddard: any idea why we set up memcached haproxy and not use it18:10
yoctozeptoand the variable controlling this was introduced by commit "Make it possible to not haproxy memcached"18:11
yoctozeptolooks like this is some artifact from the ancient time18:11
yoctozeptomaybe switch the default?18:12
yoctozepto(and deprecate and remove)18:12
*** hjensas has quit IRC18:16
*** igordc has joined #openstack-kolla18:24
*** igordc has quit IRC18:32
*** hamzaachi has quit IRC18:33
stackedsaxmnasiadka: would love an in-person meeting!  But only if we run it exactly like the IRC meeting now: speak only in short, single sentences, occasionally hold up a +1 sign, and start and stop exactly on the hour.  Better yet, we could all meet in person but continue to hold the meeting via IRC, so it'll be an entirely silent in-person meeting.18:36
*** pcaruana has joined #openstack-kolla18:38
*** pcaruana has quit IRC18:44
*** k_mouza has joined #openstack-kolla18:48
*** hamzaachi has joined #openstack-kolla18:56
*** amoralej is now known as amoralej|off19:03
*** kplant has quit IRC19:03
*** k_mouza has quit IRC19:07
*** k_mouza has joined #openstack-kolla19:21
*** k_mouza has quit IRC19:26
*** radeks has quit IRC19:35
*** priteau has quit IRC19:40
*** gouthamr has quit IRC19:43
*** igordc has joined #openstack-kolla19:47
*** hamzaachi has quit IRC19:51
*** gouthamr_ has joined #openstack-kolla19:54
*** hamzaachi has joined #openstack-kolla19:55
*** hamzaachi has quit IRC20:09
*** jbadiapa has quit IRC20:10
openstackgerritPierre Riteau proposed openstack/kayobe master: Add new variables for customising the deployment image in Bifrost  https://review.opendev.org/69055920:13
*** dr_feelgood has quit IRC20:15
*** hjensas has joined #openstack-kolla20:20
*** k_mouza has joined #openstack-kolla20:43
*** k_mouza has quit IRC20:56
*** henriqueof has quit IRC21:15
*** schwicht has quit IRC21:31
*** k_mouza has joined #openstack-kolla21:36
*** goldyfruit has joined #openstack-kolla21:38
*** schwicht has joined #openstack-kolla22:02
*** goldyfruit has quit IRC22:04
*** k_mouza has quit IRC22:10
*** gfidente has quit IRC22:15
*** schwicht has quit IRC22:32
*** igordc has quit IRC23:23

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!