Wednesday, 2020-01-29

*** k_mouza has joined #openstack-kolla00:17
*** munimeha1 has quit IRC00:21
*** k_mouza has quit IRC00:22
*** threestrands has joined #openstack-kolla00:50
*** billp_ has quit IRC00:53
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Copy CA into containers.  https://review.opendev.org/69988801:18
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Use kolla_toolbox to execute REST methods  https://review.opendev.org/70078801:19
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Generate self signed TLS certificates  https://review.opendev.org/70132301:19
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: CI: Add TLS tests  https://review.opendev.org/70141401:19
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic  https://review.opendev.org/66451601:19
*** igordc has quit IRC01:23
*** hamzy has quit IRC01:56
*** hamzy has joined #openstack-kolla01:56
*** jrist has quit IRC01:57
*** weshay has joined #openstack-kolla01:57
*** rphillips has quit IRC01:57
*** weshay|ruck has quit IRC01:57
*** rphillips has joined #openstack-kolla01:58
*** zzzeek has quit IRC02:03
*** zzzeek has joined #openstack-kolla02:08
*** jrist has joined #openstack-kolla02:40
*** goldyfruit_ has quit IRC04:04
*** skramaja has joined #openstack-kolla04:17
*** k_mouza has joined #openstack-kolla04:18
*** k_mouza has quit IRC04:22
*** factor has joined #openstack-kolla04:24
openstackgerritSaravanan KR proposed openstack/kolla master: Add support to build netcontrold binary image  https://review.opendev.org/70472304:40
osmanlicilegimorning05:24
*** sri_ has joined #openstack-kolla05:56
*** Tony31 has joined #openstack-kolla06:06
Tony31hi - is this the right place for Kayobe?06:06
*** waleedm has joined #openstack-kolla06:11
sri_morning06:17
*** k_mouza has joined #openstack-kolla06:18
Tony31hi sri06:18
Tony31Can we try and trade help? :)06:19
*** k_mouza has quit IRC06:23
sri_Tony31, this is right place for Kayobe, but I am not the right person ;(06:28
Tony31ok thanks06:29
Tony31i would like to discuss a possible documentation error (or if the documentation is fine, it could be my misunderstanding of it)06:29
Tony31(will post anyway)06:29
sri_Tony31, just give a 30 min folks  will be here by the time06:29
Tony31ok will do - thank you06:30
sri_np06:32
yoctozeptomorning06:34
yoctozeptowe usually read logs, channel is always open for relevant questions :-)06:35
Tony31hi yoctozepto  - I'm trying to get started with Kayobe and I am stuggling.06:35
Tony31I have installed kayobe in a venv with pip - kayobe commands are working fine. This page mentions config path and config files https://docs.openstack.org/kayobe/train/configuration/kayobe.html#configuring-kayobe06:36
Tony31I dont have any config files or config path as yet and I am confused how to go about getting it.06:36
Tony31On the same page it mentions "<kayobe repo>/ansible/group_vars/all/*" but in the kayobe repo, the paths described in the documentation I am unable to see them..06:37
Tony31is the documentation correct?06:37
Tony31kayobe repo - https://opendev.org/openstack/kayobe/src/branch/stable/train/ansible06:37
Tony31https://opendev.org/openstack/kayobe/src/branch/stable/train  (without the ansible dir)06:37
Tony31I may have figured it out...06:46
*** pbing19 has joined #openstack-kolla06:48
yoctozeptoTony31: cool :-)07:05
Tony31:-s still going around in circles here 🙈07:07
yoctozeptoTony31: what's the issue then?07:19
Tony31so in the repo I can see `etc/kayobe` and I can configure that as the `KAYOBE_CONFIG_PATH`. - but I am confused how the previous section applies in this case since it's outside of the config path. The previous section mentions:07:19
Tony31> Playbook group variables for the all group in <kayobe repo>/ansible/group_vars/all/* set global defaults. These files should not be modified.07:19
*** luksky has joined #openstack-kolla07:20
*** cah_link has joined #openstack-kolla07:27
*** pbing19 has quit IRC07:45
*** pbing19 has joined #openstack-kolla07:45
mnasiadkamorning07:47
mnasiadkayoctozepto: can you reply in https://review.opendev.org/#/c/698303/6 - and let's get over with it? ;-)07:51
patchbotpatch 698303 - kolla-ansible - External Ceph: add ceph_*_user variables - 6 patch sets07:51
yoctozepto08:19:53 <Tony31> > Playbook group variables for the all group in <kayobe repo>/ansible/group_vars/all/* set global defaults. These files should not be modified.07:52
yoctozeptothis is true, no touching :-)07:52
Tony31but how would it be modified when it's in a git within opendev07:53
Tony31i think there is a step or two missing, when following the installation using pip07:53
yoctozeptomnasiadka: both are grammatically correct07:53
Tony31i've taken a few steps back and now gone forward with installation from source. this then clones the repo into the venv/src07:53
yoctozeptoTony31: well, "repo" might not be precise here, true07:54
yoctozeptoit means wherever the source files are07:54
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: External Ceph: add ceph_*_user variables  https://review.opendev.org/69830307:54
mnasiadkayoctozepto: ok, so pushing ;)07:54
Tony31yoctozepto yes I understand it, in the guide there arent any steps to clone the repo... except when using the installation from source.07:54
Tony31Am I correct with understanding that in either case, we need to clone the repo into the venv /src ?07:56
yoctozeptomnasiadka: formatting only07:57
yoctozeptoTony31: yes, this: https://opendev.org/openstack/kayobe-config07:59
yoctozeptothis is kayobe-config07:59
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: External Ceph: add ceph_*_user variables  https://review.opendev.org/69830307:59
mnasiadkayoctozepto: done :)07:59
Tony31🙈 I have not yet come across that link in the docs08:00
Tony31I'll try and go forward as best I can08:00
yoctozeptomnasiadka: +2, hopefully it does not exceed the line length08:00
mnasiadkaoh it does, let me fix08:01
yoctozeptoTony31: it's in the one you linked yourself08:01
yoctozepto07:36:21 <Tony31> I have installed kayobe in a venv with pip - kayobe commands are working fine. This page mentions config path and config files https://docs.openstack.org/kayobe/train/configuration/kayobe.html#configuring-kayobe08:01
yoctozeptojust under the heading08:01
yoctozeptohard to miss :-)08:01
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: External Ceph: add ceph_*_user variables  https://review.opendev.org/69830308:02
Tony31🙈 thanks08:03
mnasiadkaok, morning review series done, I can fetch coffee now ;)08:04
*** sri_ has quit IRC08:06
*** sri_ has joined #openstack-kolla08:07
*** rpittau|afk is now known as rpittau08:08
*** bengates has joined #openstack-kolla08:08
*** Tony31 has quit IRC08:24
*** luksky has quit IRC08:28
*** gfidente has joined #openstack-kolla08:29
*** luksky has joined #openstack-kolla08:31
yoctozeptomgoddard, mnasiadka: I noticed we actually have a mix-and-match when it comes to mod_wsgi or not - e.g. glance is running in eventlet08:44
mnasiadkayoctozepto: well, maybe we need some summary/analysis what is run how - and think about unification (if possible)08:47
*** rgogunskiy has quit IRC08:49
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix Prometheus template generation  https://review.opendev.org/70460108:54
yoctozeptomnasiadka: yeah, true09:09
yoctozeptomnasiadka: whiteboard enriched09:11
openstackgerritMerged openstack/kolla-ansible stable/train: Fix Prometheus template generation  https://review.opendev.org/70460009:11
*** dougsz has joined #openstack-kolla09:14
openstackgerritMerged openstack/kolla-ansible stable/stein: Add also_notifies to Infoblox backend for Designate  https://review.opendev.org/70460309:17
openstackgerritMerged openstack/kolla-ansible stable/train: Add also_notifies to Infoblox backend for Designate  https://review.opendev.org/70460209:17
openstackgerritMerged openstack/kolla-ansible stable/train: Allow ironic_ipxe to serve instance images  https://review.opendev.org/70460509:17
mnasiadkayoctozepto: splendid09:19
*** luksky has quit IRC09:26
*** jbadiapa has joined #openstack-kolla09:27
*** tonythomas has joined #openstack-kolla09:29
mgoddardmorning09:30
mgoddardTony31 gone already?09:30
openstackgerritMerged openstack/kolla-ansible stable/train: External Ceph: copy also cinder keyring to nova services  https://review.opendev.org/70460609:36
openstackgerritMerged openstack/kolla-ansible stable/stein: Enable Glance to use Cinder iSCSI backend  https://review.opendev.org/70460809:36
openstackgerritMerged openstack/kolla-ansible stable/train: Enable Glance to use Cinder iSCSI backend  https://review.opendev.org/70460709:36
openstackgerritMerged openstack/kolla master: Docs: extend release management with "static links to docs" step  https://review.opendev.org/70367509:36
openstackgerritMerged openstack/kolla master: Add error message when merge fails  https://review.opendev.org/70417809:42
openstackgerritMerged openstack/kolla master: Add cinderlib RPM to cinder-volume  https://review.opendev.org/70307509:42
mgoddardmerge merge merge09:42
mgoddardmnasiadka: I think I've worked out that weird DNF issue09:46
mgoddardwe weren't importing the right GPG key for rmq09:46
mnasiadkamgoddard: I would assume dnf can print out an error message though ;-) but good you found the issue :)09:46
mgoddardthat would be nice09:47
*** threestrands has quit IRC09:47
*** luksky has joined #openstack-kolla10:29
openstackgerritPierre Riteau proposed openstack/kayobe stable/stein: Use {{ openstack_branch }} as version of kolla-ansible in ansible tests  https://review.opendev.org/70477110:29
openstackgerritPierre Riteau proposed openstack/kayobe stable/rocky: Use {{ openstack_branch }} as version of kolla-ansible in ansible tests  https://review.opendev.org/70477210:30
openstackgerritPierre Riteau proposed openstack/kayobe stable/rocky: Use stable/rocky as version of kolla-ansible in ansible tests  https://review.opendev.org/70477410:47
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use listen port for Placement precheck  https://review.opendev.org/70459510:47
yoctozepto10:46:13 <mgoddard> we weren't importing the right GPG key for rmq10:54
yoctozeptomight be worth sharing with podman folks10:54
mgoddardyoctozepto: it's not quite that simple10:55
mgoddardof course :)10:55
mgoddardwill explain in a commit message10:55
yoctozeptomgoddard: I see10:55
*** pcaruana has quit IRC10:57
*** rgogunskiy has joined #openstack-kolla10:57
*** priteau has joined #openstack-kolla10:59
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: External Ceph: add ceph_*_user variables  https://review.opendev.org/69830311:07
mnasiadkamgoddard: updated ^^ ;)11:07
mgoddardthanks11:08
mnasiadkamgoddard: git a bit mislead by the filename, and then found out it's an include in nova.conf.j211:08
yoctozeptomnasiadka: yeah, I should have placed in some snippets directory11:10
yoctozeptoit in*11:10
osmanlicilegifolks, unfortunately I won't be able to attend todays meeting. please let me know if there's something related with me cause I'll be out after 14:00 utc and will be back tomorrow.11:11
*** rgogunskiy has quit IRC11:11
yoctozeptoosmanlicilegi: no problem :-)11:11
osmanlicilegiyoctozepto: you all know I'm having issues with the current datacenter. I need to have a quick site visit to equinix today. wishing good luck to myself :]11:14
yoctozepto:-(11:14
yoctozeptogood luck11:14
mnasiadkadata center drama11:17
mnasiadkayoctozepto: nova.conf.d sounds fine, but I think we are not doing that in any other place :)11:17
mgoddardit was a bit of a hack to workaround an ansible issue11:18
mgoddardI wonder if said issue is still present in 2.811:18
*** pcaruana has joined #openstack-kolla11:40
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: Update packages in images  https://review.opendev.org/69941411:42
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: base image fixes for RabbitMQ, fluentd & InfluxDB  https://review.opendev.org/70478211:42
*** calbers has quit IRC11:49
*** calbers has joined #openstack-kolla11:50
*** rpittau is now known as rpittau|bbl11:53
openstackgerritMerged openstack/kolla master: CentOS 8: base and openstack-base images  https://review.opendev.org/69941311:55
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684111:55
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684112:00
*** cah_link has quit IRC12:14
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684112:26
*** rgogunskiy has joined #openstack-kolla12:29
*** kplant has joined #openstack-kolla12:32
*** gfidente has quit IRC12:56
*** gfidente has joined #openstack-kolla12:57
hrwmgoddard: will skip meeting today12:57
*** goldyfruit_ has joined #openstack-kolla13:01
*** pbing19 has quit IRC13:08
*** ktibi has joined #openstack-kolla13:13
*** pbing19 has joined #openstack-kolla13:14
yoctozepto12:18:33 <mgoddard> I wonder if said issue is still present in 2.813:21
yoctozeptocould be gone now, worth checking13:21
yoctozeptothough it's a nice split anyway ;-)13:21
*** skramaja has quit IRC13:24
ktibiHi mgoddard, I see your review about centos8 for images. Any plans for centos8 in KayoB ? (for bifrost)13:29
*** rpittau|bbl is now known as rpittau13:30
yoctozeptoktibi: plans yes, realization pending13:31
ktibiok thx ;)13:32
*** rgogunskiy has quit IRC13:44
*** strigazi has quit IRC13:56
openstackgerritMark Goddard proposed openstack/kayobe stable/rocky: Fix multiple CI failures  https://review.opendev.org/70477413:56
mgoddardapparently docker legacy repo URLs are accessible again13:57
mgoddardhttps://yum.dockerproject.org/repo/main/centos/7/Packages/13:57
mgoddardhttps://github.com/moby/moby/issues/40376#issuecomment-57492749413:58
*** strigazi has joined #openstack-kolla14:02
mnasiadkadoesn't mean we should use them :)14:09
*** gfidente has quit IRC14:13
*** gfidente has joined #openstack-kolla14:13
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: Remove shellinabox from ironic-conductor  https://review.opendev.org/70481314:19
hrwwe moved from them so -1 for going back14:20
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/69354414:20
*** KeithMnemonic has joined #openstack-kolla14:20
*** TrevorV has joined #openstack-kolla14:21
*** zzzeek has quit IRC14:22
openstackgerritMerged openstack/kayobe stable/stein: Use {{ openstack_branch }} as version of kolla-ansible in ansible tests  https://review.opendev.org/70477114:23
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: Use upstream Ceph/master  https://review.opendev.org/70433714:23
*** zzzeek has joined #openstack-kolla14:23
openstackgerritMerged openstack/kolla-ansible master: External Ceph: add ceph_*_user variables  https://review.opendev.org/69830314:26
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/69354414:28
*** sri_ has quit IRC14:32
*** priteau has quit IRC14:50
*** priteau has joined #openstack-kolla14:51
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi14:52
mgoddard^ meeting in 8 minutes14:52
*** Trevor_V has joined #openstack-kolla14:55
*** priteau has quit IRC14:56
*** Trevor__V has joined #openstack-kolla14:57
osmanlicilegihola! I'm here for half hour14:57
*** iputra has joined #openstack-kolla14:58
*** TrevorV has quit IRC14:58
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Jan 29 15:00:05 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:00
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:00
mgoddard\o15:00
osmanlicilegio/15:00
*** Trevor_V has quit IRC15:00
*** scottsol has joined #openstack-kolla15:01
mgoddardmnasiadka, hrw, yoctozepto: around?15:03
yoctozeptoo/15:03
mgoddard#topic agenda15:04
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:04
mgoddard* Roll-call15:05
mgoddard* Announcements15:05
mgoddard* Review action items from last meeting15:05
mgoddard* CI status15:05
mgoddard* Ussuri release planning15:05
mgoddard* HAProxy tag - dougsz15:05
mgoddard* Ussuri goal: https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-contrib-docs.html15:05
mgoddard* CI scenarios: https://etherpad.openstack.org/p/KollaAnsibleScenarios15:05
mgoddard#topic announcements15:05
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:05
mgoddardAnyone have any?15:05
* yoctozepto devstacking15:05
osmanlicileginope15:06
osmanlicilegicongrats yoctozepto again for his devstack journey :)15:06
mgoddard#info yoctozepto now devstack core. All devstack queries to him15:06
mgoddardNo action items last time15:06
yoctozepto:D15:06
yoctozeptothanks15:06
mgoddard#topic CI status15:06
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:06
mgoddardWhiteboard says rocky kolla is broken15:07
mgoddardpost-failure (fluentd)15:07
*** priteau has joined #openstack-kolla15:07
yoctozeptoI *think* it is fixed now...15:07
osmanlicilegishould be fixed15:07
mgoddardlooks ok15:07
yoctozeptoindeed15:08
mgoddardkayobe rocky has an issue, fix proposed15:08
mgoddard#topic Ussuri release planning15:08
*** openstack changes topic to "Ussuri release planning (Meeting topic: kolla)"15:08
yoctozeptok-a clear as well15:08
yoctozeptoah, topic changed ;d15:08
mgoddardI've been spending some time with CentOS 8 recently15:08
yoctozeptoyeah, gj with that :-)15:09
mgoddardmade a status etherpad15:09
mgoddard#link https://etherpad.openstack.org/p/kolla-centos815:09
yoctozeptoand coordinating on the ml15:09
mgoddardyeah, nice team work. would have been nice to add a [kolla] tag though15:10
yoctozeptoyeah, but I also match on my fellow stackers :-)15:10
mgoddardwith our current patches I think we should be able to get the majority of deploy jobs passing15:10
yoctozeptoyeah, just need some reviews15:11
osmanlicilegifrom my side, rabbitmq is ok, just waiting for tripleo-common patches. elk stack is on the way.15:11
yoctozeptomight be able today15:11
yoctozeptohoping so15:11
yoctozeptotired of research stuff15:11
mgoddardwhat are the tripleo-common patches?15:11
yoctozepto(phd in progress)15:11
yoctozeptomgoddard: it's one of their repos15:11
mgoddardyeah I know15:12
osmanlicilegipatches for template-overrides. https://review.opendev.org/#/q/topic:bp/rabbitmq-version-upgrade+(status:open+OR+status:merged)15:12
yoctozeptowhere they override our templates aiar15:12
mgoddardis it for their template overrides/15:12
yoctozeptoafair*15:12
*** TrevorV has joined #openstack-kolla15:12
yoctozeptono need to ask any questions15:12
* yoctozepto delivers answers before questions15:13
mgoddardso related question, how strongly should we pin rmq versions?15:13
mgoddard3.8.2 seems quite specific15:13
yoctozeptoi'd go with 3.8.x15:13
mgoddardif there was a 3.8.3 would we really want to ignore it?15:13
yoctozeptobut rmq is not stricly semver15:13
osmanlicilegithat's what I did at start, 3.8.*15:13
yoctozeptoas they drop erlang support with minors15:13
mgoddard:)15:13
yoctozeptoI'd go 3.8.x anyway15:14
osmanlicilegiif we all agree to go with 3.8.x, I'll do it15:14
mgoddardwhy did you add .2?15:14
yoctozeptodo it!15:14
yoctozeptoyes, why?!15:14
yoctozepto:D15:14
osmanlicilegiI thought someone will give -1 for 3.8.x :)))15:14
yoctozeptoosmanlicilegi: who?! names!15:14
osmanlicilegisecret! :)15:15
yoctozeptowe'll deal with 'em15:15
yoctozeptowith one core15:15
*** Trevor__V has quit IRC15:15
osmanlicilegibtw I'm ok with 3.8.x, will be in ci tonight15:15
mgoddardcool15:15
yoctozepto+115:15
osmanlicilegiso another question related with elk stack15:15
mgoddardto complicate things, are you working on CentOS 8?15:16
osmanlicilegiI was planning to pin it to 6.8.615:16
mgoddardbecause we will drop c7 in master soon15:16
yoctozeptowhy exact again? ;d15:16
osmanlicilegiI will not pin it too :)15:16
yoctozeptohaha, ok15:16
osmanlicilegithe question for rabbitmq is, ci fails on tripleo-build-containers-centos-7 job15:17
yoctozeptoyeah, they need their patches15:17
mnasiadkaI'm here if there were any questions to me, had an internet outage at home ;)15:17
osmanlicilegiand I think we cannot make tripleo-build-containers-centos-7 non-voting15:17
mgoddardhi mnasiadka15:18
yoctozeptomnasiadka: :O15:18
yoctozeptomnasiadka: no questions so far15:18
yoctozeptoosmanlicilegi: we can't permanently15:18
mnasiadkaosmanlicilegi: ask tripleo guys to work with you on a fix in their almighty template overrides?15:18
yoctozeptowe disable it very rarely15:18
yoctozeptowhen things are really bad on ooo side15:18
yoctozeptoso not now15:18
*** pbing19 has quit IRC15:19
osmanlicilegimnasiadka: I often ask on #tripleo but no one replies. I'll keep trying15:19
osmanlicilegianyway, got my notes for rmq and elk15:20
mgoddardosmanlicilegi: our ooo ambassador is cloudnull15:20
cloudnullo/15:21
mgoddardI was about to type 'he's usually responsive'15:21
* cloudnull regrets raising his hand 15:21
cloudnull:D15:21
mgoddard:)15:21
* cloudnull reading back ,15:22
cloudnullsomething I can help with ?15:22
osmanlicilegicloudnull: I'll need your help for https://review.opendev.org/#/q/topic:bp/rabbitmq-version-upgrade+(status:open+OR+status:merged)15:22
mgoddardosmanlicilegi: do you need to add Depends-On in your patch?15:22
yoctozepto(and make it 3.8.*)15:23
mgoddardthen it will pull in the tripleo patch when running the tripleo job15:23
osmanlicilegiI'll send another update and will let you know15:23
osmanlicilegimgoddard: seems I missed it, thx for reminding15:23
mgoddardcloudnull: I think you're off the hook for the moment. It was really a general question about how to contact tripleo15:23
mgoddardthanks for raising your hand :)15:24
yoctozeptocloudnull: thanks for being responsive!15:24
mgoddardany way to mention without ping in IRC/15:25
mgoddard?15:25
yoctozeptomgoddard: nope, unless you want c**loudnull15:25
yoctozepto(remove **)15:25
yoctozeptoor something along these lines15:25
mgoddardyeah15:25
yoctozeptoirc clients are pretty dumb15:26
mgoddardosmanlicilegi: you had an ELK question?15:26
yoctozeptoyou can set them to react to other keywords15:26
yoctozeptolike infra does for config-core, infra-core, infra-root etc.15:26
cloudnull++ if something comes up let me know, im always around and available to provide snark and sarcasm on most subjects :D15:26
osmanlicilegimgoddard: got my answer about pinning :)15:26
yoctozeptoI thought about kolla-core to call us and have it working :-)15:26
yoctozeptocloudnull: snark and sarcasm is what we feed on!15:26
mgoddardcould be useful, could be annoying :)15:27
yoctozeptomgoddard: personal preference, each core configures it by himself15:27
mgoddardosmanlicilegi: so you are going with unpinned for ELK?15:27
yoctozeptoyeah, back to ELK15:27
osmanlicilegimgoddard: yes15:27
yoctozeptoplease keep unpinned15:27
mgoddardok, makes sense15:27
mgoddardwe don't usually pin things, it's just rabbit is a little 'sensitive'15:27
mgoddardwhile it's on topic, should we use bintray for rmq as well as erlang?15:28
mgoddardat the moment we get rmq from packagecloud and erlang from bintray15:28
yoctozeptomgoddard: or just we forgettin to unpin on time ;-)15:28
yoctozeptohmm, weird15:29
mgoddardhttps://www.rabbitmq.com/install-rpm.html15:29
yoctozeptormq provides both compatible15:29
osmanlicilegiusing bintray makes sense to me15:29
yoctozeptomhm15:29
osmanlicilegione repo to rule 'em all15:30
yoctozeptoepel? we already ruled it out15:30
dougsz osmanlicilegi: ELK unpinned means we will currently get 7.x ?15:30
*** bjoernt has quit IRC15:30
yoctozeptodougsz: good q15:30
mnasiadkayeah, rmq is also on GitHub - but I guess erlang is missing there :)15:30
*** bjoernt has joined #openstack-kolla15:30
osmanlicilegidougsz: it will be oss-6.x15:30
yoctozeptook15:31
dougszosmanlicilegi: cool, I have a patch for 7.x if we need it (both kolla and kolla ansible)15:31
yoctozeptocan we go 7.x directly?15:31
dougszUpgrade issues15:31
yoctozeptoand 6 is fine?15:31
yoctozepto(ah, I remember I already asked you)15:31
osmanlicilegi6 is not painful15:31
dougszWe can, but it either involves going to 6 in between, or spinning up two clusters and ingesting the old one15:32
mgoddard5->6 ok, 6->7 ok, 5->7 not ok15:32
yoctozeptoah ok15:32
mgoddardwe have discussed providing both images15:32
mgoddardelasticsearch & elasticsearch715:32
mgoddardsince monasca seems to want 715:33
osmanlicilegiwe can upgrade to 7 on v cycle15:33
mnasiadka6.8 is going to be eol in November 2020, so we can go from 5 to 6, and then from 6 to 7 in V15:33
osmanlicilegigood roadmap for elk15:33
mgoddardmove on?15:33
osmanlicilegiyep15:34
mgoddard#topic HAProxy tag - dougsz15:34
*** openstack changes topic to "HAProxy tag - dougsz (Meeting topic: kolla)"15:34
mgoddarddougsz:15:34
dougszSo in Rocky something like: kolla-ansible reconfigure -t glance15:34
dougszWas well scoped to Glance (and common services)15:35
dougszIn Stein, kolla-ansible reconfigure -t glance can now restart HAProxy, iff the HAProxy config is changed15:35
dougszSo an operation with limited scope, can now take out the whole deployment if things go badly15:36
dougszIt seems sensible to me that we retain the old behaviour15:36
mgoddardThis changed in stein when we split haproxy config out15:37
yoctozeptohmm15:37
yoctozeptothat looks bad indeed15:37
mgoddardWorth pointing out that this behaviour was intentional, and part of the reason for the change15:38
*** Trevor_V has joined #openstack-kolla15:38
mgoddard(although mostly it was to improve haproxy config sanity)15:38
mnasiadkayeah, but it would make sense to allow the user to specify if he wants to restart haproxy or not in the process15:38
dougszThe main reason being to break the HAProxy config file up?15:38
yoctozeptoyeah, it makes sense to enable glance in haproxy when glance tag is called15:38
dougszIt does, but I think the most common operation in a running deployment is to tweak service config files15:39
osmanlicilegimaybe k-a should have an option like --restart-related-services15:39
dougszwhich rarely requires reconfiguring haproxy15:39
dougszSo i would rather do kolla-ansible reconfigure -t glance,haproxy as an exception15:39
mgoddardwhat about scaling out?15:39
kplantosmanlicilegi: shouldn't 'reconfigure' imply that all impacted services will be restarted?15:39
mgoddardI think there are two schools of thought15:40
dougszI think scaling out is likely to be less common than tweaking service config15:40
mgoddard1. --tags glance should only touch glance, and I understand this limitation15:40
yoctozepto1 is more expected I believe15:41
mgoddard2. --tags glance should configure glance and the things it depends on15:41
yoctozepto2 - is it only haproxy atm anyway?15:41
mgoddardand the common stuff15:41
mgoddardwhich has always been the case15:41
*** TrevorV has quit IRC15:41
yoctozeptoah yeah15:42
yoctozeptowell, it would be fine if it needed only a reload15:42
mgoddardan option which might keep everyone happy is to add some new tags15:42
mgoddardglance-haproxy-config15:42
mgoddardso if you want haproxy config you do --tags glance,glance-haproxy-config15:42
yoctozeptolet me look up current docs on tags15:43
mgoddardotherwise, just --tags glance15:43
mgoddardor even more simple, just remove those tags for the haproxy-config imports15:43
mgoddard--tags glance,haproxy15:43
yoctozeptonah, no new features around15:43
*** waleedm has quit IRC15:44
yoctozeptoyeah, that makes sense15:44
yoctozeptoand update docs accordingly15:44
dougszThat sounds reasonable. My vote is to not implicitly reconfigure HAProxy when a service is reconfigured via tags to reduce the change of taking out the deployment15:44
mgoddardbut that would prevent just generating haproxy config for glance15:44
yoctozeptoI like my tags limited15:44
mgoddardhow so?15:45
priteauIs it maybe time to make deploy and reconfigure tasks behave differently? deploy = implicit reconfiguration of dependencies, reconfigure = touch only services listed in tags15:45
dougszalso sounds good ^15:45
osmanlicilegi+115:46
yoctozepto-2, feels complicated15:46
kplanti think that might introduce an opportunity to break things down the line and be very difficult to diagnose. if you call a reconfigure --tags svc and the scoped changes to "svc" are successful BUT the cascaded changes, which were not made, later break the deployment down the line15:46
yoctozepto^15:47
mgoddardthis is the argument against using tags at all15:47
yoctozeptofor the haproxy case I think it's an overkill15:47
yoctozeptomgoddard: yeah, but it's easier to reason when you know all your paths and not need to analyse both deploy and reconfigure15:48
yoctozeptoI'm reluctant15:48
yoctozeptothough I would accept it if proposed and looking sane15:48
yoctozeptofor the haproxy case -15:49
mgoddardansible conditionals and tags don't mix too well15:49
yoctozeptocan we just remove those tags from haproxy and be done?15:49
mgoddardwe could15:50
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: doc: fix bullets in external_ceph.rst  https://review.opendev.org/70483215:51
priteauAnother issue I noticed with the new HAProxy approach: you cannot customise your main haproxy.cfg template to refer to backends from haproxy/services.d, as haproxy is first started with just the single haproxy.cfg file15:51
mgoddardis there ever a case where you would only want to modify haproxy config for one service?15:51
kplanti recently had to change the balance algorithm for keystone only15:51
yoctozeptomgoddard: when you deploy it15:52
yoctozeptobut you still end up restarting the dummy15:52
mgoddardI mean more at deployment time15:52
mgoddarde.g. if you want to roll out changes slowly, keystone, then glance15:52
mnasiadkaI don't think it would be a problem, if we would have --tags haproxy to generate configs for all enabled services and restart haproxy15:52
mnasiadkaalthough mgoddard has a case for doing it one by one :)15:53
mgoddardI'm just worried about removing that fine grained control15:53
yoctozeptoit's far from finegrained15:53
*** Satarel has quit IRC15:53
mgoddardwell it's more than just all haproxy15:53
yoctozeptoif you end up restarting haproxy anyways15:54
mnasiadkawhatever it is now, it seems we need to have an option to modify haproxy config for one service at a time (or choose not to)15:54
yoctozeptowhich is why it's problematic in the first place15:54
yoctozeptothen it's pita more than benefit15:54
mgoddardwhy is that problematic?15:54
* osmanlicilegi bbl15:54
yoctozeptofor dougsz ?15:54
mgoddardhaproxy restart shouldn't be a problem15:54
yoctozeptotaking down the deploy15:54
yoctozeptowell, you drop connections to all the services15:55
yoctozeptoand services and mariadb15:55
mgoddardand if you want to make fine-grained changes then its necessary15:55
mnasiadkawell, if there would be an option for real graceful reload of haproxy - we should be fine then15:56
yoctozeptoyeah15:56
yoctozeptomaybe there is15:56
mgoddardI think it accepts SIGHUP15:56
mnasiadkayoctozepto: there is this socket transfer reload15:56
yoctozeptoalso, if we used a separate haproxy for mysql...15:56
mgoddardmaybe we should look into HUP15:56
mnasiadkayoctozepto: well, maybe we don't need to use haproxy, I think there are more intelligent solutions for load balancing galera15:57
mnasiadkamgoddard: here is the whole story about haproxy reloads: https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/15:57
mgoddardok15:58
mgoddardwe have 2 mins15:58
mgoddardnot sure we have consensus15:59
mgoddardbut let's move on15:59
dougszIt looks like there is also a check mode which can validate the config - not sure if it will catch everything15:59
mgoddardwell we have consensus to change current behaviour at least15:59
mgoddard#topic Ussuri goal: https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-contrib-docs.html15:59
*** openstack changes topic to "Ussuri goal: https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-contrib-docs.html (Meeting topic: kolla)"15:59
mgoddardBefore we leave, there is a 2nd ussuri cross-project goal16:00
mgoddardper-project PTL and contributor docs16:00
mgoddardwe have some, we could have more16:00
mgoddardAnyone want to take this on?16:00
mnasiadkamaybe some fresh contributor could look up the docs and think about what can be improved?16:01
mnasiadkaI don't have a problem in writing something there, but I need to have somebody with a fresh mindset tell us what's missing :)16:01
mgoddardit would be nice if we had docs for contributors, cores and PTL16:01
mgoddardosmanlicilegi is the most recent core, but he left16:02
mnasiadkaif everybody could spent a couple of minutes and add to the Ideas section of the whiteboard what could be improved there - it's a start16:02
mgoddard(the meeting)16:02
mgoddardwhere have these ideas come from?16:03
mnasiadkafrom mine and yoctozepto's heads - blame us :)16:03
*** pbing19 has joined #openstack-kolla16:03
mgoddardwe do not have ideas here. we do as we are told16:04
mgoddardI moved them below kayobe feature status16:04
yoctozeptomgoddard: :<16:04
mgoddardSeems like a good plan to line up things that are not yet priorities but could be in future16:04
mgoddardOr just other things we might want to do16:05
*** iputra has quit IRC16:05
mgoddard#action mnasiadka to herd cats for contributor guide goal16:06
mgoddardcryptic action16:06
mnasiadkaoh crap, let it be16:06
mgoddardanyway, we're over time16:06
mnasiadkaI'll chase osmanlicilegi16:06
mgoddardthanks all16:06
yoctozeptoxD16:06
mgoddard#endmeeting16:06
*** openstack changes topic to "Kolla IRC meetings on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL | Whiteboard: https://etherpad.openstack.org/p/KollaWhiteBoard | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:06
openstackMeeting ended Wed Jan 29 16:06:30 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:06
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-01-29-15.00.html16:06
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-01-29-15.00.txt16:06
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-01-29-15.00.log.html16:06
*** billp has joined #openstack-kolla16:07
mnasiadkayoctozepto: did we follow up with tripleo to deprecate ODL image in Kolla (since they stopped supporting ODL deployments)?16:09
yoctozeptomnasiadka: not remember16:10
yoctozeptoprobably not16:10
openstackgerritMark Goddard proposed openstack/kayobe stable/rocky: Fix multiple CI failures  https://review.opendev.org/70477416:11
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: doc: fix bullets in external_ceph.rst  https://review.opendev.org/70483216:14
yoctozeptomgoddard: you were running kayobe with master k-a16:19
yoctozeptogood lord16:19
mgoddardyoctozepto: only in 'unit tests' for ansible roles16:20
yoctozeptomgoddard: still scary16:20
mgoddardand then it was only minimally using k-a to generate passwords16:20
yoctozeptoglad you have tests though ;p16:20
*** numans has quit IRC16:28
*** ktibi has quit IRC16:29
*** ktibi has joined #openstack-kolla16:30
cosmicsound?resolve_outputs=False&resolve_outputs=False why would be the need for same values?16:32
openstackgerritMichal Nasiadka proposed openstack/kolla master: Deprecate Opendaylight  https://review.opendev.org/70483816:34
mnasiadkayoctozepto, mgoddard: ^^16:34
mgoddardboom16:35
yoctozeptocosmicsound: huh?16:36
yoctozeptomnasiadka: +316:37
*** bengates has quit IRC16:37
openstackgerritMerged openstack/kolla-ansible master: doc: fix bullets in external_ceph.rst  https://review.opendev.org/70483216:38
mnasiadkamgoddard: do we want to wait for removal in V? we build ODL Carbon, which is like 5 releases back compared to current release16:40
yoctozeptomnasiadka: odl deserves a fast drop indeed16:41
yoctozeptoespecially being n/a in c816:41
*** luksky has quit IRC16:42
openstackgerritMichal Nasiadka proposed openstack/kolla master: Deprecate OpenDaylight  https://review.opendev.org/70483816:44
mgoddardmnasiadka: does it hurt us to do a slow drop?16:46
mgoddardif we ignore c816:47
yoctozeptomgoddard: apart from wasting resources and still getting occasional failures on ussuri :-)16:47
yoctozeptonone16:47
yoctozepto:D16:47
mgoddardit's just polite16:47
mgoddardI guess I don't care too much16:47
*** tumble has joined #openstack-kolla16:50
cosmicsoundyoctozepto , haproxy has that in its requests it seems16:52
yoctozeptocosmicsound: to where?16:53
*** dking_desktop has joined #openstack-kolla16:53
cosmicsoundheat_api16:54
cosmicsoundits related to debuging that missing magnum logs in kibana16:54
cosmicsoundJan 29 17:26:46 haproxy[45]: 172.22.100.10:33848 [29/Jan/2020:17:26:46.932] heat_api heat_api/sc 0/0/0/35/35 200 6836 - - ---- 188/1/0/1/0 0/0 "GET /v1/fa1b477c2f0c4bbfa6f82f45980fc317/stacks/kubernetes-4w4sxj4ye7ro/b233374c-a904-475d-8ea6-818dc7d10666?resolve_outputs=False&resolve_outputs=False HTTP/1.1"16:55
cosmicsoundThis is how it looks16:55
yoctozeptoah, heat16:56
tumblehello guys. I'm struggling with "ERROR neutron.agent.rpc RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5". Any idea what I could do solve this mismatch? I'm actually running a kolla-ansible cluster which is why I don't really get how I ended up with a version mismatch16:56
yoctozeptofrom magnum I presume?16:56
*** ktibi has quit IRC16:57
*** ktibi has joined #openstack-kolla16:57
cosmicsoundyep16:59
mnasiadkamgoddard: as long as the repo will not disappear again we will be fine :)17:01
*** ktibi has quit IRC17:04
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: Allow SSH access to keystone_ssh container  https://review.opendev.org/70484817:06
*** ktibi has joined #openstack-kolla17:06
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/69354417:06
yoctozeptocosmicsound: then it's magnum doing it17:13
*** rlljorge has joined #openstack-kolla17:16
rlljorgeHello there, someone running train version with ceph, Can confirm if the cinder restore from snapshot backups are ok ?17:17
*** rlljorge has left #openstack-kolla17:20
*** rlljorge has joined #openstack-kolla17:20
mgoddardstable releases proposed: https://review.opendev.org/70484917:20
patchbotpatch 704849 - releases - Release kolla and kolla-ansible stable branches - 2 patch sets17:20
*** vmixor has joined #openstack-kolla17:29
*** rpittau is now known as rpittau|afk17:33
*** gfidente is now known as gfidente|afk17:38
*** dougsz has quit IRC17:57
*** luksky has joined #openstack-kolla18:09
*** waleedm has joined #openstack-kolla18:09
*** tonythomas has quit IRC18:10
*** negronjl has quit IRC18:18
*** negronjl has joined #openstack-kolla18:21
hrwre18:28
hrwwent quickly through meeting notes18:28
hrwcloudnull: how tripleo looks with move to c8? you wait for the same stuff as we do iirc. so no way to get s/7/8 in tripleo job until c8 is ready for it, right?18:30
hrwosmanlicilegi, mgoddard, yoctozepto: we can grab rmq and erlang from RDO iirc.18:30
cloudnullhrw ++ we're ready to move, but c8 is just not ready yet.18:32
cloudnullso, when will then be now?18:32
cloudnullsoon.18:32
cloudnull:D18:32
*** priteau has quit IRC18:33
hrw;D18:33
*** benj_ has quit IRC18:47
*** benj_ has joined #openstack-kolla18:47
*** vmixor has quit IRC18:50
*** jonaspaulo has quit IRC18:52
dking_desktopCould anybody give me some tips on troubleshooting a baremetal introspection timeout. I'm not really seeing anything useful in ironic-inspector.log. I'm not entirely sure how introspection works, but I can see that using IPMI, when I start introspection, it is successfully able to reboot the server.18:55
kplantyeah, can you edit the image used for introspection and add a plaintext password for root?18:58
kplantthat way you can actually pop into the server via ipmi and look around18:58
kplantand not just watch it boot18:58
*** igordc has joined #openstack-kolla19:02
*** rlljorge has quit IRC19:14
*** oyrogerg has quit IRC19:14
*** pbing19 has quit IRC19:18
*** pbing19 has joined #openstack-kolla19:18
*** waleedm has quit IRC19:23
*** waleedm has joined #openstack-kolla19:23
*** gmann is now known as gmann_afk19:57
dking_desktopkplant: Sorry, I didn't notice that there was a reply. Actually, the server still has an old OS on it, and I can reach it with KVM. I see that it rebooted, but it booted back into the old OS, and then nothing further.20:01
kplantright, what i mean is edit the image that's used for introspection20:01
*** ktibi has quit IRC20:01
kplantso after it PXEs you can log into that image20:01
kplantand poke around the logs on the baremetal machine20:01
kplantironic calls for the node to pxe, it feeds it an image and waits for that image to collect introspection data and shoot it back to the ironic api20:02
kplantif it's not communicating back with ironic, you're not going to see anything on your conductor logs20:02
dking_desktopThat's helpful. So, perhaps it never got it to boot the PXE image. That sounds likely, since I'm still trying to figure out the networking.20:03
dking_desktopI believe that the PXE image is served over a different network?20:04
kplantit would be whatever network ironic is using for provisioning20:04
*** waleedm_ has joined #openstack-kolla20:07
*** waleedm has quit IRC20:07
dking_desktopThat makes sense. I'm going to look at that further. Thanks for the suggestion.20:09
kplantnp. if you're not seeing the machine(s) pxe boot20:09
*** waleedm__ has joined #openstack-kolla20:09
kplanti would look at the network with tcpdump and see if you see the machine soliciting for BOOTP/DHCP20:09
kplantthat would be the first step of pxe20:09
kplantbootp/dhcp -> tftp/http(s) spam20:09
yoctozeptoand: is it configured to boot via pxe20:11
*** waleedm_ has quit IRC20:12
yoctozeptoipmi may fail to set boot method20:12
yoctozeptoor set it to wrong cards20:12
yoctozeptocards=NICs20:12
dking_desktopkplant: Thanks. I'll look at that20:12
dking_desktopyoctozepto: It's set to boot_interface=pxe20:13
dking_desktopWell, it has that and no-bios20:13
dking_desktopI also attempted to set a port with the MAC address, as per the article.20:14
dking_desktopI'm thinking it's the provisioning network, as I don't think I set that up properly.20:15
yoctozeptoit most likely needs to be the untagged network20:16
yoctozeptoconnected to host20:16
kplant^20:16
kplantand if your nics do support vlan tagging for pxe... still go with untagged20:16
kplantbecause it will be transparent to the OS and that's a nightmare20:16
yoctozeptoxD20:16
*** waleedm__ has quit IRC20:16
yoctozeptokplant: yeah, I believe ironic would fail that20:17
yoctozeptoone would need custom setup to support this switch from preboot to boot20:17
yoctozeptoas boot suddenly wants untagged20:17
kplantyeah for sure20:18
kplantduring pxe it would think the network was untagged and then... surprise20:18
*** rlljorge has joined #openstack-kolla20:18
dking_desktopEverything on the control node uses a VLAN. Only the BMC network is accessible outside of a VLAN.20:20
kplantthe node that you're trying to use in ironic, the provisioning network should be native20:21
dking_desktopThat might be an issue.20:24
yoctozeptoyeah, it's always some vlan20:35
yoctozeptodepending on terminology it might be native, default or untagged20:36
*** kplant has quit IRC20:55
*** rlljorge has quit IRC21:07
*** pbing19 has quit IRC21:10
*** gmann_afk is now known as gmann21:17
*** dking_desktop has quit IRC21:41
*** gfidente|afk is now known as gfidente21:46
*** k_mouza has joined #openstack-kolla22:15
*** k_mouza has quit IRC22:19
*** Trevor_V has quit IRC22:31
*** ktibi has joined #openstack-kolla22:52
*** ktibi has quit IRC22:57
*** gfidente has quit IRC23:33

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!