Wednesday, 2021-06-30

jingvarmorning06:51
jingvarvictoria/stable etc/kayobe/kolla.yml kolla_enable_cinder: yes kolla_enable_cinder_backend_nfs: yes kolla_enable_glance: yes 06:53
jingvarcreated /home/centos/kayobe/config/src/kayobe-config/etc/kolla/config/nfs_shares 06:57
jingvarservice deploy is succsesful, but no nfs_shares  in controller1 /etc/kolla/cinder06:59
jingvarTASK [Copying over nfs_shares files for cinder_volume]  skipping: [controller1]07:00
jingvarI think something chandged  enable_cinder_backend_nfs or controllers are not in cinder-volume07:02
jingvarbut I have cinder container07:03
mnasiadkacinder-volume container or cinder-api container?07:13
jingvarhmm, cinder-api07:18
mnasiadkawell then you have the answer, add desired hosts to cinder-volume or storage group07:18
jingvarok, where I can read about kayobe and cinder + nfs, bacause kayobe doc is too short, kolla tells just about enable_cinder_backend_nfs: "yes" and exports file, there is nothing about host groups07:26
jingvarhttps://docs.openstack.org/kolla-ansible/latest/doc-kolla-ansible.pdf  reate/etc/kolla/config/nfs_shares  enable_cinder_backend_nfs:"yes" it is all07:28
jingvarwhat is a reason to place cinder-volume on storages instead of controllers? 07:40
yoctozeptojingvar: cinder-volume actually manages the storage nodes; it is kind of tricky because in the lvm case it has to be colocated with the block devices; in other cases, it might be better off staying on controllers (e.g., with ceph/rbd)07:41
jingvarmain probem is no high level architecture defenition07:45
yoctozeptoyou mean - lack of high level architecture documentation?07:46
yoctozeptowe know the problem but there's always something more important :-(07:47
jingvarI my  head are typical architectures and some times I have spend too much time to resolve unexpected questions07:48
kevkohi, anyone for +2 and +w for this tiny patch ? https://review.opendev.org/c/openstack/kolla-ansible/+/797576 ?08:09
yoctozeptokevko: you are looking for smugglers08:29
mnasiadkawhy do we even need that?08:30
mnasiadkaI remember we had this discussion before, and we decided to direct users to do config overrides instead of setting variables?08:31
mnasiadkaand if we want that, where are the docs? :)08:31
mnasiadkaoh, we already merged in master? doh08:32
mnasiadkathen I'm not touching it :)08:32
yoctozeptomnasiadka: this file cannot merged, hence we (with Mark) decided to allow users to simply override this one var rather than to force them to copy the template and keep it in sync08:37
kevkomnasiadka: well, I can say same about merged change which changed default behaviour of rabbitmq, why it was merged and changed if it can be changed via config override ? 08:38
jingvarconfig/src/kayobe-config/etc/kolla/inventory/overcloud/hosts [cinder-volume:children] -> I changed storage to controll , but after service deploy there is storage instead control08:38
mnasiadkayoctozepto, kevko: so why there's no reno/commit message stating that?08:38
mnasiadkaand no docs how users can use that?08:39
yoctozeptomnasiadka: rare requirement I guess? we should recommend 3-node rabbitmqs08:41
yoctozeptothis is for the power users browsing code08:41
yoctozeptoso it's fine by me08:41
mnasiadkamaybe let's drop docs, because kolla-ansible is for power users08:41
kevko+1 :D 08:42
mgoddardmnasiadka: next patch you propose without docs & reno will get torn to shreds :D08:43
mnasiadkamgoddard: fine by me, I don't have to propose any patches :)08:43
mnasiadkaBut seriously, a one line commit message is probably below our standards.08:44
kevkofrom my perspective of view, if functionality, or changed configuration is not touched ..why there is need to have reno if nothing changed ? defaults is same as hardcoded value08:45
mnasiadkaFine, reno ticked out.08:45
mnasiadkacommit message doesn't explain why, and we agreed we are dropping kolla-ansible docs, right? :)08:45
mgoddardkevko: what mnasiadka is trying to say is, "would you be so kind as to follow up with some documentation?"08:57
jingvarAs I uderstand I can't change [cinder-volume:children ] storage to control beacsue it managed by template in Kolla09:04
mnasiadkajingvar: https://docs.openstack.org/kayobe/latest/control-plane-service-placement.html#custom-kolla-ansible-inventories09:08
mnasiadkaif you're using Kayobe, which I understand you do09:08
jingvarThanks mnasiadka , I'll try it09:11
opendevreviewMaksim Malchuk proposed openstack/kayobe master: Set defaults for openstack_cacert  https://review.opendev.org/c/openstack/kayobe/+/79370309:17
opendevreviewMaksim Malchuk proposed openstack/kayobe master: TLS certificates management sync with Kolla-Ansible  https://review.opendev.org/c/openstack/kayobe/+/79369709:24
yoctozeptomnasiadka: bad day? I understand your love for docs but I believe we have bigger issues in other parts than telling power users how to override the safety vaults ;-)09:32
yoctozeptovalves*09:32
mnasiadkayoctozepto: I'm not a fan of easing overriding of something, that shouldn't be overridden :)09:41
yoctozeptomnasiadka: fair point; we need to find the golden mean though09:50
kevkomnasiadka: I'm not a fan of hardcoding values to configs, if it is default .. it should be in defaults 09:51
mnasiadkaAnyway, my 5 cents is I don't like hiding variables from users, it should end up in the reference rabbitmq docs - maybe we should do it for all reference docs - like a table on top/bottom which vars can you set and what are they for (and if we discourage of setting them for some reason)09:52
jingvarMirantis has an approach09:53
yoctozeptomnasiadka: I agree with you, we should do better in the docs dep09:54
yoctozeptodepartment*09:54
jingvarsystem model with all variables witch use salt formulas and cluster model - it is overrides09:55
jingvaryou can easely find any variable and its value09:55
jingvaryou don't need dive into salt formulas to find something hardcoded09:56
jingvarit is my 5 cent09:57
kevkoyoctozepto: just trying to install masakari  ..do i need to enable also hacluster ? 10:09
mnasiadkakevko: no Masakari guide? :)10:18
kevkomnasiadka: already read .. guide just saying ..turn on masakari by enable_masakari: "yes" ..that's why I am asking :) 10:18
kevko:D10:19
kevkomnasiadka: and of course no docs for hacluster :)10:22
kevkobut no big deal ..i will grep a code :D 10:22
yoctozeptokevko, mnasiadka: oh! I actually forgot about those, sorry! yes, you need to enable hacluster if you want hostmonitor to work ;-)10:28
kevkoyeah, i just realized it when i saw error message in log :) 10:28
yoctozeptokhe khe10:30
yoctozeptocould you report this as a bug?10:30
yoctozeptoand include the error message10:30
yoctozeptoit should work better10:30
kevkoyeah10:39
opendevreviewMaksim Malchuk proposed openstack/kayobe master: TLS certificates management sync with Kolla-Ansible  https://review.opendev.org/c/openstack/kayobe/+/79369711:50
jingvarI removed ~/kayobe, undefine seed and started from the begin12:15
jingvarbut ./dev/seed-hypervisor-deploy.sh  kayobe: 'control host bootstrap' is not a kayobe command.12:15
opendevreviewMaksim Malchuk proposed openstack/kayobe master: TLS certificates management sync with Kolla-Ansible  https://review.opendev.org/c/openstack/kayobe/+/79369712:15
opendevreviewScott Solkhon proposed openstack/kolla-ansible master: Support storing passwords in Hashicorp Vault  https://review.opendev.org/c/openstack/kolla-ansible/+/79299712:30
kevkoyoctozepto: using masakari debian image, it looks like kolla is not adding masakari user to sudoers 12:58
kevkooh, sorry, to kolla group12:58
kevkohmm, no, it is ..weird13:00
opendevreviewMark Goddard proposed openstack/kolla-ansible master: CI: Avoid generating a nova key in cephadm scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/79885013:28
opendevreviewMark Goddard proposed openstack/kolla-ansible master: nova: Use cinder user for Ceph  https://review.opendev.org/c/openstack/kolla-ansible/+/79885113:29
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Add disable_firewall variable  https://review.opendev.org/c/openstack/kolla-ansible/+/79885413:37
opendevreviewMark Goddard proposed openstack/kolla-ansible master: DNM: test disable_firewall  https://review.opendev.org/c/openstack/kolla-ansible/+/79885513:37
opendevreviewScott Solkhon proposed openstack/kolla-ansible master: Support storing passwords in Hashicorp Vault  https://review.opendev.org/c/openstack/kolla-ansible/+/79299714:16
kevkoyoctozepto: should tcpdump be masakari binary dependency or it can be installed via dockerfile  ? 14:29
mnasiadkayoctozepto: https://review.opendev.org/c/openstack/kolla-ansible/+/79885014:55
opendevreviewVerification of a change to openstack/kolla-ansible failed: Make rabbitmq cluster_partition_handling configurable  https://review.opendev.org/c/openstack/kolla-ansible/+/79757614:55
opendevreviewMark Goddard proposed openstack/kolla-ansible master: nova: Use cinder user for Ceph  https://review.opendev.org/c/openstack/kolla-ansible/+/79885114:57
opendevreviewMark Goddard proposed openstack/kolla-ansible master: CI: stop setting ceph_nova_user  https://review.opendev.org/c/openstack/kolla-ansible/+/79891014:57
yoctozeptono meeting today?15:07
yoctozepto#startmeeting kolla15:07
opendevmeetMeeting started Wed Jun 30 15:07:41 2021 UTC and is due to finish in 60 minutes.  The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot.15:07
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:07
opendevmeetThe meeting name has been set to 'kolla'15:07
yoctozepto#chair mgoddard15:07
opendevmeetCurrent chairs: mgoddard yoctozepto15:07
yoctozepto#topic Roll-call15:07
yoctozeptoo/15:07
mgoddarderm15:07
mgoddardoop15:08
mgoddards15:08
yoctozeptono problem15:08
mgoddardI'm not sure where today went :)15:08
yoctozeptolet's harvest participants15:08
yoctozeptocrazy day here as well :-)15:08
mgoddardmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt15:08
mgoddardkoalas assemble15:08
yoctozeptomeeting now, folks :-)15:08
yoctozepto<315:08
mnasiadkaseriously?15:08
mnasiadka:)15:09
yoctozeptobut you have to whisper "assemble"15:09
mgoddard#topic agenda15:09
mgoddard* Roll-call15:09
mgoddard* Agenda15:09
mgoddard* Announcements15:09
mgoddard* Review action items from the last meeting15:09
mgoddard* CI status15:09
mgoddard* Release tasks15:09
mgoddard* Xena cycle planning15:09
mgoddard* Open discussion15:09
mgoddard#topic announcements15:09
mgoddard#info Kolla Wallaby releases now available15:10
yoctozeptoYEAH15:11
mgoddard#link http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023289.html15:11
mgoddard#topic  Review action items from the last meeting15:11
mgoddardthere were none15:12
mgoddard#topic CI status15:12
yoctozeptoneed to merge patches I think?15:12
mgoddardhmm15:13
yoctozeptoon the stable branches15:13
yoctozeptofor kolla15:14
mgoddard#link https://review.opendev.org/q/topic:%22ci-emergency-fix-for-zuul-4-6%2215:14
mgoddardadded my +215:15
mgoddardupgrades to Ussuri fail sporadically on Ubuntu due to Neutron migrations failing15:16
mgoddardmonasca scenario fails all the time15:16
yoctozeptoyes, sir15:16
mgoddardsomething to do with the ES index15:16
mgoddardunsure of the cause15:16
mgoddardI think dougsz is aware15:17
yoctozeptook15:17
mgoddard#topic Release tasks15:17
yoctozeptoubuntu ussuri upgrades are a mystery though15:17
mgoddardI think we still have some R-17 release patches open15:18
mnasiadkaYes, and Heat is a mystery15:19
mgoddardhttps://review.opendev.org/c/openstack/kolla-ansible/+/79642215:19
yoctozeptorefresh our memory please15:19
mnasiadkaand only on debuntu15:20
mnasiadkahttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d4c/796415/3/check/kolla-ansible-debian-source/d4c358b/primary/logs/kolla/heat/heat-api-error.txt15:20
mgoddardCan we at least merge this one? https://review.opendev.org/c/openstack/kolla-ansible/+/79642215:20
mnasiadkahttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d4c/796415/3/check/kolla-ansible-debian-source/d4c358b/primary/logs/kolla/heat/apache-error.txt15:20
mnasiadkamgoddard: I guess we can15:21
yoctozeptowhy enable chrony again?15:24
mgoddardhas anyone tried speaking to the heat team?15:24
mgoddardyoctozepto: only for upgrades, to test the cleanup command15:25
mnasiadkamgoddard: well, once somebody will debug it properly, and find some meaningful error message - it would make sense, but I haven't made it to this point.15:25
yoctozeptomgoddard: are not we testing that in wallaby?15:28
mgoddardyoctozepto: yes, testing the wallaby code15:28
yoctozeptoah, you want to avoid regressions15:28
mgoddardin master we test master code15:28
mgoddardindeed15:29
yoctozeptothere is one issue15:29
yoctozeptoenabled will kill the host one15:29
mgoddard"we disabled CI testing in master because we test ussuri" :)15:29
yoctozepto?15:29
mgoddardI'm being silly15:30
yoctozeptoI see15:30
mgoddardif that is an issue in master then it is an issue in wallaby also15:30
yoctozeptoyeah, we don't restore it for users15:30
yoctozeptook15:31
yoctozeptoas long as it works in ci15:31
mgoddardhold on, don't we disable the host one first, then start it after cleaning the container?15:32
mgoddardto avoid having them both running15:32
mgoddardon the heat issue, should we disable testing of heat in CI for now?15:33
mnasiadkathat's also an option15:34
mnasiadkaI'll do a recheck to check if the issue still persists, if it does will disable heat testing in CI15:34
mgoddardit's only doing openstack orchestration service list15:34
yoctozeptodoes it break on the api call?15:35
yoctozeptoor earlier?15:35
mgoddardlooks like api call https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d4c/796415/3/check/kolla-ansible-debian-source/d4c358b/primary/logs/kolla/heat/heat-api-error.txt15:36
mnasiadkayeah, it's breaking on that15:36
mgoddardAny other release tasks?15:37
yoctozeptocan we call something else?15:37
yoctozeptoon that heat api15:37
mnasiadkait's crashing, so what do you want to call?15:38
yoctozeptoI'm just curious if it's failing listing these15:38
mnasiadkalike the whole api wsgi service is not being run, it's crashing on mod_wsgi running the binary15:39
yoctozeptoyeah, but mgoddard said only after that api call15:39
yoctozeptoand it looks like it15:39
yoctozepto- - - [22/Jun/2021:06:36:44 +0000] "GET / HTTP/1.1" 500 531 1321112 "-" "curl-healthcheck"15:40
mgoddardseems to affect ubuntu only15:40
yoctozeptohmm15:40
yoctozeptoit seems to work on version discovery from curl15:40
yoctozeptoor does it?15:40
mgoddard50015:40
mnasiadkawell, docker healthcheck does something15:40
* yoctozepto blind15:40
yoctozeptolet's see the same for centos15:40
mnasiadkaand gets 50015:41
mgoddardI think we should move on15:42
mgoddardI propose we disable the heat service list on ubuntu, and add it to the workarounds list to be fixed before release15:43
mnasiadkawill do15:44
yoctozepto- - - [22/Jun/2021:06:35:03 +0000] "GET / HTTP/1.1" 300 117 2108380 "-" "curl-healthcheck"15:45
yoctozeptoworking gets 30015:45
yoctozeptoso it's utterly broken15:45
yoctozeptohmm15:45
yoctozeptomust be mod_wsgi version15:45
mgoddardre other release tasks, we should probably do some stable releases soon15:45
mgoddard#action mgoddard check open backports & propose stable releases15:46
mgoddard#topic Xena cycle planning15:48
mgoddardDo we need to take some time to look at open patches, and try to agree some things to focus on15:49
mgoddard?15:49
headphoneJamesWould like to get the Lets Encrypt integration in Xena :)15:49
mgoddard+115:49
headphoneJameshttps://review.opendev.org/c/openstack/kolla-ansible/+/74134015:49
headphoneJamespart 1 is there15:50
mgoddardwe could borrow an idea from ironic15:50
yoctozeptohow so?15:50
mgoddardcore reviewers can sign up to review particular features15:50
yoctozeptoah15:51
mgoddardwhich means that the developer of the feature has an idea if their patch is likely to get merged, and who to ask/bug for help/reviews15:51
mgoddardit might also help focus attention of cores15:52
mgoddardand maybe get more cores involved in reviewing15:52
mgoddardsince the review firehose can be a bit overwhelming15:52
yoctozeptotrue that15:52
mgoddardand it is easy to focus on easy wins15:53
mgoddardjust to get them out of the queue :)15:53
mgoddardany thoughts?15:53
yoctozepto(sorry for mentioning but I need to before I forget, there is a thread on ml which could use mgoddard replying - some kayobe slides are gone)15:53
yoctozeptomgoddard: I agree we could adopt this strategy15:53
mgoddardk, thanks15:53
yoctozeptoyw15:54
yoctozepto(the one from Amit)15:54
mgoddardI see it15:54
mgoddardright now we don't have a great view of features being developed15:56
mgoddardwell, something to think about15:57
mgoddardbright ideas on a postcard please15:57
mnasiadkaWell, it would be nice to have a list of features that need reviewing on some board/etherpad15:57
mgoddardwe previously had priorities on https://etherpad.opendev.org/p/KollaWhiteBoard15:58
mgoddardbut we stopped doing priorities :)15:58
mgoddardI guess they had two flaws15:58
mgoddard1. no guarantee that anyone picks up priorities15:58
mgoddard2. people often contribute unanticipated features that do not get added to the list15:59
yoctozeptoyeah, need to adapt that approach15:59
yoctozeptoto reality15:59
mgoddardso maybe we need more of a separation between 'community priorities as a guide for project direction'15:59
yoctozeptook, we have to wrap up I guess15:59
mgoddardvs work in progress16:00
yoctozeptoyup16:00
mgoddardsounds like a good challenge for the next PTL :)16:00
mgoddardon that note, thanks all16:00
mgoddard#endmeeting16:00
opendevmeetMeeting ended Wed Jun 30 16:00:52 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-06-30-15.07.html16:00
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-06-30-15.07.txt16:00
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-06-30-15.07.log.html16:00
mnasiadkawell, community priorities don't really mean anything now16:01
mnasiadkaat least that's what I see16:01
yoctozeptoI didn't get what you mean mnasiadka16:01
mgoddardmnasiadka: why not?16:02
mnasiadkamgoddard, yoctozepto: there's one more stupid question from me - https://review.opendev.org/c/openstack/kolla-ansible/+/792997/23 - do we want to run that CI job everytime? Maybe we could squeeze it in some other job?16:02
mnasiadkamgoddard: people commit stuff what they think is important to them, not really looking at the priorities16:03
mnasiadkawe can have a list of features we HAVE TO merge this cycle, like upgrade to CentOS 9 Stream16:03
mgoddardmnasiadka: it's nice and short, ~5m - treat it like a unit test job?16:03
mnasiadkabut the rest is nice to have and nobody ever picks it up :)16:03
mnasiadkamgoddard: it takes more time to prepare the environment than to run it - maybe we could run it in the unit tests job? :)16:04
mgoddardmnasiadka: wuchunyang has been quite good at picking up priorities16:05
mgoddardmnasiadka: but largely you are right16:05
mnasiadkamgoddard: it's nice to have a list of ,,nice to have'' features, that we just have for contributors without an agenda - I've been one of them when I worked in IBM and Nokia16:06
mnasiadkabut we can't count anybody will work on them for sure :)16:07
mnasiadkaI think I signed up for doing ansible-core in kolla-toolbox and bumping up haproxy to ease pain of letsencrypt, and that's my plan for August16:07
opendevreviewVerification of a change to openstack/kolla failed: [CI] Do not set ansible_python_interpreter for Zuul  https://review.opendev.org/c/openstack/kolla/+/79810716:08
mnasiadkaI think it would be nice to just have a list of priorities that we want to merge in a particular month or two-month period, if an author is not actively responding/updating - we just put it off the list16:10
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Add disable_firewall variable  https://review.opendev.org/c/openstack/kolla-ansible/+/79885416:10
mnasiadkabut then somebody needs to maintain it, so I don't know :)16:13
mnasiadkayoctozepto: https://review.opendev.org/c/openstack/kolla/+/797275 - another stackhpc change, you need to review :)16:19
opendevreviewVerification of a change to openstack/kolla failed: [CI] Do not set ansible_python_interpreter for Zuul  https://review.opendev.org/c/openstack/kolla/+/79810616:19
headphoneJamesis there a chance I can get a core to look at https://review.opendev.org/c/openstack/kolla-ansible/+/741340 in next few weeks?16:23
mnasiadkaset RP +1, will try to look at it on Friday (and then vanish for three weeks)16:25
headphoneJamesthx!16:26
opendevreviewVerification of a change to openstack/kolla failed: [CI] Do not set ansible_python_interpreter for Zuul  https://review.opendev.org/c/openstack/kolla/+/79810817:11
yoctozeptodoh, ci is broken on epel17:23
opendevreviewMerged openstack/kolla-ansible master: CI: Avoid generating a nova key in cephadm scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/79885017:53
opendevreviewVerification of a change to openstack/kolla failed: Bump up cAdvisor's version  https://review.opendev.org/c/openstack/kolla/+/79727517:57
*** gfidente is now known as gfidente|afk18:11
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Make rabbitmq cluster_partition_handling configurable  https://review.opendev.org/c/openstack/kolla-ansible/+/79757620:01
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Fix the Tempest image url variable  https://review.opendev.org/c/openstack/kolla-ansible/+/78032820:48

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!