Wednesday, 2021-10-27

opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone  https://review.opendev.org/c/openstack/kolla-ansible/+/81557701:24
*** amoralej|off is now known as amoralej06:25
JjayHello Team, 06:55
JjayI deployed openstack with Magnum and it works ok until I create a volume on the cluster template.. The cluster fails with the error "cinder" volume driver needs "cloud_provider_enabled" label to be true or unset.06:56
JjayCan someone assist on how to enable it in Kolla ?06:56
em_Jjay, you have to set https://github.com/EugenMayer/openstack-lab/blob/master/deploy/3_configure_kolla.sh#L5907:07
em_and then pick a storage type, see  https://docs.openstack.org/kolla-ansible/latest/reference/storage/cinder-guide.html07:07
em_and then you will need to define where to put the storage https://github.com/EugenMayer/openstack-lab/blob/master/config/multinode#L2507:08
Jjay<em> Cinder is already enabled and the backend is ceph which is working OK07:08
JjayThe issue is only when i use Magnum 07:10
em_i have zero experience with magnum (for today, guess it comes on my roadmap tomorrow)07:10
em_So cannot help you with that07:11
JjaySure thanks a lot <em_>07:11
em_Since i think kolla has no magnum integration, you should rather check the magnum docs, it's all post kolla deployment AFAICS - see https://docs.openstack.org/magnum/latest/user/#storage07:13
DK4hello, is it possible to deploy an DVR Setup with kolla?07:46
jingvarDK4: yes07:52
DK4but theres no guide for it as far i can see :) all trial and error i guess?07:58
jingvarhttps://docs.openstack.org//kolla-ansible/latest/doc-kolla-ansible.pdf 08:06
jingvarenable_neutron_provider_networks: yes08:07
DK4can you even have the SNAT Routing on computes with dvr?08:11
em_DK4, are you using OVS or OVN? (be cause that differs)08:26
DK4im using the kolla default (ovs)08:29
em_In this case, what jingvar said, AFAIR enable_neutron_provider_networks: yes should be enough for OVS dvr08:33
em_why should you not be able to use SNAT on the compute? Every compute has it's own interface with the public/floating subnet. Packages leaving to the provider network are the SNAT on the compute itself08:34
DK4thanks for the explanation08:42
DK4in the inventory of kolla what would be the [network] section then be? do i add my compute hosts there?08:42
opendevreviewWill Szumski proposed openstack/kayobe master: Do not fail when admin openrc file does not exist  https://review.opendev.org/c/openstack/kayobe/+/81525908:52
opendevreviewMerged openstack/kayobe stable/wallaby: Remove stale config dump from seed hypervisor host configure  https://review.opendev.org/c/openstack/kayobe/+/81523008:57
opendevreviewMark Goddard proposed openstack/kayobe master: Drop become in stackhpc.libvirt-vm for seed vm provision  https://review.opendev.org/c/openstack/kayobe/+/81268709:00
opendevreviewMerged openstack/kayobe stable/victoria: Remove stale config dump from seed hypervisor host configure  https://review.opendev.org/c/openstack/kayobe/+/81523109:20
opendevreviewMerged openstack/kayobe stable/ussuri: Remove stale config dump from seed hypervisor host configure  https://review.opendev.org/c/openstack/kayobe/+/81523209:21
opendevreviewMerged openstack/kayobe master: Fix link syntax in release note  https://review.opendev.org/c/openstack/kayobe/+/81551509:21
jingvarDK4: do you use clean kolla-ansible or kayobe09:28
DK4jingvar: im using clean kolla-ansible currently. but as im in testing still i could look into kayobe if it that makes sense.09:32
opendevreviewPierre Riteau proposed openstack/kayobe stable/xena: Fix link syntax in release note  https://review.opendev.org/c/openstack/kayobe/+/81544909:43
opendevreviewPierre Riteau proposed openstack/kayobe stable/wallaby: Set proxy option in early dnf invocation  https://review.opendev.org/c/openstack/kayobe/+/81465809:45
opendevreviewPierre Riteau proposed openstack/kayobe stable/wallaby: CI: enable DNF tests on CentOS Stream 8  https://review.opendev.org/c/openstack/kayobe/+/81475209:45
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: Debian: enable building Bifrost images  https://review.opendev.org/c/openstack/kolla/+/81478310:04
hrwrebased10:04
em_what is the best way to wipe kolla from a compute (to restart a deployment)? Remove /etc/kolla and wipe all containers?10:22
em_are there any othere leftovers / secrets?10:22
em_DK4, for the network to compute question, i'am new so take this with caution. IMHO network decides where to put the neutron-server. In common, this is put only on the controller10:26
em_DK4, the computes only get the so called neutron-agents (depending on your network type)10:26
em_Please be sure to take this information with caution. Way to new to all thyis10:26
jpichi all, if I have 2 AZs with 3 controlers each, must I have 1 mariadb cluster of 6, or 2 mariadb clusters of 3? I mean, must I have 1 cluster for all AZ, or 1 cluster per AZ? thanks10:32
hrwem_: "kolla-ansible destroy"10:32
adrian-ahi guys, any feedback on https://review.opendev.org/c/openstack/kolla-ansible/+/815043 ?10:32
jingvarHave someone an deployment without central logging ? I need to check harpoxe log config and fluentd -  for somereasons fluentd try to connnet to ELK10:33
hrwadrian-a: will leave some10:34
adrian-ahrw: thanks10:37
hrwdone10:40
em_hrw, is kolla-ansible-deploy available on the actual compute nodes? looks like a binary the deployer would have. or is this then done remotely via ssh?10:45
em_hrw, we found out that beside nuking containers/docker and /etc/kolla one needs to reboot due to the /run/ sockets and the libvirt socket check during verify10:46
hrwem_: I do not know where from you deploy.10:46
hrwem_: kolla-ansible is a tool to do all heavy jobs. check, bootstrap, deploy, upgrade, destroy.10:47
em_hrw, yeah, understood, using it the entire time. My question is, if destroy is nuking the deployer and wiping kolla there, or nuking the nodes (i need the latter)10:48
jingvarkoll-ansible works with containers and their config - maybe volumes if wipe10:50
hrwem_: if you need to completely erase nodes then no, k-a does not have that part10:51
jingvarkayobe configures host software, nic's configuration10:52
opendevreviewAdrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch  https://review.opendev.org/c/openstack/kolla-ansible/+/81504311:04
em_when deploying with kolla, non DVR mode OVN, it seems like kolla forces to create a provider interface on the compute instances, even though it is not required at all. Anybody aware of that issue?11:22
em_So i did not expose the external interface for the compute instances https://github.com/EugenMayer/openstack-lab/blob/stable/ovn/config/group_vars/compute/compute.yml#L3 .. only for the controller (where the gateway service should run on) https://github.com/EugenMayer/openstack-lab/blob/stable/ovn/config/group_vars/control/control.yml#L311:23
em_now, during deployment, kolla tries to createe br-ex on computes ot 11:24
em_to the defaults eth1 (why so ever this interface is used as default) and fails, since it does not exists. Did i get the non DVR mode wrong or is this a kolla glitch?11:24
jpicthis infra i'm looking at has different regions with a mix of boths, some regions have 1 cluster per AZ, others have 1 cluster per region with all AZ, and both /seem/ to work them, any idea is welcome11:33
em_understood, in this mode enable_neutron_provider_networks: no is required11:36
opendevreviewAdrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch  https://review.opendev.org/c/openstack/kolla-ansible/+/81504311:55
opendevreviewWill Szumski proposed openstack/kayobe master: Do not fail when admin openrc file does not exist  https://review.opendev.org/c/openstack/kayobe/+/81525912:15
opendevreviewMerged openstack/kolla-ansible stable/wallaby: mariadb: Do not use wsrep-notify.sh on Debian  https://review.opendev.org/c/openstack/kolla-ansible/+/81466812:32
*** amoralej is now known as amoralej|lunch12:39
opendevreviewWill Szumski proposed openstack/kayobe master: Do not fail when admin openrc file does not exist  https://review.opendev.org/c/openstack/kayobe/+/81525912:41
jingvarwhy /etc/kolla/fluentd/config.json was generated from  kolla-ansible/ansible./roles/common/templates/td-agent.conf.j2 ? but  somehow /etc/kolla/fluentd/td-agent.conf from  ./roles/common/templates/conf12:51
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: CI: make Debian source jobs voting  https://review.opendev.org/c/openstack/kolla/+/81565712:59
priteaujingvar: fluentd config.json is generated from ansible/roles/common/templates/fluentd.json.j213:00
priteaujingvar: td-agent.conf is generated from several input, filter, format and output files.13:02
priteauby td-agent.conf.j2 13:02
jingvar[root@ctl01 fluentd]# grep -r /var/log/kolla/fluentd/fluentd.log13:06
jingvarconfig.json:    "command": "/usr/sbin/td-agent -o /var/log/kolla/fluentd/fluentd.log"13:06
opendevreviewMargaritaShakhova proposed openstack/kolla-ansible master: Fix broken deploy of placement service  https://review.opendev.org/c/openstack/kolla-ansible/+/81552413:08
jingvarpriteau: ohh, I was blind13:08
*** amoralej|lunch is now known as amoralej13:14
jingvar- name: "conf/output/01-es.conf.j2" -> enabled: "{{ log_direct_to_elasticsearch }}" ,  log_direct_to_elasticsearch:  enable_elasticsearch: "{{ 'yes' if enable_central_logging | bool or enable_osprofiler | bool or enable_skydive | bool or enable_monasca | bool else 'no' }}"13:14
jingvaropt/kayobe/src/kayobe-config/etc/kolla/globals.yml:enable_elasticsearch: False13:14
jingvarIf I right understand, but I have td-conf with enabled elastic13:15
priteauDo you have any of these variables set? enable_central_logging / enable_osprofiler / enable_skydive / enable_monasca13:18
jingvaropt/kayobe/src/kayobe-config/etc/kolla/globals.yml:enable_central_logging: False , enable_skydive: False , enable_monasca: False, enable_osprofiler: False13:18
jingvarIt looks starnge - I don't have anything that could trigger elastic13:20
opendevreviewVerification of a change to openstack/kayobe stable/wallaby failed: CI: enable DNF tests on CentOS Stream 8  https://review.opendev.org/c/openstack/kayobe/+/81475213:25
em_is there no way to add port_forward via kolla-ansible? It's missing here https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutron-extensions.html13:35
em_hmm, according to https://blueprints.launchpad.net/kolla-ansible/+spec/enable-l3-port-forwarding it has been implemented13:37
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: make Debian/source voting  https://review.opendev.org/c/openstack/kolla-ansible/+/81566613:37
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: make Debian/source jobs voting  https://review.opendev.org/c/openstack/kolla-ansible/+/81566613:38
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: make Debian source jobs voting  https://review.opendev.org/c/openstack/kolla-ansible/+/81566613:38
opendevreviewMichal Nasiadka proposed openstack/kolla master: docs: weekly meetings page  https://review.opendev.org/c/openstack/kolla/+/81549413:43
opendevreviewPierre Riteau proposed openstack/kolla-ansible master: Add support for deploying prometheus-msteams  https://review.opendev.org/c/openstack/kolla-ansible/+/81267913:44
em_Anybody has port-forwarding up and running while deploying via kolla-ansible? I guess it might differ w ro w/o DVR, but i do not understand how to enable it on the agent13:45
opendevreviewMichal Nasiadka proposed openstack/kayobe master: Build overcloud host image directly with DIB  https://review.opendev.org/c/openstack/kayobe/+/77260913:49
em_ok we found out about enable_neutron_port_forwarding: 'yes' - seems to be missing in the docs (or we cannot find it f13:50
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: CI: make Debian source jobs voting  https://review.opendev.org/c/openstack/kolla/+/81565714:00
opendevreviewAdrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch  https://review.opendev.org/c/openstack/kolla-ansible/+/81504314:01
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: CI: make Debian source jobs voting  https://review.opendev.org/c/openstack/kolla-ansible/+/81566614:02
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: Fix wrong opts in cyborg.config  https://review.opendev.org/c/openstack/kolla-ansible/+/81567214:13
opendevreviewAdrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch  https://review.opendev.org/c/openstack/kolla-ansible/+/81504314:20
opendevreviewAdrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch  https://review.opendev.org/c/openstack/kolla-ansible/+/81504314:38
mnasiadkamgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt14:51
mnasiadkameeting in 914:51
mnasiadka#startmeeting kolla15:00
opendevmeetMeeting started Wed Oct 27 15:00:11 2021 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
opendevmeetThe meeting name has been set to 'kolla'15:00
fricklermnasiadka: where do you keep that ping list? I'd like to add myself15:00
mnasiadka#topic rollcall15:00
priteauo/15:00
mgoddard\o15:00
mnasiadkafrickler: it's currently at https://wiki.openstack.org/wiki/Meetings/Kolla15:00
mnasiadkao/15:00
fricklermnasiadka: thx15:00
hrw[°][_]15:00
yoctozeptoo/15:02
mnasiadka#topic agenda15:03
em_My management network (provider based vswitch), which is used to also tunnel the (non dvr) ovn network, has an mtu of 1400. When adding the provider network, i used a mtu of 1400 - but that seems not to be enough. Do i need to set something with kolla in general? I have found https://docs.openstack.org/neutron/queens/admin/config-mtu.html but not sure that applies here15:03
yoctozeptomnasiadka: need to move it to the whiteboard (the official ping list)15:03
mnasiadkaem_: we have a weekly meeting now - please wait until it ends (around 1hr)15:03
mnasiadkayoctozepto: yup15:03
em_(oh sorry, was not aware about the irc based meeting, will shut up. Excuse me)15:03
mnasiadka* Roll-call15:03
mnasiadka* Agenda15:03
mnasiadka* Announcements15:03
mnasiadka* Review action items from the last meeting15:03
mnasiadka* CI status15:03
mnasiadka* Release tasks15:03
mnasiadka* Yoga cycle planning15:03
mnasiadka* Open discussion15:03
mnasiadka#topic Announcements15:04
mnasiadkaI have none - anyone anything?15:04
mgoddardRC!?15:04
mgoddardRC1?15:04
mgoddardPTG?15:04
yoctozeptoRC!!!!!1111oneoneoneeleven15:04
opendevreviewwu.chunyang proposed openstack/kolla-ansible master: Fix wrong opts in cyborg.conf  https://review.opendev.org/c/openstack/kolla-ansible/+/81567215:04
mnasiadkaAh, right - RC1 for Kolla, Kolla-Ansible and Kayobe has been cut.15:05
mgoddard#info15:05
mnasiadka#info RC1 for Kolla, Kolla-Ansible and Kayobe has been cut.15:05
mnasiadkaOk then, let's move on I guess - unless somebody else wants to announce anything?15:06
mnasiadka#topic Review action items from the last meeting15:07
mnasiadkaSeems there were none.15:07
mnasiadka#topic CI Status15:07
mnasiadkaAre we green?15:07
mnasiadkaSeems we are - based on the whiteboard.15:08
mnasiadka#topic Release tasks15:09
mnasiadkaSo, do we have a list of blockers for doing RC2?15:09
mnasiadkaI think all MariaDB related patches have been merged?15:09
mnasiadkayoctozepto: ?15:10
yoctozeptoyeah, I think so15:11
yoctozeptoany release tasks still to do?15:11
yoctozeptocentos-openstack-release done?15:11
mgoddardthe gerrit dashboards seem a bit broken15:12
mgoddardno project filtering15:12
yoctozeptomayhaps we need also https://review.opendev.org/c/openstack/kolla-ansible/+/81427615:12
yoctozeptofor mariadb15:12
fricklermgoddard: for dashboards you need to remove the /#/ from the path15:13
opendevreviewUwe Grawert proposed openstack/kolla-ansible master: [Grafana] Add unified alerting and smtp options  https://review.opendev.org/c/openstack/kolla-ansible/+/81569415:14
frickleror reload15:14
fricklersee https://gerrit-review.googlesource.com/c/gerrit/+/32153515:14
mgoddardfrickler: thanks15:14
hrwyoctozepto: c-r-o-xena exists15:14
mnasiadkaand we use it15:15
yoctozeptook15:15
yoctozeptoso only that mariadb patch15:15
yoctozeptobut I'm not sure what the impact is15:15
yoctozeptoperhaps it garbles the config15:15
mnasiadkaaround gerrit dashboards - I see master branch in stable branch backports section of Kolla dashboard, so maybe we need to revisit those and check what's going on15:15
priteauAbout CI status, we are amber on wallaby for Kayobe (just updated the booard)15:16
priteauIt's caused by frequent disk full issues15:16
priteauThe wallaby images must be a bit bigger than other releases15:17
priteauWe have a proposed workaround which is to disable heat from CI upgrade jobs15:17
mnasiadkaOk, I think one of the changes to make it better for Wallaby is merging/merged today15:18
mnasiadkayoctozepto: I don't see a bug report in that patch, so can't really tell we should wait to get it reviewed and merged.15:18
mnasiadkaSo, should we post RC2 for Kolla/Kolla-Ansible/Kayobe?15:18
mgoddardhttps://review.opendev.org/c/openstack/kolla-ansible/+/81494215:20
yoctozeptomnasiadka: it's in the reno15:20
yoctozeptomgoddard: good catch15:20
mnasiadkayoctozepto: what about closes-bug?15:21
yoctozeptomnasiadka: yeah, you can comment that on it15:21
yoctozeptobut the bug report is not satisfactory tbh15:21
yoctozeptoit's like writing "it doesn't work."15:21
hrwmnasiadka: let https://review.opendev.org/c/openstack/kolla/+/815440 merge and then RC2?15:22
mnasiadkaok, so two changes are +w and we need to wait for them to merge15:23
mgoddardif we're going to merge this revert then let's do it before release https://review.opendev.org/c/openstack/kolla-ansible/+/81494915:23
mgoddard(discuss)15:23
opendevreviewMerged openstack/kayobe stable/wallaby: Set proxy option in early dnf invocation  https://review.opendev.org/c/openstack/kayobe/+/81465815:24
mnasiadkamgoddard: I think you owe us some more description and reason ;-)15:24
mgoddardI would say the same about the original patch :)15:25
mnasiadkaI added enable_host_ntp and cinder-volume fix as RC2 blockers in the whiteboard (L297)15:25
mnasiadkaWell, the original patch stated everything works, and CI didn't explode - so I'd like to know what does it break ;-)15:25
yoctozeptomgoddard: like mnasiadka said - it was well described :-) 15:26
priteauIf we are issuing RC2 for Kayobe we may want to merge https://review.opendev.org/c/openstack/kayobe/+/812687 in xena15:27
yoctozeptoI don't mind reverting if it really breaks something; but then again we should probably rewrite it to work differently as it does not make sense to use it with ovs native firewall and ovn15:27
mnasiadkapriteau: especially that the comment says "since Xena"... ;-)15:28
mgoddardI don't see where neutron is loading that module15:28
mgoddardI do see neutron will print a warning if it is not loaded15:29
priteauI think br_netfilter can be loaded by docker15:30
mgoddardif it uses iptables15:31
mnasiadkawe default to disable iptables now I think15:31
mgoddardI can try to justify, but not in real time15:31
yoctozeptomgoddard: did it break somewhere IRL?15:31
mgoddardnot yet15:31
mnasiadkabut that means we could have non-working SGs?15:32
yoctozeptothen try to make it break :-) 15:32
yoctozeptoI tried and it works15:32
yoctozeptohence did not bother to improve, just removed15:32
mgoddardit jumped out as one of those patches that could bite us15:32
yoctozeptowell, at least we know how to fix it quickly15:33
mgoddardand my gut has often been right on those in the past but I let them slide then suffer latet15:33
yoctozeptoand we fix when it bites15:33
mnasiadkaSo why change a tradition?15:33
mgoddardanyway15:33
kevkoguys , is this visible in CI ?15:33
kevko2021-10-27 15:11:40.879 25 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: No module named 'ovn_octavia_provider.common': ModuleNotFoundError: No module named 'ovn_octavia_provider.common'15:33
yoctozeptooh well, that's it for the scientific method :D15:33
yoctozeptokevko: guy, we are in a meeting15:33
yoctozepto:-)15:33
kevkooh, sorry :D 15:33
priteauI just have a freshly deployed kolla using xena branches, br_netfilter is loaded on compute hosts15:34
priteauI can try and see what enabled it15:34
mgoddarddo you know how/when it got loaded?15:34
priteauI don't know yet15:35
priteau[Wed Oct 27 14:11:49 2021] Bridge firewalling registered 15:35
yoctozeptonow we know when15:36
yoctozeptoyour mileage may vary though15:36
priteauBut in neutron-openvswitch-agent logs:15:36
priteau2021-10-27 14:08:13.127 7 WARNING neutron.agent.linux.iptables_firewall [req-60c2a81f-edaa-4c34-a2ae-37017aeac72f - - - - -] Kernel module br_netfilter is not loaded.15:36
priteau2021-10-27 14:08:13.128 7 WARNING neutron.agent.linux.iptables_firewall [req-60c2a81f-edaa-4c34-a2ae-37017aeac72f - - - - -] Please ensure that netfilter options for bridge are enabled to provide working security groups.15:36
mnasiadkawell, so after the warning something did it ;-)15:37
hrwthis should be done by something on host before containers start15:37
hrwotherwise we would need to have hostos == containeros15:38
mgoddard    Systems that don't override default settings for those knobs would work15:38
mgoddard    fine except for this exception in the log file and agent resync. This is15:38
mgoddard    because the first attempt to add a iptables rule using 'physdev' module15:38
mgoddard    (-m physdev) will trigger the kernel module loading. In theory, we could15:38
mgoddard    silently swallow missing knobs, and still operate correctly. But on15:38
mgoddard    second thought, it's quite fragile to rely on that implicit module15:38
mgoddard    loading. In the case where we can't detect whether firewall is enabled,15:38
mgoddard    it's better to fail than hope for the best.15:38
mgoddardneutron commit e83a44b96a8e3cd81b7cc684ac90486b283a350715:38
mgoddardwhich I linked to in the br_netfilter patch 2 weeks ago15:39
priteauIt was loaded when I launched a VM15:39
priteaucreated                             | 2021-10-27T14:11:45Z15:40
priteauLoaded 4 seconds late15:40
priteaulater15:40
mgoddardwe should move on15:40
mnasiadkaYup, what's the plan? Leave it as is since it seems to work?15:41
priteauIt works but it produces WARNING messages in logs, that's not nice15:42
priteauWhat do we gain from not loading it?15:43
yoctozeptopriteau: this is for ovs native firewall and ovn to not have this oddity15:44
yoctozeptobut we can revert, it does not hurt15:44
yoctozeptoI can make a better version of it15:44
mnasiadkawell, we can tweak it to at least not enable when neutron_plugin_agent=ovn15:44
yoctozeptoat some point ;p15:44
mnasiadkaok, let's revert for now and post a tweak15:45
yoctozeptoyeah, that makes sense15:45
mnasiadka#agreed to revert https://review.opendev.org/c/openstack/kolla-ansible/+/814949 and post a better version (to skip loading when not required e.g. ovn)15:46
mnasiadka#topic Yoga cycle planning15:46
mgoddardsorry, laptop died15:46
mnasiadkaI have a draft summary mail I'm going to send to openstack-discuss after the PTG and then will populate the Whiteboard with priorities/tasks15:47
mnasiadkaI'm also planning to use Kolla Klub mailing list to get feedback around our single-distro plans for Kolla15:47
hrwcool15:47
mnasiadkaAnything else that needs to be done?15:48
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Revert "Do not load br_netfilter"  https://review.opendev.org/c/openstack/kolla-ansible/+/81494915:48
hrwshould we deprecate CentOS now in Xena to be able to drop it with Yoga or we deprecate in Yoga to drop in Zeus?15:48
mgoddardI think we should wait until yoga15:49
yoctozeptomgoddard, mnasiadka: I improved the error message there15:49
mgoddardstill a lot of uncertainty15:49
yoctozeptosorry, the revert reason15:49
mgoddardthanks yoctozepto 15:49
* yoctozepto tired lol15:49
yoctozeptomgoddard: yw15:49
mnasiadkaWell, we need to make sure we're not going cs915:49
hrwyoctozepto: tired like old lamb?15:49
yoctozeptohrw: yup15:49
mnasiadkaDoes that mean we need to drop centos-binary in Yoga?15:50
hrwmnasiadka: is RDO goes cs9 only for yoga then no binary in yoga for us15:50
hrwdeprecate & drop in one cycle15:50
mnasiadkaOk, so we need to carry out the plan to deprecate all binary, and then if RDO goes cs9 (which they most probably will) in Yoga - we will drop that with an appropriate message?15:51
hrwlooks like15:51
mnasiadkaOk, at least that's clear :)15:51
yoctozeptook, makes sense15:52
mnasiadkaI'll phrase it like this in the summary mail and in the whiteboard items, so it's clear for everybody.15:52
mnasiadkaAnd then we also deprecate CentOS as whole in Yoga and drop it in Zeus, right?15:53
mnasiadkaBut still we need to rework centos-source to not pull in anything from RDO packages, so centos-source on cs8 works...15:53
mgoddardI thought we said A for dropping?15:54
hrwmgoddard: I hoped for A to be Debian/source only iirc15:54
hrwas "Y drops binary, Z drops distros" but that can be one release too short15:55
mnasiadkaI remember we wanted a clean slate from A, but maybe we're just rushing it.15:55
mgoddardI think we need a period where we're deploying the chosen containers on all host distros by default15:56
mgoddardideally have that released and in real world use for a while before dropping15:56
mgoddardof course this is going to explode the test matrix15:57
yoctozeptoonly debian on all15:57
yoctozeptoothers dedicated15:57
yoctozeptothe question was about cs915:57
mgoddardok, double the test matrix15:57
yoctozeptoas we can't rely on rdo then ;d15:58
yoctozeptoyeah, double15:58
mgoddardon the libvirt question, I found a bit more context on the issue I mentioned15:58
yoctozeptoI guess in practice we don't need to verify anything other than qemu/kvm15:58
hrwinfra from xena times would probably fine15:58
yoctozeptomgoddard: which one?15:59
mgoddardwe had problems with centos 8.2 libvirt containers on an 8.1 host15:59
mgoddardqemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe0400617215:59
mgoddardqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs\' failed.15:59
mnasiadkaOk, I think we need to continue the discussion next week (or after the meeting).16:00
mnasiadkaThanks for attending.16:00
mnasiadka#endmeeting16:00
opendevmeetMeeting ended Wed Oct 27 16:00:34 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-10-27-15.00.html16:00
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-10-27-15.00.txt16:00
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-10-27-15.00.log.html16:00
mgoddardI don't think we have much more evidence than that, but there's certainly nervousness here about mixing host & container distros with libvirt in a container16:00
hrwmgoddard: you have qemu 6.0.0 in cs816:00
hrwand 5.2.0 in c8.416:02
opendevreviewMerged openstack/kayobe master: CI: Disable heat in upgrade jobs to save disk space  https://review.opendev.org/c/openstack/kayobe/+/81492016:02
yoctozeptomgoddard: I see, thanks for letting me know16:03
yoctozeptowe do test kvm in CI16:03
yoctozeptoit seems passing now with debian in containers16:03
mgoddardpossibly it would work if the host was newer than the containers16:04
fricklerwe do test kvm? I thought that that was very unstable in most of our providers16:04
yoctozeptomnasiadka: https://review.opendev.org/c/openstack/kolla-ansible/+/814949 +A16:05
yoctozeptofrickler: it tests fine for us16:06
yoctozeptofrickler: https://review.opendev.org/c/openstack/kolla-ansible/+/81461016:06
yoctozeptothis with debian on whatever16:06
opendevreviewPierre Riteau proposed openstack/kayobe stable/xena: CI: Disable heat in upgrade jobs to save disk space  https://review.opendev.org/c/openstack/kayobe/+/81563616:13
frickleryoctozepto: ah, that's with the special nested nodesets, that's fine then16:13
em_I think you guys are doing a fairly unsual job with moving low-level system services like libvirt while at the same time not locking container distro to host distro. IMHO that is doomed16:14
em_*moving low-level services like libvirt into docker images16:15
hrwem_: libvirt runs qemu which talks to kvm part of kernel. none of it is distro related16:16
em_at least it's powerlifting a problem sector that might not need to be looked at (i would find it odd that someone picks OS y for host and then OS x for the images .. if there is a freedome of choice, i think it would be the same choice for both16:16
em_hrw, i understand, but there will be posix layout differences, socket name differences and all that. And i have seen that there is quiet some host mounting done (mostly to /etc/kolla but also to /run) - the latter can be problematic and thus then be extra effort to mount .e.g /run/libvirt/libvirt.sock to /run/libvirt/daemon.sock (or whatever) - so host-to-image mount specific overrides. At least a lot of effort16:19
em_At least you guys seem to have a love for that detail! :)16:21
em_hrw, i would not be shocked that some libvirt versions required at least some specific kernel version++, e.g. going with debian the kernel is oldish. Well anyway, i assume you guys tested this out anyways16:23
hrwem_: please, check facts before playing 'debian == ancient' ok?16:23
em_hrw, dont be rude. I'am using debian for about 20 years now :) And i love it.16:24
hrwubuntu 20.04 uses 5.4, centos stream 8 uses 4.18 (with backports), debian 11 has 5.1016:25
em_And it is a distro which is slow moving, by design. And a like that design.16:25
hrwgood things comes to those who wait ;D16:25
em_i understand the kernel argument - debian changed a lot in this field since i think 4.9 past, they moved a lot fast, esp with 5.x -16:26
hrwmy debian systems run 5.14 from bullseye-backports16:26
em_is there a good reason for that? i run the stable 5.10 ones right now16:27
em_i used backport kernels for stretch and buster (due to docker engine deps), but with bullseye it seems to be nice and smooth16:27
hrwsome of my boxes needs 5.14 to have all components supported16:31
em_Interesting16:32
em_speaking of debian, i'am not able to deploy debian-buster-genericcloud with my ssh keys - it is just not deploying those using (debian-11-genericcloud-amd64-20211011-792)16:32
em_i see that in the logs16:33
em_[   24.013961] cloud-init[620]: ci-info: no authorized SSH keys fingerprints found for user debian.16:33
em_ci-info: no authorized SSH keys fingerprints found for user debian.16:33
* hrw off16:33
opendevreviewMerged openstack/kolla-ansible stable/xena: Drop enable_host_ntp  https://review.opendev.org/c/openstack/kolla-ansible/+/81494216:38
opendevreviewMerged openstack/kolla stable/xena: cinder-volume/ubuntu: add lsscsi and nvme  https://review.opendev.org/c/openstack/kolla/+/81544016:53
opendevreviewUwe Grawert proposed openstack/kolla-ansible master: [Grafana] Add unified alerting and smtp options  https://review.opendev.org/c/openstack/kolla-ansible/+/81569417:14
opendevreviewUwe Grawert proposed openstack/kolla-ansible master: [Grafana] Add unified alerting and smtp options  https://review.opendev.org/c/openstack/kolla-ansible/+/81569417:15
*** amoralej is now known as amoralej|off17:29
opendevreviewMerged openstack/kayobe master: Drop become in stackhpc.libvirt-vm for seed vm provision  https://review.opendev.org/c/openstack/kayobe/+/81268717:37
yoctozeptokevko, mnasiadka, mgoddard: heads-up - osa is going proxysql only the next cycle (in z; for mariadb)17:49
kevkoopenstack ansible  17:49
kevko? 17:49
kevkoyoctozepto: well, we are running proxysql as LB for mariadb  from stein :D .. 17:50
kevkoi mean locally 17:51
kevkoso what does it mean for kolla ? can we be first (finally) ? :D 17:51
mnasiadkayoctozepto: what does it mean only - they are not using haproxy at all from Z? ;-)17:58
mnasiadkayoctozepto: regarding rocky linux - do we really need to test it on rocky linux opendev dib image? I feel it's going to be a pain to convince opendev (since stream is basically newer packages, and we would probably be the only ones wanting that image to be served in nodepool)18:07
yoctozeptomnasiadka: for mariadb18:13
yoctozeptomnasiadka: I don't really need it, but then we need to mark it as not tested upstream, just kind-of-supported18:13
yoctozeptokevko: they already support proxysql I think18:15
yoctozeptoso we can't be first18:15
yoctozeptomnasiadka: we forgot to include custom meeting agenda from the whiteboard18:16
yoctozeptoor, well, missed to get to it :-)18:16
kevkohmm, so will you finally find some time ? :/ 18:17
kevkoi can be online and answer every question :) 18:17
opendevreviewRadosław Piliszek proposed openstack/kolla master: [DNM] test something  https://review.opendev.org/c/openstack/kolla/+/72210218:28
jingvar      {{ ( enable_elasticsearch | bool or ( elasticsearch_address != kolla_internal_vip_address )) and   not enable_monasca | bool }}18:34
jingvarwhat will be there with enabled fqdn ^^^18:35
jingvarkolla-ansible/ansible/group_vars/all.yml:elasticsearch_address: "{{ kolla_internal_fqdn }}" 18:36
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone  https://review.opendev.org/c/openstack/kolla-ansible/+/81557718:50
opendevreviewPierre Riteau proposed openstack/kayobe stable/xena: Drop become in stackhpc.libvirt-vm for seed vm provision  https://review.opendev.org/c/openstack/kayobe/+/81564219:10
opendevreviewMerged openstack/kolla-ansible master: Revert "Do not load br_netfilter"  https://review.opendev.org/c/openstack/kolla-ansible/+/81494919:14
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Fix missing Ansible version in the error message  https://review.opendev.org/c/openstack/kolla-ansible/+/81573519:17
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible stable/xena: Revert "Do not load br_netfilter"  https://review.opendev.org/c/openstack/kolla-ansible/+/81564319:18
opendevreviewRadosław Piliszek proposed openstack/kolla master: CI: Add missed Debian upgrade jobs  https://review.opendev.org/c/openstack/kolla/+/81573719:30
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Use Docker healthchecks for keystone-fernet container  https://review.opendev.org/c/openstack/kolla-ansible/+/80651019:59
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Trivial fix shebang in keystone's fernet-node-sync.sh.j2  https://review.opendev.org/c/openstack/kolla-ansible/+/81381620:01
opendevreviewVerification of a change to openstack/kolla stable/wallaby failed: cinder-volume/ubuntu: add lsscsi and nvme  https://review.opendev.org/c/openstack/kolla/+/81544120:08
opendevreviewMerged openstack/kolla stable/victoria: cinder-volume/ubuntu: add lsscsi and nvme  https://review.opendev.org/c/openstack/kolla/+/81544220:30
opendevreviewMerged openstack/kolla stable/ussuri: cinder-volume/ubuntu: add lsscsi and nvme  https://review.opendev.org/c/openstack/kolla/+/81544320:39
opendevreviewMerged openstack/kolla master: CI: make Debian source jobs voting  https://review.opendev.org/c/openstack/kolla/+/81565720:39
opendevreviewMerged openstack/kolla-ansible master: CI: make Debian source jobs voting  https://review.opendev.org/c/openstack/kolla-ansible/+/81566620:39
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: Debian: enable building Bifrost images  https://review.opendev.org/c/openstack/kolla/+/81478321:12
hrwrebased so it will fail zuul21:13
opendevreviewMerged openstack/kolla-ansible stable/xena: Revert "Do not load br_netfilter"  https://review.opendev.org/c/openstack/kolla-ansible/+/81564322:02
opendevreviewSeena Fallah proposed openstack/kolla-ansible master: mariadb: use add_host to include inactive hosts in shard grouping  https://review.opendev.org/c/openstack/kolla-ansible/+/81427623:23
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone  https://review.opendev.org/c/openstack/kolla-ansible/+/81557723:32

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!