| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 01:24 |
|---|---|---|
| *** amoralej|off is now known as amoralej | 06:25 | |
| Jjay | Hello Team, | 06:55 |
| Jjay | I deployed openstack with Magnum and it works ok until I create a volume on the cluster template.. The cluster fails with the error "cinder" volume driver needs "cloud_provider_enabled" label to be true or unset. | 06:56 |
| Jjay | Can someone assist on how to enable it in Kolla ? | 06:56 |
| em_ | Jjay, you have to set https://github.com/EugenMayer/openstack-lab/blob/master/deploy/3_configure_kolla.sh#L59 | 07:07 |
| em_ | and then pick a storage type, see https://docs.openstack.org/kolla-ansible/latest/reference/storage/cinder-guide.html | 07:07 |
| em_ | and then you will need to define where to put the storage https://github.com/EugenMayer/openstack-lab/blob/master/config/multinode#L25 | 07:08 |
| Jjay | <em> Cinder is already enabled and the backend is ceph which is working OK | 07:08 |
| Jjay | The issue is only when i use Magnum | 07:10 |
| em_ | i have zero experience with magnum (for today, guess it comes on my roadmap tomorrow) | 07:10 |
| em_ | So cannot help you with that | 07:11 |
| Jjay | Sure thanks a lot <em_> | 07:11 |
| em_ | Since i think kolla has no magnum integration, you should rather check the magnum docs, it's all post kolla deployment AFAICS - see https://docs.openstack.org/magnum/latest/user/#storage | 07:13 |
| DK4 | hello, is it possible to deploy an DVR Setup with kolla? | 07:46 |
| jingvar | DK4: yes | 07:52 |
| DK4 | but theres no guide for it as far i can see :) all trial and error i guess? | 07:58 |
| jingvar | https://docs.openstack.org//kolla-ansible/latest/doc-kolla-ansible.pdf | 08:06 |
| jingvar | enable_neutron_provider_networks: yes | 08:07 |
| DK4 | can you even have the SNAT Routing on computes with dvr? | 08:11 |
| em_ | DK4, are you using OVS or OVN? (be cause that differs) | 08:26 |
| DK4 | im using the kolla default (ovs) | 08:29 |
| em_ | In this case, what jingvar said, AFAIR enable_neutron_provider_networks: yes should be enough for OVS dvr | 08:33 |
| em_ | why should you not be able to use SNAT on the compute? Every compute has it's own interface with the public/floating subnet. Packages leaving to the provider network are the SNAT on the compute itself | 08:34 |
| DK4 | thanks for the explanation | 08:42 |
| DK4 | in the inventory of kolla what would be the [network] section then be? do i add my compute hosts there? | 08:42 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Do not fail when admin openrc file does not exist https://review.opendev.org/c/openstack/kayobe/+/815259 | 08:52 |
| opendevreview | Merged openstack/kayobe stable/wallaby: Remove stale config dump from seed hypervisor host configure https://review.opendev.org/c/openstack/kayobe/+/815230 | 08:57 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Drop become in stackhpc.libvirt-vm for seed vm provision https://review.opendev.org/c/openstack/kayobe/+/812687 | 09:00 |
| opendevreview | Merged openstack/kayobe stable/victoria: Remove stale config dump from seed hypervisor host configure https://review.opendev.org/c/openstack/kayobe/+/815231 | 09:20 |
| opendevreview | Merged openstack/kayobe stable/ussuri: Remove stale config dump from seed hypervisor host configure https://review.opendev.org/c/openstack/kayobe/+/815232 | 09:21 |
| opendevreview | Merged openstack/kayobe master: Fix link syntax in release note https://review.opendev.org/c/openstack/kayobe/+/815515 | 09:21 |
| jingvar | DK4: do you use clean kolla-ansible or kayobe | 09:28 |
| DK4 | jingvar: im using clean kolla-ansible currently. but as im in testing still i could look into kayobe if it that makes sense. | 09:32 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/xena: Fix link syntax in release note https://review.opendev.org/c/openstack/kayobe/+/815449 | 09:43 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/wallaby: Set proxy option in early dnf invocation https://review.opendev.org/c/openstack/kayobe/+/814658 | 09:45 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/wallaby: CI: enable DNF tests on CentOS Stream 8 https://review.opendev.org/c/openstack/kayobe/+/814752 | 09:45 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Debian: enable building Bifrost images https://review.opendev.org/c/openstack/kolla/+/814783 | 10:04 |
| hrw | rebased | 10:04 |
| em_ | what is the best way to wipe kolla from a compute (to restart a deployment)? Remove /etc/kolla and wipe all containers? | 10:22 |
| em_ | are there any othere leftovers / secrets? | 10:22 |
| em_ | DK4, for the network to compute question, i'am new so take this with caution. IMHO network decides where to put the neutron-server. In common, this is put only on the controller | 10:26 |
| em_ | DK4, the computes only get the so called neutron-agents (depending on your network type) | 10:26 |
| em_ | Please be sure to take this information with caution. Way to new to all thyis | 10:26 |
| jpic | hi all, if I have 2 AZs with 3 controlers each, must I have 1 mariadb cluster of 6, or 2 mariadb clusters of 3? I mean, must I have 1 cluster for all AZ, or 1 cluster per AZ? thanks | 10:32 |
| hrw | em_: "kolla-ansible destroy" | 10:32 |
| adrian-a | hi guys, any feedback on https://review.opendev.org/c/openstack/kolla-ansible/+/815043 ? | 10:32 |
| jingvar | Have someone an deployment without central logging ? I need to check harpoxe log config and fluentd - for somereasons fluentd try to connnet to ELK | 10:33 |
| hrw | adrian-a: will leave some | 10:34 |
| adrian-a | hrw: thanks | 10:37 |
| hrw | done | 10:40 |
| em_ | hrw, is kolla-ansible-deploy available on the actual compute nodes? looks like a binary the deployer would have. or is this then done remotely via ssh? | 10:45 |
| em_ | hrw, we found out that beside nuking containers/docker and /etc/kolla one needs to reboot due to the /run/ sockets and the libvirt socket check during verify | 10:46 |
| hrw | em_: I do not know where from you deploy. | 10:46 |
| hrw | em_: kolla-ansible is a tool to do all heavy jobs. check, bootstrap, deploy, upgrade, destroy. | 10:47 |
| em_ | hrw, yeah, understood, using it the entire time. My question is, if destroy is nuking the deployer and wiping kolla there, or nuking the nodes (i need the latter) | 10:48 |
| jingvar | koll-ansible works with containers and their config - maybe volumes if wipe | 10:50 |
| hrw | em_: if you need to completely erase nodes then no, k-a does not have that part | 10:51 |
| jingvar | kayobe configures host software, nic's configuration | 10:52 |
| opendevreview | Adrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch https://review.opendev.org/c/openstack/kolla-ansible/+/815043 | 11:04 |
| em_ | when deploying with kolla, non DVR mode OVN, it seems like kolla forces to create a provider interface on the compute instances, even though it is not required at all. Anybody aware of that issue? | 11:22 |
| em_ | So i did not expose the external interface for the compute instances https://github.com/EugenMayer/openstack-lab/blob/stable/ovn/config/group_vars/compute/compute.yml#L3 .. only for the controller (where the gateway service should run on) https://github.com/EugenMayer/openstack-lab/blob/stable/ovn/config/group_vars/control/control.yml#L3 | 11:23 |
| em_ | now, during deployment, kolla tries to createe br-ex on computes ot | 11:24 |
| em_ | to the defaults eth1 (why so ever this interface is used as default) and fails, since it does not exists. Did i get the non DVR mode wrong or is this a kolla glitch? | 11:24 |
| jpic | this infra i'm looking at has different regions with a mix of boths, some regions have 1 cluster per AZ, others have 1 cluster per region with all AZ, and both /seem/ to work them, any idea is welcome | 11:33 |
| em_ | understood, in this mode enable_neutron_provider_networks: no is required | 11:36 |
| opendevreview | Adrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch https://review.opendev.org/c/openstack/kolla-ansible/+/815043 | 11:55 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Do not fail when admin openrc file does not exist https://review.opendev.org/c/openstack/kayobe/+/815259 | 12:15 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: mariadb: Do not use wsrep-notify.sh on Debian https://review.opendev.org/c/openstack/kolla-ansible/+/814668 | 12:32 |
| *** amoralej is now known as amoralej|lunch | 12:39 | |
| opendevreview | Will Szumski proposed openstack/kayobe master: Do not fail when admin openrc file does not exist https://review.opendev.org/c/openstack/kayobe/+/815259 | 12:41 |
| jingvar | why /etc/kolla/fluentd/config.json was generated from kolla-ansible/ansible./roles/common/templates/td-agent.conf.j2 ? but somehow /etc/kolla/fluentd/td-agent.conf from ./roles/common/templates/conf | 12:51 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: CI: make Debian source jobs voting https://review.opendev.org/c/openstack/kolla/+/815657 | 12:59 |
| priteau | jingvar: fluentd config.json is generated from ansible/roles/common/templates/fluentd.json.j2 | 13:00 |
| priteau | jingvar: td-agent.conf is generated from several input, filter, format and output files. | 13:02 |
| priteau | by td-agent.conf.j2 | 13:02 |
| jingvar | [root@ctl01 fluentd]# grep -r /var/log/kolla/fluentd/fluentd.log | 13:06 |
| jingvar | config.json: "command": "/usr/sbin/td-agent -o /var/log/kolla/fluentd/fluentd.log" | 13:06 |
| opendevreview | MargaritaShakhova proposed openstack/kolla-ansible master: Fix broken deploy of placement service https://review.opendev.org/c/openstack/kolla-ansible/+/815524 | 13:08 |
| jingvar | priteau: ohh, I was blind | 13:08 |
| *** amoralej|lunch is now known as amoralej | 13:14 | |
| jingvar | - name: "conf/output/01-es.conf.j2" -> enabled: "{{ log_direct_to_elasticsearch }}" , log_direct_to_elasticsearch: enable_elasticsearch: "{{ 'yes' if enable_central_logging | bool or enable_osprofiler | bool or enable_skydive | bool or enable_monasca | bool else 'no' }}" | 13:14 |
| jingvar | opt/kayobe/src/kayobe-config/etc/kolla/globals.yml:enable_elasticsearch: False | 13:14 |
| jingvar | If I right understand, but I have td-conf with enabled elastic | 13:15 |
| priteau | Do you have any of these variables set? enable_central_logging / enable_osprofiler / enable_skydive / enable_monasca | 13:18 |
| jingvar | opt/kayobe/src/kayobe-config/etc/kolla/globals.yml:enable_central_logging: False , enable_skydive: False , enable_monasca: False, enable_osprofiler: False | 13:18 |
| jingvar | It looks starnge - I don't have anything that could trigger elastic | 13:20 |
| opendevreview | Verification of a change to openstack/kayobe stable/wallaby failed: CI: enable DNF tests on CentOS Stream 8 https://review.opendev.org/c/openstack/kayobe/+/814752 | 13:25 |
| em_ | is there no way to add port_forward via kolla-ansible? It's missing here https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutron-extensions.html | 13:35 |
| em_ | hmm, according to https://blueprints.launchpad.net/kolla-ansible/+spec/enable-l3-port-forwarding it has been implemented | 13:37 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: make Debian/source voting https://review.opendev.org/c/openstack/kolla-ansible/+/815666 | 13:37 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: make Debian/source jobs voting https://review.opendev.org/c/openstack/kolla-ansible/+/815666 | 13:38 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: make Debian source jobs voting https://review.opendev.org/c/openstack/kolla-ansible/+/815666 | 13:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: docs: weekly meetings page https://review.opendev.org/c/openstack/kolla/+/815494 | 13:43 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: Add support for deploying prometheus-msteams https://review.opendev.org/c/openstack/kolla-ansible/+/812679 | 13:44 |
| em_ | Anybody has port-forwarding up and running while deploying via kolla-ansible? I guess it might differ w ro w/o DVR, but i do not understand how to enable it on the agent | 13:45 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: Build overcloud host image directly with DIB https://review.opendev.org/c/openstack/kayobe/+/772609 | 13:49 |
| em_ | ok we found out about enable_neutron_port_forwarding: 'yes' - seems to be missing in the docs (or we cannot find it f | 13:50 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: CI: make Debian source jobs voting https://review.opendev.org/c/openstack/kolla/+/815657 | 14:00 |
| opendevreview | Adrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch https://review.opendev.org/c/openstack/kolla-ansible/+/815043 | 14:01 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: make Debian source jobs voting https://review.opendev.org/c/openstack/kolla-ansible/+/815666 | 14:02 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Fix wrong opts in cyborg.config https://review.opendev.org/c/openstack/kolla-ansible/+/815672 | 14:13 |
| opendevreview | Adrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch https://review.opendev.org/c/openstack/kolla-ansible/+/815043 | 14:20 |
| opendevreview | Adrian Andreias proposed openstack/kolla-ansible master: docs: Parameterize kolla-ansible version and branch https://review.opendev.org/c/openstack/kolla-ansible/+/815043 | 14:38 |
| mnasiadka | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak parallax Fl1nt | 14:51 |
| mnasiadka | meeting in 9 | 14:51 |
| mnasiadka | #startmeeting kolla | 15:00 |
| opendevmeet | Meeting started Wed Oct 27 15:00:11 2021 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 15:00 |
| frickler | mnasiadka: where do you keep that ping list? I'd like to add myself | 15:00 |
| mnasiadka | #topic rollcall | 15:00 |
| priteau | o/ | 15:00 |
| mgoddard | \o | 15:00 |
| mnasiadka | frickler: it's currently at https://wiki.openstack.org/wiki/Meetings/Kolla | 15:00 |
| mnasiadka | o/ | 15:00 |
| frickler | mnasiadka: thx | 15:00 |
| hrw | [°][_] | 15:00 |
| yoctozepto | o/ | 15:02 |
| mnasiadka | #topic agenda | 15:03 |
| em_ | My management network (provider based vswitch), which is used to also tunnel the (non dvr) ovn network, has an mtu of 1400. When adding the provider network, i used a mtu of 1400 - but that seems not to be enough. Do i need to set something with kolla in general? I have found https://docs.openstack.org/neutron/queens/admin/config-mtu.html but not sure that applies here | 15:03 |
| yoctozepto | mnasiadka: need to move it to the whiteboard (the official ping list) | 15:03 |
| mnasiadka | em_: we have a weekly meeting now - please wait until it ends (around 1hr) | 15:03 |
| mnasiadka | yoctozepto: yup | 15:03 |
| em_ | (oh sorry, was not aware about the irc based meeting, will shut up. Excuse me) | 15:03 |
| mnasiadka | * Roll-call | 15:03 |
| mnasiadka | * Agenda | 15:03 |
| mnasiadka | * Announcements | 15:03 |
| mnasiadka | * Review action items from the last meeting | 15:03 |
| mnasiadka | * CI status | 15:03 |
| mnasiadka | * Release tasks | 15:03 |
| mnasiadka | * Yoga cycle planning | 15:03 |
| mnasiadka | * Open discussion | 15:03 |
| mnasiadka | #topic Announcements | 15:04 |
| mnasiadka | I have none - anyone anything? | 15:04 |
| mgoddard | RC!? | 15:04 |
| mgoddard | RC1? | 15:04 |
| mgoddard | PTG? | 15:04 |
| yoctozepto | RC!!!!!1111oneoneoneeleven | 15:04 |
| opendevreview | wu.chunyang proposed openstack/kolla-ansible master: Fix wrong opts in cyborg.conf https://review.opendev.org/c/openstack/kolla-ansible/+/815672 | 15:04 |
| mnasiadka | Ah, right - RC1 for Kolla, Kolla-Ansible and Kayobe has been cut. | 15:05 |
| mgoddard | #info | 15:05 |
| mnasiadka | #info RC1 for Kolla, Kolla-Ansible and Kayobe has been cut. | 15:05 |
| mnasiadka | Ok then, let's move on I guess - unless somebody else wants to announce anything? | 15:06 |
| mnasiadka | #topic Review action items from the last meeting | 15:07 |
| mnasiadka | Seems there were none. | 15:07 |
| mnasiadka | #topic CI Status | 15:07 |
| mnasiadka | Are we green? | 15:07 |
| mnasiadka | Seems we are - based on the whiteboard. | 15:08 |
| mnasiadka | #topic Release tasks | 15:09 |
| mnasiadka | So, do we have a list of blockers for doing RC2? | 15:09 |
| mnasiadka | I think all MariaDB related patches have been merged? | 15:09 |
| mnasiadka | yoctozepto: ? | 15:10 |
| yoctozepto | yeah, I think so | 15:11 |
| yoctozepto | any release tasks still to do? | 15:11 |
| yoctozepto | centos-openstack-release done? | 15:11 |
| mgoddard | the gerrit dashboards seem a bit broken | 15:12 |
| mgoddard | no project filtering | 15:12 |
| yoctozepto | mayhaps we need also https://review.opendev.org/c/openstack/kolla-ansible/+/814276 | 15:12 |
| yoctozepto | for mariadb | 15:12 |
| frickler | mgoddard: for dashboards you need to remove the /#/ from the path | 15:13 |
| opendevreview | Uwe Grawert proposed openstack/kolla-ansible master: [Grafana] Add unified alerting and smtp options https://review.opendev.org/c/openstack/kolla-ansible/+/815694 | 15:14 |
| frickler | or reload | 15:14 |
| frickler | see https://gerrit-review.googlesource.com/c/gerrit/+/321535 | 15:14 |
| mgoddard | frickler: thanks | 15:14 |
| hrw | yoctozepto: c-r-o-xena exists | 15:14 |
| mnasiadka | and we use it | 15:15 |
| yoctozepto | ok | 15:15 |
| yoctozepto | so only that mariadb patch | 15:15 |
| yoctozepto | but I'm not sure what the impact is | 15:15 |
| yoctozepto | perhaps it garbles the config | 15:15 |
| mnasiadka | around gerrit dashboards - I see master branch in stable branch backports section of Kolla dashboard, so maybe we need to revisit those and check what's going on | 15:15 |
| priteau | About CI status, we are amber on wallaby for Kayobe (just updated the booard) | 15:16 |
| priteau | It's caused by frequent disk full issues | 15:16 |
| priteau | The wallaby images must be a bit bigger than other releases | 15:17 |
| priteau | We have a proposed workaround which is to disable heat from CI upgrade jobs | 15:17 |
| mnasiadka | Ok, I think one of the changes to make it better for Wallaby is merging/merged today | 15:18 |
| mnasiadka | yoctozepto: I don't see a bug report in that patch, so can't really tell we should wait to get it reviewed and merged. | 15:18 |
| mnasiadka | So, should we post RC2 for Kolla/Kolla-Ansible/Kayobe? | 15:18 |
| mgoddard | https://review.opendev.org/c/openstack/kolla-ansible/+/814942 | 15:20 |
| yoctozepto | mnasiadka: it's in the reno | 15:20 |
| yoctozepto | mgoddard: good catch | 15:20 |
| mnasiadka | yoctozepto: what about closes-bug? | 15:21 |
| yoctozepto | mnasiadka: yeah, you can comment that on it | 15:21 |
| yoctozepto | but the bug report is not satisfactory tbh | 15:21 |
| yoctozepto | it's like writing "it doesn't work." | 15:21 |
| hrw | mnasiadka: let https://review.opendev.org/c/openstack/kolla/+/815440 merge and then RC2? | 15:22 |
| mnasiadka | ok, so two changes are +w and we need to wait for them to merge | 15:23 |
| mgoddard | if we're going to merge this revert then let's do it before release https://review.opendev.org/c/openstack/kolla-ansible/+/814949 | 15:23 |
| mgoddard | (discuss) | 15:23 |
| opendevreview | Merged openstack/kayobe stable/wallaby: Set proxy option in early dnf invocation https://review.opendev.org/c/openstack/kayobe/+/814658 | 15:24 |
| mnasiadka | mgoddard: I think you owe us some more description and reason ;-) | 15:24 |
| mgoddard | I would say the same about the original patch :) | 15:25 |
| mnasiadka | I added enable_host_ntp and cinder-volume fix as RC2 blockers in the whiteboard (L297) | 15:25 |
| mnasiadka | Well, the original patch stated everything works, and CI didn't explode - so I'd like to know what does it break ;-) | 15:25 |
| yoctozepto | mgoddard: like mnasiadka said - it was well described :-) | 15:26 |
| priteau | If we are issuing RC2 for Kayobe we may want to merge https://review.opendev.org/c/openstack/kayobe/+/812687 in xena | 15:27 |
| yoctozepto | I don't mind reverting if it really breaks something; but then again we should probably rewrite it to work differently as it does not make sense to use it with ovs native firewall and ovn | 15:27 |
| mnasiadka | priteau: especially that the comment says "since Xena"... ;-) | 15:28 |
| mgoddard | I don't see where neutron is loading that module | 15:28 |
| mgoddard | I do see neutron will print a warning if it is not loaded | 15:29 |
| priteau | I think br_netfilter can be loaded by docker | 15:30 |
| mgoddard | if it uses iptables | 15:31 |
| mnasiadka | we default to disable iptables now I think | 15:31 |
| mgoddard | I can try to justify, but not in real time | 15:31 |
| yoctozepto | mgoddard: did it break somewhere IRL? | 15:31 |
| mgoddard | not yet | 15:31 |
| mnasiadka | but that means we could have non-working SGs? | 15:32 |
| yoctozepto | then try to make it break :-) | 15:32 |
| yoctozepto | I tried and it works | 15:32 |
| yoctozepto | hence did not bother to improve, just removed | 15:32 |
| mgoddard | it jumped out as one of those patches that could bite us | 15:32 |
| yoctozepto | well, at least we know how to fix it quickly | 15:33 |
| mgoddard | and my gut has often been right on those in the past but I let them slide then suffer latet | 15:33 |
| yoctozepto | and we fix when it bites | 15:33 |
| mnasiadka | So why change a tradition? | 15:33 |
| mgoddard | anyway | 15:33 |
| kevko | guys , is this visible in CI ? | 15:33 |
| kevko | 2021-10-27 15:11:40.879 25 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: No module named 'ovn_octavia_provider.common': ModuleNotFoundError: No module named 'ovn_octavia_provider.common' | 15:33 |
| yoctozepto | oh well, that's it for the scientific method :D | 15:33 |
| yoctozepto | kevko: guy, we are in a meeting | 15:33 |
| yoctozepto | :-) | 15:33 |
| kevko | oh, sorry :D | 15:33 |
| priteau | I just have a freshly deployed kolla using xena branches, br_netfilter is loaded on compute hosts | 15:34 |
| priteau | I can try and see what enabled it | 15:34 |
| mgoddard | do you know how/when it got loaded? | 15:34 |
| priteau | I don't know yet | 15:35 |
| priteau | [Wed Oct 27 14:11:49 2021] Bridge firewalling registered | 15:35 |
| yoctozepto | now we know when | 15:36 |
| yoctozepto | your mileage may vary though | 15:36 |
| priteau | But in neutron-openvswitch-agent logs: | 15:36 |
| priteau | 2021-10-27 14:08:13.127 7 WARNING neutron.agent.linux.iptables_firewall [req-60c2a81f-edaa-4c34-a2ae-37017aeac72f - - - - -] Kernel module br_netfilter is not loaded. | 15:36 |
| priteau | 2021-10-27 14:08:13.128 7 WARNING neutron.agent.linux.iptables_firewall [req-60c2a81f-edaa-4c34-a2ae-37017aeac72f - - - - -] Please ensure that netfilter options for bridge are enabled to provide working security groups. | 15:36 |
| mnasiadka | well, so after the warning something did it ;-) | 15:37 |
| hrw | this should be done by something on host before containers start | 15:37 |
| hrw | otherwise we would need to have hostos == containeros | 15:38 |
| mgoddard | Systems that don't override default settings for those knobs would work | 15:38 |
| mgoddard | fine except for this exception in the log file and agent resync. This is | 15:38 |
| mgoddard | because the first attempt to add a iptables rule using 'physdev' module | 15:38 |
| mgoddard | (-m physdev) will trigger the kernel module loading. In theory, we could | 15:38 |
| mgoddard | silently swallow missing knobs, and still operate correctly. But on | 15:38 |
| mgoddard | second thought, it's quite fragile to rely on that implicit module | 15:38 |
| mgoddard | loading. In the case where we can't detect whether firewall is enabled, | 15:38 |
| mgoddard | it's better to fail than hope for the best. | 15:38 |
| mgoddard | neutron commit e83a44b96a8e3cd81b7cc684ac90486b283a3507 | 15:38 |
| mgoddard | which I linked to in the br_netfilter patch 2 weeks ago | 15:39 |
| priteau | It was loaded when I launched a VM | 15:39 |
| priteau | created | 2021-10-27T14:11:45Z | 15:40 |
| priteau | Loaded 4 seconds late | 15:40 |
| priteau | later | 15:40 |
| mgoddard | we should move on | 15:40 |
| mnasiadka | Yup, what's the plan? Leave it as is since it seems to work? | 15:41 |
| priteau | It works but it produces WARNING messages in logs, that's not nice | 15:42 |
| priteau | What do we gain from not loading it? | 15:43 |
| yoctozepto | priteau: this is for ovs native firewall and ovn to not have this oddity | 15:44 |
| yoctozepto | but we can revert, it does not hurt | 15:44 |
| yoctozepto | I can make a better version of it | 15:44 |
| mnasiadka | well, we can tweak it to at least not enable when neutron_plugin_agent=ovn | 15:44 |
| yoctozepto | at some point ;p | 15:44 |
| mnasiadka | ok, let's revert for now and post a tweak | 15:45 |
| yoctozepto | yeah, that makes sense | 15:45 |
| mnasiadka | #agreed to revert https://review.opendev.org/c/openstack/kolla-ansible/+/814949 and post a better version (to skip loading when not required e.g. ovn) | 15:46 |
| mnasiadka | #topic Yoga cycle planning | 15:46 |
| mgoddard | sorry, laptop died | 15:46 |
| mnasiadka | I have a draft summary mail I'm going to send to openstack-discuss after the PTG and then will populate the Whiteboard with priorities/tasks | 15:47 |
| mnasiadka | I'm also planning to use Kolla Klub mailing list to get feedback around our single-distro plans for Kolla | 15:47 |
| hrw | cool | 15:47 |
| mnasiadka | Anything else that needs to be done? | 15:48 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Revert "Do not load br_netfilter" https://review.opendev.org/c/openstack/kolla-ansible/+/814949 | 15:48 |
| hrw | should we deprecate CentOS now in Xena to be able to drop it with Yoga or we deprecate in Yoga to drop in Zeus? | 15:48 |
| mgoddard | I think we should wait until yoga | 15:49 |
| yoctozepto | mgoddard, mnasiadka: I improved the error message there | 15:49 |
| mgoddard | still a lot of uncertainty | 15:49 |
| yoctozepto | sorry, the revert reason | 15:49 |
| mgoddard | thanks yoctozepto | 15:49 |
| * yoctozepto tired lol | 15:49 | |
| yoctozepto | mgoddard: yw | 15:49 |
| mnasiadka | Well, we need to make sure we're not going cs9 | 15:49 |
| hrw | yoctozepto: tired like old lamb? | 15:49 |
| yoctozepto | hrw: yup | 15:49 |
| mnasiadka | Does that mean we need to drop centos-binary in Yoga? | 15:50 |
| hrw | mnasiadka: is RDO goes cs9 only for yoga then no binary in yoga for us | 15:50 |
| hrw | deprecate & drop in one cycle | 15:50 |
| mnasiadka | Ok, so we need to carry out the plan to deprecate all binary, and then if RDO goes cs9 (which they most probably will) in Yoga - we will drop that with an appropriate message? | 15:51 |
| hrw | looks like | 15:51 |
| mnasiadka | Ok, at least that's clear :) | 15:51 |
| yoctozepto | ok, makes sense | 15:52 |
| mnasiadka | I'll phrase it like this in the summary mail and in the whiteboard items, so it's clear for everybody. | 15:52 |
| mnasiadka | And then we also deprecate CentOS as whole in Yoga and drop it in Zeus, right? | 15:53 |
| mnasiadka | But still we need to rework centos-source to not pull in anything from RDO packages, so centos-source on cs8 works... | 15:53 |
| mgoddard | I thought we said A for dropping? | 15:54 |
| hrw | mgoddard: I hoped for A to be Debian/source only iirc | 15:54 |
| hrw | as "Y drops binary, Z drops distros" but that can be one release too short | 15:55 |
| mnasiadka | I remember we wanted a clean slate from A, but maybe we're just rushing it. | 15:55 |
| mgoddard | I think we need a period where we're deploying the chosen containers on all host distros by default | 15:56 |
| mgoddard | ideally have that released and in real world use for a while before dropping | 15:56 |
| mgoddard | of course this is going to explode the test matrix | 15:57 |
| yoctozepto | only debian on all | 15:57 |
| yoctozepto | others dedicated | 15:57 |
| yoctozepto | the question was about cs9 | 15:57 |
| mgoddard | ok, double the test matrix | 15:57 |
| yoctozepto | as we can't rely on rdo then ;d | 15:58 |
| yoctozepto | yeah, double | 15:58 |
| mgoddard | on the libvirt question, I found a bit more context on the issue I mentioned | 15:58 |
| yoctozepto | I guess in practice we don't need to verify anything other than qemu/kvm | 15:58 |
| hrw | infra from xena times would probably fine | 15:58 |
| yoctozepto | mgoddard: which one? | 15:59 |
| mgoddard | we had problems with centos 8.2 libvirt containers on an 8.1 host | 15:59 |
| mgoddard | qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172 | 15:59 |
| mgoddard | qemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs\' failed. | 15:59 |
| mnasiadka | Ok, I think we need to continue the discussion next week (or after the meeting). | 16:00 |
| mnasiadka | Thanks for attending. | 16:00 |
| mnasiadka | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Wed Oct 27 16:00:34 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-10-27-15.00.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-10-27-15.00.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2021/kolla.2021-10-27-15.00.log.html | 16:00 |
| mgoddard | I don't think we have much more evidence than that, but there's certainly nervousness here about mixing host & container distros with libvirt in a container | 16:00 |
| hrw | mgoddard: you have qemu 6.0.0 in cs8 | 16:00 |
| hrw | and 5.2.0 in c8.4 | 16:02 |
| opendevreview | Merged openstack/kayobe master: CI: Disable heat in upgrade jobs to save disk space https://review.opendev.org/c/openstack/kayobe/+/814920 | 16:02 |
| yoctozepto | mgoddard: I see, thanks for letting me know | 16:03 |
| yoctozepto | we do test kvm in CI | 16:03 |
| yoctozepto | it seems passing now with debian in containers | 16:03 |
| mgoddard | possibly it would work if the host was newer than the containers | 16:04 |
| frickler | we do test kvm? I thought that that was very unstable in most of our providers | 16:04 |
| yoctozepto | mnasiadka: https://review.opendev.org/c/openstack/kolla-ansible/+/814949 +A | 16:05 |
| yoctozepto | frickler: it tests fine for us | 16:06 |
| yoctozepto | frickler: https://review.opendev.org/c/openstack/kolla-ansible/+/814610 | 16:06 |
| yoctozepto | this with debian on whatever | 16:06 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/xena: CI: Disable heat in upgrade jobs to save disk space https://review.opendev.org/c/openstack/kayobe/+/815636 | 16:13 |
| frickler | yoctozepto: ah, that's with the special nested nodesets, that's fine then | 16:13 |
| em_ | I think you guys are doing a fairly unsual job with moving low-level system services like libvirt while at the same time not locking container distro to host distro. IMHO that is doomed | 16:14 |
| em_ | *moving low-level services like libvirt into docker images | 16:15 |
| hrw | em_: libvirt runs qemu which talks to kvm part of kernel. none of it is distro related | 16:16 |
| em_ | at least it's powerlifting a problem sector that might not need to be looked at (i would find it odd that someone picks OS y for host and then OS x for the images .. if there is a freedome of choice, i think it would be the same choice for both | 16:16 |
| em_ | hrw, i understand, but there will be posix layout differences, socket name differences and all that. And i have seen that there is quiet some host mounting done (mostly to /etc/kolla but also to /run) - the latter can be problematic and thus then be extra effort to mount .e.g /run/libvirt/libvirt.sock to /run/libvirt/daemon.sock (or whatever) - so host-to-image mount specific overrides. At least a lot of effort | 16:19 |
| em_ | At least you guys seem to have a love for that detail! :) | 16:21 |
| em_ | hrw, i would not be shocked that some libvirt versions required at least some specific kernel version++, e.g. going with debian the kernel is oldish. Well anyway, i assume you guys tested this out anyways | 16:23 |
| hrw | em_: please, check facts before playing 'debian == ancient' ok? | 16:23 |
| em_ | hrw, dont be rude. I'am using debian for about 20 years now :) And i love it. | 16:24 |
| hrw | ubuntu 20.04 uses 5.4, centos stream 8 uses 4.18 (with backports), debian 11 has 5.10 | 16:25 |
| em_ | And it is a distro which is slow moving, by design. And a like that design. | 16:25 |
| hrw | good things comes to those who wait ;D | 16:25 |
| em_ | i understand the kernel argument - debian changed a lot in this field since i think 4.9 past, they moved a lot fast, esp with 5.x - | 16:26 |
| hrw | my debian systems run 5.14 from bullseye-backports | 16:26 |
| em_ | is there a good reason for that? i run the stable 5.10 ones right now | 16:27 |
| em_ | i used backport kernels for stretch and buster (due to docker engine deps), but with bullseye it seems to be nice and smooth | 16:27 |
| hrw | some of my boxes needs 5.14 to have all components supported | 16:31 |
| em_ | Interesting | 16:32 |
| em_ | speaking of debian, i'am not able to deploy debian-buster-genericcloud with my ssh keys - it is just not deploying those using (debian-11-genericcloud-amd64-20211011-792) | 16:32 |
| em_ | i see that in the logs | 16:33 |
| em_ | [ 24.013961] cloud-init[620]: ci-info: no authorized SSH keys fingerprints found for user debian. | 16:33 |
| em_ | ci-info: no authorized SSH keys fingerprints found for user debian. | 16:33 |
| * hrw off | 16:33 | |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Drop enable_host_ntp https://review.opendev.org/c/openstack/kolla-ansible/+/814942 | 16:38 |
| opendevreview | Merged openstack/kolla stable/xena: cinder-volume/ubuntu: add lsscsi and nvme https://review.opendev.org/c/openstack/kolla/+/815440 | 16:53 |
| opendevreview | Uwe Grawert proposed openstack/kolla-ansible master: [Grafana] Add unified alerting and smtp options https://review.opendev.org/c/openstack/kolla-ansible/+/815694 | 17:14 |
| opendevreview | Uwe Grawert proposed openstack/kolla-ansible master: [Grafana] Add unified alerting and smtp options https://review.opendev.org/c/openstack/kolla-ansible/+/815694 | 17:15 |
| *** amoralej is now known as amoralej|off | 17:29 | |
| opendevreview | Merged openstack/kayobe master: Drop become in stackhpc.libvirt-vm for seed vm provision https://review.opendev.org/c/openstack/kayobe/+/812687 | 17:37 |
| yoctozepto | kevko, mnasiadka, mgoddard: heads-up - osa is going proxysql only the next cycle (in z; for mariadb) | 17:49 |
| kevko | openstack ansible | 17:49 |
| kevko | ? | 17:49 |
| kevko | yoctozepto: well, we are running proxysql as LB for mariadb from stein :D .. | 17:50 |
| kevko | i mean locally | 17:51 |
| kevko | so what does it mean for kolla ? can we be first (finally) ? :D | 17:51 |
| mnasiadka | yoctozepto: what does it mean only - they are not using haproxy at all from Z? ;-) | 17:58 |
| mnasiadka | yoctozepto: regarding rocky linux - do we really need to test it on rocky linux opendev dib image? I feel it's going to be a pain to convince opendev (since stream is basically newer packages, and we would probably be the only ones wanting that image to be served in nodepool) | 18:07 |
| yoctozepto | mnasiadka: for mariadb | 18:13 |
| yoctozepto | mnasiadka: I don't really need it, but then we need to mark it as not tested upstream, just kind-of-supported | 18:13 |
| yoctozepto | kevko: they already support proxysql I think | 18:15 |
| yoctozepto | so we can't be first | 18:15 |
| yoctozepto | mnasiadka: we forgot to include custom meeting agenda from the whiteboard | 18:16 |
| yoctozepto | or, well, missed to get to it :-) | 18:16 |
| kevko | hmm, so will you finally find some time ? :/ | 18:17 |
| kevko | i can be online and answer every question :) | 18:17 |
| opendevreview | Radosław Piliszek proposed openstack/kolla master: [DNM] test something https://review.opendev.org/c/openstack/kolla/+/722102 | 18:28 |
| jingvar | {{ ( enable_elasticsearch | bool or ( elasticsearch_address != kolla_internal_vip_address )) and not enable_monasca | bool }} | 18:34 |
| jingvar | what will be there with enabled fqdn ^^^ | 18:35 |
| jingvar | kolla-ansible/ansible/group_vars/all.yml:elasticsearch_address: "{{ kolla_internal_fqdn }}" | 18:36 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 18:50 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/xena: Drop become in stackhpc.libvirt-vm for seed vm provision https://review.opendev.org/c/openstack/kayobe/+/815642 | 19:10 |
| opendevreview | Merged openstack/kolla-ansible master: Revert "Do not load br_netfilter" https://review.opendev.org/c/openstack/kolla-ansible/+/814949 | 19:14 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Fix missing Ansible version in the error message https://review.opendev.org/c/openstack/kolla-ansible/+/815735 | 19:17 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible stable/xena: Revert "Do not load br_netfilter" https://review.opendev.org/c/openstack/kolla-ansible/+/815643 | 19:18 |
| opendevreview | Radosław Piliszek proposed openstack/kolla master: CI: Add missed Debian upgrade jobs https://review.opendev.org/c/openstack/kolla/+/815737 | 19:30 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Use Docker healthchecks for keystone-fernet container https://review.opendev.org/c/openstack/kolla-ansible/+/806510 | 19:59 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Trivial fix shebang in keystone's fernet-node-sync.sh.j2 https://review.opendev.org/c/openstack/kolla-ansible/+/813816 | 20:01 |
| opendevreview | Verification of a change to openstack/kolla stable/wallaby failed: cinder-volume/ubuntu: add lsscsi and nvme https://review.opendev.org/c/openstack/kolla/+/815441 | 20:08 |
| opendevreview | Merged openstack/kolla stable/victoria: cinder-volume/ubuntu: add lsscsi and nvme https://review.opendev.org/c/openstack/kolla/+/815442 | 20:30 |
| opendevreview | Merged openstack/kolla stable/ussuri: cinder-volume/ubuntu: add lsscsi and nvme https://review.opendev.org/c/openstack/kolla/+/815443 | 20:39 |
| opendevreview | Merged openstack/kolla master: CI: make Debian source jobs voting https://review.opendev.org/c/openstack/kolla/+/815657 | 20:39 |
| opendevreview | Merged openstack/kolla-ansible master: CI: make Debian source jobs voting https://review.opendev.org/c/openstack/kolla-ansible/+/815666 | 20:39 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Debian: enable building Bifrost images https://review.opendev.org/c/openstack/kolla/+/814783 | 21:12 |
| hrw | rebased so it will fail zuul | 21:13 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Revert "Do not load br_netfilter" https://review.opendev.org/c/openstack/kolla-ansible/+/815643 | 22:02 |
| opendevreview | Seena Fallah proposed openstack/kolla-ansible master: mariadb: use add_host to include inactive hosts in shard grouping https://review.opendev.org/c/openstack/kolla-ansible/+/814276 | 23:23 |
| opendevreview | James Kirsch proposed openstack/kolla-ansible master: Use system scoped tokens with Keystone https://review.opendev.org/c/openstack/kolla-ansible/+/815577 | 23:32 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!