| mnasiadka | morning | 06:19 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: systemd: handle running container without systemd unit https://review.opendev.org/c/openstack/kolla-ansible/+/890198 | 06:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Test with docker_restart_policy default https://review.opendev.org/c/openstack/kolla-ansible/+/890658 | 06:21 |
| mnasiadka | frickler, bbezak, kevko: would love to get reviews on the systemd patch ^^ - that solves kayobe upgrades in CI, so it should solve them for users | 06:22 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: rabbitmq: bump version to 3.12 https://review.opendev.org/c/openstack/kolla/+/887225 | 06:55 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873913 | 07:04 |
| kevko | mnasiadka morning , i will check it ... and what about LE ? :P | 07:14 |
| mnasiadka | will have a look, have been off for the last two days, today is monday in Poland ;) | 07:14 |
| kevko | mnasiadka check also the kolla part SvenKieske left some comments about if conditional for ownership of some user or check for some mode and then set both ... | 07:15 |
| kevko | he had argumentation that there is no need to check ... on the first side it's true ..but on the other side ..it's extend start ..so if there are ifs ... most of the cases it's set on the first start ..and never again .. | 07:17 |
| mnasiadka | kevko: frankly speaking that amount of bash is going to be fun to review | 07:25 |
| kevko | it's not that lot :D | 07:27 |
| hrw | elo | 07:29 |
| hrw | which patch you are talking about? | 07:29 |
| kevko | \o | 07:29 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/887347 | 07:29 |
| mnasiadka | +555 -50 - not that lot? | 07:29 |
| mnasiadka | come on | 07:30 |
| kevko | well, yeah, but most of that it's checks and lot of are just echos to log .. | 07:30 |
| kevko | aaand it's divided into functions .. | 07:30 |
| kevko | and 200 lines are parsers for example | 07:31 |
| kevko | 30 lines is call lego command on multiline | 07:32 |
| kevko | if you quickly check by an eye ...it's not that bad :) | 07:32 |
| kevko | hrw feel free to check also :) | 07:33 |
| hrw | left some comments | 07:37 |
| hrw | I dislike those shell scripts | 07:37 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: systemd: handle running container without systemd unit https://review.opendev.org/c/openstack/kolla-ansible/+/890198 | 07:38 |
| kevko | hrw: well, that scripts are importing cert into haproxy, backup old and copy new ... i thought that some python is overkill | 07:40 |
| kevko | and original was also bash ... | 07:40 |
| kevko | but i tried to write it bulletproof | 07:41 |
| hrw | kevko: read my comments | 07:41 |
| kevko | i've already read ...thanks ..good point | 07:41 |
| hrw | if something is done 2+ times then it should be a function | 07:44 |
| kevko | agree | 07:44 |
| hrw | if something is done nearly the same then also worth refactoring | 07:44 |
| hrw | if something then lego_funtion="renew" else lego_funtion="newcert" fi ; lego --things $lego_function | 07:45 |
| hrw | etc | 07:45 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Test with docker_restart_policy default https://review.opendev.org/c/openstack/kolla-ansible/+/890658 | 07:55 |
| mnasiadka | ok, reworking that so it's more readable is nice - but we have a way to test what happens when ssh rsync fails? are we retrying it? are we just failing? | 07:55 |
| kevko | mnasiadka: fail and log ... it's cron ..in configurable time it wil run again (default 12 hours) .. | 07:56 |
| mnasiadka | kevko: thanks for reminding about fluentd logs, commented on the k-a patch :) | 08:01 |
| yusufgungor | Hi everyone, we are using zed version openstack with oslo.messaging version is 14.0.1 for all services. | 08:18 |
| yusufgungor | Our rabbitmq installed and configured via kolla-ansible. | 08:18 |
| yusufgungor | "amqp_durable_queues = true" is exist for all services under oslo_messaging_rabbit sections. | 08:18 |
| yusufgungor | Also there exist a ha-all policy with "ha-mode: all", "ha-promote-on-shutdown: always" and pattern ^(?!(amq\.)|(.*_fanout_)|(reply_)).* | 08:18 |
| yusufgungor | After rebooting the all controller nodes one by one, some reply queues is not recreated. So the not HA queues like reply is not recreated. | 08:18 |
| yusufgungor | Do you have any idea to fix? | 08:18 |
| yusufgungor | Logs like below: | 08:18 |
| yusufgungor | /var/log/kolla/nova/nova-conductor.log:2023-08-16 10:52:13.970 27 WARNING oslo_messaging._drivers.amqpdriver [None req-bda2bd6c-5c18-48ce-924c-6d9af2fb28b6 - - - - - -] reply_60d6e5b31df946a391621751856800ce doesn't exist, drop reply to 29951416f27546cdbbc70146e619439b: oslo_messaging.exceptions.MessageUndeliverable | 08:18 |
| yusufgungor | /var/log/kolla/nova/nova-conductor.log:2023-08-16 10:52:13.971 27 ERROR oslo_messaging._drivers.amqpdriver [None req-bda2bd6c-5c18-48ce-924c-6d9af2fb28b6 - - - - - -] The reply 29951416f27546cdbbc70146e619439b failed to send after 60 seconds due to a missing queue (reply_60d6e5b31df946a391621751856800ce). Abandoning...: oslo_messaging.exceptions.MessageUndeliverable | 08:18 |
| yusufgungor | /var/log/kolla/neutron/neutron-server.log:2023-08-15 17:36:24.333 40 ERROR oslo_messaging._drivers.amqpdriver [None req-0b7ed5e1-9488-4095-8d92-c9aeb333f011 - - - - - -] The reply d79b3d1485574cf9b0f418f0e920421c failed to send after 60 seconds due to a missing queue (reply_c8d743b4e7184a7ca0be09c43ed03608). Abandoning...: oslo_messaging.exceptions.MessageUndeliverable | 08:18 |
| yusufgungor | /var/log/kolla/neutron/neutron-server.log:2023-08-15 17:36:47.533 54 WARNING oslo_messaging._drivers.amqpdriver [None req-be715e9b-7139-4e92-9e78-a103ec828d9d - - - - - -] reply_5c0c508ae6a44d6c859b21b9eec8a828 doesn't exist, drop reply to 14f52a5591a24f0890b0c956c799b750: oslo_messaging.exceptions.MessageUndeliverable | 08:18 |
| mnasiadka | mattcrees: ^^ - any idea? | 08:20 |
| mnasiadka | maybe some oslo.messaging bug? | 08:20 |
| kevko | ha-mode all ? it should be removed no ? | 08:22 |
| mattcrees | I've not seens queues fail to recreate before. yusufgungor: is there anything helpful in the rabbitmq logs? | 08:23 |
| mnasiadka | ha-all policy? in zed? huh | 08:24 |
| mattcrees | That'll be there if rabbitmq_ha_replica_count is unset. https://github.com/openstack/kolla-ansible/blob/9f8c454484f1656443a43b0df4e9f287cd41df0d/ansible/roles/rabbitmq/templates/definitions.json.j2#L21 | 08:25 |
| mnasiadka | but that's not default | 08:27 |
| mattcrees | True it's not | 08:27 |
| kevko | yusufgungor: share your globals without sensitive data | 08:28 |
| kevko | and potentionally some overrides | 08:28 |
| kevko | *config overrides | 08:28 |
| mnasiadka | yusufgungor: paste them to paste.openstack.org, not on the channel please | 08:32 |
| yusufgungor | Hi, thanks for your replies. globals.yaml is here: https://paste.opendev.org/show/bCkk2AtoD4Fxrbi4CUvO/ | 08:44 |
| yusufgungor | We had to enable rabbitmq ha-all policy and durable queues because there exist issues on rabbitmq HA tests. After one rabbitmq node is down then services are getting timeouts from rabbitmq even thought the rabbitmq cluster is healthy. Solution is to stop all services and clear all queues. So we had to return back to the HA queue policy | 08:44 |
| yusufgungor | om_enable_rabbitmq_high_availability: true | 08:44 |
| yusufgungor | rabbitmq_ha_promote_on_shutdown: "always" | 08:45 |
| yusufgungor | But we still encounter problems for "reply" queues which are not HA. Policy pattern ignore reply, fanout and amq queues | 08:45 |
| kevko | that's ok i think | 08:45 |
| kevko | these queues shouldn't be ha i think | 08:46 |
| yusufgungor | you are right but we had no HA problems with transient queues and ha-all policy on wallaby. The problem can be reproduced easily, when HA policy is not defined at all, try to stop the rabbitmq node which has the "conductor" queue. After that even the stopped rabbit node is up again and joined the cluster, conductor logs oslo.messaging timeouts. Restart is not enought to resolve, we had to stop all services and clear | 08:50 |
| yusufgungor | all queues. It is not acceptable for us on production. | 08:50 |
| yusufgungor | This situation also asked on #openstack-oslo but no reply yet | 08:51 |
| kevko | yusufgungor: try to turn off durable queues via config override | 08:51 |
| kevko | and deploy rabbit transient with ha all ... | 08:52 |
| kevko | om_enable_rabbitmq_high_availability: true | 08:52 |
| kevko | this should work | 08:53 |
| yusufgungor | yes we can do, but we want to learn your thoughts about this situation | 08:54 |
| yusufgungor | kolla-ansible by default disables the ha-all policy now and it seems there exist problems on rabbitmq failovers, do we aggree? | 08:57 |
| yusufgungor | Also still there exist problems even we follow the documentation about rabbitmq HA: https://docs.openstack.org/kolla-ansible/latest/reference/message-queues/rabbitmq.html#high-availability | 08:57 |
| mnasiadka | kevko: https://review.opendev.org/c/openstack/kolla/+/873913 - let's merge this given new setuptools might go out and break yoga in future (and still have no clue why this works in CI, but not locally) :) | 09:00 |
| hrw | {% if (base_distro == 'centos' and base_distro_tag.startswith('stream9')) or base_distro == 'rocky' or distro_python_version == '3.9' %} | 09:05 |
| hrw | argh. | 09:05 |
| hrw | why not just "if python 3.9" then? | 09:06 |
| hrw | left comments | 09:06 |
| mnasiadka | makes sense | 09:07 |
| mnasiadka | we need more reviewers like hrw :) | 09:07 |
| hrw | picky and lazy ones? | 09:08 |
| opendevreview | Merged openstack/kolla stable/yoga: Use erlang-25 from copr on aarch64 https://review.opendev.org/c/openstack/kolla/+/886966 | 09:09 |
| opendevreview | Merged openstack/kolla stable/yoga: ovsdpdk: add libdpdk-dev https://review.opendev.org/c/openstack/kolla/+/880317 | 09:09 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: Move to Debian 12 'bookworm' https://review.opendev.org/c/openstack/kolla/+/886088 | 09:10 |
| hrw | as no one rebased it... | 09:10 |
| kevko | mnasiadka done | 09:22 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873913 | 09:23 |
| kevko | hrw, mnasiadka ups, i've approved ... | 09:23 |
| hrw | mnasiadka: where is NOTE? | 09:23 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873913 | 09:25 |
| mnasiadka | hrw: added now | 09:25 |
| hrw | "NOTE: horizon fails with newer setuptools on Python 3.9" maybe? | 09:25 |
| hrw | a = 1 # NOTE: set a to one | 09:26 |
| hrw | comment should say why not what | 09:26 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873913 | 09:27 |
| hrw | +2 | 09:35 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/883716 | 11:30 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: opensearch: move to yum/apt repos https://review.opendev.org/c/openstack/kolla/+/883716 | 11:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: opensearch-dashboards: Fix permissions https://review.opendev.org/c/openstack/kolla/+/884375 | 11:32 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Test some more scenarios on debian https://review.opendev.org/c/openstack/kolla-ansible/+/879132 | 11:34 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ovn: Improve clustering https://review.opendev.org/c/openstack/kolla-ansible/+/868929 | 11:41 |
| opendevreview | Xing Zhang proposed openstack/kolla-ansible master: Fix default_share_type for manila-api https://review.opendev.org/c/openstack/kolla-ansible/+/891588 | 12:12 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Test haproxy single external frontend https://review.opendev.org/c/openstack/kolla-ansible/+/841239 | 12:33 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Move to Debian 12 'bookworm' https://review.opendev.org/c/openstack/kolla/+/886088 | 12:38 |
| opendevreview | Merged openstack/kolla stable/yoga: Pin setuptools=67.2.* https://review.opendev.org/c/openstack/kolla/+/873913 | 12:47 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/zed: CI: Add rocky9 experimental jobs https://review.opendev.org/c/openstack/kolla/+/891457 | 12:47 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: CI: Add rocky9 experimental jobs https://review.opendev.org/c/openstack/kolla/+/891458 | 12:48 |
| mnasiadka | mgoddard mnasiadka hrw bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter - meeting in 10 | 12:50 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: systemd: handle running container without systemd unit https://review.opendev.org/c/openstack/kolla-ansible/+/890198 | 12:53 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Test with docker_restart_policy default https://review.opendev.org/c/openstack/kolla-ansible/+/890658 | 12:53 |
| mmalchuk | cores, please merge backports: https://review.opendev.org/q/I3ec1af6353c3ecc64589599abe375b0ae9b14d5c | 12:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Move to Debian 12 'bookworm' https://review.opendev.org/c/openstack/kolla/+/886088 | 12:59 |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Aug 16 13:00:01 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:00 |
| mnasiadka | o/ | 13:00 |
| jangutter | o\ | 13:00 |
| bbezak | o/ | 13:00 |
| mmalchuk | \o | 13:01 |
| mattcrees | o/ | 13:01 |
| frickler | \o | 13:01 |
| mnasiadka | #topic agenda | 13:02 |
| mnasiadka | * Announcements | 13:02 |
| mnasiadka | * CI status | 13:02 |
| mnasiadka | * Release tasks | 13:02 |
| mnasiadka | * Current cycle planning | 13:02 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:02 |
| mnasiadka | * Open discussion | 13:02 |
| mnasiadka | #topic Announcements | 13:02 |
| mnasiadka | the PTL and TC election has begun for 2024.1 - I've put my name for Kolla PTL for another cycle | 13:03 |
| frickler | yay | 13:03 |
| mmalchuk | vote +1 | 13:03 |
| mnasiadka | I thought so you're not going to oppose ;-) | 13:04 |
| mnasiadka | #topic CI Status | 13:04 |
| mgoddard | \o | 13:04 |
| mnasiadka | Kayobe CI (upgrade) is still red | 13:04 |
| mnasiadka | there's a patch ongoing review, hopefully we can merge it this week | 13:04 |
| mmalchuk | give a link please | 13:05 |
| mnasiadka | the magnum CI job is red as well (everywhere) - anybody with designate knowledge wants to have a look? | 13:05 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/890198 | 13:05 |
| mnasiadka | if nobody wants to have a look - should we disable the designate part in Magnum CI jobs? | 13:06 |
| frickler | hmm, I'll have a look | 13:06 |
| mmalchuk | me too | 13:06 |
| frickler | have some interest in designate being tested | 13:06 |
| mnasiadka | thanks | 13:06 |
| mmalchuk | there several patches for manbum today | 13:06 |
| mmalchuk | magnum* | 13:07 |
| mnasiadka | #topic Release tasks | 13:07 |
| mnasiadka | We're close to standard projects releasing | 13:07 |
| mnasiadka | It's time to switch to UCA and RDO Bobcat | 13:08 |
| mnasiadka | UCA patch: https://review.opendev.org/c/openstack/kolla/+/891018 | 13:08 |
| mnasiadka | RDO repo is still empty for Bobcat | 13:09 |
| mnasiadka | UCA patch needs second core to merge | 13:09 |
| mnasiadka | #topic Current cycle planning | 13:09 |
| mnasiadka | So, kevko has put some work into Let's Encrypt - hrw has been so kind to do a review of a lot of bash scripting | 13:10 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla/+/887347 | 13:10 |
| mnasiadka | and here is the kolla-ansible part | 13:10 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 13:10 |
| mnasiadka | It would be nice if some other cores (or non-cores) would do a thorough review - we should aim for getting this merged in the upcoming weeks | 13:10 |
| mmalchuk | a lot of comments did SvenKieske and me too) | 13:11 |
| hrw | UCA/Bobcat +2+w | 13:12 |
| mnasiadka | Ok, thanks - let's keep it rolling | 13:12 |
| mnasiadka | Another thing, there's a container called openvswitch-netcontrold | 13:12 |
| mnasiadka | For some reason we've been installing the package via pip outside of a venv | 13:12 |
| mnasiadka | The container is only used by tripleo | 13:12 |
| mnasiadka | Is there anybody that is using ovs-dpdk? | 13:12 |
| mnasiadka | netcontrold is a tool for optimizing PMD threads in OVS | 13:13 |
| mnasiadka | From my perspective - it would be easier to drop it | 13:13 |
| mmalchuk | +1 | 13:13 |
| hrw | +2 | 13:13 |
| mnasiadka | frickler: any dpdk usage on your side and you'd like to use netcontrold? ;-) | 13:14 |
| frickler | nope | 13:15 |
| jangutter | https://github.com/netcontrold/netcontrold-py <--- last update july 2020 .... | 13:15 |
| mnasiadka | Ok, I'll send a mail to ML and drop this image | 13:15 |
| mnasiadka | #action mnasiadka to send a mail to ML and drop openvswitch-netcontrold | 13:15 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:16 |
| mnasiadka | so, regarding jobboard we have second patch to review - https://review.opendev.org/c/openstack/kolla-ansible/+/888588 | 13:17 |
| mnasiadka | it has RP+12 | 13:17 |
| mnasiadka | it has RP+1 | 13:17 |
| mnasiadka | ok, second thing is debian bookworm | 13:18 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla/+/886088 | 13:18 |
| mnasiadka | that should pass now and from my perspective we can merge that - and work on the kolla-ansible side of things | 13:18 |
| mmalchuk | it can be merged afaik | 13:19 |
| frickler | we need haproxy fix first | 13:20 |
| frickler | I guess I should rework https://review.opendev.org/c/openstack/kolla-ansible/+/883559/5 | 13:20 |
| mmalchuk | why? whats wrong with haproxy? | 13:20 |
| frickler | if we kind of agree to do it all in one step | 13:20 |
| frickler | the option we use is no longer there in bookworm | 13:20 |
| mmalchuk | another version there? | 13:21 |
| frickler | newer version, yes, removed deprecated option | 13:21 |
| mnasiadka | newer versions of haproxy does not support that option | 13:21 |
| mmalchuk | lets stack changes | 13:21 |
| mnasiadka | so basically we could fine tune those three patches and merge them, unless frickler wants to work on making out of those three patches (I'm lazy and I wouldn't) ;-) | 13:22 |
| frickler | can't stack k-a on k | 13:22 |
| frickler | they need rebasing anyway afaict | 13:22 |
| mnasiadka | and there's also the thing about python version in wsgi paths | 13:22 |
| mnasiadka | I did have a stupid approach on linking /var/lib/kolla/venv/lib/python3.X to /var/lib/kolla/venv/lib/python | 13:23 |
| mnasiadka | to stop this madness of doing a lot of changes just to pass the variable properly in CI | 13:23 |
| mmalchuk | lets merge stupid approach first) | 13:23 |
| frickler | which patch is that? | 13:24 |
| kevko | is /var/lib/kolla/venv/lib/python good name ? | 13:24 |
| mnasiadka | so, in the past we did: https://review.opendev.org/c/openstack/kolla/+/788304/2/.zuul.d/debian.yaml#9 | 13:25 |
| mnasiadka | and now I was thinking of https://review.opendev.org/c/openstack/kolla/+/874989 and https://review.opendev.org/c/openstack/kolla-ansible/+/874990 | 13:25 |
| kevko | wouldn't be better to include some custom path into path ? | 13:25 |
| mmalchuk | you have another path to venv? | 13:26 |
| frickler | I'm lacking context as to why this is/was all needed | 13:26 |
| kevko | it's not my case ..but what if you are just not using venv ? | 13:27 |
| frickler | but I can read up later | 13:27 |
| mnasiadka | mainly because of this: https://review.opendev.org/c/openstack/kolla-ansible/+/874990/3/ansible/roles/horizon/templates/horizon.conf.j2 | 13:27 |
| mnasiadka | (and other wsgi services) | 13:27 |
| mnasiadka | we need to pass the whole path to mod_wsgi | 13:27 |
| mmalchuk | kevko pip requieres venev now | 13:27 |
| mnasiadka | other option is just a play in kolla-ansible that find proper python version | 13:28 |
| mnasiadka | and set distro_python_version to that | 13:28 |
| mnasiadka | (the one in container images) | 13:28 |
| mmalchuk | bad idea | 13:28 |
| mmalchuk | what if there are two suitable versions? | 13:28 |
| mnasiadka | two versions in a venv? | 13:29 |
| mmalchuk | in system. but search in venv for other path? lol | 13:30 |
| frickler | it seems wrong to me to run the wsgi from site-packages, why is this not in xxx/bin? | 13:30 |
| mmalchuk | according the venv the path always lib/... | 13:31 |
| kevko | frickler +1 | 13:31 |
| mmalchuk | there can't be several versions so the path is constant | 13:31 |
| mmalchuk | frickler why? | 13:32 |
| mmalchuk | it run within venv | 13:32 |
| mmalchuk | if there is no bin/... file - this is an issue of the package itself | 13:33 |
| frickler | because of the mess we are seeing now. seems like a bug in horizon to me. but I need to take a closer look, might be I'm missing something | 13:33 |
| frickler | mmalchuk: exactly | 13:33 |
| mmalchuk | lets fix python lib | 13:33 |
| frickler | but that's not from python lib, it is horizon packaging | 13:34 |
| mmalchuk | show ma your bug later... will take a look | 13:34 |
| mnasiadka | frickler: horizon freezer and skyline seem to be using it | 13:34 |
| frickler | one bad precedent copied two times, doesn't make it more valid IMHO | 13:34 |
| frickler | but we don't need to decide on this now, do we? | 13:35 |
| mmalchuk | lets dig deeper... later | 13:36 |
| mnasiadka | yup | 13:36 |
| mnasiadka | ok then | 13:36 |
| kevko | btw, can we also drop kolla_distro ? :) | 13:37 |
| mnasiadka | we can drop kolla if you like | 13:37 |
| mnasiadka | :) | 13:37 |
| mmalchuk | lol | 13:37 |
| mmalchuk | an move to the k8s))) | 13:38 |
| kevko | sometimes i am trying something with different image locally to just be sure that some implementation is working on both distros ... | 13:38 |
| mnasiadka | that's how it is and have been for long time ;) | 13:38 |
| mnasiadka | anyway, let's continue | 13:38 |
| mnasiadka | mattcrees would like us to review https://review.opendev.org/c/openstack/kolla-ansible/+/882825 | 13:38 |
| mattcrees | Yes please | 13:39 |
| mnasiadka | and https://review.opendev.org/c/openstack/kolla-ansible/+/890391/4 | 13:39 |
| mattcrees | Would also be good to get a non-shpc core to review this: https://review.opendev.org/c/openstack/kolla-ansible/+/881392 | 13:39 |
| kevko | i would like to see some test with rabbitmq nodes stop/start to see everything is working ok | 13:39 |
| kevko | queues recreated , clients reconnected ...etc | 13:40 |
| kevko | can we add something like this into ci ? | 13:40 |
| frickler | kevko wants to be our chaos monkey ;) | 13:40 |
| mmalchuk | mattcrees I'm not shpc and would like to be core) as you wish | 13:41 |
| frickler | I'm not sure if it is possible to do in a stable way in CI | 13:41 |
| mmalchuk | anyway reviewed | 13:41 |
| mnasiadka | let's first get the CI in a stable form :) | 13:41 |
| kevko | well, today there was some guy asking for these types of error ... | 13:42 |
| kevko | *problems | 13:42 |
| mnasiadka | true, but we all know that not all errors are bugs in our configuration | 13:42 |
| mnasiadka | there's also oslo.messaging bugs and RMQ code bugs | 13:42 |
| mattcrees | Does sound like it would cause CI problems. The recent rabbit changes improve stability, but don't guarantee it ;) | 13:43 |
| mattcrees | Perhaps a not-voting job would be useful, to at least test regularly even if it falls over sometimes? | 13:43 |
| frickler | but who would look at that regularly? | 13:44 |
| mnasiadka | that's the problem, now there's not a lot of people looking at periodic publish jobs, not counting anything else :) | 13:44 |
| mnasiadka | I think this can be a topic for the 2024.1 PTG, let's focus on existing patches to review now :) | 13:45 |
| mnasiadka | #topic Open Discussion | 13:45 |
| mmalchuk | this can be linked to whiteboard, and we check this atleast each week | 13:45 |
| mnasiadka | ok then, anybody has any additional topic? | 13:45 |
| mmalchuk | yep | 13:46 |
| jangutter | (me after that?) | 13:46 |
| mmalchuk | https://review.opendev.org/c/openstack/kolla/+/891098 | 13:46 |
| mmalchuk | handler to RH distros to make server-status everywhere identical | 13:46 |
| mmalchuk | the same as in Ubuntu/Debian | 13:47 |
| mmalchuk | same way | 13:47 |
| mmalchuk | and also, folks please review https://review.opendev.org/q/I3ec1af6353c3ecc64589599abe375b0ae9b14d5c | 13:47 |
| mmalchuk | X and Y | 13:47 |
| mnasiadka | maybe let's merge the kolla-ansible patch first, so we don't introduce a security bug :) | 13:48 |
| mmalchuk | we don't | 13:48 |
| mmalchuk | and we're talking in master first | 13:48 |
| mmalchuk | the fix is already there | 13:48 |
| mnasiadka | I don't know if doing that in httpd_setup is the best way to do it, we could just do it in openstack-base by COPYing the file - but I have no strong opinion, let's have some other people voice in that patch | 13:49 |
| mmalchuk | in Ubuntu/Debian this did in the kolla image so this is the same way | 13:49 |
| mmalchuk | it configured and enabled right after apt-get install apache2 | 13:50 |
| mmalchuk | so for RH distros the httpd_setup the only place | 13:50 |
| mnasiadka | not really - you can do it in here as well: https://github.com/openstack/kolla/blob/7f12d216dc4de2c8d32291c3d6223185ecf2b510/docker/openstack-base/Dockerfile.j2#L59 (we install apache here) | 13:51 |
| mnasiadka | but let's discuss in the patch | 13:52 |
| mnasiadka | 9 minutes left | 13:52 |
| mnasiadka | jangutter: ? | 13:52 |
| jangutter | etcd 3.3 -> 3.4 ... looks like fixing it in tooz will take a while. | 13:52 |
| mmalchuk | apache not installed in base | 13:52 |
| mnasiadka | mmalchuk: openstack-base | 13:52 |
| mnasiadka | jangutter: and before it is fixed we can't do the upgrade? | 13:52 |
| mmalchuk | mnasiadka RH only | 13:53 |
| jangutter | So, it will require a change in both kolla and kolla-ansible if we can't wait for tooz. It passes all CI so far (flaky job just rarely) | 13:53 |
| mmalchuk | mnasiadka it still kolla not kolla-ansible | 13:53 |
| jangutter | If tooz can be updated, then kolla-ansible doesn't need changes. | 13:54 |
| mnasiadka | mmalchuk: your patch is in kolla ;) | 13:54 |
| frickler | tooz won't be updated in time for bobcat very likely | 13:54 |
| mnasiadka | jangutter: what kind of changes? a lot? | 13:54 |
| mmalchuk | mnasiadka but first you propose fix this in k-a | 13:54 |
| jangutter | config file changes, minimal.... but until the endpoint is updated, cinder won't work, for example. | 13:55 |
| mnasiadka | mmalchuk: that was long time ago, not fix, but allow users to configure it | 13:55 |
| frickler | does cinder need etcd? or is it just an alternative to redis? | 13:55 |
| mnasiadka | alternative - tooz supports both | 13:55 |
| kevko | i think it's alternative, isn't it ? | 13:55 |
| jangutter | It can use anything that tooz supports as a backend, but etcd seems to be the default in the CI. | 13:55 |
| mnasiadka | some people like etcd, have no clue why :) | 13:56 |
| frickler | but other services use redis, maybe we should make that consistent as a default? | 13:56 |
| kevko | mnasiadka for similar reasons as some people like rh clones :) | 13:56 |
| kevko | don't know why :D | 13:56 |
| mnasiadka | kevko: I don't like them, they pay my rent :) | 13:56 |
| mnasiadka | frickler: with tooz we support both, but I thought we have a recommendation for redis, not etcd | 13:57 |
| frickler | so at least CI should be consistent? will need to look into that, too | 13:58 |
| mnasiadka | basically we use what is enabled | 13:58 |
| mnasiadka | #cinder_coordination_backend: "{{ 'redis' if enable_redis|bool else 'etcd' if enable_etcd|bool else '' }}" | 13:58 |
| jangutter | I think it doesn't really matter what coordination backend is in use ... but at the moment everything in the CI (masakari and something else) points to etcd... | 13:58 |
| jangutter | Aha | 13:58 |
| mnasiadka | masakari? I thought only zun | 13:59 |
| mnasiadka | but zun for a different reason (not tooz, docker overlay something) | 13:59 |
| * frickler needs to run, will check back later | 13:59 | |
| mnasiadka | anyway, time is up | 14:00 |
| jangutter | --- masakari, cinder, designate, ironic (ironically!) | 14:00 |
| mnasiadka | I think I have a patch for enabling redis in the ceph jobs | 14:00 |
| mnasiadka | let's talk about this next week or somewhere outside of the meeting | 14:00 |
| mnasiadka | need to run for a different meeting | 14:00 |
| mnasiadka | thanks for coming! | 14:00 |
| jangutter | thanks! | 14:00 |
| mnasiadka | #endmeeting | 14:00 |
| opendevmeet | Meeting ended Wed Aug 16 14:00:37 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-16-13.00.html | 14:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-16-13.00.txt | 14:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-08-16-13.00.log.html | 14:00 |
| mmalchuk | mnasiadka thanks | 14:00 |
| mmalchuk | cores please +W last two: https://review.opendev.org/q/I3ec1af6353c3ecc64589599abe375b0ae9b14d5c | 14:01 |
| mmalchuk | there need second +2 | 14:01 |
| jangutter | Oh, I forgot, 77 lines to review if anyone likes ARA: https://review.opendev.org/c/openstack/kolla-ansible/+/891097 | 14:06 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Deny access to public /server-status in http Openstack services https://review.opendev.org/c/openstack/kolla-ansible/+/890857 | 14:17 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Deny access to public /server-status in http Openstack services https://review.opendev.org/c/openstack/kolla-ansible/+/890858 | 14:17 |
| opendevreview | Merged openstack/kolla master: [release] Use UCA Bobcat https://review.opendev.org/c/openstack/kolla/+/891018 | 14:35 |
| opendevreview | Simon Dodsley proposed openstack/kolla-ansible master: Add Pure Storage FlashBlade as Manila backend https://review.opendev.org/c/openstack/kolla-ansible/+/879846 | 14:39 |
| opendevreview | Simon Dodsley proposed openstack/kolla-ansible master: Add Pure Storage FlashBlade as Manila backend https://review.opendev.org/c/openstack/kolla-ansible/+/879846 | 14:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Stop Galera container after bootstrap https://review.opendev.org/c/openstack/kolla-ansible/+/891606 | 14:57 |
| opendevreview | Simon Dodsley proposed openstack/kolla-ansible master: Add Pure Storage FlashBlade as Manila backend https://review.opendev.org/c/openstack/kolla-ansible/+/879846 | 15:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Stop Galera container after bootstrap https://review.opendev.org/c/openstack/kolla-ansible/+/891606 | 15:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Stop Galera container after bootstrap https://review.opendev.org/c/openstack/kolla-ansible/+/891606 | 15:56 |
| opendevreview | Maximilian Sesterhenn proposed openstack/kolla master: WIP: Add ovn-bgp-agent / FRR / Horizon BGPVPN dashboard support https://review.opendev.org/c/openstack/kolla/+/891617 | 17:55 |
| opendevreview | Maximilian Sesterhenn proposed openstack/kolla-ansible master: WIP: Add ovn-bgp-agent / FRR / neutron-bgpvpn / Horizon BGPVPN dashboard support https://review.opendev.org/c/openstack/kolla-ansible/+/891622 | 18:19 |
| *** bjoernt_ is now known as bjoernt | 21:41 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!