| mnasiadka | yoctozepto, hrw, frickler, mgoddard: I'm not able to run the meeting today start to end - can one of you take over? (I'll be available on the meeting, but mainly via mobile). | 06:06 |
|---|---|---|
| frickler | mnasiadka: ack, can do | 06:33 |
| mnasiadka | frickler: thanks | 06:45 |
| mnasiadka | https://review.opendev.org/c/openstack/kayobe/+/860406 - can we merge this? | 06:48 |
| mnasiadka | ah, it's kayobe | 06:48 |
| mnasiadka | mgoddard, priteau ^^ | 06:48 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Deprecate Monasca and dependencies https://review.opendev.org/c/openstack/kolla-ansible/+/860405 | 07:03 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Skip ansible-lint fqcn for module actions https://review.opendev.org/c/openstack/kolla-ansible/+/861039 | 07:41 |
| mnasiadka | and ansible-lint broke again, now FQCN for module actions | 07:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Skip ansible-lint fqcn for module actions and role-name https://review.opendev.org/c/openstack/kolla-ansible/+/861039 | 07:51 |
| *** ramona-rautenberg[m] is now known as ramona-beermann[m] | 07:57 | |
| opendevreview | Verification of a change to openstack/kayobe master failed: added support for ANSIBLE_VAULT_PASSWORD_FILE https://review.opendev.org/c/openstack/kayobe/+/858703 | 07:57 |
| mnasiadka | ansible/roles/elasticsearch/tasks/upgrade.yml:4: jinja[invalid]: Unexpected templating type error occurred on ({{ elasticsearch_shard_body | to_json }}): Object of type PosixPath is not JSON serializable | 07:58 |
| mnasiadka | geez | 07:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Skip ansible-lint fqcn for module actions and role-name https://review.opendev.org/c/openstack/kolla-ansible/+/861039 | 08:02 |
| opendevreview | Merged openstack/kayobe master: added option to skip hooks https://review.opendev.org/c/openstack/kayobe/+/858159 | 09:13 |
| admin1 | hi all .. i am looking for direction on deploying lbaas wtih kolla-ansible .. in the network section, do I use a separate interface in all or do I use providers network ? | 09:17 |
| yoctozepto | mnasiadka, frickler: maybe let's just pin ansible-lint and bump as we see fit? | 09:21 |
| frickler | yoctozepto: last PTG we discussed and decided to stick with what we have now until the next incident | 09:29 |
| yoctozepto | it's the next incident (-: | 09:29 |
| frickler | we've already added this as PTG topic again, seems ansible-lint seems to support that timing | 09:30 |
| yoctozepto | indeed | 09:30 |
| yoctozepto | they want us to pin them | 09:30 |
| frickler | but we don't easily give others what they want | 09:30 |
| frickler | anyway, is it enough to have the skip now and discuss next week? | 09:31 |
| yoctozepto | yeah, yeah | 09:33 |
| *** dasTor_ is now known as dasTor | 09:34 | |
| yoctozepto | btw, I agree with the move to FQCNs - we should do it; the other subtleties of ansible-lint is what makes me want to pin them (going too much into style territory) | 09:34 |
| frickler | +1 | 09:35 |
| mnasiadka | seems it passes now | 09:37 |
| mnasiadka | but yes, ansible-lint greets us with new crap every release | 09:37 |
| frickler | they kind of made that their mission https://github.com/ansible/ansible-lint/blob/main/docs/philosophy.md#style-and-formatting | 09:42 |
| ramona-beermann[m] | For Ansible FQCN and maybe for ansible-lint i can help. For the FQCN's if you want to use all, you have to implement a custom rule in ansible-lint. Because at this time ansible-lint only checks if ansible.builtin.xxx is in use. Not the other collections. I built one in the past for our company. | 09:43 |
| mnasiadka | Now they check for more FQCNs | 09:44 |
| mnasiadka | We don't need to implement them all | 09:44 |
| mnasiadka | We can start with what ansible-lint requires | 09:45 |
| mnasiadka | let's discuss that next week, shall we? | 09:45 |
| ramona-beermann[m] | Them only checks builtin and legacy according to the docs https://ansible-lint.readthedocs.io/rules/fqcn/ and that is not so many. | 09:46 |
| ramona-beermann[m] | ok :) | 09:46 |
| yoctozepto | I left some related notes on the ptg etherpad | 09:47 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: Deprecate Monasca and dependencies https://review.opendev.org/c/openstack/kolla-ansible/+/860405 | 10:33 |
| opendevreview | Merged openstack/kolla-ansible master: Skip ansible-lint fqcn for module actions and role-name https://review.opendev.org/c/openstack/kolla-ansible/+/861039 | 11:34 |
| opendevreview | Merged openstack/kolla-ansible master: Deprecate Monasca and dependencies https://review.opendev.org/c/openstack/kolla-ansible/+/860405 | 11:35 |
| zbig | Hey folks, I am looking for bit help with HA setup. My cluster consist out of 8 machines with 3 in controller role (Inventory file https://paste.opendev.org/show/blFqF5uQtjDvfj6Q1E6f/ ) I am deploying Openstack Yoga with kolla-ansible with following settings (globals.yaml https://paste.opendev.org/show/bPhVP7i2es2NEsLxRwMu/ ). I was testing HA capabilities by doing hard power down of one of the controller machine an | 11:48 |
| zbig | d observing how openstack will behave. I have noticed following: Horizon was not usable failing with “something went wrong” page, after debugging I figured out that changing this value here https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/horizon/templates/local_settings.j2#L13 to False is help to made horizon loading properly. But even then, creating new instance was failing and horizon | 11:48 |
| zbig | was super slow. At the same time I have notice that cli openstack client was working ok for read operations (didn’t try write operations). Grafana and Kibana were both fine. So my question is - is there anything specific to HA setup is should consider? I was hoping that taking out one controller node may results in some some issues with network or compute but Horizon and scheduling new workloads would work fin | 11:48 |
| zbig | e. | 11:48 |
| kevko | yoctozepto: I started to implement mechanism for db cleanup for cinder and nova, but I realized that it is not so easy to calculate cron runs for every host ... so i implemented very simple mechanism where cron file is copied to OS (for cinder-api and nova-conductor) but it is commented as template ... and operator can override it for his needs ... is this mergable ? or not ..because semi-kolish ? | 12:14 |
| kevko | i also asked if i need to implement some locking mechanism for this in openstack-cinder and openstack-nova channels ...if no .. this cron can run everywhere and i will implement kolish containers :P ..but need to wait | 12:15 |
| kevko | what do you think ? | 12:15 |
| kevko | and others ? | 12:16 |
| opendevreview | Alex Welsh proposed openstack/kayobe master: adding support for ANSIBLE_VAULT_PASSWORD_FILE https://review.opendev.org/c/openstack/kayobe/+/858703 | 13:02 |
| frickler | this seems to be failing two weeks in a row, is there a known issue? maybe disable this job for now? https://zuul.openstack.org/builds?job_name=kolla-publish-debian-source-aarch64-quay&project=openstack%2Fkolla&branch=stable%2Fyoga&skip=0 | 13:19 |
| mnasiadka | seems like some erlang debian issue | 13:27 |
| mnasiadka | hrw: ^^ | 13:27 |
| hrw | let me check | 13:33 |
| hrw | yeah. erlang 24 or lack of it | 13:38 |
| mnasiadka | should we disable those jobs and wait for better future, or what's the approach? | 13:41 |
| hrw | moment | 13:41 |
| hrw | @%@^@ 3rd party repos | 13:43 |
| hrw | disable | 13:46 |
| hrw | I hate rabbitmq and erlang more and more | 13:46 |
| frickler | mgoddard mnasiadka hrw yoctozepto bbezak parallax Fl1nt frickler adrian-a kevko 6 minute warning | 13:54 |
| opendevreview | Alex Welsh proposed openstack/kayobe master: Fix bug trying to decrypt a plaintext file https://review.opendev.org/c/openstack/kayobe/+/858925 | 13:56 |
| frickler | #startmeeting kolla | 14:00 |
| opendevmeet | Meeting started Wed Oct 12 14:00:16 2022 UTC and is due to finish in 60 minutes. The chair is frickler. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 14:00 |
| frickler | #topic rollcall | 14:00 |
| TimBeermann[m] | \o | 14:00 |
| ramona-beermann[m] | o/ | 14:00 |
| frickler | \o/ | 14:00 |
| mgoddard | \o | 14:00 |
| m_hiner | o/ | 14:00 |
| hrw | /o\ | 14:02 |
| frickler | #topic agenda | 14:02 |
| frickler | * Announcements | 14:02 |
| frickler | * Review action items from the last meeting | 14:02 |
| frickler | * CI status | 14:02 |
| frickler | * Release tasks | 14:02 |
| frickler | * Current cycle planning | 14:03 |
| frickler | * Additional agenda (from whiteboard) | 14:03 |
| frickler | * Open discussion | 14:03 |
| frickler | #topic Announcements | 14:03 |
| frickler | next week is the PTG | 14:03 |
| frickler | anything else to announce? | 14:03 |
| frickler | #topic Review action items from the last meeting | 14:04 |
| hrw | nothing safe enough to put in logs | 14:04 |
| frickler | didn't see any action items | 14:05 |
| frickler | #topic CI status | 14:05 |
| frickler | ansible-lint was releasing funny things again | 14:06 |
| frickler | should be fixed for now and we'll discuss next week in more detail | 14:06 |
| frickler | I mentioned a periodic job failure earlier (kolla-publish-debian-source-aarch64-quay) | 14:06 |
| yoctozepto | o/ | 14:07 |
| frickler | otherwise I think we are fine (don't say that too loud) | 14:07 |
| yoctozepto | (makes sense) | 14:08 |
| frickler | #topic Release tasks | 14:08 |
| frickler | I missed that topic in my preparation, I think we are at RC+1 | 14:08 |
| frickler | anything to mention here? | 14:08 |
| yoctozepto | we are lagging with opensearch and systemd, no? | 14:09 |
| frickler | I think so, yes, not sure what deadline we have to get those landed | 14:09 |
| frickler | but maybe deferring to after the PTG is fine? | 14:10 |
| hrw | yep | 14:11 |
| frickler | related: | 14:11 |
| frickler | #topic Current cycle planning | 14:11 |
| frickler | opensearch and systemd | 14:11 |
| yoctozepto | indeed | 14:11 |
| frickler | anything in particular to say about those? | 14:12 |
| frickler | is opensearch just pending reviews or does it still need coding work? | 14:12 |
| yoctozepto | mnasiadka: opensearch question ^ | 14:12 |
| frickler | so I guess we defer that question to when mnasiadka is more available | 14:14 |
| yoctozepto | agreed | 14:14 |
| frickler | IIUC systemd had another update, I didn't get to look at it again, though | 14:15 |
| m_hiner | I would like to discuss the checks and installation of python3-dbus for the systemd | 14:15 |
| m_hiner | In my VMs, usually the Centoses didn't have the the dbus python package available and it had to be installed | 14:16 |
| frickler | that's about the thread that I started here I guess | 14:17 |
| frickler | #link https://review.opendev.org/c/openstack/kolla-ansible/+/816724/34/ansible/module_utils/kolla_systemd_worker.py | 14:17 |
| m_hiner | yes, but also here: https://review.opendev.org/c/openstack/kolla-ansible/+/816724/32/ansible/roles/prechecks/defaults/main.yml#13 | 14:17 |
| yoctozepto | but we can install it on bootstrap and check for in prechecks | 14:18 |
| hrw | m_hiner: python3-dbus is present in CS9 | 14:18 |
| m_hiner | i would like to come to some agreement here so I know how to continue | 14:18 |
| hrw | m_hiner: and master does not support any previous version | 14:18 |
| yoctozepto | what is the problem? | 14:19 |
| frickler | is the question whether to use pypi or distro versions? | 14:19 |
| hrw | also in RL9 | 14:19 |
| m_hiner | frickler: yes | 14:19 |
| m_hiner | it may be that I was just using outdated images, so in some cases it was necessary to install the python-dbus | 14:20 |
| frickler | I would strongly suggest distro version in this case, since IMO it should match the installed dbus things | 14:20 |
| frickler | unless there is some issue with that which I'm missing? | 14:21 |
| m_hiner | i see, so just check for dbus availability in python should be sufficient? | 14:21 |
| frickler | I would think so, yes. plus maybe making sure that that pkg gets installed | 14:22 |
| m_hiner | okay, thanks | 14:22 |
| frickler | yoctozepto: I think your comments essentially said the same? | 14:22 |
| m_hiner | yes, the checks were implemented first but i got sidetracked wit it's unavailability in my VMs | 14:23 |
| m_hiner | other than that, i would like to ask more people to take a look at the patchset and possibly give their opinion on the template discussion there | 14:24 |
| yoctozepto | frickler: yes | 14:24 |
| frickler | I'll have a look at the template thing, too | 14:25 |
| m_hiner | thank you, that's all from me | 14:25 |
| frickler | thx m_hiner. anything else about the remaining short time in this cycle? | 14:26 |
| mnasiadka | Opensearch is in progress on k-a side, will shout when reviews will be needed | 14:27 |
| m_hiner | I think the patchset is more or less ready to be merged | 14:27 |
| m_hiner | although there are some random fails in ARM64 pipelines for which I cannot pinpoint the issue | 14:28 |
| frickler | is there some fixed deadline? or will we delay zed until whenever this is ready? | 14:29 |
| frickler | I think random failures on arm64 may well be unrelated | 14:30 |
| m_hiner | I also think so because sometimes they appear, sometimes they don't | 14:31 |
| mnasiadka | We are a trailing project, iirc we can delay 2-3 months - opensearch is a must to have before release, the rest is very nice to have (if there are reviewers happy to merge and author is responding in reasonable timeframe) | 14:32 |
| frickler | ok, then let's go on for now | 14:33 |
| frickler | #topic Additional agenda (from whiteboard) | 14:33 |
| frickler | (yoctozepto) Deprecating Monasca's friends mentioned in https://review.opendev.org/c/openstack/kolla/+/860404 | 14:33 |
| yoctozepto | done | 14:34 |
| frickler | great | 14:34 |
| frickler | m_hiner: I think we already discussed your topic, too | 14:34 |
| m_hiner | yes | 14:34 |
| frickler | #topic Open discussion | 14:35 |
| frickler | anybody anything? | 14:35 |
| mnasiadka | Should we deprecate Elasticsearch or remove it with opensearch addition? | 14:36 |
| frickler | is there a chance to migrate things? then maybe a deprecation cycle might be useful? assuming we can sustain that | 14:37 |
| frickler | but I'd be open to either option | 14:39 |
| mnasiadka | Yes, we are working on migration, but that’s going to be a direct upgrade from es to opensearch | 14:39 |
| mnasiadka | Will probably propose removal once we iron out the upgrade path | 14:40 |
| frickler | ok | 14:41 |
| frickler | another question from me: does anyone want to use zoom for the PTG? or can we agree on meetpad? | 14:41 |
| frickler | also there is still room on the etherpad to add topics | 14:43 |
| frickler | #link https://etherpad.opendev.org/p/kolla-antelope-ptg | 14:43 |
| frickler | and I guess we'll obviously skip this meeting next week, not sure if that needs to be explicitly announced, mnasiadka? | 14:44 |
| * frickler will assume silent approval of everything | 14:46 | |
| frickler | so thx everyone, we'll be around in case anything else comes up | 14:47 |
| frickler | #endmeeting | 14:47 |
| opendevmeet | Meeting ended Wed Oct 12 14:47:13 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:47 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-10-12-14.00.html | 14:47 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-10-12-14.00.txt | 14:47 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-10-12-14.00.log.html | 14:47 |
| mnasiadka | frickler: I’ll send a mail to ML that we skip the meeting | 14:49 |
| mnasiadka | frickler: thanks for running | 14:49 |
| frickler | anytime | 14:49 |
| hrw | I am looking at preparing some patches for zed- to set rabbitmq/erlang at *one* version | 14:49 |
| hrw | zed will get 3.10.8 instead of 3.10.* | 14:50 |
| hrw | etc | 14:50 |
| opendevreview | Ivan Halomi proposed openstack/ansible-collection-kolla master: Adding podman support https://review.opendev.org/c/openstack/ansible-collection-kolla/+/852240 | 15:08 |
| yoctozepto | hrw: we will forget to bump | 15:17 |
| yoctozepto | why is that necessary? | 15:18 |
| hrw | yoctozepto: look how yoga exploded | 15:18 |
| hrw | rmq team changes erlang requirements when they want | 15:18 |
| hrw | we do it once per release | 15:18 |
| yoctozepto | I did not notice | 15:18 |
| hrw | yoctozepto: 15:19 < frickler> this seems to be failing two weeks in a row, is there a known issue? maybe disable this job for now? https://zuul.openstack.org/builds?job_name=kolla-publish-debian-source-aarch64-quay&project=openstack%2Fkolla&branch=stable%2Fyoga&skip=0 | 15:19 |
| yoctozepto | ah, aarch64 | 15:24 |
| hrw | yeah, the arch no one give a.. | 15:26 |
| yoctozepto | rmq does not | 15:28 |
| yoctozepto | seems vmware is not interested in arm | 15:28 |
| yoctozepto | sadly | 15:29 |
| yoctozepto | hrw: maybe we could build erlang for aarch64? | 15:29 |
| hrw | yoctozepto: want to maintain it? | 15:30 |
| yoctozepto | hrw: could be a nice experiment | 15:30 |
| yoctozepto | we have no idea how bad it is | 15:30 |
| yoctozepto | btw, if we actually manage to add nats support to oslo.messaging, then all rmq problems are gone - even the aarch64 one as nats builds for arm | 15:31 |
| yoctozepto | https://github.com/nats-io/nats-server/releases | 15:31 |
| hrw | Go. builds to binary. I like it | 15:32 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: rabbitmq: fix version to 3.10.8 for all distros/archs https://review.opendev.org/c/openstack/kolla/+/861092 | 15:35 |
| hrw | that's latest 3.10 for now | 15:35 |
| hrw | ubuntu/aarch64 zed is unsafe due to old erlang: https://nvd.nist.gov/vuln/detail/CVE-2022-37026 | 15:39 |
| hrw | centos/aarch64 yoga similar | 15:39 |
| yoctozepto | yeah, go is awesome all around | 15:39 |
| hrw | yoctozepto: anything which builds to binary | 15:40 |
| yoctozepto | it is so easy to please hrw! | 15:44 |
| hrw | yoctozepto: I like you, but today it is easy to get me pass the line | 15:45 |
| hrw | ;P | 15:45 |
| yoctozepto | I see, it's been a hard day for me as well; and it's still not over ;d | 15:49 |
| hrw | and I have lovely shitshow with rmq team on twitter: https://twitter.com/haerwu/status/1580193816128393217 | 15:51 |
| yoctozepto | they have arm64 docker image but not the packages, huh? | 15:59 |
| yoctozepto | I bet zigo would consider this insane :-) | 16:00 |
| yoctozepto | (/me just realised he is in this channel; hi zigo!) | 16:00 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: ubuntu: use PPA to get Erlang 25 for AArch64 https://review.opendev.org/c/openstack/kolla/+/861096 | 16:00 |
| hrw | meh. need to do same for yoga | 16:19 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Add support of podman deployment https://review.opendev.org/c/openstack/kolla-ansible/+/799229 | 16:26 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla stable/yoga: ubuntu: use PPA to get Erlang 25 for AArch64 https://review.opendev.org/c/openstack/kolla/+/861097 | 16:33 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla master: ubuntu: use PPA to get Erlang 25 for AArch64 https://review.opendev.org/c/openstack/kolla/+/861096 | 16:42 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla stable/yoga: ubuntu: use PPA to get Erlang 25 for AArch64 https://review.opendev.org/c/openstack/kolla/+/861097 | 16:43 |
| hrw | updated commit | 16:43 |
| hrw | messages | 16:43 |
| hrw | yoctozepto: https://etherpad.opendev.org/p/KollaWhiteBoard#L640 | 16:45 |
| hrw | updated | 16:49 |
| hrw | need to check xena and wallaby versions | 16:55 |
| * hrw out | 16:56 | |
| hrw | ubuntu/aarch64 is not buildable for wallaby/xena/yoga | 18:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!