Wednesday, 2022-10-12

mnasiadkayoctozepto, hrw, frickler, mgoddard: I'm not able to run the meeting today start to end - can one of you take over? (I'll be available on the meeting, but mainly via mobile).06:06
fricklermnasiadka: ack, can do06:33
mnasiadkafrickler: thanks06:45
mnasiadkahttps://review.opendev.org/c/openstack/kayobe/+/860406 - can we merge this?06:48
mnasiadkaah, it's kayobe06:48
mnasiadkamgoddard, priteau ^^06:48
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Deprecate Monasca and dependencies  https://review.opendev.org/c/openstack/kolla-ansible/+/86040507:03
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Skip ansible-lint fqcn for module actions  https://review.opendev.org/c/openstack/kolla-ansible/+/86103907:41
mnasiadkaand ansible-lint broke again, now FQCN for module actions07:41
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Skip ansible-lint fqcn for module actions and role-name  https://review.opendev.org/c/openstack/kolla-ansible/+/86103907:51
*** ramona-rautenberg[m] is now known as ramona-beermann[m]07:57
opendevreviewVerification of a change to openstack/kayobe master failed: added support for ANSIBLE_VAULT_PASSWORD_FILE  https://review.opendev.org/c/openstack/kayobe/+/85870307:57
mnasiadkaansible/roles/elasticsearch/tasks/upgrade.yml:4: jinja[invalid]: Unexpected templating type error occurred on ({{ elasticsearch_shard_body | to_json }}): Object of type PosixPath is not JSON serializable07:58
mnasiadkageez07:58
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Skip ansible-lint fqcn for module actions and role-name  https://review.opendev.org/c/openstack/kolla-ansible/+/86103908:02
opendevreviewMerged openstack/kayobe master: added option to skip hooks  https://review.opendev.org/c/openstack/kayobe/+/85815909:13
admin1hi all .. i am looking for direction on deploying lbaas wtih kolla-ansible ..  in the network section, do I use a separate interface in all or do I use providers network ? 09:17
yoctozeptomnasiadka, frickler: maybe let's just pin ansible-lint and bump as we see fit?09:21
frickleryoctozepto: last PTG we discussed and decided to stick with what we have now until the next incident09:29
yoctozeptoit's the next incident (-:09:29
fricklerwe've already added this as PTG topic again, seems ansible-lint seems to support that timing09:30
yoctozeptoindeed09:30
yoctozeptothey want us to pin them09:30
fricklerbut we don't easily give others what they want09:30
frickleranyway, is it enough to have the skip now and discuss next week?09:31
yoctozeptoyeah, yeah09:33
*** dasTor_ is now known as dasTor09:34
yoctozeptobtw, I agree with the move to FQCNs - we should do it; the other subtleties of ansible-lint is what makes me want to pin them (going too much into style territory)09:34
frickler+109:35
mnasiadkaseems it passes now09:37
mnasiadkabut yes, ansible-lint greets us with new crap every release09:37
fricklerthey kind of made that their mission https://github.com/ansible/ansible-lint/blob/main/docs/philosophy.md#style-and-formatting09:42
ramona-beermann[m]For Ansible FQCN and maybe for ansible-lint i can help. For the FQCN's if you want to use all, you have to implement a custom rule in ansible-lint. Because at this time ansible-lint only checks if ansible.builtin.xxx is in use. Not the other collections. I built one in the past for our company.09:43
mnasiadkaNow they check for more FQCNs09:44
mnasiadkaWe don't need to implement them all09:44
mnasiadkaWe can start with what ansible-lint requires09:45
mnasiadkalet's discuss that next week, shall we?09:45
ramona-beermann[m]Them only checks builtin and legacy according to the docs https://ansible-lint.readthedocs.io/rules/fqcn/ and that is not so many.09:46
ramona-beermann[m]ok :) 09:46
yoctozeptoI left some related notes on the ptg etherpad09:47
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: Deprecate Monasca and dependencies  https://review.opendev.org/c/openstack/kolla-ansible/+/86040510:33
opendevreviewMerged openstack/kolla-ansible master: Skip ansible-lint fqcn for module actions and role-name  https://review.opendev.org/c/openstack/kolla-ansible/+/86103911:34
opendevreviewMerged openstack/kolla-ansible master: Deprecate Monasca and dependencies  https://review.opendev.org/c/openstack/kolla-ansible/+/86040511:35
zbigHey folks, I am looking for bit help with HA setup. My cluster consist out of 8 machines with 3 in controller role (Inventory file https://paste.opendev.org/show/blFqF5uQtjDvfj6Q1E6f/ ) I am deploying Openstack Yoga with kolla-ansible with following settings (globals.yaml https://paste.opendev.org/show/bPhVP7i2es2NEsLxRwMu/ ). I was testing HA capabilities by doing hard power down of one of the controller machine an11:48
zbigd observing how openstack will behave. I have noticed following: Horizon was not usable failing with “something went wrong” page, after debugging I figured out that changing this value here https://github.com/openstack/kolla-ansible/blob/stable/yoga/ansible/roles/horizon/templates/local_settings.j2#L13 to False is help to made horizon loading properly. But even then, creating new instance was failing and horizon11:48
zbig was super slow. At the same time I have notice that cli openstack client was working ok for read operations (didn’t try write operations). Grafana and Kibana were both fine. 
So my question is - is there anything specific to HA setup is should consider? I was hoping that taking out one controller node may results in some some issues with network or compute but Horizon and scheduling new workloads would work fin11:48
zbige. 
   11:48
kevkoyoctozepto: I started to implement mechanism for db cleanup for cinder and nova, but I realized that it is not so easy to calculate cron runs for every host ... so i implemented very simple mechanism where cron file is copied to OS (for cinder-api and nova-conductor) but it is commented as template ... and operator can override it for his needs ... is this mergable ? or not ..because semi-kolish ? 12:14
kevkoi also asked if i need to implement some locking mechanism for this in openstack-cinder and openstack-nova channels ...if no .. this cron can run everywhere and i will implement kolish containers :P ..but need to wait 12:15
kevkowhat do you think ? 12:15
kevkoand others ? 12:16
opendevreviewAlex Welsh proposed openstack/kayobe master: adding support for ANSIBLE_VAULT_PASSWORD_FILE  https://review.opendev.org/c/openstack/kayobe/+/85870313:02
fricklerthis seems to be failing two weeks in a row, is there a known issue? maybe disable this job for now? https://zuul.openstack.org/builds?job_name=kolla-publish-debian-source-aarch64-quay&project=openstack%2Fkolla&branch=stable%2Fyoga&skip=013:19
mnasiadkaseems like some erlang debian issue13:27
mnasiadkahrw: ^^13:27
hrwlet me check13:33
hrwyeah. erlang 24 or lack of it13:38
mnasiadkashould we disable those jobs and wait for better future, or what's the approach?13:41
hrwmoment13:41
hrw@%@^@ 3rd party repos13:43
hrwdisable13:46
hrwI hate rabbitmq and erlang more and more13:46
fricklermgoddard mnasiadka hrw yoctozepto bbezak parallax Fl1nt frickler adrian-a kevko 6 minute warning13:54
opendevreviewAlex Welsh proposed openstack/kayobe master: Fix bug trying to decrypt a plaintext file  https://review.opendev.org/c/openstack/kayobe/+/85892513:56
frickler#startmeeting kolla14:00
opendevmeetMeeting started Wed Oct 12 14:00:16 2022 UTC and is due to finish in 60 minutes.  The chair is frickler. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
opendevmeetThe meeting name has been set to 'kolla'14:00
frickler#topic rollcall14:00
TimBeermann[m]\o14:00
ramona-beermann[m]o/14:00
frickler\o/14:00
mgoddard\o14:00
m_hinero/14:00
hrw /o\14:02
frickler#topic agenda14:02
frickler* Announcements14:02
frickler* Review action items from the last meeting14:02
frickler* CI status14:02
frickler* Release tasks14:02
frickler* Current cycle planning14:03
frickler* Additional agenda (from whiteboard)14:03
frickler* Open discussion14:03
frickler#topic Announcements14:03
fricklernext week is the PTG14:03
frickleranything else to announce?14:03
frickler#topic Review action items from the last meeting14:04
hrwnothing safe enough to put in logs14:04
fricklerdidn't see any action items14:05
frickler#topic CI status14:05
frickleransible-lint was releasing funny things again14:06
fricklershould be fixed for now and we'll discuss next week in more detail14:06
fricklerI mentioned a periodic job failure earlier (kolla-publish-debian-source-aarch64-quay)14:06
yoctozeptoo/14:07
fricklerotherwise I think we are fine (don't say that too loud)14:07
yoctozepto(makes sense)14:08
frickler#topic Release tasks14:08
fricklerI missed that topic in my preparation, I think we are at RC+114:08
frickleranything to mention here?14:08
yoctozeptowe are lagging with opensearch and systemd, no?14:09
fricklerI think so, yes, not sure what deadline we have to get those landed14:09
fricklerbut maybe deferring to after the PTG is fine?14:10
hrwyep14:11
fricklerrelated:14:11
frickler#topic Current cycle planning14:11
frickleropensearch and systemd14:11
yoctozeptoindeed14:11
frickleranything in particular to say about those?14:12
frickleris opensearch just pending reviews or does it still need coding work?14:12
yoctozeptomnasiadka: opensearch question ^14:12
fricklerso I guess we defer that question to when mnasiadka is more available14:14
yoctozeptoagreed14:14
fricklerIIUC systemd had another update, I didn't get to look at it again, though14:15
m_hinerI would like to discuss the checks and installation of python3-dbus for the systemd14:15
m_hinerIn my VMs, usually the Centoses didn't have the the dbus python package available and it had to be installed14:16
fricklerthat's about the thread that I started here I guess14:17
frickler#link https://review.opendev.org/c/openstack/kolla-ansible/+/816724/34/ansible/module_utils/kolla_systemd_worker.py14:17
m_hineryes, but also here: https://review.opendev.org/c/openstack/kolla-ansible/+/816724/32/ansible/roles/prechecks/defaults/main.yml#1314:17
yoctozeptobut we can install it on bootstrap and check for in prechecks14:18
hrwm_hiner: python3-dbus is present in CS914:18
m_hineri would like to come to some agreement here so I know how to continue14:18
hrwm_hiner: and master does not support any previous version14:18
yoctozeptowhat is the problem?14:19
frickleris the question whether to use pypi or distro versions?14:19
hrwalso in RL914:19
m_hinerfrickler: yes14:19
m_hinerit may be that I was just using outdated images, so in some cases it was necessary to install the python-dbus14:20
fricklerI would strongly suggest distro version in this case, since IMO it should match the installed dbus things14:20
fricklerunless there is some issue with that which I'm missing?14:21
m_hineri see, so just check for dbus availability in python should be sufficient?14:21
fricklerI would think so, yes. plus maybe making sure that that pkg gets installed14:22
m_hinerokay, thanks14:22
frickleryoctozepto: I think your comments essentially said the same?14:22
m_hineryes, the checks were implemented first but i got sidetracked wit it's unavailability in my VMs14:23
m_hinerother than that, i would like to ask more people to take a look at the patchset and possibly give their opinion on the template discussion there14:24
yoctozeptofrickler: yes14:24
fricklerI'll have a look at the template thing, too14:25
m_hinerthank you, that's all from me14:25
fricklerthx m_hiner. anything else about the remaining short time in this cycle?14:26
mnasiadkaOpensearch is in progress on k-a side, will shout when reviews will be needed14:27
m_hinerI think the patchset is more or less ready to be merged14:27
m_hineralthough there are some random fails in ARM64 pipelines for which I cannot pinpoint the issue14:28
frickleris there some fixed deadline? or will we delay zed until whenever this is ready?14:29
fricklerI think random failures on arm64 may well be unrelated14:30
m_hinerI also think so because sometimes they appear, sometimes they don't14:31
mnasiadkaWe are a trailing project, iirc we can delay 2-3 months - opensearch is a must to have before release, the rest is very nice to have (if there are reviewers happy to merge and author is responding in reasonable timeframe)14:32
fricklerok, then let's go on for now14:33
frickler#topic Additional agenda (from whiteboard)14:33
frickler(yoctozepto) Deprecating Monasca's friends mentioned in https://review.opendev.org/c/openstack/kolla/+/86040414:33
yoctozeptodone14:34
fricklergreat14:34
fricklerm_hiner: I think we already discussed your topic, too14:34
m_hineryes14:34
frickler#topic Open discussion14:35
frickleranybody anything?14:35
mnasiadkaShould we deprecate Elasticsearch or remove it with opensearch addition?14:36
frickleris there a chance to migrate things? then maybe a deprecation cycle might be useful? assuming we can sustain that14:37
fricklerbut I'd be open to either option14:39
mnasiadkaYes, we are working on migration, but that’s going to be a direct upgrade from es to opensearch14:39
mnasiadkaWill probably propose removal once we iron out the upgrade path14:40
fricklerok14:41
frickleranother question from me: does anyone want to use zoom for the PTG? or can we agree on meetpad?14:41
frickleralso there is still room on the etherpad to add topics14:43
frickler#link https://etherpad.opendev.org/p/kolla-antelope-ptg14:43
fricklerand I guess we'll obviously skip this meeting next week, not sure if that needs to be explicitly announced, mnasiadka?14:44
* frickler will assume silent approval of everything14:46
fricklerso thx everyone, we'll be around in case anything else comes up14:47
frickler#endmeeting14:47
opendevmeetMeeting ended Wed Oct 12 14:47:13 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:47
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-10-12-14.00.html14:47
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-10-12-14.00.txt14:47
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2022/kolla.2022-10-12-14.00.log.html14:47
mnasiadkafrickler: I’ll send a mail to ML that we skip the meeting14:49
mnasiadkafrickler: thanks for running14:49
frickleranytime14:49
hrwI am looking at preparing some patches for zed- to set rabbitmq/erlang at *one* version14:49
hrwzed will get 3.10.8 instead of 3.10.*14:50
hrwetc14:50
opendevreviewIvan Halomi proposed openstack/ansible-collection-kolla master: Adding podman support  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/85224015:08
yoctozeptohrw: we will forget to bump15:17
yoctozeptowhy is that necessary?15:18
hrwyoctozepto: look how yoga exploded 15:18
hrwrmq team changes erlang requirements when they want15:18
hrwwe do it once per release15:18
yoctozeptoI did not notice15:18
hrwyoctozepto: 15:19 < frickler> this seems to be failing two weeks in a row, is there a known issue? maybe disable this job for now? https://zuul.openstack.org/builds?job_name=kolla-publish-debian-source-aarch64-quay&project=openstack%2Fkolla&branch=stable%2Fyoga&skip=015:19
yoctozeptoah, aarch6415:24
hrwyeah, the arch no one give a..15:26
yoctozeptormq does not15:28
yoctozeptoseems vmware is not interested in arm15:28
yoctozeptosadly15:29
yoctozeptohrw: maybe we could build erlang for aarch64?15:29
hrwyoctozepto: want to maintain it?15:30
yoctozeptohrw: could be a nice experiment15:30
yoctozeptowe have no idea how bad it is15:30
yoctozeptobtw, if we actually manage to add nats support to oslo.messaging, then all rmq problems are gone - even the aarch64 one as nats builds for arm15:31
yoctozeptohttps://github.com/nats-io/nats-server/releases15:31
hrwGo. builds to binary. I like it15:32
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: rabbitmq: fix version to 3.10.8 for all distros/archs  https://review.opendev.org/c/openstack/kolla/+/86109215:35
hrwthat's latest 3.10 for now15:35
hrwubuntu/aarch64 zed is unsafe due to old erlang: https://nvd.nist.gov/vuln/detail/CVE-2022-3702615:39
hrwcentos/aarch64 yoga similar15:39
yoctozeptoyeah, go is awesome all around15:39
hrwyoctozepto: anything which builds to binary15:40
yoctozeptoit is so easy to please hrw!15:44
hrwyoctozepto: I like you, but today it is easy to get me pass the line15:45
hrw;P15:45
yoctozeptoI see, it's been a hard day for me as well; and it's still not over ;d15:49
hrwand I have lovely shitshow with rmq team on twitter: https://twitter.com/haerwu/status/158019381612839321715:51
yoctozeptothey have arm64 docker image but not the packages, huh?15:59
yoctozeptoI bet zigo would consider this insane :-)16:00
yoctozepto(/me just realised he is in this channel; hi zigo!)16:00
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: ubuntu: use PPA to get Erlang 25 for AArch64  https://review.opendev.org/c/openstack/kolla/+/86109616:00
hrwmeh. need to do same for yoga16:19
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Add support of podman deployment  https://review.opendev.org/c/openstack/kolla-ansible/+/79922916:26
opendevreviewMarcin Juszkiewicz proposed openstack/kolla stable/yoga: ubuntu: use PPA to get Erlang 25 for AArch64  https://review.opendev.org/c/openstack/kolla/+/86109716:33
opendevreviewMarcin Juszkiewicz proposed openstack/kolla master: ubuntu: use PPA to get Erlang 25 for AArch64  https://review.opendev.org/c/openstack/kolla/+/86109616:42
opendevreviewMarcin Juszkiewicz proposed openstack/kolla stable/yoga: ubuntu: use PPA to get Erlang 25 for AArch64  https://review.opendev.org/c/openstack/kolla/+/86109716:43
hrwupdated commit16:43
hrwmessages16:43
hrwyoctozepto: https://etherpad.opendev.org/p/KollaWhiteBoard#L64016:45
hrwupdated16:49
hrwneed to check xena and wallaby versions16:55
* hrw out16:56
hrwubuntu/aarch64 is not buildable for wallaby/xena/yoga18:52

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!