Wednesday, 2023-05-24

opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/74239906:11
fricklermnasiadka: sorry I missed your patch, I would've abandoned mine as duplicate, but now you've approved it already06:52
opendevreviewMerged openstack/kolla-ansible master: Fix log collection  https://review.opendev.org/c/openstack/kolla-ansible/+/88395006:59
mnasiadkafrickler: no problems - glad it's fixed07:35
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: opensearch: alter path after using rpm/deb packaging  https://review.opendev.org/c/openstack/kolla-ansible/+/88394207:35
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/74239907:48
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/74239907:49
Fl1ntHi folks!07:50
opendevreviewMichal Nasiadka proposed openstack/kolla master: ubuntu: mark collectd and telegraf as buildable  https://review.opendev.org/c/openstack/kolla/+/88400107:56
Fl1ntguys, did you catched up that dnf install <package><wildcard_version> doesn't work anymore? dnf can't find the appropriate package using wildcard somehow weirdly enough it doesn't seems to affect the CI jobs Oo08:02
opendevreviewMerged openstack/kolla master: CI: Move useful rocky/ubuntu jobs out of experimental  https://review.opendev.org/c/openstack/kolla/+/88393708:03
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: build opensearch in venus scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/88400308:03
opendevreviewGraeme Moss proposed openstack/kolla-ansible master: Fix the fluentd regexp to collect the logs  https://review.opendev.org/c/openstack/kolla-ansible/+/87603908:14
opendevreviewGraeme Moss proposed openstack/kolla-ansible master: Fix the fluentd regexp to collect the logs  https://review.opendev.org/c/openstack/kolla-ansible/+/87603908:15
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: opensearch: alter path after using rpm/deb packaging  https://review.opendev.org/c/openstack/kolla-ansible/+/88394208:48
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/74239908:56
opendevreviewlishuliang proposed openstack/kolla-ansible master: Fix the fluentd regexp to collect the logs  https://review.opendev.org/c/openstack/kolla-ansible/+/87603909:00
opendevreviewlishuliang proposed openstack/kolla-ansible master: Fix the fluentd regexp to collect the logs  https://review.opendev.org/c/openstack/kolla-ansible/+/87603909:04
SvenKieskeFl1nt: can you post an error message somewhere with a complete command line? I doubt that something fundamental has changed in dnf10:02
mmalchukcores, please review and merge https://review.opendev.org/q/topic:fix-bash-expansion-issue-openrc10:31
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors  https://review.opendev.org/c/openstack/kolla-ansible/+/74239910:40
Fl1ntSvenKieske, will do for sure, just tested it with rabbitmq-server-3.8.* and the rabbitmq repo enabled, cache appropriatly find any available package using dnf provides rabbitmq-server and list me all available version, dnf provides rabbitmq-server-3.8.* return a match not found, same with install.10:48
mnasiadkaFl1nt: it works in kolla-build, why wouldn't that work anywhere else?10:58
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Refactor MariaDB restart procedure  https://review.opendev.org/c/openstack/kolla-ansible/+/88418211:36
fricklermnasiadka: I'm a bit worried about the opensearch-upgrade failures in https://review.opendev.org/c/openstack/kolla-ansible/+/883942 , but didn't look into them yet, do you happen to know what is going wrong there?11:38
mnasiadkafrickler: well, it doesn't work in zed, so that's why it fails in upgrade jobs I guess - but will have a look in a minute11:38
fricklermnasiadka: I just had the same idea, that would be plausible, yes.11:39
mnasiadkafrickler: I think we still need a reno that log location for opensearch has changed, but will add that to the k-a change11:40
fricklerso we first need to merge this and then backport11:40
SvenKieskeFl1nt: I believe "dnf provides" is a dnf plugin and not in core? not sure though11:47
Fl1ntSvenKieske, actually it is, I've found out what's going on. Upstream rabbitmq repofile is incomplete, it start at 3.9 and doesn't list 3.8 even if it provide 3.8 release package.11:51
Fl1ntmnasiadka, frickler why did we switched from Elastic to opensearch?11:54
mnasiadkaFl1nt: please don't ask that question again11:54
mnasiadkabecause Elastic changed license11:54
Fl1ntwoot??11:54
SvenKieskeFl1nt: elastic licence is not open source compatible, it's source available with some weird restrictions, I guess it's the "business software licence" or something like that?11:55
mnasiadkahttps://www.elastic.co/blog/license-change-clarification11:55
Fl1ntyeah just saw that mixed apache-2.0/SSPL shit...11:56
SvenKieskesee e.g. here https://www.coss.community/cossc/elastic-license-2-0-and-the-evolution-of-open-source-licensing-3jb311:56
SvenKieskeunder what rock did you live? :D that was all over the news at the time it changed and resulted in amazon forking elasticsearch into opensearch11:57
Fl1ntWhat the fuck is that suppose to mean: "giving users the choice of which license to apply" I do choose MIT11:57
Fl1ntSvenKieske, A rock in front of the Ocean, full of coconut trees and sunshine my friend ^^ 11:58
Fl1ntDidn't had time to care about licenses :D11:58
SvenKieskesounds nicer than looking at licences for sure :D 11:59
Fl1ntI'm rather working on our infra and how it kick asses while not being a settings plumbing hell, and we do reach a sweat spot lately automating offline kolla workflows, services rework such as designate and few others.12:00
Fl1ntBTW: I'll patch few missing blocks and improper macros call in few days as we review the wallaby release in here :D12:00
Fl1ntThen Xena and Yoga following to reach Zed before the end of the year.12:01
Fl1ntok guys, I'll need to offlinize this: https://opendev.org/openstack/kolla/src/tag/12.7.0/docker/kolla-toolbox/Dockerfile.j2#L96 <-- Can't work in offline mode build ^^12:16
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Fix fluentd haproxy/syslog logs  https://review.opendev.org/c/openstack/kolla-ansible/+/88418312:21
mnasiadkaFl1nt: how? setting up a local galaxy server? ;-)12:21
SvenKieskeI mean you can just pre-download and local install ansible-galaxy requirements, no?12:23
* SvenKieske never quite understood the security advantage of air gapped installations.12:24
opendevreviewMerged openstack/kolla-ansible stable/zed: Fix Bash variable expansion issues in openrc file  https://review.opendev.org/c/openstack/kolla-ansible/+/88322812:24
opendevreviewMerged openstack/kolla-ansible stable/xena: Fix Bash variable expansion issues in openrc file  https://review.opendev.org/c/openstack/kolla-ansible/+/88323012:24
opendevreviewMerged openstack/kolla-ansible stable/yoga: Fix Bash variable expansion issues in openrc file  https://review.opendev.org/c/openstack/kolla-ansible/+/88322912:24
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Fix Bash variable expansion issues in openrc file  https://review.opendev.org/c/openstack/kolla-ansible/+/88323112:25
mnasiadkafrickler: https://review.opendev.org/c/openstack/kolla-ansible/+/876039 - that would be also good to get in12:26
Fl1ntSvenKieske, don't ask, I'll not fight against the whole planet, Enterprise do it that way, we're crafting a profesionnal oriented solution, let's deal with it ^^12:29
SvenKieskesure; I mean I like offline installs, air gap is another thing imho :) still a lot of work12:30
Fl1ntmnasiadka, as SvenKieske said, I'll introduce a galaxy_repo_url that let you point out at a local galaxy server or simple git/http web server + a conditional route on the dockerfile depending on this variable.12:30
Fl1ntSvenKieske, per see, it's not an airgapped zone in here, the zone just don't get access to internet, only the mirror have way to replicate data from the internet through a special proxy+ids+whatever.12:31
SvenKieskenice, that could be useful for more people I guess :)12:34
fricklermnasiadka: what about https://review.opendev.org/c/openstack/kolla-ansible/+/876785 ? seems to be the same issue with a different solution? I don't want to actually check those regexes though12:34
Fl1ntSvenKieske, at least for me and Ubisoft at first glance :D Probably a bit for Societe Generale and Redhat too I guess :D12:35
SvenKieskeyeah; I also was a little bit initmidated by the regexes, even if those are not the longest I've seen in my life.12:35
mnasiadkafrickler: I actually checked the one I mentioned - with the change that is stacked on top of it - I haven't checked the one you linked to ;-)12:36
mnasiadkagrami[m]: ^^ - care to elaborate why your fix is better? ;)12:36
fricklermnasiadka: I'm just worrying a bit about the submitter of the other patch. essentially getting ignored for some weeks and then superseded by another patch isn't a nice experience12:37
mmalchukFl1nt ask me about offline builds) I can help12:37
mnasiadkafrickler: true, but the other one mentions kafka :)12:38
Fl1ntmmalchuk, will do as soon as I'll drop my downstream patches, for now I've few of them for fluentD/Kolla-toolbox and Openstack-base.12:38
Fl1ntfrickler, Definitely do agree with that statement ^^12:39
fricklermnasiadka: ah, grami[m] already commented on the other bug, so that's ok-ish I think12:40
grami[m]mnasiadka: so this one follows the log format standard where as the other ones are a hack with no regex grouping this way you can now get system scope Id and removes the errors in fluentd that is caused by the missing system scope12:43
mnasiadkagrami[m]: thanks, it's merging - remember to refresh the cherry pick once it's merged12:43
grami[m]Thanks mnasiadka learning so much. Yes will need to get this down to zed at least. 12:44
mnasiadkamgoddard mnasiadka hrw  bbezak frickler kevko SvenKieske mmalchuk - weekly meeting in 1012:50
opendevreviewMerged openstack/kolla master: opensearch-dashboards: Fix permissions  https://review.opendev.org/c/openstack/kolla/+/88394112:52
mnasiadka#startmeeting kolla13:01
opendevmeetMeeting started Wed May 24 13:01:45 2023 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.13:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:01
opendevmeetThe meeting name has been set to 'kolla'13:01
mnasiadka#topic rollcall13:01
frickler\o13:01
mnasiadkao/13:01
mattcreeso/13:02
mhinero/13:02
mmalchuk\o13:02
ihalomi\o13:02
mgoddard\o13:02
SvenKieskeo/13:03
mnasiadka#topic agenda13:04
mnasiadka* CI status13:04
mnasiadka* Release tasks13:04
mnasiadka* Regular stable releases (first meeting in a month)13:04
mnasiadka* Current cycle planning13:04
mnasiadka* Additional agenda (from whiteboard)13:04
mnasiadka* Open discussion13:04
mnasiadka#topic CI Status13:04
fricklerlots of last minute fun ;)13:04
mnasiadkaSo, not counting the OpenSearch breakage, I think it's not bad13:04
SvenKieskeand we have more tests now not in experimental, no? so should stay better?13:04
mnasiadkaat least we'll notice that something is wrong13:05
fricklermore debian testing might still be nice https://review.opendev.org/c/openstack/kolla-ansible/+/879132?usp=dashboard13:05
mnasiadkawe should probably aim to mark more jobs as voting (prometheus-opensearch should be stable)13:06
mnasiadkafrickler: octavia seems unhappy?13:06
fricklerI just saw that, will check later, ack13:06
mnasiadkaok then13:07
mnasiadka#topic Release tasks13:07
mnasiadkaSo, kolla, kolla-ansible and ansible-collection-kolla are on the way for rc113:07
mnasiadkakayobe needs some patches to be in that shape13:07
mnasiadkabut doesn't sound very bad, at least for now ;-)13:08
mnasiadkamgoddard is looking at reworking the mariadb role to get around the Ansible breakage - because it seems they think it shouldn't work the way we used it ;-)13:08
mnasiadka#link https://github.com/ansible/ansible/issues/8084813:09
mnasiadkado we have any other urgent things to fix?13:09
mgoddardhttps://review.opendev.org/c/openstack/kolla-ansible/+/88418213:09
mgoddard(mariadb restart)13:10
mgoddarddoesn't work yet13:10
mmalchukwe have CI issue with images build13:11
mnasiadkanice13:11
mmalchukfix: https://review.opendev.org/c/openstack/kayobe/+/88167913:11
mmalchukjobs non-voting so it ok for now but blocks other changes13:12
opendevreviewMichal Nasiadka proposed openstack/kolla master: mariadb: Bump to current LTS (10.11)  https://review.opendev.org/c/openstack/kolla/+/88292413:12
mnasiadkamattcrees, mgoddard can you have a look in that kayobe patch?13:12
SvenKieskemgoddard: is https://review.opendev.org/c/openstack/kolla-ansible/+/884182 ready for review? didn't really understand your comment "doesn't work yet" as zuul says it's all good?13:13
mnasiadkanot good, check the failed mariadb and multinode jobs13:14
mgoddardmmalchuk: will take a look13:15
mmalchukmgoddard thanks13:15
SvenKieskemhm, we really should make more jobs voting, it's very confusing for new contributors..13:15
mmalchuk+113:15
mnasiadkaSvenKieske: that's what we should aim in B cycle13:16
mnasiadkalet's focus on releasing for now13:16
SvenKieskesure :)13:16
fricklerjust teach new contributors that "non-voting" != "ignore me"13:18
frickleralso remind old cores of that from time to time again ;)13:19
SvenKieske(but only sometimes; it's really easy ;) )13:19
mnasiadkatrue, but jobs like prometheus-opensearch or others that are relatively stable could be marked as voting13:19
mnasiadkaand we could try to focus to get the multinode jobs less failing13:20
fricklerack13:21
mnasiadkaanyway, kolla and kolla-ansible don't seem to have any more RP+1 patches13:22
mmalchukthere have couple from me)13:23
Fl1ntRP?13:23
mmalchukhttps://review.opendev.org/c/openstack/kolla-ansible/+/88210013:23
mmalchukhttps://review.opendev.org/c/openstack/kolla-ansible/+/88349513:23
SvenKieskeFl1nt: review priority13:24
Fl1ntaaah yes ^^13:24
SvenKieskemmalchuk: I gave all the +1 I had :)13:24
mmalchukcool13:24
SvenKieskeactually I didn't, looking at the second changeset13:24
Fl1ntI still need another +1 in here: https://review.opendev.org/c/openstack/kolla-ansible/+/87827013:25
mmalchukdone13:26
Fl1ntnice, thx!13:26
mnasiadkaok then13:26
mnasiadkalet's move on13:26
mnasiadka#topic Additional agenda (from whiteboard)13:26
mnasiadkaihalomi: it's podman and /run again - what's up?13:27
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Drop support for openEuler  https://review.opendev.org/c/openstack/kolla-ansible/+/87912913:27
ihalomii had exams so i didnt have time to look more into it but basically the only option i see is to manually create all subdirs inside run before running deployment 13:28
SvenKieskeyeah, pretty ugly; I even asked upstream podman, but have not seen any indication of change to be more compatible with docker :/13:28
Fl1ntgave all my +1 available on the two ticket mnasiadka and SvenKieske 13:29
mnasiadkaihalomi: ok, so docker creates them by itself, podman doesn't - can we create it in kolla_container Ansible module?13:30
mnasiadkathe other question is - why are those /run subdirs absent?13:30
SvenKieskewell, technically podman can create them, but it's a security thing regarding one subdir which is incompatible how we spawn the container..13:31
SvenKieskethis is the issue: https://github.com/containers/podman/issues/16305#issuecomment-129981000413:33
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Refactor MariaDB restart procedure  https://review.opendev.org/c/openstack/kolla-ansible/+/88418213:33
ihalomii guess we can create it inside module but wouldnt it be better to create all of them at once before deployment and not try to do it before each container?13:33
SvenKieskewe mount "shared" and this doesn't work the second time, because it's a tmpfs, AFAIK13:33
mnasiadkaso - docker is not using /run/secrets - or what is the difference?13:34
SvenKieskeI _think_ docker shares secrets via ENV? but not 100% sure13:34
mnasiadkaihalomi: didn't somebody give you a list of all /run dirs that are required for libvirt?13:35
ihalomiyes, but there are some others that are needed under specific conditions13:36
SvenKieskeuhm, I might have found a workaround, if really only /run/secret is the problem13:37
mnasiadkawhat specific conditions?13:37
SvenKieskeyou can configure recent podman versions to supply secrets via env: https://docs.podman.io/en/latest/markdown/options/secret.html#secret-secret-opt-opt13:37
ihalomifor example openvswitch and linuxbridge13:37
SvenKieske"type=mount|env" default is mount13:38
SvenKieskeat least since podman 4.313:38
ihalomiyes but from what i understood you have to mount each secret manually 13:38
SvenKieskewith the "env" variant you don13:39
SvenKieske't have to mount anything13:39
SvenKieskelet's discuss this seperately, maybe?13:39
ihalomiand second problem is it that it will mount all secrets from user to container anyway and thats security issue i guess 13:40
mnasiadkahmm, how tripleo is/was doing that13:40
mnasiadkahttps://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/train/deployment/nova/nova-libvirt-container-puppet.yaml13:40
mnasiadkamaybe that will shed some light13:40
mnasiadka(they supported podman in train)13:41
ihalomithey are mounting run as not shared and only mounting /run/libvirt as shared 13:42
mnasiadkamaybe that's a solution - but that needs testing13:42
ihalomior /var/run/libvirt to be precise 13:42
SvenKieskeand they are running virsh commands to set nova secrets: https://opendev.org/openstack/tripleo-heat-templates/src/branch/stable/train/deployment/nova/nova-libvirt-container-puppet.yaml#L79713:42
mnasiadkaSvenKieske: I think we push that in a file13:44
mnasiadkaanyway, some homework to do on podman ;-)13:45
mnasiadka#topic Open discussion13:46
mnasiadkaAnybody anything?13:46
mmalchukKayobe still lack of reviewers13:46
kevkoo/ sorry, another meeting :(13:46
opendevreviewMerged openstack/kolla-ansible master: opensearch: alter path after using rpm/deb packaging  https://review.opendev.org/c/openstack/kolla-ansible/+/88394213:46
mmalchukwho can review my changes?13:46
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/88386713:47
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/87955413:47
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/79369713:47
mmalchukhttps://review.opendev.org/c/openstack/kayobe/+/84003313:47
mmalchukenough for now13:47
opendevreviewMark Goddard proposed openstack/kayobe master: dev: Improve error checking for config check functions  https://review.opendev.org/c/openstack/kayobe/+/88420213:48
opendevreviewMichal Nasiadka proposed openstack/kolla stable/zed: opensearch-dashboards: Fix permissions  https://review.opendev.org/c/openstack/kolla/+/88392813:48
opendevreviewMichal Nasiadka proposed openstack/kolla stable/zed: opensearch-dashboards: Fix permissions  https://review.opendev.org/c/openstack/kolla/+/88392813:49
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/zed: opensearch: alter path after using rpm/deb packaging  https://review.opendev.org/c/openstack/kolla-ansible/+/88392913:49
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/zed: opensearch: alter path after using rpm/deb packaging  https://review.opendev.org/c/openstack/kolla-ansible/+/88392913:49
mnasiadkaok then, thanks for coming :)13:50
mnasiadka#endmeeting13:50
opendevmeetMeeting ended Wed May 24 13:50:07 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:50
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-05-24-13.01.html13:50
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-05-24-13.01.txt13:50
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-05-24-13.01.log.html13:50
mmalchukthanks mnasiadka 13:50
kevkothanks13:50
opendevreviewMerged openstack/kolla-ansible master: Fix the fluentd regexp to collect the logs  https://review.opendev.org/c/openstack/kolla-ansible/+/87603913:50
SvenKieskethanks all13:50
opendevreviewGraeme Moss proposed openstack/kolla-ansible stable/zed: Fix the fluentd regexp to collect the logs  https://review.opendev.org/c/openstack/kolla-ansible/+/88391513:59
grami[m]^^ I think i refreshed the cherrypick looks like it's on the right patch set. 14:02
opendevreviewMark Goddard proposed openstack/kolla master: mariadb-clustercheck: ensure /var/log/kolla/mariadb exists  https://review.opendev.org/c/openstack/kolla/+/88420814:19
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Refactor MariaDB restart procedure  https://review.opendev.org/c/openstack/kolla-ansible/+/88418214:21
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Fix passwords.yml permissions  https://review.opendev.org/c/openstack/kolla-ansible/+/88210014:25
opendevreviewMerged openstack/kolla stable/xena: fluentd: pin fluent-plugin-elasticsearch to <5.3.0  https://review.opendev.org/c/openstack/kolla/+/88340814:53
opendevreviewMerged openstack/kolla stable/wallaby: fluentd: pin fluent-plugin-elasticsearch to <5.3.0  https://review.opendev.org/c/openstack/kolla/+/88353614:53
opendevreviewMerged openstack/kayobe master: Show an error when *_bridge_ports is not a list  https://review.opendev.org/c/openstack/kayobe/+/88386715:38
opendevreviewMaksim Malchuk proposed openstack/kayobe stable/zed: Show an error when *_bridge_ports is not a list  https://review.opendev.org/c/openstack/kayobe/+/88424216:02
opendevreviewMaksim Malchuk proposed openstack/kayobe stable/yoga: Show an error when *_bridge_ports is not a list  https://review.opendev.org/c/openstack/kayobe/+/88424316:02
opendevreviewMaksim Malchuk proposed openstack/kayobe stable/xena: Show an error when *_bridge_ports is not a list  https://review.opendev.org/c/openstack/kayobe/+/88424416:03
opendevreviewMaksim Malchuk proposed openstack/kayobe stable/wallaby: Show an error when *_bridge_ports is not a list  https://review.opendev.org/c/openstack/kayobe/+/88424516:03
opendevreviewMerged openstack/kolla stable/zed: opensearch-dashboards: Fix permissions  https://review.opendev.org/c/openstack/kolla/+/88392816:47
opendevreviewMaksim Malchuk proposed openstack/kayobe master: Fix an issue when user forgot combine custom passwords  https://review.opendev.org/c/openstack/kayobe/+/86139718:12
opendevreviewMaksim Malchuk proposed openstack/kayobe master: Fix an issue when user forgot combine custom passwords  https://review.opendev.org/c/openstack/kayobe/+/86139718:28

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!