| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Generate hostnqn https://review.opendev.org/c/openstack/kolla-ansible/+/897227 | 06:33 |
|---|---|---|
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Revert "CI: Disable bare metal testing on RL9/c9s" https://review.opendev.org/c/openstack/kayobe/+/897260 | 06:55 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Revert "CI: Disable bare metal testing on RL9/c9s" https://review.opendev.org/c/openstack/kayobe/+/897260 | 06:57 |
| mnasiadka | morning | 07:19 |
| SvenKieske | morning | 07:24 |
| SvenKieske | I need some vacation after my vacation :D | 07:25 |
| mnasiadka | nothing new | 07:28 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Default keystone user role changed to member https://review.opendev.org/c/openstack/kolla-ansible/+/897222 | 07:30 |
| mnasiadka | SvenKieske: welcome back ;) | 07:33 |
| SvenKieske | thanks :) | 07:35 |
| SvenKieske | that's a nice feature addition: https://review.opendev.org/c/openstack/kolla-ansible/+/897227 | 07:36 |
| mnasiadka | well, that's a bug I would say :) | 07:37 |
| SvenKieske | alright, so it could be a bug worth a "fixes" relnote then? ;) | 07:38 |
| rockey | isn't it easier to utilize copy module with content, instead of a one-row template? | 07:49 |
| rockey | i mean, the complexity of that template isn't really there :) | 07:49 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Fix data file path detection with new pip https://review.opendev.org/c/openstack/kayobe/+/896379 | 07:56 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/zed: Fix data file path detection with new pip https://review.opendev.org/c/openstack/kayobe/+/896380 | 07:59 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/yoga: Fix data file path detection with new pip https://review.opendev.org/c/openstack/kayobe/+/896381 | 07:59 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Generate hostnqn https://review.opendev.org/c/openstack/kolla-ansible/+/897227 | 08:15 |
| SvenKieske | mnasiadka: didn't we already have a planning etherpad for ptg? I can't find it at https://etherpad.opendev.org/p/KollaWhiteBoard | 08:48 |
| mnasiadka | https://etherpad.opendev.org/p/kolla-caracal-ptg (https://ptg.opendev.org/etherpads.html) | 08:49 |
| SvenKieske | added it to the list on the WhiteBoard | 08:50 |
| mnasiadka | thanks | 08:50 |
| SvenKieske | should probably add some content for the PTG as well; I have no prior experience how much we will be able to discuss, to me it seems a little empty? | 08:52 |
| hrw | SvenKieske: check previous ones for examples? | 09:00 |
| SvenKieske | hrw: that's a good idea! | 09:04 |
| SvenKieske | will try to do that later, currently swamped in meetings :) | 09:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Use importlib.metadata instead of importlib_metadata https://review.opendev.org/c/openstack/kayobe/+/897306 | 09:13 |
| opendevreview | Pedro Henrique Pereira Martins proposed openstack/kolla-ansible master: Fix the docker container dimensions comparison for short notation https://review.opendev.org/c/openstack/kolla-ansible/+/886500 | 09:26 |
| opendevreview | Célestin Matte proposed openstack/kolla-ansible master: Ansible-lint: crash when warnings occur https://review.opendev.org/c/openstack/kolla-ansible/+/897312 | 10:03 |
| SvenKieske | did we already enable rabbitmq quorum queues? I lost track.. | 10:14 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Use importlib.metadata instead of importlib_metadata https://review.opendev.org/c/openstack/kayobe/+/897313 | 10:20 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/zed: Use importlib.metadata instead of importlib_metadata https://review.opendev.org/c/openstack/kayobe/+/897314 | 10:20 |
| kevko | mnasiadka: i jus tbuilt images wih kolla and i am missing ironic-dnsmasq | 10:23 |
| mnasiadka | kevko: why? | 10:24 |
| kevko | what i am doing wrong ... do we miss it ? | 10:24 |
| kevko | aaaa, it is different name ..it's just dnsmasq ..not ironic-dnsmasq | 10:29 |
| mnasiadka | haha | 10:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Generate hostnqn https://review.opendev.org/c/openstack/kolla-ansible/+/897227 | 10:59 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: Default keystone user role changed to member https://review.opendev.org/c/openstack/kolla-ansible/+/897222 | 11:07 |
| mnasiadka | mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo - meeting in 7 | 12:53 |
| mnasiadka | #startmeeting koll | 13:00 |
| opendevmeet | Meeting started Wed Oct 4 13:00:54 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'koll' | 13:00 |
| mnasiadka | #endmeeting | 13:00 |
| opendevmeet | Meeting ended Wed Oct 4 13:00:58 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/koll/2023/koll.2023-10-04-13.00.html | 13:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/koll/2023/koll.2023-10-04-13.00.txt | 13:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/koll/2023/koll.2023-10-04-13.00.log.html | 13:00 |
| mnasiadka | #startmeeting kolla | 13:01 |
| opendevmeet | Meeting started Wed Oct 4 13:01:02 2023 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:01 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:01 |
| mnasiadka | #topic rollcall | 13:01 |
| mnasiadka | o/ | 13:01 |
| SvenKieske | o/ | 13:01 |
| jangutter | o/ | 13:01 |
| bbezak | o/ | 13:01 |
| mmalchuk | o/ | 13:01 |
| frickler | \o | 13:02 |
| mhiner | o/ | 13:02 |
| mnasiadka | #topic agenda | 13:03 |
| mnasiadka | * CI status | 13:03 |
| mnasiadka | * Release tasks | 13:03 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 13:03 |
| mnasiadka | * Current cycle planning | 13:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:04 |
| mnasiadka | * Open discussion | 13:04 |
| mnasiadka | #topic CI status | 13:04 |
| mnasiadka | After RMQ drama I think we're back into business as usual | 13:04 |
| mnasiadka | There have been some post failures on pushing to quay.io | 13:05 |
| mnasiadka | but let's monitor that | 13:05 |
| mnasiadka | #topic Release tasks | 13:06 |
| mnasiadka | I think none really, we're waiting for RDO to publish stable/2023.2 | 13:06 |
| mnasiadka | to switch to those repos | 13:07 |
| mnasiadka | nothing apart this | 13:07 |
| mnasiadka | #topic Current cycle planning | 13:07 |
| mnasiadka | So, I didn't have time looking into LE or Podman | 13:07 |
| mnasiadka | But I'll try to make kevko happier by doing it this week | 13:07 |
| mnasiadka | anybody else having cycles to look into those two? | 13:08 |
| SvenKieske | I'll try to, but probably later this week, at least not today I think, still other stuff to do | 13:08 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla/+/887347 | 13:08 |
| mnasiadka | and | 13:09 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/741340 | 13:09 |
| mnasiadka | kevko: the k-a patch has a merge conflict - please update :) | 13:09 |
| mnasiadka | SvenKieske: how are we with td-agent upgrade? need help? | 13:09 |
| SvenKieske | I'd hope to finish that, I'm not aware of "more" issues then the currently known ones: file paths, user changes | 13:10 |
| opendevreview | Juan Pablo Suazo proposed openstack/kolla master: Adds TAAS Neutron plugin to support OVS port mirrors https://review.opendev.org/c/openstack/kolla/+/885151 | 13:10 |
| SvenKieske | I guess I asked last week about whether we care about UID|GID stability and never received an answer | 13:11 |
| SvenKieske | I know there is some code which pins users to certain UIDs I can't imagine a way how this could work with the new fluent package | 13:12 |
| SvenKieske | so in that area, if we care about it, help would be nice, I guess it might be configurable in the new fluentd-package, but I didn't have time to trace it down just yet | 13:12 |
| mnasiadka | do we need to change the user? | 13:12 |
| frickler | well we need some way of upgrading existing things I guess? pinning would be easier than having a migration script | 13:13 |
| SvenKieske | yes, the new username is "_fluentd" if my memory is correct and this is going to be set during "postinst" phase during package install | 13:13 |
| mnasiadka | fantastic | 13:13 |
| SvenKieske | I did already mention this on IRC and in the changeset discussion with no feedback so far :) | 13:13 |
| mnasiadka | well then we need to add that user with new UID | 13:14 |
| mnasiadka | does that mean we need to change some permissions on upgrade? uhh | 13:14 |
| SvenKieske | mhm | 13:14 |
| SvenKieske | not sure what is stored in these paths tbh | 13:14 |
| mnasiadka | do we have any other choice? I don't think so | 13:14 |
| SvenKieske | question is what would be lost if we have /var/lib/fluentd/ instead of old path (don't nail me down on it, the important bit is /var/lib/$bla changes) | 13:15 |
| mnasiadka | probably nothing, but if you don't wander this route - you won't find out I guess | 13:16 |
| SvenKieske | actually don't know what fluentd stores there, beside in flight data, sockets etc | 13:16 |
| SvenKieske | I'll check it out | 13:16 |
| mnasiadka | we should be fine, some buffers, in flight data, sockets | 13:19 |
| mnasiadka | I've also been working on https://review.opendev.org/c/openstack/kolla-ansible/+/897227 - it's a bugfix, but given the amounts of comments in the bug it seems to be important | 13:20 |
| mnasiadka | Seems people are using nvmeof after all | 13:20 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:21 |
| mnasiadka | let's go through that, there's some | 13:21 |
| mnasiadka | (jsuazo) 844614: Deploy Glance and Cinder Backup with S3 backend support | 13:21 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/844614 | 13:21 |
| jsuazo | That proposal has addressed all comments and should be ready to merge :D | 13:22 |
| mmalchuk | confirm that | 13:22 |
| mnasiadka | What about https://review.opendev.org/c/openstack/kolla-ansible/+/844998/1 ? | 13:23 |
| mnasiadka | Should I abandon this? | 13:23 |
| mmalchuk | yep | 13:24 |
| jsuazo | Yes, those changes are included in the proposal | 13:24 |
| mnasiadka | I still don't know if I like the s3_* variables | 13:25 |
| mnasiadka | frickler: willing to have a review? | 13:25 |
| mmalchuk | norm variables) | 13:25 |
| SvenKieske | are they to short? :D | 13:25 |
| mnasiadka | well, usually we should have role specific variables | 13:26 |
| SvenKieske | just add it to the review and discuss there? :) | 13:26 |
| mnasiadka | but if we want to make it easier for users | 13:26 |
| mnasiadka | I guess it's ok | 13:26 |
| kevko | \o/ | 13:26 |
| SvenKieske | could make sense | 13:26 |
| kevko | sorry for delay | 13:26 |
| mnasiadka | no worries kevko | 13:26 |
| mnasiadka | next one | 13:27 |
| mnasiadka | (jsuazo) 885417: Configures the tap-as-a-service neutron plugin | 13:27 |
| SvenKieske | could also be more complicated for the users if they want different s3 URLs/buckets :) | 13:27 |
| mmalchuk | the review is an year old, lets merge it | 13:27 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/885417 | 13:27 |
| mnasiadka | SvenKieske: it's covered in the docs | 13:27 |
| SvenKieske | ok :) | 13:27 |
| mnasiadka | jsuazo: did we merge the Kolla side? | 13:28 |
| mnasiadka | Ah no, we didn't | 13:28 |
| jsuazo | not yet. its the next point on the list | 13:28 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla/+/885151 | 13:28 |
| mmalchuk | not yet | 13:28 |
| jsuazo | Zuul failing on unrelated image builds, rebased again and waiting for the jobs to run | 13:28 |
| mnasiadka | ok then | 13:29 |
| jsuazo | But made the changes to make the installation simply through pip | 13:29 |
| mmalchuk | we change from source to pypi and I'm not sure | 13:29 |
| frickler | please don't rebase if there is no merge conflict | 13:29 |
| * frickler feels like a broken record repeating that | 13:29 | |
| mmalchuk | it work perfect from source | 13:29 |
| mnasiadka | reviewed | 13:31 |
| jsuazo | It did, but as the source is not very well maintained (branch,version and tag-wise) its simpler to do it through pip | 13:31 |
| kevko | why you don't like rebase ? | 13:31 |
| kevko | if no merge conflict | 13:31 |
| frickler | because it breaks patch history and is plain unneccessary | 13:31 |
| mmalchuk | frickler care about CI cpu's ;_ | 13:31 |
| frickler | CI can be triggered with a recheck comment just as well | 13:31 |
| SvenKieske | I'm always a big fan of not doing unneccessary work :) | 13:31 |
| frickler | and reviewers don't have to go looking what might have changed through umpteen unneeded rebases | 13:31 |
| mmalchuk | frickler we don't have history) with git amend)) | 13:31 |
| mnasiadka | frickler: we need a recheck button ;-) | 13:31 |
| mmalchuk | big and red ))))) | 13:32 |
| frickler | no, people need to think before they recheck and add a meaningful reason | 13:32 |
| frickler | else I'll complain about that, too ;-P | 13:32 |
| mnasiadka | haha | 13:33 |
| mnasiadka | ok, let's move on | 13:33 |
| SvenKieske | that's also true | 13:33 |
| mnasiadka | (jsuazo) 875983: Enable the Fluentd Plugin Systemd | 13:33 |
| SvenKieske | but what about that pip bit now? | 13:33 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/875983 | 13:33 |
| mnasiadka | SvenKieske: nothing, should work | 13:33 |
| SvenKieske | ok | 13:33 |
| SvenKieske | the systemd plugin is waiting on your feedback, I did propose to make this dependent on "enable_central_logging" which defaults to "no" | 13:34 |
| SvenKieske | ah no, I see the feedback was already addressed. | 13:35 |
| jsuazo | Added a variable to opt in to the fluentd systemd, "enable_fluentd_systemd",it defaults to the value of "enable_central_logging" | 13:35 |
| mnasiadka | reviewed, docs oneliner and reno fixup needed and looks fine | 13:36 |
| SvenKieske | that sentence should go into the release notes imho | 13:36 |
| SvenKieske | and|or in the docs | 13:36 |
| SvenKieske | not sure how we cover every variable, I guess we don' | 13:36 |
| SvenKieske | t have everything in the docs.. | 13:36 |
| mnasiadka | we don't, but let's try to make docs better, not worse :) | 13:36 |
| jsuazo | agree, i'll add it while addressing the new coments | 13:36 |
| SvenKieske | nice, ty | 13:37 |
| mnasiadka | #topic Open discussion | 13:37 |
| mnasiadka | thanks | 13:37 |
| mnasiadka | so - anybody anything? | 13:37 |
| mnasiadka | Ah right | 13:37 |
| mnasiadka | I have one | 13:37 |
| mnasiadka | PTG timing | 13:37 |
| mnasiadka | frickler: you didn't vote on the poll - so can you state your preferences here please? | 13:38 |
| frickler | I don't care, open to anything | 13:39 |
| SvenKieske | yeah, good news: our downstream meetings are mostly canceled so I guess I should've (more) time for PTG :) | 13:39 |
| mnasiadka | ok, seems everybody is open to anything | 13:39 |
| SvenKieske | I did vote :) | 13:39 |
| mnasiadka | usually we did a very late slot to allow US to join | 13:39 |
| mmalchuk | me too | 13:39 |
| mnasiadka | but I haven't seen anybody from US since long time | 13:40 |
| mmalchuk | anybody is here?) | 13:40 |
| jangutter | Does that mean we can do s/color/colour/g finally? | 13:40 |
| kevko | i have something | 13:40 |
| mnasiadka | seems not | 13:41 |
| mnasiadka | Bobcat PTG was 13:00 - 17:00 UTC | 13:41 |
| mnasiadka | so 15:00 - 19:00 CEST | 13:41 |
| mnasiadka | are we sticking to that, or should we shift a bit earlier? | 13:41 |
| frickler | if you are asking it that way, I'd say earlier | 13:42 |
| mmalchuk | 12:59 UTC ? | 13:42 |
| SvenKieske | one or two hours earlier would be nice I guess. | 13:42 |
| mmalchuk | be more specific | 13:43 |
| SvenKieske | 12:00 - 16:00 UTC ? | 13:43 |
| mmalchuk | one hour is ok | 13:43 |
| mnasiadka | ok, let's shift one hour | 13:43 |
| mnasiadka | it's still east coast US friendly | 13:43 |
| frickler | just note that that won't appear in the official timetable | 13:44 |
| mmalchuk | still nobody from US | 13:44 |
| frickler | but the TC has been being similar things, so it should be ok | 13:44 |
| SvenKieske | mhm, is it to late to change the official timetable? | 13:44 |
| frickler | yes | 13:45 |
| SvenKieske | then we should at least send out an email etc. to notify ppl not living on IRC everyday ;) | 13:45 |
| kevko | Guys, I need to go out of PC for 10 minutes ..but I would like to also ask several things, so let me ask now and we can discuss after several minutes ... | 13:45 |
| SvenKieske | ok | 13:46 |
| mnasiadka | 23-27 October 2023: | 13:46 |
| mnasiadka | Monday - 12.00 - 15.00 UTC (general, Kolla and Kolla-Ansible) | 13:46 |
| mnasiadka | Tuesday - 12.00 - 14.00 UTC (Kolla-Ansible) | 13:46 |
| mnasiadka | Tuesday - 14.00 - 15.00 UTC (Operator Hour) | 13:46 |
| mnasiadka | Thursday - 12.00 - 14.00 UTC (Kayobe) | 13:46 |
| mnasiadka | Is that basically what we would like? | 13:46 |
| SvenKieske | seems fine to me | 13:46 |
| mnasiadka | 3 hours on Monday, 2+1 hours on Tuesday and Kayobe on Thursday | 13:46 |
| frickler | +1 | 13:46 |
| mmalchuk | fine | 13:46 |
| SvenKieske | not 12-16? | 13:47 |
| SvenKieske | one hour less? | 13:47 |
| frickler | with no topics to discuss, we'll be even faster | 13:47 |
| SvenKieske | I don't know, just asking, my first PTG | 13:47 |
| mnasiadka | well, last time we didn't have that many topics, we can do 12-16 and just finish earlier | 13:47 |
| kevko | 1. Rabbitmq/erlang incompatibility ... We found incompatibility in erlang/rabbitmq in yoga.. For example now I am checking master rabbitmq ...paste here -> | 13:47 |
| kevko | https://paste.openstack.org/show/b2voIvMhqW4IblzVE5jD/ | 13:47 |
| mnasiadka | 3.12 in Yoga? | 13:48 |
| kevko | no .. | 13:48 |
| kevko | that paste is for master | 13:48 |
| mmalchuk | kevko may be move to etherpad ? | 13:48 |
| kevko | because currently i've decided to check all debian images from master -> antelope -> ... | 13:49 |
| mnasiadka | kevko: can you post a summary instead of shredded bits? | 13:49 |
| kevko | okay i will | 13:49 |
| SvenKieske | mhm, but shouldn't master be on erlang vm 26 already? | 13:49 |
| mnasiadka | that would be easier to understand and fix :) | 13:49 |
| mnasiadka | probably should be, and kevko is only checking Debian | 13:49 |
| mnasiadka | can we get a full view on all distros and then think if we need to fix anything? | 13:49 |
| kevko | yeah, but question was that i think we should switch erlang/rabbit to novemberrain ..and pin every version in every branch to avoid this in future | 13:49 |
| mnasiadka | rabbit is from novemberain | 13:50 |
| kevko | and erlang should be also i think | 13:50 |
| mnasiadka | erlang can't be switched, because they don't mirror aarch64 | 13:50 |
| kevko | special case is arm | 13:50 |
| mnasiadka | and they use launchpad to build the packages | 13:50 |
| mnasiadka | so why would we download from a mirror that downloads from launchpad? | 13:50 |
| kevko | but arm has different config in kolla | 13:50 |
| mnasiadka | what do you mean by different? | 13:51 |
| kevko | repos.yaml | 13:52 |
| mnasiadka | I don't understand, please describe the situation fully somewhere - so we can all analyse | 13:53 |
| kevko | Second thing - there is no option to configure logging in kolla-ansible .. (i mean, formatters, handlers ...etc ....) ..so i will probably propose a new shared role | 13:53 |
| kevko | okay i will | 13:53 |
| kevko | i will add to etherpad probably | 13:54 |
| mnasiadka | kevko: that's a PTG topic most probably | 13:54 |
| mmalchuk | big plans | 13:54 |
| kevko | and last question ..is it ok to add repo for python3-podman for ansible collection which is installing this package ? | 13:54 |
| mnasiadka | sure | 13:54 |
| kevko | because i will need to create a ticket for debian release team and kindly ask them to include my package into stable | 13:55 |
| kevko | for now it is in unstable .. | 13:55 |
| mnasiadka | ok, coming back to the topic | 13:55 |
| mnasiadka | 3 hours times two for the PTG is enough, or do we prefer 4? | 13:55 |
| mnasiadka | (I'm fine with both) | 13:55 |
| mmalchuk | 3 hours for rabbitmq fair enough | 13:55 |
| mnasiadka | For RabbitMQ 3 weeks are not enough | 13:56 |
| mmalchuk | do we have more topics?) | 13:56 |
| * SvenKieske actually debugged some rabbit stuff earlier this day | 13:56 | |
| SvenKieske | I asked earlier today about the status of quorum queues | 13:57 |
| mnasiadka | ok, let's go for 3 hours on Mon and Tue, I'm pretty sure you'll all be sleeping on those sessions anyway :) | 13:57 |
| mmalchuk | sure | 13:57 |
| SvenKieske | I did only find out it's apparently support in oslo now, but I think we didn't implement it yet in kolla? | 13:57 |
| mnasiadka | SvenKieske: not yet in Kolla, feel free to propose ;) | 13:57 |
| mnasiadka | ok then | 13:58 |
| SvenKieske | might be a topic for PTG? or is it a no-brainer? :D | 13:58 |
| mnasiadka | probably more implementation detail than topic for PTG, we could discuss if that should be default or not :) | 13:58 |
| mnasiadka | but sure, add it to the list | 13:58 |
| SvenKieske | okay then | 13:58 |
| mnasiadka | the more topics we have, the less chance we finish in 7 minutes | 13:58 |
| mnasiadka | kevko: does the PTG timeline suit you? | 13:59 |
| SvenKieske | PTG board, btw: | 13:59 |
| SvenKieske | #link https://etherpad.opendev.org/p/kolla-caracal-ptg | 13:59 |
| mnasiadka | thanks SvenKieske | 13:59 |
| mnasiadka | ok, time is up | 13:59 |
| mnasiadka | thanks for coming :) | 13:59 |
| mnasiadka | #endmeeting | 13:59 |
| opendevmeet | Meeting ended Wed Oct 4 13:59:49 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-10-04-13.01.html | 13:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-10-04-13.01.txt | 13:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-10-04-13.01.log.html | 13:59 |
| mmalchuk | thanks mnasiadka | 13:59 |
| SvenKieske | thanks | 14:01 |
| mnasiadka | of course ptg bot does not have the 12.00 UTC slot | 14:13 |
| greatgatsby | sorry I'm reading back in the logs here and trying to understand the rabbitmq issue. We're on kolla-ansible yoga and upgraded our docker images/containers yesterday and we're seeing all kinds of rabbitmq "crashed queue" type messages. Is this related? Is there a ticket somewhere we can follow what's happening? | 14:14 |
| mnasiadka | I would suggest filing a ticket in Launchpad for Kolla | 14:16 |
| greatgatsby | kevko mentioned a "incompatibility in erlang/rabbitmq in yoga" ... just trying to understand if there's a known issue with rabbitmq and yoga right now? | 14:17 |
| mnasiadka | SvenKieske, frickler: that probably means we need to switch to 13:00 UTC or work against the official PTG schedule | 14:29 |
| frickler | well that's what I was trying to imply earlier. but who cares about the ptg bot? just add the timeframe to our etherpad and be done | 14:37 |
| mnasiadka | yeah, added that | 14:38 |
| opendevreview | Merged openstack/kolla-ansible master: Default keystone user role changed to member https://review.opendev.org/c/openstack/kolla-ansible/+/897222 | 14:58 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/2023.1: Default keystone user role changed to member https://review.opendev.org/c/openstack/kolla-ansible/+/897287 | 14:58 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/zed: Default keystone user role changed to member https://review.opendev.org/c/openstack/kolla-ansible/+/897288 | 14:58 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/yoga: Default keystone user role changed to member https://review.opendev.org/c/openstack/kolla-ansible/+/897289 | 14:59 |
| SvenKieske | well I'd care about it, because it should be discoverable to external people when we meet, so they can participate, no? | 15:09 |
| frickler | they can read the agenda and discover it, if they don't do that, they can still join an hour late IMO | 15:25 |
| kevko | greatgatsby: yeah, there is different unsupported version of erlang :D | 15:32 |
| greatgatsby | kevko: thanks for the reply, sorry for the poorly worded questions. We had a number of VMs crash/power down last night. We're seeing a lot of rabbit errors (no surprise there) but we're also seeing "Instance shutdown by itself" from nova logs during a kolla-ansible deploy. | 15:41 |
| greatgatsby | the deploy was just upgrading the kolla containers, which would have restarted most services | 15:42 |
| kevko | greatgatsby: that's exactly the same error i've seen before i fixed image | 15:48 |
| greatgatsby | what fix? Could you please point me to a ticket or something with more info? Greatly appreciated! | 15:49 |
| kevko | greatgatsby: well, i still didn't send to upstream ...but i just pinned erlang version to proper version | 15:51 |
| kevko | greatgatsby: currently i am working on a fix | 15:51 |
| kevko | greatgatsby: fix is very easy ...but i would like to provide some script which will check it automatically and fail in case it's bad | 15:51 |
| greatgatsby | I see, so this is due to the rabbitmq/erlang versions in the rabbitmq image? I'm seeing rabbitmq-server 3.9.29-1 and erlang 1:25.3.2.6-1 in ours. | 15:56 |
| greatgatsby | kevko: thanks again for the help, never would have discovered this on my own. Are you saying I need to poing erlange to 25.2? That seems to be what the rabbitmq docs show as the max erlang version? | 16:04 |
| kevko | greatgatsby: https://www.rabbitmq.com/which-erlang.html | 16:24 |
| greatgatsby | kevko: excellent, thanks. That's where I got the 25.2 version from (for 3.9.29) which does seem to indicate the 25.3 in the container is a version mismatch? Just want to make sure I have my facts straight. | 16:25 |
| kevko | greatgatsby: yep | 16:26 |
| greatgatsby | thanks! | 16:27 |
| mnasiadka | kevko: so from where do we take erlang 25.2 in Yoga? ;-) | 16:29 |
| mnasiadka | or we pin to 24 | 16:29 |
| kevko | i just created a pin to 25.2 :D | 16:30 |
| kevko | novemberrain ? | 16:30 |
| mnasiadka | no aarch64 | 16:30 |
| mnasiadka | launchpad only has latest build | 16:30 |
| mnasiadka | so that leaves us with 24 | 16:30 |
| mnasiadka | what concerns me a bit, is why the heck CI works :D | 16:31 |
| kevko | mnasiadka: so why we have ./kolla/template/repos.yaml ??? | 16:31 |
| kevko | there are separate blocks for aarch64 ? | 16:31 |
| mnasiadka | kevko: And you want to have different versions for x86 and aarch64? | 16:32 |
| mnasiadka | While we strived for cycles to have the same versions on every distro and arch? | 16:32 |
| kevko | mnasiadka: another option is to have it non-working :) | 16:32 |
| mnasiadka | erlang 24 is supported with rmq 3.9 | 16:33 |
| mnasiadka | so what's the problem? :) | 16:33 |
| kevko | CI is working ..and actually ... openstack is working ...but it is crashing from time to time ...and it's definitely rabbitmq/erlang incompatibility | 16:33 |
| kevko | i am not expert for rabbitmq/erlang | 16:33 |
| kevko | but what if you run upgrade from erlang 25 -> 24 ? | 16:33 |
| kevko | i would say that it won't be without problems :) | 16:34 |
| mnasiadka | Nobody is an expert, nobody knows :) | 16:34 |
| kevko | i am 90 percent sure that it will be problem :D | 16:35 |
| mnasiadka | well, there's 25.3 already | 16:35 |
| mnasiadka | so either way, we will downgrade | 16:36 |
| mnasiadka | ;) | 16:36 |
| mnasiadka | hmm, the same is on centos/rocky | 16:36 |
| mnasiadka | fantastic | 16:36 |
| kevko | now i don't understand | 16:37 |
| mnasiadka | we published 25.3 already | 16:37 |
| mnasiadka | so if somebody moved to 25.3 | 16:37 |
| mnasiadka | and now he gets 25.2 | 16:37 |
| mnasiadka | isn't that a downgrade? | 16:37 |
| kevko | i think .2 is less than .3 no ? :D | 16:38 |
| mnasiadka | and centos already has erlang-25.3.2.6-1.el8.x86_64 | 16:38 |
| mnasiadka | and nobody complained for months | 16:38 |
| kevko | yeah, that's the issue we are talking about :D :D | 16:38 |
| kevko | i was complaining and greatgatsby also :D | 16:39 |
| kevko | and documentation saying "maximum supported" ... | 16:39 |
| kevko | erlang/rabbitmq is very sensitive | 16:39 |
| mnasiadka | well, it's been like that for long | 16:39 |
| mnasiadka | I'm pretty sure erlang 25.3 didn't show up even this year | 16:39 |
| mnasiadka | well, 25.3 was released in March | 16:40 |
| mnasiadka | this year | 16:40 |
| mnasiadka | so probably anybody that did a deploy on a new container since march - should be observing that | 16:41 |
| mnasiadka | question why there's no bugs raised :) | 16:41 |
| kevko | when you realize how often people upgrade the openstack release, and especially rabbitmq ... 6 months is not so long that it is possible that people are not aware of this error | 16:42 |
| mnasiadka | probably | 16:42 |
| mnasiadka | I'll run a test tomorrow what happens when we downgrade ;) | 16:42 |
| kevko | well, because i haven't reported yet :D | 16:42 |
| mnasiadka | I hope downgrading from 25.3 to 25.2 won't blow out your cluster | 16:42 |
| kevko | we found it on new deployment on customer side | 16:42 |
| kevko | i've already done it | 16:43 |
| kevko | and it's fixed .. | 16:43 |
| mnasiadka | where is it fixed? | 16:43 |
| kevko | no crash, no turn off the rabbitmq ..nothing ..working like a charm several days from fix | 16:43 |
| kevko | my downstream repo and my image ... | 16:43 |
| mnasiadka | yes, but that might not be the fix for everybody, unless we drop aarch64 :) | 16:44 |
| kevko | because you know ..sometimes fix in upstream can take days ...and you have to convince many people that you are right :D | 16:44 |
| kevko | and i am : | 16:44 |
| kevko | :) | 16:44 |
| kevko | well, i think we already have different situation with aarch64 | 16:45 |
| mnasiadka | the version is the same there | 16:45 |
| kevko | i mean it's is somehow specific | 16:45 |
| mnasiadka | I know that probably nobody is using that :) | 16:45 |
| mnasiadka | You can say that out loud | 16:46 |
| kevko | so why to not focus to debian/ubuntu centos/rocky ...... to have same compatible versions and working ..and aarch64 also even if they has different version | 16:46 |
| kevko | i know that it is not ideal situation ...but ... | 16:46 |
| kevko | :D | 16:46 |
| kevko | haha | 16:46 |
| kevko | yeah , probably nobody :D | 16:46 |
| kevko | ^^^^^ then you can use jinja apt and pin to supported versions for both archs but different versions 24/25 for erlang for example | 16:47 |
| mnasiadka | just writing the patch | 16:48 |
| kevko | i also :D | 16:48 |
| kevko | AND ...i am working on script which will check every time the compatibility | 16:48 |
| kevko | we will just need to update supported matrix | 16:48 |
| kevko | and build will fail/pass based on check | 16:48 |
| kevko | what do you think about it ? | 16:49 |
| mnasiadka | actually debian aarch64 uses erlang 24 | 16:49 |
| mnasiadka | ah no | 16:49 |
| mnasiadka | I missed that | 16:49 |
| mnasiadka | kevko: script? we could think of a framework for testing for correct versions of a lot of things in the container, but that's rather PTG topic | 16:51 |
| mnasiadka | we just need to pin erlang correctly, which we have never done | 16:51 |
| kevko | Well ..something as RUN check-compat ....which will compare some table of compatibility versions | 16:52 |
| kevko | I've pinned it in my downstream right after I figured out it is a problem | 16:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions https://review.opendev.org/c/openstack/kolla/+/897352 | 16:53 |
| kevko | We had same errors as great Gatsby had | 16:53 |
| mnasiadka | I haven't seen them on any deployment, and we have some fresh ones | 16:53 |
| mnasiadka | But better to stay on the safe side with EOL RMQ | 16:53 |
| kevko | So you don't believe it ? 🤣 | 16:54 |
| kevko | *me | 16:54 |
| kevko | Why 23 ? | 16:54 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions https://review.opendev.org/c/openstack/kolla/+/897352 | 16:54 |
| mnasiadka | too late | 16:54 |
| mnasiadka | ;) | 16:54 |
| mnasiadka | kevko: I can't believe even fresh stable releases have EOL RMQ and we should do something about it | 16:55 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions https://review.opendev.org/c/openstack/kolla/+/897352 | 16:55 |
| kevko | Yep but that is another issue | 16:56 |
| mnasiadka | ok, that should be enough | 16:56 |
| mnasiadka | and let's try to solve this mess on the PTG | 16:58 |
| mnasiadka | kevko: you're going to join this time? ;) | 16:58 |
| kevko | 28 leaving to vacation on scheyschellas for 3 weeks :D | 17:00 |
| kevko | sorry - 18th | 17:00 |
| kevko | so :( | 17:00 |
| mnasiadka | great timing :) | 17:01 |
| mnasiadka | but at least try to populate some ideas on the PTG etherpad :) | 17:01 |
| kevko | i can somehow connect via mobile *maybe | 17:01 |
| mnasiadka | And have fun on vacation :) | 17:01 |
| mnasiadka | naah, vacation is vacation | 17:01 |
| kevko | but i will be on a boat during that 3 weeks ..almost all the time | 17:01 |
| kevko | do you remember my old patch for logging ? | 17:02 |
| kevko | that would be very nice to revive | 17:02 |
| mnasiadka | If you're going to convince us this time this is needed - sure :) | 17:02 |
| mnasiadka | need to finish for today, talk to you another day ;) | 17:02 |
| kevko | well - we definetly should provide option to pass own logging.conf | 17:03 |
| kevko | because by default there are several loggers in openstack ...and most of them are warns ... that's ok ...but sometimes you want to reconfigure loggers ... | 17:04 |
| kevko | especially when debugging ... | 17:04 |
| kevko | and that's not possible right now | 17:04 |
| kevko | also - we are using fluentd ..but instead of fluentlogger we are writing logs to disk and tailing by fluentd :D | 17:05 |
| kevko | which is not good .. | 17:05 |
| mnasiadka | Put that to the ptg etherpad please :) | 17:06 |
| kevko | mnasiadka: last thing -> your pin will not work i think ..because ppa has only 25.3.2 | 17:06 |
| kevko | while novemberrain has all version of erlang .. | 17:06 |
| mnasiadka | Might be, we need separate apt prefs for aarch64 | 17:07 |
| mnasiadka | Will update later | 17:07 |
| kevko | that's why I've been talking that we should use mirrors for both rabbit/erlang packages | 17:07 |
| kevko | ()[root@pixla /]# apt-cache policy erlang-base | 17:08 |
| kevko | erlang-base: | 17:08 |
| kevko | Installed: 1:25.3.2.6-1rmq1ppa1~ubuntu20.04.1 | 17:08 |
| kevko | Candidate: 1:25.3.2.6-1rmq1ppa1~ubuntu20.04.1 | 17:08 |
| kevko | Version table: | 17:08 |
| kevko | *** 1:25.3.2.6-1rmq1ppa1~ubuntu20.04.1 500 | 17:08 |
| kevko | 500 https://ppa.launchpadcontent.net/rabbitmq/rabbitmq-erlang-25/ubuntu focal/main amd64 Packages | 17:08 |
| kevko | 100 /var/lib/dpkg/status | 17:08 |
| kevko | 1:24.2.1+dfsg-1~bpo11+1 500 | 17:08 |
| kevko | 500 http://osbpo.debian.net/debian bullseye-zed-backports-nochange/main amd64 Packages | 17:08 |
| kevko | 1:23.2.6+dfsg-1+deb11u1 500 | 17:08 |
| kevko | 500 http://deb.debian.org/debian bullseye/main amd64 Packages | 17:08 |
| kevko | ()[root@pixla /]# apt download erlang-base | 17:08 |
| mnasiadka | Yoga is going EM and then EOL so a couple more months only | 17:08 |
| kevko | this is not about yoga ..but this is about rabbitmq packages | 17:09 |
| kevko | most of the time it's simple to cherry-pick fix from upstream | 17:09 |
| kevko | but ppa repo for erlang 25 don't have all versions .. | 17:10 |
| kevko | mirror has | 17:10 |
| mnasiadka | But only for x86 | 17:10 |
| kevko | yeah - but it's another issue | 17:10 |
| kevko | if we separate x86 and aarch64 ... we can control both versions | 17:11 |
| kevko | apt url / pin | 17:11 |
| mnasiadka | And I’m not keen to move other branches unless we need to | 17:11 |
| kevko | but we need it i think :D | 17:11 |
| mnasiadka | Why? | 17:12 |
| kevko | hmm, maybe i didn't understand ...you said move other branches ...what does it mean ? | 17:12 |
| mnasiadka | As in move other branches to use erlang from novemberain | 17:13 |
| kevko | i think it is broken for all branches currently | 17:13 |
| kevko | and your pin will not work ..because there is not version what you need | 17:14 |
| kevko | (erlang) | 17:14 |
| mnasiadka | Broken for all branches? How is that possible? | 17:16 |
| mnasiadka | 3.10 supports 25.3, so that works | 17:17 |
| kevko | i didn't check all ... i just wanted to say that we were using url for apt ..but we need to take into account that ppa don't have all versions ..so you don't have a control | 17:18 |
| kevko | you don't have a control which version will be installed .. | 17:19 |
| kevko | and if you add pin ... it can fail because there is no version which will fit your pinned version | 17:19 |
| kevko | (novemberrain is something different ..they have all ..so you can just pin it ) | 17:19 |
| mnasiadka | I just prefer it fails build than breaks somebody’s deployment | 17:27 |
| kevko | mnasiadka: yeah, agree ...but when it fail (beacuse of pin and not available version in ppa) ...you will just need to switch to novemberrain | 17:35 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions https://review.opendev.org/c/openstack/kolla/+/897352 | 18:42 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions https://review.opendev.org/c/openstack/kolla/+/897352 | 18:44 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/yoga: Fix data file path detection with new pip https://review.opendev.org/c/openstack/kayobe/+/896381 | 20:16 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/yoga: Fix data file path detection with new pip https://review.opendev.org/c/openstack/kayobe/+/896381 | 20:23 |
| opendevreview | Verification of a change to openstack/kayobe stable/2023.1 failed: Fix data file path detection with new pip https://review.opendev.org/c/openstack/kayobe/+/896379 | 22:56 |
| opendevreview | Merged openstack/kayobe stable/zed: Fix data file path detection with new pip https://review.opendev.org/c/openstack/kayobe/+/896380 | 23:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!