| clarkb | just about meeting time | 18:59 |
|---|---|---|
| clarkb | the agenda for today is a bit shorter than some of the recent agendas. Maybe that means we'll be able to find $meal a bit early today | 19:00 |
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Jan 30 19:00:10 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/L6CVNQQLCEZCK3I7I7FQUC7VV2UWSBHM/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | Service coordinator nominations open February 6, 2024 - February 20, 2024 | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/ | 19:01 |
| clarkb | That is one week away | 19:01 |
| clarkb | rouhgly | 19:01 |
| clarkb | OpenDev PrePTG February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service. | 19:02 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/KUZG7DPU77GC3M3MKKTSYY655JLIYV4Z/ | 19:02 |
| clarkb | and the Pre PTG is two weeks away | 19:02 |
| clarkb | #topic Server Upgrades | 19:02 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/905510 Upgrading meetpad service to jammy | 19:03 |
| clarkb | fungi: frickler maybe you have time to review those changes? Otherwise I think I may work with tonyb and proceed with things with just my reviews | 19:03 |
| clarkb | (which coincidentally is one of the agenda items for the pre ptg, whether or not we want to do more singe reviewer approvals) | 19:03 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades | 19:04 |
| clarkb | this etherpad also has notes on a wiki replacement plan. If you haven't read them that may be a good idea. | 19:05 |
| * frickler makes notes to take a look, also at those reviews | 19:05 | |
| clarkb | thanks! | 19:06 |
| clarkb | #topic Python container updates | 19:06 |
| clarkb | Nothing new to mention here. I think we're pretty much stalled out on the zuul oeprator stuff. But it isn't urgent. I may just remove this from the next meeting's agenda | 19:07 |
| clarkb | #topic AFS quota issues | 19:08 |
| clarkb | somewhere on my todo list that keeps getting overlooked is ubuntu ports cleanup. /me adds it again and maybe it will get looked at | 19:09 |
| clarkb | #topic Broken wheel build issues | 19:10 |
| clarkb | the wheels are building and being written to afs now | 19:10 |
| clarkb | The remaining problem has to do with volume selection for vos release | 19:10 |
| clarkb | #link https://review.opendev.org/c/openstack/project-config/+/907150 Will fix centos 8 and 9 stream | 19:11 |
| clarkb | In theory that change will have fixed these problems and everything centos 8/9 stream wheel related will be happy | 19:11 |
| clarkb | I did notice thatwhile centos 7 was working when this first came up it is not working now | 19:11 |
| clarkb | I haven't had a chance to look at it | 19:11 |
| clarkb | one thing I noticed is that I'm not sure these wheels were ever really working properly for stream | 19:12 |
| clarkb | that may put more weight into the "maybe we should just shut it down side of things" | 19:13 |
| clarkb | however if things work now then letting it continue to be happy is least effort | 19:13 |
| frickler | til next time it breaks ;) | 19:14 |
| clarkb | #topic Keycloak upgrade struggles | 19:14 |
| clarkb | ya exactly :) | 19:14 |
| clarkb | recently fungi and I started poking at keycloak upgrades. A little while back keycloak moved from wildfly as its runtime to quarkus. When they did this they didn't really make the container image migration a simple direct one | 19:15 |
| clarkb | this resulted in us getting stuck behind on a legacy set of images and now we are trying to figure out how to get onto the new quarkus stuff | 19:15 |
| fungi | but also some useful discoveries | 19:15 |
| clarkb | The biggest issue we've run into is that the existing service uses an h2 databse but keycloak considers h2 to not be production worthy so there is no migration path for that | 19:16 |
| clarkb | file paths and names and db credentials have all changed | 19:16 |
| clarkb | this seems to be a clear sign we should move to a real db so that is what fungi has been doing most recently. Getting mariadb up alongside keyloack | 19:16 |
| fungi | i did try some basic migration which made it into production briefly over the weekend before i rolled it back due to breakage | 19:16 |
| clarkb | ya including renaming and moving files | 19:16 |
| fungi | it would be good for someone who has successfully used keycloak.opendev.org recently to test that it's working as expected again | 19:16 |
| corvus | yep wfm | 19:17 |
| fungi | anyway, for a number of reasons, it appears that trying to "migrate" our existing data will be extremely involved (switching database format, upgrading through several keycloak releases, also we should move to latest ubuntu lts while we're at it) | 19:18 |
| tonyb | and me | 19:18 |
| fungi | given that the service is used by at most half a dozen people in a non-production-critical path right now, it will probably be less work to set it up from scratch on the latest keycloak images on new ubuntu | 19:18 |
| clarkb | ya the end result of all this wheel spinning seems to be we should do a new deployment with that we have laerned then rebuild the currently small set of integration we have using it | 19:18 |
| clarkb | then we should be in a much better position for the future and being able to upgrade things | 19:19 |
| fungi | i've got a change underway (close to complete i think) to do that | 19:19 |
| fungi | just adding some more hygiene to the testing now | 19:19 |
| fungi | like switching to group_vars and using keycloak99 for the test node, etc | 19:19 |
| fungi | tests so far are passing, i've been adding some safety like making sure keycloak and mariadb only listen on the loopback address | 19:20 |
| fungi | anyway, hopefully we can have a new keycloak02.opendev.org up by the time for our pre-ptg and start talking about moving forward some more on the sso spec | 19:21 |
| clarkb | sounds good | 19:21 |
| tonyb | +1 | 19:21 |
| fungi | i didn't have anything else for this topic, but maybe folks have questions | 19:22 |
| corvus | plan sgtm, thanks! | 19:22 |
| clarkb | #topic InMotion/OpenMetal Cloud RabbitMQ Problems | 19:23 |
| clarkb | wanted to follow up on this. frickler's hunch was correct. rabbitmq had a sad | 19:23 |
| clarkb | tonyb: can speak more to this than I can but initially a simple docker restart of rabbitmq containers was done to try and get rabbit to cooperate and that didn't help at all | 19:23 |
| clarkb | then later a restart using kolla tooling was done and that got things working again | 19:24 |
| clarkb | however there are still a number of leaked nodes we need to clean up which likely involve deeper admin work | 19:24 |
| tonyb | Yup that one. | 19:24 |
| clarkb | one thing I have had to do in the past to make nodes deletable is to reset their task state if they are already in a deleting task state because nova won't retry if task state is already deleting | 19:24 |
| clarkb | then there is another issue we've encountered which had to do with a mismatch in cell accounting | 19:25 |
| clarkb | melwitt helped the cells issue if we need help again | 19:25 |
| tonyb | I'll get more details and then reach out on nova/melwitt | 19:26 |
| tonyb | in the next 24-48 hours e should have things "sorted" | 19:26 |
| clarkb | thanks! | 19:27 |
| clarkb | #topic Open Discussion | 19:27 |
| clarkb | That was all I had on the agenda. A reminder to add your content and interest to the pre ptg etherpad | 19:27 |
| corvus | #link https://etherpad.opendev.org/p/opendev-preptg-202402 | 19:29 |
| clarkb | sounds like that may be everything | 19:31 |
| clarkb | a rare short meeting. THank you for your time everyone! | 19:31 |
| clarkb | I'll close this out in a few more minutes if nothing else pops up | 19:31 |
| fungi | thanks clarkb! | 19:31 |
| tonyb | Thanks all! | 19:31 |
| corvus | thanks clarkb | 19:33 |
| clarkb | #endmeeting | 19:35 |
| opendevmeet | Meeting ended Tue Jan 30 19:35:21 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:35 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2024/infra.2024-01-30-19.00.html | 19:35 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-01-30-19.00.txt | 19:35 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2024/infra.2024-01-30-19.00.log.html | 19:35 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!