| clarkb | meeting time | 19:00 |
|---|---|---|
| clarkb | #startmeeting | 19:00 |
| opendevmeet | clarkb: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 19:00 |
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Mar 18 19:00:47 2025 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/YM5MZF2IHG6P4FFTRVMNVLJHYOIBVFUD/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:02 |
| clarkb | Anything to announce? | 19:02 |
| clarkb | I dno't have any concrete plans but next week the kids are home from school for a break and I may try to do an easy day or two and get out of the house with them | 19:02 |
| tonyb | FWIW, next week 25th I head back to AU | 19:05 |
| clarkb | ack | 19:05 |
| clarkb | #topic Zuul-launcher image builds | 19:05 |
| clarkb | I'm not aware of any new updates on this subject | 19:05 |
| clarkb | though if corvus is around I'd be curious to know if the quota handling has improved the reliability | 19:06 |
| fungi | also i've still got a change up to add flex dfw3 images/quota | 19:06 |
| clarkb | oh do you have a link to that? | 19:06 |
| fungi | #link https://review.opendev.org/943104 Add the DFW3 region for Rackspace Flex | 19:06 |
| clarkb | thanks! | 19:07 |
| clarkb | I'll review that after the meeting | 19:07 |
| clarkb | #topic Updating Flavors in OVH | 19:08 |
| clarkb | The other related item was the OVH flavor update process | 19:08 |
| clarkb | #link https://etherpad.opendev.org/p/ovh-flavors | 19:08 |
| clarkb | we proposed starting that yesterday but they came back and said that timing didn't work for them. They will be in touch with timing that does work at some point but I haven't seen any proposed dates from their end | 19:08 |
| clarkb | we can probably pull this off of the agenda next week. But I wanted to make sure we called out it wasn't happening and we're waiting on timing from ovh | 19:09 |
| clarkb | #topic Container hygiene tasks | 19:09 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/944799 drop python 3.10 image builds | 19:09 |
| clarkb | according to codesearch we don't have anything using python 3.10 iamges anymore | 19:10 |
| clarkb | I think this is a safe change to land at any time | 19:10 |
| clarkb | next up is rebuilding the images we do use | 19:10 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/944789 rebuild python 3.11 and 3.12 base images | 19:10 |
| clarkb | unfortunately this change has been in uWSGI purgatory | 19:10 |
| clarkb | it seems that compiling current uwsgi on top of current bookworm on aarch64 segfaults | 19:11 |
| clarkb | uwsgi is only barely maintained anymore so I've begun looking at alternatives (which is the next topic) | 19:11 |
| clarkb | I don't think we can safely udpate python-builder and python-base without also updating uwsgi-base as the wheel builds for things will see mismatched binary packages and we could break things relying on uwsgi today | 19:11 |
| clarkb | so I think our main options here are either to drop uwsgi, fix uwsgi builds somehow, or retry until we get lucky? (they failures don't seem to be 100% consistent) | 19:12 |
| clarkb | maybe it is worth trying to enqueue to the gate a few times to see if we can get it to go through | 19:12 |
| clarkb | then finally I've got changes up to move images from python 3.11 to 3.12 | 19:13 |
| clarkb | #link https://review.opendev.org/q/topic:%22opendev-python3.12%22+status:open Update images to use python3.12 | 19:13 |
| frickler | I didn't look at details yet, but could we use distro pkgs like devstack does? | 19:13 |
| clarkb | while those changes don't strictly depend on the new image updates it might be nice to use these rebuilds to also get new base images so we address two things with one set of service restarts | 19:13 |
| clarkb | frickler: the container images are based on debian bookworm. If there is a bookworm uwsgi package then yes I think that is possible | 19:14 |
| clarkb | we would add uwsgi to lodgeit's bindep requirements and then switch the base container image from uwsgi-base to python-base | 19:14 |
| fungi | #link https://packages.debian.org/bookworm/uwsgi | 19:14 |
| clarkb | this will likely downgrade uwsgi for us but we aren't doing anything too crazy with it so thats probably fine (also uwsgi itself isn't super stateful) | 19:14 |
| frickler | I think that would likely only work with distro python. so it might work for py3.11 then and we'd have to use noble for py3.12 | 19:15 |
| clarkb | actually it wouldn't work for either if that is the cas | 19:15 |
| clarkb | since both 3.11 and 3.12 are custom builds on top of debian | 19:15 |
| clarkb | another option would be to try and pin uwsgi versions to something older than latest to see if we can get that to build more reliably | 19:16 |
| frickler | and we do need custom python builds? | 19:16 |
| clarkb | it dramatically simplifies supporting a range of pythons | 19:16 |
| clarkb | since we can use the upstream python container images and not need to have different platforms for different python versions or even wait for new distro releases | 19:17 |
| clarkb | I don't think uwsgi is important enough to warrant dramatically changing how we run python services | 19:17 |
| clarkb | it is basically unmaintained and can't be built on modern platforms reliably | 19:18 |
| clarkb | if we can get by with simple fixes to uwsgi that is fine. But I would rather invest my time in replacing uwsgi than completely rearchitecting our container setup | 19:18 |
| frickler | ack | 19:18 |
| clarkb | which is maybe a good segue into the next topic. Reviwes on the above changes are very much appreciated too | 19:19 |
| clarkb | #topic Dropping uWSGI | 19:19 |
| * corvus arrives late | 19:19 | |
| clarkb | there are a number of factors have me thinking uWSGI is no longer a viable option for python web servers. First is that the project is minimally maintained and bug fixes are slow and build errors are common. Next is the python world is becoming more async which often means ASGI instead of WSGI | 19:20 |
| clarkb | rather than try and keep the old barely working thing going I suspect a better use of our time is looking forward and picking a modern tool that is maintained and can support asgi and wsgi | 19:20 |
| clarkb | in my investigation I discovered granian which seems to fit this criteria. However, frickler and others have pointed out granian is maintained laregly by one person (there are a number of contributors but the vast majority of work is one person) and there is no distro packaging for it | 19:21 |
| clarkb | I think that the lack of distro packaging is less meaningful for us as we install from pypi and they publish x86 and arm wheels | 19:21 |
| clarkb | #link https://review.opendev.org/c/opendev/lodgeit/+/944805 lodgeit granian container update | 19:22 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/944806 lodgeit system-config deployment update | 19:22 |
| corvus | we only use uwsgi for lodgeit, right? | 19:22 |
| clarkb | corvus: the uwsgi-base container image is lodgeit only. mailman3 also uses uwsgi but not via the base image | 19:22 |
| clarkb | anyway I'm open to other alternatives we think might be more appropriate, but I do think we should try and stop using uwsgi | 19:23 |
| corvus | my feeling is -- for something like lodgeit -- whatever works and requires the least time. if granian works, that's good enough for me; i'd be in favor of adding a granian image and declaring uwsgi unmaintained by us | 19:24 |
| fungi | in the mm3 case uwsgi is part of upstream images we're reusing (well, rebuilding), right? | 19:24 |
| clarkb | fungi: yes | 19:24 |
| fungi | in which case we can probably wait for or help with upstream's decisions on whether to continue relying on it | 19:24 |
| clarkb | ya I think being in sync with them and possibly starting a conversation with them about alternatives is a good thing and also less urgent | 19:25 |
| clarkb | also I think our needs differ from openstack's needs. While it might be more important for openstack to use distro packaged wsgi servers that isn't as important for us | 19:25 |
| frickler | can we simply re-use the mm3 container and install lodgeit in there? or is that setup too different? | 19:26 |
| clarkb | I'm not sure. But it is fairly different (uses alpine instead of debian, and has django installed) | 19:26 |
| fungi | https://github.com/maxking/docker-mailman is the mailman containers. they're arranged a bit differently from how we usually do our container images | 19:26 |
| tonyb | I think that granian is a good step forward and the single maintainer is better that no maintainer. The switch to granian looks pretty nice | 19:26 |
| clarkb | gunicorn was also mentioend in discussion in the tc channel. But gunicorn doesn't do asgi. uvicorn was mentioned but that is a separate code base aiui and doesn't do wsgi | 19:27 |
| clarkb | anyway I noted on the change why I thought granian was a good option and I think being able to switch to gunicorn or similar later means this si fairly low risk | 19:28 |
| tonyb | ++ | 19:28 |
| frickler | iiuc our uwsgi build issues were on aarch64, too? https://github.com/maxking/docker-mailman/blob/main/web/Dockerfile#L20-L21 | 19:29 |
| clarkb | frickler: aha thats a good find | 19:29 |
| clarkb | I can pin our uwsgi image to that version and see if it works | 19:29 |
| clarkb | and then we can migrate away from uwsgi on a less urgent schedule | 19:29 |
| fungi | corvus summed up my position quite nicely, if it works with minimal effort, then great, lodgeit hopefully shouldn't eat a ton of our admin time | 19:29 |
| clarkb | ya so I think plan is try that workaround from mm3, if that works land it. Concurrently we can make a plan to migrate lodgeit (as written the chagnes I have require coordination between container and system-cofnig so we may have an outage) | 19:30 |
| frickler | ack, I wasn't aware that this is "only" for lodgeit, I'd withdraw my -1 then | 19:30 |
| clarkb | if the workaround doesn't work we can speed up the granian switch. If the workaround works then we can be a bit more cautious and double check for alternatives first | 19:30 |
| clarkb | frickler: oh cool | 19:30 |
| clarkb | that gives me a path forward I'll work on that | 19:31 |
| clarkb | thanks for listening to me on this subject | 19:31 |
| fungi | and yeah, i did a search for uwsgi in the open/closed issues and pull requests for the mailman containers and saw they'd been doing work recently to get it working for aarch64 | 19:31 |
| fungi | so potentially related | 19:31 |
| frickler | ah, yes, git blame shows this https://github.com/maxking/docker-mailman/pull/743, so rather recent | 19:32 |
| clarkb | #topic Upgrading old servers | 19:33 |
| clarkb | I should probably just merge this and the sprint idea topics together | 19:33 |
| tonyb | clarkb: Yeah it seems to be basically the same topic at this point | 19:33 |
| clarkb | I built nb05 and nb06 yesterday and got them deployed | 19:34 |
| clarkb | they have built every image but bookworm, centos 9 stream, gentoo, and openeuler. The last two are paused and don't build. Just before the meeting I requested rebuilds of the first two | 19:34 |
| clarkb | everything is looking ok to me so far. I'll plan to clean out the old servers late this week when we're confident we won't need them anymore | 19:34 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/944867 fix nodepool image export cron | 19:35 |
| clarkb | this is a related fix for a change I made to be backward and forward compatible between docker-compose and docker compose | 19:35 |
| clarkb | we should land that and make sure we are exporting things properly before deleting the old servers too | 19:35 |
| clarkb | frickler: ^ you had a concern about updating cron's PATH instead but I don't think that is necessary as docker-compose should be a temporary shim to make things compatible between old and new systems | 19:36 |
| clarkb | oh also good news is the nested container stuff to build rocky linux images seems to work fine | 19:37 |
| clarkb | anyway I'm going to keep working on server replacements as time goes on and help is very much welcome | 19:37 |
| clarkb | anyone else have updates on this topic? | 19:37 |
| frickler | yes, I was only worried about what happens on nb04, but if it works there, I'm fine | 19:37 |
| fungi | there was also some error you spotted on the new servers, frickler? | 19:38 |
| clarkb | it should. on the old servers we pip install docker-compose which puts the executable in /usr/local/bin/docker-compose and on noble and newer we write out a shim for docker-compose to docker compose that we write to /usr/local/bin/docker-compose | 19:38 |
| clarkb | fungi: its the same error on both I think. /usr/local/bin isn't in cron's path so we don't find the pip installed executable or our shim | 19:39 |
| clarkb | your fix uses the rooted path and should fix it for both I think | 19:39 |
| fungi | "/etc/nodepool-builder-compose/docker-compose.yaml: `version` is obsolete" | 19:39 |
| clarkb | oh thats a warning | 19:39 |
| clarkb | `docker compose` doesn't use versioned docker-compose.yaml files but `docker-compose` does | 19:39 |
| fungi | okay, so benign then, but something we can clean up after the transition | 19:39 |
| clarkb | ya | 19:39 |
| frickler | ack | 19:40 |
| clarkb | #topic Running certcheck on bridge | 19:41 |
| clarkb | I haven't had a chance to look into running this out of an infra-prod job | 19:41 |
| clarkb | but I still intend to as I think that is likely a good way to do it | 19:41 |
| clarkb | #topic Working through our TODO list | 19:42 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-january-2025-meetup | 19:42 |
| clarkb | I marked parallel infra-prod job execution done on that list | 19:42 |
| clarkb | and the python 3.12 effort from earlier is also out of that list | 19:42 |
| clarkb | feel free to pick things off of their and work on them when you have time | 19:43 |
| clarkb | #topic Packaging updates for bindep | 19:43 |
| clarkb | fungi has two proposed changes to bindep that serve as examples for modernizing python packaging with pbr | 19:43 |
| clarkb | #link https://review.opendev.org/938570 Drop requirements.txt | 19:43 |
| fungi | just wanted to touch base quickly on those two remaining changes | 19:43 |
| clarkb | #link https://review.opendev.org/940711 Drop auxiliary requirements files | 19:43 |
| fungi | and either merge or abandon them | 19:43 |
| clarkb | I'm happy to reduce the delta between us and upstream PyPA expectations and this is a good step in showing people using PBR how to do that so +2 from me | 19:44 |
| fungi | once we have a decision one way or the other, we'll tag a new bindep version and then i can work on porting the various packaging updates from bindep to our other tools | 19:44 |
| fungi | ideally i'd like to get a bindep release out this week | 19:45 |
| frickler | I can take a closer look at those tomorrow if you want to wait for that | 19:45 |
| fungi | but since they're stylistic changes for packaging to serve as a possible template for our other tools and wider pbr user base, i want to be sure we have consensus | 19:45 |
| fungi | yeah, tomorrow would be great | 19:45 |
| fungi | this came about in part because of problems openstack was facing using pbr, so it would be great to have a real-world example to point them to instead of something fabricated | 19:46 |
| fungi | anyway, that's all i had for this topic | 19:47 |
| clarkb | #topic Open Discussion | 19:47 |
| clarkb | as mentioned I marked parallel infra-prod deplyoments done. This has dramatically sped up our deploy buildsets in many situations | 19:47 |
| clarkb | thank you to everyone who helped get that over the finish line | 19:48 |
| corvus | what's the mutex at? | 19:48 |
| clarkb | 4 | 19:48 |
| frickler | a generic certcheck job in zuul-jobs might also be interesting for other #zuul users, maybe ask if someone there is interested in helping? | 19:48 |
| clarkb | there is potential to bump that up but I think we'll see diminishing returns | 19:48 |
| corvus | thinking of increasing it more? | 19:48 |
| corvus | ok | 19:48 |
| clarkb | mutix of 1 ran periodic in 2 hours or so (usually a little over). Mutex of 2 ran in about 1 hour. Mutex of 4 got it to 40 minutes | 19:49 |
| fungi | watching the load on bridge we could almost certainly increase it, but there are coalescent event horizons like the letsencrypt job which would need to be refactored to take advantage of much more parallelism | 19:49 |
| clarkb | bumping to 6 I think we'd only get to 35 minutes at best and then ya ^ | 19:49 |
| clarkb | I'm happy to increase it and see what happens if people would like to do that | 19:49 |
| clarkb | but it doesn't seem as urgent as the prior increases | 19:49 |
| corvus | meh. sounds lie time might be better spent thinking about the next blocker | 19:50 |
| clarkb | I guess if anyone feels strongly about it or wants to experiment push up a change. I don't think anyone will object | 19:50 |
| frickler | if we're voting I'd rather keep it conservative and stick to 4 | 19:50 |
| corvus | is there a reason to be conservative? | 19:50 |
| fungi | i have a change open documenting how i prepped the openstack-discuss ml to switch it to moderating new subscribers by default: | 19:50 |
| fungi | #link https://review.opendev.org/944893 docs: Switch a mailing list to default moderation | 19:50 |
| fungi | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/L4OG3TJ5JBVS4IS7KCQKXER736PWEITB/ [administrivia] Recent change in moderation for new subscribers | 19:51 |
| corvus | (it sounds like the only reason not to do it is because it won't actually help, and we'll just be waiting on other resources outside the mutex) | 19:51 |
| clarkb | the main reason I can think of is limiting blast radius if we decide we need to shut off ansible runs | 19:52 |
| clarkb | but the speedup tradeoffs if there would be worth it imo | 19:52 |
| clarkb | and if not there then ya not much incentive | 19:52 |
| clarkb | Gerrit meets is happening at 00:00 UTC between tuesday and wednesday. That is in just over 4 hours | 19:53 |
| clarkb | if you want to participate they stream it to gerritforge's youtube channel | 19:53 |
| clarkb | they will discuss gerrit caches and I intend on listening in and sending questions (likely to discord) | 19:53 |
| clarkb | also the openinfra foundation is putting together a newsletter for the end of march and wants to spotlight openduev | 19:53 |
| clarkb | draft work is going in https://etherpad.opendev.org/p/opendev_newsletter | 19:54 |
| clarkb | feel free to add ideas or write something if you are intested or have somethign you want to communicate in that format. I'll probably draft things early next week if no one else beats me to it | 19:54 |
| frickler | fungi: btw. did you check the load on wiki before rebooting? just wondering whether the issue really was related to that | 19:55 |
| fungi | frickler: i didn't, but in the past have observed that the openid login problem persists even after load dies down | 19:55 |
| fungi | it's probably restored by restarting apache and mariadb/mysql, guessing there's something going on between them that gets stuck | 19:57 |
| fungi | but a reboot takes care of all of that | 19:57 |
| frickler | ah, ok, maybe next time I'm confident enough to try that on my own, then | 19:58 |
| fungi | feel free | 19:58 |
| fungi | we're really trying not to waste too much of our time on it, and would instead prefer to spend it reviewing tonyb's replacement work | 19:58 |
| frickler | well this morning a preferred a running wiki with broken login to a possibly completely broken one after a reboot | 19:58 |
| fungi | fair enough | 19:59 |
| clarkb | fwiw the wiki did work for me yseterday when prepping the agenda | 19:59 |
| clarkb | so whatever broke it occured between about 00:00 and when frickler noticed it | 19:59 |
| fungi | same, i edited the agenda during my afternoon yesterday as well | 19:59 |
| clarkb | and we are at time. Thank you everyone! | 19:59 |
| fungi | thanks clarkb! | 19:59 |
| frickler | actually bauzas noticed, but anyway ;) | 19:59 |
| clarkb | we'll be back here same time and location next week | 19:59 |
| clarkb | feel free to continue discussion in our normal comms channels | 20:00 |
| clarkb | #endmeeting | 20:00 |
| opendevmeet | Meeting ended Tue Mar 18 20:00:13 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2025/infra.2025-03-18-19.00.html | 20:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2025/infra.2025-03-18-19.00.txt | 20:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2025/infra.2025-03-18-19.00.log.html | 20:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!