| clarkb | hello everyone | 19:00 |
|---|---|---|
| clarkb | the meeting will get started shortly | 19:00 |
| fungi | ohai | 19:00 |
| tonyb | o/ | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Aug 1 19:01:35 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TGHMCISZUPXZ6QOAOOXRSAIHN6WYKPP4/ Our Agenda | 19:02 |
| clarkb | #topic Announcements | 19:03 |
| clarkb | A reminder that I won't be able to attend next week due to travel | 19:03 |
| clarkb | I also looked at service coordinator things and it appears I did not propose an election time frame for the next election during the last one | 19:04 |
| * tonyb has no idea how to parse that statement | 19:04 | |
| clarkb | we are theoretically supposed to run elections every 6 months and I have tried to list dates for the next period during the current election | 19:05 |
| clarkb | last january/february I did not do this | 19:05 |
| clarkb | Doing a direct january 31 - Feburary 14, 2023 + 6 months set of math I think the nomination period would open today and end in two weeks. | 19:06 |
| tonyb | Oh okay, I think I get it | 19:06 |
| clarkb | I'm happy to do that though for selfish reasons would prefer to delay by a week | 19:06 |
| clarkb | Delaying would allow me to get emails out on time and travel without being in the middle of nomination period | 19:07 |
| fungi | i'm in favor of time travel, yep | 19:07 |
| fungi | or time and travel separately, whatever works | 19:08 |
| tonyb | FWIW I'm fine with a delay. | 19:08 |
| clarkb | :) cool in that case I'll send emails to make August 8 to 22 as a nomination period with a week long voting period afterwards should that be necessary | 19:08 |
| fungi | thanks! | 19:08 |
| clarkb | #topic Bastion Host | 19:09 |
| clarkb | #link https://review.opendev.org/q/topic:bridge-backups | 19:09 |
| clarkb | This stack needs reviews. Other than that I think things are going well with the bastion | 19:09 |
| clarkb | #topic Mailman 3 | 19:09 |
| fungi | two steps forward, one step back | 19:09 |
| fungi | while waiting for reviews on the open topic:mailman3 changes, i was going to do a new held node and run through more imports to make sure the import process isn't broken by the new mailman releases | 19:10 |
| fungi | however that timed out, then a recheck failed | 19:10 |
| fungi | mailman rest api wasn't starting up, on the second failed node i found tracebacks in the container logs | 19:10 |
| fungi | story short, a recent importlib_resources release removed a bunch of deprecated stuff, breaking mailman-core | 19:11 |
| fungi | i proposed a new change to pin back importlib_resources<6 and restacked the other changes on top of that | 19:11 |
| clarkb | fungi: was that sufficient to get a held node and get back onto the original plan? | 19:12 |
| fungi | i think so, will know shortly | 19:12 |
| fungi | in better news, i added a section to the migration pad detailing the manual django site/mail host creation steps | 19:12 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/890220 pin importlib_resources for mm3 | 19:12 |
| fungi | #link https://etherpad.opendev.org/p/mm3migration "Add Django Site and Postorius Mail Host" | 19:12 |
| fungi | once i get a good held node, i'll do those manual steps on it and then try to run through some imports from production data again | 19:13 |
| clarkb | and that will be on top of the upgraded version right? | 19:13 |
| fungi | but in the meantime, assuming those changes pass latest testing, they should still be ready to merge | 19:13 |
| fungi | correct | 19:13 |
| fungi | since we decided we wanted to upgrade before scheduling the remaining imports | 19:14 |
| clarkb | sounds good, thank you | 19:14 |
| clarkb | ++ | 19:14 |
| fungi | so i just want to be sure it will go as smoothly as the previous imports | 19:14 |
| fungi | but the topic:mailman3 changes still need reviews if anyone gets a spare moment or two | 19:14 |
| clarkb | #topic Gerrit Updates | 19:16 |
| clarkb | As planned last week we (mostly fungi) managed to push some of this forward. We are running on Gerrit 3.7.4 images with updated jeepyb now. During the restart process we cleared out the replication plugins' on disk waiting queue and the plugin seemed fine starting back up that way | 19:17 |
| clarkb | fungi: do we have confirmation yet of happy lp bug updates? | 19:17 |
| fungi | i haven't checked, and nobody's said | 19:18 |
| clarkb | ack | 19:19 |
| clarkb | The only other remaining gerrit todos are deciding if we want to do anything different with the replication plugin (I think what we've got now is a decent on the fly workaround actually) and a start towards a 3.8 upgrade | 19:20 |
| clarkb | neither of which are super urgent | 19:20 |
| clarkb | #topic Server Upgrades | 19:21 |
| clarkb | I have nothing new on this topic | 19:21 |
| clarkb | The last week has generally been full of real world distractions. Both good and bad :) | 19:21 |
| tonyb | I got distracted with python and zuul mirror updates | 19:21 |
| clarkb | sound slike w ecan go to the next topic then | 19:22 |
| clarkb | #topic Fedora Cleanup | 19:22 |
| clarkb | I noticed some discussion around general base job cleanup which I suspect is related to the fedora clenaup work | 19:23 |
| tonyb | I've been working on the zuul side of this | 19:23 |
| frickler | devstack just dropped fedora support in master fwiw | 19:23 |
| tonyb | I'll publish something this week | 19:23 |
| tonyb | nothing super interesting | 19:24 |
| clarkb | ok will look forward to reviewing those changes | 19:24 |
| clarkb | #topic Gitea 1.20 Upgrade | 19:24 |
| clarkb | They are moving faster than I can keep up right now | 19:25 |
| clarkb | there is a 1.20.2 release already | 19:25 |
| tonyb | yikes | 19:25 |
| clarkb | I need ot update my change to 1.20.2 and hold a node to figure out access log file locations so that we can cross check the changes they made to that log file befor ethey end up in production | 19:26 |
| clarkb | unfortunately I don't think any of these releases hvae done anything to make oauth2 disablement easier to configure | 19:26 |
| clarkb | But I guess it is a good thing they ar efixing bugs generally | 19:26 |
| clarkb | #topic Etherpad 1.9.1 upgrade | 19:27 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/887006 Etherpad 1.9.1 | 19:27 |
| clarkb | I feel like this is ready but it would be good to have others check the held system works for them too | 19:27 |
| clarkb | the comments on that change have the held node ip and ianw appears to have checked it. Thank you ianw | 19:28 |
| clarkb | if another core can check it out we can schedule a day to land and monitor that change? | 19:28 |
| clarkb | let me know if you think it looks good and I'll try to find a day that seems safe for that (but I'v egot ~6 days left on my trip that aren't consumed by flying so I'm in crunch time for doing things here too) | 19:30 |
| clarkb | #topic Python container image updates | 19:30 |
| clarkb | the changes to update irc bots did land iirc and as far as I can tell the bots are all still functional? | 19:30 |
| tonyb | I think so :-) | 19:31 |
| tonyb | I'll get back to that after the zuul stuff | 19:31 |
| tonyb | as discussed last time I'll target as much as possible to be on bookworm 3.10 | 19:31 |
| clarkb | at this point moving consumer images to the new stuff then cleaning up the old stuff should be mostly smooth sailing at least as far as the base image is concerned. We may still need sort out $service on newer python problems | 19:32 |
| tonyb | yeah. it looks like there are many that will be very simple | 19:33 |
| tonyb | ptgbot will need small amounts of work | 19:33 |
| tonyb | gotta find a core for that project :-P | 19:33 |
| clarkb | zuul and nodepool should be easy transitions and will allow them to clean up backported package installs | 19:34 |
| clarkb | but ya it will be an iterative process to get through the list | 19:34 |
| tonyb | yup. | 19:34 |
| clarkb | #topic Meetpad LE Cert Update | 19:34 |
| fungi | we're 12 days out from expiration, fwiw | 19:35 |
| clarkb | I saw frickler and ianw discussing and debugging this. It appears that some sort of cache is preventing a new DNS verification key from being issued which prevents our ansible machinery from triggering appropriately? | 19:35 |
| frickler | ianw and me noticed that this is an interesting edge case not currently handled properly in driver.sh, yes | 19:35 |
| clarkb | frickler: can you give us a quick overview of that edge case? | 19:36 |
| frickler | essentially what you wrote, the dns-01 is still valid from a previous attempt, so acme.sh actually issues a new cert at the first stage ("issue" iirc) | 19:37 |
| frickler | while driver.sh only expects some dns auth verification record to be produced at that point | 19:37 |
| clarkb | aha | 19:37 |
| frickler | we can retry in a couple of days to see if the old auth expired by then | 19:37 |
| clarkb | maybe wait until 7 days before expiry and if the cache is still stale we can manually copy the cert and restart the docker services? | 19:38 |
| frickler | other option are either generation a new key or changing the cert content like adding a different hostname, meetpad02 maybe | 19:38 |
| frickler | or doing the manual path, right | 19:38 |
| clarkb | I wonder if changing the order of the names in the cert would do it | 19:39 |
| clarkb | that might be an easy option, if that fails then do the manual copy | 19:39 |
| clarkb | mostly thinking that we can debug acme and improve it on a longer time frame than 12 days | 19:39 |
| fungi | as far as avoiding it in the future, do we have the ability to pick the validity period for those tokens? | 19:40 |
| clarkb | so decoupe that from making the service happy again | 19:40 |
| frickler | I don't think we can choose an interval there | 19:40 |
| frickler | also it likely isn't easy to intentionally trigger the current situation | 19:40 |
| clarkb | ya so fixing the service is a higher priority than fixing our acme.sh integration | 19:41 |
| clarkb | our own evidence would indicate this is rare | 19:41 |
| frickler | root cause seems to have been an internal failure at LE at the initial renew attempt | 19:41 |
| clarkb | neat | 19:41 |
| clarkb | my suggestion is wait until ~7 days before expiry. Attempt reissue automatically and if that fails manually copy the file and restart containers | 19:42 |
| fungi | wfm | 19:42 |
| frickler | ack | 19:42 |
| clarkb | #topic Open Discussion | 19:42 |
| clarkb | I think fungi wanted to bring up matrix room and space creation on our homeserver | 19:43 |
| fungi | the starlingx community is interested in making use of our matrix homeserver | 19:43 |
| fungi | sounds like they may want as many as 20 channels (some general discussion channels, per-project channels, separate channels for gerritbot) | 19:43 |
| ildikov | o/ | 19:44 |
| ildikov | more along the lines of 12, but yes, the idea is to have rooms/channels per project team | 19:44 |
| fungi | er, "rooms" i guess they're termed in matrix parlance. and with the number of rooms they're thinking about, they also want to know if grouping those with a "space" makes sense | 19:44 |
| clarkb | I'm not sure we're in a spot to say using a space makes sense since we've not done it before | 19:45 |
| clarkb | but we can certainly put the rooms in a space and find out | 19:45 |
| fungi | also there was some question as to the creation workflow. for rooms (and spaces) on our opendev.org homeserver, that's only doable by our matrix admin account, so requests for new rooms would need to come to one of us | 19:45 |
| tonyb | is there an API we can use later? | 19:46 |
| clarkb | tonyb: yes I think so | 19:46 |
| tonyb | okay. that's nice | 19:46 |
| ildikov | yeah, I tried to click around, but I can only create a room on matrix.org | 19:47 |
| clarkb | it is intentional to restrict what rooms can be created | 19:47 |
| clarkb | er who can create rooms | 19:48 |
| fungi | anyway, it sounds like there's no objection to hosting multiple rooms and a space for the starlingx community if they decide that's what they want | 19:48 |
| clarkb | but as mentioned we could potentially automate the actual creation after writing a tool to do it from reviewed inputs | 19:48 |
| tonyb | https://element.io/blog/spaces-the-next-frontier/ also introduces subspaces | 19:49 |
| clarkb | I don't have any objections from a service hosting standpoint. My only concern is as a user I get frustrated when projects have a bunch of channels nad they are all dead or say not my problem | 19:49 |
| clarkb | Openstack is particularly bad about this for example | 19:49 |
| fungi | yeah, we can definitely provide them with feedback/recommendations that room proliferation can lead to user confusion | 19:49 |
| ildikov | people in the StarlingX community are currently saying that they are uncomfortable throwing every topic into one channel on IRC | 19:50 |
| ildikov | so for them it seems more appealing to have channels/rooms for more focused discussions, and we landed on grouping by project teams | 19:50 |
| tonyb | I guess all we can do is explain the options that we can host/manage and let the community decide | 19:50 |
| corvus | yes i think a space would be appropriate | 19:50 |
| fungi | ildikov: sounds like you can let me know what the names are they want (now that i've figured out where to create those). with that many rooms they probably need to pick a common name prefix for clarity, like #starlingx-whatever or #stx-whatever | 19:51 |
| clarkb | ildikov: yes I think those optimizations prioritize long term core developers over new contributors or people trying to debug a problem. Its a choice and one I personally dislike but the homeserver can handle X rooms just fine | 19:51 |
| fungi | we can also start out with just one so they can try it out, i suppose | 19:51 |
| fungi | whatever works better | 19:51 |
| ildikov | clarkb: the plan is to have a general channel, where new people can bring up any topic, etc | 19:52 |
| tonyb | ++ | 19:52 |
| clarkb | fungi: the initial test room we made cleaned up just fine so Ithink we can go ahead and creat ethem all | 19:52 |
| clarkb | if we need to they can be cleaned up later | 19:52 |
| ildikov | we've tried the one channel option through IRC, but people didn't really get into it | 19:52 |
| tonyb | so the homeserver will be OpenDev.org | 19:53 |
| ildikov | @fungi I'll get that info to you | 19:53 |
| ildikov | tonyb: +1 | 19:53 |
| tonyb | and There will be a stx space | 19:53 |
| tonyb | with several rooms? | 19:53 |
| fungi | yes | 19:53 |
| fungi | where several is maybe a dozen | 19:54 |
| tonyb | yup | 19:54 |
| clarkb | tonyb: that was my take away. And fungi's idea to use a consistent room prefix in #opendev like starlingx or stx is also a good idea imo | 19:54 |
| ildikov | I would call the space StarlingX, and yes, several rooms, I think there are currently 11 project teams, I'll double check with the team leads who objects to having a room | 19:54 |
| fungi | mainly, i think "spaces" don't operate like room namespaces, so you still end up needing to namespace the rooms themselves for clarity | 19:54 |
| tonyb | yeah, depending how the space and room names interact | 19:54 |
| ildikov | clarkb: +1, that's how I wrote up the proposal to the community | 19:54 |
| tonyb | cool beans | 19:55 |
| corvus | spaces are just groups of rooms, no namespacing | 19:55 |
| tonyb | okay | 19:55 |
| corvus | (also, users can make their own spaces, server-hosted spaces are basically just suggestions of how users can discover rooms -- which is appropriate here) | 19:55 |
| corvus | room logging and gerrit bot announcements are available; the only service we're missing in matrix is meetbot | 19:57 |
| ildikov | corvus: that's great info, thank you for sharing | 19:57 |
| fungi | yeah, i dug up the links to the right files for the bots and pasted them in #opendev earlier but i'll try to collect all this into a rudimentary document | 19:57 |
| ildikov | fungi: +1, I'm happy to help | 19:59 |
| clarkb | sound slike a plan | 19:59 |
| clarkb | and we are at time. Thank you everyone! | 19:59 |
| fungi | thanks clarkb! | 19:59 |
| tonyb | thanks clarkb | 20:00 |
| clarkb | reminder I can't host a meeting next week | 20:00 |
| tonyb | thanks all | 20:00 |
| clarkb | but happy for osmeone else to chair without me | 20:00 |
| clarkb | #endmeeting | 20:00 |
| opendevmeet | Meeting ended Tue Aug 1 20:00:12 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-08-01-19.01.html | 20:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-08-01-19.01.txt | 20:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-08-01-19.01.log.html | 20:00 |
| ildikov | thank you all!! | 20:00 |
| corvus | ildikov: https://zuul-ci.org/docs/zuul/latest/howtos/matrix.html may be useful | 20:00 |
| ildikov | corvus: it's a good one, I've been using it already :) | 20:03 |
| ildikov | thank you for sharing | 20:03 |
| ildikov | I was more lost with regards to setting up rooms, etc | 20:03 |
| corvus | ildikov: understood :) | 20:03 |
| ildikov | and I'm very grateful for everyone's help here! :) | 20:06 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!