| clarkb | just about meeting time | 18:59 |
|---|---|---|
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Aug 20 19:00:55 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/XHQIBDONZ6RMOINLKWHDFGKVN5ADB3N3/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | The openinfra summit in korea is just under 2 weeks away | 19:01 |
| clarkb | I'm planning to attend so will miss the September 3 meeting. Do plan to chair the August 27 meeting though | 19:02 |
| clarkb | #topic Upgrading Old Servers | 19:03 |
| clarkb | Looking at Gerrit it appears tonyb has been pushing noble server replacements along | 19:03 |
| clarkb | apologies that these changes got lost in the shuffle | 19:03 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/925447/2 | 19:04 |
| tonyb | Slowly. I have some updates to make to https://review.opendev.org/q/topic:noble-mirror+is:open which will be the first focal upgrade | 19:04 |
| clarkb | #link https://review.opendev.org/q/topic:noble-mirror+is:open | 19:04 |
| tonyb | Also I've finally finished testing the recaptcha and bulk delete extensions on the held wiki node | 19:05 |
| fungi | frickler had some comments on 925438 or else i'd just insta-approve it | 19:05 |
| clarkb | ya I +2'd those changes but didn't approve since there were comment sfrom frickler | 19:05 |
| tonyb | I plan to respin the mediawiki role with a bunch of questions. | 19:05 |
| clarkb | tonyb: I take it there weren't any major concerns that popped up in the wiki testing? | 19:05 |
| clarkb | at least for those particular items? | 19:06 |
| fungi | my cursory testing was good with the wiki held node | 19:06 |
| tonyb | Yeah I don't know how I missed that's what the pause was on the noble updates | 19:06 |
| fungi | i didn't find time to make edits with a separate account and test partrolling them, but i'd also be fine just "testing in production" for that bit since i'm the only one currently doing it anyway | 19:07 |
| tonyb | clarkb, fungi: Nope once I creted a valid recatchpa token (it's a personal one) recaptcha worked as expected and once that was done the there was stuff to bulk delete | 19:07 |
| clarkb | that is reassuring that things are working as you get to them | 19:07 |
| tonyb | I'd like another review of https://etherpad.opendev.org/p/opendev-wiki-announce and then we can float some timelines | 19:08 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-wiki-announce Draft announcement for wiki server replacement | 19:08 |
| tonyb | Yeah the only think that doesn't "just work" is the skin which was also expected | 19:08 |
| fungi | and also purely cosmetic | 19:09 |
| tonyb | We can think about how to handle the database update/migration inline on the review once they're updated | 19:09 |
| clarkb | sounds good. I'll take a look at that announcement draft today. I know i've looked at it before but there appear to be new edits | 19:09 |
| tonyb | Yup I incorporated your feedback and added some extra details | 19:10 |
| clarkb | anything else? | 19:10 |
| clarkb | #topic AFS Mirror Cleanups | 19:12 |
| clarkb | This item is the first one to fall off my list as I get distracted by other things. Unfortanately, that means I haven't made any real progress on this since the removal of projects from the zuul tenant | 19:12 |
| clarkb | I think the next step here is going to be continuing to do project removals from the zuul tenant config and also cleaning up jobs in projects that are still active | 19:13 |
| clarkb | but I don't have a sense yet for what is left to do in the current state of things | 19:13 |
| clarkb | That said there is some interest in mirroring rocky packages so it would be good to get xenial cleared out | 19:13 |
| clarkb | However, it would be good to determine if we need to mirror rocky packages before proceeding with that (this came up in the TC meeting about an hour ago) | 19:14 |
| clarkb | #topic Testing Rax's New Cloud Offering | 19:14 |
| clarkb | Progress is slowly being made here. I'm hoping to have a quick chat with some folks on thursday to get some base details | 19:15 |
| clarkb | So while I don't have anything concrete yet I expect to have more info this week | 19:15 |
| tonyb | That'd be good. It's all very "handwavy" | 19:16 |
| fungi | yeah, basically hoping cloudnull can clarify it all for us | 19:16 |
| clarkb | #topic Etherpad 2.2.2 Upgrade | 19:17 |
| clarkb | Recently Etherpad made a 2.2.2 release. There are actually tags for releases between 2.2.2 and the release we are running but not official release objects on github | 19:17 |
| clarkb | Testing quickly showed that this doesn't just work for us like some previous updates. In particular they have completely rewored how js code is imported/loaded into the browser and this appears to break our ep_headings plugin | 19:18 |
| clarkb | There is an alternative plugin called ep_headings2 that does similar things, but an ancient github issue indicates ep_headings2 is not compatible with ep_headings markup. It isn't clear to me yet if this incompatibility is fatal to etherpad or if pads will just require some manual reformatting | 19:19 |
| clarkb | I think we can live with manual reformatting as people can do that for pads that are active and that they care about. But if this is fatal for the service or just pads using ep_headings we will need to do more work to fix things | 19:19 |
| fungi | the data is fully encapsulated in mysql right? so in theory we can dump/source production into a held node and see what happens? | 19:20 |
| clarkb | Anyway that all requires some testing. I think we can hold a node on the version we are running today. Make a pad with all the variou headings content, then update to 2.2.2 and see what breaks if anything | 19:20 |
| clarkb | yes an alternative to ^ is to restore the prod db into the current 2.2.2 held node | 19:20 |
| clarkb | I just haven't prioritized this testing yet as I've been largely focused on getting prepared for the summit | 19:20 |
| clarkb | if anyone else wants to poke at this that would be appreciated | 19:21 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/926078 WIP Change implementing the upgrade | 19:21 |
| fungi | i'll see how my tomorrow shapes up | 19:21 |
| clarkb | that change is WIP until we know we won't break things with an upgrade | 19:21 |
| * fungi is still catching up from emergency travel madness | 19:21 | |
| fungi | seems like it would be straightforward to test though so i'll try to give it a shot | 19:22 |
| clarkb | thanks | 19:22 |
| clarkb | also worth noting we did confirm that ep_headings2 works with 2.2.2. The main question is what compatibilty looks like coming from ep_headings | 19:22 |
| clarkb | I suspect in an absolute worst case we might end up needing to export and reimport all the pads in some sort of automated fashion. That will almost certainly lose all the headings formatting but should preserve the content | 19:23 |
| tonyb | and the etherpad team aren't interested in the fact ep_headings is broken? | 19:24 |
| fungi | it's "just a plugin" | 19:24 |
| tonyb | Ah | 19:24 |
| clarkb | ya ist an ancient plugin that was archived | 19:24 |
| tonyb | that's a little sad | 19:24 |
| tonyb | but understandable | 19:24 |
| fungi | i think the idea that there are people running decade+ old etherpad instances and not auto-expiring pads is surprising to them | 19:25 |
| clarkb | and more generally the new maintainer is far more interested in modernizing things than keeping compatibility with really old stuff | 19:25 |
| fungi | the project itself has had several leadership turnovers and renames in that span of time, after all | 19:25 |
| clarkb | for example the new auth system. Though they did eventually add the old auth system abck into etherpad | 19:25 |
| clarkb | #topic Service Coorindator Election | 19:27 |
| clarkb | Today is the last day for the nomination period. I haven't seen any nominations. Assuming I haven't missed one does that mean I'm it again? | 19:27 |
| frickler | congratulations :) | 19:28 |
| clarkb | I can make it official after the meeting if no one else wants it I guess | 19:28 |
| fungi | amd you have my condolences | 19:29 |
| fungi | s/amd/and/ | 19:29 |
| fungi | congratudolences | 19:29 |
| tonyb | clarkb: Thank you for your bravery | 19:29 |
| clarkb | the crowd is making their will clear | 19:30 |
| clarkb | #topic Updating the Default Zuul Nodeset | 19:30 |
| clarkb | last week I announced we would set ubuntu-noble as the default nodeset tomorrow | 19:30 |
| clarkb | #link https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/DWF57QP75BC6GBIG7RV6PRLBDOVANET3/ | 19:30 |
| fungi | and there was much rejoicing | 19:30 |
| clarkb | #link https://review.opendev.org/c/opendev/base-jobs/+/926360 | 19:31 |
| clarkb | plan is still to merge that tomorrow as announced | 19:31 |
| clarkb | I intend on doing that first thing in the morning so I have plenty of time to be around to help debug if necessary | 19:31 |
| clarkb | do ya'll think that is early enough in the day or would you prefer to merge it before my day starts? | 19:31 |
| clarkb | I can drop the -W if that is the case | 19:31 |
| frickler | I'm busy in my morning, so fine with waiting for you | 19:32 |
| fungi | seems fine. i can approve it earlier if you want though | 19:32 |
| clarkb | no I think that is enough time. I just wanted to give the option if we felt strongly about it | 19:33 |
| clarkb | #topic openstack.org DNS Hosting Moved to CloudFlare | 19:33 |
| fungi | it happened | 19:33 |
| clarkb | as of very recently this manuever has been completed | 19:33 |
| fungi | a few hours ago now | 19:33 |
| clarkb | please be on the lookout for unexpected behavior that can be attributed to DNS | 19:33 |
| tonyb | noted | 19:34 |
| clarkb | fungi is able to edit the zone directly and while I can't yet do things myself I can also message other people who can | 19:34 |
| fungi | thanks frickler for spotting the problem with the proxied openstack.org redirect to www.openstack.org | 19:34 |
| clarkb | I've got on my todo list to create an account so that I can make edits in addition to fungi. In general though we don't make a ton of edits so I don't expect issues | 19:34 |
| fungi | more generally, we should avoid making changes to that domain for a few days anyway while authoritative dns settles out | 19:34 |
| fungi | also it looks like we can probably adapt our raxdns backup to use the cloudflare api, we'll just need an api key created and (obviously) code changes to enable it | 19:35 |
| clarkb | I wonder if cloudflare has direct export apis | 19:36 |
| clarkb | the raxdns backup has to go record by record and construct the file itself iirc | 19:36 |
| frickler | so are all non-opendev websites now proxied by cloudflare? | 19:36 |
| fungi | they definitely have a rest api because i perused the docs for it | 19:36 |
| fungi | frickler: no, only openstack.org | 19:36 |
| clarkb | (and it was before this change. I think some of the motivation for the change was to streamline that existing integration) | 19:37 |
| fungi | basically, the foundation relies on cloudflare for cdn services with www.openstack.org but cloudflare makes that less expensive if you host the domain with them, while rackspace charges the foundation for dns hosting | 19:37 |
| frickler | I think the good thing about it is that it is reachable via IPv6 now | 19:38 |
| tonyb | yeah it looks like you can basically get a bind zone file via the API | 19:38 |
| fungi | so moving the hosting for that specific domain was a cost-cutting measure | 19:38 |
| frickler | openstack.org did point to a rax IP before, but I didn't check www. | 19:38 |
| clarkb | frickler: oh ya I think only one of them pointed to cloudflare before | 19:39 |
| clarkb | and the other redirected to the cdn or something. | 19:39 |
| fungi | actually openstack.org was going to a vexxhost ip address i think, but regardless it was just hosting a redirect to the cloudflare cdn for www.o.o | 19:39 |
| frickler | didn't occur to me that anyone is actually still using www.* these days | 19:39 |
| fungi | welcome to the wonders of the world wide web | 19:40 |
| frickler | I'd also still prefer for us to have higher TTLs on our CNAMEs again (like 1h), but also not mission critical I guess | 19:41 |
| clarkb | I think that is a generally good idea. At least historically we saw dns resolution failures at a higher rate with low ttls from hosts behind NAT | 19:41 |
| fungi | yeah, as i pointed out, an http patch call can adjust the ttl per record so we could do that to the (currently) 104 records which refer to opendev services | 19:41 |
| clarkb | all of those udp packets can easily get lost in the NAT state tables I guess | 19:42 |
| fungi | but also we try not to depend on the openstack.org domain where we can, so it should eventually just be for openstack project content (afs/kerberos is the biggest hold-out at the moment) | 19:42 |
| frickler | well I don't think we will change docs.openstack.org, which I'd consider the most important one | 19:43 |
| fungi | docs.openstack.org is openstack-specific content hosted from static.opendev.org via a cname in dns | 19:43 |
| fungi | so basically the sort of thing i said the domain should eventually consist of once we're done moving opendev services out | 19:44 |
| clarkb | #topic OpenMetal Cloud Cert Refresh | 19:45 |
| fungi | with my openstack tact sig chair hat on i do think the dns and hosting for docs.openstack.org is important, but opendev's own operations don't depend on it | 19:45 |
| clarkb | ya and we've got a path towards addressing that | 19:45 |
| clarkb | might just take a moment | 19:46 |
| frickler | the openmetal issue looks like it could be fixed | 19:46 |
| clarkb | yup the background here is LE emailed us warning us that the cert was going to expire soon | 19:46 |
| fungi | we'll know tomorrow-ish | 19:46 |
| frickler | I checked logs earlier and they no longer showed the error | 19:46 |
| fungi | oh good | 19:46 |
| clarkb | frickler looked at the kolla logs and saw that the config appeared to have been updated to use the wrong email account | 19:46 |
| frickler | instead there was a ratelimit error from LE | 19:46 |
| clarkb | we passed this info along to openmetal and they made some changes which hopefully will resolve things | 19:46 |
| frickler | but hopefulle it will be all resolved after the next cron trigger | 19:47 |
| fungi | stupid rate limits. doesn't everyone know that time and space are relative anyway? | 19:47 |
| clarkb | fungi: did you change to add it to certcheck land? | 19:47 |
| fungi | not yet | 19:47 |
| fungi | #link https://review.opendev.org/926488 "Track our OpenMetal environment HTTPS cert expiry" | 19:47 |
| clarkb | maybe we should go ahead and +W that since its a minor change with minimal impact if we got it wrong | 19:48 |
| fungi | looks like frickler just did. thanks! | 19:48 |
| frickler | +3 | 19:48 |
| frickler | good idea to check port 5000 | 19:48 |
| fungi | agreed, it hadn't dawned on me that they could diverge in the future | 19:49 |
| clarkb | cool, that should help us easily confirm things are issuing properly in the next day or so | 19:49 |
| clarkb | #topic Open Discussion | 19:49 |
| clarkb | Anything else with our last ~10 minutes? | 19:49 |
| tonyb | not from me. | 19:50 |
| * fungi hungers | 19:50 | |
| frickler | there was some discussion about the "opendev team" name in https://review.opendev.org/c/openstack/contributor-guide/+/926511 | 19:51 |
| frickler | and we do reference that multiple times in our own docs | 19:51 |
| fungi | oh, yeah, i think what's there now was the result of a quick stream edit | 19:52 |
| frickler | so if someone feels strongly to not use that term, an update would be in order IMO | 19:52 |
| fungi | i'll try to take a look tomorrow and adjust the terminology to match what we settled on when we established our new identity | 19:53 |
| clarkb | https://codesearch.opendev.org/?q=OpenDev%20Team&i=nope&literal=nope&files=&excludeFiles=&repos= shows a couple of small instances | 19:53 |
| clarkb | might be good to link to others if you see them | 19:53 |
| frickler | I also failed to find a good reference for "all (openstack/opendev related) IRC channels" | 19:53 |
| frickler | do we intentionally not have that? | 19:53 |
| clarkb | frickler: openstack/project-config/accessbot/channels.yaml is probably the most reliable | 19:54 |
| fungi | we don't run any irc servers, so technically speaking, no | 19:54 |
| frickler | there were some refs in system-config | 19:54 |
| fungi | we have channels where we maintain access controls, channels where we provide logging and meeting services, channels where we emit code review event data, channels where we supply announcements... | 19:55 |
| tonyb | there was a master list on the wiki but that's certainly wrong. apart from that you could trall meetings.opendev.org/irclogs | 19:55 |
| frickler | https://docs.opendev.org/opendev/system-config/latest/project.html and https://docs.opendev.org/opendev/system-config/latest/irc.html | 19:55 |
| clarkb | frickler: thanks | 19:55 |
| fungi | but to be clear, we haven't attempted to offer an index of channels indicating what's on topic for each of them | 19:55 |
| clarkb | and I think thats more the purvue of the projects themselves if they choose to have 100 channels | 19:56 |
| frickler | yes, I resorted to referring to individual project's contributor docs | 19:56 |
| clarkb | (I'm much more a fan of collapsing things down into more central channels) | 19:56 |
| fungi | i agree that's the most correct guidance | 19:56 |
| fungi | referring to project documentation i mean | 19:57 |
| clarkb | and we are at time. Thank you everyone! we'll be back here next week at the same time and location. | 20:00 |
| clarkb | #endmeeting | 20:00 |
| opendevmeet | Meeting ended Tue Aug 20 20:00:22 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2024/infra.2024-08-20-19.00.html | 20:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-08-20-19.00.txt | 20:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2024/infra.2024-08-20-19.00.log.html | 20:00 |
| fungi | thanks clarkb! | 20:00 |
| frickler | thx all o/ | 20:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!