| clarkb | hello it is meeting time | 19:00 |
|---|---|---|
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue May 7 19:00:33 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/HDXZPH7XJI5O5SDOCC35RGUALV7SUDZS/ Our Agenda | 19:00 |
| clarkb | We have an agenda but I half expect this to go quickly as I believe attendance may be light | 19:00 |
| clarkb | #topic Announcements | 19:02 |
| clarkb | I don't have anythign to announce other than that I expect this to go quickly and attendance to be light | 19:02 |
| clarkb | #topic Upgrading old servers | 19:02 |
| clarkb | With the release of Noble the year long countdown to getting off of focal has started | 19:03 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-focal-server-upgrades | 19:03 |
| clarkb | tonyb has started putting notes together for the needs there. I added some annotations and flavor to try and categorize things | 19:03 |
| clarkb | tonyb: would it be preferable for me to just go ahead and rearrange or do you like having the notes? | 19:03 |
| tonyb | Whatever works for you. | 19:04 |
| tonyb | I certainly have no objection to you moving things around | 19:04 |
| tonyb | I just did the bit I knew and figured I'd slowly figure the rest out | 19:04 |
| clarkb | sounds good | 19:05 |
| clarkb | The other idea I had here was to maybe try and consolidate the various server upgrade todo lists into one big list? | 19:06 |
| clarkb | then we don't need new documents over time as we have different upgrades to do | 19:06 |
| clarkb | I guess one reason not to do that is things might get too cluttered. I'm open to feedback on that | 19:07 |
| tonyb | Yup we can do that. Should be easy enough. I'm happy to give it a try and see what it looks like. | 19:07 |
| clarkb | thanks! | 19:07 |
| clarkb | I did also want to note that your changes to prep meetpad upgrades to jammy have landed and the changes to add the new servers are up | 19:08 |
| tonyb | Yup. | 19:08 |
| clarkb | as mentioned in review on the inventory update I think we want to do that with the old servers in the emergency file so that we don't end up with a weird cluster state but otherwise I think this should be straightforward | 19:08 |
| tonyb | As you pointed out we probably need a little care before takign the next step. | 19:08 |
| tonyb | I'll double check, but I think we're good to go. perhaps Thursday sometime? | 19:09 |
| clarkb | that should work for me | 19:10 |
| tonyb | cool beans | 19:10 |
| clarkb | mornings are generally better simply because then I feel more confident we'll have time to work through any issues over the day | 19:10 |
| clarkb | #topic MariaDB Upgrades | 19:11 |
| tonyb | Yup. We can figure out a time that works with TZ, and real world commitments in #opendev | 19:11 |
| clarkb | ++ | 19:12 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/916848 Update Gerrit's Mariadb | 19:12 |
| clarkb | this is still lacking reviews. I'm happy to approve this change if others review it and give it +2s | 19:12 |
| tonyb | I thought I had. | 19:12 |
| tonyb | I have now | 19:12 |
| clarkb | specifically I think we should plan to do a short gerrit downtime after the change merges to ensure we've got the upgrade in place after merging it as this won't happen automatically | 19:12 |
| clarkb | thanks! | 19:12 |
| clarkb | but other than that change and the manual tasks to deploy it all mariadb upgrades are complete | 19:13 |
| clarkb | #topic AFS Mirror Cleanups | 19:13 |
| clarkb | topic:drop-ubuntu-xenial has changes up for rewiew. | 19:13 |
| clarkb | I think this is largely where we're at now. I'm hoping to avoid a large backlog of changes like this sitting in my open queue. Ideally I write a few and we merge a few and slowly chip away | 19:14 |
| clarkb | Looks like a number of them have reviews and I just need to approve them when I can monitor. I think I'll try to do some of that today/tomorrow. We'll see how the days go | 19:14 |
| tonyb | Sounds good to me. | 19:14 |
| clarkb | #topic Building Ubuntu Noble Images and Test Nodes | 19:15 |
| clarkb | As mentioned last week the next step here is adding Noble to our mirror content and babysitting what is likely to be a large set of data that needs to be downloaded and released in afs | 19:16 |
| clarkb | in particular i Know we'll need a quota bump on the volume at least. | 19:16 |
| clarkb | I don't thik anyone has volunteered for that yet and I'm trying to focus on the cleanups until that is more fully done before I take on adding stuff. But if anyone else has time for this I think we can add content | 19:16 |
| tonyb | I doubt I'll have time before this meeting next week but I can probably do it after that. | 19:17 |
| tonyb | I have spent some time reading and learning more about AFS so I can try to pick up work items in that area. | 19:17 |
| clarkb | cool we can resync at the next meeting then | 19:19 |
| clarkb | #topic Gerrit 3.9 Upgrade Planning | 19:19 |
| clarkb | #link https://etherpad.opendev.org/p/gerrit-upgrade-3.9 Upgrade prep and process notes | 19:19 |
| clarkb | I spent a bit of yesterday doing some testing of this and I have learned new things | 19:19 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/918333 Need updated and promoted 3.9 image before doing more testing. | 19:19 |
| clarkb | First up is that when I thought I had updated the 3.9 image previously I didn ot because the promote job didn't run. Pretty sure this happend for the same reason it didn't run against an etherpad change recently: I only modified the job definitions and not the dockerfile | 19:20 |
| clarkb | this will also update our 3.8 image so we should plan a restart for that. Maybe combine the 3.8 update and the mariadb upgrade after merging both of those changes? | 19:20 |
| clarkb | Once that is done I can resume testing particularly of the downgrade using images that are likely to end up in production rather than old buggy images | 19:21 |
| clarkb | One of the questions from last week is whether or not hte change topic limit limitation of 5k changes per topic by default would be a problem for us | 19:21 |
| tonyb | Yeah I agree WRT combining the timing for MariaDB and the new 3.8 container restart | 19:21 |
| clarkb | through testing I am confident that the limit only applies to open changes and not the total of open and closed changes. This means we're fine for the new-release topic which is known to have more than 5k total but never near that many open | 19:22 |
| clarkb | I also tested an upgrade from 3.8 to 3.9 with the limit being exceeded and that did not seem to impact the upgrade process at all | 19:22 |
| tonyb | which is all goodness | 19:23 |
| clarkb | The only thing the limit seems to affect is when at or over the limit and you try to add a chagne to the topic you get an error. But even that is pretty forgiving. If you restore an abandoned change with a topic that would be over the limit the restore works just fine | 19:23 |
| clarkb | yup I'm fairly confident now that we can do the upgrade and not worry about the topic limit. Then if we ever find a reason to have more than 5k open changes in a topic we can revisit and adjust the config. But for now we're good as is | 19:24 |
| clarkb | The other thing I've tested and tried to improve upstream is building the Gerrit docs without webfonts | 19:24 |
| tonyb | Thanks Clark | 19:24 |
| clarkb | Testing shows some slight rendering differences but they are all minor and I think I'd rather have small rendering differences than load a bunch of fonts from google | 19:24 |
| tonyb | on gerrit tangent did our without-webfonts patch get any feedback? | 19:25 |
| clarkb | By default the gerrit release war does not know how to remove the webfont though so I wrote a change that I pushed upstream to do that | 19:25 |
| clarkb | #link https://gerrit-review.googlesource.com/c/gerrit/+/424404 Upstream change to have release war build target without webfonts | 19:25 |
| clarkb | tonyb: not yet the last time I checked | 19:25 |
| corvus | i wonder if we should start considering discouraging topics for things like that and encourage tags instead | 19:25 |
| corvus | since topics do have a use in gerrit for submitwholetopic. i don't have any expectation we would ever turn that on for openstack, however, it's starting to feel like the wrong tool for the job now that it has that additional functionality in gerrit | 19:26 |
| corvus | ("things like that" == release process) | 19:26 |
| clarkb | corvus: ya I think that gerrit diverged on how topics are used and never really communicated that explicitly | 19:27 |
| clarkb | it just happened naturally due to the same topic merge functionality I think | 19:27 |
| corvus | exactly | 19:27 |
| corvus | anyway, no need to discuss further or decide now; just throwing out a brainstorm :) | 19:27 |
| tonyb | corvus: We could probably do that. We'd just need to ensure that the tags stuff is as easy to use as the topics stuff ATM | 19:27 |
| tonyb | I've never really looked a tags | 19:28 |
| clarkb | tonyb: ya thats an acl update. But also I think we can address that separately to the upgrade | 19:28 |
| tonyb | Yup | 19:28 |
| corvus | tonyb: yeah, and i think that means really doing that acl update. which would be a good side effect of the process. because tags rock. | 19:28 |
| clarkb | basically we just need to allow registered users or whatever to globally set hashtags then let projects limit that if they wish | 19:28 |
| clarkb | That was the end of what I learned about the upgrade process. The good news is everything I was concerned about so far has been a non issue | 19:29 |
| clarkb | I'll resume testing once we have new images and make sure the revert process is viable for us | 19:29 |
| clarkb | #topic Wiki ssl cert renewal | 19:30 |
| clarkb | This is on my todo list to do friday ish. Just a few more days then the notifications should go away | 19:30 |
| clarkb | as before noting it in the meeting to make sure it isn't overlooked and everyone knows that it will be dealt with before it expires | 19:30 |
| clarkb | #topic Open Discussion | 19:30 |
| clarkb | I maybe should've considered this an announcement but I've been asked to let people know the CFP for the openinfra summit in korea is open until may29 | 19:31 |
| tonyb | Oh and git-review does hashtags already | 19:31 |
| clarkb | #link https://openinfrafoundation.formstack.com/forms/openinfra_asia_summit_2024 OpenInfra Summit CFP is open | 19:31 |
| tonyb | Thanks. | 19:32 |
| clarkb | The event happens at the beginning of September in a city outside of Seoul (I think within metro service distance) | 19:32 |
| tonyb | I've been asked by dpawlik to see if there are objecstions to enbing the elasticsearch reporter in the opendev zuul? | 19:33 |
| tonyb | I'm guessing it's report to the existing opensearch site | 19:33 |
| * tonyb would really like to go to Korea for the summit | 19:33 | |
| clarkb | it would probably be good to describe what the use case is. My initial reponse having run a large elasticsearch cluster for years is that I want nothing to do with that | 19:34 |
| tonyb | LOL | 19:34 |
| clarkb | I'm also not sure we should have reporters to sinks that we don't run (smtp is a weird one we report to the local smtp server iirc but then normal smtp happens to deliver elsewhere) | 19:35 |
| clarkb | but understanding the use case would be helpful before saying anythign definitive | 19:35 |
| tonyb | I think it is about consistency with other elasticserach site used by some team members | 19:35 |
| tonyb | so the same data is available for opendev as is for rdoproject | 19:36 |
| tonyb | but yes I'll get more details | 19:36 |
| clarkb | I think the answer to reporting to a third party elasticsearch is no. Reporting to an elasticsearhc/opensearch we run is a maybe with the side note that in theory the data reported is far less of a deluge than the log data was and it might be easier to maintain | 19:36 |
| tonyb | Okay. | 19:37 |
| tonyb | and opensearch.logs.openstack.org would be considered 3rd party? | 19:37 |
| clarkb | yes, opendev doesn't run it | 19:37 |
| tonyb | Cool. Just checking. | 19:38 |
| clarkb | and to justify running something like that ourslves we'd need to understand what it enabels that existing zuul apis do not | 19:38 |
| clarkb | which goes back to use cases | 19:39 |
| tonyb | Makes sense | 19:39 |
| tonyb | I think that's all I had | 19:40 |
| clarkb | for me as well | 19:41 |
| clarkb | I'll give it another minute before calling it a meeting | 19:41 |
| * tonyb twiddles thumbs | 19:42 | |
| clarkb | and time! | 19:43 |
| clarkb | #endmeeting | 19:43 |
| opendevmeet | Meeting ended Tue May 7 19:43:39 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:43 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2024/infra.2024-05-07-19.00.html | 19:43 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-05-07-19.00.txt | 19:43 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2024/infra.2024-05-07-19.00.log.html | 19:43 |
| clarkb | Thank you tonyb for joining me today :) | 19:43 |
| clarkb | oh and corvus | 19:43 |
| clarkb | I'm going to go eat lunch now | 19:43 |
| tonyb | Nice | 19:44 |
| corvus | yay! | 19:44 |
| corvus | i'm going to sleep! | 19:44 |
| clarkb | good night! | 19:44 |
| corvus | good lunch! | 19:44 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!