| clarkb | Meeting time | 18:59 |
|---|---|---|
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Mar 19 19:00:12 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/2KGXT47R66KRZPHD6I2HXB23SLIKCYOZ/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | fungi: made a git-review 2.4.0 release yesterday. Now is a good time to update if you haven't already | 19:00 |
| clarkb | having us use the latest version helps find any issues that might be present through our day to day use | 19:01 |
| clarkb | And the PTG is fast approaching | 19:01 |
| clarkb | #link https://etherpad.opendev.org/p/apr2024-ptg-opendev Get your PTG agenda items on the agenda doc | 19:01 |
| fungi | though also installing the branch tip should always be safe, since we do test it | 19:01 |
| fungi | and helps us find problems prior to tagging a new release | 19:01 |
| clarkb | ++ | 19:02 |
| clarkb | for the PTG I've gone ahead and added soem stuff. But as previously mentioend I expect ti to be fairly low key on our side and more of an outreach to our users sort of thing | 19:02 |
| clarkb | help them with their items rather than push any specific thigns from our side | 19:02 |
| clarkb | that said feel free to add content to that etherpad | 19:02 |
| clarkb | #topic Server Upgrades | 19:03 |
| clarkb | I haven't seen any new movement on this and I've been plenty distracted by old distro cleanups and rax auth and zuul reviews | 19:04 |
| clarkb | I suspect that the rest of us may similarly be distracted and not have anything new here. I'll give it a few minuets for others to chime in if I've missed anything before we move on | 19:04 |
| clarkb | #topic MariaDB Upgrades | 19:07 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11 | 19:07 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11 | 19:07 |
| clarkb | if I can get reviews on these changes I'm happy to approve and watch the rollout when it is convenient | 19:07 |
| clarkb | I think thats mostlywhere this is at. Just sanity checks that we're ok with the process applied to these services and then finding time to do it | 19:08 |
| clarkb | #topic AFS Mirror cleanups | 19:09 |
| clarkb | CentOS 7 is gone | 19:09 |
| fungi | and there was much rejoicing | 19:09 |
| clarkb | thank you everyone for the help getting things into shape to make that possible | 19:09 |
| fungi | if that experience has taught me anything, it's that i'm not looking forward to untangling ubuntu-xenial removal | 19:10 |
| clarkb | I did notice that I made some mistakes in cleaning up the opensuse obs centos 7 stuff whcih led to even more cleanups to the opensuse volume | 19:10 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/913608 Final OpenSUSE mirror cleanup | 19:10 |
| clarkb | this is opensuse cleanup but it is related to centos 7 :) and I think is the last step in finally clearing up both opensuse and centos 7 | 19:10 |
| clarkb | and ya after that is done it is time to start looking at ubuntu xenial removal which is going to be "fun" | 19:11 |
| clarkb | I've already got some changes up under topic:drop-ubuntu-xenial but that is going to be just the tip of the iceberg I expect | 19:11 |
| fungi | more like a continental ice sheet | 19:11 |
| fungi | or at least a modest glacier | 19:12 |
| clarkb | The upside to all of this is our afs volume capacity is looking far moer healthy. Which means we should have plenty of space for things like ubuntu 24.04 and maybe others | 19:12 |
| clarkb | so ya reviews welcome on that last cleanup change then be on the lookout for xenial removal stuff probably more serisouly after the PTG? I think between now and then I may have too much other stuff to work on | 19:13 |
| clarkb | #topic Rebuilding Gerrit Images | 19:13 |
| clarkb | Of which this is one :) | 19:13 |
| clarkb | I'd like to get our gerrit 3.9 image updated to 3.9.2. One big reason for this is it will make it easier to test 3.9's broken change reindexing to better understand if and how that affects us before we upgrade | 19:13 |
| fungi | sounds like a great idea | 19:14 |
| clarkb | but that will also upate our gerrit 3.8 image which means we should couple landing that change with a production restart of gerrit. Might be a good thing for late this week | 19:14 |
| clarkb | similar to the db upgrade changes I'm happy if others review the changes then I can approve and babysit as necessary (in this case do a docker-compose pull and down and up -d) | 19:15 |
| clarkb | re the 3.9 reindexing issue I also put it on the April 4th gerrit community meeting agenda to discuss the impact and better understand the issue | 19:15 |
| clarkb | it seems like a fairly serious problem but they aren't rushing to fix it so I may just lack understanding | 19:15 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2 | 19:16 |
| clarkb | I forgot to link the change. That chagne is the one that needs reviews :) | 19:16 |
| clarkb | #topic Rackspace MFA | 19:18 |
| clarkb | We will be required to setup MFA in rax March 26th but last week we decided to opt into it early to find any expected problems in a more controlled manner | 19:18 |
| clarkb | as noted yesterday fungi and I have meetings basically all day today so we said we'd do this tomorrow | 19:19 |
| clarkb | fungi: I'll just catch up with you tomorrow moring on that and we can work through it? | 19:19 |
| fungi | yep, sounds good to me | 19:19 |
| clarkb | #topic Project Renames | 19:20 |
| clarkb | preparing for this also in the list of things I need to look at before worrying too much about ubuntu xenial removal | 19:20 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames. | 19:21 |
| clarkb | this is a prep change for the playbook itself | 19:21 |
| clarkb | Once we're a bit closer we'll need to records changes too but don't need to do that too early | 19:23 |
| clarkb | #topic Linaro Cloud Cert Renewal | 19:24 |
| clarkb | Our helpful certchecker tool is warning us that this cert will expire on April 15 | 19:24 |
| clarkb | I've written down the process for renewing the cert on the node itself which si reachable from bridge | 19:25 |
| clarkb | its fairly straightforward we run acme.sh to get a new LE cert. Then copy its contents where kolla can find it and run kolla | 19:25 |
| frickler | where is that writeup? I could take a look at this | 19:26 |
| clarkb | frickler: its in the root homedir on the server itself | 19:26 |
| fungi | should we port that process document into system-config's docs for posterity, or is it too full of sensitive info? | 19:27 |
| clarkb | fungi: we probably could | 19:27 |
| clarkb | I don't think it has any sensitive info in the process. | 19:27 |
| clarkb | the server is access via bridge (and the server is the last one in our ansible inventory). You ssh in via bridge and then the doc is there. Happy to help port to our proper docs now that we've used thep rocess more than once and didn't need to make big changes ot it | 19:28 |
| clarkb | the first time I wrote it down I was distilling from a couple of emails. | 19:28 |
| clarkb | but I think tonyb did the last one without any issue off the local doc so ya proper hosting of that is fine | 19:28 |
| fungi | if discoverability is a concern, replacing it with a one-line file linking to the appropriate section of our docs would likely work | 19:28 |
| clarkb | ++ | 19:29 |
| frickler | I'll take a look at it later this week, seems it isn't urgent yet | 19:29 |
| clarkb | frickler: ya not urgent. Let me know if you have any questions | 19:29 |
| frickler | except for the daily warning mail, which is a good reminder ;) | 19:29 |
| clarkb | #topic Nodepool Image Delete After Upload | 19:29 |
| clarkb | corvus wanted to point out that this new feature has landed in nodepool and we can make use of it if we like | 19:30 |
| clarkb | basically you can update the configs to opt into having nodepool builder clean up on disk image files after they have been uploaded to clouds. Additionally you can opt to keep certain image formats and clear out others. | 19:30 |
| clarkb | In our case I think we could choose to delete all but the qcow2 format of the image since that file is the smallest and we can generate the other two (raw and vhd) from the qcow2 if necessary | 19:30 |
| corvus | might be worth asking what we want out of local images; recovery in case of cloud issues? user access for reproducibility? | 19:31 |
| clarkb | corvus: currently we do host files for users to do local reproduction but we only serve the qcow2 files to force people into the lower bandwidth option | 19:31 |
| clarkb | for this reason I don't think we want to delete all of the image files after upload, but I do think we can get away with keeping only the qcow2 file | 19:31 |
| corvus | so qcow2 for user access would be great then; and i think we're probably not too worried about cloud issues? | 19:31 |
| fungi | we did make them downloadable for user reproducibility reasons, i point people to those with some regularity | 19:31 |
| clarkb | corvus: ya I think for cloud issues we have enough clouds that we would fetch from one of them to push into another if absolutely necessary | 19:32 |
| clarkb | or just rebuild and let other clouds deal with that particular image type in the interim | 19:32 |
| corvus | like, if we lose a cloud we could probably live with either "let the next rebuild fix it" or "convert it ourselves" if we're in a dire situation. | 19:32 |
| clarkb | exactly | 19:32 |
| fungi | or remove that cloud from out configuration temporarily | 19:33 |
| fungi | s/out/our/ | 19:33 |
| clarkb | maybe go ahead and push up a change to keep only the qcow2s post upload and if there are any further concerns they can be posted in review? | 19:34 |
| frickler | with a similar reasoning we might be ok with only keeping the latest local image and not multiple iterations? | 19:35 |
| clarkb | yes, though i'm not sure if nodepool can express that yet | 19:35 |
| frickler | yes, this was more of like another feature idea | 19:36 |
| clarkb | #topic Open Discussion | 19:37 |
| clarkb | That took us to the end of the listed agenda but I forgot to add a couple things before sending it | 19:37 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/913686 Update Gitea to 1.21.8 | 19:37 |
| clarkb | This fixes a number of bugs in gitea supposedly. Would be good to get that in to keep up to date | 19:38 |
| clarkb | And then separately etherpad released a 2.0 (and then very quickly a 2.0.1) | 19:38 |
| clarkb | The changes to etehrpad appear to largely be in how etherpad is installed and managed (they use `pnpm` now I don't know what that means really yet) | 19:38 |
| clarkb | It appears that this means we'll have to completely redo our docker image builds but I think the upgrade itself is straightforward as the db doesn't change | 19:39 |
| fungi | pnpm seems to be a package manager for npm | 19:39 |
| clarkb | if anyone else wants to look into that please feel free. Our current image is basically a fork of their image because they weren't reliably rebuilding their images for nodejs security issues | 19:39 |
| clarkb | So we need to port our fork over to the new stuff and then test the result | 19:39 |
| fungi | https://pnpm.io/ | 19:39 |
| clarkb | you'll note that npm is a package manager itsefl | 19:40 |
| clarkb | so now we have package manager managers | 19:40 |
| fungi | looks like a meta package manager that tries to use some sort of caching file tree | 19:40 |
| fungi | throws around terms like "content-addressable storage" | 19:40 |
| clarkb | if I end up looking at this it will almost certainly be a post PTG task for me | 19:41 |
| clarkb | and post PTG is probably fine since big update to etherpad pre PTG are scary | 19:41 |
| fungi | also looks like pnpm positions itself as a more efficient replacement for npm | 19:41 |
| clarkb | Anything else? | 19:42 |
| clarkb | I feel like there is a lot but we're moving slowly due to the openstack release. I bring that up to reassure people its ok | 19:43 |
| clarkb | priorities and all that | 19:43 |
| fungi | pnpm seems to also try t replace yarn | 19:43 |
| clarkb | fungi: the js community never found a tool that was too good to replace | 19:44 |
| fungi | i secretly replaced the js community with folgers crystals | 19:44 |
| clarkb | sounds like that may be it | 19:46 |
| clarkb | Thanks everyone! | 19:46 |
| clarkb | #endmeeting | 19:46 |
| opendevmeet | Meeting ended Tue Mar 19 19:46:15 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:46 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2024/infra.2024-03-19-19.00.html | 19:46 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-03-19-19.00.txt | 19:46 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2024/infra.2024-03-19-19.00.log.html | 19:46 |
| fungi | thanks clarkb! | 19:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!