| clarkb | hello | 19:00 |
|---|---|---|
| clarkb | time for our weekly team meeting | 19:00 |
| corvus | aloha | 19:01 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Oct 10 19:01:37 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/7WZKZSWIX2W3OHQFM6OZNJU54BUD4IIT/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | The OpenInfra PTG will take place October 23-27. Please keep this in mind when making changes to tools like etherpad and meetpad | 19:02 |
| clarkb | #topic Mailman 3 | 19:02 |
| clarkb | and now we can dive right in to our agenda | 19:02 |
| clarkb | The plan is still to migrate lists.openstack.org on Thursday starting around 15:30 UTC | 19:03 |
| clarkb | yesterday I asked fungi if there aws anythign we should do to help prepare and sounded like it was well in hand? | 19:03 |
| fungi | still on track for maintenance thursday, yes. i sent a reminder to openstack-discuss on friday | 19:03 |
| fungi | i'm working on the changes we'll merge as part of the maintenance and will ping infra-root once they're ready for review today or worst case tomorrow | 19:04 |
| fungi | i'll also finish fleshing out the maintenance plan on the etherpad | 19:04 |
| clarkb | sounds good | 19:04 |
| fungi | but it's mostly a copy/paste of the earlier maintenances with timelines adjusted | 19:04 |
| fungi | there are a few steps to do tomorrow, including initial rsync and checking dns record ttls | 19:04 |
| clarkb | ping me if/when I can help and I can dig into things | 19:05 |
| fungi | will do | 19:05 |
| clarkb | The other mailman3 item of note is we created a new mailing list through our automation otuside of a migration. That seems to ahve worked as welll | 19:05 |
| clarkb | we expected it to as our testing covers that but always good to confirm | 19:05 |
| fungi | yes, the list owner confirmed they were able to create an account and got granted control of the settings | 19:06 |
| fungi | they do not yet seem to have posted anything though, empty archive | 19:06 |
| fungi | (i subscribed successfully when it was first created, and haven't seen any messages for it either, so no reason to think the archiving is broken) | 19:07 |
| clarkb | ack | 19:07 |
| clarkb | anything else mailman 3 related? | 19:07 |
| fungi | i have nothing | 19:08 |
| clarkb | #topic Updating Our OpenMetal Cloud | 19:08 |
| clarkb | #link https://etherpad.opendev.org/p/redeploying-opendevs-openmetal-cloud Notes from discussion with Yuriy | 19:08 |
| clarkb | This was an informative discussion. There are 3 timeframes we can upgrade within which will produce different results. If we upgrade today we'll get a newer base OS. If we upgrade in about a month we'll get the same newer base os and the same openstack versiosn as before but with more robust configs. And finally if we upgrade early next year (sounded lik February) we'll get | 19:09 |
| clarkb | an even newer base OS and openstack 2023.1 or 2023.2 | 19:09 |
| frickler | did you discuss whether they might offer debian or ubuntu as base os? | 19:10 |
| fungi | upgrading today would also get us more ip addresses, right? | 19:10 |
| clarkb | given that I think we shouldn't upgrade until the more robust openstack configs are available. That means not upgrading right away. But then we have to decide if we want to upgrade twice or just once | 19:10 |
| clarkb | fungi: oh yes a few more addresses since we would reduce the total number of subnets in use which helps with vrrp | 19:10 |
| clarkb | frickler: yes that came up. It didn't sound like they have fully decided which base os would be used in the February update | 19:11 |
| clarkb | frickler: but currently its all a variation of centos | 19:11 |
| clarkb | we don't need to decide anything right now, but I think the decision we need to make is if we want to upgrade twice (once in november and once in february) or just once and if just once decide if we wait for february or do it in november ish | 19:12 |
| fungi | sounds like they're more comfortable with red hat based platforms from a sheer familiarity standpoint but are considering alternatives due to recent changing winds in that regard | 19:12 |
| clarkb | I don't think there is much value to doing three upgrades and doing an upgrade right now | 19:12 |
| frickler | I would be fine with waiting until feb | 19:13 |
| clarkb | considering we seem to lack time as a major resource I think I'm leaning that way too | 19:13 |
| clarkb | other useful bits of info: we do have to shutdown the existing cloud to build a new one because we would recycle the same hardware. The cloud should be renamed to openmetal from inmotion particularly important on the grafana side so we may update that dashboard with a new name earlier | 19:14 |
| clarkb | maybe something like "OpenMetal (formerly InMotion)" to reduce confusion | 19:15 |
| clarkb | maybe think this over and we can put together a plan next week | 19:15 |
| clarkb | and then relay that back to Yuriy as necessary to ensure we're all on the same page | 19:16 |
| fungi | worth noting, the openstack feature freeze for their next release will be at the end of february, so that will probably be the busiest time for quota usage | 19:16 |
| fungi | though the openmetal cloud being offline at that time wouldn't severely diminish available quota | 19:16 |
| clarkb | its about 10% of our quota? maybe a little less | 19:16 |
| fungi | right | 19:16 |
| clarkb | probably noticeable but we should be able to limp along | 19:16 |
| fungi | maybe a little more after the reprovision | 19:17 |
| clarkb | alright please think it over during the next week and bring up any other concerns like the rlease schedule timing for openstack and we can put a rough plan in place soon | 19:18 |
| clarkb | #topic Python Container Updates | 19:18 |
| clarkb | The Gerrit update is complete. We are now running Gerrit on bookworm with java 17 | 19:19 |
| clarkb | I haven't noticed any problems, but please say something if you notice anything off or weird | 19:19 |
| clarkb | In theory GC performance is much improved under java 17 so we should see Gerrit being more responsive | 19:19 |
| clarkb | #link https://review.opendev.org/q/(+topic:bookworm-python3.11+OR+hashtag:bookworm+)status:open | 19:19 |
| clarkb | There are some other containers that need updates though. Within OpenDev we have lodgeit and gear. I think both of those should be good to go. In lodgeit's case I had to do some python3.11 porting | 19:19 |
| clarkb | On the zuul side I updated zuul-registry to simply move to python3.11 and stay on bullseye for now | 19:20 |
| clarkb | My hope is that we can get everything to python3.11 and then drop python3.9 and python3.10 builds on both bullseye and bookworm. And that will be a big impact on the total number of images we need to juggle | 19:20 |
| clarkb | not as great as having just bookworm but still a big improvement | 19:20 |
| clarkb | if we want to do lodgeit today I should be around to keep an eye on it | 19:21 |
| corvus | seems reasonable; i looked a bit at the zuul-registry issue, and i didn't see a quick fix. it could be a problem in the future. | 19:21 |
| clarkb | and gear is something we don't use anymore so should be fine wheneever | 19:21 |
| corvus | (i mean, it's a problem now; could be a bigger one later) | 19:21 |
| clarkb | corvus: ya I too looked at rehash and decided this would require actually undersanding openssl internals which felt like a pain. One crazy idea I had was to use golang's sha256sum implementation instead since it too is resumable | 19:22 |
| corvus | (so if anyone wants to explore python/c libssl bindings, there's a good puzzle for you!) | 19:22 |
| corvus | good idea | 19:22 |
| clarkb | but then you are shuffling bits around between python and go instead of python and C and that may be more painful | 19:22 |
| fungi | i take it pyca/cryptography doesn't have resumable hash primitives | 19:22 |
| corvus | not that i have found | 19:24 |
| clarkb | cryptography does allow you to make copies of hash objects in order to get intermediate results. But what we really need is the ability to serialize the objects in that state and I don't know that they support this | 19:24 |
| fungi | right, openssl lets you get intermediate results too | 19:24 |
| fungi | (even in v3 i mean) | 19:24 |
| fungi | but i also didn't see anywhere that it allowed exporting that state | 19:25 |
| clarkb | https://github.com/stevvooe/resumable is the golang thing I found which is deprecated and says stdlib can do it directly | 19:26 |
| clarkb | appears to do something similar to pickling the hash object then unpickles it later | 19:26 |
| clarkb | in any case reviews on those outstanding changes are helpful and I'm happy to help shepherd things through with approvals once reviewed | 19:27 |
| clarkb | #topic Etherpad 1.9.3 | 19:27 |
| clarkb | Pretty sure I mentioned this last week? But tonyb ran into similar cache related problems with 1.9.3 so I'm wary of upgrading prior to the PTG. My big concern is that people won't use etherpad until the PTG then it won't work due to cache issues | 19:28 |
| clarkb | For this reason I think we defer this upgrade until after the PTG and then peopel can clear caches when not in a rush to attend sessions | 19:28 |
| fungi | fwiw, i don't believe this is the first time i've seen persisted cookies/storage/state get confused after an etherpad upgrade | 19:28 |
| fungi | but i agree avoiding that right before a ton of users swamp the system is probably wise | 19:28 |
| frickler | +1 | 19:29 |
| tonyb | +1 | 19:29 |
| clarkb | fungi: ya I think we've hit it at least once before | 19:29 |
| clarkb | #topic Gitea 1.21 | 19:29 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/897679 A test change for gitea 1.21.0-rc1 | 19:29 |
| clarkb | I went ahead and got this started since the template updates tend to be annoying and not need many updates after the first RC | 19:30 |
| clarkb | there is still no changelog but this change found at least one problem we will need to address: Gitea by default requires rsa keys to be 3072 bits or longer in 1.21 | 19:30 |
| clarkb | The key that trips over this is the gerrit replication key | 19:31 |
| clarkb | In the change I've disabled key length checking, but I think we should also take a todo to rotate the key out with something larger or with another key type | 19:31 |
| fungi | which has been the default size ssh-keygen emits for a while, just not as long as when we generated that key | 19:31 |
| clarkb | to do that we should be able to prestage the new key alongside the old key in gitea then add it to the gerrit user's home dir and remove the old key from gerrit | 19:32 |
| clarkb | but we need to check that the key isn't used anywhere else before removing it | 19:32 |
| fungi | and yeah, we could either rotate the key before upgrading, or upgrade with the restriction disabled, rotate after, and then remove the override | 19:32 |
| clarkb | We can probably wait for the changelog to exist before decidign just in case there is any extra info they have to share | 19:32 |
| fungi | replacing the key first might be less work, but could delay the upgrade | 19:33 |
| fungi | and i have no real preference either way | 19:33 |
| clarkb | #topic Gerrit Replication File Leaks | 19:33 |
| * fungi pops some popcorn for this one | 19:34 | |
| fungi | any new plot twists? | 19:34 |
| clarkb | I was really hopeful that this bugfix would be in place by the time we updated Gerrit to bookworm. Unfortunately I got feedback late last week for a redesign on the approach. I'm a bit frustrated by that because I am not responsible for the approach in the current code base I'm merely patching it to stop elaking files (which my change does successfully do with three test | 19:34 |
| clarkb | cases...) | 19:34 |
| clarkb | I spent Friday trying to make the suggested approach work with minimal refactoring. I'm missing something clearly important in the handling of the events because I can get it to work when replication actually does need to happen but if you filter out the project using the replication config I'm now leaking more files | 19:35 |
| fungi | who new trying to fix one bug was going to result in you being the new maintainer? | 19:35 |
| clarkb | one step forward two back sort of deal. The latest thing I'm attempting is getting eclipse going so that I can mroe easily use a debugger to understand that new behavior. Unfortunately I broke eclipse in about 30 seconds | 19:36 |
| clarkb | I now have like 5 different eclipse workspace and two separate gerrit clones... I think I'm understanding eclipse better but its been interesting | 19:36 |
| clarkb | If anyone understands eclipse and wants to walk through it with me I would be grateful :) | 19:36 |
| clarkb | If that doesn't go anywhere I'll fallback to jdb which is not my favorite debugger but should be able to do what I need it to do | 19:37 |
| corvus | System.out.println is my fav | 19:38 |
| clarkb | I'm speedrunning java developer setup | 19:38 |
| fungi | you should twitch stream that | 19:38 |
| tonyb | and we're enjoying the show | 19:38 |
| tonyb | I'd watch! | 19:38 |
| clarkb | corvus: oh that is the other issue. The test suite appaers to eat the replication logs. Normal error logs are emitted though so I've thought about a mass replacement of the replication log calls with normal log calls so that I can see what is happening | 19:38 |
| corvus | oO | 19:39 |
| clarkb | I think it is writing them to a virtual in memory disk location that goes away when the test finishes | 19:39 |
| clarkb | similar to how w do it with python except they aren't capturing the contents so they can be given to you on completion | 19:39 |
| fungi | yeah, gerrit plugins have independent logging, right? i vaguely recall configuring that somewhere | 19:39 |
| clarkb | whereas the normal log goes to stdout | 19:39 |
| clarkb | fungi: the replication plugin does | 19:39 |
| clarkb | anyway I haven't given up. I have some avenues to explore and will continue to poke at them as time permits | 19:40 |
| clarkb | #topic Re-evaluating the meeting time | 19:41 |
| clarkb | frickler mentioned that now that ianw is no longer attending the meetings we might be able to reschedule to be friendlier to other timezones. One problem with that is I'd like to continue to encourage tonyb to participate and moving the meeting earlier may make that difficult for tonyb | 19:42 |
| clarkb | that said, it is still pretty early for tonyb and maybe tonyb would rather catch up asynchronously anyway? | 19:42 |
| clarkb | thoughts? | 19:42 |
| fungi | i'm open to different times, though i do have (rather a lot of) other meetings on tuesdays. up to 4 other meetings besides this one depending on how the weeks/months fall | 19:42 |
| tonyb | selfishly I'd like to keep the meeting at a time I can attend | 19:43 |
| clarkb | when frickler brought the idea up to me I think we both decided that 1700-1800 UTC would work for us | 19:43 |
| fungi | and some of my tuesday meetings follow daylight/summer time shifts, which add to the complexity | 19:43 |
| clarkb | tonyb: ok that is good to know. I assume 1900 UTC is about as early as is practical for you too? | 19:43 |
| tonyb | and with AU having done DST it's 6am which is about as early as I can do | 19:43 |
| tonyb | correct | 19:44 |
| tonyb | the winter 5am block was hard for me to get to | 19:44 |
| frickler | hmm, o.k., so let's stick to this time, then | 19:45 |
| clarkb | another approach would be to shift the meeting so that it is more convenient for australia and europe and those of us in north maerica take the hit | 19:45 |
| tonyb | we could alternate, but that sucks for different reasons | 19:45 |
| fungi | i guess there are better au/de overlap times which are like 3am for some folks in the us | 19:45 |
| clarkb | I would need to look at a map and do some maths to figure out when ^ those times are | 19:45 |
| fungi | can we shrink the planet, or turn it at a different speed to fix this? | 19:46 |
| fungi | wait, no, quantum tunneling | 19:46 |
| tonyb | we could all move to NZ? | 19:46 |
| clarkb | I hear it is nice there | 19:46 |
| fungi | i would find that an acceptable solution as well | 19:47 |
| corvus | best suggestion so far | 19:47 |
| clarkb | but ya no easy solutions. Lets keep it at 19:00 for now and I can look at a map and see if anything jumps out as a good alternative while also enabling those who which to attend time to attend | 19:47 |
| fungi | i'm okay with alternating times too | 19:47 |
| fungi | spread the pain around | 19:48 |
| corvus | fungi: you'd have to adjust your meeting orrery | 19:48 |
| tonyb | I can do some research and dump it to an etherpad | 19:48 |
| clarkb | my concern with alternating times is that they are already so few of us that we might end up with two meetings with significantly fewer people | 19:48 |
| frickler | well the EU/AU overlap would be around 6 UTC likely | 19:48 |
| clarkb | frickler: that is approximately midnight where I am and 3am where fungi is so ya fungi's numebr above seems spot on | 19:49 |
| fungi | 0600z is 2am local for me at the moment, right. in about a month that will change to 1am | 19:49 |
| clarkb | clearly we should do more brainstorming. At least now we know where the interest is and can do more accurate brainstorming | 19:49 |
| clarkb | feedback welcome if/when we have a great idea on how to make this better :) | 19:50 |
| clarkb | #topic Open Discussion | 19:50 |
| clarkb | I know fungi had one item to discuss before we end the meeting so jumping to open discussion now | 19:50 |
| fungi | folks in the #openstack-kolla channel were talking about it being maybe nice if we could start mirroring the "osbpo" (debian openstack backports) package repository | 19:50 |
| clarkb | functionally that repo is very similar to Ubuntu Cloud Archive but for Debian? | 19:51 |
| frickler | yes | 19:51 |
| fungi | i realized we have a mirror.deb-openstack volume in afs which is 5 years stale, the predecessor of osbpo essentially | 19:51 |
| fungi | none of the files in there are of any use now | 19:51 |
| clarkb | I have no objections to mirroring it. Maybe we can get the kolla team to modify configs for mirror.deb-openstack to write out the current content? | 19:51 |
| fungi | size estimates for the whole of osbpo are around 22gb at the moment, but with reprepro we'd probably only mirror a fraction of that (like we don't care about openstack victoria packages for debian stretch) | 19:52 |
| fungi | i expect the amount of data would come out about the same as uca, that order of magnitude anyway | 19:52 |
| fungi | and yeah, i'll work with kolla folks on it, just making sure there were no immediate objections | 19:53 |
| clarkb | none from me. | 19:53 |
| frickler | if it is only for kolla, just bookworm and maybe bullseye would be enough | 19:53 |
| tonyb | seems like a reasonable ask to me | 19:53 |
| frickler | I fear this may end up as task for me, but no objection | 19:53 |
| fungi | frickler: yeah, i was going to start suggesting openstack bobcat packages for bookworm and then adding on if they realize they need more | 19:54 |
| frickler | ack | 19:54 |
| fungi | okay, that's all i needed to confirm | 19:54 |
| fungi | thanks! | 19:54 |
| tonyb | frickler: I'm willing to help learn if you're willing to instruct. | 19:55 |
| clarkb | Depending on how the next coupel of days go I may try to disappear on Friday. Kids have it randomly off (teacher inservice day or soething) and will attempt to take advantage. But I expect wednesday and thursday to be busy | 19:55 |
| tonyb | assuming infra-root isn't a prerequisite | 19:55 |
| clarkb | tonyb: root shouldn't be necessary for most of it | 19:55 |
| fungi | i can also provide pointers. it's pretty straightforward since we don't need to make any changes to afs other than delete some old files out of the volume for our own sake | 19:55 |
| clarkb | root may be required to delete the old files if reprepro can't be convinced to do it | 19:55 |
| fungi | tonyb: it'll be 100% config files, maybe a root sysadmin running the initial mirror pulse without a timeout just to make sure it populates safely | 19:56 |
| tonyb | that was my expectation just wanted to be clear | 19:56 |
| fungi | but logs are also public | 19:56 |
| fungi | so you can quickly see what's going wrong if something is | 19:56 |
| frickler | on the topic of being away, I'll be mostly offline tomorrow | 19:57 |
| fungi | thanks for the heads up | 19:57 |
| fungi | also i'll be absent the first half of next week | 19:57 |
| fungi | all things open runs sunday through tuesday so i'll be driving saturday and wednesday | 19:58 |
| clarkb | fungi: have fun! | 19:58 |
| tonyb | enjoy | 19:58 |
| fungi | but i should be around again thursday/friday next week | 19:58 |
| fungi | also i've got some family obligations that following weekend up into wednesday of ptg week, so will have limited availability then as well | 19:58 |
| clarkb | and that is all we have time for. Feel free to continue discussion on the mailing list or in #opendev | 19:59 |
| clarkb | Thank you everyoen for your time | 19:59 |
| fungi | thanks clarkb! | 19:59 |
| clarkb | #endmeeting | 19:59 |
| opendevmeet | Meeting ended Tue Oct 10 19:59:49 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-10-10-19.01.html | 19:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-10-10-19.01.txt | 19:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-10-10-19.01.log.html | 19:59 |
| tonyb | thanks all | 19:59 |
| frickler | o/ | 20:00 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!