| clarkb | just about meeting time. I suspect this willbe another short one | 18:58 |
|---|---|---|
| corvus | o/ | 18:59 |
| fungi | i have something for open discussion which came up during the openstack tc meeting, if there's time | 19:00 |
| * tonyb waves | 19:00 | |
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue May 28 19:00:35 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/HUXFNIX5ZHYFHSF4SRXSKN25P3CXES4E/ Our Agenda | 19:00 |
| clarkb | fungi: there should be plenty of time I suspect | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | The OpenInfra Summit CFP closes tomorrow. I'm not sure what timezone that is relative to but get your proposals in if you plan to submit one | 19:01 |
| clarkb | #topic Upgrading Old Servers | 19:02 |
| clarkb | Let's dive right in | 19:02 |
| clarkb | tonyb it sounds like you have learned things about the wiki migration/upgrade plan. Anything new you'd like to share? | 19:02 |
| tonyb | I have the basics fleshed out | 19:02 |
| tonyb | I should have the container stuff published this week and hopefully enough of the service-mediawiki stuff up that we can hold a node and start looking more carefully | 19:03 |
| clarkb | that sounds like good progress. Anything concerning other than the theme may have to go away? | 19:04 |
| tonyb | .... Unless I should also target Noble and just make mediawiki a real challenge | 19:04 |
| tonyb | I haven't gotten far enough to verify if all the extensions work but they load so that's something | 19:05 |
| clarkb | hopefully the container stuff isolates us quite a bit from the underlying platform | 19:05 |
| clarkb | so even if we deploy on jammy moving to noble later should be more straightforward | 19:05 |
| tonyb | Yup for sure. | 19:05 |
| tonyb | That's the main aim | 19:06 |
| tonyb | getting to a newer mediawiki is kinda secondary | 19:06 |
| fungi | we haven't tested deploying any servers on noble yet, so i wouldn't recommend gating this work behind figuring that out | 19:06 |
| clarkb | thats a good point too. | 19:07 |
| fungi | one problem (or set of problems) at a time | 19:07 |
| tonyb | Okay | 19:07 |
| tonyb | I'm hoping that cacti will be easy after this ;P | 19:07 |
| clarkb | anything we can do to help at this point or are we waiting for the held node to poke around on? | 19:07 |
| tonyb | I think its that one | 19:08 |
| clarkb | great I guess let us know when you get to that point. Anything else related to upgrading servers? | 19:08 |
| tonyb | I have some "local hacks" that let me spin up a VM and then run roles etc from system-config so it's going okay and slightly faster than doing everythign in the gate | 19:09 |
| tonyb | nothing else from me | 19:09 |
| fungi | as far as deploying things on noble, the first one should probably be an upgrade of something we've already got in containers | 19:09 |
| clarkb | fungi: or a mirror node | 19:09 |
| fungi | yep. just to limit the amount of effort involved | 19:09 |
| fungi | i wouldn't mix "make this work on noble" with "make this work in containers, and with ansible" was my point | 19:10 |
| clarkb | ++ | 19:10 |
| tonyb | Sounds fair | 19:10 |
| fungi | jammy has plenty of support lifetime still | 19:11 |
| clarkb | #topic AFS Mirror Cleanups | 19:11 |
| clarkb | Speaking of things with less lifetime I've sort of stalled on the xenial cleanup being distracted by things like gerrit and gitea upgrades | 19:12 |
| clarkb | however there is one more change in topic:drop-ubuntu-xenial that I think is landable now. Just need to recheck the change now that its depends-on has merged | 19:12 |
| clarkb | And then its back to pruning usage of xenial out where appropriate. I'll see if I find time for that soon (maybe tomorrow) | 19:13 |
| clarkb | #topic Gerrit 3.9 Upgrade | 19:13 |
| clarkb | #link https://etherpad.opendev.org/p/gerrit-upgrade-3.9 Upgrade prep and process notes | 19:13 |
| clarkb | If you haven't yet it would be great if you can take a look at this document to ensure my plan looks sane and doesn't have any obvious issues | 19:13 |
| clarkb | we have up to date image builds for 3.8 and 3.9 so we're upgrading from latest to latest of each release branch. And my concerns seem to be largely non issues after testing | 19:14 |
| clarkb | assuming nothing comes up before May 31 (Friday) at 1600 UTC I'll proceed with doing the upgrade then as previously announced | 19:14 |
| clarkb | The upgrade doesn't require any schema changes to the db and all reindexing can be done online. Should make the actual process relatively quick but I still allocated an hour in case we decide to rollback (which requires offline reindexing) | 19:15 |
| fungi | it looks good to me, but as previously mentioned i'll be in a car all day so can't help. take my evaluation with a grain of salt ;) | 19:16 |
| tonyb | I'll be around and and moved meetings out of the conflict zone | 19:16 |
| fungi | i do still think this is good timing relative to release cycles of the larger communities we're hosting | 19:16 |
| clarkb | ya I did cross check against the openstack release cycle and it looks clear. Starlingx just did a big release so expect them to be relatively quiet | 19:17 |
| fungi | if there are any short-term hitches, it shouldn't be too hard for them to absorb gracefully | 19:17 |
| fungi | but also the preliminary upgrade (and downgrade) testing has eliminated most concerns | 19:18 |
| clarkb | cool I peeked at 3.10 upgrade notes and I think the lucene stuff that 3.9 initially tripped over may now be somewhat intentional in the 3.10 release so that upgrade is likely to be more interesting than this one | 19:19 |
| clarkb | they say the indexes are automatically rebuilt when upgrading to 3.10 from 3.9 but I get the sense that isn't via normal online reindexing and its automatic offline reindexing instead | 19:19 |
| clarkb | but we'll figure that out in future testing. No need to worry about it now | 19:20 |
| clarkb | #topic Linaro Cloud Cert Renewal | 19:20 |
| clarkb | Just a reminder that this is still nearing its expiry. I think fungi mentioned possibly looking at it later as it became more urgent | 19:20 |
| fungi | yeah, i'm happy to serve as a last-minute backup if nobody else takes it | 19:21 |
| clarkb | thanks | 19:21 |
| tonyb | I can do it later today or tomorrow | 19:21 |
| clarkb | double thanks! | 19:21 |
| clarkb | #topic Gitea 1.22 Upgrade | 19:22 |
| clarkb | The long awaited Gitea 1.22 release has finally arrived (yesterday) | 19:22 |
| fungi | i'll stop gripping my seat now | 19:22 |
| clarkb | This change does include the updates to the doctor tool to fixup mysql dbs that we will want to apply to our databases | 19:22 |
| clarkb | s/change/release/ | 19:22 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/920580/ Change implementing the upgrade | 19:22 |
| clarkb | I've gone ahead and pushed a change up that tests the upgrade (and will perform the upgrade when merged) and most things seem to be working | 19:23 |
| clarkb | the appearance is changed slightly as part of some web css refactoring that I had to accomodate in our template overrides | 19:23 |
| clarkb | Anyway I plan to test the doctor tool on a held node for that upgrade before we upgrade anything. That said I suspect the plan will be to do the upgrade and then do the doctoring as separate steps | 19:24 |
| clarkb | since the doctor fixups are not strictly required for this upgrade we'll just continue to have the same case sensitivity problems until we fix things | 19:24 |
| clarkb | #link https://104.130.74.99:3081/opendev/system-config Held Node | 19:24 |
| tonyb | I did notice that 'code search' is back on the test server. I don't know if we want to hide that in css again | 19:24 |
| clarkb | this is the held node if anyone wants to take a look at the new gitea appearance or check things otherwise | 19:24 |
| clarkb | tonyb: were we intentioanlly hiding it before? | 19:25 |
| clarkb | https://opendev.org/explore/code I think its there today as well | 19:25 |
| tonyb | I thought so, as it was generally not "right" in our usecase | 19:25 |
| tonyb | Hmm okay I'll look again | 19:25 |
| fungi | i wasn't aware there was a code search link previously, maybe that's new? | 19:25 |
| clarkb | fungi: its in the current release too | 19:26 |
| fungi | it's always been reachable via the explore tab | 19:26 |
| clarkb | oh the difference is whether it is available in the in repo view | 19:26 |
| fungi | ahh | 19:26 |
| clarkb | I don't think it is a problem to have, we haven't removed codesearch because the gitea version is deficient but it isn't actively harmful as far as I can tell | 19:26 |
| tonyb | Okay. | 19:27 |
| clarkb | oh no its in the code repo view too just in a different spot (they really refactored the UI) | 19:27 |
| tonyb | Ooooo so it is | 19:28 |
| clarkb | But ya I'm not sure its a problem and I wasn't aware that we had taken any active steps to disable it. It was more of a "we can't remove codesearch until gitea works better with search" | 19:28 |
| fungi | right, in my opinion we've continued to operate hound as a fallback option | 19:28 |
| fungi | people can (and do) use either | 19:28 |
| clarkb | keep calling out unexpected changes though. This release doesn't have a ton of "breaking" changes listed but it does seem to have a lot of user facing changes | 19:29 |
| fungi | they both return slightly different results | 19:29 |
| fungi | i think the main deficiency in gitea's search is a lack of regex support, yeah? | 19:29 |
| clarkb | that and it doesn't reindex things aggressively so some stuff just doesn't show up in results | 19:30 |
| fungi | also that it doesn't return all matches within each repo for a cross-repo search, but that's probably more of a pet peeve of mine | 19:30 |
| clarkb | however, I wonder/suspect if the db problems are related to that and fixing up the db might fix a number of these weird slightly broken behaviors we've seen | 19:30 |
| fungi | definitely worth investigating further | 19:30 |
| clarkb | I'll dig into this upgrade more after gerrit is done | 19:31 |
| clarkb | until then let me know if you notice anything weird on the held node | 19:31 |
| clarkb | #topic Open Discussion | 19:31 |
| clarkb | Anything else? | 19:31 |
| fungi | the openstack tc is working on removing individual collaborators on pypi projects | 19:32 |
| * gouthamr is lurking | 19:32 | |
| fungi | the pypi interface for doing that is clunky and there is no programmatic api available for it, so removing hundreds of collaborators on openstack deliverables is a lot of work clicking things | 19:33 |
| fungi | one option would be to temporarily share access to the openstackci pypi account with openstack tc members so they can share the workload | 19:33 |
| fungi | though thinking through it, that will also be somewhat hindered by the new mfa requirement for pypi | 19:34 |
| fungi | since they would each need some way to access multi-factor auth credentials to that account | 19:35 |
| gouthamr | ah; bummer | 19:35 |
| clarkb | two concerns: the account is used for more than just openstack (due to zuul multitenancy migrations being half done) and there is no way to promote an existing user like we can with gerrit group membership (for example we in the past promoted tonyb temporarily for brnach management before we grew the features and acls to just delegate that properly) | 19:35 |
| fungi | the fallback option is that tact sig members who have access to our openstackci account do all the work | 19:35 |
| clarkb | this means that once we share the credenitals the only way to unshare them is to rotate credentials | 19:35 |
| fungi | yes, we'd presumably need to change the password and re-enroll the mfa with a new secret | 19:36 |
| clarkb | I think I'm generally ok with the idea given previous escalations of privileges to people like tonyb for other tasks as long as we keep the group small like that. But am concerned that the amount of work rotating and testing and fixing things after the fact is worse than just clicking buttons | 19:36 |
| clarkb | fungi: yes and then update zuul configs and test that it all still works with new new stray newlines involved | 19:36 |
| fungi | well, the zuul configs should be using api keys now | 19:36 |
| clarkb | if we could do this programmatically then we could create a token that we revoke I think, but I guess there is no way to do that via a token and requires an actual user session? | 19:37 |
| clarkb | fungi: I'm operating under the assumtion we would have to roll those too | 19:37 |
| clarkb | but maybe that is a bad assumption | 19:37 |
| fungi | yeah, from what i gather there's an open warehouse feature request to add api methods for collaborator maintenance | 19:37 |
| fungi | i don't think password and mfa changes invalidate api keys that have been issued, but i suppose it's possible | 19:38 |
| clarkb | maybe we can test that with a different account and if it doesn't invalidate/force rolling them due to exposure then the effort involved is likely much less than I feared | 19:39 |
| clarkb | then we can get a small set of volunteers for a small time window before we roll the main credentials. The other concern I have is that it is difficult to audit this stuff (again due to the lack of groups and separate accounts) so keepign the window involved small helps | 19:40 |
| corvus | so... the end state would be that only the opendev credential has owner or maintainer status? | 19:41 |
| fungi | yes, for official openstack deliverables | 19:41 |
| corvus | i feel like that is slightly misaligned | 19:41 |
| fungi | it would probably be an improvement for openstack to also have more direct control over these packages | 19:42 |
| corvus | (in that it means that opendev is on the hook for any maintenance of the ownership list (including this situation)) | 19:42 |
| corvus | would it be appropriate for openstack to have its own (essentially never-used) credential for this? | 19:42 |
| fungi | the way i see it for now is that the openstack tact sig and opendev sysadmins are sharing that responsibility (and account), but i can see room for improvement | 19:43 |
| clarkb | corvus: yes that could possibly work, but we'd still need to help in the transition work I think | 19:43 |
| corvus | hrm, so other tenants should get their own accounts? | 19:43 |
| clarkb | but in theory we'd do it once then be done | 19:43 |
| fungi | making a new account for openstack raises the question of who stores and safeguards the password and mfa secret | 19:43 |
| tonyb | Can we simulate the clicky clicky with something ike selenium? | 19:43 |
| clarkb | corvus: I think long term that would be ideal. THe current account is called "openstackci" iirc | 19:44 |
| clarkb | yes confirmed | 19:44 |
| fungi | tonyb: possibly, but https://xkcd.com/1205 | 19:44 |
| corvus | yeah... i'd be more comfortable with opendev being the only holder of these credentials if there were some low-impact way of us doing that maintenance (ie, script) | 19:44 |
| fungi | our irc accessbot is probably the closest example we have currently | 19:45 |
| frickler | I can check until next week how much work this actually would be and whether I could volunteer to execute it in a reasonable time frame | 19:45 |
| corvus | agree, and that's pretty self-service | 19:45 |
| tonyb | fungi: fair point | 19:45 |
| fungi | frickler: it's about 3 clicks per package plus re-entering the package name, and potentially several seconds wait between each operation | 19:46 |
| fungi | er, 3 clicks per collaborator on a package, plus re-entering the collaborator's username i mean | 19:47 |
| fungi | on each package | 19:47 |
| frickler | but usually there is only one other co-owner or multiple ones? do we know? | 19:47 |
| fungi | pypi has a "are you really sure, type the name into this text input box to confirm you know what you're doing" step | 19:47 |
| clarkb | I think we can get that programmitcally as you can list all of the maintainers + owners together (it doesn't distinguish them though) | 19:47 |
| fungi | frickler: on some packages there are more than one | 19:47 |
| fungi | gouthamr has a list | 19:48 |
| gouthamr | yes; i have the extra-maintainers in a yaml dump here: | 19:48 |
| gouthamr | #link https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L52 | 19:49 |
| clarkb | side note looking at that list you should probably sort it by priority | 19:49 |
| clarkb | a number of those are pacakges we don't care about much anymore so are less important | 19:49 |
| frickler | so I can just start to work and see how it goes. and then judge how many weeks I think it might take me to go through gathering enough concentration in between | 19:49 |
| frickler | and that would avoid all issues with creds rotation being needed | 19:51 |
| fungi | also, to be clear, the triggering event for this effort was that a historical collaborator on a package which was considered to be under the governance of openstack decided to push their own release for that project outside of what code was being maintained in gerrit, and the interest in preventing that in the future was amplified somewhat by the xz-utils compromise more recently | 19:51 |
| corvus | ballpark that's maybe ~175 people to remove? | 19:51 |
| clarkb | and more recently people have asked people who have moved on from poenstack to take over openstack packages because they are still listed on the pypi package | 19:51 |
| clarkb | anyway seems like frickler putting some effort into it to get a sense of the scale is a good first step | 19:53 |
| corvus | i agree with the effort; just not sure i love the idea that if anything comes up, opendev admins may be on the hook for updating pypi as packages are removed from openstack official governance. | 19:53 |
| frickler | gouthamr: do we actually care about (possibly soon to be) retired projects? | 19:53 |
| frickler | corvus: well that's the case already for the majority of the projects anyway? | 19:54 |
| corvus | i have no better ideas given the current constraints; only a desire to change the constraints. :) | 19:54 |
| fungi | corvus: i'm happy to reframe that as openstack tact sig representatives within the opendev sysadmins will take on those tasks as they arise, at least until we come up with a more self-service option like accessbot | 19:54 |
| clarkb | corvus: what sorts of updates are you concerned about? I think much of the actual package properties and attributes can be driven through the release process. The main bit that can't is the maintainers/owners aiui | 19:54 |
| corvus | the only concrete example i can think of is "hey opendev, can you add <username> to <project> because it's being released from openstack governance". | 19:56 |
| clarkb | and yes I agree getting us out of this business is ideal, but currently we're stuck in it one way or another (we could add a second account for openstack to manage things with but that would flow through the openstackci account anyway) | 19:56 |
| clarkb | corvus: ack. One option there may be to say you can't publish to the old name on pypi. But that may be problematic for other reasons (needing to update deps everywhere for example) | 19:56 |
| fungi | yeah, i mean, the majority of official openstack packages were *already* in this situation | 19:56 |
| clarkb | ya I think the current proposal is still an improvement if not the ideal state. And we can revisit if we need to take additional steps to get closer to that ideal | 19:57 |
| fungi | because an initial upload of a release on pypi creates the project there and adds the uploader as the only owner | 19:57 |
| corvus | to be clear, i'm not objecting to this work, or to frickler, or anyone else doing it. just noting that we may be signing up for more work in the future depending on openstack project lifecycle, and that this is not as self-service as we would like. | 19:59 |
| clarkb | we are just about at time now. I think it is reasonable for frickler to investigate the effort required to do this directly. Then next week at the TC meeting we can discuss the idea of package ownership having a secondary account to move us out of the dependency tree | 19:59 |
| clarkb | corvus: ++ | 19:59 |
| fungi | so anyway, the current state seems to be that frickler and i can look into sharing the work the tc is requesting, and separately we can brainstorm something like accessbot to create and add accounts on pypi (though the lack of an api for that in warehouse makes that a bit of a challenge at present) | 19:59 |
| clarkb | fungi: or just do another pass of adding openstackowner to all the packages | 20:00 |
| fungi | yeah | 20:00 |
| fungi | which i can also volunteer for if needed | 20:00 |
| clarkb | then we can state "openstackci" is only used for publication. Other actions like changing owners when packages retire within openstack should be handled by the openstackowner account | 20:00 |
| clarkb | and we are at time. Thank you everyone! | 20:00 |
| tonyb | Thanks all | 20:00 |
| corvus | and if there ever is an api, we can change the name of the publishing account :) | 20:01 |
| clarkb | I'm going to end it here since I need to eat lunch, but feel free to continue discussion in our other venues if you'd like to finish up on any of these thoughts | 20:01 |
| clarkb | corvus: ++ | 20:01 |
| clarkb | #endmeeting | 20:01 |
| opendevmeet | Meeting ended Tue May 28 20:01:25 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2024/infra.2024-05-28-19.00.html | 20:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-05-28-19.00.txt | 20:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2024/infra.2024-05-28-19.00.log.html | 20:01 |
| fungi | yes, definitely | 20:01 |
| * gouthamr responds to frickler late.. | 20:04 | |
| gouthamr | frickler: soon-to-be retired ones, yes | 20:04 |
| gouthamr | thanks for the discussion here folks.. agree bringing up the "openstackowner" change in the TC room next week.. | 20:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!