| ianychoi | Hi team, would it be possible to get a backup of translate.openstack.org database? I have two main reasons: 1/ I want to move forward on Weblate migration by analyzing manual data and 2/ Now I cannot approve Korean language team members after I delete one Zanata user.. I need to double-check some table/data integrity issues. | 10:37 |
|---|---|---|
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Remove networking-midonet https://review.opendev.org/c/openstack/project-config/+/919415 | 11:01 |
| tonyb | ianychoi: we can do that. safer to give you a copy of the backup than do live investigation. | 11:02 |
| fungi | though it looks like the local db backups on the server are empty | 13:54 |
| fungi | the database we're dumping definitely has content and the defaults file we tell mysqldump to use works to give me access with mysqlclient | 13:56 |
| fungi | mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces | 13:57 |
| fungi | i think that has something to do with it | 13:58 |
| fungi | looks like it's because we're trying to back up all databases with the zanata user even though it only has access to the zanata db | 14:00 |
| fungi | if i drop the --all-databases option and specify the "zanata" database, i get the dump i expect from it | 14:01 |
| fungi | though the tablespaces error is somewhat orthogonal, i have to add --no-tablespaces to silence that | 14:02 |
| fungi | unrelated... infra-root: config-core: i've self-approved https://review.opendev.org/954374 (Remove CLA enforcement from all projects and lock) since we said today is when we were going to do the other gerrit config changes depending on it | 14:06 |
| fungi | okay, back to the zanata db backup topic, it looks like the exact edits described above are already implemented for the stream backup borg is doing, so it looks like our remote copies are fine, it's just the local dumps that aren't | 14:19 |
| opendevreview | Merged openstack/project-config master: Remove CLA enforcement from all projects and lock https://review.opendev.org/c/openstack/project-config/+/954374 | 14:20 |
| fungi | aha, this is apparently already known, and covered in the code comment at https://opendev.org/opendev/system-config/src/commit/a88a6f5/modules/openstack_project/manifests/translate.pp#L133-L138 | 14:21 |
| fungi | so anyway, if we're going to hand off a db dump, it'll either have to be fetched from one of the backup servers or use mysqldump to make a one-off with the options described there | 14:22 |
| Clark[m] | fungi: I'm slowly rolling in (doing system updates now). Once clas are removed from projects the next step is updating all projects acls to remove the clas themselves. Do we want to do one at a time and confirm that the accepted target group isn't removed or otherwise affected? | 14:28 |
| Clark[m] | I think that info may still be relevant going forward. I don't expect that removing the cla itself will affect the group existence | 14:28 |
| fungi | well, that's all stored in notedb in git repos anyway, right? so if it does remove them then there will be a transaction history regardless, so if we still need the data in the future we can pull it from there anyway | 14:30 |
| Clark[m] | Good point. I believe this is the case | 14:30 |
| Clark[m] | Looks like manage projects may be slow again (likely due to an overwhelmed gitea but I haven't checked yet) | 14:35 |
| clarkb | gitea13 does have somewhat high load (not the highest I've seen but high enough to indicate it is probablythe thing slowing down manage-projects) | 14:46 |
| clarkb | acls should be updated in gerrit now though (gitea management is done according to the playbook log) | 14:46 |
| fungi | Build succeeded (deploy pipeline). | 14:56 |
| fungi | all good | 14:56 |
| clarkb | I guess next step is manually applying https://review.opendev.org/c/opendev/system-config/+/954376/2/doc/source/gerrit.rst to all projects then we can approve that change? | 15:00 |
| clarkb | I've reapplied my +2 to that change and I think I'm ready for the manual updates and change approval if you are | 15:04 |
| clarkb | then once we're happy with the results of that we can consider if we also want to proceed with the last omnibus gerrit image update: https://review.opendev.org/c/opendev/system-config/+/882900 before restarting the service to pick upthe changes | 15:04 |
| fungi | yeah, i'll work on applying the contributor-agreement config removals... i guess doing it through the ui is still the simplest approach? or have we been pushing config changes like that into git lately? | 15:07 |
| clarkb | fungi: I think if you go through the web ui it forces you to code review the chagnes now | 15:07 |
| fungi | ah, it's admittedly been a while | 15:07 |
| clarkb | which is doable but you may have to modify your group membership to review and approve such a change. I think when hashtags were updated globally corvus pushed to all projects directly for that | 15:08 |
| fungi | yeah, looks like i did that a couple of years ago for receive.rejectImplicitMerges | 15:09 |
| corvus | yeah, the web ui didn't work for some reason; i think we need some permissions changes for that... we should probably do it, but it's a diversion. i burned some cycles on it last time before just pushing like we usually do | 15:09 |
| fungi | mainly i just have to remember the summoning and binding incantations for pushing to the right ref in all-projects | 15:10 |
| corvus | i'm sure i left details in the irc log | 15:10 |
| corvus | (about the red herring of web ui review... i probably didn't do anything useful like leave the correct incantation) | 15:10 |
| fungi | git fetch origin refs/meta/config && git checkout FETCH_HEAD [...] git push origin HEAD:refs/meta/config | 15:13 |
| fungi | that's the bookends | 15:13 |
| fungi | HEAD is now at 283e0b0 Allow any gerrit user to edit hashtags | 15:14 |
| fungi | that meets expectations | 15:14 |
| fungi | infra-root: this is what i'm preparing to push https://paste.opendev.org/show/btHxpa3asThwid10tfht/ | 15:18 |
| clarkb | that diff looks correct to me | 15:19 |
| fungi | i corrected some typos in the commit message just now but otherwise that's what i'm planning to push | 15:20 |
| fungi | i think i'll have to grant my admin account additional permissions | 15:22 |
| clarkb | yes I think you need to add your admin account to project bootstrappers temporarily | 15:22 |
| fungi | 283e0b0..76473b6 HEAD -> refs/meta/config | 15:24 |
| fungi | #status log Manually injected project.config edits indicated in https://review.opendev.org/954376 for CLA removal from Gerrit | 15:25 |
| clarkb | I wonder if it validates that nothing is using the cla at this point. We believe that to be the case so unlikely to get an error if they do validate it | 15:25 |
| opendevstatus | fungi: finished logging | 15:25 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/954376 is next once we're happy that cla removal hasn't caused problems. Not sure what we would check though. Pushing code to a project that long used the cla to ensure we didn't leave some vestigal config laying around? | 15:26 |
| fungi | looks like we've got a bit of documentation and example cleanup to do still, but nothing urgent: https://codesearch.opendev.org/?q=requireContributorAgreement | 15:27 |
| fungi | and none of those repos required a cla, so can't really use them to test things regardless | 15:28 |
| fungi | Jul 18 15:29:47 eavesdrop01 docker-gerritbot[646]: 2025-07-18 15:29:47,538 INFO gerritbot: Sending "Dan Smith proposed openstack/nova master: WIP: Parallelize s-g generators https://review.opendev.org/c/openstack/nova/+/955091" to #openstack-nova | 15:30 |
| fungi | good enough? | 15:30 |
| clarkb | ya I suspect that is about as good as we'll get | 15:30 |
| clarkb | I've noticed that I don't think 954376 will trigger infra-prod-run-review. But that is ok since all it does is update the conatiner image which we manually deploy for gerrit anyawy. Also the quay followup would trigger infra-prod-run-review if we proceed with that one and bundel up all this into a single restart | 15:33 |
| clarkb | should I approve 954376 or do you want to do that? | 15:33 |
| fungi | go for it | 15:33 |
| clarkb | done | 15:34 |
| fungi | thanks! | 15:35 |
| clarkb | I'll let others weigh in on https://review.opendev.org/c/opendev/system-config/+/882900 to move gerrit images to quay.io while we wait on that change | 15:35 |
| fungi | yeah, looks like it hasn't changed since i last reviewed it 3 months ago | 15:36 |
| clarkb | ya I think the main thing there is confirming we're in a spot where we are comfortable to make the switch (should be with gerrit running on podman now) and that nothing has changed since originally pushed/updated and now that would impact the change (rechecking it and having it come back green is a good indication that this is also fine) | 15:39 |
| clarkb | but if others think its still a good idea and the change itself is sound we should probably approve that soonish as well. | 15:40 |
| opendevreview | Antoine Musso proposed opendev/git-review master: Command to delete applied local branches https://review.opendev.org/c/opendev/git-review/+/955094 | 15:45 |
| clarkb | other than the planned gerrit updates is there anything else I should be looking at or catching up on this morning? Seems like it was fairly quiet other than the unexpected gerrit shutdown tuesday morning? | 15:49 |
| corvus | clarkb: on the niz front https://review.opendev.org/q/hashtag:+opendev-niz+status:open has 2 interesting things: | 15:52 |
| corvus | 1) the changes to clean up the bionic arm64 stuff | 15:52 |
| corvus | 2) the changes to remove nodepool | 15:52 |
| clarkb | looks like the bionic arm64 ozj cleanup is still hung up on the openstack/requirements unmaintained branches | 15:53 |
| corvus | for item #1, i think the state is we're waiting for those depends-on changes to merge | 15:53 |
| corvus | yeah, so to progress that, those changes either need to merge, or we need to time out and say we waited long enough, and just merge https://review.opendev.org/954761 without the dependencies | 15:54 |
| corvus | (we can do that; it will just add some config errors to the relevant repos) | 15:54 |
| corvus | i did merge the use-nodepool switch, so missing labels are now config errors | 15:54 |
| corvus | (i updated 954761 to account for that, so it removes the xenial labels instead of making them "xenial-invalid" like we had before) | 15:54 |
| clarkb | that is good to know. Let me review the changes I haven't reviewed yet and I know fungi was trying to get feedback from elodilles_pto on cleaning up the unmaintained branches. However _pto indicates that maynot happen quickyl I suspecy | 15:55 |
| corvus | for item #2 -- merging that is not something for today, but reviewing those 2 changes so that they're ready to merge next week would be good | 15:55 |
| corvus | fungi: you reviewed the child, but you may want to look at the parent too: https://review.opendev.org/955228 removes the nodepool docs | 15:56 |
| corvus | (from system-config) | 15:56 |
| clarkb | corvus: https://review.opendev.org/c/opendev/zuul-providers/+/951018 appears to be in merge conflict as well (its not part of 1 or 2) | 15:56 |
| fungi | aha, thanks | 15:56 |
| corvus | clarkb: yeah, i'll catch up on that later; not important right now. | 15:57 |
| corvus | i think i'll shut down the nodepool servers tomorrow when i can better keep an eye on them. no changes needed for that -- that'll just be a manual docker compose down | 15:58 |
| clarkb | corvus: did you want to weigh in on https://review.opendev.org/c/opendev/system-config/+/882900 as part of the gerrit omnibus update? | 15:58 |
| clarkb | corvus: sounds good re nodepool shutdown. I would put the servers in the emergency file list as I think ansible may start them up again | 15:58 |
| corvus | (to recap, at this point zuul should not be interacting with nodepool at all, which is why it's safe to shut the servers down any time, and then delete them some time after that) | 15:59 |
| corvus | clarkb: ack re emergency file | 15:59 |
| clarkb | corvus: https://review.opendev.org/c/opendev/system-config/+/955229/2/playbooks/roles/zuul-executor/tasks/main.yaml the change to add the key material under a new name needs a zuul executor restart to pick up right? But we're not actually changing the content and we don't remove the old file so things should transition gracefully | 16:02 |
| clarkb | (just want to make suer that is the expectation and intention) | 16:02 |
| corvus | clarkb: 882900 +2 with comment -- will leave it up to you to decide how important that is -- but also, i can't remember if our scripts actually change that value, so the initial value may be difficult to change? | 16:03 |
| corvus | clarkb: re key exactly -- material is the same | 16:04 |
| corvus | (also, that change is complete in private hostvars) | 16:04 |
| clarkb | ack thanks | 16:04 |
| clarkb | I'll finish this review of the nodepool cleanup change then update the gerrit image descriptions for quay | 16:05 |
| clarkb | corvus: is there a openstack/project-config cleanup to fix the zuul config error on 955229 ? I'm not seeing that one yet | 16:06 |
| corvus | think it merged; one sec | 16:07 |
| corvus | https://review.opendev.org/955235 | 16:08 |
| corvus | i rechecked 229 | 16:08 |
| clarkb | thanks | 16:08 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Drop devstack-gate documentation https://review.opendev.org/c/opendev/system-config/+/955386 | 16:09 |
| fungi | noticed that ^ when reviewing the nodepool docs removal | 16:09 |
| corvus | https://review.opendev.org/q/hashtag:+opendev-niz+status:merged if you want to see other stuff over the last few days | 16:09 |
| corvus | i did a bunch of cleanup | 16:09 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Migrate gerrit images to quay.io https://review.opendev.org/c/opendev/system-config/+/882900 | 16:11 |
| frickler | clarkb: corvus: as quasi-reqs-core I'd say to force-merge the open changes there for the unmaintained branches. I've still enabled my acc after the devstack cleanup so I can just do it? | 16:11 |
| clarkb | fungi: corvus ^ now with better repo descriptions | 16:11 |
| clarkb | frickler: ya I think from the opendev side of things we're hapyp for that as the changes are purely mechanical and don't impact the content of the repo as it were | 16:11 |
| clarkb | frickler: so if openstack/requirements are also happy with that approach I am too | 16:12 |
| corvus | sgtm | 16:14 |
| fungi | agreed | 16:14 |
| frickler | done, except for https://review.opendev.org/c/openstack/requirements/+/954774 which seems to be in normal progress now | 16:23 |
| clarkb | the cla cleanup change should merge in a few minutes. I guess if the quay change passes check testing I'll go ahead and approve it too | 16:29 |
| clarkb | then after that change lands and deploys we can start on a gerrit restart to switch the image source and pull the latest updates | 16:30 |
| opendevreview | Merged opendev/system-config master: Stop installing and configuring CLAs https://review.opendev.org/c/opendev/system-config/+/954376 | 16:44 |
| opendevreview | Merged opendev/system-config master: Remove nodepool documentation https://review.opendev.org/c/opendev/system-config/+/955228 | 16:44 |
| clarkb | fungi: I'm guestimating that the earliest we could restart gerrit is around 1900 UTC if we wait for the quay move | 16:47 |
| fungi | that's fine by me, i'll still be around | 16:52 |
| fungi | going to push up a few changes to clean up requireContributorAgreement references so they're less likely to get cargo-culted | 16:52 |
| opendevreview | Jeremy Stanley proposed opendev/infra-manual master: Replace CLA references with DCO https://review.opendev.org/c/opendev/infra-manual/+/955391 | 17:04 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Drop a CLA reference in jeepyb docs ACL example https://review.opendev.org/c/opendev/system-config/+/955392 | 17:07 |
| fungi | those are the main ones | 17:07 |
| fungi | there's also a test fixture in gerritlib that expressly turns requireContributorAgreement off, and similar in zuul's quickstart test gerrit setup, which i didn't bother with as they're not enabling it | 17:09 |
| fungi | and in an airship gerrit deployment script too | 17:10 |
| fungi | but nothing else in any repos we host that turns it on, once the above changes merge | 17:10 |
| clarkb | double checking build logs for 882900 I realized that quay.io/opendevorg/gerrit-base:latest does already exist but is a couple years old from our previous attempt | 17:18 |
| clarkb | but looking at the build logs I'm pretty sure that the gerrit 3.10 image build is pulling the gerrit-base image from the intermediate registry properly | 17:18 |
| clarkb | https://zuul.opendev.org/t/openstack/build/19b9f9f95ab9430383e5105e91fe7345/log/job-output.txt#1342-1365 this is the build side pulling the image and the hashes there seem to line up with the hashes on the gerrit-base image build side here: https://zuul.opendev.org/t/openstack/build/555df3b64b444858ad0318d329141bc7/log/job-output.txt#2681-2727 | 17:19 |
| fungi | yeah, we had a brief attempt to use it from quay back then | 17:20 |
| clarkb | I'm going to approve the change now as the only job remaining is gitea so unaffected by the move | 17:20 |
| fungi | sounds good, thanks | 17:20 |
| clarkb | but if others want to confirm that we appear to be building the image with non stale data that would be appreciated | 17:20 |
| clarkb | oh hrm there was a gerrit 3.10.7 release | 17:21 |
| fungi | of course there was | 17:27 |
| fungi | do we want to hold up for that? | 17:27 |
| fungi | i could go either way | 17:27 |
| clarkb | I'm working on a change for it now | 17:27 |
| clarkb | but looking at the changelog in https://gerrit.googlesource.com/plugins/replication/+/refs/tags/v3.10.7 I don't think any of those ~3 changes are critical for us | 17:28 |
| clarkb | and we don't use webhooks at all so the updates there shouldn't affect us at all | 17:28 |
| fungi | yeah, i was just pulling that up | 17:28 |
| clarkb | also note that the way we build gerrit proper we're going to update gerrit proper | 17:30 |
| clarkb | but the plugins are pinned to tagged versions | 17:30 |
| clarkb | https://www.gerritcodereview.com/3.10.html#3107 is the release notes for gerrit proper | 17:30 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gerrit images to 3.10.7 and 3.11.4 https://review.opendev.org/c/opendev/system-config/+/955395 | 17:32 |
| fungi | yeah all that looks fine to me | 17:32 |
| clarkb | I couldn't rebase things because that will restart the testing of 882900 so I did a depends on instead | 17:32 |
| fungi | wfm | 17:33 |
| clarkb | 955395 doesn't appear to trigger the gitea job so I think we could sneak it in today without losing a bunch of time if we think that is safest | 17:34 |
| fungi | i'm okay with it either way, and can stick around later if needed too, i don't have any evening plans other than cooking dinner at some point | 17:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gerrit images to 3.10.7 and 3.11.4 https://review.opendev.org/c/opendev/system-config/+/955395 | 17:35 |
| clarkb | actually I realized that I should force an image rebuild for publication purposes and edited the change. Not sure if gitea is triggered now | 17:35 |
| clarkb | nope that still doesn't cause it to do gitea things | 17:36 |
| clarkb | fungi: I think the main reason to try and get 955395 in as well is simply to ensure the plugins all match the gerrit core version. There may be something we don't understand in that replication change log that is important (though that seems unlikely) | 17:37 |
| clarkb | and ya its only 10:37am local time here. Maybe we should just get this out of the way too. Otherwise its another wait for a good time to restart gerrit loop | 17:37 |
| clarkb | so I think I'm leaning towards including 955395 in the restart as well | 17:37 |
| fungi | i'm good with that plan | 17:39 |
| fungi | and already +2 on the change | 17:40 |
| fungi | i guess i can go ahead and approve it | 17:40 |
| clarkb | its interesting that in the zuu.l queue it still says it is waiting on repo state | 17:40 |
| clarkb | I wonder fi that depends on has confused something? | 17:41 |
| clarkb | though it has the jobs figured out so in must've merged something | 17:41 |
| clarkb | now it is running jobs. I just needed to be patient | 17:47 |
| fungi | intermittent failure pulling haproxy container images? https://zuul.opendev.org/t/openstack/build/be5d9e300ed7429799840d594a2cf444 | 17:58 |
| fungi | hard to tell from the log whether it was haproxy or haproxy-statsd that hit it | 17:59 |
| clarkb | you may be able to tell by looking up the blob hash to see which image it belongs to. We're mirroring haproxy to quay.io but haproxy-statsd is still on docker hub | 18:07 |
| clarkb | I just pulled both locally and they both worked | 18:09 |
| clarkb | 3da95a905ed5 belongs to haproxy | 18:09 |
| fungi | ah okay | 18:09 |
| clarkb | maybe a hiccup with quay ? | 18:10 |
| clarkb | or possibly a race with our mirroring jobs? THough those run periodically and should've been done for hours by the time this job ran | 18:10 |
| mnasiadka | While we’re at mirroring - any chance to get reviews on https://review.opendev.org/c/opendev/system-config/+/954703 ? | 18:19 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/954978 is the other change on my backlog for this week. This one checks afs is mounted before starting zuul executor containers during weekly updates. Not sure that is urgent but probably worth doing at some point | 18:19 |
| opendevreview | Merged opendev/system-config master: Migrate gerrit images to quay.io https://review.opendev.org/c/opendev/system-config/+/882900 | 18:35 |
| opendevreview | Merged opendev/system-config master: docker-mirror: Add Ubuntu 24.04 and Debian Bookworm/Trixie mirrors https://review.opendev.org/c/opendev/system-config/+/954703 | 18:36 |
| clarkb | the quay.io image builds promoted to quay.io and skimming https://quay.io/repository/opendevorg/gerrit/manifest/sha256:92c6933f69e3c0c5cca3013453308de11fb9f36288cbaf8839d71d71a823072d the python version is 3.12 in there so I'm like 99% certain we used the correct base image to build that image | 18:40 |
| clarkb | at this point I think we could restart with the original planned set of changes. The 3.10.7 change just entered the gate though. SO maybe we aim for more of a 2000 UTC restart cc fungi | 18:41 |
| clarkb | I am happy that my original estimate was reasonably accurate until we added another change to the list | 18:41 |
| clarkb | that gives me time to eat lunch first too so I'm happy | 18:41 |
| opendevreview | Merged opendev/system-config master: Wait for AFS to mount when rebooting executors https://review.opendev.org/c/opendev/system-config/+/954978 | 18:48 |
| fungi | sounds fine to me | 18:51 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Fix indentation on Gitea splash page https://review.opendev.org/c/opendev/system-config/+/952407 | 19:17 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Hyperlink service icons on Gitea splash page https://review.opendev.org/c/opendev/system-config/+/952408 | 19:17 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Link Rackspace donor logo to testimonial article https://review.opendev.org/c/opendev/system-config/+/952861 | 19:17 |
| fungi | i expect those ^ should probably be squashed once people are satisfied with them individually, so as to not churn our gitea services with unnecessary restarts on deploy | 19:22 |
| clarkb | ++ | 19:23 |
| clarkb | manage projects for 882900 is still running. load on gitea13 is similar to what it was before so I'm not super concerned it should eventually complete | 19:24 |
| clarkb | but it has created a small deploy traffic jam we may wish to wait for completion of before starting gerrit restarts later (assuming they aren't done by then) | 19:25 |
| fungi | sure, makes sense | 19:25 |
| fungi | deploy reported for 882900 (success) | 19:32 |
| opendevreview | Merged opendev/system-config master: Update gerrit images to 3.10.7 and 3.11.4 https://review.opendev.org/c/opendev/system-config/+/955395 | 19:35 |
| clarkb | looks like everything has deployed according to zuul | 19:45 |
| clarkb | pulled the image locally which matches what I see in quay.io and it appears to have the correct content in it | 19:49 |
| clarkb | (I just checked the timestamp on the python3 binary and its newer than the years old old base image on quay) | 19:49 |
| clarkb | fungi: so I think our process for restarting gerrit is going to be `docker compose pull; docker inspect $imageid and confirm it matches latest on quay ; then send an announcement; docker compose down ; move gerrit waiting replication queue dir aside / delete it; delete large gerrit h2 caches ; docker compose up -d ; then after things are running initiate reindexing of changes | 19:50 |
| fungi | assuming we're going to want a root screen session on review, i've started one | 19:51 |
| fungi | and that sequence lgtm, i can start the pull | 19:51 |
| clarkb | gerrit_file_diff and git_file_diff appear to be the caches to be worried about this time | 19:51 |
| clarkb | fungi: I've attached to the screen. | 19:52 |
| fungi | quay.io/opendevorg/gerrit 3.10 2e7da5290a2d 54 minutes ago | 19:52 |
| fungi | previous one from dockerhub looks like 50502a8647b5 | 19:53 |
| clarkb | fungi: ya so if you `docker inspect 2e7da5290a2d | grep RepoDigests -A 1` that should show you a digest that matches https://quay.io/repository/opendevorg/gerrit/manifest/sha256:c931c215a837a0bd51133eb57f3ef6fbc930300e177c746ed4d5ccca9d3db8ec | 19:53 |
| fungi | quay.io/opendevorg/gerrit@sha256:c931c215a837a0bd51133eb57f3ef6fbc930300e177c746ed4d5ccca9d3db8ec | 19:54 |
| fungi | yep | 19:54 |
| clarkb | and ya the current container appears to be running off of 50502a8647b5 (noting that in case we have to revert) | 19:54 |
| clarkb | cool that is also the image I checked looking and looked correct after a brief a glance | 19:55 |
| clarkb | so I think I'm ready to proceed whenever you are. Do we want ot wait for the hourly jobs to run in about 5 minutes first? | 19:55 |
| fungi | status notice The Gerrit service on review.opendev.org will be offline briefly for a configuration and version update, but should return to service momentarily | 19:55 |
| fungi | something like that? ^ | 19:56 |
| clarkb | ++ | 19:56 |
| fungi | and yeah, let's wait for hourlies to finish | 19:56 |
| fungi | and they're enqueued now | 20:01 |
| clarkb | shouldn't be long | 20:02 |
| clarkb | the queued command looks correct to me | 20:03 |
| clarkb | those cache names seem to match the two I pasted above in here | 20:03 |
| fungi | cool, it's apparently basically the same thing i ran for a restart in may, sans the date on the file copy | 20:03 |
| fungi | should i go ahead and do the status notice now or wait for the hourly jobs to get closer to done? | 20:04 |
| clarkb | ya I think we can send it now | 20:04 |
| clarkb | also we need to be careful about the difference between opendevorg/gerrit and opendevmirror/gerrit | 20:05 |
| fungi | #status notice The Gerrit service on review.opendev.org will be offline briefly for a configuration and version update, but should return to service momentarily | 20:05 |
| opendevstatus | fungi: sending notice | 20:05 |
| clarkb | in this case I believe we've got the correct string and are using opendevorg's gerrit not the gerrit gerrit mirrored in opendevmirror | 20:05 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline briefly for a configuration and version update, but should return to service momentarily | 20:05 | |
| clarkb | I just wanted to call that out as a potential mixup gotcha | 20:05 |
| fungi | yeah | 20:05 |
| opendevstatus | fungi: finished sending notice | 20:08 |
| clarkb | hourlies are finishing up too | 20:08 |
| clarkb | they are gone (so done) | 20:08 |
| clarkb | I'm ready if you are | 20:08 |
| fungi | okay, starting | 20:09 |
| clarkb | I'm going to be sad if this shutdown takes 5 minutes | 20:09 |
| fungi | yeah | 20:09 |
| fungi | it's still taking far longer that i was expecting after we dropped the db pruning | 20:10 |
| fungi | 78 seconds | 20:10 |
| fungi | it should be on its way back up now | 20:10 |
| clarkb | at least it didn't timeout and it shutdown on its own (so sigint is working) | 20:10 |
| fungi | Powered by Gerrit Code Review (3.10.7-1-g5f52a4e3d5-dirty) | 20:11 |
| fungi | webui is up and responding | 20:11 |
| clarkb | [2025-07-18T20:10:41.373Z] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.10.7-1-g5f52a4e3d5-dirty ready | 20:11 |
| clarkb | anyone have a change/patchset to push up? We'll be able to verify replication and pushes work afterwards | 20:11 |
| fungi | i see file diffd | 20:11 |
| fungi | diff | 20:11 |
| fungi | s | 20:12 |
| clarkb | ya I see diffs as well | 20:12 |
| fungi | i don't have anything to push (should have saved some of the updates i was working on a few minutes ago but didn't think about it) | 20:13 |
| fungi | i'm tailing the gerritbot log though | 20:13 |
| clarkb | I've started digging through my backlog but we just merged all of them :) | 20:13 |
| clarkb | I think there is a new gitea release. I'll push a change for that | 20:14 |
| fungi | looks like gerritbot gets flooded by replicaton events | 20:14 |
| clarkb | ya those are normal on the event stream iirc | 20:15 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to gitea 1.24.3 https://review.opendev.org/c/opendev/system-config/+/955411 | 20:17 |
| clarkb | https://opendev.org/opendev/system-config/commit/e03b19448e80d9833fbe19778eb7f4f5ec22b7ba I think replication is working (as is my push) | 20:18 |
| clarkb | fungi: anything else you think I should check on? | 20:18 |
| fungi | no, everything looks fine to me | 20:18 |
| fungi | 953624,1 is about 10 minutes from merging, i forgot that was likely to result in a gitea redeploy | 20:19 |
| fungi | (updating the well-known file for element/matrix) | 20:19 |
| clarkb | thats probably fine. The main thing is gitea13 has been busy | 20:19 |
| clarkb | lunch actually arrived late so I'm going to eat that quickly now. But I'm around this afternoon and can monitor that gitea deployment and ensure the 1.24.3 change doesn't need more updates | 20:20 |
| clarkb | oh right we need to reindex changes for that 78s shutdown period | 20:20 |
| clarkb | fungi: ^ do you want to trigger that or should I? | 20:20 |
| clarkb | `ssh -p 29418 admin@review.opendev.org gerrit reindex start --force changes` is the incantation I believe | 20:21 |
| clarkb | s/reindex/index/ | 20:22 |
| fungi | i can | 20:23 |
| fungi | it's just "index" not "reindex" but close! | 20:24 |
| fungi | anyway it's running now | 20:24 |
| clarkb | fungi: I just saw a traceback for an openid login attempt | 20:26 |
| clarkb | but it is for elastic recheck | 20:26 |
| clarkb | something that I don't expect to acutally succeed at auth. Do you think you should login as fungi3 and double check login works typically? | 20:27 |
| clarkb | I should really set up the test account for myself already | 20:28 |
| fungi | i think i did something a while back that broke that test account when playing around with merges and the like, but i can certainly try logging in with my normal account | 20:30 |
| fungi | signed in successfully | 20:31 |
| clarkb | ok thanks. I think the issue is on whatever is trying to log in as that account (we haven't used it in a long time) | 20:31 |
| fungi | interestingly, my name shows up in all-caps in the top-right corner now, i don't recall it appearing that way in the past | 20:31 |
| clarkb | does for me as well. If I click on my name for the drop down to get settings etc it shows my name with normal casing | 20:32 |
| clarkb | I can't remember if this is new behavior. I awnt to say it may have done this for a while | 20:32 |
| clarkb | reindexing is about 1/4 of the way done | 20:33 |
| clarkb | I detached from the screen and will let you decide when it can be shutdown | 20:33 |
| fungi | thanks, i think we can tear it down safely. i'll do that now | 20:35 |
| clarkb | once reindexing is done I'm going to look at some local network updates. The one thing I've noticed switching from pfsense to opnsense is that opnsense is much more consistent about regular updates | 20:38 |
| clarkb | a good thing but I also don't want to fall behind | 20:38 |
| opendevreview | Merged opendev/system-config master: Update .well-known/matrix/client for mobile OIDC https://review.opendev.org/c/opendev/system-config/+/953624 | 20:42 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add iptables rule blocks to drop traffic from specific IPs https://review.opendev.org/c/opendev/system-config/+/955414 | 20:54 |
| clarkb | fungi: I think gitea is serving the updated matrix client well know data | 20:56 |
| clarkb | gitea itself was not restarted | 20:56 |
| fungi | ah good, the deploy is smart enough not to restart gitea unless the image contents change | 20:57 |
| clarkb | fungi: in https://opendev.org/opendev/system-config/src/commit/f4c176ebdf26746744f6caae0df540ae48f99429/playbooks/roles/iptables/templates/rules.v4.j2#L28 is the `-p {{ host.protocol }}` a bug? or does the --dport rule ensure that we're checking the correct port? | 20:57 |
| clarkb | I guess I'm confused over why we have both -p and --dport specified there. As written it would do something like -m tcp -p tcp -s some.add.res.s --dport 8443 -j ACCEPT | 20:58 |
| fungi | i think the module and protocol are separate options | 20:59 |
| clarkb | [2025-07-18T20:56:55.488Z] [Reindex changes v86-v86] INFO com.google.gerrit.server.index.OnlineReindexer : Reindex changes to version 86 complete | 20:59 |
| clarkb | fungi: gotcha | 20:59 |
| fungi | clarkb: or are you confusing the -p (protocol) option with --dport (destination port)? | 21:00 |
| clarkb | three changes failed to reindex. Same number as before and they were all old changes that I Think we expect this from unfortunately | 21:00 |
| clarkb | fungi: ya I guess I'm confused since -p also takes a protocl number | 21:00 |
| fungi | fwiw, this is way simpler with netfilter, if we ever redo our rules | 21:00 |
| clarkb | like maybe we have some redundancy in there but I guess saying tcp twice and then finally filtering on port whatever is workable | 21:00 |
| fungi | it's saying iptables should use the tcp module with the tcp protocol | 21:01 |
| clarkb | gotcha | 21:01 |
| fungi | i don't know if the tcp module supports additional protocols besides tcp, but our rules assume it doesn't | 21:02 |
| clarkb | and for the drop rule we don't care about tcp or udp or whatever so don't need any of that just the source ip address | 21:02 |
| fungi | right | 21:03 |
| clarkb | thanks for double checking | 21:04 |
| fungi | could argue for rejecting connections with icmp admin-prohibited errors instead or something, but in cases where you're blocking spoofed attackers that's bad and enables reflection attacks or inadvertent backscatter | 21:04 |
| clarkb | I think the stuff we had in flight is looking happy at the moment so I'm going to proceed with local network gear updates | 21:04 |
| clarkb | fungi: yup that is why I didn't use reject | 21:04 |
| fungi | [insert thumbs-up emoji here] | 21:05 |
| clarkb | then next week I guess I'm diving into gerrit 3.11 upgrade prep properly | 21:08 |
| clarkb | that was quick. I like that updates come more frequently so that the update process itself is better exercised | 21:20 |
| clarkb | https://687a0a9b9a79cc21edde-9d697031d66865af8e6eba8e2a3ea98e.ssl.cf1.rackcdn.com/openstack/89bf4bb7cd59426db4e8c920c923fdbf/bridge99.opendev.org/screenshots/ gitea 1.24.3 screenshots look good | 21:25 |
| clarkb | I'm not in a rush to do that upgrade and we can probably wait for monday | 21:25 |
| clarkb | Mostly I'm worried that gitea13 might have a sad | 21:25 |
| fungi | yeah, seems like a good idea this close to everyone's weekend | 21:26 |
| clarkb | we got a fair bit done. Gerrit should be all caught up on the backlog of issues surrounding it which is nice. We also confirmed that sigint works but it is still somewhat slow. I did start running a strace but I couldn't confirm the file number mapping via lsof before it stopped | 21:28 |
| clarkb | I suspect its still trying to deal with those large db files on shutdown | 21:28 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!