Wednesday, 2024-04-17

clarkbhttps://iscinumpy.dev/post/bound-version-constraints/ 2015 me is surprised it took so long for the rest of the python world to discover this si a problem00:05
fungia frog cannot explain dry land to a tadpole00:06
fungino matter how many times you try to convince someone they're going to run into problems you've experienced, they'll fail to comprehend or refuse to believe until it happens to them00:08
fungiand also conveniently forget you encountered and came up with solutions for the problem00:09
clarkbI'm happy someone took the time to write all that down though because ya its basically all the wisdom. Don't cap deps, security fixes aren't backported, dep solver gets sad, etc etc00:12
clarkb"Artificially limiting versions will always reduce the chances of it working in the future." This is well and simply said and something I think I've failed to articulate well00:13
clarkb"If you don’t cap your dependencies, you are immediately notified when a dependency releases a new version, probably by your CI, the first time you build with that new version." I love this post00:14
fungiby extension, your ci jobs breaking on a new version of something isn't a problem, it's a signal00:15
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Overwrite EFI grub.cfg when it exists  https://review.opendev.org/c/openstack/diskimage-builder/+/91602101:19
opendevreviewMerged openstack/diskimage-builder master: fix(rocky): don't uninstall linux-firmware  https://review.opendev.org/c/openstack/diskimage-builder/+/91281601:21
*** dmellado030 is now known as dmellado0302:00
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Overwrite EFI grub.cfg when it exists  https://review.opendev.org/c/openstack/diskimage-builder/+/91602102:21
opendevreviewAlberto Gonzalez proposed openstack/project-config master: Add new repository powertrain-build  https://review.opendev.org/c/openstack/project-config/+/91603807:06
opendevreviewAlberto Gonzalez proposed openstack/project-config master: Add new repository powertrain-build  https://review.opendev.org/c/openstack/project-config/+/91603807:29
opendevreviewAlberto Gonzalez proposed openstack/project-config master: Add new repository powertrain-build  https://review.opendev.org/c/openstack/project-config/+/91603807:31
opendevreviewAlberto Gonzalez proposed openstack/project-config master: Add new repository powertrain-build  https://review.opendev.org/c/openstack/project-config/+/91603807:43
opendevreviewAlberto Gonzalez proposed openstack/project-config master: Add new repository powertrain-build  https://review.opendev.org/c/openstack/project-config/+/91603808:05
opendevreviewJan Marchel proposed openstack/project-config master: Add repository for NebulOuS scripts  https://review.opendev.org/c/openstack/project-config/+/91604808:29
opendevreviewRadosław Piliszek proposed openstack/project-config master: Add repository for NebulOuS scripts  https://review.opendev.org/c/openstack/project-config/+/91604808:43
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Support Ubuntu 24.04 in nodepool elements  https://review.opendev.org/c/openstack/project-config/+/91605009:19
opendevreviewDr. Jens Harbott proposed openstack/diskimage-builder master: Add Ubuntu 24.04 (noble) build to testing  https://review.opendev.org/c/openstack/diskimage-builder/+/91591509:35
opendevreviewMerged openstack/project-config master: Fix the ACL associated with charm-keystone-ldap-k8s  https://review.opendev.org/c/openstack/project-config/+/90366711:59
opendevreviewMerged openstack/project-config master: Retire PowerVMStacker SIG: Remove Project from Infrastructure Systems  https://review.opendev.org/c/openstack/project-config/+/90958412:02
opendevreviewDr. Jens Harbott proposed openstack/diskimage-builder master: Add Ubuntu 24.04 (noble) build to testing  https://review.opendev.org/c/openstack/diskimage-builder/+/91591512:14
opendevreviewMerged openstack/project-config master: Ironic UM group access to bifrost UM branches  https://review.opendev.org/c/openstack/project-config/+/91028712:17
opendevreviewMerged openstack/project-config master: Add warning to nodepool configs about changing cloud name  https://review.opendev.org/c/openstack/project-config/+/91195912:29
opendevreviewMerged openstack/project-config master: Add openstack-helm-plugin git repository  https://review.opendev.org/c/openstack/project-config/+/91600712:29
opendevreviewMerged openstack/project-config master: Add repository for NebulOuS scripts  https://review.opendev.org/c/openstack/project-config/+/91604812:33
fungiugh, manage-projects is failing13:38
fungiTASK [gitea-git-repos : Create Gitea Repos and Org]13:39
fungiNo usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/root']13:39
fungifatal: [gitea10.opendev.org]: FAILED!13:39
fungi/dev/vda1       155G  155G     0 100% /13:40
fungi115G    /var/gitea/data/gitea/repo-archive13:41
fungihttps://meetings.opendev.org/irclogs/%23opendev/%23opendev.2024-01-05.log.html#t2024-01-05T14:45:3313:43
fungithat's what we did to solve a similar situation on gitea09 in january13:43
fungii'll give it a try13:44
fungii'm deleting ~clarkb/giteaXY_transplant_db.sql from 2023-03-03 on gitea10 like i did on gitea09, and then rebooting it13:50
fungithen i'll proceed with the archive cleanup button clicking13:50
Clark[m]It looks like we did end up landing the updated robots.txt which was part of the upstream suggestion from matrix/discord. I guess insufficient for us though13:53
Clark[m]Or at least the file currently served looks new (even has a January 2024 timestamp in it)13:54
fungiStarted Task: Delete all repositories' archives (ZIP, TAR.GZ, etc..)13:55
fungi/dev/vda1       155G   41G  115G  27% /13:57
fungimuch better13:57
fungi#status log Rebooted and deleted repository archives on gitea10 after they filled up its rootfs13:58
opendevstatusfungi: finished logging13:58
funginow we either need to re-enqueue a deploy to get manage-projects run again, or merge something else that will trigger it13:59
Clark[m]Don't forget to have Gerrit rereplicate to 10 and we should probably spot check the others. Thank you for fixing this13:59
Clark[m]Does manage projects run hourly? That may be another option if so. Simply wait for the next hourly run. I want to say it doesn't though14:00
opendevreviewDr. Jens Harbott proposed openstack/diskimage-builder master: Add Ubuntu 24.04 (noble) build to testing  https://review.opendev.org/c/openstack/diskimage-builder/+/91591514:00
fungidaily i think14:00
fungiClark[m]: i think our only options are replicate all repos or specific repos, but we don't have a way to limit which remote servers get replicated to14:02
fungii can kick off a full replication14:02
Clark[m]We do14:02
fungiany idea what that syntax is?14:02
Clark[m]You can specify a URL filter when you ask for replication14:02
fungioh, so that's what "--url PATTERN  : pattern to match URL on" means14:02
fungiany idea what that pattern would look like, do we have any examples?14:03
Clark[m]Ya I think if you do --url gitea10 that will work14:03
fungior can i nab an example from the task queue i guess?14:03
Clark[m]Any substring of the name in our replication config file14:03
fungiokay, so no wildcarding required14:03
fungi2410 tasks14:04
Clark[m]Yes the docs imply it is substring matching only not a regex or glob14:04
fungii see a bunch of retries in the queue for gitea1114:05
fungi/dev/vda1       155G  155G     0 100% /14:05
fungiwax on, wax off14:05
fungidoing the same dance with that server now14:06
Clark[m]Didn't we also have a change to run the cron job to clear the archive daily?14:07
fungii saw discussion of cron tasks in the irc log14:08
Clark[m]https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gitea/templates/app.ini.j2#L130 weekly14:08
fungigitea11 cleaned up and re-replication started for it as well14:09
Clark[m]And ya the disk usage is within the week. Maybe we want to run that daily14:09
fungi#status log Rebooted and deleted repository archives on gitea11 after they filled up its rootfs14:09
opendevstatusfungi: finished logging14:10
Clark[m]I think we just go ahead with daily deletion ... Unfortunately not very efficient but should keep things down to a more manageable state14:10
fungiworking on that now14:11
fungithe cron timespec this uses doesn't seem aligned with vixie cron14:13
fungioh, it's using optional seconds as the first field14:14
Clark[m]It's a golang specific cron lib iirc14:15
opendevreviewJeremy Stanley proposed opendev/system-config master: Switch Gitea archive cleanup from weekly to daily  https://review.opendev.org/c/opendev/system-config/+/91611614:17
clarkb9, 12, 13, and 14 appear to not be suffering the same issue14:22
fungitheir disks didn't look filled up when i checked14:22
clarkbya sorry the way I worded that was not super claer. I'm saying they are fine14:23
fungioh, right yes14:23
fungiafter i saw 11 was also full i checked the others, should have said as much14:23
fungimy guess is that the crawler(s) got persisted to one or two backends14:25
clarkbthe robots.txt rules seem to match for the archive download paths too14:26
clarkbso ya this must be bad crawlers? I feel like humans wouldn't be able to click buttons fast enough14:27
fungiright, i expect it's crawlers that gon't care about being polite or respecting our wishes14:27
fungis/gon't/don't/14:27
fungii suppose we could try to find common user agent strings or something and see if they're unique enough to add to our filter14:28
clarkbfungi: ++ and you can start by grepping 'archive' out of the access logs to see what if anything is fetching those14:28
fungigoing to try to get a quick walk in now that things have quieted down, and then we can work on landing upgrades i guess14:29
clarkbsounds good14:29
clarkbI'm still in hte process of waking up myself14:29
Clark[m]fungi: I suspect the cron update won't auto deploy. We can land it first then the 1.21.11 upgrade to have Ansible update the config in the running services for us14:36
fungimakes sense, agreed14:38
*** blarnath is now known as d34dh0r5315:04
fricklerclarkb: noble dib passing now https://zuul.opendev.org/t/openstack/build/53135a39694e4c2c8de0505a9126359815:15
clarkbfrickler: in the symlink change should we provide fully rooted paths for both target and link name? It isn't clear to me what gutsy is on the fs15:26
fricklerclarkb: this is copying how the other symlinks look like. a relative path is relative to the target, this is where ln is different from cp15:27
clarkbTIL15:29
clarkbI +2'd it anyway. I also left a nit on the project-config change that I +2 as well.15:29
clarkbThe only other thing I notice si that we should probably plan to make a glean release after the glean change lands15:31
clarkbthank you for putting this together15:31
clarkbfrickler: any interest in reviewing https://review.opendev.org/c/opendev/system-config/+/916116 to try and counter the gitea archive ballooning?16:12
fungialso i'm ready to start approving upgrades any time. should we start with the mailman mariadb container?16:17
clarkbfungi: wfm16:19
fungifire in the hole!16:20
opendevreviewClark Boylan proposed opendev/system-config master: Add more User Agent filters  https://review.opendev.org/c/opendev/system-config/+/91612516:35
opendevreviewMerged opendev/system-config master: Upgrade Mailman's MariaDB to 10.11  https://review.opendev.org/c/opendev/system-config/+/91518317:05
fungideploy for that's going to be waiting a bit while the hourly builds complete17:05
clarkbfungi: the lists deploy job is running now17:20
fungiyep17:20
fungilooks like the mariadb container restarted a minute agio17:21
fungior it was upgrading17:21
fungidatabase_1      | 2024-04-17 17:22:26 0 [Note] Starting MariaDB 10.11.7-MariaDB-1:10.11.7+maria~ubu2204 source revision 87e13722a95af5d9378d990caf48cb687443934717:23
fungi as process 117:23
fungimailman services are up again, i can browse list details (postorius) and archives (hyperkitty)17:24
clarkbfungi: tehre are a few connection errors in the mariadb log that surprise me17:25
clarkbbut maybe that is related to how we test if things are up in ansible?17:25
fungii think we've seen those in the past too17:25
clarkbI think fi services are working then it isn'y a big deal17:25
fungiwe only see them at startup, from what i recall17:25
clarkband I too can confirm services seem to be working for me. Maybe we want to send a test email through?17:25
clarkbya so perhaps related to how we do the db based startup checking17:26
clarkbwe dont' appear to log these services to /var/log/containers though so we don't have the log from when the actual mariadb upgrade occured?17:26
clarkbI think because we must stop and start mariadb a couple times?17:26
fungithat's what it looked like when i was grepping ps output17:26
clarkbI do see the small backup in the database dir though which confirms the upgrade ran17:27
fungiand yeah, we could add syslog logging to the compose file, what we use now is almost identical to upstream mailman-docker's compose file17:27
clarkback17:28
clarkbI think the only other thing I can think of to test is sending an email through17:28
clarkbotherwise what I've checked lgtm17:28
fungiyeah, i figured i'd wait a bit to see if one came through naturally17:28
fungiwhat upgrade did you want to do next? merge the outstanding gitea patches and then upgrade it, or do etherpad?17:29
clarkbI'm thinking that maybe we should do gitea in order to mitigate disk consumption concerns17:30
clarkbthat seems more ugent than etherpad17:30
fungimakes sense, yep17:30
fungiso we want to go ahead with the archive cleanup cron and ua filter additions, then the gitea version update?17:30
clarkbI think the ua filter addition can go in any order, but the cron update should go before the gitea version update17:31
clarkbthat way we don't need to manually restart anything17:31
fungiclarkb: should i wait to approve the gitea upgrade change until we're sure the cron config has merged?17:42
fungior just risk it?17:42
clarkbit is probably ok to risk it?17:46
fungiyeah, was trying to decide if the potential time savings outweighed the potential time loss if we hit an unexpected test failure17:49
fungianyway, approved it now too17:49
clarkbwe should have a window of time where we can -W the upgrade change if the parent fails17:49
fungigood point17:49
clarkbthat way we can still rely on the upgrade to pick up the new config if we like17:49
*** dmellado033 is now known as dmellado0317:54
opendevreviewMerged opendev/system-config master: Cleanup lingering Mailman 2 playbook  https://review.opendev.org/c/opendev/system-config/+/91518417:54
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Support Ubuntu 24.04 in nodepool elements  https://review.opendev.org/c/openstack/project-config/+/91605017:58
clarkbfungi: it occurred to me that manage-projects may not have run yet? I don't think that is a problem since it is a job distinct from the infra-prod-service-gitea job?18:23
clarkbtwo new emails arrived in openstack-discuss too and a third is in the archive that hasn't made it to my inbox so that lgtm18:26
clarkbits also my parents last day in town and they just asked us to pop over for lunch. I'll be about 5-10 minutes away from my computer should anything go wrong and keep an eye on this channe via matrix18:28
clarkbya its taking about 7-8 minutes for new openstack-discuss emails to arrive in my inbox. But email is async so not a big deal18:32
clarkbI suspect this is not new either18:32
fungiyeah, fairly typical18:32
fungithere are a lot of subscribers18:32
fungii'll be around for the gitea upgrade, go enjoy your lunch18:33
fungiand no, we haven't merged or retriggered any new manage-projects runs afaik, checking the builds list for confirmation18:33
fungihttps://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-manage-projects&project=openstack/project-config18:35
fungilast build started at 12:42 utc, no successful runs yet18:35
clarkback and ya I don't think that is a problem for the infra-prod-service-gitea job18:37
clarkbbut getting that to run after wards is probably a good idea18:37
fungiit's unfortunate that i approved a slew of changes that ran the deploy job before noticing it was consistently failing, so there's a good chunk of work queued up for it the next time it runs18:39
opendevreviewMerged opendev/system-config master: Switch Gitea archive cleanup from weekly to daily  https://review.opendev.org/c/opendev/system-config/+/91611618:55
Clark[m]The deploy job is running for that now. Should be able to check the updated config on the servers well before the next change lands18:58
fungiyep18:59
fungifungi@gitea09:~$ ls -l /var/gitea/conf/app.ini 19:01
fungi-rw-r--r-- 1 root root 3833 Apr 17 18:56 /var/gitea/conf/app.ini19:01
fungifungi@gitea09:~$ grep SCHEDULE /var/gitea/conf/app.ini 19:01
fungiSCHEDULE = 0 0 3 * * *19:01
fungii'll make sure the other 5 match19:01
fungiall 6 lgtm19:02
Clark[m]Excellent 19:02
opendevreviewMerged opendev/system-config master: Add more User Agent filters  https://review.opendev.org/c/opendev/system-config/+/91612519:15
opendevreviewMerged openstack/diskimage-builder master: Write fedora download redirect info to stderr  https://review.opendev.org/c/openstack/diskimage-builder/+/90820719:17
opendevreviewMerged openstack/diskimage-builder master: Overwrite EFI grub.cfg when it exists  https://review.opendev.org/c/openstack/diskimage-builder/+/91602119:18
opendevreviewMerged opendev/system-config master: Upgrade gitea to v1.21.11  https://review.opendev.org/c/opendev/system-config/+/91600419:23
Clark[m]The job that should upgrade things has just started19:28
Clark[m]Looks like gitea09 is currently down for the update19:30
Clark[m]And its up. System-config renders for me19:31
fungiyep, looking right to me19:38
fungiPowered by Gitea Version: v1.21.1119:38
fungiand my test clones have completed successfully19:39
fungii guess etherpad is next to look at? do we want to do that today or tomorrow?19:39
fungii'm flexible19:39
Clark[m]I don't mind proceeding with etherpad I'm just not around for a bit longer19:40
Clark[m]Tomorrow would be fine too if you prefer19:41
funginah, the sooner the better in my opinion19:41
Clark[m]The gitea job was successful and all 6 back ends lgtm19:42
fungii'm just mowing in short bursts19:42
fungino real scheduled plans for a couple more hours and then i need to cook dinner19:42
fungishould i approve it now so we get the gate jobs underway, or wait a bit for you to free up?19:43
Clark[m]I think it's fine if you go for it now. Worst case I'm not far from home19:43
fungioh, or did we want to wait for an actual etherpad release?19:44
fungioh, it's the etherpad mariadb upgrade we wanted to do first19:44
Clark[m]The database update change doesn't depend on the etherpad release. But we shouldn't land the etherpad upgrade until etherpad has a release just in case they change stuff in non compatible ways19:44
fungihttps://review.opendev.org/91100019:44
fungiyeah, that just dawned on me19:45
Clark[m]Ya the DB upgrade is the one fine to do now if we are happy with the change19:45
fungiafter several of these i'm not too worried. went ahead and approved it19:45
Clark[m]The etherpad proper upgrade needs to wait19:45
fungiagreed19:45
fungiso many upgrades19:46
Clark[m]Maybe double check that flag is correct against the other mariadb updates but otherwise it should be very similar to the other updates. Services go down. DB comes up and upgrades then etherpad connects to it19:46
fungilooks the same as the others19:51
fungiprobably at least another half hour until it merges20:14
fungibut also the ducks don't want me to keep mowing20:15
fungithere's this mated pair of mallards that comes to hang out in the yard a few hours a day and scavenge the area below the deck where birds drop seed from the feeders upstairs20:17
clarkbsmart20:18
fungithey tolerate me sitting here a few feet away, but are not fans of the lawn mower20:18
fungithough now they're getting into a territorial dispute with the canada geese hanging out a few lots up the water from us20:19
fungiso maybe i can finish up after all20:20
fungialso i don't think either the ducks or the geese are fans of the ospreys which are nesting out near the end of the dock20:21
opendevreviewMerged opendev/system-config master: Upgrade Etherpad's MariaDB to 10.11  https://review.opendev.org/c/opendev/system-config/+/91100020:23
JayFI'm trying to paste a large amount of preexisting text into an etherpad, and it's just spamming "connected" over and over, and I can confirm via another browser the text is not getting uploaded20:25
JayFI assume this is some kinda abuse prevention trick? but I'd like to know how you all get around it when you need to20:26
JayFI got it to work via batching, but if there's a neat trick (or  a concrete line/character count), I'd still be interested20:29
clarkbJayF: depending on the timing you may have run into the db ugprade behind etherpad that just happened a few minutes ago20:29
clarkbbut yes I do believe there is a per connection rate limit20:29
JayFI think the answer is "I hit both"20:29
JayFwhich made the behavior extra weird :D 20:29
fungiJayF: have you tried the import feature?20:29
JayF*blink*20:29
JayF:-O20:29
fungicheck the menu icons at the top-right20:30
fungiit has import/export features20:30
JayFthat is what I was looking for :D 20:30
JayFthank you20:30
fungino guarantees it will work better, but probably more efficient than pasting ginormous amounts of data20:30
clarkbfungi: fwiw the db upgrade seems to have gone well. We don't appear to set etherpad having a dependency on the db so we didn't restart it?20:31
JayFGiven how much I had to break it up, I think it's technically 3 ginormous datas :P 20:31
clarkbbut it seems to have reconnected after the db was happy again so nto a huge deal20:31
fungiyeah, it's working for me now20:32
clarkbhttps://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/etherpad/templates/settings.json.j2#L574-L589 and https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/etherpad/templates/settings.json.j2#L599-L613 are the rate limit configs20:33
clarkbJayF: ^ basically its 10 edits per second per IP20:33
clarkbI'm guessing that your copy paste buffer gets flushed multiple times pushing it over that limit maybe?20:33
JayFthat's what I'd suspect, yeah20:34
JayFit also acted different with middle-click vs right-click paste20:34
fungithat's good to know, but probably browser-specific as to how it treats the primary and selection buffers20:39
fungiclarkb: so what was the plan for the gerrit upgrade? yolo this week or hold until after the rename maintenance is knocked out?20:40
clarkbfungi: I'm leaning towards after the rename20:40
fungii'm good with either20:40
clarkbbut I wonder stop you if you want to get it done sooner20:40
clarkb*I won't stop you20:40
fungiit's nice to be nearing the end of this tunnel at least20:41
clarkbya huge progress today, thanks for helping drive it along20:41
fungiyeah, i could plan to do the gerrit upgrade tomorrow so we have the weekend to shake out any unexpected behavior changes20:41
clarkbI don't think there is a change for gerrit yet and it would have to be manually restarted too20:47
clarkbsince the ansible for gerrit doesnt' auto restart things iirc20:47
clarkbfungi: https://review.opendev.org/c/opendev/system-config/+/911622 is the rename prep change we should consider landing soonish too20:49
clarkbmaybe we should recheck to get up to date test job logs and we can check that things get copied to the correct location?20:50
clarkbthat said it should have fairly low impact even if it doesn't work, we'll just end up with files copied somewhere we don't expect and maybe extra errors in the error log20:50
fungiyeah, i'll get that merged after dinner21:12
fungier, rechecked i mean21:12
fungiwe don't merge it until the end of the maintenance if memory serves21:13
fungioh, wait, *that* prep change21:13
fungiyeah, approved now21:13
fungiwe still need the opendev/project-config change to reflect the repo redirects, but that comes later21:14
Clark[m]Yup, I stepped away again but it is on my to-do list to write that change. 21:18
opendevreviewMerged opendev/system-config master: Move gerrit replication waiting queue aside during project renames  https://review.opendev.org/c/opendev/system-config/+/91162222:48

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!