| ianw | i've got nb04 cleanup running in a screen ... it's taken all day. i stopped the builder, but will let it restart with any next ansible run as the running rm will only delete the old files, it should have enough space to try new builds | 06:33 |
|---|---|---|
| fungi | thanks for doing that ianw! | 13:47 |
| *** ykarel_ is now known as ykarel | 15:02 | |
| clarkb | ya usually if you start an rm that gets enough headroom going before the next hourly jobs begin | 15:45 |
| frickler | seems like the rm finished leaving ~50% free on /opt and one successful build finished 1h ago, too | 15:49 |
| clarkb | my distro has all the updates today.... the post holiday rush. Once I've rebooted for that I'm going to dive into paste02 booting and enrollment and hoping for gitea upgrade reviews if they haven't been posted yet. We can probably do that upgrade soonish | 15:49 |
| clarkb | then I need to look into meeting prep and our meetup planning | 15:49 |
| clarkb | for record keeping purposes `image = conn.create_image('Ubuntu Noble OpenDev 20250110', filename='/home/tonyb/noble-server-cloudimg-amd64.vhd', wait=True)` is what I ended up doing for rax image uploads | 15:50 |
| clarkb | then manual boots look like: `conn.create_server('clarkb-test-sdk', image='Ubuntu 22.04 LTS (Jammy Jellyfish) (Cloud)', flavor='4 GB Performance', key_name='infra-root-keys-2024-04-08', auto_ip=True)` | 15:51 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/938826 is the gitea upgrade change. There is a held node here: https://104.239.145.201:3081/opendev/system-config | 15:55 |
| clarkb | they did change the UI a bit. Made it narrower with a panel on the left for search and language breakdown rather than having that in the header? | 16:01 |
| clarkb | but seems to functiona for me | 16:01 |
| clarkb | if you try to do a fuzzy search you get "Fuzzy search is disabled due to performance reasons" in a banner. This is the fix for the memory leak I think | 16:02 |
| fungi | that's unfortunate, i guess we'll need to keep hound around for even longer | 16:03 |
| clarkb | I don't think hound does fuzzy search fwiw. But hound does do regexes | 16:04 |
| fungi | yes, fuzzy searching was the closest thing gitea had to regex searching | 16:05 |
| fungi | even if it wasn't as flexible | 16:05 |
| clarkb | 1242/2078 ... this may take a while | 16:06 |
| clarkb | this is what happens when you have java and ghc installed | 16:12 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Add paste02 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/939123 | 16:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add paste02 to inventory as a new paste server https://review.opendev.org/c/opendev/system-config/+/939124 | 16:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Backup paste02 to the backup servers https://review.opendev.org/c/opendev/system-config/+/939126 | 17:00 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Retire paste01 backups on the smaller backup server https://review.opendev.org/c/opendev/system-config/+/939128 | 17:02 |
| clarkb | infra-root ^ I think we should wait for CI results to come back on 939124 before we land the dns update just to be sure noble and container and lodgeit are all still happy since december's efforts. | 17:03 |
| clarkb | and you may want to jump on the server just to check that noble image looks good as well as launch node against it | 17:04 |
| frickler | clarkb: paste02 lgtm with a quick check. 939123 doesn't update the cname yet, so should be safe to merge? | 17:32 |
| clarkb | frickler: yes it should be safe to merge. We need to manually stop paste01 services, dump the db, and restore it on 02 then land the CNAME update which si why it isn't done early | 17:33 |
| clarkb | I'm only dropping the cname ttl to 300 to make that process simpler when we get there | 17:33 |
| clarkb | as a reminder this image was the one found in tonyb's homedir. I don't expect any problems doing that but was hoping tonyb could confirm it was good to use | 17:34 |
| clarkb | as fungi mentioend last week we don't expect people to put inappropriate images in their homedir on bridge. And also the timestamps align with the uploads to other clouds so I think it is fine | 17:35 |
| clarkb | frickler: do you know yet if you might be able to be around for ~January 21-23 timeframe? If not the whoel time thats fine but it would be useful to know if I should try and schedule some early (for me) time during the meetup block | 18:20 |
| clarkb | a reminder that https://etherpad.opendev.org/p/opendev-january-2025-meetup is starting to come together with topics and organizational info | 18:20 |
| frickler | clarkb: not completely sure yet, but I'd say you don't need to schedule anything specifically for me | 18:27 |
| clarkb | ok thanks | 18:29 |
| clarkb | fungi: any thoughts on paste02 and/or gitea 1.23.1? | 18:36 |
| fungi | hoping to test them out as soon as the package updates on my workstation complete, but they're taking an age because of kernel module build failures. probably just a few more minutes | 18:38 |
| clarkb | ack thanks. Good luck with the kernel modules | 18:38 |
| clarkb | thankfully my updates this morning seem to have gone well. I haven't noticed any problems anyway | 18:39 |
| fungi | unfortunately it's already upgraded firefox which is therefore refusing to open new tabs until i restart it, which i'd rather do when i reboot for the kernel upgrade that's in progress | 18:39 |
| clarkb | huh. I use mozilla's FF "stream" and it updates itself in place and it will happily continue to make new tabs after an update and before restarting | 18:40 |
| fungi | yeah, if i try to open something in a new tab it tells me "Gah. Your tab just crashed." and if i try to "restore this tab" it says "Restart to Keep Using Firefox" | 18:41 |
| clarkb | ansible is aggressively pruning python support. Curiously the powershell version they support goes back to windows 7.... | 19:30 |
| fungi | i suspect part of it is that the python module ecosystem has adopted a pattern of dropping support for any python versions as soon as the corresponding upstream cpython reaches eol, and a majority of those also don't backport security fixes, so this drives ansible to perpetually update to the newest versions of their dependencies and that cuts them out of being able to work with | 19:33 |
| fungi | older python | 19:33 |
| clarkb | could be. Its been a while since I looked at hte remote python ansible generates, but at the time it was very simple and largely relied on stdlib | 19:34 |
| fungi | clarkb: is 104.239.145.201 the held gitea node? if so, i don't see the new sidebar you were mentioning? | 19:40 |
| clarkb | fungi: its the right side not left (I think I typed left for whatever reason) | 19:40 |
| clarkb | but the search bar is there and the language breakdown instead of being in the same vertical column as the files | 19:40 |
| fungi | and it's in the code search results view? | 19:40 |
| clarkb | its here: https://104.239.145.201:3081/opendev/system-config | 19:41 |
| clarkb | compared to https://opendev.org/opendev/system-config | 19:41 |
| fungi | ah, that space. got it | 19:41 |
| fungi | okay, looks like the same information as in the previous version, just some of it moved over to the right | 19:43 |
| clarkb | ya I think the idea there is to mimic github... | 19:43 |
| fungi | unfortunately that's a pretty massive waste of screen space, but whatever | 19:43 |
| clarkb | it works but ya its wastes space | 19:43 |
| fungi | i already use a portrait orientation screen, so this just leaves a huge stripe of unused space down the right side | 19:44 |
| fungi | oh well | 19:44 |
| clarkb | I'm going to work on lunch nowish. I'm also hoping to go out on a bike ride later this afternoon. But otherwise I'm around and can pay attention to paste and/or gitea | 19:45 |
| fungi | anyway, new gitea version seems to be working for me | 19:45 |
| opendevreview | Merged opendev/zone-opendev.org master: Add paste02 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/939123 | 19:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit images to 3.10.4 and 3.11.1 https://review.opendev.org/c/opendev/system-config/+/939167 | 20:49 |
| clarkb | these releases were just published | 20:49 |
| clarkb | as noted in the commit message bundling the new images up with restarts for h2 compaction config updates may be a good idea | 20:50 |
| clarkb | fungi: looks like wiki may be sad again? | 20:54 |
| clarkb | I wish there was a more effective "please be kind" sign/tool/roadblock for the ai crawlers | 20:54 |
| fungi | checking | 20:55 |
| fungi | when i try to authenticate via ubuntuone, mw responds with 'Verification failed: Verification of the OpenID URL failed. Error message: "Bad signature"' | 20:57 |
| clarkb | ya I just got that too | 20:57 |
| fungi | ssh is being very slow to connect too | 20:57 |
| clarkb | loading was really slow before I gto that point though | 20:57 |
| fungi | load average is very low though | 20:58 |
| fungi | oh wow, swap is exhausted | 20:58 |
| fungi | memcached and java are eating a ton of memory, but collectively so are all the apache processes | 20:59 |
| clarkb | java is elasticsearch iirc | 20:59 |
| fungi | i'm just going to reboot the server | 20:59 |
| clarkb | ack | 20:59 |
| fungi | who knows what state some of those processes are in due to memory allocation failures | 20:59 |
| clarkb | the paste02 change hit docker hub rate limits | 21:02 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Mirror mariadb:10.11 to quay https://review.opendev.org/c/opendev/system-config/+/939169 | 21:04 |
| fungi | grr | 21:05 |
| clarkb | and ^ is a response to that. We should be able to swap our deployments over to the mirrored image if we do that | 21:05 |
| clarkb | fungi: I was able to login just now | 21:12 |
| clarkb | managed to do a first pass update on a meeting agenda for tomorrow | 21:18 |
| fungi | excellent, thanks for confirming! | 21:20 |
| opendevreview | Merged opendev/system-config master: Mirror mariadb:10.11 to quay https://review.opendev.org/c/opendev/system-config/+/939169 | 21:22 |
| opendevreview | Merged openstack/project-config master: Add refs/tags/* acls for openstack-helm projects https://review.opendev.org/c/openstack/project-config/+/938698 | 21:38 |
| corvus | clarkb fungi : can you review https://review.opendev.org/938989 to add a container mirror job pls? | 21:41 |
| clarkb | done | 21:49 |
| clarkb | corvus: I realized another reason to run the mirroring job prior to periodic is it that it would ensure the opendev periodic jobs pull a more up to date image if we switch opendev over to the quay mirror for things like mariadb | 21:49 |
| clarkb | I don't think that is urgent. Updates for things like mariadb are few and far between and we can manually enqueue the jobs if we need thing faster (or fetch directly from docker hub etc0 | 21:50 |
| opendevreview | Merged opendev/system-config master: Run the jaegertracing mirror job https://review.opendev.org/c/opendev/system-config/+/938989 | 21:55 |
| corvus | good point; delayed by 1 day is probably okay for us 99% of the time, but a smidge earlier would be better. | 21:57 |
| clarkb | does anyone else have updates for the meeting agenda? | 23:46 |
| clarkb | I'll get that sent out soonish if not | 23:46 |
| fungi | argh, 939124 is failing in the gate again for the same reason, i think | 23:46 |
| clarkb | fungi: after tonights periodic job runs we should be able toswitch that image over to pulling from quay | 23:46 |
| clarkb | and just land the change tomorrow | 23:46 |
| fungi | fatal: unable to access 'https://github.com/Matty9191/ssl-cert-check/': The requested URL returned error: 500 | 23:47 |
| fungi | huh, not tyhe same! | 23:47 |
| clarkb | hrm what fetches taht? | 23:47 |
| fungi | TASK [install-certcheck : Pull latest ssl-cert-check from git] | 23:47 |
| clarkb | is that what we use to check our ssl certs? | 23:47 |
| corvus | probably github under ai training spider dos | 23:47 |
| clarkb | ah yup that is what we use it for | 23:47 |
| fungi | i wonder why we don't install a distro package of that | 23:48 |
| corvus | it's in noble | 23:48 |
| fungi | and jammy, and focal... | 23:49 |
| corvus | Package: ssl-cert-check | 23:49 |
| corvus | Homepage: https://prefetch.net/articles/checkcertificate.html | 23:49 |
| fungi | same version in all (4.14) | 23:49 |
| clarkb | re switching images over to quay we do want to be careful that we don't go backwards in time. Though I suspect that is generally fine | 23:49 |
| corvus | that points to the same github repo | 23:49 |
| clarkb | so basically we check docker hub and quay have the same image before approving the transition in git | 23:49 |
| corvus | or verify we haven't restarted in prod since last successful mirror? | 23:50 |
| fungi | for ssl-cert-check, we're basically grabbing a 4-year-stale git repo over and over. version is the same as what's packaged in ubuntu/debian | 23:52 |
| fungi | i'll see if i can clean that up | 23:52 |
| fungi | we have an entire role dedicated to it | 23:52 |
| clarkb | thanks | 23:53 |
| clarkb | its also probably small enoguh we could vendor it too | 23:53 |
| fungi | well, we have a bunch of extra stuff around it (cronjob, dedicated user, config...) | 23:56 |
| fungi | i think i can just improve the installation method in the role and call it done | 23:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!