| clarkb | @ERROR: Unknown module 'centos-stream-full' rsync error: error starting client-server protocol (code 5) at main.c(1863) [Receiver=3.2.7] | 00:09 |
|---|---|---|
| clarkb | I think this is due to the remote saying no and not really a problem on our end. But I need to reload ssh keys in order to check the logs on the old server | 00:09 |
| clarkb | https://static.opendev.org/mirror/logs/rsync-mirrors/centos-stream.log so its working there | 00:10 |
| clarkb | (note that log file is likely to be overwritten with the one I pasted the error from shortly so that link may not always be useful) | 00:10 |
| clarkb | but I've just remembered that we got our mirror into some allow list to mirror from some more upstream location rather than the normal mirrors | 00:10 |
| clarkb | I bet this is the issue. We need to get the new server's ip on that list | 00:11 |
| clarkb | tonyb dpawlik maybe you know who to talk to about that? otherwise I Think we'll hvae to revert to the locations we were using previously | 00:11 |
| clarkb | amoralej: looking at git losg you were the one who proposed the change so maybe you would know about that | 00:12 |
| clarkb | the new server is mirror-update03.opendev.org replacing mirror-update02.opendev.org. Its A and AAAA records are in DNS if you need them instead of a name | 00:12 |
| clarkb | the yum puppetlabs rsync looks to have run successfully so ya no issues with rsync on the new platform. This is just a "you're not allowed to sync from centos mirrors directly form that ip problem" I think | 00:17 |
| clarkb | tomorrow we can switch to the normal mirrors to sync from again if there is no movement on getting the new ip in the allowed list | 00:17 |
| clarkb | I realized that the script that releases docs/tarballs/etc does have a lockfile at /var/run/release-volumes.lock so I went ahead and flock -n /var/run/release-volumes.lock bash on mirror-update02 for completeness even though it shouldn't be required at this point | 00:21 |
| clarkb | before I go find dinner the current state is: mirror-update02 has been removed from our inventory, cron.service is stopped and disabled on mirror02, and I'm holding flocks on the 17 lock files used by mirror-update02. On mirror-update03 everything is running and mostly working as far as I can tell. The one exception is the centos stream sync are failing because we need toget the new | 00:24 |
| clarkb | server allowed to fetch from the centos official mirrors. Otherwise we can switch back to one of the less official mirrors | 00:24 |
| clarkb | I think we can probably proceed with shutting down/deleting mirror-update02 in the morning if nothing comes up between now and then. We can also likely update dns for mirror-update.o.o to point at mirror-update03.o.o at this point | 00:25 |
| clarkb | and now dinner | 00:25 |
| tonyb | I'll escalate to apevec and the centos infra team | 01:22 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 02:02 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 02:02 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 02:02 |
| opendevreview | Tony Breeds proposed opendev/glean master: Switch to new devstack based testing https://review.opendev.org/c/opendev/glean/+/953163 | 02:22 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 02:24 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 02:24 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 02:33 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 02:33 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 03:49 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 03:49 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 04:03 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 04:04 |
| *** ykarel_ is now known as ykarel | 04:18 | |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 04:47 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 04:47 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 04:47 |
| amoralej | tonyb, clarkb I'll ping the centos infra team who maintain that in their irc channel | 06:21 |
| amoralej | ftr https://pagure.io/centos-infra/issue/1706 | 07:13 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 08:32 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 10:52 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 10:54 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 10:55 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 11:08 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 11:37 |
| fungi | clarkb: tonyb: dpawlik6: apevec: amoralej: alternatively, we keep a mirror-update.opendev.org cname pointed to the active server, if that's even more convenient for them | 12:58 |
| amoralej | it should be fixed now | 13:08 |
| amoralej | if you can test it | 13:08 |
| fungi | thanks! i should be able to, gimme a sec | 13:08 |
| fungi | amoralej: CentOS.org rsync service (centoss8.centos.org) @ERROR: Unknown module 'centos-stream-full' | 13:11 |
| fungi | still seeing that when i try | 13:11 |
| fungi | specifically testing from mirror-update03.opendev.org 104.130.124.190 2001:4800:7818:104:be76:4eff:fe01:325c | 13:13 |
| opendevreview | Merged opendev/zone-opendev.org master: Point mirror-update to mirror-update03 https://review.opendev.org/c/opendev/zone-opendev.org/+/953140 | 13:15 |
| amoralej | that's the last log entry in https://static.opendev.org/mirror/logs/rsync-mirrors/centos-stream.log ? | 13:17 |
| fungi | my manual attempt at 13:12 utc isn't reflected there but the error looks the same | 13:18 |
| amoralej | ah, ok | 13:19 |
| fungi | the last one in the log from about an hour ago was connecting to mref1.uw2.stream.centos.org while mine ended up at centoss8.centos.org | 13:19 |
| amoralej | but you are using rsync://rsync.stream.centos.org/centos-stream-full/9-stream/ right? | 13:19 |
| amoralej | so centoss8 must be some mirror in centos infra i guess | 13:19 |
| fungi | rsync://rsync.stream.centos.org/centos-stream-full/9-stream/ yes | 13:20 |
| fungi | i was basically running the same script that we run from cron | 13:20 |
| amoralej | may you retry now? | 13:20 |
| fungi | this time it hit multacom-1.sponsored.centos.org and seems to be working | 13:21 |
| fungi | thanks amoralej! | 13:21 |
| amoralej | it can take some time to apply in all servers, but Fabien from centos infra told me it should be fine now | 13:21 |
| fungi | cool, much appreciated | 13:22 |
| fungi | i'll keep an eye out to make sure the cron for it around 18:00 utc also runs clean | 13:23 |
| fungi | my manual run of the script completed successfully, i'm running it one more time as a double check (should roughly no-op unless there were package updates within the past few minutes). seems to be reaching centosb20.centos.org this time | 13:31 |
| fungi | and it already finished, yep | 13:31 |
| fungi | looks good, i'm closing my screen session on that server but i was redirecting all output to /var/log/rsync-mirrors/centos-stream.log if anyone's curius | 13:32 |
| fungi | infra-root: i'm pruning /opt/backups-202010 on backup01.ord.rax in a root screen session there. will log when complete | 13:34 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 13:44 |
| mnasiadka | clarkb, tonyb: Any idea why changing nodeset on the devstack based centos 10 job definition ends up with probably loosing all other variables from parent job? devstack seems to ask for database password on the console - see https://zuul.opendev.org/t/openstack/build/f5d52942f66245e996a676bd12c3389d ;-) | 14:38 |
| mnasiadka | although it seems the vars are there looking at https://03a54a18e5e7f9d6e9a2-ece6d16851dc34e48db19a7f0145a6c3.ssl.cf1.rackcdn.com/openstack/f5d52942f66245e996a676bd12c3389d/zuul-info/inventory.yaml | 14:39 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 14:45 |
| mnasiadka | Ok, I think I got it - it seems the node needs to be named controller | 14:45 |
| fungi | mnasiadka: yes, devstack jobs normally have custom nodesets that name the nodes, which becomes important in multinode versions of the jobs | 14:46 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 14:48 |
| clarkb | infra-root looks like centos stream mirroring should be happy now and the mirror-update.o.o cname update change landed. Any objections to me deleting mirror-update02 at somepoint between meetings today? | 14:58 |
| clarkb | then I'll approve https://review.opendev.org/c/opendev/zone-opendev.org/+/953141/ and we should be done with mirror-update upgrade stuff | 14:59 |
| fungi | sounds good, thanks! | 15:02 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 15:05 |
| mnasiadka | fungi or clarkb - can you mark https://review.opendev.org/c/openstack/diskimage-builder/+/934045 as ready? I'm not the original author so can't move it out of WIP | 15:05 |
| fungi | mnasiadka: done | 15:06 |
| mnasiadka | thx | 15:06 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 15:06 |
| fungi | or will be once gertty finishes syncing my edit | 15:06 |
| fungi | it hit an error the first time, for some reason, but looks like it took when i retries | 15:07 |
| fungi | retried | 15:07 |
| fungi | nope, still showing wip | 15:07 |
| clarkb | I think you may need to be admin to do that (or the original author?) we can also update acls to allow cores to do it too but I doubt that was done here | 15:08 |
| fungi | aha, i need to elevate to admin privs, the acl for dib doesn't grant wip toggle to the core group | 15:08 |
| fungi | yes, that exactly | 15:09 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 15:09 |
| mnasiadka | ok then, centos stream 10 and rocky linux 10 should be good, let me go the rabbit hole of reviewing the whole chain | 15:11 |
| clarkb | I've responded to the cfn repo question on the list. Let me know if there is any additional followup that you think we should do from our side | 15:11 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 15:11 |
| clarkb | mnasiadka: exciting. I should be able to review them after meetings today as well | 15:11 |
| fungi | mnasiadka: okay, it's marked ready now | 15:13 |
| clarkb | re mirror-update02 I'll plan to clean that server up between the openstack tc meeting and the infra meeting. so 1800-1900 hour block. Let me know if you see a reason not to or if you need to copy any data etc. cc infra-root | 15:17 |
| clarkb | mnasiadka: will you rebase the centos 10 change(s) on top of the new ps for 949942? | 15:23 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 15:24 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 15:24 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 15:24 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 15:24 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 15:24 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 15:25 |
| clarkb | aha there they are | 15:25 |
| mnasiadka | yup, all rebased now | 15:25 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 16:04 |
| clarkb | looking at grafana mirror.debian and mirror.debian-security were last released about a day ago. I'll double check logs for those (I suspect that may simply be due to a lack of updatse and reprepro is generally working for ubuntu and others) before I delete the old server | 16:07 |
| corvus | it looks like only the ze servers were updated in the zuul restart... there's a good chance that's related to my server replacement and maybe we do still need to clear a cache somewhere... but... | 16:25 |
| corvus | the weird thing is that the zuul_reboot.log is empty. | 16:25 |
| corvus | oh, hrm... /var/log/ansible/zuul_reboot.log.1 is not empty and has the same timestamp | 16:25 |
| corvus | okay, not sure what's up with the rotation, but the .1 log confirms that the issue was a docker command problem (so, fallout from the noble upgrade -- not server replacement cache issue) | 16:27 |
| corvus | fix inbound | 16:27 |
| fungi | thanks for checking that | 16:27 |
| clarkb | oh! the container names change | 16:35 |
| clarkb | the commands themselves don't but some of the outputs do. I bet that is related | 16:35 |
| opendevreview | James E. Blair proposed opendev/system-config master: Use docker-compose ps -qa for graceful restart https://review.opendev.org/c/opendev/system-config/+/953225 | 16:38 |
| corvus | weirder than that, it turns out | 16:39 |
| clarkb | I +2'd but had a super duper minor nit | 16:41 |
| opendevreview | James E. Blair proposed opendev/system-config master: Use docker-compose ps -qa for graceful restart https://review.opendev.org/c/opendev/system-config/+/953225 | 16:42 |
| corvus | clarkb: i do want to fix that :) | 16:42 |
| corvus | that comment was really key to understanding how this script was intended to work; i certainly don't want to undo that | 16:43 |
| corvus | (i couldn't figure out how there wasn't a race condition between stopping and then listing; that comment explained it) | 16:43 |
| corvus | i'm manually restarting the rest of the cluster | 16:47 |
| clarkb | ya we basically list all containers that exist running and stopped. Then ask them all to stop ignoring errors from those already stopped. Then wait for that list to enter a stopped state | 16:48 |
| corvus | hrm, looks like the non-executors got an extra change snuck in, and it is, unfortunately, a change that affects the executors and schedulers. but we don't use the feature (nodeset alternatives) so it's not a big deal. we're not looking to make a release with this, so running split for a bit doesn't matter. | 16:52 |
| clarkb | good to know | 16:53 |
| clarkb | we could trigger the playbook out of band with the weekly update once your fix lands if you want to get everything back in sync and confirm the update works | 16:54 |
| corvus | i think i'll just wait for the weekend | 16:55 |
| clarkb | wfm | 16:55 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 16:56 |
| clarkb | fungi: the debian and debian-security repro last updated date from grafana is pointing to a real issue | 16:56 |
| clarkb | fungi: the log indicates the keys were using are unknown | 16:56 |
| clarkb | fungi: give me a couple mintues and I'll call out specific | 16:57 |
| fungi | clarkb: good catch, maybe they were manually added to a keychain before or something | 16:57 |
| clarkb | I think I just found a new gitea bug: You can't click on line numbers and get links to those lines anymore? | 16:58 |
| clarkb | https://opendev.org/opendev/system-config/src/commit/7a4e0f5340498a44a7876c7c1c72f8c91ae784c5/playbooks/roles/reprepro/files/debian/config/updates line 7 the first key there is complained about | 16:58 |
| clarkb | https://opendev.org/opendev/system-config/src/commit/7a4e0f5340498a44a7876c7c1c72f8c91ae784c5/playbooks/roles/reprepro/files/debian-security/config/updates line 6 the first key there is complained about | 16:59 |
| clarkb | hold on testing gitea now really quickly | 16:59 |
| clarkb | https://opendev.org/opendev/system-config/src/commit/7a4e0f5340498a44a7876c7c1c72f8c91ae784c5/playbooks/roles/reprepro/files/debian-security/config/updates#L6 If I navigate via the file browser on the left the resulting rendered page doesn't let me click on links. If I navigate the old school way it does | 16:59 |
| fungi | that does sound like a bug, but at least there's a workaround | 17:00 |
| clarkb | https://opendev.org/opendev/system-config/src/commit/7a4e0f5340498a44a7876c7c1c72f8c91ae784c5/playbooks/roles/reprepro/tasks/debian.yaml#L6-L8 for debain we definitely only have that second key managed | 17:00 |
| fungi | okay, so we're missing a keyfile i guess | 17:01 |
| clarkb | and this https://opendev.org/opendev/system-config/commit/7136db339e9438d12d3f08021a5bab0c0d199c48 commit removed the old keys | 17:01 |
| clarkb | fungi: based on ^ I Think the issue is that I removed the keys from management but didn't fully remove them where we are using them. Do you think the fix is simply to remove the key from each of othose updates files? | 17:01 |
| clarkb | to force it to use the key we do manage instead? | 17:01 |
| fungi | yeah, i think that should work as long as nothing is being signed exclusively by the old keys any longer | 17:01 |
| clarkb | if so I can make that change. I just want to make sure I understand the issue before I start pushing buttons | 17:02 |
| fungi | and would be the preferred fix | 17:02 |
| clarkb | ok lets try that then. I'll work on an update | 17:02 |
| fungi | thanks! | 17:02 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 17:04 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 17:05 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 17:05 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Stop using old debian keys in reprepro https://review.opendev.org/c/opendev/system-config/+/953226 | 17:06 |
| clarkb | something like that maybe | 17:06 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 17:06 |
| clarkb | I will hold off on deleting mirror-update02 for now as a result of the problem 953226 will hopefully fix | 17:10 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 17:15 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 17:15 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 17:16 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 17:16 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 17:18 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 17:19 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 17:19 |
| mnasiadka | ok then, sorry for the flood - now the removal patch is really only removing | 17:20 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 17:20 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 17:20 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 17:20 |
| clarkb | trying to narrow down this gitea behavior I'm beginning to wonder if it is possibly a cached assets problem. I can't reproduce anymore after a hard refresh | 17:24 |
| clarkb | I was going to file an issue upstream if I could narrow it down but now I think it may just be a browser cache issue. | 17:24 |
| corvus | clarkb: i followed one of your links, and clicking line numbers worked; using the file browser on the left to change to a new file, still worked, clicking "code", picking a file there, and clicking lines also worked. | 17:28 |
| clarkb | corvus: ok thats more evidence in the it was probably a cache or pebkac issue | 17:29 |
| corvus | okay, i'm happy with the newly updated state of the launchers, i'm ready to switch the other tenants to start using niz as previously discussed... any objections? | 17:47 |
| corvus | ("other" including "openstack" in this case) | 17:48 |
| fungi | no objections from me | 17:50 |
| clarkb | corvus: mechanically nodepool will keep running with the existing max-servers values but due to nodeset construction we expect the vast majority of load to switch to zuul launcher right? | 17:50 |
| clarkb | no objection from me. I just want to understand the expected outcome from the initial changes | 17:50 |
| corvus | yes -- except i'm not sure about the mix.. it might be closer to 50% of the load until we merge everything in https://review.opendev.org/q/hashtag:opendev-niz+status:open | 17:51 |
| corvus | so roughly speaking, when https://review.opendev.org/952712 merges openstack may go from 0% niz to 50% niz | 17:52 |
| clarkb | got it | 17:52 |
| corvus | then progressively up to 99%-100% as we merge that stack | 17:52 |
| corvus | (not sure how advantageous that really is... but it could help us identify load issues before they become a problem) | 17:53 |
| corvus | oh, i think there is a problem with my "reduce launches to 2GB-ram" plan... they need a lot of temporary space, and they only get an 18G /opt... the 8GB-ram nodes had 70G... | 17:58 |
| fungi | mmm, so it's less about the memory on those flavors and more about the disk | 17:59 |
| fungi | and not wanting to attach cinder volumes i guess | 17:59 |
| clarkb | ya I guess our options are bigger nodes or using cinder volumes | 17:59 |
| corvus | yeah; ram and cpu look great with the smaller size | 18:00 |
| corvus | i wonder if we should update launch-node to be able to launch and attach a cinder volume on opt | 18:00 |
| clarkb | corvus: it can | 18:00 |
| clarkb | or wait maybe not for /opt because that is where the ephemeral goes? | 18:01 |
| corvus | oh neat | 18:01 |
| clarkb | in general it can. But /opt may be special when we have epehermal drives too | 18:01 |
| corvus | easy to move the launcher temp dir | 18:01 |
| clarkb | use --volume and --mount-path flags | 18:01 |
| clarkb | corvus: ya the make_swap.sh script is hardcoded to put the extra space of the ephemeral drive on /opt | 18:02 |
| clarkb | then I think we run mount_volume.sh after it with the --volume and --mount-path info and it may be able to take over /opt effectively orphaning the ephemeral drive area | 18:03 |
| clarkb | which is probably fine for something like this | 18:03 |
| clarkb | oh you also need to set --fs-label | 18:04 |
| corvus | okay, so how about i relaunch the zl nodes with --volume 80 --mount-path /var/lib/zuul --fs-label zuul | 18:04 |
| corvus | then after that, we'll switch the launcher temp dir to /var/lib/zuul/launcher-tmp | 18:05 |
| clarkb | corvus: that wfm. Note I think --volume expects a volume uuid from a volume you have precreated | 18:05 |
| clarkb | so basically you create the volume then pass its uuid to launch node and it should configure lvm and fstab etc | 18:05 |
| corvus | oooh | 18:06 |
| corvus | is this better, or should we switch back to the larger node size? | 18:06 |
| clarkb | I feel like it is probably 6 one way half a dozen the other. THis way more closely reflects our intent, but the larger node size is probably simpler overall | 18:07 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 18:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Add support for building Fedora 40 https://review.opendev.org/c/openstack/diskimage-builder/+/922109 | 18:08 |
| corvus | okay, given how this is set up, if we just want to add a volume to the 2gb nodes, i'll just manually do that and then run the mount_volume.sh script. it should be close enough to what we get from launching the node. | 18:09 |
| opendevreview | Merged opendev/system-config master: Stop using old debian keys in reprepro https://review.opendev.org/c/opendev/system-config/+/953226 | 18:10 |
| clarkb | corvus: agreed that should produce a basically identical result | 18:10 |
| clarkb | 953226 has been deployed. It deployed after the 18:10 debian mirror run but should be in place for the 18:31 debian-security run | 18:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/diskimage-builder master: Add support for building Fedora 40 https://review.opendev.org/c/openstack/diskimage-builder/+/922109 | 18:17 |
| fungi | good enough | 18:21 |
| fungi | i can always manually run the other if there's a problem | 18:21 |
| clarkb | I think I'm ok to wait. My main concern is that I want mirror-update02 out of the way before it can accidentally create conflicts but my conflict resolution system has been in place for almost 24 hours and it seems to eb working great so far | 18:23 |
| opendevreview | James E. Blair proposed opendev/system-config master: Move zuul launcher temp dir https://review.opendev.org/c/opendev/system-config/+/953236 | 18:27 |
| clarkb | corvus: if the launcher data was important I would say lets use a cinder volume for sure. But since its largely ephemeral data shuffling from one place to another I think we're ok with either option (just to clarify furhter on my 6 one way half dozen the other comment. It isn't always the case) | 18:27 |
| corvus | yes i agree. i'm torn between right-sizing this (cinder) and make it easy to replace (oversize) | 18:28 |
| clarkb | corvus: one quick question posted to that change | 18:28 |
| corvus | but i have completed all the work to use cinder | 18:28 |
| corvus | replied | 18:29 |
| corvus | the new volumes are in place, and we can/should merge that change now to use them. | 18:29 |
| clarkb | cool +2 from me | 18:30 |
| fungi | and approved | 18:31 |
| clarkb | debian-security is doing work now and not immediately bailing out | 18:32 |
| fungi | seems fixed in that case | 18:32 |
| clarkb | Released volume mirror.debian-security successfully | 18:35 |
| clarkb | afsmon hasn't run yet to reflect that in grafana yet, but I think this is happy now | 18:36 |
| opendevreview | Merged openstack/project-config master: Switch all Zuul tenants to use niz nodesets https://review.opendev.org/c/openstack/project-config/+/952712 | 18:43 |
| opendevreview | James E. Blair proposed openstack/project-config master: Revert "Switch all Zuul tenants to use niz nodesets" https://review.opendev.org/c/openstack/project-config/+/953241 | 19:06 |
| corvus | flex-sjc3 got better, so i'm going to let the revert go in on its own | 19:19 |
| opendevreview | Merged opendev/system-config master: Use docker-compose ps -qa for graceful restart https://review.opendev.org/c/opendev/system-config/+/953225 | 19:25 |
| clarkb | mnasiadka we've noticed that openstack is relying on ubuntu bionic and focal for translation sync jobs (bionic) and older stable branch jobs (focal). We were curious if you wanted to port those two images to zuul launcher since you did a good chunk of the others | 19:26 |
| fungi | technically not "stable" branch, but only "unmaintained" branch | 19:27 |
| fungi | i also mentioned it in #openstack-release in case the unmaintainers are interested, as this would impact jobs for unmaintained/zed and older branches | 19:27 |
| mnasiadka | clarkb: can do, will post some patches tomorrow | 19:35 |
| clarkb | mnasiadka: thank you! | 19:35 |
| corvus | remote: https://review.opendev.org/c/zuul/zuul/+/953244 Launcher: handle API errors in request assignment [NEW] | 19:42 |
| corvus | that should take care of the issues just observed | 19:42 |
| opendevreview | Merged openstack/project-config master: Revert "Switch all Zuul tenants to use niz nodesets" https://review.opendev.org/c/openstack/project-config/+/953241 | 19:51 |
| clarkb | the debian sync should start in about 10 minutes. I'm going to grab lunch in the meantime and if that was successful afterwards I'll proceed wtih mirror-update02 deletion | 19:59 |
| tonyb | are we syncing stretch and buster? | 20:00 |
| clarkb | just bullseye and bookworm iirc | 20:00 |
| clarkb | double checking | 20:00 |
| clarkb | confirmed https://opendev.org/opendev/system-config/src/commit/7a4e0f5340498a44a7876c7c1c72f8c91ae784c5/playbooks/roles/reprepro/files/debian/config/distributions | 20:01 |
| tonyb | Awesome. That explains the errors I saw in DIB | 20:01 |
| opendevreview | Merged opendev/system-config master: Move zuul launcher temp dir https://review.opendev.org/c/opendev/system-config/+/953236 | 20:11 |
| clarkb | the debian mirror process is running now too. Still in progress though | 20:28 |
| clarkb | Released volume mirror.debian successfully | 20:33 |
| clarkb | I am going to delete mirror-update02 now | 20:34 |
| clarkb | there are no volumes attached so no extra cleanups other than the server | 20:34 |
| clarkb | #status log Deleted mirror-update02.opendev.org (ad3289c8-8eeb-4031-b723-c16b6a57b7e3) it has been replaced by mirror-update03.opendev.org. | 20:36 |
| opendevstatus | clarkb: finished logging | 20:37 |
| opendevreview | Merged opendev/zone-opendev.org master: Remove mirror-update02 from DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/953141 | 20:39 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 20:49 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 20:49 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Disable nodepool testing https://review.opendev.org/c/openstack/diskimage-builder/+/953246 | 20:49 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 20:55 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 20:55 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 21:05 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 21:05 |
| clarkb | tonyb: mnasiadka I think the dib testing and centos 10 stack lgtm. I did have one note on the rockylinux 10 change | 21:22 |
| clarkb | there were a few other nits here and there but at this point I think what we have is largely functional and we should try to land it and then do followup changes for those nits as necesasry | 21:22 |
| clarkb | I'm going to pop out for a bike ride. And then afterwards I need to head over to my remote backup location and find out why the backup device is not accepting ssh connections anymore | 21:35 |
| clarkb | my brother moved and afterwards the small machine I have set up to collect my backups at his place rejects ssh. I'm guessing it wants to fsck ro something along those lines after being powered down in a way it didn't expect | 21:37 |
| clarkb | clearly I should upgrade this setup with an ilo | 21:37 |
| clarkb | no rush but I didn't want to forget: fungi did backup pruning complete yet? | 21:38 |
| clarkb | I don't recall how long that takes on the larger backup server (thinking of my own backup server reminded me that was ongoing) | 21:38 |
| corvus | reverting the provider out of the openstack tenant also left a handful of nodes in the "outdated" state. https://review.opendev.org/953251 should correct that (or, alternately, putting the provider back in the tenant) | 22:49 |
| corvus | i monkeypatched that in so that the nodes are cleaned up | 23:04 |
| fungi | #status log Pruned /opt/backups-202010 on backup01.ord.rax reducing volume usage from 90% to 33% | 23:04 |
| fungi | clarkb: it did finally, thanks for the reminder! | 23:04 |
| opendevstatus | fungi: finished logging | 23:05 |
| fungi | slow wiki api is slow... thanks ai! | 23:05 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add support for CentOS Stream 10 https://review.opendev.org/c/openstack/diskimage-builder/+/934045 | 23:11 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add Rocky Linux 10 support to rocky-container element https://review.opendev.org/c/openstack/diskimage-builder/+/952548 | 23:11 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 23:12 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 23:12 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove mirror from experimental debian jobs https://review.opendev.org/c/openstack/diskimage-builder/+/953256 | 23:12 |
| tonyb | One last CI run and then I think we can merge 934045 and ancestors | 23:13 |
| corvus | both zuul launchers are running on their new tmp space now and i cleaned up the old stuff | 23:38 |
| clarkb | corvus: the provider node fix up lgtm | 23:48 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!