| clarkb | I couldn' | 01:03 |
|---|---|---|
| clarkb | er | 01:03 |
| clarkb | I couldn't help myself and double cehcked that review02 is happily parked in the emergency file and only running the mariadb container | 01:03 |
| clarkb | and now I smell good so will disappear until tomorrow | 01:04 |
| Clark[m] | *smell food. But maybe I smell good too | 01:23 |
| Clark[m] | fungi: any idea why openstack/aetos is only just now getting created on GitHub? | 02:28 |
| Clark[m] | Looks like the jobs that should mirror to GitHub were added a while back and the last change merged was on the 11th. Not sure what would have tripped it over today. But slightly concerned it's related to the server switch somehow | 02:31 |
| Clark[m] | Ok the reason is that openstack/aetos was not added to openstack governance until today | 02:40 |
| Clark[m] | So just a coincidt | 02:40 |
| Clark[m] | *coincidence | 02:40 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: Add limit-log-file-size role https://review.opendev.org/c/zuul/zuul-jobs/+/945795 | 07:18 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: mirror-workspace-git-repos: Allow deleting current branch https://review.opendev.org/c/zuul/zuul-jobs/+/946033 | 07:26 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: emit-job-header: add region information https://review.opendev.org/c/zuul/zuul-jobs/+/946035 | 07:45 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: mirror-workspace-git-repos: Allow deleting current branch https://review.opendev.org/c/zuul/zuul-jobs/+/946033 | 07:48 |
| *** Adri2000_ is now known as Adri2000 | 08:11 | |
| *** ralonsoh_ is now known as ralonsoh | 10:19 | |
| opendevreview | Merged openstack/project-config master: Remove noop jobs for openstack-ansible-tests https://review.opendev.org/c/openstack/project-config/+/947665 | 12:49 |
| mnasiadka | corvus/clarkb : Any pointers where to start with remaning zuul-launcher images (e.g. rocky)? | 14:47 |
| corvus | mnasiadka: yeah, the two links at the bottom of my email: https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/ZLZ7OUFAOAZ7OS2PO2MHGJJKOBYVWB3G/ | 14:48 |
| corvus | mnasiadka: one of them is the nodepool config, and the other is the new zuul config | 14:48 |
| mnasiadka | corvus: gah, I got blind after being on the other end of the world for three weeks, sorry :) | 14:48 |
| mnasiadka | (and I read that email like 2 minutes ago) | 14:49 |
| corvus | no problem! | 14:49 |
| corvus | since the debuntu images are in both, you can probably see the structure and similarity. but of course let me know if anything is unclear | 14:49 |
| clarkb | ya in theory the process is roughly mapping the configs and pushing a change up. Then wait for zuul to produce results that either look good or require tweaking. Update and repeat | 14:50 |
| mnasiadka | clarkb: sounds like my everyday job :) | 14:50 |
| clarkb | one of the big beneifts to the new system is you get pre merge feedback on the image builds which we don't hav with nodepool | 14:50 |
| fungi | yeah, we can even add jobs to exercise each build of an image before putting it into wider use | 14:55 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:21 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:22 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:22 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:24 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:27 |
| corvus | mnasiadka: oh one new thing: there needs to be an image definition here: https://opendev.org/opendev/zuul-providers/src/branch/master/zuul.d/images.yaml | 15:28 |
| mnasiadka | corvus: yeah, getting there :) | 15:28 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:29 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:32 |
| mnasiadka | corvus: seems that's not enough - do I also need to add it to providers and labels? | 15:34 |
| corvus | oof, i think we may need to merge the image definition before the job. sorry, and thanks for your patience as we work through this for the first time. :) | 15:35 |
| corvus | mnasiadka: so i think if you just split the images.yaml part of that into its own change, we can merge that real quick | 15:36 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu-noble-arm64 image definition https://review.opendev.org/c/opendev/zuul-providers/+/947843 | 15:37 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:38 |
| opendevreview | Merged opendev/zuul-providers master: Add ubuntu-noble-arm64 image definition https://review.opendev.org/c/opendev/zuul-providers/+/947843 | 15:38 |
| corvus | (basically we're sort of clamining that this repo is going to own the ubuntu-noble-arm64 image, and that needs to happen before any build jobs for it can run) | 15:39 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:39 |
| mnasiadka | Seems binary-arm64 is not mirrored, let me try to unset the mirror | 15:55 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 15:55 |
| clarkb | mnasiadka: we do have a mirror but it uses a different path iirc | 15:58 |
| clarkb | https://mirror.dfw.rax.opendev.org/ ubuntu vs ubuntu-ports there | 15:58 |
| fungi | yeah, unlike debian, ubuntu puts their non-x86 packages in a different repository entirely | 15:59 |
| fungi | so in debian the amd64 and arm64 packages share a repository, but in ubuntu they don't | 15:59 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 16:03 |
| clarkb | corvus: fungi: to followup on yesterdays' gerrit quay image build problem I switched the container runtime to podman and everything looks happy at a quick glance. I think the issue was the docker build itself was unaawre of the mirrors so not getting the correct image version from the buildset registry and instead getting the 2 year old version from quay | 16:04 |
| clarkb | I think maybe I should go ahead and convert our base non container image builds jobs (not the docker image build jobs) to use the podman container_runtime and make that default? | 16:05 |
| clarkb | this will affect codesearch and maybe lodgeit at this point since they already switched but I don't expect problems. Do we know of any other image build issues when going from docker to podman to look out for? | 16:05 |
| corvus | clarkb: i thought we were using buildx | 16:07 |
| clarkb | corvus: I think we are because that is the default for docker, but you have to configure mirrors for it specially using /etc/buildkitd.toml or something | 16:08 |
| clarkb | corvus: as an alternative we could add that in I guess | 16:08 |
| corvus | if we want to use zuul as a model: https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L276-L278 | 16:09 |
| corvus | maybe we can get all the new build goodness with podman? i'm not sure what the current state is | 16:10 |
| corvus | (istr that there were some new features we wanted on the docker side which is why we did that, but maybe they were already there on the podman side because of buildkit? i don't recall the details) | 16:11 |
| clarkb | corvus: based on https://review.opendev.org/c/opendev/system-config/+/882900/5..6/zuul.d/docker-images/gerrit.yaml it seems like podman just worked at least for the mirror situation. I'm not sure if podman also just gives us the buildkit/buildx functionality | 16:11 |
| clarkb | corvus: ya I'm wondering too. It wouldn't surprise me if that is available by default with podman | 16:12 |
| clarkb | I think multi stage builds and maybe some variants of COPY commands were what zuul needed. Might be worth toggling and see if it just works? | 16:12 |
| corvus | remote: https://review.opendev.org/c/zuul/zuul/+/947848 DNM: test podman image build [NEW] | 16:13 |
| corvus | whichever we do, i do think it's valuable for zuul and opendev to try to make the same decision here | 16:14 |
| corvus | (or, at least, figure out a good reason why not to) | 16:14 |
| corvus | just so we have a good well-tested approach to image building an use | 16:15 |
| clarkb | ++ | 16:20 |
| fungi | clarkb: are we okay to go ahead with 944408 now? (followed by 944409) | 16:35 |
| fungi | just trying to clean up some lingering reviews | 16:36 |
| clarkb | fungi: I think we probably are. The main thing is we should plan to restart gerrit when we land that | 16:37 |
| clarkb | so maybe we want to wait anotherday or two to spread that one? | 16:37 |
| clarkb | s/spread that one/spread that out/ | 16:37 |
| fungi | ah, good point. also there's 944401 for the limnoria image which could probably go in, though we don't want to interrupt meetings so need a quiet time for that as well | 16:37 |
| fungi | and there's some changes for refstack testing/image, should those be abandoned? | 16:38 |
| fungi | or do we want to land them before we wind it down? | 16:38 |
| fungi | looks like the service is still up for the moment | 16:39 |
| clarkb | ++ on limnoria. For refstack we should probably just abandon the chagne we have for that | 16:39 |
| clarkb | I think we should make an argument to wind it down and if we keep supporting it then we're not arguing very well :) | 16:40 |
| fungi | that's 944403 in refstack and 944404 in system-config | 16:40 |
| fungi | they're both your changes but i can abandon them if you like | 16:40 |
| clarkb | I can do it | 16:41 |
| clarkb | fungi: do you think I should leave the two refstack changes open and only abandon system-config? | 16:43 |
| clarkb | or abandon the whole lot? | 16:43 |
| fungi | abandon the whole lot. i'm going to push changes to retire the repo | 16:43 |
| clarkb | on it | 16:43 |
| clarkb | and done | 16:44 |
| fungi | thanks! i'm working on the series of retirement changes now, before i forget | 16:45 |
| clarkb | corvus: I don't know what our multiarch story is when building with podman. That would be another thing to consider. However, the only container image we currently build that depends on that is nodepool-builder. With the move to zuul-launcher maybe we're ok with letting that die on the vine and then addressing it later if necessary with podman | 16:54 |
| clarkb | (I know podman can do it, but I'm pretty sure the mechanics are different) | 16:54 |
| corvus | agree | 16:55 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu noble arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/947841 | 17:17 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Wind down refstack.openstack.org https://review.opendev.org/c/opendev/system-config/+/947856 | 17:18 |
| mnasiadka | corvus: I've noticed OSUOSL is not in zuul.d/providers.yaml - and I think that's the only ARM64 cloud that OpenDev uses? Should I just add section at the bottom of the file? | 17:23 |
| clarkb | oh yup we'll need that provider to run the arm64 image builds (since we don't cross build) | 17:24 |
| corvus | right now we're building on nodepool nodes (bootstrapping and all) | 17:24 |
| corvus | but we will need that in the future for sure | 17:24 |
| corvus | mnasiadka: so yes, feel free to do that, but you can do it as a separate change | 17:25 |
| corvus | mnasiadka: (but also, feel free to leave it to me/us if you want) | 17:25 |
| corvus | (basically, once this job succeeds and produces the first image build, it will be useful to add the osuosl provider, and zuul will upload the build to it) | 17:27 |
| mnasiadka | I can do that tomorrow, or at least fail trying :) | 17:27 |
| mnasiadka | Is zuul-launcher also something I could use in a new zuul deployment outside of OpenDev? I haven't seen any docs on the Zuul site about it. | 17:28 |
| corvus | mnasiadka: not yet; opendev is piloting it, and there are intentionally no docs yet. we're still radically changing the configuration of it from day to day. | 17:28 |
| mnasiadka | Ok, so I'll stick to nodepool and migrate one day | 17:29 |
| corvus | yeah, hopefully within a few months it'll be settled enough to recommend | 17:29 |
| corvus | (and then there will be a long period where both will work) | 17:30 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Set review.opendev.org TTL back to default of 1 hour https://review.opendev.org/c/opendev/zone-opendev.org/+/947858 | 17:32 |
| clarkb | I realized ^ that is probably safe enough to do at this point even if we're not 100% certain a rollback won't happen | 17:32 |
| clarkb | but I'll let reviewers decide if they want to wait on that for when we start pulling stuff out of the inventory etc | 17:33 |
| fungi | oh, speaking of dns, the refstack dns entries are in cloudflare so i'll clean those up once the system-config change merges and we shut the server down | 17:41 |
| clarkb | fungi: I +1'd but didn't +2 the change because i think we should announce the turn down | 17:41 |
| clarkb | but I wasn't sure where that communication should be sent. I'm +2 once we do that though | 17:41 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Prepare for retirement of RefStack repositories https://review.opendev.org/c/openstack/project-config/+/947859 | 17:42 |
| fungi | clarkb: agreed, i was just digging for the last time that was discussed | 17:42 |
| fungi | most recent mention of refstack on openstack-discuss (not including the confused-looking not-really-a-vulnerability report) was this interop wg ptg summary from 2 years ago: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/ADOH77CXJAFQRF4CVHNPQCLIEXRLHMQJ/ | 17:45 |
| fungi | so i don't think the foundation staff ever announced anything about the trademark policy changes | 17:46 |
| fungi | i'll wip the system-config change while we get some sort of public announcement out | 17:46 |
| clarkb | sounds good thanks | 17:46 |
| fungi | https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/Q4OS2JXADLFG4YK75NH2QORZZRFINUKL/ doesn't mention refstack by name but is the closest thing i can find to a tacit agreement to get rid of it | 17:53 |
| fungi | "...doesn't look like the community is interested in running actual interoperability testing. Let us know if you disagree..." [crickets] | 17:54 |
| fungi | i did find "RESOLVED, that the Interoperability Working Group be disbanded." in https://board.openinfra.org/meetings/minutes/07022024 | 18:04 |
| clarkb | so maybe we just need to make a note that due to ^ the service is shutting down? | 18:05 |
| fungi | well, that's far from being an announcement. i'm asking the trademark folks on openinfra staff whether we need something more explicit first | 18:07 |
| clarkb | ++ | 18:13 |
| clarkb | I'm going to pop out for some early lunch to sneak that in before our meeting in ~47 minutes | 18:13 |
| clarkb | back by then | 18:13 |
| fungi | i plan to go out for an early dinner as soon as the opendev meeting concludes | 18:14 |
| fungi | but am around in the meantime | 18:14 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Migrate gerrit images to quay.io https://review.opendev.org/c/opendev/system-config/+/882900 | 19:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use podman to build non docker hub container images https://review.opendev.org/c/opendev/system-config/+/947872 | 19:59 |
| opendevreview | Clark Boylan proposed opendev/lodgeit master: Update lodgeit to use podman to build its container image https://review.opendev.org/c/opendev/lodgeit/+/947873 | 20:02 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing podman build of lodgeit https://review.opendev.org/c/opendev/system-config/+/945143 | 22:40 |
| clarkb | following up on system-config-run- jobs on noble and whether or not they need updates I think we're ok or at least we're pulling the right image from the buildset registry | 22:46 |
| clarkb | you can see the registry log here: https://zuul.opendev.org/t/openstack/build/60f1d3f12c384f7e90a0edc635ac6069/log/docker/buildset_registry.txt#4768-4849 matches with the system-config-run-review-3.10 job here: https://zuul.opendev.org/t/openstack/build/8987bde25f0c43d9a8099cfdfebaaaf1/log/job-output.txt#17793-17885 based on timestamps and some urls | 22:46 |
| clarkb | one thing I do have a question about is why https://zuul.opendev.org/t/openstack/build/8987bde25f0c43d9a8099cfdfebaaaf1/console#2/0/13/localhost pulls artifacts from the intermediate registry to localhost (which i think is the zuul executor) | 22:47 |
| clarkb | corvus: ^ you don't happen to know off the top of your head why that is happening do you? | 22:47 |
| clarkb | oh actually I think maybe that is the ipv6 proxy socat tunnel thing we do to populate the buildset registry | 22:48 |
| clarkb | so its "localhost" but really what we're doing is shuttling the bits into the buildset registry where we will fetch them later | 22:48 |
| clarkb | ya I think that is it. So while maybe less ideal in terms of confusing playbook/role names I don't think we're doign the wrong thing there | 22:49 |
| clarkb | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_898/openstack/8987bde25f0c43d9a8099cfdfebaaaf1/bridge99.opendev.org/screenshots/gerrit-main-page.png and this shows the gerrit version is ahead of any other builds we've properly published | 22:50 |
| clarkb | so tl;dr is that I think with podman building we correct use of the buildset registry on the build side then we're already good on the deployment side | 22:50 |
| clarkb | so really the main question is "are podman produced images good" and I have no realize to believe they are not based on the autoamted testing we've performed so far | 22:51 |
| clarkb | fungi: unrelated to ^ I notice we've been getting email responses from att to service-discuss-bounces because maybe someone signed up with a phone number email to sms gateway that is being shutdown? | 22:52 |
| clarkb | fungi: if that is the case should we go ahead and unsubscribe those email addresses? | 22:52 |
| clarkb | I guess if I login as list admin on that I should be able to see if that address is subscribed /me checks | 22:55 |
| clarkb | ok that address is not present so maybe this is just scam/phishing/spam trying to get us to follow the link or something | 22:58 |
| clarkb | (however, that link looks legit to me...) | 22:59 |
| clarkb | but I'm not following it | 22:59 |
| clarkb | oh maybe someone has a forward set up on their email client and the bounce is ending up all the way back to lists? that may explain it | 22:59 |
| clarkb | side note: the sign out button ends up very close to the remove all members button... | 23:02 |
| tonyb | JayF: Random gentoo question. I'm building a bunch of local container images including a gentoo one. It is >1.5Gb, I suspect because of the ports data and possibly locally build caches (I did `emerge --sync` and `emerge install sudo`) What's the safe way of removing all of that to reduce the size of the image? (on fedora I'm doing `dnf clean all; rm -fr /var/lib/dnf/repos/*` | 23:02 |
| clarkb | heh I just realized that we could text those numbers and ask. Maybe ths is the phish | 23:04 |
| clarkb | tonyb: https://wiki.gentoo.org/wiki/Knowledge_Base:Freeing_disk_space and https://wiki.gentoo.org/wiki/FAQ#Source_tarballs_are_collecting_in_.2Fvar.2Fcache.2Fdistfiles.2F._Is_it_safe_to_delete_these_files.3F has some suggestions | 23:07 |
| clarkb | looks like eclean automates some of it for you if given the appropriate commands/flags | 23:08 |
| tonyb | clarkb: Thanks | 23:09 |
| clarkb | though eclean will keep packge info for currently installed packages even when you supply the --deep flag (for deep clean). in The case of a container that may be all that is present? | 23:10 |
| clarkb | probably worth testing and falling back to manual cleanup if eclean is not aggressive enough | 23:10 |
| tonyb | Yeah, that's a solid start thank clarkb | 23:10 |
| corvus | clarkb: i'm back at the kb, but it looks like you talked yourself into an answer; i'll stand down unless you have followups or i misunderstood | 23:19 |
| Clark[m] | corvus: I think I understand it all and it's largely working as expected | 23:22 |
| Clark[m] | The only unexpected thing still is the names say "docker" but podman interprets the config properly anyway | 23:22 |
| corvus | names of what? | 23:25 |
| Clark[m] | One of the roles. Let me see if I can find it | 23:26 |
| corvus | yeah let's check that, because i'm almost certain that the non-docker path should avoid the word docker everywhere | 23:26 |
| Clark[m] | https://zuul.opendev.org/t/openstack/build/8987bde25f0c43d9a8099cfdfebaaaf1/console it's the playbook not role | 23:27 |
| Clark[m] | docker-image/pre.yaml | 23:27 |
| Clark[m] | So maybe that's just a bug in base-jobs. Using a docker specific name when things are generic enough for podman too | 23:27 |
| Clark[m] | Not a big deal other than potential confusion. But also something I think we can easily rename | 23:28 |
| tonyb | LOL: somehow the container image is bigger now ;P | 23:31 |
| tonyb | https://paste.opendev.org/show/bNYklnBSaA9gdUdnRgnN/ | 23:32 |
| tonyb | I guess it's not worth much more effort as the base image (gentoo/stage3) is already 1.11GB | 23:32 |
| corvus | Clark: yeah, i think this is doing the right thing, but renaming might be weird... the opendev-buildset-registry-consumer job (which is part of the parent hierarchy) is designed to work with both. but it runs the "docker" playbook. it doesn't really matter, because it's purpose is just to pull from the intermediate registry, and that role will work with either. | 23:44 |
| corvus | we could probably switch to the "container" version of that playbook, but that version has an extra task to install either docker or podman. which means if we used it from a "docker" job, because of the variable defaults, it would end up installing podman, which is weird. | 23:44 |
| corvus | i don't know why we have the ensure-docker/podman roles included in playbooks/container-image/pre.yaml. but i think we would need to figure that out and remove them if we were to switch opendev-buildset-registry-consumer to use that playbook. | 23:45 |
| corvus | or we just ignore it and try to remember this the next time we think it's weird. | 23:45 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
