| clarkb | infra-root we can meet here to keep logs separated and also to have the bot track notes for us | 19:02 |
|---|---|---|
| fungi | wfm | 19:02 |
| clarkb | #startmeeting infra | 19:02 |
| opendevmeet | Meeting started Thu Mar 16 19:02:34 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:02 |
| opendevmeet | The meeting name has been set to 'infra' | 19:02 |
| clarkb | #topic What do we do about docker deleting free orgs | 19:02 |
| clarkb | #link https://etherpad.opendev.org/p/MJTzrNTDMFyEUxi1ReSo | 19:03 |
| clarkb | Long story short Docker will remove access to free orgs on April 14. I believe this means we won't be able to manage our images including push new version of them to opendevorg/* and zuul/* | 19:03 |
| clarkb | If I've read their comms properly the public images already there will be accessible for 30 days after April 14 | 19:04 |
| clarkb | This puts us in a position where we need to find a new home for the images we host at opendevorg/ and zuul/ | 19:04 |
| clarkb | The etherpad linked above attempts to capture what our options are | 19:05 |
| clarkb | We can A) Move to a different free public registry like quay or github etc B) Host our own registry C) Apply for docker's DSOS open source program D) pay Docker to keep our existing orgs as is. THere is also a E) which is do more than one of options A-D | 19:06 |
| clarkb | informal discussion between different infra-root leave me to believe that none of us are too thrilled about C or D | 19:06 |
| fungi | and for completeness, f) stop hosting/publishing images | 19:06 |
| NeilHanlon | ... and move to somewhere and raise goats | 19:06 |
| clarkb | yup that is technically an option though likely to be the most effort of all of them due to our reliance on containers for our service deployments | 19:07 |
| fungi | goat farming has started to become tempting | 19:07 |
| clarkb | considering all that I believe A and B are the serious contenders here. | 19:07 |
| corvus | for clarity: if we did want to do (F), i think we could still run our own tiny registry for our own operational needs | 19:07 |
| corvus | i'm not pushing for (F) | 19:08 |
| clarkb | My personal inclination is to avoid running more services if we can. To this end I created a quay account and tested things there. I believe that quay, despite some oddities, would be a viable option for us | 19:08 |
| fungi | your progress reports on quay experience in #opendev seemed promising | 19:08 |
| corvus | yes, based on that progress, i like: A (quay); B, F in that order. | 19:09 |
| fungi | and while it doesn't get us away from "some vendor-run registry that could be pulled out from under us at any moment" our communities at least have fairly strong relationships with red hat, unlike docker inc | 19:09 |
| corvus | [but if someone has a herd of goats ready to go, that really does make F more tempting] | 19:09 |
| clarkb | I think the rough setup I would advocate there is that infra-root create personal accounts via https://quay.io (you click sign in then register an account). We would then create an org for opendevorg/ and another for zuul/ and add our personal accounts to those orgs | 19:09 |
| clarkb | One thing to note is that orgs appear to require an email address and that email address must be distinct | 19:10 |
| fungi | do + extensions count as distinct? | 19:10 |
| NeilHanlon | they do | 19:10 |
| clarkb | which means we'll probably want the opendevorg/ email addr to be something like infra-root+quayopendevorg@... | 19:10 |
| fungi | good enough for me | 19:10 |
| clarkb | and then I'll defer to corvus on what he feels is most appropriate for zuul/ but could do similar there | 19:11 |
| NeilHanlon | that is the setup, too, that we use for Rocky Linux, which seems to work well | 19:11 |
| corvus | that's a little awkward for other orgs, in that it requires some interaction with infra-root | 19:11 |
| clarkb | NeilHanlon: thats good input that others are able to do similar | 19:11 |
| clarkb | corvus: agreed | 19:11 |
| corvus | but that's probably scalable to the number of orgs we're contemplating for the next little while | 19:11 |
| clarkb | corvus: yup and it sounds like a lot of other ajdacent orgs/communities are already on quay and have solved this themsevles | 19:12 |
| corvus | (or we could ask projects like zuul to get their own secret shared mailbox, but that's a lot of work that's probably not necessary) | 19:12 |
| fungi | it can certainly be handled the same way as it is for dockerhub: let community leadership create accounts, set up orgs, encode secrets for zuul, done | 19:12 |
| corvus | fungi: right, just the email address thing is new | 19:12 |
| clarkb | Then once we have an org we could create an appliction/robot within that org for api access and push credentials. This is what our CI jobs would be configured with | 19:12 |
| fungi | yeah, they'd need e-mail addresses. most of them will probably create a gmail inbox (eww) and call it done, but whatever | 19:13 |
| corvus | clarkb: did the email get used for anything? confirmation dance? | 19:13 |
| clarkb | Another oddity is that if you docker image push to create a new image in quay it is by default private even if your account is private only. You can manually set this as public via the web UI or emilienM wrote a script and blog post on how to do it via the api | 19:13 |
| clarkb | #link https://my1.fr/blog/moving-container-images-from-docker-io-to-quay-io/ | 19:13 |
| clarkb | corvus: they didn't ask for confirmation now that I've double checked. When I set a docker cli password on the account to docker image push they sent me an email to alert me that this event occurred | 19:14 |
| clarkb | oh account sign up does require a phone number, email address, and physical address. I don't know how much validation of those fields they do. They have not called or texted my phone number | 19:15 |
| fungi | my assumption is that those and handed directly to your friendly red hat sales team for followup | 19:15 |
| clarkb | but if you sign up via https://quay.io there are fewer questions to answer than when you sign up via sso.redhat.com so I suggest signing up via quay | 19:16 |
| corvus | okay, so the org email address is basically something that's unused for now, but could be important later. i think letting our orgs use infra-root+ORG@.. is reasonable in that case, and better than asking people to make gmail accounts and losing the passwords | 19:16 |
| clarkb | corvus: wfm | 19:16 |
| fungi | yeah, i'm cool with it | 19:16 |
| clarkb | it is possible that the org creation process is different than the individual user process and they will validate it. I can't be sure of that yet since I did a slightly different process | 19:16 |
| clarkb | Another thing I noticed is that quay's security scanning is very aggressive to the point of not being useful imo. It calls out a bunch of issues with up to date debian packages | 19:17 |
| clarkb | But ya I would say that if others are comfortable creating personal accounts for quay/red hat we can start there then create orgs and then figure out what zuul job updates are required to migrate our images over | 19:18 |
| clarkb | everything I've seen indicates it should be possible to automate what we need in order to create new images on quay and push to them via zuul | 19:18 |
| corvus | sounds like we have 2 tasks there: 1) do the create public repo api call at the start of the promote job; and 2) update the docker-specific tag promote to something that works with quay | 19:19 |
| corvus | i can volunteer for #2 | 19:19 |
| fungi | i'm assuming their security scanning is really only accurate if the images are rhel/centos/fedora based | 19:19 |
| clarkb | corvus: and 3) create org in quay for opendevorg and 4) create org in quay for zuul | 19:20 |
| clarkb | I'm happy to do 3 once we have general agreement and more than one user account that has managed to sign up successfully | 19:20 |
| corvus | oh yeah that too, sorry i was just thinking of job related work | 19:20 |
| NeilHanlon | in my experience, it's on par with most other scanners... which is to say: mediocre on a good day | 19:20 |
| clarkb | corvus: I can do 4 too fwiw but thought you might prefer to do it | 19:20 |
| corvus | yeah i can do #4 | 19:21 |
| fungi | do we want to push in parallel to both dockerhub and quay for some period of time, or just make a hard cut-over? i guess for opendev's server images it can be a hard cut. other projects will likely migrate at their own pace | 19:21 |
| clarkb | I think 1) becomes easier to test once 3 and/or 4 are done since we'll be able to create an application that can talk to the API at that point | 19:21 |
| clarkb | I'm willing to drive 1 and can holler for help if I need it | 19:21 |
| corvus | my read on the zuul community is that we probably don't really need to get specific feedback and whatever we decide here will work for zuul. | 19:21 |
| corvus | (specifically, everyone i've talked to in the past about this either (a) would prefer quay to dockerhub or (b) doesn't care) | 19:22 |
| clarkb | fungi: I think I'd just hard cutover the publication location, then we update opendev's pull locations | 19:22 |
| fungi | presumably zuul will do an announcement at least, opendev's server deployments likely don't even need that | 19:22 |
| clarkb | its not like the docker hub images are going away immediately and if we need to update for security updates anyway we're doing a build and publish and pull anyway | 19:22 |
| corvus | fungi: re announcement, yep, at least one | 19:22 |
| clarkb | we also need to decide if we care about rebuilding imgaes multiple times or not. If we do care we should start with our base images and then their consumers | 19:23 |
| clarkb | but I think we can rebuild if those move later and it will be fine too | 19:23 |
| corvus | i'd like to float the idea of deleting our repos from dockerhub shortly before we lose access. that way there isn't an "official looking" repo serving content that we don't control. | 19:23 |
| clarkb | thats a reaosnable thing to do. I wonder if that would impact their promise to not allow name reuse. eg if you intentionally delete it vs them doing it | 19:24 |
| clarkb | but I do agree not having stale images that we know are bad is preferable to potentially having bad images later | 19:24 |
| fungi | we could always delete the orgs and then try to squat the same names with personal accounts | 19:25 |
| fungi | but yeah there could be some delay in them releasing the names, if they do | 19:25 |
| clarkb | it is probably sufficient to just delete our existing images within the orgs and let them do what they are going to do to the org | 19:25 |
| corvus | i think they're talking about the org.... i assume that would stay. we could always create a new repo so there's at least one repo in the org. | 19:25 |
| corvus | "dear-docker-please-do-not-release-this-namespace" | 19:26 |
| fungi | oh, got it, delete the repos but not necessarily the orgs | 19:26 |
| corvus | fungi: i'm not sure that's possible (deleting orgs) | 19:26 |
| NeilHanlon | someone in the github issue commented they removed an older org the had and the namespace wasn't cleaned up within 30 mins | 19:26 |
| NeilHanlon | corvus: throw in some zero width spaces or something to make it hard | 19:26 |
| corvus | lol | 19:26 |
| clarkb | I'm going to try and summarize where I think the plan is headed on the etherpad | 19:26 |
| NeilHanlon | just use chatgpt /s | 19:27 |
| clarkb | oh looks like corvus already is thanks! | 19:27 |
| corvus | oh i started a section at the bottom | 19:27 |
| NeilHanlon | (just wanted to make fungi's blood pressure rise ;) ) | 19:27 |
| fungi | they make medication for that these days | 19:27 |
| corvus | new task on that list: copy existing repos and all tags to quay.io -- i think that would be a good idea, yeah? | 19:27 |
| clarkb | corvus: It was something I wondered about. I think for opendev its far less important but mostly because we just have :latest on most images except gerrit | 19:28 |
| clarkb | But for zuul I think that makes a lot of sense | 19:28 |
| clarkb | and if we are doing it for zuul we may as well do it for opendev | 19:28 |
| corvus | true; that might be something i manually do for zuul then | 19:28 |
| clarkb | emilienm's blog/script will be useful for that | 19:29 |
| clarkb | since that is essentially what it was for | 19:29 |
| corvus | ah cool | 19:29 |
| fungi | is that linked in the pad? | 19:30 |
| clarkb | fungi: not yet I'll add it | 19:31 |
| fungi | thanks | 19:31 |
| clarkb | heh corvus wins again | 19:31 |
| fungi | i saw it in #opendev, just want to make sure we don't have to go digging it back out of our irc logs | 19:31 |
| clarkb | ok I think we've got a good foundation for zuul/ and opendevorg/ things. This doesn't address what we'll do with other things like zookeeper mariadb etc | 19:33 |
| clarkb | or graphite/grafana | 19:33 |
| ianw | i don't want to be annoying, but i just wonder if we are setting ourselves up here to once again be disappointed by basically doing the same thing over again. since we are have a rather large consortium behind us, perhaps we should consider using some of the resources available to the project to start this on a paid plan, and then we seem less likely to be back here having another meeting | 19:34 |
| fungi | which will depend on what decisions their communities make, i suppose? | 19:34 |
| clarkb | there are two classes of image there. THe first are the "docker official" images like python, zookeeper, mariadb, debian, ubuntu and it sounds like most people expect docker to continue to maintain those | 19:34 |
| clarkb | but graphite and jitsi meet we may need to watch for new image locations for those as well | 19:34 |
| clarkb | ianw: I think we can make a decision like that further down the road (there is a "upgrade" button to do that in account settings and I assume org settings) | 19:35 |
| fungi | ianw: assuming you mean pay for quay, we can talk to the foundation executive team and board of directors, but it may get weird with red hat also being a member of the organization | 19:35 |
| clarkb | ianw: my concern with trying to start that way is it will slow down the move while we sort out if that is ok or even appropriate and who to pay and so on. | 19:35 |
| clarkb | also once you sign up its really clear they don't expect open source orgs to pay | 19:36 |
| corvus | also before you sign up that's pretty clear | 19:36 |
| clarkb | in particular when you create a new org the free tier is public images only and labeled "open source" | 19:36 |
| clarkb | I don't think any part of this plan prevents us from doing that later | 19:36 |
| fungi | as far as paying for things, i wouldn't be surprised if red hat paying for membership in a non-profit which in turn buys services from red hat might have tax law implications, so will want to make sure we talk to the right people to nail down all the gotchas if we want to go that direction | 19:37 |
| ianw | nothing really stops us paying docker either, but i feel like the major concern with that is not really technical but more "they bait and switched us". which, ok, but if RH starts imposing rate limits etc. we do have to sort of say "well we walked into that" | 19:39 |
| corvus | putting on my free software hippie hat for a moment -- if the norm in open source communities becomes "pay to host container images" then i do think a serious consideration of (F) -- don't publish images just like we don't publish .rpm or .deb files is worth having. "docker build" is not actually hard for end users to run. | 19:39 |
| clarkb | ianw: yes we could pay docker hub as well and there is potential for this or something like it to occur with quay as well. I've heard an explicit interest in not paying docker from other infra rooters. And I think those opinions are valid | 19:40 |
| fungi | and we do already operate on a ton of donated services which could be pulled out from under us at any time, so free dockerhub or quay aren't all that different from those | 19:40 |
| clarkb | Where it gets weird paying quay is they explicitly sya they don't want money for what we are planning to do | 19:41 |
| ianw | i don't really want to argue this, i know everybody understands what we're getting into. quay seems fine and i'm happy to be on board with helping switching. just if it turns out that unlimited free repos isn't sustainable, i don't think we should be surprised, again | 19:41 |
| clarkb | ianw: agreed | 19:41 |
| clarkb | fool me once etc | 19:42 |
| corvus | ianw: i agree too. and i don't think we have any implied or explicit SLA to our users here either. moving stuff like this around quickly, with little notice, and even having interruptions in service should be expected, i think. | 19:42 |
| fungi | it's definitely worth calling out, and i wouldn't be surprised if quay is getting inundated with a flood of dockerhub evacuees too which might change their viewpoint on financial matters | 19:42 |
| clarkb | the short timeline involved makes it difficult for me to say we should do something like f which is almost certainly going to be more effort | 19:42 |
| corvus | yeah, f is a bit "change how people think" which, you know, we don't shy away from, but also tends to have an extended time line | 19:43 |
| ianw | ok, the fact that we even discussed this, which i don't feel like was ever discussed with dockerhub as it was pretty much assumed it would remain around forever, is probably enough | 19:43 |
| clarkb | ok cool. And please do bring up concerns if we run into anything unexpected and new | 19:44 |
| corvus | i may have joked about dockerhub pulling the plug while drunk... but yes, i don't recall a serious discussion :) | 19:44 |
| fungi | fwiw, i don't assume any gratis service will be around/available forever. we've seen enough examples ourselves that's not reasonable | 19:44 |
| fungi | transifex did the same to us years ago | 19:44 |
| corvus | yep, sounds like we're eyes wide open on this | 19:45 |
| NeilHanlon | fwiw, i did start conversations with some of Rocky's sponsors about the possibility of a community-owned/operated registry. of course: with all the problems which come along with that... but, I think it's not a bad idea in general to have something without a commercial interest... | 19:45 |
| clarkb | re other images. I'm thinking for now we should maybe keep our ears open for news on any changes to those images. And update quickly assuming they move. | 19:45 |
| clarkb | But I think it may be too early to say "we need to build our own ubuntu images from scratch" and I think it highly unlikely that docker just deletes those | 19:45 |
| clarkb | er I guess it would be debian since that is what we predominantly use for our container base | 19:46 |
| clarkb | #link https://github.com/jitsi/docker-jitsi-meet/issues/1502 jitsi meet move to github container registry will happen soon | 19:46 |
| clarkb | it wouldn't surprise me if grafana stays on docker hub | 19:47 |
| clarkb | seems like discussion is slowing down. Is there anything else we want to discuss around the docker hub org deletion? | 19:48 |
| clarkb | Maybe we've got enough sorted out to beging to make progress then resync on tuesday | 19:48 |
| clarkb | *to begin | 19:49 |
| corvus | sounds like we have consensus (maybe even complete agreement) and a task list? | 19:49 |
| fungi | i agree on the task list | 19:49 |
| corvus | i concur | 19:49 |
| fungi | #link https://wiki.debian.org/Cloud/CreateDockerImage if we do decide we need to build our own | 19:50 |
| clarkb | fungi: we can also fork what docker is/was doing too | 19:50 |
| clarkb | (I don't have an opinion on the best method but they are all open source and forkable aiui) | 19:50 |
| NeilHanlon | #link https://github.com/docker-library/official-images/ "Library"/"Official" image sources for many images that can be rebuilt | 19:51 |
| NeilHanlon | this is "just" references to other repos, so in theory they should be reproducible | 19:51 |
| clarkb | sounds like that is probably it for now. We can sync up Tuesday at our normal meeting time | 19:52 |
| fungi | i guess an #agreed is in order? | 19:52 |
| clarkb | sure. How about #agreed Move opendevorg and zuul container images from docker hub to quay.io. This will require modifications to zuul jobs and management of accounts/orgs in quay.io. We recognize there is a risk that we may end up in the same spot later down the road but this seems the least painful option for today. | 19:54 |
| clarkb | ianw: ^ want to make sure I'm representing what you've said accurately | 19:54 |
| fungi | wfm | 19:54 |
| NeilHanlon | sgtm | 19:54 |
| corvus | ++ | 19:55 |
| clarkb | #link https://www.docker.com/blog/we-apologize-we-did-a-terrible-job-announcing-the-end-of-docker-free-teams/ | 19:55 |
| clarkb | "We’d also like to clarify that public images will only be removed from Docker Hub if their maintainer decides to delete them." | 19:56 |
| clarkb | I thought we had waited long enough for clarification from them. However reading this I don't really think it changes my inclination for the plan | 19:56 |
| clarkb | maybe makes it slightly less urgent? but its still not super clear so I'm more happy to move to something that is clearer | 19:57 |
| NeilHanlon | obviously i'm external to opendev.. but.. IMO: too little, too late | 19:57 |
| corvus | i think the only thing that warrants a reconsideration is: Can I migrate to a Personal account? | 19:57 |
| corvus | You can migrate to a Free Team organization to a Personal account by opening a support ticket. No action will be taken against your account while your ticket is being processed. | 19:57 |
| corvus | that was a question we had that was unanswered until now. | 19:57 |
| corvus | but i'm still ready to proceed as agreed. | 19:58 |
| fungi | NeilHanlon: you don't seem all that external to me. you're participating in our discussions and helping support our testing efforts | 19:58 |
| clarkb | me too after skimming this. Idon't think it functionally changes anything for us | 19:58 |
| corvus | aside from that clarification, it's basically "here's what we said but with more words" | 19:58 |
| NeilHanlon | fungi: well, thank you for that :) I'm glad to be involved with an org like opendev, and try to model what we do at Rocky after y'alls model | 19:59 |
| fungi | "We have assigned more staff to promptly review all applications." i guess that's their "we're sorry we ignored all your applications previously" | 19:59 |
| clarkb | ianw: fwiw I'm happy if you nack that draft as well :) | 20:01 |
| clarkb | Maybe I'll leave it out of the official meeting notes just to avoid implication everyone ack'd it. But the log captures that we have general consensus if not complete consensus | 20:01 |
| clarkb | ya I'll leave it there. Thank you everyone for the input. | 20:02 |
| clarkb | #endmeeting | 20:02 |
| opendevmeet | Meeting ended Thu Mar 16 20:02:45 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-03-16-19.02.html | 20:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-03-16-19.02.txt | 20:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-03-16-19.02.log.html | 20:02 |
| fungi | thanks! | 20:03 |
| corvus | clarkb: thanks (and also for the quay research)! | 20:03 |
| NeilHanlon | thanks all | 20:05 |
| ianw | clarkb: very sorry called into another thing. that's fine. i don't think you even really need the last sentence, we discussed it, it's fine, we know what we're getting into | 20:20 |
| clarkb | ack | 20:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!