| *** sboyron has quit IRC | 00:23 | |
| *** hamalq has quit IRC | 06:21 | |
| *** hamalq has joined #opendev-meeting | 06:22 | |
| *** zbr has quit IRC | 06:52 | |
| *** sboyron has joined #opendev-meeting | 06:52 | |
| *** zbr has joined #opendev-meeting | 06:53 | |
| *** hashar has joined #opendev-meeting | 08:10 | |
| *** zbr has quit IRC | 09:51 | |
| *** zbr has joined #opendev-meeting | 09:56 | |
| *** zbr has quit IRC | 10:58 | |
| *** zbr has joined #opendev-meeting | 10:59 | |
| *** zbr has quit IRC | 11:00 | |
| *** zbr has joined #opendev-meeting | 11:01 | |
| *** zbr has quit IRC | 11:03 | |
| *** zbr has joined #opendev-meeting | 11:05 | |
| *** zbr has quit IRC | 11:07 | |
| *** zbr has joined #opendev-meeting | 11:17 | |
| *** hashar is now known as hasharAway | 15:20 | |
| clarkb | anyoen else here for the meeting? | 19:00 |
|---|---|---|
| diablo_rojo | o/ | 19:00 |
| ianw | o/ | 19:00 |
| clarkb | cool we shall get started momentarily | 19:00 |
| clarkb | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Dec 1 19:01:13 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.opendev.org/pipermail/service-discuss/2020-November/000137.html Our Agenda | 19:01 |
| clarkb | We have an agenda, trying to get things back to normal after an eventful few weeks | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:01 | |
| clarkb | Wallaby cycle signing key has been activated https://review.opendev.org/760364 | 19:01 |
| clarkb | Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html | 19:01 |
| clarkb | at this point this is there mostly as a reminder for myself as I have failed to sign it sofar :( | 19:02 |
| clarkb | #topic Actions from last meeting | 19:02 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:02 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-11-24-19.01.txt minutes from last meeting | 19:02 |
| clarkb | Last meeting we didn't haev a formal agenda and instead went through gerrit upgrade related items. | 19:02 |
| clarkb | There are still a few of those to talk through which we will get to shortly | 19:03 |
| clarkb | #topic Priority Efforts | 19:03 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:03 | |
| clarkb | #topic OpenDev | 19:03 |
| *** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:03 | |
| clarkb | We've been working through the debugging of system load on Gerrit. We've had a few good leads so far but nothing that has made it go completely away | 19:03 |
| fungi | ohai | 19:03 |
| clarkb | In particular someone else on the Gerrit mailing list was struggling with similar on Gerrit 2.16 and the discussion there pointed to caches | 19:04 |
| corvus | o/ | 19:04 |
| clarkb | fungi and I have since been trying to tune our cache sized based on the info that ssh review gerrit show-caches gives us | 19:04 |
| clarkb | I think this has helped but it hasn't completely made things happy yet | 19:04 |
| fungi | worth noting, we think there is correlation between the "missing tree" errors people get on push and the elevated system load | 19:05 |
| clarkb | We also noticed that there is a jgit recieve.autogc setting that runs git gc when code is pushed. We set that but literally just 5 minutes ago I realized we set it in the wrong config file | 19:05 |
| clarkb | there is not a jgit config file so I imagine next ups is getting that moved into the correct config file | 19:05 |
| fungi | though it's so far only been observed on large repos, generally while or immediately after pushing change series | 19:05 |
| clarkb | whcih could be related to the autogc thing maybe? the gerrit docs note that disabling it is recommended (despite being enabled by default) due to the load impact it has | 19:05 |
| fungi | yeah, i'll work on adding the jgit.config after this meeting | 19:06 |
| clarkb | 3.3.0 release notes imply that it will not be disabled by default though so maybe they decided making things bad by default was not recommended | 19:06 |
| fungi | the release note on that is a little vague/confusing, to be perfectly honest | 19:06 |
| clarkb | In conversation with Luca on the Gerrit slack he says that Java 11 is likely also to have some performance benefits. Gerrithub has been running 3.2 on java 11 since the beginning of this release | 19:06 |
| clarkb | fungi: yup | 19:06 |
| clarkb | I think this means we should also look at landing java 11 support in our images then switch over prod to java 11. Fungi switched review-test over to java 11 this morning | 19:07 |
| fungi | yeah, if folks want to beat on review-test at all, that's helpful | 19:07 |
| clarkb | And I'm hoping that this afternoon I'll have time to update the image jobs to build a 3.3 as well | 19:07 |
| fungi | i feel like we should land the openjdk 11 patch before trying the upgrade to gerrit 3.3, fwiw | 19:08 |
| clarkb | I agree | 19:08 |
| fungi | that way if we see new issues we have a better idea of what brought them in | 19:08 |
| ianw | ++ | 19:08 |
| ianw | not to derail, but how important do we think upgrading the hsot from xenial is too? | 19:08 |
| fungi | also we need to do openjdk 11 before we upgrade to (not yet existent) gerrit 3.4 | 19:09 |
| fungi | since they're planning to drop support for <11 at that release | 19:09 |
| clarkb | ianw: I think that is reasonably important, but not urgent. eg we should be able to schedule that and warn people of the upcoming new IP address | 19:09 |
| clarkb | if someone wants to start looking at what that would require I would be grateful :) | 19:09 |
| corvus | my understanding is it should only be important for OS support reasons | 19:10 |
| corvus | not for java version/performance reasons | 19:10 |
| corvus | is that correct or do we think there's a perf benefit? | 19:10 |
| clarkb | corvus: generally linux benchmarking gets worse as you get newer kernels | 19:10 |
| clarkb | I would actually expect a performance impact (if I had to guess without testing) | 19:10 |
| fungi | yeah, i think the os upgrade would just be mre because xenial reaches eol in a few months | 19:11 |
| clarkb | phoronix does generic benchmarking of linux over time if people want to see what I would assume that | 19:11 |
| ianw | yep, and also if you're spending time debugging things and it does get down to the kernel/container-ish layer better to be debugging something current | 19:11 |
| clarkb | ianw: ya thats true | 19:12 |
| fungi | and on that note, sometime soon we should also talk out a plan for how we would actually do the upgrading to focal... options are to build a new vm and then we have new ip addresses to warn folks about (given how many we know are stuck behind corporate firewalls with special rules allowing 29418/tcp to our server's current address) or do in-place upgrades | 19:12 |
| clarkb | I think I still strongly prefer the new host method | 19:12 |
| ianw | i feel like last time we went with in-place | 19:12 |
| corvus | it sounds like it's a wildcard and could go either way, so i'd lean towards deferring os upgrade until we've stabilized or run out of other things | 19:12 |
| fungi | i do too, but in that case we need to decide on a communication schedule | 19:12 |
| clarkb | corvus: ++ | 19:12 |
| fungi | corvus: yes, i agree we should hold off the os upgrade until we have known performance for the container on the current os version | 19:13 |
| corvus | do we want to see about putting together an http-only recommendation for third-party ci before host replacement? | 19:13 |
| clarkb | The other thing I wanted to bring up is tristanC has done some plugin work to do zuul results table rendering. I've been too distracted by other things, but do others think that is in a place that we should consume it? I think if I had any concerns its that it is written in another esoteric alnguage that compiles to js/java aiui | 19:13 |
| clarkb | corvus: based on some of the responses I've gotten so far I think a lot of third party CIs would struggle with that | 19:14 |
| clarkb | a non zero number are still stuck on zuul v2 | 19:14 |
| corvus | they would have a choice about what kind of struggle | 19:14 |
| fungi | also how would http-only work? are we planning to add the checks plugin? | 19:15 |
| corvus | fight internal network rules or upgrade software to supported versions | 19:15 |
| clarkb | fungi: that is a good question | 19:15 |
| corvus | fungi: that's the question; i'm not sure checks has a long-term future, but it does exist and has no limitations for the third-party ci use-case (it does for a full gating system); an alternative may be webhooks. | 19:15 |
| fungi | right now we're not offering them an alternative for the stream-events cli | 19:15 |
| clarkb | corvus: is webhooks another plugin option? | 19:16 |
| corvus | yep | 19:16 |
| fungi | so while i think http-only sounds great, we'd probably need to decide what that looks like and get it available first | 19:16 |
| corvus | afaik, its supporters do have a long-term interest | 19:16 |
| clarkb | fungi: ya sounds like something to do more investigating for | 19:16 |
| ianw | there's also now the "findings" tab? if i've understood, you're supposed to put "autogenerated" on your review comment to be in there? | 19:17 |
| corvus | fungi: agreed (is why i raised it -- do we want to look into setting that as a goal?) | 19:17 |
| clarkb | ianw: I think zuul is doing that? | 19:17 |
| corvus | yes has been for some time | 19:17 |
| corvus | i believe findings are different (at least, last time i was exposed to the design doc) | 19:17 |
| fungi | ianw: robot comments are toggleable, zuul has done that by default for ~ a year | 19:18 |
| fungi | and yes, robot comments and findings are separate things | 19:18 |
| ianw | i haven't yet managed to find the documentation on how to get anything into "findings" | 19:18 |
| corvus | ianw: are you suggesting findings tab as alternative to results table rendering? | 19:18 |
| fungi | the checks plugin puts thnigs in findings | 19:18 |
| ianw | corvus: not really as i don't understand it, but i mean it does seem like a summary of the latest zuul results is a "finding" | 19:20 |
| corvus | clarkb: i haven't seen tristanC's table; is there a ml message or other link or something? | 19:20 |
| corvus | ianw: have a link to an example? | 19:20 |
| fungi | ianw: what "robot comments" (autogenerated) do is hide things when you switch the "only comments" slider in the "change log" section of the change view | 19:20 |
| ianw | corvus: yes, let me did, it was rolled out on a test instance | 19:21 |
| ianw | dig | 19:21 |
| clarkb | corvus: https://review.opendev.org/c/opendev/system-config/+/763891 is the change | 19:21 |
| clarkb | and ya the job that test gerrit installation on ^ was held aiui for people to test it | 19:21 |
| corvus | fungi: (at some point i understood "robot comments" to be a new type of comment associated with checks plugin vs the "regular old comments" which may or may not have the 'autogenerated' tag | 19:22 |
| ianw | https://104.130.172.52/c/openstack/diskimage-builder/+/554002 | 19:22 |
| ianw | are we onto talking about the table? because i'd like to run some things about gerrit gate testing by the peanut gallery | 19:23 |
| fungi | corvus: oh, interesting, it's possible i've confused them but i kept seeing them mentioned as the same thing | 19:23 |
| corvus | clarkb, tristanC: there are 2 zuul plugins for gerrit | 19:23 |
| corvus | clarkb, tristanC: is there any way maybe we could contribute to one or more of those? | 19:23 |
| clarkb | corvus: yes I strongly encouraged tristanC to do so, but was told there is no interest in learning java or js | 19:24 |
| corvus | i believe tristanC knows js | 19:24 |
| corvus | unless tristanC forgot js? | 19:24 |
| clarkb | which is one of my concerns with using the sf thing, its in a random language that tristanC finds acceptable rather than the upstream tooling | 19:24 |
| clarkb | corvus: I dunno that is just what I was told last week when it came up | 19:24 |
| clarkb | I believe this particular plugin is written in some language that compiles to js | 19:25 |
| ianw | yeah, but "javascript" these days is similar to assembly language really | 19:25 |
| fungi | the main thing i've wondered about scope-wise is whether a pg plugin for displaying a summary table of arbitrary third-party ci comments/votes is relevant to the zuul plug-in, but maybe if zuul is the reference for the comment format then it could be | 19:25 |
| corvus | ianw: i think that's a bit of a stretch | 19:25 |
| corvus | fungi: displaying zuul results is absolutely relevant | 19:25 |
| fungi | yep, and so if other ci systems leave comments which look like zuul results, then supporting that as part of the zuul plugin seems sane enough | 19:26 |
| corvus | https://gerrit.googlesource.com/plugins/zuul-status/ | 19:26 |
| ianw | corvus: maybe, but i mean https://104.130.172.52/plugins/zuul-results/static/zuul-results.js | 19:26 |
| corvus | Displays zuul status on PolyGerrit change | 19:26 |
| clarkb | corvus: http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-11-23.log.html#t2020-11-23T15:10:55 | 19:26 |
| corvus | ianw: i'm not sure what the point you're making is | 19:27 |
| corvus | ianw: that is clearly a minimized and obfuscated file; i don't deny the existence of such things | 19:27 |
| corvus | i only say that plenty of people write javascript as the input to creating such files | 19:27 |
| corvus | the fact that there are minimized js files doesn't mean we need to learn new languages; the upstream polygerrit plugins are written in something resembling js, right? so collaboration with others could be done that way, and since we've managed to teach some zuul devs how to do some basic js, they may be able to contribute too | 19:28 |
| clarkb | corvus: yup agreed. Maybe the best thing here is to hold out and see if we can upstream support for this into an existing plugin first | 19:29 |
| ianw | right, anyway i guess the exact point at hand is this is this concrete proposal for adding this table is written in https://reasonml.github.io/ and we probably have to decide if we want to incorporate that | 19:29 |
| corvus | clarkb: based on that convo, it seems like we're saying "someone needs to learn polygerrit" vs "someone needs to learn reasonml" | 19:29 |
| ianw | in terms of the bigger picture, of testing plugins, i think we should do some work there too. fungi suggested on-list that we should hold a node to test the plugins, which sort of works | 19:30 |
| clarkb | right which I still think would be better if that is the toolchain gerrit has attached to | 19:30 |
| corvus | ianw: i've already -2d one change to add reasonml to zuul based on the lack of support for our last experiment with an esoteric language | 19:30 |
| clarkb | because then we're collaborating in that ecosystem rather tah nsetting off on our own and being different | 19:30 |
| ianw | however, getting reviews into that held gerrit that look useful enough to test the plugin is a bit of a pain | 19:30 |
| fungi | ianw: yeah, what i didn't consider at the time was that we also need to get some representative content into the held gerrit somehow | 19:31 |
| clarkb | ianw: fungi could we autogenerate some content? | 19:31 |
| fungi | we could instead demo things on review-test for now, i suppose, and hold off deleting it | 19:31 |
| corvus | i mean, i like playing with esoteric functional languages, don't get me wrong, but as a group we don't have the best track record there, whereas i think there's a bigger chance we can get more long-term collaboration/support by sticking with how upstream does plugins | 19:31 |
| clarkb | make a project, push some changes, merge a change or two, etc | 19:31 |
| corvus | clarkb: ++ 'collaborating in that ecosystem' | 19:31 |
| ianw | clarkb: yes, i think so ... but we need to figure out adding the first admin user automatically | 19:31 |
| clarkb | ianw: the zuul all in one stuff does that, I bet we can reuse it | 19:32 |
| fungi | ianw: i have that figured out | 19:32 |
| corvus | you just need to leave a comment to test this, right? | 19:32 |
| clarkb | corvus: ya a zuul formatted comment I think (maybe the username matters too? I'm not sure) | 19:32 |
| fungi | there are probably multiple ways to create an initial admin account, but one is to use the gerrit cli with the built-in "gerrit code review" user | 19:32 |
| ianw | fungi: ok, i think we should go through together out of meeting maybe, and see if we can get the test job doing it | 19:32 |
| corvus | for hideci, yes; but hopefully we can omit that in the future -- comment tags are a thing :) | 19:32 |
| fungi | i think the zuul quickstart just uses become auth right? | 19:33 |
| fungi | been a while since i looked at that bit | 19:33 |
| ianw | at that point, it seems like it would also be easy to use a headless browser to take a screenshot of a review, which would make it easy to have an artifact confirming plugins working | 19:33 |
| ianw | and we can also hold the node for manual fiddling | 19:33 |
| fungi | that also sounds really awesome | 19:33 |
| ianw | there's some flag, DEVELOPMENT_BECOME_ANY_ACCOUNT which i didn't fully get to understanding last week | 19:34 |
| fungi | ianw: the alternative is the mechanism i describe in the gerrit admins section of our system-config docs. that works even on a gerrit with no existing accounts | 19:35 |
| corvus | also, ftr, i suspect it's perfectly fine to make a new plugin if this doesn't fit with zuul-status; i don't get the impression that lots of small plugins are necessarily bad. | 19:35 |
| clarkb | corvus: that is a good point. It seems the more important bit is using the toolchains upstream is using then they may get involved and help us | 19:35 |
| ianw | fungi: ok, that was what i was trying but wasn't getting an admin account. i think we should try again | 19:35 |
| clarkb | I think the gerrit maintainers do actually do a reasonable amount of plugin work to keep them working as things change ing errit | 19:35 |
| clarkb | supporting that work would be a good idea imo | 19:36 |
| ianw | i've already engaged on the thread; i can write a summary to respond if we like | 19:36 |
| clarkb | that sounds like a good way to recap this discussion for those who may not be hear | 19:36 |
| fungi | that reminds me, paladox contributed an opendev theme override (with light and dark mode support) as what i think is a pg plugin, but it's just an sgml/html blob in a paste. i was going to try to learn how to integrate that | 19:36 |
| clarkb | s/hear/here/ | 19:36 |
| ianw | it sounds like basically a) we're not currently convinced on the separate project, especially in a language that doesn't have a lot of exposure, and would like to investigate integrating with upstream more | 19:37 |
| ianw | and b) we'd like to expand the overall plugin testing environment to make it easier | 19:37 |
| clarkb | ianw: ++ | 19:37 |
| ianw | i'll draft something and loop people back | 19:37 |
| clarkb | thank you | 19:37 |
| fungi | also if that discussion thread wasn't on service-discuss, could it be redirected there? | 19:38 |
| fungi | i have a feeling it might have ended up on openstack-discuss | 19:38 |
| ianw | i can cc, i think it was openstack discuss only | 19:38 |
| corvus | yeah, fwiw i have no idea what thread is being discussed :( | 19:38 |
| corvus | also, friendly reminder that there is a zuul running for the purposes of testing plugins in the upstream gerrit; i have no idea what testing means for polygerrit plugins; that may be interesting to learn | 19:39 |
| ianw | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051.html | 19:39 |
| fungi | i do recall replying on it, but in retrospect i should have asked people to follow up to service-discuss | 19:39 |
| ianw | for reference | 19:39 |
| corvus | (it's mostly testing java plugins) | 19:39 |
| fungi | thanks ianw | 19:39 |
| clarkb | alright anything else on Gerrit before we move on? | 19:40 |
| fungi | part of the problem is i subscribe to lots of mailing lists and dump them into the same folder, so sometimes it's not immediately apparent to me if people have started discussions in the wrong ml | 19:40 |
| fungi | maybe we should agree to move forward with the jdk update asap? | 19:41 |
| fungi | other than that, no i think we've got things pretty well covered | 19:41 |
| clarkb | I'm on board, its being tested on review-test. If others can give that a quick check then we're probably good to proceed on that | 19:41 |
| clarkb | thinking out loud here: do the jgit autogc config first maybe? then do java 11 next? | 19:41 |
| clarkb | just to do one thing at a time and autogc fix seems simpler | 19:41 |
| fungi | yeah, i'll push that change up after the meeting | 19:42 |
| clarkb | thanks | 19:42 |
| clarkb | #topic Update Config Management | 19:42 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:42 | |
| clarkb | Is there anything new on this effort to call out? I don't think so but I'm double checking | 19:43 |
| fungi | the codesearch rebuild maybe? | 19:44 |
| clarkb | oh ya ianw ^ is that complete at this point? | 19:44 |
| fungi | we have two servers at the moment still, right? | 19:44 |
| fungi | oh, actually it's a cname now | 19:44 |
| ianw | no i cleaned the old one up, that should be all finished now | 19:44 |
| fungi | awesome, thanks! | 19:44 |
| ianw | nobody has complained so i assume it's working perfectly :) | 19:44 |
| clarkb | excellent | 19:45 |
| fungi | yes, i was making a point to use the opendev one so i would test it | 19:45 |
| fungi | and have had no problems | 19:45 |
| clarkb | #topic General topics | 19:45 |
| *** openstack changes topic to "General topics (Meeting topic: infra)" | 19:45 | |
| clarkb | #topic Bup and Borg Backups | 19:45 |
| corvus | ianw: ++ thanks! | 19:45 |
| *** openstack changes topic to "Bup and Borg Backups (Meeting topic: infra)" | 19:45 | |
| clarkb | I think we're getting more and more comfortable with borg? I've unfortunately had little time to interact with it mroe recently | 19:46 |
| clarkb | ianw: I know at some point you wanted to do verification then drop bup? | 19:46 |
| fungi | i should practice with restoring something i guess | 19:46 |
| clarkb | maybe a good thing to try and do before dropping bup is having other admins do things like ^ | 19:46 |
| ianw | yeah, i was thinking what i'll do is a config change to remove the bup cron jobs; people can audit the borg changes and approve that when happy | 19:46 |
| clarkb | ianw: that sounds like a reasonable plan | 19:47 |
| fungi | i'm down | 19:47 |
| clarkb | and that is important for the focal upgrades we were talking about earlier too | 19:47 |
| ianw | then we can kill all the puppet bits and maybe just attach the old backup volumes to the new server for a bit | 19:47 |
| clarkb | since bup and pytho3n don't mix | 19:47 |
| clarkb | thank you for getting this moving and doing all that work, really appreciated | 19:48 |
| clarkb | #topic Docker Rate Limits are Being Seen in CI | 19:48 |
| *** openstack changes topic to "Docker Rate Limits are Being Seen in CI (Meeting topic: infra)" | 19:48 | |
| clarkb | This is mostly a heads up/fyi | 19:49 |
| ianw | mostly in NAT environments? | 19:49 |
| clarkb | jobs particularly those running on limestone seem to hit this | 19:49 |
| clarkb | ianw: ya, though I would've expected it to hit all environments fairly equally due to our use of mirrors? But maybe we aren't using the mirrors the way I thought we were | 19:49 |
| fungi | yeah, we're not seeing it so much on our proxies as on limestone nat for jobs not using the proxy | 19:49 |
| clarkb | a few weeks back I pushed up changes to switch our zuul mirror config for docker over to just using the host addrs rather than the mirror. I don't think we need to land those yet since its NAT getting us | 19:50 |
| clarkb | but something to be aware of and maybe we need to bring that conversation for getting our images open source specialled again | 19:50 |
| clarkb | jbryce was going to look at the agreement in more detail and get back to us but I think like us has been busy | 19:50 |
| clarkb | another option is to use quay which does not rate limit | 19:51 |
| clarkb | but does have outages when aws east goes down | 19:51 |
| clarkb | I don't have answers, just info for people to digest :) | 19:51 |
| corvus | or make a new kind of pass-through proxy/mirror | 19:51 |
| clarkb | ya one that understands it needs to be a sort of lru cache | 19:52 |
| corvus | yup; i believe that's doable and much of the code in zuul-registry can be repurposed for that | 19:52 |
| corvus | (but still, it's not a trivial project, so one that we should deliberately choose) | 19:52 |
| fungi | also possibly a more useful effort than trying to bend something like squid to cache "authenticated" requests | 19:52 |
| ianw | so that would authenticate with a higher-limited key, and transparently pass through all our requests? | 19:53 |
| clarkb | though possibly squid would be better for our http caching we do on those hosts in general | 19:53 |
| clarkb | since in theory it can be more flexible than what apache is currently doing | 19:53 |
| corvus | ianw: or even anonymously but just stay under the limit? | 19:53 |
| clarkb | ya docker hub sends the required cache control headers to cache publicly those manifests | 19:53 |
| clarkb | the issue is that apache will not cache any authenticated request even with those headers | 19:53 |
| fungi | there's no "anonymously" really through right? | 19:54 |
| clarkb | we believe squid can be convinced to do so though | 19:54 |
| corvus | clarkb: do you think the squid approach will work with all the weird auth stuff? | 19:54 |
| clarkb | corvus: I think so if we can make it respect the cache-control: public or whatever header it is that is sent back by docker hub | 19:54 |
| corvus | fungi: in the way i intended to use it, yes (an auth credential obtained with no identifying information) | 19:54 |
| ianw | what we have not traditionally done is limit our mirrors to only be connectable from their respective clouds; we might want to think about that if we're using a opendev specific key | 19:54 |
| corvus | fungi: (authz without authn i guess?) | 19:54 |
| fungi | it's been years since i've done esoteric things with squid (including trivially patching it to ignore some things which would cause it not to cache but that it lacked configuration for), so it would need a poc regardless | 19:55 |
| clarkb | I think the major issue with apache as we use it for this problem space is that it will never cache a request that had an authorization header even if cache control says it is ok to do so | 19:55 |
| clarkb | if apache could be convinced to do ^ it would probably be fine too. Since it is now the manifest data that we need to cache | 19:55 |
| corvus | based on my estimation of effort, it sounds like spending a couple of days attempting to get squid to work should take precedence over a couple of weeks to implement a smart registry proxy | 19:56 |
| corvus | (or, you know, convince everyone to use quay.io :) | 19:56 |
| fungi | yeah, like i said, there have been times when i had to patch and recompile squid to get it to cache some stuff too, so i don't want to say it's necessarily better than apache mod_proxy, and i don't personally think being stuck maintaining our own patched build of either of those is particularly wise | 19:57 |
| fungi | the first thing it really needs is exploration | 19:58 |
| clarkb | ++ | 19:58 |
| clarkb | part of the issue in the past is the info from docker was a bit vague | 19:58 |
| clarkb | but now we've got a bit more real world data and we should be able to work with that to find a reasonable solution | 19:58 |
| clarkb | alright we are just about at time so I'll call it here | 19:59 |
| clarkb | thanks everyone | 19:59 |
| fungi | the other part of the issue was that it was an advance warning about stuff they weren't actually doing yet, yeah | 19:59 |
| fungi | now it's observable and testable at least | 19:59 |
| clarkb | feel free to continue any/all of these conversations in #opendev | 19:59 |
| clarkb | #endmeeting | 19:59 |
| *** openstack changes topic to "Incident management and meetings for the OpenDev sysadmins; normal discussions are in #opendev" | 19:59 | |
| openstack | Meeting ended Tue Dec 1 19:59:39 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-12-01-19.01.html | 19:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-12-01-19.01.txt | 19:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-12-01-19.01.log.html | 19:59 |
| fungi | thanks clarkb! | 19:59 |
| *** hasharAway is now known as hashar | 20:14 | |
| *** hashar has quit IRC | 22:16 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!