| *** ianychoi__ has joined #airshipit | 00:14 | |
| *** ianychoi_ has quit IRC | 00:17 | |
| *** sreejithp has quit IRC | 01:54 | |
| *** aedan9902 has quit IRC | 01:54 | |
| *** aedan9902 has joined #airshipit | 02:20 | |
| *** aedan9902 has quit IRC | 02:27 | |
| *** aedan9902 has joined #airshipit | 03:14 | |
| *** aedan9902 has quit IRC | 03:24 | |
| *** aedan1 has joined #airshipit | 03:28 | |
| *** aedan has quit IRC | 03:31 | |
| *** uzumaki has joined #airshipit | 04:08 | |
| *** lemko2 has joined #airshipit | 04:28 | |
| *** lemko has quit IRC | 04:29 | |
| *** lemko2 is now known as lemko | 04:29 | |
| *** aedan9902 has joined #airshipit | 05:20 | |
| *** aedan9902 has quit IRC | 05:33 | |
| *** lemko has quit IRC | 05:46 | |
| *** lemko7 has joined #airshipit | 05:46 | |
| *** aedan9902 has joined #airshipit | 05:48 | |
| *** aedan9902 has quit IRC | 05:53 | |
| *** aedan9902 has joined #airshipit | 06:06 | |
| *** uzumaki has quit IRC | 06:07 | |
| *** aedan9902 has quit IRC | 06:19 | |
| *** aedan1 has quit IRC | 06:47 | |
| *** uzumaki has joined #airshipit | 06:52 | |
| *** dasp has joined #airshipit | 07:00 | |
| *** dasp_ has quit IRC | 07:01 | |
| *** aedan9902 has joined #airshipit | 07:48 | |
| *** aedan9902_1 has joined #airshipit | 09:04 | |
| *** aedan9902 has quit IRC | 09:04 | |
| *** muhaha has joined #airshipit | 09:11 | |
| *** uzumaki has quit IRC | 11:39 | |
| *** muhaha has quit IRC | 11:51 | |
| *** uzumaki has joined #airshipit | 13:05 | |
| *** sreejithp has joined #airshipit | 14:40 | |
| *** uzumaki has quit IRC | 14:40 | |
| *** arijitbose_ has joined #airshipit | 14:45 | |
| *** arijitbose_ has quit IRC | 14:45 | |
| *** aedan9902_1 has quit IRC | 14:46 | |
| *** arijitbose has joined #airshipit | 14:47 | |
| *** SRao has joined #airshipit | 16:02 | |
| airship-irc-bot1 | <alexander.hughes> hitting docker pull rate limits again in airshipctl, are the images not being cached/are we not using the zuul cache? https://zuul.opendev.org/t/openstack/build/9f8cd3f32d0745cfa6993a308b7f7ad3 | 16:21 |
|---|---|---|
| clarkb | docker subverted our cache | 16:24 |
| clarkb | the way our caching works is that we cache the image blobs (the image layer data essentially). Before ~novemeber docker hub rate limited blob downloads which makes sense since they represent the bulk of the data transfer. We were good citizens and cached them and avoided rate limiting. When they changed their rate limits the major change they made was to stop rate limiting blobs and instead rate limit | 16:25 |
| clarkb | manifests | 16:25 |
| clarkb | our cache does not currently cache manifests because you have to be authenticated to fetch them and apache's cache layer refuses to cache things with authentication headers (even if the cache-control header says the data is public and cacheable) | 16:26 |
| clarkb | (and yes even if fetching anonymously you have to create an auth token for the anonymous fetch so from the http layer's perspective it is all authenticated even if that doesn't imply any real authorization) | 16:26 |
| clarkb | from opendev's own job perspective we seem to avoid being rate limited most of the time. We appear to struggle the most in clouds that NAT ipv4 (which makes sense as we'll funnel all ipv4 traffic through fewer addrs and docker hub does not do ipv6) | 16:28 |
| clarkb | if the problem presists or worsens we likely need to look at alternatives. One thought is that squid can likely be made to cache the manifests in addition to the blobs as it can likely be told to honor the cache-control settings for authenticated requests | 16:30 |
| clarkb | docker-registry itself is often mentioned, but the issue there is it doesn't seem to have the ability to do online pruning of its cache set. However, the docs are a bit conflicted on this subject and maybe testing will reveal it works properly | 16:33 |
| airship-irc-bot1 | <alexander.hughes> so this is a common issue, though seemingly infrequent, across all of opendev - and the potential alternatives are being explored across opendev not at project levels? | 16:36 |
| clarkb | well it is an issue for any users of docker hub due to their rate limits | 16:36 |
| clarkb | including moby's own CI aiui :) | 16:36 |
| clarkb | explored is probably a bit strong. We've identified some avenues that could be pursued but no one has volunteered to take them further at this point that I know of | 16:37 |
| *** SRao has quit IRC | 16:37 | |
| airship-irc-bot1 | <alexander.hughes> I see, is the brainstorming session and signup list public that I can watch for developments? | 16:38 |
| clarkb | it has come up in a couple of our weekly team meetings and been discussed there. We publsih an agenda here https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting and meet at 19:00UTC Tuesdays in #opendev-meeting on freenode | 16:39 |
| clarkb | (this issue isn't currently on the agenda but could be readded as necessary) | 16:40 |
| clarkb | if people would prefer a more asynchronous communication on the topic starting a thread at service-discuss@lists.opendev.org would be a reasonable appraoch | 16:40 |
| airship-irc-bot1 | <alexander.hughes> that's fair, thanks :slightly_smiling_face: | 16:46 |
| clarkb | largely at this point I think it needs someone interested in addressing it to figure out if squid or docker-registry or something else can be made to work, and then work with us to update the config management for our mirror nodes to deploy that alternative | 16:47 |
| airship-irc-bot1 | <rishabh.k.jain> Hi, Can I get you few more core reviews. Got one +2. Almost there :slightly_smiling_face: | 16:55 |
| *** uzumaki has joined #airshipit | 17:18 | |
| airship-irc-bot1 | <rishabh.k.jain> Hi, Can I get some more reviews here. Thank you :slightly_smiling_face: https://review.opendev.org/c/airship/airshipctl/+/772425 https://review.opendev.org/c/airship/airshipctl/+/771839 | 17:20 |
| airship-irc-bot1 | <ratnopam.chakrabarti> Is airship CI gate non-functional today? https://jenkins.nc.opensource.att.com/ seems not accessible | 20:19 |
| airship-irc-bot1 | <ratnopam.chakrabarti> It's back up now. Thanks Siraj. | 20:52 |
| *** uzumaki has quit IRC | 21:06 | |
| *** muhaha has joined #airshipit | 22:11 | |
| *** muhaha has quit IRC | 23:11 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!