| *** tetsuro has joined #openstack-infra | 00:02 | |
| *** ryohayakawa has joined #openstack-infra | 00:06 | |
| *** rfolco has quit IRC | 00:22 | |
| *** armax has joined #openstack-infra | 00:33 | |
| *** yamamoto has joined #openstack-infra | 00:39 | |
| *** rcernin has quit IRC | 00:42 | |
| *** rcernin has joined #openstack-infra | 00:46 | |
| *** dtroyer has quit IRC | 01:03 | |
| *** rlandy has quit IRC | 01:28 | |
| *** tetsuro has quit IRC | 01:32 | |
| *** ricolin has joined #openstack-infra | 01:40 | |
| *** tetsuro has joined #openstack-infra | 01:41 | |
| *** tetsuro has quit IRC | 01:43 | |
| *** tetsuro has joined #openstack-infra | 01:44 | |
| *** rcernin has quit IRC | 02:13 | |
| *** apetrich has quit IRC | 02:14 | |
| *** antonym has quit IRC | 02:15 | |
| *** rcernin has joined #openstack-infra | 02:15 | |
| *** antonym has joined #openstack-infra | 02:31 | |
| *** rcernin has quit IRC | 02:33 | |
| *** dave-mccowan has quit IRC | 02:41 | |
| *** rcernin has joined #openstack-infra | 02:55 | |
| *** dtroyer has joined #openstack-infra | 02:58 | |
| *** tetsuro has quit IRC | 03:11 | |
| *** ociuhandu has joined #openstack-infra | 03:16 | |
| *** ociuhandu has quit IRC | 03:20 | |
| *** armax has quit IRC | 03:34 | |
| *** markvoelker has joined #openstack-infra | 04:03 | |
| *** markvoelker has quit IRC | 04:07 | |
| *** ykarel|away is now known as ykarel | 04:14 | |
| *** udesale has joined #openstack-infra | 04:28 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: write-inventory: add per-host variables https://review.opendev.org/739892 | 05:03 |
|---|---|---|
| *** raukadah is now known as chandankumar | 05:09 | |
| *** marios has joined #openstack-infra | 05:17 | |
| *** ryo_hayakawa has joined #openstack-infra | 05:19 | |
| *** ryo_hayakawa has quit IRC | 05:19 | |
| *** ryo_hayakawa has joined #openstack-infra | 05:19 | |
| *** tobiash has joined #openstack-infra | 05:20 | |
| *** ryohayakawa has quit IRC | 05:21 | |
| *** lmiccini has joined #openstack-infra | 05:24 | |
| *** abhishekk|away is now known as abhishekk | 05:33 | |
| *** dklyle has quit IRC | 06:03 | |
| *** eolivare has joined #openstack-infra | 06:11 | |
| *** xek has joined #openstack-infra | 06:25 | |
| *** xek_ has joined #openstack-infra | 06:27 | |
| *** xek has quit IRC | 06:30 | |
| *** ianw is now known as ianw_pto | 06:38 | |
| *** trident has quit IRC | 06:39 | |
| *** ccamacho has joined #openstack-infra | 06:42 | |
| *** trident has joined #openstack-infra | 06:42 | |
| *** markvoelker has joined #openstack-infra | 06:43 | |
| *** markvoelker has quit IRC | 06:48 | |
| *** rpittau has joined #openstack-infra | 06:50 | |
| *** rcernin has quit IRC | 06:59 | |
| *** rcernin has joined #openstack-infra | 07:06 | |
| *** jcapitao has joined #openstack-infra | 07:15 | |
| *** lbragstad_ has joined #openstack-infra | 07:19 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: write-inventory: add per-host variables https://review.opendev.org/739892 | 07:21 |
| *** lbragstad has quit IRC | 07:21 | |
| *** rpittau has quit IRC | 07:23 | |
| *** rpittau has joined #openstack-infra | 07:25 | |
| *** gfidente has joined #openstack-infra | 07:29 | |
| *** xek_ has quit IRC | 07:30 | |
| *** rpittau has quit IRC | 07:31 | |
| *** tosky has joined #openstack-infra | 07:33 | |
| *** zbr has joined #openstack-infra | 07:35 | |
| *** hashar has joined #openstack-infra | 07:39 | |
| *** rcernin has quit IRC | 07:41 | |
| *** rpittau has joined #openstack-infra | 07:42 | |
| *** ralonsoh has joined #openstack-infra | 07:47 | |
| openstackgerrit | Merged zuul/zuul-jobs master: fetch-coverage-output: direct link to coverage data https://review.opendev.org/739099 | 07:59 |
| *** dtantsur|afk is now known as dtantsur | 08:10 | |
| *** ociuhandu has joined #openstack-infra | 08:10 | |
| *** Lucas_Gray has joined #openstack-infra | 08:11 | |
| *** AJaeger has quit IRC | 08:13 | |
| *** jaicaa_ has quit IRC | 08:14 | |
| *** Wryhder has joined #openstack-infra | 08:16 | |
| *** Lucas_Gray has quit IRC | 08:16 | |
| *** Wryhder is now known as Lucas_Gray | 08:17 | |
| *** derekh has joined #openstack-infra | 08:19 | |
| *** tkajinam has quit IRC | 08:23 | |
| *** diablo_rojo has quit IRC | 08:41 | |
| *** zbr7 has joined #openstack-infra | 08:41 | |
| *** pkopec has joined #openstack-infra | 08:54 | |
| *** jaicaa has joined #openstack-infra | 09:00 | |
| *** AJaeger has joined #openstack-infra | 09:00 | |
| *** xek_ has joined #openstack-infra | 09:07 | |
| *** rcernin has joined #openstack-infra | 09:26 | |
| *** ryo_hayakawa has quit IRC | 09:33 | |
| *** yamamoto has quit IRC | 09:39 | |
| *** rcernin has quit IRC | 09:41 | |
| *** yamamoto has joined #openstack-infra | 09:42 | |
| *** xek_ has quit IRC | 09:44 | |
| *** derekh has quit IRC | 09:57 | |
| *** derekh has joined #openstack-infra | 09:58 | |
| *** yamamoto has quit IRC | 10:00 | |
| *** yamamoto has joined #openstack-infra | 10:02 | |
| *** yamamoto has quit IRC | 10:18 | |
| *** rfolco has joined #openstack-infra | 10:38 | |
| *** bhagyashris is now known as bhagyashris|brb | 10:42 | |
| *** apetrich has joined #openstack-infra | 10:44 | |
| *** xarses has joined #openstack-infra | 10:45 | |
| *** xarses_ has quit IRC | 10:47 | |
| dulek | Hi! Just wanted to report that codesearch.openstack.org fails with "too many open files". | 10:48 |
| *** armax has joined #openstack-infra | 10:48 | |
| dulek | Whoops, it started to work again. | 10:49 |
| *** yamamoto has joined #openstack-infra | 10:51 | |
| *** yamamoto has quit IRC | 10:56 | |
| *** bhagyashris|brb is now known as bhagyashris | 10:59 | |
| *** jcapitao is now known as jcapitao_lunch | 11:00 | |
| *** udesale_ has joined #openstack-infra | 11:03 | |
| *** udesale has quit IRC | 11:06 | |
| *** ysandeep is now known as ysandeep|brb | 11:11 | |
| *** ysandeep|brb is now known as ysandeep | 11:25 | |
| *** markvoelker has joined #openstack-infra | 11:34 | |
| *** markvoelker has quit IRC | 11:39 | |
| *** rlandy has joined #openstack-infra | 11:41 | |
| *** xek_ has joined #openstack-infra | 11:44 | |
| *** hashar is now known as hasharNap | 12:02 | |
| *** xek_ has quit IRC | 12:04 | |
| *** rcernin has joined #openstack-infra | 12:08 | |
| *** jcapitao_lunch is now known as jcapitao | 12:14 | |
| *** rcernin has quit IRC | 12:16 | |
| *** tinwood is now known as tinwood-afk | 12:17 | |
| *** ysandeep is now known as ysandeep|mtg | 12:29 | |
| *** ramishra has quit IRC | 12:31 | |
| *** smarcet has joined #openstack-infra | 12:51 | |
| *** smarcet has quit IRC | 12:51 | |
| *** smarcet has joined #openstack-infra | 12:53 | |
| *** smarcet has quit IRC | 12:56 | |
| *** hasharNap is now known as hashar | 12:57 | |
| *** rpittau has quit IRC | 12:57 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: mirror-workspace-git-repos: enable pushing to windows host https://review.opendev.org/739977 | 12:58 |
| *** rpittau has joined #openstack-infra | 13:00 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: mirror-workspace-git-repos: enable pushing to windows host https://review.opendev.org/739977 | 13:01 |
| *** ramishra has joined #openstack-infra | 13:02 | |
| *** xek_ has joined #openstack-infra | 13:04 | |
| *** adriant has quit IRC | 13:05 | |
| *** adriant has joined #openstack-infra | 13:05 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: mirror-workspace-git-repos: enable pushing to windows host https://review.opendev.org/739977 | 13:07 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: mirror-workspace-git-repos: enable pushing to windows host https://review.opendev.org/739977 | 13:09 |
| *** kota_ has quit IRC | 13:11 | |
| *** kota_ has joined #openstack-infra | 13:11 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: mirror-workspace-git-repos: enable pushing to windows host https://review.opendev.org/739977 | 13:14 |
| *** lpetrut_ has joined #openstack-infra | 13:21 | |
| *** ccamacho has quit IRC | 13:23 | |
| *** ysandeep|mtg is now known as ysandeep | 13:30 | |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Define maintain-github-openstack-mirror job https://review.opendev.org/738228 | 13:40 |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Define maintain-github-openstack-mirror job https://review.opendev.org/738228 | 13:42 |
| *** tinwood-afk is now known as tinwood | 13:45 | |
| *** Goneri has joined #openstack-infra | 13:48 | |
| *** ysandeep is now known as ysandeep|afk | 13:52 | |
| *** rpittau_ has joined #openstack-infra | 13:56 | |
| *** rpittau has quit IRC | 13:58 | |
| *** ysandeep|afk is now known as ysandeep | 14:00 | |
| *** bhagyashris is now known as bhagyashris|afk | 14:07 | |
| *** dklyle has joined #openstack-infra | 14:11 | |
| *** bhagyashris|afk is now known as bhagyashris | 14:17 | |
| *** rlandy is now known as rlandy|ruck | 14:19 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git-repos: enable pushing to windows host https://review.opendev.org/739977 | 14:22 |
| *** dave-mccowan has joined #openstack-infra | 14:25 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git-repos: enable pushing to windows host https://review.opendev.org/739977 | 14:25 |
| *** lpetrut_ has quit IRC | 14:28 | |
| *** dave-mccowan has quit IRC | 14:31 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git-repos: enable pushing to windows host https://review.opendev.org/739977 | 14:34 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: DNM: Add unified synchronize-repos role that works with linux and windows https://review.opendev.org/740005 | 14:39 |
| *** dmsimard3 has joined #openstack-infra | 14:41 | |
| *** dmsimard has quit IRC | 14:41 | |
| *** dmsimard3 is now known as dmsimard | 14:41 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: DNM: test ensure-podman https://review.opendev.org/740006 | 14:41 |
| fungi | dulek: i don't recall for sure, but i think the "too many open files" situation occurs when hound is reindexing/restarting | 14:46 |
| fungi | so it tends to clear up on its own | 14:46 |
| fungi | hopefully once the elasticsearch backend work for gitea progresses further we'll be able to finally retire the old codesearch service | 14:47 |
| dulek | fungi: Interesting, thanks! | 14:47 |
| *** priteau has joined #openstack-infra | 14:48 | |
| dulek | I actually like that Hound thing, it's incredibly useful when going through zuul yamls that are scattered through the repos. | 14:49 |
| *** ysandeep is now known as ysandeep|away | 14:55 | |
| *** ykarel is now known as ykarel|away | 15:04 | |
| *** priteau has quit IRC | 15:04 | |
| *** Xuchu has quit IRC | 15:05 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: DNM: test ensure-podman https://review.opendev.org/740006 | 15:11 |
| *** ccamacho has joined #openstack-infra | 15:20 | |
| *** ccamacho has quit IRC | 15:20 | |
| *** ccamacho has joined #openstack-infra | 15:20 | |
| *** __ministry1 has joined #openstack-infra | 15:22 | |
| *** lmiccini has quit IRC | 15:25 | |
| *** armax has quit IRC | 15:25 | |
| *** hamalq has joined #openstack-infra | 15:44 | |
| *** hamalq_ has joined #openstack-infra | 15:45 | |
| fungi | dulek: yep, ideally the search feature in gitea will be similarly useful (once it works) | 15:47 |
| *** hamalq has quit IRC | 15:49 | |
| *** gyee has joined #openstack-infra | 15:52 | |
| *** ociuhandu_ has joined #openstack-infra | 15:55 | |
| *** ociuhandu has quit IRC | 15:58 | |
| *** armax has joined #openstack-infra | 15:58 | |
| *** hashar has quit IRC | 15:58 | |
| *** gyee has quit IRC | 15:59 | |
| *** ociuhandu_ has quit IRC | 15:59 | |
| *** dtantsur is now known as dtantsur|afk | 16:01 | |
| *** jcapitao has quit IRC | 16:03 | |
| *** diablo_rojo has joined #openstack-infra | 16:03 | |
| *** marios is now known as marios|out | 16:06 | |
| openstackgerrit | Merged openstack/project-config master: Define maintain-github-openstack-mirror job https://review.opendev.org/738228 | 16:07 |
| *** ociuhandu has joined #openstack-infra | 16:12 | |
| *** udesale_ has quit IRC | 16:16 | |
| *** ociuhandu has quit IRC | 16:18 | |
| cgoncalves | hey infra team. could some core review and hopefully give a second +2 on https://review.opendev.org/#/c/738161/ ? thanks! | 16:18 |
| *** gouthamr has quit IRC | 16:26 | |
| openstackgerrit | Merged openstack/project-config master: Add nested-virt-centos-8 label https://review.opendev.org/738161 | 16:33 |
| *** derekh has quit IRC | 16:41 | |
| *** chandankumar has quit IRC | 16:44 | |
| *** xek_ has quit IRC | 16:45 | |
| *** soniya29|rover has quit IRC | 16:46 | |
| *** ykarel|away has quit IRC | 16:46 | |
| *** akahat has quit IRC | 16:46 | |
| *** ociuhandu has joined #openstack-infra | 16:47 | |
| *** hashar has joined #openstack-infra | 16:47 | |
| *** gouthamr has joined #openstack-infra | 16:52 | |
| *** markvoelker has joined #openstack-infra | 16:53 | |
| *** ociuhandu has quit IRC | 16:53 | |
| *** markvoelker has quit IRC | 16:53 | |
| *** Lucas_Gray has quit IRC | 16:57 | |
| *** gyee has joined #openstack-infra | 17:07 | |
| *** marios|out has quit IRC | 17:12 | |
| *** ociuhandu has joined #openstack-infra | 17:16 | |
| *** __ministry1 has quit IRC | 17:20 | |
| *** ociuhandu has quit IRC | 17:23 | |
| *** akahat has joined #openstack-infra | 17:27 | |
| *** eolivare has quit IRC | 17:30 | |
| *** hashar has quit IRC | 17:43 | |
| *** ricolin has quit IRC | 18:00 | |
| *** ralonsoh has quit IRC | 18:07 | |
| *** rlandy|ruck is now known as rlandy|ruck2 | 18:10 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 18:25 | |
| *** ociuhandu has joined #openstack-infra | 18:35 | |
| *** hashar has joined #openstack-infra | 18:36 | |
| *** ociuhandu has quit IRC | 18:41 | |
| *** xek_ has joined #openstack-infra | 18:48 | |
| *** mattw4 has joined #openstack-infra | 19:00 | |
| *** mattw4 has quit IRC | 19:00 | |
| *** rpittau_ has quit IRC | 19:01 | |
| *** gmann_ has joined #openstack-infra | 19:17 | |
| *** gmann_ is now known as gmann | 19:18 | |
| *** piotrowskim has joined #openstack-infra | 19:19 | |
| *** zxiiro has joined #openstack-infra | 19:28 | |
| *** masayukig has joined #openstack-infra | 19:29 | |
| *** arne_wiebalck_ has joined #openstack-infra | 19:30 | |
| *** apetrich has quit IRC | 19:31 | |
| *** donnyd has joined #openstack-infra | 19:31 | |
| *** apetrich has joined #openstack-infra | 19:31 | |
| *** davidlenwell has joined #openstack-infra | 19:32 | |
| *** portdirect has joined #openstack-infra | 19:32 | |
| *** mnaser has joined #openstack-infra | 19:32 | |
| *** cyberpear has joined #openstack-infra | 19:32 | |
| *** dougwig has joined #openstack-infra | 19:33 | |
| *** jberg-dev has joined #openstack-infra | 19:33 | |
| *** lathiat has joined #openstack-infra | 19:34 | |
| *** TheJulia has joined #openstack-infra | 19:35 | |
| *** jungleboyj_ has joined #openstack-infra | 19:35 | |
| *** niedbalski has joined #openstack-infra | 19:37 | |
| *** Ng has joined #openstack-infra | 19:39 | |
| *** rpittau has joined #openstack-infra | 19:40 | |
| *** jamespage has joined #openstack-infra | 19:40 | |
| *** arne_wiebalck_ is now known as arne_wiebalck | 19:41 | |
| *** evgenyl has joined #openstack-infra | 19:41 | |
| *** piotrowskim has quit IRC | 19:42 | |
| *** beisner has joined #openstack-infra | 19:42 | |
| *** howell has joined #openstack-infra | 19:44 | |
| *** tonyb[m] has joined #openstack-infra | 19:44 | |
| *** jrosser has joined #openstack-infra | 19:48 | |
| *** cjohnston has joined #openstack-infra | 19:48 | |
| *** csatari has joined #openstack-infra | 19:49 | |
| *** nicolasbock has joined #openstack-infra | 19:50 | |
| *** zbr_ has joined #openstack-infra | 19:51 | |
| *** kopecmartin is now known as kopecmartin|pto | 19:52 | |
| *** hemna has quit IRC | 19:52 | |
| *** rm_work has joined #openstack-infra | 19:53 | |
| *** vishalmanchanda has joined #openstack-infra | 19:53 | |
| *** rpioso has joined #openstack-infra | 19:54 | |
| *** jbryce has joined #openstack-infra | 19:55 | |
| *** gagehugo has joined #openstack-infra | 19:57 | |
| *** hemna has joined #openstack-infra | 19:59 | |
| *** tdasilva has joined #openstack-infra | 19:59 | |
| *** srwilkers has joined #openstack-infra | 19:59 | |
| *** mnasiadka has joined #openstack-infra | 20:04 | |
| *** knikolla has joined #openstack-infra | 20:05 | |
| *** johnsom_ has joined #openstack-infra | 20:10 | |
| *** mwhahaha has joined #openstack-infra | 20:10 | |
| *** thedac has joined #openstack-infra | 20:12 | |
| *** coreycb has joined #openstack-infra | 20:14 | |
| *** samueldmq has joined #openstack-infra | 20:16 | |
| *** erbarr has joined #openstack-infra | 20:16 | |
| *** seongsoocho has joined #openstack-infra | 20:17 | |
| *** vkmc_ has joined #openstack-infra | 20:17 | |
| *** philroche has joined #openstack-infra | 20:22 | |
| *** xek_ has quit IRC | 20:22 | |
| *** crodriguez has joined #openstack-infra | 20:23 | |
| *** hrybacki has joined #openstack-infra | 20:23 | |
| *** gfidente is now known as gfidente|afk | 20:31 | |
| *** lbragstad_ is now known as lbragstad | 20:42 | |
| *** vdrok has joined #openstack-infra | 20:44 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Update synchronize-repos https://review.opendev.org/740110 | 20:49 |
| *** Lucas_Gray has joined #openstack-infra | 20:52 | |
| *** mnaser is now known as mnaser|ic | 20:52 | |
| *** hashar has quit IRC | 20:55 | |
| rlandy|ruck2 | hi - did anyone report any issues on the RAX provider? we're seeing timeouts and " HTTPError: 429 Client Error: Too Many Requests for url: http://mirror.iad.rax.opendev.org" errors | 21:04 |
| *** mnaser|ic has quit IRC | 21:08 | |
| *** mnaser|ic has joined #openstack-infra | 21:08 | |
| *** mnaser|ic has quit IRC | 21:08 | |
| *** mnaser|ic has joined #openstack-infra | 21:08 | |
| *** mnaser|ic is now known as vexxhost | 21:08 | |
| *** gouthamr has quit IRC | 21:09 | |
| *** gouthamr has joined #openstack-infra | 21:11 | |
| *** vexxhost is now known as mnaser | 21:14 | |
| *** mnaser is now known as mnaser|ic | 21:14 | |
| *** gouthamr_ has joined #openstack-infra | 21:15 | |
| *** johnsom_ is now known as johnsom | 21:19 | |
| *** johnsom has joined #openstack-infra | 21:19 | |
| *** aedc has quit IRC | 21:33 | |
| fungi | rlandy|ruck2: first we've heard of it... gouthamr did mention yesterday in #opendev having a job which suffered spontaneous service restarts and maybe node reboots in rax regions | 21:36 |
| rlandy|ruck2 | fungi: we're going to try slow down our rate of promotions. it's possible the mirrors can't keep up with how fast we are changing the container hashes | 21:36 |
| fungi | rlandy|ruck2: are you seeing that from attempts at distro package installs, or something else? | 21:38 |
| rlandy|ruck2 | mostly from container downloads | 21:39 |
| fungi | ahh, those are proxied, so it may be that the volume of requests shoved through that proxy are greater than dockerhub will allow | 21:39 |
| fungi | a sample log would help narrow it down | 21:40 |
| * rlandy|ruck2 gets | 21:40 | |
| rlandy|ruck2 | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8e5/739595/2/check/tripleo-ci-centos-8-undercloud-containers/8e53807/logs/undercloud/home/zuul/undercloud_install.log | 21:40 |
| fungi | i assume the "tripleou" in there is not a typo | 21:43 |
| fungi | but yeah, port 8082 is definitely a proxy | 21:43 |
| rlandy|ruck2 | https://61e80585713a4e519d20-6c739d88c8cc9fcc2582598a6751723e.ssl.cf5.rackcdn.com/739587/8/gate/tripleo-ci-centos-7-containers-multinode/6b239b1/logs/undercloud/home/zuul/undercloud_install.log | 21:44 |
| rlandy|ruck2 | ^^ another log | 21:44 |
| rlandy|ruck2 | "tripleou" is not a typo :) | 21:44 |
| fungi | still on kitchen duty, but can take a closer look at our mirror logs in a bit | 21:44 |
| rlandy|ruck2 | someone took the tripleoussuri namesapce | 21:44 |
| rlandy|ruck2 | so we shortened it | 21:44 |
| rlandy|ruck2 | k - thanks | 21:44 |
| rlandy|ruck2 | fungi: we're tracking this in a LP bug: https://bugs.launchpad.net/tripleo/+bug/1886867 | 21:47 |
| openstack | Launchpad bug 1886867 in tripleo "Tripleo upstream jobs are failing undercloud install/standalone deploy with "requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url"" [High,Triaged] | 21:47 |
| *** rfolco has quit IRC | 21:52 | |
| *** slaweq has quit IRC | 21:54 | |
| *** portdirect has quit IRC | 21:56 | |
| *** slaweq has joined #openstack-infra | 21:56 | |
| *** portdirect has joined #openstack-infra | 21:56 | |
| *** slaweq has quit IRC | 22:00 | |
| *** vishalmanchanda has quit IRC | 22:03 | |
| *** sshnaidm|afk has quit IRC | 22:03 | |
| fungi | rlandy|ruck2: so one thnig which jumps out at me there... it starts out with "Error detected in auth headers: error invalid_token" and then retries multiple times 8.5 minutes apart, the the *last* error is i think the requests library signalling you've exceeded your desired number of retries | 22:18 |
| fungi | "429 Client Error: Too Many Requests for url" isn't a particularly useful error message to focus on, that's what requests is going to throw any time you max out a retry | 22:19 |
| rlandy|ruck2 | right - we looked at that ... 8.5 minutes apart seemed like a far enough retry | 22:22 |
| rlandy|ruck2 | we'll pick this up tomorrow | 22:23 |
| rlandy|ruck2 | thanks | 22:23 |
| *** armax has quit IRC | 22:23 | |
| fungi | yw, and yeah i think "Error detected in auth headers: error invalid_token" is going to be the thread to pull on | 22:24 |
| fungi | quite possible it's dockerhub returning that | 22:25 |
| *** rlandy|ruck2 is now known as rlandy|bbl | 22:27 | |
| fungi | the access log on the mirror for it is just multiple instances of: | 22:27 |
| fungi | 174.143.130.154 - - [2020-07-08 14:10:22.491] "GET /v2/tripleou/centos-binary-barbican-api/blobs/sha256:15a3874f2bc6e7228f99587b6d7a6cd95a92d85bc85f118503e734b68b129c68 HTTP/1.1" 307 - - "-" "python-requests/2.22.0" | 22:27 |
| fungi | oh, actually that's from an earlier run which happened to use a node with the same address | 22:27 |
| fungi | for that job it was: | 22:28 |
| fungi | 174.143.130.154 - - [2020-07-08 17:13:15.911] "GET /v2/tripleou/centos-binary-barbican-api/blobs/sha256:15a3874f2bc6e7228f99587b6d7a6cd95a92d85bc85f118503e734b68b129c68 HTTP/1.1" 429 226 - "-" "python-requests/2.22.0" | 22:28 |
| fungi | so yeah, confirmed it's a 429 response code each time | 22:28 |
| *** tosky has quit IRC | 22:30 | |
| fungi | hrm, okay, so the 429 does look like it's coming from dockerhub, https://docs.docker.com/registry/spec/api/ indicates they'll return it | 22:34 |
| fungi | https://docs.docker.com/docker-hub/download-rate-limit/ | 22:35 |
| fungi | "The download rate limit introduced by Docker caps the number of objects that users can download within a specified timeframe. Any downloads beyond this limit will result in the 429 Too Many Requests error message." | 22:35 |
| fungi | "Docker will gradually impose download rate limits with an eventual limit of 300 downloads per six hours for anonymous users." | 22:36 |
| fungi | so it's likely our proxy is getting thwapped by dockerhub's rate limit | 22:36 |
| *** rfolco has joined #openstack-infra | 22:52 | |
| *** armax has joined #openstack-infra | 22:56 | |
| *** rfolco has quit IRC | 22:58 | |
| *** tkajinam has joined #openstack-infra | 22:58 | |
| *** rcernin has joined #openstack-infra | 23:01 | |
| *** gfidente|afk has quit IRC | 23:06 | |
| *** hamalq_ has quit IRC | 23:09 | |
| *** hamalq has joined #openstack-infra | 23:10 | |
| *** evrardjp has quit IRC | 23:25 | |
| *** evrardjp has joined #openstack-infra | 23:26 | |
| *** mnaser|ic has quit IRC | 23:38 | |
| *** mnaser|ic has joined #openstack-infra | 23:38 | |
| *** mnaser|ic has quit IRC | 23:38 | |
| *** mnaser|ic has joined #openstack-infra | 23:38 | |
| *** mnaser|ic is now known as mnaser | 23:38 | |
| *** rfolco has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!