Thursday, 2019-11-14

« Wednesday, 2019-11-13 Index Friday, 2019-11-15 »
*** tosky has quit IRC00:36
*** Goneri has quit IRC01:20
*** jpena|off has quit IRC01:54
openstackgerritIan Wienand proposed zuul/nodepool master: [dnm] testing docker registry with dib  https://review.opendev.org/69417702:06
*** rlandy|rover|bbl has quit IRC02:14
*** rlandy|rover|bbl has joined #zuul02:15
*** bhavikdbavishi has joined #zuul02:58
*** bhavikdbavishi1 has joined #zuul03:00
*** bhavikdbavishi has quit IRC03:02
*** bhavikdbavishi1 is now known as bhavikdbavishi03:02
openstackgerritIan Wienand proposed zuul/nodepool master: [dnm] testing docker registry with dib  https://review.opendev.org/69417703:26
*** rlandy|rover|bbl is now known as rlandy|rover03:43
ianwmordred: when we use the intermediate registry, does it *only* have the jobs from the prior job?  i.e. you can't also build an image with opendev/python-builder as well?04:06
ianwthe images from the prior job, i mean04:06
clarkbit will proxy to dockerhub for images not built in the buildset04:07
ianwok, cool, i must have stuffed something up then04:08
mordredianw: yah - what clarkb said04:12
ianwyeah, https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dc7/694177/5/check/nodepool-build-image/dc7c1cf/job-output.txt04:12
ianwweirdly, watching the console you don't see the full output of the docker build i think, it looks different in the logs04:13
openstackgerritIan Wienand proposed zuul/nodepool master: [dnm] testing docker registry with dib  https://review.opendev.org/69417704:14
ianwok, my fault -- i dropped "as nodepool" accidentally, that makes sense with the output04:14
ianwit couldn't find the nodepool image04:14
*** rlandy|rover has quit IRC04:15
*** raukadah is now known as chandankumar04:27
ianw"They mapped those commits back to the mailing list and found 60 that did not have a mailing-list thread identified. Some of those were errors in their tool, but a manual review showed that 24 patches were off-list patches."04:36
ianwhttps://lwn.net/Articles/804511/04:36
ianwonce you've zuuled, this stuff just sounds insane04:36
openstackgerritIan Wienand proposed zuul/nodepool master: Dockerfile: create APP_DIR  https://review.opendev.org/69364604:47
openstackgerritIan Wienand proposed zuul/nodepool master: docker: Base nodepool images on diskimage-builder base image  https://review.opendev.org/69417704:47
openstackgerritIan Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon  https://review.opendev.org/69346404:47
*** sshnaidm|ruck has quit IRC05:11
*** sshnaidm_ has joined #zuul05:12
*** swest has joined #zuul05:55
*** igordc has joined #zuul05:55
*** bhavikdbavishi has quit IRC06:21
*** bhavikdbavishi has joined #zuul06:30
*** sgw has joined #zuul06:34
*** sanjayu__ has joined #zuul06:35
*** sgw has quit IRC07:34
*** igordc has quit IRC07:39
*** pleia2 has quit IRC08:08
*** pleia2 has joined #zuul08:09
*** tosky has joined #zuul08:16
*** jangutter has joined #zuul08:19
*** dmellado has quit IRC08:21
*** dmellado has joined #zuul08:23
*** dmellado has quit IRC08:25
*** dmellado has joined #zuul08:27
*** themroc has joined #zuul08:56
*** jpena has joined #zuul09:09
*** dmellado has quit IRC09:20
*** dmellado has joined #zuul09:24
*** sshnaidm_ is now known as sshnaidm|ruck09:26
*** bhavikdbavishi has quit IRC09:34
*** pcaruana has joined #zuul09:54
*** reiterative has joined #zuul10:00
openstackgerritSimon Westphahl proposed zuul/zuul master: Add optional support for circular dependencies  https://review.opendev.org/68535410:35
*** rfolco has joined #zuul11:37
*** jpena is now known as jpena|lunch12:00
*** rfolco has quit IRC12:51
*** sanjayu__ has quit IRC12:51
*** ryanpetrello has quit IRC12:52
*** smyers has quit IRC12:52
*** paladox has quit IRC12:52
*** ianw has quit IRC12:52
*** jlk has quit IRC12:52
*** amotoki has quit IRC12:52
*** calebb has quit IRC12:52
*** fbo has quit IRC12:52
*** clayg has quit IRC12:52
*** openstackstatus has quit IRC12:53
*** rfolco has joined #zuul12:53
*** sanjayu__ has joined #zuul12:53
*** ryanpetrello has joined #zuul12:53
*** smyers has joined #zuul12:53
*** paladox has joined #zuul12:53
*** ianw has joined #zuul12:53
*** calebb has joined #zuul12:53
*** jlk has joined #zuul12:53
*** amotoki has joined #zuul12:53
*** fbo has joined #zuul12:53
*** clayg has joined #zuul12:53
*** rlandy has joined #zuul12:54
*** rlandy is now known as rlandy|rover12:55
*** rlandy|rover is now known as rlandy|rover|mtg13:03
*** jpena|lunch is now known as jpena13:08
*** avass has joined #zuul13:21
*** sanjayu__ has quit IRC14:28
*** themroc has quit IRC14:42
*** Goneri has joined #zuul14:48
*** chandankumar is now known as raukadah14:51
*** avass has quit IRC15:22
*** rlandy|rover|mtg is now known as rlandy|rover15:28
*** swest has quit IRC15:40
*** michael-beaver has joined #zuul15:43
*** bhavikdbavishi has joined #zuul15:53
*** bhavikdbavishi1 has joined #zuul15:59
*** bhavikdbavishi has quit IRC16:02
*** bhavikdbavishi1 is now known as bhavikdbavishi16:02
openstackgerritFabien Boucher proposed zuul/zuul master: Gitlab - Minimal reporter ables to comment on MR  https://review.opendev.org/69434616:27
openstackgerritFabien Boucher proposed zuul/zuul master: Gitlab - Minimal reporter ables to comment on MR  https://review.opendev.org/69434616:31
fboyeah ! first report from Zuul to gitlab.com https://gitlab.com/fabien.dot.boucher/demo-zuul/merge_requests/7#note_24478249316:43
fungiwoo!16:44
*** sshnaidm|ruck is now known as sshnaidm|off16:56
*** reiterative has quit IRC17:23
*** jangutter has quit IRC17:24
*** tosky has quit IRC17:49
*** michael-beaver has quit IRC17:52
*** reiterative has joined #zuul17:57
*** jpena is now known as jpena|off18:04
ShrewstristanC: where are we on the zuul-stream-functional failures?18:12
pabelangerso, I'm starting to thing zuul.a.c is deadlocked for some reason18:27
pabelangerbut, I am not sure why18:27
pabelangerI cannot see anything in debug logs18:27
pabelangerhttps://dashboard.zuul.ansible.com/t/ansible/status18:27
pabelangermy current guess is it happen just at the time that ansible-network/releases entered the gate change queue18:28
pabelangerhttps://pastebin.com/raw/tkN2gBbc18:28
pabelangeris also list of threads18:28
pabelangermy other thought, is maybe github ratelimited us for some reason18:29
pabelangerThread: 139952270055168 for example18:30
pabelangerhowever, I can see zuul processing github events still18:30
pabelanger2019-11-14 18:30:39,697 DEBUG zuul.GithubRequest: GET https://api.github.com/repos/ansible/ansible/branches/devel result: 200, size: 3852, duration: 21118:31
pabelangerfor example18:31
pabelangermaybe we are just trying to process ansible/ansible, and it is taking a really long time18:31
clarkbpabelanger: there are logs of the start and stop times for processing github events18:33
clarkbif you have astart but no stop it is likely that18:33
clarkbthe post merge jobs ansible runs are very bad for caching and make things slow18:33
pabelangerin that case, I'd expect zuul.o.o to have the same issue for post jobs?18:34
pabelangerbut doesnt' look to be18:34
tristanCShrews: i couldn't figure why ansible would try to exec 'auto', that should be prevented by https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L138118:37
tristanCthough i didn't spent too much time on this18:37
tristanCpabelanger: isn't windmill-config-deploy doing a zuul reconfigure?18:38
pabelangerno, shouldn't have. Just a redeploy18:39
pabelangerif reading right https://pastebin.com/raw/jNHV6RhL is the event when ansible-network/releases entered gate18:41
pabelanger2019-11-14 18:01:46,935 DEBUG zuul.GithubRateLimitHandler: GitHub API rate limit resource: core, remaining: 0, reset: 157375809318:43
pabelangeryah, hit ratelimit it seems18:43
clarkbpabelanger: gerrit triggers independently of github and ee dont run post for github sources iirc18:44
pabelangerso, if I am reading that right, it should start processing again at  Thursday, November 14, 2019 7:01:33 PM18:45
pabelangerso, 15mins~18:45
pabelangerlets see what happens18:45
pabelangeryah, something odd happening. I can see zuul trying to hit apis of repos we don't have github app on: and we then hit the ratelimit18:48
tobiashpabelanger: your scheduler main loop seems to be blocked during addchange because of rate limit18:49
tobiashpabelanger: just to be sure, I guess you're using app auth?18:50
pabelangeryah18:50
pabelanger2019-11-14 18:01:47,248 WARNING zuul.GithubRateLimitHandler: API rate limit reached, need to wait for 3587 seconds18:50
tobiashThats a lot18:50
pabelangerso, the question is, why didn't it use app18:50
tobiashWait time normally is around 30s18:51
pabelangerhttp://paste.openstack.org/show/786131/18:51
pabelangerthat is when I see it18:51
pabelangerbut, I have no idea why we are hitting all those other repos18:51
pabelangerwe don't have ansible-role-bacula-fd in our system18:51
clarkbis your app installed on that repo?18:52
tobiashNo installation ID available for project infOpen/ansible-role-redis18:52
pabelangerno18:52
pabelangerI don't think so18:52
pabelangerit happens after the search query18:52
ShrewstristanC: umm, wait... when were these stream jobs added? I don't see them on any past merges.18:53
pabelangerhttps://api.github.com/search/issues?q=github.com%2Fansible-network%2Freleases%2Fpull%2F4+type%3Apr+is%3Aopen+in%3Abody&per_page=10018:53
tobiashThat explains it, anonymous api is *much* more restricted18:53
pabelangerthat is returning it18:53
tobiashpabelanger: you need the app on all repos your zuul acesses18:54
pabelangertobiash: that is the thing, we don't access those18:54
pabelangernothing in our system does18:54
*** igordc has joined #zuul18:54
pabelangerbut, the search api is returning them18:54
pabelangerso, question is, why does zuul use search API and why are they returned?18:54
tobiashAre those repos backnreferenxing to your repos?18:55
pabelangershouldn't be18:55
pabelangerhttps://github.com/ansible-network/releases18:55
tobiashI guess we should filter the search results to what zuul knows18:55
pabelangeranother 5 mins, and should start again18:56
tristanCShrews: they only ran when ansible files are updated18:56
pabelangertobiash: yah, filtering the return sounds like right idea18:57
ShrewstristanC: ah18:57
pabelangerwell, zuul processed again19:02
pabelangerbut, was rate limited again19:02
pabelanger2019-11-14 19:01:47,630 WARNING zuul.GithubRateLimitHandler: API rate limit reached, need to wait for 3588 seconds19:02
ShrewstristanC: i think i spotted the issue19:09
openstackgerritDavid Shrewsbury proposed zuul/zuul master: WIP: Use correct ansible version var name  https://review.opendev.org/69438819:10
ShrewstristanC: ^^^ hopefully19:10
*** bhavikdbavishi has quit IRC19:12
tristanCShrews: hum, iirc the arguments vars is named 'ansible_version', not zuul_ansible_version19:16
tristanCi'm not sure to get how the zuul_ansible_version from the .zuul.yaml gets all the way to this executor procedure19:17
ShrewstristanC: yeah, i'm making an assumption there. i find it difficult to trace it back, but it comes from: self.arguments = json.loads(job.arguments)19:20
ShrewstristanC: but the var in .zuul.yaml is named zuul_ansible_version19:21
openstackgerritPaul Belanger proposed zuul/zuul master: Don't search github api for projects not in Zuul  https://review.opendev.org/69439019:24
pabelangertobiash: first attempt, totally untested. Need to see if I can confirm that is right approach19:25
pabelangertobiash: given the amount of development you've done with github, open to idea how to test19:25
clarkbpabelanger: I think that breaks the existing depends on contract19:28
clarkbbut that may be necessary to get this woeking19:28
pabelangerclarkb: yah, I was thinking about that too. depends-on external project of zuul19:28
tobiashafaik dependson external project was a thought at some point in time but never implemented19:29
openstackgerritDavid Shrewsbury proposed zuul/zuul master: WIP: Use correct ansible version var name  https://review.opendev.org/69438819:30
tobiashpabelanger: for testing enhance the search_issues method in the fakegithub and return something unrelated19:31
openstackgerritPaul Belanger proposed zuul/zuul master: Don't search github api for projects not in Zuul  https://review.opendev.org/69439019:31
pabelangertobiash: good idea19:32
tobiashfrom my point of view the approach lgtm. If we support external deps at some point (should probably optional) one can still tweak the if clause to accomodate for this19:34
*** themroc has joined #zuul19:38
*** Goneri has quit IRC19:44
ShrewstristanC: hrm, nope, doesn't appear to be the fix19:46
*** zbr is now known as zbr|ooo19:46
pabelangerokay, zuul.a.c cleared the rate-limit this time20:03
*** Goneri has joined #zuul20:43
*** tosky has joined #zuul20:57
*** igordc has quit IRC21:45
*** igordc has joined #zuul21:45
*** rfolco has quit IRC21:55
*** themroc has quit IRC22:14
*** rlandy|rover is now known as rlandy|rover|bia23:14
« Wednesday, 2019-11-13 Index Friday, 2019-11-15 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!