| *** tosky has quit IRC | 00:10 | |
| jhesketh | corvus: \o/, that's awesome. Congrats to tristanC and everybody else who got it over the line! Sorry I couldn't be there to help :-( | 00:14 |
|---|---|---|
| *** hamalq has quit IRC | 01:25 | |
| *** evrardjp has quit IRC | 02:33 | |
| *** evrardjp has joined #zuul | 02:33 | |
| *** saneax has joined #zuul | 03:36 | |
| *** ricolin has quit IRC | 04:47 | |
| *** y2kenny has quit IRC | 04:51 | |
| *** jfoufas1 has joined #zuul | 04:51 | |
| *** ykarel|away has joined #zuul | 05:03 | |
| *** ykarel|away is now known as ykarel | 05:07 | |
| *** ajitha has joined #zuul | 05:25 | |
| *** ykarel_ has joined #zuul | 05:49 | |
| *** ykarel has quit IRC | 05:49 | |
| *** cloudnull has quit IRC | 05:54 | |
| *** jangutter has joined #zuul | 06:05 | |
| *** jangutter_ has quit IRC | 06:08 | |
| *** vishalmanchanda has joined #zuul | 06:19 | |
| *** jcapitao has joined #zuul | 06:25 | |
| *** ykarel__ has joined #zuul | 06:29 | |
| *** ykarel_ has quit IRC | 06:31 | |
| *** cloudnull has joined #zuul | 06:42 | |
| *** hashar has joined #zuul | 06:57 | |
| *** rpittau|afk is now known as rpittau | 07:29 | |
| *** hashar_ has joined #zuul | 07:43 | |
| *** hashar has quit IRC | 07:46 | |
| *** hashar_ is now known as hashar | 07:51 | |
| *** ricolin has joined #zuul | 07:52 | |
| *** wuchunyang has joined #zuul | 07:55 | |
| *** ykarel__ is now known as ykarel | 07:59 | |
| *** harrymichal has joined #zuul | 08:11 | |
| *** tosky has joined #zuul | 08:11 | |
| *** bodgix has quit IRC | 08:14 | |
| *** bodgix has joined #zuul | 08:14 | |
| *** arxcruz has quit IRC | 08:14 | |
| *** arxcruz has joined #zuul | 08:18 | |
| *** nils has joined #zuul | 08:31 | |
| *** tobberydberg has quit IRC | 08:58 | |
| *** jangutter_ has joined #zuul | 08:59 | |
| *** tobberydberg has joined #zuul | 09:00 | |
| *** harrymichal has quit IRC | 09:01 | |
| *** harrymichal has joined #zuul | 09:01 | |
| *** jangutter has quit IRC | 09:02 | |
| *** harrymichal has quit IRC | 09:03 | |
| *** harrymichal has joined #zuul | 09:03 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Dispatch Gerrit events via Zookeeper https://review.opendev.org/c/zuul/zuul/+/783816 | 09:17 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Ensure single instance for active event gathering https://review.opendev.org/c/zuul/zuul/+/783817 | 09:17 |
| *** hashar has quit IRC | 09:20 | |
| *** ykarel is now known as ykarel|lunch | 09:45 | |
| *** jcapitao is now known as jcapitao_lunch | 10:00 | |
| *** wuchunyang has quit IRC | 10:14 | |
| *** harrymichal_ has joined #zuul | 10:19 | |
| *** harrymichal has quit IRC | 10:23 | |
| *** harrymichal_ is now known as harrymichal | 10:23 | |
| *** ykarel|lunch is now known as ykarel | 10:52 | |
| *** harrymichal_ has joined #zuul | 11:08 | |
| *** harrymichal has quit IRC | 11:10 | |
| *** harrymichal_ is now known as harrymichal | 11:10 | |
| *** openstackgerrit has quit IRC | 11:21 | |
| *** hashar has joined #zuul | 11:39 | |
| *** jcapitao_lunch is now known as jcapitao | 11:48 | |
| *** jangutter_ is now known as jangutter | 11:49 | |
| *** jcapitao_ has joined #zuul | 11:50 | |
| jangutter | Newbie question regarding job.provides and job.requires: do I also need to add an entry for job.required-projects? (Or is provides and requires merely a sequencing thing)? | 11:52 |
| *** harrymichal has quit IRC | 11:52 | |
| *** harrymichal has joined #zuul | 11:53 | |
| *** jcapitao has quit IRC | 11:54 | |
| *** harrymichal has quit IRC | 12:05 | |
| *** jcapitao_ has quit IRC | 12:35 | |
| avass | jangutter: currently provides/requires only works across changes and not between two jobs in the same buildset. however you can use zuul_return with job.dependencies instead | 12:36 |
| *** jcapitao_ has joined #zuul | 12:36 | |
| *** jcapitao_ is now known as jcapitao | 12:36 | |
| avass | jangutter: and yeah it's a sequencing thing. a job with "requires: foo" will wait for any jobs for changes ahead of it in the queue that has "provides: foo" | 12:37 |
| avass | zuul will also populate zuul.artifacts for the job with "requires: foo" with any artifacts returned by "provides: foo" | 12:39 |
| avass | https://zuul-ci.org/docs/zuul/reference/jobs.html#var-zuul.artifacts | 12:39 |
| jangutter | avass: aha - job.dependencies is the one I'm looking for -> yeah, I'm passing artifacts. | 12:44 |
| jangutter | avass: Aaah! Now I see what happens "Unable to freeze job graph: Job test1-use-artifacts depends on test2-upload-artifacts which was not run." | 12:53 |
| *** dpawlik6 is now known as dpawlik | 12:58 | |
| avass | jangutter: yeah in that case zuul_return is probably what you need :) | 13:19 |
| jangutter | avass: Thanks, so I structured it like this: project test1 runs test2-build-and-upload (from project test2), then test1-use. Job test1-use has dependencies: test2-build-and-upload. | 13:22 |
| *** hashar has quit IRC | 13:24 | |
| jangutter | avass: Result: single buildset where the "build-and-upload" job publishes to our artifact repository and uses zuul_return to pass the references to the "use" job. | 13:25 |
| jangutter | avass: works like a charm. | 13:25 |
| *** spotz has joined #zuul | 13:30 | |
| corvus | jangutter: you can also make the dependency "soft" if you have a case where you sometimes don't need the build-and-upload job to run; then zuul will wait for it and use it if it runs, but if it doesn't, then it won't fail like above | 13:49 |
| *** ykarel is now known as ykarel|away | 13:50 | |
| *** ykarel|away has quit IRC | 13:54 | |
| *** vishalmanchanda has quit IRC | 14:19 | |
| *** tosky has quit IRC | 14:39 | |
| *** tosky has joined #zuul | 14:39 | |
| *** jangutter has quit IRC | 14:39 | |
| *** jangutter has joined #zuul | 14:40 | |
| fungi | GomathiselviS: i think i see what happened with incomplete testing of 773474. the base-test exercise in 782864 included a roles/remove-build-sshkey-fork/vars/main.yaml which defined zuul_temp_ssh_key, and the original change to zuul-jobs was missing that file for some reason (forgot a git add?). that's why the exercise worked, though i'm not sure why we didn't see the same failure in opendev as others saw | 14:55 |
| fungi | immediately after 773474 merged (could some var caching setting have carried that definition over from add-build-sshkey?) | 14:55 |
| fungi | no, i take that back, the file was there, it was zuul_ssh_key_algorithm which was missing from it | 14:57 |
| fungi | so i'm back to being confused. what we tested is what we merged, there was no zuul_ssh_key_algorithm defined in the copy of remove-build-sshkey we tried in opendev/base-jobs either, so it had to be getting it from the definition in add-build-sshkey somehow | 14:58 |
| fungi | also we need a followup to 783717 which adds that to the role's readme, i'll put that together after my next meeting | 15:00 |
| fungi | but i'm still really curious as to what circumstances caused that to be defined or not | 15:01 |
| *** etp has quit IRC | 15:10 | |
| *** hashar has joined #zuul | 15:32 | |
| *** rpittau is now known as rpittau|afk | 15:33 | |
| *** tosky has quit IRC | 15:36 | |
| *** jfoufas1 has quit IRC | 15:53 | |
| *** wuchunyang has joined #zuul | 16:05 | |
| *** iurygregory_ has joined #zuul | 16:06 | |
| *** iurygregory has quit IRC | 16:06 | |
| *** GomathiselviS has joined #zuul | 16:08 | |
| GomathiselviS | corvus, fungi: https://review.opendev.org/c/opendev/base-jobs/+/783468 - This is to revert the changes made to test a PR using base-test. | 16:14 |
| *** wuchunyang has quit IRC | 16:14 | |
| *** hamalq has joined #zuul | 16:15 | |
| *** iurygregory_ is now known as iurygregory | 16:15 | |
| *** hamalq_ has joined #zuul | 16:19 | |
| *** hamalq has quit IRC | 16:20 | |
| guillaumec | corvus, fungi https://review.opendev.org/c/opendev/gear/+/741288/4..6 | 16:24 |
| *** jcapitao has quit IRC | 16:28 | |
| *** nils has quit IRC | 16:34 | |
| fungi | thanks GomathiselviS. did you catch my comments in here earlier before you joined? http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2021-03-30.log.html#t2021-03-30T14:55:25 | 16:50 |
| fungi | i really can't figure out why it was working for us in opendev but not for some other deployments | 16:50 |
| fungi | i also replied to a message on the zuul-discuss ml about the error, feel free to follow up there with corrections or additional info | 16:52 |
| fungi | http://lists.zuul-ci.org/pipermail/zuul-discuss/2021-March/001546.html | 16:52 |
| *** GomathiselviS has quit IRC | 16:55 | |
| fungi | guillaumec: thanks, that looks good. it's too bad tls v1.3 is broken like that, but from zuul's perspective tls v1.2 is fine for what little remaining time we rely on gear. if there are other users to whom 1.3 support is important this at least gives them an idea of what needs to be done to add it robustly | 16:57 |
| fungi | but i suspect it would need some significant changes to its current socket handling to get there | 16:57 |
| clarkb | guillaumec: am I reading the upsteram bug correctly that it seems you must do non blocking reads, but even that doesn't work ? | 17:02 |
| *** tosky has joined #zuul | 17:03 | |
| clarkb | seems liek a bug if you do a poll, poll says you can read, you do a read, and then you must ignore an error to proceed | 17:03 |
| clarkb | hopefully upstream agrees (I think doing a non blocking read isn't a terrible thing, but it shouldn't return an error in this case) | 17:04 |
| corvus | guillaumec: change lgtm -- is it ready to merge? (have you tested it with zuul via depends on? or do we still need to do that) | 17:06 |
| *** jangutter_ has joined #zuul | 17:09 | |
| *** jangutter has quit IRC | 17:13 | |
| fungi | there was a depends-on change which demonstrated the error, let's see if i can find it | 17:19 |
| *** GomathiselviS has joined #zuul | 17:26 | |
| GomathiselviS | fungi: I did see the conversation. My bad, I missed the issue. | 17:27 |
| fungi | GomathiselviS: it was more that i was curious why we didn't see it break that way in opendev, and what differed in other deployments to expose the error | 17:28 |
| *** hashar is now known as hasharDinner | 17:36 | |
| *** ykarel|away has joined #zuul | 17:46 | |
| *** GomathiselviS has quit IRC | 18:06 | |
| *** ykarel|away has quit IRC | 18:16 | |
| guillaumec | corvus, yes: https://review.opendev.org/c/zuul/zuul/+/780261 | 18:29 |
| corvus | guillaumec: great, thanks; clarkb, tobiash: do you want to review https://review.opendev.org/741288 or should i +w it? | 18:32 |
| zbr|rover | corvus: i was looking at an old change related to yarn and building docs. we cannot add yarn to bindep for dpkg platforms because it will not install it (it needs extra repo) | 18:36 |
| zbr|rover | so in that case bindep is not of much help, likely this is the reason why it was added only for alpine. | 18:36 |
| zbr|rover | quite a popular tool not to be included in default set of repos | 18:37 |
| tobiash | corvus: I think we should probably adopt that to zuul and nodepool | 18:37 |
| corvus | tobiash: you mean merge that, release gear, and bump the reqs? | 18:37 |
| tobiash | corvus: I mean do the same on the tls stuff in zuul, but release as well | 18:38 |
| corvus | tobiash: where? i don't think i understand | 18:40 |
| corvus | iiuc, only gear tickles this python bug, so if we make the fix in gear it should be sufficient | 18:40 |
| fungi | zbr|rover: looks like ubuntu focal (current lts) and debian buster (current stable) both have yarnpkg packages | 18:41 |
| zbr|rover | fungi: thanks, adding it. | 18:41 |
| fungi | zbr|rover: 1.22.4 in focal, 1.13.0 in buster | 18:41 |
| fungi | buster-backports and the currently frozen upcoming debian bullseye release both have yarnpkg 1.22 (1.22.4 in buster-backports, 1.22.10 in bullseye) | 18:43 |
| *** saneax has quit IRC | 18:45 | |
| *** y2kenny has joined #zuul | 19:07 | |
| y2kenny | Hi, I am still a bit confused about the security model for roles include. Can roles from untrusted project be included by playbook in trusted project if the roles are checked-in/non-speculative or is that inclusion direction (untrusted roles from trusted) blocked entirely? | 19:11 |
| avass | y2kenny: you can still include roles from an untrusted project but they won't load speculatively | 19:12 |
| y2kenny | avass:... Oh... when you say won't load speculatively meaning the roles won't be available for certain type of pipeline? (like check for example?) | 19:13 |
| fungi | proposed changes for the role won't be incorporated by playbooks in trusted projects | 19:13 |
| y2kenny | like the roles will be available for 'submit' pipeline? | 19:13 |
| avass | y2kenny: I mean that if you update a role those changes won't actually be used until they're merged | 19:14 |
| fungi | they'll have to merge before that state will be used | 19:14 |
| fungi | we attempted to convey that in the first paragraph of https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.roles | 19:14 |
| y2kenny | um... ok... then I must be missing something else. I still get roles not found even after the role has been submitted | 19:14 |
| y2kenny | do I need to set "include" in the tenant file? | 19:15 |
| y2kenny | for roles? | 19:15 |
| y2kenny | fungi: I have that setup but for some reason I still get role not found. | 19:15 |
| avass | no those should be included by default, but you need to set job.roles that fungi linked and the roles directory need to be in the root of the project | 19:15 |
| fungi | and the project containing the role needs to be included | 19:16 |
| fungi | e.g. as a required-projects entry, except in certain implicit cases like it's in the project for the triggering event | 19:17 |
| avass | oh I didn't know that was needed | 19:17 |
| y2kenny | fungi: OOOOooooo... I think that's what I missed | 19:18 |
| y2kenny | the required-projects bit | 19:18 |
| fungi | well, i may be wrong about that too. i need to double-check when i'm not in a meeting | 19:19 |
| avass | I don't think it's needed since I'm not doing that here: https://github.com/albinvass/zuul-config/blob/master/zuul.d/jobs.yaml#L16 | 19:21 |
| avass | y2kenny: looking at my config, are you using project canonical name or just the short name of the project? | 19:22 |
| y2kenny | Oh... that may be it... I used short | 19:23 |
| avass | y2kenny: I have a vague memory of there being a problem if there are slashes in the project name like "zuul/zuul" and it will only work with the canonical name in that case | 19:24 |
| y2kenny | avass: ok, I will give that a try | 19:24 |
| avass | so the project "zuul" would work for short name but not "zuul/zuul", that would need to be "opendev.org/zuul/zuul" | 19:24 |
| y2kenny | Ah... yea, my project has slashes... | 19:25 |
| fungi | yeah, i think i was remembering that you need the project containing the role to be in the tenant config, and if you restrict what zuul loads from that project you need it to specify to at least include roles from there | 19:26 |
| y2kenny | fungi: ok... I wasn't sure if roles are one of the configurable item since it wasn't listed here: https://zuul-ci.org/docs/zuul/reference/tenants.html#attr-tenant.untrusted-projects.%3Cproject%3E.include | 19:27 |
| y2kenny | although I think my current issue might be the canonical name. | 19:28 |
| fungi | oh, yeah i think you're right you don't have to specify inclusion of roles, they're technically not zuul configuration | 19:32 |
| fungi | so the project just needs to be in the tenant | 19:32 |
| avass | yeah the logic for loading roles just splits on slashes and that borks if the short project name has a slash: https://opendev.org/zuul/zuul/src/branch/master/zuul/configloader.py#L956 | 19:41 |
| avass | actually it's this part that's causing the issue: https://opendev.org/zuul/zuul/src/branch/master/zuul/model.py#L4960 | 19:47 |
| y2kenny | um... the canonical name didn't seem to fix it. I still get: | 19:50 |
| y2kenny | ERROR! the role 'test/baremetal-libdrm' was not found in /var/lib/zuul/builds/b13bcabe844a4a91891caa4b882da53b/trusted/project_0/github.com/osg/ec/infra/zuul-config/playbooks/test-baremetal-role/roles:/var/lib/zuul/builds/b13bcabe844a4a91891caa4b882da53b/ansible/playbook_0/role_0/zuul-config/roles:/var/lib/zuul/builds/b13bcabe844a4a91891caa4b882da5 | 19:50 |
| y2kenny | 3b/ansible/playbook_0/role_1/zuul-jobs/roles:/var/lib/zuul/builds/b13bcabe844a4a91891caa4b882da53b/trusted/project_0/github.com/osg/ec/infra/zuul-config/playbooks/test-baremetal-role | 19:50 |
| y2kenny | my role is in osg/ec/infra/zuul-job/roles | 19:51 |
| y2kenny | I have 'zuul: github.com/osg/ec/infra/zuul-jobs' defined in the job | 19:52 |
| y2kenny | may be I do have to add required-project | 19:52 |
| y2kenny | but it's weird because I am the zuul-job project is the triggering project currently | 19:52 |
| fungi | are there branches involved? | 19:54 |
| fungi | do the repositories have multiple branches, and is the build triggered for a branch with a matching branch name containing the role in the other project? | 19:54 |
| y2kenny | no special branches. All just master | 19:54 |
| y2kenny | if I have the role with the same name in the zuul-config (trusted) project then the role is matched | 19:55 |
| fungi | okay, that's not it then | 19:55 |
| y2kenny | for some reason the role is not matched even though the error message listed out the location of the role | 19:56 |
| y2kenny | (zuul-job/roles) | 19:57 |
| corvus | y2kenny: sure that was the right zuul-jobs? maybe that was opendev's zuul-jobs rep | 19:59 |
| corvus | repo | 19:59 |
| y2kenny | corvus: that's possible... I do include the opendev one | 20:00 |
| y2kenny | for "- zuul: myorg/our-roles-project" what is the meaning of the name? is that just to differentiate between zuul roles and galaxy roles? | 20:01 |
| y2kenny | oh nevermind, I see the docs scrolling down | 20:02 |
| avass | this could be relevant: https://review.opendev.org/c/zuul/zuul/+/783965 I'm not sure if that's going to affect anything but I guess that's what's actually wanted there | 20:02 |
| corvus | y2kenny: check the debug log from the executor, it may have chosen to exclude the roles for some reason | 20:02 |
| y2kenny | corvus: ok... doing that now... | 20:03 |
| corvus | y2kenny: look for lines near "Prepare zuul role for" | 20:04 |
| y2kenny | corvus: ok so I see a lot of those for a specific event. I see my zuul-jobs being added as 'implicit' | 20:16 |
| y2kenny | one of those are add roles to ansible/playbook_0/role_0/zuul-jobs/roles | 20:17 |
| y2kenny | the other add roles to playbook_0/role_2/zuul-jobs/roles | 20:17 |
| avass | corvus, y2kenny: I think my fix is correct: https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L1868 | 20:18 |
| y2kenny | but my earlier error message, there's only one role_#/zuul-jobs/roles | 20:18 |
| *** hasharDinner has quit IRC | 20:20 | |
| y2kenny | avass: are you saying I should upgrade my executor? | 20:20 |
| corvus | avass: fix for what bug? i don't think we have a bug identified yet | 20:20 |
| corvus | y2kenny: can you share the actual log entries (and possibly job definitions?) | 20:24 |
| y2kenny | corvus: sure. what's the best way to do it? (I forgot your paste bin url again...) | 20:24 |
| corvus | http://paste.openstack.org/ | 20:24 |
| y2kenny | corvus: so the other odd thing is that there's a lot of Prepare zuul role (like 50+ of them) most of them just repeating for the same project | 20:26 |
| y2kenny | corvus: job definition http://paste.openstack.org/show/II0YmSvb4RR8ksbwkVeZ/ | 20:29 |
| y2kenny | log snip: http://paste.openstack.org/show/2iMOlL3o09fIINOXy1yQ/ | 20:29 |
| y2kenny | looking at the pattern of the logs, seems like roles are prepared per play but specified roles are only added in a very narrow circumstances | 20:32 |
| corvus | y2kenny: is git.amd.com/osg/ec/infra/zuul-jobs/roles/test/baremetal-libdrm/ where your role is? | 20:33 |
| y2kenny | corvus: yes | 20:33 |
| y2kenny | from the executor log, I see 53 instances of "Prepare zuul role" for event bf5c14eb | 20:34 |
| corvus | y2kenny: can you paste the "ERROR! the role 'test/baremetal-libdrm' was not found in" from the 4a85cf5c453344d39e38428456758c76 build? | 20:34 |
| y2kenny | 25 are for opendev | 20:34 |
| y2kenny | corvus: yes, one moment | 20:35 |
| y2kenny | corvus: ok so that build does not have error because it actually does not use the role | 20:37 |
| y2kenny | it's one of the build job before the test job | 20:38 |
| corvus | y2kenny: we need the logs from the job that's failing :) | 20:38 |
| y2kenny | yes... I will get you that | 20:38 |
| y2kenny | the build that failed are: b13bcabe844a4a91891caa4b882da53b | 20:38 |
| y2kenny | 29c07fc0d9b44e56b7f7f6aa88acd205 | 20:38 |
| y2kenny | and e4eab5562a84408ab5d2048d856a702f | 20:38 |
| y2kenny | let me start a new paste... | 20:39 |
| corvus | let's just stick with b13 | 20:39 |
| avass | corvus, y2kenny: there are two target_name: zuul-jobs and I believe the second overwrites the first one | 20:39 |
| avass | my change should fix that | 20:40 |
| corvus | avass: i don't believe that's the case | 20:40 |
| avass | oh? | 20:40 |
| corvus | avass: the name is only used for bare roles | 20:40 |
| corvus | afaict, these are contained roles, so the target name doesn't matter; ansible will never see it | 20:41 |
| avass | it looks like it's symlinked to: ansible/pre_playbook_0/role_1/zuul-jobs | 20:41 |
| avass | oh | 20:41 |
| corvus | avass: ansible sees a role dir at /var/lib/zuul/builds/4a85cf5c453344d39e38428456758c76/ansible/post_playbook_0/role_1/zuul-jobs/roles | 20:41 |
| y2kenny | corvus: all 3 fails the same way: http://paste.openstack.org/show/03WsuykOkZ9ahUlekrvW/ | 20:42 |
| avass | corvus: but isn't that a symlink? | 20:42 |
| corvus | avass: so to ansible that looks like "/some/path/that/does/not/matter/" and under that if the user asks for role "test/baremetal-libdrm" it will look under that directory | 20:42 |
| corvus | avass: you're suggesting the target of the symlink is overwritten? | 20:43 |
| corvus | the targets of the symlinks look like "var/lib/zuul/builds/4a85cf5c453344d39e38428456758c76/trusted/project_1" so the role names really don't matter there | 20:44 |
| avass | no actually after a second look I think the symlink is pointing to the wrong target | 20:44 |
| avass | or no, the symlink is getting overwritten. I'm getting my src/dest mixed up | 20:44 |
| corvus | y2kenny: can you get me the "Prepare zuul role for" for build b13bcabe844a4a91891caa4b882da53b ? | 20:44 |
| corvus | y2kenny: we're trying to correlate the directories in that error message to the roles that zuul prepped; we need a matched pair of logs for that | 20:45 |
| avass | I'm gonna continue digging into that | 20:46 |
| y2kenny | corvus: um... I guess it's possible for same event to be served by multiple executors right? (I am not seeing that build on executor_0... I guess it's on another executor... one sec... | 20:47 |
| corvus | correct, different builds for the same buildset can be (and usually are) served from different executors | 20:48 |
| avass | corvus: oh got it, I wasn't seeing the role_# in the source anywhere so forgot about that part. but that was a bit clearer after looking at the executor workdir | 20:50 |
| y2kenny | corvus: this should be all of it: http://paste.openstack.org/show/MgDdTmsRNdp3YUMFfgx9/ | 20:52 |
| y2kenny | for some reason I only see opendev ones | 20:53 |
| y2kenny | for zuul-job | 20:53 |
| y2kenny | s | 20:53 |
| y2kenny | so seems like whatever is feeding to "Prepare zuul role" already not have my role repo | 20:54 |
| y2kenny | and the earlier zuul-jobs inclusion via implicit may have been a red herring... | 20:56 |
| avass | corvus: and I cannot reproduce the issue with slashes we had either :/ | 20:56 |
| corvus | looking at logs now | 20:57 |
| avass | y2kenny: oh! | 20:59 |
| corvus | y2kenny: yeah, it does look like zuul did not prepare the ec/infra/zuul-jobs repo as a role directory for that job (but in the other logs you pasted, it did). i am assuming that b13bcabe844a4a91891caa4b882da53b is a build of the "test-libdrm" job that you pasted here http://paste.openstack.org/show/II0YmSvb4RR8ksbwkVeZ/ , and that job definition is already merged (ie, the change that you're testing isn't | 21:01 |
| corvus | the change that adds that job); is that all correct? | 21:01 |
| avass | the __eq__ for class Role compares target_name: https://opendev.org/zuul/zuul/src/branch/master/zuul/model.py#L1125 | 21:02 |
| avass | and that might override in addRole because of that?: https://opendev.org/zuul/zuul/src/branch/master/zuul/model.py#L1427 | 21:03 |
| avass | oh but that's a ZuulRole and not a Role instance so nvm | 21:03 |
| y2kenny | corvus: um... that's may be a bit complicated. The job definition already exist but the event that triggered the jobs came from changing some comment inside that job definition... I am not sure if that matters. | 21:04 |
| y2kenny | there's no new role or job | 21:05 |
| y2kenny | those are all already there. The trigger was just a comment change as a test trigger | 21:05 |
| y2kenny | corvus: I don't know if this matter but I pasted the job definition again: http://paste.openstack.org/show/2iKjrZlUHeZ5WXddm349/ | 21:07 |
| y2kenny | test-libdrm itself is actually a parent itself... there are 3 other jobs that inherit its definition but it's really just for the nodeset variation | 21:08 |
| y2kenny | (that's why there are 3 build that fail the same way.) | 21:08 |
| corvus | y2kenny: as long as the version of the job that is currently merged has the "roles:" definition in it (and the current state of the zuul-jobs repo also has that role in it), that should be fine -- it's only if the change in flight is adding one of those things that we need to get into the weeds about whether something isn't being speculatively tested. | 21:10 |
| corvus | y2kenny: what version of the scheduler are you running? 4.1.0 or a recent master? | 21:10 |
| y2kenny | corvus: 4.0.0 actually... I can upgrade and retry if needed | 21:11 |
| y2kenny | the upgrade from 3.x was so smooth that I am not hesitating in upgrading :) | 21:11 |
| corvus | y2kenny: don't worry about it, it shouldn't be relevant. there's an additional debug tool available to us in current master, but it also has a memory leak, so i'm not going to ask you to upgrade. :) | 21:16 |
| y2kenny | ok | 21:16 |
| corvus | 4.1.0 should be safe, btw, but doesn't have any relevant changes. | 21:16 |
| y2kenny | I will upgrade after I finish my snacks;) | 21:17 |
| corvus | y2kenny: i am starting to get a little stumped here. we've at least established that the issue is that zuul didn't prepare the role repo; which i think puts us in the realm of looking for something in the job definitions. like perhaps http://paste.openstack.org/show/II0YmSvb4RR8ksbwkVeZ/ didn't apply to that branch or something like that. | 21:18 |
| y2kenny | corvus: um... but that's only one branch for all these repos... I am wondering if it has to do with the location of various jobs | 21:21 |
| avass | y2kenny: does the job definition look like that or does it also have playbooks configured? | 21:21 |
| y2kenny | so test-libdrm is in my zuul-jobs but test-baremetal-role-fedora is in zuul-config | 21:22 |
| y2kenny | avass: the test-libdrm does not have playbook associated with it. It's the test-baremetal-role-fedora job that has playbook associate with it | 21:22 |
| y2kenny | and it is in zuul-config (trusted) project | 21:23 |
| avass | corvus: roles are associated with specific playbooks as I understand it, am I right? | 21:23 |
| y2kenny | the playbook in zuul-config is trying to call the role in zuul-jobs | 21:23 |
| y2kenny | but the triggering repo change is zuul-jobs | 21:24 |
| y2kenny | what I don't get is why some of the 'prior-job' of test-libdrm has the roles inclusion properly | 21:25 |
| corvus | avass: oh, i think you've got it | 21:25 |
| avass | so those roles are never added to the parents playbooks execution | 21:26 |
| corvus | y2kenny: the "roles:" stanza needs to go in the same job definition as the "run:" playbook that uses it | 21:26 |
| corvus | basically, you're saying "the playbooks in this job use these roles" | 21:26 |
| corvus | and in http://paste.openstack.org/show/II0YmSvb4RR8ksbwkVeZ/ there are no playbooks, so zuul did nothing | 21:27 |
| y2kenny | Oooooo | 21:27 |
| y2kenny | that can be fix and tested fairly quickly | 21:27 |
| y2kenny | so the roles needs to be in test-baremetal-role-fedora | 21:27 |
| corvus | but presumably test-baremetal-role-fedora or maybe its parent has a "run: something.yaml" that executes "{{ testrole }}"; that's the place to add "roles: ..." | 21:27 |
| corvus | yep | 21:28 |
| y2kenny | ok... I will give it a try | 21:28 |
| corvus | y2kenny: (and if test-baremetal-role-fedora is in a trusted repository, you'll need to merge that change for it to take effect) | 21:28 |
| corvus | avass: yep it's in paragraph 2 of the docs: https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.roles :) | 21:29 |
| avass | corvus: oh there's documentation for that :) | 21:30 |
| y2kenny | my appology for not reading the doc properly. | 21:32 |
| avass | well there is a lot to read | 21:33 |
| corvus | my apologies for not writing it properly :) | 21:35 |
| corvus | y2kenny: is it working now? | 21:35 |
| y2kenny | working on that... will have the result shortly. | 21:35 |
| corvus | cool, let me know :) | 21:36 |
| y2kenny | corvus: works! | 21:46 |
| y2kenny | thanks you corvus and avass for goose chasing with me | 21:46 |
| corvus | y2kenny: \o/ | 21:48 |
| avass | y2kenny: no problem! | 21:49 |
| fungi | excellent, i hadn't even thought to ask if there was inheritance/variance in play, and whether the roles were defined along with the relevant playbook | 21:59 |
| *** ajitha has quit IRC | 22:04 | |
| y2kenny | something unrelated - for jobs that is SUCCESS/FAILURE/TIMED_OUT, I see logs, etc. being captured but for some reason there is nothing when the job has the RETRY/RETRY_LIMIT, is that intentional? Reading from the docs, RETRY_LIMIT status is supposed to be "The pre-run playbook failed more than the maximum number of retry attempts." but from what | 22:13 |
| y2kenny | I can tell, may be it's not just pre-run? By watching the stream logs, looks like the job reached the run playbook but then subsequently become unreachable (the stream ended with RESULT_UNREACHABLE.) Is there a way to keep the job log even when the job ends with RESULT_UNREARCHABLE? | 22:13 |
| y2kenny | is cleanup-run something that can be leveraged here or that's not quite it | 22:16 |
| corvus | y2kenny: base job post-run playbooks should still run and upload whatever logs (like the console log) are on the executor, so the intent is there should still be logs there for retry_limit (retry jobs that haven't hit retry_limit yet don't report). | 22:19 |
| y2kenny | corvus: ok. I will double check my post-run configurations. | 22:21 |
| corvus | (basically, the only thing cleanup playbooks get you is they run even if the job is aborted; we don't want to save logs from aborted jobs -- it's a waste -- zuul aborts a lot of jobs) | 22:23 |
| y2kenny | understood. | 22:23 |
| y2kenny | corvus: um... this is weird... I do have post-run defined in the base job but they don't seem to run. The last line from the stream is: "RUN END RESULT_UNREACHABLE: [trusted : git.amd.com/osg/ec/infra/zuul-config/playbooks/test-baremetal-role/run.yaml@master]" | 22:28 |
| tristanC | with https://review.opendev.org/c/zuul/zuul/+/780964 , zuul-runner can now reproduce job locally with this command: `tox -evenv -- zuul-runner --api https://zuul.opendev.org --tenant zuul --job zuul-tox-py36 --pipeline check --project zuul/zuul --nodes ssh:ubuntu-bionic:192.168.243.42:zuul:/home/zuul --skip-playbook post --skip-playbook opendev/base-jobs/playbooks/base/pre.yaml` | 22:29 |
| y2kenny | (just for some context, the test ran on the baremetal node but it probably crashed the kernel/driver so badly that the node is not reachable by ansible/executor.) | 22:29 |
| tristanC | unfortunately, opendev base pre job needs to be removed because the mirror configuration is not properly skipped and it setups invalid url (e.g. index-url = https://__omit_place_holder__7726381c6f190bdb7700279c420829b0d8255d20/pypi/simple ) | 22:32 |
| *** dpawlik has quit IRC | 22:37 | |
| *** holser has quit IRC | 22:42 | |
| *** bschanzel has quit IRC | 22:42 | |
| *** openstackstatus has quit IRC | 22:42 | |
| *** aluria has quit IRC | 22:42 | |
| *** dpawlik0 has joined #zuul | 22:42 | |
| *** bschanzel_ has joined #zuul | 22:42 | |
| *** openstack has joined #zuul | 22:46 | |
| *** ChanServ sets mode: +o openstack | 22:46 | |
| *** tosky has quit IRC | 23:14 | |
| *** openstackgerrit has joined #zuul | 23:31 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix ZK-related race condition in github driver https://review.opendev.org/c/zuul/zuul/+/783726 | 23:31 |
| clarkb | opendev is seeing a scenario where zuul decides it doesn't need to update a repo because the ref exists and the rev is present. However, it seems that if you merge a fastforwardable change to a ref zuul doesn't use that new change state because the ref exists and the existing change state is present for the rev. But the ref is never updated to point at rev which causes subsequent jobs to fail | 23:56 |
| clarkb | has anyone seen something like this? I think the underlying cause is the merger's isUpdateNeeded() isn't checking if ref points at rev, just that ref and rev exist separately | 23:56 |
| clarkb | that said I would have expected to see these problems more frequently if this was the case | 23:56 |
| clarkb | fungi: pointed out we should be able to reproduce this issue in the unittest framework. I'll try to spend some time with that tomorrow. And then see if checking that ref points at rev fixes it. Just thought I would call it out in case this is a known issue and I should reivew changes instead :) | 23:59 |
| fungi | so basically we need two projects, one where we merge a change, then merge another fast-forwardable change on the same branch, then a second project which triggers a job consuming that branch of the first project | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!