| *** armstrongs has quit IRC | 00:04 | |
| *** saneax has quit IRC | 00:27 | |
| *** wuchunyang has joined #zuul | 00:59 | |
| *** Goneri has quit IRC | 01:02 | |
| *** swest has quit IRC | 01:56 | |
| *** swest has joined #zuul | 02:11 | |
| *** saneax has joined #zuul | 02:15 | |
| *** rfolco has joined #zuul | 02:55 | |
| *** mugsie has quit IRC | 02:59 | |
| *** rfolco has quit IRC | 02:59 | |
| *** rfolco has joined #zuul | 03:00 | |
| *** mugsie has joined #zuul | 03:02 | |
| *** rfolco has quit IRC | 03:05 | |
| *** sgw has quit IRC | 03:07 | |
| *** Goneri has joined #zuul | 03:28 | |
| *** bhavikdbavishi has joined #zuul | 03:43 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined https://review.opendev.org/738952 | 03:47 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined https://review.opendev.org/738952 | 03:52 |
| *** Goneri has quit IRC | 03:53 | |
| *** wuchunyang has quit IRC | 03:56 | |
| *** bhavikdbavishi1 has joined #zuul | 04:00 | |
| *** wuchunyang has joined #zuul | 04:01 | |
| *** bhavikdbavishi has quit IRC | 04:02 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:02 | |
| *** wuchunyang has quit IRC | 04:04 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #zuul | 04:33 | |
| *** bhavikdbavishi has quit IRC | 04:34 | |
| *** bhavikdbavishi has joined #zuul | 04:36 | |
| *** ysandeep|away is now known as ysandeep | 04:37 | |
| *** wuchunyang has joined #zuul | 04:45 | |
| *** vishalmanchanda has joined #zuul | 04:50 | |
| *** wuchunyang has quit IRC | 05:00 | |
| *** bhagyashris|afk is now known as bhagyashris | 05:11 | |
| *** saneax has quit IRC | 05:19 | |
| *** saneax has joined #zuul | 05:40 | |
| *** reiterative has quit IRC | 05:42 | |
| *** reiterative has joined #zuul | 05:43 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined https://review.opendev.org/738952 | 05:43 |
| *** marios has joined #zuul | 05:44 | |
| *** bhavikdbavishi has quit IRC | 06:00 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: emit-job-header: add inventory hostname https://review.opendev.org/738963 | 06:01 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 06:03 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined https://review.opendev.org/738952 | 06:26 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined https://review.opendev.org/738952 | 06:37 |
| *** hashar has joined #zuul | 06:39 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined https://review.opendev.org/738952 | 06:54 |
| *** newbie2020 has joined #zuul | 06:59 | |
| *** wuchunyang has joined #zuul | 06:59 | |
| *** jcapitao has joined #zuul | 07:13 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined https://review.opendev.org/738952 | 07:16 |
| *** harrymichal has joined #zuul | 07:21 | |
| *** bhagyashris is now known as bhagyashris|lunc | 07:28 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined https://review.opendev.org/738952 | 07:29 |
| *** tosky has joined #zuul | 07:46 | |
| *** jpena|off is now known as jpena | 07:56 | |
| *** ysandeep is now known as ysandeep|lunch | 08:02 | |
| *** nils has joined #zuul | 08:11 | |
| *** harrymichal has quit IRC | 08:19 | |
| *** wuchunyang has quit IRC | 08:39 | |
| *** wuchunyang has joined #zuul | 08:47 | |
| *** bhagyashris|lunc is now known as bhagyashris | 08:50 | |
| *** wuchunyang has quit IRC | 09:01 | |
| *** ysandeep|lunch is now known as ysandeep | 09:01 | |
| *** hashar has quit IRC | 09:16 | |
| *** wuchunyang has joined #zuul | 10:12 | |
| *** wuchunyang has quit IRC | 10:17 | |
| *** jcapitao is now known as jcapitao_lunch | 10:32 | |
| *** wuchunyang has joined #zuul | 10:46 | |
| *** wuchunyang has quit IRC | 10:59 | |
| *** newbie2020 has quit IRC | 11:20 | |
| *** hashar has joined #zuul | 11:34 | |
| *** jpena is now known as jpena|lunch | 11:43 | |
| *** rfolco has joined #zuul | 11:48 | |
| *** ysandeep is now known as ysandeep|afk | 12:01 | |
| *** jcapitao_lunch is now known as jcapitao | 12:04 | |
| *** rlandy has joined #zuul | 12:07 | |
| *** wuchunyang has joined #zuul | 12:08 | |
| *** rlandy is now known as rlandy|ruck | 12:10 | |
| *** wuchunyang has quit IRC | 12:31 | |
| *** ysandeep|afk is now known as ysandeep | 12:33 | |
| *** jpena|lunch is now known as jpena | 12:42 | |
| *** LLIU82 has joined #zuul | 13:29 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: Fix branch name and project name for ref-updated event https://review.opendev.org/738320 | 13:44 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: scheduler: Fix event process abide hasUnparsedBranchCache argument https://review.opendev.org/739042 | 13:44 |
| *** vishalmanchanda has quit IRC | 13:48 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add a job for publishing a site to netlify https://review.opendev.org/739047 | 13:55 |
| *** sgw has joined #zuul | 13:57 | |
| mordred | corvus, avass: ^^ I'm not sure if there is a good way to test that | 13:58 |
| *** Goneri has joined #zuul | 14:01 | |
| avass | mordred: what does the '--prod' flag do | 14:04 |
| avass | mordred: tried to check it but it seems 'netlify publish' isn't a command :) | 14:04 |
| mordred | avass: actually publishes it live. if you leave that off it with publish to a preview site - kind of similar to how our per-build previews work - but since we have that in zuul already it doesn't seem as exciting to support | 14:05 |
| avass | mordred: ah yeah, just thought that would be a way to test it | 14:06 |
| mordred | avass: you should be able to run "npx -p netlify-cli netlify publish --help" ... oh, wait - it's deploy not publish. nice catch | 14:07 |
| avass | well, unless you want to fake their api :) | 14:07 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add a job for publishing a site to netlify https://review.opendev.org/739047 | 14:07 |
| mordred | avass: so you should be able to run "npx -p netlify-cli netlify deploy --help" | 14:08 |
| mordred | :) | 14:08 |
| avass | mordred: yep, can't find anything that would help so I guess it looks good | 14:10 |
| AJaeger | mordred: did you see my comment on https://review.opendev.org/#/c/739047/1/roles/netlify-publish/README.rst ? | 14:10 |
| avass | mordred: unless this helps: https://www.netlify.com/products/dev/ | 14:13 |
| mordred | AJaeger: ah - missed that. thanks! | 14:13 |
| mordred | avass: I don't think so - I think that's more akin to "yarn develop" except running a netlfy - I don't think the netlify deploy command has support for pointing to an alternate api location | 14:15 |
| *** bhagyashris is now known as bhagyashris|afk | 14:15 | |
| avass | mordred: ah alright | 14:15 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add a job for publishing a site to netlify https://review.opendev.org/739047 | 14:16 |
| zbr | can we start publishing zuul-jobs as an ansible collection? i have a very good use-case for enabling local testing. | 14:27 |
| *** LLIU82 has quit IRC | 14:34 | |
| *** ysandeep is now known as ysandeep|away | 14:43 | |
| openstackgerrit | Alex Schultz proposed zuul/zuul-jobs master: Make persist-iptables more robust https://review.opendev.org/739061 | 14:45 |
| fungi | i thought we had made decisions which imply that zuul-jobs was not intended as a collection | 14:51 |
| fungi | for example, we have hyphens in role names, and have disabled the ansible-lint error for that condition on the grounds that we're not making a collection from zuul-jobs | 14:52 |
| avass | fungi: collections can't use hyphens? that seems arbitrary | 14:52 |
| *** ysandeep|away is now known as ysandeep | 14:53 | |
| fungi | avass: i think it's that they want them to mirror python module naming requirements | 14:53 |
| fungi | (you can't import a module with a hyphen in its name because that's a reserved character for an operator) | 14:54 |
| fungi | but yeah, this was all hotly debated in the ansible user community after the developers informed them that's how it was going to be | 14:55 |
| avass | yeah I guess so, but it's not python so it doesn't need to act like python | 14:55 |
| fungi | i don't recall the exact reason honestly, so it's entirely likely i've misremembered that explanation | 14:56 |
| mordred | collections introduce the ability to include module_utils and other things like that | 14:56 |
| mordred | and to use those things you have to import them - so pretty much all of the aspects of the naming wind up being expose in python import statements | 14:56 |
| *** sshnaidm|ruck is now known as sshnaidm|mtg | 14:57 | |
| mordred | https://opendev.org/openstack/ansible-collections-openstack/src/branch/master/plugins/modules/server.py#L454 | 14:57 |
| mordred | for instance | 14:57 |
| avass | yeah, that seems like it would make it hard to turn zuul-jobs into a collection | 14:59 |
| *** bolg has quit IRC | 15:01 | |
| *** harrymichal has joined #zuul | 15:06 | |
| *** rf0lc0 has joined #zuul | 15:09 | |
| *** dpawlik2 has joined #zuul | 15:12 | |
| *** yoctozepto2 has joined #zuul | 15:13 | |
| *** rfolco has quit IRC | 15:14 | |
| *** irclogbot_1 has quit IRC | 15:14 | |
| *** Goneri has quit IRC | 15:14 | |
| *** etp has quit IRC | 15:14 | |
| *** mnasiadka has quit IRC | 15:14 | |
| *** avass has quit IRC | 15:14 | |
| *** Goneri has joined #zuul | 15:14 | |
| *** mnasiadka_ has joined #zuul | 15:14 | |
| *** dpawlik4 has quit IRC | 15:14 | |
| *** sgw has quit IRC | 15:14 | |
| *** etp has joined #zuul | 15:14 | |
| *** yoctozepto has quit IRC | 15:15 | |
| *** yoctozepto2 is now known as yoctozepto | 15:15 | |
| *** sgw1 has joined #zuul | 15:16 | |
| corvus | mordred: i'd love exception handling in ansible -- like a way to say "include this role, but we expect this task to fail". because honestly, we can run everything but the last task of that role for testing. | 15:17 |
| corvus | i guess we could add a private "test only" variable... _netlify_publish_dry_run=true | 15:18 |
| corvus | and skip the task | 15:18 |
| corvus | that would at least get all the templating and stuff tested. might be worthwhile? | 15:18 |
| *** irclogbot_2 has joined #zuul | 15:19 | |
| mwhahaha | can someone point to me where the failure is on these jobs https://review.opendev.org/#/c/739061/ ? the logs point to they were all successful as far as i can see | 15:20 |
| corvus | mwhahaha: that's a good question. i don't have an answer yet, but will continue to look | 15:25 |
| mwhahaha | thanks, let me know if you figure it out :D | 15:26 |
| *** ysandeep is now known as ysandeep|away | 15:26 | |
| *** avass has joined #zuul | 15:28 | |
| AJaeger | corvus, mwhahaha I don't see the task "List current ipv4 rules" in the log files at all, did ansible got confused? | 15:32 |
| mwhahaha | is there a check for that somewhere? | 15:33 |
| mwhahaha | i didn't see persist-iptables invoked anywhere actually | 15:33 |
| mwhahaha | i did see TASK [Persist iptables rules] tho | 15:34 |
| mwhahaha | but that comes from multi-node-firewall? | 15:34 |
| *** harrymichal has quit IRC | 15:37 | |
| *** marios is now known as marios|out | 15:38 | |
| EmilienM | mwhahaha: yes and it seems to be called in pre/multinode https://opendev.org/zuul/zuul-jobs/src/branch/master/playbooks/multinode/pre.yaml#L18 | 15:38 |
| mwhahaha | yea but the patch was to persist-firewall instead of that one | 15:39 |
| mwhahaha | doesn't look like it's even exercised in those jobs | 15:39 |
| mwhahaha | rm nm the role is persistent-firewall | 15:40 |
| mordred | corvus: yeah - that's not a terrible idea | 15:40 |
| mordred | (the private test only variable) | 15:40 |
| corvus | mwhahaha: from the executor log: 2020-07-02 15:03:37,973 DEBUG zuul.AnsibleJob: [e: 0501b323688044368f6af4a72ed83ec3] [build: ea61ed0736f74ae3ac618dbc8e5b5b26] Ansible complete, result RESULT_UNREACHABLE code None | 15:41 |
| corvus | mwhahaha: oh wait, that may just be the cleanup playbook; nevermind | 15:41 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: scheduler: Fix event process abide hasUnparsedBranchCache argument https://review.opendev.org/739042 | 15:41 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: Fix branch name and project name for ref-updated event https://review.opendev.org/738320 | 15:41 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: WIP: Improve reconfigure for gerrit ref-updated & various enhancements https://review.opendev.org/739078 | 15:41 |
| corvus | mwhahaha: here we go: | 15:41 |
| corvus | 2020-07-02 15:00:37,954 DEBUG zuul.AnsibleJob.output: [e: 0501b323688044368f6af4a72ed83ec3] [build: ea61ed0736f74ae3ac618dbc8e5b5b26] Ansible output: b'ERROR! conflicting action statements: shell, retry' | 15:42 |
| mwhahaha | ah | 15:42 |
| corvus | 2020-07-02 15:00:37,955 DEBUG zuul.AnsibleJob.output: [e: 0501b323688044368f6af4a72ed83ec3] [build: ea61ed0736f74ae3ac618dbc8e5b5b26] Ansible output: b"The error appears to be in '/var/lib/zuul/builds/ea61ed0736f74ae3ac618dbc8e5b5b26/untrusted/project_0/opendev.org/zuul/zuul-jobs/roles/persistent-firewall/tasks/main.yaml': line 1, column 3, but may" | 15:42 |
| mwhahaha | it's retries | 15:42 |
| mwhahaha | not retry | 15:42 |
| corvus | i thought we had those errors showing up in the job output, but i guess not | 15:42 |
| * mwhahaha = not so smart | 15:42 | |
| openstackgerrit | Alex Schultz proposed zuul/zuul-jobs master: Make persist-iptables more robust https://review.opendev.org/739061 | 15:43 |
| mwhahaha | that would be helpful to have in output tho :D | 15:43 |
| corvus | yeah, and this is an easily reproducible test case, so we should be able to do that | 15:43 |
| *** saneax has quit IRC | 15:43 | |
| *** marios|out has quit IRC | 15:47 | |
| *** hamalq has joined #zuul | 15:57 | |
| *** hamalq_ has joined #zuul | 15:58 | |
| *** dmellado has joined #zuul | 16:00 | |
| *** hamalq has quit IRC | 16:02 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: WIP: Improve reconfigure for gerrit ref-updated & various enhancements https://review.opendev.org/739078 | 16:04 |
| *** dmellado has quit IRC | 16:20 | |
| *** dmellado has joined #zuul | 16:22 | |
| mwhahaha | corvus: can you look for the MODULE FAILURE in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a7f/739061/2/check/zuul-jobs-test-multinode-roles-debian-stretch/a7f2f40/job-output.txt that's actually the bug we're hitting so it would be helpful to know what the actual error is | 16:23 |
| mwhahaha | but there's no useful output other than 'MODULE FAILURE' in the job-output.txt so maybe there's a trace in the executor logs? | 16:23 |
| corvus | mwhahaha: do you have a link to the zuul build page? | 16:29 |
| mwhahaha | https://zuul.opendev.org/t/zuul/build/a7f2f40f7ebb4db6b275168013e4f1ec | 16:29 |
| corvus | mwhahaha: ah that. yeah, i checked that yesterday. there is no more information on the executor; this is absolutely all the information we have: https://zuul.opendev.org/t/zuul/build/a7f2f40f7ebb4db6b275168013e4f1ec/console#1/3/33/primary | 16:30 |
| corvus | mwhahaha: exit code -13 | 16:30 |
| mwhahaha | stupid modules | 16:30 |
| corvus | mwhahaha: 1 sec let me get you 1 more link | 16:31 |
| mwhahaha | ok let me see if i can find the -13 in ansible | 16:31 |
| corvus | mwhahaha: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/persistent-firewall/tasks/main.yaml#L3 | 16:31 |
| corvus | mwhahaha: so clearly this has been happening for a while and someone wanted to debug it | 16:32 |
| mwhahaha | ha, it fails later as currently written anyway | 16:32 |
| mwhahaha | so let me go see if i can figure out what's happening to give the -13 | 16:32 |
| corvus | mwhahaha: exactly, but this is the underlying error. | 16:32 |
| mwhahaha | thanks | 16:32 |
| corvus | (the later error is just an undefined var from this one) | 16:32 |
| corvus | mwhahaha: looks like clarkb wrote that comment 9 months ago: https://opendev.org/zuul/zuul-jobs/commit/3c60b35a1933c9a2a95b4439a4a6f276cc49970b | 16:33 |
| mwhahaha | yea i saw that. the failed_when false isn't helping | 16:33 |
| clarkb | ya I think the idea was maybe running under a shell would emit more infomration | 16:34 |
| corvus | agree; i'm not sure what that shouldn't be considered an error; seems like it should be an error, or we need a comment :) | 16:34 |
| clarkb | because maybe python subprocess in ansible for command was masking something | 16:34 |
| mwhahaha | the ansible modules usually end up eating the logs unfortunately | 16:34 |
| mwhahaha | unless it properly catches exceptions | 16:34 |
| *** nils has quit IRC | 16:35 | |
| corvus | in this case, could we really just be getting iptables-save returning -13 with no output? that's what it seems like to me | 16:35 |
| corvus | so maybe we need to look at iptables-save | 16:35 |
| mwhahaha | yea i was going to look to see if it returns -13 or does ansible via python return -13 | 16:35 |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: WIP fetch-coverage-output: direct link to coverage data https://review.opendev.org/739099 | 16:36 |
| mwhahaha | given that it's a MODULE FAILURE, it's likely something in the ansible execution and not the iptables-save command itself | 16:38 |
| corvus | hrm that's a good point | 16:38 |
| corvus | so maybe -13 is from python function in the module | 16:39 |
| mwhahaha | it's a good thing i've spent the last few weeks in the ansible internals | 16:41 |
| * mwhahaha dies a bit more inside | 16:41 | |
| corvus | (ftr, i did double check on ze04 to confirm that like the ones we looked up yesterday, there is no additional info in the logs for a7f2f40f7ebb4db6b275168013e4f1ec) | 16:45 |
| mwhahaha | so the -13 is likely comming from the command execution in ansible's ActionModule | 16:45 |
| mwhahaha | i found where that error message comes from and it's spitting out the rc from that function | 16:45 |
| corvus | mwhahaha: if we assume (this is a bit of a leap, but not uncommon with negative rc's) that it's errno, that would be eacces: permission denied | 16:50 |
| mwhahaha | yea i was wondering about that | 16:51 |
| mwhahaha | of course eacces can be spit out from a bunch of things | 16:51 |
| mwhahaha | specifically something in here https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/basic.py#L2603-L2670 | 16:53 |
| corvus | mwhahaha, clarkb, fungi: this happens right after "bring subnode bridge interface up" -- is it at all possible that if you run iptables-save right after that, an eacces could happen due to some bringup-related process still happening? | 16:54 |
| avass | corvus: there is exception handling, block, always, rescue :) | 16:54 |
| mwhahaha | it feels like it's a connectivity or internal communication problem. if iptables-save had the eaccess, it wouldn't be a module failure but rather a task failure | 16:55 |
| avass | oh, well except that you can't catch a specific error | 16:55 |
| corvus | avass, mordred: genius! there is an ansible_failed_task variable | 16:55 |
| corvus | avass: you can! :) | 16:55 |
| avass | oh well, you're welcome ) | 16:55 |
| corvus | avass: so we can verify that the failed task is exactly the task we expect to fail | 16:56 |
| fungi | i'd have to dig into what operations iptables-save calls, but it's entirely possible that reading the ruleset may be blocked temporarily while interfaces are added | 16:56 |
| avass | corvus: yeah just saw it. must have missed that part earlier | 16:56 |
| fungi | and yeah, narrowing it down further that way, if we can, would help | 16:56 |
| corvus | avass: it takes a village :) | 16:57 |
| mwhahaha | maybe it's getting eacess from the fcntl? https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/basic.py#L2618-L2620 | 17:00 |
| mwhahaha | it'll return eaces as an error according to https://pubs.opengroup.org/onlinepubs/009695399/functions/fcntl.html | 17:00 |
| mwhahaha | it's also not consistent and isn't specific to a target os (we see it on centos, that was a debian failure) | 17:01 |
| *** jcapitao has quit IRC | 17:05 | |
| corvus | mwhahaha: does the latest error mean the retry idea didn't work? | 17:06 |
| mwhahaha | yea | 17:07 |
| mwhahaha | because the module itself failed (probably in the fnctl) | 17:07 |
| corvus | i guess module failures are exempt from retrying? | 17:07 |
| corvus | ya | 17:07 |
| mwhahaha | and not the iptables-save command | 17:07 |
| mwhahaha | yea module retires are hard fails | 17:07 |
| mwhahaha | er modules failures are hard failures | 17:07 |
| mwhahaha | me no word well today | 17:07 |
| corvus | mwhahaha: we can try to turn on verbose mode; but we have to do it on all the executors and then turn it off real quick, otherwise disks fill up with logs | 17:08 |
| corvus | that might get us a traceback | 17:08 |
| mwhahaha | that being said, isn't that an IOError or an OSError? because that should end up with an 'Error Executing'... log line | 17:08 |
| mwhahaha | https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/basic.py#L2665-L2667 | 17:08 |
| mwhahaha | that might not be spit out unless you turn verbose on | 17:09 |
| avass | corvus, mwhahaha: module failures are exceptions rather than task failures. ignore_errors would work but not using failed_when | 17:09 |
| corvus | avass: actually failed_when can be used to ignore this error; it's retries that didn't do what we hoped | 17:10 |
| avass | okay that's strange | 17:10 |
| corvus | yeah, everything about this is strange :) | 17:10 |
| corvus | mwhahaha: want me to verbosify? | 17:10 |
| mwhahaha | we can try it | 17:10 |
| corvus | switching to #opendev | 17:10 |
| mwhahaha | and then recheck my patch to see if it triggers | 17:10 |
| *** jpena is now known as jpena|off | 17:14 | |
| *** bhavikdbavishi has joined #zuul | 17:24 | |
| *** bolg has joined #zuul | 17:27 | |
| *** LLIU82 has joined #zuul | 17:27 | |
| *** olaph has joined #zuul | 17:28 | |
| LLIU82 | hi, guys. I met an error on revoke sudo role. 2020-07-02 16:42:13.242101 | TASK [revoke-sudo : Check if zuul is sudoer] | 17:29 |
| LLIU82 | 2020-07-02 16:42:14.267118 | | 17:29 |
| LLIU82 | 16:42:14.561017", | 17:29 |
| LLIU82 | does it mean I set the sudo access right for the nodeset in a wrong way? | 17:31 |
| clarkb | LLIU82: are the job logs something that you can link to so that we can see more context? | 17:32 |
| clarkb | if not maybe you can use a paste service toshare a bit more of the log content? | 17:32 |
| avass | LLIU82: wrote to you on teams :) | 17:33 |
| LLIU82 | PRE-RUN END RESULT_NORMAL: [untrusted : opendev.org/zuul/zuul-jobs/playbooks/tox/pre.yaml@master] | 17:34 |
| LLIU82 | [revoke-sudo : Check if zuul is sudoer] | 17:34 |
| *** LLIU82 has quit IRC | 17:34 | |
| *** LLIU82 has joined #zuul | 17:38 | |
| LLIU82 | sorry. Since I paste too much. I was moved out of the chat | 17:39 |
| avass | LLIU82: yeah you might want to use something like: http://paste.openstack.org/ and link that instead | 17:39 |
| LLIU82 | PRE-RUN END RESULT_NORMAL: [untrusted : opendev.org/zuul/zuul-jobs/playbooks/tox/pre.yaml@master] | 17:40 |
| LLIU82 | [revoke-sudo : Check if zuul is sudoer] | 17:40 |
| LLIU82 | that general sudo access is actually revoked.] | 17:40 |
| LLIU82 | 2020-07-02 16:42:14.800262 | doc-web | "rc": 1, | 17:40 |
| LLIU82 | ignored: 0 | 17:40 |
| fungi | LLIU82: that looks like the revoke-sudo role is being executed on a system where zuul's user did not start with permission to call `sudo -n true` though the stdout/stderr fields from the task result would help confirm what happened with it | 17:46 |
| avass | fungi: yeah that's exactly it | 17:46 |
| fungi | i thought we had written it so that the "Check if zuul is sudoer" task was allowed to fail, and that was simply used to skip the removal tasks | 17:47 |
| avass | fungi: the user had sudo access but not from the /etc/sudoers.d/zuul file. in a dev environment | 17:47 |
| fungi | ahh | 17:47 |
| fungi | was it the "Prove that general sudo access is actually revoked." task which was failing then? | 17:48 |
| LLIU82 | Thanks for helping =D | 17:48 |
| avass | I believe so yep | 17:48 |
| LLIU82 | yes, exactly | 17:48 |
| fungi | it was hard to tell from those bits of logs without the full context in paste.openstack.org or somewhere | 17:48 |
| avass | fungi: but I guess I should get working on making that configurable through nodepool, since we really don't want to revoke sudo on static nodes :) | 17:49 |
| LLIU82 | sudo right is set in /etc/sudoers instead of /etc/sudoers.d/zuul | 17:50 |
| LLIU82 | so revoke failed | 17:50 |
| LLIU82 | avass suggested to stop the sudo right | 17:50 |
| LLIU82 | I am going to do further test | 17:51 |
| fungi | got it, i agree the revoke-sudo role makes some basic assumptions, but at least it failed safely and did not allow the job to proceed since it detected that it was unable to actually revoke sudo access | 17:51 |
| fungi | so i consider that a good test of the current design for the role, at least ;) | 17:52 |
| *** bolg has quit IRC | 17:53 | |
| avass | we don't use the /etc/sudoers.d/zuul file on static nodes just to make sure it fails instead of revoking sudo, and we keep a copy of zuul-jobs with some quickfixes but I'm trying to sync everything up so we can use zuul-jobs directly :) | 17:55 |
| avass | I think being able to configure a revoke_sudo variable in nodepool is the last thing that needs to be done | 17:55 |
| *** zenkuro has quit IRC | 18:00 | |
| *** bhavikdbavishi has quit IRC | 18:36 | |
| mnaser | avass: i tihnk revoke-sudo makes a lot of assumptions unfortunately right now | 18:42 |
| clarkb | well in this case it seems that the setup was intentionally done to prevent revoke sudo from working? | 18:44 |
| clarkb | reading scrollback I think the best option is to not run revoke-sudo on those static nodes | 18:44 |
| clarkb | one way to do that is remove the role from the node, another would be to have zuul static nodes without sudo in the first place | 18:44 |
| avass | yeah but it's more complicated than that | 18:45 |
| *** armstrongs has joined #zuul | 19:02 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: WIP: Improve reconfigure for gerrit ref-updated & various enhancements https://review.opendev.org/739078 | 19:05 |
| *** armstrongs has quit IRC | 19:09 | |
| *** harrymichal has joined #zuul | 19:14 | |
| fungi | mnaser: also the role actually did what it was supposed to: when it identified that it had not revoked sudo permission it raised a failure so the (potentially dangerous) job payload could not proceed | 19:15 |
| AJaeger | there's https://review.opendev.org/703065 to "improve remove-sudo", not sure whether that would have helped | 19:17 |
| avass | AJaeger: there's also: https://review.opendev.org/#/c/706248/ ;) | 19:18 |
| AJaeger | ;) | 19:18 |
| avass | but it's mostly that we need sudo access on static nodes for reasons and we're lazy so we want to re-use jobs from zuul-jobs without revoking sudo | 19:19 |
| *** wuchunyang has joined #zuul | 19:28 | |
| fungi | yeah, sounds more like there should be a nice switch you can flip to skip revoke-sudo | 19:32 |
| *** LLIU82 has quit IRC | 19:35 | |
| *** wuchunyang has quit IRC | 19:40 | |
| *** harrymichal has quit IRC | 19:41 | |
| *** harrymichal has joined #zuul | 19:55 | |
| *** hashar has quit IRC | 19:58 | |
| *** harrymichal has quit IRC | 20:06 | |
| *** wuchunyang has joined #zuul | 20:11 | |
| *** sshnaidm|mtg is now known as sshnaidm|afk | 20:27 | |
| *** wuchunyang has quit IRC | 20:38 | |
| *** wuchunyang has joined #zuul | 20:39 | |
| *** wuchunyang has quit IRC | 20:43 | |
| *** wuchunyang has joined #zuul | 20:45 | |
| *** y2kenny has joined #zuul | 21:12 | |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-coverage-output: direct link to coverage data https://review.opendev.org/739099 | 21:13 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: WIP: Improve reconfigure for gerrit ref-updated & various enhancements https://review.opendev.org/739078 | 21:18 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: multi-node-hosts-file: add ipv6 address if defined https://review.opendev.org/738952 | 21:26 |
| ianw | clarkb: ^ interesting on your thoughts on this one -- it came up in the graphite testing where i'm querying "graphite02.opendev.org" and relying on /etc/hosts to sort it out | 21:32 |
| ianw | you did something similar in the mirror scripts, poking at the host addresses | 21:32 |
| ianw | mirror test scripts i mean | 21:32 |
| *** bolg has joined #zuul | 21:36 | |
| clarkb | ianw: I'm not sure I recall the mirror context, but I can review that | 21:38 |
| clarkb | ianw: ok, there is a potential problem there/here. I think that may break on clouds like ovh where the cloud saiys "you've got an ipv6 address" but the hots has no way of configuring it bceaues its not in config drive or metadata service | 21:40 |
| clarkb | and they don't RA etc | 21:40 |
| clarkb | I think the idea is good, but I'm not sure how safe it is due to clouds | 21:41 |
| clarkb | things should still fallback to ipv4 eventually but that will be slow | 21:41 |
| *** bolg has quit IRC | 21:45 | |
| fungi | yeah, there are definitely clouds which report addresses in the api but don't provide any way for the instance to know about them (expecting an out of band service with access to query the api to add the configuration in the instance) | 21:58 |
| *** bolg has joined #zuul | 22:15 | |
| ianw | clarkb/fungi: will they appear in the nodepool vars though? | 22:21 |
| ianw | clarkb: also it looks like https://zuul.opendev.org/t/zuul/build/0aadd709bf7a4e22b23c1a832d8c1016/console caught the iptables problem on suse? | 22:23 |
| clarkb | I think they do appearin nodepool vars. Maybe the solution is to change that first | 22:25 |
| ianw | hrm i had a vague sense we filtered that out in the clouds.yaml | 22:27 |
| *** pabelanger has joined #zuul | 22:28 | |
| ianw | # OVH has a weird new ipv6 setup that we can't handle properly | 22:28 |
| ianw | # for now ignore ipv6 | 22:28 |
| ianw | force_ipv4: true | 22:28 |
| ianw | type stuff | 22:28 |
| clarkb | ya that forces the useof ipv4 but the ipv6 info is still there | 22:29 |
| pabelanger | hello, we are in the process of renaming all master branches to main for ansible-collections. I've automated most of the renaming process on github side, but dealing with some stuggles on the zuul side. I was wondering what people thought of the idea of a new zuul-merger command to expire repo state? Otherwise, if not, I can write an ansible-playbook to stop all zuul mergers, and delete the content | 22:30 |
| pabelanger | on disk | 22:30 |
| pabelanger | most of the issue is around github repos that are in zuul, but humans don't understand how the mergers work. So every so often tags or default branches get delete, and merged fall into a bad state | 22:31 |
| clarkb | pabelanger: I want to say branch renames should be picked up. The default branch doesnt matter much to mergers | 22:34 |
| clarkb | the scheduler tells the mergers what to merge and where iirc independent of any in repo state | 22:35 |
| clarkb | you may have a stale master branch on the mergersuntil you clear them out next though | 22:35 |
| pabelanger | http://paste.openstack.org/show/795512/ | 22:36 |
| pabelanger | is what I see, unless I delete the repo from disk | 22:36 |
| pabelanger | and zuul doesnt' seem to be able to recover from it | 22:36 |
| clarkb | aha | 22:37 |
| clarkb | its specifically HEAD changing | 22:37 |
| clarkb | for that we may want to check head and update it on repo reset | 22:38 |
| clarkb | pabelanger: I think we can do ^ safely always without human input | 22:39 |
| pabelanger | k, I'm not sure how to do that :) | 22:40 |
| clarkb | we'd need to query the remote HEAD value then update our local HEADs to match | 22:42 |
| clarkb | I am not sure of hiw to do that in gitpython but should be possible | 22:42 |
| pabelanger | k, maybe I'll check back next week for more help | 22:43 |
| pabelanger | I know people are off for july 4th | 22:43 |
| pabelanger | have to run now, thanks for info | 22:43 |
| *** tosky has quit IRC | 22:54 | |
| *** rlandy|ruck has quit IRC | 22:57 | |
| clarkb | pabelanger: git remote set-head -a origin is how to do it with cli tools | 22:59 |
| *** saneax has joined #zuul | 23:01 | |
| *** Goneri has quit IRC | 23:08 | |
| *** bolg has quit IRC | 23:32 | |
| *** hamalq_ has quit IRC | 23:52 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!