| @serjossj:matrix.org | Hi all, I've an issue with my pipeline and want to know better if it's zuul-executor related and if my setting/pipeline is ok. Is this the proper channel to ask? | 08:47 |
|---|---|---|
| -@gerrit:opendev.org- Christian Mueller proposed: [zuul/nodepool] 865531: WIP: enable EC2 Spot usage in AWS https://review.opendev.org/c/zuul/nodepool/+/865531 | 11:38 | |
| @serjossj:matrix.org | need some help in how executor works. Does it restrict repo access to protected and PR branches for the project triggering the pipeline? | 11:43 |
| To explain better, here is the problem I encountered (i'm really not an executor exper and just know ZUUL since 1 year). | ||
| I've a main repository with multiple submodule. | ||
| - MAIN REPO | ||
| |_SUB #1 | ||
| |_SUB #2 | ||
| |_SUB #3 | ||
| I want to do the following: | ||
| - job #1: do some logic and create a branch (for main and submodules #1 and #2), whose name is "exported" with zuul_return to job #2 | ||
| - job #2: dependent from job #1. Execute a playbook to checkout the just created branch from job #1 and build. | ||
| What I see is that I cannot find the created branch just in the repository that started the pipeline, ie | ||
| Triggering both job from MAIN REPO, I get an error when checking out the created branch in MAIN REPO. | ||
| Triggering both job from SUB #2, I get an error when checking out the created branch in SUB #2. | ||
| What's more interesting is that, if I list all branches fro the repo above, where I've the error I only get protected branches. | ||
| I've tried to add remote again to the repo and re-fetching but nothing changes. | ||
| So my guess is that is related to the way executor or some other zuul component works. Is there a way to accomplish what I want to do? Previously job #1 and #2 were on the same job but now I need to add more jobs to the chain so, I'll have many jobs depending from job #1. | ||
| @serjossj:matrix.org | Thanks in advance for any idea/help | 11:43 |
| @clarkb:matrix.org | > <@serjossj:matrix.org> need some help in how executor works. Does it restrict repo access to protected and PR branches for the project triggering the pipeline? | 15:20 |
| > | ||
| > To explain better, here is the problem I encountered (i'm really not an executor exper and just know ZUUL since 1 year). | ||
| > I've a main repository with multiple submodule. | ||
| > | ||
| > - MAIN REPO | ||
| > |_SUB #1 | ||
| > |_SUB #2 | ||
| > |_SUB #3 | ||
| > | ||
| > I want to do the following: | ||
| > - job #1: do some logic and create a branch (for main and submodules #1 and #2), whose name is "exported" with zuul_return to job #2 | ||
| > - job #2: dependent from job #1. Execute a playbook to checkout the just created branch from job #1 and build. | ||
| > | ||
| > What I see is that I cannot find the created branch just in the repository that started the pipeline, ie | ||
| > | ||
| > Triggering both job from MAIN REPO, I get an error when checking out the created branch in MAIN REPO. | ||
| > Triggering both job from SUB #2, I get an error when checking out the created branch in SUB #2. | ||
| > | ||
| > What's more interesting is that, if I list all branches fro the repo above, where I've the error I only get protected branches. | ||
| > | ||
| > I've tried to add remote again to the repo and re-fetching but nothing changes. | ||
| > | ||
| > So my guess is that is related to the way executor or some other zuul component works. Is there a way to accomplish what I want to do? Previously job #1 and #2 were on the same job but now I need to add more jobs to the chain so, I'll have many jobs depending from job #1. | ||
| It's a holiday weekend for many of us so responses may be slow. | ||
| That said Zuul wants to operate on the git state as presented by the code review system. That (speculative) state is then configured for each job by zuul and your base jobs should synchronize it. Updating git state beyond that in one job isn't going to automatically reflect in a subsequent job of the same buildset. | ||
| My first thought is I would avoid this type of workflow. You want the git state to be consistent across jobs for reproduce ability and all that. | ||
| Separately, there are settings for handling repos with protected branches, including only considering protected branches. I don't use GitHub though so am not super familiar with them. You should check your connection settings for them. | ||
| @picog:matrix.org | Hi folks. | 15:59 |
| I'm trying to diagnose an occasional problem we get on our CI system. I think at times of heavy load, jobs error out with RETRY_LIMIT. | ||
| The problem is that I can't seem to get the output of the command that is failing. | ||
| I've done: ```zuul-executor keep``` and `zuul-executor verbose``` | ||
| But the log file that get created don't contain any useful output of the command | ||
| For example: | ||
| ``` | ||
| cat /var/lib/zuul/builds/999b55bd860b474ba3a89319a6e0506d/work/logs/job-output.txt | ||
| 2022-11-24 17:27:51.943644 | Job console starting... | ||
| 2022-11-24 17:28:59.232255 | Running Ansible setup... | ||
| ``` | ||
| Any suggestions on how I can get more info about what is going wrong? | ||
| @picog:matrix.org | * Hi folks. | 16:00 |
| I'm trying to diagnose an occasional problem we get on our CI system. I think at times of heavy load, jobs error out with RETRY\_LIMIT. | ||
| The problem is that I can't seem to get the output of the command that is failing. | ||
| I've done: `zuul-executor keep` and `zuul-executor verbose` | ||
| But the log file that get created don't contain any useful output of the command | ||
| For example: | ||
| ``` | ||
| cat /var/lib/zuul/builds/999b55bd860b474ba3a89319a6e0506d/work/logs/job-output.txt | ||
| 2022-11-24 17:27:51.943644 | Job console starting... | ||
| 2022-11-24 17:28:59.232255 | Running Ansible setup... | ||
| ``` | ||
| Any suggestions on how I can get more info about what is going wrong? | ||
| @picog:matrix.org | I do see, `Ansible timeout exceeded: 60`, so I know the command times out. | 16:03 |
| @picog:matrix.org | * I do see, `Ansible timeout exceeded: 60` in the log , so I know the command times out. | 16:03 |
| @picog:matrix.org | Ah, just noticed the holiday message above. Happy thanksgiving to those who celebrate it :D | 16:08 |
| @clarkb:matrix.org | > <@picog:matrix.org> Hi folks. | 17:05 |
| > I'm trying to diagnose an occasional problem we get on our CI system. I think at times of heavy load, jobs error out with RETRY\_LIMIT. | ||
| > The problem is that I can't seem to get the output of the command that is failing. | ||
| > I've done: `zuul-executor keep` and `zuul-executor verbose` | ||
| > But the log file that get created don't contain any useful output of the command | ||
| > For example: | ||
| > | ||
| > ``` | ||
| > cat /var/lib/zuul/builds/999b55bd860b474ba3a89319a6e0506d/work/logs/job-output.txt | ||
| > 2022-11-24 17:27:51.943644 | Job console starting... | ||
| > 2022-11-24 17:28:59.232255 | Running Ansible setup... | ||
| > ``` | ||
| > | ||
| > Any suggestions on how I can get more info about what is going wrong? | ||
| zuul-executor keep will keep the workspace. zuul-executor verbose enables ansible -vvv on the ansible runs but this doesn't show up in the job-output butrather the executor log. Grep for the build uuid in that log file and you should get very verbose ansible information that hopefully points at the issue | ||
| @clarkb:matrix.org | Note RETRY_LIMIT indicates the job is being retried. THe two major reasons for this are failures in pre-run whcih are always retryable or a failure due to ansible exit code 4 which indicates a connectivity problem. | 17:05 |
| @picog:matrix.org | > <@clarkb:matrix.org> zuul-executor keep will keep the workspace. zuul-executor verbose enables ansible -vvv on the ansible runs but this doesn't show up in the job-output butrather the executor log. Grep for the build uuid in that log file and you should get very verbose ansible information that hopefully points at the issue | 17:11 |
| When you say grep the logfile, this is would be journalctl for the zuul-executor service? | ||
| @clarkb:matrix.org | > <@picog:matrix.org> When you say grep the logfile, this is would be journalctl for the zuul-executor service? | 17:12 |
| Yes, the log for the zuul-executor service wherever that is stored | ||
| @picog:matrix.org | Unfortunately mine contains not info except for the: Ansible timeout exceeded: 60 | 17:13 |
| @picog:matrix.org | * Unfortunately mine contains no info except for the: Ansible timeout exceeded: 60 | 17:13 |
| @clarkb:matrix.org | I wonder if you need debug logging enabled | 17:13 |
| @clarkb:matrix.org | I don't remember if Ansible is logged at a debug level in the executor and I don't have easy access to check today but that would be what I would check next | 17:14 |
| @picog:matrix.org | I actually tried for a while to turn that on and then gave up with a small source code hack to print the command :D | 17:14 |
| @clarkb:matrix.org | There is a command line flag I think | 17:14 |
| @clarkb:matrix.org | https://opendev.org/zuul/zuul/src/branch/master/zuul/cmd/__init__.py#L196 yes there is | 17:16 |
| @clarkb:matrix.org | Use the -d or --debug flags looks like | 17:22 |
| @picog:matrix.org | I think my service starts with that by default, but I see in the yaml config file there are different handlers | 17:24 |
| @picog:matrix.org | One logs to /var/log/zuul/executor.log, and that contains useful info! | 17:24 |
| @picog:matrix.org | This is really weird: | 17:27 |
| ``` | ||
| b'ansible 2.8.7' | ||
| b' config file = /var/lib/zuul/builds/db80b8c797424b058285604082ca2ba5/ansible/setup_playbook/ansible.cfg' | ||
| b" configured module search path = ['/var/lib/zuul/ansible/2.8/zuul/ansible/library']" | ||
| b' ansible python module location = /usr/lib/zuul/ansible/2.8/lib/python3.6/site-packages/ansible' | ||
| b' executable location = /usr/lib/zuul/ansible/2.8/bin/ansible' | ||
| b' python version = 3.6.8 (default, Aug 7 2019, 17:28:10) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]' | ||
| b'Using /var/lib/zuul/builds/db80b8c797424b058285604082ca2ba5/ansible/setup_playbook/ansible.cfg as config file' | ||
| b"host_list declined parsing /var/lib/zuul/builds/db80b8c797424b058285604082ca2ba5/ansible/setup-inventory.yaml as it did not pass it's verify_file() method" | ||
| b"script declined parsing /var/lib/zuul/builds/db80b8c797424b058285604082ca2ba5/ansible/setup-inventory.yaml as it did not pass it's verify_file() method" | ||
| b'Parsed /var/lib/zuul/builds/db80b8c797424b058285604082ca2ba5/ansible/setup-inventory.yaml inventory source with yaml plugin' | ||
| b'[WARNING]: Skipping plugin (/usr/lib/python3.6/site-' | ||
| b'packages/ara/plugins/actions/ara_read.py) as it seems to be invalid: module' | ||
| b"'ansible.plugins.callback./usr/lib/python3.6/site-" | ||
| b"packages/ara/plugins/actions/ara_read' has no attribute 'CallbackModule'" | ||
| b'[WARNING]: Skipping plugin (/usr/lib/python3.6/site-' | ||
| b'packages/ara/plugins/actions/ara_record.py) as it seems to be invalid: module' | ||
| b"'ansible.plugins.callback./usr/lib/python3.6/site-" | ||
| b"packages/ara/plugins/actions/ara_record' has no attribute 'CallbackModule'" | ||
| ``` | ||
| @picog:matrix.org | Let me try format is nicer and put on a pastebin | 17:29 |
| @picog:matrix.org | Perhaps there is nothing of interest there and ansible simply timed out network wise | 17:39 |
| @vlotorev:matrix.org | Hi, do I correctly understand that specifying job.branches in abstract job doesn't make sense as this field is not inherited? | 18:20 |
| @nhicher:matrix.org | Hello, I'd like to know if you plan to release new version of grafyaml on pypi, for 0.0.8 doesn't work with grafana 8 and 9, thanks =) | 19:25 |
| -@gerrit:opendev.org- Guillaume Chauvel proposed: [zuul/zuul] 864828: tutorial: Update gerrit configuration to gerrit 3.7 https://review.opendev.org/c/zuul/zuul/+/864828 | 20:02 | |
| @iwienand:matrix.org | > <@nhicher:matrix.org> Hello, I'd like to know if you plan to release new version of grafyaml on pypi, for 0.0.8 doesn't work with grafana 8 and 9, thanks =) | 22:49 |
| sure, i just tagged a 0.0.9. I guess we're mostly focused on running from git, but we can tag as needed | ||
| @iwienand:matrix.org | also probably better to bring up in #opendev channel, it's more that place that would be watched for requests (there's a lot of cross over of people, but it's not really a zuul project) | 22:54 |
| @nhicher:matrix.org | ianw: ok thanks, I will go on opendev next time =) | 23:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!