| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 00:19 | |
| - [zuul/zuul] 823587: Add some ZK debug scripts https://review.opendev.org/c/zuul/zuul/+/823587 | ||
| - [zuul/zuul] 823739: Offload FrozenJob secrets https://review.opendev.org/c/zuul/zuul/+/823739 | ||
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 823744: just for debuggin https://review.opendev.org/c/zuul/zuul/+/823744 | 02:57 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 823744: just for debuggin https://review.opendev.org/c/zuul/zuul/+/823744 | 07:01 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 823744: just for debuggin https://review.opendev.org/c/zuul/zuul/+/823744 | 08:13 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/nodepool] 823389: Export current tenant limit stats https://review.opendev.org/c/zuul/nodepool/+/823389 | 08:28 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 823782: Compress ZKObject data https://review.opendev.org/c/zuul/zuul/+/823782 | 10:18 | |
| @jim:acmegating.com | Kenny Ho: question on 823732 | 15:07 |
|---|---|---|
| @jpew:matrix.org | I just upgraded from zuul 4.3.0 -> 4.11.0 (and yes, I made a temporary stop at 4.6.0), but now I get an error trying to upload the logs in my jobs; I'm using the site_logs configuration shown here: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs but my jobs fail with: `The error was: 'site_logs' is undefined` on the line with `hosts: "{{ site_logs.fqdn }}"` | 16:08 |
| @jim:acmegating.com | jpew: depending on circumstances, that could be related to https://zuul-ci.org/docs/zuul/releasenotes.html#security-issues | 16:24 |
| @jpew:matrix.org | @corvus That was my suspicion... but I'm wasn't sure how to make it work again (the release notes were not clear to me :/ ) and I saw the config fragement in zuul-jobs was still the same | 16:26 |
| @jpew:matrix.org | Would something like | 16:31 |
| ``` | ||
| - hosts: localhost | ||
| tasks: | ||
| - set_fact: | ||
| site_logs_fqdn: "{{ hostvars['localhost'].site_logs.fqdn }}" | ||
| - hosts: "{{ site_logs_fqdn }}" | ||
| ... | ||
| ``` | ||
| work and be safe? | ||
| @jim:acmegating.com | i don't know; i'd probably start by making a local ansible-only reproducer so i didn't have to go through the whole zuul debug cycle. that release note should have the details for how variables are set with various "!unsafe" flags depending on how and where site_logs is deifned | 16:36 |
| @jpew:matrix.org | Ok, I _think_ this will work; the thing I'm unsure of is if it's safe to do.... I can't quite grok what the documentation is saying about when it is safe to do so and when it is not: | 16:44 |
| ``` | ||
| - hosts : "{{ hostvars['localhost'].site_logs.fqdn }}" | ||
| roles: | ||
| @jpew:matrix.org | * Ok, I _think_ this will work; the thing I'm unsure of is if it's safe to do.... I can't quite grok what the documentation is saying about when it is safe to do so and when it is not: | 16:44 |
| ``` | ||
| - hosts : "{{ hostvars['localhost'].site_logs.fqdn }}" | ||
| roles: | ||
| ``` | ||
| @jpew:matrix.org | If someone can help me by telling me if the above is "safe", I can update the zuul-jobs documentation :) | 16:50 |
| @gobi_g:matrix.org | How to run playbook in verbose mode? | 16:52 |
| @jim:acmegating.com | jpew: it's the workaround that the release note says is almost never safe to do. if we were to update the docs to suggest that, they would have to be full of caveats | 16:53 |
| @jim:acmegating.com | i think it would be better to update the docs to avoid using the secret fqdn in the hosts entry | 16:54 |
| @jpew:matrix.org | @corvus Duplicate the FQDN in another variable in the job? | 16:54 |
| @jim:acmegating.com | i don't know; like i said earlier, i'd need to make a reproducer to understand it fully | 16:55 |
| @jpew:matrix.org | @corvus I can reproduce it pretty easily (I think) | 16:56 |
| @jim:acmegating.com | jpew: if you'd like to dig into it more to understand it fully, here are some of the stories and changes: https://storyboard.openstack.org/#!/story/2008682 https://storyboard.openstack.org/#!/story/2008672 https://storyboard.openstack.org/#!/story/2008664 | 16:57 |
| @jim:acmegating.com | jpew: maybe just hard code the site? | 17:01 |
| @jim:acmegating.com | karthi: see the last paragraph of https://zuul-ci.org/docs/zuul/operation.html#id3 | 17:02 |
| @gobi_g:matrix.org | corvus: but there I can't find reference for setting verbose mode. | 17:08 |
| @avass:vassast.org | jpew: Is it possible to use include_role together with delegate_to instead of uploading logs as a separate play? | 17:10 |
| @avass:vassast.org | something like: | 17:12 |
| ``` | ||
| - hosts: localhost | ||
| tasks: | ||
| - include_role: | ||
| name: upload-logs | ||
| apply: | ||
| delegate_to: "{{ site_logs.fqdn }}" | ||
| @avass:vassast.org | * something like: | 17:13 |
| ``` | ||
| - hosts: localhost | ||
| roles: | ||
| - role: add-fileserver | ||
| fileserver: "{{ site_logs }}" | ||
| tasks: | ||
| - include_role: | ||
| name: upload-logs | ||
| apply: | ||
| delegate_to: "{{ site_logs.fqdn }}" | ||
| ``` | ||
| @clarkb:matrix.org | karthi: there is a command to toggle it. `zuul-executor verbose` then `zuul-executor unverbose`. https://zuul-ci.org/docs/zuul/operation.html#id3 | 17:13 |
| @clarkb:matrix.org | * karthi: there is a command to toggle it. `zuul-executor verbose` then `zuul-executor unverbose` when you are done collecting the extra verbose information. https://zuul-ci.org/docs/zuul/operation.html#id3 | 17:13 |
| @gobi_g:matrix.org | Zuul run playbook in verbose mode by default but when I used copy module it says changed. But in verbose it should show the details about stdout results. | 17:13 |
| @clarkb:matrix.org | > <@gobi_g:matrix.org> Zuul run playbook in verbose mode by default but when I used copy module it says changed. But in verbose it should show the details about stdout results. | 17:14 |
| The zuul console in the build report page should also include info like that for individual tasks | ||
| @gobi_g:matrix.org | Clark: so that console page won't show the verbose details? | 17:15 |
| @clarkb:matrix.org | > <@gobi_g:matrix.org> Clark: so that console page won't show the verbose details? | 17:17 |
| If you click on the individual tasks in the console they expand and should show you additional details | ||
| @gobi_g:matrix.org | Is there any way to display task details in verbose mode in console?. Example if I'm running copy module I want to see which file copied to which location like that without helping of debug. | 17:17 |
| @fungicide:matrix.org | looking at a copy task example, i don't see stdout included, though there is a fair amount of other detail: https://zuul.opendev.org/t/zuul/build/b083c5ba56f84fe4b5eb4e4622012840/console#3/0/40/ubuntu-focal | 17:17 |
| @fungicide:matrix.org | karthi: you're clicking on the magnifying glass icon next to the task status in the console tab, right? | 17:18 |
| @fungicide:matrix.org | in the example i linked, it does seem to show the source and destination arguments, at least | 17:23 |
| @gobi_g:matrix.org | fungi: https://zuul.opendev.org/t/zuul/stream/748ff6916fd749209922b41a61a2c532?logfile=console.log | 17:24 |
| I'm asking whether we can able to see module execution in verbose mode like ansible verbose mode -vvv | ||
| @jim:acmegating.com | karthi: it's not possible to see verbose mode in the live console stream | 17:25 |
| @jim:acmegating.com | karthi: however the console view here has a lot of details: https://zuul.opendev.org/t/zuul/build/4e6a166494114262834769ea14b8674d/console | 17:25 |
| @jim:acmegating.com | more details than the live log streaming, but still not quite as much as "-vvv" | 17:26 |
| @jim:acmegating.com | karthi: to see "-vvv" you need to run the command i linked in the docs. but that will only output to the zuul-executor debug log | 17:26 |
| @jim:acmegating.com | that's because -vvv may output sensitive information | 17:26 |
| @jim:acmegating.com | so it's really there only to help zuul operators debug tricky jobs | 17:27 |
| @jim:acmegating.com | Albin Vass: jpew if that works, that looks like a good solution | 17:27 |
| @gobi_g:matrix.org | Thanks for the details corvus | 17:28 |
| @gobi_g:matrix.org | Do we need to do any additional steps to avail that console option? Because I see "this build does not provide any console information" in my dashboard | 17:30 |
| @jpew:matrix.org | Albin Vass: OK, I'll have to try it out | 17:32 |
| @jim:acmegating.com | karthi: yeah you need the logs uploaded along with the manifest, like the `post-logs.yaml` playbook in https://zuul-ci.org/docs/zuul/tutorials/quick-start.html#configure-a-base-job | 17:38 |
| @avass:vassast.org | another option is to have a job inheritance layer that only runs: | 17:38 |
| ``` | ||
| - hosts: localhost | ||
| tasks: | ||
| - set_fact: | ||
| cacheable: true | ||
| site_logs_fqdn: "{{ hostvars['localhost'].site_logs.fqdn }}" | ||
| ``` | ||
| because that would be safe to do since anyone trying to override `site_logs.fqdn` would also remove the ssh key from the secret var, and from what I remember lookups wouldn't be executed in that set_fact unless you ran it twice (or maybe they wouldn't run at all?). | ||
| but I can't say that I like that solution :) | ||
| @jim:acmegating.com | me neither. that's a workaround for really advanced users; not something we should put in docs | 17:39 |
| @jim:acmegating.com | * me neither. that's a workaround for really advanced users; not something we should put in docs for something we expect widely used by new users | 17:39 |
| @gobi_g:matrix.org | corvus: thank you | 17:42 |
| @gobi_g:matrix.org | Where we have to define ansible.cfg file for the playbooks? Or is there any way to pass values to ansible.cfg in zuul? | 18:22 |
| @jim:acmegating.com | karthi: you can't define ansible.cfg in zuul. what do you want to change? | 18:25 |
| @gobi_g:matrix.org | Retry job using retry_file_enabled option | 18:32 |
| @jim:acmegating.com | karthi: zuul expects to retry the entire job in cases like that. if you need to set something like that, you might consider using zuul's ansible to run your own ansible, then you can set whatever options you want. opendev does that for its CD jobs. | 18:34 |
| @gobi_g:matrix.org | Oh okay corvus | 18:36 |
| @tristanc_:matrix.org | Adam Krpan (he/him/his): was able to spawn instance attached to multiple network with nodepool version 4.0.0, and it seems like this is no longer working with the version 4.1.0. Is this feature was ever supported? | 19:45 |
| @akrpan-pure:matrix.org | ^ specifically in 4.0.0 the interface order was deterministic and always picked the correct one for SSH, in 4.1.0 it's randomized and it's picking the wrong interface for SSH | 19:46 |
| @clarkb:matrix.org | If this is for openstack instances I think you are supposed to annotate the network config to indicate which is the accessible interface | 19:47 |
| @clarkb:matrix.org | is routable or public or something like that | 19:47 |
| @akrpan-pure:matrix.org | I tried doing it by clouds.yaml but if you know how to annotate the networks that would be fantastic, yeah. I'll search for that as well | 19:47 |
| @clarkb:matrix.org | Yes, I think it is via clouds.yaml | 19:48 |
| @akrpan-pure:matrix.org | Tried this: | 19:49 |
| ``` | ||
| clouds: | ||
| cinderci: | ||
| auth: | ||
| # credentials n stuff | ||
| user_domain_name: Default | ||
| project_domain_name: Default | ||
| identity_api_version: 3 | ||
| volume_api_version: 3 | ||
| floating_ip_source: 'nova' | ||
| regions: | ||
| - name: cinderci | ||
| values: | ||
| networks: | ||
| - name: public_mgmt | ||
| routes_externally: true | ||
| default_interface: true | ||
| - name: public_iscsi1 | ||
| routes_externally: false | ||
| - name: public_iscsi2 | ||
| routes_externally: false | ||
| ``` | ||
| @clarkb:matrix.org | nodepool is asking openstacksdk "give me the routable public address". I wonder if the update got a new version of the sdk and that is what chagned the behavior | 19:49 |
| @clarkb:matrix.org | I would try setting routes_externally: false on the second network | 19:49 |
| @clarkb:matrix.org | I'm not sure what the defaults are | 19:49 |
| @akrpan-pure:matrix.org | > <@clarkb:matrix.org> I would try setting routes_externally: false on the second network | 19:50 |
| It's already set, do you mean `default_interface: false`? | ||
| @clarkb:matrix.org | oh apparently I can scroll in that test block | 19:50 |
| @clarkb:matrix.org | I would expect that to work. And it not working is likely an openstacksdk bug | 19:50 |
| @akrpan-pure:matrix.org | Got it, cool to hear I'm not insane at least | 19:51 |
| @clarkb:matrix.org | https://opendev.org/zuul/nodepool/src/branch/master/nodepool/driver/openstack/handler.py#L188-L231 is the block of code that I suspect is failing | 19:54 |
| @clarkb:matrix.org | I believe interface_ip is meant to be ipv6 if available and not disabled via config drive and ipv4 otherwise. if you are able to inspecting the value of the public ipv4 and ipv6 values there may be helpful | 19:55 |
| @clarkb:matrix.org | (it is logged at a debug level in the launcher debug log) | 19:55 |
| @clarkb:matrix.org | * oh apparently I can scroll in that text block | 19:57 |
| @akrpan-pure:matrix.org | The `Node is running` line? | 20:03 |
| @clarkb:matrix.org | yes | 20:06 |
| @clarkb:matrix.org | I would expect interface_ip to == either one of the two public ipv4 or ipv6 addrs but if they are different that might be a clue to where this is occuring | 20:06 |
| @akrpan-pure:matrix.org | Well, the address is still wrong but I may have another smoking gun | 20:13 |
| @akrpan-pure:matrix.org | `region: None` | 20:13 |
| @akrpan-pure:matrix.org | If the region is `None`, seems reasonable that it wouldn't pull those settings | 20:13 |
| @akrpan-pure:matrix.org | * If the region is `None`, seems reasonable that it wouldn't pull those interface settings | 20:13 |
| @akrpan-pure:matrix.org | And that I know I can set, so I can try that and come back | 20:13 |
| @clarkb:matrix.org | sounds good | 20:13 |
| @akrpan-pure:matrix.org | Maybe the old openstack cluster we had automatically set default region | 20:14 |
| @akrpan-pure:matrix.org | * Maybe the old openstack cluster we had automatically set default region, or it was in the clouds file for it | 20:14 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!