| @westphahl:matrix.org | corvus: question on https://review.opendev.org/c/zuul/zuul/+/838047 | 12:17 |
|---|---|---|
| @jpew:matrix.org | I'm seeing a case with Zuul 5.0.0 where we've updated a `job.vars` to add a new element to an array variable, but it's not using the new value in a periodic pipeline | 14:02 |
| @jpew:matrix.org | e.g. when I check `inventory.yaml` in the periodic job, it's still using the old value of the variable | 14:02 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 838047: Conditionally avoid refreshing jobs and builds from ZK https://review.opendev.org/c/zuul/zuul/+/838047 | 15:27 | |
| @jim:acmegating.com | swest: thanks, updated! | 15:59 |
| @jim:acmegating.com | tristanC: Clark swest thanks for the ansible reviews -- it looks like there's one important change that could use reviews -- adding ansible 5: https://review.opendev.org/837629 | 16:00 |
| @clarkb:matrix.org | corvus: yup I intend on working my way up the stack a bit more today. Hopefully to get through the whole thing | 16:02 |
| @jim:acmegating.com | i think we're probably at a good place to maybe restart opendev and make a (probably) final 5.x before we land the ansible stuff for 6.0... that sound good? | 17:59 |
| @clarkb:matrix.org | works for me | 18:02 |
| @clarkb:matrix.org | corvus: for testing this switch to build in the assemble script the comments note that zuul's js build is the main thing to watch out for. When I locally ran build against zuul it appears to have included all that js/html stuff. Do any of the zuul CI jobs run a dashboard and test/screenshot it using the docker iamges? maybe quickstart? Just thinking I could depends on and try to extra confirm it is happy that way | 18:06 |
| @jim:acmegating.com | Clark: i'm not sure about depends-on because of the tenant barrier... and i don't think the quick-start job verifies js (probably only api). so you might need to just build both things manually to test (or publish 'build' under a new tag) | 18:15 |
| @clarkb:matrix.org | corvus: ok what I can do is grab the archive copy of the test built image then tell zuul to build its image on that | 18:16 |
| @clarkb:matrix.org | and then confirm that there is js stuff in the installation | 18:16 |
| @clarkb:matrix.org | or something along those lines | 18:16 |
| @clarkb:matrix.org | locally I mean | 18:16 |
| @jim:acmegating.com | Clark: yeah -- though that made me realize you could change the FROM line in zuul's dockerfile to point to insecure-ci-registry. but local would be faster if that's an option. :) | 18:17 |
| @clarkb:matrix.org | ya I'm juggling a few things right now so wanted to see if there was an easy way. I'll try to swing back around and do the local thing when I have a chance | 18:19 |
| @fungicide:matrix.org | > <@jpew:matrix.org> I'm seeing a case with Zuul 5.0.0 where we've updated a `job.vars` to add a new element to an array variable, but it's not using the new value in a periodic pipeline | 18:53 |
| Did you "update" it by merging a change/PR/MR to adjust the configuration, or some other way? I think you're saying your periodic buidls are using an outdated version of the job configuration, but just trying to confirm that's what you meant. | ||
| @fungicide:matrix.org | > <@jpew:matrix.org> I'm seeing a case with Zuul 5.0.0 where we've updated a `job.vars` to add a new element to an array variable, but it's not using the new value in a periodic pipeline | 18:54 |
| * Did you "update" it by merging a change/PR/MR to adjust the configuration, or some other way? I think you're saying your periodic builds are using an outdated version of the job configuration, but just trying to confirm that's what you meant. | ||
| @jpew:matrix.org | We made a change to the variable and gated it, but the periodic (cron based) pipeline didn't see it | 18:55 |
| @mnaser:matrix.org | > <@jpew:matrix.org> We made a change to the variable and gated it, but the periodic (cron based) pipeline didn't see it | 18:56 |
| would be curious to see what the job info in the dashboard reports the variable.. or the dashboard doesnt report that.. i dont remember :) | ||
| @jpew:matrix.org | mnaser: I've only ever seen it in inventory.yaml, but i can poke around more | 18:57 |
| @mnaser:matrix.org | jpew: yeah the UI shows the vars -- https://zuul.opendev.org/t/vexxhost/job/ansible-role-openmanage-integration | 18:58 |
| @jpew:matrix.org | Ah let me check | 18:58 |
| @jpew:matrix.org | It's correct now, but I did a `full-reconfigure` to try and get us back working | 19:02 |
| @jpew:matrix.org | so that may have kicked it back out | 19:02 |
| @fungicide:matrix.org | > <@jpew:matrix.org> We made a change to the variable and gated it, but the periodic (cron based) pipeline didn't see it | 19:07 |
| and the periodic pipeline trigger is using the same branch of the repo as the branch where you updated the configuration? just want to make sure there's no branched configuraton (implicit branch matching) at work | ||
| @jpew:matrix.org | Yes, both on master | 19:07 |
| @fungicide:matrix.org | andthe periodic build was triggered after the configuration change merged, not before or around the same time and maybe queued waiting for a node assignment? | 19:09 |
| @fungicide:matrix.org | * and the periodic build was triggered after the configuration change merged, not before or around the same time and maybe queued waiting for a node assignment? | 19:09 |
| @jpew:matrix.org | fungi: The periodioc change was triggered several hours later | 19:09 |
| @fungicide:matrix.org | sounds likely to be a bug in that case. just asking since i know in opendev most of our tenants put periodic pipeline priority dead last and i've seen periodic jobs queued awaiting node assignments for over a day when resources are otherwise occupied by higher-priority pipelines | 19:14 |
| @fungicide:matrix.org | so that could have been a simple explanation for running with older configuration | 19:14 |
| @fungicide:matrix.org | jpew: if you can isolate the bit of the scheduler debug log where the merge event was reacted to, which should have resulted in a soft configuration update, maybe you'll find an error buried in there | 19:16 |
| @jpew:matrix.org | I'll see if I can reproduce it | 19:22 |
| -@gerrit:opendev.org- Zuul merged on behalf of Vasyl Vavrychuk: [zuul/zuul] 838166: doc: Add command_socket attribute in web section https://review.opendev.org/c/zuul/zuul/+/838166 | 19:50 | |
| @clarkb:matrix.org | if zuul/web/build exists you cannot build the docker image. Should the docker image rm web/build after copying it if present? | 20:32 |
| @clarkb:matrix.org | I ended up just doing a git clean -xfd in the repo to get moving again. I can see an argument for failing if not clean to remind you that producing a build from unclean sources might produce unexpected results | 20:33 |
| @jim:acmegating.com | i think either of those things would be a good improvement | 20:39 |
| @clarkb:matrix.org | ok once I've finished checking this assemble update locally I'll work on a patch for that | 20:46 |
| @clarkb:matrix.org | hrm this isn't as straightforward as I thought it might be. We don't want to install git until after bindep runs so that we can record git being installed as a bindep dep | 21:04 |
| @clarkb:matrix.org | but we need git to run `git clean -xfd | 21:05 |
| @clarkb:matrix.org | and we don't want assemble to run git clean because there may be out of tree content we actually do want like web/build | 21:05 |
| @iwienand:matrix.org | we're debugging some missing console logs in #openstack-infra that's looking more like a zuul issue collecting output | 22:50 |
| @iwienand:matrix.org | e.g. https://zuul.opendev.org/t/openstack/build/f90ff1b3cb484237873dbc8277543305/console | 22:50 |
| @clarkb:matrix.org | I want to say this came up before fwiw and is a known issue | 22:50 |
| @iwienand:matrix.org | does not show "Add centos-release-openstack for CentOS 9" from https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/838763/1/playbooks/wheel-cache/pre.yaml#39 | 22:51 |
| @iwienand:matrix.org | looking at the json output we can see hosts: is blank -> https://imgur.com/a/8Y4Wpxs | 22:52 |
| @iwienand:matrix.org | that's a "yum" call. but the same thing happens with "packages", e.g. | 22:53 |
| @iwienand:matrix.org | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/bindep/tasks/packages.yaml#L11 -> this step | 22:53 |
| @iwienand:matrix.org | also does not have content in the json, as above | 22:54 |
| @jim:acmegating.com | ianw: iirc, the yum module is unusual and doesn't include output in the usual way. | 22:55 |
| @iwienand:matrix.org | hrm, it's intereting that "package:" also falls into this then | 22:56 |
| @iwienand:matrix.org | i would have thought that would wrap things differently | 22:56 |
| @clarkb:matrix.org | I think package may call into yum on yum platforms? something like that | 22:56 |
| @clarkb:matrix.org | its possible that package works on apt platforms for this reason | 22:56 |
| @iwienand:matrix.org | yeah, i'm sure it does, just i would have thought it would wrap the output more generically | 22:57 |
| @iwienand:matrix.org | I also seem to remember having a discussion about displaying this output, but i don't think it was this issue specifically.... | 22:58 |
| @iwienand:matrix.org | https://review.opendev.org/c/zuul/zuul/+/734833 is what i'm thinking of | 23:01 |
| @iwienand:matrix.org | that is more about picking up failures. this is a different case of no output being recorded at all | 23:03 |
| @jim:acmegating.com | yeah, no output in json could suggest this is a different issue | 23:06 |
| @iwienand:matrix.org | given the impending merge of many ansible changes, is chasing this further at this point likely not worth it? | 23:08 |
| @iwienand:matrix.org | i'm guessing it must be somewhere in the zuul_json plugin | 23:08 |
| @jim:acmegating.com | ianw: hard to say, but i feel it's likely independent. no major changes to zuul_json were needed for ansible 5, so this could still be present when we upgrade (because it's a bug in zuul_json), or it could be fixed in ansible-core >= 2.12 because yum/package changed. | 23:10 |
| @iwienand:matrix.org | i should be able to run a local ansible test, with a recent-ish ansible and a simple playbook that just does a "yum:" and just put zuul_json in as a standalone plugin, right? | 23:13 |
| @iwienand:matrix.org | istr trying to debug these plugins before it was impossible to get debugging output, i think i ended up writing things out to a file in /tmp. but modulo that should be tractable | 23:14 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!