Monday, 2022-12-19

-@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 866863: Store pause and resume events on the build and report them https://review.opendev.org/c/zuul/zuul/+/86686309:17
-@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 866863: Store pause and resume events on the build and report them https://review.opendev.org/c/zuul/zuul/+/86686309:30
@raaar:matrix.org> <@newbie23:matrix.org> Got it. So provided the specific merge method is allowed, configured within GH, then is the zuul api call that will decide the merge type (e.g. squash merge)11:10
Hi, just wanted to follow up on this -- we've recently enable squash-merging for a number of sub-projects that are hooked via submodules to a base-project (creating a virtual monorepo). We have some dedicated logic to bump all submodules in the base repo after any PR has been merged as a part of the post pipeline in all subproject repos. However, we see that the state synced from the executor contains a "merge" commit instead of the squash commit that was actually created on GitHub. As a result we end up pushing effectively pushing a dangling reference to a non-existing commit.
Is there a way to make sure the repo state on the executor is up-to-date with the remote when running the post pipeline?
@raaar:matrix.org> <@newbie23:matrix.org> Got it. So provided the specific merge method is allowed, configured within GH, then is the zuul api call that will decide the merge type (e.g. squash merge)11:14
* Hi, just wanted to follow up on this -- we've recently enable squash-merging for a number of sub-projects that are hooked via submodules to a base-project (creating a virtual monorepo). We have some dedicated logic to bump all submodules in the base repo after any PR has been merged as a part of the post pipeline in all subproject repos. However, we see that the state synced from the executor contains a "merge" commit instead of the squash commit that was actually created on GitHub. As a result we end up pushing a dangling reference to a non-existing commit.
Is there a way to make sure the repo state on the executor is up-to-date with the remote when running the post pipeline?
@raaar:matrix.org * Hi, just wanted to follow up on this -- we've recently enabled squash-merging for a number of sub-projects that are hooked via submodules to a base-project (creating a virtual monorepo). We have some dedicated logic to bump all submodules in the base repo after any PR has been merged as a part of the post pipeline in all subproject repos. However, we see that the state synced from the executor contains a "merge" commit instead of the squash commit that was actually created on GitHub. As a result we end up pushing a dangling reference to a non-existing commit.11:22
Is there a way to make sure the repo state on the executor is up-to-date with the remote when running the post pipeline?
@raaar:matrix.orgIn case it helps, we use "closed" action of the pull_request event as the trigger to kick off the post pipeline13:39
@g_gobi:matrix.orgHi,13:51
I'm seeing this error recently in my zuul setup.
@g_gobi:matrix.orgweb_1        | 2022-12-19 13:49:32,445 ERROR zuul.web:   Traceback (most recent call last):13:52
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/site-packages/zuul/web/__init__.py", line 134, in received_message
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: self._streamLog(req)
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/site-packages/zuul/web/__init__.py", line 173, in _streamLog
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: self.streamer = LogStreamer(
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/site-packages/zuul/web/__init__.py", line 194, in __init__
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: self.finger_socket = socket.create_connection(
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/socket.py", line 787, in create_connection
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: for res in getaddrinfo(host, port, 0, SOCK_STREAM):
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
web_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: socket.gaierror: [Errno -2] Name or service not known
@g_gobi:matrix.org * ```13:54
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: Traceback (most recent call last):
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/site-packages/zuul/web/**init**.py", line 134, in received\_message
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: self.\_streamLog(req)
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/site-packages/zuul/web/**init**.py", line 173, in \_streamLog
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: self.streamer = LogStreamer(
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/site-packages/zuul/web/**init**.py", line 194, in **init**
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: self.finger\_socket = socket.create\_connection(
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/socket.py", line 787, in create\_connection
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: for res in getaddrinfo(host, port, 0, SOCK\_STREAM):
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: for res in \_socket.getaddrinfo(host, port, family, type, proto, flags):
web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: socket.gaierror: \[Errno -2\] Name or service not known
```
Any idea on how to resolve this?
-@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/nodepool] 868090: Adapt load-test to new zk structure https://review.opendev.org/c/zuul/nodepool/+/86809014:23
@clarkb:matrix.org> <@g_gobi:matrix.org> ```16:13
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: Traceback (most recent call last):
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/site-packages/zuul/web/**init**.py", line 134, in received\_message
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: self.\_streamLog(req)
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/site-packages/zuul/web/**init**.py", line 173, in \_streamLog
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: self.streamer = LogStreamer(
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/site-packages/zuul/web/**init**.py", line 194, in **init**
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: self.finger\_socket = socket.create\_connection(
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/socket.py", line 787, in create\_connection
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: for res in getaddrinfo(host, port, 0, SOCK\_STREAM):
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: for res in \_socket.getaddrinfo(host, port, family, type, proto, flags):
> web\_1 | 2022-12-19 13:49:32,445 ERROR zuul.web: socket.gaierror: \[Errno -2\] Name or service not known
> ```
>
> Any idea on how to resolve this?
>
This looks like it is trying to make a connection to the finger server that will stream the data off the test node and DNS resolution isn't working for that name.
@g_gobi:matrix.orgClark: I checked the logs Looks like the scheduler received the event and completed the node assignment and triggered the executor client but the executor not started the request.16:20
It stucked in `scheduler_1 | 2022-12-19 16:15:30,557 INFO zuul.ExecutorClient: xxxx`
I didn’t got any logs after this.
@clarkb:matrix.orgDo you have debug logging enabled on the executor? If not maybe enable that to see if there is any more logging via the more verbose logs16:25
@g_gobi:matrix.orgCan you please let me know the steps to enable `debug` log?16:53
@clarkb:matrix.orgtdlaw: if you run the zuul-executor process with `-d` or `--debug` it should log at a debug level17:12
@clarkb:matrix.orgcorvus: for the nox stuff, I'm thinking I'll get around to writing some readme content and job descriptions then maybe we should look at landing changes to add jobs? Or do you think it would be better to refine with zuul first (looks like a couple of things aren't happy yet) before we commit to interfaces in zuul-jobs?17:16
@jim:acmegating.comClark: do you think we should include siblings in the first round?17:19
@clarkb:matrix.orghrm ya siblings and inline comment support would be the big missing pieces. I think if we make it clear that is a regression compared to the tox role its probably fine to not include? But I could see waitong to include it just to avoid unexpected problems17:20
@g_gobi:matrix.org> <@clarkb:matrix.org> tdlaw: if you run the zuul-executor process with `-d` or `--debug` it should log at a debug level17:21
I'm running it in docker setup. Do we have any env variable to enable debug log?
@clarkb:matrix.orgtdlaw: I don't think so. You can modify the command that is executed in the container though17:23
@clarkb:matrix.orgthe quickstart docker-compose.yaml file does this if you need an example17:24
@g_gobi:matrix.org> <@clarkb:matrix.org> the quickstart docker-compose.yaml file does this if you need an example17:25
I'm also using that only. Looks like we need to update the command in image itself.
@clarkb:matrix.orgtdlaw: no, the docker-compose file in quickstart modifies the command, You should need to modify it to include the -d17:26
@clarkb:matrix.orgit doesn't modify it to set debugging, but it does modify it. I'm saying you can modify it further17:26
@g_gobi:matrix.orgThanks. I'll check that one.17:31
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 868131: Track object versions in the Buildset object https://review.opendev.org/c/zuul/zuul/+/86813122:08
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 868002: Add nox role and some simple jobs https://review.opendev.org/c/zuul/zuul-jobs/+/86800222:11
@clarkb:matrix.orgcorvus: ^ that adds some docs and a siblings implementation that I haven't tested yet (I just rechecked my test bindep change but that won't exercise siblings super well I don't think)22:12
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 868002: Add nox role and some simple jobs https://review.opendev.org/c/zuul/zuul-jobs/+/86800222:17
@clarkb:matrix.orgcorvus: any chance you might have time to look at the inline comment stuff? I think you implemented that originalyl and may grok it better22:19
@jim:acmegating.comClark: yes, i was just about to check in on that now that i've come up for air, and i can look into comments, thanks!22:23
@clarkb:matrix.orgI'm going to push another ps here shortly as somethign is wrong in mapping the envs with siblings22:23
@clarkb:matrix.orgcorvus: by the way I think there is a bug in the tox siblings implementation where it will only do siblings in the first env found if multiple envs are selected. The nox implementation corrects this but if we agree that is an issue we should port the fix to tox too22:31
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 868002: Add nox role and some simple jobs https://review.opendev.org/c/zuul/zuul-jobs/+/86800222:32
@jim:acmegating.comClark: agree that sounds like a bug22:32
@clarkb:matrix.orgspecifically it does `changed = changed or install_siblings` and python or logic should short circuit so the install_siblings function isn't called more than once22:33
@clarkb:matrix.orgwhile I don't think that bindep is actually exercising siblings it does seem to run through the isblings code enough to have caught these problems with it. That is reassuring22:36
@jim:acmegating.com`nox-linters                               DISK_FULL in 9m 13s  `  oO22:36
@clarkb:matrix.orgya I haven't had time to look at that.22:37
@clarkb:matrix.orgrunning flake8 on bindep doesn't do that, but it may just be a matter of scale or noxfile.py implementation or something22:37
@jim:acmegating.comi'll be running that locally to test this out, so maybe i'll see what it is :)22:37
@clarkb:matrix.org++22:37
@clarkb:matrix.orgok that last patchset doesn't blow up on silly problems. I think it is reviewable and testable in more depth using actual siblings.22:40
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 867057: WIP: play with noxfile https://review.opendev.org/c/zuul/zuul/+/86705722:45
@jim:acmegating.comthe disk_full is probably due to not excluding .nox.  fix included in latest ps ^22:45
@jim:acmegating.com(i'm moving the flake8 config to setup.cfg so we're not relying on tox.ini for that)22:45
@clarkb:matrix.org++ I did that for bindep too22:47
@clarkb:matrix.orgcorvus: also I had to update that zuul change to not try and use a zuul specific docs jobs. I think that is done for publication reasons but seems to rely on some sort of opendev/base-jobs stuff I haven't fully processed yet. I suspect the path here is land the zuul-jobs change, update base-jobs, then update zuul once we getto that point22:49
@clarkb:matrix.orgI've got it running a generic docs job now just to exercise the sphinx build aspect (it also captures a site preview via swift logs)22:49
@clarkb:matrix.orghrm and now it is failing to find myslq beacuse we are skipping binary install dependencies for some reaosn. I did update the condition on the unittests parent job for that in a recent patchset. Maybe I got it wrong?22:51
@clarkb:matrix.orgthe actual unittests don't seem to have this problem and they all share a common job ancestry. Odd22:52
@clarkb:matrix.orgah you set tox_install_bindep to false on that job. I wonder why22:54
@clarkb:matrix.orgBut its failing beacuse it tries to do regular test setup which requires mysql22:55
@jim:acmegating.comClark: re tox_install_bindep -- i thought i just matched the old tox job behavior22:59
@jim:acmegating.comoh you mean it's set in base-jobs?23:01
@clarkb:matrix.orgcorvus: no its set on nox-linters as a variant in the zuul project config23:01
@clarkb:matrix.orgbut something in that path is looking to interact with mysql via the test setup script23:01
@jim:acmegating.comoh i get it.  i missed the context switch from docs to linters.  23:01
@clarkb:matrix.orgwe might need to compare a tox run to a nox run to see why they differ23:02
@clarkb:matrix.orgthe unittests are running, but I'm not sure we are properly suppressing warning output as there seems to be a lot of that23:02
@jim:acmegating.com> <@clarkb:matrix.org> the unittests are running, but I'm not sure we are properly suppressing warning output as there seems to be a lot of that23:03
Clark: i think that may be happening with tox now too, we should double check that
@clarkb:matrix.orgah23:03
@jim:acmegating.com> <@clarkb:matrix.org> we might need to compare a tox run to a nox run to see why they differ23:03
i don't think we should run test-setup.sh in linters
@clarkb:matrix.orgI see it23:04
@jim:acmegating.comClark: https://zuul.opendev.org/t/zuul/build/935c820fab9e41908b6db39f69cca6ee/console#1/0/3/ubuntu-jammy old run for comparison23:04
@clarkb:matrix.orgits a bug in my zuul-jobs change. I'll work on a fix23:04
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 868002: Add nox role and some simple jobs https://review.opendev.org/c/zuul/zuul-jobs/+/86800223:05
@clarkb:matrix.orgthat should fix it23:05
@jim:acmegating.comClark: new subject: most of the work in the tox line comment handling is really just "flake8 and sphinx output parsing", and i don't think it has much to do with tox.  should we (a) symlink that python library into nox so they share the same code?  (b) copy it so that if tox or nox introduce their own output decorations in the future the implementations can diverge?23:06
@clarkb:matrix.orgcorvus: I copied the siblings stuff even though some of the low level deps management is the same because the high level interfaces are different (for selecting sessions in particular). Symlinking is probably fine in this case if nothing changes between them?23:08
@jim:acmegating.comokay, i'll symlink and if they ever diverge in the future we can turn that symlink into a copy23:08
@clarkb:matrix.orgI was also able to simplify things with nox for two reasons. The first is nox doesn'y run under python2 at all and neither do our executors so the old python2 stuff could be dropped. And since nox doens't have an explicit config dump api like tox we basically just hae to use the convention of where to find the env dir rather than looking it up out of config23:10
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 864235: Report a config error for unsupported merge mode https://review.opendev.org/c/zuul/zuul/+/86423523:11
@clarkb:matrix.orgcorvus: now the linters target fails when trying to build the libre2 wheel because libre2-dev is not installed (because we skip the insallation). I think that is fine and instead we need to update the linters sessions to not install that stuff and just install the linter tools23:13
@clarkb:matrix.orgleft a note inline on that one23:14
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 868134: Add inline comment support to nox https://review.opendev.org/c/zuul/zuul-jobs/+/86813423:15
@jim:acmegating.comack, will incorporate into next ps23:15
@clarkb:matrix.orglooks like the zuul-client test job exercises siblings and it mostly worked. The new issue is that we need to pass `--no-install` after we configure siblings otherwise it redoes the installations.23:16
@clarkb:matrix.orgI can edit that after you push your ps.23:16
@jim:acmegating.comClark: hrm, i think we do expect zuul to be installed for docs, since we run zuul commands and capture output23:17
@jim:acmegating.comoh linters not docs, sorry, brain fart23:17
@clarkb:matrix.orgalso we should add -v to the nox invocations as it adds a bunch of helpful info like what versions of deps are being installed23:19
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 867057: WIP: play with noxfile https://review.opendev.org/c/zuul/zuul/+/86705723:19
@jim:acmegating.comClark: ^ that includes an intentional pep8 failure23:19
@clarkb:matrix.orgcorvus: there is an issue in the zuul-jobs change with the old tox name not being findable.23:24
@jim:acmegating.comClark: i think we need an __init__.py in `nox/roles`23:24
@jim:acmegating.com * Clark: i think we need an `__init__.py` in `nox/roles`23:25
@jim:acmegating.comi think that needs to go in https://review.opendev.org/86800223:25
@clarkb:matrix.orgin the library dir you mean?23:25
@jim:acmegating.comnope you got that one.  i mean also one level up23:25
@jim:acmegating.comi think that's why py27 is failing23:25
@clarkb:matrix.orggotcha I can add that. Give me a minute I was adding the other bits and I think there is naother issue too that I'm looking at23:26
@clarkb:matrix.orgbut I can do that update shortly23:26
@jim:acmegating.com> <@clarkb:matrix.org> corvus: there is an issue in the zuul-jobs change with the old tox name not being findable.23:27
maybe symlink won't work and we need to copy
@clarkb:matrix.orghttps://zuul.opendev.org/t/zuul/build/2e91fc49afa04218b0842d71c0bd7f2f/log/job-output.txt#551 is the issue23:27
@jim:acmegating.comoh nope i see it sorry23:28
@clarkb:matrix.orgoh I don't yet23:29
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 868134: Add inline comment support to nox https://review.opendev.org/c/zuul/zuul-jobs/+/86813423:29
@jim:acmegating.comi forgot to git add the symlink23:29
@clarkb:matrix.orgah23:29
@clarkb:matrix.orggive me a sec and I'll address the other things in the parent and then we can recheck the zuul change23:30
@jim:acmegating.comkk23:30
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 868002: Add nox role and some simple jobs https://review.opendev.org/c/zuul/zuul-jobs/+/86800223:31
-@gerrit:opendev.org- Clark Boylan proposed on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-jobs] 868134: Add inline comment support to nox https://review.opendev.org/c/zuul/zuul-jobs/+/86813423:31
@clarkb:matrix.org868002 has the `__init__.py` now and I added -v by default and have it run with `--no-install` when we run under siblings during the last invocation23:32
@jim:acmegating.comi don't think `-v` will interfere with the comment parsing, so should be good there.23:34
@clarkb:matrix.orglinting fails on 868002 due to some unused imports. Shouldn't affect how things run for this pass though so I'll avoid updating to fix that23:37
@clarkb:matrix.orgoh heh all theimports come after the doc string that is why I missed them earlier23:38
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 865809: Document file-matcher behavior for refs w/o files https://review.opendev.org/c/zuul/zuul/+/86580923:40
@jim:acmegating.comhttps://zuul.opendev.org/t/zuul/build/7513185da23c414c87aef31f3e612ff9/console#3/0/12/ubuntu-jammy looks promising23:41
@jim:acmegating.comClark: i think if you want to update the zuul-jobs stack now is a good time23:41
@jim:acmegating.comi do want to see that comment make it all the way back to zuul, but let's not wait for the unit test run to finish before you update zuul-jobs23:42
@jim:acmegating.comthe 2 jobs failing on 868134 are also due to the imports, so hopefully that one is gtg23:43
@clarkb:matrix.orgwell I'm looking at the zuul client job and trying to make sense of the isblings stuff first. I think my --no-install may have been ignored?23:43
@clarkb:matrix.orgbut ya I'm working on sorting that out and fixing the linters23:43
@jim:acmegating.comClark: might want to add the --no-install to the 'emit nox command' task?23:46
@clarkb:matrix.orgya23:46
@jim:acmegating.com(just to aid debugging -- that confused me)23:46
@clarkb:matrix.orgI'm also going to double check it works locally23:46
@jim:acmegating.comClark: i haven't checked this, but we may need "-R" or something like it23:47
@clarkb:matrix.orgI have confirmed it seems towork with my local nox installation23:47
@clarkb:matrix.orgoh I'm testing in bindep and I set the reuse existing venvs globally on bindep's config23:48
@clarkb:matrix.orgI'm guessing zuul doesn't have that and I should set that flag here just to be extra sure for CI23:48
@jim:acmegating.comah yeah, i think it would be good not to rely on that setting in the noxfile23:49
@clarkb:matrix.orgya the role should be generic enough to handle it either way23:50
@jim:acmegating.comi'll go ahead and set that on zuul though since i think it's desirable23:51
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 868002: Add nox role and some simple jobs https://review.opendev.org/c/zuul/zuul-jobs/+/86800223:51
-@gerrit:opendev.org- Clark Boylan proposed on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-jobs] 868134: Add inline comment support to nox https://review.opendev.org/c/zuul/zuul-jobs/+/86813423:51
@clarkb:matrix.orgcorvus: ^ hopefully that addresses both linting issues and the siblings behavior23:51
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 867057: WIP: play with noxfile https://review.opendev.org/c/zuul/zuul/+/86705723:52
@jim:acmegating.comClark: cool how does that update look ^?23:52
@clarkb:matrix.orgya that looks right23:53

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!