| @jim:acmegating.com | guillaumec: thanks! that sounds useful for us to keep in mind. and that must not have been easy to solve! | 00:24 |
|---|---|---|
| -@gerrit:opendev.org- Zuul merged on behalf of yatin: [zuul/zuul-jobs] 832497: [multi-node-bridge] Allow to skip openvswitch installation https://review.opendev.org/c/zuul/zuul-jobs/+/832497 | 03:26 | |
| -@gerrit:opendev.org- Simon Westphahl proposed on behalf of Benedikt Löffler: [zuul/nodepool] 802255: Add option for an upload script https://review.opendev.org/c/zuul/nodepool/+/802255 | 06:51 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 834324: Create remote ref when it does not exist https://review.opendev.org/c/zuul/zuul/+/834324 | 07:23 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 832757: debug test for bundle id https://review.opendev.org/c/zuul/zuul/+/832757 | 07:40 | |
| -@gerrit:opendev.org- Simon Westphahl proposed on behalf of Benedikt Löffler: [zuul/nodepool] 802255: Add option for an upload script https://review.opendev.org/c/zuul/nodepool/+/802255 | 07:49 | |
| @gobi_g:matrix.org | > <@gobi_g:matrix.org> I have 2 nodes. In one zuul server running. On that node I can able to view logs. On the other I'm facing this problem. | 08:13 |
| I checked ports using both nodes using lsof. | ||
| I see that in one node port 19885 was available(used by /usr/bin/python3) and another one it was not available. | ||
| How to make port 19885 available to the executor? | ||
| Note: I disabled the firewall | ||
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 832757: debug test for bundle id https://review.opendev.org/c/zuul/zuul/+/832757 | 08:42 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/nodepool] 834109: Pass requestor data to Nodes https://review.opendev.org/c/zuul/nodepool/+/834109 | 09:20 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul-jobs] 834432: mirror git repos: cleanup submodule checkouts https://review.opendev.org/c/zuul/zuul-jobs/+/834432 | 09:23 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul-jobs] 834432: mirror git repos: cleanup submodule checkouts https://review.opendev.org/c/zuul/zuul-jobs/+/834432 | 09:24 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul-jobs] 834432: mirror git repos: cleanup submodule checkouts https://review.opendev.org/c/zuul/zuul-jobs/+/834432 | 09:26 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 832757: debug test for bundle id https://review.opendev.org/c/zuul/zuul/+/832757 | 09:31 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 831925: Inject bundle id to inventroy zuul.items https://review.opendev.org/c/zuul/zuul/+/831925 | 09:56 | |
| @mhuin:matrix.org | > <@harrymichal:matrix.org> Hi folks! Just rerun a pipeline and one of the builds failed with 'POST_FAILURE' and did not provide any logs at all. How should I interpret such a result and is it even expected? | 10:09 |
| > https://softwarefactory-project.io/zuul/t/local/build/894d0a1757ac40a8b29f2f9735d18a47 | ||
| I'm looking at it currently. Here's a channel for troubleshooting problems on the public software factory instance: https://matrix.to/#/#sf-ops:matrix.org | ||
| @mhuin:matrix.org | If needed we can escalate to the public zuul channel here | 10:10 |
| @mhuin:matrix.org | And FYI it looks like you hit a random network error during the logs upload: 2022-03-20 21:31:46,287 DEBUG zuul.AnsibleJob.output: [e: 0cb04420-a893-11ec-94c3-d4084cec5c47] [build: 894d0a1757ac40a8b29f2f9735d18a47] Ansible output: b'failed: [ci-node-35] (item={\'dest\': \'/var/lib/zuul/builds/894d0a1757ac40a8b29f2f9735d18a47/work/artifacts\', \'src\': \'artifacts\'}) = | 10:20 |
| > {"ansible_loop_var": "zj_output", "changed": false, "cmd": "/bin/rsync --delay-updates -F --compress --archive --no-owner --no-group --rsh=/bin/ssh -S none -o Port=22 --out-format=<<CHANGED>>%i %n%L zuul-worker@<<REDACTED>>:/home/zuul-worker/zuul-output/artifacts/ /var/lib/zuul/builds/894d0a1757ac40a8b29f2f9735d18a | ||
| 47/work/artifacts/", "msg": "ssh_exchange_identification: Connection closed by remote host\\r\\nrsync: connection unexpectedly closed (0 bytes received so far) [Receiver]\\nrsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.2]\\n", "rc": 255, "zj_output": {"dest": "/var/lib/zuul/builds/894d0a1757ac4 | ||
| 0a8b29f2f9735d18a47/work/artifacts", "src": "artifacts"}}' | ||
| @mhuin:matrix.org | so a bit of bad luck | 10:21 |
| @gchauvel:matrix.org | * corvus: if it can be of any help to someone later on, my errors were introduced by https://review.opendev.org/c/zuul/zuul/+/816176, related to https://github.com/util-linux/util-linux/commit/93de9f687d1640fff963f26b7db474eef3746532, my ubuntu focal was running 5.13.x and distro setpriv was likely built against user space kernel headers linux-libc-dev package which is 5.4.x | 10:51 |
| - with 5.13.x, compiling a newer setpriv, it worked | ||
| - switching back to a 5.4.x kernel, it worked with distro setpriv | ||
| @q:fricklercloud.de | looking at https://zuul.opendev.org/t/openstack/project/opendev.org/openstack/devstack# I see 12 tabs named "master", but from looking at the jobs, some of those should correlate to stable branches. | 12:17 |
| @q:fricklercloud.de | if I query the API directly, the branch information seems to be in place | 12:30 |
| curl -s https://zuul.opendev.org/api/tenant/openstack/project/openstack/devstack | jq '.configs[] | select(.source_context.project == "openstack/devstack") | .source_context' | ||
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 831933: gerritdriver: enable filtering on wip state https://review.opendev.org/c/zuul/zuul/+/831933 | 13:09 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 831933: gerritdriver: enable triggering on wip state https://review.opendev.org/c/zuul/zuul/+/831933 | 13:11 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 831933: gerritdriver: enable triggering on wip state https://review.opendev.org/c/zuul/zuul/+/831933 | 13:31 | |
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 831933: gerritdriver: enable triggering on wip state https://review.opendev.org/c/zuul/zuul/+/831933 | 13:37 | |
| @avass:vassast.org | Sorry for the spam. I'm having some troubles getting unittests to connect to zookeeper setup with `./tools/test-setup-docker.sh` not really sure why :/ | 13:39 |
| @d-j-j:matrix.org | > <@avass:vassast.org> Sorry for the spam. I'm having some troubles getting unittests to connect to zookeeper setup with `./tools/test-setup-docker.sh` not really sure why :/ | 13:52 |
| I had similar issues for some time. The only thing I've found to help is adding an explicit zookeeper start command in tools/docker-compose.yaml: | ||
| volumes: | ||
| - "./ca:/var/certs:z" | ||
| - "./zoo.cfg:/conf/zoo.cfg:z" | ||
| + command: "sh -c 'zkServer.sh start-foreground'" | ||
| I couldn't find the root cause yet. Could you try if this workaround helps in your case as well? | ||
| @avass:vassast.org | > <@d-j-j:matrix.org> I had similar issues for some time. The only thing I've found to help is adding an explicit zookeeper start command in tools/docker-compose.yaml: | 13:53 |
| > | ||
| > volumes: | ||
| > - "./ca:/var/certs:z" | ||
| > - "./zoo.cfg:/conf/zoo.cfg:z" | ||
| > + command: "sh -c 'zkServer.sh start-foreground'" | ||
| > | ||
| > I couldn't find the root cause yet. Could you try if this workaround helps in your case as well? | ||
| yeah apparently that works, thanks! | ||
| -@gerrit:opendev.org- Albin Vass proposed: [zuul/zuul] 831933: gerritdriver: enable triggering on wip state https://review.opendev.org/c/zuul/zuul/+/831933 | 14:14 | |
| -@gerrit:opendev.org- Albin Vass marked as active: [zuul/zuul] 831933: gerritdriver: enable triggering on wip state https://review.opendev.org/c/zuul/zuul/+/831933 | 14:14 | |
| @fungicide:matrix.org | > <@gobi_g:matrix.org> I checked ports using both nodes using lsof. | 14:29 |
| > I see that in one node port 19885 was available(used by /usr/bin/python3) and another one it was not available. | ||
| > How to make port 19885 available to the executor? | ||
| > Note: I disabled the firewall | ||
| was there a job running at the time? if the log streamer is started by the start-zuul-console role, for example in your job's pre-run phase playbook, then i would only expect it to be listening on that port when a job is underway | ||
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: | 15:21 | |
| - [zuul/zuul] 834507: DNM: fix broken enqueue when buildset.newrev is null https://review.opendev.org/c/zuul/zuul/+/834507 | ||
| - [zuul/zuul] 834508: DNM Do not show sign-in button if no IdP is available https://review.opendev.org/c/zuul/zuul/+/834508 | ||
| @jim:acmegating.com | pushed 5.1.0 | 16:03 |
| @gobi_g:matrix.org | > <@fungicide:matrix.org> was there a job running at the time? if the log streamer is started by the start-zuul-console role, for example in your job's pre-run phase playbook, then i would only expect it to be listening on that port when a job is underway | 16:44 |
| In that vm logs are streaming. I have another VM in that I can't able to connect port 19885 | ||
| @fungicide:matrix.org | karthi: right, but you said you used lsof on that one to check whether anything was listening on the port and saw nothing, so i was asking whether you did that _while_ a job was running on the node | 17:20 |
| @fungicide:matrix.org | if you checked at a time when no job was running on the node, it may be normal that no process is listening on the port, so doesn't rule out whether the problem is somehwere else | 17:21 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 834518: Correct tagged version docs build https://review.opendev.org/c/zuul/zuul/+/834518 | 17:23 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 834519: Correct tagged version docs build https://review.opendev.org/c/zuul/nodepool/+/834519 | 17:23 | |
| @jim:acmegating.com | zuul-maint: ^ those are high-priority -- our release docs job broke | 17:23 |
| @jim:acmegating.com | i think to correct that, i should just do a manual docs build of 5.1.0 and copy it into afs... the only other thing i can think to do would be to just tag and release 5.1.1 and leave 5.1.0 an undocumented mystery release. i don't love that idea. | 17:25 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-client] 834522: Correct tagged version docs build https://review.opendev.org/c/zuul/zuul-client/+/834522 | 17:27 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-operator] 834524: Correct tagged version docs build https://review.opendev.org/c/zuul/zuul-operator/+/834524 | 17:28 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-sphinx] 834525: Correct tagged version docs build https://review.opendev.org/c/zuul/zuul-sphinx/+/834525 | 17:28 | |
| @jim:acmegating.com | zuul-maint: that should be the full set ^ | 17:28 |
| @jim:acmegating.com | i'll wait for some more feedback before i execute the manual build/afs copy procedure. | 17:29 |
| @clarkb:matrix.org | corvus: on it. I did leave some questions on https://review.opendev.org/c/zuul/zuul/+/832361 just now if you have time to look | 17:33 |
| @fungicide:matrix.org | corvus: i'm not opposed to a manually uploaded build of the docs, given we probably can't reasonably reenqueue the tag into the release pipeline | 17:40 |
| @fungicide:matrix.org | though if the docs job isn't dependent on the pypi upload job, maybe it would be simpler? the pypi upload job will simply fail as the re-upload will be rejected by the api | 17:41 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-sphinx] 834525: Correct tagged version docs build https://review.opendev.org/c/zuul/zuul-sphinx/+/834525 | 17:43 | |
| @fungicide:matrix.org | corvus: oh, except that the fixed sphinx config won't be present in the tagged commit, so we'll just get an identically broken docs build. never mind! the manual upload does make sense in that case | 17:45 |
| @jim:acmegating.com | fungi: exactly, yeah; the content is wrong :/ | 17:49 |
| @jim:acmegating.com | Clark: thx, replied! | 18:00 |
| @clarkb:matrix.org | corvus: oh yup I see the _by_ difference in those list names now. For the question related to error reporting I guess I don't see where we generate the error? Since it is possible to use submitWholeTopic without a cycle is it not possible that those get into the pipeline without erroring if the cycle support is disabled? | 18:16 |
| @jim:acmegating.com | Clark: correct, it will be enqueued and then reported back to gerrit with a warning in the report message that says "Dependency cycle detected" (enqueing is the only way we can get a message back to the user since that's the only place we'll have all the information to determine if we should leave a message) | 18:37 |
| @jim:acmegating.com | Clark: the next change in the stack actually makes a change to that process, so you can see all the relevant code: https://review.opendev.org/832362 | 18:37 |
| @gobi_g:matrix.org | > <@fungicide:matrix.org> if you checked at a time when no job was running on the node, it may be normal that no process is listening on the port, so doesn't rule out whether the problem is somehwere else | 18:52 |
| Got it. I'll check fungi: | ||
| @clarkb:matrix.org | corvus: I think my confusion is that all the code seems to care about actual cycles but the submitTogether stuff in Gerrit doesn't actually require a cycle? For example https://review.opendev.org/c/zuul/zuul/+/832362/2/zuul/manager/__init__.py shows that we detect cycles with cycleForChange() and report them back. But why would that detect a cycle with submitTogether if there isn't a cycle? | 19:01 |
| @clarkb:matrix.org | oh are we building a cycle in needs_changes and needed_by_changes? So that there will always be a cycle by the time the pipeline manager processes it even if there wasn't one in gerrit? | 19:03 |
| @clarkb:matrix.org | If that is the acse that might be worth a comment in https://review.opendev.org/c/zuul/zuul/+/832361/1/zuul/driver/gerrit/gerritconnection.py around line 1044? | 19:04 |
| @jim:acmegating.com | Clark: well, we're not really building a cycle there -- we're just annotating that gerrit says that A needs B, and B needs A. the pipeline manager will figure out that's a cycle. i think maybe that's the source of the confusion? the gerrit driver is trying to do the least amount of work possible beyond just providing information about what gerrit says about the changes. | 19:08 |
| @jim:acmegating.com | it's up to the pipeline manager to figure out that the dependencies are actually a cycle, and then dtrt | 19:08 |
| @clarkb:matrix.org | Yes but there may not be a cycle is my point. Since submit together doesn't imply that. So I'm confused in those cases why we would return an error if you haven't allowed cycles | 19:10 |
| @jim:acmegating.com | i don't see how submit together can not be a cycle | 19:10 |
| @clarkb:matrix.org | I think it is because we are effectively lying in the driver and saying there is a cycle there when we construct those lists | 19:10 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 834518: Correct tagged version docs build https://review.opendev.org/c/zuul/zuul/+/834518 | 19:10 | |
| @clarkb:matrix.org | Aiui Gerrit submit together allows you to do it on a normal stack of changes with no cycles | 19:10 |
| @clarkb:matrix.org | A<-B<-C and then they merge together but without a cycle | 19:11 |
| @jim:acmegating.com | sure, but they are submitted simultaneously, which is more or less zuul's definition of a dependency cycle | 19:11 |
| @clarkb:matrix.org | Yes I agree with that. But Gerrit won't show there is a cycle. I think it's that code I call out that essentially lies saying there is one though so that will handles it properly | 19:11 |
| @clarkb:matrix.org | Since it will end up saying C needs A and A needs C | 19:12 |
| @jim:acmegating.com | does gerrit have a visualization for a cycle? i thought it would just say "Submitted Together: A, B, C" | 19:12 |
| @clarkb:matrix.org | I don't think there is visualization. They don't even really visualize normal trees either :/ | 19:13 |
| @clarkb:matrix.org | Anyway I suspect the code is correct, it just isn't super clear why it is correct in the case of no explicit cycles. Hence my suggestion for a comment | 19:13 |
| @jim:acmegating.com | i'm happy to write a comment, i just don't know what to write :) | 19:13 |
| @clarkb:matrix.org | I'll draft something after lunch and we can see if what I end up with makes sense | 19:14 |
| @jim:acmegating.com | thanks :) | 19:14 |
| @jim:acmegating.com | (i tore down my test setup for that a few weeks ago, so it's not easy for me to see what the "submitted together" section says for that right now, but i could set it up again) | 19:16 |
| @jim:acmegating.com | (but my recollection is that it would say a,b,c) | 19:16 |
| @jim:acmegating.com | so it's saying both things: there's a git relationship, and they will be submitted together | 19:22 |
| @clarkb:matrix.org | corvus: ok I posted a suggestion. Does that make sense? | 19:49 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 19:54 | |
| - [zuul/zuul] 832361: Support submitWholeTopic in Gerrit https://review.opendev.org/c/zuul/zuul/+/832361 | ||
| - [zuul/zuul] 832362: Report per-branch cyclic-dependency conflicts https://review.opendev.org/c/zuul/zuul/+/832362 | ||
| - [zuul/zuul] 832363: Add queue.dependencies-by-topic https://review.opendev.org/c/zuul/zuul/+/832363 | ||
| @jim:acmegating.com | Clark: done, thanks! | 19:55 |
| @clarkb:matrix.org | +2 thanks | 19:55 |
| @jim:acmegating.com | i'm going to manually update the docs now | 19:56 |
| @jim:acmegating.com | done; https://zuul-ci.org/docs/zuul/5.1.0/ exists | 20:02 |
| -@gerrit:opendev.org- Tristan Cacqueray proposed: [zuul/zuul] 834570: Update axios to fix CVE-2022-0536 https://review.opendev.org/c/zuul/zuul/+/834570 | 22:02 | |
| -@gerrit:opendev.org- Tristan Cacqueray proposed: [zuul/zuul] 834571: Update swagger-ui https://review.opendev.org/c/zuul/zuul/+/834571 | 22:11 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-client] 834522: Correct tagged version docs build https://review.opendev.org/c/zuul/zuul-client/+/834522 | 22:16 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-operator] 834524: Correct tagged version docs build https://review.opendev.org/c/zuul/zuul-operator/+/834524 | 22:29 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 828917: Add support for Microsoft login https://review.opendev.org/c/zuul/zuul/+/828917 | 22:58 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!