| @mnaser:matrix.org | Clark: i think maybe i have too new of local git version compared to zuul, since it very much happily reports no merge conflicts ;( | 00:02 |
|---|---|---|
| @mnaser:matrix.org | yeah, git 2.47.0 locally vs 2.39.2 in the container | 00:08 |
| @fungicide:matrix.org | well, also zuul may be specifying a particular (more conservative) conflict resolution scheme in an attempt not to be looser with merges than gerrit is | 00:14 |
| @mnaser:matrix.org | it seems that the current version of zuul we run (11.1.0) has 2.39.2 and 11.2.0 has 2.39.5 -- i'll have an attempt at bumping it and seeing if it helps. | 00:15 |
| @mnaser:matrix.org | i've tried every single attempt of trying to make it happy but nothing.. i cant reproduce it locally either | 00:15 |
| @mnaser:matrix.org | the conflicting file is this https://review.vexxhost.dev/c/atmosphere/+/214/1/zuul.d/project.yaml and the only thing i can imagine conflicting from the parent change is https://review.vexxhost.dev/c/atmosphere/+/208/4/zuul.d/project.yaml so i think it may be the end of line that got it unhappy | 00:16 |
| @clarkb:matrix.org | you might try having gerrit do a rebase maybe? But ya I'm not sure. I have git 2.47.1 locally. Theoretically could be a bug in new git impacting compatibility with old git. That would probably get more attention if it is a known issue though | 00:28 |
| @clarkb:matrix.org | oh something else to check would be that permissions and file tree don't change (I don't think they do based on your links) but we've seen those create odd conflicts in the past. And also maybe less obvious characters sneaking in and changing (like utf8 quotes or something) | 00:56 |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 939421: web: Fix null reference exception for cancelled builds https://review.opendev.org/c/zuul/zuul/+/939421 | 09:03 | |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 939421: web: Fix null reference exception for cancelled builds https://review.opendev.org/c/zuul/zuul/+/939421 | 09:11 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 937895: Add "autohold_matches" var in job inventory https://review.opendev.org/c/zuul/zuul/+/937895 | 09:16 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 938463: sqlreporter: ensure build end data is stored even if log URL is too long https://review.opendev.org/c/zuul/zuul/+/938463 | 09:50 | |
| -@gerrit:opendev.org- Andy Ladjadj proposed: [zuul/zuul-jobs] 836744: fix(packer): prevent task failure when packer_variables is not defined https://review.opendev.org/c/zuul/zuul-jobs/+/836744 | 15:20 | |
| -@gerrit:opendev.org- Andy Ladjadj proposed: [zuul/zuul-jobs] 834045: [upload-logs-base] add ipa extension to known mime types https://review.opendev.org/c/zuul/zuul-jobs/+/834045 | 15:22 | |
| -@gerrit:opendev.org- Andy Ladjadj proposed wip: [zuul/zuul-jobs] 834046: [upload-logs-base] add android mime-type https://review.opendev.org/c/zuul/zuul-jobs/+/834046 | 15:22 | |
| -@gerrit:opendev.org- Andy Ladjadj proposed wip: [zuul/zuul-jobs] 834046: [upload-logs-base] add android mime-type https://review.opendev.org/c/zuul/zuul-jobs/+/834046 | 15:23 | |
| -@gerrit:opendev.org- Andy Ladjadj marked as active: [zuul/zuul-jobs] 834046: [upload-logs-base] add android mime-type https://review.opendev.org/c/zuul/zuul-jobs/+/834046 | 15:23 | |
| -@gerrit:opendev.org- Andy Ladjadj proposed wip: [zuul/zuul-jobs] 834043: [upload-logs-base] add public url attribute https://review.opendev.org/c/zuul/zuul-jobs/+/834043 | 15:27 | |
| -@gerrit:opendev.org- Andy Ladjadj proposed wip: [zuul/zuul-jobs] 834043: [upload-logs-base] add public url attribute https://review.opendev.org/c/zuul/zuul-jobs/+/834043 | 15:30 | |
| -@gerrit:opendev.org- Andy Ladjadj proposed: [zuul/zuul-jobs] 834046: [upload-logs-base] add android mime-type https://review.opendev.org/c/zuul/zuul-jobs/+/834046 | 15:36 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 938685: Use opendevmirror for container images https://review.opendev.org/c/zuul/zuul/+/938685 | 15:48 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Simon Westphahl: [zuul/zuul] 939201: Don't invalidate zk context on stop event https://review.opendev.org/c/zuul/zuul/+/939201 | 16:49 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 939470: Reduce font-size used for the pipeline name https://review.opendev.org/c/zuul/zuul/+/939470 | 17:04 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Felix Edel: [zuul/zuul] 937112: Reduce empty space on status page https://review.opendev.org/c/zuul/zuul/+/937112 | 17:04 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 939472: Reword the reconfigured toolbar stat https://review.opendev.org/c/zuul/zuul/+/939472 | 17:15 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Felix Edel: | 17:15 | |
| - [zuul/zuul] 938551: Make job-list on status page more compact https://review.opendev.org/c/zuul/zuul/+/938551 | ||
| - [zuul/zuul] 937876: Incorporate tenants stats in Toolbar section https://review.opendev.org/c/zuul/zuul/+/937876 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 939473: Reverse order of name/value in toolbar stats https://review.opendev.org/c/zuul/zuul/+/939473 | 17:16 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Felix Edel: | 17:22 | |
| - [zuul/zuul] 938551: Make job-list on status page more compact https://review.opendev.org/c/zuul/zuul/+/938551 | ||
| - [zuul/zuul] 937876: Incorporate tenants stats in Toolbar section https://review.opendev.org/c/zuul/zuul/+/937876 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 939473: Reverse order of name/value in toolbar stats https://review.opendev.org/c/zuul/zuul/+/939473 | 17:24 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 939472: Reword the reconfigured toolbar stat https://review.opendev.org/c/zuul/zuul/+/939472 | 17:24 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Felix Edel: [zuul/zuul] 938677: Summarize named queues in default (collapsed) view on status page https://review.opendev.org/c/zuul/zuul/+/938677 | 17:25 | |
| -@gerrit:opendev.org- Michael Kelly proposed: | 17:32 | |
| - [zuul/zuul-operator] 883595: deploy: Surface disk_limit_per_job in operator CRD https://review.opendev.org/c/zuul/zuul-operator/+/883595 | ||
| - [zuul/zuul-operator] 883596: Broaden scope of potential config updates https://review.opendev.org/c/zuul/zuul-operator/+/883596 | ||
| - [zuul/zuul-operator] 853695: Prefix zuul-specific resources with instance name https://review.opendev.org/c/zuul/zuul-operator/+/853695 | ||
| - [zuul/zuul-operator] 853696: Prefix nodepool specific resources with instance name https://review.opendev.org/c/zuul/zuul-operator/+/853696 | ||
| - [zuul/zuul-operator] 867938: Prefix managed resources with instance name https://review.opendev.org/c/zuul/zuul-operator/+/867938 | ||
| - [zuul/zuul-operator] 861488: helm: Add a basic helm chart for zuul-operator https://review.opendev.org/c/zuul/zuul-operator/+/861488 | ||
| - [zuul/zuul-operator] 862390: helm: Add cert-manager as optional dependency https://review.opendev.org/c/zuul/zuul-operator/+/862390 | ||
| - [zuul/zuul-operator] 863191: helm: Add pxc-operator as optional dependency https://review.opendev.org/c/zuul/zuul-operator/+/863191 | ||
| - [zuul/zuul-operator] 865945: test: Restructure zuul-operator-functional-k8s layout https://review.opendev.org/c/zuul/zuul-operator/+/865945 | ||
| - [zuul/zuul-operator] 863579: test: Introduce Helm-based functional test https://review.opendev.org/c/zuul/zuul-operator/+/863579 | ||
| - [zuul/zuul-operator] 863474: k8s: Provide an option to disable PXC operator installer https://review.opendev.org/c/zuul/zuul-operator/+/863474 | ||
| - [zuul/zuul-operator] 866231: k8s: Clean up cert-manager installer https://review.opendev.org/c/zuul/zuul-operator/+/866231 | ||
| - [zuul/zuul-operator] 863475: k8s: Provide an option to disable cert-manager installation https://review.opendev.org/c/zuul/zuul-operator/+/863475 | ||
| - [zuul/zuul-operator] 863586: helm: Remove unnecessary CRD access from clusterrole https://review.opendev.org/c/zuul/zuul-operator/+/863586 | ||
| - [zuul/zuul-operator] 863476: k8s: Enable administrator to limit the watched namespace scope https://review.opendev.org/c/zuul/zuul-operator/+/863476 | ||
| - [zuul/zuul-operator] 863477: k8s: Allow use of a default image version besides latest https://review.opendev.org/c/zuul/zuul-operator/+/863477 | ||
| - [zuul/zuul-operator] 863571: web: Enable custom metadata for Service resources https://review.opendev.org/c/zuul/zuul-operator/+/863571 | ||
| - [zuul/zuul-operator] 861279: bug: Select scheduler pod based on instance name on update https://review.opendev.org/c/zuul/zuul-operator/+/861279 | ||
| - [zuul/zuul-operator] 863572: bug: Properly parameterize zookeeper-client-tls everywhere https://review.opendev.org/c/zuul/zuul-operator/+/863572 | ||
| - [zuul/zuul-operator] 866295: k8s: Remove unused ClusterRole from rbac-admin https://review.opendev.org/c/zuul/zuul-operator/+/866295 | ||
| - [zuul/zuul-operator] 866296: helm: Support clusteradmin role binding https://review.opendev.org/c/zuul/zuul-operator/+/866296 | ||
| - [zuul/zuul-operator] 866297: k8s: Added deploy cluster admin template https://review.opendev.org/c/zuul/zuul-operator/+/866297 | ||
| - [zuul/zuul-operator] 866406: k8s: Inject rbac.yaml into operator.yaml https://review.opendev.org/c/zuul/zuul-operator/+/866406 | ||
| - [zuul/zuul-operator] 866407: k8s: Provide tools and checker for deploy templates https://review.opendev.org/c/zuul/zuul-operator/+/866407 | ||
| - [zuul/zuul-operator] 863439: doc: Rework install doc to cover both template and helm install https://review.opendev.org/c/zuul/zuul-operator/+/863439 | ||
| @joao15130:matrix.org | Clark: Two things: | 18:05 |
| - Inheritance resolved, it was a parameter (MTU) which was causing the bug. Thank you. | ||
| - Synchronize: the rsync issue comes from the session ansible is trying to set up. When setting hosts: all it tries to execute the playbook on the remote host (which makes sense) but the command below fails. When it's set to localhost it doesn't recognize the key. | ||
| /usr/bin/rsync --delay-updates -F --compress --delete-after --archive --no-owner --no-group --rsh=\'/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null\' --out-format=\'<<CHANGED>>%i %n%L\' /var/lib/zuul/builds/f9d3fa4427c94809a1717e8af29546ad/work/src zuul@10.228.237.212:. | ||
| Feel free to advise. | ||
| @clarkb:matrix.org | as mentioned the rsync module doesn't use the controlpersist setup so it has to create a new ssh connection. My hunch is that something during the job runtime is preventing that from happening (potentially new firewall rules that allow the existing connection to run, changes to the accepted ssh keys, etc) | 18:07 |
| @joao15130:matrix.org | no firewall set up | 18:08 |
| @joao15130:matrix.org | and here's what I can debug | 18:08 |
| @joao15130:matrix.org | TASK [prepare-workspace : debug current host] | 18:08 |
| localhost | 6687c91eb78e | ||
| localhost | ok: Runtime: 0:00:00.006902 | ||
| TASK [prepare-workspace : debug user] | ||
| localhost | root | ||
| localhost | ok: Runtime: 0:00:00.007961 | ||
| @joao15130:matrix.org | so synchronize is trying to connect with zuul user from the executor container (logged as root) | 18:09 |
| @joao15130:matrix.org | however, the error has now evolved | 18:10 |
| @joao15130:matrix.org | 2025-01-16 18:07:28.709996 | localhost | "msg": "rsync: [generator] recv_generator: mkdir \"/var/lib/zuul/builds/e27954e542db48e780a869ceaee85822/trusted/project_0/gerrit/zuul-config/roles/prepare-workspace/tasks/src\" failed: Read-only file system (30)\n*** Skipping any contents from this failed directory ***\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]\n", | 18:10 |
| @joao15130:matrix.org | I've set ssh key to allow root user from executor to log into the worker as zuul | 18:10 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 938579: Act on image build artifact unlock instead of image uploads https://review.opendev.org/c/zuul/zuul/+/938579 | 18:10 | |
| @clarkb:matrix.org | the reason I suspect something like that is you said normal ssh for regular tasks does work. That uses controlpersist by default (and maybe only with zuul?). That creates a single ssh connection at the beginning of your job that gets reused over and over again as long as there isn't a gap in usage that causes it to timeout. rsync does not use this system so when you get to an rsync/synchronize task it has to create a new ssh connection which is apparently failing | 18:16 |
| @clarkb:matrix.org | this can happen if something during job runtime impacts your ability to ssh while keeping the existing controlpersist connection alive | 18:16 |
| @joao15130:matrix.org | I'm confused with the way the synchronize must work | 18:17 |
| @joao15130:matrix.org | does it have to ssh to the remote host and synchronized or is it ran on the locahost (exec container) | 18:17 |
| @joao15130:matrix.org | from the title: | 18:18 |
| Synchronize src repos to workspace directory | ||
| @clarkb:matrix.org | it runs rsync | 18:19 |
| @clarkb:matrix.org | depending on how you write the tasks it can connect from test node to executor or executor to test node. It doesn't need to use ssh but that is likely to be the most common setup | 18:19 |
| @joao15130:matrix.org | From my understanding zuul via ansible clone some repos that will be sync to the remote host | 18:20 |
| @joao15130:matrix.org | ok | 18:20 |
| @clarkb:matrix.org | it can also synchronize from testnode A to testnode B. It all depends on how you use it | 18:20 |
| @clarkb:matrix.org | it is worth noting that prepare-workspace-git might be preferable to prepare-workspace. prepare-workspace-git uses git operations instead of rsync and this can be useful if you have a cache on your test nodes. However, that doesn't answer the question of why rsync is failing | 18:21 |
| @joao15130:matrix.org | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/prepare-workspace-git | 18:22 |
| @joao15130:matrix.org | ok | 18:22 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 938146: Limit command stdout/stderr to 1GiB https://review.opendev.org/c/zuul/zuul/+/938146 | 18:34 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 938608: Re-elect James Blair as project lead https://review.opendev.org/c/zuul/zuul/+/938608 | 19:22 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 938783: Remove config object freezing https://review.opendev.org/c/zuul/zuul/+/938783 | 19:22 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [zuul/zuul] 934089: Add a note about Zuul not handling Gerrit chagne number collisions https://review.opendev.org/c/zuul/zuul/+/934089 | 20:13 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 931847: Don't join streaming threads when skipped before connect https://review.opendev.org/c/zuul/zuul/+/931847 | 20:13 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 938463: sqlreporter: ensure build end data is stored even if log URL is too long https://review.opendev.org/c/zuul/zuul/+/938463 | 20:51 | |
| @mhuin:matrix.org | having a bit of trouble testing this one. I'd like to set a long log url and check what gets recorded in the sql table for builds, so I use zuul_return in my test playbook as described in the doc. It doesn't seem to work that way though. Anybody can tell me what I'm doing wrong? | 20:55 |
| @clarkb:matrix.org | mhu: not all test cases actually run ansible iirc. I would double check your test case is doing so | 21:02 |
| -@gerrit:opendev.org- Zuul merged on behalf of Benjamin Schanzel: [zuul/zuul] 939421: web: Fix null reference exception for cancelled builds https://review.opendev.org/c/zuul/zuul/+/939421 | 21:26 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 937661: Add provider list web API endpoint https://review.opendev.org/c/zuul/zuul/+/937661 | 21:30 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 938146: Limit command stdout/stderr to 1GiB https://review.opendev.org/c/zuul/zuul/+/938146 | 21:51 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 23:17 | |
| - [zuul/zuul] 937662: Add images web endpoint https://review.opendev.org/c/zuul/zuul/+/937662 | ||
| - [zuul/zuul] 938519: Fix dict mutation in web info endpoint https://review.opendev.org/c/zuul/zuul/+/938519 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 937817: Add provider image web ui https://review.opendev.org/c/zuul/zuul/+/937817 | 23:17 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!