Tuesday, 2024-04-23

@jim:acmegating.comzuul-maint: how does this look for a zuul release? commit 4d6db3602fee4383bf92445e434047f77adfaa4e (HEAD -> master, tag: 10.1.0, origin/master, gerrit/master)00:23
@anoop-jose:matrix.orgIs it possible to have a stand-alone zuul installed, and set Gerrit, GitHub and GitLab connected to it? I meant, if someone use Gerrit for code push and review the events must be streamed to zuul. And same for GitHub and GitLab.03:26
All these should be connected at the same time to same zuul installation.
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 916709: Fix Gerrit query for topics with special chars https://review.opendev.org/c/zuul/zuul/+/91670906:50
@anoop-jose:matrix.org * I have a stand-alone Zuul installed, and its currently connected to Gerrit. Is it also possible to connect GitHub and GitLab connect to the same Zuul installation? 06:50
I meant, if someone uses Gerrit for code push and review the events must be streamed to Zuul. And the same for GitHub and GitLab. All these should be connected at the same time to the same Zuul installation.
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 916709: Fix Gerrit query for topics with special chars https://review.opendev.org/c/zuul/zuul/+/91670906:55
@sjal:matrix.org> <@anoop-jose:matrix.org> I have a stand-alone Zuul installed, and its currently connected to Gerrit. Is it also possible to connect GitHub and GitLab connect to the same Zuul installation? 07:26
> I meant, if someone uses Gerrit for code push and review the events must be streamed to Zuul. And the same for GitHub and GitLab. All these should be connected at the same time to the same Zuul installation.
you can setup different connections in the config, it won't stop you from adding several different ones. Each project is tied to a different connection also
@sjal:matrix.orggerrit/gitlab etc. are just drivers for those connections07:28
-@gerrit:opendev.org- Felix Edel proposed:08:22
- [zuul/zuul] 916284: Implement new status page https://review.opendev.org/c/zuul/zuul/+/916284
- [zuul/zuul] 916285: Make helper functions available to other components https://review.opendev.org/c/zuul/zuul/+/916285
- [zuul/zuul] 916286: Implement QueueItemPopover https://review.opendev.org/c/zuul/zuul/+/916286
- [zuul/zuul] 916287: Implement PipelineDetails view https://review.opendev.org/c/zuul/zuul/+/916287
- [zuul/zuul] 916288: Sort pipelines on status page by number of queue items https://review.opendev.org/c/zuul/zuul/+/916288
- [zuul/zuul] 916289: Align QueueItem and QueueItemPopover https://review.opendev.org/c/zuul/zuul/+/916289
- [zuul/zuul] 916290: Align job progress bars and job result labels https://review.opendev.org/c/zuul/zuul/+/916290
- [zuul/zuul] 916744: Visualize branches in ChangeQueues https://review.opendev.org/c/zuul/zuul/+/916744
@drichardt:matrix.orgHi guys. After moving our Zuul to new domain and updateing certs and everything I'm getting strange error message from zuul-web 08:42
```
2024-04-23 08:21:28,893 INFO zuul.GithubConnection.GithubClientManager: Authing to GitHub
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/jwt/algorithms.py", line 257, in prepare_key
key = load_pem_private_key(key, password=None)
File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/primitives/serialization/base.py", line 24, in load_pem_private_key
return ossl.load_pem_private_key(
File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 949, in load_pem_private_key
return self._load_key(
File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1201, in _load_key
self._handle_key_loading_error()
File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1257, in _handle_key_loading_error
raise ValueError(
ValueError: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [_OpenSSLErrorWithText(code=503841036, lib=60, reason=524556, reason_text=b'error:1E08010C:DECODER routines::unsupported')])
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/zuul-scheduler", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.10/site-packages/zuul/cmd/scheduler.py", line 127, in main
Scheduler().main()
File "/usr/local/lib/python3.10/site-packages/zuul/cmd/__init__.py", line 248, in main
self.run()
File "/usr/local/lib/python3.10/site-packages/zuul/cmd/scheduler.py", line 92, in run
self.connections.load(self.sched.zk_client,
File "/usr/local/lib/python3.10/site-packages/zuul/lib/connections.py", line 82, in load
connection.onLoad(zk_client, component_registry)
File "/usr/local/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 1344, in onLoad
self._github_client_manager.initialize()
File "/usr/local/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 993, in initialize
self._prime_installation_map()
File "/usr/local/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 1159, in _prime_installation_map
headers = self._get_app_auth_headers()
File "/usr/local/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 1081, in _get_app_auth_headers
app_token = jwt.encode(data,
File "/usr/local/lib/python3.10/site-packages/jwt/api_jwt.py", line 67, in encode
return api_jws.encode(json_payload, key, algorithm, headers, json_encoder)
File "/usr/local/lib/python3.10/site-packages/jwt/api_jws.py", line 153, in encode
key = alg_obj.prepare_key(key)
File "/usr/local/lib/python3.10/site-packages/jwt/algorithms.py", line 259, in prepare_key
key = load_pem_public_key(key)
File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/primitives/serialization/base.py", line 34, in load_pem_public_key
return ossl.load_pem_public_key(data)
File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 996, in load_pem_public_key
self._handle_key_loading_error()
File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1257, in _handle_key_loading_error
raise ValueError(
ValueError: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [_OpenSSLErrorWithText(code=75497580, lib=9, reason=108, reason_text=b'error:0480006C:PEM routines::no start line')])
```
does someone of you has an Idea which key he tries to use there and what could cause this error ?
-@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 916744: Visualize branches in ChangeQueues https://review.opendev.org/c/zuul/zuul/+/91674408:44
-@gerrit:opendev.org- Felix Edel proposed:09:17
- [zuul/zuul] 916287: Implement PipelineDetails view https://review.opendev.org/c/zuul/zuul/+/916287
- [zuul/zuul] 916288: Sort pipelines on status page by number of queue items https://review.opendev.org/c/zuul/zuul/+/916288
- [zuul/zuul] 916289: Align QueueItem and QueueItemPopover https://review.opendev.org/c/zuul/zuul/+/916289
- [zuul/zuul] 916290: Align job progress bars and job result labels https://review.opendev.org/c/zuul/zuul/+/916290
- [zuul/zuul] 916744: Visualize branches in ChangeQueues https://review.opendev.org/c/zuul/zuul/+/916744
@drichardt:matrix.org> <@drichardt:matrix.org> Hi guys. After moving our Zuul to new domain and updateing certs and everything I'm getting strange error message from zuul-web 09:33
>
> ```
> 2024-04-23 08:21:28,893 INFO zuul.GithubConnection.GithubClientManager: Authing to GitHub
> Traceback (most recent call last):
> File "/usr/local/lib/python3.10/site-packages/jwt/algorithms.py", line 257, in prepare_key
> key = load_pem_private_key(key, password=None)
> File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/primitives/serialization/base.py", line 24, in load_pem_private_key
> return ossl.load_pem_private_key(
> File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 949, in load_pem_private_key
> return self._load_key(
> File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1201, in _load_key
> self._handle_key_loading_error()
> File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1257, in _handle_key_loading_error
> raise ValueError(
> ValueError: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [_OpenSSLErrorWithText(code=503841036, lib=60, reason=524556, reason_text=b'error:1E08010C:DECODER routines::unsupported')])
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
> File "/usr/local/bin/zuul-scheduler", line 8, in <module>
> sys.exit(main())
> File "/usr/local/lib/python3.10/site-packages/zuul/cmd/scheduler.py", line 127, in main
> Scheduler().main()
> File "/usr/local/lib/python3.10/site-packages/zuul/cmd/__init__.py", line 248, in main
> self.run()
> File "/usr/local/lib/python3.10/site-packages/zuul/cmd/scheduler.py", line 92, in run
> self.connections.load(self.sched.zk_client,
> File "/usr/local/lib/python3.10/site-packages/zuul/lib/connections.py", line 82, in load
> connection.onLoad(zk_client, component_registry)
> File "/usr/local/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 1344, in onLoad
> self._github_client_manager.initialize()
> File "/usr/local/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 993, in initialize
> self._prime_installation_map()
> File "/usr/local/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 1159, in _prime_installation_map
> headers = self._get_app_auth_headers()
> File "/usr/local/lib/python3.10/site-packages/zuul/driver/github/githubconnection.py", line 1081, in _get_app_auth_headers
> app_token = jwt.encode(data,
> File "/usr/local/lib/python3.10/site-packages/jwt/api_jwt.py", line 67, in encode
> return api_jws.encode(json_payload, key, algorithm, headers, json_encoder)
> File "/usr/local/lib/python3.10/site-packages/jwt/api_jws.py", line 153, in encode
> key = alg_obj.prepare_key(key)
> File "/usr/local/lib/python3.10/site-packages/jwt/algorithms.py", line 259, in prepare_key
> key = load_pem_public_key(key)
> File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/primitives/serialization/base.py", line 34, in load_pem_public_key
> return ossl.load_pem_public_key(data)
> File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 996, in load_pem_public_key
> self._handle_key_loading_error()
> File "/usr/local/lib/python3.10/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1257, in _handle_key_loading_error
> raise ValueError(
> ValueError: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [_OpenSSLErrorWithText(code=75497580, lib=9, reason=108, reason_text=b'error:0480006C:PEM routines::no start line')])
> ```
>
> does someone of you has an Idea which key he tries to use there and what could cause this error ?
Found issue. Certifikate was not in pem format. Creating new app credential and adding it solved issue
@f2ked:matrix.orghow is [`zuul.executor.work_root`](https://zuul-ci.org/docs/zuul/latest/job-content.html#var-zuul.executor.work_root) specified? Docs seem to suggest it should be `${HOME}/work` but my executors are using `$HOME`.14:02
@jim:acmegating.comf2ked: can you link to the docs that suggest it should be $home/work?14:03
-@gerrit:opendev.org- Christian Mueller proposed: [zuul/nodepool] 916801: WIP: enable EC2 Fleet API https://review.opendev.org/c/zuul/nodepool/+/91680115:12
-@gerrit:opendev.org- Christian Mueller proposed: [zuul/nodepool] 916801: WIP: enable EC2 Fleet API https://review.opendev.org/c/zuul/nodepool/+/91680115:28
@f2ked:matrix.orgtop of the same page15:31
https://zuul-ci.org/docs/zuul/latest/job-content.html#working-directory
@f2ked:matrix.orgit doesn't talk about `$HOME` but I'm assuming it is wherever ssh15:33
@f2ked:matrix.org * it doesn't talk about `$HOME` but I'm assuming it is wherever ssh'in lands you15:33
@jim:acmegating.comssh isn't involved here; the executor is the host that ssh is executed *from*, not *to*.15:38
thanks for the pointer to that text. i could see why one might assume that was in relation to $HOME, but the text actually says `work/` is under "a directory to hold all the content related to the job" -- internally, that's the build directory. we set $HOME to the build directory + work/ because it's the highest level directory writable by the job. we could explicitly add a note about $HOME, however, i would advise against using $HOME and just consider the behavior undefined, and instead just use the zuul variables for generating paths.
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 915091: Add --disable-pipelines option https://review.opendev.org/c/zuul/zuul/+/91509116:42
@f2ked:matrix.orgok.. but I'm still confused (could be terminology)18:43
I have a worker node that I get from a static node pool.
when the source "appears" on that node it is in `./src` not `./work/src` as this document suggests
Associating `.` with `$HOME` is me.
I guess my real question is, how can I specify the directory where all "work" will be done on that node?
@jim:acmegating.comf2ked: yeah i think there's a terminology confusion.  `executor` means the zuul-executor process.  that's where the ansible-playbook process is run, and the ssh connection to the remote worker node originates from.  that's also where the zuul.executor.* ansible variables in the docs are relevant.18:58
the worker node is the static node that receives the ssh connection. strictly speaking, the responsibility of the zuul program ends there, and what happens next is up to the user, but we generally expect users to use the roles in the zuul-jobs repo, and by shared convention, they copy the git source code from the executor to the remote node and put it it `$HOME/src/...`. essentially they mirror `.../work/src/` on the executor to `$HOME/src/` on the remote worker node.
@picog:matrix.orgI'm trying to debug a very weird job where there seems to be no output of the job and the build seems to fail for no reason.19:20
Any clues what I should be looking for?
If I grep for the event-id of the job, I see:
```
a718dabae24b 2024-04-23 16:54:02,499 INFO zuul.AnsibleJob: [e: 374b69fc2bd14c6f8d1582013faad159] [build: 2bd641dd880d49adab1d23309d09645b] Checking out gerrit/zuul/zuul-jobs branch master
a718dabae24b ----------------------------------------
a718dabae24b Exception occurred during processing of request from ('::ffff:10.89.0.149', 48860, 0, 0)
a718dabae24b Traceback (most recent call last):
a718dabae24b File "/usr/local/lib/python3.11/site-packages/zuul/lib/log_streamer.py", line 77, in handle
a718dabae24b self.stream_log(log_file)
a718dabae24b File "/usr/local/lib/python3.11/site-packages/zuul/lib/log_streamer.py", line 98, in stream_log
a718dabae24b if self.follow_log(log):
a718dabae24b ^^^^^^^^^^^^^^^^^^^^
a718dabae24b File "/usr/local/lib/python3.11/site-packages/zuul/lib/log_streamer.py", line 144, in follow_log
a718dabae24b ret = self.request.recv(1024)
a718dabae24b ^^^^^^^^^^^^^^^^^^^^^^^
a718dabae24b ConnectionResetError: [Errno 104] Connection reset by peer
a718dabae24b
a718dabae24b During handling of the above exception, another exception occurred:
a718dabae24b
a718dabae24b Traceback (most recent call last):
a718dabae24b File "/usr/local/lib/python3.11/socketserver.py", line 691, in process_request_thread
a718dabae24b self.finish_request(request, client_address)
a718dabae24b File "/usr/local/lib/python3.11/socketserver.py", line 361, in finish_request
a718dabae24b self.RequestHandlerClass(request, client_address, self)
a718dabae24b File "/usr/local/lib/python3.11/socketserver.py", line 755, in __init__
a718dabae24b self.handle()
a718dabae24b File "/usr/local/lib/python3.11/site-packages/zuul/lib/log_streamer.py", line 82, in handle
a718dabae24b self.request.sendall(msg.encode("utf-8"))
a718dabae24b BrokenPipeError: [Errno 32] Broken pipe
a718dabae24b ----------------------------------------
a718dabae24b 2024-04-23 17:14:39,876 INFO zuul.AnsibleJob: [e: 374b69fc2bd14c6f8d1582013faad159] [build: cab6dce863a44eaeb2083b4f1477b8e1] Early failure in job
a718dabae24b 2024-04-23 17:14:48,329 INFO zuul.ExecutorServer: [e: 374b69fc2bd14c6f8d1582013faad159] [build: cab6dce863a44eaeb2083b4f1477b8e1] Held status set to False
a718dabae24b 2024-04-23 17:14:50,327 INFO zuul.nodepool: [e: 374b69fc2bd14c6f8d1582013faad159] Returning nodeset <NodeSet [<Node 0000000891 ['sslva-builder']:node>]>
a718dabae24b 2024-04-23 17:14:50,449 INFO zuul.nodepool: [e: 374b69fc2bd14c6f8d1582013faad159] Nodeset <NodeSet [<Node 0000000891 ['sslva-builder']:node>]> with 1 nodes was in use for 1253.2204751209356 seconds for build <BuildRequest cab6dce863a44eaeb2083b4f1477b8e1, job=293e9d082a83470c930c74fa66bd6be1, state=completed, path=/zuul/executor/unzoned/requests/cab6dce863a44eaeb2083b4f1477b8e1 zone=None> for project gerrit.netronome.com/nse/sslva-build
a718dabae24b 2024-04-23 17:14:50,449 INFO zuul.AnsibleJob: [e: 374b69fc2bd14c6f8d1582013faad159] [build: cab6dce863a44eaeb2083b4f1477b8e1] Job execution took: 1253.220 seconds
a718dabae24b 2024-04-23 17:23:25,910 INFO zuul.ExecutorServer: [e: 374b69fc2bd14c6f8d1582013faad159] [build: 2bd641dd880d49adab1d23309d09645b] Held status set to False
a718dabae24b 2024-04-23 17:23:26,649 INFO zuul.nodepool: [e: 374b69fc2bd14c6f8d1582013faad159] Returning nodeset <NodeSet [<Node 0000000894 ['sslva-builder']:oracle9>]>
a718dabae24b 2024-04-23 17:23:26,686 INFO zuul.nodepool: [e: 374b69fc2bd14c6f8d1582013faad159] Nodeset <NodeSet [<Node 0000000894 ['sslva-builder']:oracle9>]> with 1 nodes was in use for 1771.3888368420303 seconds for build <BuildRequest 2bd641dd880d49adab1d23309d09645b, job=4f43e36fd87047bba203dc4aa4baa2fe, state=completed, path=/zuul/executor/unzoned/requests/2bd641dd880d49adab1d23309d09645b zone=None> for project gerrit.netronome.com/nse/sslva-build
a718dabae24b 2024-04-23 17:23:26,686 INFO zuul.AnsibleJob: [e: 374b69fc2bd14c6f8d1582013faad159] [build: 2bd641dd880d49adab1d23309d09645b] Job execution took: 1771.389 seconds
```
@picog:matrix.orgIs this log streamer error something that I should be worried about? 19:21
@sylvass:albinvass.seI think that is the zuul-executor trying to connect to the log-streaming on the remote node, but eventually fails. I don't think that should be something critical, but running this role early in the job should give you live log streaming for shell and command tasks:19:27
https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L45
@picog:matrix.org> <@sylvass:albinvass.se> I think that is the zuul-executor trying to connect to the log-streaming on the remote node, but eventually fails. I don't think that should be something critical, but running this role early in the job should give you live log streaming for shell and command tasks:19:32
> https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L45
Thanks, will add that.
-@gerrit:opendev.org- Zuul merged on behalf of Dr. Jens Harbott: [zuul/nodepool] 916053: Make debootstrap support noble https://review.opendev.org/c/zuul/nodepool/+/91605320:27
@f2ked:matrix.orgwhen running with the tutorial setup, should I expect21:06
```
http://XXX:9000/t/example-tenant/stream/XXX?logfile=console.log
```
to show more than
```
--- END OF STREAM ---
```
while the job is running?
@clarkb:matrix.orgYour base job needs to run the role that starts the log streamer daemon21:16
@clarkb:matrix.orgLike https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L4521:18
-@gerrit:opendev.org- dasm proposed: [zuul/zuul-client] 916845: [zuulclient] Implemented buildsets api https://review.opendev.org/c/zuul/zuul-client/+/91684521:57
@f2ked:matrix.orgcould you point me to the clause that does it?22:17
the [docs](https://zuul-ci.org/docs/zuul/latest/developer/ansible.html#capturing-live-command-output) say
> The global job configuration should run the zuul_console task as a very early prerequisite step.
so is it a task or a role?
@f2ked:matrix.orgI guess it's22:21
```
- hosts: all
roles:
- add-build-sshkey
+ - start-zuul-console
- prepare-workspace
```
@clarkb:matrix.orgThe line I linked to is the role `start-zuul-console`22:21
@clarkb:matrix.orgThere is an underlying task that does it within the role whcih is what you actually need to run. But the role encapsulates it nicely for you22:22
@f2ked:matrix.orgoops, I see that highlight now 😬22:22
@f2ked:matrix.orgClark: that did seem to be enough :( still only see22:38
```
--- END OF STREAM ---
```
@f2ked:matrix.organd yes port 19885 on the worker node is accessible22:40
@clarkb:matrix.orgMaybe look at the zuul executor debug log to see if the ansible output there indicates why it may be failing22:43
@clarkb:matrix.orgalso ensure the hosts you run that on are `all` not `localhost`22:43

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!