| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 856523: Add span for builds and propagate via request https://review.opendev.org/c/zuul/zuul/+/856523 | 08:07 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 857421: Trace merge requests and merger operations https://review.opendev.org/c/zuul/zuul/+/857421 | 08:11 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Tristan Cacqueray: [zuul/zuul] 850575: doc: fix liveness probes path rendering https://review.opendev.org/c/zuul/zuul/+/850575 | 08:47 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [zuul/zuul-jobs] 858961: Reduce the number of loops in prepare-workspace-git https://review.opendev.org/c/zuul/zuul-jobs/+/858961 | 15:34 | |
| -@gerrit:opendev.org- Tristan Cacqueray proposed: [zuul/zuul] 859711: Add executor.ssh_args configuration https://review.opendev.org/c/zuul/zuul/+/859711 | 19:25 | |
| @tristanc_:matrix.org | I'm not sure how to proceed, but we need to set `-o Ciphers=aes256-ctr` on the ssh command to access a FIPS enabled host. Thus I've proposed 859711 ^, but perhaps there is another way? | 19:30 |
|---|---|---|
| @clarkb:matrix.org | > <@tristanc_:matrix.org> I'm not sure how to proceed, but we need to set `-o Ciphers=aes256-ctr` on the ssh command to access a FIPS enabled host. Thus I've proposed 859711 ^, but perhaps there is another way? | 19:34 |
| OpenDev's zuul is able to ssh to fips enabled test nodes from the debian bullseye based zuul container images without trouble. A bit more info on why this is necessary would be helpful. | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: | 19:38 | |
| - [zuul/zuul] 858096: Trace node request phase https://review.opendev.org/c/zuul/zuul/+/858096 | ||
| - [zuul/zuul] 858372: Create link to previous buildset span https://review.opendev.org/c/zuul/zuul/+/858372 | ||
| @tristanc_:matrix.org | Clark: I don't have much more info, beside that this option is required on our deployment, running ubi8 container on a centos7 host. | 19:38 |
| @clarkb:matrix.org | I think before we had a whole new mechanism to configure ssh we should understand this better. Running ssh -vvv from your container to a fips sshd should show which ciphers both ends support and where the mismatch is. I'm surprised ubi8 ssh can't ssh to a fips host though | 19:41 |
| @clarkb:matrix.org | Looks like fips 140-2 requires aes128-ctr or aes192-ctr or aes256-ctr. I guess this means your client isn't offering one of those by default. Maybe a better approach is to modify the client config to allow those ciphers and then let ssh negotiate them rather than hard coding? In particular it is my understanding that clients and servers should negotiate the best mutual option and keeping that available for non fips hosts is probably ideal | 19:48 |
| @tristanc_:matrix.org | Clark: but how can we modify the ssh client config for the executor? | 19:51 |
| @clarkb:matrix.org | I think you just configure it via /etc/ssh or /home/zuul/.ssh ? | 19:51 |
| @clarkb:matrix.org | It is using openssh | 19:51 |
| @clarkb:matrix.org | I'm really curious to know what cipher list you have configured if it doesn't include aes256-ctr | 19:52 |
| @tristanc_:matrix.org | Clark: I meant these paths are not mounted in bubblewrap, so you mean by adding them to the ro_path list ? | 19:58 |
| @clarkb:matrix.org | Oh if we don't mount an ssh config are we just using compiled defaults then? | 19:59 |
| @clarkb:matrix.org | In that case I'm really surprised you don't have aes256-ctr | 20:00 |
| @tristanc_:matrix.org | Clark: i'm waiting for another reproducer, but it looked like the connection was dropped because the client was offering deprecated ciphers | 20:02 |
| @clarkb:matrix.org | right, this is why I'm saying we need to be cautious. It seems like there may be a problem with your openssh installation and working around it with zuul here may just cover up that problem. We should understand this and if the client is deficient fix the client | 20:03 |
| @clarkb:matrix.org | running ssh -vvv in your container within the zuul bwrap environment (zuul has a command to get one of those iirc) will probably show you what the client is offering. ssh -Q cipher will show you what the client supports | 20:05 |
| @tristanc_:matrix.org | Clark: thank you for the pointer, i'll keep on debugging | 20:07 |
| @clarkb:matrix.org | If I install openssh-clients in redhat/ubi8 and execute `ssh -Q cipher` it reports aes256-ctr | 20:07 |
| @clarkb:matrix.org | The red hat openssh config seems to use this Ciphers list: `Ciphers aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc` | 20:09 |
| @tristanc_:matrix.org | Clark: the other parameter is that the zuul's ssh key is quite ancient, it was created with sha256 | 20:10 |
| @clarkb:matrix.org | tristanC: the key itself shouldn't matter for this aiui | 20:10 |
| @clarkb:matrix.org | this is the cipher used to bulk encrypt data after the connection is established | 20:10 |
| @clarkb:matrix.org | in the ssh -vvv output I think you are looking for the lines like `ciphers ctos` and `ciphers stoc` to show client to server (ctos) and server to client (stoc). Then later `kex: server->client cipher:` and `kex: client->server cipher:` to see what is chosen. | 20:14 |
| @clarkb:matrix.org | In your case you may only get the first type of log lines and not the second if negotiation is failing | 20:14 |
| @tristanc_:matrix.org | well, I can't reproduce the issue anymore, and perhaps it was not ssh after all, here is the failure we get: `"msg": "Connection closed by 38.102.83.73 port 22\r\nrsync: connection unexpectedly closed (0 bytes received so far) [sender]\nrsync error: unexplained error (code 255) at io.c(226) [sender=3.1.3]\n",` from: https://review.rdoproject.org/zuul/build/163d0147c214443b860d6c20268702a2/log/job-output.txt#187 | 20:19 |
| @clarkb:matrix.org | tristanC: once sec | 20:22 |
| @clarkb:matrix.org | tristanC: https://opendev.org/openstack/project-config/commit/a6d4fae07098546f5d61b32e62b7053946064e9a that is what I would look at next | 20:22 |
| @clarkb:matrix.org | I can't say for sure if you are hitting that problem but the symptoms are similar. Basically openssh sshd has a number of slots that it will start negotiating new connections on and that number is quite low by default. When you get people port scanning you those may all fill up and then valid ssh connections fail. To check on that you can look at your sshd logs (probably in syslog or equivalent) to see if you've got port scanning occuring at the same time as your failures | 20:23 |
| @clarkb:matrix.org | we worked around it by increasing the number of new connections | 20:24 |
| @clarkb:matrix.org | and it tends to be temporary as the port scanning roves around the internet | 20:24 |
| @tristanc_:matrix.org | Clark: I see, thank you for the help, it's much appreciated! | 20:24 |
| @clarkb:matrix.org | https://matrix.org/blog/2022/09/28/upgrade-now-to-address-encryption-vulns-in-matrix-sdks-and-clients fyi since we're on matrix here | 20:28 |
| @clarkb:matrix.org | this room isn't encrypted though so I don't think your use of this room is affected | 20:29 |
| @tristanc_:matrix.org | Clark: alright, so bwrap seems to be the issue, here is the command that fails: https://paste.opendev.org/show/bvhLUSqyiXONVQ3VdUww/ , and adding `-c aes256-ctr` fixes it : https://paste.opendev.org/show/bUNcTSfaZuSZ6rjqTnox/ | 20:42 |
| @tristanc_:matrix.org | and bind mounting the ssh_config from the container to the bwrap seems to fix it | 20:47 |
| @clarkb:matrix.org | tristanC: lines 35-36 and 47-48 are the relevant lines | 20:47 |
| @clarkb:matrix.org | I think that is showing both the server and client wanting to use chacha20-poly1305@openssh.com and then failing after | 20:48 |
| @clarkb:matrix.org | if your fips server can't do chacha20-poly1305@openssh.com why is it advertising it. That seems like maybe a fips bug | 20:48 |
| @clarkb:matrix.org | note that chacha and the gcm variants provide extra security over the aes ctr variants aiui. So openssh is properly negotiating the more secure method | 20:50 |
| @clarkb:matrix.org | overriding to aes256-ctr is going to work because that is what fips wants, but ideally you'd get that to work through properly configured software (in this case I think the server is misconfigured?) | 20:50 |
| @tristanc_:matrix.org | or perhaps because of that first choice, then it picks `<implicit>` MAC which is the actual issue? I don't know, the server is configured using the standard fips enabling tool, and the issue only happens with zuul jobs | 20:51 |
| @tristanc_:matrix.org | unfortunately we can't simply mount /etc/ssh and /etc/crypto-policy in bwrap because they get assigned to nobody and ssh is complaining about bad permission | 20:52 |
| @clarkb:matrix.org | > <@tristanc_:matrix.org> or perhaps because of that first choice, then it picks `<implicit>` MAC which is the actual issue? I don't know, the server is configured using the standard fips enabling tool, and the issue only happens with zuul jobs | 20:53 |
| MAC is implicit because it is built into the chacha cipher. It is not built in with aes256-ctr so there is a separate negotiation for that. The reason it doesn't work for zuul is we don't bind mount an openssh config in and so use compiled defaults. I think your config outside of bwrap is overriding the priority list of ciphers (see my list above it puts aes256-gcm before chacha20) and that must work | ||
| @tristanc_:matrix.org | right, thanks. So we'll add that to an extra trusted_ro_paths | 20:54 |
| @clarkb:matrix.org | well I still think the correct thing is to fix the remote side if it can't actually do chacha | 20:54 |
| @clarkb:matrix.org | Both sides agree they can do aes256-ctr. But the side that is picky (the server fips side) is not prioritizing its ciphers properly | 20:55 |
| @clarkb:matrix.org | at least that is how I read the log | 20:56 |
| @jim:acmegating.com | (*i don't disagree with clark -- just pointing out that you'd want it for both trusted_ and untrusted_ro_paths if you decide to do that) | 20:57 |
| @tristanc_:matrix.org | corvus: unfortunately we can't really change the fips configuration, so we'll mount a custom ssh config to fix the client. Thanks for the tips. For what its worth, here is a fix proposal: https://softwarefactory-project.io/r/c/software-factory/sf-config/+/26225 | 21:04 |
| @clarkb:matrix.org | tristanC: well, maybe it indicates a bug in the sshd? | 21:05 |
| @clarkb:matrix.org | tristanC: let me put it another way: a default openssh client compiled by redhat should be able to ssh to an openssh server compiled by redhat with fips enabled and disabled | 21:05 |
| @clarkb:matrix.org | That appears to be broken for you and I would say that constitutes and issue likely in the sshd | 21:06 |
| @tristanc_:matrix.org | Clark: I think the expectation is that the ssh client uses the provided crypto-policies | 21:06 |
| @clarkb:matrix.org | I can kind of understand that except that the ssh protocol relies heavily on negotiating these items between client and server. If a server cannot support a particular setup then it shouldn't negotiation them. | 21:08 |
| @clarkb:matrix.org | In any case I'm glad that existing zuul configuration constructs allow this to be addressed | 21:09 |
| @clarkb:matrix.org | running `zuul-bwrap` out of the zuul published executor image then running ssh in that against a random server I haev at home seems to show it defaulting to chacha20 here too. I'm surprised that opendev's fips jobs all seem fine in that case. | 21:18 |
| @tristanc_:matrix.org | alternatively, we could always bind mount /etc/ssh and /etc/crypto-policies, but that would require adding the `-o StrictModes=no` to the ssh_args. | 21:21 |
| @jim:acmegating.com | https://tracing.opendev.org/trace/73f394c92d9fbe1943178eabfa6da6b1 is working now | 21:25 |
| @jim:acmegating.com | opendev is a few commits behind master, so we only have the queueitem and buildset traces now. | 21:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!