| *** rlandy|ruck|bbl is now known as rlandy|ruck | 00:35 | |
| *** rlandy|ruck is now known as rlandy|out | 00:44 | |
| *** ysandeep|out is now known as ysandeep | 04:47 | |
| fnordahl | Since yesterday, my SSH keys are no longer accepteed for git access to Gerrit. I tried to re-create the keys through the web UI but that does not appear to have helped. Anything up with the gerrit ssh key auth system atm? | 06:22 |
|---|---|---|
| *** amoralej|off is now known as amoralej | 07:34 | |
| frickler | fnordahl: did you possibly update your local ssh client or configuration? there was an issue which affected fedora 35 users amongst others | 07:53 |
| fnordahl | frickler: I'm running on the development release of Ubuntu, Jammy, so that is quite possible. Do you have any details on what changed? | 07:55 |
| frickler | fnordahl: let me try to find something | 07:56 |
| *** ysandeep is now known as ysandeep|lunch | 07:58 | |
| fnordahl | I see from the changelog that RSA signatures using SHA-1 is dropped, but neither my key or review.opendev.org keys appear to be in that category | 08:01 |
| frickler | fnordahl: it's a question of bad negotiation. one workaround is to use en ed25519 key instead. and I just did a local test and it seems that indeed jammy has recently updated to openssh 8.8 which changed the client behavior | 08:04 |
| frickler | I only found https://lists.opendev.org/pipermail/service-discuss/2021-October/000291.html so far which describes the issue. there is also some other workaround setting some ssh option | 08:05 |
| fnordahl | frickler: great, thank you for the pointer. I'll try to find a workaround from that. The opendev gerrit does have a ED25519 key as one of its server keys, but it also have others: https://pastebin.ubuntu.com/p/CmpYh8J4qW/ | 08:08 |
| frickler | fnordahl: well the workaround would be for you to use an ed25519 key instead of your rsa key. the other option is to add "-o 'PubkeyAcceptedKeyTypes +ssh-rsa'" to your ssh command or ssh config | 08:11 |
| fnordahl | frickler: that SSH option worked! | 08:19 |
| fnordahl | For future travelers with OpenSSH 8.8, popping this into your ~/.ssh/config will get you sorted: https://pastebin.ubuntu.com/p/dG3Qws4MQs/ | 08:19 |
| *** jpena|off is now known as jpena | 08:33 | |
| *** ysandeep|lunch is now known as ysandeep | 08:41 | |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: DNM Add option to download log files to the directory https://review.opendev.org/c/openstack/ci-log-processing/+/830337 | 09:48 |
| *** bshephar1 is now known as bshephar | 10:58 | |
| *** ysandeep is now known as ysandeep|dr_appt | 10:59 | |
| *** rlandy|out is now known as rlandy|ruck | 11:15 | |
| *** dviroel|out is now known as dviroel | 11:21 | |
| fungi | fnordahl: yes, the reason we haven't been specifically recommending that is that it does downgrade host authentication security compared to newer openssh defaults (though by downgrading them to the previous defaults, so it's probably reasonably safe if that's your preference) | 12:24 |
| fungi | we have a long-standing bug with gerrit upstream trying to get their host key negotiation fixed | 12:25 |
| fungi | this started cropping up in fedora 23 over a year ago | 12:25 |
| fungi | if the built-in sshd for gerrit supported host key negotiation, it could continue to use rsa with a stronger hash and openssh would happily use that | 12:26 |
| fnordahl | fungi: I understand the reluctance to document that workaround, and great that there is a bug with gerrit to fix the root of the issue. | 12:43 |
| fnordahl | fungi: I don't really have a preference nor a choice, downgrading OpenSSH is not an option, so using this option specifically for the host review.opendev.org appears like a pragmatic path forward for me. | 12:44 |
| fungi | sure, it's what i'll probably end up doing too now that openssh 8.8 has landed in debian/unstable | 12:54 |
| *** afaranha_ is now known as afaranha | 12:59 | |
| fungi | oh! looks like it's probably fixed in gerrit's master branch as of last month: https://bugs.chromium.org/p/gerrit/issues/detail?id=12758#c7 | 13:01 |
| fungi | "Bump sshd version to 2.8.0 and update jgit to 56f45e36d [...] SSHD-1216: Server-side implementation of the RFC 8332 server-sig-algs extension: the server announces that it prefers the SHA-2 signatures for RSA keys." | 13:02 |
| *** ysandeep|dr_appt is now known as ysandeep | 13:02 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:04 | |
| fungi | likely not trivially backportable, so may have to wait for us to upgrade to 3.6 | 13:04 |
| fungi | yeah, i don't see it in the stable-3.5 branch history | 13:07 |
| fungi | (and we're still on 3.4 at the moment) | 13:07 |
| *** amoralej is now known as amoralej|lunch | 13:08 | |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: DNM Add option to download log files to the directory https://review.opendev.org/c/openstack/ci-log-processing/+/830337 | 13:25 |
| *** amoralej|lunch is now known as amoralej | 14:13 | |
| frickler | hmm, I tested with 3.5.0.1-1504-gd4e5d1cbb8 which should be the latest master and it doesn't seem to fix the issue | 14:15 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 14:23 | |
| fungi | frickler: and it included mina-sshd 2.8.0? | 14:26 |
| fungi | maybe it also needs to be configured in gerrit | 14:26 |
| frickler | I'm not sure how to check what it includes, I just built a container according to https://www.github.com/GerritCodeReview/docker-gerrit and ran it in the zuul quickstart environment | 14:31 |
| frickler | actually it says so: Remote protocol version 2.0, remote software version GerritCodeReview_3.5.0.1-1504-gd4e5d1cbb8 (APACHE-SSHD-2.8.0) | 14:34 |
| fungi | huh, so it's got the fix in theory, but maybe the extension has to be explicitly enabled by gerrit or something | 14:35 |
| fungi | or maybe the fix is broken | 14:35 |
| *** ysandeep is now known as ysandeep|dinner | 14:39 | |
| frickler | my money is on the latter option. I tried setting sshd.enableDeprecatedKexAlgorithms = true but that didn't help, either | 14:43 |
| fungi | i guess followup to the chromium and/or apache bug trackers is in order if we're sure it still doesn't work | 14:47 |
| *** ysandeep|dinner is now known as ysandeep | 15:10 | |
| *** dviroel is now known as dviroel|lunch | 15:11 | |
| opendevreview | yatin proposed openstack/project-config master: Update Neutron's Dashboard as per recent changes https://review.opendev.org/c/openstack/project-config/+/830440 | 15:15 |
| clarkb | fungi: frickler: I think mina 2.8.0 only fixes it for the client side which gerrit uses for replication. The server side, which our users run into, needs the next mina update | 15:48 |
| clarkb | the problem there originates in mina itself marking the original bug that ianw filed as fixed when they only fixed the client side | 15:48 |
| clarkb | I had to respond (and maybe I craeted a new issue?) basically saying this only fixed half the problem and not the half of the problem we were concerned about | 15:49 |
| fungi | clarkb: not sure then why it mentions "Server-side implementation" in the release node from mina | 15:50 |
| fungi | the SSHD-1216 jira ticket seemed to be about fixing it for the server side once it was pointed out that the original fix was only on the client side | 15:50 |
| clarkb | https://issues.apache.org/jira/browse/SSHD-1141 is the original. THen 1216 is what should've addressed the server side | 15:50 |
| fungi | right | 15:51 |
| clarkb | https://issues.apache.org/jira/browse/SSHD-1216 and that does report 2.8.0 includes the fix | 15:51 |
| fungi | which should be included in 2.8.0 if i'm reading correctly | 15:51 |
| clarkb | so ya maybe the fix isn't complete | 15:51 |
| clarkb | https://bugs.chromium.org/p/gerrit/issues/detail?id=13930 is the gerrit side issue. Maybe we should update that issue? | 15:53 |
| clarkb | frickler: if you still have the up to date gerrit install looking at the ssh -vvv output is helpful. You should see the client try to do the kex negotiation and then the server ignore it. you can compare against the production gerrit to see the difference if there is one | 15:55 |
| frickler | clarkb: I already did that and found no difference | 15:55 |
| clarkb | My hunch then is this is due to how gerrit is contructing the server | 15:56 |
| clarkb | since the change on the mina side does seem to implement it, but it is in a default kex handler and maybe gerrit isn't applying that properly | 15:56 |
| opendevreview | Merged openstack/project-config master: Move missed repos under openstack-ansible-roles ACL https://review.opendev.org/c/openstack/project-config/+/829278 | 16:09 |
| clarkb | frickler: yes I think that is the issue. I can work on a change for upstream between meetings | 16:10 |
| clarkb | Harder for me to test as I don't have a gerrit master test setup right now though | 16:10 |
| frickler | clarkb: I just used the zuul quickstart setup and replaced the gerrit container | 16:11 |
| frickler | only change needed was to add "user: root" in the docker-compose | 16:12 |
| clarkb | frickler: what did you replace the container with? But also thats a good hint. Thanks | 16:14 |
| *** dviroel|lunch is now known as dviroel | 16:15 | |
| frickler | clarkb: built locally with https://www.github.com/GerritCodeReview/docker-gerrit and the instructions for the master branch there with Dockerfile-dev | 16:15 |
| frickler | so only "docker build -t gerritcodereview/gerrit:dev -f Dockerfile-dev ." | 16:16 |
| frickler | in ubuntu/20 | 16:16 |
| clarkb | thanks! | 16:17 |
| frickler | oh, there was some apt failure, too, needed to add DEBIAN_FRONTEND=noninteractive to the apt command | 16:18 |
| * frickler goes away for a bit, bbl | 16:19 | |
| clarkb | github is having issues which means I can't run my local build :) I suspect pushing to upstream won't help either. | 16:39 |
| clarkb | https://www.githubstatus.com/ reports everything is fine but https://github.com/bazelbuild/rules_nodejs/releases/download/5.1.0/rules_nodejs-5.1.0.tar.gz is a 503 for me. Is anyone else able to reach that tarball? | 16:44 |
| * clarkb finds breakfast and hopes that github manages to fix this in the meantime | 16:45 | |
| fungi | 200 OK | 16:48 |
| fungi | maybe there's a bad cdn endpoint | 16:48 |
| fungi | i was able to download it fine | 16:48 |
| clarkb | hrm ya still failing here. Probably their cdn then | 17:02 |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: DNM Add option to download log files to the directory https://review.opendev.org/c/openstack/ci-log-processing/+/830337 | 17:03 |
| clarkb | https://github.com/github/feedback/discussions/11915 I'm not the only one | 17:08 |
| *** ysandeep is now known as ysandeep|out | 17:16 | |
| *** Guest7 is now known as diablo_rojo_phone | 17:24 | |
| *** diablo_rojo_phone is now known as Guest229 | 17:24 | |
| clarkb | ok github doesn't explode anymore but now my bazel is too old. Time to look at their directions for master builds | 17:41 |
| *** jpena is now known as jpena|off | 17:42 | |
| *** amoralej is now known as amoralej|off | 17:45 | |
| opendevreview | Merged openstack/project-config master: Update Neutron's Dashboard as per recent changes https://review.opendev.org/c/openstack/project-config/+/830440 | 18:51 |
| clarkb | debug1: kex_input_ext_info: server-sig-algs=<...rsa-sha2-512,rsa-sha2-256,ssh-rsa> | 18:57 |
| clarkb | That shows up when I try to ssh to my local build. I haven't managed to configure it to accept a specific key (turns out this is a huge pita when you use an agent without key material on disk) | 18:58 |
| clarkb | I think I'll go ahead and push what I have after the infra meeting and then people can look at it closer | 18:58 |
| clarkb | I really wish that there was a way to say "use this key" without having material on disk | 18:58 |
| clarkb | seems like you have to have at least the pubkey on disk | 18:59 |
| clarkb | anyway that log line doesn't show up when I talk to review.opendev.org | 18:59 |
| clarkb | I think that implies this is working | 18:59 |
| clarkb | even if I don't have fully successful authentication | 18:59 |
| clarkb | also when you specify -i with a key that is on disk it tries your agent keys anyway | 19:00 |
| clarkb | why is ssh so obtuse about this | 19:00 |
| fungi | hiding details from users is apparently a sign of "usability" | 19:03 |
| clarkb | fungi: since you use rsa keys can you double check if that log line debug1: kex_input_ext_info: server-sig-algs shows up for you or not? | 19:04 |
| clarkb | I'm worried that since I talked ed25519 to review.o.o it may have short circuited and I'm missing something | 19:04 |
| clarkb | but since you use rsa it not showing up would be clear indication my change makes a difference? | 19:04 |
| clarkb | and I'll also try to figure out adding my test key to my local test setup to see if it can rsa with sha2 | 19:05 |
| fungi | what should i be trying to ssh into? | 19:05 |
| clarkb | fungi: review.o.o port 29418 using your normal rsa ssh key | 19:06 |
| clarkb | using at least -v | 19:06 |
| clarkb | I was doing gerrit ls-projects but any other command is probably fine too | 19:06 |
| fungi | clarkb: it does not show up for me with ssh -v to our gerrit, no | 19:07 |
| clarkb | thanks! I think that is pretty clear indication my fix does something :) | 19:07 |
| clarkb | I'll get the change pushed up after the meeting | 19:08 |
| fungi | cool, thanks | 19:08 |
| *** tbarron is now known as Guest252 | 19:40 | |
| clarkb | I was able to confirm using rsa to a build of my change worked after adding my key to the account. Now I'm building an upstream build without my change in it to perform the same test which should fail | 19:43 |
| clarkb | cool and now confirmed what frickler was seeing with the upstream war | 19:47 |
| clarkb | frickler: good catch on that and thank you for saying something. Would've been frustrating if 3.6 released and we assumed the problem was fixed only for it to still be broken :) | 19:54 |
| fungi | indeed! | 19:55 |
| frickler | well I had the zuul setup still up from some other test, and I hoped to be able to simply confirm the fix instead of falsifying | 20:07 |
| *** dviroel is now known as dviroel|brb | 21:23 | |
| *** dviroel|brb is now known as dviroel | 21:40 | |
| *** rlandy|ruck is now known as rlandy|out | 23:57 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!