| *** promethe- is now known as prometheanfire | 02:47 | |
| *** ralonsoh_ is now known as ralonsoh | 07:36 | |
| bauzas | morning folkks | 08:45 |
|---|---|---|
| opendevreview | Sylvain Bauza proposed openstack/nova master: Add the 2023.1 Antelope prelude section https://review.opendev.org/c/openstack/nova/+/875380 | 09:27 |
| sean-k-mooney | bauzas: https://review.opendev.org/c/openstack/nova/+/875380/2..3/releasenotes/notes/antelope-prelude-4a99907b00e739f8.yaml is still tecnilaly not correct | 09:35 |
| sean-k-mooney | 2024.1 is the first SLURP release offically for all project | 09:36 |
| bauzas | no | 09:36 |
| sean-k-mooney | yes | 09:36 |
| bauzas | see the TC resolution | 09:36 |
| bauzas | 2023.1 is the first 'SLURP release' but upgrade from yoga is considered a test | 09:37 |
| bauzas | we indeed need to guarantee 2023.1 to 2024.1 upgrades | 09:37 |
| bauzas | but both are 'officially' SLURP releases | 09:37 |
| sean-k-mooney | calling it a SLURP release imples its supproted by all project form yoga | 09:38 |
| sean-k-mooney | which was never a requirement | 09:38 |
| bauzas | the resolution is clear about the requirements | 09:38 |
| bauzas | but there is a TC meeting today, we can clarify the resolution there | 09:38 |
| sean-k-mooney | the requirement is to support upgrades form A->C | 09:38 |
| sean-k-mooney | but that means C is the first SLURP release | 09:39 |
| sean-k-mooney | its the release you are upgrading too that has to support it | 09:39 |
| bauzas | "Communication: We will use “SLURP” word to designate a SLURP release in release page, release notes page or any other place we want to communicate it. We can also use its full form “Skip Level Upgrade Release Process” if needed. A “not-SLURP” release will not be designated with anything and not having “SLURP” word is enough to communicate that this is not “SLURP” release. Also, the number schema in the releas | 09:39 |
| bauzas | e naming process will help all of us to relate which release is “SLURP”." | 09:39 |
| bauzas | " Testing: Just as we test and guarantee that upgrades are supported between adjacent releases today, we will also test and guarantee that upgrades between two “SLURP” releases are supported. Upgrades are tested for most projects today with grenade. A skip-level job will be maintained in the grenade repository that tests a normal configuration between the last two “SLURP” releases. The job will be updated on every new | 09:40 |
| bauzas | “SLURP” release, and there will always be a regular single-release grenade job testing between the previous release and current one, as we have today." | 09:40 |
| sean-k-mooney | that why i made the disticiton wiht A being the first __base__ release you upgrade form to the first SLURP release | 09:40 |
| bauzas | so we need to guarantee between *two* SLURP releases | 09:40 |
| bauzas | Yoga being *not* a SLURP release, we don't need to guarantee it, despite 2023.1 *is* a SLURP release | 09:40 |
| bauzas | " Our letter-based release naming scheme is about to wrap back around to A, so the proposal is that the “new A” release be the first one where we enforce this scheme. Y->A should be a “dress rehearsal” where we have the jobs enabled to help smoke out any issues, but where hard guarantees are not yet made." | 09:41 |
| sean-k-mooney | again i disagre not on the funtionalty but that the resolution is clear an the terminology is defiend properly | 09:41 |
| sean-k-mooney | yes i read that and that does not make A a SLURP release | 09:41 |
| bauzas | As a reminder, OpenStack 2023.1 is our first `Skip-Level-Upgrade Release`__ (starting from now, we name it a `SLURP release`) where you can rolling-upgrade your compute services from OpenStack Yoga as an experimental feature. Next SLURP release will be 2024.1. | 09:42 |
| bauzas | I just said two informations : | 09:42 |
| bauzas | 1/ 2023.1 *is* a SLURP release | 09:42 |
| bauzas | 2/ upgrade from Yoga is not guaranteed | 09:42 |
| bauzas | IIUC, you disagree on #1 | 09:43 |
| bauzas | but IMHO the resolution is quite clear | 09:43 |
| bauzas | https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adjustment.html#example-sequence even shows it | 09:43 |
| sean-k-mooney | correct with the caveat that 2024.1 is a slurp relase and upgrade will be guarenteed form 2023.1 | 09:43 |
| bauzas | I don't disagree on your last sentence | 09:44 |
| bauzas | both are SLURP releases, but the only guaranteed skip-level upgrade is from 2023.1 to 2024.1 | 09:44 |
| sean-k-mooney | for me a SLURP release guarnetes upgrade of more then one release | 09:44 |
| bauzas | that's what I disagree, based on my reading of the TC resolution :) | 09:45 |
| sean-k-mooney | that the thing the guarentee is a majory part of the definiton fo a slurp release | 09:45 |
| sean-k-mooney | if we dont have that in my book its wrong to call it one | 09:45 |
| sean-k-mooney | which is why i dont think we should call 2023.1 a SLURP release | 09:45 |
| bauzas | the 'P' is important | 09:45 |
| bauzas | 2023.1 is in the process of a skip-level-upgrade | 09:45 |
| sean-k-mooney | it is the base releae for a skiplevel upgrade | 09:46 |
| bauzas | yes | 09:46 |
| sean-k-mooney | but it is not a skip level upgrade release its self | 09:46 |
| bauzas | it's a 'SLURP release' as per the TC charter :) | 09:46 |
| sean-k-mooney | i think there is ambiguity in the wrodign and the curent statement is controditory | 09:46 |
| bauzas | hence me saying that the rollingupgrade feature is officially experimental | 09:47 |
| bauzas | it clarifies the expectations | 09:47 |
| bauzas | this is just a demo showcase | 09:47 |
| sean-k-mooney | if A cant guarentee a skip level upgrade form a previos release i dont think it can be considerd a slurp release so i think "Deployments wishing to move to a one year upgrade cycle will synchronize on a “SLURP” release, and then skip the following “not-SLURP” release, upgrading when the subsequent “SLURP” is released" is not sufficent | 09:48 |
| bauzas | since Yoga *isn't* a SLURP release | 09:48 |
| bauzas | well, | 09:48 |
| bauzas | "we will also test and guarantee that upgrades between two “SLURP” releases are supported. " is the key information | 09:48 |
| sean-k-mooney | i think that sentance should be changed and we should add 2 new deffintion to the details section | 09:48 |
| bauzas | that's a TC amendment, feel free to drop a patch | 09:49 |
| bauzas | I don't disagree this can be confusing, but I think I avoided it by the prelude | 09:49 |
| sean-k-mooney | i can but as it stand i dont think the prelude is correct. i wont block it but i think its a little misleading | 09:49 |
| bauzas | sean-k-mooney: and honestly, we have an internal meeting conflicting at the same time of the TC meeting, but I'd like to attend this TC call then to clarify the wordings | 09:50 |
| bauzas | lemme ask the TC folks in their chan what they think about the namings | 09:50 |
| bauzas | sean-k-mooney: feel free to drop again a comment on my prelude patch | 09:53 |
| bauzas | I'll show it to the TC folks | 09:53 |
| bauzas | sean-k-mooney: anyway, pointed it out in the TC chan | 09:57 |
| sean-k-mooney | i am going to push a patch to the governace repo for review | 10:01 |
| bauzas | cool thanks | 10:01 |
| bauzas | sean-k-mooney: the odds of reno make me think that we should merge the prelude without waiting for the TC clarification | 10:02 |
| bauzas | but if the TC agrees on saying 'A isn't a SLURP release' then we could later modify the prelude | 10:03 |
| bauzas | given the first commit will be included in the 2023.1 branch, we won't miss it | 10:03 |
| sean-k-mooney | https://review.opendev.org/c/openstack/governance/+/875853 | 10:06 |
| sean-k-mooney | yes we can backport a change to the prelude | 10:06 |
| bauzas | sean-k-mooney: fwiw, said +1 to placement rc1 https://review.opendev.org/c/openstack/releases/+/875452 even if I know that we have a concurrent db cleanup patch | 10:12 |
| *** ralonsoh__ is now known as ralonsoh | 10:18 | |
| opendevreview | Sylvain Bauza proposed openstack/nova master: Add service version for Antelope https://review.opendev.org/c/openstack/nova/+/874932 | 10:19 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: DNM (yet) Update min support for Bobcat https://review.opendev.org/c/openstack/nova/+/875621 | 10:19 |
| opendevreview | 周众 proposed openstack/nova master: Instance unexpected shutdown when source node startup https://review.opendev.org/c/openstack/nova/+/875859 | 10:22 |
| opendevreview | 周众 proposed openstack/nova master: Instance unexpected shutdown when source node startup https://review.opendev.org/c/openstack/nova/+/875859 | 10:25 |
| sean-k-mooney | bauzas: ah the unique constatit one | 10:27 |
| sean-k-mooney | ya that is an existing issue that was not intoduced in this release so it woudl not be an RC candiate bug anyway | 10:28 |
| sean-k-mooney | it can be backported after we do the release | 10:28 |
| sean-k-mooney | so im fine with the +1 there | 10:28 |
| opendevreview | zhouzhong proposed openstack/nova master: Instance unexpected shutdown when source node startup https://review.opendev.org/c/openstack/nova/+/875859 | 10:30 |
| opendevreview | zhouzhong proposed openstack/nova master: Instance unexpected shutdown when source node startup https://review.opendev.org/c/openstack/nova/+/875859 | 10:32 |
| opendevreview | zhouzhong proposed openstack/nova master: Instance unexpected shutdown when source node startup https://review.opendev.org/c/openstack/nova/+/875859 | 10:33 |
| opendevreview | OpenStack Release Bot proposed openstack/placement stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/placement/+/875873 | 10:50 |
| opendevreview | OpenStack Release Bot proposed openstack/placement stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/placement/+/875874 | 10:50 |
| opendevreview | OpenStack Release Bot proposed openstack/placement master: Update master for stable/2023.1 https://review.opendev.org/c/openstack/placement/+/875875 | 10:50 |
| opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | 10:58 |
| bauzas | folks, placement-rc1 tag is done, master is now on Bobcat and Antelope is branched for Placement, I repeat, Antelope is branched for Placement | 11:05 |
| gibi | bauzas: \0/ | 11:55 |
| sean-k-mooney | a day ahead of schdule and all :P | 12:00 |
| sean-k-mooney | but cool | 12:00 |
| sean-k-mooney | the non client libs like os-vif and os-traits are also already branched | 12:01 |
| sean-k-mooney | well at least os-vif is but i assume the same is true for the rest | 12:01 |
| bauzas | sean-k-mooney: indeed, I approved https://review.opendev.org/c/openstack/releases/+/874450 | 13:00 |
| *** lowercase is now known as Guest6298 | 14:48 | |
| *** lowercase_ is now known as lowercase | 14:48 | |
| dansmith | bauzas: so my skip-level job failed because we can't ssh to the guest | 15:07 |
| dansmith | it's clearly working (reachable) but rejecting maybe the key or something | 15:07 |
| dansmith | can you think of what about running on jammy would affect that? | 15:08 |
| sean-k-mooney | could it be related to sha1/rsa keys | 15:12 |
| sean-k-mooney | i think tempest is using ec keys now | 15:12 |
| sean-k-mooney | but thats a guess | 15:12 |
| dansmith | that's what I was thinking too, but we're generating the keys with osc | 15:12 |
| dansmith | oh, but maybe jammy as the host refuses to use them? | 15:13 |
| dansmith | I would think that if we needed a change, that'd be wrapped up in grenade already though... | 15:13 |
| sean-k-mooney | fedora reject rsa keys. osc will only generate rsa keys because that all nova does | 15:13 |
| sean-k-mooney | so unless its gnereated in tempest itself it would fail if jammy rejects them | 15:13 |
| sean-k-mooney | with that said im expecting that job to still use tempest master right | 15:14 |
| dansmith | this is pre-tempest, this is grenade resource creation | 15:14 |
| sean-k-mooney | we dont pin it or anything strange for grenade | 15:14 |
| bauzas | dansmith: lemme look at your change | 15:14 |
| dansmith | sean-k-mooney: https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L186 | 15:15 |
| dansmith | sean-k-mooney: https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L121 | 15:15 |
| dansmith | and when we try to ssh: | 15:15 |
| dansmith | 2023-02-28 20:55:58.894 | debug1: Will attempt key: /opt/stack/save/cinder_key.pem explicit | 15:15 |
| dansmith | 2023-02-28 20:55:58.897 | debug1: SSH2_MSG_SERVICE_ACCEPT received | 15:15 |
| dansmith | ahh | 15:15 |
| dansmith | 2023-02-28 20:55:58.902 | sign_and_send_pubkey: no mutual signature supported | 15:16 |
| dansmith | there ^ | 15:16 |
| dansmith | but surely grenade is running on jammy nodes already, so I'm not sure why it wouldn't have been configured to support the old keys | 15:16 |
| sean-k-mooney | dansmith: proably not | 15:18 |
| sean-k-mooney | for antelop i think it used focal | 15:18 |
| dansmith | looks like not yeah | 15:18 |
| dansmith | okay, well, I can fix that then I guess | 15:19 |
| sean-k-mooney | so likely we jsut need to call ssh-keygen directly | 15:19 |
| sean-k-mooney | instead of osc | 15:19 |
| bauzas | wait | 15:19 |
| dansmith | well, I was going to configure the host to allow that key type | 15:19 |
| bauzas | https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L121 | 15:19 |
| bauzas | we no longer support this by Zed | 15:19 |
| bauzas | oh no, sorry | 15:20 |
| dansmith | bauzas: but it works if you use the old microversion, which osc is doing right? | 15:20 |
| bauzas | you're passing the key | 15:20 |
| bauzas | dansmith: ah right too | 15:20 |
| dansmith | also that :) | 15:20 |
| bauzas | but I wonder, that's probably why we have a problem with rsa | 15:20 |
| bauzas | could we maybe try to create ed keys ? | 15:21 |
| dansmith | I mean, I guess, but it seems more upgrade-y to allow the use of the older key type during the upgrade | 15:22 |
| dansmith | hmm, that keypair create *is* generating the key in nova right? | 15:23 |
| dansmith | $CINDER_KEY is just the key name | 15:23 |
| bauzas | yeah | 15:24 |
| bauzas | and by default it creates a ssh-rsa key | 15:24 |
| dansmith | looks like that's the only place we run keypair create, so maybe it's best to just generate it properly | 15:24 |
| bauzas | but you're creating it in Yoga, right? | 15:24 |
| bauzas | or in Zed ? | 15:24 |
| dansmith | it's master grenade | 15:25 |
| bauzas | against a master devstack then ? | 15:25 |
| bauzas | oh | 15:25 |
| bauzas | it defaults to the previous release before upgrading | 15:26 |
| bauzas | right? | 15:26 |
| dansmith | not a master devstack | 15:26 |
| bauzas | I mean the grenade base | 15:26 |
| dansmith | at this point in the phase, we've run devstack from yoga, we're running grenade from master to do some things against it, then upgrade to devstack master | 15:27 |
| bauzas | https://github.com/openstack/grenade/blob/master/grenaderc#L36 | 15:27 |
| bauzas | ok | 15:28 |
| bauzas | so | 15:28 |
| bauzas | we created a yoga devstack, we're running grenade against it | 15:28 |
| bauzas | which will create a key | 15:28 |
| bauzas | and after this, we would upgrade to devstack master | 15:28 |
| bauzas | so the environment that OSC calls is a Yoga nova-API service | 15:28 |
| bauzas | amirite ? | 15:29 |
| dansmith | yes | 15:29 |
| * bauzas looks at https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f59/875773/3/check/grenade-skip-level-continuous/f597bdf/job-output.txt | 15:30 | |
| dansmith | https://review.opendev.org/c/openstack/grenade/+/875940 | 15:31 |
| dansmith | that ^ is what I'm thinking | 15:31 |
| bauzas | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f59/875773/3/check/grenade-skip-level-continuous/f597bdf/controller/logs/grenade.sh_log.txt | 15:32 |
| bauzas | I see the problem | 15:32 |
| bauzas | but this is using jammy | 15:32 |
| opendevreview | Dan Smith proposed openstack/nova master: Add continuous skip-level job for nova https://review.opendev.org/c/openstack/nova/+/875773 | 15:33 |
| bauzas | so I guess ssh-rsa keys are no longer accepted | 15:33 |
| bauzas | since you created them with yoga, it was accepted by Nova | 15:33 |
| bauzas | and the guest was accordingly created | 15:33 |
| dansmith | I don't think it has anything to do with yoga, | 15:34 |
| dansmith | because even if we were running on jammy, we'd still be able to generate because of the old microversion right>? | 15:34 |
| bauzas | you're true | 15:34 |
| bauzas | nothing related | 15:34 |
| bauzas | we continue to accept to blindly create ssh-rsa keys by default | 15:35 |
| bauzas | if you don't opt(in | 15:35 |
| bauzas | but, jammy guests should no longer accept it | 15:35 |
| bauzas | lemme verify this | 15:35 |
| dansmith | it's not jammy guests, it's hosts | 15:35 |
| bauzas | oooh, then I understand | 15:35 |
| dansmith | the host won't even use the ssh key to try to ssh to a remote if it's not one of the allowed key types | 15:35 |
| dansmith | the error messaging for this has been terrible | 15:36 |
| * bauzas was wondering why grenade by hit, compared to the ssh connections we make in tempest | 15:36 | |
| dansmith | because it shows that it's trying the key, but it's not | 15:36 |
| dansmith | I struggled with this when I upgraded myself | 15:36 |
| dansmith | yeah | 15:36 |
| bauzas | dansmith: -1 on https://review.opendev.org/c/openstack/grenade/+/875940 | 15:42 |
| bauzas | IMHO, you need to specify another keytype to be safe | 15:43 |
| bauzas | I mean another algorithm | 15:43 |
| dansmith | that's been the default for years now right? | 15:43 |
| bauzas | https://transang.me/ssh-handshake-is-rejected-with-no-mutual-signature-algorithm-error/ | 15:43 |
| dansmith | oh, maybe not: | 15:44 |
| dansmith | "If invoked without any arguments, ssh-keygen will gen‐ | 15:44 |
| bauzas | no, I think we continue defaulting to ssh-rsa | 15:44 |
| dansmith | erate an RSA key." | 15:44 |
| dansmith | how stupid, when ssh will reject the default | 15:44 |
| dansmith | ffs :) | 15:44 |
| bauzas | another option on the client side is to invoke to accept the key | 15:44 |
| bauzas | +o PubkeyAcceptedKeyTypes +ssh-rsa | 15:45 |
| bauzas | when you ssh | 15:45 |
| dansmith | right I mentioned that above as another option | 15:45 |
| bauzas | or add it in your ssh config | 15:45 |
| bauzas | but I'd prefer to switch to the latest ed25519 if not edcsa | 15:45 |
| dansmith | but since we're already using a deprecated thing, it seems better to just move to what people should be doing, and since this wasn't generated by the old devstack, | 15:45 |
| bauzas | yeah | 15:45 |
| dansmith | it's not an upgrade-y thing like I mentioned | 15:45 |
| bauzas | agreed | 15:45 |
| opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | 15:48 |
| opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | 15:55 |
| opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Have schema for 'lock' action be applied to all microversions. https://review.opendev.org/c/openstack/nova/+/875653 | 15:58 |
| bauzas | gmann: fwiw, I added the new RBAC policies mention in the prelude https://review.opendev.org/c/openstack/nova/+/875380/3/releasenotes/notes/antelope-prelude-4a99907b00e739f8.yaml#49 | 15:59 |
| bauzas | feel free to vote on it | 15:59 |
| bauzas | gmann: also, I forgot about it because the original change wasn't correctly tracked on the nova side : no blueprint at least | 15:59 |
| bauzas | hence my miss | 15:59 |
| clarkb | bauzas: dansmith: RSA is fine with openssh. THe issue is openssh + rsa + sha1 | 16:10 |
| bauzas | I see | 16:10 |
| bauzas | ssh-rsa | 16:10 |
| dansmith | clarkb: oh, so rsa with ssh-keygen that uses sha256 would be fine/ | 16:10 |
| bauzas | by default ssh-rsa uses sha1 yes | 16:11 |
| clarkb | dansmith: at keygen time the sha version isn't a concern. It is negotiated at connection time | 16:11 |
| clarkb | but ya ssh-rsa is ssh + sha1. rsa-sha-256 or whatever the name is is the sha2 variant | 16:11 |
| dansmith | okay I'm confused then | 16:12 |
| bauzas | clarkb: I -1 on https://review.opendev.org/c/openstack/grenade/+/875940/1/projects/70_cinder/resources.sh#121 | 16:12 |
| clarkb | for a concrete example gerri's sshd didn't know how to negotiate rsa + sha2 with clients until recently (we opendev addressed that and now it works). This broke fedora who deprecated rsa + sha1 before everyone else | 16:12 |
| bauzas | clarkb: do you think it would be better to rather use rsa with sha256 ? | 16:12 |
| clarkb | today fedora users can use those same exact rsa keys to talk to gerrit because the gerrit sshd will now negotiate with the fedora ssh client to use sha2 | 16:12 |
| clarkb | bauzas: if ou want to use rsa then ya that is he non deprecated option | 16:13 |
| clarkb | one consideration here is fips testing. fips doesn't allow ed25519 keys. Which means you're stuck using ecdsa or rsa | 16:13 |
| clarkb | Personally when all this started becoming a problem I switched my personal things to ed25519 as I don't care about fips at home | 16:14 |
| bauzas | clarkb: ok, then your preference would be "-t rsa -E sha256 " correct ? | 16:14 |
| bauzas | for ssh-keygen | 16:14 |
| clarkb | bauzas: the sha version shouldn't matter at keygen time its all dynamic protocol negoatiation | 16:14 |
| clarkb | bauzas: is this talking to cirros? | 16:15 |
| clarkb | if so cirros' dropbear server may not support sha2 negotiation which would be a similar problem to what we addressed in Gerrit | 16:16 |
| bauzas | clarkb: context https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f59/875773/3/check/grenade-skip-level-continuous/f597bdf/controller/logs/grenade.sh_log.txt | 16:17 |
| bauzas | clarkb: coming from https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L186 failing | 16:17 |
| clarkb | ya thats cirros. And the test node must be jammy? | 16:18 |
| bauzas | we generate ssh-rsa keys by default with jammy | 16:18 |
| bauzas | yup | 16:18 |
| clarkb | I think your options are: use newer cirros with newer dropbear, use ecdsa/ed25519 key if dropbear supports these, explicitly override configs to allow rsa + sha1 on the jammy side to talk to dropbear that only knows this combo | 16:18 |
| clarkb | Part of the problem here is that when everyone deprecated sha1 they kept the fallback default in protocol negotiation as rsa + sha1 despite knowing it will not work. It would've been better if they updated the fallback default to be rsa + sha2 which would have worked with gerrit its only problem was understanding how to negotiate things, not lack of sha2 support | 16:19 |
| clarkb | dropbear version 2020.79 added support for both ed25519 keys and rsa sha2 | 16:21 |
| clarkb | the easiest thing is probably using ecdsa keys and that would also satisfiy fips | 16:22 |
| bauzas | clarkb: cool, then | 16:27 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/train: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833438 | 16:33 |
| gmann | bauzas: thanks for RBAC things in prelude. I will check | 17:08 |
| bauzas | cool | 17:08 |
| dansmith | clarkb: okay, so you're saying the different key type ends up using a different hash algo in the negotiation and that's what sidesteps the problem? | 17:09 |
| dansmith | I thought it was the key type itself | 17:09 |
| clarkb | dansmith: correct | 17:10 |
| dansmith | hrm | 17:10 |
| clarkb | all the newer keys use hashes that are secure enough to have not been replaced | 17:10 |
| clarkb | rsa didn't so has extra negotiation requirements when setting up a connection | 17:10 |
| dansmith | but I thought you said the hash wasn't part of the key? | 17:10 |
| clarkb | its the key exchange extensions bit of the ssh protocol | 17:10 |
| clarkb | sorry correct to the first statement not to the second | 17:10 |
| clarkb | this is not part of the key itself. It is part of how the key is used at connection time | 17:11 |
| dansmith | because I thought even ssh'ing between two newer machines still fails with the old key | 17:11 |
| clarkb | it shouldn't | 17:11 |
| dansmith | that hasn't been my experience, but perhaps I was just missing what the linkage was | 17:12 |
| clarkb | I think cirros + dropbear and gerrit + mina sshd may have created confusion around that? But new openssh to new opensshd should be fine | 17:13 |
| dansmith | I had to enable two things over the course of the last few years: | 17:13 |
| dansmith | PubkeyAcceptedKeyTypes +ssh-rsa | 17:13 |
| dansmith | and | 17:13 |
| dansmith | KexAlgorithms +diffie-hellman-group1-sha1,diffie-hellman-group14-sha1 | 17:13 |
| dansmith | seems like the latter must be the part you're talking about | 17:14 |
| dansmith | but at some point, something (one of my systems, which might have been macos) stopped even considering rsa keys | 17:14 |
| dansmith | although some docs make the PubkeyAcceptedTypes sound like maybe it's not the key type? I dunno, it's just confusing (to me) :) | 17:17 |
| bauzas | iiuc, it's a matter of negotiation like clarkb said | 17:26 |
| bauzas | the current interim solution allows the existing keys to continue to work by asking the server to negociate with them | 17:26 |
| bauzas | later, "they" could drop support of those keys and then the pubkeyacceptedtypes hint wouldn't work | 17:27 |
| bauzas | the idea of that option hint is to leave a graceful period to ops to generate new keys and deliver their public ones into the servers | 17:27 |
| bauzas | but like every other migration helper, people will rather use the workaround instead of migrating their keys | 17:28 |
| bauzas | until the full removal arrives | 17:28 |
| dansmith | bauzas: you see that documented somewhere/ | 17:35 |
| dansmith | because if I read what clarkb is saying, there's nothing wrong with rsa keys themselves... | 17:35 |
| bauzas | well, I need to remember that situation | 17:35 |
| dansmith | back in the day, I migrated all my keys to dsa because that was "the way" and then dsa was dropped and I had to migrate back | 17:36 |
| dansmith | so forgive me if I'm a bit skeptical :D | 17:36 |
| clarkb | correct you should not need to generate new keys to fix this | 17:37 |
| clarkb | its purely a I'm making a connection we need to negotiate details and can't agree on those details problem | 17:37 |
| bauzas | https://www.rfc-editor.org/rfc/rfc8332#section-5.2 | 17:37 |
| bauzas | "Nevertheless, implementations SHOULD start to disable "ssh-rsa" in their default configurations as soon as the implementers believe that new RSA signature algorithms have been widely adopted." | 17:38 |
| dansmith | clarkb: so why do I need the PubkeyAcceptedTypes change? | 17:39 |
| clarkb | dansmith: because ssh-rsa means sha1 | 17:40 |
| clarkb | dansmith: the protocl uses something liks rsa-sha-256 to denote the sha2 varints | 17:40 |
| dansmith | clarkb: so is the option really "PubkeyExchangeNegotiationTypes" ? | 17:40 |
| clarkb | and the server doesn't know to negotiate rsa-sha-256 because the default when you don't know how to negotiate is ssh-rsa | 17:40 |
| dansmith | like, the key is rsa, and ssh-rsa is not the key type *on disk* but the key type "in our negotiation" ? | 17:41 |
| clarkb | dansmith: sort of? I think what goes on is both sides need to agree on the pubkey type they will accept. The server if old like dropbear says ssh-rsa but not rsa-sha-256. Your client is new and only says rsa-sha-256 and there is no overlp and it fails | 17:41 |
| dansmith | I guess the thing that is confusing is that I consider the "type of the key" to be a property or characteristic of the content of my file on disk | 17:42 |
| dansmith | but it sounds like that's not what "Pubkey .. Type" means both to ssh over the wire and in its config | 17:42 |
| clarkb | correct | 17:42 |
| dansmith | all I want is you to hug me and tell me I'm not crazy for finding that confusing :D | 17:43 |
| clarkb | you are not crzy. When we first ran into this with gerrit and fedora making the change early there was a couple days of major head scratching | 17:43 |
| sean-k-mooney | fedora used to have a tool to cofniuure thigns https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 | 17:54 |
| *** jamesdenton_ is now known as jamesdenton | 17:54 | |
| sean-k-mooney | and yes you can use rsa-sha2-256 or rsa-sha2-512 | 17:55 |
| bauzas | clarkb: you have way more background than me on this one, do you agree with me on the fact that the current pubkey hint will be later dropped, if I read carefully https://www.rfc-editor.org/rfc/rfc8332#section-5.2 ? | 17:55 |
| sean-k-mooney | so you would add PubkeyAcceptedKeyTypes +rsa-sha2-256 to the sshconfig | 17:55 |
| sean-k-mooney | after i had it working and then it broke with gerrit later i jsut gave in and stopped using rsa keys | 17:56 |
| bauzas | sean-k-mooney: my thought was that PubkeyAcceptedKeyTypes +ssh-rsa was only a temporary workaround until all OpenSSH clients were supposed new enough to drop this compat natively | 17:58 |
| sean-k-mooney | yes but you can disable his on the server side too | 17:58 |
| sean-k-mooney | for gerrit it did not supprot swaping to sha2 until very recently | 17:59 |
| clarkb | bauzas: what that is saying is that ssh-rsa is going away which means rsa + sha1 is going away. There are no plans to remove rsa + sha2 as far as I know | 18:00 |
| bauzas | anyway, for the grenade need, generating a ecdsa key is OK | 18:00 |
| sean-k-mooney | bauzas: i fyou look at https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 | 18:00 |
| bauzas | clarkb: yeah that's what I found | 18:01 |
| sean-k-mooney | fedor aplanned to drop rsa for key excachange in a future defualt policy that was for f33 | 18:01 |
| sean-k-mooney | i think that came intor affect aroudn f35 | 18:01 |
| bauzas | that's not the RSA keys which are deprecated, that's the SHA1 signature algorithm which is (hence the +o PubkeyAcceptedKeyTypes +ssh-rsa be only a temporary workaround) | 18:01 |
| clarkb | sean-k-mooney: thats only rsa + sha1 | 18:02 |
| sean-k-mooney | key exchange: ECDHE, DHE | 18:02 |
| sean-k-mooney | under FUture | 18:02 |
| clarkb | wut | 18:02 |
| clarkb | oh dhe is rsa iirc | 18:03 |
| sean-k-mooney | anyway it sould liek a this is related to rsa/sha1 and we shoudl be able to fix that by using a ecdsa key which will work for fips too | 18:03 |
| sean-k-mooney | instead of hacking in the old key type which wont | 18:03 |
| clarkb | and if anyone can grok the openssh code better than I a PR to default to rsa + sha2 as teh fallback might generate interesting conversations | 18:05 |
| sean-k-mooney | so we should just replace https://github.com/openstack/grenade/blob/master/projects/70_cinder/resources.sh#L121 | 18:05 |
| sean-k-mooney | with a call to ssh-keygen | 18:05 |
| clarkb | the rfc suggests this happen eventually but as far as I can tell it hasn't happened yet which leads to annoying failures n a lot of cases | 18:05 |
| bauzas | sean-k-mooney: patch is up to change grenade https://review.opendev.org/c/openstack/grenade/+/875940/ | 18:06 |
| sean-k-mooney | cool | 18:07 |
| sean-k-mooney | so ye were just discussing why this is needed | 18:07 |
| bauzas | yeah | 18:07 |
| sean-k-mooney | rahter then trying to find a solution | 18:07 |
| sean-k-mooney | ok | 18:07 |
| dansmith | yeah because I definitely didn't | 18:07 |
| bauzas | well | 18:07 |
| bauzas | we are trying to untangle the oddness of ssh negociation | 18:07 |
| dansmith | and I have a bunch of hacks in my own ssh_config that probably need revisiting now that more of my systems are upgraded | 18:07 |
| bauzas | me too | 18:07 |
| bauzas | this discussion is half-workwise, halp-personalwise | 18:08 |
| sean-k-mooney | ok i got burned by this years ago and have helped other fix it in the past so i mostly just accpet it at this point | 18:08 |
| bauzas | I just shamelessly tuned the signature negociation on the fly with my ssh_config | 18:08 |
| bauzas | as I didn't wanted to generate a new pair of ECDSA or something else keys | 18:09 |
| sean-k-mooney | ya i still have PubkeyAcceptedKeyTypes +ssh-rsa for one site in my config | 18:09 |
| sean-k-mooney | but i think thats offline | 18:09 |
| bauzas | but now if I understand correctly, I could rather continue to use my RSA keys but ask for a different signature | 18:09 |
| sean-k-mooney | i also still have | 18:09 |
| sean-k-mooney | host review.opendev.org | 18:10 |
| bauzas | using rsa-sha2 | 18:10 |
| sean-k-mooney | HostKeyAlgorithms ssh-rsa | 18:10 |
| sean-k-mooney | KexAlgorithms +diffie-hellman-group1-sha1 | 18:10 |
| sean-k-mooney | Ciphers +aes128-cbc | 18:10 |
| sean-k-mooney | for some reason witch i relly dont need | 18:10 |
| clarkb | bauzas: correct. The problem is that some servers don't know how to negotiate that with you like old dropbear and old gerrit | 18:10 |
| sean-k-mooney | thats what i treid when PubkeyAcceptedKeyTypes +ssh-rsa stopped working for gerrit | 18:10 |
| bauzas | clarkb: so the per-host config is still required, gotcha | 18:11 |
| clarkb | the gerrit we have deployed at review.opendev.org should work fine though. ianw and I worked with gerrit and mina sshd upstream and got that all fixed and eventually got it backported to gerrit 3.5 (it was always on 3.6) | 18:11 |
| clarkb | bauzas: if the servers don't know how to negotiate rsa + sha2 | 18:11 |
| clarkb | review.opendev.org should not need this anymore | 18:11 |
| bauzas | yeah got it | 18:11 |
| bauzas | clarkb: how to enable rsa+sha2 globally in the config ? | 18:12 |
| clarkb | bauzas: if your openssh client is new (like on fedora or jammy etc) then thats the only version they will use by default | 18:12 |
| bauzas | because I assume my current OS doesn't have its defaults changed | 18:12 |
| bauzas | oh | 18:12 |
| clarkb | thats why it failed to talk to cirros. jammy will only use rsa + sha2 by default, but cirros dropbear will only do rsa + sha1 | 18:12 |
| clarkb | this mismatch leads to a failure to negotiate between them and no ssh connection | 18:13 |
| sean-k-mooney | i think you would add PubkeyAcceptedKeyTypes rsa-sha2-256 | 18:13 |
| sean-k-mooney | under "HOST *" | 18:13 |
| clarkb | sean-k-mooney: that shouldn't be necessary its automatic | 18:13 |
| clarkb | since all the new lcients only do sha2 | 18:13 |
| bauzas | hmmm | 18:13 |
| sean-k-mooney | ya it should not but if you wanted to force it that would be how | 18:13 |
| bauzas | yeah got it | 18:14 |
| bauzas | this chat is definitely a helper | 18:14 |
| bauzas | clarkb: thanks a lot ! | 18:14 |
| * sean-k-mooney still wishise osc/keystone support using ssh keys for auth | 18:15 | |
| * bauzas got those weird connection issues since Fedora 33 and never took decent time to understand more the problem despite thinking he would eventually need to generate a new pair of keys, which is now I know a wrong assumption | 18:15 | |
| sean-k-mooney | i really would like to just upload a pulic key to keystone and use that to authenticate with osc | 18:15 |
| sean-k-mooney | bauzas: for what its woth its in te seciryt enhancmetns secation fo the 22.04 release notes https://discourse.ubuntu.com/t/jammy-jellyfish-release-notes/24668 | 18:23 |
| sean-k-mooney | it was disabled by defult in openssh https://www.openssh.com/txt/release-8.8 | 18:25 |
| sean-k-mooney | after being deprecated in 8.3 https://lwn.net/Articles/821544/ | 18:26 |
| clarkb | right but they didn't change the fallback key | 18:42 |
| clarkb | so there are layers of problems here. The first is that you can no longer negotiate ssh-rsa with servers that need it. Second is that when that negotiate fails both sides expect ssh-rsa | 18:42 |
| clarkb | which of course fails making the fallback useless | 18:42 |
| clarkb | they should've changed the fallback to rsa-sha-256 so that clients would attempt that after a failed negotiation. This wold still fail on cirros but would've worked with gerrit | 18:43 |
| opendevreview | Merged openstack/nova stable/victoria: Fix the wrong exception used to retry detach API calls https://review.opendev.org/c/openstack/nova/+/866086 | 18:52 |
| opendevreview | Dan Smith proposed openstack/nova master: Add grenade-skip-level-always to nova https://review.opendev.org/c/openstack/nova/+/875773 | 22:39 |
| opendevreview | melanie witt proposed openstack/nova master: testing: Reset affinity support global variables https://review.opendev.org/c/openstack/nova/+/875991 | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!