| *** ociuhandu has joined #openstack-nova | 00:20 | |
| *** ociuhandu has quit IRC | 00:21 | |
| *** ociuhandu has joined #openstack-nova | 00:22 | |
| *** ociuhandu has quit IRC | 00:27 | |
| *** eandersson has quit IRC | 00:44 | |
| *** tbachman_ has joined #openstack-nova | 00:58 | |
| *** tbachman has quit IRC | 01:01 | |
| *** tbachman_ is now known as tbachman | 01:01 | |
| *** nanzha has joined #openstack-nova | 01:17 | |
| *** spsurya has joined #openstack-nova | 01:18 | |
| *** ociuhandu has joined #openstack-nova | 01:18 | |
| *** ociuhandu has quit IRC | 01:23 | |
| *** eandersson has joined #openstack-nova | 01:25 | |
| *** eandersson is now known as eandersson2 | 01:39 | |
| *** eandersson has joined #openstack-nova | 01:40 | |
| *** mlavalle has joined #openstack-nova | 01:44 | |
| *** mlavalle has quit IRC | 01:44 | |
| *** ceryx has joined #openstack-nova | 01:56 | |
| *** ociuhandu has joined #openstack-nova | 01:57 | |
| *** ociuhandu has quit IRC | 02:02 | |
| *** ociuhandu has joined #openstack-nova | 02:33 | |
| *** ociuhandu has quit IRC | 02:38 | |
| *** ociuhandu has joined #openstack-nova | 02:40 | |
| *** ociuhandu has quit IRC | 02:44 | |
| *** nanzha has quit IRC | 03:21 | |
| *** nanzha has joined #openstack-nova | 03:21 | |
| *** mkrai has joined #openstack-nova | 03:23 | |
| *** Liang__ has joined #openstack-nova | 03:23 | |
| *** Liang__ is now known as LiangFang | 03:25 | |
| *** psachin has joined #openstack-nova | 03:28 | |
| *** mkrai has quit IRC | 03:45 | |
| *** boxiang has joined #openstack-nova | 03:54 | |
| *** eandersson has quit IRC | 03:56 | |
| *** ceryx has quit IRC | 03:56 | |
| *** ceryx has joined #openstack-nova | 03:57 | |
| *** eandersson has joined #openstack-nova | 03:57 | |
| *** ceryx has quit IRC | 03:57 | |
| *** eandersson has quit IRC | 03:57 | |
| *** ceryx has joined #openstack-nova | 03:57 | |
| *** eandersson has joined #openstack-nova | 03:57 | |
| *** mkrai has joined #openstack-nova | 03:58 | |
| *** nanzha has quit IRC | 04:04 | |
| *** nanzha has joined #openstack-nova | 04:04 | |
| *** bhagyashris has joined #openstack-nova | 04:30 | |
| *** mkrai has quit IRC | 04:38 | |
| *** ociuhandu has joined #openstack-nova | 04:44 | |
| *** ociuhandu has quit IRC | 04:50 | |
| *** ricolin has joined #openstack-nova | 04:54 | |
| *** bhagyashris has quit IRC | 04:56 | |
| *** psachin has quit IRC | 05:10 | |
| *** zhanglong has joined #openstack-nova | 05:15 | |
| *** nanzha has quit IRC | 05:23 | |
| *** nanzha has joined #openstack-nova | 05:27 | |
| *** ratailor has joined #openstack-nova | 05:29 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Update keypairs in saving an instance object https://review.opendev.org/683043 | 05:30 |
|---|---|---|
| *** ociuhandu has joined #openstack-nova | 05:30 | |
| *** ociuhandu has quit IRC | 05:39 | |
| *** mkrai has joined #openstack-nova | 05:41 | |
| *** bhagyashris has joined #openstack-nova | 05:42 | |
| *** links has joined #openstack-nova | 06:17 | |
| *** bhagyashris_ has joined #openstack-nova | 06:24 | |
| *** bhagyashris has quit IRC | 06:27 | |
| *** bhagyashris__ has joined #openstack-nova | 06:28 | |
| *** bhagyashris_ has quit IRC | 06:29 | |
| *** takashin has left #openstack-nova | 06:30 | |
| *** tetsuro has joined #openstack-nova | 06:36 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | 06:58 |
| *** ociuhandu has joined #openstack-nova | 07:01 | |
| *** ociuhandu has quit IRC | 07:05 | |
| *** rcernin has quit IRC | 07:08 | |
| *** bhagyashris__ is now known as bhagyashris | 07:09 | |
| *** nanzha has quit IRC | 07:10 | |
| *** nanzha has joined #openstack-nova | 07:17 | |
| *** jmlowe has quit IRC | 07:19 | |
| *** ccamacho has joined #openstack-nova | 07:27 | |
| *** nanzha has quit IRC | 07:32 | |
| openstackgerrit | ya.wang proposed openstack/nova-specs master: Add spec for live migration no performance impact. https://review.opendev.org/693655 | 07:36 |
| *** danil has joined #openstack-nova | 07:37 | |
| danil | hello folks. Please help me. Is there a way to get nova client through openstacksdk? I mean that I can not just use novaclient library due to dependencies. Thanks | 07:38 |
| *** ccamacho has quit IRC | 07:39 | |
| *** jangutter has joined #openstack-nova | 07:45 | |
| *** boxiang has quit IRC | 07:47 | |
| *** zhubx has joined #openstack-nova | 07:47 | |
| *** gibi_ptg is now known as gibi | 07:58 | |
| * gibi is back but jetlagged | 08:00 | |
| *** maciejjozefczyk has joined #openstack-nova | 08:03 | |
| *** awalende has joined #openstack-nova | 08:09 | |
| *** tkajinam has quit IRC | 08:19 | |
| *** zhubx has quit IRC | 08:27 | |
| *** zhubx has joined #openstack-nova | 08:28 | |
| *** ccamacho has joined #openstack-nova | 08:33 | |
| *** ircuser-1 has quit IRC | 08:34 | |
| *** zhanglong has quit IRC | 08:37 | |
| *** zhanglong has joined #openstack-nova | 08:39 | |
| *** trident has quit IRC | 08:45 | |
| *** ociuhandu has joined #openstack-nova | 08:46 | |
| *** zhanglong has quit IRC | 08:49 | |
| *** ralonsoh has joined #openstack-nova | 08:49 | |
| *** zhanglong has joined #openstack-nova | 08:51 | |
| *** ociuhandu has quit IRC | 08:51 | |
| *** gshippey has joined #openstack-nova | 08:55 | |
| *** zhanglong has quit IRC | 08:56 | |
| *** slaweq has joined #openstack-nova | 08:57 | |
| *** zhanglong has joined #openstack-nova | 08:58 | |
| *** trident has joined #openstack-nova | 08:59 | |
| *** nanzha has joined #openstack-nova | 09:01 | |
| *** rcernin has joined #openstack-nova | 09:03 | |
| *** lpetrut has joined #openstack-nova | 09:03 | |
| *** luyao has joined #openstack-nova | 09:04 | |
| *** maciejjozefczyk has quit IRC | 09:10 | |
| *** maciejjozefczyk has joined #openstack-nova | 09:10 | |
| *** brinzhang has joined #openstack-nova | 09:15 | |
| *** trident has quit IRC | 09:16 | |
| *** ivve has joined #openstack-nova | 09:17 | |
| *** maciejjozefczyk has quit IRC | 09:17 | |
| *** zhanglong has quit IRC | 09:20 | |
| *** trident has joined #openstack-nova | 09:24 | |
| *** zhanglong has joined #openstack-nova | 09:30 | |
| *** slaweq has quit IRC | 09:31 | |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| *** avolkov has joined #openstack-nova | 09:39 | |
| *** maciejjozefczyk has joined #openstack-nova | 09:44 | |
| danil | hello folks. Please help me. Is there a way to get nova client through openstacksdk? I mean that I can not just use novaclient library due to dependencies. Thanks | 09:46 |
| *** maciejjozefczyk has quit IRC | 09:49 | |
| *** zhanglong has quit IRC | 09:55 | |
| *** rcernin has quit IRC | 09:56 | |
| *** ygk_12345 has joined #openstack-nova | 09:59 | |
| *** brinzhang has quit IRC | 10:01 | |
| *** brinzhang has joined #openstack-nova | 10:02 | |
| *** LiangFang has quit IRC | 10:03 | |
| *** brinzhang_ has joined #openstack-nova | 10:07 | |
| *** luksky has joined #openstack-nova | 10:09 | |
| *** brinzhang has quit IRC | 10:11 | |
| *** rcernin has joined #openstack-nova | 10:12 | |
| *** cdent has joined #openstack-nova | 10:43 | |
| *** mkrai has quit IRC | 10:46 | |
| *** dlbewley has quit IRC | 10:49 | |
| *** rcernin has quit IRC | 10:49 | |
| *** dlbewley has joined #openstack-nova | 10:50 | |
| *** brinzhang has joined #openstack-nova | 10:55 | |
| *** brinzhang_ has quit IRC | 10:58 | |
| *** bhagyashris has quit IRC | 11:06 | |
| *** ociuhandu has joined #openstack-nova | 11:10 | |
| *** ociuhandu has quit IRC | 11:15 | |
| *** ygk_12345 has quit IRC | 11:17 | |
| *** FlorianFa has quit IRC | 11:26 | |
| *** bhagyashris has joined #openstack-nova | 11:29 | |
| *** ociuhandu has joined #openstack-nova | 11:29 | |
| *** ociuhandu has quit IRC | 11:34 | |
| *** ygk_12345 has joined #openstack-nova | 11:46 | |
| *** pcaruana has joined #openstack-nova | 11:47 | |
| *** awalende has quit IRC | 11:48 | |
| *** ygk_12345 has left #openstack-nova | 11:49 | |
| *** awalende has joined #openstack-nova | 11:49 | |
| *** bhagyashris has quit IRC | 11:53 | |
| *** bhagyashris has joined #openstack-nova | 11:53 | |
| *** ociuhandu has joined #openstack-nova | 12:00 | |
| *** dviroel has joined #openstack-nova | 12:03 | |
| *** ociuhandu has quit IRC | 12:04 | |
| *** pcaruana has quit IRC | 12:09 | |
| *** danil has quit IRC | 12:09 | |
| *** CeeMac has joined #openstack-nova | 12:12 | |
| *** bhagyashris has quit IRC | 12:13 | |
| *** rcernin has joined #openstack-nova | 12:16 | |
| *** dave-mccowan has joined #openstack-nova | 12:26 | |
| *** dave-mccowan has quit IRC | 12:31 | |
| *** awalende has quit IRC | 12:33 | |
| *** awalende has joined #openstack-nova | 12:39 | |
| *** ratailor has quit IRC | 12:41 | |
| *** slaweq has joined #openstack-nova | 12:43 | |
| *** brinzhang has joined #openstack-nova | 12:44 | |
| *** rcernin has quit IRC | 12:44 | |
| brinzhang | dansmith: efried: In nova PTG in Shanghai we talked about this spec https://review.opendev.org/#/c/663563/, and we reached the agreement on this feature, you can see ML/Etherpad https://etherpad.openstack.org/p/nova-shanghai-ptg http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010643.html | 12:47 |
| *** rouk has joined #openstack-nova | 12:47 | |
| jroll | efried: I don't have anything, may be able to get something but not sure it'll be simple :) this might be a reasonable starting point: https://github.com/tpm2-software/tpm2-tools/wiki/Getting-Started | 12:47 |
| jroll | efried: if you don't like the looks of that, I can bug an expert here | 12:47 |
| *** slaweq has quit IRC | 12:47 | |
| brinzhang | dansmith: efried: I was updated (re-write) this spec, please review again (there are -2 on it now) | 12:48 |
| *** sean-k-mooney has quit IRC | 12:52 | |
| *** sean-k-mooney has joined #openstack-nova | 12:58 | |
| *** mkrai has joined #openstack-nova | 13:00 | |
| *** ociuhandu has joined #openstack-nova | 13:00 | |
| *** slaweq has joined #openstack-nova | 13:04 | |
| *** ociuhandu has quit IRC | 13:05 | |
| *** slaweq has quit IRC | 13:08 | |
| *** bbowen has joined #openstack-nova | 13:15 | |
| *** mriedem has joined #openstack-nova | 13:15 | |
| *** mkrai has quit IRC | 13:20 | |
| *** ociuhandu has joined #openstack-nova | 13:24 | |
| *** jaosorior has joined #openstack-nova | 13:35 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove TODO from ComputeTaskManager._live_migrate https://review.opendev.org/693696 | 13:38 |
| *** ociuhandu has quit IRC | 13:56 | |
| *** ociuhandu has joined #openstack-nova | 13:57 | |
| *** mkrai has joined #openstack-nova | 13:59 | |
| *** Erdosip has joined #openstack-nova | 14:01 | |
| Erdosip | Hy! | 14:01 |
| *** zhubx has quit IRC | 14:03 | |
| *** zhubx has joined #openstack-nova | 14:03 | |
| Erdosip | I would like to investigate about CEPH multiple pool support in the same nova process.. AFAIK this is not supported now, since nova only able to use one RBD, but we would like to use multiple... Are there any technical problems with this, to implement? | 14:03 |
| *** brinzhang_ has joined #openstack-nova | 14:04 | |
| *** tbachman has quit IRC | 14:04 | |
| Erdosip | I would like to investigate about CEPH multiple pool support in the same nova process.. AFAIK this is not supported now, since nova only able to use one RBD pool, but we would like to use multiple... Are there any technical problems with this, to implement? | 14:04 |
| Erdosip | (sorry, i thinked, i can edit my previous message...) | 14:05 |
| *** ociuhandu has quit IRC | 14:06 | |
| *** brinzhang has quit IRC | 14:07 | |
| *** nicholas has joined #openstack-nova | 14:13 | |
| *** slaweq has joined #openstack-nova | 14:15 | |
| *** mmethot has quit IRC | 14:15 | |
| *** priteau has joined #openstack-nova | 14:17 | |
| *** mmethot has joined #openstack-nova | 14:18 | |
| *** slaweq has quit IRC | 14:19 | |
| *** nweinber has joined #openstack-nova | 14:19 | |
| mriedem | Erdosip: this is likely a better question for the openstack-discuss mailing list, tag the subject line with [nova] so it's filtered properly | 14:23 |
| Erdosip | okay, thanks! | 14:24 |
| sean-k-mooney | Erdosip: the only way to connect to multiple cephs form the same nova host today is via cinder today as far as i know | 14:29 |
| *** bnemec has joined #openstack-nova | 14:29 | |
| *** bnemec has quit IRC | 14:29 | |
| sean-k-mooney | i dont know what it would take to support it with the rbd image backend but as mriedem says if you ask on the mailing list someone more familar with that may respond | 14:30 |
| *** maciejjozefczyk has joined #openstack-nova | 14:32 | |
| dansmith | brinzhang_: skimming your changes, it does not look like my core concerns are addressed, so I'll leave my -2 on there | 14:33 |
| * mriedem drives back to MN, back on in 4 hours or so | 14:38 | |
| *** mriedem has quit IRC | 14:38 | |
| *** luksky has quit IRC | 14:41 | |
| *** ociuhandu has joined #openstack-nova | 14:43 | |
| *** ociuhandu has quit IRC | 14:48 | |
| *** bnemec has joined #openstack-nova | 14:49 | |
| efried | brinzhang_: I'll look again | 14:52 |
| efried | jroll: oh, nice, good link, will read. | 14:53 |
| *** brinzhang has joined #openstack-nova | 14:53 | |
| jroll | :) | 14:54 |
| dansmith | Erdosip: I've planned to add a tiny bit of multiple-pool support to nova when glance becomes able to replicate images in multiple pools | 14:55 |
| *** brinzhang has quit IRC | 14:55 | |
| *** brinzhang has joined #openstack-nova | 14:55 | |
| dansmith | Erdosip: but it would just be the ability to know about multiple pools, there'd still be one "primary" pool that nova tries to get everything into in order to use them | 14:55 |
| *** brinzhang_ has quit IRC | 14:56 | |
| efried | jroll: I've gotten to the point where I can create an encrypted vTPM. I had found another "hello world" type script out there. It runs seamlessly whether encrypted or not -- that is, afaict the encryption/decryption happens at the libvirt level, and is transparent from the perspective of the VM. | 14:56 |
| *** brinzhang has quit IRC | 14:56 | |
| jroll | \o/ | 14:56 |
| jroll | efried: cool, that makes sense to me | 14:57 |
| *** brinzhang has joined #openstack-nova | 14:57 | |
| efried | There have been some points of awkwardness thus far, including the need to recycle the libvirtd service before it'll pick up the "secret" | 14:57 |
| jroll | or rather, aligns with how I understood the encrypted vtpm bits | 14:57 |
| jroll | eek | 14:57 |
| efried | Which would kind of be ... yeah, eek. | 14:57 |
| jroll | I hope that's an unintentional bug :) | 14:57 |
| efried | I'm still playing; it may be the case that creating the secret via the API rather than CLI could circumvent that -- not sure. | 14:57 |
| *** brinzhang has quit IRC | 14:58 | |
| efried | but even then, I'm still worried about your security requirements being satisfied here. | 14:58 |
| *** brinzhang has joined #openstack-nova | 14:58 | |
| efried | You create a "secret" in libvirt and give it a value. And then you create the libvirt XML with the UUID of that secret. And... that's it. There never seems to be a need to retype that secret value. | 14:59 |
| dansmith | efried: libvirt's cli is just an api client, in case you didn't know | 14:59 |
| dansmith | efried: virsh is, I mean | 14:59 |
| *** brinzhang has quit IRC | 14:59 | |
| efried | dansmith: I figured, point being to do it from the same process as wants to use the secret | 14:59 |
| *** brinzhang has joined #openstack-nova | 15:00 | |
| efried | sorry, I wasn't clear on that: use the libvirt API from within nova, rather than me typing in the CLI for my PoC. | 15:00 |
| jroll | efried: huh, yeah, I wonder where libvirt keeps that secret | 15:00 |
| efried | jroll: well, it keeps it in a file in a directory. | 15:00 |
| jroll | sigh. | 15:00 |
| dansmith | efried: yeah and my point is virsh and nova look the same to libvirtd, which is the thing that manipulates the guests, xmls, etc | 15:00 |
| jroll | the secret itself is in a file? | 15:00 |
| * jroll needs to play with this as well | 15:00 | |
| *** brinzhang has quit IRC | 15:01 | |
| sean-k-mooney | efried: ideally you would want to store the secret in the hosts tpm rather then on disk | 15:01 |
| *** brinzhang has joined #openstack-nova | 15:01 | |
| efried | dansmith: Okay, I think I understand what you're saying. There's just one libvirtd "server side" and it hears things the same way whether through API or virsh. | 15:01 |
| dansmith | efried: virsh is just an api client like nova, but yes that's what I'm saying | 15:02 |
| efried | jroll: Yes, the secret is in a file. Presumably the fact that said file is encrypted is the thing that makes this more "secure". | 15:02 |
| sean-k-mooney | efried: ya kind of virsh is just like the nova client | 15:02 |
| sean-k-mooney | all of the logic is implemetne in the libvirt deamon | 15:02 |
| efried | sean-k-mooney: we're 100% virtual here, there is no TPM on the host. | 15:02 |
| *** brinzhang has quit IRC | 15:02 | |
| sean-k-mooney | efried: well there could be | 15:02 |
| efried | for coding purposes, there isn't. | 15:02 |
| efried | jroll: but the fact that it's just sitting around there on the disk means that your disk thief could just... boot up the VM? | 15:03 |
| *** brinzhang has joined #openstack-nova | 15:03 | |
| efried | and would then have to have root access to the VM I guess | 15:03 |
| jroll | efried: the TPM state is in an encrypted file. there is a passphrase to decrypt the file, which we give to libvirt via `virsh secret-set-value` or the API. I'm wondering where libvirt stores that secret, is that also in a file? how does libvirt encrypt that file? (these are rhetorical questions, fwiw, digging now) | 15:03 |
| sean-k-mooney | efried: shoudl we maybe look at barbican to store the secreat rather then libvirt | 15:04 |
| sean-k-mooney | i havent really looked at how this worksin in libvirt yet | 15:04 |
| *** brinzhang has quit IRC | 15:04 | |
| sean-k-mooney | but ideally i think we dont want it sotred on the compute host right | 15:04 |
| efried | jroll: I'm still not 100% on any of this, but my understanding is that libvirt uses whatever's in the "secret" to encrypt/decrypt the contents of the vTPM. | 15:05 |
| *** brinzhang has joined #openstack-nova | 15:05 | |
| efried | but once you've done the secret-set-value... that's it. | 15:05 |
| jroll | efried: right, trying to figure out where "whatever's in the secret" is stored | 15:05 |
| jroll | where/how | 15:05 |
| * jroll upgrades to latest libvirt | 15:05 | |
| sean-k-mooney | it might just be in memory | 15:06 |
| sean-k-mooney | not everythin you do to a libvirt domain has to be stored to disk | 15:06 |
| *** jaosorior has quit IRC | 15:06 | |
| efried | jroll: | 15:06 |
| efried | stack@breakfast:~/nova$ sudo ls -l /etc/libvirt/secrets/ | 15:06 |
| efried | total 8 | 15:06 |
| efried | -rw------- 1 root root 4 Nov 9 12:36 d3e91e0e-20a5-4f60-ae85-9c8bfed6939f.base64 | 15:06 |
| efried | -rw------- 1 root root 211 Nov 9 12:36 d3e91e0e-20a5-4f60-ae85-9c8bfed6939f.xml | 15:06 |
| jroll | oh goody | 15:06 |
| jroll | thanks. | 15:06 |
| sean-k-mooney | efried: is that encrypted or plain text? | 15:07 |
| jroll | I'm gonna go find a table or three. definitely not to flip. | 15:07 |
| efried | The .base64 is just the b64-encoded value of the passphrase I set | 15:07 |
| efried | the XML is the XML chunk I started with. | 15:07 |
| sean-k-mooney | so plain texted effectivly | 15:07 |
| efried | yup | 15:07 |
| efried | So I've been thinking of a model where we secret-set-value just long enough to boot the VM, then blow away the secret. | 15:08 |
| sean-k-mooney | so you might as well not encrypt the tpm at that point | 15:08 |
| lyarwood | that isn't passed as plain text to QEMU btw | 15:08 |
| efried | ...assuming the tpm stays "unencrypted" while the domain is running. | 15:08 |
| lyarwood | it's the same as with LUKS, it's encypted using the master key for the domain | 15:08 |
| dansmith | efried: ahh, so... fake security then? :D | 15:08 |
| efried | but .... what dansmith said. | 15:09 |
| dansmith | heh | 15:09 |
| efried | that doesn't seem like real security. | 15:09 |
| Erdosip | I think it's under /etc/libvirt/security | 15:09 |
| efried | I mean, I guess your disk thief would have to steal the disk while the VM is booting. | 15:09 |
| *** ociuhandu has joined #openstack-nova | 15:10 | |
| *** Erdosip has quit IRC | 15:10 | |
| sean-k-mooney | im not sure the vm woudl have to be running | 15:10 |
| sean-k-mooney | there are cases were we leave the domain intact | 15:10 |
| sean-k-mooney | such as a showdown form inside the vm | 15:10 |
| sean-k-mooney | or suspend/pause | 15:11 |
| sean-k-mooney | maybe even stop | 15:11 |
| efried | sean-k-mooney: the secret in the domain xml is just the uuid of the secret. So presumably (hopefully) the decryption happens in-memory when the xml is loaded up to actually start the guest. | 15:11 |
| sean-k-mooney | start calls power on which calls reboot which recreates the domain | 15:11 |
| sean-k-mooney | sure but the issue is the secret is not encrypted on disk | 15:12 |
| efried | so regardless of whether the domain xml is there or needs to be recreated, it's when the guest starts up that we would need the secret to be present. | 15:12 |
| lyarwood | again, the secret isn't provided to QEMU as a plaintext string | 15:12 |
| sean-k-mooney | efried: im wondering is there a way to pass it at that point and never save it to disk | 15:13 |
| efried | lyarwood: but it's on disk in cleartext. (base-64 encoded, but effectively plaintext) | 15:13 |
| sean-k-mooney | lyarwood: sure but efried just said it in /etc/libvirt/secrets/ and is just base64 encoded | 15:13 |
| lyarwood | right and that's pretty locked down given the listing above | 15:13 |
| efried | lyarwood: I suspect I'm missing what you're getting at. | 15:13 |
| *** canori01 has joined #openstack-nova | 15:13 | |
| efried | lyarwood: you mean 600 owned by root locked down? | 15:14 |
| lyarwood | if someone had access to root it's already game over | 15:14 |
| sean-k-mooney | lyarwood: that not good enough in my view | 15:14 |
| efried | pretty sure in jroll's view too. | 15:14 |
| sean-k-mooney | the operator shoudl not be able to see the secreate | 15:14 |
| lyarwood | they could already get in and decrypt various things from the running QEMU process | 15:14 |
| canori01 | Hello, would I evacuate an instance that's running into 'AllocationUpdateFailed: Failed to update allocations for consumer' running on Stein | 15:14 |
| lyarwood | well that just isn't within libvirt's security model AFAIK | 15:15 |
| canori01 | I mean, how would I recover from that | 15:15 |
| *** ociuhandu has quit IRC | 15:15 | |
| efried | lyarwood: Actually I think we're drawing the line at "compromised root is acceptable risk; stolen disk must be secure". | 15:15 |
| jroll | more like we realize compromised root on a running hypervisor is game over anyway, but yeah | 15:16 |
| efried | IOW I think it's the nova admin creds that are the root of trust. | 15:16 |
| lyarwood | efried: kk, danpb had a blog about secret handling a while ago that might cover a way around that ./me checks | 15:16 |
| *** munimeha1 has joined #openstack-nova | 15:16 | |
| efried | thanks lyarwood | 15:16 |
| sean-k-mooney | efried: https://libvirt.org/formatsecret.html#SecretAttributes | 15:16 |
| jroll | lyarwood: thanks, I was just going to ask that | 15:16 |
| sean-k-mooney | so it look like you can mare a secret as ephemeral | 15:16 |
| efried | having displayed any understanding whatsoever, lyarwood, you're now permanently on call for support. | 15:16 |
| sean-k-mooney | so its only kept in memory and never on disk | 15:17 |
| sean-k-mooney | so if we store it in barbican and then do that that would be better | 15:17 |
| efried | oh, cool, I saw that but hadn't played with it yet, will have to try that out and see if it does the trick. | 15:17 |
| sean-k-mooney | well ephemeral=yes private=yes | 15:17 |
| efried | The secret I created while playing was private but not ephemeral. | 15:17 |
| jroll | can confirm ephemeral=yes leaves it off of disk | 15:18 |
| sean-k-mooney | so form a nova point of view can we use barbican to store it | 15:18 |
| efried | cool, so the nova user is the root of trust because has access to the barbican entry. | 15:18 |
| efried | yeah, what sean-k-mooney said. | 15:18 |
| jroll | and restarting libvirt nukes an ephemeral secret | 15:18 |
| jroll | \o/ | 15:18 |
| sean-k-mooney | jroll: that is proably ok/good | 15:19 |
| efried | now we just gotta hope you don't have to restart libvirt to make it pick up the secret :P | 15:19 |
| lyarwood | thanks sean-k-mooney | 15:19 |
| jroll | efried: fwiw, I didn't need to restart, but I'm on older libvirt playing with tls secrets, not tpm | 15:19 |
| *** awalende has quit IRC | 15:19 | |
| sean-k-mooney | jroll: i dont know if libvirt makes a difference | 15:20 |
| efried | jroll: didn't need to restart to do what? | 15:20 |
| *** awalende has joined #openstack-nova | 15:20 | |
| efried | in my case everything appeared to be moving along fine until I tried booting the guest, then the nova log got an exception complaining it couldn't find my secret. | 15:20 |
| efried | anyway, ephemeral would be entirely useless in that case, so it MUST work (famous last words) | 15:21 |
| jroll | efried: ah, I was just doing 'virsh secret-list' | 15:21 |
| jroll | heh | 15:21 |
| jroll | sean-k-mooney: if there's a bug in newer libvirt it might :) | 15:21 |
| sean-k-mooney | oh i ment between secrete usage/types | 15:22 |
| sean-k-mooney | i think it just has a secret that is use for many things | 15:22 |
| efried | jroll: btw, there was a question as to whether we needed to bother providing the non-encrypted version of vTPM support at all. | 15:22 |
| jroll | older libvirt won't let me create a vtpm secret type ¯\_(ツ)_/¯ | 15:22 |
| efried | right, that needs to be >=5.6.0, which afaict you have to compile yourself. | 15:22 |
| jroll | efried: I'd be pro-not-supporting that, I don't like footguns | 15:22 |
| sean-k-mooney | e.g. i dont think a secreate for tls or tpm is a different thing internally in libvirt | 15:22 |
| *** jaosorior has joined #openstack-nova | 15:22 | |
| efried | penick indicated the same. Just that the requirements are going to be way more stringent for it if so. | 15:23 |
| *** rchurch has joined #openstack-nova | 15:23 | |
| *** awalende_ has joined #openstack-nova | 15:23 | |
| sean-k-mooney | for what its worth rhel/fedora will be shiping it relitvly quickly | 15:23 |
| sean-k-mooney | im not sure when exactly | 15:23 |
| sean-k-mooney | but we have an advance virt stream we use for openstack | 15:24 |
| efried | I guess "with ussuri" is soon enough <shrug> | 15:24 |
| sean-k-mooney | that ships a newer version then base rhel | 15:24 |
| efried | but also the barbican requirement | 15:24 |
| *** ociuhandu has joined #openstack-nova | 15:24 | |
| *** awalende has quit IRC | 15:24 | |
| efried | anyway, glad you agree jroll, it makes the code & test surface easier. | 15:25 |
| efried | lyarwood: since you mentioned the qemu side -- do you have any idea what's the minimum qemu version to support encrypted vTPM? I couldn't find that in the docs anywhere. | 15:25 |
| efried | canori01: it would help to know why the allocation update is failing (out of resources? provider conflict? other?), and what action is being performed when that happens. | 15:26 |
| *** awalende_ has quit IRC | 15:27 | |
| canori01 | efried: I am replacing old hardware. So I powered off the old hypervisor and then did a nova-evacuate. All the guests evacuated except for two which gave me the provider conflict | 15:28 |
| lyarwood | efried: I don't sorry | 15:29 |
| *** ociuhandu has quit IRC | 15:29 | |
| lyarwood | efried: 2.12 is listed in the Stein spec FWIW | 15:29 |
| efried | lyarwood: yeah, that was for non-encrypted, and idk if there's a higher req for encrypted. | 15:30 |
| efried | canori01: If it was a provider conflict, you should just be able to "try again". | 15:30 |
| lyarwood | efried: no idea sorry, kashyap is out today otherwise I'd ask him if he could work that out for us. | 15:30 |
| efried | ah, good plan. I'll stalk him, thanks. | 15:30 |
| sean-k-mooney | efried: tpm emulation was added in 2.11 but im not sure if that included encryption | 15:30 |
| efried | it did not | 15:31 |
| efried | But I don't even know whether qemu required any update for that aspect. It might not have. | 15:31 |
| sean-k-mooney | also this is an intersting site https://repology.org/project/libvirt/versions | 15:31 |
| sean-k-mooney | look like apline has the depencyies you need but very little else at teh moment | 15:33 |
| *** TxGirlGeek has joined #openstack-nova | 15:34 | |
| sean-k-mooney | Fedora Rawhide would "work" too although i suspect nova wont be happy with it in general | 15:34 |
| efried | sean-k-mooney: oh, that reminds me, I ran into another weird quirk with the home-built libvirt... | 15:35 |
| efried | sean-k-mooney: When I restarted the libvirt service, it blew up because it was looking in /usr/lib/x86_64-linux-gnu/ which contained the old libvirt 4.xx | 15:35 |
| efried | I had to go into the service and set LD_LIBRARY_PATH=/usr/lib explicitly to make it work. | 15:36 |
| efried | which seems... weird. | 15:36 |
| efried | had to do the same with n-cpu. | 15:36 |
| sean-k-mooney | ya that does seam strange | 15:36 |
| sean-k-mooney | well clearly i have more work to do | 15:37 |
| dansmith | efried: why is that surprising? | 15:37 |
| dansmith | efried: if you have an old version installed in the standard system library path, and start a new daemon binary that requires the new libs, | 15:38 |
| dansmith | you have to tell it where they are | 15:38 |
| dansmith | (if I'm understanding what you're saying) | 15:38 |
| efried | dansmith: Is /usr/lib/x86_64-linux-gnu/ the "standard system library path"? | 15:38 |
| efried | That was the surprising part to me. I would have thought /usr/lib was. | 15:38 |
| dansmith | efried: for multi-arch stuff on fedora-like distros it is I think yeah | 15:39 |
| sean-k-mooney | dansmith: well the devstack plugin uninstalles the packages and libvirt was compiles with --system | 15:39 |
| efried | sean-k-mooney: Doesn't your plugin just use `make install`? | 15:39 |
| dansmith | sean-k-mooney: clearly not if the old version was still there | 15:39 |
| efried | I would have thought that should have covered symlinking or otherwise replacing versions in any "standard" paths. | 15:39 |
| dansmith | efried: not generally | 15:40 |
| sean-k-mooney | efried: yes but the autogen sript was passed --system so when it invoked configure it should have set it up to instll in the default system lib path | 15:40 |
| dansmith | efried: make install usually puts them in /usr/local/lib unless you tell it otherwise | 15:40 |
| dansmith | or, --system would yeah | 15:40 |
| sean-k-mooney | dansmith: well i dont try an nuke the old verions | 15:40 |
| dansmith | efried: cat /etc/ld.so.conf or /etc/ld.so.conf.d/* | 15:41 |
| dansmith | efried: will show you where the loader looks for libs | 15:41 |
| sean-k-mooney | dansmith: this shoudl do the right thing yes? https://opendev.org/x/devstack-plugin-libvirt-qemu/src/branch/master/devstack/libs/libvirt#L94-L96 | 15:41 |
| dansmith | efried: even on my ubuntu system, /usr/lib/$arch is in there | 15:41 |
| dansmith | sean-k-mooney: should do what? land it in /usr/lib? probably, but he's saying that happened AFAICT | 15:42 |
| efried | ubuntu is what I'm using. And yeah, same. In fact, /usr/lib isn't there at all. | 15:42 |
| sean-k-mooney | /usr/lib is not there at all? | 15:43 |
| efried | stack@breakfast:~/nova$ cat /etc/ld.so.conf.d/* | 15:43 |
| efried | /usr/lib/x86_64-linux-gnu/libfakeroot | 15:43 |
| efried | # libc default configuration | 15:43 |
| efried | /usr/local/lib | 15:43 |
| efried | # Multiarch support | 15:43 |
| efried | /usr/local/lib/x86_64-linux-gnu | 15:43 |
| efried | /lib/x86_64-linux-gnu | 15:43 |
| efried | /usr/lib/x86_64-linux-gnu | 15:43 |
| efried | canori01: I want to say we were working on this, or something very similar, recently, around ignoring *provider* conflicts and just retrying, but it might have been for a different lifecycle operation, I can't remember. Have you looked for an existing launchpad bug? | 15:45 |
| sean-k-mooney | efried: this is what i have http://paste.openstack.org/show/785944/ | 15:45 |
| canori01 | efried: I was lookin at https://bugs.launchpad.net/nova/+bug/1798688 but I'm not 100% sure if it's related | 15:47 |
| openstack | Launchpad bug 1798688 in OpenStack Compute (nova) "AllocationUpdateFailed_Remote: Failed to update allocations for consumer. Error: another process changed the consumer after the report client read the consumer state during the claim" [High,Fix released] - Assigned to Matt Riedemann (mriedem) | 15:47 |
| efried | sean-k-mooney: yeah, that's what I have in /usr/lib as well, point is that the libvirt-bin and n-cpu services were looking in the x86... dir. | 15:47 |
| sean-k-mooney | oh ok | 15:47 |
| *** jaosorior has quit IRC | 15:47 | |
| efried | sean-k-mooney: iow I assume we should either be installing there, or be blowing those away with symlinks. | 15:47 |
| sean-k-mooney | it might be best to just drop the --system | 15:48 |
| sean-k-mooney | it clearly not doing it write | 15:49 |
| sean-k-mooney | then it will install in /usr/local/lib which is in the lib path | 15:49 |
| efried | I want to say other things blew up when I tried that | 15:49 |
| *** tbachman has joined #openstack-nova | 15:50 | |
| *** awalende has joined #openstack-nova | 15:50 | |
| efried | anyway, hopefully this is all moot irl. Unless I want to try to run a CI job around this thing. And I'm not sure if there's going to be value in that. | 15:50 |
| sean-k-mooney | well ill try and spend some more time on this and see if i can make it a bit more robost. | 15:50 |
| *** links has quit IRC | 15:52 | |
| efried | sean-k-mooney: I appreciate that. Though it's probably not worth trying to make it perfect. I think the sweet spot will be having it work in devstack CI (again, if I end up wanting to do something there), which is of course a pretty narrow subset of usages. | 15:52 |
| *** mkrai has quit IRC | 15:52 | |
| canori01 | efried: Is there a way, I coudl just force the instance to take a new allocation and spawn somewhere? I have plenty of spare capacity | 15:53 |
| efried | ...excluding, among other things, "works in my dirty devstack, containing who-knows-what packages, that I've restacked a zillion times" | 15:53 |
| sean-k-mooney | efried: ya i would like to get a ci job in that plugin repo at a minium but i have need to use this plugin alot in the past when enableing things for openstack so i want it to be atleast usable for that | 15:53 |
| *** tbachman_ has joined #openstack-nova | 15:53 | |
| efried | canori01: At this point you should still be able to evacuate it, no? | 15:53 |
| efried | oh, you mean YOU have use cases too?? Well, then, by all means :P | 15:54 |
| *** tbachman has quit IRC | 15:54 | |
| *** tbachman_ is now known as tbachman | 15:54 | |
| sean-k-mooney | well not currently | 15:54 |
| *** awalende has quit IRC | 15:54 | |
| sean-k-mooney | i orginally wrote the first version of that plugin back in 2014 | 15:54 |
| sean-k-mooney | i have used it for at least 4 feature at this point | 15:55 |
| canori01 | efried: I've tried resetting the state of the instance and evacuating again, but it consistently gives me this http://paste.openstack.org/show/785945/ | 15:55 |
| sean-k-mooney | so i like keping it working for when i need it | 15:55 |
| efried | canori01: Oh, hm, that's a reproducible failure? That is not expected. I think we need a bug. | 15:56 |
| canori01 | ok, thanks | 15:56 |
| sean-k-mooney | canori01: you dont have two copies of the nova compute agent running on the compute node by any chacce? | 15:57 |
| *** jaosorior has joined #openstack-nova | 15:57 | |
| sean-k-mooney | oh never mind | 15:58 |
| sean-k-mooney | this is in the scduler | 15:58 |
| *** artom has joined #openstack-nova | 16:02 | |
| *** jaosorior has quit IRC | 16:04 | |
| *** jaosorior has joined #openstack-nova | 16:05 | |
| dansmith | cdent: don't we have some guideline around not making fundamental api behaviors change based on policy or config? (well, I know config) | 16:07 |
| dansmith | cdent: meaning something like a user is granted some policy element, which enables a deep feature that they don't even know about nor can detect, which makes all calls to a specific API suddenly return 202 and be async, where normally they are 200 and sync | 16:08 |
| cdent | dansmith: there's a "what should cause a microversion bump" guideline somewhere that I can dig up. Is that what you're thinkin gof? | 16:08 |
| dansmith | cdent: and two users with differing policies for this element would see the different sync/async behavior | 16:09 |
| cdent | I don't know of anythign with that level of detail, but that sounds like a definite no no | 16:09 |
| dansmith | cdent: no because this would differ within one microversion for two users with different endowments | 16:09 |
| dansmith | cdent: seems like it to me as well | 16:09 |
| *** ociuhandu has joined #openstack-nova | 16:10 | |
| dansmith | cdent: might be more than you want to digest at the moment, but this is the reference: https://review.opendev.org/#/c/635684/53/nova/compute/api.py@3852 | 16:10 |
| cdent | does this apply to something you're looking at (I just got back from being away, so not in context) | 16:10 |
| cdent | jinx | 16:10 |
| * cdent looks | 16:10 | |
| dansmith | cdent: the shortcut is: we do a cast if allow_cross_cell_resize is true, and (in a later patch) that will be true based on your policy | 16:10 |
| dansmith | and that seems bad | 16:11 |
| cdent | so cloud A wouild be cast and B would be call, dependent on policy settings? would it be discoverable in any way? | 16:11 |
| cdent | hmmm. however, is a user ever supposed to that cells exist? | 16:12 |
| *** ociuhandu has quit IRC | 16:13 | |
| cdent | because this isn't preventing or allowing resize, correct? just resize of a particular type? | 16:13 |
| *** ociuhandu has joined #openstack-nova | 16:14 | |
| dansmith | cdent: no, it'd be *User* A and user B that see differences, in the same cloud | 16:14 |
| dansmith | cdent: and no, a user would not know they have this ability, nor know why they see different behavior from their peer | 16:14 |
| cdent | hmmm. is there a short explanation for the "why" on controlling by policy? | 16:15 |
| dansmith | cdent: it's literally gating whether or not the scheduler will return all hosts, or only hosts within the current cell, not something the user knows about | 16:15 |
| cdent | (I'm trying to read the code, but there's a lot here) | 16:15 |
| dansmith | cdent: the why is so you can control it per-user, where config would be per-cloud | 16:15 |
| cdent | yes, but _why_ would you control is per user? | 16:15 |
| cdent | s/is/it/ | 16:16 |
| dansmith | cdent: because it | 16:16 |
| dansmith | is more expensive, and/or you may have other reasons, | 16:16 |
| *** nanzha has quit IRC | 16:16 | |
| dansmith | it's a little weird that it's a policy knob, but it is that way just so that it can be controlled per user for whatever reason | 16:17 |
| cdent | I can see that people would want that, even though they shouldn't ;) | 16:17 |
| cdent | everybody wants the knobs | 16:17 |
| *** tbachman has quit IRC | 16:18 | |
| cdent | okay, I think I have enough info to think about this with some level of legitimacy | 16:18 |
| dansmith | well, that's a different argument I guess, but ... yeah. | 16:18 |
| dansmith | my other argument would be that this is potentially more failure-prone, | 16:18 |
| dansmith | so you may not want to give it to "simple people" or something | 16:18 |
| dansmith | or just let admins do it so they can move things around the cloud | 16:18 |
| dansmith | like the policy knob that lets you direct migrations to a specific host vs. just use the scheduler | 16:18 |
| * cdent nods | 16:19 | |
| dansmith | but regardless, that discussion is somewhat separate from the api behavior thing | 16:19 |
| * cdent nods | 16:19 | |
| cdent | yeah, just trying to add a bit of definition so I'm not filling in the blanks poorly | 16:19 |
| dansmith | yup | 16:19 |
| dansmith | explaining it to you here makes me feel more like my +0 should be a -1 | 16:20 |
| *** bnemec has quit IRC | 16:20 | |
| cdent | what it is doing seems like the right thing, but I agree that it should have a microversion for the reason you state: change in behavior when same code has a different user using it | 16:21 |
| cdent | i'll say as much on the review | 16:21 |
| sean-k-mooney | dansmith: i dont know if you could take extention list approch like neutron | 16:21 |
| *** ociuhandu has quit IRC | 16:21 | |
| sean-k-mooney | dansmith: and have that reterun different values based on policy | 16:21 |
| cdent | what neutron does is terrifying from an api standpoint | 16:21 |
| sean-k-mooney | well at least you can detect it | 16:21 |
| dansmith | cdent: the microversion would have to say "after microversion 2.X, the method could be sync or async so watch your return code" | 16:21 |
| sean-k-mooney | but yes | 16:21 |
| sean-k-mooney | it still feels weird | 16:21 |
| dansmith | cdent: which I thought is one of the reasons we *don't* add a microversion | 16:22 |
| *** maciejjozefczyk has quit IRC | 16:22 | |
| dansmith | sean-k-mooney: we specifically do not version our api via extensions anymore, so I'm not sure why you'd suggest that | 16:22 |
| dansmith | that's the whole point of microversions :) | 16:22 |
| sean-k-mooney | dansmith: neutron uses bot | 16:22 |
| sean-k-mooney | *both | 16:22 |
| sean-k-mooney | on your return code point | 16:22 |
| sean-k-mooney | it woudl be 200 for syc and 202 or 203 for async? | 16:23 |
| dansmith | sean-k-mooney: I said that above | 16:23 |
| sean-k-mooney | would they send the same payload in both cases? | 16:23 |
| dansmith | sean-k-mooney: you might want to read the backscroll and the code/comment I linked above | 16:24 |
| sean-k-mooney | i kind of feel like haveing a flag in the post body would be cleaner | 16:24 |
| dansmith | it'd be good context for the conversation | 16:24 |
| dansmith | sean-k-mooney: because I've said several times that this is not something user knows about or can/would opt into | 16:24 |
| sean-k-mooney | am i will after my next 1:1 in 5 mins | 16:24 |
| sean-k-mooney | dansmith: ah ok | 16:24 |
| *** tbachman has joined #openstack-nova | 16:25 | |
| sean-k-mooney | oh this is for cross cell resize | 16:26 |
| sean-k-mooney | oh its related to the downcall to the compute. hum ya thats tricky | 16:27 |
| sean-k-mooney | well maybe "down call" is not right but the thing we were talking to matt about last week | 16:28 |
| *** mkrai has joined #openstack-nova | 16:29 | |
| *** tbachman has quit IRC | 16:30 | |
| *** ociuhandu has joined #openstack-nova | 16:32 | |
| *** ivve has quit IRC | 16:32 | |
| *** ricolin has quit IRC | 16:34 | |
| *** ociuhandu has quit IRC | 16:35 | |
| *** ociuhandu has joined #openstack-nova | 16:36 | |
| *** mkrai has quit IRC | 16:36 | |
| *** maciejjozefczyk has joined #openstack-nova | 16:43 | |
| *** pcaruana has joined #openstack-nova | 16:45 | |
| *** bnemec has joined #openstack-nova | 16:47 | |
| *** ociuhandu has quit IRC | 16:52 | |
| *** ociuhandu has joined #openstack-nova | 16:52 | |
| *** slaweq has joined #openstack-nova | 16:58 | |
| *** ociuhandu has quit IRC | 17:00 | |
| *** slaweq has quit IRC | 17:02 | |
| *** cdent has quit IRC | 17:04 | |
| *** ociuhandu has joined #openstack-nova | 17:04 | |
| *** mlavalle has joined #openstack-nova | 17:05 | |
| *** tetsuro has quit IRC | 17:23 | |
| *** maciejjozefczyk has quit IRC | 17:25 | |
| *** rosmaita has joined #openstack-nova | 17:27 | |
| *** rosmaita has left #openstack-nova | 17:28 | |
| *** ociuhandu has quit IRC | 17:32 | |
| *** ociuhandu has joined #openstack-nova | 17:39 | |
| *** ociuhandu has quit IRC | 17:44 | |
| *** priteau has quit IRC | 17:45 | |
| *** lpetrut has quit IRC | 17:55 | |
| *** jaosorior has quit IRC | 18:06 | |
| *** ociuhandu has joined #openstack-nova | 18:18 | |
| *** dtantsur is now known as dtantsur|afk | 18:20 | |
| *** maciejjozefczyk has joined #openstack-nova | 18:26 | |
| *** slaweq has joined #openstack-nova | 18:28 | |
| *** ociuhandu has quit IRC | 18:31 | |
| *** nweinber_ has joined #openstack-nova | 18:33 | |
| *** nweinber has quit IRC | 18:36 | |
| *** jmlowe has joined #openstack-nova | 18:38 | |
| *** zhubx has quit IRC | 18:54 | |
| *** zhubx has joined #openstack-nova | 18:54 | |
| *** ivve has joined #openstack-nova | 18:55 | |
| *** pcaruana has quit IRC | 18:57 | |
| *** pcaruana has joined #openstack-nova | 18:58 | |
| *** maciejjozefczyk has quit IRC | 19:02 | |
| *** slaweq has quit IRC | 19:06 | |
| *** nweinber__ has joined #openstack-nova | 19:08 | |
| *** nweinber_ has quit IRC | 19:10 | |
| *** slaweq has joined #openstack-nova | 19:11 | |
| efried | jroll: progress: using ephemeral="yes" nothing shows up on the disk. This means we'll have to define & set the secret on the fly once per guest per service-process-instance, which seems reasonable. | 19:12 |
| efried | jroll: Next sticky design point: bootstrapping the secret UUID. | 19:13 |
| efried | We don't want it to be in the flavor, because then you need a new flavor per secret. | 19:14 |
| efried | I think we can do it this way: Create it with a random secret in the key manager. I guess neither the VM user nor the operator needs to know the secret value; they just have to be able to access it from the keymgr. | 19:15 |
| *** slaweq has quit IRC | 19:16 | |
| efried | So we just have to figure out a way to associate the secret UUID with the instance. | 19:16 |
| efried | We could use the instance UUID, but that seems fragile. | 19:16 |
| *** ircuser-1 has joined #openstack-nova | 19:17 | |
| efried | otherwise I suppose we need to use a new field in the instance. Which makes me a little bit sad, but it's not the end of the world. | 19:17 |
| efried | dansmith: do you have any other suggestions? | 19:17 |
| efried | TL;DR: I'm going to generate a UUID when I create the instance, and need to be able to associate that UUID with the instance for the life of the instance. | 19:18 |
| dansmith | efried: system_metadata ? | 19:18 |
| efried | is that a place I can stuff arbitrary key=value without a schema change? | 19:18 |
| dansmith | efried: yup | 19:18 |
| efried | sounds perfect :) | 19:18 |
| efried | thanks. | 19:19 |
| efried | okay, I think I have enough answers that I can take a break from PoCing and fill in a bunch of the open questions on the spec. (<== TxGirlGeek FYI) | 19:19 |
| *** awalende has joined #openstack-nova | 19:21 | |
| sean-k-mooney | efried: you could just use 1 the instance uuid or 1 a uuid5 based on the instance uuid as a seed | 19:21 |
| efried | anything derived from the instance UUID seems brittle, because other things might assume they can do the same. | 19:22 |
| efried | A separate random uuid is safest | 19:23 |
| sean-k-mooney | uuid5 takes a seed uuid + a string | 19:23 |
| sean-k-mooney | so the seed uuid would be the instance uuid | 19:23 |
| efried | oh, that could work. | 19:23 |
| sean-k-mooney | that defines a namespace | 19:23 |
| sean-k-mooney | and then you can generate uuids form that | 19:23 |
| efried | yeah, the string could be "emulated tpm secret" or something unique. | 19:23 |
| sean-k-mooney | ya | 19:23 |
| sean-k-mooney | https://docs.python.org/2/library/uuid.html#uuid.uuid5 | 19:24 |
| *** ociuhandu has joined #openstack-nova | 19:25 | |
| *** awalende has quit IRC | 19:25 | |
| dansmith | efried: definitely prefer generation + storing in sysmeta | 19:26 |
| efried | okay | 19:26 |
| *** mriedem has joined #openstack-nova | 19:27 | |
| sean-k-mooney | yeah you can do both | 19:27 |
| efried | wow cool https://docs.openstack.org/api-ref/key-manager/ | 19:31 |
| dansmith | efried: it's for managing secrets..including the definition of its api | 19:32 |
| efried | hahahaha | 19:32 |
| mriedem | heh, thought that was fixed awhile ago | 19:32 |
| mriedem | this page last updated: 2018-10-18 16:43:23 | 19:32 |
| mriedem | https://docs.openstack.org/barbican/latest/api/ | 19:33 |
| sean-k-mooney | dont you use castalain or something rather then barbican directly form within nova? | 19:33 |
| mriedem | ^ is where https://docs.openstack.org/api-quick-start/ takes you | 19:33 |
| mriedem | castallan is the interface, barbican is a backend implementation | 19:34 |
| sean-k-mooney | ya that works https://docs.openstack.org/barbican/latest/api/reference/secrets.html | 19:34 |
| NostawRm | I'm exploring the possibility of adding another image backend for Glance and Nova before working out a blueprint, spec, etc. Adding a driver to glance seemed straight forward, but the way nova does image backends doesn't seem as if its intended to add more backends since its all in one file. Is this the case and using boot from volume is intended or am I making bad assumptions? | 19:34 |
| mriedem | NostawRm: if it's adding a backend that is already supported by cinder than it's likely not going to get much traction | 19:35 |
| *** henriqueof has joined #openstack-nova | 19:35 | |
| mriedem | EMC ScaleIO was the last to try and fail at that | 19:35 |
| sean-k-mooney | NostawRm: it also depends on your usecase. | 19:35 |
| *** ociuhandu has quit IRC | 19:36 | |
| sean-k-mooney | e.g. what advantage would it bring to have a new iamge backend over cinder/bfv | 19:36 |
| dansmith | NostawRm: are you the one that asked on the ML earlier today? | 19:37 |
| *** tbachman has joined #openstack-nova | 19:38 | |
| NostawRm | that's a shame, but it makes. When using Ceph/RBD, we used it with ephemeral storage, but StorPool only has volumes, so having to boot from volume. Keep doing work arounds and was looking at attacking it from this angle instead | 19:38 |
| *** ociuhandu has joined #openstack-nova | 19:38 | |
| NostawRm | Right now, its because you can't rescue an instance booted from volume | 19:38 |
| NostawRm | dansmith, nah, wasn't me | 19:38 |
| mriedem | so fix rescue to work with volume-backed servers instead... | 19:38 |
| mriedem | like rebuilding volume-backed servers | 19:38 |
| dansmith | NostawRm: ack, well, you might go look at my response then :) | 19:38 |
| mriedem | etc | 19:38 |
| NostawRm | But if that's the case, I suppose it is just what it is | 19:39 |
| dansmith | mriedem: heh, exactly what I said in my response | 19:39 |
| mriedem | we are sympatico | 19:39 |
| *** ociuhandu has quit IRC | 19:39 | |
| sean-k-mooney | were is the spec for detaching a root volume | 19:39 |
| *** ociuhandu has joined #openstack-nova | 19:39 | |
| dansmith | NostawRm: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010678.html | 19:39 |
| sean-k-mooney | resucing a boot form volume instance is a similar usecase | 19:40 |
| dansmith | sean-k-mooney: not really, rescue wouldn't require a detach | 19:40 |
| dansmith | just a boot order change | 19:40 |
| sean-k-mooney | ya thats true | 19:40 |
| NostawRm | That's hilarious | 19:40 |
| NostawRm | that its also StorPool | 19:40 |
| sean-k-mooney | but if you can detact it you can detach and add it to another instance | 19:41 |
| dansmith | sean-k-mooney: that's not what rescue is | 19:41 |
| sean-k-mooney | i know rescue is booting and existing instance in place so you can fix it | 19:41 |
| TxGirlGeek | efried got it :) | 19:41 |
| sean-k-mooney | but you can detach attach fix detach retatach to original instnace | 19:41 |
| sean-k-mooney | *you could | 19:41 |
| sean-k-mooney | any fixing resucre for bfv is userful too | 19:42 |
| *** tbachman has quit IRC | 19:43 | |
| *** tbachman has joined #openstack-nova | 19:45 | |
| sean-k-mooney | by the way anyone know if the topic of a generic cinder image backend come up again at the PTG. bfv is not quite the same thing but if we were to add another imgage backend that would be my goto approach | 19:47 |
| sean-k-mooney | fixing bfv to do what you want feels like a lot less work however | 19:47 |
| mriedem | dansmith: didn't lee's old spec cover volume-backed rescue? https://review.opendev.org/#/c/651151/ | 19:49 |
| dansmith | mriedem: I dunno, it'd be an important part of it though | 19:50 |
| jroll | efried: that all sounds good, although I don't even think the VM user or operator needs to be able to access the key in the secret manager. but yeah +1 to all the rest | 19:51 |
| NostawRm | So, I'm fairly certain that the Mailing list thread was created because of me bothering StorPool. I was looking at possible solutions to propose to them, seems like they were already on it too :) | 19:52 |
| sean-k-mooney | jroll: i think the operator definetly shound not be able to | 19:52 |
| sean-k-mooney | the user maybe but proably not | 19:53 |
| jroll | yep | 19:53 |
| sean-k-mooney | the user might have to be able to acess it if we use there auth token to manage it | 19:53 |
| efried | jroll: I don't know what ability is conferred upon the "operator", by virtue of being the same user as is booting the instance | 19:53 |
| efried | n-cpu is going to access the key manager service and be able to create & retrieve the passphrase based on being credentialed as the operator, no? | 19:54 |
| sean-k-mooney | no | 19:54 |
| efried | so couldn't the operator, using the same creds, simply go retrieve same? | 19:54 |
| efried | oh? | 19:54 |
| dansmith | generally the user's creds would be passed to barbican right? | 19:54 |
| sean-k-mooney | efried: nova compute should be useing the autoken form the boot request | 19:54 |
| efried | is that not what I said? | 19:55 |
| dansmith | which means you can't start the instance in isolation, but when the user does a start, you can auth and get the deets | 19:55 |
| sean-k-mooney | efried: operaor generaly refers to the cloud admin | 19:55 |
| dansmith | efried: you said "credentialed as the operator" | 19:55 |
| efried | I thought "operator" in this context was "guy creating instance". | 19:55 |
| efried | as opposed to "admin". | 19:55 |
| dansmith | hah | 19:55 |
| dansmith | no | 19:55 |
| efried | did I want "user"? | 19:56 |
| dansmith | yes | 19:56 |
| efried | sigh, okay. | 19:56 |
| efried | So | 19:56 |
| sean-k-mooney | user or tenant | 19:56 |
| efried | "user" is the one with access to the secret | 19:56 |
| mriedem | NostawRm: fwiw i replied to the thread and linked in a bunch of pre-existing design work on several volume-backed server feature gaps in the api if you're so inclined to push your vendor to fund a developer to work on those again | 19:56 |
| sean-k-mooney | yes ideally | 19:56 |
| efried | doesn't this have implications for certain lifecycle operations? | 19:56 |
| sean-k-mooney | yes | 19:56 |
| efried | ...that are done by admins? | 19:56 |
| dansmith | efried: see what I said above about restart | 19:56 |
| sean-k-mooney | efried: this is already the case for encyped instances | 19:57 |
| dansmith | right | 19:57 |
| efried | what operations are done by admins? | 19:57 |
| dansmith | live migrate, restarting a compute host | 19:57 |
| sean-k-mooney | live migrate might be ok | 19:58 |
| dansmith | (and cold migrate of course | 19:58 |
| efried | *can* live migrate be done by the user? | 19:58 |
| sean-k-mooney | assuming libvirt woudl transfer the key | 19:58 |
| dansmith | efried: if granted, but not usually | 19:58 |
| dansmith | sean-k-mooney: yeah, fair | 19:58 |
| sean-k-mooney | efried: not unless you change the policy.json | 19:58 |
| dansmith | but cold migrate wouldn't work | 19:58 |
| efried | "libvirt would transfer the key" -- meaning the secret I loaded into the libvirtd on host1 would transfer over to the libvirtd on host2? | 19:59 |
| sean-k-mooney | well cold migrate would result in the vm being ofline? | 19:59 |
| sean-k-mooney | we would not be able to start it | 19:59 |
| sean-k-mooney | a user could do a resize | 19:59 |
| sean-k-mooney | that should work | 19:59 |
| dansmith | sean-k-mooney: usually it is done on a running instance | 19:59 |
| dansmith | yeah, user-initiated resize should be doable | 19:59 |
| dansmith | cold migrate would need to either fail early, or else it would result in an offlined vm that the admin can't restart | 20:00 |
| dansmith | so not useful for maintenance without significant disruption | 20:00 |
| NostawRm | efried: I really appreciate that. I also don't mind putting in the work to contribute, I just like to check here to make sure the contributions wouldn't be immediately shot down if I can't find it being discussed anywhere else. Thanks! | 20:00 |
| dansmith | no way around that, but ideally it would refuse to move it so they don't accidentally break instances | 20:00 |
| efried | I think the secret stays loaded up in libvirtd as long as that process is running, so admin could do pretty much anything that keeps the instance on the same host; as long as they don't restart libvirtd the instance will come back up. | 20:00 |
| efried | I guess I'll have to test the secret-goes-with-live-migrate theory. | 20:01 |
| sean-k-mooney | efried: ya maybe that would be correct | 20:01 |
| dansmith | efried: hopefully libvirt won't show you te secret right? | 20:01 |
| sean-k-mooney | however hard reboot deletes the domain | 20:01 |
| sean-k-mooney | and recreates it | 20:02 |
| efried | dansmith: it won't, as long as private='yes'. | 20:02 |
| sean-k-mooney | so that might clear the secret | 20:02 |
| efried | sean-k-mooney: I'm saying specifically that doesn't matter. | 20:02 |
| dansmith | efried: yeah, so you can't use that to do an unauthorized cold migration, if that's what you're implying | 20:02 |
| efried | dansmith: understood for cold, but still wondering about live. | 20:02 |
| dansmith | efried: live should work I would think | 20:03 |
| dansmith | efried: if not, graceful fail, otherwise you'd migrate into an unrunnable state | 20:03 |
| efried | setting up the secret and creating the instance are totally different things, from what I understand. I wouldn't expect libvirt to transfer the actual secret to the destination. However, if it's transferring the "state" of the instance, which includes an already-unencrypted vtpm, that might work. But then the VM on the target is kind of in a different condition than it was on the source, in that the secret is not loaded. | 20:05 |
| dansmith | efried: oh, well in that case maybe worth more thinking, | 20:05 |
| efried | so like the reboot-recreate-xml scenario would *not* work in that case. | 20:05 |
| sean-k-mooney | i can ask danpb tomorow when he is online if he knows what is expected to happen | 20:06 |
| dansmith | efried: but since the instances can't be restarted without the user's creds anyway, even if it migrates and can't reboot | 20:06 |
| dansmith | that would be the same | 20:06 |
| sean-k-mooney | but testing might give you an answer quiciker | 20:06 |
| sean-k-mooney | what is the state of migrations for instnace with encyption today | 20:07 |
| efried | dansmith: well, what I was saying is, once the secret is loaded up in libvirtd, anybody would be able to restart, without user's creds. | 20:07 |
| sean-k-mooney | the vtpm would likely have the same constraints | 20:07 |
| dansmith | efried: ...and break the ability to restart the instance in place you mean right? | 20:08 |
| efried | unless we do a step to undefine the secret when rebooting or something. | 20:08 |
| sean-k-mooney | efried: well we proably shoudl do that as part of stop | 20:08 |
| *** avolkov has quit IRC | 20:08 | |
| sean-k-mooney | it also depens on if its a soft or hard reboot | 20:09 |
| sean-k-mooney | soft reboot shoudl be fine | 20:09 |
| dansmith | a soft reboot would persist I'd think, | 20:09 |
| dansmith | but yeah, hard reboot and stop should nuke the secret if you want it safe at rest | 20:09 |
| efried | The dom xml contains just a pointer to the secret. The secret has to be loaded up in libvirtd's memory in order to start the guest. Once it's loaded up, it stays loaded up until/unless you undefine it (or restart the service). Clearly I would undefine the secret on instance deletion. But anything where the instance needs to start, no matter who starts it, should work as long as the secret is still loaded. | 20:10 |
| sean-k-mooney | efried: well you should undeife the secrete when you stop the instance without deleting it too | 20:10 |
| sean-k-mooney | to not keep it in memory | 20:11 |
| efried | why? | 20:11 |
| efried | I mean sure... but why? | 20:11 |
| *** slaweq has joined #openstack-nova | 20:11 | |
| sean-k-mooney | to prevent people reading it form libvirt memory. we could make it a config option or chose not too | 20:11 |
| sean-k-mooney | but if you really care about security you want the key decyped for as short a time as possible | 20:12 |
| efried | that sounds in the same category as "delete the file from disk right quick" | 20:12 |
| efried | i.e. "fake security" as dansmith calls it. | 20:12 |
| sean-k-mooney | not really | 20:12 |
| sean-k-mooney | you are say if i have not deleted the isntance but stop it keep the key in memory untill either the vm is deleted or you restart libvirt | 20:13 |
| dansmith | efried: I think the user would expect that it's safe at rest when stopped | 20:13 |
| *** tbachman has quit IRC | 20:13 | |
| *** slaweq has quit IRC | 20:16 | |
| dansmith | efried: this is definitely all a bit of theater of course, because none of the places where the key lives unencrypted for any amount of time are "secure", | 20:17 |
| efried | dansmith: We have apparently accepted that "compromised root" menas all bets are off anyway. | 20:18 |
| dansmith | but I think it's different to say "okay you stopped it, so I forgot all the secrets until you give them to me again" vs "I wasn't very careful with your secret while I had it" | 20:18 |
| sean-k-mooney | i dont think that is really true | 20:18 |
| efried | the important thing being, if someone walks off with the disk, they can't get at the secret, because they don't have the user's creds. | 20:19 |
| sean-k-mooney | in some cases maybe but we should not make it easy even in that case | 20:19 |
| dansmith | efried: compromised root of the host you mean? that's definitely game-over I think with all the other assumptions that prop up an openstack cluster | 20:19 |
| dansmith | efried: yep and that's what I call 'safe at rest | 20:19 |
| efried | conceivably I could delete the secret as soon as the guest starts. | 20:19 |
| sean-k-mooney | efried: i have been wondering about that | 20:20 |
| efried | I assume the tpm stays unlocked while the guest is up. Testing... | 20:20 |
| sean-k-mooney | but that could break one usecase | 20:20 |
| sean-k-mooney | maybe | 20:20 |
| dansmith | efried: I don't think that buys you anything real, other than avoiding forgetting/failing to clean it up, which is good of course | 20:20 |
| sean-k-mooney | what hapens when you ssh in and do a reboot in the guest | 20:20 |
| efried | dansmith: yeah, that's all I meant, just simplifies some of the assumptions and operations. | 20:20 |
| dansmith | sean-k-mooney: IIRC modern kvm reboot is within the qemu process, not respawned by libvirt, so I would expect that works | 20:21 |
| efried | Makes it harder for not-the-user to do things to the instance, but at least makes it harder consistently. | 20:21 |
| dansmith | efried: well, I'd see the main gain as avoiding a case where we fail during cleanup and don't get to the secret-forgetting part | 20:21 |
| dansmith | but yeah | 20:21 |
| sean-k-mooney | dansmith: ya i thinkit sends an ahci shutdown and does nto exit the qemu process too so i think it would be ok i was just not sure | 20:21 |
| dansmith | sean-k-mooney: acpi you mean? | 20:22 |
| sean-k-mooney | sorry yes | 20:22 |
| sean-k-mooney | not storage :) | 20:22 |
| dansmith | otherwise you're talking about usb or firewire | 20:22 |
| dansmith | usb I think | 20:22 |
| sean-k-mooney | ahci is used for sata too | 20:22 |
| efried | ...yeah, confirmed deleting the secret after the guest starts doesn't affect ability of guest to get at the vtpm. | 20:22 |
| dansmith | oh sata right right | 20:22 |
| dansmith | mixed up my *hci acronyms | 20:23 |
| dansmith | efried: almost impossible for that to happen, since qemu and libvirt are fairly separate at runtime | 20:23 |
| dansmith | efried: try a soft reboot from inside the guest tho | 20:23 |
| efried | meaning, from within the guest, just 'reboot'? | 20:23 |
| sean-k-mooney | yes | 20:23 |
| sean-k-mooney | a soft reboot from the api would almost be the same excpet that will escalte to a hard reboot after a time out | 20:24 |
| efried | that seems to have worked fine. Guest came back up, no errors in n-cpu, the tpm is still there and still works. | 20:25 |
| dansmith | figured | 20:25 |
| dansmith | so delete after start is probably the way to go | 20:25 |
| efried | that's what we want though, right? | 20:25 |
| sean-k-mooney | yes | 20:25 |
| dansmith | makes it very clear how long we have the key | 20:25 |
| dansmith | yep | 20:26 |
| efried | like. | 20:26 |
| efried | so assume live migration would carry the VM state, including its decrypted vtpm, and now the state on the dest is consistent: the key is not loaded on either side. | 20:27 |
| dansmith | exactly | 20:27 |
| efried | um, meaning this is an example of an admin operation resulting in a functional vtpm on an instance that didn't "exist" before (on the dest host) where we didn't need the user creds to retrieve the key. | 20:27 |
| efried | I... guess that's still fine. | 20:27 |
| mriedem | dansmith: replied on the cross-cell change https://review.opendev.org/#/c/635684/53/nova/compute/api.py@3852 | 20:28 |
| dansmith | efried: yes, but they still can't cold migrate or start the guest after a host reboot, but that's expected and "desired" | 20:28 |
| dansmith | from the security perspective | 20:28 |
| efried | I guess it's a slightly different attack surface: if I have admin, and I have compromised root on one host in your cloud, I can get your secret by live-migrating your instance to that host and reading it out of memory. | 20:29 |
| efried | um | 20:29 |
| efried | I can read the *tpm* out of memory, not the secret. | 20:29 |
| sean-k-mooney | it depnds on how qemu store it | 20:29 |
| efried | which is just as bad. | 20:29 |
| dansmith | efried: that's more work than you need to do to get the secret really | 20:29 |
| sean-k-mooney | well the tpm data shoudl stilly be encyrpted | 20:30 |
| dansmith | efried: if you're root, you can read all of host memory, you can trigger a crash dump on the qemu, or ptrace it, etc | 20:30 |
| sean-k-mooney | but read and writes to it shoudl be decypted by qemu | 20:30 |
| efried | yeah, point is if you don't have root wherever the instance is currently running, but you do have root somewhere else in the cloud. | 20:30 |
| dansmith | efried: ah yes, sorry.. | 20:30 |
| efried | Is there already a way to mark an instance not migratable? | 20:31 |
| dansmith | you don't even need root anywhere in the cluster to do that attack, fwiw | 20:31 |
| dansmith | all you need is the ability to connect to rabbit and it's over | 20:31 |
| dansmith | given that you can convince a compute you don't even have shell access to to send you the guest's memory contents to a non-secure port | 20:32 |
| sean-k-mooney | so without going on too much of a tangent if you use SGX or other sercure memory enclave technologies you can protect form that | 20:33 |
| sean-k-mooney | but i think that is out of scope fo vTPM | 20:33 |
| sean-k-mooney | with SGX root cant read all memeory | 20:33 |
| sean-k-mooney | and the kernel lockdown feature in 5.4 will also prevent that in some cases | 20:34 |
| dansmith | sean-k-mooney: my attack doesn't require root to be able to read the memory | 20:35 |
| dansmith | I said you don't even need shell on the box with the instance | 20:35 |
| dansmith | my attack might not work with encrypted memory for other reasons, but not because of root | 20:35 |
| sean-k-mooney | sure that was not the point i was makeing | 20:36 |
| sean-k-mooney | https://thenewstack.io/linux-kernel-finally-gets-its-lockdown/ | 20:36 |
| sean-k-mooney | ^ that is a cool feature in 5.4 that shoudl help | 20:36 |
| sean-k-mooney | but anyway i like the idea of removing the key once we boot the instnace | 20:36 |
| efried | So is there a way to indicate I don't want my instance live-migratable? | 20:37 |
| sean-k-mooney | i used to say use sriov/pinning or hugepage but we fixed that :) | 20:37 |
| dansmith | efried: meaning as a user asking the operator to not handle your memory contents over a migration? | 20:37 |
| sean-k-mooney | ill check the libvirt xml | 20:37 |
| sean-k-mooney | maybe | 20:37 |
| dansmith | efried: the user isn't supposed to know that a live migration can or has happened really anyway, so what you're asking for is like an SLA which would happen outside of nova I think | 20:38 |
| efried | dansmith: I guess I meant more like something on the instance so that n-api would simply refuse to migrate | 20:38 |
| dansmith | efried: I know, and I'm saying that's not an appropriate thing for nova to be doing IMHO | 20:38 |
| efried | other than incidentals, like PCI devices (right?), I mean something specific. | 20:38 |
| sean-k-mooney | efried: i mean we coudl just check for vtpm and reject | 20:39 |
| dansmith | no, | 20:39 |
| dansmith | sean-k-mooney: he's saying some user wanting to not be live migrated for risk, | 20:39 |
| dansmith | not nova refusing to migrate vtpm-having instances | 20:39 |
| sean-k-mooney | ah | 20:39 |
| efried | either user (so in the flavor) or admin (conf opt) | 20:39 |
| dansmith | if we can migrate them (which we can I think) then we should | 20:39 |
| dansmith | efried: user doesn't control the flavor | 20:39 |
| sean-k-mooney | thats a similar discustion to the spec fo exposing post copy | 20:39 |
| dansmith | the *details* of the flavor | 20:39 |
| dansmith | but yeah, I'd say the cloud should say "Flavors that end in -secu will never be live migrated for security reasons" and then just handle that policy themselves | 20:40 |
| sean-k-mooney | in that spec they wanted a way to mark a vm as not using post copy to avoid the risk of remote execution over a netwrok link | 20:40 |
| umbSublime | That's something I was asked today. Is there a way to make a vm/flavor _NOT_ live-migratable. Some of our users would for some reason want to opt-out of being live-migratable and understand what that means if we need to do maintenance on the host | 20:42 |
| sean-k-mooney | fyi just looking at the libvirt level i dont see anything in https://libvirt.org/formatdomain.html that would alow us to mark a vm as not live migratable but we could do int at the nova level | 20:45 |
| efried | hm, there's a side effect of the hack that's necessary to transfer the vtpm data meaning that you could disable migration by simply not setting those conf opts. | 20:45 |
| dansmith | sean-k-mooney: I assumed he was looking for a nova tunable not libvirt | 20:45 |
| sean-k-mooney | ya i just said i would look at both levels | 20:46 |
| dansmith | ack | 20:46 |
| sean-k-mooney | efried: which conf options? | 20:47 |
| efried | https://review.opendev.org/#/c/639934/14/nova/conf/libvirt.py | 20:47 |
| sean-k-mooney | the nova migration uri? | 20:47 |
| sean-k-mooney | that would block you form using swtpm too however right | 20:48 |
| sean-k-mooney | why is swtpm_group a choices field | 20:49 |
| sean-k-mooney | that kindof ties distos hands | 20:49 |
| sean-k-mooney | same for user i guess | 20:49 |
| sean-k-mooney | efried: with may kolla-ansible hat on it also forces me to run the swtpm as a specific user or group which is not ideal | 20:50 |
| efried | sean-k-mooney: IIUC the libvirt process takes care of starting the swtpm. | 20:52 |
| efried | I mean, I didn't have to do anything special to start it. | 20:52 |
| sean-k-mooney | ok but im not sure we should be resticting the options in nova | 20:53 |
| sean-k-mooney | e.g. i think that should just be a string field | 20:53 |
| sean-k-mooney | if you do "ps aux | grep swtpm" what user/group is it running as | 20:54 |
| efried | some processes have a `tss` user, some have `root`. | 20:55 |
| efried | sorry, the root ones are just qemu. They have swtpm CLI opts in 'em. | 20:56 |
| * efried ==> school run | 21:01 | |
| *** efried is now known as efried_afk | 21:01 | |
| *** ociuhandu has quit IRC | 21:06 | |
| *** ociuhandu has joined #openstack-nova | 21:07 | |
| *** spsurya has quit IRC | 21:07 | |
| *** ociuhandu has quit IRC | 21:14 | |
| *** ociuhandu has joined #openstack-nova | 21:14 | |
| *** TxGirlGeek has quit IRC | 21:19 | |
| *** mlavalle has quit IRC | 21:22 | |
| *** ociuhandu has quit IRC | 21:24 | |
| *** mlavalle has joined #openstack-nova | 21:41 | |
| *** pcaruana has quit IRC | 21:42 | |
| *** gshippey has quit IRC | 21:44 | |
| *** takashin has joined #openstack-nova | 21:46 | |
| *** efried_afk is now known as efried | 21:53 | |
| *** ociuhandu has joined #openstack-nova | 22:07 | |
| *** slaweq has joined #openstack-nova | 22:11 | |
| *** ociuhandu has quit IRC | 22:11 | |
| *** slaweq has quit IRC | 22:15 | |
| *** dviroel has quit IRC | 22:16 | |
| efried | Not sure why I shouldn't determine the uid/gid based on the swtpm process running on the dest host. | 22:21 |
| efried | I guess discovering which process that is could be a bit tricky. | 22:22 |
| *** ociuhandu has joined #openstack-nova | 22:25 | |
| efried | oh, ahem, the swtpm processes correspond to instances; they don't exist until the tpm does, chicken/egg. | 22:26 |
| sean-k-mooney | efried: is this in relattion to the config options or something else? | 22:27 |
| sean-k-mooney | *relation | 22:27 |
| efried | yeah, I was trying to figure out a way not to need the config options. | 22:28 |
| efried | because they suck. | 22:28 |
| sean-k-mooney | well why do we need them? | 22:28 |
| sean-k-mooney | nova is not startign the swtpm binary | 22:28 |
| sean-k-mooney | right? | 22:28 |
| sean-k-mooney | that is handeled by libvirt | 22:28 |
| efried | the tpm data file needs to be owned by the uid/gid of the swtpm server process | 22:28 |
| sean-k-mooney | will libvirt create that for us? | 22:29 |
| efried | It creates the process | 22:29 |
| efried | it doesn't help us for it to create the file, cause it ain't the right one. | 22:29 |
| sean-k-mooney | im not really a fan of nova having to manage those file permissions | 22:31 |
| sean-k-mooney | espcially if the file is not own by the nova user | 22:31 |
| efried | I agree. Summed up above as "suck". | 22:31 |
| sean-k-mooney | could we make the files be owned by nova? | 22:32 |
| sean-k-mooney | i have not really looked at how this works at a lowerer level | 22:32 |
| *** ociuhandu has quit IRC | 22:33 | |
| sean-k-mooney | also we really should not have a generic recursive_chown function in nova. | 22:34 |
| sean-k-mooney | *need to have | 22:34 |
| *** dklyle has quit IRC | 22:35 | |
| *** david-lyle has joined #openstack-nova | 22:35 | |
| sean-k-mooney | i dont nessisarly know of a better way but from a privsep point of view it sucks to have generic functions like that that take a path user and group | 22:35 |
| sean-k-mooney | efried: is this just needed for the save_and_migrate_vtpm_dir and restore_vtpm_dir functions | 22:39 |
| efried | yes | 22:39 |
| sean-k-mooney | your kind of assuming that nova will have read acess to those folders in places | 22:41 |
| sean-k-mooney | although you might be using privladged functions | 22:41 |
| efried | I haven't really looked at that patch yet (I didn't write it) but I would assume I would need to use privileged operations on both sides. | 22:42 |
| sean-k-mooney | i gues when you do fileutils.ensure_tree(swtpm_dir) to create the folder its in the libvirt instance dir and nova should have write acess to that | 22:42 |
| sean-k-mooney | no all operations would need to be privaladged | 22:43 |
| sean-k-mooney | so that patch is mainly for resize since cold migrate wont work and normal users cant call it | 22:45 |
| efried | On the source, moving the data from its normal location to within the instance dir needs to be privileged to get at the data in its original location | 22:46 |
| efried | Conversely on the dest, creating the real location needs to be privileged; and we should really make sure it's already owned by the right user before moving the contents back out. | 22:46 |
| efried | so I don't see which operations there wouldn't need to be privileged. | 22:46 |
| sean-k-mooney | the ensure_tree function call here https://review.opendev.org/#/c/639934/15/nova/virt/libvirt/utils.py@654 | 22:47 |
| efried | cool, looks like that's the case, no? | 22:47 |
| sean-k-mooney | it is i was just wondering if that need privlages and it was missing or not | 22:48 |
| sean-k-mooney | i would not consider it that unusaly to assume nova can read and write to the virt driver instance folder without needing to elevate privaldges | 22:49 |
| efried | yeah, that's a fair point. | 22:51 |
| sean-k-mooney | its proably fine i was just trying to figure out in what context the config parmanter were used and the surronding context | 22:51 |
| sean-k-mooney | in this case the reason nova has to hanel this is that in the cold migrate case libvirt wont copy these files for us. at least not by default | 22:52 |
| efried | in the live migrate case it won't copy these files either. | 22:52 |
| sean-k-mooney | the commit does not suggest that this is need for live migration | 22:53 |
| efried | what's a "resize between nodes"? | 22:54 |
| sean-k-mooney | your chageing the flavor | 22:54 |
| sean-k-mooney | resize is generally between hosts | 22:54 |
| sean-k-mooney | cold migrate is a special calse fo resize where the flaovr remains the same | 22:54 |
| sean-k-mooney | at least in the libvirt driver | 22:54 |
| dansmith | mriedem: if you're going to flip that to an cast always, I can +W what you have and you can do the error handling cleanup and cast in a change after this but before we turn it on | 22:55 |
| dansmith | mriedem: if you're cool with that | 22:55 |
| dansmith | mriedem: also, I think you'd've needed a reno to explain how migrate _can_ be different sometimes for invisible reasons anyway, so doing one to explain that it's always a little different now is equivalent/better | 22:56 |
| *** rcernin has joined #openstack-nova | 22:58 | |
| mriedem | ok, i can build that into the series before the policy change at the end. as for docs, it would have been part of the cross-cell docs patch that i haven't fleshed out yet, but yeah it's easier to just say generically 'resize / cold migrate is now a cast all the time' | 22:58 |
| mriedem | sean-k-mooney: the flavor always remains the same for cold migrate | 22:59 |
| dansmith | mriedem: I'd prefer it go early in the series the next time you rebase, just so it merges closer to this conversation for context, but.. ack | 22:59 |
| mriedem | the song....that's a different story | 22:59 |
| mriedem | dansmith: ack, i likely need to rebase soonish anyway so i'll push it toward the bottom when i do | 23:00 |
| dansmith | k | 23:00 |
| sean-k-mooney | mriedem: yes that is what i intened to say | 23:01 |
| *** tkajinam has joined #openstack-nova | 23:01 | |
| sean-k-mooney | i was just trying to indicate that when a commit say "When doing a resize or a cold migration between nodes, this data needs to be copied ..." its not refering to live migration | 23:03 |
| sean-k-mooney | efried: this is the qemu docs on the migration process https://github.com/qemu/qemu/blob/9f2ce35dfa4ea4a31dbb765dd02bed2500891887/docs/specs/tpm.txt#L324-L384 | 23:08 |
| efried | IIUC all of that is handled under the covers | 23:09 |
| sean-k-mooney | yes | 23:10 |
| sean-k-mooney | but looking at it i am not sure they every transfer the tpm state independetly | 23:10 |
| sean-k-mooney | it seams to be contained in the testvm.bin create by the migrate command | 23:11 |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460 | 23:14 |
| sean-k-mooney | this would also seam to mesh with our assumtion of automatic stat transfer for the live migration case https://github.com/stefanberger/libvirt-tpm/commit/72299db63690aa4b74131f54398a665a51e3592f | 23:16 |
| *** mlavalle has quit IRC | 23:16 | |
| *** nweinber__ has quit IRC | 23:21 | |
| *** awalende has joined #openstack-nova | 23:21 | |
| *** munimeha1 has quit IRC | 23:25 | |
| *** awalende has quit IRC | 23:26 | |
| *** ociuhandu has joined #openstack-nova | 23:30 | |
| *** ivve has quit IRC | 23:32 | |
| *** ociuhandu has quit IRC | 23:35 | |
| *** maciejjozefczyk has joined #openstack-nova | 23:38 | |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Spec: Ussuri: Encrypted Emulated Virtual TPM https://review.opendev.org/686804 | 23:41 |
| efried | jroll, dansmith, sean-k-mooney: I think that's ready for review now ^ | 23:42 |
| sean-k-mooney | ack. ill review it tomorrow | 23:42 |
| *** ralonsoh has quit IRC | 23:44 | |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Spec: Ussuri: Encrypted Emulated Virtual TPM https://review.opendev.org/686804 | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!