| *** lbragstad has quit IRC | 00:23 | |
| *** tosky has quit IRC | 00:28 | |
| *** martinkennelly has quit IRC | 00:50 | |
| *** lbragstad has joined #openstack-nova | 00:52 | |
| *** bauzas has quit IRC | 01:15 | |
| yonglihe | gibi: thanks raise the VIC TYPE concerns, I tested by wire that in local repo. I try to verify Rodolfo's patch today: https://review.opendev.org/c/openstack/neutron/+/779292 | 01:17 |
|---|---|---|
| *** mlavalle has quit IRC | 01:21 | |
| *** hamalq has quit IRC | 01:24 | |
| gmann | stephenfin: gibi updated the policy patch for modernize-os-hypervisors-api BP https://review.opendev.org/c/openstack/nova/+/765798/5 | 01:31 |
| *** k_mouza has joined #openstack-nova | 01:40 | |
| *** ociuhandu has joined #openstack-nova | 01:42 | |
| *** k_mouza has quit IRC | 01:45 | |
| *** ociuhandu has quit IRC | 01:46 | |
| *** macz_ has quit IRC | 01:55 | |
| *** bauzas has joined #openstack-nova | 02:14 | |
| *** spatel_ has joined #openstack-nova | 02:15 | |
| *** rcernin has quit IRC | 02:25 | |
| *** rcernin has joined #openstack-nova | 02:42 | |
| *** rcernin has quit IRC | 02:48 | |
| *** rcernin has joined #openstack-nova | 02:51 | |
| *** macz_ has joined #openstack-nova | 03:12 | |
| *** macz_ has quit IRC | 03:16 | |
| *** rcernin has quit IRC | 03:20 | |
| *** psachin has joined #openstack-nova | 03:29 | |
| *** ociuhandu has joined #openstack-nova | 03:30 | |
| *** ociuhandu has quit IRC | 03:34 | |
| *** rcernin has joined #openstack-nova | 03:40 | |
| *** lbragstad has quit IRC | 03:43 | |
| *** zzzeek has quit IRC | 03:46 | |
| *** zzzeek has joined #openstack-nova | 03:47 | |
| *** sapd1_x has joined #openstack-nova | 03:50 | |
| *** efried1 has joined #openstack-nova | 03:50 | |
| *** david-lyle has joined #openstack-nova | 03:50 | |
| *** eandersson8 has joined #openstack-nova | 03:51 | |
| *** adriant4 has joined #openstack-nova | 03:51 | |
| *** dpawlik6 has joined #openstack-nova | 03:51 | |
| *** rcernin_ has joined #openstack-nova | 03:51 | |
| *** gary_perkins_ has joined #openstack-nova | 03:56 | |
| *** owalsh_ has joined #openstack-nova | 03:58 | |
| *** rcernin has quit IRC | 03:59 | |
| *** gyee has quit IRC | 03:59 | |
| *** amodi has quit IRC | 03:59 | |
| *** sapd1 has quit IRC | 03:59 | |
| *** hoonetorg has quit IRC | 03:59 | |
| *** mjturek has quit IRC | 03:59 | |
| *** openstackgerrit has quit IRC | 03:59 | |
| *** efried has quit IRC | 03:59 | |
| *** dklyle has quit IRC | 03:59 | |
| *** adriant has quit IRC | 03:59 | |
| *** gary_perkins has quit IRC | 03:59 | |
| *** eandersson has quit IRC | 03:59 | |
| *** tbarron has quit IRC | 03:59 | |
| *** cgoncalves has quit IRC | 03:59 | |
| *** dpawlik has quit IRC | 03:59 | |
| *** owalsh has quit IRC | 03:59 | |
| *** adriant4 is now known as adriant | 03:59 | |
| *** efried1 is now known as efried | 03:59 | |
| *** eandersson8 is now known as eandersson | 03:59 | |
| *** dpawlik6 is now known as dpawlik | 03:59 | |
| *** gyee has joined #openstack-nova | 04:01 | |
| *** vishalmanchanda has joined #openstack-nova | 04:16 | |
| *** vishalmanchanda has quit IRC | 04:21 | |
| *** vishalmanchanda has joined #openstack-nova | 04:21 | |
| *** songwenping__ has joined #openstack-nova | 04:28 | |
| *** songwenping_ has quit IRC | 04:32 | |
| *** ratailor has joined #openstack-nova | 04:34 | |
| *** whoami-rajat_ has joined #openstack-nova | 04:41 | |
| *** whoami-rajat_ is now known as whoami-rajat | 04:46 | |
| *** jangutter has quit IRC | 05:15 | |
| *** jangutter has joined #openstack-nova | 05:16 | |
| *** ociuhandu has joined #openstack-nova | 05:18 | |
| *** ociuhandu has quit IRC | 05:22 | |
| *** iurygregory has quit IRC | 05:26 | |
| *** k_mouza has joined #openstack-nova | 05:37 | |
| *** k_mouza has quit IRC | 05:41 | |
| *** macz_ has joined #openstack-nova | 05:48 | |
| *** rcernin_ has quit IRC | 05:48 | |
| *** rcernin_ has joined #openstack-nova | 05:49 | |
| *** macz_ has quit IRC | 05:52 | |
| *** spatel_ has quit IRC | 06:42 | |
| *** slaweq has joined #openstack-nova | 06:50 | |
| *** ociuhandu has joined #openstack-nova | 07:00 | |
| *** ociuhandu has quit IRC | 07:06 | |
| *** cgoncalves has joined #openstack-nova | 07:09 | |
| *** bauzas has quit IRC | 07:16 | |
| *** gyee has quit IRC | 07:29 | |
| *** vishalmanchanda has quit IRC | 07:36 | |
| *** ociuhandu has joined #openstack-nova | 07:43 | |
| *** david-lyle has quit IRC | 07:46 | |
| *** dklyle has joined #openstack-nova | 07:48 | |
| *** xek has joined #openstack-nova | 07:48 | |
| *** bauzas has joined #openstack-nova | 07:54 | |
| *** khomesh24 has joined #openstack-nova | 07:56 | |
| *** tesseract has joined #openstack-nova | 08:01 | |
| *** dklyle has quit IRC | 08:01 | |
| *** lpetrut has joined #openstack-nova | 08:04 | |
| *** tesseract has quit IRC | 08:12 | |
| *** tesseract has joined #openstack-nova | 08:13 | |
| *** songwenping_ has joined #openstack-nova | 08:14 | |
| *** songwenping__ has quit IRC | 08:18 | |
| *** andrewbonney has joined #openstack-nova | 08:19 | |
| *** rpittau|afk is now known as rpittau | 08:24 | |
| *** rcernin_ has quit IRC | 08:24 | |
| *** macz_ has joined #openstack-nova | 08:41 | |
| *** martinkennelly has joined #openstack-nova | 08:41 | |
| *** macz_ has quit IRC | 08:46 | |
| *** ociuhandu has quit IRC | 08:48 | |
| *** ociuhandu has joined #openstack-nova | 08:49 | |
| *** ociuhandu has quit IRC | 08:49 | |
| yonglihe | gibi: alex_xu:, F.Y.I I've talke with ralonsoh and slaweq, that neutron patch is likely merged today: https://review.opendev.org/c/openstack/neutron/+/779292 | 08:52 |
| *** sapd1 has joined #openstack-nova | 08:57 | |
| *** derekh has joined #openstack-nova | 09:00 | |
| lyarwood | melwitt: ack np looking | 09:01 |
| *** tosky has joined #openstack-nova | 09:02 | |
| lyarwood | elod: https://review.opendev.org/c/openstack/nova/+/777217/ - I think you forgot to vote on this? | 09:04 |
| *** ociuhandu has joined #openstack-nova | 09:04 | |
| *** lucasagomes has joined #openstack-nova | 09:05 | |
| *** ftarasenko has joined #openstack-nova | 09:06 | |
| *** ociuhandu has quit IRC | 09:14 | |
| *** ociuhandu has joined #openstack-nova | 09:15 | |
| elod | lyarwood: wow. apparently. thx. :S | 09:20 |
| *** ociuhandu has quit IRC | 09:20 | |
| lyarwood | elod: np I've done that lots with the new UI | 09:20 |
| elod | yeah, let's blame that o:) I don't know how i missed to vote :S sorry :S | 09:22 |
| *** ociuhandu has joined #openstack-nova | 09:28 | |
| *** k_mouza has joined #openstack-nova | 09:32 | |
| *** k-s-dean has joined #openstack-nova | 09:37 | |
| stephenfin | oh, no Gerrit bot? | 09:48 |
| stephenfin | trivial fix to address some issues coming down the pipeline with setuptools here https://review.opendev.org/c/openstack/nova/+/779449 | 09:48 |
| jrosser | could i get some help with this when debugging why i can't rescue boot-from-volume images https://opendev.org/openstack/nova/src/branch/master/nova/api/openstack/compute/rescue.py#L60 | 09:52 |
| jrosser | if i print req.api_version_request i get "API Version Request Major: 2, Minor: 1" | 09:53 |
| jrosser | that seems always going to fail if passed to api_version_request.is_supported(req, '2.87') | 09:53 |
| stephenfin | jrosser: How are you making the request/ | 09:55 |
| jrosser | stephenfin: either through horizon or `openstack server rescue --image 14600a9b-a240-4210-8a32-cea43d0499ac 4f2b0a96-7b6b-4d01-bb75-1248574e71d6` | 09:57 |
| stephenfin | you need to request API microversion 2.87, as that code would suggest | 09:57 |
| stephenfin | openstack --os-compute-api-version 2.87 server rescue ... | 09:57 |
| stephenfin | should do the trick | 09:57 |
| stephenfin | OSC currently defaults to 2.1. We're working on fixing that | 09:58 |
| *** jawad_axd has joined #openstack-nova | 09:58 | |
| stephenfin | I don't know how to do the equivalent in Horizon | 09:58 |
| jrosser | ooooohhhh - well thats explains it! | 09:58 |
| jrosser | perhaps that warrants a docs fix here https://docs.openstack.org/nova/latest/user/rescue.html | 09:59 |
| stephenfin | Yes, good point | 09:59 |
| stephenfin | lyarwood: Maybe if you have time in the next few weeks? ^ | 09:59 |
| *** k_mouza has quit IRC | 10:04 | |
| *** k_mouza has joined #openstack-nova | 10:05 | |
| *** k-s-dean has quit IRC | 10:10 | |
| jawad_axd | Hi folks, how luks password is generated for instance booted from encrypted volume? Barbican is used in environment. I can grab luks password like “virsh secret-get-value SECRET-UUID | —base64 —decode” when instance is running on compute node. How can I generate this password on my own? Use case is: if instance is backed up outside somewhere and needs to boot standalone, thus needs luks passwor | 10:11 |
| jawad_axd | d.Any comments on this?Thanks | 10:11 |
| jrosser | jawad_axd: there was a thread on the mailing list about this recently http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020374.html | 10:15 |
| *** k_mouza has quit IRC | 10:17 | |
| jawad_axd | @jrosser I missed that one. I look into it. Thanks | 10:17 |
| *** k_mouza has joined #openstack-nova | 10:20 | |
| lyarwood | stephenfin: sorry was afk, reading | 10:21 |
| lyarwood | huh how did I miss that | 10:23 |
| *** rcernin_ has joined #openstack-nova | 10:27 | |
| lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/779479 | 10:32 |
| lyarwood | jawad_axd: hey sorry reading | 10:32 |
| lyarwood | jawad_axd: yeah that post on the ML spells out how to grab the passphrase from the key manager, it's awful I know but thanks to design choices made before my time :/ | 10:33 |
| lyarwood | jawad_axd: http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020421.html specifically lists the steps | 10:34 |
| *** jangutter has quit IRC | 10:41 | |
| jawad_axd | @lyarwood Thanks. Apparently, it requires volume to to be mapped. I am wondering if there is an easy way to decrypt/decode it from barbican cli, when encrypted volume exists in environment and is in in AVAILABLE state.a | 10:42 |
| *** jangutter has joined #openstack-nova | 10:42 | |
| *** jangutter has quit IRC | 10:43 | |
| *** jangutter has joined #openstack-nova | 10:44 | |
| lyarwood | jawad_axd: No, you can't stream/decode an unmapped encrypted volume. | 10:44 |
| lyarwood | jawad_axd: unlike Glance, Cinder doesn't send actual volume data via the API | 10:45 |
| lyarwood | jawad_axd: so you will need to map the encrypted volume to a host before being able to decrypt it | 10:45 |
| *** iurygregory_ has joined #openstack-nova | 10:45 | |
| lyarwood | jawad_axd: as the owner of the volume you could snapshot it and download the encrypted volume snapshot from glance I guess | 10:46 |
| *** iurygregory_ is now known as iurygregory | 10:46 | |
| lyarwood | jawad_axd: that would give you a local file you could then decrypt but it's not the live volume | 10:46 |
| gibi | stephenfin: question in https://review.opendev.org/c/openstack/nova/+/779449/1/setup.cfg#11 | 10:47 |
| *** k_mouza has quit IRC | 10:56 | |
| jawad_axd | @lyarwood Ok.. if I understand correctly, luks header is attached to 'rbd disk in my case' when we create instance from encrypted volume, is that right ? My understanding is, when we create encrypted volume from image, it attaches luks header to volume_from_image that time, if its true then it should be possible to retrieve luks password when it is not mapped. | 10:57 |
| lyarwood | jawad_axd: the header is always attached but the actual passphrase is stored outside of that in the key manager and doesn't require access to the volume or image | 10:59 |
| lyarwood | jawad_axd: to be clear, fetching the passphrase and decrypting data are two seperate steps | 11:01 |
| lyarwood | jawad_axd: you don't need access to the volume to fetch the passphrase from the key manager | 11:01 |
| lyarwood | jawad_axd: you do need access to the volume or a downloaded image to decrypt the actual data held within them | 11:01 |
| jawad_axd | @lyarwood I am actually concerned about passphrase here. How to get it from key manager.. I | 11:02 |
| lyarwood | jawad_axd: kk then it's this part | 11:03 |
| lyarwood | openstack secret get --payload_content_type 'application/octet-stream' http://192.168.122.208/key-manager/v1/secrets/6fd4f879-005d-4b7d-9e5f-2505f010be7c --file mysecret.key | 11:03 |
| lyarwood | jawad_axd: ^ where the URL is provided by `openstack secret list` and lists your secret UUID | 11:03 |
| * lyarwood forgets if cinder lists the secret uuid somewhere | 11:03 | |
| lyarwood | ah yeah it does | 11:04 |
| lyarwood | https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=show-a-volume-s-details-detail#show-a-volume-s-details | 11:04 |
| lyarwood | if you do an `openstack volume show $volume` | 11:04 |
| lyarwood | encryption_key_id is the secret uuid | 11:04 |
| lyarwood | jawad_axd: `hexdump -e '16/1 "%02x"' mysecret.key` will then give you the passphrase used to unlock the LUKS header | 11:05 |
| lyarwood | which is awful | 11:05 |
| lyarwood | but here we are | 11:05 |
| lyarwood | jawad_axd: FWIW by default only the owner of the volume should have access to this secret | 11:06 |
| *** jangutter has quit IRC | 11:07 | |
| *** jangutter has joined #openstack-nova | 11:07 | |
| jawad_axd | @lyarwood Yesss... I just tested and got passphrase from "hexdump -e '16/1 "%02x"' mysecret.key" . Very cool. Thanks alot. | 11:12 |
| *** sapd1 has quit IRC | 11:16 | |
| *** dtantsur|afk is now known as dtantsur | 11:25 | |
| *** rcernin_ has quit IRC | 11:26 | |
| *** Luzi has joined #openstack-nova | 11:27 | |
| *** jangutter has quit IRC | 11:30 | |
| *** jangutter has joined #openstack-nova | 11:30 | |
| *** jangutter has quit IRC | 11:42 | |
| *** jangutter has joined #openstack-nova | 11:42 | |
| *** ociuhandu_ has joined #openstack-nova | 11:49 | |
| *** artom has quit IRC | 11:51 | |
| *** slaweq_ has joined #openstack-nova | 11:52 | |
| *** ociuhandu has quit IRC | 11:53 | |
| *** ociuhandu_ has quit IRC | 11:53 | |
| *** brinzhang has quit IRC | 11:56 | |
| *** lpetrut has quit IRC | 11:56 | |
| *** k_mouza has joined #openstack-nova | 11:56 | |
| *** ociuhandu has joined #openstack-nova | 11:57 | |
| *** lpetrut has joined #openstack-nova | 11:58 | |
| *** slaweq has quit IRC | 11:59 | |
| *** ociuhandu has quit IRC | 12:03 | |
| *** ociuhandu has joined #openstack-nova | 12:13 | |
| *** ociuhandu has quit IRC | 12:18 | |
| *** artom has joined #openstack-nova | 12:25 | |
| *** slaweq_ is now known as slaweq | 12:38 | |
| *** psachin has quit IRC | 12:47 | |
| stephenfin | gibi: Done | 12:50 |
| *** __ministry1 has joined #openstack-nova | 12:52 | |
| jrosser | stephenfin: thankyou for the api version tip before, i am now able to put a boot-from-volume instance into rescue | 12:52 |
| stephenfin | hth | 12:52 |
| jrosser | it doesnt quite behave as i expect though :) | 12:53 |
| jrosser | i seem to get the original instance root disk as the root disk in the rescue instance | 12:53 |
| jrosser | this is the xml i get for the disks http://paste.openstack.org/show/803376/ | 12:54 |
| *** zzzeek has quit IRC | 13:00 | |
| *** zzzeek has joined #openstack-nova | 13:02 | |
| *** lucasagomes has quit IRC | 13:04 | |
| lyarwood | jrosser: yup that's by design, it's using stable device rescue so the original disks are presented first | 13:05 |
| lyarwood | jrosser: with the rescue disk last | 13:05 |
| jrosser | but i would expect it to boot off the USB disk in that case? | 13:06 |
| lyarwood | jrosser: note the boot order element, it boots from the rescue disk | 13:06 |
| gibi | stephenfin: thanks +2 | 13:06 |
| jrosser | yeah, so with hw_rescue_bus=usb i don't see it boot from the usb device | 13:06 |
| lyarwood | jrosser: is it the same image in the volume and rescue image? | 13:07 |
| jrosser | it should be, +/- the snapshots in the ceph backend | 13:07 |
| lyarwood | jrosser: right so are you sure it's booting from the original volume? | 13:07 |
| lyarwood | this caught be out a few times while testing this | 13:08 |
| jrosser | yes, its confusing, one moment | 13:08 |
| lyarwood | hmm actually I wonder if this is a valid bug when using a different bus | 13:09 |
| jrosser | ok before i check that, i had another thing with hw_rescue_bus=scsi | 13:09 |
| lyarwood | is there the original boot element higher in the XML? | 13:09 |
| jrosser | in the instance console bus=scsi gives "no bootable device" | 13:09 |
| * jrosser puts this back to USB | 13:10 | |
| jrosser | lyarwood: i've just redone it with the bus=usb http://paste.openstack.org/show/803378/ | 13:14 |
| sean-k-mooney | lyarwood: we dont really suppor t mixing buses properly | 13:14 |
| sean-k-mooney | i dont think we generate the contoler properly in all cases | 13:14 |
| lyarwood | sean-k-mooney: the instance was already using SCSI in the bus=scsi case so that should work | 13:15 |
| lyarwood | sean-k-mooney: and bus=usb worked for me while I was testing this back in the day with virtio disks attached | 13:16 |
| *** vishalmanchanda has joined #openstack-nova | 13:16 | |
| sean-k-mooney | yep but i thik we have some edge cases with scsi and virtio blk | 13:16 |
| lyarwood | yeah | 13:16 |
| sean-k-mooney | usb would likely have worked because the usb contoler was previously always added by libvirt | 13:17 |
| sean-k-mooney | i have seen issue where the scsi contoler was not alwasy added but that might have been fixed by now | 13:17 |
| lyarwood | jrosser: just building an env now to play with this | 13:17 |
| jrosser | lyarwood: oh cool, thankyou :) | 13:18 |
| lyarwood | jrosser: oh wait, does this reproduce if you use a different image? | 13:18 |
| jrosser | i can try that | 13:18 |
| lyarwood | yeah please, it might be the rescue disk is finding the original disk first and mounting it as the root filesystem as the labels match | 13:19 |
| lyarwood | so we are booted into the kernel from the rescue disk using the filesystem from the original | 13:19 |
| jrosser | also somewhat contrary to the stuff right at the end of here, leaving --image off does some fail-y thing i've not yet found https://docs.openstack.org/nova/latest/user/rescue.html | 13:20 |
| lyarwood | hmm with a boot from volume instance it should try to boot from the original image referenced by the volume, if one is present. I forget what the behaviour is if an image wasn't used to create the volume. | 13:22 |
| * lyarwood grabs some lunch while devstack builds | 13:24 | |
| *** lbragstad has joined #openstack-nova | 13:24 | |
| jrosser | from the wording i'd inferred that no --image would make it use the 'default', i.e the one from the instance being rescued | 13:24 |
| sean-k-mooney | do we support rescue for BFV | 13:28 |
| sean-k-mooney | we didnt for a long time | 13:28 |
| sean-k-mooney | i think it was added in the last 2-3 cycle but cant recall if it landed | 13:28 |
| jrosser | ussuri i think | 13:28 |
| sean-k-mooney | ya i know we still have no supprot for rebuild with bfv | 13:29 |
| jrosser | ah thats interesting, changing the rescue image to one != the original instance makes things work a whole lot better | 13:30 |
| sean-k-mooney | jrosser: the image for rescue if you dont pass an image is the image use to boot the vm unless a rescue image is set in the nova.conf https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.rescue_image_id | 13:30 |
| jrosser | i rescued a focal vm with bionic image and thats now as i expect | 13:30 |
| sean-k-mooney | jrosser: yep it normally does | 13:30 |
| sean-k-mooney | it should work in both casees but you often dont have the same disk lable issues | 13:31 |
| jrosser | theres a bunch of trap doors for the unwary here :) | 13:31 |
| sean-k-mooney | for what its worth i have generally not had issues with this. its typeically just worked | 13:32 |
| sean-k-mooney | even before the stable rescuse work | 13:32 |
| *** jangutter has quit IRC | 13:34 | |
| *** jangutter has joined #openstack-nova | 13:34 | |
| jrosser | heres what i get if i don't pass --image http://paste.openstack.org/show/803379/ | 13:35 |
| lyarwood | jrosser: kk that's a bug | 13:36 |
| lyarwood | jrosser: but glad the original issue is resolved at least | 13:36 |
| lyarwood | https://github.com/openstack/nova/blob/31889ce296d1e1a62fe5825292479009118ddfab/nova/compute/manager.py#L4123-L4130 doesn't look right | 13:38 |
| jrosser | lyarwood: would you expect hw_rescue_bus=scsi to work? | 13:38 |
| lyarwood | jrosser: with a different image and an instance that already had a disk attached via SCSI yes | 13:39 |
| jrosser | feels like something else there as i get "No Bootable device" in the instance console | 13:39 |
| lyarwood | jrosser: would you mind raising a bug for that and the API error above when --image is missing? | 13:44 |
| lyarwood | I'm not sure about the SCSI failure to find a boot device tbh | 13:44 |
| *** jangutter has quit IRC | 13:44 | |
| lyarwood | unless again it's something weird with the image | 13:44 |
| sean-k-mooney | jrosser: which image did you set the rescue bus on? | 13:44 |
| *** jangutter has joined #openstack-nova | 13:44 | |
| jrosser | the one i'm specifying with --image | 13:45 |
| sean-k-mooney | if you dont specify it it would have to be on the original image | 13:45 |
| sean-k-mooney | ya ok that should be the image whos metadata we use | 13:45 |
| jrosser | in this case turns out those options are set on both images i've tried as the rescue image now | 13:46 |
| jrosser | one of which is the original instance image | 13:47 |
| *** sapd1 has joined #openstack-nova | 13:48 | |
| *** spatel_ has joined #openstack-nova | 13:48 | |
| *** spatel_ has quit IRC | 13:50 | |
| *** ratailor has quit IRC | 13:53 | |
| *** tkajinam has quit IRC | 13:53 | |
| *** amodi has joined #openstack-nova | 13:54 | |
| *** spatel_ has joined #openstack-nova | 13:56 | |
| *** songwenping_ has quit IRC | 14:01 | |
| *** songwenping_ has joined #openstack-nova | 14:01 | |
| jrosser | lyarwood: bug reports done | 14:13 |
| jrosser | thankyou again for your help, i have something usable now with the usb bus and understanding the need for a different rescue image | 14:14 |
| *** lucasagomes has joined #openstack-nova | 14:16 | |
| *** lucasagomes has quit IRC | 14:16 | |
| *** jangutter has quit IRC | 14:16 | |
| lyarwood | jrosser: np and thanks for the bugs, I'll try to get them resolved after feature freeze later this week | 14:17 |
| *** lucasagomes has joined #openstack-nova | 14:17 | |
| *** jangutter has joined #openstack-nova | 14:17 | |
| *** abhishekk is now known as konan | 14:26 | |
| *** konan is now known as abhishekk | 14:28 | |
| *** Luzi has quit IRC | 14:34 | |
| sean-k-mooney | lyarwood: by the way do you know why the volumn detach is sometime failing in the live migration job | 14:36 |
| sean-k-mooney | looks like its hitting nova.exception.DeviceDetachFailed: Device detach failed for vdb: Unable to detach the device from the live config. | 14:40 |
| *** __ministry1 has quit IRC | 14:41 | |
| lyarwood | sean-k-mooney: no, I've been trying to push gibi's rework along to see if that resolved it tbh | 14:41 |
| sean-k-mooney | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unable%20to%20detach%20the%20device%20from%20the%20live%20config%5C%22%20AND%20loglevel%3A%20ERROR | 14:42 |
| lyarwood | sean-k-mooney: there's nothing obvious in the logs but I wonder if it could be related to https://bugs.launchpad.net/cinder/+bug/1917750 | 14:43 |
| openstack | Launchpad bug 1917750 in Cinder "Running parallel iSCSI/LVM c-vol backends is causing random failures in CI" [Undecided,New] | 14:43 |
| sean-k-mooney | ~300 hits in 30 days | 14:43 |
| sean-k-mooney | im seeing some libvirt issue on the contoler too not the node with teh detach issue | 14:43 |
| lyarwood | do you have an example to hand? | 14:44 |
| sean-k-mooney | am its hitting my vdpa pataches but also neutron let me get one | 14:45 |
| sean-k-mooney | so ya https://review.opendev.org/c/openstack/nova/+/778350/4 https://zuul.opendev.org/t/openstack/build/fb643b53835341ac8589afeadfa7044d/logs | 14:47 |
| * lyarwood cracks knuckles | 14:48 | |
| sean-k-mooney | its showing up in neutron too https://review.opendev.org/c/openstack/neutron/+/777785 | 14:49 |
| *** artom has quit IRC | 14:49 | |
| sean-k-mooney | in https://zuul.opendev.org/t/openstack/build/32f3dd64008b469eb9fb8b13ed33f137 | 14:49 |
| sean-k-mooney | so i think this is just an issue with master in general | 14:50 |
| sean-k-mooney | it could be related to https://bugs.launchpad.net/cinder/+bug/1917750 maybe havent looked at it yet | 14:50 |
| openstack | Launchpad bug 1917750 in Cinder "Running parallel iSCSI/LVM c-vol backends is causing random failures in CI" [Undecided,New] | 14:50 |
| *** ociuhandu has joined #openstack-nova | 14:51 | |
| lyarwood | sean-k-mooney: yeah I think it's related | 14:57 |
| *** ociuhandu has quit IRC | 14:57 | |
| lyarwood | gah | 15:01 |
| lyarwood | yeah it's that | 15:01 |
| lyarwood | so we end up in a situation where one test attaches a volume to the host as /dev/sda from c-vol LVM/iSCSI backend #1 | 15:02 |
| lyarwood | another test then attaches another volume with the same WWN to the hsot as /dev/sdb from c-vol LVM/iSCSI backend #2 | 15:02 |
| lyarwood | the first test finishes and removes what it thinks is the first volume attachment | 15:03 |
| lyarwood | but it's actually the second | 15:03 |
| lyarwood | leaving libvirt unable to detach the device from the instance as I assume it can't flush | 15:03 |
| lyarwood | I guess our current code swallows or ignores the failure from the libvirt and just retries? | 15:04 |
| sean-k-mooney | porbaly ya | 15:04 |
| sean-k-mooney | we dont use the events yet | 15:04 |
| sean-k-mooney | so we need to revert running those n parrallel | 15:04 |
| sean-k-mooney | although is this not a os-brick bug | 15:05 |
| sean-k-mooney | i mean the concurrent attach should work | 15:05 |
| lyarwood | yeah it's a job topology bug | 15:05 |
| lyarwood | it `works` but the first test will end up sending I/O to the volume from the second test | 15:05 |
| lyarwood | something has changed recently somewhere in the stack to allow both backends to return the same WWN tbh | 15:06 |
| bauzas | stephenfin: apologies for forgetting that a string is immutable | 15:06 |
| * bauzas hides | 15:06 | |
| lyarwood | unless people have always missed these failures | 15:06 |
| sean-k-mooney | lyarwood: would not not be possible thouhg in general | 15:06 |
| sean-k-mooney | for them to have the same WWN | 15:06 |
| lyarwood | not with a real world backend | 15:07 |
| lyarwood | https://en.wikipedia.org/wiki/World_Wide_Name | 15:08 |
| *** ociuhandu has joined #openstack-nova | 15:08 | |
| sean-k-mooney | lyarwood: ya i tought if you have iscsi portals from different sotrage backend thw WWN was only uniq within any given portal | 15:12 |
| sean-k-mooney | lyarwood: i dont think cinder is choosing the WWN for the volume | 15:13 |
| *** mjturek has joined #openstack-nova | 15:14 | |
| lyarwood | right yeah that would make sense | 15:16 |
| * lyarwood builds a test env to prove wtf is going on here | 15:21 | |
| stephenfin | bauzas: nw. I've replied to the rest of your comments also | 15:22 |
| *** lpetrut has quit IRC | 15:29 | |
| bauzas | yeah I'll review again | 15:42 |
| gibi | lyarwood: do I understand correclty from the scrollback that making some cinder test serial would help merging patches? | 15:42 |
| lyarwood | gibi: removing c-vol from the computes would | 15:42 |
| lyarwood | gibi: if we really need to run another c-vol service then it needs to be on the controller alongside the original | 15:43 |
| lyarwood | gibi: just building an env now to prove things | 15:43 |
| stephenfin | lyarwood, gibi, sean-k-mooney: any aversion to adding mypy to pre-commit? | 15:43 |
| sean-k-mooney | not speciriclly no | 15:43 |
| stephenfin | assuming I can do it while respecting mypy-files | 15:44 |
| lyarwood | stephenfin: against the files that are changing? | 15:44 |
| gibi | lyarwood: thanks for looking into that, sign me up for review where there is something I can push | 15:44 |
| sean-k-mooney | i have been bitten by flake8 passed by pep8 didnt | 15:44 |
| stephenfin | yeah | 15:44 |
| gibi | lyarwood: does having c-vol along with n-cpu is an invalid config? | 15:44 |
| lyarwood | stephenfin: no issues assuming it isn't adding a huge amount of delay | 15:44 |
| sean-k-mooney | i have gotten used to relying on pre-commit to do that form me | 15:44 |
| stephenfin | sean-k-mooney: me too :( I ran it on HEAD but it turns out I broke something then fixed it in the next change | 15:44 |
| stephenfin | me too * 2 :) | 15:44 |
| stephenfin | pre-commit FTW | 15:44 |
| gibi | stephenfin: no problem for me I dont use pre-commit :) | 15:44 |
| *** ociuhandu has quit IRC | 15:45 | |
| stephenfin | gibi: You're missing out. It's pretty great :) | 15:45 |
| lyarwood | gibi: no, the issue here is with the default c-vol backend, LVM/iSCSI, that you can't have >1 running version of in an env AFAICT | 15:45 |
| *** ociuhandu has joined #openstack-nova | 15:45 | |
| * bauzas disappears for around 1.5h | 15:45 | |
| gibi | stephenfin: I do run fast8 py39 and functional-py39 on every commit I push (except on old stable branches where there is no py39 or py38 support) | 15:46 |
| bauzas | wife is sick and I need to taxi her to the doctor | 15:46 |
| lyarwood | gibi: we end up mapping different volumes from different backends to the computes with the same WWN (world wide name) that should be unqiue | 15:46 |
| gibi | lyarwood: so we cannot have multiple backend per compute? | 15:46 |
| stephenfin | gibi: That's better than me. I run what I think are relevant tests and then let the CI do the rest. Our tests take too long to run locally | 15:46 |
| lyarwood | gibi: we can't have multiple LVM/iSCSI backed c-vol's in the same env | 15:46 |
| * stephenfin might do some work on that in Xena | 15:47 | |
| gibi | stephenfin: I have a beefy blade in a lab to run unit and func test (and devstack on baremetal to test sriov) | 15:47 |
| * lyarwood has some _unit_last and _func_last commands that run unit and func for any test files touched | 15:47 | |
| *** ociuhandu has quit IRC | 15:47 | |
| lyarwood | it's not great but better than nothing | 15:47 |
| stephenfin | I've a four year old laptop 0:) | 15:47 |
| *** ociuhandu has joined #openstack-nova | 15:47 | |
| lyarwood | refresh is 3 soooooooooooo ;) | 15:48 |
| *** dklyle has joined #openstack-nova | 15:48 | |
| gibi | still a laptop is slow, you should ask for a lab :) | 15:48 |
| * stephenfin might not have managed to switch everything over yet... | 15:48 | |
| gibi | lyarwood: so in a real deployment there can only one c-vol service using the LVM/iSCSI backend? | 15:49 |
| gibi | lyarwood: sorry that I'm slow to understant this :) | 15:49 |
| lyarwood | gibi: I don't think that has ever been enforced but the LVM/iSCSI backend is only ever used for testing | 15:50 |
| gibi | OK, then I stop worrying :) | 15:50 |
| gibi | it is just test :) | 15:50 |
| *** jangutter_ has joined #openstack-nova | 15:51 | |
| lyarwood | I'll just caveat all of the above with the fact that no one from Cinder has agreed with any of that in the bug as yet | 15:52 |
| lyarwood | so I might be missing the point entirely | 15:52 |
| lyarwood | but two volumes with the same WWN connected to the same host smells like something that will bork devicemapper | 15:52 |
| *** stephenfin has quit IRC | 15:53 | |
| gibi | if it fixes the CI and let us merge the api db compaction before FF then I'm happy to take the hit if we need to revert the change later | 15:53 |
| gibi | :) | 15:53 |
| *** jangutte_ has joined #openstack-nova | 15:53 | |
| lyarwood | gibi: is that stuck in a recheck loop? | 15:53 |
| gibi | pretty much yes | 15:54 |
| lyarwood | gah okay | 15:54 |
| *** stephenfin has joined #openstack-nova | 15:54 | |
| lyarwood | let me confirm and then I can push some changes | 15:54 |
| gibi | the Queens one is recheckd through the whole weekend and still bouncing back | 15:54 |
| gibi | mostly with the detach issue | 15:54 |
| gibi | but also kernel panic, and recenlty with POST_FAILURe | 15:54 |
| *** jangutter has quit IRC | 15:54 | |
| *** jangutter_ has quit IRC | 15:56 | |
| stephenfin | sean-k-mooney: because I didn't write it down, can you remind me again how you were suggesting me map a project to a hypervisor in '/os-hypervisors'? Was it metadata? | 16:07 |
| stephenfin | sean-k-mooney: to clarify, if I say "get all hypervisors relevant to this user", what should I be filtering on? | 16:07 |
| sean-k-mooney | use the existing metadtaa keys for tenant isolation | 16:10 |
| sean-k-mooney | one sec ill get it | 16:11 |
| sean-k-mooney | https://docs.openstack.org/nova/latest/admin/aggregates.html#tenant-isolation-with-placement | 16:12 |
| sean-k-mooney | you would be looking for filter_tenant_id* | 16:12 |
| lyarwood | sigh, why is devstack writing out /etc/cinder/cinder-api-uwsgi.ini when we only deploy c-vol | 16:12 |
| stephenfin | sean-k-mooney: great | 16:12 |
| sean-k-mooney | basically if the host is not a member of an aggreate with filter_tenant_id* then anyone coudl view it | 16:14 |
| sean-k-mooney | if it is then only does listed in that can view it | 16:14 |
| sean-k-mooney | the way i was suggsing doing it was look for all host wiht filter_tenant_id=<my-project> and if that is none then allow all hosts | 16:16 |
| sean-k-mooney | you coudl do it other ways fo course but i think that is what i suggested in the past | 16:16 |
| stephenfin | gmann, gibi: replied on https://review.opendev.org/c/openstack/nova/+/765798 | 16:16 |
| stephenfin | the policy checks are correct, but we're not filtering the compute nodes retrieved | 16:16 |
| * sean-k-mooney didnt write that down either unless i left it in a comment | 16:16 | |
| gibi | stephenfin: good potin | 16:22 |
| gibi | point | 16:22 |
| sean-k-mooney | stephenfin: by the way are you ok with https://review.opendev.org/c/openstack/nova/+/773792 now? i did not remove the extension check since bauzas suggested i should add it when i discussed it previously | 16:37 |
| sean-k-mooney | it hit the cinder issue i was disucssing with lyarwood eairler so i have not rechecked it yet | 16:37 |
| stephenfin | Holy s***, OSC has a REPL?? | 16:38 |
| stephenfin | sean-k-mooney: I'd like to remove it if we can. I'm literally trying to test if it will work as we speak :) | 16:39 |
| sean-k-mooney | yes.... | 16:39 |
| sean-k-mooney | you didnt know that | 16:39 |
| stephenfin | but tbf I didn't read your replies yet. Looking | 16:39 |
| sean-k-mooney | stephenfin: we can remove it | 16:39 |
| sean-k-mooney | i tested it locally | 16:39 |
| sean-k-mooney | if you request a feild that does not exist it does not break anything | 16:39 |
| sean-k-mooney | also it passed tempest for the run where i did not have the extension test | 16:40 |
| stephenfin | hmm, so we can avoid a second API check with no consequences? | 16:40 |
| sean-k-mooney | so if bauzas is ok with me removing it and gibi is oke to review it again i can remove it | 16:40 |
| sean-k-mooney | stephenfin: its cached | 16:40 |
| sean-k-mooney | we will only ever check once right | 16:40 |
| sean-k-mooney | oh way not we refresh the cache | 16:41 |
| stephenfin | yup :) | 16:41 |
| sean-k-mooney | ya ok we can kill the check | 16:41 |
| stephenfin | though tbf, that refresh is behind a timer | 16:41 |
| sean-k-mooney | i just didnt want to revert it and get -2 again | 16:41 |
| stephenfin | so it's not as harmful as a I thought | 16:41 |
| stephenfin | *as I | 16:41 |
| *** jawad_axd has quit IRC | 16:42 | |
| sean-k-mooney | cool weell i need to recheck due to the cinder issue so i can respin instead but dont want to keep reving it for no reason | 16:42 |
| sean-k-mooney | well not no reason but going back an forth | 16:42 |
| *** jawad_axd has joined #openstack-nova | 16:42 | |
| stephenfin | Thanks fair | 16:43 |
| sean-k-mooney | stephenfin: did i do https://review.opendev.org/c/openstack/nova/+/773792/8/nova/network/neutron.py#2043 correctly by the way | 16:43 |
| stephenfin | bauzas is AFK at the moment | 16:43 |
| stephenfin | so maybe gibi can weigh in? | 16:43 |
| stephenfin | nah, what gibi suggested is what I was expecting | 16:43 |
| stephenfin | :param: foo foo foo | 16:44 |
| stephenfin | foo foo foo | 16:44 |
| *** jawad_axd has quit IRC | 16:44 | |
| stephenfin | well, :return: | 16:44 |
| stephenfin | you know what I mean :) | 16:44 |
| sean-k-mooney | ah so just 4 spaces instead of like 8 | 16:44 |
| stephenfin | yup | 16:44 |
| sean-k-mooney | instead of removing the indet entirely | 16:44 |
| stephenfin | please | 16:44 |
| gibi | I can rerewiew | 16:44 |
| gibi | if that was the question | 16:44 |
| sean-k-mooney | well would you prefer i keep the extention check or remove it | 16:45 |
| sean-k-mooney | since the call wont fail in either case and we dont check for the port resouces extension | 16:45 |
| sean-k-mooney | gibi: specificlaly this if https://review.opendev.org/c/openstack/nova/+/773792/8/nova/network/neutron.py#2050 | 16:46 |
| gibi | I'm OK to remove the check as neutron does not bark on nonexistent field and you access the field in the response conditionally | 16:47 |
| sean-k-mooney | ya i use get and default to None and handel that properly later | 16:48 |
| sean-k-mooney | in that case ill remove that and put back the correct indent for the return comment | 16:48 |
| sean-k-mooney | sorry for the churn i go do that now | 16:49 |
| sean-k-mooney | stephenfin: anything else for me to adress while i do that our are you happy other then that? | 16:49 |
| stephenfin | sean-k-mooney: I'd really like to see that Wallaby reference dropped from the reno too /o\ | 16:53 |
| stephenfin | I just left suggestions for other ways to store/get that metadata if you really rely on it | 16:54 |
| sean-k-mooney | i know how to get it from git i just hate seeing release notes without it | 16:54 |
| sean-k-mooney | but fine i can drop it | 16:54 |
| sean-k-mooney | i almost never ead release notes outside of git | 16:55 |
| stephenfin | Yeah, figured you would | 16:55 |
| stephenfin | you could do a comment | 16:55 |
| sean-k-mooney | hum i guess i could. whats the yaml comment syntax // | 16:56 |
| sean-k-mooney | it follows c right | 16:56 |
| stephenfin | as I noted, release notes should avoid version information in general since it doesn't make sense for backported fixes. That's not applicable here but in general, it's a good guide | 16:56 |
| sean-k-mooney | i think ill just drop it for now | 16:56 |
| stephenfin | '#' I think, but I'm not sure | 16:56 |
| sean-k-mooney | well this is a feature | 16:56 |
| sean-k-mooney | so it wont be backported but sure | 16:57 |
| stephenfin | yeah, like I said, not applicable here but a good guide | 16:57 |
| stephenfin | just never include version information in the release note and you never need to think about it | 16:57 |
| sean-k-mooney | im not sure i actully agree on the backport thing but i also dont wnat to spend time debating it FF week :) | 16:57 |
| *** takamatsu has quit IRC | 17:00 | |
| *** lucasagomes has quit IRC | 17:02 | |
| *** rpittau is now known as rpittau|afk | 17:05 | |
| *** khomesh24 has quit IRC | 17:09 | |
| lyarwood | has anyone deployed a local multinode devstack env recently? for some reason I can't curl keystone on the controller from the compute but I can ssh and ping between the hosts just fine | 17:17 |
| *** lpetrut has joined #openstack-nova | 17:18 | |
| sean-k-mooney | lyarwood: likely iptables | 17:18 |
| lyarwood | sean-k-mooney: disabled, as is firewalld | 17:18 |
| sean-k-mooney | do "sudo iptables -F; sudo iptables -X" | 17:18 |
| lyarwood | what the flying | 17:18 |
| lyarwood | >< | 17:18 |
| lyarwood | ggwp systemd | 17:19 |
| lyarwood | the service was dead but for what ever reason systemd didn't flush the rules | 17:19 |
| sean-k-mooney | devstack/neutorn? add some iptable rules directly | 17:19 |
| *** openstackgerrit has joined #openstack-nova | 17:19 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Use improved guest CPU config APIs https://review.opendev.org/c/openstack/nova/+/762330 | 17:19 |
| sean-k-mooney | i used to keep that command in my local.sh | 17:19 |
| sean-k-mooney | so that devstack would just do it every time i stacked | 17:20 |
| sean-k-mooney | i assume its working now? | 17:20 |
| lyarwood | yeah not a bad idea | 17:20 |
| lyarwood | yeah hopefully just stacking the compute again | 17:20 |
| sean-k-mooney | i never bother to check what exacatly add the iptables ruels athat break it but the fix was simple so i never felt the need | 17:21 |
| *** jangutter has joined #openstack-nova | 17:26 | |
| kashyap | lyarwood: gibi: The above patch (I resolved conflicts from a rebase) is chunky ... but solves a real live migration problem. In the past it was a blueprint and I wrote some docs in a "spec". But there's a case to be made for it to be an "advanced bug-fix" | 17:28 |
| kashyap | I now need to go out, perhaps I can bring this up on the upstream meeting for discussion | 17:28 |
| *** jangutte_ has quit IRC | 17:30 | |
| kashyap | [That needs to be split up for easier review] | 17:31 |
| gibi | kashyap: ack, I have to look at it tomorrow | 17:32 |
| *** ociuhandu has quit IRC | 17:41 | |
| *** tesseract has quit IRC | 17:47 | |
| *** ociuhandu has joined #openstack-nova | 17:54 | |
| *** ociuhandu has quit IRC | 17:54 | |
| *** ociuhandu has joined #openstack-nova | 17:55 | |
| *** amodi has quit IRC | 17:56 | |
| *** lpetrut has quit IRC | 18:00 | |
| *** derekh has quit IRC | 18:00 | |
| *** artom has joined #openstack-nova | 18:01 | |
| *** ociuhandu has quit IRC | 18:02 | |
| *** ociuhandu has joined #openstack-nova | 18:02 | |
| *** ociuhandu has quit IRC | 18:02 | |
| *** ociuhandu has joined #openstack-nova | 18:03 | |
| *** ociuhandu has quit IRC | 18:12 | |
| *** dtantsur is now known as dtantsur|afk | 18:13 | |
| *** ociuhandu has joined #openstack-nova | 18:16 | |
| *** gryf has quit IRC | 18:22 | |
| *** gryf has joined #openstack-nova | 18:25 | |
| *** ralonsoh has quit IRC | 18:28 | |
| *** hamalq has joined #openstack-nova | 18:30 | |
| sean-k-mooney | stephenfin: gibi i am doing some testing of vdpa for different api actions tl;dr i need to block shelve in addtion to livemigrate which was already planned. | 18:33 |
| sean-k-mooney | but the reason i need to block shelve is because the vdpa change shared the same code for shelve as normal neutron vf ports | 18:33 |
| sean-k-mooney | e.g. vnic_type=direct | 18:33 |
| sean-k-mooney | so it also hits https://bugzilla.redhat.com/show_bug.cgi?id=1767797 | 18:33 |
| openstack | bugzilla.redhat.com bug 1767797 in openstack-nova "When unshelving an SR-IOV instance, the binding profile isn't reclaimed or rescheduled, and this might cause PCI-PT conflicts" [High,Assigned] - Assigned to alifshit | 18:34 |
| sean-k-mooney | nova is correctly claiming the device in the pci tracker but not updating the neutron port | 18:34 |
| sean-k-mooney | before we regenerate the xml | 18:34 |
| sean-k-mooney | im wondering if i should block shelve for all sriov port types including vdpa,vf and pfs | 18:35 |
| stephenfin | I think that would be wise | 18:35 |
| sean-k-mooney | or even do it in two patches? | 18:35 |
| sean-k-mooney | 1 for the vdpa related things and one for the other sriov port type so that could be backported? | 18:35 |
| sean-k-mooney | if i only have one vm unshelve works fine but its just because it can use the same vf/vdpa device | 18:36 |
| sean-k-mooney | so if we fix that bug it will work | 18:36 |
| stephenfin | Two patches make sense | 18:37 |
| sean-k-mooney | i kind of feel this is like the numa live migration case. it sort of works but fundementally its broken, although this time i think we can backport a fix for unshleve with sriov | 18:37 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Follow up from bp/pci-socket-affinity series https://review.opendev.org/c/openstack/nova/+/779556 | 18:39 |
| lyarwood | sean-k-mooney: so it looks like the issue is with focal nodes still using tgtadm, just going to use your cloud to rebuild a multinode env if that's okay | 18:45 |
| sean-k-mooney | ya it shoudl be fine | 18:46 |
| sean-k-mooney | assuming i has free space go for it if not tell me an i can shelve something | 18:46 |
| lyarwood | sean-k-mooney: ah nvm, virt-builder supports focal now | 18:47 |
| sean-k-mooney | ah ya it has about 20-30G of hugepages free | 18:47 |
| sean-k-mooney | well there is space if you want to boot a coulpel of 8G vms you should be able ot spawn 3-4 | 18:47 |
| sean-k-mooney | lyarwood: im planning to redpeloy the cloud in a month or two and enable memory over subsciption. curretly everything uses hugepages but for how lightly used the vms are it proably makes snese to not do that | 18:48 |
| *** ociuhandu has quit IRC | 18:53 | |
| *** ociuhandu has joined #openstack-nova | 18:54 | |
| *** takamatsu has joined #openstack-nova | 18:55 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: fakelibvirt: make kB_mem default not laughable https://review.opendev.org/c/openstack/nova/+/779559 | 18:56 |
| *** k_mouza has quit IRC | 18:56 | |
| lyarwood | sean-k-mooney: kk | 18:58 |
| sean-k-mooney | artom: lyarwood since ye appare to be around could ye way in on https://review.opendev.org/c/openstack/nova/+/778347/4 just trying to get more input before i respin | 18:59 |
| sean-k-mooney | artom: lyarwood basicaly too questions should we use a diffferent name e.g.hw:mem_lock or hw:locked_memoy instead of hw:mlock | 18:59 |
| sean-k-mooney | and should that require hw:mem_page_size to be set | 19:00 |
| lyarwood | sean-k-mooney: I'll look once I've kicked off devstack | 19:00 |
| sean-k-mooney | thanks | 19:00 |
| lyarwood | sean-k-mooney: did you move your jump host again btw? | 19:00 |
| sean-k-mooney | no it should still be dyn.seanmooney.info | 19:01 |
| *** ociuhandu has quit IRC | 19:01 | |
| sean-k-mooney | openstack.seanmooney.info is loadbalanced by cloudflares cdn | 19:01 |
| lyarwood | yeah there we go, I still had openstack.seanmooney.info | 19:02 |
| lyarwood | .ssh/config updated | 19:02 |
| sean-k-mooney | dyn.seanmooney.info seams to be working for me but because of nat i cant really test that properly | 19:02 |
| sean-k-mooney | ah ok | 19:03 |
| artom | sean-k-mooney, I'm obviously missing context here, but do we need that patch at all right now? | 19:04 |
| sean-k-mooney | artom: yep | 19:04 |
| artom | As in, why not continue to add it implicitly when we detect a VDPA device? | 19:04 |
| artom | Sorry, not "continue to add", but just "add implicitly" | 19:05 |
| sean-k-mooney | libvirt does not do it right now | 19:05 |
| artom | Like you're saying we do for SEV and realtime | 19:05 |
| sean-k-mooney | and the way libvirt currently does it is a big problem for us | 19:05 |
| artom | Ah, so Nova doesn't do it, *libvirt* does it | 19:05 |
| sean-k-mooney | yep | 19:05 |
| artom | (For SEV and realtime) | 19:05 |
| sean-k-mooney | for sev and realtime nova does it | 19:05 |
| sean-k-mooney | but that is slightly differnt | 19:05 |
| artom | Tbh, I think having to set an extra spec that you have no choice for is bad UX, no? | 19:05 |
| artom | What's preventing Nova from detecting VDPA devices and adding the required XML? | 19:06 |
| sean-k-mooney | am kind of but we shoudl not be changing the memory we sare using based on a neutorn port | 19:06 |
| sean-k-mooney | artom: tl;dr our current memory tracking is really broken | 19:06 |
| artom | Ah, because not all VDPA devices require it? | 19:06 |
| sean-k-mooney | and vm that vfio(sriov port or pci passthough) vgpu or nvmeof device is being locked in memory by libvirt | 19:07 |
| sean-k-mooney | meaning oversubcript does not work | 19:07 |
| sean-k-mooney | artom: am the simulator does not require it | 19:08 |
| sean-k-mooney | artom: its not clear if dpdk based vdpa devices would | 19:08 |
| *** andrewbonney has quit IRC | 19:09 | |
| *** hamalq has quit IRC | 19:09 | |
| sean-k-mooney | so i dont know maybe we could auto add it | 19:09 |
| *** hamalq has joined #openstack-nova | 19:09 | |
| artom | You're saying "let's not change the memory based on Neutron port", except libvirt kinda does that already for vGPU, to use your own example | 19:11 |
| sean-k-mooney | my conern right now is to track locked memory proertly we might need severly restrict what type of vms can use neutron sriov port or passthough of pci or vgpus deivces | 19:11 |
| sean-k-mooney | artom: well it does it for neutron VF ports | 19:11 |
| sean-k-mooney | so we are already inadvertely doing it based on ports | 19:11 |
| artom | Yeah | 19:12 |
| sean-k-mooney | well libvirt is | 19:12 |
| artom | UX-wise, if something needs doing regardless, we should be asking the user to do it for us | 19:12 |
| artom | The fact that our memory tracking is broken is a tangential problem :P | 19:12 |
| sean-k-mooney | well im concerned we might need to block vms that dont use hw:mem_page_size form using sriov port in the future to solve this issue | 19:13 |
| sean-k-mooney | artom: ya it is but i was trying not to boil the ocean i guess doing it based on the port type make sense | 19:14 |
| sean-k-mooney | i was orginaly hoping this was just a bug that would go away | 19:14 |
| artom | I get it, the deadline is looming and you're rushing | 19:14 |
| sean-k-mooney | partly that | 19:14 |
| sean-k-mooney | and partly in theory mellonx/nvidia coudl fix this is there driver suppport page faults | 19:15 |
| sean-k-mooney | they dont right now and may never but if they did it would not need to be locked | 19:15 |
| sean-k-mooney | the vdpa sim module doe snot need locking but it likely either support page fause or just does not use dma memory | 19:15 |
| artom | My paranoid conservative opinion is that this needs to be figured out in a spec next cycle ;) | 19:16 |
| artom | Instead of panic-merging stuff ;) | 19:17 |
| sean-k-mooney | well we discussed locked memory before for sev and realtime but did not have a usecase for it. | 19:17 |
| sean-k-mooney | i could also just mark the guest as realtime today | 19:17 |
| sean-k-mooney | that requirement would go away when libvirt start treating vdpa like a vf and upping the mlock limit | 19:18 |
| sean-k-mooney | maybe that a better idea for now. | 19:18 |
| sean-k-mooney | so no new extra spec but require a newer libvirt or a realtime guest. | 19:19 |
| sean-k-mooney | i agree though i would like to not rush this | 19:19 |
| *** ociuhandu has joined #openstack-nova | 19:27 | |
| sean-k-mooney | artom: for what its worth to use ovs-dpdk and vhost user you have to set hw:mem_page_size=large | 19:28 |
| sean-k-mooney | other wise it wont work the same way that vdpa breaks today | 19:28 |
| sean-k-mooney | if you dont add locked | 19:28 |
| artom | Yeah, I agree there's precedent | 19:28 |
| artom | (In terms for breaking unless the user does something they strictly-speaking should not have to do) | 19:29 |
| artom | But... doesn't mean we shouldn't strive to improve on that :) | 19:29 |
| sean-k-mooney | well again we did not wnat reqouce usage to change basked on port type | 19:30 |
| sean-k-mooney | that is why you were required to enable hugepages in the flavor or image | 19:30 |
| sean-k-mooney | there are othere issue this create for attach too | 19:30 |
| artom | Well you clearly shouldn't be allowed to attach ports that require locked memory to a running instance :) | 19:31 |
| artom | (Unless it already has locked memory) | 19:31 |
| sean-k-mooney | yep | 19:32 |
| artom | Which we would need to track, and it could come from not only the extra spec, but also other source that caused libvirt to do it for us, etc etc | 19:32 |
| sean-k-mooney | but not just running | 19:32 |
| artom | Hence: spec discussion :) | 19:32 |
| sean-k-mooney | it would be incorrect to change it on hard reboot | 19:32 |
| sean-k-mooney | or iff it was off | 19:32 |
| sean-k-mooney | e.g. you cant add to any instance already on the host | 19:33 |
| sean-k-mooney | as it change the memory usage of the guest | 19:33 |
| *** k_mouza has joined #openstack-nova | 19:33 | |
| sean-k-mooney | same way addign a vhost-user port shoudl not make the guest suddenly use hugepages | 19:33 |
| *** ociuhandu has quit IRC | 19:37 | |
| *** k_mouza has quit IRC | 19:37 | |
| *** amodi has joined #openstack-nova | 19:44 | |
| *** ociuhandu has joined #openstack-nova | 19:49 | |
| *** ociuhandu has quit IRC | 19:52 | |
| *** ociuhandu has joined #openstack-nova | 19:52 | |
| openstackgerrit | sean mooney proposed openstack/nova master: Support per port numa policies with SR-IOV https://review.opendev.org/c/openstack/nova/+/773792 | 19:57 |
| *** lbragstad has quit IRC | 20:01 | |
| lyarwood | sean-k-mooney: sorry had to go afk, so devstack is still running somehow. Is your host pretty loaded at the moment? | 20:05 |
| sean-k-mooney | no load is at 4.38 on the host | 20:06 |
| sean-k-mooney | it has 24cores/48threads | 20:07 |
| lyarwood | weird | 20:07 |
| sean-k-mooney | it might be ipv6 being slow | 20:07 |
| lyarwood | I'm using the medium flavor with 12 vcpus | 20:07 |
| sean-k-mooney | i dont have native ipv6 and the tunnel is sometimes slow | 20:07 |
| lyarwood | I'm not using the async stuff but still, it's been running for over an hour now | 20:07 |
| lyarwood | kk | 20:08 |
| sean-k-mooney | ya that weird what does the load look like in the vm | 20:08 |
| sean-k-mooney | io wait is also not that high in the host %0.90 | 20:09 |
| lyarwood | kk let me try again with DEVSTACK_PARALLEL=true | 20:10 |
| sean-k-mooney | it should only take about 20 mings without it | 20:11 |
| lyarwood | takes about 6 on a local f32 vm | 20:12 |
| lyarwood | kk, it's looking quicker tbh | 20:12 |
| *** lbragstad has joined #openstack-nova | 20:15 | |
| sean-k-mooney | ya something is wrong with my ipv6 routing http://paste.openstack.org/show/803402/ | 20:19 |
| sean-k-mooney | dns is working but i cant ping | 20:19 |
| sean-k-mooney | thats form the server that is hosting the vm | 20:19 |
| *** jamesden_ is now known as jamesdenton | 20:20 | |
| *** jamesdenton has quit IRC | 20:21 | |
| *** jamesden_ has joined #openstack-nova | 20:21 | |
| *** jamesden_ is now known as jamesdenton | 20:22 | |
| *** jamesdenton has quit IRC | 20:29 | |
| openstackgerrit | Merged openstack/nova master: pci manager: replace node_id parameter with compute_node https://review.opendev.org/c/openstack/nova/+/778747 | 20:30 |
| openstackgerrit | Merged openstack/nova master: apidb: Compact Queens database migrations https://review.opendev.org/c/openstack/nova/+/759404 | 20:30 |
| sean-k-mooney | lyarwood: problems on my routeer the tunnel is not working properly. for now if you do | 20:31 |
| sean-k-mooney | sysctl -w net.ipv6.conf.all.disable_ipv6=1 | 20:31 |
| sean-k-mooney | sysctl -w net.ipv6.conf.default.disable_ipv6=1 | 20:31 |
| sean-k-mooney | it will work around the issue | 20:31 |
| sean-k-mooney | with sudo of course | 20:31 |
| *** irclogbot_0 has quit IRC | 20:31 | |
| sean-k-mooney | it shoudl be falling back to ipv4 anyway but that can slow down installs | 20:31 |
| sean-k-mooney | this is one of the reason im going to be reinstallin my cloud in a month or too | 20:32 |
| openstackgerrit | Merged openstack/nova master: pci: track host NUMA topology in stats https://review.opendev.org/c/openstack/nova/+/774149 | 20:32 |
| sean-k-mooney | ipv6 is nice but somethiem it has issue so untill i have it nativly im going to remove it | 20:32 |
| sean-k-mooney | am what? ^ | 20:32 |
| *** irclogbot_0 has joined #openstack-nova | 20:33 | |
| *** ociuhandu has quit IRC | 20:33 | |
| sean-k-mooney | stephenfin: gibi the numa node is already in the pci_stats in the compute node table | 20:34 |
| sean-k-mooney | as is the host numa toplogy blob | 20:35 |
| sean-k-mooney | {"product_id": "101e", "vendor_id": "15b3", "numa_node": 0, "tags": {"dev_type": "vdpa", "physical_network": null, "parent_ifname": "enp6s0f0_0"}, "count": 3} | 20:36 |
| *** jamesdenton has joined #openstack-nova | 20:38 | |
| sean-k-mooney | artom: i really wish you asked me to review that | 20:39 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: fakelibvirt: make kB_mem default not laughable https://review.opendev.org/c/openstack/nova/+/779559 | 20:40 |
| sean-k-mooney | these were class method because we did not want to store this data in an object | 20:40 |
| artom | sean-k-mooney, the NUMA node yes, but not the socket | 20:40 |
| sean-k-mooney | right but the socket you were goint to store/lookup seperately | 20:41 |
| artom | sean-k-mooney, I save the host numa_topology in the pci stats object | 20:41 |
| artom | (I forget what it's called exactly) | 20:41 |
| sean-k-mooney | im reviewing https://review.opendev.org/c/openstack/nova/+/774149 now to fiture out what you cahgned but this is going to conflcit with most of my patches | 20:41 |
| *** whoami-rajat has quit IRC | 20:42 | |
| openstackgerrit | Merged openstack/nova master: conf: Clean up docs for scheduler options https://review.opendev.org/c/openstack/nova/+/773639 | 20:42 |
| artom | Storing it (the numa_topology) in the object was cleaner than passing it through | 20:42 |
| artom | Only the socket policy filter() method needs it, but it'd have been passed through like 42 other methods to get there | 20:43 |
| sean-k-mooney | but those function were ment to be pur fucntion of there input | 20:43 |
| artom | They still are | 20:43 |
| artom | They're just in the object now | 20:43 |
| sean-k-mooney | but do they use any data via self | 20:44 |
| sean-k-mooney | if so they are not | 20:44 |
| artom | Nope | 20:44 |
| artom | Only the new filter for the socket policy | 20:44 |
| sean-k-mooney | then they can stay class methods | 20:44 |
| sean-k-mooney | you can call class methods via self | 20:44 |
| artom | No, because they *call* the new socket filter method | 20:44 |
| artom | Which needs to be on self, not cls | 20:44 |
| artom | Because of the aforementioned requirement for numa_topology | 20:44 |
| artom | You can call cls from self, but not the other way around | 20:45 |
| sean-k-mooney | sure but you could pass in the numa object | 20:46 |
| artom | Yeah, it's what I was saying | 20:46 |
| sean-k-mooney | where are you usin ghtis the next patch | 20:46 |
| artom | Yeah, next patch | 20:46 |
| artom | It's what I was saying - storing self.numa_topology was much cleaner than passing it rhough | 20:47 |
| sean-k-mooney | i would not have done what you have in the one that just merged | 20:47 |
| artom | *through | 20:47 |
| sean-k-mooney | ya but it break the design of the class | 20:47 |
| sean-k-mooney | and that make me uncofrotable | 20:47 |
| sean-k-mooney | we intentionally did not do what you are not doing | 20:47 |
| sean-k-mooney | so ya you are now using self here https://review.opendev.org/c/openstack/nova/+/772779/17/nova/pci/stats.py#336 | 20:48 |
| artom | I figured there was a reason, on the other hand, there was also a thing where node_id was optional which looks like it was only for testing | 20:50 |
| artom | So it was hard to tell what was legit reason and what was programmer laziness ;) | 20:50 |
| melwitt | sean-k-mooney: curious how you get ~5 devstack times? do you disable certain services or something? | 20:50 |
| sean-k-mooney | i get time of about 15 mins | 20:51 |
| sean-k-mooney | artom: the numa node is optional | 20:51 |
| *** jamesdenton has quit IRC | 20:51 | |
| sean-k-mooney | not all devices have one | 20:51 |
| artom | sean-k-mooney, not the numa node, the compute node_id | 20:51 |
| sean-k-mooney | oh thats required | 20:52 |
| artom | It didn't use to be | 20:52 |
| sean-k-mooney | where is that optional | 20:52 |
| melwitt | hm ok. when I do it with DEVSTACK_PARALLEL=1 it takes 25-30 minutes but that is with default things enabled, nearly empty local.conf | 20:52 |
| melwitt | I was wondering how people are getting these really fast times | 20:52 |
| artom | sean-k-mooney, https://review.opendev.org/c/openstack/nova/+/778747/2/nova/pci/manager.py | 20:52 |
| sean-k-mooney | melwitt: it has to be DEVSTACK_PARALLEL=True | 20:52 |
| sean-k-mooney | unless they fixed that 1 will not work | 20:53 |
| melwitt | sean-k-mooney: I used it, it gave me the async time report at the end | 20:53 |
| melwitt | maybe I used True, I don't know for sure | 20:53 |
| sean-k-mooney | this is also my default contoler http://paste.openstack.org/show/803405/ | 20:53 |
| sean-k-mooney | i have some service disabled yes | 20:53 |
| melwitt | thanks | 20:53 |
| sean-k-mooney | you could disable tempest and horizon form that list if you anted | 20:54 |
| sean-k-mooney | or cinder i guess | 20:54 |
| sean-k-mooney | but that a faily simple compute cloud | 20:54 |
| sean-k-mooney | baisclaly no heat or swift form the default set | 20:55 |
| sean-k-mooney | artom: i think that was needed | 20:56 |
| artom | sean-k-mooney, only for testing as far as I could find | 20:56 |
| sean-k-mooney | artom: we initalise the pci tracker before the comptue service is registered | 20:56 |
| artom | That's the only place where it was ever not passed in | 20:56 |
| sean-k-mooney | artom: no i think its need for a fresh install | 20:56 |
| artom | sean-k-mooney, wouldn't that then explode in testing? | 20:57 |
| sean-k-mooney | its not that is passed i think its set | 20:57 |
| artom | sean-k-mooney, well, you're in luck, the top-most patched failed the gate :) | 20:58 |
| artom | So any feedback you have it's now or never | 20:59 |
| melwitt | sean-k-mooney: thanks | 20:59 |
| sean-k-mooney | artom: https://github.com/openstack/nova/blame/421c52d9d341b07d850c21e0e702a008a8e1d3b7/nova/compute/resource_tracker.py#L493-L495 | 21:01 |
| sean-k-mooney | artom: when that runs it possible we have not registered teh compute node yet | 21:02 |
| sean-k-mooney | if i remeber correctly | 21:02 |
| artom | sean-k-mooney, I think that's changed since then... | 21:03 |
| openstackgerrit | Merged openstack/nova master: Differentiate between InstanceNotFound and ConstraintNotMet https://review.opendev.org/c/openstack/nova/+/775309 | 21:03 |
| openstackgerrit | Merged openstack/nova master: Add functional test for bug 1837995 https://review.opendev.org/c/openstack/nova/+/775449 | 21:04 |
| openstack | bug 1837995 in OpenStack Compute (nova) ""Unexpected API Error" when use "openstack usage show" command" [Undecided,In progress] https://launchpad.net/bugs/1837995 - Assigned to melanie witt (melwitt) | 21:04 |
| artom | Now it's only called from _setup_pci_tracker, which has the compute_node object available to it | 21:04 |
| artom | sean-k-mooney, anyways, I've -1'ed the top-most patch that hasn't merged yet | 21:05 |
| sean-k-mooney | yes but i think that has a similar beahvior | 21:05 |
| artom | Take your time to review it | 21:05 |
| artom | We can revisit this tomorrow | 21:05 |
| sean-k-mooney | i was tryign to find it but you updated it and its hard to fid the old version :) | 21:05 |
| artom | I *only* changed node_id to compute_node | 21:06 |
| sean-k-mooney | actully i can just go in the history i gues | 21:06 |
| artom | Nothing else | 21:06 |
| sean-k-mooney | this is the placves where its called that i was worred about | 21:08 |
| sean-k-mooney | https://github.com/openstack/nova/blob/1273c5ee0b18974d9837e9221fc9270429d428bf/nova/compute/resource_tracker.py#L727-L761 | 21:08 |
| sean-k-mooney | well i think those case its fine | 21:08 |
| sean-k-mooney | assuming create has the sideffect of it having the id | 21:08 |
| sean-k-mooney | artom: why did you pass the compute node object in | 21:10 |
| sean-k-mooney | instead of the id | 21:10 |
| artom | Don't need to look it up then | 21:10 |
| artom | Saves a DB query - gibi suggested it | 21:10 |
| sean-k-mooney | id does not | 21:11 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/778747/2/nova/pci/manager.py#64 | 21:11 |
| sean-k-mooney | we were passing in the compute node id before | 21:11 |
| sean-k-mooney | now we dont save it and just extract the id | 21:11 |
| artom | We don't save what? | 21:12 |
| sean-k-mooney | the compute node object in this object | 21:12 |
| *** gyee has joined #openstack-nova | 21:16 | |
| artom | We don't need to | 21:17 |
| sean-k-mooney | you split out the chage that refactored the interface form the cahgne that used the compute node object | 21:17 |
| artom | We save the numa_topology in PciDeviceStats | 21:17 |
| artom | https://review.opendev.org/c/openstack/nova/+/774149/12/nova/pci/stats.py | 21:17 |
| sean-k-mooney | that should not have been done imo | 21:18 |
| artom | Debate with stephenfin on that, his idea :) | 21:18 |
| artom | But to not throw him under the bus too much, I agree with it | 21:18 |
| artom | Makes the changes cleaner | 21:18 |
| artom | One to always pass compute_node instead of the optional node_id=None | 21:18 |
| sean-k-mooney | not really | 21:18 |
| sean-k-mooney | it was not optional | 21:18 |
| artom | And another to pull the numa_topology from that and pass it to PciDeviceStats | 21:19 |
| sean-k-mooney | it was a key word argument | 21:19 |
| sean-k-mooney | but the comptue node was required in all the production code | 21:19 |
| sean-k-mooney | *id | 21:19 |
| artom | How is https://review.opendev.org/c/openstack/nova/+/778747/2/nova/pci/manager.py#54 not optional? | 21:19 |
| artom | It literally says node_id=None | 21:19 |
| sean-k-mooney | right but all uses of it out side fo test always set it | 21:19 |
| artom | Yeah, my point exactly | 21:20 |
| artom | Codify that it's always expected | 21:20 |
| artom | And because we'll need the full compute_node later on, replace node_id with the full object | 21:20 |
| sean-k-mooney | sure but not in a patch seperate form the new usage or the full object | 21:20 |
| sean-k-mooney | artom: sorry this just annoys me because you were chanign someint i did not think you were going to change. it conflicts with my changes and it break my mental model of how the pci trakcer works | 21:22 |
| artom | sean-k-mooney, that changes aren't that dramatic... | 21:23 |
| sean-k-mooney | the main one was that it never store state in the pci tracker object directly | 21:24 |
| sean-k-mooney | the numa toplogy object will have to be keep consitent now | 21:25 |
| sean-k-mooney | ok we do sotre state but differently | 21:26 |
| artom | We're talking about the *host* numa_topology | 21:26 |
| sean-k-mooney | yes | 21:26 |
| artom | When is that ever going to change... | 21:27 |
| sean-k-mooney | within th elife time of the agent i guess it not going to | 21:27 |
| sean-k-mooney | i mean memoy and cpu hotplug are thigns and you can reconfigure hyperthreading on the fly or hugepagers for that matter | 21:28 |
| sean-k-mooney | hugepage is actully the most likely ot change at runtime | 21:28 |
| sean-k-mooney | but to have that picked up you need to restart libvirtd | 21:28 |
| *** k_mouza has joined #openstack-nova | 21:29 | |
| sean-k-mooney | actully se also store the currently pinne cpu in the host numa toplogy blob | 21:29 |
| sean-k-mooney | so its update every time we boot new vms | 21:30 |
| *** k_mouza has quit IRC | 21:33 | |
| artom | sean-k-mooney, the PCI tracker never uses that information though | 21:36 |
| artom | I could add a comment to warn future programmers | 21:36 |
| artom | We just need the socket/node mapping, and that's effectively constant | 21:37 |
| lyarwood | melwitt / sean-k-mooney ; so in my defence, it's late and I shouldn't be working but when I said ~5mins earlier what I actually wanted to say was ~500 seconds. http://paste.openstack.org/show/803406/ | 21:40 |
| sean-k-mooney | lyarwood: ay right that about right with paralle | 21:41 |
| sean-k-mooney | i think i missed where you said ti too ~500 though | 21:42 |
| sean-k-mooney | sub ten mintues is doable if you have good netowrking, io and a fast cpu | 21:44 |
| lyarwood | yup I'm just running a 4 vCPU, 16GB, 50GB RAW disk VM on my p1 gen2 with a 1Gbps connection | 21:46 |
| melwitt | lyarwood: heh, sorry, it wasn't only you, I had seen other mentions of 5-6 min about it before and your mention made me think to ask what am I doing wrong to not get this result 😆 | 21:52 |
| sean-k-mooney | melwitt: yep dansmith was around the 5-8 minute mark | 21:52 |
| sean-k-mooney | that i think was on baremetal | 21:53 |
| sean-k-mooney | rather then nested virt but its doable | 21:53 |
| dansmith | talking about devstack time? | 21:54 |
| sean-k-mooney | yep | 21:54 |
| sean-k-mooney | i think you were geting about 430 ish second if im not mistaken | 21:55 |
| dansmith | yeah, I can do about 5mins with a less-than-full devstack config | 21:55 |
| dansmith | with OCaaS plus parallel I can get 3xxx yeah | 21:55 |
| dansmith | er, 3xx | 21:56 |
| melwitt | what are the main things you disable? | 21:56 |
| dansmith | disable_service c-bak etcd3 c-api c-vol c-sch swift horizon dstat | 21:56 |
| melwitt | thanks | 21:56 |
| dansmith | tempest if I don't need it | 21:56 |
| sean-k-mooney | swift i think is kind of slow to set up | 21:56 |
| sean-k-mooney | dstat should not make much of a differnce horizon take a while to complie and compress the static pages | 21:57 |
| dansmith | well, not all of the systemctl commands are super fast | 21:57 |
| dansmith | sometimes depending on what is running, daemon-reload can take a couple seconds, and start if it waits for the first child, etc | 21:58 |
| sean-k-mooney | ya i notice that more on unstack then anything esle | 22:00 |
| sean-k-mooney | some service take a long time to stop randomly | 22:00 |
| dansmith | yeah, that's another good reason though, | 22:00 |
| dansmith | more shtuff to unstack makes the process slower when you're iterating | 22:01 |
| sean-k-mooney | yep although i normlaly see how long i can go with just doing sudo systemctl restart devstack@n-* | 22:02 |
| sean-k-mooney | if im hacking on stuff i generally dont restack unless i have too | 22:02 |
| dansmith | well, when you're working on stuff that crosses multiple projects, as I have been lately, unstack/stack time is important | 22:04 |
| dansmith | especially if one of those _is_ devstack :) | 22:04 |
| sean-k-mooney | yep i used to restack multiple times a day | 22:04 |
| sean-k-mooney | now i just have different envs for different tings | 22:05 |
| sean-k-mooney | so i restack less | 22:05 |
| sean-k-mooney | still important to be quick | 22:05 |
| dansmith | yeah just depends on what you're doing | 22:06 |
| dansmith | obviously hacking on a single project, service restart is by far the most efficient :) | 22:06 |
| sean-k-mooney | basically if i dont need db change i try to jsut checkout the patch i need and restart it | 22:07 |
| sean-k-mooney | if i get error i restack | 22:07 |
| sean-k-mooney | it works more times then it proably should | 22:07 |
| dansmith | it *should* work for most things, so .. I'd be concerned if it didn't ;) | 22:09 |
| sean-k-mooney | i sometime get bitten by the compute service verion if i change to a different series that i want to test | 22:10 |
| sean-k-mooney | i know i can fix that but i never do | 22:10 |
| sean-k-mooney | i just restack | 22:10 |
| *** ociuhandu has joined #openstack-nova | 22:21 | |
| *** rcernin has joined #openstack-nova | 22:24 | |
| *** spatel_ has quit IRC | 22:27 | |
| *** rcernin has quit IRC | 22:30 | |
| *** rcernin has joined #openstack-nova | 22:30 | |
| *** ociuhandu has quit IRC | 22:38 | |
| *** grandchild has joined #openstack-nova | 22:56 | |
| *** tkajinam has joined #openstack-nova | 23:00 | |
| *** adriant has quit IRC | 23:11 | |
| *** adriant has joined #openstack-nova | 23:12 | |
| *** mlavalle has joined #openstack-nova | 23:22 | |
| *** grandchild has quit IRC | 23:53 | |
| *** grandchild has joined #openstack-nova | 23:54 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!