| opendevreview | Sylvain Bauza proposed openstack/nova-specs master: Proposes mdev live-migration support in libvirt https://review.opendev.org/c/openstack/nova-specs/+/900636 | 08:37 |
|---|---|---|
| opendevreview | sean mooney proposed openstack/nova master: add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | 12:38 |
| opendevreview | sean mooney proposed openstack/nova master: add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | 13:24 |
| opendevreview | sean mooney proposed openstack/nova master: add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844 | 13:24 |
| opendevreview | sean mooney proposed openstack/nova master: add healthcheck tracker to nova context https://review.opendev.org/c/openstack/nova/+/829468 | 13:24 |
| opendevreview | sean mooney proposed openstack/nova master: add healthcheck utils and constants https://review.opendev.org/c/openstack/nova/+/829469 | 13:39 |
| opendevreview | sean mooney proposed openstack/nova master: add healthcheck endpoint to proxy commands https://review.opendev.org/c/openstack/nova/+/830703 | 13:39 |
| bauzas | sean-k-mooney: gibi: fwiw, I'm about to start writing the series for the mdev live-migration RFE | 14:02 |
| gibi | bauzas: good luck! | 14:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: Allow live migrate paused instance when post copy is enabled https://review.opendev.org/c/openstack/nova/+/444517 | 14:11 |
| artom | melwitt, sean-k-mooney, gibi, btw, I updated https://review.opendev.org/c/openstack/nova/+/883682/ to hopefully reflect what was agreed on in the review comments | 14:38 |
| artom | So now there's the notifications best effort patch on top: https://review.opendev.org/c/openstack/nova/+/903807/3 | 14:38 |
| gibi | artom: thanks. Both looks good to me | 15:07 |
| sean-k-mooney | artom: ack ill look at it shortly | 15:08 |
| artom | Thank _you_! | 15:09 |
| gibi | sean-k-mooney: if you have a moment later, then I verified https://review.opendev.org/c/openstack/nova/+/444517 and it solves the live-migration of paused instance when post-copy is configured in nova.conf. | 16:29 |
| gibi | artom: ^^ \o/ | 16:29 |
| kashyap | gibi: Also the "double live migration of a paused instance" thing - did you see that bz too? | 16:33 |
| kashyap | Although, why would one want to run a double LM, though | 16:33 |
| artom | gibi, ~~o~~ | 16:34 |
| kashyap | "If a VM is in paused, and it live-migrated twice, it is lost" | 16:34 |
| kashyap | https://bugs.launchpad.net/nova/+bug/1947725 | 16:34 |
| kashyap | "Lost"? | 16:34 |
| artom | Here instance instance, here boy! | 16:35 |
| gibi | kashyap: I can try | 16:46 |
| gibi | kashyap: nope, that but is still reproducible https://paste.opendev.org/show/bswQVIMJddVkrUtC1Pk0/ | 16:55 |
| gibi | kashyap: but it make sens as the related qemu bug is also open https://gitlab.com/qemu-project/qemu/-/issues/686 | 16:56 |
| kashyap | gibi: Ohh, I missed the attached QEMU bug | 16:58 |
| artom | sean-k-mooney, so, tracing our snapshot upload call, we actually call it with a file handler for image_file/data | 17:06 |
| artom | https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L3182 | 17:07 |
| artom | https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L3176 | 17:07 |
| artom | The thing is, on the sender side (Nova) passing a file handler to Python requests results in a streaming upload: https://docs.python-requests.org/en/latest/user/advanced.html#streaming-uploads | 17:09 |
| artom | For chunks we have to use a generator or iterator: https://docs.python-requests.org/en/latest/user/advanced.html#chunk-encoded-requests | 17:09 |
| sean-k-mooney | ack so we likely need a [glance]uplod_type=streaming|chunked config option | 17:10 |
| sean-k-mooney | default to streamign since that is ths current behavior and allow opting into chunked | 17:11 |
| sean-k-mooney | and we can consider changing the default in the future | 17:11 |
| sean-k-mooney | melwitt: bauzas gibi can ye look at https://review.opendev.org/c/openstack/nova/+/903530/2 and the patch below to unblock mypy in teh requiremetns job for stephenfin | 17:20 |
| sean-k-mooney | seperatly this https://review.opendev.org/c/openstack/nova/+/897218/2 and the 3 patches below it will finish the codespell and sphinx-lint series | 17:22 |
| sean-k-mooney | if we can get the mypy codespell and sphinx-lint serise merged this week it would be great | 17:22 |
| sean-k-mooney | gibi: how did you test https://review.opendev.org/c/openstack/nova/+/444517 in the openstack-k8s-operators CI | 17:27 |
| sean-k-mooney | did you submit a pr? | 17:27 |
| sean-k-mooney | i know they were working on makign the conent provide supprot buildign with upstream patches | 17:27 |
| sean-k-mooney | btu i didnt think that was done yet | 17:27 |
| melwitt | sean-k-mooney: sure, I can look (if others don't get to it first) | 17:28 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: POC: attempting glance chunked uploads https://review.opendev.org/c/openstack/nova/+/903611 | 17:29 |
| gibi | sean-k-mooney: sorry I was missleading I applied the patch locally | 17:42 |
| sean-k-mooney | oh ok | 17:42 |
| sean-k-mooney | that is something we will be able to do eventually | 17:42 |
| sean-k-mooney | with a depends on | 17:42 |
| sean-k-mooney | but not for a few months yet | 17:42 |
| sean-k-mooney | its on the todolist and somethign ill be supprotign with rhex-ci too eventually | 17:43 |
| melwitt | sean-k-mooney: hrm.. I notice that artom's patch is going to fail CI for a guest kernel panic 🫤 and it looks to be using the split image | 18:14 |
| artom | :( | 18:15 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: POC: attempting glance chunked uploads https://review.opendev.org/c/openstack/nova/+/903611 | 18:17 |
| sean-k-mooney | melwitt: i guess we can reviwe the logs when it finishes | 18:18 |
| melwitt | and it seems like it's often (always?) this volume backed server resize test(s). so I wonder if there's something about it that's different than everything else. I'll see if I can find anything | 18:18 |
| sean-k-mooney | is that in nova-next where we are not using split image or one of the other jobs | 18:18 |
| melwitt | no it's in tempest-integrated-compute-rbac-old-defaultsf | 18:19 |
| sean-k-mooney | well the volume test obviouly has some pci hotplug events | 18:19 |
| sean-k-mooney | so maybe we can mitigate that by adding hw_disk_bus=scisi | 18:19 |
| melwitt | logs are at https://zuul.opendev.org/t/openstack/build/fcc86981ee0e488f9f8b05a5969bfacb | 18:19 |
| melwitt | hm ok | 18:19 |
| sean-k-mooney | i.e. chagne from virtio-blk to virtio-scisi as that will not be a pci hotplu and instead be a scsi attach/detach | 18:20 |
| melwitt | gotcha | 18:20 |
| sean-k-mooney | the sttack trace looks like its in the page fault handeler | 18:21 |
| sean-k-mooney | so this looks like the same failure we saw before | 18:21 |
| sean-k-mooney | i wonder if this is bfv | 18:22 |
| sean-k-mooney | and if we replace the image in that case | 18:22 |
| melwitt | it is bfv | 18:23 |
| sean-k-mooney | ok so we shoudl double check that tempest used the correct cirros image in that case | 18:23 |
| sean-k-mooney | its calling into the share create_server https://github.com/openstack/tempest/blob/ab3686d28d2728001e3bd2fd543575087bf00137/tempest/api/compute/servers/test_server_actions.py#L491 | 18:25 |
| sean-k-mooney | which end up using https://github.com/openstack/tempest/blob/ab3686d28d2728001e3bd2fd543575087bf00137/tempest/common/compute.py#L204 | 18:25 |
| sean-k-mooney | image_id = CONF.compute.image_ref | 18:26 |
| sean-k-mooney | melwitt: have we confirued that to be the split image? | 18:26 |
| sean-k-mooney | its https://zuul.opendev.org/t/openstack/build/fcc86981ee0e488f9f8b05a5969bfacb/log/controller/logs/tempest_conf.txt#24 | 18:26 |
| sean-k-mooney | 86a0308f-80a0-4582-8a64-f2b54e487232 | 18:27 |
| sean-k-mooney | i guess we coudl check the devstack log and conrim what image that is | 18:27 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/fcc86981ee0e488f9f8b05a5969bfacb/log/controller/logs/devstacklog.txt#15755-15773 | 18:27 |
| sean-k-mooney | ok so that is the uec image | 18:28 |
| sean-k-mooney | so ya my best suggestion is lets try adding hw_disk_bus=scsi in devstack adn see if that helps or disabling https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.libvirt_disable_apic | 18:29 |
| melwitt | ok, I can try that | 18:31 |
| sean-k-mooney | thinking about this its failign in the really early boot | 18:32 |
| sean-k-mooney | so im not sure that will help | 18:32 |
| sean-k-mooney | the other thing we coudl tyr is adding swap to the tempest flavors to see if that help with the OOM issues | 18:32 |
| melwitt | ah right | 18:34 |
| artom | sean-k-mooney, bleah, glanceclient chunks it for us: https://opendev.org/openstack/python-glanceclient/src/branch/master/glanceclient/common/http.py#L110 | 19:17 |
| sean-k-mooney | ack streaming might still be good to enable | 19:17 |
| artom | Which... I guess I learned stuff today, but then I'm no closer to understanding why g-api OOM'ed :P | 19:17 |
| sean-k-mooney | ya not sure | 19:18 |
| opendevreview | Merged openstack/nova master: Allow live migrate paused instance when post copy is enabled https://review.opendev.org/c/openstack/nova/+/444517 | 19:59 |
| opendevreview | Merged openstack/nova master: [codespell] fix final typos and enable ci https://review.opendev.org/c/openstack/nova/+/897214 | 21:20 |
| opendevreview | Merged openstack/nova master: Bump hacking version https://review.opendev.org/c/openstack/nova/+/903529 | 21:20 |
| JayF | sean-k-mooney: https://blueprints.launchpad.net/nova/+spec/ironic-guest-metadata is updated; if you have a preliminary review before tomorrow's meeting I'm happy to do a round trip of feedback ahead of it :) | 22:22 |
| opendevreview | Jay Faulkner proposed openstack/nova master: [ironic] Partition & use cache for list_instance* https://review.opendev.org/c/openstack/nova/+/900831 | 23:05 |
| opendevreview | Jay Faulkner proposed openstack/nova master: Limit nodes by ironic shard key https://review.opendev.org/c/openstack/nova/+/903915 | 23:05 |
| opendevreview | Jay Faulkner proposed openstack/nova master: Add nova-manage ironic-compute-node-move https://review.opendev.org/c/openstack/nova/+/903916 | 23:05 |
| opendevreview | Jay Faulkner proposed openstack/nova master: Make compute node rebalance safter https://review.opendev.org/c/openstack/nova/+/903917 | 23:05 |
| opendevreview | Merged openstack/nova master: [codespell] ignore codespell in git blame https://review.opendev.org/c/openstack/nova/+/897215 | 23:29 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!