*** ociuhandu has joined #openstack-nova | 00:17 | |
*** ociuhandu has quit IRC | 00:21 | |
*** brinzhang has joined #openstack-nova | 00:23 | |
*** bbowen has quit IRC | 00:30 | |
*** bbowen has joined #openstack-nova | 00:30 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Moving functional jobs to Victoria testing runtime https://review.opendev.org/725372 | 00:59 |
---|---|---|
gmann | stephenfin: melwitt ^^ it need 3.15 version as min to have testenv section expandable. I was about to do PR in tox before i saw it is already fixed and there in 3.15 | 01:01 |
*** Liang__ has joined #openstack-nova | 01:09 | |
*** jazeltq has joined #openstack-nova | 01:38 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove the duplicate policy initialization https://review.opendev.org/726272 | 01:40 |
gmann | melwitt: ^^ this will fix the unit tests policy warning you observed. i searched in all unit tests and this is the only test doing the duplicate policy.init() which caused those warnings. | 01:40 |
*** lbragstad_ has joined #openstack-nova | 01:51 | |
*** lbragstad has quit IRC | 01:53 | |
*** lbragstad_ has quit IRC | 01:59 | |
*** tetsuro has quit IRC | 02:25 | |
openstackgerrit | Merged openstack/nova master: Silence amqp heartbeat warning https://review.opendev.org/724188 | 02:32 |
openstackgerrit | Merged openstack/nova master: remove support of oslo.messaging 9.8.0 warning message https://review.opendev.org/724725 | 02:37 |
*** sapd1_x has joined #openstack-nova | 02:41 | |
*** psachin has joined #openstack-nova | 03:00 | |
*** lbragstad has joined #openstack-nova | 03:05 | |
*** lbragstad has quit IRC | 03:09 | |
*** jazeltq has quit IRC | 03:10 | |
*** udesale has joined #openstack-nova | 03:33 | |
melwitt | gmann: cool thanks. warnings down to 64 from 4488. are these warnings left intended then? https://zuul.opendev.org/t/openstack/build/a185bd1d5c0f4d4280a8c1dc35020240/log/job-output.txt#13255 | 03:34 |
*** songwenping__ has joined #openstack-nova | 03:48 | |
*** brinzhang_ has joined #openstack-nova | 03:48 | |
*** brinzhang has quit IRC | 03:51 | |
*** songwenping_ has quit IRC | 03:51 | |
*** tetsuro has joined #openstack-nova | 04:08 | |
*** tetsuro has quit IRC | 04:10 | |
*** brinzhang has joined #openstack-nova | 04:11 | |
*** songwenping_ has joined #openstack-nova | 04:11 | |
*** tetsuro has joined #openstack-nova | 04:13 | |
*** brinzhang_ has quit IRC | 04:14 | |
*** songwenping__ has quit IRC | 04:14 | |
*** udesale_ has joined #openstack-nova | 04:16 | |
*** udesale has quit IRC | 04:18 | |
*** evrardjp has quit IRC | 04:36 | |
*** evrardjp has joined #openstack-nova | 04:36 | |
*** brinzhang has quit IRC | 04:38 | |
*** hamzy__ has joined #openstack-nova | 04:38 | |
*** brinzhang has joined #openstack-nova | 04:39 | |
*** hamzy_ has quit IRC | 04:41 | |
*** psachin has quit IRC | 04:51 | |
*** ociuhandu has joined #openstack-nova | 04:53 | |
*** ociuhandu has quit IRC | 04:58 | |
*** songwenping_ has quit IRC | 05:05 | |
*** songwenping_ has joined #openstack-nova | 05:05 | |
*** udesale_ has quit IRC | 05:13 | |
*** udesale has joined #openstack-nova | 05:22 | |
*** jamesdenton has quit IRC | 05:42 | |
*** jamesdenton has joined #openstack-nova | 05:43 | |
*** huaqiang has quit IRC | 05:51 | |
*** dpawlik has joined #openstack-nova | 05:55 | |
*** Yumeng has joined #openstack-nova | 06:03 | |
*** maciejjozefczyk has joined #openstack-nova | 06:10 | |
openstackgerrit | Jie Li proposed openstack/nova master: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/722187 | 06:12 |
openstackgerrit | Luyao Zhong proposed openstack/nova-specs master: spec update: virtual-persistent-memory https://review.opendev.org/726284 | 06:24 |
*** ociuhandu has joined #openstack-nova | 06:25 | |
*** udesale has quit IRC | 06:28 | |
*** udesale has joined #openstack-nova | 06:30 | |
*** ociuhandu has quit IRC | 06:34 | |
*** dklyle has quit IRC | 06:40 | |
*** slaweq has joined #openstack-nova | 06:43 | |
*** jamesdenton has quit IRC | 06:54 | |
*** jamesdenton has joined #openstack-nova | 06:54 | |
gibi | good morning | 06:54 |
gibi | bauzas: ack have a nice PTO | 06:54 |
*** belmoreira has joined #openstack-nova | 06:55 | |
*** nightmare_unreal has joined #openstack-nova | 06:59 | |
*** toabctl has joined #openstack-nova | 07:05 | |
*** ttsiouts has joined #openstack-nova | 07:05 | |
*** ratailor has joined #openstack-nova | 07:14 | |
*** tosky has joined #openstack-nova | 07:17 | |
*** Liang__ has quit IRC | 07:22 | |
*** udesale_ has joined #openstack-nova | 07:25 | |
*** udesale has quit IRC | 07:28 | |
*** xek has joined #openstack-nova | 07:32 | |
*** dtantsur|afk is now known as dtantsur | 07:39 | |
*** ralonsoh has joined #openstack-nova | 07:44 | |
*** sapd1 has quit IRC | 07:48 | |
*** Liang__ has joined #openstack-nova | 07:53 | |
gibi | sean-k-mooney: hi! do you have bandwidth to re-fresh the spec moving patch? https://review.opendev.org/#/c/721278/ I can take over if you are busy with other things | 07:54 |
*** ttsiouts has quit IRC | 07:56 | |
*** avolkov has joined #openstack-nova | 07:57 | |
*** ttsiouts has joined #openstack-nova | 08:01 | |
*** maciejjozefczyk has quit IRC | 08:03 | |
*** martinkennelly has joined #openstack-nova | 08:09 | |
*** tkajinam has quit IRC | 08:13 | |
*** takamatsu has joined #openstack-nova | 08:29 | |
*** tetsuro has quit IRC | 08:33 | |
*** ociuhandu has joined #openstack-nova | 08:36 | |
*** ttsiouts has quit IRC | 09:01 | |
*** ociuhandu has quit IRC | 09:01 | |
*** ociuhandu has joined #openstack-nova | 09:01 | |
openstackgerrit | Merged openstack/nova master: Wait for all servers to be active when testing vGPUs https://review.opendev.org/726087 | 09:11 |
*** kevinz has joined #openstack-nova | 09:11 | |
openstackgerrit | Xinran WANG proposed openstack/os-resource-classes master: Add new resource class for QAT card. https://review.opendev.org/726314 | 09:15 |
*** belmoreira has quit IRC | 09:18 | |
openstackgerrit | Stephen Finucane proposed openstack/nova master: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/722187 | 09:19 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Resolve issue with deprecated extra specs https://review.opendev.org/726315 | 09:19 |
*** belmoreira has joined #openstack-nova | 09:26 | |
*** ccamacho has joined #openstack-nova | 09:31 | |
*** ttsiouts has joined #openstack-nova | 09:35 | |
*** rcernin has quit IRC | 09:39 | |
*** ttsiouts has quit IRC | 09:40 | |
*** threestrands has quit IRC | 09:41 | |
*** Liang__ has quit IRC | 09:51 | |
*** owalsh has quit IRC | 10:15 | |
*** maciejjozefczyk has joined #openstack-nova | 10:19 | |
*** rcernin has joined #openstack-nova | 10:22 | |
*** owalsh has joined #openstack-nova | 10:22 | |
stephenfin | gibi: Could you take a look at https://review.opendev.org/#/c/724381/ today? | 10:26 |
gibi | stephenfin: added to my queue | 10:26 |
stephenfin | thanks | 10:26 |
*** ttsiouts has joined #openstack-nova | 10:34 | |
*** maciejjozefczyk has quit IRC | 10:39 | |
openstackgerrit | Stephen Finucane proposed openstack/nova master: tox: Integrate mypy https://review.opendev.org/676208 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Update and correct typing information https://review.opendev.org/714694 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add typing information https://review.opendev.org/714695 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Split instance NUMA object tests https://review.opendev.org/714696 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Replace 'cpu_pinning_requested' helper https://review.opendev.org/714697 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Don't consider overhead CPUs for unpinned instances https://review.opendev.org/714698 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Remove handling of pre-Train compute nodes https://review.opendev.org/714699 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Add validation for 'cpu_realtime_mask' https://review.opendev.org/468203 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Tweak the 'cpu_realtime_mask' handling slightly https://review.opendev.org/461456 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Rework 'get_realtime_constraint' https://review.opendev.org/714700 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Invert order of NUMA topology generation https://review.opendev.org/714701 | 10:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Remove '_numa_fit_instance_cell_with_pinning' https://review.opendev.org/714703 | 10:41 |
*** hrw has joined #openstack-nova | 10:45 | |
hrw | morning | 10:45 |
hrw | can someone take a look at https://review.opendev.org/#/c/723900/ patch? it is stable/train backport to set aarch64 cpu model | 10:46 |
kashyap | hrw: I can give a look; just wondering of any migration concerns | 10:54 |
kashyap | I have it open; about to +1, but made me pause: | 10:55 |
kashyap | Before there was no default CPU model on AAch64 'Train' VM, but now an "update" brings a default. | 10:55 |
*** ttsiouts has quit IRC | 11:13 | |
openstackgerrit | Merged openstack/nova master: config: Explicitly register 'remote_debug' CLI opts https://review.opendev.org/677444 | 11:14 |
*** ttsiouts has joined #openstack-nova | 11:16 | |
kashyap | hrw: Commented in the change | 11:19 |
*** tetsuro has joined #openstack-nova | 11:46 | |
*** tetsuro has quit IRC | 11:47 | |
*** raildo has joined #openstack-nova | 11:57 | |
*** adriant has quit IRC | 12:02 | |
*** adriant has joined #openstack-nova | 12:03 | |
*** slaweq_ has joined #openstack-nova | 12:16 | |
*** slaweq has quit IRC | 12:18 | |
sean-k-mooney | gibi: i was going to ask you about that actully | 12:20 |
sean-k-mooney | gibi: i can update it | 12:20 |
sean-k-mooney | has the policy stuff been merged | 12:20 |
sean-k-mooney | we were waiting for that to complete i think | 12:20 |
gibi | sean-k-mooney: do you mean we need a policy related spec update? | 12:22 |
sean-k-mooney | gibi: no gmann or bauzas i think had a spec that was close and i was asked to wait a few days before respinning | 12:22 |
sean-k-mooney | i was ment to do it last week but got distracted | 12:23 |
sean-k-mooney | i assume which ever one i was ment to wait for is not done | 12:23 |
sean-k-mooney | e.g. we wanted to see if it could get completed before rc1 or not | 12:23 |
sean-k-mooney | gibi: ill update it shortly | 12:24 |
gibi | looking at the open spec reviews I don't see anything open now | 12:24 |
gibi | that is close | 12:24 |
gibi | sean-k-mooney: thanks | 12:24 |
*** READ10 has joined #openstack-nova | 12:25 | |
*** jangutter_ has joined #openstack-nova | 12:25 | |
*** tkajinam has joined #openstack-nova | 12:27 | |
*** jangutter has quit IRC | 12:28 | |
*** lpetrut has joined #openstack-nova | 12:33 | |
*** kevinbenton has quit IRC | 12:38 | |
*** kevinbenton has joined #openstack-nova | 12:38 | |
*** ratailor has quit IRC | 12:57 | |
*** lbragstad has joined #openstack-nova | 13:13 | |
*** nightmare_unreal has quit IRC | 13:18 | |
*** nweinber has joined #openstack-nova | 13:27 | |
*** brinzhang has quit IRC | 13:33 | |
*** brinzhang has joined #openstack-nova | 13:34 | |
hrw | kashyap: thx | 13:41 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add MigrationTypeField https://review.opendev.org/706013 | 13:46 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'NovaObjectDictCompat' from 'Migration' https://review.opendev.org/723572 | 13:46 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'NovaObjectDictCompat' from 'InstancePCIRequest' https://review.opendev.org/723573 | 13:46 |
*** mlavalle has joined #openstack-nova | 13:57 | |
*** slaweq_ is now known as slaweq | 14:00 | |
*** ratailor has joined #openstack-nova | 14:03 | |
*** lpetrut has quit IRC | 14:08 | |
*** mriedem has joined #openstack-nova | 14:22 | |
*** ttsiouts has quit IRC | 14:23 | |
*** ttsiouts has joined #openstack-nova | 14:23 | |
*** ttsiouts has quit IRC | 14:28 | |
*** ratailor has quit IRC | 14:37 | |
*** lpetrut has joined #openstack-nova | 14:41 | |
*** nightmare_unreal has joined #openstack-nova | 14:42 | |
*** lpetrut_ has joined #openstack-nova | 14:49 | |
*** lpetrut has quit IRC | 14:52 | |
*** dklyle has joined #openstack-nova | 14:58 | |
*** brinzhang has quit IRC | 15:00 | |
*** brinzhang has joined #openstack-nova | 15:00 | |
*** lpetrut_ has quit IRC | 15:14 | |
*** sapd1_x has quit IRC | 15:15 | |
*** sangeet has joined #openstack-nova | 15:17 | |
sangeet | Does anyone know what triggers (or paramater) that marks service deleted in Services Table in Nova Database? Somehow consoleauth service was maked deleted and I want to undersatnd how it could have happened, while my service was still up | 15:18 |
*** sangeet has left #openstack-nova | 15:19 | |
*** sangeet has joined #openstack-nova | 15:20 | |
sangeet | Is there a paramater to mark service deleted automatically in services table in nova database? | 15:21 |
*** tkajinam has quit IRC | 15:22 | |
*** udesale_ has quit IRC | 15:28 | |
*** artom has quit IRC | 15:29 | |
sangeet | Any suggestion for above questions will be much apprecaited. | 15:31 |
*** dpawlik has quit IRC | 15:32 | |
*** bnemec is now known as beekneemech | 15:37 | |
*** artom has joined #openstack-nova | 15:40 | |
gmann | melwitt: should be 0 :), let me check | 15:41 |
*** gyee has joined #openstack-nova | 15:41 | |
gmann | sean-k-mooney: gibi that was delete-on-termination spec update which is merged- https://review.opendev.org/#/c/722194/ | 15:43 |
*** belmoreira has quit IRC | 15:45 | |
*** martinkennelly has quit IRC | 15:55 | |
*** martinkennelly has joined #openstack-nova | 15:56 | |
*** ttsiouts has joined #openstack-nova | 15:57 | |
openstackgerrit | Merged openstack/nova master: objects: Add migrate-on-load behavior for legacy NUMA objects https://review.opendev.org/724381 | 15:59 |
*** amodi has quit IRC | 16:03 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Moving functional jobs to Victoria testing runtime https://review.opendev.org/725372 | 16:06 |
*** ttsiouts has quit IRC | 16:13 | |
*** ttsiouts has joined #openstack-nova | 16:13 | |
*** ociuhandu has quit IRC | 16:15 | |
*** ociuhandu has joined #openstack-nova | 16:16 | |
*** ttsiouts has quit IRC | 16:18 | |
*** ociuhandu has quit IRC | 16:20 | |
*** ociuhandu has joined #openstack-nova | 16:20 | |
*** ygk_12345 has joined #openstack-nova | 16:23 | |
*** evrardjp has quit IRC | 16:36 | |
*** evrardjp has joined #openstack-nova | 16:36 | |
*** dtantsur is now known as dtantsur|afk | 16:42 | |
*** hamzy__ has quit IRC | 16:43 | |
ygk_12345 | @gibi hi | 16:44 |
*** hamzy has joined #openstack-nova | 16:47 | |
*** ttsiouts has joined #openstack-nova | 16:49 | |
*** ttsiouts has quit IRC | 16:53 | |
ygk_12345 | hi all | 16:54 |
ygk_12345 | i have upgraded the nova from stein to train and I have rebooted the computes. but after the compute is up , it is not listing all the vms in the virsh list | 16:54 |
ygk_12345 | but the vm's disk files are intact as /var/lib/nova/instances folder | 16:55 |
ygk_12345 | why is it so ? | 16:55 |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Suppress remaining policy warnings in unit tests https://review.opendev.org/726272 | 16:55 |
ygk_12345 | but the vm's disk files are intact in /var/lib/nova/instances folder | 16:56 |
ygk_12345 | can someone help me please | 16:57 |
*** ttsiouts has joined #openstack-nova | 16:57 | |
stephenfin | ygk_12345: are the instances appearing if you do 'nova list'? | 16:58 |
ygk_12345 | stephenfin yes in ERROR state | 16:58 |
ygk_12345 | i tried reset and rebooted bbut still not coming in the virsh list | 16:58 |
stephenfin | ygk_12345: do you have 'resume_guests_state_on_host_boot' configured? I assume not? | 16:59 |
ygk_12345 | stephenfin where to check that ? | 16:59 |
stephenfin | nova.conf | 16:59 |
ygk_12345 | let me check | 16:59 |
stephenfin | If that is left to it's default value, nova won't restart VMs for you | 16:59 |
openstackgerrit | Takashi Natsume proposed openstack/os-vif master: Remove translation sections from setup.cfg https://review.opendev.org/724974 | 16:59 |
stephenfin | though the domains should still be there, I would have thought | 17:00 |
ygk_12345 | so how do I recover those now ? the disk files are intact in the /var/lib/nova folder though | 17:00 |
ygk_12345 | stephenfin the virsh is not listing all the domain though | 17:00 |
ygk_12345 | stephenfin yes it is configured "resume_guests_state_on_host_boot = True" | 17:01 |
stephenfin | Odd. What do the logs say? | 17:01 |
ygk_12345 | stephenfin the computes log is complaining about just one missing disk of a vm at "nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/e7fefd4e-f2cf-4048-9b37-4ff807243291/disk" | 17:02 |
*** ttsiouts has quit IRC | 17:02 | |
stephenfin | can you rebuild that instance? | 17:03 |
ygk_12345 | stephenfin how ? | 17:03 |
stephenfin | nova rebuild SERVER_ID IMAGE_ID | 17:04 |
ygk_12345 | stephenfin ok let me try. please hold on | 17:04 |
ygk_12345 | stephenfin will I loose any data on this vm ? | 17:05 |
stephenfin | Yes. Rebuild is a destructive operation | 17:05 |
ygk_12345 | in that case I can also extract the disk file as image and spawn the vm right ? | 17:06 |
ygk_12345 | i dont want ot loose data | 17:06 |
ygk_12345 | stephenfin is there a way to restore those vms ? | 17:07 |
ygk_12345 | stephenfin I see lot of disk files in the location /var/lib/nova with the uudi's folders | 17:07 |
ygk_12345 | vm uuids | 17:08 |
ygk_12345 | can someone help me with this issue ? | 17:10 |
ygk_12345 | i see active base files still at the folder | 17:10 |
sean-k-mooney | gmann: ah yes that is the spec i was tinking of. | 17:12 |
sean-k-mooney | ygk_12345: you can snapshot the vm | 17:12 |
ygk_12345 | somehow it is not lsiting other virsh domains | 17:13 |
ygk_12345 | sean-k-mooney even when it is in error state u mean ? | 17:13 |
sean-k-mooney | oh no im justre reading back now | 17:13 |
sean-k-mooney | so in its curent state i dont think snapshot would work. | 17:14 |
sean-k-mooney | but that is generaly how you would preseve the content if you were going to rebuild | 17:14 |
ygk_12345 | sean-k-mooney why is it not listing the other virsh domains ? | 17:15 |
sean-k-mooney | ygk_12345: do you have your /var/lib/nova on nfs or other shared storage? | 17:15 |
sean-k-mooney | ygk_12345: are you using sudo | 17:15 |
ygk_12345 | sean-k-mooney yes it is an lvm share | 17:15 |
ygk_12345 | we mounted it | 17:15 |
sean-k-mooney | lvm is not a shared file system | 17:16 |
sean-k-mooney | as in shared over the netwrok between multiple hosts | 17:16 |
ygk_12345 | sean-k-mooney yes | 17:16 |
ygk_12345 | not in that sense | 17:16 |
ygk_12345 | it is deidcted to taht server | 17:16 |
ygk_12345 | *that host | 17:16 |
sean-k-mooney | there was a bug related to sometimes incorrectly deleting the vm disk if migration or evac failed on nfs not too loong ago | 17:17 |
sean-k-mooney | i was wondering if that is who you got into this state | 17:17 |
sean-k-mooney | it woudl appear not to be relatecxd | 17:17 |
ygk_12345 | sean-k-mooney this is just lvm and the uuid folders are still present in the /var/lib/nova folder which is an lvm | 17:18 |
sean-k-mooney | yep but the disk file is not correct | 17:18 |
ygk_12345 | sean-k-mooney what do u mean by correct ? | 17:18 |
sean-k-mooney | /var/lib/nova/instances/e7fefd4e-f2cf-4048-9b37-4ff807243291/disk is not present wehn you check | 17:19 |
ygk_12345 | sean-k-mooney yes its there | 17:19 |
sean-k-mooney | oh its there but nova thinks its not there | 17:19 |
ygk_12345 | oh yes that only one file is absent | 17:19 |
ygk_12345 | rest are there | 17:19 |
sean-k-mooney | ok | 17:20 |
ygk_12345 | but the domains have vanished | 17:20 |
ygk_12345 | thats what I am trying to say | 17:20 |
sean-k-mooney | sudo if you do "sudo virsh list -all" | 17:20 |
sean-k-mooney | there is nothing | 17:20 |
ygk_12345 | let me check | 17:20 |
sean-k-mooney | its either -all or --all i cant rememeber | 17:20 |
sean-k-mooney | nova creates the vms in the system namespace | 17:21 |
ygk_12345 | sean-k-mooney it is showing only two domains, but it shol;d have 30 + domains | 17:21 |
sean-k-mooney | if you dont use sudo on some operating sytems be default you wont see domains in the system namespace | 17:21 |
sean-k-mooney | oh just read back you just upgraded? | 17:22 |
sean-k-mooney | i need to check one thing | 17:22 |
ygk_12345 | yes | 17:22 |
ygk_12345 | from stein to train | 17:22 |
ygk_12345 | one compute node is fine | 17:22 |
ygk_12345 | the rest three are having this problem | 17:23 |
ygk_12345 | not listing all the domains | 17:23 |
ygk_12345 | after a reboot | 17:23 |
ygk_12345 | i have to manually mount the lvm | 17:23 |
sean-k-mooney | ya so im not sure which release we changed it in but we fixed a bug with evacuation a couple of release ago where we did not properly clean up domains. but the side effect of that is if you did an evacuate and it failed and you did a reset state | 17:23 |
ygk_12345 | on to the /var/lib/nova/instances | 17:23 |
sean-k-mooney | then after upgrade the domain could be deleted | 17:24 |
ygk_12345 | but they are intact in one compute | 17:24 |
ygk_12345 | i still see the disk files with nova uuids present | 17:24 |
sean-k-mooney | it only affected instance where the operator incoorect did reset state | 17:24 |
ygk_12345 | but onl;;y virsh domains vanished | 17:24 |
sean-k-mooney | basically im wondering if the period jobs deleted the vms | 17:25 |
ygk_12345 | many vms went into ERROR state | 17:25 |
ygk_12345 | no such crons | 17:25 |
sean-k-mooney | ygk_12345: its not a cron job we have a period task in the nova compute agent | 17:26 |
ygk_12345 | ok | 17:26 |
sean-k-mooney | that looks for stale domains form evacuation or migratiosn and deletes them | 17:26 |
ygk_12345 | can we restore these vms now ? | 17:26 |
ygk_12345 | these were active vms before the upgrade | 17:26 |
ygk_12345 | i still see the active base files in the log for these vanished domaisn | 17:28 |
sean-k-mooney | yes but there are cases where they can be active because you did "nova reset-state" to put them in active but did not clean up migrations or evac | 17:28 |
ygk_12345 | no ididn't do any reset either | 17:29 |
ygk_12345 | i still see the base disk files intact | 17:29 |
sean-k-mooney | ygk_12345: you dont see a log message like this do you https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9851-L9853 | 17:30 |
ygk_12345 | which message to be exact ? | 17:31 |
ygk_12345 | i see these vms in ERROR state | 17:32 |
sean-k-mooney | "Destroying instance as it has been evacuated from this host but still exists in the hypervisor ..." | 17:32 |
sean-k-mooney | or "Destroying instance with name label " | 17:32 |
sean-k-mooney | "'%s' which is marked as " | 17:32 |
sean-k-mooney | "DELETED but still present on host." | 17:33 |
ygk_12345 | nope | 17:33 |
ygk_12345 | its just in ERROR state | 17:33 |
sean-k-mooney | ok i was makeing sure it was not related too https://github.com/openstack/nova/commit/c4988cdabf311d29cf64af732091068cfabeedaa | 17:36 |
sean-k-mooney | which fixes https://bugs.launchpad.net/nova/+bug/1764883 | 17:36 |
openstack | Launchpad bug 1764883 in OpenStack Compute (nova) queens "Evacuation fails if the source host returns while the migration is still in progress" [Medium,Fix committed] - Assigned to Lee Yarwood (lyarwood) | 17:36 |
sean-k-mooney | but its not | 17:37 |
sean-k-mooney | that cause some vms in the active state to be deleted if operators incorrectly started an evacuation which failed and then reset the vm state. | 17:38 |
sean-k-mooney | ygk_12345: in this case im not aware of a bug that would result in the domains being deleted as part of a upgrade | 17:38 |
ygk_12345 | we did not evacuate. just upgraded from stein to train. one of the four computes is fine | 17:39 |
ygk_12345 | only virsh domains on these three computes vanished | 17:40 |
ygk_12345 | but the disk files are there | 17:40 |
*** ygk_12345 has quit IRC | 17:41 | |
artom | ygk_12345, so `virsh list` is empty? What about `virsh list --all`? | 17:41 |
*** ralonsoh has quit IRC | 18:08 | |
*** nweinber has quit IRC | 18:15 | |
*** ttsiouts has joined #openstack-nova | 18:19 | |
*** ttsiouts has quit IRC | 18:24 | |
sean-k-mooney | artom: it only has 2 domains out of like 30 | 18:33 |
sean-k-mooney | so something undefiend all the domains | 18:33 |
sean-k-mooney | and since the log message for the only code that i know of in nova that will do that is not present i dont really know why that happened | 18:33 |
sean-k-mooney | e.g. it was not the evac cleanup code or the preiodic take for deleted instances | 18:34 |
sean-k-mooney | so something undefiend the domains | 18:35 |
artom | sean-k-mooney, and since ygk are gone, we'll never know what happened :P | 18:39 |
sean-k-mooney | ya i dont really know how you woudl fix this either | 18:39 |
sean-k-mooney | i think the root disk files are already deleted | 18:40 |
sean-k-mooney | so im not sure there is much you can do at this point | 18:40 |
*** nweinber has joined #openstack-nova | 18:47 | |
*** dpawlik has joined #openstack-nova | 19:03 | |
*** jamesdenton has quit IRC | 19:43 | |
*** jamesdenton has joined #openstack-nova | 19:54 | |
*** benj_ has quit IRC | 19:55 | |
*** benj_ has joined #openstack-nova | 19:56 | |
*** nweinber has quit IRC | 20:05 | |
*** mgariepy has quit IRC | 20:12 | |
*** dpawlik has quit IRC | 20:12 | |
*** slaweq has quit IRC | 20:22 | |
*** amodi has joined #openstack-nova | 20:40 | |
*** raildo has quit IRC | 20:48 | |
*** KeithMnemonic has quit IRC | 21:01 | |
*** ccamacho has quit IRC | 21:02 | |
*** Yumeng has quit IRC | 21:23 | |
*** dustinc has joined #openstack-nova | 21:34 | |
*** songwenping_ has quit IRC | 21:39 | |
*** songwenping_ has joined #openstack-nova | 21:39 | |
*** jharris has joined #openstack-nova | 21:56 | |
jharris | hey, I am trying to get a nova profile working, and having trouble with blockmapping | 21:59 |
jharris | do I need to put the image in the block mapping as well as the volume I want to mount? | 21:59 |
*** jharris has quit IRC | 22:03 | |
*** jharris has joined #openstack-nova | 22:04 | |
openstackgerrit | Merged openstack/os-vif master: Remove translation sections from setup.cfg https://review.opendev.org/724974 | 22:07 |
*** jharris has quit IRC | 22:11 | |
*** songwenping__ has joined #openstack-nova | 22:20 | |
*** songwenping_ has quit IRC | 22:24 | |
*** mlavalle has quit IRC | 22:33 | |
*** mlavalle has joined #openstack-nova | 22:36 | |
*** martinkennelly has quit IRC | 22:37 | |
*** tosky has quit IRC | 23:11 | |
*** mriedem has left #openstack-nova | 23:15 | |
*** artom has quit IRC | 23:20 | |
*** songwenping_ has joined #openstack-nova | 23:21 | |
*** songwenping__ has quit IRC | 23:24 | |
*** mlavalle has quit IRC | 23:50 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!