| *** brinzhang has joined #openstack-nova | 00:09 | |
| *** tommylikehu has joined #openstack-nova | 00:58 | |
| *** sapd1 has joined #openstack-nova | 01:30 | |
| *** tiendc has joined #openstack-nova | 01:35 | |
| *** tbachman has joined #openstack-nova | 01:37 | |
| *** tetsuro has joined #openstack-nova | 01:45 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:47 | |
| *** Kevin_Zheng has joined #openstack-nova | 01:50 | |
| *** tetsuro has quit IRC | 01:57 | |
| *** tetsuro has joined #openstack-nova | 02:00 | |
| *** pvc has joined #openstack-nova | 02:01 | |
| pvc | hi anyone use cyborg? | 02:01 |
|---|---|---|
| *** Dinesh_Bhor has quit IRC | 02:05 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:13 | |
| *** mhen has quit IRC | 02:17 | |
| *** mhen has joined #openstack-nova | 02:23 | |
| *** hongbin has joined #openstack-nova | 02:24 | |
| *** hshiina has joined #openstack-nova | 02:44 | |
| *** Nel1x has joined #openstack-nova | 02:45 | |
| *** fried_rice has quit IRC | 02:53 | |
| *** fried_rice has joined #openstack-nova | 02:54 | |
| *** mrsoul has joined #openstack-nova | 02:55 | |
| *** trident has quit IRC | 02:56 | |
| *** trident has joined #openstack-nova | 02:56 | |
| *** Nel1x has quit IRC | 03:09 | |
| *** Nel1x has joined #openstack-nova | 03:09 | |
| *** tetsuro has quit IRC | 03:18 | |
| *** Dinesh_Bhor has quit IRC | 03:25 | |
| *** brinzhang has quit IRC | 03:27 | |
| *** brinzhang has joined #openstack-nova | 03:28 | |
| *** Dinesh_Bhor has joined #openstack-nova | 03:29 | |
| *** udesale has joined #openstack-nova | 03:54 | |
| *** kevinbenton has quit IRC | 04:00 | |
| *** kevinbenton has joined #openstack-nova | 04:00 | |
| *** ratailor has joined #openstack-nova | 04:14 | |
| *** hshiina_ has joined #openstack-nova | 04:15 | |
| *** hshiina has quit IRC | 04:18 | |
| *** Dinesh_Bhor has quit IRC | 04:19 | |
| openstackgerrit | Merged openstack/nova master: Add restrictions on updated_at when getting instance action records https://review.openstack.org/607801 | 04:33 |
| *** janki has joined #openstack-nova | 04:34 | |
| *** zzzeek_ has quit IRC | 04:36 | |
| *** hemna_ has quit IRC | 04:37 | |
| *** zzzeek has joined #openstack-nova | 04:41 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:41 | |
| *** hemna has joined #openstack-nova | 04:47 | |
| *** Nel1x has quit IRC | 04:47 | |
| *** hongbin has quit IRC | 04:52 | |
| *** ShilpaSD has joined #openstack-nova | 04:55 | |
| *** ircuser-1 has joined #openstack-nova | 05:02 | |
| *** markmcclain has quit IRC | 05:40 | |
| *** markmcclain has joined #openstack-nova | 05:42 | |
| *** Bhujay has joined #openstack-nova | 05:46 | |
| *** ratailor has quit IRC | 05:56 | |
| *** Bhujay has quit IRC | 06:15 | |
| *** tetsuro has joined #openstack-nova | 06:30 | |
| *** ratailor has joined #openstack-nova | 06:48 | |
| *** Luzi has joined #openstack-nova | 06:50 | |
| *** ccamacho has joined #openstack-nova | 07:07 | |
| *** tetsuro has quit IRC | 07:08 | |
| *** alexchadin has joined #openstack-nova | 07:15 | |
| *** dpawlik has joined #openstack-nova | 07:16 | |
| *** adrianc has joined #openstack-nova | 07:18 | |
| *** dpawlik has quit IRC | 07:20 | |
| *** dpawlik has joined #openstack-nova | 07:25 | |
| *** dpawlik has quit IRC | 07:27 | |
| *** dpawlik has joined #openstack-nova | 07:28 | |
| *** jangutter has joined #openstack-nova | 07:33 | |
| *** pcaruana has joined #openstack-nova | 07:46 | |
| *** skatsaounis has joined #openstack-nova | 07:48 | |
| *** lpetrut has joined #openstack-nova | 07:49 | |
| bauzas | good morning Nova | 07:51 |
| bauzas | pvc: nova integration for cyborg is way not done | 07:51 |
| bauzas | that's still a huge WIP | 07:51 |
| *** slaweq has joined #openstack-nova | 07:53 | |
| *** Dinesh_Bhor has quit IRC | 07:54 | |
| *** slaweq has quit IRC | 07:54 | |
| *** alex_xu has joined #openstack-nova | 07:59 | |
| *** slaweq has joined #openstack-nova | 07:59 | |
| *** slaweq has quit IRC | 07:59 | |
| pvc | noted on this bauzas | 08:01 |
| *** sahid has joined #openstack-nova | 08:03 | |
| *** wangy__ has joined #openstack-nova | 08:03 | |
| *** pvradu has joined #openstack-nova | 08:04 | |
| *** oanson has joined #openstack-nova | 08:05 | |
| *** pvradu has quit IRC | 08:06 | |
| *** pvradu has joined #openstack-nova | 08:06 | |
| *** psachin has joined #openstack-nova | 08:07 | |
| *** moshele has joined #openstack-nova | 08:07 | |
| *** tetsuro has joined #openstack-nova | 08:09 | |
| *** slaweq has joined #openstack-nova | 08:11 | |
| *** tetsuro has quit IRC | 08:13 | |
| *** maciejjozefczyk has joined #openstack-nova | 08:14 | |
| openstackgerrit | Lucian Petrut proposed openstack/nova stable/rocky: Fix os-simple-tenant-usage result order https://review.openstack.org/613853 | 08:15 |
| *** wznoinsk has quit IRC | 08:16 | |
| pvc | bauzas but can i use it manually? | 08:19 |
| gibi | finucannot, bauzas: hi! This weeks nova meeting is an EU timezone one and I'm regularly chair the EU timezone meetings. However 1st of November is a public holiday here so I'm looking for a way to pass this occasion to somebody else. Are you guys available on 1st of November? | 08:19 |
| *** tetsuro has joined #openstack-nova | 08:20 | |
| *** takashin has left #openstack-nova | 08:22 | |
| *** tetsuro has quit IRC | 08:27 | |
| *** pvc has quit IRC | 08:30 | |
| *** hshiina has joined #openstack-nova | 08:37 | |
| *** adrianc has quit IRC | 08:37 | |
| *** adrianc has joined #openstack-nova | 08:38 | |
| *** hshiina_ has quit IRC | 08:40 | |
| *** brinzhang has quit IRC | 08:41 | |
| *** wangy__ has quit IRC | 08:44 | |
| bauzas | gibi: Nov 1st is also an holiday for me | 08:44 |
| bauzas | so I need to pass | 08:45 |
| *** temka is now known as artom | 08:45 | |
| gibi | bauzas: OK, no worries. If everybody is out on 1st then I can chair a very short meeting anyhow :) | 08:47 |
| jangutter | Is November 1 diabetes day? | 08:48 |
| jangutter | or is halloween + candy not a major thing in Europe? | 08:49 |
| gibi | jangutter: pumpkin art is getting popular in Hungary but candy not really. For me it is going to cemeteries and lit candles for the dead | 08:51 |
| gibi | jangutter: but I think the root of halloween and all saints' day are the same | 08:53 |
| jangutter | gibi: it's weird how things go round, Samhain morphed to All Saint's Eve went overseas to the New World and came back. | 08:54 |
| *** jpena|off is now known as jpena | 08:57 | |
| *** Dinesh_Bhor has joined #openstack-nova | 08:58 | |
| gibi | jangutter: yeah, for me it is a bit similar to biologycal evolution but couple of order of magnitude faster | 08:58 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix nits in I7cbd5d9fb875ebf72995362e0b6693492ce32051 https://review.openstack.org/613863 | 09:01 |
| *** sridharg has joined #openstack-nova | 09:04 | |
| *** sridharg has quit IRC | 09:10 | |
| *** panda|off is now known as panda | 09:19 | |
| openstackgerrit | Jan Gutter proposed openstack/os-vif master: Extend port profiles with datapath offload type https://review.openstack.org/572081 | 09:45 |
| *** slaweq has quit IRC | 09:56 | |
| *** cdent has joined #openstack-nova | 09:57 | |
| *** slaweq has joined #openstack-nova | 09:59 | |
| *** hshiina has quit IRC | 10:00 | |
| *** sambetts_ is now known as sambetts | 10:23 | |
| *** slaweq has quit IRC | 10:25 | |
| *** slaweq has joined #openstack-nova | 10:27 | |
| *** dtantsur|afk is now known as dtantsur | 10:35 | |
| *** moshele has quit IRC | 10:36 | |
| *** adrianc has quit IRC | 10:39 | |
| *** mdbooth has joined #openstack-nova | 10:42 | |
| *** mdbooth is now known as mdbooth_390 | 10:43 | |
| *** tbachman has quit IRC | 10:46 | |
| *** dave-mccowan has joined #openstack-nova | 10:51 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Integration tests using gabbi-tempest https://review.openstack.org/613386 | 10:54 |
| *** moshele has joined #openstack-nova | 10:57 | |
| *** Dinesh_Bhor has quit IRC | 11:00 | |
| *** rdopiera has joined #openstack-nova | 11:02 | |
| *** rdopiera has left #openstack-nova | 11:02 | |
| *** wxy-xiyuan has quit IRC | 11:15 | |
| *** wxy-xiyuan has joined #openstack-nova | 11:15 | |
| *** spsurya has quit IRC | 11:16 | |
| *** MasterofJOKers has quit IRC | 11:16 | |
| *** mdbooth_390 has quit IRC | 11:16 | |
| *** zigo has quit IRC | 11:17 | |
| *** spsurya has joined #openstack-nova | 11:18 | |
| *** MasterofJOKers has joined #openstack-nova | 11:22 | |
| *** adrianc has joined #openstack-nova | 11:23 | |
| *** udesale has quit IRC | 11:26 | |
| *** jamesdenton has joined #openstack-nova | 11:32 | |
| *** janki has quit IRC | 11:32 | |
| *** erlon has joined #openstack-nova | 11:35 | |
| *** ccamacho has quit IRC | 11:44 | |
| *** ccamacho has joined #openstack-nova | 11:45 | |
| openstackgerrit | Merged openstack/nova master: Add post-test hook for testing evacuate https://review.openstack.org/602174 | 11:47 |
| openstackgerrit | Merged openstack/nova master: Add volume-backed evacuate test https://review.openstack.org/604397 | 11:47 |
| openstackgerrit | Merged openstack/nova master: Add functional regression test for bug 1794996 https://review.openstack.org/606106 | 11:47 |
| openstack | bug 1794996 in OpenStack Compute (nova) "_destroy_evacuated_instances fails and kills n-cpu startup if lazy-loading flavor on a deleted instance" [High,In progress] https://launchpad.net/bugs/1794996 - Assigned to Matt Riedemann (mriedem) | 11:47 |
| *** rdopiera has joined #openstack-nova | 12:02 | |
| *** tbachman has joined #openstack-nova | 12:07 | |
| *** tbachman_ has joined #openstack-nova | 12:11 | |
| *** tbachman_ has joined #openstack-nova | 12:11 | |
| *** tbachman has quit IRC | 12:12 | |
| *** tbachman_ is now known as tbachman | 12:12 | |
| *** rdopiera has left #openstack-nova | 12:21 | |
| *** edleafe- has joined #openstack-nova | 12:29 | |
| *** edmondsw has quit IRC | 12:29 | |
| *** edleafe- is now known as edleafe | 12:29 | |
| *** adrianc has quit IRC | 12:31 | |
| *** StevenK has quit IRC | 12:32 | |
| *** StevenK has joined #openstack-nova | 12:32 | |
| *** jmlowe has quit IRC | 12:34 | |
| *** edmondsw has joined #openstack-nova | 12:36 | |
| openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal https://review.openstack.org/591607 | 12:37 |
| *** tiendc has quit IRC | 12:45 | |
| *** udesale has joined #openstack-nova | 12:47 | |
| *** adrianc has joined #openstack-nova | 12:50 | |
| *** ratailor has quit IRC | 12:51 | |
| *** awaugama has joined #openstack-nova | 12:54 | |
| openstackgerrit | Jan Gutter proposed openstack/nova-specs master: Spec to implement generic HW offloads for os-vif https://review.openstack.org/607610 | 12:57 |
| *** ralonsoh has joined #openstack-nova | 12:58 | |
| *** cburgess has joined #openstack-nova | 13:01 | |
| *** awaugama has quit IRC | 13:02 | |
| *** awaugama has joined #openstack-nova | 13:02 | |
| *** munimeha1 has joined #openstack-nova | 13:02 | |
| *** jpena is now known as jpena|lunch | 13:03 | |
| *** cdent has quit IRC | 13:03 | |
| *** rdopiera has joined #openstack-nova | 13:07 | |
| *** rdopiera has left #openstack-nova | 13:08 | |
| *** dpawlik has quit IRC | 13:12 | |
| *** zigo has joined #openstack-nova | 13:13 | |
| *** mrch has joined #openstack-nova | 13:19 | |
| *** mriedem has joined #openstack-nova | 13:19 | |
| *** jmlowe has joined #openstack-nova | 13:22 | |
| *** elbragstad is now known as lbragstad | 13:22 | |
| *** takashin has joined #openstack-nova | 13:25 | |
| *** dpawlik has joined #openstack-nova | 13:26 | |
| *** dpawlik has quit IRC | 13:26 | |
| *** dpawlik has joined #openstack-nova | 13:27 | |
| *** jaosorior has quit IRC | 13:30 | |
| *** mdbooth_390 has joined #openstack-nova | 13:33 | |
| openstackgerrit | Jack Ding proposed openstack/nova master: Add I/O Semaphore to limit concurrent disk ops https://review.openstack.org/609180 | 13:36 |
| bauzas | edleafe: jaypipes_ : for obvious internal reasons, I won't be able to attend today's scheduler meeting | 13:44 |
| bauzas | which is in 15 mins | 13:45 |
| jaypipes_ | bauzas: IBM buys RH and you can't attend the scheduler meeting? :) | 13:46 |
| bauzas | we have our big guru that tells us the story follow-up in 10 mins | 13:46 |
| jaypipes_ | bauzas: Paul Cormier? | 13:46 |
| bauzas | so I'll take popcorn and carefully listen | 13:47 |
| bauzas | jaypipes_: nope, Jim | 13:47 |
| jaypipes_ | bauzas: I'll tl;dr it for you: "execs make out like bandits. IBM and RH have such amazing synergistic realities that it will enable the bottom line to expand into unforeseen markets and mesh amazing benches of copesetic technology." | 13:48 |
| bauzas | heh | 13:48 |
| jangutter | I find it an incredibly sad day: AIX-bashing will no longer be possible on the Nova chat :-( | 13:51 |
| jaypipes_ | jangutter: lol | 13:51 |
| edleafe | jangutter: it never went out of style, even among us IBMers | 13:51 |
| *** MasterofJOKers has quit IRC | 13:52 | |
| bauzas | on the good news, Europe is closer to the US for 2 weeks | 13:54 |
| bauzas | we shifted yesterday | 13:54 |
| *** jpena|lunch is now known as jpena|mtg | 13:55 | |
| *** takashin has quit IRC | 13:57 | |
| fried_rice | n-sch meeting now in #openstack-meeting-alt | 13:59 |
| *** fried_rice is now known as efried | 13:59 | |
| *** takashin has joined #openstack-nova | 14:00 | |
| *** artom has quit IRC | 14:00 | |
| *** jaosorior has joined #openstack-nova | 14:05 | |
| *** moshele has quit IRC | 14:08 | |
| *** alexchadin has quit IRC | 14:10 | |
| *** tbachman has quit IRC | 14:14 | |
| SteelyDan | jangutter: I bashed AIX more when I was at IBM than ever :) | 14:16 |
| *** SteelyDan is now known as dansmith | 14:16 | |
| *** jmlowe has quit IRC | 14:16 | |
| *** tbachman has joined #openstack-nova | 14:17 | |
| *** jaypipes_ is now known as jaypipes | 14:17 | |
| tobias-urdin | struggling how we should be able to update flavor specs, the ideal way would be to use teh `disabled` field on flavors and just create new ones since that keeps all rows in db and relationships still work, but there is a unique constraint on flavor names so cant use the same names | 14:18 |
| tobias-urdin | like updating disk and the flavor name is small.2x4 i cannot change the name, would have been good if the constraint was on the name, disabled, deleted combination (what i can see deleted field is deprecated though) | 14:20 |
| *** cdent has joined #openstack-nova | 14:20 | |
| tobias-urdin | and overall there isn't much info about the disabled field on flavors, thought it was deprecated at first but doesn't seem so? openstackclient doesn't even have support for setting it what i can see | 14:20 |
| mriedem | tobias-urdin: there is no REST API support for modifying the disabled field | 14:22 |
| mriedem | flavors, once they are in the API DB, do not use the SoftDeleteMixin in the model so they don't have the deleted/deleted_at columns anymore either | 14:22 |
| mriedem | tobias-urdin: you can delete the flavors and recreate though | 14:23 |
| *** tbachman has quit IRC | 14:23 | |
| mriedem | the flavor used to create a server is stored along with the instances record in the db | 14:23 |
| mriedem | so nova knows the info about the flavor used to create the instance even if the original flavor is gone | 14:23 |
| *** tbachman has joined #openstack-nova | 14:23 | |
| tobias-urdin | mriedem: ah great, then i can probably pull out that info from there if the referenced flavor_id doesn't exist anymore | 14:24 |
| tobias-urdin | is the disable flavor functionality supported and just somewhat implemented? or is it one of those things that will get pulled soon | 14:25 |
| mriedem | starting with microversion 2.47, showing a server will also show it's embedded flavor details | 14:25 |
| mriedem | the flavors.disabled column seems to be an afterthought from rax from several years ago | 14:26 |
| mriedem | was never plumbed into the api to change it, or nova-manage for that matter | 14:26 |
| BlackDex | Hello there | 14:26 |
| BlackDex | I'm having some issues with live-migrate of a large instances | 14:26 |
| mriedem | tobias-urdin: sorrison was having these same types of questions awhile back, and i think i suggested that we could potentially use the flavor update API (PUT) to set/unset the disabled field while changing the flavor | 14:26 |
| BlackDex | it has a large local storage, and takes a long time to transfer | 14:26 |
| mriedem | but i'm not sure how much that would be accepted | 14:27 |
| BlackDex | it looks like it timesout or something | 14:27 |
| BlackDex | can i change this timeout or something? | 14:27 |
| mriedem | BlackDex: https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_completion_timeout ? | 14:28 |
| mriedem | BlackDex: you might be interested in https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/live-migration-force-after-timeout.html as well but that's not implemented yet | 14:31 |
| tobias-urdin | mriedem: appreciate the help, i'll look if we can solve our issue taking the flavor data from the instance blob if the flavor doesn't exist | 14:31 |
| tobias-urdin | not related but horizon will show "Flavor not available" as well so need to fix up a patch there as well probably, if >= 2.47 microversion is supported | 14:32 |
| mriedem | when showing the server details in horizon you mean? | 14:33 |
| mriedem | yeah that's true | 14:33 |
| *** mlavalle has joined #openstack-nova | 14:34 | |
| BlackDex | mriedem: Seems to be the value i'm searching for | 14:34 |
| *** jmlowe has joined #openstack-nova | 14:37 | |
| *** psachin has quit IRC | 14:43 | |
| *** dklyle has joined #openstack-nova | 14:47 | |
| *** mchlumsky_ has quit IRC | 14:53 | |
| *** ralonsoh has quit IRC | 14:55 | |
| *** psachin has joined #openstack-nova | 14:55 | |
| *** hshiina has joined #openstack-nova | 14:57 | |
| *** jpena|mtg is now known as jpena | 14:59 | |
| *** beekneemech is now known as bnemec | 15:00 | |
| openstackgerrit | Jack Ding proposed openstack/nova-specs master: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | 15:00 |
| *** MasterofJOKers has joined #openstack-nova | 15:00 | |
| *** cfriesen has joined #openstack-nova | 15:01 | |
| *** artom has joined #openstack-nova | 15:02 | |
| *** takashin has left #openstack-nova | 15:04 | |
| *** Luzi has quit IRC | 15:04 | |
| *** pvradu has quit IRC | 15:06 | |
| *** psachin has quit IRC | 15:07 | |
| *** irclogbot_2 has joined #openstack-nova | 15:08 | |
| *** itlinux has quit IRC | 15:09 | |
| *** dtantsur is now known as dtantsur|brb | 15:10 | |
| *** wznoinsk has joined #openstack-nova | 15:11 | |
| *** jmlowe has quit IRC | 15:14 | |
| *** wznoinsk has quit IRC | 15:17 | |
| *** spatel has joined #openstack-nova | 15:21 | |
| spatel | sean-k-mooney: morning | 15:22 |
| *** cdent has quit IRC | 15:22 | |
| *** artom has quit IRC | 15:22 | |
| *** wznoinsk has joined #openstack-nova | 15:22 | |
| spatel | Look at this http://paste.openstack.org/show/733564/ | 15:23 |
| spatel | I found my compute nodes getting reboot itself, look like kernel bug with iommu | 15:23 |
| spatel | My kernel version is 3.10.0-862.14.4.el7.x86_64 #1 SMP | 15:23 |
| *** wznoinsk has quit IRC | 15:24 | |
| *** artom has joined #openstack-nova | 15:25 | |
| *** wznoinsk has joined #openstack-nova | 15:26 | |
| *** jmlowe has joined #openstack-nova | 15:29 | |
| lbragstad | jaypipes o/ wondering if you want to go through https://review.openstack.org/#/c/600266/ at some point | 15:35 |
| *** dpawlik has quit IRC | 15:35 | |
| lbragstad | we (keystone+oslo) want to make sure the interface is sane before merging that approach | 15:35 |
| *** dpawlik has joined #openstack-nova | 15:35 | |
| lbragstad | figured you would have some valuable feedback there regarding the discussions from the PTG | 15:36 |
| lbragstad | cc bnemec ^ | 15:36 |
| *** dpawlik has quit IRC | 15:37 | |
| *** dpawlik has joined #openstack-nova | 15:38 | |
| *** jaosorior has quit IRC | 15:38 | |
| *** lpetrut has quit IRC | 15:40 | |
| *** gyee has joined #openstack-nova | 15:40 | |
| *** dpawlik has quit IRC | 15:42 | |
| *** awaugama has quit IRC | 15:42 | |
| gibi | mriedem: hi! This weeks nova meeting is an EU timezone one and I'm regularly chair the EU timezone meetings. However 1st of November is a public holiday here so I'm looking for a way to pass this occasion to somebody else. Could you chair the Thursday's meeting? | 15:45 |
| *** artom has quit IRC | 15:46 | |
| *** Shilpa has joined #openstack-nova | 15:46 | |
| mriedem | gibi: sure | 15:46 |
| gibi | mriedem: thanks | 15:46 |
| mriedem | np | 15:46 |
| bauzas | FWIW, most of EU countries will respect this holiday | 15:47 |
| bauzas | but I guess mriedem won't be surprised to see me off because $holiday | 15:47 |
| *** cdent has joined #openstack-nova | 15:48 | |
| *** ShilpaSD has quit IRC | 15:49 | |
| mriedem | because .* :) | 15:49 |
| *** dpawlik has joined #openstack-nova | 15:54 | |
| *** hshiina has quit IRC | 15:54 | |
| *** mlavalle has quit IRC | 15:54 | |
| *** jmlowe has quit IRC | 15:56 | |
| *** mlavalle has joined #openstack-nova | 15:57 | |
| *** dpawlik has quit IRC | 15:58 | |
| *** kashyap has joined #openstack-nova | 15:58 | |
| *** jmlowe has joined #openstack-nova | 15:58 | |
| *** Shilpa has quit IRC | 15:59 | |
| jaypipes | lbragstad: yes, will try to take a look this afternoon. currently stuck working on $work shit in Chef. :( | 16:01 |
| *** itlinux has joined #openstack-nova | 16:01 | |
| *** jangutter has quit IRC | 16:10 | |
| cdent | "When we last saw our hero, he was stuck working in Chef. Today, he's still stuck working in Chef." | 16:11 |
| jaypipes | cdent: tomorrow, he shall be working in Chef. | 16:11 |
| cdent | womp womp | 16:11 |
| mriedem | efried: i guess https://docs.openstack.org/nova/latest/reference/update-provider-tree.html is out of date given reshaper and the allocations kwarg | 16:16 |
| BlackDex | Hello there. Something went wrong with a live-migrate. The instance is running on compute03 now, but openstack/horizon still thinks it is on compute01 | 16:16 |
| BlackDex | how can i fix this? | 16:16 |
| jaypipes | BlackDex: please see /topic | 16:16 |
| BlackDex | ah sorry | 16:16 |
| *** dtantsur|brb is now known as dtantsur | 16:16 | |
| mriedem | BlackDex: likely something failed in post-live migration and compute01 logs should tell you | 16:16 |
| finucannot | gibi: I don't know if you found someone but I can chair it if not, sure | 16:17 |
| mriedem | failed before updating the instance.host/node values to reflect compute03 | 16:17 |
| * finucannot is off today but accidentally opened IRC and will forget to reply tomorrow | 16:17 | |
| *** finucannot is now known as stephenfin | 16:17 | |
| * stephenfin goes back to making an IBM voodoo doll | 16:18 | |
| jaypipes | heh | 16:18 |
| *** eharney has joined #openstack-nova | 16:20 | |
| lbragstad | jaypipes sounds good - i'll be available this afternoon if you have questions/advice, just ping | 16:21 |
| gibi | stephenfin: mriedem already took it but thanks for the offer | 16:23 |
| jaypipes | lbragstad: will do, thx | 16:25 |
| *** beagles is now known as beagles_brb | 16:29 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Provide allocation_ratio/reserved amounts from update_provider_tree() https://review.openstack.org/613126 | 16:32 |
| mriedem | bauzas: efried: jaypipes: cdent: i think this is much more sane now ^ | 16:32 |
| cdent | mriedem: huzzah, on my list for today | 16:33 |
| mnaser | do we have someone who's a libvirt expert around here? | 16:36 |
| mnaser | i'm fairly certain i found a bug that breaks live migration because of libvirt | 16:36 |
| mnaser | but i need a bit of guidance in nailing down the root cause :\ | 16:36 |
| *** adrianc has quit IRC | 16:37 | |
| mnaser | or heck, could be a nova bug if it's nova that generates the xml at the destination libvirt | 16:37 |
| mnaser | but afaik that's not how it works | 16:37 |
| mdbooth_390 | mnaser: Shoot. I might know something. | 16:37 |
| mdbooth_390 | mnaser: Or kashyap might. | 16:37 |
| mnaser | mdbooth_390: i have a live migration that is failing with "qemu-kvm: get_pci_config_device: Bad config data: i=0x10 read: 61 device: 1 cmask: ff wmask: c0 w1cmask:0" | 16:38 |
| mnaser | upon doing research, it looks like that issue would come up if you change host_mtu, so i realized that the VMs that fail to migrate don't have host_mtu defined in the source, but for some reason, it's defined in the destination | 16:38 |
| mnaser | so cat /var/log/libvirt/qemu/instance-being-migrated.log shows the new process being created (the one waiting for memory to be streamed in) with host_mtu=1500 defined | 16:39 |
| mnaser | and then the migration fails because there's something something that becomes of a different size that causes that issue | 16:39 |
| efried | mriedem: Re upt doc, yes, agree it should get updated. Guess that was missed when we did the docstrings. | 16:39 |
| mnaser | so ideally: vms shouldn't have host_mtu in the qemu command, so the libvirt xml shouldn't have that... and i'm not sure how we're getting to that state | 16:40 |
| openstackgerrit | Chris Friesen proposed openstack/nova-specs master: Add support for emulated virtual TPM https://review.openstack.org/571111 | 16:40 |
| mdbooth_390 | mnaser: Different libvirt versions source and dest? | 16:40 |
| mnaser | mdbooth_390: yes | 16:40 |
| mnaser | mdbooth_390: also, new instances booted in new libvirt contain host_mtu=1500 | 16:41 |
| mnaser | which means newly booted instances post-upgrade are live-migrateable, but pre-upgrade are not | 16:41 |
| mdbooth_390 | mnaser: Any chance it's this: https://bugzilla.redhat.com/show_bug.cgi?id=1449346 ? | 16:41 |
| openstack | bugzilla.redhat.com bug 1449346 in libvirt "Addition of host_mtu=XXXX changes PCI ioport size & addresses for virtio-net device" [Unspecified,Closed: errata] - Assigned to laine | 16:41 |
| mnaser | mdbooth_390: i actaully saw this one, the revert is included in my local branch *however* the symptoms of the failure is exactly that (and that's what made me look for host_mtu) | 16:42 |
| mnaser | so both my src and dest already have that patch | 16:42 |
| *** tomtom001 has joined #openstack-nova | 16:43 | |
| mnaser | my question is: are xml files generated during a live migration by nova? | 16:43 |
| mdbooth_390 | mnaser: No, they aren't. | 16:43 |
| mdbooth_390 | They're tweaked by Nova, though. | 16:44 |
| mnaser | ok so that eliminates nova possibly sending an mtu when it starts up | 16:44 |
| mnaser | Aaaaah | 16:44 |
| mdbooth_390 | But I don't think we set host_mtu anywhere | 16:44 |
| mnaser | you do | 16:44 |
| mnaser | one sec | 16:44 |
| mnaser | mdbooth_390: https://github.com/openstack/nova/commit/f02b3800051234ecc14f3117d5987b1a8ef75877 | 16:44 |
| cfriesen | mnaser: we do generate xml files during live migration | 16:44 |
| mnaser | so this ignores the fact that the source might *not* have host_mtu set | 16:45 |
| cfriesen | the source generates the xml and passes it to libvirt as part of the migration | 16:45 |
| mdbooth_390 | cfriesen: Nah, we do it post. | 16:45 |
| mdbooth_390 | cfriesen: IIRC the xml sent to the dest is the running xml. | 16:45 |
| cfriesen | mdbooth_390: we save the xml post, but we *generate* the xml and pass it to libvirt as part of the live migration call | 16:45 |
| mdbooth_390 | cfriesen: So not generated by nova during the LM. | 16:45 |
| tomtom001 | Hello, I ran into an issue in queens where if I shutdown CPU-A and try to evacuate the evacuation fails, then I delete the VM that I tried to evacuate. When I startup CPU-A nova-cmopute service fails to start because it can't find remote instance (the VM). What data is nova trying to use to determine what instances it needs to look for in it's startup process? | 16:45 |
| mnaser | right, well if you look at the commit above, it does set the host_mtu stuff | 16:46 |
| mdbooth_390 | cfriesen: Otherwise we'd lose all address info, for eg, and the lm would fail. | 16:46 |
| mnaser | so.. could it be the culprit ? | 16:46 |
| *** sambetts is now known as sambetts|afk | 16:46 | |
| mnaser | added in rocky too and this is a rocky deployment | 16:46 |
| cfriesen | mdbooth_390: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L604 note the "destination_xml" arg | 16:47 |
| *** N3l1x has joined #openstack-nova | 16:47 | |
| *** burt has joined #openstack-nova | 16:47 | |
| melwitt | 16:48 | |
| mnaser | oh | 16:48 |
| mdbooth_390 | cfriesen: Look how it's generated | 16:48 |
| mdbooth_390 | cfriesen: 'generated' | 16:48 |
| mdbooth_390 | It's from the running guest | 16:49 |
| mnaser | https://github.com/openstack/nova/blob/d87852ae6a1987b6faa3cb5851f9758b47ef4636/nova/virt/libvirt/driver.py#L7092-L7124 | 16:49 |
| cfriesen | mnaser: that commit looks suspicious to me | 16:49 |
| mnaser | yes | 16:49 |
| mnaser | but | 16:49 |
| mnaser | it refreshes all the nic info | 16:49 |
| mnaser | from that xml | 16:49 |
| mnaser | https://github.com/openstack/nova/blob/f02b3800051234ecc14f3117d5987b1a8ef75877/nova/virt/libvirt/vif.py#L602-L624 | 16:49 |
| mnaser | https://github.com/openstack/nova/blob/f02b3800051234ecc14f3117d5987b1a8ef75877/nova/virt/libvirt/vif.py#L566-L598 | 16:50 |
| cfriesen | sean-k-mooney: possible problem with one of your commits: ^ | 16:50 |
| mnaser | https://github.com/openstack/nova/blob/9ae0cb67fabe09d6114680602f13b24da9dc88e2/nova/virt/libvirt/migration.py#L90 | 16:50 |
| mnaser | and that's how it breaks | 16:50 |
| mnaser | we upgrade the vif config so if you're trying to migrate, host_mtu will be set in the destination which results in live migration failing | 16:51 |
| mdbooth_390 | mnaser: So the guest was created before mtu was added, then nova was upgraded on the source, then we added mtu before lm? | 16:51 |
| mnaser | mdbooth_390: yes | 16:51 |
| mnaser | guest was long living, nova upgraded to rocky, live migration attempted, mtu is added before lm starts, lm fails at destination because host_mtu affects some internal kvm stuff that are beyond me :) | 16:52 |
| mdbooth_390 | mnaser: Still sounds like a libvirt bug to me that this changes addressing, no? | 16:52 |
| cfriesen | mdbooth_390: no, we're feeding the dest xml additional parameters that weren't there before | 16:52 |
| mnaser | HOWEVER this also confirms the theory that instances started AFTER rocky would not be affected by this issue, i see it here | 16:52 |
| mnaser | mdbooth_390: the libvirt bug there did the same behaviour we just started doing in nova, it started feeding host_mtu all the time and broke live migrations, so they reverted it | 16:53 |
| cfriesen | in a live migration you need to create the dest guest with xml that is compatible with the source | 16:53 |
| mnaser | we're doing the same mistake.. just one layer above :) | 16:53 |
| mdbooth_390 | mnaser: I feel like I was at best a catalyst in that discussion. A figurative rubber duck :) | 16:55 |
| cfriesen | I think we'll need to special-case this in the virt driver to ensure we don't add/remove the MTU setting over a live migration | 16:55 |
| cfriesen | and we get to make fun of sean-k-mooney and mriedem for missing this scenario | 16:56 |
| mnaser | well i needed to know about the nova-touching-xml part and then it made more sense :) | 16:56 |
| mnaser | okay so | 16:56 |
| mnaser | i'll file a bug | 16:56 |
| mnaser | and i can hack on this because it's blocking a huge upgrade right now.. | 16:56 |
| mnaser | if anyone has suggestions.. maybe try to see if there's no mtu set and it's an lm, not set it ? | 16:57 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: No longer call _normalize_inventory_from_cn_obj from upt flow https://review.openstack.org/613991 | 16:57 |
| cfriesen | I don't think we store the nova service version at the time an instance moved to a given compute node, so I think you might have to look at that instance xml before updating the vif-related xml stuff and ensure the mtu part stays the same | 16:58 |
| mriedem | what did i miss? mixed version hypervisors with host mtu stuff on one of the 100 different vif type configurations in neutron? oops. | 16:58 |
| mdbooth_390 | mriedem: Yeah, come on. It's so obvious! | 16:59 |
| *** udesale has quit IRC | 17:00 | |
| *** jmlowe has quit IRC | 17:01 | |
| *** sahid has quit IRC | 17:03 | |
| cfriesen | I updated the vTPM spec, I think it should address all the issues that were raised in the pervious version. | 17:04 |
| mnaser | mriedem: not even mixed version hypervisor | 17:05 |
| mnaser | i think this breaks live migrations of any vm created before pre-rocky to post-rocky | 17:05 |
| mdbooth_390 | mnaser: Sorta. Mixed on the same host. | 17:05 |
| mnaser | yeah i guess | 17:05 |
| * mnaser is writing bug | 17:05 | |
| cfriesen | mixed version nova, not hypervisor | 17:05 |
| mriedem | so live migrating a queens vm to a stein? | 17:06 |
| cfriesen | I think queens to rocky, potentially. | 17:06 |
| cfriesen | for vifs of type ethernet and bridge, on libvirt 3.3.0+ | 17:07 |
| cfriesen | (assuming it's https://github.com/openstack/nova/commit/f02b3800051234ecc14f3117d5987b1a8ef75877 that is the culprit) | 17:07 |
| tomtom001 | https://bugs.launchpad.net/nova/+bug/1583504 This is still an issue when: You shutoff CPU-A - then evacuate VM-B, then if it fails you decide to delete VM-B. When you start CPU-A, it cannot find VM-B and fails to start nova-compute | 17:08 |
| openstack | Launchpad bug 1713783 in OpenStack Compute (nova) pike "duplicate for #1583504 After failed evacuation the recovered source compute tries to delete the instance" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | 17:08 |
| mdbooth_390 | mriedem: You create an instance on Queens, then upgrade nova on that compute to Rocky, then try to migrate it somewhere. | 17:08 |
| mdbooth_390 | mnaser: ^^^^ ? | 17:09 |
| *** markvoelker has joined #openstack-nova | 17:12 | |
| mnaser | mdbooth_390: correct | 17:13 |
| mnaser | mriedem: cfriesen migrating a vm that was running before nova was upgraded to rocky would be affected | 17:14 |
| *** tommylikehu has quit IRC | 17:28 | |
| mnaser | oh wait | 17:33 |
| mnaser | that might not even be the commit that broke it | 17:33 |
| mnaser | https://github.com/openstack/nova/blob/f02b3800051234ecc14f3117d5987b1a8ef75877/nova/virt/libvirt/driver.py#L6900-L6904 | 17:34 |
| mnaser | because at the time we didnt regen network | 17:34 |
| mnaser | https://github.com/openstack/nova/commit/2b52cde565d542c03f004b48ee9c1a6a25f5b7cd | 17:36 |
| mnaser | this one specifically broke it | 17:36 |
| *** jmlowe has joined #openstack-nova | 17:36 | |
| *** READ10 has joined #openstack-nova | 17:39 | |
| mnaser | https://bugs.launchpad.net/nova/+bug/1800511 | 17:39 |
| openstack | Launchpad bug 1800511 in OpenStack Compute (nova) "VMs started before Rocky upgrade cannot be live migrated" [Undecided,New] | 17:39 |
| mnaser | i think this might be an easy fix actaully | 17:39 |
| *** awaugama has joined #openstack-nova | 17:40 | |
| mnaser | looks like _update_vif_xml is only used in migrations | 17:40 |
| *** bnemec has quit IRC | 17:42 | |
| *** beagles_brb is now known as beagles | 17:44 | |
| mriedem | it is | 17:44 |
| mriedem | live migrations | 17:44 |
| mnaser | ok, so i will try to get a failing test case there | 17:45 |
| mnaser | where no mtu => adds an mtu, seems straightforward | 17:46 |
| mriedem | is the libvirt version the same on both hosts? | 17:46 |
| mriedem | >=3.3? | 17:47 |
| mnaser | mriedem: nope its 3.9 | 17:47 |
| mriedem | which is >3.3 where mtu is supported for those vif types | 17:47 |
| mnaser | yeah but the problem is if you provide an mtu the internals of qemu change and the instance becomes non-livemigrateable | 17:47 |
| mnaser | and actually if you do virsh dumpxml --migratable <instance> it doesnt even include mtu | 17:48 |
| mnaser | so libvirt kinda conciously doesnt put it, but we do | 17:48 |
| *** bnemec has joined #openstack-nova | 17:50 | |
| mriedem | ok so https://github.com/openstack/nova/commit/2b52cde565d542c03f004b48ee9c1a6a25f5b7cd exposes the issue because it calls the get_config code for the vif which returns the mtu now | 17:51 |
| mriedem | and we try sending that to the dest | 17:51 |
| mnaser | mriedem: correct | 17:51 |
| cdent | jaypipes: if you loving on the gabbi integration tests, I made this one for nova (the required tempest and gabbi-tempest changes have merged): https://review.openstack.org/#/c/613386/ | 17:51 |
| mriedem | i think this would have been an issue regardless of https://github.com/openstack/nova/commit/2b52cde565d542c03f004b48ee9c1a6a25f5b7cd if the guest xml on the source had the mtru in the vif xml | 17:51 |
| mriedem | *mtu | 17:51 |
| mriedem | since we'd just send that over unchanged | 17:52 |
| mnaser | mriedem: if mtu exists on both sides, it's fine | 17:52 |
| mnaser | if mtu doesn't in one, it blows up | 17:52 |
| mriedem | sure but i don't think https://github.com/openstack/nova/commit/2b52cde565d542c03f004b48ee9c1a6a25f5b7cd changes that | 17:52 |
| mnaser | so if source has mtu, then dest will get it .. if source doesnt, dest won't.. and we're good | 17:52 |
| mnaser | it calls self.vif_driver.get_config | 17:53 |
| mnaser | which gets mtu configs | 17:53 |
| mriedem | *on the source host* | 17:53 |
| mriedem | we don't know what is configured for mtu on the dest | 17:53 |
| mriedem | since that's not in the LiveMigrateData object | 17:53 |
| mriedem | as far as i know | 17:53 |
| mriedem | _update_vif_xml is called from the source | 17:53 |
| mriedem | using migrate data from the dest | 17:53 |
| *** lpetrut has joined #openstack-nova | 17:54 | |
| mnaser | yes, to "modify" the xml which will be shipped to dest to start the instance to recieve data.. right? | 17:54 |
| mriedem | yes | 17:54 |
| mriedem | but if the xml on the source already had mtu in it, and the dest doesn't support that, then kaboom anyway right? | 17:54 |
| cfriesen | mriedem: I think the issue is that the instance originally didn't have mtu in it | 17:54 |
| cfriesen | then we upgraded nova, and now it would have mtu | 17:54 |
| mnaser | the host_mtu was added really long time ago in qemu | 17:55 |
| mnaser | so its not likely that qemu will crash for lack of that feature | 17:55 |
| *** eharney has quit IRC | 17:55 | |
| mnaser | but if the destination doesnt get a host_mtu.. it just .. doesnt set one and moves on | 17:55 |
| cfriesen | nova on the dest doesn't need to have the mtu, but the xml that we pass the live migration needs to match the xml that was used to start the running instance. | 17:55 |
| mnaser | cfriesen: doing a much better job explaining this than i am anyways :p | 17:56 |
| mriedem | "but the xml that we pass the live migration needs to match the xml that was used to start the running instance." | 17:58 |
| mriedem | ok i guess i didn't know that was a thing | 17:58 |
| mriedem | i'm not sure why it cares about the xml from when the guest was last started if we're sending over the xml that we want | 18:00 |
| mriedem | same is true for things like block devices | 18:01 |
| mriedem | so the proposed fix is going to be checking within _update_vif_xml if the source guest xml had mtu set before we call get_config to get the new dest xml vif config? | 18:02 |
| mriedem | and if not, omit it from the dest xml? | 18:02 |
| mnaser | mriedem: i think that is where im going | 18:02 |
| mnaser | i think i have a failing test too | 18:02 |
| mnaser | another reason it was missed was because the tests were testing bridge => vhostuser | 18:04 |
| mnaser | vhostuser doesnt set mtus | 18:04 |
| mnaser | so the tests didnt show mtu appearing | 18:04 |
| *** eharney has joined #openstack-nova | 18:05 | |
| *** mvkr has quit IRC | 18:05 | |
| mriedem | i'm pretty sure the live migration tests for _update_vif_xml are just using fake xml aren't they? | 18:07 |
| *** jpena is now known as jpena|off | 18:07 | |
| mriedem | well maybe not, but i'm not sure they would have cared much about the delta given part of the reason for that change was live migrating between vif types which is going to yield different xml | 18:08 |
| mnaser | mriedem: kinda, also only bridge and ethernet net_type support mtu | 18:11 |
| mnaser | and the tests was testing bridge=>vhostuser which meant mtu was never added anyways | 18:12 |
| openstackgerrit | Jack Ding proposed openstack/nova-specs master: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | 18:19 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: Add tests for bug #1800511 https://review.openstack.org/614004 | 18:21 |
| openstack | bug 1800511 in OpenStack Compute (nova) rocky "VMs started before Rocky upgrade cannot be live migrated" [High,Triaged] https://launchpad.net/bugs/1800511 | 18:21 |
| mnaser | mriedem: cfriesen mdbooth_390 ^ round 1.. that's a failing test that's commented out, i'll work on the actual fix now | 18:21 |
| *** mrch has quit IRC | 18:28 | |
| mnaser | mriedem: sorry to bother ya, do you think _update_vif_xml() should make sure "mtu" is there (if it was before) and removed if it wasn't there before.. or should i pass an extra arg to get_vif_config() inside _update_vif_xml() like migration=True and handle it in the vif layer? | 18:29 |
| mriedem | i think it would probably be simpler to isolate it within _update_vif_xml rather than plumb down through to the vif get_config stuff | 18:30 |
| mriedem | i just wonder how whack-a-mole this is going to be going forward with any changes to that os-vif code | 18:31 |
| mriedem | or if this mtu thing is an outlier | 18:31 |
| mnaser | mriedem: it looks like the mtu is an outlier because $kvm things .. but i dunno | 18:32 |
| mriedem | yeah i was just wondering based on that bz | 18:34 |
| mriedem | which i skimmed | 18:34 |
| mriedem | sounds like libvirt had to revert that | 18:34 |
| mriedem | s/that/something/ | 18:34 |
| mnaser | yep, they did back in 3.3 | 18:34 |
| mnaser | so we really just repeated their bug a layer above | 18:34 |
| mriedem | sweet | 18:35 |
| *** erlon has quit IRC | 18:41 | |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: Add tests for bug #1800511 https://review.openstack.org/614004 | 18:44 |
| openstack | bug 1800511 in OpenStack Compute (nova) rocky "VMs started before Rocky upgrade cannot be live migrated" [High,Triaged] https://launchpad.net/bugs/1800511 | 18:44 |
| *** jmlowe has quit IRC | 18:46 | |
| cfriesen | mriedem: missed your comment earlier. the XML that we send as part of the migration is used to start up qemu on the dest, this creates data structures in memory, and those data structures need to match the data structures for the guest on the source node. | 18:47 |
| mriedem | cfriesen: sure but this isn't the only thing we change on the dest xml when we send it over | 18:48 |
| mriedem | which is why i'm hoping the mtu thing is an outlier | 18:49 |
| cfriesen | yeah, unless it changes the in-memory layout we don't really care. | 18:50 |
| cfriesen | so we preserve things like device types, virtual machine type, CPU/RAM topology, etc. | 18:51 |
| cfriesen | I wonder if the MTU size affects the layout of the NIC ring buffers | 18:51 |
| mnaser | i mean i guess this explains why/what | 18:52 |
| mnaser | https://bugzilla.redhat.com/show_bug.cgi?id=1449346 | 18:52 |
| openstack | bugzilla.redhat.com bug 1449346 in libvirt "Addition of host_mtu=XXXX changes PCI ioport size & addresses for virtio-net device" [Unspecified,Closed: errata] - Assigned to laine | 18:52 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: Avoid setting MTU if none was configured https://review.openstack.org/614008 | 18:52 |
| *** jaosorior has joined #openstack-nova | 18:52 | |
| mnaser | anyhow | 18:52 |
| cfriesen | ah, that would do it | 18:53 |
| mnaser | thats a fix with the test in the patch above it, i'd be more than happy to iterate if anyone has comments | 18:53 |
| mnaser | considering this has put a big upgrade project to a halt :< | 18:53 |
| *** jmlowe has joined #openstack-nova | 18:55 | |
| mriedem | i'm on a call for the next hour or so but will try to take a look | 18:57 |
| mnaser | cool, thank you, please highlight me if anything so i can iterate | 18:58 |
| *** jmlowe has quit IRC | 19:08 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930 | 19:08 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012 | 19:08 |
| cfriesen | mnaser: got a possible problem scenario for you. boot instance on upgraded nova, live-migrate to non-upgraded nova (this will preserve the MTU), then live-migrate back. I suspect the running guest will have MTU, but the generated XML won't. | 19:10 |
| mnaser | cfriesen: isn’t that covered by the _with_mtu test case? | 19:11 |
| *** dtantsur is now known as dtantsur|afk | 19:11 | |
| mnaser | Where we have a source mtu and it is maintained for the live migration | 19:11 |
| mnaser | It only omits it if the domain XML doesn’t have it, and if the domain XML doesn’t have it then the host_mtu is not set and we’re okay | 19:13 |
| cfriesen | no, in the scenario I just described the running guest would have the MTU, but it's on an older nova that doesn't even try to generate the mtu in the xml | 19:13 |
| cfriesen | unless we actually dump the xml from the running guest, in which case it might be okay | 19:14 |
| cfriesen | can't remember the exact details right now | 19:14 |
| mnaser | cfriesen: I’m assuming in all of this that the XML is being dumped from the running guest | 19:15 |
| mnaser | Let me check | 19:16 |
| cfriesen | yeah, it does. get_xml_desc() in guest.py | 19:17 |
| *** lpetrut has quit IRC | 19:18 | |
| mnaser | Yup. And it specifies dump_migratable=True | 19:19 |
| mnaser | Which I assume means libvirt will be taking care of not making us do bad things | 19:19 |
| *** tbachman has quit IRC | 19:22 | |
| *** eharney has quit IRC | 19:32 | |
| mriedem | mnaser: comments inline | 19:32 |
| mriedem | mostly just nits | 19:32 |
| *** artom has joined #openstack-nova | 19:32 | |
| mriedem | cfriesen: you're talking about this right? https://github.com/openstack/nova/blob/a0eacbf7fff60282007ddca705ef7331e8a4a6f8/nova/virt/libvirt/migration.py#L83 | 19:36 |
| mriedem | yeah that's what i was referring to earlier - before _update_vif_xml we'd just always send the source vif xml unchanged | 19:36 |
| *** READ10 has quit IRC | 19:37 | |
| cfriesen | mriedem: yes | 19:37 |
| *** artom has quit IRC | 19:40 | |
| mnaser | mriedem: cfriesen thanks for the reviews both, i tried to address but in case it wasn't clear in terms of comments, please let me know | 19:44 |
| * mnaser is bad at explaining things | 19:44 | |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: Add tests for bug #1800511 https://review.openstack.org/614004 | 19:44 |
| openstack | bug 1800511 in OpenStack Compute (nova) "VMs with vif_type bridge/tap started before Rocky upgrade cannot be live migrated" [High,In progress] https://launchpad.net/bugs/1800511 - Assigned to Mohammed Naser (mnaser) | 19:44 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: libvirt: Avoid setting MTU during live migration if unset https://review.openstack.org/614008 | 19:44 |
| *** jmlowe has joined #openstack-nova | 19:46 | |
| cfriesen | mnaser: much better, one little comment | 19:50 |
| mnaser | cfriesen: i guess i wanted to make sure that a normal reboot wouldn't do the trick (because the qemu-kvm process doesnt get restarted) | 19:52 |
| mnaser | but i guess the follow up comment clears it up | 19:52 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: libvirt: Avoid setting MTU during live migration if unset https://review.openstack.org/614008 | 19:52 |
| mnaser | cfriesen: voila | 19:52 |
| cfriesen | mnaser: a "nova reboot" without the "hard" option will reset things | 19:52 |
| mnaser | really | 19:52 |
| mnaser | i thought i tried to do an acpi reboot | 19:53 |
| cfriesen | we try an ACPI shutdown | 19:54 |
| cfriesen | then if that doesn't work we do a hard shutdown | 19:54 |
| mnaser | aaaah | 19:54 |
| mnaser | gotcha | 19:54 |
| mnaser | so the reboot always restarts the process | 19:54 |
| mnaser | cool, TIL! | 19:54 |
| mriedem | pep8 is going to fail on https://review.openstack.org/#/c/614004/ | 19:56 |
| mnaser | ah shit | 19:56 |
| mnaser | ill run that locally | 19:57 |
| mriedem | +2 on the top one | 19:58 |
| *** erlon has joined #openstack-nova | 20:00 | |
| *** lbragstad has quit IRC | 20:03 | |
| *** erlon has quit IRC | 20:08 | |
| *** lbragstad has joined #openstack-nova | 20:18 | |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: Add tests for bug #1800511 https://review.openstack.org/614004 | 20:21 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: libvirt: Avoid setting MTU during live migration if unset https://review.openstack.org/614008 | 20:21 |
| openstack | bug 1800511 in OpenStack Compute (nova) "VMs with vif_type bridge/tap started before Rocky upgrade cannot be live migrated" [High,In progress] https://launchpad.net/bugs/1800511 - Assigned to Mohammed Naser (mnaser) | 20:21 |
| mnaser | mriedem: fixed your nit too if you feel like readding your +2 and cfriesen +1 :) | 20:21 |
| * mnaser goes to wander around | 20:21 | |
| mriedem | +Ws | 20:24 |
| mriedem | oops | 20:24 |
| mriedem | +2s | 20:24 |
| mriedem | time to play the "find another core" game | 20:24 |
| *** dpawlik has joined #openstack-nova | 20:28 | |
| *** ralonsoh has joined #openstack-nova | 20:31 | |
| *** imacdonn has quit IRC | 20:33 | |
| *** imacdonn has joined #openstack-nova | 20:33 | |
| *** eharney has joined #openstack-nova | 20:34 | |
| mnaser | mriedem: I guess all the IBM ones are busy today :p | 20:36 |
| mriedem | too soon | 20:38 |
| mnaser | It’s been quiet here today | 20:42 |
| mnaser | And it’s getting late so I might have to settle for an EU core tomorrow :p | 20:42 |
| melwitt | I can take a look. things have been a bit chaotic today | 20:45 |
| *** ralonsoh has quit IRC | 20:47 | |
| *** eharney has quit IRC | 20:48 | |
| mriedem | the one day sean-k-mooney isn't working 20 hours | 20:48 |
| mriedem | mnaser: you've deployed this patch now and verified it fixes your upgrade issue? | 20:49 |
| melwitt | do we want sean-k-mooney to test this first? | 20:49 |
| mriedem | i think mnaser is testing it in prod | 20:49 |
| mnaser | I have not tested in the environment to be honest | 20:49 |
| mnaser | I can do that now tho | 20:50 |
| mnaser | Gimme a few | 20:50 |
| dansmith | seems pretty clear from the bug that this is the right thing to do | 20:50 |
| melwitt | ok, asked in case that's why you mentioned sean-k-mooney | 20:50 |
| dansmith | obviously a manual test would be nice | 20:50 |
| dansmith | I dropped a +W on the top meaty one.. I can put an upgradeable +1 on the bottom one | 20:51 |
| *** tbachman has joined #openstack-nova | 20:55 | |
| mnaser | (taking a bit of time, trying to find VMs that have been launched before which are testing) | 21:03 |
| mnaser | dansmith: mriedem melwitt i've tested this one now locally on a vm that confirmed had no mtu set (virsh dumpxml, /proc/pid/cmdline had no host_mtu), live migrated with patch and migration completed succesfully | 21:14 |
| mriedem | i mentioned sean-k-mooney because he added the mtu stuff for bridge/tap types | 21:18 |
| mriedem | and is familiar with this live migration _update_vif_xml stuff | 21:19 |
| melwitt | ack | 21:20 |
| *** mvkr has joined #openstack-nova | 21:21 | |
| *** dpawlik has quit IRC | 21:22 | |
| *** dpawlik has joined #openstack-nova | 21:23 | |
| *** awaugama has quit IRC | 21:23 | |
| mriedem | while we're all here and having fun, we need to do a rocky release for several upgrade related issues, but need https://review.openstack.org/#/q/topic:bug/1798188+(status:open+OR+status:merged) as part of that, | 21:23 |
| mriedem | and will also want the fix that mnaser just pushed | 21:23 |
| mriedem | mnaser: are you working backports? | 21:24 |
| efried | mriedem: Allocation ratio fix will need to be part of --^ ? | 21:24 |
| mriedem | efried: i don't think that is critical | 21:24 |
| mriedem | annoying, yes | 21:24 |
| mriedem | but not critical | 21:24 |
| efried | okay | 21:24 |
| *** dpawlik_ has joined #openstack-nova | 21:26 | |
| *** dpawlik_ has quit IRC | 21:26 | |
| *** dpawlik has quit IRC | 21:27 | |
| melwitt | mriedem: I was debating how to update the reno as mentioned by mnaser on the review | 21:28 |
| melwitt | because once all compute hosts are rocky, should be OK to stop running nova-consoleauth with the workaround BUT you also have to make sure you deploy your console proxies per cell and you will still experience all consoles being reset that the point you stop using the workaround (if not all auths already expired) | 21:29 |
| melwitt | it's kind of complicated so I was feeling stuck on whether the reno is conveying all of that well enough | 21:29 |
| mriedem | well, it says use the workaround while you're doing rolling upgrade, so not all computes are rocky yet, as mohammed pointed out, | 21:31 |
| mriedem | and then there is the "Operators may unset the configuration option when:" section which says once you're upgraded (all hosts are Rocky), and stuff is expired, you can drop the workaround, | 21:31 |
| mriedem | if you need to add "and you're running console proxy per cell" to that list, then just add that | 21:32 |
| melwitt | yeah, ok. thanks. I've felt so mired in it that I was having a hard time thinking about it | 21:32 |
| mriedem | i left comments, | 21:32 |
| mriedem | but i think it's just (1) describing what rolling upgrade means and (2) add the part about console proxy being deployer per cell before unsetting the workaround | 21:33 |
| mriedem | we have checks for the former in the nova-status upgrade check, but not the latter, but i'm not sure if it's possible to check the latter via nova-status is it? | 21:34 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930 | 21:39 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Destination.cross_cell_move field https://review.openstack.org/614035 | 21:39 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036 | 21:39 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Change HostManager to allow scheduling to other cells https://review.openstack.org/614037 | 21:39 |
| melwitt | mriedem: I don't think so. the only possible way to check is if the nova.conf on the console proxy host has [database]connection configured (implying access to a cell database) | 21:39 |
| mnaser | mriedem: sorry checking things in and out, let me push up backport | 21:42 |
| *** dpawlik has joined #openstack-nova | 21:42 | |
| mnaser | I wonder if the UI adds the cherry picked from if it hasn’t merged yet | 21:42 |
| mriedem | it does not | 21:42 |
| mnaser | Boo. Alright | 21:42 |
| mnaser | This needs to go back to rocky or further? | 21:42 |
| *** mriedem is now known as mriedem_away | 21:42 | |
| mriedem_away | rocky | 21:43 |
| mnaser | ok cool | 21:43 |
| mriedem_away | _update_vif_xml was added in rocky, | 21:43 |
| mriedem_away | setting the mtu was backported to queens. if that could fail for different reasons, like live migrating a guest from source host which set mtu to something and then to a dest host that is not new enough to support that mtu config, well that's a different issue | 21:44 |
| mnaser | mriedem_away: most likely not, this feature was added a while back ago | 21:44 |
| mnaser | let me verify that | 21:44 |
| mriedem_away | libvirt 3.3 | 21:45 |
| mnaser | oh yes only that got support for it | 21:45 |
| mriedem_away | but our minimum libvirt support is 1.3.1 | 21:45 |
| mnaser | so unless someone is running 1.3.1 alongside 3.3.0 | 21:45 |
| mriedem_away | i think this might catch it anyway https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L235 | 21:46 |
| mriedem_away | alright really leaving now | 21:46 |
| *** itlinux has quit IRC | 21:46 | |
| openstackgerrit | Mohammed Naser proposed openstack/nova stable/rocky: Add tests for bug #1800511 https://review.openstack.org/614040 | 21:46 |
| openstack | bug 1800511 in OpenStack Compute (nova) "VMs with vif_type bridge/tap started before Rocky upgrade cannot be live migrated" [High,In progress] https://launchpad.net/bugs/1800511 - Assigned to Mohammed Naser (mnaser) | 21:46 |
| openstackgerrit | Mohammed Naser proposed openstack/nova stable/rocky: libvirt: Avoid setting MTU during live migration if unset https://review.openstack.org/614041 | 21:46 |
| mnaser | ^ thats for later | 21:46 |
| * mnaser & | 21:46 | |
| *** dpawlik has quit IRC | 21:46 | |
| *** tbachman has quit IRC | 21:52 | |
| *** pcaruana has quit IRC | 21:57 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Kill @safe_connect in _get_provider_traits https://review.openstack.org/613613 | 22:08 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Kill @safe_connect in _get_provider_traits https://review.openstack.org/613613 | 22:10 |
| *** spatel has quit IRC | 22:11 | |
| *** cdent has quit IRC | 22:13 | |
| *** mlavalle has quit IRC | 22:19 | |
| *** munimeha1 has quit IRC | 22:23 | |
| *** tbachman has joined #openstack-nova | 22:23 | |
| *** kevinbenton has quit IRC | 22:26 | |
| *** kevinbenton has joined #openstack-nova | 22:27 | |
| *** kevinbenton has quit IRC | 22:28 | |
| *** kevinbenton has joined #openstack-nova | 22:29 | |
| *** kevinbenton has quit IRC | 22:31 | |
| *** kevinbenton has joined #openstack-nova | 22:31 | |
| *** kevinbenton has quit IRC | 22:32 | |
| *** kevinbenton has joined #openstack-nova | 22:34 | |
| *** eandersson has joined #openstack-nova | 22:36 | |
| *** kevinbenton has joined #openstack-nova | 22:43 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Add rolling upgrade info to enable_consoleauth workaround reno https://review.openstack.org/613174 | 23:11 |
| melwitt | mnaser: updated to address your comments ^ let me know if there's anything else unclear in the message | 23:12 |
| openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Add rolling upgrade info to enable_consoleauth workaround reno https://review.openstack.org/613174 | 23:14 |
| *** spatel has joined #openstack-nova | 23:24 | |
| *** spatel has quit IRC | 23:29 | |
| *** takashin has joined #openstack-nova | 23:37 | |
| *** itlinux has joined #openstack-nova | 23:37 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!