Thursday, 2020-07-16

« Wednesday, 2020-07-15 Index Friday, 2020-07-17 »
*** rcernin has quit IRC00:06
*** rcernin has joined #openstack-nova00:13
*** tetsuro has joined #openstack-nova00:35
*** tetsuro_ has joined #openstack-nova00:36
*** rouk has joined #openstack-nova00:39
*** tetsuro has quit IRC00:40
rouki cant seem to make nova-serialproxy work correctly past stein, on train it logs that its connecting to the backend, but my client gets punted. with "remote host closed connection: Connection is already closed"00:42
roukdont see any bugreports on it, and the logs are pretty silent, can anyone verify that it works for them on train?00:43
*** artom has quit IRC00:57
openstackgerritmelanie witt proposed openstack/nova master: Use unittest.mock instead of mock in test_websocketproxy  https://review.opendev.org/74135101:02
*** jangutter_ has quit IRC01:03
*** brinzhang has joined #openstack-nova01:09
*** Liang__ has joined #openstack-nova01:17
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.PY2 and six.PY3  https://review.opendev.org/72651201:45
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.add_metaclass  https://review.opendev.org/72656501:45
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.reraise  https://review.opendev.org/72689801:45
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.moves  https://review.opendev.org/72722401:53
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys  https://review.opendev.org/72775701:53
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte  https://review.opendev.org/72777701:54
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types  https://review.opendev.org/72809401:54
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.text_type (1/2)  https://review.opendev.org/72810901:54
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.text_type (2/2)  https://review.opendev.org/72811701:54
*** tkajinam has quit IRC01:55
*** tkajinam has joined #openstack-nova01:55
*** mkrai has joined #openstack-nova02:19
*** mkrai has quit IRC02:33
*** sapd__x has joined #openstack-nova02:39
*** Yumeng has joined #openstack-nova02:46
*** ociuhandu has joined #openstack-nova02:46
*** ociuhandu has quit IRC02:50
*** songwenping__ has joined #openstack-nova02:58
*** songwenping__ has quit IRC02:59
*** tetsuro_ has quit IRC03:02
*** tetsuro has joined #openstack-nova03:03
*** mkrai has joined #openstack-nova03:04
*** tetsuro has quit IRC03:07
*** dave-mccowan has quit IRC03:30
*** psachin has joined #openstack-nova03:39
*** brinzhang_ has joined #openstack-nova03:43
*** brinzhang has quit IRC03:47
*** brinzhang0 has joined #openstack-nova04:14
*** gyee has quit IRC04:15
*** brinzhang_ has quit IRC04:16
*** brinzhang_ has joined #openstack-nova04:19
*** brinzhang0 has quit IRC04:22
*** yaawang has quit IRC04:25
*** yaawang has joined #openstack-nova04:26
*** tetsuro has joined #openstack-nova04:39
*** tetsuro has quit IRC04:39
*** udesale has joined #openstack-nova04:57
*** tetsuro has joined #openstack-nova05:00
*** ratailor has joined #openstack-nova05:19
*** yaawang has quit IRC05:26
*** yaawang has joined #openstack-nova05:27
*** links has joined #openstack-nova05:29
*** tetsuro has quit IRC05:30
*** jsuchome has joined #openstack-nova05:46
*** tetsuro has joined #openstack-nova06:08
*** tetsuro has quit IRC06:25
*** tetsuro has joined #openstack-nova06:25
*** tetsuro has quit IRC06:27
*** dklyle has quit IRC06:36
*** tetsuro has joined #openstack-nova06:42
*** mkrai has quit IRC06:43
*** mkrai_ has joined #openstack-nova06:43
gibijsuchome: will check06:45
* gibi needed to take the day off yesterday on short notice06:45
*** tetsuro has quit IRC06:47
*** tetsuro has joined #openstack-nova06:48
*** tetsuro has quit IRC06:52
*** mkrai_ has quit IRC06:57
*** maciejjozefczyk has joined #openstack-nova07:01
*** ociuhandu has joined #openstack-nova07:02
*** tesseract has joined #openstack-nova07:13
openstackgerritHuaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs  https://review.opendev.org/68893607:17
openstackgerritHuaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance  https://review.opendev.org/73699707:17
openstackgerritHuaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance  https://review.opendev.org/74139507:17
*** udesale has quit IRC07:26
*** udesale has joined #openstack-nova07:31
*** yaawang has quit IRC07:32
*** yaawang has joined #openstack-nova07:33
*** ralonsoh has joined #openstack-nova07:33
*** ociuhandu has quit IRC07:35
openstackgerritBrin Zhang proposed openstack/nova-specs master: migrate testing to ubuntu focal  https://review.opendev.org/74086307:45
*** rcernin has quit IRC07:48
*** yaawang has quit IRC08:02
*** tosky has joined #openstack-nova08:02
*** yaawang has joined #openstack-nova08:03
*** k_mouza has joined #openstack-nova08:04
*** jangutter has joined #openstack-nova08:11
*** bnemec has quit IRC08:14
*** rcernin has joined #openstack-nova08:14
*** tosky has quit IRC08:16
*** bnemec has joined #openstack-nova08:17
*** martinkennelly has joined #openstack-nova08:20
*** happyhemant has joined #openstack-nova08:23
*** mkrai has joined #openstack-nova08:24
*** avolkov has joined #openstack-nova08:32
*** tosky has joined #openstack-nova08:33
*** xek has joined #openstack-nova08:38
bauzasgood morning Nova08:39
* bauzas pops his head up08:39
bauzas(after a few PTOs)08:40
aarentsgood morning08:43
gibi 08:44
gibibauzas, aarents o/08:44
huaqianggibi: morning08:46
*** nightmare_unreal has joined #openstack-nova08:47
huaqiangcan you review https://review.opendev.org/728480, stephen and alex mentioned that it's better to let you have a look.08:47
gibihuaqiang: ack, I will check08:48
huaqiangthanks!08:48
*** derekh has joined #openstack-nova08:53
aarentsbauzas: FYI, I updated https://etherpad.opendev.org/p/nova-libvirt-subteam, I re-propose something to fix this bug: https://bugs.launchpad.net/nova/+bug/1732428 here: https://review.opendev.org/#/c/696084/3 I implemented rebase of backing file suggested by lyarwood kashyap08:54
openstackLaunchpad bug 1732428 in OpenStack Compute (nova) "Unshelving a VM breaks instance metadata when using qcow2 backed images" [Medium,In progress] - Assigned to Alexandre arents (aarents)08:54
kashyapaarents: Hi08:54
bauzasaarents: ack, will look08:55
aarentskashyap: Hi08:55
aarentsbauzas: thks08:55
kashyapThx for the reminder; been buried w/ other stuff.  WIll look08:55
aarentskashyap: ok cool08:56
gibihuaqiang: approved the patch08:57
huaqianggibi: cool, appreciate!08:57
gibihuaqiang, stephenfin, alex_xu: extended the runway slot of bp/use-pcpu-and-vcpu-in-one-instance with an additional week as I see good progress there09:01
*** xek has quit IRC09:01
huaqianggibi: nice to hear this!09:03
gibinice to see the progress :)09:04
*** rcernin has quit IRC09:07
*** dtantsur|afk is now known as dtantsur09:15
*** ociuhandu has joined #openstack-nova09:35
*** ociuhandu has quit IRC09:40
stephenfinbauzas: I'm working on https://review.opendev.org/#/c/741282/ and noticed a potential bug in one of the vGPU functional tests09:42
bauzasstephenfin: a-ha ?09:43
stephenfinsec, getting the links09:43
bauzasyeah, i tried to provide some methods for the func tests09:43
stephenfinbauzas: very small one09:43
stephenfinthese tests use v2.1 + latest microversion https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_vgpu.py#L132-L13709:44
stephenfinthese tests do not https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_vgpu.py#L30009:44
stephenfinboth are inheriting from nova.tests.functional.test_servers.ServersTestBase https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_vgpu.py#L4109:44
stephenfinwhich uses API version 2 (.0) by default09:44
stephenfinany reason those later tests shouldn't be using v2.1 + latest microversion too?09:45
stephenfinyou need the latest microversion so you can request a host https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_vgpu.py#L33009:45
stephenfinwell, a reasonably new one09:45
bauzasstephenfin: yeah, we need a specific microversion for at least one test AFAICU09:46
bauzasstephenfin: at least for the resize tests : https://github.com/openstack/nova/commit/d2e0afc1f72db5cb56ed987e2873798fc1e8954509:47
stephenfinbauzas: okay, so I'm going to move this https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_vgpu.py#L134-L13709:49
stephenfinto the base class https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_vgpu.py#L4209:49
stephenfindoesn't sound like you wanted to use the legact API v2 for the VGPUMultipleTypesTests tests09:49
bauzasstephenfin: I just wanted to provide some tests for vGPU by using the v2 API, but then when I wanted to provide the resize ones, it needed the microversion09:50
bauzasstephenfin: so, no worries if you want to use the latest API for all of them, it should work09:50
stephenfinokay, sweet. Thanks09:51
*** k_mouza has quit IRC09:51
bauzasthe main issue I had with the tests was the fact that I needed to move some methods to the InstanceHelperMixin one09:52
stephenfinYou'll probably be interested in https://review.opendev.org/#/c/741283/1 so09:52
bauzasstephenfin: hence the https://github.com/openstack/nova/blob/d2e0afc1f72db5cb56ed987e2873798fc1e89545/nova/tests/functional/integrated_helpers.py#L53809:53
stephenfinbauzas: You're *definitely* interested in the above so09:53
bauzasstephenfin: :)09:55
*** k_mouza has joined #openstack-nova09:57
*** jangutter_ has joined #openstack-nova10:00
*** jangutter has quit IRC10:02
*** xek has joined #openstack-nova10:06
*** Liang__ has quit IRC10:12
*** martinkennelly has quit IRC10:13
*** rcernin has joined #openstack-nova10:15
*** rcernin has quit IRC10:30
openstackgerritMerged openstack/nova master: notifications: add the pcpuset info to instance numa cell payload  https://review.opendev.org/72848011:06
openstackgerritMerged openstack/nova master: objects: Introduce the 'CPUAllocationPolicy.MIXED' enum  https://review.opendev.org/71626711:06
*** mgariepy has quit IRC11:10
*** Yumeng has quit IRC11:16
*** xiaolin has joined #openstack-nova11:18
openstackgerritMerged openstack/nova master: hardware: refactor the code for CPU policy sanity check  https://review.opendev.org/74092511:18
lyarwoodstephenfin / gibi ; https://review.opendev.org/#/c/699291/ - Would you mind taking a look at this fix from mriedem that I've been trying to push over the line for a while?11:21
* lyarwood isn't going to vote on it given the small changes he made11:21
stephenfinsure11:22
lyarwoodthanks, I'll get back to your series later today btw11:24
gibilyarwood: I will check it11:26
*** sapd__x has quit IRC11:28
*** JamesBenson has joined #openstack-nova11:36
*** xek has quit IRC11:47
openstackgerritBalazs Gibizer proposed openstack/nova master: Regenerate device metadata during interface detach  https://review.opendev.org/74143611:56
*** raildo has joined #openstack-nova11:59
openstackgerritBalazs Gibizer proposed openstack/nova master: [WIP] Support SRIOV interface attach and detach  https://review.opendev.org/74099512:10
stephenfingibi, sean-k-mooney: Do either of you know why we do this in the resize operation instead of accessing instance.image_meta? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3886-L388712:10
stephenfinI suspect it's something to do with filtering out undesirable image metadata properties but I'm not sure12:11
sean-k-mooneylet me check12:11
*** takamatsu has joined #openstack-nova12:12
sean-k-mooneystephenfin: we are using the cached metadta12:12
sean-k-mooneywe store the image metadta in the system metadata table12:12
*** happyhemant has quit IRC12:12
sean-k-mooneyif we looked up the image by its uuid and got the metadata it could have change or the image could have been deleted12:13
sean-k-mooneystephenfin: we only have the image uuid in the instance https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L12712:15
sean-k-mooneythre is no image_meta12:15
sean-k-mooneybut if you want to add a property that just calls that function i dont have any issue with that12:16
*** mgariepy has joined #openstack-nova12:17
stephenfinHmm, I think they may be equivalent. 'Instance.image_meta' calls 'ImageMeta.from_instance' https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L250-L25112:17
sean-k-mooneyoh we do have a property https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L12712:18
stephenfinwhich calls 'get_image_from_system_metadata' https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L12712:18
stephenfinhmm, those sound equivalent so. Probably legacy stuff12:18
sean-k-mooneyyes https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/objects/image_meta.py#L126-L12812:18
gibiseem equivalent, yes12:18
* gibi is late to the game12:18
stephenfinSo my series is gaining yet another tech debt reduction patch \o/12:18
stephenfinevery. damn. time.12:18
sean-k-mooneyyou could pretend you did not see it :P12:19
sean-k-mooneyalso this seams backwards to me12:19
sean-k-mooneyi would have had instance.image_meta just  call  utils.get_image_from_system_metadata(sysmeta)12:20
gibistephenfin: you are a hero removing those depts12:20
gibidebts12:20
sean-k-mooneyand had image_meta.from_instance(instance) return instance.image_meta12:20
stephenfinfair point12:21
stephenfinmuch of a muchness12:21
*** maciejjozefczyk has quit IRC12:21
stephenfinworth noting the 'ImageMeta.from_instance' also handles multiple different ways of storing system_metadata in the instance object12:21
sean-k-mooneythere are extra db queries curently12:21
stephenfinif indeed instance is an object12:21
*** rcernin has joined #openstack-nova12:21
*** maciejjozefczyk has joined #openstack-nova12:21
stephenfinI've no idea if that call to 'utils.instance_sys_meta' is necessary, vs. just using 'instance.system_metadata'12:22
sean-k-mooneyit should not bey system_metadata is always loaded by default12:22
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/objects/instance.py#L70-L7112:23
stephenfinAn it's a 'DictOfNullableStringsField' type field12:23
stephenfinso it couldn't be anything *except* a dict12:23
stephenfinso this check seems like garbage https://github.com/openstack/nova/blob/master/nova/utils.py#L547-L54912:23
sean-k-mooneyya it might be empty but it will be a dict12:23
stephenfinyup, undefined or a dict. Nothing else since it's not nullable12:24
* stephenfin marks that down for the chopping block too12:24
sean-k-mooneywell i dont know it might be there for some reason in the past12:24
stephenfinpre-o.vo, most likely12:25
sean-k-mooneyhttps://github.com/openstack/nova/commit/96164cecc6b7bf447d06896f723dab416e63ee0612:25
openstackgerritMerged openstack/nova stable/rocky: Reject boot request for unsupported images  https://review.opendev.org/70860812:25
sean-k-mooneyso ya it was part of the ovo migration12:26
sean-k-mooneyits proably not needed 7 years later.12:27
*** sapd__x has joined #openstack-nova12:28
*** ratailor has quit IRC12:30
*** xek has joined #openstack-nova12:53
*** xek_ has joined #openstack-nova12:55
*** k_mouza has quit IRC12:57
*** xek has quit IRC12:58
*** lbragstad has quit IRC12:58
*** lbragstad has joined #openstack-nova13:01
*** spatel has joined #openstack-nova13:04
*** ociuhandu has joined #openstack-nova13:15
*** artom has joined #openstack-nova13:23
*** k_mouza has joined #openstack-nova13:24
*** dave-mccowan has joined #openstack-nova13:25
*** rcernin has quit IRC13:31
*** sapd__x has quit IRC13:34
artomgibi, how did you notice https://bugs.launchpad.net/nova/+bug/1887799 ?13:35
openstackLaunchpad bug 1887799 in OpenStack Compute (nova) "interface metadata is not regenerated after interface detach" [Low,In progress] - Assigned to Balazs Gibizer (balazs-gibizer)13:35
artomCode inspection, or "real life"?13:36
artomBecause... we *do* update device metadata on restart13:36
artomJust no in the driver13:36
artomLemme comment on the patch13:36
gibiartom: working on the SRIOV interface attach feature and during detach I see that there is metadata left in the database13:36
gibithen I inspected the code13:36
artomgibi, something else is going on then: https://opendev.org/openstack/nova/src/branch/master/nova/network/neutron.py#L175513:37
sean-k-mooneygibi: your actully workign on that13:37
sean-k-mooneygibi: do you have any patches up13:38
gibisean-k-mooney: I've spent two day on it and I have a WIP patch, now writin a small spec13:38
sean-k-mooneygibi: cool13:38
gibisean-k-mooney: https://review.opendev.org/#/c/740995/13:38
gibisean-k-mooney: it works in devstack already13:38
artomgibi, shot in the dark, but maybe your bug is related to https://review.opendev.org/#/c/670593/ ?13:39
sean-k-mooneythe code is more or less already there in regards to actully doing the attach. all that was missing was the claim which wew can do form the api using the new rpc calls added for live migration13:39
gibiartom: thanks for the pointers I wil dig13:39
gibisean-k-mooney: yes, I needed to put together the pieces13:40
gibibut most of the pieces exist13:40
sean-k-mooneyyep we say that when we filed the bug a few year ago but it was too close to code freeze to get it in13:40
sean-k-mooneyit was deemed to need a spec and we were past m213:40
sean-k-mooneyso the curent api check was added but we never went and fixed it. that was pre sriov migration so it was more work then13:41
jsuchomegibi: thanks for your +2, what/who do we need for starting the workflow?13:42
sean-k-mooneygibi: ill try an test out your change today or tomorow im currently working on the fucntial test for my pci device lookup patches but ill test yours when i test that on real hardware13:43
*** yaawang has quit IRC13:49
*** yaawang has joined #openstack-nova13:50
gibijsuchome: you need one more +2 . But I also suggest that pull this back to the original series so that the reviewers of the original series will see it and review it14:07
gibisean-k-mooney: thanks. any test is useful14:08
gibisean-k-mooney: I now have access with a physical machine with SRIOV card14:08
jsuchomelyarwood: hi, could you please take a look at https://review.opendev.org/#/c/738738/ ? that's a new prerequisite to direct-rbd-download to make things more readable14:13
lyarwoodjsuchome: ack queued, might not get to it today however14:14
jsuchomegreat, thanks!14:14
*** martinkennelly has joined #openstack-nova14:17
*** mkrai has quit IRC14:21
*** links has quit IRC14:32
*** martinkennelly has quit IRC14:32
huaqianghi, I have a question, do we support specify such flavor and image metadata  conbination：vcpus=4, hw:cpu_realtime=yes, hw_cpu_policy=dedicated , hw_cpu_realtime_mask=^014:34
huaqiangI don't the general rule using extra_specs with image properties14:35
openstackgerritArtom Lifshitz proposed openstack/nova master: Add regression test for bug 1879787  https://review.opendev.org/74123014:36
openstackbug 1879787 in OpenStack Compute (nova) "post_live_migration does not handle Neutron errors" [Medium,In progress] https://launchpad.net/bugs/1879787 - Assigned to Artom Lifshitz (notartom)14:36
openstackgerritArtom Lifshitz proposed openstack/nova master: Handle Neutron errors in _post_live_migration()  https://review.opendev.org/72976314:36
huaqiangs/"I don't"/"I don't find'"/14:37
sean-k-mooneygibi: sorry was on a call but ya having sriov capably hardware is both a blessing and a curse14:44
sean-k-mooneyvcpus=4 is not a valid extraspc14:45
sean-k-mooneybut yes the rest would be valid14:45
sean-k-mooneywe can only assert that the realtime requireemtn for a mask are fullfiled when we have both the flavor and image14:46
sean-k-mooneyso we cant use the current flavor validation feature to asset it its correct14:46
*** dklyle has joined #openstack-nova14:46
sean-k-mooneyand if you use the flavor with hw:cpu_realtime=yes14:47
sean-k-mooneywith a flavor that does not declare the cpu policy and mask it will be an error14:47
*** tkajinam has quit IRC14:47
sean-k-mooneyhuaqiang: https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/compute/api.py#L598 is where the requiremtn will be checked14:49
sean-k-mooneyfor realtime it validated here https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/hardware.py#L171914:50
*** mkrai has joined #openstack-nova14:51
huaqiangsean-k-mooney: I think the combination I listed is allowed by code14:52
huaqiangbut looks ugly14:52
huaqianghw_cpu_policy=dedicated , hw_cpu_realtime_mask=^014:52
huaqiangthe cpu policy and realtime mask are set in iamge14:53
huaqiangbut realtime switch is in flavor :hw:cpu_realtime=yes14:53
openstackgerritAndrew Bonney proposed openstack/nova master: Fix misleading documentation for live_migration_inbound_addr  https://review.opendev.org/74147314:54
sean-k-mooneyyes it is allow14:54
sean-k-mooneyand yes we would generall recommend that you set all 3 in the falvor14:54
sean-k-mooneyand then ovrride the mask in the image if needed14:54
bauzasgibi: I won't be around tonight for the team meeting14:54
bauzasgibi: nothing to report but aarents's ask for reviews in https://etherpad.opendev.org/p/nova-libvirt-subteam14:55
*** sapd__x has joined #openstack-nova14:55
bauzas(for the libvirt subteam)14:55
bauzasgibi: also, we are now 2 weeks close to the Victoria-2 milestone and we only have 10 approved specs14:55
*** derekh has quit IRC14:55
*** devfaz has quit IRC14:55
*** hoonetorg has quit IRC14:55
*** irclogbot_3 has quit IRC14:55
*** gibi has quit IRC14:55
bauzasah shit, netsplit14:56
*** derekh has joined #openstack-nova14:57
*** devfaz has joined #openstack-nova14:57
*** hoonetorg has joined #openstack-nova14:57
*** irclogbot_3 has joined #openstack-nova14:57
*** gibi has joined #openstack-nova14:57
huaqiangsean-k-mooney: That's recommendation, but code must cope with any corner case14:58
*** mloza has joined #openstack-nova14:58
sean-k-mooneyhuaqiang: correct because we wanted to suppot both explcitly in the spec14:59
sean-k-mooneythis is working as desigined14:59
sean-k-mooneywe do handel the corner cases in the code already14:59
huaqiangclear now. Thanks14:59
huaqiangI need to deal with another such case in rebuild15:00
sean-k-mooneyah no15:00
sean-k-mooneyin rebuild this shoudl not change15:01
huaqiangrebuild from realtime-dedicated instance to realtime-mixed instance15:01
sean-k-mooneythere should already be a check that compares the numa toplogy objects with the flavor and old image vs flavor and new image15:01
sean-k-mooneyhuaqiang: that is not allowed15:01
sean-k-mooneyrebuild are not allowed to change resouce allocattions15:02
sean-k-mooneygoing form dedicated to mixed changes that15:02
sean-k-mooneythe current cod ei added for numa rebuidl should block that15:02
bauzasgibi: so, you're back from the netsplit15:02
bauzasretelling my story15:03
bauzasgibi: I won't be around tonight for the team meeting15:03
bauzasgibi: nothing to report but aarents's ask for reviews in https://etherpad.opendev.org/p/nova-libvirt-subteam15:03
bauzas(for the libvirt subteam)15:03
sean-k-mooneyhuaqiang: https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/compute/api.py#L3538-L357815:03
bauzasgibi: also, we are now 2 weeks close to the Victoria-2 milestone and we only have 10 approved specs15:03
bauzasgibi: should we do a Spec review day ?15:03
*** spatel has quit IRC15:04
gibiohh I was away?15:04
gibibauzas: do you see open specs that needs our focus?15:04
gibiif there are open specs that are ready for review then sure we can have a spec review day15:04
* gibi is on the community call for the 10th birthday of openstack15:05
bauzasgibi: I have a pretty short specs dashboard15:05
sean-k-mooneyhuaqiang: i have functional test that assert the correct behavior here https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_numa_servers.py#L105115:06
sean-k-mooneyi am not modifying cpu policy but that shoudl also be blocked by that code15:06
sean-k-mooneyit should be trivial to add a functional test to assert that it does not change15:07
sean-k-mooneyor rather that an attepmt to rebuild form dedicated to mixed will be rejected15:08
gibibauzas: I will ask the others on the meeting about a spec review day15:08
gibibauzas: next week is good for you?15:08
bauzasyup15:08
gibicool15:09
bauzasanyway, it's just a question, since we don't have a lot of open specs15:09
bauzashttps://review.opendev.org/#/q/project:openstack/nova-specs+is:open15:09
gibiat most it won't be a super busy day :)15:09
bauzasthis said, I'll create a new spec for the reshape tool15:09
sean-k-mooneyits proably good to do before m2 in either case15:09
gibibauzas: and I will push one for SRIOV attach15:09
bauzasack, all good15:10
gibi:)15:10
bauzasgibi: I'd also write a spec for GPU NUMA affinity using the NUMA topology in placement, but it's more for discussing about it this cycle, not wanting to merge it15:10
sean-k-mooneygibi: im proably going to try an add support for sriov PF detach as a bug fix this cycle too15:11
sean-k-mooneyim not sure if it would make sense to add it to your spec or not15:11
huaqiangsean-k-mooney: I need time to digest your code.15:12
sean-k-mooneygibi: when vnic_type=direct-physical was added detach was never updated15:12
sean-k-mooneyit tries to find the nic by its mac but we dont have a mac in this case15:12
gibisean-k-mooney: before you go into that check my detach code if that solves your case too ro not15:12
gibibauzas: ack15:13
sean-k-mooneygibi: it might we just need to fine the device by its pci address not it mac15:13
sean-k-mooneysince we use a <hostdev> element instead of <interface type=hostdev> for pfs15:14
sean-k-mooneysriov detach works for vnic_type=direct|macvtap|virtio-forwarder today just not direct-physical15:15
gibisean-k-mooney: works and removes the pci claim too?15:15
*** johnthetubaguy has quit IRC15:16
sean-k-mooneygibi: ill admit i cant rememebr i think so but we dont currently block it because it complete succefully15:16
sean-k-mooneyso its updateing the neuton port and removing the port form the virtual interfaces table15:16
gibisean-k-mooney: I had to add https://review.opendev.org/#/c/740995/2/nova/compute/manager.py@7416 to make the pci device freed15:17
sean-k-mooneyi belive it free the device but it might be "working" but not doing the right thing15:17
gibibut it can be that some periodic would free that15:17
dansmithmelwitt: the last dependency for this is about to merge in the gate.. wanna send it? https://review.opendev.org/#/c/734184/15:17
sean-k-mooneygibi: ya i think the periodic is what we rely on15:17
sean-k-mooneydoing it premtivly is better but i think we get back into a consitent state15:18
sean-k-mooneyill test it before i apply your patch and let you know15:18
gibisean-k-mooney: cool15:20
*** johnthetubaguy has joined #openstack-nova15:20
*** hamalq has joined #openstack-nova15:22
*** maciejjozefczyk has quit IRC15:23
*** hamalq_ has joined #openstack-nova15:23
*** jobewan has quit IRC15:24
*** gyee has joined #openstack-nova15:25
sean-k-mooneygibi: by the way if i confim that we did not free the pci device do we want to backport a fix to block detach the same way we backported a patch to block attach for older releases?15:25
sean-k-mooneyi.e. if the periodic does not fix it15:26
*** hamalq has quit IRC15:27
*** dave-mccowan has quit IRC15:28
gibisean-k-mooney: I'm not sure. If people using it then blocking it would be bad15:29
sean-k-mooneyit cloud be done vai a conig option i guess like numa live migration15:30
sean-k-mooneywe can cross that bridge when we come to it15:30
gibiyeah15:30
*** mlavalle has joined #openstack-nova15:30
*** udesale_ has joined #openstack-nova15:32
*** udesale has quit IRC15:34
gibinova meeting starts in 15 minutes on #openstack-meeting-315:45
*** martinkennelly has joined #openstack-nova15:46
*** xek_ has quit IRC15:47
*** sapd__x has quit IRC16:05
*** jsuchome has quit IRC16:06
*** psachin has quit IRC16:12
*** johnthetubaguy has quit IRC16:17
*** johnthetubaguy has joined #openstack-nova16:20
*** sapd__x has joined #openstack-nova16:24
*** ociuhandu_ has joined #openstack-nova16:24
*** udesale_ has quit IRC16:25
*** ociuhandu has quit IRC16:27
*** ociuhandu_ has quit IRC16:28
artomHey, if we set a bug as incomplete, can the reporter then switch it back once they come back with new logs?16:30
artomMy thinking is: to avoid other people looking at it needlessly, set it as incomplete, and let the reporter come back with the info. Is that how we do i?16:31
artomit16:31
gibiartom: they can switch it back16:31
gibijust ask them to set it back to New16:31
openstackgerritElod Illes proposed openstack/nova stable/rocky: Check cherry-pick hashes in pep8 tox target  https://review.opendev.org/74149216:32
*** JamesBen_ has joined #openstack-nova16:37
*** JamesBenson has quit IRC16:40
*** dtantsur is now known as dtantsur|afk16:40
artomIs (anti-)affinity in placement a thing we track?16:44
sean-k-mooneyartom: no16:44
artomContext: https://bugs.launchpad.net/nova/+bug/1886160 looks like ye olde "the late anti-affinity check on the host is super racy" problem16:45
openstackLaunchpad bug 1886160 in OpenStack Compute (nova) "Anti-affinity instance creation failed and the scheduling node was incorrect." [Undecided,New]16:45
artomSo I'm looking for something to tell them other than "tough luck"16:45
sean-k-mooneyhow about dont use multi create if you also use numa or sriov16:45
*** mgariepy has quit IRC16:45
sean-k-mooneythe fact we now precompute a series of alternate host insead of rescudling on retry actully makes anti affinity worse16:46
artomRight, we're < 30 untriaged, and I need lunch16:53
*** martinkennelly has quit IRC16:55
*** derekh has quit IRC17:00
*** martinkennelly has joined #openstack-nova17:00
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Simplify '_create_domain' function  https://review.opendev.org/73920817:04
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add vTPM config support  https://review.opendev.org/73920917:04
openstackgerritStephen Finucane proposed openstack/nova master: scheduler: Request vTPM trait based on flavor or image  https://review.opendev.org/73921017:04
openstackgerritStephen Finucane proposed openstack/nova master: crypto: Add support for creating, destroying vTPM secrets  https://review.opendev.org/73921117:04
openstackgerritStephen Finucane proposed openstack/nova master: manager: Prevent compute startup on invalid vTPM config  https://review.opendev.org/73921217:04
openstackgerritStephen Finucane proposed openstack/nova master: tests: Rename tests for '_create_guest_with_network'  https://review.opendev.org/74046417:04
openstackgerritStephen Finucane proposed openstack/nova master: tests: Move single use constants to their callers  https://review.opendev.org/74128017:04
openstackgerritStephen Finucane proposed openstack/nova master: tests: Define constants in '_IntegratedTestBase'  https://review.opendev.org/74128117:04
openstackgerritStephen Finucane proposed openstack/nova master: tests: Remove 'test_servers.ServersTestBase'  https://review.opendev.org/74128217:04
openstackgerritStephen Finucane proposed openstack/nova master: tests: Add 'PlacementHelperMixin', 'PlacementInstanceHelperMixin'  https://review.opendev.org/74128317:04
openstackgerritStephen Finucane proposed openstack/nova master: tests: Make '_IntegratedTestBase' subclass 'PlacementInstanceHelperMixin'  https://review.opendev.org/74128417:04
openstackgerritStephen Finucane proposed openstack/nova master: tests: Add helpers for suspend, resume and reboot of server  https://review.opendev.org/74128517:04
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Pass context, instance to '_create_domain'  https://review.opendev.org/74128617:04
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add emulated TPM support to Nova  https://review.opendev.org/63136317:04
openstackgerritStephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files  https://review.opendev.org/63993417:04
openstackgerritStephen Finucane proposed openstack/nova master: docs: Add docs for vTPM support  https://review.opendev.org/73921317:04
openstackgerritStephen Finucane proposed openstack/nova master: api: Reject non-spawn operations for vTPM  https://review.opendev.org/74150017:04
*** k_mouza has quit IRC17:16
*** k_mouza has joined #openstack-nova17:25
*** mgariepy has joined #openstack-nova17:26
*** ralonsoh has quit IRC17:32
*** k_mouza has quit IRC17:34
openstackgerritsean mooney proposed openstack/nova master: lookup nic feature by pci address  https://review.opendev.org/73913117:38
*** nightmare_unreal has quit IRC17:39
*** mriedem has joined #openstack-nova18:03
*** sapd__x has quit IRC18:11
*** mkrai has quit IRC18:13
*** tesseract has quit IRC18:41
*** avolkov has quit IRC18:41
sean-k-mooneygibi:  some comments in the patch. im not sure the interface you are using for the new fucntions is correct18:44
sean-k-mooneyoverall the logic seam pretty sound but its not quite how i would do it18:44
sean-k-mooneygibi: ya so the perodic task does not free the claim and they are not freed by the detach without your patch19:03
sean-k-mooneygibi: deleting the instance does19:03
sean-k-mooneyso right now if you detatch an sriov interface it leaks the pci device until the vm is deleted19:04
sean-k-mooneyim going to check if we fix it with live migrations or cold migrations19:04
sean-k-mooneyyes live migration frees the claims19:07
sean-k-mooneycold migration does not fix the issue an result in claiming a vf on the dest host that is unused19:11
sean-k-mooneyshevle and unshelve also claims an extra vf19:11
sean-k-mooneygibi: so it look like anything that uses move claims does not correct it but sriov live migration which intentionally does not use move claims and caluatles the required pci device by looking at the attached interfaces does correct the issue19:13
*** nicolasbock has quit IRC19:14
sean-k-mooneyok now that i have tested my patch ill test your and see if it fixes it.19:14
*** nicolasbock has joined #openstack-nova19:15
openstackgerritTobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration  https://review.opendev.org/74152919:33
sean-k-mooneytobias-urdin: ^ that is not really a fix for the issue its just going to mask the race condition19:51
sean-k-mooneyi left a comment inline but you need to use the python bindings not virsh19:55
*** vesper11 has quit IRC20:20
*** vesper11 has joined #openstack-nova20:21
mnaserhrm20:51
melwittmnaser: did you see my reply to you yesterday?20:51
mnasermelwitt: oh yes, sorry, i mentally said "yes"20:52
melwittnp, just wanted to make sure :)20:52
* mnaser has been knee-deep in a lot of issues20:52
mnasermy fun thing now is a vm that takes 725 seconds to provision20:52
melwittyikes20:52
mnaserand the only strange thing is like20:52
mnaserCreating image .... nothing from there until VM started20:52
mnaser12 minutes until it starts20:53
mnaserand here's the thing.. the image it uses?  it's 1 megabyte.  and it's cached locally (verified by sha1sum the image uuid)20:53
mnaserhttps://www.irccloud.com/pastebin/s9lKj7n7/20:53
mnaser(that's grepping the nova-compute log for that specific vm)20:54
melwitthm, I've seen that hang before in cases where there was something like an environmental error where there was no OSD under the root20:54
melwittthe 'rbd import' command can hang silently and I have not yet figured out a good way to time it out or anything20:55
melwittbut in your case it's not stuck, it just takes forever20:55
mnasermelwitt: well and also, sorry i should clarify, its not even using ceph20:56
mnaserso local storage, so should be trivial?20:56
melwittshould be...20:56
mnaserand afaik the port plugging should happen after the vm get started (then resumed when the callback arrives)20:56
mnaserwhich means it isn't waiting on neutron20:57
*** xek_ has joined #openstack-nova20:57
mnaserbut this is a case where like a single compute node is starting up 15 vms with 5 ports each potentially20:58
*** raildo has quit IRC21:00
melwittmnaser: looks like the "Instance spawned successfully." is after port plugging and everything, it's after it's 100% done21:05
mnasermelwitt: i thought the pause/resume on start that happens was for networking to get plumbed?21:06
melwittif you're able to turn on DEBUG, you could look for this log marker, this marks the end of the creating image piece https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/libvirt/driver.py#L4070-L407321:06
openstackgerritMerged openstack/nova master: libvirt: Simplify '_create_domain' function  https://review.opendev.org/73920821:07
mnaseri think i might have to switch to debug21:07
melwittyeah I'm still looking re: what the pause/resume happens. I'm not familiar with that other than knowing it happens21:07
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Convert libvirt_volume_drivers to a dict  https://review.opendev.org/74154521:08
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Reintroduce native QEMU iSCSI volume support  https://review.opendev.org/74154621:08
melwittyeah looks like resume is after plugging https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/libvirt/driver.py#L655421:08
mnaserso technically its not waiting for neutron..21:08
melwittit is, right? that message is the resume, after plugging no?21:09
melwitt(Based on seeing this before the resume https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/libvirt/driver.py#L6513)21:09
mnasermelwitt: 2020-07-16 20:37:40.144 => vm started, 2020-07-16 20:37:40.283 => vm paused, 2020-07-16 20:37:40.360 => vm resumed21:09
mnaseranyhow we also run default vif_plugging_timeout which is 300s so it would have bombed then (there are logs of other vms bombing)21:10
melwittoh, hm21:11
mnaserso i wonder the fact other threads are waiting for things are causing it to stall out21:11
melwittyeah... I guess we have seen things like that, where we had to dial down the concurrent builds setting21:13
openstackgerritLee Yarwood proposed openstack/nova stable/queens: libvirt: Don't delete disks on shared storage during evacuate  https://review.opendev.org/73271721:13
melwittthere was also a bug around some libvirt driver green thread proxying but it was fixed awhile back https://review.opendev.org/67773621:14
mnasermelwitt: interesting, that's not in this stein deployemnt21:18
melwittyeah, the backports all got nacked awhile back :\21:19
mnaseryep, i see that21:19
mnaserseems like train came out with that21:19
melwittyes21:19
openstackgerritLee Yarwood proposed openstack/nova master: Ensure source compute is up when confirming a resize  https://review.opendev.org/69929121:19
mnaserinterestingly enough, all os_vif ops are taking almost 4-5s21:20
mnaserand i think os_vif is just local only?21:20
melwittyeah, I think so21:21
*** markvoelker has joined #openstack-nova21:23
*** martinkennelly has quit IRC21:24
*** markvoelker has quit IRC21:26
*** markvoelker has joined #openstack-nova21:33
dansmithmelwitt: ersum, thanks21:37
melwittyer welcome21:37
*** markvoelker has quit IRC21:38
openstackgerritStephen Finucane proposed openstack/nova master: api: Reject non-spawn operations for vTPM  https://review.opendev.org/74150021:42
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add emulated TPM support to Nova  https://review.opendev.org/63136321:42
openstackgerritStephen Finucane proposed openstack/nova master: docs: Add docs for vTPM support  https://review.opendev.org/73921321:42
openstackgerritStephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files  https://review.opendev.org/63993421:42
openstackgerritTobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration  https://review.opendev.org/74152921:56
openstackgerritTobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration  https://review.opendev.org/74152921:57
*** xiaolin has quit IRC22:24
*** ociuhandu has joined #openstack-nova22:25
*** ociuhandu has quit IRC22:29
*** rcernin has joined #openstack-nova22:43
*** rcernin has quit IRC22:47
*** mlavalle has quit IRC22:53
*** rcernin has joined #openstack-nova22:54
*** tkajinam has joined #openstack-nova23:02
*** tosky has quit IRC23:04
openstackgerritArtom Lifshitz proposed openstack/nova master: Handle Neutron errors in _post_live_migration()  https://review.opendev.org/72976323:04
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Check if device is listed in the inactive config during detach  https://review.opendev.org/74155723:12
*** gyee has quit IRC23:16
*** devfaz has quit IRC23:16
*** hoonetorg has quit IRC23:16
*** irclogbot_3 has quit IRC23:16
*** gibi has quit IRC23:16
*** gyee has joined #openstack-nova23:23
*** devfaz has joined #openstack-nova23:23
*** hoonetorg has joined #openstack-nova23:23
*** irclogbot_3 has joined #openstack-nova23:23
*** gibi has joined #openstack-nova23:23
mnasermelwitt: further investigation shows that that the vms are taking a long time because of plugging ports23:44
mnaserbut .. https://www.irccloud.com/pastebin/WAiTYOJE/ .. vif_plugging_timeout is 300, except it took almost 10 minutes to plug em23:44
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices  https://review.opendev.org/74156123:52
« Wednesday, 2020-07-15 Index Friday, 2020-07-17 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!