Thursday, 2019-05-09

« Wednesday, 2019-05-08 Index Friday, 2019-05-10 »
*** _alastor_ has quit IRC00:09
*** ttsiouts has quit IRC00:12
*** mriedem has quit IRC00:15
*** artom has joined #openstack-nova00:30
*** _hemna has joined #openstack-nova00:33
*** gyee has quit IRC00:38
*** Sundar has quit IRC00:45
*** brinzhang has joined #openstack-nova00:53
*** cdent has quit IRC01:06
*** _hemna has quit IRC01:08
*** zhongjun2_ has joined #openstack-nova01:26
*** tiendc has joined #openstack-nova01:28
*** _alastor_ has joined #openstack-nova01:37
*** _alastor_ has quit IRC01:42
*** sean-k-mooney has quit IRC01:42
*** tbachman has quit IRC01:49
*** tbachman has joined #openstack-nova01:51
*** ttsiouts has joined #openstack-nova02:09
*** itlinux has joined #openstack-nova02:28
*** partlycloudy has joined #openstack-nova02:35
*** itlinux has quit IRC02:38
*** ttsiouts has quit IRC02:42
*** ricolin has joined #openstack-nova02:42
*** JamesBenson has joined #openstack-nova02:59
*** _hemna has joined #openstack-nova03:04
*** psachin has joined #openstack-nova03:34
*** _hemna has quit IRC03:38
*** udesale has joined #openstack-nova03:51
*** lpetrut has joined #openstack-nova03:53
*** wwriverrat has quit IRC04:03
openstackgerritBrin Zhang proposed openstack/nova master: Change the migration warning log  https://review.opendev.org/65791604:13
*** ratailor has joined #openstack-nova04:14
*** lpetrut has quit IRC04:18
openstackgerritMerged openstack/nova stable/stein: Use migration_status during volume migrating and retyping  https://review.opendev.org/65757504:33
*** janki has joined #openstack-nova04:38
*** ttsiouts has joined #openstack-nova04:39
*** ivve has quit IRC04:45
*** JamesBenson has quit IRC05:01
*** ttsiouts has quit IRC05:12
*** jaosorior has joined #openstack-nova05:12
*** samueldmq has quit IRC05:19
*** yaawang has quit IRC05:19
*** yaawang has joined #openstack-nova05:20
*** _hemna has joined #openstack-nova05:34
*** _alastor_ has joined #openstack-nova05:39
*** _alastor_ has quit IRC05:43
*** ivve has joined #openstack-nova05:49
*** belmoreira has joined #openstack-nova06:06
*** slaweq has joined #openstack-nova06:06
*** _hemna has quit IRC06:07
*** udesale has quit IRC06:13
*** udesale has joined #openstack-nova06:14
*** whoami-rajat has joined #openstack-nova06:20
*** Luzi has joined #openstack-nova06:26
openstackgerritzhongshengping proposed openstack/nova master: Update Python 3 test runtimes for Train  https://review.opendev.org/65794106:29
*** brinzh has joined #openstack-nova06:48
*** brinzhang has quit IRC06:52
*** udesale has quit IRC06:53
*** udesale has joined #openstack-nova06:53
*** maciejjozefczyk has joined #openstack-nova07:09
*** tesseract has joined #openstack-nova07:09
*** ttsiouts has joined #openstack-nova07:09
*** shilpasd has joined #openstack-nova07:18
*** panda|off is now known as panda07:19
*** jaosorior has quit IRC07:27
*** johnthetubaguy has quit IRC07:28
*** johnthetubaguy has joined #openstack-nova07:30
*** rpittau|afk is now known as rpittau07:35
*** helenafm has joined #openstack-nova07:41
*** ttsiouts has quit IRC07:42
*** ttsiouts has joined #openstack-nova07:46
*** belmoreira has quit IRC07:55
*** belmoreira has joined #openstack-nova07:58
*** brinzh has quit IRC08:08
*** brinzh has joined #openstack-nova08:08
*** ralonsoh has joined #openstack-nova08:20
*** kaisers1 has joined #openstack-nova08:27
*** tkajinam has quit IRC08:30
*** jangutter has joined #openstack-nova08:39
*** tobias-urdin has joined #openstack-nova08:41
*** panda has quit IRC08:55
*** panda has joined #openstack-nova08:55
*** gmann_pto has joined #openstack-nova08:58
*** _hemna has joined #openstack-nova09:12
*** ttsiouts has quit IRC09:15
*** tssurya has joined #openstack-nova09:18
*** wxy-xiyuan has quit IRC09:26
*** belmoreira has quit IRC09:33
openstackgerritzhongshengping proposed openstack/nova master: Update Python 3 test runtimes for Train  https://review.opendev.org/65794109:37
*** _alastor_ has joined #openstack-nova09:40
*** _alastor_ has quit IRC09:45
*** _hemna has quit IRC09:46
*** belmoreira has joined #openstack-nova09:48
*** tssurya has quit IRC09:57
openstackgerritSylvain Bauza proposed openstack/nova master: Pass allocations to virt drivers when resizing  https://review.opendev.org/58908510:07
*** helenafm has quit IRC10:10
*** mugsie has quit IRC10:25
*** ttsiouts has joined #openstack-nova10:31
*** jaosorior has joined #openstack-nova10:34
*** jaosorior has quit IRC10:35
*** jaosorior has joined #openstack-nova10:35
*** mugsie has joined #openstack-nova10:35
*** mugsie has quit IRC10:35
*** mugsie has joined #openstack-nova10:36
*** ttsiouts has quit IRC10:36
*** mugsie has quit IRC10:38
*** mugsie has joined #openstack-nova10:39
*** tesseract has quit IRC10:40
*** tesseract has joined #openstack-nova10:41
*** tssurya has joined #openstack-nova10:42
*** priteau has joined #openstack-nova10:43
*** tesseract has quit IRC10:45
*** tesseract has joined #openstack-nova10:45
*** tbachman has quit IRC10:48
*** belmoreira has quit IRC10:55
*** belmoreira has joined #openstack-nova10:56
*** panda is now known as panda|lunch11:02
openstackgerritSurya Seetharaman proposed openstack/nova stable/ocata: [DNM] Populate key_pairs from top_cell to child cells  https://review.opendev.org/65799511:09
openstackgerritSurya Seetharaman proposed openstack/nova stable/ocata: [DNM] Populate key_pairs from top_cell to child cells  https://review.opendev.org/65799511:13
tssuryasorrison: ^^ its an ugly fix, but it worked for us :) I don't think it will be merged upstream but you can use it downstream. Let me know if you have any questions11:14
*** belmoreira has quit IRC11:15
*** udesale has quit IRC11:15
openstackgerritSurya Seetharaman proposed openstack/nova stable/ocata: [DNM] Populate key_pairs from top_cell to child cells  https://review.opendev.org/65799511:19
*** priteau has quit IRC11:23
*** helenafm has joined #openstack-nova11:38
*** _hemna has joined #openstack-nova11:42
*** belmoreira has joined #openstack-nova11:48
*** gmann_pto has quit IRC11:50
*** janki has quit IRC11:54
*** zigo has quit IRC11:59
*** belmoreira has quit IRC12:03
*** tbachman has joined #openstack-nova12:05
*** panda|lunch is now known as panda12:06
*** _hemna has quit IRC12:16
*** zigo has joined #openstack-nova12:23
*** mgariepy has quit IRC12:26
*** mgariepy has joined #openstack-nova12:30
*** mgariepy has quit IRC12:42
*** brinzh has quit IRC12:44
*** belmoreira has joined #openstack-nova12:49
*** udesale has joined #openstack-nova12:50
*** mriedem has joined #openstack-nova12:51
*** ratailor has quit IRC12:54
*** mgariepy has joined #openstack-nova12:57
*** artom has quit IRC13:04
shilpasdefried: Hi13:10
efriedo/ shilpasd13:10
shilpasdHi need bit input, i am currently working on implementation of https://review.opendev.org/#/c/650188/, Allow compute nodes to use shared storage provider for DISK_GB resources13:10
shilpasdGone through RESHAPE functionality implemented and here which deals with reshaping inventory and allocations13:11
shilpasdIn Shared storage Provider, similarly doing reshape where moving allocations of DISK_GB from compute node to shared storage provider13:11
shilpasdnow i want to correct local_gb_used at compute node at nova side13:12
shilpasdso can you help me how to proceed here13:12
shilpasdpre_start_hook() first update_available_resource() and then calls update_provider_tree()13:13
shilpasdand during update_provider_tree() i am doing RESHAPE for DISK_GB similarly done for VGPU13:14
efriedshilpasd: let me go look at some code...13:16
efriedIIRC local_gb_used is a bit tricky13:16
*** samueldmq has joined #openstack-nova13:16
efriedshilpasd: I'm sort of thinking we may want local_gb_used to be set to zero when the host is using shared disk.13:17
efriedmriedem, gibi, dansmith: You guys have a take on this? ^13:18
* gibi is lost in downstream rabbit holes, please send help13:21
*** ttsiouts has joined #openstack-nova13:24
mriedemidk, i'd think local_gb_used would at least report reserved_host_disk_mb (which defaults to 0)13:24
mriedemotherwise i'd tend to agree that it should be 0 for instance usage in the resource tracker if you're using shared storage on that node13:25
mriedemmdbooth might have other wrinkles in mind there13:25
dansmithyea, what mriedem said... because in the future we could have the compute *also* reporting its own disk inventory if it gained the ability to do that thing13:28
mdboothHmm, I'd need to look at code. Depends how local_gb_used is used.13:28
*** lbragstad_ has joined #openstack-nova13:29
* mdbooth doesn't understand how we handle thin provisioning in the placement world.13:29
*** lbragstad_ is now known as lbragstad13:30
*** _alastor_ has joined #openstack-nova13:30
*** _alastor_ has quit IRC13:35
*** tesseract has quit IRC13:35
*** tesseract has joined #openstack-nova13:35
efriedmdbooth: The only nod to thin provisioning is allocation_ratio13:36
efriedwhich amounts to a wild-ass guess13:36
efriedand is why cinder has shown minimal enthusiasm for adopting placement to do their tracking13:37
mdboothIIRC local_gb_used is all about thin provisioning.13:37
efriedoh?13:37
mdboothIIRC it answers the question: can I migrate here?13:37
*** KH-Jared has quit IRC13:38
mdboothIt's weird that we don't track actual usage in placement.13:38
efriedmdbooth: you would have to be sending updates all the time13:39
mdboothYeah13:40
efriedthe actual usage of thin-provisioned storage changes every time the guest does a write13:40
efriedalso, somebody told me that the storage doesn't ever actually tell you how much space it really has anyway.13:40
efriedthe more expensive the storage, the less accurate13:40
mdbooth*some* storage13:40
efriedwhich seems bizarre to me13:40
mdboothWe don't use that in Nova, though.13:40
efried"we" don't?13:40
efriedI was assuming we would eventually be using this shared storage stuff for cinder volumes too13:41
mdboothThe question we want to answer is: "do I have enough space to write X bytes here"?13:41
mdboothIt seems weird to me that any useful storage would not be able to answer that question.13:41
efriedagree with that13:41
efriedbut apparently it is so13:41
dansmithmdbooth: uh, we do if we're on nfs backed by it13:41
mdboothThat's a different question to "how much storage do I have?"13:41
dansmithbut the most expensive and most impressive storage gives you no useful numbers13:41
dansmithreportedly some of them give you nothing more than "buy more disks around october"13:42
mdboothDISK_GB_PURCHASED_OCTOBER13:42
dansmithobviously that would be a trait not a resource class :)13:42
mdboothHehe13:43
*** altlogbot_3 has quit IRC13:43
*** altlogbot_3 has joined #openstack-nova13:45
gansomriedem: good morning! =) I replied to your comment in https://review.opendev.org/#/c/657870 Could you please confirm if the test case is really the same?13:45
kashyapefried: Nice work on doing the summary emails.  Tireless donkey work it is.13:45
kashyapThanks!13:45
efriedkashyap: :) thanks13:49
*** psachin has quit IRC13:51
efriedshilpasd: afaict, the libvirt driver is setting the DISK_GB inventory based on the result from _get_local_gb_info()13:52
efriedAt a glance, it seems like it would make sense to split that into two methods: _get_local_disk_info and _get_shared_disk_info.13:53
efriedIf we start off supporting either local or shared but not both, then in a shared scenario the locals ought to report zero.13:53
efriedIf the provider tree you are given has its disk local, and _get_local_disk_info returns zeros but _get_shared_disk_info returns nonzeros, you know you need to reshape.13:54
efriedAnd local_gb_used (really local_gb<anything>) should simply be fed from _get_local_disk_info, no matter what, meaning in the above case it will be zeros.13:54
stephenfinanyone know if/when jaypipes will be back around. It sure would be lovely give the a once over and get it in (he reviewed in depth previously) https://review.opendev.org/#/c/629589/13:56
*** mgariepy has quit IRC13:57
*** mgariepy has joined #openstack-nova13:58
jaypipesstephenfin: I am here.13:59
jaypipesstephenfin: +2 from me. feel free to +W it.14:03
shilpasdefried: thanks for input14:04
*** ttsiouts has quit IRC14:08
*** rpittau is now known as rpittau|afk14:08
openstackgerritEric Fried proposed openstack/nova-specs master: Train Cycle Themes  https://review.opendev.org/65717114:09
*** _hemna has joined #openstack-nova14:13
gansoHi folks! I got a question about the num_instances value in the compute_nodes table. I have run into a situation where I have 4 VMs and "openstack hypervisor show" displays running_vms = 2. I found that this value is incremented/decremented in nova/compute/stats.py. Is there a way to recalculate this value or is the only way to fix this to update the database directly?14:13
*** mlavalle has joined #openstack-nova14:13
*** JamesBenson has joined #openstack-nova14:17
*** tiendc has quit IRC14:17
*** itlinux has joined #openstack-nova14:17
mriedemganso: are all 4 of those vms running?14:20
gansomriedem: yes14:21
*** ccamacho has joined #openstack-nova14:21
*** JamesBenson has quit IRC14:21
mriedemlooks like the update_available_resource periodic task in the compute should recalculate that value on each run of the task, which is every 1 minute by default14:23
*** itlinux has quit IRC14:23
*** ttsiouts has joined #openstack-nova14:23
mriedemwhich release are you on?14:23
*** JamesBenson has joined #openstack-nova14:23
*** tesseract has quit IRC14:24
gansomriedem: mitaka14:24
*** liuyulong has joined #openstack-nova14:25
mriedemganso: ok there was a stats related bug that probably only got fixed back to ocata, se14:27
mriedem*sec14:27
*** ttsiouts has quit IRC14:28
*** JamesBenson has quit IRC14:28
openstackgerritSurya Seetharaman proposed openstack/nova master: Disable max_placement_results if affinity (or anti) is requested  https://review.opendev.org/65811014:28
*** JamesBenson has joined #openstack-nova14:29
mriedemganso: i think you might need https://review.opendev.org/#/q/I0b9e5b711878fa47ba90e43c0b41437b57cf8ef614:32
mriedemoh but looking at that again, it was fixing a regression in ocata, so maybe not14:32
mriedemnvm14:32
efriedtssurya: Was your intent to backport that ----^14:32
tssuryaefried: yea was hoping to once it got merged in master14:33
efriedtssurya: Okay. Because we talked about enabling some placement-isms to help with this in Train itself, yah?14:33
tssuryaefried: yep, that would be a spec14:33
tssuryaI'll get onto that too soon-ish14:33
mriedemin_tree for strict affinity but that's not backportable14:33
mriedemit's also racy14:33
gansomriedem: so, this problem is likely caused by a bug that got fixed, correct? or was the periodic task added in ocata?14:34
efriedcool, just making sure I had the big picture14:34
tssuryamriedem: no no ^ that fix is just making limits None if affinity/anti is requested14:34
tssuryalike we did earlier on for force_hosts/nodes14:34
tssuryaonly that needs to be backported14:34
mriedemganso: the periodic was always around,14:34
mriedemthe stats thing i fixed was a regression in ocata so shouldn't affect you on mitaka14:34
mriedemganso: for that num_instances thing, you'd likely need to add some debug logging to that update_available_resource flow to see what it's doing,14:36
mriedembut it should 0 out num_instances in the stats at the start of the run, and then add instances to the stats as it processes instances for that host/node14:36
mriedemganso: does https://developer.openstack.org/api-ref/compute/#list-hypervisor-servers also show 4 servers on that hypervisor?14:37
*** mvkr has quit IRC14:37
efriedtssurya: We have in_tree in Stein though...14:38
efried...so you could make a backportable patch that adds in_tree for positive affinity and removes the limit for anti-affinity.14:38
efried...and then if you want to backport further, use this one.14:39
gansomriedem: hmmm is that equivalent of "openstack hypervisor list" ?14:39
efriedtssurya: is that the plan?14:39
mriedemefried: tssurya: it should be a 2 part series at least,14:39
gansomriedem: I have confirmed that the database has the value "2", so the database is wrong14:39
*** BjoernT has joined #openstack-nova14:39
tssuryaefried: oh didn't realise in_tree was there in Stein since it was used only in train14:39
tssuryabut what you said makes sense14:39
mriedemtssurya: but the bug goes back further than that14:39
efriedtssurya: I *think* it's in stein, sec...14:39
mriedemin_tree usage in nova is new in train14:40
mriedemfrom tetsuro14:40
tssuryamriedem: yup I think the bug is from rocky, and it would be good for us to have it in rocky14:40
*** BjoernT has quit IRC14:40
mriedemit probably goes back further than that14:41
mriedemmy guess is pike14:41
efriedtssurya: Confirmed, in_tree is in Stein.14:41
efriedSo you could do this in three parts.14:41
mriedemefried: in_tree is in stein in placement right?14:41
efriedyes --^14:41
mriedembut the nova object usage of it is not14:42
efried1.3114:42
efrieddoes that matter?14:42
mriedemif you're using versioned objects it does yes14:42
efriedversioned objects14:42
efriedyou mean the RequestSpec?14:42
mriedemRequestGroup14:42
mriedembut yes14:42
mriedemhttps://review.opendev.org/#/q/topic:bug/1777591+(status:open+OR+status:merged)14:42
mriedemit's really 2 changes, one to backport which just disables the limit if the server is in a group14:43
efriedgroan, okay.14:43
mriedemand another to be smarter about in_tree if it's in a strict affinity policy14:43
mriedembut noting that ^ is racy14:43
tssuryamriedem, efired: ok so we stick to what we discussed in the ptg right ?14:44
tssuryathe new in_tree affinity thing stays in train then14:44
*** _alastor_ has joined #openstack-nova14:44
tssuryamriedem: yea you might be right, I can trace it back to at least queens14:45
mriedemmax_placement_results was new in queens so i guess it goes to queens14:45
mriedemganso: no not hypervisor list, sec14:45
efriedtssurya: That's not my favorite misspelling of my handle.14:45
mriedemshe's channeling her inner trump14:46
tssuryaefried: yikes, I didn't even realised I missplled it :D14:46
mriedemtssurya is a big fan, some are saying the biggest fan14:46
mriedemfolks14:46
efriedanyway, +2 on this patch, thanks for explaining14:46
tssuryamriedem: :P14:46
tssuryathanks efried, mriedem14:47
mriedemganso: more like this https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-hypervisor-servers14:47
*** _hemna has quit IRC14:47
mriedemganso: i'm wondering if ^ will show 4 for that node while running_vms shows 214:47
gansomriedem: oh I see!14:47
gansomriedem: will run it and report back!14:47
mriedemif ^ shows 2 servers as well, then you have some issue14:47
mriedemlike, the db might say 2 of those guests you think are on that node actually aren't14:48
mriedeme.g. failed live migration or something14:48
gansomriedem: it does list 4 VMs! =)14:50
*** tesseract has joined #openstack-nova14:50
mriedemok so yeah the stats stuff is busted somehow, but you'd likely need to add logging statements to the code to debug it14:51
*** dpawlik has quit IRC14:51
*** Luzi has quit IRC14:51
*** tesseract has quit IRC14:51
*** tesseract has joined #openstack-nova14:51
gansomriedem: ok, so... I am curious on how I could reset the value, and double check if it is being recalculated incorrectly, or if it is not being recalculated at all14:51
gansomriedem: like, if I restart the service (nova-cpu I'd assume), it would reset to 0 and recalculate? Does it help if I set the value to 0 in the database and then restart the service?14:52
*** ratailor has joined #openstack-nova14:52
mriedemthe periodic should set it every time14:53
mriedemso you could set it to 0 in the db if you want and wait for the periodic to run, or restart the service14:53
gansomriedem: ok, will try that, thank you!14:54
openstackgerritMerged openstack/nova stable/rocky: Use migration_status during volume migrating and retyping  https://review.opendev.org/65757714:54
*** BjoernT has joined #openstack-nova14:54
*** hongbin has joined #openstack-nova14:56
mriedemtssurya: comments in that change, but it's not just the group hint14:59
gansomriedem: sorry, one more question. I am trying to track the name of the periodic task, I haven't seen periodic_task decorators or where they are registered in resource_tracker.py. But it seems the periodic task is either "update_available_resource" or another method that invokes this one. Could you please confirm?14:59
mriedemwe also have same_host/different_host hints/filters14:59
mriedemganso: it's the ComputeManager.update_available_resource method14:59
mriedemcalled on start of the compute service and in a periodic14:59
gansomriedem: oh it starts in the ComputeManager, thank you!14:59
*** ttsiouts has joined #openstack-nova14:59
tssuryamriedem: oh hmm looking14:59
*** tesseract has quit IRC15:01
mriedemhttps://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#samehostfilter15:01
mnaserso it looks like the kernel that enables nested by default (4.19) also added a gate that blocks live migrations if nested virt is enabled p15:01
jangutterDid the ptg etherpad finally die?15:01
mnaserERROR nova.virt.libvirt.driver [-] [instance: 738eb188-1545-4126-b05b-54d384e55f73] Live Migration failure: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet: libvirtError: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet15:02
mnaserso by default vmx is enabled, so every instance is not live migratable...15:02
tssuryamriedem: I have never used that filter before, but since I am fixing it for the other two filters, I will update the patch for this one as well15:02
tssuryaand https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#differenthostfilter like you have said15:02
mriedemi really wish we could functionally test this code to recreate the bug but i don't think there is an easy way to do that15:03
tssuryajangutter: I have been trying to access it too :( not sure if someone has a backup15:03
*** tesseract has joined #openstack-nova15:03
*** ttsiouts has quit IRC15:04
tssuryamriedem: you mean the bug I am working on ?15:04
*** guilhermesp has joined #openstack-nova15:11
mriedemtssurya: yes15:11
*** tesseract has quit IRC15:11
mriedembut there isn't really a good way to do that with functional tests that don't involve stubbing placement responses, which defeats the purpose of a functional test15:11
*** wwriverrat has joined #openstack-nova15:11
*** imacdonn has quit IRC15:12
*** imacdonn has joined #openstack-nova15:12
mnaserI’m trying to figure out how I can make both nested virt available as an opt in15:14
mnaserBut also without making complicated host aggregates and flavour explosion15:15
*** tesseract has joined #openstack-nova15:16
mriedemrequired trait on vmx?15:18
*** tesseract has quit IRC15:18
mriedemlibvirt computes with vmx enabled would report the HW_CPU_X86_VMX trait right?15:20
*** tesseract has joined #openstack-nova15:20
mriedemand you could have flavors and/or images which require that trait so the user can opt into nested virt that way15:20
*** jaosorior has quit IRC15:23
mriedemlive migration is going to be hairy though right? because you could have instances land on those hosts that don't want nested virt and then you can't live migrate them?15:30
mriedemnot sure if "Live Migration failure: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet: libvirtError: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet" is a problem if vmx is on the source, dest or both15:31
mnaserright, but then the idea is to make it opt-in and have a note that says "you won't be live migrated if you enable nested virtualize"15:31
mnaservirt*15:31
mriedemmy point is,15:31
mriedemthe people that aren't opting in could still land on those hosts15:31
mriedemi.e. this is the exclusion problem in the scheduler15:31
mriedemfor which forbidden aggregates are going to be used15:31
mriedemhttps://specs.openstack.org/openstack/nova-specs/specs/train/approved/placement-req-filter-forbidden-aggregates.html15:32
mriedemi'm not sure how you get around this without aggregates15:33
*** trident has quit IRC15:33
mnaserhmm, so its not necessarily possible to "request" a custom cpu flag for a specific instance15:34
*** trident has joined #openstack-nova15:35
openstackgerritRodrigo Barbieri proposed openstack/nova stable/queens: [DEBUG] Add functional confirm_migration_error test  https://review.opendev.org/65813615:36
openstackgerritDan Smith proposed openstack/nova master: Update the contributor doc for macos  https://review.opendev.org/65813715:37
dansmithtrivial contributor doc update ^15:37
*** ivve has quit IRC15:40
*** mvkr has joined #openstack-nova15:42
*** helenafm has quit IRC15:43
bnemecefried: Would you be able to add the nova-specific freeze dates to https://releases.openstack.org/train/schedule.html ?15:43
bnemecThe keystone patch is a good example: https://review.opendev.org/#/c/653544/15:44
bnemecIt came up last week that people were surprised by some Nova freeze dates so it would be good to get them on the common schedule.15:44
efriedbnemec: can do. I think melwitt conveniently created the schedule a month or two ago in the wiki, please hold.15:44
efriedbnemec: https://wiki.openstack.org/wiki/Nova/Train_Release_Schedule15:44
mriedemmnaser: the user would request via the flavor/image which would have the required cpu flag as a trait15:44
bnemecefried: Cool, thanks.15:45
bnemecI guess the main one would be spec freeze then.15:46
mnasermriedem: would it be appended to the 'existing' cpu flags or it would have to match a hypervisor that has those cpu flags (i.e I would have a bunch of nested-virt enabled computes, and those that aren't)15:46
mriedemmnaser: my understanding from https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/report-cpu-features-as-traits.html is that the compute reports the cpu features as traits, and you can configure each compute with additional flags using https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu_model_extra_flags15:48
*** priteau has joined #openstack-nova15:48
mriedemi think cpu_model_extra_flags get reported as traits as well, but i would have to dig into the code to confirm15:48
mnaserbut if I add `cpu_model_extra_flags` that would then make every instance get nested virt15:49
mnaser(on that host)15:49
*** belmoreira has quit IRC15:49
mriedemoh i see, you want per-instance15:49
mnaseryeah, that way I don't have to maintain a whole bunch of host aggrgates15:50
mriedemwell, geez, that sounds like starlingx craziness bro!15:50
mnaseris that something they do / looking to do ?15:50
mriedemi want to say yes15:50
mriedemi thought somewhat related to https://review.opendev.org/#/c/637834/ but not really what you want15:51
mnaserit looks like the workflow of google is per-image based -- https://cloud.google.com/compute/docs/instances/enable-nested-virtualization-vm-instances15:51
*** cdent has joined #openstack-nova15:52
mriedemyar, i know we have things like hw_vif_model, hw_pointer_model, hw_scsi_model but no hw_cpu_model15:53
mnaseryeah I guess the weird thing is `hw_cpu_model` actually ties into physical infra instead of virtual only15:54
mriedemnot sure where vcpu_model comes from on the instance15:54
*** gyee has joined #openstack-nova15:54
mnaseri.e. `hw_cpu_model` depends on the physical CPU having a certain capability so it might not always work15:54
mnaserwhereas the rest will probably always work as they are virtualized things15:55
mriedemhttps://review.opendev.org/#/c/148138/15:55
*** yan0s has quit IRC15:56
*** priteau has quit IRC15:56
mriedemallowing the user to specify the cpu features they want sounds like something kashyap might be aware of if someone is working on it15:57
*** bnemec has quit IRC15:58
*** bnemec has joined #openstack-nova15:58
mnaserbecause right now I think the assumption is: if a host advertises support for a specific trait, it will always boot vms with that cpu flag15:58
mriedemmnaser: ah here you go, starlingx did have a spec for it at one point https://review.opendev.org/#/c/168982/ (liberty)15:59
*** mvkr has quit IRC15:59
mriedemthank my handy dandy starlingx diffs spreadsheet15:59
mnaserwhere as I'm trying to have it be: if a host advertises support for a specific trait, it can be an opt-in one, so all vms boot without it, unless it's requested explicitly16:00
*** tesseract has quit IRC16:01
mnaseryeah.. this seems to pretty much cover the majority of the use case, but huge scope16:01
mriedemlast week starlingx claimed to be down to 11 critical patches outside of nova and when i asked about that they said their were all for numa live migration, so if they have support for the user specifying cpu model in the flavor/image, i'm not sure what they are doing now16:01
mriedemand cfriesen isn't around16:01
mriedemnote that spec also predates traits and all that by years16:02
mnaserI was thinking maybe just add something like cpu_models_optin or something .. compute manager will advertise all traits (including ones in cpu_models_optin) and when it gets a vm, it'll decide to add/remove that flag if the flavor/image has that trait added16:02
*** ratailor has quit IRC16:03
mnaserwhich would make the scheduler side of things fairly untouched, and it would be a virt-level thing... and the traits stuff will still work too16:03
mnaserfor t in requested_traits: if t in cpu_optional_features then add cpu flag else noop -- pretty much,16:04
*** ttsiouts has joined #openstack-nova16:09
openstackgerritBrian Haley proposed openstack/nova stable/queens: Teardown networking when rolling back live migration even if shared disk  https://review.opendev.org/65814916:14
dansmithmriedem: I took that claim to mean they have only 11 patches of stuff planned to merge into nova and the rest is "value add" they're planning to just keep in their fork16:18
dansmithwhich is noticeably different from the original stance on the "we don't want to maintain a fork" stuff that none of us believed anyway :)16:19
mriedemyeah maybe16:19
mriedemi know they have lots of other stuff like this, like specifying vif_model when attaching a port16:22
openstackgerritDan Smith proposed openstack/nova master: Add extra logging to request filters  https://review.opendev.org/65762916:25
*** dtantsur has quit IRC16:26
*** BjoernT has quit IRC16:26
*** dtantsur has joined #openstack-nova16:26
openstackgerritSurya Seetharaman proposed openstack/nova master: Disable limit if affinity(anti)/same(different)host is requested  https://review.opendev.org/65811016:30
*** stephenfin has quit IRC16:30
mriedemgibi: comments and questions and nits in your 'move ops for qos ports' spec https://review.opendev.org/#/c/652608/ but nothing worth holding it back16:32
*** dtantsur has quit IRC16:32
*** roukoswarf has joined #openstack-nova16:34
roukoswarfhey all, i have a fun one... so i boot an instance using an image and a volume, works great, hard resets fine. but i boot an instance with a volume as the boot disk, add a volume, then hard reset, and then it has "no bootable device" till the volume is detached16:35
roukoswarfi cant find a decernable difference in the libvirt cfg that would cause it to not check all disks for mbr... but maybe theres some cfg im missing? replication steps are pretty simple, boot from a volume, add a volume, do a full power reset on the vm, fail.16:36
*** dtantsur has joined #openstack-nova16:37
*** tssurya has quit IRC16:38
*** maciejjozefczyk has quit IRC16:42
*** _hemna has joined #openstack-nova16:43
*** ricolin has quit IRC16:45
*** irclogbot_3 has quit IRC16:46
*** munimeha1 has joined #openstack-nova16:46
*** irclogbot_0 has joined #openstack-nova16:47
*** cdent_ has joined #openstack-nova16:56
openstackgerritDan Smith proposed openstack/nova master: Add extra logging to request filters  https://review.opendev.org/65762916:57
*** cdent has quit IRC16:57
*** cdent_ is now known as cdent16:57
*** stephenfin has joined #openstack-nova17:01
*** liuyulong has quit IRC17:03
openstackgerritMohammed Naser proposed openstack/nova master: [rfc] ability to use optional cpu flags  https://review.opendev.org/65815817:04
mnasermriedem: ^ well, I'd love to hear comments, I know this probably needs a spec and all sorts of other stuff.. but the 'general' idea17:05
*** ralonsoh has quit IRC17:08
*** macza has joined #openstack-nova17:12
*** udesale has quit IRC17:13
*** _hemna has quit IRC17:17
openstackgerritMerged openstack/nova master: libvirt: auto detach/attach sriov ports on migration  https://review.opendev.org/62958917:35
*** artom has joined #openstack-nova17:41
*** ivve has joined #openstack-nova17:42
*** _hemna has joined #openstack-nova17:53
*** hongbin has quit IRC17:54
*** BjoernT has joined #openstack-nova17:55
*** tbachman has quit IRC18:13
*** ttsiouts has quit IRC18:25
*** _hemna has quit IRC18:27
*** BjoernT has quit IRC18:34
*** igordc has joined #openstack-nova18:35
*** jaypipes has quit IRC18:37
*** ttsiouts has joined #openstack-nova18:40
openstackgerritArtom Lifshitz proposed openstack/python-novaclient master: Use SHA56 instead of MD5 in completion cache  https://review.opendev.org/65818118:44
*** igordc has quit IRC18:44
*** BjoernT has joined #openstack-nova18:44
*** ttsiouts has quit IRC18:45
*** tbachman has joined #openstack-nova18:45
*** igordc has joined #openstack-nova18:47
openstackgerritMerged openstack/nova master: Remove ComputeDriver.macs_for_instance method  https://review.opendev.org/65273718:55
mriedemhttps://review.opendev.org/#/q/topic:bp/libvirt-neutron-sriov-livemigration+(status:open+OR+status:merged) is all merged now but i'm not sure what docs are needed or where18:59
mriedemah yes https://docs.openstack.org/neutron/latest/admin/config-sriov.html#known-limitations19:00
mriedemstephenfin: is ^ something you want to handle while sean is out on pto?19:02
openstackgerritDustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for validating instance and node  https://review.opendev.org/65602819:11
*** ttsiouts has joined #openstack-nova19:11
*** maciejjozefczyk has joined #openstack-nova19:15
artommriedem, sean's back Monday, and stephenfin is probably drinking and/or sleeping now19:18
mriedemyeah, it was for his bouncer19:19
mriedemirc, not the "bloke" at the "pub"19:19
artom(Yeah, not sure why I thought you didn't know his tz)19:20
*** jobewan has joined #openstack-nova19:22
openstackgerritMerged openstack/nova master: Add --instance option to heal_allocations  https://review.opendev.org/65194519:22
*** maciejjozefczyk has quit IRC19:23
*** tbachman has quit IRC19:30
openstackgerritArtom Lifshitz proposed openstack/python-novaclient master: Use SHA256 instead of MD5 in completion cache  https://review.opendev.org/65818119:35
*** hongbin has joined #openstack-nova19:44
*** BjoernT has quit IRC19:45
*** ttsiouts has quit IRC20:14
redkriegI'm having an issue with lxd on my deployment where creating instances fails with "LXDAPIException: Profile is currently in use" in nova-compute.log.  is there any good way to troubleshoot what could be causing that?20:17
*** Nick_A has joined #openstack-nova20:21
*** _hemna has joined #openstack-nova20:23
*** ttsiouts has joined #openstack-nova20:29
*** tssurya has joined #openstack-nova20:31
*** boxiang has quit IRC20:32
*** ttsiouts has quit IRC20:32
*** ttsiouts has joined #openstack-nova20:33
*** boxiang has joined #openstack-nova20:33
openstackgerritMerged openstack/nova master: Update the contributor doc for macos  https://review.opendev.org/65813720:35
*** ttsiouts has quit IRC20:47
*** takashin has joined #openstack-nova20:52
*** _hemna has quit IRC20:58
*** logan- has quit IRC21:00
mriedemefried: nova meeting today?21:01
efriedyes, sorry, starting...21:01
edleafeheh, was just typing that :)21:01
mriedemredkrieg: lxd isn't in tree, you'd have to ask in their channel21:01
redkriegcool, thanks21:01
mriedemredkrieg: https://github.com/openstack/nova-lxd#support-and-discussions21:01
redkriegthat helps, thank you mriedem.  I'm doing a reinstall right now so I'll follow up there if I still have an issue21:03
mriedemnp21:03
*** logan- has joined #openstack-nova21:03
*** mchlumsky has quit IRC21:04
*** cdent has quit IRC21:11
*** cdent has joined #openstack-nova21:26
*** Nick_A has left #openstack-nova21:28
*** ttsiouts has joined #openstack-nova21:29
*** ttsiouts has quit IRC21:34
*** lbragstad has quit IRC21:45
*** dasp has quit IRC21:52
*** tbachman has joined #openstack-nova21:58
*** JamesBenson has quit IRC21:59
*** hongbin has quit IRC22:04
*** hongbin has joined #openstack-nova22:05
openstackgerritMatt Riedemann proposed openstack/nova master: DNM: Test theory behind bug 1822884  https://review.opendev.org/64946422:05
openstackbug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,In progress] https://launchpad.net/bugs/1822884 - Assigned to sean mooney (sean-k-mooney)22:05
*** hongbin has quit IRC22:05
*** hongbin has joined #openstack-nova22:06
*** hongbin has quit IRC22:06
*** hongbin has joined #openstack-nova22:06
*** hongbin has quit IRC22:08
*** hongbin has joined #openstack-nova22:08
*** trident has quit IRC22:19
*** tbachman has quit IRC22:19
mriedemsorry22:21
mriedemnot sorry22:21
openstackgerritMatt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize  https://review.opendev.org/64180622:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid  https://review.opendev.org/61401222:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method  https://review.opendev.org/61403622:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add Instance.hidden field  https://review.opendev.org/63112322:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add TargetDBSetupTask  https://review.opendev.org/62789222:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask  https://review.opendev.org/63158122:22
openstackgerritMatt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask  https://review.opendev.org/63385322:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method  https://review.opendev.org/62131322:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method  https://review.opendev.org/63329322:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask  https://review.opendev.org/62789022:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method  https://review.opendev.org/63483222:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image  https://review.opendev.org/63760522:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask  https://review.opendev.org/62789122:22
openstackgerritMatt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection  https://review.opendev.org/64218322:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add power_on kwarg to ComputeDriver.spawn() method  https://review.opendev.org/64259022:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method  https://review.opendev.org/63508022:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask  https://review.opendev.org/63564622:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add Destination.allow_cross_cell_move field  https://review.opendev.org/61403522:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add CrossCellWeigher  https://review.opendev.org/61435322:22
*** trident has joined #openstack-nova22:22
openstackgerritMatt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API  https://review.opendev.org/63826922:22
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job  https://review.opendev.org/65665622:22
*** markmcclain has quit IRC22:35
*** icey has quit IRC22:35
*** icey has joined #openstack-nova22:36
*** markmcclain has joined #openstack-nova22:37
*** ttsiouts has joined #openstack-nova22:46
*** mlavalle has quit IRC22:49
*** ttsiouts has quit IRC22:51
*** slaweq has quit IRC22:51
*** munimeha1 has quit IRC22:52
*** _hemna has joined #openstack-nova22:53
*** macza has quit IRC22:57
*** slaweq has joined #openstack-nova22:58
redkriegI reinstalled one of my compute nodes and thought I had removed it from openstack first but I think it reregistered before I shut it down.  is there a good way to get around this?  error:22:58
redkrieg2019-05-09 18:57:34.505 12028 ERROR nova.scheduler.client.report [req-d7fa7b90-ada9-4296-8f2f-977b8e041dfc - - - - -] [req-ce0a1ceb-1e0f-486f-afe6-c1dc9ecdbad5] Failed to create resource provider record in placement API for UUID 5f5eecf0-9805-4b0c-892e-cddb70b6a4eb. Got 409: {"errors": [{"status": 409, "request_id": "req-ce0a1ceb-1e0f-486f-afe6-c1dc9ecdbad5", "detail": "There was a conflict when22:58
redkriegtrying to complete your request.\n\n Conflicting resource provider name: nyc-slxc-1 already exists.  ", "title": "Conflict"}]}.22:59
*** Nick_A has joined #openstack-nova23:01
*** slaweq has quit IRC23:03
*** tkajinam has joined #openstack-nova23:03
cdentredkrieg: quick and dirty: find the resource provider that represent the older compute node (its uuid) and delete that from placement. you can do that with curl calls to placement, the osc-placement client, or looking into the database directly23:04
*** slaweq has joined #openstack-nova23:11
redkriegcdent: thanks, I found the entry and deleted it.  looks like it registered correctly after.23:12
cdentredkrieg: great. I think there's been some work or at least discussion on making that less of a problem, but quick and dirty seemed like the right way to go in this case23:13
redkriegyep, it's not a common scenario.  I generally avoid reinstalling when I can.23:13
*** slaweq has quit IRC23:16
mriedemthe proper thing would have been (1) stop the nova-compute process on the host, (2) delete the compute service in the API which (depending on your release) should cleanup the provider in placement automatically23:16
*** ttsiouts has joined #openstack-nova23:17
mriedemhttps://developer.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service23:18
redkriegI actually tried to use the openstack-ansible-ops playbook to remove the host first, but I think it left a placement entry instead of properly cleaning it up.  I'm just glad it's working now.  thanks for looking!23:19
*** ttsiouts has quit IRC23:21
mriedemhmm, i hope the ansible stuff isn't screwing things up now23:22
mriedemwhich release are you using?23:22
*** _hemna has quit IRC23:26
Nick_Awe're on rocky23:27
*** samueldmq has quit IRC23:31
*** whoami-rajat has quit IRC23:34
*** jobewan has quit IRC23:36
mriedemam i being blizzarded again?23:48
mriedemanyway, DELETE /os-services/{service_id} should have cleaned that stuff up for you assuming the nova-compute service was not running (it will recreate records on a periodic task)23:49
mriedemso i don't know what that ansible play does, but might be worth investigating, it should stop the service first and then delete it via the api23:49
artommriedem, you're seriously getting snow?23:51
mriedemartom: no blizzard the company23:51
artomThey're a verb now?23:51
mriedemyes23:52
mriedem"it's a blizzard of blizzard operators"23:52
mriedemblizzarded23:52
* artom stops asking questions23:53
openstackgerritMatt Riedemann proposed openstack/nova master: Handle inactive port binding in _update_port_binding_for_instance  https://review.opendev.org/65350623:53
mriedemfwiw i have no idea who redkrieg and Nick_A work for, i've just noticed the blizzard gang runs in packs23:54
openstackgerritMatt Riedemann proposed openstack/nova master: Handle inactive port binding in _update_port_binding_for_instance  https://review.opendev.org/65350623:59
mriedemtobberydberg: i know your guys are interested in this ^23:59
mriedemmight be a bear to backport but that should have it passing23:59
« Wednesday, 2019-05-08 Index Friday, 2019-05-10 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!