Thursday, 2020-12-17

*** tosky has quit IRC00:01
*** JamesBenson has quit IRC00:08
*** LinPeiWen has joined #openstack-nova00:37
*** JamesBenson has joined #openstack-nova00:43
brinzhang0gmann: ack, thanks, you have added in the cleanup apis etherpad, works for me, keep the consistency with other apis is better ^^00:48
*** JamesBenson has quit IRC00:49
*** derekh has quit IRC00:50
*** tbachman has quit IRC00:52
*** tbachman has joined #openstack-nova00:53
*** hemna has quit IRC00:54
*** hemna has joined #openstack-nova00:59
*** jamesdenton has joined #openstack-nova01:02
*** brinzhang has joined #openstack-nova01:03
*** brinzhang0 has quit IRC01:06
*** JamesBenson has joined #openstack-nova01:21
*** hamalq_ has quit IRC01:24
*** gyee has quit IRC01:24
*** evrardjp has quit IRC01:24
*** knikolla has quit IRC01:24
*** amodi has quit IRC01:24
*** CeeMac has quit IRC01:24
*** ftarasenko has quit IRC01:24
*** fnordahl has quit IRC01:24
*** tinwood has quit IRC01:24
*** nicolasbock has quit IRC01:24
*** mnasiadka has quit IRC01:24
*** nautik has quit IRC01:24
*** flaviof has quit IRC01:24
*** pas-ha has quit IRC01:24
*** cap has quit IRC01:24
*** Hazelesque_ has quit IRC01:24
*** jroll has quit IRC01:24
*** JamesBenson has quit IRC01:25
*** mlavalle has quit IRC01:26
*** hamalq_ has joined #openstack-nova01:30
*** gyee has joined #openstack-nova01:30
*** evrardjp has joined #openstack-nova01:30
*** amodi has joined #openstack-nova01:30
*** ftarasenko has joined #openstack-nova01:30
*** knikolla has joined #openstack-nova01:30
*** CeeMac has joined #openstack-nova01:30
*** fnordahl has joined #openstack-nova01:30
*** tinwood has joined #openstack-nova01:30
*** mnasiadka has joined #openstack-nova01:30
*** nicolasbock has joined #openstack-nova01:30
*** nautik has joined #openstack-nova01:30
*** flaviof has joined #openstack-nova01:30
*** pas-ha has joined #openstack-nova01:30
*** cap has joined #openstack-nova01:30
*** Hazelesque_ has joined #openstack-nova01:30
*** jroll has joined #openstack-nova01:30
*** LinPeiWen has quit IRC01:38
*** dave-mccowan has joined #openstack-nova01:49
*** abhishekk has quit IRC02:03
*** macz_ has quit IRC02:05
*** macz_ has joined #openstack-nova02:05
*** k_mouza has joined #openstack-nova02:07
*** tinwood has quit IRC02:08
*** tinwood has joined #openstack-nova02:11
*** k_mouza has quit IRC02:12
*** hemanth_n has joined #openstack-nova02:28
openstackgerritWenping Song proposed openstack/nova master: Remove redundant code in instance delete  https://review.opendev.org/c/openstack/nova/+/76744902:45
*** LinPeiWen has joined #openstack-nova02:50
*** Yumeng has joined #openstack-nova02:53
*** hamalq_ has quit IRC03:00
*** xinranwang has joined #openstack-nova03:02
*** gyee has quit IRC03:11
*** mkrai has joined #openstack-nova03:37
*** mgoddard has quit IRC04:03
*** mkrai has quit IRC04:27
*** macz_ has quit IRC04:28
*** mkrai has joined #openstack-nova04:29
*** songwenping_ has quit IRC04:44
*** songwenping_ has joined #openstack-nova04:44
*** mgoddard has joined #openstack-nova05:05
*** abhishekk has joined #openstack-nova05:05
*** k_mouza has joined #openstack-nova05:09
*** k_mouza has quit IRC05:13
*** mkrai has quit IRC05:21
*** mkrai_ has joined #openstack-nova05:21
*** tbachman has quit IRC05:22
*** tbachman has joined #openstack-nova05:22
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-nova05:33
*** mgoddard has quit IRC06:00
*** nautik has quit IRC06:06
*** diconico07 has quit IRC06:06
*** xinranwang has quit IRC06:11
*** vishalmanchanda has joined #openstack-nova06:17
*** LinPeiWen has quit IRC06:27
*** dasp has quit IRC06:30
*** LinPeiWen has joined #openstack-nova06:41
*** dasp has joined #openstack-nova06:47
*** zzzeek has quit IRC06:57
*** zzzeek has joined #openstack-nova06:58
*** mgoddard has joined #openstack-nova07:04
*** ralonsoh has joined #openstack-nova07:14
*** rcernin has quit IRC07:14
*** rcernin has joined #openstack-nova07:30
*** mkrai_ has quit IRC07:31
*** slaweq has joined #openstack-nova07:34
*** slaweq has quit IRC07:38
*** belmoreira has joined #openstack-nova07:45
*** slaweq has joined #openstack-nova07:50
*** mkrai has joined #openstack-nova07:56
*** slaweq has quit IRC08:01
*** mgoddard has quit IRC08:01
*** slaweq has joined #openstack-nova08:08
*** andrewbonney has joined #openstack-nova08:11
openstackgerritzhulipeng proposed openstack/nova master: Fixed multipath residue when evacuating the VM  https://review.opendev.org/c/openstack/nova/+/76746808:14
*** songwenping_ has quit IRC08:17
*** songwenping_ has joined #openstack-nova08:18
*** sapd1 has joined #openstack-nova08:18
*** tesseract has joined #openstack-nova08:19
bauzasgood morning Nova08:40
*** songwenping__ has joined #openstack-nova08:41
*** jawad_axd has joined #openstack-nova08:42
*** rcernin has quit IRC08:44
*** songwenping_ has quit IRC08:44
gibibauzas: good morning to you too08:53
bauzasgibi: last 2 days for you too ?08:55
gibibauzas: yepp, I'm already a bit in vacation mood08:55
bauzasLOL08:55
bauzasI'm still trying to remove the -W flag from the routed networks change before I'm going south :)08:56
bauzasand wow, Zuul gave me +1 on https://review.opendev.org/c/openstack/nova/+/749068/5 \o/08:57
gibiI know I will looking back at gerrit during the vacation periods as my wife has to work on some of the days08:57
bauzasyeepeekay :)08:57
gibithat is your present from Zuul :D08:57
bauzaswell, given how I battled with Zuul for this change, I'm more like this https://www.youtube.com/watch?v=YfpDSNNgYhI08:59
gibihehe, that is very vacation specific movie at least in my country :)09:01
gibialmost like home alone09:01
*** mgoddard has joined #openstack-nova09:02
*** ociuhandu has joined #openstack-nova09:06
bauzasagreed, same here09:06
*** derekh has joined #openstack-nova09:07
bauzasI don't expect Christmas periods without Die Hard 2 or 1 movies being broadcasted on at least one of them09:07
bauzasthe 2 gets more broadcasted tho09:07
bauzaswow, TIL the reasoning behind https://www.youtube.com/watch?v=KAYIgEpds_Y09:09
bauzasthe French dubbing totally trampled this phrase09:09
*** tosky has joined #openstack-nova09:12
*** ociuhandu has quit IRC09:16
*** teoobo_ has joined #openstack-nova09:18
gibibauzas: I will get you the hungarian dub after my morning call09:20
gibi:)09:20
*** sapd1 has quit IRC09:27
*** ociuhandu has joined #openstack-nova09:46
*** ociuhandu has quit IRC09:53
*** mgoddard has quit IRC10:01
gibibauzas: https://www.youtube.com/watch?v=BeH-ODa-41o10:03
gibiinterestingly there is two different hun dub for die hard 110:04
*** mgoddard has joined #openstack-nova10:05
gibigmann, sean-k-mooney, melwitt, bauzas, brinzhang: I'm OK with the proposal in the API etherpad. I'm glad we found an agreeable way forward10:08
bauzasme too10:08
bauzasfwiw, I just modified my change10:08
*** ociuhandu has joined #openstack-nova10:08
bauzashttps://review.opendev.org/c/openstack/nova/+/749068/5/nova/tests/functional/integrated_helpers.py10:09
bauzasbut I'll split this hairy change into multiple pieces now that everything works10:09
bauzasgibi: sharpen your pen, hopefully will upload the series later the day10:09
gibibauzas: sure, I will try to review it before I go10:10
bauzasno rushes, we're half the cycle10:10
bauzasbut I'm glad this works, as proven by the functests10:10
*** ociuhandu has quit IRC10:12
gibiI'm happy too, it was something I promised making happen in the past so I'm happy to sink time on reviewing it10:12
*** ociuhandu has joined #openstack-nova10:12
*** k_mouza has joined #openstack-nova10:15
stephenfingibi: sorry, was out yesterday. Looking now10:15
gibistephenfin: ack, no worries10:16
stephenfinin the meantime, bauzas/gibi: any chance you'd look at https://review.opendev.org/c/openstack/nova-specs/+/765796 to sync spec and code?10:16
brinzhanggibi: yeah, me too, happy to see the ethpad for clean multi apis ^^10:16
gibistephenfin: on it10:16
bauzasstephenfin: kids taxi, but I'll look10:16
gibibauzas: you don't have to, gmann and I took care of it now10:17
bauzaskk, ttyl10:17
gibilyarwood: I'm look at another possible race in tempest about volume handling https://zuul.opendev.org/t/openstack/build/fce0115deae94e38aaef4a349e271e20/log/job-output.txt#6907010:18
stephenfinThanks! There's also a part 3, but that's got an open question on it from sean-k-mooney. I wonder if we should discuss in the team meeting today? https://review.opendev.org/c/openstack/nova-specs/+/765797/1/specs/wallaby/approved/modernize-os-hypervisors-api.rst10:19
lyarwoodgibi: oh fun10:21
lyarwoodgibi: let me take a look10:21
*** mkrai has quit IRC10:23
*** k_mouza has quit IRC10:24
*** k_mouza has joined #openstack-nova10:25
elodlyarwood, similarly like yesterday: can you hit now the bandit patch in queens: https://review.opendev.org/c/openstack/nova/+/767364 o:)10:25
*** mkrai has joined #openstack-nova10:27
lyarwoodelod: ack will do10:28
openstackgerritMerged openstack/nova-specs master: Update modernize-os-hypervisors-api spec, pt. 2  https://review.opendev.org/c/openstack/nova-specs/+/76579610:28
openstackgerritAlexandre arents proposed openstack/nova master: Revert "nova-multi-cell: Skip test_cold_migrate_unshelved_instance"  https://review.opendev.org/c/openstack/nova/+/76593310:30
zigobauzas: Hi man!10:31
zigoOne quick question about GPU in passthrough mode: what's the reason for moving from "type-PCI" to "type-PF" ?10:31
zigoWe had the issue when we upgraded from Rocky to Train ...10:32
bauzaszigo: that depends on which GPU card you have10:37
bauzasif it supports SR-IOV, you need to modify the config because Nova will see the PF10:37
zigobauzas: If it has SR-IOV support, then type-PF ?10:37
bauzasyes10:37
zigoAh ok...10:37
bauzasnvidia T4s10:37
zigobauzas: That's the model we have, indeed.10:37
bauzasyeah, we had a problem with nvidia10:38
zigobauzas: Is this for both the alias in the scheduler and in the compute ?10:38
bauzasthey only say that SR-IOV is experimental for T4, but given this, you actuallly need to change nova for it10:38
bauzaszigo: hem, yes AFAIK10:38
bauzasstephenfin: right ? ^10:38
lyarwoodgibi: odd, so one of the tests makes two attempts to detach the same volume10:40
stephenfinzigo, bauzas: correct - if the device reports SR-IOV capabilities, you need to set the device type10:40
zigobauzas: Thanks a lot. I'll add a per-compute option to select PCI / PF then.10:42
stephenfinOn that note, I still have two patches open to improve SR-IOV testing, if anyone has a few minutes. Happy to review other stuff in exchange https://review.opendev.org/q/topic:%2522sriov-functional-tests%2522+status:open10:42
lyarwoodgibi: https://review.opendev.org/c/openstack/tempest/+/760357 - this added a cleanup to detach the volume that these tests already detaches10:43
lyarwoodI'll write up a bug shortly10:44
*** tosky has quit IRC10:47
*** tosky has joined #openstack-nova10:47
*** zzzeek has quit IRC10:47
*** zzzeek has joined #openstack-nova10:49
zigobauzas: IMO, it'd be really nice to automate this, with placement reporting the correct type and the scheduler being auto-magic. It took us 2 days to figure out what was going on...10:49
elodthx lyarwood !10:50
gibistephenfin: ack, I can add that to today's agenda10:51
*** lbragstad has quit IRC10:52
gibilyarwood: thanks! (sorry I was pulled after I pinged you)10:53
sean-k-mooneyzigo: we do not report pci devices in placement yet10:53
zigosean-k-mooney: But that would be a nice improvement ! :)10:54
sean-k-mooneyzigo: also this is because of a upgrade in your t4 device firmware10:54
sean-k-mooneyzigo: yes its on our todolist for a few years10:54
sean-k-mooneythere is a series that does most of it that was abandoned 2 years ago10:54
sean-k-mooneywe just need to revive it10:54
sean-k-mooneyit was blocked because placmenet did not support nested resouce providers proerly at the time10:54
sean-k-mooneythat has now meen fixed on the placment side10:55
gibilyarwood: i don't see the double detach here: tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario10:55
*** lbragstad has joined #openstack-nova10:55
zigosean-k-mooney: Is type-PCI and type-PF the only possible values?10:55
gibilyarwood: nvm, I see it now10:55
sean-k-mooneyno also type-VF10:55
zigo(I'm implementing this in my deployment software and need to validate input)10:55
zigoOk, thanks.10:56
gibinova_volume_attach adds a cleanup and the test case itself adds the same cleanup10:56
sean-k-mooneywe are adding type-VDPA but i plan to not allow that in the alias10:56
sean-k-mooneyVDPA devices will be VF that cant be use with alias based passthough as they require an additonal contolplain to function10:56
sean-k-mooneyzigo: by the way are you currently supporting devname in the pci whitelist?10:57
sean-k-mooneyzigo: we woudl really like to remove that in the future10:57
zigosean-k-mooney: Yeah, I do !10:57
sean-k-mooneyok what we would prefer is for installer tools to convert that to the pci adress10:58
zigoFor each compute node, I can set: use_gpu, gpu_name, gpu_vendor_id, gpu_product_id and gpu_device_type.10:58
sean-k-mooneygpu_name?10:58
zigoI just need to expose gpu_device_type in my API which I didn't do...10:58
zigoYeah, a free name which goes in the alias.10:59
sean-k-mooneyoh that is differnt10:59
sean-k-mooneyim not talking about the alias10:59
sean-k-mooneyim talking about the whitelist where we support devname for nics e.g. eht110:59
*** rpittau|afk is now known as rpittau10:59
sean-k-mooneyeth110:59
zigoWe don't do SR-IOV yet.10:59
zigoJust GPUs...11:00
sean-k-mooneyvGPU or pci passthough11:00
sean-k-mooneyif your using pci passthough via the alias and pci whitelist that is most of the config for neutron sriov11:01
sean-k-mooneywell sriov is jsut a tech its not related to networkign some gpus supprot it too11:01
sean-k-mooneye.g. you can create VF on some gpus and passthough those to the guest11:02
sean-k-mooneyinstead of the PF11:02
zigosean-k-mooney: Maybe for later, I'll implement it then... Thanks for the details!11:04
sean-k-mooneyno worries i just want to make sure you dont implement someitng we want to remove and then have to deal with migratin users off the use of devname11:05
sean-k-mooneyin the pci whitelist11:05
openstackgerritMerged openstack/nova stable/queens: [stable-only] Cap bandit to 1.6.2  https://review.opendev.org/c/openstack/nova/+/76736411:07
*** teoobo_ has quit IRC11:11
*** sapd1 has joined #openstack-nova11:14
*** sapd1 has quit IRC11:18
*** k_mouza has quit IRC11:19
*** k_mouza has joined #openstack-nova11:24
*** k_mouza has quit IRC11:25
*** k_mouza has joined #openstack-nova11:26
*** lbragstad has quit IRC11:38
*** lbragstad has joined #openstack-nova11:42
stephenfinsean-k-mooney: fyi I closed this as WONTFIX https://bugs.launchpad.net/nova/+bug/190850711:47
openstackLaunchpad bug 1908507 in OpenStack Compute (nova) "vif quotas not set for tap interface" [Undecided,Won't fix]11:47
stephenfinlemme know if you disagree11:48
sean-k-mooneyhum for calico that should technically work11:48
sean-k-mooneyi agree we are unlikely to fix it and they shoudl just use neutron qos11:49
openstackgerritElod Illes proposed openstack/nova stable/rocky: [stable-only] Cap bandit to 1.6.2  https://review.opendev.org/c/openstack/nova/+/76649211:54
*** ociuhandu has quit IRC11:55
*** brinzhang_ has joined #openstack-nova11:57
*** brinzhang has quit IRC12:00
*** JamesBenson has joined #openstack-nova12:01
*** JamesBenson has quit IRC12:07
*** JamesBenson has joined #openstack-nova12:07
elodlyarwood, sorry, one more thing o:) the ussuri gate fixing patch looks good now... if you have a minute: https://review.opendev.org/c/openstack/nova/+/76673812:07
lyarwoodelod: LGTM, could you +2/+W as I own it12:13
elodlyarwood: ACK12:14
*** raildo has joined #openstack-nova12:23
gibistephenfin: thanks for the lot of bugtriage the numbers now looks really good12:23
stephenfinnp12:23
gibiI think we have an all time low untriaged bug count12:27
gibi(all time low since I'm looking)12:27
*** ociuhandu has joined #openstack-nova12:31
*** ociuhandu has quit IRC12:39
*** ociuhandu has joined #openstack-nova12:47
*** hemanth_n has quit IRC12:51
*** ociuhandu has quit IRC12:52
Yumenghi gibi, good afternoon.12:56
gibiYumeng: hi12:56
Yumengare you almost in holiday? :)12:56
YumengI saw nova is going to cancel the next two weekly meeting.12:57
gibiI'm off from next week12:57
gibiYumeng: yes, today is the last meeting this yera12:57
gibiyear12:57
Yumengwow, sounds excited and happy.12:58
YumengI'm gonna catch you to discuss vGPU before you off. lol12:58
gibiOK12:59
*** macz_ has joined #openstack-nova12:59
Yumengcurrent nova code has this to support get accel_info to resume guest state when a host is booted. https://review.opendev.org/c/openstack/nova/+/767273/1/nova/virt/libvirt/driver.py#349013:01
Yumengand your concern is if cyborg-agent service starts after nova-compute, nova can not get acc_info as expected,how should we solve this issue?13:01
sean-k-mooneywe shoudl not in nova, at least not entirely13:02
*** k_mouza_ has joined #openstack-nova13:02
sean-k-mooneythe deployment tools shoudl use systemd before/after to order the service starts13:03
gibiI guess we need to make clear in the doc that we have an service restart ordering dependency on cyborg13:03
gibiother than that nova could simply fail to reboot those VMs during service startup that needs cyborg13:04
sean-k-mooneywell ideally the cyborg agent shoudl have a "before: nova-compute.service" requirement13:04
sean-k-mooneynot the other way around13:04
*** macz_ has quit IRC13:04
sean-k-mooneygibi: well we can and should call cyborgs api and do the async wait for the arq bidnigns13:04
sean-k-mooneyif the agent comes online in that tiem and responds it would be fine but yes we coudl skip them if not13:05
sean-k-mooneythe resume guest on host reboot feature does not work unless you use system to force the ordering anyway13:05
sean-k-mooneylibvirtd and openvswitch for example both need to be started before nova-compute13:05
Yumeng"the resume guest on host reboot feature does not work unless you use system to force the ordering anyway" +113:06
gibiToday we don't do async in this code path13:06
gibiat least nova.compute.manager.ComputeManager._get_accel_info does not do that13:06
*** k_mouza has quit IRC13:06
sean-k-mooneyare we just getting the info or rebining on reboot13:06
gibijust getting infor13:07
gibino rebind as far as I see13:07
sean-k-mooneyif its just getting the info there is no depenciy on the agent being running13:07
sean-k-mooneythat info shoudl come form the db13:07
gibiyeah, good point13:07
sean-k-mooneyif we however need the agent to create mdevs13:07
gibibut I'm not familiar with cyborg internal arch13:07
sean-k-mooneyfor exampel then we cant use GET13:07
sean-k-mooneyor at least if we did we would need to have cyborg reflect the status fo the binding as not active or complete13:08
YumengI think we need use GET and also the agent to create mdevs13:08
sean-k-mooneythen we coudl poll or better wait for the externa event form the agent to signal cojmplete of its startup13:08
sean-k-mooneylike we do for spawn13:09
sean-k-mooneyYumeng: did we reject or at lest strong advise agaisnt makeing GET magic in the cyborg api13:09
*** spatel has joined #openstack-nova13:09
sean-k-mooneywhere it would cause the agent to reporvison the attemnet/device if the host restarted13:10
sean-k-mooneyYumeng: i though we were goign to reflect the status in the db and have the agent automaticaly create the devices on start up and signle that its not finised provioning in the api via a status field13:11
sean-k-mooneygibi: that is why i tought we were doing the async event waiting by the way ^13:11
gibiack13:12
sean-k-mooneyYumeng: if we allow GET on the bindings to change state it basically means that cyborg cannot support the READONLY api personces as part of the keyston RBAC/policys effort13:13
openstackgerritLee Yarwood proposed openstack/nova master: WIP db: Add machine_type to instance extras  https://review.opendev.org/c/openstack/nova/+/76753113:14
openstackgerritLee Yarwood proposed openstack/nova master: WIP objects: Add machine_type to instance  https://review.opendev.org/c/openstack/nova/+/76753213:14
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Record the machine_type of instances during init_host  https://review.opendev.org/c/openstack/nova/+/76753313:14
lyarwoodgibi / sean-k-mooney / stephenfin ; http://paste.openstack.org/show/801124/ ^ I'm getting this on instance.save() and can't for the life of me work out why, any ideas? I think I'm missing something basic in the db layer.13:14
* sean-k-mooney clicks13:14
*** spatel has quit IRC13:15
Yumengsean-k-mooney: nope. it is not allowed to change sate in GET13:15
lyarwoodI *think* instance_update_and_get_original is trying to update machine_type in the instance db for some reason13:15
brinzhang_gibi, sean-k-mooney: we found this issue from you comments in vGPU support spec, depends on the "resume_guests_state_on_host_boot=True" config13:15
brinzhang_https://review.opendev.org/c/openstack/nova-specs/+/750116/9/specs/wallaby/approved/support-vGPU-nova-cyborg-interaction.rst#18313:15
sean-k-mooneylyarwood: ill take a look but we shoudl not have a machine_type field. it is ment to be img_machine_type in the instance_system_metadata table13:16
Yumengsean-k-mooney: "the agent automaticaly create the devices on start up" Does this mean cyborg will need another periodic task to sync arq in db with mdevs in the sys path?13:16
sean-k-mooneyso "mapper.column_attrs[key], value"  looks wrong to me13:16
lyarwoodsean-k-mooney: the above series adds machine_type as a field to the instance_extras table etc13:16
lyarwoodsean-k-mooney: I've just missed something somewhere leading to this error when I set it in the instance object and try to save13:17
sean-k-mooney lyarwood right but you shoudl not be doing that13:17
sean-k-mooneylyarwood: wasnt the plan to have no db migraiton requireed for this13:18
sean-k-mooneythat is what we dicussed at the ptg13:18
*** k_mouza_ has quit IRC13:18
lyarwoodI think you suggested adding this to system metadata but I then said it may as well go into instance extras13:19
lyarwoodit's in the spec as an instance extra13:19
openstackgerritElod Illes proposed openstack/nova stable/stein: [stable-only] Cap bandit to 1.6.2  https://review.opendev.org/c/openstack/nova/+/76648713:19
sean-k-mooneyok i think that is not the right way to do this13:19
sean-k-mooneyat least not as a general pattern if we are adding more fileds13:19
sean-k-mooneywhich is why it wanted it in system metadata to begin with13:20
sean-k-mooneysince we will likely want to do this again going forward when we change defaults13:20
*** k_mouza has joined #openstack-nova13:20
*** k_mouza has quit IRC13:20
lyarwoodwhat's the issue with this being an instance extra?13:20
*** k_mouza has joined #openstack-nova13:20
sean-k-mooneyit will require a new column and db migration for every filed for not real value13:21
sean-k-mooneyif we use system metadata we dont require either13:21
sean-k-mooneyits also where the image metadata is currently sotred13:22
sean-k-mooneyso if we recored itn in img_machine_type we dont need to update any code that currently uses it13:22
sean-k-mooneythat way we dont miss anythign and it minimsies the code changes13:22
lyarwoodoh hell no, I'm not overloading img_machine_type13:22
sean-k-mooneythat was the whole point13:23
*** zzzeek has quit IRC13:23
*** k_mouza_ has joined #openstack-nova13:23
*** k_mouza has quit IRC13:23
lyarwoodI didn't get that memo tbh, I've always wanted to track the machine type of the instance seperate to anything else13:23
lyarwoodso we can move it forward in the future without overwriting the image meta etc13:24
sean-k-mooneywe could use a different prefix in the system metadata13:24
sean-k-mooneybut im pretty stongly -1 on a new db column13:25
*** zzzeek has joined #openstack-nova13:25
lyarwoodkk let me look into using it instead13:25
lyarwoodI honestly didn't think a new extra col for this would be a big deal13:26
lyarwoodgiven what's already in there13:26
*** mkrai has quit IRC13:26
sean-k-mooneywell given we want to evenually recored the value of all image properties and the machine_type propsoal was just a scoped down verion of that i think its a problematic design13:26
lyarwoodit wasn't a scoped down version of that at all13:27
lyarwoodI think you're crossing wires here tbh13:27
sean-k-mooneyit was ment to be13:27
lyarwoodit was always just about recording the machine type of existing instances13:28
sean-k-mooneythats what i discussed with stephen when we were deciding if i or you would write the spec13:28
*** k_mouza_ has quit IRC13:28
lyarwoodthe image properties are part of it but not all instances will have the machine type set that way13:28
sean-k-mooneyyes i know13:29
lyarwoodthis was always about capturing the currently used machine type and stashing it somewhere13:29
sean-k-mooneyyep13:29
lyarwoodallowing the config to change13:29
lyarwoodI didn't take from that the need to use the image properties13:29
lyarwoodI wasn't even planning to look at the image properties tbh13:30
lyarwoodJust the defined domains13:30
sean-k-mooneywell i had a very differnet plan for implementing it then13:30
sean-k-mooneyi will also point out form a downstream perspecitve we may need to backport this to 16.2 to support 17 upgrades13:31
sean-k-mooneythat would depend on how we handel the defualt machine type in 17/wallaby13:32
sean-k-mooneyits intended to be q35 for all new installs13:32
lyarwoodI wasn't planning on backporting this13:32
sean-k-mooneyfor upgrade we need to ensure that the machine type does not change13:32
lyarwoodnew installs would be q3513:32
lyarwoodupgrades would be the old default13:33
*** tesseract has quit IRC13:33
lyarwooduntil the env was fully upgraded13:33
lyarwoodand the original machine types stashed13:33
sean-k-mooneyok but that means the ooo work is more involed but that is fine13:33
lyarwoodyeah it's a wrinkle we would need to document13:33
lyarwoodif we go with system metadata then I could backport it13:33
*** tesseract has joined #openstack-nova13:33
sean-k-mooneyyep13:34
lyarwoodso I'll take another look at that13:34
*** hemanth_n has joined #openstack-nova13:35
*** hemanth_n has quit IRC13:35
gibilyarwood, sean-k-mooney: I barely followed the machine type dicussion above, if your decide on a change in direction then please propose a spec update accordingly13:41
sean-k-mooneyack im not sure it sa change in direction at elast form the ptg discussion13:42
sean-k-mooneyhttps://etherpad.opendev.org/p/nova-wallaby-ptg line 196 is the aggreed statement13:42
sean-k-mooney*79613:42
gibisean-k-mooney: anyhow, please keep the spec in sync with the impl :13:43
gibi:)13:43
sean-k-mooneybut the main delta is lyarwood is going to look at using system metatdata table instead of a new column which would be a change in spec13:43
gibiack13:43
sean-k-mooneyi guess its tbd how easy that is but i think it shoudl be a smaller code change in general13:44
*** k_mouza has joined #openstack-nova13:47
*** k_mouza_ has joined #openstack-nova13:50
*** k_mouza has quit IRC13:51
*** k_mouza has joined #openstack-nova13:51
*** k_mouza has quit IRC13:52
*** k_mouza has joined #openstack-nova13:54
*** k_mouza has quit IRC13:55
*** k_mouza_ has quit IRC13:55
*** k_mouza has joined #openstack-nova13:56
*** nweinber has joined #openstack-nova13:58
*** ociuhandu has joined #openstack-nova14:07
lyarwoodsean-k-mooney: sorry I can't recall why I dropped the generic record all the metadata props part of this14:08
sean-k-mooneyit was likely not in the downstream bz/dod14:09
lyarwoodsean-k-mooney: I think it might be due to that approach not capturing the config side14:09
lyarwoodsean-k-mooney: yeah or that14:09
sean-k-mooneyit was going to capture the config too14:09
sean-k-mooneythe idea was to sotre the config value in the imgage prerty filed if it was not set14:09
sean-k-mooneybut its fine to strat with just machien_type14:09
sean-k-mooneyif its easy to repate the same pattern for the other value when we need them14:10
lyarwoodyup, let me rewrite this and update the spec14:11
gibi^^ +114:11
gibi:)14:11
gibion the spec side14:11
lyarwoodyup sorry about this14:11
gibino problem on my side14:12
*** ociuhandu has quit IRC14:13
*** ociuhandu has joined #openstack-nova14:17
*** dklyle has quit IRC14:19
*** ociuhandu_ has joined #openstack-nova14:20
*** jkulik has quit IRC14:20
*** lbragstad has quit IRC14:21
*** ociuhandu has quit IRC14:23
*** lbragstad has joined #openstack-nova14:23
*** k_mouza has quit IRC14:29
openstackgerritYumengBao proposed openstack/nova-specs master: libvirt supports composing cyborg owned vGPU accelerator into domain XML  https://review.opendev.org/c/openstack/nova-specs/+/75011614:39
*** mkrai has joined #openstack-nova14:40
stephenfinbauzas: any chance you could look at https://review.opendev.org/q/topic:%2522sriov-functional-tests%2522+status:open too14:40
stephenfin??14:40
* bauzas clicks14:40
Yumenggibi:  just updated vGPU spec according to today's discuss. Please help to mention in nova's meeting today so cores can know about the update! need their review and comment,Thanks! https://review.opendev.org/c/openstack/nova-specs/+/75011614:45
gibiYumeng: thanks for the update. I will mention the fact that the spec needs core review on today's meeting14:45
*** brinzhang0 has joined #openstack-nova14:46
*** brinzhang_ has quit IRC14:49
*** k_mouza has joined #openstack-nova14:51
openstackgerritLance Bragstad proposed openstack/placement master: Bump oslo.log version to 4.3.0  https://review.opendev.org/c/openstack/placement/+/76022914:52
openstackgerritLance Bragstad proposed openstack/placement master: Pass context objects to oslo.policy directly  https://review.opendev.org/c/openstack/placement/+/76678514:53
openstackgerritAleksey Ulyanov proposed openstack/nova master: Fix graceful shutdown timeout  https://review.opendev.org/c/openstack/nova/+/76492714:54
*** sean-k-mooney has quit IRC14:56
*** ociuhandu_ has quit IRC15:00
*** ociuhandu has joined #openstack-nova15:00
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers  https://review.opendev.org/c/openstack/placement/+/76024015:02
openstackgerritAleksey Ulyanov proposed openstack/nova master: Fix graceful shutdown timeout  https://review.opendev.org/c/openstack/nova/+/76492715:02
*** ociuhandu has quit IRC15:12
*** ociuhandu has joined #openstack-nova15:13
bauzasYumeng: gibi: saw the update but honestly, I won't have time for reviewing this large spec before end of this year15:13
bauzaswhich is for me in 24 hours (with Friday afternoon full of meetings)15:13
gibibauzas: no worries15:17
gibiYumeng knows that we are just about to go for PTO for two weeks15:18
*** ociuhandu has quit IRC15:18
*** sean-k-mooney has joined #openstack-nova15:21
bauzasgibi: actually, I lied, I just commented out loud15:23
bauzasbecause I was horrified by what I saw15:24
*** nautik has joined #openstack-nova15:25
*** ociuhandu has joined #openstack-nova15:33
openstackgerritLee Yarwood proposed openstack/nova stable/victoria: WIP run-evacuate-hook: Use osc to evacuate instances  https://review.opendev.org/c/openstack/nova/+/76757015:35
gibisean-k-mooney: you were right and I was wrong. https://bugs.launchpad.net/nova/+bug/1823251 is a timeout issue really. Now I can reporduce it locally.15:36
openstackLaunchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed]15:36
sean-k-mooneygibi: oh ok so can we do what manial did or is it different?15:36
*** sapd1 has joined #openstack-nova15:36
gibisean-k-mooney: I think we can try what manial did, but I'm affraid at some point we will run out the time budget for the whole test run15:37
gibitest job run15:37
sean-k-mooneyyou mentioned something about scale factor in the bug15:37
gibiyeah, we have a 160sec per test case timeout15:37
sean-k-mooneyah  self.useFixture(timeout.Timeout(15:37
sean-k-mooney            scaling_factor=self.TIMEOUT_SCALING_FACTOR))15:37
gibibut the mysql tests uses 4 * 160 sec15:38
gibiand still not enough15:38
*** macz_ has joined #openstack-nova15:38
gibisomehow the test VM is pretty IO limited15:38
sean-k-mooneyhum well they defalt to 300 second for manila15:38
sean-k-mooneygibi: this only happens on limestone right15:39
gibithe original report said so15:39
gibibut I havent validated that recently15:39
sean-k-mooneyim not sure how new the hardware is, we have had io issue in the past where qos was limiting the disk to 100 IOPS on ovh?15:39
*** macz_ has quit IRC15:42
gibisean-k-mooney: now I see more this happening on vexxhost-ca-ymq-1 provider15:45
*** mlavalle has joined #openstack-nova15:45
gibibut maybe it is just because we run test more there15:45
gibii dont know15:45
gibihm, here is a neutron migration test with postgres failing for the same reason https://b2b5edd1153190007318-36597f05313fd3fdc073d75819a76269.ssl.cf1.rackcdn.com/766204/7/check/neutron-functional-with-uwsgi/8533ed9/job-output.txt15:47
gibiso it is not at all nova or mysql specific15:47
gibithey have ~ 200sec timeout15:47
sean-k-mooneyya it proably affect all project with a lot of migrations15:47
gibi(nova has 160 for the api db test, and 640 for the nova db test)15:47
sean-k-mooneystephenfin's patch shoudl help with that15:47
gibithe squashing migration one?15:48
sean-k-mooneyyep15:48
sean-k-mooneystephenfin: have you pushed that yet ^15:48
gibinova meeting starts in 10 minutes...15:50
stephenfinsean-k-mooney: yup, there's a whole series here https://review.opendev.org/q/topic:%22bp%252Fcompact-db-migrations-wallaby%22+(status:open%20OR%20status:merged) Just waiting for reviews :)15:53
gibistephenfin: nice15:54
openstackgerritStephen Finucane proposed openstack/nova master: tests: Rework some CPU pinning tests to use new config  https://review.opendev.org/c/openstack/nova/+/76757515:54
openstackgerritStephen Finucane proposed openstack/nova master: Remove support for '[DEFAULT] vcpu_pin_set'  https://review.opendev.org/c/openstack/nova/+/76757615:54
openstackgerritStephen Finucane proposed openstack/nova master: Drop support for pinning of host siblings  https://review.opendev.org/c/openstack/nova/+/76757715:54
openstackgerritStephen Finucane proposed openstack/nova master: Remove '[workarounds] disable_fallback_pcpu_query'  https://review.opendev.org/c/openstack/nova/+/76757815:54
openstackgerritStephen Finucane proposed openstack/nova master: Drop VCPU -> PCPU reshape  https://review.opendev.org/c/openstack/nova/+/76757915:54
gibiI can check how much faster the test on my intentionally loaded machine with those compacted migrations15:55
stephenfinsean-k-mooney: Series to drop 'vcpu_pin_set' there ^15:56
*** dklyle has joined #openstack-nova15:56
stephenfinI gave up trying to keep the reshaper around and just removed it15:56
sean-k-mooneyok ill try and look at both today i might start with the vcpu one15:57
openstackgerritMerged openstack/nova master: functional: Add live migration tests for PCI, SR-IOV servers  https://review.opendev.org/c/openstack/nova/+/74695016:05
*** markguz_ has quit IRC16:08
*** macz_ has joined #openstack-nova16:09
gmannstephenfin: sean-k-mooney gibi any final direction on this ?, I am holding the code review for this which is very close to merge.  https://review.opendev.org/c/openstack/nova-specs/+/765797/1/specs/wallaby/approved/modernize-os-hypervisors-api.rst#20516:11
gibigmann: we will touch it today on the meeting16:12
gmanngibi: sorry, forgot about meeting :) thanks for reminder16:13
openstackgerritMerged openstack/nova master: functional: Expand SR-IOV live migration tests with NUMA  https://review.opendev.org/c/openstack/nova/+/74936016:13
*** martinkennelly has joined #openstack-nova16:33
martinkennellyare you in charge of SDN room?16:33
*** mkrai has quit IRC16:35
gibistephenfin: your db migration compating patches make the runtime of those test down by more than 75%16:48
stephenfinhurrah16:48
stephenfinthat makes sense tbf16:48
stephenfinthere is also an API DB compaction, but I guess those are separate tests16:49
gibiyes, there are unit tests for the nova db and functional test for the api db (dont ask why)16:49
gibiI measured the nova db ones now16:50
gibianyhow your series will eventually solve https://bugs.launchpad.net/nova/+bug/182325116:50
openstackLaunchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed]16:50
gibiI think16:50
gibibut until that I will push timeout increases16:50
openstackgerritGhanshyam proposed openstack/nova master: DNM: testing minbw new tests  https://review.opendev.org/c/openstack/nova/+/76758716:57
gibigmann: good idea ^^17:01
openstackgerritBalazs Gibizer proposed openstack/nova master: Increase timeout for db migration tests  https://review.opendev.org/c/openstack/nova/+/76759017:06
gibisean-k-mooney, stephenfin ^^17:07
stephenfinLGTM. Do you want to run that a couple of times through CI or merge and let it test itself?17:08
gibistephenfin: merge it then we can revert if it does not help17:09
gibi:)17:09
stephenfinack, +2 from moi17:09
sean-k-mooneyah so we increase the scaling factor for the migration and decresss it for other test17:09
sean-k-mooneyoh no17:10
gibisean-k-mooney: nope, it is increase it for all our migration test just different amount17:10
sean-k-mooneywe set it to 5 for the main db and 2 for the api17:10
gibithe api db mifration tests were defaulted to 117:10
sean-k-mooneyright so increased for both17:10
sean-k-mooneyok cool17:10
*** belmoreira has quit IRC17:10
gibiyepp17:10
sean-k-mooneyand you have reference stephens series too17:11
sean-k-mooneyya this looks good to me17:11
gibiif it doesn't really help then I can try to look at serializing them17:11
gibibut it would be better to merge stephen's series instead17:11
gibianyhow this is the end of my day see you all tomorrow17:12
stephenfino/17:12
* stephenfin is on PTO again tomorrow, fwiw, so have a good weekend o/17:12
gibistephenfin: o/17:12
*** k_mouza has quit IRC17:17
*** k_mouza has joined #openstack-nova17:18
*** ralonsoh has quit IRC17:23
*** jawad_axd has quit IRC17:33
*** jawad_axd has joined #openstack-nova17:33
*** jawad_axd has quit IRC17:35
*** jawad_axd has joined #openstack-nova17:36
*** jawad_axd has quit IRC17:41
*** ociuhandu_ has joined #openstack-nova17:41
*** ociuhandu has quit IRC17:44
*** ociuhandu_ has quit IRC17:45
*** gyee has joined #openstack-nova17:49
*** hamalq has joined #openstack-nova17:50
*** hamalq_ has joined #openstack-nova17:52
*** derekh has quit IRC17:53
*** hamalq has quit IRC17:56
gmanngibi: yeah, we can check in advance specially if those new test needs to skip in multi-cell job17:57
*** vishalmanchanda has quit IRC18:06
*** rpittau is now known as rpittau|afk18:20
sean-k-mooneyartom: i see you added your self to https://review.opendev.org/c/openstack/nova/+/76736818:21
sean-k-mooneythat is required before https://review.opendev.org/c/openstack/neutron/+/766277 can merge on the neutron side18:22
sean-k-mooneyand we need to backport it to at least train18:22
*** martinkennelly has quit IRC18:22
sean-k-mooneyits related to https://review.opendev.org/c/openstack/nova/+/602432 also18:23
sean-k-mooneywe are currenly up to 6 pathces we need to backport to fix live migration18:23
sean-k-mooney3 on nova and 3 on neutron18:23
sean-k-mooney2 are merged on the neutron side already and 1 on the nova side18:24
sean-k-mooneyif we need to go before train we will need to backport thing in os-vif also18:24
sean-k-mooneyanyway im going to be calling it a day soon then going on PTO till january so ill respond it comments when im back if there are any18:25
sean-k-mooneygibi: just saw this in the conflicts list https://review.opendev.org/c/openstack/nova/+/709658 we proably should arcive that just to not distract peopel with it18:27
*** tesseract has quit IRC18:28
*** k_mouza has quit IRC18:29
sean-k-mooneyoh enjoy the break all o/ i may or may not be around over the next 2 weeks but im on pto until the 4th of january18:29
lyarwood\o enjoy either way, catch you in the new year18:55
* lyarwood -> drops18:55
artomsean-k-mooney, yeah, seemed like a thing I need to look at19:10
artom(Sorry for the lag, was doing some errands and getting a haircut, before Shutdown 2 Electric Boogalloo after Christmas)19:10
sean-k-mooneyno worries19:11
*** andrewbonney has quit IRC19:11
sean-k-mooneytl;dr it only every worked because of buggy behavior in neutron that when fix means we need to actully wait properly in nova19:11
sean-k-mooneybasically we were migration before because fo a network-vif-plugged event form the dhcp agent19:12
sean-k-mooneynot form the l2 agent that we were waiting for19:12
artomsean-k-mooney, I suspect this'll be another "Artom and Sean spent 2 hours on GMeet" kind of situation :)19:12
sean-k-mooneyabout an hour and with radolfo but ya19:12
sean-k-mooneyi had a suption this was true for a while but we found the atcual code that was at fault friday last week19:13
sean-k-mooneywe have had a much of unxepcect event in the logs for a few releases and this is why many of them happened19:14
sean-k-mooneyits all related to the multipe port binding work in rocky19:14
*** dviroel has quit IRC19:43
*** spatel has joined #openstack-nova20:06
*** slaweq has quit IRC20:59
*** ociuhandu has joined #openstack-nova21:35
*** k_mouza has joined #openstack-nova21:39
*** nweinber has quit IRC21:40
*** k_mouza has quit IRC21:47
*** ociuhandu has quit IRC21:59
*** tbachman has quit IRC22:01
*** tbachman has joined #openstack-nova22:03
openstackgerritGhanshyam proposed openstack/placement master: Remove deprecated [placement]/policy_file config option  https://review.opendev.org/c/openstack/placement/+/76440922:12
openstackgerritGhanshyam proposed openstack/placement master: [goal] Deprecate the JSON formatted policy file  https://review.opendev.org/c/openstack/placement/+/76439422:12
*** rcernin has joined #openstack-nova22:15
*** ociuhandu has joined #openstack-nova22:17
*** slaweq has joined #openstack-nova22:21
*** ociuhandu has quit IRC22:28
*** ociuhandu has joined #openstack-nova22:29
*** slaweq has quit IRC22:33
*** ociuhandu has quit IRC22:33
openstackgerritGhanshyam proposed openstack/nova stable/rocky: DNM: testing gate  https://review.opendev.org/c/openstack/nova/+/76702722:38
openstackgerritmelanie witt proposed openstack/nova master: Add functional regression test for bug 1853009  https://review.opendev.org/c/openstack/nova/+/69501222:38
openstackbug 1853009 in OpenStack Compute (nova) ussuri "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard)22:38
openstackgerritmelanie witt proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker  https://review.opendev.org/c/openstack/nova/+/69518722:38
openstackgerritmelanie witt proposed openstack/nova master: Invalidate provider tree when compute node disappears  https://review.opendev.org/c/openstack/nova/+/69518822:38
openstackgerritmelanie witt proposed openstack/nova master: Prevent deletion of a compute node belonging to another host  https://review.opendev.org/c/openstack/nova/+/69480222:38
openstackgerritmelanie witt proposed openstack/nova master: Fix inactive session error in compute node creation  https://review.opendev.org/c/openstack/nova/+/69518922:38
*** martinkennelly has joined #openstack-nova22:43
*** dviroel has joined #openstack-nova22:58
*** ociuhandu has joined #openstack-nova23:00
openstackgerritLance Bragstad proposed openstack/placement master: Bump oslo.log version to 4.3.0  https://review.opendev.org/c/openstack/placement/+/76022923:07
openstackgerritLance Bragstad proposed openstack/placement master: Pass context objects to oslo.policy directly  https://review.opendev.org/c/openstack/placement/+/76678523:07
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers  https://review.opendev.org/c/openstack/placement/+/76024023:07
*** ociuhandu has quit IRC23:12
*** lbragstad has quit IRC23:18
*** raildo has quit IRC23:31
*** tbachman has quit IRC23:31
*** tbachman has joined #openstack-nova23:34
*** ociuhandu has joined #openstack-nova23:41
*** tosky has quit IRC23:43
*** ociuhandu has quit IRC23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!