Tuesday, 2020-11-17

*** tosky has quit IRC00:06
*** lemko4 has joined #openstack-nova00:20
*** lemko has quit IRC00:21
*** lemko4 is now known as lemko00:21
*** tbachman has quit IRC00:28
*** tbachman has joined #openstack-nova00:36
*** k_mouza has joined #openstack-nova00:45
*** k_mouza has quit IRC00:49
*** macz_ has quit IRC00:56
*** zzzeek has quit IRC01:06
*** zzzeek has joined #openstack-nova01:09
*** mlavalle has quit IRC01:10
*** iurygregory has quit IRC01:32
*** sapd1_x has joined #openstack-nova01:46
*** sapd1_x has quit IRC01:51
*** sapd1_x has joined #openstack-nova01:58
*** sapd1_x has quit IRC02:02
*** sapd1_x has joined #openstack-nova02:06
*** sapd1_x has quit IRC02:15
*** yingjisun has joined #openstack-nova02:24
*** sapd1_x has joined #openstack-nova02:37
*** sapd1_x has quit IRC02:41
*** rcernin has quit IRC02:44
*** artom has quit IRC02:55
*** xinranwang has joined #openstack-nova03:00
xinranwanggibi:  Hi gibi, could you please review the smartnic spec when you got time? ;)03:00
*** rcernin has joined #openstack-nova03:06
*** rcernin has quit IRC03:13
*** sapd1_x has joined #openstack-nova03:14
*** sapd1_x has quit IRC03:14
*** mkrai has joined #openstack-nova03:16
*** sapd1 has joined #openstack-nova03:17
*** adrianc has quit IRC03:22
*** adrianc has joined #openstack-nova03:38
*** rcernin has joined #openstack-nova03:43
*** adrianc has quit IRC03:47
*** rcernin has quit IRC03:48
*** vishalmanchanda has joined #openstack-nova03:52
*** rcernin has joined #openstack-nova03:53
*** rcernin has quit IRC03:54
*** rcernin has joined #openstack-nova03:54
*** donnyd has quit IRC04:07
*** donnyd has joined #openstack-nova04:08
*** adrianc has joined #openstack-nova04:17
*** adrianc has quit IRC04:23
openstackgerritBrin Zhang proposed openstack/nova master: Cyborg shelve/unshelve support  https://review.opendev.org/72956304:26
*** ratailor has joined #openstack-nova04:29
*** k_mouza has joined #openstack-nova04:46
*** johnsom has quit IRC04:46
*** johnsom has joined #openstack-nova04:49
*** k_mouza has quit IRC04:50
*** rcernin has quit IRC04:56
*** rcernin has joined #openstack-nova04:57
*** sapd1 has quit IRC05:06
*** fyx has quit IRC05:11
*** fyx has joined #openstack-nova05:14
*** johnsom has quit IRC05:18
*** johnsom has joined #openstack-nova05:19
*** zzzeek has quit IRC05:25
*** zzzeek has joined #openstack-nova05:27
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-nova05:33
*** zzzeek has quit IRC05:50
*** yingjisun has quit IRC05:51
*** zzzeek has joined #openstack-nova05:51
*** sapd1 has joined #openstack-nova05:52
*** JamesBenson has quit IRC05:53
*** zzzeek has quit IRC06:10
*** zzzeek has joined #openstack-nova06:11
*** flaviof has quit IRC06:11
*** flaviof has joined #openstack-nova06:12
*** johnsom has quit IRC06:27
*** johnsom has joined #openstack-nova06:27
*** yingjisun has joined #openstack-nova06:52
*** rcernin has quit IRC07:10
*** rcernin has joined #openstack-nova07:13
*** zzzeek has quit IRC07:14
*** adrianc has joined #openstack-nova07:15
*** zzzeek has joined #openstack-nova07:16
*** rm_work has quit IRC07:26
*** rm_work has joined #openstack-nova07:28
*** zzzeek has quit IRC07:37
*** zzzeek has joined #openstack-nova07:38
*** ociuhandu has joined #openstack-nova07:40
*** ociuhandu has quit IRC07:42
*** ociuhandu has joined #openstack-nova07:43
*** ociuhandu has quit IRC07:43
*** ralonsoh has joined #openstack-nova07:43
*** rcernin has quit IRC07:44
*** rcernin has joined #openstack-nova07:47
*** macz_ has joined #openstack-nova07:57
gibixinranwang: when I got time I will review :)07:58
*** slaweq has joined #openstack-nova07:58
gibiI mean I'm trying to get to it07:58
*** sapd1 has quit IRC07:58
*** macz_ has quit IRC08:02
*** rpittau|afk is now known as rpittau08:05
*** dklyle has quit IRC08:06
*** sapd1 has joined #openstack-nova08:08
*** tesseract has joined #openstack-nova08:14
*** tobiash has quit IRC08:15
*** tobiash has joined #openstack-nova08:16
*** andrewbonney has joined #openstack-nova08:17
*** rcernin has quit IRC08:20
*** ociuhandu has joined #openstack-nova08:22
*** zzzeek has quit IRC08:26
xinranwanggibi:  thanks~08:29
*** zzzeek has joined #openstack-nova08:30
*** rcernin has joined #openstack-nova08:33
*** ociuhandu has quit IRC08:38
*** zzzeek has quit IRC08:38
*** zzzeek has joined #openstack-nova08:40
*** rcernin has quit IRC08:40
*** zzzeek has quit IRC08:48
*** zzzeek has joined #openstack-nova08:51
*** mgoddard has joined #openstack-nova08:52
*** ociuhandu has joined #openstack-nova08:52
*** iurygregory has joined #openstack-nova08:52
*** tosky has joined #openstack-nova08:56
*** k_mouza has joined #openstack-nova09:00
*** dtantsur|afk is now known as dtantsur09:04
*** k_mouza has quit IRC09:05
bauzasgood spec review day, Nova09:08
*** links has joined #openstack-nova09:15
*** links has quit IRC09:22
*** martinkennelly has joined #openstack-nova09:25
*** sean-k-mooney has quit IRC09:27
*** sean-k-mooney has joined #openstack-nova09:27
*** mgoddard has quit IRC09:29
*** rcernin has joined #openstack-nova09:32
*** xek has joined #openstack-nova09:36
*** rcernin has quit IRC09:40
*** ociuhandu has quit IRC09:42
*** ociuhandu has joined #openstack-nova09:51
*** ociuhandu has quit IRC09:56
*** zzzeek has quit IRC09:57
*** ociuhandu has joined #openstack-nova09:58
bauzasgibi: easy peasy to review for a spec review day https://review.opendev.org/#/c/756242/10:00
*** zzzeek has joined #openstack-nova10:02
*** rcernin has joined #openstack-nova10:04
*** rcernin has quit IRC10:09
*** jamesdenton has quit IRC10:10
*** jamesdenton has joined #openstack-nova10:10
*** whoami-rajat__ has joined #openstack-nova10:12
*** songwenping_ has joined #openstack-nova10:14
*** yingjisun has left #openstack-nova10:15
*** artom has joined #openstack-nova10:19
*** mgoddard has joined #openstack-nova10:19
*** zzzeek has quit IRC10:38
*** zzzeek has joined #openstack-nova10:40
gibibauzas: done10:43
*** k_mouza has joined #openstack-nova10:51
stephenfingibi, bauzas: I know it's spec review day, but could take a real quick look at https://review.opendev.org/#/c/762898/ while it's fresh in my mind?10:54
bauzasgibi: cool, thanks10:54
bauzasstephenfin: ack, I clicked for it, I'm just looking at a spec now10:54
stephenfinall good, thanks :)10:55
openstackgerritMerged openstack/nova-specs master: Re-proposes Routed Networks  https://review.opendev.org/75624210:58
*** rcernin has joined #openstack-nova10:58
*** ociuhandu has quit IRC11:07
*** ociuhandu has joined #openstack-nova11:07
gibistephenfin: So the intention of this exception is to fail the startup of the compute service this why it is never caught?11:09
stephenfingibi: Yes, exactly. It's presence indicates a misconfiguration meaning we should hard fail11:10
stephenfintbc, we will only attempt to retrieve this information on the host if nova.conf tells us to11:13
*** zzzeek has quit IRC11:17
*** zzzeek has joined #openstack-nova11:20
*** zzzeek has quit IRC11:27
*** zzzeek has joined #openstack-nova11:28
*** mkrai has quit IRC11:29
*** macz_ has joined #openstack-nova11:33
*** ociuhandu has quit IRC11:35
*** ociuhandu has joined #openstack-nova11:36
gibistephenfin: thanks. Then my only question is about underscores :) https://review.opendev.org/#/c/762898/1/nova/tests/unit/virt/libvirt/test_driver.py@2750511:38
sean-k-mooneygibi: just reading the comment11:38
stephenfinoh, there shouldn't be three /o\11:38
sean-k-mooneygibi: that is the style that stephenfin  uses11:38
*** macz_ has quit IRC11:38
gibisean-k-mooney: I know he likes underscores, but I think there is one extra now11:38
openstackgerritStephen Finucane proposed openstack/nova master: Add missing exception  https://review.opendev.org/76289811:39
sean-k-mooneyyep11:39
stephenfindone; my mistake /o\11:39
sean-k-mooneytest___ instead of test__11:39
sean-k-mooneyim still fine with this11:40
gibistephenfin: now you have one underscore, which is fine by me, but I guess your style would cause two underscores11:40
stephenfinit would, but I know not everyone likes that so I'm okay to stick with one11:40
*** ociuhandu has quit IRC11:41
*** sapd1 has quit IRC11:41
gibiok11:41
bauzassean-k-mooney: FWIW, I think I drafted a good nova-cyborg relationship in https://review.opendev.org/#/c/750116/9/specs/wallaby/approved/support-vGPU-nova-cyborg-interaction.rst@18311:41
bauzasbased on the open issues we have atm11:42
bauzaswith the mdev persistence and the likes11:42
bauzasthis would hugely benefit to Cyboth11:42
bauzascyborg*11:42
bauzassean-k-mooney: anyway, lunch11:42
gibibauzas: does it have an impact on https://review.opendev.org/#/c/742785/6..9/specs/wallaby/approved/support-sriov-smartnic.rst too?11:42
openstackgerritStephen Finucane proposed openstack/nova master: api-ref: Move 'os-agents' API to obsolete section  https://review.opendev.org/75572911:42
openstackgerritStephen Finucane proposed openstack/nova master: virt: Remove 'change_instance_metadata' API  https://review.opendev.org/74931611:42
openstackgerritStephen Finucane proposed openstack/nova master: virt: Remove 'reset_network' API  https://review.opendev.org/74931511:42
openstackgerritStephen Finucane proposed openstack/nova master: virt: Remove 'get_all_bw_counters' API  https://review.opendev.org/74931211:42
openstackgerritStephen Finucane proposed openstack/nova master: objects: Remove 'BandwidthUsage', 'BandwidthUsageList'  https://review.opendev.org/75911411:42
bauzasgibi: I haven't reviewed this spec yet11:42
gibibauzas: nvm, I will read your comment in the vGPU spec11:43
bauzasgibi: but tl;dr: I propose to leave cyborg manage the mdevs and just ask nova to bind them to the guest definition11:43
bauzaswhich is sometimes we already have11:43
bauzass/sometimes/something11:43
sean-k-mooneybauzas: ill go read it shortly11:43
bauzasgibi: sean-k-mooney: the only difference would be the inventory reporting11:44
bauzasbut this way, this would allow cyborg to keep a persisted state of mdevs, which is something I don't wanna managed in nova and which creates problems for us11:44
*** rcernin has quit IRC11:45
bauzaswe haven't discussed this at the PTG, I reckon, but I feel we would all benefit of this solution11:45
*** tbachman has quit IRC11:45
bauzaslike, my customers would have the choice to either manage the mdev fleet by themselves and just use nova, or play with cyborg11:45
sean-k-mooney bauzas im not sure they __need__ to precreate the mdev but i agree they could and it would ok to do so11:46
bauzassean-k-mooney: precreating is way better for many reasons11:46
bauzasI mean, precreating after a config modification, that's it11:46
sean-k-mooneyi just read you last comment i need to read the rest of the doc and your comments for context11:46
bauzasbecause once you're done with the config, your inventory won't change11:47
sean-k-mooneyi think it was on v2 the last time i looked11:47
bauzaseither way, I need to go lunching11:47
bauzasmy kids need to go to school11:47
sean-k-mooneyya cyborg might want to be more dynmaic but i agree it has advantages11:47
sean-k-mooneysimplcity being one of them11:47
*** brinzhang0 has joined #openstack-nova11:50
*** ociuhandu has joined #openstack-nova11:53
*** brinzhang_ has quit IRC11:53
*** brinzhang_ has joined #openstack-nova11:55
*** songwenping__ has joined #openstack-nova11:55
*** spatel has joined #openstack-nova11:56
*** slaweq has quit IRC11:56
*** slaweq has joined #openstack-nova11:57
*** brinzhang0 has quit IRC11:58
*** rcernin has joined #openstack-nova11:58
*** songwenping_ has quit IRC11:59
*** spatel has quit IRC12:01
*** mgoddard has quit IRC12:01
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types  https://review.opendev.org/72809412:08
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.text_type (1/2)  https://review.opendev.org/72810912:08
*** xinranwang has quit IRC12:09
*** JamesBenson has joined #openstack-nova12:12
*** raildo_ has joined #openstack-nova12:14
*** ratailor has quit IRC12:14
*** rcernin has quit IRC12:14
*** mgoddard has joined #openstack-nova12:15
*** raildo has quit IRC12:16
openstackgerritTakashi Natsume proposed openstack/nova master: Remove six.text_type (2/2)  https://review.opendev.org/72811712:20
*** lpetrut has joined #openstack-nova12:24
openstackgerritStephen Finucane proposed openstack/nova-specs master: Add modernize-os-hypervisors-api spec  https://review.opendev.org/75510912:28
*** ociuhandu has quit IRC12:28
*** Yumeng has joined #openstack-nova12:28
Yumengsean-k-mooney, @gi12:29
*** xinranwang has joined #openstack-nova12:30
Yumenggood morning nova ^12:31
sean-k-mooneyo/12:31
Yumenghi sean, I wanna discuss some issues in the vGPU specs with you. https://review.opendev.org/#/c/750116/12:33
sean-k-mooneyim reviewing it currently12:36
*** Luzi has joined #openstack-nova12:36
sean-k-mooneybut sure if you have topics bring them up12:36
*** zzzeek has quit IRC12:39
Yumengsean-k-mooney: take your time.  You can directly leave comments on the patch or ping me later here(if I am still here ^^). My question will be mainly on:  3. Who creates mdev device in the sys path?(proposed change part) which do you prefer12:41
*** zzzeek has joined #openstack-nova12:41
sean-k-mooneysorry had to step away for a phone call, im stell reading the options now, i think im leanign twords cyborg creating them but i need to read your content and sylvain's comments12:44
sean-k-mooney/stell/still/12:44
bauzasYumeng: fwiw, I already provided my thoughts12:46
bauzasand i prefer a solution with cyborg precreating the mdevs12:46
bauzas(and just ask nova to use an existing mdev)12:47
gibibauzas, Yumeng: I've also just pushed my comments on the vGPU spec12:48
Yumengbauzas:  I am reviewing your comments now.   yes, solution 2 in the spec is a cyborg precreating the mdevs. seems that's what you guys like.12:49
Yumengbauzas: ok.thank you, I will take a look first12:50
Yumenggibi: ok.thank you, I will take a look first12:50
*** ociuhandu has joined #openstack-nova12:59
gibibauzas: if you want an easy spec: https://review.opendev.org/#/c/755477/ :)13:01
*** rcernin has joined #openstack-nova13:02
*** rcernin has quit IRC13:02
*** ociuhandu has quit IRC13:04
*** ociuhandu has joined #openstack-nova13:04
bauzasgibi: nit : s/Sylvian/Sylvain (Sylvian(e) is a woman name ;) )13:11
gibibauzas: sorry13:11
bauzasgibi: heh no worries ;)13:11
* bauzas doesn't know how to say your first name ;)13:12
sean-k-mooneygibi: bauzas  i dont think we shoudl be treating mdevs and vf similarly unless you are suggesting we start tracking mdevs in teh pci tracker13:12
sean-k-mooneyso that we persist them there and have nova precreate them13:12
sean-k-mooneyto provide stable names13:13
sean-k-mooneyand other things like numa affinity13:13
sean-k-mooneye.g. just add a type_mdev to the list13:13
bauzaswhy ?13:13
bauzasanyway, I need to look at other specs13:13
gibisean-k-mooney: I don't suggest to plug mdevs into the pci tracker, especially that cyborgs would like to track them13:14
sean-k-mooneybecause mdev are not ment to work like vf in that they framework was desigined to have them dynmicaly created at runtime by higher level orcestors13:14
sean-k-mooneygibi: well my point is that without cycborg i dont think its valid to say they should be precreated13:14
bauzasgibi: me too13:15
*** ociuhandu has quit IRC13:15
sean-k-mooneythat is  just not how they were ment to be used13:15
bauzasgibi: sean-k-mooney: honestly, I would like to stop supporting to create mdevs in nova13:15
bauzasbut first, we would need to support it by TripleO13:15
gibisean-k-mooney: does libvirt handles mdevs differently than pci?13:15
sean-k-mooneyright i dont think we shoudl do that not without providing a real alternitive13:15
sean-k-mooneygibi: yes it does13:15
bauzasgibi: mdevs are a bit different13:16
bauzasbut,13:16
sean-k-mooneygibi: libvirt alows them to be allcoated dymically ti does not do the same for vfs13:16
bauzasI don't see why we couldn't just tell 'sorry, but please precreate them'13:16
sean-k-mooneybauzas: that would be a major regression in functionality for one13:16
sean-k-mooneyand be an upgrade issue13:16
bauzasgiven we only support one type for GPU, when you tell which type you want to get for a pGPU, then we could just create them13:16
sean-k-mooneybut the main reason is most device dont support one type13:17
bauzasanyway, I need to be off for 10 mins13:17
bauzas(someone is around my home)13:17
sean-k-mooneythe whole we only support one type per PGPU is a limiatieon in nvidias solution that is not representivie of the mdev framework as a whole13:18
*** artom has quit IRC13:18
sean-k-mooneyif we support generic mdevs in teh future and in the cyborg case in particalar it dose not hold true that all device will only support 1 mdev type13:18
taccosean-k-mooney: fyi, still no luck. but all in all some more informations. I can see 248available cpus in the vm but everything above id63 are marked as offline.13:22
gibihm, I more and more like the idea to hide this complexity from nova in cyborg. Then cyborg is free to pre-create or create on the fly the mdev at arq mind. Nova don't have to worry about it (except the recreation case at reboot, see my comment in the spec)13:22
gibis/mind/bind/13:23
sean-k-mooneytacco: ok so that is pointing to the guest kernel13:23
sean-k-mooneytacco: well most likely the kernel13:23
sean-k-mooneyhave you tried onlining them via /sys13:23
sean-k-mooneygibi: yep i was noting that in the spec13:24
sean-k-mooneysolution 2 id fine if cyborg does the precreate when we bind at teh conductor13:24
sean-k-mooneyits not ok if we need to do an api call on the compute13:24
sean-k-mooneythey can also do option 3 and precreate them at agent start13:25
gibisean-k-mooney: yeah, I assumed that cyborg provides the mdev to as in the same way as provides the pci for an fpga13:25
gibis/to as/ to us/13:25
sean-k-mooneyyep in the arq when it completes the binding13:25
taccosean-k-mooney: yes, but then i got a mem error.13:25
sean-k-mooneywhcih we poll for on the compute13:25
gibisean-k-mooney: exactly13:25
sean-k-mooneyso its really just the uuid which we then stick in the xml13:25
gibisean-k-mooney: and this uuid needs to be stable from nova perspective at reboot, or else nova needs to update the xml13:27
sean-k-mooneywell we regenerate teh xml form scratch on reboot so13:27
sean-k-mooneyit  could change13:27
sean-k-mooneywe wont notice13:27
*** Yumeng has quit IRC13:28
openstackgerritMerged openstack/nova-specs master: Support interface attach with qos ports  https://review.opendev.org/75547713:28
gibiI mean hypervisor reboot not VM reboot13:28
sean-k-mooneyoh same power-on for reasons calls hard reboot13:28
gibihm, that could work then13:29
sean-k-mooneyreasons being we required spwan and hard-reboot before we required stop start in the virt driver13:29
sean-k-mooneyand we never changed libvirt because it just worked13:29
gibido we rebind or requery the arq from cyborg at VM hard reboot?13:30
gibiwe need that to pick up a the new uuid13:30
sean-k-mooneythat was a question i was going to ask in the spec13:31
sean-k-mooneyi cant rember if we get the arq or not13:31
bauzas(still on and off, but I see your convo, will be back in 20 mins-ish)13:31
sean-k-mooneywe do not rebind ports on hard reboot13:31
sean-k-mooneyfor cinder block devices lyarwood  would have to correct me but i think we cached the attachmemnt but maybe we recreate them13:32
sean-k-mooneygibi: point being that we had to solve this for fpgas already13:32
sean-k-mooneyso i think what ever we did there should be suffienct13:32
gibisean-k-mooney: yeah, I agree13:33
gibiI have to jump on a call back later13:33
sean-k-mooneycool13:33
sean-k-mooneyim leaving comments on the spec but i think we agree cyborg should handel the mdev creation and just tell us13:34
*** nweinber has joined #openstack-nova13:34
sean-k-mooneywhere it dose that (agent start or bind) is up to them so long as it work for all lifecycle events13:34
lyarwoodsean-k-mooney: we don't recreate volume attachments on hard reboot13:35
gibisean-k-mooney: agre13:35
sean-k-mooneywe cache the conection info in the db13:35
sean-k-mooney?13:35
lyarwoodsean-k-mooney: the libvirt driver disconnects volumes from the host and reconnects them but that's it13:35
lyarwoodsean-k-mooney: yeah we cache the attachment id and associated connection_info13:35
sean-k-mooneyya13:36
sean-k-mooneyso i tought we could hard reboot without external api calls13:36
lyarwoodsean-k-mooney: the only way to refresh that at the moment is via a move operation, I've argued that we should do it during a hard reboot in the past13:36
sean-k-mooneybut im not sure what the case is for cyborg13:36
sean-k-mooneylyarwood: yep that is why i was asking i knew you wanted to change it but didnt knwo if you wanted to add or remove the caching13:37
lyarwoodsean-k-mooney: remove the connection_info caching13:37
lyarwoodsean-k-mooney: but if we ever get there is another thing13:38
sean-k-mooneyso we would then have a call to cinder on every hard reboot13:38
lyarwoodsean-k-mooney: yeah13:38
sean-k-mooneyi guess if the cinder service is down the storeage backend may also be down13:38
sean-k-mooneyso even with the cache we are not guareteed to be able to connect13:39
lyarwoodsean-k-mooney: really depends on the backend, most could be up still13:39
lyarwoodsean-k-mooney: we could at least attempt to refresh tbh13:39
*** k_mouza has quit IRC13:41
sean-k-mooneyya ceph and other likely are not running on the same hosts as the cinder services13:41
*** ociuhandu has joined #openstack-nova13:45
*** ociuhandu has quit IRC13:50
*** mlavalle has joined #openstack-nova13:56
*** ociuhandu has joined #openstack-nova13:57
taccohm.. struggling aroung with aggregates, strange behavior. i tought if i create a aggreagate with a key and attatch that key to a flavor only hosts with the specific flavors are on the aggregate hosts.14:01
sean-k-mooneytacco: only if you use the placement version14:02
taccoor is there anything else to do? For now only a couple of HVs already have the new filter. But i guess only the hosts in the aggregate group needs this filter, or to be aware of.14:02
sean-k-mooneyif you use the fitler you need to add key=false to every other flavor14:02
taccoah.. ok, thanks. that could help for the moment :)14:02
sean-k-mooneyhttps://docs.openstack.org/nova/latest/reference/isolate-aggregates.html14:02
taccoi tought absence of this flavor also implies that it can not be on the hosts.14:03
taccothanks for pointing me there.14:03
*** songwenping__ has quit IRC14:03
sean-k-mooneytacco: nope that what many assume but its not how it works14:03
openstackgerritLee Yarwood proposed openstack/nova-specs master: WIP - Image and flavor defined ephemeral storage encryption  https://review.opendev.org/75228414:03
sean-k-mooneytacco: this is basically unmaintained and im not sure if it still works(it should) but https://opendev.org/x/nfv-filters provdes a filter that does what you want14:04
*** songwenping__ has joined #openstack-nova14:04
sean-k-mooneyhttps://opendev.org/x/nfv-filters/src/branch/master/nfv_filters/nova/scheduler/filters/aggregate_instance_type_filter.py14:04
*** songwenping__ has quit IRC14:04
sean-k-mooneydocs are here https://opendev.org/x/nfv-filters/src/branch/master/doc/source/scheduler_filters/aggregate-instance-type-filter.rst14:04
sean-k-mooneyif you dont have placment avaiable with the prefilter then that shoudl bridge the gap14:05
*** songwenping__ has joined #openstack-nova14:05
taccook, will read first and decide then. :) but.. yeah.. found a bug that was already reported. yay14:05
sean-k-mooneyits actully more powerful the what you can do with placemnt or the code in tree but its out of tree and not maintained by me really anymore14:06
taccoyes.. can understand that. I would like to stick as close as i can to upstream and maintained versions.14:08
taccoonly if there is no other way out i would think about that.. but our first solution we had was way to full with cherry-picked stuff.. and deployment was only done by manual actions.. i hate it. Now we have a nice fully automated and functional setup i don't want to miss that anymore. :D14:09
lyarwoodexit14:15
lyarwoodargh14:15
kashyap"You can leave anytime, but you can never exit."14:16
* kashyap ducks14:16
*** xek has quit IRC14:18
*** Luzi has quit IRC14:18
*** xek has joined #openstack-nova14:18
*** macz_ has joined #openstack-nova14:20
*** k_mouza has joined #openstack-nova14:24
*** k_mouza has quit IRC14:24
*** macz_ has quit IRC14:24
*** k_mouza has joined #openstack-nova14:24
*** brinzhang_ has quit IRC14:26
*** ociuhandu has quit IRC14:30
*** lbragstad__ has quit IRC14:37
*** sapd1 has joined #openstack-nova14:38
*** lbragstad has joined #openstack-nova14:40
*** tbachman has joined #openstack-nova14:42
taccosean-k-mooney: where do i have to set the  scheduler.enable_isolated_aggregate_filtering nova.conf on the HVs?14:50
sean-k-mooneyno nova.conf on the schduler14:50
*** songwenping_ has joined #openstack-nova14:50
sean-k-mooneyso on the contollers14:51
openstackgerritMerged openstack/nova master: Restore retrying the RPC connection to conductor  https://review.opendev.org/76263314:51
openstackgerritMerged openstack/nova master: Fix the vGPU dynamic options race  https://review.opendev.org/75847014:51
openstackgerritMerged openstack/nova master: api-ref: Move 'os-agents' API to obsolete section  https://review.opendev.org/75572914:51
openstackgerritLee Yarwood proposed openstack/nova-specs master: WIP libvirt: Allow the default machine type to be changed  https://review.opendev.org/76219914:52
lyarwoodkashyap: ^ btw you might be interested in this, I've used some text from a former spec of yours14:53
taccook i see. thansk will give it a try. :)14:53
*** songwenping__ has quit IRC14:53
kashyaplyarwood: Ack; will queue14:53
openstackgerritLee Yarwood proposed openstack/nova-specs master: WIP libvirt: Allow the default machine type to be changed  https://review.opendev.org/76219914:58
*** ociuhandu has joined #openstack-nova15:01
*** artom has joined #openstack-nova15:04
sean-k-mooneymelwitt:i saw your questions/comments on https://review.opendev.org/#/c/602432/ by the way ill adress them shortly and respin it.15:09
sean-k-mooneymelwitt: tl;dr plugin a port for vif_type=ovs and hybrid-plug=false is currently a noop unless you are on windows where they pass crete-port to have os-vif create teh ovs port15:10
*** ociuhandu has quit IRC15:10
*** ociuhandu has joined #openstack-nova15:10
sean-k-mooneymelwitt: it was a noop because libvirt was previously creating the ovs port which is racy and what im changing in my patch15:11
sean-k-mooneyso we now need to pass create-port=true but ill likely remove that in a futre release of os-vif15:11
sean-k-mooneyi orginally made it unconditional but it triggered a race in the ovs agent which has sicne been fixed15:12
sean-k-mooneyin hignsight i should not have reverted the cahnge and added create-port but it was too late in the cyle to make it unconditonal again15:13
sean-k-mooneyill deprecate it this cycle and likely remove create-port in x15:13
sean-k-mooneysince all knwon callers will now be passing true15:14
*** lpetrut has quit IRC15:16
*** ociuhandu has quit IRC15:21
openstackgerritLee Yarwood proposed openstack/nova-specs master: WIP - Image and flavor defined ephemeral storage encryption  https://review.opendev.org/75228415:21
*** ociuhandu has joined #openstack-nova15:27
taccowhats the best way to figure out which hosts are given back by the API if i request a certain flavor? because try+error could be verry boring over the time :D15:32
tacco--debug to the cli command and see the api result? :)15:33
*** dklyle has joined #openstack-nova15:34
sean-k-mooneytacco: you cant really15:37
sean-k-mooneytacco: even if you were to ask placment that woudl not take into account features15:37
sean-k-mooneythere is no dry run option15:37
taccoi see.15:37
sean-k-mooneyor similar to get the list15:37
*** ociuhandu has quit IRC15:38
hemanth_nHi appreciate any reviews on the following backports https://review.opendev.org/#/q/status:open+project:openstack/nova+topic:bug/1892361 thank you15:39
stephenfinsean-k-mooney: when did we talk about the project_admin behaviour /o\ https://review.opendev.org/#/c/755109/15:49
stephenfinI agree it makes sense but I can't remember discussing it, heh15:49
*** mkrai has joined #openstack-nova15:50
sean-k-mooneyin the post ptg review15:52
sean-k-mooneylast week15:52
sean-k-mooneyas an alternivie to gmann use system_reader+proejct_admin proposal15:53
sean-k-mooneyalso in the call with lance15:53
sean-k-mooneyon the nova rbac stuff15:53
sean-k-mooneyso not upstream unfortunetly but i suggested puting it into your spec on the call15:53
sean-k-mooneywe did talk about alot of stuff on that call to be fair15:54
*** artom has quit IRC15:54
stephenfinsean-k-mooney: fair fair. Let me amend it real quick16:00
sean-k-mooneyi was happy with the rest of the spec for what its worth16:01
sean-k-mooneyi said it was fine but i ment it looked good16:01
*** ociuhandu has joined #openstack-nova16:02
*** k_mouza has quit IRC16:07
taccosean-k-mooney: thanks that helped, now i have to see how to get the config part into os_nova for openstack-ansible to persist my change somewhere upstream. :D16:12
sean-k-mooney:)16:12
*** bnemec has quit IRC16:16
*** hemna has quit IRC16:18
*** mkrai has quit IRC16:21
*** hemna has joined #openstack-nova16:23
*** k_mouza has joined #openstack-nova16:23
*** artom has joined #openstack-nova16:28
*** iurygregory has quit IRC16:29
*** xek has quit IRC16:32
gmannstephenfin: sean-k-mooney replied on review. are we allowing hypervisor info for projects for use case of boot server on host?16:35
openstackgerritStephen Finucane proposed openstack/nova-specs master: Update modernize-os-hypervisors-api spec  https://review.opendev.org/76304316:35
stephenfingmann, sean-k-mooney, bauzas, gibi: I've split that potential policy change out so we can debate the merits of it separately ^16:36
gibistephenfin: ack, thanks16:37
gmannstephenfin: +1,16:37
sean-k-mooneygmann: yes16:38
sean-k-mooneygmann: i dont think we shoudl be using system_read+proejct_admin16:38
gmann what we can do is keep GET  /os-hypervisors for ['system' and 'projects'] and only return list of hypervisors for project what all they are access to via aggregate metadata otherwise emtpy16:38
stephenfinsean-k-mooney: You okay to discuss separately, yeah? Can you toggle your -1 if so?16:39
gmannsean-k-mooney: well if we make only project then system would not be able to list hypervisor at all16:39
sean-k-mooneystephenfin: yep if you file a second spec im happy to defer to there16:39
gmannand adding/removing the host are system level operation right16:39
stephenfinSee https://review.opendev.org/76304316:39
sean-k-mooneygmann: system would16:39
sean-k-mooneygmann: its an additive change16:39
sean-k-mooneyfithe now yould admins can call /os-hyperviors16:40
sean-k-mooney....16:40
sean-k-mooneyright now only admins can call /os-hypervisors16:40
gmannbut they need scope as project. so any system scope token would not be able to do 'project'-only scope16:40
sean-k-mooneyright16:41
sean-k-mooneysystem scoped tokens should not be project only16:41
gmannyeah so how they get hyperviors ino16:41
gmanninfo16:41
sean-k-mooneyim suggesting project_admin shoudl be able to list the host summaries for all hosts tehy can boot too16:41
gmannyeah that i agree. but keep scope_type as ['system', 'project'] sop that system can list all hyperviros and project can only list what they have access to16:42
sean-k-mooneywith role admin16:42
gmannand default to SYSTEM_READER_OR_PROJECT_ADMIN16:43
sean-k-mooneywe dont want project member or reader to have acess16:43
gmannyes16:43
sean-k-mooneyya16:43
sean-k-mooneySYSTEM_READER_OR_PROJECT_ADMIN woudl be correct16:43
sean-k-mooneyif we do that we dont need to change the api to pass a proejct id16:43
gmanncool and in code we can check and return only the project accessible hypervisors  but in that case we need to check the token's scope in nova code16:44
gmannsean-k-mooney: +1 on that.16:44
sean-k-mooneyyes on just that one api where we are adding thei fucntionality16:44
sean-k-mooneyif we need to expose something lese to project admins we can do that in a similar way16:44
gmannchecking token's cope part is little different and new which we need to check how to do16:44
sean-k-mooneybut we dont need full system_reader for project admin usecases16:45
gmannagree.16:45
sean-k-mooneyi belive it should be in the context object16:45
gmannyes we can do from context only difference is, it is 'system_scope' in context and 'system' in oslo policy side. but not big deal16:46
sean-k-mooneystephenfin: updated https://review.opendev.org/#/c/755109/4 to a +1 if someone wants to +w16:47
*** bnemec has joined #openstack-nova16:50
gmannstephenfin: sean-k-mooney replied on this what we disucssed - https://review.opendev.org/#/c/763043/116:53
stephenfinAck, will respin shortly. Thanks :)16:53
*** ociuhandu has quit IRC16:55
*** ociuhandu has joined #openstack-nova16:55
*** hamalq has quit IRC16:56
*** hamalq has joined #openstack-nova16:57
*** ociuhandu has quit IRC17:00
lyarwoodkashyap / stephenfin ; thinking about the machine type enumeration problem a little more, do we care if an instance is using a versioned machine type?17:02
sean-k-mooneylyarwood: kashyap and i disagree17:03
kashyaplyarwood: Do you mean, should we record that or not?17:03
sean-k-mooneylyarwood: i prefer to use the unversioned ones17:03
lyarwoodkashyap / stephenfin ; if we record the verioned machine type and always use it wouldn't that stop users getting security updates for free?17:03
sean-k-mooneythere are reason to use the versioned ones17:03
sean-k-mooneylivemigration/upgrades mainly17:03
lyarwoodsean-k-mooney: right I think I'm with you if the ABI remains the same17:03
sean-k-mooneythe machine type partly defines the abi17:04
sean-k-mooneyso it wont nessisarly remian the same but its largly the same17:04
sean-k-mooneything that can change are the max number of cores supported for example17:04
sean-k-mooneybut it should be backward compatible17:04
lyarwoodyeah17:04
kashyaplyarwood: Just so I get you: we're referring to CentOS/RHEL-based versioned machine types, yeah?  (And not the upstream QEMU's per-release machine types)17:04
sean-k-mooneye.g. a q35-1 guest shoudl be upgradable to a q35-2 machine type17:05
kashyapsean-k-mooney: Well, wait.  That sentence doesn't make sense17:05
kashyapTo remind: when you migrate an instance with machine type q35-1, it *remains* q35-1 on the destination17:05
sean-k-mooneythe abi between verions should change additivly17:06
kashyap... until one explicitly changes it via nova.conf.17:06
sean-k-mooneykashyap: yes i never said migrate17:06
*** sapd1 has quit IRC17:06
sean-k-mooneyi said upgrade i actully ment via a hard reboot17:06
kashyapsean-k-mooney: Err, my mind read the "upgradable" as "migratable".  Silly me!17:06
kashyapsean-k-mooney: Even in the case of "upgrade", libvirt won't gratuitously update it, you have to explicitly request it.17:07
sean-k-mooneyyou can expect that a guest booed on an older vers should be able to boot in the newer verion of the same overall type17:07
kashyaplyarwood: Not sure if we're on track with your original question17:07
lyarwoodkashyap: yeah this is still on the original question17:07
sean-k-mooneykashyap: well if you just use q35 in your nova.conf17:07
openstackgerritMerged openstack/nova-specs master: Add modernize-os-hypervisors-api spec  https://review.opendev.org/75510917:07
lyarwoodkashyap: so my spec as written will track the versioned machine type17:07
sean-k-mooneythen since we destroy and recreate teh domain it will use the new one that the q35 alis now points too17:07
kashyaplyarwood: So ... on the security updates: they're not tied to machine types often; most fixes in machine types are typically garden-variety "bug fixes"17:08
lyarwoodkashyap: but the issue with that is that you can never move from that specific version without rebuilding the ntire instance17:08
lyarwoodentire*17:08
sean-k-mooneylyarwood: well really it should track the unversioned one17:08
sean-k-mooneyunless its set in the nova.conf17:08
lyarwoodsean-k-mooney: that's what I'm getting at17:08
sean-k-mooneyin which case it should use that17:08
lyarwoodsean-k-mooney: even then I think we should track the unversioned machine type17:08
kashyapsean-k-mooney: Lee's question is, what *if* the guest is explicitly using the versioned machine type17:08
sean-k-mooneyif you had not set it the expecation is that it will change when you hard-reboot if you did a yum update and updated libvirt/qemu17:09
sean-k-mooneykashyap: if the guest is useing a verioned machinve type in the image you dont need to do anything17:09
sean-k-mooneyif the guest is useing a version machine tyep from the config then use the config value17:09
sean-k-mooneythe upgrade to a newer one was ment to be handeled by the new recreate api17:10
sean-k-mooneywhich we rejected so there is no upgrade path now17:10
lyarwoodwell just using the alias17:10
lyarwoodthat's the only way to provide an upgrade path to users17:10
sean-k-mooneyright but on waht operation would it upgrade17:10
kashyaplyarwood: I think the clearest way to get understanding here is to lay out some example scenarios17:11
lyarwoodmove or hard reboot17:11
lyarwoodassuming the underlying QEMU had been updated17:11
kashyaplyarwood: I've writte up some for RHOS QE for cold migration, live and evacuate scenarios17:11
sean-k-mooneyi guess i could by that17:11
*** ociuhandu has joined #openstack-nova17:11
kashyapI can clean that up and provide a link, as lay out the impliciations.  And that allows one to better reason about it, IMHO17:12
sean-k-mooneyresize,hardreboot, unshelve, rebuild and cold migrate17:12
lyarwooddoes anyone actually care about running on a specific version of a machine type?17:12
sean-k-mooneylyarwood: do you have my hw:stable_abi extra spec in your spec17:12
lyarwoodsean-k-mooney: no17:12
sean-k-mooneylyarwood: we do for ffu17:12
sean-k-mooneybut beyond that im not sure17:13
lyarwoodsean-k-mooney: why? Wouldn't that be covered by LM17:13
sean-k-mooneyyes kind of17:13
sean-k-mooneyit more we want to pin the new host to the old verion untill the rolling upgrade is complete17:13
sean-k-mooneyi think17:14
sean-k-mooneyso that we can technically migrate form new to old if we need too eventhough we say dont do that17:14
sean-k-mooneychanging the machine_type is ment to be a post upgrade step17:14
lyarwoodwell you could still technically do that with FFU17:14
lyarwoodactually no you couldn't ignore me17:15
lyarwoodand agreed with post upgrade for changing default machine types17:15
kashyapA factor to consider: if you track the unversioned 'pc' or 'q35', it always aliases to the latest available versioned machine type on the box.17:16
sean-k-mooneyfrom a donwstream point of view what we said was new installs should use the new default, existing installs should be upgrade with the old default then the default shoudl be changed as a post upgrade action17:16
sean-k-mooneykashyap: yes but if you have not express an opipion in the image and the config is either unset or set to the aliase i think that is the right thing to track anyway17:17
kashyapsean-k-mooney: Yep, that I agree with.17:17
sean-k-mooneythe edge case is if you have a versioned machine type in the config17:17
sean-k-mooneyi was suggesting recodign that17:18
sean-k-mooneybut then we cant really allow it to change on hard reboot17:18
lyarwoodit wouldn't17:18
sean-k-mooneyit would have to be resize or rebuld only17:18
sean-k-mooneymaintain the machine type on cold or live migate17:18
lyarwoodif a versioned machine type is recorded the only way to change it would be with a rebuild17:18
*** mlavalle has quit IRC17:18
sean-k-mooneylyarwood: only rebuild or rebuild/resize17:19
lyarwoodsean-k-mooney: rebuild/resize sorry17:19
*** iurygregory has joined #openstack-nova17:19
sean-k-mooneyso the elephant in the room17:19
sean-k-mooneywhat about evacuate17:19
lyarwoodsean-k-mooney: actually can you proivide the machine type via a flavor17:19
sean-k-mooneylyarwood: no17:19
sean-k-mooneybut we could17:19
lyarwoodsean-k-mooney: then just rebuild17:19
sean-k-mooneythat does not work then17:20
sean-k-mooneyrebuild is distructive17:20
lyarwoodsean-k-mooney: and evacuate uses rebuild17:20
sean-k-mooneyand for secuity reason we do need a way to change this17:20
sean-k-mooneywhich is one of the reason i had recreate17:20
lyarwoodwell that's no different to the situation we are in today17:20
sean-k-mooneyno today you jsut change the config option and hardreboot17:21
lyarwoodthat a user has no access to17:21
sean-k-mooneyor if its using the aliase just package update and reboot17:21
lyarwoodthat's basically the same as a db update17:21
sean-k-mooneyit is yes17:21
sean-k-mooneywe need a way for an admin to update it17:21
sean-k-mooneythat is not a db update17:21
lyarwoodif you're using an alias it's just a hard reboot with the new approach still17:21
sean-k-mooneywe dont need a way for a user17:21
sean-k-mooneylyarwood: yep which is why i like aliasis amoung other things17:22
sean-k-mooneylyarwood: what about a nova manage command17:22
lyarwoodsean-k-mooney: yup that could work17:22
sean-k-mooneyto update the machine type in a suppotable way17:22
sean-k-mooneyideally support either the instance uuid/uuids or a host17:23
lyarwoodbut it still looks like we can't enumerate all of the possible machine types so I'm back to using a StringField17:23
lyarwoodgah17:23
sean-k-mooneywell ya its a sting filed17:23
sean-k-mooneythey are different per disto17:23
lyarwoodyup17:24
sean-k-mooneyand you can add your own by just adding a file17:24
lyarwoodI indeeed, right need to help with childcare and then I'll update the spec17:24
sean-k-mooneyi know some people have done that to work around some default in the past realted to >2TB vms17:24
lyarwoodthanks stephenfin, kashyap, sean-k-mooney!17:24
*** tesseract has quit IRC17:25
*** rpittau is now known as rpittau|afk17:27
*** k_mouza has quit IRC17:30
*** mlavalle has joined #openstack-nova17:31
ralonsohsean-k-mooney, just a heads-up: https://review.opendev.org/74006717:36
sean-k-mooneyoh nice17:36
sean-k-mooneyso now just the nova bit17:37
sean-k-mooneythanks17:37
sean-k-mooneyralonsoh: ill respin https://review.opendev.org/#/c/760047/ this week hopefully too17:37
ralonsohsean-k-mooney, perfect!17:37
* gibi leaves the keyboard17:42
gibio/17:42
*** ociuhandu_ has joined #openstack-nova17:43
*** ociuhandu has quit IRC17:46
*** ociuhandu_ has quit IRC17:47
*** k_mouza has joined #openstack-nova17:50
*** martinkennelly has quit IRC17:57
*** jangutter_ has quit IRC17:58
stephenfindansmith: Is there any way to identify what images have been cached for an aggregate?17:58
stephenfinAssuming such a request even makes sense?17:58
*** jangutter has joined #openstack-nova17:58
dansmithstephenfin: no, the caching thing is all point-in-time requests, akin to asking for boots that don't end up spawning actual instances17:59
dansmithso no accounting or recording17:59
stephenfinokay, so not displaying anything in response to an 'openstack aggregate cache image' call makes sense?18:00
stephenfine.g. there's no point showing the aggregate since nothings has "changed"18:00
dansmithlike a table of data or something? no, nothing to display. obviously if it returns 404 because there's no aggregate or something an error makes sense18:01
dansmithright, no point in showing the image or aggregate, IMHO18:01
stephenfindansmith++ great, thanks :)18:01
* stephenfin copy-pastes to https://review.opendev.org/#/c/762134/18:01
dansmithfrom context it sounds like you're ...cool18:02
*** mgoddard has quit IRC18:02
dansmithstephenfin: in case you hadn't seen it, I had good intentions: https://review.opendev.org/#/c/688960/18:02
dansmithbut never followed through because I suck18:02
*** andrewbonney has quit IRC18:05
stephenfinthankfully gtema is a machine 🤗18:07
*** k_mouza has quit IRC18:10
*** xek has joined #openstack-nova18:15
*** k_mouza has joined #openstack-nova18:19
*** xinranwang has quit IRC18:49
*** k_mouza has quit IRC18:56
*** gyee has joined #openstack-nova19:18
*** mgoddard has joined #openstack-nova19:21
*** jawad_axd has joined #openstack-nova19:36
*** xek has quit IRC19:44
*** xek has joined #openstack-nova19:44
*** k_mouza has joined #openstack-nova19:57
*** k_mouza has quit IRC20:01
*** gryf has quit IRC20:09
*** gryf has joined #openstack-nova20:09
*** k_mouza has joined #openstack-nova20:16
*** dtantsur is now known as dtantsur|afk20:17
*** k_mouza has quit IRC20:21
*** smcginnis has quit IRC20:24
*** mgoddard has quit IRC20:31
*** nweinber has quit IRC20:58
*** nweinber has joined #openstack-nova20:58
*** rcernin has joined #openstack-nova21:02
*** nweinber has quit IRC21:30
*** smcginnis has joined #openstack-nova21:59
*** ralonsoh has quit IRC22:03
*** xek has quit IRC22:22
*** songwenping__ has joined #openstack-nova22:50
*** songwenping_ has quit IRC22:53
rm_workhey, what config does nova-manage pull the db connection stuff from by default? is it /etc/nova.conf?23:18
rm_workerr, /etc/nova/nova.conf23:19
rm_workit seems to not be respecting all of the sql connection config23:19
rm_workwe use x509 auth for our sql connection, which is in the connection string in nova.conf as &ssl_cert=...&ssl_key=...23:19
rm_workwhich works for the actual service, but the nova-manage db stuff doesn't seem to use it correctly23:20
*** slaweq has quit IRC23:21
rm_workyeah, verified even with the config explicitly set, it's not respecting the x509 args, so the db migration scripts fail on the sql connection23:23
rm_workhmmm, well that was my assumption, but maybe it's actually that it is missing some other sql permission? trying to dig in deeper, I just know I get this error when it tries to run `_get_marker_for_migrate_instances`23:28
rm_work`(pymysql.err.OperationalError) (1045, "Access denied for user 'nova'@'sqlhost' (using password: YES)")`23:28
rm_workbut it does seem to be able to get SOME data, as it printed some numbers about the migration counts correctly23:29
*** mloza has quit IRC23:46
*** tosky has quit IRC23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!