Thursday, 2022-02-17

opendevreviewmelanie witt proposed openstack/nova master: libvirt: Register defaults for undefined hw image properties  https://review.opendev.org/c/openstack/nova/+/80070803:22
opendevreviewGhanshyam proposed openstack/nova master: Separate flavor extra specs policy for server APIs  https://review.opendev.org/c/openstack/nova/+/82962603:38
opendevreviewGhanshyam proposed openstack/nova master: Separate flavor extra specs policy for server APIs  https://review.opendev.org/c/openstack/nova/+/82962604:16
opendevreviewGhanshyam proposed openstack/nova master: Test PROJECT_ADMIN APIs with no legacy rule case  https://review.opendev.org/c/openstack/nova/+/82484505:19
opendevreviewGhanshyam proposed openstack/nova master: Move rule_if_system() method to base test class  https://review.opendev.org/c/openstack/nova/+/82447505:20
opendevreviewGhanshyam proposed openstack/nova master: Convert SYSTEM_ADMIN|READER to Admin and system scope  https://review.opendev.org/c/openstack/nova/+/81939005:20
opendevreviewGhanshyam proposed openstack/nova master: Make more project level APIs scoped to project only  https://review.opendev.org/c/openstack/nova/+/82867005:20
opendevreviewGhanshyam proposed openstack/nova master: Server actions APIs scoped to project scope  https://review.opendev.org/c/openstack/nova/+/82435805:22
opendevreviewGhanshyam proposed openstack/nova master: Modify remaining APIs as per RBAC new guidelines  https://review.opendev.org/c/openstack/nova/+/82899405:22
opendevreviewGhanshyam proposed openstack/nova master: Modify remaining APIs as per RBAC new guidelines  https://review.opendev.org/c/openstack/nova/+/82899405:39
opendevreviewGhanshyam proposed openstack/nova master: Separate flavor extra specs policy for server APIs  https://review.opendev.org/c/openstack/nova/+/82962605:39
opendevreviewGhanshyam proposed openstack/nova master: Modify remaining APIs as per RBAC new guidelines  https://review.opendev.org/c/openstack/nova/+/82899406:01
*** akekane_ is now known as abhishekk06:02
opendevreviewGhanshyam proposed openstack/nova master: Separate flavor extra specs policy for server APIs  https://review.opendev.org/c/openstack/nova/+/82962606:03
opendevreviewGhanshyam proposed openstack/nova master: Separate flavor extra specs policy for server APIs  https://review.opendev.org/c/openstack/nova/+/82962606:03
*** amoralej|off is now known as amoralej08:16
gibisean-k-mooney: re nvme multipath:  I'm not against an exception if it is low risk (and it seems so). But I defer to bauzas to make the decision with the PTL hat.08:23
sean-k-mooney[m]ack08:24
bauzasgibi: sean-k-mooney: I agree with sean's proposal for a specless bp approval during Feb 22th meeting08:34
bauzasthis way, we will see whether we have a consensus08:34
bauzassean-k-mooney: gibi: procedural -2 for https://review.opendev.org/c/openstack/nova/+/82394108:38
gibibauzas: ack08:38
bauzassean-k-mooney: if you want, you can review this change and +2 it 08:39
bauzasso, in case we agree on accepting it for Yoga, I'd just remove the -2 and do +W08:39
sean-k-mooney[m]i have actully reviewed it and the lack of bluepinrt/bug was the main thing that stopped me form +2ing before08:40
sean-k-mooney[m]but sure ill take a look again08:40
sean-k-mooney[m]and ya we can decide on tuesday08:41
bauzas++08:42
bauzassean-k-mooney: thanks btw. for having reviewed it08:42
yuvalhello09:18
yuvalI am just making sure the feature freeze is the week of the 21 meaning -> 25.2 right?09:18
gibiyuval: I think the cut date for us will be 2.24 EOB but things that are approved can be rebased and merged after it09:26
yuvalwhats EOB stand for?09:35
yuvalI have this: https://review.opendev.org/c/openstack/nova/+/82160609:36
yuvalits not complicate - its just waiting for os-brick release to pass zuul09:36
gibiend of business 09:40
gibiso basically end of the day of cores on the west coast09:41
yuvalgot it09:42
gibiyuval: as far as I see your os-brick patch is not part of that release https://review.opendev.org/c/openstack/os-brick/+/82365409:42
gibias this is not landed yetr09:42
yuvalgibi: this is not the important patch for me, although I do think brian want it in the release09:43
yuvalos-brick 5.2 dont have the lightos connector in it09:44
yuvalso zuul is failing in nova for it09:44
gibiahh so you only need https://review.opendev.org/c/openstack/os-brick/+/82160309:44
yuvalyes09:44
gibiand that is already landed and will be in os-brick 5.209:45
gibicool09:45
gibithen I think you are on the good track09:45
yuvalok, thanks 09:45
gibiI will check you nova patch soon 09:45
yuvalthat would be great09:45
sean-k-mooneygibi: regarding the placment changes i think your current implemenation is likely ok to proceed with. we could add followup patches to try the alternitive approch if we do think the perfload resulst are refective fo a real regression10:15
gibisean-k-mooney: I have to figure out how to make a less noisy perfload measurement because currenty the noise is comparabel with the amount of degradation I can measure 10:16
gibibut at least I did not make the performance halved :D10:17
gibisean-k-mooney: I'm happy to make followup patches trying to improve especially as you gave directions with the SQL I can try to apply and measure10:18
sean-k-mooneyya i dont really want to hold up the review on the subquery vs join approch escpially since i dont have performacne data one way or another10:19
gibisean-k-mooney: do you have any other comment that I should fix before FF?10:19
sean-k-mooneyso i woudl prefer to take the "get it working; then make it fast" approch provided there is no large regression 10:19
gibiI'm happy to if there is anythign10:19
sean-k-mooneyam not so far i have not got to the end of the series yet10:20
sean-k-mooneybut so far things look pretty good10:20
gibisean-k-mooney: thanks. let me know if I need to fix up thing quickly :) 10:21
gibiand thanks for the review10:21
sean-k-mooneyhuh i didnt know that tuple unpacking work with dictionaries10:22
sean-k-mooneyhttps://review.opendev.org/c/openstack/placement/+/826491/8/placement/handlers/resource_provider.py10:22
sean-k-mooneyi understand how it could i guess i just tought that would be a syntax error10:22
gibiyeah that was a nice touch from the python language that you can pack out into a dict10:24
gibiinto a dict value10:24
sean-k-mooneygibi: its out of scope of the current spec but do you want to eventually support any_traits for root_required by the way10:24
gibiI have no need for it right now, but if there is some use case from others then I can do it, most of the infra is in place for it10:24
gibiprobably the DB query needs some touching and the API needs to call the new trait parsing code10:25
sean-k-mooneyok do we use root_required in nova today. i dont think so but perhaps we do10:26
sean-k-mooneyim not sure i have a usecase for any traits with it currently either but your serise will serve as a template for adding it if we need it in the future anyway10:27
gibisean-k-mooney: require_image_type_support, accelerators_filter and remote_managed_ports_filter prefilters are using it in nova today10:29
gibiit is mosly to filter for compute capabilities10:29
gibiand for that the simple AND trait list is enough10:29
sean-k-mooneyreally i tought those were just in the unnamed group10:29
gibithey could be there too10:29
gibiboth works today as we always allocate from the root10:29
gibithe code is a bit simpler with root_required as you don't need to find the unnamed group10:30
sean-k-mooneyya i just was not aware we converted to useing root_requried for that10:30
sean-k-mooneyya10:30
sean-k-mooneygibi: which patch are you adding the 1.39 micorversion is it the last patch10:35
gibiyes10:35
sean-k-mooneyah yes it is10:35
sean-k-mooneyok i was just looking at the func test changes and wondering why you only had the negitive tests for 1.3810:36
sean-k-mooneyok that makes sense10:36
gibiyuval: left comments in https://review.opendev.org/c/openstack/nova/+/82160611:16
yuvalThanks11:19
sean-k-mooneyyuval: how much work woudl it be to entirely factor out the monitor thread and "message queue" out of the nova code11:32
sean-k-mooneyi really dont like the idea of os-brick or the volume driver spawnign a background thread11:33
sean-k-mooneyim pretty sure i have objected to adding that to os-brick on the mailing list in the past11:33
gibichateaulav: I have an idea what is wrong with the object compatibility code still in https://review.opendev.org/c/openstack/nova/+/82836911:34
gibichateaulav: left a comment11:34
sean-k-mooneyhum we likely can move the call to super to the end11:35
sean-k-mooneygibi: actully looking at the version im a littel confused why was a version bump done for 1.18 and 1.1911:36
gibisean-k-mooney: I think remotable methods need a bump11:36
sean-k-mooneyi have not looked at the patches that added those but adding a new method does not require an object version bump11:36
sean-k-mooneyah11:37
sean-k-mooneythey are not jsut local11:37
sean-k-mooneyperhaps11:37
gibiabout the super call, I think yes we can move even way down 11:38
sean-k-mooneyi would prefer not to put it in the midel of the compat ifs11:38
sean-k-mooneyso either have it first or after all of them ideally11:38
yuvalsean-k-mooney: can you add your comment to the patchset I discuss it with my colleagues11:40
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/821606/8/nova/virt/libvirt/volume/lightos.py#5911:42
sean-k-mooneyyuval: this is the feedback i left on the cinder spec over a year ago https://review.opendev.org/c/openstack/cinder-specs/+/766732/12/specs/wallaby/nvme-agent.rst#7311:46
sean-k-mooneyyuval: os-bricks  executor.Thread is just a monkey patched green thread so if you are doing long running work that never yeild it will block nova11:53
chateaulavgibi: sounds good, I'll take a look as soon as get in in about an hour11:56
gibichateaulav: ack11:56
sean-k-mooneyyuval: this for exampel will block nova until its finished processing it https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/lightos.py#L145-L16211:58
sean-k-mooneyso these sleeps https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/lightos.py#L174-L18011:58
sean-k-mooneyare the only thing that cause that to yeild back to nova11:59
sean-k-mooneysome of the operation in https://github.com/openstack/os-brick/blob/5902166149ba46da5f36eb352e62f3361269fb2c/os_brick/initiator/connectors/lightos.py#L107 also do io which will cause eventlet to yeild but in genrealy i dont think we want to allow that to run in a thread in the compute agent12:02
sean-k-mooneylightos is not the only problematic driver that is now tryign to do this12:04
sean-k-mooneybut i really dong like this design parrent in os-brick and i think all montors/deamon threads shoudl be seperate processes or not part of os-brick at all12:05
sean-k-mooney... os-brick is doing this in several places.... that is really not good we should probaly make this a ptg topic12:07
yuvalok, thanks for the feedback, we will think what our options12:19
sean-k-mooneygibi: by the way nova still has root wrap solely for os-brick right. is there a reason we cant move this to os-brick?12:23
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/etc/nova/rootwrap.d/compute.filters12:23
sean-k-mooneygibi: looking at what os-brick is doing really make me think we shoudl not be loading it in the compute agent directly since its problematic in several ways12:25
gibiI think only os-brick using rootwrap we converted the rest of privsep12:26
gibiso I think we can move it12:26
gibibut probably we need a volunteer12:26
sean-k-mooneyyep its just techdebth12:26
gibias of what os-brick does under the hood, I'm not that familiar with. I saw that iscsi connector also spawn threads so I assume that works other12:26
gibiotherwise we would have failures12:27
sean-k-mooneyit works but they are green threads12:27
sean-k-mooneynot native threads12:27
sean-k-mooneybecause we monkey patch and its running in the same process12:27
sean-k-mooneyos-brick does not use eventlets iteself12:27
sean-k-mooneywhich is true for os-vif alos both both inherit it becasue they are imported into nova which does12:28
gibiahh you have a good point12:28
gibithose just look like native threads but we monkey patched them12:28
gibibut then I don't fully get the problem12:29
gibiwhy don't they spawn greenlets freely?12:29
gibiunder the hood12:29
sean-k-mooneygibi: simple since they are monkey patched if they dont yeild nova's event loop is blocked12:29
gibithey have thight loops without io?12:30
sean-k-mooneyyes in some places12:30
gibiahh so they have some thight loops 12:30
gibithen those needs some careful yields12:30
sean-k-mooneyhttps://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/lightos.py#L164-L18212:30
sean-k-mooneyso this yeild because monitor_db does some writes to a file and those sleeps12:31
sean-k-mooneymonitor_message_queue wont yeild unless the log satement casue it to happen12:32
sean-k-mooneybut it will terminate when the queue is empty12:32
gibiI see12:32
sean-k-mooneyisci is potentally worse12:33
sean-k-mooneyhttps://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L723-L78012:33
sean-k-mooneyit spawns a bunch of threds then joins in the multipath code12:34
sean-k-mooneythe join shoudl cause a yeild and those threads will be interleaved with nova12:34
sean-k-mooneybut this type of raw threadign is really not a good thing with eventlets in the mix12:35
gibijeah that seem a lot more complex. but still it works or we would have strange bugs 12:35
sean-k-mooneythey also have sleeps https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L77712:35
sean-k-mooneyya so that will do io an dthe sleep will yeild which the do on each iteration12:36
sean-k-mooneyi just think the use of thread in general in os-brick is problematic12:37
opendevreviewImran Hussain proposed openstack/nova master: [nova/libvirt] Support for checking and enabling SMM when needed  https://review.opendev.org/c/openstack/nova/+/82549612:45
*** amoralej is now known as amoralej|lunch13:21
IPOHello, all ! What are the next steps toward https://review.opendev.org/c/openstack/nova/+/805649 ? Or it is ok,it is needed just wait a bit ?14:15
gibiIPO: I guess we need a second core14:18
gibisean-k-mooney: you were +2 before on that patch ^^14:18
*** amoralej|lunch is now known as amoralej14:19
IPOgibi: I see, thanks for info !14:26
yuval_sean-k-mooney other the monitor thread - anything is a concern?14:42
sean-k-mooneyyuval_: that is my main one.15:05
sean-k-mooneyyuval_: gibi listed other issue liek py2 support ectra inline15:06
sean-k-mooneyyuval_: so other then the monitor thread and the comments that gibi has already reaied i dont really have other issues with it15:07
bauzasmelwitt: gibi: question about InstanceMapping records vs. shelved-offloaded instances and how we quota them15:11
bauzasmelwitt: gibi: https://review.opendev.org/c/openstack/nova/+/713301/34/nova/limit/placement.py#8015:12
bauzasiirc, we don't have quota usage for shelved-offloaded instances, right?15:12
sean-k-mooneyshelved instance shoudl consume the same quota for disk as running but they should not consume quota for cpu or ram15:12
sean-k-mooneybauzas: it depend on if it BFV really i think15:13
bauzassean-k-mooney: my question is about the instances/servers quota usage15:13
sean-k-mooneywell i t should consume isntance quota15:13
sean-k-mooneythat is a limit on the number of non deleted instance you can have in the db15:14
bauzassurely, but I don't remember about the current behaviour15:14
sean-k-mooneyits independt fo the state15:14
sean-k-mooneyit wont be tracked in placment currently15:14
sean-k-mooneysince we dont consume allocations of instance15:14
sean-k-mooneybut i guess15:14
sean-k-mooneyyou coudl count allocation by consumer type maybe15:14
sean-k-mooneywhen its in shelve offloaded i dont know if we still have any allcoation in placment at all15:15
sean-k-mooneyif we had local disk i think we should for storage15:15
sean-k-mooneye.g. if you were usign the rbd image backend15:16
sean-k-mooneybut sicne we dont have a host it might not be there15:16
sean-k-mooneymelwitt: how does that work with unified limits?15:16
bauzassean-k-mooney: look at the change15:17
bauzassean-k-mooney: there is a TODO for consumer types15:17
bauzassean-k-mooney: for the moment, we look at instance mappings15:17
bauzashence my question15:17
sean-k-mooneyright but consuemr types is not enough15:17
sean-k-mooneya shelve offloaded instance does not have a host15:18
sean-k-mooneyso it cant have allcoation against an rp today15:18
sean-k-mooneysince the only resouce usage it has are floating ips, and disk really15:18
sean-k-mooneyso without creating a RP per compute with inventories of instance,floating ip ectra we cant tack the useage in palcment15:19
sean-k-mooneyso we woudl have to tack this in keystone?15:19
bauzasI don't know, we haven't discussed it in https://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/unified-limits-nova.html15:20
bauzastbc, I don't want to trample this series because of this question15:20
sean-k-mooneywell for now we can just count it in nova15:20
bauzasmy only concern is about making sure we continue to have the same behaviour from the before15:21
sean-k-mooneyright so we shoudl continue to count instnace using the instance cell mapings in the nova api db proably for yoga15:21
sean-k-mooneyin Zed we could eitehr create an RP per porject with invetoires of diffent quata type like instance or tack it somewhere else15:22
sean-k-mooneythe question really is should you be able to get all usage just by looking at placment or not15:23
sean-k-mooneybauzas: ""All the resource class usage will be counted using placement, but server count will make use of instance mappings.""15:25
sean-k-mooneyhttps://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/unified-limits-nova.html#enforcing-unified-limits15:25
bauzassean-k-mooney: nah I was saying about shelved instances15:26
sean-k-mooneyyes15:26
sean-k-mooneythat fact its shelved does nto change that15:26
bauzasok, then it was a nit15:26
sean-k-mooneyi honestly dont see an impedment to counting server ectra via sharing rps at some point in the futre if we need too or other things like floating ip that are out of scope of the current spec15:28
* gibi is a bit late to the party15:29
gibiI agree that offloaded servers if holding floating ips then it should consume quota15:30
gibiand yes we cannot count that with consumer types as no instance allocation exists for offloaded servers in placement15:30
melwittbauzas, sean-k-mooney: the tl;dr is that unified limits is not meant to have the same behavior as before. shelved offloaded will not consume quota16:01
bauzasmelwitt: cool then, I'll switch to +216:02
melwittbauzas: I dunno if you've seen this but this is one place where we document the expected differences when counting from placement https://docs.openstack.org/nova/latest/admin/quotas.html#quota-usage-from-placement16:09
melwittand unified limits builds on top of that16:09
bauzasmelwitt: oh thanks, I think I saw it already but forgot 16:10
bauzasmy fucking brain is so bad...16:10
melwittand changes a few more things like no more user_id scoped quotas16:10
bauzasand you documented the shelved thing, good16:10
bauzas"Behavior will be different for servers in SHELVED_OFFLOADED state. A server in SHELVED_OFFLOADED state will not have placement allocations, so it will not consume quota usage for cores and ram. Note that because of this, it will be possible for a request to unshelve a server to be rejected if the user does not have enough quota available to support the cores and ram needed by the server to be unshelved."16:10
bauzasperfect, definitely +2 once we're done with the internal meeting16:11
melwittyeah I aimed to document all differences for operators, it's also in the config option help and was in the release note16:11
bauzasmelwitt: my bad, I haven't yet looked at the reno change16:23
opendevreviewFelix Huettner proposed openstack/nova stable/ussuri: Gracefull recovery when attaching volume fails  https://review.opendev.org/c/openstack/nova/+/82950616:23
melwittbauzas: sorry I meant the release note for the counting from placement in the past16:23
bauzasah16:24
bauzasgotchaz16:24
melwittI was saying I tried to put the info everywhere :)16:24
melwittreno for the counting from placement is in this list https://docs.openstack.org/releasenotes/nova/train.html#relnotes-20-0-0-stable-train-upgrade-notes16:25
opendevreviewJulia Kreger proposed openstack/nova master: WIP Ironic - Handle instance/node host on rebalance  https://review.opendev.org/c/openstack/nova/+/81389716:35
opendevreviewJulia Kreger proposed openstack/nova master: Ironic - Don't query the API for instance counts  https://review.opendev.org/c/openstack/nova/+/82961316:37
opendevreviewJonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82836916:38
opendevreviewJonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82205316:38
opendevreviewJonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82837216:38
opendevreviewJonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82836916:51
opendevreviewJonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82205316:51
opendevreviewJonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82837216:51
*** artom__ is now known as artom17:11
*** efried1 is now known as efried17:19
*** eandersson8 is now known as eandersson17:19
opendevreviewBalazs Gibizer proposed openstack/nova master: Record SRIOV PF MAC in the binding profile  https://review.opendev.org/c/openstack/nova/+/82924818:07
opendevreviewMerged openstack/nova master: Document remote-managed port usage considerations  https://review.opendev.org/c/openstack/nova/+/82751318:10
*** amoralej is now known as amoralej|off18:28
sean-k-mooneygibi: im going to call it a day there ill try and review the rest of your placment patches on monday18:43
*** kopecmartin_ is now known as kopecmartin19:11
chateaulavgibi: are there more verbose logs from grenade, or something that i am missing besides `tempest.lib.exceptions.ServerFault: Got server fault`19:14
opendevreviewJonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82836919:41
opendevreviewJonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82205319:41
opendevreviewJonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82837219:41
*** ministry is now known as __ministry20:07
opendevreviewMerged openstack/nova master: Fix to implement 'pack' or 'spread' VM's NUMA cells  https://review.opendev.org/c/openstack/nova/+/80564920:28
opendevreviewIlya Popov proposed openstack/nova stable/xena: Fix to implement 'pack' or 'spread' VM's NUMA cells  https://review.opendev.org/c/openstack/nova/+/82980420:33
opendevreviewMerged openstack/nova master: neutron: Allow to spawn VMs with port without IP address  https://review.opendev.org/c/openstack/nova/+/66941120:35
opendevreviewJonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82836921:18
opendevreviewJonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82205321:18
opendevreviewJonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support  https://review.opendev.org/c/openstack/nova/+/82837221:18
opendevreviewMerged openstack/nova master: [nova/libvirt] Support for checking and enabling SMM when needed  https://review.opendev.org/c/openstack/nova/+/82549621:43
*** haleyb_ is now known as haleyb22:04
*** dasm is now known as dasm|off23:07

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!