| *** mnasiadka_ is now known as mnasiadka | 05:08 | |
| *** tkajinam_ is now known as tkajinam | 05:11 | |
| songwenping_ | sean-k-mooney:can we support multi-create vgpu vms? | 07:56 |
|---|---|---|
| *** songwenping_ is now known as songwenping | 08:25 | |
| gibi | kashyap: I played with a fedora36 container but I cannot reproduce the slow pip resolving issue either there | 09:35 |
| gibi | so I think you might have extra python3- packages installed that creates some version confusion | 09:35 |
| gibi | it should now do it | 09:35 |
| gibi | as venv is separate | 09:36 |
| gibi | but I have no other ideas | 09:36 |
| sean-k-mooney | songwenping: no that is not supported | 09:40 |
| sean-k-mooney | songwenping: we do not suppport multi create with nested resouce providres | 09:41 |
| sean-k-mooney | songwenping: https://bugs.launchpad.net/nova/+bug/1874664 | 09:42 |
| sean-k-mooney | songwenping: actully it looks like the patches are merged | 09:42 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/723884/ and https://review.opendev.org/c/openstack/nova/+/723858/ | 09:43 |
| sean-k-mooney | songwenping: i dont think this fully resolves the issue bauzas ^ | 09:45 |
| sean-k-mooney | songwenping: ya reading them we only added documentation and test to show that it does not work in some cases | 09:46 |
| sean-k-mooney | so form the comit | 09:46 |
| sean-k-mooney | - you can ask for a flavor with 2 VGPU with --max 2 | 09:46 |
| sean-k-mooney | - but you can't ask for a flavor with 4 VGPU and --max 2 | 09:46 |
| kashyap | gibi: Hey, thanks a ton for trying! Also, what do you mean by "it should now do it as venv is separate"? I can't quite parse it :) | 09:47 |
| kashyap | Pls rephrase it | 09:47 |
| gibi | tox should build a new virtual env that is sealed off from python packages on the host | 09:47 |
| kashyap | Yeap; thanks! | 09:49 |
| gibi | interestingly the dockerfile in the bugreport shows the issue to me but my dockerfile doesn't | 09:49 |
| gibi | craxy | 09:49 |
| opendevreview | Rajat Dhasmana proposed openstack/nova-specs master: Repropose volume backed server rebuild spec https://review.opendev.org/c/openstack/nova-specs/+/840155 | 09:53 |
| whoami-rajat | sean-k-mooney, ^^ updated the spec | 09:54 |
| sean-k-mooney | whoami-rajat: thanks ill review it shortly | 09:57 |
| whoami-rajat | thanks! | 09:57 |
| songwenping | sean-k-mooney: no, not this bug. | 10:28 |
| sean-k-mooney | songwenping: are you refering to creating a vm with multipel vgpus? | 10:28 |
| songwenping | i create two vms with flavor that has 1 vgpu metadata | 10:28 |
| sean-k-mooney | ok | 10:29 |
| songwenping | libvirt allocate the same vgpu for these two vms | 10:29 |
| sean-k-mooney | we only support doing that via 2 differnt api request today | 10:29 |
| sean-k-mooney | we do not supprot creating 2 vms that use vgpus in the same api request | 10:30 |
| songwenping | does the api check the request? | 10:30 |
| sean-k-mooney | no | 10:30 |
| sean-k-mooney | we will not block it at the api | 10:30 |
| sean-k-mooney | we docuement it in the api as part of the previous bug | 10:31 |
| *** carloss_ is now known as carloss | 10:31 | |
| songwenping | why donot we block it at api? | 10:31 |
| sean-k-mooney | gibi: speaking of which if you use pci device in placment for now at least multi create will not work for the same reason | 10:31 |
| sean-k-mooney | songwenping: becuase at the api we dont know fi it uses nested resouce provder or not | 10:32 |
| sean-k-mooney | so we documented the limistaion | 10:32 |
| songwenping | could you please find the doc? | 10:32 |
| sean-k-mooney | songwenping: https://docs.openstack.org/nova/yoga/admin/virtual-gpu.html#caveats | 10:35 |
| sean-k-mooney | we added this section https://review.opendev.org/c/openstack/nova/+/723884/6/doc/source/admin/virtual-gpu.rst | 10:36 |
| sean-k-mooney | we coud make that more clear that we do not support multi creat a all but that is our current stnace | 10:37 |
| sean-k-mooney | if a server uses neste reseouce proverder we do not support multi create | 10:37 |
| sean-k-mooney | this applys to vgpu, cyborg, neuton bandwith or packet per second qos or any other usage of resouce form nested resouce providers | 10:38 |
| sean-k-mooney | like pmem | 10:38 |
| *** songwenping__ is now known as songwenping | 10:45 | |
| opendevreview | Rajesh Tailor proposed openstack/nova master: Fix typos https://review.opendev.org/c/openstack/nova/+/843127 | 10:50 |
| songwenping | sean-k-mooney: i see, thanks. | 10:50 |
| gibi | sean-k-mooney: I don't have the context why the multi create fail on nested allocations but I can take a look later | 10:59 |
| sean-k-mooney | gibi: we just dont have any code for it to handel mappign the allcoation to the differnt instance properly | 11:00 |
| sean-k-mooney | we woudl need to do multipel placment queries and orchstrate that in the schduler | 11:00 |
| gibi | hm, I thought that normal multicreate uses the alternatives from the scheduler | 11:00 |
| sean-k-mooney | it does i think but apprently that is not enough | 11:01 |
| sean-k-mooney | this has basicaly never worked if you have nested rps | 11:02 |
| sean-k-mooney | it work fine if you dont | 11:02 |
| kevko | Hi nova team ! | 11:04 |
| kevko | I would like to ask if there is any chance to do something as i will describe below : | 11:05 |
| kevko | I really would like to schedule 2 instances in server group with anti-affinity policy to be scheduled to different host but also in different datacentre (or az) | 11:06 |
| kevko | something similar as it is here | 11:06 |
| kevko | https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/complex-anti-affinity-policies.html | 11:06 |
| kevko | instead of max_server_per_host key should be something as max_server_per_az | 11:07 |
| sean-k-mooney | kevko: that is not currently possibel or trivial to do | 11:07 |
| kevko | yeah, i was checking code :/ | 11:07 |
| sean-k-mooney | the request spec currently modles an az requst as a string | 11:07 |
| kevko | it's mainly because of octavia and two datacentres .. | 11:07 |
| sean-k-mooney | it cant accpet a list of stings so you cant do that in a singel request | 11:08 |
| sean-k-mooney | kevko: in the past we have declared this out of scope of nova | 11:08 |
| sean-k-mooney | kevko: as somethign a higher level orchestrator shoudl mange | 11:08 |
| kevko | hmm | 11:08 |
| kevko | it's mainly because of octavia loadbalancers .. | 11:09 |
| sean-k-mooney | yep so octavia coudl orchestrate this | 11:09 |
| sean-k-mooney | they already mange the vm creation | 11:09 |
| sean-k-mooney | via calling nova | 11:09 |
| sean-k-mooney | so they could make requsts with differnt AZs | 11:10 |
| sean-k-mooney | i woudl have ot refresh my memory of server goups but i cant recall if you can add more server to the group in later request or not | 11:11 |
| sean-k-mooney | if you can you could just group the loadbalnce request per AZ | 11:11 |
| kevko | i am not sure if I understand | 11:12 |
| sean-k-mooney | i was trying to recall if you needed to populate the server group in one go or coudl you make multiple server create request and resuse the same server group | 11:13 |
| sean-k-mooney | i think its the latter | 11:13 |
| sean-k-mooney | we have 2 or 3 ways to do anti afinity in nova | 11:13 |
| kevko | you can set az in octavia ...an if you have az created from two datacentres ... server group and anti affinity will not work ..there is big chance that nova just schedule to two hosts ..but in same datacentre | 11:13 |
| sean-k-mooney | and some requrie it all in one request and other allow it to be exteded. | 11:13 |
| sean-k-mooney | kevko: right so ocativa could have an az anti affintiy policy in its api | 11:14 |
| kevko | currently I think there is only one way to do it ... have 1 az from two hosts from two different datacentres ... but this is no way ..because we have hundreds of amphoraes :( | 11:14 |
| sean-k-mooney | so what i would expect is you would creat at least 1 AZ per datacenter right | 11:15 |
| sean-k-mooney | and then create a singel server group | 11:16 |
| sean-k-mooney | with anti affintiy | 11:16 |
| sean-k-mooney | and then in 2 differnet calls to nova create the vms for datacenter A and B | 11:16 |
| kevko | it's not true i think | 11:17 |
| sean-k-mooney | im describing the change you need to make to ocativa not how it works today | 11:18 |
| kevko | anti-affinity works per host ...so there could be situation that in one datacentre instance will be scheduled on two hosts | 11:18 |
| sean-k-mooney | kevko: correct | 11:18 |
| kevko | yeah, | 11:18 |
| sean-k-mooney | that is why you need ot use differnt AZ in the two diffenrt calls to nova | 11:18 |
| kevko | octavia has availability_zone as one parameter in configuration | 11:18 |
| kevko | if this can be list and setup list of AZs where AZ is 2 hosts from two datacentres ... it will be work i think | 11:19 |
| sean-k-mooney | right if you want to supprot this you need to supprot this at the octavia api | 11:19 |
| sean-k-mooney | not config driven | 11:19 |
| sean-k-mooney | kevko: az in nova are nto the same as AZs in aws | 11:19 |
| sean-k-mooney | the are jsut metadtaa on a a host aggreate. | 11:19 |
| sean-k-mooney | there is not gurantee or expectiaon of a sperate falut domain | 11:20 |
| sean-k-mooney | and conversily there is not guranteeor or expecation fo network connectivy between them | 11:20 |
| sean-k-mooney | that is all determin by how you deploy your cloud | 11:20 |
| kevko | yeah i know that it is only metadata defined for host (or how to say :P ...i really don't know how it looks like in deep) | 11:21 |
| sean-k-mooney | its a strign defiend on a host aggareate | 11:21 |
| sean-k-mooney | and then host are mapped to aggreates | 11:21 |
| kevko | what about regions ? | 11:23 |
| kevko | can be helpfull ? | 11:23 |
| sean-k-mooney | the do not existin in nova | 11:23 |
| sean-k-mooney | they are a keystone only construct | 11:23 |
| kevko | yeah ... correct | 11:24 |
| kevko | cells ? | 11:24 |
| sean-k-mooney | cells are internal to nova | 11:24 |
| sean-k-mooney | and not expsoed at the api to end users | 11:24 |
| sean-k-mooney | they are a way to shared the db and message bus | 11:24 |
| kevko | yeah i know .. | 11:24 |
| sean-k-mooney | and are explictly not for fault tollerance | 11:24 |
| sean-k-mooney | but they can kind fo help in a way | 11:25 |
| kevko | so currently i don't have way how to deal with it | 11:25 |
| sean-k-mooney | we coudl support az anti affintiy in nova | 11:25 |
| sean-k-mooney | but we have rejected it in the past the last 2 times it came up | 11:25 |
| sean-k-mooney | kevko: filters and weigher can be added out fo tree by the way | 11:27 |
| sean-k-mooney | but all instnace will be mapped ot an az even if oenis not requested | 11:28 |
| sean-k-mooney | so really if you want to use AZs to map to datacenters | 11:28 |
| sean-k-mooney | then octavia need to accpet the AZ as an api parameter when creating loadblancers | 11:29 |
| sean-k-mooney | and if you want ti to supprot spreaing loadbalncer between azs then it allso need to supprot doing that iself really | 11:29 |
| sean-k-mooney | since AZ are not really visable to neutron by the way there is no guarentee at teh api level that a newton network can span azs | 11:30 |
| opendevreview | sean mooney proposed openstack/os-vif master: [trivial] update job template to zed https://review.opendev.org/c/openstack/os-vif/+/843432 | 11:31 |
| kevko | why it was rejected ? | 11:33 |
| kevko | this would be super usefull | 11:33 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/doc/source/contributor/project-scope.rst#no-more-orchestration= | 11:42 |
| sean-k-mooney | kevko: it came up on the ptg in the past i can see if there was a draft spec | 11:44 |
| sean-k-mooney | kevko: https://review.opendev.org/c/openstack/nova-specs/+/756380 | 11:44 |
| sean-k-mooney | kevko: as i noted here https://review.opendev.org/c/openstack/nova-specs/+/756380/5#message-ba89445e7b2b033fe39f9bbd32a49271e4780e75= | 11:45 |
| sean-k-mooney | doing this in a filter or weigher is far far too late | 11:45 |
| sean-k-mooney | as the az is alreay asigned at that point | 11:46 |
| sean-k-mooney | kevko: so the only way to do it today woudl be to have an AZ span both datacenters and then use somethign else to model the datacenters and provide anti afinity | 11:46 |
| kevko | hmm | 11:47 |
| kevko | understand | 11:47 |
| sean-k-mooney | you coudl do that with metadata on host aggrats and then a weigher or filter that used that datacenter metadata | 11:47 |
| sean-k-mooney | so 1 AZ that spans both datacheners 1 host aggreate per datacenter and a filter/weigher that used a datacenter tag on the host aggreate + server group infor to implment antiafinity | 11:49 |
| sean-k-mooney | but its much simpelr to have 1 AZ per datacenter and have octaivr just schdule differnt loadbanceer to differnt AZs | 11:49 |
| kevko | and this is not possible :/ | 11:49 |
| sean-k-mooney | its cleaner form an api level and more efficnt form a schdulign perspecitve | 11:50 |
| kevko | because octavia has only one AZ to configure | 11:50 |
| sean-k-mooney | right you woudl have to change octavia | 11:50 |
| sean-k-mooney | so im suggestign you make a featur equest to octavia instead of nova | 11:50 |
| kevko | yeah | 11:50 |
| kevko | thank you ! | 11:50 |
| kevko | you are always replying ;-) | 11:50 |
| sean-k-mooney | perhaps i reply too much on ocation | 11:52 |
| sean-k-mooney | but when people have ligtimate usecase i try to help them find a solution | 11:52 |
| sean-k-mooney | you do i just dont think the clean solution is in nova | 11:52 |
| sean-k-mooney | for older release you coud do this as i descibe with an out of tree weigher/filter if you have 1 AZ and multiple host aggreates but thats messy for a number of reasons | 11:53 |
| kevko | still thank you | 11:56 |
| opendevreview | Miguel proposed openstack/nova master: Pin autopep8 to 1.5.5 in tox https://review.opendev.org/c/openstack/nova/+/843443 | 12:16 |
| kevko | sean-k-mooney last question : Can i import octaviaclient in my own custom nova filter ? :P | 12:26 |
| sean-k-mooney | its your filter so :) | 12:26 |
| sean-k-mooney | upstrem no | 12:27 |
| sean-k-mooney | filters are not ment to make any db or api calls in general | 12:27 |
| sean-k-mooney | filters run per host so we really dont want to do that on large clouds | 12:27 |
| kevko | hmm, ok | 12:28 |
| kevko | can I write filter which will check security group assigned ? | 12:28 |
| sean-k-mooney | security groups? or server groups | 12:29 |
| kevko | both :D | 12:31 |
| kevko | i mean, if it is possible to check server groups it will be better, if no .. security group is same also | 12:32 |
| sean-k-mooney | for server groups you can use the existin filter as a refernce | 12:32 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py#L82-L165= | 12:33 |
| sean-k-mooney | but you cant add new policies | 12:33 |
| sean-k-mooney | so you cant jsut add an AZ-anti-affinity policy | 12:33 |
| sean-k-mooney | kevko: we dont have any example of the security gorups but if its in teh request spec then yes | 12:34 |
| kevko | i will have to try | 12:35 |
| sean-k-mooney | kevko: filters are passed 2 objects the hoststate object and a request spec object | 12:35 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L42= | 12:35 |
| sean-k-mooney | kevko: which has security groups https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L98= | 12:35 |
| sean-k-mooney | but only the security groups used for nova created ports | 12:36 |
| kevko | but no server group :/ | 12:36 |
| sean-k-mooney | it will not have the security groups of neutron ports that are passed in as uuids | 12:36 |
| sean-k-mooney | kevko: its called instance_group i think https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L91= | 12:37 |
| kevko | yeah, i just saw it | 12:37 |
| sean-k-mooney | the network request whil it has the port uuid https://github.com/openstack/nova/blob/e44b1a940fdc45cc9dbb08e193a8c25052cf64e7/nova/objects/network_request.py#L39= does not have any port secuirty groups | 12:40 |
| *** mnaser_ is now known as mnaser | 12:41 | |
| *** hemna2 is now known as hemna | 13:15 | |
| sean-k-mooney | gibi: stephenfin hopefully a quick one https://review.opendev.org/c/openstack/os-vif/+/843432 | 13:21 |
| * gibi clicks | 13:21 | |
| sean-k-mooney | just changing the template to zed since the py36 job is failing | 13:21 |
| gibi | sean-k-mooney: do you want to drop 3.6 support from the setup.cfg at the same time as we stop testing with 3.6? | 13:24 |
| sean-k-mooney | oh ya i can do that | 13:24 |
| sean-k-mooney | should i add a release note then too | 13:25 |
| gibi | yepp that a reno would make sense | 13:26 |
| sean-k-mooney | ok ill do that now | 13:26 |
| stephenfin | sean-k-mooney: replied | 13:40 |
| opendevreview | sean mooney proposed openstack/os-vif master: update job template to zed https://review.opendev.org/c/openstack/os-vif/+/843432 | 13:40 |
| sean-k-mooney | stephenfin: hehe well ^ is an updated version | 13:40 |
| sean-k-mooney | stephenfin: no we are not release independent | 13:41 |
| sean-k-mooney | i raised at the ptg and we said no keep it as release-with-intermidary | 13:41 |
| sean-k-mooney | so we shoudl not use the unversioned one | 13:41 |
| sean-k-mooney | if we want to make it release independent then sure | 13:41 |
| opendevreview | sean mooney proposed openstack/os-vif master: update job template to zed https://review.opendev.org/c/openstack/os-vif/+/843432 | 13:43 |
| sean-k-mooney | actully some typos fixed ^ | 13:44 |
| sean-k-mooney | ill leave it for now for ye to review but if you want me to change anything i can | 13:44 |
| sean-k-mooney | stephenfin: by the way your dont happen to be a bindep core? | 13:44 |
| gibi | sean-k-mooney: so we say py 3.10 is supported, which probably works, but I'm not sure the overall Zed release will go out with py3.10 supportr | 13:45 |
| sean-k-mooney | gibi: debain and ubuntu ship yoga on 3.10 | 13:45 |
| sean-k-mooney | so zed will be released on at least 3.10 by them | 13:46 |
| sean-k-mooney | its technially not in the testing runtimes but we have tests using it so i think it will be fine | 13:46 |
| gibi | yeah, hence my statement that it probably works without problem | 13:46 |
| gibi | OK, I'm convinced +@ | 13:46 |
| gibi | +2 | 13:46 |
| sean-k-mooney | i can drop that from the list if you like untile we have a 22.04 based job | 13:46 |
| sean-k-mooney | but i could also do that as a follow up | 13:47 |
| sean-k-mooney | i.e. add a 22.04 job | 13:47 |
| gibi | no, it is OK, we run py3.10 on os-vif so we know it works | 13:47 |
| sean-k-mooney | cool | 13:47 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: DNM: Testing live migration with local attach https://review.opendev.org/c/openstack/nova/+/843146 | 13:49 |
| *** artom_ is now known as artom | 14:04 | |
| *** bhagyashris is now known as bhagyashris|ruck | 14:22 | |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/xena: Reproduce live migration rollback w/o multi port bindings error https://review.opendev.org/c/openstack/nova/+/843336 | 14:50 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/xena: Fix LM rollback w/o multi port bindings extension https://review.opendev.org/c/openstack/nova/+/843337 | 14:50 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Reproduce live migration rollback w/o multi port bindings error https://review.opendev.org/c/openstack/nova/+/843338 | 14:58 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Fix LM rollback w/o multi port bindings extension https://review.opendev.org/c/openstack/nova/+/843339 | 14:58 |
| kashyap | gibi: Hey, before I go out on PTO I can't get this to completion this week :-( - https://review.opendev.org/q/topic:bp%252Fcpu-selection-with-hypervisor-consideration | 15:34 |
| gibi | kashyap: is there anything I can help with there? | 15:35 |
| kashyap | gibi: Stephen has a bunch of valid comments; need to address those and get it in. | 15:36 |
| gibi | ack | 15:37 |
| kashyap | gibi: Also, one more is the separate config class to parse the "cpu" bit from `virsh domcapabilitles` | 15:38 |
| gibi | yeah I remember | 15:38 |
| gibi | that | 15:38 |
| kashyap | Also have to address your own good comments on this patch: https://review.opendev.org/c/openstack/nova/+/762330 | 15:39 |
| kashyap | gibi: Ah, forgot that I already have the patch here for: https://review.opendev.org/c/openstack/nova/+/838191 (libvirt/config: Parse the 'cpu' element in domainCapabilities) | 15:40 |
| kashyap | (Modulo test) | 15:40 |
| gibi | yeah only test needed | 15:41 |
| kashyap | Actually, no that's not it...it's not adding a separate config class there | 15:41 |
| kashyap | gibi: From IRC log | 15:42 |
| kashyap | (Of this channel): | 15:42 |
| kashyap | 17:13 < kashyap> gibi: To tie up the lose end on caps vs domCaps -- the guidance from the libvirt folks is (a) no, we can't treat the caps == domCaps w/ 'host-model' mode; and (b) we should use domCaps wherever possible. | 15:42 |
| kashyap | 17:13 < kashyap> gibi: So that means, we should introduce a new config object | 15:42 |
| kashyap | 17:13 < gibi> kashyap: ack, make sense | 15:42 |
| gibi | hehh, I'm just following you guid here :) | 15:43 |
| kashyap | So, to sum up: | 15:43 |
| kashyap | (1) address your + Stephen's comments; (2) add the separate config object class to parse "cpu" element from domCaps, and work that in "properly" | 15:44 |
| kashyap | I could definitely use some help, if you have some time for this! I keep duking around this and never quite managing it fully | 15:44 |
| gibi | I cannot promise I can jump on this right away but I keep it in mind | 15:45 |
| gibi | but thanks for summing it up it helps keeping the context | 15:46 |
| opendevreview | Merged openstack/os-vif master: update job template to zed https://review.opendev.org/c/openstack/os-vif/+/843432 | 17:44 |
| opendevreview | Rajat Dhasmana proposed openstack/nova-specs master: Repropose volume backed server rebuild spec https://review.opendev.org/c/openstack/nova-specs/+/840155 | 18:41 |
| opendevreview | Miguel Lavalle proposed openstack/os-vif master: Delete trunk bridges to avoid race with Neutron https://review.opendev.org/c/openstack/os-vif/+/841499 | 19:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!