Wednesday, 2021-01-06

« Tuesday, 2021-01-05 Index Thursday, 2021-01-07 »
*** tosky has quit IRC00:23
*** LinPeiWen has joined #openstack-nova00:29
*** spatel has joined #openstack-nova00:46
*** spatel has quit IRC00:52
*** lifeless has quit IRC00:53
*** lifeless has joined #openstack-nova00:55
*** tbachman has quit IRC01:13
*** tbachman has joined #openstack-nova01:14
openstackgerritmelanie witt proposed openstack/nova master: Enable test_volume_backed_live_migration in tempest  https://review.opendev.org/c/openstack/nova/+/52810401:17
*** gyee has quit IRC01:25
*** sapd1 has joined #openstack-nova01:30
brinzhang0bauzas, sean-k-mooney: I have replied the cyborg shelve/unshleve patch, pls review again, and I think there is no need to update in, one comments need to move the comments, and I will addressed in the followed up patch01:30
*** martinkennelly has quit IRC01:43
*** zenkuro has joined #openstack-nova02:04
*** macz_ has joined #openstack-nova02:12
*** spatel has joined #openstack-nova02:26
*** spatel has quit IRC02:28
*** psachin has joined #openstack-nova03:30
*** hemanth_n has joined #openstack-nova03:38
*** mkrai has joined #openstack-nova03:40
*** mgoddard has joined #openstack-nova04:08
*** sapd1_x has joined #openstack-nova04:39
*** ratailor has joined #openstack-nova04:40
*** sapd1 has quit IRC04:43
*** ociuhandu has joined #openstack-nova05:08
*** ociuhandu has quit IRC05:13
*** dave-mccowan has quit IRC05:19
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-nova05:33
*** sapd1_x has quit IRC05:38
*** whoami-rajat__ has joined #openstack-nova05:41
*** zzzeek has quit IRC05:42
*** zzzeek has joined #openstack-nova05:43
*** macz_ has quit IRC05:45
*** zzzeek has quit IRC05:50
*** zzzeek has joined #openstack-nova05:52
*** sapd1_x has joined #openstack-nova05:57
*** zzzeek has quit IRC05:59
*** zzzeek has joined #openstack-nova06:00
*** zzzeek has quit IRC06:05
*** zzzeek has joined #openstack-nova06:11
*** zzzeek has quit IRC06:18
*** zzzeek has joined #openstack-nova06:19
*** zzzeek has quit IRC06:46
*** zzzeek has joined #openstack-nova06:49
*** sapd1_x has quit IRC06:51
openstackgerritMamduh proposed openstack/os-vif stable/queens: Refactor code of linux_net to more cleaner and increase performace  https://review.opendev.org/c/openstack/os-vif/+/76594106:53
openstackgerritMamduh proposed openstack/os-vif stable/queens: Fix - os-vif fails to get the correct UpLink Representor  https://review.opendev.org/c/openstack/os-vif/+/76598306:53
*** mkrai has quit IRC06:56
*** sapd1_x has joined #openstack-nova07:06
*** zzzeek has quit IRC07:09
*** zzzeek has joined #openstack-nova07:12
*** zzzeek has quit IRC07:21
*** zzzeek has joined #openstack-nova07:22
*** dklyle has quit IRC07:46
*** macz_ has joined #openstack-nova07:46
*** macz_ has quit IRC07:50
*** zzzeek has quit IRC07:50
*** zzzeek has joined #openstack-nova07:54
*** mkrai has joined #openstack-nova07:59
*** sapd1_x has quit IRC07:59
*** rpittau|afk is now known as rpittau08:01
*** tesseract has joined #openstack-nova08:03
*** zzzeek has quit IRC08:16
*** zzzeek has joined #openstack-nova08:19
openstackgerritBalazs Gibizer proposed openstack/nova master: Remove dead code from SchedulerReportClient  https://review.opendev.org/c/openstack/nova/+/76946708:21
*** andrewbonney has joined #openstack-nova08:23
*** elod_pto is now known as elod08:48
*** CeeMac has joined #openstack-nova09:01
*** zzzeek has quit IRC09:03
*** zzzeek has joined #openstack-nova09:04
*** zenkuro has quit IRC09:26
*** mkrai has quit IRC09:27
*** zenkuro has joined #openstack-nova09:27
*** ociuhandu has joined #openstack-nova09:34
*** alexe9191 has joined #openstack-nova09:41
alexe9191Hello There,09:42
*** zzzeek has quit IRC09:42
alexe9191I was wondering if I can run a newer version of nova-api/scheduler/conductor on an older version of nova-compute. Nova-compute would be kili for instance and the newer version of the API services would be on Pike or Rocky for instance?09:42
*** zzzeek has joined #openstack-nova09:43
gibialexe9191: nova keeps compatibility between version N+1 controller and version N compute service09:44
gibiso you can have Rocky controller services and Pike compute services09:45
gibisorry, so you can have Rocky controller services and Queens compute serivces09:45
*** mkrai has joined #openstack-nova09:46
gibias Q + 1 = R09:46
alexe9191interesting, what out of curiosity. Does this also work the other way around?09:46
alexe9191so nova-compute R and nova-api Q ?09:46
*** macz_ has joined #openstack-nova09:47
*** derekh has joined #openstack-nova09:48
gibino, the controller services need to be upgraded first09:49
alexe9191I am also guessing that I can't immediately upgrade the nova-compute from kilo to rocky in one go? This also has to be done in a rolling fashion? So basically, K -> L -> etc > R on the nova-computes ?09:50
*** zzzeek has quit IRC09:51
gibialexe9191: what we test is upstream is rolling upgrade as you guessed. There are some support for fast forward upgrade where you move more than one release forward in single step09:51
gibihttps://wiki.openstack.org/wiki/Fast_forward_upgrades09:51
*** martinkennelly has joined #openstack-nova09:52
*** macz_ has quit IRC09:52
*** zzzeek has joined #openstack-nova09:53
alexe9191I honestly fail to understand that a little bit as the documentation is not really clear on fast forward upgrades09:54
*** slaweq has joined #openstack-nova09:55
sean-k-mooneyalexe9191: fast forward upgrades are not really a feature of the comonet project more an installer feature that the compoent project like nova try to not intentionally break09:55
sean-k-mooneystictly speaking nova only support n to n+109:55
alexe9191So basically this fast forward *feature* is more of a wrapper around what nova support ?09:56
sean-k-mooneynot quite it a way of runnign all the db updates and migration for the intermidat release so that you can skip them09:57
sean-k-mooneyon the nova side we try to keep that logic around for multiple release so that if you skip 3 version of the compute agent it will still have the compatibley code09:57
*** ociuhandu has quit IRC09:57
sean-k-mooneyso nova "supports" fast forward upgrades09:58
alexe9191So for instance running the DB migration scripts from newton while one is running Kilo ?09:58
sean-k-mooneyby keeping the compat code for several release beyond when it was striclty reuired09:58
sean-k-mooneywhen you do an FFU there is a period where the contolplane is in accessable.09:59
sean-k-mooneybasiclly you move the contoler services to newton while the computes are on kilo09:59
sean-k-mooney then you move the compute to newton09:59
sean-k-mooneywhen you do that in principal the newton contoler cannot manage the kilo compute but the vms keep running10:00
sean-k-mooneywhen you bring the compute back up to newton manageablity is restored10:00
*** zzzeek has quit IRC10:00
*** zzzeek has joined #openstack-nova10:00
sean-k-mooneyin pratice if there has been no RPC major version bump between the two version you have deploy the contoler could tehoretically mange compute more then 1 releasee old10:01
sean-k-mooneybut we dont test that10:01
alexe9191ok, and rolling upgrade being moving everything at once from one release to the other, both the APi & the nova-compute agents, K, L, N, ...10:01
sean-k-mooneythere is typically 5-6 release betwwen major rpc version bumps10:01
alexe9191That's very helpful to know!10:02
*** ociuhandu has joined #openstack-nova10:02
sean-k-mooneyalexe9191: rolling upgrades are where you move the contolplane in one go to n+1, then move the computes in a "rolling" or incremental fashion 1 by 110:02
sean-k-mooneyrolling upgrades allow you to maintain managmeablity of the computes with mixed verions10:03
sean-k-mooneybut we only test a delta of 1 verions10:03
sean-k-mooneywe write the code such that it could be a larger delta but the test matix is too large for use to offialy support that10:04
alexe9191Right, so if upgrading K to L. I could basically roll the upgrade to the compute nodes one rack at a time from K to L10:04
sean-k-mooneyyes10:04
alexe9191So in theory one could do N+2 but that's not tested.10:04
sean-k-mooneyyes in princiapl n->n+2 should work provided there is not an rpc major version bump in between10:05
sean-k-mooneybut really you would be the first to test that10:05
alexe9191I'd rather stick to what you guys are doing upstream in that case.10:06
sean-k-mooneywe have been on RPC version 5.X for about 5 releases now10:06
*** mkrai has quit IRC10:06
sean-k-mooneyalexe9191: ya if this is a prod deployment that is likely your safest option10:06
alexe9191I am guessing there are major changes between K and R in RPC versions, is that documented in the release note?10:06
*** mkrai_ has joined #openstack-nova10:06
alexe9191Yes, which sounds that doing the rolling-upgrades is the safest supported solution.10:07
sean-k-mooneyi think it would be in the release notes but i can quickly check the code10:08
alexe9191Fantastic, if I could also know which file you are checking in the source tree that would be great as well.10:08
sean-k-mooneywe have each verion bump documented with a comment https://github.com/openstack/nova/blame/master/nova/compute/rpcapi.py#L7610:09
sean-k-mooneyhttps://github.com/openstack/nova/blame/master/nova/compute/rpcapi.py#L38510:09
alexe9191Awesome!10:09
sean-k-mooneythe aliase are the quickest way to check10:09
alexe9191Interesting, so K to P is on the same Major RPC release.10:10
sean-k-mooneyso kilo was 4.010:10
sean-k-mooneyrocky was 5.010:10
sean-k-mooneytechnically 5.0 is the ame as 4.2210:10
sean-k-mooneyso if rocky still has the 4.22 compat code then you could pin the  rpc version to 4.0 in principal10:11
sean-k-mooneybut it proably will have bugs10:11
alexe9191This is really helpful!10:12
sean-k-mooneyhonestly i doubt we actully manged to keep perfect rpc compate and i suspect rocky has had the 4.x compat code removed10:13
sean-k-mooneybut looking quickly it looks like the compat code was maybe removed in stine10:14
*** zzzeek has quit IRC10:14
sean-k-mooneywithout testign it first however i would not bet on it it fully working10:14
alexe9191If I am not mistaken, this could mean that in theory I could run Newton API on kilo nova compute. Since the RPC API is on the same major version.10:15
sean-k-mooneyin theory yes10:15
sean-k-mooneyin practice you would likely be the first to try that and you would have to let us know if it works :)10:16
alexe9191I will most definitely do that :)10:16
sean-k-mooneyyou would pin the RPC version of the newton contolplane to 4.010:16
alexe9191This should be done in nova.conf I could imagine ?10:16
sean-k-mooneyyou can do that via the upgrade levels in the nova.conf10:16
*** zzzeek has joined #openstack-nova10:17
alexe9191I am gonna let you guys know once I test that! Thanks a lot for all of the helpful information.10:19
bauzassean-k-mooney: alexe9191: well, Kilo is very prehistoric10:19
bauzasand from the RPC PoV, we only started to introduce massively nova objects by this time10:20
sean-k-mooneyit is but so is newton :P10:20
bauzasat newton, we mostly stopped to use dicts on RPC calls10:20
bauzaswhile during kilo, most of the values were those10:20
sean-k-mooneymostly but we still have a few even on master10:20
bauzasspeaking of RPC compat, I'd say a nova boot *could* work10:21
sean-k-mooneybauzas: having random dicts of stings migth actully help in this case10:21
sean-k-mooneyprovide the required data is there10:21
bauzasmaybe10:21
bauzasanyway, all of this is very experimental10:22
bauzasbut I sincerely hope alexe9191 is not expecting live-migration to work with this env :)10:23
sean-k-mooneyoh yes it is.10:23
alexe9191bauzas no and this is not a production environment where I will be testing. Our of curiosity i am going to run an experiment but for production I am probably gonna do the rolling upgrade.10:26
bauzasalexe9191: anyway, I'm very curious about your experiment results :)10:28
bauzasI'm personnally not opposed to have operators breaking the N+1 rule by running very old computes if they wish10:29
bauzasprovided they understand the maintenance price10:29
bauzasbut... Kilo/Rocky is like a gymnast front oversplit to me :)10:31
bauzasfront side* even10:31
sean-k-mooneyone of the cost is really you can only safely use the max api version of the oldest compute10:31
sean-k-mooneyas in microversion10:31
sean-k-mooneyyou might be able to use later microverions but if they needed an rpc change it wont work10:32
sean-k-mooneywe will backlevel the request but you wont get teh behavior the new microverion implies10:32
gibistephenfin: I finished reviewing the migration compacting series, nice work!10:33
stephenfingibi: Thanks :)10:33
gibiI have couple of questions in the middle but overall I think we are good10:33
stephenfinIf you want me to review anything, just let me know. I'm almost done on the os-hypervisors fixup (many rabbit holes)10:33
stephenfinSweet. Will take a look shortly10:33
sean-k-mooneybauzas: by the way i have not got around to doing the full code review for the cyborg shevle patches but did my respoce make sense10:35
*** dtantsur|afk is now known as dtantsur10:35
sean-k-mooneyregarding the conductor chagnes10:35
*** zzzeek has quit IRC10:35
bauzassean-k-mooney: I'll look at them later today, I was on a meeting just before and I want to rebase my own routed networks series10:37
sean-k-mooneyno worries10:37
bauzasin other words, I lag :D10:37
*** slaweq has quit IRC10:38
*** zzzeek has joined #openstack-nova10:38
gibistephenfin: there is a trivial review https://review.opendev.org/c/openstack/nova/+/769467 you can lok at10:38
sean-k-mooneyhehe well i was indening to do a fully review yesterady and havent started other then to respond ot your comments so your still ahead of me10:39
stephenfingibi: have it open already :)10:39
gibibesides that I have nothing :)10:40
gibiI need to do a bunch of rebases10:40
sean-k-mooneyspeaking of rebases i wil be rebasing my vdpa spec in a day or so but if people have time it would be nice if ye could review that spec https://review.opendev.org/c/openstack/nova-specs/+/764999 and also the numa port polices spec https://review.opendev.org/c/openstack/nova-specs/+/76590110:49
sean-k-mooneyi intend to start pushing some standalone patches for those this week namely the xml generation for vdpa and some constants for the newtuon extensions10:50
sean-k-mooneythe larger change will still be a few weeks out but i would like to start submitting an merging some of the smaller patches which would be dead code but easilly reviewable while i work on the more complex part but the spec need to be approved first before that can happen10:51
*** mkrai_ has quit IRC11:03
openstackgerritBalazs Gibizer proposed openstack/nova master: Use the non polling notification waiter in func test  https://review.opendev.org/c/openstack/nova/+/75844511:07
openstackgerritBalazs Gibizer proposed openstack/nova master: Create a fixture around fake_notifier  https://review.opendev.org/c/openstack/nova/+/75844611:07
openstackgerritBalazs Gibizer proposed openstack/nova master: Use NotificationFixture for legacy notifications too  https://review.opendev.org/c/openstack/nova/+/75844811:07
openstackgerritBalazs Gibizer proposed openstack/nova master: Test the NotificationFixture  https://review.opendev.org/c/openstack/nova/+/75845011:09
openstackgerritBalazs Gibizer proposed openstack/nova master: Move fake_notifier impl under NotificationFixture  https://review.opendev.org/c/openstack/nova/+/75845111:09
openstackgerritStephen Finucane proposed openstack/nova master: api: Drop statistics-style fields from os-hypervisors  https://review.opendev.org/c/openstack/nova/+/76404011:16
openstackgerritStephen Finucane proposed openstack/nova master: tests: Remove 'test_extended_hypervisors'  https://review.opendev.org/c/openstack/nova/+/76951911:16
openstackgerritStephen Finucane proposed openstack/nova master: api: Normalize exception handling for os-hypervisors  https://review.opendev.org/c/openstack/nova/+/76952011:16
stephenfingibi: Done ^ Sorry for the delay11:17
* stephenfin looks at DB patches11:17
gibion it11:17
*** masterpe has quit IRC11:20
*** LinPeiWen has quit IRC11:25
*** masterpe has joined #openstack-nova11:29
openstackgerritSylvain Bauza proposed openstack/nova master: Add requested_networks field to RequestSpec object  https://review.opendev.org/c/openstack/nova/+/74997711:29
openstackgerritSylvain Bauza proposed openstack/nova master: WIP: Add a routed networks scheduler pre-filter  https://review.opendev.org/c/openstack/nova/+/74906811:29
sean-k-mooneydoes anywone know if shareing resouce providers actully work by the way11:43
sean-k-mooneyits in relateion to http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019647.html11:43
sean-k-mooneyspecifically for the RBD image backend11:43
sean-k-mooneyif we used provider.yaml to set the local disk_gb inventory to 0 and created a shareing resocue provider with a disk_gb inventory for the ceph pool11:44
lyarwoodI don't think anyone was working on that11:44
sean-k-mooneythen added the MISC_SHARES_VIA_AGGREGATE trait to the RP and adedd the RP and the comptue to an aggreate would it work11:45
sean-k-mooneylyarwood: they have not been for like 2 years11:45
sean-k-mooneybut i tought the placment side was done11:45
sean-k-mooneyi just dont know if anyone has ever tested it11:45
gibisean-k-mooney: we don't have functional or other integration tests for it so I assume that it does not work11:45
lyarwoodno idea sorry11:46
sean-k-mooneyhehe ya that my base assumtion with openstack too11:46
sean-k-mooneyif its not tested its broken11:46
sean-k-mooneyif it did work technially operators/installer tooll could configure it and actully get correct reporting for shared storage this way11:47
sean-k-mooneythe real question is will the placement allocation candiate include the inventory form the sharing provider11:47
sean-k-mooneyusing the request we currently make11:48
sean-k-mooneyit "should" but i have no idea if it will actully work11:48
sean-k-mooneyplacment has functrion api test for shared resouces https://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/shared-resources.yaml11:49
sean-k-mooneyhttps://github.com/openstack/placement/blob/master/placement/tests/functional/gabbits/shared-resources.yaml#L135-L14311:50
sean-k-mooneyit looks like it should work11:50
sean-k-mooneyif you read those top to bottom11:51
gibisean-k-mooney: I'm not sure that our resize and migrate logic works properly with sharing providers11:52
sean-k-mooneyif i had time i would really like to test that and write that up in docs because it would be greate do document how to correctly deploy using sharing agggreats to model storage11:52
gibicurrently we copy the allocation including the part that is allocated from  sharing providers11:53
gibieven if the same sharing provider shares with the new destination11:53
sean-k-mooneywe would just do double allcoations then11:53
sean-k-mooneywhich i think is ok11:53
sean-k-mooneyit would reconsile the vaules wehn we do resize confirm11:53
gibihm, yes11:54
sean-k-mooneyi mean its not perfect but its not dangourous11:54
sean-k-mooneyit would jsut cause some failures wehn near capasity11:54
*** ociuhandu has quit IRC11:59
*** evrardjp has left #openstack-nova12:00
*** ociuhandu has joined #openstack-nova12:00
gibistephenfin: left feedback in https://review.opendev.org/c/openstack/nova/+/76404012:00
sean-k-mooneygibi: do we have any ci tests using provider.yaml12:01
gibisean-k-mooney: hm, good question12:01
gibiI don't think we have tempest tesd12:01
gibitest12:02
sean-k-mooneyim wondering if i could maybe test this theory with a change to the ceph job12:02
gibiwe have some functional test12:02
gibihttps://github.com/openstack/nova/blob/ccb2e11129d4a0730b0ba19a7f49ed8fc5a05f6d/nova/tests/functional/compute/test_resource_tracker.py#L66512:03
*** ociuhandu has quit IRC12:05
sean-k-mooneyspecificly i was thinking of configure both compute nodes to  report 0 disk_gb and having a pretest hook create the sharing resouce provider12:05
gibiI think it can be done12:06
sean-k-mooneyya i think it would be worth testing at least12:07
sean-k-mooneyill have to try and make time to do it.12:08
openstackgerritStephen Finucane proposed openstack/nova master: api: Drop statistics-style fields from os-hypervisors  https://review.opendev.org/c/openstack/nova/+/76404012:08
stephenfingibi: done12:08
lyarwoodsean-k-mooney: I'd be interested in helping with that12:09
lyarwoodonce I've finished this machine type stuff12:10
stephenfingibi++ thanks :)12:11
stephenfingmann: When you're around, could do with some input on https://review.opendev.org/c/openstack/nova/+/765798/ I'm kind of lost as to what the next steps are /o\12:12
sean-k-mooneyhehe i was hoping you would say that. long term it would be nice to be able to do it automatically but if this worked we could do it via an install and docuemnt it for this release and figure out how to make it work out of the box in the future12:12
gibistephenfin: further comment in https://review.opendev.org/c/openstack/nova/+/76404012:12
*** ratailor has quit IRC12:14
lyarwoodsean-k-mooney: yup agreed12:14
*** tosky has joined #openstack-nova12:15
*** lpetrut has joined #openstack-nova12:17
*** slaweq has joined #openstack-nova12:17
sean-k-mooneythe main downside to this approch is without a reshape this would only work for new installs12:19
sean-k-mooneybut it would work for new installs so that would at least be progress12:19
sean-k-mooneyfor existing deployments we would need to reshape the allcoation form the compute node rp to the shareing one12:20
sean-k-mooneyor sharing ones, you coudl have 1 ceph cluster per az or cell or something consiveably12:21
sean-k-mooneyaggrates would take care of modeling that however12:21
lyarwoodyeah the upgrade case isn't going to be fun to handle tbh12:23
*** sapd1 has joined #openstack-nova12:24
sean-k-mooneyno matter what we do i dont see that improving much. sure if we can write a reshap but it will he complicated12:25
sean-k-mooneyi suspect a nova manage command would acatully be a better approch12:25
*** ociuhandu has joined #openstack-nova12:26
*** ociuhandu has quit IRC12:26
sean-k-mooneywhere you spcify the sharing rp and the compute host to reshape12:26
sean-k-mooneyhum actully i guess placment-mange not nova mange as this is all on the placement side12:27
*** ociuhandu has joined #openstack-nova12:27
*** ociuhandu has quit IRC12:28
*** ociuhandu has joined #openstack-nova12:29
*** ociuhandu has quit IRC12:35
*** zzzeek has quit IRC12:46
*** zzzeek has joined #openstack-nova12:49
*** ociuhandu has joined #openstack-nova12:51
*** martinkennelly has quit IRC12:54
*** ociuhandu has quit IRC13:02
*** hemanth_n has quit IRC13:06
*** ociuhandu has joined #openstack-nova13:07
*** brinzhang0 has quit IRC13:12
*** ociuhandu has quit IRC13:12
*** slaweq has quit IRC13:15
*** sapd1 has quit IRC13:19
*** ociuhandu has joined #openstack-nova13:33
sean-k-mooneylyarwood: by the way have you reviewd https://review.opendev.org/c/openstack/cinder-specs/+/76673213:35
sean-k-mooneyits propsoing extending os-brick to spawn a deamon process ot monitor and heal NVMEoF volules created over MD raids13:36
sean-k-mooneye.g. havign it actily monitoing the underlying raid config and healing it13:37
*** ociuhandu has quit IRC13:37
sean-k-mooneyunfortunetly that is now approved on the cinder cide but i dont think this should be in the scope of os-brick to do personlally13:37
sean-k-mooneygibi: lyarwood  did this come up in the nova cinder cross project dicusstion at the ptg?13:38
sean-k-mooneygibi: lyarwood  i dont see any nova review on the spec at all13:38
sean-k-mooneythis is the os-brick patch https://review.opendev.org/c/openstack/os-brick/+/76857613:39
sean-k-mooneygibi: we had some discussion with qqmber about this on monday13:41
lyarwoodsean-k-mooney: iirc it came up in the cinder track ahead of the nova track starting up13:50
lyarwoodsean-k-mooney: it's definitently not in n-cpu's wheel house to look after stuff like this so I'm not sure where you would have it if not os-brick?13:51
sean-k-mooneyin a standalone agnet13:52
sean-k-mooneythe fact is it would be executing in the n-cpu process13:52
sean-k-mooneygranted as a child process spawned by os-brick13:53
sean-k-mooneybut in any case it would be in the nova-compute contianer13:53
*** alexe9191 has quit IRC13:53
sean-k-mooneyfor me it makes far more sense to make this its own standalone deamon13:53
*** ociuhandu has joined #openstack-nova13:55
lyarwoodYeah I get that but to rearch that layer between os-brick and another local agent would be a huge amount of work13:55
lyarwoodah wait13:55
lyarwoodisn't that the proposal anyway13:55
* lyarwood opens the spec again13:55
sean-k-mooneythe propsoal is to add an agent into os-brick and have os-bick spawn the agent into a new process from the nova compute agent13:56
sean-k-mooneyso a seperate deamon coudl use os-brick to do all the work13:56
sean-k-mooneywhat i dont like is the magic spawning of a seperate process . granted we kind of do that for privsep but  that is differnt13:57
sean-k-mooneythe lifetime fo rhte privesep process is related to the lifetiem of the nova-compute service13:57
lyarwoodright13:57
sean-k-mooneythe lifetime or this agent would not be13:57
lyarwoodsorry I had this the wrong way around13:58
lyarwoodI thought they already had a deamon somewhere13:58
* lyarwood adds the spec to his list to review in full later13:58
sean-k-mooneyits already approved on the cinder side13:58
sean-k-mooneyi would have -1 it if it wasnt merged.13:59
*** nweinber has joined #openstack-nova14:10
*** dave-mccowan has joined #openstack-nova14:11
openstackgerritStephen Finucane proposed openstack/nova master: api: Drop statistics-style fields from os-hypervisors  https://review.opendev.org/c/openstack/nova/+/76404014:13
stephenfingibi: third time lucky :D ^14:14
openstackgerritLee Yarwood proposed openstack/nova-specs master: libvirt: Update instance machine type stash spec  https://review.opendev.org/c/openstack/nova-specs/+/76954714:42
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Record the machine_type of instances in system_metadata  https://review.opendev.org/c/openstack/nova/+/76753314:42
openstackgerritLee Yarwood proposed openstack/nova master: WIP nova-manage: Add commands for managaing instance machine type  https://review.opendev.org/c/openstack/nova/+/76954814:42
sean-k-mooneystephenfin: how is your review queue at the moment14:59
stephenfinsean-k-mooney: Manageable. What's up?15:00
sean-k-mooneycan you take a look at this small bug fix https://review.opendev.org/c/openstack/nova/+/767368 and when you have time my spec for port numa polices https://review.opendev.org/c/openstack/nova-specs/+/76590115:01
sean-k-mooneyi still have to update the vdpa spec but that what im going to do now15:01
stephenfinsure thing15:02
gibisean-k-mooney: if nova needs to know (communicate with) a new agent (spawned by os-brick) then we need a nova spec about it. If the os-brick change is transparent to nova then meh15:02
gibisean-k-mooney: do we define what is in a nova compute container upstream?15:03
gibistephenfin: will look in a sec15:03
sean-k-mooneypersonally im a little uncomfortably with our dependient libs effectivly injecting arbitry code that executes as a persitent deamon15:03
sean-k-mooneygibi: no nova does not kolla/ooo/loci do15:04
sean-k-mooneywe do try to keep it at 1 process per container for the most part bar privsep15:04
gibithen those projects needs to be involved to the dicussion as they are impacted by a new agent15:04
gibistephenfin: I'm +2 on https://review.opendev.org/c/openstack/nova/+/76404015:05
sean-k-mooneythe way its proposed it would jsut run in the continer without there interventions or knowladge15:05
sean-k-mooneyalthough in the contier case the new agent would still implictly share the same lifetime as the nova agent since tehy are in the same container15:05
stephenfingibi: Great. Thanks!15:06
gibistephenfin: new you are the closest to grab the microversion 2.8815:06
gibisean-k-mooney: that is actually a good argument for a separate service as they want independent lifetime15:07
stephenfingibi: Yup. I'm going to ask gmann to take a look at that later, assuming he's back from his PTO15:07
gibistephenfin: cool15:07
*** sapd1 has joined #openstack-nova15:20
*** dklyle has joined #openstack-nova15:26
*** derekh has quit IRC15:27
*** lpetrut has quit IRC15:27
*** ociuhandu has quit IRC15:34
*** mgoddard has quit IRC15:41
*** mgoddard has joined #openstack-nova15:41
*** spotz has quit IRC15:44
*** ociuhandu has joined #openstack-nova15:46
*** alexe9191 has joined #openstack-nova15:46
*** mgoddard has quit IRC15:47
*** ociuhandu has quit IRC15:56
openstackgerritKashyap Chamarthy proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities  https://review.opendev.org/c/openstack/nova/+/67379015:59
*** mlavalle has joined #openstack-nova16:01
*** macz_ has joined #openstack-nova16:10
*** macz_ has quit IRC16:10
*** macz_ has joined #openstack-nova16:11
dansmithlyarwood: I was just about to +2 that machine type sysmeta spec rev, but saw that stephenfin already did.. I have annoying unrelated questions, one of which might be an accidental state omission16:11
dansmithlyarwood: if it 's cool, I'll just +2 and let you quickly look if you want to throw the other state in there to avoid a rev-to-the-rev,16:12
dansmithand I can just +W if I'm wrong or you want it separate16:12
lyarwooddansmith: just on a call now, happy to answer your questions before this lands. I'll take a look once this is over.16:16
dansmithlyarwood: ack16:16
*** jamesden_ is now known as jamesdenton16:46
*** gyee has joined #openstack-nova16:50
*** ociuhandu has joined #openstack-nova16:51
*** ociuhandu has quit IRC16:52
*** ociuhandu has joined #openstack-nova16:52
*** ociuhandu has quit IRC16:52
*** tesseract has quit IRC17:02
*** sapd1 has quit IRC17:07
*** mgoddard has joined #openstack-nova17:10
*** ircuser-1 has joined #openstack-nova17:20
*** sapd1 has joined #openstack-nova17:21
lyarwooddansmith: replied, need to drop for dinner but I'll respin this evening.17:22
dansmithack17:23
*** psachin has quit IRC17:23
*** ociuhandu has joined #openstack-nova17:27
gmannstephenfin: ack, I will review today.17:30
*** ociuhandu has quit IRC17:33
*** rpittau is now known as rpittau|afk17:43
*** sapd1 has quit IRC17:47
stephenfingmann: thanks17:47
*** dtantsur is now known as dtantsur|afk18:27
*** zenkuro has quit IRC18:37
*** andrewbonney has quit IRC18:56
openstackgerritsean mooney proposed openstack/nova master: [WIP] test numa and vcpu topologies  https://review.opendev.org/c/openstack/nova/+/76960118:58
sean-k-mooneymelwitt:^18:58
melwittthanks18:59
sean-k-mooneyi wonder is there an easy way to print the xml that would be created19:00
*** ociuhandu has joined #openstack-nova19:13
*** tbachman has quit IRC19:17
*** bnemec has quit IRC19:18
sean-k-mooneyah i can use OS_DEBUG=1 and add assert False19:20
sean-k-mooneyand it then dumps all the logs19:20
sean-k-mooneyhttp://paste.openstack.org/show/801461/19:22
sean-k-mooneythe numa toplogy is correct but the cpu toplogy is not19:24
sean-k-mooney    <topology sockets="8" cores="1" threads="1"/>19:24
melwitta-ha, cool (about the logs)19:24
*** bnemec has joined #openstack-nova19:24
sean-k-mooneythat was the xml for https://review.opendev.org/c/openstack/nova/+/769601/1/nova/tests/functional/libvirt/test_numa_servers.py#15819:24
melwittack19:25
sean-k-mooneyoh my extra specs are wrong19:25
sean-k-mooneywiat did i copy those form the customer19:25
sean-k-mooneyno they have the correct ones im missing cpu_19:26
*** ociuhandu has quit IRC19:26
*** tbachman has joined #openstack-nova19:26
sean-k-mooneyok i repoduced it now19:28
sean-k-mooneyhttp://paste.openstack.org/show/801463/19:28
melwittomg !19:29
sean-k-mooneythat is the error19:29
melwittyes, looks like you've got it. sweet19:29
*** bnemec has quit IRC19:30
melwittyeah there's the IndexError which is a separate thing but once that is fixed it will still be an erroneous failure to schedule. need to open I guess two bugs, one for the IndexError and one for the scheduling fail19:30
openstackgerritsean mooney proposed openstack/nova master: [WIP] test numa and vcpu topologies  https://review.opendev.org/c/openstack/nova/+/76960119:31
sean-k-mooneythe index error i think is jsut use trying to get the first entry in the list of toplogies19:32
*** bnemec has joined #openstack-nova19:32
sean-k-mooneybut its empty so that is simple to fix19:32
melwittcorrect19:32
sean-k-mooneyactully just lookking a littel more up in the logs http://paste.openstack.org/show/801464/19:33
sean-k-mooneywe have the limits printed19:33
sean-k-mooneyso it has the flaovr limits there  Flavor limits 2:2:819:34
melwittyeah. that is showing on the customer case too19:34
sean-k-mooneyya so the issue is the preference of 0,0,019:34
melwittyeah but we use 0 (on master) and -1 (on queens) to mean "no preference"19:35
sean-k-mooneyyep so we need to adjust that logic slightly19:35
sean-k-mooneyso that the closet match code works19:35
*** bnemec has quit IRC19:37
*** bnemec has joined #openstack-nova19:39
melwittsean-k-mooney: did you want to work on the fix? or did you want me to stack a change on top of your test?19:41
sean-k-mooneyim testing a quick hack but sofar its not working so if you want to stack a change on top go for it19:43
sean-k-mooneythat is what im trying currrntly but the two things i have tired so far have no effect :)19:43
melwittoh heh. well it's up to you19:44
sean-k-mooneyoh its becasue its using the numa toplogy object i think19:46
openstackgerritMerged openstack/nova master: tests: Remove 'test_extended_hypervisors'  https://review.opendev.org/c/openstack/nova/+/76951919:46
sean-k-mooneyill give it one more try then im going for dinner and its all yours19:46
melwitthehe, k. well it was your idea on how to fix it :) so if you want to stick with it that wfm. if not, I can give it a try19:48
openstackgerritsean mooney proposed openstack/nova master: [WIP] fix max cpu topologies with numa affinity  https://review.opendev.org/c/openstack/nova/+/76961419:51
sean-k-mooneythis is probaly not right but it passes the new test i added19:51
melwittack19:52
sean-k-mooneythe vcpu toplogy cannot cahnge per numa node19:52
sean-k-mooneyso i  dont actully think we shoul dbe looking at the numa node there19:52
sean-k-mooneyhum i just ran all the libvirt func tests and they passed19:53
sean-k-mooneyim running the full set now19:53
sean-k-mooney:( i hit the subunit parser bug....20:00
sean-k-mooney  File "/opt/repos/nova/.tox/functional/lib/python3.8/site-packages/subunit/v2.py", line 227, in _write_packet20:01
sean-k-mooney    raise ValueError("Length too long: %r" % base_length)20:01
sean-k-mooneyValueError: Length too long: 507254820:01
sean-k-mooneythere are dissadvatages to OS_DEBUG=120:01
sean-k-mooneyok im going to have dinner and we can see what the ci says but i thinnk that working acourdign to to the func tests that have run but i think i have disabled too much20:03
sean-k-mooneymelwitt: wem might actully be able to just remove https://github.com/openstack/nova/blob/ccb2e11129d4a0730b0ba19a7f49ed8fc5a05f6d/nova/virt/hardware.py#L616-L63720:07
sean-k-mooneyit was added by https://github.com/openstack/nova/commit/770ab8eeb72b184ac6164aeabb89c4bf45f938a920:07
sean-k-mooneybut in not conviced the resoning is correct20:07
melwitthm ok20:08
sean-k-mooneythe guest vcpu toplogy is indepenet of its numa toplogy20:09
sean-k-mooneythis patch was trying to support different guest vcpu toplogies per numa node which we have never supported20:10
sean-k-mooneyat least looking at https://github.com/openstack/nova/commit/770ab8eeb72b184ac6164aeabb89c4bf45f938a9#diff-027adff210e1ed4c63524a486cb988702337880b5c0773482200a136e391ecfbR77320:11
sean-k-mooneyit looks like we still have some of those unit tests https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/test_hardware.py#L792-L88120:16
sean-k-mooneythe last one is definetly not requesatable via the api20:17
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/tests/unit/virt/test_hardware.py#L858-L88320:17
*** ociuhandu has joined #openstack-nova20:17
sean-k-mooneyoh my func test run passed locally too.20:17
sean-k-mooneyok really getting dinner now o/20:18
*** ociuhandu has quit IRC20:18
*** ociuhandu has joined #openstack-nova20:19
*** ociuhandu has quit IRC20:19
*** ociuhandu has joined #openstack-nova20:20
*** whoami-rajat__ has quit IRC20:21
*** ociuhandu has quit IRC20:29
*** ociuhandu has joined #openstack-nova20:34
*** ociuhandu has quit IRC20:43
openstackgerritLance Bragstad proposed openstack/placement master: Bump oslo.log version to 4.3.0  https://review.opendev.org/c/openstack/placement/+/76022920:53
*** ociuhandu has joined #openstack-nova20:56
*** ociuhandu has quit IRC21:03
*** jmlowe has quit IRC21:33
*** ociuhandu has joined #openstack-nova21:36
*** ociuhandu has quit IRC21:40
*** ociuhandu has joined #openstack-nova21:44
*** ociuhandu has quit IRC21:51
*** ociuhandu has joined #openstack-nova21:54
*** ociuhandu has quit IRC21:58
*** ociuhandu has joined #openstack-nova22:10
gmannstephenfin: done, left comment for https://review.opendev.org/c/openstack/nova/+/769520/1/nova/api/openstack/compute/hypervisors.py#36322:15
gmannstephenfin: and other two patches of this BP also22:15
*** nweinber has quit IRC22:17
*** ociuhandu has quit IRC22:19
*** rcernin has joined #openstack-nova22:27
*** alexe9191 has quit IRC22:42
*** bnemec has quit IRC22:45
*** zzzeek has quit IRC22:53
*** zzzeek has joined #openstack-nova22:55
*** bnemec has joined #openstack-nova23:01
*** zzzeek has quit IRC23:08
*** eharney has quit IRC23:09
*** zzzeek has joined #openstack-nova23:09
*** ociuhandu has joined #openstack-nova23:25
*** ociuhandu has quit IRC23:30
*** tosky has quit IRC23:51
« Tuesday, 2021-01-05 Index Thursday, 2021-01-07 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!