Friday, 2020-10-23

« Thursday, 2020-10-22 Index Saturday, 2020-10-24 »
*** martinkennelly has quit IRC00:25
*** swp20 has quit IRC00:28
*** macz_ has joined #openstack-nova00:37
*** macz_ has quit IRC00:42
*** gyee has quit IRC00:44
*** dklyle has quit IRC01:29
*** dklyle has joined #openstack-nova01:38
*** jawad_axd has joined #openstack-nova01:54
*** jawad_axd has quit IRC01:58
*** swp20 has joined #openstack-nova02:01
*** aarents has quit IRC02:07
*** brinzhang has joined #openstack-nova02:09
*** aarents has joined #openstack-nova02:14
*** jawad_axd has joined #openstack-nova02:15
*** jawad_axd has quit IRC02:20
*** dklyle has quit IRC02:22
*** jawad_axd has joined #openstack-nova02:35
*** jawad_axd has quit IRC02:40
*** hamalq_ has quit IRC02:45
*** rcernin has quit IRC02:51
*** jawad_axd has joined #openstack-nova02:56
*** jawad_axd has quit IRC03:01
*** jawad_axd has joined #openstack-nova03:17
*** rcernin has joined #openstack-nova03:19
*** rcernin has quit IRC03:19
*** rcernin has joined #openstack-nova03:19
*** jawad_axd has quit IRC03:21
*** psachin has joined #openstack-nova03:33
*** jawad_axd has joined #openstack-nova03:38
*** jawad_axd has quit IRC03:42
*** jawad_axd has joined #openstack-nova03:58
*** jawad_axd has quit IRC04:03
*** swp20 has quit IRC04:06
openstackgerritmelanie witt proposed openstack/nova master: WIP Omit resource inventories from placement update if zero  https://review.opendev.org/75934804:09
*** evrardjp has quit IRC04:34
*** evrardjp has joined #openstack-nova04:34
*** swp20 has joined #openstack-nova04:47
*** ratailor has joined #openstack-nova04:51
*** jawad_axd has joined #openstack-nova05:00
*** jawad_axd has quit IRC05:05
*** bhagyashris is now known as bhagyashris|sick05:09
*** sapd1 has joined #openstack-nova05:17
*** sapd1 has quit IRC05:29
*** vishalmanchanda has joined #openstack-nova05:57
*** rpittau|afk is now known as rpittau06:21
*** mvorwerk has joined #openstack-nova06:35
*** jawad_axd has joined #openstack-nova06:40
*** mvorwerk has quit IRC06:42
*** slaweq has joined #openstack-nova06:52
*** tesseract has joined #openstack-nova06:56
*** sapd1 has joined #openstack-nova07:01
*** ralonsoh has joined #openstack-nova07:02
*** hoonetorg has quit IRC07:07
*** psachin has quit IRC07:08
*** psachin has joined #openstack-nova07:11
*** andrewbonney has joined #openstack-nova07:13
*** hoonetorg has joined #openstack-nova07:19
*** tesseract has quit IRC07:25
*** xek has joined #openstack-nova07:26
*** tesseract has joined #openstack-nova07:28
*** tosky has joined #openstack-nova07:36
openstackgerritJeffrey Zhang proposed openstack/nova master: Support inject-nmi action in watchdog  https://review.opendev.org/74107207:42
*** sapd1 has quit IRC08:19
*** sapd1 has joined #openstack-nova08:20
*** xek has quit IRC08:24
*** martinkennelly has joined #openstack-nova08:25
*** k_mouza has joined #openstack-nova08:31
openstackgerritJorhson Deng proposed openstack/nova stable/rocky: optimize the shelve code flow  https://review.opendev.org/75937308:37
*** derekh has joined #openstack-nova08:37
*** jawad_axd has quit IRC08:37
*** LinPeiWen has quit IRC08:38
*** jawad_axd has joined #openstack-nova08:38
openstackgerritJorhson Deng proposed openstack/nova stable/ussuri: optimize the shelve code flow  https://review.opendev.org/75937408:38
*** LinPeiWen has joined #openstack-nova08:38
openstackgerritJorhson Deng proposed openstack/nova stable/train: optimize the shelve code flow  https://review.opendev.org/75937508:39
openstackgerritJorhson Deng proposed openstack/nova stable/stein: optimize the shelve code flow  https://review.opendev.org/75937608:39
openstackgerritJorhson Deng proposed openstack/nova stable/queens: optimize the shelve code flow  https://review.opendev.org/75937708:41
openstackgerritJorhson Deng proposed openstack/nova stable/pike: optimize the shelve code flow  https://review.opendev.org/75937808:41
openstackgerritJorhson Deng proposed openstack/nova stable/ocata: optimize the shelve code flow  https://review.opendev.org/75937908:41
*** salmankhan has joined #openstack-nova08:43
lyarwood^ needs to start from stable/victoria btw, I've updated the stable/ussuri commit if you could take a look.08:44
*** deke997 has joined #openstack-nova08:49
*** ratailor has quit IRC08:49
deke997hey guys08:49
deke997I'm trying to get some updated info on multi-function pci passthrough08:49
deke997last I checked it was not really supported08:50
deke997anyone know more on this?08:50
*** slaweq has quit IRC08:52
*** slaweq has joined #openstack-nova08:52
*** jawad_ax_ has joined #openstack-nova09:13
*** salmankhan1 has joined #openstack-nova09:13
*** salmankhan has quit IRC09:15
*** salmankhan1 is now known as salmankhan09:15
*** jawad_axd has quit IRC09:15
*** ociuhandu has joined #openstack-nova09:19
*** xek has joined #openstack-nova09:29
*** sapd1 has quit IRC09:30
*** deke997 has quit IRC09:45
*** brinzhang has quit IRC09:53
*** swp20 has quit IRC09:59
*** ociuhandu has quit IRC10:08
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Liberty database migrations  https://review.opendev.org/75839710:13
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Mitaka database migrations  https://review.opendev.org/75839810:13
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Newton database migrations  https://review.opendev.org/75908510:13
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Ocata database migrations  https://review.opendev.org/75908610:13
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Pike database migrations  https://review.opendev.org/75908710:13
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Queens database migrations  https://review.opendev.org/75908810:13
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Rocky database migrations  https://review.opendev.org/75908910:13
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Stein database migrations  https://review.opendev.org/75909010:13
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Train database migrations  https://review.opendev.org/75909110:13
*** k_mouza has quit IRC10:23
*** k_mouza has joined #openstack-nova10:23
*** recyclehero has joined #openstack-nova10:32
recycleherohey guys, I was referred here from my deploment method team OSA10:32
recycleherothere is a problem with cpu allocation ratio10:33
recycleheroat first as OSA was templating xxx_allocation_ratio it was normal that subsequent changes to aloocation ratio via palcement api get discarded10:34
recycleherobut we did this patch10:34
recycleherohttps://review.opendev.org/#/c/758029/10:35
recycleheroso we let xxx_allocation_ratios to be None10:36
recycleherothen I have set inital_xxx_allocation_ratio to a floating number and did a clean deploy10:36
recyclehero1- in horizon I am seeing again the actual PCPU instead of VCPU. some people say this is the way it is and I am okay with that10:37
*** sapd1 has joined #openstack-nova10:38
recyclehero2- using openstack resource cli I see the allocation ratio for cpu is equal to number tht I have set for initial_cpu_allocation_ratio10:38
recycleherobut the problem is with *total*10:38
recycleherototal is eqaul to 1*PCPU10:38
*** dtantsur|afk is now known as dtantsur10:39
recyclehero3-I tried palcement api to put the inventory for the resource provider. its sucsess as I check just after with openstack resource provider inventory show. resource_provider_generation is also ++. then in a minute it gets overwritten and another generation++10:41
recycleherocurl -XPUT -H 'X-Auth-Token: gAAAAABfkgravKJ1xAyjbHjY3hTy4wOMfufVKfafIIb5KUqcJufJ82mEIorUC_FwnOgHWX4uLDr_W22PHc28JouKhIknGkSJyP0dAPYa3DwFNA5dRgugMi9DcycwUPqLYoBQ3h6zv8M-VWCFFEJAuc-9ggV5ZTfkkB_vvXYDdd5dc765WhLZr2s' -H "Content-type: application/json" -d '{"inventories": {"MEMORY_MB": {"allocation_ratio": 2.0,"reserved": 1024,"total": 24014},"VCPU": {"allocation_ratio": 16.0,"reserved":10:41
recyclehero0,"total": 128}},"resource_provider_generation": 4}' 'http://172.29.236.11:8780/resource_providers/c8ac01e1-0b52-41f0-bd52-484070ea3aa9/inventories'10:41
*** k_mouza has quit IRC10:43
*** sapd1 has quit IRC10:45
*** sapd1 has joined #openstack-nova10:51
*** tbachman has quit IRC10:55
*** sapd1 has quit IRC11:00
stephenfinrecyclehero: 'cpu_allocation_ratio' applies only to VCPU; you can't overcommit PCPU11:07
stephenfinAssuming you're talking about the different resource classes?11:10
stephenfinTo be clear, VCPU and PCPU are resource classes, along with DISK_GB, MEMORY_MB etc. Maybe you meant "total is equal to 1 * host CPUs?"11:11
stephenfinrecyclehero: the way that's supposed to work is that 'initial_*_allocation_ratio' will be used to set the initial ratios, allowing a user to overrides them manually using e.g. osc-placement. By comparison, '*_allocation_ratio' (no 'initial_' prefix) will always override whatever the user sets11:12
*** k_mouza has joined #openstack-nova11:13
stephenfinrecyclehero: e.g. https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.cpu_allocation_ratio11:14
*** k_mouza has quit IRC11:17
*** k_mouza has joined #openstack-nova11:17
*** xek has quit IRC11:22
*** takamatsu has quit IRC11:28
*** takamatsu has joined #openstack-nova11:32
stephenfingibi: https://review.opendev.org/#/c/759114/ failed but it doesn't seem related; I guess we're good to remove that?11:34
*** takamatsu has quit IRC11:34
stephenfinassuming so, could you revisit https://review.opendev.org/#/c/749312/ and we can address the comments in a follow-up?11:35
stephenfin*the follow-up11:35
*** takamatsu has joined #openstack-nova11:35
*** xek has joined #openstack-nova11:35
openstackgerritStephen Finucane proposed openstack/nova master: api-ref: Move 'os-agents' API to obsolete section  https://review.opendev.org/75572911:40
stephenfinlyarwood: Any chance you could close out the last two xenapi changes? https://review.opendev.org/#/c/749315/ https://review.opendev.org/#/c/755729/11:40
stephenfinThere's one more, but I think I've been too eager in removing stuff with it https://review.opendev.org/#/c/749316/11:41
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Liberty database migrations  https://review.opendev.org/75939911:42
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Mitaka database migrations  https://review.opendev.org/75940011:42
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Newton database migrations  https://review.opendev.org/75940111:42
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations  https://review.opendev.org/75940211:42
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Pike database migrations  https://review.opendev.org/75940311:42
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Queens database migrations  https://review.opendev.org/75940411:42
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Rocky database migrations  https://review.opendev.org/75940511:42
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Stein database migrations  https://review.opendev.org/75940611:42
lyarwoodstephenfin: ack will try today, trying to finish some PoC stuff for the PTG next week11:47
*** tesseract has quit IRC11:52
*** xek has quit IRC11:54
*** xek has joined #openstack-nova11:55
*** xek has quit IRC12:00
*** xek has joined #openstack-nova12:01
*** xek has quit IRC12:03
*** xek has joined #openstack-nova12:03
*** tesseract has joined #openstack-nova12:04
*** tesseract has quit IRC12:04
*** xek has quit IRC12:04
*** xek has joined #openstack-nova12:05
*** tesseract has joined #openstack-nova12:09
*** tesseract has quit IRC12:09
*** xek has quit IRC12:11
*** tesseract has joined #openstack-nova12:12
*** nightmare_unreal has joined #openstack-nova12:21
*** swp20 has joined #openstack-nova12:23
*** songwenping_ has joined #openstack-nova12:27
*** tbachman has joined #openstack-nova12:27
*** swp20 has quit IRC12:29
recycleherostephenfin: its obvious one could not overcommit PCPUs.though I have problems with overcommiting vcpus12:30
recycleheroand yes total is 1 * host cpus12:31
recycleherowe have not defined cpu_allocation_ration which is equall to it being None.12:31
recycleheroand set initial_cpu_allocation_ratio: 1612:32
recycleheroI even used the api as I mentioned above.12:32
recycleheroto be clear what is the value that is shown in horizon hypervisor dashboard?12:32
sean-k-mooneyrecyclehero:are you using any falvor extra spec12:34
sean-k-mooneyrecyclehero: also people have proposed overcommiting PCPUs in the past but we have rejected it so its obvious but not to eveyone :)12:35
recycleheroi have 1000 cpus, deal with it nova! :D12:36
sean-k-mooney:)12:36
recycleheroI didnt understnd the flavor part? how is it related with me not having allocation_ratio*PCPU=VCPUs?12:37
sean-k-mooneyi was wondering was this related to schduling but your actully asking about horizon right12:37
sean-k-mooneyhorizon pull its infor from the hypervios api12:37
recycleheroaha so if I even be successful with overcommiting via placement I dont get to see it in horizon12:38
*** k_mouza has quit IRC12:39
sean-k-mooneyhttps://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-details-detail,show-hypervisor-statistics-detail#list-hypervisors-details12:39
sean-k-mooneyso it s either hitting the details endpoint or the stats endpoint12:40
sean-k-mooneyin both cases it just reports the vcpus i belive12:40
sean-k-mooneythat value is not multiplied by the allocation ratio12:41
recycleheroso its actually PCPUs right?12:41
sean-k-mooneywell no PCPUs are a different thing12:41
recycleheroThe number of vcpu in this hypervisor. This does not take allocation ratios used for overcommit into account so there may be disparity between this and the used count.12:41
recyclehero"VCPU = PCPU - reserverd" or am I completely wrong12:42
sean-k-mooneyit either  the number of cpus on the host or it the number of cpu in the host defined by the vcpu_pin_set or cpu_shared_set12:42
sean-k-mooneyrecyclehero: its not calulated that way12:42
sean-k-mooneyif you use cpu_dedicated_set and dont defien cpu_shared_set then vcpu=012:43
sean-k-mooneyif you use cpu_shared_set or vcpu_pin_set and dont use cpu_dedicated_set then pcpus are 012:44
*** xek has joined #openstack-nova12:45
recycleherosean-k-mooney: aha, but actually I dont care what I see in horzion. I have 8 cores which I am seeing in horizon. I need 128 cores ready for allocation.12:45
recycleherohttps://docs.openstack.org/placement/latest/user/index.html12:46
sean-k-mooneyrecyclehero: stephenfin has a short blog on the topic https://that.guru/blog/cpu-resources-redux/12:46
sean-k-mooneyrecyclehero: by the way we normally don suggest setting allocation raitos over 412:46
sean-k-mooneyim planning to push a patch to change the inital allcoation ration to 4 next week12:46
sean-k-mooneyill be bringin it up at the ptg12:47
sean-k-mooneybut back to your issue12:47
sean-k-mooneycan you check what is set in placment12:47
recycleherook 4 will even do. but I will be rather be typing virt command with allocation ratio 112:47
recycleherohow? api cli?12:47
sean-k-mooneyyou can do it either way12:48
sean-k-mooneythere is an osc-placment plugin for the openstack client12:48
sean-k-mooneyso you can pip install that12:48
sean-k-mooneyopenstack --os-placement-api-version 1.18 resource provider list12:48
recycleheroopenstack resource provider inventory show c8ac01e1-0b52-41f0-bd52-484070ea3aa9 VCPU12:49
sean-k-mooneythen openstack --os-placement-api-version 1.18 resource provider inventory show <uuid of host> VCPU12:49
recyclehero allocation_ratio | 16.012:49
recycleherototal            | 812:49
sean-k-mooneythere you go12:49
noonedeadpunksean-k-mooney: oh, btw, if we're talking about this:) Is usage of initial_*_allocation_ratio can be considered as unified and recommended approach instead of *_allocation_ratio?12:49
sean-k-mooneynoonedeadpunk: it depends on if you want to manage it via config or api12:50
sean-k-mooneynoonedeadpunk: some operators like to commit there configuration to git so want to use *allocation_ratio12:50
sean-k-mooneyother want to manage it via the api so shoudl use initail_*12:51
sean-k-mooneyto set the defualt12:51
sean-k-mooneythat basically how you chose between them12:51
noonedeadpunksean-k-mooney: just of the docs it seems that the idea of initial was thet it can be stored in git but can be overriden in api?12:51
noonedeadpunkah, I guess I see what you mean...12:51
sean-k-mooneynoonedeadpunk: yes it will set the inital value in the config and then over write it12:52
recycleheronoonedeadpunk: https://review.opendev.org/#/c/758029/ we have takeled with jrosser and came to this. there are 2 refrence to related documentaion part.12:52
sean-k-mooneyinital is only used when creating an inventory for the first time12:52
recycleherohttps://docs.openstack.org/nova/ussuri/admin/configuration/schedulers.html#usage-scenarios12:52
noonedeadpunksean-k-mooney: aha, ok, I missed that part I guess12:52
sean-k-mooney*_allocation_ration take efect every time the resouce tracker periodic task runs12:52
noonedeadpunkso like if initial_ is overriden during runtime, it won't be really apllied12:53
recycleheroif resource_tracker.py dont found the host it will set its inital flag to true12:53
noonedeadpunkwell, not during runtime, but after host is discovered12:53
sean-k-mooneyya if you calll the api to change a value on the inventory then inital will not have any effect on that12:54
sean-k-mooneyinital is just used when creating an inventory and never after that12:55
noonedeadpunksean-k-mooney: ok, cool, thanks!12:55
recycleherosean-k-mooney: will u be in half an hour? I feel you are the one who save me from this misery.12:55
recycleherohere12:55
sean-k-mooneyill be here yes12:55
recycleherogreat thanks12:55
noonedeadpunkrecyclehero: yeah, it was exactly about that patch as I was thinking to just replace cpu_allocation_ratio with initial_cpu_allocation_ratio but it doesn't make sense for this scenario12:56
*** Diavel has joined #openstack-nova12:56
sean-k-mooneyrecyclehero: this is the spec by the way that added inital_* https://specs.openstack.org/openstack/nova-specs/specs/stein/implemented/initial-allocation-ratios.html12:57
sean-k-mooneynoonedeadpunk: idealy you would support both12:57
sean-k-mooneyalthough only one of the two ways is acutlly used in any one deployment12:57
DiavelHi guys, currently working on an app which consumes rabbitmq messages from nova-compute and I am wondering - Is there a reason why there is no notification sent on `UpdateMetadata` of an instance?12:58
Diavelthere are for imo every other operation you can run from horizon, but there is no for metadata update12:59
noonedeadpunksean-k-mooney: well, deployers are able to set overrides and define initial ones. and this patch will remove regular ones. so kind of both supported but not in very straightforward way....12:59
sean-k-mooneynoonedeadpunk: ah i see well that makes sense i guess13:00
sean-k-mooneynoonedeadpunk: i still prefer how kolla ansible does it with config overrides rahter then provideing wrappers around things13:00
noonedeadpunksean-k-mooney: we encourage to use config overrides https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/defaults/main.yml#L50113:01
sean-k-mooneyDiavel: when you ser updateMetadata what do you mean13:01
noonedeadpunkbut I mean we should have some defaults set?13:01
sean-k-mooneyDiavel: is it the server proerties13:01
sean-k-mooneyDiavel: or image/flaovr metadata13:01
sean-k-mooneynoonedeadpunk: oh you have both13:02
noonedeadpunkyep :p13:02
sean-k-mooneyya its better in the long run i think since the user become more familar with the upstream project names and there for the release note make more sense to them13:02
noonedeadpunkand these vars can be overriden with config_overrides so overrides have prescedence13:02
Diavelsean-k-mooney: it's the server metadata, not flavour13:04
sean-k-mooneyDiavel: are you refering to this metadata https://github.com/openstack/nova/blob/master/nova/notifications/objects/instance.py#L5813:04
sean-k-mooneyah ok13:04
sean-k-mooneywell that is updated when you do an openstack server set --properties13:04
sean-k-mooneywhich is not an instance action13:04
sean-k-mooneyit might be part of https://github.com/openstack/nova/blob/master/nova/notifications/objects/instance.py#L28713:05
sean-k-mooneywe dont have the filed ther ebut that might be triggered form the openstack server set13:05
noonedeadpunkbtw, need to remove *_allocation_ratio from config (so that value would be None) is not super convinient from our prespective:))) being able to set it as just empty string to make it not used is more neat and I guess from python prespective is not harder in implementation?13:06
sean-k-mooneyDiavel: wa it think its the update https://github.com/openstack/nova/blob/master/nova/notifications/objects/instance.py#L590-L59813:06
noonedeadpunkas `if str()` and `if None` are both false....13:07
noonedeadpunkbut whatever13:07
sean-k-mooneyDiavel: so i think you just need to addd it to the payload and extend https://github.com/openstack/nova/blob/6a5e158756edc6c01bb23c26dcfc4c3a3df703dc/nova/notifications/base.py#L170-L20713:08
noonedeadpunkah, well, I guess it might be limitation of oslo.config... as it should be int, so can't be just empty13:08
sean-k-mooneynoonedeadpunk: i think you can set it to "cpu_allocation_ratio="13:09
*** jawad_ax_ has quit IRC13:10
noonedeadpunkoh, really? I thought we can't....13:10
Diavelsean-k-mooney: thanks a lot ;) will take a look at that part13:10
sean-k-mooneynoonedeadpunk: you can use 0.0 https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L420-L45313:10
sean-k-mooneynoonedeadpunk: but i think y ou can also set it to None13:11
*** jawad_axd has joined #openstack-nova13:11
noonedeadpunki thought that None in config is not None, but str('None')?13:12
noonedeadpunkbut yeah13:12
*** jawad_axd has quit IRC13:12
sean-k-mooneynot it None unquoted i think13:12
noonedeadpunk0.0 would be an option out of the box:)13:12
sean-k-mooneyanyway 0.0 shoudl work13:12
*** jawad_axd has joined #openstack-nova13:12
sean-k-mooneyand its still the same type13:13
noonedeadpunkyeah13:13
*** jawad_axd has quit IRC13:13
sean-k-mooneyare you thinking of just templating the value13:13
*** jawad_axd has joined #openstack-nova13:13
noonedeadpunkwell, we do it now13:14
noonedeadpunkand why I started asking question was that change https://review.opendev.org/#/c/758029/2/templates/nova.conf.j213:15
*** vishalmanchanda has quit IRC13:16
noonedeadpunkanyway thanks a lot for helping sean-k-mooney!13:18
sean-k-mooneyno worries13:27
*** jangutter_ has joined #openstack-nova13:27
*** jangutter has quit IRC13:28
*** priteau has quit IRC13:28
*** k_mouza has joined #openstack-nova13:29
Diavelsean-k-mooney: if I am not mistaken the metadata should already be included in the payload.. https://github.com/openstack/nova/blob/6a5e158756edc6c01bb23c26dcfc4c3a3df703dc/nova/notifications/base.py#L182-L183 -- https://github.com/openstack/nova/blob/6a5e158756edc6c01bb23c26dcfc4c3a3df703dc/nova/notifications/base.py#L46713:31
sean-k-mooney in the unversion one perhaps13:32
sean-k-mooneybut its not defiend in the versiond payload13:32
sean-k-mooneyhttps://github.com/openstack/nova/blob/6a5e158756edc6c01bb23c26dcfc4c3a3df703dc/nova/notifications/base.py#L211-L25213:33
sean-k-mooneythat never read it and the object does nto hold that value https://github.com/openstack/nova/blob/6a5e158756edc6c01bb23c26dcfc4c3a3df703dc/nova/notifications/objects/instance.py#L287-L31713:33
sean-k-mooneyDiavel: so if you are using the versioned notificaiton which is what you should be using you wont get it13:34
sean-k-mooneyit can be added but it cannot be bakcported13:34
recycleheronoonedeadpunk, sean-k-mooney : I set inital_cpu_allocation_ratio via this config overrides.13:35
recycleherosean-k-mooney: I read the spec but dont know why its still overriding my config13:35
sean-k-mooneyDiavel: so you will need to use the unversion notification and then add it to the versioned notificaton for wallaby13:35
recycleherobecause I have set it None, I mean they are not there. thats None right?13:36
Diavelsean-k-mooney: got it.. shit I missed that, sorry and thanks again13:36
Diavelsean-k-mooney: I am using the versioned one13:36
sean-k-mooneyDiavel: no worries we can totally add it but they are versioned so we cant backprot it13:37
sean-k-mooneyDiavel: it would be good to add it so that you dont have to use the unversioned ones13:37
Diavelsean-k-mooney: yeah I totally agree13:37
sean-k-mooneyrecyclehero: initial is only used when a node if first deployed13:38
sean-k-mooneysince it has a default setting it to none will result in it still using the default13:38
recycleherosean-k-mooney: I didnt completly get the second one. I did the patch which I sent. so cpu_allocation_ratio is not present in nova.conf. also I ahve set the inital_cpu_allocation_ratio:16 which is present in nova.conf13:41
recycleheroit was a clean deploy13:41
recycleheroand I think it did use the iniita ratio because it is set to 1613:41
recycleherothe problem lies it total I guess13:41
recycleheroyou were asking me before to do some tests.13:41
sean-k-mooneywell total shoudl be equal to the number of cores13:41
sean-k-mooneynot 16*number of cores13:42
sean-k-mooneyand horizon  should not change13:42
recycleherooh so It means I have it set now. let me check lunching a 10 core instance13:42
sean-k-mooneyyou cant13:42
sean-k-mooneyyou have only 8 cores on the host13:43
recycleherobut I want to overcommit :((13:43
sean-k-mooneyyou cant oversubseribe against your self13:43
sean-k-mooneylaunch 2 5 core ones13:43
recycleheroaha ok13:43
openstackgerritStephen Finucane proposed openstack/nova master: virt: Remove 'change_instance_metadata' API  https://review.opendev.org/74931613:49
*** k_mouza has quit IRC13:51
*** nweinber has joined #openstack-nova13:56
*** nweinber has quit IRC13:59
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Liberty database migrations  https://review.opendev.org/75839713:59
*** nweinber has joined #openstack-nova13:59
*** jawad_axd has quit IRC14:00
*** nweinber has quit IRC14:02
*** k_mouza has joined #openstack-nova14:03
*** nweinber has joined #openstack-nova14:03
*** tesseract has quit IRC14:05
*** rpittau is now known as rpittau|afk14:06
*** tesseract has joined #openstack-nova14:06
*** tesseract has quit IRC14:12
*** tesseract has joined #openstack-nova14:13
*** k_mouza has quit IRC14:14
stephenfinthis looks new :-\ https://zuul.opendev.org/t/openstack/build/c85b0734fb524eefb67449105528b0a2/log/logs/screen-n-cpu.txt#86114:14
recycleherostephenfin:  I am seeing 10 VCPU on my project so I guess it is ok. thank you14:19
*** jawad_axd has joined #openstack-nova14:20
recycleherobut on hoizon in hypervisor VCPU usage is 10 of 814:20
recycleherostephenfin: before getting of your shoulder, what is the logic that one cat overcommit self?14:20
recycleherocant14:20
sean-k-mooneyrecyclehero: 10 of 8 is correct14:20
sean-k-mooneythat is what should be in horizon14:21
sean-k-mooneystephenfin: nope14:21
sean-k-mooneystephenfin: we have seen that downstream before14:21
sean-k-mooneystephenfin: we belive it happens due to out of memory14:21
recycleherosean-k-mooney: I mean why is that that I cant have a single x>8 cores instance?14:22
recycleherostephenfin: wrong mention14:22
*** k_mouza has joined #openstack-nova14:23
stephenfinrecyclehero: PCPU is how we track pinned CPUs. Pinned CPUs means each instance CPU is assigned its own host CPU14:23
stephenfinSo no overcommit. If an instance is using a host CPU, no other instance can use it14:23
sean-k-mooneywell for non pcpus its also not required14:24
sean-k-mooney*alllow14:24
stephenfinrecyclehero: As for the 10 of 8, Horizon is probably retrieving stats from nova's 'os-hypervisors' API which is notoriously broken14:24
sean-k-mooneywe do not allow 2 guest cpus to use the same host cpu14:24
sean-k-mooneystephenfin: horizon is doing the right thing14:24
sean-k-mooneyits ment to report how many cpus are avaiable and it doe suse os hyperviros14:24
sean-k-mooneyit is not ment to the cpus multiplied by the  allocation ration14:25
stephenfinThat's wrong14:25
sean-k-mooneyno that is how it was ment to work14:25
sean-k-mooneyand how it has always worked14:25
sean-k-mooneyit has never used the allocation ration14:25
stephenfinCreate a load of guests with 'hw:numa_nodes=1' on two node host with 6 CPUs per node14:25
stephenfinHow many host CPUs are free if you boot those guests?14:26
stephenfinThe os-hypervisors information makes no sense in a world of overcommit14:26
stephenfinwhich is why we're planning to drop it in Wallaby14:26
sean-k-mooneystephenfin: recyclehero  is try ing to boot a 10 vcpu guest on an 8 cpu host14:26
sean-k-mooneywhich we have never supported14:27
sean-k-mooneybecause 2 gust cpus cant consome the same host cpu enven if they are floating14:27
stephenfinsean-k-mooney: yes, that's true14:27
stephenfinsean-k-mooney: but we have 'max_unit' to track that14:28
stephenfinrecyclehero: Oh, that was an interesting point on the output of e.g. 'openstack resource provider inventory list'14:29
stephenfinI'd have expected the total column to show the total inclusive of allocation_ratio14:29
stephenfinbut it doesn't; it shows the total before the allocation ratio is applied14:29
sean-k-mooneyno14:29
stephenfinso you have to do that maths yourself :)14:29
sean-k-mooneyyes14:29
stephenfinsean-k-mooney: make up your mind, man!14:30
stephenfin:P14:30
sean-k-mooneyso that the different values are not depenent on each other14:30
sean-k-mooneyyou have to do the math yourself14:30
sean-k-mooneymultipleying total by allocation ratio - reserved14:30
*** dklyle has joined #openstack-nova14:31
stephenfinsean-k-mooney: as for that earlier issue, this entire stack is failing with it :( https://review.opendev.org/#/q/topic:bp/compact-db-migrations-wallaby14:32
stephenfinI don't think it's something I've done wrong, but I'm waiting to see if another patch fails first14:32
sean-k-mooneystephenfin: the import issue if its the same one it due to an oom issue causeing the import to fail14:32
sean-k-mooneyif not then rbd is not isntalled14:33
*** tesseract has quit IRC14:33
sean-k-mooneybecause ceph-common is not installed14:33
stephenfinoh, no, look down14:33
*** tesseract has joined #openstack-nova14:33
sean-k-mooneyi didnt look too closelon14:33
stephenfinignore the rbd thing14:33
stephenfinthat's lyarwood's work14:33
sean-k-mooneyoh line 86114:34
sean-k-mooneyit did not scole down fro some reason14:35
sean-k-mooneyTypeError: Parameterized generics cannot be used with class or instance checks14:35
stephenfinYeah14:35
stephenfinI've never seen that before14:35
stephenfinit's buried in eventlet too14:35
sean-k-mooneyso its coming from the ytpeing model14:36
stephenfinmypy hasn't been updated recently though, and typing is stdlib now14:37
sean-k-mooneyform the libvirt treadpool proxy14:37
sean-k-mooneystephenfin: yes it si from the standardlib14:37
stephenfincrap, new eventlet release yesterday14:37
sean-k-mooneyto fix dnspython support?14:38
sean-k-mooneywe likely need to cap it untill we figure this out14:38
stephenfingibi: Looks like we might have a problem with the new release of eventlet ^14:39
stephenfinhttps://zuul.opendev.org/t/openstack/build/c85b0734fb524eefb67449105528b0a2/log/logs/screen-n-cpu.txt#86114:39
sean-k-mooneyhttps://github.com/eventlet/eventlet/commit/9d731373e2325588a33229b425100ad538eed76614:40
sean-k-mooneythis is apparently what changed14:40
sean-k-mooneyoh they fixed my ssl bug i think https://github.com/eventlet/eventlet/commit/7b1aa58d77550caabba3ca2e0b1bb79626c83d9c14:42
sean-k-mooneyoh no its a different one14:42
sean-k-mooneyhttps://github.com/eventlet/eventlet/issues/632 was our issue14:43
stephenfinyeah, the last working build for https://review.opendev.org/#/c/758396/ used eventlet===0.26.114:44
stephenfinhttps://zuul.opendev.org/t/openstack/build/bf1f2b730848408b99c61e82068eb73b/log/job-output.txt#635514:44
stephenfinthe failing one uses eventlet===0.28.014:44
stephenfinhttps://zuul.opendev.org/t/openstack/build/00e3a2793a1040b29b944b944c9da795/log/job-output.txt14:44
stephenfinso eventlet has borked us \o/14:44
sean-k-mooneythey have done a buch of release in the last month14:45
sean-k-mooneyi guess need to file a bug14:46
*** Diavel has quit IRC14:52
stephenfinHmm, we also bumped our version of mypy, greenlet and libvirt-python recently, so it could be those too. This will be fun14:52
*** songwenping_ has quit IRC15:07
*** macz_ has joined #openstack-nova15:10
*** takamatsu has quit IRC15:15
*** ralonsoh has quit IRC15:20
*** ralonsoh has joined #openstack-nova15:20
recycleherosean-k-mooney stephenfin thanks guys. I really appreciate ur help. I guess it was okay after we did the 'None' patch in OSA but the data displayed was confusing.15:24
sean-k-mooneyit was a design choice that they made very early on in horizon15:24
sean-k-mooneythey wanted ti to deisplay your actual resouce avaiablyt not your over commit availablity15:25
sean-k-mooneyim sure it could be changed but  it would be a change in behavior of what that dashboard shows15:25
recycleherobut anyway IMO 10 out of 8 is not a good thing to show especially in red. its like ur doing something wrong15:26
sean-k-mooneyits in red because you are over commiting15:27
sean-k-mooneywhich in maycases is wrong15:27
sean-k-mooneyfor disk or memory its potentially dangrous15:27
sean-k-mooneydisk can cause data loss and vm hangs15:27
sean-k-mooneymemory without swap can cause the vms to be killed15:28
sean-k-mooneycpu if its too high will also eventually cause nova to stop working properly due to to high load15:28
recycleheroBTW I am using kernel shared memory + lots of swap. i hope it works well15:29
sean-k-mooneyif you are using non numa guest then it can15:29
recycleherowith this and using same distros for instances I hope I can get near 2.0 for memory15:30
sean-k-mooneyrecyclehero: you need to have enough swap to fully account for the over commit15:30
recycleherolots of swap ssd15:30
sean-k-mooneyso you have allcoated at least 2 times your memroy for swap15:30
recycleheroequal memory wont work. I have 12GB Ram + 12 GB swap. you are suggesting 24GB swap for allocation ratio of 2.0?15:32
sean-k-mooneythe guest uses more memory then in the falvor for the qemu process to use and back device es with15:32
sean-k-mooneyrecyclehero: yes15:32
sean-k-mooneyyou shoudl have 24G15:32
sean-k-mooneyof swap15:32
sean-k-mooneyyou could go a little lowwer but you wotn get to 2.0 with only 12G15:32
sean-k-mooneyyou will hit oom issues15:33
sean-k-mooneyyou have not accounted for the qemu overhead or nova/os memory usage15:33
recycleherogood to know. I should I make changes while its a fresh deployement.15:33
recycleheroI have counted nova/os memory in the reserverd.15:34
recycleherobut not for the qemu overhead15:34
sean-k-mooneythe qemu over head can be small or lager depening on how my cores or devices an instance has15:34
sean-k-mooneyit can be as small as ~100MB or well over a GB15:35
sean-k-mooneyper instance15:35
sean-k-mooneyso you need to caulate what the maxium amount of overhead will be for you smallest and larges flavor by booting them then figure out how many of them will run on each host and set teh reserved value accourdignly15:36
sean-k-mooneyor allocate enough swap to copenstate15:36
recycleheromy strategy is reserved for nova+os and swap to conpensate for overcommiting15:39
sean-k-mooneyya but again that is ignoring qemu overhead15:40
sean-k-mooneywhich is ment to be acounted for in reserved or swap15:40
sean-k-mooneyin generall i advise never overcommiting memroy or disks for what its worth15:40
sean-k-mooneyonly cpus15:40
sean-k-mooneyand even then only cpus at a max of 4.015:41
sean-k-mooneyif you want to hevialy over commit memory you might find https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/libvirt-file-backed-memory.html intersting too15:42
sean-k-mooneyits uses files on disk that are cached in ram instead of relying on swap15:42
recycleherothanks sean-k-mooney for sharing ur xp15:45
*** gyee has joined #openstack-nova15:48
sean-k-mooneywe get enough of bug reported by customer related to OOM that i always advise people against it unless the really really know what they are doing15:49
sean-k-mooneypeople coming form vmware often dont know that vmware always uses file backed memroy wehn you are over commiting behind the scenes15:50
*** takamatsu has joined #openstack-nova15:53
openstackgerritStephen Finucane proposed openstack/nova master: docs: Make JSON valid  https://review.opendev.org/75946615:58
stephenfinmelwitt: lyarwood: That will fix our currently broken docs build ^15:58
stephenfinI don't know what changed :( Neither Sphinx nor Pygments have seen releases recently :-\15:59
stephenfinand that doc hasn't been changed15:59
melwitttbh I don't like the sound of "don't know what changed" :/16:00
stephenfinneither do I, but it's the best I can offer atm :)16:01
melwittalso I don't understand how the change fixes it, what is it doing, shorten the string?16:01
stephenfinno, what's there isn't valid JSON16:02
stephenfinyou can't escape newlines with backslash16:02
melwittohh16:02
stephenfinso the parser is rightly complaining16:02
melwittI see16:02
stephenfinI just don't know why it's suddenly doing it now16:02
melwittso something started validating it16:02
melwittyeah16:03
stephenfinbut Sphinx has has this validation for years16:03
stephenfinit's more than something has stopped ignoring that16:03
* stephenfin looks at openstackdocstheme16:03
stephenfinhmm, new release of that16:03
melwittyeah last time I helped with a docs job break it was bc of a new release of openstackdocstheme. so seems likely to be the cause16:04
melwittand I agree with you on what's likely happened, something started being strict(er) about validation16:06
stephenfinnot that, 2.2.6 hasn't made it to u-c yet16:10
stephenfinAhh, pygments 2.7.1 came in recently. That's our culprit (change I1bc1fe988e3cfd4824dd7b4a9327a67db46c1533)16:12
* stephenfin updates commit message16:12
melwitt\o/16:14
*** psachin has quit IRC16:15
openstackgerritStephen Finucane proposed openstack/nova master: docs: Make JSON valid  https://review.opendev.org/75946616:16
stephenfinI didn't see the change because I was searching for pygments (lower case p) :(16:17
stephenfinvim--16:17
stephenfinmelwitt: ^16:17
melwittheh16:17
melwitton it16:17
stephenfinThe gate is still hopelessly broken due to this libvirt/eventlet issue but that's one thing off the list16:17
melwitt++16:20
lyarwoodstephenfin: sorry wasn't watching irc upstream16:20
lyarwoodstephenfin: what's the eventlet issue?16:21
stephenfin<stephenfin> yeah, the last working build for https://review.opendev.org/#/c/758396/ used eventlet===0.26.116:21
stephenfin<stephenfin> https://zuul.opendev.org/t/openstack/build/bf1f2b730848408b99c61e82068eb73b/log/job-output.txt#635516:21
stephenfin<stephenfin> the failing one uses eventlet===0.28.016:21
stephenfin<stephenfin> https://zuul.opendev.org/t/openstack/build/00e3a2793a1040b29b944b944c9da795/log/job-output.txt16:21
stephenfin<stephenfin> so eventlet has borked us \o/16:21
stephenfinI think it's eventlet to blame, though the diff between those versions doesn't show anything particularly egregious16:21
stephenfinmypy and greenlet were also bumped so they might be more to blame16:22
*** psachin has joined #openstack-nova16:23
*** nweinber has quit IRC16:30
*** jmlowe has quit IRC16:30
stephenfinartom++ Good spot on https://review.opendev.org/#/c/758157/. A flaw in my testing16:33
*** jmlowe has joined #openstack-nova16:35
*** ralonsoh has quit IRC16:36
*** k_mouza has quit IRC16:40
*** martinkennelly has quit IRC16:46
*** recyclehero has quit IRC16:50
*** arne_wiebalck_ has joined #openstack-nova16:50
*** ianw has quit IRC16:51
*** Hazelesque_ has joined #openstack-nova16:52
*** jawad_axd has quit IRC16:52
*** artom has quit IRC16:54
*** nweinber has joined #openstack-nova16:56
*** eharney has quit IRC16:57
*** arne_wiebalck has quit IRC16:57
*** Hazelesque has quit IRC16:57
*** arne_wiebalck_ is now known as arne_wiebalck16:57
*** eharney has joined #openstack-nova17:04
*** andrewbonney has quit IRC17:06
*** mlavalle has joined #openstack-nova17:07
*** recyclehero has joined #openstack-nova17:10
*** dtantsur is now known as dtantsur|afk17:12
*** hamalq has joined #openstack-nova17:17
*** hamalq_ has joined #openstack-nova17:20
*** hamalq has quit IRC17:23
*** ianw has joined #openstack-nova17:23
*** derekh has quit IRC17:33
*** nightmare_unreal has quit IRC17:44
*** nweinber has quit IRC17:58
*** nweinber has joined #openstack-nova17:58
*** artom has joined #openstack-nova18:02
*** salmankhan1 has joined #openstack-nova18:02
*** artom has quit IRC18:02
openstackgerritMerged openstack/nova stable/stein: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available  https://review.opendev.org/74761118:02
*** artom has joined #openstack-nova18:02
*** salmankhan has quit IRC18:05
*** salmankhan1 has quit IRC18:06
melwittdansmith: if you're around, fix for the gate is at https://review.opendev.org/75946618:10
dansmithmelwitt: ack18:11
melwittthanks!18:11
*** nweinber has quit IRC18:14
*** nweinber has joined #openstack-nova18:15
johnsomYeah, as mentioned above, nova is crashing in the Octavia gates. The traceback does point to eventlet.18:16
johnsomhttps://zuul.opendev.org/t/openstack/build/2828ba97f9d54d458facd777d91dd36e/log/controller/logs/screen-n-cpu.txt#89918:16
melwitturgh. so that's what stephenfin was talking about earlier. I didn't realize it was a 100% rate failure18:18
melwittlooking into it18:20
artomstephenfin, I do what I can:)18:28
*** lbragstad has quit IRC18:29
*** dklyle has quit IRC18:35
*** ociuhandu has joined #openstack-nova18:36
*** nweinber has quit IRC18:38
*** nweinber has joined #openstack-nova18:38
johnsomHere is a paste with the packages that are different between the last good run (yesterday) and the failing run today: http://paste.openstack.org/show/799337/18:46
openstackgerritmelanie witt proposed openstack/nova master: DNM test eventlet 0.26.1  https://review.opendev.org/75950618:56
melwittah, thank you18:56
openstackgerritmelanie witt proposed openstack/nova master: DNM test eventlet 0.26.1  https://review.opendev.org/75950619:00
*** ociuhandu has quit IRC19:14
openstackgerritMerged openstack/nova stable/train: compute: Validate a BDMs disk_bus when provided  https://review.opendev.org/74455219:34
openstackgerritMerged openstack/nova stable/ussuri: Fix invalid assert_has_calls  https://review.opendev.org/75175219:34
openstackgerritmelanie witt proposed openstack/nova master: DNM try using autowrap_names instead of autowrap  https://review.opendev.org/75951319:36
*** nweinber has quit IRC19:55
*** rcernin has quit IRC19:55
*** tesseract has quit IRC20:03
openstackgerritMerged openstack/nova master: docs: Make JSON valid  https://review.opendev.org/75946620:08
*** vesper has joined #openstack-nova20:44
*** bhagyashris has joined #openstack-nova20:47
*** ianw has quit IRC20:50
*** mgariepy has quit IRC20:50
*** ircuser-1 has quit IRC20:50
*** bhagyashris|sick has quit IRC20:50
*** vesper11 has quit IRC20:50
*** ganso has quit IRC20:50
*** ianw has joined #openstack-nova20:50
*** ircuser-1 has joined #openstack-nova20:50
*** ganso has joined #openstack-nova20:53
openstackgerritBilly Olsen proposed openstack/nova stable/ussuri: add functional regression test for bug #1888395  https://review.opendev.org/75952220:56
openstackbug 1888395 in OpenStack Compute (nova) ussuri "shared live migration of a vm with a vif is broken in train" [High,In progress] https://launchpad.net/bugs/188839520:56
*** mgariepy has joined #openstack-nova21:00
openstackgerritBilly Olsen proposed openstack/nova stable/ussuri: Set migrate_data.vifs only when using multiple port bindings  https://review.opendev.org/75915121:10
*** ociuhandu has joined #openstack-nova21:14
*** ociuhandu has quit IRC21:19
*** eharney has quit IRC21:32
*** artom has quit IRC21:39
*** macz_ has quit IRC21:40
*** mlavalle has quit IRC21:48
*** raildo has quit IRC21:53
*** rcernin has joined #openstack-nova22:00
*** xek has quit IRC22:05
*** slaweq has quit IRC22:33
*** k_mouza has joined #openstack-nova22:42
*** k_mouza has quit IRC22:46
*** dklyle has joined #openstack-nova22:58
*** rcernin has quit IRC22:59
*** macz_ has joined #openstack-nova23:03
*** macz_ has quit IRC23:07
*** sean-k-mooney has quit IRC23:20
*** sean-k-mooney has joined #openstack-nova23:21
openstackgerritBilly Olsen proposed openstack/nova stable/train: add functional regression test for bug #1888395  https://review.opendev.org/75953323:22
openstackbug 1888395 in OpenStack Compute (nova) ussuri "shared live migration of a vm with a vif is broken in train" [High,In progress] https://launchpad.net/bugs/1888395 - Assigned to Billy Olsen (billy-olsen)23:22
openstackgerritBilly Olsen proposed openstack/nova stable/train: Set migrate_data.vifs only when using multiple port bindings  https://review.opendev.org/75915323:23
*** artom has joined #openstack-nova23:24
*** whoami-rajat__ has quit IRC23:36
*** owalsh_ has joined #openstack-nova23:40
*** hamalq_ has quit IRC23:43
*** owalsh has quit IRC23:43
*** owalsh has joined #openstack-nova23:45
*** tosky has quit IRC23:46
*** owalsh_ has quit IRC23:48
*** rcernin has joined #openstack-nova23:50
« Thursday, 2020-10-22 Index Saturday, 2020-10-24 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!