Thursday, 2020-03-12

*** mlavalle has quit IRC00:03
*** tetsuro has joined #openstack-nova00:13
*** TxGirlGeek has quit IRC00:22
*** nicolasbock has quit IRC00:30
*** nicolasbock has joined #openstack-nova00:30
*** nicolasbock has quit IRC00:43
brinzhanggmann: I already granular os-instance-actions GET API, you can review https://review.opendev.org/#/c/711791/300:44
*** openstackstatus has joined #openstack-nova00:44
*** ChanServ sets mode: +v openstackstatus00:44
*** tbachman_ has joined #openstack-nova00:50
*** tbachman has quit IRC00:52
*** tbachman_ is now known as tbachman00:52
*** hongbin has joined #openstack-nova01:12
gmannbrinzhang: we need to add in same patch when adding the new defaults.01:29
gmannlike this  -https://review.opendev.org/#/c/706672/7/nova/policies/attach_interfaces.py01:30
brinzhanggmann: I was already separate, and the granular beased on this patch, does need I rollback?01:31
brinzhanggmann: split it from the base patch, I think it's clear to me to know what happened01:32
gmannbrinzhang: yeah it will be easy for otherwise we endup doing half things per patches. also if you do in single patch we can see tests better way about what behavior is changed01:33
brinzhanggmann: ok, I can combine them.01:34
gmannthanks. because it end up changing the policy twice. first in default role patch and then again changing in new patch./01:36
brinzhanggmann: about you comments of https://review.opendev.org/#/c/706470/10/nova/tests/unit/cmd/test_policy.py01:38
brinzhangI think I should keep my changes, otherwise the policy seem cannot access request01:38
gmannI do not think it will make any difference if so then something wrong which we need to debug01:42
brinzhanggmann: ok, let me try01:42
gmannbecause old rules should keep working as it is.01:42
*** gyee has quit IRC01:51
*** vishalmanchanda has joined #openstack-nova01:51
*** zhanglong has joined #openstack-nova02:23
*** Xuchu has joined #openstack-nova02:27
*** sapd1_x has joined #openstack-nova02:45
*** hongbin has quit IRC03:03
*** mkrai has joined #openstack-nova03:04
*** Xuchu_ has joined #openstack-nova03:20
*** Xuchu has quit IRC03:22
*** Xuchu_ is now known as Xuchu03:22
*** KeithMnemonic1 has joined #openstack-nova03:28
*** KeithMnemonic has quit IRC03:31
openstackgerritKevin Zhao proposed openstack/nova master: fix scsi disk unit number of the attaching volume when cdrom bus is scsi  https://review.opendev.org/71260703:36
*** psachin has joined #openstack-nova03:37
*** tetsuro has quit IRC03:41
*** yaawang has quit IRC03:44
*** yaawang has joined #openstack-nova03:45
*** tetsuro has joined #openstack-nova03:45
*** dave-mccowan has quit IRC03:58
*** KeithMnemonic1 has quit IRC04:06
*** zhanglong has quit IRC04:10
*** tobiash has quit IRC04:16
*** udesale has joined #openstack-nova04:17
*** tobiash has joined #openstack-nova04:17
*** mkrai has quit IRC04:21
*** mkrai has joined #openstack-nova04:21
openstackgerritBrin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies  https://review.opendev.org/70647004:24
*** factor has joined #openstack-nova04:34
openstackgerritBrin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies  https://review.opendev.org/70647004:40
*** tobiash has quit IRC04:43
brinzhanggmann: I was merged these patches, and leave comments in that failed case, would please review this after you online https://review.opendev.org/70647004:46
*** tobiash has joined #openstack-nova04:46
*** tetsuro_ has joined #openstack-nova05:23
*** tetsuro has quit IRC05:27
*** zhanglong has joined #openstack-nova05:31
*** lbragstad__ has joined #openstack-nova05:33
*** lbragstad_ has quit IRC05:35
*** evrardjp has quit IRC05:35
*** evrardjp has joined #openstack-nova05:36
openstackgerritKevin Zhao proposed openstack/nova master: fix scsi disk unit number of the attaching volume when cdrom bus is scsi  https://review.opendev.org/71260705:57
*** links has joined #openstack-nova05:58
*** Luzi has joined #openstack-nova06:02
*** bbowen_ has quit IRC06:48
*** bbowen_ has joined #openstack-nova06:48
*** LiangFang has quit IRC07:02
*** smcginnis has quit IRC07:03
*** dpawlik has joined #openstack-nova07:04
*** smcginnis has joined #openstack-nova07:04
*** Liang__ has joined #openstack-nova07:04
*** dSrinivas has joined #openstack-nova07:08
dSrinivasHi All, i am getting this error http://paste.openstack.org/show/790571/  from March 3 in nova-compute logs in our CI FC HBA's are Online and Connectivity exists between vm and Storage07:08
*** rpittau|afk is now known as rpittau07:17
*** tetsuro has joined #openstack-nova07:25
*** tetsuro_ has quit IRC07:29
*** tetsuro_ has joined #openstack-nova07:35
*** tetsuro has quit IRC07:37
*** damien_r has joined #openstack-nova07:54
*** slaweq has joined #openstack-nova08:02
*** amoralej|off is now known as amoralej08:06
openstackgerritMerged openstack/nova stable/queens: Add retry_on_deadlock to migration_update DB API  https://review.opendev.org/71221208:07
*** tesseract has joined #openstack-nova08:08
*** maciejjozefczyk has joined #openstack-nova08:15
*** tosky has joined #openstack-nova08:18
*** sapd1_x has quit IRC08:23
gibimorning08:28
*** zhanglong has quit IRC08:44
*** tetsuro_ has quit IRC08:44
*** tetsuro has joined #openstack-nova08:47
*** rpittau is now known as rpittau|bbl08:52
*** tetsuro_ has joined #openstack-nova08:53
*** tetsuro has quit IRC08:57
*** ralonsoh has joined #openstack-nova08:58
*** xek has joined #openstack-nova08:58
*** dtantsur|afk is now known as dtantsur09:13
*** tetsuro has joined #openstack-nova09:14
*** damien_r has quit IRC09:16
*** damien_r has joined #openstack-nova09:16
*** tetsuro_ has quit IRC09:17
*** tetsuro_ has joined #openstack-nova09:20
*** tetsuro has quit IRC09:23
*** priteau has joined #openstack-nova09:25
lyarwoodhttps://review.opendev.org/#/q/status:open+topic:bug/1861071 - I'd appreciate reviews of this bugfix series from any non-RH cores with bandwidth if possible, there's a followup series on top of this as well as I'd like to backport this.09:25
gibilyarwood: ack, added to my growing review queue09:26
lyarwoodgibi: awesome thanks :) anything I can help with in return please let me know09:27
gibilyarwood: sure. I have two patch series to be backported furter, as soon I as I did the backport I will ping you09:28
lyarwoodgibi: ack thanks09:33
*** masterpe has joined #openstack-nova09:34
*** derekh has joined #openstack-nova09:41
*** martinkennelly has joined #openstack-nova09:42
masterpeHi, when I try to live migrate a instance I get the following error: No valid host was found. Unable to move instance ae88f5d0-28be-4234-86bf-d89a33a2380b to host compute2. There is not enough capacity on the host for the instance09:42
masterpecompute2 has been host-evacuate, after that I deleted all the instance config files of the compute node09:43
masterpeThe nova logs gets filled with messgae: Instance 9111eb48-2291-457c-88e2-57f5c4491aa4 has been moved to another host compute3 (compute3.example.tld). There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 4, u'MEMORY_MB': 16384}}.: InstanceNotFound_Remote: Instance 0b02dcda-b283-435c-9986-891323411973 could not be found.09:46
*** martinkennelly has quit IRC09:46
masterpeThe DB table allocations is empty09:46
*** martinkennelly has joined #openstack-nova09:47
*** Xuchu_ has joined #openstack-nova09:49
*** Xuchu has quit IRC09:50
*** Xuchu_ is now known as Xuchu09:50
*** sapd1 has joined #openstack-nova09:50
gibimasterpe: did nova-compute service on compute2 was recovered after you evacuated everything out of compute2?09:52
masterpesorry?09:53
gibiI think the nova-compute on compute2 should clean up the allocations09:53
gibimasterpe: I assume that when compute2 was host-evacuated the nova-compute service on that host was down09:53
gibimasterpe: and then you recovered that service09:54
gibimasterpe: and then you tried to live migrate to compute209:54
masterpeyes09:54
*** martinkennelly has quit IRC09:55
openstackgerritJohn Garbutt proposed openstack/nova master: Add tests for API using unified limits driver  https://review.opendev.org/71214009:55
gibimasterpe: the "allocations remaining" logs ae coming from the compute2 log?09:55
masterpeyes09:56
openstackgerritJohn Garbutt proposed openstack/nova master: Make unified limits APIs return reserved of 0  https://review.opendev.org/71214109:56
gibimasterpe: what nova versions do you use?09:56
masterpeI use it from openstack-ansible tag 18.1.1209:58
masterpeit is comming from source09:58
*** martinkennelly has joined #openstack-nova09:58
*** ociuhandu has joined #openstack-nova09:59
gibibased on https://releases.openstack.org/teams/openstackansible.html that means you use Rocky09:59
masterpenova-compute --version gives me 18.2.309:59
masterpeyes09:59
gibidid you check the allocations table in the placement database? (I'm not sure how ansible deploys placement in rocky)10:00
masterpethere is no placement database10:02
openstackgerritJohn Garbutt proposed openstack/nova master: WIP: Enforce api and db limits  https://review.opendev.org/71214210:02
gibimasterpe: then I guess there is a nova_api database with an allocations table10:02
openstackgerritJohn Garbutt proposed openstack/nova master: Add unified limits configuration  https://review.opendev.org/71213710:02
gibialso you can try to look at the allocations from the placement aPI10:03
gibi$ openstack resource provider list --name compute210:04
gibigrab the uuid from the output10:04
masterpeah yes, thanks. There is a allocations table in the database nova and nova_api10:04
gibiopenstack resource provider allocation show <uuid>10:04
gibido you see allocations on your emptied compute?10:06
masterpeon the openstack cli I don't see a resource provider10:07
gibi'openstack resource provider list'  without the name filter should list your compute2 maybe you use fqdn and that makes the name compute2.example.tld10:17
*** zhanglong has joined #openstack-nova10:17
gibimasterpe: when you recovered compute2 did you restarted the nova-compute service? If yes did you see logs like https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L729 during the startup of the service?10:22
masterpeThe system was completly powerd off, after evacuated I started the server10:23
gibiduring the service startup nova-compute should have cleaned up the allocations of the evacuated instances and log the above linket INFO log10:25
*** ociuhandu has quit IRC10:25
*** ociuhandu has joined #openstack-nova10:28
*** dSrinivas has quit IRC10:38
*** tbachman has quit IRC10:38
*** tetsuro_ has quit IRC10:39
*** ociuhandu has quit IRC10:49
*** ociuhandu has joined #openstack-nova10:50
openstackgerritBrin Zhang proposed openstack/python-novaclient master: WIP Microversion 2.83: Add volume-patch CLI  https://review.opendev.org/71265110:53
*** ociuhandu has quit IRC10:53
*** nicolasbock has joined #openstack-nova10:56
*** ociuhandu has joined #openstack-nova10:58
*** vishalmanchanda has quit IRC11:00
*** sapd1 has quit IRC11:00
*** jamesdenton has joined #openstack-nova11:01
*** mkrai has quit IRC11:02
*** mkrai_ has joined #openstack-nova11:02
*** ociuhandu has quit IRC11:03
brinzhanggmann: would you have time to review os-instance-actions --- add default policy patch? as I said before, the cmd/test_policy for test_filter_rules_non_admin and test_filter_rules_instance_owner, I think the default policy need to be update.11:06
*** derekh has quit IRC11:10
*** nicolasbock has quit IRC11:15
*** nicolasbock has joined #openstack-nova11:16
*** derekh has joined #openstack-nova11:20
*** ociuhandu has joined #openstack-nova11:20
*** ociuhandu has quit IRC11:27
*** ociuhandu has joined #openstack-nova11:27
*** tetsuro has joined #openstack-nova11:28
*** mkrai_ has quit IRC11:37
*** nicolasbock has quit IRC11:38
*** nicolasbock has joined #openstack-nova11:39
*** nicolasbock has quit IRC11:42
*** nicolasbock has joined #openstack-nova11:43
*** nicolasbock has quit IRC11:51
*** eharney has joined #openstack-nova11:52
*** ociuhandu has quit IRC11:54
*** nicolasbock has joined #openstack-nova11:54
*** ociuhandu has joined #openstack-nova11:55
*** ociuhandu has quit IRC11:55
*** ociuhandu has joined #openstack-nova11:55
*** nicolasbock has quit IRC11:56
*** nicolasbock has joined #openstack-nova11:57
*** Luzi has quit IRC12:10
*** udesale_ has joined #openstack-nova12:14
*** zhanglong has quit IRC12:15
*** udesale has quit IRC12:17
*** tetsuro has quit IRC12:22
*** rcernin has quit IRC12:25
*** tkajinam has quit IRC12:27
*** zhanglong has joined #openstack-nova12:30
openstackgerritJohn Garbutt proposed openstack/nova master: Enforce api and db limits  https://review.opendev.org/71214212:31
openstackgerritJohn Garbutt proposed openstack/nova master: Add logic to enforce local api and db limits  https://review.opendev.org/71213912:37
openstackgerritJohn Garbutt proposed openstack/nova master: Add tests for the UnifiedLimits quota driver  https://review.opendev.org/71213812:37
openstackgerritJohn Garbutt proposed openstack/nova master: Add logic to enforce local api and db limits  https://review.opendev.org/71213912:37
openstackgerritJohn Garbutt proposed openstack/nova master: Add tests for API using unified limits driver  https://review.opendev.org/71214012:37
openstackgerritJohn Garbutt proposed openstack/nova master: Make unified limits APIs return reserved of 0  https://review.opendev.org/71214112:38
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: YAML file loading and schema validation  https://review.opendev.org/67334112:38
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs  https://review.opendev.org/67602912:38
openstackgerritJohn Garbutt proposed openstack/nova master: Enforce api and db limits  https://review.opendev.org/71214212:38
openstackgerritJohn Garbutt proposed openstack/nova master: WIP: Update quota_class APIs for db and api limits  https://review.opendev.org/71214312:39
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs  https://review.opendev.org/67602912:41
*** psachin has quit IRC12:47
*** zhanglong has quit IRC12:49
*** ociuhandu has quit IRC12:50
*** ociuhandu has joined #openstack-nova12:51
*** eharney has quit IRC12:53
*** mmethot has joined #openstack-nova12:56
*** ociuhandu has quit IRC12:57
*** tbachman has joined #openstack-nova12:57
*** nweinber has joined #openstack-nova12:59
*** ociuhandu has joined #openstack-nova13:00
*** rpittau|bbl is now known as rpittau13:02
openstackgerritJohn Garbutt proposed openstack/nova master: Update quota_class APIs for db and api limits  https://review.opendev.org/71214313:12
*** psachin has joined #openstack-nova13:12
*** amoralej is now known as amoralej|lunch13:15
*** ociuhandu has quit IRC13:15
*** ociuhandu has joined #openstack-nova13:16
*** Xuchu has quit IRC13:16
openstackgerritBalazs Gibizer proposed openstack/nova master: Ensures that COMPUTE_RESOURCE_SEMAPHORE usage is fair  https://review.opendev.org/71267413:17
*** Xuchu has joined #openstack-nova13:17
gibimelwitt: ^^13:17
gibimelwitt: I created some test and I'm +2 on Jason's patch13:19
*** ociuhandu has quit IRC13:21
*** spatel has joined #openstack-nova13:21
masterpegibi: I have just rebooted the compute node and search in the nova-compute.log, but I could not find any "Cleaning up allocations" message13:22
*** ociuhandu has joined #openstack-nova13:22
gibimasterpe: interesting13:22
*** Xuchu has quit IRC13:24
gibimasterpe: did you managed to find the allocations in placement either via the CLI or in the db?13:25
*** spatel has quit IRC13:26
gibimasterpe: are the evacuated instances still exists (not deleted) on compute3 ?13:27
masterpecli does not work, I will search in the DB.13:27
masterpeyes13:27
*** derekh has quit IRC13:29
*** ociuhandu has quit IRC13:30
dansmithgibi: I think it's safe to merge the vmware deprecation package now.. quite a bit of support on it13:31
*** ociuhandu has joined #openstack-nova13:32
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.83: Add volume-patch CLI  https://review.opendev.org/71265113:33
brinzhangdansmith: can you leave some comments in https://review.opendev.org/#/c/694428/3/nova/objects/instance_action.py@20113:34
brinzhangdansmith: I am confusing of mriedem said in comments ^^13:34
gibidansmith: done13:34
dansmithbrinzhang: yeah I remember not fully grasping that as well, but I will read and try to grok13:35
dansmithgibi: thanks13:35
gibidansmith: thanks for proposing the deprecation13:35
dansmithnp13:35
brinzhangdansmith: thanks, I know little about serialize_args.13:36
*** dtantsur is now known as dtantsur|brb13:36
masterpegibi: yes the instance are still on compute313:37
*** eharney has joined #openstack-nova13:38
gibimasterpe: was the evacuation successfull? what does nova instance-action-list <uuid> shows regarding the evacuation?13:38
*** mkrai_ has joined #openstack-nova13:39
lyarwoodsean-k-mooney / stephenfin / kashyap: https://review.opendev.org/#/c/696834/ - would you be able to hit this today if you have time?13:40
*** tbachman has quit IRC13:40
kashyaplyarwood: I have already read bits of the patch yesterday, will respond13:41
masterpenova instance-action-list b4a3a998-e8b4-482c-ae4d-dddaee707ff1 ( the uuid is from a instance )13:41
masterpeevacuate      | req-552e7f33-8f94-4bc3-a992-8f59b1f92f11 | -       | 2020-03-06T17:26:14.00000013:41
lyarwoodkashyap: thanks13:42
gibimasterpe: yeah so the evacuation was successful. I have no further idea how https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L729 is not triggered.13:43
gibimasterpe: if you find the offending allocations against compute2 in allocations table of the nova_api db then you can deleted them manually. But I have no idea how to prevent them to happen13:44
dansmithbrinzhang: so, the thing I don't get about what he said is the "pass a format_exc_val kwarg" which isn't really an option here13:44
masterpeI can not find that code in the file13:44
dansmithbrinzhang: but we can make serialize_args pass the cleaned-up exception message-or-type thing for exc_val13:45
*** amoralej|lunch is now known as amoralej13:46
masterpegibi: In my manager.py the commit https://opendev.org/openstack/nova/commit/09de94e39bbcfc7f8130638e73a8248e49cb6ab7 is missing, so probebly the file is from before that.13:48
*** dklyle has quit IRC13:49
*** dklyle has joined #openstack-nova13:49
brinzhangdansmith: what do you mean? Can you elaborate? I still don't quite understand. In other words, I don't know what I should do about @serialize_args13:50
gibimasterpe: ahh. you have 18.2.3 and that patch is in 18.313:50
dansmithbrinzhang: I'm cooking up a patch for you13:50
gibimasterpe: that explains what you see13:50
brinzhangdansmith: thanks13:50
masterpewhen I do select * from allocations where consumer_id = 'b4a3a998-e8b4-482c-ae4d-dddaee707ff1';13:51
masterpeI see 4 enty's13:51
masterpethe collum resource_provider_id is the compute node?13:51
gibimasterpe: yes, the resource_provider_id is the compute node uuid13:52
gibimasterpe: most probably you have allocation for that instance on two computes, compute2 and compute313:52
gibithe bugfix that is missing from your rocky version is fixing that13:53
gibiif you could upgrade to 18.3 and restart the compute that would also fix your deployment as well13:53
*** jraju__ has joined #openstack-nova14:01
*** dave-mccowan has joined #openstack-nova14:02
*** links has quit IRC14:02
*** derekh has joined #openstack-nova14:03
dansmithgibi: fwiw, normally I can make the 1600 meeting but not today because I already have a thing I have to do14:03
dansmither, s/fwiw/fyi/14:03
gibidansmith: ack, thanks for the heads up14:03
masterpeThe id in resource_providers is linked with resource_provider_id of allocations14:04
gibimasterpe: yes sorry, the resource_provider_id is an internal id pointing to the resource_providers table and there is an uuid in that table that points to the compute node14:05
masterperesource_provider_id has id 914:05
masterpeso I can delete all records in allocations with resource_provider_id = 9?14:06
*** tbachman has joined #openstack-nova14:08
gibimasterpe: if you evacuated all the servers from compute2 then yes14:08
masterpeyes compute2 is emty14:09
*** ociuhandu has quit IRC14:11
*** eharney has quit IRC14:13
*** kplant has joined #openstack-nova14:19
*** dSrinivas has joined #openstack-nova14:21
dSrinivasHi all, while attaching the volume to the Nova intance it is failing with this errorhttp://paste.openstack.org/show/790596/ can anyone please me14:23
dSrinivasFor FC not for iscsi14:25
*** dave-mccowan has quit IRC14:25
*** dave-mccowan has joined #openstack-nova14:29
sean-k-mooneyso that look like an os_brick issue but have you check if the FC deivce is mounted properly on the host at the expected path14:31
dSrinivasFC devices are Online14:31
dSrinivasIt is attaching the volume and again it is deleting after 20seconds14:32
sean-k-mooneyand if you run the grep command yourself does it find the deivice14:32
dSrinivasCinder volume logs it is showing attachment updated successful14:32
sean-k-mooneyhum well the deleing it proably part of the clean up14:32
dSrinivasSystool -c fc_host -v | grep Online it is showing the ports as online14:33
*** mriedem has joined #openstack-nova14:33
sean-k-mooneyon the storage backend14:34
sean-k-mooneythat does not mean that the compute node was able to connect to it and mount it right?14:34
sean-k-mooneyit just means the sotrage backend thing it available to be mounted?14:34
dSrinivasAt storage side it is showing attached again after 20se14:34
sean-k-mooneyFC is not my thing14:34
lyarwooddSrinivas: which version is this?14:35
dSrinivasDeleting14:35
dSrinivasMaster14:35
openstackgerritDan Smith proposed openstack/nova master: Make serialize_args handle exception messages safely  https://review.opendev.org/71269714:35
lyarwoodyeah sorry no idea then, can you raise this in an os-brick bug and ask for help in #openstack-cinder ?14:35
sean-k-mooneydSrinivas: the only thing that comes to mind is maybe it is timing out waithg for it to be avialble14:35
dansmithbrinzhang: mriedem ^14:35
mriedemooo14:36
dSrinivasHow to increase the retries14:36
sean-k-mooneydSrinivas: im not sure.14:38
dSrinivasIt is attaching and then deleting immediately14:38
sean-k-mooneyand you can see that on the compute node not the storage backend14:38
lyarwooddSrinivas: ignore the volume state, the underlying block devices aren't present.14:39
lyarwooddSrinivas: raise an os-brick bug for this and reach out to the folks in #openstack-cinder14:40
lyarwooddSrinivas: we don't directly support the os-brick code in this channel14:40
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree  https://review.opendev.org/67652214:40
dSrinivasI have check the n-cpu logs and pasted the link above14:41
lyarwooddSrinivas: yes, n-cpu is calling os-brick and it is failing to connect the volume here14:41
sean-k-mooneydSrinivas: which is showing an error form os-bricks which is a lib that is maintained by both cinder and nova14:41
lyarwoodsean-k-mooney: well we chip in with the odd bugfix ;)14:42
sean-k-mooneyyes14:42
*** ociuhandu has joined #openstack-nova14:42
sean-k-mooneywell you chip in with the odd bugfix i avoid it14:42
*** gyee has joined #openstack-nova14:43
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs  https://review.opendev.org/69346014:44
kplantanyone know what would cause "No valid host was found. No valid host found for resize " when trying to resize an instance?14:44
kplanti have plenty of resources available and can deploy new instances with the same flavor i'm trying to resize to14:45
mriedemdansmith: left some comments, thanks14:45
dansmithmriedem: thanks14:47
*** ociuhandu has quit IRC14:47
*** TxGirlGeek has joined #openstack-nova14:48
openstackgerritTakashi NATSUME proposed openstack/python-novaclient master: Microversion 2.82 - nova cyborg interaction  https://review.opendev.org/71270014:50
jkulikkplant, if you only have one host, you'd explicitly have to allow resize on the same host, iirc14:51
kplanti did see that, i have more than one host though14:51
jkulikthen you'll have to look into the logs of nova-scheduler I guess14:51
kplantunfortunately it gives me the same vague error message14:51
*** psachin has quit IRC14:52
jkulikenabling DEBUG logging always helps me understanding why the scheduler couldn't find an appropriate host. you need to search the logs for the request-id, not the instance uuid though.14:53
kplantaah debug is a good idea14:53
kplantty14:53
*** spatel has joined #openstack-nova14:54
*** ociuhandu has joined #openstack-nova14:57
*** spatel has quit IRC14:57
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: YAML file loading and schema validation  https://review.opendev.org/67334114:57
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs  https://review.opendev.org/67602914:58
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree  https://review.opendev.org/67652214:58
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs  https://review.opendev.org/69346014:58
*** spatel has joined #openstack-nova14:58
sean-k-mooney^ that is just a clean rebase against master to allo the test to be run locally14:58
sean-k-mooneyit was complaing about moxs stubs14:58
*** Xuchu has joined #openstack-nova15:01
*** maciejjozefczyk has quit IRC15:02
*** ociuhandu has quit IRC15:02
*** spatel has quit IRC15:03
mriedemsmcginnis: i hope my characterization of you being fluent in executive is accurate https://review.opendev.org/#/c/712498/15:06
*** tbachman has quit IRC15:06
*** mkrai_ has quit IRC15:13
*** mkrai has joined #openstack-nova15:13
*** Xuchu has left #openstack-nova15:13
iurygregoryHi nova team, is this a know issue in the grenade tests http://paste.openstack.org/show/790610/ we started to see this in the ironic-grenade job while running the Smoke Tests http://zuul.openstack.org/build/3b20be2558c044628361f34eae06d52015:14
openstackgerritJohn Garbutt proposed openstack/nova master: WIP: Update limit APIs  https://review.opendev.org/71270715:15
iurygregoryops not with grenade tests/ smoke tests =)15:15
lyarwoodiurygregory: http://status.openstack.org/elastic-recheck/#1844929 - I can't tell if it's the same issue but I've seen this recently in the grenade job15:18
*** maciejjozefczyk has joined #openstack-nova15:20
*** prometheanfire has joined #openstack-nova15:21
prometheanfireI think the mock update is causing some pain15:21
prometheanfirehttps://1b8c54b5d412a3af0fef-40bd60678638a1db566d5d37b438f20d.ssl.cf1.rackcdn.com/712615/2/check/cross-nova-py36/8e15404/testr_results.html15:21
prometheanfirehttps://review.opendev.org/71261515:21
iurygregorywell I see some Timed out waiting for response from cell in the logs also...15:22
prometheanfirewoo15:22
prometheanfirefun time all around15:22
iurygregorylyarwood, maybe .-. it's since the is the same message in logs15:23
iurygregorybut seems "random" so far in the ironic jobs15:23
iurygregorysome times the tests in grenade just works .-.15:23
lyarwoodyup that's what we are seeing with this issue15:24
*** dSrinivas has quit IRC15:24
openstackgerritLee Yarwood proposed openstack/nova master: workarounds: Add option to disable native LUKSv1 decryption by QEMU  https://review.opendev.org/70803015:25
openstackgerritLee Yarwood proposed openstack/nova master: workarounds: Connect RBD volumes to the compute host as block devices  https://review.opendev.org/70802915:25
*** mlavalle has joined #openstack-nova15:26
prometheanfirehttps://review.opendev.org/712713 created to test just mock-415:28
prometheanfirethat being said, nova still fails tests on the alembic update15:29
prometheanfirethat test is here https://review.opendev.org/70870615:30
openstackgerritJohn Garbutt proposed openstack/nova master: Update limit APIs  https://review.opendev.org/71270715:38
sean-k-mooneyanyone else seeing http://paste.openstack.org/show/790613/ with master if you try to run tox15:40
sean-k-mooneywell tox -e py36  resource_tracker15:41
*** sapd1 has joined #openstack-nova15:42
sean-k-mooneyim also seeign this in my second nova repo15:44
sean-k-mooneyill try a fresh clone just in case15:44
*** TxGirlGeek has quit IRC15:45
gibisean-k-mooney: tox -e py36 resource_tracker -r passed on master for me15:46
*** TxGirlGeek has joined #openstack-nova15:47
sean-k-mooneyit looks like im missing https://github.com/openstack/nova/commit/91973ec26c3ae77ffe349bc4c5c03f13065d9b6f15:48
sean-k-mooneybut i did a rebase against master just before running the tests15:48
kashyaplyarwood: Responded...15:48
sean-k-mooneyi wonder if the repos are all in sync and what one im using as my origin15:48
openstackgerritBalazs Gibizer proposed openstack/nova stable/pike: Mask the token used to allow access to consoles  https://review.opendev.org/70887615:49
*** eharney has joined #openstack-nova15:49
sean-k-mooneyill rebase against the gerrit remote which shoudl be up to date15:49
lyarwoodkashyap: thanks15:50
gibisean-k-mooney: I have 91973ec26c3ae77ffe349bc4c5c03f13065d9b6f in my repo15:50
kashyaplyarwood: Not sure how much of what I wrote makes sense :-); I'll check back in later15:50
lyarwoodkashyap: yup it does, I wasn't aware of the other API15:52
lyarwoodkashyap: I'll take a look after $meeetings15:52
kashyaplyarwood: It's part of the (you wouldn't believe) 5-ish year effort in QEMU/libvirt to get the incremental backup stuff and all the relevant things15:53
sean-k-mooneygibi: my origin was set to https://review.openstack.org/p/openstack/nova i wonder if one fo the servers is out of sync15:53
gibisean-k-mooney: my setup is a bit complicated as I push through nordix, gerrit.nordix.org15:54
gibiand I'm not sure what nordix mirrors15:54
sean-k-mooneygibi: ya no worries i can fix this by swaping my origin to opendev or gitub15:54
sean-k-mooneyinstead of directly form gerrit15:54
sean-k-mooneybut its just strange that it thinks im up to date15:55
gibinova team meeting starts in 5 minutes on #openstack-meeting-3 (please note the place!)15:55
*** dtantsur|brb is now known as dtantsur15:56
sean-k-mooneyoh it moved15:56
sean-k-mooneyas part of the time chagne15:56
gibiyeah, #openstack-meeting was booked this time15:57
*** mkrai has quit IRC15:58
*** spatel has joined #openstack-nova16:00
*** spatel has quit IRC16:01
*** spatel has joined #openstack-nova16:01
*** tbachman has joined #openstack-nova16:02
openstackgerritJohn Garbutt proposed openstack/nova master: Add tests for API using unified limits driver  https://review.opendev.org/71214016:14
*** ociuhandu has joined #openstack-nova16:15
openstackgerritJohn Garbutt proposed openstack/nova master: Make unified limits APIs return reserved of 0  https://review.opendev.org/71214116:15
openstackgerritJohn Garbutt proposed openstack/nova master: Enforce api and db limits  https://review.opendev.org/71214216:15
openstackgerritJohn Garbutt proposed openstack/nova master: Update quota_class APIs for db and api limits  https://review.opendev.org/71214316:16
openstackgerritJohn Garbutt proposed openstack/nova master: Update limit APIs  https://review.opendev.org/71270716:16
*** ociuhandu has quit IRC16:19
openstackgerritMerged openstack/nova master: Deprecate the vmwareapi driver  https://review.opendev.org/71210116:28
*** dpawlik has quit IRC16:28
*** dpawlik has joined #openstack-nova16:28
*** sapd1 has quit IRC16:39
*** jangutter has joined #openstack-nova16:40
*** tesseract has quit IRC16:40
gibiartom, bauzas: I have to leave for today, but let's discuss the meeting time clash at some point16:45
bauzasgibi: we're trying to fix it16:45
artomgibi, we *have* fixed it :)16:46
bauzasyou shouldn't impact the whole community for a specific internal-only issue16:46
artomJust not this week, because it was too last minute16:46
artomNext week our call in on Wednesday, so no conflict16:46
artom(Well, unless someone strongly objects internally)16:46
bauzasgibi: technically, for a worldwide distributed team, timeslots around 5-6pm our time are strategic16:46
gibiartom, bauzas: thanks16:47
gibiand sorry for the short notice16:47
bauzasbut we only have 5 of those slots per week :)16:47
gibiI would not able to chari the night meeting16:47
bauzasso, conflics are usual :)16:47
bauzasconflicts* even16:47
* gibi leaves the building16:48
*** udesale_ has quit IRC16:49
mnaserhi nova friends -- this is kind-of-a-big-change and it'd be nice if opinions came in here -- https://review.opendev.org/#/c/712696/416:52
bauzasmnaser: that's surely something I can comment on16:54
mnaserbauzas: please :)16:54
sean-k-mooneygibi: i dont think we will have an issue aft the eu moves to DST16:55
sean-k-mooneygibi: we are also looking at another time internally16:55
sean-k-mooneygibi: so we shoudl be about to sort it out internally16:55
*** damien_r has quit IRC16:58
*** priteau has quit IRC17:01
openstackgerritSylvain Bauza proposed openstack/nova master: WIP: Allocate mdevs when resizing or reverting resize  https://review.opendev.org/71274117:09
openstackgerritJohn Garbutt proposed openstack/nova master: Update limit APIs  https://review.opendev.org/71270717:13
openstackgerritJohn Garbutt proposed openstack/nova master: WIP: Enforce unified limits using oslo.limit  https://review.opendev.org/61518017:13
openstackgerritJohn Garbutt proposed openstack/nova master: DNM: Turn on unified limits to see if tests pass  https://review.opendev.org/70838717:15
*** jraju__ has quit IRC17:16
kplantjkulik: for the record, the problem seems to have been that the AZ which the instance was created for no longer existed17:22
kplantthe resize function kept reqesting an AZ with the original name17:22
kplantso i just created an AZ with that name. not sure if that's intended behavior17:23
*** rpittau is now known as rpittau|afk17:27
johnthetubaguykplant: if you requested the AZ in the first place, I believe that is expected17:30
kplantjohnthetubaguy: you're everywhere :-)17:32
johnthetubaguyheh, sometimes17:33
*** jangutter_ has joined #openstack-nova17:33
*** dtantsur is now known as dtantsur|afk17:34
*** evrardjp has quit IRC17:35
*** evrardjp has joined #openstack-nova17:36
*** jangutter has quit IRC17:36
openstackgerritJohn Garbutt proposed openstack/nova master: WIP: Fix up quota classes APIs  https://review.opendev.org/71274917:41
openstackgerritJohn Garbutt proposed openstack/nova master: WIP: Fix up quota sets APIs  https://review.opendev.org/71274917:42
*** kozhukalov has joined #openstack-nova17:46
openstackgerritElod Illes proposed openstack/nova stable/rocky: Remove global state from the FakeDriver  https://review.opendev.org/71275117:49
stephenfinjohnthetubaguy: if you're about for another few minutes, care to finish off the remove-nova-network series? https://review.opendev.org/#/c/703974/17:51
* stephenfin wants to mark the BP complete17:51
*** derekh has quit IRC18:01
openstackgerritmelanie witt proposed openstack/nova master: Add config option for neutron client retries  https://review.opendev.org/71222618:06
sean-k-mooneyanyone know where the placement inventory class is defiend in nova18:15
sean-k-mooneyok its not a class or object its just a random dict https://github.com/openstack/nova/blob/2f88e09609ee3075d675c1c41358f23901e56bce/nova/compute/provider_tree.py#L6318:21
sean-k-mooneyah  # dict of inventory records, keyed by resource class18:22
sean-k-mooneythat is what i was looking for18:22
*** ralonsoh has quit IRC18:29
openstackgerritSasha Andonov proposed openstack/nova master: rbd_utils: increase _destroy_volume timeout  https://review.opendev.org/70576418:33
*** martinkennelly has quit IRC18:39
*** maciejjozefczyk has quit IRC18:42
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: YAML file loading and schema validation  https://review.opendev.org/67334118:43
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs  https://review.opendev.org/67602918:43
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree  https://review.opendev.org/67652218:43
openstackgerritsean mooney proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs  https://review.opendev.org/69346018:43
openstackgerritStephen Finucane proposed openstack/nova master: Handle flavor disk mismatches when resizing  https://review.opendev.org/71276618:46
stephenfinmelwitt: did you # scsi > 7 patch land? Have you a link?18:49
melwittstephenfin: what? the scsi limited to 7 thing is a libvirt bug. previous to that bug, the limit was 26 and a patch of mine from stein changed it to be configurable18:50
melwittwhich thing are you looking for?18:51
stephenfinAh, I got that assways, sorry /o\18:51
stephenfinI was conflating the two18:52
stephenfinhttps://bugs.launchpad.net/nova/+bug/186427918:52
openstackLaunchpad bug 1864279 in OpenStack Compute (nova) "Unable to attach more than 6 scsi volumes" [Undecided,New]18:52
stephenfintrying to find a dupe to associate with that18:52
melwittstephenfin: ok, just looked. yeah, I remember we discussed downstream whether to propose a patch that basically checks "!= 7" and skips, but the consensus was for libvirt to fix its bug18:54
melwittand apparently that did not happen yet? this has been going on for a long time and it looks like we need to talk about it again18:56
melwittit doesn't really seem right for us to have to cover over the libvirt bug. and I don't know why it's not getting fixed on their side18:57
melwittlast I heard, they said they were fixing it18:57
stephenfinI've no idea whether it has been fixed yet or not. Could be that the fix just hasn't been backported? I can poke danpb or kashyap about it in the AM18:58
stephenfinI'm tempted to close this bug as not an issue here though. It's clearly a libvirt issue18:58
melwittyeah I mean it seems not fixed if these ppl are hitting it18:58
stephenfinThe only resolution I feel we could do is add a warning for the affected versions. We have done that before, in fairness18:59
melwittbut yeah def let's ask danpb or kashyap tomorrow18:59
stephenfincool, I'll do that and consider adding the nova runtime version check if it's worth it18:59
*** gmann is now known as gmann_lunch19:00
openstackgerritMerged openstack/nova master: Run sdk functional tests on nova changes  https://review.opendev.org/71209219:00
openstackgerritMerged openstack/nova master: [Trivial] Fix code comment of admin password tests  https://review.opendev.org/71173419:00
melwittstephenfin: here's the bug ticket https://bugzilla.redhat.com/show_bug.cgi?id=1744079 says it's fixed, so need to look at what versions this is showing up in? maybe they just need to update or maybe a backport is missing like you said?19:06
openstackmelwitt: Error: Error getting bugzilla.redhat.com bug #1744079: NotPermitted19:06
melwittsorry that was a clone https://bugzilla.redhat.com/show_bug.cgi?id=174178219:07
openstackmelwitt: Error: Error getting bugzilla.redhat.com bug #1741782: NotPermitted19:07
kplantif i have a instance on say "compute-01" and virsh shows it there, but nova thinks it's on "compute-02" how can i fix that?19:09
stephenfinmelwitt: Yup, looks like this is the commit we need https://github.com/libvirt/libvirt/commit/c8007fdc5d2ce43fec2753cda60fb4963f55abd519:12
stephenfin$ git tag --contains c8007fdc5d2ce43fec2753cda60fb4963f55abd5 | head -119:12
stephenfinv5.10.019:12
*** kozhukalov has quit IRC19:13
melwittstephenfin: hm that looks like it. but v5.10.0 doesn't match the fixed in versions on the rhbzs? I don't understand that19:15
stephenfinbackport, I'd say19:17
melwittcause according to that there should be a fix in the 4.5.x series too19:17
melwittyeah..19:17
stephenfinokay, I definitely don't think we need to do anything here in nova so19:18
stephenfinThe bug's there. People can find it via their favourite search engine and annoy their vendors to backport the fix if needed19:18
sean-k-mooneywe cant rely on backport in downstream version upstream19:20
melwittyeah, agreed nothing to do in nova. I was trying to find some links to leave in the lp bug but I realize I don't know how upstream libvirt works, how to tell them what version etc19:20
sean-k-mooneyso until we can raise the min libvirt version to 5.10 we cant assume that commit is there19:20
melwittwell, shouldn't upstream libvirt be backporting like we do for stable? or do they not do that19:21
sean-k-mooneymelwitt: upstream libvirt does not do backports19:21
stephenfinI left some words. Hopefully they help19:21
* stephenfin goes home19:22
sean-k-mooneymelwitt: which is why each os vendor needs to do it downstream which is annoying19:22
melwittdang19:22
*** amoralej is now known as amoralej|off19:24
*** spatel has quit IRC19:29
*** jangutter_ has quit IRC19:29
*** spatel has joined #openstack-nova19:29
melwittdansmith: I added you to this review bc it's updating code that has a NOTE with yo name on it https://review.opendev.org/70576419:35
*** kplant has quit IRC19:50
*** trident has quit IRC19:51
*** trident has joined #openstack-nova19:59
*** gmann_lunch is now known as gmann20:01
*** jamesdenton has quit IRC20:03
*** jamesdenton has joined #openstack-nova20:04
*** jangutter has joined #openstack-nova20:09
*** jangutter has quit IRC20:14
*** nweinber has quit IRC20:15
*** tbachman has quit IRC20:16
*** spatel has quit IRC20:27
*** TxGirlGeek has quit IRC20:32
*** ociuhandu has joined #openstack-nova20:42
*** ociuhandu has quit IRC21:02
*** ociuhandu has joined #openstack-nova21:03
*** ociuhandu has quit IRC21:08
*** N3l1x has quit IRC21:09
*** stampeder has joined #openstack-nova21:21
*** spatel has joined #openstack-nova21:31
*** mriedem has left #openstack-nova21:35
*** spatel has quit IRC21:36
*** stampeder has quit IRC21:37
*** ociuhandu has joined #openstack-nova21:41
*** xek has quit IRC21:45
*** CeeMac has quit IRC21:50
*** CeeMac has joined #openstack-nova21:56
openstackgerritmelanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929  https://review.opendev.org/70147821:58
openstackbug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/184492921:58
*** amoralej|off is now known as amoralej22:00
*** ociuhandu has quit IRC22:03
*** ociuhandu has joined #openstack-nova22:04
*** rcernin has joined #openstack-nova22:04
*** rcernin has quit IRC22:04
*** rcernin has joined #openstack-nova22:05
*** ociuhandu has quit IRC22:09
*** etp has quit IRC22:15
*** etp has joined #openstack-nova22:16
*** factor has quit IRC22:27
*** slaweq has quit IRC22:28
*** tbachman has joined #openstack-nova22:42
*** nicolasbock has quit IRC22:43
*** dave-mccowan has quit IRC22:50
*** tkajinam has joined #openstack-nova22:56
*** amoralej is now known as amoralej|off23:29
*** jangutter has joined #openstack-nova23:43
*** liuyulong has quit IRC23:47
*** jangutter has quit IRC23:47
*** threestrands has joined #openstack-nova23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!