Monday, 2021-01-11

« Sunday, 2021-01-10 Index Tuesday, 2021-01-12 »
*** pfire has joined #openstack-nova00:13
pfireresized a powered off instance, cannot start instance while it is in task state resize finish, cannot confirmresize while in vmstate stopped00:14
pfireshould a resize have been allowed in the first place00:15
*** prometheanfire has joined #openstack-nova00:22
*** prometheanfire has quit IRC00:24
*** prometheanfire has joined #openstack-nova00:27
prometheanfiregot around it by manually starting the instance, then resize could be confirmed (but instance showed up as stopped), and then once instance start was issued the instance was restarted (instead of the state being updated in the db)00:28
*** pfire has quit IRC00:30
*** sapd1 has quit IRC00:40
*** swp20 has joined #openstack-nova00:45
*** LinPeiWen has joined #openstack-nova01:23
*** brinzhang has joined #openstack-nova01:35
*** songwenping_ has joined #openstack-nova01:38
*** swp20 has quit IRC01:41
*** dklyle has joined #openstack-nova01:46
*** iurygregory has joined #openstack-nova02:07
*** rcernin has quit IRC02:59
*** rcernin has joined #openstack-nova03:08
*** mkrai has joined #openstack-nova03:08
*** rcernin has quit IRC03:19
*** sapd1 has joined #openstack-nova03:20
*** rcernin has joined #openstack-nova03:30
*** psachin has joined #openstack-nova03:33
*** hemanth_n has joined #openstack-nova03:44
*** vishalmanchanda has joined #openstack-nova04:00
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Fix nova live migration task failed due respone value changed  https://review.opendev.org/c/openstack/python-novaclient/+/76987205:39
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.88: Replace tenant* with project* in usage* commands  https://review.opendev.org/c/openstack/python-novaclient/+/76967505:39
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.88: Replace tenant* with project* in quota* commands  https://review.opendev.org/c/openstack/python-novaclient/+/76986905:39
openstackgerritBrin Zhang proposed openstack/python-novaclient master: DNM/TEST: function test  https://review.opendev.org/c/openstack/python-novaclient/+/77006005:39
*** macz_ has joined #openstack-nova05:44
*** macz_ has quit IRC05:48
*** sapd1 has quit IRC06:00
*** mkrai has quit IRC07:13
*** sapd1 has joined #openstack-nova07:14
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.88: Replace tenant* with project* in quota* commands  https://review.opendev.org/c/openstack/python-novaclient/+/76986907:33
openstackgerritHemanth N proposed openstack/nova stable/train: Update pci stat pools based on PCI device changes  https://review.opendev.org/c/openstack/nova/+/76172507:36
*** ralonsoh has joined #openstack-nova07:40
*** hoonetorg has joined #openstack-nova07:46
*** slaweq has joined #openstack-nova07:51
*** rcernin has quit IRC07:51
*** mkrai has joined #openstack-nova07:53
*** sapd1 has quit IRC08:04
*** tesseract has joined #openstack-nova08:10
*** andrewbonney has joined #openstack-nova08:13
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.88: Replace tenant* with project* in limits commands  https://review.opendev.org/c/openstack/python-novaclient/+/77006508:14
*** chengsheng1 has joined #openstack-nova08:14
*** sapd1 has joined #openstack-nova08:18
*** lee1 has joined #openstack-nova08:19
*** lee1 is now known as lyarwood08:19
gibigood morning nova08:19
*** xek has joined #openstack-nova08:21
chengsheng1Please help review the code: https://review.opendev.org/c/openstack/nova/+/762330/08:26
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.88: Replace tenant* with project* in usage* commands  https://review.opendev.org/c/openstack/python-novaclient/+/76967508:27
openstackgerritManik Sidana proposed openstack/nova master: Fix typo  https://review.opendev.org/c/openstack/nova/+/77006708:32
*** rpittau|afk is now known as rpittau08:33
openstackgerritManik Sidana proposed openstack/nova master: Fix typo in warning message  https://review.opendev.org/c/openstack/nova/+/77006708:36
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.88: Replace tenant* with project* in quota* commands  https://review.opendev.org/c/openstack/python-novaclient/+/76986908:38
*** ralonsoh has quit IRC08:38
*** ralonsoh has joined #openstack-nova08:41
brinzhanggibi: good morning, noVNC patch has updated, pls review again https://review.opendev.org/c/openstack/nova/+/62233608:44
gibibrinzhang: ack,08:45
*** lpetrut has joined #openstack-nova08:45
*** yoctozepto has quit IRC08:48
*** chengsheng1 has quit IRC08:51
*** chengsheng1 has joined #openstack-nova08:51
*** kevinz has quit IRC08:53
*** sapd1 has quit IRC09:16
*** ociuhandu has joined #openstack-nova09:18
*** zenkuro has joined #openstack-nova09:18
*** vishalmanchanda has quit IRC09:18
*** martinkennelly has joined #openstack-nova09:31
gibilyarwood: if you have time could you check a tempest failure like https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_637/764040/8/check/nova-live-migration/637f5c6/testr_results.html based on logstash we have similar failures pretty fequrently09:33
gibihttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%20to%20reach%20available%20status%20(current%20detaching)%20within%20the%20required%20time%5C%2209:33
lyarwoodgibi: oh fun, not this again09:43
*** ociuhandu has quit IRC09:43
gibimonday fun09:43
lyarwoodDevice detach failed for vdb: Unable to detach the device from the live config.09:44
*** derekh has joined #openstack-nova09:44
gibiI'm not sure that it is only that test case failing with the same symptome09:44
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.88: Replace tenant* with project* in limits commands  https://review.opendev.org/c/openstack/python-novaclient/+/77006509:52
*** ociuhandu has joined #openstack-nova09:53
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.88: Replace tenant* with project* in quota* commands  https://review.opendev.org/c/openstack/python-novaclient/+/76986909:55
lyarwoodgibi: melwitt wrote an ER query for this before the break in https://bugs.launchpad.net/nova/+bug/188252109:57
openstackLaunchpad bug 1882521 in OpenStack Compute (nova) ussuri "Failing device detachments on Focal" [Undecided,New]09:57
lyarwoodgibi / melwitt ; looks different to the previous hit within libvirtd logs at least, we only see the device_del commands from libvirtd to QEMU and not the DEVICE_DELETED events being emitted by QEMU.09:58
lyarwoodgibi / melwitt ; we can try increasing the initial wait again but that's another hackaround09:58
*** ociuhandu has quit IRC09:58
lyarwoodgibi / melwitt ; the ultimate fix for this is waiting for libvirtd to emit events when detaching devices https://review.opendev.org/c/openstack/nova/+/74992909:59
*** jangutter_ has joined #openstack-nova09:59
gibilyarwood: could I help with ^^ somehow?10:00
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.88: Replace tenant* with project* in limits commands  https://review.opendev.org/c/openstack/python-novaclient/+/77006510:01
lyarwoodgibi: I don't have any cycles to spend on it at the moment, feel free to take a look if you do, that's only a PoC change I was using to work out how to tap into the event framework within the libvirt driver10:01
gibilyarwood: ack, I might need more context but I will take a look at that patch10:02
*** jangutter has quit IRC10:03
lyarwoodgibi: https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainEventID & https://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainEventDeviceRemovedCallback for the libvirt specific docs10:03
*** ociuhandu has joined #openstack-nova10:09
*** ociuhandu has quit IRC10:14
*** ociuhandu has joined #openstack-nova10:15
*** jangutter_ has quit IRC10:17
*** jangutter has joined #openstack-nova10:20
gibilyarwood: was this https://review.opendev.org/c/openstack/nova/+/749827 superseeded by https://review.opendev.org/c/openstack/nova/+/755799 ?10:21
*** jangutter_ has joined #openstack-nova10:21
lyarwoodgibi: yeah sorry I'm not sure why I eneded up with a different change there10:22
gibilyarwood: OK I will drop that10:23
lyarwoodgibi: one thing I've just noticed, there's a 5 second gap between the first and second device_del request in libvirtd, my change was supposed to add a 10 second delay10:24
*** jangutter has quit IRC10:24
gibiI think on master we have inc_sleep_time=1010:25
lyarwoodyeah I think the retry decorator only uses that for the second wait, ./me checks10:26
*** jangutter_ is now known as jangutter10:26
*** ociuhandu has quit IRC10:29
*** ociuhandu has joined #openstack-nova10:30
gibiit seem so10:31
openstackgerritLucas Alvares Gomes proposed openstack/nova master: DO NOT REVIEW: Test OVN devstack module  https://review.opendev.org/c/openstack/nova/+/74822610:32
openstackgerritBalazs Gibizer proposed openstack/nova master: WIP/DNM libvirt: Start emitting DeviceRemovedEvent and DeviceRemovalFailedEvent events  https://review.opendev.org/c/openstack/nova/+/74992910:32
*** zenkuro has quit IRC10:33
*** zenkuro has joined #openstack-nova10:33
openstackgerritBalazs Gibizer proposed openstack/nova master: WIP/DNM libvirt: Start emitting DeviceRemovedEvent and DeviceRemovalFailedEvent events  https://review.opendev.org/c/openstack/nova/+/74992910:37
*** dtantsur|afk is now known as dtantsur10:37
openstackgerritBalazs Gibizer proposed openstack/nova master: WIP/DNM libvirt: Start emitting DeviceRemovedEvent and DeviceRemovalFailedEvent events  https://review.opendev.org/c/openstack/nova/+/74992910:40
*** mkrai has quit IRC10:54
*** nightmare_unreal has joined #openstack-nova10:56
*** avolkov has joined #openstack-nova11:07
*** ociuhandu has quit IRC11:18
*** ociuhandu has joined #openstack-nova11:23
*** ociuhandu has quit IRC11:28
*** ociuhandu has joined #openstack-nova11:30
*** zzzeek has quit IRC11:33
*** ociuhandu has quit IRC11:35
*** zzzeek has joined #openstack-nova11:35
*** ociuhandu has joined #openstack-nova11:43
stephenfingibi: Could you re +2 this, please? https://review.opendev.org/c/openstack/nova/+/769520/11:44
gibistephenfin: oops I missed that, done noew11:46
gibinow11:46
stephenfinAlso, gibi, lyarwood, sean-k-mooney: I addressed your concerns on that OSC change, I think https://review.opendev.org/c/openstack/python-openstackclient/+/762396/11:46
stephenfingibi++ thanks11:46
gibiI'm +1 on the osc patch too11:48
stephenfinhurrah11:49
*** rcernin has joined #openstack-nova11:51
*** ociuhandu has quit IRC11:51
*** raildo has joined #openstack-nova11:53
gibigmann: I still see tempest failing with "Multiple possible networks found, use a Network ID to be more specific." on master. e.g.: https://74ab9f21dcdd2e895607-6eadae60e9a1b03c6789f42d51c17d3a.ssl.cf2.rackcdn.com/756894/5/check/nova-next/1fb33ef/testr_results.html11:55
gibidansmith: I have two tempest failures in test_glance_direct_import_image_to_specific_stores[id-82fb131a-dd2b-11ea-aec7-340286b6c574] I tried to create a logstash signature for it but I failed11:56
gibi1) https://zuul.opendev.org/t/openstack/build/4a78ba5901474187a3bdc49cacf84b47/logs11:56
gibi2) https://zuul.opendev.org/t/openstack/build/9488788b59904e6f9bc1280f13fc0a6b/logs11:56
gibidansmith: is this the timeout failure you mentioned on the last week's meeting?11:57
sean-k-mooneystephenfin: +1 also11:57
gibidansmith: I also see a full nova-ceph-multistore job timeout here https://zuul.opendev.org/t/openstack/build/8807547b038b4dd983fd7e0bb4cb4b40 it might be related too12:02
*** ociuhandu has joined #openstack-nova12:05
*** hemanth_n has quit IRC12:06
*** hemanth_n has joined #openstack-nova12:07
*** mgariepy has quit IRC12:09
*** ociuhandu has quit IRC12:11
*** dviroel has joined #openstack-nova12:11
*** zzzeek has quit IRC12:19
*** rcernin has quit IRC12:21
*** zzzeek has joined #openstack-nova12:21
*** Oriz has joined #openstack-nova12:23
*** ociuhandu has joined #openstack-nova12:24
*** songwenping_ has quit IRC12:31
*** songwenping_ has joined #openstack-nova12:32
*** ociuhandu has quit IRC12:37
*** Luzi has joined #openstack-nova12:56
*** ociuhandu has joined #openstack-nova13:10
*** mgariepy has joined #openstack-nova13:13
*** ociuhandu has quit IRC13:14
*** ociuhandu_ has joined #openstack-nova13:14
*** artom has joined #openstack-nova13:21
*** ociuhandu_ has quit IRC13:22
*** lemko has quit IRC13:24
*** lemko has joined #openstack-nova13:24
*** lemko has quit IRC13:34
*** lemko has joined #openstack-nova13:35
*** ociuhandu has joined #openstack-nova13:35
*** hemanth_n has quit IRC13:36
*** ociuhandu has quit IRC13:40
*** ociuhandu has joined #openstack-nova13:40
*** ociuhandu has quit IRC13:40
*** ociuhandu has joined #openstack-nova13:41
*** ociuhandu has quit IRC13:41
*** yoctozepto has joined #openstack-nova13:45
openstackgerritBalazs Gibizer proposed openstack/nova master: Refactor ResourceRequest constructor  https://review.opendev.org/c/openstack/nova/+/76972013:45
openstackgerritBalazs Gibizer proposed openstack/nova master: Support interface attach with QoS ports  https://review.opendev.org/c/openstack/nova/+/75653013:46
*** ociuhandu has joined #openstack-nova13:47
openstackgerritBalazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/report.py  https://review.opendev.org/c/openstack/nova/+/76515213:48
gibistephenfin: fix your comment in the qos series ^^13:48
*** tbachman has quit IRC13:50
*** tbachman has joined #openstack-nova13:52
*** ralonsoh_ has joined #openstack-nova13:59
*** ralonsoh has quit IRC13:59
*** sapd1 has joined #openstack-nova14:00
*** ralonsoh_ is now known as ralonsoh14:01
*** Luzi has quit IRC14:04
openstackgerritBalazs Gibizer proposed openstack/nova stable/victoria: Add upgrade check about old computes  https://review.opendev.org/c/openstack/nova/+/76192414:05
stephenfingibi: Done. Think I'm +2 on that whole series. I'll leave it to you  to find another reviewer14:06
stephenfinsean-k-mooney would probably be good to have reviewing it since I think they're re-using part of it for another spec?14:07
gibistephenfin: thanks14:07
sean-k-mooneystephenfin: am the routed networks stuff bauzas is doing?14:12
stephenfinsean-k-mooney: no, gibi's "attach ports with QoS policies" series14:12
sean-k-mooneystephenfin: the only part im reusing form that is the network info in the requst spec14:12
stephenfinoh, it's something from bauzas' series you're reusing. nvm so14:13
sean-k-mooneyoh no i dont need that for anything as far as i am aware but i can review it in anycase14:13
bauzasatm working on adding UTs for the WIP patch, btw. ;)14:13
sean-k-mooneybauzas: did you see my comments on it14:13
sean-k-mooneythe functional tests are asserting incorrect behavior14:13
bauzassean-k-mooney: err, no14:14
sean-k-mooneylook at the move operation tests https://review.opendev.org/c/openstack/nova/+/749068/6/nova/tests/functional/test_servers.py14:15
sean-k-mooneythe test have 3 host and 2 segments14:15
sean-k-mooneythe first host is not on any segemnt and the other two are on one each14:15
sean-k-mooneyso the migrate live migrate and evacuate test should not pass14:15
bauzasyup14:15
bauzaswhat?14:16
sean-k-mooneysince there is not another host on the same segment14:16
sean-k-mooneyyou cant change segments on any move operation14:16
bauzashmmm14:16
bauzasthen the implementation is not good14:16
bauzasI asked for it last month14:16
sean-k-mooneyunfortunetly it appears so14:16
bauzasand I thought we were supporting to move to other segments if they are in the same networks14:17
sean-k-mooneyno that would change the ip14:17
sean-k-mooneysegment are mapped to subnets14:17
bauzaslook at https://review.opendev.org/c/openstack/nova/+/749068/6/nova/scheduler/request_filter.py#31614:17
sean-k-mooneyso changing a the segment changes the subnet14:17
bauzaswe get the all aggregates for the same network14:17
sean-k-mooneyyes that is for boot14:18
sean-k-mooneyfor move opertation you have to get the aggreate for the current segment14:18
bauzasokay, but then for moves, we should only ask for the existed aggregate then14:18
bauzashmmmm, ok14:18
bauzasthen we need to change this14:18
bauzasthere are no ways to have different behaviours for request filters14:19
sean-k-mooneyya sorry for not seeing that before. we allso will need to add a 4th host that is connected to the same segment as host214:19
sean-k-mooneybauzas: well if the port has an ip then you know its a move operation or they specified an ip14:20
sean-k-mooneyin both  cases you need to limit the aggreates to the segment corresponing to that ip14:20
bauzassean-k-mooney: aaaaaah yeah14:20
bauzasL310 then14:20
sean-k-mooneyso that can be your check i belive14:20
bauzasyeah okay14:20
bauzasthen I need to know how to get the segment for the port14:21
bauzaslemme look if we have it14:21
sean-k-mooneyyou get the via the subnet the ip is from14:21
sean-k-mooneythen you can get the segment form the subnet14:21
bauzasmmmm14:21
sean-k-mooneyi belive the port will have a subnet id in it14:22
bauzashopefully yeah IIRC14:22
gmanngibi: ack, will check today14:22
gibithanks14:22
gmannscenario tests should not have this error as Tempest explicitly pass the net id but I will debug if something else causing this or we miss some tests14:23
sean-k-mooneybauzas: let me know if you have any issues with that and i can look into that more closely if needed14:23
* bauzas looks at https://docs.openstack.org/api-ref/network/v2/index.html?expanded=list-subnets-detail#subnets14:24
bauzasto know whether we can the segment from the subnet14:25
bauzasas we already get the one from the network14:25
*** dave-mccowan has quit IRC14:25
bauzasactually wait14:25
bauzassean-k-mooney: I don't think we need to call Neutron for this14:25
sean-k-mooneyhttps://docs.openstack.org/api-ref/network/v2/index.html?expanded=list-subnets-detail#id26314:25
sean-k-mooneysegment_id is there14:26
bauzassean-k-mooney: we could just get the network related to the subnet and lookup all the segments in this to match the related subnet14:26
bauzassee my point ?14:26
sean-k-mooneyyou could but that is more calls to neutron14:26
bauzasno14:27
bauzashttps://review.opendev.org/c/openstack/nova/+/749068/6/nova/network/neutron.py14:27
bauzaswe already get all the segments14:27
sean-k-mooneyah right14:27
bauzasin https://review.opendev.org/c/openstack/nova/+/749068/6/nova/scheduler/utils.py we ask the network14:27
sean-k-mooneyso just filter them14:27
bauzasso we get all the segments14:28
bauzasand then we filter them14:28
bauzasyeah14:28
*** dave-mccowan has joined #openstack-nova14:28
bauzasokay, I think it's simple to fix then14:28
sean-k-mooneyunfortunetly the segment dont have the subnet id14:28
bauzaswe have the subnet name14:29
bauzasand the id actually14:29
sean-k-mooneyit does not have that eitehr14:29
sean-k-mooneyhttps://docs.openstack.org/api-ref/network/v2/index.html?expanded=show-segment-details-detail#show-segment-details14:29
bauzashttps://docs.openstack.org/api-ref/network/v2/index.html?expanded=list-subnets-detail#id264 is what we get14:30
*** nweinber has joined #openstack-nova14:30
bauzaswhen calling Neutron for a list of segments related to the network14:30
bauzasand I can see both the name and id fields14:30
bauzasbecause the object is the subnet, not the segment14:30
sean-k-mooneyright but right now we dont list all the subnets14:31
sean-k-mooneywe just get all the segment by the network id14:31
sean-k-mooneyso we need to have 1 call to get teh subnet14:31
sean-k-mooneyit has the segment id in it14:31
bauzasoh shit, you're right14:32
sean-k-mooneyso instead of calling _get_segment_ids_for_network14:32
bauzasI was looking at the wrong endpoint14:32
sean-k-mooneywe just need get_segment_id_for_port14:32
sean-k-mooneythat uses the subnet_id in the port to do a subnet show14:32
bauzasyeah we can't filter14:32
sean-k-mooneyand pull out the id14:32
bauzasok I can try to fix this14:33
bauzasanyway, good catch14:33
sean-k-mooneyya its still just one call to neutron either gettign the segment by network or if it has an ip geting the segment for the current ip14:33
sean-k-mooneybauzas: fyi the subnet in the port is in port[fixed_ips][0]['subnet_id']14:35
sean-k-mooneywell port['fixed_ips'][0]['subnet_id'] for got the ''14:35
bauzasyeah14:36
bauzasI'm literrally hands-on :)14:36
bauzasdid a git stash for the UT work I was doing14:36
bauzasand I'll have to change a few things14:36
bauzasfortunately, you pinged me before I went too far14:37
*** ralonsoh has quit IRC14:37
bauzasbehavioural-driven-development, that is.14:37
*** ralonsoh has joined #openstack-nova14:38
*** tosky has joined #openstack-nova14:39
*** ralonsoh has quit IRC14:40
bauzassean-k-mooney: can we assume a port always have a fixed-ips list with ip addresses in it ?14:42
bauzasI can be conservative and double-check we have a list of IPs, and if not, fallback to getting the network id14:43
bauzasas we could boot with an unbound port14:43
bauzasright ?14:43
sean-k-mooneybauzas: no i t wont have a fix ip if it was a new boot14:45
sean-k-mooneyit will if it has an ip e.g. a move operation or they specified one14:45
bauzasokay, so we need to be conservative14:45
sean-k-mooneyyep but if it has no fixed ip14:45
bauzassure, sec14:45
*** ociuhandu has quit IRC14:45
sean-k-mooneythen any segment in the network is valid14:45
sean-k-mooneyso you can use the lent of the list to determin how to look it up14:46
*** ociuhandu has joined #openstack-nova14:46
sean-k-mooneye.g. if port['fixed_ips']: lookup_by_port_subnet else lookup_by_network14:46
bauzasthat's exactly what I wrotre14:48
bauzaspasting it14:48
bauzashmpf, paste.o.o is super slow14:48
sean-k-mooneyya it is weird14:50
bauzassean-k-mooney: https://pastebin.com/x0g0anRS14:50
bauzasagreed with the logic ?14:50
bauzasif so, adding the new neutron knob14:50
bauzas(nevermind the pep8 indenting issue, fwiw :) )14:51
sean-k-mooneyya that works14:51
sean-k-mooneyalthough i kind of feel like lines 17-22 should be under the if on 1314:52
sean-k-mooneywell if/else14:52
sean-k-mooneyi mean you dont need to do the if network_id as that will always  be true14:52
sean-k-mooneyyou could keep the if subnet_id but that will only be set if we go to line 14 in which case we know it wont be empty14:53
bauzasyeah, I can improve this14:54
sean-k-mooneyso you can put lines 21 and 22 on line 15 and lines 18 and 19 on line 1714:54
sean-k-mooneybut ya the logic looks correct to me14:54
*** spotz has joined #openstack-nova14:57
*** lpetrut has quit IRC15:10
*** ralonsoh has joined #openstack-nova15:16
openstackgerritBalazs Gibizer proposed openstack/nova stable/victoria: Reproduce bug 1907522 in functional test  https://review.opendev.org/c/openstack/nova/+/76973615:23
openstackbug 1907522 in OpenStack Compute (nova) "test_*_with_qos_min_bw_allocation fails in the nova-multi-cell job with: nova.exception.MigrationPreCheckError: Migration pre-check error: Failed to create port bindings for host <host>" [High,In progress] https://launchpad.net/bugs/1907522 - Assigned to Balazs Gibizer (balazs-gibizer)15:23
openstackgerritBalazs Gibizer proposed openstack/nova stable/victoria: Fallback to same-cell resize with qos ports  https://review.opendev.org/c/openstack/nova/+/76972715:24
*** mgariepy has quit IRC15:30
*** martinkennelly has quit IRC15:38
*** brinzhang has quit IRC15:55
*** dklyle has quit IRC15:55
*** dklyle has joined #openstack-nova15:55
*** macz_ has joined #openstack-nova15:59
*** mgariepy has joined #openstack-nova16:06
lyarwooddansmith: https://review.opendev.org/c/openstack/nova-specs/+/769547 - would you mind hitting PS3 today? Should be good to go now.16:10
dansmithroger16:11
lyarwoodcheers16:11
openstackgerritDan Smith proposed openstack/nova-specs master: libvirt: Update instance machine type stash spec  https://review.opendev.org/c/openstack/nova-specs/+/76954716:15
*** ociuhandu has quit IRC16:16
*** zenkuro has quit IRC16:17
*** zenkuro has joined #openstack-nova16:17
*** ociuhandu has joined #openstack-nova16:23
*** efried has quit IRC16:29
*** kashyap has joined #openstack-nova16:31
*** Oriz has quit IRC16:32
*** psachin has quit IRC16:33
openstackgerritMerged openstack/nova-specs master: libvirt: Update instance machine type stash spec  https://review.opendev.org/c/openstack/nova-specs/+/76954716:38
*** efried has joined #openstack-nova16:38
gibilyarwood: I'm looking at detach_device_with_retry() codepath detach_device_with_retry16:39
gibilyarwood: and wondering why we have the code set up in a way that we always call an intial detach then after it a detach in a loop16:40
*** mlavalle has quit IRC16:41
gibilyarwood: also, I'm wondering what is the way _do_wait_and_retry_detach() normally exit the tool16:41
gibiloop16:41
*** mlavalle has joined #openstack-nova16:41
gibiis it raises DeviceDetachFailed to signal that the detach is done?16:42
lyarwoodgibi: just in a call16:42
*** sapd1 has quit IRC16:43
gibino worries, I will drop soon, so we can talk about it tomorrow morning16:43
lyarwoodgibi: ack if we could that would be great16:43
*** sapd1 has joined #openstack-nova16:44
gibiOK16:46
melwittgibi: IIRC it's because the initial detach "should" detach it from persistent + live config but the guest can refuse the live detach as it's an ACPI request. so we retry in case that happens (guest refusal by it's busy). under nominal conditions I think the loop detects it's detached in the first iteration and moves on16:48
melwitt*bc it's busy16:48
gibimelwitt: thanks.16:51
gibimelwitt: both the initial and the retry loop calls _try_detach_device() _try_detach_device16:52
gibihttps://github.com/openstack/nova/blob/745f835e9d8931261501747130165824334abb4b/nova/virt/libvirt/guest.py#L40416:52
gibimelwitt: so if the first all to _try_detach_device() finishes without exception then when we call the retry loop it calls _try_detach_device() again that will detach again16:53
gibi* first call16:54
melwittyeah, IIRC the second call will be a no-op if the guest detached the live config the first call16:55
*** gyee has joined #openstack-nova16:55
melwittif the first call did not result in detach of the live config, it will loop N times to keep trying16:55
gibimelwitt: ohh, is it because we have the condition in the loop           config = get_device_conf_func(device)16:55
gibi            if config is not None:16:56
gibi           config = get_device_conf_func(device)16:56
gibi            if config is not None:16:56
gibi sorry16:56
gibiback copy paste16:56
gibihere https://github.com/openstack/nova/blob/745f835e9d8931261501747130165824334abb4b/nova/virt/libvirt/guest.py#L47616:56
melwittmmmaybe. I think that only checks whether the persistent config got detached, not the live16:57
melwittI think that only checks whether the persistent detach succeeded and so it will go ahead and try the live detach and get DeviceNotFound17:00
lyarwoodmelwitt++17:00
*** ociuhandu_ has joined #openstack-nova17:02
melwittand DeviceNotFound gets ignored as a no-op17:03
gibimelwitt: what is the exit condition of _do_wait_and_retry_detach() ? the retry decorator retries on DeviceDetachFailed exception and if _try_detach_device() succeed without exception then _do_wait_and_retry_detach() raises a DeviceNotFound so that also leads to retry17:03
gibiso the exit condition must be that the config become None17:05
*** ociuhandu has quit IRC17:05
*** ociuhandu_ has quit IRC17:07
melwitthmm.. yeah I see what you're saying. I hadn't thought the detach from live would be reflected in that get_disk but maybe it is17:08
melwittgibi: (it seems like it has to be, like you said)17:09
gibiohh we have two different exceptions in the control flow DeviceDetachFailed and DeviceNotFound, I think I mixed the two17:10
*** rpittau is now known as rpittau|afk17:10
gibifor the initial detach DeviceNotFound means we failed the detaching17:11
gibiin one of the domains17:11
*** tesseract has quit IRC17:11
gibiand hence we move to retry17:12
gibibut in the retry loop we only retry on DeviceDetachFailed not on DeviceNotFound17:12
*** nightmare_unreal has quit IRC17:12
melwittgibi: ahhhh yes you got it17:12
gibiso the retry loop does not care about DeviceNotFound (partial detach failure) but only cares about the config being not None17:13
melwittthat sneaky RetryDecorator17:13
gibithis code is hard17:13
*** haleyb has quit IRC17:14
melwittyes. this is exactly why lyarwood has plans to refactor it :)17:14
gibibut then if the retry loop sees any exception (other than the self inflicted DeviceDetachFailed) e.g. DeviceNotFound then it just let the exception buble up and not retry. So I don't know how this really retries a partial detach failure in libvirt17:15
gibimelwitt: yeah, and I try to help him, hence my questions :)17:15
gibi(no the exceptions are not inherinting from each other)17:16
lyarwoodFWIW with an event based approach we would just ask libvirt to detach once and then wait17:16
*** haleyb has joined #openstack-nova17:16
gibilyarwood: and if if sends a failed event, we call detach again?17:16
lyarwoodtrue yeah we could but it should be easier to write and maintain than this mess17:17
gibilyarwood: or we consider the DeviceRemovalFailedEvent as a permanent failure that we buble up?17:18
gibis/than/and/17:18
gibinvm17:18
gibithat is end of my day. I will continue looking into this tomorrow17:18
gibimelwitt, lyarwood: thanks for the help17:18
*** tosky has quit IRC17:24
*** tosky has joined #openstack-nova17:25
*** martinkennelly has joined #openstack-nova17:26
melwittit seems like get_device_conf_func has to be reflecting the live config too, else this doesn't make sense17:31
*** ociuhandu has joined #openstack-nova17:36
*** openstackgerrit has quit IRC17:37
*** ociuhandu has quit IRC17:41
*** dklyle has quit IRC17:47
*** dklyle has joined #openstack-nova18:01
*** derekh has quit IRC18:05
*** mgoddard has quit IRC18:14
*** mgoddard has joined #openstack-nova18:15
*** hamalq has joined #openstack-nova18:16
*** mgoddard has quit IRC18:24
gmanngibi: melwitt this should fix the multiple-network issue in nova-next job occurring in tempest scenario manager.https://review.opendev.org/c/openstack/tempest/+/77016918:39
gmann Previously i thought it cannot happen in scenario manager but if 'CONF.network.port_vnic_type' and 'CONF.network.port_profile' are not configured then it can occur which is what we saw in nova-next job18:39
melwittwoot, thanks gmann18:40
gmannthis is where it falling on tenant default network which are note present https://zuul.opendev.org/t/openstack/build/ce402610f88a45928ff808bbb4a6303f/log/controller/logs/tempest_log.txt#5343918:41
gmannnot18:41
*** andrewbonney has quit IRC18:43
melwittI vaguely remember trying to fix it with https://review.opendev.org/c/openstack/tempest/+/716809 a long time ago18:43
gmannmelwitt: ah i remember now that discussion. base class also need to setup the resources.18:45
gmannin my patch I am creating default network resource by default for scenario tests and provide flag for subclass to stop that if needed18:46
gmanncreating by default should be right thing for scenario testing18:46
melwittyeah, I see. cool18:46
*** _mlavalle_1 has joined #openstack-nova19:05
*** mlavalle has quit IRC19:08
*** dtantsur is now known as dtantsur|afk19:15
*** nweinber has quit IRC19:17
*** gmann is now known as gmann_afk19:25
*** nweinber has joined #openstack-nova20:00
*** tobias-urdin has joined #openstack-nova20:17
*** avolkov has quit IRC20:36
*** elod has quit IRC20:43
*** ralonsoh has quit IRC20:44
*** elod has joined #openstack-nova20:45
*** zenkuro has quit IRC20:51
*** zenkuro has joined #openstack-nova20:52
*** eharney has quit IRC20:58
*** ociuhandu has joined #openstack-nova21:01
*** eharney has joined #openstack-nova21:10
*** gmann_afk is now known as gmann21:13
*** zenkuro has quit IRC21:19
*** zenkuro has joined #openstack-nova21:19
*** lemko has quit IRC21:21
*** lemko has joined #openstack-nova21:21
*** hoonetorg has quit IRC21:22
*** openstackgerrit has joined #openstack-nova21:32
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers  https://review.opendev.org/c/openstack/placement/+/76024021:32
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for aggregates  https://review.opendev.org/c/openstack/placement/+/76023521:32
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for allocations  https://review.opendev.org/c/openstack/placement/+/76023621:32
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for allocation candidates  https://review.opendev.org/c/openstack/placement/+/76023721:32
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for inventories  https://review.opendev.org/c/openstack/placement/+/76023821:32
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource classes  https://review.opendev.org/c/openstack/placement/+/76023921:32
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for traits  https://review.opendev.org/c/openstack/placement/+/76024121:32
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for usage  https://review.opendev.org/c/openstack/placement/+/76024221:32
*** ociuhandu has quit IRC21:36
*** hoonetorg has joined #openstack-nova21:40
*** raildo has quit IRC21:43
*** nweinber has quit IRC21:44
*** xek has quit IRC21:53
*** zenkuro has quit IRC22:05
*** zenkuro has joined #openstack-nova22:06
*** rcernin has joined #openstack-nova22:07
*** rcernin has quit IRC22:09
*** rcernin has joined #openstack-nova22:11
openstackgerritMerged openstack/nova master: api: Drop statistics-style fields from os-hypervisors  https://review.opendev.org/c/openstack/nova/+/76404022:19
*** zenkuro has quit IRC22:49
*** zenkuro has joined #openstack-nova22:50
*** slaweq has quit IRC22:51
*** tkajinam has joined #openstack-nova22:57
*** zenkuro has quit IRC23:06
*** zenkuro has joined #openstack-nova23:06
*** rcernin_ has joined #openstack-nova23:54
*** rcernin has quit IRC23:55
« Sunday, 2021-01-10 Index Tuesday, 2021-01-12 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!