Monday, 2021-01-18

*** tosky has quit IRC00:03
*** jangutter has quit IRC00:06
*** brinzhang has joined #openstack-nova00:17
*** jangutter has joined #openstack-nova00:23
*** LinPeiWen86 has joined #openstack-nova00:40
*** zzzeek has quit IRC00:44
*** zzzeek has joined #openstack-nova00:46
*** macz_ has joined #openstack-nova01:24
*** macz_ has quit IRC01:28
*** zoharm1 has joined #openstack-nova01:29
*** zoharm has quit IRC01:33
openstackgerritKeigo Noha proposed openstack/nova stable/rocky: Change default num_retries for glance to 3  https://review.opendev.org/c/openstack/nova/+/77112002:06
*** zenkuro has quit IRC02:17
*** benj_ has quit IRC02:18
*** benj_ has joined #openstack-nova02:19
*** LinPeiWen86 has quit IRC02:21
*** tobberydberg has quit IRC02:23
*** tobberydberg has joined #openstack-nova02:35
*** rcernin has quit IRC02:41
*** rcernin has joined #openstack-nova02:58
*** rcernin has quit IRC03:03
*** rcernin has joined #openstack-nova03:03
*** mkrai has joined #openstack-nova03:06
*** zoharm has joined #openstack-nova03:10
*** zoharm1 has quit IRC03:12
*** jangutter_ has joined #openstack-nova03:20
*** zzzeek has quit IRC03:22
*** jangutter has quit IRC03:22
*** zzzeek has joined #openstack-nova03:23
*** psachin has joined #openstack-nova03:33
*** links has joined #openstack-nova04:27
*** ratailor has joined #openstack-nova04:39
*** LinPeiWen1 has joined #openstack-nova04:49
*** Mamduhala has joined #openstack-nova04:54
*** vishalmanchanda has joined #openstack-nova05:10
*** zzzeek has quit IRC05:10
*** zzzeek has joined #openstack-nova05:14
*** psachin has quit IRC05:18
*** psachin has joined #openstack-nova05:22
*** bbowen has quit IRC05:54
*** bbowen_ has joined #openstack-nova05:54
*** Mamduhala has quit IRC06:26
openstackgerritAdit Sarfaty proposed openstack/nova master: Retry on vmware create_vm when it fails  https://review.opendev.org/c/openstack/nova/+/76458607:04
*** lpetrut has joined #openstack-nova07:14
*** mkrai has quit IRC07:20
*** CeeMac has joined #openstack-nova07:22
*** ralonsoh has joined #openstack-nova07:28
*** manuvakery1 has joined #openstack-nova07:41
*** lpetrut_ has joined #openstack-nova07:45
*** lpetrut_ has quit IRC07:45
*** lpetrut_ has joined #openstack-nova07:46
*** lpetrut has quit IRC07:48
*** danpawlik5 has quit IRC07:51
*** dpawlik7 has joined #openstack-nova07:53
*** ociuhandu has joined #openstack-nova07:53
*** rcernin has quit IRC07:53
*** rcernin has joined #openstack-nova07:57
*** ociuhandu has quit IRC07:58
*** rpittau|afk is now known as rpittau07:59
*** hemanth_n has joined #openstack-nova07:59
*** slaweq has joined #openstack-nova08:04
*** tesseract has joined #openstack-nova08:07
*** andrewbonney has joined #openstack-nova08:16
*** mkrai has joined #openstack-nova08:23
*** dtantsur|afk is now known as dtantsur08:32
*** tosky has joined #openstack-nova08:44
*** ociuhandu has joined #openstack-nova09:01
*** rcernin has quit IRC09:13
openstackgerritLee Yarwood proposed openstack/nova master: api: Log os-resetState as an instance action  https://review.opendev.org/c/openstack/nova/+/77092609:20
lyarwood^ stephenfin should be good now, there was however a weird but unrelated grenade failure in the last run09:20
lyarwoodhttps://bugs.launchpad.net/tempest/+bug/191217809:20
openstackLaunchpad bug 1912178 in tempest "Links are not allowed as constraints (pip 20.2.4)" [Undecided,New]09:20
stephenfinack09:20
*** rcernin has joined #openstack-nova09:24
*** derekh has joined #openstack-nova09:38
*** martinkennelly has joined #openstack-nova09:53
*** ociuhandu has quit IRC10:00
*** ociuhandu has joined #openstack-nova10:01
*** ociuhandu has quit IRC10:01
*** ociuhandu has joined #openstack-nova10:01
lyarwooddid we depreacte xen support in the libvirt driver?10:08
* lyarwood is currently dealing with the min version fallout and the horror that is nova.tests.unit.virt.libvirt.test_driver.LibvirtConnTestCase._check_xml_and_uri10:08
lyarwoodhttps://review.opendev.org/c/openstack/nova/+/749055 yup10:10
lyarwoodstephenfin: you didn't have a series to remove xen already did you?10:10
stephenfinfrom libvirt?10:10
stephenfinI did, yeah10:10
lyarwoodah where? I only see the xenapi stuff10:11
* lyarwood looks harder10:11
stephenfinhttps://review.opendev.org/q/topic:%22bp%252Fremove-deprecated-libvirt-virt-types%22+(status:open%20OR%20status:merged)10:11
lyarwood*facepalm* sorry the same topic, of course!10:11
lyarwoodI'll rebase my version bump on that series and give it a review10:12
*** mkrai has quit IRC10:33
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING  https://review.opendev.org/c/openstack/nova/+/75470210:36
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Drop support for UML  https://review.opendev.org/c/openstack/nova/+/74323010:36
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Drop support for Xen  https://review.opendev.org/c/openstack/nova/+/74323110:36
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION  https://review.opendev.org/c/openstack/nova/+/75470010:36
lyarwoodokay done ^ gibi I've put the MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING removal patch at the base of that series so we don't hold up your device detach rewrite10:36
*** ociuhandu has quit IRC10:53
*** ociuhandu has joined #openstack-nova10:53
*** ociuhandu has quit IRC11:00
*** ociuhandu has joined #openstack-nova11:01
*** ftarasenko has joined #openstack-nova11:03
*** dpawlik7 has quit IRC11:06
*** ociuhandu has quit IRC11:06
*** ociuhandu has joined #openstack-nova11:07
*** dpawlik7 has joined #openstack-nova11:09
*** songwenping__ has joined #openstack-nova11:10
gibilyarwood: thanks. I rebased my effort top of that11:11
gibilyarwood: btw, the event handling patch seems to work well on the gate, I still have to add some unit tests and I will consider splitting the patch as it is getting huge11:11
*** ociuhandu has quit IRC11:13
lyarwoodgibi: ack cool, it's on my list to review later today either way11:13
openstackgerritLee Yarwood proposed openstack/nova stable/train: trivial: Resolve (most) flake8 3.x issues  https://review.opendev.org/c/openstack/nova/+/77094311:14
openstackgerritLee Yarwood proposed openstack/nova stable/train: Switch to hacking 2.x  https://review.opendev.org/c/openstack/nova/+/77094411:14
gibilyarwood: have you heard back from libvirt folks about the timeout length in libvirt?11:14
lyarwoodelod: ^ not sure how you feel about this btw11:14
lyarwoodgibi: oh I didn't reach out to them after our discussion last week as it looks like the failure case is sync and doesn't have a timeout11:15
lyarwoodgibi: I can confirm with them if you'd like11:15
songwenping__sean-k-mooney, gibi, bauzas: hi guys, i review your chat record, do you means when the host is power off, we should set arqs to UNKNOWN state through cyborg-api; when the host is power on and during the cyborg-agent init, we should set arqs to PROVISIONING and create mdevs, and then set them to BOUND when the mdevs created.11:15
gibilyarwood: I trust your judgement. I misunderstood you previously about the timeout then, I though we have sync and async case as well11:15
lyarwoodkashyap: ^ not sure if you'd know, basically we are trying to work out if libvirt has an internal timeout before it emits the DeviceRemovalFailedEvent event11:16
openstackgerritYumengBao proposed openstack/nova-specs master: Libvirt supports composing cyborg owned vGPU accelerator into domain XML  https://review.opendev.org/c/openstack/nova-specs/+/75011611:16
kashyaplyarwood: Hi, incidentally, just reading here.  :-)11:16
gibibut then I we got back to the point where we don't know how long to wait for the event. if we timeout too soon then the retry will cause the same issue in libvirt as before11:16
lyarwoodkashyap: AFAICT it doesn't and would just emit that event synchronously on a failure11:16
*** ociuhandu has joined #openstack-nova11:17
lyarwoodgibi: but I think it's safer to wait for longer now11:17
kashyaplyarwood: Hmm, what's the exact event name?  This?  VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED11:17
gibikashyap: yes11:17
lyarwoodkashyap: but the failure one11:18
* lyarwood checks11:18
kashyapRight; just trying to find the precise name11:18
lyarwoodVIR_DOMAIN_EVENT_ID_DEVICE_REMOVAL_FAILED11:18
gibilyarwood: we can wait longer yes, I can bump the current 60sec to longer and see what happens on the gate. current I see some timeout in tempest with 60sec (then retry succeeds) so we can at least see if the timeout case go away if I bump the timeout value in the driver11:18
kashyaplyarwood: Okay; Peter Krempa / Michal from upstream libvirt might know; let me check on OFTC11:19
lyarwoodgibi: oh it doesn't need to be 60 seconds between attempts11:19
lyarwoodgibi: our previous issues were caused with a 5 second retry or lower iirc11:20
gibilyarwood: it is 60sec now and still see one or two timeout in a tempest run11:20
lyarwoodgibi: not between requests11:20
lyarwoodgibi: it builds up, the first is 5 seconds11:20
openstackgerritsean mooney proposed openstack/nova-specs master: add port scoped numa affinity spec  https://review.opendev.org/c/openstack/nova-specs/+/76590111:21
kashyaplyarwood: gibi: So, from the libvirt ocmmit that introduced the said event, it says the following:11:21
kashyap[quote]11:21
kashyapSince we didn't opt to use one single event for device lifecycle for a11:21
kashyapVM we are missing one last event if the device removal failed. This11:21
kashyapevent will be emitted once we asked to eject the device but for some11:21
kashyapreason it is not possible.11:21
kashyap[/quote]11:21
lyarwoodright, I've also looked at the code and it looks like it's just a direct synchronous failure with that call, IOW it doesn't come after a timeout.11:22
kashyapLooking at the code, I don't see particularly any timeout: https://www.redhat.com/archives/libvir-list/2016-April/msg00196.html11:22
kashyapJinx11:22
kashyapBut let me confirm w/ Peter11:22
gibilyarwood: but then who the event base code can help compared to the current retry code. that retry also wait syncronosly for libvirt to return11:23
gibiif the event also sync then waiting for the event is equivalent to wait for the detach call to return from libvirt11:23
gibior do I miss something?11:23
*** tbachman has quit IRC11:24
gibiI can verify something from the logs. I can check if there is a case when the detach returns before and then the detach event arrives significantly later11:26
gibiif there is such case then waiting for the event make sense11:26
kashyapgibi: Can you give a link to your patch, again, please?11:26
gibihttps://review.opendev.org/c/openstack/nova/+/77024611:26
kashyapThx11:26
*** xarlos has joined #openstack-nova11:32
kashyapgibi: lyarwood: So ... it indeed looks like a sync failure; danpb on #virt also says this: that event is issued when ACPI refuses the unplug explicitly.11:33
kashyapgibi: lyarwood: But if the guest OS is just slow or not responding, we don't get that event.11:34
*** Yumeng has joined #openstack-nova11:34
gibikashyap: so there is a case when we eventually need to time out an retry?11:34
*** ociuhandu has quit IRC11:34
gibithe case when the guest OS did not respond11:34
*** ociuhandu has joined #openstack-nova11:35
kashyapgibi: Yeah, looks liks so -- how else do we take into acount the the guest OS's slowness or unresponsiveness11:35
gibibut we should not retry too soon to avoid interrupting an ongoing detach in libvirt as that caused our issue in detach in the first place11:35
kashyapYeah, exactly that too...11:36
kashyapgibi: Also, wait:11:36
kashyapgibi: DanPB reminds: "if you get that event I don't htink you want to retry - that's an unconditional failure IIUC"11:36
gibikashyap: so if we get the failure event then we fail, if we get timeout we retry. if we get a sync failure about device missing then we retry with persistent=False11:38
Yumenghi sean-k-mooney, gibi, bauzas. Thank you for the discussion on Jan 15 about vGPU spec. I agree sean's suggestion about host reboot case and  updated a new patch . Pls help to review at your convenience.Thanks! https://review.opendev.org/c/openstack/nova-specs/+/75011611:39
lyarwoodgibi: yeah sorry had to go afk11:40
hemanth_nlyarwood: could you please review https://review.opendev.org/c/openstack/nova/+/761725 when you get some time11:40
lyarwoodhemanth_n: ack will do11:40
kashyapgibi: How does the sync failure of the missing device manifest?11:40
gibikashyap: https://review.opendev.org/c/openstack/nova/+/770246/7/nova/virt/libvirt/driver.py#4511:41
* kashyap clicks11:41
lyarwoodkashyap: if the device has already been removed from the persistent config but remains in the live config11:41
gibiwe get a libvirtError with VIR_ERR_DEVICE_MISSING11:41
gibias far as I see there is no failed event in this case11:41
lyarwoodgibi: yeah I don't think libvirt gets as far as calling QEMU11:42
lyarwoodgibi: for QEMU to then emit the event back to libvirt etc11:42
kashyapThx, both11:42
kashyapAh, looks like VIR_ERR_DEVICE_MISSING was added in libvirt for Nova :)11:44
gibiooh :)11:44
kashyapOtherwise, we were `grep`ing for error message; which is unreliable11:44
kashyapProper error code was due anyway; Nova just uncovered the bug11:45
*** xek has joined #openstack-nova11:45
gibiahh I know that. I thought there was some deeper reason. Yeas lyarwood just removed the grepping from nova as we have new enough libvirt to rely on the error code instead11:46
*** ociuhandu has quit IRC11:49
*** ociuhandu has joined #openstack-nova11:49
gibiLogs look good. I see cases where the detach sequence is the following:11:50
gibi* first attempt times out after 60sec waiting for the event from libvirt11:50
gibi* second attempt fails synchronously with device not found11:50
gibi* third attempt with persistent=False succeeds11:50
gibiso I assume the first attempt only removed the device from the persistent config11:51
gibihence the second attempt fail11:51
gibithen the third succeed as it does not try to remove it from the persistent config11:51
lyarwoodyeah the first should always remove it from the persistent config11:53
lyarwoodso you can call the second with persistent=False11:53
*** raildo has joined #openstack-nova11:53
lyarwoodas we did previously11:53
kashyapgibi: FWIW, I summarized the IRC chat here: https://review.opendev.org/c/openstack/nova/+/770246/7/nova/virt/libvirt/driver.py#233011:54
*** ociuhandu has quit IRC11:54
gibilyarwood: still that first timeout seems wrong, why not we get an event11:55
* kashyap --> lunch11:55
lyarwoodgibi: I think it's a single event once detached from both the persistent and live config11:55
gibilyarwood: and we need to send two detach to have detach it from both config?11:55
gibiif yes then it make sense11:56
*** jangutter has joined #openstack-nova11:56
*** jangutter has quit IRC11:56
*** jangutter has joined #openstack-nova11:57
*** jangutter_ has quit IRC11:59
*** zenkuro has joined #openstack-nova12:08
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove 'hypervisor_version' from 'libvirt_info'  https://review.opendev.org/c/openstack/nova/+/74419912:08
lyarwoodgibi: one request can detach the device from both12:13
lyarwoodgibi: it just depends on the guestOS for the device to be detached from the live config12:13
gibilyarwood: so the first detach attempt triggers the detach from both the persistent and the live config. The detach call returns, but we timeout waiting for the event. So I have to assume that either a) we didn't wait long enough to get the event b) we _have to_ send two detach to get the device detached from the live config.12:15
gibior c) libvirt does not send the event in some case12:16
gibibut that seems wrong12:16
lyarwoodright so for a and b we retry the detach12:17
lyarwoodwe can even check if the device is in either config before doing that12:17
lyarwoodbut it should always be gone from the persistent config at that point12:18
lyarwoodjust to go back to the original blind retry issue12:18
lyarwoodthe issue with that approach in CI was the time between actual detach calls to libvirt and then down into QEMU was tiny12:19
lyarwoodso much so that they would overlap and cause the undocumented QEMU behaviour12:19
lyarwoodwith your approach I really think it's safe to wait for longer between attempts to detach now, so something like 20 to 30 seconds12:20
*** ociuhandu has joined #openstack-nova12:20
lyarwoodI know you're waiting for 60 at the moment but as I said before the previous behaviour was to start at 5 seconds and increase that with each iteration12:20
*** lpetrut__ has joined #openstack-nova12:22
sean-k-mooneyYumeng: +1 on the vgpu spec. https://review.opendev.org/c/openstack/nova-specs/+/75011612:22
sean-k-mooneygibi: bauzas ^ if you have time to re review i think that is ready12:22
gibiI still don't see the difference between the old and the new approach. In both code wait a predefined time before retry, and in both code this waiting is only happen if the first try fails, if the first try succeeds then no waiting happens. So I don't see why we are in the better situation with the new code12:22
*** mkrai has joined #openstack-nova12:23
gibisean-k-mooney: ack12:23
sean-k-mooneygibi: didnt the old code retyr regardless of failure or success12:23
sean-k-mooneyi tought it just started teh time out and retired without waiting12:23
gibisean-k-mooney: the old case also checked the existence of the device in the live config before retry12:23
sean-k-mooneyfor an event12:23
sean-k-mooneygibi: right but that is not a vaild check12:24
gibisean-k-mooney: that is what we do both in the old and the new code12:24
sean-k-mooneythe issue is that with new qemu issuing a second detach cancles the operation12:24
gibisean-k-mooney: but it seems waiting for the event times out12:24
sean-k-mooneyso it may still be in the live domain but that does not mean you can send a second detach12:24
lyarwoodright, the main thing I wanted here was to allow n-cpu to wait longer between requests to libvirt to detach12:25
*** lpetrut_ has quit IRC12:25
gibisean-k-mooney: it seems that the libvirt detach call returns at the same time when the detach event is sent, so the whole thing seems synchronous12:25
lyarwoodthus making it harder if not impossible for us to hit the weird undocumented QEMU behaviour12:25
gibisean-k-mooney: but then I don't know how we was able to step on the first detach handling with the retry12:26
sean-k-mooneygibi: i think that depends on the load12:26
gibilyarwood: but then finding the right timeout is the goal both in the new and the old code. but for that we don't have to blow up the old code12:26
sean-k-mooneyi dont think it is synconos jsut look that way with light load12:27
sean-k-mooneygibi: you could simulate that with cgroups/taskset12:27
lyarwoodgibi: so the old code is also a mess12:27
gibisean-k-mooney: above kashyap and lyarwood said it looks synchronous12:27
sean-k-mooneyyou could limit the execution time of  libvirtd12:27
gibilyarwood: fair the old code is a mess and can be factored to be cleaner12:27
sean-k-mooneygibi: i see havent looked at ti too closely12:28
lyarwoodgibi: and about things being sync or async, I was talking about the failure event12:28
gibilyarwood: ohh, I missed that then12:28
sean-k-mooneyso the even is asyc right12:28
lyarwoodgibi: I think the success event can be async12:28
gibilyarwood: so the happy case still can be async?12:28
sean-k-mooneybut on a fast host it will be almost instant12:28
lyarwoodright12:28
gibicool, then the event waiting code is better12:28
lyarwoodright that allows us to wait longer12:29
lyarwoodas we should get an event back when the device does eventually detach12:29
gibiI will do more test to see why we don't get any event back from libvirt at the first detach attempt12:30
gibiin the above scenario12:30
gibias that missing event is bothering me12:30
sean-k-mooneygibi: you could try doing "sudo cpulimit -l 1 -e libvirtd" or "sudo cpulimit -l 1 -p <libvirtd pid>"12:31
gibisean-k-mooney: thanks I will try12:31
sean-k-mooneythat should limit libvirt to 1% of 1 core12:31
lyarwoodgibi: we don't get the event as the device is still attached to the live config12:33
lyarwoodgibi: and failure is for sync failures where QEMU rejects the request to detach12:34
gibilyarwood: but we expect that libvirt will detach from the live config as we asked for it12:34
gibiin the first request12:34
lyarwoodgibi: we expect that it should try12:34
gibiand if fails then it should tell us it failed, isn't it?12:34
kashyapgibi: lyarwood: So, Peter Krempa came back; let me post the chat summary snippet here, from #virt, OFTC:12:35
lyarwoodgibi: no, again the failure event is for when QEMU tells libvirt that it can't detach the device when the request is initially made12:35
lyarwoodgibi: if that request is accepted but the guestos doesn't allow the device to detach then we don't get anything back12:35
lyarwoodthat's my understanding at least, lets see what kashyap has12:35
kashyaplyarwood: Yeah; now I hear some additional subtleties; I'm going to post the full summary on the change; and we all can read and discuss here12:36
*** rcernin has quit IRC12:36
*** ociuhandu has quit IRC12:38
*** ociuhandu has joined #openstack-nova12:38
*** LinPeiWen1 has quit IRC12:42
*** eandersson has quit IRC12:44
*** eandersson has joined #openstack-nova12:44
*** zoharm1 has joined #openstack-nova12:45
*** Luzi has joined #openstack-nova12:46
*** ociuhandu has quit IRC12:46
*** brinzhang_ has joined #openstack-nova12:46
*** bhagyashris has quit IRC12:47
*** bhagyashris has joined #openstack-nova12:47
kashyapgibi: lyarwood: Done; see here (I also like DanPB's advice on dealin w/ active vs. inactive config separately): https://review.opendev.org/c/openstack/nova/+/770246/7/nova/virt/libvirt/driver.py#233012:48
*** zoharm has quit IRC12:48
*** brinzhang_ has quit IRC12:48
*** brinzhang_ has joined #openstack-nova12:48
*** purplerbot has quit IRC12:49
*** brinzhang_ has quit IRC12:50
*** brinzhang_ has joined #openstack-nova12:51
kashyapLet me know if that answers your question; if not, please note there, if anything doesn't add up :-)12:51
gibisorry, I'm on a call12:51
kashyapNo rush at all12:51
*** rcernin has joined #openstack-nova12:55
*** CeeMac has quit IRC12:56
*** vishalmanchanda has quit IRC12:56
*** tobberydberg has quit IRC12:56
*** brinzhang has quit IRC12:56
*** jrosser has quit IRC12:56
*** efried has quit IRC12:56
*** cgoncalves has quit IRC12:56
*** johanssone has quit IRC12:56
*** logan- has quit IRC12:56
*** ociuhandu has joined #openstack-nova12:57
*** artom has joined #openstack-nova12:58
*** CeeMac has joined #openstack-nova12:59
*** vishalmanchanda has joined #openstack-nova12:59
*** tobberydberg has joined #openstack-nova12:59
*** jrosser has joined #openstack-nova12:59
*** efried has joined #openstack-nova12:59
*** cgoncalves has joined #openstack-nova12:59
*** johanssone has joined #openstack-nova12:59
*** logan- has joined #openstack-nova12:59
*** arne_wiebalck has quit IRC12:59
*** logan- has quit IRC12:59
*** logan- has joined #openstack-nova13:00
gibikashyap: thanks for the summary, I will work DanPB's suggestion into the patch as handling the live an the persistent config separately feels cleaner to me too13:00
*** rcernin has quit IRC13:00
*** arne_wiebalck has joined #openstack-nova13:01
kashyapNp13:03
*** masterpe has quit IRC13:03
*** ociuhandu has quit IRC13:05
*** purplerbot has joined #openstack-nova13:06
*** links has quit IRC13:08
*** links has joined #openstack-nova13:13
*** rcernin has joined #openstack-nova13:13
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Drop support for Xen  https://review.opendev.org/c/openstack/nova/+/74323113:22
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION  https://review.opendev.org/c/openstack/nova/+/75470013:22
*** eharney has joined #openstack-nova13:25
*** rcernin has quit IRC13:27
*** ociuhandu has joined #openstack-nova13:32
*** ratailor has quit IRC13:35
*** ociuhandu has quit IRC13:36
*** ociuhandu has joined #openstack-nova13:36
*** ratailor has joined #openstack-nova13:38
openstackgerritBrin Zhang proposed openstack/nova master: Nova supports password encrypted VNC  https://review.opendev.org/c/openstack/nova/+/62233613:43
*** ratailor has quit IRC13:44
*** Yumeng has quit IRC13:48
*** masterpe has joined #openstack-nova13:56
*** lbragstad has joined #openstack-nova14:03
*** sapd1 has joined #openstack-nova14:12
*** lbragstad has quit IRC14:16
*** lbragstad has joined #openstack-nova14:19
*** lbragstad has quit IRC14:20
*** lbragstad has joined #openstack-nova14:20
*** zoharm has joined #openstack-nova14:21
*** zoharm1 has quit IRC14:23
stephenfinsean-k-mooney: Can you fix the failures on https://review.opendev.org/c/openstack/nova-specs/+/765901 real quick?14:24
sean-k-mooneyoh it failed yes ill do it now14:25
*** whoami-rajat__ has joined #openstack-nova14:25
sean-k-mooneysorry should have ran pep814:26
*** ociuhandu has quit IRC14:29
sean-k-mooneystephenfin: for the leading underscore14:30
sean-k-mooneyfor [2]14:30
*** ociuhandu has joined #openstack-nova14:30
sean-k-mooneyshoudl it be [_2]14:30
stephenfin*trailing14:30
stephenfin[2]_14:30
sean-k-mooneyoh14:30
sean-k-mooneyand the first one does not need that?14:31
stephenfinyou already have it on the first one :)14:31
sean-k-mooneyoh for [1]_14:31
sean-k-mooneyi have another instance of [2] before that14:31
sean-k-mooneycool ill fix them all14:31
*** ociuhandu has quit IRC14:34
*** ociuhandu has joined #openstack-nova14:34
*** arturo has left #openstack-nova14:34
openstackgerritsean mooney proposed openstack/nova-specs master: add port scoped numa affinity spec  https://review.opendev.org/c/openstack/nova-specs/+/76590114:35
sean-k-mooneystephenfin: i think ^ fixes everything14:36
sean-k-mooneylet me know if there is anything else and ill respin14:36
stephenfinwill do14:36
*** hemanth_n has quit IRC14:39
*** hemanth_n has joined #openstack-nova14:42
stephenfinsean-k-mooney: I have my doubt about that policy amendment to the modernize os-hypervisors API spec14:44
stephenfin"namely the idea that we shoudl allow users with the PROECT_ADMIN role to list all hyperviors there project is allowed to acess via the summary endpoint"14:44
stephenfinthere doesn't appear to be anything that restrict projects to specific hypervisor(s)14:45
stephenfinat least there's no tenant or project attribute of the ComputeNode or Service objects14:45
stephenfindid you have something specific in mind?14:46
*** Luzi has quit IRC14:48
*** dpawlik7 has quit IRC14:52
*** hemanth_n has quit IRC14:53
*** jangutter_ has joined #openstack-nova14:53
*** jangutter has quit IRC14:56
openstackgerritBalazs Gibizer proposed openstack/nova master: Move query param generation to RequestGroup o.vo  https://review.opendev.org/c/openstack/nova/+/75689415:00
openstackgerritBalazs Gibizer proposed openstack/nova master: Refactor _claim_pci_device_for_interface_attach to prepare for qos  https://review.opendev.org/c/openstack/nova/+/75689515:01
openstackgerritBalazs Gibizer proposed openstack/nova master: Allow extending an existing instance allocation  https://review.opendev.org/c/openstack/nova/+/75710915:03
openstackgerritBalazs Gibizer proposed openstack/nova master: Make remove allocation symmetric with add allocation  https://review.opendev.org/c/openstack/nova/+/75711015:04
*** mkrai has quit IRC15:05
openstackgerritBalazs Gibizer proposed openstack/nova master: Factor out port resource deallocation  https://review.opendev.org/c/openstack/nova/+/75711115:06
openstackgerritBalazs Gibizer proposed openstack/nova master: Refactor update_pci_request_spec_with_allocated_interface_name  https://review.opendev.org/c/openstack/nova/+/75761415:08
openstackgerritBalazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/utils.py  https://review.opendev.org/c/openstack/nova/+/76994215:09
openstackgerritBalazs Gibizer proposed openstack/nova master: Refactor ResourceRequest constructor  https://review.opendev.org/c/openstack/nova/+/76972015:10
openstackgerritBalazs Gibizer proposed openstack/nova master: Support interface attach with QoS ports  https://review.opendev.org/c/openstack/nova/+/75653015:12
*** nightmare_unreal has joined #openstack-nova15:12
openstackgerritBalazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/report.py  https://review.opendev.org/c/openstack/nova/+/76515215:13
openstackgerritBalazs Gibizer proposed openstack/nova master: Remove unused Instance method  https://review.opendev.org/c/openstack/nova/+/75761515:14
*** psachin has quit IRC15:14
sean-k-mooneystephenfin: i intened to reuse the aggreate metadtaa used for the teannt afinity filters15:15
sean-k-mooneywe use the same metadat for both the placment prefilter or the post filter15:15
sean-k-mooneythe prefilter allwos you to do stict tenant isolation15:15
sean-k-mooneyso that only a given tenatn can use member of a give aggreate15:15
*** dpawlik7 has joined #openstack-nova15:17
sean-k-mooneystephenfin: https://docs.openstack.org/nova/rocky/admin/configuration/schedulers.html#tenant-isolation-with-placement15:18
sean-k-mooneyyou do openstack aggregate set --property filter_tenant_id=9691591f913949818a514f95286a6b90 myagg15:18
sean-k-mooneyto restict the usage15:19
sean-k-mooneystephenfin: so the check you would do is simple.15:19
sean-k-mooneygretrive all aggreate with the current teant id15:19
sean-k-mooneyshow only host in those aggreates15:19
sean-k-mooneystephenfin: if the list is of aggrates is empty then list all hosts15:20
sean-k-mooneysince that tenant is unconfiged15:20
stephenfinmakes sense15:20
*** dpawlik7 has quit IRC15:26
*** lpetrut__ has quit IRC15:27
*** dpawlik9 has joined #openstack-nova15:27
*** dpawlik9 is now known as dpawlik15:27
openstackgerritDan Smith proposed openstack/nova master: Do not inherit os_glance properties on snapshot  https://review.opendev.org/c/openstack/nova/+/77123415:31
*** ociuhandu has quit IRC15:32
*** songwenping__ has quit IRC15:32
*** ociuhandu has joined #openstack-nova15:33
*** manuvakery1 has quit IRC15:33
*** nweinber has joined #openstack-nova15:35
*** ociuhandu has quit IRC15:37
*** efried has quit IRC15:39
*** efried has joined #openstack-nova15:40
*** efried has quit IRC15:41
*** efried has joined #openstack-nova15:47
*** nweinber_ has joined #openstack-nova15:49
stephenfingibi: Could you stick https://review.opendev.org/c/openstack/nova/+/764482/ on your review queue, please?15:49
*** nweinber_ has quit IRC15:50
gibistephenfin: on it15:50
lyarwoodhas anyone hit https://bugs.launchpad.net/tempest/+bug/1912178 locally today?15:53
openstackLaunchpad bug 1912178 in tempest "Links are not allowed as constraints (pip 20.2.4)" [Undecided,New]15:53
stephenfinnot me15:53
gibilyarwood: yes15:53
lyarwoodweird, I'm not sure what is adding the URI to the constraints list15:53
lyarwoodoh cool not just me then, cool15:54
gibilyarwood: I think devstack edits the constraints list based on what repos/libs you asked in the local.conf to be cloned15:54
gibilyarwood: as far as I understand the issue appears if somehow the pip version gets updated in your devstack vm15:55
gibilyarwood: for me, in a clean ubuntu 20.04 the first devstack stach.sh works15:55
gibibut subsequnt unstack.sh && stack.sh fails with the above rror15:55
*** swp20 has joined #openstack-nova15:56
lyarwoodgibi: yeah I was about to say I've not touched the version of pip in this env15:56
gibilyarwood: I suspect that the stach.sh runs some update if it is not in a clean vM15:57
gibibut I have no proof15:57
*** swp20 has quit IRC15:57
*** swp20 has joined #openstack-nova15:58
*** swp20 has quit IRC16:00
openstackgerritBalazs Gibizer proposed openstack/nova master: WIP/DNM libvirt: Start emitting DeviceRemovedEvent and DeviceRemovalFailedEvent events  https://review.opendev.org/c/openstack/nova/+/74992916:00
openstackgerritBalazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach  https://review.opendev.org/c/openstack/nova/+/77024616:00
*** swp20 has joined #openstack-nova16:00
lyarwoodgibi: fun, so I've updated pip and it seems to be working now16:02
gibihm, interesting16:02
*** nweinber has quit IRC16:03
*** macz_ has joined #openstack-nova16:04
*** ociuhandu has joined #openstack-nova16:04
*** macz_ has quit IRC16:04
*** macz_ has joined #openstack-nova16:05
*** nweinber has joined #openstack-nova16:07
*** ociuhandu has quit IRC16:09
stephenfinbauzas: If you've a chance this week, care to look at https://review.opendev.org/q/topic:bp/compact-db-migrations-wallaby ? dansmith and gibi have already looked through it, so I need a final pair of eyes16:14
bauzassure, will do16:14
stephenfinyou can ignore the apidb series at the bottom for now - I suspect I need to rework some of that16:14
stephenfincool, thanks16:15
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove 'hypervisor_version' from 'libvirt_info'  https://review.opendev.org/c/openstack/nova/+/74419916:25
*** ociuhandu has joined #openstack-nova16:29
openstackgerritStephen Finucane proposed openstack/nova master: Add generate schemas tool  https://review.opendev.org/c/openstack/nova/+/76979616:29
*** sapd1 has quit IRC16:31
*** ociuhandu has quit IRC16:33
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Drop support for Xen  https://review.opendev.org/c/openstack/nova/+/74323116:35
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION  https://review.opendev.org/c/openstack/nova/+/75470016:35
*** raildo_ has joined #openstack-nova16:42
*** sapd1 has joined #openstack-nova16:42
*** eandersson8 has joined #openstack-nova16:43
*** kashyap has quit IRC16:44
*** raildo has quit IRC16:44
*** slaweq has quit IRC16:44
*** slaweq has joined #openstack-nova16:45
*** eandersson has quit IRC16:45
*** zzzeek has quit IRC16:45
*** ociuhandu has joined #openstack-nova16:45
*** eandersson8 is now known as eandersson16:45
*** zzzeek has joined #openstack-nova16:46
*** rpittau is now known as rpittau|afk16:49
*** kashyap` has joined #openstack-nova16:50
*** tesseract has quit IRC16:51
openstackgerritMerged openstack/python-novaclient master: Deprecate agent commands and APIs  https://review.opendev.org/c/openstack/python-novaclient/+/76906816:51
gibilyarwood, kashyap`: back from some testing. If a domain has both a persistent and a live config, then detaching from only the persistent domain first works but does not generate an event. detaching from the live first then detaching from the persistent second does not work none of them generate event (but if the 3rd attempt is live again then that works).16:52
kashyap`gibi: On a call; but reading (as I'm not speaking) ...16:53
gibiso it seems we always need to detach from the persistent first, don't wait for any event, then detach from the live, and wait for event16:53
*** ociuhandu has quit IRC16:53
kashyap`Wait, you're contrasting "persistent" vs. "live" -- my brain was expecting either persistent vs. transient; or live vs offline :-)16:54
gibithe code call them persistent and live16:54
gibithis is where I learned :)16:54
gibidetach from persistent is detach with VIR_DOMAIN_AFFECT_CONFIG16:55
gibidetach from live is detach with VIR_DOMAIN_AFFECT_LIVE flag16:55
kashyap`(Sometimes libvirt terminology gives a damn headache)16:56
kashyap`(For ther record, I was referring to: https://wiki.libvirt.org/page/VM_lifecycle#Transient_guest_domains_vs_Persistent_guest_domains)16:56
kashyap`gibi: Hmm, but I _think_ I see what you mean16:57
kashyap`"live config" often implies it's not persistent (i.e. it is "transient")16:57
kashyap`Either way; back to the original issue16:57
gibiso17:00
gibiI can make it work it just wierdly order dependent17:00
kashyap`gibi: Hmm, so that behaviour of "no event when detaching from live config first" sounds strange and inconsistent.17:00
gibiand also the persistent side does not generate event17:00
*** ociuhandu has joined #openstack-nova17:00
gibikashyap`: yeah17:00
kashyap`gibi: Let me ask Peter; if this is a "bug or a feature" ;-)17:01
*** ociuhandu has quit IRC17:01
*** ociuhandu has joined #openstack-nova17:02
*** takamatsu has quit IRC17:02
*** ociuhandu has quit IRC17:02
gibikashyap`: could be some kind of complication in my test as well. I try to re-test it with a simpler tempest test as now there is rescue involved as well17:03
kashyap`gibi: I want to phrase the (complex) problem in my own words, so tell me if this is correct:17:05
kashyap`(Because I'm not clear in _which_ combination the event is generated and in which not)17:06
kashyap`- Assume a domain has both a persistent (i.e. inactive) and a live config.17:06
kashyap`- Detach a device from the inactive config first --> detach successful; but does not generate an event17:06
kashyap`- Detach a device from the live config first, then detach it from the persistent config --> the second detach fails; and no event generated in both cases17:06
kashyap`gibi: Is that correct?  (With the final point being: if the third attempt in the last step is a live config --> then the detach works)17:07
gibikashyap`: Detach a device from the live config first, then detach it from the17:08
gibi                  persistent config17:08
gibiin this case both detach returns without generating event, and the device remains attached (at least in the live config)17:08
kashyap`Aah --> both of them fail (i.e. detach not succesful) and nor do they generate an event?17:09
gibinova only checks the live config so I can only say that the detach failed from the live config as teh device remains there.17:10
gibibut meanwhile I simplifed the test17:10
gibiand now17:10
gibiI can start with live detach first and it succeeds and generates event as well17:11
gibiso there is some complication my first test scenario that makes the behavior different17:11
kashyap`gibi: The event is _REMOVAL_FAILED, yeah?17:11
kashyap`gibi: If you like, I can start an upstream thread w/ the libvirt folks on libvir-list, and Cc you17:12
gibikashyap`: when I get event it is VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED17:13
gibikashyap`: I think I've never seen VIR_DOMAIN_EVENT_ID_DEVICE_REMOVAL_FAILED in my tests ever17:13
kashyap`Err, sorry; it's a succesful removal17:13
gibikashyap`: let's not rush with that mail, I think I tested too much first17:13
kashyap`gibi: No rush at all :-) Just outlining as an option17:14
gibiand I have to figure out what extra the nova rescue does17:14
gibithat makes the behavior different17:14
gibibecause with a simple ACTIVE instance the live first scenario works17:14
gibias expected17:14
gibiremoves the device and generates event17:14
kashyap`Right; gibi - while it's fresh in your mind, do you want to quickly add a note on the scenarios you've tested and the ones that are "weird"?17:15
kashyap`Doesn't have to be now; I know it's EoD in CET.  Tomm is fine too17:15
gibisure I will comment on the patch17:15
gibilyarwood: you mentioned that I could check the device both in the persistent config and in the live config. do you have a pointer how to check in the persistent config?17:16
kashyap`Excellent; this hot unplug thing is always muddy; so many variables involved.17:16
kashyap`gibi: I wonder if lyarwood meant 'nodedev-list'17:22
kashyap`(No, wait - that's for listing devices on the host)17:22
kashyap`Normally, there should be a '--persitent' flag for most of the listing ('domiflist', 'domblklist', et al) APIs.17:27
gibikashyap`: yeah, I'm doing this from the nova code17:27
kashyap`gibi: So what do you want to check?  Query the presence of a device from peristent config?17:27
gibiyes17:28
gibinova currently query the live config only17:28
kashyap`gibi: Are you referring to this?17:31
kashyap`                device_name_for_logging=self.vif_driver.get_vif_devname(vif),17:31
kashyap`                live=live,17:31
gibikashyap`: https://github.com/openstack/nova/blob/5b65812ef2c6ac46bea30e2e603284eac232339e/nova/virt/libvirt/guest.py#L31517:33
gibithis is how nova checks if a device is in the config and I think it is checking the live config17:33
* kashyap` clicks17:33
gibikashyap`: for vifs the code is here https://github.com/openstack/nova/blob/5b65812ef2c6ac46bea30e2e603284eac232339e/nova/virt/libvirt/guest.py#L23017:34
gibiI guess the key is self._domain.XMLDesc(0) in both case but I failed to find the proper doc what 0 means and what else I can plug there17:35
*** links has quit IRC17:35
gibianyhow I have to drop off17:35
gibifor today17:35
kashyap`gibi: Ah, it's all custom parsing of the XML17:35
*** ociuhandu has joined #openstack-nova17:35
gibiyep17:35
kashyap`Yeah, I need to head out shortly too; we can work it out tomm17:35
gibikashyap`: thanks for the help, let's continue this tomorrow17:35
gibio/17:35
*** ociuhandu has quit IRC17:40
*** ociuhandu has joined #openstack-nova17:41
*** artom has quit IRC17:42
*** ociuhandu has quit IRC17:51
*** artom has joined #openstack-nova17:52
*** ociuhandu has joined #openstack-nova17:53
*** ociuhandu has quit IRC17:58
*** ralonsoh has quit IRC17:58
*** vishalmanchanda has quit IRC17:59
*** kashyap` is now known as kashyap18:01
*** dasp has quit IRC18:13
*** nweinber has quit IRC18:16
*** ociuhandu has joined #openstack-nova18:22
*** nweinber has joined #openstack-nova18:24
*** dasp has joined #openstack-nova18:25
*** ociuhandu has quit IRC18:35
gmanngibi: stephenfin can either of you check this-  https://review.opendev.org/c/openstack/placement/+/76439418:42
*** ociuhandu has joined #openstack-nova18:53
*** whoami-rajat__ has quit IRC18:55
elodlyarwood: sorry, I was off today, but will look into the patches tomorrow18:56
*** ociuhandu has quit IRC19:01
openstackgerritsean mooney proposed openstack/nova-specs master: libvirt: support vdpa kernel networking  https://review.opendev.org/c/openstack/nova-specs/+/76499919:01
sean-k-mooneyartom: im proably going to call it there for today let me know if there is anything else you want me to adress in ^19:04
sean-k-mooneyit will be tomorrow before i get to it proably19:04
artomsean-k-mooney, cool, thanks!19:04
artomGuyze.19:04
artomGuyze guyze guyzes.19:05
*** zoharm has quit IRC19:05
artomhttp://www.hardware.py/ is available!19:05
artomAww, it's restricted :(19:06
*** raildo_ is now known as raildo19:09
sean-k-mooneyhehe while agree the docs to that moduel are lacking im not sure it need its own webseit even thouh it could definetly fill one19:10
openstackgerritArtom Lifshitz proposed openstack/nova-specs master: `socket` PCI NUMA-affinity Policy  https://review.opendev.org/c/openstack/nova-specs/+/76555119:12
*** andrewbonney has quit IRC19:25
*** artom has quit IRC19:28
*** nightmare_unreal has quit IRC19:29
*** vishalmanchanda has joined #openstack-nova19:30
*** artom has joined #openstack-nova19:30
*** elod has quit IRC19:33
*** elod has joined #openstack-nova19:35
*** ociuhandu has joined #openstack-nova19:56
*** ociuhandu has quit IRC20:01
*** slaweq has quit IRC20:24
*** slaweq has joined #openstack-nova20:56
*** ociuhandu has joined #openstack-nova20:58
*** amodi has joined #openstack-nova21:07
*** ociuhandu has quit IRC21:09
*** ociuhandu has joined #openstack-nova21:10
*** ociuhandu has quit IRC21:13
*** ociuhandu has joined #openstack-nova21:13
*** xek has quit IRC21:26
*** ociuhandu has quit IRC21:26
*** nweinber has quit IRC21:56
*** vishalmanchanda has quit IRC22:09
*** raildo has quit IRC22:16
*** slaweq has quit IRC22:20
*** slaweq has joined #openstack-nova22:24
*** rcernin has joined #openstack-nova22:39
*** slaweq has quit IRC23:14
*** brinzhang_ has quit IRC23:20
*** brinzhang has joined #openstack-nova23:31
*** derekh has quit IRC23:41
*** ociuhandu has joined #openstack-nova23:49
*** macz_ has quit IRC23:52
*** ociuhandu has quit IRC23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!