| *** yingjisun has joined #openstack-nova | 00:18 | |
| openstackgerrit | norman shen proposed openstack/nova stable/rocky: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/747612 | 00:26 |
|---|---|---|
| *** sapd1 has quit IRC | 00:31 | |
| *** gyee has quit IRC | 00:56 | |
| *** hamalq has quit IRC | 01:21 | |
| *** Liang__ has joined #openstack-nova | 01:25 | |
| *** sapd1 has joined #openstack-nova | 01:27 | |
| *** HagunKim has quit IRC | 01:38 | |
| *** spatel has joined #openstack-nova | 01:53 | |
| *** sapd1 has quit IRC | 01:57 | |
| *** spatel has quit IRC | 01:58 | |
| *** rnoriega_ has quit IRC | 02:03 | |
| *** rnoriega_ has joined #openstack-nova | 02:04 | |
| *** sapd1 has joined #openstack-nova | 02:06 | |
| *** aj_mailing has quit IRC | 02:13 | |
| *** martinkennelly has quit IRC | 02:16 | |
| *** rcernin has quit IRC | 02:26 | |
| *** yingjisun_ has joined #openstack-nova | 02:32 | |
| *** yingjisun has quit IRC | 02:33 | |
| *** yingjisun_ is now known as yingjisun | 02:33 | |
| *** sapd1 has quit IRC | 02:38 | |
| *** sapd1 has joined #openstack-nova | 02:43 | |
| *** rcernin has joined #openstack-nova | 02:50 | |
| *** rcernin has quit IRC | 02:57 | |
| *** rcernin has joined #openstack-nova | 02:59 | |
| *** rcernin has quit IRC | 02:59 | |
| *** rcernin has joined #openstack-nova | 03:00 | |
| *** brinzhang0 has joined #openstack-nova | 03:04 | |
| *** brinzhang_ has quit IRC | 03:08 | |
| openstackgerrit | Jorhson Deng proposed openstack/nova master: add the decorator revert function for attach_volume and detach_volume https://review.opendev.org/754258 | 03:29 |
| *** sapd1 has quit IRC | 03:31 | |
| *** psachin has joined #openstack-nova | 03:37 | |
| *** aj_mailing has joined #openstack-nova | 03:43 | |
| *** ratailor has joined #openstack-nova | 03:51 | |
| *** Liang__ has quit IRC | 04:31 | |
| *** Liang__ has joined #openstack-nova | 04:32 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-nova | 04:33 | |
| *** sapd1 has joined #openstack-nova | 04:35 | |
| *** aj_mailing has quit IRC | 04:53 | |
| *** LinPeiWen has quit IRC | 05:00 | |
| *** ociuhandu has joined #openstack-nova | 05:01 | |
| *** ociuhandu has quit IRC | 05:10 | |
| *** bnemec has quit IRC | 05:10 | |
| *** aj_mailing has joined #openstack-nova | 05:18 | |
| *** LinPeiWen has joined #openstack-nova | 05:21 | |
| *** aj_mailing has quit IRC | 06:13 | |
| *** maciejjozefczyk_ has joined #openstack-nova | 06:22 | |
| *** slaweq has joined #openstack-nova | 06:22 | |
| *** ralonsoh has joined #openstack-nova | 06:30 | |
| *** slaweq has quit IRC | 06:33 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:36 | |
| *** maciejjozefczyk_ has quit IRC | 06:36 | |
| *** dklyle has quit IRC | 06:37 | |
| *** viks____ has joined #openstack-nova | 06:37 | |
| *** aj_mailing has joined #openstack-nova | 06:43 | |
| *** slaweq has joined #openstack-nova | 06:52 | |
| *** martinkennelly has joined #openstack-nova | 07:11 | |
| *** aj_mailing has quit IRC | 07:29 | |
| *** aj_mailing has joined #openstack-nova | 07:29 | |
| *** rcernin has quit IRC | 07:39 | |
| *** iurygregory has quit IRC | 07:40 | |
| *** tosky has joined #openstack-nova | 07:43 | |
| *** links has joined #openstack-nova | 07:44 | |
| *** maciejjozefczyk_ has joined #openstack-nova | 07:58 | |
| *** maciejjozefczyk has quit IRC | 08:01 | |
| *** nightmare_unreal has joined #openstack-nova | 08:10 | |
| *** links has quit IRC | 08:16 | |
| *** links has joined #openstack-nova | 08:16 | |
| *** maciejjozefczyk has joined #openstack-nova | 08:18 | |
| *** maciejjozefczyk_ has quit IRC | 08:21 | |
| *** maciejjozefczyk_ has joined #openstack-nova | 08:22 | |
| *** lpetrut has joined #openstack-nova | 08:23 | |
| *** derekh has joined #openstack-nova | 08:23 | |
| *** maciejjozefczyk has quit IRC | 08:24 | |
| *** aj_mailing has quit IRC | 08:26 | |
| *** sapd1_x has joined #openstack-nova | 08:26 | |
| *** sapd1 has quit IRC | 08:30 | |
| *** salmankhan has joined #openstack-nova | 08:33 | |
| *** k_mouza has joined #openstack-nova | 08:35 | |
| *** salmankhan has quit IRC | 08:40 | |
| *** rcernin has joined #openstack-nova | 08:40 | |
| *** salmankhan1 has joined #openstack-nova | 08:40 | |
| openstackgerrit | OpenStack Release Bot proposed openstack/nova stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/754319 | 08:40 |
| openstackgerrit | OpenStack Release Bot proposed openstack/nova stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320 | 08:40 |
| openstackgerrit | OpenStack Release Bot proposed openstack/nova master: Update master for stable/victoria https://review.opendev.org/754321 | 08:40 |
| openstackgerrit | OpenStack Release Bot proposed openstack/nova master: Add Python3 wallaby unit tests https://review.opendev.org/754322 | 08:40 |
| stephenfin | hurrah \o/ | 08:41 |
| stephenfin | gibi++ | 08:41 |
| *** salmankhan1 is now known as salmankhan | 08:42 | |
| *** rcernin has quit IRC | 08:45 | |
| gibi | yepp, RC1 is out | 08:47 |
| CeeMac | lyarwood: Morning. I've been trying to investigate the state of play with some other instances that were having their volumes retyped and I see a lot of errors. I found an instance with 'clean' volumes and tried to do a retype on the first volume attached. This failed with a different error "libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job" which I have seen in the | 09:05 |
| CeeMac | logs from some of the other tasks as I've been investigating | 09:05 |
| CeeMac | lyarwood: full error log output: http://paste.openstack.org/show/798374/ | 09:05 |
| CeeMac | lyarwood: looking at the cinder-volume logs the retype completes successfully then goes through an error clean up and deletes the temporary volume again. http://paste.openstack.org/show/798375/ | 09:07 |
| CeeMac | so I'm not sure if there is something else more seriously wrong that is linked to the corruption, or if this a separate issue | 09:08 |
| CeeMac | I'm going to enable debug on cinder-volume and try another one to see if there is any better info on what is happening. | 09:08 |
| CeeMac | if anyone else feels like chipping in, don't be shy :) This is perplexing me somewhat | 09:10 |
| *** brinzhang0 has quit IRC | 09:20 | |
| *** maciejjozefczyk has joined #openstack-nova | 09:20 | |
| *** maciejjozefczyk_ has quit IRC | 09:23 | |
| openstackgerrit | Merged openstack/python-novaclient stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/750932 | 09:23 |
| openstackgerrit | Merged openstack/python-novaclient stable/victoria: Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/750933 | 09:26 |
| *** ociuhandu has joined #openstack-nova | 09:28 | |
| *** links has quit IRC | 09:33 | |
| *** maciejjozefczyk_ has joined #openstack-nova | 09:36 | |
| *** rcernin has joined #openstack-nova | 09:37 | |
| *** maciejjozefczyk has quit IRC | 09:39 | |
| *** yingjisun has quit IRC | 09:43 | |
| *** rcernin has quit IRC | 10:10 | |
| lyarwood | CeeMac: morning sorry was AFK | 10:11 |
| *** salmankhan has quit IRC | 10:11 | |
| lyarwood | CeeMac: We would need to see the libvirtd logs associated with that rebase tbh | 10:12 |
| *** Liang__ has quit IRC | 10:14 | |
| CeeMac | lyarwood: how would I go about enabling logging for libvirt? | 10:14 |
| lyarwood | kashyap: did you have a public doc on enabling libvirt debugging anywhere? :) | 10:14 |
| *** salmankhan has joined #openstack-nova | 10:31 | |
| openstackgerrit | Harshavardhan Metla proposed openstack/nova stable/rocky: Updated the weights section of Compute schedulers in nova https://review.opendev.org/754349 | 10:33 |
| *** noonedeadpunk has quit IRC | 10:33 | |
| *** noonedeadpunk has joined #openstack-nova | 10:35 | |
| *** harsha24 has joined #openstack-nova | 10:45 | |
| *** maciejjozefczyk has joined #openstack-nova | 10:46 | |
| *** maciejjozefczyk_ has quit IRC | 10:49 | |
| *** dtantsur|afk is now known as dtantsur | 10:49 | |
| kashyap | lyarwood: Sure: https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt | 10:51 |
| kashyap | lyarwood: It my most referred public "doc" :D | 10:52 |
| kashyap | lyarwood: If you have 'virt-admin', take that route, instead (also documented above) | 10:52 |
| * kashyap --> back to kitchen | 10:52 | |
| lyarwood | CeeMac: ^ | 10:53 |
| lyarwood | kashyap: thanks | 10:53 |
| lyarwood | CeeMac: also I've been thinking about your issues, these are Windows Domain Controllers right? | 10:53 |
| kashyap | CeeMac: Oh, that "pivot" error brings back bad memories :D | 10:54 |
| CeeMac | kashyap: lyarwood thanks, i'll take a look | 10:54 |
| kashyap | CeeMac: Ah-ha, here it goes; I recall debugging it on the list: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html | 10:54 |
| CeeMac | lyarwood: yes, the ones that have been affected the worst so far | 10:54 |
| lyarwood | CeeMac: and the actual corruption you're seeing is that they can't boot after a reboot? Does Windows even load? | 10:54 |
| CeeMac | lyarwood: correct, and no, the AD DB appears to have been corrupted which causes a BSOD | 10:55 |
| CeeMac | lyarwood: tried all of the startup recovery options for DSRM etc, no dice | 10:55 |
| kashyap | CeeMac: Also, BTW, "one should not see that error anymore"(TM), because a bug that came out of debugging problem was fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1382165 | 10:55 |
| openstack | bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa | 10:55 |
| * kashyap --> kitchen, again | 10:55 | |
| CeeMac | lyarwood: I've discovered a few other instances with failed retype jobs, I'm going to try reboot them see if they're broken too or not | 10:55 |
| lyarwood | CeeMac: is the AD DB on the same volume as the OS? | 10:55 |
| CeeMac | lyarwood: it is not | 10:56 |
| CeeMac | kashyap: thanks, i'll go read that too! | 10:56 |
| lyarwood | CeeMac: right, I wonder if this is due to the disk serial changing during the retype | 10:56 |
| * lyarwood checks the code again | 10:56 | |
| CeeMac | lyarwood: thats what I was wondering too. And this seems to be the case for recreating a DC from its original disks, or from a snapshot | 10:59 |
| *** harsha24 has quit IRC | 10:59 | |
| CeeMac | lyarwood: although I haven't yet validated if the same corruption occurs, but the end result is absolutely the same. | 11:00 |
| lyarwood | CeeMac: is it not possible to point at a different disk for the AD DB to confirm this has happened and also recover? | 11:04 |
| lyarwood | CeeMac: or just list the disks in Windows before and after a retype | 11:04 |
| lyarwood | CeeMac: looking at the code things should persist in the DB | 11:05 |
| lyarwood | https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7299-L7314 - we use the return from a call to c-api here to determine what the volume_id and serial should be | 11:05 |
| lyarwood | https://github.com/openstack/nova/blob/2931516b75ab1eda8deefec6910efee82a0a7ae2/nova/compute/manager.py#L7364-L7389 - then stash it in our db here | 11:05 |
| * lyarwood -> lunch then I'll try to reproduce this brb | 11:07 | |
| CeeMac | lyarwood: the problem is if the AD DB corrupts either the server reboots itself and won't start back up, or stays online but you can't log in to it because the authentication is tied to the AD DB. | 11:07 |
| CeeMac | lyarwood: no worries, enjoy your lunch :) | 11:07 |
| *** sapd1_x has quit IRC | 11:11 | |
| *** xek has joined #openstack-nova | 11:14 | |
| CeeMac | kashyap: was this resolved in a paticular release do you know? is there some patch i can/should apply for rocky? | 11:16 |
| CeeMac | or, which version of libvirt was the patch introduced in may be a better question kashyap | 11:18 |
| *** sapd1_x has joined #openstack-nova | 11:19 | |
| kashyap | CeeMac: It is fixed in libvirt v5.0.0 and above | 11:22 |
| lyarwood | CeeMac: https://review.opendev.org/#/q/I0c52917a5555a70c4973f37dea1aebf878dd73b4 - I fixed that years ago, sorry I missed that kashyap suggested that | 11:22 |
| CeeMac | ah | 11:22 |
| CeeMac | i'm on v4 | 11:22 |
| CeeMac | which explains why i still see it | 11:22 |
| kashyap | OH, yes; that too - the 'mirror' element patch from lyarwood | 11:22 |
| CeeMac | lyarwood: should that not be present in rocky anyhow if it was introduced around ocata? (presumably)? | 11:23 |
| lyarwood | CeeMac: yes my change should be in stable/rocky | 11:24 |
| kashyap | Yes, it should be in Rocky | 11:24 |
| CeeMac | hmm | 11:24 |
| CeeMac | then i'm still confused how I'm gettting it :/ | 11:24 |
| kashyap | lyarwood: Unrelated - I can't reproduce the Focal / DEVICE_DEL thing for the life of me :-( Maybe my chops for setting up synthetic workload are bad | 11:25 |
| CeeMac | unless it would still require libvirt v5? | 11:25 |
| kashyap | CeeMac: Hmm, for "lulz", can you try with libvirt-v5, if you can afford to? | 11:25 |
| CeeMac | kashyap: you make that sound so easy :D | 11:25 |
| *** xek has quit IRC | 11:25 | |
| kashyap | I know; I'm a terribly bad person for phrasing it that way :D | 11:25 |
| *** sapd1_x has quit IRC | 11:26 | |
| CeeMac | is it a straight forward upgrade? or are there bits and bobs that plug in that would be affected | 11:26 |
| CeeMac | i have a test environment which i have these issues in that i also got in prod, that is on the same version (running OSA) | 11:26 |
| CeeMac | i was planning an upgrade from rocky to train anyhow | 11:27 |
| CeeMac | not sure which libvirt comes down with that, will check | 11:27 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1896463 in func env https://review.opendev.org/754100 | 11:27 |
| openstack | bug 1896463 in OpenStack Compute (nova) ussuri "evacuation failed: Port update failed : Unable to correlate PCI slot " [Low,Triaged] https://launchpad.net/bugs/1896463 | 11:27 |
| kashyap | CeeMac: Good question; when upgrading libvirt (assuming you're not using any container-based deployments): | 11:27 |
| CeeMac | no containers kashyap | 11:27 |
| kashyap | CeeMac: ... if you also see a QEMU update being pulled in: then you need to cold-reboot (stop + start) the instances for the libvirt to pick up the new QEMU | 11:28 |
| kashyap | (If there's no QEMU update; then 'just' restarting the libvirtd will suffice - yes, your instance will stay intact during libvirtd restart) | 11:29 |
| CeeMac | its like you've been asked these questions before :D | 11:29 |
| CeeMac | just need to find out if powercycling the other instances that have failed retype on their attached volumes also breaks the instance, then I'll ponder options for upgrades etc | 11:30 |
| openstackgerrit | kuang congxian proposed openstack/nova master: Replace six with python 3 style code in nova/test https://review.opendev.org/754354 | 11:30 |
| kashyap | CeeMac: Yes, very many times. Also some important bits here: https://kashyapc.fedorapeople.org/Effective-Virtual-CPU-Configuration-in-Nova-Berlin2018.pdf | 11:31 |
| kashyap | CeeMac: (Slightly updated variant of the above, from Jan-2020: https://kashyapc.fedorapeople.org/Secure-CPU-config-for-KVM-based-guests-RHT_TechDay_Brussels_2020.pdf) | 11:31 |
| *** rcernin has joined #openstack-nova | 11:41 | |
| *** harsha24 has joined #openstack-nova | 11:41 | |
| *** xek has joined #openstack-nova | 11:48 | |
| CeeMac | kashyap: thanks, appreciate it | 11:50 |
| kashyap | CeeMac: I mentioned the above because you also need to take into account "security" :D - (thanks to the CPU hardware flaws that flew by, and are still flowing by) | 11:51 |
| CeeMac | soooo many security flaws in the last few years! | 11:52 |
| *** raildo has joined #openstack-nova | 11:52 | |
| kashyap | Yea, are you seeing the timeline slide? ;-) | 11:53 |
| CeeMac | didn't look at it yet, was just thinking it terms of meltdown/spectre variants in general | 11:55 |
| CeeMac | which I now see is on the slide :D | 11:56 |
| *** harsha24 has quit IRC | 11:58 | |
| *** tbachman has quit IRC | 12:09 | |
| *** tbachman has joined #openstack-nova | 12:10 | |
| openstackgerrit | Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update .gitreview for stable/victoria https://review.opendev.org/754319 | 12:20 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/victoria: [stable-only] Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320 | 12:20 |
| openstackgerrit | Harshavardhan Metla proposed openstack/nova master: Updated the weights section of Compute schedulers in nova https://review.opendev.org/754365 | 12:22 |
| lyarwood | elod: the stable-only tags aren't technically required there at the moment as it isn't a full stable branch just yet | 12:22 |
| lyarwood | elod: until GA | 12:22 |
| lyarwood | elod: we will use the stable/$release branch to cut another rc tag if required | 12:23 |
| lyarwood | etc | 12:23 |
| lyarwood | at least that's my understanding | 12:24 |
| elod | lyarwood: the pep8 tools/check-cherry-picks.sh fails without the tag | 12:24 |
| lyarwood | ah my bad sorry | 12:26 |
| lyarwood | I forgot that we had that check in there | 12:26 |
| elod | yes, it's good most of the cases, but sometimes makes our life harder :) | 12:27 |
| elod | at least the fix is just a commit message update, so not that painful :) | 12:28 |
| lyarwood | yup true | 12:29 |
| *** kaisers has joined #openstack-nova | 12:33 | |
| *** nweinber has joined #openstack-nova | 12:35 | |
| *** k_mouza has quit IRC | 12:37 | |
| *** k_mouza has joined #openstack-nova | 12:38 | |
| *** yingjisun has joined #openstack-nova | 12:40 | |
| *** gmann is now known as gmann_pto | 12:41 | |
| *** k_mouza has quit IRC | 12:42 | |
| lyarwood | CeeMac: so I can reproduce the race in https://bugs.launchpad.net/nova/+bug/1896621 | 12:44 |
| openstack | Launchpad bug 1896621 in OpenStack Compute (nova) "instance corrupted after volume retype" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood) | 12:44 |
| lyarwood | CeeMac: looking at the corruption now, I assume serials are going to change once I reboot | 12:44 |
| CeeMac | lyarwood: glad you were able to reproduce it | 12:45 |
| lyarwood | CeeMac: so in that case the serials are correct after a hard reboot | 12:46 |
| *** maciejjozefczyk has quit IRC | 12:46 | |
| lyarwood | CeeMac: actually did you use --hard in the reboot command? | 12:46 |
| *** maciejjozefczyk has joined #openstack-nova | 12:46 | |
| lyarwood | CeeMac: I wonder if the issue is with a soft reboot here | 12:46 |
| CeeMac | Lyarwood well | 12:47 |
| lyarwood | oh wow the volume hasn't rolled back | 12:47 |
| CeeMac | lyarwood: in one instance the server restarted itself and then powered itself off | 12:48 |
| lyarwood | the instance is pointing to the original volume but cinder thinks that has retyped/migrated to the new type | 12:48 |
| CeeMac | lyarwood: another one I soft rebooted | 12:48 |
| CeeMac | lyarwood: is that visible somewhere? | 12:49 |
| CeeMac | lyarwood: when I checked openstack volume show it had migration state error but original volume id | 12:49 |
| CeeMac | And the disk mappings seemed correct in /dev/disk/by-path matching lun id etc | 12:50 |
| lyarwood | CeeMac: ah sorry I was looking at the wrong volume | 12:50 |
| lyarwood | CeeMac: the first volume is in that state | 12:50 |
| lyarwood | CeeMac: migration status error and listed against the original type | 12:50 |
| CeeMac | lyarwood: right | 12:51 |
| *** yingjisun has quit IRC | 12:55 | |
| lyarwood | CeeMac: yeah I can't make this fail and change the disk serials | 12:58 |
| lyarwood | CeeMac: I'll update the bug with some ideas | 12:58 |
| CeeMac | Thanks lyarwood | 12:59 |
| *** ratailor has quit IRC | 13:06 | |
| *** k_mouza has joined #openstack-nova | 13:08 | |
| lyarwood | CeeMac: was there much of a gap between the failure and the instances being rebooted? | 13:08 |
| *** spatel has joined #openstack-nova | 13:08 | |
| lyarwood | CeeMac: I think I see the issue now, we don't actually tell libvirt to revert back to the original volumes within the domain itself | 13:08 |
| lyarwood | CeeMac: if we fail late when dumping the domain XML | 13:09 |
| *** spatel has quit IRC | 13:12 | |
| *** dave-mccowan has joined #openstack-nova | 13:15 | |
| *** maciejjozefczyk_ has joined #openstack-nova | 13:16 | |
| *** jmlowe has quit IRC | 13:16 | |
| *** jmlowe has joined #openstack-nova | 13:19 | |
| *** maciejjozefczyk has quit IRC | 13:19 | |
| CeeMac | lyarwood: in the instance that broke itself its hard to say as it rebooted and powered itself off. | 13:20 |
| *** dave-mccowan has quit IRC | 13:20 | |
| CeeMac | lyarwood: one of the other instances there was a day or two before I manually rebooted | 13:21 |
| *** rcernin has quit IRC | 13:21 | |
| *** martinkennelly has quit IRC | 13:25 | |
| *** hemna has quit IRC | 13:25 | |
| CeeMac | lyarwood: sounds like you're on to something | 13:25 |
| *** hemna has joined #openstack-nova | 13:25 | |
| lyarwood | CeeMac: kk, I think the issue is that libvirt leaves the new volume connected after this failure, the instance keeps writing to the new volume until we rip out the volume from underneath it likely sending the disk into a read-only state within the instance. By the time the instance reboots it's out of sync with the original volume that is now reconnected and that's where the failure comes | 13:25 |
| lyarwood | from. | 13:25 |
| CeeMac | lyarwood: during the swap_volume process you mean? | 13:26 |
| lyarwood | let me see if I can reproduce this on master with a Fedora VM so I can debug this a little more and prove that's the case | 13:26 |
| CeeMac | lyarwood: so the new device is attached for a period until the job fails and is reverted? | 13:27 |
| lyarwood | CeeMac: so we fail *after* the swap volume operation has completed | 13:27 |
| lyarwood | CeeMac: yeah I believe so | 13:27 |
| lyarwood | CeeMac: there's nothing in the failure path within Nova that would revert the active domain configuration AFAICT | 13:27 |
| * lyarwood assumes he will be writing a load of functional tests for this very soon | 13:28 | |
| openstackgerrit | Merged openstack/nova master: libvirt: make mdev types name attribute be optional https://review.opendev.org/753574 | 13:33 |
| CeeMac | lyarwood: that could explain it. Appreciate you looking into this so diligently :) | 13:34 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224 | 13:47 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757 | 13:47 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777 | 13:48 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094 | 13:49 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109 | 13:50 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Add placeholder migrations for Victoria backports https://review.opendev.org/754396 | 13:54 |
| *** mlavalle has joined #openstack-nova | 13:59 | |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117 | 14:05 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/victoria: libvirt: make mdev types name attribute be optional https://review.opendev.org/754401 | 14:16 |
| stephenfin | bauzas: ^ | 14:17 |
| *** dklyle has joined #openstack-nova | 14:18 | |
| lyarwood | stephenfin: do we want to land that in stable before we are certain we don't want another rc? | 14:18 |
| stephenfin | um, I don't understand the question | 14:19 |
| stephenfin | are you saying we should wait to land this in 22.0.1? | 14:19 |
| gibi | if we merge someting meaningful to stable/victoria now then we have to cut RC2 I guess | 14:19 |
| stephenfin | ah | 14:19 |
| stephenfin | gotcha | 14:19 |
| stephenfin | then no, that's not critical IMO | 14:19 |
| stephenfin | I just know bauzas was eager for it to land at some point | 14:19 |
| gibi | yeah I would only land critical things to stable/v at the moment | 14:20 |
| stephenfin | makes sense | 14:20 |
| *** zzzeek has quit IRC | 14:21 | |
| gibi | also landing things on master should be limited to make a possible critical bug backport easy | 14:21 |
| *** zzzeek has joined #openstack-nova | 14:24 | |
| *** zzzeek has quit IRC | 14:29 | |
| *** zzzeek has joined #openstack-nova | 14:29 | |
| *** mlavalle has quit IRC | 14:32 | |
| *** mlavalle has joined #openstack-nova | 14:33 | |
| *** k_mouza has quit IRC | 14:33 | |
| *** zzzeek has quit IRC | 14:34 | |
| *** zzzeek has joined #openstack-nova | 14:36 | |
| *** slaweq has quit IRC | 14:37 | |
| *** slaweq has joined #openstack-nova | 14:44 | |
| *** k_mouza has joined #openstack-nova | 14:45 | |
| *** zzzeek has quit IRC | 14:46 | |
| *** zzzeek has joined #openstack-nova | 14:46 | |
| melwitt | lyarwood: hey, did you miss adding a NOTE here for the backport conflicts? https://review.opendev.org/747978 | 14:50 |
| *** zzzeek has quit IRC | 14:51 | |
| lyarwood | melwitt: looking | 14:52 |
| lyarwood | melwitt: oh is this the pike one | 14:52 |
| *** zzzeek has joined #openstack-nova | 14:53 | |
| melwitt | yeah, I wasn't sure if a NOTE was left out intentionally or not | 14:53 |
| lyarwood | melwitt: I didn't write them up for this sorry, I posted it as a courtesy for someone | 14:53 |
| lyarwood | that and the Newton patch attached to the bug | 14:53 |
| melwitt | ah gotcha | 14:54 |
| *** maciejjozefczyk has joined #openstack-nova | 14:56 | |
| *** maciejjozefczyk_ has quit IRC | 14:56 | |
| *** zzzeek has quit IRC | 14:58 | |
| *** zzzeek has joined #openstack-nova | 15:00 | |
| *** tkajinam has quit IRC | 15:00 | |
| *** k_mouza has quit IRC | 15:04 | |
| *** martinkennelly has joined #openstack-nova | 15:13 | |
| *** dklyle has quit IRC | 15:13 | |
| *** dklyle has joined #openstack-nova | 15:14 | |
| *** k_mouza has joined #openstack-nova | 15:14 | |
| *** k_mouza has quit IRC | 15:18 | |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Update contributor guide for Wallaby https://review.opendev.org/754427 | 15:20 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747978 | 15:26 |
| *** k_mouza has joined #openstack-nova | 15:28 | |
| *** k_mouza has quit IRC | 15:32 | |
| openstackgerrit | Merged openstack/os-vif stable/victoria: Update .gitreview for stable/victoria https://review.opendev.org/750924 | 15:34 |
| openstackgerrit | Merged openstack/nova master: Use absolute path during qemu img rebase https://review.opendev.org/739246 | 15:41 |
| openstackgerrit | Merged openstack/nova stable/ussuri: Move revert resize under semaphore https://review.opendev.org/751354 | 15:41 |
| openstackgerrit | Merged openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982 | 15:41 |
| *** lpetrut has quit IRC | 15:48 | |
| openstackgerrit | George Melikov proposed openstack/nova master: Instance snapshot: fallback to usual upload on ImageNotAuthorized https://review.opendev.org/754442 | 15:49 |
| *** salmankhan1 has joined #openstack-nova | 15:54 | |
| *** gyee has joined #openstack-nova | 15:55 | |
| *** salmankhan has quit IRC | 15:57 | |
| *** salmankhan1 is now known as salmankhan | 15:57 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055 | 16:00 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444 | 16:00 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444 | 16:02 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055 | 16:02 |
| *** k_mouza has joined #openstack-nova | 16:05 | |
| *** ralonsoh has quit IRC | 16:11 | |
| stephenfin | melwitt: Care to review this functional reproducer for me, please? https://review.opendev.org/#/c/749713/ | 16:13 |
| melwitt | stephenfin: will do | 16:13 |
| stephenfin | thanks :) | 16:13 |
| stephenfin | lyarwood: If you're still about, this is a follow-up to a series you looked at https://review.opendev.org/#/c/749319/ | 16:14 |
| * stephenfin is skimming through his list of "almost there" patches | 16:14 | |
| lyarwood | stephenfin: yup still here, if I don't get to today I'll make sure I do on Monday | 16:15 |
| stephenfin | Good enough for me. Thanks | 16:16 |
| *** xek has quit IRC | 16:19 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Rework how we check for neutron extensions https://review.opendev.org/705792 | 16:23 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: neutron: Remove references to out-of-tree 'qos-queue' extension https://review.opendev.org/754448 | 16:23 |
| openstackgerrit | Merged openstack/python-novaclient stable/ussuri: Add a cleanup for a server in a functional test https://review.opendev.org/743756 | 16:24 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Don't use generic 'Field' container https://review.opendev.org/738239 | 16:28 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove unnecessary type aliases, exceptions https://review.opendev.org/738240 | 16:28 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Use imports instead of type aliases https://review.opendev.org/738018 | 16:28 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove wrappers around ovo mixins https://review.opendev.org/738019 | 16:28 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: objects: Add type hints https://review.opendev.org/738020 | 16:28 |
| *** zzzeek has quit IRC | 16:29 | |
| *** zzzeek has joined #openstack-nova | 16:30 | |
| lyarwood | CeeMac: took a while but I've confirmed what I said before is what is happening btw, writing it up now. | 16:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove use of pkg_resources https://review.opendev.org/740661 | 16:32 |
| CeeMac | lyarwood: good work, thanks | 16:35 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to 'nova.pci' https://review.opendev.org/676209 | 16:36 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console https://review.opendev.org/705657 | 16:36 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: rpc: Rework 'get_notifier', 'wrap_exception' https://review.opendev.org/741663 | 16:36 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to top-level modules https://review.opendev.org/705658 | 16:36 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Expand type hints for nova.block_device https://review.opendev.org/742170 | 16:36 |
| *** salmankhan has quit IRC | 16:37 | |
| stephenfin | melwitt: Oh, also, if you have a chance to look at https://review.opendev.org/#/c/743976/ again and see if you agree with my thoughts, that'd be lovely | 16:38 |
| *** nweinber has quit IRC | 16:41 | |
| *** ociuhandu has quit IRC | 16:41 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Add type hints for 'nova.compute.api' https://review.opendev.org/741623 | 16:50 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Duplicate '_test_resize' to '_test_migrate' https://review.opendev.org/743206 | 16:50 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Remove unnecessary logic from resize, migrate tests https://review.opendev.org/743207 | 16:50 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Add 'nova.compute.API.migrate' https://review.opendev.org/741624 | 16:50 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Split 'nova.compute.API._resize' https://review.opendev.org/743208 | 16:50 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Add type hints https://review.opendev.org/744869 | 16:53 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Remove 'kwargs' from 'get_instance_nw_info' https://review.opendev.org/744872 | 16:53 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Remove unused 'affect_auto_assigned' parameter https://review.opendev.org/744871 | 16:53 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Add type hints for various allocate functions https://review.opendev.org/744870 | 16:53 |
| *** nweinber has joined #openstack-nova | 16:57 | |
| *** dtantsur is now known as dtantsur|afk | 16:58 | |
| *** JamesBenson has joined #openstack-nova | 17:01 | |
| CeeMac | lyarwood: thanks again for spending your time on this, it is much appreciated | 17:02 |
| *** aj_mailing has joined #openstack-nova | 17:03 | |
| *** ozzzo has joined #openstack-nova | 17:04 | |
| *** hamalq has joined #openstack-nova | 17:07 | |
| *** jangutter has quit IRC | 17:18 | |
| *** derekh has quit IRC | 17:18 | |
| *** jangutter has joined #openstack-nova | 17:19 | |
| *** hamalq has quit IRC | 17:21 | |
| *** hamalq has joined #openstack-nova | 17:21 | |
| *** abhishekk is now known as abhishekk|away | 17:27 | |
| *** nightmare_unreal has quit IRC | 17:32 | |
| *** maciejjozefczyk has quit IRC | 17:48 | |
| *** k_mouza has quit IRC | 17:53 | |
| melwitt | stephenfin: question for you re: whether we could add the correct asserts commented out or not https://review.opendev.org/#/c/749713/2/nova/tests/functional/libvirt/test_numa_live_migration.py@421 | 18:06 |
| *** xek has joined #openstack-nova | 18:07 | |
| *** xek has quit IRC | 18:07 | |
| openstackgerrit | Merged openstack/nova stable/victoria: [stable-only] Update .gitreview for stable/victoria https://review.opendev.org/754319 | 18:15 |
| openstackgerrit | Merged openstack/nova stable/victoria: [stable-only] Update TOX_CONSTRAINTS_FILE for stable/victoria https://review.opendev.org/754320 | 18:16 |
| *** ociuhandu has joined #openstack-nova | 18:33 | |
| *** psachin has quit IRC | 18:34 | |
| *** ociuhandu has quit IRC | 18:41 | |
| *** dklyle has quit IRC | 18:41 | |
| *** dklyle has joined #openstack-nova | 18:42 | |
| *** jdillaman has quit IRC | 18:48 | |
| *** kaisers has quit IRC | 19:00 | |
| *** k_mouza has joined #openstack-nova | 19:33 | |
| *** k_mouza has quit IRC | 19:37 | |
| *** nweinber has quit IRC | 19:38 | |
| openstackgerrit | Merged openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883 | 19:40 |
| *** JamesBenson has quit IRC | 19:42 | |
| *** nweinber has joined #openstack-nova | 19:42 | |
| *** JamesBenson has joined #openstack-nova | 19:43 | |
| *** JamesBenson has quit IRC | 19:48 | |
| *** k_mouza has joined #openstack-nova | 19:51 | |
| *** k_mouza has quit IRC | 19:56 | |
| *** spatel has joined #openstack-nova | 20:09 | |
| *** spatel has quit IRC | 20:13 | |
| *** JamesBenson has joined #openstack-nova | 20:18 | |
| *** nweinber has quit IRC | 20:22 | |
| *** JamesBenson has quit IRC | 20:23 | |
| *** slaweq has quit IRC | 20:54 | |
| *** JamesBenson has joined #openstack-nova | 20:58 | |
| *** nweinber has joined #openstack-nova | 20:58 | |
| *** nweinber has quit IRC | 21:00 | |
| *** JamesBenson has quit IRC | 21:03 | |
| *** JamesBenson has joined #openstack-nova | 21:31 | |
| *** k_mouza has joined #openstack-nova | 21:35 | |
| *** JamesBenson has quit IRC | 21:36 | |
| *** k_mouza has quit IRC | 21:39 | |
| *** JamesBenson has joined #openstack-nova | 22:00 | |
| *** k_mouza has joined #openstack-nova | 22:04 | |
| *** k_mouza has quit IRC | 22:08 | |
| *** JamesBenson has quit IRC | 22:28 | |
| *** k_mouza has joined #openstack-nova | 22:32 | |
| *** k_mouza has quit IRC | 22:37 | |
| *** tosky has quit IRC | 22:58 | |
| *** JamesBenson has joined #openstack-nova | 23:04 | |
| *** mlavalle has quit IRC | 23:07 | |
| *** JamesBenson has quit IRC | 23:08 | |
| openstackgerrit | Merged openstack/nova stable/pike: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747978 | 23:35 |
| *** hamalq has quit IRC | 23:39 | |
| *** k_mouza has joined #openstack-nova | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!