| openstackgerrit | Jeremy Stanley proposed openstack/nova master: DNM: Testing PDF builds https://review.opendev.org/752585 | 00:00 |
|---|---|---|
| *** yingjisun has joined #openstack-nova | 00:26 | |
| *** hamalq has quit IRC | 00:30 | |
| *** LinPeiWen has joined #openstack-nova | 00:31 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: Testing PDF builds https://review.opendev.org/752585 | 00:34 |
| *** gyee has quit IRC | 01:03 | |
| *** Liang__ has joined #openstack-nova | 01:20 | |
| *** zzzeek has quit IRC | 01:21 | |
| *** zzzeek has joined #openstack-nova | 01:22 | |
| *** gregwork has quit IRC | 01:27 | |
| *** yingjisun has quit IRC | 01:32 | |
| *** hamalq has joined #openstack-nova | 01:37 | |
| *** hamalq_ has joined #openstack-nova | 01:41 | |
| *** hamalq has quit IRC | 01:44 | |
| *** sapd__x has joined #openstack-nova | 01:48 | |
| *** jmlowe has quit IRC | 01:56 | |
| *** brinzhang_ has joined #openstack-nova | 02:11 | |
| *** brinzhang0 has quit IRC | 02:14 | |
| *** brinzhang0 has joined #openstack-nova | 02:17 | |
| *** songwenping_ has joined #openstack-nova | 02:17 | |
| *** songwenping__ has quit IRC | 02:20 | |
| *** brinzhang_ has quit IRC | 02:20 | |
| *** rcernin has quit IRC | 02:29 | |
| *** lbragstad has quit IRC | 02:31 | |
| *** links has joined #openstack-nova | 02:35 | |
| *** sapd__x has quit IRC | 02:50 | |
| *** zzzeek has quit IRC | 02:54 | |
| *** zzzeek has joined #openstack-nova | 02:54 | |
| *** sapd__x has joined #openstack-nova | 02:59 | |
| *** zzzeek has quit IRC | 03:01 | |
| *** zzzeek has joined #openstack-nova | 03:04 | |
| *** rcernin has joined #openstack-nova | 03:15 | |
| *** dave-mccowan has quit IRC | 03:16 | |
| *** hamalq_ has quit IRC | 03:18 | |
| *** mkrai has joined #openstack-nova | 03:25 | |
| *** psachin has joined #openstack-nova | 03:37 | |
| *** mkrai has quit IRC | 04:06 | |
| *** mkrai_ has joined #openstack-nova | 04:06 | |
| *** mkrai_ has quit IRC | 04:32 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-nova | 04:33 | |
| *** mkrai has joined #openstack-nova | 04:34 | |
| *** mkrai has quit IRC | 04:50 | |
| *** mkrai_ has joined #openstack-nova | 04:50 | |
| *** sapd__x has quit IRC | 04:55 | |
| *** zzzeek has quit IRC | 05:19 | |
| *** zzzeek has joined #openstack-nova | 05:19 | |
| *** xek has joined #openstack-nova | 05:26 | |
| *** sapd__x has joined #openstack-nova | 05:30 | |
| *** vishalmanchanda has joined #openstack-nova | 05:33 | |
| *** hamalq has joined #openstack-nova | 05:44 | |
| *** ratailor has joined #openstack-nova | 05:54 | |
| *** xek has quit IRC | 06:04 | |
| *** kukacz has joined #openstack-nova | 06:06 | |
| *** ralonsoh has joined #openstack-nova | 06:08 | |
| *** slaweq has joined #openstack-nova | 06:14 | |
| *** hamalq has quit IRC | 06:29 | |
| *** hamalq has joined #openstack-nova | 06:32 | |
| *** Yumeng has joined #openstack-nova | 06:39 | |
| *** mkrai_ has quit IRC | 06:40 | |
| *** mkrai_ has joined #openstack-nova | 06:40 | |
| *** dklyle has quit IRC | 06:41 | |
| *** hamalq has quit IRC | 06:44 | |
| *** jawad_axd has joined #openstack-nova | 06:45 | |
| *** amorin has quit IRC | 06:46 | |
| *** hamalq has joined #openstack-nova | 06:52 | |
| *** yonglihe has quit IRC | 06:53 | |
| *** guilhermesp has quit IRC | 06:53 | |
| *** johnsom has quit IRC | 06:53 | |
| *** vdrok has quit IRC | 06:53 | |
| *** rpittau|afk has quit IRC | 06:53 | |
| *** nicolasbock has quit IRC | 06:53 | |
| *** fyx has quit IRC | 06:53 | |
| *** pas-ha has quit IRC | 06:53 | |
| *** donnyd has quit IRC | 06:53 | |
| *** csatari has quit IRC | 06:53 | |
| *** rm_work has quit IRC | 06:53 | |
| *** masayukig has quit IRC | 06:53 | |
| *** TheJulia has quit IRC | 06:53 | |
| *** bcm has quit IRC | 06:53 | |
| *** nicolasbock has joined #openstack-nova | 06:55 | |
| *** vdrok has joined #openstack-nova | 06:56 | |
| *** yonglihe has joined #openstack-nova | 06:56 | |
| *** guilhermesp has joined #openstack-nova | 06:56 | |
| *** rpittau|afk has joined #openstack-nova | 06:56 | |
| *** masayukig has joined #openstack-nova | 06:56 | |
| *** bcm has joined #openstack-nova | 06:56 | |
| *** csatari has joined #openstack-nova | 06:57 | |
| *** TheJulia has joined #openstack-nova | 06:57 | |
| *** johnsom has joined #openstack-nova | 06:57 | |
| *** donnyd has joined #openstack-nova | 06:57 | |
| *** fyx has joined #openstack-nova | 06:57 | |
| *** brinzhang_ has joined #openstack-nova | 07:01 | |
| *** brinzhang0 has quit IRC | 07:04 | |
| *** hamalq has quit IRC | 07:05 | |
| *** mkrai_ has quit IRC | 07:06 | |
| *** rm_work has joined #openstack-nova | 07:08 | |
| *** pas-ha has joined #openstack-nova | 07:09 | |
| *** tesseract has joined #openstack-nova | 07:14 | |
| *** derekh has joined #openstack-nova | 07:20 | |
| *** tosky has joined #openstack-nova | 07:25 | |
| *** ociuhandu has joined #openstack-nova | 07:45 | |
| bauzas | good Friday Nova | 07:51 |
| lyarwood | Morning \o | 08:04 |
| lyarwood | sean-k-mooney / bauzas / stephenfin / gibi ; so some good news, the libvirt version bump https://review.opendev.org/#/c/746981/ works with the train UCA https://review.opendev.org/#/c/747123/ | 08:05 |
| lyarwood | the only issue remaining appears to be with how the grenade jobs are deployed | 08:05 |
| * lyarwood assumes they aren't using bionic | 08:05 | |
| lyarwood | oh they are, weird | 08:06 |
| lyarwood | ah right I need to backport this to stable/ussuri | 08:09 |
| lyarwood | fun chicken and egg problem here | 08:10 |
| kashyap | lyarwood: Mornin, I still haven't gotten to the Focal thing; was beset with a worser storm downstream, if you know what I mean :D | 08:11 |
| lyarwood | np | 08:12 |
| nightmare_unreal | Hello is there a glance dedicated channel ? | 08:14 |
| nightmare_unreal | nvm found it | 08:14 |
| gibi | lyarwood: what exactly need to be backported to ussir? | 08:19 |
| gibi | ussuri | 08:19 |
| lyarwood | gibi: https://review.opendev.org/#/c/752641/ I believe, do devstack uses the train UCA when deplying the stable/ussuri env during a grenade run | 08:21 |
| lyarwood | that or I could hack enabling the UCA into grenade on master somehow | 08:21 |
| lyarwood | so* devstack sorry | 08:22 |
| gibi | hm so in ussuri the train UCA has newer libvirt than the default packages? | 08:22 |
| lyarwood | yes correct | 08:24 |
| lyarwood | the train UCA has the new MIN_{LIBVIRT,QEMU}_VERSIONs we are looking to set in the current bump on master | 08:24 |
| lyarwood | as well | 08:24 |
| gibi | this means that we now change what libvirt version we use for testing the upgrade from ussuri | 08:26 |
| *** kukacz has quit IRC | 08:26 | |
| *** dtantsur|afk is now known as dtantsur | 08:27 | |
| *** kukacz has joined #openstack-nova | 08:28 | |
| lyarwood | gibi: we would also change the version of libvirt installed when testing stable/ussuri as a whole | 08:36 |
| lyarwood | gibi: but tbh I don't think anyone is actually using the default versions installed in bionic | 08:36 |
| bauzas | sorry I was focusing on some downstream bugs | 08:38 |
| bauzas | a tl:dr: please ? | 08:39 |
| lyarwood | bauzas: sean-k-mooney suggested that we use the Train UCA in Victoria so we can land the libvirt version bump *without* hitting the QEMU issue found in focal and the Ussuri UCA. | 08:40 |
| bauzas | lyarwood: for all jobs ? | 08:41 |
| lyarwood | bauzas: This mostly works but we need to also use the Train UCA on stable/ussuri to allow grenade jobs on master to pass | 08:41 |
| lyarwood | bauzas: yes for all jobs | 08:41 |
| bauzas | lyarwood: so it would be a Focal + train UCA for all jobs, right? | 08:41 |
| *** k_mouza has joined #openstack-nova | 08:41 | |
| lyarwood | bauzas: no just bionic | 08:41 |
| lyarwood | bauzas: this doesn't change the focal situation | 08:41 |
| lyarwood | bauzas: it just means we can land the libvirt version bump ahead of the focal change | 08:42 |
| lyarwood | bauzas: so we don't end up landing the version bump after rc | 08:42 |
| bauzas | lyarwood: I raised a concern yesterday about us bumping minimums so close to RC1 | 08:42 |
| bauzas | but I heard people wanting us to bump | 08:42 |
| lyarwood | bauzas: we don't have a choice if the TC are pushing us to move to focal | 08:43 |
| bauzas | well, TC aren't operators | 08:43 |
| bauzas | some of them are | 08:43 |
| bauzas | but I don't want us to hold all our gate so close to RC1 because we found some bugs unexpectedly | 08:43 |
| bauzas | telling "we had to do it because $TC" is a terrible explanation | 08:44 |
| lyarwood | I also hate making changes like this so close to RC tbh | 08:44 |
| lyarwood | but if we don't support bionic | 08:44 |
| bauzas | okay, let's be pragmatic and discuss about the change surface to be as minimal as we can | 08:44 |
| gibi | do we need to change the min libvirt version to be able to switch to Focal? | 08:44 |
| bauzas | there are two different concerns here | 08:44 |
| bauzas | 1/ people want to bump versions | 08:45 |
| bauzas | 2/ TC wants us to move to Focal | 08:45 |
| bauzas | those two concerns aren't interrelated | 08:45 |
| bauzas | (AFAICT) | 08:45 |
| lyarwood | right, we did however say that we would bump the min versions in V | 08:46 |
| lyarwood | and that we would also move the next versions in V | 08:46 |
| lyarwood | gibi: correct we don't need to change the min to move to focal | 08:46 |
| bauzas | bumping minimums is somehow understandable | 08:46 |
| bauzas | more than us moving to Focal, tbh | 08:47 |
| openstackgerrit | Liang Fang proposed openstack/nova master: [WIP] rbd patch for volume local cache https://review.opendev.org/726762 | 08:47 |
| stephenfin | I personally class the bumping of minimums more important and less risky than the focal move | 08:47 |
| bauzas | anyway, I want us to stop focusing on distros, and rather be discussing on QEMU versions | 08:47 |
| lyarwood | tbh I'd personally drop the entire move to focal at this point | 08:47 |
| stephenfin | oh, jinx :) | 08:47 |
| lyarwood | \o/ | 08:47 |
| gibi | droping Focal needs to be agreed with the TC | 08:48 |
| bauzas | stephenfin: yeah I'm kinda leading to this conclusion | 08:48 |
| lyarwood | I'm amazed that we continued with it after M2 tbh | 08:48 |
| lyarwood | move early or don't move at all | 08:48 |
| stephenfin | apologies if this was discussed yesterday, but couldn't we have a single non-voting tempest job on Focal? | 08:48 |
| bauzas | gibi: I don't understand why the OpenStack community makes an hard sell on a OS thing | 08:48 |
| stephenfin | to satisfy that request | 08:48 |
| bauzas | I thought we only considered dependencies and not OSes | 08:49 |
| lyarwood | yeah but I get the feeling that gmann's changes are to the base jobs | 08:49 |
| bauzas | to be frank | 08:49 |
| lyarwood | we could overwrite the node labels in our tree however | 08:49 |
| gibi | bauzas: me neither but I think to make the communication clear we have to be on the same side with the TC | 08:49 |
| bauzas | okay, let's be pragmatic | 08:49 |
| bauzas | the problem is with a specific QEMU version that races on volume attach/detach, right? | 08:50 |
| stephenfin | ah, you mean the parent 'openstack-*' and 'legacy-*' jobs? | 08:50 |
| bauzas | and past this version, QEMU isn't fixed yet | 08:50 |
| bauzas | am i correct? | 08:50 |
| lyarwood | stephenfin: yeah | 08:50 |
| bauzas | so, we can legitamately bump the minimums without being impacted by the QEMU issue | 08:51 |
| bauzas | legitimatly | 08:51 |
| lyarwood | bauzas: that's the issue with focal and the ussuri UCA yes | 08:51 |
| lyarwood | bauzas: but we can only bump the mins *if* we use the train UCA, ussuri UCA or move to focal | 08:51 |
| bauzas | lyarwood: because the ussuri UCA ships a qemu version that's impacted, right? | 08:51 |
| *** Liang__ is now known as LiangFang | 08:51 | |
| lyarwood | bauzas: correct it's the same versions as shipped in focal | 08:51 |
| lyarwood | bauzas: with the train UCA being a little older and not impacted by the bug | 08:52 |
| bauzas | ok, so the problem is with UCA, not the distro | 08:52 |
| lyarwood | bauzas: well both, it's in the core distro from focal | 08:52 |
| bauzas | ah I see | 08:52 |
| bauzas | gotcha | 08:52 |
| bauzas | so | 08:52 |
| bauzas | I'm all documenting this | 08:52 |
| gibi | buming minimum means we change the libvirt version on stable/ussuri too | 08:53 |
| gibi | is it OK? | 08:53 |
| bauzas | gibi: why ? | 08:53 |
| gibi | did we do that before in other stable branches? | 08:53 |
| lyarwood | gibi: yeah we've done that in the past | 08:53 |
| stephenfin | I guess we'll change the version tested if we backport, yes. We won't change the versions allowed though | 08:53 |
| lyarwood | bauzas: grenade | 08:53 |
| gibi | lyarwood: so it is sort of a normal thing at a min bump? | 08:53 |
| brinzhang_ | gibi: IMO, in V release is enough | 08:53 |
| lyarwood | gibi: iirc the last time we did this was due to QEMU/libvirt bugs | 08:54 |
| bauzas | lyarwood: what? | 08:54 |
| lyarwood | gibi: so we enabled the UCA to workaround these and had to do it on an older stable release for grenade as well | 08:54 |
| bauzas | lyarwood: I thought grenade was upgrading on the target | 08:54 |
| brinzhang_ | Cyborg done this today https://review.opendev.org/#/q/project:openstack/cyborg+branch:master+topic:migrate-to-focal | 08:54 |
| lyarwood | bauzas: it does but it doesn't rerun devstack | 08:54 |
| brinzhang_ | bump to the min version | 08:54 |
| lyarwood | bauzas: so we'd need to use the train UCA on stable/ussuri to ensure n-cpu from master would work | 08:55 |
| lyarwood | stephenfin: correct, it's just the tested version in stable/ussuri being changed | 08:55 |
| lyarwood | and again I don't think anyone is using the min versions of libvirt and QEMU at the moment | 08:55 |
| *** derekh has quit IRC | 08:55 | |
| lyarwood | if you're using Ubuntu you're using the UCAs | 08:56 |
| lyarwood | if you're using el distro you;re already way past these versions on stable/ussuri | 08:56 |
| bauzas | ok, either way | 08:56 |
| bauzas | if we merge the bumps | 08:57 |
| bauzas | are we sure that the new minimums are satisfied by the train UCA ? | 08:57 |
| bauzas | because | 08:57 |
| bauzas | we could just don't care about train UCA | 08:57 |
| lyarwood | bauzas: yes see https://review.opendev.org/#/c/746981/ | 08:57 |
| bauzas | but if we would like to tackle the other issue, we would need to deploy a Bionic with Train UCA | 08:58 |
| lyarwood | bauzas: specifically https://zuul.opendev.org/t/openstack/build/318b52b1aaf6473787405c9ed1ca0bc0/log/controller/logs/dpkg-l.txt | 08:58 |
| bauzas | perfect, then I'm clear | 08:58 |
| lyarwood | bauzas: that change Depends-On the devstack change pulling in the train UCA | 08:58 |
| lyarwood | bauzas: all of the non grenade jobs are working | 08:58 |
| gibi | OK, so it is not unprecedentet to change the tested libvirt version on a stable branch | 08:58 |
| bauzas | okay, so I personnally feel that bumping minimums are reasonably safe | 08:59 |
| gibi | so lets do it then | 08:59 |
| bauzas | +1 | 08:59 |
| gibi | bump the minimum and turn on train UCA in stable/ussuri | 08:59 |
| bauzas | and let's do what we approved yesterday for the Focal thing | 08:59 |
| bauzas | the gmann's work on jobs | 08:59 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224 | 09:00 |
| bauzas | lyarwood: sean-k-mooney: that being said, we need to document the gap | 09:00 |
| bauzas | I guess we can use https://bugs.launchpad.net/nova/+bug/1882521 as a base start | 09:00 |
| openstack | Launchpad bug 1882521 in OpenStack Compute (nova) "Failing device detachments on Focal" [High,Confirmed] - Assigned to Lee Yarwood (lyarwood) | 09:00 |
| *** mkrai has joined #openstack-nova | 09:01 | |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.iteritems/itervalues/iterkeys https://review.opendev.org/727757 | 09:01 |
| gibi | bauzas: could you drop the -2 from https://review.opendev.org/#/c/749707 then? | 09:01 |
| bauzas | gibi: I was on it | 09:01 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.byte2int/int2byte https://review.opendev.org/727777 | 09:01 |
| gibi | coolio | 09:01 |
| lyarwood | ubuntu-- https://zuul.opendev.org/t/openstack/build/fc4fd2d6fcd14618899060c4e84edca0/log/job-output.txt#6579 one small thing to fix here for the evacuation tests also | 09:01 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094 | 09:01 |
| lyarwood | bauzas: and you mean outside of https://review.opendev.org/#/c/746981/6/releasenotes/notes/victoria-libvirt-version-bump-e1a09b3a72ee56a4.yaml | 09:01 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109 | 09:01 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117 | 09:02 |
| *** martinkennelly has joined #openstack-nova | 09:02 | |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: doc: correct the link to user/flavor.rst https://review.opendev.org/740176 | 09:02 |
| bauzas | lyarwood: no, it's a matter of communicating the caveat we found with Focal | 09:03 |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: doc: Fix rendering in the accelerator support doc https://review.opendev.org/751574 | 09:03 |
| bauzas | because TC and the Foundation would communicate on the fact that service projects do support Focal | 09:03 |
| bauzas | but we regress on a thing | 09:03 |
| bauzas | so, ideally, I would comment this in a prelude section | 09:03 |
| *** Yumeng has quit IRC | 09:03 | |
| lyarwood | bauzas: ah sorry right can we include a known issues section? | 09:04 |
| bauzas | lyarwood: I guess that RHEL-based distros are also impacted by the volume attach issue since it's a qemu problem ? | 09:04 |
| bauzas | lyarwood: that'd be the right position I guess | 09:04 |
| *** k_mouza has quit IRC | 09:04 | |
| bauzas | lyarwood: reno supports an 'issues' section https://docs.openstack.org/reno/latest/user/usage.html#editing-a-release-note | 09:05 |
| lyarwood | bauzas: I've not been able to reproduce it on el distros yet | 09:06 |
| bauzas | ah shit | 09:06 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | 09:06 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982 | 09:06 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_NATIVE_TLS_VERSION https://review.opendev.org/746983 | 09:06 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_BETTER_SIGKILL_HANDLING https://review.opendev.org/746984 | 09:06 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985 | 09:06 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986 | 09:06 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: test_evacuate.sh: Support libvirt-bin and libvirtd systemd services https://review.opendev.org/752650 | 09:06 |
| bauzas | so it's not purely a qemu thing | 09:06 |
| lyarwood | we believe it is but it also appears to be caused by overall load on the host etc so it's super awkward to reproduce | 09:07 |
| *** martinkennelly has quit IRC | 09:07 | |
| bauzas | gibi: removed the -2 and +W the base patch | 09:08 |
| bauzas | lyarwood: what would be a recommendation for operators using Focal ? | 09:09 |
| bauzas | if we document a known issue | 09:09 |
| *** martinkennelly has joined #openstack-nova | 09:09 | |
| bauzas | I mean, is there any found workaround with Focal ? | 09:09 |
| lyarwood | no, there's not. | 09:09 |
| lyarwood | they could hit this issue | 09:09 |
| bauzas | or should we document as a known issue 'hold your OS upgrade if you consider running Victoria" | 09:10 |
| lyarwood | well that's the thing we can't if we drop support for bionic | 09:10 |
| bauzas | lyarwood: can't they try to override the base qemu versions by downgrading them using a Train UCA ? | 09:10 |
| lyarwood | if we support both then we can say that | 09:10 |
| lyarwood | bauzas: I doubt ubuntu would support that for a few release tbh | 09:10 |
| bauzas | lyarwood: I'm in favor of continuing to support Bionic for that exact reason | 09:11 |
| bauzas | this whole thing sucks. | 09:11 |
| lyarwood | bauzas: if we can it would make our lives alot easier | 09:11 |
| *** k_mouza has joined #openstack-nova | 09:12 | |
| bauzas | honestly, we would then have to write "if you wanna use Victoria and you feel you're impacted by the bug, please consider running another distro that's not impacted" ? | 09:12 |
| bauzas | this is absolutely serious. | 09:12 |
| bauzas | can't we try to find Canonical engineers to help us ? | 09:13 |
| bauzas | sahid isn't here AFAICS | 09:13 |
| bauzas | jamespage neither. | 09:13 |
| stephenfin | sahid left Canonical and is now...somewhere else | 09:14 |
| lyarwood | kashyap has been trying to engage their QEMU engineer | 09:14 |
| lyarwood | yes I did mean engineer | 09:14 |
| lyarwood | 8| | 09:14 |
| bauzas | gibi: have you engaged with the TC about this very critical issue ? | 09:14 |
| lyarwood | brb I need more coffee | 09:15 |
| bauzas | hopefully gmann did the liaison | 09:15 |
| kashyap | lyarwood: Is this the Focal topic? | 09:15 |
| * kashyap reads back | 09:15 | |
| kashyap | stephenfin: gibi: lyarwood: In general: in the past I've gone to Ubuntu's -stable IRC channel to point out missing virt-related backports | 09:16 |
| *** k_mouza has quit IRC | 09:16 | |
| kashyap | But that's not scalable... | 09:16 |
| bauzas | I'm sure that raising the problem to the TC and eventually the Foundation would send a signal that Canonical would be foolish to not capture | 09:17 |
| bauzas | because if we eventually say "sorry, but you can race on volume attach/detach with Focal and you have no other options but to consider other distros for this release if you BFV", that would certainly put some prios | 09:18 |
| *** arjunbaindur has joined #openstack-nova | 09:18 | |
| bauzas | that's my expectation at least | 09:18 |
| *** k_mouza has joined #openstack-nova | 09:18 | |
| bauzas | lyarwood: do you feel brave enough to write the problem and the absence of solution in a reno note ? | 09:20 |
| lyarwood | bauzas: I can try | 09:25 |
| bauzas | thanks | 09:26 |
| *** derekh has joined #openstack-nova | 09:26 | |
| gibi | so on Focal the only solution is to downgrade libvirt? | 09:27 |
| gibi | so on Focal the only solution is to downgrade qemu | 09:27 |
| bauzas | gibi: there is no solution for Focal, IIUC | 09:27 |
| bauzas | that's the problem | 09:27 |
| bauzas | or I'd rather say "there is no solution found yet for Focal" | 09:27 |
| bauzas | this leaves people investigating a potential workaround | 09:28 |
| gibi | there is no way to use an older qemu there? | 09:28 |
| bauzas | gibi: lyarwood said this is though to downgrade | 09:28 |
| bauzas | due to the canonical support model | 09:28 |
| *** brinzhang_ has quit IRC | 09:28 | |
| bauzas | we would then ask to install an old qemu version that's not supporter | 09:28 |
| bauzas | supported | 09:28 |
| lyarwood | gibi / bauzas ; Yeah I can't speak for Canonical but we (RH) wouldn't support that with our distros | 09:29 |
| bauzas | which makes sense tbh | 09:29 |
| gibi | OK, then simply state what we know in the reno. | 09:29 |
| gibi | don't have to be sugarcoated | 09:29 |
| lyarwood | kk | 09:29 |
| gibi | there is an open qemu bug to link to | 09:29 |
| bauzas | gibi: we also need to state we haven't identified yet a solution. | 09:29 |
| bauzas | if I was an operator, I'd surely reconsider upgrading if I feel I could be impacted | 09:30 |
| gibi | being it a qemu bug I don't really think we as openstack can find a solution | 09:30 |
| bauzas | I do agree | 09:30 |
| gibi | but yes, be clear that we don't know a viable workaround on Focal | 09:30 |
| bauzas | the workaround could be "don't use volumes ?" | 09:31 |
| bauzas | shit, I can't emoji | 09:31 |
| gibi | that is not a WA :D | 09:31 |
| kashyap | gibi: So for the Focal QEMU problem, the moment we can figure out a libvirt-level reproducer that can really speeden up the root-cause-analysis | 09:31 |
| kashyap | gibi: The tricky part here is, it is reproducible only in a Tempest-based setup :-( | 09:31 |
| gibi | kashyap: I see | 09:32 |
| gibi | bauzas: to answer your earlier question about talking to the TC, I haven't doen that (yet). I will raise the issue to them | 09:33 |
| kashyap | gibi: One approach I want to try (once I get these other pressing thing off my chest) is to setup a CPU-intensive workload, and do an attach+detach on a guest launched w/ plain libvirt | 09:33 |
| kashyap | ... and then pray to the Juju under the sea that the issue will reprocuce | 09:33 |
| bauzas | gibi: thanks | 09:34 |
| *** LiangFang has quit IRC | 09:35 | |
| gibi | bauzas: will you add the limitation to our relesenotes in a separate patch from the https://review.opendev.org/#/c/751045/3/releasenotes/notes/victoria-prelude-9b4c16ff8c6e7f3e.yaml ? | 09:39 |
| gibi | can I approve ^^? | 09:39 |
| bauzas | gibi: yes, I'll add another prelude item in a separate change once lyarwood documents the issue in another reno change | 09:39 |
| gibi | ack | 09:40 |
| gibi | thanks | 09:40 |
| bauzas | gibi: lyarwood: tbc, this is what I see : a documented 'issues' note that explains the problem and the absence of known solution | 09:40 |
| bauzas | and a prelude item that would only refer to this problem | 09:40 |
| gibi | sure, that works for me | 09:40 |
| *** vishalmanchanda has quit IRC | 09:43 | |
| * bauzas bails out for 2 hours, back to sports | 09:46 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: releasenote: Add known issue for bug #1894804 https://review.opendev.org/752654 | 09:46 |
| openstack | bug 1894804 in qemu (Ubuntu) "Second DEVICE_DELETED event missing during virtio-blk disk device detach" [Undecided,New] https://launchpad.net/bugs/1894804 | 09:46 |
| lyarwood | bauzas: ^ | 09:46 |
| bauzas | hah | 09:46 |
| bauzas | looking now | 09:46 |
| gibi | lyarwood: looking too | 09:46 |
| bauzas | lyarwood: absolutely well written | 09:47 |
| bauzas | lyarwood: I would +3 if I could | 09:47 |
| gibi | looks good to me too | 09:47 |
| * bauzas leaves now this time | 09:49 | |
| *** vishalmanchanda has joined #openstack-nova | 09:57 | |
| lyarwood | stephenfin: https://review.opendev.org/#/c/746984/ thoughts on this btw | 09:57 |
| *** martinkennelly has quit IRC | 10:00 | |
| openstackgerrit | Merged openstack/nova master: fakelibvirt: Use versionutils to set min versions found in the driver https://review.opendev.org/749707 | 10:02 |
| openstackgerrit | Merged openstack/nova master: Victoria 22.0.0 prelude section https://review.opendev.org/751045 | 10:03 |
| *** psachin has quit IRC | 10:09 | |
| *** rcernin has quit IRC | 10:09 | |
| *** psachin has joined #openstack-nova | 10:11 | |
| gibi | bauzas, lyarwood, gmann: pinged the TC about the our Focal limitation http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017390.html | 10:11 |
| stephenfin | gibi: Do you still have your SR-IOV environment available? If so, could you run a test for me? | 10:16 |
| gibi | stephenfin: sure | 10:16 |
| lyarwood | gibi: ack thanks | 10:16 |
| *** psachin has quit IRC | 10:16 | |
| lyarwood | melwitt: https://review.opendev.org/#/c/747361/ - would you mind hitting this stable backport series today? | 10:16 |
| stephenfin | gibi: Could you try (a) pausing and (b) stopping a guest and then attaching an SR-IOV-based VIF to it? | 10:17 |
| *** psachin has joined #openstack-nova | 10:18 | |
| gibi | so try to attach it while it is pause or stopped | 10:18 |
| stephenfin | yup | 10:18 |
| stephenfin | and ideally detach also, but that's less critical | 10:18 |
| gibi | OK, I will check it soon | 10:19 |
| stephenfin | I want to see if libvirt complains and if the guest handles it when it's resumed | 10:19 |
| stephenfin | ack, thanks :) | 10:19 |
| *** vinay_m has joined #openstack-nova | 10:19 | |
| vinay_m | hi , regarding this bug : https://bugs.launchpad.net/nova/+bug/1893238 | 10:20 |
| openstack | Launchpad bug 1893238 in OpenStack Compute (nova) "nova document search result links not working" [Medium,Triaged] | 10:20 |
| stephenfin | (reason being that we call attach_device without the live or persistent flags, so I suspect libvirt might crash and burn without them https://github.com/openstack/nova/blob/f47bde539c8/nova/virt/libvirt/driver.py#L4425) | 10:20 |
| vinay_m | is search implementation is diff/seperate from nova service ? | 10:21 |
| vinay_m | means like is it not involved with nova code part | 10:22 |
| sean-k-mooney | stephenfin: i kind of figured out why the func test failed. its not the fakelibvirt changes | 10:24 |
| sean-k-mooney | at least not directly | 10:24 |
| stephenfin | vinay_m: the search functionality is a Sphinx feature so no, it's not a nova issue | 10:25 |
| sean-k-mooney | the source_dev elemnt is different in one of the comarisons so it does not find the device to detach | 10:25 |
| gibi | stephenfin: boot, pause, attach, unpause. No error, the running guest sees the new device | 10:27 |
| stephenfin | gibi: Okay, cool. I wonder what the point of those flags is so :-\ | 10:28 |
| stephenfin | gibi: Could you run one final check? If you attach the device then stop and start the guest, does it stick around? | 10:28 |
| sean-k-mooney | stephenfin: im really thinking we might want to remove tehm and pass true in all cases | 10:29 |
| gibi | detaching while paused leads to a failure and a retry loop | 10:29 |
| sean-k-mooney | gibi: yep it should | 10:30 |
| vinay_m | stephenfin: thx | 10:30 |
| sean-k-mooney | libvirt will wait for the guest to respond to the acpi hot unplug interupt | 10:30 |
| sean-k-mooney | and since its paused it wont | 10:30 |
| stephenfin | sean-k-mooney: Yeah, I'm not even sure what the point of them is. By passing 'live=True', we'll be passing the 'VIR_DOMAIN_AFFECT_LIVE' flag to the 'attachDeviceFlags' call | 10:32 |
| stephenfin | libvirt says that "VIR_DOMAIN_AFFECT_LIVE specifies that the device shall be allocated to the active domain instance only and is not added to the persisted domain configuration." | 10:32 |
| stephenfin | but if the instance is shutoff, then by definition there isn't an active domain so that should fail | 10:33 |
| *** xek has joined #openstack-nova | 10:33 | |
| stephenfin | "Note that the target hypervisor must return an error if unable to satisfy flags." | 10:33 |
| sean-k-mooney | ah ok | 10:33 |
| sean-k-mooney | i didnt know about the error requiremetn | 10:33 |
| *** mkrai has quit IRC | 10:34 | |
| *** mkrai_ has joined #openstack-nova | 10:34 | |
| sean-k-mooney | well i did but not with that wording | 10:34 |
| stephenfin | my thinking was simply to set the 'VIR_DOMAIN_AFFECT_LIVE' and 'VIR_DOMAIN_AFFECT_CONFIG' flags based on checking 'domain.isActive()' and 'domain.isPersistent()' respectively | 10:34 |
| stephenfin | but if it's that simple, why do you even need the flags | 10:35 |
| sean-k-mooney | ya | 10:35 |
| * stephenfin is confused | 10:35 | |
| sean-k-mooney | i think that should work | 10:35 |
| sean-k-mooney | can you add a patch to try it | 10:35 |
| gibi | stephenfin: boot, attach, stop, start: guest keeps the attached device | 10:35 |
| sean-k-mooney | as i said i know we have had some issues where the live and persistent domains get out of sync | 10:35 |
| stephenfin | weeeeeird | 10:36 |
| stephenfin | might have to ask danpb about this one | 10:36 |
| sean-k-mooney | gibi: start recreated the xml form start so its going to work in the nova cases | 10:36 |
| sean-k-mooney | even if we dont update the persistent domain | 10:36 |
| sean-k-mooney | i kindof wish we did not have a persistnet domain in nova | 10:37 |
| sean-k-mooney | just use teh traisiant one | 10:37 |
| sean-k-mooney | it would remove a subset of bugs | 10:37 |
| stephenfin | sean-k-mooney: ah, that's a fair point | 10:37 |
| * lyarwood reads up | 10:37 | |
| lyarwood | sean-k-mooney: I thought that was the plan in W? | 10:37 |
| lyarwood | sean-k-mooney: to look into removing the persistent domain that is | 10:38 |
| stephenfin | I was thinking of reboot in the libvirt sense, where you have to have a persistent domain | 10:38 |
| sean-k-mooney | lyarwood: add it to the ptg/spec list | 10:38 |
| sean-k-mooney | lyarwood: that is what we talked about after that CVE | 10:38 |
| lyarwood | yup doing it now | 10:38 |
| sean-k-mooney | did you bring it up upstream yet | 10:38 |
| *** jangutter has joined #openstack-nova | 10:38 | |
| sean-k-mooney | lyarwood: stephenfin is looking at the device attach/detach flag we use and tryign to figure out why we pass live/persited true/false in places | 10:39 |
| lyarwood | I thought we had talked about it here but obviously not sorry | 10:39 |
| lyarwood | ah | 10:39 |
| lyarwood | so | 10:39 |
| lyarwood | that's because the initial attempt to detach | 10:39 |
| lyarwood | might only detach from the inactive config | 10:40 |
| lyarwood | after that we only want to try the live config | 10:40 |
| stephenfin | lyarwood: not for PCI devices | 10:40 |
| lyarwood | that's async and can fail if the guestos doesn't work with us | 10:40 |
| stephenfin | we're not using the detach_device_with_retry helper for those | 10:40 |
| stephenfin | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4396 | 10:40 |
| stephenfin | vs https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2306-L2309 | 10:41 |
| stephenfin | also not for mdev devices https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7512 | 10:41 |
| *** kukacz has quit IRC | 10:41 | |
| lyarwood | what's the reason for not using the helper with PCI devices? | 10:41 |
| lyarwood | or mdevs? | 10:41 |
| lyarwood | shouldn't the semantics be the same | 10:42 |
| *** jangutter_ has quit IRC | 10:42 | |
| stephenfin | not a clue :) that's what I was thinking but I can't find proof | 10:42 |
| stephenfin | nor can I find proof that what we do for the other cases is even correct | 10:42 |
| sean-k-mooney | so pci device detach was really only done in suspend until recently | 10:42 |
| lyarwood | yeah I would think they should be the same tbh, virtio-blk devices are PCI devices after all | 10:42 |
| *** jangutter_ has joined #openstack-nova | 10:42 | |
| lyarwood | oh right | 10:42 |
| stephenfin | for example, we don't pass the live or persistent flags to the attach_device call for PCI devices | 10:42 |
| stephenfin | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4425 | 10:42 |
| lyarwood | hotunplug wasn't always a thing with PCI passthrough | 10:43 |
| stephenfin | yet we do for other devices like network interfaces https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2247 | 10:43 |
| stephenfin | why? I don't know | 10:43 |
| sean-k-mooney | for pci passthough its still not a thing for neutron sriov port it partly worked btu it never freed the devices | 10:43 |
| sean-k-mooney | in the pci tracker | 10:43 |
| lyarwood | kk then that might be why | 10:43 |
| lyarwood | a single call would be enough if the domain was paused | 10:44 |
| lyarwood | and iirc live == running or paused right | 10:44 |
| stephenfin | yeah | 10:44 |
| sean-k-mooney | a paused domian is runnign yes | 10:44 |
| sean-k-mooney | well live | 10:44 |
| stephenfin | I want to replace that with a call to isActive | 10:44 |
| stephenfin | same thing, fwict | 10:44 |
| sean-k-mooney | its still defiend and qemu is running but the cpus are stopped | 10:44 |
| *** rcernin has joined #openstack-nova | 10:44 | |
| lyarwood | is this all from the dom.xml(PAUSED) patch btw stephenfin ? | 10:45 |
| stephenfin | yup :( I'm way down this rabbit hole now | 10:45 |
| * lyarwood hasn't got back to that after dumping stale comments in there the other day | 10:45 | |
| *** jangutter has quit IRC | 10:45 | |
| lyarwood | kk | 10:45 |
| lyarwood | well just to confuse you even more | 10:45 |
| lyarwood | https://review.opendev.org/#/c/749929/ is how we should be doing this | 10:46 |
| lyarwood | eventually | 10:46 |
| lyarwood | that's incomplete but events are apparently a better way of checking when things are actually detached | 10:46 |
| stephenfin | oh, very nice | 10:46 |
| lyarwood | urgh I want to rewrite detach_device_with_retry so much in W | 10:49 |
| *** martinkennelly has joined #openstack-nova | 10:49 | |
| lyarwood | stephenfin: so sorry I lost track of what the issue is here now | 10:51 |
| stephenfin | The question is whether we should be setting the live parameter here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4425 | 10:52 |
| stephenfin | gibi's testing suggests it's not necessary, in which case I guess the question is do we need to be setting it here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2247 | 10:53 |
| *** rcernin has quit IRC | 10:54 | |
| sean-k-mooney | stephenfin: if we just remove the persistnt domain then this is a moot point | 10:54 |
| stephenfin | true | 10:54 |
| sean-k-mooney | well maybe not | 10:55 |
| sean-k-mooney | we might need to backport something | 10:55 |
| sean-k-mooney | for issue on older branches | 10:55 |
| lyarwood | stephenfin: yeah odd, I wonder if it's something in https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainAttachDeviceFlags that defaults to attaching it to all configs? | 10:55 |
| stephenfin | VIR_DOMAIN_AFFECT_CURRENT sounds reasonable | 10:56 |
| lyarwood | https://github.com/openstack/nova/blob/c49cd3b9669da3f8da377369442bbe749fcb3c13/nova/virt/libvirt/driver.py#L1921 we also do that for disks FWIW | 10:56 |
| lyarwood | provide persistent and live that is | 10:56 |
| sean-k-mooney | lyarwood: ya i was wondering if currently we shoudl jsut set both to true | 10:57 |
| sean-k-mooney | but was not sure if that would fail in some cases | 10:57 |
| stephenfin | Or drop both flags and use VIR_DOMAIN_AFFECT_CURRENT (modify whatever state the instance is currently in) | 10:57 |
| lyarwood | stephenfin: right but we would need to ensure that would attach or detach from both configs at the moment | 11:01 |
| stephenfin | can the two get out of sync? | 11:01 |
| lyarwood | stephenfin: if we dropped the inactive persistent config then yeah we could just use that | 11:01 |
| lyarwood | stephenfin: yup | 11:01 |
| stephenfin | I thought an instance was either persistent or transient | 11:02 |
| stephenfin | *domain | 11:02 |
| * stephenfin wonders who on earth would want them to be different | 11:02 | |
| lyarwood | stephenfin: https://bugzilla.redhat.com/show_bug.cgi?id=1861142 for an example btw | 11:03 |
| openstack | bugzilla.redhat.com bug 1861142 in openstack-nova "[OSP 13.0.z] Unable to detach volume from instance when previously removed from the inactive config" [High,Verified] - Assigned to lyarwood | 11:03 |
| *** xek has quit IRC | 11:05 | |
| stephenfin | fun | 11:05 |
| lyarwood | so much | 11:05 |
| sean-k-mooney | how would people feel about backporting droping the persistent domain? is that doable or no? | 11:13 |
| sean-k-mooney | im not sure how that and rolling upgrades would interact | 11:14 |
| stephenfin | let's get it in master before we even think about | 11:14 |
| stephenfin | that | 11:14 |
| sean-k-mooney | ya im just wondering for some of the downstream bugs. | 11:15 |
| sean-k-mooney | although lyarwood you are fixign that bug via the events right | 11:15 |
| *** rcernin has joined #openstack-nova | 11:18 | |
| lyarwood | We could for new domains | 11:19 |
| lyarwood | we would just need to retain support for persisted domains for a while | 11:20 |
| lyarwood | but that's the same in master | 11:20 |
| lyarwood | actually I don't know if we could ever drop that support really | 11:20 |
| sean-k-mooney | we should be able too | 11:20 |
| lyarwood | unless we wait for major OS changes | 11:20 |
| lyarwood | like bionic to focal | 11:20 |
| lyarwood | that would need LM anyway | 11:20 |
| lyarwood | that would need an upgrade check on the computes | 11:21 |
| sean-k-mooney | so if all new domains are trasiant. and we make all move operations transitant and all hard reboots convert them to transient | 11:21 |
| lyarwood | so ops could be told something like `you're upgrading Nova to a version without support for persistent domains, LM instances from this compute host now before moving` | 11:21 |
| sean-k-mooney | then i think we only need 1 or 2 releases to drop it | 11:21 |
| lyarwood | yeah that's cool | 11:22 |
| sean-k-mooney | we would want a nova status check or something | 11:22 |
| sean-k-mooney | to say hay these vms still need to be moved | 11:22 |
| *** rcernin has quit IRC | 11:23 | |
| lyarwood | yup | 11:23 |
| sean-k-mooney | from a down stream persective it woudl be nice if this was in wallaby | 11:25 |
| sean-k-mooney | since the upgrade to wallaby will be an FFU and will involve migrations | 11:25 |
| lyarwood | yup | 11:25 |
| kashyap | sean-k-mooney: I'm on the Focal box you gave me; I'm doing a bunch of tests in it, I hope it won't disappear under my feet? :-) | 11:26 |
| sean-k-mooney | kashyap: not unless you or lyarwood delete it | 11:27 |
| sean-k-mooney | i wont be doing any maintance on it until kolla releases victoria or i hit a kernel issue that requires a reboot | 11:28 |
| kashyap | sean-k-mooney: Ah-ha, thank you | 11:29 |
| sean-k-mooney | its the same cloud i do all my dev on too | 11:29 |
| kashyap | And thanks for doing the gratis maintenance of this cloud for us | 11:29 |
| kashyap | I feel like I should be paying you out in a beverage of your choice, for using your electricity remotely | 11:29 |
| sean-k-mooney | its going to be running anyway | 11:30 |
| sean-k-mooney | at some point im probaly going to redploy a limited third party ci on it again | 11:30 |
| vinay_m | stephenfin : gone through bug comments ,discussed like it would be sphinx version related issue | 11:32 |
| sean-k-mooney | when i do ill proably try and replace fort nebula for multi numa testing of the whitebox plugin | 11:32 |
| vinay_m | i would like to explore how a search ref link working ,added in nova/doc/source/index.rst , how can i ? | 11:33 |
| sean-k-mooney | vinay_m: you can propose a patch via gerrit and the docs job will render a preview sight | 11:34 |
| sean-k-mooney | you can also build the docs locally | 11:34 |
| sean-k-mooney | using tox | 11:34 |
| sean-k-mooney | if you donwload the repos | 11:34 |
| sean-k-mooney | i think that woudl be enouch to test the search capablity | 11:35 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Introduce nova-evacuate https://review.opendev.org/744883 | 11:40 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557 | 11:40 |
| lyarwood | gmann: https://review.opendev.org/#/c/752641/ would you mind taking a look at this if you have time | 11:42 |
| *** k_mouza has quit IRC | 11:45 | |
| *** suryasingh has quit IRC | 11:48 | |
| *** k_mouza has joined #openstack-nova | 11:51 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: WIP/DNM - Image defined ephemeral storage encryption https://review.opendev.org/752284 | 11:54 |
| *** rcernin has joined #openstack-nova | 11:54 | |
| *** ygk_12345 has joined #openstack-nova | 11:54 | |
| *** kukacz has joined #openstack-nova | 11:54 | |
| ygk_12345 | hi all | 11:54 |
| ygk_12345 | i am facing a unique problem with rocky | 11:55 |
| *** k_mouza has quit IRC | 11:55 | |
| ygk_12345 | when I create an instance, it is having 4 vnics inside and they are getting ips from four dhcp tenant networks. but after sometime the vm is lossing its two network interfaces | 11:56 |
| ygk_12345 | so when I rebooth it then, then the vnics are coming back but again after 100 secs they are disappearing again . is this a knownw issue ? that image worked earlier | 11:57 |
| *** k_mouza has joined #openstack-nova | 11:58 | |
| *** rcernin has quit IRC | 11:59 | |
| sean-k-mooney | it sounds like you have a neutron issue | 12:01 |
| sean-k-mooney | ygk_12345: if neutron is retruning incosistent data then the force refesh of the netwrok info cache will result in the cache being populated with that inconsitent data | 12:02 |
| *** links has quit IRC | 12:02 | |
| sean-k-mooney | we have seen this downstream with contrail and aci before | 12:02 |
| sean-k-mooney | i have not see it be an issue with intree network backends | 12:02 |
| ygk_12345 | sean-k-mooney I see these messages in the compute node where that instance is | 12:03 |
| ygk_12345 | Received unexpected event network-vif-plugged-86f1034d-837d-4e67-ad5e-63d9642a0b2a for instance with vm_state active and task_state None. | 12:03 |
| ygk_12345 | Received unexpected event network-vif-unplugged-86f1034d-837d-4e67-ad5e-63d9642a0b2a for instance with vm_state active and task_state None. | 12:03 |
| sean-k-mooney | what neutron backend are you using | 12:04 |
| *** sapd__x has quit IRC | 12:04 | |
| ygk_12345 | ovs | 12:04 |
| ygk_12345 | ovs | 12:04 |
| sean-k-mooney | with iptables or contrack | 12:04 |
| ygk_12345 | ovs | 12:04 |
| sean-k-mooney | for the firewall driver | 12:04 |
| ygk_12345 | how to check it ? | 12:04 |
| sean-k-mooney | /etc/neutron/plugins/ml2/ml2_conf.ini | 12:05 |
| ygk_12345 | sean-k-mooney let me check | 12:05 |
| ygk_12345 | sean-k-mooney firewall_driver = openvswitch | 12:06 |
| sean-k-mooney | ok that is conntrack | 12:06 |
| ygk_12345 | sean-k-mooney so whats the workaround for this issue ? | 12:06 |
| sean-k-mooney | you have not provided enough info to know what the issue is yet | 12:07 |
| ygk_12345 | why i s the vif intermittently loosing connection and dropping ? | 12:07 |
| sean-k-mooney | those event wont casue the vif to be removed form the guest | 12:07 |
| ygk_12345 | sean-k-mooney so what logs to check in particulaer ? | 12:08 |
| sean-k-mooney | well first you need to look at the instance action logs for the affected instance and determin that no api action was performend on the guest | 12:08 |
| ygk_12345 | sean-k-mooney its happening only with a particular image | 12:08 |
| sean-k-mooney | the image cant affect this unless you are using the watchdog and the vm hung and was rebooted | 12:09 |
| ygk_12345 | where can I find those logs ? | 12:09 |
| sean-k-mooney | openstack server event list <uuid> | 12:10 |
| ygk_12345 | no actions except two reboots which I did | 12:11 |
| ygk_12345 | create,reboot,reboot | 12:11 |
| sean-k-mooney | ok if there are no action at the time of the events you next need to look at the neutron agent logs | 12:11 |
| sean-k-mooney | basically you need to check why the agent updated the port status | 12:12 |
| ygk_12345 | which agent in particular ? | 12:12 |
| ygk_12345 | dhcp ? | 12:12 |
| sean-k-mooney | if it was devstack it would be devstack@q-agt but its the neutron ovs l2 agent | 12:12 |
| ygk_12345 | ok | 12:12 |
| sean-k-mooney | if nova is not perfroming any instance actions then the port vif:host-id should not be updating | 12:13 |
| ygk_12345 | how can I track the port status of an instance in the ovs logs ? | 12:14 |
| sean-k-mooney | so the only thing that would cause those event to be emitted is a change in state form the ovs agent | 12:14 |
| sean-k-mooney | you need to have the neutron port uuid if the agent is in debug mode it will print the uuid of the port it is processign at differnt times | 12:14 |
| ygk_12345 | ok let me check | 12:15 |
| sean-k-mooney | what you are looking for is if the port is listed in teh added or deleted list of ports around teh time of the events | 12:15 |
| sean-k-mooney | that would indicate the port being removed or added to the ovs bridge | 12:15 |
| sean-k-mooney | simialrly if the admin state on the newon port was changed it would also show up there | 12:15 |
| sean-k-mooney | ygk_12345: the other thing that you cold check before going into the agent logs is the qemu instance log | 12:16 |
| ygk_12345 | oh ok | 12:16 |
| sean-k-mooney | and confirm that there are no restarts | 12:16 |
| sean-k-mooney | ygk_12345: do you enabel the watchdog or qemu guest agent in teh image that is affected | 12:17 |
| ygk_12345 | no idea | 12:18 |
| sean-k-mooney | they would be listed in the image metadata in glance | 12:18 |
| sean-k-mooney | ygk_12345: those event typically mean 1 of two things. eighet the tap device was removed and added to ovs, or the port was updated (host-id or admin status) | 12:20 |
| ygk_12345 | i dont think they are set for that image | 12:20 |
| sean-k-mooney | ok | 12:20 |
| sean-k-mooney | this sound like its not a nova issue for what its worth | 12:20 |
| sean-k-mooney | but check the qemu log to confim there is not reboot of the instace when it happened. if so then you need to look at the l2 agent logs if that does not show anyting then you need to check the dhcp agent and neutron server logs for the port uuid | 12:21 |
| sean-k-mooney | to track down why the event was sent | 12:21 |
| *** ratailor has quit IRC | 12:25 | |
| ygk_12345 | ok | 12:26 |
| ygk_12345 | sean-k-mooney even when the vnics have disappeared, those ports are being shown as active in nova | 12:28 |
| ygk_12345 | how can this be ? | 12:28 |
| sean-k-mooney | without logs i cant really help you resolve this. can you file a bug and attach some logs specificly the nova compute and l2 agent logs for aroudn the time when this happened | 12:31 |
| *** links has joined #openstack-nova | 12:31 | |
| ygk_12345 | ok | 12:31 |
| sean-k-mooney | what your describing does not really match to any known broken behviaor that comes to mind | 12:32 |
| sean-k-mooney | the only way to revmoe an interface form a vm that is runnign other then a nova interface detach which is not present in the event logs | 12:32 |
| sean-k-mooney | is to delete the neutron port | 12:32 |
| sean-k-mooney | that will send a network-vif-deleted event to nova | 12:33 |
| sean-k-mooney | but presuably you did not delete the nuetorn port | 12:33 |
| sean-k-mooney | so im not aware of any code path that could result in this | 12:33 |
| ygk_12345 | strange | 12:33 |
| sean-k-mooney | the info cache currption issue requires a hard reboot for the instance to be lost or a move operations | 12:34 |
| *** k_mouza_ has joined #openstack-nova | 12:40 | |
| *** k_mouza has quit IRC | 12:42 | |
| ygk_12345 | sean-k-mooney when the ping is unreachable for the vm , I dont see any messages for that port in the neutron-server except when I do a manual reboot | 12:44 |
| lyarwood | ubuntu-- | 12:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: test_evacuate.sh: Support libvirt-bin and libvirtd systemd services https://review.opendev.org/752650 | 12:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | 12:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982 | 12:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_NATIVE_TLS_VERSION https://review.opendev.org/746983 | 12:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_BETTER_SIGKILL_HANDLING https://review.opendev.org/746984 | 12:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985 | 12:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986 | 12:46 |
| stephenfin | lyarwood: https://media.tenor.com/images/a53dc07bf0a6f8822f16c0760299f915/tenor.gif | 12:47 |
| gibi | :) | 12:49 |
| lyarwood | yup sorry all | 12:51 |
| * lyarwood needs to head offline for a few hours | 12:53 | |
| lyarwood | back this evening to clean up any fallout caused by all of this | 12:53 |
| gibi | ack | 12:56 |
| ygk_12345 | sean-k-mooney i have posted this bug https://bugs.launchpad.net/nova/+bug/1896226 | 12:58 |
| openstack | Launchpad bug 1896226 in OpenStack Compute (nova) "The vnics are disappearing in the vm" [Undecided,New] | 12:58 |
| *** nweinber has joined #openstack-nova | 13:00 | |
| *** martinkennelly has quit IRC | 13:00 | |
| *** arjunbaindur has quit IRC | 13:02 | |
| kashyap | lyarwood: Hey, so do you know how many CPU cores did that reproducer host had? | 13:02 |
| kashyap | lyarwood: I'm asking because, depending on that I may need to fork my CPU workload to spread across multiple (CPU) cores | 13:03 |
| *** dave-mccowan has joined #openstack-nova | 13:04 | |
| kashyap | Meanwhile ... lovely: I just created a fresh F32 VM on Ubuntu Focal, and the bootloader is filling up little yellow squares on my screen. | 13:05 |
| gibi | kashyap: I think it is 8 https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/zuul-info/host-info.controller.yaml#440 | 13:10 |
| gibi | kashyap: at least this is the CI job run linked to the bug report | 13:11 |
| * kashyap clicks; thank you | 13:11 | |
| *** owalsh has quit IRC | 13:11 | |
| kashyap | gibi: So that's the "host VM" where Tempest is running, yeaH? | 13:17 |
| bauzas | gibi: thanks | 13:17 |
| kashyap | gibi: I'm trying to recreate the _exact_ damn setup :D | 13:18 |
| bauzas | (for the TC ping) | 13:18 |
| *** k_mouza_ has quit IRC | 13:18 | |
| *** tosky_ has joined #openstack-nova | 13:18 | |
| *** tosky is now known as Guest83047 | 13:18 | |
| *** tosky_ is now known as tosky | 13:18 | |
| gibi | kashyap: yes I think that is the "host" where tempest runs | 13:18 |
| sean-k-mooney | kashyap: it has 8 i think | 13:18 |
| kashyap | gibi: Okay, 8 vCPUs, and 7599 MB of memory | 13:19 |
| kashyap | gibi: And do you see what's the disk size? | 13:19 |
| sean-k-mooney | yep ist my small flavor | 13:19 |
| kashyap | sean-k-mooney: The disk size is ~16GB? | 13:20 |
| gibi | size: 80.00 GB | 13:20 |
| kashyap | gibi: Oh, right | 13:20 |
| gibi | https://zuul.opendev.org/t/openstack/build/4c56def513884c5eb3ba7b0adf7fa260/log/zuul-info/host-info.controller.yaml#259-293 | 13:20 |
| kashyap | Okay; I need to tweak the setup so that I'm also doing QEMU-on-KVM nested and not KVM-on-KVM nested | 13:20 |
| kashyap | Thanks; /me back in a few | 13:20 |
| gibi | thanks kashyap | 13:20 |
| openstackgerrit | Merged openstack/nova master: releasenote: Add known issue for bug #1894804 https://review.opendev.org/752654 | 13:22 |
| openstack | bug 1894804 in qemu (Ubuntu) "Second DEVICE_DELETED event missing during virtio-blk disk device detach" [Undecided,New] https://launchpad.net/bugs/1894804 | 13:22 |
| *** k_mouza has joined #openstack-nova | 13:22 | |
| *** LinPeiWen has quit IRC | 13:29 | |
| sean-k-mooney | kashyap: you dont need 80G of disk by the way 25-40G is typeically more then enough | 13:33 |
| *** ygk_12345 has quit IRC | 13:36 | |
| *** xek has joined #openstack-nova | 13:36 | |
| kashyap | sean-k-mooney: Yeah, I have something like that anyway | 13:38 |
| kashyap | sean-k-mooney: Do we have the full guest command-line of the Focal VM "host" upstream is using? | 13:40 |
| sean-k-mooney | if you want an account on my home cloud by the way let me know and i can create one for you | 13:40 |
| sean-k-mooney | no bug its just an openstack vm | 13:40 |
| *** k_mouza has quit IRC | 13:40 | |
| sean-k-mooney | with 8G of ram and 8 cpus and 80G fo disk | 13:41 |
| sean-k-mooney | im not sure that really matters too much | 13:41 |
| kashyap | sean-k-mooney: Okay; so what you gave me suffices | 13:42 |
| kashyap | sean-k-mooney: For now, stable access (and it _is_ stable) to this VM is sufficient :) Thx! | 13:42 |
| sean-k-mooney | ya my small flaovr is basicaly a proxy for the ci vms | 13:42 |
| sean-k-mooney | except i add hw:cpu_sockets=2 hw:cpu_threads=2 hw:numa_nodes=2 and hw:mem_page_size=large | 13:43 |
| sean-k-mooney | that should not change the behvior in this case but just an fyi | 13:43 |
| kashyap | sean-k-mooney: One more: what's the CirrOS version being run here? | 13:44 |
| *** vinay_m has quit IRC | 13:45 | |
| kashyap | "here" as in, in the Tempest env | 13:45 |
| sean-k-mooney | the default one form devstack so like 4.2 ish | 13:45 |
| kashyap | I'd like to know the exact version, although it shouldn't matter _that_ much | 13:45 |
| *** vinay_m has joined #openstack-nova | 13:45 | |
| sean-k-mooney | 0.5.1 | 13:45 |
| sean-k-mooney | https://github.com/openstack/devstack/blob/master/stackrc#L670 | 13:45 |
| kashyap | Excellent | 13:45 |
| sean-k-mooney | it hasnt been updated in 7 months https://github.com/openstack/devstack/commit/7a0fa4fd9e5db7253fee0820fc002703d43bca3c so that should be what its using | 13:46 |
| sean-k-mooney | the fine i think shoudl still be in /opt/stack/data i think | 13:46 |
| *** k_mouza has joined #openstack-nova | 13:50 | |
| *** k_mouza has quit IRC | 13:55 | |
| *** rcernin has joined #openstack-nova | 13:55 | |
| *** artom has joined #openstack-nova | 13:56 | |
| *** rcernin has quit IRC | 14:00 | |
| *** k_mouza has joined #openstack-nova | 14:02 | |
| kashyap | Also, annoyingly enough, CirrOS from 0.4.0 onwards, it's only initrd/kernel, so I can't use 'guestfish' to look around in it: https://paste.centos.org/view/10bee1e1 | 14:08 |
| *** xek has quit IRC | 14:09 | |
| *** links has quit IRC | 14:11 | |
| *** vinay_m has quit IRC | 14:14 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/ussuri: tests: Add regression test for bug 1894966 https://review.opendev.org/752371 | 14:21 |
| openstack | bug 1894966 in OpenStack Compute (nova) ussuri "Create servergroup failed with unexpected error" [Undecided,Confirmed] https://launchpad.net/bugs/1894966 | 14:21 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/ussuri: api: Set min, maxItems for server_group.policies field https://review.opendev.org/752702 | 14:21 |
| *** songwenping_ has quit IRC | 14:28 | |
| *** songwenping_ has joined #openstack-nova | 14:28 | |
| *** owalsh has joined #openstack-nova | 14:29 | |
| *** sapd1 has joined #openstack-nova | 14:30 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: tests: Add regression test for bug 1894966 https://review.opendev.org/752706 | 14:31 |
| openstack | bug 1894966 in OpenStack Compute (nova) ussuri "Create servergroup failed with unexpected error" [Undecided,In progress] https://launchpad.net/bugs/1894966 - Assigned to Stephen Finucane (stephenfinucane) | 14:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: api: Set min, maxItems for server_group.policies field https://review.opendev.org/752707 | 14:31 |
| *** mkrai_ has quit IRC | 14:34 | |
| *** mkrai has joined #openstack-nova | 14:36 | |
| mnaser | i'm curious as to why nova needs to ssh for cold migrations if using shared storage | 14:46 |
| mnaser | maybe no one just worked on that test path or? | 14:47 |
| mnaser | aka if shared storage: skip ssh | 14:47 |
| gibi | mnaser: I might be wrong but I remember nova detects that the instance is on share path by checking that the same file exists on both the source and the dest | 14:54 |
| mnaser | gibi: yeah i think that might be the only reason it still does it | 14:54 |
| mnaser | i think breaking out early if the VM is using volumes only would make sense cause that would eliminate the whole ssh key thing for us | 14:55 |
| mnaser | the only reason we still have it is just for that | 14:55 |
| gmann | gibi: bauzas lyarwood what is consensus for Focal migration? same as we discussed in meeting? | 14:55 |
| sean-k-mooney | mnaser: we could do that via an rpc i guess | 14:55 |
| mnaser | sean-k-mooney: can one n-cpu rpc to another n-cpui | 14:55 |
| sean-k-mooney | yes | 14:56 |
| sean-k-mooney | it happens alot in move operations | 14:56 |
| sean-k-mooney | well not a lot but we do rpcs between teh computes invovled | 14:56 |
| gibi | mnaser: hm, only volume instances still can have config drives | 14:57 |
| mnaser | gibi: right but i think we dont actually move that we just rebuild it on the target with a cold migrate | 14:57 |
| mnaser | i _think_ | 14:57 |
| gibi | most probably yes | 14:57 |
| mnaser | https://github.com/openstack/nova/blob/cff7382fb3de4eef7aeddc34b6ae3409cbe50ea0/nova/virt/libvirt/driver.py#L8340-L8376 | 14:58 |
| gibi | so, yeah we can stop doing that ssh shared storage check if the instance is only using volumes | 14:58 |
| mnaser | hmm this doesnt seem to use ssh | 14:58 |
| *** nightmare_unreal has quit IRC | 15:00 | |
| mnaser | i cant seem to honestly find the code that does the ssh to remote host.. | 15:00 |
| gibi | then i don't know what step needs the ssh | 15:00 |
| mnaser | i should go look for a traceback | 15:01 |
| gibi | gmann: we merged the libvirt min version bump, and we go with what we discussed on the meeting yesterday as far as I understand | 15:01 |
| *** dklyle has joined #openstack-nova | 15:02 | |
| gmann | gibi: ok, thanks for confirmation I will check and reply the ML after my breakfast | 15:02 |
| mnaser | ok i dont have the actual traceback but it looks like we try to mkdir remotely? https://bugzilla.redhat.com/show_bug.cgi?id=1624709 | 15:02 |
| openstack | bugzilla.redhat.com bug 1624709 in openstack-packstack "[OSP 13] Need to change the permission of /etc/nova/migration/identity after installation" [High,Closed: errata] - Assigned to ratailor | 15:02 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/queens: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747976 | 15:04 |
| mnaser | maybe a cold migrate does move the configdrive after all | 15:04 |
| *** mkrai has quit IRC | 15:05 | |
| sean-k-mooney | shelve recreates it as does cross cell resize | 15:06 |
| sean-k-mooney | but for same cell resize/migrate i think we copy them | 15:06 |
| gibi | mnaser: I only see the RemoteFilesystem.create_dir used for non shared instances | 15:08 |
| gibi | driver.py#L690 | 15:08 |
| gibi | https://github.com/openstack/nova/blob/90777d790d7c268f50851ac3e5b4e02617f5ae1c/nova/virt/libvirt/driver.py#L10347-L10349 | 15:08 |
| gibi | but during livemigrate we copy the config drive https://github.com/openstack/nova/blob/90777d790d7c268f50851ac3e5b4e02617f5ae1c/nova/virt/libvirt/driver.py#L9731 | 15:10 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM Testin Focal with integrated-gate-compute on Bionic https://review.opendev.org/752711 | 15:11 |
| *** mkrai has joined #openstack-nova | 15:15 | |
| *** xek has joined #openstack-nova | 15:16 | |
| *** xek has quit IRC | 15:17 | |
| *** mkrai has quit IRC | 15:22 | |
| *** mkrai_ has joined #openstack-nova | 15:22 | |
| *** tesseract has quit IRC | 15:35 | |
| *** mlavalle has joined #openstack-nova | 15:41 | |
| *** tkajinam has quit IRC | 15:44 | |
| *** martinkennelly has joined #openstack-nova | 15:47 | |
| *** jawad_axd has quit IRC | 15:49 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use cell targeted context to query instance metadata https://review.opendev.org/752459 | 15:51 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Clean up the DynamicVendorData constructor https://review.opendev.org/752718 | 15:51 |
| gibi | that was all from me for this week. See you on Monday | 15:53 |
| gibi | o/ | 15:53 |
| *** rcernin has joined #openstack-nova | 15:56 | |
| *** rcernin has quit IRC | 16:03 | |
| *** dtantsur is now known as dtantsur|afk | 16:10 | |
| openstackgerrit | Claudiu Belu proposed openstack/nova master: hyperv: Configures chassis asset tags for VMs. https://review.opendev.org/752723 | 16:12 |
| *** mkrai_ has quit IRC | 16:19 | |
| *** vishalmanchanda has quit IRC | 16:23 | |
| sean-k-mooney | stephenfin: look like this issue im facing is that the fakelibvirt dirver is incomplete | 16:23 |
| *** ociuhandu_ has joined #openstack-nova | 16:23 | |
| sean-k-mooney | stephenfin: when we generate teh fake instance domain its not provided with the info related to the vm pci devices just that the nic is type direct or whatever. | 16:24 |
| stephenfin | Hmm, yeah, that's not surprising | 16:25 |
| sean-k-mooney | we are also not generating network node_dev elements for any of the pci PF/VFs | 16:25 |
| sean-k-mooney | im going to hard code a few thigs to see if i can make some minimal chagnes to get it to pass | 16:25 |
| sean-k-mooney | but we likely need to flesh this out more in the future | 16:26 |
| stephenfin | yeah, I did the minimal amount necessary to get the sr-iov migration tests working | 16:26 |
| stephenfin | but it's full of holes, for sure. I just didn't have SR-IOV hardware handy to inspect the flow and copy the XML from | 16:27 |
| sean-k-mooney | so the issue is i need to backprot this to queens | 16:27 |
| *** ociuhandu has quit IRC | 16:27 | |
| sean-k-mooney | so im hesitent to depend on your refactors | 16:27 |
| sean-k-mooney | but they might also fix some of the issues | 16:27 |
| *** ociuhandu_ has quit IRC | 16:28 | |
| sean-k-mooney | this is what causes the detach to fail to find the device currently | 16:28 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/libvirt/fakelibvirt.py#L1826-L1830 | 16:28 |
| sean-k-mooney | that ses the souce_dev to fake_pf_interface_name | 16:28 |
| sean-k-mooney | but in https://review.opendev.org/#/c/602432/17/nova/tests/unit/virt/libvirt/fakelibvirt.py | 16:29 |
| *** jmlowe has joined #openstack-nova | 16:29 | |
| sean-k-mooney | both before and after my change we dont set source_dev when generatign the vm domain | 16:29 |
| *** k_mouza has quit IRC | 16:29 | |
| sean-k-mooney | im going to hard code if for now but im concerned then next thing ill hit is the pci adress dont match adresses form the host pci_info object | 16:30 |
| sean-k-mooney | since in both cases we are just incrementing the adresses | 16:30 |
| sean-k-mooney | ill corss that bridge later i guess. | 16:30 |
| *** ociuhandu has joined #openstack-nova | 16:33 | |
| sean-k-mooney | the issue with that fixture is that the comment and assumtion is wrong | 16:34 |
| sean-k-mooney | we also use this with macvtal to get the vf name not jsut the parent | 16:34 |
| sean-k-mooney | and we also dont have the required info to work this out when we generate the domain stuff currently. | 16:34 |
| *** ociuhandu has quit IRC | 16:37 | |
| *** k_mouza has joined #openstack-nova | 16:40 | |
| *** k_mouza has quit IRC | 16:44 | |
| *** nweinber has quit IRC | 16:44 | |
| *** psachin has quit IRC | 16:49 | |
| *** k_mouza has joined #openstack-nova | 16:51 | |
| *** k_mouza has quit IRC | 16:55 | |
| *** derekh has quit IRC | 17:01 | |
| *** hamalq has joined #openstack-nova | 17:01 | |
| openstackgerrit | Merged openstack/nova master: test_evacuate: Wait until subnode is down before starting tests https://review.opendev.org/751065 | 17:03 |
| *** k_mouza has joined #openstack-nova | 17:04 | |
| *** k_mouza has quit IRC | 17:09 | |
| *** iurygregory has quit IRC | 17:10 | |
| sean-k-mooney | stephenfin: ... https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/libvirt/fakelibvirt.py#L1087-L1093 | 17:14 |
| *** ralonsoh has quit IRC | 17:14 | |
| sean-k-mooney | detach device in the fake libvirt fixture only support disks | 17:14 |
| *** k_mouza has joined #openstack-nova | 17:14 | |
| sean-k-mooney | so i dont know how/why gibis func test could be passing on master. | 17:14 |
| *** k_mouza has quit IRC | 17:19 | |
| *** k_mouza has joined #openstack-nova | 17:26 | |
| *** k_mouza has quit IRC | 17:30 | |
| *** k_mouza has joined #openstack-nova | 17:34 | |
| *** gyee has joined #openstack-nova | 17:35 | |
| *** k_mouza has quit IRC | 17:39 | |
| *** k_mouza has joined #openstack-nova | 17:42 | |
| *** k_mouza has quit IRC | 17:47 | |
| *** k_mouza has joined #openstack-nova | 17:50 | |
| *** kukacz has quit IRC | 17:54 | |
| *** k_mouza has quit IRC | 17:55 | |
| *** jawad_axd has joined #openstack-nova | 18:11 | |
| *** jawad_axd has quit IRC | 18:15 | |
| *** irclogbot_1 has quit IRC | 18:19 | |
| *** irclogbot_1 has joined #openstack-nova | 18:24 | |
| *** irclogbot_1 has quit IRC | 18:31 | |
| *** jawad_axd has joined #openstack-nova | 18:32 | |
| *** jawad_axd has quit IRC | 18:36 | |
| *** irclogbot_0 has joined #openstack-nova | 18:36 | |
| *** k_mouza has joined #openstack-nova | 18:51 | |
| *** jawad_axd has joined #openstack-nova | 18:52 | |
| *** k_mouza has quit IRC | 18:56 | |
| *** jawad_axd has quit IRC | 18:56 | |
| *** nweinber has joined #openstack-nova | 18:57 | |
| *** sapd1 has quit IRC | 18:59 | |
| lyarwood | gmann: sorry was afk, are you still around? | 19:19 |
| lyarwood | gmann: AFAIK the plan is to still move to focal in V and skip the tests | 19:20 |
| lyarwood | gmann: in the meantime we wanted to try to move bionic onto the train UCA to allow us to move the MIN_ versions in the libvirt driver | 19:20 |
| lyarwood | gmann: https://review.opendev.org/#/c/752641/ needs to land first in stable/ussuri and then master to allow grenade to work correct | 19:21 |
| lyarwood | correctly* | 19:21 |
| lyarwood | then the rest of https://review.opendev.org/#/q/topic:bump-libvirt-qemu-victoria+(status:open+OR+status:merged) can land | 19:21 |
| *** k_mouza has joined #openstack-nova | 19:21 | |
| lyarwood | gmann: when do you think th focal change is actually going to land btw? | 19:22 |
| *** k_mouza has quit IRC | 19:26 | |
| *** slaweq has quit IRC | 19:41 | |
| *** k_mouza has joined #openstack-nova | 19:47 | |
| *** k_mouza has quit IRC | 19:51 | |
| gmann | lyarwood: hi, sorry i was away | 19:56 |
| gmann | lyarwood: ok, and I am keeping tempest-integrated-compute job on bionic to have those test coverage running somewhere | 19:59 |
| gmann | lyarwood: let me review the devstack patches | 19:59 |
| gmann | plan for Focal migration is pending on Barbican issue which is the only blocker left https://storyboard.openstack.org/#!/story/2007732 | 20:00 |
| *** nweinber has quit IRC | 20:01 | |
| *** zzzeek has quit IRC | 20:06 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: testing Train UCA https://review.opendev.org/752759 | 20:09 |
| *** spatel has joined #openstack-nova | 20:12 | |
| *** spatel has quit IRC | 20:17 | |
| *** k_mouza has joined #openstack-nova | 20:30 | |
| *** spatel has joined #openstack-nova | 20:33 | |
| *** k_mouza has quit IRC | 20:34 | |
| *** Corwin has quit IRC | 20:39 | |
| *** zzzeek has joined #openstack-nova | 20:42 | |
| *** owalsh has quit IRC | 20:46 | |
| *** zzzeek has quit IRC | 20:48 | |
| *** nweinber has joined #openstack-nova | 20:50 | |
| *** nweinber has quit IRC | 20:53 | |
| *** zzzeek has joined #openstack-nova | 20:54 | |
| *** spatel has quit IRC | 20:56 | |
| *** zzzeek has quit IRC | 20:59 | |
| *** ociuhandu has joined #openstack-nova | 21:02 | |
| *** zzzeek has joined #openstack-nova | 21:03 | |
| *** zzzeek has quit IRC | 21:05 | |
| *** zzzeek has joined #openstack-nova | 21:06 | |
| *** ociuhandu has quit IRC | 21:06 | |
| *** spatel has joined #openstack-nova | 21:11 | |
| *** zzzeek has quit IRC | 21:13 | |
| *** owalsh has joined #openstack-nova | 21:14 | |
| *** spatel has quit IRC | 21:16 | |
| *** zzzeek has joined #openstack-nova | 21:16 | |
| *** zzzeek has quit IRC | 21:23 | |
| *** zzzeek has joined #openstack-nova | 21:24 | |
| *** rcernin has joined #openstack-nova | 21:27 | |
| *** rcernin has quit IRC | 21:35 | |
| *** rcernin has joined #openstack-nova | 21:36 | |
| openstackgerrit | Merged openstack/nova stable/ussuri: Correctly disable greendns https://review.opendev.org/751621 | 21:40 |
| *** rcernin has quit IRC | 21:42 | |
| *** bbowen has joined #openstack-nova | 21:50 | |
| *** bbowen_ has quit IRC | 21:53 | |
| *** rcernin has joined #openstack-nova | 22:00 | |
| *** spatel has joined #openstack-nova | 22:22 | |
| *** dave-mccowan has quit IRC | 22:22 | |
| *** zzzeek has quit IRC | 22:25 | |
| *** zzzeek has joined #openstack-nova | 22:26 | |
| *** spatel has quit IRC | 22:27 | |
| *** zzzeek has quit IRC | 22:30 | |
| *** zzzeek has joined #openstack-nova | 22:31 | |
| *** eharney has quit IRC | 22:35 | |
| *** mlavalle has quit IRC | 22:38 | |
| *** martinkennelly has quit IRC | 22:38 | |
| *** hamalq has quit IRC | 23:02 | |
| openstackgerrit | Merged openstack/nova master: test_evacuate.sh: Support libvirt-bin and libvirtd systemd services https://review.opendev.org/752650 | 23:03 |
| openstackgerrit | Merged openstack/nova stable/rocky: Improve CinderFixtureNewAttachFlow https://review.opendev.org/749222 | 23:03 |
| *** rcernin has quit IRC | 23:06 | |
| *** zzzeek has quit IRC | 23:08 | |
| *** zzzeek has joined #openstack-nova | 23:11 | |
| *** zzzeek has quit IRC | 23:16 | |
| *** zzzeek has joined #openstack-nova | 23:17 | |
| *** tosky has quit IRC | 23:31 | |
| *** rcernin has joined #openstack-nova | 23:39 | |
| *** hamalq has joined #openstack-nova | 23:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
