| *** salv-orlando has joined #openstack-nova | 00:00 | |
| cfriesen_ | eandersson: you might try enabling debug logs and redoing the test. I'd expect that on the dest nova-compute rebuild_instance() would call ResourceTracker.rebuild_claim() | 00:00 |
|---|---|---|
| cfriesen_ | eandersson: that in turn should emit logs like "Attempting claim: ....." | 00:00 |
| cfriesen_ | Spaz-Home: no, it should claim resources for scheduler-chosen destinations as well since even now placement doesn't do cpu pinning, hugepages, etc. | 00:02 |
| cfriesen_ | eandersson: just to be clear, the "Attempting claim" log should come out at "info" level | 00:02 |
| eandersson | Is that in the compute logs? | 00:03 |
| cfriesen_ | eandersson: yes, on the dest node | 00:03 |
| eandersson | I only see three of those, and none from the evacuated nodes | 00:04 |
| cfriesen_ | eandersson: that claim is what calls _test_numa_topology(), which is what handles setting up the CPU pinning for the instance that is being evacuated. | 00:05 |
| *** yangyapeng has joined #openstack-nova | 00:05 | |
| *** tbachman has quit IRC | 00:05 | |
| *** salv-orlando has quit IRC | 00:05 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Rename network.utils to network.linux_utils. https://review.openstack.org/559197 | 00:06 |
| cfriesen_ | eandersson: I'd expect to see the "Attempting claim" log on instance boot as well. | 00:07 |
| *** tbachman has joined #openstack-nova | 00:08 | |
| eandersson | > Creating Image | 00:09 |
| eandersson | > Instance spawned successfully. | 00:09 |
| *** yangyapeng has quit IRC | 00:09 | |
| eandersson | That is it for INFO logging | 00:10 |
| eandersson | I'll enable debug logging and see in a bit | 00:11 |
| cfriesen_ | do you see a "Rebuilding instance" log on the dest compute node when doing the evacuate? | 00:11 |
| *** odyssey4me has quit IRC | 00:11 | |
| *** odyssey4me has joined #openstack-nova | 00:11 | |
| *** mdrabe has joined #openstack-nova | 00:13 | |
| *** takashin has joined #openstack-nova | 00:15 | |
| eandersson | Yea - missed that line > [instance: xxxx] Rebuilding instance | 00:16 |
| eandersson | > disk not on shared storage, rebuilding from: 'yyyy' | 00:16 |
| cfriesen_ | okay, so pretty much right after that I'd expect to see an "Attempting claim" log, followed by a "Claim successful" log | 00:16 |
| *** hamzy has quit IRC | 00:19 | |
| cfriesen_ | if you want you could add extra logging in nova.compute.manager.ComputeManager.rebuild_instance() around where it sets "rebuild_claim" and then calls it. | 00:19 |
| *** hamzy has joined #openstack-nova | 00:20 | |
| cfriesen_ | eandersson: crazy question maybe, but did you live-migrate any of these pinned instances? | 00:21 |
| eandersson | nope | 00:21 |
| eandersson | I created a lab and 5 new VMs. | 00:21 |
| eandersson | with two computes | 00:21 |
| eandersson | evacuated the two vms on compute2 | 00:22 |
| *** mdrabe has quit IRC | 00:22 | |
| *** yangyapeng has joined #openstack-nova | 00:24 | |
| eandersson | > Lock "compute_resources" acquired by "nova.compute.resource_tracker.rebuild_claim" | 00:24 |
| *** mdrabe has joined #openstack-nova | 00:24 | |
| *** hshiina has joined #openstack-nova | 00:25 | |
| *** chyka has joined #openstack-nova | 00:25 | |
| *** r-daneel has quit IRC | 00:26 | |
| cfriesen_ | eandersson: that's promising, it corresponds to the locking decorator on ResourceTracker.rebuild_claim | 00:28 |
| *** yangyapeng has quit IRC | 00:28 | |
| *** chyka has quit IRC | 00:30 | |
| cfriesen_ | eandersson: just had a thought...what virt driver are you using? | 00:31 |
| cfriesen_ | eandersson: do you see a log like "Virt driver does not support 'get_available_resource'. Compute tracking is disabled." ? | 00:32 |
| *** Dinesh_Bhor has joined #openstack-nova | 00:32 | |
| eandersson | nope | 00:34 |
| *** mdrabe has quit IRC | 00:34 | |
| eandersson | cfriesen_, default kvm | 00:35 |
| *** hoangcx has quit IRC | 00:39 | |
| *** hoangcx has joined #openstack-nova | 00:40 | |
| *** mdrabe has joined #openstack-nova | 00:44 | |
| *** Swami has quit IRC | 00:44 | |
| *** yangyapeng has joined #openstack-nova | 00:45 | |
| eandersson | libvirt, qemu-kvm | 00:47 |
| *** yangyapeng has quit IRC | 00:49 | |
| *** _ix has quit IRC | 00:50 | |
| *** yamamoto has joined #openstack-nova | 00:53 | |
| *** yamamoto has quit IRC | 00:58 | |
| *** gjayavelu has quit IRC | 00:59 | |
| *** vladikr has quit IRC | 00:59 | |
| *** vladikr has joined #openstack-nova | 01:00 | |
| *** mdrabe has quit IRC | 01:03 | |
| *** mdrabe has joined #openstack-nova | 01:05 | |
| *** tiendc has joined #openstack-nova | 01:07 | |
| *** phuongnh has joined #openstack-nova | 01:07 | |
| *** s1061123 has quit IRC | 01:08 | |
| *** s1061123 has joined #openstack-nova | 01:10 | |
| *** baoli has joined #openstack-nova | 01:10 | |
| *** mdrabe has quit IRC | 01:10 | |
| *** baoli has quit IRC | 01:15 | |
| *** hongbin has joined #openstack-nova | 01:16 | |
| *** yangyapeng has joined #openstack-nova | 01:18 | |
| *** yangyapeng has quit IRC | 01:18 | |
| *** suresh12 has quit IRC | 01:20 | |
| *** harlowja has quit IRC | 01:21 | |
| *** kaisers has quit IRC | 01:22 | |
| *** baoli has joined #openstack-nova | 01:29 | |
| melwitt | logan-: here's the link http://logs.openstack.org/84/534384/24/check/tempest-full/ae551f8/ | 01:31 |
| *** mdrabe has joined #openstack-nova | 01:33 | |
| *** baoli has quit IRC | 01:37 | |
| *** baoli has joined #openstack-nova | 01:37 | |
| *** amodi has quit IRC | 01:38 | |
| *** mdrabe has quit IRC | 01:38 | |
| *** baoli has quit IRC | 01:42 | |
| *** mdrabe has joined #openstack-nova | 01:45 | |
| *** baoli has joined #openstack-nova | 01:51 | |
| *** baoli has quit IRC | 01:51 | |
| *** yangyapeng has joined #openstack-nova | 01:52 | |
| *** yangyapeng has quit IRC | 01:52 | |
| *** yamamoto has joined #openstack-nova | 01:55 | |
| *** mdrabe has quit IRC | 01:58 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove non-libguestfs file injection for libvirt https://review.openstack.org/324720 | 01:58 |
| *** fragatina has quit IRC | 01:59 | |
| *** yamamoto has quit IRC | 02:00 | |
| *** salv-orlando has joined #openstack-nova | 02:02 | |
| *** mdrabe has joined #openstack-nova | 02:03 | |
| *** annp has joined #openstack-nova | 02:04 | |
| *** salv-orlando has quit IRC | 02:07 | |
| *** suresh12 has joined #openstack-nova | 02:10 | |
| *** tinwood has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-nova | 02:11 | |
| *** owalsh_ has joined #openstack-nova | 02:14 | |
| *** awaugama has quit IRC | 02:14 | |
| *** suresh12 has quit IRC | 02:14 | |
| *** mdrabe has quit IRC | 02:15 | |
| *** mdrabe has joined #openstack-nova | 02:16 | |
| *** owalsh has quit IRC | 02:17 | |
| *** mriedem has quit IRC | 02:22 | |
| *** germs has quit IRC | 02:23 | |
| *** germs has joined #openstack-nova | 02:24 | |
| *** mdrabe has quit IRC | 02:24 | |
| *** mdrabe has joined #openstack-nova | 02:28 | |
| *** claudiub|2 has quit IRC | 02:28 | |
| *** chyka has joined #openstack-nova | 02:33 | |
| *** mdrabe has quit IRC | 02:34 | |
| gmann | alex_xu_: ping | 02:34 |
| *** mdrabe has joined #openstack-nova | 02:37 | |
| *** chyka has quit IRC | 02:38 | |
| *** sridharg has joined #openstack-nova | 02:40 | |
| *** mdrabe has quit IRC | 02:41 | |
| *** david-lyle has quit IRC | 02:42 | |
| *** david-lyle has joined #openstack-nova | 02:42 | |
| *** ekhugen has quit IRC | 02:45 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (3) https://review.openstack.org/559258 | 02:45 |
| *** egarbade has quit IRC | 02:45 | |
| *** patriciadomin has quit IRC | 02:45 | |
| *** Spazmotic has joined #openstack-nova | 02:49 | |
| *** Spaz-Work has quit IRC | 02:49 | |
| *** psachin` has joined #openstack-nova | 02:49 | |
| *** ekhugen has joined #openstack-nova | 02:50 | |
| *** patriciadomin has joined #openstack-nova | 02:50 | |
| *** egarbade has joined #openstack-nova | 02:50 | |
| *** _ix has joined #openstack-nova | 02:51 | |
| *** mdrabe has joined #openstack-nova | 02:51 | |
| *** janki has joined #openstack-nova | 02:52 | |
| *** _ix has quit IRC | 02:55 | |
| *** mdrabe has quit IRC | 02:56 | |
| *** _ix has joined #openstack-nova | 02:56 | |
| *** yamamoto has joined #openstack-nova | 02:56 | |
| *** yamamoto has quit IRC | 03:01 | |
| *** mdrabe has joined #openstack-nova | 03:01 | |
| *** salv-orlando has joined #openstack-nova | 03:03 | |
| *** gouthamr has quit IRC | 03:05 | |
| *** gjayavelu has joined #openstack-nova | 03:06 | |
| *** salv-orlando has quit IRC | 03:08 | |
| *** mdrabe has quit IRC | 03:09 | |
| *** lbragstad has joined #openstack-nova | 03:11 | |
| *** fragatina has joined #openstack-nova | 03:25 | |
| *** andreas_s has joined #openstack-nova | 03:25 | |
| *** fragatin_ has joined #openstack-nova | 03:26 | |
| *** gjayavelu has quit IRC | 03:26 | |
| *** fragatina has quit IRC | 03:30 | |
| *** abhishekk has joined #openstack-nova | 03:30 | |
| *** andreas_s has quit IRC | 03:30 | |
| *** nicolasbock has quit IRC | 03:39 | |
| *** annp has quit IRC | 03:39 | |
| *** hongbin has quit IRC | 03:41 | |
| *** armaan has quit IRC | 03:45 | |
| *** armaan has joined #openstack-nova | 03:46 | |
| *** suresh12 has joined #openstack-nova | 03:48 | |
| *** links has joined #openstack-nova | 03:48 | |
| *** annp has joined #openstack-nova | 03:49 | |
| *** harlowja has joined #openstack-nova | 03:53 | |
| *** vladikr has quit IRC | 03:56 | |
| *** yamamoto has joined #openstack-nova | 03:57 | |
| *** yamamoto has quit IRC | 04:02 | |
| *** voelzmo has quit IRC | 04:04 | |
| *** salv-orlando has joined #openstack-nova | 04:04 | |
| *** rcernin has quit IRC | 04:04 | |
| *** ratailor has joined #openstack-nova | 04:05 | |
| *** germs has quit IRC | 04:07 | |
| *** salv-orlando has quit IRC | 04:09 | |
| *** udesale has joined #openstack-nova | 04:12 | |
| *** udesale has quit IRC | 04:13 | |
| *** udesale has joined #openstack-nova | 04:13 | |
| *** lbragstad has quit IRC | 04:18 | |
| *** rcernin has joined #openstack-nova | 04:19 | |
| *** sree has joined #openstack-nova | 04:33 | |
| homeski | is osprofiler working in Nova --Newton? | 04:36 |
| homeski | can't find any docs on it .. | 04:37 |
| *** armaan has quit IRC | 04:43 | |
| *** lpetrut has joined #openstack-nova | 04:48 | |
| *** armaan has joined #openstack-nova | 04:52 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_user_data.py https://review.openstack.org/559264 | 04:53 |
| *** yamamoto has joined #openstack-nova | 04:58 | |
| *** Dinesh_Bhor has quit IRC | 05:01 | |
| *** salv-orlando has joined #openstack-nova | 05:02 | |
| *** abhishekk has quit IRC | 05:02 | |
| *** yamamoto has quit IRC | 05:04 | |
| *** Dinesh_Bhor has joined #openstack-nova | 05:04 | |
| *** _ix has quit IRC | 05:05 | |
| *** harlowja has quit IRC | 05:13 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/api/openstack/compute/test_hosts.py https://review.openstack.org/559266 | 05:23 |
| *** mikal_ has joined #openstack-nova | 05:23 | |
| *** sar has quit IRC | 05:24 | |
| *** lpetrut has quit IRC | 05:26 | |
| *** mikal has quit IRC | 05:27 | |
| *** tbachman has quit IRC | 05:28 | |
| *** jchhatbar has joined #openstack-nova | 05:30 | |
| *** bhagyashris has joined #openstack-nova | 05:31 | |
| *** tbachman has joined #openstack-nova | 05:32 | |
| *** bhagyashri_s has quit IRC | 05:32 | |
| *** moshele has joined #openstack-nova | 05:33 | |
| *** janki has quit IRC | 05:33 | |
| *** Zames has joined #openstack-nova | 05:34 | |
| *** Zames has quit IRC | 05:37 | |
| *** yamamoto has joined #openstack-nova | 05:40 | |
| *** ShilpaSD has joined #openstack-nova | 05:43 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/api/*/test_instance_actions.py https://review.openstack.org/559269 | 05:45 |
| *** Shilpa has quit IRC | 05:46 | |
| *** Zames has joined #openstack-nova | 05:48 | |
| *** Zames has quit IRC | 05:52 | |
| *** suresh12 has quit IRC | 05:56 | |
| *** suresh12 has joined #openstack-nova | 06:02 | |
| *** Dinesh__Bhor has joined #openstack-nova | 06:04 | |
| *** Dinesh_Bhor has quit IRC | 06:04 | |
| *** jaosorior has joined #openstack-nova | 06:05 | |
| *** lajoskatona has joined #openstack-nova | 06:05 | |
| *** suresh12 has quit IRC | 06:07 | |
| *** sahid has joined #openstack-nova | 06:18 | |
| *** sree has quit IRC | 06:19 | |
| *** sridharg has quit IRC | 06:23 | |
| *** sridharg has joined #openstack-nova | 06:24 | |
| *** sar has joined #openstack-nova | 06:28 | |
| *** jistr has quit IRC | 06:30 | |
| *** jistr has joined #openstack-nova | 06:32 | |
| *** phuongnh has quit IRC | 06:35 | |
| *** phuongnh has joined #openstack-nova | 06:37 | |
| *** Zames has joined #openstack-nova | 06:38 | |
| *** andreas_s has joined #openstack-nova | 06:40 | |
| *** andreas_s_ has joined #openstack-nova | 06:41 | |
| *** ccamacho has quit IRC | 06:43 | |
| *** markvoelker has quit IRC | 06:43 | |
| *** markvoelker has joined #openstack-nova | 06:44 | |
| *** andreas_s has quit IRC | 06:45 | |
| *** phuongnh has quit IRC | 06:46 | |
| *** phuongnh has joined #openstack-nova | 06:47 | |
| *** pcaruana has joined #openstack-nova | 06:47 | |
| *** markvoelker has quit IRC | 06:48 | |
| *** damien_r has joined #openstack-nova | 06:50 | |
| *** belmoreira has joined #openstack-nova | 06:53 | |
| *** ccamacho has joined #openstack-nova | 06:53 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/548772 | 06:53 |
| *** imacdonn has quit IRC | 07:04 | |
| *** imacdonn has joined #openstack-nova | 07:05 | |
| *** phuongnh has quit IRC | 07:08 | |
| *** phuongnh has joined #openstack-nova | 07:09 | |
| kashyap | zigo: Thanks. Will look in a few. | 07:11 |
| *** ralonsoh has joined #openstack-nova | 07:17 | |
| openstackgerrit | Roman Dobosz proposed openstack/nova master: Pass hosts or nodes to HostStates update method depending on a driver. https://review.openstack.org/556761 | 07:19 |
| openstackgerrit | Roman Dobosz proposed openstack/nova master: Added weight for aggregate soft (anti) affinity. https://review.openstack.org/556762 | 07:19 |
| openstackgerrit | Roman Dobosz proposed openstack/nova master: Added node field for InstanceGroup objects https://review.openstack.org/559284 | 07:19 |
| *** tesseract has joined #openstack-nova | 07:20 | |
| *** phuongnh has quit IRC | 07:23 | |
| *** phuongnh has joined #openstack-nova | 07:23 | |
| *** armaan has quit IRC | 07:33 | |
| *** armaan has joined #openstack-nova | 07:33 | |
| *** Zames has quit IRC | 07:35 | |
| *** stakeda has joined #openstack-nova | 07:35 | |
| *** threestrands has quit IRC | 07:36 | |
| *** rcernin has quit IRC | 07:37 | |
| *** phuongnh has quit IRC | 07:40 | |
| *** AlexeyAbashkin has joined #openstack-nova | 07:41 | |
| *** amoralej|off is now known as amoralej | 07:45 | |
| *** tssurya has joined #openstack-nova | 07:48 | |
| *** moshele has quit IRC | 07:48 | |
| *** salv-orlando has quit IRC | 07:49 | |
| *** salv-orlando has joined #openstack-nova | 07:49 | |
| *** phuongnh has joined #openstack-nova | 07:52 | |
| *** salv-orlando has quit IRC | 07:54 | |
| *** jpena|off is now known as jpena | 07:54 | |
| *** ratailor_ has joined #openstack-nova | 07:55 | |
| *** ratailor has quit IRC | 07:58 | |
| *** voelzmo has joined #openstack-nova | 08:02 | |
| *** takashin has left #openstack-nova | 08:02 | |
| *** alexchadin has joined #openstack-nova | 08:02 | |
| *** mgoddard has joined #openstack-nova | 08:03 | |
| *** rmart04 has joined #openstack-nova | 08:04 | |
| *** jogo has quit IRC | 08:06 | |
| *** ratailor__ has joined #openstack-nova | 08:07 | |
| *** armaan has quit IRC | 08:08 | |
| *** armaan has joined #openstack-nova | 08:08 | |
| *** ratailor_ has quit IRC | 08:09 | |
| *** avolkov has joined #openstack-nova | 08:13 | |
| *** lucas-afk is now known as lucasagomes | 08:13 | |
| *** derekh has joined #openstack-nova | 08:15 | |
| *** suresh12 has joined #openstack-nova | 08:16 | |
| *** hoangcx has quit IRC | 08:19 | |
| *** jogo has joined #openstack-nova | 08:19 | |
| *** suresh12 has quit IRC | 08:21 | |
| *** lajoskatona has quit IRC | 08:25 | |
| *** sree has joined #openstack-nova | 08:29 | |
| kaisers1 | mikal_: ping | 08:32 |
| *** mdbooth has joined #openstack-nova | 08:33 | |
| *** voelzmo has quit IRC | 08:37 | |
| *** lajoskatona has joined #openstack-nova | 08:42 | |
| *** markvoelker has joined #openstack-nova | 08:44 | |
| kashyap | It's often best to just ask the question -- when the person wakes up, they could respond directly to the question. | 08:46 |
| *** armaan has quit IRC | 08:49 | |
| *** armaan has joined #openstack-nova | 08:50 | |
| *** salv-orlando has joined #openstack-nova | 08:51 | |
| *** alexchadin has quit IRC | 08:53 | |
| *** salv-orl_ has joined #openstack-nova | 08:54 | |
| *** voelzmo has joined #openstack-nova | 08:56 | |
| *** salv-orlando has quit IRC | 08:57 | |
| *** voelzmo has quit IRC | 09:01 | |
| *** ratailor_ has joined #openstack-nova | 09:03 | |
| *** ralonsoh_ has joined #openstack-nova | 09:05 | |
| *** ratailor__ has quit IRC | 09:05 | |
| *** ralonsoh has quit IRC | 09:05 | |
| *** alexchadin has joined #openstack-nova | 09:07 | |
| *** cdent has joined #openstack-nova | 09:14 | |
| *** jchhatbar has quit IRC | 09:15 | |
| *** janki has joined #openstack-nova | 09:15 | |
| *** markvoelker has quit IRC | 09:18 | |
| *** fragatin_ has quit IRC | 09:20 | |
| *** Sandy619 has joined #openstack-nova | 09:20 | |
| *** jchhatbar has joined #openstack-nova | 09:20 | |
| *** fragatina has joined #openstack-nova | 09:20 | |
| *** janki has quit IRC | 09:23 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Cleanup patch for the cell-disable series https://review.openstack.org/559299 | 09:27 |
| *** claudiub|2 has joined #openstack-nova | 09:30 | |
| *** sdague has joined #openstack-nova | 09:32 | |
| *** sar has quit IRC | 09:34 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add missing encryption_secret_uuid tests https://review.openstack.org/540679 | 09:34 |
| *** owalsh_ is now known as owalsh | 09:35 | |
| *** StevenK has quit IRC | 09:39 | |
| *** StevenK has joined #openstack-nova | 09:42 | |
| *** armaan has quit IRC | 09:42 | |
| *** armaan has joined #openstack-nova | 09:43 | |
| kashyap | zigo: Hi there | 09:44 |
| kashyap | zigo: I'm just about to respond to your email, although it's easy to use the versions in Stretch | 09:45 |
| zigo | kashyap: Great ! :) | 09:45 |
| kashyap | zigo: Thanks for the compilation trying; it looks like a fairly small one. | 09:45 |
| kashyap | zigo: But -- it would be nice to use 3.2.0, as I just spent 30 minutes comparing the release notes of 3.0.0 and 3.2.0 | 09:46 |
| zigo | As I wrote, right now, not much of a problem. In a year from now, I don't know ! | 09:46 |
| zigo | kashyap: Oh, interesting. What's the added features? | 09:46 |
| kashyap | That's why instead of such new versions from Buster / Sid | 09:46 |
| kashyap | zigo: So, one of the core things we'd get with 3.2.0 is this: | 09:46 |
| kashyap | zigo: https://www.redhat.com/archives/libvir-list/2017-February/msg01295.html -- "Detect host CPU model by askingQEMU on x86_64" | 09:47 |
| kashyap | zigo: It fixes a ton of bugs in that area of CPU model comparision | 09:47 |
| zigo | Oh ... | 09:47 |
| zigo | I see, interesting indeed. | 09:47 |
| zigo | Though that's bugs in libvirt itself, not really a new API feature, right? | 09:48 |
| kashyap | zigo: Yes, indeed | 09:49 |
| kashyap | zigo: Bug hang on, let me look at the details | 09:49 |
| zigo | What I was refering was missing feature on the way of the enhancement of Nova. | 09:49 |
| kashyap | zigo: Some new APIs too | 09:49 |
| zigo | Is there any incompatibilities? | 09:50 |
| zigo | If there are bugfixes, then I can try to do my best to get the latest version from Buster backported. But that's a whole of a different story to have this backport being mandatory because of API breaks. | 09:51 |
| kashyap | zigo: That's a broad question, you mean API incompat? | 09:51 |
| kashyap | zigo: No, not really | 09:51 |
| zigo | Yeah... | 09:51 |
| kashyap | zigo: Isn't it too much to backport the 'Buster' versions? Rather than introducing the bit more conservative 3.2.0 and 2.9.0 (QEMU)? | 09:52 |
| zigo | If 3.2.0 is still backward compatible with 3.0.0, then just require 3.0.0 and I'll do what I can to get the backport done. It's just nicer to not have it mandatory. | 09:52 |
| kashyap | zigo: libvirt guarantees API compatibility "forever" | 09:52 |
| zigo | Oh ! | 09:52 |
| zigo | So why are you willing to even require anything? | 09:52 |
| kashyap | Wait, let me retract that :-) | 09:53 |
| kashyap | Read here about API guarantees: https://libvirt.org/support.html | 09:53 |
| * zigo reads | 09:53 | |
| zigo | Another thing: isn't Nova only interested in the python-libvirt API, rather than the engine behind? | 09:54 |
| kashyap | zigo: I meant, the "forever" part applies to ABI compatibility from the primary public API | 09:54 |
| zigo | " IOW they will be ABI compatible forever." <--- Read this, indeed ! | 09:55 |
| zigo | That's the Linux kernel userland API policy (though here, ABI, which is even nicer). | 09:55 |
| *** alexchadin has quit IRC | 09:55 | |
| kashyap | zigo: Each Compute node runs libvirt daemon | 09:55 |
| zigo | Linuz told Debian folks (at Debconf in portland) that he didn't understand why it wasn't a common practice on all the Linux echosystem, and I very much agree ! :) | 09:56 |
| zigo | Ok, so libvirt C lib is super-stable, but is there anything new in 3.2.0 that you would need that isn't in 3.0.0 ? | 09:58 |
| *** dtantsur|afk is now known as dtantsur | 09:59 | |
| kashyap | zigo: Yeah, like I said 3.2.0 libvirt (in conjunction with QEMU 2.9.0) by default saves a ton of headache in the CPU modelling area | 09:59 |
| *** esberglu has joined #openstack-nova | 09:59 | |
| kashyap | zigo: But if it's too much time and effort, it's alright, can settle on 3.0.0 & 2.8.0 | 10:02 |
| * kashyap responds on the list; so others can catch up too | 10:03 | |
| *** esberglu has quit IRC | 10:04 | |
| zigo | Great, thanks. Just do that, and advise for 3.0.0, then I'll keep in mind that backporting is prefered. | 10:04 |
| zigo | The real issue isn't "too much effort", right now, it's not. The issue is: I don't know if it's going to be hard or even impossible in one year from now. | 10:04 |
| zigo | Anyway, thanks so much for your care. | 10:04 |
| * zigo goes for lunch | 10:04 | |
| * kashyap too | 10:06 | |
| kashyap | zigo: Email sent. | 10:07 |
| *** ralonsoh_ has quit IRC | 10:08 | |
| *** markvoelker has joined #openstack-nova | 10:15 | |
| *** Dinesh__Bhor has quit IRC | 10:16 | |
| *** stakeda has quit IRC | 10:23 | |
| *** armaan has quit IRC | 10:23 | |
| *** armaan has joined #openstack-nova | 10:24 | |
| *** sar has joined #openstack-nova | 10:24 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Move test_report_client out of placement namespace https://review.openstack.org/558911 | 10:26 |
| *** nicolasbock has joined #openstack-nova | 10:27 | |
| *** jistr has quit IRC | 10:27 | |
| *** annp has quit IRC | 10:28 | |
| *** armaan has quit IRC | 10:28 | |
| *** armaan has joined #openstack-nova | 10:28 | |
| *** jistr has joined #openstack-nova | 10:36 | |
| *** tesseract has quit IRC | 10:37 | |
| *** tesseract has joined #openstack-nova | 10:38 | |
| *** sahid has quit IRC | 10:40 | |
| *** _ix has joined #openstack-nova | 10:48 | |
| *** markvoelker has quit IRC | 10:49 | |
| *** tbachman has quit IRC | 10:54 | |
| *** tesseract-RH has joined #openstack-nova | 10:59 | |
| *** tesseract has quit IRC | 10:59 | |
| *** lajoskatona has quit IRC | 11:05 | |
| *** alexchadin has joined #openstack-nova | 11:09 | |
| *** damien_r has quit IRC | 11:12 | |
| *** phuongnh has quit IRC | 11:16 | |
| *** cdent has quit IRC | 11:17 | |
| *** lajoskatona has joined #openstack-nova | 11:19 | |
| *** Roamer`_ is now known as Roamer` | 11:20 | |
| *** cdent has joined #openstack-nova | 11:22 | |
| *** Sandy619 has quit IRC | 11:26 | |
| *** lucasagomes is now known as lucas-hungry | 11:28 | |
| *** _ix has quit IRC | 11:33 | |
| *** jpena is now known as jpena|lunch | 11:38 | |
| *** tiendc has quit IRC | 11:42 | |
| *** gouthamr has joined #openstack-nova | 11:45 | |
| *** markvoelker has joined #openstack-nova | 11:46 | |
| *** sree has quit IRC | 11:49 | |
| *** sree has joined #openstack-nova | 11:50 | |
| *** sree has quit IRC | 11:50 | |
| *** sree has joined #openstack-nova | 11:51 | |
| *** sree_ has joined #openstack-nova | 11:52 | |
| *** sree_ is now known as Guest13036 | 11:52 | |
| *** hshiina has quit IRC | 11:53 | |
| *** armaan has quit IRC | 11:54 | |
| *** sree has quit IRC | 11:56 | |
| kashyap | Anyone also seeing these timeout issues: | 11:58 |
| kashyap | http://logs.openstack.org/84/534384/24/check/neutron-tempest-multinode-full/594b6f8/job-output.txt.gz#_2018-04-06_05_44_18_306935 | 11:58 |
| efried | kashyap: Things have been pretty flaky the last couple of days. I haven't been paying close enough attention to say whether it's that same issue over and over, but could very well be. | 12:01 |
| efried | I bet mriedem knows more. | 12:01 |
| *** jistr is now known as jistr|mtg | 12:02 | |
| * kashyap nods | 12:03 | |
| kashyap | I just wanted to be mindful before I blindly hit 'recheck' | 12:03 |
| *** ratailor_ has quit IRC | 12:04 | |
| *** armaan has joined #openstack-nova | 12:05 | |
| *** udesale has quit IRC | 12:05 | |
| *** liverpooler has joined #openstack-nova | 12:05 | |
| cdent | efried, edleafe : a) what day is it? b) do either of you recall what the state of multiple member_of params is? | 12:06 |
| *** efried is now known as fried_rice | 12:07 | |
| jroll | kashyap: there's been something about a certain provider giving lots of timeouts | 12:07 |
| kashyap | jroll: Ah, I see. | 12:07 |
| kashyap | That explains the 'recheck's by others while I was asleep | 12:07 |
| * jroll tries to remember the name, not sure if it's fixed yet | 12:07 | |
| fried_rice | a) Thanks cdent; b) Spec merged: https://review.openstack.org/#/c/555413/ original series nearing completion, new work not started. | 12:08 |
| fried_rice | as far as I know. | 12:08 |
| jroll | ah yes, limestone, which is the provider for that job | 12:08 |
| cdent | thanks fried_rice | 12:09 |
| fried_rice | cdent: Original bp topic https://review.openstack.org/#/q/topic:bp/alloc-candidates-member-of+(status:open+OR+status:merged) | 12:09 |
| cdent | yeah, had that, was more a question of "is someone on the hook for that?" | 12:09 |
| fried_rice | cdent: Not sure, actually. Either dansmith or edleafe I would imagine. | 12:14 |
| cdent | fried_rice: I've noted the need for someone in the weekly report | 12:14 |
| fried_rice | ack | 12:14 |
| *** suresh12 has joined #openstack-nova | 12:17 | |
| *** markvoelker has quit IRC | 12:19 | |
| *** pooja_jadhav has quit IRC | 12:21 | |
| *** suresh12 has quit IRC | 12:21 | |
| *** pooja_jadhav has joined #openstack-nova | 12:22 | |
| *** markvoelker has joined #openstack-nova | 12:24 | |
| *** _ix has joined #openstack-nova | 12:25 | |
| *** swamireddy has quit IRC | 12:29 | |
| *** lucas-hungry is now known as lucasagomes | 12:30 | |
| *** jistr|mtg is now known as jistr | 12:31 | |
| *** mriedem has joined #openstack-nova | 12:32 | |
| *** armaan has quit IRC | 12:39 | |
| *** armaan has joined #openstack-nova | 12:39 | |
| *** jpena|lunch is now known as jpena | 12:41 | |
| *** armaan_ has joined #openstack-nova | 12:42 | |
| *** armaan_ has quit IRC | 12:44 | |
| *** armaan_ has joined #openstack-nova | 12:45 | |
| *** armaan has quit IRC | 12:45 | |
| *** odyssey4me has quit IRC | 12:47 | |
| *** odyssey4me has joined #openstack-nova | 12:47 | |
| *** jmlowe has quit IRC | 12:48 | |
| *** pchavva has joined #openstack-nova | 12:53 | |
| mriedem | arvindn05: dansmith: i just thought of something fun with the image-based traits spec, | 12:53 |
| mriedem | if we rebuild with a new image that has some required traits on it, | 12:53 |
| *** _ix has quit IRC | 12:53 | |
| mriedem | will we run through placement during scheduling again? | 12:54 |
| mriedem | i.e. https://review.openstack.org/#/c/546357/ | 12:54 |
| *** tbachman has joined #openstack-nova | 12:54 | |
| mriedem | we flagged our individual filters on what can run during rebuild based on quantitative resources or not, but traits on the image would be...qualitative? | 12:54 |
| *** ArchiFleKs has joined #openstack-nova | 12:55 | |
| *** baoli has joined #openstack-nova | 12:56 | |
| *** baoli has quit IRC | 12:58 | |
| *** artom has quit IRC | 12:58 | |
| *** baoli has joined #openstack-nova | 12:58 | |
| mriedem | arvindn05: i left a comment in the spec https://review.openstack.org/#/c/541507/9/specs/rocky/approved/glance-image-traits.rst@151 - we'll likely need an amendment for that | 12:59 |
| *** edleafe is now known as figleaf | 13:01 | |
| *** avolkov has quit IRC | 13:01 | |
| ArchiFleKs | Hi, I have a weird issue, on a compute node I cannot launch nova-compute, I tried "/openstack/venvs/nova-16.0.10/bin/python -v /openstack/venvs/nova-16.0.10/bin/nova-compute --debug" manually and it just exit, I have no log on the nova side, just python loading : http://paste.openstack.org/show/718596/ (I'm running openstack-ansible) | 13:01 |
| *** Guest13036 has quit IRC | 13:02 | |
| *** sree has joined #openstack-nova | 13:02 | |
| *** ralonsoh has joined #openstack-nova | 13:02 | |
| *** alexchadin has quit IRC | 13:02 | |
| * figleaf reads back | 13:03 | |
| figleaf | cdent: fried_rice: I believe that dansmith was working on that | 13:03 |
| fried_rice | ack | 13:04 |
| figleaf | with the notion that member_of(a,b) is an OR and member_of(a)&member_of(b) is an AND | 13:04 |
| fried_rice | right. The spec update that dansmith merged should say that, I believe. (Haven't reread it since the review.) | 13:06 |
| *** alexchadin has joined #openstack-nova | 13:07 | |
| *** edmondsw has joined #openstack-nova | 13:07 | |
| *** avolkov has joined #openstack-nova | 13:08 | |
| *** egarbade has quit IRC | 13:09 | |
| *** david-lyle has quit IRC | 13:10 | |
| *** patriciadomin has quit IRC | 13:10 | |
| *** derekh has quit IRC | 13:10 | |
| *** ekhugen has quit IRC | 13:11 | |
| mriedem | claudiub|2: i'm seeing a failed hyperv CI run from last night, http://cloudbase-ci.com/nova/324720/5/Hyper-V_logs/192.168.3.143-compute01/neutron-hyperv-agent.log.gz | 13:12 |
| mriedem | looks like this is the culprit | 13:12 |
| mriedem | AttributeError: 'NoneType' object has no attribute 'startswith' | 13:12 |
| mriedem | in File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1864, in _device_to_port_id | 13:12 |
| *** derekh has joined #openstack-nova | 13:13 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: add lower-constraints job https://review.openstack.org/555961 | 13:13 |
| jroll | is there work being done (already done?) to move the instance groups (and thus affinity) work to use placement aggregates instead of host aggregates? | 13:13 |
| mriedem | instance groups don't use aggregates | 13:13 |
| *** sree has quit IRC | 13:13 | |
| jroll | mmm, maybe I'm misinformed then - let's just say affinity | 13:14 |
| mriedem | figleaf had a distance etherpad i think for ideas on modeling affinity in placement | 13:14 |
| mriedem | but there is no planned work for this in rocky as far as i know, we didn't talk about it at the ptg | 13:14 |
| jroll | okay, thanks mriedem | 13:15 |
| *** cdent has quit IRC | 13:15 | |
| figleaf | mriedem: jroll: I think the closest we are getting to affinity is the member_of work, using placement aggs | 13:16 |
| mriedem | placement aggregates are on resource providers though, and server group affinity filtering is on instances, so i'm not sure how that translates | 13:17 |
| mriedem | since placement doesn't have a concept of consumer groups | 13:17 |
| *** avolkov has quit IRC | 13:17 | |
| *** jmlowe has joined #openstack-nova | 13:18 | |
| *** fragatina has quit IRC | 13:18 | |
| jroll | right, though I don't care as much about the server group concept as I do aggregate affinity | 13:18 |
| *** felipemonteiro_ has joined #openstack-nova | 13:19 | |
| jroll | figleaf: that is not exposed to the user (yet), correct? | 13:19 |
| figleaf | mriedem: yeah, that's what I meant by "the closest we are getting". We're not re-creating every detail of Nova (at least not this cycle) | 13:19 |
| figleaf | jroll: no, not yet | 13:19 |
| jroll | cool. thanks | 13:19 |
| jroll | :) | 13:19 |
| *** armaan_ has quit IRC | 13:20 | |
| *** fragatina has joined #openstack-nova | 13:20 | |
| mriedem | claudiub|2: https://bugs.launchpad.net/neutron/+bug/1761748 - not sure if that's what is causing the nova-compute service to say it's down in this CI run though | 13:21 |
| openstack | Launchpad bug 1761748 in neutron "hyperv: Unable to get ports details for devices: AttributeError: 'NoneType' object has no attribute 'startswith'" [Undecided,New] | 13:21 |
| mriedem | jroll: ok, i don't know what aggregate affinity is then :) | 13:21 |
| mriedem | i think that's an oath only thing right now | 13:21 |
| mriedem | and https://review.openstack.org/#/c/529135/ | 13:21 |
| jroll | mriedem: well, ironic users in general want to create groups of baremetal machines based on locality/power/rack/etc, and do affinity between those groups. aggregates seem the most sensible | 13:22 |
| sean-k-mooney | mriedem: jroll i would assume just passing member_of as part of resouce request to allocation_candiades | 13:22 |
| *** felipemonteiro__ has joined #openstack-nova | 13:23 | |
| *** cdent has joined #openstack-nova | 13:23 | |
| mriedem | jroll: i think that's this https://review.openstack.org/#/c/529135/ | 13:23 |
| jroll | sean-k-mooney: yes, that's what the spec provides, I'm trying to map that in my head to the future, as /allocation_candidates is not exposed to users | 13:23 |
| mriedem | jroll: currently none of placement is exposed to users, per policy anyway | 13:24 |
| jroll | mriedem: yeah, I'm hearing that dan hates this | 13:24 |
| jroll | right | 13:24 |
| sean-k-mooney | mriedem: jroll e.g. Resouce1:PCPU4,MEMORY_MB=1024 Resource1:member_of=<aggreagte uuid here> | 13:24 |
| jroll | I'm aware of this :) | 13:24 |
| *** david-lyle has joined #openstack-nova | 13:24 | |
| *** vladikr has joined #openstack-nova | 13:24 | |
| mriedem | jroll: L17 https://etherpad.openstack.org/p/nova-ptg-rocky-placement | 13:25 |
| sean-k-mooney | jroll: its not exposed to users but you can model this in the flavor | 13:25 |
| mriedem | i didn't think this was the same thing as the "allow host or node uuid for aggregates in the API" thing | 13:25 |
| mriedem | i thought that was a different spec, but couldn't find it | 13:25 |
| jroll | mriedem: right, saw the etherpad, wasn't in the session. I don't see an explicit f this in the etherpad, so maybe it'll still happen | 13:26 |
| jroll | trying to get a read on what to expect in the future, I guess | 13:26 |
| mriedem | i didn't like it either | 13:26 |
| mriedem | fwiw | 13:26 |
| *** felipemonteiro_ has quit IRC | 13:26 | |
| jroll | heh | 13:26 |
| jroll | sean-k-mooney: I'm not doing a flavor per system type per rack, but thanks :) | 13:27 |
| *** psachin` has quit IRC | 13:28 | |
| *** lbragstad has joined #openstack-nova | 13:28 | |
| mriedem | i remember one thing from the ptg was at least some sentiment that if we did do this in nova, we wouldn't want to munge the host and node concept into the existing host aggregate API today, since that could be really confusing | 13:28 |
| *** links has quit IRC | 13:28 | |
| mriedem | you'd need like a hypervisor (node) aggregate API or something so it's clear that it's different | 13:28 |
| sean-k-mooney | jroll: sure but somthing line Resouce1:PCPU4,MEMORY_MB=1024 Resource1:member_of=<aggreagte uuid here> should work the question then becomes how to get that to placement | 13:28 |
| mriedem | os-hypervisors is the API today for compute nodes | 13:28 |
| mriedem | os-services is the one for hosts | 13:29 |
| jroll | mriedem: yeah, totally understand that, I was fairly surprised when jay introduced that spec to be honest | 13:29 |
| mriedem | as for the comments in the etherpad about using the placement-req-filter stuff, the question is how do you transfer the user request to the nova scheduler, and i think that's what the policy part was for in jay's spec | 13:29 |
| sean-k-mooney | jroll: flavors is the only way that works today but we have discussed allowing resouce requests via neutron port and glance images too | 13:29 |
| *** sar has quit IRC | 13:30 | |
| jroll | sean-k-mooney: I really don't want to design a feature over irc right now, sorry | 13:30 |
| mriedem | so i'm not sure how we leverage placement-req-filter with ironic node aggregates in placement yet | 13:31 |
| *** avolkov has joined #openstack-nova | 13:31 | |
| sean-k-mooney | jroll: no worries just giving one option. i think we agree this is a valid usecase that should be supported. how is a question that can be adressed later | 13:31 |
| mriedem | not sure if traits would get us there, traits are the new metadata bastard child | 13:31 |
| *** esberglu has joined #openstack-nova | 13:32 | |
| *** _ix has joined #openstack-nova | 13:32 | |
| *** alexchadin has quit IRC | 13:33 | |
| openstackgerrit | Tyler Blakeslee proposed openstack/nova master: Use six.text_type instead of str in NovaException __repr__ https://review.openstack.org/559169 | 13:33 |
| jroll | mriedem: yeah, makes sense, thank you | 13:33 |
| sean-k-mooney | mriedem: i think jay would argue this is not a qualitive capablity of the resouce accounting so traits are not the best way but use it would work with traits | 13:33 |
| sean-k-mooney | e.g. custom_rack1 but again that is not user facing | 13:34 |
| *** alexchadin has joined #openstack-nova | 13:34 | |
| *** lbragstad has quit IRC | 13:35 | |
| *** stephenfin is now known as finucannot | 13:36 | |
| mriedem | jroll: i think i get what you meant by "alloc candidates aren't exposed to the user" - in that there is no user-facing compute API to translate a request for node aggregates (in placement) to the nova scheduler | 13:37 |
| *** _ix has quit IRC | 13:38 | |
| *** jaypipes has joined #openstack-nova | 13:38 | |
| *** jaypipes is now known as leakypipes | 13:38 | |
| cdent | mriedem: traits can take us anywhere we want to go, if we're willing. Such is the danger of traits. | 13:38 |
| finucannot | leakypipes: I was literally just tab completing your name. Morning o/ | 13:38 |
| jroll | mriedem: yes, exactly. maybe there's a new api eventually, maybe instance groups affinity moves to use placement aggregates, maybe it's traits. dunno, need to think about it | 13:38 |
| finucannot | leakypipes: Does this sound like something you could tackle today? https://review.openstack.org/#/c/553072/ | 13:39 |
| mriedem | jroll: yeah, needs some thought - gut feeling is traits isn't the answer here | 13:39 |
| jroll | agree | 13:39 |
| mriedem | munging aggregates with traits i mean | 13:39 |
| mriedem | cdent: yes that's readily apparent already :) | 13:39 |
| *** lbragstad has joined #openstack-nova | 13:39 | |
| mriedem | bbiab | 13:40 |
| *** _ix has joined #openstack-nova | 13:40 | |
| *** dansmith is now known as superdan | 13:40 | |
| * jroll makes a note to smack people downstream that say "oh we can hack in a patch to leverage traits for this" when it inevitably happens | 13:40 | |
| cdent | jroll: meh, if people want that kind of rope, maybe that's what the rope should do? | 13:40 |
| jroll | cdent: not when they leave me to clean up the mess :P | 13:41 |
| cdent | traits are effectively the same thing as labels in k8s, aren't they? | 13:41 |
| jroll | I mean my specific downstream, to be clear :) | 13:41 |
| cdent | that's a good point. If it's you on the hook to suffer, slap away | 13:41 |
| *** mlavalle has joined #openstack-nova | 13:44 | |
| *** david-lyle has quit IRC | 13:44 | |
| *** alexchadin has quit IRC | 13:45 | |
| *** awaugama has joined #openstack-nova | 13:52 | |
| *** swamireddy has joined #openstack-nova | 13:52 | |
| *** Zames has joined #openstack-nova | 13:55 | |
| *** sridharg has quit IRC | 13:55 | |
| *** bnemec has joined #openstack-nova | 13:59 | |
| *** bnemec is now known as beekneemech | 13:59 | |
| *** maciejjozefczyk has joined #openstack-nova | 14:01 | |
| *** sree has joined #openstack-nova | 14:01 | |
| *** melwitt is now known as jgwentworth | 14:02 | |
| *** lajoskatona has quit IRC | 14:02 | |
| leakypipes | finucannot: morning sir! :) | 14:02 |
| * leakypipes coffee-fills | 14:02 | |
| *** Zames has quit IRC | 14:04 | |
| *** artom has joined #openstack-nova | 14:05 | |
| *** sree has quit IRC | 14:05 | |
| *** artom_ has joined #openstack-nova | 14:08 | |
| *** maciejjozefczyk has quit IRC | 14:09 | |
| *** adriano has quit IRC | 14:09 | |
| *** adriano__ has joined #openstack-nova | 14:10 | |
| *** artom has quit IRC | 14:10 | |
| *** germs has joined #openstack-nova | 14:10 | |
| *** germs has quit IRC | 14:11 | |
| *** germs has joined #openstack-nova | 14:11 | |
| kashyap | mriedem: I think you're the closest to ACK this: https://review.openstack.org/#/c/544238/ ("libvirt: Block swapping to an encrypted volume when using QEMU to decrypt") | 14:11 |
| kashyap | Trivial diff-stat, though. | 14:11 |
| *** hongbin has joined #openstack-nova | 14:12 | |
| mriedem | cdent: the other thing with giving that rope is people build solutions off it, and then when the time comes that we want to change anything related to it, we break those solutions | 14:14 |
| mriedem | and if it's fairly nebulous we don't really know how anyone is using it | 14:14 |
| cdent | traits are part of the (immutable?) api of placement | 14:15 |
| cdent | so how it is used in placement is stuck | 14:15 |
| cdent | how it is used in nova, though, that's different | 14:15 |
| *** germs has quit IRC | 14:15 | |
| mriedem | with microversions the placement api isn't immutable | 14:15 |
| cdent | (not disagreeing with you) | 14:15 |
| mriedem | but yes i don't expect how placement handles traits to change much | 14:15 |
| cdent | it is in the sense that we'll never raise the minimum | 14:15 |
| mriedem | and yes how they are incorporated into nova is the messy part | 14:15 |
| mriedem | or could be messy | 14:16 |
| * cdent tries to think of some alliterative to say and gives up | 14:16 | |
| mriedem | right now traits-based scheduling from queens is pretty straight forward | 14:16 |
| mriedem | 'put me on a host with trait=foo, got it' | 14:16 |
| * cdent nods | 14:17 | |
| *** jogo has quit IRC | 14:19 | |
| *** jogo has joined #openstack-nova | 14:20 | |
| *** amodi has joined #openstack-nova | 14:22 | |
| *** Zames has joined #openstack-nova | 14:22 | |
| mriedem | mmedvede: i got a pkvm ci failure setting up devstack here https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/20/324720/5/check/tempest-dsvm-full-xenial/b59cb03/devstacklog.txt.gz | 14:22 |
| mriedem | Connecting to cirros-builder.novalocal (cirros-builder.novalocal)|9.114.111.40|:80... failed: No route to host. | 14:22 |
| mriedem | jgwentworth: fyi, i had emailed the virtuozzo team about their CI failing at least a week ago, so it's a known issue | 14:23 |
| mriedem | ImagePropertiesFilter always kicks out whatever their trying to use for an image | 14:23 |
| *** damien_r has joined #openstack-nova | 14:24 | |
| *** baoli has quit IRC | 14:24 | |
| *** Zames has quit IRC | 14:24 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip placement on rebuild in same host https://review.openstack.org/546357 | 14:25 |
| *** armaan has joined #openstack-nova | 14:26 | |
| *** r-daneel has joined #openstack-nova | 14:27 | |
| *** armaan has quit IRC | 14:30 | |
| *** hamzy has quit IRC | 14:30 | |
| *** udesale has joined #openstack-nova | 14:31 | |
| *** suresh12 has joined #openstack-nova | 14:31 | |
| odyssey4me | howdy everyone, thanks again for your help so far - we have an upgrade task failing which might be due to new changes in rocky... would this be a change in something that needs to happen? http://logs.openstack.org/78/558078/2/check/openstack-ansible-upgrade-ubuntu-xenial-nv/3dc2358/logs/ara-report/result/b9fe5b27-00e6-455b-9a10-71b2046663c5/ | 14:33 |
| edmondsw | fyi, someone is trying to add suppz support to the libvirt driver, but seems to be doing that off on their own | 14:33 |
| edmondsw | https://ask.openstack.org/en/question/113484/add-support-for-a-new-driver-in-nova/ | 14:33 |
| edmondsw | I tried to point them in the right direction | 14:33 |
| *** yamahata has quit IRC | 14:34 | |
| jgwentworth | mriedem: okay, so virtuozzo CI currently broken and being worked on | 14:34 |
| *** sree has joined #openstack-nova | 14:34 | |
| *** suresh12 has quit IRC | 14:35 | |
| *** hamzy has joined #openstack-nova | 14:36 | |
| mriedem | lyarwood: so in https://review.openstack.org/#/c/544238/4/nova/virt/libvirt/driver.py we also can't do the swap if both volumes are encrypted? | 14:36 |
| mriedem | jgwentworth: it's a known issue, i'm not sure how actively they are working to fix it | 14:36 |
| *** baoli has joined #openstack-nova | 14:37 | |
| mriedem | odyssey4me: that looks like the new 'disabled' colum in the nova_api.cell_mappings table, has the nova_api db been synced before running this? | 14:38 |
| lyarwood | mriedem: correct | 14:38 |
| *** udesale has quit IRC | 14:38 | |
| mriedem | tssurya: ^ | 14:38 |
| *** sree has quit IRC | 14:38 | |
| odyssey4me | mriedem well, that's a queens to rocky upgrade, so yes - it was done in the queens deployment | 14:39 |
| kashyap | lyarwood: Probably that should point also be documented as a note in the code? (With current limitations.) | 14:40 |
| mriedem | odyssey4me: if you're upgrading to rocky and you already created the cell0 mapping in queens, you don't need to do it again | 14:40 |
| mriedem | odyssey4me: although, map_cell0 should be smarter, it has a check for DBDuplicateEntry for running it when it already exists | 14:41 |
| jgwentworth | mriedem: okay. would you say it's too soon to mail the ML to describe the issue and ask if anyone on their side can look into it? | 14:41 |
| mriedem | jgwentworth: i emailed the vz CI email a couple of weeks ago, and got a reply this week, but deleted that reply | 14:42 |
| mriedem | i'd give it maybe another week? | 14:42 |
| jgwentworth | k, will do. thanks | 14:42 |
| mriedem | odyssey4me: so i don't think we're getting the duplicate entry b/c it's trying to create the cell0 mapping in the api db, and that fails because the object code doesn't match the api db schema | 14:42 |
| mriedem | odyssey4me: which tells me that nova-manage api_db sync hasn't been run on rocky yet | 14:43 |
| lyarwood | mriedem: thanks, btw https://review.openstack.org/#/c/543569/ is the change covering this all prior to Queens where we would hit https://bugs.launchpad.net/nova/+bug/1739593, it would be great to land this in stable/ocata and pike sometime soon to close that all out. | 14:43 |
| openstack | Launchpad bug 1739593 in OpenStack Security Advisory "Swapping encrypted volumes can lead to data loss and a possible compute host DOS attack (CVE-2017-18191)" [Undecided,Incomplete] | 14:43 |
| mriedem | lyarwood: i've tried to purge that from my memory | 14:44 |
| odyssey4me | mriedem hmm, IIRC on an upgrade the steps are different - we have this now, so you're right: https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml | 14:44 |
| *** david-lyle has joined #openstack-nova | 14:45 | |
| lyarwood | mriedem: you and me both, definitely a mistake on my part to label this as a security issue | 14:45 |
| *** lyan has joined #openstack-nova | 14:45 | |
| * lyarwood now has ~7 downstream bugs about this | 14:45 | |
| *** lyan is now known as Guest81836 | 14:45 | |
| mriedem | odyssey4me: so https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml is getting run all over again on an upgrade from queens to rocky? | 14:46 |
| *** suresh12 has joined #openstack-nova | 14:46 | |
| odyssey4me | mriedem yes, that sequence is run for greenfield, or upgrade - but the result of 'Get the api_db version' would be different if it was an upgrade | 14:46 |
| *** Spazmotic has quit IRC | 14:46 | |
| mriedem | odyssey4me: oh i see the problem https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml#L29 | 14:47 |
| mriedem | yeah looking at http://logs.openstack.org/78/558078/2/check/openstack-ansible-upgrade-ubuntu-xenial-nv/3dc2358/logs/ara-report/reports/68ffa953-cc0e-494b-89b9-dcf7ca4a8b59.html | 14:47 |
| mriedem | so "Perform a Nova API DB sync" doesn't get run | 14:47 |
| mriedem | that's a problem | 14:47 |
| odyssey4me | ok, but that gets run later | 14:47 |
| odyssey4me | IIRC this relates to the difference in procedure for greenfield vs upgrade | 14:48 |
| odyssey4me | this might only have been an ocata/pike thing though, when cells v2 was all new | 14:48 |
| mriedem | how did you upgrade from ocata to pike, or pike to queens? | 14:48 |
| mriedem | i guess we haven't added a new cell mappings field sine then | 14:48 |
| mriedem | so that's why | 14:48 |
| odyssey4me | this same procedure is used - ocata->pike: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/pike/tasks/nova_db_setup.yml | 14:48 |
| odyssey4me | pike->queens: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/queens/tasks/nova_db_setup.yml | 14:49 |
| *** baoli has quit IRC | 14:49 | |
| mriedem | yeah, we just have'nt added a new cell_mappings column in queens or pike | 14:49 |
| odyssey4me | aha, ok | 14:49 |
| mriedem | so, we can fix this in one of two ways: | 14:49 |
| *** baoli has joined #openstack-nova | 14:49 | |
| *** armaan has joined #openstack-nova | 14:49 | |
| *** Spazmotic has joined #openstack-nova | 14:49 | |
| odyssey4me | so this conditional set is only important for the newton->ocata upgrade? | 14:49 |
| mriedem | 1. nova-manage cell_v2 map_cell0 can do a db query for cell0 rather than rely on a DBDuplicateEntry | 14:50 |
| mriedem | 2. you can run nova-manage api_db sync unconditionally | 14:50 |
| odyssey4me | we're very happy to shorten and simplify the steps taken, so option 2 sounds good to me | 14:50 |
| odyssey4me | I'll push a patch up shortly. I'll ping you to verify the review if that's ok? | 14:51 |
| mriedem | fwiw, grenade also runs map_cell0 on upgrade and cell0 is already mapped http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_34_277 | 14:51 |
| mriedem | sure. i'm not ansible guy, but i can see if the non-voting upgrade job is green or not :) | 14:51 |
| odyssey4me | :) hopefully we can make it as readable as a shell task procedure | 14:52 |
| mriedem | oh looky here http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_25_031 | 14:52 |
| mriedem | grenade hits the same thing but doesn't fail for some reason | 14:52 |
| *** eharney has joined #openstack-nova | 14:53 | |
| *** sar has joined #openstack-nova | 14:53 | |
| *** salv-orlando has joined #openstack-nova | 14:55 | |
| *** sree has joined #openstack-nova | 14:55 | |
| *** felipemonteiro__ has quit IRC | 14:55 | |
| tssurya | sorry was afk, that doesn't look goo | 14:56 |
| tssurya | good* | 14:56 |
| *** felipemonteiro__ has joined #openstack-nova | 14:56 | |
| odyssey4me | mriedem maybe someone couldn't figure it out and added || true ;) | 14:57 |
| *** salv-orl_ has quit IRC | 14:58 | |
| odyssey4me | jokes aside, that's quite serious... but I'm glad we're not alone, and that we could find a red flag :) | 14:58 |
| mriedem | it's hitting htis https://github.com/openstack-dev/grenade/blob/master/projects/60_nova/upgrade.sh#L74 | 14:58 |
| mriedem | oh i know why it's not failing | 14:59 |
| mriedem | the nova-manage command doesn't return a non-0 | 14:59 |
| mriedem | that's why it dumps out the breadcrumb | 14:59 |
| mriedem | "Has "nova-manage api_db sync" been run?" | 14:59 |
| openstackgerrit | Merged openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | 15:00 |
| tssurya | mriedem: is this because api DB was not synced before running db sync ? | 15:00 |
| *** suresh12 has quit IRC | 15:00 | |
| *** sree has quit IRC | 15:00 | |
| mriedem | yes | 15:00 |
| tssurya | ah ok | 15:00 |
| *** tbachman has quit IRC | 15:02 | |
| *** sree has joined #openstack-nova | 15:02 | |
| mriedem | the order in grenade for the db sync matches what we have in our upgrade docs too https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process | 15:04 |
| mriedem | tssurya: odyssey4me: superdan: jgwentworth: https://bugs.launchpad.net/grenade/+bug/1761775 | 15:07 |
| openstack | Launchpad bug 1761775 in grenade "InternalError during nova-manage db sync when upgrading to Rocky" [Undecided,Confirmed] | 15:07 |
| mriedem | i listed 2 options in there for fixing grenade | 15:07 |
| *** suresh12 has joined #openstack-nova | 15:08 | |
| mriedem | trying to channel what sdague would say... | 15:08 |
| fried_rice | mriedem: Care to +W https://review.openstack.org/#/c/555961/ (lower-constraints.txt) quick before we break it again? | 15:08 |
| tssurya | hmm question.. why wasn't this an issue before me adding a new column to cell_mappings ? | 15:08 |
| tssurya | I mean no other column change in api_db before has needed a api sync before db sync situation ? | 15:09 |
| odyssey4me | mriedem yeah, that does seem a bit chicken-and-egg | 15:09 |
| cdent | finucannot: +many on the -W stuff. pain now for less pain later | 15:09 |
| mriedem | tssurya: we just haven't modified the cell_mappings table in forever | 15:10 |
| *** felipemonteiro_ has joined #openstack-nova | 15:10 | |
| tssurya | mriedem: cell_mappings yes, what about other tables ? | 15:10 |
| mriedem | db sync doesn't hit other tables | 15:10 |
| *** rmart04 has quit IRC | 15:10 | |
| mriedem | in the api db | 15:11 |
| *** armaan has quit IRC | 15:11 | |
| jgwentworth | mriedem: interesting, I always thought the order was api_db first in the upgrade docs. I would have thought that one should be done first and I can't immediately think of why it's not | 15:11 |
| superdan | mriedem: oh is it trying to read the cell mappings to do the cell databases before it has sync'd the api db? | 15:11 |
| superdan | yeah I'm not sure why api isn't first | 15:11 |
| finucannot | cdent: Yup, only way to do it, unfortunately | 15:11 |
| mriedem | superdan: nova-manage db sync by default tries to also sync cell0- | 15:11 |
| tssurya | mriedem: ah heh yes, anyways I thought api_sync was run before db sync until now | 15:11 |
| *** armaan has joined #openstack-nova | 15:11 | |
| superdan | mriedem: and it looks up the mapping for it before it has sync'd the api db? | 15:11 |
| *** artom_ has quit IRC | 15:12 | |
| tssurya | superdan: yes | 15:12 |
| mriedem | yeah | 15:12 |
| *** suresh12 has quit IRC | 15:12 | |
| superdan | er, I guess t's just that we call db sync before api_db sync in grenade you mean | 15:12 |
| mriedem | right | 15:13 |
| mriedem | that's one possible fix | 15:13 |
| superdan | mriedem: tbh, I imagine that it's second because the main db sync was there first, and we were adding stuff, so appending it to the end | 15:13 |
| superdan | I don't think that was an intentional decision | 15:13 |
| mriedem | superdan: maybe https://review.openstack.org/#/c/190399/ | 15:14 |
| mriedem | wow mitaka | 15:14 |
| superdan | I'm kinda surprised actually since you would have to sync the api db at day zero to get cell0 to work at all, unless it's just using config for that one | 15:14 |
| *** felipemonteiro__ has quit IRC | 15:14 | |
| mriedem | ocata->pike would use devstack to setup ocata first though wouldn't it? | 15:14 |
| mriedem | and devstack does the base install with the api db sync first | 15:14 |
| *** sree has quit IRC | 15:14 | |
| superdan | ah, right, right | 15:15 |
| mriedem | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L724 | 15:15 |
| mriedem | then later: https://github.com/openstack-dev/devstack/blob/master/lib/nova#L741 | 15:15 |
| superdan | yeah I was thinking about my superconductor changes and was like "I was sure api goes first" | 15:15 |
| *** fullmetaljackiet has joined #openstack-nova | 15:15 | |
| superdan | but this is grenade, which matters less | 15:15 |
| superdan | because devstack has already run | 15:15 |
| superdan | so that makes sense | 15:15 |
| mriedem | fried_rice: someone is goign to have to tl;dr that for me - will that lower-constraints.txt file get synced automagically by the reqs bot? | 15:16 |
| superdan | someone else should write the grenade patch so I can +2 it | 15:16 |
| fried_rice | mriedem: No, it's manual. | 15:16 |
| fried_rice | mriedem: There was a ML thread by dhellmann | 15:17 |
| mriedem | yeah i saw the novel | 15:17 |
| tssurya | superdan: I can.. | 15:17 |
| mriedem | again, tl;dr | 15:17 |
| mriedem | fried_rice: also, is that job supposed to actually run tests? http://logs.openstack.org/61/555961/3/check/openstack-tox-lower-constraints/91b6d17/job-output.txt.gz#_2018-04-06_13_19_18_865706 | 15:17 |
| mriedem | b/c it doesn't | 15:17 |
| superdan | and we should change those docs, I imagine they're in that order for the same reason | 15:17 |
| mriedem | fried_rice: so -1 on that | 15:18 |
| *** armaan has quit IRC | 15:18 | |
| superdan | tssurya: okay I'll tweak the doc | 15:19 |
| fried_rice | mriedem: Okay. The requirements-check job sure works (cause that's what held up PS1 and PS2). But yeah, I don't know anything beyond that. | 15:19 |
| tssurya | although mriedem says he prefers a "Add a from-queens script" to swapping | 15:19 |
| mriedem | no i don't prefer that | 15:19 |
| mriedem | i said, "Option #1 seems a bit shady, but option #2 seems excessive, especially if we're going to be changing things in the API DB in future releases." | 15:20 |
| mriedem | in other words, i'm hedging | 15:20 |
| tssurya | okay sorry so its the shady one then | 15:20 |
| superdan | I don't think it's shady, I think it's been wrong for a long time, hidden by the fact that we run devstack (in the proper order) first before grenade | 15:21 |
| mriedem | if we added a from-queens script, per grenade guidelines, we'd also need a nova release note in rocky saying you need to run nova-manage api_db sync before nova-manage db sync | 15:21 |
| jgwentworth | heh, I think it was shady assuming there was a good reason for the current ordering, which there apparently isn't | 15:21 |
| superdan | yeah | 15:21 |
| odyssey4me | mriedem alright, I've pushed up an OSA patch to test out the shady option: https://review.openstack.org/#/c/559354/1/tasks/nova_db_setup.yml | 15:21 |
| *** sree has joined #openstack-nova | 15:21 | |
| tssurya | superdan : should I do the docs in the same one then ? | 15:21 |
| odyssey4me | that'll make api_db sync before before db | 15:22 |
| superdan | tssurya: I'm doing the doc patch, if you can do grenade | 15:22 |
| tssurya | superdan: okay :) | 15:22 |
| odyssey4me | go.... TEAM! ;) | 15:22 |
| mriedem | odyssey4me: that will affect both base install and upgrade? | 15:22 |
| odyssey4me | mriedem yep | 15:22 |
| jgwentworth | \o/ | 15:23 |
| *** jmlowe_ has joined #openstack-nova | 15:23 | |
| *** jmlowe has quit IRC | 15:23 | |
| fried_rice | mriedem: Looking over dhellmann's email again, it's possible that the no-op-ness of that job is intentional; to be switched on at some point in the future once the dust has settled. This is just setting up the framework. | 15:24 |
| mriedem | fried_rice: well the commit message is clearly misleading then | 15:24 |
| fried_rice | mriedem: But... I guess I'll let dhellmann respond to your -1. I was just trying to take over since he's got a few hundred of these up in the air. | 15:25 |
| jgwentworth | so we set our constraints manually and the job makes sure all of the components work together with the versions they've required? | 15:25 |
| odyssey4me | mriedem if install must do a different order to upgrade, then that's not awesome - but we can adjust... but as it stands now it looks like our api_db sync happened first on upgrade only, but the db sync didn't happen before the cell mapping... so I can adjust the sequence to just move when the db sync happens if that's more sensible | 15:25 |
| *** suresh12 has joined #openstack-nova | 15:25 | |
| mriedem | odyssey4me: this might be fine, we'll see; i was wondering why there were redundant api db syncs in there before | 15:25 |
| mriedem | i guess the first one was trying to be sly | 15:26 |
| mriedem | and a one time only thing on install | 15:26 |
| *** sree has quit IRC | 15:26 | |
| odyssey4me | yeah, I believe there was a sequence difference between new environment vs upgrade | 15:27 |
| mriedem | odyssey4me: left a comment, i think this might fail | 15:28 |
| mriedem | we'll see | 15:28 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Clarify/correct the ordering of API and Cell database schema updates https://review.openstack.org/559357 | 15:28 |
| superdan | ooh, look how many words! | 15:28 |
| jgwentworth | you dun good | 15:29 |
| *** belmoreira has quit IRC | 15:30 | |
| odyssey4me | mriedem question to clarify before I revise | 15:30 |
| odyssey4me | (in review) | 15:30 |
| *** artom has joined #openstack-nova | 15:33 | |
| *** armaan has joined #openstack-nova | 15:33 | |
| *** openstackgerrit has quit IRC | 15:33 | |
| tssurya | https://review.openstack.org/#/c/559358/ | 15:33 |
| *** tbachman has joined #openstack-nova | 15:34 | |
| superdan | tssurya: comments on the commit message | 15:35 |
| tssurya | looking | 15:35 |
| superdan | also probably need jgwentworth's sign-off on the final version before we put it in | 15:36 |
| superdan | we usually try to require that for things like this in grenade | 15:36 |
| tssurya | sure :) | 15:36 |
| jgwentworth | I shall ack it | 15:37 |
| *** ShilpaSD has quit IRC | 15:37 | |
| superdan | tssurya: if my friday-speak wasn't clear enough on that review let me know and I'll try to plain it up a little | 15:38 |
| *** tssurya is now known as sususuryashines | 15:39 | |
| sususuryashines | superdan: yes please | 15:39 |
| jgwentworth | hah, good friday nick | 15:40 |
| sususuryashines | for starters did you only want a change in explanation or the title also seems odd to you ? | 15:40 |
| superdan | no, the title seems okay | 15:40 |
| sususuryashines | jgwentworth: heh thanks ;) | 15:40 |
| superdan | let me just write what I think it should be | 15:40 |
| mriedem | sususuryashines: you could just lift from this https://review.openstack.org/#/c/559357/ | 15:41 |
| *** lajoskatona has joined #openstack-nova | 15:42 | |
| sususuryashines | mriedem: yes I thought so at first, but that commit message is a little different from superdan's recommendations on my commit message. | 15:42 |
| mriedem | it's different in that it doesn't have all the meh and stuff :) | 15:43 |
| superdan | sususuryashines: left a suggested block of text | 15:44 |
| *** vladikr has quit IRC | 15:44 | |
| sususuryashines | superdan: historical :) | 15:44 |
| sususuryashines | ack, will read and change | 15:45 |
| sususuryashines | thanks | 15:45 |
| superdan | sususuryashines: it's a lot like my doc patch, but a little more grenade-y.. feel free to re-write it in your words, I just want it to sound more like "oops, this was wrong all along" and not "we decided to change this when something broke" | 15:45 |
| sususuryashines | superdan: ack | 15:45 |
| *** lajoskatona has quit IRC | 15:46 | |
| *** pcaruana has quit IRC | 15:49 | |
| *** chyka has joined #openstack-nova | 15:50 | |
| *** armaan has quit IRC | 15:53 | |
| *** danpawlik has quit IRC | 15:59 | |
| *** artom has quit IRC | 16:01 | |
| *** r-daneel has quit IRC | 16:03 | |
| *** suresh12 has quit IRC | 16:04 | |
| *** chyka has quit IRC | 16:04 | |
| *** artom has joined #openstack-nova | 16:04 | |
| *** chyka has joined #openstack-nova | 16:05 | |
| *** sree has joined #openstack-nova | 16:08 | |
| *** germs has joined #openstack-nova | 16:11 | |
| *** germs has quit IRC | 16:11 | |
| *** germs has joined #openstack-nova | 16:11 | |
| *** AlexeyAbashkin has quit IRC | 16:12 | |
| mriedem | fried_rice: thanks for going over https://review.openstack.org/#/c/558001/ - i left some replies, will wait on you before making any changes there | 16:12 |
| fried_rice | mriedem: ack | 16:13 |
| mriedem | i think at this point it's likely easiest to say, "if you use OVS everywhere, just set this to True" | 16:13 |
| *** sree has quit IRC | 16:13 | |
| mriedem | if you're mnaser and enjoy pain, then leave it False | 16:14 |
| *** dtantsur is now known as dtantsur|pto | 16:14 | |
| mnaser | hi | 16:14 |
| mriedem | just making a little ODL joke | 16:14 |
| mriedem | at your expense | 16:14 |
| mnaser | oh | 16:14 |
| mnaser | mriedem: if you're really into pain then you're run nova with multiple network backends | 16:15 |
| *** mdrabe has joined #openstack-nova | 16:15 | |
| mnaser | s/run/running/ | 16:15 |
| mnaser | s/running/trying to run/ | 16:15 |
| mriedem | yeah, that's what the new config option in that patch is for | 16:15 |
| *** patriciadomin has joined #openstack-nova | 16:15 | |
| *** ekhugen has joined #openstack-nova | 16:15 | |
| mnaser | thanks for the ping about it though | 16:15 |
| *** egarbade has joined #openstack-nova | 16:16 | |
| *** germs has quit IRC | 16:16 | |
| mriedem | kashyap: ever seen this live migration failure? http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-multinode-live-migration/8de6e74/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_05_21_48_43_258043 i've seen this in a few different live migration runs this week | 16:19 |
| mriedem | VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8 | 16:19 |
| mriedem | error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk' | 16:19 |
| *** derekh has quit IRC | 16:20 | |
| *** Spazmotic has quit IRC | 16:22 | |
| *** Spaz-Work has joined #openstack-nova | 16:22 | |
| mriedem | http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-multinode-live-migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz | 16:22 |
| mriedem | found https://bugzilla.redhat.com/show_bug.cgi?id=1450524 | 16:24 |
| openstack | bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang | 16:24 |
| arvindn05 | mriedem: i added comments to https://review.openstack.org/#/c/541507/9 | 16:24 |
| fried_rice | mriedem: Responded | 16:24 |
| arvindn05 | also had a question on https://review.openstack.org/#/c/546357/ - Skip placement on rebuild in same host | 16:25 |
| arvindn05 | why are we allowing the skipping placement given we know the image can now contain qualitative aspects? | 16:26 |
| arvindn05 | also what if the qualitative aspects in the flavor changes? would we want to run placement in this case as well?? | 16:26 |
| mriedem | arvindn05: for one, we aren't yet allowing images with required traits | 16:28 |
| mriedem | that patch fixes a bug which has to go back to pike | 16:28 |
| mriedem | two, rebuild doesn't take a new flavor | 16:28 |
| arvindn05 | ahh...missed that it was pike... | 16:29 |
| mriedem | rebuild does take a new image, and the image meta could change, so we have to re-run some scheduler filters | 16:29 |
| mriedem | this patch is master, but fixes a bug that was introduced in pike | 16:29 |
| arvindn05 | for #2, it doesnt need to take a new flavor right? if the flavor was updated for the instance, then we might want to run placement | 16:30 |
| mriedem | arvindn05: my point was, once we do start accepting images with required qualitative traits, the logic in this bug fix will likely need to change, as part of your blueprint | 16:30 |
| mriedem | arvindn05: the flavor doesn't get updated for the instance unless you resize the instance | 16:30 |
| *** germs has joined #openstack-nova | 16:30 | |
| *** germs has quit IRC | 16:30 | |
| *** germs has joined #openstack-nova | 16:30 | |
| mriedem | in which case you're not doing a rebuild, you're doing a cold migration with a new flavor | 16:30 |
| mriedem | to a new host | 16:30 |
| *** germs has quit IRC | 16:30 | |
| kashyap | mriedem: Was AFK; looking | 16:31 |
| fried_rice | jroll: Was this just a suggestion, or is this the way we're supposed to be setting these up? https://review.openstack.org/#/c/555961/3/tox.ini@243 | 16:31 |
| *** germs has joined #openstack-nova | 16:31 | |
| *** germs has quit IRC | 16:31 | |
| *** germs has joined #openstack-nova | 16:31 | |
| arvindn05 | got it...will keep in mind to change the patch | 16:32 |
| fried_rice | jroll: I don't know if you're involved in the whole requirements shift thingy, or if you're just able to look at it and know that that will dtrt | 16:32 |
| jroll | fried_rice: that's how the other tox environments wind up running tests, look at the testenv:py35 section of that file | 16:32 |
| kashyap | mriedem: That "Guest index YYYYY inconsistent with Host index" error rings a bell. And there _was_ a QEMU fix, let me look | 16:32 |
| jroll | fried_rice: this is assuming the goal is to run unit tests there | 16:32 |
| jroll | fried_rice: I'm not involved, just saw you mention here that you don't know what you need there, and thought I'd help :) | 16:33 |
| fried_rice | jroll: Right, I get that it will do a thing, and that that thing is probably useful; but I didn't know if ^ assumption is actually true yet. | 16:33 |
| mriedem | kashyap: yeah you're on https://bugzilla.redhat.com/show_bug.cgi?id=1450524 | 16:33 |
| openstack | bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang | 16:33 |
| kashyap | Am I? | 16:33 |
| jroll | fried_rice: ah, gotcha. question for someone else then :) | 16:33 |
| * kashyap clicks | 16:33 | |
| fried_rice | jroll: Okay, thanks. I'll wait for dhellmann to respond (either saying, "verily, do as jroll says," or "yeah, we don't want to do that yet - amend the commit message"). | 16:33 |
| jroll | fried_rice: though based on the commit message, I'd guess unit tests were the goal: "Create a tox environment for running the unit tests against the lower bounds of the dependencies." | 16:34 |
| fried_rice | jroll: yuh; but the email chain hints that maybe we're just setting up the framework right now. I dunno. Was just trying to ease Doug's burden a little. | 16:34 |
| jroll | ยฏ\_(ใ)_/ยฏ | 16:35 |
| arvindn05 | mriedem: i might be missing something....lets say you have instance X with flavor F, while X is running, you go in and change the flavor F to F'(only adding 1 trait, no resource changes), so when its rebuild are we still using F or F'? | 16:35 |
| mriedem | arvindn05: F | 16:35 |
| mriedem | the flavor used to originally create the instance is stored in the db with the instance, | 16:35 |
| *** openstackgerrit has joined #openstack-nova | 16:36 | |
| openstackgerrit | Merged openstack/nova master: Use a pythonic delete, with a retry. https://review.openstack.org/554793 | 16:36 |
| mriedem | so if the admin deletes or changes extra specs on the original flavor, the instance is still using what it was created with | 16:36 |
| openstackgerrit | Merged openstack/nova master: Clarify/correct the ordering of API and Cell database schema updates https://review.openstack.org/559357 | 16:36 |
| mriedem | i believe it's also persisted with the request spec | 16:36 |
| arvindn05 | ahh...didnt know that. so basically we have a snapshot of the flavor with the instance....makes more sense now :) | 16:36 |
| mriedem | yes | 16:36 |
| *** ralonsoh has quit IRC | 16:36 | |
| kashyap | mriedem: So ... this seems to be the instance that crashed: | 16:37 |
| kashyap | http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-multinode-live-migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz | 16:37 |
| mriedem | kashyap: tracking in nova with https://bugs.launchpad.net/nova/+bug/1761798 | 16:38 |
| openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | 16:38 |
| mriedem | kashyap: yup | 16:38 |
| kashyap | mriedem: So, that RHBZ bug was closed with this (unsatisfying) conclusion: | 16:39 |
| kashyap | [quote] | 16:39 |
| kashyap | "Well, we know half of their problem was the old virtio-balloon bug - that's already fixed. | 16:39 |
| kashyap | There is an unexplained 'virtio-net' problem they were seeing - but we've never managed to recreate it, so I don't know if there's anything we can do. | 16:39 |
| kashyap | [/quote] | 16:39 |
| kashyap | "their | 16:39 |
| kashyap | (Err, "their" == the reporter) | 16:39 |
| *** r-daneel has joined #openstack-nova | 16:41 | |
| mriedem | yeah, | 16:41 |
| mriedem | hoping that maybe this goes away when we start using the queens UCA packages | 16:41 |
| mriedem | so far it's only 4 hits in 7 days | 16:41 |
| kashyap | Nod; I'm trying to see which version is that fix in | 16:41 |
| kashyap | ("that fix" alluded to in the above quote) | 16:41 |
| *** vladikr has joined #openstack-nova | 16:41 | |
| kashyap | mriedem: Look at that comment: https://bugzilla.redhat.com/show_bug.cgi?id=1450524#c21 | 16:42 |
| openstack | bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang | 16:42 |
| kashyap | "I'm not having any luck replicating this here; it's ran ~230 iterations of a migration [...]" | 16:42 |
| mriedem | kashyap: yeah, don't really care about how often someone ran something in parallel in isolation | 16:43 |
| kashyap | (I recall trying to get this reproducing; but was nigh impossible .) | 16:43 |
| kashyap | Yeah; OpenStack Gate is singularly unique in identifying weird bugs | 16:43 |
| mriedem | with openstack ci the tests are run concurrently on the same 2 hosts, and get run thousands of times per day | 16:43 |
| kashyap | Yep, /me remembers the horrors of the "live snapshot bug" that was only reproducible on the Gate | 16:44 |
| mriedem | anyway, not high priority, i wasn't sure if there was any link from the rhev package versions to versions of qemu we use | 16:44 |
| *** lucasagomes is now known as lucas-afk | 16:44 | |
| mriedem | i don't know how the rhev package versions translate | 16:44 |
| *** armaan has joined #openstack-nova | 16:44 | |
| *** danpawlik has joined #openstack-nova | 16:44 | |
| kashyap | ('qemu-kvm-rhev' is what CentOS uses) | 16:44 |
| mriedem | otherwise the versions are the same? | 16:45 |
| kashyap | It is the "enterprise" version that has all the 'advanced' features. (And 'qemu-kvm' is the plain binary.) | 16:45 |
| kashyap | mriedem: Not exactly; I'll figure out the version translations and how they map to Ubuntu | 16:45 |
| kashyap | (And write a comment) | 16:45 |
| mriedem | thanks. we're hitting this with qemu 2.10 | 16:46 |
| kashyap | The eagle-eyed would notice the "NB2" noted by DanPB here: https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix | 16:46 |
| kashyap | At the end of the page | 16:46 |
| mriedem | i haven't had to care about rhev since i was at ibm and even then, about 8 years ago | 16:47 |
| kashyap | mriedem: Me neither; the "-rhev" is just the QEMU binary split | 16:47 |
| mriedem | superdan: gimme some MCP builds! | 16:47 |
| * kashyap has never even tried "RHEV" or oVirt in his life | 16:47 | |
| kashyap | Real People use raw QEMU directly | 16:47 |
| superdan | mriedem: no. | 16:47 |
| * kashyap runs | 16:47 | |
| superdan | mtreinish: can you hook us up? https://review.openstack.org/#/c/559358/3 | 16:48 |
| *** sambetts is now known as sambetts|afk | 16:49 | |
| *** suresh12 has joined #openstack-nova | 16:49 | |
| *** danpawlik has quit IRC | 16:50 | |
| *** salv-orlando has quit IRC | 16:51 | |
| arvindn05 | mriedem: you got me thinking on instance migration as welll | 16:51 |
| *** salv-orlando has joined #openstack-nova | 16:51 | |
| mriedem | arvindn05: images don't change on instance migration | 16:51 |
| arvindn05 | mriedem: is the placement api called anytime during the migrations? | 16:52 |
| mriedem | but, when migrating to a new host, the scheduler will validate the image used to create the instance as well | 16:52 |
| mriedem | yes | 16:52 |
| mriedem | as part of scheduling to find a new dest host | 16:52 |
| mriedem | rebuild is weird in that the instance doesn't move | 16:52 |
| mriedem | but the image can change so we have to re-validate that the new image can work on the same host | 16:52 |
| mriedem | the original image used to create the instance is stored in the request_spec for that instance | 16:53 |
| mriedem | so that's what the scheduler uses during a move operation | 16:53 |
| *** fried_rice is now known as fried_rolls | 16:53 | |
| arvindn05 | so when you say the scheduler will validate the image, the placement API will be requested for a matching compute host? | 16:54 |
| *** suresh12 has quit IRC | 16:54 | |
| *** jchhatbar has quit IRC | 16:54 | |
| arvindn05 | i see there is a "Manual selection of the destination host" per https://docs.openstack.org/nova/pike/admin/live-migration-usage.html | 16:54 |
| mriedem | yes. scheduling asks placement for compute node resource providers that can fit the flavor constraints, and optionally qualitative traits defined in the flavor extra specs | 16:55 |
| mriedem | arvindn05: yes, but that still runs through the scheduler but with the requested host | 16:55 |
| mriedem | if you specify the force parameter to live migration with a host, then we bypass the scheduler and, well, you get what you ask for | 16:55 |
| *** mgoddard has quit IRC | 16:56 | |
| arvindn05 | ok...and we would get no valid host if when run through the scheduler, the host does not match | 16:56 |
| mriedem | correct | 16:56 |
| *** salv-orlando has quit IRC | 16:56 | |
| *** fragatina has quit IRC | 16:56 | |
| arvindn05 | can you point me to the code where the live migration calls the scheduler?(assuming it calls select destination method) | 16:57 |
| *** links has joined #openstack-nova | 16:57 | |
| mriedem | arvindn05: https://github.com/openstack/nova/blob/0de6ebd086656066c4dcc5c093b5cc7e92e035ac/nova/conductor/tasks/live_migrate.py#L319 | 16:58 |
| *** gjayavelu has joined #openstack-nova | 16:59 | |
| arvindn05 | mriedem: thank you very much :) | 16:59 |
| *** mdbooth has quit IRC | 16:59 | |
| *** jpena is now known as jpena|off | 17:01 | |
| arvindn05 | mriedem: btw https://review.openstack.org/#/c/557795/ -Update ImageMetaProp object to expose traits is ready for review as well | 17:01 |
| *** sdague has quit IRC | 17:04 | |
| *** links has quit IRC | 17:06 | |
| mriedem | arvindn05: yeah i wanted to post this https://review.openstack.org/#/c/557795/14/nova/objects/image_meta.py@560 and get the ack from superdan and leakypipes first so we're on the same page | 17:06 |
| *** sususuryashines has quit IRC | 17:07 | |
| *** links has joined #openstack-nova | 17:08 | |
| *** david-lyle has quit IRC | 17:09 | |
| mtreinish | superdan: is that ordering documented anywhere? | 17:13 |
| superdan | mtreinish: it's documented in the right order | 17:14 |
| superdan | mtreinish: this matches the docs | 17:14 |
| mriedem | mtreinish: it is now https://review.openstack.org/#/c/559357/ | 17:14 |
| superdan | mriedem: dude, shsssshh | 17:14 |
| mriedem | ha | 17:14 |
| mtreinish | heh, ok well as long as it's written down now | 17:14 |
| mriedem | mtreinish is like the substitute teacher on a friday that lets things slide | 17:15 |
| mtreinish | +A | 17:15 |
| mriedem | see!? | 17:15 |
| superdan | hah | 17:15 |
| *** gouthamr has quit IRC | 17:17 | |
| mtreinish | lol | 17:17 |
| sean-k-mooney | by the way when ye refer to cells in that doc does the ordering of api first apply equally to v1 and v2 | 17:17 |
| *** sdague has joined #openstack-nova | 17:19 | |
| sean-k-mooney | i generally try to not know how cells v1 works in case anyone will ask me to help fix somthing with cellsv1 in the furutre but just taught i would ask. | 17:20 |
| *** armaan has quit IRC | 17:20 | |
| mriedem | don't ask | 17:21 |
| mriedem | the rule with "helping people with cells v1" is, don't help them | 17:21 |
| sean-k-mooney | :) | 17:22 |
| mriedem | failing that, goto superdan or jgwentworth | 17:22 |
| *** suresh12 has joined #openstack-nova | 17:22 | |
| sean-k-mooney | i approch cellsv1 the opisitway i approch most of the rest of the codebase. i want to know how most of the codebase works so i can help others but with cellsv1 i tend to follow the above advice | 17:23 |
| mriedem | finucannot: wtf does this mean? http://logs.openstack.org/70/559370/1/check/build-sphinx-docs/bda526f/job-output.txt.gz#_2018-04-06_16_47_02_376553 | 17:25 |
| mriedem | and how do i fix it? | 17:25 |
| *** suresh12 has quit IRC | 17:26 | |
| ildikov | mriedem: it looks a little like when the 'todo' extension was missing from the config | 17:27 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanup _get_request_spec_for_select_destinations for live migrate https://review.openstack.org/559381 | 17:27 |
| *** yassine_ has quit IRC | 17:27 | |
| *** Swami has joined #openstack-nova | 17:27 | |
| mriedem | ildikov: it's something with this new PTI docs change i think, but i don't have the heart to read that thread right now | 17:28 |
| *** yassine_ has joined #openstack-nova | 17:28 | |
| *** mgoddard has joined #openstack-nova | 17:28 | |
| *** vladikr has quit IRC | 17:29 | |
| *** fullmetaljackiet has quit IRC | 17:33 | |
| *** tesseract-RH has quit IRC | 17:35 | |
| sean-k-mooney | mriedem: do you need the "-" in query: >- also gr say sphinx>=1.6.2,!=1.6.6,!=1.6.7 # BSD but the test requirements in elastic-rechec is sphinx>=1.1.2,<1.2 | 17:36 |
| sean-k-mooney | mriedem: so maybe just use a non achient version of sphinx? | 17:37 |
| sean-k-mooney | mriedem: it does not look like the openstack requirement proposal bot keeps the repo in sync so when they bumped to new pbr on march 1st perhaps that has issues since pbr is trying to move to sphinx >1.6 | 17:38 |
| *** fragatina has joined #openstack-nova | 17:40 | |
| *** fragatina has quit IRC | 17:41 | |
| sean-k-mooney | lower constratings also reqiures 1.6.5 https://github.com/openstack/requirements/blob/master/lower-constraints.txt#L359 so the 1.2 max in openstack-infra/elastic-recheck is proable the issue | 17:41 |
| *** fragatina has joined #openstack-nova | 17:41 | |
| *** fragatina has quit IRC | 17:41 | |
| *** fragatina has joined #openstack-nova | 17:42 | |
| *** fragatina has quit IRC | 17:43 | |
| *** fragatina has joined #openstack-nova | 17:43 | |
| arvindn05 | mriedem: thanks. jaypipes said he would approve as well...will work on my next patch for the BP and also start thinking about the resize issue for rocky | 17:43 |
| *** fragatina has quit IRC | 17:43 | |
| *** suresh12 has joined #openstack-nova | 17:44 | |
| *** fragatina has joined #openstack-nova | 17:44 | |
| *** voelzmo has joined #openstack-nova | 17:45 | |
| *** AlexeyAbashkin has joined #openstack-nova | 17:46 | |
| *** mdrabe has quit IRC | 17:48 | |
| *** AlexeyAbashkin has quit IRC | 17:50 | |
| *** mdrabe has joined #openstack-nova | 17:52 | |
| sean-k-mooney | mriedem: https://review.openstack.org/559390 fixes your issue. do you want me to rebase your patch on top. | 17:55 |
| *** voelzmo has quit IRC | 17:59 | |
| mriedem | sean-k-mooney: i can, was looking at the pti email and doing some stuff, but if this works, sure | 18:01 |
| sean-k-mooney | mriedem: locally at least this fixes it for me. i cherry picked you patch locallly and that passed the docs job too after i bumped the version | 18:02 |
| *** armaan has joined #openstack-nova | 18:02 | |
| cdent | leakypipes, fried_rolls, figleaf : this might be of interest, a sort of placement test env: https://github.com/cdent/placecat (not fully fleshed out yet, just playing so far) | 18:03 |
| *** brad[] has quit IRC | 18:03 | |
| sean-k-mooney | mriedem: by the way pti do you mean the "[openstack-dev] Following the new PTI for document build, broken local builds" thread from finucannot | 18:04 |
| leakypipes | cdent: nice :) | 18:04 |
| *** brad[] has joined #openstack-nova | 18:04 | |
| sean-k-mooney | mriedem: hum the chages for pti seam simple ill try making those instead | 18:06 |
| sean-k-mooney | mriedem: then i need to go home because its 7 on a friday | 18:06 |
| *** tssurya has joined #openstack-nova | 18:07 | |
| *** tssurya is now known as sususuryashines | 18:08 | |
| *** voelzmo has joined #openstack-nova | 18:09 | |
| *** baoli_ has joined #openstack-nova | 18:11 | |
| *** armaan has quit IRC | 18:11 | |
| *** adriano__ has quit IRC | 18:12 | |
| *** gouthamr has joined #openstack-nova | 18:13 | |
| *** gouthamr has quit IRC | 18:13 | |
| *** baoli has quit IRC | 18:14 | |
| *** voelzmo has quit IRC | 18:14 | |
| *** gouthamr has joined #openstack-nova | 18:14 | |
| *** sree has joined #openstack-nova | 18:16 | |
| *** danpawlik has joined #openstack-nova | 18:18 | |
| *** sree has quit IRC | 18:21 | |
| *** r-daneel has quit IRC | 18:23 | |
| *** r-daneel has joined #openstack-nova | 18:25 | |
| *** danpawlik has quit IRC | 18:28 | |
| *** baoli_ has quit IRC | 18:29 | |
| *** felipemonteiro_ has quit IRC | 18:31 | |
| *** felipemonteiro_ has joined #openstack-nova | 18:32 | |
| *** bauzas has quit IRC | 18:35 | |
| figleaf | cdent: sweet. Wish I had time to play with it. :) | 18:35 |
| *** vladikr has joined #openstack-nova | 18:36 | |
| *** ssurana has joined #openstack-nova | 18:38 | |
| *** danpawlik has joined #openstack-nova | 18:39 | |
| *** david-lyle has joined #openstack-nova | 18:40 | |
| *** sree has joined #openstack-nova | 18:42 | |
| openstackgerrit | Merged openstack/python-novaclient master: Fix validation for command arguments https://review.openstack.org/558324 | 18:43 |
| *** danpawlik has quit IRC | 18:44 | |
| *** avolkov has quit IRC | 18:47 | |
| *** sree has quit IRC | 18:47 | |
| *** danpawlik has joined #openstack-nova | 18:49 | |
| openstackgerrit | Merged openstack/python-novaclient master: Fix AttributeError in getting a resource ID https://review.openstack.org/558328 | 18:51 |
| openstackgerrit | Merged openstack/python-novaclient master: Replace GB with GiB https://review.openstack.org/558364 | 18:51 |
| *** damien_r has quit IRC | 18:51 | |
| *** amoralej is now known as amoralej|off | 18:51 | |
| *** mgoddard has quit IRC | 18:53 | |
| openstackgerrit | sean mooney proposed openstack/os-vif master: convert os-vif docs to follow PTI https://review.openstack.org/559398 | 18:54 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: add a link in the install guides about configuring neutron https://review.openstack.org/559115 | 18:54 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Log a more useful error when cinder auth isn't configured https://review.openstack.org/559399 | 18:54 |
| *** danpawlik has quit IRC | 18:54 | |
| *** bauzas has joined #openstack-nova | 18:55 | |
| *** logan- has quit IRC | 18:56 | |
| *** logan- has joined #openstack-nova | 18:56 | |
| *** baoli has joined #openstack-nova | 18:57 | |
| *** fried_rolls is now known as fried_rice | 18:59 | |
| *** bauzas has quit IRC | 19:00 | |
| *** felipemonteiro__ has joined #openstack-nova | 19:00 | |
| fried_rice | cdent: Neat! | 19:00 |
| fried_rice | cdent: It's basically a fully functional but not-for-production placement service that you can set up to have something to test/play against? | 19:01 |
| cdent | fried_rice: since you're not one with the twitters, you may not be aware of the placedock (or the blog posts associated with it) stuff, if you go to the placedock repo there's loads of links to context | 19:01 |
| cdent | fried_rice: pretty much | 19:01 |
| cdent | depending on the variables you pass into the container | 19:02 |
| cdent | (and I recently found a bug with that) | 19:02 |
| cdent | and in terms of production, you could, except that the container builds from non master code | 19:03 |
| *** felipemonteiro_ has quit IRC | 19:03 | |
| *** cz2 has quit IRC | 19:03 | |
| openstackgerrit | sean mooney proposed openstack/os-vif master: make legacy-tempest-dsvm-nova-os-vif voteing https://review.openstack.org/559401 | 19:07 |
| *** claudiub|2 has quit IRC | 19:21 | |
| *** cz2 has joined #openstack-nova | 19:31 | |
| *** mgoddard has joined #openstack-nova | 19:31 | |
| sean-k-mooney | cdent: i have been following your tweets on it on and off its a cool idea | 19:31 |
| *** yamamoto_ has joined #openstack-nova | 19:33 | |
| sean-k-mooney | cdent: have you considerd useing any of this work as part of functional testing for placemnet | 19:33 |
| cdent | sean-k-mooney: not realy: because of the way the existing gabbi tests work with wsgi-intercept, it's not really needed | 19:34 |
| cdent | using it for some kind of integration tests might be useful thought | 19:34 |
| sean-k-mooney | cdent: when placement has its own repo this could be an interesting way for folks to do placement dev/testing | 19:34 |
| cdent | yes | 19:34 |
| sean-k-mooney | cdent: well i was thinking it would be kind of like neutron fullstack tests | 19:34 |
| sean-k-mooney | they are more then fuctional tests but not quite intergration tests | 19:35 |
| cdent | sean-k-mooney: what do you see as the difference between using the container and what the current wsgi-intercept driven tests do? | 19:35 |
| sean-k-mooney | cdent: noting really other then the fact that you are using sqlite right so i dont need to setup a db first so it would mean a smaller testing requiremtes set | 19:36 |
| *** yamamoto has quit IRC | 19:36 | |
| *** mgoddard has quit IRC | 19:37 | |
| cdent | the container can use any database, depending on environment variables | 19:38 |
| cdent | the functional tests use sqlite, yeah? | 19:38 |
| sean-k-mooney | am im not sure what they use but were you not tweeting about using sqlight | 19:39 |
| sean-k-mooney | i could be getting confused between what can work and what that suff is using today | 19:39 |
| *** salv-orlando has joined #openstack-nova | 19:40 | |
| sean-k-mooney | anyway this time im actully going to leave since its 8:40 on a friday. enjoy your weekends o/ | 19:41 |
| cdent | o/ | 19:43 |
| *** bauzas has joined #openstack-nova | 19:58 | |
| *** baoli has quit IRC | 20:02 | |
| *** mchlumsky has quit IRC | 20:10 | |
| *** esberglu has quit IRC | 20:11 | |
| *** mchlumsky has joined #openstack-nova | 20:12 | |
| *** mikal_ is now known as mikal | 20:14 | |
| *** cdent has quit IRC | 20:16 | |
| *** artom has quit IRC | 20:21 | |
| *** links has quit IRC | 20:28 | |
| *** yamamoto_ has quit IRC | 20:28 | |
| *** esberglu has joined #openstack-nova | 20:28 | |
| *** esberglu has quit IRC | 20:34 | |
| *** gjayavelu has quit IRC | 20:36 | |
| *** hemna_ has joined #openstack-nova | 20:41 | |
| *** pchavva has quit IRC | 20:44 | |
| mriedem | for the life of me i can't figure out why this isn't a problem anymore https://bugs.launchpad.net/nova/+bug/1718512 | 20:45 |
| openstack | Launchpad bug 1718512 in OpenStack Compute (nova) "migration fails if instance build failed on destination host" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | 20:45 |
| mriedem | or, how it was a problem to begin with - since i don't know where we save 'retry' changes to a request spec during the initial build | 20:46 |
| *** edmondsw has quit IRC | 20:47 | |
| *** beekneemech has quit IRC | 20:48 | |
| mriedem | superdan: i think i see a fun little weirdness, we update the requestspec with a new flavor during resize before we've actually confirmed that the resize happened https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L349 | 20:49 |
| mriedem | b/c at that point we've rpc cast to the dest compute to start the resize | 20:49 |
| *** damien_r has joined #openstack-nova | 20:50 | |
| mriedem | if it fails on the compute, the request spec now says that the instance is using a new flavor | 20:50 |
| *** danpawlik has joined #openstack-nova | 20:50 | |
| superdan | mriedem: and that matters because why? if you later evacuate maybe? | 20:51 |
| mriedem | it might not matter right now, | 20:51 |
| mriedem | i was thinking, | 20:51 |
| mriedem | if resize does a reschedule to the cell conductor, we could try saving changes to the request spec, because of the retry stuff | 20:51 |
| superdan | actually, I think evac is probably fine since it likely uses the instance | 20:51 |
| mriedem | which would be an upcall to the api db | 20:51 |
| superdan | okay, but it shouldn't need to change the flavor | 20:52 |
| superdan | on reschedule | 20:52 |
| superdan | (from resize) | 20:52 |
| mriedem | right the flavor in the requestspec wouldn't change on a reschedule during resize, | 20:52 |
| mriedem | the RequestSpec.retry field might | 20:52 |
| *** damien_r has quit IRC | 20:53 | |
| *** AlexeyAbashkin has joined #openstack-nova | 20:53 | |
| mriedem | although, | 20:53 |
| mriedem | it looks like on a reschedule for resize, the compute doesn't send the request spec back up to the cell conductor | 20:53 |
| mriedem | which is dumb, but likely the only thing saving this | 20:54 |
| superdan | you mean "that's by design" right? :D | 20:54 |
| mriedem | f no | 20:54 |
| mriedem | god _reschedule_resize_or_reraise in the compute manager is about as convoluted as it gets | 20:55 |
| mriedem | it's not a bdm turducken, but it's close | 20:55 |
| superdan | are you just trying to start trouble on a friday afternoon? | 20:55 |
| *** claudiub|2 has joined #openstack-nova | 20:55 | |
| *** danpawlik has quit IRC | 20:56 | |
| mriedem | no, i've had a patch up for this bug since it came up in early november, but realized i needed a functional test for it which stalled it out for awhile, and then earlier in the week i triaged a duplicate of the same bug, so started working on writing that functional test, | 20:56 |
| mriedem | which now that i have the functional test written, it doesn't fail as expected... | 20:56 |
| mriedem | i pulled the reqspec out of the api db and the 'retry' field isn't saved, so i'm trying to figure out wth the reqspec gets saved during a reschedule, during build | 20:57 |
| *** AlexeyAbashkin has quit IRC | 20:57 | |
| superdan | yeah, never go trawling through your backlog on a friday | 20:58 |
| superdan | that's like tuesday stuff | 20:58 |
| mriedem | the tab has been open since thursday, if that helps | 20:58 |
| *** damien_r has joined #openstack-nova | 20:58 | |
| superdan | hah | 20:58 |
| mriedem | far as i can tell, when we're still at the top during build, we populate the filter_properties dict with the initial host we're going to try for the build here https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155 | 20:59 |
| mriedem | but that never gets persisted back into the request_spec | 21:00 |
| superdan | yeah, but it shouldn't right? | 21:01 |
| superdan | because otherwise next time we went to do something we might think they requested that host specifically? | 21:01 |
| mriedem | requesting the host specifically is the forced_hosts stuff, | 21:01 |
| mriedem | that gets reset before all move operations | 21:01 |
| mriedem | and yeah, i don't see a good reason to persist the retry hosts either | 21:01 |
| mriedem | but this bug is asserting that they are/were and that causes problems with later move operations | 21:02 |
| mriedem | time to get back to the irc logs from last september | 21:02 |
| *** awaugama has quit IRC | 21:03 | |
| *** itlinux has joined #openstack-nova | 21:03 | |
| * superdan hopes that takes longer than 27 minutes | 21:03 | |
| mriedem | oh i already found the convo http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-09-20.log.html#t2017-09-20T15:25:23 | 21:04 |
| superdan | doesn't seem to be loading for me | 21:05 |
| superdan | my browser might have fridayitus | 21:05 |
| superdan | it's a real plague | 21:05 |
| superdan | lol, I just read over the dazed and confused references in that log | 21:07 |
| superdan | good times | 21:07 |
| mriedem | heh | 21:07 |
| mriedem | the creepy guy that now drives a lincoln everywhere | 21:07 |
| superdan | lol | 21:08 |
| mriedem | this is his original build request spec http://paste.openstack.org/show/621557/ | 21:11 |
| mriedem | so, i'm wondering if either we used to save reqspec changes during reschedules back in mitaka? | 21:14 |
| mriedem | otherwise, he might have special sauce | 21:14 |
| *** danpawlik has joined #openstack-nova | 21:14 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771 | 21:18 |
| mriedem | ^ is the attempt at a recreate | 21:18 |
| *** itlinux has quit IRC | 21:18 | |
| *** baoli has joined #openstack-nova | 21:18 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Don't ignore original retried hosts when moving an instance https://review.openstack.org/505771 | 21:19 |
| *** danpawlik has quit IRC | 21:19 | |
| imacdonn | fried_rice mriedem For https://review.openstack.org/#/c/558089/ , do we need to do anything further around unit tests? I can do a new PS with the requested comment changes, but wanted to see if anything else fell out of the discussion... | 21:22 |
| *** baoli has quit IRC | 21:22 | |
| *** gouthamr has quit IRC | 21:22 | |
| *** baoli has joined #openstack-nova | 21:22 | |
| mriedem | imacdonn: i have to go back and look at replies, in the middle of debugging something | 21:22 |
| *** artom has joined #openstack-nova | 21:23 | |
| fried_rice | imacdonn: If mriedem is satisfied with my response at https://review.openstack.org/#/c/558089/1/nova/tests/unit/compute/test_compute_mgr.py@3730 then I think the remaining question is https://review.openstack.org/#/c/558089/1/nova/tests/unit/test_test.py@45 -- have you tried that out locally to see if it works? | 21:23 |
| *** lpetrut has joined #openstack-nova | 21:24 | |
| mriedem | looking back at mitaka, the api creates the request spec, but doesn't send it to conductor.build_instances, that just builds it's own fake request spec to pass to the scheduler; conductor passes that fake reqspec down to compute, where it's ignored | 21:25 |
| *** felipemonteiro__ has quit IRC | 21:25 | |
| mriedem | there are no request_spec.save() calls in mitaka, but there is that one in newton for a resize | 21:26 |
| imacdonn | fried_rice: I haven't ... because I don't really know what that test is intended to accomplish | 21:26 |
| * fried_rice looks again... | 21:26 | |
| *** sdague has quit IRC | 21:27 | |
| mriedem | so i wonder if this is what happened: built the server on host1, resized - failed on host2, rescheduled to host3, then tried to live migrate to host2 and that was kicked out because the request_spec.save() during the resize reschedule was persiste | 21:27 |
| mriedem | *persiste | 21:27 |
| mriedem | gdi | 21:27 |
| mriedem | this thing https://github.com/openstack/nova/blob/stable/pike/nova/conductor/manager.py#L1155 | 21:27 |
| mriedem | wrong link, this https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L349 | 21:27 |
| fried_rice | imacdonn: Does stuff break if you take out that particular PlacementFixture instantiation? | 21:27 |
| *** eharney has quit IRC | 21:28 | |
| imacdonn | fried_rice: yes, I had to add the PlacementFixture to get the test to not fail | 21:28 |
| *** yamamoto has joined #openstack-nova | 21:28 | |
| cfriesen_ | mriedem: seems plausible | 21:29 |
| mriedem | cfriesen_: where the hell did you come from | 21:30 |
| fried_rice | imacdonn: I think mriedem is correct that start_service('compute') would accomplish the same thing (just looking at what start_service('compute') now does, as amended by you). But then as you say, I'm not sure what we're gaining by that test in the first place, since there's any number of other tests that start_service('compute') already. If it were up to me, I'd say nix the test. So yeah, I guess let's wait for mrie | 21:30 |
| *** Guest81836 has quit IRC | 21:30 | |
| cfriesen_ | mriedem: lurking all day, got bored with trying to solve internal race conditions | 21:31 |
| imacdonn | fried_rice: ack | 21:32 |
| mriedem | imacdonn: fried_rice: replied | 21:34 |
| fried_rice | thx | 21:34 |
| imacdonn | mriedem fried_rice Thanks .. I'll start wordsmithing comments then ;) | 21:35 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597 | 21:40 |
| *** yamamoto has quit IRC | 21:41 | |
| *** hongbin has quit IRC | 21:42 | |
| imacdonn | # compute should always be configured to use placement, for reporting | 21:43 |
| imacdonn | # purposes, even if the scheduler doesn't require it. Verify this by | 21:43 |
| imacdonn | # requesting a bogus resource_provider (to minimise cost). If a | 21:43 |
| imacdonn | # MissingAuthPlugin results, it's likely due to lack of placement | 21:43 |
| imacdonn | # configuration. If the request fails in any other way, it should be | 21:43 |
| imacdonn | # obvious from the exception. | 21:43 |
| imacdonn | mriedem fried_rice any objections to that? or edits? | 21:43 |
| *** slaweq_ has joined #openstack-nova | 21:44 | |
| fried_rice | imacdonn: lgtm, but mriedem is the ocd one | 21:44 |
| fried_rice | :P | 21:44 |
| imacdonn | :) | 21:44 |
| *** slaweq_ has quit IRC | 21:46 | |
| mriedem | that's the pot calling the kettle black if i've ever heard it | 21:46 |
| *** slaweq has quit IRC | 21:46 | |
| *** mriedem is now known as mriedem_afk | 21:47 | |
| *** slaweq has joined #openstack-nova | 21:47 | |
| *** Nil_ has quit IRC | 21:47 | |
| *** leakypipes has quit IRC | 21:49 | |
| *** slaweq_ has joined #openstack-nova | 21:50 | |
| *** yassine_ has quit IRC | 21:53 | |
| *** slaweq has quit IRC | 21:55 | |
| openstackgerrit | iain MacDonnell proposed openstack/nova master: Update check to ensure compute is using placement https://review.openstack.org/558089 | 21:56 |
| *** yassine_ has joined #openstack-nova | 21:56 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Move test_report_client out of placement namespace https://review.openstack.org/558911 | 21:56 |
| *** _ix has quit IRC | 21:58 | |
| *** burt has quit IRC | 22:01 | |
| *** lpetrut has quit IRC | 22:02 | |
| *** lpetrut has joined #openstack-nova | 22:17 | |
| *** sar has quit IRC | 22:20 | |
| *** tbachman_ has joined #openstack-nova | 22:20 | |
| *** mriedem_afk has quit IRC | 22:21 | |
| *** tbachman has quit IRC | 22:22 | |
| *** tbachman_ is now known as tbachman | 22:22 | |
| *** damien_r has quit IRC | 22:22 | |
| fried_rice | รถ/ | 22:23 |
| *** fried_rice is now known as efried | 22:23 | |
| *** superdan is now known as dansmith | 22:25 | |
| *** lpetrut has quit IRC | 22:30 | |
| *** lpetrut has joined #openstack-nova | 22:30 | |
| *** mriedem has joined #openstack-nova | 22:35 | |
| *** figleaf is now known as edleafe | 22:37 | |
| *** lpetrut has quit IRC | 22:42 | |
| mriedem | dansmith: i know you want to know this, but i figured out the bug | 22:45 |
| *** liverpooler has quit IRC | 22:49 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - Add a microversion for consumer generation https://review.openstack.org/558958 | 22:49 |
| *** gjayavelu has joined #openstack-nova | 22:52 | |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | 22:55 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | 22:55 |
| *** danpawlik has joined #openstack-nova | 23:15 | |
| *** danpawlik has quit IRC | 23:19 | |
| *** r-daneel has quit IRC | 23:20 | |
| *** hongbin has joined #openstack-nova | 23:28 | |
| *** lbragstad has quit IRC | 23:34 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771 | 23:35 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771 | 23:37 |
| *** chyka has quit IRC | 23:39 | |
| *** itlinux has joined #openstack-nova | 23:40 | |
| *** sususuryashines has quit IRC | 23:51 | |
| *** AlexeyAbashkin has joined #openstack-nova | 23:53 | |
| *** danpawlik has joined #openstack-nova | 23:55 | |
| *** moshele has joined #openstack-nova | 23:57 | |
| *** AlexeyAbashkin has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!