| *** jaypipes_ is now known as jaypipes | 00:04 | |
| *** jaypipes has quit IRC | 00:05 | |
| *** yassine has joined #openstack-nova | 00:05 | |
| *** yassine is now known as Guest57292 | 00:05 | |
| *** acormier has joined #openstack-nova | 00:08 | |
| *** mtreinish has quit IRC | 00:12 | |
| *** edmondsw has joined #openstack-nova | 00:21 | |
| *** gongysh has joined #openstack-nova | 00:21 | |
| *** acormier has quit IRC | 00:24 | |
| *** yangyapeng has joined #openstack-nova | 00:25 | |
| *** edmondsw has quit IRC | 00:25 | |
| *** r-daneel has quit IRC | 00:29 | |
| *** yangyapeng has quit IRC | 00:30 | |
| *** amodi has quit IRC | 00:33 | |
| *** sdague has quit IRC | 00:36 | |
| *** hoangcx has quit IRC | 00:37 | |
| *** mtreinish has joined #openstack-nova | 00:37 | |
| *** hiro-kobayashi has joined #openstack-nova | 00:37 | |
| *** hoangcx has joined #openstack-nova | 00:37 | |
| *** chyka has quit IRC | 00:37 | |
| *** dave-mccowan has joined #openstack-nova | 00:42 | |
| *** lei-zh has quit IRC | 00:45 | |
| *** slaweq has joined #openstack-nova | 00:46 | |
| *** edleafe has quit IRC | 00:48 | |
| ameeda | jaypipes: are you around ? | 00:50 |
|---|---|---|
| *** edleafe has joined #openstack-nova | 00:51 | |
| *** slaweq has quit IRC | 00:51 | |
| *** Swami has quit IRC | 00:53 | |
| *** Dinesh_Bhor has joined #openstack-nova | 00:54 | |
| *** edleafe has quit IRC | 00:55 | |
| openstackgerrit | Ameed Ashour proposed openstack/nova master: detach instance volumes when VM creation fails https://review.openstack.org/528385 | 00:56 |
| *** Dinesh_Bhor has quit IRC | 00:56 | |
| *** claudiub has quit IRC | 00:58 | |
| *** chyka has joined #openstack-nova | 01:01 | |
| *** acormier has joined #openstack-nova | 01:04 | |
| *** acormier has joined #openstack-nova | 01:04 | |
| *** chyka has quit IRC | 01:06 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:07 | |
| *** gongysh has quit IRC | 01:09 | |
| *** edleafe has joined #openstack-nova | 01:11 | |
| *** Dinesh_Bhor has quit IRC | 01:17 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:18 | |
| *** Dinesh_Bhor has quit IRC | 01:21 | |
| *** yangyapeng has joined #openstack-nova | 01:22 | |
| *** stakeda has joined #openstack-nova | 01:22 | |
| *** liverpooler has joined #openstack-nova | 01:23 | |
| *** acormier has quit IRC | 01:28 | |
| mriedem | efried: there will be the normal project update which will talk about some placement stuff done in queens but not a deep dive | 01:30 |
| *** zhaochao has joined #openstack-nova | 01:31 | |
| mriedem | mgagne: you know this whole aggregate core/ram/disk filter thread? if you're using the CachingScheduler, it doesn't affect you | 01:31 |
| mriedem | melwitt: can you take another pass on the luks fix? https://review.openstack.org/#/c/539739/ | 01:32 |
| mriedem | we need that in before rc1 | 01:32 |
| melwitt | mriedem: sure | 01:32 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug nova-api service version check during boot from volume https://review.openstack.org/540992 | 01:34 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug nova-next image not found failures https://review.openstack.org/541008 | 01:37 |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.openstack.org/507486 | 01:43 |
| *** naichuans_ has joined #openstack-nova | 01:44 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Store a RequestContext on service to reuse for periodic tasks https://review.openstack.org/524306 | 01:44 |
| openstackgerrit | melanie witt proposed openstack/nova master: Improve the CellDatabases test fixture and usage https://review.openstack.org/508432 | 01:44 |
| *** naichuans_ has quit IRC | 01:44 | |
| *** salv-orlando has joined #openstack-nova | 01:45 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:47 | |
| openstackgerrit | Hironori Shiina proposed openstack/nova master: virt/ironic: Implement rescue and unrescue https://review.openstack.org/416487 | 01:48 |
| *** hongbin has joined #openstack-nova | 01:49 | |
| *** Dinesh_Bhor has quit IRC | 01:50 | |
| *** chyka has joined #openstack-nova | 01:50 | |
| *** slaweq has joined #openstack-nova | 01:50 | |
| *** gjayavelu has quit IRC | 01:52 | |
| *** hiro-kobayashi has quit IRC | 01:54 | |
| *** slaweq has quit IRC | 01:55 | |
| *** chyka has quit IRC | 01:55 | |
| openstackgerrit | baiwenteng proposed openstack/nova-specs master: modify spelling error https://review.openstack.org/541090 | 01:57 |
| *** sapd__ has quit IRC | 02:04 | |
| mriedem | Kevin_Zheng: yikun: easy docs bug to fix https://bugs.launchpad.net/nova/+bug/1747562 | 02:05 |
| openstack | Launchpad bug 1747562 in OpenStack Compute (nova) "CPU topologies in nova - wrong link for "Manage Flavors"" [Medium,Triaged] | 02:05 |
| *** awaugama has quit IRC | 02:05 | |
| *** sapd has joined #openstack-nova | 02:06 | |
| Kevin_Zheng | mriedem can you check https://review.openstack.org/#/c/540796/ ? Seems your idea in original patch is incorrect | 02:07 |
| mriedem | what was my idea in what original patch? | 02:07 |
| mriedem | i guess you mean https://review.openstack.org/#/c/330285/ | 02:09 |
| *** edmondsw has joined #openstack-nova | 02:09 | |
| *** acormier has joined #openstack-nova | 02:09 | |
| mriedem | anyway +W | 02:10 |
| Kevin_Zheng | https://review.openstack.org/#/c/330285/173/nova/compute/manager.py yeah | 02:10 |
| Kevin_Zheng | here | 02:10 |
| Kevin_Zheng | seems you were wrong and here response was correct but somehow the result is wrong :) | 02:11 |
| mriedem | yeah i guess i had a typo; reviewing a change with 176 patch sets will do that | 02:12 |
| Kevin_Zheng | heh | 02:12 |
| *** Dinesh_Bhor has joined #openstack-nova | 02:12 | |
| *** itlinux has joined #openstack-nova | 02:13 | |
| *** edmondsw has quit IRC | 02:14 | |
| *** esberglu has quit IRC | 02:14 | |
| *** Dinesh_Bhor has quit IRC | 02:15 | |
| *** gongysh has joined #openstack-nova | 02:18 | |
| *** harlowja has quit IRC | 02:18 | |
| *** markvoelker has joined #openstack-nova | 02:22 | |
| *** slaweq has joined #openstack-nova | 02:24 | |
| *** markvoelker has quit IRC | 02:24 | |
| *** mriedem has quit IRC | 02:25 | |
| *** acormier has quit IRC | 02:26 | |
| *** acormier has joined #openstack-nova | 02:27 | |
| *** mriedem has joined #openstack-nova | 02:28 | |
| *** slaweq has quit IRC | 02:29 | |
| *** markvoelker has joined #openstack-nova | 02:31 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:31 | |
| *** salv-orlando has quit IRC | 02:33 | |
| *** lyan has joined #openstack-nova | 02:33 | |
| *** salv-orlando has joined #openstack-nova | 02:33 | |
| yikun | mriedem, OK, thanks, I will try to fix. :D | 02:35 |
| Kevin_Zheng | yikun already assigned to you :) | 02:35 |
| mriedem | good luck | 02:36 |
| *** salv-orlando has quit IRC | 02:38 | |
| yikun | https://review.openstack.org/#/c/522161/ | 02:39 |
| yikun | : ), ^^ and this is a issue make us can't create vm with unicode server name. | 02:39 |
| yikun | I update the patch again~ | 02:39 |
| *** masahisa has quit IRC | 02:40 | |
| yikun | take a look if you have time, :) | 02:40 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Transform servergroup.addmember notification https://review.openstack.org/541101 | 02:41 |
| *** itlinux has quit IRC | 02:43 | |
| *** Tom-Tom has joined #openstack-nova | 02:45 | |
| *** dave-mccowan has quit IRC | 02:48 | |
| openstackgerrit | pangliye proposed openstack/nova master: fix misspelling of 'projectUser' https://review.openstack.org/541108 | 02:49 |
| *** bkopilov has quit IRC | 02:50 | |
| *** tuanla____ has joined #openstack-nova | 02:53 | |
| *** markvoelker has quit IRC | 02:56 | |
| *** markvoelker has joined #openstack-nova | 02:59 | |
| *** slaweq has joined #openstack-nova | 03:00 | |
| *** slaweq has quit IRC | 03:04 | |
| *** dave-mccowan has joined #openstack-nova | 03:08 | |
| *** gjayavelu has joined #openstack-nova | 03:09 | |
| *** markvoelker has quit IRC | 03:11 | |
| *** mriedem has quit IRC | 03:14 | |
| *** markvoelker has joined #openstack-nova | 03:14 | |
| *** lyan has quit IRC | 03:14 | |
| *** gjayavelu has quit IRC | 03:22 | |
| *** acormier_ has joined #openstack-nova | 03:23 | |
| *** markvoelker has quit IRC | 03:25 | |
| *** gcb has joined #openstack-nova | 03:26 | |
| *** liverpooler has quit IRC | 03:27 | |
| *** acormier has quit IRC | 03:27 | |
| *** gyee has quit IRC | 03:29 | |
| *** markvoelker has joined #openstack-nova | 03:30 | |
| *** dave-mccowan has quit IRC | 03:33 | |
| *** gongysh has quit IRC | 03:33 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova master: Use check_string_length from oslo_utils https://review.openstack.org/286390 | 03:34 |
| openstackgerrit | LiChunlin proposed openstack/nova master: update the description of hypervisor statistics response https://review.openstack.org/462355 | 03:35 |
| *** masahisa has joined #openstack-nova | 03:35 | |
| *** acormier_ has quit IRC | 03:37 | |
| *** chyka has joined #openstack-nova | 03:39 | |
| *** slaweq has joined #openstack-nova | 03:40 | |
| *** markvoelker has quit IRC | 03:42 | |
| *** chyka has quit IRC | 03:44 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Fix wrong link for "Manage Flavors" in CPU topologies doc https://review.openstack.org/541116 | 03:44 |
| *** slaweq has quit IRC | 03:44 | |
| *** crushil has joined #openstack-nova | 03:48 | |
| *** sree has joined #openstack-nova | 03:50 | |
| *** jdurgin has quit IRC | 03:50 | |
| *** jdurgin has joined #openstack-nova | 03:52 | |
| *** hongbin has quit IRC | 03:55 | |
| *** edmondsw has joined #openstack-nova | 03:58 | |
| *** gongysh has joined #openstack-nova | 04:01 | |
| *** edmondsw has quit IRC | 04:02 | |
| openstackgerrit | Merged openstack/nova master: [placement] Add sending global request ID in post https://review.openstack.org/526823 | 04:05 |
| openstackgerrit | Merged openstack/nova master: Trivial fix a missleading comment https://review.openstack.org/540796 | 04:05 |
| *** bkopilov has joined #openstack-nova | 04:08 | |
| *** pooja_jadhav has joined #openstack-nova | 04:11 | |
| *** gongysh has quit IRC | 04:11 | |
| *** udesale has joined #openstack-nova | 04:13 | |
| *** gongysh has joined #openstack-nova | 04:14 | |
| *** abhishekk has joined #openstack-nova | 04:21 | |
| *** ratailor has joined #openstack-nova | 04:22 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Adds view builders for keypairs controller https://review.openstack.org/347289 | 04:22 |
| *** psachin has joined #openstack-nova | 04:26 | |
| *** hiro-kobayashi has joined #openstack-nova | 04:29 | |
| *** gongysh has quit IRC | 04:30 | |
| *** yamamoto has joined #openstack-nova | 04:35 | |
| *** Tom-Tom has quit IRC | 04:36 | |
| *** Tom-Tom has joined #openstack-nova | 04:36 | |
| *** crushil has quit IRC | 04:38 | |
| *** lpetrut has joined #openstack-nova | 04:41 | |
| *** Tom-Tom has quit IRC | 04:41 | |
| *** crushil has joined #openstack-nova | 04:42 | |
| *** slaweq has joined #openstack-nova | 04:44 | |
| *** harlowja has joined #openstack-nova | 04:46 | |
| *** slaweq has quit IRC | 04:48 | |
| *** sree_ has joined #openstack-nova | 04:53 | |
| *** sree_ is now known as Guest77194 | 04:54 | |
| *** sree has quit IRC | 04:57 | |
| *** trinaths has joined #openstack-nova | 05:02 | |
| *** zhurong has quit IRC | 05:07 | |
| *** lpetrut has quit IRC | 05:08 | |
| *** Tom-Tom has joined #openstack-nova | 05:10 | |
| *** harlowja has quit IRC | 05:11 | |
| *** links has joined #openstack-nova | 05:12 | |
| *** janki has joined #openstack-nova | 05:15 | |
| *** lpetrut has joined #openstack-nova | 05:16 | |
| *** itlinux has joined #openstack-nova | 05:17 | |
| *** gongysh has joined #openstack-nova | 05:19 | |
| *** chyka has joined #openstack-nova | 05:22 | |
| *** lyan has joined #openstack-nova | 05:22 | |
| *** tuanla____ has quit IRC | 05:23 | |
| *** hoangcx has quit IRC | 05:26 | |
| *** moshele has joined #openstack-nova | 05:27 | |
| *** sdake_ is now known as sdake | 05:30 | |
| *** moshele has quit IRC | 05:30 | |
| *** sree has joined #openstack-nova | 05:31 | |
| *** lpetrut has quit IRC | 05:32 | |
| *** Guest77194 has quit IRC | 05:34 | |
| *** acormier has joined #openstack-nova | 05:38 | |
| *** acormier has quit IRC | 05:42 | |
| *** mdnadeem has joined #openstack-nova | 05:44 | |
| *** edmondsw has joined #openstack-nova | 05:46 | |
| *** andrewbogott has quit IRC | 05:49 | |
| *** andrewbogott has joined #openstack-nova | 05:49 | |
| *** edmondsw has quit IRC | 05:50 | |
| *** sree_ has joined #openstack-nova | 05:54 | |
| *** sree_ is now known as Guest54253 | 05:54 | |
| *** itlinux has quit IRC | 05:55 | |
| *** sree has quit IRC | 05:58 | |
| *** gcb has quit IRC | 06:00 | |
| *** slaweq has joined #openstack-nova | 06:02 | |
| *** Dinesh_Bhor has quit IRC | 06:04 | |
| *** Dinesh_Bhor has joined #openstack-nova | 06:05 | |
| *** jchhatbar has joined #openstack-nova | 06:06 | |
| *** janki has quit IRC | 06:06 | |
| *** slaweq has quit IRC | 06:07 | |
| *** Dinesh__Bhor has joined #openstack-nova | 06:09 | |
| *** Dinesh_Bhor has quit IRC | 06:10 | |
| *** lpetrut has joined #openstack-nova | 06:11 | |
| *** hiro-kobayashi has quit IRC | 06:12 | |
| *** chyka has quit IRC | 06:15 | |
| *** takedakn has joined #openstack-nova | 06:18 | |
| *** lpetrut has quit IRC | 06:18 | |
| *** takedakn has quit IRC | 06:19 | |
| *** moshele has joined #openstack-nova | 06:20 | |
| jianghuaw | bauzas, you're right. XenServer has the same problem for VM resizing and migration. The new VM has no vGPU after these operations. We need fix that in Rocky. | 06:33 |
| *** lyan has quit IRC | 06:35 | |
| jianghuaw | bauzas, for "rescue VM", the instance being rescued won't have vGPU. But after unrescue it, it will get the vGPU again. Also placement will hold the allocation. So I think it's not a problem. | 06:36 |
| jianghuaw | I guess there is no need to use workaround to rebuild an instance after rescue. | 06:36 |
| *** ratailor_ has joined #openstack-nova | 06:36 | |
| *** claudiub has joined #openstack-nova | 06:37 | |
| jianghuaw | Maybe kvm is different for this item. Need you help to clarify. | 06:37 |
| *** crushil has quit IRC | 06:38 | |
| *** ratailor has quit IRC | 06:38 | |
| *** threestrands has quit IRC | 06:40 | |
| *** kholkina has joined #openstack-nova | 06:49 | |
| *** sridharg has joined #openstack-nova | 06:53 | |
| *** Dinesh__Bhor has quit IRC | 06:57 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Avoid raise InstanceNotFound exception https://review.openstack.org/541152 | 07:00 |
| *** Eran_Kuris_ has joined #openstack-nova | 07:02 | |
| *** sree has joined #openstack-nova | 07:04 | |
| *** Eran_Kuris_ has quit IRC | 07:05 | |
| *** Guest54253 has quit IRC | 07:07 | |
| *** josecastroleon has quit IRC | 07:08 | |
| *** Dinesh__Bhor has joined #openstack-nova | 07:08 | |
| *** zhurong has joined #openstack-nova | 07:09 | |
| *** Dinesh__Bhor has quit IRC | 07:10 | |
| *** gcb has joined #openstack-nova | 07:13 | |
| *** andreas_s has joined #openstack-nova | 07:14 | |
| *** khappone has quit IRC | 07:17 | |
| *** crushil has joined #openstack-nova | 07:19 | |
| *** jchhatba_ has joined #openstack-nova | 07:20 | |
| *** jchhatbar has quit IRC | 07:23 | |
| *** lpetrut has joined #openstack-nova | 07:24 | |
| openstackgerrit | Nakanishi Tomotaka proposed openstack/nova master: Test Compute API in multiple cells https://review.openstack.org/534116 | 07:25 |
| *** AlexeyAbashkin has joined #openstack-nova | 07:32 | |
| *** rcernin has quit IRC | 07:37 | |
| *** AlexeyAbashkin has quit IRC | 07:39 | |
| *** khappone has joined #openstack-nova | 07:39 | |
| *** bhagyashris has quit IRC | 07:42 | |
| *** AlexeyAbashkin has joined #openstack-nova | 07:48 | |
| *** slaweq has joined #openstack-nova | 07:49 | |
| *** jpena|off is now known as jpena | 07:53 | |
| *** sree has quit IRC | 07:59 | |
| *** slaweq has quit IRC | 08:02 | |
| *** alexchadin has joined #openstack-nova | 08:02 | |
| *** takashin has left #openstack-nova | 08:03 | |
| *** slaweq has joined #openstack-nova | 08:03 | |
| *** ilyashakhat has quit IRC | 08:08 | |
| *** jchhatba_ has quit IRC | 08:12 | |
| *** jchhatba_ has joined #openstack-nova | 08:12 | |
| *** ccamacho has joined #openstack-nova | 08:13 | |
| *** pcaruana has joined #openstack-nova | 08:14 | |
| openstackgerrit | Jianghua Wang proposed openstack/nova master: XenAPI: Provide support matrix and doc for VGPU https://review.openstack.org/540808 | 08:19 |
| *** abhishekk has quit IRC | 08:21 | |
| *** tesseract has joined #openstack-nova | 08:22 | |
| openstackgerrit | Merged openstack/nova master: [placement] Add sending global request ID in put (1) https://review.openstack.org/531258 | 08:23 |
| *** sahid has joined #openstack-nova | 08:23 | |
| *** ralonsoh has joined #openstack-nova | 08:23 | |
| hrw | morning | 08:25 |
| *** bhagyashris has joined #openstack-nova | 08:25 | |
| *** damien_r has joined #openstack-nova | 08:26 | |
| *** lajoskatona has joined #openstack-nova | 08:32 | |
| *** zhurong has quit IRC | 08:38 | |
| *** yamahata has quit IRC | 08:38 | |
| *** Dinesh__Bhor has joined #openstack-nova | 08:39 | |
| *** abhishekk has joined #openstack-nova | 08:41 | |
| *** rodolof has joined #openstack-nova | 08:47 | |
| *** crushil has quit IRC | 08:49 | |
| *** gjayavelu has joined #openstack-nova | 08:49 | |
| *** tssurya has joined #openstack-nova | 08:51 | |
| *** hiro-kobayashi has joined #openstack-nova | 08:53 | |
| *** gongysh has quit IRC | 08:53 | |
| *** mgoddard_ has joined #openstack-nova | 08:54 | |
| *** amoralej|off is now known as amoralej | 08:56 | |
| *** priteau has joined #openstack-nova | 08:56 | |
| *** alexchadin has quit IRC | 08:57 | |
| *** rodolof has quit IRC | 08:58 | |
| *** alexchadin has joined #openstack-nova | 08:58 | |
| *** Dinesh__Bhor has quit IRC | 09:00 | |
| *** cdent has joined #openstack-nova | 09:02 | |
| *** zhurong has joined #openstack-nova | 09:03 | |
| *** gongysh has joined #openstack-nova | 09:04 | |
| *** Dinesh__Bhor has joined #openstack-nova | 09:04 | |
| *** threestrands has joined #openstack-nova | 09:06 | |
| *** jaosorior has quit IRC | 09:06 | |
| *** ragiman has joined #openstack-nova | 09:08 | |
| *** sree has joined #openstack-nova | 09:12 | |
| *** masahisa_ has joined #openstack-nova | 09:15 | |
| *** masahisa has quit IRC | 09:16 | |
| *** threestrands has quit IRC | 09:21 | |
| *** edmondsw has joined #openstack-nova | 09:22 | |
| *** wxy has quit IRC | 09:22 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.openstack.org/507486 | 09:25 |
| stephenfin | o/ | 09:25 |
| *** edmondsw has quit IRC | 09:27 | |
| *** masahisa has joined #openstack-nova | 09:28 | |
| *** hoonetorg has quit IRC | 09:28 | |
| *** jaosorior has joined #openstack-nova | 09:29 | |
| *** masahisa_ has quit IRC | 09:30 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/nova master: Make sure that we have usable input for graphical console https://review.openstack.org/538003 | 09:32 |
| *** pooja_jadhav is now known as neha_alhat | 09:32 | |
| bauzas | good morning | 09:35 |
| bauzas | stephenfin: could you please give me again the change you want me to review ? | 09:35 |
| stephenfin | bauzas: It was that backport patch from mdbooth, but I'm told there's a small issue with it that should be resolved on master first | 09:36 |
| stephenfin | bauzas: I'll ping you if/when it's ready to go again :) | 09:36 |
| bauzas | k | 09:38 |
| hrw | 14th version pushed. hope that it will be final ;D | 09:38 |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Fix wrong link for "Manage Flavors" in CPU topologies doc https://review.openstack.org/541116 | 09:40 |
| *** hoonetorg has joined #openstack-nova | 09:41 | |
| *** derekh has joined #openstack-nova | 09:45 | |
| *** takedakn has joined #openstack-nova | 09:49 | |
| hrw | lovely comment from sahid ;) | 09:52 |
| *** stakeda has quit IRC | 09:54 | |
| *** takedakn has quit IRC | 09:54 | |
| *** Dinesh__Bhor has quit IRC | 09:57 | |
| *** rmart04 has joined #openstack-nova | 09:58 | |
| *** hiro-kobayashi has quit IRC | 10:02 | |
| *** trinaths has quit IRC | 10:05 | |
| *** janki has joined #openstack-nova | 10:06 | |
| *** jchhatba_ has quit IRC | 10:07 | |
| gibi | ameeda: hi! Thanks for updating the cleanup patch. I have one small nit: https://review.openstack.org/#/c/528385/17/nova/compute/manager.py@2508 | 10:12 |
| *** gjayavelu has quit IRC | 10:13 | |
| ameeda | gibi: I will add it now, Thank you !! | 10:13 |
| *** sree has quit IRC | 10:13 | |
| *** sree has joined #openstack-nova | 10:14 | |
| *** jaianshu has joined #openstack-nova | 10:14 | |
| *** sree has quit IRC | 10:18 | |
| *** nmathew has joined #openstack-nova | 10:19 | |
| *** sree has joined #openstack-nova | 10:19 | |
| gibi | ameeda: I like the fact that the current patch does not change the notification functional tests :) | 10:21 |
| vivsoni_ | Hi Team, i have just now created devstack newton | 10:21 |
| vivsoni_ | cinder service-list works fine | 10:22 |
| vivsoni_ | but nova service-list | 10:22 |
| vivsoni_ | gives error | 10:22 |
| vivsoni_ | ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. | 10:22 |
| vivsoni_ | <type 'exceptions.ImportError'> (HTTP 500) (Request-ID: req-8d0cdb9c-4ae4-4544-9368-7934404aac53) | 10:22 |
| *** trinaths has joined #openstack-nova | 10:22 | |
| *** alexchadin has quit IRC | 10:22 | |
| vivsoni_ | not nova service-list... but it faild while creating nova boot | 10:22 |
| vivsoni_ | command failed : nova boot --image cirros-0.3.4-x86_64-uec --flavor m1.tiny inst_1 | 10:23 |
| *** sree has quit IRC | 10:23 | |
| *** artom has quit IRC | 10:24 | |
| *** artom has joined #openstack-nova | 10:24 | |
| *** dtantsur|afk is now known as dtantsur | 10:25 | |
| *** alexchadin has joined #openstack-nova | 10:25 | |
| ameeda | gibi: it's my pleasure to do thing which keeps openstack stable :) | 10:26 |
| *** azaid__ has joined #openstack-nova | 10:26 | |
| *** bkopilov has quit IRC | 10:27 | |
| openstackgerrit | Ameed Ashour proposed openstack/nova master: detach instance volumes when VM creation fails https://review.openstack.org/528385 | 10:27 |
| ameeda | gibi: patch uploaded ^ :) | 10:27 |
| *** abhishekk has quit IRC | 10:27 | |
| *** alexchadin has quit IRC | 10:30 | |
| azaid__ | ameeda: Okay i will check it | 10:30 |
| -openstackstatus- NOTICE: Our Zuul infrastructure is currently experiencing some problems and processing jobs very slowly, we're investigating. Please do not approve or recheck changes for now. | 10:30 | |
| *** alexchadin has joined #openstack-nova | 10:30 | |
| *** alexchadin has quit IRC | 10:30 | |
| stephenfin | vivsoni_: You need to provide more detailed, debug-level logs, for us to have any hope with that | 10:31 |
| stephenfin | vivsoni_: Looks like you're missing some Python package though | 10:31 |
| *** alexchadin has joined #openstack-nova | 10:31 | |
| stephenfin | vivsoni_: Report the bug as requested and we will take a look | 10:31 |
| *** alexchadin has quit IRC | 10:31 | |
| hrw | vivsoni_: newton? | 10:32 |
| bauzas | vivsoni_: stephenfin: that really looks like a packaging issue, hence not something we should mark a bug | 10:32 |
| *** alexchadin has joined #openstack-nova | 10:32 | |
| bauzas | as* | 10:32 |
| *** alexchadin has quit IRC | 10:32 | |
| bauzas | for the first resolution stage, I'd discuss that in #openstack or ask.openstack.org | 10:32 |
| gibi | ameeda: looking | 10:32 |
| stephenfin | bauzas: Good point | 10:33 |
| *** alexchadin has joined #openstack-nova | 10:33 | |
| *** alexchadin has quit IRC | 10:33 | |
| * bauzas really trying to reduce the open backlog :) | 10:33 | |
| vivsoni_ | hrw: yes newton | 10:33 |
| *** azaid__ has quit IRC | 10:33 | |
| vivsoni_ | stephenfin: sure i am logging a bug | 10:33 |
| vivsoni_ | will share the launchpad bug ID | 10:34 |
| stephenfin | vivsoni_: Before you do, maybe see bauzas' suggestion above | 10:34 |
| hrw | I hope to be able to forget about <queens in 2-3 months | 10:34 |
| vivsoni_ | stephenfin: ok | 10:34 |
| stephenfin | vivsoni_: Someone on #openstack would be able to help. Also note that 'newton' is unsupported now so, unless it's still an issue with recent releases, your best bet would be your vendor | 10:34 |
| *** dtruong has quit IRC | 10:35 | |
| *** dtruong has joined #openstack-nova | 10:35 | |
| vivsoni_ | stephenfin: ok | 10:35 |
| gibi | ameeda: +2, thank you for the work and the perseverance :) | 10:36 |
| * gibi is off to the pool and then to lunch | 10:37 | |
| hrw | can someone tell me at which moment instance xml is passed to libvirt? | 10:39 |
| *** trinaths has quit IRC | 10:42 | |
| ameeda | gibi: thank you very much !! | 10:42 |
| bauzas | hrw: in a shit number of places :p | 10:43 |
| *** andreas_s has quit IRC | 10:43 | |
| bauzas | in spawn() you mean ? | 10:43 |
| hrw | bauzas: sort of, yes | 10:43 |
| *** trinaths has joined #openstack-nova | 10:43 | |
| hrw | bauzas: I would like to add few pcie-root-port entries but they should be added after libvirt creates own xml | 10:43 |
| ameeda | gibi: I have another bug which has some problem , please note this https://review.openstack.org/#/c/526900/28/nova/tests/functional/db/test_archive.py. this caused after I wrote sqlite migration script https://review.openstack.org/#/c/526900/29/nova/db/sqlalchemy/migrate_repo/versions/378_instance_system_metadata_sqlite_upgrade.sql | 10:43 |
| bauzas | hrw: _get_guest_config() is where we build the XML IIRC | 10:44 |
| hrw | bauzas: too early | 10:44 |
| hrw | bauzas: I am adding usb host controller and usb keyboard in _get_guest_config() already | 10:44 |
| hrw | but pci(e) is added by libvirt not nova ;( | 10:45 |
| bauzas | I need to disappear because $gym | 10:45 |
| bauzas | but let's wrap that up when I'm back | 10:45 |
| *** andreas_s has joined #openstack-nova | 10:47 | |
| *** abalutoiu__ has joined #openstack-nova | 10:49 | |
| *** cdent has quit IRC | 10:49 | |
| * hrw off for ~2h | 10:51 | |
| *** chyka has joined #openstack-nova | 10:52 | |
| *** lucas-afk is now known as lucasagomes | 10:52 | |
| *** abalutoiu_ has quit IRC | 10:52 | |
| *** sambetts|afk is now known as sambetts | 10:54 | |
| *** chyka has quit IRC | 10:56 | |
| *** Tom-Tom has quit IRC | 10:57 | |
| *** andreas_s has quit IRC | 10:57 | |
| *** Tom-Tom has joined #openstack-nova | 10:58 | |
| *** andreas_s has joined #openstack-nova | 10:58 | |
| *** yamamoto has quit IRC | 10:59 | |
| *** Tom-Tom has quit IRC | 11:02 | |
| *** yamamoto has joined #openstack-nova | 11:03 | |
| *** Tom-Tom has joined #openstack-nova | 11:04 | |
| openstackgerrit | nalini proposed openstack/nova master: Modify show aggregate to display 'updated_at' value https://review.openstack.org/537334 | 11:04 |
| *** andreas_s has quit IRC | 11:07 | |
| *** andreas_s has joined #openstack-nova | 11:08 | |
| *** Tom-Tom has quit IRC | 11:08 | |
| *** edmondsw has joined #openstack-nova | 11:10 | |
| *** alexchadin has joined #openstack-nova | 11:11 | |
| openstackgerrit | Jianghua Wang proposed openstack/nova master: XenAPI: Provide support matrix and doc for VGPU https://review.openstack.org/540808 | 11:12 |
| *** edmondsw has quit IRC | 11:15 | |
| *** udesale has quit IRC | 11:22 | |
| *** cdent has joined #openstack-nova | 11:22 | |
| *** andreas_s has quit IRC | 11:22 | |
| *** andreas_s has joined #openstack-nova | 11:23 | |
| *** nicolasbock has joined #openstack-nova | 11:26 | |
| *** andreas_s has quit IRC | 11:27 | |
| *** andreas_s has joined #openstack-nova | 11:27 | |
| openstackgerrit | Chris Dent proposed openstack/nova-specs master: Add generation support in aggregate association https://review.openstack.org/540447 | 11:29 |
| stvnoyes | mriedem: ok I'll look into it. Have you seen any problems or is your concern theoretical at the moment? | 11:30 |
| *** nmathew has quit IRC | 11:38 | |
| *** trinaths has quit IRC | 11:40 | |
| *** links has quit IRC | 11:41 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Remove microversion fallback code from report client https://review.openstack.org/528794 | 11:42 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Make bdms querying in multiple cells use scatter-gather https://review.openstack.org/541246 | 11:43 |
| *** Tom-Tom has joined #openstack-nova | 11:47 | |
| *** sdague has joined #openstack-nova | 11:51 | |
| *** sree has joined #openstack-nova | 11:51 | |
| *** sree_ has joined #openstack-nova | 11:51 | |
| *** sree_ is now known as Guest61310 | 11:52 | |
| *** links has joined #openstack-nova | 11:54 | |
| *** sree has quit IRC | 11:55 | |
| *** Eran_Kuris has quit IRC | 11:56 | |
| *** tbachman has quit IRC | 11:56 | |
| *** gongysh has quit IRC | 11:57 | |
| *** Eran_Kuris has joined #openstack-nova | 11:57 | |
| *** Eran_Kuris has quit IRC | 12:01 | |
| *** Eran_Kuris has joined #openstack-nova | 12:02 | |
| *** moshele has quit IRC | 12:03 | |
| *** acormier has joined #openstack-nova | 12:03 | |
| *** masahisa has quit IRC | 12:04 | |
| *** moshele has joined #openstack-nova | 12:04 | |
| *** acormier has quit IRC | 12:04 | |
| *** fyxim has quit IRC | 12:11 | |
| *** fyxim has joined #openstack-nova | 12:11 | |
| *** Guest61310 has quit IRC | 12:14 | |
| *** sree has joined #openstack-nova | 12:15 | |
| *** sree has quit IRC | 12:19 | |
| *** ragiman has quit IRC | 12:24 | |
| *** vladikr has joined #openstack-nova | 12:31 | |
| *** ragiman has joined #openstack-nova | 12:31 | |
| *** ratailor_ has quit IRC | 12:33 | |
| *** nsingh has joined #openstack-nova | 12:36 | |
| *** cdent has quit IRC | 12:36 | |
| nsingh | andymccr: is there any framework to develop ansible plugin??? | 12:39 |
| *** jaianshu has quit IRC | 12:50 | |
| *** jpena is now known as jpena|lunch | 12:51 | |
| *** yangyapeng has quit IRC | 12:52 | |
| nsingh | evrardjp:is there any framework to develop ansible plugin??? | 12:52 |
| *** moshele has quit IRC | 12:53 | |
| *** yangyapeng has joined #openstack-nova | 12:53 | |
| evrardjp | what do you mean? | 12:53 |
| evrardjp | nsingh: ^ | 12:53 |
| *** moshele has joined #openstack-nova | 12:53 | |
| *** yangyapeng has quit IRC | 12:53 | |
| evrardjp | ansible is the framework, you can simply use it for writing your own plugins... | 12:54 |
| evrardjp | if you need stuff to be included in ansible and/or openstack-ansible, we can help you indeed | 12:54 |
| evrardjp | depending on the time ofc | 12:54 |
| *** yangyapeng has joined #openstack-nova | 12:54 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add regression test for bug 1735407 https://review.openstack.org/526095 | 12:54 |
| openstack | bug 1735407 in OpenStack Compute (nova) "[Nova] Evacuation doesn't respect anti-affinity rules" [Medium,In progress] https://launchpad.net/bugs/1735407 - Assigned to Balazs Gibizer (balazs-gibizer) | 12:54 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add late server group policy check to rebuild https://review.openstack.org/525242 | 12:54 |
| *** gongysh has joined #openstack-nova | 12:54 | |
| *** links has quit IRC | 12:55 | |
| nsingh | evrardjp:ok.. https://github.com/ansible/ansible is this the frmework you talking about? | 12:55 |
| gibi | alex_xu, mriedem: I've fixed your comments ^^ | 12:56 |
| evrardjp | nsingh: what are you trying to achieve? | 12:56 |
| *** takamatsu has joined #openstack-nova | 12:57 | |
| nsingh | evrardjp: i am new to ansible. i want to develop ansible plugin for project(masakari). | 12:58 |
| *** eharney has joined #openstack-nova | 12:58 | |
| evrardjp | to do what? | 12:58 |
| evrardjp | I think we are probably not on the right channel :) | 12:59 |
| *** AlexeyAbashkin has quit IRC | 12:59 | |
| nsingh | ok.. thanks | 12:59 |
| *** zhurong_ has joined #openstack-nova | 13:01 | |
| *** AlexeyAbashkin has joined #openstack-nova | 13:01 | |
| *** yangyapeng has quit IRC | 13:02 | |
| *** udesale has joined #openstack-nova | 13:02 | |
| *** liverpooler has joined #openstack-nova | 13:04 | |
| *** links has joined #openstack-nova | 13:08 | |
| *** zhurong has quit IRC | 13:08 | |
| *** tbachman has joined #openstack-nova | 13:10 | |
| *** jaypipes has joined #openstack-nova | 13:10 | |
| *** pchavva has joined #openstack-nova | 13:12 | |
| sean-k-mooney | o/ jaypipes how is your new laptop treating you? | 13:12 |
| sean-k-mooney | jaypipes: its rather early for you no? | 13:13 |
| *** edmondsw has joined #openstack-nova | 13:13 | |
| jaypipes | sean-k-mooney: no, 8am. new lappie is fantastic. :) | 13:17 |
| sean-k-mooney | let me know how the battery life is that the main thing that has been putting me off getting one. | 13:18 |
| *** cdent has joined #openstack-nova | 13:19 | |
| *** amoralej is now known as amoralej|lunch | 13:19 | |
| sean-k-mooney | im looking at the new lg gram mainly for battery but i like system76 as a company so plus the galago looks like the perfect dev laptop if you travel even a little | 13:20 |
| *** acormier has joined #openstack-nova | 13:20 | |
| *** udesale has quit IRC | 13:21 | |
| *** udesale has joined #openstack-nova | 13:22 | |
| *** alexchadin has quit IRC | 13:22 | |
| *** Eran_Kuris has quit IRC | 13:22 | |
| *** alexchadin has joined #openstack-nova | 13:23 | |
| jaypipes | sean-k-mooney: it is excellent. and the damn keys are in the right place! /me shakes fist at Macbook. | 13:23 |
| *** jaosorior has quit IRC | 13:23 | |
| jaypipes | sean-k-mooney: also, the screen is fantastic. | 13:24 |
| jaypipes | sean-k-mooney: and for $1100 including a 3-year warranty, you can't beat the price IMHO. | 13:24 |
| *** acormier has quit IRC | 13:25 | |
| *** AlexeyAbashkin has quit IRC | 13:25 | |
| hrw | zuul... 4h from sending patch and still no results ;( | 13:25 |
| *** rodolof has joined #openstack-nova | 13:26 | |
| *** AlexeyAbashkin has joined #openstack-nova | 13:27 | |
| sean-k-mooney | jaypipes: well not haveing to pay 23% vat on import certenly helps | 13:27 |
| *** bkopilov has joined #openstack-nova | 13:27 | |
| jaypipes | sean-k-mooney: heh, well I can't help you with that :) | 13:29 |
| *** dave-mccowan has joined #openstack-nova | 13:29 | |
| jaypipes | hrw: your patch hasn't even made it to the zuul birthing canal yet. | 13:29 |
| jaypipes | hrw: contractions will likely begin in about another hour. | 13:30 |
| *** yangyapeng has joined #openstack-nova | 13:30 | |
| hrw | jaypipes: looks like | 13:32 |
| *** dave-mccowan has quit IRC | 13:35 | |
| *** dave-mcc_ has joined #openstack-nova | 13:35 | |
| *** hemna_ has joined #openstack-nova | 13:36 | |
| *** acormier has joined #openstack-nova | 13:37 | |
| *** yangyapeng has quit IRC | 13:38 | |
| *** AlexeyAbashkin has quit IRC | 13:38 | |
| *** acormier_ has joined #openstack-nova | 13:41 | |
| *** acormier has quit IRC | 13:42 | |
| *** janki has quit IRC | 13:44 | |
| *** yangyapeng has joined #openstack-nova | 13:44 | |
| *** acormier_ has quit IRC | 13:46 | |
| *** jpena|lunch is now known as jpena | 13:47 | |
| bauzas | hrw: so you had a point | 13:47 |
| bauzas | about when nova is passing the XML to libvirt | 13:47 |
| hrw | bauzas: yes | 13:47 |
| bauzas | did you found it ? | 13:47 |
| hrw | not yet. digged in other stuff | 13:48 |
| *** yangyapeng has quit IRC | 13:50 | |
| *** masahisa has joined #openstack-nova | 13:50 | |
| hrw | bauzas: _create_domain() :D | 13:50 |
| bauzas | there is another method AFAIR | 13:50 |
| hrw | this one calls libvirt_guest.Guest | 13:51 |
| bauzas | at least _create_domain_and_network() AFAIK | 13:51 |
| hrw | _create_domain_and_network calls _create_domain | 13:52 |
| *** READ10 has joined #openstack-nova | 13:52 | |
| *** zhurong_ has quit IRC | 13:52 | |
| bauzas | oh my bad yeah | 13:52 |
| bauzas | for spawn at leasy | 13:52 |
| hrw | bauzas: that's the only point I need :) | 13:52 |
| *** esberglu has joined #openstack-nova | 13:53 | |
| bauzas | k | 13:53 |
| *** masahisa_ has joined #openstack-nova | 13:54 | |
| *** masahisa has quit IRC | 13:55 | |
| *** Sukhdev_ has joined #openstack-nova | 13:55 | |
| *** jmlowe has quit IRC | 13:55 | |
| openstackgerrit | Jacek Tomasiak proposed openstack/python-novaclient master: Fix listing of instances above API max_limit https://review.openstack.org/534222 | 13:56 |
| ericyoung | We have started to see failures in the tempest.scenario.test_encrypted_cinder_volume tests with the ScaleIo backend. | 13:58 |
| ericyoung | Any known issues with Luks encrypted volumes? | 13:58 |
| *** masahisa_ has quit IRC | 13:59 | |
| *** hrw has quit IRC | 13:59 | |
| *** hrw has joined #openstack-nova | 14:01 | |
| ameeda | Hello, could you please give me another +2 here https://review.openstack.org/#/c/528385/ | 14:01 |
| hrw | jaypipes: thanks for +2+W :) | 14:03 |
| *** moshele has quit IRC | 14:03 | |
| *** jaosorior has joined #openstack-nova | 14:04 | |
| jaypipes | hrw: np. | 14:04 |
| *** psachin has quit IRC | 14:04 | |
| efried | Quick and easy approval: https://review.openstack.org/#/c/533237/ (just registering what PowerVM did in Q) | 14:05 |
| jaosorior | anybody knows what actions trigger the compute.instance.update notification? | 14:07 |
| gibi | jaosorior: hi | 14:07 |
| gibi | jaosorior: let me dig a bit | 14:07 |
| jaosorior | thanks gibi! | 14:08 |
| ameeda | HELP: when I run .tox/functional/bin/python -m testtools.run nova.tests.functional.db.test_archive.TestDatabaseArchive.test_archive_deleted_rows | 14:09 |
| ameeda | I got this error esttools.matchers._impl.MismatchError: 'instances' not in {'instance_system_metadata': 11, 'instance_actions_events': 2, 'block_device_mapping': 1, 'instance_info_caches': 1, 'instance_actions': 2, 'instance_extra': 1, 'security_group_instance_association': 1} | 14:09 |
| ameeda | this caused after I wrote migration script for sqllite here https://review.openstack.org/#/c/526900/29/nova/db/sqlalchemy/migrate_repo/versions/378_instance_system_metadata_sqlite_upgrade.sql | 14:09 |
| ameeda | what I did wrong to get that ? | 14:09 |
| *** yangyapeng has joined #openstack-nova | 14:11 | |
| *** abhishekk has joined #openstack-nova | 14:13 | |
| gibi | jaosorior: hard to answer in general. That notification is eventually emited from https://github.com/openstack/nova/blob/2c1874a0ecdd1b5ce7670cdfc42396e90e3a55aa/nova/notifications/base.py#L233 | 14:13 |
| gibi | jaosorior: and tha is for example called for every action that changes a persistent attribute of an Instance object https://github.com/openstack/nova/blob/2c1874a0ecdd1b5ce7670cdfc42396e90e3a55aa/nova/objects/instance.py#L826 | 14:14 |
| jaosorior | gibi: so, from what I can gather, updates are triggered by metadata changes (which is exactly what I need). but I wanted to know what else would trigger it | 14:14 |
| jaosorior | I see | 14:14 |
| *** andreas_s has quit IRC | 14:15 | |
| gibi | jaosorior: yes, metadata update on the Instance also triggers this code path | 14:15 |
| *** andreas_s has joined #openstack-nova | 14:15 | |
| efried | stephenfin or gibi, care to push this? https://review.openstack.org/#/c/533237/ | 14:17 |
| *** andreas_s has quit IRC | 14:18 | |
| stephenfin | efried: Sure, done | 14:18 |
| *** mriedem has joined #openstack-nova | 14:18 | |
| efried | Thanks! | 14:18 |
| *** andreas_s has joined #openstack-nova | 14:18 | |
| gibi | efried: I missed it :0 | 14:20 |
| gibi | :) | 14:20 |
| efried | gibi: Don't worry, I've got other patches to satisfy your cravings. | 14:21 |
| *** chyka has joined #openstack-nova | 14:23 | |
| bauzas | mriedem: are we considering https://bugs.launchpad.net/nova/+bug/1747582 as a Won't Fix ? | 14:24 |
| openstack | Launchpad bug 1747582 in OpenStack Compute (nova) "delete a resize instance change quota_usages incorrectly" [Undecided,New] - Assigned to guolidong (guolidong) | 14:24 |
| bauzas | I mean, we did a huge work on quotas thanks to melwitt after Newton | 14:24 |
| *** liverpooler has quit IRC | 14:24 | |
| *** amoralej|lunch is now known as amoralej | 14:24 | |
| *** liverpooler has joined #openstack-nova | 14:25 | |
| openstackgerrit | Jianghua Wang proposed openstack/nova master: XenAPI: Provide support matrix and doc for VGPU https://review.openstack.org/540808 | 14:25 |
| *** chyka has quit IRC | 14:28 | |
| jianghuaw | bauzas, ^ | 14:28 |
| bauzas | jianghuaw: cool, will review it as soon as I can | 14:29 |
| jianghuaw | bauzas, thanks. | 14:29 |
| openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900 | 14:34 |
| *** lucasagomes is now known as lucas-hungry | 14:34 | |
| *** david-lyle has quit IRC | 14:34 | |
| *** AlexeyAbashkin has joined #openstack-nova | 14:35 | |
| *** dtantsur is now known as dtantsur|bbl | 14:37 | |
| *** Eran_Kuris has joined #openstack-nova | 14:37 | |
| *** links has quit IRC | 14:38 | |
| *** rodolof has quit IRC | 14:39 | |
| *** artom has quit IRC | 14:45 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Add 'numa-aware-vswitches' spec https://review.openstack.org/541290 | 14:45 |
| stephenfin | sean-k-mooney: It's very rough, but fancy taking a look at ^ at some point? | 14:45 |
| stephenfin | efried: You might be able to weight in from the generic device management aspect too ^ | 14:46 |
| mriedem | bauzas: depends on if it's recreatable in master | 14:46 |
| efried | ack | 14:46 |
| bauzas | mriedem: just told that actually | 14:46 |
| stephenfin | Try as I may, I can't think of a reason to need that, but I'm clearly missing something as both you and sean-k-mooney suggested it :) | 14:46 |
| bauzas | hence Incomplete | 14:46 |
| * stephenfin disappears for an hour | 14:46 | |
| *** alexchadin has quit IRC | 14:48 | |
| sean-k-mooney | stephenfin: reason to need ? | 14:48 |
| sean-k-mooney | stephenfin: also first taught on https://review.openstack.org/541290 is it should proably be an neutron spec but i have only got to line 10 so ill keep reading | 14:50 |
| *** mdnadeem has quit IRC | 14:52 | |
| *** lpetrut has quit IRC | 14:53 | |
| *** elod has joined #openstack-nova | 14:54 | |
| sean-k-mooney | stephenfin: of were you refering to a need for numa-aware-vswitches when you said you can think why you need that? | 14:55 |
| *** jmlowe has joined #openstack-nova | 14:57 | |
| *** acormier has joined #openstack-nova | 14:59 | |
| mriedem | bauzas: can you review https://review.openstack.org/#/c/539739/ please? it's an rc1 blocker due to a regression in the native encrypted luks stuff, blocking some cinder 3rd party CIs | 15:05 |
| bauzas | mriedem: roger. | 15:05 |
| bauzas | LUKS, I am your father. | 15:05 |
| *** 7IZAAAWPC has joined #openstack-nova | 15:07 | |
| *** 17SAAOQME has joined #openstack-nova | 15:07 | |
| *** cdent has quit IRC | 15:09 | |
| edleafe | bauzas: <groan!> | 15:11 |
| *** mnaser has quit IRC | 15:11 | |
| *** mnaser has joined #openstack-nova | 15:11 | |
| *** rodolof has joined #openstack-nova | 15:11 | |
| *** 7IZAAAWPC has quit IRC | 15:12 | |
| *** 17SAAOQME has quit IRC | 15:12 | |
| bauzas | that's possibly one of the top said jokes, I know | 15:12 |
| *** amodi has joined #openstack-nova | 15:13 | |
| *** abhishekk is now known as abhishekk|meetin | 15:13 | |
| *** dtantsur|bbl is now known as dtantsur | 15:14 | |
| *** mlavalle has joined #openstack-nova | 15:15 | |
| openstackgerrit | Lajos Katona proposed openstack/nova master: WIP: ServerMovingTests with custom resources https://review.openstack.org/497399 | 15:16 |
| *** awaugama has joined #openstack-nova | 15:16 | |
| *** artom has joined #openstack-nova | 15:24 | |
| *** lucas-hungry is now known as lucasagomes | 15:30 | |
| *** Eran_Kuris has quit IRC | 15:30 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Make the InstanceMapping marker UUID-like https://review.openstack.org/539323 | 15:31 |
| *** burt has joined #openstack-nova | 15:32 | |
| *** lpetrut has joined #openstack-nova | 15:34 | |
| *** sree has joined #openstack-nova | 15:36 | |
| *** andrewbogott has quit IRC | 15:36 | |
| *** andrewbogott has joined #openstack-nova | 15:37 | |
| *** artom has quit IRC | 15:38 | |
| *** med_ has quit IRC | 15:38 | |
| *** yamamoto has quit IRC | 15:39 | |
| *** sree has quit IRC | 15:41 | |
| stephenfin | sean-k-mooney: Yup, keep reading. It seems like a nova thing in that current design :) | 15:41 |
| stephenfin | though you may have left comments in the interim | 15:41 |
| * stephenfin goes to check | 15:41 | |
| *** yamamoto has joined #openstack-nova | 15:42 | |
| *** karlamrhein has quit IRC | 15:42 | |
| *** karlamrhein has joined #openstack-nova | 15:42 | |
| *** andrewbogott has quit IRC | 15:44 | |
| *** andrewbogott has joined #openstack-nova | 15:44 | |
| *** kholkina has quit IRC | 15:48 | |
| *** yamahata has joined #openstack-nova | 15:48 | |
| *** dtantsur is now known as dtantsur|bbl | 15:49 | |
| *** david-lyle has joined #openstack-nova | 15:51 | |
| *** pcaruana has quit IRC | 15:51 | |
| *** samueldmq has quit IRC | 15:52 | |
| sean-k-mooney | stephenfin: well im commenting back. the current design will not work unfrotunetly for example have you taught about how you handel bonds. | 15:52 |
| *** samueldmq has joined #openstack-nova | 15:52 | |
| *** tbachman has quit IRC | 15:52 | |
| * stephenfin meant to note that bonding was out of scope for this | 15:53 | |
| *** yamamoto has quit IRC | 15:53 | |
| *** abalutoiu_ has joined #openstack-nova | 15:55 | |
| *** claudiub|2 has joined #openstack-nova | 15:55 | |
| *** claudiub|2 has quit IRC | 15:55 | |
| *** abalutoiu_ has quit IRC | 15:55 | |
| *** abalutoiu_ has joined #openstack-nova | 15:56 | |
| *** claudiub|2 has joined #openstack-nova | 15:56 | |
| *** jaypipes has quit IRC | 15:56 | |
| *** salv-orlando has joined #openstack-nova | 15:57 | |
| *** ragiman has quit IRC | 15:58 | |
| sean-k-mooney | stephenfin: sorry in advance https://review.openstack.org/#/c/541290/1 comments in line | 15:58 |
| *** claudiub has quit IRC | 15:58 | |
| stephenfin | sean-k-mooney: :) | 15:58 |
| *** abalutoiu__ has quit IRC | 15:58 | |
| sean-k-mooney | stephenfin: by the way where did the 50% performanc number come from? the numa effect on ovs-dpdk should be in the singel figure if you turn on the numa support in dpdk | 15:59 |
| *** lajoskatona has left #openstack-nova | 15:59 | |
| stephenfin | sean-k-mooney: I was wondering if anyone was going to call me on that. It's been quoted numerous times to me by fbaudin, but I think it's based on internal analysis | 16:00 |
| stephenfin | I can root out the details and add them in some form to the spec | 16:00 |
| stephenfin | ...which clearly needs more work :) | 16:00 |
| sean-k-mooney | stephenfin: ya but i have mention to him in the past that we have messured much lower numbers in the past if you turn on the numa support in dpdk/ovs which is off by default | 16:01 |
| *** r-daneel has joined #openstack-nova | 16:01 | |
| *** salv-orlando has quit IRC | 16:03 | |
| *** salv-orlando has joined #openstack-nova | 16:03 | |
| stephenfin | sean-k-mooney: Thanks, I wasn't aware of that. There's some great info in there | 16:03 |
| stephenfin | sean-k-mooney: You aren't, by any chance, going to be in Dublin, are you? | 16:03 |
| sean-k-mooney | i proably will be. im not sure if i will be there on behalf of intel though as my travel has not been approved so i might not be the there the whole week if i go up myself | 16:04 |
| *** Sukhdev_ has quit IRC | 16:04 | |
| *** icey has quit IRC | 16:05 | |
| *** icey has joined #openstack-nova | 16:06 | |
| *** zioproto has quit IRC | 16:06 | |
| *** jaosorior has quit IRC | 16:06 | |
| *** zioproto has joined #openstack-nova | 16:06 | |
| *** gjayavelu has joined #openstack-nova | 16:07 | |
| *** cdent has joined #openstack-nova | 16:09 | |
| *** trinaths has joined #openstack-nova | 16:10 | |
| sean-k-mooney | stephenfin: just looking at the at the dpdk-devel packages for centos and fedora and they appear to have the the numa support turned on. and the ovs packages depend on libnuma so they "should" have numa support but if the testing was not done with at least 1 pmd per numa node then ovs will not be able to compensate internally if the vm and nic are not on the same numa node | 16:14 |
| *** redondo-mk has quit IRC | 16:15 | |
| stephenfin | sean-k-mooney: Could you stick that in the review too? This is all good data to use while working on this | 16:15 |
| *** redondo-mk has joined #openstack-nova | 16:15 | |
| stephenfin | sean-k-mooney: Planning on setting up an AIO deployment this evening to start hacking on this and see what I can get working | 16:15 |
| efried | sean-k-mooney: Are you from Dublin originally? | 16:16 |
| *** abhishekk|meetin has quit IRC | 16:16 | |
| sean-k-mooney | stephenfin: yep will do. if you use networking-ovs-dpdk/devstack just be aware we dont acally compile in the ovs support you will have to add it in but patches are welcome. | 16:16 |
| sahid | mriedem can you ack this https://review.openstack.org/#/c/515008/, it's a bugfix | 16:16 |
| *** andreas_s has quit IRC | 16:17 | |
| sean-k-mooney | efried: no i lived there for a year im on the other cost in shannon but its only about a 3 hour drive or so | 16:17 |
| *** andreas_s has joined #openstack-nova | 16:17 | |
| efried | I thought pronouncing 'th' as 't' was a Dublin thing. | 16:17 |
| stephenfin | sean-k-mooney: You mean you don't compile in DPDK support...right? | 16:18 |
| stephenfin | If not, what about compiling in? | 16:18 |
| stephenfin | efried: For example? | 16:18 |
| efried | stephenfin: "taught" for "thought" | 16:18 |
| *** yamamoto has joined #openstack-nova | 16:18 | |
| sean-k-mooney | efried: they pronounce 'th' more like a 'd' but 'th' as a 't' is more of a tipperary thing that i generally avoid doing in speach | 16:19 |
| *** trinaths has quit IRC | 16:19 | |
| *** yamamoto has quit IRC | 16:19 | |
| stephenfin | Ah, no. My father, who's also from the west coast, does that too | 16:20 |
| stephenfin | One, two, tree... | 16:20 |
| *** yamamoto has joined #openstack-nova | 16:20 | |
| sean-k-mooney | ya losing the th is a munster thing mainly | 16:21 |
| sean-k-mooney | efried: also i spell phonetically not correctly so the acent i write with is not the one i speak with. at least not entirely | 16:22 |
| stephenfin | Pronouncing 's' as 'sh' is also a thing. Fun | 16:22 |
| *** damien_r1 has joined #openstack-nova | 16:24 | |
| *** damien_r has quit IRC | 16:25 | |
| *** yamamoto has quit IRC | 16:25 | |
| sean-k-mooney | stephenfin: speaking of fun did you see this yet https://bugs.launchpad.net/nova/+bug/1747496 all those mtu patch last year and this is still a thing ... | 16:25 |
| openstack | Launchpad bug 1747496 in OpenStack Compute (nova) "MTUs are not set for VIFs if using kernel ovs + hybrid plug = false" [Undecided,New] - Assigned to sean mooney (sean-k-mooney) | 16:25 |
| *** zhaochao has quit IRC | 16:26 | |
| stephenfin | Oh, I saw the ping for that last night and forgot | 16:26 |
| stephenfin | Fun indeed | 16:26 |
| *** andreas_s has quit IRC | 16:26 | |
| sean-k-mooney | isint nice when libvirt tries to be helpful and plug vif for us to not over burden os-vif with all the work | 16:27 |
| openstackgerrit | James E. Blair proposed openstack/python-novaclient stable/pike: Zuul: Remove project name https://review.openstack.org/541370 | 16:27 |
| *** gjayavelu has quit IRC | 16:28 | |
| sean-k-mooney | stephenfin: i found it while i was deploying or new internal devcloud with jumbo frames and no security group dirver | 16:28 |
| *** trinaths has joined #openstack-nova | 16:28 | |
| *** gjayavelu has joined #openstack-nova | 16:28 | |
| stephenfin | sean-k-mooney: Oh, that crap | 16:29 |
| *** artom has joined #openstack-nova | 16:29 | |
| stephenfin | I want to remove that _entire_ code path | 16:29 |
| stephenfin | In fact, I think I started on doing it about a year ago and got distracted/confused | 16:29 |
| sean-k-mooney | stephenfin: ya so do i but as a backportable solution first im going to set the mtu in the xml for libvirt >3.2 and then stop libvirt form helping and let os-vif do it as a followup patch | 16:30 |
| stephenfin | sean-k-mooney: Yup, I can get onboard with that approach | 16:30 |
| *** yamamoto has joined #openstack-nova | 16:30 | |
| *** tbachman has joined #openstack-nova | 16:32 | |
| *** gjayavelu has quit IRC | 16:32 | |
| *** gmann has quit IRC | 16:33 | |
| *** gmann has joined #openstack-nova | 16:33 | |
| *** itlinux has joined #openstack-nova | 16:34 | |
| *** damien_r has joined #openstack-nova | 16:38 | |
| *** sahid has quit IRC | 16:39 | |
| cfriesen | sean-k-mooney: just curious, why is numa support in dpdk off by default? seems like that'd be a good thing. | 16:39 |
| *** damien_r1 has quit IRC | 16:39 | |
| mriedem | melwitt: comments in https://review.openstack.org/#/c/515008/ since sahid asked for review and left | 16:41 |
| *** ccamacho has quit IRC | 16:41 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | 16:42 |
| *** priteau has quit IRC | 16:42 | |
| *** hamzy has quit IRC | 16:43 | |
| *** brad[] has quit IRC | 16:45 | |
| *** damien_r has quit IRC | 16:45 | |
| stephenfin | cfriesen: I'm not sean-k-mooney, but I'm guessing it's the need for an extra dependency? | 16:46 |
| *** chyka has joined #openstack-nova | 16:49 | |
| *** sree has joined #openstack-nova | 16:49 | |
| bauzas | mriedem: I have a noob question, how can you know that vol.volume_id is set in https://review.openstack.org/#/c/539739/4/nova/virt/libvirt/driver.py@7430 . | 16:49 |
| bauzas | ? | 16:49 |
| mriedem | bauzas: because when we attach a volume to an instance, we save the volume_id on the bdm record, | 16:51 |
| mriedem | see nova.virt.block_device.DriverVolumeBlockDevice.attach | 16:51 |
| mriedem | DriverVolumeBlockDevice wraps an proxies a BlockDeviceMapping object, | 16:52 |
| mriedem | and the @update_db decorator on attach() updates the wrapped bdm object | 16:52 |
| *** jmlowe has quit IRC | 16:53 | |
| mriedem | i should say that is the case for bfv with source type blank/image/snapshot where nova creates the volume and then attaches it, | 16:53 |
| mriedem | for normal volume attach to an existing instance, the bdm is created in compute here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5220 | 16:53 |
| bauzas | ok, looking at the code | 16:53 |
| mriedem | with the volume_id | 16:53 |
| *** jmlowe has joined #openstack-nova | 16:53 | |
| *** tssurya has quit IRC | 16:54 | |
| *** salv-orlando has quit IRC | 16:54 | |
| *** salv-orlando has joined #openstack-nova | 16:55 | |
| hrw | zuul forgot about me ;( 7h since patch upload and no results | 16:56 |
| *** chyka_ has joined #openstack-nova | 16:56 | |
| bauzas | hrw: you probably haven't see the gate status | 16:56 |
| bauzas | they had problems in the morning with very slow processing | 16:56 |
| hrw | bauzas: probably | 16:56 |
| kashyap | dansmith: Okay, after talking to libvirt and QEMU migration maintainers, you're right in pointing out here: https://review.openstack.org/#/c/497457/18/nova/virt/libvirt/driver.py | 16:57 |
| *** chyka has quit IRC | 16:57 | |
| kashyap | That... | 16:57 |
| hrw | ok I see | 16:57 |
| kashyap | dansmith: It is an absolutely horrible hack | 16:57 |
| kashyap | dansmith: ...to slow down migration via by limiting the speed to 1 MiB/s | 16:58 |
| dansmith | kashyap: Well, I know it's a hack, but wasn't sure if there was another option :) | 16:58 |
| hrw | bauzas: will see. in worst case 'recheck' in the morning tomorrow | 16:58 |
| bauzas | hrw: you know the status page or not ? | 16:58 |
| kashyap | dansmith: Discussing the design aspects w/ them. I have a half-comment in draft, will complete it in a bit :-) | 16:59 |
| dansmith | kashyap: excellent, thanks! | 16:59 |
| kashyap | dansmith: Just wanted to confirm your instinct :P | 16:59 |
| hrw | bauzas: https://wiki.openstack.org/wiki/Infrastructure_Status one? | 16:59 |
| dansmith | kashyap: awesome :P | 16:59 |
| *** salv-orlando has quit IRC | 16:59 | |
| * stephenfin awaits kashyap's findings | 16:59 | |
| stephenfin | (I looked at the underlying bug and also assumed there wasn't an alternative) | 17:00 |
| bauzas | hrw: http://zuul.openstack.org/ and http://status.openstack.org/openstack-health/#/ | 17:00 |
| hrw | thx | 17:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Bump compute RPC API to version 5.0 https://review.openstack.org/541005 | 17:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Compute RPC client bump to 5.0 https://review.openstack.org/541035 | 17:01 |
| kashyap | dansmith: At least, QEMU migratoin upstream said: "If Nova can define a need, we can add a tunable that libvirt can use" | 17:01 |
| * kashyap goes back to writing | 17:02 | |
| dansmith | ++ | 17:02 |
| *** gongysh has quit IRC | 17:02 | |
| *** gyee has joined #openstack-nova | 17:02 | |
| stephenfin | dansmith: <3 I've been waiting for RPC version bumps for aaaaages | 17:03 |
| stephenfin | Mostly thanks to lots of comments from ndipanov saying we can cruft in virt/hardware when that happens | 17:03 |
| dansmith | stephenfin: don't review it yet.. still working out the bugs. I think this iteration is close, but need tempest tests | 17:03 |
| dansmith | stephenfin: well, then you need to look at those notes and make sure I've got the bits that need to be removed | 17:04 |
| stephenfin | I can do that | 17:04 |
| dansmith | stephenfin: I've deprecated the things I know about, but if there are others, now's the time... | 17:04 |
| *** salv-orlando has joined #openstack-nova | 17:04 | |
| *** slaweq has quit IRC | 17:05 | |
| * hrw off | 17:05 | |
| *** slaweq has joined #openstack-nova | 17:05 | |
| * bauzas enjoys reading BDM code | 17:06 | |
| gibi | bauzas: could you look at https://review.openstack.org/#/c/525242 again? (and the one below) | 17:06 |
| *** gongysh has joined #openstack-nova | 17:06 | |
| gibi | bauzas: mriedem is already +2 on both | 17:06 |
| bauzas | fortunately I have https://docs.openstack.org/nova/latest/user/block-device-mapping.html for help | 17:06 |
| bauzas | gibi: only if that's super quick | 17:07 |
| gibi | bauzas: you already reviewed it :) | 17:07 |
| gibi | bauzas: so I hope it is a quick one | 17:07 |
| bauzas | gibi: I'm reviewing mriedem's change https://review.openstack.org/#/c/539739/4 that requires some BDM knowledge | 17:07 |
| bauzas | ^ RC1 blocker, your call, core | 17:07 |
| mriedem | did you lay down a guantlet just now? | 17:08 |
| mriedem | *gauntlet even | 17:08 |
| gibi | bauzas: sure, the LUKS patch has priority | 17:08 |
| *** gongysh has quit IRC | 17:08 | |
| bauzas | hah, no | 17:09 |
| bauzas | for the first time of my poor life, I have the opportunity to understand the BDM mess^H^H implementation | 17:09 |
| *** slaweq has quit IRC | 17:10 | |
| dansmith | only 341 things in check.. I look forward to my CI results later in the week | 17:10 |
| bauzas | dansmith: yeah like I said to hrw, that GMT morning was awful | 17:12 |
| *** jpena is now known as jpena|off | 17:18 | |
| kashyap | stephenfin: Quickly, for your point on that live migration issue -- yeah, for the immediate term, we may have to go with the ugly hack of slowing the n/w bandwidth. Until we figure out a way to stop migration much earlier in Nova. | 17:21 |
| *** r-daneel has quit IRC | 17:22 | |
| bauzas | mriedem: nit on https://review.openstack.org/#/c/539739/4/nova/tests/unit/virt/libvirt/test_driver.py@10523 that prevents me to +2 except if you add it in a follow-up (given the gate delays) | 17:22 |
| *** slaweq has joined #openstack-nova | 17:22 | |
| *** r-daneel has joined #openstack-nova | 17:22 | |
| bauzas | mriedem: do you agree with me on the fact that serial and volume_id should have the same values ? | 17:23 |
| *** AlexeyAbashkin has quit IRC | 17:25 | |
| *** AlexeyAbashkin has joined #openstack-nova | 17:25 | |
| *** udesale has quit IRC | 17:25 | |
| *** slaweq has quit IRC | 17:26 | |
| *** AlexeyAbashkin has quit IRC | 17:29 | |
| dansmith | hmm | 17:31 |
| dansmith | looks like we just had a zuul reset? | 17:31 |
| mriedem | bauzas: replied | 17:31 |
| mriedem | i specifically wanted those to be different | 17:32 |
| mriedem | so that we take volume_id over serial if both are in connection_info | 17:32 |
| bauzas | okay, I don't want to overthink on that | 17:33 |
| bauzas | +2ing | 17:33 |
| *** sree has quit IRC | 17:33 | |
| mriedem | the DriverVolumeBlockDevice.attach code only sets the serial to the volume_id iff 'serial' isn't already in the connection_info, | 17:33 |
| mriedem | so once again, some cinder volume driver could theoretically be returning a 'serial' value | 17:34 |
| *** sree has joined #openstack-nova | 17:34 | |
| bauzas | yeah I saw the conditional | 17:34 |
| mriedem | so if volume_id, use it, else hope that serial is correct | 17:34 |
| bauzas | I love those conditions actually | 17:34 |
| bauzas | tbh, the best would be to tell the cinder storage drivers "please, don't give us information, we'll find out by ourselves, it's better ;) ' | 17:35 |
| mriedem | a couple of years ago i talked about adding a versioned object for the connection_info dict to cinder, | 17:35 |
| bauzas | since we set the right value in serial if that's not already in there | 17:35 |
| mriedem | to try and normalize a lot of that stuff | 17:35 |
| mriedem | but, i didn't pursue it | 17:36 |
| bauzas | your fault, then | 17:36 |
| bauzas | boooo | 17:36 |
| mriedem | sort of like the MigrateData thing that dansmith did for live migration | 17:36 |
| bauzas | yup | 17:36 |
| mriedem | yes it's my fault, sadly enough, | 17:36 |
| mriedem | it would have also meant converting like 80 volume drivers over to that new model... | 17:36 |
| bauzas | :) | 17:37 |
| ildikov | yeah, this comes up every half year and we note that harmonize the drivers is just not likely to happen | 17:38 |
| *** sree has quit IRC | 17:38 | |
| ildikov | or you end up with a free-form part where most of the info sits anyway... | 17:39 |
| *** rmart04 has quit IRC | 17:39 | |
| bauzas | mriedem: I haven't seen https://etherpad.openstack.org/p/nova-aggregate-filter-allocation-ratio-snafu somehow planned for the PTG in the PTG etherpad | 17:43 |
| *** mtreinish has quit IRC | 17:43 | |
| bauzas | mriedem: ack if I'm adding it? | 17:44 |
| mriedem | jaypipes had a thing in the ptg etherpad for this issue, | 17:45 |
| mriedem | i'll link it in | 17:45 |
| mriedem | starts at L34 in https://etherpad.openstack.org/p/nova-ptg-rocky | 17:46 |
| *** traskat has joined #openstack-nova | 17:48 | |
| bauzas | cool | 17:48 |
| mriedem | it's enchilada time! | 17:50 |
| *** mtreinish has joined #openstack-nova | 17:50 | |
| *** jose-phillips has left #openstack-nova | 17:51 | |
| dansmith | mriedem: remind me what the legacy-tempest job is? | 17:53 |
| dansmith | it's failing my service version patches for seemingly related reasons, but the regular job is fine | 17:53 |
| mriedem | hmm, legacy-tempest shouldn't be running on our stuff | 17:54 |
| mriedem | i don't think | 17:54 |
| mriedem | that was replaced by tempest-full | 17:55 |
| dansmith | legacy-tempest-dsvm-py35FAILURE in 1h 14m 39s | 17:55 |
| *** gjayavelu has joined #openstack-nova | 17:55 | |
| dansmith | on one of my patches | 17:55 |
| dansmith | https://review.openstack.org/#/c/539590/ | 17:55 |
| mriedem | oh that's different | 17:55 |
| mriedem | it's just tempest with devstack on py35 | 17:55 |
| * tbachman wonders if he’s the only one who can’t access zuul status atm | 17:56 | |
| dansmith | tbachman: no | 17:56 |
| dansmith | mriedem: hmm, wonder why it failed | 17:56 |
| tbachman | dansmith: thx | 17:56 |
| mriedem | dansmith: http://logs.openstack.org/90/539590/2/check/legacy-tempest-dsvm-py35/79137ee/job-output.txt.gz#_2018-01-31_19_33_29_779861 | 17:56 |
| mriedem | likely something in the reserve_block_device_name rpc call? | 17:56 |
| dansmith | mriedem: right, hence my "seemingly related reason" | 17:56 |
| dansmith | no, it's because we check for service version before allowing tags | 17:57 |
| dansmith | which is what that patch is doing | 17:57 |
| *** NostawRm has quit IRC | 17:57 | |
| dansmith | s/doing/changing/ | 17:57 |
| dansmith | s/changing/touching/ | 17:57 |
| *** NostawRm has joined #openstack-nova | 17:57 | |
| bauzas | mriedem: mgagne: just added a thought on a possible workaround for https://review.openstack.org/#/c/541018/ | 17:57 |
| bauzas | mriedem: mgagne: tl;dr just set allocation ratios on every compute to be 99999.0 | 17:58 |
| sean-k-mooney | cfriesen: stephenfin yes numa support is off by default in dpdk so that libnuma can remain an optional dependecy not a required depency of dpdk | 17:58 |
| mriedem | dansmith: maybe "if result is" is different in py35 | 17:58 |
| mriedem | i don't see much "foo is bar" | 17:58 |
| bauzas | mriedem: mgagne: that will allow Placement to accept all the hosts, and just keep the filtering logic in the appropriate AggregateRAMFilter and so on | 17:58 |
| *** yamamoto has quit IRC | 17:59 | |
| *** yamamoto has joined #openstack-nova | 18:00 | |
| dansmith | mriedem: um, which thing are you talking about? | 18:00 |
| kashyap | I'm Neutron-challenged, can anyone tell at _which_ of these points is Nova waiting (by pausing) on Neutron to let it complete things? http://git.openstack.org/cgit/openstack/neutron/commit/?h=master&id=b7c303e ("Notify nova with network-vif-plugged in case of live migration") | 18:00 |
| dansmith | the sentinel check has to be "is" | 18:00 |
| *** tesseract has quit IRC | 18:00 | |
| kashyap | "during live migration on pre migration" is terribly unclear | 18:01 |
| * kashyap goes to look at the code | 18:01 | |
| *** sambetts is now known as sambetts|afk | 18:01 | |
| *** derekh has quit IRC | 18:02 | |
| dansmith | kashyap: nova waits when it uses the virtapi wait_for_events context manager | 18:02 |
| dansmith | it waits to exit the context until timeout or the event arrives | 18:02 |
| kashyap | I see. | 18:03 |
| kashyap | Is this what the Neutron events telling Nova: | 18:03 |
| kashyap | "Hey, I've done with doing all the stuff I need to do, so I'm signalling that by sending out this event (from the above Neutron commit). Over to you now." | 18:04 |
| kashyap | I.e. the destination Neutron has completed virtual interface plugging. | 18:04 |
| dansmith | yes | 18:04 |
| dansmith | I mean, that's what vif-plugged means | 18:04 |
| kashyap | Noted. | 18:05 |
| stvnoyes | mriedem: matt, I added a change to enable multiattach on the zfssa array, but when tempest runs during the zfssa ci, it's not running with multiattach enabled (in tempest.conf). Where would I make a change to have this enabled when running the Oracle ZFSSA CI? thanks... | 18:05 |
| sean-k-mooney | dansmith: actully vif-plugged has different menanings | 18:05 |
| sean-k-mooney | vif-plugged in the nova-compute agent log means os-vif finisshed plugging the interface | 18:06 |
| sean-k-mooney | vif-plugged event form neutron means neutron finished wireing it up | 18:06 |
| dansmith | sean-k-mooney: we're talking about when neutron does a POST of the event to nova's external event interface | 18:06 |
| *** salv-orlando has quit IRC | 18:07 | |
| dansmith | and it better (still) mean that the wiring is done, else we have a problem :) | 18:07 |
| mriedem | stvnoyes: do you have these set in devstack? https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-multiattach/run.yaml#L31 | 18:07 |
| *** salv-orlando has joined #openstack-nova | 18:07 | |
| sean-k-mooney | dansmith: ah then yes that is sent when neutron finishes wiring up the port that was plugged by nova/os-vif/libvirt into the network backend so your correct in what you said | 18:07 |
| mriedem | stvnoyes: also, is tempest running against master nova so the 2.60 microversion is available? | 18:08 |
| dansmith | sean-k-mooney: thanks :) | 18:08 |
| kashyap | sean-k-mooney: Hey, I'm about to write some questions on this change: https://review.openstack.org/#/c/497457/18 | 18:09 |
| kashyap | sean-k-mooney: Post some design discussion w/ QEMU & libvirt folks | 18:09 |
| kashyap | Would appreciate if you also look at it | 18:09 |
| sean-k-mooney | kashyap: am i am behind in some internal paper work but ill try and take a look tomorow | 18:10 |
| kashyap | sean-k-mooney: Sure, tomm is fine; I'm already late for dinner by an hour | 18:10 |
| kashyap | Just want to 'commit this to paper' before my "cache gets flushed" | 18:10 |
| kashyap | As I've spent the last 2 hours in this rabbit hole :P | 18:11 |
| *** amodi has quit IRC | 18:11 | |
| mriedem | efried: on that service user / CONF.glance.api_servers bug https://bugs.launchpad.net/nova/+bug/1747511 - it's definitely related to using a service user; i removed that code and now the nova-next job passes | 18:11 |
| openstack | Launchpad bug 1747511 in OpenStack Compute (nova) "legacy-tempest-dsvm-neutron-nova-next-full fails with "Can not find requested image" since 1/31" [Undecided,New] | 18:11 |
| stvnoyes | mriedem: i didn't make any changes to the CI devstack (although I can run locally ok). I believe the CI is against nova master, but will check. The zfssa CI test is running for this review - https://review.openstack.org/#/c/541368/ | 18:12 |
| *** salv-orlando has quit IRC | 18:12 | |
| mriedem | stvnoyes: so this job result right? https://openstackci.oracle.com/zfssa_ci/refs-changes-68-541368-1-iscsi/ | 18:12 |
| *** slaweq has joined #openstack-nova | 18:13 | |
| stvnoyes | mriedem: yes. I'm seeing tempest.api.compute.volumes.test_attach_volume.AttachVolumeMultiAttachTest) ... SKIPPED: Volume multi-attach is not available. | 18:13 |
| *** hamzy has joined #openstack-nova | 18:13 | |
| mriedem | stvnoyes: that's b/c it's not set in tempest.conf | 18:13 |
| mriedem | 2018-02-06 09:16:16.276 12703 DEBUG tempest [-] compute-feature-enabled.volume_multiattach = False log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2898 | 18:13 |
| mriedem | https://openstackci.oracle.com/zfssa_ci/refs-changes-68-541368-1-iscsi/tempest.log | 18:13 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make service all-cells min version helper use scatter-gather https://review.openstack.org/539590 | 18:13 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/539617 | 18:13 |
| openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | 18:13 |
| mriedem | stvnoyes: where is the code that controls the local.conf for the devstack setup in this CI? | 18:14 |
| mriedem | is it internal to oracle? or on github? | 18:14 |
| *** hamzy_ has joined #openstack-nova | 18:15 | |
| *** odyssey4me has joined #openstack-nova | 18:15 | |
| stvnoyes | mriedem: good question. idk. I haven't played with CI stuff so all this is pretty new to me. I will see if there's someone around on the zfssa team who would know. | 18:15 |
| mriedem | stvnoyes: https://wiki.openstack.org/wiki/ThirdPartySystems/Oracle_ZFSSA_CI | 18:16 |
| odyssey4me | howdy all - we're having some trouble in the OSA deployment test for upgrades from newton (without cells v2) to ocata (with cells v2), it looks like there's some weird state which has been hit where the compute registration isn't working right any more. I'm seeing this: https://pastebin.com/C9ji6vdS | 18:16 |
| odyssey4me | Our last working test was the last patch we did which was in early December. | 18:17 |
| *** slaweq has quit IRC | 18:17 | |
| odyssey4me | Any idea what may have changed to cause the previously working implementation to no longer work? | 18:17 |
| *** hamzy has quit IRC | 18:17 | |
| *** tssurya has joined #openstack-nova | 18:18 | |
| mriedem | odyssey4me: i don't see anything related to that in https://github.com/openstack/nova/commits/stable/ocata going back to november | 18:19 |
| mriedem | are you sure it wasn't something that changed on the OSA side? | 18:19 |
| odyssey4me | mriedem yep, very sure - see https://review.openstack.org/#/q/branch:stable/ocata+project:openstack/openstack-ansible-os_nova | 18:19 |
| *** david-lyle has quit IRC | 18:20 | |
| mriedem | https://github.com/openstack/nova/commits/stable/ocata/nova/cmd/manage.py | 18:21 |
| *** brad[] has joined #openstack-nova | 18:21 | |
| mriedem | oldest thing in stable there is from october | 18:21 |
| odyssey4me | yeah, I'm wondering if something has perhaps changed in the database tables either in newton (the source), or in ocata (the destination) | 18:22 |
| mriedem | have you verified if the instances you're verifying actually exist? | 18:22 |
| *** dtantsur|bbl is now known as dtantsur | 18:22 | |
| mriedem | i.e. not just not mapped, but don't actually exist | 18:22 |
| odyssey4me | well, I got stuck there - given that nova-manage doesn't appear to have a way to list the instances :/ | 18:23 |
| odyssey4me | any tips for getting a listing out? | 18:23 |
| odyssey4me | the closest I could find is "nova-manage cell_v2 discover_hosts --verbose" | 18:23 |
| *** amodi has joined #openstack-nova | 18:24 | |
| mriedem | that doesn't dump the instances | 18:24 |
| mriedem | the api isn't up? | 18:25 |
| odyssey4me | it gives me a set of uuid's which look rather like they belong to instances: https://pastebin.com/mSwpZybQ | 18:25 |
| mriedem | Getting compute nodes from cell 'cell1': 16443e92-e33b-487e-8069-7c80e5bdbc33 | 18:25 |
| mriedem | that's a cell mapping uuid | 18:25 |
| *** lpetrut has quit IRC | 18:25 | |
| mriedem | Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637 | 18:25 |
| mriedem | that's a compute node uuid | 18:25 |
| mriedem | instances aren't the only things that have uuids anymore | 18:26 |
| odyssey4me | the API is up, so I can query things that way if it helps | 18:26 |
| mriedem | worth a shot if you have a local recreate, | 18:27 |
| mriedem | but my guess is the api won't find them either if nova-manage can't | 18:27 |
| odyssey4me | my apologies for dumb questions - it's been a very long time since I actually worked with nova directly :) | 18:27 |
| mriedem | if they don't have instance_mappings in nova_api i mean | 18:27 |
| mriedem | totally fine | 18:27 |
| mriedem | not dumb at all | 18:27 |
| mriedem | if you have a local recreate, you could check the nova_api.instance_mappings table directly | 18:28 |
| mriedem | or the cell db instances table | 18:28 |
| odyssey4me | sure, can do that | 18:28 |
| mriedem | looks like you have 2 cells, so you'd have to look in each | 18:28 |
| mriedem | so you've got 4 dbs (nova_api, nova_cell0, cell1 and then cell 9461149a-52a9-495d-8021-d2cda1645d28) | 18:29 |
| odyssey4me | right, so I have several DB's here: nova, nova_api, nova_cell0; nova_placement | 18:29 |
| mriedem | nova is likely cell1 | 18:29 |
| mriedem | yes? | 18:29 |
| mriedem | nova_placement isn't a thing...not sure what that is | 18:29 |
| *** ralonsoh has quit IRC | 18:29 | |
| odyssey4me | ok, that's empty - I'll look into why that's there later | 18:31 |
| mriedem | i probably know why | 18:31 |
| odyssey4me | it might be some leftovers from previous work before things matured | 18:32 |
| *** lpetrut has joined #openstack-nova | 18:33 | |
| mriedem | odyssey4me: yeah https://review.openstack.org/#/q/I31293ac4689630e4113588ab2c6373cf572b8f38 | 18:33 |
| odyssey4me | haha, ok - thanks for the reference :) | 18:34 |
| mriedem | odyssey4me: so looking at https://pastebin.com/mSwpZybQ there is something screwed up with the host mappings, | 18:34 |
| mriedem | it looks like this compute is in two cells | 18:34 |
| mriedem | Checking host mapping for compute host 'ocata-nova1': 6e6d71ab-0b18-416d-8a3a-ce713ac59637 | 18:34 |
| *** brad[] has quit IRC | 18:34 | |
| odyssey4me | if it'd make things simpler I can get your pub key on this host for you to poke around directly? | 18:35 |
| *** harlowja has joined #openstack-nova | 18:35 | |
| odyssey4me | the host is a temp instance, so nothing special on it | 18:35 |
| mriedem | i don't think our relationship has hit that level of maturity yet | 18:35 |
| odyssey4me | hahaha, fair enough | 18:35 |
| odyssey4me | otherwise, I'll need some guidance with db queries to get data out - I can gist the results as we go | 18:36 |
| mriedem | well, it appears you have 2 nova_api.host_mappings entries for host "ocata-nova1" | 18:36 |
| mriedem | which would be wrong | 18:36 |
| odyssey4me | yeah, that seemed weird to me too | 18:37 |
| cfriesen | is anyone aware of an issue where running "'wget http://169.254.169.254/latest/meta-data/instance-id" in the guest gives a result that is *not* the same as that instance's OS-EXT-SRV-ATTR:instance_name in "nova show"? | 18:37 |
| mriedem | tssurya: dansmith: melwitt: any idea why we don't have a unique constraint across the cell_id and host colums in the host_mappings table? | 18:37 |
| dansmith | mriedem: I thought there was some argument about that with overlapping hostnames (which won't work anyway) ? | 18:38 |
| mriedem | odyssey4me: is this CI supposed to have 2 cells? because multi-cell wasn't supported in ocata | 18:38 |
| odyssey4me | mriedem interestingly enough, I only see one entry in the DB for it | 18:38 |
| odyssey4me | nah, it only has cell0 and cell1 | 18:39 |
| mriedem | odyssey4me: do you have a CI run with logs posted? | 18:39 |
| *** stvnoyes has quit IRC | 18:39 | |
| mriedem | https://pastebin.com/mSwpZybQ is saying there are 3 cell mappings | 18:39 |
| mriedem | "Found 3 cell mappings." | 18:39 |
| mriedem | select uuid from nova_api.cell_mappings; | 18:39 |
| odyssey4me | mriedem unfortunately our log collection is broken for ocata, so all we have is console output which only shows brokenness when tempest runs, so that's not very useful | 18:40 |
| odyssey4me | I can fix up the CI to collect logs, but that'll take a few days to make its way through... | 18:40 |
| odyssey4me | hmm, that is odd - three cells showing | 18:42 |
| odyssey4me | it might be my bad, I re-ran 'nova-manage cell_v2 discover_hosts' and 'nova-manage cell_v2 map_instances --cell_uuid ...' a few times | 18:43 |
| *** gmann has quit IRC | 18:43 | |
| odyssey4me | also ran 'nova-manage cell_v2 simple_cell_setup' after the initial build | 18:43 |
| odyssey4me | that gave me 'Cell0 is already setup', so I don't think that broke anything | 18:44 |
| mriedem | discover_hosts is idempotent and doesn't create mappings | 18:45 |
| mriedem | cell mappings i mean | 18:45 |
| mriedem | map_instances should also be ok | 18:45 |
| mriedem | my guess is something go f'ed up when running simple_cell_setup a few times | 18:45 |
| odyssey4me | I only did it once :) | 18:46 |
| *** brad[] has joined #openstack-nova | 18:46 | |
| mriedem | was create_cell ever called? | 18:48 |
| odyssey4me | yep - this is the basic set of steps that would have executed before the discovery: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L16-L55 | 18:48 |
| odyssey4me | so in this case, the initial api_db sync would have been skipped as this was an existing DB | 18:49 |
| odyssey4me | then the cell0 map done: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L31 | 18:49 |
| odyssey4me | then the creation of cell1: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_setup.yml#L37 | 18:49 |
| odyssey4me | then the api_db sync, and db sync | 18:49 |
| mriedem | where does map_instances happen? | 18:50 |
| mriedem | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L38 | 18:50 |
| *** pooja-jadhav has joined #openstack-nova | 18:50 | |
| mriedem | found it | 18:50 |
| odyssey4me | we wait for a compute instance to be ready in the handler: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/handlers/main.yml#L67-L76 | 18:51 |
| odyssey4me | then yes, https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml gets run | 18:51 |
| *** niraj_singh has joined #openstack-nova | 18:52 | |
| odyssey4me | you'll find those tasks, and their stdout/stderr in this console log: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz | 18:53 |
| odyssey4me | it's a bit verbose, apologies in advance | 18:53 |
| *** nsingh has quit IRC | 18:55 | |
| *** neha_alhat has quit IRC | 18:55 | |
| kashyap | dansmith: When you get a moment, wrote this here: https://review.openstack.org/#/c/497457/18 | 18:56 |
| kashyap | Please correct / critique / answer as you see fit. | 18:56 |
| mriedem | odyssey4me: hmm, for https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L39 | 18:57 |
| mriedem | --cell_uuid {{ cell1_uuid['stdout'].split()[3] }}" | 18:57 |
| mriedem | are you sure that's hitting cell1 and not cell0? | 18:57 |
| mriedem | oh i guess because of https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/tasks/nova_db_post_setup.yml#L32 | 18:58 |
| mriedem | so nvm | 18:58 |
| odyssey4me | yep, exactly | 18:58 |
| odyssey4me | if someone names their cell "cell12" then it might not be right - but we're using safe defaults here | 18:58 |
| kashyap | dansmith: To finish up, for completeness' sake: The QEMU / libvirt 3.9.0 "pause-before-switchover" thing -- it won't be useful in Nova's case, as that will stop the guest CPUs, which will extend the guest down time.) | 18:58 |
| odyssey4me | although the spacing in that grep should be reasonable to protect the boundaries | 18:59 |
| * kashyap goes to feed himself | 18:59 | |
| *** amodi has quit IRC | 18:59 | |
| *** AlexeyAbashkin has joined #openstack-nova | 19:00 | |
| *** lucasagomes is now known as lucas-afk | 19:02 | |
| mriedem | odyssey4me: just to verify, this is running on the newton code before upgrading to ocata yes? | 19:03 |
| mriedem | because http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_10_32_079153 returns version 21 for the nova_api db which was the version in newton | 19:03 |
| odyssey4me | yep, the initial build from be from what was the head of stable/newton at the time the test ran | 19:04 |
| odyssey4me | I'll push a patch up to make it use the EOL branch now :) | 19:04 |
| odyssey4me | oh hang on, I think it mighth be sha pinned | 19:04 |
| mriedem | there was only one nova_api db change in newton before eol https://github.com/openstack/nova/blob/newton-eol/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/022_request_specs_spec_mediumtext.py | 19:04 |
| mriedem | and shouldn't have anything to do with what you're seeing | 19:04 |
| odyssey4me | heh, that makes it even weirder that this failed: https://github.com/openstack/openstack-ansible-tests/blob/stable/newton/test-vars.yml#L157 | 19:05 |
| dansmith | kashyap: yeah that's why I said it didn't seem like it would :) | 19:05 |
| mriedem | odyssey4me: so everything in that console output looks ok to me, | 19:06 |
| mriedem | where do instances actually get created in newton before the upgrade is started? | 19:07 |
| dansmith | kashyap: I was also really looking for you to comment on the potential race with the setting of the bandwidth limit in two places.. did you look at that at all? | 19:07 |
| mriedem | odyssey4me: i.e. how do you know that you've actually got instances from newton to map when map_instances runs? | 19:07 |
| kashyap | dansmith: No, not yet. After I finish dinner | 19:07 |
| * kashyap bbiab | 19:07 | |
| *** r-daneel_ has joined #openstack-nova | 19:08 | |
| *** david-lyle has joined #openstack-nova | 19:08 | |
| odyssey4me | mriedem by 'instances' do you mean the compute host? The terminology is confusing to me here... because in this CI run no instance (by this I mean a cloud instance, not a nova hypervisor) is created until *after* the upgrade when tempest executes. | 19:08 |
| *** yamamoto has quit IRC | 19:08 | |
| *** david-lyle has quit IRC | 19:08 | |
| *** david-lyle has joined #openstack-nova | 19:09 | |
| *** r-daneel has quit IRC | 19:09 | |
| *** r-daneel_ is now known as r-daneel | 19:09 | |
| *** amodi has joined #openstack-nova | 19:09 | |
| mriedem | instances == vms | 19:09 |
| mriedem | odyssey4me: ok then map_instances isn't going to map anything :) | 19:09 |
| mriedem | map_instances means create nova_api.instance_mappings records in the db | 19:10 |
| mriedem | based on VMs in the 'nova' db | 19:10 |
| mriedem | the 'nova' db is the cell1 db | 19:10 |
| mriedem | odyssey4me: so looking at https://pastebin.com/C9ji6vdS again, when/how did either of those instances get created? | 19:11 |
| mriedem | the ones passed to verify_instance | 19:11 |
| odyssey4me | ok, btw if we're doing something dumb here then please feel free to say so with any improvement suggestions :) | 19:11 |
| mriedem | odyssey4me: the ansible tasks all look fine | 19:11 |
| mriedem | the ansible is like a step for step copy of what's in the install guide here https://docs.openstack.org/nova/latest/user/cells.html | 19:11 |
| odyssey4me | ok, given that I did those commands after the test failed, those instances would have been created by tempest after the upgrade completed | 19:11 |
| odyssey4me | the one thing that may not have run here is the online migrations | 19:12 |
| dansmith | kashyap: okay thanks | 19:12 |
| mriedem | odyssey4me: do you see any errors in the nova-api logs? | 19:12 |
| mriedem | or nova-conductor? | 19:12 |
| mriedem | also, based on that git hash for nova, the CI isn't picking up any bug fixes since the newton GA | 19:13 |
| mriedem | which seems less than ideal | 19:13 |
| odyssey4me | nova-conductor: No host-to-cell mapping found for selected host ocata-nova1. Setup is incomplete. | 19:16 |
| odyssey4me | Failed to compute_task_build_instances: Host 'ocata-nova1' is not mapped to any cell | 19:16 |
| hrw | https://marcin.juszkiewicz.com.pl/2018/02/06/graphical-console-in-openstack-aarch64/ - please read ;) | 19:17 |
| *** Tom-Tom has quit IRC | 19:20 | |
| *** jafeha has quit IRC | 19:21 | |
| *** Tom-Tom has joined #openstack-nova | 19:22 | |
| tssurya | odyssey4me: can you check the value of the "mapped" column inside compute_nodes table of the nova db (cell1 db), ? if you have this host there, then there should be a record | 19:24 |
| odyssey4me | hmm, select mapped from compute_nodes; gives me an unknown column error | 19:26 |
| odyssey4me | yup, none of those are working | 19:26 |
| odyssey4me | none of the db's have that column | 19:27 |
| *** Tom-Tom has quit IRC | 19:27 | |
| *** salv-orlando has joined #openstack-nova | 19:28 | |
| tssurya | there should be a table called compute_nodes only in your nova db | 19:29 |
| tssurya | not in the api | 19:29 |
| tssurya | odyssey4me: that is in the cell1's db | 19:30 |
| *** salv-orlando has quit IRC | 19:30 | |
| odyssey4me | yup, the nova db has that table - but the table has no 'mapped' column | 19:31 |
| odyssey4me | tssurya here're the columns present: https://pastebin.com/grmGWEQe | 19:32 |
| odyssey4me | tssurya in case you missed it, this is a newton build (with no cells) being upgraded to an ocata build (with cells being setup) | 19:33 |
| odyssey4me | cells v2 to be clear | 19:33 |
| *** slaweq has joined #openstack-nova | 19:33 | |
| tssurya | odyssey4me: oh okay, ocata.. | 19:35 |
| odyssey4me | tssurya yep :) ye olde crusty stable code ;) | 19:35 |
| *** ccamacho has joined #openstack-nova | 19:36 | |
| odyssey4me | mriedem any further thoughts or ideas? if not I'll work on fixing up the log capturing so that I can point you at a proper set of logs to peruse | 19:37 |
| mriedem | odyssey4me: sorry was eating lunch, | 19:37 |
| mriedem | i think the compute_nodes.mapped column was added in ocata | 19:38 |
| mriedem | the host 'ocata-nova1' is a problem i think, since an earlier paste showed it was discovered in 2 different cell mappings | 19:38 |
| *** slaweq has quit IRC | 19:38 | |
| mriedem | https://pastebin.com/mSwpZybQ | 19:38 |
| mriedem | at this point i'd probably need the logs from an untouched env | 19:39 |
| odyssey4me | alright, thanks much for your time and input so far | 19:39 |
| mriedem | yw | 19:39 |
| odyssey4me | I'll work on getting the log collection fixed up so that we can debug better. | 19:40 |
| *** baoli has joined #openstack-nova | 19:41 | |
| *** baoli has quit IRC | 19:42 | |
| efried | mriedem: I'm going to put up three pairs of patches to dig a bit deeper, see which part of ServiceTokenAuthWrapper is actually busted. | 19:44 |
| efried | mriedem: Unless you have some other plan. | 19:44 |
| mriedem | so 6 patches? | 19:44 |
| mriedem | i don't, no :) | 19:45 |
| mriedem | we might have to put this in the release notes as a known issue | 19:45 |
| efried | Yeah, going to pass through each of the service_auth-y bits of ServiceTokenAuthWrapper - which I have to do in ksa - and then a blank Nova patch for each that Depends-On its ksa buddy. | 19:45 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | 19:45 |
| mriedem | ildikov: ^ cleans up that bfv legacy attach stuff | 19:45 |
| ildikov | mriedem: tnx, looking | 19:47 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug nova-api service version check during boot from volume https://review.openstack.org/540992 | 19:47 |
| *** mgoddard_ has quit IRC | 19:47 | |
| *** lpetrut has quit IRC | 19:48 | |
| *** tssurya has quit IRC | 19:50 | |
| mriedem | holy f 679 check queue length | 19:51 |
| melwitt | I saw there was a status message about zuul from 02:30 having problems | 19:52 |
| melwitt | "[02:30:17] -openstackstatus-NOTICE: Our Zuul infrastructure is currently experiencing some problems and processing jobs very slowly, we're investigating. Please do not approve or recheck changes for now." | 19:52 |
| mriedem | yeah i knew about that | 19:53 |
| mriedem | melwitt: you might want to weigh in on this https://review.openstack.org/#/c/532361/ | 19:53 |
| mriedem | given you'll be the ptl of the project that ruined removing mox in rocky for all projects | 19:53 |
| melwitt | -_- | 19:53 |
| mriedem | as the lame duck ptl, i can only fire shots across the bow | 19:54 |
| *** imacdonn has quit IRC | 19:55 | |
| mriedem | bauzas: don't forget https://review.openstack.org/#/c/526095/ | 19:56 |
| cdent | melwitt, mriedem : I recommend either lying ("yeah, sure we can will do it") or just do it | 19:56 |
| *** imacdonn has joined #openstack-nova | 19:56 | |
| artom | Strictly speaking, *removing* mox is easy | 19:56 |
| mriedem | cdent: i assumed we'd do what we have done in previous releases, | 19:56 |
| artom | *Replacing* it with mock, OTOH... | 19:56 |
| mriedem | which is we can work on it as a low priority thing | 19:56 |
| mriedem | so it would be what doug said, which is forward progress | 19:56 |
| cdent | mriedem: why not, since it is a goal™, make it higher priority and use the fact that it is a goal as an excuse to de-prioritize other stuff? | 19:57 |
| mriedem | i don't consider it high priority | 19:58 |
| cdent | isn't that what "openstack wide goals" means? | 19:59 |
| melwitt | yeah. while I agree it's a fine thing to do, it's not trivial to review and takes away from reviewing important things like bug fixes and placement work that our users want and need | 19:59 |
| cdent | (to be clear, I'm not too wed to the mox thing, but I do think that the goals process should mean _something_) | 19:59 |
| melwitt | and I personally don't get the argument of "maintaining mox3" because, what is there to maintain? | 19:59 |
| mriedem | right, i don't think 'retiring mox3' is a high priority | 20:00 |
| * kashyap wishes there was a way to update the messy formatting in a long Gerrit comment | 20:00 | |
| *** jafeha has joined #openstack-nova | 20:00 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: user headers https://review.openstack.org/541427 | 20:00 |
| cdent | melwitt: that seems a far more valid response to the goal process | 20:00 |
| mriedem | ffs, sqlalchemy-migrate was forked over for openstack usage, and we're still using that gd thing forever | 20:00 |
| cdent | much to the chagrin of zzzeek | 20:01 |
| mriedem | this would be like a goal to migrate every project from sqla-migrate to that other thing | 20:01 |
| mriedem | alembic | 20:01 |
| mriedem | which is arguably a more useful goal than removing mox tests | 20:01 |
| *** owalsh has quit IRC | 20:02 | |
| cdent | perhaps one of you should respond to zane with the real reasons you don't want to do it? | 20:02 |
| *** slaweq has joined #openstack-nova | 20:02 | |
| dansmith | didn't mriedem already do that? | 20:02 |
| *** owalsh has joined #openstack-nova | 20:02 | |
| *** amoralej is now known as amoralej|off | 20:02 | |
| cdent | dansmith: I'm not entirely sure. I just read his most recent response and that doesn't seem to. | 20:03 |
| *** dtantsur is now known as dtantsur|afk | 20:03 | |
| mriedem | i did, | 20:03 |
| dansmith | wasn't his first resposne "because we can't possibly get it done" ? | 20:03 |
| openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: invalidate https://review.openstack.org/541430 | 20:03 |
| cdent | But, dansmith, I was just trying to say "instead of talking about it here, maybe talk about it on the review" | 20:03 |
| mriedem | i left comments on the review | 20:03 |
| cdent | right, dansmith, that's not the same as "because we don't think it is worth doing because why not support mox3 forever" | 20:03 |
| melwitt | I will be commenting on the review. I'm just talking here for now since we were talking about it | 20:03 |
| mriedem | i won't -1 it because i don't want to further the notion that nova is the project that blocks progress on everything in openstack | 20:04 |
| cdent | yeah, I'm not saying we shouldn't talk about it here too, just hoping it gets reflected there too (it's not like I'd ever want to stop talking about stuff) | 20:04 |
| dansmith | cdent: okay I don't see the difference between what he said (and the implications on mox3) and what we're saying here, but... that's fine | 20:04 |
| cdent | mriedem: but isn't that exactly what zane has implied? | 20:04 |
| dansmith | cdent: that nova is being a blocker on purpose? | 20:05 |
| openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: connection params https://review.openstack.org/541432 | 20:05 |
| mriedem | cdent: yes it is, but i don't think that means that other projects which actually can get it done and find value in doing it, can/should do it | 20:05 |
| dansmith | I don't think he said that | 20:05 |
| *** tssurya has joined #openstack-nova | 20:05 | |
| *** tssurya has quit IRC | 20:05 | |
| cdent | dansmith: I know it is stupid that for some reason the details of this stuff become such a thing, but they do. The conversations that various members of the TC have had about the goals process have been _involved_ | 20:05 |
| mriedem | *shouldn't do it | 20:05 |
| cdent | dansmith: not on purpose, but sort of de facto | 20:05 |
| cdent | but yeah, whatever, I personally don't really care about the mox goal (or any of them really) | 20:07 |
| bauzas | mriedem: +Wd, thanks | 20:07 |
| efried | mriedem: It occurs to me that disabling bits of the service auth token will have impact outside of glance. | 20:07 |
| bauzas | oh man, mox | 20:07 |
| cdent | mriedem: thanks for that final explicit comment | 20:07 |
| bauzas | wow, https://review.openstack.org/#/c/534605/ too | 20:09 |
| *** rmart04 has joined #openstack-nova | 20:09 | |
| mriedem | bauzas: that one is much more focused and easy | 20:09 |
| mriedem | all it is is make the debug option mutable | 20:09 |
| mriedem | with a sighup | 20:09 |
| mriedem | it's not "make all options mutable" | 20:10 |
| cdent | it was initially | 20:10 |
| bauzas | I'm still chilling | 20:10 |
| *** rmart04 has quit IRC | 20:10 | |
| cdent | but I nipped it | 20:10 |
| bauzas | anyway, time will see | 20:10 |
| bauzas | the problem is not about having a mutable conf opt :) | 20:11 |
| mriedem | cdent: good - that's how i first read that one, until i re-read it | 20:11 |
| bauzas | it's rather about making sure nova can sighub | 20:11 |
| kashyap | dansmith: So, on that potential race-thing, did you see the comment with more Neutron-educated Sean Mooney? He makes a good point when he says: | 20:11 |
| kashyap | "it would be nicer to have a solution that is always guaranteed to work but even with a race it much less likely to be noticed then the race we have today were the RARP packets from qemu are often lost as the nic is not wired up." | 20:11 |
| mriedem | sighub - the special interest group for github | 20:11 |
| bauzas | oops | 20:12 |
| bauzas | sighup :p | 20:12 |
| bauzas | kill -1 :p | 20:12 |
| * kashyap --> Off for the night | 20:12 | |
| bauzas | mriedem: oh btw. why have I a plumage for our team ? :p | 20:13 |
| mriedem | bauzas: you're the only one with a high enough plumage score | 20:14 |
| mriedem | french | 20:14 |
| mriedem | mandatory minimum level of plumage | 20:14 |
| dansmith | kashyap: that has nothing to do with the race I was asking about | 20:15 |
| bauzas | mriedem: because we are all cocks ? :) | 20:15 |
| mriedem | ha | 20:15 |
| * kashyap goes to re-read dansmith's comment in the scroll | 20:15 | |
| kashyap | dansmith: Okay, I see -- you're asking the impact of setting it in two places | 20:16 |
| dansmith | kashyap: yes and in two threads | 20:16 |
| odyssey4me | mriedem actually, we have some logs from just the standard upgrade test: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/ | 20:16 |
| kashyap | So in the existing code in _live_migration(), guest.migrate() is already called with 'CONF.libvirt.live_migration_bandwidth'. And then... | 20:16 |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Isolate placement database config https://review.openstack.org/541435 | 20:16 |
| kashyap | dansmith: A test result with log would convince you I suppose? | 20:16 |
| odyssey4me | mriedem the 'host' folder will have the nova-compute logs, and the 'openstack' folder will have the other nova services logs | 20:16 |
| *** r-daneel_ has joined #openstack-nova | 20:16 | |
| dansmith | kashyap: no of course not.. if it's a race then a passing test means nothing :) | 20:17 |
| mriedem | odyssey4me: holy wingdings batman http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/openstack/openstack1/nova/nova-api-os-compute.log.txt.gz.txt.gz | 20:17 |
| odyssey4me | ouch | 20:17 |
| mriedem | oh here we go http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/openstack/openstack1/nova/nova-api-os-compute.log.txt.gz | 20:17 |
| kashyap | dansmith: An automated test that repeats, say 100 times? | 20:17 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: TrivialFix: Add a space between messages https://review.openstack.org/541436 | 20:17 |
| dansmith | kashyap: are you joking? I can't tell | 20:17 |
| kashyap | :P | 20:17 |
| *** r-daneel has quit IRC | 20:17 | |
| *** r-daneel_ is now known as r-daneel | 20:17 | |
| kashyap | Okay, it's clearly too late for me | 20:17 |
| odyssey4me | looks like we have a bug in the log collection - I'll figure that out | 20:17 |
| * kashyap wil come back afresh tomm. | 20:17 | |
| dansmith | kashyap: and if not, no. we have races that sit dormant for years until something changes in infra, things go in a different order, and we start getting breakage | 20:17 |
| dansmith | kashyap: ack | 20:18 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: TrivialFix: Add a space between messages https://review.openstack.org/541436 | 20:18 |
| *** slaweq has quit IRC | 20:18 | |
| mriedem | odyssey4me: ok here was that error you mentioned http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/openstack/openstack1/nova/nova-conductor.log.txt.gz#_2018-02-05_18_19_11_428 | 20:18 |
| odyssey4me | mriedem yep, eith the instance error state just a little later | 20:19 |
| *** slaweq has joined #openstack-nova | 20:19 | |
| mriedem | odyssey4me: i see that nova-manage cell_v2 create_cell didn't exist in newton, so when those commands run, are they against ocata? | 20:21 |
| odyssey4me | yes, in our newton deploy there is no cell management done | 20:21 |
| kashyap | dansmith: Scooping up my final active neurons: You want to see clear _functional test_ evidence that the override of CONF.libvirt.live_migration_bandwidth doesn't happen, yes? (Though, Sahid claims it doesn't.) | 20:22 |
| odyssey4me | mriedem newton only does these: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/newton/tasks/nova_db_setup.yml | 20:22 |
| dansmith | kashyap: no. | 20:22 |
| mriedem | odyssey4me: there are a bunch of errors in the nova-compute logs http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz | 20:22 |
| dansmith | kashyap: I want human analysis of whether or not setting the thing to 1 and 100 in two threads is asking for trouble | 20:23 |
| dansmith | kashyap: as far as I can tell, the migration is started with the CONF value, and then set to 1 to slow things down, but there is no locking or mechanism to ensure the slowdown call is done after the value is set to the one from conf | 20:23 |
| *** guest___ has joined #openstack-nova | 20:24 | |
| odyssey4me | mriedem hmm, even late in the log, yes: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz#_2018-02-05_18_15_57_449 | 20:24 |
| kashyap | (Okay, between the two threads where it is being set. Needs more code reading...) | 20:24 |
| odyssey4me | mriedem actually, that seems to recover: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz#_2018-02-05_18_15_57_469 | 20:24 |
| dansmith | kashyap: I can ask someone else, I thought you would be able to dig into the libvirt docs and validate or discredit sahid's assertion that it doesn't matter (which doesn't make sense to me) | 20:24 |
| *** guest___ has quit IRC | 20:24 | |
| kashyap | dansmith: Yeah, it is started with the CONF value | 20:24 |
| dansmith | kashyap: right, but I don't think there's anything that ensures that that is set before we make the call to slow it down | 20:25 |
| *** mgoddard_ has joined #openstack-nova | 20:25 | |
| mriedem | odyssey4me: right it logs a warning on the first time it doesn't find it | 20:25 |
| dansmith | kashyap: so it seems to me like it could be started, not reach the point at which the conf value is used, we set the speed limit, then the conf value gets applied | 20:25 |
| odyssey4me | these are somewhat odd though: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz#_2018-02-05_18_14_01_860 | 20:25 |
| kashyap | dansmith: Okay, you'd want to know how the two values that are being set are coordinated with some locking mechanism with the underlying libvirt API | 20:25 |
| *** ameeda_ has joined #openstack-nova | 20:26 | |
| dansmith | kashyap: no. nevermind, go to sleep, I'll try to dig into it | 20:26 |
| * kashyap stops being stupid and goes to sleep | 20:26 | |
| mriedem | odyssey4me: ok so in the console output, we see that compute show up in the service list here http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_15_43_557748 | 20:26 |
| ameeda_ | mriedem: Hello :) | 20:27 |
| mriedem | odyssey4me: it might be worthwhile to run discover_hosts with --verbose http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_15_48_024543 | 20:28 |
| mriedem | to see what it dumps out | 20:29 |
| mriedem | odyssey4me: i do remember that with devstack-based CI, we'd have issues in grenade where nova-compute would take awhile to bring up and create the compute_nodes record, and discover_hosts would miss it | 20:29 |
| *** jamesdenton has joined #openstack-nova | 20:29 | |
| mriedem | https://review.openstack.org/#/c/488381/ | 20:30 |
| odyssey4me | mriedem yeah, that's why we added the wait until a compute agent was found - that seems to have worked for us so far, but perhaps we need more sheps | 20:30 |
| mriedem | however, you have that "openstack compute service list" check | 20:30 |
| mordred | efried: looks like you have a fun debugging task on your plate | 20:31 |
| *** awaugama has quit IRC | 20:31 | |
| odyssey4me | but yeah, lemme add the --verbose CLI option when our debug flag is enabled, which would be very useful here | 20:31 |
| efried | mordred: indeeed | 20:31 |
| mriedem | odyssey4me: are the /etc/ files captured? | 20:33 |
| odyssey4me | mriedem unfortunately not - we had to turn that off 'cos we were eating up all the infra inodes :/ | 20:33 |
| odyssey4me | we could add that back, but tar.gz them so we eat fewer inodes :) | 20:33 |
| mriedem | ok, i can't tell if the [database]/connection value in nova.conf that the nova-compute is using is the same database connection as the cell1 this creates | 20:33 |
| *** ameeda_ has quit IRC | 20:36 | |
| *** vladikr has quit IRC | 20:36 | |
| mriedem | odyssey4me: anyway i think we can see that the compute_nodes entry is created here for the compute host http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/host/nova/nova-compute.log.txt.gz#_2018-02-05_18_15_57_469 | 20:36 |
| *** READ10 has quit IRC | 20:36 | |
| mriedem | at 18:15:57.469 | 20:36 |
| mriedem | it shows up in the service list here: | 20:36 |
| mriedem | http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_15_43_557748 | 20:36 |
| mriedem | "end": "2018-02-05 18:15:43.505982", | 20:37 |
| odyssey4me | mriedem db config template is here, if that helps a bit: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/ocata/templates/nova.conf.j2#L213-L233 | 20:37 |
| odyssey4me | each db will have a unique name, and each db user/password will also be unique | 20:37 |
| *** mgoddard_ has quit IRC | 20:38 | |
| dansmith | kashyap: I explained my concern in more detail, maybe you could dig into docs/code tomorrow with that extra info to validate one way or the other? | 20:38 |
| mriedem | odyssey4me: looks ok | 20:39 |
| openstackgerrit | melanie witt proposed openstack/nova master: Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442 | 20:39 |
| dansmith | melwitt: nice | 20:41 |
| mriedem | odyssey4me: also throw the --verbose option on create_cell | 20:41 |
| *** sridharg has quit IRC | 20:42 | |
| mriedem | ultimately my guess is discover_hosts isn't picking up the compute host and mapping it to the cell | 20:43 |
| mriedem | just not sure why | 20:43 |
| mriedem | could be a timing issue, but that's what the openstack service list check is for | 20:43 |
| melwitt | dansmith: thanks for pointing it out on the original review | 20:44 |
| dansmith | melwitt: yar | 20:44 |
| efried | mriedem: What's your trick for seeing those nova-new results before they're reported to gerrit? | 20:44 |
| dansmith | melwitt: that race is the whole reason I wrote this eventing stuff so I get all sad when it gets broken | 20:44 |
| odyssey4me | yeah, that's my one suspicion... a race condition of some sort - the other is that perhaps there's an online migration required which wasn't executed | 20:45 |
| mriedem | efried: status.openstack.org/zuul | 20:45 |
| mriedem | odyssey4me: i see that the online data migrations are skipped | 20:45 |
| efried | Oh, look, those are links! | 20:45 |
| odyssey4me | mriedem thanks again for looking through all that, and for sanity checking the process to verify that everything looks like it's doing what it should... I'll get a better set of test logs/configs going and we can revisit when those are good | 20:45 |
| mriedem | odyssey4me: http://logs.openstack.org/29/540329/5/check/openstack-ansible-upgrade-ubuntu-xenial/30ae3e2/job-output.txt.gz#_2018-02-05_18_03_50_709904 | 20:46 |
| mriedem | <openstack1> Task "Disable the online migrations requirement" has been omitted from the job because the conditional "[u"inventory_hostname == groups['nova_conductor'][0]", u'not data_migrations | skipped', u'data_migrations | succeeded']" was evaluated as "False" | 20:46 |
| odyssey4me | mriedem yep, they're skipped due to the check to see whether all hosts are upgraded before running them... and I don't think we have something in these role tests which actually check for that and execute them, so I think I'll patch that up | 20:46 |
| *** traskat has quit IRC | 20:46 | |
| odyssey4me | I thought we had one, but perhaps it got patched in later and not ported back | 20:47 |
| efried | eff me, guess I need to wait for some of those four hundred patches to clear out of the check queue. | 20:47 |
| melwitt | dansmith: understandable. I'm pretty new to the neutron event scene and have been learning | 20:47 |
| *** vladikr has joined #openstack-nova | 20:49 | |
| *** chyka_ has quit IRC | 20:49 | |
| *** hongbin has joined #openstack-nova | 20:50 | |
| efried | D'oh. The check queue seems to be smart enough not to run functional tests if no .py files were touched. | 20:54 |
| mriedem | someone want to fast approve this? https://review.openstack.org/#/c/539978/ | 20:55 |
| openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: user headers https://review.openstack.org/541427 | 20:56 |
| openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: invalidate https://review.openstack.org/541430 | 20:56 |
| openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Debug nova-next failure: connection params https://review.openstack.org/541432 | 20:56 |
| efried | mriedem: There is no fast. There is only zuul. | 20:57 |
| sean-k-mooney | efried: zull used to be fast for small projects. now zuul is fair to all projects. as some who also works on small projects this make me sad | 20:58 |
| efried | Seems snappy for nova-powervm - unless the whole world is broken. | 20:58 |
| sean-k-mooney | efried: ah well the upstream zuul used to round robin taking the first job from each queue in a pipline so small pojects with short to no queue were more likely to have jobs run. now it handels jobs in the order it arrived being fair to all pojects | 21:00 |
| *** takamatsu has quit IRC | 21:01 | |
| cdent | melwitt, mriedem: would you have any objection to me setting a little goal for myself for nova: remove code from all the __init__.py files (to avoid inadvertent imports)? That is, if I were to do it, is there a chance it would get merged. | 21:01 |
| sean-k-mooney | efried: if you have your own zuul instance for thirdparty testing then it is still fast for your on defintion(policy of job selection) of fast | 21:01 |
| mriedem | cdent: that's somehow related to splitting out placement right? | 21:02 |
| cdent | mriedem: I discovered it because of that, yes, but it's also beneficial for the rest of nova: the fact that it is a common behavior means that various processes are often importing many more modules than they actually need. | 21:03 |
| mriedem | i'd say it has as good a chance at getting merged as anything else | 21:03 |
| sean-k-mooney | cdent: mriedem even if it not related to splitting out placement can we do it anyway. haveing side effect form importing moduels makes testing harder and is really unintuitive to people who have not worked with that module before | 21:04 |
| cdent | mriedem: for latest example see my note to self on https://review.openstack.org/#/c/541435/ | 21:04 |
| mriedem | sean-k-mooney: you act like i said no one could push patches | 21:04 |
| cdent | would a spec or blueprint be of any value? | 21:05 |
| sean-k-mooney | mriedem: hehe sorry i know that is not what you ment. i just hate seeing __init__.py files with anything but constants in them | 21:05 |
| mriedem | cdent: idk | 21:05 |
| sean-k-mooney | mriedem: its a pet pev of mine. | 21:06 |
| mriedem | cdent: if it's a non-trivial amount of work, and is a depednency for the eventual placement split, then i could see a specless bp | 21:06 |
| cdent | yeah, sadly it would probably be non-trivial as it often ends with wiggle here causing ripples way over there | 21:07 |
| cdent | I'll put it my queue, thanks. | 21:07 |
| cdent | and I think that can be the end of my work day | 21:08 |
| sean-k-mooney | cdent: that part of why i dont like that patten in python. for example import anyting in nova.cmd results in your process being monkey patched https://github.com/openstack/nova/blob/master/nova/cmd/__init__.py | 21:08 |
| cdent | yup | 21:09 |
| cdent | 'night all | 21:09 |
| *** cdent has quit IRC | 21:09 | |
| *** artom has quit IRC | 21:13 | |
| openstackgerrit | Merged openstack/nova master: Add PowerVM to feature-classification https://review.openstack.org/533237 | 21:16 |
| efried | edleafe: yt? | 21:16 |
| *** hemna_ has quit IRC | 21:17 | |
| *** AlexeyAbashkin has quit IRC | 21:21 | |
| kashyap | dansmith: Okay, after watching the Falcon launch, I came back to IRC | 21:21 |
| kashyap | dansmith: Saw your comment in the review. Much clearer. | 21:22 |
| kashyap | So, I asked a libvirt dev who does migration to look into it -- Will the 'bandwidth' parameter of migrateToURI2() will be ignored if it was previously set by the API migrateSetMaxSpeed()? | 21:22 |
| mriedem | dansmith: i'm going to announce this here publicly, but melwitt has never been asked to give a sizing | 21:23 |
| mriedem | sdague: ^ is that an ibm only thing? | 21:23 |
| edleafe | efried: ? | 21:23 |
| mriedem | melwitt: this was much more fun before you said LOEs | 21:24 |
| efried | edleafe: Wanted to do a quick brainstorm if you have a mo. | 21:24 |
| melwitt | mriedem: haha, sorry | 21:24 |
| sean-k-mooney | mriedem: a sizing of a task? if so how, i get ask that all the time for stuff i dont even work on | 21:24 |
| kashyap | dansmith: Will respond on that review once I dig into the Python bindings and upstream relevant APIs to confirm what you asked. (Also double-confirm with a libvirt upstream dev.) | 21:24 |
| edleafe | efried: sure | 21:24 |
| * kashyap now _really_ off | 21:24 | |
| mriedem | sean-k-mooney: of course you do, because you work at a corporation | 21:24 |
| sean-k-mooney | mriedem: :) ah so thats where i went wrong. | 21:25 |
| efried | edleafe: Trying to think through an easy way to get NUMA affinity in placement modeling. First off, do you understand the stuff Jay has been talking about, something about aggregate "distance"? | 21:25 |
| edleafe | efried: vaguely | 21:25 |
| efried | edleafe: Can you help me understand it? | 21:26 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: List/show all server migration types https://review.openstack.org/540256 | 21:26 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration https://review.openstack.org/334732 | 21:26 |
| sean-k-mooney | mriedem: also because of that we are ment to track everything we do upstream and in jira and in downstream tools with vendors ... corporate processes :) | 21:26 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Transform live_migration_post_dest notification https://review.openstack.org/469784 | 21:26 |
| edleafe | efried: my weak understanding is that it would favor RAM that is associated with a given NUMA node, but that if there wasn't enough, it would use RAM that was "further away" | 21:27 |
| efried | edleafe: Okay, but how does this "distance" thing get defined? | 21:27 |
| edleafe | efried: that's the part that I don't have any clue about | 21:27 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells https://review.openstack.org/521400 | 21:28 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells https://review.openstack.org/521400 | 21:28 |
| mriedem | sean-k-mooney: do you track how much time you aren't spending working on grant requests when you should be? | 21:28 |
| efried | edleafe: Okay, me neither. So starting from first principles, I was thinking about how one could model NUMA affinity. And here's what I came up with: | 21:28 |
| edleafe | efried: you'd have to have Jay give a lesson on this (to all of us) | 21:28 |
| efried | edleafe: GET /allocation_candidates?<...the usual resource requests...>&affine_resources=<resource_class>,<resource_class>,... | 21:29 |
| mriedem | efried: L142 https://etherpad.openstack.org/p/nova-ptg-queens-placement | 21:29 |
| sean-k-mooney | mriedem: hehe if it did i would cry | 21:29 |
| *** salv-orlando has joined #openstack-nova | 21:30 | |
| efried | mriedem: Thanks. Will dig. | 21:30 |
| efried | edleafe: So first of all, it would only apply to the unnumbered request group. Because you're already getting same-provider "affinity" in the numbered groups. | 21:30 |
| *** felipemonteiro has joined #openstack-nova | 21:31 | |
| efried | edleafe: So within the unnumbered request group, the API would force the resources from classes you list to come from either the same provider, or the same aggregate. | 21:31 |
| efried | edleafe: I'm not sure an anti_affine_resources makes any sense, but could be added if it does. | 21:32 |
| *** threestrands has joined #openstack-nova | 21:32 | |
| edleafe | efried: "affine"? | 21:32 |
| edleafe | efried: How would that specify RAM, PCI deveice, etc., associated with a given NUMA? | 21:33 |
| efried | edleafe: Okay, let me give a more complete example, make more sense. | 21:33 |
| *** salv-orlando has quit IRC | 21:33 | |
| sean-k-mooney | edleafe: anti_affine_resources for nic bonding e.g. antiafintiy of the pf for the VF request | 21:34 |
| sean-k-mooney | efried: ^ | 21:34 |
| efried | sean-k-mooney: ack | 21:34 |
| efried | edleafe: To start with, model NUMA nodes as subtrees under the root compute host RP (which may not in fact provide any resources). So maybe the first tier children provide CPU & memory, and the second tier (grandchildren) are the network devices providing VFs. | 21:35 |
| efried | edleafe: Associate each child/grandchildren in a given NUMA node with the same aggregate UUID. | 21:35 |
| efried | edleafe: Then your request would be like GET /allocation_candidates?resources=VCPU:2,MEMORY_MB:2048,SRIOV_NET_VF:1,DISK_GB=2048&affine_resources=VCPU,MEMORY_MB,SRIOV_NET_VF | 21:36 |
| edleafe | why can't the network devices be children of the NUMA node? | 21:37 |
| efried | They are | 21:37 |
| efried | swhat I said | 21:37 |
| efried | Sorry, to rephrase: | 21:37 |
| sean-k-mooney | efried: but the cpu and memory shoudl also be childeren of the numa node too | 21:37 |
| edleafe | "and the second tier (grandchildren) are the network devices providing VFs." | 21:37 |
| efried | sean-k-mooney: The CPU/mem are resources provided by the numa node which is the RP. | 21:37 |
| sean-k-mooney | efried: ah they are inventories of the numa node rather then sub resouce providres but that means you can have traints on the cpu and ram | 21:38 |
| sean-k-mooney | *can't have traits | 21:39 |
| efried | So root RP is the compute node. It provides no resources. It has two child RPs, representing NUMA nodes. They provide VCPU and memory inventory. Then those guys have child RPs representing network devices, providing VF inventory. | 21:39 |
| *** pchavva has quit IRC | 21:39 | |
| edleafe | efried: one thing we learned back in Atlanta was that this stuff was much better discussed with a drawing surface so that all could see the relationships | 21:39 |
| edleafe | Those of us who think visually need that for anything other than the trivial stuff | 21:40 |
| mriedem | https://awwapp.com/# ? | 21:40 |
| efried | sean-k-mooney: Yeah, that would be another way to do it: model the NUMA provider as an empty child of the empty root compute host and give the NUMA provider three grandchild tiers which provide CPU, memory, and VFs. | 21:40 |
| efried | But not to get hung up on those details. | 21:40 |
| edleafe | mriedem: sure. efried, have at it! | 21:40 |
| mriedem | or https://webwhiteboard.com/ | 21:41 |
| edleafe | And I'm still not clear how "distance" figures into any of this | 21:41 |
| mriedem | i won't be joining, but just being super helpful | 21:41 |
| *** dave-mcc_ has quit IRC | 21:41 | |
| efried | edleafe: It doesn't. This is purely one-dimensional affinity. Either they're in the same agg or they're not. | 21:41 |
| edleafe | efried: ok, that's where I was not clear | 21:42 |
| *** jmlowe has quit IRC | 21:42 | |
| *** hemna_ has joined #openstack-nova | 21:42 | |
| *** jafeha__ has joined #openstack-nova | 21:42 | |
| edleafe | efried: won't the current NRP stuff handle this without additional work? IOW, the CPU,RAM,VF all have to | 21:43 |
| edleafe | be from the same provider | 21:43 |
| efried | Only if we can model them as belonging to the same provider. | 21:43 |
| sean-k-mooney | edleafe: distance was discused in 2 forms. 1 a distance in nodes along the provider tree and 2 a distence to a shared resouce provider and an aggreate member to model things like network latency for share storage | 21:43 |
| efried | Which we can do for just CPU/memory, but not for VFs. | 21:44 |
| *** jafeha has quit IRC | 21:44 | |
| edleafe | efried: yeah, I guess it could think of the compute node as the root provider, and then stuff from different NUMA nodes would match | 21:44 |
| efried | And also splitting things up more allows us to tie traits more specifically to their resources, if that matters. | 21:44 |
| edleafe | sean-k-mooney: I remember the shared distance thing. I thought that was discarded as not needed, because you could just create tighter aggregates so that only "near" storage would ever be used | 21:45 |
| efried | sean-k-mooney: I tried thinking about distance along a tree, but kept painting myself into a corner where that would severely restrict how things would have to be modeled. | 21:46 |
| sean-k-mooney | edleafe: yes proably its been a while since denver | 21:46 |
| efried | sean-k-mooney: Do you understand the concept of distance as relates to aggregates? Cause I don't. | 21:46 |
| efried | https://awwapp.com/b/umrhofjje/ | 21:47 |
| *** hemna_ has quit IRC | 21:48 | |
| edleafe | efried: so we would use aggs to denote a NUMA node? | 21:49 |
| efried | aggzactly. | 21:50 |
| edleafe | ugh | 21:50 |
| *** itlinux has quit IRC | 21:50 | |
| edleafe | you'd probably have to extend the aggs to include the compute node, no? | 21:51 |
| efried | I don't know how we get anti-affinity with this, though. Anything down the anti_affine_resources thought path is bringing my head crashing against a brick wall. | 21:51 |
| efried | The least of the problems being the lack of symmetry. | 21:52 |
| sean-k-mooney | efried: not very well. the distance along the tree i understand. all inventoies of the same resouce provider have a distance of 0, parent/direct chile is distance or 1 and so on | 21:52 |
| efried | edleafe: I don't think so, in this case, since the compute node isn't providing any resources. | 21:52 |
| edleafe | efried: I was just thinking of the eventual monster SQL that would have to return the CN | 21:52 |
| * edleafe has to step away for a few minutes | 21:53 | |
| efried | edleafe: That's another topic, but IMO we could be doing stuff with way simpler (though possibly less efficient) SQL that would be way more flexible and provably correct. | 21:53 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Address comments from I51adbbdf13711e463b4d25c2ffd4a3123cd65675 https://review.openstack.org/541455 | 21:54 |
| efried | sean-k-mooney: Right. And I suppose as long as the model is somewhat symmetrical (that is, all providers of a given resource are always at the same level of the tree - which shouldn't be too hard to encourage) and mutually understood by the dev and the op for a given driver (which pretty much has to be the case anyway) it could probably be made to work. | 21:54 |
| *** itlinux has joined #openstack-nova | 21:55 | |
| sean-k-mooney | edleafe: well if recall correctly jay did not want this to be handeled in the sql code but rater in a filter due to the complexity of the sql | 21:55 |
| sean-k-mooney | that said perhaps that was just for numa and not distance in general | 21:56 |
| sean-k-mooney | if there is a relitivly effcient way to handel numa/distance in sql without lots of complexity and ram the sql would certenly be better. | 21:59 |
| sean-k-mooney | actuly when i said filter i think i should have said python e.g. placement would still do the filting not the nova filters | 22:00 |
| *** ccamacho has quit IRC | 22:01 | |
| efried | sean-k-mooney: I'm on board with that. | 22:01 |
| openstackgerrit | melanie witt proposed openstack/nova master: WIP Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442 | 22:02 |
| efried | Swhat I was hinting at earlier: we should be doing more of the processing in python, when it comes to the really complex stuff. The SQL is too hard to understand, too hard to test/prove, and too few people grok it. | 22:02 |
| * edleafe is back | 22:04 | |
| edleafe | efried: 100% on the non-SQL processing | 22:04 |
| edleafe | efried: I think Jay likes the challenge. :) | 22:04 |
| *** salv-orlando has joined #openstack-nova | 22:05 | |
| efried | No doubt. And no doubt when we get it right it's way more efficient. But over this release I've come to think it's not worth the tradeoff. | 22:05 |
| sean-k-mooney | edleafe: perhaps buy would you like the code review :) | 22:05 |
| edleafe | I'd rather review python than sqla | 22:06 |
| efried | ++ | 22:06 |
| efried | Okay, well, I was coming into this thinking about proposing a spec to handle affinity in Rocky. | 22:07 |
| edleafe | I also think that subtle bugs are more readily spotted in code | 22:07 |
| edleafe | I used to be a SQL DBA, and I still can't catch all the implications of some of the sqla used in placement | 22:08 |
| efried | ...But I don't think I have a good enough handle yet on the use cases; and I think I need to understand this idea of "aggregate distance" before putting anything on paper. | 22:09 |
| edleafe | at some point you have to specify NUMA in the request, no? | 22:09 |
| efried | Well, no. | 22:09 |
| efried | With the model as I've described it, you would never have to say NUMA in any context. | 22:10 |
| efried | You just have to say "keep these resources together" | 22:10 |
| efried | and placement understands what you mean by "together" (because that's how we design it) | 22:11 |
| edleafe | efried: sure, but what I meant is that you don't want to be giving out NUMA nodes to someone who just wants a basic VM | 22:11 |
| efried | edleafe: Totally. In which case you don't say affine_resources=... at all. And it operates like it does today. You get your resources from wherever in the tree. | 22:11 |
| *** hemna_ has joined #openstack-nova | 22:12 | |
| efried | edleafe: Now it's likely I'm still completely misunderstanding how NUMA is actually wired, hardware-wise. | 22:12 |
| edleafe | efried: what I'm saying is that if you request NUMA, the code would ensure that all the resources are provided by a single node. | 22:12 |
| efried | edleafe: But I'm saying you request NUMA not by saying NUMA but by saying affine_resources=<whatever resources need to come from the same NUMA node> | 22:12 |
| edleafe | efried: according to the people who seemed to know more about it than I do, a machine with NUMA can be configured in a variety of ways. | 22:13 |
| edleafe | efried: so if all the resources were provided by a plain compute node, wouldn't that also get returned? | 22:14 |
| sean-k-mooney | edleafe: yes it they can. i know more about this topic then is good for my health | 22:14 |
| sean-k-mooney | edleafe: if that compute node has only one memoy controler then by definiton all resocues on that node belong to the same numa node | 22:15 |
| efried | edleafe: Okay, that's a fair point. Which we could resolve, as you say, by tagging the NUMA RPs with a NUMA trait. | 22:15 |
| edleafe | ah good. So efried, just direct your questions to sean-k-mooney | 22:15 |
| sean-k-mooney | so all compute nodes always have at least 1 numa node | 22:15 |
| *** openstackgerrit has quit IRC | 22:16 | |
| efried | Okay, so the only way to get non | 22:16 |
| efried | non-NUMA-ness is to go to a system that has more than one NUMA node and get resources spread out across them. | 22:16 |
| efried | In which case we *don't* need the trait. | 22:17 |
| sean-k-mooney | efried: and if you incldue a distance to restict it | 22:17 |
| *** salv-orlando has quit IRC | 22:17 | |
| sean-k-mooney | if you have multiple numa node and put no constraitn on the distance between resocues in the tree then you get resouce from different numa nodes | 22:18 |
| sean-k-mooney | your performace will be less but it will still work | 22:18 |
| sean-k-mooney | and you never asked for any afintiy so it fine | 22:18 |
| efried | Okay, so this is a workable thing for NUMA affinity. I don't think the same mechanism can be made to work for affinity in general, or for anti-affinity. | 22:21 |
| *** openstackgerrit has joined #openstack-nova | 22:21 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Move the nova-next job in-tree and update it https://review.openstack.org/541474 | 22:21 |
| sean-k-mooney | efried: if you dont have to deal with aggreates then i think it will work not sure how distance is compute for aggregates however. | 22:22 |
| efried | sean-k-mooney: I'm talking about using one aggregate per NUMA node (so nothing to do with distance). | 22:22 |
| sean-k-mooney | when modeling afinity via distance affinty becoures deistnace < X and anti afinity is distance > X | 22:23 |
| sean-k-mooney | efried: oh then ya if numa nodes are aggrages then that is different | 22:23 |
| efried | Yeah, I get that, but *modeling* and *expressing* distance seems really complicated for people to understand. | 22:23 |
| cfriesen | efried: one aggregate per numa node? would you then associate a pool of RAM, CPUs, PCI devices, etc, with that aggregate? | 22:24 |
| cfriesen | at probably 2-4 numa nodes per compute node that seems like a lot of aggregates | 22:24 |
| sean-k-mooney | you would make the cpus/ram/pci devices resouce providers members of the aggregate | 22:24 |
| efried | cfriesen: You associate the resource *providers* with the aggregate. So whatever providers provide the resources associated with your NUMA node. | 22:25 |
| efried | yeah, what sean said. | 22:25 |
| efried | No, it would be one aggregate per NUMA node, period. | 22:25 |
| sean-k-mooney | efried: not one aggregate per numa node per host? | 22:25 |
| efried | sean-k-mooney: Yes. | 22:25 |
| efried | One aggregate per NUMA node, in the global sense :) | 22:26 |
| sean-k-mooney | oh ame that aggreate will be am large | 22:26 |
| efried | Is that "a lot" of aggregates? Meh. Aggregates are cheap. | 22:26 |
| efried | I feel like we're not talking in the same terms. Here: If you have 10 hosts, and each host has 2 NUMA nodes, you have 20 aggregates total. | 22:27 |
| sean-k-mooney | i think i would prefer lots of small aggregate then a few giant cluster wide ones | 22:27 |
| sean-k-mooney | efried: ah ok that what i was thinking we should have if we used aggregates so im cool with that | 22:27 |
| *** jmlowe has joined #openstack-nova | 22:28 | |
| *** rcernin has joined #openstack-nova | 22:28 | |
| sean-k-mooney | we dont have traits on aggregates but at least for numa i dont think we need them on teh aggreages so it could work | 22:29 |
| cfriesen | so you'd have these per-numa-node aggregates for system use, then per-host aggregates to match against flavor/image properties and act as availability zones? | 22:29 |
| efried | sean-k-mooney: We can always stamp the involved providers with traits. But not sure we need 'em. At least to mark NUMA nodes. | 22:29 |
| efried | cfriesen: I know nothing about host aggregates or availability zones. | 22:30 |
| *** amodi has quit IRC | 22:30 | |
| sean-k-mooney | cfriesen: well an aggreaget is just a bag of resouce providers nothing more or less. you can use them to group anything you want | 22:30 |
| efried | cfriesen: Except that placement aggregates != host aggregates | 22:30 |
| efried | cfriesen: The other main use case for aggregates is to associate sharing resource providers, like a shared storage pool. | 22:30 |
| efried | ^ placement aggregates, that is. | 22:31 |
| cfriesen | efried: okay...I'm not up on the placement terminology. :) | 22:32 |
| efried | cfriesen: I'm not up on... pretty much everything else. Between the two of us, we're gonna need a pidgin. | 22:32 |
| *** tbachman_ has joined #openstack-nova | 22:33 | |
| sean-k-mooney | cfriesen: a sharing resouce provider is jsut a normal resouce provider with a trait and then its inventores are consumable by anything in the same aggreates as it | 22:33 |
| cfriesen | so how do we handle the idea of an instance consuming cpu/ram from one placement aggregate but a PCI device from another placement aggregate? (for the non-strict PCI affinity case) | 22:34 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/513160 | 22:34 |
| mriedem | melwitt: cleaned that up ^ | 22:34 |
| *** tbachman has quit IRC | 22:34 | |
| *** tbachman_ is now known as tbachman | 22:35 | |
| *** chyka has joined #openstack-nova | 22:35 | |
| sean-k-mooney | cfriesen: instance consume resocues form invetories provided by resouce providers | 22:35 |
| melwitt | mriedem: thanks. sorry I lost track of that one | 22:35 |
| sean-k-mooney | cfriesen: and resouce provires can be member of 0-n aggregates | 22:35 |
| sean-k-mooney | so you are not consuming form the aggregates | 22:36 |
| efried | cfriesen: In the design I'm strawpersoning, if you care about CPU/mem affinity and you don't care about VF affinity, you would say affine_resources=VCPU,MEMORY_MB but not SRIOV_NET_VF | 22:37 |
| *** edmondsw has quit IRC | 22:37 | |
| efried | cfriesen: s/VF/PCI device/ if you wish. | 22:37 |
| *** edmondsw has joined #openstack-nova | 22:38 | |
| sean-k-mooney | efried: that may break down a bit with anti afinity | 22:39 |
| efried | sean-k-mooney: I'm saying we're not handling anti-affinity at all with this model. | 22:39 |
| sean-k-mooney | e.g. i want 2 VF whoes parent PF are anti affined | 22:39 |
| sean-k-mooney | efried: ah ok | 22:39 |
| efried | sean-k-mooney: Yeah, I get that. But that's not a NUMA use case, right? | 22:39 |
| efried | It's more about HA | 22:40 |
| efried | redundancy | 22:40 |
| efried | powerfail grouping | 22:40 |
| efried | that kind of thing. | 22:40 |
| sean-k-mooney | ya its for ha bonding so my bond dont come from the same card | 22:40 |
| sean-k-mooney | e.g. pulling out one cable wont bring donw the bond | 22:40 |
| efried | So yeah, this doesn't cover that. And if there's a generic solution that encompasses both, great. But I'm gonna have to pick Jay's brain for that. | 22:40 |
| *** edmondsw has quit IRC | 22:42 | |
| sean-k-mooney | efried: if you can show your propasl support affinity in a clean way and does not block future addtion of anti afinity by design then its better then nothing | 22:42 |
| sean-k-mooney | e.g. we dont have to support everything in one go in rocky | 22:43 |
| sean-k-mooney | it would be nice however | 22:43 |
| efried | That's kinda what I was thinking. Propose it this way, show that it wouldn't prevent a more generic solution in the future, and then Jay can propose (or not) that more generic solution. | 22:44 |
| cfriesen | efried: what about multi-numa-node guests? they would need to essentially do numa anti-affinity (but host affinity) between the two pools of memory | 22:46 |
| efried | cfriesen: What does that mean? Multi-numa-node guests? You mean guests that don't care about NUMA affinity? Or do they explicitly care about *anti* affinity? | 22:47 |
| sean-k-mooney | cfriesen: multi numa guest do not need to be on multiple host numa nodes | 22:47 |
| cfriesen | efried: no, I mean guests with multi numa nodes | 22:47 |
| sean-k-mooney | cfriesen: i consider that limitation in libvirt to be a bug not a feature | 22:47 |
| *** salv-orlando has joined #openstack-nova | 22:47 | |
| sean-k-mooney | libvirt virt driver that is | 22:47 |
| cfriesen | sean-k-mooney: one reason why you might "require" multi-numa-node guests is to get access to more memory bandwidth | 22:47 |
| sean-k-mooney | cfriesen: yes but there is a flavour extra spec defined for that that only the hyperv dirver support | 22:48 |
| sean-k-mooney | cfriesen: i think the libvirt one should too | 22:49 |
| cfriesen | sean-k-mooney: sorry, don't follow. | 22:49 |
| sean-k-mooney | cfriesen: hw:numa_mempolicy | 22:50 |
| cfriesen | sean-k-mooney: I can currently specify a multi-numa-node guest with libvirt, and I get access to that many host numa nodes, with accordingly more host memory bandwidth. (presumably combined with dedicated cpus) | 22:50 |
| cfriesen | numa_mempolicy isn't mentioned anywhere in nova | 22:50 |
| sean-k-mooney | cfriesen: i used to be in the hyperv driver | 22:51 |
| sean-k-mooney | anyway that is off topic | 22:52 |
| sean-k-mooney | efried: a multi numa gues is a guest with multiple virtual numa nodes | 22:52 |
| cfriesen | efried: and currently they map to different physical numa nodes of the same host. | 22:52 |
| sean-k-mooney | efried: you can choose what virtual cores are associated to each virtual numa node and you can device the memoy between each virtual numa node | 22:53 |
| *** itlinux has quit IRC | 22:53 | |
| efried | Well, if you wanted to get that specific, you could probably swang something with granular request groups. | 22:54 |
| efried | But we still wouldn't be able to guarantee you anti-affinity. | 22:54 |
| cfriesen | if you can't guarantee numa-antiaffinity it'll change the current behaviour | 22:54 |
| cfriesen | (ie if both virtual numa nodes could end up on the same host numa node) | 22:55 |
| sean-k-mooney | placement does not need to give that guarentee | 22:55 |
| sean-k-mooney | the virt driver can via the resouce tracker | 22:55 |
| efried | sean-k-mooney: Yeahhhh, but we eventually *should* be able to implement that filter in placement. | 22:55 |
| sean-k-mooney | or the nova numa topology filter | 22:55 |
| *** burt has quit IRC | 22:55 | |
| sean-k-mooney | efried: talk to jay about that. in principal i agree | 22:56 |
| *** pchavva has joined #openstack-nova | 22:56 | |
| sean-k-mooney | i would like placement to fully handel numa. but if it can handel 80% of numa in rocky with the rest in the filetr i would be happy with that too | 22:57 |
| efried | Okay. I'll try to put up that spec later this week. | 22:57 |
| sean-k-mooney | cfriesen: im sure you know this also but just requesting guest numa nodes was never ment to give any form of cpu affinity either. from an api perspcetive it does not but for libvir it pince the cores of that numa node to float over a host numa node | 22:59 |
| sean-k-mooney | cfriesen: similay requsting hugepage was not ment to create a numa topology for the guest and also restice the floating cores to a host numa node | 23:00 |
| sean-k-mooney | cfriesen: all of those implit behaviors are internal implentation behaviors of the libvirt driver | 23:01 |
| cfriesen | sean-k-mooney: the hugepage one is a qemu limitation though since you can only point at a single backing file for memory | 23:01 |
| cfriesen | (on a given guest numa node) | 23:01 |
| sean-k-mooney | cfriesen: yes and no you numa node number is optionaly and numactl will be used to determin the optimal numa node to use of you dont specify one | 23:03 |
| cfriesen | sean-k-mooney: how do you tell qemu to use hugepages without specifying a backing file? | 23:04 |
| *** slaweq has quit IRC | 23:04 | |
| sean-k-mooney | cfriesen: you do specify a backing file but the backing file itesf does not have numa affintiy | 23:05 |
| sean-k-mooney | cfriesen: when you create a hugepage backing file under /dev/hugepages/libvrt/qemu the numa node is not part of the path | 23:06 |
| *** gjayavelu has quit IRC | 23:06 | |
| sean-k-mooney | cfriesen: checkout https://libvirt.org/formatdomain.html#elementsMemoryBacking the note on the nodeset being optional | 23:06 |
| *** felipemonteiro has quit IRC | 23:06 | |
| *** hemna_ has quit IRC | 23:07 | |
| *** felipemonteiro has joined #openstack-nova | 23:07 | |
| sean-k-mooney | cfriesen: tacking hugepages per numa node made the resouce track code simpeler which is why it was doen per numa node originally | 23:10 |
| *** sdague has quit IRC | 23:10 | |
| *** r-daneel has quit IRC | 23:12 | |
| *** r-daneel has joined #openstack-nova | 23:13 | |
| *** moshele has joined #openstack-nova | 23:14 | |
| *** gjayavelu has joined #openstack-nova | 23:15 | |
| cfriesen | sean-k-mooney: ah, right. I was thinking that mapping hugetlbfs was numa-affined, but it's not necessarily. | 23:17 |
| sean-k-mooney | cfriesen: you might be able to do somthing via the mount options but by defualt no | 23:18 |
| *** felipemonteiro has quit IRC | 23:18 | |
| *** Trident has quit IRC | 23:18 | |
| sean-k-mooney | efried: actully speaking of memory and numa we should really be tracking 4k pages e.g. non hugepage memory per numa node too | 23:19 |
| sean-k-mooney | the kernel out of memory killer is triggered per numa node not gloably | 23:19 |
| sean-k-mooney | so the reserved memory config option does not work properly on a numa system | 23:20 |
| sean-k-mooney | vms can still be killed if a numa node has no memoy left and the kernel need to allocate some on that node but the host othersise has free ram | 23:20 |
| sean-k-mooney | our just make all your vms use hugepages that works too | 23:21 |
| *** Tom-Tom has joined #openstack-nova | 23:21 | |
| cfriesen | that's what we do by default (2M pages) | 23:25 |
| *** Tom-Tom has quit IRC | 23:26 | |
| sean-k-mooney | cfriesen: ya we spent 6 months trying to track down what was causeing OOM to kill our vms in the the intel nfv ci and it turned out to be this. hugepages fixed it | 23:26 |
| sean-k-mooney | the anoying thing was it took about 2-3 weeks to start showing up as it need the memory to become fragmented enough to force it | 23:27 |
| *** acormier has quit IRC | 23:28 | |
| *** acormier has joined #openstack-nova | 23:29 | |
| *** brad[] has quit IRC | 23:29 | |
| *** acormier has quit IRC | 23:30 | |
| *** acormier has joined #openstack-nova | 23:30 | |
| *** slaweq has joined #openstack-nova | 23:31 | |
| *** acormier has quit IRC | 23:35 | |
| *** pchavva has quit IRC | 23:36 | |
| *** slaweq has quit IRC | 23:37 | |
| *** masahisa has joined #openstack-nova | 23:41 | |
| openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | 23:42 |
| *** masahisa has quit IRC | 23:44 | |
| openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | 23:44 |
| *** moshele has quit IRC | 23:50 | |
| *** rodolof has quit IRC | 23:53 | |
| *** stakeda has joined #openstack-nova | 23:54 | |
| *** artom has joined #openstack-nova | 23:58 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add a prelude release note for the 17.0.0 Queens GA https://review.openstack.org/541495 | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!