| *** yamahata has quit IRC | 00:00 | |
| *** jogo has quit IRC | 00:00 | |
| *** markvoelker has quit IRC | 00:01 | |
| *** crushil has quit IRC | 00:01 | |
| *** crushil has joined #openstack-nova | 00:02 | |
| *** markvoelker has joined #openstack-nova | 00:02 | |
| *** markvoelker has quit IRC | 00:04 | |
| *** markvoelker has joined #openstack-nova | 00:04 | |
| *** baoli has quit IRC | 00:07 | |
| *** liverpooler has quit IRC | 00:08 | |
| *** yamahata has joined #openstack-nova | 00:15 | |
| *** chyka has joined #openstack-nova | 00:19 | |
| *** tjones has joined #openstack-nova | 00:19 | |
| *** nic has quit IRC | 00:23 | |
| *** thorst has joined #openstack-nova | 00:25 | |
| *** thorst has quit IRC | 00:27 | |
| *** crushil has quit IRC | 00:32 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Use VIR_DOMAIN_BLOCK_REBASE_COPY_DEV when rebasing https://review.openstack.org/465205 | 00:33 |
|---|---|---|
| *** ijw has quit IRC | 00:36 | |
| *** ijw has joined #openstack-nova | 00:36 | |
| *** hongbin has quit IRC | 00:37 | |
| *** fragatin_ has joined #openstack-nova | 00:38 | |
| *** fragatin_ has quit IRC | 00:39 | |
| *** fragatin_ has joined #openstack-nova | 00:39 | |
| *** dimtruck is now known as zz_dimtruck | 00:40 | |
| *** gianpietro has quit IRC | 00:41 | |
| *** fragatina has quit IRC | 00:41 | |
| *** crushil has joined #openstack-nova | 00:41 | |
| *** zz_dimtruck is now known as dimtruck | 00:42 | |
| *** fragatin_ has quit IRC | 00:44 | |
| *** crushil has quit IRC | 00:50 | |
| *** dixiaoli has joined #openstack-nova | 00:51 | |
| *** hieulq has joined #openstack-nova | 00:51 | |
| *** ijw has quit IRC | 00:54 | |
| *** tjones has quit IRC | 00:54 | |
| armax | mriedem: ping | 00:55 |
| mriedem | armax: pong | 00:55 |
| armax | quick question | 00:55 |
| armax | since I know how much you love numa and sriov code in nova | 00:55 |
| armax | do you happen to know anything about this bug https://bugs.launchpad.net/mos/+bug/1658070? | 00:56 |
| openstack | Launchpad bug 1658070 in Mirantis OpenStack "Failed SR_IOV evacuation with host" [Medium,Confirmed] - Assigned to Sergey Nikitin (snikitin) | 00:56 |
| armax | apparently it’s targeted to MOS | 00:56 |
| armax | but I fear that’s plainly affecting nova | 00:56 |
| *** tbachman has quit IRC | 00:56 | |
| mriedem | hmm, doesn't ring a bell, but i bet cfriesen might | 00:56 |
| mriedem | i think snikitin is off openstack now too | 00:57 |
| armax | ok thanks | 00:57 |
| armax | I’ll poke around | 00:57 |
| *** thorst has joined #openstack-nova | 00:57 | |
| mriedem | doesn't mention which release either | 00:58 |
| mriedem | mitaka i guess? | 00:58 |
| *** diablo_rojo has joined #openstack-nova | 01:01 | |
| mriedem | armax: hmm https://github.com/openstack/nova/commit/dfdae01bd03f522ffab7876b253ec41641934702 | 01:01 |
| mriedem | ^ went back to newton | 01:01 |
| armax | I see | 01:02 |
| *** trungnv has joined #openstack-nova | 01:02 | |
| armax | that’s good to know | 01:02 |
| mriedem | but that bug was around awhile | 01:02 |
| mriedem | i remember | 01:02 |
| mriedem | it was blocking migration with pci | 01:02 |
| mriedem | https://bugs.launchpad.net/nova/+bug/1512880 | 01:03 |
| openstack | Launchpad bug 1512880 in OpenStack Compute (nova) newton "Failed cold migration with SR-IOV" [Medium,Fix released] | 01:03 |
| armax | that seems like it’s a slightly different use case | 01:03 |
| *** tbachman has joined #openstack-nova | 01:03 | |
| mriedem | maybe | 01:04 |
| mriedem | evacuate and migrate are pretty similar though | 01:04 |
| armax | yeah | 01:04 |
| mriedem | could also be some stuff in the resource tracker in the compute, but the bug report said that if they specify a host it fails, but if they don't it works | 01:04 |
| armax | perhaps bug https://bugs.launchpad.net/mos/+bug/1658070 is newton? | 01:04 |
| openstack | Launchpad bug 1658070 in Mirantis OpenStack "Failed SR_IOV evacuation with host" [Medium,Confirmed] | 01:04 |
| mriedem | moshele and cfriesen are definitely the guys to ask | 01:04 |
| armax | as the workaround says to let nova figure out the host | 01:05 |
| mriedem | yeah that part confuses me | 01:05 |
| mriedem | unless when you don't specify a host, the scheduler picks the right how to migrate to | 01:05 |
| mriedem | *host | 01:05 |
| armax | right, though we’d have to figure out whether all hosts are equal or the failure is genuine | 01:06 |
| *** itlinux has quit IRC | 01:07 | |
| armax | thanks for the feedback, I’ll try to get confirm whether this is an issue still affecting master | 01:07 |
| mriedem | sure, if all else fails moshele in the morning (he's in israel i think) | 01:08 |
| mriedem | cfriesen has no excuse for not being around right now working at all hours, he's in canada | 01:08 |
| *** harlowja has quit IRC | 01:08 | |
| artom | Yeah, our country's so big we encompass all timezones, and therefore work around the clock | 01:11 |
| mriedem | https://review.openstack.org/#/q/topic:bug/1512880+status:merged+branch:master | 01:11 |
| mriedem | armax: ^ was really a whole series of changes to get cold migration working with pci devices | 01:11 |
| mriedem | artom: amen | 01:11 |
| armax | mriedem: so I guess we’ll have to see whether the issue exists on master or not | 01:12 |
| mriedem | i wouldn't be surprised if it does, evacuate isn't integration tested | 01:13 |
| mriedem | anything with pci/sriov is not tested heavily, it's all 3rd party and a limited scope i think | 01:13 |
| armax | yup | 01:13 |
| armax | makes sense | 01:13 |
| mriedem | but the mellanox guys were running cold migration with sriov in their ci i think | 01:14 |
| mriedem | but we can't test evacuate in tempest since you have to take down the single compute service | 01:14 |
| armax | yeah, that’s not really reasonable on a continuous basis | 01:14 |
| artom | Surely we can hack something around that... | 01:14 |
| artom | But maybe it's just my late night brain talking | 01:15 |
| mriedem | artom: yeah we've talked about ways to hack that | 01:15 |
| mriedem | could be in the multinode job but you have to basically lock all other tests from running at the same time | 01:15 |
| mriedem | would be best if we just had some integration tests to run serially in-tree | 01:15 |
| artom | Nothing in evacuate actually depends on the compute being down, no? | 01:16 |
| *** thorst has quit IRC | 01:16 | |
| *** itlinux has joined #openstack-nova | 01:16 | |
| mriedem | yes, the api :) | 01:16 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4082 | 01:16 |
| artom | Right... we could, err, jiggle that check a bit ;) | 01:16 |
| artom | It's extremely ugly | 01:16 |
| *** gjayavelu has quit IRC | 01:17 | |
| artom | But it would allow us to run evacuations tests in parallel | 01:17 |
| mriedem | artom: for the last time i wish you'd stop talking about jiggling things around me | 01:17 |
| artom | I made the conscious effort of saying jiggle and not diddle, you should thank me | 01:17 |
| mriedem | i'm sensitive because my wife said i looked apple shaped after i got back from boston | 01:17 |
| mriedem | those are nowhere near as bad as what that tubaguy says constantly | 01:18 |
| mriedem | which i shall not repeat | 01:18 |
| artom | Can Nova know it's running in CI? | 01:18 |
| mriedem | probably shouldn't, | 01:18 |
| mriedem | i wouldn't do that for tempest | 01:19 |
| mriedem | we could probably do that with in-tree functional tests | 01:19 |
| artom | I said it was gonna be ugly :/ | 01:19 |
| mriedem | where you can have multiple compute services running | 01:19 |
| artom | That seems like a whole lot of scaffolding | 01:19 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py | 01:20 |
| mriedem | ^ is an example of a multi-compute functional test | 01:20 |
| mriedem | to test retries in conductor | 01:20 |
| armax | this use case probably falls in the category of issues that are two expensive to test on a continuous basis | 01:20 |
| armax | two -> too | 01:21 |
| mriedem | we test live migration and cold migration and resize on a continuous basis | 01:21 |
| mriedem | anything involving snapshots would probably take more time than this | 01:22 |
| mriedem | evacuate is just rebuild on another host | 01:22 |
| *** Daisy has joined #openstack-nova | 01:22 | |
| mriedem | idk, you could probably test most of the functionality through the entire stack with what's setup in https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py | 01:22 |
| *** gyee has quit IRC | 01:22 | |
| mriedem | anything external like neutron and cinder are faked out, but... | 01:22 |
| mriedem | anywho, i need to go spend time with http://www.adultswim.com/videos/your-pretty-face-is-going-to-hell/ | 01:23 |
| armax | mriedem: have fun and thanks for the feedback | 01:24 |
| *** cNilesh has joined #openstack-nova | 01:24 | |
| artom | Whoa, I parsed 'adultswim' waaay wrong | 01:24 |
| *** gianpietro has joined #openstack-nova | 01:24 | |
| *** oomichi has quit IRC | 01:25 | |
| *** salv-orlando has quit IRC | 01:25 | |
| mriedem | i highly recommend http://www.adultswim.com/videos/your-pretty-face-is-going-to-hell/hammerman/ | 01:25 |
| *** oomichi has joined #openstack-nova | 01:25 | |
| *** gyee has joined #openstack-nova | 01:26 | |
| *** tovin07_ has joined #openstack-nova | 01:26 | |
| *** dave-mccowan has quit IRC | 01:28 | |
| *** Apoorva_ has joined #openstack-nova | 01:29 | |
| *** gianpietro has quit IRC | 01:29 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Use plain routes list for server-password endpoint instead of stevedore https://review.openstack.org/462049 | 01:30 |
| *** dixiaoli has quit IRC | 01:30 | |
| *** thorst has joined #openstack-nova | 01:31 | |
| *** Apoorva has quit IRC | 01:32 | |
| *** Apoorva_ has quit IRC | 01:33 | |
| *** thorst has quit IRC | 01:33 | |
| *** Kevin_Zheng has joined #openstack-nova | 01:36 | |
| *** gianpietro has joined #openstack-nova | 01:37 | |
| *** suresh12 has quit IRC | 01:41 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Use plain routes list for server-password endpoint instead of stevedore https://review.openstack.org/462049 | 01:41 |
| *** Daisy has quit IRC | 01:42 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | 01:42 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | 01:42 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | 01:42 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | 01:42 |
| *** Daisy has joined #openstack-nova | 01:42 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | 01:42 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | 01:42 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | 01:42 |
| *** gcb has joined #openstack-nova | 01:43 | |
| *** gianpietro has quit IRC | 01:45 | |
| *** Daisy has quit IRC | 01:46 | |
| *** gongysh has joined #openstack-nova | 01:47 | |
| *** Daisy has joined #openstack-nova | 01:51 | |
| *** itlinux has quit IRC | 01:51 | |
| *** dave-mccowan has joined #openstack-nova | 01:52 | |
| *** Sukhdev has joined #openstack-nova | 01:55 | |
| *** coreywright has quit IRC | 01:55 | |
| *** Sukhdev has quit IRC | 01:56 | |
| *** itlinux has joined #openstack-nova | 02:00 | |
| *** tjones has joined #openstack-nova | 02:00 | |
| *** takashin has quit IRC | 02:04 | |
| *** suresh12 has joined #openstack-nova | 02:04 | |
| *** moshele has joined #openstack-nova | 02:09 | |
| *** suresh12 has quit IRC | 02:09 | |
| *** slaweq has joined #openstack-nova | 02:10 | |
| *** smatzek has joined #openstack-nova | 02:10 | |
| *** coreywright has joined #openstack-nova | 02:14 | |
| *** slaweq has quit IRC | 02:15 | |
| *** yamahata has quit IRC | 02:16 | |
| *** thorst has joined #openstack-nova | 02:18 | |
| *** thorst has quit IRC | 02:19 | |
| *** gianpietro has joined #openstack-nova | 02:21 | |
| *** tjones has quit IRC | 02:21 | |
| *** bkopilov_ has quit IRC | 02:21 | |
| *** bkopilov has quit IRC | 02:22 | |
| *** smatzek has quit IRC | 02:25 | |
| *** gianpietro has quit IRC | 02:25 | |
| *** itlinux has quit IRC | 02:25 | |
| *** jaypipes has joined #openstack-nova | 02:27 | |
| *** thorst has joined #openstack-nova | 02:29 | |
| *** thorst has quit IRC | 02:29 | |
| openstackgerrit | Huan Xie proposed openstack/nova master: WIP: XenAPI use os-xenapi v2 in nova https://review.openstack.org/453493 | 02:30 |
| *** tbachman has quit IRC | 02:32 | |
| *** gyee has quit IRC | 02:35 | |
| *** itlinux has joined #openstack-nova | 02:36 | |
| *** moshele has quit IRC | 02:37 | |
| *** moshele has joined #openstack-nova | 02:37 | |
| *** liusheng has quit IRC | 02:39 | |
| *** jaypipes has quit IRC | 02:40 | |
| *** fragatina has joined #openstack-nova | 02:41 | |
| *** dave-mccowan has quit IRC | 02:42 | |
| *** Shunli has joined #openstack-nova | 02:44 | |
| *** fragatina has quit IRC | 02:45 | |
| *** itlinux has quit IRC | 02:48 | |
| *** sree has joined #openstack-nova | 02:49 | |
| *** nicolasbock has quit IRC | 02:51 | |
| *** sree has quit IRC | 02:53 | |
| *** xinliang has quit IRC | 02:54 | |
| *** abhi89 has joined #openstack-nova | 02:54 | |
| *** tjones has joined #openstack-nova | 02:54 | |
| *** crushil has joined #openstack-nova | 02:55 | |
| *** tbachman has joined #openstack-nova | 02:56 | |
| *** fragatina has joined #openstack-nova | 02:56 | |
| *** hongbin has joined #openstack-nova | 02:57 | |
| *** fragatina has quit IRC | 02:57 | |
| *** itlinux has joined #openstack-nova | 02:57 | |
| *** fragatina has joined #openstack-nova | 02:58 | |
| *** diablo_rojo has quit IRC | 02:59 | |
| *** abhishek has joined #openstack-nova | 03:00 | |
| *** thorst has joined #openstack-nova | 03:00 | |
| *** tjones has quit IRC | 03:03 | |
| *** abhi89 has quit IRC | 03:03 | |
| *** xinliang has joined #openstack-nova | 03:06 | |
| *** abhishek has quit IRC | 03:06 | |
| *** abhi89 has joined #openstack-nova | 03:07 | |
| *** ociuhandu has joined #openstack-nova | 03:08 | |
| *** ociuhandu has quit IRC | 03:10 | |
| *** liusheng_ has joined #openstack-nova | 03:12 | |
| *** oomichi has quit IRC | 03:14 | |
| *** ociuhandu has joined #openstack-nova | 03:15 | |
| *** thorst has quit IRC | 03:18 | |
| *** oomichi has joined #openstack-nova | 03:19 | |
| *** itlinux has quit IRC | 03:24 | |
| *** fragatina has quit IRC | 03:24 | |
| *** hongbin has quit IRC | 03:24 | |
| *** Sukhdev has joined #openstack-nova | 03:27 | |
| *** crushil has quit IRC | 03:31 | |
| *** Daisy has quit IRC | 03:34 | |
| *** Daisy has joined #openstack-nova | 03:34 | |
| *** slaweq has joined #openstack-nova | 03:38 | |
| *** Daisy has quit IRC | 03:39 | |
| *** bkopilov_ has joined #openstack-nova | 03:39 | |
| *** dikonoor has joined #openstack-nova | 03:42 | |
| *** bkopilov has joined #openstack-nova | 03:42 | |
| *** links has joined #openstack-nova | 03:42 | |
| *** kfarr has quit IRC | 03:43 | |
| *** slaweq has quit IRC | 03:43 | |
| *** takashin has joined #openstack-nova | 03:44 | |
| *** zsli_ has joined #openstack-nova | 03:49 | |
| *** links has quit IRC | 03:51 | |
| *** Shunli has quit IRC | 03:52 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/465283 | 03:53 |
| *** links has joined #openstack-nova | 03:54 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/451049 | 03:56 |
| *** gouthamr has quit IRC | 03:57 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/465371 | 03:58 |
| *** deepak has quit IRC | 03:58 | |
| *** vks1 has joined #openstack-nova | 04:01 | |
| *** abhi89 has quit IRC | 04:04 | |
| *** xinliang has quit IRC | 04:04 | |
| *** xinliang has joined #openstack-nova | 04:05 | |
| *** suresh12 has joined #openstack-nova | 04:07 | |
| *** gomarivera_ has joined #openstack-nova | 04:13 | |
| *** chyka has quit IRC | 04:13 | |
| *** chyka has joined #openstack-nova | 04:15 | |
| *** thorst has joined #openstack-nova | 04:15 | |
| *** gongysh has quit IRC | 04:15 | |
| *** chyka has quit IRC | 04:19 | |
| *** thorst has quit IRC | 04:20 | |
| *** tbachman has quit IRC | 04:23 | |
| *** moshele has quit IRC | 04:24 | |
| *** links has quit IRC | 04:24 | |
| *** dikonoor has quit IRC | 04:32 | |
| *** adisky_ has joined #openstack-nova | 04:32 | |
| *** harlowja has joined #openstack-nova | 04:33 | |
| *** vancsics has quit IRC | 04:36 | |
| *** ayogi has joined #openstack-nova | 04:37 | |
| *** dikonoor has joined #openstack-nova | 04:50 | |
| *** Daisy has joined #openstack-nova | 04:52 | |
| *** gongysh has joined #openstack-nova | 04:53 | |
| *** prateek has joined #openstack-nova | 04:54 | |
| *** Daisy_ has joined #openstack-nova | 04:56 | |
| *** Daisy has quit IRC | 04:56 | |
| *** ociuhandu has quit IRC | 05:00 | |
| *** links has joined #openstack-nova | 05:06 | |
| *** suresh12 has quit IRC | 05:11 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy description for 'os-hide-server-addresses' https://review.openstack.org/461728 | 05:16 |
| *** thorst has joined #openstack-nova | 05:16 | |
| *** abhi89 has joined #openstack-nova | 05:17 | |
| *** trinaths has joined #openstack-nova | 05:19 | |
| *** abhishek has joined #openstack-nova | 05:20 | |
| *** gjayavelu has joined #openstack-nova | 05:20 | |
| openstackgerrit | jichenjc proposed openstack/nova-specs master: WIP: add z/VM Driver https://review.openstack.org/464915 | 05:21 |
| *** thorst has quit IRC | 05:21 | |
| *** abhishek has quit IRC | 05:22 | |
| *** Sukhdev has quit IRC | 05:23 | |
| *** abhi89 has quit IRC | 05:23 | |
| *** dikonoor has quit IRC | 05:26 | |
| *** yamahata has joined #openstack-nova | 05:28 | |
| *** sree has joined #openstack-nova | 05:29 | |
| *** ratailor has joined #openstack-nova | 05:29 | |
| openstackgerrit | Luo Gangyi proposed openstack/nova master: convert unicode to string before we connect to rados https://review.openstack.org/465404 | 05:32 |
| *** salv-orlando has joined #openstack-nova | 05:32 | |
| *** armax has quit IRC | 05:40 | |
| *** dikonoor has joined #openstack-nova | 05:43 | |
| *** litao__ has joined #openstack-nova | 05:47 | |
| *** pcaruana has joined #openstack-nova | 05:48 | |
| openstackgerrit | Songming Yan proposed openstack/nova-specs master: Sriov-bond https://review.openstack.org/463526 | 05:49 |
| openstackgerrit | Songming Yan proposed openstack/nova-specs master: Sriov-bond https://review.openstack.org/463526 | 05:49 |
| *** sridharg has joined #openstack-nova | 05:53 | |
| *** harlowja has quit IRC | 05:54 | |
| *** ralonsoh has joined #openstack-nova | 05:58 | |
| *** fragatina has joined #openstack-nova | 05:58 | |
| *** liusheng_ has quit IRC | 06:12 | |
| *** liusheng_ has joined #openstack-nova | 06:13 | |
| *** Oku_OS-away is now known as Oku_OS | 06:13 | |
| *** fragatina has quit IRC | 06:15 | |
| *** thorst has joined #openstack-nova | 06:17 | |
| *** Daisy_ has quit IRC | 06:17 | |
| *** cNileshc has joined #openstack-nova | 06:17 | |
| *** Daisy has joined #openstack-nova | 06:18 | |
| *** rcernin has joined #openstack-nova | 06:18 | |
| *** cNilesh has quit IRC | 06:19 | |
| *** Daisy has quit IRC | 06:22 | |
| *** thorst has quit IRC | 06:22 | |
| *** markus_z has joined #openstack-nova | 06:22 | |
| *** Daisy has joined #openstack-nova | 06:28 | |
| *** gongysh has quit IRC | 06:31 | |
| *** Daisy has quit IRC | 06:32 | |
| *** liusheng_ has quit IRC | 06:34 | |
| *** Daisy has joined #openstack-nova | 06:35 | |
| openstackgerrit | Wang Shilong proposed openstack/nova master: Lustre support https://review.openstack.org/446288 | 06:36 |
| *** gongysh has joined #openstack-nova | 06:43 | |
| *** satyar has joined #openstack-nova | 06:48 | |
| *** belmoreira has joined #openstack-nova | 06:51 | |
| *** mgagne has quit IRC | 06:53 | |
| *** mgagne has joined #openstack-nova | 06:56 | |
| *** mgagne is now known as Guest24103 | 06:56 | |
| *** abalutoiu has quit IRC | 06:57 | |
| alex_xu | johnthetubaguy: gmann , I'm at bug smash, and there is event at evening, not sure I can back to the hotel in time. If i'm late on the api meeting, would you mind to run the api meeting first? | 06:57 |
| *** rcernin has quit IRC | 06:57 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Example verification for servers-actions.inc https://review.openstack.org/454565 | 07:00 |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Catch exception.OverQuota when create image for volume backed instance https://review.openstack.org/464548 | 07:03 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy description for 'limits' https://review.openstack.org/465426 | 07:03 |
| *** salv-orlando has quit IRC | 07:09 | |
| *** ltomasbo|away is now known as ltomasbo | 07:10 | |
| *** mdnadeem has joined #openstack-nova | 07:11 | |
| *** suresh12 has joined #openstack-nova | 07:11 | |
| *** hieulq_ has joined #openstack-nova | 07:12 | |
| *** hieulq_ has quit IRC | 07:13 | |
| *** rcernin has joined #openstack-nova | 07:13 | |
| *** tetsuro has joined #openstack-nova | 07:13 | |
| *** danpawlik_absent is now known as dpawlik | 07:14 | |
| *** tetsuro has quit IRC | 07:14 | |
| *** karimb has joined #openstack-nova | 07:15 | |
| *** suresh12 has quit IRC | 07:16 | |
| *** thorst has joined #openstack-nova | 07:18 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy description for multinic https://review.openstack.org/465433 | 07:19 |
| *** thorst has quit IRC | 07:22 | |
| *** sridharg has quit IRC | 07:23 | |
| *** cfriesen has quit IRC | 07:26 | |
| openstackgerrit | fengzhr proposed openstack/nova master: Modify the description of flat_injected in nova.conf https://review.openstack.org/458789 | 07:26 |
| *** mriedem has quit IRC | 07:27 | |
| *** burt has quit IRC | 07:27 | |
| *** burt has joined #openstack-nova | 07:28 | |
| *** rcernin has quit IRC | 07:29 | |
| *** sridharg has joined #openstack-nova | 07:30 | |
| *** prateek is now known as prateek|lunch | 07:32 | |
| *** rcernin has joined #openstack-nova | 07:35 | |
| *** rcernin has quit IRC | 07:35 | |
| *** rcernin has joined #openstack-nova | 07:35 | |
| *** jpena|off is now known as jpena | 07:36 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy description for server_usage https://review.openstack.org/465437 | 07:42 |
| *** hshiina has joined #openstack-nova | 07:48 | |
| *** jaosorior_away is now known as jaosorior | 07:48 | |
| *** Daisy has quit IRC | 07:50 | |
| *** Daisy has joined #openstack-nova | 07:50 | |
| *** prateek|lunch is now known as prateek | 07:53 | |
| *** lpetrut has joined #openstack-nova | 07:54 | |
| *** Daisy has quit IRC | 07:54 | |
| *** gongysh has quit IRC | 07:56 | |
| *** damien_r has joined #openstack-nova | 07:58 | |
| *** damien_r has quit IRC | 07:58 | |
| openstackgerrit | Luo Gangyi proposed openstack/nova master: convert unicode to string before we connect to rados https://review.openstack.org/465404 | 07:59 |
| *** zzzeek has quit IRC | 08:00 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy description for networks_associate https://review.openstack.org/465443 | 08:00 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: placement: test for agg association not sharing https://review.openstack.org/461901 | 08:03 |
| *** sridharg has quit IRC | 08:03 | |
| *** zzzeek has joined #openstack-nova | 08:04 | |
| *** djohnsto has joined #openstack-nova | 08:04 | |
| *** mlakat has joined #openstack-nova | 08:04 | |
| *** dimtruck is now known as zz_dimtruck | 08:08 | |
| *** salv-orlando has joined #openstack-nova | 08:10 | |
| *** Daisy has joined #openstack-nova | 08:10 | |
| *** gszasz has joined #openstack-nova | 08:12 | |
| *** lucas-afk is now known as lucasagomes | 08:13 | |
| *** salv-orlando has quit IRC | 08:14 | |
| johnthetubaguy | alex_xu: no worries, I will try be around for the meeting | 08:14 |
| alex_xu | johnthetubaguy: thanks! | 08:15 |
| *** karimb has quit IRC | 08:16 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy description for os-networks https://review.openstack.org/465452 | 08:17 |
| *** trinaths has quit IRC | 08:18 | |
| *** thorst has joined #openstack-nova | 08:18 | |
| *** yamahata has quit IRC | 08:20 | |
| *** thorst has quit IRC | 08:23 | |
| *** dikonoo has joined #openstack-nova | 08:24 | |
| *** dikonoor has quit IRC | 08:24 | |
| *** Drankis has joined #openstack-nova | 08:27 | |
| *** gszasz has quit IRC | 08:27 | |
| *** udesale has joined #openstack-nova | 08:27 | |
| *** derekh has joined #openstack-nova | 08:30 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy description for os-host https://review.openstack.org/465458 | 08:31 |
| gmann | alex_xu: sure, i will also be there | 08:31 |
| alex_xu | gmann: thanks a lot | 08:31 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy description for os-fixed-ips https://review.openstack.org/461724 | 08:32 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add policy description for 'os-hide-server-addresses' https://review.openstack.org/461728 | 08:34 |
| *** gongysh has joined #openstack-nova | 08:34 | |
| gmann | alex_xu: johnthetubaguy ^^ with this policy doc are finished : https://review.openstack.org/#/q/topic:bp/policy-docs+status:open+project:+openstack/nova | 08:35 |
| johnthetubaguy | gmann: superstar, I should take a look at those soon | 08:35 |
| gmann | johnthetubaguy: thanks | 08:36 |
| *** trinaths has joined #openstack-nova | 08:36 | |
| alex_xu | gmann: yea, superstar :) | 08:38 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Use VIR_DOMAIN_BLOCK_REBASE_COPY_DEV when rebasing https://review.openstack.org/465205 | 08:39 |
| *** karimb has joined #openstack-nova | 08:39 | |
| *** gjayavelu has quit IRC | 08:39 | |
| artom | kashyap, mind taking a look at ^^ when you have a moment? You offered help with the downstream bug ) | 08:40 |
| artom | ;) | 08:40 |
| *** trinaths has left #openstack-nova | 08:49 | |
| *** aarefiev_afk is now known as aarefiev | 08:51 | |
| *** karimb has quit IRC | 09:00 | |
| *** salv-orlando has joined #openstack-nova | 09:00 | |
| *** takashin has quit IRC | 09:02 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Support tag instances when boot(3/4) https://review.openstack.org/459593 | 09:03 |
| *** gszasz has joined #openstack-nova | 09:03 | |
| *** Daisy has quit IRC | 09:04 | |
| *** Daisy has joined #openstack-nova | 09:04 | |
| *** ralonsoh has quit IRC | 09:04 | |
| *** ralonsoh_ has joined #openstack-nova | 09:04 | |
| *** salv-orlando has quit IRC | 09:06 | |
| *** satyar has quit IRC | 09:06 | |
| *** karimb has joined #openstack-nova | 09:07 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Catch exception.OverQuota when create image for volume backed instance https://review.openstack.org/464548 | 09:09 |
| *** djohnsto has quit IRC | 09:11 | |
| *** gongysh has quit IRC | 09:14 | |
| *** gongysh has joined #openstack-nova | 09:15 | |
| *** thorst has joined #openstack-nova | 09:19 | |
| openstackgerrit | Tao Li proposed openstack/nova master: Fix the display of updated_at time when using memcache driver. https://review.openstack.org/414463 | 09:23 |
| *** gcb has quit IRC | 09:27 | |
| *** karimb has quit IRC | 09:27 | |
| *** zsli_ has quit IRC | 09:30 | |
| *** karimb has joined #openstack-nova | 09:33 | |
| *** markus_z has quit IRC | 09:33 | |
| *** yingjun has quit IRC | 09:38 | |
| *** thorst has quit IRC | 09:39 | |
| openstackgerrit | Béla Vancsics proposed openstack/nova master: [WIP] Transform keypair.delete notification https://review.openstack.org/463946 | 09:40 |
| openstackgerrit | Ferenc Horváth proposed openstack/nova master: Add AggregateMultiTenancyIsolation filter tests https://review.openstack.org/464002 | 09:41 |
| openstackgerrit | Alex Szarka proposed openstack/nova master: Transform instance.resize.error notifications https://review.openstack.org/417882 | 09:47 |
| *** links has quit IRC | 09:48 | |
| *** Daisy has quit IRC | 09:49 | |
| *** markus_z has joined #openstack-nova | 09:50 | |
| *** tbachman has joined #openstack-nova | 09:53 | |
| *** tovin07_ has quit IRC | 09:56 | |
| *** slaweq has joined #openstack-nova | 09:56 | |
| *** yamamoto has quit IRC | 09:56 | |
| *** markus_z has quit IRC | 09:57 | |
| *** yingjun has joined #openstack-nova | 09:58 | |
| *** links has joined #openstack-nova | 10:00 | |
| openstackgerrit | jichenjc proposed openstack/nova-specs master: WIP: add z/VM Driver https://review.openstack.org/464915 | 10:01 |
| *** GnomeKing has quit IRC | 10:01 | |
| *** slaweq has quit IRC | 10:01 | |
| *** johnthetubaguy has left #openstack-nova | 10:06 | |
| *** cNilesh has joined #openstack-nova | 10:06 | |
| *** cNileshc has quit IRC | 10:09 | |
| *** salv-orlando has joined #openstack-nova | 10:10 | |
| *** suresh12 has joined #openstack-nova | 10:12 | |
| *** dikonoo has quit IRC | 10:12 | |
| *** faizy has joined #openstack-nova | 10:14 | |
| *** salv-orlando has quit IRC | 10:15 | |
| *** Daviey_ is now known as Daviey | 10:15 | |
| *** markus_z has joined #openstack-nova | 10:16 | |
| *** suresh12 has quit IRC | 10:16 | |
| *** cNilesh has quit IRC | 10:19 | |
| *** sdague has joined #openstack-nova | 10:29 | |
| *** hferenc has quit IRC | 10:32 | |
| *** slaweq has joined #openstack-nova | 10:32 | |
| *** thorst has joined #openstack-nova | 10:35 | |
| *** slaweq has quit IRC | 10:37 | |
| *** bkopilov_ has quit IRC | 10:39 | |
| *** bkopilov has quit IRC | 10:39 | |
| *** thorst has quit IRC | 10:40 | |
| *** yingjun has quit IRC | 10:43 | |
| *** yamamoto has joined #openstack-nova | 10:45 | |
| *** moshele has joined #openstack-nova | 10:46 | |
| openstackgerrit | Sandeep Pawar proposed openstack/nova master: Libvirt volume driver for Veritas HyperScale https://review.openstack.org/443951 | 10:52 |
| *** ayogi has quit IRC | 10:52 | |
| -openstackstatus- NOTICE: gerrit is being restarted to help stuck git replication issues | 10:54 | |
| *** nicolasbock has joined #openstack-nova | 10:54 | |
| openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Remove the `--volume-service-name` option https://review.openstack.org/465502 | 11:01 |
| *** links has quit IRC | 11:05 | |
| *** gszasz has quit IRC | 11:05 | |
| *** gomarivera_ has quit IRC | 11:05 | |
| openstackgerrit | jichenjc proposed openstack/nova master: WIP: remove flavor policy https://review.openstack.org/465505 | 11:07 |
| *** salv-orlando has joined #openstack-nova | 11:11 | |
| *** smatzek has joined #openstack-nova | 11:12 | |
| *** hferenc has joined #openstack-nova | 11:15 | |
| *** salv-orlando has quit IRC | 11:16 | |
| *** gongysh has quit IRC | 11:18 | |
| *** gongysh has joined #openstack-nova | 11:18 | |
| *** rfolco has joined #openstack-nova | 11:19 | |
| *** gongysh has quit IRC | 11:19 | |
| *** cdent has joined #openstack-nova | 11:19 | |
| *** gongysh has joined #openstack-nova | 11:19 | |
| openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Remove --cell_name, --tenant options https://review.openstack.org/465509 | 11:19 |
| *** gongysh has quit IRC | 11:19 | |
| openstackgerrit | jichenjc proposed openstack/nova master: WIP: Remove os_compute_api:os-server-groups policy https://review.openstack.org/465510 | 11:19 |
| *** gongysh has joined #openstack-nova | 11:20 | |
| *** gongysh has quit IRC | 11:20 | |
| *** links has joined #openstack-nova | 11:22 | |
| *** gianpietro has joined #openstack-nova | 11:22 | |
| openstackgerrit | jichenjc proposed openstack/nova master: WIP: remove flavor policy https://review.openstack.org/465505 | 11:23 |
| *** gianpietro has quit IRC | 11:27 | |
| *** TTris has joined #openstack-nova | 11:31 | |
| *** sean-k-mooney has quit IRC | 11:32 | |
| *** Kevin_Zheng has quit IRC | 11:36 | |
| *** zul has quit IRC | 11:39 | |
| openstackgerrit | Alex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification https://review.openstack.org/403660 | 11:40 |
| *** zul has joined #openstack-nova | 11:41 | |
| *** karimb has quit IRC | 11:44 | |
| *** abalutoiu has joined #openstack-nova | 11:44 | |
| *** Jack_Iv has joined #openstack-nova | 11:46 | |
| *** thorst has joined #openstack-nova | 11:46 | |
| openstackgerrit | Andrey Volkov proposed openstack/nova master: PoC: Select PCI devices with distinct tag values https://review.openstack.org/448008 | 11:48 |
| *** gongysh has joined #openstack-nova | 11:49 | |
| *** litao__ has quit IRC | 11:50 | |
| *** gianpietro has joined #openstack-nova | 11:50 | |
| *** udesale has quit IRC | 11:52 | |
| *** yolkfull has quit IRC | 11:52 | |
| openstackgerrit | Alex Szarka proposed openstack/nova master: Add method for verify multiple notifications https://review.openstack.org/465526 | 11:53 |
| *** tbachman has quit IRC | 11:54 | |
| *** gianpietro has quit IRC | 11:56 | |
| openstackgerrit | Alex Szarka proposed openstack/nova master: Add method for verify multiple notifications https://review.openstack.org/465526 | 11:57 |
| *** claudiub has joined #openstack-nova | 11:57 | |
| *** gongysh has quit IRC | 11:58 | |
| *** lucasagomes is now known as lucas-hungry | 11:59 | |
| *** Alex______ has joined #openstack-nova | 11:59 | |
| *** sean-k-mooney has joined #openstack-nova | 12:02 | |
| *** ratailor has quit IRC | 12:03 | |
| openstackgerrit | jichenjc proposed openstack/nova master: WIP: remove flavor policy https://review.openstack.org/465505 | 12:07 |
| *** Daisy has joined #openstack-nova | 12:07 | |
| *** gianpietro has joined #openstack-nova | 12:10 | |
| *** ericyoung has quit IRC | 12:11 | |
| *** Daisy has quit IRC | 12:12 | |
| *** vks1 has quit IRC | 12:15 | |
| *** claudiub has quit IRC | 12:15 | |
| *** jwcroppe has quit IRC | 12:16 | |
| *** jwcroppe has joined #openstack-nova | 12:16 | |
| *** mriedem has joined #openstack-nova | 12:17 | |
| *** mriedem has quit IRC | 12:18 | |
| *** mriedem has joined #openstack-nova | 12:18 | |
| *** liverpooler has joined #openstack-nova | 12:19 | |
| *** jwcroppe has quit IRC | 12:20 | |
| *** gszasz has joined #openstack-nova | 12:21 | |
| openstackgerrit | Alex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification https://review.openstack.org/403660 | 12:21 |
| openstackgerrit | Alex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification https://review.openstack.org/403660 | 12:22 |
| *** gianpietro has quit IRC | 12:31 | |
| *** karimb has joined #openstack-nova | 12:31 | |
| *** gianpietro has joined #openstack-nova | 12:32 | |
| *** ayogi has joined #openstack-nova | 12:33 | |
| *** Miouge has joined #openstack-nova | 12:34 | |
| *** zz_dimtruck is now known as dimtruck | 12:34 | |
| Miouge | Is there an extra spec (flavor or image) to configure <cpu mode='host-passthrough'> with libvirt? I know of cpu_mode in nova.conf but that’s for a whole compute node | 12:35 |
| *** gianpietro has quit IRC | 12:36 | |
| openstackgerrit | Alex Szarka proposed openstack/nova master: Add method for verify multiple notifications https://review.openstack.org/465526 | 12:37 |
| openstackgerrit | Alex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification https://review.openstack.org/403660 | 12:38 |
| *** gouthamr has joined #openstack-nova | 12:39 | |
| *** salv-orlando has joined #openstack-nova | 12:39 | |
| sfinucan | Miouge: not that I'm aware of, and it's not something we'd be eager to enable | 12:39 |
| artom | mriedem, I thought it was OK to continue using mox when it's already in place? | 12:40 |
| artom | I essentially copy-pasted an existing test and tweaked it a bit | 12:40 |
| *** salv-orlando has quit IRC | 12:40 | |
| *** edmondsw has joined #openstack-nova | 12:41 | |
| *** salv-orlando has joined #openstack-nova | 12:41 | |
| Miouge | sfinucan: curious to why not something eager to enable, recommended to use cpu_nova.conf and a separate host aggregate instead? | 12:42 |
| mriedem | artom: not for new tests | 12:42 |
| artom | mriedem, anyways, I think we had this discussion previously, and it was only dansmith who persuaded you to let it go, so I'll just fix my test, shall I | 12:42 |
| artom | ;) | 12:42 |
| mriedem | artom: new tests should be written in mock | 12:42 |
| mriedem | artom: using mox to tweak existing tests that are using mox is acceptable | 12:42 |
| *** mdnadeem has quit IRC | 12:43 | |
| sfinucan | Miouge: The comments in this spec should explain the position :) https://review.openstack.org/#/c/387295/ | 12:43 |
| sfinucan | ...and host aggregates would be preferred, yes | 12:44 |
| artom | mriedem, fair enough. If I ever find the discussion I'm thinking of I'll wave it in your face, but yeah, I won't be lazy and do the new test with mock ;) | 12:44 |
| mriedem | artom: i remember it, it was some nova-network live migration thing | 12:45 |
| *** bhagyashris has quit IRC | 12:45 | |
| mriedem | and i believe dansmith went back afterward and re-wrote the tests in mock | 12:45 |
| *** tbachman has joined #openstack-nova | 12:45 | |
| *** kylek3h has quit IRC | 12:46 | |
| mriedem | artom: see #4 here https://docs.openstack.org/infra/manual/developers.html#peer-review | 12:46 |
| *** dane-fichter has joined #openstack-nova | 12:46 | |
| *** jpena is now known as jpena|lunch | 12:47 | |
| *** sree has quit IRC | 12:48 | |
| *** sree has joined #openstack-nova | 12:49 | |
| *** hshiina has quit IRC | 12:50 | |
| dane-fichter | hey dansmith, could you take a look at my trusted certs patch when you get a chance? | 12:51 |
| dane-fichter | https://review.openstack.org/#/c/457711/ | 12:51 |
| dane-fichter | It's passing unit tests, not sure if there's anything else I need to implement here | 12:52 |
| sdague | dansmith / mriedem: thinking about the thing that came up in the cells forum session around searching by ip .... | 12:53 |
| sdague | it's really pathologically bad in our code | 12:53 |
| *** sree has quit IRC | 12:54 | |
| *** sree has joined #openstack-nova | 12:54 | |
| mriedem | i looked it up after the session, | 12:54 |
| mriedem | and it wasn't mysql or pg specific | 12:54 |
| sdague | nope | 12:54 |
| mriedem | it was originally just about a security bug | 12:54 |
| sdague | mriedem: what was the security bug? | 12:54 |
| mriedem | api would call down to nova-network which would pull all instances from the db and iterate them filtering on ip, something like that | 12:55 |
| sdague | ok, so removing nova-network out of the flow, we're now just loading every instance, loading every network info cache, and grepping through each json? | 12:56 |
| sdague | especially in the all_tenants case | 12:56 |
| *** lucas-hungry is now known as lucasagomes | 12:57 | |
| mriedem | https://github.com/openstack/nova/commit/ae781ee97947c33d6d43e4c21df4f338c875bf1c was the original change | 12:57 |
| *** lyan has joined #openstack-nova | 12:58 | |
| *** liangy has joined #openstack-nova | 12:58 | |
| *** lyan has quit IRC | 12:58 | |
| mriedem | sdague: not sure how you'd do the filtering in the db when the instance_info_cache.network_data is serialized json | 12:58 |
| *** xyang1 has joined #openstack-nova | 12:58 | |
| mriedem | *network_info | 12:59 |
| sdague | right, in order to make it efficient we'd need to extract the search keys that we wanted to hit into a column | 12:59 |
| *** mdnadeem has joined #openstack-nova | 12:59 | |
| sdague | which you could do, just denormalize it a bit | 12:59 |
| *** ayogi has quit IRC | 13:00 | |
| *** esberglu has joined #openstack-nova | 13:00 | |
| sdague | given that the ip search is regex, you could even just make it a comma separated list, and still do the second stage filter, but it would drop a lot of db bloat | 13:00 |
| sdague | anyway, just pondering that, given it seemed like it was a query many wanted to do, and it's one of our most pathologically bad ones | 13:01 |
| mriedem | and it's also been this way forever.. | 13:01 |
| mriedem | if we had numbers on how long it takes to do the filter on an all_tenants call with 1000 instances or something, then it might be more interesting | 13:02 |
| *** gomarivera_ has joined #openstack-nova | 13:02 | |
| sdague | mriedem: yeh data would be good | 13:02 |
| *** eharney has joined #openstack-nova | 13:03 | |
| sdague | it does also mean though if you exceed 1000 instances in your filter set before ips, you'll get the wrong answer, right? | 13:03 |
| *** bhagyashris has joined #openstack-nova | 13:03 | |
| sdague | that's actually the thing I'm more concerned about | 13:03 |
| *** johnthetubaguy has joined #openstack-nova | 13:03 | |
| *** salv-orlando has quit IRC | 13:03 | |
| mriedem | true, if the ip you're looking for is the 1001 instance, you're not going to get anything back | 13:03 |
| *** salv-orlando has joined #openstack-nova | 13:04 | |
| sdague | if nothing else, we should describe that issue | 13:04 |
| mriedem | yeah, definitely, | 13:05 |
| mriedem | could go into the api-ref | 13:05 |
| mriedem | i feel like there was something else we said we'd doc in the api-ref but i'm forgetting - i think notes are in etherpads though, and i haven't digested those yet | 13:05 |
| *** tuan_luong has joined #openstack-nova | 13:06 | |
| sdague | well, in mysql we could - https://dev.mysql.com/doc/refman/5.7/en/json-search-functions.html#function_json-contains | 13:06 |
| mriedem | ip (Optional) query string An IPv4 address to filter results by. | 13:06 |
| mriedem | ^ is the api-ref and doesn't mention anything about this | 13:06 |
| sdague | mriedem: yep, I know | 13:06 |
| sdague | I honestly didn't realize how bad it was | 13:06 |
| *** rabel has joined #openstack-nova | 13:07 | |
| *** gomarivera_ has quit IRC | 13:07 | |
| mriedem | looks like postgresql does too https://www.postgresql.org/docs/9.3/static/functions-json.html | 13:07 |
| mriedem | http://schinckel.net/2014/05/25/querying-json-in-postgres/ | 13:08 |
| *** salv-orlando has quit IRC | 13:08 | |
| rabel | Hi there, I'm a little bit confused about quotas in nova. To start with: Can anyone tell me, what a quota class is? And how are those related to default quotas? How do default quotas work in detail? For example why does a project have default quotas? I thought default quotas are global quotas that take effect for projects without quotas? | 13:09 |
| *** artom has quit IRC | 13:11 | |
| mriedem | rabel: se | 13:12 |
| mriedem | *sec | 13:12 |
| *** Jack_Iv has quit IRC | 13:12 | |
| mriedem | rabel: some notes here https://docs.openstack.org/developer/nova/quotas.html | 13:12 |
| mriedem | rabel: there is only one quota class and it's "default" | 13:12 |
| *** Jack_Iv has joined #openstack-nova | 13:12 | |
| *** Jack_Iv has quit IRC | 13:13 | |
| *** Jack_Iv has joined #openstack-nova | 13:13 | |
| mriedem | some more info here https://review.openstack.org/#/c/411035/ | 13:13 |
| *** Jack_Iv has quit IRC | 13:13 | |
| rabel | mriedem: thank you very much! do you know if other openstack components differ in this regard? | 13:14 |
| *** gianpietro has joined #openstack-nova | 13:14 | |
| *** mdrabe has joined #openstack-nova | 13:14 | |
| *** kfarr has joined #openstack-nova | 13:15 | |
| *** pchavva has joined #openstack-nova | 13:15 | |
| *** mkoderer has joined #openstack-nova | 13:17 | |
| *** marst has quit IRC | 13:18 | |
| mkoderer | mriedem: ping | 13:18 |
| *** marst has joined #openstack-nova | 13:18 | |
| bhagyashris | Hi all, Can anyone please guide me How to check the memory consumption or memory during the process initialization suppose for n-cpu service? Is there any command to check this? | 13:18 |
| *** kylek3h has joined #openstack-nova | 13:19 | |
| *** jamesdenton has joined #openstack-nova | 13:20 | |
| rabel | mriedem: what is still confusing to me is the --default option of "openstack quota show", which shows default quotas for a given project or class. that does not seem to make a lot of sense. what do you think? | 13:22 |
| *** vladikr has joined #openstack-nova | 13:22 | |
| *** baoli has joined #openstack-nova | 13:23 | |
| *** cleong has joined #openstack-nova | 13:30 | |
| mriedem | rabel: i think that's trying to match the nova quota-defaults CLI | 13:30 |
| *** sree has quit IRC | 13:31 | |
| mriedem | mkoderer: pong | 13:31 |
| mkoderer | mriedem: dgonzalez and I wanted to discuss a tricky bug with you: https://bugs.launchpad.net/nova/+bug/1691131 | 13:32 |
| openstack | Launchpad bug 1691131 in OpenStack Compute (nova) "IO stuck causes nova compute agent outage" [Undecided,New] | 13:32 |
| mkoderer | nova-compute stops sending it's heartbeats if we have a NFS overload | 13:32 |
| mkoderer | and with this we can't control the cloud anylonger since stopping of vms is not possible anylonger | 13:33 |
| mriedem | mkoderer: which release? | 13:33 |
| mkoderer | we are on liberty - but the code is the same on master | 13:33 |
| mriedem | "The nova-compute services scans over all attached volume paths" | 13:34 |
| mriedem | which part specifically are you referring to there? | 13:34 |
| rabel | mriedem: so, let me summarize: the only quota class is "default" and the concept of quota classes will probably soon be removed. default quotas are quotas that take effect if there is no project-specific quotas. and having default quotas for a project/class does not make sense. is this correct in your opinion? | 13:34 |
| mriedem | "and having default quotas for a project/class does not make sense" - why does that not make sense? | 13:35 |
| mriedem | rabel: having default quota (limits) is fine | 13:35 |
| mriedem | having an api and database for quota classes when there is only ever a single entry though, is weird | 13:36 |
| mriedem | *database table | 13:36 |
| *** thorst is now known as thorst_afk | 13:36 | |
| mkoderer | mriedem: this part blocks: https://github.com/openstack/nova/blob/902b7bf6f5425824a0661b8e4beac4a894749c03/nova/virt/libvirt/driver.py#L7022 | 13:36 |
| *** johnthetubaguy has left #openstack-nova | 13:37 | |
| dansmith | sdague: mriedem: setting the precedent of loading json in mysql seems not awesome | 13:37 |
| rabel | mriedem: default quotas do make sense. but how are they connected to a specific project? isn't the idea of default quotas that they are global? | 13:37 |
| dansmith | sdague: mriedem: I'd almost rather just build a separate index of that thing, if it's a thing people do often | 13:37 |
| dansmith | sdague: mriedem: also, it's something rather easy that a client can do right? find the port/device_id in neutron and then show the instance in nova? | 13:38 |
| *** markus_z has quit IRC | 13:39 | |
| *** cleong has quit IRC | 13:41 | |
| bauzas | dansmith: edleafe: tbc, I'm working on updating my series given dan's comments | 13:41 |
| *** mdnadeem has quit IRC | 13:41 | |
| dansmith | bauzas: sweet :) | 13:42 |
| bauzas | just has a problem with a 2-instances request | 13:42 |
| bauzas | because I need more hosts than just 4 | 13:42 |
| edleafe | bauzas: cool. I'll be pushing an update to my series, but it's just a pep8 fix, so don't freak out | 13:42 |
| sdague | dansmith: expect, neutron api isn't always exposed correctly directly to users | 13:42 |
| *** zzzeek has quit IRC | 13:43 | |
| *** diablo_rojo has joined #openstack-nova | 13:43 | |
| sdague | dansmith: yeh, secondary index would be fine as well | 13:43 |
| bauzas | edleafe: which one do you think you want to update ? the ones about passing the UUIDs and the scheduler claims? | 13:43 |
| *** jpena|lunch is now known as jpena | 13:44 | |
| edleafe | bauzas: yep | 13:44 |
| bauzas | ok | 13:44 |
| dansmith | sdague: could we also just special case that search by asking neutron ourselves and then pulling the relevant instances out ? | 13:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/465283 | 13:45 |
| sdague | dansmith: they support search by regex? | 13:45 |
| sdague | I think it gets complicated when there are multiple filters | 13:46 |
| sdague | but especially what we allow in our filters today | 13:46 |
| *** johnthetubaguy has joined #openstack-nova | 13:46 | |
| dansmith | sdague: I dunno, they'd have to for the client approach to work, but I didn't realize that searching by regex was a thing, I thought the use-case was "find me the instance with this ip" | 13:46 |
| sdague | it's definitely a regex today, which is kind of suck | 13:47 |
| sdague | but that's what it is | 13:47 |
| sdague | and has been | 13:47 |
| *** dimtruck is now known as zz_dimtruck | 13:47 | |
| *** tbachman has quit IRC | 13:47 | |
| mriedem | mkoderer: that's in a ploop block, are you using ploop? | 13:48 |
| *** mlavalle has joined #openstack-nova | 13:48 | |
| *** zzzeek has joined #openstack-nova | 13:49 | |
| mriedem | mkoderer: is there a way to detect a bad mountpoint with nfs? | 13:49 |
| *** READ10 has joined #openstack-nova | 13:50 | |
| *** lucasagomes is now known as lucas-brb | 13:52 | |
| dgonzalez | mriedem: mkoderer sent the wrong line, the correct line is https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7025 (below the loop block) | 13:53 |
| openstackgerrit | Béla Vancsics proposed openstack/nova master: Transform keypair.delete notification https://review.openstack.org/463946 | 13:53 |
| *** tbachman has joined #openstack-nova | 13:53 | |
| dgonzalez | mriedem: But both lines basically have the same content, so I guess the issue could occur in both blocks | 13:53 |
| *** kfarr has quit IRC | 13:54 | |
| mriedem | i'm going to be away for a couple of hours | 13:54 |
| *** mriedem is now known as mriedem_afk | 13:54 | |
| *** baoli_ has joined #openstack-nova | 13:55 | |
| *** Drankis has quit IRC | 13:55 | |
| *** baoli has quit IRC | 13:57 | |
| alex_xu | johnthetubaguy: sdague it would be great to merge this one first when you have free time, it can reduce some conflict in later patch. thanks in advance | 13:58 |
| sdague | alex_xu: which patch? | 13:59 |
| alex_xu | sdague: oops, https://review.openstack.org/462825 | 13:59 |
| *** crushil has joined #openstack-nova | 13:59 | |
| *** salv-orlando has joined #openstack-nova | 14:00 | |
| sdague | alex_xu: done | 14:00 |
| alex_xu | sdague: thanks! | 14:01 |
| johnthetubaguy | will take a look at that series next, looking a the policy docs first | 14:02 |
| johnthetubaguy | I also need to work out why that highlight didn't make any noise in this (franctionally less stupid) IRC client | 14:02 |
| *** rabel has quit IRC | 14:03 | |
| *** jwcroppe has joined #openstack-nova | 14:04 | |
| *** p01nt3r75 has joined #openstack-nova | 14:04 | |
| *** djohnsto has joined #openstack-nova | 14:05 | |
| *** vks1 has joined #openstack-nova | 14:06 | |
| *** rabel has joined #openstack-nova | 14:07 | |
| *** coreywright has quit IRC | 14:07 | |
| *** zz_dimtruck is now known as dimtruck | 14:08 | |
| rabel | mriedem: I was shortly disconnected. have you by chance answered my last question? | 14:10 |
| *** awaugama has joined #openstack-nova | 14:12 | |
| *** suresh12 has joined #openstack-nova | 14:12 | |
| *** itlinux has joined #openstack-nova | 14:12 | |
| *** tbachman has quit IRC | 14:14 | |
| *** suresh12 has quit IRC | 14:16 | |
| dims | sdague : Do we need to suppress this ERROR traceback? http://logs.openstack.org/58/465458/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d240f9b/logs/screen-n-sch.txt.gz#_May_17_08_54_38_996958 | 14:19 |
| *** coreywright has joined #openstack-nova | 14:20 | |
| sdague | dims: yeh, probably | 14:22 |
| *** adisky_ has quit IRC | 14:22 | |
| sdague | we should support at least a 60 second grace before we emit that | 14:22 |
| sdague | it's a service race startup | 14:22 |
| *** marst has quit IRC | 14:23 | |
| *** tbachman has joined #openstack-nova | 14:24 | |
| *** tjones has joined #openstack-nova | 14:24 | |
| *** karimb has quit IRC | 14:25 | |
| *** trinaths has joined #openstack-nova | 14:26 | |
| *** tuan_luong has quit IRC | 14:26 | |
| *** gszasz has quit IRC | 14:27 | |
| *** moshele has quit IRC | 14:28 | |
| *** jwcroppe has quit IRC | 14:28 | |
| *** udesale has joined #openstack-nova | 14:29 | |
| *** jwcroppe has joined #openstack-nova | 14:29 | |
| *** karimb has joined #openstack-nova | 14:29 | |
| *** marst has joined #openstack-nova | 14:30 | |
| *** gjayavelu has joined #openstack-nova | 14:31 | |
| *** Guest24103 is now known as mgagne | 14:32 | |
| *** mgagne has quit IRC | 14:32 | |
| *** mgagne has joined #openstack-nova | 14:32 | |
| *** yamamoto_ has joined #openstack-nova | 14:32 | |
| *** jwcroppe_ has joined #openstack-nova | 14:32 | |
| *** gyee has joined #openstack-nova | 14:33 | |
| *** lucas-brb is now known as lucasagomes | 14:33 | |
| *** jwcroppe has quit IRC | 14:34 | |
| *** yamamoto has quit IRC | 14:36 | |
| *** dane-fichter has quit IRC | 14:36 | |
| edleafe | This isn't the best forum for discussion, but I wanted some clarification about the request from ops for additional host for retries. Was this simply for Nova? I remember talking about getting rid of the dependence on max_attempts in the scheduler, because it was a nova-ism. | 14:37 |
| edleafe | Basing the number of returned hosts on max_attempts seems like we're once again linking nova to the scheduler. IOW, if max_attempts is 5, then *every* service that might ever use the scheduler will get back 5 resource providers per request, whether they need them or not. Wouldn't it be a better design to have the service calling select_destinations ask for more if they want more than 1? | 14:37 |
| edleafe | ^^ bauzas dansmith cdent jaypipes | 14:38 |
| *** dave-mccowan has joined #openstack-nova | 14:38 | |
| bauzas | edleafe: because the conductor would be cell-local | 14:38 |
| *** moshele has joined #openstack-nova | 14:39 | |
| bauzas | edleafe: so it would be an upcall which we disagree with | 14:39 |
| *** slaweq has joined #openstack-nova | 14:39 | |
| edleafe | bauzas: you're missing the point | 14:39 |
| edleafe | I'm not saying returning more than one host is a bad thing | 14:39 |
| *** itlinux has quit IRC | 14:39 | |
| edleafe | I'm asking if we want to tie the number of returned RPs to nova's needs | 14:39 |
| bauzas | no | 14:39 |
| bauzas | the RP call returns the full list | 14:40 |
| cdent | edleafe: when you say "returned RPs" do you mean placement -> scheduler or scheduler -> something else | 14:40 |
| *** armax has joined #openstack-nova | 14:40 | |
| bauzas | then the filters/weighters filters out that list | 14:40 |
| edleafe | cdent: returned from scheduler | 14:40 |
| cdent | ✔ | 14:40 |
| edleafe | cdent: I still try to think of scheduler as independent from nova | 14:41 |
| edleafe | (as a design goal) | 14:41 |
| *** artom has joined #openstack-nova | 14:41 | |
| cdent | seems a reasonable practice, although I've always assumed that while placement would always be independent, it would have as clients lots of different "things that do filtering" (including nova-scheduler) | 14:41 |
| edleafe | cdent: oh definitely for placement, since we got to create it separately | 14:42 |
| *** prateek has quit IRC | 14:42 | |
| bauzas | edleafe: no, we clearly said during Forum that while placement could be eventually independent, scheduler is a nova-ism | 14:42 |
| edleafe | I'm wondering if we're giving up on the notion of the scheduler ever being useful for anything besides nova | 14:42 |
| bauzas | edleafe: at least jay told that in his presentations | 14:43 |
| cdent | I think maintaining a strong contractual boundary is always good | 14:43 |
| edleafe | so it was decided that every service that needs scheduling will create its own? | 14:43 |
| sean-k-mooney | sfinucan: can you look at https://review.openstack.org/#/c/451049/9 the os-vif requirements file on master has conflicting with the global requirement for a while so we should probably merge this | 14:44 |
| edleafe | or just that nova doesn't want to waste cycles on making that happen | 14:44 |
| cdent | so I would think that the answer to your original question is "yes, we should be explicit" | 14:44 |
| sfinucan | sean-k-mooney: on it | 14:44 |
| bauzas | edleafe: every service could be using Placement differently | 14:44 |
| edleafe | bauzas: nobody's talking about placement | 14:45 |
| bauzas | edleafe: and scheduler could be a library by Queens | 14:45 |
| sean-k-mooney | sfinucan: cool thanks, just going true my review queue which is always fun :) | 14:45 |
| bauzas | but that's still a nova thing | 14:45 |
| edleafe | I'm talking about making a change that ties the scheduler closer to nova | 14:45 |
| edleafe | that we will decide to undo in N cycles because the winds have changed direction | 14:46 |
| *** vks1 has quit IRC | 14:47 | |
| *** burt has quit IRC | 14:48 | |
| *** burt has joined #openstack-nova | 14:48 | |
| *** slaweq has quit IRC | 14:48 | |
| *** udesale has quit IRC | 14:50 | |
| dansmith | edleafe: I think we've been on a path to keep scheduler closely tied to nova and split out placement as the generic thing, ever since placement was a thing | 14:50 |
| *** udesale has joined #openstack-nova | 14:50 | |
| dansmith | that's been my understanding | 14:50 |
| *** ociuhandu has joined #openstack-nova | 14:50 | |
| *** vks1 has joined #openstack-nova | 14:50 | |
| edleafe | dansmith: so it's OK to add more things to tie them together? | 14:53 |
| dansmith | to tie nova and the scheduler together? | 14:53 |
| dansmith | or nova and "nova's scheduler" I should say | 14:54 |
| *** udesale has quit IRC | 14:54 | |
| edleafe | I think you've answered my question | 14:54 |
| dansmith | heh | 14:54 |
| edleafe | bleh | 14:54 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: Scheduler returning alternatives per destination https://review.openstack.org/465175 | 14:54 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: Delete existing allocations when retrying https://review.openstack.org/460433 | 14:54 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Allocate instance by the conductor https://review.openstack.org/460455 | 14:54 |
| dansmith | making the claim in the scheduler is tying them closer together right? | 14:54 |
| edleafe | no | 14:54 |
| *** cfriesen has joined #openstack-nova | 14:54 | |
| dansmith | hmm, okay :) | 14:55 |
| bauzas | dansmith: edleafe: just in case you're interested in, I just updated my series | 14:55 |
| *** moshele has quit IRC | 14:55 | |
| *** links has quit IRC | 14:55 | |
| dansmith | bauzas: cool | 14:55 |
| bauzas | working now back on the conductor change now that we have the alternatives | 14:55 |
| edleafe | assuming an independent scheduler, if a service requests some resources, the scheduler would make a selection, claim the resources to ensure that nothing else uses them, and then returns the selected RP | 14:55 |
| edleafe | Nothing nova-y about it | 14:56 |
| bauzas | edleafe: again, like I said, we plan to make scheduler a library for Queens | 14:56 |
| bauzas | so it would eventually be possible for other services to use that lib | 14:56 |
| bauzas | but for the moment, consider it as a nova-ism | 14:56 |
| edleafe | bauzas: that seems kind of silly if only Nova can use it | 14:56 |
| bauzas | baby steps at least | 14:57 |
| bauzas | ie. the plan is | 14:57 |
| bauzas | 1/ claims in Pike | 14:57 |
| bauzas | 2/ scheduler as a lib in Queens | 14:57 |
| edleafe | bauzas: baby steps forward, while taking full steps back | 14:57 |
| bauzas | 3/ conductor uses that lib for filtering/weighting in Queens/Rocky | 14:57 |
| bauzas | 4/ other services could possibly be using that scheduler lib by Rocky | 14:58 |
| *** tbachman has quit IRC | 14:58 | |
| bauzas | (c) by dansmith for the library thing | 14:58 |
| bauzas | which I totally agree with | 14:58 |
| edleafe | bauzas: no they couldn't, if we start tying scheduler to nova more and more | 14:58 |
| edleafe | instead of getting rid of a nova-ism and making it part of the interface, we're now making scheduler more dependent on that nova-ism | 14:59 |
| *** salv-orlando has quit IRC | 14:59 | |
| bauzas | edleafe: that's a truism, and I don't disagree with your statement | 14:59 |
| dansmith | bauzas: no, it was johnthetubaguy's idea, I just fell in love with it | 14:59 |
| edleafe | making it even less likely that any other service will ever use scheduler | 14:59 |
| bauzas | what I disagree is with you saying that we go into that direction | 14:59 |
| *** salv-orlando has joined #openstack-nova | 15:00 | |
| bauzas | dansmith: oops, hail to johnthetubaguy then :) | 15:00 |
| edleafe | dansmith: that contradicts what you were saying earlier: that the scheduler is nova's scheduler | 15:00 |
| bauzas | anyway, back to work for updating the conductor now | 15:00 |
| edleafe | why do any of those steps if the end goal isn't to make it useful for other services? | 15:00 |
| dansmith | edleafe: I don't want it as a library for other projects to use, I want it to be nothing but code in nova (i.e. not a service) | 15:01 |
| dansmith | edleafe: because placement is the generic thing useful for other services | 15:01 |
| johnthetubaguy | placement is for everyone right, and +1 what dansmith just said | 15:01 |
| *** ociuhandu has quit IRC | 15:01 | |
| edleafe | ok, then let's tie scheduler to nova ever more tightly | 15:01 |
| cdent | library is a bit of a misnomer, I guess? We mean something more like "module"? | 15:02 |
| dansmith | cdent: yeah | 15:02 |
| edleafe | cdent: yeah, "library" means generally useful | 15:02 |
| cdent | library would be fun, but seems unlikely. Also, why would we replicate that confusing mass? | 15:02 |
| johnthetubaguy | my think was, stop making RPC API calls to it, since it the process is no longer special, but thats a different thing | 15:03 |
| edleafe | cdent: because the concept of choosing one RP over another is rather generic | 15:03 |
| dansmith | edleafe: I think we've been on this "keep scheduler in nova, split out placement" path for quite a while, but I'm not sure the distinction was made explicit and given the "split the scheduler" talk before that, you're probably not the only one confused by the "plan" | 15:03 |
| cdent | there could be a library for things that filter resource providers that is implemented outside of nova, using more clear techniques | 15:03 |
| cdent | I think that would be useful | 15:04 |
| dansmith | could be, | 15:04 |
| edleafe | cdent: sure, but there was never any work even considered for that | 15:04 |
| johnthetubaguy | cdent: yeah, thats a sepearate, quite interesting thing, that is a library | 15:04 |
| *** rcernin has quit IRC | 15:04 | |
| dansmith | and you could also build a generic scheduling library that makes it clear we should dump much of what nova's scheduler does | 15:04 |
| *** salv-orlando has quit IRC | 15:04 | |
| dansmith | and when you guys build that, we can evaluate :) | 15:04 |
| edleafe | cdent: whereas we've spent a lot of effort on cleaning up the interface between nova and the scheduler | 15:04 |
| cdent | edleafe: I still think that interface should be kept as clean and explicit as possible. That's simply good practice. | 15:05 |
| * dansmith nods | 15:05 | |
| edleafe | cdent: and if the scheduler were an independent library, it could be cleaned up radically, as long as the interface contract was maintained | 15:05 |
| * johnthetubaguy nods at cdent | 15:05 | |
| edleafe | cdent: well, that's why I started this conversation. | 15:05 |
| * cdent nods at everyone | 15:05 | |
| edleafe | We're making the interface dirtier | 15:06 |
| cdent | can someone point me at this dirtying? | 15:06 |
| johnthetubaguy | edleafe: where? | 15:06 |
| edleafe | basing the number of returned hosts on max_attempts | 15:06 |
| edleafe | which is a nova-ism that we've discussed removing in the past | 15:06 |
| bauzas | I would certainly appreciate reviews on my changes | 15:06 |
| edleafe | IOW, any other potential user of scheduler would now get back the number of hosts that nova wants | 15:07 |
| edleafe | IMO, it would be cleaner to make the number of hosts to return per request explicit | 15:07 |
| edleafe | and move max_attempts out of the scheduler config | 15:08 |
| johnthetubaguy | edleafe: so the retry configuration needs to be set by the operator where? | 15:08 |
| edleafe | johnthetubaguy: somewhere in nova's config, not in scheduler's config | 15:08 |
| johnthetubaguy | scheduler config is nova config right now, and there is no plan to change that right? | 15:08 |
| edleafe | johnthetubaguy: Unless, of course, we are deciding to keep the two locked together | 15:09 |
| johnthetubaguy | placement config is a totally different thing | 15:09 |
| edleafe | johnthetubaguy: no plan, correct. It's been pointed out several times as something we need to change | 15:09 |
| edleafe | if we want an independent scheduler | 15:09 |
| cdent | fwiw, edleafe, I agree with you: the config of attempts should be outside of the nova-scheduler's domain, it should accept an arg | 15:09 |
| *** itlinux has joined #openstack-nova | 15:09 | |
| cdent | _even_ if that code never becomes a lib | 15:10 |
| *** vks1 has quit IRC | 15:10 | |
| *** karimb has quit IRC | 15:10 | |
| bauzas | cdent: well, technically, we said to return alternatives by the number of retries | 15:11 |
| johnthetubaguy | seems like you want the retry conf set centrally not different on every compute node, thats the bit that matters I guess? | 15:11 |
| bauzas | cdent: and atm we have that flag in the scheduler group | 15:11 |
| bauzas | although it's settable per compute given it's used by the scheduler.utils module | 15:12 |
| johnthetubaguy | bauzas: hmm, joy | 15:12 |
| edleafe | johnthetubaguy: by not passing the number of hosts to return to the scheduler, it doesn't matter what config file it's set in. | 15:13 |
| bauzas | again, I beg people to review https://review.openstack.org/#/c/465175/ if they have matters | 15:13 |
| edleafe | it's set and unchangeable for any other service | 15:13 |
| edleafe | it seems like since keeping the interface clean is more work, we'll just keep doing things the old way with tying scheduler to nova | 15:13 |
| edleafe | which dansmith says is what we're going to do | 15:14 |
| *** salv-orlando has joined #openstack-nova | 15:14 | |
| johnthetubaguy | I am all for keeping the interface clean, it just feels like that belongs inside the scheduler to me, but I am kinda just stuck in the old ways I guess | 15:14 |
| edleafe | johnthetubaguy: so you don't ever see any service other than nova ever ever ever using the scheduler? | 15:15 |
| *** karimb has joined #openstack-nova | 15:15 | |
| johnthetubaguy | edleafe: basically no, I could see another seam being added to extract the filter/weigher stuff thats been cut and paste all over the place, and some placement helper utils, but that feels like a set of problems we should ignore for now, so we can make quicker progress | 15:16 |
| edleafe | johnthetubaguy: agree that we shouldn't spend time on it now | 15:17 |
| dansmith | johnthetubaguy: agree | 15:17 |
| edleafe | johnthetubaguy: disagree that we should actively make things worse | 15:17 |
| edleafe | worse == tighter coupling | 15:17 |
| johnthetubaguy | right now there is random util dust sprinkled all over, thats bad, I need to go review the proposed code | 15:19 |
| sean-k-mooney | dansmith: johnthetubaguy out of interest do you ever see a case where i would be allowed to pass a set of claime to nova boot? not saying that is desireable but has that come up as a topic before? | 15:20 |
| dansmith | sean-k-mooney: it has | 15:20 |
| johnthetubaguy | sean-k-mooney: yeah, came up with the watcher folks I think | 15:20 |
| johnthetubaguy | reservations, etc | 15:20 |
| *** trinaths has left #openstack-nova | 15:20 | |
| sean-k-mooney | cool was any desion reached on that or was it defered to later | 15:20 |
| dansmith | sean-k-mooney: what do you think? :P | 15:21 |
| sean-k-mooney | much later haha | 15:21 |
| dansmith | we can't even agree which color to paint the bike shed | 15:21 |
| dansmith | well, some of us can, but .. :P | 15:21 |
| *** trinaths1 has joined #openstack-nova | 15:21 | |
| *** hongbin has joined #openstack-nova | 15:22 | |
| *** salv-orlando has quit IRC | 15:22 | |
| *** felipemonteiro has joined #openstack-nova | 15:22 | |
| sean-k-mooney | ok cool as long as we can at least all agree to use the placement api for track resources and allocations of those resources thats better then what we have today | 15:23 |
| *** tbachman has joined #openstack-nova | 15:23 | |
| sean-k-mooney | well thats perhaps unfair neutron is already using it for ip pools for the l3 routing stuff so we have made more progress then just agreeing to use the placement api | 15:24 |
| *** trinaths1 has quit IRC | 15:24 | |
| *** rabel_b1 has joined #openstack-nova | 15:27 | |
| *** rabel has quit IRC | 15:27 | |
| sean-k-mooney | dansmith: by the way you summit talk on the cells v2 work was really good. jays talks on the placement apis were also good to watch | 15:27 |
| dansmith | sean-k-mooney: thanks :) | 15:28 |
| *** armax has quit IRC | 15:28 | |
| *** p01nt3r75 has quit IRC | 15:29 | |
| *** abalutoiu has quit IRC | 15:30 | |
| *** p01nt3r75 has joined #openstack-nova | 15:31 | |
| *** shewless has quit IRC | 15:31 | |
| *** Oku_OS is now known as Oku_OS-away | 15:33 | |
| *** mdrabe has quit IRC | 15:36 | |
| *** Miouge has quit IRC | 15:37 | |
| *** hieulq_ has joined #openstack-nova | 15:38 | |
| *** psachin has joined #openstack-nova | 15:38 | |
| *** Sukhdev has joined #openstack-nova | 15:40 | |
| *** gomarivera_ has joined #openstack-nova | 15:40 | |
| cfriesen | kashyap: what would you think of https://review.openstack.org/#/c/461456 ? The rationale being that it's defined as a realtime mask so if the user specifies "1-2" they should get vCPUs 1 and 2 as realtime. | 15:43 |
| *** mdrabe has joined #openstack-nova | 15:44 | |
| sfinucan | cfriesen: I took a look at it. Looks sensible to me. Do we document support image metadata anywhere though? | 15:45 |
| sfinucan | ...or is that up to Glance to do? | 15:45 |
| sfinucan | dansmith: RE: https://review.openstack.org/#/c/408242/, what would be the preferred pattern - | 15:47 |
| *** mlakat has quit IRC | 15:48 | |
| sfinucan | hardening all the access to various InstanceNUMACell attributes to check if the attribute is set first | 15:48 |
| johnthetubaguy | sfinucan: did you see the image meta object stuff? | 15:48 |
| sfinucan | or always setting those attributes anywhere we create an object? | 15:48 |
| johnthetubaguy | sfinucan: we should probably generate docs from that at some point | 15:48 |
| *** sree has joined #openstack-nova | 15:48 | |
| *** chyka has joined #openstack-nova | 15:48 | |
| cfriesen | sfinucan: not actually sure | 15:48 |
| sfinucan | johnthetubaguy: To which "stuff" do you refer? :) | 15:48 |
| *** gjayavelu has quit IRC | 15:48 | |
| johnthetubaguy | sfinucan: https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py | 15:49 |
| sfinucan | johnthetubaguy: Oh, yeah - I knew where that was defined, alright. I meant docs that you don't need to read the code for :) | 15:49 |
| johnthetubaguy | sfinucan: we should generate some docs off that I think, I suspect a few have got into the admin guides, but unsure on the details there | 15:50 |
| cfriesen | sfinucane: glance has this https://docs.openstack.org/developer/glance/common-image-properties.html but there's not much there | 15:50 |
| johnthetubaguy | sfinucan: its better than extra specs though... | 15:50 |
| cfriesen | sfinucan: johnthetubaguy: image_meta.py just says "CPU mask indicates which vCPUs will have realtime enable" which doesn't really match the current code (hence my patch) | 15:51 |
| dansmith | sfinucan: what do you mean? those patches do the right thing.. we merged one already right? | 15:51 |
| openstackgerrit | Daniel Gonzalez Nothnagel proposed openstack/nova master: WIP: Get disk size of instance in separate process https://review.openstack.org/465653 | 15:51 |
| sfinucan | johnthetubaguy: I'll go look for stuff now. Admin guide seems like the correct place to put them, even though we're going to move that into nova soon | 15:52 |
| sfinucan | dansmith: Aye, but those patches that merged mix the two. https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=99f3bbc1ac7ddcde36931f590b6df3811e596d8c | 15:52 |
| *** sree_ has joined #openstack-nova | 15:53 | |
| sfinucan | dansmith: So in some cases, we've done "if ('key' in object) and object.key:" | 15:53 |
| *** lucasagomes is now known as lucas-afk | 15:53 | |
| *** sree_ is now known as Guest39263 | 15:53 | |
| sfinucan | ...but in others, we just set the various fields to None in the tests, sidestepping the issue | 15:53 |
| dansmith | sfinucan: being defensive in the code that uses those fields is the wy | 15:54 |
| dansmith | *way | 15:54 |
| sfinucan | OK, I'll do that so | 15:54 |
| dansmith | sfinucan: i.e. the "if foo in obj and obj.foo | 15:54 |
| dansmith | sfinucan: cool, thanks for picking those back up | 15:54 |
| * sfinucan is trying to fix https://review.openstack.org/#/c/408242/ | 15:54 | |
| sfinucan | dansmith: no problemo | 15:54 |
| *** sree has quit IRC | 15:55 | |
| *** gomarivera_ has quit IRC | 15:55 | |
| *** sree has joined #openstack-nova | 15:56 | |
| *** Apoorva has joined #openstack-nova | 15:57 | |
| *** Guest39263 has quit IRC | 15:58 | |
| *** gouthamr_ has joined #openstack-nova | 15:58 | |
| *** karimb_ has joined #openstack-nova | 15:58 | |
| *** gouthamr has quit IRC | 15:59 | |
| *** nic has joined #openstack-nova | 16:00 | |
| *** karimb has quit IRC | 16:00 | |
| *** nic1 has joined #openstack-nova | 16:00 | |
| *** lpetrut has quit IRC | 16:01 | |
| *** tjones has quit IRC | 16:03 | |
| *** moshele has joined #openstack-nova | 16:03 | |
| *** sree has quit IRC | 16:03 | |
| *** sree has joined #openstack-nova | 16:04 | |
| *** nic has quit IRC | 16:05 | |
| mriedem_afk | rabel_b1: you can have global default quotas (limits), project-specific default quotas, and then just project-specific quotas, and i understand that the latter two are confusing, and i'm not sure if/how we treat them differently, unless you have default project limits and user-specific limits within that project | 16:05 |
| *** mriedem_afk is now known as mriedem | 16:05 | |
| mriedem | mkoderer: dgonzalez: so anytime anything hits https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7025 in the compute it locks up the entire process? | 16:06 |
| mriedem | mkoderer: dgonzalez: because that method is called from a few different operations | 16:06 |
| rabel_b1 | mriedem: ok, thank you very much. | 16:08 |
| *** hieulq_ has quit IRC | 16:09 | |
| *** hieulq_ has joined #openstack-nova | 16:09 | |
| mriedem | dgonzalez: ah i see your patch now | 16:09 |
| *** hieulq_ has quit IRC | 16:10 | |
| *** hieulq_ has joined #openstack-nova | 16:10 | |
| *** gyee_ has joined #openstack-nova | 16:11 | |
| *** pcaruana has quit IRC | 16:11 | |
| *** mdnadeem has joined #openstack-nova | 16:12 | |
| *** hieulq__ has joined #openstack-nova | 16:13 | |
| *** ociuhandu has joined #openstack-nova | 16:16 | |
| *** jaosorior is now known as jaosorior_away | 16:17 | |
| *** mdnadeem has quit IRC | 16:17 | |
| *** hieulq__ has quit IRC | 16:18 | |
| *** belmoreira has quit IRC | 16:20 | |
| *** p01nt3r75 has quit IRC | 16:20 | |
| *** gyee has quit IRC | 16:20 | |
| *** ociuhandu has quit IRC | 16:25 | |
| *** gouthamr_ has quit IRC | 16:28 | |
| *** marst has quit IRC | 16:28 | |
| *** aarefiev is now known as aarefiev_afk | 16:31 | |
| *** sree_ has joined #openstack-nova | 16:32 | |
| *** sree_ is now known as Guest16420 | 16:32 | |
| *** harlowja has joined #openstack-nova | 16:32 | |
| *** Guest16420 has quit IRC | 16:33 | |
| *** sree has quit IRC | 16:34 | |
| *** ltomasbo is now known as ltomasbo|away | 16:40 | |
| *** rabel_b1 has quit IRC | 16:41 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | 16:43 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | 16:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | 16:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | 16:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | 16:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | 16:44 |
| openstackgerrit | sean mooney proposed openstack/nova master: do not set vcpu_pin elements when virt_type=qemu https://review.openstack.org/465160 | 16:44 |
| *** eharney has quit IRC | 16:44 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | 16:47 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | 16:47 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | 16:47 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | 16:47 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | 16:47 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | 16:47 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | 16:47 |
| *** armax has joined #openstack-nova | 16:48 | |
| *** eharney has joined #openstack-nova | 16:48 | |
| *** gianpietro has quit IRC | 16:55 | |
| *** karimb_ has quit IRC | 16:59 | |
| *** gouthamr has joined #openstack-nova | 17:02 | |
| *** derekh has quit IRC | 17:02 | |
| *** fragatina has joined #openstack-nova | 17:02 | |
| *** fragatina has quit IRC | 17:02 | |
| *** fragatina has joined #openstack-nova | 17:03 | |
| *** dimtruck is now known as zz_dimtruck | 17:03 | |
| *** hieulq__ has joined #openstack-nova | 17:03 | |
| *** hieulq_ has quit IRC | 17:04 | |
| *** tbachman has quit IRC | 17:04 | |
| *** zz_dimtruck is now known as dimtruck | 17:04 | |
| cfriesen | where does ServerExternalEvents run? is this part of nova-api? | 17:04 |
| cfriesen | bah, never mind, just noticed the full path to the file | 17:05 |
| *** hieulq__ has quit IRC | 17:07 | |
| *** sambetts is now known as sambetts|afk | 17:09 | |
| *** stvnoyes1 has left #openstack-nova | 17:11 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add a method signature test for all virt drivers https://review.openstack.org/465676 | 17:11 |
| mdbooth | Honestly I was expecting that to find things. Surprised it didn't. | 17:12 |
| *** jwcroppe_ has quit IRC | 17:13 | |
| mdbooth | lyarwood: Added ^^^ when I noticed your change to swap_volume, also your previous reluctance to change detach_volume. This test should make those things easier. | 17:14 |
| *** suresh12 has joined #openstack-nova | 17:15 | |
| *** hieulq_ has joined #openstack-nova | 17:15 | |
| *** fragatina has quit IRC | 17:16 | |
| *** hieulq_ has quit IRC | 17:18 | |
| *** felipemonteiro has quit IRC | 17:18 | |
| *** gianpietro has joined #openstack-nova | 17:19 | |
| artom | mdbooth, I thought we already tested virt drivers's method signatures... Were some drivers missing from the list? | 17:20 |
| *** ralonsoh_ has quit IRC | 17:21 | |
| mdbooth | artom: Where do we test it? | 17:22 |
| mdbooth | Although that would explain why they all match :) | 17:22 |
| artom | mdbooth, ah, I think they're individual tests in each driver | 17:22 |
| artom | http://paste.openstack.org/show/609828/ | 17:23 |
| artom | I know I remembered getting method signature errors | 17:23 |
| *** gianpietro has quit IRC | 17:24 | |
| mdbooth | That's incredibly similar to mine, except not recursive | 17:25 |
| mdbooth | I needed recursive for the volume drivers | 17:25 |
| mdbooth | Although I hadn't noticed that existed already | 17:26 |
| *** harlowja has quit IRC | 17:26 | |
| mdbooth | I wonder why it doesn't have to handle wrapped methods | 17:26 |
| *** ericyoung has joined #openstack-nova | 17:28 | |
| artom | mdbooth, I agree your way is better - it doesn't depend on each driver doing their own tests correctly | 17:28 |
| mdbooth | Ah... HyperV munges it. | 17:28 |
| mdbooth | artom: Meh, it's only important that it's checked, tbh. | 17:29 |
| artom | mdbooth, I suppose the only issue with the current tests is if a new virt driver gets added and forgets them | 17:29 |
| artom | But that doesn't seem likely | 17:29 |
| *** jpena is now known as jpena|off | 17:30 | |
| *** gaurangt has joined #openstack-nova | 17:30 | |
| mdbooth | Well powervm.PowerVMDriver already wasn't added to the DriverMethodTestCase | 17:30 |
| artom | Also, you've included PowerVM | 17:30 |
| mdbooth | Although that case is stupid | 17:30 |
| artom | Which isn't in the list I got | 17:30 |
| mdbooth | artom: Can you note on the review that we already do that? I'll probably abandon it, and/or update my recursive test to use the existing method. | 17:32 |
| artom | mdbooth, sure | 17:32 |
| mdbooth | artom: Thanks | 17:32 |
| *** Sukhdev has quit IRC | 17:35 | |
| *** tbachman has joined #openstack-nova | 17:35 | |
| artom | mdbooth, there. I didn't even -1 it ;) | 17:35 |
| *** crushil has quit IRC | 17:37 | |
| mriedem | cfriesen: did you happen to catch the sriov bug armax was asking about last night? | 17:38 |
| *** kaisers has joined #openstack-nova | 17:39 | |
| *** felipemonteiro has joined #openstack-nova | 17:39 | |
| *** yamamoto_ has quit IRC | 17:39 | |
| *** kfarr has joined #openstack-nova | 17:40 | |
| armax | mriedem: btw talking to gyee_ it seems the issue happens in Newton still | 17:40 |
| *** fragatina has joined #openstack-nova | 17:40 | |
| *** chyka_ has joined #openstack-nova | 17:41 | |
| *** yamamoto has joined #openstack-nova | 17:43 | |
| *** jwcroppe has joined #openstack-nova | 17:43 | |
| *** salv-orlando has joined #openstack-nova | 17:43 | |
| gyee_ | I am curious why we are using NopClaim if host is specified https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2771 | 17:43 |
| gyee_ | does specifying a host implies force evacuate? | 17:44 |
| *** crushil has joined #openstack-nova | 17:44 | |
| *** chyka has quit IRC | 17:45 | |
| *** salv-orlando has quit IRC | 17:48 | |
| *** gjayavelu has joined #openstack-nova | 17:49 | |
| *** felipemonteiro has quit IRC | 17:49 | |
| gyee_ | meanwhile if SR-IOV is enabled, we are trying to get the PCI mapping via the migration context, which is only populated at rebuild_claim. https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L2462 | 17:49 |
| *** itlinux has quit IRC | 17:50 | |
| *** jamesdenton has quit IRC | 17:50 | |
| mriedem | https://github.com/openstack/nova/commit/dc0221d7240326a2d1b467e2a367bebb7e764e61 added that code in the compute manager about the nop claim, which implies resources were already claimed, but i'd have to dig into that | 17:51 |
| *** felipemonteiro has joined #openstack-nova | 17:52 | |
| *** kfarr has quit IRC | 17:54 | |
| mriedem | yikes, that was part of a big series https://review.openstack.org/#/q/topic:bug/1417667 | 17:56 |
| *** felipemonteiro has quit IRC | 17:56 | |
| *** gianpietro has joined #openstack-nova | 17:57 | |
| gyee_ | but I don't see where the resource is being claimed, if the host is specified | 17:57 |
| openstackgerrit | Gage Hugo proposed openstack/nova master: Admin API Policy contingent on is_admin_project https://review.openstack.org/384148 | 17:57 |
| mriedem | gyee_: i'm also looking for that | 17:58 |
| *** jamesdenton has joined #openstack-nova | 17:58 | |
| *** jamielennox is now known as jamielennox|away | 17:59 | |
| *** rabel_b1 has joined #openstack-nova | 18:02 | |
| *** liangy has quit IRC | 18:02 | |
| gyee_ | mriedem: I don't think that's being done. The migration_context in instance_extra table is empty if host specified, regardless if SR-IOV is enabled or not | 18:03 |
| *** smatzek is now known as smatzek_away | 18:05 | |
| *** jamielennox|away is now known as jamielennox | 18:05 | |
| mriedem | gyee_: as far as i can tell, the only way to get the noop claim is if the user specifies the host and --force | 18:06 |
| *** zul has quit IRC | 18:06 | |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4115 | 18:07 |
| mriedem | if the host is specified but force is not, then the api nulls out host when calling conductor | 18:07 |
| gyee_ | I didn't use the --force option. Just merely specifying the host. | 18:07 |
| mriedem | which gets to this else block https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L751 | 18:07 |
| mriedem | node would eventually be set here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L770 | 18:08 |
| mriedem | and passed to compute as scheduled_node | 18:08 |
| mriedem | which should get the rebuild_claim here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2769 | 18:08 |
| mriedem | dansmith: do you remember anything about not doing a claim if the host is forced during evacuate? | 18:08 |
| *** yamahata has joined #openstack-nova | 18:09 | |
| dansmith | mriedem: no | 18:09 |
| dansmith | mriedem: I would think you'd get a new claim on the new compute node, and then drop the old one when the dead compute comes up a deletes the old instance | 18:10 |
| *** rabel_b1 has quit IRC | 18:11 | |
| *** Sukhdev has joined #openstack-nova | 18:11 | |
| *** yamamoto has quit IRC | 18:11 | |
| mriedem | i'm not sure why we have this then https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2771 | 18:12 |
| *** yamamoto has joined #openstack-nova | 18:12 | |
| dansmith | mriedem: well for a regular rebuild | 18:12 |
| mriedem | oh...yeah | 18:12 |
| dansmith | regular rebuild is in place | 18:12 |
| mriedem | original commit message says "Also - if we called the scheduler, we also know the node of the chosen host, and the limits set, and this information is needed for doing claims." | 18:13 |
| mriedem | but yeah, | 18:13 |
| *** claudiub has joined #openstack-nova | 18:13 | |
| mriedem | i suppose the noop claim is there for rebuild on the same host | 18:13 |
| mriedem | not evacuate | 18:13 |
| sdague | efried: I'm pushing in your devstack console color fix, thanks again for that | 18:13 |
| mriedem | god | 18:13 |
| mriedem | at some point we should just split the evacuate and rebuild code apart in the compute manager | 18:13 |
| gyee_ | ++ | 18:14 |
| mriedem | i think you can also specify a host for rebuild | 18:15 |
| mriedem | oh no nvm | 18:15 |
| mriedem | i was thinking of another bug, | 18:15 |
| mriedem | where you change the image and the new image meta for the rebuild doesn't work with the original host, but we don't validate that | 18:16 |
| mriedem | gyee_: then i think all you need to do is add an "or recreate" to this conditional https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2767 | 18:16 |
| mriedem | gyee_: do you have a recreate of the bug? | 18:17 |
| *** moshele has quit IRC | 18:17 | |
| gyee_ | mriedem: I haven't file a bug for Nova yet | 18:19 |
| gyee_ | about to do so | 18:19 |
| mriedem | gyee_: oh we have a bug | 18:19 |
| mriedem | well MOS does | 18:19 |
| mriedem | but we can add nova to that | 18:19 |
| cfriesen | question about RPC (with standard rabbitmq/kombu messaging): are incoming messages on nova-compute immediately grabbed by an eventlet? I'm seeing huge lock contention on the 'refresh_cache-%s' lock between the live migration code and external events coming in from neutron | 18:19 |
| gyee_ | do I have perm to do that? Sorry I am still new to Nova. | 18:19 |
| mriedem | gyee_: https://bugs.launchpad.net/mos/+bug/1658070 | 18:20 |
| openstack | Launchpad bug 1658070 in Mirantis OpenStack "Failed SR_IOV evacuation with host" [Medium,Confirmed] | 18:20 |
| *** sree has joined #openstack-nova | 18:20 | |
| gyee_ | oh good :-) | 18:20 |
| *** psachin has quit IRC | 18:23 | |
| *** jamesdenton has quit IRC | 18:23 | |
| *** chyka has joined #openstack-nova | 18:23 | |
| *** moshele has joined #openstack-nova | 18:24 | |
| cdent | bauzas, dansmith: sorry for beating a dead horse, but I need some help understanding https://review.openstack.org/#/c/465175/ (returning alternatives) | 18:24 |
| *** sree has quit IRC | 18:25 | |
| cdent | edleafe too | 18:25 |
| gyee_ | mriedem: thanks for the help! | 18:25 |
| *** chyka_ has quit IRC | 18:25 | |
| dansmith | cdent: what about it? | 18:25 |
| mriedem | gyee_: np, you did the hard debug already, | 18:25 |
| cdent | in the collection of hosts being returned from select_destinations there is a list of dicts | 18:25 |
| cdent | in each dict is one host which is the chosen host | 18:25 |
| mriedem | gyee_: if you get a recreate and have that fix handy ping me to review | 18:25 |
| cdent | and then a key 'alternatives' which are other hosts that might work | 18:26 |
| cdent | is each chosen host (across the entire selected_hosts) different or does every single host satisfy the request_spec? | 18:26 |
| gyee_ | mriedem: definitely, just need to add unit tests before pushing it upstream | 18:27 |
| cdent | or to put it another way: what is the reason for not having simply "more hosts" instead of the nested "alternatives"? | 18:27 |
| *** crushil has quit IRC | 18:27 | |
| dansmith | cdent: each host could satisfy the request.. meaning all the primaries and alternates | 18:27 |
| dansmith | cdent: I had thought about it in terms of just a list of hosts, and not the hierarchy, however: | 18:28 |
| dansmith | cdent: I think the point is that we will, for a single num_instances>1 request, not choose the same primary host for two instances, but the alternates can overlap | 18:28 |
| dansmith | cdent: so the distinction is made such that we consume the primaries | 18:28 |
| mriedem | gyee_: i'm still a bit confused about how this happens if that force flag isn't specified | 18:29 |
| cdent | dansmith: is the additional code complexity worth the win? | 18:29 |
| dansmith | cdent: well, if we don't, then we'll choose hostA, (hostB, C) for the first instance and hostB, (HostA, C) for the second, right? that's a bad decision if we fail on hostB and the first instance filled up hostA | 18:30 |
| *** suresh12 has quit IRC | 18:30 | |
| dansmith | cdent: also, I think we have to consume the hosts as primaries to keep the existing scheduler behavior, right? | 18:30 |
| dansmith | the previous code did that such that num_instances>1 would not pack them all onto the same host, IIUC | 18:31 |
| *** salv-orlando has joined #openstack-nova | 18:31 | |
| dansmith | I dunno, I guess I'd wait for either bauzas or johnthetubaguy to comment on that | 18:31 |
| * dansmith shuts up | 18:31 | |
| cdent | thanks dansmith, I'll have to read through it a bit more | 18:32 |
| cdent | it feels like it could be simpler, but feelings may not matter | 18:32 |
| *** jamesdenton has joined #openstack-nova | 18:32 | |
| dansmith | well, FWIW, when I first read that code I was kinda horrified with the nested dicting, | 18:33 |
| dansmith | but I didn't complain because I think it probably makes the smallest behavioral change | 18:33 |
| dansmith | it's more code, but I'm not sure it's _that_ much more complex | 18:33 |
| *** suresh12 has joined #openstack-nova | 18:34 | |
| dansmith | it's not an unreasonable opinion though, IMHO | 18:34 |
| *** jwcroppe has quit IRC | 18:34 | |
| dansmith | recording it, even if it is shot down with technical reasons is probably a good idea | 18:34 |
| cdent | roger that | 18:34 |
| gyee_ | mriedem: maybe none != False? | 18:35 |
| EmilienM | mriedem, cdent: super quick fyi: I tested again nova/wsgi and it works fine | 18:35 |
| cdent | yay! | 18:35 |
| EmilienM | this is our config: http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache_config/10-nova_api_wsgi.conf.txt.gz | 18:35 |
| EmilienM | if you have any suggestion, feel free to let me know | 18:35 |
| EmilienM | the proof it works http://logs.openstack.org/81/462281/3/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/ce2e89d/logs/apache/nova_api_wsgi_access_ssl.txt.gz | 18:35 |
| *** itlinux has joined #openstack-nova | 18:36 | |
| EmilienM | mwhahaha: ^ fyi also | 18:36 |
| *** moshele has quit IRC | 18:36 | |
| mriedem | gyee_: i think you might be right https://github.com/openstack/nova/blob/stable/newton/nova/api/openstack/compute/evacuate.py#L92 | 18:37 |
| cdent | EmilienM: is cgi-bin/nova/nova-api the new wsgi script that pbr creates? | 18:37 |
| EmilienM | cdent: yes, nova-api-wsgi or something | 18:37 |
| *** zul has joined #openstack-nova | 18:37 | |
| mriedem | gyee_: gah, yeah, that's it | 18:37 |
| edleafe | cdent: my understanding is that if N instances are requested, we'd pick N hosts from the list of weighed hosts. Then for each, we'd get the cell uuid and from the remaining, find enough in the same cell to satisfy the number of alternates needed | 18:37 |
| EmilienM | cdent: /usr/bin/nova-api-wsgi | 18:37 |
| gyee_ | mriedem: two bugs then | 18:37 |
| mriedem | so if you're using microversion<2.29 then force is None | 18:37 |
| mriedem | yup | 18:37 |
| EmilienM | cdent: I need to go now, but you can PM me if you have more comments, i'll catch up later | 18:38 |
| edleafe | So I didn't envision overlap at all | 18:38 |
| edleafe | but I suppose it wouldn't be terrible, since the claim would simply fail on a retry | 18:38 |
| *** salv-orlando has quit IRC | 18:39 | |
| melwitt | dansmith, sdague: I created a WIP devstack patch to add the --tcp option to dstat and added Depends-On it for the caching db connections patch, and compared the output (using active tcp as a hint to db connections) | 18:40 |
| sdague | melwitt: cool, honestly, just add that if you like | 18:40 |
| melwitt | I put them in a spreadsheet here https://docs.google.com/spreadsheets/d/1DIfFfX3kaA_SRoCM-aO7BN4IBEShChXLztOBFeKryt4/edit?usp=sharing | 18:40 |
| sdague | dstat is really there to cross check all these things for runs, so no reason to not report it all the time | 18:40 |
| melwitt | cool. it makes the csv less human readable but I think it's probably worth it for us to be able to have a high level indicator of how many connections we're using http://logs.openstack.org/94/465194/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/7a6b700/logs/dstat-csv_log.txt.gz | 18:42 |
| dansmith | melwitt: not sure I get it.. both tabs top out at ~40 right? | 18:43 |
| sdague | yeh, the non csv version is still emitted as well | 18:43 |
| dansmith | oh | 18:44 |
| melwitt | dansmith: that's listening. I highlighted the "active/established" column, col AC. comparing a gate-tempest-dsvm-neutron-full-ubuntu-xenial in both cases, without caching active connections were 1495 at the end of the run, with caching it was 445 | 18:44 |
| dansmith | yeah | 18:44 |
| dansmith | I hadn't scrolled over enough | 18:45 |
| *** yamamoto has quit IRC | 18:45 | |
| dansmith | well, I guess that's it | 18:45 |
| dansmith | and needing of a backport | 18:45 |
| dansmith | I just dunno why we're never gc'ing those | 18:45 |
| melwitt | ah, I had hoped it would show the scrolling I did to ppl shared with | 18:45 |
| melwitt | yeah, I don't get it either. I thought it should have worked fine the way it was | 18:46 |
| mriedem | melwitt: for your todo in the wip you could use oslo.cache which i think has a default timer | 18:46 |
| mriedem | and can be configured for memcache | 18:46 |
| dansmith | mriedem: will that put it in memcache or dogpile? | 18:47 |
| dansmith | because we probably don't want that | 18:47 |
| mriedem | defaults to dogpile | 18:47 |
| mriedem | in memory | 18:47 |
| mriedem | well, i think | 18:47 |
| *** MVenesio has joined #openstack-nova | 18:47 | |
| mriedem | https://docs.openstack.org/developer/oslo.cache/opts.html | 18:47 |
| dansmith | we just don't want those shared across machines in any way, and not serialized either | 18:47 |
| *** salv-orlando has joined #openstack-nova | 18:48 | |
| melwitt | mriedem: I have looked at oslo.cache briefly before and wasn't sure about whether or not to use it. I don't think I found examples in nova | 18:49 |
| *** felipemonteiro has joined #openstack-nova | 18:49 | |
| dansmith | well, for the backport we can ignore the cache purge, | 18:49 |
| dansmith | since the other lists aren't purged either | 18:49 |
| melwitt | dansmith: I added a thing to clear it upon SIGHUP. as far as expiring based on a timer, the utility of it seems to be mostly around not keeping a connection open to a cell that's seldom used | 18:51 |
| dansmith | right, I'm just saying we needn't worry about any of this for the first patch that we want to be simple for backporting | 18:52 |
| dansmith | I wouldn't even hook up SIGHUP since the other lists are't either | 18:52 |
| melwitt | okay | 18:53 |
| dansmith | related tangent, | 18:55 |
| dansmith | reporting the tcp connection count like this all the time seems like a good thing, and is another thing we could measure and report on independent of worker performance | 18:55 |
| dansmith | i.e "ending tcp connections is >100% higher than expected" | 18:55 |
| dansmith | re: the perf-monitoring forum session | 18:55 |
| melwitt | yeah, I was thinking the same | 18:55 |
| dansmith | I also wonder if we should open a bug against oslo-db for this | 18:56 |
| melwitt | going to un-WIP the devstack patch. sdague, where is the non-csv dstat in the logs? I was wondering should we put --tcp in for both csv and non-csv? I did but I wasn't sure | 18:56 |
| *** mtanino_ has joined #openstack-nova | 18:58 | |
| melwitt | dansmith: yeah, should at least ask zzzeek about it and open a bug if it's not an expected thing | 18:58 |
| *** catintheroof has joined #openstack-nova | 18:59 | |
| *** salv-orlando has quit IRC | 18:59 | |
| sdague | melwitt: it's in a screen | 19:00 |
| sdague | well, it's in the journal :) | 19:00 |
| sdague | but we dump it out to screen | 19:00 |
| sdague | files | 19:00 |
| sdague | for compat in gate runs | 19:00 |
| melwitt | cool, thanks | 19:01 |
| *** crushil has joined #openstack-nova | 19:03 | |
| *** READ10 has quit IRC | 19:03 | |
| *** lpetrut has joined #openstack-nova | 19:03 | |
| *** Vek has quit IRC | 19:04 | |
| *** mordred has quit IRC | 19:05 | |
| *** jwcroppe has joined #openstack-nova | 19:06 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | 19:06 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | 19:06 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | 19:06 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | 19:06 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | 19:06 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | 19:06 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | 19:06 |
| *** mordred has joined #openstack-nova | 19:06 | |
| openstackgerrit | Steve Noyes proposed openstack/nova master: WIP: update live migration to use v3 cinder api https://review.openstack.org/463987 | 19:10 |
| melwitt | sdague: patch has been un-WIP https://review.openstack.org/#/c/465194 | 19:11 |
| sdague | melwitt: +2, thanks | 19:12 |
| melwitt | thx | 19:12 |
| *** artom has quit IRC | 19:13 | |
| *** stvnoyes has joined #openstack-nova | 19:14 | |
| cdent | dansmith, bauzas, edleafe I tried to leave some questions on https://review.openstack.org/#/c/465175/ to capture my concerns, but not sure if they are clear | 19:16 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Destroy the ReqSpec object when deleting the instance https://review.openstack.org/391060 | 19:17 |
| *** gomarivera_ has joined #openstack-nova | 19:22 | |
| *** karimb has joined #openstack-nova | 19:26 | |
| *** gomarivera_ has quit IRC | 19:27 | |
| *** mkoderer has quit IRC | 19:27 | |
| *** awaugama has quit IRC | 19:28 | |
| *** jose-phillips has quit IRC | 19:34 | |
| *** clenimar has joined #openstack-nova | 19:34 | |
| *** gianpietro has quit IRC | 19:35 | |
| *** NostawRm has quit IRC | 19:36 | |
| *** yamahata has quit IRC | 19:37 | |
| *** harlowja has joined #openstack-nova | 19:39 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | 19:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | 19:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | 19:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | 19:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | 19:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | 19:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | 19:40 |
| *** xyang1 has quit IRC | 19:43 | |
| *** xyang1 has joined #openstack-nova | 19:45 | |
| melwitt | cfriesen: had you filed a bug for the increase in DB connections in Newton? I'll need one to refer to in my patch | 19:45 |
| *** yamamoto has joined #openstack-nova | 19:45 | |
| *** jose-phillips has joined #openstack-nova | 19:46 | |
| *** Sukhdev has quit IRC | 19:47 | |
| *** lpetrut has quit IRC | 19:47 | |
| *** crushil has quit IRC | 19:49 | |
| *** jose-phillips has quit IRC | 19:52 | |
| *** harlowja has quit IRC | 19:53 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove DeviceIsBusy exception https://review.openstack.org/465723 | 19:56 |
| *** karimb has quit IRC | 19:56 | |
| *** nic1 is now known as nic | 19:57 | |
| *** harlowja has joined #openstack-nova | 19:57 | |
| *** karimb has joined #openstack-nova | 19:57 | |
| *** karimb has quit IRC | 19:57 | |
| *** harlowja has quit IRC | 20:00 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Cache database and message queue connection objects https://review.openstack.org/465042 | 20:01 |
| *** yamamoto has quit IRC | 20:02 | |
| *** cdent has quit IRC | 20:05 | |
| zzzeek | melwitt: can you tell me about https://review.openstack.org/#/c/465042/3/nova/context.py | 20:06 |
| *** crushil has joined #openstack-nova | 20:06 | |
| melwitt | zzzeek: yes! I was hoping you would be able to comment on this. recently we've seen a significant increase in number of DB connections since we've been doing DB connection switching for cells | 20:07 |
| zzzeek | melwitt: so...did you try changing pool_size and max_overflow ? | 20:08 |
| zzzeek | melwitt: also when you say "switching" - are you retrieving independent connections per cell DB or are you "switching" a single connection to refer to different databases within a single MySQL server? | 20:08 |
| zzzeek | IIUC "cells" would be in different DB servers altogether so they'd have to have independent DB connections | 20:09 |
| dansmith | zzzeek: correct, we're just surprised that when we create and stop using one, they don't seem to be gc'd | 20:09 |
| melwitt | zzzeek: independent connection. to make a connection we create a new oslo.db transaction context manager | 20:09 |
| dansmith | zzzeek: i.e. every time we do that switch (i.e. new connection), we get a connection that never goes away | 20:09 |
| zzzeek | dansmith / melwitt OK, so, the SQLAlchemy engine by default uses a connection pool | 20:09 |
| zzzeek | if you don't want it to pool anything, you need to turn it off | 20:10 |
| *** smatzek_away has quit IRC | 20:10 | |
| *** dnovosel has joined #openstack-nova | 20:10 | |
| dansmith | zzzeek: the pool is keeping references to all the connections then? | 20:10 |
| *** pchavva has quit IRC | 20:10 | |
| zzzeek | dansmith: it keeps refernces to as many connections as you've told it to. our default in oslo.db is probably 5 at the moment | 20:10 |
| dansmith | zzzeek: we end up with ~1400 connections after a tempest run | 20:11 |
| dansmith | I'm guessing it doesn't know that each re-connection to the same db is reusable though? so it's keeping 5 connections of every transaction manager? | 20:11 |
| zzzeek | dansmith: we end up with 1.4K connections on an openstack even w/o separate cell DBs if we have dozens of processes. it's kind of a problem to which there are a few solutions | 20:11 |
| dansmith | if that's the case, we should just not do that thing (i.e. do what melwitt has proposed) | 20:11 |
| melwitt | zzzeek: I had thought the pool would hand back an object that is the same DB as before | 20:11 |
| dansmith | zzzeek: well, it doesn't happen with melwitt's patch | 20:12 |
| zzzeek | dansmith: right but, you don't need a patch. just turn down the pooling | 20:12 |
| *** suresh12 has quit IRC | 20:12 | |
| zzzeek | dansmith: with one exception, which is that to disable the pool entirely, you need to use the "nullpool" implementation and we might need to add a hook to oslo.db if you really want that. but that is not what melwitt's patch is doing. | 20:12 |
| dansmith | well, we get different behavior than what we have today.. meaning we defeat the benefits of the connection pooling , if there are benefits | 20:13 |
| zzzeek | dansmith: just in the last few IRC lines im not sure if all three of us are clear on every concept here, for example, I don't know what a "Transaction manager" is | 20:13 |
| dansmith | zzzeek: the oslo.db context | 20:13 |
| melwitt | it's the _TransactionContextManager class in oslo.db | 20:14 |
| dansmith | https://review.openstack.org/#/c/465042/3/nova/context.py | 20:14 |
| dansmith | L384 there ^ | 20:14 |
| zzzeek | also, the idea of a "pool" is that yes, it has a connection that it will reuse as needed | 20:14 |
| melwitt | okay, so I don't know what we're doing to not get the reusability | 20:14 |
| dansmith | melwitt: the pool manager has to know that a repeated connection attempt is "the same" in order to re-use an existing connection | 20:15 |
| dansmith | so I'm wondering if we're confusing it such that it isn't re-using every time we create a new context there | 20:15 |
| zzzeek | dansmith: ok more terms that confuse me, what is "the pool manager" | 20:15 |
| melwitt | right. how does it normally know that I guess? yeah | 20:15 |
| dansmith | zzzeek: I'm sure I'm not using the right term.. whatever the thing is in sqla that does the pooling.. you said "nullpool" above, which would be one of those things | 20:16 |
| dansmith | melwitt: well, it can do connection pooling with one context easily | 20:16 |
| melwitt | right ... | 20:16 |
| dansmith | and that's all we've needed in the past | 20:16 |
| dansmith | but now we're creating lots of those things | 20:17 |
| zzzeek | dansmith: OK. so we are talking aobut SQLAlchemy QueuePool or NullPool. it doesn't know about connection attempts being "the same" or anything like that, it basically is asked for a DB connection, and it gives you one from its pool. if it doesnt have any, it either decides to give you an ad-hoc connection based on what you've configured "max_overflow" towards, or, it makes you wait | 20:17 |
| melwitt | zzzeek: this is how we establish db connections (static) and what we did to "switch" is just create a new one of those each time https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L79-L80 | 20:17 |
| mriedem | i'd think it would key off the connection url | 20:17 |
| melwitt | mriedem: that's what I had thought | 20:17 |
| dansmith | maybe, but that has creds in it | 20:17 |
| dansmith | zzzeek: but is that pool global, or per-context or what? | 20:17 |
| *** dimtruck is now known as zz_dimtruck | 20:18 | |
| zzzeek | dansmith: if nova is architected here per best practices, you'd have one pool per unique database URL | 20:18 |
| dansmith | heh | 20:18 |
| zzzeek | dansmith: if OTOH you are creating brand new engines every time you get a requset, then you're making new pools each time and that is the wrong way to do that | 20:18 |
| dansmith | we're a long ways away from the pool in nova, so it's hard to say | 20:19 |
| zzzeek | dansmith: so in this patch i see "db_connection", and if that is in fact an "Engine" object, or say the enginefacade factory around that, then that's OK, that's what should happen here. | 20:19 |
| zzzeek | that's not a "connection", it's a factory for connections | 20:19 |
| zzzeek | if that's the case, I dont really know what this is | 20:19 |
| dansmith | enginefacade.transaction_context() | 20:20 |
| dansmith | it's that ^ | 20:20 |
| melwitt | it's a _TransactionContextManager | 20:20 |
| zzzeek | dansmith / melwitt ok | 20:20 |
| *** gianpietro has joined #openstack-nova | 20:20 | |
| *** marst has joined #openstack-nova | 20:20 | |
| melwitt | I modeled it off how it's done in our db/sqlalchemy/api.py for the static connections | 20:20 |
| zzzeek | melwitt: OK then that's fine let me update my thing | 20:21 |
| efried | sdague Thinking I should blast the ML to let 'em know about journalctl -a, yah? | 20:21 |
| zzzeek | melwitt: to clarify, previosly, everyime new cell DB were used, you'd build up whole new _TransactionContextManager per request ? | 20:21 |
| efried | I mean, it's in the doc now (when does that get flushed to docs.openstack.org?) but who's gonna go reread it? | 20:22 |
| *** suresh12 has joined #openstack-nova | 20:23 | |
| melwitt | zzzeek: yeah. and that was based on my limited knowledge of how to use oslo.db, if I want to establish a new connection from only a connection url, it seemed like that's the only way? | 20:23 |
| melwitt | is to use enginefacade.transaction_context() | 20:23 |
| zzzeek | melwitt: OK so, is that version of the code what's released in production and all that? because that will perform very, very badly | 20:23 |
| cfriesen | who knows neutron and live-migration? Do we really need to process "network-changed" events on the dest compute node during a migration when we're going to explicitly call network_info = self.network_api.get_instance_nw_info() anyways in post_live_migration_at_destination()? | 20:23 |
| melwitt | zzzeek: yes. that's why at the moment, it seems like caching these is the way to fix this, and backport it all the way to newton | 20:24 |
| zzzeek | melwitt: it is. So, background, the transactioncontextmanager is a facade around the concept of the SQLAlchemy engine | 20:24 |
| mriedem | cfriesen: we probably don't need to, but does refreshing the network info twice do any harm? | 20:25 |
| zzzeek | this is why it has a method "get_engine()". when you use it as a decorator or context manager, it goes in to a "block" in which scope it actually procures an ORM session which uses an actual sqlalchemy.engine.Connection object | 20:25 |
| zzzeek | melwitt: the concept of the Engine is described at http://docs.sqlalchemy.org/en/rel_1_1/core/engines.html http://docs.sqlalchemy.org/en/rel_1_1/core/connections.html http://docs.sqlalchemy.org/en/rel_1_1/core/pooling.html and would provide some background on what's going on | 20:26 |
| *** suresh12 has quit IRC | 20:27 | |
| melwitt | zzzeek: thanks | 20:27 |
| zzzeek | melwitt: the history of "enginefacade" itself is that when I came to work on openstack, there was already an "EngineFacade" local to openstack / oslo that evreyone was using, so the curent oslo.db version tries to improve upon that. however, like all layers of indirection, it is confusing. I wrote it and I never remember how it is organized when I go back and look at it, because it had to work on top of all these existing | 20:27 |
| zzzeek | patterns | 20:27 |
| cfriesen | mriedem: turns out that it does when you have a lot (like 16) vifs on an instance. The live migration code can end up blocking for like 35+ seconds waiting to get the 'refresh_cache-%s' lock because it's racing against the incoming network-change events | 20:28 |
| zzzeek | melwitt: so short story, if you make a new engine (and therefore a new connection pool) on every request, then yes, the "pool" is not working because you are making a whole new pool each time | 20:29 |
| melwitt | zzzeek: I see. I too, find confusing :) | 20:29 |
| mriedem | cfriesen: ok i wondered about the lock | 20:29 |
| cfriesen | mriedem: the lack of fair locking means that the events are likely to drop/regain the lock before the live migration code gets it | 20:29 |
| zzzeek | melwitt: ive tried for many years to get the docs / API to express this concept that when you have a URL in SQLAclhemy, you get a *factory*, and not the connection directly, but then with enginefacade in the mix, the bar for explaining is higher. | 20:29 |
| cfriesen | mriedem: adding a greenthread.sleep(0) in the event path helps a lot, but that only works if you're using the "blocking" rpc executor | 20:30 |
| melwitt | zzzeek: gotcha. thanks for explaining. I had been expecting something like a tcp socket pool where it would see us connecting to the same place repeatedly and reuse objects. but I think I understand now why it wouldn't be able to match those things up | 20:30 |
| mriedem | cfriesen: what is the 'blocking' rpc executor? | 20:30 |
| cfriesen | mriedem: oslo_messaging/_executors/impl_blocking.py | 20:32 |
| cfriesen | mriedem: default RPC executor in oslo messaging | 20:32 |
| mriedem | cfriesen: i see, https://docs.openstack.org/developer/oslo.messaging/executors.html | 20:34 |
| *** itlinux_ has joined #openstack-nova | 20:34 | |
| mriedem | cfriesen: well, we could potentially ignore the event if we know the instance is undergoing a certain task state transition | 20:34 |
| mriedem | cfriesen: looking at nova.rpc, we use eventlet for the executor | 20:36 |
| mriedem | executor='eventlet', | 20:36 |
| cfriesen | mriedem: we were just wondering internally whether we could ignore them if it was in the middle of a live migration. do you think the state contained in "instance" at the time it was pulled from the DB in ServerExternalEventsController would be recent enough or would we need to check the DB | 20:37 |
| *** itlinux has quit IRC | 20:37 | |
| mriedem | cfriesen: between the time that the api pulls the instance from the db and the time it rpc casts to the compute and checks the state, i don't imagine there would be a huge difference, but it does become a race and you could end up ignoring an event that you actually wanted to process | 20:38 |
| *** slaweq has joined #openstack-nova | 20:38 | |
| *** slaweq has quit IRC | 20:38 | |
| zzzeek | melwitt: oh, like an implicit global pool per URL. yeah, we don't do it that way. SQLA doesn't have any global registries. However, you're right that oslo.db *could* serve this purpose. it's just that a URL is bundled with all kinds of other things like pooling config, tranasction isolation, SQL mode stuff, encodings, etc. | 20:38 |
| *** jose-phillips has joined #openstack-nova | 20:38 | |
| cfriesen | mriedem: hmm...that's sort of odd, I wasn't seeing a bunch of events getting bunched up all waiting on the lock. In that case the yield isn't guaranteed to buy you anything and only "fair locks" would provide a guarantee | 20:39 |
| zzzeek | (sqla has some global registries but not linked to things like strings) | 20:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | 20:41 |
| *** slaweq has joined #openstack-nova | 20:41 | |
| melwitt | zzzeek: out of curiosity, is there a better way to do something like connecting to several different DBs? or is what we have reasonable | 20:41 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | 20:41 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | 20:41 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | 20:41 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | 20:41 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | 20:41 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | 20:41 |
| *** zz_dimtruck is now known as dimtruck | 20:41 | |
| zzzeek | melwitt: in here I see "db.create_context_manager()" which seems pretty clean. what determines the other settings for this CELL db, they are taken from the global settings? like how do we know it's utf-8 encoding, stuff like that ? | 20:42 |
| melwitt | zzzeek: they all use the same config, they're just clones of what we call the "main DB" they all have the same schema | 20:43 |
| dansmith | clone in the schema sense | 20:43 |
| melwitt | yeah, sorry | 20:43 |
| zzzeek | melwitt: yup. OK so oslo.db could have a featrue like this. like in enginefacade, you give it the URL and some flag, "use_existing" or something, pulls it from a dictionaar like you have here keyed to URL | 20:44 |
| *** gouthamr has quit IRC | 20:44 | |
| melwitt | zzzeek: okay, cool. thanks. just wanted a sanity check of whether there were more appropriate objects to use that we didn't use | 20:47 |
| sdague | efried: totally feel free to, it was a nice add | 20:48 |
| *** zul has quit IRC | 20:51 | |
| efried | sdague Pursuant to your colors-by-hash idea, it occurred to me that it might be more useful to do that by service, now that we've got converged logs. | 20:52 |
| sdague | efried: you don't have access that early | 20:53 |
| efried | Just thinking whether consumer is more likely to grep away by req-id or by service - and I'm thinking it's the former by a mile. | 20:53 |
| sdague | that's set by systemd itself | 20:53 |
| sdague | efried: you don't need to grep by service though | 20:53 |
| sdague | because -u does that for you | 20:53 |
| efried | sdague Yeah, exactly my point. | 20:54 |
| efried | It would be a visual break showing you when you're jumping services. | 20:54 |
| cfriesen | task_states.MIGRATING is only for live-migration, right? | 20:54 |
| sdague | right, so the reason I thought it would be nice to rainbow hash the req-id is that it becomes more clear when you have competing flows happening at the some time | 20:54 |
| sdague | most of our races are actually related to that | 20:54 |
| efried | sdague Makes sense. | 20:54 |
| sdague | it's potentially totally gratuitous, but it was enough to wonder "hey, that might be useful" | 20:55 |
| efried | sdague If it's just coloring the req-id string itself (not the whole message), it should be harmless at worst... unless you wind up printing black-on-black. | 20:55 |
| melwitt | cells meeting in a few min | 20:57 |
| sdague | efried: yeh, I was thinking just the req-id string | 20:57 |
| *** Vek has joined #openstack-nova | 21:00 | |
| *** catintheroof has quit IRC | 21:01 | |
| *** Apoorva_ has joined #openstack-nova | 21:01 | |
| *** armax has quit IRC | 21:02 | |
| *** thorst_afk has quit IRC | 21:03 | |
| *** rfolco has quit IRC | 21:03 | |
| *** jwcroppe has quit IRC | 21:03 | |
| *** dnovosel has quit IRC | 21:05 | |
| *** Apoorva has quit IRC | 21:05 | |
| *** Sukhdev has joined #openstack-nova | 21:05 | |
| *** hongbin has quit IRC | 21:08 | |
| *** gomarivera_ has joined #openstack-nova | 21:11 | |
| *** MVenesio has quit IRC | 21:11 | |
| *** itlinux_ has quit IRC | 21:14 | |
| *** gouthamr has joined #openstack-nova | 21:14 | |
| *** ijw has joined #openstack-nova | 21:15 | |
| *** gomarivera_ has quit IRC | 21:15 | |
| *** hongbin has joined #openstack-nova | 21:16 | |
| *** kylek3h has quit IRC | 21:16 | |
| *** artom has joined #openstack-nova | 21:17 | |
| *** ijw has quit IRC | 21:19 | |
| *** esberglu has quit IRC | 21:19 | |
| *** ijw has joined #openstack-nova | 21:20 | |
| *** ijw has quit IRC | 21:20 | |
| *** ijw has joined #openstack-nova | 21:20 | |
| *** mnestratov has joined #openstack-nova | 21:22 | |
| *** sdague has quit IRC | 21:22 | |
| *** kylek3h has joined #openstack-nova | 21:22 | |
| *** ijw has quit IRC | 21:25 | |
| *** ijw has joined #openstack-nova | 21:26 | |
| *** edmondsw has quit IRC | 21:27 | |
| *** edmondsw has joined #openstack-nova | 21:28 | |
| *** edmondsw_ has joined #openstack-nova | 21:31 | |
| *** edmondsw has quit IRC | 21:32 | |
| *** marst has quit IRC | 21:33 | |
| melwitt | dansmith: got a bunch of unit test fails on my patch bc I think I need to clear that cache between tests | 21:34 |
| dansmith | melwitt: yep | 21:34 |
| dansmith | just like we do for the others | 21:34 |
| *** ijw has quit IRC | 21:34 | |
| *** rfolco has joined #openstack-nova | 21:34 | |
| melwitt | I just grepped for CELLS in the tests and didn't find anything. where are you doing it? | 21:34 |
| melwitt | er, didn't find a clearing of it | 21:34 |
| *** ijw has joined #openstack-nova | 21:35 | |
| *** edmondsw_ has quit IRC | 21:36 | |
| melwitt | oh, derp, I grepped in nova/tests but it's in nova/test.py | 21:36 |
| melwitt | nvm | 21:36 |
| dansmith | yepo | 21:36 |
| *** avolkov has quit IRC | 21:36 | |
| *** ijw has quit IRC | 21:39 | |
| *** ijw has joined #openstack-nova | 21:40 | |
| cfriesen | mriedem: dansmith: would something like this seem okay to you? http://paste.openstack.org/show/609851/ I'm a little bit worried about _rollback_live_migration() which only calls setup_networks_on_host() and not get_instance_nw_info() like the network-changed notifier does | 21:41 |
| *** baoli_ has quit IRC | 21:41 | |
| *** burt has quit IRC | 21:41 | |
| *** mtanino_ has quit IRC | 21:44 | |
| *** ijw has quit IRC | 21:47 | |
| *** ijw has joined #openstack-nova | 21:48 | |
| *** Sukhdev has quit IRC | 21:50 | |
| *** ijw has quit IRC | 21:52 | |
| *** ijw has joined #openstack-nova | 21:52 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Cache database and message queue connection objects https://review.openstack.org/465042 | 21:53 |
| *** mnestratov has quit IRC | 21:53 | |
| *** ijw has quit IRC | 21:54 | |
| *** slaweq has quit IRC | 21:54 | |
| *** mdrabe has quit IRC | 21:55 | |
| *** ijw has joined #openstack-nova | 21:55 | |
| *** jamesdenton has quit IRC | 21:57 | |
| *** xyang1 has quit IRC | 21:58 | |
| *** rfolco has quit IRC | 21:58 | |
| *** thorst_afk has joined #openstack-nova | 22:00 | |
| mriedem | cfriesen: i don't really care for the instance.refresh thing | 22:02 |
| *** harlowja has joined #openstack-nova | 22:02 | |
| *** kylek3h has quit IRC | 22:12 | |
| *** dimtruck is now known as zz_dimtruck | 22:14 | |
| *** zz_dimtruck is now known as dimtruck | 22:14 | |
| *** harlowja has quit IRC | 22:18 | |
| *** thorst_afk has quit IRC | 22:20 | |
| *** dimtruck is now known as zz_dimtruck | 22:23 | |
| *** mlavalle has quit IRC | 22:29 | |
| *** Apoorva_ has quit IRC | 22:33 | |
| *** Apoorva has joined #openstack-nova | 22:33 | |
| *** itlinux has joined #openstack-nova | 22:33 | |
| *** suresh12 has joined #openstack-nova | 22:38 | |
| cburgess | Is it actually possible to select additional scheduler filters to use on the boot command? | 22:46 |
| cfriesen | mriedem: the point of that is to query from the DB the current task_state to see if it's migrating...got a better way to do that? | 22:46 |
| *** baoli has joined #openstack-nova | 22:48 | |
| *** baoli has quit IRC | 22:48 | |
| *** baoli has joined #openstack-nova | 22:49 | |
| dansmith | cfriesen: I don't see why we should skip refreshes altogether for migrating instances, nor do I think it's reasonable to do that heavy lookup every time | 22:49 |
| dansmith | we already send events to both host and destination during migration | 22:49 |
| *** itlinux has quit IRC | 22:49 | |
| dansmith | if you ask for any pre-review, can you do it in gerrit please? it's annoying to see only ten lines of context | 22:49 |
| cfriesen | dansmith: we've seen the refresh take 35+ seconds due to lock contention with the incoming network-change events when there are 16 vifs. the events kept on acquiring the lock even though the migration code tried to get it first. | 22:50 |
| cfriesen | dansmith: lack of fair locking was giving grief | 22:50 |
| *** eharney has quit IRC | 22:52 | |
| cfriesen | dansmith: with the 2-sec delay due to neutron batching, the DB lookup from instance.refresh() is way faster | 22:52 |
| cfriesen | dansmith: and the reason to skip them for live-migrating instances is that after the migration is done we're going to refresh the cache explicitly | 22:53 |
| dansmith | but for everyone else that's an extra db lookup on every event | 22:53 |
| *** zz_dimtruck is now known as dimtruck | 22:54 | |
| cfriesen | dansmith: got any suggestions? fair locking would help unblock the migration code but we'd still have the 2-sec neutron delays (and that seems serialized somehow so it takes ~8sec to finish getting all the network-changed events after a migration | 22:55 |
| *** erhudy has quit IRC | 22:55 | |
| mriedem | cfriesen: do you really need the refresh at all? | 22:56 |
| mriedem | what are the chances that the live migration completes and the task_state changes in the window of time between when the api gets the instances for the events and when you check in the compute? | 22:56 |
| cfriesen | mriedem: that's why I asked you earlier...seems like there'd be a small race window where we might ignore an event we should have processed | 22:58 |
| cfriesen | whether that would actually cause problems...I don't know | 22:58 |
| melwitt | cburgess: what do you mean by select? other than adding them to enabled_filters in your nova.conf? | 22:58 |
| *** itlinux has joined #openstack-nova | 22:59 | |
| cfriesen | mriedem: dansmith: to ignore it the instance would have been migrating at the time nova-api pulled the instance from the DB, so I *think* it should be valid to ignore the event | 23:00 |
| cburgess | melwitt Exactly. We have available_filters and default_filters in nova.conf. default gets used when you do a launch but is there a way as part of a boot to add a filter to that list or use a different filter? | 23:00 |
| cfriesen | cburgess: I don't see a way...get_filtered_hosts() calls self.filter_handler.get_filtered_objects(self.enabled_filters....) where self.enabled_filters is used as the list of filters to evaluate | 23:01 |
| melwitt | cburgess: you mean potentially use different filters for different boot requests? I don't think so | 23:01 |
| dansmith | cfriesen: I don't see why it's valid to ignore it ever, but especially not just because you might've lost the race where we already looked it up in the api | 23:02 |
| cfriesen | dansmith: the event handler for the network-changed message just calls self.network_api.get_instance_nw_info(). the live-migration code will call that explicitly when the migration finishes. | 23:04 |
| cfriesen | that's a relatively expensive call, so we'd like to minimize the number of times we call it | 23:04 |
| cfriesen | and we'd like to avoid the main migration code contending on the lock with the events | 23:05 |
| cburgess | melwitt cfriesen So whats what I thought. Which begs the question.. why do we have 2 config values? | 23:05 |
| melwitt | cburgess: I think the only way to get different behavior for different boot would be scheduler hints (and add a filter that handles that hint) or extra specs to do it via flavor | 23:05 |
| cburgess | melwitt Ahh that could be why hints. I haven't looked at that. | 23:06 |
| melwitt | cburgess: that's a good question ... checking | 23:06 |
| cfriesen | cburgess: 2 config values for what/ | 23:06 |
| cfriesen | ? | 23:06 |
| melwitt | like available filters vs enabled filters | 23:06 |
| cburgess | cfriesen scheduler_default_filters vs scheduler_available_filters | 23:07 |
| dansmith | cfriesen: some virt driver could need to wait for some update during live migration and need to see that refresh to proceed, no? | 23:07 |
| melwitt | I actually don't know why | 23:07 |
| dansmith | cfriesen: sounds like you want to fix the locking problem instead of just whack-a-mole'ing your immediate problem | 23:07 |
| *** liverpooler has quit IRC | 23:08 | |
| cfriesen | dansmith: there are a few issues: 1) lock isn't fair, 2) the call to neutron is expensive, 3) what does it buy us to call self.network_api.get_instance_nw_info() a bunch of times during a single live-migration? | 23:08 |
| dansmith | cfriesen: fix #1, skip duplicate instances as you are, and I just wrote a virt driver that depends on those refreshes to complete phases of my live migration. | 23:09 |
| *** lbragstad has quit IRC | 23:10 | |
| cburgess | melwitt Yeah ok thats what I thought.. superfluous... I'll keep digging into the hints thing but it really look to me like its redundant. | 23:12 |
| cfriesen | dansmith: would a virt driver ever use InstanceInfoCache? | 23:12 |
| *** gouthamr has quit IRC | 23:12 | |
| dansmith | cfriesen: sure, the one I just wrote does | 23:13 |
| melwitt | cburgess: yeah, I'm not seeing any clues as to why there are two options. maybe a legacy thing that used to have a purpose. jaypipes, bauzas, or edleafe might know | 23:13 |
| cfriesen | :) | 23:13 |
| *** mtanino has joined #openstack-nova | 23:13 | |
| cfriesen | dansmith: okay, so assuming we do the locking thing...do you think we would want to modify the standard lock (likely making it somewhat slower in the fastpath) or introduce a new "fair" lock and hope services switch to using it? | 23:14 |
| dansmith | cfriesen: there's lots of network_info walking throughout the xen driver at least.. | 23:15 |
| dansmith | cfriesen: I have no idea what you want to do, and would suggest you ask the lockutils people what they think | 23:15 |
| cfriesen | dansmith: I think it'd be nuts to rely on incoming network-changed events during a live migration since they can be processed in any order due to RPC and locking contention. But who knows what crazy stuff xen might do. | 23:16 |
| cfriesen | dansmith: fair enough | 23:16 |
| dansmith | if you're waiting for some property on a vif to become a value, I see no reason why that wouldn't be reasonable | 23:17 |
| dansmith | anyway, EOD and EOP for me | 23:17 |
| * dansmith & | 23:17 | |
| cfriesen | thanks for the feedback | 23:17 |
| *** dikonoor has joined #openstack-nova | 23:19 | |
| *** gouthamr has joined #openstack-nova | 23:20 | |
| *** thorst_afk has joined #openstack-nova | 23:20 | |
| *** claudiub has quit IRC | 23:23 | |
| *** seba has quit IRC | 23:24 | |
| * mriedem wonders what virt driver dan wrote | 23:24 | |
| mriedem | SteelyDanHypervisor? | 23:24 |
| *** jgrassler has quit IRC | 23:25 | |
| mriedem | i'm going to need someone to run the nova team meeting tomorrow... | 23:26 |
| cfriesen | 10thDanBlackBeltHypervisor | 23:26 |
| *** itlinux has quit IRC | 23:27 | |
| mriedem | also, i wonder if dan is aware of dan's opinions on out of tree virt drivers | 23:27 |
| mriedem | oomichi: were you interested in taking over either https://blueprints.launchpad.net/nova/+spec/live-migration-per-instance-timeout or https://blueprints.launchpad.net/nova/+spec/live-migration-force-after-timeout ? | 23:31 |
| *** crushil has quit IRC | 23:32 | |
| *** seba has joined #openstack-nova | 23:34 | |
| *** karthiks has quit IRC | 23:35 | |
| *** jgrassler has joined #openstack-nova | 23:36 | |
| *** dimtruck is now known as zz_dimtruck | 23:37 | |
| *** pewp has quit IRC | 23:38 | |
| *** pewp has joined #openstack-nova | 23:38 | |
| *** ijw has quit IRC | 23:38 | |
| *** ijw has joined #openstack-nova | 23:39 | |
| *** thorst_afk has quit IRC | 23:41 | |
| *** mtanino has quit IRC | 23:44 | |
| *** chyka has quit IRC | 23:45 | |
| *** chyka has joined #openstack-nova | 23:45 | |
| *** felipemonteiro has quit IRC | 23:49 | |
| *** chyka has quit IRC | 23:50 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Have scheduler claim resources https://review.openstack.org/463993 | 23:51 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Remove selecting host based on host_subset_size https://review.openstack.org/465174 | 23:51 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Make the method to put allocations public https://review.openstack.org/464086 | 23:51 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Don't delete allocation if instance being scheduled https://review.openstack.org/460177 | 23:51 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Pass a list of instance UUIDs to scheduler https://review.openstack.org/465171 | 23:51 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - Pass the number of hosts to return explicitly https://review.openstack.org/465767 | 23:51 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - return multiple hosts from select_destinations https://review.openstack.org/465768 | 23:51 |
| mriedem | melwitt: do you want to tweak the reno at all in this? https://review.openstack.org/#/c/465042/ | 23:51 |
| mriedem | if not, i'm +2 | 23:51 |
| melwitt | mriedem: yeah, I can change it to your suggest and flip the test assertions. I wasn't thinking when I copied them | 23:52 |
| melwitt | *suggestion | 23:52 |
| *** armax has joined #openstack-nova | 23:53 | |
| openstackgerrit | Hongbin Lu proposed openstack/nova master: Handle exception on adding secgroup https://review.openstack.org/465173 | 23:53 |
| cfriesen | mriedem: so it looks like _build_network_info_model() costs about 200ms plus about 125ms per port since we query each port separatly from neutron. and the refresh_cache lock is held the whole time | 23:55 |
| cfriesen | mriedem: I wonder if we could issues those requests in parallel for all the ports? | 23:56 |
| mriedem | cfriesen: we've talked for years about needing to do bulk queries from neutron | 23:56 |
| *** hongbin has quit IRC | 23:56 | |
| mriedem | well, | 23:57 |
| mriedem | where do we re-query per port, after we get the list of ports from data = client.list_ports(**search_opts) | 23:57 |
| cfriesen | _nw_info_get_ips and _nw_info_get_subnets both take noticeable time | 23:58 |
| cfriesen | for each port | 23:58 |
| mriedem | the first is going to iterate over all fixed_ips for a port and get floating IPs for each fixed_ip | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!