| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: NUMA-aware live migration https://review.openstack.org/552722 | 00:02 |
|---|---|---|
| *** Swami has quit IRC | 00:02 | |
| *** odyssey4me has quit IRC | 00:02 | |
| *** odyssey4me has joined #openstack-nova | 00:03 | |
| *** felipemonteiro__ has joined #openstack-nova | 00:07 | |
| *** suresh12 has quit IRC | 00:11 | |
| *** danpawlik has joined #openstack-nova | 00:18 | |
| *** tetsuro has joined #openstack-nova | 00:21 | |
| *** danpawlik has quit IRC | 00:22 | |
| *** suresh12 has joined #openstack-nova | 00:26 | |
| *** salv-orlando has joined #openstack-nova | 00:27 | |
| *** mvk has joined #openstack-nova | 00:27 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: SchedulerReportClient.update_from_provider_tree https://review.openstack.org/533821 | 00:27 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use update_provider_tree from resource tracker https://review.openstack.org/520246 | 00:27 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Fix nits in update_provider_tree series https://review.openstack.org/531260 | 00:27 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Move refresh time from report client to prov tree https://review.openstack.org/535517 | 00:27 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Make generation optional in ProviderTree https://review.openstack.org/539324 | 00:27 |
| openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Add nested resources to server moving tests https://review.openstack.org/527728 | 00:27 |
| efried | blayum | 00:27 |
| *** salv-orlando has quit IRC | 00:32 | |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Mention (no) granular support for image traits https://review.openstack.org/554305 | 00:32 |
| *** Zames has joined #openstack-nova | 00:36 | |
| *** felipemonteiro__ has quit IRC | 00:40 | |
| *** jichen has joined #openstack-nova | 00:41 | |
| *** hongbin has joined #openstack-nova | 00:44 | |
| *** zhurong has joined #openstack-nova | 00:45 | |
| *** danpawlik has joined #openstack-nova | 00:52 | |
| *** yamamoto has joined #openstack-nova | 00:52 | |
| tetsuro | Good morning. | 00:53 |
| *** phuongnh has joined #openstack-nova | 00:56 | |
| *** danpawlik has quit IRC | 00:57 | |
| *** yamamoto has quit IRC | 00:57 | |
| Spazmotic | +1 more blayum? :p | 00:58 |
| Spazmotic | Morning | 00:58 |
| efried | Hi, and goodbye. The wife's getting pretty antsy. | 01:00 |
| * efried waves | 01:00 | |
| * Spazmotic waves | 01:00 | |
| *** zhaochao has joined #openstack-nova | 01:00 | |
| *** liverpooler has quit IRC | 01:00 | |
| *** crushil has joined #openstack-nova | 01:02 | |
| *** liverpooler has joined #openstack-nova | 01:05 | |
| *** vladikr has quit IRC | 01:05 | |
| *** vladikr has joined #openstack-nova | 01:06 | |
| openstackgerrit | Merged openstack/nova master: Add disabled column to cell_mappings table. https://review.openstack.org/552505 | 01:07 |
| openstackgerrit | Merged openstack/nova master: libvirt: move get_numa_memnode in designer module https://review.openstack.org/554850 | 01:07 |
| openstackgerrit | Merged openstack/nova master: libvirt: move vpu_realtime_scheduler in designer https://review.openstack.org/554851 | 01:07 |
| *** fragatina has quit IRC | 01:09 | |
| *** hshiina has joined #openstack-nova | 01:09 | |
| sean-k-mooney[m] | mriedem: bauzas can ye take a look at this backport for os-vif when ye get time https://review.openstack.org/#/c/531465/1 its minor but has been up for a while. just wondering if i should keep it open or abandon it. | 01:10 |
| *** fragatina has joined #openstack-nova | 01:11 | |
| sean-k-mooney[m] | !cmd freenode AWAY until noon gmt | 01:11 |
| openstack | sean-k-mooney[m]: Error: "cmd" is not a valid command. | 01:11 |
| sean-k-mooney[m] | data did not work... | 01:11 |
| sean-k-mooney[m] | *that | 01:11 |
| *** tiendc has joined #openstack-nova | 01:11 | |
| *** tbachman has quit IRC | 01:11 | |
| *** fragatin_ has joined #openstack-nova | 01:12 | |
| *** fragatina has quit IRC | 01:15 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:15 | |
| *** fragatin_ has quit IRC | 01:16 | |
| *** tbachman has joined #openstack-nova | 01:17 | |
| *** Zames has quit IRC | 01:20 | |
| *** Guest90893 has quit IRC | 01:22 | |
| openstackgerrit | zhufl proposed openstack/nova master: Fix api-ref: nova image-meta is deprecated from 2.39 https://review.openstack.org/554813 | 01:22 |
| openstackgerrit | zhufl proposed openstack/nova master: Fix api-ref: nova image-meta is deprecated from 2.39 https://review.openstack.org/554813 | 01:24 |
| *** hiro-kobayashi has joined #openstack-nova | 01:24 | |
| *** liusheng has quit IRC | 01:26 | |
| *** liusheng has joined #openstack-nova | 01:27 | |
| *** suresh12 has quit IRC | 01:27 | |
| *** salv-orlando has joined #openstack-nova | 01:27 | |
| *** tbachman has quit IRC | 01:29 | |
| *** harlowja has quit IRC | 01:31 | |
| *** danpawlik has joined #openstack-nova | 01:31 | |
| *** salv-orlando has quit IRC | 01:32 | |
| *** Dinesh_Bhor has quit IRC | 01:35 | |
| *** oomichi has quit IRC | 01:35 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:36 | |
| *** danpawlik has quit IRC | 01:36 | |
| *** gjayavelu has quit IRC | 01:40 | |
| *** wolverineav has quit IRC | 01:43 | |
| *** wolverineav has joined #openstack-nova | 01:44 | |
| *** danpawlik has joined #openstack-nova | 01:47 | |
| *** wolverineav has quit IRC | 01:48 | |
| *** tbachman has joined #openstack-nova | 01:49 | |
| *** gaoyan has joined #openstack-nova | 01:50 | |
| *** danpawlik has quit IRC | 01:51 | |
| *** yamamoto has joined #openstack-nova | 01:53 | |
| *** hoonetorg has quit IRC | 01:56 | |
| *** hoonetorg has joined #openstack-nova | 01:57 | |
| *** yamamoto has quit IRC | 01:59 | |
| openstackgerrit | zhufl proposed openstack/nova master: Fix api-ref: nova image-meta is deprecated from 2.39 https://review.openstack.org/554813 | 02:02 |
| *** zhurong has quit IRC | 02:02 | |
| *** Tom-Tom has joined #openstack-nova | 02:02 | |
| *** suresh12 has joined #openstack-nova | 02:06 | |
| *** mriedem has quit IRC | 02:07 | |
| *** gjayavelu has joined #openstack-nova | 02:07 | |
| *** gjayavelu has quit IRC | 02:08 | |
| *** danpawlik has joined #openstack-nova | 02:08 | |
| *** wolverineav has joined #openstack-nova | 02:10 | |
| *** suresh12 has quit IRC | 02:10 | |
| *** danpawlik has quit IRC | 02:13 | |
| *** wolverineav has quit IRC | 02:14 | |
| *** wolverineav has joined #openstack-nova | 02:16 | |
| *** owalsh_ has joined #openstack-nova | 02:16 | |
| *** owalsh has quit IRC | 02:20 | |
| *** idlemind has quit IRC | 02:23 | |
| *** idlemind has joined #openstack-nova | 02:24 | |
| *** Spaz-Work has quit IRC | 02:24 | |
| *** Spaz-Work has joined #openstack-nova | 02:25 | |
| *** salv-orlando has joined #openstack-nova | 02:28 | |
| *** yingjun has joined #openstack-nova | 02:32 | |
| *** salv-orlando has quit IRC | 02:34 | |
| *** _ix has joined #openstack-nova | 02:36 | |
| *** yamamoto has joined #openstack-nova | 02:37 | |
| *** edmondsw has joined #openstack-nova | 02:42 | |
| *** danpawlik has joined #openstack-nova | 02:44 | |
| *** yamamoto has quit IRC | 02:46 | |
| *** wolverineav has quit IRC | 02:46 | |
| *** phuongnh has quit IRC | 02:48 | |
| *** phuongnh has joined #openstack-nova | 02:48 | |
| *** andreas_s has joined #openstack-nova | 02:48 | |
| *** danpawlik has quit IRC | 02:49 | |
| *** edmondsw has quit IRC | 02:49 | |
| *** germs has quit IRC | 02:52 | |
| *** germs has joined #openstack-nova | 02:52 | |
| *** germs has quit IRC | 02:52 | |
| *** germs has joined #openstack-nova | 02:52 | |
| *** germs has quit IRC | 02:52 | |
| *** germs has joined #openstack-nova | 02:53 | |
| *** germs has quit IRC | 02:53 | |
| *** germs has joined #openstack-nova | 02:53 | |
| *** andreas_s has quit IRC | 02:53 | |
| *** yamamoto has joined #openstack-nova | 02:56 | |
| *** psachin has joined #openstack-nova | 02:56 | |
| *** zhurong has joined #openstack-nova | 02:56 | |
| *** Tom-Tom has quit IRC | 02:58 | |
| *** Tom-Tom has joined #openstack-nova | 02:59 | |
| *** Tom-Tom has quit IRC | 03:03 | |
| *** Tom-Tom has joined #openstack-nova | 03:04 | |
| *** bkopilov has quit IRC | 03:06 | |
| *** phuongnh has quit IRC | 03:07 | |
| *** phuongnh has joined #openstack-nova | 03:08 | |
| *** sree has joined #openstack-nova | 03:09 | |
| *** dave-mccowan has quit IRC | 03:11 | |
| *** amodi has quit IRC | 03:12 | |
| *** annp has joined #openstack-nova | 03:14 | |
| *** AlexeyAbashkin has joined #openstack-nova | 03:17 | |
| *** danpawlik has joined #openstack-nova | 03:18 | |
| *** AlexeyAbashkin has quit IRC | 03:21 | |
| Spaz-Work | Morning again :P | 03:22 |
| *** danpawlik has quit IRC | 03:23 | |
| *** _ix has quit IRC | 03:23 | |
| *** phuongnh has quit IRC | 03:30 | |
| *** hongbin has quit IRC | 03:30 | |
| *** salv-orlando has joined #openstack-nova | 03:30 | |
| *** Tom-Tom has quit IRC | 03:31 | |
| *** harlowja has joined #openstack-nova | 03:32 | |
| openstackgerrit | Jie Li proposed openstack/nova-specs master: Support volume-backed server rebuild https://review.openstack.org/532407 | 03:32 |
| *** Tom-Tom has joined #openstack-nova | 03:33 | |
| *** Tom-Tom_ has joined #openstack-nova | 03:34 | |
| *** salv-orlando has quit IRC | 03:35 | |
| *** Tom-Tom has quit IRC | 03:38 | |
| yikun | Anyone knows why NUMACell obj add the fields (1.1, 1.2) but without obj_make_compatible changing? | 03:42 |
| yikun | https://github.com/openstack/nova/blob/master/nova/objects/numa.py#L51-L53 | 03:43 |
| yikun | It seems only this one is an exception in all Nova object. | 03:43 |
| *** hiro-kobayashi has quit IRC | 03:49 | |
| *** danpawlik has joined #openstack-nova | 03:51 | |
| *** harlowja has quit IRC | 03:54 | |
| *** fragatina has joined #openstack-nova | 03:54 | |
| *** tuanla____ has joined #openstack-nova | 03:55 | |
| *** danpawlik has quit IRC | 03:56 | |
| *** amodi has joined #openstack-nova | 03:59 | |
| *** udesale has joined #openstack-nova | 04:03 | |
| *** edmondsw has joined #openstack-nova | 04:05 | |
| *** tidwellr has joined #openstack-nova | 04:06 | |
| *** tidwellr_ has joined #openstack-nova | 04:08 | |
| *** tidwellr has quit IRC | 04:08 | |
| *** edmondsw has quit IRC | 04:10 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: trivial: omit condition evaluations https://review.openstack.org/545248 | 04:14 |
| *** tidwellr_ has quit IRC | 04:17 | |
| *** hshiina has quit IRC | 04:21 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: WIP: Add host info to instance action events https://review.openstack.org/555146 | 04:21 |
| *** hshiina has joined #openstack-nova | 04:21 | |
| *** bkopilov has joined #openstack-nova | 04:23 | |
| openstackgerrit | Merged openstack/nova master: Preserve multiattach flag when refreshing connection_info https://review.openstack.org/554667 | 04:28 |
| *** salv-orlando has joined #openstack-nova | 04:31 | |
| *** Dinesh_Bhor has quit IRC | 04:34 | |
| *** salv-orlando has quit IRC | 04:36 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:36 | |
| *** zhurong has quit IRC | 04:45 | |
| *** felipemonteiro__ has joined #openstack-nova | 04:45 | |
| *** crushil has quit IRC | 04:46 | |
| *** germs has quit IRC | 04:49 | |
| *** germs has joined #openstack-nova | 04:49 | |
| *** germs has quit IRC | 04:49 | |
| *** germs has joined #openstack-nova | 04:49 | |
| *** germs has quit IRC | 04:54 | |
| *** amodi has quit IRC | 04:55 | |
| *** abhishekk has joined #openstack-nova | 04:55 | |
| *** hiro-kobayashi has joined #openstack-nova | 04:56 | |
| *** felipemonteiro__ has quit IRC | 04:57 | |
| *** gjayavelu has joined #openstack-nova | 04:59 | |
| *** Dinesh__Bhor has joined #openstack-nova | 05:03 | |
| *** Dinesh_Bhor has quit IRC | 05:03 | |
| *** suresh12 has joined #openstack-nova | 05:03 | |
| *** isssp has joined #openstack-nova | 05:03 | |
| *** tssurya has joined #openstack-nova | 05:04 | |
| *** idlemind has quit IRC | 05:05 | |
| *** burned has quit IRC | 05:06 | |
| *** tianhui_ has quit IRC | 05:07 | |
| *** tssurya has quit IRC | 05:08 | |
| *** lpetrut has joined #openstack-nova | 05:08 | |
| *** isssp has quit IRC | 05:09 | |
| *** isssp has joined #openstack-nova | 05:12 | |
| *** sridharg has joined #openstack-nova | 05:14 | |
| *** ratailor has joined #openstack-nova | 05:14 | |
| *** imacdonn has quit IRC | 05:15 | |
| *** imacdonn has joined #openstack-nova | 05:15 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: add check before adding cpus to cpuset_reserved https://review.openstack.org/539865 | 05:16 |
| *** moshele has joined #openstack-nova | 05:16 | |
| openstackgerrit | jichenjc proposed openstack/nova master: fix race condition of instance host https://review.openstack.org/494458 | 05:24 |
| *** gyankum has joined #openstack-nova | 05:26 | |
| *** jaosorior_ is now known as jaosorior | 05:26 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Move placement test cases from db to placement https://review.openstack.org/553149 | 05:28 |
| *** salv-orlando has joined #openstack-nova | 05:29 | |
| *** wolverineav has joined #openstack-nova | 05:30 | |
| *** rcernin has quit IRC | 05:34 | |
| *** suresh12 has quit IRC | 05:35 | |
| *** udesale_ has joined #openstack-nova | 05:36 | |
| *** zhurong has joined #openstack-nova | 05:37 | |
| *** udesale__ has joined #openstack-nova | 05:37 | |
| *** tetsuro has left #openstack-nova | 05:39 | |
| *** udesale has quit IRC | 05:39 | |
| *** udesale_ has quit IRC | 05:41 | |
| *** wolverineav has quit IRC | 05:41 | |
| *** wolverineav has joined #openstack-nova | 05:42 | |
| *** Tom-Tom_ has quit IRC | 05:43 | |
| *** Tom-Tom has joined #openstack-nova | 05:44 | |
| *** wolverineav has quit IRC | 05:46 | |
| *** rcernin has joined #openstack-nova | 05:46 | |
| *** Tom-Tom has quit IRC | 05:48 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Avoid live migrate to same host https://review.openstack.org/542689 | 05:48 |
| *** links has joined #openstack-nova | 05:53 | |
| *** edmondsw has joined #openstack-nova | 05:54 | |
| *** Tom-Tom has joined #openstack-nova | 05:58 | |
| *** edmondsw has quit IRC | 05:59 | |
| *** Tom-Tom has quit IRC | 06:00 | |
| *** lpetrut has quit IRC | 06:03 | |
| *** Tom-Tom has joined #openstack-nova | 06:03 | |
| *** Tom-Tom_ has joined #openstack-nova | 06:06 | |
| *** Tom-Tom_ has quit IRC | 06:06 | |
| *** rcernin has quit IRC | 06:06 | |
| *** Tom-Tom_ has joined #openstack-nova | 06:08 | |
| *** Tom-Tom has quit IRC | 06:08 | |
| *** rcernin has joined #openstack-nova | 06:08 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Not instance to ERROR if set_admin_password failed https://review.openstack.org/555160 | 06:10 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/548772 | 06:10 |
| *** lpetrut has joined #openstack-nova | 06:12 | |
| *** Dinesh__Bhor has quit IRC | 06:12 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Trivial fix: move log to earlier https://review.openstack.org/555164 | 06:14 |
| *** Dinesh__Bhor has joined #openstack-nova | 06:16 | |
| *** AlexeyAbashkin has joined #openstack-nova | 06:16 | |
| *** ratailor has quit IRC | 06:17 | |
| *** ratailor has joined #openstack-nova | 06:17 | |
| *** rgb00 has joined #openstack-nova | 06:19 | |
| *** lucas-afk has quit IRC | 06:20 | |
| *** fragatina has quit IRC | 06:21 | |
| *** AlexeyAbashkin has quit IRC | 06:21 | |
| *** bswrchrd has quit IRC | 06:22 | |
| *** kholkina has joined #openstack-nova | 06:23 | |
| *** lucasagomes has joined #openstack-nova | 06:23 | |
| *** threestrands has quit IRC | 06:25 | |
| openstackgerrit | Merged openstack/nova stable/queens: Handle not found error on taking snapshot https://review.openstack.org/550660 | 06:30 |
| *** lpetrut has quit IRC | 06:33 | |
| *** zhurong has quit IRC | 06:34 | |
| *** sidx64 has joined #openstack-nova | 06:35 | |
| *** lajoskatona has joined #openstack-nova | 06:35 | |
| *** zer0c00l has joined #openstack-nova | 06:42 | |
| *** Zames has joined #openstack-nova | 06:43 | |
| *** tianhui has joined #openstack-nova | 06:44 | |
| openstackgerrit | pippo proposed openstack/nova master: Update links in README https://review.openstack.org/555169 | 06:45 |
| zer0c00l | if my nova endpoint is at https://novaapi.example.com:4443/nova_api, when i curl that endpoint i get something like http://novaapi.example.com/v2/ instead of https://novaapi.example.com/nova_api/v2 | 06:45 |
| zer0c00l | why is that? is that because some kind of misconfiguration or is it expected | 06:45 |
| *** ccamacho has quit IRC | 06:45 | |
| *** Zames has quit IRC | 06:46 | |
| *** udesale_ has joined #openstack-nova | 06:48 | |
| *** udesale__ has quit IRC | 06:50 | |
| *** claudiub has joined #openstack-nova | 06:57 | |
| *** mayur_ind has joined #openstack-nova | 06:58 | |
| mayur_ind | hi folks | 07:00 |
| *** Elixer_dota has joined #openstack-nova | 07:00 | |
| mayur_ind | getting following error in nova http://paste.openstack.org/show/708826/ | 07:01 |
| mayur_ind | what may be the reason for this, is there way to solve it.? | 07:01 |
| *** sahid has joined #openstack-nova | 07:02 | |
| *** masber has quit IRC | 07:05 | |
| *** masber has joined #openstack-nova | 07:06 | |
| *** sar has joined #openstack-nova | 07:08 | |
| *** sidx64_ has joined #openstack-nova | 07:13 | |
| *** zhurong has joined #openstack-nova | 07:14 | |
| *** sidx64 has quit IRC | 07:14 | |
| *** links has quit IRC | 07:15 | |
| *** sidx64 has joined #openstack-nova | 07:16 | |
| *** sidx64_ has quit IRC | 07:17 | |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: Initial change set of z/VM driver https://review.openstack.org/523387 | 07:18 |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: Spawn and destroy function of z/VM driver https://review.openstack.org/527658 | 07:18 |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: add snapshot function https://review.openstack.org/534240 | 07:18 |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: add power actions https://review.openstack.org/543340 | 07:18 |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: add get console output https://review.openstack.org/543344 | 07:18 |
| *** alexchadin has joined #openstack-nova | 07:19 | |
| *** andreas_s has joined #openstack-nova | 07:21 | |
| *** belmoreira has joined #openstack-nova | 07:22 | |
| *** links has joined #openstack-nova | 07:23 | |
| *** andreas_s has quit IRC | 07:25 | |
| *** claudiub has quit IRC | 07:25 | |
| *** isssp has quit IRC | 07:25 | |
| *** jichen has quit IRC | 07:25 | |
| *** jaosorior has quit IRC | 07:25 | |
| *** Hazelesque has quit IRC | 07:25 | |
| *** slaweq has quit IRC | 07:25 | |
| *** sambetts|afk has quit IRC | 07:25 | |
| *** test222__ has quit IRC | 07:25 | |
| *** markmc has quit IRC | 07:25 | |
| *** rajinir has quit IRC | 07:25 | |
| *** Anticimex has quit IRC | 07:25 | |
| *** karimull has quit IRC | 07:25 | |
| *** Kvisle has quit IRC | 07:25 | |
| *** rajinir_ has joined #openstack-nova | 07:26 | |
| *** rajinir_ has quit IRC | 07:26 | |
| *** rajinir_ has joined #openstack-nova | 07:26 | |
| *** rajinir_ is now known as rajinir | 07:26 | |
| *** jichen has joined #openstack-nova | 07:26 | |
| *** Hazelesque has joined #openstack-nova | 07:26 | |
| *** slaweq has joined #openstack-nova | 07:26 | |
| *** jaosorior has joined #openstack-nova | 07:26 | |
| *** Kvisle has joined #openstack-nova | 07:26 | |
| *** Hazelesque has quit IRC | 07:26 | |
| *** Hazelesque has joined #openstack-nova | 07:26 | |
| *** isssp has joined #openstack-nova | 07:26 | |
| openstackgerrit | Merged openstack/nova master: Add unit tests for EmulatorThreadsTestCase https://review.openstack.org/538699 | 07:27 |
| *** karimull has joined #openstack-nova | 07:27 | |
| *** andreas_s has joined #openstack-nova | 07:27 | |
| *** markmc has joined #openstack-nova | 07:27 | |
| *** test222__ has joined #openstack-nova | 07:27 | |
| *** sambetts_ has joined #openstack-nova | 07:29 | |
| *** rcernin has quit IRC | 07:31 | |
| *** afaranha has joined #openstack-nova | 07:32 | |
| *** salv-orlando has quit IRC | 07:33 | |
| *** markvoelker has quit IRC | 07:35 | |
| *** ccamacho has joined #openstack-nova | 07:38 | |
| *** ralonsoh has joined #openstack-nova | 07:39 | |
| *** edmondsw has joined #openstack-nova | 07:42 | |
| *** pcaruana has joined #openstack-nova | 07:42 | |
| *** pcaruana has quit IRC | 07:44 | |
| *** pcaruana has joined #openstack-nova | 07:44 | |
| *** pcaruana has quit IRC | 07:45 | |
| *** danpawlik has joined #openstack-nova | 07:45 | |
| *** pcaruana has joined #openstack-nova | 07:45 | |
| *** edmondsw has quit IRC | 07:46 | |
| *** pcaruana has quit IRC | 07:47 | |
| *** pcaruana has joined #openstack-nova | 07:47 | |
| *** pcaruana has quit IRC | 07:48 | |
| *** pcaruana has joined #openstack-nova | 07:48 | |
| *** pcaruana has quit IRC | 07:50 | |
| *** pcaruana has joined #openstack-nova | 07:50 | |
| *** masber has quit IRC | 07:51 | |
| *** pcaruana has quit IRC | 07:51 | |
| *** pcaruana has joined #openstack-nova | 07:51 | |
| *** pcaruana has quit IRC | 07:53 | |
| *** pcaruana has joined #openstack-nova | 07:53 | |
| *** pcaruana has quit IRC | 07:54 | |
| *** pcaruana has joined #openstack-nova | 07:55 | |
| *** AlexeyAbashkin has joined #openstack-nova | 07:56 | |
| *** pcaruana has quit IRC | 07:56 | |
| *** Tom-Tom_ has quit IRC | 07:56 | |
| *** Tom-Tom has joined #openstack-nova | 07:57 | |
| *** belmorei_ has joined #openstack-nova | 07:57 | |
| *** Tom-Tom_ has joined #openstack-nova | 07:58 | |
| *** ispp has joined #openstack-nova | 07:58 | |
| *** asettle has quit IRC | 07:59 | |
| *** belmoreira has quit IRC | 07:59 | |
| *** andymccr has quit IRC | 07:59 | |
| *** Roamer` has quit IRC | 07:59 | |
| *** isssp has quit IRC | 08:00 | |
| *** Roamer` has joined #openstack-nova | 08:01 | |
| *** Tom-Tom has quit IRC | 08:02 | |
| *** belmorei_ has quit IRC | 08:02 | |
| *** belmore__ has joined #openstack-nova | 08:02 | |
| *** lpetrut has joined #openstack-nova | 08:03 | |
| *** hoangcx has quit IRC | 08:03 | |
| *** hoangcx has joined #openstack-nova | 08:04 | |
| *** lpetrut_ has joined #openstack-nova | 08:04 | |
| *** lpetrut has quit IRC | 08:04 | |
| *** andymccr has joined #openstack-nova | 08:05 | |
| *** asettle has joined #openstack-nova | 08:06 | |
| *** pcaruana has joined #openstack-nova | 08:06 | |
| *** asettle is now known as Guest66969 | 08:06 | |
| *** FL1SK has quit IRC | 08:07 | |
| *** pcaruana has quit IRC | 08:07 | |
| *** pcaruana has joined #openstack-nova | 08:08 | |
| *** pcaruana has quit IRC | 08:09 | |
| *** pcaruana has joined #openstack-nova | 08:09 | |
| *** pcaruana has quit IRC | 08:10 | |
| *** pcaruana has joined #openstack-nova | 08:11 | |
| *** tesseract has joined #openstack-nova | 08:11 | |
| *** masber has joined #openstack-nova | 08:11 | |
| *** pcaruana has quit IRC | 08:12 | |
| *** pcaruana has joined #openstack-nova | 08:13 | |
| *** pcaruana has quit IRC | 08:15 | |
| *** pcaruana has joined #openstack-nova | 08:15 | |
| *** pcaruana has quit IRC | 08:16 | |
| *** AlexeyAbashkin has quit IRC | 08:17 | |
| *** pcaruana has joined #openstack-nova | 08:18 | |
| *** masber has quit IRC | 08:18 | |
| *** AlexeyAbashkin has joined #openstack-nova | 08:18 | |
| *** pcaruana has quit IRC | 08:20 | |
| *** pcaruana has joined #openstack-nova | 08:20 | |
| *** sidx64 has quit IRC | 08:20 | |
| *** pcaruana has quit IRC | 08:21 | |
| *** pcaruana has joined #openstack-nova | 08:21 | |
| *** pcaruana has quit IRC | 08:22 | |
| *** ragiman has joined #openstack-nova | 08:23 | |
| *** tuanla____ has quit IRC | 08:25 | |
| *** tuanla____ has joined #openstack-nova | 08:26 | |
| *** alexchadin has quit IRC | 08:27 | |
| *** pcaruana has joined #openstack-nova | 08:29 | |
| *** sidx64 has joined #openstack-nova | 08:30 | |
| *** pcaruana has quit IRC | 08:30 | |
| *** tianhui_ has joined #openstack-nova | 08:31 | |
| *** tianhui has quit IRC | 08:32 | |
| *** sidx64 has quit IRC | 08:33 | |
| *** salv-orlando has joined #openstack-nova | 08:34 | |
| *** markvoelker has joined #openstack-nova | 08:34 | |
| *** hoangcx has quit IRC | 08:35 | |
| *** hoangcx has joined #openstack-nova | 08:36 | |
| *** pcaruana has joined #openstack-nova | 08:36 | |
| *** pcaruana has quit IRC | 08:37 | |
| *** pcaruana has joined #openstack-nova | 08:38 | |
| *** pcaruana has quit IRC | 08:39 | |
| *** pcaruana has joined #openstack-nova | 08:40 | |
| *** salv-orlando has quit IRC | 08:40 | |
| *** pcaruana has quit IRC | 08:40 | |
| *** pcaruana has joined #openstack-nova | 08:41 | |
| *** pcaruana has quit IRC | 08:41 | |
| *** damien_r has joined #openstack-nova | 08:43 | |
| *** tianhui_ has quit IRC | 08:43 | |
| *** sapd has quit IRC | 08:44 | |
| *** amoralej|off is now known as amoralej | 08:44 | |
| *** sapd has joined #openstack-nova | 08:45 | |
| *** tuanla____ has quit IRC | 08:46 | |
| *** hiro-kobayashi has quit IRC | 08:46 | |
| *** tianhui has joined #openstack-nova | 08:46 | |
| *** tuanla____ has joined #openstack-nova | 08:47 | |
| *** avolkov has joined #openstack-nova | 08:48 | |
| *** jpena|off is now known as jpena | 08:48 | |
| *** tianhui has quit IRC | 08:51 | |
| *** tianhui has joined #openstack-nova | 08:55 | |
| *** masber has joined #openstack-nova | 08:55 | |
| *** pcaruana has joined #openstack-nova | 09:02 | |
| *** mdnadeem has joined #openstack-nova | 09:03 | |
| *** pcaruana has quit IRC | 09:03 | |
| *** pcaruana has joined #openstack-nova | 09:03 | |
| *** pcaruana has quit IRC | 09:05 | |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: Spawn and destroy function of z/VM driver https://review.openstack.org/527658 | 09:06 |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: add snapshot function https://review.openstack.org/534240 | 09:06 |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: add power actions https://review.openstack.org/543340 | 09:06 |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: add get console output https://review.openstack.org/543344 | 09:06 |
| *** hshiina has quit IRC | 09:11 | |
| *** owalsh_ is now known as owalsh | 09:13 | |
| *** zhurong has quit IRC | 09:16 | |
| *** masber has quit IRC | 09:17 | |
| *** gaoyan has quit IRC | 09:20 | |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: add snapshot function https://review.openstack.org/534240 | 09:21 |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: add power actions https://review.openstack.org/543340 | 09:21 |
| openstackgerrit | jichenjc proposed openstack/nova master: z/VM Driver: add get console output https://review.openstack.org/543344 | 09:21 |
| *** Dinesh__Bhor has quit IRC | 09:25 | |
| gmann_ | mayur_ind: will be good if you attach logs etc on bug. with 500 it is not possible to check what went wring | 09:26 |
| *** masber has joined #openstack-nova | 09:28 | |
| *** mdbooth has joined #openstack-nova | 09:28 | |
| *** edmondsw has joined #openstack-nova | 09:30 | |
| *** mayur_ind has quit IRC | 09:31 | |
| *** hemna_ has quit IRC | 09:31 | |
| *** alexchadin has joined #openstack-nova | 09:34 | |
| *** edmondsw has quit IRC | 09:35 | |
| *** salv-orlando has joined #openstack-nova | 09:35 | |
| Kevin_Zheng | gmann_ Hi, could you spare a few minutes and help me with some test issue? I'm working on some functional tests that have to verify the request_id, it seems that the req['openstack.request_id'] is not translated to context.request_id in functional tests, could you tell me where do we generate the context in functional tests? I have difficulties to find it | 09:37 |
| *** salv-orlando has quit IRC | 09:38 | |
| *** salv-orlando has joined #openstack-nova | 09:38 | |
| *** yingjun has quit IRC | 09:41 | |
| *** mgoddard has joined #openstack-nova | 09:43 | |
| *** jichen has quit IRC | 09:44 | |
| Kevin_Zheng | gmann_ Never mind, I got it | 09:45 |
| gmann_ | Kevin_Zheng: ohk :), in QA meeting which is about to close | 09:45 |
| *** Guest66969 is now known as asettle | 09:46 | |
| *** bkopilov has quit IRC | 09:46 | |
| *** ragiman has quit IRC | 09:46 | |
| *** chyka has joined #openstack-nova | 09:47 | |
| *** chyka has quit IRC | 09:51 | |
| *** ragiman has joined #openstack-nova | 10:02 | |
| *** mgoddard has quit IRC | 10:05 | |
| *** sidx64 has joined #openstack-nova | 10:07 | |
| *** liverpooler has quit IRC | 10:10 | |
| *** mdnadeem has quit IRC | 10:10 | |
| *** tssurya has joined #openstack-nova | 10:11 | |
| openstackgerrit | sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | 10:11 |
| *** udesale_ has quit IRC | 10:16 | |
| *** gjayavelu has quit IRC | 10:17 | |
| *** FL1SK has joined #openstack-nova | 10:17 | |
| *** alexchadin has quit IRC | 10:26 | |
| *** masuberu has joined #openstack-nova | 10:27 | |
| *** sree has quit IRC | 10:30 | |
| *** abhishekk has quit IRC | 10:30 | |
| *** sree has joined #openstack-nova | 10:30 | |
| *** masber has quit IRC | 10:30 | |
| *** alexchadin has joined #openstack-nova | 10:31 | |
| *** bjolo has quit IRC | 10:32 | |
| *** Tom-Tom_ has quit IRC | 10:32 | |
| *** Tom-Tom has joined #openstack-nova | 10:33 | |
| *** derekh has joined #openstack-nova | 10:33 | |
| *** sree has quit IRC | 10:34 | |
| *** dtantsur|afk is now known as dtantsur | 10:36 | |
| *** mgoddard has joined #openstack-nova | 10:37 | |
| *** Tom-Tom has quit IRC | 10:38 | |
| *** bjolo has joined #openstack-nova | 10:39 | |
| *** vladikr has quit IRC | 10:39 | |
| gmann_ | gibi: can you check if all ok from notification wise - https://review.openstack.org/#/c/554090/2 | 10:44 |
| gibi | gmann_: I opened it and I will try to check it today | 10:49 |
| *** _ix has joined #openstack-nova | 10:54 | |
| *** AlexeyAbashkin has quit IRC | 11:00 | |
| *** AlexeyAbashkin has joined #openstack-nova | 11:00 | |
| *** logan- has quit IRC | 11:03 | |
| *** logan- has joined #openstack-nova | 11:03 | |
| *** sidx64 has quit IRC | 11:05 | |
| stephenfin | lyarwood: When you're about, any chance you could test the combination of these two patches to see if it resolves your '[pci] passthrough_whitelist' concerns? https://review.openstack.org/#/c/554632/ https://review.openstack.org/#/c/552874/ | 11:07 |
| *** sar has quit IRC | 11:09 | |
| *** sidx64 has joined #openstack-nova | 11:13 | |
| *** abhishekk has joined #openstack-nova | 11:19 | |
| *** liverpooler has joined #openstack-nova | 11:21 | |
| *** sar has joined #openstack-nova | 11:22 | |
| *** belmore__ has quit IRC | 11:23 | |
| *** Zames has joined #openstack-nova | 11:24 | |
| *** chichi2 has joined #openstack-nova | 11:24 | |
| *** chichi2 has left #openstack-nova | 11:24 | |
| *** Zames has quit IRC | 11:26 | |
| gibi | gmann_: left a response and +2 in the review :) | 11:27 |
| gmann_ | gibi: got it, thanks for confirmation | 11:33 |
| *** liverpooler has quit IRC | 11:34 | |
| *** tssurya has quit IRC | 11:35 | |
| *** sidx64 has quit IRC | 11:39 | |
| *** zhurong has joined #openstack-nova | 11:39 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Noauth should also use request_id from compute_req_id.py https://review.openstack.org/555266 | 11:39 |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add host field to InstanceActionEvent https://review.openstack.org/555146 | 11:39 |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: t https://review.openstack.org/555267 | 11:39 |
| *** sidx64 has joined #openstack-nova | 11:42 | |
| *** sidx64 has quit IRC | 11:42 | |
| *** elmaciej has joined #openstack-nova | 11:44 | |
| *** sidx64 has joined #openstack-nova | 11:44 | |
| *** udesale has joined #openstack-nova | 11:46 | |
| *** _ix has quit IRC | 11:49 | |
| *** dklyle has quit IRC | 11:49 | |
| *** masuberu has quit IRC | 11:52 | |
| *** mhenkel has quit IRC | 11:52 | |
| *** masuberu has joined #openstack-nova | 11:52 | |
| *** pchavva has joined #openstack-nova | 11:52 | |
| *** sambetts_ is now known as sambetts | 11:54 | |
| *** arvindn05 has joined #openstack-nova | 11:54 | |
| *** zhouyaguo has joined #openstack-nova | 12:03 | |
| *** amoralej is now known as amoralej|lunch | 12:05 | |
| *** hrw has quit IRC | 12:05 | |
| *** ragiman has quit IRC | 12:06 | |
| *** odyssey4me has quit IRC | 12:07 | |
| *** odyssey4me has joined #openstack-nova | 12:08 | |
| *** mgoddard has quit IRC | 12:08 | |
| *** tiendc has quit IRC | 12:09 | |
| *** belmoreira has joined #openstack-nova | 12:09 | |
| *** alexchadin has quit IRC | 12:12 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: WIP https://review.openstack.org/553288 | 12:13 |
| *** edmondsw has joined #openstack-nova | 12:15 | |
| *** mgoddard has joined #openstack-nova | 12:15 | |
| *** alexchadin has joined #openstack-nova | 12:15 | |
| openstackgerrit | Raoul Hidalgo Charman proposed openstack/nova master: Expose shutdown retry interval as config setting https://review.openstack.org/552483 | 12:16 |
| *** hrw has joined #openstack-nova | 12:16 | |
| *** sree has joined #openstack-nova | 12:17 | |
| *** alexchadin has quit IRC | 12:18 | |
| *** alexchadin has joined #openstack-nova | 12:18 | |
| *** mdnadeem has joined #openstack-nova | 12:18 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: WIP https://review.openstack.org/553288 | 12:18 |
| *** alexchadin has quit IRC | 12:18 | |
| *** ragiman has joined #openstack-nova | 12:19 | |
| *** Zames has joined #openstack-nova | 12:19 | |
| *** alexchadin has joined #openstack-nova | 12:19 | |
| *** alexchadin has quit IRC | 12:19 | |
| *** alexchadin has joined #openstack-nova | 12:20 | |
| *** alexchadin has quit IRC | 12:20 | |
| *** alexchadin has joined #openstack-nova | 12:21 | |
| *** alexchadin has quit IRC | 12:21 | |
| openstackgerrit | Eric Young proposed openstack/nova master: Support extending attached ScaleIO volumes https://review.openstack.org/554679 | 12:21 |
| *** alexchadin has joined #openstack-nova | 12:21 | |
| *** alexchadin has quit IRC | 12:22 | |
| *** alexchadin has joined #openstack-nova | 12:22 | |
| *** alexchadin has quit IRC | 12:22 | |
| *** alexchadin has joined #openstack-nova | 12:23 | |
| *** lucasagomes is now known as lucas-hungry | 12:23 | |
| *** Zames has quit IRC | 12:23 | |
| sean-k-mooney | bauzas: o/ | 12:24 |
| *** yassine has quit IRC | 12:24 | |
| *** yassine has joined #openstack-nova | 12:25 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: WIP https://review.openstack.org/553288 | 12:25 |
| efried | morning nova | 12:25 |
| sean-k-mooney | efried: morning :) | 12:25 |
| openstackgerrit | Matthew Edmonds proposed openstack/nova master: make PowerVM capabilities explicit https://review.openstack.org/547169 | 12:27 |
| *** tssurya has joined #openstack-nova | 12:29 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Extract generate_hostid method into utils.py https://review.openstack.org/555282 | 12:30 |
| Spaz-Work | Morning daylight novaers | 12:31 |
| *** vladikr has joined #openstack-nova | 12:31 | |
| edmondsw | efried lost your +1 on https://review.openstack.org/#/c/547169 with a commit message update | 12:31 |
| efried | ... | 12:31 |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: WIP https://review.openstack.org/553288 | 12:31 |
| edmondsw | since we decided to use specless bps, needed to remove the old bp reference | 12:31 |
| efried | edmondsw: Oh, other drivers do it this way? | 12:32 |
| edmondsw | yep | 12:32 |
| efried | cool. | 12:32 |
| efried | +1 | 12:32 |
| edmondsw | tx | 12:32 |
| edmondsw | stephenfin that would be a quick review if you have a minute... https://review.openstack.org/#/c/547169 | 12:32 |
| *** alexchad_ has joined #openstack-nova | 12:33 | |
| *** alexchadin has quit IRC | 12:33 | |
| *** cdent has joined #openstack-nova | 12:34 | |
| *** markvoelker has quit IRC | 12:34 | |
| *** liverpooler has joined #openstack-nova | 12:34 | |
| *** markvoelker has joined #openstack-nova | 12:34 | |
| sean-k-mooney | efried: fyi i left some comments on https://review.openstack.org/#/c/554305 last night/30 seconds ago. do you think allowing traits by resouces_class is resonable? not full granular support but just enough to cover 80% of usecases | 12:39 |
| efried | responding right now. | 12:39 |
| efried | sean-k-mooney: done | 12:39 |
| efried | sean-k-mooney: Oh, I missed your 30-seconds-ago comment - we crossed in the mail. Looking... | 12:41 |
| sean-k-mooney | cool reading. an ya im aware 99% of the time traits are specific to resouces classes to you wont get conflicts | 12:41 |
| efried | okay, you brought up a more viable example. Is it possible for GPUs and CPUs to have the same traits? | 12:41 |
| efried | I wonder if it makes sense for us to name those traits differently. HW_CPU_X vs HW_GPU_X | 12:42 |
| sean-k-mooney | efried: ya i think come gpus support sse instructions | 12:42 |
| efried | Be interested to see how jaypipes feels about that. | 12:42 |
| sean-k-mooney | efried: i see pros and cons to that | 12:42 |
| efried | yeah | 12:42 |
| efried | but again, in that case maybe it makes sense to force them to use granular in the flavor for the time being, just so we can defer this discussion and get the majority of the function we need right away. | 12:43 |
| sean-k-mooney | so ya clip notes version i dont think we need GPU1:trait:X=required,GPU2:trait:Y=required in image but gpu:trait:x=required,cpu:trait:y=reqiured would be nice unless we namspace all the traits | 12:43 |
| sean-k-mooney | efried: well images are use creatable but flavors are not | 12:44 |
| efried | Hum. I wonder if that's a problem in and of itself. | 12:44 |
| efried | Aren't we giving the user the power to hog resources now? | 12:44 |
| sean-k-mooney | HW_CPU_X vs HW_GPU_X would "fix" this but we duplicate the traits in some cases | 12:45 |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: WIP https://review.openstack.org/553288 | 12:45 |
| sean-k-mooney | am no | 12:45 |
| sean-k-mooney | the user can not request a gpu via the image | 12:45 |
| sean-k-mooney | they would have to select a flavor with a gpu as we dont have resaouce request on the image just traits | 12:45 |
| cdent | good morning jaypipes, edleafe, efried: I believe we had some discussions about what counts as a valid uuid recently, I have some related concerns within placement: We save uuids as strings of length 36, but in most (but not all) places we json schema validate incoming uuids to accept both the '-' and not '-' forms. In the dict format of putting allocations we do _not_, we only accept '-'. | 12:46 |
| efried | do we always save them with the hyphens? | 12:46 |
| *** sree has quit IRC | 12:47 | |
| efried | even if they're sent in without? | 12:47 |
| cdent | efried: as far as I know we dont' process them as uuids, so we take what's given | 12:47 |
| cdent | that's why I'm raising the issue | 12:47 |
| efried | So it'd be possible for me to create e.g. two separate resource providers with UUIDs A-B-C-D-E and ABCDE?? | 12:47 |
| sean-k-mooney | cdent: minus is not valide for uuids only hyphens so we should either convert or rais an exception and return a 300 | 12:47 |
| sean-k-mooney | efried: both of those would be invalid | 12:48 |
| sean-k-mooney | the asci represention of a uuid is not arbitrary it has a fixed format | 12:48 |
| cdent | efried: That is my concern, yes, but I haven't had a chance to check it yet | 12:48 |
| efried | I'm shorthanding. A{8}-B{4}-C{4}-D{4}-E{12} | 12:48 |
| efried | cdent: Okay, sounds like a thing to do. Should be an easy enough func test to write. | 12:49 |
| sean-k-mooney | efried: ah wel that format requires the hypens | 12:49 |
| *** tbachman has quit IRC | 12:49 | |
| cdent | efried: yeah, was just checking in first before digging harder | 12:50 |
| *** mriedem has joined #openstack-nova | 12:50 | |
| *** arvindn05 has quit IRC | 12:50 | |
| efried | sean-k-mooney: What cdent is saying is that the placement API is allowing either 12345678-ABCD-ABCD-ABCD-12345678ABCD or 12345678ABCDABCDABCD12345678ABCD as inputs, but may in fact be interpreting those as *different* values. | 12:50 |
| efried | ...which would be bad. Like crossing the streams. | 12:51 |
| sean-k-mooney | efried: ya so is placement using ovo for its data sturctures | 12:51 |
| *** ratailor has quit IRC | 12:52 | |
| sean-k-mooney | ovo has 2 uuid fields one is a strict check and the other just emits a warnign if the format is invalid. if we use ovo internally in the strict form we can prevent incorrect uuids form getting to the db | 12:52 |
| sean-k-mooney | efried: for the 12345678ABCDABCDABCD12345678ABCD case i would be happy if the api returned a 400 bad request in that case | 12:53 |
| efried | Which we can't do without a new microversion. | 12:53 |
| sean-k-mooney | efried: ya but i would be in favor of a microversion for this | 12:54 |
| efried | Although part of what cdent is about to find out is, even though the schema will pass that, maybe something further down will reject it. | 12:54 |
| *** yamamoto has quit IRC | 12:54 | |
| *** arvindn05 has joined #openstack-nova | 12:54 | |
| sean-k-mooney | efried: well the db field is a varchar(36) so it wont so it would have to be somthing in the python code before it hits the db layer | 12:55 |
| efried | sean-k-mooney: I would too (be in favor of a new microversion to lock this down), but think about it from a consumer standpoint. They don't have to use the new microversion in order to start passing in their UUIDs with hyphens. | 12:55 |
| *** _ix has joined #openstack-nova | 12:55 | |
| efried | It'd be kind of weird, like "use this new microversion so you can make sure I'm using hyphens in my UUIDs for me." | 12:55 |
| sean-k-mooney | yes the microversion is just stopping them passing without hypens | 12:56 |
| efried | I sorta doubt consumers will bother with it, considering they would then have to do a 406-and-retry-with-lower-microversion branch. | 12:56 |
| sean-k-mooney | efried: well i gues we need to first check what the behavior is today | 12:57 |
| efried | yuh | 12:57 |
| efried | iiuc, cdent is on that. | 12:57 |
| sean-k-mooney | perhaps we normalise it at some point | 12:57 |
| efried | if we'd just quit bugging him :P | 12:57 |
| sean-k-mooney | :) but if we didnt bug him that would just give him more time to get pulled into internal meetings | 12:58 |
| cdent | this came about because of internal discussions | 12:58 |
| * cdent squares the circle | 12:58 | |
| *** lyan has joined #openstack-nova | 12:58 | |
| * efried shouts over cdent's shoulder: "Oi, cdent's manager, this is important, cancel all his meetings!" | 12:58 | |
| *** lyan is now known as Guest29400 | 12:59 | |
| efried | (cdent probably works from home, huh. I'm shouting at his dog.) | 12:59 |
| *** zhurong has quit IRC | 13:00 | |
| cdent | no dog, and apparently you weren't loud enough to wake the cat | 13:00 |
| *** udesale has quit IRC | 13:00 | |
| *** sq4ind has joined #openstack-nova | 13:04 | |
| sq4ind | hi guys | 13:04 |
| sq4ind | have a problem after upgrade to queens | 13:04 |
| sq4ind | I cannot live migrate instances, I am getting error on nova-conductor: | 13:05 |
| *** idlemind has joined #openstack-nova | 13:05 | |
| sq4ind | Setting instance to ACTIVE state.: NoValidHost: No valid host was found. Unable to move instance 220a6584-02ae-4a22-9940-6f64bbb4a1d8 to host nova0 There is not enough capacity on the host for the instance. | 13:05 |
| sq4ind | but there is planty of resources | 13:05 |
| sq4ind | in the placement-api : Over capacity for MEMORY_MB on resource provider 52c0c39e-30f9-4bd8-84e9-af5c35aac61f. Needed: 2048, Used: 175104, Capacity: 122355.0 | 13:06 |
| *** crushil has joined #openstack-nova | 13:06 | |
| sq4ind | Placement API returning an error response: Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider '52c0c39e-30f9-4bd8-84e9-af5c35aac61f'. The requested amount would exceed the capacity. | 13:06 |
| sq4ind | any idea ? | 13:06 |
| efried | allocation ratio thing? | 13:06 |
| sq4ind | default | 13:06 |
| sq4ind | 1.5 | 13:06 |
| sean-k-mooney | sq4ind: did you set allocation in aggregates or on compute node nova.conf | 13:07 |
| sq4ind | on compute | 13:07 |
| sean-k-mooney | sq4ind: oh ok. we broke the aggregate allocation ratios | 13:07 |
| *** suresh12 has joined #openstack-nova | 13:07 | |
| sean-k-mooney | sq4ind: can you share the full resouce provider info for 52c0c39e-30f9-4bd8-84e9-af5c35aac61f | 13:08 |
| sq4ind | it looks like the resources are not being properly updated | 13:08 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Change compute mgr placement check to region_name https://review.openstack.org/554759 | 13:09 |
| sean-k-mooney | sq4ind: well you resocce useage exceed you cappasity currently 175104>122355.0 by a ratio of 1.4 | 13:09 |
| sq4ind | sean-k-mooney, but how: | 13:10 |
| sq4ind | [root@nova0 ~]# free -m | 13:10 |
| sq4ind | total used free shared buff/cache available | 13:10 |
| sq4ind | Mem: 120694 6338 113954 9 402 107727 | 13:10 |
| sq4ind | Swap: 3815 0 3815 | 13:10 |
| sq4ind | ? | 13:10 |
| sean-k-mooney | if the ratio is not set in the RP and is at the default of 1.0 then it would fail with that message | 13:10 |
| sean-k-mooney | free -m show current inuse | 13:10 |
| sean-k-mooney | memory | 13:10 |
| sean-k-mooney | not the reseved memory | 13:10 |
| sean-k-mooney | if you are using kvm it does not preallocate the vm memory and only allocates as guests use it | 13:11 |
| efried | mriedem: I went ahead and made that change -----^ | 13:11 |
| efried | ...and rechecked the devstack side - although there's no way the devstack change fails because of this tweak. | 13:11 |
| sean-k-mooney | sq4ind: what does the hyperviors api say is used on nova0 | 13:12 |
| *** suresh12 has quit IRC | 13:12 | |
| efried | because now we're both setting and checking the new value. | 13:12 |
| *** bjolo has quit IRC | 13:12 | |
| efried | mriedem: I think we'd be looking for the nova patch itself to fail tempest now, because it's using devstack with os_region_name set. | 13:12 |
| *** mdnadeem has quit IRC | 13:13 | |
| mriedem | aye aye | 13:13 |
| sq4ind | sean-k-mooney, | 13:13 |
| sq4ind | | current_workload | 0 | | 13:13 |
| sq4ind | | disk_available_least | 252971 | | 13:13 |
| sq4ind | | free_disk_gb | 492681 | | 13:13 |
| sq4ind | | free_ram_mb | 110067 | | 13:13 |
| sq4ind | | host_ip | 10.252.16.190 | | 13:13 |
| sq4ind | | host_time | 13:12:45 | | 13:13 |
| sq4ind | | hypervisor_hostname | nova0.linguamatics.com | | 13:13 |
| sq4ind | | hypervisor_type | QEMU | | 13:13 |
| sq4ind | | hypervisor_version | 2009000 | | 13:13 |
| sq4ind | | id | 1 | | 13:13 |
| sean-k-mooney | sq4ind: pastbing might be simpler | 13:13 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | 13:13 |
| sq4ind | | load_average | 0.10, 0.05, 0.06 | | 13:13 |
| sq4ind | | local_gb | 492834 | | 13:13 |
| sq4ind | | local_gb_used | 153 | | 13:14 |
| sq4ind | | memory_mb | 122867 | | 13:14 |
| kashyap | sq4ind: Please use pastebin :-( | 13:14 |
| sq4ind | | memory_mb_used | 12800 | | 13:14 |
| sq4ind | | running_vms | 4 | | 13:14 |
| sq4ind | | service_host | nova0.linguamatics.com | | 13:14 |
| sq4ind | | service_id | 12 | | 13:14 |
| sq4ind | | state | up | | 13:14 |
| sq4ind | | status | enabled | | 13:14 |
| sq4ind | | uptime | 1:59 | | 13:14 |
| sq4ind | | users | 1 | | 13:14 |
| sq4ind | | vcpus | 16 | | 13:14 |
| sq4ind | | vcpus_used | 6 | 13:14 |
| sq4ind | sorry | 13:14 |
| sq4ind | sean-k-mooney, sorry for pasting here | 13:14 |
| sq4ind | https://pastebin.com/aprda4We | 13:14 |
| sean-k-mooney | sq4ind: thats ok | 13:15 |
| mriedem | lyarwood: can you check https://review.openstack.org/#/c/555029/ before we do a queens release? | 13:15 |
| *** yingjun has joined #openstack-nova | 13:15 | |
| mriedem | melwitt: might want to throw the spec review day on the schedule wiki https://wiki.openstack.org/wiki/Nova/Rocky_Release_Schedule#Special_review_days | 13:17 |
| sean-k-mooney | sq4ind: so the capsity on the RP more or less matache the hypervisor api 122355.0 vs 122867 | 13:17 |
| sean-k-mooney | sq4ind: but the used ram is way off 12800 vs 175104 | 13:18 |
| *** udesale has joined #openstack-nova | 13:18 | |
| sean-k-mooney | efried: any idea what could cause ^ other then leaked allocations | 13:19 |
| *** amodi has joined #openstack-nova | 13:20 | |
| sean-k-mooney | the difference between 122355.0 vs 122867 is placement is in MiB and hypervior api is in MB i think | 13:20 |
| efried | sean-k-mooney: TBH I never understood the allocation ration breakage issue. | 13:20 |
| efried | s/ration/ratio/ | 13:20 |
| sean-k-mooney | efried: if you set them in the host aggregate we did not use that value to set the RP allocation ratio and instead only used the nova conf version | 13:21 |
| *** dtantsur is now known as dtantsur|brb | 13:21 | |
| sean-k-mooney | at least i think that was the issue | 13:21 |
| *** yamamoto has joined #openstack-nova | 13:21 | |
| efried | In this case it's nowhere close, though, as sq4ind points out. | 13:22 |
| sean-k-mooney | well the capasity is right it the used that is wrong | 13:22 |
| efried | Although in the original statement it's a lot closer: "in the placement-api : Over capacity for MEMORY_MB on resource provider 52c0c39e-30f9-4bd8-84e9-af5c35aac61f. Needed: 2048, Used: 175104, Capacity: 122355.0" | 13:22 |
| sean-k-mooney | so this is noting to do with alocation ratios | 13:22 |
| efried | sq4ind: Are you able to query the placement API directly? | 13:23 |
| sq4ind | efried, let me try | 13:23 |
| efried | I would like to see what placement thinks the allocation ratio is for that MEMORY_MB inventory record on provider 52c0c39e-30f9-4bd8-84e9-af5c35aac61f | 13:23 |
| edleafe | cdent: reading scrollback | 13:23 |
| sean-k-mooney | efried: yes the orignial error is correct allocating an addtion 2048 on top of 175104 would violate the over commit ratio | 13:23 |
| efried | sean-k-mooney: Unless the alloc ratio is 1.5 | 13:24 |
| efried | or higher | 13:24 |
| edleafe | cdent: we should accept either, but normalize how we store them. Is that the change you are proposing? | 13:24 |
| *** lucas-hungry is now known as lucasagomes | 13:24 | |
| efried | edleafe: It is unclear whether we are normalizing them or not. | 13:24 |
| efried | cdent is finding out. | 13:24 |
| edleafe | efried: gotcha. We definitely *should* be normalizing | 13:25 |
| efried | edleafe: Or only accepting one format. | 13:25 |
| edleafe | no, I don't think we need to do that | 13:26 |
| cdent | here's the bug: https://bugs.launchpad.net/nova/+bug/1758057 | 13:28 |
| openstack | Launchpad bug 1758057 in OpenStack Compute (nova) "When creating uuid-based entities we can duplicate UUIDs" [Undecided,Triaged] | 13:28 |
| cdent | we do not normalize, they are treated as different resource providers | 13:28 |
| sq4ind | efried, sorry but I am not able to query placement api directly | 13:29 |
| *** awaugama has joined #openstack-nova | 13:30 | |
| cdent | efried, edleafe: gonna have my lunch while that settlles in | 13:31 |
| edleafe | cdent: chew thoroughly! | 13:32 |
| *** psachin has quit IRC | 13:32 | |
| *** eharney has joined #openstack-nova | 13:32 | |
| *** germs has joined #openstack-nova | 13:33 | |
| *** germs has quit IRC | 13:33 | |
| *** germs has joined #openstack-nova | 13:33 | |
| *** germs has quit IRC | 13:33 | |
| *** germs has joined #openstack-nova | 13:34 | |
| *** germs has quit IRC | 13:34 | |
| *** germs has joined #openstack-nova | 13:34 | |
| *** burt has joined #openstack-nova | 13:36 | |
| efried | cdent: Okay, so I think we have to fix the bug by normalizing UUIDs for all the APIs, and I don't think we need a microversion for that. Afterwards we can consider whether we want a microversion to further restrict the acceptable input formats. | 13:36 |
| *** amoralej|lunch is now known as amoralej | 13:37 | |
| sq4ind | is there any way to repopulate cells in the placement ? (or is it safe to remove cell and recreate it ) | 13:38 |
| bauzas | folks, gentle notice I'm under the water with serious vGPU testing | 13:38 |
| *** kholkina has quit IRC | 13:40 | |
| jaypipes | cdent: hey, got your question on UUIDs answered? | 13:41 |
| mriedem | kashyap: how's the libvirt min version bump thing going? | 13:42 |
| kashyap | mriedem: Hi | 13:43 |
| kashyap | mriedem: First fixing the last unit test of this, as we speak: https://review.openstack.org/#/c/534384/4/ | 13:43 |
| kashyap | (Then I'll get to it.) | 13:43 |
| kashyap | Just duking around the last test for the conditional in driver.py. The existing patch & tests all 'pass' | 13:43 |
| kashyap | mriedem: You got a deadline for me? Or was it yesterday? :-) | 13:44 |
| mriedem | would be nice to have that done by milestone 1 in case anything crops up, then we have time later in the release to deal with it | 13:44 |
| kashyap | Ah, true. /me goes to look when is Milestone-1 | 13:44 |
| cdent | jaypipes is probably a bug so made one: https://bugs.launchpad.net/nova/+bug/1758057 | 13:45 |
| openstack | Launchpad bug 1758057 in OpenStack Compute (nova) "When creating uuid-based entities we can duplicate UUIDs" [Undecided,Triaged] | 13:45 |
| mriedem | kashyap: april 19 | 13:45 |
| kashyap | mriedem: Ah, I'll be starting tomm or at most Monday. | 13:45 |
| * kashyap even got a task reminder on phone for it this morning :P | 13:45 | |
| jaypipes | cdent: cool. looks like a relatively simple fix. | 13:45 |
| cdent | efried: I pretty much agree with you, but what do we do any extant providers that are the same uuid with different reps and have since diverged? | 13:46 |
| cdent | jaypipes: if you look above in scrollback there are some differing opinions on the right fix, but not vastly so | 13:46 |
| kashyap | mriedem: In your "copious free time", wonder if you can punch any holes in the above change: https://review.openstack.org/#/c/534384 | 13:46 |
| kashyap | If you can't get to it; it's fine. Got enough attention so far, can wait | 13:46 |
| mriedem | kashyap: i already did wrt backports | 13:46 |
| mriedem | so don't really want to talk about that one honestly | 13:47 |
| kashyap | Okay, no prob; dansmith suggested on the post a bit more palatable. | 13:47 |
| efried | cdent: That's a neat question. Let the user clean 'em up? | 13:47 |
| kashyap | (But, IMHO, it just seems like (even Tony) is trying to overly stick to the "letter of the law") | 13:47 |
| efried | cdent: ...which entails allowing DELETE APIs to accept and use non-normalized UUIDs... | 13:47 |
| lyarwood | mriedem: sorry missed your ping earlier, looking now | 13:48 |
| mriedem | dansmith: question in tssurya's scheduler sighup patch about locking https://review.openstack.org/#/c/550527/ | 13:48 |
| mriedem | dansmith: i have a feeling we should be locking on that new attribute when we're resetting it | 13:48 |
| *** esberglu has joined #openstack-nova | 13:48 | |
| efried | cdent: Gotta say it's pretty doubtful that something like this would have happened IRL, because the client - whatever it is - will generally be using one code path to create providers. So they'll have created 'em all with the same UUID format. | 13:49 |
| dansmith | mriedem: okay I failed to circle back to those so I'll try to do that this morning dodging meetings | 13:50 |
| *** gouthamr has joined #openstack-nova | 13:50 | |
| cdent | efried: that's probably true, I'm just asking the questions for completeness. Also, just sake of the record and all that (in the sense that it is not really germane for today's reality): the whole point of having an http api is so that there are and will be multiple clients and different code | 13:51 |
| efried | yup, I get that. | 13:51 |
| efried | cdent: I say we just fix the glitch, and not bother making a cleanup tool until someone claims they need it. | 13:52 |
| *** yamamoto has quit IRC | 13:53 | |
| efried | cdent, melwitt, mriedem: This would be a good bug for new contributors. | 13:53 |
| edleafe | efried: +1 on not worrying about it until we need to worry about it | 13:54 |
| *** Tom-Tom has joined #openstack-nova | 13:54 | |
| *** felipemonteiro__ has joined #openstack-nova | 13:55 | |
| mriedem | tag it with low-hanging-fruit then | 13:55 |
| *** mlavalle has joined #openstack-nova | 13:55 | |
| *** jianghuaw has joined #openstack-nova | 13:56 | |
| efried | done | 13:57 |
| *** tbachman_ has joined #openstack-nova | 13:57 | |
| sq4ind | sean-k-mooney, efried any idea ? | 13:58 |
| efried | sq4ind: None here, sorry. | 13:58 |
| sq4ind | efried, thanks | 13:58 |
| efried | sq4ind: But you may want to recap for jaypipes - this sounds up his alley to me. | 13:58 |
| gibi | nova meeting starts in about a minute | 13:59 |
| *** bkopilov has joined #openstack-nova | 13:59 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | 14:03 |
| *** tidwellr has joined #openstack-nova | 14:03 | |
| *** felipemonteiro_ has joined #openstack-nova | 14:03 | |
| efried | mriedem: Poetic patch title ^ | 14:04 |
| *** tuanla____ has quit IRC | 14:04 | |
| mriedem | is it? | 14:04 |
| *** felipemonteiro__ has quit IRC | 14:07 | |
| *** yamamoto has joined #openstack-nova | 14:08 | |
| *** moshele has quit IRC | 14:08 | |
| *** hamzy has quit IRC | 14:09 | |
| openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: Network interface attach/detach https://review.openstack.org/546813 | 14:10 |
| *** links has quit IRC | 14:10 | |
| *** itlinux has quit IRC | 14:11 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Add request_id to isntance action notifications https://review.openstack.org/553288 | 14:13 |
| *** yamamoto has quit IRC | 14:13 | |
| *** hongbin has joined #openstack-nova | 14:13 | |
| *** zhouyaguo has quit IRC | 14:13 | |
| *** dtantsur|brb is now known as dtantsur | 14:15 | |
| *** dklyle has joined #openstack-nova | 14:15 | |
| *** germs_ has joined #openstack-nova | 14:18 | |
| *** germs has quit IRC | 14:20 | |
| *** alexchad_ has quit IRC | 14:22 | |
| *** Spaz-Work has quit IRC | 14:22 | |
| *** yamamoto has joined #openstack-nova | 14:24 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Add request_id to instance action notifications https://review.openstack.org/553288 | 14:25 |
| *** Spaz-Work has joined #openstack-nova | 14:25 | |
| bauzas | mriedem: come to France and you'll enjoy the day | 14:25 |
| bauzas | well, if you can... | 14:25 |
| *** r-daneel has joined #openstack-nova | 14:27 | |
| *** yamamoto has quit IRC | 14:28 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Add request_id to instance action notifications https://review.openstack.org/553288 | 14:28 |
| *** ccamacho1 has joined #openstack-nova | 14:31 | |
| *** ccamacho1 has quit IRC | 14:31 | |
| *** alexchadin has joined #openstack-nova | 14:31 | |
| *** ccamacho1 has joined #openstack-nova | 14:32 | |
| mriedem | need another core for the bottom 2 patches in this series to move the nova-cells-v1 job in-tree and then change it to use neutron, which is the first part of removing nova-network https://review.openstack.org/#/c/549780/ | 14:33 |
| *** ccamacho has quit IRC | 14:34 | |
| *** amodi has quit IRC | 14:34 | |
| stephenfin | mriedem: I'll grab em | 14:36 |
| *** sree has joined #openstack-nova | 14:37 | |
| *** yamamoto has joined #openstack-nova | 14:39 | |
| *** hamzy has joined #openstack-nova | 14:40 | |
| *** hrw has quit IRC | 14:41 | |
| *** yamamoto has quit IRC | 14:43 | |
| sq4ind | I think I've found why I was having issues... basically when listing cells in placement I had one out of three rabbitmq servers in the transport url, and only hosts connected to this rabbitmq server were updating their resource usage. Now I've fixed the url, I am also moving VMs from hosts and removing hosts from placement cells, and then recreating them. After that everything works as it should be | 14:44 |
| sq4ind | hope that makes sense | 14:44 |
| *** zhaochao has quit IRC | 14:45 | |
| efried | sq4ind: Glad you figured it out - sorry we couldn't be more help. | 14:46 |
| *** yamamoto has joined #openstack-nova | 14:46 | |
| *** yamamoto has quit IRC | 14:46 | |
| *** jackie-truong has joined #openstack-nova | 14:49 | |
| *** felipemonteiro__ has joined #openstack-nova | 14:53 | |
| sq4ind | efried, no problem :D I like to dig more and more into openstack internals :D And you were very helpful: you've gave some clues where to look at it :D, thanks ! :) | 14:55 |
| *** felipemonteiro_ has quit IRC | 14:57 | |
| *** Swami has joined #openstack-nova | 14:57 | |
| *** gyankum has quit IRC | 14:58 | |
| *** alexchadin has quit IRC | 14:58 | |
| *** zhaochao has joined #openstack-nova | 14:59 | |
| *** abhishekk has quit IRC | 15:00 | |
| *** sree has quit IRC | 15:00 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Support extending attached ScaleIO volumes https://review.openstack.org/554679 | 15:01 |
| mriedem | stephenfin: thanks for hitting those cells ci patches | 15:01 |
| *** sree has joined #openstack-nova | 15:01 | |
| stephenfin | mriedem: np | 15:03 |
| *** tidwellr_ has joined #openstack-nova | 15:04 | |
| *** tidwellr has quit IRC | 15:04 | |
| *** dave-mccowan has joined #openstack-nova | 15:05 | |
| *** sree has quit IRC | 15:05 | |
| mriedem | edmondsw: we don't need both a powervm-resize and powervm-cold-migrate blueprint | 15:06 |
| mriedem | if you implement resize, you have to have cold migrate | 15:06 |
| mriedem | so i'm going to mark the cold migrate one as superseded | 15:07 |
| edmondsw | mriedem yep | 15:07 |
| mriedem | esberglu: ^ | 15:07 |
| edmondsw | rebuild and evac would go in there as well | 15:07 |
| *** sree has joined #openstack-nova | 15:08 | |
| edmondsw | esberglu update the description on that one? | 15:08 |
| edmondsw | (resize) | 15:08 |
| mriedem | eh? | 15:08 |
| mriedem | those are not the same as resize | 15:08 |
| mriedem | nor do they require a blueprint | 15:08 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Use microversion parse 0.2.1 https://review.openstack.org/550265 | 15:08 |
| mriedem | if your driver can spawn and destroy, you support rebuild | 15:08 |
| *** elmaciej has quit IRC | 15:08 | |
| *** zhaochao has quit IRC | 15:09 | |
| *** dave-mccowan has quit IRC | 15:09 | |
| mriedem | https://blueprints.launchpad.net/nova/rocky should be accurate now right? | 15:09 |
| mriedem | 5 powervm-* blueprints | 15:09 |
| edmondsw | mriedem sorry otp | 15:10 |
| *** sidx64 has quit IRC | 15:10 | |
| *** sree has quit IRC | 15:12 | |
| openstackgerrit | Raoul Hidalgo Charman proposed openstack/nova master: Expose shutdown retry interval as config setting https://review.openstack.org/552483 | 15:14 |
| mriedem | does anyone from virtuozzo still work on nova? | 15:14 |
| mriedem | because their CI is borked | 15:14 |
| edmondsw | mriedem yeah, looks right | 15:14 |
| *** moshele has joined #openstack-nova | 15:14 | |
| esberglu | edmondsw: mriedem: Will update resize | 15:15 |
| edmondsw | mriedem I meant evac would come only when we support migration | 15:15 |
| edmondsw | esberglu I think mriedem wants it left as-is | 15:15 |
| mriedem | edmondsw: those aren't the same | 15:15 |
| mriedem | or related | 15:15 |
| mriedem | evac is not the same as cold migrate, | 15:15 |
| mriedem | it's a rebuild of the instance on another host | 15:15 |
| mriedem | using spawn and destroy | 15:15 |
| edmondsw | oh, right... | 15:16 |
| *** dave-mccowan has joined #openstack-nova | 15:16 | |
| *** sidx64 has joined #openstack-nova | 15:16 | |
| edmondsw | sorry, too many different uses of the same term | 15:16 |
| dansmith | mriedem: should I link him or will you? | 15:16 |
| dansmith | edmondsw: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | 15:17 |
| edmondsw | dansmith +1 | 15:17 |
| *** sidx64 has quit IRC | 15:17 | |
| edmondsw | I remember reading that when it was new | 15:17 |
| edmondsw | esberglu ^ | 15:18 |
| kashyap | dansmith: At this point it warrants to be put in the channel topic ;-) | 15:18 |
| dansmith | heh | 15:18 |
| kashyap | "Evacuate", please see: $short-url | 15:18 |
| kashyap | dansmith: Then you can start chiding people: "Don't you read channel topic?!" | 15:19 |
| dansmith | kashyap: make a bot, that will waste more time | 15:19 |
| kashyap | True; it reminds me of the bot on #fedora-devel, where if someone just says: | 15:20 |
| kashyap | Person: Ping | 15:20 |
| kashyap | Bot: https://fedoraproject.org/wiki/No_naked_pings | 15:20 |
| *** germs_ has quit IRC | 15:20 | |
| *** sidx64 has joined #openstack-nova | 15:20 | |
| *** amodi has joined #openstack-nova | 15:20 | |
| *** germs has joined #openstack-nova | 15:20 | |
| *** germs has quit IRC | 15:20 | |
| *** germs has joined #openstack-nova | 15:20 | |
| *** felipemonteiro__ has quit IRC | 15:21 | |
| *** sidx64 has quit IRC | 15:21 | |
| *** felipemonteiro__ has joined #openstack-nova | 15:21 | |
| efried | wow | 15:22 |
| efried | TIL | 15:22 |
| dansmith | efried: not everyone agrees with those assertions (like myself) | 15:22 |
| efried | ...that some people are just too damn touchy | 15:22 |
| dansmith | hehe | 15:22 |
| dansmith | yeah that | 15:22 |
| eric-young | efried: thanks for the quick edit on https://review.openstack.org/554679 | 15:22 |
| mriedem | can we now say that nova people aren't as bad as fedora people? | 15:22 |
| efried | eric-young: Yahyoubetcha | 15:23 |
| eric-young | efried: I am seeing another issue right now, and am testing to see if it's just me | 15:23 |
| mriedem | when it comes to the every 6 month nova hate-a-thon? | 15:23 |
| efried | eric-young: Mm, I didn't test or anything, just fixed obvious problem. | 15:23 |
| mriedem | eric-young: that requires a minimum os-brick right? | 15:23 |
| eric-young | efried, yeah. silly mistake but it did expose something else. | 15:23 |
| kashyap | efried: Yeah, I don't mind such bare pings. But prefer "dressed up" pings | 15:24 |
| mriedem | you've been riedmeann'ed | 15:24 |
| mriedem | gd i can't even spell my own name | 15:24 |
| mriedem | ffs | 15:24 |
| kashyap | mriedem: You slipped in "mean" there; I think they call it: "Freudian" | 15:25 |
| * kashyap stops trolling | 15:25 | |
| *** lajoskatona has quit IRC | 15:28 | |
| sean-k-mooney | edleafe: just on the uuids its not a uuid anymore if you acept a string with our hypentens. if we decalre the filed as an int at the api then we could accpet 3c3770f6-e1a6-4c3c-ac1d-ba2aa2f231c4 as a hex int 3c3770f6e1a64c3cac1dba2aa2f231c4 | 15:28 |
| mriedem | fyi all things are going to be blocked right now https://review.openstack.org/555314 | 15:28 |
| mriedem | until ^ merges | 15:28 |
| *** Swami has quit IRC | 15:29 | |
| dansmith | efried: so, on this aggregate thing | 15:29 |
| dansmith | efried: what if we had member_of=in:foo,bar&member_of=in:baz -- which would turn into (foo OR bar) AND baz | 15:30 |
| dansmith | efried: that would let me have infinite filters that AND together several "any one of these would be fine for this requirement" | 15:30 |
| jaypipes | cfriesen: I'm dead set against "implicit" or "hidden" resources. | 15:30 |
| dansmith | efried: logically "tenant_aggregates AND az_aggregates AND gpu_aggregates AND shared_storage_aggregates" for some fictitious request with all those restrictions | 15:31 |
| eric-young | mriedem, yes, that patch requires an unreleased version of os-brick. Is there a way to instigate a release or just wait? | 15:31 |
| efried | dansmith: Sure, that seems simple-but-powerful. But we still need to address the question of whether to allow a queryparam key to be repeated. We haven't so far. | 15:32 |
| bauzas | jianghuaw_: mriedem: FWIW, I filed a bug with a new 'vgpu' tag | 15:33 |
| efried | cdent, edleafe: In your capacity as API SIGgers, what do you think? | 15:33 |
| dansmith | efried: yeah, but it's a common thing for query strings, which is why we have it as a list right? | 15:33 |
| bauzas | jianghuaw_: mriedem: https://bugs.launchpad.net/nova/+bug/1758086 | 15:33 |
| openstack | Launchpad bug 1758086 in OpenStack Compute (nova) "nvidia driver limits to one single GPU per guest" [Low,Triaged] - Assigned to Sylvain Bauza (sylvain-bauza) | 15:33 |
| bauzas | so we could track all VGPU related bugs | 15:33 |
| efried | dansmith: I agree it's a common thing for query strings in general. But we've explicitly avoided doing it in placement so far. | 15:33 |
| *** idlemind_ has joined #openstack-nova | 15:33 | |
| cdent | query parameter repetition is expected and normal, which is why under the covers the library return a list of values | 15:33 |
| efried | Meaning you can break the API with that. | 15:33 |
| *** idlemind has quit IRC | 15:33 | |
| dansmith | cdent: yeah, that | 15:33 |
| dansmith | this: https://pastebin.com/pbgv9vux | 15:34 |
| efried | Yeah yeah, I know it's supported by HTTP and the tooling. | 15:34 |
| cdent | I'm more worried by what kind of impact this would have on already very challenging for mortals to understand query code | 15:34 |
| cdent | s/query/database query/ | 15:35 |
| efried | dansmith: correct me if I'm wrong, but we can't avoid crossing that bridge *somehow*. | 15:35 |
| dansmith | efried: well, we can by just not doing things :) | 15:35 |
| dansmith | but yeah, if we want it to be more than trivially useful... | 15:36 |
| efried | I mean, we don't _have_ to implement it in a monster JOIN; we can do individual queries and do the set math in python. | 15:36 |
| efried | So cdent/edleafe y'all don't have a problem introducing a repeatable queryparam key, where we've avoided them in the past? | 15:36 |
| *** itlinux has joined #openstack-nova | 15:37 | |
| efried | e.g. resources=VCPU:3,MEMORY_MB:1024 instead of resources=VCPU:3&resources=MEMORY_MB:1024 -- if you do the latter, it does *not* work. | 15:37 |
| edleafe | sean-k-mooney: the hyphens in a UUID are for humans. And Python doesn't require them: http://paste.openstack.org/show/708977/ | 15:38 |
| cdent | well that's kind of the issue: if we're going to allow it in place A, it would be better to allow it all places | 15:38 |
| edleafe | efried: reading back... | 15:38 |
| efried | swhat I'm sayin | 15:38 |
| dansmith | efried: and is that a microversion change or a bug? | 15:38 |
| dansmith | the CGI guy in me says it is a bug | 15:38 |
| efried | dansmith: Oh, definitely would be a microversion change, and no it's not a bug. | 15:38 |
| dansmith | because I always forget and then have to fix my things :) | 15:38 |
| dansmith | okay | 15:39 |
| efried | dansmith: It's explicitly the way we designed the API. | 15:39 |
| *** itlinux has quit IRC | 15:39 | |
| mriedem | eric-young: you could propose an os-brick release to the openstack/releases repo, | 15:39 |
| dansmith | so I guess you need to decide what it means to split them like that | 15:39 |
| efried | I brought it up like a year ago and someone (possibly cdent) assured me that it was the way things were intended. | 15:39 |
| mriedem | eric-young: or i can, it's pretty easy, | 15:39 |
| mriedem | then just need PTL sign off | 15:39 |
| *** itlinux has joined #openstack-nova | 15:39 | |
| dansmith | whether you just merge them as if they were one argument, or make some other assumption about why the caller is doing that | 15:39 |
| cdent | efried: i do not recall senator | 15:39 |
| dansmith | as is the case in member_of | 15:39 |
| efried | just so. | 15:39 |
| dansmith | s/is/would be/ | 15:40 |
| cdent | efried: currently how do repeated params get represented in req.GET | 15:40 |
| efried | Or we can go with using punctuation join | 15:40 |
| cdent | I think WebOb may be just "taking care of it" | 15:40 |
| edleafe | efried: ok, repeating a query param is not a bad thing. It's pretty much the only way to AND things | 15:40 |
| efried | cdent: nope, I remember checking that when I ran across this originally. | 15:40 |
| cdent | in which case the doubling may be challenging | 15:40 |
| eric-young | I'll take a look. no harm in me knowing how to do it :) | 15:40 |
| bauzas | jaypipes: are you still available for an hangout ? | 15:41 |
| *** sar has quit IRC | 15:41 | |
| efried | edleafe: Yeah, the issue is whether we deviate from the rest of the *placement* API specifically, which uses punctuation for lists, and does *not* support repeating keys. | 15:41 |
| cdent | efried: you certain? https://docs.pylonsproject.org/projects/webob/en/stable/api/multidict.html?highlight=multidict | 15:41 |
| efried | It's not about webob. It's about how we process in the handlers. | 15:42 |
| efried | hold on, I'll find code. | 15:42 |
| *** pcaruana has joined #openstack-nova | 15:42 | |
| cdent | you should get the double list back on req.GET, but using items() (as done in RequestGroup parsing) may be throwing things off, dunno | 15:43 |
| *** pcaruana has quit IRC | 15:44 | |
| dansmith | cdent: efried edleafe: sorry I didn't think about this enough during spec review, but I hadn't gotten to the second use case in my head | 15:44 |
| cdent | such is the way of the world, ain't no thing | 15:45 |
| efried | cdent: Well, I can't immediately suss what the handler is doing, cause it clearly thinks it's getting a single string value back from req.GET. | 15:45 |
| *** jackie-truong has quit IRC | 15:45 | |
| efried | it's possible that GET is special and returns a single value if there's only one, but a list if there's more than one. Which seems goofy, but sounds vaguely familiar. And you have to use something else, like GETALL, if you want it to be a list every time. | 15:46 |
| *** yamamoto has joined #openstack-nova | 15:47 | |
| efried | cdent: Oh, no, it looks like GET returns the first one, period. | 15:47 |
| efried | ...where of course "first" could be anything, because dict hashing. | 15:48 |
| *** pcaruana has joined #openstack-nova | 15:48 | |
| cdent | so we know it's a fixable problem, but there's a fair bit of semantis wrangling associated with it | 15:49 |
| cdent | dan wants a specific meaning out of two different member_of keys, which is different than the presumed concatenation of two resources keys | 15:49 |
| cdent | it's that semantic difference of duplication that is a probelm | 15:50 |
| cdent | (if it actually exists, I'm struggling to get my brain right on this because context switching) | 15:50 |
| efried | cdent: "fixable problem" what are we talking about? IMO the fact that we don't accept multiple instances of qparam keys at the moment isn't a problem - it's just the way we designed it. | 15:51 |
| dansmith | cdent: yeah, agree that member_of and resources should not behave in opposite ways | 15:51 |
| efried | If we're talking about the actual issue dansmith is trying to solve, then yeah, fixable. | 15:51 |
| efried | dansmith: So my suggestion was to use punctuation rather than duplicating qparam keys. | 15:52 |
| efried | member_of=in:A,B;in:C | 15:52 |
| dansmith | yeah, I like that less, fwiw | 15:52 |
| dansmith | but it's fine if so | 15:52 |
| efried | Oh, me too, but it doesn't make member_of and resources behave differently. | 15:52 |
| dansmith | yeah | 15:52 |
| efried | We can pick some other punctuation maybe. | 15:53 |
| efried | But & and + both have special meaning already :( | 15:53 |
| *** yamamoto has quit IRC | 15:53 | |
| efried | (since 'AND' is what we're trying to express) | 15:53 |
| dansmith | the symbol used doesn't really matter | 15:53 |
| cdent | efried: "fixable problem" was the immediate thing of "being able to accept duplicates on keys in general". I agree that addressign dan's problem in the least disruptive way is probably syntax in the param | 15:53 |
| dansmith | (to me) | 15:53 |
| cdent | ; can't work | 15:53 |
| cdent | it is equivalent to & (and actually more correct) | 15:53 |
| jaypipes | bauzas: yes sir | 15:54 |
| dansmith | cdent: in a url? really? | 15:54 |
| cdent | but yeah, as long as we are consistent, doesn't matter | 15:54 |
| cdent | dansmith: yeah cgi processing was modernized in the late 90s to use ; for split of params, but it never really took, but library are supposed to support it | 15:54 |
| dansmith | huh, interesting | 15:54 |
| *** chyka has joined #openstack-nova | 15:54 | |
| cdent | it looks nicer too :) | 15:55 |
| * jroll learned something new today | 15:55 | |
| efried | okay, so ':'? | 15:55 |
| efried | oh, no. | 15:55 |
| efried | cause we're using that for in: | 15:55 |
| efried | sigh | 15:56 |
| dansmith | let's go super wacky and use a grave accent | 15:56 |
| dansmith | didn't see that coming didja! | 15:56 |
| cdent | unicode snowman | 15:56 |
| efried | dà nsmith | 15:57 |
| efried | from now on | 15:57 |
| dansmith | heh | 15:57 |
| cdent | since the values of member_of can't have a space, why not a + (which is space)? | 15:57 |
| dansmith | [08:57:29] Message(432): dà nsmith Erroneous Nickname | 15:57 |
| dansmith | bummer | 15:57 |
| *** tblakes has joined #openstack-nova | 15:57 | |
| bauzas | jaypipes: argh, I have a meeting starting in 2 mins | 15:57 |
| efried | cdent: + comes through as a space? That could work. | 15:58 |
| jaypipes | bauzas: I'm free the rest of the day. just ping me when you're free. | 15:58 |
| bauzas | jaypipes: I just want to make sure we can discuss on the direction for all the NUMA things before the next specs day | 15:58 |
| efried | it's a bit hacky-overloady | 15:58 |
| bauzas | jaypipes: okay, cool | 15:58 |
| dansmith | do not like | 15:58 |
| edleafe | cdent: what is the reason why you want to avoid using &? | 15:58 |
| cdent | so we dont' have to encode it | 15:58 |
| openstackgerrit | sahid proposed openstack/nova-specs master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/539605 | 15:58 |
| cdent | and it wasn't me that was someone else | 15:58 |
| edleafe | cdent: no, I mean multiple query params | 15:58 |
| efried | edleafe: Because we don't support repeating queryparams elsewhere in the API where it would also make sense. E.g. resources= | 15:59 |
| efried | and the inconsistency would make sadness | 15:59 |
| * edleafe wonders what "makes sense" means | 15:59 | |
| efried | edleafe: e.g. resources=VCPU:3&resources=MEMORY_MB:1024 | 16:00 |
| efried | ^ does not work. | 16:00 |
| edleafe | efried: there isn't a good reason why it shouldn't | 16:00 |
| efried | edleafe: other than that the code isn't set up to make it work. | 16:01 |
| efried | If we wanted to support that, we would have to make code changes. | 16:01 |
| jroll | if only we knew someone that could change code :) | 16:01 |
| edleafe | jroll: impossible!! | 16:01 |
| efried | jroll: This would be a pretty pervasive change, for IMO negative benefit. | 16:01 |
| sean-k-mooney | edleafe: that is correct python does not require them to have hyphens but https://tools.ietf.org/html/rfc4122 spcifies the reqiurements for the string representation on page 4 which reuires the use of - when serialising the uuid as a string to be standars complient | 16:01 |
| dansmith | jroll: are you helping? :) | 16:01 |
| efried | Now we've got two different ways to express the same thing - double the test matrix, double the bugs, etc. | 16:02 |
| cfriesen | jaypipes: the emulator threads and I/O threads are separate from the vcpu threads, so it seems odd to make the user ask for a "VCPU" to run them on. | 16:02 |
| jroll | dansmith: I'd be happy to make that work, it seems silly that it doesn't :) | 16:02 |
| dansmith | jroll: no, I mean helping by taking shots from the stands.. and I'm just joking of course :) | 16:02 |
| jroll | I can do that too :P | 16:02 |
| sahid | mriedem: btw i updated the vf trusted spec to address your suggestion about metadata | 16:02 |
| dansmith | heh | 16:02 |
| mriedem | sahid: ok | 16:03 |
| jaypipes | cfriesen: it's all just a processor, IMO. | 16:03 |
| jaypipes | cfriesen: either dedicated or shared CPU resource. | 16:03 |
| jroll | I'm just casting my vote for having the key twice in the url being the sane thing to do imho ¯\_(ツ)_/¯ | 16:03 |
| *** udesale has quit IRC | 16:03 | |
| jaypipes | cfriesen: if the emulator thread is dedicated to a physical host CPU, I really don't understand why that isn't considered a "resource consumption". | 16:03 |
| dansmith | jroll: for member_of you mean? I agree, that's what I like the best for my use case | 16:04 |
| jroll | dansmith: yep | 16:04 |
| *** hemna_ has joined #openstack-nova | 16:04 | |
| sean-k-mooney | cfriesen: well normally the emulator tread floats over teh vcpus of the guest so they dont have to ask for extra cpus if they want that behavior | 16:04 |
| *** harlowja has joined #openstack-nova | 16:04 | |
| *** felipemonteiro_ has joined #openstack-nova | 16:05 | |
| sahid | dansmith: i updated tx/rx queue spec I hope you can have a look so I could address any issues | 16:05 |
| efried | To be completely clear on my position: | 16:05 |
| efried | I love repeatable queryparam keys in general. | 16:05 |
| efried | BUT | 16:05 |
| efried | I'm opposed to making all the existing APIs accept repeated keys in addition to accepting their current syntax. | 16:05 |
| efried | and | 16:05 |
| dansmith | sahid: ack, after the call | 16:05 |
| efried | I'm pretty neutral on making the (new) member_of key repeatable, even though it deviates from the norm. | 16:05 |
| mriedem | if you're talking about AND semantics in the API, glance already has some for filtering https://developer.openstack.org/api-ref/image/v2/index.html#show-images | 16:05 |
| sahid | dansmith: cool thanks | 16:05 |
| dansmith | johnthetubaguy: ^ | 16:06 |
| edleafe | sean-k-mooney: of course we should be strict in how we create and store them. But there is no harm in accepting a non-hyphenated UUID string - it's unambiguous | 16:06 |
| cfriesen | jaypipes: I fully agree that it's resource consumption if the emulator thread is running "isolated". I'm just not sure it makes sense to use the "VCPU" resource given the name which strongly implies "virtual CPU". | 16:06 |
| jroll | efried: that's fair | 16:06 |
| openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Update spec to reflect reality https://review.openstack.org/555000 | 16:06 |
| jaypipes | cfriesen: so what sean-k-mooney just said " normally the emulator tread floats over teh vcpus of the guest" is wrong then? | 16:06 |
| sean-k-mooney | edleafe: i disagree it does cause harm as we need to have normalistion code | 16:06 |
| efried | mriedem: Unfortunately I don't see anything in there that helps us in this case. | 16:07 |
| cfriesen | jaypipes: normally the emulator threads are allowed to run on all the host cpus that the vCPU threads run on | 16:07 |
| sean-k-mooney | jaypipes: also the emulator threads is libvirt specific so we have to be carful not to leak too much of the implentation details via the api | 16:07 |
| johnthetubaguy | sahid: thanks for the quick turn around on that, taking a look at it again now | 16:08 |
| dansmith | cdent: efried: so, resources=CPU:1 AND resources=MEM:1024 would be the same semantic meaning as resources=CPU:1,MEM:1024 actually right? | 16:08 |
| mriedem | coreycb: https://bugs.launchpad.net/nova/+bug/1758060 - i don't see that test in the upstream repo | 16:08 |
| openstack | Launchpad bug 1758060 in nova (Ubuntu) "16.1.0 test failure UEFI is not supported" [Low,Triaged] - Assigned to Corey Bryant (corey.bryant) | 16:08 |
| mriedem | coreycb: special sauce? | 16:08 |
| dansmith | I think someone said that wouldn't make sense, butnow I'm not sure I get that | 16:09 |
| *** felipemonteiro__ has quit IRC | 16:09 | |
| efried | dansmith: It makes sense. I just don't think we should do it. Because testability. | 16:09 |
| jaypipes | sean-k-mooney: that's why I called it overhead_dedicated_Set | 16:09 |
| sean-k-mooney | dansmith: i would equate them as teh same form reading | 16:09 |
| mriedem | efried: "To find images tagged with ready and approved, include tag=ready&tag=approved in your query string. (Note that only images containing both tags will be included in the response.)" | 16:09 |
| dansmith | efried: hmm, okay | 16:09 |
| sahid | johnthetubaguy: ack thanks | 16:09 |
| *** harlowja has quit IRC | 16:09 | |
| cfriesen | jaypipes: currently if you enable "isolate" for the emulator then nova will internally allocate another physical host cpu to run the emulator threads | 16:09 |
| jaypipes | cfriesen: right, and I'm opposed to that, because it f**ks with resource tracking and inventory management. | 16:10 |
| cdent | I said it didn't make sense, but that may be because I'm not fully understanding dan's use case | 16:10 |
| *** claudiub has joined #openstack-nova | 16:10 | |
| dansmith | cdent: I'm on a call right now, but maybe a hangout with interested parties when I'm done? | 16:10 |
| cdent | and also because I wasn't really thinking and on the resources case, more concatenation of value | 16:10 |
| dansmith | if it's really a misunderstanding thing | 16:11 |
| cdent | dansmith: I'd like to understand more but today's not great, i'm in a meeting right now and have been in one or another since I started today | 16:11 |
| efried | The problem is simply that we used ',' to mean 'AND' in the `resources` qparam; but ',' is already taken in `member_of` (to mean 'OR', unfortunately, but disambiguated by requiring `in:`, which consumes another nice piece of punctuation, the ';') | 16:11 |
| jaypipes | cfriesen: I am attempting to create a new resource class that represents a physical dedicated CPU (sahid would rather this resource class be called CPU_DEDICATED insetad of PCPU) that can be used to inventory these kinds of resources (which are currently not tracked in the same way as other resources, much to my dismay) | 16:11 |
| efried | s/';'/':'/ | 16:11 |
| efried | sigh | 16:11 |
| dansmith | efried: ah okay, although that's also kindof already done, but yeah I see that now | 16:12 |
| sean-k-mooney | jaypipes: yes, if we track the overhead sepreatly form the pcpus/vcpu resuces it could imply that the cpus the guest sees is the resouce:*CPU-overhead or we ask for resouce:*cpus + over head form placement | 16:12 |
| cfriesen | jaypipes: the problem is if I have X "dedicated" guest vcpus, and I want to allocate X+1 physical cpus, and run the emulator threads on a physical CPU that isn't running any vCPU thread. | 16:12 |
| jaypipes | sean-k-mooney: this has nothing to do with "what the guest sees". this is a resource accounting issue only. | 16:12 |
| efried | dansmith: so let's move ahead with repeatable member_of. Human reading query string will interpret '&' as 'AND', as it should be. | 16:12 |
| coreycb | mriedem: special sauce indeeed! my bad. | 16:12 |
| mriedem | nice | 16:12 |
| *** sar has joined #openstack-nova | 16:13 | |
| cdent | wait, efried, did you just make a new different decision? I thought we had gone the other way ten minutes ago? | 16:13 |
| efried | vay | 16:13 |
| cfriesen | jaypipes: I'd be fine with CPU_DEDICATED. Would we also have CPU_SHARED instead of VCPU? | 16:13 |
| efried | cdent: Where's your purple thingy? | 16:14 |
| jaypipes | sean-k-mooney: this is only about answering the question "how many dedicated CPUs are left on the host (or on a specific NUMA node)?" and "how many shared CPUs are left on the host (or specific NUMA node)" | 16:14 |
| sean-k-mooney | jaypipes: well it could what im asking is the resouces we counting are track soly by the vaules of resources:PCPU and resources:VCPU and that the overhead extra spec dont effect how we claim in placement | 16:14 |
| *** eharney has quit IRC | 16:14 | |
| cdent | efried: do p!help | 16:14 |
| efried | p!help | 16:14 |
| jaypipes | cfriesen: no, we can't change VCPU to CPU_SHARED, unfortunately. | 16:14 |
| cdent | p!help | 16:14 |
| efried | p!log | 16:14 |
| jaypipes | cfriesen: which is why I wanted PCPU to be the class name for the dedicated CPU resource, since it matched VCPU | 16:14 |
| cdent | you can do it in a privmsg too, so we don't have to see it. there you can leave off the p! prefic | 16:15 |
| efried | cdent: http://p.anticdent.org/logs/openstack-nova?dated=2018-03-22%2016:05:18.169011#4hT5 | 16:15 |
| *** tblakes has quit IRC | 16:16 | |
| efried | cdent: Basically, we collectively on average seem to hate the available options for intra-param punctuation more than we hate deviating from the rest of the API by supporting repeatable for this one new case. | 16:16 |
| sean-k-mooney | jaypipes: can you clear up somthing for me. if i have resources:VCPU=4 resources:PCPU=1 and hw:overhead_dedicated_set=0 do i claim 5 cpus in placement (4 shared and 1 dedicated) | 16:16 |
| cdent | oh, that's not how intepreted that | 16:16 |
| openstackgerrit | Eric Young proposed openstack/nova master: Support extending attached ScaleIO volumes https://review.openstack.org/554679 | 16:16 |
| cdent | I got what I said here about being least disruptive: [t 3cRQ] | 16:17 |
| purplerbot | <cdent> efried: "fixable problem" was the immediate thing of "being able to accept duplicates on keys in general". I agree that addressign dan's problem in the least disruptive way is probably syntax in the param [2018-03-22 15:53:38.038412] [n 3cRQ] | 16:17 |
| *** yamamoto has joined #openstack-nova | 16:17 | |
| *** ragiman has quit IRC | 16:17 | |
| jaypipes | sean-k-mooney: no, you claim 4 VCPU and 1 PCPU resources. | 16:17 |
| jaypipes | sean-k-mooney: there's no such things as a generic "CPU resource" | 16:17 |
| sean-k-mooney | jaypipes: yes that is what i ment | 16:17 |
| *** udesale has joined #openstack-nova | 16:18 | |
| sean-k-mooney | i just wanted to make sure we dont claim 4VCPUs and 2 PCPUS | 16:18 |
| cfriesen | sean-k-mooney: jaypipes: hw:overhead_dedicated_set doesn't make sense since we may want to run emulator threads separate from all vCPU threads (this is what "isolate" does today) | 16:18 |
| sean-k-mooney | and in this case the guest recived 5 logical cpus and the emulator tread is pinned to the PCPU that was allocated to the vm | 16:19 |
| cdent | efried: it probably doesn't matter _that_ much as long as documentation etc | 16:19 |
| cdent | (whichever choice we make) | 16:19 |
| jaypipes | cfriesen: ack. I'm reworking this in the spec after your feedback. | 16:19 |
| *** andreas_s has quit IRC | 16:19 | |
| sean-k-mooney | cfriesen when you say isolate which isolate do you mean | 16:19 |
| jaypipes | cfriesen, sean-k-mooney: can there ever be >1 emulator thread, though? | 16:20 |
| efried | cdent, dansmith, edleafe: Okay, let's make a call one way or another. And send an email. And propose a spec delta. And fix the code. And close the books. | 16:20 |
| cfriesen | jaypipes: yes | 16:20 |
| edleafe | cdent: efried: I didn't realize that we didn't accept multiple resources= params. But I do agree that changing that now is not a good use of time | 16:20 |
| jaypipes | sean-k-mooney: he is referring to hw:emulator_threads_policy=isolate | 16:20 |
| cfriesen | jaypipes: and there can be multiple I/O threads too | 16:20 |
| jaypipes | cfriesen: and how does "hw:emulator_threads_policy=isolate" account for >1 emulator thread? that was the whole point of the overhead_dedicated_set being a pinset -- allowing >1 emulator thread to be specified. | 16:20 |
| sean-k-mooney | jaypipes: ah ok hw:cpu_threads_policy=isolate is different just makeing sure | 16:20 |
| cfriesen | jaypipes: all the emulator threads get affined to a single physical cpu | 16:21 |
| jaypipes | cfriesen: that sounds like a libvirt-specific assumption that will likely change at any given time (as most of these things tend to do) | 16:21 |
| sahid | jaypipes: oh so you can't change VCPU? in that case i'm ok with PCPU | 16:22 |
| *** ftersin has joined #openstack-nova | 16:22 | |
| cfriesen | jaypipes: the emulator threads don't usually have a lot of work to do (the exception is during live migration I think). the goal of "hw:emulator_threads_policy=isolate" is to ensure that the minimal work they do doesn't interrupt the guest cpu threads | 16:22 |
| *** yamamoto has quit IRC | 16:22 | |
| *** ftersin has left #openstack-nova | 16:22 | |
| jaypipes | sahid: I mean, we *could*, but that would be a seriously annoying code change ;) | 16:22 |
| mriedem | melwitt: if you approve a blueprint like https://blueprints.launchpad.net/nova/+spec/hurrah-for-privsep-again you should set the 'series goal' to rocky so it shows up on the actual rocky blueprints page: https://blueprints.launchpad.net/nova/rocky | 16:23 |
| *** udesale has quit IRC | 16:23 | |
| melwitt | mriedem: sorry, I missed that it wasn't set. setting now | 16:23 |
| jaypipes | cfriesen: so it never makes sense to have emulator threads on dedicated CPUs and guest vCPU threads on shared CPUs, right? | 16:24 |
| cfriesen | jaypipes: yes, I suppose that's qemu-specific | 16:24 |
| cfriesen | jaypipes: no, that wouldn't make sense. | 16:24 |
| sahid | jaypipes: yes i'm agreed with cfriesen, that never make sense | 16:24 |
| jaypipes | cfriesen: furthermore, from what I gather from your and sean-k-mooney's feedback, it also would never make sense to have emulator threads policy set to *anything* if the guest vCPU threads were *not* pinned to dedicated CPUs. | 16:25 |
| cfriesen | jaypipes: yes | 16:25 |
| jaypipes | k | 16:25 |
| cfriesen | jaypipes: but it might make sense to have vcpu threads on dedicated cpu, and emulator threads on shared cpu (lumped together with other emulator threads and shared vcpu threads) | 16:26 |
| stephenfin | yeah, that's one of the ideas that came up in reviews of sahid's spec ^ | 16:27 |
| cfriesen | jaypipes: sahid: I'm just not sure if we need to keep the "allocate a whole host cpu for my emulator threads because I'm super important and don't want other guests to impact my emulator threads" | 16:27 |
| sahid | cfriesen: yes, you have asked me, but did not have responded. I think that is not necessary | 16:27 |
| stephenfin | assuming we gain 'pcpu_pin_set' option (or whatever tetsuro's spec called for) | 16:27 |
| *** yingjun has quit IRC | 16:27 | |
| sahid | we just want in some cases the emulthreads to run somewhere | 16:28 |
| jaypipes | stephenfin: CONF.cpu_dedicated_set | 16:28 |
| cfriesen | If we don't need that, then the simplest thing would be to enforce that we always have at least one shared host CPU, and if you ask for hw:cpu_threads_policy=isolate it runs there. | 16:28 |
| stephenfin | that's the one | 16:28 |
| jaypipes | stephenfin: is my proposal, along with CONF.cpu_shared_set being the other disjoint set. | 16:28 |
| stephenfin | or maybe not, but the idea's the same | 16:28 |
| *** masayukig has quit IRC | 16:29 | |
| jaypipes | cfriesen: you mean emulator_threads_policy, not cpu_threads_policy. | 16:29 |
| *** gjayavelu has joined #openstack-nova | 16:29 | |
| cfriesen | bah, yes | 16:29 |
| sahid | cfriesen: i see you point, we can't have cpu_shared_set empty | 16:29 |
| jaypipes | yet another reason all this stuff is so confusing... | 16:29 |
| stephenfin | So we'd completely remove the ability to have one core per guest for emulator threads (e.g. what we currently have) | 16:30 |
| stephenfin | not that I'm against that. Just making sure I understand | 16:30 |
| sahid | stephenfin: yes, no need of that anymore | 16:30 |
| cfriesen | stephenfin: that's really an implementation detail currently, but yes | 16:30 |
| stephenfin | (y) | 16:30 |
| jaypipes | stephenfin: I don't understand that last statement... | 16:31 |
| jaypipes | stephenfin: why would we be removing that ability? | 16:31 |
| cfriesen | jaypipes: if we assume that the emulator thread work scales with the number of vcpus, then we don't need to account for it separately, it's already in cpu_allocation_ratio. This would simplify things since you wouldn't need to ask for an extra VCPU from placement. | 16:32 |
| stephenfin | jaypipes: Because you can't account for the extra per-guest CPU | 16:32 |
| cfriesen | jaypipes: we're saying we'd remove the code that allocates a whole extra host CPU to run the emulator threads | 16:32 |
| *** yamamoto has joined #openstack-nova | 16:32 | |
| stephenfin | what cfriesen says | 16:32 |
| jaypipes | cfriesen: that's exactly *not* what I want. | 16:32 |
| cfriesen | jaypipes: sorry, which conversation thread is that for? | 16:33 |
| cfriesen | :) | 16:33 |
| jaypipes | cfriesen: *something* needs to account for CPU resources consumed by emulator threads. right now, nothing is accounted for in placement. | 16:33 |
| *** moshele has quit IRC | 16:33 | |
| cfriesen | jaypipes: there are two possibilities. 1) emulator thread work scales with the number of guests. 2) emulator thread work scales with the number of vcpus | 16:33 |
| *** andreas_s has joined #openstack-nova | 16:33 | |
| stephenfin | jaypipes: Does it though? I mean, if those threads are running on cores meant for 'shared' workloads, do we care? | 16:34 |
| *** AlexeyAbashkin has quit IRC | 16:34 | |
| cfriesen | jaypipes: if 1 is true, then yes, we need to account for the work done by the emulator threads | 16:34 |
| stephenfin | After all, we don't account for the overhead of emulator threads _without_ the policy applied. We could just ignore it | 16:34 |
| openstackgerrit | Matt Riedemann proposed openstack/osc-placement master: Resolve nits from I552688b9ee32b719a576a7a9ed5e4d5aa31d7b3f https://review.openstack.org/537971 | 16:34 |
| jaypipes | stephenfin: what's the difference between an emulator thread running on a shared CPU and a guest vCPU thread running on a shared CPU, then? | 16:34 |
| stephenfin | You charge for the latter | 16:35 |
| jaypipes | stephenfin: we account for the latter, not the formter. | 16:35 |
| cfriesen | jaypipes: if 2 is true, then the work done by the emulator thread is "pooled" with the work done by the vCPU threads and it's all accounted for in cpu_allocation_ratio | 16:35 |
| jaypipes | stephenfin: but that's my point... we're not accounting for the former when we should be. | 16:35 |
| stephenfin | Also, the potential cycles consumed by emulator thread <<< guest vCPU thread | 16:36 |
| jaypipes | stephenfin: except during live migration? | 16:36 |
| *** masuberu has quit IRC | 16:36 | |
| stephenfin | good point | 16:36 |
| jaypipes | my point being, we're currently not accounting for any of this stuff. | 16:36 |
| *** eharney has joined #openstack-nova | 16:36 | |
| *** yamamoto has quit IRC | 16:36 | |
| cfriesen | jaypipes: stephenfin: I think the interesting case would be if you have a bunch of "dedicated" guest cpus, and the emulator thread running on the "shared" host cpu pool | 16:37 |
| *** crushil has quit IRC | 16:37 | |
| cfriesen | because in that case we really don't account for the emulator thread work currently | 16:37 |
| mriedem | johnthetubaguy: if you're still around, https://review.openstack.org/#/c/520248/ | 16:37 |
| stephenfin | cfriesen: I thought we did. sahid extended some 'overhead' function to do that | 16:37 |
| stephenfin | *currently did | 16:37 |
| johnthetubaguy | mriedem: been meaning to catch you about that | 16:38 |
| cfriesen | stephenfin: right now the emulator threads for an instance run on a dedicated host cpu | 16:38 |
| cfriesen | and we do account for that | 16:38 |
| mriedem | stephenfin: he did https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L804 | 16:38 |
| cfriesen | (if "isolate" is enable) | 16:38 |
| stephenfin | cfriesen: Ah, ok. I missed that nuance | 16:38 |
| cfriesen | stephenfin: but we're talking about running the emulator threads on the "shared" host cpus now | 16:39 |
| mriedem | note that we don't account for overhead in placement at all | 16:39 |
| mriedem | since it's per-compute, and done on the compute | 16:39 |
| jaypipes | cfriesen: well, *all* emulator threads run on the same dedicated CPU? or *each* emulator thread runs on a dedicated host CPU? | 16:39 |
| cfriesen | jaypipes: all on the same | 16:39 |
| openstackgerrit | sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | 16:39 |
| jaypipes | cfriesen: k | 16:39 |
| stephenfin | cfriesen: At the moment? | 16:39 |
| jaypipes | mriedem: this is a different overhead... | 16:40 |
| johnthetubaguy | mriedem: ah https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/remove-configurable-hide-server-address-feature.html would fix that | 16:40 |
| cfriesen | jaypipes: but each instance with "isolate" enabled gets a separate host CPU | 16:40 |
| jaypipes | cfriesen: that's what I just asked you... | 16:40 |
| stephenfin | jaypipes: A guest can have multiple emulator threads | 16:40 |
| cfriesen | jaypipes: all the emulator threads for a given instance run on a single dedicated host CPU | 16:40 |
| jaypipes | cfriesen: ok, so it's each instance gets a separate dedicated host CPU for its emulator thread if emulator_threads_policy=isolate. | 16:40 |
| stephenfin | so your question was confusing :) | 16:40 |
| stephenfin | jaypipes: *for its emulator threads | 16:41 |
| stephenfin | (plural) | 16:41 |
| jaypipes | stephenfin: we don't currently support >1 emulator thread. | 16:41 |
| cfriesen | jaypipes: qemu has multiple emulator threads, so we do | 16:41 |
| mriedem | johnthetubaguy: yeah mentioned that in reply to the patch | 16:41 |
| stephenfin | indeed | 16:41 |
| * stephenfin pipes down | 16:41 | |
| cfriesen | jaypipes: but they're all pinned the same | 16:41 |
| jaypipes | cfriesen: but *we* (i.e. nova libvirt virt driver) never sets >1 right? | 16:41 |
| johnthetubaguy | mriedem: yeah, that is where I got that from | 16:41 |
| cfriesen | jaypipes: it's not up to libvirt, I don't think. it's internal to qemu. | 16:42 |
| stephenfin | this is where sahid should butt in | 16:42 |
| cfriesen | jaypipes: the existance of multiple threads is an implementation detail that is unimportant to the present discussion. | 16:42 |
| sahid | jaypipes: by default when using cpu_policy:dedicated we pin the emulatore threads to the set of pCPUs dedicated for guest | 16:43 |
| johnthetubaguy | mriedem: I think I am about to +W this given we hide it during building, which was my main concern, that seem reasonable? | 16:43 |
| sahid | if that is your question | 16:43 |
| mriedem | johnthetubaguy: i think you +Wing my patch is certainly reasonable yes | 16:43 |
| cfriesen | jaypipes: libvirt will only let you set a single emulator thread affinity for a given domain | 16:43 |
| mriedem | thank you for being british | 16:43 |
| johnthetubaguy | mriedem: heh | 16:43 |
| cfriesen | jaypipes: you *can* set individual IO thread affinities, but we don't muck with those at all at the moment | 16:43 |
| stephenfin | cfriesen: Agreed - it's not important. Sorry for bogging us down there | 16:44 |
| stephenfin | jaypipes: So yeah, one guest = one extra dedicated CPU | 16:44 |
| jaypipes | HOW MANY FRIGGIN CPU RESOURCES IS THIS GUEST CONSUMING? <-- why is it so hard to answer this damn question. | 16:44 |
| *** gjayavelu has quit IRC | 16:44 | |
| stephenfin | without emulator and CPU thread policies, N | 16:45 |
| cfriesen | jaypipes: with qemu you could easily have a dozen or more host threads for an instance with a single vCPU | 16:45 |
| *** yamahata has quit IRC | 16:45 | |
| stephenfin | with emulator thread policy but no CPU thread policy, N + 1 | 16:45 |
| jaypipes | cfriesen: all I care about is the resource accounting. | 16:45 |
| * cdent blinks and steps slowly away | 16:45 | |
| stephenfin | with emulator thread policy and CPU thread policy == isolate, (N * M) + 1 | 16:45 |
| stephenfin | where N = requested CPUs for guest | 16:45 |
| stephenfin | and M is size of the sibling sets of the host | 16:46 |
| *** cdent has left #openstack-nova | 16:46 | |
| stephenfin | typically 2 for x86 platforms (HyperThreading) | 16:46 |
| jaypipes | cfriesen: I don't care how the instance knows which of its virtual processors are pinned to dedicated pCPUs. I don't care about which OS threads are used for emulator processing. All I care about is **how many CPU resources** is the guest consuming. | 16:46 |
| * stephenfin is impressed by cdent's commitment there | 16:46 | |
| *** yamamoto has joined #openstack-nova | 16:47 | |
| *** andreas_s has quit IRC | 16:47 | |
| cfriesen | jaypipes: stephenfin's comments are basically valid, I think. The reason why it's slippery is that the amount of work done by the emulator thread is usually quite small, except for exceptions like live migration. | 16:47 |
| jaypipes | cfriesen: if there isn't a way to calculate that simple resource accounting question, then something is completely f**ked about all of this. | 16:47 |
| sean-k-mooney | jaypipes: well as stephenfin said its for emulator thread policy and CPU thread policy == isolate, (N * M) + 1 but N and M depend on the compute host that is selected | 16:48 |
| jaypipes | cfriesen: furthermore, if there isn't a simple way to do resource accounting, all of this smells like over-engineering to me. | 16:49 |
| sean-k-mooney | well actully N is the numer of guest cpus but M is host dependent | 16:49 |
| jaypipes | ugh | 16:49 |
| stephenfin | jaypipes: It's totally hardware specific and we don't want to leak that level of detail to the user | 16:50 |
| sean-k-mooney | jaypipes: to your privous point however if we have a PCPU resouce and VCPU resouce then we can get rid fo the host depency | 16:50 |
| sean-k-mooney | actully no we cant. we would need to get teh host info to the scheduer filter still | 16:50 |
| cfriesen | jaypipes: currently with dedicated cpus it's deterministic because the emulator threads get a whole separate host CPU | 16:50 |
| jaypipes | stephenfin: I *also* do not want to leak *any* of this crap to the end user. My whole spec is intended to convey a simple to understand concept about consumable CPU resources. | 16:51 |
| jaypipes | stephenfin: i.e. the user wants a dedicated CPU, they ask for that. if they want 4 shared CPUs, they ask for that. | 16:51 |
| cfriesen | jaypipes: If you set the CPU threading policy to "ISOLATE", then the number of host CPUs actually consumed will depend on whether the host has hyperthreading enabled or not. | 16:52 |
| *** yamamoto has quit IRC | 16:52 | |
| jaypipes | stephenfin: the problem is that for some reason, everyone wants to complicate the situation endlessly with hardware-specific doodads that nobody outside of Intel and a couple folks at Red Hat even understand. | 16:52 |
| jaypipes | stephenfin: sorry if I'm frustrated, but by God, why is this stuff so complicated? | 16:53 |
| sahid | jaypipes: i think you are making it more complicated by mixing VCPU/PCPU also that, two option overhed_dedicated/shared_set, we should keep all of that easy, you ask for PCPU or VCPU and you ask to isolate or not the overhead | 16:53 |
| stephenfin | Yup ^ ISOLATE gives us an additional case, where a user ask for entire CPU including thread siblings | 16:53 |
| sahid | one case that run the same as the guest pCPUs assigned and the other torun on CODN.cpu_shared_set | 16:53 |
| cfriesen | jaypipes: realistically, it's complicated because people can use the complexity to get better performance. | 16:53 |
| jaypipes | sahid: how is that simpler than just having the user ask for a quantity of dedicated CPU and a quantity of shared CPU resources? | 16:53 |
| stephenfin | cores, in 'lscpu' terminology | 16:53 |
| sean-k-mooney | stephenfin: ya isolate is teh real issue here | 16:53 |
| sahid | because how are you doing the pinning | 16:54 |
| stephenfin | We could kill the 'ISOLATE' feature, but then folks, me included, are going to wonder what we're gaining for the people that use this | 16:54 |
| jaypipes | sahid: again, I don't care about the pinning. all I care about here is the resource accounting. how *many* of dedicated CPU and shared CPU resources are being consumed on the host. | 16:54 |
| *** wolverineav has joined #openstack-nova | 16:54 | |
| sean-k-mooney | jaypipes: and jay ya i know this is hardware defined infrastruct at its best. sorry to bring up the edgecases | 16:54 |
| *** wolverineav has quit IRC | 16:54 | |
| *** wolverineav has joined #openstack-nova | 16:55 | |
| sean-k-mooney | stephenfin: well we could replace isolate with avoid. whic would meen we just make sure you land on a host that has no hyperthreads so you get teh same performance | 16:55 |
| cfriesen | jaypipes: for the simple case I think it should be possible to make it simple. | 16:55 |
| sean-k-mooney | that could be modled as a trait | 16:55 |
| stephenfin | jaypipes: and it's understandably frustrating but it's telcos that are the main driver of this stuff. We're just enabling it | 16:55 |
| jaypipes | sahid: and by "I", I mean "this spec doesn't care about the pinning". Not that the virt driver doesn't care. Just that the placement service doesn't care or know at all which guest CPU is pinned to which host CPU. | 16:56 |
| sahid | jaypipes: i think we care about the pinnig, because the end user will not see the CPU that will be dedicated for running emultreads | 16:56 |
| cfriesen | sean-k-mooney: then you're back to compute-node level granularity and may as well use aggregates | 16:56 |
| jaypipes | stephenfin: telcos are *not* demanding an utterly incomprehensible way of configuring workloads. they are demanding that there high performance workloads be set up in a way that takes advantage of the hardware as much as possible. | 16:57 |
| stephenfin | aye, what sahid said is the main thing bothering me. The user isn't getting a CPU. They're getting their emulator threads offloaded | 16:57 |
| dansmith | mriedem: sorry I keep missing those comments.. unintentional | 16:57 |
| sean-k-mooney | cfriesen: ya i know... | 16:57 |
| jaypipes | stephenfin: trust me, nobody in Verizon HQ Planning likes trying to understand these CONF and extra spec options. In fact, they pretty much have to rely on RH's triple-o people to configure stuff for them because nobody can understand any of it. | 16:58 |
| cfriesen | stephenphin: agreed. but *internally* we could model that by bumping the VCPU resource count (assuming we ran the emulator thread on the "shared" host cpus). | 16:58 |
| cfriesen | stephenfin: this would basically be giving a rough estimate that the emulator work is roughly equal to a "shared" vcpu | 16:59 |
| jaypipes | cfriesen: isn't that precisely what my spec is proposing? bumping the # of requested VCPU resources to account for emulator threads? | 16:59 |
| sean-k-mooney | jaypipes: true but onap, ecomp osm and open mano all use these things today so we cant break existing users. well we can but we need to know what we are breaking | 16:59 |
| johnthetubaguy | jaypipes: I still think we should have a guide that says "make me go predictably fast and isolated" and another that says "let me go as dense with the packing as I can to get max utilisation" | 16:59 |
| cfriesen | jaypipes: the difference is that I don't want the extra "VCPU" resource to be specified in the flavor. I want nova to add it if the flavor asks for emulator threads to be isolated. | 17:00 |
| jaypipes | sean-k-mooney: ONAP and eCOMP both go around nova and inventory the hosts entirely externally of nova. :( | 17:00 |
| *** gyee has joined #openstack-nova | 17:00 | |
| jaypipes | cfriesen: what is the difference? | 17:00 |
| jaypipes | johnthetubaguy: totes agree. | 17:00 |
| sean-k-mooney | jaypipes: ya i hate that too but they use the flavour extra specs also for things like emulator threads and cpu pinning | 17:00 |
| jaypipes | johnthetubaguy: clearly you've never read the RH NFV OpenStack tuning guide. | 17:01 |
| cfriesen | jaypipes: clarity to the user. If I look at a flavor that specifies 5 cpus, but then the resource in the extra specs asks for 6 cpus, I'm going to think there's a mistake | 17:01 |
| johnthetubaguy | jaypipes: nope, should I? | 17:01 |
| jaypipes | johnthetubaguy: no | 17:01 |
| sahid | cfriesen: yes that is why the additional CPU used when using policy=isolate is not take into account in quota | 17:02 |
| johnthetubaguy | jaypipes: cool, gtk | 17:02 |
| sean-k-mooney | cfriesen: that depends. if the resouces asks for 6 cpus but 5 for the vm the the vms sees and 1 for emulator thread that it does not i think its ok | 17:02 |
| stephenfin | jaypipes: Heh, yeah. I'm not saying it's _not_ all crap as it is (quite the opposite, tbh) but I just don't see yet how we can abstract this kind of stuff better than we do | 17:02 |
| *** yamamoto has joined #openstack-nova | 17:02 | |
| jaypipes | sahid: and that is a resource accounting problem, no? | 17:02 |
| *** fragatina has joined #openstack-nova | 17:02 | |
| sahid | we ask for 4 dedicated pCPUs but actually 5 are dedicated | 17:02 |
| stephenfin | I'm just seeing "if we do this, stuff that does work at the moment breaks" | 17:02 |
| *** idlemind_ has quit IRC | 17:02 | |
| johnthetubaguy | stephenfin: what about having those two options I just mentioned? | 17:02 |
| sahid | jaypipes: it is yes, but saying I want 5 vCPUs and the user only have 4vCPU is also a mistake | 17:03 |
| sean-k-mooney | cfriesen: we accounted for the extra cpu for the emulator for in placement and the guest still got what it expected | 17:03 |
| *** idlemind has joined #openstack-nova | 17:03 | |
| *** fragatin_ has joined #openstack-nova | 17:03 | |
| stephenfin | johnthetubaguy: The problem is, what VZW wants is different from what E/// want | 17:03 |
| openstackgerrit | Merged openstack/osc-placement master: Do not depend on jenkins user in devstack gate https://review.openstack.org/552476 | 17:03 |
| sahid | we don't have any unit in our resource management to take that extra CPU into account | 17:03 |
| cfriesen | sean-k-mooney: I want to distinguish between what's in the flavor and what nova asks from placement. If the flavor says "4 VCPU and I want isolated emulator threads" then I'm fine with nova asking for 5 VCPU. | 17:04 |
| cfriesen | sean-k-mooney: but I don't want the flavor to have to ask for 5 VCPU | 17:04 |
| stephenfin | We could also stop treating emulator threads as a VCPU and add another resource class to that | 17:04 |
| stephenfin | *for that | 17:04 |
| sean-k-mooney | sahid: again it depends if the flavour vcpu field is set to 4 the that means the guest will see 4 cpus. if the resouce:VCPUS is 5 we will claim 5 cpus in placemnt 4 of which would be used by the guest and on that could be used for the emulator | 17:04 |
| jaypipes | sahid: so perhaps the way forward is indeed to have a completely separate CPU_SHARED and CPU_DEDICATED resource class, separte from VCPU. VCPU would represent guest vCPU threads that are using *shared* host CPUs. CPU_SHARED would represent emulator threads (and anything else not guest-related) that are on shared host CPUs, and CPU_DEDICATED would be the amount of dedicated CPUs. | 17:04 |
| stephenfin | (for the emulator_threads_policy issue) | 17:04 |
| johnthetubaguy | stephenfin: I guess I am trying to suggest we document the two extreme cases really well, then document a few ways you might want to dial back in either direction from the extreme? | 17:05 |
| cfriesen | stephenfin: just to simplify things. :) | 17:05 |
| stephenfin | So CPU_SHARED, CPU_DEDICATED, and CPU_EMULATOR_THREADS | 17:05 |
| jaypipes | stephenfin: CPU_EMULATOR_THREADS is not a consumable resource, though... | 17:05 |
| jaypipes | stephenfin: CPU_SHARED is the consumable resource that emulator threads consume. | 17:06 |
| cfriesen | stephenfin: fundamentally right now, emulator threads run on the same host CPUs as the guest CPUs do by default. | 17:06 |
| sahid | jaypipes: yes basically i thought that was you idea since the beginning :) | 17:06 |
| stephenfin | jaypipes: It could be though, if we added the 'emulator_pin_set' list to nova.conf that sahid suggests | 17:06 |
| dansmith | sahid: I left you minor comments in that spec | 17:06 |
| *** yamamoto has quit IRC | 17:06 | |
| sahid | thanks dansmith i will address them | 17:07 |
| cfriesen | stephenfin: I don't like the idea of having a separate dedicated pool just for emulator threads when the "shared" pool is perfectly fine | 17:07 |
| *** fragatina has quit IRC | 17:07 | |
| jaypipes | stephenfin: you are conflating two different things. one is allocation of resources. the other is assignment of emulator threads to a particular set of host CPUs. | 17:07 |
| cfriesen | I think accounting for the emulator threads as a VCPU (specifically if all the cpus in a guest are "dedicated") is a reasonable approximation | 17:08 |
| cfriesen | If any of the cpus in a guest are "shared", then we can run the emulator threads with them. | 17:08 |
| jaypipes | omg, chris you're killing me. | 17:09 |
| stephenfin | jaypipes: Am I? We're saying for the resources for CPU_DEDICATED will come from CONF.cpu_dedicated_set. I'm saying the resources for CPU_EMULATOR_THREADS could come from CONF.cpu_emulator_thread_set | 17:09 |
| *** fragatin_ has quit IRC | 17:09 | |
| sean-k-mooney | jaypipes: can i sugges we create an etherpad and start listing exampels and then clearly state what each means in terms of host,guest and placement allocatiosn | 17:09 |
| cfriesen | jaypipes: ah, forget it. just account for the emulator threads as a VCPU. | 17:09 |
| stephenfin | Then, as a guest, I want N CPU_EMULATOR_THREADS | 17:09 |
| stephenfin | sean-k-mooney: Not a bad idea | 17:10 |
| jaypipes | stephenfin: but according to cfriesen, you can't request that. | 17:10 |
| jaypipes | stephenfin: you can't say "I want 3 emulator threads". | 17:10 |
| bauzas | jaypipes: stephenfin: I'm done with my meeting | 17:10 |
| bauzas | can we jump into a hangout ? | 17:10 |
| jaypipes | bauzas: uhm,.... | 17:10 |
| cfriesen | stephenfin: yeah, there's just some random number of emulator threads that are all pinned the same. | 17:11 |
| jaypipes | bauzas: you've missed quite a bit. | 17:11 |
| bauzas | jaypipes: yeah, that's what I just see... | 17:11 |
| *** jackie-truong has joined #openstack-nova | 17:12 | |
| stephenfin | Riight. OK, forget that | 17:12 |
| sean-k-mooney | maybe this would help https://libvirt.org/formatdomain.html#elementsCPUTuning | 17:12 |
| bauzas | jaypipes: stephenfin: any possible tl;dr ? | 17:12 |
| sean-k-mooney | and this https://libvirt.org/formatdomain.html#elementsIOThreadsAllocation | 17:12 |
| jaypipes | sean-k-mooney: that's the problem... | 17:12 |
| stephenfin | You can tell libvirt what CPUs the emulator threads will be placed on. Currently we say to pin only to one dedicated CPU | 17:12 |
| jaypipes | sean-k-mooney: I am not interested in the pinning questions. I am only concerned with how to do resource accounting. | 17:12 |
| stephenfin | But in this new world, we'd be affining to the entire pool of shared CPUs | 17:13 |
| stephenfin | That first bit was what I was mixing up | 17:13 |
| sean-k-mooney | jaypipes: yes i know. | 17:13 |
| cfriesen | jaypipes: if every guest CPU is "shared" then the emulator threads just float across the same set of host cpus. It's when you have "dedicated" guest CPUs that the question of how to account for the emulator thread work arises. | 17:13 |
| jaypipes | cfriesen: ack, understood. | 17:13 |
| openstackgerrit | Nguyen Hai proposed openstack/nova-specs master: Enhance nova-specs webpage and clean up repo https://review.openstack.org/551802 | 17:13 |
| sean-k-mooney | so looking at the docs you can only have 1 emulator thread. you can have multiple io thread and multiple vcpu trheads | 17:13 |
| cfriesen | sean-k-mooney: don't we spawn another emulator thread when live migrating? | 17:14 |
| stephenfin | bauzas: We're enraging jaypipes to the point that I fear for his pugs' lives | 17:14 |
| jaypipes | cfriesen: but it's not like that is something dynamically calculated, right? I mean, a guest either has dedicated CPUs or it doesn.t' | 17:14 |
| jaypipes | https://tenor.com/view/angry-panda-mascot-mad-gif-3456638 | 17:14 |
| sean-k-mooney | cfriesen: i dont know but in that case to we care | 17:14 |
| cfriesen | jaypipes: correct. | 17:14 |
| bauzas | stephenfin: hold my beer | 17:14 |
| sean-k-mooney | cfriesen: its live migrating all sla are out the window at that point | 17:14 |
| stephenfin | bauzas: Also, we're trying to figure out how we map things like emulator threads policy and the ISOLATE CPU threads policy features to placement | 17:14 |
| bauzas | k | 17:15 |
| jaypipes | cfriesen: so it would be a request for, say resources:CPU_DEDICATED=4 (the vCPU threads) and resources:CPU_SHARED=1 (the emulator thread) | 17:15 |
| *** belmoreira has quit IRC | 17:15 | |
| openstackgerrit | Nguyen Hai proposed openstack/nova-specs master: Enhance nova-specs webpage and clean up repo https://review.openstack.org/551802 | 17:15 |
| cfriesen | jaypipes: if we assume that the emulator thread is doing a non-trivial amount of work then that would be accurate. | 17:15 |
| *** mgoddard has quit IRC | 17:15 | |
| jaypipes | cfriesen: and no request for resources:VCPU, since there's no "VCPU" resources being consumed by the guest. | 17:15 |
| cfriesen | jaypipes: if we assumed the emulator thread is doing a trivial amount of work then we could avoid allocating resources for the emulator thread entirely | 17:16 |
| *** yamahata has joined #openstack-nova | 17:16 | |
| jaypipes | cfriesen: it's not about whether the emulator thread is doing a lot of work or not. it's about whether we want to *account* for it -- which obviously would be up to the operator, yes? | 17:16 |
| jaypipes | cfriesen: if the operator didn't care, they wouldn't put resources:CPU_SHARED=1 in the flavor extra spec | 17:17 |
| *** yamamoto has joined #openstack-nova | 17:17 | |
| cfriesen | jaypipes: okay, I get you. | 17:17 |
| cfriesen | keeping resource allocation separate from the nitty-gritty of actually affining things, I think what you say is true | 17:17 |
| jaypipes | ++ | 17:17 |
| stephenfin | I wonder what operators _would_ care about this? It's not exactly a thing you'd use in a bog-standard cloud | 17:17 |
| sean-k-mooney | jaypipes: i dodnt think we should assume resources:CPU_SHARED=1 is the emulator thread though | 17:18 |
| * bauzas sits at the back and listens | 17:18 | |
| stephenfin | Another idea: we could add sahid's emulator_thread_pin_set option and forget the whole thing accounting thing | 17:18 |
| jaypipes | sean-k-mooney: we're not assuming that. | 17:18 |
| stephenfin | Same way we don't account for things that currently run on the host outside of 'vcpu_pin_set' | 17:18 |
| *** afaranha has quit IRC | 17:18 | |
| jaypipes | sean-k-mooney: it could be anything associated with the guest that is consuming some amount of shared CPU resources. | 17:19 |
| *** mdbooth has quit IRC | 17:19 | |
| stephenfin | That's a variant of the "use the shared pool and ignore the impact idea", but this would ensure our emulator threads wouldn't impact the guests running on said shared pool | 17:19 |
| sean-k-mooney | ok can i sugges we document this here https://etherpad.openstack.org/p/cpu-resource-accounting | 17:19 |
| stephenfin | e.g. during a live migration | 17:19 |
| sean-k-mooney | then we can pull it into the spec | 17:19 |
| jaypipes | sure | 17:20 |
| cfriesen | jaypipes: the thing here though is that "emulator_thread_policy=isolate" is what we currently ask for. I'm not enthusiastic about saying that they *also* need to set CPU_SHARED=1 or similar | 17:20 |
| jaypipes | I need to take a quick walk to screw my head back on, though. | 17:20 |
| cfriesen | I need coffee | 17:20 |
| jaypipes | cfriesen: they wouldn't *need* to. if they didn't, the resource accounting would just continue to be messed up and inaccurate. | 17:20 |
| bauzas | I need to catch up the convo... | 17:20 |
| jaypipes | anyway, I will now take a walk. | 17:21 |
| openstackgerrit | sahid proposed openstack/nova-specs master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/539605 | 17:21 |
| sean-k-mooney | bauzas: the reason i suggested the etherpad is i think we all need to catch up and get it donw on "paper" | 17:21 |
| *** yamamoto has quit IRC | 17:22 | |
| bauzas | sean-k-mooney: etherpads are good, but I tend to think hangouts are way better | 17:22 |
| *** archit has joined #openstack-nova | 17:22 | |
| bauzas | for reaching to a conclusion | 17:22 |
| sahid | hum.. I just noticed your comments johnthetubaguy, let me update the spec | 17:22 |
| *** amodi has quit IRC | 17:22 | |
| stephenfin | bauzas: I think we need to get our ideas in order first though | 17:22 |
| *** archit is now known as amodi | 17:22 | |
| * bauzas opening a new tab then | 17:22 | |
| sean-k-mooney | bauzas: we can do both | 17:22 |
| stephenfin | Also, it's almost home time for me and I have dinner plans this evening :) | 17:23 |
| *** dtantsur is now known as dtantsur|afk | 17:23 | |
| bauzas | your fault | 17:23 |
| bauzas | I just spent the whole day between meetings and troubleshooting | 17:23 |
| bauzas | I just feel I need to pass my nerves on something | 17:23 |
| *** sridharg has quit IRC | 17:24 | |
| bauzas | ah, and kids-sitting, thanks to our glorifous country that doesn't like work | 17:24 |
| *** danpawlik has quit IRC | 17:27 | |
| *** sambetts is now known as sambetts|afk | 17:27 | |
| sahid | mikal: sorry to annoy you each time I have CI problem, but our CI is really slow, no? | 17:30 |
| *** mgoddard has joined #openstack-nova | 17:30 | |
| sahid | wrong channel sorry | 17:30 |
| *** tblakes has joined #openstack-nova | 17:31 | |
| *** yamamoto has joined #openstack-nova | 17:32 | |
| * stephenfin has to bounce. Will pick up on https://etherpad.openstack.org/p/cpu-resource-accounting and jaypipes' spec in the morning | 17:35 | |
| *** derekh has quit IRC | 17:35 | |
| *** damien_r has quit IRC | 17:36 | |
| *** yamamoto has quit IRC | 17:37 | |
| rybridges | Hello. Got a question for you guys. I am trying to setup routed provider networks with Neutron. I created a network with a segment and a subnet associated with that segment. Then the host aggregate for the segment gets created automatically in nova when I do those things. However, when I do openstack aggregate show on the aggregate, I don't see any hosts associated with the aggregate that gets | 17:37 |
| rybridges | created. Is this normal? Or a bug? Or some problem on our end? | 17:37 |
| rybridges | I am running the openvswitch agent on my hypervisors. Dont have any other pre-existing aggregates | 17:37 |
| *** sahid has quit IRC | 17:38 | |
| *** esberglu_ has joined #openstack-nova | 17:39 | |
| *** lucasagomes is now known as lucas-afk | 17:40 | |
| *** esberglu has quit IRC | 17:41 | |
| sean-k-mooney | stephenfin: by the way we should keep https://etherpad.openstack.org/p/cpu-resource-accounting and turn it into docs for this stuff | 17:43 |
| *** READ10 has joined #openstack-nova | 17:47 | |
| *** yamamoto has joined #openstack-nova | 17:47 | |
| *** tblakes has quit IRC | 17:48 | |
| openstackgerrit | Merged openstack/osc-placement master: Resolve nits from I552688b9ee32b719a576a7a9ed5e4d5aa31d7b3f https://review.openstack.org/537971 | 17:49 |
| *** Anticimex has joined #openstack-nova | 17:50 | |
| melwitt | rybridges: I'm not familiar with the details of the feature but at a high level, I would think you (the admin) would have to add hosts to the host aggregates. that is, how would neutron know which hosts to put in which aggregates for you? mlavalle, can you confirm how one is expected to configure on the nova side to use routed provider networks? ^ | 17:51 |
| *** jackie-truong has quit IRC | 17:51 | |
| *** yamamoto has quit IRC | 17:52 | |
| *** danpawlik has joined #openstack-nova | 17:52 | |
| rybridges | Thanks so much for the response melwitt. I was thinking nova might know because of the bridge_mappings in the openvswitch conf. Since all of the bridge mappings on my hypervisor point to the same physical network, and I created the neutron network with that physical network, I was thinking nova / neutron might be able to pick that up. | 17:53 |
| rybridges | We can add the hosts manuall to the aggregate. I just wanted to make sure that is normal and we aren't expecting them to be added for us | 17:53 |
| melwitt | yeah, understood. hopefully mlavalle can confirm when he's around. there are some docs and summit videos about the feature (https://etherpad.openstack.org/p/nova-ptg-rocky L240) but from skimming them I don't see mention of what to do on the nova side | 17:56 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | 17:56 |
| *** felipemonteiro has joined #openstack-nova | 17:56 | |
| *** ccamacho1 has quit IRC | 17:56 | |
| *** danpawlik has quit IRC | 17:57 | |
| *** felipemonteiro_ has quit IRC | 17:59 | |
| *** yankcrime has quit IRC | 18:00 | |
| *** gjayavelu has joined #openstack-nova | 18:00 | |
| *** jpena is now known as jpena|off | 18:00 | |
| openstackgerrit | Merged openstack/nova stable/pike: Revert "Refine waiting for vif plug events during _hard_reboot" https://review.openstack.org/553818 | 18:02 |
| *** yamamoto has joined #openstack-nova | 18:02 | |
| rybridges | Ya we found a talk about it from a summit in 2016 here: https://youtu.be/HwQFmzXdqZM?t=33m5s The guy does a demo and shows that when you create the network and subnet, the aggregate is automatically created and has the compute hosts in there | 18:04 |
| *** yamamoto has quit IRC | 18:06 | |
| *** yamamoto has joined #openstack-nova | 18:06 | |
| *** yamamoto has quit IRC | 18:06 | |
| *** mgoddard has quit IRC | 18:10 | |
| *** suresh12 has joined #openstack-nova | 18:10 | |
| *** boris_42_ has joined #openstack-nova | 18:13 | |
| *** harlowja has joined #openstack-nova | 18:15 | |
| *** AlexeyAbashkin has joined #openstack-nova | 18:15 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add AggregateList.get_by_metadata() query method https://review.openstack.org/544728 | 18:16 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add aggregates list to Destination object https://review.openstack.org/544729 | 18:16 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add request filter functionality to scheduler https://review.openstack.org/544730 | 18:16 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make get_allocation_candidates() honor aggregate restrictions https://review.openstack.org/547990 | 18:16 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add require_tenant_aggregate request filter https://review.openstack.org/545002 | 18:16 |
| openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282 | 18:16 |
| sean-k-mooney | bauzas: oh by the way are you around? regarding the nic feature based scheduling. https://review.openstack.org/#/c/545951/ if i was to start it from scratch today i would totally use placement and tratis and model the requests as tratis on the neutron port for a resouce class of type VIF | 18:16 |
| sean-k-mooney | bauzas: that said i was only ment to work on this feature this cycle as it was assume that little to no code change would be needed since it was approved for the last 2 cycles. | 18:17 |
| sean-k-mooney | bauzas: so im not really sure what to do with https://review.openstack.org/#/c/545951/. | 18:17 |
| *** harlowja_ has joined #openstack-nova | 18:17 | |
| sean-k-mooney | bauzas: 50% of the feature(all the discovery and stroage fo the nic feature in the nova db) has been merged since pike. the only bit that was missing was using it in the schduler and the change to the pcirequest spec. | 18:18 |
| *** oomichi has joined #openstack-nova | 18:18 | |
| *** harlowja has quit IRC | 18:19 | |
| *** AlexeyAbashkin has quit IRC | 18:20 | |
| *** dave-mccowan has quit IRC | 18:20 | |
| openstackgerrit | melissaml proposed openstack/nova master: fix a typo in provider_tree.py https://review.openstack.org/555404 | 18:21 |
| *** danpawlik has joined #openstack-nova | 18:25 | |
| *** danpawlik has quit IRC | 18:30 | |
| *** fragatina has joined #openstack-nova | 18:31 | |
| *** ralonsoh has quit IRC | 18:31 | |
| *** tesseract has quit IRC | 18:32 | |
| openstackgerrit | Dan Smith proposed openstack/nova-specs master: Amend the member_of spec for multiple query sets https://review.openstack.org/555413 | 18:32 |
| dansmith | efried: cdent: edleafe: ^ proposal to argue over | 18:32 |
| *** mdurrant_ is now known as mdurrant | 18:33 | |
| *** gjayavelu has quit IRC | 18:33 | |
| *** gjayavelu has joined #openstack-nova | 18:34 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Allow scheduling only to enabled cells (Filter Scheduler) https://review.openstack.org/550527 | 18:34 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Modify nova-manage cell_v2 list_cells to display "disabled" column https://review.openstack.org/555415 | 18:34 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add --enable and --disable options to nova-manage update_cell https://review.openstack.org/555416 | 18:34 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add disabled option to create_cell command https://review.openstack.org/555417 | 18:34 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Always deallocate networking before reschedule if using Neutron https://review.openstack.org/555418 | 18:36 |
| *** germs has quit IRC | 18:36 | |
| *** germs has joined #openstack-nova | 18:37 | |
| *** germs has quit IRC | 18:37 | |
| *** germs has joined #openstack-nova | 18:37 | |
| kmalloc | dansmith: i wanted to ask you a question, would you (as a nova team/core) be opposed to the concept of an API that (say for pure administration purposes) would let you cleanup all resources (notably for the case of "keystone project has been deleted, but i want all instances cleanup) -- letting nova schedule the "cleanup" (/cleanup-all-resources-for-project/<project-id>). Trying to avoid a "let someoen scope to | 18:38 |
| kmalloc | whatever project, even if it doesn't exist" api, to let folks do that kind of cleanup/iterate over the instances in nova. | 18:38 |
| kmalloc | mriedem: ^ cc (same question for you) | 18:38 |
| jroll | rybridges: not sure if this was in ocata, but routed networks should be using placement aggregates for that, might be worth hitting the placement api and checking those. (again that's definitely how it is in queens, idk about ocata though) | 18:39 |
| *** germs has quit IRC | 18:39 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Network bandwidth resource provider https://review.openstack.org/502306 | 18:39 |
| *** germs has joined #openstack-nova | 18:39 | |
| *** germs has quit IRC | 18:39 | |
| *** germs has joined #openstack-nova | 18:39 | |
| *** lpetrut_ has quit IRC | 18:39 | |
| jaypipes | kmalloc: I would be opposed to that, yes. there are notification event queues that external agents/workers can listen to and do the needful cleanup via the Compute API. | 18:40 |
| mriedem | kmalloc: IMO that is something that can be dealt with in mistral | 18:40 |
| dansmith | kmalloc: you can do that today with a few lines of bash and novaclient right? | 18:40 |
| melwitt | kmalloc: that's not something that's going to be in nova (or any of the other projects themselves) but in I think this openstackclient purge CLI does what you're describing https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/project-purge.html | 18:40 |
| *** READ10 has quit IRC | 18:40 | |
| *** tssurya has quit IRC | 18:41 | |
| sean-k-mooney | melwitt: there is noting preventing an external service/agent/cli tool with admin previlages from doing that in pricipal however correct. its just not implented | 18:41 |
| kmalloc | melwitt: right, but if you can't get a scoped token for a project anymore... | 18:41 |
| mriedem | admin? | 18:41 |
| jaypipes | kmalloc: and administrative token can be procured, no? | 18:41 |
| kmalloc | i was looking at the forward idea of system-scope (for administrative actions) | 18:42 |
| kmalloc | it is tying to eliminate the concept of an "admin" project you scope to for this kind of thing | 18:42 |
| kmalloc | but if nova can be made aware of system-scope for the "admin" cases, i guess that works just as well | 18:42 |
| *** felipemonteiro has quit IRC | 18:42 | |
| sean-k-mooney | kmalloc: e.g. an admin token that only allowed to make requets against a specific host | 18:42 |
| kmalloc | ok, thanks! | 18:43 |
| *** felipemonteiro has joined #openstack-nova | 18:43 | |
| *** tbachman_ has quit IRC | 18:43 | |
| kmalloc | (again, forward looking things) | 18:43 |
| mriedem | https://review.openstack.org/#/c/525772/ | 18:43 |
| kmalloc | yeah | 18:43 |
| kmalloc | ty! | 18:43 |
| kmalloc | jaypipes: sortof, system-scope is designed to not let you do things like make resources (create a VM) - but still do actions like [for example] disable a hypervisor (clearly not a project-scoped action, very much infrastructure admin) [these are examples, not meaning that api would exist] | 18:45 |
| jaypipes | kmalloc: not sure how forward looking it is, considering the bug report is from 2012-03-29. | 18:45 |
| jaypipes | ;) | 18:45 |
| kmalloc | jaypipes: forward looking in keystone-and-ks-middleware support vs "when the bug was filed and we suck at fixing that across openstack's history" ;) | 18:46 |
| sean-k-mooney | kmalloc: ah so a system-scoped token could set the datapath_down field on a neutorn port or admin_state but not delete or create a neutron port | 18:47 |
| kmalloc | sean-k-mooney: right. that would be the kindof idea | 18:48 |
| *** tblakes has joined #openstack-nova | 18:48 | |
| sean-k-mooney | kmalloc: would you also envision it allowing migrate actions also or jsut service level apis | 18:48 |
| kmalloc | sean-k-mooney: the plan is really for things that are meant to be done administratively vs by an end user. Migrate is a bit gray | 18:49 |
| kmalloc | i've seen it used in non-administrative ways. | 18:49 |
| sean-k-mooney | kmalloc: im just trying to tink of why a full could admin token would not be correct in this case | 18:49 |
| kmalloc | though haven't looked at common cases lately (my knowledge there is out dated) | 18:49 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add require_tenant_aggregate request filter https://review.openstack.org/545002 | 18:49 |
| openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282 | 18:49 |
| *** moshele has joined #openstack-nova | 18:50 | |
| kmalloc | sean-k-mooney: trying to break the "manage infrastructure" and "admin can do things for an account/domain/project" | 18:50 |
| sean-k-mooney | kmalloc: ah ok makes sense. | 18:50 |
| kmalloc | yeah, it's something we've needed for a long time | 18:51 |
| kmalloc | because admin via a magic project scope is ... kindof awful | 18:51 |
| kmalloc | it leads to a lot of potential "oopse, this scope bled through and let something happen" making the concept of say "domain admin" hard. | 18:51 |
| jaypipes | kmalloc: kinda awful that is the same as it was >6 years ago... | 18:51 |
| kmalloc | jaypipes: yes, and through many many many many iterations... we have successfully moved the needle to "yeah we still need to fix it" | 18:52 |
| kmalloc | jaypipes: which makes me a bit more than sad. | 18:52 |
| jaypipes | kmalloc: not saying it's not a good idea, just that clearly everyone has worked around it or lived with admin-ness for time eternal at this point. | 18:52 |
| sean-k-mooney | kmalloc: https://review.openstack.org/#/c/525772 will result in a api microversion change also right | 18:52 |
| kmalloc | sean-k-mooney: hm. | 18:52 |
| *** amoralej is now known as amoralej|off | 18:53 | |
| *** cdent has joined #openstack-nova | 18:53 | |
| kmalloc | sure? but how are old microversions managed in policy... wait wait, going to stop going down this rabbithole for now | 18:53 |
| mlavalle | rybridges, melwitt: Neutron creates the aggragates for you: https://github.com/openstack/neutron/blob/master/neutron/services/segments/plugin.py#L216 | 18:53 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Always deallocate networking before reschedule if using Neutron https://review.openstack.org/520248 | 18:54 |
| mriedem | oomichi: can you re-approve this? ^ it sat so long that the test needed to be updated. | 18:54 |
| kmalloc | jaypipes: right, but i think it's time to provide a real fix rather than relying on "everyone has their own workaround" | 18:54 |
| sean-k-mooney | kmalloc: do unscoped tokens work for all scoped apis. if so then you proable dont need on but it might be nice to have a microverion | 18:54 |
| kmalloc | jaypipes: and with policy-in-code, and scope-types we're really moving the right way | 18:54 |
| kmalloc | sean-k-mooney: no, unscoped tokens don't work anywhere | 18:54 |
| kmalloc | excpet to get a scoped token in keystone. | 18:54 |
| jaypipes | kmalloc: ok | 18:54 |
| kmalloc | sean-k-mooney: we're trying to allow RBAC for system-scope actions, unscoped tokens are explicitly no-roles. | 18:55 |
| sean-k-mooney | kmalloc: ok so ya that would be a micro version bump then for backwards compatiblity. | 18:55 |
| melwitt | mlavalle: right, but what about compute hosts that go into those aggregates? is that something neutron can do automatically or is it left up to the admin to have to do that? | 18:55 |
| *** tblakes has quit IRC | 18:55 | |
| melwitt | thanks for chiming in, btw :) | 18:55 |
| kmalloc | sean-k-mooney: yeah i'm not sure how a microversion affects policy enforcement -- it becomes weird. | 18:55 |
| jaypipes | kmalloc: I struggle to have confidence in the firmness of the ideas after seeing keystone v3 domains, federation, PKI and other things go back and forth. | 18:55 |
| kmalloc | jaypipes: well, please chime in on the scope types stuff. | 18:56 |
| mlavalle | melwitt, rybridges: Neutron adds the hosts to the aggregate: https://github.com/openstack/neutron/blob/master/neutron/services/segments/plugin.py#L223 | 18:56 |
| sean-k-mooney | mlavalle: that is for routed networks correct | 18:57 |
| mlavalle | sean-k-mooney: yes it is | 18:57 |
| kmalloc | jaypipes: there are a number of things we have not changed once other services lean on it. [also not sure what you mean firmness of federation, that's stayed the same and been built on, it hasn't been taken out] | 18:57 |
| melwitt | mlavalle: a-ha, okay. so rybridges is seeing behavior where no hosts get added to the aggregate | 18:58 |
| kmalloc | jaypipes: but i'd rather have more eyes/comments than less on the scope-types work. | 18:58 |
| *** avolkov has quit IRC | 18:58 | |
| rybridges | Hey mlavalle! Thanks so much for the reply. We just checked the segmenthostmappings table in the DB and we see that the host mappings are in the table and our compute hosts are listed. But they are not added in the outpute of aggregate show | 18:58 |
| kmalloc | sean-k-mooney: i'll have to see how the policy stuff shakes out with microversions, i'm sure it'll be 100% ok, just not clicking right this moment. | 18:58 |
| *** moshele has quit IRC | 18:59 | |
| *** danpawlik has joined #openstack-nova | 18:59 | |
| kmalloc | sean-k-mooney: which means, i need to poke at things more. hehe :) | 18:59 |
| *** itlinux has quit IRC | 18:59 | |
| *** itlinux has joined #openstack-nova | 18:59 | |
| kmalloc | sean-k-mooney: thanks for your time! | 18:59 |
| *** tbachman has joined #openstack-nova | 19:00 | |
| cfriesen | jaypipes: cpu thread isolation is actually a valid issue for resource accounting, because the amount of resources consumed is variable depending on whether the host has enabled HT or not. | 19:00 |
| *** tblakes has joined #openstack-nova | 19:00 | |
| sean-k-mooney | kmalloc: ya no worres i just didnot notice any microversion bumps in https://review.openstack.org/#/c/525772/4 and sice it has 1+2 already i taught i would ask since it proably should not merge until that is checked | 19:00 |
| *** voelzmo has joined #openstack-nova | 19:00 | |
| jaypipes | cfriesen: I don't doubt that. I'm saying that the cpu_threads_policy has nothing to do with resource accounting. | 19:01 |
| kmalloc | sean-k-mooney: yeah, it might also not require a microversion, because it doesn't actually change anything yet, the change might need a microversion. | 19:01 |
| cfriesen | jaypipes: but it does, since it determines whether we need to allocate additional resources on some compute nodes | 19:01 |
| kmalloc | sean-k-mooney: i'll make sure to get some time to ensure we aren't introducing any breaking changes there (i don't think we are) that would require microversion bumps and if we are... make sure it's added | 19:01 |
| sean-k-mooney | kmalloc: oh adding the scope_types=['project'] does not change the behavior | 19:02 |
| kmalloc | sean-k-mooney: yeah it shouldn't. | 19:02 |
| jaypipes | cfriesen: whether the host has HT enable impacts the reporting of CPU inventory for the host (or its NUMA nodes), but cpu_threads_policy only affects which host processors the guest is willing to be pinned to. | 19:02 |
| *** r-daneel_ has joined #openstack-nova | 19:02 | |
| *** tbachman_ has joined #openstack-nova | 19:02 | |
| jaypipes | cfriesen: are you referring to cpu_allocation_policy? | 19:02 |
| *** r-daneel has quit IRC | 19:02 | |
| *** r-daneel_ is now known as r-daneel | 19:02 | |
| jaypipes | or cpu_threads_policy? | 19:02 |
| kmalloc | sean-k-mooney: yeah it's meant to just allow for richer policy allowing for defining the scope-types in policy as well | 19:02 |
| *** gjayavelu has quit IRC | 19:03 | |
| cfriesen | jaypipes: no. If I ask for 1 PCPU and set cpu_thread_policy=ISOLATE then it will actually consume two host CPUs (the main one and the HT sibling) | 19:03 |
| kmalloc | sean-k-mooney: but that should just be allowing for more specific policy enforcement, not actually making a change to behavior. | 19:03 |
| kmalloc | sean-k-mooney: that is why i was feeling confused, but you know, i am 100% willing to assume I was mis-understanding something since it's in nova's code base and i don't spend as much time there. | 19:04 |
| *** tbachman has quit IRC | 19:04 | |
| *** tbachman_ is now known as tbachman | 19:04 | |
| sean-k-mooney | kmalloc: ok cool sound like you have it well in hand in anycase. just taught i would ask for my own knollage i avoid api changes unless i have no choice | 19:04 |
| *** danpawlik has quit IRC | 19:04 | |
| kmalloc | sean-k-mooney: cool. and again ty very much! | 19:04 |
| cfriesen | jaypipes: basically by enabling cpu_thread_policy=ISOLATE I'm asking for the entire host core, which maps to two ht siblings. But each of those siblings was exposed to placement as a PCPU. | 19:04 |
| jaypipes | cfriesen: it doesn't always map to 2 HTs. | 19:05 |
| cfriesen | jaypipes: true, it could be more | 19:05 |
| jaypipes | cfriesen: yet more hardware vendor undefined randomness. | 19:05 |
| sean-k-mooney | jaypipes: unfrotuently yes | 19:05 |
| cfriesen | jaypipes: or it could be 1, if HT is disabled on the host | 19:05 |
| jaypipes | cfriesen: no, that is the host inventory of things. | 19:06 |
| sean-k-mooney | or 8 HT if its powerpc | 19:06 |
| jaypipes | cfriesen: again, I understand the host inventory part of this. | 19:06 |
| jaypipes | cfriesen: my problem is with flavors that consume different amounts of resources on different hosts. | 19:06 |
| cfriesen | yep, that's exactly what this does, since it depends on the host config | 19:06 |
| *** yamamoto has joined #openstack-nova | 19:07 | |
| sean-k-mooney | jaypipes: ya thats what happens for cpu_thread_policy=ISOLATE today | 19:07 |
| jaypipes | cfriesen: which, due to the design of cpu_threads_policy, being a string of "isolate|prefer|share" is entirely impossible to predict an integer amaount of some CPU resources that will *actually* be consumed by the guest. | 19:07 |
| sean-k-mooney | jaypipes: yes. this was not an issue in icehouse but its biting us now | 19:08 |
| mlavalle | rybridges, melwitt: Look at slide 28 in https://www.slideshare.net/MiguelLavalle/routed-networks-sydney. That shows you what you should see in Placement. For each segment in a routed network, that is the Placement structure that you should see | 19:08 |
| cfriesen | jaypipes: the original goal was to improve flexibility by enabling hyperthreads, while allowing instances to ask for whole cores if they need it for performance. | 19:08 |
| tblakes | mriedem: For bug https://bugs.launchpad.net/nova/+bug/1756360, it looks like we're going to need to implement __repr__ for NovaExceptions. Do you have any input on the format we want to return? | 19:08 |
| openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | 19:08 |
| sean-k-mooney | jaypipes: cfriesen ya the original intel proposal made pinning a host config option with no flavor extra specs. then you would just use aggregate to make teh desision | 19:09 |
| cfriesen | jaypipes: actually, it *is* possible to predict what will be consumed by the guest given the host information | 19:09 |
| cfriesen | jaypipes: it's just that it could be different from one compute node to another | 19:09 |
| mlavalle | rybridges, melwitt: a routed network is a network where segments are associated to its subnets. In other words, if you do a GET of those subnets, all of them should have a valua in their 'sehment_id' attribute | 19:09 |
| sean-k-mooney | cfriesen: not before the placement allocate_candidates request | 19:10 |
| *** voelzmo has quit IRC | 19:10 | |
| mlavalle | rybridges, melwitt: 'segment_id'^^^^ | 19:10 |
| cfriesen | sean-k-mooney: correct, unless we wanted to model siblings in placement. :) | 19:10 |
| *** Swami has joined #openstack-nova | 19:10 | |
| * cfriesen ducks | 19:11 | |
| *** voelzmo has joined #openstack-nova | 19:11 | |
| mriedem | tblakes: can you reply to gibi's question about reproducing this in comment 1 | 19:11 |
| sean-k-mooney | cfriesen: lets not unless its a trait | 19:11 |
| melwitt | rybridges: did you verify whether the nova_api.aggregate_hosts table has anything in it? in case it's some kind of scoping issue with doing the 'openstack aggregate show' not showing it? | 19:11 |
| sean-k-mooney | event then its a state ful trait which is kind of a bad thing | 19:11 |
| cfriesen | sean-k-mooney: actually, a trait of "number of HT siblings" might make sense | 19:11 |
| openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | 19:11 |
| sean-k-mooney | cfriesen: it could but its missueing tratits | 19:12 |
| sean-k-mooney | and it changes the request | 19:12 |
| *** yamamoto has quit IRC | 19:13 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | 19:13 |
| sean-k-mooney | i guess you could tag the PCPU RP with HW_1_HT and HW_2_HT if HT was on | 19:13 |
| *** sdeath has joined #openstack-nova | 19:14 | |
| sean-k-mooney | then maybe combine that with forbiden tratis to avoid HT hosts | 19:14 |
| sdeath | Q: upgrade from Ocata->Pike; now services are showing up as duplicates (same ID, UUID, etc). Any ideas as to what might be causing it? | 19:14 |
| sdeath | nova service-list, openstack compute servicec list | 19:15 |
| sdeath | (asked last couple days running on #openstack) | 19:15 |
| sean-k-mooney | its a strech jay im assuming you would not like tratis for HT amount | 19:15 |
| sdeath | interesting development today: evidently I can't delete them, bombs out, "Service id X refers to multiple services" (although the UUID is the same) | 19:15 |
| sean-k-mooney | melwitt: ^ is the db race you were debugging | 19:16 |
| cfriesen | how bad would it be if the initial pre-check didn't account for siblings properly but the accounting after we picked a compute node did? | 19:16 |
| sean-k-mooney | melwitt: the reader writer lock upgrde thing | 19:16 |
| cfriesen | the NUMATopology filter would still check for siblings properly | 19:16 |
| cfriesen | sean-k-mooney: jaypipes: ^ | 19:17 |
| melwitt | sean-k-mooney: hm, I thought that bug was preventing services without uuids from receiving new uuids. not resulting in duplicates of them? | 19:18 |
| sdeath | there are no duplicate entries in nova.services; I suspect a join against that table is returning duplicate rows, maybe related to the upgrade? two versions of the API present at once? | 19:18 |
| sean-k-mooney | melwitt: oh ok i taught i saw duplicate uuid in the title maybe not | 19:18 |
| dansmith | sdeath: do you have two cells defined pointing at the same db? | 19:18 |
| sean-k-mooney | sdeath: is this an ironic deployment? | 19:18 |
| sdeath | dansmith: I do, it turns out… that my problem? | 19:18 |
| dansmith | sdeath: yup | 19:18 |
| sdeath | ah - very well then; cure for this being, then…? | 19:19 |
| dansmith | sdeath: delete one | 19:19 |
| sean-k-mooney | dansmith: wait sdeath do you also have duplicate host names across cells | 19:19 |
| sean-k-mooney | dansmith: how would you get teh same uuid otherwise? | 19:19 |
| dansmith | sean-k-mooney: two cell mappings pointing at the same db will cause us to list from it twice | 19:20 |
| sdeath | sean-k: it returns the same rows both times… I can paste what I pasted to #openstack (and got kickbanned, thank you Freenode, I AM NOT A SPAMMER grumble mumble bah) | 19:20 |
| jaypipes | cfriesen: "it *is* possible to predict what will be consumed by the guest given the host information" <-- but it's not possible to do scheduling that way. | 19:20 |
| dansmith | sdeath: pastebin, yo | 19:20 |
| melwitt | use paste.openstack.org | 19:20 |
| melwitt | or pastebin | 19:20 |
| sean-k-mooney | dansmith: ah ok so its not two compute services with the same uuid its two cell mappings | 19:20 |
| melwitt | all of that said, you might hit the bug sean-k-mooney mentioned after that, and if so, the fix is up for review currently and will be backported https://bugs.launchpad.net/nova/+bug/1746509 | 19:21 |
| openstack | Launchpad bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] - Assigned to melanie witt (melwitt) | 19:21 |
| sdeath | so OK… I can't kill the cell because it's got hosts in it... | 19:21 |
| sdeath | evidently… | 19:21 |
| cfriesen | jaypipes: right, so what if we ignore siblings for the placement prefiltering, run through the scheduler filters as usual (which will look at siblings properly), then once we pick a host we update the actual allocations in placement based on the knowledge we have of the host. | 19:21 |
| dansmith | sdeath: you'll have to delete it from sql I guess | 19:21 |
| sdeath | DS: is that safe, then? | 19:21 |
| sdeath | if I remove from the nova.cells table? | 19:21 |
| *** voelzmo has quit IRC | 19:22 | |
| dansmith | sdeath: from nova_api.cell_mappings | 19:22 |
| jaypipes | cfriesen: gross. | 19:22 |
| sean-k-mooney | cfriesen: well that is what we were going to do anywya in jays current spec right? | 19:22 |
| mriedem | dansmith: sdeath: we have --force flag on delete-cell i thought? | 19:22 |
| cfriesen | sean-k-mooney: except for the final allocations part at the end | 19:22 |
| rybridges | mlavalle: melwitt: nova_api.aggregate_hosts is empty in the db. that is likely our problem. But as I said earlier the segmenthostmappings table in the neutron db has the right hosts in it.. | 19:22 |
| dansmith | mriedem: that will delete all the hosts | 19:22 |
| dansmith | mriedem: which he doesn't want | 19:22 |
| jaypipes | sean-k-mooney: we weren't going to dynamically adjust the number of resources requested in the allocation requests depending on which host was picked! | 19:22 |
| sean-k-mooney | cfriesen: the allcoation stil happen in the scheduler right | 19:22 |
| mriedem | the mappings, right | 19:23 |
| mriedem | nvm | 19:23 |
| sean-k-mooney | jaypipes: ah ya but it could be done in the schduler and ask placement to validate it | 19:23 |
| sean-k-mooney | jaypipes: we would have to have a down call to the compute host however to figure out if we need to adjust the claim amount | 19:24 |
| dansmith | sdeath: figure out which one is right right one and delete the other | 19:24 |
| cfriesen | sean-k-mooney: isnt' that already in the host infomration? | 19:24 |
| mlavalle | rybridges: sehgmenthostmappings has a lot of rows that will naver make it to Placement. Only those segments that are part of a routed network will have a RP in Placement | 19:24 |
| jaypipes | sean-k-mooney: no. I'm not willing to change the design of the placement service (and the allocation request atomicity/guarantees) just to meet these wack-o requirements. | 19:24 |
| dansmith | sdeath: hopefully all your instance and host mappings refer to one of the two cell mappings | 19:24 |
| melwitt | rybridges: yeah, so that implies the nova API call to add the host must be failing https://github.com/openstack/neutron/blob/master/neutron/services/segments/plugin.py#L224 else segment_host_mappings is empty there | 19:24 |
| sean-k-mooney | cfriesen: what how many ht the host has | 19:24 |
| sdeath | m: let's see if —force exists... | 19:24 |
| sean-k-mooney | cfriesen: its in the numa toplogy blob kindof but not really | 19:25 |
| *** tidwellr_ has quit IRC | 19:25 | |
| dansmith | sdeath: you do not want --force | 19:25 |
| mlavalle | rybridges: in other words. Have you created a routed network yet? | 19:25 |
| dansmith | sdeath: --force on that is actually --recursive | 19:25 |
| *** tidwellr has joined #openstack-nova | 19:25 | |
| cfriesen | jaypipes: once we select a compute node in the scheduler we need to make a call to placement to actually consume the resources, right? | 19:25 |
| sdeath | ds: so maybe we won't do that then | 19:25 |
| sdeath | [backs slowly away from angry bomb with ominous red light glowing on it] | 19:25 |
| dansmith | sdeath: PSA: I dunno what irc client you're using but I bet it supports tab nick completion | 19:25 |
| melwitt | heh | 19:25 |
| cfriesen | jaypipes: oh, but we tell it that we want to actually consume a specific allocation from earlier, | 19:26 |
| sdeath | adium, evidently it does | 19:26 |
| sdeath | so, backtrcing: nova_api.cell_mappings | 19:26 |
| cfriesen | jaypipes: I think I see what you mean, we can't retroactively change the size of an earlier allocation request | 19:26 |
| sean-k-mooney | cfriesen: ya so in this case we would have to make another allocation_candiates request and then find the host n that again and consume that | 19:27 |
| cfriesen | sean-k-mooney: right. Or else we throw out the cpu_thread_policy option in the flavor, or else we model siblings in placement. | 19:27 |
| jaypipes | cfriesen: right. the whole point of the allocation candidates list was "these are the nodes that met your request for resources". if we then just modify that resource amount, we invalidate the result of allocation candidates. :( | 19:28 |
| sdeath | dansmith: OK, rows delete (from host_mappings and from cell_mappings; host_mappings due to foreign_key constraint) | 19:28 |
| sdeath | restarting nova… let's see if this work. | 19:28 |
| sdeath | *wrks | 19:28 |
| sdeath | hey, how about that. only one row per hypervisor! | 19:28 |
| sdeath | OK, so... | 19:28 |
| dansmith | sdeath: so you'll need to do discover_hosts again, but it probably won't find them all because they've been marked as already mapped | 19:29 |
| dansmith | sdeath: you should have changed the id on the others to be the cell you kept | 19:29 |
| sdeath | will it barf if I manually reinsert? | 19:30 |
| dansmith | sdeath: no | 19:30 |
| sean-k-mooney | cfriesen: i dont really see a good way to keep cpu_thread_policy=isolate with out having an allocation on the compute host | 19:30 |
| tblakes | mriedmen: I responded to his comment in bug 1756360. He was seeing that issue because he wasn't passing in a kwarg that nova passes in. | 19:30 |
| jaypipes | cfriesen: or... we just go with my spec and just track dedicated CPUs as their own thing and deal with the workloads that can't tolerate having their dedicated CPUs be hyperthreads as the snowflakey thing that it is when we get to the destination host the scheduler selected. | 19:30 |
| sdeath | dansmith: actually, map_cell_and_hosts seems to have picked them back up.. | 19:30 |
| openstack | bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] https://launchpad.net/bugs/1756360 - Assigned to Tyler Blakeslee (tblakes) | 19:30 |
| sdeath | I see fresh IDs and happiness | 19:30 |
| dansmith | sdeath: that's not what you wanted I think | 19:30 |
| cfriesen | jaypipes: so we'd get to the compute node and find out "oops, it doesn't have as many PCPU resources as we thought" and fail the claim? | 19:31 |
| dansmith | sdeath: didn't that create a new cell mapping again? | 19:31 |
| sdeath | dansmith: nope | 19:31 |
| sean-k-mooney | cfriesen: well claim would happen in scuduler before getting to compute host | 19:31 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Add unit test for non-placement resize https://review.openstack.org/537614 | 19:31 |
| sean-k-mooney | cfriesen: we could at the very end try to extend the claim and only fail if we could not extend | 19:32 |
| sdeath | cell that I wanted to keep is there; new host_mapping IDs mapping the previously-mapped compute nodes to that cell; nova service-list and openstack compute service list show the expected list of hypervisors. | 19:32 |
| dansmith | sdeath: ah, I see where it went, okay | 19:32 |
| dansmith | sdeath: so you're good now? | 19:32 |
| sdeath | nova-manage cell_v2 list_cells shows right number of cells... | 19:32 |
| sdeath | I think it's good. | 19:33 |
| sdeath | so... | 19:33 |
| cfriesen | sean-k-mooney: I'm talking about ResourceTracker.instance_claim() | 19:33 |
| *** danpawlik has joined #openstack-nova | 19:33 | |
| dansmith | sdeath: cool, now read the channel topic :) | 19:33 |
| jaypipes | cfriesen: no... it *would* have as many pCPUs as we thought... it's just *that snowflake of a workload considers hyperthreads not to be good enough pCPUs for it*. | 19:33 |
| cfriesen | jaypipes: heh...that's one way of thinking about it. :) | 19:34 |
| sean-k-mooney | jaypipes: well one exampel of that is realtime cpus and hyperthread dont mix | 19:34 |
| jaypipes | cfriesen: it's the only way to think about it. but I digress.. | 19:34 |
| sdeath | Apologies; I tried #openstack two days running, no soap… wasn't sure where to head from there. There a better "general guidance" channel? | 19:34 |
| jaypipes | sean-k-mooney: I don't care? | 19:34 |
| tblakes | mriedem: I responded to the comment in https://bugs.launchpad.net/nova/+bug/1756360. They were seeing that issue because they were not passing in a kwarg that nova passes in. | 19:34 |
| openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | 19:34 |
| cfriesen | jaypipes: I think that'd work, it wouldn't be any more racy than it is now. | 19:34 |
| dansmith | sdeath: nope, and I feel your pain, but still.. normally we run people out of here _before_ answering their questions | 19:35 |
| sean-k-mooney | jaypipes: similar feeling... | 19:35 |
| cfriesen | sean-k-mooney: the numatopologyfilter would still catch stuff like that I think | 19:35 |
| jaypipes | sean-k-mooney: no, in all seriousness, it doesn't impact the accounting of CPU resources. | 19:35 |
| jaypipes | cfriesen: yes, it would indeed. | 19:35 |
| sean-k-mooney | jaypipes: yes that is true i shoudl be placement | 19:35 |
| sean-k-mooney | it does not care becase it doesn't impact the accounting of CPU resources. numa topology filter might | 19:36 |
| *** awaugama has quit IRC | 19:36 | |
| sdeath | dansmith: then thanks for the assistance! there any objection to lurking? | 19:37 |
| dansmith | sdeath: nope | 19:37 |
| sean-k-mooney | jaypipes: so jay for the snowflake. it lands on a host with HT off and boots fine. it lands on a host with HT on. what happens. retry? | 19:37 |
| melwitt | sdeath: fwiw the openstack@lists.openstack.org is better than the channel for getting responses about deployment/config issues, IMHO | 19:37 |
| *** danpawlik has quit IRC | 19:37 | |
| melwitt | mailing list | 19:38 |
| cfriesen | jaypipes: so if we ignore ht in placement we don't get the atomicity of reservation benefits, but it's no worse than it is currently. we might fail ResourceTracker.instance_claim() or rebuild_claim() or resize_claim() once we get to the compute node. | 19:38 |
| jaypipes | sean-k-mooney: NUMATopologyFilter has no impact on resource accounting whatsoever, for two reasons: a) the results of numa_fit_instance_to_host() are immediately discarded and b) it looks at *assignment*, not allocation. | 19:38 |
| melwitt | gah, more POST_FAILURE on zuul >:| | 19:38 |
| *** patriciadomin has quit IRC | 19:38 | |
| sdeath | melwitt: will give that one a shot if/when needed... thanks! | 19:38 |
| cfriesen | sean-k-mooney: if it lands on a host with HT on, but there's enough free host CPUs, then we behave as now and we're good | 19:39 |
| jaypipes | sean-k-mooney: see what cfriesen just said... the NUMATopologyFilter will catch the HT-snowflake stuff in the scheduler before going to the compute host. | 19:39 |
| sean-k-mooney | jaypipes: yes i know. im aske about the usecase boot me a vm with 2 pinned cpus and tread policy isolate. will that still work with out a retry if the host has hypertreads on | 19:39 |
| *** _nick has joined #openstack-nova | 19:39 | |
| cfriesen | sean-k-mooney: actually no, we'd need to account in placement for the extra CPUs we consume | 19:39 |
| cfriesen | jaypipes: given that the snowflake instance will actually consume more CPUs, we need to account for those extra ones in placement | 19:40 |
| *** tbachman has quit IRC | 19:40 | |
| *** _nick is now known as yankcrime | 19:40 | |
| sean-k-mooney | cfriesen: jaypipes yes so there are two options at that point, 1 we fail to boot and retry, or 2 we ask placement to extend the allocation and fail if it would not fit | 19:40 |
| jaypipes | cfriesen, sean-k-mooney: the fundamental problem with the cpu_thread_policy is that it leads to non-deterministic amounts of requested resources. | 19:40 |
| cfriesen | jaypipes: it's deterministic, but it depends on the host | 19:40 |
| jaypipes | sean-k-mooney: I'm not going to change the allocation request. | 19:40 |
| jaypipes | cfriesen: omg, I'm gonna slap you. | 19:41 |
| jaypipes | :) | 19:41 |
| cfriesen | jaypipes: what about a whole new allocation request | 19:41 |
| jaypipes | cfriesen: it's non-deterministic from the viewpoint of the scheduler | 19:41 |
| sean-k-mooney | jaypipes: ok so we just define it a retry and maybe you could use a weigher to minimies the change it would happen | 19:41 |
| sean-k-mooney | e.g. you land on a host. figure out you need more resouce to run on that host then you asked for and retry on next host | 19:42 |
| cfriesen | sean-k-mooney: so now we're saying that ISOLATE can't possible run on a host with HT enabled. how is this different from an aggregate? | 19:42 |
| jaypipes | cfriesen: for a whole new allocation request, we'd need to re-submit to GET /allocation_candidates with a new requested resource amount, which would give us back a different set of compute hosts, which we would send to the NUMATopologyFilter, which would re-work the allocation request again, and we'd end up in an infinite loop of sadnsees. | 19:42 |
| cfriesen | jaypipes: can we specify a particular compute node when doing the allocation request? | 19:42 |
| sean-k-mooney | cfriesen: actully you can make it work but it doubles the amount of flavors | 19:43 |
| jaypipes | cfriesen: no. | 19:43 |
| cfriesen | jaypipes: how do we handle specifying the compute node when doing a migration? | 19:43 |
| sean-k-mooney | cfriesen: you set flavor.vcpu=4 and resouce[vcpu]=8 and it will work only on ht systems | 19:43 |
| jaypipes | cfriesen: you can ask for an aggregate, an amount of resources, required traits, but not a specific provider (since that would defeat the entire purpose of the GET /allocation_candidates endpoint. | 19:43 |
| cfriesen | sean-k-mooney: that'd work on non-ht as well, technically | 19:43 |
| *** esberglu_ is now known as esberglu | 19:44 | |
| jaypipes | cfriesen: we don't call GET /allocation_candidates when specifying a compute node (force_host). | 19:44 |
| sean-k-mooney | cfriesen: yes but it really expecive maybe add trait:HT_COUNT_2=required to avoid that | 19:44 |
| rybridges | mlavalle: This is how I am creating the network / segment / subnet: http://paste.openstack.org/show/708997/ As you can see the aggregate host list is empty. I was under the impression that this process creates a routed network. Is there something I am missing? | 19:45 |
| jaypipes | sean-k-mooney: that's a trait that looks suspiciously like a quantity of resources. | 19:45 |
| sean-k-mooney | jaypipes: the trait is based on how we said we would do cpu frequency | 19:45 |
| jaypipes | sean-k-mooney: you mean vGPU display heads? | 19:45 |
| sean-k-mooney | e.g. tag it with multipel tratis so 4GHZ cpu would have 1GHZ,2GHZ,3GHZ and 4GHZ traits applied | 19:46 |
| mriedem | melwitt: that's a known issue http://status.openstack.org/elastic-recheck/#1758054 | 19:46 |
| mriedem | everything is blocked until that's merged | 19:46 |
| sean-k-mooney | jaypipes: ya i think that does the same thing but did not look at that that closely | 19:46 |
| jaypipes | sean-k-mooney: but in this case, the amount of resources being *requested* changes depending on which host a workload ends up on :( that's the whole problem with this... | 19:46 |
| melwitt | mriedem: ah, thanks | 19:47 |
| dansmith | is gerrit sucking hard for everyone else? | 19:47 |
| *** tblakes has quit IRC | 19:47 | |
| sean-k-mooney | jaypipes: yes so if you wanted an isoleated vm on a host with HT on you would have a flavor like this | 19:47 |
| cfriesen | jaypipes: what do we call when specifying a compute node on a migration? | 19:48 |
| jaypipes | dansmith: I'm too busy wanting to shoot myself in the head with a bazooka to feel any pain from gerrit. | 19:48 |
| sean-k-mooney | flavor.vcpu=4,resouce[vcpu]=8:traits:HT_count_2=forbid thread_policy=isolate | 19:48 |
| dansmith | jaypipes: roger that | 19:48 |
| cfriesen | jaypipes: and why couldn't we do that in the scheduler if we realize after selecting a host that we need to account for some extra PCPU resources? | 19:49 |
| cfriesen | sean-k-mooney: at that point you may as well use a host aggregate | 19:49 |
| jaypipes | cfriesen: we call PUT /allocations/{migration_uuid} to reserve resources on the source host for the migration and PUT /allocations/{instance_uuid} to consume the instance resources on the destination host. | 19:49 |
| *** r-daneel_ has joined #openstack-nova | 19:49 | |
| jaypipes | cfriesen: we don't go through the shceduler at all when force_host. | 19:49 |
| cfriesen | jaypipes: we do for migrations (when it's really a "suggested host" rather than force) | 19:50 |
| *** tblakes has joined #openstack-nova | 19:50 | |
| *** r-daneel has quit IRC | 19:50 | |
| *** r-daneel_ is now known as r-daneel | 19:50 | |
| sean-k-mooney | cfriesen: well you can have lavor.vcpu=4,resouce[vcpu]=8:traits:ht_count=require thread_policy=isolate and flavor.vcpu=4,resouce[vcpu]=4:traits:HT_count_2=forbid thread_policy=isolate | 19:51 |
| sean-k-mooney | cfriesen: it should have been HT_count_2=require when resouce[vcpu]=8 not forbid originally | 19:51 |
| *** sidx64 has joined #openstack-nova | 19:52 | |
| jaypipes | sean-k-mooney: that just doesn't seem right to me. | 19:52 |
| cfriesen | I really don't want to have multiple extra-spec keys that depend on the value of other extra-spec keys | 19:52 |
| *** sidx64 has quit IRC | 19:52 | |
| jaypipes | and certainly isn't very understandable to me. | 19:52 |
| *** rgb00 has quit IRC | 19:52 | |
| cfriesen | agreed, that's a mess. :) | 19:52 |
| mriedem | we don't go through the scheduler when a host is forced, but conductor does the resource allocation 'claim' | 19:53 |
| mriedem | for live migrate and evacuate | 19:53 |
| *** suresh12 has quit IRC | 19:53 | |
| sean-k-mooney | so we keep coming back to we deprecate and remove tread_policy=isolate or extend the allocation on the compute host which breaks the workflow | 19:53 |
| sean-k-mooney | or we change isolate to mean "host with HT off" | 19:54 |
| cfriesen | mriedem: for "cold migrate to this compute node" we need to be going through the scheduler to do cpu pinning, pci, etc. | 19:55 |
| mriedem | cfriesen: we do go through the scheduler for cold migrate | 19:57 |
| mriedem | there is no force for cold migrate | 19:57 |
| mriedem | remember | 19:57 |
| mriedem | -5 | 19:57 |
| cfriesen | jaypipes: is there an actual API spec somewhere for placement? https://docs.openstack.org/nova/latest/user/placement.html doesn't seem to document the HTTP calls. | 19:57 |
| mriedem | cfriesen: https://developer.openstack.org/api-ref/placement/ | 19:57 |
| mriedem | https://docs.openstack.org/nova/latest/user/placement.html#rest-api goes to ^ | 19:58 |
| cfriesen | mriedem: we allow an optional host, which is not a "force" but is a suggested destination | 19:58 |
| mriedem | cfriesen: yeah, and | 19:58 |
| mriedem | ? | 19:58 |
| mriedem | it goes through the scheduler filters | 19:58 |
| mriedem | and placement | 19:58 |
| cfriesen | mriedem: right, but for that placement call do we ask for all the possible allocation candidates or do we specify a particular host? | 19:58 |
| jaypipes | sean-k-mooney: at this point, I'd much prefer a single trait called HW_CPU_HYPERTHREADING whose absence indicates that hyperthreads are not enabled on the host. | 19:59 |
| mriedem | cfriesen: placement doesn't know about 'hosts' | 19:59 |
| jaypipes | sean-k-mooney: and using the forbidden traits stuff to find compute hosts that don't have hyperthreading enabled. | 19:59 |
| mriedem | cfriesen: we say, 'hey placement, here is my request, give me your shit' | 19:59 |
| mriedem | and then we restrict to just that requested host for the filtering | 19:59 |
| jaypipes | mriedem: and placement goes 'sorry, I don't speak jive'. | 20:00 |
| mriedem | is that a 418? | 20:00 |
| cfriesen | mriedem: okay, so the call to placement still filters all the RPs, then the scheduler filters narrow it down to the requested host? | 20:00 |
| jaypipes | correct. | 20:00 |
| mriedem | cfriesen: yes | 20:00 |
| jaypipes | cfriesen: yes | 20:00 |
| sean-k-mooney | jaypipes: ya im leaning that way too. just get rid of hw:cpu_thread_policy as technical debt but i know that will piss of several people | 20:00 |
| *** damien_r has joined #openstack-nova | 20:00 | |
| jaypipes | sean-k-mooney: if by several people you do accurately mean 2-3 people in the world, I'm willing to live with that. | 20:01 |
| mriedem | doesn't stephenfin have a tattoo for that extra spec? | 20:01 |
| sean-k-mooney | mriedem: i dont think so | 20:01 |
| cfriesen | jaypipes: can we update the allocation? | 20:01 |
| *** gjayavelu has joined #openstack-nova | 20:01 | |
| jaypipes | mriedem: no, but dansmith told me he just got an NFV tattoo. | 20:02 |
| sean-k-mooney | mriedem: if he does that a life lesson in its self | 20:02 |
| jaypipes | on his butt cheek. | 20:02 |
| mriedem | oh i was thinking of https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/share-pci-between-numa-nodes.html | 20:02 |
| dansmith | do not do no | 20:02 |
| sean-k-mooney | cfriesen: we could but this would be the only place we update teh allocation from the compute node in the future | 20:02 |
| sean-k-mooney | cfriesen: hence why we should not | 20:02 |
| cfriesen | sean-k-mooney: I was thinking update it from the scheduler after picking the destination | 20:03 |
| *** eharney has quit IRC | 20:03 | |
| sean-k-mooney | cfriesen: but then again we need to down call to the compute node to know if we need to update it and then have to go to placement and make another allocation_candiates request | 20:03 |
| cfriesen | sean-k-mooney: no, I'm pretty sure that the scheduler has information on number of ht siblings | 20:04 |
| sean-k-mooney | cfriesen: in the host numa topology blob yes | 20:04 |
| cfriesen | sean-k-mooney: I'm looking at the PUT /allocations/{consumer_uuid} operation, which seems to allow updating an existing allocation | 20:05 |
| sean-k-mooney | but im not sure we still have the host_state object at the point we are selecting the node | 20:05 |
| mlavalle | rybridges: what microversion of Placement you are using? | 20:05 |
| *** suresh12 has joined #openstack-nova | 20:05 | |
| cfriesen | sean-k-mooney: that's easy to solve | 20:05 |
| * mriedem goes back to focusing on harakiri | 20:06 | |
| sean-k-mooney | jaypipes: oh an when i say several people i mainly mean my management chain so meh | 20:06 |
| jaypipes | cfriesen: no, we really can't update the allocation... | 20:06 |
| cfriesen | jaypipes: what is that PUT operation then? | 20:06 |
| *** danpawlik has joined #openstack-nova | 20:06 | |
| jaypipes | cfriesen: are you referring to within the scheduler or somewhere else? | 20:06 |
| *** tssurya has joined #openstack-nova | 20:06 | |
| cfriesen | https://developer.openstack.org/api-ref/placement/#update-allocations | 20:06 |
| sean-k-mooney | jaypipes: yes he means update the claim form the schduler by checking the numa toplogy blob from the host state object for the selected host at the very end before we down call to the compute node | 20:07 |
| jaypipes | cfriesen: that call overwrites the allocations that were made for an instance against one or more providers. | 20:08 |
| *** josecastroleon has joined #openstack-nova | 20:08 | |
| sean-k-mooney | cfriesen: its really for things like interface or volumen attach in the future | 20:08 |
| *** yamamoto has joined #openstack-nova | 20:09 | |
| cfriesen | jaypipes: right, so we do the initial allocation, then if HT is enabled on the selecte host and the flavor has cpu_thread_policy=ISOLATE we update the PCPU resource allocation to use twice as many | 20:09 |
| jaypipes | sean-k-mooney: well... not really. it exists to fully replace the set of allocations for a consumer. | 20:09 |
| sean-k-mooney | jaypipes: yes but we do an atomic update using the generation ides so the caller is respocible for doing the merge | 20:09 |
| jaypipes | cfriesen: that is technical debt I do not wish to continue carrying. | 20:10 |
| jaypipes | cfriesen: I do not wish to get into the business of treating some special snowflake workloads differently -- with regards to the allocation behaviour -- from every other instance. | 20:11 |
| *** suresh12 has quit IRC | 20:11 | |
| mriedem | melwitt: small things for https://review.openstack.org/#/c/555092/ | 20:11 |
| sean-k-mooney | cfriesen: technically you could proably do it with an out of tree weigher if you really wanted too. sort by host without ht and if there are none left increase allocation... | 20:11 |
| cfriesen | fair enough. we have a decent number of clients with all-in-one single-node installs that will be unhappy if we tell them that HT is going to be an all-or-nothing kind of thing. | 20:11 |
| melwitt | mriedem: thx | 20:11 |
| jaypipes | cfriesen: all-in-one single-node installs of a "cloud". hmm... | 20:12 |
| *** sidx64 has joined #openstack-nova | 20:12 | |
| *** danpawlik has quit IRC | 20:12 | |
| mriedem | sean-k-mooney: cfriesen: which of you are going to summarize the last 4 hours talking about this in the ML tonight? | 20:12 |
| mriedem | per the PTG retrospective? | 20:12 |
| cfriesen | https://etherpad.openstack.org/p/cpu-resource-accounting | 20:12 |
| sean-k-mooney | mriedem: we have a partial summary here | 20:12 |
| sean-k-mooney | :) we need to add the last case though e.g. deprecate cpu_thread_policy | 20:13 |
| cfriesen | jaypipes: I suspect a number of distrib-cloud folks would be not thrilled with HT becoming a per-compute-node thing. | 20:13 |
| jaypipes | cfriesen: technically, with the HW_CPU_HYPERTHREADING trait solution, one could feasibly set the trait against only one NUMA node provider on the single compute host and effectively carve out one NUMA node for HT-tolerant workloads and the other NUMA node for non HT-tolerant workloads... | 20:14 |
| sean-k-mooney | cfriesen: well there is one other way around this | 20:14 |
| cfriesen | yeah, that's true | 20:14 |
| cfriesen | better than per compute node | 20:14 |
| sean-k-mooney | you can exclude the HT form the new shared and dedicate pin sets | 20:14 |
| *** yamamoto has quit IRC | 20:14 | |
| mriedem | 555314 has been promoted to top of gate now btw | 20:14 |
| cfriesen | sean-k-mooney: doesn't buy you anything because then nothing can run on the ones you excluded | 20:14 |
| sean-k-mooney | cfriesen: it means you can have HT for you shared cores and no HT for your dedicated cores | 20:15 |
| jaypipes | I still think the "apply the HW_CPU_HYPERTHREADING trait to one of the NUMA nodes" is a better, simpler solution. | 20:16 |
| sean-k-mooney | jaypipes: oh had not read that when i typed that am yes that would work | 20:16 |
| jaypipes | and have the flavors that cannot tolerate their pCPUs being on hyperthread siblings add a forbidden=HW_CPU_HYPERTHREADING to the request to placement. | 20:17 |
| sean-k-mooney | jaypipes: you would not even have to do it at the numa level | 20:17 |
| cfriesen | jaypipes: something else...looking at line 80 in the etherpad. if you have a multi-numa-node guest with some of the vcpus being "shared" and some being "dedicated", it actually matters which are which because it affects how many of each you need for the numa-specific RPs | 20:17 |
| sean-k-mooney | you could have 2 RP of VCPU under the same numa node with different tratis | 20:17 |
| *** sshwarts has quit IRC | 20:17 | |
| *** suresh12 has joined #openstack-nova | 20:17 | |
| jaypipes | cfriesen: sure, but doesn't the CONF.cpu_dedicated_set and CONF.cpu_shared_set pinning strings allow you to specify that properly? | 20:18 |
| sean-k-mooney | cfriesen: the numa toplogy filter will fix that | 20:18 |
| jaypipes | sean-k-mooney: keep it simple... | 20:18 |
| sean-k-mooney | jaypipes: well i was assuming ... actully no i was going to say the operator created teh RPs but the virtdriver/compute agent does | 20:19 |
| jaypipes | sean-k-mooney: right. | 20:19 |
| cfriesen | jaypipes: no. suppose I ask for two numa nodes, with 3 "shared" and 1 "dedicated" in virtual numa node 1, and 1 "shared" and 3 "dedicated" in virtual numa node 2. | 20:19 |
| cfriesen | jaypipes: each virtual numa node must map to a physical numa node | 20:19 |
| sean-k-mooney | cfriesen: yes following so far. what part of that will the numa topology filter not validate | 20:20 |
| jaypipes | cfriesen: no, that's not how things currently work, according to sahid and sean-k-mooney at least... multiple virtual NUMA nodes do *not* necessarily map to multiple *host* NUMA nodes. | 20:20 |
| cfriesen | jaypipes: that's true, but each virtual numa node must not cross physical numa node boundaries | 20:21 |
| jaypipes | cfriesen: ah, yes, for sure. | 20:21 |
| sean-k-mooney | jaypipes: in the libvirt driver they do but its not required too. its a result of a icehose bug | 20:21 |
| jaypipes | cfriesen: but, as sean-k-mooney says, the NUMATopologyFilter will catch that stuff. | 20:21 |
| jaypipes | cfriesen: none of that will impact the *amount* of PCPU or VCPU resources that are requested. | 20:21 |
| cfriesen | jaypipes: sure it does, because it affects how many of each I need to ask for from the RP that represents a single numa node | 20:22 |
| sean-k-mooney | cfriesen: but jay is suggesting we dont ask placement for x pcpus in numa node 1 and y on node 2 we just ask for x+y pcpus | 20:23 |
| jaypipes | sean-k-mooney: we *could* ask for 1 PCPU on one provider and 2 PCPU on another provider using granular request groups if we wanted. | 20:24 |
| cfriesen | jaypipes: suppose I have vcpus 0-3 on virtual numa node 0, and vcpus 4-7 on virtual numa node 1. If I specify that I want vpus 0-3 to be shared, that's fundamentally a different request than if I say I want 0,4 to be shared | 20:24 |
| openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | 20:24 |
| sean-k-mooney | jaypipes: yes we could. and as a step 2 we might want to for the numa reasons but what your proposing today would not block us doing that in the future | 20:24 |
| jaypipes | cfriesen: could we do granular request groups for that? i.e. resources1:VCPU=1, resources2:VCPU:1 | 20:24 |
| jaypipes | cfriesen: would say "find my a provider tree that has providers that can serve 1 VCPU each | 20:25 |
| cfriesen | is that how we're representing virtual numa nodes? | 20:25 |
| sean-k-mooney | jaypipes: that kind of what i was thinking of whith the numa stuff on line 203 of https://etherpad.openstack.org/p/cpu-resource-accounting | 20:26 |
| jaypipes | k | 20:26 |
| jaypipes | sean-k-mooney: ack | 20:26 |
| cfriesen | because we're going to need to be able to ask for "X PCPU, Y VCPU, Z memory pages of size A" all on one host NUMA node | 20:26 |
| *** sidx64_ has joined #openstack-nova | 20:26 | |
| sean-k-mooney | cfriesen: that should be fine with granular requrest | 20:27 |
| sean-k-mooney | i think | 20:27 |
| jaypipes | cfriesen: we haven't considered memory pages yet, but yeah, that's what granular request groups are for. | 20:27 |
| *** ircuser-1 has joined #openstack-nova | 20:27 | |
| jaypipes | btw, I proposed standardizing memory pages as resource classes a year ago: https://review.openstack.org/#/c/442718/ | 20:27 |
| sean-k-mooney | jaypipes: the mem_page_size can just be a trait | 20:28 |
| jaypipes | please no. | 20:28 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add functional regression test for bug 1746509 https://review.openstack.org/555092 | 20:28 |
| openstack | bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] https://launchpad.net/bugs/1746509 - Assigned to melanie witt (melwitt) | 20:28 |
| openstackgerrit | melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093 | 20:28 |
| jaypipes | sean-k-mooney: are they consumable things? | 20:28 |
| sean-k-mooney | jaypipes: the mempages are | 20:28 |
| jaypipes | right, so they should be resource classes. | 20:28 |
| cfriesen | okay. so I think the end-user would find it convenient to specify *which* guest CPUs are shared/dedicated. If we specify only "how many" then they have to discover it at boot time and have sufficiently flexible software to handle setting up dynamic affinity based on what they discovered. | 20:28 |
| jaypipes | sean-k-mooney: but, meh, another day for that discussion on memory pages. | 20:29 |
| sean-k-mooney | so we have a rp with an inventory of mempages with a 2MB trait as a child of the numa node | 20:29 |
| *** sidx64 has quit IRC | 20:29 | |
| *** tbachman has joined #openstack-nova | 20:29 | |
| jaypipes | cfriesen: no disagreement from me, but again.... not related to placement/resource accounting. | 20:29 |
| sean-k-mooney | jaypipes: oh i was agreeing the mempage should be a resouce class. i just dont know if we want mempage2MB and mempage1G | 20:30 |
| jaypipes | sean-k-mooney: let's discuss the mempages stuff another day, eh? :) | 20:30 |
| sean-k-mooney | jaypipes: sure | 20:30 |
| sean-k-mooney | can we put this in https://etherpad.openstack.org/p/cpu-resource-accounting then copy it into the spec | 20:31 |
| cfriesen | jaypipes: so if we allow the end user to specify something like "hw:shared_vcpus=0,1,4,5", it seems to me that nova should internally map that to the desired granular request rather than needing to specify it explicitly in the flavor extra spec. | 20:31 |
| cfriesen | jaypipes: given this, it is implied that we want 4 VCPU resources, with the remainder being PCPU | 20:32 |
| *** syjulian has quit IRC | 20:33 | |
| *** MikeG451 has quit IRC | 20:33 | |
| cfriesen | jaypipes: alternately, if we explicitly specify the VCPU and PCPU count, and make them discover the mapping at boot time, then we wouldn't need the "hw:shared_vcpus" extra spec | 20:33 |
| jaypipes | cfriesen: lemme make sure I understand you... | 20:33 |
| *** awaugama has joined #openstack-nova | 20:33 | |
| jaypipes | cfriesen: so you are saying that, just for VCPU and PCPU resource classes, that if nova sees the magic hw:shared_vcpus extra spec, that nova should figure out how many VCPU and how many PCPU it should ask for from placement instead of requiring the admin to put a "resources:VCPU=X" and "resources:PCPU=X" extra spec in the flavor? | 20:34 |
| sean-k-mooney | cfriesen: so flavor.vcpus=8, hw:shared_vcpus=0,1,4,5 resouce[vcpu]=4 resouces[pcpu]=4 meaning all odd guest cpus are shared and all even cores(those not in that list) are dedicate cores. | 20:35 |
| cfriesen | jaypipes: I think that would be the most convenient option for the end user | 20:36 |
| sean-k-mooney | cfriesen: the hw:shared_vcpus=0,1,4,5 dose not change the placement request however right its just for the virt diriver/numa topology filter | 20:37 |
| jaypipes | cfriesen: I prefer to have them discover the mapping at boot time and just specify VCPU and PCPU counts. | 20:37 |
| cfriesen | jaypipes: that would be more generic, yes. | 20:37 |
| jaypipes | cfriesen: that gives the virt driver the freedom to map the guest CPUs whatever way it needs, and once the virt driver makes that mapping decision, it could just write it to the instance metadata for the guest to read on boot. | 20:38 |
| cfriesen | jaypipes: in that case I'd suggest that we make the lower-numbered vCPUs in a virtual numa node be "shared", and the higher-numbered ones "dedicated" | 20:38 |
| *** danpawlik has joined #openstack-nova | 20:39 | |
| cfriesen | but yeah, up to the virt driver | 20:39 |
| *** moshele has joined #openstack-nova | 20:39 | |
| rybridges | mlavalle: {"versions": [{"min_version": "1.0", "max_version": "1.4", "id": "v1.0"} | 20:39 |
| jaypipes | cfriesen, sean-k-mooney: ok, I'm going to update my spec and attempt as best as possible to recap the above decisions. | 20:40 |
| cfriesen | we'd have to persist the mapping somewhere to preserve it over live migration. and it'd be nice to keep it over cold migration/evacuate too | 20:40 |
| jaypipes | cfriesen: instance metadata... | 20:40 |
| jaypipes | cfriesen: just like how we save device metadata/tags today, right? | 20:40 |
| cfriesen | should work, I think | 20:41 |
| *** boris_42_ has quit IRC | 20:41 | |
| sean-k-mooney | cfriesen: you say per numa node but you realsie libvirt does not map guest cores to virtual numa nodes right | 20:42 |
| sean-k-mooney | actully you kind of can. | 20:42 |
| sean-k-mooney | <numa> | 20:42 |
| sean-k-mooney | <cell id='0' cpus='0-3' memory='512000' unit='KiB'/> | 20:42 |
| sean-k-mooney | </numa> | 20:43 |
| *** danpawlik has quit IRC | 20:43 | |
| sean-k-mooney | cpus in the cell id is the guest logical cpu | 20:43 |
| *** damien_r has quit IRC | 20:45 | |
| *** cdent has quit IRC | 20:45 | |
| *** AlexeyAbashkin has joined #openstack-nova | 20:45 | |
| sean-k-mooney | jaypipes: the gotcha with the metadata preseting the mapping is that on live migration we would want to make sure the mapping did not change as the running workload would not likely call the metadata api again | 20:45 |
| cfriesen | sean-k-mooney: yeah, that's doable | 20:46 |
| sean-k-mooney | that said NFV + live migration does not mix no matter how much telcos want it to | 20:46 |
| cfriesen | sean-k-mooney: it just means that NUMATopologyFilter needs to check for that case | 20:47 |
| jaypipes | sean-k-mooney: we'd still want to update the instance metadata to reflect the pinning on the destination host, though, so when the workload rebooted, it was able to configure itself. | 20:47 |
| sean-k-mooney | cfriesen: meaning the numa toplogy filter would have to query the metatdata service | 20:47 |
| cfriesen | jaypipes: no, we'd need to keep the "which vcpus are dedicated" mapping the same over the live migration | 20:47 |
| sean-k-mooney | cfriesen: maybe put that in its own filter. | 20:47 |
| cfriesen | jaypipes: we don't need to store the virtual-to-physical mapping in the metadata | 20:48 |
| *** damien_r has joined #openstack-nova | 20:48 | |
| sean-k-mooney | cfriesen: ok i should have left 2 hours ago. letse leave the edgecase to live migration of snowflake vnf to another spec? | 20:49 |
| cfriesen | sean-k-mooney: lol...me too. got an issue to debug | 20:49 |
| sean-k-mooney | cfriesen: it has no effect on placement just the numatoplogy+everything_else_super filter | 20:49 |
| sean-k-mooney | jaypipes: are you ok writing this up in the spec? | 20:50 |
| cfriesen | sean-k-mooney: correct | 20:50 |
| *** damien_r has quit IRC | 20:51 | |
| sean-k-mooney | ok im off tomorow but if you ping me ill proably check irc at some point | 20:52 |
| *** AlexeyAbashkin has quit IRC | 20:52 | |
| *** edmondsw has quit IRC | 20:56 | |
| *** edmondsw has joined #openstack-nova | 20:56 | |
| cfriesen | if we report the "which vcpus are dedicated" mapping via metadata, what happens if the virtual routing is such that the guest has no access to the metadata server? will the config drive get suitably updated? | 20:57 |
| *** damien_r has joined #openstack-nova | 20:57 | |
| sean-k-mooney | config drive is read only so no | 20:57 |
| sean-k-mooney | you have to proxy to metadata via dhcp server if its an isolated neutron network | 20:57 |
| sean-k-mooney | config dirive might get update on cold migrate or hard reboot | 20:58 |
| cfriesen | sean-k-mooney: readonly is fine, the guest isn't going to be updating it | 20:58 |
| cfriesen | and the mapping of which guest cpus are shared/dedicated shouldn't change dynamically once booted | 20:59 |
| sean-k-mooney | yes but qemu can unplug it and update it on livemigrate either | 20:59 |
| *** josecastroleon has quit IRC | 20:59 | |
| cfriesen | we need to keep it constant over live migrate | 20:59 |
| cfriesen | so no problem | 20:59 |
| *** damien_r has quit IRC | 21:00 | |
| mlavalle | rybridges: the microversion required on the Neutron side is 1.1. So it seems right. At the time of creating your subnet, do you you see any tracebacks in the Neutron server log? If so, please share it in a paste | 21:01 |
| sean-k-mooney | sure. it just means more complicated code so when we calluate the pinning for the dest we have to maintian the shared/dedicated logical core mappings which is technical debt but ok | 21:01 |
| *** damien_r has joined #openstack-nova | 21:01 | |
| *** edmondsw has quit IRC | 21:01 | |
| cfriesen | sean-k-mooney: not really technical debt. logically the guest shouldn't need to be constantly checking which guest cpus are shared/dedicated. should only need to happen at boot time, or ideally on creation/rebuild | 21:02 |
| dansmith | efried: excellent use of proper ITU phonetics | 21:02 |
| *** hamzy has quit IRC | 21:03 | |
| *** sidx64_ has quit IRC | 21:03 | |
| sean-k-mooney | cfriesen: yes but it does mean that this is yes another thing the fit_instance_to_host function needs to enforce | 21:04 |
| cfriesen | sean-k-mooney: yes, but it's required as soon as you allow mixed shared/dedicated in one instance. | 21:04 |
| *** tblakes has quit IRC | 21:05 | |
| *** damien_r has quit IRC | 21:05 | |
| sean-k-mooney | cfriesen: yes which we dont allow today so once its added that fucntion need to be extended to allow passing in a set of mappings and the validate that the would still be correct for the new host. | 21:05 |
| *** suresh12 has quit IRC | 21:08 | |
| sean-k-mooney | cfriesen: again its doable but https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4364-L4510 is not exactly the most plesant code to debug currently. anyway this time im really leaving | 21:08 |
| sean-k-mooney | o/ | 21:08 |
| *** suresh12 has joined #openstack-nova | 21:08 | |
| *** suresh12 has quit IRC | 21:10 | |
| *** suresh12 has joined #openstack-nova | 21:10 | |
| *** damien_r has joined #openstack-nova | 21:10 | |
| *** yamamoto has joined #openstack-nova | 21:10 | |
| *** danpawlik has joined #openstack-nova | 21:14 | |
| *** itlinux has quit IRC | 21:14 | |
| *** yamamoto has quit IRC | 21:16 | |
| *** AlexeyAbashkin has joined #openstack-nova | 21:17 | |
| cfriesen | jaypipes: so is your overall goal to get rid of the current hw:numa_mem.X=Y, hw:numa_cpu.X=Y, hw:mem_page_size=2048, etc. and specify it all explicitly as placement resources? | 21:17 |
| cfriesen | jaypipes: because the alternative would be to keep them as-is and have nova calculate the allocation | 21:18 |
| dansmith | cfriesen: fwiw, I have't been following all of this today, | 21:18 |
| dansmith | but I'm highly allergic to plans for us trying to automate that fitting of things onto numa nodes | 21:19 |
| dansmith | I don't like that it's manual, but it's a headache I don't really want to own | 21:19 |
| *** tidwellr has quit IRC | 21:19 | |
| *** danpawlik has quit IRC | 21:19 | |
| *** salv-orlando has quit IRC | 21:19 | |
| dansmith | so convincing me is going to take some damn fine, simple code and a lot of tests | 21:19 |
| *** gouthamr has quit IRC | 21:19 | |
| *** salv-orlando has joined #openstack-nova | 21:19 | |
| cfriesen | dansmith: so are you suggesting we keep what we have now or require explicit resource specification in the flavor? | 21:20 |
| dansmith | I would expect we'd take the thing we have now and turn it into a granular resource request based on the total, and the pieces you specified per node | 21:20 |
| dansmith | cfriesen: I don't really have a suggestion, I'm just saying, every time I think I could genericify that someone brings up a case they want to support which is hard | 21:21 |
| *** AlexeyAbashkin has quit IRC | 21:21 | |
| dansmith | and I expect people that need/use that level of fine-grained control over what the system looks like are doing it for suuuuper specific reasons | 21:22 |
| openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | 21:22 |
| dansmith | like "I have a custom irreplaceable application that only runs on windows nt 4.0 which has a never-going-to-be-fixed bug where it requires 2MB of memory on the same node as the cdrom's IDE controller or my application fails to read from the light pen" | 21:23 |
| *** salv-orlando has quit IRC | 21:24 | |
| cfriesen | dansmith: nice. | 21:25 |
| * dansmith takes a bow | 21:25 | |
| *** salv-orlando has joined #openstack-nova | 21:26 | |
| cfriesen | If we're going to allow shared/dedicated CPUs within a single instance then we'll need to add some way of specifying at a minimum how many "shared" or "dedicated" vcpus we want in each virtual numa node. Whether that's explicitly via resources or via some other thing that nova factors into the resource allocation request is sort of up in the air. | 21:28 |
| cfriesen | But if we keep the existing fields we have now (which would be nice to avoid breaking people) nova will need to translate that into allocation requests for VCPUS or PCPUS (to use jay's terminology) | 21:30 |
| melwitt | dansmith, mriedem: I was just looking at https://blueprints.launchpad.net/nova/+spec/libvirt-cpu-model-extra-flags again. is the ability to enable various cpu flags a feature that we would want anyway regardless of the meltdown mitigation? wondering why one of the choices isn't just "enable_pcid = True/False" and leave it at that, for the backport and going forward? that way there's no migration path nor concern about opening up more | 21:31 |
| melwitt | bug possibilities for stable branches | 21:31 |
| melwitt | sigh | 21:31 |
| dansmith | melwitt: yeah I think people have wanted that for other reasons | 21:32 |
| dansmith | melwitt: like, choose a lower base cpu but add in avx2 | 21:32 |
| dansmith | or something like that | 21:32 |
| melwitt | okay, I see | 21:32 |
| rybridges | mlavalle: We are not seeing any stacktraces in neutron-server.log or in nova-api.log or in nova-placement-api.log. It looks like the association is not working on this line: https://github.com/openstack/neutron/blob/stable/ocata/neutron/services/segments/plugin.py#L225 Nothing after that line is being executed. But we dont see errors or stack traces. When we look at the resource_providers in | 21:32 |
| rybridges | placement, the segment is there and registered as a resource provider, but the aggregate is not associated with it. | 21:32 |
| dansmith | melwitt: you can already configure your cpu model to be different and break live migration, so this wouldn't be any different | 21:33 |
| melwitt | okay. if it's a feature that has utility going forward (specifying various flags) then I'm inclined to favor the idea of backporting it as the full-fledged feature being that the risk is lower than the upgrade pain for operators to go from a [workarounds] option -> cpu_model_extra_flags | 21:35 |
| dansmith | melwitt: did you see my latest suggestion on the patch? | 21:35 |
| melwitt | no, looking now | 21:36 |
| dansmith | melwitt: backporting it with a restriction that pcid is the only thing you can put in that option would eliminate the general-purpose use of it without causing the deployment pain | 21:36 |
| melwitt | ah, I see. that's a nice idea | 21:36 |
| melwitt | I think that addresses all of the concerns | 21:37 |
| dansmith | me too | 21:37 |
| melwitt | noice | 21:37 |
| melwitt | what do you think of that mriedem? | 21:38 |
| *** moshele has quit IRC | 21:39 | |
| mriedem | so the backport has the choices kwarg with a single item? | 21:39 |
| mriedem | and we drop choices in master? | 21:39 |
| dansmith | that's one way yeah | 21:39 |
| dansmith | well, I'm asking right now if we want a list anyway | 21:40 |
| *** artom has quit IRC | 21:40 | |
| dansmith | or some sort of reasonable validation | 21:40 |
| dansmith | if we can get the list of all options from libvirt or something | 21:40 |
| *** artom has joined #openstack-nova | 21:40 | |
| mriedem | cpu_map.xml comes back in the libvirt host capabilities stuff we already have i think | 21:41 |
| mriedem | we dump it on startup to the logs | 21:41 |
| melwitt | yeah, I was thinking maybe we have choices and there's only one choice in the backport and going forward there will be a set of choices we know are good? or are there just too many to reasonable do it that way | 21:41 |
| dansmith | yeah, so I would think we should validate what they give us anyway, so we don't just generate broken xml | 21:41 |
| dansmith | but that would be later than in conf choices | 21:41 |
| mriedem | http://logs.openstack.org/84/534384/9/check/tempest-full/a66bf1a/controller/logs/screen-n-cpu.txt.gz#_Mar_22_13_34_01_146931 | 21:42 |
| mriedem | i would say, (1) backportable version has just a single choice, pcied or whatever, and then (2) another patch, master-only, drops choices and we validate on startup of the driver based on the host capabilities | 21:43 |
| mriedem | not in config | 21:43 |
| dansmith | that's cool, | 21:43 |
| dansmith | although I think just landing it with the validation and restricted choice would be fine too | 21:43 |
| dansmith | but if you like it being smaller with no validation then that's cool | 21:43 |
| *** esberglu has quit IRC | 21:44 | |
| mriedem | oh i don't care if we add the validation in the backport change too | 21:44 |
| mriedem | if that's reliable | 21:44 |
| mriedem | i'm not sure what in ^ is the thing we would use to validate | 21:44 |
| *** esberglu has joined #openstack-nova | 21:44 | |
| mriedem | the cpu features? | 21:44 |
| mriedem | http://logs.openstack.org/84/534384/9/check/tempest-full/a66bf1a/controller/logs/screen-n-cpu.txt.gz#_Mar_22_13_34_01_148516 | 21:45 |
| melwitt | yeah, if there's a chance that pcid could fail validation, then it would be good to validate it before going ahead | 21:45 |
| dansmith | I think, but kashyap did say at one point it wasn't obvious and we couldn't automate the enabling of this too, | 21:45 |
| dansmith | so we might be barking up the wrong tree | 21:45 |
| dansmith | either way it doesn't matter.. if we can, we should, if we can't, then we won't :) | 21:45 |
| dansmith | I already asked on the review so we'll see what he says | 21:45 |
| mriedem | good luck | 21:45 |
| melwitt | okay, I'll approve the bp with a summary of what we agreed on the approach | 21:46 |
| dansmith | oh I thought we had already agreed to approve it | 21:46 |
| melwitt | we did, I meant more the approach part | 21:46 |
| melwitt | I hadn't approved it yet while we were debating how to go forward | 21:47 |
| dansmith | alright | 21:47 |
| *** tssurya has quit IRC | 21:47 | |
| *** danpawlik has joined #openstack-nova | 21:49 | |
| *** esberglu has quit IRC | 21:49 | |
| *** suresh12 has quit IRC | 21:51 | |
| *** Drankis has quit IRC | 21:52 | |
| *** pcaruana has quit IRC | 21:53 | |
| sean-k-mooney | discussing https://review.openstack.org/#/c/534384/ ? | 21:53 |
| mlavalle | rybridges: that's odd. I have a meeting in 5 min and then I have to run. can I ping you tomorrow? | 21:53 |
| *** danpawlik has quit IRC | 21:54 | |
| *** yamamoto has joined #openstack-nova | 21:55 | |
| rybridges | mlavalle: sure dude! Thanks for the help today | 21:55 |
| mlavalle | rybridges: :-) what's your time zone? I'm in US Central (Austin) | 21:56 |
| *** suresh12 has joined #openstack-nova | 21:59 | |
| openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | 22:02 |
| *** amodi has quit IRC | 22:03 | |
| *** awaugama has quit IRC | 22:04 | |
| *** archit has joined #openstack-nova | 22:06 | |
| rybridges | mlavalle PST (California) | 22:08 |
| rybridges | I'll be in 8am-7pm PST | 22:09 |
| mlavalle | ok cool | 22:09 |
| *** itlinux has joined #openstack-nova | 22:10 | |
| *** burt has quit IRC | 22:12 | |
| *** archit has quit IRC | 22:12 | |
| *** wolverineav has quit IRC | 22:17 | |
| *** damien_r has quit IRC | 22:18 | |
| *** wolverineav has joined #openstack-nova | 22:18 | |
| *** damien_r has joined #openstack-nova | 22:20 | |
| *** danpawlik has joined #openstack-nova | 22:23 | |
| *** crushil has joined #openstack-nova | 22:25 | |
| *** crushil has left #openstack-nova | 22:26 | |
| *** danpawlik has quit IRC | 22:29 | |
| *** pchavva has quit IRC | 22:29 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Teardown networking when rolling back live migration even if shared disk https://review.openstack.org/555481 | 22:32 |
| openstackgerrit | Patricia Domingues proposed openstack/nova master: load up the volume drivers by checking architecture https://review.openstack.org/541393 | 22:34 |
| *** rcernin has joined #openstack-nova | 22:34 | |
| *** damien_r has quit IRC | 22:34 | |
| *** suresh12 has quit IRC | 22:35 | |
| *** suresh12 has joined #openstack-nova | 22:36 | |
| *** suresh12 has quit IRC | 22:44 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Teardown networking when rolling back live migration even if shared disk https://review.openstack.org/555481 | 22:44 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DRY up test_rollback_live_migration_set_migration_status https://review.openstack.org/555489 | 22:44 |
| *** edmondsw has joined #openstack-nova | 22:46 | |
| *** andreas_s has joined #openstack-nova | 22:47 | |
| *** andreas_s has quit IRC | 22:52 | |
| *** liverpooler has quit IRC | 22:52 | |
| *** masber has joined #openstack-nova | 22:56 | |
| *** hongbin has quit IRC | 23:01 | |
| *** felipemonteiro has quit IRC | 23:03 | |
| *** danpawlik has joined #openstack-nova | 23:03 | |
| *** mlavalle has quit IRC | 23:03 | |
| *** mlavalle has joined #openstack-nova | 23:03 | |
| *** Guest29400 has quit IRC | 23:04 | |
| *** suresh12 has joined #openstack-nova | 23:07 | |
| *** danpawlik has quit IRC | 23:07 | |
| *** elmaciej has joined #openstack-nova | 23:08 | |
| *** suresh12 has quit IRC | 23:11 | |
| *** r-daneel has quit IRC | 23:12 | |
| *** masuberu has joined #openstack-nova | 23:13 | |
| *** AlexeyAbashkin has joined #openstack-nova | 23:16 | |
| *** masber has quit IRC | 23:17 | |
| *** masber has joined #openstack-nova | 23:18 | |
| *** masuberu has quit IRC | 23:18 | |
| *** AlexeyAbashkin has quit IRC | 23:20 | |
| *** fragatina has quit IRC | 23:22 | |
| *** fragatina has joined #openstack-nova | 23:23 | |
| *** archit has joined #openstack-nova | 23:24 | |
| *** Swami has quit IRC | 23:26 | |
| *** masber has quit IRC | 23:29 | |
| *** masber has joined #openstack-nova | 23:29 | |
| *** chyka has quit IRC | 23:32 | |
| *** danpawlik has joined #openstack-nova | 23:33 | |
| *** danpawlik has quit IRC | 23:38 | |
| *** mlavalle has quit IRC | 23:38 | |
| *** _ix has quit IRC | 23:44 | |
| Spazmotic | Morning folks | 23:45 |
| *** calebb has quit IRC | 23:50 | |
| *** calebb has joined #openstack-nova | 23:52 | |
| *** danpawlik has joined #openstack-nova | 23:54 | |
| *** esberglu has joined #openstack-nova | 23:54 | |
| *** liverpooler has joined #openstack-nova | 23:55 | |
| *** xinliang has quit IRC | 23:58 | |
| *** esberglu has quit IRC | 23:59 | |
| *** danpawlik has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!