| *** thorst has joined #openstack-nova | 00:04 | |
| *** mingyu has joined #openstack-nova | 00:05 | |
| *** mingyu has quit IRC | 00:10 | |
| *** rajathagasthya has quit IRC | 00:13 | |
| *** yangyapeng has quit IRC | 00:16 | |
| *** yassine has joined #openstack-nova | 00:19 | |
| *** thingee_ has quit IRC | 00:19 | |
| *** suresh12 has joined #openstack-nova | 00:20 | |
| *** itlinux has joined #openstack-nova | 00:20 | |
| *** kornicameister has quit IRC | 00:25 | |
| *** kornicameister has joined #openstack-nova | 00:26 | |
| *** edmondsw has joined #openstack-nova | 00:30 | |
| masber | good morning nova team, I am not sure whether this is the right place to ask this question (please forgive me if not) I am testing openstack vs raw performance in terms of disk latency using ioping and realized that disk latency from the host is around 159us whole on the vm is around 388us. Could someone please help me to understand why is that? thank you very much | 00:30 |
|---|---|---|
| masber | by the way I am using openstack all in one flavor using packstack so instance storage is local lvm | 00:31 |
| *** thorst has quit IRC | 00:31 | |
| *** mamandle has joined #openstack-nova | 00:34 | |
| *** edmondsw has quit IRC | 00:34 | |
| *** Nel1x has joined #openstack-nova | 00:35 | |
| *** yamamoto_ has quit IRC | 00:37 | |
| *** yamamoto has joined #openstack-nova | 00:39 | |
| *** kiennt has joined #openstack-nova | 00:39 | |
| *** kornicameister has quit IRC | 00:40 | |
| *** Sukhdev has quit IRC | 00:41 | |
| *** tyrefors has joined #openstack-nova | 00:42 | |
| *** Apoorva_ has quit IRC | 00:43 | |
| *** sbezverk has quit IRC | 00:44 | |
| *** kornicameister has joined #openstack-nova | 00:45 | |
| *** xinliang has joined #openstack-nova | 00:45 | |
| *** xinliang has quit IRC | 00:45 | |
| *** xinliang has joined #openstack-nova | 00:45 | |
| *** markvoelker has joined #openstack-nova | 00:45 | |
| *** mdnadeem has joined #openstack-nova | 00:46 | |
| openstackgerrit | Merged openstack/nova master: Add release note for shared storage known issue https://review.openstack.org/491582 | 00:48 |
| *** kristian__ has joined #openstack-nova | 00:49 | |
| openstackgerrit | Merged openstack/nova master: add cli to support matrix https://review.openstack.org/481498 | 00:51 |
| *** kristian__ has quit IRC | 00:54 | |
| *** eharney has joined #openstack-nova | 00:59 | |
| *** tetsuro has joined #openstack-nova | 00:59 | |
| *** mingyu has joined #openstack-nova | 01:01 | |
| *** suresh12 has quit IRC | 01:02 | |
| *** jichen has joined #openstack-nova | 01:04 | |
| *** phuongnh has joined #openstack-nova | 01:07 | |
| *** dave-mccowan has joined #openstack-nova | 01:07 | |
| *** Shunli has joined #openstack-nova | 01:08 | |
| *** thorst has joined #openstack-nova | 01:08 | |
| *** thorst has quit IRC | 01:08 | |
| *** thorst has joined #openstack-nova | 01:09 | |
| alex_xu | git show ls | 01:12 |
| alex_xu | oops | 01:12 |
| edleafe | alex_xu: good thing I've never typed in the wrong window! | 01:13 |
| alex_xu | edleafe: :) | 01:13 |
| *** thorst has quit IRC | 01:13 | |
| *** gbarros has joined #openstack-nova | 01:14 | |
| alex_xu | that is fault of multiple screens | 01:14 |
| *** yangyapeng has joined #openstack-nova | 01:17 | |
| *** tuanluong has joined #openstack-nova | 01:19 | |
| *** crushil has joined #openstack-nova | 01:23 | |
| *** mamandle has quit IRC | 01:24 | |
| *** thorst has joined #openstack-nova | 01:25 | |
| *** thorst has quit IRC | 01:27 | |
| *** litao__ has joined #openstack-nova | 01:29 | |
| *** kornicameister has quit IRC | 01:30 | |
| *** kornicameister has joined #openstack-nova | 01:35 | |
| *** bkopilov has quit IRC | 01:43 | |
| *** gongysh has joined #openstack-nova | 01:46 | |
| openstackgerrit | huangtianhua proposed openstack/python-novaclient master: Allow boot server with multiple nics https://review.openstack.org/492003 | 01:58 |
| *** crushil has quit IRC | 01:59 | |
| *** diga has quit IRC | 02:18 | |
| *** chyka has joined #openstack-nova | 02:19 | |
| *** yuchenghui has quit IRC | 02:20 | |
| *** chyka has quit IRC | 02:23 | |
| *** Sukhdev_ has quit IRC | 02:23 | |
| *** yamahata has quit IRC | 02:25 | |
| *** eharney has quit IRC | 02:26 | |
| *** dave-mccowan has quit IRC | 02:27 | |
| *** rajathagasthya has joined #openstack-nova | 02:28 | |
| *** thorst has joined #openstack-nova | 02:28 | |
| *** hongbin has joined #openstack-nova | 02:28 | |
| *** thorst has quit IRC | 02:29 | |
| *** hemna_ has quit IRC | 02:29 | |
| *** kornicameister has quit IRC | 02:29 | |
| *** kornicameister has joined #openstack-nova | 02:35 | |
| *** esberglu has quit IRC | 02:40 | |
| *** rajathagasthya has quit IRC | 02:44 | |
| *** kbaegis has quit IRC | 02:45 | |
| *** rajathagasthya has joined #openstack-nova | 02:47 | |
| *** hongbin has quit IRC | 02:48 | |
| *** hongbin has joined #openstack-nova | 02:49 | |
| *** rajathagasthya has quit IRC | 02:49 | |
| *** dklyle has quit IRC | 02:49 | |
| *** zhurong has joined #openstack-nova | 02:52 | |
| *** hongbin has quit IRC | 02:52 | |
| *** hongbin has joined #openstack-nova | 02:53 | |
| *** Nel1x has quit IRC | 02:53 | |
| *** dave-mccowan has joined #openstack-nova | 02:54 | |
| *** yamahata has joined #openstack-nova | 02:54 | |
| *** kbaegis has joined #openstack-nova | 02:55 | |
| *** rajathagasthya has joined #openstack-nova | 02:58 | |
| *** markvoelker has quit IRC | 03:02 | |
| *** markvoelker has joined #openstack-nova | 03:03 | |
| *** bkopilov has joined #openstack-nova | 03:04 | |
| *** mamandle has joined #openstack-nova | 03:08 | |
| *** mamandle has quit IRC | 03:19 | |
| *** itlinux has quit IRC | 03:20 | |
| *** david-lyle has joined #openstack-nova | 03:22 | |
| *** vks1 has joined #openstack-nova | 03:25 | |
| *** Shunli has quit IRC | 03:25 | |
| *** kbaegis1 has joined #openstack-nova | 03:26 | |
| *** nicolasbock has joined #openstack-nova | 03:29 | |
| *** kbaegis has quit IRC | 03:29 | |
| *** suresh12 has joined #openstack-nova | 03:32 | |
| *** yamamoto has quit IRC | 03:36 | |
| *** suresh12 has quit IRC | 03:37 | |
| *** gbarros has quit IRC | 03:37 | |
| *** udesale has joined #openstack-nova | 03:40 | |
| *** hongbin has quit IRC | 03:43 | |
| *** rajathagasthya has quit IRC | 03:47 | |
| *** yamamoto has joined #openstack-nova | 03:47 | |
| *** dave-mccowan has quit IRC | 03:48 | |
| *** yamamoto has quit IRC | 03:48 | |
| *** yamamoto has joined #openstack-nova | 03:49 | |
| openstackgerrit | Merged openstack/nova master: Test resize with too big flavor https://review.openstack.org/490814 | 03:49 |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Live migration failed in xapi pool https://review.openstack.org/489451 | 03:49 |
| *** kornicameister has quit IRC | 03:53 | |
| *** kornicameister has joined #openstack-nova | 03:54 | |
| *** diga has joined #openstack-nova | 03:56 | |
| *** markvoelker has quit IRC | 04:01 | |
| *** markvoelker has joined #openstack-nova | 04:01 | |
| *** mamandle has joined #openstack-nova | 04:01 | |
| *** thorst has joined #openstack-nova | 04:04 | |
| *** rmcall has quit IRC | 04:07 | |
| *** suresh12 has joined #openstack-nova | 04:09 | |
| *** thorst has quit IRC | 04:09 | |
| *** sridharg has joined #openstack-nova | 04:11 | |
| *** suresh12 has quit IRC | 04:14 | |
| *** moshele has joined #openstack-nova | 04:18 | |
| *** Sukhdev has joined #openstack-nova | 04:20 | |
| *** thorst has joined #openstack-nova | 04:26 | |
| *** moshele has quit IRC | 04:28 | |
| *** gongysh has quit IRC | 04:30 | |
| *** thorst has quit IRC | 04:30 | |
| *** gouthamr has joined #openstack-nova | 04:31 | |
| *** shan has joined #openstack-nova | 04:33 | |
| *** trinaths has joined #openstack-nova | 04:35 | |
| *** sbezverk has joined #openstack-nova | 04:35 | |
| *** adisky__ has joined #openstack-nova | 04:37 | |
| *** hareesh has joined #openstack-nova | 04:38 | |
| *** sbezverk has quit IRC | 04:40 | |
| *** rmcall has joined #openstack-nova | 04:40 | |
| *** claudiub has joined #openstack-nova | 04:40 | |
| *** ekuris has joined #openstack-nova | 04:40 | |
| *** suresh12 has joined #openstack-nova | 04:42 | |
| *** ratailor has joined #openstack-nova | 04:43 | |
| *** david-lyle has quit IRC | 04:44 | |
| *** rmcall has quit IRC | 04:45 | |
| *** dklyle has joined #openstack-nova | 04:45 | |
| *** gyee has joined #openstack-nova | 04:48 | |
| *** mingyu has quit IRC | 04:49 | |
| *** mingyu has joined #openstack-nova | 04:50 | |
| *** rajathagasthya has joined #openstack-nova | 04:50 | |
| *** cfriesen has quit IRC | 04:50 | |
| *** mingyu has quit IRC | 04:50 | |
| *** rajathagasthya has quit IRC | 04:51 | |
| *** rajathagasthya has joined #openstack-nova | 04:51 | |
| *** suresh12 has quit IRC | 04:53 | |
| openstackgerrit | Spencer Yu proposed openstack/python-novaclient master: Nova client should retry with Retry-After value https://review.openstack.org/447766 | 04:56 |
| *** gongysh has joined #openstack-nova | 05:09 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Live migration failed in xapi pool https://review.openstack.org/489451 | 05:10 |
| jianghuaw | mriedem, thanks for the comments for https://bugs.launchpad.net/nova/+bug/1709594. | 05:11 |
| openstack | Launchpad bug 1709594 in OpenStack Compute (nova) "live-migration without '--block-migrate" failed with "No sql_connection parameter is established"" [Undecided,New] | 05:11 |
| jianghuaw | Yes, this is a devstack configuration and it's running in superconductor mode. | 05:11 |
| *** sree has joined #openstack-nova | 05:11 | |
| jianghuaw | the database configuration in nova.conf is: | 05:12 |
| jianghuaw | [database] | 05:12 |
| jianghuaw | connection = mysql+pymysql://root:citrix@127.0.0.1/nova_cell0?charset=utf8 | 05:12 |
| jianghuaw | [api_database] | 05:12 |
| jianghuaw | connection = mysql+pymysql://root:citrix@127.0.0.1/nova_api?charset=utf8 | 05:12 |
| jianghuaw | mriedem, the nova_api database connection settings seems correct. | 05:14 |
| *** gouthamr has quit IRC | 05:16 | |
| *** moshele has joined #openstack-nova | 05:16 | |
| *** gmann has joined #openstack-nova | 05:17 | |
| *** litao__ has quit IRC | 05:19 | |
| *** gjayavelu has joined #openstack-nova | 05:20 | |
| *** suresh12 has joined #openstack-nova | 05:21 | |
| *** mamandle has quit IRC | 05:22 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it https://review.openstack.org/480379 | 05:24 |
| *** Sukhdev has quit IRC | 05:30 | |
| *** mamandle has joined #openstack-nova | 05:30 | |
| *** kiennt has quit IRC | 05:34 | |
| *** jamesdenton has quit IRC | 05:36 | |
| *** abalutoiu has quit IRC | 05:39 | |
| *** mingyu has joined #openstack-nova | 05:46 | |
| *** suresh12 has quit IRC | 05:47 | |
| *** armax has joined #openstack-nova | 05:48 | |
| *** abalutoiu has joined #openstack-nova | 05:48 | |
| *** armax has quit IRC | 05:48 | |
| *** mvk has joined #openstack-nova | 05:49 | |
| *** mingyu has quit IRC | 05:49 | |
| *** psachin has joined #openstack-nova | 05:52 | |
| *** kristian__ has joined #openstack-nova | 05:52 | |
| *** suresh12 has joined #openstack-nova | 05:53 | |
| *** kristian__ has quit IRC | 05:56 | |
| *** jamesdenton has joined #openstack-nova | 05:57 | |
| *** kiennt has joined #openstack-nova | 06:00 | |
| *** mamandle_ has joined #openstack-nova | 06:01 | |
| *** yamamoto has quit IRC | 06:01 | |
| *** mamandle has quit IRC | 06:01 | |
| *** mamandle_ is now known as mamandle | 06:01 | |
| *** slaweq has quit IRC | 06:05 | |
| *** rcernin has joined #openstack-nova | 06:05 | |
| *** lpetrut has joined #openstack-nova | 06:05 | |
| *** yamamoto has joined #openstack-nova | 06:05 | |
| *** mamandle has quit IRC | 06:09 | |
| *** gmann has quit IRC | 06:11 | |
| *** Oku_OS-away is now known as Oku_OS | 06:12 | |
| *** junbo has quit IRC | 06:12 | |
| *** gmann has joined #openstack-nova | 06:12 | |
| *** junbo has joined #openstack-nova | 06:15 | |
| *** jamesdenton has quit IRC | 06:16 | |
| *** suresh12 has quit IRC | 06:20 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it https://review.openstack.org/480379 | 06:22 |
| *** thorst has joined #openstack-nova | 06:26 | |
| *** slaweq has joined #openstack-nova | 06:29 | |
| *** thorst has quit IRC | 06:31 | |
| *** moshele has quit IRC | 06:33 | |
| *** jamesdenton has joined #openstack-nova | 06:34 | |
| *** kristian__ has joined #openstack-nova | 06:34 | |
| *** rajathagasthya has quit IRC | 06:36 | |
| *** yuchenghui has joined #openstack-nova | 06:37 | |
| *** itlinux has joined #openstack-nova | 06:38 | |
| *** jamesdenton has quit IRC | 06:39 | |
| *** sridharg has quit IRC | 06:39 | |
| *** kukacz has joined #openstack-nova | 06:43 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it https://review.openstack.org/480379 | 06:43 |
| openstackgerrit | Alex Xu proposed openstack/nova master: placement: avoid to return duplicated alloc_reqs when no sharing rp https://review.openstack.org/492395 | 06:43 |
| *** yuchenghui has quit IRC | 06:43 | |
| *** kukacz_ has joined #openstack-nova | 06:44 | |
| *** gjayavelu has quit IRC | 06:45 | |
| *** sridharg has joined #openstack-nova | 06:46 | |
| *** kukacz has quit IRC | 06:47 | |
| *** jamesdenton has joined #openstack-nova | 06:48 | |
| *** itlinux has quit IRC | 06:54 | |
| *** sshwarts has joined #openstack-nova | 06:56 | |
| *** lpetrut has quit IRC | 06:57 | |
| openstackgerrit | Merged openstack/nova master: [placement] Add api-ref for usages https://review.openstack.org/480563 | 06:58 |
| *** abalutoiu has quit IRC | 06:59 | |
| *** sridharg has quit IRC | 06:59 | |
| *** sridharg has joined #openstack-nova | 07:00 | |
| openstackgerrit | Merged openstack/nova master: Raise NoValidHost if no allocation candidates https://review.openstack.org/491491 | 07:03 |
| openstackgerrit | Alex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it https://review.openstack.org/480379 | 07:04 |
| *** sridharg has quit IRC | 07:07 | |
| *** gjayavelu has joined #openstack-nova | 07:09 | |
| *** sridharg has joined #openstack-nova | 07:09 | |
| *** kukacz_ has quit IRC | 07:10 | |
| *** kukacz has joined #openstack-nova | 07:10 | |
| *** psachin has quit IRC | 07:12 | |
| *** gcb has joined #openstack-nova | 07:13 | |
| *** tesseract has joined #openstack-nova | 07:13 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it https://review.openstack.org/480379 | 07:14 |
| *** sahid has joined #openstack-nova | 07:14 | |
| alex_xu | emm...finally | 07:14 |
| *** sridharg has quit IRC | 07:17 | |
| *** sridharg has joined #openstack-nova | 07:19 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Enhance support matrix document https://review.openstack.org/482013 | 07:20 |
| *** markmc has quit IRC | 07:21 | |
| *** markus_z has joined #openstack-nova | 07:23 | |
| *** aarefiev_afk is now known as aarefiev | 07:29 | |
| *** mdnadeem has quit IRC | 07:42 | |
| *** xinliang has quit IRC | 07:48 | |
| *** ralonsoh has joined #openstack-nova | 07:48 | |
| *** alexchadin has joined #openstack-nova | 07:48 | |
| *** gjayavelu has quit IRC | 07:57 | |
| *** xinliang has joined #openstack-nova | 08:00 | |
| *** mdnadeem has joined #openstack-nova | 08:01 | |
| *** markus_z has quit IRC | 08:03 | |
| *** mdnadeem has quit IRC | 08:03 | |
| *** mdnadeem has joined #openstack-nova | 08:03 | |
| *** markus_z has joined #openstack-nova | 08:04 | |
| *** gjayavelu has joined #openstack-nova | 08:09 | |
| *** jpena|off has quit IRC | 08:12 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: replace chance with filter scheduler in func tests https://review.openstack.org/491529 | 08:13 |
| *** mriedem has quit IRC | 08:16 | |
| *** moshele has joined #openstack-nova | 08:17 | |
| *** Drankis has joined #openstack-nova | 08:17 | |
| *** abalutoiu has joined #openstack-nova | 08:18 | |
| gibi | good morning | 08:19 |
| maciejjozefczyk | hello | 08:20 |
| *** suresh12 has joined #openstack-nova | 08:21 | |
| *** xinliang has quit IRC | 08:23 | |
| *** gjayavelu has quit IRC | 08:24 | |
| *** lucas-afk is now known as lucasagomes | 08:25 | |
| *** derekh has joined #openstack-nova | 08:25 | |
| *** suresh12 has quit IRC | 08:25 | |
| *** thorst has joined #openstack-nova | 08:27 | |
| *** yamamoto has quit IRC | 08:27 | |
| *** yamamoto has joined #openstack-nova | 08:28 | |
| *** psachin has joined #openstack-nova | 08:29 | |
| *** yamamoto has quit IRC | 08:31 | |
| *** yamamoto has joined #openstack-nova | 08:32 | |
| *** hoonetorg has quit IRC | 08:32 | |
| *** thorst has quit IRC | 08:32 | |
| *** gjayavelu has joined #openstack-nova | 08:33 | |
| *** egonzalez has joined #openstack-nova | 08:36 | |
| *** jaosorior has quit IRC | 08:37 | |
| *** ekuris has quit IRC | 08:37 | |
| *** moshele has quit IRC | 08:37 | |
| *** avolkov has joined #openstack-nova | 08:38 | |
| *** tetsuro has quit IRC | 08:38 | |
| *** xinliang has joined #openstack-nova | 08:39 | |
| *** xinliang has quit IRC | 08:39 | |
| *** xinliang has joined #openstack-nova | 08:39 | |
| bauzas | morning Novaists | 08:40 |
| *** jaosorior has joined #openstack-nova | 08:41 | |
| *** ociuhandu has quit IRC | 08:41 | |
| *** tosika has quit IRC | 08:46 | |
| *** hoonetorg has joined #openstack-nova | 08:49 | |
| *** tosika has joined #openstack-nova | 08:52 | |
| *** xinliang has quit IRC | 08:52 | |
| *** jaosorior has quit IRC | 08:54 | |
| *** ekuris has joined #openstack-nova | 08:54 | |
| ratailor | alex_xu, you around ? | 09:05 |
| alex_xu | ratailor: yea | 09:05 |
| ratailor | alex_xu, regarding this bug https://bugs.launchpad.net/nova/+bug/1709260 | 09:06 |
| openstack | Launchpad bug 1709260 in OpenStack Compute (nova) "Addition of host to host-aggregate should be case -sensitive" [Low,Confirmed] - Assigned to Rajesh Tailor (ratailor) | 09:06 |
| *** ralonsoh has quit IRC | 09:06 | |
| *** xinliang has joined #openstack-nova | 09:06 | |
| ratailor | alex_xu, I have done something like this.. https://github.com/openstack/nova/commit/402b3abf990d08d2af8331079d36a92d84d84b80#diff-342854d9b83486a3db6189bc21be6266L1026 | 09:06 |
| *** ralonsoh has joined #openstack-nova | 09:07 | |
| ratailor | alex_xu, for host_mappings table in api_db. | 09:07 |
| ratailor | alex_xu, I was able to pass through this code block. https://github.com/openstack/nova/blob/master/nova/objects/host_mapping.py#L91-L93 | 09:08 |
| alex_xu | ratailor: not sure your first link is correct, that is pointed to a empty migration check | 09:09 |
| alex_xu | ratailor: oh, sorry, I saw the entire patch now | 09:09 |
| alex_xu | ratailor: I'm not sure we need to convert the table to case-sensitive | 09:10 |
| ratailor | alex_xu, but again I got stuck in this code block. https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L571-L575 | 09:10 |
| alex_xu | at least, the hostname should be case-insensitive | 09:10 |
| *** tetsuro has joined #openstack-nova | 09:11 | |
| ratailor | alex_xu, in that case, do I need to look for alternative approach if possible ? | 09:11 |
| alex_xu | ratailor: I guess it won't work after you change the collation of the table? since the in-memory copy of host<->aggregation mapping in the scheduler is still case-sensitive | 09:14 |
| *** yuchenghui has joined #openstack-nova | 09:15 | |
| alex_xu | ratailor: ah, I see, you want to stop the api user input a hostname which differnet with the Db one | 09:16 |
| ratailor | alex_xu, I also applied the collation logic to service table in nova_cell1 db, but here, case-insensitive check is getting passed. | 09:16 |
| alex_xu | but the hostname should be case-insensitive? whatever the user input COMPUTE0.example.com or compute0.example.com, it should works | 09:16 |
| *** gjayavelu has quit IRC | 09:16 | |
| ratailor | alex_xu, yes, I wan't to prevent from adding the case-insensitive host to host-aggregate. | 09:17 |
| ratailor | but aggregates store the hostname as COMPUTE0.example.com, whereas in db its compute0.example.com | 09:17 |
| ratailor | alex_xu, so that is what might be confusing scheduler. | 09:18 |
| *** gongysh has quit IRC | 09:19 | |
| alex_xu | ratailor: there is a related bug https://bugs.launchpad.net/nova/+bug/1538011 | 09:20 |
| openstack | Launchpad bug 1538011 in OpenStack Compute (nova) "Metadata keys are not case sensitive" [Medium,In progress] - Assigned to Augustina Ragwitz (auggy) | 09:20 |
| *** yuchenghui has quit IRC | 09:20 | |
| ratailor | alex_xu, Thanks! let me check it. | 09:20 |
| alex_xu | ratailor: it would be great to check those two bugs, see the best way to fix | 09:21 |
| ratailor | alex_xu, sure. | 09:21 |
| *** abhishekk has joined #openstack-nova | 09:22 | |
| alex_xu | ratailor: and there is a spec for that bug https://review.openstack.org/#/c/350843/, but didn't get merged | 09:22 |
| *** gmann has quit IRC | 09:28 | |
| *** xinliang has quit IRC | 09:31 | |
| *** sree has quit IRC | 09:34 | |
| *** sree has joined #openstack-nova | 09:34 | |
| *** cdent has joined #openstack-nova | 09:35 | |
| *** jichen has quit IRC | 09:35 | |
| *** sambetts|afk is now known as sambetts | 09:36 | |
| openstackgerrit | Vladyslav Drok proposed openstack/nova master: [placement] Add api-ref for RP usages https://review.openstack.org/450105 | 09:37 |
| *** gongysh has joined #openstack-nova | 09:37 | |
| *** kukacz has quit IRC | 09:39 | |
| *** sree has quit IRC | 09:39 | |
| *** sree has joined #openstack-nova | 09:40 | |
| *** markvoelker has quit IRC | 09:42 | |
| *** xinliang has joined #openstack-nova | 09:44 | |
| *** kukacz has joined #openstack-nova | 09:47 | |
| *** sdague has joined #openstack-nova | 09:50 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Add datapath type information to OVS vif objects https://review.openstack.org/474892 | 09:51 |
| *** jaosorior has joined #openstack-nova | 09:54 | |
| openstackgerrit | Vladyslav Drok proposed openstack/nova master: [placement] Add api-ref for allocation_candidates https://review.openstack.org/481112 | 09:57 |
| openstackgerrit | Vladyslav Drok proposed openstack/nova master: [placement] Make placement_api_docs.py failing https://review.openstack.org/480924 | 09:57 |
| alex_xu | cdent: bauzas this is ready for review https://review.openstack.org/#/q/topic:bug/1702420 | 10:01 |
| cdent | alex_xu: thanks alex, I’ve got that in my queue to look at soon. Was it hard to resolve? | 10:01 |
| bauzas | alex_xu: thanks, will look to it asap | 10:02 |
| *** mvk_ has joined #openstack-nova | 10:02 | |
| bauzas | just have some internal paperwork to do | 10:02 |
| alex_xu | cdent: after I pull some code into a sub-method, then thing gets a little better | 10:02 |
| cdent | ah, yeah, good idea | 10:02 |
| *** gcb has quit IRC | 10:04 | |
| maciejjozefczyk | cdent: please add also this to your queue https://review.openstack.org/#/c/491808/ | 10:04 |
| cdent | maciejjozefczyk: already there | 10:04 |
| maciejjozefczyk | cdent: :) thx | 10:04 |
| *** mvk has quit IRC | 10:05 | |
| *** kiennt has quit IRC | 10:07 | |
| *** phuongnh has quit IRC | 10:10 | |
| *** kukacz_ has joined #openstack-nova | 10:11 | |
| *** kukacz has quit IRC | 10:12 | |
| *** kukacz_ has quit IRC | 10:13 | |
| *** mdnadeem has quit IRC | 10:16 | |
| *** bkopilov has quit IRC | 10:19 | |
| *** yamamoto has quit IRC | 10:21 | |
| *** kukacz has joined #openstack-nova | 10:22 | |
| *** lpetrut has joined #openstack-nova | 10:22 | |
| *** kukacz has quit IRC | 10:22 | |
| *** yamamoto has joined #openstack-nova | 10:25 | |
| *** kukacz has joined #openstack-nova | 10:26 | |
| *** kukacz has quit IRC | 10:26 | |
| *** moshele has joined #openstack-nova | 10:26 | |
| *** yamamoto has quit IRC | 10:28 | |
| *** yamamoto has joined #openstack-nova | 10:28 | |
| *** thorst has joined #openstack-nova | 10:28 | |
| *** ekuris has quit IRC | 10:29 | |
| *** mdnadeem has joined #openstack-nova | 10:30 | |
| *** thorst has quit IRC | 10:33 | |
| *** yangyapeng has quit IRC | 10:33 | |
| *** dtantsur|afk is now known as dtantsur | 10:35 | |
| *** sree has quit IRC | 10:35 | |
| *** sree has joined #openstack-nova | 10:35 | |
| *** udesale has quit IRC | 10:36 | |
| *** kristian__ has quit IRC | 10:37 | |
| *** kristian__ has joined #openstack-nova | 10:38 | |
| *** zhurong has quit IRC | 10:39 | |
| *** tetsuro has quit IRC | 10:39 | |
| *** sree has quit IRC | 10:40 | |
| *** yamamoto has quit IRC | 10:41 | |
| openstackgerrit | Sean Dague proposed openstack/nova master: Clarify that vlan feature means nova-network support https://review.openstack.org/478551 | 10:42 |
| *** ekuris has joined #openstack-nova | 10:44 | |
| *** yamahata has quit IRC | 10:48 | |
| openstackgerrit | Merged openstack/nova master: Remove ram/disk sched filters from default list https://review.openstack.org/491854 | 10:54 |
| *** shan has quit IRC | 10:54 | |
| openstackgerrit | Merged openstack/nova master: Mark Chance and Caching schedulers as deprecated https://review.openstack.org/492210 | 10:57 |
| *** gongysh has quit IRC | 11:01 | |
| *** mkrcmari__ has joined #openstack-nova | 11:01 | |
| *** psachin has quit IRC | 11:03 | |
| *** mvk_ has quit IRC | 11:04 | |
| *** mdnadeem has quit IRC | 11:06 | |
| *** yamamoto has joined #openstack-nova | 11:09 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: replace chance with filter scheduler in func tests https://review.openstack.org/491529 | 11:10 |
| *** mkucia has joined #openstack-nova | 11:11 | |
| *** mdnadeem has joined #openstack-nova | 11:12 | |
| openstackgerrit | Merged openstack/python-novaclient master: Remove substitutions for command error msg https://review.openstack.org/490705 | 11:12 |
| *** mdnadeem has quit IRC | 11:19 | |
| *** yamamoto_ has joined #openstack-nova | 11:20 | |
| *** diga has quit IRC | 11:22 | |
| *** yamamoto has quit IRC | 11:24 | |
| *** yangyapeng has joined #openstack-nova | 11:25 | |
| openstackgerrit | Merged openstack/nova master: [placement] Avoid error log on 405 response https://review.openstack.org/490021 | 11:26 |
| openstackgerrit | Vladyslav Drok proposed openstack/nova master: [placement] Add api-ref for allocation_candidates https://review.openstack.org/481112 | 11:26 |
| openstackgerrit | Vladyslav Drok proposed openstack/nova master: [placement] Make placement_api_docs.py failing https://review.openstack.org/480924 | 11:27 |
| openstackgerrit | Merged openstack/nova master: api-ref: fix security_groups response parameter in os-security-groups https://review.openstack.org/489274 | 11:27 |
| *** mkrcmari__ has quit IRC | 11:28 | |
| openstackgerrit | Merged openstack/nova master: api-ref: requested security groups are not applied to pre-existing ports https://review.openstack.org/489275 | 11:28 |
| *** mkrcmari__ has joined #openstack-nova | 11:29 | |
| *** kbaegis1 has quit IRC | 11:34 | |
| *** kbaegis has joined #openstack-nova | 11:34 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | 11:34 |
| openstackgerrit | Merged openstack/nova master: Remove translation of log messages https://review.openstack.org/466637 | 11:39 |
| maciejjozefczyk | sdague: Could you check this one? https://review.openstack.org/#/c/491808/ please? | 11:39 |
| openstackgerrit | Merged openstack/nova master: remove mox from unit/virt/vmwareapi/test_driver_api.py https://review.openstack.org/452128 | 11:39 |
| sdague | maciejjozefczyk: how does this handle the synchronization problem that now computes might be trying to delete the same instances at the same time? | 11:41 |
| sdague | previously, by being host scoped, this was a sharded problem | 11:41 |
| *** lucasagomes is now known as lucas-hungry | 11:42 | |
| *** dave-mccowan has joined #openstack-nova | 11:43 | |
| *** alexchadin has quit IRC | 11:43 | |
| *** alexchadin has joined #openstack-nova | 11:44 | |
| *** alexchadin has quit IRC | 11:44 | |
| *** mkucia has quit IRC | 11:45 | |
| *** alexchadin has joined #openstack-nova | 11:45 | |
| *** alexchadin has quit IRC | 11:45 | |
| *** alexchadin has joined #openstack-nova | 11:45 | |
| *** alexchadin has quit IRC | 11:46 | |
| openstackgerrit | Merged openstack/nova master: imagebackend: cleanup constructor args to Rbd https://review.openstack.org/490499 | 11:46 |
| *** alexchadin has joined #openstack-nova | 11:46 | |
| openstackgerrit | Merged openstack/nova master: Add policy granularity to the Flavors API https://review.openstack.org/449288 | 11:47 |
| *** mtanino has joined #openstack-nova | 11:47 | |
| sdague | ah, I see now | 11:49 |
| *** kbaegis has quit IRC | 11:49 | |
| *** smatzek has joined #openstack-nova | 11:49 | |
| *** kbaegis has joined #openstack-nova | 11:49 | |
| maciejjozefczyk | sdague: The problem is about instance (which is deleted from nova side) is still running on compute A, but nova says that its deleted and it was on host B | 11:49 |
| sdague | maciejjozefczyk: yeh, I get the problem | 11:50 |
| sdague | I was just trying to make sure that this didn't make it so that multiple computes were trying to delete the same instance | 11:50 |
| sdague | but _get_instances_on_driver should keep us sharded | 11:50 |
| maciejjozefczyk | yes | 11:51 |
| *** alexchadin has quit IRC | 11:51 | |
| maciejjozefczyk | I'm going to work on patch to rollback migration if deletion of instance will be triggered, in near future | 11:52 |
| sdague | cool | 11:52 |
| maciejjozefczyk | but this fix for already 'lost' and 'working' zombiee instances i think should be in nova | 11:52 |
| maciejjozefczyk | in my installation I have hundreds of them | 11:53 |
| *** kbaegis has quit IRC | 11:53 | |
| *** kbaegis has joined #openstack-nova | 11:54 | |
| *** trinaths has quit IRC | 11:56 | |
| *** esberglu has joined #openstack-nova | 11:59 | |
| sdague | maciejjozefczyk: yep, +2 on this fix | 11:59 |
| *** tbachman has quit IRC | 11:59 | |
| *** _pewp_ has quit IRC | 12:00 | |
| maciejjozefczyk | sdague: thx | 12:00 |
| *** thorst has joined #openstack-nova | 12:02 | |
| *** esberglu has quit IRC | 12:04 | |
| *** abhishekk has quit IRC | 12:04 | |
| *** alexchadin has joined #openstack-nova | 12:04 | |
| *** tuanluong has quit IRC | 12:06 | |
| *** kbaegis has quit IRC | 12:06 | |
| *** kbaegis has joined #openstack-nova | 12:06 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: placement: avoid returning duplicated alloc_reqs when no sharing rp https://review.openstack.org/492395 | 12:06 |
| openstackgerrit | Alex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it https://review.openstack.org/480379 | 12:06 |
| alex_xu | cdent: thanks | 12:06 |
| openstackgerrit | Ilya Popov proposed openstack/nova master: Tests: Add cleanup of 'instances' directory https://review.openstack.org/491589 | 12:06 |
| sdague | alex_xu: can I tempt you with doc patches? :) | 12:08 |
| *** _pewp_ has joined #openstack-nova | 12:08 | |
| sdague | mostly I'd like to get the manuals stuff merged before I go on vacation next week | 12:08 |
| *** alexchadin has quit IRC | 12:09 | |
| *** sree has joined #openstack-nova | 12:09 | |
| sdague | https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/doc-migration | 12:09 |
| *** vks1 has quit IRC | 12:09 | |
| alex_xu | sdague: yea, let me try | 12:10 |
| *** jpena has joined #openstack-nova | 12:10 | |
| *** kbaegis1 has joined #openstack-nova | 12:12 | |
| *** kbaegis has quit IRC | 12:12 | |
| sdague | alex_xu: thank you | 12:13 |
| *** sree has quit IRC | 12:13 | |
| *** kbaegis1 has quit IRC | 12:17 | |
| *** kbaegis has joined #openstack-nova | 12:17 | |
| *** xinliang has quit IRC | 12:20 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it https://review.openstack.org/480379 | 12:20 |
| *** MVenesio has joined #openstack-nova | 12:20 | |
| *** sree has joined #openstack-nova | 12:20 | |
| *** mtanino has quit IRC | 12:25 | |
| *** edmondsw has joined #openstack-nova | 12:27 | |
| *** slaweq has quit IRC | 12:31 | |
| *** slaweq has joined #openstack-nova | 12:32 | |
| *** lucas-hungry is now known as lucasagomes | 12:32 | |
| *** xinliang has joined #openstack-nova | 12:33 | |
| *** moshele has quit IRC | 12:34 | |
| *** jamesdenton has quit IRC | 12:35 | |
| *** slaweq has quit IRC | 12:36 | |
| *** jamesdenton has joined #openstack-nova | 12:36 | |
| *** karthiks has quit IRC | 12:37 | |
| *** moshele has joined #openstack-nova | 12:40 | |
| *** ratailor has quit IRC | 12:41 | |
| *** gbarros has joined #openstack-nova | 12:42 | |
| *** alexchadin has joined #openstack-nova | 12:44 | |
| *** catintheroof has joined #openstack-nova | 12:45 | |
| *** lyan has joined #openstack-nova | 12:45 | |
| *** mriedem has joined #openstack-nova | 12:45 | |
| *** mkrcmari__ has quit IRC | 12:46 | |
| *** sbezverk has joined #openstack-nova | 12:48 | |
| *** moshele has quit IRC | 12:49 | |
| *** slaweq has joined #openstack-nova | 12:51 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Reconnect volumes and encryptors during a hard reboot https://review.openstack.org/400384 | 12:54 |
| *** mvk has joined #openstack-nova | 12:54 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Detach volumes on _rebuild_default_impl failure https://review.openstack.org/442105 | 12:54 |
| *** markmc has joined #openstack-nova | 12:55 | |
| *** mdnadeem has joined #openstack-nova | 12:57 | |
| *** jpena is now known as jpena|mtg | 12:59 | |
| cdent | mriedem: i was partly thinking in terms of “don’t add more churn to zuul, now” | 12:59 |
| *** esberglu has joined #openstack-nova | 13:00 | |
| *** moshele has joined #openstack-nova | 13:00 | |
| *** pchavva has joined #openstack-nova | 13:01 | |
| mriedem | mmm zuul churn | 13:01 |
| cdent | fresh and tasty | 13:02 |
| mriedem | artom: you love evacuate right? | 13:02 |
| mriedem | gibi: you love finding bugs right? | 13:02 |
| gibi | mriedem: I would put it I like finding them now than getting it from production :) | 13:05 |
| artom | mriedem, in the same way I love, err... | 13:05 |
| artom | Crap, it's too early for witty wordplay | 13:06 |
| artom | mriedem, anyways, what's up? | 13:06 |
| mriedem | my main worry evacuate from an ocata compute messing this up https://review.openstack.org/#/c/491012/ | 13:06 |
| mriedem | artom: i don't know how much you've followed this | 13:06 |
| mriedem | but basically the filter scheduler creates allocations in placement now, | 13:06 |
| mriedem | on both the source and dest computes during a move | 13:06 |
| mriedem | like evacuate | 13:06 |
| *** mingyu has joined #openstack-nova | 13:07 | |
| mriedem | the problem is that the resource tracker has no concept of other providers than itself, so during it's periodic accounting updates, it overwrites allocations in placement for any other provider | 13:07 |
| mriedem | that patch ^ attempts to resolve that | 13:07 |
| mriedem | by using a minimum compute service version check - so once all of the computes are pike, it will stop doing it's local accounting | 13:08 |
| mriedem | and overwriting the stuff the scheduler created | 13:08 |
| mriedem | one of my worries is that we have an ocata compute that is forced-down, which takes it out of the service version check, but could still be running and trampling on things | 13:08 |
| *** lucasxu has joined #openstack-nova | 13:09 | |
| mriedem | i think it's probably a small window because if you are forcing a compute down and evacuating from it, (1) you're likely to stop that host at some point and (2) once the instances move, the dest compute should be accounting for them - and the scheduler will also do that | 13:09 |
| artom | I'm fuzzy on the resource tracker having no concept of other providers than itself | 13:10 |
| artom | I thought compute nodes were resource providers? | 13:11 |
| mriedem | they are | 13:11 |
| openstackgerrit | Merged openstack/nova master: Keep the code consistent https://review.openstack.org/490304 | 13:11 |
| mriedem | cdent: am i correct in saying that https://review.openstack.org/#/c/491012/ only applies to the periodic update_available_resource task? | 13:12 |
| mriedem | looks like that's the only place that _update_usage_from_instances is called from | 13:12 |
| openstackgerrit | Merged openstack/nova master: add description about key_name https://review.openstack.org/489525 | 13:12 |
| * cdent refreshes himself | 13:12 | |
| *** bkopilov has joined #openstack-nova | 13:12 | |
| cdent | mriedem: yes | 13:13 |
| mriedem | just thinking that if that code thinks everything is pike and doesn't auto-heal, | 13:13 |
| mriedem | and it misses the instance moving from the ocata compute, | 13:13 |
| mriedem | then we have to be sure that the rebuild_claim handles it | 13:13 |
| *** moshele has quit IRC | 13:14 | |
| cdent | murgh | 13:14 |
| gibi | mriedem: if a compute host is forced_down it should mean that that compute host is fenced | 13:14 |
| gibi | mriedem: therefore it cannot tramp on allocations | 13:15 |
| *** kylek3h has joined #openstack-nova | 13:15 | |
| *** baoli has joined #openstack-nova | 13:15 | |
| mriedem | gibi: that doesn't mean the nova-compute service is not running on that host | 13:15 |
| mriedem | and if the service is running, it's update_available_resource periodic is running and could be overwriting allocations for the instance that's being evacuated | 13:16 |
| mriedem | the forced_down flag doesn't do anything besides let the evacuate API proceed before the servicegroup api checkin says the compute is down | 13:16 |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack/nova master: docs: Explain the flow of the "serial console" feature https://review.openstack.org/476188 | 13:16 |
| *** tbachman has joined #openstack-nova | 13:17 | |
| mriedem | cdent: so i don't think rebuild_claim will update allocations at all | 13:17 |
| sdague | mriedem: the contract with the user is forced_down means they killed that compute | 13:17 |
| gibi | mriedem: if the compute is stull running but the admin set force-down then it is a user error | 13:17 |
| gibi | admin should fence first then set forced-down flag | 13:17 |
| mriedem | rebuild_call calls _move_call which calls _update_usage_from_migration which calls _update_usage which doesn't call the report client | 13:17 |
| sdague | it is only meant to be used if they've taken that system out of communication | 13:17 |
| sdague | agree with gibi, that's admin error, and we've never attempted to correct for that | 13:18 |
| mriedem | that's not documented anywhere https://developer.openstack.org/api-ref/compute/#update-forced-down | 13:18 |
| sdague | mriedem: ok, well we should document it, that was the whole point of that feature | 13:18 |
| *** ociuhandu has joined #openstack-nova | 13:18 | |
| sdague | for HA systems to override nova when it knew better | 13:19 |
| cdent | mriedem: yeah, it looks like the allocation creation is all happening outside the various _claim* methods | 13:19 |
| mriedem | i had reported a bug related to docs on this at one point https://bugs.launchpad.net/nova/+bug/1691871 | 13:19 |
| openstack | Launchpad bug 1691871 in OpenStack Compute (nova) "forced-down vs service disable is not documented well in the compute API reference" [Medium,Confirmed] | 13:19 |
| gibi | mriedem: it is at least in the original spec https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/mark-host-down.html | 13:19 |
| cdent | which is somewhat weird | 13:19 |
| cdent | but probably good given what we want eventually | 13:20 |
| sdague | mriedem: sure but as you know, there aren't many idle folks looking for bugs to fix. Docs bugs mostly languish in our tracker. | 13:21 |
| mriedem | gibi: ok so the doctor project is supposed to get an alarm that something is wrong with the host, fence it and then force it down and start evacuating? | 13:21 |
| mriedem | sdague: i planned on fixing that docs gap myself | 13:21 |
| mriedem | but $time | 13:21 |
| *** tbachman has quit IRC | 13:21 | |
| gibi | mriedem: I think so, yes | 13:21 |
| sdague | mriedem: sure, that's fine | 13:22 |
| mriedem | what does 'fencing' mean in this case? | 13:22 |
| bauzas | mriedem: do we have a devstack change for https://review.openstack.org/#/c/491854/ ? | 13:22 |
| *** tbachman has joined #openstack-nova | 13:22 | |
| cdent | re: $time [t 1zqO] | 13:22 |
| purplerbot | <cdent> When do we start asking if the concept of PTL, as currently constructed, is sustainable? [2017-08-10 13:21:20.802315] [n 1zqO] | 13:22 |
| mriedem | bauzas: no | 13:22 |
| gibi | mriedem: power off, or cut the network | 13:22 |
| gibi | mriedem: mostly power off via IPMI | 13:22 |
| bauzas | mriedem: okay, I'll write it | 13:22 |
| sdague | what gibi said | 13:22 |
| mriedem | ok powering off would be ideal | 13:22 |
| sdague | mriedem: but, it could be lots of things. They could also decide to network fence the node | 13:23 |
| mriedem | but network is also good so the compute couldn't send changes to the report client (to conductor i mean) | 13:23 |
| mriedem | as long as it can't get to the placement api then that's sufficient | 13:23 |
| cdent | that comment from bauzas reminds me of something I read while reading gerrit messages last night: did some test have to be changed so that it was running one of the filters we have no declared no longer default? | 13:23 |
| gibi | mriedem: I'm not 100% sure doctor also starts the evacuation automatically after force_down | 13:23 |
| mriedem | gibi: ok but some project, maybe watcher | 13:23 |
| gibi | mriedem: sure | 13:23 |
| sdague | mriedem: I can take a spin on the api-ref, if you review my other doc patches :) | 13:23 |
| gibi | mriedem: we have our on internally :) | 13:24 |
| bauzas | what's the problem with evacuation ? | 13:24 |
| mriedem | sdague: i have one change i need to get in today and that's https://review.openstack.org/#/c/491012/ | 13:24 |
| mriedem | bauzas: see https://review.openstack.org/#/c/491012/ | 13:24 |
| bauzas | mriedem: ah, this... | 13:25 |
| *** kbaegis1 has joined #openstack-nova | 13:27 | |
| mriedem | cdent: ok left more comments in there | 13:29 |
| mriedem | i think i'm ok with this now, or as much as i can be, | 13:29 |
| mriedem | cdent: i'll fix the functional test quick and push that up, just that change | 13:29 |
| *** kbaegis has quit IRC | 13:29 | |
| mriedem | and then i'll probably spend some time this afternoon working on a functional test for evacuate | 13:29 |
| *** kfarr has joined #openstack-nova | 13:29 | |
| mriedem | with the service version stuff | 13:30 |
| mriedem | bauzas: devstack sets this | 13:30 |
| mriedem | iniset $NOVA_CONF filter_scheduler enabled_filters "RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,RamFilter,DiskFilter" | 13:30 |
| *** mdnadeem has quit IRC | 13:30 | |
| *** jaypipes has joined #openstack-nova | 13:30 | |
| bauzas | mriedem: I know | 13:30 |
| mriedem | nvm, that's only for the fake hypervisor | 13:30 |
| bauzas | mriedem: I'm just writing atm the change to remove this | 13:30 |
| *** kbaegis has joined #openstack-nova | 13:31 | |
| mriedem | which was only used in the large tests, which we don't run anymore | 13:31 |
| bauzas | we run Core and Ram by default AFAIK | 13:31 |
| *** kbaegis1 has quit IRC | 13:31 | |
| bauzas | not Disk tho | 13:31 |
| mriedem | FILTERS="RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter" | 13:31 |
| mriedem | ^ is what we run by default | 13:32 |
| bauzas | so, we need to remove Ram *and* Diks | 13:32 |
| bauzas | I was just remembering we still run 2 legacy filters | 13:32 |
| mriedem | http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n104 | 13:32 |
| mriedem | ok | 13:33 |
| bauzas | anyway, patch is on its way | 13:33 |
| *** kbaegis has quit IRC | 13:33 | |
| mriedem | sdague: oh right, that docs bug was because cfriesen was asking about this, see the last paragraph or two in that bug report, | 13:34 |
| *** kbaegis has joined #openstack-nova | 13:34 | |
| *** kbaegis has quit IRC | 13:34 | |
| mriedem | someone had forced the service down, upgraded everything else, and then when they tried to set forced_down=False, it failed with ServiceTooOld | 13:34 |
| *** kbaegis has joined #openstack-nova | 13:34 | |
| mriedem | which i think just caused some confusion in how that all works | 13:34 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Update api-guide and api-ref to be clear about forced-down https://review.openstack.org/492533 | 13:35 |
| mriedem | the ServiceTooOld is i'm pretty sure a 500 out of the PUT /os-services API too | 13:35 |
| *** moshele has joined #openstack-nova | 13:35 | |
| sdague | mriedem: yeh, I don't know what the recovery path is on that all, but at least we can be clear on what the state transition into it should look like | 13:35 |
| cdent | does anyone have anything specific for me to do, or shall I continue chasing reviews and randomly testing things by hand? | 13:35 |
| *** kbaegis1 has joined #openstack-nova | 13:36 | |
| *** kbaegis2 has joined #openstack-nova | 13:36 | |
| *** kbaegis has quit IRC | 13:36 | |
| edleafe | cdent: sudo make me a sandwich | 13:37 |
| * cdent flies to new york | 13:37 | |
| *** kbaegis has joined #openstack-nova | 13:38 | |
| *** egonzalez has quit IRC | 13:38 | |
| bauzas | sdague: mriedem: https://review.openstack.org/#/c/492537/ | 13:38 |
| bauzas | ^ devstack change removing legacy filters FTW | 13:39 |
| sdague | bauzas: ok, cool | 13:39 |
| sdague | bauzas: is there a reason we override those in the first place? | 13:39 |
| bauzas | sdague: correct me if I'm wrong but we don't need to modify grenade since it uses default config optoions ? | 13:39 |
| sdague | would we just use defaults? | 13:39 |
| *** moshele has quit IRC | 13:39 | |
| mriedem | sdague: because devstack had the default + SameHost + DifferentHost | 13:39 |
| bauzas | what mriedem said | 13:39 |
| mriedem | tempest has tests for SameHost/DifferentHost filters | 13:39 |
| sdague | ah, cool, good reason | 13:39 |
| mriedem | which aren't defaults | 13:40 |
| bauzas | we could tho make += "SameHost" | 13:40 |
| bauzas | since it's a listopt | 13:40 |
| *** kbaegis has quit IRC | 13:40 | |
| mriedem | in bash? | 13:40 |
| *** kbaegis3 has joined #openstack-nova | 13:40 | |
| *** kbaegis2 has quit IRC | 13:40 | |
| *** kbaegis3 has quit IRC | 13:40 | |
| *** kbaegis1 has quit IRC | 13:40 | |
| sdague | bauzas: that doesn't work in setting nova.conf | 13:40 |
| bauzas | ah, right | 13:40 |
| bauzas | I usually do this directly :) | 13:40 |
| bauzas | sdague: anyway, my question still remains wrt grenade | 13:41 |
| bauzas | sdague: do we need to s// something when we upgrade the node ? | 13:41 |
| sdague | bauzas: grenade should be fine as long as those things didn't get deleted | 13:41 |
| bauzas | in terms of nova.conf -ism | 13:41 |
| sdague | it will use the ocata config in pike | 13:41 |
| bauzas | yeah, my thoughts | 13:41 |
| bauzas | not super then | 13:41 |
| bauzas | we don't remove those filters because CachingScheduler still requires them | 13:42 |
| bauzas | but in theory, you don't need to run those filters since Ocata | 13:42 |
| sdague | right as long as there is normal deprecation cycles you don't need to do anything in grenade | 13:42 |
| bauzas | that's just we wanted to be gentle in Ocata and not ask operators to change their conf when upgrading from Newton | 13:42 |
| sdague | as long as you remove the deprecated setting of things in devstack in the release where you deprecate them | 13:42 |
| bauzas | okay, I think it's reasonable then | 13:43 |
| sdague | it just naturally rolls through it all | 13:43 |
| bauzas | k | 13:43 |
| bauzas | mriedem: looping over https://etherpad.openstack.org/p/nova-pike-release-candidate-todo | 13:44 |
| sdague | mriedem: I think putting the entire recovery path inside the API ref for forced-down is not idea - https://review.openstack.org/#/c/492533/1/api-ref/source/os-services.inc | 13:44 |
| bauzas | mriedem: AFAICT, the top prio for reviewing is then https://review.openstack.org/#/c/491012/ ? | 13:45 |
| sdague | there probably needs to be a specific HA recovery document for the whole thing | 13:45 |
| sdague | this is just "don't use this unless you really are sure" | 13:45 |
| bauzas | it's a force thing | 13:45 |
| bauzas | :p | 13:45 |
| bauzas | 'force' is meaningful in Linux terminology :) | 13:45 |
| mriedem | bauzas: yes | 13:46 |
| bauzas | don't expect good things to happen magically if you use a force flag | 13:46 |
| mriedem | sdague: i didn't say to put the entire recovery path in there... | 13:46 |
| mriedem | just 'never come back' is not really correct, | 13:46 |
| mriedem | i think maybe you mean 'never come back in it's current state' | 13:46 |
| mriedem | or something | 13:46 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Resource tracker compatibility with Ocata and Pike https://review.openstack.org/491012 | 13:47 |
| mriedem | cdent: jaypipes: ^ fixed the functional test failure | 13:47 |
| bauzas | you can "force enable" back | 13:47 |
| cdent | mriedem: roger | 13:47 |
| bauzas | but honestly, that forced-down thing is just a hack because of our SG API | 13:47 |
| bauzas | I'm not sure we should be explicit here | 13:48 |
| *** egonzalez has joined #openstack-nova | 13:51 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954 | 13:51 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Update api-guide and api-ref to be clear about forced-down https://review.openstack.org/492533 | 13:53 |
| *** marst_ has joined #openstack-nova | 13:54 | |
| *** alexchadin has quit IRC | 13:55 | |
| *** takedakn has joined #openstack-nova | 13:56 | |
| *** takashin has joined #openstack-nova | 13:58 | |
| mriedem | nova meeting in 1 minute | 13:59 |
| bauzas | sdague: tried to help with defining why we have a force-down flag https://review.openstack.org/#/c/492533/2 | 13:59 |
| bauzas | seeing that being used by operators scares me | 13:59 |
| bauzas | it was never the intent | 13:59 |
| *** takedakn has quit IRC | 13:59 | |
| bauzas | and like the bug report mentions, we have the disabling thing | 14:00 |
| *** ioggstream has joined #openstack-nova | 14:00 | |
| ioggstream | hi@all | 14:00 |
| *** gouthamr has joined #openstack-nova | 14:00 | |
| sdague | bauzas: right, so I think we have to be careful about putting too much into api-ref because it's supposed to be a reference | 14:00 |
| sdague | we probably need an HA guide here that explains the whole thing | 14:00 |
| ioggstream | about OS:: Nova::ServerGroup policy, how will anti-affinity (default) policy work if I have more vms than compute nodes? | 14:00 |
| bauzas | sdague: well the problem is that our API can be consumed by both end-users and automation tools | 14:00 |
| mriedem | bauzas: the api-ref could link to a more detailed guide | 14:01 |
| sdague | bauzas: I'll be honest, I'm not going to spend all day redrafting this docs patch. I think what I have up there is an improvement, if folks want to take it over and write it instead, I'm good with that | 14:01 |
| bauzas | sdague: to be frank, I do wonder if we should just cut that flag from the nova CLI | 14:01 |
| mriedem | sdague: there is, or was, a maintenance guide | 14:01 |
| bauzas | sdague: okay, I'll push a rev then | 14:01 |
| mriedem | planned and unplanned | 14:01 |
| sdague | mriedem: sure | 14:01 |
| mriedem | i'd put it in there, but it might have been in the now defunct ops guide | 14:01 |
| mriedem | would have to dig | 14:01 |
| bauzas | mriedem: sdague: what are you thinking of just removing that method from our nova CLI ? | 14:01 |
| bauzas | since only scripts should call it | 14:01 |
| mriedem | meeting time | 14:02 |
| sdague | bauzas: people might have manually downed their nodes as well | 14:02 |
| sdague | bauzas: I think it's fine to be in the nova cli | 14:02 |
| sdague | that's an admin tool | 14:02 |
| *** gongysh has joined #openstack-nova | 14:02 | |
| sdague | they just need to realize what they are doing | 14:02 |
| bauzas | agreed | 14:02 |
| bauzas | sdague: I'll try a new rev | 14:02 |
| *** gongysh has quit IRC | 14:03 | |
| sdague | bauzas: I think your assumption that no one should ever call this manually is wrong | 14:03 |
| jianghuaw | jianghua | 14:03 |
| sdague | the important point is they met preconditions | 14:04 |
| sdague | the service is fenced | 14:04 |
| sdague | we don't care how they met those | 14:04 |
| sdague | but they are expressing to nova that they did | 14:04 |
| bauzas | sdague: we have a service group API for that | 14:04 |
| sdague | it might have been a tool | 14:04 |
| bauzas | sdague: the only usecase I heard of was that the SG API was lacking of functionality and either lagging or totally missing the host being down | 14:05 |
| bauzas | sdague: so, eventually, the SG API would meet those preconds | 14:05 |
| bauzas | that's just because Nova isn't intented to be a Nagios system, we allow other tools to fence the host for us | 14:05 |
| sdague | bauzas: that's not good enough if I need it now | 14:05 |
| *** crushil has joined #openstack-nova | 14:05 | |
| bauzas | from our behalf, I mean | 14:06 |
| *** awaugama has joined #openstack-nova | 14:06 | |
| dtantsur | edleafe: hi! my last attempt to use resource classes in the CI ended up with RamFilter removing the nodes | 14:06 |
| *** moshele has joined #openstack-nova | 14:06 | |
| dtantsur | I wonder if I'm missing something.. I thought I disabled requesting RAM/disk/CPU | 14:06 |
| bauzas | dtantsur: Nova by default was still running those filters until yesterday | 14:07 |
| dtantsur | oh | 14:08 |
| edleafe | dtantsur: probably the request was for 1 of the resource class, along with the disk/ram/cpu in the flavor | 14:08 |
| dtantsur | edleafe: I assume I'm removing the request for disk/ram/cpu from flavor | 14:08 |
| dtantsur | lemme get a link | 14:08 |
| dtantsur | edleafe: https://review.openstack.org/#/c/476968/13/devstack/lib/ironic@1889 | 14:09 |
| dtantsur | bauzas: this one, right? https://github.com/openstack/nova/commit/2fe96819c24eff5a9493a6559f3e8d5b4624a8c9 | 14:09 |
| edleafe | dtantsur: ok, then that should work | 14:09 |
| bauzas | dtantsur: correct | 14:09 |
| edleafe | dtantsur: do you have the call to placement anywhere in logs? | 14:10 |
| dtantsur | thanks, I'll see how it looks nowadays | 14:10 |
| bauzas | dtantsur: oh wait | 14:10 |
| bauzas | dtantsur: Ironic is special-case IIRC | 14:10 |
| bauzas | dtantsur: you folks have your own czay filters list :) | 14:10 |
| bauzas | crazy | 14:10 |
| bauzas | in nova | 14:10 |
| dtantsur | yeah, we did change something around it.. lemme check | 14:10 |
| * bauzas is facepalming | 14:11 | |
| dtantsur | https://review.openstack.org/#/c/490459/ | 14:11 |
| dtantsur | this is why I'm seeing the RamFilter, not the ExactRamFilter | 14:12 |
| dtantsur | oh, and by the way. should we kill Exact filters with fire | 14:12 |
| dtantsur | (well, I meant deprecate) | 14:12 |
| dtantsur | ? | 14:12 |
| edleafe | dtantsur: those were there only to support the pretense that an ironic node was a vm | 14:13 |
| edleafe | dtantsur: so yeah, kill 'em! | 14:13 |
| dtantsur | edleafe: wanna get a deprecation patch or should I? | 14:13 |
| edleafe | dtantsur: we should also deprecate the separate ironic filter options, no? | 14:13 |
| dtantsur | edleafe: yep | 14:14 |
| *** jpena|mtg is now known as jpena|off | 14:14 | |
| edleafe | dtantsur: I may have time, but not much | 14:14 |
| edleafe | dtantsur: if you want to start and post a WIP, I can pick it up | 14:14 |
| dtantsur | ENOTMUCHTIME is a common error code nowadays | 14:14 |
| dtantsur | sure, will do | 14:14 |
| * dtantsur git clone nova | 14:15 | |
| *** sree has quit IRC | 14:18 | |
| *** slaweq has quit IRC | 14:19 | |
| bauzas | dtantsur: those Exact* filters could just be treated like the other legacy filters | 14:20 |
| dtantsur | and how do you treat legacy filters? :) | 14:20 |
| bauzas | dtantsur: being removed from the list of filters to run by default, but still in tree for upgrade concerns | 14:20 |
| bauzas | when you upgrade from Ocata, you certainly don't want to update nova.conf for that | 14:21 |
| dtantsur | they're not on by default, there is an option to enable them.. | 14:21 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: test server evacuation with placement https://review.openstack.org/492548 | 14:21 |
| *** sshwarts has quit IRC | 14:21 | |
| dtantsur | folks, what's your next version? (to use with deprecated_since) | 14:22 |
| bauzas | well, in theory you could use CachingScheduler with IroncHostManager I guess | 14:22 |
| bauzas | in that case, you'd still require Exact* filters | 14:22 |
| gibi | mriedem: just out of curiosity create a Pike -> Pike evac test and it seems the allocation on the source host has never cleaned up https://review.openstack.org/#/c/492548/ | 14:22 |
| bauzas | yet another call for deprecating the other scheduler driver we have in tree | 14:22 |
| dtantsur | bauzas: I've never heard of people using it, but yeah. For every crazy feature there are people to try it in production.. | 14:23 |
| ioggstream | does anybody knows if soft-anti-affinity may be enabled in newton ? | 14:23 |
| bauzas | ioggstream: IIRC, yes | 14:23 |
| bauzas | ioggstream: https://blueprints.launchpad.net/nova/+spec/soft-affinity-for-server-group is Mitaka complete | 14:24 |
| ioggstream | bauzas: by default it doesn't work but I saw that mitaka has an ERRATA | 14:24 |
| mriedem | gibi: because the periodic task doesn't cleanup allocations anymore | 14:24 |
| mriedem | gibi: it assumes the scheduler has everything correct | 14:24 |
| mriedem | and the source node is 'down' | 14:24 |
| gibi | mriedem: but not even the source compute cleans up? | 14:24 |
| gibi | mriedem: after started up again? | 14:24 |
| mriedem | if it's down we probably don't care about it | 14:24 |
| mriedem | oh, we'll talk after the meeting | 14:24 |
| gibi | mriedem: sure | 14:24 |
| *** sridharg has quit IRC | 14:26 | |
| mriedem | but yeah the update_available_resource code in pike now does'nt overwrite the allocatoins | 14:26 |
| mriedem | per that change | 14:26 |
| mriedem | so that's why the source compute won't cleanup once it comes back up | 14:26 |
| mriedem | it should remove though.... | 14:26 |
| mriedem | gibi: this one https://review.openstack.org/#/c/491850/ | 14:27 |
| *** lbragstad has quit IRC | 14:28 | |
| gibi | mriedem: my test https://review.openstack.org/#/c/492548/ is top of https://review.openstack.org/#/c/491850/ and I still see the allocation on the source host | 14:28 |
| *** tbachman has quit IRC | 14:34 | |
| *** takashin has left #openstack-nova | 14:35 | |
| *** takedakn has joined #openstack-nova | 14:35 | |
| mriedem | gibi: i think that's probably due to https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1047 | 14:35 |
| gibi | mriedem: checking the debug log... | 14:36 |
| mriedem | gibi: but we should get into https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1145 | 14:36 |
| mriedem | _remove_deleted_instances_allocations | 14:36 |
| gibi | mriedem: I see the debug log you pointed at | 14:37 |
| mriedem | oh it could be https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1187 | 14:38 |
| mriedem | if (instance.host == cn.host and | 14:38 |
| mriedem | instance.node == cn.hypervisor_hostname): | 14:38 |
| mriedem | we continue there | 14:38 |
| mriedem | or if instance.host != cn.host: | 14:38 |
| mriedem | we also continue there | 14:38 |
| *** moshele has quit IRC | 14:38 | |
| mriedem | seems we should check to see if the instance is in self.tracked_migrations | 14:38 |
| gibi | I can insert some extra log to confirm | 14:38 |
| *** priteau has joined #openstack-nova | 14:39 | |
| gibi | ahh there is logs already | 14:40 |
| gibi | it is the instance.host == cn.host where we continue | 14:41 |
| jaypipes | mriedem: ty | 14:43 |
| jaypipes | mriedem: fyi, kinda vacationing today... | 14:43 |
| jaypipes | mriedem: will work on my patches thouhg | 14:43 |
| mriedem | jaypipes: don't think you have anything to work on | 14:45 |
| mriedem | except follow ups for additional testing and whatnot | 14:45 |
| *** yamahata has joined #openstack-nova | 14:46 | |
| *** cleong has joined #openstack-nova | 14:46 | |
| mriedem | gibi: ok so self.tracked_migrations probably won't help after we restart the compute service since that dict will probably be empty | 14:46 |
| mriedem | gibi: i left some comments in your test change, | 14:46 |
| mriedem | we could maybe do some allocation cleanup in https://github.com/openstack/nova/blob/9a66d039a14afd591f4a3b6e655580aeeed17d29/nova/compute/manager.py#L649 | 14:46 |
| jaypipes | mriedem: yeah | 14:46 |
| mriedem | gibi: this is similar to https://bugs.launchpad.net/nova/+bug/1679750 where we don't delete the allocations for the instance on the compute host during a 'local delete' in the API | 14:47 |
| openstack | Launchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,In progress] | 14:47 |
| mriedem | when we can't get to the compute host b/c it's down | 14:47 |
| sdague | mriedem: ok, stepping away from the computer for a bit... | 14:47 |
| *** vks1 has joined #openstack-nova | 14:48 | |
| bauzas | mriedem: https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1134 isn't a cellv2-compatible call since it would be an upcall, no ? | 14:48 |
| gibi | mriedem: should I file a bug or we will squeez the fix into https://review.openstack.org/#/c/491012 ? | 14:48 |
| mriedem | gibi: i think it's a new bug | 14:49 |
| mriedem | gibi: similar to bug 1679750 | 14:49 |
| openstack | bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,In progress] https://launchpad.net/bugs/1679750 | 14:49 |
| mriedem | but slightly different | 14:49 |
| gibi | mriedem: OK, I will report it | 14:49 |
| mriedem | bug 1679750 is about how we don't remove the allocations when we delete the instance and the compute is down | 14:49 |
| mriedem | in your case, the instance isn't deleted, it's evacuated, and we just don't cleanup the allocations from the original host when it comes back up | 14:49 |
| mriedem | bauzas: services are per-cell | 14:50 |
| mriedem | so it's not an upcall | 14:50 |
| *** PsionTheory has joined #openstack-nova | 14:50 | |
| gibi | mriedem: thank for the pointers | 14:50 |
| mriedem | bauzas: this is why we needed to expose the service uuids out of the API in 2.53 | 14:50 |
| mriedem | to identify nova-compute 1 from nova-compute 1 in cell A and cell B | 14:51 |
| bauzas | mriedem: so it only verifies if all the local-cell computes are having that version, right? | 14:51 |
| bauzas | mriedem: since we're in a local compute | 14:52 |
| mriedem | yes | 14:52 |
| mriedem | moves are also restricted to the cell that the instance already lives in | 14:52 |
| *** armax has joined #openstack-nova | 14:53 | |
| ioggstream | bauzas: https://bugzilla.redhat.com/show_bug.cgi?id=1447798 thx++ | 14:53 |
| openstack | bugzilla.redhat.com bug 1447798 in python-openstackclient "soft-anti-affinity is not accepted by python-openstackclient but is with python-novaclient" [High,New] - Assigned to jruzicka | 14:53 |
| ioggstream | fyi | 14:53 |
| mriedem | https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1166 | 14:53 |
| mriedem | ioggstream: i don't think osc is handling nova microversions | 14:53 |
| bauzas | mriedem: correct, so it's probably not a problem | 14:53 |
| mriedem | and the soft affinity policy rules were in a microversion | 14:53 |
| mriedem | bauzas: yes of all the problems we could have with this series of changes, that's one i'm not worried about :) | 14:54 |
| bauzas | fortunately because we don't support multi-cell moves | 14:54 |
| bauzas | okay | 14:54 |
| bauzas | gtk :) | 14:54 |
| mriedem | and likely won't for a long time | 14:54 |
| ioggstream | mriedem: you mean osc challenges the server for the supported options? | 14:54 |
| mriedem | ioggstream: no, i mean i believe osc only ever requests compute API things with 2.1 | 14:55 |
| *** sbezverk has quit IRC | 14:55 | |
| mriedem | and the soft affinity group policy was added in 2.15 https://docs.openstack.org/nova/latest/reference/api-microversion-history#id13 | 14:55 |
| mriedem | so if you're using osc and try to create a server group with policy 'soft-anti-affinity' using microversion 2.1, it's going to fail with a 400 | 14:56 |
| mriedem | saying it doesn't know what that policy is | 14:56 |
| gibi | mriedem: I filed https://bugs.launchpad.net/nova/+bug/1709902 | 14:56 |
| openstack | Launchpad bug 1709902 in OpenStack Compute (nova) "source host allocation not cleaned up in placement after evacuation" [Undecided,New] | 14:56 |
| ioggstream | mriedem: ok, I didn't know the microversion stuff, let me read about it | 14:56 |
| mriedem | ioggstream: you might be able to pass an --openstack-compute-version or something option to osc cli, idk | 14:56 |
| *** Swami has joined #openstack-nova | 14:58 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use ksa adapter for placement conf & requests https://review.openstack.org/492247 | 14:58 |
| *** cfriesen has joined #openstack-nova | 14:58 | |
| ioggstream | mriedem: I'll check and let you know, thx++ | 14:59 |
| openstackgerrit | Dmitry Tantsur proposed openstack/nova master: Deprecate bare metal filters https://review.openstack.org/492563 | 14:59 |
| *** lbragstad has joined #openstack-nova | 14:59 | |
| dtantsur | edleafe: ^^^ | 14:59 |
| * dtantsur did not expect nova unit tests to run 15 minutes, wow :D | 14:59 | |
| *** sree has joined #openstack-nova | 15:00 | |
| edleafe | dtantsur: get a faster machine! :) | 15:00 |
| *** felipemonteiro has joined #openstack-nova | 15:00 | |
| bauzas | mriedem: do you agree with my comment on https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1073 and if so, possibly me passing a new revision ? | 15:03 |
| mriedem | dtantsur: heh yeah you need more VCPUs | 15:05 |
| dtantsur | LOOL | 15:05 |
| *** jaypipes has quit IRC | 15:05 | |
| bauzas | t470s FTW | 15:05 |
| *** jaypipes has joined #openstack-nova | 15:05 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use ksa adapter for placement conf & requests https://review.openstack.org/492247 | 15:06 |
| openstackgerrit | Eric Fried proposed openstack/nova master: nova.utils.get_ksa_adapter() https://review.openstack.org/488137 | 15:06 |
| * bauzas just need to migrate from his old t440s | 15:06 | |
| mriedem | bauzas: given the amount of shit we log continually all the time, like lazy-loading pci request and pci devices for every claim, then i'd say leave that for a follow up patch | 15:06 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Get auth from context for glance endpoint https://review.openstack.org/490057 | 15:06 |
| bauzas | mriedem: ok, firing it up | 15:06 |
| bauzas | and then I'll +W that one | 15:06 |
| *** derekh has quit IRC | 15:06 | |
| *** suresh12 has joined #openstack-nova | 15:07 | |
| *** sapcc-bot1 has joined #openstack-nova | 15:07 | |
| bauzas | (story-telling, if you remember my current laptop, you'll understand how much I'll enjoy to have a new Enter key w/ my new laptop :D ) | 15:07 |
| *** yamamoto_ has quit IRC | 15:08 | |
| dtantsur | bauzas: I have t460s. too old for nova, I guess :D | 15:08 |
| mriedem | bauzas: if you're going to +W anyway, just do it now | 15:08 |
| mriedem | with the gate time as it is | 15:09 |
| bauzas | mriedem: sure | 15:09 |
| mriedem | i don't really want to be doing rc1 at midnight | 15:09 |
| mriedem | midnight is matty's time | 15:09 |
| *** sapcc-bot has quit IRC | 15:09 | |
| dtantsur | oh, rc1.. does this ironic-related stuff have a chance to get in? | 15:09 |
| bauzas | mriedem: yeah I should consider relocating to Hawaii | 15:10 |
| bauzas | given the TZ | 15:10 |
| *** yamamoto has joined #openstack-nova | 15:10 | |
| bauzas | or French Polynesia, that could help too | 15:10 |
| *** slaweq has joined #openstack-nova | 15:10 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: test server evacuation with placement https://review.openstack.org/492548 | 15:10 |
| mriedem | gibi: per your bug, i'm curious why TestLocalDeleteAllocations isn't failing then | 15:11 |
| *** yamamoto has quit IRC | 15:11 | |
| mriedem | oh i know why | 15:11 |
| mriedem | it's deleted | 15:11 |
| mriedem | and when the RT gets InstanceNotFound, it deletes the allocations | 15:11 |
| *** hareesh has quit IRC | 15:12 | |
| jaypipes | is it just me or is the gate getting better... | 15:12 |
| mriedem | dtantsur: i've been leaning on dansmith for that change | 15:12 |
| mriedem | dtantsur: but he's out the rest of the week | 15:12 |
| mriedem | dtantsur: so it's likely rc2 at this point | 15:12 |
| dtantsur | yeah, it's fine, as long as it gets into Pike at all | 15:12 |
| *** burt has joined #openstack-nova | 15:12 | |
| mriedem | jaypipes: except for the slow nodes and timeouts, it's not bad | 15:12 |
| *** kfarr has quit IRC | 15:12 | |
| mriedem | gibi: i wonder why https://review.openstack.org/#/c/492548/ keeps saying it can't merge? | 15:14 |
| mriedem | but don't rebase the entire series... | 15:14 |
| gibi | mriedem: I think it is because I adding the test at the end of the test_server.py | 15:15 |
| jaypipes | mriedem, bauzas, sdague: should be simple review: https://review.openstack.org/#/c/491822/ | 15:15 |
| gibi | mriedem: but I can try locally | 15:15 |
| *** Oku_OS is now known as Oku_OS-away | 15:15 | |
| gibi | mriedem: yes, the evac patch conflicts on the file ending | 15:16 |
| gibi | mriedem: master has test_resize_not_enough_resource at the end | 15:16 |
| gibi | mriedem: but Jay's branch doesn't contain that | 15:16 |
| gibi | mriedem: I will have no time today to do the evac fix in the init_host so if this is important somebody can take over. But tomorrow I can continue | 15:17 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: Make compute log less verbose with allocs autocorrection https://review.openstack.org/492571 | 15:17 |
| bauzas | mriedem: HTH ^ | 15:18 |
| openstackgerrit | Dmitry Tantsur proposed openstack/nova master: Deprecate bare metal filters https://review.openstack.org/492563 | 15:18 |
| *** itlinux has joined #openstack-nova | 15:19 | |
| dtantsur | bauzas, mriedem, easy win for your review stats ^^ (like if you needed more, heh) | 15:19 |
| *** suresh12 has quit IRC | 15:20 | |
| *** annegentle has joined #openstack-nova | 15:21 | |
| gibi | mriedem, dansmith: I think this is ready to go in, Jay was +2 already but I needed to rebase: https://review.openstack.org/#/c/491529/ | 15:21 |
| jaypipes | gibi: hey, there's an urgent request for your assistance here: http://lists.openstack.org/pipermail/openstack/2017-August/045339.html | 15:22 |
| * jaypipes runs away | 15:22 | |
| *** namnh has joined #openstack-nova | 15:22 | |
| gibi | jaypipes: seems like s spam :) | 15:26 |
| *** annegentle has quit IRC | 15:26 | |
| mriedem | bauzas: sdague: melwitt: i triaged https://bugs.launchpad.net/nova/+bug/1709594 | 15:27 |
| openstack | Launchpad bug 1709594 in OpenStack Compute (nova) "live-migration without '--block-migrate" failed with "No sql_connection parameter is established"" [High,Triaged] | 15:27 |
| mriedem | it's a superconductor issue, | 15:27 |
| mriedem | the xenapi driver tries to get host aggregates during a live migration, | 15:27 |
| mriedem | and the aggregates are in the API DB now, | 15:27 |
| jaypipes | gibi: was just joking you. | 15:27 |
| mriedem | so this is an upcall attempt from the compute service, in a cell with no access to the api db | 15:27 |
| *** sdake_ is now known as sdake | 15:27 | |
| gibi | jaypipes: ohh. OK. You got me :) | 15:28 |
| *** annegentle has joined #openstack-nova | 15:28 | |
| *** thingee_ has joined #openstack-nova | 15:31 | |
| mriedem | gibi: i think we could deal with the source allocation cleanup bug as a backport | 15:34 |
| *** takedakn has quit IRC | 15:37 | |
| gibi | mriedem: ack | 15:39 |
| *** moshele has joined #openstack-nova | 15:40 | |
| mriedem | it's only a problem unless the compute comes back online | 15:41 |
| mriedem | s/unless/until/ | 15:41 |
| *** moshele has quit IRC | 15:43 | |
| ioggstream | mriedem: +1 [stack@c1f-ops-dirp00 openshift-on-openstack]$ o server group create deleteme-sg-3 --policy soft-anti-affinity --os-compute-api-version 2.15 | 15:44 |
| sdague | mriedem: good call | 15:44 |
| *** gbarros has quit IRC | 15:45 | |
| mriedem | ioggstream: cool | 15:46 |
| *** chyka has joined #openstack-nova | 15:46 | |
| *** tbachman has joined #openstack-nova | 15:49 | |
| openstackgerrit | Spencer Yu proposed openstack/python-novaclient master: Nova client should retry with Retry-After value https://review.openstack.org/447766 | 15:49 |
| mriedem | ioggstream: going to update that bugzilla? | 15:49 |
| mriedem | btw, it's nice to see my ibm red hat bugzilla account is still active | 15:50 |
| mriedem | and has permissions to ibm bugs | 15:50 |
| *** jpena|off is now known as jpena | 15:50 | |
| ioggstream | mriedem: yes, I'll do it. I've an rh account: I'll try with it. | 15:51 |
| ioggstream | mriedem: my issue is now that heat doesn't seem to support soft-anti-affinity | 15:51 |
| bauzas | dtantsur: edleafe: I'm confused by the ironic upgrade path (see https://review.openstack.org/#/c/492563/2) | 15:52 |
| bauzas | dtantsur: edleafe: we could potentially have old ironic nodes to schedule with a pike scheduler | 15:52 |
| dtantsur | bauzas: well, we're not removing anything, are we? | 15:53 |
| bauzas | dtantsur: sure, but you send a signal with those deprecations | 15:53 |
| bauzas | I just don't want operators to freak out | 15:53 |
| dtantsur | well, in Queens the old style of scheduling is not going to be possible | 15:53 |
| dtantsur | we have to send some signal about that coming | 15:53 |
| bauzas | so it's just a wording issue | 15:54 |
| dtantsur | re upgrade https://docs.openstack.org/ironic/latest/admin/upgrade-guide.html#upgrading-from-ocata-to-pike | 15:54 |
| dtantsur | we should change s/recommended/required/, I'm going to have another update anyway | 15:54 |
| bauzas | but the point is, if you deprecate in Pike, that means you accept operators to not use those opts by the Pike timeframe, which could be a bit of concern if they roll out upgrades | 15:54 |
| bauzas | dtantsur: I briefly looked at your relnotes too | 15:55 |
| bauzas | dtantsur: and haven't found any clear ask for providing resource classes for nodes | 15:55 |
| bauzas | (talking of https://docs.openstack.org/releasenotes/ironic/unreleased.html )à | 15:55 |
| dtantsur | bauzas: this is coming as part of https://review.openstack.org/#/c/491773/ | 15:55 |
| bauzas | ah cool | 15:55 |
| bauzas | dtantsur: so I guess some linkage between relnotes could be appreciated | 15:56 |
| bauzas | like, ironic has to be upgraded before compute nodes obviously | 15:56 |
| bauzas | but ironic can be upgraded after placement and scheduler, right | 15:56 |
| *** Apoorva has joined #openstack-nova | 15:56 | |
| dtantsur | I guess so, yeah | 15:56 |
| bauzas | and before you run a fresh pike nova-compute, you have to update your ironic cloud to set the resource classes | 15:57 |
| bauzas | that is the upgrade ordering I care, since operators would query for | 15:57 |
| dtantsur | right, we can do it while still running Ocata | 15:57 |
| dtantsur | should this patch be finished today? I have meetings, then I'd prefer to bail out (I'm in EU time) | 15:58 |
| *** aarefiev is now known as aarefiev_afk | 16:00 | |
| bauzas | dtantsur: you mean the notes ? | 16:00 |
| ildikov | mriedem: quick meeting if you're available | 16:00 |
| dtantsur | bauzas: this nova patch | 16:00 |
| bauzas | dtantsur: well, mriedem will cut the rc1 tag tonight for us (as well, I'm CEST) so that would mean those notes would require a backport if we want them in the pike tree | 16:01 |
| *** annegentle has quit IRC | 16:01 | |
| dtantsur | this sounds like "today".. | 16:01 |
| *** suresh12 has joined #openstack-nova | 16:01 | |
| dtantsur | edleafe: is it possible you take on fixing release note wording for ^^^? | 16:01 |
| bauzas | dtantsur: ideally, I'd have appreciated to see https://review.openstack.org/#/c/491773/ landed first, but I guess we need to send them concurrently to the gate | 16:02 |
| *** dklyle is now known as david-lyle | 16:02 | |
| dtantsur | bauzas: ironic is not branched today, so it may wait a bit, I think | 16:03 |
| edleafe | bauzas: dtantsur: I'll review after API-WG meeting | 16:03 |
| dtantsur | cool | 16:03 |
| bauzas | dtantsur: oh right, not the same cadence than us | 16:03 |
| bauzas | well, I can push a new rev | 16:03 |
| bauzas | edleafe: ^ | 16:04 |
| dtantsur | if you don't mind! | 16:04 |
| edleafe | I never mind | 16:04 |
| edleafe | :) | 16:04 |
| mriedem | pushing what now? | 16:06 |
| mriedem | we already have stuff in the nova release notes that say ironic gets upgraded before nova | 16:06 |
| *** lyan has quit IRC | 16:06 | |
| mriedem | for some unrelated features, like boot from volume | 16:06 |
| mriedem | we don't need to say that 5 times | 16:06 |
| bauzas | mriedem: context is https://review.openstack.org/#/c/492563/ | 16:07 |
| edleafe | mriedem: it's the sixth time that will sink in | 16:10 |
| *** suresh12 has quit IRC | 16:10 | |
| *** sahid has quit IRC | 16:11 | |
| *** yamamoto has joined #openstack-nova | 16:12 | |
| *** crushil has quit IRC | 16:12 | |
| dtantsur | edleafe, bauzas: updated ironic part (docs and reno) | 16:13 |
| dtantsur | wording suggestions are welcome, my wording can be awful sometimes | 16:13 |
| mriedem | this doesn't have to be done in pike does it? | 16:14 |
| mriedem | seems like too many moving parts | 16:14 |
| dtantsur | mriedem: these filters won't work in Queens, no? | 16:14 |
| mriedem | idk | 16:14 |
| mriedem | i just, | 16:14 |
| * dtantsur +100 to "too many moving parts" | 16:14 | |
| mriedem | i can only have my head wrapped around 10 different RC1 stop ship omfg blocking issues at one time | 16:14 |
| mriedem | and this is #11 | 16:15 |
| edleafe | unless we drag it out further, the plan is that in queens, ironic will only be scheduled using custom resource classes | 16:15 |
| * dtantsur feels sorry for mriedem, and shares his pain with his own PTL struggles | 16:15 | |
| edleafe | mriedem: this doesn't change functionality, right? Just marks things as deprecated | 16:15 |
| *** Drankis has quit IRC | 16:15 | |
| efried | If the placement API gives me a 500, and I'm using mod_wsgi, where do I find the log that's gonna tell me WHY? (devstack) | 16:16 |
| mriedem | efried: the placement-api logs | 16:16 |
| sdague | efried: uwsgi? | 16:16 |
| efried | mod-wsgi | 16:16 |
| efried | I know, I know, it's deprecated. Is that my actual problem? | 16:16 |
| sdague | ummm... I thought we deleted that bit | 16:17 |
| *** yamamoto has quit IRC | 16:17 | |
| sdague | efried: mod-wsgi is going to be an apache log | 16:17 |
| efried | Heh. If it's deleted, not just deprecated, I guess that'd do it. | 16:17 |
| efried | Okay, where's that guy? | 16:17 |
| sdague | efried: we may not have deleted it | 16:17 |
| sdague | efried: whereever your apache logs | 16:17 |
| efried | I mean, our CI is running with it and (mostly) succeeding. | 16:17 |
| sdague | /var/log/apache/placement* | 16:17 |
| sdague | on an ubuntu like thing | 16:17 |
| sdague | it will be a different place on a redhat like thing | 16:18 |
| openstackgerrit | Merged openstack/nova master: remove log message with potential stale info https://review.openstack.org/492242 | 16:18 |
| *** kristian__ has quit IRC | 16:18 | |
| efried | nyaha! Thanks sdague | 16:18 |
| *** Swami has quit IRC | 16:19 | |
| *** mriedem is now known as mriedem_away | 16:20 | |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: Deprecate bare metal filters https://review.openstack.org/492563 | 16:21 |
| bauzas | dtantsur: edleafe: tried some rewording ^ | 16:21 |
| dtantsur | thanks, looking | 16:21 |
| efried | That placement log has no entries since two weeks ago :( | 16:21 |
| efried | (and the 500 happened two minutes ago) | 16:21 |
| openstackgerrit | Merged openstack/nova master: placement: refactor healing of allocations in RT https://review.openstack.org/491850 | 16:23 |
| *** pcaruana has quit IRC | 16:25 | |
| sdague | efried: it is hard for me to debug remotely, if there is a box I can connect to I can look | 16:26 |
| sdague | I have IBM vpn access so hopefully a path | 16:26 |
| *** psachin has joined #openstack-nova | 16:27 | |
| *** itlinux has quit IRC | 16:27 | |
| *** lucasxu has quit IRC | 16:27 | |
| *** gjayavelu has joined #openstack-nova | 16:28 | |
| cdent | efried: there may still be a non wsgi log where the other nova logs are. another place to check are any other apache log you can find (grep for ‘resource_providers’). where things end up gets weird | 16:29 |
| *** s1061123 has quit IRC | 16:29 | |
| cdent | and where a 500 ends up with mod_wsgi in the first place (even outside openstack) can be odd | 16:29 |
| cdent | sometimes it will be the central apache error.log | 16:29 |
| *** namnh has quit IRC | 16:30 | |
| efried | cdent Yeah, I'm finding it in the horizon_access.log (which is odd - I'm not using horizon at all). But just the request/response headers, not the error trace. | 16:30 |
| cdent | is a horizon_error.log? or error.log? | 16:30 |
| efried | There's an error.log | 16:31 |
| *** saphi has joined #openstack-nova | 16:31 | |
| efried | it has nothing for the last 6h | 16:31 |
| *** lucasagomes is now known as lucas-afk | 16:32 | |
| cdent | efried: I missed the earlier discussion, what time does this code come from? | 16:32 |
| efried | I'm using latest master nova. nova-powervm driver plugged in, but wouldn't think that's in the code path. | 16:32 |
| cdent | what installed placement? | 16:33 |
| *** hongbin has joined #openstack-nova | 16:33 | |
| sdague | cdent: I'm going to try to get on the box and poke to see if we can iterate through it | 16:33 |
| *** rcernin has quit IRC | 16:33 | |
| cdent | If it’s nova master there should still be a non apache log for errors, based on the work you did last summer sdague | 16:34 |
| cdent | _unless_ the 500 is in the calling of the wsgi app (rather than within the wsgi app) | 16:35 |
| cdent | I’d be curious to hear what it turns out to be | 16:35 |
| efried | esberglu We're still using mod-wsgi in our CI, right? | 16:40 |
| bauzas | folks, bailing out for a few hours, \o | 16:41 |
| efried | sdague cdent Ya know, my nova is current, but my other stacky stuff is a couple weeks old. Could that do it? | 16:41 |
| efried | I mean, regardless, I would like to be able to figure out where to look for the real cause of a generic 500... | 16:42 |
| *** mingyu has quit IRC | 16:43 | |
| cdent | efried: that’s why I asked about what’s doing your deployment (devstack, whatever) | 16:43 |
| efried | oh, sorry, I misunderstood the question then. Yeah, I did a devstack a couple weeks ago, been dinking around with the nova code since then. But hadn't yet had occasion to get this far (been just working conf stuff, so stopping early). | 16:44 |
| *** lyan has joined #openstack-nova | 16:44 | |
| cdent | a devstack from two weeks ago that is still mod_wsgi surprises me | 16:45 |
| cdent | so it may be that your error is in journalctl | 16:45 |
| cdent | journalctl --unit devstack@placement-api | 16:46 |
| efried | cdent That journalctl unit doesn't exist. | 16:48 |
| cdent | I’ll leave you in sdague’s good hands then. You seem to have some weird. :) | 16:49 |
| efried | cdent The placement log is in /var/log/apache2 | 16:49 |
| efried | cdent But it doesn't have anything in it. Leading me to believe we aren't getting that far. | 16:49 |
| cdent | oh yeah, I forgot that you have that file | 16:49 |
| cdent | look in all the other files in /var/log/apache2 for whatever is the central error file | 16:50 |
| sdague | efried: yeh, it's super weird, it looks like placement isn't running | 16:50 |
| *** Swami has joined #openstack-nova | 16:50 | |
| efried | sdague I'm not real savvy here, but I thought placement was (as of yet) part of n-cpu. | 16:50 |
| sdague | efried: it's a wsgi script | 16:51 |
| sdague | so, actually it looks like it's there under apache, but not running for some reason | 16:51 |
| sdague | or running but not connected | 16:51 |
| cdent | or died | 16:51 |
| efried | Should I try restarting httpd? | 16:52 |
| sdague | efried: wait a sec before doing that | 16:52 |
| efried | suresure. | 16:52 |
| sdague | I want to figure out if there is any other postmortem here | 16:52 |
| sdague | http://paste.openstack.org/show/618073/ - apache status | 16:52 |
| *** slaweq has quit IRC | 16:53 | |
| efried | FWIW, the stack itself would have been done a couple weeks ago, and the only service I've been mucking with is n-cpu. | 16:53 |
| *** marst_ has quit IRC | 16:54 | |
| *** marst_ has joined #openstack-nova | 16:54 | |
| sdague | hmmm... well all the access for all services is in the horizon access log | 16:55 |
| sdague | which, is surely a bug | 16:55 |
| sdague | however, it's a bread trail | 16:55 |
| efried | sdague If it's unique to mod-wsgi, it might not be a bug that'll get attention. | 16:55 |
| sdague | yeh | 16:55 |
| sdague | we also don't enable horizon in normal testing | 16:56 |
| cdent | that’s not a mod-wsgi bug, that’s an apache misconfiguration, and is part of the driver to switch to uwsgi in devstack | 16:56 |
| cdent | which ever section ends up listening on port 80 (or 443) with a log configuration will end up getting lots of things that listen on that port | 16:56 |
| efried | I can try to restack with uwsgi, but last time I tried, it didn't work. (Which is why we're still using mod-wsgi in our CI.) | 16:57 |
| *** yamahata has quit IRC | 16:57 | |
| efried | (esberglu We either need to figure out the uwsgi thing or add /var/log/apache2/* to our CI log dump.) | 16:57 |
| sdague | ah, you know, I think the crux of it is after the uwsgi cut over we started dropping ports | 16:58 |
| cdent | apache makes the assumption that if you are onthe same port, you’re using the same log | 16:58 |
| sdague | yeh | 16:58 |
| sdague | yeh, this is part of why mod-wsgi is such a pain on the dev side | 16:59 |
| cdent | and when horizon is enabled it is consuming port 80 in a weird way | 16:59 |
| sdague | efried: so my starting point was "nova-manage upgrade check" | 16:59 |
| sdague | which went all 500 stack tracy | 16:59 |
| sdague | yes, placement is returning 500s, however I can't find them landing in a log anywhere | 17:00 |
| *** dtantsur is now known as dtantsur|afk | 17:00 | |
| sdague | http://paste.openstack.org/show/618074/ horizon_access.log | 17:01 |
| *** baoli has quit IRC | 17:01 | |
| efried | yeah, just what I was seeing. | 17:01 |
| *** ralonsoh has quit IRC | 17:01 | |
| efried | Mine were coming from the compute service trying to suss out the resource providers | 17:01 |
| efried | sdague Well, if you're game to help me debug why uwsgi stack is failing, I can try restacking thusly. | 17:02 |
| sdague | efried: yeh, I would do that | 17:02 |
| efried | That would be a big help in general. | 17:02 |
| sdague | make sure you do a ./clean.sh | 17:02 |
| efried | Okay, rippinit. | 17:02 |
| *** igordcard is now known as igordc | 17:02 | |
| sdague | yeh, I don't think anything else useful can come from this install | 17:02 |
| efried | I did a lot of useful stuff wrt service catalog lookups. | 17:03 |
| efried | But I was breaking right at driver init to muck around in pdb | 17:03 |
| efried | But now I'm working on converting the placement API over (https://review.openstack.org/#/c/492247/) so I kinda wanted to see it working :) | 17:04 |
| *** baoli has joined #openstack-nova | 17:04 | |
| efried | ...and you can see from ^^ that our CI isn't having any trouble with it. | 17:04 |
| sdague | :) | 17:05 |
| efried | (just checked the compute logs to be sure - as if we could pass without placement being happy in the first place - and there's no 500s) | 17:05 |
| *** ociuhandu has quit IRC | 17:06 | |
| *** kfarr has joined #openstack-nova | 17:06 | |
| *** abalutoiu has quit IRC | 17:06 | |
| sdague | yeh, it's in a weird intermediate state I think, there is a reason why mod_wsgi is something i wanted to remove from the dev/test stack. It just hits a bunch of odd apachisms that don't fit well with our other assumptions | 17:06 |
| efried | sdague To be clear, all I need to do to switch is remove WSGI_MODE=mod_wsgi from my local.conf? | 17:07 |
| sdague | yep | 17:07 |
| efried | k. Refreshing other project clones... | 17:07 |
| *** sree has quit IRC | 17:08 | |
| efried | aaand stacking... | 17:10 |
| melwitt | mriedem_away: I was wondering if we need to document that for multi-cell with nova-network, upcalls from compute are required for quota checks. I was thinking we might not have to because IIRC we're not supporting multi-cell + nova-network | 17:10 |
| melwitt | I think single cell would be okay because there's no isolation from the API DB there | 17:11 |
| efried | Meanwhile, any idea what glorious magic makes a .txt.gz journalctl log show up in the browser with color codes translated? | 17:13 |
| *** marst_ has quit IRC | 17:13 | |
| *** Apoorva_ has joined #openstack-nova | 17:14 | |
| *** mvk has quit IRC | 17:14 | |
| *** annegentle has joined #openstack-nova | 17:14 | |
| *** slaweq has joined #openstack-nova | 17:16 | |
| *** itlinux has joined #openstack-nova | 17:17 | |
| *** Apoorva has quit IRC | 17:18 | |
| *** lpetrut has quit IRC | 17:19 | |
| *** rajathagasthya has joined #openstack-nova | 17:20 | |
| sdague | efried: magic yet to be written | 17:21 |
| *** mingyu has joined #openstack-nova | 17:21 | |
| efried | sdague But it works | 17:22 |
| efried | At least in my browser | 17:22 |
| sdague | efried: interesting | 17:22 |
| efried | Except that for powervm logs, it doesn't *quite* work. | 17:22 |
| efried | sdague What, it doesn't do that for you? | 17:22 |
| sdague | efried: oh, it's not color codes translated | 17:23 |
| sdague | you mean the coloring of - http://logs.openstack.org/81/488381/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/2b8b331/logs/screen-c-api.txt.gz ? | 17:23 |
| efried | sdague Yup. | 17:23 |
| sdague | https://github.com/openstack-infra/os-loganalyze | 17:23 |
| *** suresh12 has joined #openstack-nova | 17:24 | |
| sdague | is a wsgi filter for the logs | 17:24 |
| efried | ...that runs under the auspices of apached? | 17:24 |
| sdague | yeh, log access is proxied through that | 17:24 |
| efried | ...and would need to exist on the log server. | 17:24 |
| sdague | yes | 17:24 |
| sdague | it processes in real time | 17:24 |
| efried | So ours must be outdated or something. Cool. esberglu ^^ ! | 17:24 |
| sdague | there is a regex on the files it will attempt to do it with | 17:25 |
| efried | sdague Cause for us, it shows up HTMLified, with the log filters, but no color translation. | 17:25 |
| efried | http://184.172.12.213/manual/rest_logs/logs/n-cpu.txt.gz | 17:25 |
| efried | (This isn't in production yet, cause we've been naming 'em .log.gz, but we're converting over) | 17:25 |
| sdague | https://github.com/openstack-infra/os-loganalyze/blob/master/os_loganalyze/filter.py#L24-L31 | 17:25 |
| sdague | yeh, you'd need to setup this code on apache | 17:25 |
| *** baoli has quit IRC | 17:26 | |
| sdague | https://github.com/openstack-infra/puppet-openstackci/blob/master/templates/logs.vhost.erb#L106 and friends to see how you'd set it up on apache | 17:27 |
| sdague | it requires a few rewrite rules | 17:27 |
| *** baoli has joined #openstack-nova | 17:27 | |
| *** baoli has quit IRC | 17:27 | |
| efried | sdague Is that not the same script that does the filtering capability? | 17:27 |
| *** jpena is now known as jpena|mtg | 17:30 | |
| efried | sdague Okay that stack failed, on image create. | 17:30 |
| *** markus_z has quit IRC | 17:30 | |
| efried | apache error log shows an ENOSPC in /tmp. I've got .8GB in there; is it really trying to copy the image through /tmp?? | 17:31 |
| *** yamahata has joined #openstack-nova | 17:31 | |
| *** hemna_ has joined #openstack-nova | 17:32 | |
| *** saphi has quit IRC | 17:34 | |
| *** baoli has joined #openstack-nova | 17:37 | |
| *** krtaylor has quit IRC | 17:38 | |
| *** vks1 has quit IRC | 17:40 | |
| sdague | efried: it might be | 17:40 |
| efried | sdague Restacking, gonna watch /tmp... | 17:41 |
| sdague | mtreinish: ^^^ | 17:41 |
| sdague | it might be specific on the uwsgi glance | 17:41 |
| sdague | and, if so, that could be an interesting issue | 17:41 |
| * efried googles how to grow /tmp... | 17:41 | |
| *** ioggstream has quit IRC | 17:42 | |
| sdague | you all are booting a real image, not a cirros, right? | 17:42 |
| sdague | I could totally imagine that issue to have slipped through because cirros | 17:42 |
| cdent | excellent point, sdague, at least one interation of the uwsgiification of glance needed to do some writes to disk, but I don’t know if that version landed, and if it did where on disk | 17:43 |
| *** annegentle has quit IRC | 17:44 | |
| efried | sdague Yeah, it's a real image - though it's pretty small (2GB) | 17:44 |
| sdague | efried: yeh... but there was all manner of tricksyness to get uwsgi glance to work | 17:45 |
| sdague | and I can totally imagine full write of the upload to tmp was part of that | 17:45 |
| efried | How... precious. | 17:45 |
| *** slaweq_ has joined #openstack-nova | 17:46 | |
| *** mriedem_away is now known as mriedem | 17:48 | |
| mriedem | melwitt: hmm, nova-network only starts in a cells v1 config | 17:49 |
| mriedem | so yeah, i don't think we probably have to worry about that for cells v2 multi-cell | 17:49 |
| *** slaweq has quit IRC | 17:50 | |
| *** ociuhandu has joined #openstack-nova | 17:50 | |
| *** gouthamr_ has joined #openstack-nova | 17:50 | |
| *** gouthamr has quit IRC | 17:51 | |
| *** lpetrut has joined #openstack-nova | 17:51 | |
| efried | sdague Yup, thar she blows. During that image create command, /tmp fills quickly, then stack bails. | 17:56 |
| mriedem | dude | 17:57 |
| mriedem | thar she blars | 17:57 |
| mriedem | bauzas: jaypipes: we want this in for pike too https://review.openstack.org/#/c/492234/ | 17:58 |
| mriedem | melwitt: ^ | 17:58 |
| *** lpetrut has quit IRC | 17:58 | |
| jaypipes | mriedem: done | 17:58 |
| *** SamYaple has joined #openstack-nova | 17:59 | |
| mriedem | thanks | 17:59 |
| SamYaple | any chance I will see this backport merge https://review.openstack.org/#/c/486272/ ? im not sure what novas backport policies are | 18:00 |
| mriedem | melwitt: can you also help go through the set of docs changes leading up to https://review.openstack.org/#/c/491853/ ? that's the target for the docs migration for rc1 | 18:00 |
| mriedem | SamYaple: maybe | 18:01 |
| mriedem | there should be a push for stable branch reviews once we are clear of pike rc1 | 18:01 |
| SamYaple | mriedem: ok thanks. i can't run ocata in my setup without this patch and im not a huge fan of carrying patches which im currently doing with it | 18:03 |
| sdague | SamYaple: +2 on that. | 18:03 |
| SamYaple | nice sdague! thanks | 18:04 |
| mtreinish | sdague, cdent: we ended up doing mod_proxy with http traffic on a local port | 18:04 |
| mtreinish | glance runs uwsgi in http mode and apache just forwards the traffic to the local uwsgi daemon | 18:04 |
| sdague | mtreinish: yeh, file upload seems to create a full temp file though | 18:04 |
| sdague | which, is actually kind of an issue | 18:05 |
| *** imadjerra has joined #openstack-nova | 18:05 | |
| cdent | yeah, that will be because of a tweak to mod proxy’s behavior | 18:05 |
| sdague | from nova bugs, 120GB snapshots are a thing | 18:05 |
| sdague | no one is planning N * 120GB free in /tmp | 18:05 |
| *** imadjerra has quit IRC | 18:07 | |
| cdent | I guess that means the question becomes: is anyone planning to duplicate devstack’s way of running glance? | 18:08 |
| mtreinish | sdague: I'm not sure how the uwsgi change would effect things there. We ran basically the same config before just instead of uwsgi running glance it was glance standalone. The apache config was more or less the same for tls | 18:08 |
| mtreinish | unless uwsgi is using /tmp as a cache | 18:08 |
| cdent | it’s mod proxy | 18:08 |
| *** Apoorva_ has quit IRC | 18:08 | |
| * cdent finds reference | 18:08 | |
| sdague | cdent: that's an impact of the tls termination? | 18:08 |
| *** Apoorva has joined #openstack-nova | 18:09 | |
| efried | mtreinish Is there a uwsgi config somewhere that I can tell to use something other than /tmp ? | 18:09 |
| efried | I'm trying again with TMPDIR=/var/tmp (where I have some more space) - but that'll only affect stuff running directly under the auspices of stack.sh | 18:10 |
| efried | mm, unless I set it in /etc/environment. Let's see if that flies... | 18:11 |
| *** annegentle has joined #openstack-nova | 18:11 | |
| cdent | sdague, mtreinish, efried : https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#request-bodies | 18:11 |
| cdent | it’s the sendcl thing | 18:11 |
| cdent | that was added to the proxy config when glance switched to uwsg | 18:12 |
| mtreinish | cdent: the mod_proxy config for glance should not be using sendcl | 18:12 |
| cdent | did you take it back out? | 18:12 |
| mtreinish | that was one of the paths I was investigating and it didn't work for various reasons (including this issue growing tmp space) | 18:13 |
| melwitt | mriedem: sure thing | 18:13 |
| mtreinish | cdent: https://github.com/openstack-dev/devstack/blob/master/lib/apache#L333 is the path for glance | 18:14 |
| cdent | it’s currently on line 276 of lib/apache | 18:14 |
| mtreinish | cdent: right that's the path that's not glance | 18:14 |
| cdent | is setenv iglobal to apache ? | 18:14 |
| cdent | It looks like it may be | 18:15 |
| *** Apoorva_ has joined #openstack-nova | 18:15 | |
| sdague | so... if you hit uwsgi directly without the tls proxy, this wouldn't be an issue? | 18:16 |
| cdent | sdague: unclear, because it may be the mod_proxy_uwsgi also attends to it? | 18:17 |
| cdent | I’m not certain this is the problem, but I do think sendcl is a factor, or something like it | 18:18 |
| cdent | hard to be clear what uwsgi does in the face of file uploads | 18:18 |
| openstackgerrit | Merged openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | 18:18 |
| sdague | yeh, it's supposed to be chunk encoded upload | 18:18 |
| sdague | but bouncing through proxies... things get tricky | 18:18 |
| *** Apoorva has quit IRC | 18:19 | |
| mtreinish | is there a running system with this issue? because it's easy to confirm if apache is passing chunked data to uwsgi with a tcpump on the local port | 18:19 |
| cdent | if you are sending chunked but demanding content-lenght (from whatever directive) a write will be required | 18:19 |
| mriedem | sdague: i've got a couple of comments in here https://review.openstack.org/#/c/491815 | 18:20 |
| mtreinish | cdent: if setenv is globally scoped across different site files we probaly just need to create a virtualhost for the glance path to isolate it from the other sites | 18:20 |
| mriedem | melwitt: take a look at my comments if you update | 18:20 |
| mtreinish | cdent: fwiw, the glance path doesn't use mod_proxy_uwsgi | 18:21 |
| cdent | I know, but it does use mod_proxy doesn’t it? At least that’s what I thought I heard you say? | 18:21 |
| mtreinish | yes it does | 18:21 |
| melwitt | mriedem: phew, in the nick of time | 18:21 |
| melwitt | looks like sdague is on it already | 18:22 |
| sdague | mriedem: ok, so an inline change to that is going to be 4 - 6 hours to get check results back | 18:22 |
| sdague | how late you staying up for the branch? | 18:22 |
| sdague | vs. fix later | 18:22 |
| mriedem | it's just docs, shouldn't run all jobs | 18:23 |
| sdague | it doesn't | 18:23 |
| sdague | the check queue is starved out | 18:23 |
| *** liverpooler has joined #openstack-nova | 18:23 | |
| melwitt | hm, yeah I didn't think about going through the check queue again. though I thought it would be fast for docs | 18:23 |
| sdague | it's hours before you get a slot | 18:24 |
| mriedem | are you already doing a dump of review comment follow up stuff at the end of the series? | 18:24 |
| mriedem | like https://review.openstack.org/#/c/491517/ | 18:24 |
| sdague | I rebased that and stuck it on the end | 18:24 |
| sdague | as it was in merge conflict, and I didn't want to generate more conflicts | 18:25 |
| mriedem | alright i'm fine with following up at the end of the series | 18:25 |
| sdague | ok, can do | 18:25 |
| mriedem | melwitt: you want the +W ? | 18:25 |
| *** rajathagasthya has quit IRC | 18:25 | |
| openstackgerrit | Merged openstack/nova master: Add format_dom for PCI device addresses https://review.openstack.org/491822 | 18:25 |
| melwitt | mriedem: hell yes I do | 18:25 |
| sdague | I also tried to set expectations here - https://review.openstack.org/#/c/492124 how we really do need to think about docs reviews a bit differently than code reviews, especially because they are merge magnets | 18:26 |
| sdague | yeh, it looks like we're at 3.5 hours hold to get check nodes | 18:27 |
| cdent | mtreinish: you added the sendcl, do you remember why? https://review.openstack.org/#/c/459451/ | 18:27 |
| melwitt | +Warthogged | 18:27 |
| cdent | it looks like maybe you accidentally left it in | 18:27 |
| cdent | after choosing a different approach | 18:27 |
| sdague | melwitt: thanks | 18:27 |
| sdague | ok, I'll work on follow up fix patch at top of stack | 18:28 |
| * cdent likes warthogs | 18:28 | |
| mtreinish | cdent: my intent there was specifically for the non-glance case. If nova or others gets a chunked request we don't want it to blow up an invalid content length error | 18:28 |
| *** liverpooler has quit IRC | 18:29 | |
| mtreinish | so I wanted sendcl for the mod_proxy_uwsgi case so it would handle those requests properly | 18:29 |
| mtreinish | none of the projects using that path should be taking large chunked uploads over the api (it's only nova, cinder, and placement) | 18:29 |
| mriedem | oh no | 18:30 |
| mriedem | sdague: you've just started two major new ML thread wars https://review.openstack.org/#/c/491834/3/doc/source/index.rst | 18:31 |
| mriedem | sdague: crisis the first: you omitted CONTAINERS! | 18:31 |
| mriedem | crisis the second: OpenSack is a new competing community of cloud hacky sack based software that will kill us all! | 18:31 |
| sdague | mriedem: the only in tree container support is untested libvirt lxc, right? | 18:32 |
| mriedem | virtuozzo | 18:32 |
| *** smatzek has quit IRC | 18:32 | |
| sdague | oh, right | 18:32 |
| mriedem | and lxc | 18:32 |
| mriedem | but yes lxc is likely a floating turd | 18:32 |
| *** krtaylor has joined #openstack-nova | 18:32 | |
| mriedem | tried getting CI running for that several releases ago | 18:32 |
| sdague | yeh, I debated it, someone else can tweak that | 18:32 |
| sdague | my goals, make this page navigable | 18:32 |
| mriedem | "limited container support" is probably ok | 18:33 |
| mriedem | heh, nova-net! https://review.openstack.org/#/c/492105/2/doc/source/figures/nova-conf-xen-flat.png,unified | 18:34 |
| *** tosika has quit IRC | 18:35 | |
| *** rcernin has joined #openstack-nova | 18:35 | |
| *** smatzek has joined #openstack-nova | 18:36 | |
| sdague | yeh, there is all kinds of fun that came over with that import | 18:36 |
| openstackgerrit | Ilya Popov proposed openstack/nova master: Tests: Add cleanup of 'instances' directory https://review.openstack.org/491589 | 18:38 |
| *** rajathagasthya has joined #openstack-nova | 18:38 | |
| mriedem | yeah, i finally know about the EMC system enabler https://review.openstack.org/#/c/492105/2/doc/source/figures/emc-enabler.png,unified | 18:40 |
| *** gbarros has joined #openstack-nova | 18:41 | |
| * cdent waves | 18:41 | |
| *** cdent has quit IRC | 18:41 | |
| mtreinish | cdent: looks like you're right the file is coming from mod_proxy | 18:42 |
| mtreinish | I'll write up a patch to create an isolated virtual host for the glance apache endpoint and hopefully that should isolate the sendcl param | 18:43 |
| openstackgerrit | Sean Dague proposed openstack/nova master: clean ups from I6815958b2533d462a2e5d27e7be57440d9f4f40a https://review.openstack.org/492645 | 18:43 |
| *** Apoorva_ has quit IRC | 18:43 | |
| sdague | mriedem / melwitt ^^^ ok, go to town ripping on that | 18:43 |
| sdague | I'll iterate until we get it good enough | 18:43 |
| *** Apoorva has joined #openstack-nova | 18:44 | |
| *** Sukhdev has joined #openstack-nova | 18:44 | |
| melwitt | cool, thanks | 18:45 |
| *** kfarr has quit IRC | 18:49 | |
| *** psachin has quit IRC | 18:52 | |
| *** lpetrut has joined #openstack-nova | 18:55 | |
| sdague | melwitt: on https://review.openstack.org/#/c/491853 realize that's just the import plus reference corrections to make it build | 18:55 |
| melwitt | sdague: ah, I see that now. I thought the Cells v1 vs v2 warning was new but I see now that it's not | 18:56 |
| sdague | yeh | 18:56 |
| sdague | let us say... the existing docs are a mixed bag. It's probably better to have them in, but I think we're going to need a doc sprint early in the cycle to clean up a bunch of this | 18:56 |
| melwitt | sdague: I think the only other thing not imported is the database config section that's not mentioned as skipped in the commit message. is that one supposed to be skipped? | 18:57 |
| sdague | yeh, it turned into a really goofy thing | 18:57 |
| *** awaugama has quit IRC | 18:57 | |
| melwitt | k | 18:57 |
| sdague | let me pull up a couple of links | 18:57 |
| sdague | the database doc in question was this - https://github.com/openstack/openstack-manuals/blob/6f9fc171800e8a435011f38cd4558e900884ce86/doc/config-reference/source/compute/database-connections.rst#L2 | 18:58 |
| sdague | which besides being kind of inaccurate with the "any sqla" bit, it was also pretty vapid in what it said | 18:59 |
| *** rcernin has quit IRC | 18:59 | |
| sdague | and was mostly a link to https://github.com/openstack/openstack-manuals/blob/6f9fc171800e8a435011f38cd4558e900884ce86/doc/config-reference/source/tables/common-database.rst#L11 | 18:59 |
| melwitt | cool. just making sure it wasn't missed. I'm kind of just skimming this | 18:59 |
| sdague | yeh | 18:59 |
| sdague | the common tables things was kind of a bear | 19:00 |
| sdague | steven spent a day or two fixing references, and I spent another day after him doing that. There was a lot more common normalization in the openstack-manuals repo that had to be denormalized for all of this. | 19:01 |
| *** adisky__ has quit IRC | 19:02 | |
| *** efried is now known as efried_afk | 19:02 | |
| sdague | ok, afk for a bit | 19:03 |
| *** jamesdenton has quit IRC | 19:03 | |
| mtreinish | efried_afk: https://review.openstack.org/#/c/492651/ | 19:03 |
| *** openstackgerrit has quit IRC | 19:03 | |
| mtreinish | sdague: ^^^ that should fix the glance thing | 19:03 |
| *** jamesdenton has joined #openstack-nova | 19:03 | |
| mriedem | sdague: melwitt: ok several comments in the config reference one https://review.openstack.org/#/c/491853/6 | 19:07 |
| mriedem | looks like some are duplicates of mel's | 19:07 |
| melwitt | mriedem: yeah, turns out this is a straight forklift and I didn't realize that at first | 19:07 |
| melwitt | so there's lots of latent issues in it | 19:07 |
| mriedem | heh, | 19:08 |
| mriedem | "The nova-conductor service is the only service that writes to the database." | 19:08 |
| mriedem | "If nova-conductor is not used, entries to the database are mostly written by the nova-scheduler service, although all services must be able to update entries in the database." | 19:08 |
| mriedem | ffs | 19:08 |
| melwitt | noice | 19:08 |
| mriedem | only conductor can write to the database, except everything else can and must be able to also, including your dog | 19:08 |
| melwitt | o local conductor, I barely remember you | 19:08 |
| mriedem | mr ruffles | 19:08 |
| melwitt | heh | 19:09 |
| mriedem | some of the other stuff i pointed out is just plain wrong, like "kvm is the default compute driver" | 19:09 |
| *** sree has joined #openstack-nova | 19:09 | |
| mriedem | "to change compute drivers, configure the [libvirt] section" | 19:09 |
| mriedem | hello libvirt, please to be giving me vmware | 19:09 |
| mriedem | thanks | 19:09 |
| melwitt | as you do | 19:10 |
| mriedem | sdague: i think we should rename that cells.rst to cellsv1.rst at some point | 19:11 |
| mriedem | since it's totally cellsv1 content | 19:11 |
| mriedem | mikal: this is for you https://review.openstack.org/#/c/491853/6/doc/source/admin/configuration/samples/rootwrap.conf.rst@5 | 19:11 |
| *** baoli has quit IRC | 19:13 | |
| *** sree has quit IRC | 19:13 | |
| *** openstackgerrit has joined #openstack-nova | 19:19 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Deprecate bare metal filters https://review.openstack.org/492563 | 19:19 |
| mnaser | jaypipes whenever you have a minute re https://review.openstack.org/#/c/491561/ -- i was thinking about this and realized that all_weighers will now include this which might mean that both ramweigher and rampercentageweigher would both be used (and maybe that might cause some weird scheduling and a change in default behaviour) .. any thoughts (more than welcome anyone else input too) | 19:20 |
| jaypipes | mnaser: hmm... | 19:21 |
| *** nicolasbock has quit IRC | 19:21 | |
| mriedem | ok the other rc1 things are approved, that just leaves https://review.openstack.org/#/c/487954/ | 19:21 |
| jaypipes | mnaser: yeah, that's a good point. keep it in the separate file then... | 19:21 |
| jaypipes | mnaser: mb add a note to the top saying "this is a separate file because..." | 19:22 |
| mnaser | jaypipes i think that even if its a separate file it'll automatically get included with all_weighers :X | 19:22 |
| mnaser | let me find the code for that again | 19:22 |
| mnaser | if i remember right, its not hard coded, used some loadable stuff to autoload all of the stuff in the folder | 19:22 |
| *** gbarros has quit IRC | 19:23 | |
| mnaser | https://github.com/openstack/nova/blob/master/nova/scheduler/weights/__init__.py#L46-L48 => https://github.com/openstack/nova/blob/master/nova/loadables.py#L78-L96 | 19:23 |
| jaypipes | mnaser: ugh... | 19:24 |
| mnaser | the reason this test didnt catch it because it doesnt check for an exact match but "if it contains" https://github.com/openstack/nova/blob/master/nova/tests/unit/scheduler/weights/test_weights_hosts.py | 19:24 |
| mnaser | yeah.. its not pretty | 19:24 |
| jaypipes | mnaser: well, even if it's included in the list of filters, that doesn't necessarily mean it's enabled, right? | 19:25 |
| mnaser | jaypipes this is weigher and afaik there is no enabled or disabled | 19:25 |
| mnaser | jaypipes ouch.. "but wait, there's more" - https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L389-L394 | 19:26 |
| mnaser | we couldnt even drop it from all_weighers lol | 19:26 |
| *** efried_afk is now known as efried | 19:27 | |
| mriedem | weighers are all enabled by default | 19:27 |
| efried | mtreinish Sweet, trying... | 19:27 |
| mriedem | mnaser: you can't set weight_classes ? | 19:28 |
| mriedem | default=["nova.scheduler.weights.all_weighers"], | 19:28 |
| mriedem | mnaser: you should be able to specify the specific list of weighers if you want to exclude some | 19:28 |
| *** kristian__ has joined #openstack-nova | 19:28 | |
| jaypipes | mnaser: yeah, you'd essentially have to list all the other weight classes, minus RamWeigher, and add in RamPercentageWeigher. ugh. | 19:28 |
| mnaser | mriedem - i could, but the idea is when introducing this change -- https://review.openstack.org/#/c/491561/ -- that weigher will become included with the list of default all_weighers | 19:28 |
| mnaser | i have them manually listed and it works fine for me right now | 19:29 |
| mriedem | so you want an excluded_weighers option | 19:29 |
| mriedem | how about, included_by_conditional_on_virt_backend_weighers? :) | 19:29 |
| mriedem | s/by/but./ | 19:29 |
| mnaser | i dont need it, i can manually list it, but if that change merges, the behaviour of all_weighers will change .. which is probably not okay? | 19:29 |
| jaypipes | mriedem: no, he needs to change the behaviour of the existing RamWeigher if, say, a [scheduler]ram_weight_use_percentage configuration option is set... | 19:29 |
| mriedem | ok, so you need a config option to change the behavior of the existing ram weigher, | 19:30 |
| jaypipes | mnaser: right, it's not ok, since both Ram weighers will produce different weights. | 19:30 |
| mriedem | and default it to behave the old way | 19:30 |
| mnaser | thats a good idea | 19:30 |
| jaypipes | yeah | 19:30 |
| mnaser | i can do that | 19:30 |
| mriedem | $5 please | 19:30 |
| jaypipes | mnaser: thanks man. :) | 19:30 |
| mnaser | jaypipes np | 19:30 |
| mnaser | mriedem i'll start a tab | 19:31 |
| mriedem | speaking of, | 19:31 |
| mriedem | craiglist person is coming over in an hour to buy a $10 kids wagon, emailed me asking if i have change for a $20 | 19:31 |
| mriedem | this will be fun | 19:31 |
| mriedem | "i do, if you're ok with singles and $1 coins" | 19:31 |
| mriedem | "here is a bag of nickels" | 19:32 |
| melwitt | mriedem: anything else need review for rc1 that I can look at? | 19:32 |
| mriedem | melwitt: hell yes you can https://review.openstack.org/#/c/487954/ | 19:32 |
| melwitt | sweet | 19:32 |
| *** slaweq has joined #openstack-nova | 19:33 | |
| *** kristian__ has quit IRC | 19:33 | |
| *** xyang1 has joined #openstack-nova | 19:35 | |
| *** slaweq_ has quit IRC | 19:36 | |
| *** kristian__ has joined #openstack-nova | 19:38 | |
| *** sbezverk has joined #openstack-nova | 19:39 | |
| *** baoli has joined #openstack-nova | 19:40 | |
| sdague | mriedem: I agree. I honestly think we should probably rename all the cells v1 stuff to legacy cells actually | 19:41 |
| *** baoli has quit IRC | 19:41 | |
| sdague | especially when dealing with bugs, people don't really understand the v1 v2 thing | 19:41 |
| sdague | legacy cells vs. cells would probably make things clearer | 19:41 |
| * bauzas is back around | 19:42 | |
| sdague | mriedem: can we get ironic folks voting on https://review.openstack.org/#/c/487954 | 19:43 |
| bauzas | do I need to chime in somewhere? | 19:44 |
| sdague | bauzas: I think 487954 is the last patch required for the branch | 19:45 |
| *** baoli has joined #openstack-nova | 19:45 | |
| *** sambetts is now known as sambetts|afk | 19:45 | |
| mriedem | sdague: they were ok with one of the earlier patch sets | 19:47 |
| mriedem | dtantsur|afk: is ok with it i think | 19:47 |
| bauzas | sdague: a bit latey for the gate, nope? | 19:49 |
| bauzas | unless mriedem is a night owl | 19:49 |
| mriedem | bauzas: i'll be up | 19:50 |
| *** kristian__ has quit IRC | 19:50 | |
| sdague | we apparently got more rax nodes too a couple hours ago as well, so we're actually starting to burn down the wait queue | 19:51 |
| *** mingyu has quit IRC | 19:51 | |
| sdague | which is good | 19:51 |
| sdague | at least its not still growing :) | 19:51 |
| bauzas | cool :) | 19:51 |
| *** lpetrut has quit IRC | 19:54 | |
| *** pchavva has quit IRC | 19:57 | |
| *** kristian__ has joined #openstack-nova | 20:00 | |
| *** smatzek has quit IRC | 20:07 | |
| *** baoli has quit IRC | 20:07 | |
| *** awaugama has joined #openstack-nova | 20:15 | |
| *** jaypipes has quit IRC | 20:20 | |
| *** Sukhdev has quit IRC | 20:21 | |
| *** gjayavelu has quit IRC | 20:22 | |
| mriedem | melwitt: are you still going through https://review.openstack.org/#/c/487954/ ? | 20:22 |
| melwitt | mriedem: yes, going through the tests, trying to understand what "dupe" means and how the first dupe test is different from the second | 20:22 |
| *** akhodos has joined #openstack-nova | 20:23 | |
| *** hamzy has quit IRC | 20:24 | |
| *** amotoki has quit IRC | 20:24 | |
| sdague | oof, citycloud lon1 fail again | 20:26 |
| edleafe | melwitt: the first verifies that only one call to save is made, since one instance already has been migrated | 20:26 |
| mnaser | 2017-08-10 20:24:19.406877 | MAGNUM_GUEST_IMAGE_URL='\''http://mirror.mtl01.internap.openstack.org/fedora/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2'\'' | 20:26 |
| mnaser | oops, wrong chan | 20:27 |
| edleafe | the second verifies that if refresh_cache is called a second time, the instances are not saved again | 20:27 |
| melwitt | thanks edleafe | 20:27 |
| edleafe | it has some old code in that second call - I'm correcting that from _refresh_hash_ring() to _refresh_cache() | 20:27 |
| mriedem | sdague: per gmann's doc change, i don't see where this stable api doc is even linked from http://docs-draft.openstack.org/26/489926/9/check/gate-nova-docs-ubuntu-xenial/24e8cc4//doc/build/html/reference/stable-api.html | 20:31 |
| mriedem | like https://developer.openstack.org/api-guide/compute/ | 20:32 |
| mriedem | or http://docs-draft.openstack.org/26/489926/9/check/gate-nova-docs-ubuntu-xenial/24e8cc4//doc/build/html/contributor/index.html#the-nova-api | 20:32 |
| sdague | reference/index | 20:33 |
| sdague | http://docs-draft.openstack.org/26/489926/9/check/gate-nova-docs-ubuntu-xenial/24e8cc4//doc/build/html/reference/index.html | 20:33 |
| mriedem | oh | 20:34 |
| mriedem | sheesh | 20:34 |
| sdague | from the reorged page it would be index -> Technical Reference Deep Dives | 20:34 |
| sdague | -> Nova Stable REST API | 20:34 |
| mriedem | yeah... | 20:34 |
| mriedem | that was a hunt | 20:34 |
| sdague | yeh, we're going to need a slice of time to figure out what the right information architecture is for all this stuff is at PTG | 20:35 |
| sdague | this first stage grouping was just that, a first stage grouping, without creating *more* 404s | 20:35 |
| *** sdake is now known as k2so | 20:37 | |
| sdague | so, the contributor and reference consolidations were first, and I made sub pages for them. But the admin subpage was already claimed as the main url for the admin guide, so I inlined the equivalent user / admin chunks into the top index page | 20:37 |
| sdague | it might be good to unwind all of that for the contributor / reference pages | 20:37 |
| *** kristian__ has quit IRC | 20:38 | |
| sdague | so you get there without the intermediate hop | 20:38 |
| *** yusef has joined #openstack-nova | 20:38 | |
| sdague | I also think we might be able to use refs in the toctree and get something more sane out of all of that | 20:39 |
| *** jamesdenton has quit IRC | 20:39 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954 | 20:41 |
| efried | mtreinish With your fix I did indeed hit https://bugs.launchpad.net/glance/+bug/1703856 | 20:41 |
| openstack | Launchpad bug 1703856 in Glance "502 Bad gateway error on image-create" [High,Confirmed] | 20:41 |
| *** jamesdenton has joined #openstack-nova | 20:43 | |
| *** cleong has quit IRC | 20:43 | |
| efried | mtreinish ...and so did the gate. | 20:45 |
| *** annegentle has quit IRC | 20:45 | |
| edleafe | mriedem: melwitt: ^^ addressed your comments (and your terrible taste in indentation) | 20:46 |
| melwitt | hah | 20:47 |
| *** esberglu has quit IRC | 20:47 | |
| melwitt | +2, looks cool to me | 20:48 |
| melwitt | thanks for changing those test names, makes it a lot clearer to someone not in-the-know | 20:49 |
| *** _pewp_ has quit IRC | 20:50 | |
| *** gouthamr_ has quit IRC | 20:50 | |
| *** _pewp_ has joined #openstack-nova | 20:50 | |
| edleafe | melwitt: yeah, well, they went through a bunch of back-and-forth as people had different opinions on how the migration should work. | 20:51 |
| openstackgerrit | Merged openstack/nova master: update policy UT fixtures https://review.openstack.org/398610 | 20:57 |
| *** elod has quit IRC | 20:58 | |
| openstackgerrit | Merged openstack/nova master: Require Placement 1.10 in nova-status upgrade check https://review.openstack.org/492234 | 20:58 |
| openstackgerrit | Merged openstack/nova master: Add For Operators section to front page https://review.openstack.org/491815 | 20:59 |
| mikal | mriedem: replying now | 20:59 |
| openstackgerrit | Merged openstack/nova master: rework index intro to describe nova https://review.openstack.org/491834 | 20:59 |
| openstackgerrit | Merged openstack/nova master: Bulk import all config reference figures https://review.openstack.org/492105 | 21:00 |
| *** esberglu has joined #openstack-nova | 21:01 | |
| mriedem | if i work from home, and your dog next to my house barks non-stop for 30+ minutes, | 21:01 |
| mriedem | i should be able to do something bad | 21:01 |
| openstackgerrit | Merged openstack/nova master: nova-manage: Deprecate '--version' parameters https://review.openstack.org/453808 | 21:02 |
| *** esberglu_ has joined #openstack-nova | 21:02 | |
| *** jpena|mtg is now known as jpena|off | 21:02 | |
| *** esberglu has quit IRC | 21:02 | |
| openstackgerrit | Merged openstack/nova master: doc: Import configuration reference https://review.openstack.org/491853 | 21:04 |
| openstackgerrit | Merged openstack/nova master: Structure cli page https://review.openstack.org/492111 | 21:04 |
| *** ekuris has quit IRC | 21:04 | |
| *** tosika has joined #openstack-nova | 21:08 | |
| *** krtaylor has quit IRC | 21:08 | |
| *** baoli has joined #openstack-nova | 21:10 | |
| *** egonzalez has quit IRC | 21:11 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: address review comments in stable-api guide updates https://review.openstack.org/492690 | 21:12 |
| mtreinish | efried: hmm, the patch would have fixed the 502 errors. It likely means something is misconfigured elsewhere in the path | 21:12 |
| mtreinish | let me take a look at the gate logs | 21:12 |
| bauzas | mriedem: le woof says hello to your neighbor :p | 21:13 |
| mriedem | god | 21:13 |
| mriedem | i hope le woof isn't as dumb as the neighbor dog | 21:13 |
| openstackgerrit | Robert Ellis proposed openstack/nova master: Clarifying node_uuid usage in ironic driver. https://review.openstack.org/485803 | 21:13 |
| *** mriedem has left #openstack-nova | 21:14 | |
| *** mriedem has quit IRC | 21:14 | |
| *** mriedem has joined #openstack-nova | 21:14 | |
| bauzas | eurasier FTW | 21:14 |
| bauzas | FWIW https://review.openstack.org/#/c/487954 looks good to me, but is the Ironic job working fine ? | 21:15 |
| *** elod has joined #openstack-nova | 21:16 | |
| mriedem | this is just an uncut newfoundland | 21:16 |
| *** suresh12 has quit IRC | 21:16 | |
| mriedem | the last ironic job run failed http://logs.openstack.org/54/487954/13/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/fa780de/console.html | 21:17 |
| mriedem | looks like due to timeout | 21:17 |
| mriedem | PS12 was ok http://logs.openstack.org/54/487954/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/041c03a/ | 21:17 |
| mriedem | PS14 was citycloud-lon1 which is a known slow node issue right now | 21:18 |
| bauzas | how many times is running the refresh cache? | 21:18 |
| bauzas | I mean the period | 21:18 |
| *** suresh12 has joined #openstack-nova | 21:19 | |
| mriedem | bauzas: well, at least every 60 seconds by default because it's called from get_available_nodes which is called from the update_available_resources periodic task | 21:21 |
| bauzas | I'm trying to understand if concurrent runs would be some problems | 21:23 |
| bauzas | looks not | 21:23 |
| bauzas | that said, a question | 21:23 |
| bauzas | if we spawn, do we have the same cache ? | 21:24 |
| bauzas | mriedem: ^ | 21:24 |
| *** gouthamr has joined #openstack-nova | 21:24 | |
| *** thorst has quit IRC | 21:24 | |
| mriedem | i don't understand the question | 21:24 |
| bauzas | we would run multiple greenlets, right? | 21:25 |
| bauzas | so my question is about the node cache | 21:25 |
| bauzas | do we share the same node cache object between greenlets ? | 21:26 |
| mriedem | oh eventlet spawn | 21:26 |
| mriedem | not driver.spawn | 21:27 |
| *** thorst has joined #openstack-nova | 21:27 | |
| *** Sukhdev has joined #openstack-nova | 21:27 | |
| bauzas | yup eventlet.spawn_n even | 21:28 |
| bauzas | maybe it's a stupid question, but I'm not remembering if coroutines accept to just share the same objects | 21:28 |
| *** annegentle has joined #openstack-nova | 21:29 | |
| mriedem | _pike_flavor_migration is passed to the spawn and updates self._migrated_instance_uuids and yes i'd assume that's all pointing back to self | 21:29 |
| mriedem | as the same object | 21:30 |
| mriedem | otherwise that would be crazy | 21:30 |
| *** thorst has quit IRC | 21:31 | |
| mriedem | you see things getting hit in the logs | 21:32 |
| mriedem | http://logs.openstack.org/54/487954/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/041c03a/logs/screen-n-cpu.txt.gz#_Aug_09_19_31_21_252127 | 21:32 |
| mriedem | because the ironic jobs don't yet set resource_class on the nodes | 21:32 |
| bauzas | mriedem: I should explain more my concern | 21:32 |
| bauzas | mriedem: say we have pike flavor migration run that takes more than 60 secs | 21:32 |
| bauzas | then, we would have 2 concurrent migrations | 21:33 |
| bauzas | that looks okay to me because we check the cache | 21:33 |
| bauzas | but the question I wonder is whether all the greenlets share the same cache, so when the first migration updates the cache, the latter gets the updates | 21:33 |
| bauzas | maybe it's pointless, and I'm just silly | 21:34 |
| bauzas | but I'm just thinking out loud | 21:34 |
| mriedem | both greenlets should be working on the same self._migrated_instance_uuids | 21:34 |
| mriedem | melwitt: edleafe: ^? | 21:34 |
| bauzas | mriedem: looking at StackOverflow, looks like yup | 21:35 |
| melwitt | yeah, I think the self._migrated_instance_uuids would be shared between them (if one was still running longer than 60 sec). so maybe we need to synchronize access to that set? | 21:35 |
| mriedem | oh if only we could be using synchronized collections from java! | 21:36 |
| mriedem | what's the worst that would happen here? | 21:37 |
| mriedem | wouldn't we just double migrate the same instance.flavor.extra_spec? | 21:37 |
| melwitt | that's what I was trying to think about. | 21:37 |
| mriedem | if resource_key in specs: | 21:38 |
| mriedem | # The compute must have been restarted, and the instance.flavor | 21:38 |
| mriedem | # has already been migrated | 21:38 |
| mriedem | continue | 21:38 |
| mriedem | ^ should save it | 21:38 |
| bauzas | mriedem: melwitt: well, the more I think about the problem, the more I think it wouldn't be a prolem | 21:38 |
| bauzas | at least because it's for flavors | 21:38 |
| melwitt | yeah | 21:38 |
| bauzas | not sure operators have a lot of flavors needing more than 60 secs for a migration | 21:38 |
| *** baoli has quit IRC | 21:38 | |
| *** baoli has joined #openstack-nova | 21:39 | |
| mriedem | well, unless you're hitting rpc timeouts on sending updates to conductor or something | 21:39 |
| bauzas | and if so, well, not a problem given the current code which is not synchronised but failproof | 21:39 |
| melwitt | it's a good point to think about though. I think mriedem is right that it would skip an already migrated one even if it got a stale view of the shared set | 21:39 |
| melwitt | and set doesn't raise if you add the same element twice, just a no-op | 21:43 |
| mriedem | looks like dtantsur|afk's change in ironic to test this is blowing up during scheduling http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621 | 21:44 |
| mriedem | edleafe: i left a couple of comments/questions in https://review.openstack.org/#/c/487954/ | 21:44 |
| mriedem | hmm, the compute node should be reporting inventory to placement which would include the CUSTOM_BAREMETAL resource class | 21:46 |
| mriedem | http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621 | 21:46 |
| *** gbarros has joined #openstack-nova | 21:48 | |
| *** thorst has joined #openstack-nova | 21:49 | |
| *** krtaylor has joined #openstack-nova | 21:49 | |
| mriedem | looks like we don't get inventory data before that | 21:50 |
| *** xyang1 has quit IRC | 21:54 | |
| *** thorst has quit IRC | 21:54 | |
| mriedem | i'm a bit torn on if we should land the nova change w/o actually seeing this working in the devstack test patch for ironic | 21:55 |
| mriedem | and i have to run right now anyway to pick up maya | 21:55 |
| melwitt | should I remove my vote for now? | 21:56 |
| mriedem | melwitt: no it's ok | 21:56 |
| mriedem | i'm on the fence about just sending it in and then fixing later if there is something wrong | 21:56 |
| melwitt | k | 21:56 |
| mriedem | would be nice if someone from ironic could say, i tested this manually and it's fine | 21:56 |
| mriedem | edleafe: ^ did anyone test manually? | 21:57 |
| edleafe | mriedem: not that I know of | 21:57 |
| *** Sukhdev has quit IRC | 21:57 | |
| mriedem | ok i'll ask in -ironic but i think most of that team is gone for the day | 21:58 |
| mriedem | bbibab | 21:58 |
| mriedem | *bbiab even | 21:58 |
| openstackgerrit | Merged openstack/nova master: Improve stable-api doc with current API state https://review.openstack.org/489926 | 22:01 |
| *** baoli has quit IRC | 22:01 | |
| *** tesseract has quit IRC | 22:02 | |
| *** gjayavelu has joined #openstack-nova | 22:02 | |
| *** Apoorva_ has joined #openstack-nova | 22:05 | |
| *** gjayavelu has quit IRC | 22:05 | |
| *** baoli has joined #openstack-nova | 22:05 | |
| *** ioggstream has joined #openstack-nova | 22:06 | |
| *** priteau has quit IRC | 22:07 | |
| *** kylek3h has quit IRC | 22:07 | |
| *** Apoorva has quit IRC | 22:09 | |
| *** Sukhdev has joined #openstack-nova | 22:09 | |
| *** suresh12 has quit IRC | 22:13 | |
| *** suresh12 has joined #openstack-nova | 22:14 | |
| *** priteau has joined #openstack-nova | 22:15 | |
| bauzas | folks, see you tomorrow | 22:18 |
| bauzas | will look at the branch if any | 22:18 |
| *** openstackgerrit has quit IRC | 22:18 | |
| *** priteau has quit IRC | 22:20 | |
| *** slaweq has quit IRC | 22:23 | |
| edleafe | So it looks like when Ironic sets the resource_class for a node, it doesn't do anything to create that in Placement | 22:27 |
| edleafe | which would explain http://logs.openstack.org/68/476968/12/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial/02053cf/logs/screen-n-sch.txt.gz#_Aug_09_20_35_41_533621 | 22:28 |
| edleafe | mriedem: should I add code to ensure that the custom resource class for a node exists in that migration? I don't see anything in ironic where it is created | 22:33 |
| *** akhodos has quit IRC | 22:33 | |
| kevinbenton | mriedem: kernel panic in VM http://logs.openstack.org/10/488510/33/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/8b65cd3/logs/testr_results.html.gz | 22:33 |
| kevinbenton | mriedem: how often does that happen? | 22:34 |
| *** felipemonteiro has quit IRC | 22:36 | |
| *** rajathagasthya has quit IRC | 22:36 | |
| mriedem | edleafe: no, not in your change. it's used in the scheduler. the custom resource class is created in placement via the periodic updates in the RT | 22:36 |
| mriedem | via the get_inventory() method to the ironic driver | 22:36 |
| edleafe | mriedem: so that's not being run before the test failure above | 22:37 |
| mriedem | edleafe: it's run on start of the compute service | 22:38 |
| mriedem | and in the update_available_resource periodic, | 22:39 |
| mriedem | the problem is we're not reporting any inventory for the node | 22:39 |
| mriedem | so yeah, maybe the problem is a chicken and egg issue, idk | 22:39 |
| mriedem | does the inventory not show up until we have a node that we're tracking with an instance? | 22:39 |
| mriedem | and we don't have the instance w/o the custom resource class that the node is using | 22:39 |
| edleafe | the error isn't that there is no inventory; it's that there is no such resource class | 22:40 |
| mriedem | that interaction is a black box to me right now | 22:40 |
| mriedem | edleafe: because we didn't PUT any inventory | 22:40 |
| mriedem | which would create the custom resource class from the node we're tracking | 22:40 |
| mriedem | the admin could pre-create the custom resource classes, sure | 22:41 |
| mriedem | but nova is also trying to create them if they don't already exist in placement and we have inventory for them | 22:41 |
| mriedem | i'm no baremetal expert though, so i do'nt know the order in which things need to happen here to auto-create the custom resource class | 22:41 |
| edleafe | well, just trying to figure out how to fix this. Gotta run out in a few minutes | 22:41 |
| mriedem | i don't know if there is something to fix on the nova side, | 22:42 |
| mriedem | and i haven't dug into the devstack changes in the ironic WIP patch | 22:42 |
| edleafe | they're selecting a flavor with the custom RC, but placement is barfing on that since it never got created. | 22:42 |
| mriedem | so i might just throw this in rc1 | 22:42 |
| mriedem | and deal with any issues in rc2 | 22:42 |
| edleafe | yeah | 22:42 |
| edleafe | i'm not sure that's the best response from placement | 22:42 |
| mriedem | rather than just not return allocation candidates you mean? | 22:43 |
| edleafe | I mean I understand preventing typos and stuff | 22:43 |
| mriedem | it seems ok, you're asking for allocation_candidates filtering on something which doesn't exist | 22:43 |
| mriedem | and you should probably make sure you can ask for something that's in GET /resource_classes | 22:44 |
| mriedem | it's a client side error somewhere | 22:44 |
| edleafe | ok, so I guess the 400 is correct | 22:44 |
| edleafe | the client can determine how to deal with that | 22:44 |
| *** catintheroof has quit IRC | 22:45 | |
| *** catintheroof has joined #openstack-nova | 22:45 | |
| edleafe | I gotta run - will read the scrollback in the morning | 22:45 |
| *** esberglu_ has quit IRC | 22:47 | |
| *** yusef has quit IRC | 22:48 | |
| *** annegentle has quit IRC | 22:48 | |
| *** annegentle has joined #openstack-nova | 22:49 | |
| *** catintheroof has quit IRC | 22:49 | |
| *** yamamoto has joined #openstack-nova | 22:51 | |
| *** mingyu has joined #openstack-nova | 22:51 | |
| *** annegentle has quit IRC | 22:53 | |
| *** edmondsw has quit IRC | 22:53 | |
| *** MVenesio has quit IRC | 22:54 | |
| *** mingyu has quit IRC | 22:55 | |
| *** rajathagasthya has joined #openstack-nova | 22:57 | |
| *** annegentle has joined #openstack-nova | 23:05 | |
| *** Swami has quit IRC | 23:05 | |
| *** gouthamr has quit IRC | 23:12 | |
| *** awaugama has quit IRC | 23:12 | |
| *** annegentle has quit IRC | 23:13 | |
| *** chyka has quit IRC | 23:15 | |
| *** hongbin has quit IRC | 23:21 | |
| *** lyan has quit IRC | 23:22 | |
| *** openstackgerrit has joined #openstack-nova | 23:22 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove "dhcp_options_for_instance" https://review.openstack.org/492707 | 23:22 |
| mtreinish | efried: ok try: https://review.openstack.org/#/c/492651/ again | 23:23 |
| mtreinish | I had to start playing with config settings locally and I think devstack should match the config which worked for me locally | 23:23 |
| *** slaweq has joined #openstack-nova | 23:23 | |
| *** suresh12 has quit IRC | 23:26 | |
| *** suresh12 has joined #openstack-nova | 23:26 | |
| *** slaweq has quit IRC | 23:28 | |
| efried | mtreinish ack. | 23:30 |
| efried | mtreinish Just got restacked with mod_wsgi, though, so I may push it out a bit. | 23:31 |
| *** ioggstream has quit IRC | 23:31 | |
| *** yamamoto has quit IRC | 23:33 | |
| *** claudiub has quit IRC | 23:35 | |
| *** Swami has joined #openstack-nova | 23:35 | |
| *** rajathagasthya has quit IRC | 23:36 | |
| *** gyee has quit IRC | 23:36 | |
| *** gyee has joined #openstack-nova | 23:37 | |
| *** thorst has joined #openstack-nova | 23:38 | |
| *** yamamoto has joined #openstack-nova | 23:40 | |
| *** sdague has quit IRC | 23:47 | |
| mtreinish | efried: no worries, I don't think it works anyway. It's still failing the gate | 23:48 |
| mtreinish | I'll have to play with it some more tomorrow | 23:48 |
| *** suresh12 has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
