| *** suresh12 has joined #openstack-nova | 00:00 | |
| *** hemna_ has quit IRC | 00:05 | |
| *** gmann has quit IRC | 00:18 | |
| *** gmann has joined #openstack-nova | 00:18 | |
| *** hoonetorg has quit IRC | 00:23 | |
| *** sbezverk has quit IRC | 00:24 | |
| *** tetsuro has joined #openstack-nova | 00:24 | |
| *** gjayavelu has joined #openstack-nova | 00:33 | |
| *** randomhack has joined #openstack-nova | 00:33 | |
| *** kiennt has joined #openstack-nova | 00:34 | |
| *** thorst has joined #openstack-nova | 00:39 | |
| *** mdnadeem has joined #openstack-nova | 00:40 | |
| *** mingyu has joined #openstack-nova | 00:41 | |
| *** mdnadeem has quit IRC | 00:41 | |
| *** mdnadeem has joined #openstack-nova | 00:41 | |
| *** thorst has quit IRC | 00:44 | |
| *** gouthamr has quit IRC | 00:48 | |
| *** owalsh has quit IRC | 00:48 | |
| *** jpena|off is now known as jpena | 00:49 | |
| *** Shunli has joined #openstack-nova | 00:49 | |
| *** owalsh has joined #openstack-nova | 00:50 | |
| *** randomhack has quit IRC | 00:53 | |
| *** owalsh_ has joined #openstack-nova | 00:54 | |
| *** owalsh has quit IRC | 00:56 | |
| *** vishwana_ has joined #openstack-nova | 00:57 | |
| *** owalsh_ is now known as owalsh | 00:58 | |
| *** larainema has quit IRC | 00:58 | |
| *** mingyu_ has joined #openstack-nova | 00:58 | |
| *** baoli_ has joined #openstack-nova | 00:58 | |
| *** thorst has joined #openstack-nova | 00:58 | |
| *** mingyu has quit IRC | 00:59 | |
| *** baoli has quit IRC | 01:00 | |
| *** phuongnh has joined #openstack-nova | 01:01 | |
| *** vishwanathj has quit IRC | 01:01 | |
| *** litao__ has joined #openstack-nova | 01:01 | |
| mriedem | dansmith: fyi, looks like we're hitting some slow nodes or something where n-cpu is auto-disabled by the servicegroup heartbeat because the report comes in late | 01:02 |
|---|---|---|
| mriedem | http://logs.openstack.org/39/491439/3/gate/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/d5c3c02/logs/subnode-2/screen-n-cpu.txt.gz#_Aug_08_20_55_49_474796 | 01:02 |
| dansmith | seriously? | 01:03 |
| dansmith | that's some crazy delay | 01:03 |
| mriedem | sometihng like that, only seeing it on live migration jobs | 01:03 |
| mriedem | they fail b/c the compute service is not available | 01:03 |
| mriedem | clarkb: ^ | 01:04 |
| mriedem | it's by far and away on citycloud-lon1 | 01:04 |
| mriedem | lots of HTTPReadTimeoutErrors in the console log too | 01:06 |
| *** hongbin has joined #openstack-nova | 01:06 | |
| dansmith | yeah | 01:06 |
| *** markvoelker has joined #openstack-nova | 01:07 | |
| *** claudiub has quit IRC | 01:08 | |
| dansmith | mriedem: well if it's clearly a bad node or something I assume the remedy is just pulling it out right? | 01:08 |
| mriedem | sure | 01:09 |
| mriedem | i leave it to the infra wizards | 01:09 |
| mriedem | https://bugs.launchpad.net/openstack-gate/+bug/1709506 | 01:09 |
| openstack | Launchpad bug 1709506 in OpenStack-Gate "Random live migration failures due to ComputeServiceUnavailable in citycloud-lon1 nodes" [Undecided,New] | 01:09 |
| dansmith | okay just wanted to make sure I can go back to my dinner :) | 01:09 |
| mriedem | yup | 01:09 |
| dansmith | sweet | 01:09 |
| mriedem | it also helped me notice that we're not indexing the super-cond/cond-cell1 logs in logstash | 01:10 |
| mriedem | fix is up for that | 01:10 |
| dansmith | oh interesting | 01:10 |
| mriedem | system-config repo has a yaml of the files to index | 01:10 |
| *** trungnv has quit IRC | 01:10 | |
| mriedem | with the new names we didn't account for those | 01:11 |
| *** yangyapeng has joined #openstack-nova | 01:14 | |
| *** kiennt has quit IRC | 01:15 | |
| *** gjayavelu has quit IRC | 01:17 | |
| *** yamamoto has quit IRC | 01:19 | |
| clarkb | mriedem: ya we emailed them today | 01:19 |
| clarkb | then being citycloud | 01:19 |
| *** zhurong has joined #openstack-nova | 01:26 | |
| *** yamamoto has joined #openstack-nova | 01:26 | |
| *** owalsh_ has joined #openstack-nova | 01:26 | |
| mriedem | cool | 01:27 |
| *** owalsh- has joined #openstack-nova | 01:30 | |
| *** owalsh has quit IRC | 01:30 | |
| *** owalsh- is now known as owalsh | 01:30 | |
| *** gcb has quit IRC | 01:32 | |
| *** owalsh_ has quit IRC | 01:33 | |
| *** awaugama has quit IRC | 01:35 | |
| *** Apoorva_ has joined #openstack-nova | 01:37 | |
| *** Apoorva has quit IRC | 01:40 | |
| *** Apoorva_ has quit IRC | 01:42 | |
| *** gongysh has joined #openstack-nova | 01:44 | |
| *** jpena is now known as jpena|off | 01:44 | |
| *** kiennt has joined #openstack-nova | 01:45 | |
| *** randomhack has joined #openstack-nova | 01:46 | |
| *** gcb has joined #openstack-nova | 01:46 | |
| *** trungnv has joined #openstack-nova | 01:53 | |
| *** esberglu has quit IRC | 01:56 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Mark max microversion for Pike in history doc https://review.openstack.org/491581 | 01:56 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add release note for shared storage known issue https://review.openstack.org/491582 | 01:56 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add a prelude section for Pike https://review.openstack.org/491424 | 01:56 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: provide more details on scheduling with placement https://review.openstack.org/491900 | 01:56 |
| *** esberglu has joined #openstack-nova | 01:57 | |
| *** suresh12 has quit IRC | 01:58 | |
| *** esberglu has quit IRC | 01:58 | |
| *** esberglu has joined #openstack-nova | 01:58 | |
| mriedem | stephenfin: can you drop this -2 now? https://review.openstack.org/#/c/482216/ | 02:01 |
| *** mdnadeem has quit IRC | 02:02 | |
| *** randomhack has quit IRC | 02:03 | |
| *** mriedem has quit IRC | 02:04 | |
| *** mdnadeem has joined #openstack-nova | 02:12 | |
| *** trungnv has quit IRC | 02:15 | |
| *** thorst has quit IRC | 02:16 | |
| *** trungnv has joined #openstack-nova | 02:18 | |
| *** sree has joined #openstack-nova | 02:20 | |
| *** trungnv has quit IRC | 02:21 | |
| *** trungnv has joined #openstack-nova | 02:21 | |
| *** saphi has joined #openstack-nova | 02:23 | |
| *** sree has quit IRC | 02:24 | |
| *** saphi has quit IRC | 02:24 | |
| *** saphi has joined #openstack-nova | 02:25 | |
| *** markvoelker has quit IRC | 02:30 | |
| *** annegentle has quit IRC | 02:33 | |
| *** markvoelker has joined #openstack-nova | 02:33 | |
| *** annegentle has joined #openstack-nova | 02:41 | |
| *** tuanluong has joined #openstack-nova | 02:41 | |
| *** esberglu has quit IRC | 02:43 | |
| *** zhurong has quit IRC | 02:44 | |
| *** larainema has joined #openstack-nova | 02:55 | |
| *** zhurong has joined #openstack-nova | 03:00 | |
| openstackgerrit | Merged openstack/nova master: Add functional test for local delete allocations https://review.openstack.org/470578 | 03:01 |
| *** sbezverk has joined #openstack-nova | 03:06 | |
| *** yamahata has quit IRC | 03:06 | |
| *** markvoelker has quit IRC | 03:07 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Live migration failed in xapi pool https://review.openstack.org/489451 | 03:10 |
| *** esberglu has joined #openstack-nova | 03:13 | |
| *** links has joined #openstack-nova | 03:15 | |
| *** annegentle has quit IRC | 03:15 | |
| *** thorst has joined #openstack-nova | 03:17 | |
| *** esberglu has quit IRC | 03:17 | |
| *** sbezverk has quit IRC | 03:18 | |
| *** mmehan has quit IRC | 03:18 | |
| *** thorst has quit IRC | 03:21 | |
| *** dave-mccowan has quit IRC | 03:28 | |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs to Instance object https://review.openstack.org/489408 | 03:28 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | 03:29 |
| *** baoli_ has quit IRC | 03:32 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: placement: ensure RP maps to those RPs that share with it https://review.openstack.org/480379 | 03:34 |
| *** coreywright has quit IRC | 03:34 | |
| *** ratailor has joined #openstack-nova | 03:37 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229 | 03:46 |
| openstackgerrit | Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166 | 03:46 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move execs of touch to privsep. https://review.openstack.org/489190 | 03:46 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move libvirts dmcrypt support to privsep. https://review.openstack.org/490737 | 03:46 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438 | 03:46 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972 | 03:46 |
| openstackgerrit | Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486 | 03:46 |
| openstackgerrit | Michael Still proposed openstack/nova master: Refactor libvirt.utils.execute() away. https://review.openstack.org/489816 | 03:46 |
| *** markvoelker_ has joined #openstack-nova | 03:49 | |
| *** jichen has joined #openstack-nova | 03:51 | |
| *** coreywright has joined #openstack-nova | 03:52 | |
| *** adisky__ has joined #openstack-nova | 03:52 | |
| *** markvoelker_ has quit IRC | 03:54 | |
| *** nicolasbock has joined #openstack-nova | 03:57 | |
| *** saphi has quit IRC | 03:57 | |
| *** suresh12 has joined #openstack-nova | 03:59 | |
| *** gongysh has quit IRC | 04:03 | |
| *** suresh12 has quit IRC | 04:03 | |
| *** suzhengwei has quit IRC | 04:10 | |
| *** saphi has joined #openstack-nova | 04:13 | |
| *** hongbin has quit IRC | 04:14 | |
| *** mamandle has joined #openstack-nova | 04:15 | |
| *** zhurong has quit IRC | 04:19 | |
| *** gcb has quit IRC | 04:19 | |
| *** zhurong has joined #openstack-nova | 04:20 | |
| *** gcb has joined #openstack-nova | 04:20 | |
| *** sree has joined #openstack-nova | 04:21 | |
| *** slaweq has joined #openstack-nova | 04:25 | |
| *** sbezverk has joined #openstack-nova | 04:26 | |
| *** gbarros has quit IRC | 04:26 | |
| *** hareesh has joined #openstack-nova | 04:29 | |
| *** coreywright has quit IRC | 04:29 | |
| *** slaweq has quit IRC | 04:30 | |
| *** diga has joined #openstack-nova | 04:30 | |
| *** suzhengwei has joined #openstack-nova | 04:32 | |
| *** gongysh has joined #openstack-nova | 04:39 | |
| *** mingyu_ has quit IRC | 04:46 | |
| *** sbezverk has quit IRC | 04:46 | |
| *** coreywright has joined #openstack-nova | 04:48 | |
| *** suzhengwei has quit IRC | 04:50 | |
| *** randomhack has joined #openstack-nova | 05:00 | |
| *** randomhack has quit IRC | 05:04 | |
| *** armax has quit IRC | 05:06 | |
| *** markvoelker has joined #openstack-nova | 05:06 | |
| *** gyee has quit IRC | 05:08 | |
| *** claudiub has joined #openstack-nova | 05:13 | |
| *** shan has joined #openstack-nova | 05:14 | |
| *** trinaths has joined #openstack-nova | 05:15 | |
| *** suzhengwei has joined #openstack-nova | 05:15 | |
| *** thorst has joined #openstack-nova | 05:17 | |
| *** yamamoto has quit IRC | 05:19 | |
| *** mingyu has joined #openstack-nova | 05:21 | |
| *** thorst has quit IRC | 05:22 | |
| *** psachin has joined #openstack-nova | 05:24 | |
| *** suresh12 has joined #openstack-nova | 05:25 | |
| *** slaweq has joined #openstack-nova | 05:26 | |
| *** mingyu has quit IRC | 05:27 | |
| *** mingyu has joined #openstack-nova | 05:29 | |
| *** slaweq has quit IRC | 05:31 | |
| *** yamamoto has joined #openstack-nova | 05:31 | |
| *** markvoelker has quit IRC | 05:32 | |
| *** markvoelker has joined #openstack-nova | 05:32 | |
| *** mingyu has quit IRC | 05:34 | |
| *** kiennt has quit IRC | 05:43 | |
| *** esberglu has joined #openstack-nova | 05:44 | |
| *** esberglu has quit IRC | 05:49 | |
| *** rcernin has joined #openstack-nova | 05:52 | |
| *** kiennt has joined #openstack-nova | 05:53 | |
| *** mingyu has joined #openstack-nova | 05:56 | |
| *** yamamoto_ has joined #openstack-nova | 06:05 | |
| *** moshele has joined #openstack-nova | 06:07 | |
| *** hoonetorg has joined #openstack-nova | 06:07 | |
| *** yamamoto has quit IRC | 06:09 | |
| *** Oku_OS-away is now known as Oku_OS | 06:15 | |
| *** udesale has joined #openstack-nova | 06:17 | |
| *** slaweq has joined #openstack-nova | 06:33 | |
| *** mamandle has quit IRC | 06:34 | |
| *** tesseract has joined #openstack-nova | 06:35 | |
| *** slaweq has quit IRC | 06:35 | |
| *** edmondsw has joined #openstack-nova | 06:35 | |
| *** edmondsw has quit IRC | 06:38 | |
| *** sridharg has joined #openstack-nova | 06:52 | |
| *** mingyu has quit IRC | 06:52 | |
| *** mingyu has joined #openstack-nova | 06:53 | |
| openstackgerrit | huangtianhua proposed openstack/python-novaclient master: Allow boot server with multiple nics https://review.openstack.org/492003 | 06:59 |
| *** cfriesen has quit IRC | 06:59 | |
| *** markvoelker has quit IRC | 07:03 | |
| *** markvoelker has joined #openstack-nova | 07:06 | |
| *** vks1 has joined #openstack-nova | 07:07 | |
| *** markvoelker has quit IRC | 07:09 | |
| *** sshwarts has joined #openstack-nova | 07:10 | |
| *** sahid has joined #openstack-nova | 07:10 | |
| *** vks1 has quit IRC | 07:12 | |
| *** mingyu has quit IRC | 07:13 | |
| openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Remove host filter for _cleanup_running_deleted_instances periodic task https://review.openstack.org/491808 | 07:24 |
| *** pcaruana has joined #openstack-nova | 07:25 | |
| *** suresh12 has quit IRC | 07:26 | |
| *** markus_z has joined #openstack-nova | 07:31 | |
| *** sahid has quit IRC | 07:34 | |
| *** udesale__ has joined #openstack-nova | 07:38 | |
| *** udesale has quit IRC | 07:39 | |
| *** vks1 has joined #openstack-nova | 07:39 | |
| *** mamandle has joined #openstack-nova | 07:40 | |
| *** mingyu has joined #openstack-nova | 07:40 | |
| *** sahid has joined #openstack-nova | 07:41 | |
| *** thorst has joined #openstack-nova | 07:42 | |
| *** thorst has quit IRC | 07:44 | |
| *** udesale has joined #openstack-nova | 07:51 | |
| *** mlakat has quit IRC | 07:51 | |
| *** udesale__ has quit IRC | 07:51 | |
| *** moshele has quit IRC | 07:52 | |
| *** slaweq has joined #openstack-nova | 07:52 | |
| *** alexchadin has joined #openstack-nova | 07:54 | |
| *** esberglu has joined #openstack-nova | 07:55 | |
| *** slaweq has quit IRC | 07:55 | |
| *** slaweq has joined #openstack-nova | 08:14 | |
| *** esberglu has quit IRC | 08:16 | |
| *** mlakat has joined #openstack-nova | 08:16 | |
| openstackgerrit | huangtianhua proposed openstack/python-novaclient master: Remove substitutions for command error msg https://review.openstack.org/490705 | 08:16 |
| gibi | good morning | 08:34 |
| *** ralonsoh has joined #openstack-nova | 08:34 | |
| *** zsli_ has joined #openstack-nova | 08:44 | |
| *** mingyu has quit IRC | 08:44 | |
| *** mwhahaha has quit IRC | 08:47 | |
| *** zioproto has quit IRC | 08:47 | |
| *** TheJulia has quit IRC | 08:47 | |
| *** betherly has quit IRC | 08:47 | |
| *** slaweq has quit IRC | 08:47 | |
| *** sridharg has quit IRC | 08:47 | |
| *** psachin has quit IRC | 08:47 | |
| *** ratailor has quit IRC | 08:47 | |
| *** tuanluong has quit IRC | 08:47 | |
| *** mdnadeem has quit IRC | 08:47 | |
| *** vishwana_ has quit IRC | 08:47 | |
| *** vladikr has quit IRC | 08:47 | |
| *** _pewp_ has quit IRC | 08:47 | |
| *** Dinesh_Bhor has quit IRC | 08:47 | |
| *** thingee has quit IRC | 08:47 | |
| *** fnordahl has quit IRC | 08:47 | |
| *** mchiappe1o has quit IRC | 08:47 | |
| *** cburgess has quit IRC | 08:47 | |
| *** antonym has quit IRC | 08:47 | |
| *** sree has quit IRC | 08:47 | |
| *** gcb has quit IRC | 08:47 | |
| *** larainema has quit IRC | 08:47 | |
| *** trungnv has quit IRC | 08:47 | |
| *** owalsh has quit IRC | 08:47 | |
| *** krtaylor has quit IRC | 08:47 | |
| *** thomasem- has quit IRC | 08:47 | |
| *** dutsmoc has quit IRC | 08:47 | |
| *** leifz has quit IRC | 08:47 | |
| *** asettle has quit IRC | 08:47 | |
| *** fmccrthy has quit IRC | 08:47 | |
| *** stvnoyes has quit IRC | 08:47 | |
| *** dims has quit IRC | 08:47 | |
| *** brault has quit IRC | 08:47 | |
| *** BlackDex has quit IRC | 08:47 | |
| *** jamielennox has quit IRC | 08:47 | |
| *** yankcrime has quit IRC | 08:47 | |
| *** dgonzalez has quit IRC | 08:47 | |
| *** ttx has quit IRC | 08:47 | |
| *** andymccr has quit IRC | 08:47 | |
| *** aignatov has quit IRC | 08:47 | |
| *** aries has quit IRC | 08:47 | |
| *** ltomasbo has quit IRC | 08:47 | |
| *** mmedvede has quit IRC | 08:47 | |
| *** ralonsoh has quit IRC | 08:47 | |
| *** vks1 has quit IRC | 08:47 | |
| *** yamamoto_ has quit IRC | 08:47 | |
| *** jichen has quit IRC | 08:47 | |
| *** abalutoiu has quit IRC | 08:47 | |
| *** hamzy has quit IRC | 08:47 | |
| *** maciejjozefczyk has quit IRC | 08:47 | |
| *** ykuo_ has quit IRC | 08:47 | |
| *** isq_ has quit IRC | 08:47 | |
| *** masber has quit IRC | 08:47 | |
| *** dosaboy has quit IRC | 08:47 | |
| *** kaisers2 has quit IRC | 08:47 | |
| *** brad[] has quit IRC | 08:47 | |
| *** gibi has quit IRC | 08:47 | |
| *** sgordon has quit IRC | 08:47 | |
| *** markmc has quit IRC | 08:47 | |
| *** ekhugen_alt has quit IRC | 08:47 | |
| *** devananda has quit IRC | 08:47 | |
| *** EmilienM has quit IRC | 08:47 | |
| *** syjulian has quit IRC | 08:47 | |
| *** zigo has quit IRC | 08:47 | |
| *** udesale has quit IRC | 08:47 | |
| *** mamandle has quit IRC | 08:47 | |
| *** tesseract has quit IRC | 08:47 | |
| *** yangyapeng has quit IRC | 08:47 | |
| *** jamesdenton has quit IRC | 08:47 | |
| *** itlinux has quit IRC | 08:47 | |
| *** jowisz has quit IRC | 08:47 | |
| *** rha has quit IRC | 08:47 | |
| *** Dave has quit IRC | 08:47 | |
| *** csuttles has quit IRC | 08:47 | |
| *** alezil has quit IRC | 08:47 | |
| *** tinwood has quit IRC | 08:47 | |
| *** sapcc-bot has quit IRC | 08:47 | |
| *** s1061123 has quit IRC | 08:47 | |
| *** hughsaunders has quit IRC | 08:47 | |
| *** csatari has quit IRC | 08:47 | |
| *** DuncanT has quit IRC | 08:47 | |
| *** jcook has quit IRC | 08:47 | |
| *** dulek has quit IRC | 08:47 | |
| *** mlakat has quit IRC | 08:47 | |
| *** gongysh has quit IRC | 08:47 | |
| *** karthiks has quit IRC | 08:47 | |
| *** zz9pzza has quit IRC | 08:47 | |
| *** aarefiev has quit IRC | 08:47 | |
| *** reedip has quit IRC | 08:47 | |
| *** mfisch` has quit IRC | 08:47 | |
| *** htruta` has quit IRC | 08:47 | |
| *** sambetts|afk has quit IRC | 08:47 | |
| *** inara has quit IRC | 08:47 | |
| *** ababich has quit IRC | 08:47 | |
| *** tristanC has quit IRC | 08:47 | |
| *** rhagarty has quit IRC | 08:47 | |
| *** frickler has quit IRC | 08:47 | |
| *** ansiwen has quit IRC | 08:47 | |
| *** toabctl has quit IRC | 08:47 | |
| *** jeblair has quit IRC | 08:47 | |
| *** tonyb has quit IRC | 08:47 | |
| *** johnthetubaguy has quit IRC | 08:47 | |
| *** lyarwood has quit IRC | 08:47 | |
| *** mhenkel has quit IRC | 08:47 | |
| *** ericyoung has quit IRC | 08:47 | |
| *** alaski has quit IRC | 08:47 | |
| *** zeroDivisible has quit IRC | 08:47 | |
| *** keekz has quit IRC | 08:47 | |
| *** bradjones has quit IRC | 08:47 | |
| *** rtjure has quit IRC | 08:47 | |
| *** sergek_ has quit IRC | 08:47 | |
| *** jpena|off has quit IRC | 08:47 | |
| *** d34dh0r53 has quit IRC | 08:47 | |
| *** d9k has quit IRC | 08:47 | |
| *** lucas-afk has quit IRC | 08:47 | |
| *** redondo-mk has quit IRC | 08:47 | |
| *** obre has quit IRC | 08:47 | |
| *** greghaynes has quit IRC | 08:47 | |
| *** ildikov has quit IRC | 08:47 | |
| *** dtantsur|afk has quit IRC | 08:47 | |
| *** tommylikehu has quit IRC | 08:47 | |
| *** geekinutah has quit IRC | 08:47 | |
| *** sulo has quit IRC | 08:47 | |
| *** StevenK has quit IRC | 08:47 | |
| *** kencjohnston_ has quit IRC | 08:47 | |
| *** andrewbogott has quit IRC | 08:47 | |
| *** zul has quit IRC | 08:47 | |
| *** Hazelesque has quit IRC | 08:47 | |
| *** ejat has quit IRC | 08:47 | |
| *** belliott_ has quit IRC | 08:47 | |
| *** rcernin has quit IRC | 08:47 | |
| *** links has quit IRC | 08:47 | |
| *** ircuser-1 has quit IRC | 08:47 | |
| *** bswartz has quit IRC | 08:47 | |
| *** jaosorior has quit IRC | 08:47 | |
| *** bhagyashris has quit IRC | 08:47 | |
| *** liusheng has quit IRC | 08:47 | |
| *** haukebruno has quit IRC | 08:47 | |
| *** mnaser has quit IRC | 08:47 | |
| *** weshay_ has quit IRC | 08:47 | |
| *** shiyer has quit IRC | 08:47 | |
| *** shaohe_feng has quit IRC | 08:47 | |
| *** edleafe has quit IRC | 08:47 | |
| *** med_ has quit IRC | 08:47 | |
| *** alex_xu has quit IRC | 08:47 | |
| *** jogo has quit IRC | 08:47 | |
| *** raginbajin has quit IRC | 08:47 | |
| *** kuzko has quit IRC | 08:47 | |
| *** mdbooth has quit IRC | 08:47 | |
| *** larsks has quit IRC | 08:47 | |
| *** migi has quit IRC | 08:47 | |
| *** Tahvok has quit IRC | 08:47 | |
| *** pcaruana has quit IRC | 08:47 | |
| *** kiennt has quit IRC | 08:47 | |
| *** claudiub has quit IRC | 08:47 | |
| *** chohoor has quit IRC | 08:47 | |
| *** Qiming has quit IRC | 08:47 | |
| *** mgariepy has quit IRC | 08:47 | |
| *** serverascode has quit IRC | 08:47 | |
| *** junbo has quit IRC | 08:47 | |
| *** s-dean has quit IRC | 08:47 | |
| *** oomichi has quit IRC | 08:47 | |
| *** robcresswell has quit IRC | 08:47 | |
| *** elod has quit IRC | 08:47 | |
| *** tyrefors has quit IRC | 08:47 | |
| *** MikeG451 has quit IRC | 08:47 | |
| *** scottda has quit IRC | 08:47 | |
| *** carl_baldwin has quit IRC | 08:47 | |
| *** patrickeast has quit IRC | 08:47 | |
| *** ameade has quit IRC | 08:47 | |
| *** samueldmq has quit IRC | 08:47 | |
| *** cargonza has quit IRC | 08:47 | |
| *** icey has quit IRC | 08:47 | |
| *** luos has quit IRC | 08:47 | |
| *** MasterofJOKers has quit IRC | 08:47 | |
| *** mgagne has quit IRC | 08:47 | |
| *** gus has quit IRC | 08:47 | |
| *** fungi has quit IRC | 08:47 | |
| *** tanee has quit IRC | 08:47 | |
| *** tomhambleton_ has quit IRC | 08:47 | |
| *** rushiagr has quit IRC | 08:47 | |
| *** johnsom has quit IRC | 08:47 | |
| *** vdrok has quit IRC | 08:47 | |
| *** fyxim has quit IRC | 08:47 | |
| *** toan has quit IRC | 08:47 | |
| *** raorn has quit IRC | 08:47 | |
| *** karlamrhein has quit IRC | 08:47 | |
| *** NobodyCam has quit IRC | 08:47 | |
| *** wolsen has quit IRC | 08:47 | |
| *** aweeks has quit IRC | 08:47 | |
| *** markus_z has quit IRC | 08:47 | |
| *** sshwarts has quit IRC | 08:47 | |
| *** trinaths has quit IRC | 08:47 | |
| *** litao__ has quit IRC | 08:47 | |
| *** rodrigods has quit IRC | 08:47 | |
| *** jdillaman has quit IRC | 08:47 | |
| *** sdake_ has quit IRC | 08:47 | |
| *** markmcclain has quit IRC | 08:47 | |
| *** efried has quit IRC | 08:47 | |
| *** danpawlik has quit IRC | 08:47 | |
| *** artom has quit IRC | 08:47 | |
| *** clarkb has quit IRC | 08:47 | |
| *** anthonyper has quit IRC | 08:47 | |
| *** andreaf has quit IRC | 08:47 | |
| *** FL1SK has quit IRC | 08:47 | |
| *** knikolla has quit IRC | 08:47 | |
| *** rm_work has quit IRC | 08:47 | |
| *** NostawRm has quit IRC | 08:47 | |
| *** mrhillsman has quit IRC | 08:47 | |
| *** swamireddy has quit IRC | 08:47 | |
| *** jhesketh has quit IRC | 08:47 | |
| *** shan has quit IRC | 08:47 | |
| *** mikal has quit IRC | 08:47 | |
| *** spotz has quit IRC | 08:47 | |
| *** hferenc has quit IRC | 08:47 | |
| *** sean-k-mooney has quit IRC | 08:47 | |
| *** strigazi has quit IRC | 08:47 | |
| *** purplerbot has quit IRC | 08:47 | |
| *** jlvillal has quit IRC | 08:47 | |
| *** kevinbenton has quit IRC | 08:47 | |
| *** Daviey has quit IRC | 08:47 | |
| *** dansmith has quit IRC | 08:47 | |
| *** kazsh has quit IRC | 08:47 | |
| *** jamiec has quit IRC | 08:47 | |
| *** doffm has quit IRC | 08:47 | |
| *** tobasco has quit IRC | 08:47 | |
| *** jgriffith has quit IRC | 08:47 | |
| *** rnoriega has quit IRC | 08:47 | |
| *** bnemec has quit IRC | 08:47 | |
| *** adreznec has quit IRC | 08:47 | |
| *** john51 has quit IRC | 08:47 | |
| *** Jeffrey4l has quit IRC | 08:47 | |
| *** alexchadin has quit IRC | 08:47 | |
| *** jmlowe has quit IRC | 08:47 | |
| *** kornicameister has quit IRC | 08:47 | |
| *** lennyb has quit IRC | 08:47 | |
| *** wxy has quit IRC | 08:47 | |
| *** hwoarang has quit IRC | 08:47 | |
| *** mdrabe has quit IRC | 08:47 | |
| *** stephenfin has quit IRC | 08:47 | |
| *** shaner_ has quit IRC | 08:47 | |
| *** wasmum has quit IRC | 08:47 | |
| *** khappone has quit IRC | 08:47 | |
| *** mordred has quit IRC | 08:47 | |
| *** afazekas has quit IRC | 08:47 | |
| *** ebbex has quit IRC | 08:47 | |
| *** hemna has quit IRC | 08:47 | |
| *** McNinja has quit IRC | 08:47 | |
| *** morgan has quit IRC | 08:47 | |
| *** Gorian has quit IRC | 08:47 | |
| *** test222 has quit IRC | 08:47 | |
| *** jamespage has quit IRC | 08:47 | |
| *** rmcadams has quit IRC | 08:47 | |
| *** auggy has quit IRC | 08:47 | |
| *** jbryce has quit IRC | 08:47 | |
| *** john5223 has quit IRC | 08:47 | |
| *** zsli_ has quit IRC | 08:47 | |
| *** suzhengwei has quit IRC | 08:47 | |
| *** coreywright has quit IRC | 08:47 | |
| *** zhurong has quit IRC | 08:47 | |
| *** saphi has quit IRC | 08:47 | |
| *** nicolasbock has quit IRC | 08:47 | |
| *** phuongnh has quit IRC | 08:47 | |
| *** Shunli has quit IRC | 08:47 | |
| *** gmann has quit IRC | 08:47 | |
| *** kylek3h has quit IRC | 08:47 | |
| *** xinliang has quit IRC | 08:47 | |
| *** clayton has quit IRC | 08:47 | |
| *** tojuvone has quit IRC | 08:47 | |
| *** david-lyle has quit IRC | 08:47 | |
| *** lifeless has quit IRC | 08:47 | |
| *** jistr has quit IRC | 08:47 | |
| *** openstackgerrit has quit IRC | 08:47 | |
| *** aloga has quit IRC | 08:47 | |
| *** rdo_ has quit IRC | 08:47 | |
| *** huangtianhua has quit IRC | 08:47 | |
| *** dr_gogeta86 has quit IRC | 08:47 | |
| *** Guest57787 has quit IRC | 08:47 | |
| *** mtreinish has quit IRC | 08:47 | |
| *** lbragstad has quit IRC | 08:47 | |
| *** bauzas has quit IRC | 08:47 | |
| *** igordcard has quit IRC | 08:47 | |
| *** sfinucan has quit IRC | 08:47 | |
| *** patriciadomin has quit IRC | 08:47 | |
| *** beagles has quit IRC | 08:47 | |
| *** szaher has quit IRC | 08:47 | |
| *** smcginnis has quit IRC | 08:47 | |
| *** slunkad has quit IRC | 08:47 | |
| *** pkoniszewski has quit IRC | 08:47 | |
| *** ioni has quit IRC | 08:47 | |
| *** Anticimex has quit IRC | 08:47 | |
| *** andreykurilin has quit IRC | 08:47 | |
| *** seba has quit IRC | 08:47 | |
| *** Guest6666 has quit IRC | 08:47 | |
| *** diga has quit IRC | 08:47 | |
| *** tetsuro has quit IRC | 08:47 | |
| *** ys__ has quit IRC | 08:47 | |
| *** ChanServ has quit IRC | 08:47 | |
| *** sahid has quit IRC | 08:47 | |
| *** adisky__ has quit IRC | 08:47 | |
| *** burt has quit IRC | 08:47 | |
| *** ujjain has quit IRC | 08:47 | |
| *** colby_ has quit IRC | 08:47 | |
| *** vipul has quit IRC | 08:47 | |
| *** gaurangt has quit IRC | 08:47 | |
| *** Kevin_Zheng has quit IRC | 08:47 | |
| *** jbernard has quit IRC | 08:47 | |
| *** Oku_OS has quit IRC | 08:47 | |
| *** rfolco has quit IRC | 08:47 | |
| *** melwitt has quit IRC | 08:47 | |
| *** sballe_ has quit IRC | 08:47 | |
| *** zhenguo has quit IRC | 08:47 | |
| *** mrodden has quit IRC | 08:47 | |
| *** masayukig has quit IRC | 08:47 | |
| *** rajinir has quit IRC | 08:47 | |
| *** amotoki has quit IRC | 08:47 | |
| jianghuaw | Is the DB nova-api allowed to be accessed by nova-compute service? | 09:05 |
| jianghuaw | I met an error as "RemoteError: Remote error: CantStartEngineError No sql_connection parameter is established" when nova-compute tries to query data from aggregate which belong to nova-api db. | 09:07 |
| *** openstack has joined #openstack-nova | 13:54 | |
| *** gaoyan has joined #openstack-nova | 13:55 | |
| *** felipemonteiro has quit IRC | 13:56 | |
| *** gaoyan has quit IRC | 13:59 | |
| *** moshele has joined #openstack-nova | 13:59 | |
| *** mamandle has joined #openstack-nova | 14:00 | |
| openstackgerrit | Matthew Edmonds proposed openstack/nova master: update policy UT fixtures https://review.openstack.org/398610 | 14:00 |
| *** suresh12 has joined #openstack-nova | 14:01 | |
| *** mingyu_ has joined #openstack-nova | 14:01 | |
| *** mingyu has quit IRC | 14:01 | |
| *** vks1 has quit IRC | 14:02 | |
| *** crushil has joined #openstack-nova | 14:03 | |
| *** cdent has quit IRC | 14:07 | |
| *** owalsh_ has joined #openstack-nova | 14:09 | |
| *** marst has joined #openstack-nova | 14:09 | |
| *** edleafe- has joined #openstack-nova | 14:10 | |
| mriedem | i'm going through https://review.openstack.org/#/c/491850/ now | 14:12 |
| *** owalsh has quit IRC | 14:12 | |
| *** edleafe has quit IRC | 14:12 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Improve stable-api doc with current API state https://review.openstack.org/489926 | 14:13 |
| *** mtanino has joined #openstack-nova | 14:14 | |
| *** udesale has joined #openstack-nova | 14:15 | |
| *** vks1 has joined #openstack-nova | 14:15 | |
| *** sridharg has quit IRC | 14:17 | |
| *** zhouyaguo has quit IRC | 14:17 | |
| *** zhouyaguo has joined #openstack-nova | 14:17 | |
| openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Remove host filter for _cleanup_running_deleted_instances periodic task https://review.openstack.org/491808 | 14:18 |
| *** udesale has quit IRC | 14:19 | |
| openstackgerrit | Matthew Edmonds proposed openstack/nova master: use conf for keystone session creation https://review.openstack.org/485121 | 14:20 |
| *** felipemonteiro has joined #openstack-nova | 14:20 | |
| *** suresh12 has quit IRC | 14:21 | |
| dtantsur | dansmith: hi! re your comment on the ironic-related patch: should we block changing node.resource_class for active nodes in Ironic? | 14:21 |
| *** suresh12 has joined #openstack-nova | 14:21 | |
| dansmith | dtantsur: if that's possible I think that would be an excellent idea | 14:22 |
| dtantsur | dansmith: it's not impossible, but our beloved API microversion will kick in here. meaning, we'll only be able to block it starting with the next API version :( | 14:22 |
| *** sridharg has joined #openstack-nova | 14:23 | |
| dtantsur | I would actually block it in all versions, given that it's going to screw up nova | 14:23 |
| *** hemna_ has joined #openstack-nova | 14:23 | |
| dansmith | dtantsur: presumably you can return a 409 for mostly any reason right? | 14:23 |
| dtantsur | but people tend to feel quite religiously about not bypassing the versioning | 14:23 |
| dtantsur | dansmith: right, but how does it help? | 14:24 |
| * dtantsur looks for a wild cdent | 14:24 | |
| *** gbarros has quit IRC | 14:25 | |
| dtantsur | it's an interesting corner case of the API WG to discuss. should we leave a feature that clearly breaks things, or should we break the versioning contract | 14:25 |
| * dtantsur assumes mordred and edleafe- might have opinions ^^^ | 14:25 | |
| dansmith | dtantsur: well can you return 409 for anything else in that call? | 14:25 |
| * mordred reads | 14:25 | |
| dtantsur | dansmith: sorry, I think I don't get the question. We cannot just randomly return 409 I think.. | 14:26 |
| *** kukacz has quit IRC | 14:26 | |
| *** gbarros has joined #openstack-nova | 14:26 | |
| *** armax has joined #openstack-nova | 14:26 | |
| mriedem | dtantsur: mearning, can the node update api return a 409 already for something else | 14:27 |
| dansmith | dtantsur: if 409 is already a valid return value then I think it's less problematic | 14:27 |
| dansmith | right | 14:27 |
| mriedem | so the user can already be expecting a 409 in some cases | 14:27 |
| dansmith | and I would expect 409 to be valid for most PUTs to cover situations like this | 14:27 |
| dansmith | like "you're violating some constraint" | 14:27 |
| dtantsur | mriedem, dansmith, I think I get where you're heading. Yes, we can. And no, according to the API versioning ideology, we cannot do it. | 14:27 |
| * dansmith shrugs | 14:28 | |
| dtantsur | it's not only about breaking users, it's more about signaling changes /me waits for mordred to jump in | 14:28 |
| mriedem | https://developer.openstack.org/api-ref/baremetal/#update-node | 14:28 |
| mordred | so... | 14:28 |
| mriedem | doesn't mention error codes | 14:28 |
| dtantsur | (for the record: I'm not the biggest fan of the API versioning here) | 14:28 |
| *** cleong has joined #openstack-nova | 14:28 | |
| mordred | for signalling changes, it's to signal whether someone can do something or not | 14:28 |
| dansmith | dtantsur: well, regardless of whether ironic lets you do it, the rule to operators should be that they will break stuff if they do it once things are populated | 14:28 |
| mordred | the thing in this case isn't a valid thing to try to do | 14:28 |
| mordred | so a user who does it today isn't actually doing a thing that works | 14:28 |
| mordred | so they don't actually have, you know, an application that is going to break when you do this | 14:29 |
| dtantsur | fair | 14:29 |
| dansmith | that's true, although the application here is *probably* their ansible playbook | 14:29 |
| mriedem | you've got a 409 right here https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/node.py#L1698 | 14:29 |
| mordred | alternately, they don't need to ask the API if they can avoid doing the bad state -they can, as a client, avoid writing broken code without setting a microversion, since this is mostly about ironic returning an error when they do something stupid | 14:30 |
| mordred | SO | 14:30 |
| dansmith | where they've set a bunch of nodes to be a given class and will blindly blow that into ironic, I'm guessing | 14:30 |
| mordred | I'd say this is a bug fix where requiring a version bump does't actually add value to anyone | 14:30 |
| dtantsur | mriedem: let's please not use 409 though. we've done a big mistake in our youth, and now ironicclient retries it | 14:30 |
| dansmith | eww | 14:30 |
| dtantsur | however, the generic 400 an also be returned from essentially any endpoint, we can use it | 14:30 |
| dansmith | mordred: agreed | 14:30 |
| dtantsur | mordred: I like the way you put it :) thanks! | 14:31 |
| *** owalsh_ is now known as owalsh | 14:31 | |
| mordred | dtantsur: I will happily advocate for the validity of this change not actually being an API break if you need me to | 14:31 |
| mordred | rules are there to help us do the right thing - they're not there for their own sake | 14:31 |
| dtantsur | right | 14:31 |
| dtantsur | next tricky question :) | 14:32 |
| *** zhurong has quit IRC | 14:32 | |
| dtantsur | what should we do about nodes that do not have any resource_class so far? I mean, active nodes? | 14:32 |
| dtantsur | dansmith: ^^ | 14:32 |
| *** psachin has quit IRC | 14:33 | |
| dansmith | dtantsur: if you take the strict meaning of mordred's comments above, | 14:33 |
| dansmith | then you aren't breaking a node that is active with no RC since we're not doing anything with those yet | 14:33 |
| dansmith | however, | 14:33 |
| dansmith | I think it's far more confusing to allow that | 14:33 |
| dansmith | not worth the confusion over consistency | 14:33 |
| mriedem | jaypipes: dansmith: i'm inclined to -1 this for missing tests https://review.openstack.org/#/c/491850/ but given dansmith is out the next two days i realize there is a need to start pushing this code through | 14:34 |
| dtantsur | this is what I'm thinking about. a user creates an instance back in Ocata, no resource_class set. they upgrade to Pike, then to Queens. still no resource_class. | 14:34 |
| dansmith | dtantsur: ah, right, for nodes with an instance | 14:34 |
| mordred | dtantsur: what would the resource_class be if they created it with an empty resource_class today? | 14:34 |
| dtantsur | mordred: None | 14:34 |
| *** ralonsoh_ has quit IRC | 14:35 | |
| dtantsur | (python None, or JSON null) | 14:35 |
| dansmith | mordred: it's a thing that is required for queens | 14:35 |
| mordred | AH | 14:35 |
| *** takedakn has joined #openstack-nova | 14:35 | |
| *** baoli has quit IRC | 14:35 | |
| dansmith | dtantsur: so the response probably needs to be "400: You cannot CHANGE the class of an active node" | 14:35 |
| *** slaweq_ has joined #openstack-nova | 14:35 | |
| dtantsur | so yeah, it was perfectly valid to not have any resource_class when we introduced it (which is sad, btw) | 14:35 |
| dansmith | dtantsur: sad indeed | 14:35 |
| * dtantsur blames jroll :D | 14:36 | |
| dansmith | heh | 14:36 |
| dtantsur | my main question is: how will nova react when a node gets a resource_class (without having it previously) | 14:37 |
| dansmith | dtantsur: it'll do the right thing I think.. currently it's just skipping those and logging a warning | 14:37 |
| dansmith | it's the changing that is a problem | 14:37 |
| dansmith | without it set, we report no inventory and don't migrate the instances | 14:37 |
| dansmith | once it's set, we do those things | 14:37 |
| dtantsur | okay, we can ban changing, I think | 14:38 |
| dansmith | the changing of that (or handling the change) is the complicated bit | 14:38 |
| dtantsur | and what above available nodes? do you see any problems with them? | 14:38 |
| jaypipes | mriedem, dansmith: sorry, wrapping up a meetng | 14:38 |
| dansmith | dtantsur: hmm? above available nodes? | 14:38 |
| dansmith | dtantsur: you mean nodes with no instance on them? | 14:38 |
| dtantsur | yep, but available for nova to deploy on | 14:38 |
| dtantsur | (sorry, /me uses our state machine terms) | 14:39 |
| dansmith | dtantsur: well, it's less problematic for sure, but I think the caching in the ironic driver still means there's a race there, which is not great | 14:39 |
| *** trinaths has joined #openstack-nova | 14:39 | |
| dansmith | dtantsur: but I'd much rather document that as a pitfall as it doesn't require extra code, | 14:39 |
| dansmith | it just means there's a window where you might schedule something with the old class to the node you just changed | 14:40 |
| dtantsur | right. I guess we have the same problems right now with e.g. changing properties (except that people should not change their CPU count too often) | 14:40 |
| dansmith | and you'll migrate the instance to the stale class most likely which will be confusing, but :/ | 14:40 |
| *** slaweq_ has quit IRC | 14:40 | |
| dansmith | I have to brb | 14:40 |
| dtantsur | thanks dansmith, I think I know what to do | 14:40 |
| * edleafe- returns from making coffee and jumps into the scrollback | 14:42 | |
| *** edleafe- is now known as edleafe | 14:42 | |
| dtantsur | mmmm, coffee :) | 14:42 |
| edleafe | dtantsur: yeah, if the API already returns a particular code, and what you're changing isn't a thing that people rely on, it's a bugfix, not a version bump | 14:43 |
| edleafe | dtantsur: so yeah, what mordred said | 14:43 |
| jaypipes | mriedem: k, sorry, done now. | 14:44 |
| dtantsur | edleafe: okie, I'll bake a patch today | 14:44 |
| edleafe | dtantsur: if a node has no resource_class set and an instance on it, and then the resource_class gets set, the next time through the instance flavor will get updated | 14:44 |
| jaypipes | mriedem: I'm not entirely sure how one would add tests for a continue block... | 14:44 |
| edleafe | dtantsur: dansmith: that's why the 'seen' cache is keyed on (instance_uuid, rc) | 14:44 |
| mriedem | jaypipes: i guess you'd assert that something *didn't* happen | 14:45 |
| mriedem | since you continued | 14:45 |
| mriedem | but anyway, it can be a follow up given the circumstances with time | 14:46 |
| jaypipes | mriedem: k. answered your comment about local delete.. | 14:46 |
| edleafe | dtantsur: I'll revise that migration patch to remove the support for a node with an instance changing its resource_class | 14:47 |
| *** gouthamr has joined #openstack-nova | 14:47 | |
| dansmith | edleafe: dtantsur thanks a bunch | 14:47 |
| *** alexchadin has quit IRC | 14:48 | |
| dtantsur | np | 14:48 |
| mriedem | dansmith: jaypipes: is this required for rc1 or just sugar? https://review.openstack.org/#/c/491012/ | 14:55 |
| openstackgerrit | Eric Fried proposed openstack/nova master: nova.utils.get_ksa_adapter() https://review.openstack.org/488137 | 14:55 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Get auth from context for glance endpoint https://review.openstack.org/490057 | 14:55 |
| dansmith | mriedem: required | 14:55 |
| jaypipes | dansmith: to be perfectly frank, I don't know. :( | 14:55 |
| jaypipes | sorry, that was for mriedem ^ | 14:55 |
| jaypipes | mriedem: see the comment I just responded to edleafe and dansmith with. | 14:56 |
| jaypipes | mriedem: basically boils down to "I totally think we probably might need to maybe do something here, but I'm not sure what..." | 14:56 |
| dansmith | jaypipes: huh? before that patch pike computes are behaving like ocata ones | 14:56 |
| dansmith | that's where it was when I left it anyway | 14:56 |
| mriedem | alright i'll review https://review.openstack.org/#/c/488510/ in the meantime then | 14:58 |
| *** felipemonteiro has quit IRC | 14:58 | |
| jaypipes | dansmith: well, I guess after going through the scenarios in my head, I'm wondering what we really need to do there. Probably just need another hangout session with you on it. | 14:58 |
| dansmith | jaypipes: L1038 is the critical bit: https://review.openstack.org/#/c/491012/7/nova/compute/resource_tracker.py | 14:58 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Get auth from context for glance endpoint https://review.openstack.org/490057 | 14:58 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Add documentation for documentation contributions https://review.openstack.org/492124 | 14:58 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Clean up *most* ec2 / euca2ools references https://review.openstack.org/492166 | 14:58 |
| dansmith | jaypipes: without that line, we don't move the ball forward with comptues just stomping all over everything on both sides of a boot or move operation | 14:59 |
| jaypipes | dansmith: oh, I totally get that. the part I'm wondering about is whether that "heal allocations" method needs to change now. | 14:59 |
| dansmith | er, without gating that line on the version | 14:59 |
| dansmith | jaypipes: that's not what he was asking, right? he was asking if the whole change is required for rc1 right? | 14:59 |
| jaypipes | dansmith: ah, sorry. mriedem, yes, it is. | 15:00 |
| jaypipes | mriedem: I was -Workflow on it to discuss that bit about the heal allocations. | 15:00 |
| jaypipes | sorry for confusion. | 15:00 |
| mriedem | np | 15:00 |
| dansmith | jaypipes: your non-functional continue in a conditional is part of your extra debug logging I assume, and that was all you so..whatever you want to do there | 15:00 |
| mnaser | so: there should never be a scenario where an instance fails to spawn with local variable referenced before assignment.. right? | 15:00 |
| mriedem | mnaser: correct | 15:01 |
| mnaser | because im trying to go through this code and i cant find how/why it's happening | 15:01 |
| mnaser | okay. i'll go do some more in depth checking then | 15:01 |
| mriedem | UnboundLocalError right? | 15:01 |
| jaypipes | dansmith: heh, yeah... I was tempted to put into the log debug message something like "we're really not sure whether we even get here any more and if we do, what we should do anyway" ;) | 15:01 |
| mriedem | seems that would be obvious | 15:01 |
| mnaser | mriedem the manager is catching the original exception so its making it a tad harder | 15:02 |
| mnaser | based on my search, it should either be in nova/objects/numa.py or nova/virt/hardware.py as those are the two references to it | 15:02 |
| mriedem | oh, LOG.exception? | 15:02 |
| dansmith | jaypipes: yeah, and it might be useful. Just don't say "We were completely burned out at the end of pike and this seems like a bad thing we probably should have handled. Sorry about that." | 15:02 |
| jaypipes | dansmith: don't give me ideas... :P | 15:03 |
| mnaser | mriedem i'll have to do, it's def not catching by default, compute logs is just showing: 2017-08-09 14:33:00.758 4242 DEBUG nova.compute.utils [req-91225f58-71c5-44d3-ae5c-734986b4a3f7 16e021e0ed5f47b68c095d6885f18f4b d7594b0298b54bcc9e4e0f252e1da2e4 - - -] [instance: 18b91008-d42f-4eac-b57d-07195e7774ba] local variable 'sibling_set' referenced before assignment notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py | 15:03 |
| mnaser | :313 | 15:03 |
| dansmith | mriedem: comment in here about the wording https://review.openstack.org/#/c/491582/6 | 15:03 |
| jaypipes | dansmith: how about this? LOG.debug("We used to think we were indecisive. Now we're not so sure.") | 15:03 |
| dansmith | jaypipes: that seems fine to me. highly truth-based | 15:04 |
| jaypipes | very truthy indeed. | 15:04 |
| dansmith | jaypipes: but I'd only +1 it and wait for others to +2 | 15:04 |
| *** sree has quit IRC | 15:06 | |
| *** mamandle has quit IRC | 15:06 | |
| mriedem | dansmith: replied https://review.openstack.org/#/c/491424/7/releasenotes/notes/pike_prelude-fedf9f27775d135f.yaml - so you want me to add those or leave it? | 15:07 |
| *** sree has joined #openstack-nova | 15:07 | |
| *** takedakn has quit IRC | 15:08 | |
| dansmith | mriedem: I didn't realize that wasn't +A.. I would have.. I was just suggesting that maybe circling back and putting it in there might be worthwhile, but it's really not a big deal | 15:08 |
| dansmith | it's on its way now | 15:08 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: replace chance with filter scheduler in func tests https://review.openstack.org/491529 | 15:08 |
| dansmith | I was just commenting on your comment | 15:08 |
| mriedem | want to send this in also https://review.openstack.org/#/c/491581/ ? | 15:10 |
| mriedem | it was previously approved | 15:10 |
| mriedem | pad your stats before i wreck your stats on these RT patches :) | 15:10 |
| *** sree has quit IRC | 15:10 | |
| *** cfriesen has joined #openstack-nova | 15:10 | |
| mnaser | would someone be kind enough to just eye this for a second with me before i dive deeper? is it possible that i get an error of sibling_set being referenced before assignment if: the siblings_set is empty, therefore the loop next occurs and it is referenced in line 805 (outside the loop) -- https://github.com/openstack/nova/blob/stable/newton/nova/virt/hardware.py#L777-L807 | 15:10 |
| *** sree has joined #openstack-nova | 15:10 | |
| mnaser | sorry poorly worded that, the loop is pretty much skipped over so sibling_set is never set to anything and it's referenced in the if statement below it | 15:11 |
| *** Sukhdev has joined #openstack-nova | 15:11 | |
| dansmith | mriedem: :( | 15:11 |
| dansmith | mnaser: looking | 15:12 |
| mriedem | mnaser: https://github.com/openstack/nova/blob/stable/newton/nova/virt/hardware.py#L805 would be the problem right? | 15:12 |
| mriedem | if sibling_sets.items() was empty | 15:12 |
| mriedem | then there is no sibling_set variable set in the for loop above | 15:12 |
| mnaser | thats what i was guessing -- kinda wanted a second pair of eyes before i dive in deeper in the wrong place | 15:12 |
| mnaser | i will check and see what the value of sibling_sets is | 15:13 |
| dansmith | yeah it's referencing the loop variable | 15:13 |
| *** Oku_OS is now known as Oku_OS-away | 15:13 | |
| mnaser | ok cool, i'll do some more checking and see what the sibling set value is when it works and when it doesnt | 15:13 |
| mnaser | (oddly enough, it fails only on the *last* instance to go in the server -- ex: if it fits 15 VMs, 14 will go in, the 15th will fail with that) | 15:13 |
| dansmith | mnaser: you could just put 798 and below inside an "if sibling_sets" conditional | 15:14 |
| dansmith | mnaser: then it won't run if the loop didn't do a thing, which would avoid the problem | 15:14 |
| *** tesseract has quit IRC | 15:14 | |
| *** lucasxu has quit IRC | 15:14 | |
| mriedem | if only stephenfin were around to harass | 15:14 |
| dansmith | it'd be much better to just set something before the loop to None, and then set it inside the loop and only run the bottom code if we found a thing | 15:15 |
| dansmith | because it's just using the last value it iterated over | 15:15 |
| mnaser | yeah that seems cleaner, but i also wonder if the issue is siblings_set being empty | 15:15 |
| dansmith | sibling_set has to be a tuple for the use on L805 | 15:15 |
| dansmith | so you can use None as the sentinel | 15:15 |
| mnaser | becuase maybe it shouldn't be and thats the issue | 15:15 |
| * mnaser doesn't fully understand the hardware codebase yet | 15:16 | |
| dansmith | mnaser: well, it's fragile code so it deserves fixing regardless, IMHO | 15:16 |
| mnaser | so i just want to make sure the original issue isnt siblings_set being empty? | 15:16 |
| mnaser | true | 15:16 |
| mnaser | i was a bit taken aback seeing a variable referenced before assignment in nova's code :p | 15:16 |
| dansmith | if it being empty is possible (which it clearly is) and that's fatal, then this needs to check for it and log a warning | 15:16 |
| *** links has quit IRC | 15:16 | |
| *** sree has quit IRC | 15:17 | |
| dansmith | we could ask sahid | 15:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add release note for shared storage known issue https://review.openstack.org/491582 | 15:17 |
| gibi | mriedem, jaypipes, dansmith: As Jay's resize confirm fix is almost done could you take a look on the other resize bugfix https://review.openstack.org/#/c/491491 | 15:18 |
| *** gmann has quit IRC | 15:19 | |
| mnaser | yum, theory validated - SIBLING_SETS: defaultdict(<type 'list'>, {}) _pack_instance_onto_cores /usr/lib/python2.7/site-packages/nova/virt/hardware.py:781 | 15:19 |
| mnaser | now time to understand the siblings_set and why its empty | 15:20 |
| *** danpb has joined #openstack-nova | 15:20 | |
| danpb | dansmith: ? | 15:20 |
| dansmith | mnaser: danpb is your huckleberry | 15:20 |
| mnaser | hi danpb -- i think i've discovered an issue in nova/virt/hardware.py | 15:21 |
| mnaser | for some reason, siblings_set is become an empty list | 15:21 |
| mnaser | https://github.com/openstack/nova/blob/stable/newton/nova/virt/hardware.py#L777-L807 | 15:21 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Structure cli page https://review.openstack.org/492111 | 15:21 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Add documentation for documentation contributions https://review.openstack.org/492124 | 15:21 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Clean up *most* ec2 / euca2ools references https://review.openstack.org/492166 | 15:21 |
| openstackgerrit | Sean Dague proposed openstack/nova master: doc: Import configuration reference https://review.openstack.org/491853 | 15:21 |
| mnaser | and when its empty (for whatever reason), line 805 reference a variable that was not assigned | 15:21 |
| *** itlinux has joined #openstack-nova | 15:21 | |
| dansmith | danpb: we know how to fix the acute issue, but mnaser is concerned that sibling_sets being empty might be indicative of some more fundamental problem | 15:21 |
| dansmith | and wanted a sanity check | 15:22 |
| sdague | mriedem: the config reference import that stephenfin was working on should be workable now, the last iteration had a wrong include stanza that worked locally because of cruft I had, but failed in the gate | 15:22 |
| dansmith | danpb: we're hoping it was just an oversight and that set can be empty for legit reasons | 15:22 |
| mnaser | is there any scenario where sibling_set should be empty? this is a server with 32 cores, vcpu_pin_set is set to 2-31 | 15:23 |
| *** gjayavelu has joined #openstack-nova | 15:23 | |
| mnaser | wait | 15:23 |
| mnaser | oh boy | 15:23 |
| mnaser | did i just forget how to math | 15:23 |
| mnaser | and the fact it had 29 cores probably messed up the math | 15:23 |
| danpb | sibling_sets gets populated from available_siblings iiuc | 15:24 |
| danpb | so presumably available_siblings is empty too ? | 15:24 |
| mnaser | danpb: let me put some log.debug's, but i confirmed that siblings_set was empty | 15:24 |
| mnaser | fyi this issue only occurs on the *last* VM to fit on the compute node | 15:24 |
| *** ralonsoh has joined #openstack-nova | 15:25 | |
| danpb | well i think you'd need to work backwards through the call stack to figure out why it becomes empty | 15:26 |
| mnaser | danpb: AVAILABLE_SIBLINGS: [CoercedSet([]), CoercedSet([]), CoercedSet([]), CoercedSet([]), CoercedSet([]), CoercedSet([]), CoercedSet([])] | 15:26 |
| mnaser | ok i guess ill have to see why host_cell.free_siblings is being set to that | 15:27 |
| dansmith | danpb: doesn't this come from libvirt? | 15:27 |
| *** crushil has quit IRC | 15:28 | |
| *** markvoelker has quit IRC | 15:28 | |
| mnaser | the only thing i can imagine which can cause a corner case is the fact that we reserve 2 cores for the OS, so vcpu_pin_set=2-31 .. maybe that's not taken in consideration (guessing) | 15:28 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: [placement] Add api-ref for usages https://review.openstack.org/480563 | 15:28 |
| danpb | dansmith: libvirt will provide info on the siblings present on the host, but iiuc free_siblings is populated by nova | 15:29 |
| danpb | so if mnaser is saying it works correctly for all VMs until the last one, it sounds like nova is filtering the info from libvirt and ending up with the empty set | 15:30 |
| dansmith | danpb: oh okay, I hadn't traced it very far up the stack because I assumed we must just be getting an empty set of things from libvirt because we had no numa info or something | 15:30 |
| dansmith | danpb: yeah, makes sense | 15:30 |
| sahid | danpb: mnaser danpb i just for information i was working on this but did not find the root cause | 15:30 |
| sahid | https://review.openstack.org/#/c/458848/ | 15:30 |
| mnaser | im checking the values of host_cell and instance_cell | 15:30 |
| mnaser | HOST_CELL: NUMACell(cpu_usage=14,cpuset=set([2,4,6,8,10,12,14,16,18,20,22,24,26,28,30]),id=0,memory=196562,memory_usage=57344,mempages=[NUMAPagesTopology,NUMAPagesTopology],pinned_cpus=set([2,4,6,8,10,12,14,18,20,22,24,26,28,30]),siblings=[set([8,24]),set([2,18]),set([10,26]),set([12,28]),set([6,22]),set([14,30]),set([4,20])]) | 15:31 |
| danpb | how many VMs are you running ? | 15:31 |
| danpb | you've got 7 pairs of siblings there, so if each VM wanted one pair, you'd be able to run 7 VMs | 15:32 |
| mnaser | booted with 120x 1GB hugepages, 30 cores available, trying to boot 15 VMs with 2 cores each + 8gb of memory each | 15:32 |
| mnaser | but im not using isolate, im using prefer which i believe should try to schedule them on the same hyperthread | 15:32 |
| mnaser | my flavor has properties: hw:cpu_policy=dedicated, hw:cpu_thread_policy=prefer, hw:mem_page_size=1048576, hw:numa_nodes=2 | 15:33 |
| mnaser | (also hw:cpu_thread_policy=prefer forgot to put that in) | 15:33 |
| mnaser | i was thinking hw:numa_nodes=2 and hw:cpu_thread_policy=prefer might be the source of the issue because they are a bit the opposite of each other | 15:34 |
| mnaser | getting memory split from 2 numa nodes but also wanting to be on the same hyperthread isnt really possible considering you'll want to be on two physical cpus to access the numa node | 15:35 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: [placement] Add api-ref for RP usages https://review.openstack.org/450105 | 15:36 |
| *** slaweq_ has joined #openstack-nova | 15:37 | |
| danpb | mnaser: how many vcpus per guest ? | 15:37 |
| danpb | (i mean what is vcpu count in the flavour ?) | 15:38 |
| mnaser | danpb: 2 | 15:38 |
| *** aarefiev is now known as aarefiev_afk | 15:38 | |
| danpb | that flavour config looks flawed then | 15:38 |
| danpb | you've got 2 vcpus, and you've said to give the VM 2 virtual numa nodes | 15:38 |
| danpb | nova will want to place each virtual numa node on a separate host numa node | 15:38 |
| danpb | so each vcpu will need to be on a separate socket | 15:39 |
| openstackgerrit | Matthew Edmonds proposed openstack/nova master: use conf for keystone session creation https://review.openstack.org/485121 | 15:39 |
| danpb | so asking nova to put the vcpus in the same hyperthread sibling is nonsensical | 15:39 |
| danpb | as you can't have siblings if you have separate sockets | 15:39 |
| mnaser | that is what i kinda theorized a bit | 15:39 |
| *** markvoelker has joined #openstack-nova | 15:39 | |
| mnaser | so will have to fallback to 1 numanode default | 15:39 |
| danpb | nova shouldn't be trying to use hypthread siblings at all in ths case | 15:40 |
| mnaser | that was my assumption that what it would do, thats why i put 'prefer' in there because i figured that was the more reasonable choice | 15:40 |
| cfriesen | with hw:cpu_thread_policy=prefer it should just give separate pCPUs. if you had hw:cpu_thread_policy=require I think it'd fail | 15:40 |
| danpb | putting in prefer should be an no-op in this case | 15:40 |
| danpb | as the need to pick separate sockets should have taken priority in nova to satisfy the numa constraint | 15:41 |
| danpb | and if you have thread_policy=require, then nova ought to raise a fatal error | 15:41 |
| *** crushil has joined #openstack-nova | 15:41 | |
| *** slaweq_ has quit IRC | 15:41 | |
| cfriesen | hw:cpu_thread_policy=prefer is always a no-op, technically. it's the default behaviour | 15:41 |
| mnaser | right: so in that case, the config seems to be ok (yes, the prefer is noop/will never work) but it should still work | 15:42 |
| mnaser | numastat -m reports 4096 free hugepages in each node (for a total of 8192) | 15:42 |
| *** randomhack has quit IRC | 15:43 | |
| *** mkrai_ has joined #openstack-nova | 15:43 | |
| *** yamamoto has quit IRC | 15:43 | |
| *** chyka has joined #openstack-nova | 15:43 | |
| *** suresh12 has quit IRC | 15:44 | |
| *** yamamoto has joined #openstack-nova | 15:45 | |
| cfriesen | too bad sfinucan is on holidays. :) | 15:45 |
| *** yamamoto has quit IRC | 15:45 | |
| mnaser | the used cores are: 2-15,18-31 | 15:45 |
| *** yamamoto has joined #openstack-nova | 15:46 | |
| mnaser | which leaves 16/17 free which are on two different sockets | 15:46 |
| mnaser | so technically it should schedule on 16,17 which will be able to get 4096MB from each numa socket | 15:46 |
| *** Apoorva has joined #openstack-nova | 15:47 | |
| *** baoli has joined #openstack-nova | 15:47 | |
| cfriesen | mnaser: which version of the code is this? | 15:47 |
| mnaser | newton | 15:47 |
| mnaser | python-nova-14.0.7-1.el7.noarch | 15:48 |
| mnaser | more specifically | 15:48 |
| *** randomhack has joined #openstack-nova | 15:48 | |
| mnaser | sahid i looked at your change however it seems siblings_set being empty is a possiblity :( | 15:49 |
| cfriesen | dansmith: for some reason I thought that pCPUs without a sibling would still be in sibling_set, just as a single-item set | 15:50 |
| efried | sdague https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L198 Am I missing something, or is this untrue? | 15:50 |
| hongbin | oomichi: hi, my team is working on deciding the default api version, a team member asked me to check with you since you did the api version work in nova, i mainly wanted to know how nova maintain the default api version (pick the latest version or a stable version? bump the default version everytime a new version is introduced? etc.) Are you the right person to ask? | 15:51 |
| *** yamamoto has quit IRC | 15:51 | |
| mnaser | does host_cell get assigned by the scheduler? | 15:51 |
| cfriesen | mnaser: danpb: dansmith: this looks sort of related, but the fix should be in newton: https://bugs.launchpad.net/nova/+bug/1578155 | 15:52 |
| openstack | Launchpad bug 1578155 in OpenStack Compute (nova) newton "'hw:cpu_thread_policy=prefer' misbehaviour" [Medium,Fix committed] - Assigned to Stephen Finucane (stephenfinucane) | 15:52 |
| dansmith | I really don't know anything about this stuff | 15:52 |
| * artom remembers something like this | 15:54 | |
| jaypipes | mriedem, dansmith: I will be afk for 2 hours, just FYI. | 15:54 |
| cfriesen | mnaser: so are 16/17 the siblings of 0/1 which are not included in vcpu_pin_set? | 15:54 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: [placement] Add api-ref for usages https://review.openstack.org/480563 | 15:55 |
| artom | About the fix, at any rate. | 15:55 |
| mnaser | cfriesen nope, 0/1 are the ones which are not included | 15:55 |
| mnaser | so our vcpu_pin_set is 2-31 | 15:55 |
| mnaser | (32 core machine) | 15:55 |
| cfriesen | mnaser: yes, but which are the HT siblings of 0/1? | 15:55 |
| *** markvoelker has quit IRC | 15:56 | |
| danpb | hmm, yes, it seems numa node 0 has even numbered cpus, and node 1 has odd numbered cpus | 15:56 |
| mnaser | cfriesen sorry, not sure i follow, but here's lscpu output if that helps? http://paste.openstack.org/show/617955/ | 15:56 |
| danpb | so excluding 0-1, kills 1 cpu each host numa node, instead of 2 cpus | 15:56 |
| cfriesen | mnaser: based on the pattern above, the HT siblings of 0/1 would be 16/17....I'm wondering if that's confusing the logic in hardware.py | 15:56 |
| mnaser | but i think that they are | 15:56 |
| cfriesen | mnaswer: running "virsh capabilities" will show sibling info | 15:57 |
| *** yamamoto has joined #openstack-nova | 15:57 | |
| danpb | so that in turn kills a entire sibling pair from each numa node | 15:57 |
| *** slaweq_ has joined #openstack-nova | 15:57 | |
| danpb | effectively making 4 cpus unavailable | 15:57 |
| *** lucasxu has joined #openstack-nova | 15:57 | |
| cfriesen | danpb: theoretically it shouldn't kill the sibling pair though...we should have two pCPUs with no siblings | 15:57 |
| mnaser | here is output of virsh caps. http://paste.openstack.org/show/617956/ | 15:58 |
| *** Sukhdev_ has joined #openstack-nova | 15:58 | |
| cfriesen | but maybe the logic can't handle a mix of sibs and no sibs | 15:58 |
| mriedem | someone want to approve this? https://review.openstack.org/#/c/491855/ | 15:58 |
| danpb | yeah that wouldn't be surprising | 15:58 |
| *** randomha1k has joined #openstack-nova | 15:58 | |
| mnaser | so am i better off reserving 2 siblings and maybe that might get rid of the 'confusion' ? | 15:58 |
| cfriesen | mnaswer: seems likely | 15:58 |
| mnaser | so instead of 0,1 .. do 0,16 instead? | 15:58 |
| danpb | yeah | 15:59 |
| mnaser | let me test that theory out | 15:59 |
| *** randomhack has quit IRC | 16:00 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Handle addition of new nodes/instances in ironic flavor migration https://review.openstack.org/487954 | 16:00 |
| edleafe | dansmith: dtantsur: ^^ updated to remove the node RC change handling | 16:00 |
| dtantsur | awesome! I'm just out of a meeting, will do the ironic part now | 16:00 |
| mnaser | so my vcpu_pin_set should technically be 1-15,17-31 in that case, correct? | 16:00 |
| mriedem | gibi: wasn't that also a problem in ocata then? CoreFilter isn't enabled by default | 16:00 |
| mnaser | leaving 0,16 for the OS | 16:00 |
| cfriesen | mnaswer: either way, please raise a bug for this issue, it'd be nice to handle it properly | 16:00 |
| danpb | mnaser: yeah | 16:01 |
| mnaser | cfriesen agreed | 16:01 |
| mnaser | ok, lets try this out | 16:01 |
| mnaser | ill reboot with the new updates isolcpus= values so itll take me a few minutes and report | 16:01 |
| *** yamamoto has quit IRC | 16:01 | |
| dansmith | mriedem: melwitt: I'm going to disappear suddenly somewhere in the hour of the cells meeting.. assume we were on track to punt anyway, but.. that okay? | 16:02 |
| *** crushil has quit IRC | 16:02 | |
| *** crushil has joined #openstack-nova | 16:02 | |
| *** lucasxu has quit IRC | 16:03 | |
| *** markvoelker has joined #openstack-nova | 16:03 | |
| dansmith | jaypipes: so what's the deal with that last patch? do you want me to update it and remove the extraneous continue? or log something there or what? | 16:03 |
| jaypipes | please, yes, go for it. | 16:04 |
| dansmith | ...which? | 16:04 |
| gibi | mriedem: you are right CoreFilter is not enabled by default in Ocata. I will quickly backport the test locally to Ocata to confirm... | 16:04 |
| jaypipes | dansmith: ^ sorry, I've been in meetings and dealing with russian visa mess all morning and now have a doctor's appt. | 16:04 |
| jaypipes | dansmith: remove the extraneous continue thign | 16:04 |
| dansmith | okay | 16:04 |
| *** dtantsur is now known as dtantsur|brb | 16:05 | |
| mriedem | dansmith: yeah planned on skipping cells v2 meeting today | 16:06 |
| dansmith | mriedem: okay | 16:06 |
| mriedem | gibi: the fix in ocata would have to be different probably since this code all got refactored in pike | 16:06 |
| mriedem | gibi: but just wanted to make sure it's not an rc1 regression blocker thing | 16:06 |
| *** moshele has quit IRC | 16:08 | |
| dansmith | mriedem: I actually think we probably should enable the cache.. I thought it was already done on the computes, but it's not. The other use on compute is to calculate the rpc pin, which is cached until restart anyway | 16:08 |
| mnaser | danpb cfriesen -- i'm now getting "Not enough available CPUs to schedule instance. Oversubscription is not possible with pinned instances. Required: 1, actual: 0" | 16:09 |
| *** sahid has quit IRC | 16:09 | |
| mnaser | HOST_CELL: NUMACell(cpu_usage=14,cpuset=set([2,4,6,8,10,12,14,18,20,22,24,26,28,30]),id=0,memory=196562,memory_usage=57344,mempages=[NUMAPagesTopology,NUMAPagesTopology],pinned_cpus=set([2,4,6,8,10,12,14,18,20,22,24,26,28,30]),siblings=[set([8,24]),set([2,18]),set([10,26]),set([12,28]),set([6,22]),set([14,30]),set([4,20])]) | 16:09 |
| *** trinaths has left #openstack-nova | 16:11 | |
| danpb | where's your other numa cell | 16:11 |
| danpb | that just shows the first cell | 16:11 |
| mnaser | i dont know why its not listed. i added LOG.debug("HOST_CELL: %s" % host_cell) to _numa_fit_instance_cell_with_pinning | 16:12 |
| *** pchavva has quit IRC | 16:12 | |
| cfriesen | mnaser: I think I know what's going on...you now have fewer pCPUs in one node than the other, and you're asking for 2-node guests | 16:12 |
| mnaser | let me get you | 16:12 |
| mnaser | both host cell output | 16:12 |
| mnaser | oh | 16:12 |
| mnaser | i think you're right | 16:13 |
| danpb | cfriesen: yep makes sense | 16:13 |
| mnaser | let me switch things back to how they were and get the output of both host cells | 16:13 |
| danpb | mnaser: why do you want the guests to have multiple virtual numa cells ? | 16:14 |
| *** cdent has joined #openstack-nova | 16:14 | |
| cfriesen | mriedem: for a reno for https://review.openstack.org/#/c/491854/ would we want to describe the removal of the two default filters in "features", "upgrade", or "other"? | 16:14 |
| danpb | its generally not something you'd do unless guest memory exceeds the amount available in a single host, or need to consume say, PCI devices from separate nods at the same time | 16:14 |
| *** lucasagomes is now known as lucas-afk | 16:14 | |
| mnaser | danpb: still experimenting but the idea was more efficent use of hardware. i have 120x 1gb large pages which means that i'll end up with 60gb on each numanode, if i put 8gb sized instances only, i'll end up with 4096mb in each numa node that's unused | 16:15 |
| cfriesen | danpb: or you want increased memory bandwidth | 16:15 |
| *** hieulq has joined #openstack-nova | 16:15 | |
| danpb | cfriesen: that's only increased if your guest workload avoids cross-node traffic | 16:15 |
| cfriesen | danpb: agreed | 16:16 |
| mnaser | by doing this, it splits 4096 into each numanode and i can fill the freepages .. but it's not something that's set in stone fully | 16:16 |
| danpb | cfriesen: otherwise you'd actually decrease throughput | 16:16 |
| cfriesen | mnaser: the downside is that your guests need to be able to avoid cross-numa traffic, which makes guest coding trickier | 16:16 |
| danpb | by having the guest all contend on the cross-node memory bus | 16:16 |
| *** crushil has quit IRC | 16:16 | |
| cfriesen | danpb: yes, it'd be a special-case | 16:16 |
| mnaser | yeah i was thinking the linux kernel already does an ok job handling multiple numa nodes | 16:17 |
| danpb | IOW unless your guest app is intelligent you want to avoid multiple numa nods | 16:17 |
| mnaser | i see | 16:17 |
| cfriesen | mnaser: the kernel does, but not all userspace code does. | 16:17 |
| mriedem | dansmith: ack - gonna be afk for about an hour | 16:17 |
| mnaser | also the only other tradeoff that comes with this is that the threads are not shared which is not ideal | 16:17 |
| mnaser | so that was something i didnt like about doing this | 16:17 |
| mriedem | can't be worst dad of the summer 2 days in a row | 16:17 |
| cfriesen | mnaser: so if you've got a single large app that wants most of that 8GB... | 16:17 |
| dansmith | mriedem: okay I'll push this up with those changes and I'll be gone before jenkins gets to it | 16:18 |
| mnaser | for completition sake btw, this is both host cells | 16:18 |
| dansmith | mriedem: jaypipes so baton back to you at that point | 16:18 |
| mnaser | http://paste.openstack.org/show/617957/ | 16:18 |
| mnaser | there are 15 sets there | 16:19 |
| mnaser | let me see what it was when we had 0-1 only | 16:19 |
| cdent | dansmith: couple questions about expected discover_hosts behavior: is it supposed to be idempotent (run it again and again, it’s okay)? It is supposed to cope if two different processes run it at the same time? | 16:20 |
| cfriesen | mnaser: but only 14 pairs of pCPUs from different numa nodes | 16:20 |
| mnaser | yeah, thats why it failed now, but in the original case | 16:20 |
| mnaser | strangely enough | 16:20 |
| mnaser | i only see one output of numacell, not two | 16:20 |
| dansmith | cdent: yeah, it's expected to just run it over and over again, from cron even | 16:20 |
| dansmith | cdent: you might get a failure if you run multiples and one loses the race to insert the record, but other than that it should be fine to run multiple threads of it | 16:21 |
| cdent | dansmith: thanks that’s what I was hoping/expecting but wanted to confirm | 16:21 |
| *** markvoelker has quit IRC | 16:23 | |
| mnaser | im going to try setting hw:numa_nodes=1 and packing the server and seeing what happens (i should be able to get 14 at least) | 16:23 |
| gibi | mriedem: ocata is not affected by the bug: https://github.com/openstack/nova/blob/stable/ocata/nova/scheduler/filter_scheduler.py#L188 | 16:25 |
| gibi | mriedem: I mean bug 1708637 | 16:26 |
| openstack | bug 1708637 in OpenStack Compute (nova) "nova does not properly claim resources when server resized to a too big flavor" [High,In progress] https://launchpad.net/bugs/1708637 - Assigned to Balazs Gibizer (balazs-gibizer) | 16:26 |
| *** crushil has joined #openstack-nova | 16:26 | |
| cfriesen | mnaser: do you actually need 8GB? if you don't actually need it all, you could drop to 2MB hugepages and divide up the memory evenly with less waste. For most things 1GB pages don't give that big of a boost. | 16:26 |
| mnaser | cfriesen have you had experience with it? i just figured that if i can have 1gb pages, it would be better than 2mb pages but there isn't much substance to it other than 'it seems right' | 16:27 |
| mnaser | dropping to 2mb would obviously make life much easier | 16:27 |
| cdent | mriedem: yeah, we’ve been pretty inconsistent about which 400s are document in the placement-api-ref. anything common like “yo, not here” and “hey, you violated schema” has frequently been dropped. I’ve not been too strict on my reviews of that stuff except where a 4xx has some particular weird sense | 16:27 |
| *** danpb has quit IRC | 16:28 | |
| cdent | the current tooling doesn’t really have as much support for handling error responses as we might want, that’s probably something we could and should address later. I think once the whole thing is documented we’ll be able to tune it as a whole, better | 16:28 |
| *** lpetrut has quit IRC | 16:28 | |
| *** danpb has joined #openstack-nova | 16:28 | |
| mnaser | i just tried to create 14x 2vcpu/8gb memory and that went ok, then 1x 1vcpu/4gb memory and that was okay, so now i have 4gb free large pages.. trying to create 1vcpu/4gb - Host does not support requested memory pagesize. Requested: 1048576 kB | 16:29 |
| *** randomha1k has quit IRC | 16:29 | |
| *** lpetrut has joined #openstack-nova | 16:29 | |
| mnaser | looks like it was assigned core 21, which is on node1 .. but my free 4096 is on node0 | 16:30 |
| mnaser | i guess this happened because im still using the same thread pair, probably wouldnt happen again if i reserve 0-1 for the os | 16:30 |
| *** jmlowe has quit IRC | 16:31 | |
| *** danpb has quit IRC | 16:31 | |
| cfriesen | mnaser: we've done some testing. there are some cases where it makes a noticeable difference, but most of the time the difference is not worth the wasted memory. it depends on the guest | 16:31 |
| *** markvoelker has joined #openstack-nova | 16:32 | |
| mnaser | cfriesen all the guests are going to be multiples of 1gb in memory so 120gb in large pages in 1gb or in 2mb .. wouldn't make much difference, no? it looks like the # of freepages is split across both numa nodes | 16:33 |
| *** slaweq_ has quit IRC | 16:34 | |
| cfriesen | mnaser: if it must be exact multiples of 1GB, then you may as well use 1GB hugepages. | 16:34 |
| *** slaweq_ has joined #openstack-nova | 16:35 | |
| mnaser | cfriesen yeah they're all multiples of 1gb.. anyways ill try going back to 0-1 and seeing if i can fully populate the server | 16:35 |
| mnaser | and then after that ill file a bug regarding that sibling_set issue | 16:35 |
| *** eharney has joined #openstack-nova | 16:37 | |
| *** crushil has quit IRC | 16:38 | |
| *** slaweq_ has quit IRC | 16:38 | |
| *** slaweq_ has joined #openstack-nova | 16:39 | |
| *** baoli has quit IRC | 16:39 | |
| *** kornicameister has quit IRC | 16:40 | |
| *** rcernin has quit IRC | 16:41 | |
| *** baoli has joined #openstack-nova | 16:42 | |
| *** Nil_ has joined #openstack-nova | 16:44 | |
| cdent | mriedem: is this still relevant or has other stuff killed it: https://review.openstack.org/#/c/488187/ | 16:44 |
| *** lyan has quit IRC | 16:45 | |
| *** kornicameister has joined #openstack-nova | 16:45 | |
| *** markvoelker has quit IRC | 16:46 | |
| *** sridharg has quit IRC | 16:47 | |
| *** crushil has joined #openstack-nova | 16:50 | |
| *** sbezverk has joined #openstack-nova | 16:52 | |
| *** markvoelker has joined #openstack-nova | 16:55 | |
| *** yamamoto has joined #openstack-nova | 16:58 | |
| *** pchavva has joined #openstack-nova | 16:58 | |
| *** gcb has quit IRC | 17:02 | |
| *** yamamoto has quit IRC | 17:03 | |
| *** krtaylor has quit IRC | 17:05 | |
| *** eharney has quit IRC | 17:08 | |
| *** david-lyle has quit IRC | 17:08 | |
| *** david-lyle has joined #openstack-nova | 17:08 | |
| *** Sukhdev_ has quit IRC | 17:09 | |
| *** jpena|off is now known as jpena | 17:12 | |
| mnaser | reserving 0-1 allows me to start 14 2vcpu/8gb, but refuses to let me start anymore with that same sibling_sets bug | 17:13 |
| *** kristian__ has quit IRC | 17:13 | |
| openstackgerrit | Chris Friesen proposed openstack/nova master: Remove ram/disk sched filters from default list https://review.openstack.org/491854 | 17:14 |
| *** kristian__ has joined #openstack-nova | 17:14 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: replace chance with filter scheduler in func tests https://review.openstack.org/491529 | 17:17 |
| *** dtantsur|brb is now known as dtantsur | 17:18 | |
| mnaser | cfriesen: doing more debugging, i believe i'm onto something -- HOST_TOPOLOGY: NUMATopology(cells=[NUMACell(UNKNOWN),NUMACell(1)]) | 17:18 |
| mnaser | for some reason the first numacell is unknown..? i'll have to check that | 17:18 |
| *** kristian__ has quit IRC | 17:18 | |
| *** mriedem has quit IRC | 17:22 | |
| *** lyan has joined #openstack-nova | 17:22 | |
| melwitt | dansmith: +1 to skipping cells meeting | 17:23 |
| *** mkrai_ has quit IRC | 17:23 | |
| *** randomhack has joined #openstack-nova | 17:25 | |
| *** mingyu_ has quit IRC | 17:26 | |
| *** mdnadeem has joined #openstack-nova | 17:26 | |
| *** mdnadeem has quit IRC | 17:26 | |
| *** mingyu has joined #openstack-nova | 17:27 | |
| *** gjayavelu has quit IRC | 17:27 | |
| *** mriedem has joined #openstack-nova | 17:29 | |
| mriedem | cdent: not sure, but not something we need to care about for rc1 | 17:29 |
| *** ralonsoh has quit IRC | 17:29 | |
| *** gszasz has quit IRC | 17:29 | |
| cdent | mriedem: ‘k, will circle back round to that after we get the main things settled | 17:29 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Resource tracker compatibility with Ocata and Pike https://review.openstack.org/491012 | 17:31 |
| *** mingyu has quit IRC | 17:31 | |
| dansmith | mriedem: jaypipes ^ | 17:31 |
| *** markvoelker has quit IRC | 17:31 | |
| *** randomhack has quit IRC | 17:32 | |
| *** sambetts is now known as sambetts|afk | 17:32 | |
| mriedem | cfriesen: commented in your change | 17:33 |
| mriedem | cfriesen: if you're using the caching scheduler, you still rely on those filters since the caching scheduler doesn't use placement | 17:33 |
| dansmith | mriedem: did we ever push up that deprecation for those schedulers? | 17:33 |
| mriedem | but the caching scheduler isn't the default filter driver, the filter_scheduler is, so i think it's fine for the default enabled filters to match the default scheduler driver | 17:33 |
| mriedem | dansmith: i didn't see one | 17:33 |
| dansmith | sounded like sdague was going to but I don't know that he did | 17:33 |
| dansmith | mriedem: is it enough to put something in the config help text or do you want to log something? | 17:34 |
| *** lpetrut has quit IRC | 17:34 | |
| mriedem | i think we'd need both | 17:34 |
| *** jmlowe has joined #openstack-nova | 17:35 | |
| *** jmlowe has quit IRC | 17:35 | |
| mnaser | cfriesen i'm onto something, with a totally empty hypervisor -- i see this -- AVAILABLE_SIBLINGS: [CoercedSet([1, 17]), CoercedSet([31, 15]), CoercedSet([23, 7]), CoercedSet([13, 29]), CoercedSet([27, 11]), CoercedSet([19, 3]), CoercedSet([9, 25]), CoercedSet([5, 21])] | 17:35 |
| *** vks1 has quit IRC | 17:35 | |
| *** jpena is now known as jpena|off | 17:35 | |
| dansmith | mriedem: okay | 17:35 |
| cdent | dansmith: was the resolution on the question of what to do when we get to the end of that set of conditionals where jay had “literally no idea what to do” to do nothing? | 17:35 |
| mriedem | dansmith: plus reno of course | 17:35 |
| mnaser | [1, 17] shouldn't be there, it should just be 17, because vcpu_pin_set is 2-31 | 17:35 |
| *** jmlowe has joined #openstack-nova | 17:35 | |
| mnaser | i dont think that codebase takes vcpu_pin_set into consideration | 17:35 |
| dansmith | cdent: yes | 17:35 |
| cdent | ✔ | 17:35 |
| *** jmlowe has quit IRC | 17:36 | |
| *** shan has joined #openstack-nova | 17:36 | |
| mriedem | gibi: ok so i guess it is a regression in pike then so i'll mark it pike-rc-potential | 17:36 |
| *** suresh12 has joined #openstack-nova | 17:37 | |
| cfriesen | mnaser: you had changed vcpu_pin_set to 0,16, no? | 17:37 |
| cfriesen | mriedem: okay, I'll respin | 17:38 |
| *** jmlowe has joined #openstack-nova | 17:38 | |
| mnaser | cfriesen i switched back. by setting it to 0,16 => i end up being unable to spin the last instance because all cores + memory are taken on numanode #2 | 17:38 |
| mnaser | s/#2/#1/ | 17:38 |
| mnaser | and numanode 0 has 4gb memory left unused | 17:38 |
| mnaser | by using 0,16 -- i have a single thread in both cores that can use the leftover 4gb | 17:39 |
| mnaser | using 0,16 effectively means 14 threads for 1 numanode and 16 threads for the other. vs 15/15 | 17:39 |
| *** mingyu has joined #openstack-nova | 17:40 | |
| *** gjayavelu has joined #openstack-nova | 17:40 | |
| cfriesen | mnaser: please double-check that you restarted nova-compute or rebooted after changing vcpu_pin_set. I agree that you shouldn't see 1 in the AVAILABLE_SIBLINGS list. | 17:40 |
| mriedem | gdi, people, tox -e fast8 before git review | 17:40 |
| mriedem | https://review.openstack.org/#/c/488510/ | 17:40 |
| mnaser | DEBUG oslo_service.service [req-39a92c29-83e3-4cb0-9f78-29b97d61147d - - - - -] vcpu_pin_set = 2-31 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2622 in the logs, /proc/cmdline contains isolcpus=2-31 | 17:41 |
| *** rcernin has joined #openstack-nova | 17:42 | |
| *** jamesdenton has quit IRC | 17:42 | |
| *** abalutoiu has quit IRC | 17:42 | |
| *** kornicameister has quit IRC | 17:43 | |
| *** sdague has quit IRC | 17:43 | |
| mnaser | http://paste.openstack.org/raw/617965/ this is what is generated from `resources` in claims.py | 17:44 |
| *** yamahata has joined #openstack-nova | 17:44 | |
| mnaser | i dont even see it in the siblings there, i wonder if it is in the process of the transformation to the DB object | 17:44 |
| *** kornicameister has joined #openstack-nova | 17:46 | |
| cfriesen | mriedem: what do you think of this wording? http://paste.openstack.org/show/617966/ | 17:47 |
| *** jamesdenton has joined #openstack-nova | 17:47 | |
| mriedem | "If you have ``scheduler_driver`` set to ``caching_scheduler``, you should probably add these entries back in along with CoreFilter." | 17:48 |
| mriedem | issue the first: scheduler_driver is the old option name | 17:48 |
| mriedem | it's now [scheduler]/driver | 17:48 |
| cfriesen | whoops, was looking at wrong window. :) | 17:49 |
| mriedem | issue the 2nd: let's not tell them what to do if they're not using the filter scheduler, but just point out that non-FilterScheduler drivers, like the CachingScheduler, may want to have these filters enabled | 17:49 |
| cfriesen | okay | 17:49 |
| mriedem | if they already have that option defined in their nova.conf, then they don't have to 'add' anything back in, as it's already there, and we're just changing the defaults | 17:50 |
| cfriesen | mriedem: the issue I'm worried about is if they were relying on the defaults | 17:50 |
| mriedem | sure, and that's why you point out that non-FilterScheduler drivers, like the CachingScheduler, may want to have these enabled | 17:50 |
| cfriesen | fair enough | 17:50 |
| mriedem | isn't there also a case that you don't want the DiskFilter enabled - if you're using shared storage? | 17:52 |
| mriedem | looks like we never documented that in the description for the DiskFilter | 17:52 |
| *** randomhack has joined #openstack-nova | 17:52 | |
| mriedem | i'm not saying you need to for this change, but just thinking of the operator that then starts looking at if they need to enable these or not - will they have enough information to make that decision | 17:53 |
| *** mamandle has joined #openstack-nova | 17:53 | |
| cdent | thanks mriedem, by being assigned the fix for that bug, I should receive a gloriously huge bug bounty from … someone | 17:53 |
| mriedem | tjones | 17:54 |
| mriedem | has the bug bounty kitty | 17:54 |
| *** gyee has joined #openstack-nova | 17:54 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Mark Chance and Caching schedulers as deprecated https://review.openstack.org/492210 | 17:54 |
| *** Sukhdev_ has joined #openstack-nova | 17:54 | |
| dansmith | mriedem: ^ | 17:54 |
| *** sshwarts has quit IRC | 17:55 | |
| *** mingyu has quit IRC | 17:55 | |
| openstackgerrit | Chris Friesen proposed openstack/nova master: Remove ram/disk sched filters from default list https://review.openstack.org/491854 | 17:57 |
| *** kbaegis has joined #openstack-nova | 17:57 | |
| mriedem | dansmith: lgtm | 17:57 |
| mnaser | cfriesen my apologies, looks like wrong config.. but i think i've narrowed it down to this: AVAILABLE_SIBLINGS: [CoercedSet([8, 24]), CoercedSet([2, 18]), CoercedSet([10, 26]), CoercedSet([28, 12]), CoercedSet([22, 6]), CoercedSet([30, 14]), CoercedSet([20, 4])] .. technically there should be more set in there with CoercedSet([16]) -- because the 0 is reserved | 17:57 |
| mnaser | it's removed the entire set rather than the single one | 17:58 |
| cfriesen | dansmith: is the performance of FilterScheduler with Placement comparable to CachingScheduler? | 17:58 |
| mnaser | (this might be obvious to you but i'm just figuring it out, heh) | 17:58 |
| dansmith | cfriesen: it should end up being much better | 17:58 |
| cfriesen | mnaser: if you've got 0/1 reserved then I think arguably there should be a set with just 16 and another set with just 17 | 17:59 |
| *** yamamoto has joined #openstack-nova | 17:59 | |
| cfriesen | it should be noted though that the performance of 16/17 is going to be variable, depending on what load is on 0/1 | 18:00 |
| cfriesen | dansmith: better? isn't CachingScheduler in memory for the most part? | 18:00 |
| mnaser | of course, that's a given | 18:00 |
| *** lucasxu has joined #openstack-nova | 18:00 | |
| *** sdague has joined #openstack-nova | 18:01 | |
| dansmith | cfriesen: we've been discussing this in this channel for like I year, so I hope I don't have to summarize all of it, but if you count the ability to multi-thread the scheduler with placement, and zero reschedules, yes, it should be much better | 18:01 |
| dansmith | s/I/a | 18:01 |
| *** MVenesio has quit IRC | 18:01 | |
| mriedem | roman numeral I | 18:02 |
| *** MVenesio has joined #openstack-nova | 18:02 | |
| mriedem | you meant | 18:02 |
| sdague | dansmith: I did not push the deprecation of the other schedulers | 18:02 |
| dansmith | heh, yeah | 18:02 |
| sdague | dansmith: if that's a patch that is needed, I could do it this afternoon | 18:02 |
| dansmith | sdague: I just did | 18:02 |
| mriedem | https://review.openstack.org/#/c/492210/1 | 18:02 |
| sdague | dansmith: ok, coolio | 18:02 |
| cfriesen | dansmith: ah, you mean overall scheduler load. got it. (I was thinking time to perform a single schedule operation.) | 18:02 |
| mnaser | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5632-L5633 -- bingo.. now to see why the decision of filtering out singles is there | 18:02 |
| mriedem | cfriesen: it's also that | 18:02 |
| *** jpena|off is now known as jpena | 18:02 | |
| mriedem | time to schedule should be faster when you don't have reschedules | 18:03 |
| mriedem | b/c of bad decisions | 18:03 |
| mriedem | cfriesen: your team seems to do some performance testing, it would be cool if your guy(s) could benchmark and compare the two now in pike | 18:03 |
| dansmith | if you really care about the milliseconds requried to place something, you should be able to do better with a bunch of filterschedulers than one in-memory caching scheduler I would think | 18:04 |
| mriedem | like single caching scheduler vs multiple filter schedulers | 18:04 |
| cdent | cfriesen: it may be the case, but I’m not sure that anyone has proven it yet, that single schedule operations will speed up because placement will help limit the initial set of available candidates | 18:04 |
| *** yamamoto has quit IRC | 18:04 | |
| dansmith | cdent: cachingscheduler only refreshes the list when it runs out of spots I think, so it varies, but it also is using very outdated information which means fast but poor decisions | 18:04 |
| cfriesen | I'd expect the FilterScheduler to speed up due to Placement. Wasn't sure that it'd be faster than CachingScheduler. Anyways, we've been using FilterScheduler since we care about accurate resource tracking. | 18:05 |
| mriedem | caching scheduler is also on a periodic | 18:05 |
| cfriesen | (We modified stuff like resize and migration to update the resource tracking immediately rather than wait for the resource audit.) | 18:05 |
| dansmith | caching scheduler is filter scheduler, with cached host list data | 18:05 |
| mriedem | dansmith: so this resize/rt series has a pep8 failure in the bottom change, | 18:06 |
| *** kbaegis has quit IRC | 18:06 | |
| mriedem | dansmith: but i guess hold off on fixing that so i can go through them now | 18:06 |
| mriedem | the last 2 i mean | 18:06 |
| dansmith | okay | 18:06 |
| mriedem | dansmith: it's T-~3 hours to 0 dan time yeah? | 18:06 |
| dansmith | check queue is 7h long | 18:06 |
| *** kbaegis has joined #openstack-nova | 18:07 | |
| dansmith | mriedem: I think you mean T-3 hours until free-of-dan time | 18:07 |
| dansmith | but yeah | 18:07 |
| dansmith | I assume everyone brought liquor and cake to work today for the post-2pm PDT party | 18:07 |
| mriedem | brought? i think it's just generally available | 18:08 |
| dansmith | I was trying to act like it was the 90s | 18:08 |
| *** felipemonteiro has joined #openstack-nova | 18:10 | |
| *** felipemonteiro_ has joined #openstack-nova | 18:12 | |
| mnaser | cfriesen thanks for all your feedback, i think the best solution for now is going to be use 0,16 to have predictable performance for the host and assign hugepages with a slight bias towards node1 because it has 2 more cores so things can get properly packed | 18:13 |
| *** dtantsur is now known as dtantsur|afk | 18:15 | |
| *** felipemonteiro has quit IRC | 18:16 | |
| *** lpetrut has joined #openstack-nova | 18:17 | |
| *** kbaegis1 has joined #openstack-nova | 18:21 | |
| *** yassine has quit IRC | 18:22 | |
| sdague | dansmith: yeh, we're down to 800 nodes in ci | 18:23 |
| sdague | which is unfortunately not sufficient | 18:23 |
| dansmith | well, it was fun while it lasted | 18:24 |
| *** kbaegis has quit IRC | 18:24 | |
| sdague | if anyone wants to do doc reviews to hopefully complete the import - https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/doc-migration | 18:25 |
| *** xinliang has quit IRC | 18:26 | |
| cfriesen | mnaser: sorry you hit a bug. :) we should be able to get it sorted out once sfinucan comes back from holidays | 18:30 |
| cfriesen | mnaser: ping me with the bug number once you file it please | 18:31 |
| *** markvoelker has joined #openstack-nova | 18:32 | |
| *** nicolasbock has quit IRC | 18:33 | |
| *** mamandle has quit IRC | 18:34 | |
| *** nicolasbock has joined #openstack-nova | 18:34 | |
| *** pchavva has quit IRC | 18:37 | |
| mriedem | dansmith: jaypipes: cdent: edleafe: problems https://review.openstack.org/#/c/488510/ | 18:41 |
| dansmith | mriedem: I +2d before we had jenkins runs of the latest version | 18:42 |
| dansmith | and it was passing (tempest) before | 18:42 |
| mriedem | it's not that it's not passing tempest | 18:42 |
| mriedem | it's just spewing ERRORs | 18:42 |
| jaypipes | hey guys, just got back. | 18:42 |
| jaypipes | hmm, yes, the dreaded ./nova/compute/resource_tracker.py:1112:17: N352 LOG.warn is deprecated, please use LOG.warning! | 18:43 |
| mriedem | heh i even pointed that out | 18:44 |
| mriedem | it's in the bottom change btw | 18:44 |
| mriedem | so the entire series needs to be rebaesd | 18:44 |
| mriedem | *rebased | 18:44 |
| mriedem | but i have issues with the 2nd change in the series | 18:44 |
| jaypipes | yeah, I see that | 18:45 |
| dansmith | mriedem: we have the placement upgrade before nova thing required in the prelude of the renos | 18:45 |
| dansmith | which should be enough to merge this, IMHO | 18:45 |
| jaypipes | "prelude of the renos" sounds like some classical music composition. | 18:45 |
| *** morgan is now known as kmalloc | 18:46 | |
| * jaypipes goes to read mriedem's comments | 18:46 | |
| mriedem | dansmith: where? | 18:46 |
| dansmith | I probably shouldn't be reviewing this anyway since I was elbow deep in it myself | 18:46 |
| mriedem | i know it's in the release notes | 18:46 |
| mriedem | but it's not in the prelude | 18:46 |
| mriedem | https://docs.openstack.org/releasenotes/nova/unreleased.html#id1 | 18:46 |
| * cdent queues up “prelude of the renos” on the hi-fi | 18:46 | |
| mriedem | A new 1.10 API microversion is added to the Placement REST API. This microversion adds support for the GET /allocation_candidates resource endpoint. This endpoint returns information about possible allocation requests that callers can make which meet a set of resource constraints supplied as query string parameters. Also returned is some inventory and capacity information for the resource providers involved in the allocation | 18:46 |
| mriedem | idates. | 18:46 |
| mriedem | wrong one | 18:47 |
| mriedem | https://docs.openstack.org/releasenotes/nova/unreleased.html#upgrade-notes | 18:47 |
| mriedem | The scheduler now requests allocation candidates from the Placement service during scheduling. The allocation candidates information was introduced in the Placement API 1.10 microversion, so you should upgrade the placement service before the Nova scheduler service so that the scheduler can take advantage of the allocation candidate information. | 18:47 |
| mriedem | the code totally falls back though if 1.10 isn't available | 18:47 |
| mriedem | if we want to make 1.10 required, that's fine, but we need to also update nova-status, which i could do lickety split | 18:48 |
| dansmith | yeah I thought it was in the prelude that I read this morning | 18:48 |
| cdent | I think we should require 1.10 | 18:48 |
| dansmith | there's something that specifically says "be sure to upgrade placement first" | 18:48 |
| dansmith | I really thought that was the prelude | 18:48 |
| mriedem | that's https://docs.openstack.org/releasenotes/nova/unreleased.html#upgrade-notes | 18:48 |
| mriedem | the language there is more clear, the behavior in the scheduler code is not | 18:49 |
| dansmith | ah it's thjs: https://review.openstack.org/#/c/491900/2/doc/source/user/placement.rst | 18:49 |
| dansmith | pike placement upgrade notes | 18:49 |
| cdent | if we haven’t got 1.10, we haven’t got alloc_candidates and we’ve not forced things forward, let’s do that | 18:49 |
| mriedem | i'll update nova-status quick | 18:49 |
| *** nicolasbock has quit IRC | 18:50 | |
| cdent | If nobody gets to it tonight, I can and will look into the stuff where tempest is spewing ERROR tomorrow morning | 18:50 |
| mriedem | it's because we have a small flavor, 1 VCPU | 18:50 |
| dansmith | mriedem: did you say you think the errors during single-host migration come from the bottom patch? | 18:50 |
| mriedem | when we resize on the same host, that subtracts the flavor from itself, resulting in 0 VCPU | 18:50 |
| mriedem | which is an invalid allocation | 18:50 |
| mriedem | dansmith: from the middle patch | 18:51 |
| mriedem | the pep8 is on the bottom patch | 18:51 |
| dansmith | okay I was going to say.. | 18:51 |
| dansmith | ah | 18:51 |
| mriedem | so we'd need some additional floor type logic | 18:51 |
| cdent | isn’t the allocation supposed to be doubling to 2 already? where’s that getting lost? | 18:51 |
| mriedem | such that an allocation can't dip below 0 | 18:51 |
| *** gszasz has joined #openstack-nova | 18:51 | |
| mriedem | cdent: good question | 18:52 |
| mriedem | Instance 6fa7b953-c1fe-4520-8564-aeba8e90aece has resources on 1 compute nodes | 18:52 |
| mriedem | Original resources from same-host allocation: {u'VCPU': 1, u'MEMORY_MB': 64} | 18:52 |
| mriedem | Subtracting old resources from same-host allocation: {u'VCPU': 0, u'MEMORY_MB': 0, 'DISK_GB': 0} | 18:52 |
| mriedem | then kablammo | 18:52 |
| dansmith | mriedem: we're doing the -1 merge which should be subtracting us from the doubled allocation | 18:52 |
| mriedem | yeah the allocation doesn't appear to be doubled | 18:53 |
| mriedem | is the problem | 18:53 |
| dansmith | so | 18:53 |
| dansmith | in that patch, | 18:53 |
| dansmith | we're still healing on both sides with reckless abandon | 18:53 |
| dansmith | does it show up in thelast patch? | 18:53 |
| dansmith | that's where we stop doing that | 18:53 |
| dansmith | although that patch has a typo in it, so probably isn't running properly anyway | 18:54 |
| dansmith | er I mean, had until I updated it a few minutes ago | 18:54 |
| mriedem | umm | 18:56 |
| mriedem | http://logs.openstack.org/10/488510/30/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/5a36c66/logs/screen-n-cpu.txt.gz#_Aug_09_16_27_56_298572 | 18:56 |
| mriedem | Sending updated allocation [{'resource_provider': {'uuid': '97ce076a-2644-465b-95dc-fc0674152976'}, 'resources': {u'VCPU': 0, u'MEMORY_MB': 0, 'DISK_GB': 0}}] for instance 6fa7b953-c1fe-4520-8564-aeba8e90aece after removing resources for 97ce076a-2644-465b-95dc-fc0674152976. {{(pid=19567) remove_provider_from_instance_allocation /opt/stack/new/nova/nova/scheduler/client/report.py:1091}} | 18:56 |
| cdent | ah, so doubling gets cleaned up too soon (until later in the series)? | 18:56 |
| mriedem | oh nvm | 18:57 |
| *** slaweq__ has joined #openstack-nova | 18:57 | |
| mriedem | that's known, i was thinking that was the bottom change | 18:57 |
| *** slaweq_ has quit IRC | 18:57 | |
| *** kristian__ has joined #openstack-nova | 18:58 | |
| *** lpetrut has quit IRC | 18:59 | |
| mriedem | the current_allocs get logged but they don't have 2 VCPU | 18:59 |
| jaypipes | dansmith: are you actively making changes to these patches? if not, I can address mriedem's comments. | 18:59 |
| mriedem | so the doubling isn't happening it appears | 18:59 |
| *** yamamoto has joined #openstack-nova | 19:00 | |
| mriedem | this is the resize in the scheduler http://logs.openstack.org/10/488510/30/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/5a36c66/logs/screen-n-sch.txt.gz#_Aug_09_16_27_37_363471 | 19:00 |
| mriedem | req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 | 19:00 |
| *** felipemonteiro_ has quit IRC | 19:01 | |
| openstackgerrit | Merged openstack/nova master: doc: provide more details on scheduling with placement https://review.openstack.org/491900 | 19:01 |
| *** baoli has quit IRC | 19:01 | |
| mriedem | req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 | 19:01 |
| mriedem | oops | 19:01 |
| mriedem | Aug 09 16:27:37.633395 ubuntu-xenial-internap-mtl01-10347573 nova-scheduler[17280]: DEBUG nova.scheduler.client.report [None req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 tempest-MigrationsAdminTest-402050297 tempest-MigrationsAdminTest-402050297] Doubling-up allocation request for move operation. {{(pid=17280) _move_operation_alloc_request /opt/stack/new/nova/nova/scheduler/client/report.py:162}} | 19:01 |
| mriedem | it says it's double stuffing it | 19:02 |
| openstackgerrit | Merged openstack/nova master: Add a prelude section for Pike https://review.openstack.org/491424 | 19:02 |
| *** baoli has joined #openstack-nova | 19:02 | |
| mriedem | and it looks like it does | 19:02 |
| mriedem | Aug 09 16:27:37.633725 ubuntu-xenial-internap-mtl01-10347573 nova-scheduler[17280]: DEBUG nova.scheduler.client.report [None req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 tempest-MigrationsAdminTest-402050297 tempest-MigrationsAdminTest-402050297] New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'97ce076a-2644-465b-95dc-fc0674152976'}, 'resource | 19:02 |
| mriedem | {u'VCPU': 2, u'MEMORY_MB': 192}}]} {{(pid=17280) _move_operation_alloc_request /opt/stack/new/nova/nova/scheduler/client/report.py:202}} | 19:02 |
| *** kristian__ has quit IRC | 19:02 | |
| openstackgerrit | Merged openstack/nova master: Mark max microversion for Pike in history doc https://review.openstack.org/491581 | 19:02 |
| mriedem | but then i see this | 19:03 |
| mriedem | Aug 09 16:27:37.713717 ubuntu-xenial-internap-mtl01-10347573 nova-scheduler[17280]: DEBUG nova.scheduler.filter_scheduler [None req-7cbe9eeb-dc55-4986-be30-99bd7a03d2c4 tempest-MigrationsAdminTest-402050297 tempest-MigrationsAdminTest-402050297] Successfully claimed resources for instance 6fa7b953-c1fe-4520-8564-aeba8e90aece using allocation request {u'allocations': [{u'resource_provider': {u'uuid': u'97ce076a-2644-465b-95dc- | 19:03 |
| mriedem | 74152976'}, u'resources': {u'VCPU': 1, u'MEMORY_MB': 128}}]} {{(pid=17280) _claim_resources /opt/stack/new/nova/nova/scheduler/filter_scheduler.py:289}} | 19:03 |
| mriedem | which is the single allocation again | 19:03 |
| mriedem | wtf | 19:03 |
| openstackgerrit | Merged openstack/nova master: Document service layout for consoles with cells https://review.openstack.org/491914 | 19:03 |
| mriedem | OHHHHHHHHH | 19:03 |
| mriedem | i know why | 19:03 |
| mriedem | mfing pass by reference | 19:04 |
| openstackgerrit | Merged openstack/nova master: Cleanup release note about ignoring allow_same_net_traffic https://review.openstack.org/491855 | 19:04 |
| jaypipes | mriedem: that's passing by reference. | 19:04 |
| mriedem | dansmith: jaypipes: cdent: problem is right here https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L198 | 19:04 |
| mriedem | return new_alloc_req | 19:04 |
| mriedem | but we're not touching new_alloc_req | 19:04 |
| mriedem | so we double stuff but don't persist | 19:04 |
| jaypipes | yep | 19:05 |
| *** yamamoto has quit IRC | 19:05 | |
| cdent | oops | 19:05 |
| *** markvoelker has quit IRC | 19:06 | |
| *** tbachman has quit IRC | 19:06 | |
| cdent | does any of gibi’s pending functional stuff cover that? | 19:06 |
| mriedem | not sure | 19:06 |
| jaypipes | func or unit should have covered this... | 19:06 |
| mriedem | anyway, i'll get this nova-status thing done | 19:06 |
| * jaypipes checks | 19:06 | |
| mriedem | and then could push a fix for that on top | 19:06 |
| jaypipes | mriedem: k, I'll fix the above. | 19:07 |
| mriedem | and then we rebase the series on top of those | 19:07 |
| mriedem | or that | 19:07 |
| jaypipes | mriedem: you tell me. what do you want? | 19:07 |
| cdent | I really need to go, but please, if stuff’s not stable by the time you guys expire tonight, let me know so I can continue it in the morning | 19:07 |
| jaypipes | mriedem: I can tack on a fix on the bottom of this series? | 19:07 |
| mriedem | jaypipes: yes it would have to be | 19:07 |
| mriedem | note i've got other issues in that 2nd change though, | 19:08 |
| mriedem | which i guess could be follow ups | 19:08 |
| *** krtaylor has joined #openstack-nova | 19:08 | |
| mriedem | i'm just kind of getting tired of making myself TODOs about chasing follow ups | 19:08 |
| mriedem | but realize we're short on time | 19:08 |
| *** markvoelker has joined #openstack-nova | 19:08 | |
| jaypipes | mriedem: I'm happy to address your comments on the second patch. just need to know if dansmith is actively working on these. | 19:08 |
| dansmith | nope | 19:09 |
| jaypipes | dansmith: gotcha. I'll get on this then. | 19:09 |
| cdent | k, in that case I’ll look at the existing main stack in the morning and see where we’re at | 19:10 |
| cdent | g’night | 19:10 |
| mriedem | o/ | 19:10 |
| *** cdent has quit IRC | 19:11 | |
| *** tbachman has joined #openstack-nova | 19:11 | |
| mriedem | so i think unit tests are covering _move_operation_alloc_request | 19:11 |
| mriedem | New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'97ce076a-2644-465b-95dc-fc0674152976'}, 'resources': {u'VCPU': 2, u'MEMORY_MB': 192}}]} | 19:11 |
| mriedem | that's from that same method, | 19:11 |
| mriedem | after merging the allocations | 19:11 |
| mriedem | by reference | 19:11 |
| mriedem | ah a red herring | 19:12 |
| mriedem | Successfully claimed resources for instance 6fa7b953-c1fe-4520-8564-aeba8e90aece using allocation request {u'allocations': [{u'resource_provider': {u'uuid': u'97ce076a-2644-465b-95dc-fc0674152976'}, u'resources': {u'VCPU': 1, u'MEMORY_MB': 128}}]} {{(pid=17280) _claim_resources /opt/stack/new/nova/nova/scheduler/filter_scheduler.py:289}} | 19:12 |
| jaypipes | not for same-host-resize, though, right? | 19:12 |
| mriedem | ^ is stale | 19:12 |
| mriedem | no we doubled correctly | 19:13 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L201 | 19:13 |
| mriedem | New allocation request containing both source and destination hosts in move operation: {'allocations': [{'resource_provider': {'uuid': u'97ce076a-2644-465b-95dc-fc0674152976'}, 'resources': {u'VCPU': 2, u'MEMORY_MB': 192}}]} | 19:13 |
| mriedem | that's doubled | 19:13 |
| *** ys__ has quit IRC | 19:13 | |
| *** slaweq_ has joined #openstack-nova | 19:13 | |
| mriedem | what we're logging here: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L293 | 19:13 |
| mriedem | is stale | 19:13 |
| mriedem | that's why i got confused | 19:13 |
| mriedem | however, when we get to the RT, and subtract, we lost the doubled allocation somewhere still | 19:14 |
| mriedem | jaypipes: so i don't know what the bug is yet | 19:14 |
| mriedem | we just have really misleading logging | 19:14 |
| *** MVenesio has quit IRC | 19:14 | |
| jaypipes | mriedem: I don't agree that the above line is "stale". | 19:15 |
| mriedem | of course it is | 19:15 |
| mriedem | alloc_req comes from placement | 19:15 |
| mriedem | we double it | 19:15 |
| *** baoli has quit IRC | 19:15 | |
| mriedem | then we log the original alloc_req from placement | 19:15 |
| mriedem | which is not double | 19:15 |
| *** slaweq__ has quit IRC | 19:15 | |
| mriedem | follow the 4 log messages starting here http://logs.openstack.org/10/488510/30/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/5a36c66/logs/screen-n-sch.txt.gz#_Aug_09_16_27_37_563749 | 19:15 |
| *** MVenesio has joined #openstack-nova | 19:16 | |
| *** kbaegis1 has quit IRC | 19:16 | |
| *** kbaegis has joined #openstack-nova | 19:16 | |
| *** kristia__ has joined #openstack-nova | 19:16 | |
| jaypipes | mriedem: yeah, right. | 19:17 |
| mriedem | so, we're still losing the doubled allocation somewhere | 19:17 |
| mriedem | my guess is the RT is overwriting it? | 19:17 |
| jaypipes | mriedem: I will change placement client claim_resources() to return the possibly-updated-for-move-operation alloc_req. | 19:17 |
| mriedem | jaypipes: we don't have to change all of that, just drop alloc_req from this log message https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L293 | 19:18 |
| mriedem | we already logged the original and new thing | 19:18 |
| *** MVenesio has quit IRC | 19:18 | |
| jaypipes | k | 19:18 |
| mriedem | well, we logged the new thing | 19:18 |
| *** adisky__ has quit IRC | 19:20 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Require Placement 1.0 in nova-status upgrade check https://review.openstack.org/492234 | 19:23 |
| *** kristia__ has quit IRC | 19:25 | |
| mriedem | dansmith: ^ thar she blar | 19:25 |
| *** kristia__ has joined #openstack-nova | 19:25 | |
| sdague | mriedem: commit message? | 19:26 |
| sdague | 1.10 right? | 19:26 |
| dansmith | mriedem: I got distracted by a rant opportunity.. did you figure out the doubling thing? | 19:26 |
| mriedem | crap | 19:26 |
| jaypipes | dansmith: still working on it. | 19:26 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Require Placement 1.10 in nova-status upgrade check https://review.openstack.org/492234 | 19:27 |
| mriedem | dansmith: are you ranting in the ops midcycle upgrades etherpad? | 19:27 |
| mriedem | perhaps | 19:27 |
| mriedem | oh -dev, | 19:27 |
| mriedem | i see | 19:27 |
| dansmith | yes | 19:27 |
| dansmith | plus screaming obscenities at the monitor that only Taylor and I can hear | 19:27 |
| dansmith | I can tell it's bad when she just starts slipping me candy to calm me down | 19:28 |
| jaypipes | lol | 19:29 |
| *** sbezverk has quit IRC | 19:29 | |
| *** kristia__ has quit IRC | 19:30 | |
| dansmith | jaypipes: "working on it" meaning "working on figuring it out" or "working on putting the fix into code" ? | 19:30 |
| mriedem | we don't have it figured out | 19:30 |
| mriedem | where the doubled allocation went | 19:30 |
| jaypipes | dansmith: what mriedem said. | 19:30 |
| dansmith | and it's not what I said? | 19:30 |
| dansmith | that healing is still blind in this patch | 19:30 |
| mriedem | it might bethat | 19:30 |
| mriedem | that's what i suspect anyway | 19:30 |
| mriedem | one node is ovewriting the doubled allocation | 19:31 |
| mriedem | i should be able to tell that from the logs | 19:31 |
| dansmith | well we're 0.25 days away from getting a run of the top one anyway | 19:31 |
| openstackgerrit | Sean Dague proposed openstack/nova master: doc: Address review comments for contributor index https://review.openstack.org/491517 | 19:32 |
| sdague | dansmith: what's the non gate exposure of this? | 19:33 |
| dansmith | sdague: what? | 19:33 |
| *** ociuhandu has joined #openstack-nova | 19:33 | |
| sdague | like which jobs are showing the issue | 19:33 |
| dansmith | none of them are failing | 19:33 |
| dansmith | if that is what you mean | 19:33 |
| sdague | dansmith: one of them is showing a funny allocation though? | 19:34 |
| dansmith | I imagine it's the single-node tempest one that's hitting the issue though, although theoretically the multinode ones should too | 19:34 |
| dansmith | sdague: logging errors as they fail to do their accounting, but nothing is fatal from tempest's point of view it sounds like | 19:34 |
| sdague | just thinking if it's more effective to do local run / debug | 19:34 |
| sdague | given the gate turn around time isn't going to get better any time soon | 19:34 |
| dansmith | probably, but I have so little time left, it'd take me all that to just stack once and start looking I think | 19:35 |
| mriedem | gd we love to lazy-load pci_requests and pci_devices | 19:35 |
| dansmith | I assume jaypipes has been running this locally | 19:35 |
| jaypipes | dansmith: not in the last few weeks, no. been relying on functional tests and logging. | 19:36 |
| jaypipes | mriedem: ok if I remove that log message in scheduler _claim_resources() (with the stale alloc_req) in the bottom patch of this series? the patch call "refactor heal..." | 19:39 |
| jaypipes | nm, I'll just throw it in another patch | 19:39 |
| jaypipes | patches are cheap. | 19:39 |
| mriedem | http://logs.openstack.org/10/488510/30/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/5a36c66/logs/screen-n-cpu.txt.gz#_Aug_09_16_27_46_100320 | 19:39 |
| mriedem | yeah so on resize to same host, it's the _instance_to_allocations_dict that overwrites the doubled allocation | 19:40 |
| mriedem | dansmith: which is what you were saying | 19:40 |
| mriedem | Sending allocation for instance {'VCPU': 1, 'MEMORY_MB': 64} {{(pid=19567) _allocate_for_instance /opt/stack/new/nova/nova/scheduler/client/report.py:924}} | 19:40 |
| *** kristia__ has joined #openstack-nova | 19:40 | |
| dansmith | yeah | 19:40 |
| jaypipes | mriedem: right, and that's the thing that doens't get "corrected" until the later patch | 19:41 |
| mriedem | that's at Aug 09 16:27:46.100320 | 19:41 |
| mriedem | the doubled up allocation in the scheduler was at Aug 09 16:27:37.633725 | 19:41 |
| mriedem | ok so maybe not a turd furgeson after all | 19:41 |
| mriedem | although, an ocata compute will trample that and then the last patch will double the allocations again? | 19:42 |
| dansmith | mriedem: while we're healing, | 19:42 |
| dansmith | we'll fix it eventually on the destination node | 19:42 |
| dansmith | once we're all pike, we don't trample and thus we're good | 19:43 |
| *** rmcall has joined #openstack-nova | 19:43 | |
| mriedem | but we'll continue to log ERRORs? | 19:44 |
| dansmith | I dunno I didn't look at the actual site where we log that, but probably so | 19:44 |
| mriedem | ok, and we probably don't have a job that would test it either, | 19:45 |
| mriedem | would require a grenade multinode job that runs a migration | 19:45 |
| mriedem | we do have a multinode grenade job that runs live migration... | 19:45 |
| mriedem | back and forth too so that actually might show it | 19:45 |
| dansmith | mriedem: you can borrow my pistol when I'm done with it | 19:45 |
| mriedem | i'll be classy and do Seppuku | 19:46 |
| dansmith | heh, less mess, how midwestern of you | 19:46 |
| mriedem | disemboweling is still a mess | 19:47 |
| mriedem | i'll do it in the soaker tub | 19:47 |
| mriedem | this error log case wouldn't be the end of the world, we could release with that and backport a fix later, | 19:48 |
| mriedem | adding some logic to determine we're about to push a 0 allocation and just not do that | 19:48 |
| *** Sukhdev_ has quit IRC | 19:48 | |
| dansmith | well, normally that would be a legit error case, | 19:49 |
| dansmith | and worth the log | 19:49 |
| *** gbarros has quit IRC | 19:49 | |
| dansmith | so I dunno.. maybe if not has_ocata and ==0, then log error or something | 19:49 |
| mriedem | my point is, it doesn't cause things to fail, so not a regression, so we could cleanup the error log later and backport if necessary, yeah? | 19:51 |
| dansmith | yes | 19:51 |
| *** kristia__ has quit IRC | 19:51 | |
| openstackgerrit | Merged openstack/nova-specs master: doc: Remove crud from Sphinx conf.py https://review.openstack.org/456129 | 19:51 |
| *** kristia__ has joined #openstack-nova | 19:52 | |
| *** baoli has joined #openstack-nova | 19:52 | |
| mriedem | jaypipes: so you're putting a patch at the bottom of the series to remove that bogus log in the filter scheduler, and then fixing the pep8 in https://review.openstack.org/#/c/491850/ on top of that? | 19:52 |
| mriedem | and then i think we can limp https://review.openstack.org/#/c/488510/ in and fix my other problems in a follow up | 19:52 |
| mriedem | and then once we get a good run on https://review.openstack.org/#/c/491012/ we call it a ball for rc1 | 19:53 |
| mriedem | oh except this of course https://review.openstack.org/#/c/491491/ | 19:53 |
| dansmith | jay needs to look at that one I think | 19:53 |
| dansmith | it makes sense to me, but I figure he should look | 19:53 |
| mriedem | gibi's? | 19:53 |
| mriedem | yes agree | 19:53 |
| dansmith | yeah | 19:53 |
| mriedem | he wrote it | 19:53 |
| jaypipes | I'm still addressing mriedem's review comments on the confirm/reverrt resize patch. | 19:54 |
| mriedem | i mean, the code that's changing | 19:54 |
| jaypipes | should be done shortly. | 19:54 |
| dansmith | mriedem: right | 19:54 |
| mriedem | although it does say, "this should really raise NoValidHost but tests...." | 19:54 |
| *** baoli has quit IRC | 19:55 | |
| *** baoli has joined #openstack-nova | 19:55 | |
| *** kristia__ has quit IRC | 19:56 | |
| *** priteau has quit IRC | 19:58 | |
| *** moshele has joined #openstack-nova | 19:58 | |
| mriedem | what i don't understand in his test https://review.openstack.org/#/c/490814/ is how the resize doesn't fail on the compute during the claim | 19:58 |
| mriedem | the claim should fail | 19:58 |
| dansmith | allocation ratio? | 20:00 |
| dansmith | he's not setting it so I would expect it's defaulting to 16 or whatever | 20:00 |
| dansmith | for vcpu | 20:00 |
| mriedem | default=0.0, | 20:00 |
| dansmith | which means 16 | 20:00 |
| mriedem | if set to 0.0, the value | 20:00 |
| mriedem | set on the scheduler node(s) or compute node(s) will be used | 20:00 |
| mriedem | and defaulted to 16.0. | 20:00 |
| mriedem | yeah | 20:00 |
| * dansmith pats us all on the back for being awesome | 20:01 | |
| mriedem | haha | 20:01 |
| mriedem | i'll pull his test down, set that to 1.0 and make sure it fails the resize, and then if that's the case i'll be cool with the test | 20:01 |
| *** yamamoto has joined #openstack-nova | 20:02 | |
| dansmith | so, um | 20:02 |
| dansmith | we really fall back to legacy scheduling in that case? | 20:02 |
| dansmith | I don't think I knew that was the plan | 20:02 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: placement: refactor healing of allocations in RT https://review.openstack.org/491850 | 20:03 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Remove provider allocs in confirm/revert resize https://review.openstack.org/488510 | 20:03 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Resource tracker compatibility with Ocata and Pike https://review.openstack.org/491012 | 20:03 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: remove log message with potential stale info https://review.openstack.org/492242 | 20:03 |
| jaypipes | mriedem, dansmith: ^. | 20:03 |
| mriedem | well we wouldn't with gibi's patch | 20:03 |
| dansmith | jaypipes: I'm not reviewing any more things today unless mriedem +2s them first | 20:04 |
| dansmith | my stats and ego just can't take it | 20:04 |
| *** liverpooler has quit IRC | 20:04 | |
| *** sbezverk has joined #openstack-nova | 20:04 | |
| mriedem | if it makes you feel better, maya told laura that she didn't want to go to camp this morning b/c she thought i'd forget her again | 20:04 |
| *** abalutoiu has joined #openstack-nova | 20:06 | |
| *** gbarros has joined #openstack-nova | 20:06 | |
| dansmith | hah that's pretty awesome | 20:06 |
| jaypipes | lol Dad of the Yea. | 20:06 |
| *** yamamoto has quit IRC | 20:06 | |
| jaypipes | Year | 20:06 |
| *** gbarros has quit IRC | 20:07 | |
| dansmith | when camp called, I'd have said "oh uh yeah I "forgot" and stuff" | 20:07 |
| mriedem | i can't remember if i swore when the front desk lady called me | 20:07 |
| mriedem | "oh shit!" | 20:07 |
| mriedem | "lemme put the bong down and i'll be right over" | 20:08 |
| mriedem | hmm, setting cpu_allocation_ratio=1.0 in gibi's test doesn't make it fail | 20:08 |
| dims | mriedem : ouch! | 20:08 |
| dansmith | mriedem: nice | 20:09 |
| dansmith | mriedem: set it before anything gets started? | 20:09 |
| mriedem | yeah, in setUp | 20:09 |
| * mnaser proposes periodic job to openstack-infra/project-config to remind mriedem to pick up kids from camp | 20:09 | |
| mriedem | before setting the virt driver | 20:09 |
| dansmith | mriedem: before compute or scheduler starts? | 20:09 |
| *** slaweq_ has quit IRC | 20:10 | |
| mriedem | yes before everything starts | 20:10 |
| dansmith | but wait, | 20:10 |
| dansmith | before _everything_ starts? like all thethings? | 20:10 |
| mriedem | http://paste.openstack.org/show/617983/ | 20:10 |
| mriedem | so you're just <1 from 4 day weekend and no longer caring, i see | 20:11 |
| dansmith | no mriedem I mean BEFORE EVERYTHING | 20:11 |
| dansmith | haha | 20:11 |
| mriedem | *1 hour | 20:11 |
| dansmith | don't worry, this four day weekend will be a special kind of hell where I'll be begging to work on unit tests by mid-day tomorrow | 20:11 |
| *** slaweq_ has joined #openstack-nova | 20:14 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use ksa adapter for placement conf & requests https://review.openstack.org/492247 | 20:15 |
| mriedem | dansmith: i'm +2 on this https://review.openstack.org/#/c/492242/ | 20:16 |
| *** suresh12_ has joined #openstack-nova | 20:18 | |
| mriedem | ok bottom 2 are approved | 20:19 |
| *** suresh12 has quit IRC | 20:21 | |
| *** links has joined #openstack-nova | 20:25 | |
| mriedem | jaypipes: easy one https://review.openstack.org/#/c/488510/ | 20:26 |
| *** moshele has quit IRC | 20:27 | |
| *** slaweq__ has joined #openstack-nova | 20:28 | |
| dansmith | mriedem: I bet it's because of the fake virt driver | 20:28 |
| dansmith | 2017-08-09 13:27:58,682 INFO [nova.compute.claims] Total vcpu: 10 VCPU, used: 0.00 VCPU | 20:28 |
| dansmith | 2017-08-09 13:27:58,682 INFO [nova.compute.claims] vcpu limit not specified, defaulting to unlimited | 20:28 |
| *** sbezverk has quit IRC | 20:29 | |
| *** smatzek has quit IRC | 20:30 | |
| *** slaweq_ has quit IRC | 20:31 | |
| edmondsw | efried see the comment I just added to https://review.openstack.org/#/c/485121/7 and let me know if you have any ideas there | 20:31 |
| dansmith | actually we're getting an empty dict of limits in filter_properties | 20:31 |
| efried | edmondsw Ack. | 20:32 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Remove provider allocs in confirm/revert resize https://review.openstack.org/488510 | 20:32 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Resource tracker compatibility with Ocata and Pike https://review.openstack.org/491012 | 20:32 |
| jaypipes | mriedem: fixe.d | 20:32 |
| mriedem | dansmith: yeah, because the only limits we'd get are from NUMATopologyFilter | 20:32 |
| dansmith | but our limit for vcpus is in there | 20:33 |
| dansmith | otherwise we won't check anything, AFAICT | 20:33 |
| mriedem | oh i see, | 20:34 |
| dansmith | mriedem: we're hitting this code: https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L240-L244 | 20:34 |
| mriedem | CoreFilter puts the vcpu limit in there | 20:34 |
| dansmith | hah | 20:34 |
| mriedem | right | 20:34 |
| dansmith | lemme try with ram or something else | 20:35 |
| mriedem | since the CoreFilter isn't enabled, as noted in the commit message, we don't care about filtering on cores... | 20:35 |
| *** markvoelker has quit IRC | 20:35 | |
| dansmith | must be same with ramfilter? | 20:36 |
| dansmith | because obviously I don't hit that either since no limit | 20:36 |
| mriedem | the flavor he's using fits the ram on the fake driver | 20:36 |
| dansmith | I mean making it not | 20:36 |
| dansmith | yeah novalidhost when I do that and enable ramfilter | 20:36 |
| *** shan has quit IRC | 20:37 | |
| *** suresh12_ has quit IRC | 20:37 | |
| mriedem | ok +2 on his test then https://review.openstack.org/#/c/490814/ | 20:38 |
| mriedem | i didn't realize the enabled filters impacted the claim code in the compute | 20:38 |
| *** rajathagasthya has joined #openstack-nova | 20:39 | |
| *** suresh12 has joined #openstack-nova | 20:39 | |
| dansmith | so I didn't realize the filters controlled the compute node claiming behavior | 20:39 |
| dansmith | not sure that really makes sense | 20:39 |
| mriedem | did you just say the same thing as me? | 20:40 |
| dansmith | however, it's nice to know that removing those base filters will stop the compute node from defeating placement's decisions | 20:40 |
| dansmith | two things to set limits with different logic will definitely lead to confusion-based bugs | 20:40 |
| dansmith | now I just gotta find a typo nit in here to get my stat back and we're good to go | 20:41 |
| *** markvoelker has joined #openstack-nova | 20:41 | |
| dansmith | found one | 20:42 |
| mriedem | which change are you talking about? | 20:42 |
| mriedem | oh gibi's patch | 20:42 |
| dansmith | the test | 20:42 |
| * dansmith wrings his hands before laying down the -1 | 20:42 | |
| mriedem | removed my +2 just in time | 20:43 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: replace chance with filter scheduler in func tests https://review.openstack.org/491529 | 20:43 |
| dansmith | I was just bluffing | 20:43 |
| dansmith | I did find one but it's not worth delaying over | 20:43 |
| mriedem | d'oh | 20:43 |
| mriedem | heh replaying my +2 | 20:44 |
| *** marst_ has joined #openstack-nova | 20:44 | |
| mriedem | are you happy with cfriesen's wording here then? https://review.openstack.org/#/c/491854/ | 20:44 |
| *** dtp has joined #openstack-nova | 20:45 | |
| dansmith | mriedem: I wouldn't say I'm happy | 20:45 |
| mriedem | jaypipes: i'm not sure why i'm the author on this https://review.openstack.org/#/c/488510/3 | 20:46 |
| mriedem | https://review.openstack.org/#/c/488510/ | 20:46 |
| *** jmlowe has quit IRC | 20:46 | |
| dansmith | mriedem: I've noticed that on a couple things you've updated recently | 20:46 |
| dansmith | resetting the author for some reason I mean | 20:46 |
| mriedem | i didn't update this | 20:46 |
| jaypipes | yeah, me too... | 20:46 |
| jaypipes | haven't cared much, though. :) | 20:46 |
| dansmith | mriedem: you did, PS15 | 20:47 |
| dansmith | and 16 | 20:47 |
| mriedem | yeah it was PS15 | 20:47 |
| mriedem | jaypipes: you want to fix that quick? | 20:47 |
| jaypipes | mriedem: I certainly didn't change the author. | 20:47 |
| jaypipes | mriedem: how do I do that? | 20:47 |
| *** marst has quit IRC | 20:47 | |
| mriedem | git commit --amend --author "Jay Pipes <jaypipes@gmail.com>" | 20:47 |
| dansmith | are we re-pushing this anyway? | 20:47 |
| *** baoli has quit IRC | 20:48 | |
| dansmith | because ... six hour check run | 20:48 |
| jaypipes | gimme a sec, will do. | 20:48 |
| mriedem | dansmith: i brought it up in the ML http://lists.openstack.org/pipermail/openstack-dev/2017-July/119801.html | 20:48 |
| mriedem | i don't know what causes it automatically, something to do with git review -d + git rebase -i + git commit | 20:48 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Remove provider allocs in confirm/revert resize https://review.openstack.org/488510 | 20:48 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Resource tracker compatibility with Ocata and Pike https://review.openstack.org/491012 | 20:48 |
| jaypipes | ok, mriedem done | 20:48 |
| dansmith | mriedem: I do that all the time and it doesn't reset for me | 20:49 |
| *** randomhack has quit IRC | 20:49 | |
| dansmith | I've had my way with all of jay's patches that way | 20:49 |
| *** cleong has quit IRC | 20:49 | |
| *** baoli has joined #openstack-nova | 20:49 | |
| mriedem | git version 2.7.4 ? | 20:49 |
| dansmith | indeed actually | 20:50 |
| mriedem | git-review Version: 1.25.0 | 20:50 |
| dansmith | same | 20:50 |
| dansmith | I'm not running Windows98 or whatever you run though | 20:50 |
| mriedem | this is xenial 16.04 | 20:51 |
| *** markvoelker has quit IRC | 20:52 | |
| * dansmith doesn't believe mriedem | 20:52 | |
| mriedem | huh | 20:52 |
| mriedem | user@ubuntu:~/git/nova$ cat /etc/debian_version | 20:52 |
| mriedem | stretch/sid | 20:52 |
| jaypipes | mriedem: quick summary of the cause of the "claim mystery" from that resize too big patch please? | 20:52 |
| dansmith | expected :) | 20:52 |
| dansmith | jaypipes: he put it in the comments | 20:53 |
| dansmith | jaypipes: corefilter notenabled == no limit == no vcpu check on the compute node | 20:53 |
| jaypipes | ah | 20:53 |
| *** markmc` has joined #openstack-nova | 20:53 | |
| dansmith | I'm embarrassed to say I didn't know we had that linkage | 20:53 |
| dansmith | and also, am horrified | 20:53 |
| mriedem | dansmith: like how i gave you credit in those comments? | 20:54 |
| mriedem | does your ego feel better? | 20:54 |
| *** burt has quit IRC | 20:54 | |
| dansmith | mriedem: yes, that's why I didn't NFO your stats over a comment grammar issue :) | 20:54 |
| *** markmc` has quit IRC | 20:54 | |
| mriedem | naval flight officer? | 20:54 |
| *** markmc has quit IRC | 20:55 | |
| *** baoli has quit IRC | 20:55 | |
| dansmith | nuke from orbit | 20:55 |
| dansmith | maybe not a fun analogy these days I guess | 20:55 |
| dansmith | as PDX has been one of the west coast targets mentioned | 20:55 |
| mriedem | heh, fire and fury baby | 20:55 |
| mriedem | urban dictionary has a fun one | 20:56 |
| mriedem | "Near Fatal Orgasm" | 20:56 |
| *** markmc has joined #openstack-nova | 20:56 | |
| * dansmith steps a few paces to the right | 20:56 | |
| *** links has quit IRC | 20:56 | |
| *** jpena is now known as jpena|off | 20:56 | |
| *** markvoelker has joined #openstack-nova | 20:59 | |
| mriedem | jaypipes: ok +2 on gibi's fix https://review.openstack.org/#/c/491491/ but letting you take the final gander | 21:00 |
| jaypipes | mriedem: k, reviewing. | 21:00 |
| *** dr_gogeta86 has quit IRC | 21:00 | |
| *** slaweq_ has joined #openstack-nova | 21:00 | |
| *** lucasxu has quit IRC | 21:01 | |
| *** krtaylor has quit IRC | 21:01 | |
| jaypipes | mriedem: done | 21:02 |
| *** markmc has quit IRC | 21:02 | |
| dansmith | I hope all these gaping holes that gibi has found in the last two weeks are coming from him poking at it for reals | 21:03 |
| *** markmc has joined #openstack-nova | 21:03 | |
| *** vstinner has joined #openstack-nova | 21:03 | |
| *** yamamoto_ has joined #openstack-nova | 21:03 | |
| dansmith | and if so, kudos | 21:03 |
| mriedem | cfriesen: something you should probably fix in here https://review.openstack.org/#/c/491854/ | 21:03 |
| mriedem | yes gibi gets the last minute testing champion award | 21:04 |
| cfriesen | dansmith: what changes would you like in https://review.openstack.org/#/c/491854/ ? | 21:04 |
| *** slaweq__ has quit IRC | 21:04 | |
| mriedem | cfriesen: see the comment i just left | 21:04 |
| *** suresh12 has quit IRC | 21:04 | |
| *** krtaylor has joined #openstack-nova | 21:05 | |
| cfriesen | mriedem: I can fix that easy. Sounded like Dan wanted something else though as well. | 21:06 |
| *** suresh12 has joined #openstack-nova | 21:06 | |
| mriedem | no he was just being dan | 21:06 |
| cfriesen | oh, in that case | 21:07 |
| *** vstinner has left #openstack-nova | 21:07 | |
| mriedem | dansmith: per your question about testing in https://review.openstack.org/#/c/487954/ - there is a change that depends on it here: https://review.openstack.org/#/c/476968/ | 21:07 |
| *** thingee_ has joined #openstack-nova | 21:08 | |
| *** gszasz has quit IRC | 21:08 | |
| efried | edmondsw Do you know how we're importing the keystone_authtoken conf group?? | 21:08 |
| *** yamamoto_ has quit IRC | 21:08 | |
| openstackgerrit | Chris Friesen proposed openstack/nova master: Remove ram/disk sched filters from default list https://review.openstack.org/491854 | 21:09 |
| *** catintheroof has quit IRC | 21:09 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove ram/disk sched filters from default list https://review.openstack.org/491854 | 21:10 |
| *** markmc has quit IRC | 21:10 | |
| cfriesen | mriedem: oh crud. :) | 21:11 |
| mriedem | fixed | 21:11 |
| mriedem | approved | 21:11 |
| edmondsw | efried I think it's this: https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_opts.py#L201 | 21:13 |
| efried | edmondsw Where is that accessed by nova, though? | 21:13 |
| edmondsw | looks like for some reason ks_loading is used to register auth options but not session options | 21:13 |
| edmondsw | efried the authtoken middleware is in the api pipeline | 21:14 |
| *** markmc has joined #openstack-nova | 21:14 | |
| efried | what does that mean? | 21:14 |
| *** gjayavelu has quit IRC | 21:15 | |
| edmondsw | efried https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini#L87 | 21:15 |
| *** rajathagasthya has quit IRC | 21:16 | |
| edmondsw | note "authtoken" here: https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini#L32 | 21:16 |
| *** yamamoto_ has joined #openstack-nova | 21:16 | |
| efried | edmondsw Brilliant. What I'm trying to find is some place in nova that we can override and/or deprecate the offending opt. | 21:16 |
| edmondsw | efried I don't think there is one... in nova. It would be in keystonemiddleware | 21:17 |
| efried | But since we don't actually have a nova/conf presence for keystone_authtoken, it would have to go in, like, workarounds or something. | 21:17 |
| *** rtjure has quit IRC | 21:17 | |
| efried | edmondsw That's probably uglier than just hacking it up the way it originally was. | 21:18 |
| efried | Though if you do that, shove in a NOTE explaining why load_session_from_conf_options doesn't work. | 21:18 |
| edmondsw | yeah, that was my thought | 21:18 |
| efried | edmondsw Unless there's a different conf group you could pull that cafile from... | 21:19 |
| edmondsw | and I can bring it up with the keystone guys so they're aware... maybe they'll want to change something, maybe not | 21:19 |
| edmondsw | we were really trying to use keystone_authtoken for more than what it's designed for, so... | 21:19 |
| efried | edmondsw Yeah, perhaps a bug. | 21:19 |
| *** jmlowe has joined #openstack-nova | 21:20 | |
| efried | Oh, well, in that case, is there a more appropriate place you could/should register real session opts? | 21:20 |
| edmondsw | don't know of any other conf options that would already have this. We could create a new [keystone] section nova.conf like we have for glance, neutron, etc. | 21:20 |
| edmondsw | that would be the *right* way to do it | 21:20 |
| edmondsw | seems very duplicative, though | 21:20 |
| edmondsw | more work for operators | 21:21 |
| *** yamamoto_ has quit IRC | 21:21 | |
| efried | edmondsw Call it [identity], make mordred happy. | 21:21 |
| mordred | I didn't do it | 21:21 |
| edmondsw | lol | 21:21 |
| edmondsw | I almost said "identity", and then remembered all the current sections are using codenames | 21:22 |
| edmondsw | odd that nobody's changed that already... | 21:22 |
| efried | edmondsw Yeah, I think mordred wants to push for that eventually. | 21:22 |
| efried | edmondsw If we talk about it in -keystone and decide it's really a bug in ksm, then we could justify leaving it in [keystone_authtoken] and making a delta in nova.conf.workarounds to... work around it. | 21:23 |
| edmondsw | efried sure, let's do that tomorrow, I have to run | 21:23 |
| efried | rgr | 21:24 |
| *** crushil has quit IRC | 21:24 | |
| *** rajathagasthya has joined #openstack-nova | 21:24 | |
| *** kbaegis1 has joined #openstack-nova | 21:26 | |
| *** kbaegis has quit IRC | 21:27 | |
| *** edmondsw has quit IRC | 21:28 | |
| *** edmondsw has joined #openstack-nova | 21:29 | |
| *** derekh has quit IRC | 21:29 | |
| *** thorst has quit IRC | 21:31 | |
| *** edmondsw has quit IRC | 21:34 | |
| *** slaweq_ has quit IRC | 21:36 | |
| *** tyrefors has quit IRC | 21:42 | |
| *** priteau has joined #openstack-nova | 21:47 | |
| *** priteau has quit IRC | 21:47 | |
| *** esberglu has quit IRC | 21:48 | |
| *** esberglu has joined #openstack-nova | 21:48 | |
| *** Sukhdev_ has joined #openstack-nova | 21:48 | |
| *** thorst has joined #openstack-nova | 21:50 | |
| *** kbaegis1 has quit IRC | 21:50 | |
| *** kbaegis has joined #openstack-nova | 21:52 | |
| *** gouthamr has quit IRC | 21:52 | |
| *** esberglu has quit IRC | 21:53 | |
| mriedem | dansmith: jaypipes: ok i went through https://review.openstack.org/#/c/491012/ and posted questions, several about evacuate from an ocata compute - i know dan is MIA now and i have to head out too, but will be back around later and tomorrow to discuss anything | 21:54 |
| *** itlinux has quit IRC | 21:54 | |
| *** thorst has quit IRC | 21:55 | |
| *** yamamoto_ has joined #openstack-nova | 21:57 | |
| *** xyang1 has quit IRC | 22:00 | |
| *** vipul has quit IRC | 22:04 | |
| *** dklyle has joined #openstack-nova | 22:06 | |
| *** david-lyle has quit IRC | 22:06 | |
| *** vipul has joined #openstack-nova | 22:10 | |
| *** kylek3h has quit IRC | 22:10 | |
| *** esberglu has joined #openstack-nova | 22:12 | |
| *** abalutoiu has quit IRC | 22:12 | |
| *** rcernin has quit IRC | 22:20 | |
| *** Apoorva_ has joined #openstack-nova | 22:20 | |
| *** sbezverk has joined #openstack-nova | 22:20 | |
| *** Sukhdev has quit IRC | 22:21 | |
| *** Apoorva has quit IRC | 22:23 | |
| *** Sukhdev has joined #openstack-nova | 22:35 | |
| *** awaugama has quit IRC | 22:36 | |
| *** kristian__ has joined #openstack-nova | 22:40 | |
| *** kristian__ has quit IRC | 22:44 | |
| *** dave-mccowan has quit IRC | 22:44 | |
| *** kornicameister has quit IRC | 22:50 | |
| *** gcb has joined #openstack-nova | 22:51 | |
| *** krtaylor has quit IRC | 22:52 | |
| *** sdague has quit IRC | 22:53 | |
| *** kornicameister has joined #openstack-nova | 22:55 | |
| *** itlinux has joined #openstack-nova | 23:00 | |
| *** lyan has quit IRC | 23:04 | |
| *** armax has quit IRC | 23:05 | |
| *** abalutoiu has joined #openstack-nova | 23:19 | |
| *** mtanino has quit IRC | 23:19 | |
| *** gcb has quit IRC | 23:21 | |
| *** itlinux has quit IRC | 23:21 | |
| *** gcb has joined #openstack-nova | 23:21 | |
| *** claudiub has quit IRC | 23:32 | |
| *** suresh12 has quit IRC | 23:32 | |
| *** chyka has quit IRC | 23:33 | |
| *** hongbin has quit IRC | 23:37 | |
| *** krtaylor has joined #openstack-nova | 23:40 | |
| *** jaypipes has quit IRC | 23:44 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Avoid chowning console logs in libvirt https://review.openstack.org/472229 | 23:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: First attempt at adding a privsep user to nova itself. https://review.openstack.org/459166 | 23:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move execs of touch to privsep. https://review.openstack.org/489190 | 23:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move libvirts dmcrypt support to privsep. https://review.openstack.org/490737 | 23:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move execs of tee to privsep. https://review.openstack.org/489438 | 23:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move libvirt usages of chown to privsep. https://review.openstack.org/471972 | 23:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486 | 23:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Refactor libvirt.utils.execute() away. https://review.openstack.org/489816 | 23:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move ploop commands to privsep. https://review.openstack.org/492325 | 23:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree() https://review.openstack.org/492326 | 23:45 |
| *** marst_ has quit IRC | 23:47 | |
| *** markvoelker has quit IRC | 23:49 | |
| *** gyee has quit IRC | 23:50 | |
| *** dtp has quit IRC | 23:52 | |
| *** diga has joined #openstack-nova | 23:55 | |
| *** gcb has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!