| *** jaosorior has quit IRC | 00:00 | |
| *** panda is now known as panda|zZ | 00:00 | |
| *** ducttape_ has joined #openstack-nova | 00:01 | |
| *** amotoki has joined #openstack-nova | 00:04 | |
| *** annegentle has quit IRC | 00:04 | |
| openstackgerrit | Jay Pipes proposed openstack/nova: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | 00:07 |
|---|---|---|
| *** jaypipes has quit IRC | 00:07 | |
| *** ijw has quit IRC | 00:09 | |
| *** dims_ has joined #openstack-nova | 00:10 | |
| *** dims has quit IRC | 00:11 | |
| *** hongbin has quit IRC | 00:14 | |
| openstackgerrit | David Medberry proposed openstack/nova: Explain how allow_resize_to_same_host is useful https://review.openstack.org/426531 | 00:18 |
| *** kaisers has joined #openstack-nova | 00:18 | |
| openstackgerrit | David Medberry proposed openstack/nova: Explain how allow_resize_to_same_host is useful https://review.openstack.org/426531 | 00:19 |
| *** kaisers has quit IRC | 00:22 | |
| *** mlavalle has quit IRC | 00:25 | |
| *** annegentle has joined #openstack-nova | 00:25 | |
| *** vsaienko has quit IRC | 00:26 | |
| *** nic has quit IRC | 00:29 | |
| *** annegentle has quit IRC | 00:30 | |
| *** sree has joined #openstack-nova | 00:32 | |
| *** jdurgin has quit IRC | 00:34 | |
| *** chyka has quit IRC | 00:34 | |
| *** sree has quit IRC | 00:36 | |
| *** kylek3h has joined #openstack-nova | 00:36 | |
| *** zhurong has joined #openstack-nova | 00:41 | |
| *** sree has joined #openstack-nova | 00:42 | |
| andreykurilin | bnemec: for some reasons CI failed. | 00:43 |
| andreykurilin | any other places of novaclient initialization? | 00:43 |
| *** zhurong has quit IRC | 00:44 | |
| *** crushil has quit IRC | 00:45 | |
| *** crushil has joined #openstack-nova | 00:45 | |
| *** jose-phillips has quit IRC | 00:45 | |
| *** gouthamr has joined #openstack-nova | 00:46 | |
| bnemec | andreykurilin: Crud, maybe I was wrong about not having to worry about older novaclients. | 00:46 |
| bnemec | It's working on our ocata jobs, but they're using the latest novaclient too. | 00:47 |
| andreykurilin | bnemec: u-c is used in all gates, even for those projects who ignores g-r :) so latest novaclient should be there | 00:47 |
| *** yamahata has quit IRC | 00:48 | |
| andreykurilin | oh...there is 6.0 | 00:48 |
| andreykurilin | http://logs.openstack.org/15/427515/2/check/gate-tripleo-ci-centos-7-undercloud/962e5b9/logs/pip2-freeze.txt.gz | 00:48 |
| *** jdurgin has joined #openstack-nova | 00:48 | |
| bnemec | andreykurilin: We don't install from source though. We use package repos, and non-tripleo projects are pinned to whatever version last passed our promotion jobs. | 00:48 |
| bnemec | And this bug is blocking the promotion jobs. :-) | 00:48 |
| *** mriedem has joined #openstack-nova | 00:48 | |
| bnemec | andreykurilin: I have to get going. I'll pick this back up tomorrow though. | 00:49 |
| andreykurilin | bnemec: give me a sec, I'll resubmit a patch | 00:52 |
| andreykurilin | bnemec: I submitted a new patch-set with hack for old release | 00:53 |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova: Implement new attach/detach Cinder flow https://review.openstack.org/330285 | 00:54 |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova: Remove check_attach https://review.openstack.org/335358 | 00:54 |
| *** ducttape_ has quit IRC | 00:55 | |
| *** claudiub has quit IRC | 00:57 | |
| *** liverpooler has quit IRC | 00:58 | |
| *** Swami has quit IRC | 01:01 | |
| *** ducttape_ has joined #openstack-nova | 01:06 | |
| *** esberglu has quit IRC | 01:21 | |
| *** baoli has joined #openstack-nova | 01:24 | |
| *** sree has quit IRC | 01:28 | |
| *** sree has joined #openstack-nova | 01:28 | |
| *** sree has quit IRC | 01:33 | |
| *** browne has quit IRC | 01:37 | |
| *** inc0 has joined #openstack-nova | 01:37 | |
| *** timello has quit IRC | 01:38 | |
| inc0 | hello guys, we (kolla) started to experience issues with nova - https://bugs.launchpad.net/kolla-ansible/+bug/1660761 | 01:38 |
| openstack | Launchpad bug 1660761 in kolla-ansible "VM stuck in scheduling" [Critical,Confirmed] - Assigned to Michał Jastrzębski (inc007) | 01:38 |
| inc0 | cells v2 landed? I assume there is new step durng deployment process right? | 01:39 |
| *** ducttape_ has quit IRC | 01:39 | |
| inc0 | and we *have to* configure cells now? | 01:39 |
| diana_clarke | inc0: yes, cells v2 setup is required as of ocata | 01:41 |
| inc0 | http://docs.openstack.org/developer/nova/cells.html#cells-v2 this is correct doc right? | 01:42 |
| diana_clarke | inc0: I'm just pulling up some links for you (sorry if that sounded short/terse a moment ago). | 01:42 |
| diana_clarke | inc0: yup, that a good one to read | 01:43 |
| diana_clarke | inc0: https://review.openstack.org/#/c/426864/5/releasenotes/notes/ocata_prelude-cfa8793d07f963e7.yaml | 01:43 |
| inc0 | diana_clarke, it did not:) it sounded nothing but helpful:) | 01:43 |
| inc0 | thank you, I'll get to writing deployment code for that | 01:43 |
| diana_clarke | inc0: http://docs.openstack.org/developer/nova/man/nova-manage.html#nova-cells-v2 | 01:45 |
| inc0 | thanks, I'll check if other deployment tools got that in their deploy as well | 01:45 |
| inc0 | this will have to be part of upgrade task too I assume? | 01:46 |
| *** timello has joined #openstack-nova | 01:46 | |
| diana_clarke | inc0: Finally, I find the codesearch tool for "cells_v2" helpful to see how others have done it: http://codesearch.openstack.org/?q=cell_v2&i=nope&files=&repos= | 01:46 |
| diana_clarke | inc0: yup, upgrade too | 01:47 |
| inc0 | once I'll write code for it, I'll leave links here too so you could double check | 01:48 |
| inc0 | thank you very much | 01:48 |
| diana_clarke | inc0: Good luck! I'm still learning the ins and outs of it, but I can try to answer any questions you have. | 01:49 |
| *** sree has joined #openstack-nova | 01:49 | |
| *** unicell has quit IRC | 01:52 | |
| *** esberglu has joined #openstack-nova | 01:52 | |
| *** ducttape_ has joined #openstack-nova | 01:54 | |
| *** esberglu has quit IRC | 01:56 | |
| *** sree has quit IRC | 01:57 | |
| *** rfolco has joined #openstack-nova | 01:58 | |
| *** rfolco has quit IRC | 01:58 | |
| *** Apoorva has quit IRC | 02:06 | |
| *** Apoorva has joined #openstack-nova | 02:06 | |
| *** Apoorva has quit IRC | 02:07 | |
| *** Apoorva has joined #openstack-nova | 02:12 | |
| *** Apoorva has quit IRC | 02:13 | |
| *** ducttape_ has quit IRC | 02:13 | |
| *** baoli has quit IRC | 02:15 | |
| *** Apoorva has joined #openstack-nova | 02:15 | |
| *** baoli has joined #openstack-nova | 02:15 | |
| *** Sukhdev has quit IRC | 02:16 | |
| *** Apoorva has quit IRC | 02:17 | |
| *** Apoorva has joined #openstack-nova | 02:18 | |
| *** kaisers has joined #openstack-nova | 02:19 | |
| *** ducttape_ has joined #openstack-nova | 02:20 | |
| *** ducttape_ has joined #openstack-nova | 02:20 | |
| *** Apoorva has quit IRC | 02:22 | |
| *** kaisers has quit IRC | 02:23 | |
| *** slaweq has quit IRC | 02:55 | |
| *** raunak has joined #openstack-nova | 03:00 | |
| *** esberglu has joined #openstack-nova | 03:05 | |
| *** armax has quit IRC | 03:05 | |
| mriedem | yay new gate bugs http://logs.openstack.org/91/426991/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f218227/logs/screen-n-api.txt.gz#_2017-02-01_01_25_02_367 | 03:07 |
| *** hfu has joined #openstack-nova | 03:08 | |
| mriedem | melwitt: dansmith: for debugging pleasure tomorrow, looks like we have a race after moving instance creation to conductor causing this failure | 03:15 |
| mriedem | https://bugs.launchpad.net/nova/+bug/1660878 | 03:15 |
| openstack | Launchpad bug 1660878 in OpenStack Compute (nova) "test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"" [High,Confirmed] | 03:15 |
| mriedem | the test creates a server, then deletes it and waits for the 404, then tries to reboot it and expects a 404 but gets a 409 because it's now building | 03:16 |
| mriedem | so my guess is we get a 404 when the build request is gone but before the instance is created, something like that | 03:16 |
| *** inc0 has quit IRC | 03:16 | |
| openstackgerrit | Oliver Walsh proposed openstack/nova: DNM - TripleO CI test https://review.openstack.org/427536 | 03:18 |
| *** owalsh is now known as owalsh-afk | 03:23 | |
| *** esberglu has quit IRC | 03:25 | |
| *** esberglu has joined #openstack-nova | 03:25 | |
| *** crushil has quit IRC | 03:28 | |
| *** efried has quit IRC | 03:28 | |
| *** esberglu has quit IRC | 03:30 | |
| dansmith | mriedem: that's not supposed to happen because of the way we do the create of the instance and build request | 03:35 |
| dansmith | but okay, let's look tomorrow | 03:35 |
| mriedem | i'm going to put up a change with a bunch of debug logging | 03:35 |
| mriedem | since the delete and get paths in the api are pretty hairy | 03:35 |
| *** chyka has joined #openstack-nova | 03:36 | |
| *** chyka has quit IRC | 03:41 | |
| *** efried has joined #openstack-nova | 03:41 | |
| *** mtanino has quit IRC | 03:43 | |
| *** nicolasbock has quit IRC | 03:46 | |
| *** ssurana has joined #openstack-nova | 03:55 | |
| *** sree has joined #openstack-nova | 03:57 | |
| *** baoli has quit IRC | 03:58 | |
| smcginnis | Since this afternoon I'm getting stacking failures with "No host found for cellv1" errors. | 03:58 |
| smcginnis | Is there something I need to do to get this to work right? | 03:59 |
| *** tlian has quit IRC | 03:59 | |
| *** owalsh has joined #openstack-nova | 03:59 | |
| *** thorst_ has quit IRC | 04:00 | |
| *** owalsh-afk has quit IRC | 04:00 | |
| mriedem | smcginnis: latest devstack? | 04:00 |
| mriedem | any special localrc? | 04:00 |
| smcginnis | mriedem: Freshly cloned. | 04:00 |
| smcginnis | mriedem: Nothing special in local.conf. | 04:00 |
| mriedem | got a paste of the rough failure log? | 04:00 |
| mriedem | are you sure that's the same error message? because i'm not finding that | 04:01 |
| *** crushil has joined #openstack-nova | 04:02 | |
| smcginnis | mriedem: local.conf file: http://paste.openstack.org/show/597124/ | 04:02 |
| *** sree has quit IRC | 04:02 | |
| smcginnis | mriedem: Shoot, looks like I'll have to try to stack again to get the exact text. | 04:02 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: WIP: Add debug logging for delete/query race bug https://review.openstack.org/427544 | 04:02 |
| mriedem | smcginnis: http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html ? | 04:03 |
| mriedem | you don't have placement-api in ENABLED_SERVICES | 04:03 |
| *** psachin has joined #openstack-nova | 04:03 | |
| smcginnis | mriedem: Mmmm, nope, I don't. | 04:03 |
| smcginnis | mriedem: Guess I should have read that one. | 04:04 |
| mriedem | dansmith: melwitt: just fyi for the morning https://review.openstack.org/#/c/427544/ | 04:04 |
| mriedem | this isn't a super high failure rate so i might have some rechecking to do | 04:04 |
| mriedem | but it's high enough | 04:04 |
| mriedem | 164 hits since 1/27 | 04:04 |
| *** ducttape_ has quit IRC | 04:05 | |
| melwitt | mriedem: thanks. I'm looking at it now. nothing jumping out at me yet | 04:07 |
| mriedem | the mess of conditionals all over create/delete/get in api and conductor makes me think something can slip in here | 04:07 |
| mriedem | and we're probably doing a pass or some other assumption in one of them | 04:07 |
| mriedem | like, | 04:07 |
| mriedem | well, | 04:07 |
| mriedem | nvm | 04:08 |
| *** sudipto_ has joined #openstack-nova | 04:08 | |
| *** sudipto has joined #openstack-nova | 04:08 | |
| mriedem | i was thinking, before the api casts to conductor to create the instance in the cell, the build request could be deleted, and then we create the instance from stale build request info, but then in conductor when we go to delete the build request, it should fail with a 404 and then we should cleanup the instance too | 04:08 |
| mriedem | however, i wonder if the GET request is hitting right in between the time that the instance is created and the time that we try to delete the build request and find out it's already deleted, then cleanup | 04:09 |
| melwitt | hmm | 04:09 |
| mriedem | because there is no mutex there at all from what i can tell, | 04:09 |
| mriedem | and after we cast to conductor, we never refresh the build request from the db to get it's current state before we actually create the instance in the cell, from what i can tell anyway | 04:09 |
| *** raunak has quit IRC | 04:10 | |
| *** mdnadeem has joined #openstack-nova | 04:10 | |
| melwitt | yeah, that sounds reasonable. I was thinking along those lines of somehow the instance getting resurrected | 04:12 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: WIP: Add debug logging for delete/query race bug https://review.openstack.org/427544 | 04:17 |
| *** crushil has quit IRC | 04:20 | |
| mriedem | well i'm going to call it a night i guess | 04:21 |
| mriedem | maybe we'll catch a fish | 04:21 |
| *** crushil has joined #openstack-nova | 04:21 | |
| melwitt | gnight o/ | 04:21 |
| *** armax has joined #openstack-nova | 04:27 | |
| *** ducttape_ has joined #openstack-nova | 04:28 | |
| *** crushil has quit IRC | 04:31 | |
| *** dikonoor has joined #openstack-nova | 04:35 | |
| *** udesale has joined #openstack-nova | 04:37 | |
| *** ducttape_ has quit IRC | 04:38 | |
| *** prateek has joined #openstack-nova | 04:41 | |
| *** browne1 has joined #openstack-nova | 04:43 | |
| *** ayogi has joined #openstack-nova | 04:44 | |
| *** pbandark has joined #openstack-nova | 04:46 | |
| *** ducttape_ has joined #openstack-nova | 04:47 | |
| *** sree has joined #openstack-nova | 04:49 | |
| *** browne has joined #openstack-nova | 04:51 | |
| *** browne has left #openstack-nova | 04:53 | |
| *** browne has joined #openstack-nova | 04:53 | |
| *** browne has left #openstack-nova | 04:53 | |
| *** dikonoor has quit IRC | 04:54 | |
| *** thorst_ has joined #openstack-nova | 05:00 | |
| *** thorst_ has quit IRC | 05:04 | |
| *** yamahata has joined #openstack-nova | 05:04 | |
| openstackgerrit | Yoshihiko Atsumi proposed openstack/nova: Add resize server functional negative tests https://review.openstack.org/420514 | 05:08 |
| *** kaisers has joined #openstack-nova | 05:10 | |
| *** dikonoor has joined #openstack-nova | 05:12 | |
| *** ducttape_ has quit IRC | 05:13 | |
| *** Sukhdev has joined #openstack-nova | 05:13 | |
| *** kaisers has quit IRC | 05:15 | |
| *** raunak has joined #openstack-nova | 05:15 | |
| *** lpetrut has joined #openstack-nova | 05:16 | |
| *** kaisers has joined #openstack-nova | 05:18 | |
| *** browne1 has quit IRC | 05:18 | |
| *** dave-mccowan has quit IRC | 05:18 | |
| *** gouthamr has quit IRC | 05:21 | |
| *** dave-mccowan has joined #openstack-nova | 05:22 | |
| *** kaisers has quit IRC | 05:22 | |
| *** diga has joined #openstack-nova | 05:26 | |
| *** thorst_ has joined #openstack-nova | 05:31 | |
| *** thorst_ has quit IRC | 05:35 | |
| *** ijw has joined #openstack-nova | 05:48 | |
| *** ijw has quit IRC | 05:53 | |
| *** ecelik has joined #openstack-nova | 05:56 | |
| *** Sukhdev has quit IRC | 05:57 | |
| *** ecelik has left #openstack-nova | 05:57 | |
| *** dave-mcc_ has joined #openstack-nova | 05:58 | |
| *** dave-mccowan has quit IRC | 06:01 | |
| *** raunak has quit IRC | 06:01 | |
| *** yarkot has quit IRC | 06:06 | |
| *** satyar has joined #openstack-nova | 06:08 | |
| *** lpetrut has quit IRC | 06:13 | |
| *** Hosam has joined #openstack-nova | 06:18 | |
| *** sridharg has joined #openstack-nova | 06:23 | |
| *** abalutoiu has quit IRC | 06:25 | |
| *** ducttape_ has joined #openstack-nova | 06:27 | |
| *** rcernin has joined #openstack-nova | 06:29 | |
| *** claudiub has joined #openstack-nova | 06:30 | |
| *** yarkot has joined #openstack-nova | 06:31 | |
| *** ducttape_ has quit IRC | 06:31 | |
| *** mjura has joined #openstack-nova | 06:33 | |
| *** claudiub|2 has joined #openstack-nova | 06:33 | |
| *** armax has quit IRC | 06:35 | |
| *** claudiub has quit IRC | 06:36 | |
| *** diga has quit IRC | 06:44 | |
| *** nkrinner_afk is now known as nkrinner | 06:44 | |
| *** unicell has joined #openstack-nova | 06:52 | |
| *** unicell has quit IRC | 06:55 | |
| *** lpetrut has joined #openstack-nova | 06:56 | |
| *** fnordahl has quit IRC | 06:57 | |
| *** fnordahl has joined #openstack-nova | 06:57 | |
| *** diga has joined #openstack-nova | 06:59 | |
| *** unicell has joined #openstack-nova | 06:59 | |
| *** andreas_s has joined #openstack-nova | 07:00 | |
| *** kaisers has joined #openstack-nova | 07:09 | |
| *** tesseract has joined #openstack-nova | 07:18 | |
| *** Oku_OS-away is now known as Oku_OS | 07:18 | |
| *** pcaruana has joined #openstack-nova | 07:19 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova: Refactor a test method including 7 test cases https://review.openstack.org/395436 | 07:22 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova: api-ref: Fix keypair API parameters https://review.openstack.org/420626 | 07:22 |
| *** markus_z has joined #openstack-nova | 07:25 | |
| *** foexle has joined #openstack-nova | 07:26 | |
| *** fnordahl has quit IRC | 07:27 | |
| *** fnordahl has joined #openstack-nova | 07:27 | |
| *** xek__ is now known as xek | 07:31 | |
| *** thorst_ has joined #openstack-nova | 07:31 | |
| *** thorst_ has quit IRC | 07:36 | |
| *** sudipto has quit IRC | 07:37 | |
| *** sudipto_ has quit IRC | 07:37 | |
| *** unicell1 has joined #openstack-nova | 07:38 | |
| *** unicell has quit IRC | 07:39 | |
| *** Jack_I has joined #openstack-nova | 07:39 | |
| lennyb | wznoinsk, I see a lot of multinode jobs fail lately (also our), I found that it's related to latest placement-api that was merged. | 07:46 |
| *** lpetrut has quit IRC | 07:48 | |
| *** Cristina_ has quit IRC | 07:50 | |
| *** esberglu has joined #openstack-nova | 07:50 | |
| *** sree has quit IRC | 07:53 | |
| *** sree has joined #openstack-nova | 07:54 | |
| *** adisky_ has joined #openstack-nova | 07:54 | |
| *** esberglu has quit IRC | 07:55 | |
| *** ducttape_ has joined #openstack-nova | 07:57 | |
| *** sree has quit IRC | 07:58 | |
| lennyb | Infra, regarding gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial where can I find nova.conf files for hypervisors of this job. I have some troubles to apply placement api to our multinode setup | 08:00 |
| *** ducttape_ has quit IRC | 08:02 | |
| *** takashin has left #openstack-nova | 08:03 | |
| *** avolkov has joined #openstack-nova | 08:05 | |
| *** slagle has quit IRC | 08:05 | |
| *** lpetrut has joined #openstack-nova | 08:06 | |
| *** rnoriega has quit IRC | 08:07 | |
| *** Prophidys has quit IRC | 08:12 | |
| *** bkopilov has quit IRC | 08:14 | |
| *** ralonsoh has joined #openstack-nova | 08:14 | |
| *** abalutoiu has joined #openstack-nova | 08:20 | |
| *** lpetrut has quit IRC | 08:20 | |
| *** dave-mcc_ has quit IRC | 08:24 | |
| *** bkopilov has joined #openstack-nova | 08:24 | |
| *** foexle has quit IRC | 08:29 | |
| *** ekuris__ has joined #openstack-nova | 08:36 | |
| *** yamahata has quit IRC | 08:39 | |
| *** ssurana has quit IRC | 08:40 | |
| *** avolkov has quit IRC | 08:43 | |
| *** avolkov has joined #openstack-nova | 08:44 | |
| *** amotoki has quit IRC | 08:49 | |
| *** unicell1 has quit IRC | 08:50 | |
| *** tbachman has quit IRC | 08:57 | |
| *** sree has joined #openstack-nova | 08:58 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** zzzeek has joined #openstack-nova | 09:00 | |
| *** tbachman has joined #openstack-nova | 09:01 | |
| *** sree has quit IRC | 09:03 | |
| *** panda|zZ is now known as panda | 09:04 | |
| *** karimb has joined #openstack-nova | 09:11 | |
| *** Hosam has quit IRC | 09:20 | |
| *** mdnadeem has quit IRC | 09:24 | |
| *** mdnadeem has joined #openstack-nova | 09:24 | |
| wznoinsk | lennyb, our people are already on it :-), thanks for pinging | 09:26 |
| lyarwood | johnthetubaguy, sfinucan, bauzas ; Would you mind taking a look at https://review.openstack.org/#/c/427267/ this morning, mriedem marked the bug as High / ocata-rc-potential so I'd like to get a few more reviews on it today if possible. | 09:27 |
| *** ducttape_ has joined #openstack-nova | 09:28 | |
| *** amotoki has joined #openstack-nova | 09:30 | |
| *** lpetrut has joined #openstack-nova | 09:32 | |
| *** thorst_ has joined #openstack-nova | 09:32 | |
| *** ducttape_ has quit IRC | 09:32 | |
| oomichi | mriedem: yeah, 422171 becomes better | 09:35 |
| *** Guest15362 has joined #openstack-nova | 09:36 | |
| *** Guest15362 has quit IRC | 09:36 | |
| *** yassine has joined #openstack-nova | 09:36 | |
| *** yassine is now known as Guest30167 | 09:36 | |
| bauzas | lyarwood: roger | 09:37 |
| *** thorst_ has quit IRC | 09:38 | |
| *** amotoki has quit IRC | 09:38 | |
| *** lucas-afk is now known as lucasagomes | 09:38 | |
| *** sudipto_ has joined #openstack-nova | 09:38 | |
| *** sudipto has joined #openstack-nova | 09:38 | |
| *** esberglu has joined #openstack-nova | 09:39 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova: Optimize time during PCI alias parsing process https://review.openstack.org/427145 | 09:42 |
| *** esberglu has quit IRC | 09:44 | |
| *** amotoki has joined #openstack-nova | 09:45 | |
| *** amotoki has quit IRC | 09:47 | |
| *** amotoki has joined #openstack-nova | 09:50 | |
| *** hfu has quit IRC | 09:54 | |
| *** hfu has joined #openstack-nova | 09:54 | |
| *** hfu has quit IRC | 09:55 | |
| *** karimb has quit IRC | 09:59 | |
| lennyb | wznoinsk, looks like your people should add placement-api to enabled services on all computes | 10:01 |
| wznoinsk | lennyb, maybe they did already, I'll pass the hint, thanks | 10:01 |
| gmann | johnthetubaguy: alex_xu : can you check(whenever get time), i think we cannot change default policy value directly - https://review.openstack.org/#/c/396420/5 | 10:02 |
| gmann | johnthetubaguy: alex_xu also if you can check this sort/filter release note. I updated this one - https://review.openstack.org/#/c/421760/ | 10:03 |
| *** udesale has quit IRC | 10:04 | |
| *** efried has quit IRC | 10:06 | |
| johnthetubaguy | gmann: I think I am +1 a lot of your comments, I am going to draft a spec around the policy changes I would like to see happen next cycle | 10:08 |
| gmann | johnthetubaguy: yea that nice. | 10:09 |
| gmann | johnthetubaguy: i tried to update the existing one but yea separate is better-https://review.openstack.org/#/c/425533/ | 10:10 |
| johnthetubaguy | working with some folks over in keystone land on that, with any luck, they have similar ideas/plans forming | 10:10 |
| gmann | ok | 10:10 |
| *** mptacekx has joined #openstack-nova | 10:10 | |
| johnthetubaguy | want to get stuff laid out with some structure so we can talk through things at the PTG, with any luck. | 10:10 |
| johnthetubaguy | lyarwood: ack | 10:11 |
| *** cdent has joined #openstack-nova | 10:11 | |
| gmann | yea that is more aligned for PTG and prepare before that | 10:12 |
| gmann | johnthetubaguy: alex_xu btw do we have meeting today? do not think alex_xu is back | 10:12 |
| johnthetubaguy | gmann: I don't know if we decided something, given its crazy o'clock for your all, I vote we skip it, but I will try start the meeting in case folks are about and want to discuss things | 10:13 |
| *** satyar has quit IRC | 10:15 | |
| gmann | ok | 10:15 |
| kashyap | raj_singh: Hi, you have assigned someone to work on this -- https://bugs.launchpad.net/nova/+bug/1605016 | 10:20 |
| openstack | Launchpad bug 1605016 in OpenStack Compute (nova) "Post copy live migration interrupts network connectivity" [High,In progress] - Assigned to Sivasathurappan Radhakrishnan (siva-radhakrishnan) | 10:20 |
| cdent | gmann, johnthetubaguy: If we're talking about api meetings I'd vote for skip too | 10:20 |
| kashyap | Do they have requisite knowledge to work on it? Or should I reassign the bug back to 'nodbody'? | 10:20 |
| *** Jack_I has quit IRC | 10:21 | |
| *** dikonoor has quit IRC | 10:22 | |
| gmann | cdent: yea, api meeting. me too to skip. | 10:22 |
| *** ababich has joined #openstack-nova | 10:24 | |
| johnthetubaguy | kashyap: we are training him up, he is struggling a bit on it though | 10:27 |
| kashyap | johnthetubaguy: Me nods. That bug is quite complicated. Maybe a lower-hanging fruit might be better? | 10:27 |
| johnthetubaguy | kashyap: he is testing out the fix in an environment at the moment I believe | 10:27 |
| kashyap | johnthetubaguy: I say that because, right now QEMU migration devs are asking: What is Nova's stance on postcopy | 10:28 |
| kashyap | So, I lost complete memory of it | 10:28 |
| kashyap | Maybe mdbooth can shed some light there? | 10:28 |
| johnthetubaguy | kashyap: support is in, works with majority of neutron back ends, but not all, AFAIK | 10:28 |
| johnthetubaguy | we discussed it alot at the summit, and siva_krishnan is implementing what we agreed would be the "quick fix", calling neutron earlier in the process | 10:29 |
| mdbooth | johnthetubaguy: IIRC the problem with that change wasn't so much working out what needed to be done, as finding somebody with enough arcane neutron knowledge to know which backends allowed it to be done. | 10:29 |
| johnthetubaguy | would love to trigger of QEMU events to get more exact timing of when the post copy occurs, but the basic structure is going in the above change | 10:29 |
| *** davidgiluk has joined #openstack-nova | 10:30 | |
| johnthetubaguy | mdbooth: I am confused then, neutron folks agreed this should be OK at the summit, what changed from that? | 10:30 |
| mdbooth | johnthetubaguy: The qemu events things isn't required at all in this instance, because the change happens synchronously on the source. | 10:30 |
| kashyap | johnthetubaguy: I have asked David Gilbert of QEMU to join here. He can ask questions to his heart's content, and he goes out of his way to explain | 10:30 |
| davidgiluk | kashyap: nod | 10:30 |
| *** edmondsw has joined #openstack-nova | 10:30 | |
| kashyap | davidgiluk: Ah, you are here :-) | 10:30 |
| mdbooth | i.e. there's no need to respond to an event from a call which you just made in code you're still executing | 10:30 |
| johnthetubaguy | mdbooth: I keep being told its not syncronus | 10:31 |
| mdbooth | What's not synchronous about it? | 10:31 |
| johnthetubaguy | you just say, please do post copy at end of next iterration, I believe | 10:31 |
| johnthetubaguy | so when we call post copy, it doesn't immediatly happen, I was told | 10:31 |
| kashyap | davidgiluk: So, this is what johnthetubaguy was saying, when I asked him about "What is Nova's stance on postcopy": < johnthetubaguy> support is in, works with majority of neutron back ends, but not all, AFAIK | 10:31 |
| johnthetubaguy | there can be a small network outage with some backends, during the post copy process (which ironically speeds up the live-migrate, of course) doing work to avoid that | 10:32 |
| davidgiluk | kashyap: Yeh so my understanding was it was waiting on a fix to lp 1605016 for the others, with https://review.openstack.org/#/c/413021/9 being a proposed fix | 10:32 |
| openstack | Launchpad bug 1605016 in OpenStack Compute (nova) "Post copy live migration interrupts network connectivity" [High,In progress] https://launchpad.net/bugs/1605016 - Assigned to Sivasathurappan Radhakrishnan (siva-radhakrishnan) | 10:32 |
| mdbooth | johnthetubaguy: In _live_migration_monitor we do: | 10:32 |
| *** Hosam has joined #openstack-nova | 10:32 | |
| mdbooth | if (is_post_copy_enabled and | 10:32 |
| mdbooth | libvirt_migrate.should_switch_to_postcopy( | 10:32 |
| mdbooth | info.memory_iteration, info.data_remaining, | 10:32 |
| mdbooth | previous_data_remaining, migration.status)): | 10:32 |
| mdbooth | libvirt_migrate.trigger_postcopy_switch(guest, | 10:32 |
| mdbooth | instance, | 10:32 |
| mdbooth | migration) | 10:32 |
| johnthetubaguy | mdbooth: its the libvirt call that I am told is async | 10:32 |
| johnthetubaguy | mdbooth: it just request post copy at some point in the future, and returns right away, before actually triggering it | 10:33 |
| mdbooth | johnthetubaguy: Well we might switch a few milliseconds early | 10:33 |
| *** Hosam has quit IRC | 10:33 | |
| johnthetubaguy | mdbooth: pkoniszewski has the details on that | 10:33 |
| mdbooth | Rather than a few milliseconds late | 10:33 |
| johnthetubaguy | mdbooth: I was told more like seconds, but yeah, I was betting on milliseconds myself | 10:33 |
| davidgiluk | the postcopy transition happens at the end of a pass, and libvirt gets involved before starting the destination | 10:33 |
| *** Hosam has joined #openstack-nova | 10:33 | |
| johnthetubaguy | mdbooth: either way, I vote for the approach siva_krishnan has in the works already, just need to iron things out | 10:34 |
| johnthetubaguy | davidgiluk: sorry, whats waiting on the fix/ | 10:34 |
| johnthetubaguy | s/ / / ? / | 10:34 |
| davidgiluk | johnthetubaguy: The support for other networking configs | 10:34 |
| *** thorst_ has joined #openstack-nova | 10:34 | |
| johnthetubaguy | davidgiluk: that bug fix should help reduce the network outage some networking configs currently see, yes | 10:35 |
| *** edmondsw has quit IRC | 10:35 | |
| davidgiluk | johnthetubaguy: < johnthetubaguy> would love to trigger of QEMU events to get more exact timing of when the post copy occurs, but the basic structure is going in the above change | 10:35 |
| davidgiluk | johnthetubaguy: We already emit the events | 10:35 |
| johnthetubaguy | davidgiluk: agreed its omitted, we first need all the structures in place to call neutron at the right time, the current patch | 10:36 |
| johnthetubaguy | oops emitted | 10:36 |
| davidgiluk | generate :-) | 10:36 |
| mdbooth | Well, this merged: https://review.openstack.org/#/c/402318/ :( | 10:37 |
| mdbooth | The libvirt driver becomes slightly less fathomable for no appreciable gain. | 10:37 |
| davidgiluk | johnthetubaguy: you should probably loop in jdenemar to check what it looks like for you from ontop of libvirt - but you should have all the events you need to know when postcopy kicks in | 10:38 |
| kashyap | mdbooth: Propose a revert? | 10:38 |
| johnthetubaguy | davidgiluk: thats the plan for the follow up fix | 10:38 |
| mdbooth | kashyap: I proposed not merging it, which didn't work. | 10:38 |
| kashyap | mdbooth: From glancing at your comments on the review | 10:39 |
| davidgiluk | johnthetubaguy: OK, great; if you do need anything more from qemu, just ask | 10:39 |
| johnthetubaguy | mdbooth: sorry, got pulled into meetings after that one, so I didn't get chance to delay it | 10:39 |
| kashyap | mdbooth: No one has bothered to respond with a counter-argument | 10:39 |
| johnthetubaguy | mdbooth: I can fast revert it if you want | 10:39 |
| *** esberglu has joined #openstack-nova | 10:39 | |
| *** thorst_ has quit IRC | 10:39 | |
| kashyap | Then, you have by all means, a right to revert it, and restart the discussion, and make whoever proposed it to respond | 10:39 |
| mdbooth | johnthetubaguy: If you wouldn't mind. I'm assuming, btw, that there are no real users being impacted by this. | 10:39 |
| *** rfolco has joined #openstack-nova | 10:39 | |
| *** dikonoor has joined #openstack-nova | 10:40 | |
| johnthetubaguy | mdbooth: no idea, the SUSE folks proposed this, and I believe they support users on Xen + Libivrt + Nova, so I was assuming yes | 10:40 |
| *** sudipto has quit IRC | 10:40 | |
| *** sudipto_ has quit IRC | 10:40 | |
| mdbooth | It's been broken forever though, right? | 10:41 |
| johnthetubaguy | mdbooth: sure, I assumed they were trying to change that for one of their users that are complaining | 10:41 |
| mdbooth | I just don't want to add more sticky tape without investigating fixing the root cause. | 10:42 |
| *** swann has joined #openstack-nova | 10:42 | |
| mdbooth | I don't see any evidence anybody looked for the root cause. | 10:42 |
| johnthetubaguy | mdbooth: I added my assumption of the cause in the review, well the root cause is Xen being very picky about the device names | 10:43 |
| *** Hosam has quit IRC | 10:43 | |
| mdbooth | We need to either be picky earlier, or translate it earlier. | 10:43 |
| *** esberglu has quit IRC | 10:44 | |
| johnthetubaguy | mdbooth: the problem is, you don't know if its Xen until it hits the compute node | 10:44 |
| mdbooth | Sticking a band-aid at the lowest possible level is just going to be a source of bugs | 10:44 |
| mdbooth | Yeah, that occurred to me. No immediate answer to that as I haven't given it any thought. | 10:44 |
| mdbooth | Seems that nobody else has either, though. | 10:44 |
| mdbooth | Might be something to be resolved in driver.spawn, and attach/detach volume? | 10:45 |
| *** karimb has joined #openstack-nova | 10:45 | |
| mdbooth | That way it's resolved at the time it's created | 10:46 |
| mdbooth | Do those names have any semantic meaning at the api level beyond 'the Nth disk' anyway? I don't think we support multiple controller types, do we? | 10:47 |
| *** cdent has quit IRC | 10:48 | |
| johnthetubaguy | we support multiple hypervisors in a single cloud, at least in theory | 10:48 |
| mdbooth | If ^^^ is a valid assumption, ComputeManager could rewrite it to a number, and the driver could change it back to whatever it wanted. | 10:48 |
| johnthetubaguy | I don't think we even respect the number any more, but it could | 10:48 |
| mdbooth | johnthetubaguy: It's in a single instance. i.e. could we have both sda and vda in the same instance? | 10:49 |
| johnthetubaguy | sda would always fail to attach, all disks need to be vd* I believe, or xvd* | 10:49 |
| *** mvk has quit IRC | 10:49 | |
| mdbooth | Right, so computemanager would see it got 'sda' from the api, translate that to '0', and the driver would change it to 'sda', 'vda', or 'xvda' as required. | 10:50 |
| johnthetubaguy | seems like a good refactor to do on top of the bug fix | 10:51 |
| mdbooth | johnthetubaguy: The bug fix makes the existing code worse, though, and I see no evidence that anybody looked for a root cause. | 10:51 |
| *** claudiub has joined #openstack-nova | 10:51 | |
| mdbooth | Given that it's obviously not urgent, I don't see why we would choose to take more tech debt for that. | 10:52 |
| mdbooth | Anyway, it's not that important in itself. | 10:53 |
| mdbooth | Just death by 1,000 cuts. | 10:53 |
| johnthetubaguy | it breaks a pretty fundamental API feature, that seemed fairly important to be from a parity point of view, but I duno really. | 10:53 |
| *** claudiub|2 has quit IRC | 10:54 | |
| mdbooth | johnthetubaguy: Yeah, I know. But it's been broken a long time and I'd have preferred to see an attempt to fix it without adding tech debt first. | 10:56 |
| *** cdent has joined #openstack-nova | 10:57 | |
| johnthetubaguy | mdbooth: that my preference too, but its a balancing act | 10:57 |
| johnthetubaguy | I was very boarderline on -1 +2 myself | 10:58 |
| mdbooth | If it's important to somebody, they'll fix it. You even suggested where to look, and I agreed. | 10:58 |
| bauzas | lyarwood: nice catch, +2d https://review.openstack.org/#/c/427267/2 | 10:58 |
| *** ducttape_ has joined #openstack-nova | 10:58 | |
| sfinucan | bauzas: Could you take a look at https://review.openstack.org/#/c/392643/, please? :) | 10:59 |
| bauzas | johnthetubaguy: in case you wanna fix a nasty bug with an easy peasy fix ^ | 10:59 |
| bauzas | sfinucan: roger | 10:59 |
| mdbooth | johnthetubaguy: Now ^^^ is a nasty bug :) | 10:59 |
| mdbooth | Ah, not that one. | 10:59 |
| mdbooth | The one before which bauzas mentioned. | 11:00 |
| *** kaisers1 has quit IRC | 11:01 | |
| * johnthetubaguy nods | 11:01 | |
| kashyap | sfinucan: Your friendly monthly reminder: If you put commit IDs in single quotes, then 'clickability" is lost :-) | 11:02 |
| sfinucan | lyarwood: Finally done, though it seems bauzas pipped me to the mark :) | 11:02 |
| sfinucan | nasty old bug indeed | 11:02 |
| kashyap | sfinucan: Just reminding for next time. | 11:02 |
| lyarwood | bauzas: thanks! | 11:02 |
| lyarwood | sfinucan: cheers, np :) | 11:02 |
| sfinucan | sorry, kashyap :'( | 11:02 |
| *** kaisers1 has joined #openstack-nova | 11:02 | |
| *** ducttape_ has quit IRC | 11:03 | |
| * sfinucan thinks maybe it's the Javascript plugins infra use that's really wrong here | 11:03 | |
| *** nicolasbock has joined #openstack-nova | 11:04 | |
| *** jbernard has quit IRC | 11:11 | |
| openstackgerrit | Sergey Nikitin proposed openstack/nova-specs: Restore V3 Diagnostics info https://review.openstack.org/427637 | 11:11 |
| sfinucan | johnthetubaguy: What are we doing about https://review.openstack.org/#/c/357190 ? | 11:15 |
| johnthetubaguy | sfinucan: not sure what you mean? | 11:16 |
| johnthetubaguy | oh, its that damm mox things | 11:16 |
| sfinucan | aye :) | 11:17 |
| *** jbernard has joined #openstack-nova | 11:17 | |
| *** amotoki has quit IRC | 11:18 | |
| johnthetubaguy | sfinucan: these bits hurt my eyes a little "mock_domain.undefineFlags.assert_has_calls([mock.call(1)])" | 11:18 |
| johnthetubaguy | assert_called_once_with(1) would be much better | 11:18 |
| sfinucan | derekh - you around? | 11:19 |
| derekh | sfinucan: yup | 11:19 |
| *** mvk has joined #openstack-nova | 11:19 | |
| sfinucan | See John's comments above - could you respin https://review.openstack.org/#/c/417227 accordingly | 11:19 |
| johnthetubaguy | its OK, we can do a follow on patch | 11:20 |
| sfinucan | johnthetubaguy: Won't argue with that | 11:20 |
| johnthetubaguy | derekh: its these bits, I think we can make it a lot simpler: https://review.openstack.org/#/c/417227/2/nova/tests/unit/virt/libvirt/test_driver.py@11908 | 11:20 |
| johnthetubaguy | derekh: I wouldn't complain normally, but folks might start copying that pattern | 11:20 |
| derekh | johnthetubaguy: ack, I'll push a follow up this morning | 11:21 |
| johnthetubaguy | derekh: awesome, thanks | 11:21 |
| derekh | johnthetubaguy: np | 11:21 |
| derekh | johnthetubaguy: sfinucan thanks both | 11:21 |
| johnthetubaguy | sfinucan: my browsers got out of hand, so I gave up this morning and closed everything! | 11:21 |
| sfinucan | (y) | 11:21 |
| sfinucan | johnthetubaguy: Also, are we proceeding with this change or punting to Pike? https://review.openstack.org/#/c/412634/ | 11:22 |
| sfinucan | mriedem's etherpad sugggests the former, but it seems a little late in the day https://etherpad.openstack.org/p/nova-ocata-feature-freeze | 11:23 |
| * sfinucan is working through his "last minute things I should check out" checklist :) | 11:23 | |
| johnthetubaguy | sfinucan: so we have merged half of it, but its tempting to wait until friday, i.e. pike | 11:24 |
| sfinucan | that would be my gut feeling - it's not critical | 11:24 |
| *** jamielennox is now known as jamielennox|away | 11:25 | |
| *** jd__ has joined #openstack-nova | 11:25 | |
| jd__ | hi there | 11:25 |
| jd__ | could someone give me a hint on what this failure means http://logs.openstack.org/82/405682/8/check/gate-ceilometer-dsvm-tempest-plugin-postgresql-ubuntu-xenial/02f896f/logs/screen-n-sch.txt.gz#_2017-02-01_10_20_33_556 ? | 11:25 |
| johnthetubaguy | sfinucan: so it does stop people from taking large snapshots | 11:25 |
| johnthetubaguy | sfinucan: I am tempted to ask mriedem to make the call | 11:26 |
| sfinucan | johnthetubaguy: sounds fine by me - he's clearly thinking about it anyway, given the Etherpad | 11:28 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: conf: Resolve TODOs in 'database' https://review.openstack.org/393695 | 11:28 |
| *** sdague has joined #openstack-nova | 11:28 | |
| johnthetubaguy | sfinucan: completeness wise, we have merged half of it already | 11:30 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: rbd: Remove unnecessary 'encode' calls https://review.openstack.org/412356 | 11:31 |
| *** dikonoor has quit IRC | 11:32 | |
| *** mdnadeem has quit IRC | 11:36 | |
| sfinucan | johnthetubaguy: True, but it's still a new feature rather than a regression fix. Definitely happy to let someone else make the decision for me :) | 11:37 |
| johnthetubaguy | sfinucan: its totally a feature | 11:37 |
| *** tbachman has quit IRC | 11:40 | |
| *** dikonoor has joined #openstack-nova | 11:41 | |
| *** mdnadeem has joined #openstack-nova | 11:42 | |
| *** edand has joined #openstack-nova | 11:46 | |
| *** ociuhandu has joined #openstack-nova | 11:46 | |
| cdent | jd__: that's probably this: http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html | 11:47 |
| *** ociuhandu has quit IRC | 11:51 | |
| *** iceyao has joined #openstack-nova | 11:52 | |
| *** edand has quit IRC | 11:57 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova: Optimize time during PCI alias parsing process https://review.openstack.org/427145 | 11:59 |
| *** iceyao has quit IRC | 11:59 | |
| openstackgerrit | John Garbutt proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | 12:00 |
| *** ociuhandu has joined #openstack-nova | 12:01 | |
| openstackgerrit | John Garbutt proposed openstack/nova: Add service_token for nova-glance interaction https://review.openstack.org/419662 | 12:03 |
| *** sgordon has joined #openstack-nova | 12:06 | |
| openstackgerrit | John Garbutt proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | 12:06 |
| openstackgerrit | John Garbutt proposed openstack/nova: Add service_token for nova-glance interaction https://review.openstack.org/419662 | 12:06 |
| *** ociuhandu has quit IRC | 12:08 | |
| openstackgerrit | John Garbutt proposed openstack/nova: DNM: Test service token in Glance https://review.openstack.org/419666 | 12:09 |
| johnthetubaguy | sfinucan: I fixed up my nits on those patches ^ | 12:10 |
| openstackgerrit | Maciej Szankin proposed openstack/nova: config options: remove libvirt imagecache https://review.openstack.org/378497 | 12:11 |
| *** cdelatte has joined #openstack-nova | 12:11 | |
| *** edand has joined #openstack-nova | 12:11 | |
| sfinucan | johnthetubaguy: I'll take a look now | 12:12 |
| openstackgerrit | Lee Yarwood proposed openstack/nova: WIP - libvirt: Remove redundant bdm serial mangling and saving during swap_volume https://review.openstack.org/427364 | 12:14 |
| *** claudiub|2 has joined #openstack-nova | 12:18 | |
| *** bvanhav has joined #openstack-nova | 12:20 | |
| *** Hosam has joined #openstack-nova | 12:20 | |
| sileht | cdent, we use the default devstack setup with additional plugin, we don't override the default service list, does the placement-api is not enabled by default ? | 12:21 |
| *** claudiub has quit IRC | 12:21 | |
| cdent | sileht: it is enabled by default, but I thought most dsvm jobs didn't do anything by default? | 12:22 |
| sileht | cdent, the failing dsvm job run the default | 12:23 |
| *** esberglu has joined #openstack-nova | 12:24 | |
| * cdent looks at the logs harder | 12:24 | |
| sileht | cdent, looks like a real postgres issue: http://logs.openstack.org/82/405682/8/check/gate-ceilometer-dsvm-tempest-plugin-postgresql-ubuntu-xenial/02f896f/logs/apache/placement-api.txt.gz?level=ERROR | 12:24 |
| cdent | feh | 12:24 |
| sileht | cdent, this error comes when you miss some field in the groupby | 12:25 |
| cdent | that's a great catch, I don't think nova tests anything in anything but mysql | 12:25 |
| sileht | cdent, when mysql adds them automaticaly | 12:25 |
| cdent | i'll make a bug | 12:26 |
| sileht | cdent, even for mysql this is better to explicit everything for groupby | 12:26 |
| * cdent nods | 12:26 | |
| cdent | the sql being used there is...insane (to me) | 12:26 |
| *** esberglu has quit IRC | 12:28 | |
| *** catintheroof has joined #openstack-nova | 12:28 | |
| *** quintela has joined #openstack-nova | 12:29 | |
| *** ducttape_ has joined #openstack-nova | 12:29 | |
| *** slagle has joined #openstack-nova | 12:29 | |
| *** ociuhandu has joined #openstack-nova | 12:31 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/nova: TEST https://review.openstack.org/427667 | 12:31 |
| *** ducttape_ has quit IRC | 12:33 | |
| *** pbandark has quit IRC | 12:33 | |
| *** ociuhandu has quit IRC | 12:36 | |
| cdent | sileht, mriedem : https://bugs.launchpad.net/nova/+bug/1660959 | 12:36 |
| openstack | Launchpad bug 1660959 in OpenStack Compute (nova) "placement resource provider filtering does not work with postgres" [Undecided,Triaged] | 12:36 |
| *** mnestratov has quit IRC | 12:38 | |
| kashyap | lyarwood: Do you have DevStack local.conf with multipath config somewhere? | 12:39 |
| lyarwood | kashyap: nope sorry | 12:40 |
| *** slaweq has joined #openstack-nova | 12:40 | |
| kashyap | lyarwood: You only test with RPM-based setups? | 12:41 |
| *** bvanhav has quit IRC | 12:42 | |
| lyarwood | kashyap: yeah, almost all of the multipath issues I've had to work on recently are for Kilo envs so the volume code doesn't even apply to upstream anymore | 12:43 |
| *** bvanhav has joined #openstack-nova | 12:43 | |
| lyarwood | kashyap: well, not since liberty and the split into os-brick | 12:43 |
| kashyap | lyarwood: Hmm, okay. Yeah, it's just a pity that it's such a pain in the neck to figure out the right configs to even test with upstream | 12:43 |
| kashyap | Seems like nobody bothers with it much | 12:44 |
| *** thorst_ has joined #openstack-nova | 12:45 | |
| *** edmondsw has joined #openstack-nova | 12:46 | |
| *** edmondsw has quit IRC | 12:46 | |
| *** edmondsw has joined #openstack-nova | 12:46 | |
| *** karimb has quit IRC | 12:46 | |
| *** mvk has quit IRC | 12:47 | |
| *** eharney has joined #openstack-nova | 12:50 | |
| *** hfu has joined #openstack-nova | 12:52 | |
| *** ociuhandu has joined #openstack-nova | 12:56 | |
| *** mtanino has joined #openstack-nova | 12:56 | |
| *** ducttape_ has joined #openstack-nova | 12:57 | |
| *** mvk has joined #openstack-nova | 13:00 | |
| *** lucasagomes is now known as lucas-hungry | 13:00 | |
| *** david_1 has joined #openstack-nova | 13:00 | |
| *** tpatzig_ has joined #openstack-nova | 13:00 | |
| *** dgonzalez_ has joined #openstack-nova | 13:00 | |
| *** ociuhandu has quit IRC | 13:01 | |
| *** tbachman has joined #openstack-nova | 13:01 | |
| *** dgonzalez_ has quit IRC | 13:02 | |
| *** tpatzig_ has quit IRC | 13:02 | |
| *** david_1 has quit IRC | 13:02 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova: console: Move proxies to 'console/proxy' https://review.openstack.org/408192 | 13:02 |
| *** claudiub|2 is now known as claudiub | 13:04 | |
| *** cdelatte has quit IRC | 13:05 | |
| *** flwang has quit IRC | 13:05 | |
| *** wiggin15 has joined #openstack-nova | 13:06 | |
| *** alineb has quit IRC | 13:06 | |
| *** bnemec has quit IRC | 13:06 | |
| *** ralonsoh_ has joined #openstack-nova | 13:07 | |
| *** bapalm has quit IRC | 13:07 | |
| *** cdelatte has joined #openstack-nova | 13:07 | |
| *** bnemec has joined #openstack-nova | 13:07 | |
| wiggin15 | Hi everyone. Something changed in Nova that fails the Cinder third party CI systems. We're getting "No hosts found to map to cell, exiting.". It started yesterday | 13:08 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: tests: Validate huge pages https://review.openstack.org/399653 | 13:08 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: tests: Combine multiple NUMA-generation functions https://review.openstack.org/402561 | 13:08 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: tests: Don't reinvent __init__ https://review.openstack.org/402560 | 13:08 |
| wiggin15 | Here's Netapp's log: http://dcf901611175aa43f968-c54047c910227e27e1d6f03bb1796fd7.r95.cf5.rackcdn.com/56/427356/1/check/cinder-7mode-iSCSI/0699f1b/devstacklog.txt | 13:08 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: libvirt: create functional test base class https://review.openstack.org/407055 | 13:08 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: tests: Remove duplicate NumaHostInfo https://review.openstack.org/402562 | 13:08 |
| wiggin15 | Here's virtuozzo: http://openstack-3rd-party-virtuozzo-ci-logs.virtuozzo.com/71/416371/9/check/check-dsvm-tempest-vz7-exe-minimal/0854760/logs/devstacklog.txt.gz | 13:08 |
| wiggin15 | others are failing too | 13:08 |
| *** tblakes has joined #openstack-nova | 13:09 | |
| *** ralonsoh has quit IRC | 13:09 | |
| wiggin15 | Does anyone know if something changed in the past 24 hours that can cause this? | 13:09 |
| *** Matias has quit IRC | 13:09 | |
| cdent | wiggin15: sounds like it could be something related to cellsv2 stuff that recently merged | 13:10 |
| *** mordred has quit IRC | 13:10 | |
| *** ralonsoh_ is now known as ralonsoh | 13:11 | |
| cdent | wiggin15: there was some discussion about some changes that need to happen, last night, but I'm not sure of the outcome or timing. when mriedem comes around he'll probably know more. but also check the os-dev list for messages from him, he's being sending gate updates there | 13:11 |
| *** bapalm has joined #openstack-nova | 13:12 | |
| *** mtanino has quit IRC | 13:13 | |
| *** vladikr has quit IRC | 13:15 | |
| macsz | was the Nova API meeting canceled for today or is my calendar messed up? | 13:16 |
| *** ociuhandu has joined #openstack-nova | 13:16 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/nova: WIP: placement-api: fix ResourceProviderList query https://review.openstack.org/427667 | 13:17 |
| *** mordred has joined #openstack-nova | 13:18 | |
| *** efried has joined #openstack-nova | 13:19 | |
| johnthetubaguy | macsz: it was cancelled today | 13:20 |
| macsz | johnthetubaguy: ah, ok | 13:20 |
| *** flwang has joined #openstack-nova | 13:22 | |
| *** mnestratov|2 has joined #openstack-nova | 13:24 | |
| *** bapalm has quit IRC | 13:24 | |
| *** rnoriega has joined #openstack-nova | 13:25 | |
| *** Guest30167 has quit IRC | 13:26 | |
| cdent | rpodolyaka: on group by thing, are you saying: "we're not using c.consumer_id, so just leave it out of the select"? | 13:28 |
| rpodolyaka | cdent: yeah, my understanding is that we calculate total usage values per RP, not "per RP per consumer" | 13:28 |
| rpodolyaka | and this total actually includes multiple consumers | 13:29 |
| rpodolyaka | so consumer_id in select does not make sense and that's what postgresql is unhappy about | 13:29 |
| *** prateek has quit IRC | 13:29 | |
| cdent | makes sense | 13:30 |
| smcginnis | wiggin15: Sounds like you, like me, ignored this in the ML and now has stacking failures: http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html | 13:30 |
| rpodolyaka | we *could* group by consumer_id, but my understanding is that it's not what we really want | 13:30 |
| cdent | rpodolyaka: I think you're right | 13:30 |
| rpodolyaka | but I'm not really familiar with this code, so jaypipes should know better :P | 13:31 |
| *** annegentle has joined #openstack-nova | 13:31 | |
| cdent | rpodolyaka: but the second group_by change is probably right? | 13:31 |
| cdent | (for postgres?) | 13:31 |
| *** vsaienko has joined #openstack-nova | 13:32 | |
| cdent | sileht: you want to carry on with that stuff. If not I can take it over. | 13:32 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/nova: placement-api: fix ResourceProviderList query https://review.openstack.org/427667 | 13:33 |
| sileht | rpodolyaka, the functional tests have catched your thought :) | 13:33 |
| *** gouthamr has joined #openstack-nova | 13:34 | |
| rpodolyaka | sileht: yay! :) | 13:35 |
| *** annegentle has quit IRC | 13:35 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/nova: placement-api: fix ResourceProviderList query https://review.openstack.org/427667 | 13:36 |
| cdent | sileht: being very nit picky on the review in the hope that by the time some core comes along it is nice and happy and we get this merged quick | 13:36 |
| *** Matias has joined #openstack-nova | 13:36 | |
| sileht | cdent, | 13:36 |
| sileht | cdent, it's not really nit picky when feedback are instant :) | 13:37 |
| rpodolyaka | cdent: hmm, the second group by is done on a unique column - it must be ok, as it uniquely identifies a particular table row. I'd expect "select rp.* ..." work fine in this case | 13:37 |
| rpodolyaka | tbh it's the first time I've seen a sqalclhemy model passed to group_by() | 13:37 |
| rpodolyaka | my understanding is that it's equivalent of grouping by all columns of a table | 13:38 |
| sileht | rpodolyaka, with postgresql you must have everything you put in the select | 13:38 |
| sileht | rpodolyaka, yes because the select does 'select rp.*' | 13:38 |
| sileht | rpodolyaka, so even if rp.uuid is enought, postgresql don't want to guess | 13:38 |
| * rpodolyaka fires up a psql shell | 13:39 | |
| sileht | rpodolyaka, I think the second groupby is here just to remove duplicate rows ? | 13:40 |
| *** jpena is now known as jpena|lunch | 13:40 | |
| mriedem | wiggin15: you need to have the placement-api service enabled, in ENABLED_SERVICES | 13:40 |
| mriedem | wiggin15: if you're overriding ENABLED_SERVICES that is - there is a note in the dev list about it | 13:40 |
| *** Guest30167 has joined #openstack-nova | 13:40 | |
| *** eharney has quit IRC | 13:41 | |
| sileht | rpodolyaka, the orignal error we have: http://logs.openstack.org/82/405682/8/check/gate-ceilometer-dsvm-tempest-plugin-postgresql-ubuntu-xenial/02f896f/logs/apache/placement-api.txt.gz?level=ERROR | 13:41 |
| sileht | rpodolyaka, the error when I have fixed the first groupby: http://logs.openstack.org/68/427668/1/check/gate-ceilometer-dsvm-tempest-plugin-postgresql-ubuntu-xenial/372f489/logs/apache/placement-api.txt.gz?level=ERROR | 13:42 |
| mriedem | johnthetubaguy: sfinucan: your call on https://review.openstack.org/#/c/412634/ since it's disabled by default an experimental now anyway | 13:42 |
| mriedem | it was already approved once, just had a busted unit test | 13:42 |
| *** ducttape_ has quit IRC | 13:42 | |
| rpodolyaka | sileht: hmm, interesting | 13:43 |
| cdent | mriedem: I tagged this https://bugs.launchpad.net/nova/+bug/1660959 as rc-potential because postgres support used to work in placement api. it's what sileht and rpodolyaka are talking over now. | 13:43 |
| openstack | Launchpad bug 1660959 in OpenStack Compute (nova) "placement resource provider filtering does not work with postgres" [Undecided,In progress] - Assigned to Mehdi Abaakouk (sileht) | 13:43 |
| mriedem | cdent: yup saw it, thanks | 13:43 |
| mriedem | silly pg | 13:43 |
| wiggin15 | mriedem we have "export OVERRIDE_ENABLED_SERVICES" with services like "c-api", "q-metering" etc., is this it? | 13:43 |
| mriedem | wiggin15: i think so yes | 13:43 |
| mriedem | you need placement-api in that list | 13:43 |
| wiggin15 | mriedem thanks | 13:43 |
| sfinucan | mriedem: I'm happy to let it in, in that case. It's technically a feature but it seems low risk to me. Can't speak for johnthetubaguy, mind | 13:43 |
| sileht | cdent, and our (future) stable/ocata is already broken too :p | 13:43 |
| * sfinucan is still reviewing that | 13:43 | |
| mriedem | i'm heading into the office, bbiab | 13:43 |
| *** mriedem has quit IRC | 13:44 | |
| cdent | sileht: I think the whole world's future is broken lately. | 13:44 |
| rpodolyaka | sileht: heh, looks like it only allows this for PKs, but not for UNIQUEs - http://paste.openstack.org/show/597180/ | 13:46 |
| *** vladikr has joined #openstack-nova | 13:47 | |
| sileht | rpodolyaka, do we have another column that can do it (instead of putting all cols) ? | 13:48 |
| *** ayogi has quit IRC | 13:49 | |
| *** lucas-hungry is now known as lucasagomes | 13:50 | |
| sileht | rpodolyaka, I will use id, then | 13:51 |
| rpodolyaka | sileht: yeah, we have the PK - `id` | 13:51 |
| *** karimb has joined #openstack-nova | 13:51 | |
| *** diga has quit IRC | 13:52 | |
| *** Prophidys has joined #openstack-nova | 13:53 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/nova: placement-api: fix ResourceProviderList query https://review.openstack.org/427667 | 13:54 |
| *** vsaienko has quit IRC | 13:54 | |
| *** tongli has joined #openstack-nova | 13:55 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/nova: placement-api: fix ResourceProviderList query https://review.openstack.org/427667 | 13:56 |
| *** Hosam has quit IRC | 13:57 | |
| *** amotoki has joined #openstack-nova | 13:59 | |
| *** alineb has joined #openstack-nova | 14:00 | |
| *** kfarr has joined #openstack-nova | 14:01 | |
| *** bvanhav_ has joined #openstack-nova | 14:03 | |
| *** bvanhav has quit IRC | 14:04 | |
| *** tlian has joined #openstack-nova | 14:07 | |
| johnthetubaguy | BobBall: seeing some XenAPI issues on this glance related patch: https://review.openstack.org/#/c/412634/24 | 14:07 |
| johnthetubaguy | BobBall: is this a general problem we are seeing at the moment? | 14:07 |
| BobBall | Checking | 14:08 |
| BobBall | general problem meaning unrelated to this patch? | 14:08 |
| *** amotoki has quit IRC | 14:08 | |
| johnthetubaguy | BobBall: it feels very related to this patch, but we haven't changed the code that XenAPI uses, AFAIK | 14:09 |
| johnthetubaguy | BobBall: just wanted to check its not a general issue | 14:09 |
| BobBall | hmmmz - actually we are failing all tests in the last couple of hours... | 14:09 |
| *** ducttape_ has joined #openstack-nova | 14:09 | |
| johnthetubaguy | BobBall: looks like the plugins have changed or something? | 14:10 |
| BobBall | Big sadness from the Citrix CI at the moment :( | 14:10 |
| BobBall | Investigating. Plugins certainly shouldn't have changed since we're pulling in from a fixed version of os-xenapi from nova's requirements... | 14:11 |
| BobBall | Made the mistake of trying to open a 1MB log file in firefox. Apparently it doesn't like a 1MB html file and has decided it needs a really long think about it | 14:12 |
| johnthetubaguy | BobBall: yeah, I had to download it | 14:13 |
| johnthetubaguy | BobBall: I am seeing 404 from glance | 14:13 |
| BobBall | Not ideal. | 14:13 |
| *** esberglu has joined #openstack-nova | 14:13 | |
| *** mdrabe has joined #openstack-nova | 14:13 | |
| *** xyang1 has joined #openstack-nova | 14:13 | |
| *** esberglu has quit IRC | 14:13 | |
| *** esberglu has joined #openstack-nova | 14:14 | |
| *** yamahata has joined #openstack-nova | 14:15 | |
| *** mriedem has joined #openstack-nova | 14:16 | |
| * mriedem hopes all things are fixed now | 14:16 | |
| *** esberglu_ has joined #openstack-nova | 14:18 | |
| BobBall | Looks like glance upload is broken | 14:20 |
| mriedem | BobBall: like this? | 14:20 |
| mriedem | 2017-02-01 12:47:40.884 20207 ERROR nova.compute.manager [instance: 797cb75d-5ab2-4231-b972-db08257716f4] Failure: ['XENAPI_PLUGIN_FAILURE', 'upload_vhd2', 'PluginError', 'Got Permanent Error response [404] while uploading image [79f5d221-0e61-4ee9-82e4-69f507219b78] to glance [http://192.168.33.1:9292/v2/images/79f5d221-0e61-4ee9-82e4-69f507219b78/file]'] | 14:20 |
| BobBall | Tis only upload, yes, not download. | 14:21 |
| *** esberglu has quit IRC | 14:21 | |
| BobBall | I think. | 14:21 |
| mriedem | what changed? | 14:21 |
| BobBall | Trying to figure that out. | 14:21 |
| *** eharney has joined #openstack-nova | 14:22 | |
| *** ducttape_ has quit IRC | 14:22 | |
| *** ducttape_ has joined #openstack-nova | 14:22 | |
| *** ducttape_ has quit IRC | 14:23 | |
| mriedem | johnthetubaguy: sfinucan: i've flip flopped and decided to nack anymore of https://blueprints.launchpad.net/nova/+spec/use-service-tokens | 14:23 |
| BobBall | first failure was 11 hours ago on change 427536 | 14:24 |
| *** liverpooler has joined #openstack-nova | 14:24 | |
| mriedem | cinder and neutron are done, let's do the glance stuff in pike | 14:24 |
| johnthetubaguy | mriedem: yeah, I am fine with that, thats like Friday right? | 14:24 |
| mriedem | johnthetubaguy: yeah | 14:24 |
| mriedem | i think so anyway | 14:24 |
| sfinucan | mriedem: What he said ^ | 14:24 |
| mriedem | i believe stable/ocata is created from rc1 | 14:24 |
| *** mkoderer has joined #openstack-nova | 14:24 | |
| *** cdelatte has quit IRC | 14:25 | |
| bauzas | mriedem: it is | 14:26 |
| bauzas | mriedem: we have to write the cut | 14:26 |
| bauzas | mriedem: have you seen dhellmann's emails about that ? | 14:26 |
| bauzas | I can do a WIP if you wish | 14:26 |
| mriedem | bauzas: yes i know about it | 14:26 |
| bauzas | and you would just update with the right SHA1 | 14:26 |
| BobBall | I have a meeting in 3 minutes... Will have to look into the CI after the meeting. | 14:27 |
| BobBall | mriedem / johnthetubaguy - suggest you remove the Citrix CI from voting until I can fix? | 14:28 |
| *** Swami has joined #openstack-nova | 14:28 | |
| bauzas | mriedem: okay, but I mean, you okay with writing the releases change ? | 14:28 |
| BobBall | (or figure out what's going on...) | 14:28 |
| bauzas | mriedem: that would create the branch | 14:28 |
| mriedem | bauzas: yes | 14:28 |
| *** ducttape_ has joined #openstack-nova | 14:28 | |
| bauzas | kewl | 14:28 |
| mriedem | we have a lot of shit to fix before then :( | 14:28 |
| mriedem | BobBall: looks like that's already taken care of https://review.openstack.org/#/admin/groups/511,members | 14:29 |
| BobBall | +1 | 14:29 |
| *** jamesdenton has joined #openstack-nova | 14:30 | |
| jroll | mriedem: think we should be using the same service token for ironic interactions (rather than separate auth configs we use today) | 14:30 |
| *** ducttape_ has quit IRC | 14:31 | |
| *** READ10 has joined #openstack-nova | 14:31 | |
| jroll | slash johnthetubaguy ^ | 14:31 |
| *** crushil has joined #openstack-nova | 14:32 | |
| johnthetubaguy | mriedem: this one seems ready now, and good to have: https://review.openstack.org/#/c/421760/ | 14:32 |
| *** jheroux has joined #openstack-nova | 14:33 | |
| mriedem | jroll: is that a question? | 14:33 |
| mriedem | johnthetubaguy: yeah it's on the list | 14:33 |
| johnthetubaguy | jroll: possibly, we need to work out what we want keystone roll wise. I quite like the "i am nova" service token being used for all the cases we want for that | 14:33 |
| johnthetubaguy | jroll: I would rather we move to user token + nova service user, for all interactions where possible, so it clear where the request came from at all times | 14:34 |
| jroll | mriedem: oh yeah, punctuation is hard, that's a question | 14:34 |
| johnthetubaguy | jroll: need some oslo.policy tweaks to get that working though | 14:34 |
| jroll | johnthetubaguy: yeah, need to think more about it, user token would be nice for e.g. hitting neutron on our side | 14:35 |
| *** esberglu has joined #openstack-nova | 14:35 | |
| *** cleong has joined #openstack-nova | 14:35 | |
| mriedem | we also don't know what the performance implications are yet | 14:35 |
| johnthetubaguy | mriedem: very true, it very much depends on that not being total suck | 14:36 |
| mriedem | i think the hope is, in pike we pick a ci job and turn the service user stuff on | 14:36 |
| * mriedem adds that to the ptg ethrepad | 14:36 | |
| mriedem | *etherpad | 14:36 |
| cdent | it's cuz you're talking to a brit | 14:36 |
| johnthetubaguy | mriedem: I am asking our internal CI to do some proper performance trials with/without, but they don't sound too excited yet | 14:36 |
| johnthetubaguy | cdent: heh | 14:36 |
| *** wiggin15 has quit IRC | 14:38 | |
| *** esberglu_ has quit IRC | 14:39 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova: WIP - libvirt: Remove redundant bdm serial mangling and saving during swap_volume https://review.openstack.org/427364 | 14:39 |
| openstackgerrit | Chris Dent proposed openstack/nova: Add placement request id to log when GET or POST rps https://review.openstack.org/427740 | 14:39 |
| *** dikonoor has quit IRC | 14:39 | |
| *** jaypipes has joined #openstack-nova | 14:41 | |
| jroll | johnthetubaguy: mriedem: ok, thanks, will revisit later on :) | 14:41 |
| * jroll hopes to do a bunch of nova/ironic improvements in pike | 14:41 | |
| mriedem | jroll: didn't you get your fill of ironic features into nova in ocata? | 14:42 |
| mriedem | i feel like i reviewed an endless stream of feature parity patches :) | 14:42 |
| jroll | mriedem: not features, quality and such | 14:43 |
| * johnthetubaguy hears sweet music in the air | 14:45 | |
| cdent | jaypipes: I'm gonna go get some air, but if you want me (because you're busy or whatever) to get those tweaks on the ironic inventory stuff, leave me a message somewhere and I will | 14:45 |
| mriedem | panda: bnemec: did you guys get your issue resolved with novaclient 7.1.0? | 14:45 |
| jaypipes | cdent: nah, I'll take care of em. thanks for reviewing. | 14:46 |
| *** sree has joined #openstack-nova | 14:46 | |
| cdent | ✔ | 14:46 |
| mriedem | johnthetubaguy: jaypipes: how realistic is that patch at this point? | 14:46 |
| *** dave-mccowan has joined #openstack-nova | 14:46 | |
| *** burt has joined #openstack-nova | 14:46 | |
| jaypipes | mriedem: trying my best. I think I should be able to get it ready today. | 14:46 |
| jaypipes | mriedem: I added a bunch of functional test to the patch yesterday. | 14:46 |
| bauzas | mriedem: mmm, just saw https://bugs.launchpad.net/nova/+bug/1660878 | 14:47 |
| openstack | Launchpad bug 1660878 in OpenStack Compute (nova) "test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"" [High,Confirmed] | 14:47 |
| mriedem | bauzas: yes it's in the ML too | 14:47 |
| mriedem | i have a debug patch up for it | 14:47 |
| mriedem | https://review.openstack.org/#/c/427544/ | 14:47 |
| *** baoli has joined #openstack-nova | 14:47 | |
| * cdent wanders | 14:47 | |
| panda | mriedem: we think yes, but we have problem merging the change since puppet syntax gate is failing due to a new deprecation warning. | 14:48 |
| bauzas | mriedem: not sure I saw this thread, looking | 14:48 |
| *** jpena|lunch is now known as jpena | 14:48 | |
| mriedem | bauzas: it's the papercuts thread | 14:49 |
| mriedem | bauzas: the thread doesn't matter, the bug has the details https://bugs.launchpad.net/nova/+bug/1660878 | 14:49 |
| openstack | Launchpad bug 1660878 in OpenStack Compute (nova) "test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"" [High,Confirmed] | 14:49 |
| bauzas | oh ok | 14:49 |
| bauzas | not [nova] tagged :( | 14:49 |
| bauzas | I saw it, but passed it | 14:50 |
| bauzas | s/passed/skipped | 14:50 |
| sfinucan | bauzas: Just spotted your review on https://review.openstack.org/#/c/392643/. Thank you :) | 14:50 |
| bauzas | I should hire an assistant | 14:50 |
| mriedem | bauzas: melwitt: dansmith: looks like i got a hit on my debug patch http://logs.openstack.org/44/427544/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/f50a5ee/logs/screen-n-cond.txt#_2017-02-01_04_49_11_702 | 14:50 |
| *** sree has quit IRC | 14:50 | |
| bauzas | like the one our French Presidency candidate is hiring | 14:50 |
| jaypipes | "Harry Harrington" is officially the best-named new OpenStack core nominee. | 14:53 |
| *** cdelatte has joined #openstack-nova | 14:53 | |
| mriedem | dansmith: so i've got a recreate on that failure, we know the build request is gone before we create the instance in conductor, now i'm just not sure what i do with it - we don't put that in cell0 since if the build request is deleted before the instance is created, we just treat it like the instance was deleted, i.e. it wouldn't be in ERROR state in cell0 | 14:54 |
| mriedem | i think i can just return | 14:55 |
| dansmith | mriedem: ah I misunderstood your assertion last night then | 14:55 |
| dansmith | mriedem: yeah, if build request is gone before we create the point is we're supposed to just bail quietly.. we do that in at least one place now | 14:55 |
| mriedem | yup here https://review.openstack.org/#/c/427544/2/nova/conductor/manager.py@936 | 14:55 |
| mriedem | which for this bug is too late | 14:55 |
| dansmith | mriedem: ah, yep, indeed | 14:56 |
| mriedem | dansmith: also, | 14:56 |
| mriedem | unrelated, but this return should be a continue right? https://review.openstack.org/#/c/427544/2/nova/conductor/manager.py@972 | 14:57 |
| mriedem | as we're in a for loop | 14:57 |
| dansmith | mriedem: that's not in a for loop | 14:57 |
| mriedem | sure it is https://review.openstack.org/#/c/427544/2/nova/conductor/manager.py@875 | 14:58 |
| mriedem | it's like a 200LOC method so it's easy to miss | 14:58 |
| dansmith | ooh, well, let me check | 14:58 |
| mriedem | ok we can fix separately, i'll clean up my patch for the actual first fix | 14:59 |
| dansmith | yeah, since we've got multiple build requests I guess so, if someone were to delete one of the multi-instances in the middle | 14:59 |
| *** zz_dimtruck is now known as dimtruck | 14:59 | |
| dansmith | aye | 14:59 |
| mriedem | yar! | 14:59 |
| mriedem | matey | 14:59 |
| sfinucan | johnthetubaguy: One question on https://review.openstack.org/#/c/421760/10/api-ref/source/servers.inc | 15:01 |
| *** satyar has joined #openstack-nova | 15:01 | |
| *** prateek has joined #openstack-nova | 15:01 | |
| mriedem | i'll even put on green grass and high tides since this should take 10 minutes | 15:01 |
| dansmith | mriedem: so, I'm not sure why you think this delete-before-create thing is related to the message in your bug about rebuild | 15:01 |
| sfinucan | In that file, the parameter is recorded as 'accessIPv4', but in the release note [1] it's recorded as 'access_ip_v4' [1] https://review.openstack.org/#/c/421760/10/releasenotes/notes/add-whitelist-for-server-list-filter-sort-params-2ae766d03ba895e5.yaml | 15:02 |
| mriedem | dansmith: the test creates the server, then before it's active it deletes it, and then tries to rebuild it, expecting a 404, but gets a 409 | 15:02 |
| sfinucan | I assume the former is automagically transformed to the latter | 15:02 |
| sfinucan | ? | 15:02 |
| mriedem | dansmith: it's a negative test, expecting the deleted instance to be deleted, | 15:02 |
| mriedem | but when we hit the 409, the build request is gone but conductor created the instance (from the stale build request) | 15:02 |
| dansmith | mriedem: ah, okay cool | 15:03 |
| mriedem | dansmith: it's not just rebuild, there is reboot and some other negative tests that have the same pattern | 15:03 |
| dansmith | ack | 15:03 |
| mriedem | i'm just happy it's not more complicated than this :) | 15:03 |
| * mriedem knocks on wood | 15:03 | |
| dansmith | def totes | 15:03 |
| *** liverpooler has quit IRC | 15:05 | |
| *** yamahata has quit IRC | 15:08 | |
| *** liverpooler has joined #openstack-nova | 15:08 | |
| *** tongli has quit IRC | 15:09 | |
| bauzas | mriedem: I have now gained more b/w, which change should I bite now ? | 15:10 |
| bauzas | looking at https://etherpad.openstack.org/p/nova-ocata-rc1-todos | 15:11 |
| mriedem | bauzas: https://review.openstack.org/#/c/427367/ | 15:12 |
| mriedem | bauzas: and https://review.openstack.org/#/c/421760/ | 15:13 |
| bauzas | ack | 15:13 |
| mriedem | and https://review.openstack.org/#/c/426810/ | 15:14 |
| mriedem | basically, just read the etherpad and pick :) | 15:14 |
| *** udesale has joined #openstack-nova | 15:14 | |
| mriedem | sdague: you may want to chime in on the postgres job thread in the dev list | 15:15 |
| mriedem | mordred: ^ | 15:15 |
| mordred | uhoh | 15:15 |
| mriedem | the support matrix that won't die :) | 15:16 |
| vsaienk0 | nova team, since today morning/yesterday evening ironic multinode job is failing due to https://bugs.launchpad.net/nova/+bug/1661014 | 15:18 |
| openstack | Launchpad bug 1661014 in OpenStack Compute (nova) "Multinode job fails with "Compute host X not found"" [Undecided,New] | 15:18 |
| sdague | mriedem: sure | 15:18 |
| *** hongbin has joined #openstack-nova | 15:19 | |
| vsaienk0 | is it known problem? | 15:19 |
| *** sridharg has quit IRC | 15:19 | |
| mriedem | vsaienk0: http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html | 15:19 |
| mriedem | vsaienk0: https://bugs.launchpad.net/nova/+bug/1660973 is a duplicate | 15:20 |
| openstack | Launchpad bug 1660973 in networking-bgpvpn ""No hosts found to map to cell, exiting" / PlacementNotConfigured exception" [Critical,In progress] - Assigned to Thomas Morin (tmmorin-orange) | 15:20 |
| dansmith | mriedem: you sure that's the same? | 15:20 |
| mriedem | it's not | 15:20 |
| mriedem | nvm | 15:20 |
| mriedem | http://logs.openstack.org/75/427675/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ipmitool-tinyipa-multinode-ubuntu-xenial-nv/3ff2401/logs/screen-n-cpu.txt.gz?level=TRACE is clean | 15:21 |
| dansmith | yeah | 15:21 |
| *** tanvirt has joined #openstack-nova | 15:21 | |
| dansmith | n-cond too, an that's the important one for the mapping bug | 15:21 |
| vsaienk0 | mriedem: on ironic multinode job placement api is enabled | 15:21 |
| mriedem | vsaienk0: then it's something else | 15:22 |
| dansmith | vsaienk0: is that test trying to target specific hosts with a boot? | 15:22 |
| *** owalsh is now known as owalsh-brb | 15:22 | |
| *** liangy has joined #openstack-nova | 15:22 | |
| mordred | mriedem: done | 15:22 |
| mordred | mriedem: I believe I said the same thing as sdague - exept worse and with more words | 15:23 |
| mriedem | mordred: that's why i pinged you :) | 15:23 |
| vsaienk0 | dansmith: yes, test_schedule_to_all_nodes is failing | 15:24 |
| dansmith | vsaienk0: I see that from the logs, I'm asking what the test is doing, but I'll go look for myself | 15:25 |
| jaypipes | sileht: thx for your fix on https://review.openstack.org/#/c/427667/. zzzeek, if you can give your attention to this patch, that would be appreciated. I will leave it to mriedem, sdague or bauzas to +W. | 15:25 |
| vsaienk0 | dansmith: that is nova test, I don't know what exactly it is doing | 15:26 |
| dansmith | vsaienk0: okay, yeah, so it's trying to specifically boot an instance targeted at a node | 15:26 |
| jaypipes | sileht: for the record, the reason that uuid was the grouping expression there was that the original query I was using only returned the UUIDs of the resource providers :) | 15:26 |
| *** owalsh-brb has quit IRC | 15:27 | |
| dansmith | vsaienk0: so it may be similar to the issue mriedem pointed above, but manifesting differently, give me a sec to look at the logs | 15:27 |
| jaypipes | rpodolyaka: thx for the heads up on https://review.openstack.org/#/c/427667/ :) | 15:27 |
| bauzas | sileht: oh, nice catch, thanks for fixing that | 15:27 |
| *** prateek has quit IRC | 15:27 | |
| dansmith | vsaienk0: is this failing all the time now or just occasionally | 15:28 |
| vsaienk0 | dansmith: all the time | 15:28 |
| dansmith | vsaienk0: okay, let me put up a hacky devstack patch to try something | 15:28 |
| vsaienk0 | dansmith thanks! | 15:30 |
| bauzas | sileht: jaypipes: and FWIW, rpodolyaka is right, I did the GROUP BY because I wanted the HAVING clause | 15:30 |
| *** liverpooler has quit IRC | 15:30 | |
| bauzas | sileht: jaypipes: but that should work by the PK too | 15:30 |
| rpodolyaka | jaypipes: np! | 15:30 |
| *** jd__ has left #openstack-nova | 15:31 | |
| dansmith | vsaienk0: can you do a canary change against ironic, depending on this? https://review.openstack.org/427772 | 15:31 |
| *** lucasagomes is now known as lucas-brb | 15:31 | |
| vsaienk0 | dansmith: done https://review.openstack.org/#/c/427675/ waiting for results, thanks! | 15:32 |
| *** artom has quit IRC | 15:33 | |
| *** breitz has quit IRC | 15:33 | |
| dansmith | vsaienk0: okay, this is the thing I was pointing to yesterday, which I suspect you're hitting. if this fixes it, we'll need to put it in a proper place of course | 15:33 |
| *** owalsh-brb has joined #openstack-nova | 15:33 | |
| *** owalsh-brb is now known as owalsh | 15:33 | |
| *** mdnadeem has quit IRC | 15:33 | |
| *** artom has joined #openstack-nova | 15:33 | |
| *** breitz has joined #openstack-nova | 15:33 | |
| *** baoli has quit IRC | 15:34 | |
| openstackgerrit | Christopher Martin proposed openstack/nova: Stop using mox in test_compute_cells.py https://review.openstack.org/389750 | 15:34 |
| *** ducttape_ has joined #openstack-nova | 15:34 | |
| dansmith | vsaienk0: oh you guys are a devstack plugin, well that's why I couldn't find stuff... going forward we can just put this in your lib/ironic thing, so .. easier | 15:34 |
| *** ducttape_ has quit IRC | 15:34 | |
| *** ducttape_ has joined #openstack-nova | 15:35 | |
| *** armax has joined #openstack-nova | 15:35 | |
| sfinucan | bauzas: Think there's a very minor logging mistake in https://review.openstack.org/#/c/427200, though otherwise LGTM | 15:36 |
| bauzas | sfinucan: you can hurt me with a -1, I won't hit you :p | 15:37 |
| sfinucan | bauzas: I don't usually -1 for things I'm not 100% sure are an issue, but you asked for it ;) | 15:37 |
| bauzas | and I even prefer or I could miss the comments :) | 15:37 |
| bauzas | sfinucan: well, I'm French so arguing is like my 2nd nature | 15:38 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: Ensure build request exists before creating instance https://review.openstack.org/427775 | 15:39 |
| mriedem | dansmith: melwitt: bauzas: ^ i'm assuming some sort of transactional lock would be best there, but not sure how to do it | 15:39 |
| dansmith | mriedem: you can't, they're in separate databases | 15:40 |
| mriedem | thought about that | 15:40 |
| mriedem | yeah | 15:40 |
| dansmith | mriedem: you should create the instance, then try to delete the buildreq, and if that fails, delete the instance | 15:40 |
| dansmith | like we do below | 15:40 |
| mriedem | we have a window either way | 15:42 |
| mriedem | i'd rather not create the instance in the first place | 15:42 |
| mriedem | if the build request doesn't exist before we create the instance, this catches it, if the build request doesn't exist after we create the instance, the existing check at the bottom handles it | 15:42 |
| dansmith | mriedem: there's no window this way, | 15:43 |
| *** mptacekx has quit IRC | 15:43 | |
| dansmith | mriedem: and the check at the bottom will have to undo a bunch of other stuff, and will have done actions, notifications, etc | 15:43 |
| dansmith | mriedem: but your call | 15:43 |
| dansmith | mriedem: actually, yeah, that's fine | 15:44 |
| mriedem | huh? why my change we don't create the instance, and continue, so the actions and notifications don't happen | 15:44 |
| dansmith | mriedem: I meant if you delay the delete until below | 15:44 |
| *** liverpooler has joined #openstack-nova | 15:44 | |
| mriedem | s/why/with/ | 15:44 |
| *** baoli has joined #openstack-nova | 15:45 | |
| openstackgerrit | Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163 | 15:45 |
| dansmith | since we don't have the instance mapping until a little later, we can't delete the build request yet up there, which means we have to wait | 15:45 |
| *** ekuris__ has quit IRC | 15:46 | |
| *** Swami_ has joined #openstack-nova | 15:47 | |
| bauzas | mriedem: dansmith: catching up your convo | 15:47 |
| openstackgerrit | Lee Yarwood proposed openstack/nova: libvirt: Remove redundant bdm serial mangling and saving during swap_volume https://review.openstack.org/427364 | 15:47 |
| dansmith | mriedem: let me try to linearize the mess that came out of my head above | 15:47 |
| dansmith | mriedem: when we create the instance where we do today, it's not find-able as an instance by the api until the instance mapping is created below | 15:48 |
| bauzas | dansmith: mriedem: I think most of the corner cases about synchronizing have been written by alaski in its conductor method | 15:48 |
| bauzas | lemme find it | 15:48 |
| dansmith | mriedem: so if we delete the buildreq due to an external delete, | 15:48 |
| dansmith | mriedem: and then create the instance, it's still not find-able until we add the instance mapping and then delete the buildreq | 15:49 |
| dansmith | mriedem: so there might be a race between creating the mapping and noticing the buildreq is gone, but that is an extremely short time period | 15:49 |
| dansmith | make sense? | 15:49 |
| *** Swami has quit IRC | 15:50 | |
| *** rcernin has quit IRC | 15:51 | |
| *** liverpooler has quit IRC | 15:51 | |
| *** Sukhdev has joined #openstack-nova | 15:51 | |
| mriedem | dansmith: yeah, and my patch makes that window even smaller yes? | 15:51 |
| dansmith | mriedem: hang on, I'm reading _get_instance() | 15:52 |
| *** liangy has quit IRC | 15:52 | |
| mriedem | dansmith: are you suggesting we move all of https://github.com/openstack/nova/blob/d308d1f70e7e840b1b0c8f4307998d89f9a5ddff/nova/conductor/manager.py#L925-L949 to right after the instance is created, before the notification and action stuff? | 15:52 |
| *** liverpooler has joined #openstack-nova | 15:53 | |
| mriedem | if we moved that up, we wouldn't need to keep track of bdms to cleanup... | 15:53 |
| mriedem | i'm going to start on fixing the return/continue bug | 15:54 |
| *** mtanino has joined #openstack-nova | 15:54 | |
| *** tuan_luong has joined #openstack-nova | 15:54 | |
| dansmith | mriedem: check this out: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2295-L2302 | 15:54 |
| dansmith | mriedem: that will short-circuit the logic we put in to make this atomic across both dbs | 15:55 |
| dansmith | mriedem: and it says we should remove it once people have to use cellsv2 | 15:55 |
| *** tbachman has quit IRC | 15:55 | |
| dansmith | that will effectively go straight to the default cell without requiring the buildreq/instmap lock that we have in place | 15:55 |
| mriedem | yeah i saw that last night, and remember specifically asking alaski to put that TODO in there | 15:55 |
| dansmith | so, can we (a) remove that and see how it goes, (b) merge it if it looks good and see if the fails go away? | 15:56 |
| *** ircuser-1 has joined #openstack-nova | 15:56 | |
| openstackgerrit | Dan Smith proposed openstack/nova: WIP: Remove pre-cellsv2 short circuit in instance get https://review.openstack.org/427782 | 15:58 |
| mriedem | dansmith: would we get into that else in this case though? the inst_map could exist and have the cell mapping set | 15:59 |
| mriedem | i guess if that's the case, the cleanup routine in conductor should have already deleted the instance | 15:59 |
| mriedem | crap i havea 1x1, bbiab | 16:00 |
| *** edand has quit IRC | 16:03 | |
| *** udesale has quit IRC | 16:03 | |
| dansmith | hmm | 16:04 |
| *** stvnoyes has quit IRC | 16:05 | |
| *** stvnoyes has joined #openstack-nova | 16:06 | |
| *** trevormc has joined #openstack-nova | 16:07 | |
| bauzas | dansmith: mriedem: so, I actually wonder why we wouldn't reproduce the logic in build_instances() where when refreshing the instance, we accept that there could be an InstanceNotFound ? | 16:08 |
| bauzas | dansmith: mriedem: but here, we would just access the BuildRequest record instead | 16:08 |
| bauzas | when we get_new_instance() | 16:08 |
| dansmith | bauzas: until we have updated the instance mapping the instance itself could not be deleted, and the buildreq is how that happens | 16:08 |
| *** hfu has quit IRC | 16:09 | |
| bauzas | I'm missing a race condition here then | 16:09 |
| *** _ducttape_ has joined #openstack-nova | 16:09 | |
| dansmith | the one on L2311 also introduces a race between deleting the map and the instance, so I'll remove that one too | 16:10 |
| *** marst has joined #openstack-nova | 16:10 | |
| *** JoseMello has joined #openstack-nova | 16:10 | |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova: Implement new attach/detach Cinder flow https://review.openstack.org/330285 | 16:11 |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova: Remove check_attach https://review.openstack.org/335358 | 16:11 |
| *** hfu has joined #openstack-nova | 16:11 | |
| *** hfu has quit IRC | 16:11 | |
| *** hfu has joined #openstack-nova | 16:12 | |
| *** hfu has quit IRC | 16:12 | |
| *** ducttape_ has quit IRC | 16:13 | |
| *** hfu has joined #openstack-nova | 16:13 | |
| *** hfu has quit IRC | 16:13 | |
| bauzas | dansmith: both conditions are for existing instances that needed to have instance mappings, right? | 16:13 |
| bauzas | dansmith: for a fresh new ocata instance, we suppose the mapping is created, right? | 16:14 |
| dansmith | bauzas: as of ocata, all instances old and new have said mappings, yes | 16:14 |
| bauzas | okay, so I agree with you that we should wipe those conditionals | 16:15 |
| bauzas | because it's confusing | 16:15 |
| dansmith | s/confusing/wrong. | 16:16 |
| *** panda is now known as panda|off | 16:18 | |
| *** hieulq has joined #openstack-nova | 16:19 | |
| *** yamahata has joined #openstack-nova | 16:22 | |
| *** mdrabe has quit IRC | 16:23 | |
| bauzas | dansmith: now, say that we remove those conditionals, a GET /servers?uuid=<> would return a 404 in case the BuildRequest is deleted, right? | 16:25 |
| bauzas | dansmith: if so, I think you found the best approach for fixing the issue, kudos | 16:25 |
| dansmith | bauzas: if it's unmapped | 16:25 |
| bauzas | dansmith: but this is mapped only after scheduling, right? | 16:26 |
| dansmith | bauzas: yes | 16:26 |
| bauzas | I mean | 16:26 |
| bauzas | someone deletes an instance | 16:27 |
| *** lucas-brb is now known as lucasagomes | 16:27 | |
| bauzas | then it deletes the mappings | 16:27 |
| bauzas | so, bim | 16:27 |
| *** xinli has joined #openstack-nova | 16:27 | |
| bauzas | we go to the mappings equal None and then baaam, BuildRequestNotFound, pif paf 404 | 16:27 |
| bauzas | unless I missed that | 16:28 |
| bauzas | s/that/something | 16:28 |
| *** mdrabe has joined #openstack-nova | 16:29 | |
| *** tbachman has joined #openstack-nova | 16:30 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova: libvirt: Remove redundant bdm serial mangling and saving during swap_volume https://review.openstack.org/427364 | 16:32 |
| *** owalsh is now known as owalsh-brb | 16:32 | |
| *** chyka has joined #openstack-nova | 16:35 | |
| kashyap | Hi folks, has anyone seen this error with today's Git? | 16:35 |
| kashyap | 017-02-01 15:55:46.453 | No hosts found to map to cell, exiting. | 16:35 |
| mriedem | when conductor cleans up the build request, if it's already gone and it deletes the instance, why doesn't it also clean up the instance mapping? | 16:35 |
| mriedem | dansmith: ^? | 16:35 |
| kashyap | My current DevStack instance is failing with the above. /me looks at the logs | 16:35 |
| mriedem | kashyap: you need to have the placement-api enabled | 16:36 |
| mriedem | kashyap: http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html | 16:36 |
| kashyap | Ah! | 16:36 |
| kashyap | That's it | 16:36 |
| * kashyap clicks | 16:36 | |
| bauzas | mriedem: kashyap: not sure this error is related | 16:37 |
| bauzas | but /me shrugs | 16:37 |
| kashyap | bauzas: But mriedem is right that I don't have the placement API | 16:37 |
| bauzas | sure | 16:37 |
| mriedem | there won't be a host if n-cpu fails to start | 16:37 |
| bauzas | ah true | 16:37 |
| mriedem | which kills map_cell_and_hosts | 16:37 |
| mriedem | or whatever | 16:37 |
| bauzas | okat | 16:37 |
| bauzas | whatever, like you said | 16:37 |
| *** trinaths1 has joined #openstack-nova | 16:37 | |
| trinaths1 | Hi | 16:38 |
| bauzas | mriedem: what do you mean "when conductor cleans up the build request" ? | 16:38 |
| bauzas | mriedem: when we found a place, and then we create the instance?N | 16:38 |
| mriedem | bauzas: https://github.com/openstack/nova/blob/d308d1f70e7e840b1b0c8f4307998d89f9a5ddff/nova/conductor/manager.py#L925-L949 | 16:38 |
| *** tongli has joined #openstack-nova | 16:38 | |
| kashyap | mriedem: This is the only thing that I need, right? - 'placement-api' (Yep, looking here - http://logs.openstack.org/06/425806/8/check/gate-tempest-dsvm-cells-ubuntu-xenial/e958e02/logs/localrc.txt.gz) | 16:39 |
| mriedem | if the build request is deleted before/during scheduling, we delete the instance from the cell, but why don't we also cleanup the instance mapping record? | 16:39 |
| mriedem | kashyap: yes | 16:39 |
| dansmith | mriedem: probably should.. we do in the compute/api if we get similarly interrupted by a delete | 16:39 |
| mriedem | ok, was just thinking about orphans | 16:39 |
| *** jdurgin1 has joined #openstack-nova | 16:40 | |
| alaski | is the instance hard deleted? If there's an instance record there should be a mapping | 16:40 |
| mriedem | alaski: it's not hard deleted no | 16:40 |
| dansmith | alaski: oh good point | 16:40 |
| bauzas | I was about to sat that | 16:40 |
| bauzas | because we soft delete | 16:40 |
| bauzas | but alaski beated me | 16:40 |
| mriedem | the build request is hard-deleted, but if the instance exists then we don't care about the build request | 16:40 |
| mriedem | yeah i suppose for nova list --deleted | 16:40 |
| dansmith | yeah | 16:40 |
| mriedem | touche | 16:41 |
| * mriedem goes back to trying to fix the return/continue thing | 16:41 | |
| dansmith | alaski: nice save from the sidelines :) | 16:41 |
| *** tbachman_ has joined #openstack-nova | 16:42 | |
| alaski | dansmith: I lurk so I can occasionally seem smart still :) | 16:42 |
| dansmith | alaski: not much danger of that being forgotten, but as you wish :) | 16:43 |
| *** tbachman has quit IRC | 16:43 | |
| *** tbachman_ is now known as tbachman | 16:43 | |
| mriedem | now if he'd only tell me how to untangle this one test... | 16:44 |
| mordred | mriedem: dammit man. now I'm participating in a database related thread on the mailing list | 16:44 |
| alaski | have you tried turning it off and back on... | 16:44 |
| mordred | mriedem: there is no way this ends well for anyone | 16:44 |
| *** mlavalle has joined #openstack-nova | 16:49 | |
| bauzas | mriedem: are you considering keeping that https://review.openstack.org/#/c/427775/1 or do you prefer to remove the conditions for the get() method ? | 16:54 |
| bauzas | conditionals* even | 16:54 |
| bauzas | I tend to prefer the latter | 16:54 |
| dansmith | bauzas: we need to remove the TODOs regardless | 16:55 |
| bauzas | we = who does that so I could +2 it ? | 16:55 |
| *** FrankZhang has quit IRC | 16:55 | |
| dansmith | I already have the patch up | 16:55 |
| bauzas | oh, nice | 16:55 |
| dansmith | WIP, so don't go look yet | 16:55 |
| bauzas | please be a gentleman, so I could CC it :) | 16:55 |
| *** tesseract has quit IRC | 16:56 | |
| *** browne has joined #openstack-nova | 16:56 | |
| dansmith | wat? | 16:56 |
| dansmith | oh fuck. that breaks every api samples test.. cripes. | 16:57 |
| *** tongli has quit IRC | 17:00 | |
| *** psachin has quit IRC | 17:01 | |
| *** slaweq has quit IRC | 17:04 | |
| bauzas | dansmith: cells meeting or should we leave you having fun with api sample tests ? | 17:06 |
| dansmith | bauzas: we just had it and you didn't show up | 17:06 |
| dansmith | so we adjourned to do work | 17:06 |
| *** markus_z has quit IRC | 17:06 | |
| *** OPNFV_ has joined #openstack-nova | 17:06 | |
| *** OPNFV_ has left #openstack-nova | 17:06 | |
| bauzas | heh | 17:06 |
| *** Swami_ has quit IRC | 17:08 | |
| *** andreas_s has quit IRC | 17:09 | |
| *** kfarr has quit IRC | 17:09 | |
| *** tuan_luong has quit IRC | 17:10 | |
| *** chyka has quit IRC | 17:11 | |
| mriedem | so i can't figure this out, | 17:11 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/tests/unit/conductor/test_conductor.py#L1513 | 17:11 |
| mriedem | i changed that to self.assertEqual(3, build_and_run_instance.call_count) | 17:11 |
| mriedem | and it always fails saying it's only called once | 17:11 |
| mriedem | but there are 3 build requests in that test | 17:12 |
| mriedem | as far as i can tell, things should be peachy | 17:12 |
| *** liangy has joined #openstack-nova | 17:12 | |
| edleafe | jaypipes: are you working on https://review.openstack.org/#/c/404472/, or do you want me to fix the tests? | 17:13 |
| *** vsaienko has joined #openstack-nova | 17:13 | |
| *** bvanhav__ has joined #openstack-nova | 17:13 | |
| *** bvanhav__ is now known as bvanhav | 17:13 | |
| *** nkrinner is now known as nkrinner_afk | 17:14 | |
| bauzas | mriedem: but we only schedule once with 3 requests in it, right ? https://github.com/openstack/nova/blob/master/nova/tests/unit/conductor/test_conductor.py#L1512 | 17:14 |
| *** bvanhav_ has quit IRC | 17:15 | |
| mriedem | bauzas: it's asserting this is called https://github.com/openstack/nova/blob/d308d1f70e7e840b1b0c8f4307998d89f9a5ddff/nova/conductor/manager.py#L959 | 17:15 |
| mriedem | which is in a for loop | 17:15 |
| mriedem | over the list of build requests | 17:15 |
| mriedem | of which there are 3 | 17:15 |
| mriedem | so unless something is failing and making it continue before that for 2 of them, i'm lost | 17:16 |
| *** lpetrut has quit IRC | 17:17 | |
| bauzas | mriedem: correct me if I'm wrong, but we're only passing one request_spec ? | 17:19 |
| bauzas | so the zip thing would only return one tuple | 17:19 |
| *** owalsh-brb is now known as owalsh | 17:20 | |
| bauzas | mriedem: yup, we're only passing one of those https://github.com/openstack/nova/blob/master/nova/tests/unit/conductor/test_conductor.py#L1414-L1415 | 17:20 |
| *** chyka has joined #openstack-nova | 17:21 | |
| *** hieulq has quit IRC | 17:22 | |
| bauzas | mriedem: so if you wanna make sure you have 3 calls, just add 2 new request_specs in the params dict | 17:22 |
| *** hieulq has joined #openstack-nova | 17:22 | |
| jaypipes | edleafe: yeah, I'm working on it. | 17:23 |
| edleafe | jaypipes: cool - didn't want it to languish if you were busy | 17:23 |
| *** dtp has joined #openstack-nova | 17:23 | |
| jaypipes | edleafe: yeah, sorry, meeting hell day.. | 17:23 |
| *** nic has joined #openstack-nova | 17:24 | |
| *** hieulq has quit IRC | 17:26 | |
| mriedem | bauzas: ah beautiful | 17:28 |
| mriedem | thanks | 17:28 |
| openstackgerrit | Dan Smith proposed openstack/nova: Remove pre-cellsv2 short circuit in instance get https://review.openstack.org/427782 | 17:28 |
| bauzas | \o/ | 17:29 |
| * bauzas takes his pen | 17:29 | |
| breitz | anybody use and IDE like pycharm to debug nova with? I'm trying but failing to find the magic to make it all work | 17:30 |
| *** Swami_ has joined #openstack-nova | 17:30 | |
| dtp | dansmith melwitt you guys need anything? | 17:30 |
| dansmith | dtp: own any guns? | 17:31 |
| dtp | no :( but have been thinking about it lately | 17:31 |
| dansmith | dtp: then, no. | 17:31 |
| dansmith | heh | 17:31 |
| dtp | :) | 17:31 |
| mriedem | dtp: review these? https://review.openstack.org/#/q/topic:man+project:openstack/nova+status:open | 17:32 |
| dtp | ok | 17:32 |
| bauzas | breitz: I think we have something about that | 17:33 |
| bauzas | in our dev doc | 17:33 |
| *** burgerk has quit IRC | 17:33 | |
| bauzas | breitz: but tbh, I just pdb when I want to introspect | 17:33 |
| marst | breitz: what's failing? | 17:33 |
| mriedem | LOG.error is my pdb | 17:33 |
| breitz | thats what I do (pdb) but was looking for getting an IDE working. | 17:33 |
| bauzas | yeah found http://docs.openstack.org/developer/nova/development.environment.html#using-a-remote-debugger | 17:33 |
| bauzas | breitz: ^ | 17:33 |
| *** kylek3h is now known as kylek3h_away | 17:34 | |
| bauzas | mriedem: sometimes you need something more dynamic :p | 17:34 |
| bauzas | like "oh shit, I knew that, I need to run again to see the next call" | 17:34 |
| mriedem | LOG.error(locals())?! | 17:35 |
| breitz | thanks for the link - I'll read through that (some of which I've followed). Basically I can get pycharm to connect to the remote - but I can't get any breakpoints in nova-compute or any exception to drop me into the debugger. | 17:35 |
| bauzas | mriedem: you won't convince me :p | 17:35 |
| mriedem | btw, just let me say, i'm a professional engineer | 17:35 |
| *** Swami_ has quit IRC | 17:35 | |
| bauzas | LOL | 17:36 |
| * dansmith never uses pdb | 17:36 | |
| * bauzas blushes | 17:36 | |
| *** raunak has joined #openstack-nova | 17:36 | |
| bauzas | I even use Atom | 17:36 |
| mriedem | so, are https://review.openstack.org/#/c/427782/ and https://review.openstack.org/#/c/427775/ coexisting? or does the first supersede the second? | 17:36 |
| *** raunak has quit IRC | 17:37 | |
| melwitt | I use pdb occasionally | 17:37 |
| dansmith | mriedem: they can co-exist.. my preference is this: | 17:37 |
| bauzas | mriedem: I don't really like https://review.openstack.org/#/c/427775/1/nova/conductor/manager.py | 17:37 |
| *** karimb has quit IRC | 17:37 | |
| dansmith | mriedem: poke mine a bit and see if we hit the bug with it. If not, I'd like to merge it and get data before we merge yours | 17:37 |
| bauzas | yeah, I think dansmith's change could help us getting a better situation | 17:38 |
| dansmith | mriedem: yours doesn't hurt anything other than performance and maybe making the race harder to catch if it's still there, so I just wanted to delay | 17:38 |
| dansmith | mriedem: but, we're also up against the deadline, so.. | 17:38 |
| mriedem | -W for now | 17:38 |
| cdent | bbl | 17:41 |
| *** cdent has quit IRC | 17:41 | |
| *** Sukhdev has quit IRC | 17:41 | |
| *** trinaths1 has quit IRC | 17:44 | |
| *** dimtruck is now known as zz_dimtruck | 17:47 | |
| openstackgerrit | Oliver Walsh proposed openstack/nova: DNM - TripleO CI test https://review.openstack.org/427536 | 17:52 |
| *** _ducttape_ has quit IRC | 17:54 | |
| *** ducttape_ has joined #openstack-nova | 17:54 | |
| *** kaisers has quit IRC | 17:54 | |
| *** ducttape_ has quit IRC | 17:54 | |
| *** ducttape_ has joined #openstack-nova | 17:55 | |
| *** esberglu has quit IRC | 17:55 | |
| *** ducttape_ has quit IRC | 17:55 | |
| *** esberglu has joined #openstack-nova | 17:56 | |
| *** ducttape_ has joined #openstack-nova | 17:56 | |
| *** aarefiev is now known as aarefiev_afk | 17:57 | |
| *** esberglu_ has joined #openstack-nova | 17:57 | |
| *** ssurana has joined #openstack-nova | 17:58 | |
| *** owalsh is now known as owalsh-afk | 17:58 | |
| dims_ | mriedem : melwitt : dansmith : ServersNegativeTestJSON failed in the gate about 8-10 at least for the past 4 hours. looks like we are waiting on https://review.openstack.org/#/c/427775 | 17:59 |
| *** trevormc has left #openstack-nova | 17:59 | |
| dims_ | ah which transitively is waiting on https://review.openstack.org/#/c/427782/ | 17:59 |
| dansmith | dims_: we've been working on it all morning | 17:59 |
| dims_ | ack dansmith | 17:59 |
| *** esberglu has quit IRC | 18:00 | |
| dims_ | just reporting back from watching the CI that's all :) | 18:00 |
| *** derekh has quit IRC | 18:01 | |
| mriedem | you're gonna get stabbed dims | 18:01 |
| *** esberglu_ has quit IRC | 18:01 | |
| mriedem | lots of trolls in this cave right now :) | 18:01 |
| dims_ | LOL | 18:01 |
| mriedem | ok got this return/continue test finally passing | 18:02 |
| mriedem | woot | 18:02 |
| melwitt | dims_: yeah, it turns out https://review.openstack.org/#/c/427782 might supersede the other patch | 18:02 |
| dims_ | thanks melwitt | 18:03 |
| *** ralonsoh has quit IRC | 18:04 | |
| *** harlowja_ has joined #openstack-nova | 18:05 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova: Continue processing build requests even if one is gone already https://review.openstack.org/427839 | 18:05 |
| mriedem | melwitt: dansmith: bauzas: dtp: ^ | 18:05 |
| dtp | k | 18:06 |
| *** harlowja has quit IRC | 18:06 | |
| *** mkoderer has quit IRC | 18:06 | |
| *** lucasagomes is now known as lucas-afk | 18:08 | |
| *** esberglu has joined #openstack-nova | 18:09 | |
| *** owalsh-afk has quit IRC | 18:09 | |
| *** esberglu has quit IRC | 18:09 | |
| *** esberglu has joined #openstack-nova | 18:09 | |
| *** Apoorva has joined #openstack-nova | 18:10 | |
| *** owalsh-afk has joined #openstack-nova | 18:12 | |
| *** kaisers has joined #openstack-nova | 18:13 | |
| * mriedem gets some food | 18:14 | |
| *** zz_dimtruck is now known as dimtruck | 18:15 | |
| *** lpetrut has joined #openstack-nova | 18:16 | |
| *** davidgiluk has quit IRC | 18:19 | |
| *** vsaienko has quit IRC | 18:20 | |
| *** vsaienko has joined #openstack-nova | 18:24 | |
| *** raunak has joined #openstack-nova | 18:24 | |
| *** pcaruana has quit IRC | 18:28 | |
| *** bvanhav_ has joined #openstack-nova | 18:30 | |
| *** bvanhav has quit IRC | 18:31 | |
| *** unicell has joined #openstack-nova | 18:32 | |
| *** dtp is now known as dtp-afk | 18:33 | |
| *** krtaylor has quit IRC | 18:33 | |
| openstackgerrit | Dan Smith proposed openstack/nova: Allow scheduler to run cell host discovery periodically https://review.openstack.org/426826 | 18:37 |
| dansmith | sfinucan: mriedem: ^^ 0 means "default" which means "every minute" which means "meltdown" | 18:37 |
| dansmith | also, I added run_immediately, which will mean the scheduler will do this on startup and make this more useful for things like devstack | 18:38 |
| sfinucan | dansmith: Oh, damn. I should have remembered that 🙈 | 18:38 |
| dansmith | me too :/ | 18:39 |
| sfinucan | Good catch, nonetheless | 18:39 |
| * sfinucan looking | 18:39 | |
| mriedem | yar https://github.com/openstack/oslo.service/blob/master/oslo_service/periodic_task.py#L108 | 18:39 |
| mriedem | https://github.com/openstack/oslo.service/blob/master/oslo_service/periodic_task.py#L122 | 18:39 |
| dansmith | yeah | 18:40 |
| mriedem | i think the last time i had to deal with the innards of the periodic task code was in ATL at the juno summit | 18:41 |
| dansmith | mriedem: but, setting it =3600 will mean it runs once an hour, but at scheduler startup which will be useful for ironic people | 18:41 |
| mriedem | i specifically remember myself and dims whispering in the back of someone's session about it | 18:41 |
| mriedem | whispering, maybe some giggling | 18:41 |
| mriedem | yeah on startup seems ok | 18:42 |
| *** bvanhav_ is now known as bvanhav | 18:42 | |
| mriedem | ha | 18:45 |
| mriedem | "I think Russian already owns the bear." | 18:45 |
| melwitt | oh yeah, the mascot angst thread | 18:46 |
| mriedem | i didn't have the context, i thought he was just saying, generally, | 18:47 |
| mriedem | russia owns all likenesses of bears | 18:47 |
| mriedem | period | 18:47 |
| mriedem | which is something i'd expect from a russian given my US stereotype of them from the 80s | 18:47 |
| dansmith | I thought Bear == Trump | 18:47 |
| mriedem | gonna need to call in rocky balboa to settle this one | 18:47 |
| melwitt | I think it's explained by a later reply about a russian meme | 18:48 |
| mriedem | melwitt: yeah i read further | 18:48 |
| mriedem | dansmith: trump is more of a peacock | 18:48 |
| mriedem | beautiful to behold, but functionally useless | 18:48 |
| dansmith | um. | 18:48 |
| mriedem | what about "donal trump is beautiful to behold" don't you get?! | 18:49 |
| melwitt | mriedem: I get what you meant now. it sounded like that to me too before someone else explained it | 18:50 |
| *** dharinic is now known as dharinic|lunch | 18:50 | |
| *** vsaienko has quit IRC | 18:50 | |
| mriedem | random thought: no one has asked me lately if i can smell what the rock is cooking | 18:55 |
| *** vsaienko has joined #openstack-nova | 18:57 | |
| *** mvk has quit IRC | 18:57 | |
| *** mjura has quit IRC | 18:58 | |
| *** sree has joined #openstack-nova | 19:02 | |
| *** ducttape_ has quit IRC | 19:02 | |
| *** ducttape_ has joined #openstack-nova | 19:03 | |
| *** vsaienko has quit IRC | 19:04 | |
| *** vsaienko has joined #openstack-nova | 19:05 | |
| artom | Oh God, Bear/Bare metal | 19:06 |
| * artom just got >_< | 19:06 | |
| *** jaosorior has joined #openstack-nova | 19:06 | |
| *** sree has quit IRC | 19:07 | |
| *** vsaienko has quit IRC | 19:07 | |
| openstackgerrit | John Garbutt proposed openstack/nova-specs: WIP: write up policy ideas for pike https://review.openstack.org/427872 | 19:08 |
| *** lpetrut has quit IRC | 19:08 | |
| *** vsaienko has joined #openstack-nova | 19:09 | |
| *** adisky_ has quit IRC | 19:09 | |
| *** tbachman_ has joined #openstack-nova | 19:11 | |
| *** vsaienko has quit IRC | 19:11 | |
| *** lpetrut has joined #openstack-nova | 19:11 | |
| *** annegentle has joined #openstack-nova | 19:12 | |
| *** yamahata has quit IRC | 19:12 | |
| *** tbachman has quit IRC | 19:12 | |
| *** tbachman_ is now known as tbachman | 19:12 | |
| *** vsaienko has joined #openstack-nova | 19:12 | |
| mriedem | johnthetubaguy: have you taken a gander at this xen live migration patch? https://review.openstack.org/#/c/424428/ | 19:13 |
| *** kfarr has joined #openstack-nova | 19:13 | |
| mriedem | it mods the rpc version on the migrate_data object so won't be able to backport it if we don't get it in | 19:13 |
| *** krtaylor has joined #openstack-nova | 19:13 | |
| *** vsaienko has quit IRC | 19:17 | |
| *** harlowja_ has quit IRC | 19:19 | |
| mriedem | dansmith: looks like your change is failing some functional tests http://logs.openstack.org/82/427782/2/check/gate-nova-tox-db-functional-ubuntu-xenial/f53a7e9/console.html#_2017-02-01_17_41_39_148413 | 19:19 |
| dansmith | mriedem: I know, already fixed, just wanted to get a devstack run before I push it up again | 19:20 |
| dansmith | looks like I'm good | 19:20 |
| mriedem | ok | 19:20 |
| mriedem | fwiw i agree with bauzas on keeping the tests but changing them to assert that InstanceNotFound is raised | 19:20 |
| dansmith | I know, and I hate you guys for that, but working on that too | 19:20 |
| mriedem | \o/ | 19:20 |
| mriedem | bauzas: https://review.openstack.org/#/c/427667/ passed after the recheck, are you happy? | 19:22 |
| mriedem | nvm, i +W | 19:24 |
| johnsom | I just did a fresh reclone devstack this morning and cirros is going to error on boot with: | 19:24 |
| johnsom | n-cpu.log:2017-02-01 11:13:32.876 ERROR os_vif [req-17c8b4e4-2197-4205-aed3-007d0f2837e4 admin admin] Failed to plug vif | 19:24 |
| johnsom | VIFBridge(active=False,address=fa:16:3e:6f:0e:84,bridge_name='qbrd3377ad5-43',has_traffic_filtering=True,id=d3377ad5-4397-474c-b6dd-e29b9a52d277,network=Network(d2582e93-d6e5-43ac-be7b-fe3fc3c7e593),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tapd3377ad5-43') | 19:24 |
| johnsom | Is this a known thing? | 19:24 |
| mriedem | johnsom: no, is there an actual error? | 19:24 |
| mriedem | either in nova or neutron? | 19:24 |
| *** krtaylor has quit IRC | 19:24 | |
| johnsom | | fault | {"message": "Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=False,address=fa:16:3e:6f:0e:84,bridge_name='qbrd3377ad5-43',has_traffic_filtering=True,id=d3377ad5-4397-474c-b6dd-e29b9a52d277,network=Network(d2582e93-d6e5-43ac-be7b-fe3fc3c7", "code": 500, "details": " File | 19:24 |
| johnsom | \"/opt/stack/nova/nova/compute/manager.py\", line 1780, in _do_build_and_run_instance | | 19:24 |
| *** efried has quit IRC | 19:25 | |
| *** sambetts is now known as sambetts|afk | 19:25 | |
| openstackgerrit | Dan Smith proposed openstack/nova: Remove pre-cellsv2 short circuit in instance get https://review.openstack.org/427782 | 19:25 |
| dansmith | mriedem: ^ | 19:25 |
| mriedem | johnsom: gonna need a stacktrace or something from n-cpu logs | 19:25 |
| mriedem | johnsom: in a paste | 19:25 |
| mriedem | the error could be on the neutron side too | 19:25 |
| johnsom | https://www.irccloud.com/pastebin/1IAgqzBE/ | 19:25 |
| johnsom | Ah, found it | 19:25 |
| mriedem | cool | 19:26 |
| *** vsaienko has joined #openstack-nova | 19:26 | |
| dansmith | lol | 19:27 |
| manjeets | hi I just hit the issue in devstack saying placement is not configured | 19:29 |
| manjeets | http://paste.openstack.org/show/597262/ | 19:29 |
| manjeets | is there any doc that explains that configuration ?? | 19:30 |
| mriedem | manjeets: http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html | 19:30 |
| *** krtaylor has joined #openstack-nova | 19:31 | |
| *** READ10 has quit IRC | 19:31 | |
| openstackgerrit | Vasyl Saienko proposed openstack/nova: Revert "Fresh resource provider in RT must have generation 0" https://review.openstack.org/427877 | 19:32 |
| *** ociuhandu has quit IRC | 19:33 | |
| mriedem | vsaienko: is that just for testing? | 19:33 |
| vsaienko | mriedem yes | 19:33 |
| *** mvk has joined #openstack-nova | 19:34 | |
| *** dharinic|lunch is now known as dharinic | 19:37 | |
| johnsom | So, looking at that trace, I'm not sure if it's a nova, neutron, or oslo issue... | 19:37 |
| *** Apoorva has quit IRC | 19:38 | |
| mriedem | dansmith: want me to work https://review.openstack.org/#/c/426826/ into https://review.openstack.org/#/c/424737/ or should we just mention the periodic task in the cells v2 devref? | 19:38 |
| manjeets | thanks mriedem | 19:38 |
| mriedem | or you couldn't care less and wish i'd stop asking about docs | 19:39 |
| ildikov | mriedem: sdague: hi | 19:41 |
| mriedem | local.conf for cinder v3 testing right? | 19:41 |
| ildikov | mriedem: sdague: just wanted to ask what would be the best timing to get to the details of figuring out the Cinder V3 testing in the placement API job? | 19:41 |
| dansmith | mriedem: probably needs to be in both places | 19:42 |
| dansmith | mriedem: I'll do it | 19:42 |
| mriedem | dansmith: ok, was just adding the "Once fully upgraded" to the first issue bullet | 19:42 |
| dansmith | mriedem: oh I can do that too, I just got distracted with real stuff | 19:42 |
| dansmith | but whatever you ant | 19:43 |
| dansmith | *want | 19:43 |
| mriedem | dansmith: or, just throw the mention of the periodic into a new patch on top of https://review.openstack.org/#/c/424737/ with it's own reno | 19:43 |
| mriedem | i don't care too much either way | 19:43 |
| mriedem | ildikov: probably next week | 19:43 |
| dansmith | mriedem: the periodic needs a recheck anyway, so if you drop the -W I can just add it in there | 19:44 |
| mriedem | ildikov: it's going to require being able to pass a post-config via local.conf from the job definition in project-config through d-g into devstack | 19:44 |
| dansmith | er, drop the +W | 19:44 |
| *** Sukhdev has joined #openstack-nova | 19:44 | |
| mriedem | dansmith: done | 19:44 |
| ildikov | mriedem: we are confident at this point that the post-config direction will be accepted, right? | 19:45 |
| ildikov | mriedem: I guess we don't have too many choices here anyhow though | 19:45 |
| *** vsaienko has quit IRC | 19:45 | |
| mriedem | ildikov: sdague is the one making it happen i believe | 19:45 |
| mriedem | because it's the direction he wants | 19:46 |
| mriedem | so yes i'm pretty confident | 19:46 |
| ildikov | mriedem: great, that sounds like a good starting point | 19:46 |
| ildikov | thanks | 19:46 |
| *** cdent has joined #openstack-nova | 19:47 | |
| *** efried has joined #openstack-nova | 19:48 | |
| sdague | yeh, the biggest issue is we're now past freeze so until everyone is on an open master again, poking at devstack-gate in that way is going to be dangerous | 19:48 |
| sdague | ildikov: https://review.openstack.org/#/c/326585/ was the change that got the base support over | 19:48 |
| mriedem | f yes http://logs.openstack.org/26/426926/6/check/gate-grenade-dsvm-neutron-ubuntu-xenial/9019244/logs/grenade.sh.txt.gz#_2017-02-01_19_01_54_881 | 19:50 |
| mriedem | dansmith: jaypipes: melwitt: sdague: bauzas: ^ nova-status upgrade check in grenade | 19:50 |
| sdague | mriedem: cool | 19:51 |
| openstackgerrit | Dan Smith proposed openstack/nova: Allow scheduler to run cell host discovery periodically https://review.openstack.org/426826 | 19:51 |
| *** mtanino has quit IRC | 19:51 | |
| ildikov | sdague: got it, thanks! | 19:51 |
| dansmith | mriedem: ^ | 19:52 |
| mriedem | yup readibg | 19:52 |
| dansmith | mriedem: that status check gives me butterflies | 19:52 |
| ildikov | sdague: so basically as soon as master gets open for Pike we can get it sorted out | 19:52 |
| *** ociuhandu has joined #openstack-nova | 19:53 | |
| mriedem | dansmith: one minor thing in https://review.openstack.org/#/c/426826/6/releasenotes/notes/scheduler-can-discover-hosts-4b799cbd14dbc7dc.yaml | 19:54 |
| dansmith | ah yeah | 19:54 |
| openstackgerrit | Dan Smith proposed openstack/nova: Allow scheduler to run cell host discovery periodically https://review.openstack.org/426826 | 19:55 |
| sdague | ildikov: yeh, it's going to take some time to work through things. Honestly, that would probably be worth PTG time on the infra / qa side to sketch out the path forward | 19:55 |
| *** sree has joined #openstack-nova | 19:55 | |
| sdague | because it's not a closed loop yet | 19:55 |
| mriedem | ildikov: i think infra has an etherpad for ptg | 19:55 |
| mriedem | ildikov: probably worth getting that on there | 19:55 |
| sdague | and honestly been having a heck of a time getting that devstack-gate patch reviewed | 19:55 |
| mriedem | https://etherpad.openstack.org/p/infra-ptg-pike | 19:55 |
| *** cdent has quit IRC | 19:55 | |
| mriedem | jaypipes: meet me in -operators | 19:56 |
| *** mtanino has joined #openstack-nova | 19:59 | |
| mriedem | dansmith: is 0 positive? | 19:59 |
| mriedem | dansmith: or did you not want to distinguish between <0 for the periodic? | 19:59 |
| mriedem | since 0 will make it run | 19:59 |
| *** claudiub has quit IRC | 19:59 | |
| dansmith | mriedem: zero is positive | 19:59 |
| *** sree has quit IRC | 19:59 | |
| *** avolkov has quit IRC | 20:00 | |
| *** vsaienko has joined #openstack-nova | 20:00 | |
| mriedem | sfinucan: https://review.openstack.org/#/c/426826/7 | 20:01 |
| mriedem | sfinucan: latest adds a release note | 20:01 |
| mriedem | dansmith: if you have butterflies about that nova-status + grenade thing, then https://review.openstack.org/#/c/427499/ is up your alley | 20:03 |
| mriedem | https://review.openstack.org/#/c/426926/ was the grenade change testing it | 20:03 |
| *** gouthamr has quit IRC | 20:03 | |
| *** tanvirt has quit IRC | 20:06 | |
| *** crushil has quit IRC | 20:06 | |
| *** crushil has joined #openstack-nova | 20:06 | |
| *** Sukhdev has quit IRC | 20:08 | |
| *** mtanino has quit IRC | 20:09 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova: doc: add some detail to the map_cell0 man page https://review.openstack.org/427450 | 20:10 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: doc: refer to the cell_v2 man pages from the cells v2 doc https://review.openstack.org/427459 | 20:10 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: doc: update the man page entry for nova-manage db sync https://review.openstack.org/427460 | 20:10 |
| ildikov | mriedem: sdague: thanks, I added the patch and a note to the etherpad | 20:10 |
| *** mtanino has joined #openstack-nova | 20:12 | |
| *** crushil has quit IRC | 20:20 | |
| *** crushil has joined #openstack-nova | 20:20 | |
| artom | http://logs.openstack.org/59/424759/12/gate/gate-nova-releasenotes/d0bf975/console.html#_2017-02-01_19_58_38_712633 | 20:21 |
| artom | *stab* | 20:21 |
| artom | *stab stab stab stab* | 20:21 |
| artom | (Unrelated to any important currently ongoing issues) | 20:21 |
| mriedem | artom: oh that sucks | 20:22 |
| mriedem | 3 in the gate | 20:22 |
| artom | The gate is Kanye'ing me | 20:23 |
| artom | "Your patch is nice and imma let in finish - but this guy - this guy has the best patch of all time!" | 20:23 |
| openstackgerrit | Dan Smith proposed openstack/nova: Allow scheduler to run cell host discovery periodically https://review.openstack.org/426826 | 20:25 |
| openstackgerrit | Dan Smith proposed openstack/nova: Reserve migration placeholders for Ocata backports https://review.openstack.org/427900 | 20:25 |
| openstackgerrit | Dan Smith proposed openstack/nova: Add ComputeNode.mapped field https://review.openstack.org/427901 | 20:25 |
| openstackgerrit | Dan Smith proposed openstack/nova: Add ability to query for ComputeNodes by their mapped value https://review.openstack.org/427902 | 20:25 |
| openstackgerrit | Dan Smith proposed openstack/nova: Make discover_hosts only query for unmapped ComputeNode records https://review.openstack.org/427903 | 20:25 |
| mriedem | bauzas: replied to your comments in https://review.openstack.org/#/c/427367/ but i didn't see anything that i needed to change | 20:26 |
| mriedem | i might be misunderstanding your points though | 20:26 |
| dansmith | mriedem: ^ is what I'm thinking, sans any tests or testing | 20:26 |
| mriedem | tests i can do without, but not testing | 20:26 |
| mriedem | :) | 20:26 |
| dansmith | yeah, well, eventually, I just wanted it out there | 20:27 |
| dansmith | in case I stab a fork in my eye between now and pike | 20:27 |
| dansmith | also, the migration backports are in that stack | 20:27 |
| dansmith | placeholders I mean | 20:27 |
| bauzas | mriedem: sorry, I meant in https://review.openstack.org/#/c/427367/2/doc/source/placement.rst@175 that a scheduler hard failure could also happen once you upgrade all your computes, not only in Pike | 20:27 |
| mriedem | bauzas: oh i see | 20:28 |
| *** nmagnezi has joined #openstack-nova | 20:28 | |
| bauzas | mriedem: and about the bullet good place, I meant that since operators need to modify their nova.conf for adding the placement section, it could be nice to let them know quite close that they also need to update ratios | 20:29 |
| bauzas | but that's a nit | 20:29 |
| mriedem | bauzas: even if n-sch can't talk to placement after you've upgraded your computes, | 20:29 |
| mriedem | we'll fallback to querying the compute_nodes table | 20:29 |
| mriedem | so it's not the same kind of hard failure | 20:29 |
| bauzas | mriedem: not exactly | 20:29 |
| bauzas | mriedem: because if you don't update the n-sch nova.conf, then it would return NoValidHosts | 20:30 |
| mriedem | none of the compute services would be running, which would be a different kind of failure | 20:30 |
| johnsom | I have opened a bug for the os_vif issue I am seeing: https://bugs.launchpad.net/nova/+bug/1661086 | 20:30 |
| openstack | johnsom: Error: Could not gather data from Launchpad for bug #1661086 (https://launchpad.net/bugs/1661086). The error has been logged | 20:30 |
| *** jamielennox|away is now known as jamielennox | 20:30 | |
| johnsom | Ha, everything is melting down... | 20:30 |
| bauzas | mriedem: I mean, if the operator modifies nova.conf for all computes but forgets to do so for n-sch, and then upgrades all their computes, then we have the scheduler calling Placement, righT? | 20:31 |
| mriedem | bauzas: oh i see this https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L184 | 20:31 |
| bauzas | mriedem: exactly | 20:31 |
| bauzas | that's what I call an hard failure | 20:32 |
| bauzas | *a* hard failure even | 20:32 |
| bauzas | oh man, why English people not doing the liaison ? | 20:32 |
| mriedem | hmm, so your point is good, and makes the earlier part of that bullet not completely accurate | 20:33 |
| mriedem | "While the FilterScheduler technically depends on the Placement service | 20:33 |
| mriedem | in Ocata, if you deploy the Placement service `after` you upgrade the | 20:33 |
| mriedem | ``nova-scheduler`` service to Ocata and restart it, things will still work." | 20:33 |
| *** bvanhav has quit IRC | 20:33 | |
| bauzas | you still could | 20:33 |
| bauzas | because n-sch won't call placement until all computes are new | 20:33 |
| mriedem | if your computes are all upgraded, then we'll try placement and if it's not there, or n-sch isn't configured, we'll fail with NoValidHost | 20:33 |
| mriedem | yeah i'm saying when all computes are good | 20:33 |
| bauzas | so you can possibly run placement *after* n-sch | 20:34 |
| mriedem | but either you (1) haven't deployed placement or (2) haven't configured n-sch for placement and restarted it | 20:34 |
| bauzas | but not aftre you're done with all your computes, that's it | 20:34 |
| mriedem | there was a whole hullabaloo about this in the ML about the order of when placement gets deployed, before or after n-sch | 20:34 |
| bauzas | yeah | 20:34 |
| bauzas | but we're still good | 20:34 |
| bauzas | because now, TripleO and Puppet folks can just upgrade in the order they want | 20:35 |
| *** JoseMello has quit IRC | 20:35 | |
| bauzas | except they need to upgrade computes last | 20:35 |
| bauzas | which is the exact sequence we already have | 20:35 |
| bauzas | so nothing is changing for them | 20:35 |
| *** browne has quit IRC | 20:36 | |
| mriedem | maybe i should just remove this entirely: "However, the scheduler not being able to make requests to Placement will | 20:36 |
| mriedem | result in a hard failure in the 16.0.0 Pike release." | 20:36 |
| bauzas | possibly | 20:36 |
| bauzas | the thing is, nova.conf has to be updated for all services in Ocata, period | 20:36 |
| bauzas | so yeah, we could say what terrible thing could happen to them | 20:37 |
| mriedem | i'll say, "However, once all computes are upgraded, the scheduler not being able to make requests to Placement will result in **NoValidHost** errors" | 20:37 |
| bauzas | fine by me | 20:37 |
| bauzas | just update it quick and I'll +2 | 20:37 |
| *** browne has joined #openstack-nova | 20:37 | |
| *** sambetts|afk has quit IRC | 20:37 | |
| bauzas | that's docs | 20:37 |
| bauzas | and we can refine that later | 20:37 |
| bauzas | hence my +1 | 20:37 |
| bauzas | I personnally even think we could fast-approve doc things - but I'm nuts :) | 20:38 |
| *** sambetts_ has joined #openstack-nova | 20:38 | |
| *** harlowja has joined #openstack-nova | 20:39 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova: doc: add upgrade notes to the placement devref https://review.openstack.org/427367 | 20:40 |
| mriedem | bauzas: done ^ | 20:40 |
| bauzas | mriedem: I had two comments about things you haven't added, so I could amend that too in a follow-up | 20:41 |
| mriedem | oh i totally missed the first 2 comments | 20:42 |
| bauzas | no worries, that's unrelated to your chnage | 20:42 |
| bauzas | but I was reading the HTML directly | 20:43 |
| bauzas | so when I saw that, I thought it would be good to change that too | 20:43 |
| mriedem | i replied, | 20:43 |
| mriedem | basically, i think they are still ok statements given we want to be a bit forceful about people enabling this stuff before upgrading to ocata | 20:44 |
| mriedem | even though we have some fallback stuff in place in ocata to soften the blow if they don't get it all done | 20:44 |
| mriedem | the ocata upgrade notes section gives the specific details on stuff you must have done before upgrading | 20:44 |
| bauzas | mriedem: fine, I'm not opiniated | 20:45 |
| mriedem | are you sure about that? :P | 20:45 |
| bauzas | I'm not opiniated *by the docs* | 20:45 |
| bauzas | :p | 20:45 |
| mriedem | oh ok | 20:46 |
| bauzas | I mean, it's fine to be a bit forceful, like you said | 20:46 |
| *** tbachman_ has joined #openstack-nova | 20:48 | |
| *** browne has quit IRC | 20:48 | |
| *** browne has joined #openstack-nova | 20:49 | |
| *** tbachman has quit IRC | 20:51 | |
| *** tbachman_ is now known as tbachman | 20:51 | |
| *** aloga_ has joined #openstack-nova | 20:56 | |
| *** aloga_ has quit IRC | 20:56 | |
| *** aloga_ has joined #openstack-nova | 20:56 | |
| *** aloga_ has quit IRC | 20:56 | |
| *** _ducttape_ has joined #openstack-nova | 20:58 | |
| mriedem | bauzas: melwitt: +2 on https://review.openstack.org/#/c/427782/ now | 20:59 |
| mriedem | ci is happy | 20:59 |
| *** Sukhdev has joined #openstack-nova | 20:59 | |
| bauzas | mriedem: dansmith: Wipped | 21:00 |
| *** liverpooler has quit IRC | 21:01 | |
| mriedem | easy +W https://review.openstack.org/#/c/426810/ | 21:01 |
| *** ducttape_ has quit IRC | 21:02 | |
| *** tbachman_ has joined #openstack-nova | 21:02 | |
| *** gouthamr has joined #openstack-nova | 21:03 | |
| *** tbachman has quit IRC | 21:04 | |
| *** tbachman_ is now known as tbachman | 21:04 | |
| *** tbachman has quit IRC | 21:04 | |
| *** jose-phillips has joined #openstack-nova | 21:05 | |
| openstackgerrit | Sylvain Bauza proposed openstack/nova: Add a status check for enabled filters https://review.openstack.org/427200 | 21:08 |
| bauzas | mriedem: looking | 21:09 |
| *** armax has quit IRC | 21:11 | |
| openstackgerrit | Sylvain Bauza proposed openstack/nova: reno for notification-transformation-ocata https://review.openstack.org/426810 | 21:11 |
| bauzas | mriedem: please +W ^ | 21:12 |
| bauzas | I just fixed a typo | 21:12 |
| mriedem | nice catch | 21:12 |
| dansmith | bauzas: please, not in public! | 21:13 |
| bauzas | what ? | 21:14 |
| bauzas | talking about some guy ? | 21:14 |
| dansmith | bauzas: about "whipping" :) | 21:15 |
| bauzas | that's where my English vocabulary epic fails | 21:15 |
| *** tbachman has joined #openstack-nova | 21:15 | |
| * bauzas goes googling | 21:16 | |
| bauzas | hah | 21:16 |
| *** liangy has quit IRC | 21:20 | |
| *** erlon has quit IRC | 21:20 | |
| openstackgerrit | Vasyl Saienko proposed openstack/nova: testing https://review.openstack.org/427930 | 21:22 |
| *** tanvirt has joined #openstack-nova | 21:23 | |
| *** vsaienko has quit IRC | 21:25 | |
| *** vsaienko has joined #openstack-nova | 21:26 | |
| *** eharney has quit IRC | 21:28 | |
| *** Guest30167 has quit IRC | 21:29 | |
| *** erlon has joined #openstack-nova | 21:30 | |
| *** kfarr has quit IRC | 21:32 | |
| jaypipes | edleafe: really sorry, I haven't had a chance to do the work on the ironic patch :( my day just got consumed with meetings. :( any chance you can pick it up? | 21:34 |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| *** claudiub has joined #openstack-nova | 21:35 | |
| *** Jeffrey4l_ has joined #openstack-nova | 21:36 | |
| *** _ducttape_ has quit IRC | 21:37 | |
| *** catintheroof has quit IRC | 21:37 | |
| *** ducttape_ has joined #openstack-nova | 21:37 | |
| *** catintheroof has joined #openstack-nova | 21:37 | |
| *** Apoorva has joined #openstack-nova | 21:38 | |
| jroll | jaypipes: edleafe: cdelatte (assume this is cdent): mriedem: fyi, once this merges in, the experimental job for nova that has "resource-class" in the name will exercise that ironic resource tracker patch | 21:38 |
| *** browne has quit IRC | 21:38 | |
| jroll | oh, link would be good https://review.openstack.org/#/c/407651/ | 21:38 |
| *** Apoorva has quit IRC | 21:40 | |
| mriedem | that patch is looking pretty high risk atm | 21:40 |
| mriedem | says a guy that hasn't looked at it since last week | 21:40 |
| *** tbachman has quit IRC | 21:41 | |
| *** Apoorva has joined #openstack-nova | 21:41 | |
| *** vsaienko has quit IRC | 21:42 | |
| *** eharney has joined #openstack-nova | 21:44 | |
| *** edmondsw has quit IRC | 21:44 | |
| *** slaweq has joined #openstack-nova | 21:46 | |
| edleafe | jaypipes: ok, I can work on it | 21:46 |
| *** READ10 has joined #openstack-nova | 21:46 | |
| *** browne has joined #openstack-nova | 21:50 | |
| *** edmondsw has joined #openstack-nova | 21:53 | |
| *** armax has joined #openstack-nova | 21:54 | |
| mriedem | edleafe: jaypipes: to be clear, barring some miracle, i think that one is probably too risky at this point given the size and complexity https://review.openstack.org/#/c/404472/ | 21:55 |
| mriedem | and we're <24 hours to rc1 | 21:55 |
| jaypipes | mriedem: yes, understood. | 21:56 |
| *** satyar has quit IRC | 21:56 | |
| mriedem | ok | 21:56 |
| mriedem | just didn't want any surprises come tomorrow | 21:56 |
| jaypipes | yeah, it's my fault anyway. | 21:56 |
| mriedem | which i realize sucks given everything that already went in to get here | 21:56 |
| mriedem | jaypipes: i'm not placing blame or anything - you've done a shitload of work to get to this point | 21:57 |
| jaypipes | mriedem: it's cool, man | 21:57 |
| *** catintheroof has quit IRC | 21:57 | |
| edleafe | mriedem: jaypipes:should have a revision within the hour | 21:57 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: Collected release notes for Ocata CellsV2 https://review.openstack.org/424737 | 21:58 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: Add release note for nova-status upgrade check CLI https://review.openstack.org/426911 | 21:58 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: Add prelude section for Ocata https://review.openstack.org/426864 | 21:58 |
| mriedem | melwitt: sfinucan: bauzas: dansmith: i've rev'ed the renos for the prelude ^ | 21:58 |
| *** catintheroof has joined #openstack-nova | 21:58 | |
| bauzas | mriedem: kewl | 21:58 |
| edleafe | jaypipes: if you could give me a quick answer to my question on https://review.openstack.org/#/c/404472/29/nova/compute/resource_tracker.py@81 | 21:59 |
| bauzas | mriedem: jaypipes: given https://review.openstack.org/#/c/404472/ would not be Ocata, maybe we should think on how it could be a problem for scheduling Ironic RCs in Pike | 22:00 |
| bauzas | because I think we could then only check those by Queens | 22:00 |
| mriedem | bauzas: i didn't say it was'nt going to happen, | 22:00 |
| mriedem | but i'm saying it's very close and tight at this point | 22:00 |
| *** tbachman has joined #openstack-nova | 22:00 | |
| mriedem | i don't really follow the issue you're bringing up though either, | 22:01 |
| mriedem | but we always have service versions.. | 22:01 |
| bauzas | honestly, I think it could be a bit difficult to have it merged by tomorow | 22:01 |
| mriedem | i do too | 22:01 |
| mriedem | which is why i said that | 22:01 |
| mriedem | but i'm not following the queens comment | 22:01 |
| bauzas | I wonder if we should then maybe add a service version for ^ | 22:02 |
| bauzas | so we could be knowing if the compute is providing Ironic versions | 22:02 |
| mriedem | bauzas: well, we could do that in pike couldn't we? | 22:02 |
| bauzas | yeah probably | 22:02 |
| mriedem | what checks the service versions? the scheduler? | 22:02 |
| bauzas | yup | 22:02 |
| mriedem | then it's the same as we did in your change | 22:03 |
| bauzas | the other problem is that we could only have ironic RCs if the Ironic API is happy with that | 22:03 |
| mriedem | we're not sure about scheduling with these yet are we? i remember something something extra specs, but | 22:03 |
| mriedem | we're already past the point of minimum ironic API for supporting node.resource_class | 22:04 |
| bauzas | sorry, it's late so that's possibly why I'm a bit confusing people | 22:04 |
| mriedem | http://docs.openstack.org/developer/ironic/dev/webapi-version-history.html | 22:04 |
| mriedem | 1.21 added node.resource_class | 22:04 |
| mriedem | we require 1.29 or something right now of ironic | 22:04 |
| bauzas | I'm trying to think how to know if this RP is providing new custom RCs | 22:04 |
| mriedem | well, i guess ocata compute requires ironic >=1.29 | 22:04 |
| mriedem | gd i have like a 15 minute deep purple live in tokyo organ solo in my ears right now | 22:05 |
| mriedem | i can't think straight either | 22:05 |
| bauzas | tbc, if we don't find custom RCs, does that mean - either the RP (compute) is not yet having the new Ironic resources , - or, that Ironic doesn't provide new RCs | 22:05 |
| mriedem | the node doesn't have to have a resource_class set, | 22:06 |
| mriedem | regardless of whether or not we support that level of the ironic api in the ironic driver | 22:06 |
| mriedem | btw, newton compute supports the minimum ironic api that supports resource classes https://github.com/openstack/nova/blob/stable/newton/nova/virt/ironic/client_wrapper.py#L37 | 22:06 |
| bauzas | cool then | 22:07 |
| mriedem | anyway, the scheduler won't use placement until computes are ocata or newer, | 22:07 |
| mriedem | which supports high enough ironic api | 22:07 |
| *** gouthamr has quit IRC | 22:07 | |
| mriedem | i thought the thing was, the flavor extra spec would have the resource class in it, and if it's there, then we filter on ironic RPs using that resource class | 22:07 |
| * jroll can't tell if there's a question he can help answer | 22:07 | |
| mriedem | jroll: i just need you to appreciate my live in tokyo reference | 22:08 |
| jroll | +1 | 22:08 |
| mriedem | because dansmith failed to do so | 22:08 |
| * jroll plays it cause why not | 22:08 | |
| mriedem | jroll: we're just talking about what the implications are for our ability to do things in pike if https://review.openstack.org/#/c/404472/ doesn't make ocata | 22:09 |
| bauzas | alright | 22:09 |
| jroll | mriedem: so we need to be putting inventory (based on resource class) in the placement API a cycle before we can filter on it | 22:09 |
| jroll | I think | 22:09 |
| *** ducttape_ has quit IRC | 22:09 | |
| mriedem | jroll: well, we can use nova-compute service versions to flag that i think | 22:09 |
| mriedem | i.e. https://review.openstack.org/#/c/404472/ bumps the compute service version | 22:09 |
| *** thorst_ has quit IRC | 22:09 | |
| jroll | maybe I shouldn't try to think this through on irc for the fifth time and just hang out with y'all for a while at ptg | 22:09 |
| mriedem | so once all of your computes are at that level, then we support filter on resource class in the scheduler | 22:09 |
| *** ijw has joined #openstack-nova | 22:09 | |
| dansmith | mriedem: I failed to do what? | 22:10 |
| *** ducttape_ has joined #openstack-nova | 22:10 | |
| jroll | mriedem: where does that bump the version? | 22:10 |
| mriedem | jroll: that patch doesn't today | 22:10 |
| * jroll doesn't really know what a service version is, unless maybe that's an rpc version | 22:10 | |
| jroll | ah | 22:10 |
| mriedem | we're saying if it doesn't land until pike, we do it in pike | 22:10 |
| jroll | that explains everything | 22:10 |
| mriedem | jroll: e.g. https://review.openstack.org/#/c/425806/ | 22:11 |
| bauzas | I'm just trying to find the implications | 22:11 |
| mriedem | then we used that here https://review.openstack.org/#/c/417961/41/nova/scheduler/filter_scheduler.py@174 | 22:11 |
| bauzas | and how if we don't merge it by Ocata, how to flag that | 22:11 |
| mriedem | so with service version 16, your nova-compute won't start unless configured for placement, | 22:12 |
| jroll | aha | 22:12 |
| mriedem | and then the scheduler checks if the minimum nova-compute in the deployment is at least 16, and if so, it goes hog wild on placement | 22:12 |
| mriedem | i'm saying we do the same for ironic resource classes if we need to in pike | 22:12 |
| bauzas | so the scheduler could possibly know whether it could verify new-way | 22:12 |
| mriedem | so if you have ocata computes, then you don't get that function yet | 22:12 |
| mriedem | but once you're computes are upgraded to pike, you get it | 22:12 |
| mriedem | *your | 22:12 |
| bauzas | exactly | 22:13 |
| jroll | and then we'll have... if 'extra_spec:custom_foo' in request_spec.flavor AND service_version > the_one_with_resource_classes: placement_client.find_nodes_with_class('custom_foo') | 22:13 |
| jroll | or whatever | 22:13 |
| jroll | else do the old crap | 22:13 |
| mriedem | yeah basically | 22:13 |
| bauzas | something like that indeed | 22:13 |
| jroll | and in queens we make the extra spec thing mandatory and maybe even fix flavor to not need that | 22:14 |
| *** cleong has quit IRC | 22:14 | |
| mriedem | maybe | 22:14 |
| mriedem | i can't think n+2 | 22:14 |
| jroll | fair | 22:14 |
| jroll | s/queens/future | 22:14 |
| mriedem | by at least pike-eol :) | 22:14 |
| jroll | regardless, pike looks like progress | 22:15 |
| jroll | heh | 22:15 |
| jroll | sounds good to me, thanks | 22:15 |
| *** ijw has quit IRC | 22:15 | |
| * jroll waits a day to update resource class docs so he knows if this lands or not | 22:15 | |
| mriedem | solid progress either way i think | 22:15 |
| *** ducttape_ has quit IRC | 22:15 | |
| *** slaweq has quit IRC | 22:15 | |
| *** thorst_ has joined #openstack-nova | 22:16 | |
| *** lpetrut has quit IRC | 22:19 | |
| *** thorst_ has quit IRC | 22:20 | |
| *** jamesdenton has quit IRC | 22:25 | |
| *** ijw has joined #openstack-nova | 22:26 | |
| *** jheroux has quit IRC | 22:27 | |
| *** browne has quit IRC | 22:28 | |
| *** tanvirt has quit IRC | 22:31 | |
| *** gouthamr has joined #openstack-nova | 22:33 | |
| *** catintheroof has quit IRC | 22:35 | |
| *** nmagnezi has left #openstack-nova | 22:37 | |
| *** dimtruck is now known as zz_dimtruck | 22:38 | |
| *** crushil has quit IRC | 22:39 | |
| *** thorst_ has joined #openstack-nova | 22:40 | |
| *** edmondsw has quit IRC | 22:41 | |
| *** thorst_ has quit IRC | 22:44 | |
| mriedem | this is wrong right? https://review.openstack.org/#/c/408571/41/nova/api/openstack/compute/schemas/servers.py@277 | 22:45 |
| mriedem | should be terminated_at? | 22:45 |
| *** tblakes has quit IRC | 22:46 | |
| *** ducttape_ has joined #openstack-nova | 22:47 | |
| *** marst has quit IRC | 22:49 | |
| *** ducttape_ has quit IRC | 22:49 | |
| *** _ducttape_ has joined #openstack-nova | 22:49 | |
| melwitt | mriedem: I think so. that's what the db column is called, anyway 'terminated_at' | 22:51 |
| *** tlian has quit IRC | 22:51 | |
| mriedem | yeah i'm going through https://review.openstack.org/#/c/421760/10/api-ref/source/parameters.yaml | 22:51 |
| *** karimb has joined #openstack-nova | 22:52 | |
| *** haplo37 has quit IRC | 22:52 | |
| *** _ducttape_ has quit IRC | 22:53 | |
| *** baoli has quit IRC | 22:54 | |
| *** yamahata has joined #openstack-nova | 22:55 | |
| *** crushil has joined #openstack-nova | 22:57 | |
| *** tlian has joined #openstack-nova | 22:58 | |
| mriedem | fixing | 22:58 |
| *** yamahata has quit IRC | 23:00 | |
| *** yamahata has joined #openstack-nova | 23:01 | |
| *** esberglu has quit IRC | 23:01 | |
| openstackgerrit | Ed Leafe proposed openstack/nova: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | 23:03 |
| edleafe | jaypipes: mriedem: ^^ tests should be fixed now | 23:03 |
| jaypipes | edleafe: ty ed! | 23:03 |
| *** raunak has quit IRC | 23:04 | |
| *** mlavalle has quit IRC | 23:04 | |
| *** haplo37 has joined #openstack-nova | 23:05 | |
| *** annegentle has quit IRC | 23:06 | |
| mriedem | can you list servers with a query parameter with a 'none' value, to make it equate to NULL in the db query, i.e. GET /servers/?terminated_at=none | 23:06 |
| mriedem | ? | 23:06 |
| mriedem | i guess let's find out | 23:07 |
| *** burt has quit IRC | 23:08 | |
| *** tlian2 has joined #openstack-nova | 23:10 | |
| *** mdrabe has quit IRC | 23:10 | |
| *** zz_dimtruck is now known as dimtruck | 23:11 | |
| *** tlian has quit IRC | 23:13 | |
| *** tlian2 has quit IRC | 23:15 | |
| tonyb | cdent, mriedem: who did the placement stuff in devstack? and is the placement implementation/upgrade from mitaka -> newton -> ocata in devref documented | 23:18 |
| mriedem | tonyb: cdent did the devstack patches, | 23:19 |
| tonyb | mriedem: okay | 23:19 |
| tonyb | cdent: shoudl expect strange questions in his future | 23:19 |
| mriedem | and http://docs-draft.openstack.org/67/427367/3/check/gate-nova-docs-ubuntu-xenial/de76755//doc/build/html/placement.html | 23:19 |
| mriedem | for devref | 23:19 |
| mriedem | see 'deployment' and 'upgrade notes' | 23:19 |
| tonyb | Thanks! | 23:20 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: Fix the terminated_at field in the server query params schema https://review.openstack.org/427964 | 23:21 |
| mriedem | oomichi: gmann: alex_xu: Kevin_Zheng: ^ | 23:21 |
| mriedem | sdague: ^ | 23:21 |
| mriedem | fwiw, filtering instances by those datetime fields doesn't seem to work anyway | 23:21 |
| mriedem | i have an instance created at 2017-02-01T23:14:26Z according to nova show, but nothing is returned from: | 23:22 |
| mriedem | curl -s -H "X-Auth-Token: $OS_TOKEN" http://9.5.124.95:8774/v2.1/servers?created_at=2017-02-01T23:14:26Z | 23:22 |
| *** browne1 has joined #openstack-nova | 23:23 | |
| mriedem | oh interesting it's a regex | 23:24 |
| mriedem | curl -s -H "X-Auth-Token: $OS_TOKEN" http://9.5.124.95:8774/v2.1/servers?created_at=^2017-02-01.* | 23:24 |
| mriedem | ^ works | 23:24 |
| *** tlian has joined #openstack-nova | 23:24 | |
| *** takashin has joined #openstack-nova | 23:29 | |
| armax | mriedem: is the large ops job running for nova changes? | 23:29 |
| mriedem | armax: the large ops job doesn't run anymore | 23:29 |
| mriedem | as far as i know | 23:30 |
| armax | mriedem: ok, thanks for confirming this | 23:30 |
| mriedem | which reminds me, i was going to kill that stuff from d-g | 23:30 |
| armax | you mean https://review.openstack.org/#/c/228636/2/stackrc? | 23:30 |
| mriedem | nope | 23:30 |
| mriedem | there is some stuff in d-g about the large ops job | 23:30 |
| armax | mriedem: ok | 23:30 |
| mriedem | which i was going to clean out since we don't run that job anymore | 23:30 |
| armax | what about that one? do we want to go back and keep a simpler API_WORKERS expression? | 23:31 |
| armax | I am thinking of setting the minimum num of workers to 1 | 23:31 |
| Kevin_Zheng | mriedem: so you are suggesting separate Reno with API ref patch? | 23:31 |
| armax | that cuts the amount of RSS memory per service quite a bit | 23:31 |
| armax | and should give us some headroom at least for now | 23:32 |
| mriedem | Kevin_Zheng: at this rate, i think we'll probably want to decouple the release note from the rest of that change | 23:32 |
| armax | what do you reckon? | 23:32 |
| mriedem | armax: maximum you mean? | 23:32 |
| mriedem | right now default in devstack is nproc / 2 right? | 23:32 |
| Kevin_Zheng | mriedem: ok, will do it, holiday is over :) | 23:33 |
| mriedem | armax: btw, i'm pretty sure that https://review.openstack.org/#/c/228636/ was all changed after that | 23:33 |
| mriedem | although that was a long time ago so i might be mixed up on when those changes all happened, mtreinish might remember | 23:33 |
| mriedem | Kevin_Zheng: cool. you'll probably want to check the comments/questions in parameters.yaml because it looks like some things are missing, but i might be confused about what's supposed to be in the server query parameter list | 23:34 |
| mriedem | and if that's just sort keys or also filter keys | 23:34 |
| mriedem | i assume that maybe some things were left out of the list in the api-ref if they are things that technically work but we don't want people using, like image_ref | 23:35 |
| mriedem | gmann: ^ | 23:35 |
| armax | mriedem: the outcome of the expression really depends on the number of processors devstack runs in | 23:35 |
| *** browne1 has quit IRC | 23:36 | |
| mriedem | armax: http://lists.openstack.org/pipermail/openstack-dev/2016-February/086232.html | 23:36 |
| mriedem | for your historical enjoyment | 23:36 |
| Kevin_Zheng | mriedem: ok | 23:36 |
| armax | mriedem: thanks :) | 23:36 |
| armax | mriedem: I am trying to figure out what’s the best strategy to set the API_WORKERS variable | 23:37 |
| mriedem | armax: you should probably talk to sdague and mtreinish about that | 23:37 |
| mriedem | since they were dealing with that heavily the last time we had an OOM issue | 23:37 |
| armax | indeed, I was hoping I’d have a concrete proposal first | 23:37 |
| armax | I don’t particularly fancy the idea of reducing the number of workers deliberately, but we’re in a bit of a bind | 23:38 |
| mriedem | dude, | 23:38 |
| mriedem | just wave your hands in the ML | 23:38 |
| mriedem | maybe you'll hook someone :) | 23:39 |
| armax | we did, but maybe not high enough | 23:39 |
| mriedem | jump and wave at the same time | 23:39 |
| mriedem | and scream | 23:39 |
| armax | I’ll give it a shot | 23:41 |
| *** raunak has joined #openstack-nova | 23:43 | |
| *** ducttape_ has joined #openstack-nova | 23:51 | |
| *** xinli has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!