| dansmith | okay then I'll put that and the quota object change up front | 00:00 |
|---|---|---|
| *** tbachman has joined #openstack-nova | 00:00 | |
| melwitt | mriedem: oh yeah, I know that song. I guess people must wave their arms when they listen to it | 00:01 |
| mriedem | i know i do | 00:02 |
| *** pumaranikar has joined #openstack-nova | 00:03 | |
| melwitt | \o\ | 00:04 |
| melwitt | dansmith: ignore my comment on the incompatible signature, just read your newest comment | 00:06 |
| dansmith | well, I jumped to conclusions since it looked like you change the order, which is only half true | 00:06 |
| dansmith | but yeah | 00:06 |
| dansmith | *changed | 00:06 |
| melwitt | yeah. agreed it would be better to add a new one instead of wedge functionality into the old one | 00:07 |
| dansmith | I'm getting a bunch of quota errors with that limit check in front of basically unchanged code | 00:07 |
| *** yingjun has joined #openstack-nova | 00:07 | |
| dansmith | like things going negative or something | 00:08 |
| melwitt | well, that's not expected :( | 00:09 |
| dansmith | about to get a report | 00:09 |
| dansmith | http://pastebin.com/raw/BZWnPULS | 00:10 |
| dansmith | for example | 00:10 |
| melwitt | okay, that's because limit_check can't be used on ReservableResources, which is part of the change in nova/quota.py | 00:12 |
| melwitt | sigh | 00:12 |
| dansmith | is that because the last hunk at the bottom of quota.py thinks everything should be reservable? | 00:13 |
| dansmith | i.e., that list should be augmented in each patch in this new list? | 00:13 |
| melwitt | so, the last hunk is the types of the resources, old stuff has a mix of Countable and Reservable. Countable uses limit_check, Reservable uses quota.reserve | 00:14 |
| melwitt | so the third patch changes everything to Countable and limit_check | 00:14 |
| dansmith | right, so each patch in my new series should add its resource type to the list of countable things | 00:14 |
| dansmith | and the limit_check patch being in front should basically have no change in that hunk, right? | 00:14 |
| melwitt | if each patch is changing a reserve to a limit_check, then yeah | 00:15 |
| melwitt | they have to go in lock-step | 00:15 |
| *** dimtruck is now known as zz_dimtruck | 00:15 | |
| dansmith | melwitt: right, so this is one patch in the series: http://pastebin.com/raw/4Lwcw7EN | 00:16 |
| dansmith | (minus its tests) | 00:16 |
| dansmith | so that one should add CountableResource('server_groups', _server_group_count, 'server_groups'), | 00:16 |
| dansmith | right? | 00:16 |
| melwitt | dansmith: ah yes, that makes it clear. Yeah, not add, but change | 00:16 |
| *** tbachman has quit IRC | 00:17 | |
| *** hongbin has quit IRC | 00:17 | |
| melwitt | change it from ReservableResource to CountableResource and include it's new counting function | 00:17 |
| melwitt | *its | 00:17 |
| dansmith | ah yeah I see | 00:17 |
| dansmith | okay removing that chunk from the first patch to add the new limit check got me much farther | 00:23 |
| dansmith | 195 fails down to 46 | 00:23 |
| dansmith | pretty much all the rest look like this: AttributeError: <nova.quota.ReservableResource object at 0x7fcbda866310> does not have the attribute 'count' | 00:24 |
| dansmith | which seems to be something in the _get_usages() change that expects something from ReservableResource? | 00:24 |
| melwitt | the opposite, expecting there to be a count attr | 00:24 |
| melwitt | like, it's assuming everything is a CountableResource | 00:25 |
| dansmith | _get_usage hard codes which things it thinks are countable I guess | 00:25 |
| melwitt | yeah, since they were part of the same change. to do both, it would have to check the resource type and do each thing accordingly | 00:25 |
| *** edmondsw has joined #openstack-nova | 00:26 | |
| melwitt | I think. I don't remember what _get_usage did originally. looking | 00:26 |
| *** tbachman has joined #openstack-nova | 00:26 | |
| melwitt | oh, it's new | 00:27 |
| dansmith | it should probably look at the resource class to see if it's countable, and maybe more details need to be in there to know if it's per-project or something? | 00:27 |
| dansmith | definitely looks like that could be generalized | 00:27 |
| melwitt | okay, yeah. I wrote that to "get counts" when it used to pull usages from the usages table in the db | 00:27 |
| dansmith | okay well I think if that was generalized then this first patch would pass | 00:28 |
| melwitt | like here it used to be a db.quota_usage_get_all_by_project_and_user() call https://review.openstack.org/#/c/416521/22/nova/quota.py@219 | 00:28 |
| melwitt | dansmith: cool, thanks. please feel free to upload as-is and I can fix them. I feel bad you're spending too much time on it | 00:29 |
| dansmith | good, you should feel bad | 00:29 |
| melwitt | :( | 00:29 |
| melwitt | that's proof, the unsmiley face | 00:29 |
| *** gcb has quit IRC | 00:30 | |
| melwitt | looks like we're not out of the woods on this bug either https://bugs.launchpad.net/nova/+bug/1670627 | 00:30 |
| openstack | Launchpad bug 1670627 in OpenStack Compute (nova) ocata "quota is always in-use after delete the ERROR instances " [Critical,In progress] - Assigned to Matt Riedemann (mriedem) | 00:30 |
| melwitt | I dunno if you saw the latest comment | 00:30 |
| *** edmondsw has quit IRC | 00:31 | |
| dansmith | melwitt: do you really want me to upload this on top of your stack? if so I will, I just don't want to explode your set | 00:31 |
| mriedem | gfdi | 00:31 |
| dansmith | I'm about to retire for the evening anyway | 00:31 |
| melwitt | dansmith: I dunno, is there any other option? | 00:32 |
| mriedem | we do a rollback on the quota change if the instance destroy fails | 00:32 |
| dansmith | melwitt: I can push it up to github or something, but I kinda think we should do this anyway, so that's just indirection | 00:32 |
| dansmith | melwitt: I was just going to try to get it a little closer before pushing, but.. it's up to you | 00:32 |
| *** Apoorva has quit IRC | 00:33 | |
| melwitt | dansmith: yeah, I guess there's a small chance it could get more gnarly. but maybe that's gonna have to be dealt with anyway. yolo | 00:33 |
| dansmith | having spent hours on the patch the way it is, I don't think it can get much gnarly-er | 00:33 |
| dansmith | I didn't really start grok'ing most of it until I was ass-deep in trying to split it up | 00:34 |
| melwitt | mriedem: if we rollback on a destroy fail, then things should be working right I thought | 00:34 |
| melwitt | dansmith: hah, that's fair. yeah, go ahead and upload | 00:34 |
| mriedem | melwitt: only if we hit InstanceNotFound | 00:34 |
| mriedem | melwitt: they didn't say what error they hit though | 00:34 |
| *** baoli has joined #openstack-nova | 00:38 | |
| *** baoli_ has joined #openstack-nova | 00:39 | |
| melwitt | hm, well in _delete_while_booting, it quota.commits immediately, before trying to delete, so that seems wrong. the rollback for InstanceNotFound also wouldn't do anything if commit already happened. but I didn't think we touched _delete_while_booting | 00:39 |
| melwitt | yeah, these are all committing immediately, which I think would make rollback do nothing | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Count resources to check quota for cells https://review.openstack.org/416521 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make Quotas object favor the API database https://review.openstack.org/410945 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add Quotas.check_deltas() and set the stage for magic to happen https://review.openstack.org/446239 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Count server group quotas https://review.openstack.org/446240 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Count tenant_networks quotas https://review.openstack.org/446241 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Count used_limits quotas https://review.openstack.org/446242 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Remove useless quota_usage_refresh from nova-manage https://review.openstack.org/446243 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add get_count_by_vm_state() to Instance object https://review.openstack.org/446244 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add SecurityGroup.get_counts() https://review.openstack.org/446245 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add FixedIP.get_count_by_project() https://review.openstack.org/446246 | 00:40 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add FloatingIP.get_count_by_project() https://review.openstack.org/446247 | 00:40 |
| melwitt | thanks dansmith | 00:40 |
| dansmith | the final patch is like half the size | 00:40 |
| melwitt | woohoo | 00:41 |
| dansmith | fwiw, this makes it so much more obvious why this is good: | 00:41 |
| dansmith | https://review.openstack.org/#/c/446239/1 | 00:41 |
| dansmith | in just quota stuff, +500,-1000 | 00:41 |
| dansmith | anyway, I've made my mess, time for dinner | 00:42 |
| melwitt | yeah, that's a win for sure | 00:42 |
| melwitt | haha, o/ | 00:42 |
| *** baoli has quit IRC | 00:43 | |
| *** liangy has quit IRC | 00:43 | |
| melwitt | mriedem: quotas.commit and quotas.rollback are mutually exclusive and I should have noticed that on the review. rollback won't work if commit already happened | 00:44 |
| *** alexpilotti has joined #openstack-nova | 00:46 | |
| *** dikonoor has joined #openstack-nova | 00:48 | |
| mriedem | melwitt: then wouldn't it also be wrong in _delete_while_booting? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1807 | 00:49 |
| mriedem | that's what i copied this from | 00:49 |
| melwitt | yes, I think it's wrong there too | 00:50 |
| melwitt | I'm looking to see what rollback does if commit already occurred | 00:51 |
| melwitt | I think it's just a no-op because it would find no reservation record | 00:51 |
| *** jianghuaw-m has joined #openstack-nova | 00:52 | |
| *** dharinic_ has joined #openstack-nova | 00:52 | |
| melwitt | commit would have deleted the reservation record. then rollback comes along and reads the reservation, finds nothing, and then does nothing | 00:52 |
| *** iceyao has joined #openstack-nova | 00:53 | |
| *** dharinic1 has joined #openstack-nova | 00:53 | |
| *** sambetts_ has quit IRC | 00:53 | |
| mriedem | well hells bells | 00:55 |
| mriedem | i'm pretty sure _delete_while_booting copied that behavior from _delete | 00:55 |
| *** sambetts_ has joined #openstack-nova | 00:55 | |
| mriedem | so normal delete we get the reservation here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1943 | 00:56 |
| mriedem | so it probably copied from https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2028 | 00:56 |
| mriedem | looks like in that flow though, we don't commit unless we delete https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2023 | 00:57 |
| mriedem | so (1) create reservation, (2) delete instance = works, then commit; else rollback | 00:57 |
| mriedem | i think that's how things are working in the old delete flow | 00:57 |
| mriedem | before the cellsv2ification | 00:57 |
| *** gyee has quit IRC | 00:59 | |
| *** dharinic_ has quit IRC | 01:00 | |
| mriedem | also, i think i'm *this* close to getting this functional test to pass | 01:01 |
| mriedem | fixtures up the wazoo | 01:01 |
| *** baoli_ has quit IRC | 01:01 | |
| mriedem | melwitt: i also went with your idea to stub out Claim.__init__, that was a good idea | 01:06 |
| *** alexpilotti has quit IRC | 01:06 | |
| *** dharinic has joined #openstack-nova | 01:08 | |
| mriedem | oh f yes it works now | 01:08 |
| *** abalutoiu has quit IRC | 01:09 | |
| *** dikonoor has quit IRC | 01:09 | |
| *** kevinz has joined #openstack-nova | 01:12 | |
| *** dharinic has quit IRC | 01:13 | |
| *** armax has joined #openstack-nova | 01:14 | |
| *** jianghuaw-m has quit IRC | 01:15 | |
| *** Sukhdev has quit IRC | 01:17 | |
| *** Apoorva has joined #openstack-nova | 01:18 | |
| mriedem | here it comes | 01:20 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add populate_retry to schedule_and_build_instances https://review.openstack.org/444106 | 01:20 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add a functional regression/recreate test for bug 1671648 https://review.openstack.org/446209 | 01:20 |
| openstack | bug 1671648 in OpenStack Compute (nova) "Instances are not rescheduled after deploy fails" [High,In progress] https://launchpad.net/bugs/1671648 - Assigned to Shunli Zhou (shunliz) | 01:20 |
| *** Apoorva has quit IRC | 01:23 | |
| *** gongysh has quit IRC | 01:25 | |
| *** litao has joined #openstack-nova | 01:27 | |
| *** wangqun has joined #openstack-nova | 01:29 | |
| *** liverpooler has quit IRC | 01:35 | |
| *** mriedem has quit IRC | 01:38 | |
| *** dharinic has joined #openstack-nova | 01:39 | |
| *** eharney has quit IRC | 01:40 | |
| *** unicell has quit IRC | 01:40 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Clarify os-start API description https://review.openstack.org/446264 | 01:41 |
| *** zioproto has quit IRC | 01:41 | |
| *** ducnc has quit IRC | 01:42 | |
| *** ducnc has joined #openstack-nova | 01:42 | |
| *** xinliang has quit IRC | 01:44 | |
| *** raunak has quit IRC | 01:45 | |
| *** jianghuaw has joined #openstack-nova | 01:45 | |
| *** Shunli has joined #openstack-nova | 01:47 | |
| *** wangqun has quit IRC | 01:52 | |
| *** wangqun has joined #openstack-nova | 01:52 | |
| *** hongbin has joined #openstack-nova | 01:53 | |
| *** jianghuaw has quit IRC | 01:53 | |
| *** jianghuaw has joined #openstack-nova | 01:55 | |
| *** xinliang has joined #openstack-nova | 01:56 | |
| *** xinliang has quit IRC | 01:56 | |
| *** xinliang has joined #openstack-nova | 01:56 | |
| *** gouthamr has quit IRC | 01:59 | |
| *** zz_dimtruck is now known as dimtruck | 02:00 | |
| *** raj_sing- has joined #openstack-nova | 02:02 | |
| *** crushil has joined #openstack-nova | 02:02 | |
| *** ssurana has left #openstack-nova | 02:03 | |
| *** alexpilotti has joined #openstack-nova | 02:04 | |
| *** yuntongjin has joined #openstack-nova | 02:04 | |
| *** nic has quit IRC | 02:09 | |
| *** tovin07_ has joined #openstack-nova | 02:09 | |
| *** kaisers_ has quit IRC | 02:10 | |
| *** fragatin_ has joined #openstack-nova | 02:13 | |
| *** marst has joined #openstack-nova | 02:14 | |
| *** edmondsw has joined #openstack-nova | 02:15 | |
| *** fragatina has quit IRC | 02:16 | |
| *** yamahata has quit IRC | 02:16 | |
| *** fragatin_ has quit IRC | 02:17 | |
| *** edmondsw has quit IRC | 02:20 | |
| *** gcb has joined #openstack-nova | 02:20 | |
| *** cfriesen has quit IRC | 02:21 | |
| *** READ10 has quit IRC | 02:21 | |
| *** cfriesen has joined #openstack-nova | 02:21 | |
| *** fragatina has joined #openstack-nova | 02:26 | |
| *** alexpilotti has quit IRC | 02:26 | |
| *** fandi has quit IRC | 02:26 | |
| *** jianghuaw_ has joined #openstack-nova | 02:28 | |
| *** fragatina has quit IRC | 02:30 | |
| *** jichen has joined #openstack-nova | 02:33 | |
| *** ijw has quit IRC | 02:34 | |
| *** ijw_ has joined #openstack-nova | 02:40 | |
| *** gongysh has joined #openstack-nova | 02:43 | |
| *** amotoki has joined #openstack-nova | 02:44 | |
| *** ijw_ has quit IRC | 02:44 | |
| *** namnh has joined #openstack-nova | 02:49 | |
| *** amotoki has quit IRC | 02:49 | |
| *** erlon has joined #openstack-nova | 02:50 | |
| *** amotoki has joined #openstack-nova | 02:54 | |
| *** moshele has joined #openstack-nova | 02:55 | |
| *** cfriesen has quit IRC | 02:57 | |
| *** cfriesen has joined #openstack-nova | 02:57 | |
| *** fragatina has joined #openstack-nova | 03:00 | |
| *** fragatina has quit IRC | 03:01 | |
| *** ducnc1 has joined #openstack-nova | 03:02 | |
| *** fragatina has joined #openstack-nova | 03:02 | |
| *** ducnc has quit IRC | 03:02 | |
| *** ducnc1 is now known as ducnc | 03:02 | |
| *** raunak has joined #openstack-nova | 03:04 | |
| *** sree_ has joined #openstack-nova | 03:12 | |
| *** sree_ is now known as Guest76738 | 03:13 | |
| *** amotoki has quit IRC | 03:19 | |
| *** ijw has joined #openstack-nova | 03:25 | |
| openstackgerrit | Wang Shilong proposed openstack/nova master: Lustre support https://review.openstack.org/446288 | 03:25 |
| *** dave-mcc_ has quit IRC | 03:29 | |
| *** fandi has joined #openstack-nova | 03:33 | |
| *** unicell has joined #openstack-nova | 03:36 | |
| *** dikonoor has joined #openstack-nova | 03:39 | |
| *** unicell has quit IRC | 03:40 | |
| *** xinliang has quit IRC | 03:41 | |
| *** diga has joined #openstack-nova | 03:43 | |
| *** diga has quit IRC | 03:43 | |
| *** zhurong has joined #openstack-nova | 03:45 | |
| *** moshele has quit IRC | 03:46 | |
| *** tbachman_ has joined #openstack-nova | 03:48 | |
| *** udesale has joined #openstack-nova | 03:48 | |
| *** nicolasbock has quit IRC | 03:49 | |
| *** tbachman has quit IRC | 03:49 | |
| *** tbachman_ is now known as tbachman | 03:49 | |
| *** diga has joined #openstack-nova | 03:51 | |
| *** xinliang has joined #openstack-nova | 03:54 | |
| *** xinliang has quit IRC | 03:54 | |
| *** xinliang has joined #openstack-nova | 03:54 | |
| *** raj_sing- has quit IRC | 03:59 | |
| *** pumaranikar has quit IRC | 03:59 | |
| *** NikhilS has joined #openstack-nova | 04:01 | |
| *** amotoki has joined #openstack-nova | 04:08 | |
| *** links has joined #openstack-nova | 04:10 | |
| *** pumaranikar has joined #openstack-nova | 04:11 | |
| *** trinaths has joined #openstack-nova | 04:13 | |
| *** dharinic1 has quit IRC | 04:14 | |
| *** diga has quit IRC | 04:17 | |
| *** adisky_ has joined #openstack-nova | 04:17 | |
| *** unicell has joined #openstack-nova | 04:19 | |
| *** sneti has joined #openstack-nova | 04:20 | |
| *** diga has joined #openstack-nova | 04:20 | |
| *** dharinic_ has joined #openstack-nova | 04:20 | |
| *** unicell has quit IRC | 04:20 | |
| *** diga has quit IRC | 04:21 | |
| *** diga has joined #openstack-nova | 04:21 | |
| *** vgadiraj has joined #openstack-nova | 04:22 | |
| *** vks1 has joined #openstack-nova | 04:22 | |
| *** ssurana has joined #openstack-nova | 04:24 | |
| *** aunnam has joined #openstack-nova | 04:25 | |
| *** hongbin has quit IRC | 04:30 | |
| *** yuntongjin has quit IRC | 04:30 | |
| *** psachin has joined #openstack-nova | 04:30 | |
| *** ssurana has quit IRC | 04:31 | |
| *** dharinic- has joined #openstack-nova | 04:32 | |
| openstackgerrit | Maho Koshiya proposed openstack/nova master: Add confirm resized server functional negative tests https://review.openstack.org/421074 | 04:34 |
| *** dharinic has quit IRC | 04:35 | |
| *** dharinic has joined #openstack-nova | 04:36 | |
| *** dharinic has quit IRC | 04:36 | |
| *** dharinic has joined #openstack-nova | 04:37 | |
| *** gongysh has quit IRC | 04:38 | |
| *** tbachman has quit IRC | 04:38 | |
| *** gongysh has joined #openstack-nova | 04:38 | |
| *** dharinic has quit IRC | 04:39 | |
| *** dharinic has joined #openstack-nova | 04:39 | |
| *** gongysh has quit IRC | 04:42 | |
| *** fragatina has quit IRC | 04:42 | |
| *** gcb has quit IRC | 04:44 | |
| *** kornicameister has joined #openstack-nova | 04:54 | |
| *** dharinic has quit IRC | 04:57 | |
| *** amotoki_ has joined #openstack-nova | 04:57 | |
| *** dharinic has joined #openstack-nova | 04:58 | |
| *** ayogi has joined #openstack-nova | 04:59 | |
| *** amotoki has quit IRC | 05:00 | |
| *** Sukhdev has joined #openstack-nova | 05:03 | |
| *** Kevin_Zheng has quit IRC | 05:03 | |
| *** ratailor has joined #openstack-nova | 05:04 | |
| *** ratailor has quit IRC | 05:07 | |
| *** gongysh has joined #openstack-nova | 05:14 | |
| *** erlon has quit IRC | 05:15 | |
| *** brault_ has joined #openstack-nova | 05:15 | |
| *** ratailor has joined #openstack-nova | 05:15 | |
| *** jith has joined #openstack-nova | 05:16 | |
| *** brault has quit IRC | 05:17 | |
| *** armax has quit IRC | 05:18 | |
| *** dharinic_ has quit IRC | 05:21 | |
| *** zhurong has quit IRC | 05:25 | |
| *** unicell has joined #openstack-nova | 05:26 | |
| *** dharinic1 has joined #openstack-nova | 05:29 | |
| jith | Hi all, I have configured openstack kilo setup in debian jessie with one controller node and two compute nodes. I have used glusterfs for shared storage. I have mounted the shared storage in /var/lib/nova/instances on both nodes. I can do migration of vm’s between nodes.. But live-migration throws following error.. Pls do guide me.. http://pastebin.com/DHPWEkaZ | 05:29 |
| *** Jeffrey4l__ has joined #openstack-nova | 05:30 | |
| *** udesale has quit IRC | 05:32 | |
| *** udesale has joined #openstack-nova | 05:32 | |
| *** dharinic has quit IRC | 05:32 | |
| *** Shunli has quit IRC | 05:33 | |
| *** dharinic has joined #openstack-nova | 05:34 | |
| *** Jeffrey4l_ has quit IRC | 05:34 | |
| *** Shunli has joined #openstack-nova | 05:35 | |
| *** claudiub has joined #openstack-nova | 05:36 | |
| *** dharinic has quit IRC | 05:38 | |
| *** ekuris has joined #openstack-nova | 05:39 | |
| *** kaisers_ has joined #openstack-nova | 05:40 | |
| *** dharinic has joined #openstack-nova | 05:41 | |
| *** guchihiro has joined #openstack-nova | 05:43 | |
| *** bkopilov has quit IRC | 05:44 | |
| *** Sukhdev has quit IRC | 05:47 | |
| *** dharinic has quit IRC | 05:47 | |
| *** edmondsw has joined #openstack-nova | 05:52 | |
| *** dharinic1 has quit IRC | 05:53 | |
| openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova master: Permit Live Migration of Rescued Instances https://review.openstack.org/308198 | 05:56 |
| *** dharinic has joined #openstack-nova | 05:57 | |
| *** edmondsw has quit IRC | 05:57 | |
| *** trinaths has quit IRC | 05:59 | |
| *** bkopilov has joined #openstack-nova | 06:00 | |
| *** yuntongjin has joined #openstack-nova | 06:02 | |
| *** liusheng has quit IRC | 06:05 | |
| *** liusheng has joined #openstack-nova | 06:05 | |
| *** gongysh has quit IRC | 06:09 | |
| *** trinaths has joined #openstack-nova | 06:10 | |
| *** moshele has joined #openstack-nova | 06:16 | |
| *** adisky_ has quit IRC | 06:18 | |
| *** adisky_ has joined #openstack-nova | 06:18 | |
| *** sridharg has joined #openstack-nova | 06:20 | |
| *** sandanar has joined #openstack-nova | 06:22 | |
| *** karthiks_afk is now known as karthiks | 06:24 | |
| *** Shunli has quit IRC | 06:24 | |
| *** Shunli has joined #openstack-nova | 06:25 | |
| *** david-lyle_ has joined #openstack-nova | 06:26 | |
| *** david-lyle has quit IRC | 06:26 | |
| *** Shunli has quit IRC | 06:26 | |
| *** Akhil has joined #openstack-nova | 06:26 | |
| *** raunak has quit IRC | 06:27 | |
| *** raunak has joined #openstack-nova | 06:27 | |
| *** raunak has quit IRC | 06:27 | |
| *** raunak has joined #openstack-nova | 06:28 | |
| *** raunak has quit IRC | 06:28 | |
| *** raunak has joined #openstack-nova | 06:29 | |
| *** raunak has quit IRC | 06:29 | |
| *** raunak has joined #openstack-nova | 06:30 | |
| *** raunak has quit IRC | 06:30 | |
| *** Shunli has joined #openstack-nova | 06:30 | |
| *** adisky_ has quit IRC | 06:30 | |
| *** raunak has joined #openstack-nova | 06:30 | |
| *** raunak has quit IRC | 06:31 | |
| *** Shunli has quit IRC | 06:31 | |
| *** Shunli has joined #openstack-nova | 06:32 | |
| *** Shunli has quit IRC | 06:33 | |
| *** Shunli has joined #openstack-nova | 06:34 | |
| *** moshele has quit IRC | 06:36 | |
| *** lpetrut has joined #openstack-nova | 06:40 | |
| *** zsli_ has joined #openstack-nova | 06:43 | |
| *** zsli_ has quit IRC | 06:46 | |
| *** Shunli has quit IRC | 06:46 | |
| jith | Hi all, I have configured openstack kilo setup in debian jessie with one controller node and two compute nodes. I have used glusterfs for shared storage. I have mounted the shared storage in /var/lib/nova/instances on both nodes. I can do migration of vm’s between nodes.. But live-migration throws following error.. Pls do guide me.. http://pastebin.com/DHPWEkaZ | 06:49 |
| *** belmoreira has joined #openstack-nova | 06:51 | |
| openstackgerrit | lcsong proposed openstack/nova master: Modify some grammatical mistakes. https://review.openstack.org/446321 | 06:55 |
| *** prateek has joined #openstack-nova | 06:56 | |
| *** nkrinner_afk is now known as nkrinner | 06:58 | |
| *** andreas_s has joined #openstack-nova | 07:01 | |
| openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova master: API changes for live migration of rescued instance https://review.openstack.org/328280 | 07:05 |
| *** hferenc has quit IRC | 07:10 | |
| *** alexey_weyl has joined #openstack-nova | 07:13 | |
| alexey_weyl | Hi, I have a small question about the availability zone. | 07:13 |
| openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova master: Port binding based on events during live migration https://review.openstack.org/434870 | 07:14 |
| alexey_weyl | When I change the name or the availability zone itself in the host aggregates I don't receive an update event on the oslo bus. Is nove supposed to send such an event? | 07:14 |
| alexey_weyl | *nove=nova | 07:15 |
| *** fandi has quit IRC | 07:15 | |
| *** amotoki has joined #openstack-nova | 07:18 | |
| *** unicell has quit IRC | 07:19 | |
| *** amotoki_ has quit IRC | 07:21 | |
| *** lpetrut has quit IRC | 07:21 | |
| *** lpetrut has joined #openstack-nova | 07:21 | |
| *** bhagyashris has joined #openstack-nova | 07:25 | |
| *** jaosorior has joined #openstack-nova | 07:26 | |
| *** yamahata has joined #openstack-nova | 07:30 | |
| *** moshele has joined #openstack-nova | 07:31 | |
| *** lpetrut has quit IRC | 07:33 | |
| *** tesseract has joined #openstack-nova | 07:33 | |
| openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova-specs master: Live Migration of Rescued Instances https://review.openstack.org/347161 | 07:36 |
| *** abalutoiu has joined #openstack-nova | 07:37 | |
| *** udesale has quit IRC | 07:40 | |
| *** Jeffrey4l__ has quit IRC | 07:41 | |
| openstackgerrit | Mikhail Feoktistov proposed openstack/nova master: Add is_vz_container function https://review.openstack.org/445947 | 07:41 |
| *** diga has quit IRC | 07:48 | |
| *** haplo37 has quit IRC | 07:48 | |
| *** voelzmo has joined #openstack-nova | 07:49 | |
| *** nirendra has joined #openstack-nova | 07:53 | |
| *** Kevin_Zheng has joined #openstack-nova | 07:53 | |
| openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova-specs master: Live Migration of Rescued Instances https://review.openstack.org/347161 | 07:55 |
| *** gongysh has joined #openstack-nova | 07:56 | |
| *** haplo37 has joined #openstack-nova | 07:57 | |
| nirendra | Nova evacuate on Ocata is failing. Is there any known bug corresponding to this or a workaround | 07:58 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-nova | 08:00 | |
| *** aarefiev_afk is now known as aarefiev | 08:05 | |
| *** adisky_ has joined #openstack-nova | 08:05 | |
| *** diga has joined #openstack-nova | 08:06 | |
| *** xinliang has quit IRC | 08:09 | |
| gmann | alex_xu: hi | 08:10 |
| alex_xu | gmann: hi | 08:10 |
| gmann | alex_xu: show_extensions seems to enforce all extension discovery policy - https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/extension_info.py#L285 | 08:11 |
| gmann | alex_xu: instead of asked extension discovery policy only | 08:11 |
| gmann | self._get_extensions(context) is being called in show and it checks all extension discovery policy | 08:12 |
| alexey_weyl | Hi, I wanted to ask please why there is no id for the availability-zone? | 08:13 |
| alex_xu | gmann: emm...looks like yes | 08:13 |
| alex_xu | gmann: but it works correctly? | 08:14 |
| gmann | alex_xu: because of fatal=False on L226 right? | 08:15 |
| *** moshele has quit IRC | 08:16 | |
| gmann | if anyone disabled other extension on policy discovery then it would not affect of getting asked extension right? | 08:16 |
| alex_xu | gmann: yes | 08:17 |
| gmann | alex_xu: yea, actually it came up on https://review.openstack.org/#/c/431740/4 i am not sure why they cannot use show_extension for all, let me check with author | 08:18 |
| *** ltomasbo|away is now known as ltomasbo | 08:18 | |
| *** moshele has joined #openstack-nova | 08:18 | |
| *** arne_r has joined #openstack-nova | 08:19 | |
| Kevin_Zheng | ping gibi | 08:20 |
| alexey_weyl | can anyone please help me | 08:20 |
| alexey_weyl | I wanted to ask please why there is no id for the availability-zone? | 08:20 |
| alex_xu | gmann: emm....strange | 08:21 |
| *** xinliang has joined #openstack-nova | 08:22 | |
| gmann | alex_xu: yea from user point of view show_extension will be same, its only internal implementation which does extra checks on policies | 08:22 |
| gmann | alexey_weyl: for AZ we have only name | 08:24 |
| alex_xu | gmann: but the discovery policy is something we deprecated? | 08:24 |
| alex_xu | why people still depend on it | 08:24 |
| gmann | alexey_weyl: i mean its always different name so id not needed | 08:24 |
| *** yamahata has quit IRC | 08:25 | |
| gmann | alex_xu: you mean extensions right or discovery policy also we deprecated ? | 08:25 |
| bauzas | good morning Nova | 08:27 |
| alex_xu | gmann: i think the discovery policy | 08:28 |
| alex_xu | gmann: https://review.openstack.org/#/c/427872/ johnthetubaguy propose to remove in it in that spec | 08:28 |
| alex_xu | gmann: not suer whether we have note to say it already deprecated | 08:28 |
| gmann | alex_xu: yea, i was thinking the same. i think we do not have as there was no good way to deprecate policy... | 08:29 |
| openstackgerrit | jichenjc proposed openstack/nova master: Prevent delete cell0 in nova-manage command https://review.openstack.org/433476 | 08:29 |
| alexey_weyl | gmann: that is correct, but if i change the name of the availability zone then vitrage don't know of which zone it was originally | 08:29 |
| alexey_weyl | gmann: meanning, Vitrage don't know what zone was changed, because the previous name doesn't appear. | 08:30 |
| alex_xu | gmann: not like config opt | 08:30 |
| *** pcaruana has joined #openstack-nova | 08:30 | |
| *** diga has quit IRC | 08:32 | |
| *** ekuris has quit IRC | 08:33 | |
| gmann | alex_xu: yea | 08:34 |
| gmann | alexey_weyl: humm, yea name change, changes the AZ completely | 08:34 |
| *** abalutoiu has quit IRC | 08:35 | |
| alexey_weyl | gmann: so how can Vitrage know which availability zone was changed besides storing a cache of all of the different availability zones? | 08:35 |
| *** abalutoiu_ has joined #openstack-nova | 08:35 | |
| gmann | alexey_weyl: but you can get the old AZ from GET server | 08:35 |
| *** namnh_ has joined #openstack-nova | 08:36 | |
| gmann | alexey_weyl: i mean servers booted with AZ will be having the same | 08:36 |
| *** namnh has quit IRC | 08:37 | |
| *** moshele has quit IRC | 08:37 | |
| alexey_weyl | gmann: do you think I can open a bug that we would like the availability zone to have an ID? | 08:39 |
| alexey_weyl | gmann: but still when Vitrage is doing get all availability zones it won't know what AZ was changed. | 08:40 |
| openstackgerrit | Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879 | 08:40 |
| bauzas | FWIW, I'm personnally opposed to accept the AZ changing its name unless there are no left instances in there | 08:40 |
| bauzas | it's a source of confusion | 08:41 |
| gmann | bauzas: yea me too, alexey_weyl is it much required use case? | 08:41 |
| bauzas | creating an AZ is possibly one of the most important things to make sure | 08:41 |
| bauzas | for example, you can default instances to have an AZ by modifying the config to name the default AZ | 08:42 |
| *** dimtruck is now known as zz_dimtruck | 08:42 | |
| gmann | AZ is something managed with name so changing name is all if you do not want to deal with old one | 08:42 |
| bauzas | it's also an end-user UX | 08:42 |
| *** abalutoiu_ has quit IRC | 08:42 | |
| bauzas | meaning that users see the AZs | 08:42 |
| gmann | yea | 08:42 |
| bauzas | when you want to modify a name, then you trample all your users | 08:43 |
| *** jpena|off is now known as jpena | 08:44 | |
| alexey_weyl | bauzas: But shouldn't the AZs have and ID. like for every entity in the Database, that its properties are changed, so we will know whos data was changed by the ID. | 08:45 |
| *** manasm has joined #openstack-nova | 08:45 | |
| alexey_weyl | bauzas: gmann: another question, how can I receive update events of the AZ on oslo messaging bus? | 08:46 |
| bauzas | alexey_weyl: AZs are just an aggregate metadata | 08:46 |
| bauzas | alexey_weyl: it's not a nova specific object, neither persisted like you think | 08:46 |
| *** sree has joined #openstack-nova | 08:47 | |
| *** ekuris has joined #openstack-nova | 08:47 | |
| alexey_weyl | bauzas: Let me explain to you our use case in Vitrage, and lets see what can be done. | 08:47 |
| alexey_weyl | bauzas: In Vitrage we have many entities of different projects in Openstack and physical entities as well that Vitrage monitors and shows to the user and to the system as well so all the alarms and the root cause alarms in the system can be shown. | 08:49 |
| *** nirendra has quit IRC | 08:49 | |
| *** moshele has joined #openstack-nova | 08:49 | |
| alexey_weyl | bauzas: Thus, when a user changes the AZ name, it needs to be reflected in the Vitrage entity graph that has the AZ vertex in it. | 08:50 |
| *** Guest76738 has quit IRC | 08:50 | |
| openstackgerrit | Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879 | 08:50 |
| alexey_weyl | bauzas: When vitrage receive the updated data from nova about the AZs in nova it doesn't know which AZ was changed or deleted or what has happened. | 08:50 |
| alexey_weyl | bauzas: Thus if the AZ will have and ID Vitrage will know what AZ was changed or deleted. | 08:51 |
| bauzas | alexey_weyl: I'm personnally in favor of not accepting to change the AZ if there are already some instances | 08:51 |
| bauzas | I could write a spec | 08:51 |
| *** guchihiro has quit IRC | 08:51 | |
| bauzas | so it would mean a behavioural change, but post that microversion, it shouldn't be possible to change an AZ name | 08:52 |
| *** phuongnh has joined #openstack-nova | 08:52 | |
| alexey_weyl | bauzas: Does it sound as a reasonable use case from your side? | 08:52 |
| *** amoralej|off is now known as amoralej | 08:52 | |
| alexey_weyl | bauzas: didn't quite understoud your last sentence. | 08:53 |
| *** ralonsoh has joined #openstack-nova | 08:54 | |
| bauzas | alexey_weyl: lemme rephrase | 08:54 |
| bauzas | alexey_weyl: since the AZ information is just an aggregate metadata that is user-facing, we already have the ID you need | 08:55 |
| gmann | alexey_weyl: "when a user changes the AZ name" user you mean admin/operator right | 08:55 |
| bauzas | changing that metadata value for that specific key is source of confusion, so I'm not in favor of accepting that | 08:55 |
| alexey_weyl | bauzas: of course | 08:55 |
| alexey_weyl | bauzas: so, you have that id already? can we get from somewhere? | 08:55 |
| bauzas | it's just an aggregate... | 08:56 |
| *** jpena is now known as jpena|off | 08:57 | |
| *** lucas-pto is now known as lucasagomes | 08:58 | |
| *** david-lyle_ has quit IRC | 08:58 | |
| alexey_weyl | bauzas: don't quite understand. what can I do? | 08:59 |
| bauzas | alexey_weyl: do you know https://docs.openstack.org/developer/nova/aggregates.html#availability-zones-azs ? | 09:00 |
| gmann | alexey_weyl: we have id for aggregate not for AZ | 09:00 |
| alexey_weyl | gmann: I am familiar with the ID in the aggregate host. | 09:01 |
| bauzas | alexey_weyl: https://developer.openstack.org/api-ref/compute/?expanded=show-aggregate-details-detail tells you that availability_zone is just a specific metadata key for an aggregate | 09:02 |
| *** iceyao has quit IRC | 09:02 | |
| bauzas | alexey_weyl: we're leaking the DB PK on the API unfortunately, but we added an aggregate UUID since 2.41 | 09:03 |
| alexey_weyl | gmann: I understand. | 09:03 |
| bauzas | https://docs.openstack.org/developer/nova/aggregates.html#availability-zones-azs is explaining that AZs are just a conceptual object that Nova doesn't deal with | 09:04 |
| *** jpena|off is now known as jpena | 09:04 | |
| *** efoley_ has joined #openstack-nova | 09:06 | |
| *** edmondsw has joined #openstack-nova | 09:08 | |
| alexey_weyl | I see | 09:10 |
| alexey_weyl | bauzas: I see | 09:10 |
| alexey_weyl | bauzas: Then I have please another question | 09:10 |
| alexey_weyl | bauzas: I saw that when i change the AZ name or the aggregated host name I dont receive any update event on the oslo messaging bus | 09:11 |
| alexey_weyl | bauzas: how can i receive those updates? | 09:11 |
| bauzas | you mean a notification ? | 09:11 |
| alexey_weyl | correct | 09:11 |
| bauzas | because it's an admin operation | 09:11 |
| bauzas | so you are supposed to know what you do | 09:11 |
| *** lpetrut has joined #openstack-nova | 09:12 | |
| alexey_weyl | bauzas: but vitrage needs to get a notification of such a change. | 09:12 |
| bauzas | I'd rather name it a callback... | 09:12 |
| alexey_weyl | bauzas: ok | 09:12 |
| bauzas | alexey_weyl: well, you can propose to add some new notifications | 09:13 |
| alexey_weyl | bauzas: but still, when even an admin changes something we need to get some update that such a thing was done, how can we get it? | 09:13 |
| alexey_weyl | i see | 09:13 |
| bauzas | gibi: do you know if we need a spec for adding notifications? my guts say yes | 09:13 |
| bauzas | johnthetubaguy: ^ | 09:14 |
| *** gszasz has joined #openstack-nova | 09:16 | |
| gibi | bauzas: so far we went with specless bp for searchlight related additions | 09:17 |
| gibi | Kevin_Zheng: ping, I'm here now | 09:18 |
| bauzas | gibi: good to know | 09:19 |
| Kevin_Zheng | gibi: Ah, Hi, I was going to ask something for notification, but I got it sorted out :) | 09:19 |
| gibi | Kevin_Zheng: OK :) | 09:19 |
| bauzas | gibi: I thought the notification object was somehow needing a consensus before writing the implementation, but okat | 09:19 |
| bauzas | gibi: here, we are talking of sending notifications about aggregate modifications | 09:20 |
| gibi | bauzas, alexey_weyl: yeah, if this is a new non trivial object then I agree that a spec for the data model is a good thing to have | 09:20 |
| bauzas | gibi: do we notify some aggregate-related objects ? | 09:20 |
| openstackgerrit | jichenjc proposed openstack/nova master: Prevent delete cell0 in nova-manage command https://review.openstack.org/433476 | 09:20 |
| bauzas | already I mean | 09:20 |
| gibi | bauzas: yeah, aggregate.create and .delete are both already transformed to versioned format | 09:20 |
| bauzas | so we already emit those ? | 09:21 |
| gibi | bauzas: aggregate.add_host and remove_host are in the pipe | 09:21 |
| gibi | bauzas: yes | 09:21 |
| * gibi looking up the code | 09:21 | |
| bauzas | gibi: okay, that's what alexey_weyl was looking for | 09:21 |
| bauzas | he would like to subscribe on aggregate metadata change | 09:21 |
| gibi | bauzas, alexey_weyl: here is the data modell https://github.com/openstack/nova/blob/master/nova/notifications/objects/aggregate.py | 09:21 |
| bauzas | gibi: perfect | 09:22 |
| gibi | bauzas, alexey_weyl: I think metadata is already part of the notification | 09:22 |
| bauzas | so alexey_weyl just needs to have the aggregate update to emit | 09:22 |
| bauzas | gibi: yup | 09:22 |
| bauzas | gibi: is there already a bp for tracking this ? | 09:23 |
| bauzas | AFAICS, a specless BP should be enough given we already have the notification model | 09:23 |
| alexey_weyl | gibi: bauzas: I am not quite familiar with all the internals of nova. | 09:23 |
| gibi | bauzas: the aggregate ones are part of the notification transformation bp | 09:24 |
| *** karimb has joined #openstack-nova | 09:24 | |
| gibi | bauzas: as legacy had those as well | 09:24 |
| bauzas | gibi: yeah, but that blueprint is not intended to add *more* notifications, just transform the existing, correct? | 09:24 |
| gibi | bauzas: the metadata update part is something that is not transformed yet | 09:24 |
| gibi | bauzas: we have metadata update legacy notification https://github.com/openstack/nova/blob/master/nova/objects/aggregate.py#L442 | 09:24 |
| bauzas | gibi: lemme verify if we emit legacy notifications | 09:25 |
| bauzas | ah-ah ! | 09:25 |
| gibi | bauzas: so we have to just transform that | 09:25 |
| bauzas | gibi: so alexey_weyl would just need to contribute to the transformation bp | 09:25 |
| gibi | bauzas, alexey_weyl: it seems so, yes. | 09:25 |
| *** Jeffrey4l has joined #openstack-nova | 09:25 | |
| bauzas | perfect | 09:25 |
| alexey_weyl | bauzas: gibi: I didn't quite understood all what you said. can you please summerize that for me? what do I need to do? | 09:25 |
| bauzas | alexey_weyl: so, your problem is solved from a nova perspective | 09:26 |
| gibi | alexey_weyl: sure | 09:26 |
| bauzas | alexey_weyl: as gibi said, we already emit notifications on aggregate update | 09:26 |
| bauzas | https://github.com/openstack/nova/blob/master/nova/objects/aggregate.py#L442 | 09:26 |
| bauzas | alexey_weyl: the thing is, we emit the information on a non-versioned dict | 09:26 |
| gibi | alexey_weyl: we are creating a new notification interface for nova, to have a proper interface. these are called versioned notifications. | 09:27 |
| bauzas | alexey_weyl: so gibi is the lead for transforming our legacy payloads into a richer format | 09:27 |
| alexey_weyl | bauzas: gibi: I am listening to all of the notifications from nova and I didn't see any notification from nova when I changed the name of the AZ or the Aggregated host | 09:27 |
| gibi | alexey_weyl: do you listen to notifications topic? | 09:27 |
| alexey_weyl | yes | 09:27 |
| *** links has quit IRC | 09:27 | |
| alexey_weyl | we have added to notification topics a vitrage_notificcation | 09:28 |
| bauzas | https://github.com/openstack/nova/blob/master/nova/objects/aggregate.py#L443 is the notification name | 09:28 |
| gibi | alexey_weyl: hm, you should get aggregate.updatemetadata.end notification | 09:28 |
| alexey_weyl | and thus we can receive all the data | 09:28 |
| openstackgerrit | Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879 | 09:28 |
| alexey_weyl | bauzas: gibi: I will check it again just to make sure. | 09:28 |
| bauzas | in theory, that notification has a payload including the metadata that changed https://github.com/openstack/nova/blob/master/nova/objects/aggregate.py#L423 | 09:28 |
| gibi | alexey_weyl: if you not see that after updating metadata of an aggregate then we have some bug | 09:29 |
| gibi | alexey_weyl: I will try to verify that as well | 09:29 |
| *** tovin07_ has quit IRC | 09:29 | |
| gibi | alexey_weyl: btw, do vitrage plans to move from the legacy nova notifications to the versioned ones? | 09:30 |
| *** jichen has quit IRC | 09:30 | |
| bauzas | gibi: AFAICS, if you do the WSGI action on os-aggregates to set_metadata, you call the Aggregate object method that emits the legacy notification | 09:31 |
| bauzas | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/aggregates.py#L196 | 09:31 |
| bauzas | and https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4490 | 09:32 |
| alexey_weyl | gibi: I have to admit that this is the first time I hear about that, so I will need to check it out. But if this is something that all the projects have then vitrage will have to use such the new notifications | 09:32 |
| *** jaosorior is now known as jaosorior_lunch | 09:32 | |
| *** namnh has joined #openstack-nova | 09:32 | |
| *** dgonzalez_ has joined #openstack-nova | 09:33 | |
| *** tpatzig_ has joined #openstack-nova | 09:33 | |
| *** seife_ has joined #openstack-nova | 09:33 | |
| *** david_1 has joined #openstack-nova | 09:33 | |
| gibi | alexey_weyl: we are in the process to define and implement the new interface so it is the good time to chime in from consumer side | 09:33 |
| gibi | bauzas: that is my understanding of the code as well | 09:33 |
| *** namnh_ has quit IRC | 09:34 | |
| gibi | bauzas: but if alexey_weyl does not see the notification then there is something we don't see | 09:34 |
| alexey_weyl | gibi: bauzas: I have checked it again, and I didn't receive any notification about AZ and Aggregated host when changed their names | 09:34 |
| *** seife_ has quit IRC | 09:35 | |
| *** dgonzalez_ has quit IRC | 09:35 | |
| *** tpatzig_ has quit IRC | 09:35 | |
| *** david_1 has quit IRC | 09:35 | |
| gibi | alexey_weyl: do you get aggregate.create when you create a new aggregate? | 09:35 |
| bauzas | alexey_weyl: and what if you change a metadata key that is *not* availability_zone ? | 09:35 |
| * gibi is digging up a running test system to try as well, but it will be Mitaka based | 09:35 | |
| alexey_weyl | gibi: bauzas: When i add, delete, update instances I receive all the notification. | 09:36 |
| *** yingjun has quit IRC | 09:37 | |
| alexey_weyl | gibi: bauzas: maybe it is related to the fact that I perform the changes in the horizon admin tab? | 09:37 |
| alexey_weyl | gibi: bauzas: I will try to create a new aggregated host via cli | 09:37 |
| gibi | alexey_weyl: I think horizon should not be the problem | 09:37 |
| *** derekh has joined #openstack-nova | 09:38 | |
| alexey_weyl | gibi: bauzas: Ok, have created a new aggregated using the cli and i see it in the horizon but I didn't receive any upate | 09:39 |
| gibi | alexey_weyl: strange | 09:40 |
| gibi | alexey_weyl: did you check the notification from vitrage logs? doesn't vitrage filter on event_type? | 09:41 |
| * gibi is still in the process to try to reproduce the issue | 09:42 | |
| alexey_weyl | I have checked, I have put a print before the filtering | 09:42 |
| alexey_weyl | gibi: I have checked, I have put a print before the filtering | 09:42 |
| *** links has joined #openstack-nova | 09:44 | |
| alexey_weyl | gibi: bauzas: | 09:44 |
| alexey_weyl | gibi: bauzas: I might know the problem | 09:44 |
| gibi | alexey_weyl: I set the notification driver to log in nova.conf and created an aggregate | 09:44 |
| gibi | alexey_weyl: and I got the log: | 09:44 |
| gibi | alexey_weyl: /var/log/nova/nova-api.log:2017-03-16T10:43:00.616385+01:00 cic-3 nova-api[22020]: 2017-03-16 10:43:00.615 22020 INFO oslo.messaging.notification.aggregate.create.start [req-0b7f199d-8d06-4d2c-bcd6-fc5a8d03b623 ad400a8540f744b5b7041a356d51b6f3 a21d93baeb5e42c8a9d1da782279e309 - - -] {"event_type": "aggregate.create.start", "timestamp": "2017-03-16 09:43:00.615696", "payload": {"name": "gibi-aggregate"}, "priority": "INF | 09:44 |
| gibi | alexey_weyl: but this is Mitaka version | 09:45 |
| alexey_weyl | gibi: bauzas: The productive system I work on is liberty. This might be the problem, right? | 09:45 |
| gibi | alexey_weyl: the legacy aggregate.create works for me on Mitaka | 09:45 |
| gibi | alexey_weyl: let me check Liberty... | 09:45 |
| *** ociuhandu has quit IRC | 09:46 | |
| bauzas | FWIW https://blueprints.launchpad.net/nova/+spec/az-block-name-update | 09:46 |
| gibi | alexey_weyl: Liberty code also has the notification https://github.com/openstack/nova/blob/liberty-eol/nova/objects/aggregate.py#L126 | 09:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add PCIWeigher https://review.openstack.org/379524 | 09:47 |
| alexey_weyl | gibi: bauzas: ok, now it is really weird. | 09:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add PCIWeigher https://review.openstack.org/379524 | 09:48 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Prefer non-PCI host nodes for non-PCI instances https://review.openstack.org/379625 | 09:48 |
| gibi | alexey_weyl: is there a way for you to reproduce the problem closer to master ? | 09:48 |
| alexey_weyl | gibi: bauzas: Yes, I am going to work on pike now, and i will check it there. | 09:49 |
| alexey_weyl | gibi: bauzas: will update you later on. | 09:49 |
| alexey_weyl | gibi: bauzas: anyway, thank you very much guys!!!! :)\ | 09:49 |
| gibi | alexey_weyl: just ping me when you have some test results | 09:49 |
| gibi | bauzas: you proposal makes sense to me. less thing to change means less confusion. So changing the AZ name would be a lot harderd after it. | 09:50 |
| gibi | bauzas: deleting aggregate and creating a new one | 09:50 |
| alexey_weyl | gibi: no problem. thanks :) | 09:51 |
| bauzas | gibi: in theory, once you accept an instance to be created on that AZ, then your contract becomes harder | 09:52 |
| bauzas | because some people trust you for caring their instances | 09:52 |
| gibi | bauzas: true. | 09:52 |
| gibi | bauzas: what if just the az name needs to be changed but not the meanin of it? I mean the renamed az still means the same failure domain. | 09:53 |
| bauzas | gibi: what I need to understand is why operators need to update their AZ names | 09:54 |
| bauzas | it's just a string | 09:54 |
| gibi | bauzas: renaming the room in the building? :) | 09:54 |
| bauzas | so they would expose their topology semantically? | 09:54 |
| jith | Hi all, I have configured openstack kilo setup in debian jessie with one controller node and two compute nodes. I have used glusterfs for shared storage. I have mounted the shared storage in /var/lib/nova/instances on both nodes. I can do migration of vms between nodes.. But live-migration throws following error.. Pls do guide me.. http://pastebin.com/DHPWEkaZ | 09:55 |
| gibi | bauzas: you are right. My point is not really valid. | 09:55 |
| gibi | bauzas: anyhow I'm just playing devils advocate here. | 09:55 |
| bauzas | jith: I'm sorry, this channel is focused on development questions | 09:55 |
| bauzas | gibi: nah it's fine, I could be wrong | 09:56 |
| gibi | bauzas: az name should not reflect real topology. It should be failure domain A , B etc | 09:56 |
| bauzas | gibi: the only problem I see with preventing the AZ rename is that it's becoming hard to migrate instances | 09:56 |
| gibi | bauzas: so your point seems OK to me | 09:56 |
| bauzas | say I made a mistake and I want to fix that | 09:57 |
| gibi | bauzas: does admin can migrate instance by ignoring AZ? | 09:57 |
| bauzas | migi: he could force the migration | 09:57 |
| bauzas | oops | 09:57 |
| bauzas | s/migi/gibi | 09:57 |
| migi | :) | 09:57 |
| gibi | bauzas: force migration could work for me | 09:57 |
| gibi | migi, bauzas: :) | 09:58 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Validate huge pages https://review.openstack.org/399653 | 09:58 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: create functional test base class https://review.openstack.org/407055 | 09:58 |
| bauzas | anyway, migrating from one AZ to another is a big deal | 09:58 |
| sfinucan | jaypipes: Fixed that hugepages functional test https://review.openstack.org/#/c/399653/ | 09:59 |
| bauzas | gibi: since one host can't be in two AZs at the same time, I'd suggest to place the host on a specific aggregate that is not AZ-facing | 10:00 |
| *** ltomasbo is now known as ltomasbo|away | 10:00 | |
| bauzas | gibi: and then move the host to the new aggregate with the right AZ name | 10:00 |
| *** trinaths has left #openstack-nova | 10:00 | |
| bauzas | rather than issuing a live-migration | 10:00 |
| *** ralonsoh_ has joined #openstack-nova | 10:01 | |
| *** ociuhandu has joined #openstack-nova | 10:01 | |
| bauzas | I should test that in devstack | 10:01 |
| *** rfolco has joined #openstack-nova | 10:01 | |
| *** rfolco has quit IRC | 10:01 | |
| *** efoley__ has joined #openstack-nova | 10:02 | |
| *** iceyao has joined #openstack-nova | 10:03 | |
| mdbooth | lyarwood (or anybody else): Do you know what would consume volume.detach instance notifications? | 10:03 |
| *** nkrinner is now known as nkrinner_afk | 10:03 | |
| *** ralonsoh has quit IRC | 10:04 | |
| gibi | bauzas: ohh, I see | 10:04 |
| * gibi is running for lunch | 10:05 | |
| lyarwood | mdbooth: no sorry | 10:05 |
| lyarwood | mdbooth: is this regarding the ordering change? | 10:06 |
| mdbooth | lyarwood: Yeah | 10:06 |
| mdbooth | Also, the notification should probably live with the detach | 10:06 |
| *** efoley_ has quit IRC | 10:06 | |
| *** wangqun has quit IRC | 10:07 | |
| *** phuongnh has quit IRC | 10:07 | |
| *** iceyao has quit IRC | 10:09 | |
| *** namnh has quit IRC | 10:09 | |
| *** sdague has joined #openstack-nova | 10:11 | |
| mdbooth | lyarwood: Yeah, reading the contract on those notifications, notifying after makes more sense to me. | 10:11 |
| *** ltomasbo|away is now known as ltomasbo | 10:11 | |
| *** hshiina has quit IRC | 10:14 | |
| sfinucan | jlvillal: https://review.openstack.org/#/c/445622/1 | 10:14 |
| *** cdent has joined #openstack-nova | 10:15 | |
| openstackgerrit | Mikhail Feoktistov proposed openstack/nova master: Add is_vz_container function https://review.openstack.org/445947 | 10:15 |
| *** kevinz has quit IRC | 10:20 | |
| *** nicolasbock has joined #openstack-nova | 10:20 | |
| sfinucan | sean-k-mooney, jaypipes: Shouldn't this be in os-vif? https://review.openstack.org/#/c/441183/ | 10:21 |
| *** Jeffrey4l has quit IRC | 10:23 | |
| *** edmondsw has quit IRC | 10:25 | |
| mdbooth | We've had references to Cinder attachment_id in Nova since Mitaka | 10:33 |
| mdbooth | lyarwood: ^^^ | 10:33 |
| jith | Hi all, I have configured openstack kilo setup in debian jessie with one controller node and two compute nodes. I have used glusterfs for shared storage. I have mounted the shared storage in /var/lib/nova/instances on both nodes. I can do migration of vms between nodes.. But live-migration throws following error.. Pls do guide me.. http://pastebin.com/DHPWEkaZ | 10:34 |
| mdbooth | lyarwood: What am I missing there? | 10:34 |
| lyarwood | mdbooth: yeah cinderv2 attachment_id's | 10:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Replace obsolete vanity openstack.org URLs https://review.openstack.org/443266 | 10:34 |
| mdbooth | How are they different? | 10:34 |
| lyarwood | mdbooth: I think they are only optional when detaching | 10:35 |
| mdbooth | Are the underlying objects the same? | 10:35 |
| lyarwood | mdbooth: on the Cinder side? I have no idea. | 10:35 |
| lyarwood | mdbooth: but in the API yes | 10:35 |
| lyarwood | mdbooth: that's all we care about right? | 10:35 |
| mdbooth | So a cinderv2 attachment_id is the same as a cinderv3 attachment_id | 10:35 |
| mdbooth | i.e. the uuid values are the same | 10:35 |
| lyarwood | mdbooth: that I don't know | 10:36 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Replace obsolete vanity openstack.org URLs https://review.openstack.org/443266 | 10:36 |
| gibi | mdbooth, lyarwood: I think we you want to change when the volume.detach is sent then it would make sense to send a detach.start and a detach.end | 10:36 |
| mdbooth | gibi: Out of scope for this change, tbh. | 10:36 |
| mdbooth | Waaaaaaaaay out of scope | 10:36 |
| *** gongysh has quit IRC | 10:36 | |
| gibi | mdbooth: then let's not change when to send the volume.detach :) | 10:36 |
| lyarwood | gibi: but I can follow up with that after | 10:36 |
| gibi | lyarwood: I think it is not super important | 10:37 |
| mdbooth | lyarwood gibi: I don't know enough about the practical uses of notifications to know how important that order change is, but if there's any doubt we should leave it alone | 10:38 |
| mdbooth | It's not at all relevant to lyarwood's change | 10:38 |
| mdbooth | lyarwood: So that means moving the notification with the detach call | 10:39 |
| lyarwood | mdbooth: sure, then the notification comes before bdm.destroy | 10:39 |
| mdbooth | lyarwood: I don't see a problem with that. | 10:40 |
| gibi | mdbooth, lyarwood: OK | 10:40 |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add policy-remove-scope-checks spec https://review.openstack.org/433037 | 10:41 |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872 | 10:41 |
| mdbooth | lyarwood: As I see it, it's more about the behaviour when detach fails, tbh | 10:41 |
| lyarwood | mdbooth: true | 10:41 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposed block accepting AZ renames https://review.openstack.org/446446 | 10:42 |
| lyarwood | mdbooth: I need to run for a dentist appointment, back in ~90mins or so. | 10:42 |
| lyarwood | mdbooth: are you around for the cinder/nova meeting later today? | 10:42 |
| mdbooth | lyarwood: kk | 10:42 |
| mdbooth | lyarwood: Possibly. Talk later. | 10:42 |
| lyarwood | mdbooth: ack, thanks | 10:42 |
| *** Jeffrey4l has joined #openstack-nova | 10:46 | |
| *** alexpilotti has joined #openstack-nova | 10:47 | |
| *** jaosorior_lunch is now known as jaosorior | 10:51 | |
| *** carthaca_2 has quit IRC | 10:52 | |
| *** sapcc-bot4 has quit IRC | 10:52 | |
| *** databus23_2 has quit IRC | 10:52 | |
| *** david_1 has joined #openstack-nova | 10:52 | |
| *** sapcc-bot has joined #openstack-nova | 10:52 | |
| *** databus23_ has joined #openstack-nova | 10:52 | |
| *** carthaca_ has joined #openstack-nova | 10:52 | |
| *** tpatzig_ has joined #openstack-nova | 10:52 | |
| *** seife_ has joined #openstack-nova | 10:52 | |
| *** sapcc-bot has quit IRC | 10:54 | |
| *** databus23_ has quit IRC | 10:54 | |
| *** carthaca_ has quit IRC | 10:54 | |
| *** seife_ has quit IRC | 10:54 | |
| *** tpatzig_ has quit IRC | 10:54 | |
| *** david_1 has quit IRC | 10:54 | |
| *** sapcc-bot has joined #openstack-nova | 10:54 | |
| *** databus23_ has joined #openstack-nova | 10:54 | |
| *** carthaca_ has joined #openstack-nova | 10:54 | |
| -openstackstatus- NOTICE: paste.openstack.org is down, due to connectivity issues with backend database. support ticket has been created. | 10:59 | |
| *** ChanServ changes topic to "paste.openstack.org is down, due to connectivity issues with backend database. support ticket has been created." | 10:59 | |
| *** iceyao has joined #openstack-nova | 11:00 | |
| *** kaisers__ has joined #openstack-nova | 11:01 | |
| openstackgerrit | Sean Dague proposed openstack/nova master: remove hacking rule that enforces log translation https://review.openstack.org/446452 | 11:03 |
| *** kaisers_ has quit IRC | 11:04 | |
| *** iceyao has quit IRC | 11:04 | |
| *** jdurgin has quit IRC | 11:18 | |
| openstackgerrit | Jianghua Wang proposed openstack/nova master: XenAPI: device tagging https://review.openstack.org/333781 | 11:22 |
| openstackgerrit | Bing Li proposed openstack/nova master: Add server-action-removefloatingip.json file and update servers-actions.inc https://review.openstack.org/446471 | 11:23 |
| *** bvanhav_ has joined #openstack-nova | 11:24 | |
| *** jdurgin has joined #openstack-nova | 11:27 | |
| gibi | dansmith: Hi! I left some suggestion in https://review.openstack.org/#/c/445697 | 11:30 |
| *** nkrinner_afk is now known as nkrinner | 11:31 | |
| *** ociuhandu has quit IRC | 11:33 | |
| *** sree has quit IRC | 11:36 | |
| *** ralonsoh__ has joined #openstack-nova | 11:36 | |
| *** ralonsoh__ is now known as ralonsoh | 11:37 | |
| *** ralonsoh_ has quit IRC | 11:40 | |
| gibi | alexey_weyl: I checked the aggregate notification in a new devstack from master. | 11:43 |
| gibi | alexey_weyl: It works for me. See the logs in http://paste.openstack.org/show/602943/ | 11:44 |
| *** ociuhandu has joined #openstack-nova | 11:44 | |
| *** ChanServ changes topic to "This channel is for Nova development. For support of Nova deployments, please use #openstack. Please see: https://wiki.openstack.org/wiki/Nova/Ocata_Release_Schedule" | 11:46 | |
| -openstackstatus- NOTICE: paste.openstack.org service is back up - turns out it was a networking issue, not a database issue. yay networks! | 11:46 | |
| *** lpetrut has quit IRC | 11:48 | |
| *** raj_singh has quit IRC | 11:49 | |
| *** vks1 has quit IRC | 11:49 | |
| *** dave-mccowan has joined #openstack-nova | 11:53 | |
| *** aysyd has joined #openstack-nova | 11:55 | |
| *** alexey_weyl has quit IRC | 11:56 | |
| *** efoley_ has joined #openstack-nova | 12:03 | |
| *** alexey_weyl has joined #openstack-nova | 12:03 | |
| alexey_weyl | gibi: Hi | 12:06 |
| *** efoley__ has quit IRC | 12:06 | |
| alexey_weyl | gibi: I have checked it also, and i saw that notifications work for vitrage with master. but still notifications don't work on liberty (notifications about the aggregated hosts) | 12:07 |
| *** namnh has joined #openstack-nova | 12:08 | |
| *** manasm has quit IRC | 12:09 | |
| gibi | alexey_weyl: I don't have access to Liberty test node right now. Also Liberty is already reached end of support | 12:10 |
| *** amoralej is now known as amoralej|lunch | 12:10 | |
| gibi | alexey_weyl: so even if it is buggy we cannot fix it on Liberty | 12:10 |
| *** jpena is now known as jpena|lunch | 12:11 | |
| alexey_weyl | gibi: ok, i see that, and it is ok. but I have some other issue | 12:14 |
| alexey_weyl | gibi: The thing is that for example in devstack when you create it, you don't have an "aggregated host" but we have AZs. | 12:15 |
| *** namnh has quit IRC | 12:15 | |
| *** gouthamr has joined #openstack-nova | 12:16 | |
| alexey_weyl | gibi: After our talk before I thought that I could use only the Aggregated hosts to get the needed data, but it seems that I would need to get the AZs by calling the availability-zone list and thus because it doesn't has id we have a problem | 12:16 |
| mdbooth | lyarwood: You back? | 12:17 |
| *** lpetrut has joined #openstack-nova | 12:17 | |
| *** jianghuaw-m has joined #openstack-nova | 12:18 | |
| robcresswell | Quick question; someone's added a patch to Horizon that hides the Soft Reboot button if the Instance state is anything other than Active; is this correct? Had a look at the API docs but they only seem to show how to form the request, not any of the conditions around its usage. | 12:19 |
| mdbooth | robcresswell: soft reboot doesn't make sense for an instance which isn't running | 12:19 |
| *** mlakat has quit IRC | 12:20 | |
| robcresswell | mdbooth: Yeah, I'm not clued in on every specific status nova supports | 12:20 |
| *** jianghuaw-m has quit IRC | 12:20 | |
| * mdbooth tries to think of non-active states where the instance is still running | 12:20 | |
| mdbooth | error, perhaps | 12:20 |
| *** ratailor has quit IRC | 12:21 | |
| mdbooth | robcresswell: I mean, regardless of what the api allows, from a UI pov that restriction makes sense to me | 12:21 |
| robcresswell | mdbooth: Cool. Good enough for me. | 12:22 |
| *** rfolco has joined #openstack-nova | 12:22 | |
| robcresswell | Thanks | 12:22 |
| * mdbooth hopes what when you mouse-over it, it says: 'Soft reboot is only available when instance is running' | 12:23 | |
| * mdbooth hates disabled options which don't explain why they're disabled | 12:23 | |
| alexey_weyl | gibi: The reason that I would need to call the availability-zone list as well is because sometimes we don't have the aggregated hosts but we have AZs | 12:24 |
| gibi | alexey_weyl: if you have an AZ then you automatically have a host aggregate behind it | 12:27 |
| gibi | alexey_weyl: the availability-zone list just iterate throught the host aggregates to see if there is a metadata key on the aggregate with name availability_zone | 12:28 |
| *** karimb has quit IRC | 12:28 | |
| *** edmondsw has joined #openstack-nova | 12:29 | |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add policy-remove-scope-checks spec https://review.openstack.org/433037 | 12:29 |
| *** esberglu has joined #openstack-nova | 12:29 | |
| gibi | alexey_weyl: what would be different for you if there would be uuid besids AZ name? | 12:31 |
| *** liverpooler has joined #openstack-nova | 12:32 | |
| lyarwood | mdbooth: back now, going to attempt to eat lunch and then catch up on your review comments | 12:32 |
| *** ayogi has quit IRC | 12:32 | |
| *** udesale has joined #openstack-nova | 12:37 | |
| openstackgerrit | Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879 | 12:40 |
| *** NikhilS has quit IRC | 12:40 | |
| alexey_weyl | gibi: on my devstack which is a devstack that works with master branch, I have AZs but don't have any aggregated hosts | 12:40 |
| mdbooth | lyarwood: You might want to start with the detach refactor, btw, because I think that requires a rethink which may affect other patches. | 12:41 |
| *** efried has joined #openstack-nova | 12:42 | |
| gibi | alexey_weyl: but you have an empty host aggregate I assume | 12:42 |
| lyarwood | mdbooth: kk looking now | 12:44 |
| gibi | alexey_weyl: actually you can have more than one empty host aggregate connected to the same AZ | 12:44 |
| *** liverpooler has quit IRC | 12:44 | |
| *** liverpooler has joined #openstack-nova | 12:44 | |
| gibi | alexey_weyl: and those aggregates have uuid | 12:45 |
| gibi | alexey_weyl: if you need something unique | 12:45 |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872 | 12:47 |
| *** voelzmo has quit IRC | 12:51 | |
| *** kevinz has joined #openstack-nova | 12:51 | |
| *** voelzmo has joined #openstack-nova | 12:51 | |
| alexey_weyl | gibi: I don't quite understand. I ran on my devstack the command "nova aggregate-list" and received that it is empty and then I ran "nova availability-zone-list" and I saw to AZs | 12:51 |
| *** voelzmo has quit IRC | 12:52 | |
| alexey_weyl | gibi: what is this empty aggregated host? | 12:52 |
| *** voelzmo has joined #openstack-nova | 12:52 | |
| *** voelzmo has quit IRC | 12:53 | |
| *** lucasagomes is now known as lucas-hungry | 12:54 | |
| *** liangy has joined #openstack-nova | 12:55 | |
| *** liangy has quit IRC | 12:56 | |
| *** kylek3h has joined #openstack-nova | 12:56 | |
| *** burt has joined #openstack-nova | 12:57 | |
| *** tbachman has joined #openstack-nova | 12:58 | |
| lyarwood | hmm does any have an idea how I can reference a line in the original file of a gerrit diff? | 12:59 |
| *** manasm has joined #openstack-nova | 12:59 | |
| *** mriedem has joined #openstack-nova | 12:59 | |
| rfolco | cdent, ping | 13:00 |
| cdent | hi rfolco | 13:00 |
| *** catintheroof has joined #openstack-nova | 13:00 | |
| *** sree has joined #openstack-nova | 13:00 | |
| rfolco | cdent, cannot find how to test concurrent update/delete. How to emulate that ? | 13:00 |
| cdent | rfolco: the idea you have before of trying to do a DELETE ../inventories with an incorrect resource_provider_generation should be good enough | 13:01 |
| *** iceyao has joined #openstack-nova | 13:01 | |
| rfolco | cdent, hmm I thought the generation would fall into a different code path, not that exception. | 13:02 |
| cdent | the generation being wrong is the only thing that can cause ConcurrentUpdateDetected to raise | 13:02 |
| jaypipes | sfinucan: well, there will need to be *some* os-vif complementary part to that, yes. | 13:02 |
| jaypipes | macsz: btw, you're comment on that "i find the lack of commit message disturbing." was most excellent ;) | 13:03 |
| rfolco | cdent, also, delete does not produce json, only post/put do. So not sure about the other suggestion on checking delete request/response with 409 | 13:03 |
| gibi | alexey_weyl: run nova aggregate-list | 13:03 |
| *** jianghuaw__ has joined #openstack-nova | 13:03 | |
| cdent | rfolco: the error's response body will be json if you send an accept header of application/json | 13:04 |
| rfolco | cdent, accept header mesans the decorator ? | 13:04 |
| *** jpena|lunch is now known as jpena | 13:04 | |
| cdent | so even though a success response is empty an error response will not | 13:04 |
| cdent | i mean in the gabbi test: request_headers:\naccept: application/json\n | 13:04 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: use context mgr in instance.delete https://review.openstack.org/443764 | 13:05 |
| rfolco | cdent, hmmm, ok, thanks | 13:05 |
| cdent | rfolco: that "hmmm" makes it sound you're not quite sure what I mean or ... ? | 13:06 |
| *** hshiina has joined #openstack-nova | 13:06 | |
| *** dharinic_ has joined #openstack-nova | 13:06 | |
| rfolco | cdent, haha hmm means, ok in theory I got it, lets see in practice. | 13:06 |
| *** iceyao has quit IRC | 13:06 | |
| cdent | ah, okay, cool. let me know if you need more info as you go along | 13:06 |
| rfolco | cdent, I'll bother you again if I get stuck | 13:07 |
| gibi | alexey_weyl: the AZ is not a real entity in nova. The host aggregate is the real one and if you have an aggregate with a special metadata 'availability_zone' then the hosts in that aggregate become part of an AZ | 13:08 |
| alexey_weyl | gibi: I have ran the nova aggregate-list and I returned empty | 13:08 |
| alexey_weyl | gibi: although I have 2 AZs | 13:08 |
| *** tbachman_ has joined #openstack-nova | 13:09 | |
| *** bvanhav__ has joined #openstack-nova | 13:10 | |
| *** liangy has joined #openstack-nova | 13:11 | |
| *** bvanhav_ has quit IRC | 13:11 | |
| *** tbachman has quit IRC | 13:12 | |
| *** tbachman_ is now known as tbachman | 13:12 | |
| *** kevinz has quit IRC | 13:14 | |
| *** kevinz has joined #openstack-nova | 13:14 | |
| *** udesale has quit IRC | 13:14 | |
| *** vladikr has joined #openstack-nova | 13:15 | |
| *** mdrabe has joined #openstack-nova | 13:16 | |
| *** mriedem has quit IRC | 13:17 | |
| *** mriedem has joined #openstack-nova | 13:20 | |
| gibi | alexey_weyl: nova and internal? | 13:24 |
| *** crushil has quit IRC | 13:24 | |
| gibi | alexey_weyl: nova is the default AZ. host not part of any AZ will belong there | 13:25 |
| gibi | alexey_weyl: you can ignore the internal, as far as I understand internal never contains compute hosts | 13:26 |
| mriedem | has anyone seen this test fail before? http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/479c7bf/console.html#_2017-03-16_01_31_33_688563 | 13:26 |
| gibi | alexey_weyl: I hope somebody correct me as I reached the limit of my knowledge | 13:26 |
| *** mlavalle has joined #openstack-nova | 13:27 | |
| *** ociuhandu has quit IRC | 13:27 | |
| *** ociuhandu has joined #openstack-nova | 13:28 | |
| mriedem | bauzas: can you take a look at the ocata regression fix here https://review.openstack.org/#/c/444106/ and the functional test below it? the test is the one i was talking about yesterday. | 13:29 |
| mriedem | we need to get those in because they are hurting the ironic ci | 13:29 |
| *** karimb has joined #openstack-nova | 13:30 | |
| *** tblakes has joined #openstack-nova | 13:31 | |
| *** NikhilS has joined #openstack-nova | 13:31 | |
| *** eharney has joined #openstack-nova | 13:33 | |
| *** Jeffrey4l has quit IRC | 13:34 | |
| *** tbachman has quit IRC | 13:36 | |
| *** tbachman has joined #openstack-nova | 13:36 | |
| *** jaosorior has quit IRC | 13:37 | |
| *** jaosorior has joined #openstack-nova | 13:37 | |
| *** jaosorior has quit IRC | 13:37 | |
| *** jaosorior has joined #openstack-nova | 13:38 | |
| *** raj_sing- has joined #openstack-nova | 13:40 | |
| *** eharney has quit IRC | 13:40 | |
| *** voelzmo has joined #openstack-nova | 13:41 | |
| openstackgerrit | Rafael Folco proposed openstack/nova master: DELETE all inventory for a resource provider https://review.openstack.org/416669 | 13:45 |
| *** Jeffrey4l has joined #openstack-nova | 13:47 | |
| *** iceyao has joined #openstack-nova | 13:47 | |
| *** crushil has joined #openstack-nova | 13:48 | |
| *** felipemonteiro_ has joined #openstack-nova | 13:50 | |
| *** alexpilotti has quit IRC | 13:50 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: doc: configurable versioned notifications topics https://review.openstack.org/446523 | 13:50 |
| jlvillal | sfinucan, Thanks for the heads up on the flake8 patch. I commented. | 13:50 |
| *** voelzmo has quit IRC | 13:51 | |
| *** iceyao has quit IRC | 13:52 | |
| *** baoli has joined #openstack-nova | 13:52 | |
| *** links has quit IRC | 13:52 | |
| mriedem | lyarwood: mdbooth: i'm debugging a failure in the gate http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/479c7bf/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-03-16_01_01_32_897 where detach of a volume to an unshelved server fails, but it actually looks like when we did the detach on the guest, we got an error from libvirt that we usually handle (device not found) and raise that back up to | 13:52 |
| mriedem | compute manager to basically ignore saying it's not actually attached, | 13:52 |
| mriedem | but then it looks like we try to detach again, and get a different unhandled i/o error from libvirt | 13:53 |
| *** rfolco_ has joined #openstack-nova | 13:53 | |
| lyarwood | looking | 13:53 |
| *** eharney has joined #openstack-nova | 13:53 | |
| mriedem | the test steps are: create a server and wait for it to be active, create a volume and wait for it to be available, shelve the server and wait for it to be offloaded, attach the volume, unshelve the server and wait for it to be active, then verify the volume is attached, and then start the teardown (detach volume and wait for it to be available, then delete the volume, then delete the server) | 13:54 |
| *** rfolco has quit IRC | 13:54 | |
| lyarwood | and this is failing during the teardown right? | 13:55 |
| openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.volume_attach notification https://review.openstack.org/401992 | 13:55 |
| mriedem | one thing i noticed is that the test specifies /dev/vdb when attaching the volume, and since it's the libvirt driver it ignores that and the volume is actually mounted at /dev/vdc | 13:55 |
| mriedem | lyarwood: yeah | 13:55 |
| mriedem | the volume detach fails | 13:55 |
| mriedem | so tempest times out waiting for the volume to go from in-use to available | 13:55 |
| mriedem | we get here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L408 | 13:56 |
| openstackgerrit | Sean Dague proposed openstack/nova master: remove hacking rule that enforces log translation https://review.openstack.org/446452 | 13:56 |
| *** yingjun has joined #openstack-nova | 13:56 | |
| openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.volume_detach notification https://review.openstack.org/408676 | 13:57 |
| *** amoralej|lunch is now known as amoralej | 13:57 | |
| mriedem | what i don't get is it looks like we hit the "no target device" error from libvirt, which we handle and reraise as DeviceNotFound, but then it looks like we try to detach again and get "libvirtError: End of file while reading data: Input/output error" which we don't handle | 13:59 |
| mriedem | ok so the device name is different also because when attaching a volume to a shelved offloaded instance, we don't set the device on the bdm https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3678 | 14:00 |
| mriedem | the tempest test should probably not even be specifying the device name | 14:01 |
| lyarwood | yeah sorry I was just looking for the device xml during the detach to confirm what the target device was | 14:02 |
| *** sree has quit IRC | 14:03 | |
| *** cfriesen has quit IRC | 14:07 | |
| *** yuntongjin has quit IRC | 14:07 | |
| *** cfriesen has joined #openstack-nova | 14:07 | |
| *** lucas-hungry is now known as lucasagomes | 14:07 | |
| *** zz_dimtruck is now known as dimtruck | 14:07 | |
| *** amotoki has quit IRC | 14:08 | |
| *** hshiina has quit IRC | 14:08 | |
| mriedem | so in the attach volume to shelved offloaded server case, we actually call os-attach in the cinder api to mark the volume as in-use, since we can't do it from the compute (since the instance isn't on a host), | 14:12 |
| openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.volume_detach notification https://review.openstack.org/408676 | 14:12 |
| mriedem | i'm wondering how that might mess up the new create attachment flows, i guess we can just update the volume attachment later when we're unshelving on the host and have the connection info, we update the vol attachment in cinder | 14:12 |
| alexey_weyl | gibi: so the nova AZ has no aggregated host that the nova AZ is part of. | 14:12 |
| alexey_weyl | gibi: is it correct to devstack only? or also to openstack in production? | 14:13 |
| lyarwood | mriedem: right, update with the connector and get the connection_info back during unshelve | 14:13 |
| gibi | alexey_weyl: the nova AZ contains by default every host that does not belong to any other AZ. | 14:14 |
| gibi | alexey_weyl: it is not devstack specific but the name of the default AZ is configurable in the nova.conf with default_availability_zone option | 14:15 |
| lyarwood | mriedem: do we have a bug for the detach from unshelve issue btw? I don't think it's an issue with the target dev, we look this up correctly and call for the detach using the correct xml AFAICT | 14:16 |
| * lyarwood attempts to reproduce locally | 14:16 | |
| mriedem | lyarwood: we don't yet no | 14:16 |
| lyarwood | mriedem: kk, I'll write one up if I can reproduce | 14:16 |
| alexey_weyl | gibi: I see. Thanks a lot | 14:16 |
| mriedem | i think we've been seeing "libvirtError: End of file while reading data: Input/output error" in the logs so much when the libvirt connection temporarily drops, that we just haven't considered this a separate issue | 14:16 |
| mriedem | but it clearly is related to detaching the device | 14:17 |
| mriedem | if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self) | 14:17 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform instance.reboot notifications https://review.openstack.org/382959 | 14:19 |
| sdague | sfinucan: fixed unit tests on - https://review.openstack.org/#/c/446452/ | 14:19 |
| mriedem | lyarwood: i think i see an issue | 14:19 |
| mriedem | lyarwood: in the normal attach volume flow, we pass do_driver_attach=True here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4799 | 14:19 |
| *** Yingxin has quit IRC | 14:19 | |
| mriedem | which is used here https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L265 | 14:20 |
| mriedem | to actually attach the block device to the guest | 14:20 |
| gibi | sdague: Hi! I left a detailed use case description in https://review.openstack.org/#/c/440580/5/specs/pike/approved/scheduler-hints-in-server-details.rst | 14:20 |
| dansmith | gibi: replied on https://review.openstack.org/#/c/445697/2 | 14:20 |
| gibi | dansmith: checking... | 14:20 |
| mriedem | lyarwood: in the case of unshelve, we don't pass that https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4445 | 14:20 |
| mriedem | and it defaults to False | 14:20 |
| *** iceyao has joined #openstack-nova | 14:20 | |
| mriedem | so i don't think we ever actually attach the block device to the guest during unshelve | 14:20 |
| lyarwood | mriedem: isn't part of the test to start the instance? | 14:21 |
| *** yuntongjin has joined #openstack-nova | 14:21 | |
| mriedem | lyarwood: the test doesn't ssh into the guest to see that the device is there | 14:21 |
| mriedem | the test just hits the compute api asking if the volume is associated with the instance, which it is via the bdms table | 14:21 |
| mriedem | that doesn't mean it's actually attached in the guest though :) | 14:21 |
| *** satyar has joined #openstack-nova | 14:22 | |
| mriedem | i'm wondering if it's always been this way and the test is new, or if this was regressed when we removed check_attach internally, | 14:22 |
| lyarwood | mriedem: right, I might be mixing up the ordering here, there's debug XML listing the device when we bring the instance up | 14:22 |
| *** voelzmo has joined #openstack-nova | 14:22 | |
| mriedem | https://github.com/openstack/nova/commit/63805735c25a54ad1b9b97e05080c1a6153d8e22 | 14:22 |
| *** lpetrut has quit IRC | 14:22 | |
| *** lucasxu has joined #openstack-nova | 14:23 | |
| *** Yingxin has joined #openstack-nova | 14:23 | |
| mriedem | nvm, i guess we just never pass do_driver_attach for unshelve | 14:23 |
| *** ekuris has quit IRC | 14:23 | |
| *** yuntongjin has quit IRC | 14:23 | |
| sfinucan | sdague: Cool. Done | 14:23 |
| mriedem | oh, probably because on unshelve we are spawning the instance | 14:23 |
| *** weshay is now known as weshay_pto | 14:24 | |
| *** pcaruana has quit IRC | 14:24 | |
| sfinucan | jaypipes: I'd figured os-vif provided the modelling while nova would just do the wiring up, e.g. altering various aspects of the libvirt XML | 14:25 |
| sfinucan | that change seemed to be more of the former, so that line is blurry :) | 14:25 |
| openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.reboot.error notification https://review.openstack.org/411791 | 14:26 |
| jaypipes | sfinucan: ya, I added a review comment on it. | 14:26 |
| sfinucan | jaypipes: Gotcha. I'll check that out now | 14:26 |
| *** alexey_weyl has quit IRC | 14:27 | |
| gibi | dansmith: your propsal OK for me, thanks for looking into this problem | 14:27 |
| dansmith | gibi: okay cool, I'll push it up.. thanks | 14:27 |
| *** efoley_ has quit IRC | 14:28 | |
| sdague | gibi: ok, further response here, I think we need to at least narrow this | 14:28 |
| *** udesale has joined #openstack-nova | 14:29 | |
| gibi | sdague: thanks, looking... | 14:29 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - add some functional tests for placement https://review.openstack.org/446123 | 14:30 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Refactor placement fixtures https://review.openstack.org/446122 | 14:30 |
| mriedem | lyarwood: ooo i think i've got it | 14:31 |
| *** udesale has quit IRC | 14:31 | |
| mriedem | we're using save_and_reraise_exception incorrectly | 14:31 |
| mriedem | in detach_device_with_retry | 14:31 |
| lbragstad | aunnam did you have a patch that commented out the policy during sample generation? | 14:31 |
| *** udesale has joined #openstack-nova | 14:31 | |
| *** erlon has joined #openstack-nova | 14:31 | |
| mriedem | lyarwood: looking at the logs, we get here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L408 | 14:31 |
| mriedem | File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 408, in _try_detach_device | 14:32 |
| bauzas | mriedem: dansmith: so I looked this CET morning around the archive_deleted_rows method | 14:32 |
| aunnam | lbragstad, i am still fixing the tests that are failing | 14:32 |
| lbragstad | aunnam i can take a stab at porting that to oslo.policy if you want | 14:32 |
| mriedem | lyarwood: but because we aren't telling " with excutils.save_and_reraise_exception():" that we want to raise something new, it re-raises the original libvirtError | 14:32 |
| bauzas | mriedem: dansmith: it seems to me that it could be a bit difficult to use it directly | 14:32 |
| lbragstad | aunnam s/if you want/if you want to keep focusing on the nova tests/ | 14:32 |
| mriedem | lyarwood: which i think then gets us into the retry loop https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L420 | 14:32 |
| mriedem | and at that point we hit the i/o error | 14:33 |
| aunnam | lbragstad, its no the nova tests that are failing, its the tests from oslo.policy | 14:33 |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova-specs master: Add spec for custom resource classes in flavors https://review.openstack.org/446570 | 14:33 |
| lbragstad | aunnam oh - i misunderstood that then | 14:33 |
| jroll | edleafe: jaypipes: cdent: ^ tell me where I'm wrong :) | 14:33 |
| aunnam | lbragstad, s/no/not | 14:33 |
| mriedem | hmm, "If another exception occurs, the saved exception is logged and the new exception is re-raised." | 14:33 |
| mriedem | i wonder if something regressed in oslo.utils | 14:34 |
| cdent | jroll: will you accept "everywhere" as an answer? | 14:34 |
| * cdent puts it in the queue | 14:34 | |
| lbragstad | aunnam for some reason i thought you were working on nova specific tests, i'll give the patch you have a look | 14:34 |
| jroll | cdent: yes, then I'll give up :) | 14:34 |
| *** vks1 has joined #openstack-nova | 14:34 | |
| lbragstad | aunnam is it up for review? | 14:35 |
| * cdent mumbles something about never surrender, on on, rub some dirt on it, etc | 14:35 | |
| aunnam | lbragstad, i'll post it now | 14:35 |
| lyarwood | mriedem: right that's weird, we shouldn't be hitting the no target device error in the first place | 14:35 |
| mriedem | lyarwood: nvm that can't be right, test_detach_device_with_retry_invalid_argument tests for that case | 14:35 |
| aunnam | lbragstad, ya can discuss with you the problem i am facing | 14:35 |
| lbragstad | aunnam cool | 14:36 |
| lbragstad | aunnam fwiw - https://github.com/openstack/oslo.policy/commit/a95606c1dfd7368a247c79d5f65a54c629ce29b2 landed yesterday | 14:36 |
| mriedem | lyarwood: could this be related to the patch you have about how we always pass persistent=True to detach_device_with_retry ? | 14:36 |
| mriedem | https://review.openstack.org/#/c/441204/ | 14:36 |
| aunnam | lbragstad, cool | 14:37 |
| lyarwood | mriedem: hmm yeah if the domain from an unshelve isn't persistent | 14:37 |
| *** VAhl has joined #openstack-nova | 14:37 | |
| mriedem | lyarwood: can you explain what a persistent domain even is? | 14:37 |
| mriedem | i've never actually know | 14:37 |
| *** nic has joined #openstack-nova | 14:38 | |
| lyarwood | mriedem: https://wiki.libvirt.org/page/VM_lifecycle#Transient_guest_domains_vs_Persistent_guest_domains | 14:38 |
| openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.reboot notifications https://review.openstack.org/382959 | 14:38 |
| aunnam | lbragstad, https://review.openstack.org/#/c/443332/1/oslo_policy/generator.py I commented out the rule at line 100 in this patch | 14:38 |
| aunnam | lbragstad, https://github.com/openstack/oslo.policy/blob/master/oslo_policy/tests/test_generator.py#L278 this test is failing | 14:39 |
| openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.reboot.error notification https://review.openstack.org/411791 | 14:39 |
| lyarwood | mriedem: but it's just a normal driver.spawn() in the compute layer so I can't see how that could happen tbh | 14:39 |
| *** hongbin has joined #openstack-nova | 14:40 | |
| aunnam | lbragstad, because in this test it is getting rules from sample policy file and merging those with the modified rules | 14:40 |
| mriedem | lyarwood: yeah that page doesn't help me out here, | 14:40 |
| mriedem | because that implies nova is making a decision when creating the domain as to whether or not it's persistent, right? | 14:40 |
| *** pumarani_ has joined #openstack-nova | 14:41 | |
| mriedem | maybe we are, but can you point out where we decide that? | 14:41 |
| lyarwood | mriedem: just checking but for spawn() I think they are always persistent, live mighration is one of the few cases I know where we start a transient domain on the destination | 14:41 |
| lbragstad | aunnam so what do you have locally? | 14:41 |
| aunnam | lbragstad, so since I commented out the rule the parser is not getting the policy rules and its failing | 14:42 |
| *** carthaca_ has quit IRC | 14:42 | |
| *** sapcc-bot has quit IRC | 14:42 | |
| *** databus23_ has quit IRC | 14:42 | |
| *** nic has quit IRC | 14:42 | |
| lbragstad | aunnam can you paste a `git diff`? | 14:42 |
| *** nic has joined #openstack-nova | 14:42 | |
| openstackgerrit | Sujitha proposed openstack/nova master: Adding tags field to InstancePayload https://review.openstack.org/407228 | 14:43 |
| aunnam | lbragstad, so was thinking to write to policy-sample.yaml file without calling _generate_sample https://github.com/openstack/oslo.policy/blob/master/oslo_policy/tests/test_generator.py#L252 | 14:44 |
| *** mhenkel has joined #openstack-nova | 14:44 | |
| openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.reboot.error notification https://review.openstack.org/411791 | 14:44 |
| aunnam | lbragstad, i am not sure if it is the right way so waiting on that | 14:44 |
| mhenkel | hello jaypipes | 14:44 |
| lyarwood | mriedem: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L833 - defineXML == creates a persistent domain | 14:44 |
| mriedem | lyarwood: ok. another odd thing is in the stacktrace and error, if we were re-raising DeviceNotFound from _try_detach_device, we should see oslo.utils log the original libvirtError before reraising the new exception http://git.openstack.org/cgit/openstack/oslo.utils/tree/oslo_utils/excutils.py#n212 | 14:45 |
| aunnam | lbragstad, thats what found when I look into code | 14:45 |
| mriedem | lyarwood: but i don't see that happen | 14:45 |
| *** yingjun has quit IRC | 14:45 | |
| mriedem | instead we get here http://git.openstack.org/cgit/openstack/oslo.utils/tree/oslo_utils/excutils.py#n220 | 14:45 |
| lbragstad | aunnam i might have missed it, but what changes have you made locally? | 14:45 |
| jaypipes | mhenkel: hi! currently on calls for another hour or so... gimme a bit? :) | 14:45 |
| mriedem | i wonder if we're using a newer eventlet that's causing some context switching bugs | 14:45 |
| *** marst has quit IRC | 14:45 | |
| aunnam | lbragstad, just commented out the rule, that all rule, haven't changed the tests yet | 14:45 |
| mhenkel | jaypipes: sure thing, ping me when you have a minute or two | 14:46 |
| openstackgerrit | Sujitha proposed openstack/nova master: Change tags to default field in Instance object. https://review.openstack.org/415298 | 14:46 |
| *** logan- has quit IRC | 14:46 | |
| aunnam | lbragstad, commented the rule in here https://github.com/openstack/oslo.policy/blob/master/oslo_policy/generator.py#L109 | 14:46 |
| openstackgerrit | Sujitha proposed openstack/nova master: Reduce calls to load_tags() to 0 https://review.openstack.org/435146 | 14:46 |
| *** logan- has joined #openstack-nova | 14:46 | |
| *** dikonoor has quit IRC | 14:47 | |
| mriedem | no that's not it, last time we bumped eventlet up was last may | 14:47 |
| *** sneti_ has joined #openstack-nova | 14:47 | |
| lbragstad | aunnam alright - this is what i have locally http://cdn.pasteraw.com/3u2pj6gigswl55mkeqdbojeifp6gzf4 | 14:48 |
| lbragstad | which is showing me 6 failures | 14:48 |
| lyarwood | mriedem: do you have a logstash query for this already btw? | 14:48 |
| mriedem | lyarwood: just this: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AttachVolumeShelveTestJSON%5C%22%20AND%20message%3A%5C%22Failed%20to%20detach%20volume%5C%22%20AND%20message%3A%5C%22libvirtError%3A%20End%20of%20file%20while%20reading%20data%3A%20Input%2Foutput%20error%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | 14:49 |
| aunnam | lbragstad, ya that's the same think that i have | 14:49 |
| mriedem | lyarwood: did you create a bug? i think we need some more logging in detach_device_with_retry because i can't make out the stacktrace from that method given all of the context managers | 14:49 |
| lbragstad | aunnam cool | 14:50 |
| lyarwood | mriedem: thanks, not yet just waiting on devstack finishing locally before I try to reproduce | 14:50 |
| *** cdent has quit IRC | 14:51 | |
| gibi | sdague: responded on https://review.openstack.org/#/c/440580/5/specs/pike/approved/scheduler-hints-in-server-details.rst I can bargain on what hints the API will return | 14:51 |
| openstackgerrit | Sujitha proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185 | 14:52 |
| openstackgerrit | Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879 | 14:53 |
| *** lpetrut has joined #openstack-nova | 14:53 | |
| *** marst has joined #openstack-nova | 14:53 | |
| mriedem | lyarwood: since we dont see "Original exception being dropped" i'm wondering if we're hitting _try_detach_device the first time, it's ok, and then when we try the 2nd time, it fails with that i/o error | 14:53 |
| mriedem | that's why i want more debug logging in there to know if we're calling _try_detach_device before we get into the retry loop | 14:54 |
| mriedem | because i'm guessing that the i/o error thing on an already detached device is newish in libvirt 1.3.1 | 14:54 |
| mdbooth | lyarwood mriedem: Just finished reading scrollback. Sounds fun. | 14:54 |
| *** tbachman has quit IRC | 14:54 | |
| *** armax has joined #openstack-nova | 14:54 | |
| mriedem | mdbooth: definitely not what i planned on doing for my first 2 hours this morning :) | 14:55 |
| mriedem | gate spelunking | 14:55 |
| *** READ10 has joined #openstack-nova | 14:56 | |
| *** Jack_Iv has joined #openstack-nova | 14:57 | |
| dansmith | gibi: if you're still around, it'd be good to get your ack on this too: https://review.openstack.org/#/c/446053/4 | 14:57 |
| Jack_Iv | Hey Folks! Since Nova 13 the hooks are DEPRECATED, but what can I use instead? | 14:58 |
| *** mlavalle has quit IRC | 14:58 | |
| *** mlavalle has joined #openstack-nova | 14:58 | |
| Jack_Iv | I need to run some code, pre- and post- instance build | 14:58 |
| mriedem | Jack_Iv: there is the dynamic vendordata metadata api and notifications | 14:59 |
| mriedem | or upstream whatever your use case is | 14:59 |
| mriedem | https://docs.openstack.org/developer/nova/vendordata.html | 14:59 |
| mriedem | https://docs.openstack.org/developer/nova/notifications.html | 15:00 |
| *** bvanhav__ is now known as bvanhav | 15:00 | |
| mdbooth | Jack_Iv: What is your use case, btw? | 15:00 |
| *** moshele has quit IRC | 15:00 | |
| gibi | dansmith: looking... | 15:00 |
| Jack_Iv | I need to execute some code on compute nodes, after vm is UP and after termination | 15:01 |
| Jack_Iv | mdbooth: ^ | 15:01 |
| mdbooth | Jack_Iv: Right, but what for? | 15:01 |
| Jack_Iv | let's say, update some iptables rules | 15:01 |
| mdbooth | What does the code do? | 15:01 |
| mdbooth | On the host? | 15:01 |
| Jack_Iv | right | 15:01 |
| kashyap | Yeah, I'm curious, too. About the use case | 15:02 |
| *** karimb has quit IRC | 15:02 | |
| mriedem | lyarwood: https://bugs.launchpad.net/nova/+bug/1673483 | 15:02 |
| openstack | Launchpad bug 1673483 in OpenStack Compute (nova) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Undecided,New] | 15:02 |
| openstackgerrit | Moshe Levi proposed openstack/nova master: HW offload support for openvswitch https://review.openstack.org/398265 | 15:02 |
| mdbooth | Jack_Iv: Is this secret squirrel, or can you share the whole use case? I'm really curious as to why. | 15:02 |
| kashyap | From the an old mailing list discussion on hooks, I see: | 15:03 |
| Jack_Iv | I want to update some iptables rules after VM is up and delete those rules after termination | 15:03 |
| kashyap | There's three core scenarios for hooks | 15:03 |
| kashyap | 1. Modifying some aspect of the Nova operation | 15:03 |
| kashyap | 2. Triggering an external action synchronously to some Nova operation | 15:03 |
| kashyap | 3. Triggering an external action asynchronously to some Nova operation | 15:03 |
| kashyap | [From danpb] | 15:03 |
| *** yingjun has joined #openstack-nova | 15:03 | |
| mriedem | Jack_Iv: you can't do that with security group rules? | 15:03 |
| mdbooth | Jack_Iv: Right. Just trying to understand why, and if there's some other way to achieve what you want. | 15:03 |
| mdbooth | e.g. what mriedem said | 15:03 |
| mriedem | Jack_Iv: i.e. listen for the notifications from nova, instance.create.start and instance.create.end, and when you get those notifications, adjust the secgroup rules in neutron | 15:04 |
| Jack_Iv | No, because clients can just edit security group rules | 15:04 |
| mriedem | Jack_Iv: so change the neutron api to make admin-only rules? | 15:04 |
| *** tblakes has quit IRC | 15:04 | |
| *** karimb has joined #openstack-nova | 15:04 | |
| mriedem | Jack_Iv: at the PTG we talked about a concept of service user locks on instances, | 15:04 |
| mriedem | you could make a similar case for service/admin level "locks" on security group rules | 15:05 |
| *** whenry has quit IRC | 15:05 | |
| *** dharinic_ has quit IRC | 15:05 | |
| dansmith | I thought neutron had some base "provider rules" concept for this | 15:05 |
| mriedem | dansmith: maybe they do | 15:05 |
| mriedem | Jack_Iv: have you talked to the neutron people? | 15:05 |
| Jack_Iv | not yet | 15:05 |
| mriedem | Jack_Iv: which version of openstack are you on? | 15:05 |
| Jack_Iv | newton | 15:05 |
| mriedem | kevinbenton: does neutron have a concept of provider rules for security groups that the tenant user can't change? | 15:06 |
| *** awaugama has joined #openstack-nova | 15:07 | |
| *** mvk has quit IRC | 15:07 | |
| mriedem | Jack_Iv: i think that's probably the thread you need to pull on though | 15:07 |
| *** manasm has quit IRC | 15:08 | |
| mriedem | listen for notifications from nova for the instance build lifecycle, and then adjust rules via the networking API as needed | 15:08 |
| *** manasm has joined #openstack-nova | 15:09 | |
| mriedem | no vmware people around huh | 15:09 |
| mriedem | their CI results come in, if at all, about a week late | 15:09 |
| *** udesale has quit IRC | 15:10 | |
| *** scottda has joined #openstack-nova | 15:11 | |
| *** sridharg has quit IRC | 15:11 | |
| *** manjeets has joined #openstack-nova | 15:13 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Move detach logic from manager into driver BDM https://review.openstack.org/439520 | 15:15 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693 | 15:15 |
| gibi | dansmith: I'm +1 on https://review.openstack.org/#/c/446053 | 15:16 |
| dansmith | gibi: awesome, thanks for looking at that | 15:16 |
| dansmith | gibi: on the other patch, | 15:16 |
| lyarwood | mdbooth: ^ updated removing some of the nits, just to confirm the LM rollback issue isn't new right? | 15:16 |
| mdbooth | lyarwood: No, the LM rollback issue looks new | 15:17 |
| dansmith | gibi: the change I made prevents projects from showing up on actual flavor create/delete, so I'm fixing that by just forcing projects to be loaded in the api before doing the notification | 15:17 |
| mdbooth | The problem is you've combined 2 functions in your move that were previously separate | 15:17 |
| dansmith | gibi: just fyi.. i'll push that up a bit later, it's not a huge rush | 15:17 |
| mdbooth | And remove_volume_connection only called one of them | 15:17 |
| mdbooth | The problem is the interaction of remove_volume_connection and the code from _detach_volume, which we previously didn't call | 15:18 |
| *** Jack_Iv has quit IRC | 15:18 | |
| gibi | dansmith: that sounds OK to me. I will keep an eye on that review | 15:18 |
| *** armax has quit IRC | 15:18 | |
| dansmith | gibi: cool, thanks a bunch :) | 15:19 |
| *** Jack_Iv has joined #openstack-nova | 15:19 | |
| mdbooth | Incidentally, I really don't like the proliferation of do_all_the_things(but_not_this_thing=True, dont_not_do_that_thing=Maybe, do_this_extra_thing='True') | 15:19 |
| *** lucasxu has quit IRC | 15:22 | |
| *** kevinz has quit IRC | 15:22 | |
| *** mvk has joined #openstack-nova | 15:22 | |
| *** eharney has quit IRC | 15:22 | |
| *** Jack_Iv has quit IRC | 15:23 | |
| mdbooth | lyarwood: So, I feel like the old _driver_detach_volume is what needs to move, with perhaps a couple of volume_api calls thrown in. Ideally, though, the code in block_device won't concern itself with CONF.host. It'll be called, or it won't be called. | 15:23 |
| *** dharinic_ has joined #openstack-nova | 15:23 | |
| *** armax has joined #openstack-nova | 15:23 | |
| *** armax has quit IRC | 15:24 | |
| *** Jack_Iv has joined #openstack-nova | 15:24 | |
| mdbooth | lyarwood: That's not 100% thought through, though. There may be practical reasons for dont_do_these_things=['foo', 'bar', 7, None, {}] | 15:25 |
| *** eharney has joined #openstack-nova | 15:26 | |
| *** dharinic_ has quit IRC | 15:26 | |
| * mdbooth just noticed 'thought through, though' in previous sentence, and marvels at his native tongue. | 15:26 | |
| lyarwood | mdbooth: sorry just went over the diff again, I see your point but just to confirm, instance.host is going to point to the source during a rollback right? | 15:28 |
| mriedem | there doesn't seem to be much they're not willing to do in their company | 15:28 |
| mdbooth | Not during rollback_at_destination | 15:28 |
| mriedem | mdbooth: ^ | 15:28 |
| mdbooth | mriedem: Hehe. | 15:28 |
| *** efried has quit IRC | 15:28 | |
| *** tbachman has joined #openstack-nova | 15:29 | |
| *** tblakes has joined #openstack-nova | 15:29 | |
| *** jaosorior has quit IRC | 15:29 | |
| *** arne_r has quit IRC | 15:30 | |
| mdbooth | lyarwood: Sorry, yes. instance.host *will* be the source. I didn't read you right there. | 15:30 |
| mdbooth | However, it's executing on the destination, which is the problem. | 15:30 |
| mdbooth | self.compute_rpcapi.remove_volume_connection( | 15:31 |
| mdbooth | context, instance, bdm.volume_id, dest) | 15:31 |
| mdbooth | In _rollback_live_migration | 15:31 |
| lyarwood | mdbooth: haha right, I just couldn't find the instance.host update | 15:31 |
| mdbooth | So CONF.host != instance.host | 15:31 |
| mdbooth | BUT, we still need to tear it down | 15:31 |
| mdbooth | So I feel like that's the sort of logic that lives in ComputeManager | 15:32 |
| lyarwood | well, we are going to call destroy that's going to do that anyway | 15:32 |
| lyarwood | but sure, I get your point | 15:32 |
| lyarwood | mriedem / johnthetubaguy ; if you have anytime before the cinder meeting the uuid and attachment_id change could really use core review - https://review.openstack.org/#/q/topic:bp/cinder-new-attach-apis & https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1489581 | 15:33 |
| mriedem | dansmith: sdague: melwitt: can we get https://review.openstack.org/#/c/444106/ and the test patch below it in so we can move forward with the backport? it would also help ironic ci that is failing on this. | 15:34 |
| bauzas | dansmith: have you seen my comment on archive_deleted_rows ? | 15:34 |
| mriedem | lyarwood: ok, i meant to go through those again today | 15:34 |
| dansmith | bauzas: you said you didn't think you could use it, but didn't say why (or that I saw) | 15:34 |
| bauzas | dansmith: because the logic in there is looping over the cell DB tables | 15:34 |
| dansmith | bauzas: obviously some refactoring will be needed, but I just would like to avoid having to add yet another command to run cleanup | 15:34 |
| dansmith | mriedem: yeah | 15:34 |
| bauzas | to push them in their shadow one | 15:34 |
| *** hshiina has joined #openstack-nova | 15:35 | |
| mdbooth | lyarwood: In case you didn't notice, I thought the problem in _from_db_object() here was pretty severe: https://review.openstack.org/#/c/242603/23/nova/objects/block_device.py | 15:35 |
| bauzas | dansmith: the main point is that we provide the number of rows per table archived | 15:35 |
| dansmith | bauzas: not per table, but that applies to the thing we need to do in reqspec too right? | 15:36 |
| *** awaugama has quit IRC | 15:36 | |
| bauzas | dansmith: so I should say how many rows for the request_spec table are 'archived', ie. killed? | 15:36 |
| bauzas | dansmith: nope, this is per table AFAICS | 15:36 |
| bauzas | you return either 0 or 1 | 15:36 |
| openstackgerrit | Moshe Levi proposed openstack/os-vif master: HW offload support for openvswitch https://review.openstack.org/398277 | 15:36 |
| dansmith | bauzas: in fact, you have to look at the cell dbs to know what you can remove from reqspec, right? | 15:36 |
| sdague | mriedem: ... once I finish updating the limits spec | 15:36 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: add debug logging in detach_device_with_retry https://review.openstack.org/446601 | 15:36 |
| mriedem | lyarwood: ^ is the debug patch for detach | 15:36 |
| bauzas | but the verbose prints that | 15:37 |
| dansmith | bauzas: are you talking about the parameter or what gets printed? | 15:37 |
| bauzas | dansmith: what's printed | 15:37 |
| *** awaugama has joined #openstack-nova | 15:37 | |
| lyarwood | mdbooth: yeah sorry I did miss that, are you sure that doesn't result in a save straight away? | 15:37 |
| bauzas | dansmith: for looking at which record to remove, I was rather thinking on looking at InstanceMapping | 15:37 |
| mdbooth | lyarwood: If it does, I don't know how | 15:37 |
| bauzas | oh wait | 15:37 |
| dansmith | bauzas: you need to purge those too | 15:37 |
| bauzas | dansmith: we don't remove yet also InstanceMapping records? | 15:37 |
| bauzas | a-ha, yeah | 15:38 |
| dansmith | we don't in the case of soft delete, but otherwise we do | 15:38 |
| bauzas | erf | 15:38 |
| bauzas | okay, then I need to look at all the cells, yeah | 15:38 |
| bauzas | I should also purging the instance_mappings too then | 15:38 |
| lyarwood | mdbooth: ah I might be mixing this up with the driver_bdm dict updates that result in a save | 15:38 |
| bauzas | well, okay, I'll see what I can do to use the same command | 15:39 |
| *** efried has joined #openstack-nova | 15:39 | |
| mdbooth | lyarwood: Also, in the previous patch I didn't like the unnecessary argument type ambiguity. But I think I was ok with everything else. | 15:40 |
| *** hferenc has joined #openstack-nova | 15:43 | |
| lyarwood | mdbooth: the uuid vs bdm_uuid issue? | 15:44 |
| mdbooth | lyarwood: No, just a sec | 15:44 |
| lyarwood | mdbooth: ah the cellv1 values thing | 15:44 |
| mdbooth | https://review.openstack.org/#/c/242602/22/nova/db/sqlalchemy/api.py | 15:44 |
| mdbooth | block_device_mapping_update | 15:45 |
| * mdbooth doesn't like trying to guess the semantics of an argument by running a regex on it | 15:45 | |
| mdbooth | The caller has full context, and there are multiple ways to pass it in unambiguously | 15:45 |
| *** tbachman has quit IRC | 15:46 | |
| *** awaugama has quit IRC | 15:46 | |
| lyarwood | mdbooth: right and at the moment that's always the id so we can just drop this for now | 15:46 |
| mdbooth | Well then you'd have to drop the change in update_or_create | 15:47 |
| *** psachin has quit IRC | 15:47 | |
| mdbooth | It shouldn't be a complex fix. | 15:47 |
| lyarwood | ah yeah got it | 15:48 |
| dansmith | bauzas: remind me, we don't delete reqspec at all right now, but will in the case of non-soft delete soon right? | 15:48 |
| dansmith | bauzas: so this command will really only be needed if operators have soft delete enabled, right? | 15:48 |
| *** cdent has joined #openstack-nova | 15:48 | |
| bauzas | dansmith: yup, we don't delete the spec yety | 15:49 |
| bauzas | dansmith: if someone hard-deletes an instance (ie. the instance now longer exists), then we should purge the entry | 15:49 |
| mriedem | mdbooth: lyarwood: in case you haven't noticed yet, the original patches to add bdm.uuid were for a cellsv1 race issue where update_or_create happened at the api cell and things got racey and crazy, | 15:49 |
| mriedem | mdbooth: lyarwood: so that's why there is the update_or_create stuff | 15:50 |
| dansmith | bauzas: you're saying you will soon delete reqspec at normal instance delete time, yes? | 15:50 |
| bauzas | dansmith: tbc, I was planning to iterate over all the spec records, and lookup over all cells to see if some instance exists | 15:50 |
| bauzas | for the purge command | 15:50 |
| dansmith | not talking about purge, I'm talking about normal api delete.. | 15:50 |
| mriedem | mdbooth: lyarwood: but i believe that update_or_create is also still called from the compute api (and/or conductor now) when creating the bdms | 15:50 |
| bauzas | dansmith: ah, that, I wasn't planning to introduce it given mriedem's point | 15:50 |
| dansmith | bauzas: his point about what? | 15:50 |
| bauzas | dansmith: the fact that we shouldn't delete the spec record in case this is an instance soft-delete | 15:51 |
| mdbooth | mriedem: IIRC that's correct. The cellsv1 caller was unique in that it passed in a versioned object instead of a list of update values, though. | 15:51 |
| bauzas | dansmith: https://review.openstack.org/#/c/391060/2/nova/compute/api.py@1804 | 15:51 |
| dansmith | bauzas: right but assuming like 1% of people use soft delete, we should do the right thing in the 99% case and only require people that use soft-delete to have to worry about this purge | 15:51 |
| dansmith | I dunno what percentage do use it, but we should always delete if we know we can | 15:52 |
| bauzas | dansmith: so, I could just verify if it's a soft-delete ? | 15:52 |
| *** david-lyle has joined #openstack-nova | 15:52 | |
| bauzas | dansmith: and if it's not a soft-delete, then call the _delete_req_spec method? | 15:52 |
| dansmith | bauzas: yeah, of course, I'm not sure why we wouldn't | 15:52 |
| bauzas | lemme verify, but delete_type seems the good argument for verify it | 15:53 |
| *** lucasxu has joined #openstack-nova | 15:53 | |
| *** tbachman has joined #openstack-nova | 15:53 | |
| bauzas | dansmith: honestly, purging using the archive command would be super-expensive | 15:53 |
| *** nic has quit IRC | 15:53 | |
| bauzas | because I need to iterate over all the spec records, and then iterate over all cells for each record, so o(n2) | 15:54 |
| dansmith | bauzas: further, in the purge command, it's going to be very intensive to compare all the current reqspecs with all the deleted=1 instances in a cell, so I think you should just do the api-db purge when you archive an instance row, which will be much more efficient. | 15:54 |
| dansmith | bauzas: yeah, that's completely not okay | 15:54 |
| dansmith | bauzas: doing the purge inside archive_deleted_rows is the only place you can do it efficiently | 15:54 |
| dansmith | a separate purge command will have to do it O(n^2) | 15:54 |
| bauzas | yeah, sorry my keybord doesn't ² :p | 15:55 |
| dansmith | a separate purge command will not be able to effectively determine which deleted instances already have a purged api-db record, | 15:55 |
| dansmith | which makes it expensive | 15:55 |
| dansmith | and after you archive, it's too late | 15:55 |
| *** david-lyle_ has joined #openstack-nova | 15:55 | |
| *** sandanar has quit IRC | 15:55 | |
| bauzas | dansmith: correct me if I'm wrong, but archive_deleted_rows isn't yet cells-aware ? | 15:55 |
| dansmith | but if you do it when you archive, then you can do it in O(n) just like archive is now | 15:55 |
| *** david-lyle has quit IRC | 15:55 | |
| mdbooth | lyarwood: On locking, I think that everything in api which checks instance/task state should be required to immediately and atomically modify task state. That's probably the level of locking required for rebuild. | 15:56 |
| dansmith | bauzas: what does it matter if it is? whether you're doing one cell db or all of them, it's the same amount of work | 15:56 |
| mriedem | bauzas: the delete_type arg is what tells you the type, | 15:56 |
| mriedem | "soft_delete" or just "delete" | 15:56 |
| mdbooth | I think we do that in many places, btw. | 15:56 |
| bauzas | dansmith: sorry if I'm unclear, but that archive command would have to be run close to a child cell DB, right? | 15:56 |
| bauzas | dansmith: if so, that would be an upcall, right? | 15:56 |
| sean-k-mooney | jaypipes: looking at https://review.openstack.org/#/c/441183 im not sure that any os-vif change is required | 15:56 |
| dansmith | bauzas: it's nova-manage, upcalls are fine | 15:57 |
| *** david-lyle__ has joined #openstack-nova | 15:57 | |
| sean-k-mooney | jaypipes: the contrial pluging for os-vif would live in the networking-contrail repo not os-vif normally | 15:57 |
| bauzas | dansmith: mmm, okay, I need to consider that | 15:57 |
| mriedem | bauzas: can we just focus on the hard delete case in the api as i pointed out in the review yesterday? and worry about how to best cleanup the soft delete orphans in a separate change? | 15:57 |
| dansmith | yes, please | 15:57 |
| dansmith | not deleting when we know we can is crazypants | 15:57 |
| mriedem | they should be separate changes anyway | 15:57 |
| lyarwood | mdbooth: would that not still allow something like volume-attach to get through? | 15:57 |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted to address hung remote replication tasks, and should return to an operable state momentarily | 15:58 | |
| lyarwood | ah, not just me then | 15:58 |
| mdbooth | lyarwood: Not if everything does it, no. | 15:58 |
| *** marst_ has joined #openstack-nova | 15:58 | |
| bauzas | mriedem: dansmith: yeah, tbc, based on dan's good point, I'm just modifying https://review.openstack.org/#/c/391060/2/nova/compute/api.py to just verify delete_type | 15:58 |
| mdbooth | It's not consistent, though. | 15:58 |
| *** awaugama has joined #openstack-nova | 15:58 | |
| bauzas | and leave the soft-delete deletions be done differently | 15:58 |
| bauzas | I need to go, ttyl | 15:58 |
| *** marst has quit IRC | 15:59 | |
| mriedem | bauzas: i said that yesterday in the review :) | 15:59 |
| *** david-lyle has joined #openstack-nova | 15:59 | |
| mriedem | "So we can hard delete the request spec in the API when we hard delete an instance (the delete_type would tell you which it is)." | 15:59 |
| mriedem | "If we care about adding a way to purge leaked request specs (request specs that are related to instances which are deleted), then we could provide a nova-manage command for that, e.g. nova-manage api_db purge_zombies, or something similar. I'd do the purge CLI in a separate patch btw." | 15:59 |
| mdbooth | lyarwood: Anyway, as you say that's not really in scope. | 15:59 |
| *** david-lyle__ has quit IRC | 15:59 | |
| *** karimb has quit IRC | 16:00 | |
| * mriedem takes kid to school | 16:00 | |
| *** david-lyle_ has quit IRC | 16:00 | |
| mhenkel | sean-k-mooney, Hello | 16:01 |
| sean-k-mooney | jaypipes: that said if the plugin was ovs then yes i would have expected a patch to os-vif | 16:02 |
| sean-k-mooney | mhenkel: hieulq | 16:02 |
| sean-k-mooney | mhenkel: *hi | 16:02 |
| sean-k-mooney | mhenkel: just dissusing your change | 16:02 |
| *** karimb has joined #openstack-nova | 16:02 | |
| sean-k-mooney | mhenkel: do im not sure jay is about currently | 16:02 |
| mhenkel | I was just debating internally if we should upstream https://github.com/Juniper/contrail-nova-vif-driver/tree/master/vif_plug_vrouter to os-vif | 16:03 |
| mhenkel | or keep it as a separate plugin delivered by contrail packages | 16:03 |
| *** voelzmo has quit IRC | 16:04 | |
| sean-k-mooney | mhenkel: currently the policy for plugins in os-vif is that they can only be fro the reference backends | 16:04 |
| mhenkel | my personal preference would be to upstream | 16:04 |
| sean-k-mooney | so ovs,linux bridge and sriov | 16:04 |
| mhenkel | ah ok | 16:04 |
| sean-k-mooney | that is something we could change but the resoning is that it allows you to maintain and update them faster yourself | 16:05 |
| *** david-lyle has quit IRC | 16:05 | |
| mhenkel | sean-k-mooney: yes, that I agree with | 16:05 |
| sean-k-mooney | mhenkel: one thing that will be changing this cycle hopefully is that neutron will start passing os-vif vif object to nova | 16:06 |
| *** r-daneel has joined #openstack-nova | 16:06 | |
| mhenkel | sean-k-mooney: so we should be good with what I upstreamed so far? | 16:07 |
| sean-k-mooney | mhenkel: when that happens the change you are proposing will nolonger be required as you will be able to spcify the plug to use for the ml2 mech dirver | 16:07 |
| sean-k-mooney | mhenkel: yes | 16:07 |
| sean-k-mooney | mhenkel: though one question | 16:07 |
| sean-k-mooney | for contrial are you managin ovs or do you support other backends | 16:08 |
| mhenkel | no ovs, we provide our own virtual router | 16:08 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Get instance availability_zone without hitting the api db https://review.openstack.org/439754 | 16:08 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Avoid lazy-loading projects during flavor notification https://review.openstack.org/445697 | 16:08 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Set instance.availability_zone whenever we schedule https://review.openstack.org/446053 | 16:08 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make conductor ask scheduler to limit migrates to same cell https://review.openstack.org/438025 | 16:08 |
| mhenkel | either as a kmod or in userspace using dpdk | 16:08 |
| sean-k-mooney | mhenkel: ah ok. in that case what you have is perfect | 16:09 |
| *** nic has joined #openstack-nova | 16:09 | |
| mhenkel | sean-k-mooney: so we just need to convince jaypipes to take back his -1 ;) | 16:09 |
| sean-k-mooney | mhenkel: if you were managing ovs(with or without dpdk) you could upstream the changes that are required to the ovs plugin | 16:09 |
| *** amotoki has joined #openstack-nova | 16:09 | |
| mhenkel | I see | 16:09 |
| sean-k-mooney | mhenkel: yes but jaypipes will likely agree when he reads my comment | 16:10 |
| openstackgerrit | Sujitha proposed openstack/nova master: Add helper method to add additional data about policy rule. https://review.openstack.org/434842 | 16:11 |
| *** karimb has quit IRC | 16:12 | |
| melwitt | mriedem: on the regression test, do you know why you didn't need RetryFilter? | 16:12 |
| openstackgerrit | Sujitha proposed openstack/nova master: Add helper method to add additional data about policy rule. https://review.openstack.org/434842 | 16:13 |
| *** NikhilS has quit IRC | 16:14 | |
| openstackgerrit | Stephen Finucane proposed openstack/os-vif master: Use Sphinx 1.5 warning-is-error https://review.openstack.org/446616 | 16:15 |
| bauzas | mriedem: sorry if I misunderstood you | 16:16 |
| *** nkrinner is now known as nkrinner_afk | 16:16 | |
| bauzas | I don't want to play the French card, but... | 16:16 |
| bauzas | I thought both of you were asking me to abandon that change and just use the manage command for purging | 16:17 |
| *** karimb has joined #openstack-nova | 16:17 | |
| *** belmoreira has quit IRC | 16:18 | |
| jroll | dansmith: got a pointer to the work to expose custom classes in allocations? (is that happening yet?) | 16:18 |
| dansmith | jroll: I think it is .. in jay's head at least | 16:18 |
| jroll | dansmith: keep in mind I mostly have no idea how allocations work, so I'm not sure what's missing here. I guess it's that we can't yet create an allocation for CUSTOM_FOO? | 16:19 |
| *** Drankis has joined #openstack-nova | 16:19 | |
| dansmith | jroll: we can we just don't | 16:19 |
| jroll | dansmith: right, so one of the work items is to create those allocations, unless I'm missing something I don't see the dep | 16:20 |
| *** sneti_ has quit IRC | 16:20 | |
| *** marst_ has quit IRC | 16:20 | |
| *** dtp has joined #openstack-nova | 16:20 | |
| *** Guest15143 has quit IRC | 16:20 | |
| *** marst has joined #openstack-nova | 16:20 | |
| dansmith | jroll: ah, okay I didn't interpret that work item as this thing, probably because the spec seems focused on just the flavor override and scheduling part | 16:21 |
| dansmith | jroll: in that case, it should probably be "RT changes to allow drivers to expose allocations of custom things" and "make ironic driver do that thing" | 16:21 |
| jroll | dansmith: feels like we need to handle it at the same time, else the node remains free to be scheduled to :) | 16:21 |
| *** r-daneel has quit IRC | 16:21 | |
| dansmith | jroll: I don't think so, we can do the allocation thing first | 16:22 |
| jroll | ah, true | 16:22 |
| dansmith | if jay is doing that he should probably be on the contributors too | 16:22 |
| *** pumarani_ has quit IRC | 16:22 | |
| *** slunkad_ has quit IRC | 16:22 | |
| jroll | dansmith: okay, so I guess I'll rearrange the work items, split it into RT/ironic items, and add jaypipes. cool? | 16:22 |
| *** Guest15143 has joined #openstack-nova | 16:23 | |
| dansmith | jroll: sure, but if you're going to include all that work in here, you probably need to add some wordy description of the allocation bits into the proposed change as well, because that's probably why I assumed this was only the flavor/scheduling work | 16:23 |
| jroll | gotcha | 16:24 |
| jroll | dansmith: I have no clue how that thing works, I'll add as a dep for now until I chat with jay | 16:24 |
| dansmith | okay if he wants it all in this spec I'm sure he can just add it himself | 16:26 |
| *** cdent has quit IRC | 16:27 | |
| jroll | ya | 16:27 |
| mriedem | lyarwood: so i was thinking, we should probably decouple the attachment_id and the rest of that series from the bdm.uuid patch, since that's getting overly complicated at this point | 16:28 |
| mriedem | lyarwood: and it's been about 2 weeks, can we just decouple that from the series? | 16:28 |
| mriedem | melwitt: adding the RetryFilter in makes it blow up for some reason | 16:29 |
| mriedem | melwitt: you could pull it down and add that to see what i mean, but it seems to think that there is only one host, | 16:29 |
| mriedem | i'm not sure if that's because both compute services have the same fake-mini node? | 16:30 |
| *** lucasxu has quit IRC | 16:30 | |
| melwitt | mriedem: yeah, I suspect it's trying to same host again. it still tests the bug since it is retrying, it's just probably not landing on the second host | 16:30 |
| lyarwood | mriedem: yup happy to, I'll do that now before the call | 16:31 |
| mriedem | melwitt: well, it dumps a message like "host1 in list, hosts already tried: host1" | 16:31 |
| lyarwood | s/call/meeting/g | 16:31 |
| mriedem | and then kicks it out, | 16:31 |
| melwitt | mriedem: oh? I pulled it down a few minutes ago and am playing with it, just out of curiosity | 16:31 |
| mriedem | i'm not sure why it doesn't see host2 | 16:31 |
| mriedem | my guess is the node values are the same, but i don't see why that should matter | 16:31 |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova-specs master: Add spec for custom resource classes in flavors https://review.openstack.org/446570 | 16:31 |
| mriedem | lyarwood: thanks | 16:32 |
| jroll | dansmith: done, thanks for the help | 16:32 |
| *** karimb has quit IRC | 16:38 | |
| *** mdrabe has quit IRC | 16:38 | |
| *** tonygunk has quit IRC | 16:40 | |
| jaypipes | mhenkel: where is the os-vif plugin for vrouter? | 16:43 |
| *** aarefiev is now known as aarefiev_afk | 16:44 | |
| jaypipes | mhenkel: nm, found it, sorry. | 16:45 |
| mhenkel | jaypipes: cool | 16:45 |
| openstackgerrit | Octave Orgeron proposed openstack/nova-specs master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446626 | 16:45 |
| *** armax has joined #openstack-nova | 16:46 | |
| *** bmace has quit IRC | 16:47 | |
| *** bmace has joined #openstack-nova | 16:48 | |
| *** iceyao has quit IRC | 16:50 | |
| *** ltomasbo is now known as ltomasbo|away | 16:50 | |
| *** catinthe_ has joined #openstack-nova | 16:52 | |
| *** mdrabe has joined #openstack-nova | 16:52 | |
| *** andreas_s has quit IRC | 16:54 | |
| *** catintheroof has quit IRC | 16:54 | |
| openstackgerrit | Octave Orgeron proposed openstack/nova-specs master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446631 | 16:55 |
| *** Jack_Iv has quit IRC | 16:56 | |
| *** unicell has joined #openstack-nova | 16:56 | |
| *** unicell has quit IRC | 16:56 | |
| *** Jack_Iv has joined #openstack-nova | 16:56 | |
| *** unicell has joined #openstack-nova | 16:56 | |
| openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Use Sphinx 1.5 warning-is-error https://review.openstack.org/446632 | 16:56 |
| *** Apoorva has joined #openstack-nova | 16:57 | |
| jaypipes | mhenkel: still -1 from me, but I understand now where the os-vif stuff is. :) | 16:58 |
| jaypipes | mhenkel: pls see my review comments for details. | 16:58 |
| mhenkel | jaypipes: ok, will check, thanks! | 16:59 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: objects: Add attachment_id to BlockDeviceMapping https://review.openstack.org/437665 | 16:59 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: db: Add attachment_id to block_device_mapping https://review.openstack.org/437597 | 16:59 |
| jaypipes | mhenkel: np! :) | 16:59 |
| jaypipes | mhenkel: if you add those unit tests and make the commit message changes, I'll re-review quickly and with sean-k-mooney's +1 pass it over to sfinucan or mriedem to +2. | 17:00 |
| *** hshiina has quit IRC | 17:00 | |
| mhenkel | jaypipes: will do! | 17:00 |
| *** Jack_Iv has quit IRC | 17:01 | |
| *** ltomasbo|away is now known as ltomasbo | 17:01 | |
| *** bvanhav has quit IRC | 17:01 | |
| *** READ10 has quit IRC | 17:01 | |
| *** catintheroof has joined #openstack-nova | 17:01 | |
| jaypipes | mriedem: https://review.openstack.org/#/c/416669/ looks sensible and has had several cdent reviews around API correctness. | 17:01 |
| *** avolkov` has quit IRC | 17:01 | |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add spec to use cinder's new attachment API https://review.openstack.org/373203 | 17:02 |
| sean-k-mooney | jaypipes: i proably should have been more stringent in my own review but the main think i was reviewing for was if the os-vif delegation looked correct. | 17:02 |
| jaypipes | sean-k-mooney: it's cool duder :) | 17:02 |
| mdbooth | mriedem lyarwood: I've been looking at that detach gate failure, discussing with danpb downstream. | 17:02 |
| *** Jack_Iv has joined #openstack-nova | 17:02 | |
| mdbooth | I currently suspect it's a libvirt issue. | 17:03 |
| mdbooth | If you look at the libvirt logs, we see the device_del, then the job disappears into a black hole, then Nova reports a disconnect. | 17:03 |
| sean-k-mooney | jaypipes: that said alot of that code that deligate to os-vif in nova is not unittested correctly so this is something i guess we should improve going forward | 17:03 |
| jaypipes | sean-k-mooney: ack | 17:04 |
| *** ralonsoh has quit IRC | 17:04 | |
| *** fragatina has joined #openstack-nova | 17:04 | |
| mriedem | jaypipes: ok, after meetings and other things, like this big ass bowl of chili in front of me | 17:04 |
| *** fragatina has quit IRC | 17:04 | |
| *** fragatina has joined #openstack-nova | 17:05 | |
| mdbooth | lyarwood mriedem: All of the above occurs within the context of, from Nova's pov, self._domain.detachDeviceFlags(device_xml, flags=flags) | 17:05 |
| mdbooth | So at the moment I don't see how it can be a Nova issue | 17:05 |
| *** catinthe_ has quit IRC | 17:05 | |
| sean-k-mooney | mhenkel: the unitest in https://review.openstack.org/#/c/334048/ may be of use to you as a reference. | 17:06 |
| *** marst_ has joined #openstack-nova | 17:07 | |
| mdbooth | mriedem: How long do those log files live, btw? If I link to them in bz will they still be there in a month or 2? | 17:07 |
| *** manasm has quit IRC | 17:08 | |
| mriedem | mdbooth: i'm not sure about that, it used to be 6 months, | 17:09 |
| mriedem | fungi: jeblair: ^ how long does infra keep ci job logs around? | 17:09 |
| *** marst has quit IRC | 17:09 | |
| fungi | mriedem: now it's down to about 45 days | 17:09 |
| fungi | mdbooth: ^ | 17:10 |
| jaypipes | mriedem: :) | 17:10 |
| mdbooth | fungi: Thanks | 17:10 |
| * mdbooth will copy them :) | 17:10 | |
| sean-k-mooney | jaypipes: actully while you are around if you get a chance to look at https://review.openstack.org/#/c/441590/ let me know if you are ok with this direction. i belive that with this change the host_info class has all the info we need to pass to neutron for the port binding negciation. | 17:10 |
| fungi | mriedem: mdbooth: that's keeping us barely afloat with the present job log volume http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 17:10 |
| jeblair | (it's directly tied to log volume; we've had to reduce it due to a significant increase in the amount of data jobs are storing in logs) | 17:10 |
| sean-k-mooney | jaypipes: that said i dont want to merge this until i have some prototype code to consume it | 17:11 |
| mriedem | mdbooth: fwiw i didn't notice this type of error when we were using trusty | 17:11 |
| mriedem | which was libvirt 1.2.2 | 17:11 |
| mdbooth | fungi: NP. I originally assumed it would be days, tbh. | 17:11 |
| mdbooth | 45 days is excellent. | 17:12 |
| sean-k-mooney | sfinucan: https://review.openstack.org/#/c/441590/ is proably of interest to you too. again im going to keep the workflow -1 until i have an end to end poc working to ensure it does capture all required info | 17:12 |
| *** rcernin has joined #openstack-nova | 17:12 | |
| fungi | mdbooth: we wish it could be more, but we generate _lots_ of logs these days | 17:14 |
| *** catintheroof has quit IRC | 17:14 | |
| *** catintheroof has joined #openstack-nova | 17:15 | |
| *** bvanhav has joined #openstack-nova | 17:17 | |
| *** manasm has joined #openstack-nova | 17:17 | |
| *** jdillaman has joined #openstack-nova | 17:19 | |
| openstackgerrit | Christopher Brown proposed openstack/nova master: Add lan9118 as valid nic for hw_vif_model property for qemu https://review.openstack.org/393489 | 17:19 |
| *** lucasagomes is now known as lucas-afk | 17:21 | |
| jaypipes | sean-k-mooney: reviewed. | 17:22 |
| *** tonygunk has joined #openstack-nova | 17:24 | |
| mriedem | fungi: totally fine | 17:25 |
| mriedem | was just wondering the timeline | 17:25 |
| openstackgerrit | Octave Orgeron proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643 | 17:25 |
| *** ociuhandu has quit IRC | 17:26 | |
| *** baoli has quit IRC | 17:26 | |
| sean-k-mooney | jaypipes: thanks that also needs units so ill adress your comments when i add them | 17:27 |
| *** tbachman has quit IRC | 17:27 | |
| *** baoli has joined #openstack-nova | 17:27 | |
| openstackgerrit | Octave Orgeron proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643 | 17:27 |
| *** tbachman has joined #openstack-nova | 17:28 | |
| *** READ10 has joined #openstack-nova | 17:28 | |
| *** rcernin has quit IRC | 17:28 | |
| *** amotoki has quit IRC | 17:32 | |
| *** dtp has quit IRC | 17:32 | |
| mdbooth | lyarwood mriedem: libvirtd crashed | 17:33 |
| mriedem | mdbooth: gah | 17:34 |
| mriedem | remember how i said i'd always chalked this failure up to libvirt crashing :) | 17:34 |
| mdbooth | mriedem: So, I think we're going to request some changes in the CI environment | 17:34 |
| mdbooth | e.g. It would be nice if libvirt didn't restart | 17:35 |
| mdbooth | Also, if we collected core dumps | 17:35 |
| mdbooth | And there are some additional logs which would be useful | 17:35 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Teach HostAPI about cells https://review.openstack.org/442162 | 17:35 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891 | 17:35 |
| mdbooth | mriedem: What's the best way to request ^^^ ? | 17:35 |
| mriedem | i.e. http://status.openstack.org/elastic-recheck/#1643911 http://status.openstack.org/elastic-recheck/#1646779 http://status.openstack.org/elastic-recheck/#1638982 | 17:35 |
| mriedem | mdbooth: i think the libvirtd logging stuff is handled in devstack | 17:36 |
| *** derekh has quit IRC | 17:36 | |
| *** ociuhandu has joined #openstack-nova | 17:36 | |
| *** tbachman has quit IRC | 17:36 | |
| mriedem | mdbooth: https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/functions-libvirt#L104 | 17:37 |
| mdbooth | http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/479c7bf/logs/syslog.txt.gz#_Mar_16_01_01_32 | 17:38 |
| mdbooth | btw | 17:38 |
| *** bvanhav has quit IRC | 17:38 | |
| openstackgerrit | John Garbutt proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693 | 17:38 |
| cfriesen | is anyone aware of a way to have libvirt/qemu live migration maintain the "sparseness" of a qcow2 disk file? After live migration I'm seeing the physical size match the virtual size even though most of that space is not actually used. | 17:38 |
| *** tbachman has joined #openstack-nova | 17:39 | |
| mdbooth | cfriesen: Is this an image-backed qcow2? | 17:42 |
| mdbooth | Because if so, it should already do that. | 17:42 |
| *** aysyd has quit IRC | 17:44 | |
| mriedem | mdbooth: ok so we can just duplicate this bug i guess, | 17:44 |
| mriedem | and i wasted everyone's time | 17:44 |
| mriedem | but thanks for digging into it, | 17:44 |
| mriedem | if we can get more details on the actual crashes, that'd be great, | 17:44 |
| openstackgerrit | sean mooney proposed openstack/nova master: remove flake8-import-order for test requirements https://review.openstack.org/445622 | 17:44 |
| mdbooth | mriedem: We should raise a bug against Ubuntu's libvirt package | 17:44 |
| mriedem | becaues it's a pretty high probability gate failure since we moved to xenial | 17:44 |
| mriedem | i wouldn't really know what to say in the bug, | 17:45 |
| mriedem | except "it crashes :P " | 17:45 |
| mdbooth | Well, we've a dump from one of those crashes | 17:45 |
| mriedem | what's that palms up idk guy? | 17:45 |
| mdbooth | Sorry, not the dump | 17:45 |
| mdbooth | The 'panic' or whatever, summary | 17:45 |
| mriedem | ¯\_(ツ)_/¯ | 17:45 |
| mriedem | that guy | 17:45 |
| *** kaisers has quit IRC | 17:45 | |
| mdbooth | Hehe | 17:45 |
| johnthetubaguy | lyarwood: I have a tiny nit on this one: https://review.openstack.org/#/c/437597/ | 17:46 |
| lyarwood | looking | 17:46 |
| johnthetubaguy | lyarwood: I am wondering if we don't actually need the get_by_attachment_id method? | 17:46 |
| mdbooth | lyarwood: In https://review.openstack.org/#/c/440693/9/nova/compute/manager.py you've still got the notification out of order | 17:48 |
| mdbooth | lyarwood: I thought we said we wouldn't do that? | 17:48 |
| lyarwood | mdbooth: didn't your review say that it was fine to raise the notification after the detach? | 17:49 |
| johnthetubaguy | yeah, I saw the note added in the commit message | 17:49 |
| mdbooth | lyarwood: It did, but then we discussed it with gibi iirc? | 17:49 |
| johnthetubaguy | it sounds sensible | 17:49 |
| mdbooth | I also said I don't really understand the contract :) | 17:49 |
| johnthetubaguy | right now its half way through the detach right, it should be either before we call cinder, or after I think | 17:50 |
| mdbooth | As it stands today, you'll get a notification even if the detach fails | 17:50 |
| mdbooth | With this patch, you *won't* get a notification if the detach fails | 17:50 |
| mdbooth | That makes sense to me, but it is an unrelated change of behaviour | 17:50 |
| johnthetubaguy | well, depends where it fails | 17:50 |
| johnthetubaguy | really, you should get an error notification if it fails | 17:50 |
| mdbooth | johnthetubaguy: In volume_api.detach() | 17:50 |
| mdbooth | johnthetubaguy: The point was also raised that we could change that entirely | 17:51 |
| mdbooth | But I thought for safety we should just leave it as is | 17:51 |
| johnthetubaguy | the real error happen in _driver_detach_volume | 17:51 |
| mdbooth | And change it deliberately at another time if that's what we want | 17:51 |
| johnthetubaguy | so I think the tweak seems good | 17:51 |
| johnthetubaguy | its tempting to do it after the BDM clean up I supose | 17:51 |
| lyarwood | I'm working on this now so it's not an issue to kick it out | 17:52 |
| johnthetubaguy | oh, wait, we can't we need to send that info as part of the instance | 17:52 |
| johnthetubaguy | so here is a reason to go lyarwood's way | 17:52 |
| johnthetubaguy | in the new API there is only one cinder API call | 17:53 |
| *** asselin has joined #openstack-nova | 17:53 | |
| *** asselin has left #openstack-nova | 17:53 | |
| *** fragatina has quit IRC | 17:54 | |
| johnthetubaguy | lyarwood: I was thinking the fake BDMs should probably have attachment_id = None, so it matches the current BDMs generated by the code: https://review.openstack.org/#/c/437665/11/nova/tests/unit/objects/test_block_device.py@36 | 17:55 |
| lyarwood | johnthetubaguy: true, I was thinking ahead to testing detach but you're right that doesn't make sense yet. | 17:56 |
| johnthetubaguy | lyarwood: I would keep the uuids for the object compat one though, that seems legit | 17:57 |
| *** satyar has quit IRC | 17:57 | |
| lyarwood | johnthetubaguy: kk | 17:57 |
| *** READ10 has quit IRC | 17:58 | |
| lyarwood | johnthetubaguy: and re get_by_attachment_id , I just assumed we would want that around tbh, if you can't think of a case where we would want that over get_by_instance* then I'll drop it | 17:58 |
| *** yamahata has joined #openstack-nova | 17:59 | |
| *** sc68cal has quit IRC | 17:59 | |
| johnthetubaguy | lyarwood: I think we will always fetch using volume-uuid and instance-uuid, because of API flow. Its easy to add when we need it though, best to leave it out for now I think. | 18:00 |
| johnthetubaguy | lyarwood: I will probably regret that tomorrow, but it seems the best way to go | 18:00 |
| lyarwood | johnthetubaguy: kk, should I leave the index on attachment_id? | 18:00 |
| mriedem | probably don't need the index if we don't query by attachment_id right? | 18:00 |
| johnthetubaguy | lyarwood: good question, I guess we can leave that off too | 18:01 |
| johnthetubaguy | yeah, what mriedem said | 18:01 |
| lyarwood | right, just checking before I pull it | 18:01 |
| lyarwood | thanks | 18:01 |
| johnthetubaguy | index will slow down write performance, so worth dropping tht | 18:01 |
| *** kaisers has joined #openstack-nova | 18:01 | |
| *** vks1 has left #openstack-nova | 18:03 | |
| *** ltomasbo is now known as ltomasbo|away | 18:03 | |
| *** cfriesen has quit IRC | 18:04 | |
| *** alexpilotti has joined #openstack-nova | 18:05 | |
| *** david-lyle has joined #openstack-nova | 18:05 | |
| *** sc68cal has joined #openstack-nova | 18:06 | |
| *** aysyd has joined #openstack-nova | 18:06 | |
| *** jpena is now known as jpena|off | 18:07 | |
| mriedem | gibi: have you ever seen this? http://logs.openstack.org/08/445308/3/check/gate-tempest-dsvm-py35-ubuntu-xenial/7bf0d72/logs/screen-n-api.txt.gz#_2017-03-16_05_31_09_399 | 18:08 |
| mriedem | ValueError: Circular reference detected | 18:08 |
| mriedem | during send_notification | 18:08 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Clarify os-stop API description https://review.openstack.org/446264 | 18:08 |
| *** cfriesen has joined #openstack-nova | 18:08 | |
| *** baoli has quit IRC | 18:09 | |
| mriedem | oomichi: left a question in ^ | 18:13 |
| mriedem | for os-start | 18:13 |
| *** baoli has joined #openstack-nova | 18:13 | |
| *** gszasz has quit IRC | 18:13 | |
| *** manasm has quit IRC | 18:14 | |
| oomichi | mriedem: oh, nice catch. I am a fun for removing it also | 18:15 |
| *** Sukhdev has joined #openstack-nova | 18:16 | |
| *** lpetrut has quit IRC | 18:16 | |
| *** tbachman has quit IRC | 18:16 | |
| *** sc68cal_ has joined #openstack-nova | 18:17 | |
| *** sc68cal has quit IRC | 18:17 | |
| mdbooth | mriedem: I've also seen Circular reference detected recently | 18:17 |
| mdbooth | After an error | 18:17 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Clarify os-stop API description https://review.openstack.org/446264 | 18:17 |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova master: Remove check_detach https://review.openstack.org/446671 | 18:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446672 | 18:18 |
| oomichi | mriedem: ^^^ thanks, done | 18:18 |
| mdbooth | mriedem: http://logs.openstack.org/42/445142/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/07e282a/logs/screen-n-api.txt.gz?level=ERROR#_2017-03-15_10_18_17_459 | 18:18 |
| mdbooth | Is that similar? | 18:18 |
| mriedem | mdbooth: yes | 18:19 |
| *** sneti_ has joined #openstack-nova | 18:19 | |
| mriedem | mdbooth: https://bugs.launchpad.net/nova/+bug/1673375/comments/3 | 18:19 |
| openstack | Launchpad bug 1673375 in oslo.messaging ""ValueError: Circular reference detected" in send_notification" [Undecided,New] | 18:19 |
| mriedem | the notification is coming from the wrap_exception decorator here https://github.com/openstack/nova/blob/2380659e358770a3f36253b93a112b9779a23958/nova/compute/api.py#L4602 | 18:19 |
| mriedem | but i don't know what the circular reference is | 18:19 |
| * mdbooth doesn't know how references work in json | 18:20 | |
| mdbooth | I expect if we knew that and decoded the big chunk of text above we'd discover it | 18:20 |
| mriedem | something is messed up in that change of yours | 18:21 |
| mriedem | TypeError: get_by_instance_mapping_list() got an unexpected keyword argument 'expected_addrs' | 18:22 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/446674 | 18:22 |
| mdbooth | mriedem: Yeah, I've fixed it since | 18:22 |
| *** armax has quit IRC | 18:22 | |
| mriedem | ah ok | 18:22 |
| mdbooth | It was just a typo | 18:22 |
| *** sc68cal_ has quit IRC | 18:22 | |
| mriedem | johnthetubaguy: https://review.openstack.org/#/c/446672/ | 18:22 |
| mriedem | yup | 18:22 |
| mriedem | addrs/attrs | 18:22 |
| mriedem | johnthetubaguy: ^ is the cinderclient min version bump | 18:23 |
| mdbooth | mriedem: Now *you* saw that straight away. I stared at it so long I became blind to it. | 18:23 |
| mriedem | pair programming | 18:23 |
| mriedem | the ibm execs were right! | 18:23 |
| * johnthetubaguy get mriedem some help, I hear crys for help | 18:24 | |
| *** sc68cal has joined #openstack-nova | 18:24 | |
| mriedem | we were told that spotify did everything correctly, and so we needed to do everything they did | 18:24 |
| mriedem | like, | 18:24 |
| mriedem | pair programming, squads, kanban, agile, | 18:25 |
| mriedem | work around a single table in a room | 18:25 |
| mriedem | eat together | 18:25 |
| mriedem | sleep together | 18:25 |
| mriedem | etc etc | 18:25 |
| mriedem | out pops innovation | 18:25 |
| mriedem | yada yada yada, 18 months later, i left ibm | 18:25 |
| mriedem | ") | 18:25 |
| mriedem | :) | 18:25 |
| mdbooth | mriedem: So, looking at that circular reference thing, there are a few python objects on those dicts | 18:26 |
| *** liusheng has quit IRC | 18:26 | |
| ildikov | mriedem: sounds like a long time to sleep together with the whole team :) | 18:26 |
| mriedem | ildikov: you wouldn't believe the VD | 18:26 |
| mdbooth | If the json dumper is trying to serialise them, and there's a circular reference in there | 18:26 |
| mdbooth | Then... that would be a circular reference | 18:26 |
| mdbooth | There are plenty to choose from | 18:26 |
| *** liusheng has joined #openstack-nova | 18:27 | |
| ildikov | mriedem: VD? | 18:27 |
| mdbooth | There's probably stuff in there it's just not useful to serialise, tbh | 18:27 |
| mriedem | venereal disease | 18:27 |
| mriedem | ildikov: a joke | 18:27 |
| *** esberglu_ has joined #openstack-nova | 18:27 | |
| *** esberglu_ has quit IRC | 18:27 | |
| *** esberglu_ has joined #openstack-nova | 18:28 | |
| ildikov | mriedem: I knew it was a joke, just had no idea what VD is, LOL | 18:28 |
| *** rcernin has joined #openstack-nova | 18:28 | |
| ildikov | mriedem: I could've guessed though :) | 18:28 |
| *** esberglu has quit IRC | 18:30 | |
| *** catinthe_ has joined #openstack-nova | 18:30 | |
| openstackgerrit | Sujitha proposed openstack/nova master: Add helper method to add additional data about policy rule. https://review.openstack.org/434842 | 18:30 |
| *** catinth__ has joined #openstack-nova | 18:30 | |
| *** baoli has quit IRC | 18:32 | |
| *** catintheroof has quit IRC | 18:32 | |
| *** catinthe_ has quit IRC | 18:34 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Fix functional regression/recreate test for bug 1671648 https://review.openstack.org/446685 | 18:35 |
| openstack | bug 1671648 in OpenStack Compute (nova) ocata "Instances are not rescheduled after deploy fails" [High,In progress] https://launchpad.net/bugs/1671648 - Assigned to Matt Riedemann (mriedem) | 18:35 |
| *** sneti_ has quit IRC | 18:35 | |
| *** Swami has quit IRC | 18:35 | |
| melwitt | mriedem: while I was working on making it consider two hosts while scheduling, I discovered a race. so I have put up a fix ^ | 18:35 |
| melwitt | the regression test, that is | 18:35 |
| mriedem | melwitt: ok, cool | 18:36 |
| *** cfriesen_ has joined #openstack-nova | 18:37 | |
| mriedem | i had backported the functional regression test and the retry fix to ocata already, they aren't merged, but we could hold them up for the race fix too | 18:37 |
| mriedem | in the test | 18:37 |
| *** cfriesen has quit IRC | 18:37 | |
| *** cfriesen__ has joined #openstack-nova | 18:38 | |
| *** yamahata has quit IRC | 18:38 | |
| melwitt | okay | 18:38 |
| *** Swami has joined #openstack-nova | 18:39 | |
| melwitt | mriedem: the "host" on the instance server attributes is the host from the start_service('compute', host=) but the "hypervisor_hostname" is the nodename, and the nodename for both hosts was the same "fake-mini" so that's how they got collapsed into only one host for scheduling | 18:40 |
| dansmith | and nodes bites us in the ass once again | 18:41 |
| melwitt | there's a fake.set_nodes() that lets you set the hostnames to return and you have to do it before starting each compute service fixture | 18:41 |
| *** cfriesen_ has quit IRC | 18:41 | |
| melwitt | yeah :( node ass-biters | 18:41 |
| mriedem | ah ok | 18:41 |
| mriedem | ouch | 18:41 |
| mriedem | also, can you all please clean it up please, | 18:42 |
| *** tesseract has quit IRC | 18:42 | |
| mriedem | i'd like to keep this channel classy | 18:42 |
| melwitt | after I set the nodes, the race reared its head, sometimes I got a Claim, sometimes I got a NopClaim | 18:42 |
| mriedem | alright, my kid just threw up in class so i'm going to the school, back in a bit | 18:43 |
| mriedem | probably all those stories i told here about nodes and local delete and quotas | 18:44 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: objects: Add attachment_id to BlockDeviceMapping https://review.openstack.org/437665 | 18:44 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: db: Add attachment_id to block_device_mapping https://review.openstack.org/437597 | 18:44 |
| openstackgerrit | melanie witt proposed openstack/nova master: Fix functional regression/recreate test for bug 1671648 https://review.openstack.org/446685 | 18:46 |
| openstack | bug 1671648 in OpenStack Compute (nova) ocata "Instances are not rescheduled after deploy fails" [High,In progress] https://launchpad.net/bugs/1671648 - Assigned to Matt Riedemann (mriedem) | 18:46 |
| melwitt | mriedem: hah, hope she's okay | 18:46 |
| *** alexpilotti has quit IRC | 18:47 | |
| *** amoralej is now known as amoralej|off | 18:48 | |
| *** dharinic has quit IRC | 18:50 | |
| *** sneti_ has joined #openstack-nova | 18:52 | |
| *** alexpilotti has joined #openstack-nova | 18:53 | |
| *** alexpilotti has quit IRC | 18:58 | |
| *** rcernin has quit IRC | 18:58 | |
| *** dharinic- is now known as dharinic | 19:02 | |
| *** dtp has joined #openstack-nova | 19:02 | |
| *** alexpilotti has joined #openstack-nova | 19:04 | |
| *** sc68cal has quit IRC | 19:05 | |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add service-protected-server spec https://review.openstack.org/438134 | 19:06 |
| *** fragatina has joined #openstack-nova | 19:06 | |
| *** fragatina has quit IRC | 19:07 | |
| *** fragatina has joined #openstack-nova | 19:07 | |
| *** alexpilotti has quit IRC | 19:09 | |
| *** cdent has joined #openstack-nova | 19:09 | |
| *** yamahata has joined #openstack-nova | 19:10 | |
| *** sc68cal has joined #openstack-nova | 19:11 | |
| mriedem | lyarwood: when are you doing for the day? | 19:13 |
| *** smatzek has joined #openstack-nova | 19:16 | |
| mriedem | *done | 19:18 |
| *** Jack_Iv has quit IRC | 19:18 | |
| mriedem | i'm going to add tests to your bottom change, and fix the test in the object change | 19:18 |
| *** xyang1 has joined #openstack-nova | 19:25 | |
| * cdent is suddenly thinking of diapers | 19:25 | |
| *** armax has joined #openstack-nova | 19:26 | |
| *** liverpooler has quit IRC | 19:27 | |
| *** cdent has quit IRC | 19:27 | |
| mriedem | his bottom is a mess | 19:27 |
| mriedem | don't want the rash to set in | 19:28 |
| *** liverpooler has joined #openstack-nova | 19:30 | |
| openstackgerrit | Steve Noyes proposed openstack/nova master: Throw exception if swap volume attempted on stopped server https://review.openstack.org/446708 | 19:30 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Sort CellMappingList.get_all() for safety https://review.openstack.org/443174 | 19:31 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add get_by_instance_uuids() to InstanceMappingList https://review.openstack.org/443292 | 19:31 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Clean up ClientRouter debt https://review.openstack.org/444487 | 19:31 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make server_groups determine deleted-ness from InstanceMappingList https://review.openstack.org/443293 | 19:31 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Remove Mitaka-era service version check https://review.openstack.org/442861 | 19:31 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add workaround to disable group policy check upcall https://review.openstack.org/442736 | 19:31 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: objects: Add attachment_id to BlockDeviceMapping https://review.openstack.org/437665 | 19:31 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: db: Add attachment_id to block_device_mapping https://review.openstack.org/437597 | 19:31 |
| *** dtp has quit IRC | 19:32 | |
| *** amotoki has joined #openstack-nova | 19:32 | |
| *** clenimar has quit IRC | 19:36 | |
| *** clenimar has joined #openstack-nova | 19:36 | |
| mriedem | melwitt: per your test fix, seems we should add some better descriptions for host and hypervisor_hostname in the api-ref https://developer.openstack.org/api-ref/compute/?expanded=show-server-details-detail#id25 | 19:36 |
| mriedem | host: The host name. Appears in the response for administrative users only. | 19:36 |
| mriedem | hypervisor_hostname: The hypervisor host name. Appears in the response for administrative users only. | 19:37 |
| mriedem | oh don't forget hostId: The ID of the host. | 19:37 |
| mriedem | thanks compute api ref! | 19:37 |
| melwitt | mriedem: yeah, I actually had a look at that. I wasn't sure if maybe it was intentional that they don't mention the distinction between host name and nodename | 19:37 |
| *** lpetrut has joined #openstack-nova | 19:38 | |
| melwitt | when you run for real, they the same unless you're running ironic | 19:38 |
| mriedem | i'm not sure what the point in making the description unclear would be | 19:38 |
| mriedem | do we get any security benefit from that? | 19:38 |
| melwitt | I dunno. I was thinking maybe if there was concern about exposing implementation details like host and nodename. I'm not sure it matters | 19:39 |
| *** cfriesen__ has quit IRC | 19:39 | |
| *** amotoki has quit IRC | 19:39 | |
| melwitt | hypervisor_hostname: either the host name or the Ironic node name if ironic_driver | 19:40 |
| melwitt | so, maybe that's reasonable to put there | 19:40 |
| mriedem | i think so | 19:40 |
| mriedem | if we can't actually explain wtf the fields are in our API response, our API sucks, right? | 19:40 |
| mriedem | i'm pretty sure if cdent were around i'd get a checkmark for that | 19:40 |
| melwitt | yeah. | 19:40 |
| melwitt | hehe | 19:40 |
| mriedem | i'll open a bug | 19:41 |
| melwitt | k | 19:41 |
| openstackgerrit | John Griffith proposed openstack/nova master: Add Cinder API version detection https://review.openstack.org/444465 | 19:42 |
| lyarwood | mriedem: lol, I'm back for a while now | 19:42 |
| lyarwood | mriedem: thanks for sorting the tests out, rushed things before dinner, always a mistake. | 19:42 |
| *** Swami has quit IRC | 19:47 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/delete #1: no-ops https://review.openstack.org/438119 | 19:48 |
| mriedem | melwitt: dansmith: jaypipes: sdague: jroll: see if this makes sense https://bugs.launchpad.net/nova/+bug/1673593 | 19:49 |
| openstack | Launchpad bug 1673593 in OpenStack Compute (nova) "api-ref: descriptions for the various host fields in server GET response are useless" [Low,Confirmed] | 19:49 |
| melwitt | mriedem: makes sense to me | 19:50 |
| jroll | mriedem: yeah, makes sense | 19:51 |
| jroll | they are indeed useless | 19:51 |
| mriedem | hostId: The ID of the host. | 19:51 |
| mriedem | thanks | 19:51 |
| jroll | # start the thread | 19:51 |
| jroll | thread.start() | 19:51 |
| jroll | love that stuff | 19:51 |
| mriedem | # start it up kris | 19:52 |
| mriedem | # it's what i was born to do | 19:52 |
| melwitt | lol | 19:53 |
| mriedem | *warm it up | 19:53 |
| mriedem | my mistake | 19:53 |
| *** smatzek has quit IRC | 19:53 | |
| dansmith | ooh, I was just about to catch you on that | 19:53 |
| melwitt | let's wear our jeans backwards | 19:54 |
| *** awaugama has quit IRC | 19:54 | |
| mriedem | that's wiggity wiggity wiggity whack | 19:54 |
| *** tbachman has joined #openstack-nova | 19:56 | |
| *** voelzmo has joined #openstack-nova | 20:02 | |
| *** Jeffrey4l has quit IRC | 20:06 | |
| *** Jeffrey4l has joined #openstack-nova | 20:07 | |
| *** Sukhdev has quit IRC | 20:11 | |
| *** Swami has joined #openstack-nova | 20:11 | |
| *** baoli has joined #openstack-nova | 20:11 | |
| *** baoli has quit IRC | 20:18 | |
| *** baoli has joined #openstack-nova | 20:18 | |
| *** sneti_ has quit IRC | 20:18 | |
| *** Drankis has quit IRC | 20:21 | |
| *** winston-d has joined #openstack-nova | 20:22 | |
| winston-d | dansmith: ping | 20:22 |
| *** eharney has quit IRC | 20:23 | |
| *** Sukhdev has joined #openstack-nova | 20:23 | |
| winston-d | dansmith: Hi, Dan. I recall you mentioned Nova would have a 'force-shutdown' or sth similar API/feature to ensure hypervisor is truely down during PTG. | 20:24 |
| winston-d | dansmith: Do you have any pointer to bp or spec? I quickly skim through nova-specs and couldn't find it. | 20:25 |
| *** cfriesen__ has joined #openstack-nova | 20:29 | |
| *** Swami has quit IRC | 20:30 | |
| sdague | mriedem: I think the bug looks fine, push me a patch and lets get the api-ref fixed! | 20:30 |
| *** dtp_ has joined #openstack-nova | 20:31 | |
| *** dtp_ is now known as dtp | 20:31 | |
| *** cfriesen has joined #openstack-nova | 20:34 | |
| *** cfriesen__ has quit IRC | 20:34 | |
| mriedem | sdague: alright | 20:35 |
| mriedem | sdague: while you're around, would be nice to +W this so i can include it in the backports for the related bug https://review.openstack.org/#/c/446685/ | 20:35 |
| *** crushil has quit IRC | 20:35 | |
| mriedem | retry on failed build is kind of important | 20:36 |
| sdague | mriedem: +W | 20:36 |
| sdague | mriedem: honestly, if you find a thing is wrong in api-ref, I would generally skip the bug and write the patch, and just be descriptive in the commit message. Definitely know a few of those things aren't quite right still. | 20:37 |
| mriedem | i only wrote the bug because i wasn't planning on doing a patch right now | 20:40 |
| *** crushil has joined #openstack-nova | 20:40 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: objects: Add attachment_id to BlockDeviceMapping https://review.openstack.org/437665 | 20:40 |
| *** amrith has quit IRC | 20:42 | |
| dansmith | winston-d: force-down | 20:42 |
| *** amrith has joined #openstack-nova | 20:43 | |
| dansmith | winston-d: https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/mark-host-down.html | 20:49 |
| openstackgerrit | John L. Villalovos proposed openstack/nova master: flake8: Specify 'nova' as name of app https://review.openstack.org/446736 | 20:49 |
| bauzas | FWIW, folks I'll be off tomorrow | 20:50 |
| *** crushil has quit IRC | 20:50 | |
| *** voelzmo has quit IRC | 20:51 | |
| mriedem | ok | 20:52 |
| mriedem | i'll have a small human in the house, so i might be distracted | 20:52 |
| mriedem | bauzas: did this get talked about in last week's nova meeting? https://blueprints.launchpad.net/nova/+spec/lvm-thin-pool | 20:54 |
| mriedem | and this https://blueprints.launchpad.net/nova/+spec/remove-openstack-api-directory | 20:54 |
| mriedem | logan-: edleafe: ^? | 20:55 |
| bauzas | mriedem: the first one was not discussed given logan- wasn't there | 20:57 |
| mriedem | ok | 20:57 |
| bauzas | mriedem: the second one was discussed | 20:57 |
| bauzas | mriedem: and we said possibly a specless BP | 20:57 |
| bauzas | given only for tracking | 20:57 |
| edleafe | mriedem: yeah, and I have some patches for that | 20:58 |
| mriedem | but the bp wasn't approved | 20:58 |
| edleafe | not yet :) | 20:58 |
| mriedem | i guess i can look back at the logs, would have been nice if someone could have summarized in the bp whiteboard | 20:58 |
| *** dimtruck is now known as zz_dimtruck | 20:58 | |
| bauzas | http://eavesdrop.openstack.org/meetings/nova/2017/nova.2017-03-09-14.00.log.html | 20:59 |
| bauzas | mmm, 1 min before the next one :) | 21:00 |
| bauzas | actually.. | 21:00 |
| * mriedem starts nova meeting | 21:01 | |
| *** sc68cal has left #openstack-nova | 21:03 | |
| efried | esberglu_ adreznec Could use a new +1 on https://review.openstack.org/438119 | 21:14 |
| *** smatzek has joined #openstack-nova | 21:15 | |
| *** Sukhdev has quit IRC | 21:16 | |
| esberglu_ | ack | 21:17 |
| *** zz_dimtruck is now known as dimtruck | 21:26 | |
| openstackgerrit | Octave Orgeron proposed openstack/nova-specs master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446631 | 21:26 |
| *** catinth__ has quit IRC | 21:33 | |
| *** Jeffrey4l has quit IRC | 21:34 | |
| dtp | dansmith / melwitt: can i get a new task? | 21:34 |
| * jroll O_o at this static network metadata BP | 21:35 | |
| jroll | mriedem: I don't see anything too crazy about it | 21:35 |
| jroll | from ironic POV | 21:35 |
| jroll | though I wonder if there's other ways to do it, hm | 21:36 |
| mriedem | dtp: i will have something i think once i have a spec written | 21:36 |
| dtp | ok | 21:37 |
| dansmith | mriedem: the services id thing? | 21:37 |
| mriedem | yes | 21:37 |
| mriedem | dtp: this needs a spec https://blueprints.launchpad.net/nova/+spec/service-hyper-pci-uuid-in-api | 21:37 |
| mriedem | dtp: if you can parse that and grok it, feel free to write the spec | 21:37 |
| dansmith | mriedem: cool, I just tweaked the hostapi this morning to do the thing we said it should do | 21:37 |
| mriedem | dansmith: ok | 21:37 |
| mriedem | high five | 21:37 |
| mriedem | \o\ | 21:37 |
| mriedem | /o/ | 21:37 |
| dansmith | o/* | 21:37 |
| mriedem | o/. | 21:37 |
| melwitt | .\o | 21:38 |
| mriedem | i'm just going to steal the classics from melwitt | 21:38 |
| jroll | o/' | 21:38 |
| dtp | armpit bump? | 21:38 |
| jroll | cheeeeeers | 21:38 |
| dansmith | ~~.\o/.~~ | 21:38 |
| melwitt | lol | 21:38 |
| mriedem | odor lines? | 21:38 |
| dansmith | smell lines yeah | 21:38 |
| mriedem | ha | 21:38 |
| dansmith | I am in portland after all, where showers are optional | 21:38 |
| dtp | ok mriedem, reading now | 21:39 |
| *** gouthamr has quit IRC | 21:40 | |
| *** smatzek has quit IRC | 21:40 | |
| *** karimb has joined #openstack-nova | 21:41 | |
| *** lpetrut has quit IRC | 21:41 | |
| *** tblakes has quit IRC | 21:42 | |
| dtp | what does the services table contain? | 21:46 |
| mriedem | services | 21:47 |
| dtp | hehe | 21:47 |
| dtp | like . . . keystone? | 21:47 |
| mriedem | no | 21:47 |
| mriedem | info about the actual nova services | 21:47 |
| mriedem | like nova-compute, nova-scheduler, etc | 21:47 |
| mriedem | the host/binary/topic fields | 21:48 |
| mriedem | service versoin | 21:48 |
| *** rfolco_ has quit IRC | 21:48 | |
| mriedem | enabled/disabled | 21:48 |
| mriedem | et | 21:48 |
| mriedem | *etc | 21:48 |
| mriedem | https://developer.openstack.org/api-ref/compute/#compute-services-os-services | 21:48 |
| dtp | ah. thank you | 21:48 |
| *** Jeffrey4l has joined #openstack-nova | 21:48 | |
| mriedem | heh "Lists all running Compute services for a tenant," that's wrong | 21:48 |
| mriedem | services are not per-tenant | 21:48 |
| mriedem | sdague: more api-ref fun ^ | 21:48 |
| mriedem | that should be "for a region" right? | 21:49 |
| mriedem | or a cell i guess | 21:49 |
| mriedem | but i'm not sure we want to start using the c word in the api ref yet | 21:49 |
| dansmith | not a cell | 21:49 |
| dansmith | yeah, don't say that | 21:49 |
| mriedem | true, | 21:49 |
| mriedem | api isn't in the cell | 21:49 |
| dansmith | say for a deployment or region or something | 21:49 |
| mriedem | yeah | 21:49 |
| *** Jeffrey4l has quit IRC | 21:50 | |
| melwitt | dtp: to see how it's used, look at nova/db/sqlalchemy/models.py at Service to see the data model, then look at nova/objects/service.py to see the object API and you can sort of trace how it's used from using that as a starting point. also look at nova/service.py | 21:50 |
| *** Jeffrey4l has joined #openstack-nova | 21:50 | |
| dtp | ok, thanks | 21:51 |
| mriedem | dansmith: guh, we need to update this at some point too https://docs.openstack.org/ops-guide/arch-scaling.html#cells-and-regions | 21:51 |
| dansmith | hmm, yeah | 21:52 |
| *** felipemonteiro_ has quit IRC | 21:56 | |
| *** dtp is now known as dtp-afk | 21:58 | |
| mriedem | dansmith: i've opened a docs bug https://bugs.launchpad.net/openstack-manuals/+bug/1673616 | 21:58 |
| openstack | Launchpad bug 1673616 in openstack-manuals "Scaling in Operations Guide - cells section needs to be updated" [Undecided,New] | 21:58 |
| mriedem | dansmith: you might want to add/update/fix whatever i put in the description | 21:58 |
| dansmith | okay | 22:00 |
| dansmith | looks alright to me at first skim but I'll think on it a bit | 22:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: fix description in os-services https://review.openstack.org/446757 | 22:01 |
| *** aysyd has quit IRC | 22:05 | |
| *** mdrabe has quit IRC | 22:05 | |
| *** xyang1 has quit IRC | 22:08 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Improve descriptions for hostId, host, and hypervisor_hostname https://review.openstack.org/446761 | 22:18 |
| *** liangy has quit IRC | 22:18 | |
| melwitt | mriedem: ^ patch for the host* api-ref stuff | 22:18 |
| openstackgerrit | Octave Orgeron proposed openstack/nova-specs master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446631 | 22:26 |
| *** alexpilotti has joined #openstack-nova | 22:27 | |
| *** alexpilotti has quit IRC | 22:32 | |
| mriedem | melwitt: thanks, comments inline | 22:33 |
| *** mriedem1 has joined #openstack-nova | 22:36 | |
| *** mriedem has quit IRC | 22:37 | |
| *** mriedem1 is now known as mriedem | 22:41 | |
| *** esberglu_ has quit IRC | 22:43 | |
| cfriesen | is there a known issue in newton where pre_live_migration() can hit an RPC timeout waiting for a really big glance image to download to the destination? | 22:45 |
| *** karimb has quit IRC | 22:46 | |
| *** ijw has quit IRC | 22:47 | |
| *** ijw has joined #openstack-nova | 22:48 | |
| *** ijw has quit IRC | 22:48 | |
| *** liangy has joined #openstack-nova | 22:48 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446777 | 22:56 |
| *** ssurana has joined #openstack-nova | 23:02 | |
| *** jamielennox is now known as jamielennox|away | 23:03 | |
| *** iceyao has joined #openstack-nova | 23:03 | |
| *** jamielennox|away is now known as jamielennox | 23:05 | |
| *** iceyao has quit IRC | 23:09 | |
| mriedem | easy bug fix https://bugs.launchpad.net/nova/+bug/1673628 | 23:10 |
| openstack | Launchpad bug 1673628 in OpenStack Compute (nova) "api-ref: 'tags' field is not in response parameters docs for "GET /servers/{server_id}"" [Medium,Confirmed] | 23:10 |
| mriedem | cfriesen: that's not a newton specific limitation | 23:11 |
| mriedem | that's a latent issue forever | 23:11 |
| mriedem | cfriesen: see https://review.openstack.org/#/c/419662/1 | 23:11 |
| *** dimtruck is now known as zz_dimtruck | 23:11 | |
| *** liangy has quit IRC | 23:11 | |
| *** zz_dimtruck is now known as dimtruck | 23:11 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Repropose tag-instance-when-boot https://review.openstack.org/415315 | 23:13 |
| mriedem | Kevin_Zheng: are you ok with the updates i made to ^? | 23:13 |
| *** Jeffrey4l has quit IRC | 23:14 | |
| *** alexpilotti has joined #openstack-nova | 23:15 | |
| *** alexpilotti has quit IRC | 23:15 | |
| *** alexpilotti has joined #openstack-nova | 23:15 | |
| mriedem | melwitt: i think your func test change might be causing a race issue with global state | 23:18 |
| mriedem | yeah you need to use fake.restore_nodes | 23:20 |
| *** yingjun has quit IRC | 23:20 | |
| *** dimtruck is now known as zz_dimtruck | 23:21 | |
| *** sbezverk has joined #openstack-nova | 23:22 | |
| sbezverk | hello, I need to check liveness of nova placement api process, could you recommen any methods to check not just that the socket is opened but also that the process is alive and sane. | 23:24 |
| *** baoli has quit IRC | 23:24 | |
| mriedem | sbezverk: make a curl request to / ? | 23:24 |
| mriedem | that's testing much more than just the process is running | 23:24 |
| sbezverk | mriedem: would it not just check apachie liveness? | 23:25 |
| mriedem | making a curl request to the actual endpoint's root would provide the versions available, which means it's up and running and accepting requests | 23:25 |
| mriedem | and serving a response | 23:25 |
| mriedem | i don't know if a token is required for GET / on the placement endpoint though | 23:25 |
| sbezverk | mriedem: cool, thank you very much. I was not sure if curling for root is sufficient | 23:26 |
| mriedem | yup that will get you the versions available | 23:26 |
| *** Jeffrey4l has joined #openstack-nova | 23:29 | |
| *** ijw has joined #openstack-nova | 23:33 | |
| *** ijw has quit IRC | 23:33 | |
| *** ijw has joined #openstack-nova | 23:33 | |
| *** masber has quit IRC | 23:36 | |
| *** amotoki has joined #openstack-nova | 23:36 | |
| *** mlavalle has quit IRC | 23:39 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add api-ref for filter/sort whitelist https://review.openstack.org/421760 | 23:39 |
| sbezverk | mriedem: any known issue with placement in ocata you are aware? | 23:39 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix functional regression/recreate test for bug 1671648 https://review.openstack.org/446685 | 23:39 |
| openstack | bug 1671648 in OpenStack Compute (nova) ocata "Instances are not rescheduled after deploy fails" [High,In progress] https://launchpad.net/bugs/1671648 - Assigned to Matt Riedemann (mriedem) | 23:39 |
| mriedem | sbezverk: nothing major for placement that i can think of | 23:40 |
| mriedem | sbezverk: we did release some fixes for ocata in 15.0.1 earlier in the week, | 23:40 |
| mriedem | so if you're upgrading, i'd use 15.0.1 | 23:40 |
| *** amotoki has quit IRC | 23:41 | |
| sbezverk | mriedem: I brining up at the gate ocata images in kolla-kubernetes project | 23:41 |
| sbezverk | what worked with stable/newton seems broken in ocata, but it might be our issue too.. | 23:41 |
| *** Sukhdev has joined #openstack-nova | 23:41 | |
| mriedem | sbezverk: i guess i'd need a more specific example of what's not working | 23:42 |
| *** hongbin has quit IRC | 23:43 | |
| Sukhdev | dear Nova folks, after fresh install of devstack, Nova continue to return "No Valid host found" error - can somebody provide some wisdom, please? See the details here - http://paste.openstack.org/show/603034/ | 23:43 |
| *** zioproto has joined #openstack-nova | 23:44 | |
| sbezverk | mriedem: :) glad you asked. When I start instance I get " Placement API service is not responding." in nova scheduler log | 23:44 |
| mriedem | sbezverk: are you starting the placement service before the nova-scheduler service? | 23:44 |
| sbezverk | mriedem: it looks like connectivity issue but nothing change other than just version of images | 23:45 |
| mriedem | is there a "placement" endpoint in the service catalog? | 23:45 |
| *** dave-mccowan has quit IRC | 23:45 | |
| *** marst_ has quit IRC | 23:45 | |
| sbezverk | mriedem: yep, service catalog shows placement entries | 23:45 |
| mriedem | is nova.conf for the scheduler configured to use placement in the [placement] section, and are the credentials in there correct? | 23:45 |
| sbezverk | mriedem: all valid points, but as I said nothing changed in the gate job other than image version | 23:46 |
| mriedem | the scheduler didn't use placement in newton, | 23:46 |
| mriedem | so that's a main different with ocata | 23:46 |
| sbezverk | mriedem: was there any change in endpoint registrations between newton and ocata? | 23:46 |
| mriedem | in newton, only the nova-compute service tried connecting to the placement service | 23:46 |
| mriedem | no there was no change in endpoint registration | 23:46 |
| sbezverk | mriedem: AHHH I did not know that | 23:47 |
| mriedem | i'd compare the nova.conf [placement] sections between your nova-compute and nova-scheduler services | 23:47 |
| mriedem | make sure that whatever you did to make it work for nova-compute, that you do that for nova-scheduler too | 23:47 |
| mriedem | sbezverk: https://docs.openstack.org/developer/nova/placement.html#upgrade-notes in case you haven't seen that yet | 23:47 |
| sbezverk | mriedem: if you have 2 minutes I can get it, would be great if you cold take a look at it | 23:48 |
| mriedem | oh boy | 23:48 |
| mriedem | :) | 23:48 |
| mriedem | Sukhdev: is the nova-compute service running and listed as up when you run "nova service-list"? | 23:48 |
| Kevin_Zheng | mriedem: yeah thanks | 23:49 |
| mriedem | Kevin_Zheng: yeah as in you agree with the changes, or yeah as in you will look? | 23:49 |
| Sukhdev | mriedem : yes - see here - http://paste.openstack.org/show/603035/ | 23:50 |
| Kevin_Zheng | agree：） | 23:50 |
| sbezverk | mriedem: http://paste.openstack.org/show/603036/ | 23:51 |
| sbezverk | mriedem: look identical to me :( | 23:51 |
| Kevin_Zheng | thanks alot | 23:52 |
| Sukhdev | mriedem : anything else that I should look for? BTW, I am running the latest code (did git pull 5-10 days ago) | 23:53 |
| *** abalutoiu has joined #openstack-nova | 23:53 | |
| *** marst has joined #openstack-nova | 23:55 | |
| mriedem | Kevin_Zheng: yw | 23:56 |
| mriedem | Sukhdev: check the nova-compute logs for errors | 23:56 |
| mriedem | sbezverk: those look the same to me too | 23:57 |
| mriedem | sbezverk: the services all run in individual containers right? | 23:57 |
| mriedem | is there anything in the deployment tooling that has to do anything specific for the compute service wrt placement that the scheduler container isn't doing? | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!