Monday, 2016-10-03

*** JaiRoc_ has joined #openstack-meeting-cp01:28
*** JaiRoc_ has left #openstack-meeting-cp01:29
*** sdake has quit IRC02:29
*** sdake has joined #openstack-meeting-cp03:21
*** sdake_ has joined #openstack-meeting-cp03:38
*** sdake has quit IRC03:40
*** coolsvap has joined #openstack-meeting-cp03:41
*** gouthamr has quit IRC04:56
*** prateek_ has joined #openstack-meeting-cp05:26
*** prateek_ has quit IRC05:31
*** mugsie has quit IRC05:50
*** mugsie has joined #openstack-meeting-cp05:51
*** sdake has joined #openstack-meeting-cp06:22
*** sdake_ has quit IRC06:23
*** sdake_ has joined #openstack-meeting-cp07:14
*** sdake_ has quit IRC07:14
*** sdake has quit IRC07:15
*** sdake has joined #openstack-meeting-cp07:26
*** zigo has quit IRC07:39
*** zigo has joined #openstack-meeting-cp07:42
*** zigo is now known as Guest2475607:42
*** sdake has quit IRC07:58
*** Guest24756 is now known as zigo08:54
*** prateek has joined #openstack-meeting-cp09:26
*** sdake has joined #openstack-meeting-cp09:44
*** rarcea has joined #openstack-meeting-cp10:50
*** rarcea has quit IRC10:51
*** rarcea has joined #openstack-meeting-cp10:51
*** sdake_ has joined #openstack-meeting-cp11:12
*** sdake has quit IRC11:14
*** jrollen has joined #openstack-meeting-cp11:36
*** gouthamr has joined #openstack-meeting-cp11:41
*** sdake_ is now known as sdake11:52
*** jrollen has quit IRC12:18
*** jrollen has joined #openstack-meeting-cp12:41
*** jrollen has quit IRC12:45
*** jrollen has joined #openstack-meeting-cp12:45
*** jroll has quit IRC12:53
*** jrollen is now known as jroll12:55
*** prateek has quit IRC13:03
*** xyang1 has joined #openstack-meeting-cp14:39
*** rhe00 has left #openstack-meeting-cp15:37
*** sdake has quit IRC16:30
*** sdake has joined #openstack-meeting-cp16:34
*** piet_ has joined #openstack-meeting-cp16:54
*** mriedem has joined #openstack-meeting-cp16:58
ildikov#startmeeting cinder-nova-api-changes17:00
openstackMeeting started Mon Oct  3 17:00:00 2016 UTC and is due to finish in 60 minutes.  The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot.17:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:00
*** openstack changes topic to " (Meeting topic: cinder-nova-api-changes)"17:00
openstackThe meeting name has been set to 'cinder_nova_api_changes'17:00
ildikovscottda ildikov DuncanT ameade cFouts johnthetubaguy jaypipes takashin alaski e0ne jgriffith tbarron andrearosa hemna erlon mriedem gouthamr ebalduf patrickeast smcginnis diablo_rojo gsilvis  xyang1 raj_singh17:00
mriedemo/17:00
cFoutshi17:00
scottdahi17:00
jgriffithpresent17:00
ildikovhi All :)17:01
mriedemi have a quick update once we're done waiting for people17:01
ildikovshould I do a quick ping on the Cinder channel?17:02
mriedemshrug17:02
ildikovok, pinged a few guys17:03
ildikovmriedem: I think we can move to your update17:03
mriedemok, we have a working nova-initiated swap volume test in tempest now http://logs.openstack.org/73/374373/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cb37dbc/console.html#_2016-10-03_16_12_03_51875717:03
mriedemnot merged yet17:04
mriedemstack starts here: https://review.openstack.org/#/c/374373/17:04
ildikovmriedem: wow, that looks great!!!17:04
hemnadoink17:04
jgriffithsweet17:04
hemnaooh nice17:05
mriedemit's gotten decent review in tempest though so i expect that to be merged this week17:05
ildikovmriedem: that sounds pretty good17:06
ildikovscottda: mriedem: did we have other tests in mind that we would need?17:06
mriedemthe only other thing i had was i did the thorough review on removing volume_api.check_attach in nova https://review.openstack.org/#/c/335358/17:06
mriedemildikov: scottda's tempest change for retype doesn't trigger volume swap in nova17:06
ildikovI mean in general, not for the new API17:06
mriedemi'm not really sure why17:06
mriedembut i don't know much about retype17:07
scottdaildikov: Yes, we need to run tempest test on 2-node cinder to call swap_volume17:07
scottdaretype from lvm->lvm on the same node won't call swap_volume17:07
scottdaJust some "smarts" in lvm driver code.17:07
ildikovscottda: ok, I'll add notes to the etherpad17:07
mriedemscottda: will it be a retype test?17:08
mriedemor volume migration?17:08
scottdaBut I *think* we can make it work for 2-node. Sounds like it would duplicate coverage of swap_volume. So useful, but not imperitive17:08
scottdamriedem: Yes, it would be cinder retype-with-migration17:08
scottdaso both17:08
mriedemok17:08
mriedemnot a total swap_volume duplicate,17:08
mriedembecause the test above that i'm working on is nova-api initiated, not cinder17:09
mriedemso there is less back and forth17:09
scottdaok, cool. So still usefull for call to novaclient, etc.17:09
mriedemyes17:09
ildikovmriedem: I've just gotten there to check your comments17:09
ildikovI don't know the answer to all questions you raised17:10
*** piet_ has quit IRC17:10
ildikovlike what happens with unreserve_volume if we never called reserve, but end up with the cleanup path17:10
*** piet_ has joined #openstack-meeting-cp17:11
ildikovdoes unreserve do volume status check?17:11
ildikovscottda: hemna: ^^17:11
mriedemi checked17:11
mriedemand it does17:11
mriedemso i think that would fail17:11
mriedemwith a conflict17:11
ildikov:(17:12
hemnacinder volume has to be in-use17:12
hemnahttps://github.com/openstack/cinder/blob/master/cinder/volume/api.py#L635-L63817:12
hemnain-use or available I think by the looks of that17:12
ildikovhemna: what if something goes wrong after calling reserve and then we call unreserve to clean things up?17:13
ildikovas reserve moves it to 'attaching' IIRC17:13
mriedemif it allows available,17:13
hemnameaning the volume is in error state before calling unreserve ?17:13
mriedemthen it seems unreserve would be fine if you never called reserve17:13
scottdaexpected status must be "attaching"17:13
hemnaif an error happens between reserve and unreserve....then...it's in error state and it's borked.17:13
hemnascottda, oh yah you are right.  I didn't see line 63217:14
ildikovhemna: we are talking more about issues on Nova side because of which we want to undo the steps like volume attach17:14
mriedemyeah https://github.com/openstack/cinder/blob/master/cinder/volume/api.py#L63217:14
hemnamy bad17:14
scottdaAnd that conditional update stuff is specifically to eliminate races. So it cannot change on the cinder side.17:15
mriedemso the reason ildikov is asking,17:16
mriedemis because in this change https://review.openstack.org/#/c/335358/17/nova/compute/api.py17:16
mriedembased on the version of the compute, the nova-api may or may not reserve the volume17:16
mriedemon the compute it's unreserving the volume https://review.openstack.org/#/c/335358/17/nova/compute/manager.py17:16
ildikovthe history is that we identified one place where reserve_volume was not called17:16
ildikovand by removing check_attach it now matters17:17
mriedemin case _prep_block_device fails17:17
ildikovor matters more and we're now trying to cover the failure scenarios for cleanup, ie unreserve17:17
mriedemmy question is more clearly asked here https://review.openstack.org/#/c/335358/17/nova/conductor/manager.py17:17
hemnaso looking at the cinder api call17:18
hemnaif it's not in attaching, you'll get an error17:18
mriedembtw, is that a 400 or 409?17:19
hemnaI'll try and track that down17:19
hemnait's not obvious17:20
scottdaI'd like to make sure that with the new API we're not creating all these exceptional cases that are all shoved into one API call.17:20
ildikovmriedem: also have you checked the other calls of _check_attach_and_reserve_volume regarding unreserve? I kinda thought those are covered, but surely haven't checked all scenarios17:20
mriedemildikov: no that's why i asked in https://review.openstack.org/#/c/335358/17/nova/compute/api.py@339417:21
ildikovok, I'll try to check then, I read your comment more like you're leaning towards it's not done17:22
mriedemildikov: i believe it's because with attach_volume in the compute api,17:22
*** tongli has joined #openstack-meeting-cp17:22
mriedemit reserves the volume in the api, then casts to the compute to attach,17:23
mriedemand if the attach on the compute fails, the compute unreserves17:23
ildikovI can fix additional occurrences if they're missing, but it would be pretty odd17:23
mriedemhowever, if the cast to the compute failed for some reason, we'd leave the volume orphaned and you'd have to reset the state to get it back17:23
ildikovyeap, that's right17:23
hemnaanyway, I can't make sense of the response to the conditional_update failure17:23
hemnaand how the wsgi converts that into a failure17:24
hemna400 vs 40917:24
mriedemildikov: this attempts to fix that https://review.openstack.org/#/c/290793/17:24
mriedemhemna: yeah i couldn't either17:24
* hemna isn't happy about it17:24
scottdagorka would have the answer, I think. But it should't require a genius to figure it out.17:24
hemnascottda, +117:25
hemnait's stupid that it's this hard to figure it out17:25
ildikovmriedem: ah ok, I got it fully now what you mean, but that's kinda independent from the check_attach removal and it's a generic how to handle reserve/unreserve17:25
mriedemildikov: yes17:26
mriedemjust more garbage that needs cleaning up17:26
hemnahttps://github.com/openstack/cinder/blob/master/cinder/api/openstack/wsgi.py#L60017:27
hemnamaybe that ?17:27
hemnadunno17:27
mriedemok, so anyway, ildikov's check_attach patch needs some work17:30
mriedemjgriffith: updates on your changes?17:30
ildikovmriedem: as I see that patch you sent is taken care of, but maybe needs more eyes to figure things out17:30
jgriffithmriedem: yeah, I pushed another update yesterday...17:30
jgriffithmriedem: so there's some things to be fixed with migrate and unshelve updating correctly17:31
ildikovmriedem: yeap, tnx for the review, I'll answer/fix this week17:31
jgriffithmriedem: I also started working on the idea of ack'ing the create/terminate connection phases as an option17:31
jgriffithmriedem: that interestingly started to make things look a lot like todays intialize--->attach :(17:32
mriedemildikov: _attach_volume_shelved_offloaded is also busted, it doesn't unreserve the volume in case attach fails17:32
jgriffithmriedem: so  I started an experiment to just overhaul/improve that type of flow (better names, not so many optional args) that uses attachment_ids and stores the connectors17:32
ildikovmriedem: is there a bug report for that? we might track these issues, just to not get lost in this17:33
scottdamaybe we'd be better off with a cleanup_failed_attach API to unreserve regardless of state.17:33
jgriffithmriedem: scottda are we talking about the same thing still?17:34
jgriffithmriedem: scottda or is this another topic that you're discussion with ildikov ?17:34
jgriffithdiscussing17:34
scottdaWe were earlier talking about cleaning up failed attaches...17:34
scottdaBut the call to unreserve only works if state is "attaching". So there were issues with ildikov 's patches to remove calls to reserve17:35
jgriffithok, carry on then17:35
*** smcginnis has joined #openstack-meeting-cp17:35
* jgriffith goes back to what he was doing17:35
mriedemildikov: i'll open a bug17:36
jgriffithscottda: FWIW you could just change that :)17:36
scottdajgriffith: Yeah, but the conditional update stuff to eliminate races would get busted.17:36
ildikovjgriffith: we finished with that, just mriedem brought up one more example of the issue :)17:36
scottdajgriffith: Anyway, didn't want to drag us back. You should carry on with your update.17:36
jgriffithscottda: by allowing it to unreserve an "error-attaching"?  I don't see how17:36
mriedemjgriffith: have you had a clean ci run yet?17:36
jgriffithmriedem: nope, down to eight failures17:37
mriedemat least for the attach flow17:37
mriedemok, shelve is all sorts of weird17:37
mriedemand not well tested probably17:37
jgriffithmriedem: I've discovered that :)17:37
jgriffithmriedem: there's something kinda funky when it calls os-brick17:37
jgriffithwill get to that today17:37
mriedemyeah, 'save all of my vm state in the db and take care of it, but destroy my guest and don't charge me until i want it back'17:37
mriedemsuper pet scenario17:38
jgriffithmriedem: yeah!17:38
jgriffithmriedem: frankly I've come to hate the *feature* :)17:38
mriedemas have several of us17:38
jgriffithmriedem: I particularly hate the concept of attaching a volume to a shelved instnace17:38
jgriffithinstance even :)17:38
mriedemyes, that's what i was just looking at17:38
mriedemit's not attached until unshelve17:38
*** sdake has quit IRC17:39
jgriffithmriedem: well in reality correct, but from Cinder's point of view :)17:39
mriedemyeah17:39
mriedemi'm not sure how well we handle detaching those when we delete a shelved offloaded instance either17:39
jgriffithmriedem: anyway, I got most of it all worked out, just something in brick maybe privsep was puking17:39
mriedembecause i think that assumes terminate_connection will work17:39
jgriffithmriedem: so that part is actually covered which is nice17:39
jgriffithmriedem: it does a cleanup of everything in the BDM talbe17:40
jgriffithtable17:40
jgriffithmriedem: oh that part :)17:40
jgriffithmriedem: so my patch added some things, because we were cheating17:40
jgriffithmriedem: my patch has a no_connect flag in the connector17:40
jgriffithmriedem: so now nova and cinder both know that it's not actually being used yet17:41
mriedemok yeah i think we might have fixed this in the api anyway17:41
mriedemyeah _get_stashed_volume_connector should return None17:41
mriedemso we don't attempt to terminate the connection17:41
jgriffithmriedem: exactlys17:41
jgriffithmriedem: anyway, I'm working on an alternate proposal17:42
jgriffithmriedem: I want to post this today or tomorrow and get some folks to weigh in on both of these options17:42
mriedemok17:42
jgriffithmriedem: see what folks think is going to be best long term17:42
ildikovjgriffith: that sounds good, thanks17:43
*** sdake has joined #openstack-meeting-cp17:44
ildikovjgriffith: when I checked your spec the last time I saw comments from johnthetubaguy17:44
jgriffithildikov: yes, the addition of a confirmation that nova finished the connection17:45
jgriffithildikov: My latest update added an acknowlodge option to the create_attachment call17:45
ildikovjgriffith: yeap, we touched on that briefly today too17:46
jgriffithildikov: so you'd leave the state in attaching until Nova comes back and says "finalize" it17:46
ildikovjgriffith: and if Nova never comes back it remains in attaching state?17:47
mriedemisn't that basically what we have today with os-reserve and os-attach?17:47
mriedemos-reserve reserves the volume for attach, and os-attach changes to in-use when we're done17:47
smcginnismriedem: We kind of realized that this morning.17:48
smcginnisOr at least I did.17:48
ildikovsmcginnis: kinda same here, I didn't put those together in a flow until now17:49
mriedemyou need my expert 3rd party input17:50
mriedem:P17:50
scottdaI think johnthetubaguy had pointed out a race last week...17:50
smcginnisBut this would still clean up what we have now.17:50
scottdawhere the physical attach to the nova host failed.17:50
scottdaAnd in the shared-attach situation the cleanup would kill a new attachment that was shared to the same host.17:50
scottdaAnd that's why he thought this Ack would be useful.17:51
ildikovmriedem: lol :) that we always need :)17:51
ildikovscottda: shared-attach as multi-attach?17:53
scottdaildikov: no, for those drivers that share a connection for >1 attach to the same host17:53
scottdaso, multi-attach is required.17:54
ildikovscottda: ah, right, got it17:54
scottdaBut it's specific to drivers17:54
jgriffithsorry, bot pulled away :)17:54
jgriffithbut looks like you all have it17:54
scottdaIf I had an intern, I would request that they write out the entire state machine for volume attach, including all the wierd corner cases.17:54
ildikovjgriffith: slowly getting to the same page :)17:55
jgriffithscottda: I've already done that :)17:55
jgriffithscottda: the only *weird* cases are customizations in the drivers17:55
jgriffiththis stuff really isn't that hard, until you have 70+ backends to figure out17:56
scottdajgriffith: By weird I also mean shelve_offload and nova live_migration17:56
ildikovscottda: there are internship programs in OpenStack, so you can have one :)17:56
jgriffithscottda: those aren't really that weird in the old sense though, other than the concept17:56
scottdaand also drivers that share connections to the same compute host. And multi-attach17:56
jgriffithscottda: right... and that's a driver/backend complication17:57
ildikovsmcginnis: mriedem: BTW before we run out of time completely, I saw on the cross-project etherpad that our topic is a bit too specific for that track17:58
*** docaedo has quit IRC17:58
smcginnis:[17:58
scottdaildikov: Yes. It seems in the past that they prefer we do a cinder-nova sync for stuff like this.17:59
scottdaJust 2 projects doesn't seem to count for cross-project17:59
smcginnismriedem: I need to finalize the Cinder sessions yet. Do you have enough to dedicate one to cinder-nova, or should I look at doing that?17:59
ildikovsmcginnis: mriedem: I saw overlaps between the Nova and Cinder sessions, so I would guess we can pick one17:59
mriedemsmcginnis: we have plenty of room18:00
mriedemsmcginnis: i already penciled in a nova/cinder session friday morning18:00
mriedemin the nova room18:00
smcginnismriedem: Awesome, thanks.18:00
ildikovjgriffith: I would guess that with a bit simpler and generic API the specifics could be handled by the drivers18:00
jgriffithildikov: +118:00
mriedemsmcginnis: https://etherpad.openstack.org/p/ocata-nova-summit-ideas18:01
ildikovmriedem: sounds great18:01
*** sdake has quit IRC18:01
ildikovmriedem: tnx!!!18:01
scottdasmcginnis: I think we wanted a cinder session for the new API that went before syncing with Nova, didn't we?18:01
ildikovscottda: I think that would be beneficial18:01
mriedemwe're over time,18:01
smcginnisscottda: Yeah, we had discussed that. Since that is Friday morning, shouldn't be a problem to fit it in somewhere.18:01
mriedembut i'd like to come into the summit with, this is the cinder api(s), and how nova is going to use them18:01
ildikovscottda: I felt earlier that there are still Cinder specific items where we don't have an agreement18:01
mriedemso we can get consensus in person that those are the right things18:02
ildikovmriedem: huge +118:02
ildikovmriedem: do we need any info sharing before the summit with at least a rough plan?18:03
ildikovmriedem: just that people have a basic idea of what we're planning to do and we could focus a bit more about some level details18:03
mriedemumm18:03
ildikovmriedem:18:03
mriedemyeah probably18:04
ildikovjust an idea/question18:04
mriedemi don't want to come into the summit blind18:04
mriedemcould we do a hangout next week?18:04
ildikovmriedem: sure!18:04
mriedembasically from a nova pov i'd like the cinder team saying, yup we're on board with the proposed api and think it will work and we understand why18:04
scottdamriedem: yes to hangout18:05
mriedemif there are warts or unknowns we should talk those through in a hangout18:05
smcginnisjgriffith: Think you'd be in good shape to discuss patches in a hangout next week?18:05
scottdamriedem: And we have  been prodding the cinder team on this. We can put something on this week's agenda as well.18:05
jgriffithsure18:05
ildikovmriedem: would that be a Cinder-Nova team Hangout? or this team and we might stream it to a broader audience?18:06
smcginnisI would think this team.18:06
smcginnisTo start18:06
ildikovok, we will figure out the info sharing later then18:06
ildikovI can either put up a Doodle poll or we re-use the meeting slot for a Hangout call18:07
mriedemlet's just use this slot i think18:07
mriedemif it works for people18:07
smcginnisThat should work for me.18:07
ildikovI'm happy to do both, but reusing the slot is easier than finding another one that works for mostly everyone18:08
scottdanow is good18:08
ildikovthen I'll just post a Hangout link here at the meeting time next week18:08
scottdaor an hour ago, to be exact18:08
ildikovscottda: 167 hours later to be super exact :)18:09
ildikovis there anything else that we should discuss today?18:10
ildikovok, I tae it as a no :)18:11
ildikovscottda: let's try to have this topic on the Cinder team's meeting agenda to get the team to the same page and move towards the inter-project agreements18:12
ildikovand next Monday 1700UTC we will have a Hangouts call as opposed to the regular IRC meeting18:12
ildikovthanks all!!!18:12
ildikov#endmeeting18:13
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings"18:13
openstackMeeting ended Mon Oct  3 18:13:11 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)18:13
openstackMinutes:        http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2016/cinder_nova_api_changes.2016-10-03-17.00.html18:13
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2016/cinder_nova_api_changes.2016-10-03-17.00.txt18:13
openstackLog:            http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2016/cinder_nova_api_changes.2016-10-03-17.00.log.html18:13
*** docaedo has joined #openstack-meeting-cp18:33
*** harlowja has quit IRC18:38
*** uxdanielle has quit IRC18:39
*** harlowja has joined #openstack-meeting-cp18:39
*** rarcea has quit IRC19:19
*** sdake has joined #openstack-meeting-cp19:22
*** david-lyle has joined #openstack-meeting-cp19:47
*** piet_ has quit IRC19:53
*** sdake has joined #openstack-meeting-cp20:05
*** david-lyle has quit IRC20:30
*** david-lyle has joined #openstack-meeting-cp20:35
*** uxdanielle has joined #openstack-meeting-cp20:44
*** david-lyle has quit IRC20:44
*** Rockyg has joined #openstack-meeting-cp20:59
*** piet_ has joined #openstack-meeting-cp21:17
*** david-lyle has joined #openstack-meeting-cp21:19
*** piet_ has quit IRC21:35
*** hpe-hj has joined #openstack-meeting-cp21:40
*** hj-hpe has quit IRC21:42
*** piet_ has joined #openstack-meeting-cp21:45
*** gouthamr has quit IRC21:49
*** piet_ has quit IRC21:52
*** xyang1 has quit IRC22:00
*** piet_ has joined #openstack-meeting-cp22:01
*** piet_ has quit IRC22:04
*** piet_ has joined #openstack-meeting-cp22:04
*** david-lyle has quit IRC22:09
*** mriedem has quit IRC22:13
*** piet_ has quit IRC22:15
*** sdake has quit IRC22:16
*** sdake has joined #openstack-meeting-cp22:33
*** david-lyle has joined #openstack-meeting-cp22:36
*** tongli has quit IRC22:57
*** markvoelker has quit IRC23:29
*** david-lyle has quit IRC23:44

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!