Monday, 2016-12-19

*** ducttape_ has joined #openstack-meeting-cp00:08
*** ducttape_ has quit IRC00:52
*** ducttape_ has joined #openstack-meeting-cp01:04
*** ducttape_ has quit IRC01:15
*** ducttape_ has joined #openstack-meeting-cp01:29
*** ducttape_ has quit IRC01:50
*** ducttape_ has joined #openstack-meeting-cp02:12
*** ducttape_ has quit IRC02:37
*** ducttape_ has joined #openstack-meeting-cp02:37
*** ducttape_ has quit IRC02:52
*** gouthamr has joined #openstack-meeting-cp02:56
*** ediardo has quit IRC03:09
*** gouthamr has quit IRC03:11
*** ediardo has joined #openstack-meeting-cp03:11
*** noama has quit IRC03:12
*** noama has joined #openstack-meeting-cp03:12
*** ducttape_ has joined #openstack-meeting-cp03:15
*** ducttape_ has quit IRC03:26
*** coolsvap has joined #openstack-meeting-cp03:59
*** ducttape_ has joined #openstack-meeting-cp04:26
*** ducttape_ has quit IRC04:31
*** prateek has joined #openstack-meeting-cp04:49
*** ducttape_ has joined #openstack-meeting-cp05:00
*** ducttape_ has quit IRC05:39
*** ducttape_ has joined #openstack-meeting-cp06:57
*** ducttape_ has quit IRC07:02
*** ducttape_ has joined #openstack-meeting-cp08:28
*** MarkBaker has quit IRC08:30
*** ducttape_ has quit IRC08:33
*** noama has quit IRC08:48
*** MarkBaker has joined #openstack-meeting-cp09:13
*** ducttape_ has joined #openstack-meeting-cp09:58
*** ducttape_ has quit IRC10:03
*** MarkBaker has quit IRC10:05
*** MarkBaker has joined #openstack-meeting-cp10:07
*** MarkBaker has quit IRC10:17
*** ducttape_ has joined #openstack-meeting-cp11:03
*** MarkBaker has joined #openstack-meeting-cp11:16
*** ducttape_ has quit IRC11:21
*** cebreidian has joined #openstack-meeting-cp11:29
*** ducttape_ has joined #openstack-meeting-cp11:33
*** sdague has joined #openstack-meeting-cp11:33
*** ducttape_ has quit IRC11:49
*** prateek has quit IRC12:24
*** ducttape_ has joined #openstack-meeting-cp12:49
*** ducttape_ has quit IRC12:54
*** lamt has quit IRC13:12
*** gouthamr has joined #openstack-meeting-cp13:26
*** prateek has joined #openstack-meeting-cp13:31
*** noama has joined #openstack-meeting-cp13:37
*** noama has left #openstack-meeting-cp13:39
*** ducttape_ has joined #openstack-meeting-cp13:50
*** ducttape_ has quit IRC13:55
*** xyang1 has joined #openstack-meeting-cp14:11
*** lamt has joined #openstack-meeting-cp14:11
*** sdague_ has joined #openstack-meeting-cp14:21
*** sdague has quit IRC14:24
*** sdague_ is now known as sdague14:27
*** ducttape_ has joined #openstack-meeting-cp14:32
*** ducttape_ has quit IRC14:42
*** prateek has quit IRC14:46
*** prateek has joined #openstack-meeting-cp14:51
*** ducttape_ has joined #openstack-meeting-cp15:17
*** jaugustine has joined #openstack-meeting-cp15:22
*** prateek has quit IRC15:44
*** TheJulia has joined #openstack-meeting-cp15:59
*** _ducttape_ has joined #openstack-meeting-cp16:00
*** ducttape_ has quit IRC16:04
*** sdague has quit IRC16:06
*** jlopezgu has quit IRC16:07
*** jlopezgu has joined #openstack-meeting-cp16:12
*** _ducttape_ has quit IRC16:27
*** ducttape_ has joined #openstack-meeting-cp16:28
*** mriedem has joined #openstack-meeting-cp16:46
ildikov#startmeeting cinder-nova-api-changes17:00
openstackMeeting started Mon Dec 19 17:00:42 2016 UTC and is due to finish in 60 minutes.  The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot.17:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:00
*** openstack changes topic to " (Meeting topic: cinder-nova-api-changes)"17:00
openstackThe meeting name has been set to 'cinder_nova_api_changes'17:00
ildikovscottda DuncanT ameade cFouts johnthetubaguy jaypipes takashin alaski e0ne jgriffith tbarron andrearosa hemna erlon mriedem gouthamr ebalduf patrickeast smcginnis diablo_rojo gsilvis  xyang1 raj_singh lyarwood17:01
mriedemo/17:01
scottdahey17:01
ildikovhi :)17:01
hemnatough17:01
johnthetubaguyo/17:02
ildikovlet's wait one more minute and then we can start17:02
ildikovon Cinder side we have the dependency of the new API patch merged, so hopefully jgriffith is busy with the API patch atm17:04
ildikovjohnthetubaguy: did you have a chance to go through the comments on the Nova side patch?17:04
* johnthetubaguy hides in the naughty corner17:04
* ildikov does not know what to say on the year's last meeting to this :)17:05
johnthetubaguyheh17:05
ildikovjohnthetubaguy: we discussed it briefly and the main comment from mriedem was to not initiate any volume state changes in Cinder from Nova17:05
ildikovjohnthetubaguy: so far everyone agreed to remove that direction form your spec17:06
johnthetubaguyOK, I am not sure I know what that means17:07
ildikovjohnthetubaguy: but we can discuss it further today if you feel the need17:07
mriedemjohnthetubaguy: yeah the main issue i had was there are several parts that rely on nova setting a vol attachment resource status to error in cinder when we hit a problem, and i thought nova would just delete the vol attachment17:07
johnthetubaguyah, delete vs update into an error state17:07
mriedemsimilar to nova rolling back and calling os-terminate_connection today on a failed attach17:07
mriedemyeah i don't like nova controlling resource state in cinder17:07
mriedemi don't think anyone does17:08
*** _ducttape_ has joined #openstack-meeting-cp17:08
johnthetubaguyI think I wanted a "tombstone" for an operator to do checks/cleanup17:08
mriedemi think you were doing it in part for evacuate making queries for attachments in error state,17:08
ildikovit also gives more items to clean up later which is not that fortunate either17:08
mriedembut there are probably other ways to handle that17:08
johnthetubaguyah, yeah, evacute17:08
johnthetubaguywhat was the replacement for evacuate?17:09
* smcginnis strolls in but stays at the back of the room17:09
mriedemit's been a couple of weeks since i left comments so they are in there17:09
mriedemi had some questions about evacuate related to the error state thing, but don't remember what they were now17:09
mriedembut was thinking there were alternatives, or just not needing the error thing17:09
johnthetubaguyoh, use the migration object instead17:09
mriedemnot sure what you mean by replacement for evacuate, but yeah we use the migration object to track state for evacuate now17:10
mriedemi think i thought about that at one point too, we could store some data in the migration object17:10
mriedemlike we have old/new instance type in the migration object17:10
mriedemif we needed, we could have old/new vol attachment in there too17:10
*** ducttape_ has quit IRC17:11
johnthetubaguyso I should go through that, and undo the error state stuff17:11
johnthetubaguyso tomorrow is my last day before next year17:12
johnthetubaguyI should try get that done tomorrow17:12
johnthetubaguybut been pushing on neutron v2 refactoring instead17:12
ildikovjohnthetubaguy: the pretty high level Cinder spec: http://specs.openstack.org/openstack/cinder-specs/specs/ocata/add-new-attach-apis.html17:13
mriedemjohnthetubaguy: i'd say if you have thoughts, just make a note in the spec review as a reminder for when you get back, but keep trucking on the neutronv2 conductor stuff17:13
ildikovjohnthetubaguy: just in case you would need to double check what calls we ended up with finally17:14
mriedembecause that's actually in plan for ocata17:14
johnthetubaguymriedem: yeah, I think thats probably the better approach17:14
johnthetubaguyI can't think of why delete the attachment wouldn't be enough17:15
johnthetubaguyI think I was worrying too much about cinder knowing about all the state changes on the Nova side17:15
mriedemwhich we don't want cinder to know about17:16
smcginnismriedem: +117:16
ildikovI'm more of a fan of single responsibility as much as we can achieve that here17:16
mriedemfrom what i remember i think the only case that it maybe made some kind of sense we evacuate cleanup, but again, it's been 2 weeks and i think we have alternatives17:16
ildikovso what mriedem says :)17:16
smcginnisI think Cinder should have minimal awareness of what's happening with consumers.17:16
mriedems/we/was/17:16
ildikovmriedem: having alternatives sounds good, we can go into details if needed on the next meeting in Jan17:18
johnthetubaguyI think I don't really understand the failure modes around disconnect that well at this point17:18
johnthetubaguyI guess if its not detached, we leave it attached, if it is detatched, we detach it17:19
johnthetubaguyanyways, simpler sounds like a better starting point17:20
ildikovjohnthetubaguy: +117:20
ildikovjohnthetubaguy: when we'll have the Cinder API merged or close to that state we can also do some PoC, like what jgriffith started earlier17:21
johnthetubaguythe tricky bit, I think, is going to be the smooth transition to the new API17:22
ildikovyou mean from the point of view of the already created volumes?17:23
smcginnisjohnthetubaguy: Or just all the changes required on the nova side, microversions, etc.?17:24
johnthetubaguyyeah, already created volumes17:25
johnthetubaguyand when to use the new API part way through operations like live-migration17:25
johnthetubaguywe have a mix of old a new compute nodes out there, etc17:26
johnthetubaguywe have patterns to follow, its just tricky17:27
ildikovwe can switch to the new API, when all the computes are upgraded17:27
mriedemjohnthetubaguy: we can globally lock that out based on minimum nova-compute service version in the deployment if needed17:27
ildikovthat should be doable17:27
ildikovmriedem: yep, that's what I meant too17:27
mriedemwe can also check per-compute service versions if we want to be fancy, but that's probably a more complicated dance17:27
johnthetubaguymriedem: yeah, we might have to do that per operation though17:27
johnthetubaguywell, I mean at the start of long running operations, like live-migrate17:28
johnthetubaguyso we don't swap half way through17:28
ildikovold volumes should have attachment_id already, it's a question what other info we would need for live migrate for instance as we will create a new attachment for the new host anyhow17:28
johnthetubaguybut the BDM pattern I proposed in the spec should do that17:28
ildikovso the question here is removing the old one I guess17:28
johnthetubaguyapparently, not all old volumes have an attachment id already17:29
mriedemjohnthetubaguy: do you mean attachment id in nova or cinder?17:29
johnthetubaguyif they were attached before we created attachment ids17:29
johnthetubaguyin cinder I mean17:29
mriedemjohnthetubaguy: i think we're talking about different things17:29
mriedemi've been told in previous meetings that each volume attached has something in the cinder volume_attachments table already17:30
johnthetubaguyonly past a certain point in time, I believe17:30
ildikovI don't remember when that got introduced, but should be there for a while now17:30
mriedemis havana old enough?17:30
ildikovit is for me!17:30
mriedemhemna wrote the schema migration so he'd know17:30
hemnahey17:31
hemnasorry guys, been on multiple meetings at once17:31
hemnaok correct.17:32
johnthetubaguyaccording to the user survey, 16% of users in October 2015 were on Havana, 12% are on older releases17:32
mriedemjohnthetubaguy: so for old volume attachments / BDMs we know we'll need to 'upgrade' those to the new model somehow17:32
hemnaeach attachment is supposed to have a volume_attachment entry17:32
johnthetubaguymriedem: yeah, basically17:32
hemnaand all the original cinder volume table information was migrated when the volume_attachment table was created.17:32
johnthetubaguymriedem: I think we previously agreed a cinder-manage cmd to fix that up17:33
mriedemjohnthetubaguy: i left some comments on that in the spec too i believe17:33
mriedemjohnthetubaguy: i was hoping for something more automatic17:33
mriedemi.e. online data migration17:33
johnthetubaguymriedem: we can call all that from the online data migrations stuff17:33
mriedembecause the spec says something about a nova-manage command calling a cinder-manager command, and that seems bad to me17:33
johnthetubaguyso the trade off was vs adding an API thats just for the upgrade hump17:33
mriedemi think we can migrate old attachments on (1) first access after rolling new code or (2) a periodic task in the computes for vols attached to instances on that upgraded compute17:34
mriedemthen maybe after n-2 or whatever we drop that compat code17:34
*** SergeyLukjanov__ has quit IRC17:34
johnthetubaguyyeah, I am thinking the same, except its nova-manage cmds to match the DB work17:35
mriedemif nova-manage has to cast to the compute it's not going to be fun17:35
mriedem*rpc call to the compute17:35
mriedemwhen i was reading the spec i thought there was some reason we'd need to call to the compute - to get the volume connector from brick17:35
johnthetubaguymriedem: true, its really about staggering the system load on upgrade17:35
*** SergeyLukjanov has joined #openstack-meeting-cp17:35
mriedemwhich is needed in the update attachment for existing attachments17:35
mriedemwe do'nt have the connector from nova-manage, and you can only git it off the compute17:36
johnthetubaguyyeah17:36
mriedemso you're talking about adding an rpc call from nova-manage to compute i think17:36
mriedemwhich is why i was talking about doing something more automatic17:36
johnthetubaguyyeah, thats a "bit" evil :(17:36
mriedemor background tasks17:36
mriedemright17:36
johnthetubaguyanyways, we can work through that17:36
mriedemyeah it's all in the spec :)17:37
ildikov:)17:37
mriedemmy wedding gift to you17:37
johnthetubaguymriedem: :)17:37
scottdaha. That's right congratulations johnthetubaguy17:37
johnthetubaguyactually just got the usb stick with all the photos on today, so thats some fun for tonight17:38
ildikovjohnthetubaguy: I guess you're asking me now to cut the meeting short :)17:38
johnthetubaguyI can't go in the living room right now, apparently, all I can here is present wrapping noises, and the odd bit of singing17:39
johnthetubaguyso I am good for now17:39
hemna:)17:39
ildikov:)17:39
johnthetubaguyanyways, we derailed fast there :)17:39
ildikovis there anything for the Nova spec we can/should discuss now?17:39
johnthetubaguycinder cmd vs cinder API?17:40
ildikov*anything else17:40
johnthetubaguyvs cinder lib17:40
johnthetubaguyfor that transition bit17:40
*** _ducttape_ has quit IRC17:40
johnthetubaguymaybe its such a mess if there is no API to "upgrade" an attachment, we should just allow us to call update?17:40
*** ducttape_ has joined #openstack-meeting-cp17:41
johnthetubaguy(and we ignore the case where there is no attachment_id)17:41
ildikovmy guess here is that it's either update or create a new attachment and remove the old17:42
johnthetubaguyoh... create a new one17:42
ildikovI'm slightly unsure at this point what update will do with an old attachment17:42
hemnaI think the idea was to just call update17:42
ildikovI can clarify this with jgriffith later17:43
ildikovbut I think the API should be able to cover this17:43
jgriffithsorry... late :(17:44
ildikovI wonder what the load will be during an upgrade and whether that can cause any issues17:44
jgriffithbeen trying to deploy OpenStack17:44
jgriffithI weep for our users17:44
scottdaha17:44
jgriffithjohnthetubaguy update is basically a "finalize" and make the actual connection17:44
ildikovjgriffith: now that you mentioned users I might accept this as an excuse :)17:45
jgriffithcreate *can* do that at the same time if you like17:45
mriedemi have to drop - sev1 in prod, ttyl17:45
jgriffithbut that doesn't work for Nova17:45
ildikovmriedem: thanks for joining, ttyl17:45
ildikovjgriffith: we are agonizing on an upgrade process17:46
jgriffithildikov how come?17:46
ildikovjgriffith: in the sense of switching to the new Cinder API and get old volume attachments updated17:46
jgriffithshould be ok with a manage script17:47
jgriffithwe've minimized the differences between the two17:47
jgriffithin terms of what's in the DB etc17:47
ildikovjgriffith: you mean on the Nova or the Cinder side?17:47
jgriffithremember a while back we had this whole rework to try and make them compatible17:47
jgriffithildikov Cinder side, Nova was a bit uglier (ie nova --> cinder OR cinder-1)17:48
johnthetubaguythere are concerns on Nova having to directly call cinder-manage17:48
jgriffithjohnthetubaguy I wouldn't have Nova do that if it were me17:48
johnthetubaguybasically because we need to get the connector from the specific nova-compute node17:48
*** sdague has joined #openstack-meeting-cp17:48
jgriffithjohnthetubaguy what i'm saying is that Nova should just check which version of the API cinder supports and use the latest call17:49
johnthetubaguyI am ok with nova-manage doing nasty things, but we can't really cheat here17:49
jgriffithas far as existing attachments and things like the stored attachment id... for that you do a get-attachments and find it17:49
johnthetubaguyjgriffith: this is about all the existing volumes, and trying to move them so we only use the new API version17:49
johnthetubaguythen drop the support in Nova for the old API17:49
johnthetubaguyright, the connector isn't stored today17:50
jgriffithjohnthetubaguy sure17:50
johnthetubaguyso that needs pushing up17:50
johnthetubaguyand connection_info I guess17:50
jgriffithjohnthetubaguy all you need to do after an upgrade (like nova-migrate) is call cinder "attachments_get_all_by_volume"17:50
jgriffithjohnthetubaguy that will give you the connector info and everything that you would get if you were using the "new" API calls17:50
johnthetubaguybut cinder doesn't have the info from os-brick?17:50
jgriffithjohnthetubaguy Oh!  Yeah... that case17:51
jgriffithjohnthetubaguy so that's doable17:51
johnthetubaguyyeah, we just haven't agreed on how yet17:51
jgriffithjohnthetubaguy we use the old call for detach etc, and just make sure we use all new calls for attach creation17:52
jgriffithjohnthetubaguy I had some code that did this... it even worked :p17:52
johnthetubaguyI think thats what ildikov was suggesting, that seems possible17:52
jgriffithjohnthetubaguy I do think we can make this work, I'll need to get the latest update to Cinder done, and get a running cloud again to move forward17:53
ildikovjohnthetubaguy: partially as I was thinking about how to update and I think it's not what jgriffith meant17:53
jgriffithildikov very hard to say "what I meant". even for me :)17:54
johnthetubaguyI like the delete/create approach though17:54
johnthetubaguymeans we do it via the API17:54
johnthetubaguybut its not APIs just created for upgrade17:54
ildikovjgriffith: hmm, good opportunity for me to plant ideas in your head now it seems :)17:54
johnthetubaguyits probably a create/delete, but thats a detail for later17:55
jgriffithjohnthetubaguy I think we can solve this with the cinder api calls17:55
ildikovI think Cinder API calls as a target is good in general17:55
jgriffithjohnthetubaguy it's annoying for a couple releases to support both but I think it's certainly possible17:55
ildikovjgriffith: as we have users still on Havana I wonder how many is that "couple"... :(17:56
ildikovjgriffith: it does not make it impossible of course, it was just a note17:56
johnthetubaguywell, maybe they just don't need the delete, because the attachment doesn't exist17:56
johnthetubaguyit looked like > 20% users where on havana or earlier in December 2015 user server17:57
johnthetubaguyso its very likely they still have volumes attached17:57
johnthetubaguyand want to upgrade at some point17:57
jgriffithjohnthetubaguy they can't upgrade H-->O anyway17:57
johnthetubaguytotally17:57
jgriffithjohnthetubaguy I can barely upgrade M-->N17:57
johnthetubaguyits just more, folks are using that thing in production17:58
johnthetubaguyjgriffith: well, I kinda think people should use OpenStack as a service, but I would say that17:58
jgriffithjohnthetubaguy well, I do think it's completely doable, and it will work a hell of a lot better than things like Keystone V3 and Glance and .......17:59
johnthetubaguyjgriffith: totally agreed, its totally doable17:59
jgriffithjohnthetubaguy after this lab move and redeploy I would agree with you!17:59
jgriffithWe've built something that requires PS to deploy at this point18:00
johnthetubaguywell, there are many competing deploy solutions, I guess18:00
johnthetubaguydoing it yourself is crazy hard18:01
johnthetubaguyanyways, at some point we should discuss private cloud deploy optimizations, but anyways18:02
ildikovjohnthetubaguy: sounds a bit like discussing what to do with global warming at this point to me18:03
johnthetubaguyI guess we are out of time18:03
ildikovjohnthetubaguy: and that too :)18:03
jgriffithand lucky me... I get to go to a meeting on Open Source contribution policy :)18:03
johnthetubaguyildikov: yeah, thats true18:03
ildikovdoes anyone here would like to have the first meeting on the 2nd?18:04
johnthetubaguyjgriffith: I am sorry18:04
ildikovor the 9th sounds just fine?18:04
johnthetubaguythats a public holiday for most folks right?18:04
ildikovjgriffith: should I create a reason for you to miss that? :)18:04
scottdaI think the 9th18:05
scottdaNot much will be changed by the 2nd18:05
ildikovjgriffith: unless you're the one holding it :)18:05
johnthetubaguyscottda: a very good point18:05
ildikovscottda: yeap, my thought exactly, just wanted to give a chance for objection if there would happen to be any :)18:05
johnthetubaguyhappy holidays, when that happens for folks18:06
ildikovHappy Holidays everyone!18:06
ildikovTalk to you on the 9th the latest!18:06
scottdayup, have a good one all18:07
ildikovThanks for all your efforts this year and have some well deserved fun and rest in the upcoming two weeks! :)18:07
ildikov#endmeeting18:08
*** openstack changes topic to " (Meeting topic: cinder-nova-api-changes)"18:08
openstackMeeting ended Mon Dec 19 18:08:26 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)18:08
openstackMinutes:        http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2016/cinder_nova_api_changes.2016-12-19-17.00.html18:08
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2016/cinder_nova_api_changes.2016-12-19-17.00.txt18:08
openstackLog:            http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2016/cinder_nova_api_changes.2016-12-19-17.00.log.html18:08
*** MarkBaker has quit IRC18:14
*** stvnoyes has quit IRC19:06
*** david-lyle_ has joined #openstack-meeting-cp19:13
*** jamespag` has joined #openstack-meeting-cp19:13
*** jamespag` has quit IRC19:13
*** jamespag` has joined #openstack-meeting-cp19:13
*** david-lyle has quit IRC19:13
*** jamespage has quit IRC19:13
*** openstack has joined #openstack-meeting-cp19:14
*** ChanServ sets mode: +o openstack19:14
*** ducttape_ has quit IRC19:37
*** breton_ is now known as breton20:04
*** ducttape_ has joined #openstack-meeting-cp20:05
*** stvnoyes has joined #openstack-meeting-cp20:35
*** mriedem has left #openstack-meeting-cp20:46
*** rockyg has joined #openstack-meeting-cp21:02
*** gouthamr has quit IRC21:17
*** _ducttape_ has joined #openstack-meeting-cp21:29
*** ducttape_ has quit IRC21:33
*** _ducttape_ has quit IRC21:34
*** xyang1 has quit IRC22:53
*** rockyg has quit IRC23:05
*** ducttape_ has joined #openstack-meeting-cp23:31
*** lamt has quit IRC23:32
*** ducttape_ has quit IRC23:34
*** ducttape_ has joined #openstack-meeting-cp23:35
*** jaugustine has quit IRC23:36
*** ducttape_ has quit IRC23:39
*** sdague has quit IRC23:44
*** ducttape_ has joined #openstack-meeting-cp23:55
*** gouthamr has joined #openstack-meeting-cp23:58
*** ducttape_ has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!