Thursday, 2017-06-08

*** lbragstad has joined #openstack-meeting-cp00:37
*** lbragstad has quit IRC01:42
*** diablo_rojo has quit IRC01:46
*** diablo_rojo has joined #openstack-meeting-cp02:19
*** Rockyg has joined #openstack-meeting-cp02:37
*** Rockyg has quit IRC02:37
*** edmondsw has joined #openstack-meeting-cp02:55
*** edmondsw has quit IRC02:59
*** gagehugo has left #openstack-meeting-cp03:02
*** aselius has quit IRC04:32
*** diablo_rojo has quit IRC05:34
*** MarkBaker has joined #openstack-meeting-cp08:55
*** sdague has joined #openstack-meeting-cp10:11
*** MarkBaker has quit IRC10:20
*** markvoelker has quit IRC10:27
*** markvoelker has joined #openstack-meeting-cp10:27
*** edmondsw has joined #openstack-meeting-cp11:40
*** jroll has left #openstack-meeting-cp12:47
*** MarkBaker has joined #openstack-meeting-cp13:23
*** lbragstad has joined #openstack-meeting-cp14:14
*** felipemonteiro_ has joined #openstack-meeting-cp14:18
*** diablo_rojo has joined #openstack-meeting-cp14:27
*** felipemonteiro__ has joined #openstack-meeting-cp14:30
*** felipemonteiro_ has quit IRC14:34
*** aselius has joined #openstack-meeting-cp14:46
*** ashishb has quit IRC15:04
*** MarkBaker has quit IRC15:06
*** lbragstad has quit IRC15:08
*** MarkBaker has joined #openstack-meeting-cp15:18
*** MarkBaker has quit IRC15:31
*** jgriffith_ is now known as jgriffith15:36
*** MarkBaker has joined #openstack-meeting-cp15:46
*** mriedem has joined #openstack-meeting-cp15:56
*** lbragstad has joined #openstack-meeting-cp15:57
ildikov#startmeeting cinder-nova-api-changes16:00
openstackMeeting started Thu Jun  8 16:00:06 2017 UTC and is due to finish in 60 minutes.  The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: cinder-nova-api-changes)"16:00
openstackThe meeting name has been set to 'cinder_nova_api_changes'16:00
mriedemo/16:00
ildikovDuncanT ameade cFouts johnthetubaguy jaypipes takashin alaski e0ne jgriffith tbarron andrearosa hemna erlon mriedem gouthamr ebalduf patrickeast smcginnis diablo_rojo gsilvis  xyang1 raj_singh lyarwood breitz jungleboyj16:00
stvnoyeso/16:00
pewphemna (=゚ω゚)ノ16:00
johnthetubaguyo/16:00
ildikovhi :)16:01
jgriffitho/16:01
ildikovok, let's start16:01
mriedemyou don't have stvnoyes in your list16:02
mriedemand should probably remove alaski :)16:02
ildikovmriedem: will update, tnx :)16:02
ildikovso the open reviews are looking good: https://review.openstack.org/#/q/topic:bp/cinder-new-attach-apis16:03
ildikovwe have two patches on the gate to handle the microversion discovery, tnx to mriedem and all the reviewers16:03
ildikovthe attach PoC is under continuous updates16:04
mriedemstvnoyes: i think we can abandon https://review.openstack.org/#/c/471142/16:04
ildikovstvnoyes: were you be able to test live_migrate at all with that one?16:04
stvnoyesagreed16:04
mriedemchanging course on the live migration stuff,16:05
mriedemgoing to try the LiveMigrateData object to stash the old volume attachment ids16:05
mriedemduring the live migration16:05
stvnoyesi am testing migrate now with the new approach of having the old attach info in the migrate_data. that seems to be working ok with one small problem that i'm tracking down now16:05
stvnoyessome strange problem with os-brick. i haven't confirmed this, but it doesn't seem to like having the 'serial' value in connection_info16:06
hemnahave any logs ?16:06
stvnoyesduring connect16:06
ildikovstvnoyes: isn't it another cast issue?16:07
stvnoyeswell, I am doing connects right now with and with that and when I seem to get a pattern I'll let you know16:07
hemnaok, let me know if I can fix something16:07
stvnoyesno, no exceptions, the volume is just not seen from within the vm16:07
stvnoyes^ with and withOUT that...16:07
hemnamaybe pastebin some logs at the time os-brick is called and returns16:07
stvnoyesok16:08
stvnoyesjust hit this within the last hour16:08
hemnawith debug logging on, you should be able to see what os-brick returns from connect_volume back to nova16:08
stvnoyesok16:09
hemnaping me if you any help.16:09
johnthetubaguyin the new flow, I guess we call os-brick more often? like when you create the second attachment?16:09
johnthetubaguyignore more, I should re-read that16:09
stvnoyesI think it's the same # of times calling, this is during pre-live_migration where you connect to the vol on the dest16:10
johnthetubaguyignore me that was meant to say16:10
johnthetubaguyyeah, I forgot about the stashed connect_info's16:10
stvnoyesbtw, what is the purpose of 'serial'?16:10
stvnoyesit has the volume_id in it16:11
mriedemnot sure, i've wondered that over time too16:11
hemnaI'm unfamiliar with that16:11
jgriffithstvnoyes that's an *excellent* question :)16:11
stvnoyes;-) I guess I'm not alone16:11
ildikovmriedem: haven't we all...16:11
jgriffithstvnoyes I've never figured out why that's there as an additional volume_id arg16:12
mriedemit's a libvirt thing16:12
mriedemset in LibvirtConfigGuestDisk16:12
hemnathe cryptsetup stuff uses it for some reason16:13
johnthetubaguyits the thing we connect the VM to, I guess?16:13
hemnahttps://github.com/openstack/os-brick/blob/master/os_brick/encryptors/cryptsetup.py#L5116:13
stvnoyesanyways, I think i'm very close to getting this migrate work up for review. the unit tests are done... Just one last thing... :-)16:13
mriedemhemna: probably just b/c it knows it's the volume id16:13
hemnaonly place it's accessed afaik16:13
mriedemno it's also used in the libvirt driver16:14
mriedemduring snapshot16:14
mriedem_volume_snapshot_create16:14
ildikovstvnoyes: never say 'just one last thing' out loud! :)16:14
mriedemfor the glusterfs stuff16:14
hemnaah ok, I wasn't looking in the nova code16:14
mriedemwhich we've removed now, but...16:14
stvnoyeslet me work on this more. it's too early to dive too deep on this yet16:14
stvnoyesildikov- that's why the smile is there...16:15
stvnoyes:-)16:15
mriedem"If present, this specify serial number of virtual hard drive.           For example, it may look           like <serial>WD-WMAP9A966149</serial>.           Not supported for scsi-block devices, that is those using           disk type 'block' using device 'lun'           on bus 'scsi'.           Since 0.7.1"16:15
ildikovstvnoyes: last time I hit iscsiadm issues in my test env for instance, so there's always something that's odd at least in my env with attach...16:15
mriedemhttps://libvirt.org/formatdomain.html#elementsDisks16:16
ildikovstvnoyes: it's still a pretty thin line :)16:16
stvnoyesthanks mriedem - this makes sense - serial - If present, this specify serial number of virtual hard drive.16:17
mriedemi don't think nova makes any distinction for "Not supported for scsi-block devices, that is those using           disk type 'block' using device 'lun'           on bus 'scsi'." though16:17
mriedemnova just always sets it i think16:17
jgriffithmriedem it does, and uses the cinder volume-id in the case of block type16:18
stvnoyeswith John's new attach patch, it's not set during attach (which works ok).16:18
stvnoyesthough I am not suing John's latest16:19
stvnoyesusing16:19
mriedemi can ask some libvirt/kvm people about serial16:19
jgriffithI mean "it does not distinguish"16:19
stvnoyesit's OK, let me verify i's the problem first.16:19
jgriffithstvnoyes I will look and see if the latest has any more *missing* translations of that16:19
jgriffithstvnoyes during attach I explicitly check if that exists and if it doesn't, set it to volume-id16:20
johnthetubaguyI think serial is just a thing the virtual device exposes to the guest, right?16:20
stvnoyeslet's move on since I think it's still too early in my debugging of this.16:22
mriedemanyway16:22
mriedemi've asked kashyap who will probably ask qemu devs16:23
mriedemhas there been any movement on the swap volume questions on the cinder side?16:23
mriedemor is that no longer a question/issue?16:23
ildikovstvnoyes: do you see this issue only with live_migrate?16:23
ildikovmriedem: the idea now is to re-write swap for the new flow16:23
ildikovon both sides16:23
ildikovwe just wanted to make attach work before doing that16:24
mriedemildikov: that's kind of a chicken and egg though,16:24
ildikovso we can test what we're doing16:24
ildikovmriedem: you mean from merging perspective?16:24
mriedemi thought we were going to do swap before the attach change at the very end16:24
mriedemswap keys off if the volume that is attached, and being swapped from, was attached with the new api flow16:25
mriedemhttps://review.openstack.org/#/c/456971/6/nova/compute/api.py16:25
ildikovthat's an old review16:27
mriedemwell,16:27
mriedemit's the only review16:27
mriedemfor swap16:27
ildikovI mean since then we realized that the migration_completion stuff on the Cinder side will not work either so we need to re-think this16:27
mriedemso now we're talking about a new cinder api microversion for swap?16:27
mriedemright, so i'm asking, has it been re-thought :)16:27
ildikovwe will look into this with jgriffith soon I think16:27
ildikovas attach is operational enough for testing now, except a few cleanups16:28
mriedemtoday is p-2 so if there needs to be a new cinder api, it's getting late for pike16:28
johnthetubaguythis is blocking us turning on the new API right?16:28
mriedemyes16:28
jgriffithI don't think there's a need for a new API to simplify that16:28
mriedembecause swap won't handle new style attachments othrewise16:28
johnthetubaguyright, ideally the new and old volumes each have a separate attachment during the swap, because nova has two attachments16:28
ildikovmriedem: +116:29
* jgriffith will look at it again today16:29
jgriffithI think there's some confusion also because there's two types of swap16:29
jgriffithLiterally swap a-->b using libvirts copy function16:29
johnthetubaguycinder started and Nova API started with a different volume16:29
jgriffithand the swap associated with migration16:29
jgriffiththey're two very different things16:29
johnthetubaguyright, we are talking about the first one right?16:30
johnthetubaguythe Nova swap_volume API?16:30
mriedemthey both result in the libvirt driver doing the blockRebase though right?16:30
jgriffithso that's easy16:30
jgriffithmriedem yes16:30
mriedemwho initiates them is what impacts the migration_completion callback to cinder16:30
mriedemand i believe is the pickle16:30
johnthetubaguyah, so its the same two things I am thinking about, just different names16:30
mriedemtoday if cinder didn't initiate, it just ignores the migration_completion callback i think16:30
jgriffithmriedem yes, but in the first case the callback is a noop anyway so can just go away16:31
jgriffiththe second case is a mess, and I honestly haven't come up with an answer for it as of yet16:31
johnthetubaguyI thought the only difference is the result volume-uuid for the thing thats left attached at the end?16:31
jgriffithjohnthetubaguy yes, the second case does a swap of the volume-uuid to make it an "invisible" thing16:31
mriedemjgriffith: sure, it can't go away on the nova side w/o us knowing if we should call it or not though, is our problem16:31
jgriffithwhich is stupid IMO16:31
jgriffithmriedem but Nova already has all of the control on that16:32
jgriffith(in the first case)16:32
johnthetubaguywe don't right?16:32
jgriffithI'm agreeing that the second case is different16:32
mriedemwe don't know who called the swap volume REST API16:32
johnthetubaguywe don't know if its a user calling us or cinder after the user called cinder right?16:32
mriedemif it was cinder or an admin16:32
jgriffithjohnthetubaguy yes, it's not written that way due to call-backs etc but the call-backs don't actually *do* anything on the Cinder side16:33
mriedemjgriffith: in the case of a non-volume migrate initiated swap, right?16:33
johnthetubaguyno, no, we are saying the REST API calls on the Nova side, and Nova state is identical in both cases16:33
jgriffithmriedem correct16:33
mriedemin the case of volume migration, i thought they completed the migration on the cinder side16:33
mriedemi think we all know what we're saying, we're just saying different things :)16:33
jgriffithhaha.. perhaps16:34
johnthetubaguymriedem: +116:34
mriedemthe issue is what migrate_completion does on the cinder side for new-style attachments for a volume-migrate initiated swap16:34
jgriffithregardless there are problems there that aren't trivial16:34
jgriffithIMHO that code should all be reverted :)16:34
johnthetubaguyif there was a new swap API, that took two attachment_ids, and was the one cinder called, and for that one we made a new callback to a new API that took the same two attachment_ids... we would be in a more sane place?16:35
ildikovjgriffith: that might not be an option :)16:35
jgriffithbummer16:35
johnthetubaguyactually... thats missing too much context16:35
johnthetubaguyto be clear, this is now blocking multi-attach, and us moving to the new API16:36
ildikovit pretty much does16:36
johnthetubaguyfor me, if we passed attachment_ids around, instead of volume-ids, for cinder and nova APIs, wouldn't we have all the state we needed?16:37
hemnaespecially in the case of multi-attach that makes sense16:37
jgriffithjohnthetubaguy yes, that was kinda the whole idea of the attachment-object16:38
johnthetubaguythat means passing Nova an attachment we would update from os-brick, then attach to the instance, and the other attachment would have to be already attached, etc16:38
ildikovjohnthetubaguy: how would that help with the we don't know who called the swap-volume API problem?16:38
johnthetubaguyildikov: it doesn't fix that, but we would be telling you what the attchement was, so cinder should know if thats a migration attachment or not16:39
hemnaattaching the attached attachment.16:39
johnthetubaguyit helps a bit in that we care less if the volume-uuid changes, because the attachment-uuid is the same16:39
ildikovjohnthetubaguy: ah ok, got it16:40
johnthetubaguyhemna: whos doing what now?16:40
hemna:P16:40
hemnaignore me16:40
johnthetubaguyheh16:40
johnthetubaguyit seems like a new cinder API to be added, and a new Nova API that cinder calls into?16:41
johnthetubaguythats one approach, I guess the question is, is there a simpler option available?16:41
jgriffithjohnthetubaguy that honestly might be the safest and most expedient answer16:41
johnthetubaguyyeah, its the safest explcit one I think16:42
ildikovjohnthetubaguy: jgriffith: +116:42
jgriffithbut that means we need to come up with something and agree on both sides in like 24 hours :)16:42
ildikovjgriffith: I don't see the issue :)16:42
johnthetubaguywell... ideally three months ago, but either way could work16:42
jgriffithha!16:43
* jgriffith fires up his way-back machine16:43
mriedemi'm not totally following here, you're suggesting a new cinder api where nova passes in the newly attached attachment uuid for the attachment we swapped *to*?16:43
mriedemto complete the migratoin?16:43
johnthetubaguya new nova API to pass the existing and new attachment id16:43
mriedemand if swap was initiated by a user, not cinder, then it's ignored like today?16:44
jgriffithmriedem I *think* a new cinder/nova interaction for the volume-migration call from Cinder16:44
johnthetubaguythen a new cinder API it calls back to, passing both the attachments16:44
mriedemnew nova api?16:44
johnthetubaguyyeah, I think its a new nova api too really16:44
jgriffithjohnthetubaguy +116:44
mriedemumm16:44
johnthetubaguyso the cinder started one, calls a new API that gets attachments16:44
johnthetubaguywe can try and stop regular using using that this time, for real16:45
mriedemcan we not do more special case service-specific APIs please16:45
mriedemyou can't stop regular users from using it16:45
johnthetubaguywell, I mean good default RBAC really16:45
mriedemdefault rbac on swap is already admin only16:45
johnthetubaguy its user safe, if they create attachments16:45
jgriffithok, I'll look again later today and see if there's a way to fix the mess we already have16:45
johnthetubaguyyeah, admin only isn't good enough, but thats a different debate16:45
mriedemanyway - my fear is, if swap depends on the new attach flow, and now we can't do swap w/o this new api interaction, which is a microversion newer than the new attach api, things start to get super f'ed16:46
johnthetubaguyso.. if we ignore multi attach, I am surpized why the cinder side can't just be "fixed up", are we missing something there?16:46
johnthetubaguymriedem: yeah, we would have to raise to this new API when checking if we can use the new flow16:47
mriedemmaybe we don't design this here,16:47
mriedemi don't understand the issues on the cinder side,16:47
mriedembut can the problem be stated in the ML?16:47
johnthetubaguythats my previous question really, whats the issue cinder side, if we keep the API the same16:48
mriedemand then we, and hopefully others, can sort it out where/when we have time to digest the actual problem?16:48
johnthetubaguyyeah, some written down form would be good16:48
johnthetubaguyit seems clear we can't move forward here and now without that16:48
mriedemi can live with new cinder apis if we have to, because then the deps are still one direction,16:49
mriedembut if we're doing new apis on both sides, which is a circular dependency, then we start to get into crazy land16:49
johnthetubaguyso proposal 1: re-write the lot to be attachment aware, propsal 2: just fail if any attachment is multi-attach and use the old APIs, question: what does proposal 2 not work for cinder?16:49
ildikovmriedem: there are old attach/detach snippets in the call-bak on the Cinder side, which should not hit in the simple case though16:49
mriedemi'm not really sure why we're talking about multi-attach at this point16:50
mriedem"swapping multi-attached volumes is not supported, fin"16:50
mriedemat least in v116:50
ildikovmriedem: +116:50
mriedemwe do'nt support multiattach today in nova anyway16:50
mriedemand likely won't in pike16:50
johnthetubaguywell, we should include swapping a volume that has multiple attachments16:50
johnthetubaguyi.e. half way through a live-migration16:51
mriedemthat's not the same operatoin though16:51
mriedemthere is no migrate_completion callback during nova live migratoin16:51
johnthetubaguyNova will just reject those API calls anyways, so no change there16:51
mriedemthe migration_completion cinder api is the issue16:51
mriedemas far as i understand16:51
mriedemand that's *only* for swap volume16:51
mriedemon the nova side16:51
ildikovmriedem: correct16:51
johnthetubaguyyes, I am talking about when you call swap volume16:51
johnthetubaguywhat is the state of the system16:51
johnthetubaguyif we have a volume attached on two hosts, its "complicated" on the cinder side, as there are two attachments16:52
johnthetubaguybut Nova doesn't care about that case16:52
mriedemjohnthetubaguy: yo'ure talking about swapping a multiattach volume though right?16:52
johnthetubaguyno16:52
johnthetubaguymulti-attach=false16:52
*** MarkBaker has quit IRC16:52
johnthetubaguytwo attachments16:53
*** mgiles has joined #openstack-meeting-cp16:53
johnthetubaguysource host and destination host16:53
mriedemhow is the same volume attached to two hosts during swap volume?16:53
mriedemif not multiattach?16:53
mriedemswap volume is literally 2 separate volumes16:53
mriedem1 attachment each16:53
johnthetubaguyon the cinder side you can call migrate at any time16:53
johnthetubaguythe VM could be in the live-migrating state16:53
johnthetubaguyNova fails the API call16:53
johnthetubaguybut theoretically, you can have two attachments in the Cinder DB16:54
johnthetubaguysame volume16:54
mriedemcan you swap volumes on an instance being migrated?16:54
johnthetubaguysame instance16:54
johnthetubaguyso if you call that, I think you get a Nova API error16:54
johnthetubaguybut its a case on the cinder side16:54
mriedemyou can't swap volumes on an instance that's undergoing a task state change i don't think16:54
mriedemwe're either talking past each other,16:54
johnthetubaguyI am trying to say cinder can ignore that case16:54
mriedemor i'm just totally not understanding what you're saying16:54
johnthetubaguyso... yeah, that16:54
mriedemwith 5 minutes left, and i need lunch, let's table this and move it to the ML,16:55
johnthetubaguyprobably a hangout thing, after its all written down16:55
johnthetubaguy+116:55
mriedembut i'd really just like jgriffith or someone from cinder to describe the problem we have today with non-multiattach volumes and the migration_callback in cinder16:55
mriedemwith the new flow16:55
ildikovmriedem: johnthetubaguy: we will get it written down with jgriffith and then we can figure out whether it's an ML thread a Hangouts call or both or what else16:55
mriedemi really need writing to digest it16:56
johnthetubaguyjgriffith: do you understand the question we are asking?16:56
jgriffithjohnthetubaguy partially, but frankly you are all over the board :)16:56
mriedemjgriffith: "i'd really just like jgriffith or someone from cinder to describe the problem we have today with non-multiattach volumes and the migration_callback in cinder"16:56
jgriffithSo I'll put a write up together on the two cases16:56
mriedemthat's step 116:56
jgriffithand what the challenges are problems etc16:56
johnthetubaguybasically, if we keep all the APIs the same, whats broken in the new world?16:56
johnthetubaguyperfect16:56
ildikovjohnthetubaguy: your last scenario is unclear to me but we will get to the base case and let's take it from there16:57
johnthetubaguyI think I am making option 2 possible, but yeah, lets get the basics written down16:57
ildikovjohnthetubaguy: I think we will get a better view with some written material first16:58
johnthetubaguycool, seems like we are done, and just waiting for that write up?16:58
ildikovjohnthetubaguy: yes16:58
stvnoyesjgriffith: can you stick around here after the mtg?  I have a couple of questions for you about the attach POC operation16:59
mriedemstvnoyes: could move to nova or cinder channels17:00
jgriffithstvnoyes sure17:00
ildikovok, let's end the meeting and move to the Cinder channel with further chats17:00
jgriffithstvnoyes mriedem yeah... I'll catch you cinder17:00
ildikovor the Nova one :)17:00
stvnoyesok see on the cinder irc17:00
ildikovthank you all!17:00
ildikovwill catch up later on swap17:00
*** mriedem has left #openstack-meeting-cp17:00
ildikov#endmeeting17:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings"17:00
openstackMeeting ended Thu Jun  8 17:00:45 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2017/cinder_nova_api_changes.2017-06-08-16.00.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2017/cinder_nova_api_changes.2017-06-08-16.00.txt17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2017/cinder_nova_api_changes.2017-06-08-16.00.log.html17:00
*** mgiles has quit IRC17:03
*** MarkBaker has joined #openstack-meeting-cp17:04
*** pewp has quit IRC18:10
*** pewp has joined #openstack-meeting-cp18:12
*** diablo_rojo has quit IRC18:38
*** MarkBaker has quit IRC18:54
*** diablo_rojo has joined #openstack-meeting-cp19:05
*** MarkBaker has joined #openstack-meeting-cp19:06
*** pewp has quit IRC19:13
*** pewp has joined #openstack-meeting-cp19:17
*** harlowja has quit IRC20:08
*** kbyrne has quit IRC20:48
*** kbyrne has joined #openstack-meeting-cp20:50
*** MarkBaker has quit IRC21:17
*** edmondsw has quit IRC22:02
*** reed has quit IRC22:02
*** reed has joined #openstack-meeting-cp22:03
*** diablo_rojo has quit IRC22:51
*** felipemonteiro__ has quit IRC23:05
*** sdague has quit IRC23:08
*** knangia_ has joined #openstack-meeting-cp23:18
*** pewp has quit IRC23:21
*** pewp has joined #openstack-meeting-cp23:22
*** markvoelker has quit IRC23:34
*** harlowja has joined #openstack-meeting-cp23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!