| *** harlowja has quit IRC | 00:05 | |
| *** sdake has quit IRC | 00:22 | |
| *** sdake has joined #openstack-meeting-cp | 00:24 | |
| *** dtroyer has left #openstack-meeting-cp | 00:36 | |
| *** harlowja has joined #openstack-meeting-cp | 00:54 | |
| *** sdake_ has joined #openstack-meeting-cp | 00:59 | |
| *** sdake has quit IRC | 01:02 | |
| *** sheel has joined #openstack-meeting-cp | 01:03 | |
| *** jklare has quit IRC | 01:31 | |
| *** jklare has joined #openstack-meeting-cp | 01:33 | |
| *** amrith is now known as _amrith_ | 01:56 | |
| *** dims has quit IRC | 01:59 | |
| *** dims has joined #openstack-meeting-cp | 02:04 | |
| *** Rocky_g has quit IRC | 02:14 | |
| *** dims_ has joined #openstack-meeting-cp | 02:45 | |
| *** dims has quit IRC | 02:46 | |
| *** sdake_ is now known as sdake | 03:07 | |
| *** sdake_ has joined #openstack-meeting-cp | 03:17 | |
| *** sdake has quit IRC | 03:20 | |
| *** vilobhmm11 has joined #openstack-meeting-cp | 03:25 | |
| *** askb has joined #openstack-meeting-cp | 03:42 | |
| *** vilobhmm11 has quit IRC | 04:34 | |
| *** markvoelker has joined #openstack-meeting-cp | 04:36 | |
| *** vilobhmm11 has joined #openstack-meeting-cp | 04:47 | |
| *** sdake_ has quit IRC | 05:20 | |
| *** sdake has joined #openstack-meeting-cp | 05:29 | |
| *** sdake has quit IRC | 05:48 | |
| *** sdake has joined #openstack-meeting-cp | 05:51 | |
| *** markvoelker has quit IRC | 06:10 | |
| *** markvoelker has joined #openstack-meeting-cp | 06:26 | |
| *** markvoelker has quit IRC | 06:39 | |
| *** vilobhmm11 has quit IRC | 06:40 | |
| *** vilobhmm11 has joined #openstack-meeting-cp | 06:40 | |
| *** markvoelker has joined #openstack-meeting-cp | 07:40 | |
| *** markvoelker has quit IRC | 07:45 | |
| *** sdake_ has joined #openstack-meeting-cp | 07:49 | |
| *** sdake has quit IRC | 07:51 | |
| *** takashin has joined #openstack-meeting-cp | 07:55 | |
| *** takashin has left #openstack-meeting-cp | 07:55 | |
| *** sdake_ has quit IRC | 08:36 | |
| *** markvoelker has joined #openstack-meeting-cp | 08:42 | |
| *** markvoelker has quit IRC | 08:47 | |
| *** vilobhmm11 has quit IRC | 08:59 | |
| *** markvoelker has joined #openstack-meeting-cp | 09:44 | |
| *** markvoelker has quit IRC | 09:49 | |
| *** sdague has joined #openstack-meeting-cp | 10:16 | |
| *** _amrith_ is now known as amrith | 10:26 | |
| *** sdague has quit IRC | 10:53 | |
| *** sdague has joined #openstack-meeting-cp | 10:53 | |
| -openstackstatus- NOTICE: Gate on project-config is currently broken due to IRC tests. The problem has been detected and we are working to fix the issue as soon as possible. | 11:14 | |
| *** markvoelker has joined #openstack-meeting-cp | 11:46 | |
| *** markvoelker has quit IRC | 11:51 | |
| *** raildo-afk is now known as raildo | 12:11 | |
| *** david-lyle_ has joined #openstack-meeting-cp | 12:17 | |
| *** david-lyle has quit IRC | 12:17 | |
| *** amrith is now known as _amrith_ | 12:44 | |
| *** markvoelker has joined #openstack-meeting-cp | 12:47 | |
| *** markvoelker has quit IRC | 12:51 | |
| *** ninag has joined #openstack-meeting-cp | 12:53 | |
| *** annegentle has joined #openstack-meeting-cp | 13:04 | |
| *** annegentle has quit IRC | 13:17 | |
| *** diablo_rojo has joined #openstack-meeting-cp | 13:18 | |
| *** nikhil_k has joined #openstack-meeting-cp | 13:18 | |
| *** nikhil has quit IRC | 13:19 | |
| *** annegentle has joined #openstack-meeting-cp | 13:38 | |
| *** ninag has quit IRC | 13:39 | |
| *** ninag has joined #openstack-meeting-cp | 13:41 | |
| *** ninag has quit IRC | 13:42 | |
| *** annegentle has quit IRC | 13:43 | |
| *** markvoelker has joined #openstack-meeting-cp | 13:48 | |
| *** markvoelker has quit IRC | 13:52 | |
| *** _amrith_ is now known as amrith | 13:58 | |
| *** Guest42028 has joined #openstack-meeting-cp | 14:00 | |
| *** Guest42028 has quit IRC | 14:07 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:12 | |
| *** markvoelker has joined #openstack-meeting-cp | 14:16 | |
| *** sdake has joined #openstack-meeting-cp | 14:44 | |
| *** annegent_ has joined #openstack-meeting-cp | 14:46 | |
| *** markvoelker has quit IRC | 14:46 | |
| *** sdake_ has joined #openstack-meeting-cp | 14:56 | |
| *** sdake has quit IRC | 14:57 | |
| *** ninag has joined #openstack-meeting-cp | 14:58 | |
| *** annegent_ has quit IRC | 15:02 | |
| *** sdake has joined #openstack-meeting-cp | 15:08 | |
| *** sdake_ has quit IRC | 15:10 | |
| *** annegentle has joined #openstack-meeting-cp | 15:22 | |
| *** askb_ has joined #openstack-meeting-cp | 15:25 | |
| *** askb_ has quit IRC | 15:27 | |
| *** askb_ has joined #openstack-meeting-cp | 15:27 | |
| *** askb has quit IRC | 15:28 | |
| *** annegentle has quit IRC | 15:28 | |
| *** markvoelker has joined #openstack-meeting-cp | 15:29 | |
| *** askb_ has quit IRC | 15:29 | |
| *** askb_ has joined #openstack-meeting-cp | 15:29 | |
| *** askb_ has quit IRC | 15:31 | |
| *** askb_ has joined #openstack-meeting-cp | 15:31 | |
| *** askb_ has quit IRC | 15:32 | |
| *** askb_ has joined #openstack-meeting-cp | 15:33 | |
| *** askb_ has quit IRC | 15:35 | |
| *** askb_ has joined #openstack-meeting-cp | 15:35 | |
| *** askb_ has quit IRC | 15:37 | |
| *** askb_ has joined #openstack-meeting-cp | 15:37 | |
| *** askb_ has quit IRC | 15:38 | |
| *** annegentle has joined #openstack-meeting-cp | 15:46 | |
| *** annegentle has quit IRC | 15:51 | |
| *** mc_nair has left #openstack-meeting-cp | 16:00 | |
| *** annegentle has joined #openstack-meeting-cp | 16:03 | |
| *** annegentle has quit IRC | 16:05 | |
| *** stevemar has quit IRC | 16:08 | |
| *** openstack has joined #openstack-meeting-cp | 17:04 | |
| *** ChanServ sets mode: +o openstack | 17:04 | |
| *** annegentle has joined #openstack-meeting-cp | 17:07 | |
| *** nikhil_k is now known as nikhil | 17:35 | |
| *** sdake has quit IRC | 17:42 | |
| *** annegentle has quit IRC | 17:48 | |
| *** annegentle has joined #openstack-meeting-cp | 17:50 | |
| *** ninag has quit IRC | 18:03 | |
| *** annegentle has quit IRC | 18:03 | |
| *** gnarld_ is now known as cfouts | 18:06 | |
| *** vilobhmm11 has joined #openstack-meeting-cp | 18:15 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 18:34 | |
| *** annegentle has joined #openstack-meeting-cp | 18:35 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:35 | |
| *** ninag has joined #openstack-meeting-cp | 18:53 | |
| *** sdake has joined #openstack-meeting-cp | 18:57 | |
| *** xyang1 has joined #openstack-meeting-cp | 19:47 | |
| *** rockyg has joined #openstack-meeting-cp | 19:53 | |
| *** angdraug has joined #openstack-meeting-cp | 19:55 | |
| *** tpatil has joined #openstack-meeting-cp | 19:58 | |
| *** sdake_ has joined #openstack-meeting-cp | 20:00 | |
| *** patrickeast has joined #openstack-meeting-cp | 20:01 | |
| *** sdake has quit IRC | 20:02 | |
| *** tyr_ has joined #openstack-meeting-cp | 20:03 | |
| *** annegentle has quit IRC | 20:03 | |
| tpatil | Is the Cinder-Nova-API meeting postponed? | 20:08 |
|---|---|---|
| thingee | tpatil: looks like it's 21:00 utc, one more hour | 20:10 |
| thingee | http://lists.openstack.org/pipermail/openstack-dev/2016-March/090320.html | 20:10 |
| *** angdraug has quit IRC | 20:12 | |
| tpatil | thingee: Thanks. 21:00 UTC is 13:00 PST, it's already 13:11, anyway I will check again after one hour | 20:12 |
| *** angdraug has joined #openstack-meeting-cp | 20:13 | |
| *** angdraug has quit IRC | 20:13 | |
| thingee | tpatil: it's actually 14:00 with daylight saving time | 20:13 |
| tpatil | thingee: You are correct! | 20:14 |
| *** tyr_ has quit IRC | 20:16 | |
| *** tyr_ has joined #openstack-meeting-cp | 20:17 | |
| *** annegentle has joined #openstack-meeting-cp | 20:21 | |
| *** ninag has quit IRC | 20:26 | |
| *** angdraug has joined #openstack-meeting-cp | 20:32 | |
| *** sdake has joined #openstack-meeting-cp | 20:32 | |
| *** tyr_ has quit IRC | 20:32 | |
| *** sdake_ has quit IRC | 20:33 | |
| *** takashin has joined #openstack-meeting-cp | 20:48 | |
| *** tyr_ has joined #openstack-meeting-cp | 20:50 | |
| *** andrearosa has joined #openstack-meeting-cp | 20:59 | |
| *** scottda has joined #openstack-meeting-cp | 21:00 | |
| ildikov | #startmeting Cinder-Nova API changes | 21:00 |
| ildikov | #startmeeting Cinder-Nova API changes | 21:00 |
| openstack | Meeting started Wed Mar 30 21:00:24 2016 UTC and is due to finish in 60 minutes. The chair is ildikov. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: Cinder-Nova API changes)" | 21:00 | |
| openstack | The meeting name has been set to 'cinder_nova_api_changes' | 21:00 |
| scottda | HI | 21:00 |
| patrickeast | hey | 21:00 |
| ildikov | scottda ildikov DuncanT ameade cFouts johnthetubaguy jaypipes takashin alaski e0ne jgriffith tbarron andrearosa hemna erlon mriedem gouthamr | 21:00 |
| DuncanT | Hi | 21:00 |
| cfouts | hi | 21:00 |
| ildikov | my typingabilities are limited, sorry :S | 21:00 |
| andrearosa | hi | 21:00 |
| takashin | Hi | 21:01 |
| ildikov | hi everyone | 21:01 |
| *** tbarron has joined #openstack-meeting-cp | 21:01 | |
| ildikov | #chair scottda | 21:01 |
| openstack | Current chairs: ildikov scottda | 21:01 |
| * alaski lurks | 21:01 | |
| scottda | #link https://etherpad.openstack.org/p/cinder-nova-api-changes | 21:01 |
| scottda | ildikov: I can type if you'd like | 21:01 |
| ildikov | let's wait one more min for folks to show up | 21:02 |
| scottda | OK | 21:02 |
| tbarron | hi | 21:02 |
| *** jungleboyj has joined #openstack-meeting-cp | 21:02 | |
| jungleboyj | o/ | 21:02 |
| *** mriedem has joined #openstack-meeting-cp | 21:02 | |
| mriedem | i'm here! | 21:02 |
| * jungleboyj cheers for mriedem | 21:03 | |
| scottda | Please add items to the etherpad under Issues Summary if y | 21:03 |
| karthikp | hi | 21:03 |
| ildikov | ok, I guess we can start | 21:03 |
| scottda | So, we'd like to fix some things in the API between Nova and Cinder | 21:03 |
| ildikov | the agenda is on the etherpad scottda posted above | 21:03 |
| ildikov | #topic Briefly discuss the current situation, summarize/agree on the issues | 21:04 |
| *** openstack changes topic to "Briefly discuss the current situation, summarize/agree on the issues (Meeting topic: Cinder-Nova API changes)" | 21:04 | |
| *** sdague has quit IRC | 21:04 | |
| scottda | One issue we faced in Mitaka is with multi-attach in Nova... | 21:04 |
| ildikov | there were a few mail threads on the mailing list about issues with the interaction between these two modules | 21:04 |
| mriedem | unfortunately i've failed at reading the multiattach detach thread | 21:05 |
| mriedem | but i think i have a general idea | 21:05 |
| scottda | We had to revert ildikov 's patches because it broke older Cinder clients by passing in a new parameter. | 21:05 |
| ildikov | multiattach is special, it highlights already existing issues pretty well | 21:06 |
| scottda | We can fix that now that Cinder has api microversions | 21:06 |
| scottda | But that will require figuring out the pattern to use when Nova needs to use a cinder microversion. So that is one thing. | 21:06 |
| mriedem | scottda: those were the patches reverted during the midcycle right? | 21:06 |
| scottda | mriedem: yes | 21:06 |
| mriedem | and they were to what, pass the connector on attach/detach? | 21:07 |
| scottda | mriedem: I believe so | 21:07 |
| ildikov | the passed the host name to Cinder | 21:07 |
| scottda | oh, right. | 21:07 |
| ildikov | *they | 21:07 |
| ildikov | Cinder can accept host name, but not host name and instance at the same time | 21:07 |
| scottda | Newer (Mitaka +) cinder can, the issue is with LIberty and older | 21:08 |
| ildikov | don't ask about the history of that check, but it's what broke Cinder | 21:08 |
| mriedem | so there is a microversion that added hostname to attach/detach? | 21:08 |
| *** raildo is now known as raildo-afk | 21:08 | |
| ildikov | scottda: yeah, exactly, hemna fixed this during Mitaka | 21:08 |
| ildikov | this is why tests passed on the gate for my patch | 21:08 |
| scottda | mriedem: Actually, not microversioned in Cinder. just an optional parameter in Mitaka | 21:08 |
| ildikov | but I think this is a small part of the whole issue | 21:09 |
| mriedem | so if i have a newton nova and liberty cinder, | 21:09 |
| mriedem | and nova passes hostname to cinder | 21:09 |
| mriedem | kablammo | 21:09 |
| mriedem | right? | 21:09 |
| mriedem | instance + hostname | 21:09 |
| scottda | yes | 21:09 |
| mriedem | hence why we have microversions | 21:09 |
| mriedem | b/c nova has to ask what microversions it can send | 21:10 |
| scottda | right | 21:10 |
| *** amrith is now known as _amrith_ | 21:10 | |
| scottda | And that paves the way for any other changes that are backward incompatible. | 21:10 |
| mriedem | well, it's also about discoverability, but yeah | 21:11 |
| scottda | Yes. I think when Nova instantiates the cinderclient, it can figure out the server version then and possible store it. | 21:12 |
| scottda | Then use that to trigger different logic in the code. maybe | 21:12 |
| * scottda is waving his arms around here | 21:12 | |
| scottda | But we'll have to figure out those details, and then use that pattern for other stuffs. | 21:12 |
| scottda | ildikov: Any other things around multiattach? | 21:13 |
| mriedem | does this fix the detach issue though? | 21:13 |
| ildikov | I'm not sure sending hostname will fix everything | 21:13 |
| mriedem | i thought there was a problem there where cinder needed to start storing the connector in the attachment db | 21:13 |
| scottda | mriedem: There is a POC by jgriffith that might help more with detach... | 21:14 |
| scottda | #link https://review.openstack.org/#/c/284867/ | 21:14 |
| ildikov | mriedem: it's kind of an open question still that in what extent we're dealing with the dead horse in each module | 21:14 |
| scottda | That stores the connector in the attachment db | 21:14 |
| ildikov | mriedem: Cinder can figure out detach based on this extra info on it's side | 21:14 |
| scottda | And could also return the attachment_id to Nova when intialize_connection is called. | 21:15 |
| ildikov | if the hostname is stored with the attachments in volume info than Nova can also use that information when disconnects a volume | 21:15 |
| ildikov | although we might face issues still due to the different Cinder drivers | 21:15 |
| ildikov | s/drivers/back ends/ | 21:15 |
| *** annegentle has quit IRC | 21:16 | |
| scottda | Yes, that is another issue. Some drivers multiplex the iscsi connection for >1 volume to the same compute host. Others create 1 connection per attachment. | 21:16 |
| mriedem | scottda: "And could also return the attachment_id to Nova when intialize_connection is called." - that's prior to nova calling os-attach in the volume api | 21:16 |
| mriedem | scottda: so that's a chicken/egg | 21:16 |
| ildikov | mriedem: that was my thinking as well | 21:17 |
| scottda | mriedem: Not sure the details of jgriffith's POC. You are right. Might return it in attach() | 21:17 |
| mriedem | we already get the attachment_id back from os-attach | 21:17 |
| ildikov | yeah, we get in before detaching | 21:17 |
| patrickeast | i think having the info was half the issue, knowing when to actually disconnect from the backend was the other part that is (in my mind) still unclear | 21:18 |
| mriedem | oh nvm | 21:18 |
| mriedem | os-attach is a 202 | 21:18 |
| mriedem | nova waits for the volume to go to in-use | 21:18 |
| mriedem | and nova can get the attachments from the volume GET | 21:18 |
| ildikov | but it's only for telling Cinder which attachment to detach in case of multiple, it does not say anything about the connector | 21:18 |
| patrickeast | like scottda mentioned some backend would have a single iscsi session open for all the attachments | 21:18 |
| patrickeast | s/backend/backends/ | 21:19 |
| patrickeast | which is a problem for multi-attach to >1 vm on the same compute node | 21:19 |
| patrickeast | well, becomes a problem | 21:19 |
| ildikov | patrickeast: yeah, exactly, tempest tests with lvm and logs show nicely in multiattach case that the session is killed in case of the first detach on a host | 21:19 |
| scottda | patrickeast: Right. So there's a question of whether Nova or Cinder should contain the logic around when to actually detach() in Cinder and remove the export | 21:20 |
| ildikov | patrickeast: and the target is also removed on Cinder side at the moment | 21:20 |
| patrickeast | yep yep | 21:20 |
| scottda | I don't think we need to answer these questions now... | 21:20 |
| scottda | I think we are better off figuring out what the questions are... | 21:20 |
| scottda | and listing the possible solutions. | 21:21 |
| patrickeast | scottda: +1 | 21:21 |
| ildikov | scottda: yeah, we already got into a few small rabbit holes here | 21:21 |
| *** _amrith_ is now known as amrith | 21:22 | |
| ildikov | I think regarding multiattach the main issue is detach | 21:22 |
| scottda | ildikov: And if we figured out when to actually remove the target and finalize the detach in Cinder, would that solve all known issues? | 21:23 |
| *** tyr__ has joined #openstack-meeting-cp | 21:23 | |
| scottda | That, and only passing in host+instance_id with Mitaka+ versions of Cinder? | 21:23 |
| *** tyr_ has quit IRC | 21:23 | |
| ildikov | Nova needs to know also when to call out to os-brick | 21:24 |
| scottda | ildikov: This is for the driver/backend specific issues? Around when to actually remove the target? | 21:24 |
| scottda | Or something else? | 21:24 |
| ildikov | as far as I understood Nova closes the session on the compute node, while Cinder removes the target, but I might messed up with the logs regarding this | 21:25 |
| mriedem | does removing the target happen in the terminate_connection API in cinder? | 21:25 |
| *** annegentle has joined #openstack-meeting-cp | 21:25 | |
| ildikov | yes | 21:25 |
| mriedem | and nova passes the connector to that, | 21:25 |
| ildikov | but we also call disconnect_volume in Nova | 21:25 |
| patrickeast | removing the target does, but you need to log out on the compute node first | 21:25 |
| mriedem | which it's getting from os-brick | 21:25 |
| patrickeast | so at some point nova needs to know what to do | 21:25 |
| ildikov | mriedem: yeah, that's what I meant | 21:26 |
| mriedem | yes, the order is the nova virt driver disconnects the volume | 21:26 |
| mriedem | which actually happens in os-brick | 21:26 |
| mriedem | then it calls the terminate-connection API in cinder | 21:26 |
| patrickeast | yep, exactly | 21:26 |
| scottda | right | 21:26 |
| mriedem | and passes the volume connector | 21:26 |
| mriedem | if that connector dict was stored with the attachment in the cinder db, nova wouldn't have to pass that | 21:26 |
| mriedem | nova also get the connector from os-brick | 21:27 |
| ildikov | mriedem: I would assume in that case Nova could also check whether there are multiple volume attachments using the same? | 21:27 |
| mriedem | using the same what? | 21:27 |
| scottda | hemna Had an idea for how to do that from the Nova side. | 21:27 |
| scottda | He is not here ATM | 21:28 |
| ildikov | yeah, but that required calling initialize_connection from detach | 21:28 |
| scottda | ildikov: Yes. But we could add a new API to Cinder to get that info | 21:28 |
| andrearosa | +1 | 21:29 |
| ildikov | scottda: can we store that info with the attachments? | 21:29 |
| scottda | Can who? Nova? | 21:29 |
| scottda | or Cinder | 21:29 |
| ildikov | scottda: if not I'm fine with the new API as well, I just would like to minimise the round trips as that's a constant concern | 21:29 |
| *** annegentle has quit IRC | 21:30 | |
| ildikov | Cinder | 21:30 |
| scottda | There are a few ideas around what to store, where, by whom.... | 21:30 |
| patrickeast | one argument in the past is that cinder may not always know about all of the places a volume was attached by some cinder user (in this case nova-compute + os-brick) | 21:30 |
| scottda | and which entity determines when to remove the target/export | 21:30 |
| patrickeast | so storing the refcount of attachments to some host wasn't really ideal to keep there | 21:30 |
| patrickeast | like the vm->attached volume mapping count, that is | 21:30 |
| ildikov | scottda: yeah, I know, would be nice to clarify that either here or on a follow up meeting/mail thread... | 21:30 |
| scottda | patrickeast: Right. There is the arguement that Cinder should not care about any of this. Just call Cinder when you want the target removed. | 21:31 |
| patrickeast | scottda: yea, i mean, there is sort of a fundamental argument about whether its right or wrong, but also concerns about whether it actually *can* consistently keep track | 21:31 |
| ildikov | patrickeast: I remember a version of that idea, live migration can mess those things up for instance if it's allowed | 21:32 |
| scottda | ildikov: I'm working on a Cinder Spec to allow update of connector_info and host in the Cinder DB. That would help with live migration issues. | 21:33 |
| ildikov | patrickeast: I was wondering to store connector info | 21:33 |
| ildikov | scottda: yeah, sounds like a good start to me, although not a solution to the debates mentioned here | 21:34 |
| scottda | ildikov: Right. Just one piece in the puzzle. And one that doesn't have a lot of different ideas floating around to confuse things. | 21:34 |
| ildikov | scottda: I was suggesting this direction as well, so I'm convinced :) | 21:35 |
| scottda | So, under etherpad Issues Summary #2: New Cinder API for Updating Connector info and Nova host is needed for Nova live migration, evacuation, and shelve offload | 21:35 |
| scottda | Is there any debate or issues with that? | 21:35 |
| ildikov | scottda: it's still a question though how closely coupled we want Nova and Cinder to be here | 21:36 |
| scottda | Well, for live migration/shelve_offload/evacuate, we are changing the host.... | 21:36 |
| scottda | And Cinder stores that host info. | 21:36 |
| scottda | IF we are using host+instance_id for detach in multi-attach, we'd need to update it. | 21:36 |
| scottda | andrearosa: Wasn't there some other bug around volumes attached during migration? | 21:37 |
| scottda | That would be fixed if we could update that info in Cinder DB? | 21:37 |
| ildikov | when the host is changed we need to do updates for sure | 21:38 |
| andrearosa | IIRC there is an issue in not closing the connection with the target when we migrte, that is an issue with shelved_offload not sure if it iw an issue for LM | 21:38 |
| scottda | OK, so that's Issue #2 | 21:38 |
| scottda | andrearosa: Ahh..right. I filed a bug for that but it was relegated to "Wishlist" | 21:38 |
| andrearosa | yeap | 21:38 |
| scottda | I wanted to add a Cinder API to close the connection/remove the target | 21:39 |
| scottda | so, I'm skipping issue #1 for a moment because it is hard.. | 21:39 |
| scottda | Issue #3 is Cinder cannot force-detach without Nova/caller help since it does not have Connector info | 21:39 |
| scottda | We can fix that by saving the connector_info | 21:39 |
| patrickeast | easy :D | 21:40 |
| scottda | using jgriffith's patch, or something similar | 21:40 |
| scottda | yup. So let's move back to #1 | 21:40 |
| scottda | Detaching volumes with multi-attach when on the same host causes problems | 21:40 |
| scottda | I'm suggesting we list the issues and alternatives, with Pros and Cons. | 21:40 |
| cfouts | +1 | 21:41 |
| patrickeast | +1 | 21:41 |
| ildikov | sounds cool | 21:41 |
| scottda | We can add that to the etherpad. And discuss a bit in IRC. Then have a meeting or wrestling match or something to get to some resolution. | 21:42 |
| scottda | Did anyone have any other issues that needed addressing? IF so, please add to the etherpad "Issues Summary" | 21:42 |
| scottda | Anything else we could discuss right now? | 21:43 |
| mriedem | ndipanov had a patch in nova i can probably add | 21:43 |
| ildikov | it would be good to see a bigger picture as opposed to focus on one particular item | 21:43 |
| mriedem | but it's more of a nova issue i think | 21:43 |
| ildikov | mriedem: which one? | 21:43 |
| mriedem | https://review.openstack.org/#/c/290793/ | 21:43 |
| ildikov | mriedem: tnx! | 21:44 |
| scottda | mriedem: OK. I think we are open to any changes in the Cinder API to make things better. So anything is fair game. | 21:44 |
| mriedem | i'm not sure this requires a cinder api change | 21:44 |
| ildikov | scottda: mriedem: do we have a picture on what to capture where? | 21:44 |
| mriedem | i think it's a nova problem, but was related to some cinder interaction on attach | 21:44 |
| scottda | I actually have a WIP spec to refactor the basic attach and detach APIs. But that is probably a lower priority. | 21:45 |
| ildikov | I mean spec wise | 21:45 |
| scottda | ildikov: Well, one spec is cinder-connector-host-update | 21:45 |
| mriedem | you probably can't write specs until you know what the current multiattach problems are and what the possible fixes are | 21:45 |
| mriedem | which i thought needed documenting somewhere (maybe the same etherpad?) | 21:45 |
| scottda | ildikov: I'm working on that one. | 21:45 |
| scottda | mriedem: Yes, I think that etherpad can work for the issues and alternatives for multiattach solution. | 21:46 |
| mriedem | i've honestly lost a lot of the context on the multiattach details since prior to the midcycle | 21:46 |
| ildikov | mriedem: I mainly meant whether we can all agree that it would be better to keep things separate a bit | 21:46 |
| scottda | I think we all have. It's a complex area. | 21:46 |
| ildikov | like don't shoehorn everything into the multiattach spec for instance | 21:47 |
| mriedem | ildikov: i'd agree with that | 21:47 |
| scottda | yup, me too. | 21:47 |
| ildikov | as storing extra info is used for other cases as well, so it would be better to have full view as opposed to fix only multiattach aspects IMHO | 21:47 |
| mriedem | we'll need some details laid out on the version constraints here too, i.e. nova newton talking to cinder liberty | 21:47 |
| scottda | WE usually start with implementing changes on the Cinder side. Nova cannot implement things until Cinder has. | 21:47 |
| ildikov | mriedem: how many version do we plan to support backwards? | 21:48 |
| ildikov | as during Mitaka I think someone mentioned Icehouse or Juno also which is kind of way back in the past to adapt to IMHO | 21:48 |
| mriedem | ildikov: well, it really comes down to the api | 21:48 |
| mriedem | nova needs a way to tell that the version of cinder it's talking to can support api x | 21:49 |
| scottda | WE cannot crash older deployments of Cinder. We'll have to make sure that any incompatible changes are safely using microversions. | 21:49 |
| mriedem | and if not, we have to fail on anything that requires nova using cinder api x | 21:49 |
| ildikov | mriedem: like everything by the end of the day :) | 21:49 |
| scottda | OK. we can get folks thinking about the best way to do that. | 21:50 |
| mriedem | but the client interaction doesn't really get flushed out until we know what the cinder api changes are, if any | 21:50 |
| scottda | Nova should be able to discover Cinder API version when the cinderclient is instantiated. | 21:50 |
| ildikov | yeap, right, let's elaborate things on the etherpad | 21:50 |
| mriedem | scottda: yup | 21:51 |
| scottda | OK. We'll get hemna and jgriffith to chime in there. | 21:51 |
| scottda | Alright. Anything else? | 21:51 |
| cfouts | I'd like to mention that the Cinder community should discuss how often the Cinder API version is updated. I've seen it updated each time there is an API change and that seems to cause quite a bit of pain | 21:51 |
| cfouts | maybe only update the API version at milestones? | 21:52 |
| ildikov | scottda: do we have a follow up plan? | 21:52 |
| ildikov | scottda: have another sync before the summit and/or talk about this on the summit? and/or bring it back to the mailing list? | 21:53 |
| scottda | cfouts: What does Nova do? I think Manila changes for each API change. We need to allow any deployer to take HEAD of master at any time and use it. | 21:53 |
| cfouts | Nova seems to change far less often but I don't know what the logic is | 21:53 |
| scottda | ildikov: I think we'll end up syncing again before the summit. That's still 3+ weeks out. | 21:53 |
| mriedem | scottda: cfouts: http://docs.openstack.org/developer/nova/api_microversion_dev.html | 21:53 |
| cfouts | mriedem: thanks | 21:53 |
| mriedem | the microversion changes when it needs to based on the change | 21:53 |
| mriedem | you can't line things up around milestones | 21:54 |
| scottda | mriedem: Yes. I stole all the Cinder code and docs from Nova. Or Manila, which got it from Nova. | 21:54 |
| mriedem | the point of microversions is the client opts into them | 21:54 |
| DuncanT | cfouts: What do you mean by 'problems'? | 21:54 |
| mriedem | so if your client code is requesting x.latest, that's a bug in the client | 21:54 |
| ildikov | scottda: I think it would be good to have a quick one, but we will organise it then when we're getting closer | 21:54 |
| DuncanT | cfouts: 'pain', sorry | 21:54 |
| scottda | ildikov: OK. Once we're happy with our listing of alternatives, we can sync again. | 21:54 |
| mriedem | scottda: ildikov: yeah we need another sync up meeting before the summit | 21:55 |
| mriedem | maybe in 2 weeks? | 21:55 |
| scottda | Sure. | 21:55 |
| mriedem | fwiw, i'm fine with weekly meetings if needed | 21:55 |
| cfouts | DuncanT: competing patches with the same microversion, client needing to be updated by developer to work with latest server version, QA wondering why things don't work because of microversion change | 21:55 |
| mriedem | most of the work right now is on the cinder team i think to lay out the plan | 21:55 |
| mriedem | cfouts: QA shouldn't be testing with the latest | 21:55 |
| scottda | mriedem: I agree. But there are the issues around alternatives to multi-attach support. and NOva will have opinions on that. | 21:56 |
| mriedem | clients need to update to support microversions in order | 21:56 |
| mriedem | scottda: yeah, agree | 21:56 |
| mriedem | scottda: you guys set up the options, and we'll knock them down | 21:56 |
| ildikov | scottda: mriedem: agreed, I will call for another meeting before the summit | 21:56 |
| cfouts | mriedem: our QA tests internally before we push upstream | 21:56 |
| cfouts | maybe that isn't a wide spread practice | 21:57 |
| DuncanT | cfouts: Competing patches, sure, it's annoying. We can at least avoid merging clashes with unit tests. Clien tonly needs updating if it wants to use the new feature, that's the point, and that change is needed whether you microversion it or not | 21:57 |
| mriedem | cfouts: i guess i don't have context | 21:57 |
| scottda | We only test in production. | 21:57 |
| scottda | :) | 21:57 |
| mriedem | for example, tempest is testing nova v2.1 | 21:57 |
| mriedem | but nova has like v2.25 | 21:57 |
| ildikov | mriedem: oh, missed the weekly meetings idea, TBH I could live with that as well, I think we can easily end up with a bigger amount of tasks here | 21:57 |
| mriedem | ildikov: at least to checkpoint on progress i think that would be fine | 21:57 |
| scottda | OK, I have to run. Thanks everyone. Weekly sounds good for now. It will put some pressure on us. | 21:57 |
| mriedem | ildikov: if there is no news, it's a short meeting | 21:57 |
| *** rockyg has quit IRC | 21:58 | |
| DuncanT | cfouts: All sorts of things can change with a patch between upstream submisison and merging, that is just the nature of the beast | 21:58 |
| ildikov | mriedem: exactly, and having a weekly sync can ensure progress as well | 21:58 |
| ildikov | scottda: thanks! | 21:58 |
| cfouts | DuncanT: maybe it is more of our process then. I can live with that :) | 21:58 |
| scottda | ildikov: Don't forget to endmeeting when we are all done. | 21:58 |
| ildikov | scottda: I will contact you about the follow-up/maybe weekly meetings | 21:59 |
| scottda | great | 21:59 |
| ildikov | scottda: no worries, it's not my first one I had in mind :) | 21:59 |
| *** vilobhmm11 has quit IRC | 21:59 | |
| ildikov | ok, we have a half minute left | 22:00 |
| takashin | I would like to talk about the nova swap volume function issue. | 22:00 |
| ildikov | is there any other aspects we would want to discuss here today? | 22:00 |
| takashin | #link https://bugs.launchpad.net/cinder/+bug/1522705 | 22:00 |
| openstack | Launchpad bug 1522705 in Cinder "Cinder volumes are stuck when non admin user executes nova swap volume API" [Undecided,In progress] - Assigned to Takashi NATSUME (natsume-takashi) | 22:00 |
| DuncanT | cfouts: We've had similar issues before microversions... I wonder if you could do something by bumping to a massive microversion in the (local only) patch, so your test can request something that will never be supported by upstream code or something? I'll have a think about it | 22:00 |
| ildikov | takashin: is this the one you wrote the mail about earlier? | 22:01 |
| andrearosa | thank you all, I have to go. Bye | 22:01 |
| takashin | I wrote in the etherpad. | 22:01 |
| ildikov | andrearosa: thanks for joining | 22:01 |
| takashin | The bug is caused because cinder 'migrate_volume_completion' API can be executed by admin only in default settings of cinder policy.json. | 22:01 |
| *** andrearosa has left #openstack-meeting-cp | 22:02 | |
| ildikov | takashin: yeah, I can see now | 22:02 |
| DuncanT | takashin: That's because it was only designed to be used by nova internals | 22:02 |
| *** vilobhmm11 has joined #openstack-meeting-cp | 22:03 | |
| *** vilobhmm11 has quit IRC | 22:03 | |
| takashin | IMO, it is the problem whether attach/detach volumes is performed in nova side or cinder side. | 22:03 |
| *** vilobhmm11 has joined #openstack-meeting-cp | 22:03 | |
| mriedem | takashin: so this is only a problem if it's a user-initiated volume migration on the cinder side right? | 22:04 |
| takashin | yes | 22:05 |
| mriedem | i don't know the details, but cinder calls the nova swap-volume API? | 22:05 |
| mriedem | as a non-adin | 22:05 |
| mriedem | *admin | 22:05 |
| mriedem | and nova then calls back to cinder as that non-admin and it fails | 22:05 |
| takashin | in cinder, volume migration is admin-only | 22:05 |
| takashin | But nova allows non-admin-user | 22:05 |
| mriedem | for swap | 22:06 |
| mriedem | because for nova it's just a detach of one volume and attach of another | 22:07 |
| takashin | Now detach/attach volume is performed in cider side. | 22:07 |
| DuncanT | So my first question would be what is the usecase for a none-admin to call this? | 22:07 |
| takashin | Originally nova allows it to non-adminuser | 22:08 |
| mriedem | if cinder is initiating the swap volume call to nova, why doesn't it do it with the same admin context? | 22:08 |
| mriedem | oh nvm | 22:08 |
| takashin | No | 22:09 |
| mriedem | so admin volume migration from cinder is not a problem, right? | 22:09 |
| mriedem | but non-user swap volume from nova is a problem | 22:09 |
| DuncanT | It was expected to be admin only, because it's dangerous in general to go swapping volumes without generating detach/attach events on the VM - it's an easy way to corrupt your volumes with old cached data | 22:09 |
| mriedem | because http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n4943 | 22:09 |
| mriedem | so it seems like nova should check if the context is admin before calling http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n4943 | 22:10 |
| mriedem | this also bleeds into a cross-project discussion about api capability discoverability | 22:10 |
| takashin | Does it means admin-context in non-admin user operation? | 22:11 |
| mriedem | so the user can find out what apis they can actually perform | 22:11 |
| DuncanT | mriedem: In this case, the answer is 'admin only', and always has been | 22:11 |
| mriedem | DuncanT: the policy file is configurable though | 22:11 |
| mriedem | unless there is something hardcoded in cinder for this | 22:12 |
| DuncanT | mriedem: Not sure without checking the code... we fixed a bunch of those but there might be more | 22:12 |
| *** vilobhmm11 has quit IRC | 22:12 | |
| takashin | Change policy in nova? | 22:12 |
| mriedem | takashin: to what? make it admin-only? | 22:12 |
| *** vilobhmm11 has joined #openstack-meeting-cp | 22:12 | |
| takashin | Yes | 22:13 |
| mriedem | idk, i guess that's an option | 22:13 |
| *** vilobhmm111 has joined #openstack-meeting-cp | 22:13 | |
| takashin | O.K. | 22:13 |
| mriedem | i honestly don't know what all of the use cases are around swap volume or when it was originally introduced | 22:13 |
| mriedem | i'd need some input from others on the nova team about that, like ndipanov | 22:13 |
| *** vilobhmm111 has quit IRC | 22:13 | |
| *** vilobhmm111 has joined #openstack-meeting-cp | 22:14 | |
| mriedem | seems it should have been implemented like the nova/neutron callback api that we have for vif plugging | 22:14 |
| mriedem | which is admin only | 22:14 |
| *** vilobhmm111 has quit IRC | 22:14 | |
| mriedem | "compute_extension:instance_actions:events": "rule:admin_api", | 22:14 |
| ildikov | we can follow up on this as well and add info to the etherpad in the meantime | 22:14 |
| *** vilobhmm111 has joined #openstack-meeting-cp | 22:14 | |
| mriedem | yeah | 22:14 |
| mriedem | i have to head out | 22:14 |
| mriedem | there is a ML thread about this? | 22:15 |
| ildikov | mriedem: second one at the bottom of the etherpad IIRC | 22:15 |
| takashin | Yes | 22:16 |
| mriedem | Fix nova swap volume (updating an attached volume) function | 22:16 |
| takashin | #link http://lists.openstack.org/pipermail/openstack-dev/2016-February/087549.html | 22:16 |
| mriedem | yeah | 22:16 |
| *** vilobhmm111 has quit IRC | 22:16 | |
| ildikov | yeah, that's the one | 22:16 |
| mriedem | i see it got 0 replies :) | 22:16 |
| *** claudiub|2 has quit IRC | 22:16 | |
| mriedem | ok, i'll try to read through that at some point and reply, or get others from nova involved | 22:16 |
| takashin | Thank you. | 22:16 |
| *** annegentle has joined #openstack-meeting-cp | 22:17 | |
| *** vilobhmm11 has quit IRC | 22:17 | |
| ildikov | great, I suggest to close the meeting now | 22:18 |
| mriedem | yes please :) | 22:18 |
| takashin | O.K. | 22:18 |
| ildikov | I will call for a follow up meeting or make it even a series | 22:18 |
| ildikov | I would encourage everyone to keep an eye and add info to the etherpad in the meantime! | 22:18 |
| ildikov | thanks everyone for joining, I think it was a good start | 22:19 |
| ildikov | so if nothing else, then good afternoon/evening/night/morning everyone :) | 22:19 |
| *** tbarron has left #openstack-meeting-cp | 22:20 | |
| mriedem | ildikov: yup, thanks for starting this | 22:20 |
| *** mriedem has left #openstack-meeting-cp | 22:20 | |
| ildikov | #endmeeting | 22:20 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings" | 22:20 | |
| openstack | Meeting ended Wed Mar 30 22:20:10 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:20 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2016/cinder_nova_api_changes.2016-03-30-21.00.html | 22:20 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2016/cinder_nova_api_changes.2016-03-30-21.00.txt | 22:20 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/cinder_nova_api_changes/2016/cinder_nova_api_changes.2016-03-30-21.00.log.html | 22:20 |
| *** takashin has left #openstack-meeting-cp | 22:20 | |
| *** markvoelker has joined #openstack-meeting-cp | 22:24 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:27 | |
| *** karthikp has quit IRC | 22:32 | |
| *** amrith is now known as _amrith_ | 22:33 | |
| *** sdake has quit IRC | 22:46 | |
| *** sdake has joined #openstack-meeting-cp | 22:46 | |
| *** vilobhmm11 has joined #openstack-meeting-cp | 22:47 | |
| *** markvoelker has quit IRC | 22:57 | |
| *** karthikp has joined #openstack-meeting-cp | 22:58 | |
| *** annegentle has quit IRC | 23:03 | |
| *** tpatil has quit IRC | 23:05 | |
| *** annegentle has joined #openstack-meeting-cp | 23:24 | |
| *** annegentle has quit IRC | 23:28 | |
| *** _amrith_ is now known as amrith | 23:29 | |
| *** annegentle has joined #openstack-meeting-cp | 23:34 | |
| *** annegentle has quit IRC | 23:38 | |
| *** angdraug has quit IRC | 23:43 | |
| *** karthikp has quit IRC | 23:48 | |
| *** annegentle has joined #openstack-meeting-cp | 23:51 | |
| *** amrith is now known as _amrith_ | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!