| *** gouthamr has joined #openstack-manila | 00:10 | |
| *** gouthamr has quit IRC | 00:16 | |
| *** a-pugachev has quit IRC | 00:27 | |
| *** gcb has joined #openstack-manila | 01:13 | |
| *** gouthamr has joined #openstack-manila | 01:13 | |
| *** ianychoi_ is now known as ianychoi | 01:35 | |
| *** hongbin has quit IRC | 01:51 | |
| *** kaisers_ has joined #openstack-manila | 01:54 | |
| openstackgerrit | zhongjun proposed openstack/python-manilaclient master: Change to share access list API https://review.openstack.org/468287 | 01:54 |
|---|---|---|
| openstackgerrit | zhongjun proposed openstack/manila master: Add export-location filter in share and share instance list API https://review.openstack.org/461712 | 01:57 |
| *** kaisers has quit IRC | 01:57 | |
| openstackgerrit | zhongjun proposed openstack/python-manilaclient master: Add export-location filter in share and share instance list https://review.openstack.org/468277 | 01:59 |
| *** jmlowe has joined #openstack-manila | 02:01 | |
| *** jmlowe has quit IRC | 02:02 | |
| *** jmlowe has joined #openstack-manila | 02:02 | |
| *** tuanluong has joined #openstack-manila | 02:46 | |
| *** markstur has quit IRC | 02:55 | |
| openstackgerrit | Merged openstack/manila master: Use ShareInstance model to access share properties https://review.openstack.org/467583 | 03:06 |
| *** esker has quit IRC | 03:22 | |
| *** kaisers_ has quit IRC | 04:06 | |
| *** arnewiebalck_ has joined #openstack-manila | 04:07 | |
| *** kaisers has joined #openstack-manila | 04:10 | |
| *** kaisers has quit IRC | 04:15 | |
| gouthamr | how does one use quota-class-show | 04:44 |
| gouthamr | GET v2/{project-id}/quota-class-set/xyzzy always returns something :P | 04:46 |
| gouthamr | weirdly | 04:46 |
| gouthamr | and it's not obvious that quota-class-update is a create as well... manilaclient help text says: "Update the quotas for a quota class (Admin only)" | 04:47 |
| gouthamr | if anyone here uses quota-classes, what is the use case? | 04:48 |
| *** gouthamr has quit IRC | 04:51 | |
| openstackgerrit | Merged openstack/manila master: Implement update_access in Isilon Driver. https://review.openstack.org/461487 | 05:02 |
| *** kaisers has joined #openstack-manila | 05:11 | |
| *** arnewiebalck_ has quit IRC | 05:14 | |
| *** jprovazn has joined #openstack-manila | 06:18 | |
| *** dsariel has joined #openstack-manila | 06:26 | |
| *** dsariel has quit IRC | 06:38 | |
| *** pcaruana has joined #openstack-manila | 07:26 | |
| *** a-pugachev has joined #openstack-manila | 07:34 | |
| zhongjun | gouthamr: hi. As I know. The quota class code in Manila was directly copied from Cinder.(You know it :) ) , and the quota class code in Cinder was directly copied from Nova (by the time it was nova-volume). | 07:41 |
| zhongjun | gouthamr: the use case is: one customer may have chosen an "Economy" package which gives them one level of quotas, while another customer may have chosen a "Premium" package with significantly higher quotas. Moreover, even these Nova users may want to be able to override specific quotas for a given customer rather than rely exclusively on which package they | 07:42 |
| zhongjun | have selected. | 07:42 |
| zhongjun | gouthamr: https://wiki.openstack.org/wiki/QuotaClass | 07:42 |
| *** arnewiebalck_ has joined #openstack-manila | 07:46 | |
| zhongjun | gouthamr: https://blueprints.launchpad.net/nova/+spec/quota-classes | 07:54 |
| *** a-pugachev has quit IRC | 07:58 | |
| openstackgerrit | qtlu proposed openstack/manila master: Add support for Guru Meditation Reports for manila https://review.openstack.org/444688 | 07:59 |
| *** lpetrut has joined #openstack-manila | 08:02 | |
| *** arnewiebalck_ has quit IRC | 08:14 | |
| *** dmellado_ has joined #openstack-manila | 09:00 | |
| *** kaisers1 has quit IRC | 09:01 | |
| *** dmellado has quit IRC | 09:01 | |
| *** andreaf has quit IRC | 09:03 | |
| *** andreaf has joined #openstack-manila | 09:06 | |
| *** kaisers1 has joined #openstack-manila | 09:09 | |
| openstackgerrit | zhongjun proposed openstack/manila master: Use uwsgi for devstack https://review.openstack.org/469389 | 09:12 |
| *** a-pugachev has joined #openstack-manila | 09:16 | |
| *** dmellado_ is now known as dmellado | 09:46 | |
| arnewiebalck | gouthamr: while the “service package” use case zhongjun described is probably the main one, another use case would be to define per share type default quotas (this is what we use quota classes on Cinder for, for instance) | 09:53 |
| openstackgerrit | zhongjun proposed openstack/manila master: Use uwsgi for devstack https://review.openstack.org/469389 | 10:01 |
| openstackgerrit | zhongjun proposed openstack/manila master: Change releasenote name '_' to '-' https://review.openstack.org/469407 | 10:01 |
| *** tuanluong has quit IRC | 10:27 | |
| *** ganso has joined #openstack-manila | 10:52 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack/manila master: [Generic driver] Fix incompatibility with novaclient https://review.openstack.org/469148 | 11:23 |
| *** esker has joined #openstack-manila | 11:55 | |
| *** esker has quit IRC | 12:00 | |
| *** MVenesio has joined #openstack-manila | 12:01 | |
| *** chlong has quit IRC | 12:03 | |
| *** kaisers has quit IRC | 12:30 | |
| arnewiebalck | I just upgraded our Manila deployment to Ocata. Went ok, it seems :) | 12:33 |
| arnewiebalck | One minor thing: did the capabilities filter become more picky? It complained about not finding any hosts which would match the requested specs of the share type (which was correct, but not a problem in Newton). | 12:35 |
| *** jmlowe has quit IRC | 12:37 | |
| ganso | arnewiebalck: hi Arne, which error is shown in manila-scheduler that does match the share type? | 12:42 |
| arnewiebalck | ganso: WIthout debug the error is “Filter CapabilitiesFilter returned 0 host(s)” | 12:50 |
| arnewiebalck | ganso: With debugg, the logs said “Share type extra spec requirement "create_share_from_snapshot_support=<is> True" does not match reported capability "False" capabilities_satisfied” | 12:51 |
| arnewiebalck | ganso: Which was correct. | 12:51 |
| arnewiebalck | ganso: I changed the type’s extra specs and all was fine. | 12:52 |
| ganso | arnewiebalck: you are using CEPH, which does not have create_share_from_snapshot_support capability | 12:52 |
| ganso | arnewiebalck: so you need to check if your share type has this capability set | 12:52 |
| arnewiebalck | ganso: Right. | 12:52 |
| arnewiebalck | ganso: Hwoever, this was not an issue in Newton. | 12:52 |
| ganso | arnewiebalck: the field was possibly added to the share type during the upgrade | 12:53 |
| ganso | arnewiebalck: it may have added with a value defaulting to True | 12:53 |
| arnewiebalck | ganso: Which is the correct thing to do, as it now detected an inconsistency between the type and the host. | 12:53 |
| arnewiebalck | ganso: Anyway, was easy enough to understand what was wrong, so overall the upgrade went w/o issues. | 12:55 |
| arnewiebalck | ganso: Wait :) | 12:57 |
| arnewiebalck | ganso: If this field was added, shouldn’t the default be False? | 12:57 |
| ganso | arnewiebalck: that's a difficult question to answer lol | 12:59 |
| ganso | arnewiebalck: the previous default behavior was that create_share_from_snapshot was always expected when snapshot_support was True | 12:59 |
| arnewiebalck | ganso: Makes sense. | 12:59 |
| ganso | arnewiebalck: if snapshot_support=False, then it should be False | 12:59 |
| arnewiebalck | ganso: I think this was True in our case (for the test share type). | 13:00 |
| ganso | arnewiebalck: if you have a backend that has snapshot_support=True, but does not create share from snapshots, then, you get the create_share_from_snapshot_support=True by default, which is what you've got | 13:00 |
| arnewiebalck | ganso: yes, it was True, so mkaes all perfect sense. | 13:00 |
| ganso | arnewiebalck: but why was it True for CEPH? it does not support snapshot AFAIK | 13:00 |
| arnewiebalck | ganso: I wanted to give it a try :) | 13:01 |
| ganso | arnewiebalck: looking back at Newton, it could create snapshots, but could never create shares from snapshots, which was kinda useless from manila point of view | 13:01 |
| ganso | arnewiebalck: oh so it is your fault :P | 13:01 |
| arnewiebalck | ganso: Absolutely! ;) | 13:01 |
| arnewiebalck | ganso: PEBKAC | 13:01 |
| ganso | arnewiebalck: lol | 13:02 |
| arnewiebalck | ganso: Thanks for clarifying this. | 13:02 |
| ganso | arnewiebalck: you're welcome, glad to have helped | 13:02 |
| *** eharney has joined #openstack-manila | 13:03 | |
| *** xyang1 has joined #openstack-manila | 13:04 | |
| *** jmlowe has joined #openstack-manila | 13:14 | |
| *** gouthamr has joined #openstack-manila | 13:19 | |
| *** kaisers has joined #openstack-manila | 13:26 | |
| *** chlong has joined #openstack-manila | 13:30 | |
| *** chlong has quit IRC | 13:43 | |
| *** chlong has joined #openstack-manila | 13:46 | |
| *** kaisers has quit IRC | 13:55 | |
| *** draynium has quit IRC | 14:02 | |
| *** FL1SK has joined #openstack-manila | 14:04 | |
| bswartz | MVenesio: the generic driver isn't that smart yet | 14:11 |
| bswartz | it needs new code to detect when it's hit the block device limit and to create more service VMs | 14:11 |
| *** a-pugachev has quit IRC | 14:22 | |
| *** hongbin has joined #openstack-manila | 14:25 | |
| MVenesio | bswartz: Ok, but can i do that manually ? add a new server to create new shares | 14:29 |
| bswartz | MVenesio: it would need to be understood by the driver, so driver changes are needed | 14:32 |
| MVenesio | bswartz: so to be clear, this driver only allows one service instance with 26 shares ? | 14:33 |
| bswartz | we should document that the generic driver has a limitation of 26 (25?) shares per share network | 14:33 |
| vponomaryov | MVenesio: you can workaround it creating new share-network | 14:33 |
| vponomaryov | MVenesio: with same net and subnet | 14:34 |
| bswartz | MVenesio: yes that's currently true -- nothing in the design prevent us from adding support for more service instances but it's code that hasn't been written | 14:34 |
| vponomaryov | bswartz: IT IS documented | 14:34 |
| bswartz | vponomaryov: :-) | 14:34 |
| vponomaryov | https://docs.openstack.org/developer/manila/devref/generic_driver.html#known-restrictions | 14:35 |
| MVenesio | vponomaryov: i see, i'll test that | 14:38 |
| MVenesio | bswartz: perfect, so the driver works well, i've tested creating more than 20 shares, taking snapshots and creating shares from snapshots without errors, so far so good. | 14:46 |
| MVenesio | bswartz: the networking limitation is not a big deal for me, i'm using DVR with neutron so it's easy to spawn an L3 agent and connect a share-server there | 14:47 |
| openstackgerrit | Tom Barron proposed openstack/manila master: Fix pep8 M325 error with python 3.5 https://review.openstack.org/469523 | 14:49 |
| MVenesio | bswartz: i've tested rebooting the service instance and all the shares back to work once the instance is up and running again | 14:49 |
| MVenesio | bswartz: so the big issue here is the HA at the service instances level | 14:50 |
| *** kaisers has joined #openstack-manila | 14:52 | |
| *** jmlowe has quit IRC | 14:53 | |
| *** markstur has joined #openstack-manila | 14:56 | |
| *** markstur has quit IRC | 14:56 | |
| *** markstur has joined #openstack-manila | 14:57 | |
| *** markstur has quit IRC | 14:57 | |
| *** markstur has joined #openstack-manila | 15:03 | |
| *** hoonetorg has quit IRC | 15:04 | |
| *** kaisers has quit IRC | 15:13 | |
| *** hoonetorg has joined #openstack-manila | 15:17 | |
| jungleboyj | bswartz: Is there a list of the supported drivers for Manila somewhere? | 15:31 |
| bswartz | jungleboyj: what do you mean supported? do you mean drivers which you can receive commercial support for? or just an enumeration of the drivers in the tree? | 15:41 |
| jungleboyj | bswartz: Just an enumeration of the drivers in tree. I just went and grabbed them out of the source. :-) | 15:42 |
| bswartz | jungleboyj: that's the most accurate method :-) | 15:42 |
| jungleboyj | I guessed as much given your process is slightly different. | 15:43 |
| ganso | is gerrit loading fine for you? | 15:47 |
| *** lpetrut has quit IRC | 15:47 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack/manila master: [Generic driver] Fix incompatibility with novaclient https://review.openstack.org/469148 | 15:47 |
| vponomaryov | jungleboyj: https://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html | 15:51 |
| vponomaryov | jungleboyj: ^ list of drivers and their features is here | 15:51 |
| jungleboyj | vponomaryov: Nice. Thank you! | 15:52 |
| *** a-pugachev has joined #openstack-manila | 15:53 | |
| vkmc | arnewiebalck, hey! if you can take a look later... I'm trying to fix the test, but your feedback on the kind of data we are recovering would be appreciated https://review.openstack.org/#/c/466586/ | 15:55 |
| *** winston-d_ has joined #openstack-manila | 16:05 | |
| *** kaisers has joined #openstack-manila | 16:10 | |
| *** eharney has quit IRC | 16:29 | |
| *** eharney has joined #openstack-manila | 16:31 | |
| *** kaisers has quit IRC | 16:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/manila master: Updated from global requirements https://review.openstack.org/469570 | 16:37 |
| *** pcaruana has quit IRC | 16:50 | |
| *** kaisers has joined #openstack-manila | 16:54 | |
| *** gaurangt_ has joined #openstack-manila | 16:59 | |
| *** gaurangt- has joined #openstack-manila | 17:01 | |
| *** gaurangt has quit IRC | 17:03 | |
| *** gaurangt_ has quit IRC | 17:04 | |
| *** gaurangt has joined #openstack-manila | 17:21 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack/manila-ui master: Add support of share groups https://review.openstack.org/468899 | 17:22 |
| *** gaurangt- has quit IRC | 17:23 | |
| *** jmlowe has joined #openstack-manila | 17:35 | |
| *** jprovazn has quit IRC | 17:55 | |
| *** eharney has quit IRC | 17:56 | |
| *** kaisers has quit IRC | 17:59 | |
| *** a-pugachev has quit IRC | 18:05 | |
| *** chlong has quit IRC | 18:05 | |
| *** eharney has joined #openstack-manila | 18:09 | |
| *** chlong has joined #openstack-manila | 18:22 | |
| *** kaisers has joined #openstack-manila | 18:37 | |
| *** arnewiebalck_ has joined #openstack-manila | 18:59 | |
| *** arnewiebalck_ has quit IRC | 19:05 | |
| *** arnewiebalck_ has joined #openstack-manila | 19:12 | |
| tbarron | zhongjun: gouthamr bswartz I want to use ensure_share(s) to force my backend to re-create access rules for shares, that is | 19:22 |
| *** lpetrut has joined #openstack-manila | 19:23 | |
| tbarron | I want to re-read all the access rules for each share and enforce them on the backend. | 19:23 |
| tbarron | zhongjun: gouthamr bswartz init_host in the manager currently iterates over all share instances and calls ensure_share, but | 19:24 |
| *** jmlowe has quit IRC | 19:24 | |
| tbarron | zhongjun: gouthamr bswartz but it passes only the share instance and its data, not the access rules themselves | 19:24 |
| tbarron | zhongjun: gouthamr bswartz with the new ensure_shares spec, I could return a dictionary with e.g. {'last-access-rule-sync': <datetime>} and | 19:26 |
| tbarron | that would force a hash-mismatch | 19:26 |
| tbarron | but the driver still just gets passed the shares and isn't allowed to retrieve the access rules from the DB itself. | 19:27 |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/manila master: Use get_notification_transport for notifications https://review.openstack.org/469630 | 19:29 |
| tbarron | zhongjun: gouthamr bswartz I can return {'access_rules_status': constants.MUMBLE} where MUMBLE is something other than STATUS_ACTIVE but -- goutham check me on this -- | 19:30 |
| tbarron | unless there are rules that are still in ACCESS_STATE_APPLYING or ACCESS_STATE_DENYING -- the driver won't end up getting an update | 19:31 |
| tbarron | (even though if it does get an update it will see the full set of rules) | 19:31 |
| tbarron | ensure_share(s) seems very close to being able to address my use case so I want to make sure there isn't something I'm missing | 19:32 |
| tbarron | Or see if there's some tweak we can make that will address this case. | 19:32 |
| *** a-pugachev has joined #openstack-manila | 19:36 | |
| *** pcaruana has joined #openstack-manila | 19:45 | |
| *** arnewiebalck_ has quit IRC | 19:47 | |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/manila master: Add share notifications https://review.openstack.org/466586 | 19:57 |
| bswartz | tbarron: | 19:57 |
| * bswartz fat fingers keyboard | 19:58 | |
| bswartz | tbarron: why do you need to recreate access rules? | 19:58 |
| bswartz | are they not stored somewhere persistent? | 19:58 |
| tbarron | bswartz: in this case the persistent store is stale. | 19:59 |
| tbarron | bswartz: it may allow accesses that are no longer valid. | 19:59 |
| openstackgerrit | Victoria Martinez de la Cruz proposed openstack/manila master: Add share notifications https://review.openstack.org/466586 | 20:00 |
| tbarron | bswartz: we failover manila-share from one node to another. On both nodes we have a ganesha service. And currently we have | 20:00 |
| bswartz | right | 20:00 |
| tbarron | bswartz: no shared state between the two. | 20:00 |
| bswartz | well the lack of shared state seems like a problem | 20:00 |
| tbarron | bswartz: so I want to (re)play the manila DB access rules into a newly started ganesha | 20:01 |
| bswartz | does this config allow for multiple nodes to be active at the same time? | 20:01 |
| tbarron | bswartz: eventually I want to address that, but for now I want to replay. | 20:01 |
| tbarron | bswartz: no, this is currently (for Pike) active/passive, just like manila share | 20:01 |
| bswartz | well my thought is that if you want multiple active NFS servers for the same data, then MUST have shared state somewhere, and that somewhere is the logical place to persist the access rules | 20:02 |
| *** pcaruana has quit IRC | 20:02 | |
| tbarron | we have shared state in the DB if I can just replay it | 20:02 |
| bswartz | you're saying that there is no somewhere, and it's not needed because there's never more than 1 active one | 20:02 |
| tbarron | yes | 20:02 |
| tbarron | eventually I do want to do active/active with shared state, but not in Pike. | 20:03 |
| bswartz | yeah the DB has only the access rules -- not the other things you would need for proper pNFS or lock manager functionality | 20:03 |
| tbarron | And one can imagine corner cases where the driver's state is out of whack and needs refresh without failover. | 20:03 |
| bswartz | tbarron: so why is it not good enough to just update the ones in transient states? | 20:04 |
| tbarron | This is the kind of thing that ensure_shares() can fix for the share(instances) table. | 20:04 |
| bswartz | I agree that ensure_share() can fix this problem but it seems like a scalability nightmare | 20:04 |
| bswartz | there has to be a way to know when it's actually necessary to fix the rules vs time when it's safe to skip it | 20:05 |
| tbarron | bswartz: if I rely at all on old saved state on the node going active it may contain non-transient state that is no longer valid. | 20:05 |
| tbarron | bswartz: yeah, I really only want to replay if I've failed over or have some other reason to suspect the local state. | 20:06 |
| bswartz | I feel like this is a case where we're trying to use manila to solve a problem that the backend should really solve on its own | 20:06 |
| tbarron | bswartz: and we might "reconcile" rather than replay everything if it's too slow. | 20:06 |
| bswartz | the backend is in a better position to know if the rules are up to date or not | 20:06 |
| tbarron | bswartz: the backend would still decide if it wants a refresh and cause a hash mismatch only if it needs the refresh | 20:07 |
| bswartz | if I was designing what I think you're designing, I would have the NFS server writing the access rule list to some stable storage somewhere, so that it could read those rule back and reapply them without involvement from a higher level | 20:07 |
| tbarron | bswartz: well, we are looking at that, but it was for a later phase ... | 20:08 |
| bswartz | consider what happens in the time between when the backend boots up and when manila starts managing it | 20:08 |
| bswartz | are the shares just in a bad state that whole time? | 20:08 |
| bswartz | can't you just have the ganesha node write the access rules table to a hidden file on the share itself? | 20:09 |
| tbarron | to the first question, we were going to synchronize the ganesha service startup after the refresh (if it's needed) | 20:10 |
| bswartz | if you had something like that then the thing managing that node could fix the access rules before ganesha even started serving data | 20:11 |
| tbarron | the hidden file suggestion is interesting. | 20:11 |
| bswartz | I think it's really worth looking into a way to stash the rules on the share itself -- so it's automatically accessible to everything serving that data | 20:11 |
| tbarron | we were thinking more of using ceph or cephfs to hold the access info and have multiple ganeshas access it. | 20:11 |
| bswartz | even in that case it should be possible to stash the rules into an auxiliary ceph object | 20:12 |
| bswartz | I would just be concerned about situation where the ganesha nodes are up and the manila nodes are down | 20:13 |
| bswartz | they need to behave correctly during that time period (minus any access changes which the driver hadn't acknowledged yet) | 20:14 |
| tbarron | yeah, post Pike we would have that situation. In pike ganesha processes will be running 1-1 with manila share on the same nodes and both under pacemaker control. | 20:14 |
| bswartz | the whole point of acknowledging the driver rule updates is to tell Manila that the backend really has those updates | 20:14 |
| bswartz | so manila doesn't send them again | 20:15 |
| tbarron | right | 20:15 |
| bswartz | sending access updates for large numbers of shares simply won't scale | 20:16 |
| tbarron | arguably there should be a way to do it in extraordinary circumstances when backend state is corrupeted or invalid and should be rebuilt. | 20:18 |
| tbarron | but I am not dismissing your remakrks about "a better way" for my immediate goal. | 20:18 |
| *** jmlowe has joined #openstack-manila | 20:32 | |
| tbarron | bswartz: I guess one could use DriverPrivateData for this too; but I agree that ideally one could bring up the backend even if manila nodes are down, which argues for keeping shared-state among ganeshas outside manila. | 20:38 |
| *** chlong has quit IRC | 20:53 | |
| *** jmlowe has quit IRC | 20:59 | |
| *** winston-d_ has quit IRC | 21:00 | |
| *** jmlowe has joined #openstack-manila | 21:03 | |
| *** chlong has joined #openstack-manila | 21:05 | |
| *** pcaruana has joined #openstack-manila | 21:12 | |
| *** kaisers has quit IRC | 21:15 | |
| gouthamr | tbarron: i like your idea of sending up the "access rules are stale" update to the share manager trying to ensure.. so we can run through the update access logic around those shares. this is the same thing as dumping your rules into the private-data or storing them in a persistent store elsewhere and running through the logic yourself, with the caveat of delaying the startup of the share service | 21:20 |
| gouthamr | if we didn't do that, there could be another problem... someone interacting with the share's access rules via manila before the driver has applied existing ones | 21:21 |
| *** harlowja has quit IRC | 21:21 | |
| gouthamr | perhaps we can weave in what exact shares need reconciling... would there ever be a subset of shares or shares belonging to a whole backend? | 21:23 |
| tbarron | gouthamr: well my case right now would be all shares belonging to the whole backend because I have no way of knowing which of them are out-of-date in a failover unless I see the current state of all shares. | 21:24 |
| tbarron | gouthamr: I can imagine other cases where the driver can know that only some share info is out of date though. | 21:24 |
| gouthamr | this would need to work nicely with the current startup check that we perform on in-flight rules | 21:25 |
| tbarron | gouthamr: I think it makes sense to have a capability to refresh some or all share access info even if we avoid doing this (because of delayed startup at scale) every time that we can. | 21:26 |
| gouthamr | either case, in case of a failover like you describe, i don't see why we can't perform update_access calls inside ensure_share... it's going to be slow but necessary | 21:26 |
| tbarron | gouthamr: probably any backend can under some circumstances get out sync with manila and manila DB will be authoritative. | 21:26 |
| gouthamr | :P you'll make ganso smile. | 21:27 |
| tbarron | gouthamr: was ganso arguing this? | 21:27 |
| gouthamr | we removed that logic from startup because we thought it doesn't exist today | 21:27 |
| tbarron | gouthamr: ah | 21:27 |
| gouthamr | we used to call it "recovery mode".. | 21:28 |
| tbarron | gouthamr: well eventually I want to keep shared state outside manila for this. But even then there could be corner cases where we'd need to recover. | 21:28 |
| gouthamr | we have a variant of this in update_access...but that only gets invoked if you switch rules from r/o to r/w or vice-versa | 21:28 |
| tbarron | or compare and reconcile. | 21:28 |
| gouthamr | tbarron: invoking manila's update_access is better than doing your own thing because we perform the state changes necessary in one place | 21:29 |
| *** pcaruana has quit IRC | 21:29 | |
| *** lpetrut has quit IRC | 21:30 | |
| tbarron | the state changes on the backend? That's really all I want to refresh in this case. | 21:31 |
| *** MVenesio has quit IRC | 21:31 | |
| gouthamr | the state changes to the rules | 21:31 |
| *** MVenesio has joined #openstack-manila | 21:31 | |
| *** MVenesio has quit IRC | 21:32 | |
| gouthamr | if the driver comes up, but the rules aren't as per manila's expectations, and someone performs access-rule interactions, does your driver do the right thing | 21:32 |
| gouthamr | ? | 21:32 |
| tbarron | gouthamr: but in this case I'm assuming the rules are correct and I just need to make the backend match what we told manila before when we applied the rules via update_access. | 21:32 |
| tbarron | gouthamr: we told manila we applied rule X and denied/deleted rule Y. But the state on the node to which we are failing over is stale. | 21:33 |
| tbarron | gouthamr: I doesn't show rule X applied or Y deleted. | 21:34 |
| tbarron | gouthamr: so I need to ignore the old state on that node and build it fresh. | 21:34 |
| tbarron | gouthamr: it was just a passive node and not getting any updates. | 21:34 |
| gouthamr | tbarron: is this failover initiated by the manila driver? | 21:36 |
| tbarron | gouthamr: no, by pacemaker. | 21:38 |
| tbarron | gouthamr: down the road, we might do something like that, but right now we're working with ganesha process on same node as manila-share. | 21:38 |
| tbarron | so if someone pulls the plug on that node, pacemaker will start up manila share and a new ganesha on a new node, and corosync will move the IP used for the export location to the new node. | 21:39 |
| tbarron | but the new ganesha instance will currently have no state or will have stale state w.r.t. access rules. | 21:40 |
| tbarron | Eventually we are going to run ganesha independent of manila share, on other nodes. | 21:40 |
| tbarron | And we may run these active-active which will require shared access-rule state outside manila anyways. | 21:41 |
| *** eharney has quit IRC | 21:42 | |
| tbarron | But in Queens we've got pacemaker control of manila-share/ganesha pairs. | 21:42 |
| gouthamr | interesting.. so, you're saying we don't need the update_access stuff, just make ensure_share pass all the access rules if we tell it to | 21:43 |
| tbarron | gouthamr: that would do it. | 21:43 |
| gouthamr | a variant of this exists with replication | 21:44 |
| *** jmlowe has quit IRC | 21:44 | |
| tbarron | gouthamr: do replicas share backend access rule state? or does each replica have its own copy? | 21:44 |
| gouthamr | we don't assume that | 21:45 |
| *** kaisers has joined #openstack-manila | 21:45 | |
| gouthamr | i.e, the share manager just dumps it in the interface, because we don't know/care if you have an efficient way of replicating the access ruels | 21:45 |
| tbarron | gouthamr: yeah, you want replicas to (potentially) be in separate cDOT clusters, right? | 21:45 |
| gouthamr | rules* | 21:45 |
| gouthamr | tbarron: yeah, and mirroring that we setup is just for data, no metadata.. | 21:46 |
| tbarron | gouthamr: but you update both replicas even though only one is active, i.e. all replicas listen for access-updates rather than having to refresh a replica when it goes active? | 21:47 |
| gouthamr | depends... for readable replicas, where the secondary can be mounted, like the ZFS driver supports, we set the rules on create and I suppose every rule update is mirrored | 21:48 |
| tbarron | gouthamr: while we are using pacemaker to control ganesha services I don't think we can have all of them accept updates; so we're stuck with loading them at startup after a failover. | 21:48 |
| gouthamr | vponomaryov would know | 21:48 |
| gouthamr | for cDOT, we support "dr", so we keep the secondary in the dark, unmounted | 21:49 |
| gouthamr | so, we only apply them on promotion | 21:49 |
| tbarron | gouthamr: ok, where do you get the info from on promotion, the manila DB? | 21:49 |
| tbarron | gouthamr: private data? | 21:49 |
| tbarron | gouthamr: or you could keep it somewhere in cDOT? | 21:50 |
| gouthamr | yep, the manager passes them in the interface: https://github.com/openstack/manila/blob/master/manila/share/manager.py#L1947 | 21:50 |
| gouthamr | tbarron: we have it on cDOT.. but the feature was designed to not make that assumption | 21:51 |
| tbarron | gouthamr: ack | 21:51 |
| gouthamr | so we wrote the driver impl with that in mind | 21:51 |
| gouthamr | we don't mirror rules, and we may not rely on the primary being around to query, so we just expect manila to tell us | 21:52 |
| tbarron | gouthamr: well, this is good food for thought | 21:53 |
| tbarron | gouthamr: atm I'm inclined to think (1) we should implement the new ensure_shares in a way that allows for a full refresh of the backend access rules from the manila DB. | 21:54 |
| tbarron | gouthamr: (2) we should be working to have our backend not depend on that except in extraordinary circumstances | 21:55 |
| tbarron | gouthamr: we need to be building shared state for our backend anyways, but from a practical standpoint with schedules I want | 21:56 |
| gouthamr | +1 i think (1) is sustainable.. it was the kind of extensibility that i was hoping ensure shares would bring.... | 21:56 |
| tbarron | gouthamr: to minimize my dependencies on external teams (e.g. the ganesha team for new features) so I'm interested in manila-based solutions as well. | 21:56 |
| gouthamr | agree... welcome back tbarron :P | 21:57 |
| gouthamr | lol, meant to say that yesterday | 21:58 |
| tbarron | gouthamr: yeah, the promise of ensure shares was that we could refresh the DB when we have backend info that is needed, and work in the other direction when the backend knows it needs a refresh. | 21:58 |
| tbarron | gouthamr: oh thanks, I don't know that I'm fully back yet, digging out of email and causing confusion to my teammates. | 21:58 |
| gouthamr | arnewiebalck zhongjun: i'm reviewing https://review.openstack.org/#/c/469148/ and the quota-class-sets logic seems weird.. thanks for the use cases and the links.. | 21:59 |
| gouthamr | tbarron: yep.. and your mind's probably back in mexico.. :) | 22:01 |
| tbarron | gouthamr: yup, at least some of the time | 22:01 |
| *** harlowja has joined #openstack-manila | 22:06 | |
| *** rpittau has quit IRC | 22:07 | |
| *** gouthamr has quit IRC | 22:20 | |
| bswartz | tbarron: sorry I blitzed out on you -- I'm back now for a bit | 22:21 |
| bswartz | tbarron: driver private data is a bad idea | 22:21 |
| bswartz | tbarron: I agree that in extraordinary cirumstances, there should be a way to refresh all the rules -- in my mind that would be something like a SQL query the forces the access rule state for all the shares to out of sync | 22:22 |
| tbarron | bswartz: hey, I have no problem with async communication on irc :) | 22:22 |
| bswartz | tbarron: I'm still hung up on the case where manila is down and ganesha is up | 22:23 |
| bswartz | what if someone restarts ganesha and does nothing at all to manila -- your ensure_share approach would not solve that case | 22:23 |
| bswartz | I can think of so many cases where ensure_share is too little or too late to solve the problem | 22:23 |
| tbarron | bswartz: I'm not really inclined to use driver private data (unless as a temporary expedience if I can't get a better solution from another group in a particular release cycle). | 22:23 |
| bswartz | all of the truly robust solutions I can imagine involve the backend doing this itself | 22:24 |
| tbarron | bswartz: tend to agree | 22:24 |
| bswartz | maybe I'm just biased by tradition though | 22:24 |
| tbarron | bswartz: but I think goutham has given good arguments for having the refresh capability as well | 22:24 |
| bswartz | I need to read the scrollback from gouthamr | 22:24 |
| bswartz | or maybe gouthamr will give me a TL;DR ;-p | 22:25 |
| tbarron | bswartz: it just shouldn't be our goto-approach for ganesha state between instances | 22:25 |
| *** rpittau has joined #openstack-manila | 22:26 | |
| tbarron | bswartz: i'd only use it (or driver private data if I had to) for the pacemaker-controlled manila-share/ganesha deployment. | 22:26 |
| bswartz | tbarron: what about having the admin execute a SQL query in cases where his ganesha instance(s) got hosed? | 22:26 |
| bswartz | to force all the shares to out of sync access state | 22:27 |
| tbarron | which we're only going to do for one release, as tech-preview downstream, aiming to move ganesha off of controllers and outside of pcs control. | 22:27 |
| tbarron | bswartz: well, I think the spirit of the new ensure_shares spec was that we could (1) avoid manila DB/backend state synchronization on ordinary startupm, but (2) | 22:28 |
| tbarron | be able to trigger a reconciliation in either direction when the backend knows best. | 22:28 |
| *** xyang1 has quit IRC | 22:28 | |
| tbarron | bswartz: so yeah, we could say, don't use manila for circumstances like #2, but that's not the way we were going. | 22:29 |
| tbarron | bswartz: gouthamr pointed out that we already do this kind of access-rule refresh of backend from DB with share replication. | 22:30 |
| tbarron | bswartz: so I'm inclined to think that yes, my backend should use shared storage outside manila to keep its access-rule state, but that | 22:31 |
| tbarron | there should be a way to refresh in an emergency, as if the backend finds that its shared (or non-shared) view of the state is corrupt for some reason. | 22:32 |
| tbarron | That second could be done outside of manila itself with sql but it seems like the kind of thing that the new ensure_shares spec was aiming at. | 22:32 |
| tbarron | bswartz: "what if someone restarts ganesha and does nothing at all to manila -- your ensure_share approach would not solve that case" | 22:38 |
| tbarron | bswartz: this is being considered only for the case where ganesha and manila share live and die together on the same node under pacemaker control where manila-share leads | 22:39 |
| tbarron | bswartz: restarting ganesha on the same node won't encounter stale state any more likely than restart of any other backend. | 22:39 |
| tbarron | bswartz: it's failover to another node where we'd hit stale state and in that case manila-share has failed over as well. | 22:40 |
| tbarron | bswartz: I guess there's a pathological case there where manila-share doesn't start up successfully and ganesha does, but I think pacmaker will force another failover in that case. | 22:41 |
| * tbarron doesn't say that he likes pacemaker control but that's what we're working with near term | 22:41 | |
| *** chlong has quit IRC | 23:00 | |
| *** a-pugachev has quit IRC | 23:09 | |
| *** gouthamr has joined #openstack-manila | 23:11 | |
| bswartz | tbarron: understood | 23:39 |
| bswartz | tbarron: so getting back to the original question -- what is needed from the share manager side to enable this use case? | 23:39 |
| *** david_8 has quit IRC | 23:52 | |
| *** carthaca_ has quit IRC | 23:52 | |
| *** databus23_ has quit IRC | 23:52 | |
| *** sapcc-bot has quit IRC | 23:52 | |
| *** david_ has joined #openstack-manila | 23:53 | |
| *** carthaca_ has joined #openstack-manila | 23:53 | |
| *** sapcc-bot has joined #openstack-manila | 23:53 | |
| *** databus23_ has joined #openstack-manila | 23:53 | |
| *** ganso has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!