| *** zengyingzhe_ has joined #openstack-smaug | 00:41 | |
| *** WANG_Feng has quit IRC | 01:37 | |
| *** WANG_Feng has joined #openstack-smaug | 01:38 | |
| *** zengyingzhe_ has quit IRC | 01:59 | |
| *** jhesketh has quit IRC | 02:05 | |
| *** jhesketh has joined #openstack-smaug | 02:05 | |
| *** zengyingzhe_ has joined #openstack-smaug | 02:35 | |
| *** zengyingzhe_ has quit IRC | 02:44 | |
| *** WANG_Feng has quit IRC | 04:54 | |
| *** WANG_Feng has joined #openstack-smaug | 05:39 | |
| *** zengyingzhe_ has joined #openstack-smaug | 07:07 | |
| zengyingzhe_ | saggi, hi | 08:10 |
|---|---|---|
| saggi | zengyingzhe_: hi | 08:30 |
| zengyingzhe_ | how are you | 08:30 |
| zengyingzhe_ | the interface create_checkpoint from ProtectionProvider, is it the entry for protecting resources? | 08:32 |
| zengyingzhe_ | Or it's just responsible for creating a checkpoint in bank? | 08:33 |
| saggi | It creates the checkpoints and asynchrniously fills it up with data | 08:37 |
| zengyingzhe_ | So it's also responsible for doing the operation by executing the task returned by resource plugins? | 08:39 |
| zengyingzhe_ | If so, why there just a root_resource parameter, shouldn't there be other parameters, such as plan info? | 08:40 |
| zengyingzhe_ | Otherwise, where does the context passed to get_pre/post_task come from? | 08:42 |
| saggi | create_checkpoint should get a checkpoint instance as a parameter | 08:43 |
| saggi | resolve the plan_id and create the full tree | 08:43 |
| saggi | zengyingzhe_: ^^^^^^ | 08:44 |
| zengyingzhe_ | You mean the checkpoint will be created first, before calling create_checkpoint in provider? | 08:45 |
| zengyingzhe_ | I'm quite confused. | 08:46 |
| *** yinweiphone has joined #openstack-smaug | 09:09 | |
| yinweiphone | saggi, are you there? | 09:11 |
| yinweiphone | wanna confirm the output of graph walker | 09:11 |
| yinweiphone | i suppose the graph walker listener will watch on the traversing of the resource graph | 09:12 |
| yinweiphone | and if we implement a listener class who will produce task per resource, and we link those tasks with the dependencies between resource nodes, then we will get a graph flow or sevreral graph flows | 09:15 |
| yinweiphone | so the output of the graph walker could be a set of graph flows by this listener implementation, right? | 09:16 |
| yinweiphone | pls leave your comment when you see the msg, saggi | 09:17 |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** openstackgerrit has joined #openstack-smaug | 09:17 | |
| *** yinweiphone has quit IRC | 09:20 | |
| saggi | yinweiphone: We would walk from each source in our graph. This would make sure we iterate all the nodes in the correct order | 09:21 |
| zengyingzhe_ | saggi, You mean the checkpoint will be created first, before calling create_checkpoint in provider? | 09:22 |
| saggi | The checkpoint is given from the user in the post request | 09:22 |
| saggi | Then we need to process it (add full plan and resource tree) and persist it with the state "in progress". | 09:23 |
| *** yinweiphone has joined #openstack-smaug | 09:23 | |
| saggi | than we asynchronously start backing up | 09:23 |
| saggi | That means that the user can now query the checkpoint ID for full status information | 09:23 |
| saggi | and if we fail we have a complete picture of what happened and how to clean up | 09:23 |
| zengyingzhe_ | The post request doesn't include checkpoint. | 09:24 |
| zengyingzhe_ | Just plan info | 09:24 |
| yinweiphone | I'm thinking the revert logic | 09:25 |
| yinweiphone | any task raises exception should revert its traces like record under checkpoint | 09:26 |
| zengyingzhe_ | The revert here means totally undo the task, such as remove the backup object? | 09:27 |
| yinweiphone | shall we revert the whole flow if server all task raised exception? or we keep the checkpoint and turn it to failure status at last | 09:27 |
| yinweiphone | zengyingzhe_: yes, revert does include that work | 09:28 |
| zengyingzhe_ | I think setting the failure status would be much easier. | 09:28 |
| yinweiphone | zengyingzhe_: we could give graph walker the checkpoint as input, and it could put it into task it generates | 09:29 |
| yinweiphone | zengyingzhe_: then user will decide whetherclean up garbage, right? | 09:30 |
| zengyingzhe_ | yinweiphone: So there will be another type of graph walker, for checkpoint? | 09:32 |
| yinweiphone | zengyingzhe_: sounds better. then we just add a retry task on retriable exception, otherwise we just tolerate the task failure until whole flow finishes | 09:32 |
| yinweiphone | zengyingzhe_: graph walker should be the same, we just implement different listeners | 09:33 |
| yinweiphone | zengyingzhe_: ok, you mean how to pass checkpoint to tasks | 09:35 |
| yinweiphone | yes, then we won't pass checkpoint to graph walker, but build a listener class with checkpoint as field. | 09:36 |
| yinweiphone | just my guessing | 09:37 |
| yinweiphone | could have saggi to confirm | 09:38 |
| saggi | yinweiphone: I'll catch up in 2 min | 09:38 |
| zengyingzhe_ | yinweiphone: Could implement like that. But we should confirm if we want to revert or just set checkpoint status first. | 09:39 |
| yinweiphone | tongue has question how to pass checkpoint to tasks built during graph walking | 09:39 |
| yinweiphone | yingzhe has question | 09:39 |
| saggi | sorry had to send the agenda for today's meeting | 09:40 |
| yinweiphone | zengyingzhe_: I prefer persist failures in checkpoint,and have user to choose delete them or not. this is the routine of openstack | 09:41 |
| yinweiphone | saggi: np | 09:41 |
| zengyingzhe_ | yinweiphone, I agree with you | 09:41 |
| yinweiphone | saggi: could talk to you later when you're free | 09:42 |
| zengyingzhe_ | And this could simplify the logic of protection service. | 09:42 |
| * saggi is catching up with chat | 09:42 | |
| yinweiphone | zengyingzhe_: sorry, I missed saggi's peeve answers | 09:43 |
| yinweiphone | zengyingzhe_: did he agree the output of graph walking is a set of flows? | 09:43 |
| * saggi is done | 09:44 | |
| saggi | OK, so we need to keep the checkpoint if it fails | 09:44 |
| saggi | otherwise the user has no way to check what the error was | 09:44 |
| zengyingzhe_ | yinweiphone, i didn't discuss that with saggi | 09:44 |
| saggi | it should be in an invalid state with all data outside the bank cleared. The user needs to manually mark it for deletion. | 09:45 |
| saggi | Otherwise the user might miss the error. | 09:45 |
| yinweiphone | saggi: is it necessary to retrieve prev task or post task from each plugin? | 09:45 |
| yinweiphone | the resource graph has built dependency already | 09:46 |
| yinweiphone | we just have build task on walking each resource, and link those tasks by the dependency of the graph | 09:47 |
| saggi | I am starting to think we might just need one call instead of two. The original two calls were for the version were we didn't have the sub-graph passed. | 09:47 |
| yinweiphone | yes | 09:47 |
| yinweiphone | agree one call per plugin would be enough, since we've already built dependency on resource graph | 09:49 |
| * saggi is thinking about it | 09:49 | |
| yinweiphone | saggi: beyond this, we maybe not involve plugin to construct task | 09:50 |
| zengyingzhe_ | saggi, you said all data outside the bank cleared, so the data already backuped should be removed before setting checkpoint to a failure status? | 09:50 |
| yinweiphone | zengyingzhe_: backed up data should be cleared together with removing checkpoint | 09:50 |
| saggi | yinweiphone: Who will construct the task than? The plugin is the only one that knows what to do. | 09:51 |
| yinweiphone | saggi: graph walker listener will | 09:51 |
| saggi | zengyingzhe_: I did but thinking about it again I'd say that we should leave everything as is when we fail. That way we know data is only deleted in deletion state which is simpler. | 09:52 |
| zengyingzhe_ | saggi, OK, understand. | 09:52 |
| yinweiphone | plugin could be a param of the task, when the task is constructed by the liestener , it will pass in the plugin | 09:52 |
| saggi | yinweiphone: It seems like it's the wrong way around. By creating the task itself it can push all the information it will need when the task is actually run. | 09:53 |
| saggi | yinweiphone: It could also just do nothing or create a chain of smaller tasks. | 09:54 |
| saggi | If the task just get the plugin how will it know the context? | 09:54 |
| yinweiphone | saggi, the execute of the task would just be call protect or restore of the plugin | 09:55 |
| zengyingzhe_ | yinweiphone, you mean just telling plugin what to do, like protect or restore, then plugin will archieve it internally? | 09:58 |
| yinweiphone | the context is still kept inside the plugin, just it doesn't construct task inside plugin | 09:58 |
| saggi | This removes the task preperation stage. This means that we can't group calls for example. | 09:58 |
| saggi | If we want to run the plugin just once for all volumes. | 09:59 |
| yinweiphone | it sounds weird to have plugin to generate task in plugin | 09:59 |
| saggi | It's just separating the planning phase from the execution phase. For example, if we want to support OceanStoreDJ we need to actually build a policy of all the resources we want to back up and pass them to OSDJ. This will actually be one task (executing the policy) for multiple disconnected resources. | 10:02 |
| yinweiphone | yes, i agree with the separation of the two phases | 10:03 |
| yinweiphone | and i understand the graph walking is to prepare tasks | 10:04 |
| zengyingzhe_ | task is generated by listener, and plugin only need to provide one interface like run_task(), and run_task will execute this task according to the task content, for example, protect or restore, is this what you said about? | 10:10 |
| saggi | By returning a Task the task could be anything. It could be Task(target=self.protect, args=(...)) or it could be Flow(Task(target=self.phaseA), Task(target=phaseB)). | 10:12 |
| saggi | We don't enforce and interface | 10:12 |
| saggi | yinweiphone: Thinking about it some more I realize that without the `post` callback we have no way of aggregating tasks since in the original design it would have been done in the post call of the parent. | 10:15 |
| zengyingzhe_ | yes, it would be more flexible if task returned by plugin, it could be anything as long as plugin can understand. | 10:15 |
| zengyingzhe_ | saggi, back to my question | 10:16 |
| saggi | zengyingzhe_: Could you repeat it :) too much talk | 10:17 |
| zengyingzhe_ | The checkpoint is given from the user in the post request. There is no this parameter in checkpoint post request. | 10:17 |
| *** yinweiphone has quit IRC | 10:22 | |
| saggi | zengyingzhe_: http://ur1.ca/oemnw | 10:22 |
| saggi | zengyingzhe_: from the API doc | 10:22 |
| saggi | zengyingzhe_: ? | 10:31 |
| zengyingzhe_ | saggi, I'm checking it. :) | 10:32 |
| zengyingzhe_ | You go to have lunch first, we can talk later. | 10:32 |
| saggi | zengyingzhe_: sure | 10:33 |
| zengyingzhe_ | See you later. | 10:33 |
| *** openstackgerrit has quit IRC | 11:17 | |
| *** openstackgerrit has joined #openstack-smaug | 11:17 | |
| *** WANG_Feng has quit IRC | 11:18 | |
| *** WANG_Feng has joined #openstack-smaug | 11:31 | |
| zengyingzhe_ | saggi, you there? | 11:42 |
| *** zengyingzhe_ has quit IRC | 12:24 | |
| openstackgerrit | chenying proposed openstack/smaug: Basic design doc for API Service https://review.openstack.org/266338 | 12:35 |
| *** zhonghua-lee has joined #openstack-smaug | 13:54 | |
| *** zengyingzhe has joined #openstack-smaug | 13:55 | |
| zengyingzhe | Hi | 13:56 |
| saggi | zengyingzhe: hi | 13:56 |
| zengyingzhe | Ready for meeting? :) | 13:56 |
| zhonghua-lee | hi,saggi | 13:57 |
| saggi | zengyingzhe: yes | 13:57 |
| saggi | there are 3 minutes on my clock until it starts | 13:57 |
| zengyingzhe | yes | 13:57 |
| *** WANG_Feng has quit IRC | 13:58 | |
| *** WANG_Feng has joined #openstack-smaug | 13:59 | |
| zhonghua-lee | who can give me the catalog link please? | 13:59 |
| *** chenzeng has joined #openstack-smaug | 14:52 | |
| *** zengyingzhe has quit IRC | 14:52 | |
| chenzeng | saggi:sorry, it is 11:00 pm in my country. can we talk tomorrow? I think, I understand your design about OperationEngine. | 14:54 |
| saggi | chenzeng: Sure, good night! | 14:55 |
| chenzeng | saggi:thanks. | 14:57 |
| *** chenzeng has quit IRC | 15:03 | |
| *** zhonghua-lee has quit IRC | 15:23 | |
| *** zhonghua-lee has joined #openstack-smaug | 15:24 | |
| *** zhonghua-lee has quit IRC | 15:28 | |
| openstackgerrit | Saggi Mizrahi proposed openstack/smaug: Pluggable protection provider doc https://review.openstack.org/262264 | 16:51 |
| *** gampel has joined #openstack-smaug | 17:31 | |
| *** gampel has quit IRC | 18:06 | |
| *** wangfeng_yellow has quit IRC | 21:05 | |
| *** wangfeng_yellow has joined #openstack-smaug | 21:06 | |
| *** gampel has joined #openstack-smaug | 23:16 | |
| *** gampel has quit IRC | 23:46 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!