| *** yinwei_computer has joined #openstack-smaug | 01:42 | |
| *** yinwei_computer has quit IRC | 02:23 | |
| *** yinwei_computer has joined #openstack-smaug | 02:24 | |
| *** chenying has joined #openstack-smaug | 02:36 | |
| *** c00281451 has joined #openstack-smaug | 02:42 | |
| *** c00281451 is now known as chenzeng | 02:43 | |
| openstackgerrit | xiangxinyong proposed openstack/smaug-dashboard: [Protection Plans] Submit index view function https://review.openstack.org/306897 | 03:34 |
|---|---|---|
| *** yuval has joined #openstack-smaug | 05:59 | |
| yuval | ping yinweiishere yinwei_computer | 06:17 |
| yuval | ping yinweiishere yinwei_computer | 07:20 |
| yinweiishere | hello yuval | 07:41 |
| yuval | hey yinweiishere | 07:51 |
| yinweiishere | hi | 07:52 |
| yinweiishere | how are you | 07:52 |
| yuval | very well, you? | 07:52 |
| yinweiishere | me too | 07:53 |
| yinweiishere | thanks | 07:54 |
| yinweiishere | do you wanna check your design about task flow with me? | 07:54 |
| yuval | have you read the mail I sent you? | 07:54 |
| yinweiishere | yes | 07:54 |
| yuval | what do you think? | 07:54 |
| yinweiishere | hmm, but I don't quite get the point to divide paralle flow and sync flow | 07:54 |
| yinweiishere | especially about the sync flow is sth. related to heat? | 07:55 |
| yuval | the sync flow is for the protection plugin to implement | 07:56 |
| yinweiishere | and how would you link those paralle flows returned by each plugin? | 07:56 |
| yinweiishere | still by dependency or just let all parallel flows run on parallel? | 07:56 |
| yuval | have you seen the graph? the parallel flow is linked to the same resource sync flow | 07:57 |
| yinweiishere | yes, I saw the graph, but still a bit confused | 07:57 |
| yuval | right: | 07:57 |
| yinweiishere | do you mean sth. like this: | 07:57 |
| yuval | ? | 07:58 |
| yinweiishere | volume.sync->volume.parallel, sever.sync->server.parallel | 07:58 |
| yinweiishere | but how to link volume.paralle and server.parallel? | 07:58 |
| yinweiishere | by dependency of resource graph or let volume.parallel run on parallel with server.parallel? | 07:59 |
| yuval | they are not linked, that is why they can run in parallel | 07:59 |
| yinweiishere | hmm | 08:00 |
| yinweiishere | what's the solution for restoration dependency? | 08:00 |
| yuval | exactly the same, except the attachment can be done by heat | 08:01 |
| yuval | I'm still not 100% familiar with heat, I'm still reading about it | 08:01 |
| yinweiishere | that's fine | 08:02 |
| yinweiishere | just try to figure out your design | 08:02 |
| yuval | sure | 08:02 |
| yinweiishere | so what's the benefit of dividing sync flow and parallel flow? | 08:02 |
| yuval | with the other design, we build a task flow, but in the end, each task just "begins" its process, and is not sync. in this case, I just can't understand why we build the taskflow. each resource can launch its process while we walk the graph | 08:03 |
| yuval | I thought we can offer the protection plugin two places where it can do job: | 08:04 |
| yinweiishere | it seems to me that the idea is sth. like each resource just return a task flow, where the flow is internally composed by several tasks with dependency; but externally, all task flows returned by each resource could run on parallel without dependency | 08:04 |
| yuval | are you speaking about my idea or the other idea | 08:04 |
| yuval | ? | 08:04 |
| yinweiishere | your idea | 08:04 |
| yinweiishere | I mean your idea looks sth. like that | 08:04 |
| yuval | no, that is not what I meant | 08:05 |
| yinweiishere | if we take sync flow and parallel as one flow | 08:05 |
| yuval | no, we don't take them as one flow | 08:05 |
| yinweiishere | since we link them as sync-flow->parallel-flow | 08:05 |
| yinweiishere | but if you link them, they will look like a flow unit | 08:06 |
| yinweiishere | pls. go ahead, that's what confused me | 08:06 |
| yuval | sure | 08:06 |
| yuval | one moment | 08:07 |
| yinweiishere | np | 08:07 |
| yinweiishere | you could share any personal storage with me now | 08:07 |
| yinweiishere | as I'm in open lab :) | 08:07 |
| yuval | great :) | 08:09 |
| yuval | I sent you a small python script | 08:10 |
| yuval | which shows a small demo of the sync-flow, parallel-flow | 08:11 |
| yuval | Basically, the idea is: | 08:11 |
| yuval | we want to let the protection plugin perform operations: | 08:11 |
| yuval | 1. In parallel, where it doesn't block other resources and protection plugins, minimizing the amount of time a protect/restore takes | 08:12 |
| yuval | 2. In sync, when we do job for a resource, we can be 100% sure its child resources were handles by their protection plugins. maybe we want to affect them, gather information about them, and we need them to be handled by their protection plugins for that | 08:13 |
| yuval | A protection plugin can say: I don't care about my child resources, I just want to do all work in parallel - and implement only the parallel flow | 08:14 |
| yuval | Or maybe, it must take action only after its child resources were handled, because it needs info about them. So it can implement the sync flow also | 08:14 |
| yuval | In the other suggestion, where every task starts the operation and returns - I can't understand why we build a taskflow. we could launch the async operation while we traverse the tree | 08:15 |
| yuval | </end block> :) | 08:15 |
| yuval | yinweiishere: ? | 08:23 |
| yinweiishere | sorry, was doing other thread | 08:24 |
| yinweiishere | back | 08:24 |
| yinweiishere | reading | 08:24 |
| yuval | did you receive the mail? | 08:25 |
| yinweiishere | no, open lab has no access to huawei mail box :( | 08:25 |
| yuval | 1 second then | 08:25 |
| yinweiishere | could you pls. send it to this mail box: yinweiishere@icloud.com | 08:25 |
| yuval | https://drive.google.com/file/d/0B8kA9zmFxeh8elplVndPWGZDcW8/view?usp=sharing | 08:26 |
| yuval | Sent | 08:26 |
| yinweiishere | got it | 08:28 |
| yinweiishere | haven't read the python code, for your descriptio above: | 08:30 |
| yinweiishere | do you mean we will have all parallel flows run on parallel, while link all sync tasks with child-parent dependency in resource graph? | 08:32 |
| yuval | yes. sync tasks are linked according to resource graph. parallel task are linked to their resource sync task | 08:34 |
| yuval | **sync tasks are linked to other sync tasks according to resource graph | 08:34 |
| yinweiishere | I think the point to divide sync and parallel tasks here is the define the sync-mode of the task execute(). | 08:39 |
| yuval | not sure I understand what you mean | 08:39 |
| yuval | oh, ok | 08:39 |
| yuval | it is not just that | 08:40 |
| yinweiishere | I mean, for sync flow, we define that the execute() of the task here is to complete its task inside the life time of execute() | 08:40 |
| yuval | it is that when a sync task execute() is reached, we _know_ that its children have finished | 08:40 |
| yinweiishere | yes, it looks more clear for the behavior of a task flow | 08:41 |
| yinweiishere | but I have conern on the complex of protection plugin | 08:41 |
| yuval | we can simplify. instead of sync-flow, make it sync-method | 08:41 |
| yuval | and parallel-method | 08:42 |
| yuval | (names can change :) ) | 08:42 |
| yuval | I agree that demanding flows from protection plugin makes it much more complex | 08:42 |
| yuval | so, we can just make it two methods, which a protection plugin can implement | 08:42 |
| yinweiishere | and it seems that one protection plugin should take care of others implementation | 08:43 |
| yuval | why is that? | 08:43 |
| yinweiishere | what if child resource protection plugin returns only parallel flow, which means it doesn't care its own child | 08:43 |
| yinweiishere | but the parent resource protection plugin returns only sync flow, which means it cares its child task has been handled or not | 08:44 |
| yuval | the only demand from the protection plugin is: the sync-flow (or method) must return when the job is complete for this resource. you must sync it | 08:44 |
| yinweiishere | yes | 08:45 |
| yuval | that is a logical request from a protection plugin. not very different from demanding a status_getter function | 08:45 |
| yinweiishere | say, sever plugin implements a sync-flow, which requires its child (volume) to return a sync flow as well | 08:45 |
| yinweiishere | but volume protection plugin only returns a parallel flow which doesn't ensure its completion during execute life time | 08:46 |
| yuval | if a protection plugin doesn't return a sync-flow, a no-op sync-flow is returned. | 08:46 |
| yinweiishere | which means the requires/provids doesn't meet in task flow | 08:46 |
| yinweiishere | but server's task has requires where its children tasks can't provide | 08:47 |
| yuval | then the volume protection plugin violates the demand that its flow is complete once the operation is compelte | 08:47 |
| yinweiishere | yes, I mean there will be dependencies among each protectio plugin of the same provider | 08:47 |
| yuval | I don't understand | 08:48 |
| yuval | can you rephrase? | 08:48 |
| yinweiishere | rethink of this problem | 08:50 |
| yinweiishere | there will be case like sync-flow->parallel-flow1, sync-flow->parallel-flow2 | 08:50 |
| yinweiishere | how to handle this case? | 08:50 |
| yuval | there is not such case | 08:51 |
| yinweiishere | where parallel-flow1 and parallel-flow2 could run on parallel but sync-flow should wait those 2 to end | 08:51 |
| yinweiishere | why | 08:51 |
| yuval | because each resource returns 1 parallel flow, which is linked to the same resource sync-flow | 08:51 |
| yinweiishere | and its sync-flow is to sync its parallel flow's status? | 08:53 |
| yuval | either the sync-flow or the parallel-flow can do it | 08:53 |
| yuval | but one of them must | 08:53 |
| openstackgerrit | chenying proposed openstack/smaug: Add fullstack test to smaug https://review.openstack.org/306996 | 08:55 |
| yinweiishere | I got your idea. It's good to make the flow organization more clear. At the same time, we have to keep the simplicity of protection plugin as well. So there should be a balance here. | 08:56 |
| yinweiishere | pls. send the mail to dev mail list | 08:56 |
| yinweiishere | maybe other guys will have suggestions here. | 08:57 |
| yuval | sure | 08:58 |
| yuval | anyway, consider the other solution - why do we need the taskflow, if dependency is only for starting the operation for each resource? | 08:58 |
| yinweiishere | for protection, if each plugin only calls async openstack API, there's no need of task flow | 08:59 |
| yinweiishere | but that's only for reference implementation | 08:59 |
| yinweiishere | say, if there's consistency requirment between server and volume, then task flow is required | 09:00 |
| yuval | and then the volume will be implemented a sync? | 09:00 |
| yuval | *as sync | 09:01 |
| yinweiishere | like server-freeze-io-task <- volume-backup; server-freeze-io-task <- server-backup-snapshot | 09:01 |
| yinweiishere | not necessarily | 09:01 |
| yinweiishere | just the depdency task is sync | 09:01 |
| yuval | so the server task needs to sync the operation started by the volume task? | 09:02 |
| yinweiishere | yes | 09:02 |
| yuval | what if they are from different protection plugins? | 09:03 |
| yinweiishere | but for restoration, the dependency graph is built to represent the rebuilt dependencies. so tasks should be sync and linked with the dependencies represented in resource graph | 09:03 |
| yinweiishere | if the server/volume want to ensure fs level consistency, the two have to operate. so the two protection plugins of the same provider must combine to ensure this feature. | 09:04 |
| yinweiishere | it's protection plugin specific | 09:05 |
| yinweiishere | the two have to *cooperate | 09:05 |
| yuval | I see | 09:06 |
| yuval | I'll summarize the two suggestions and send to the mailing list | 09:06 |
| *** yuval has quit IRC | 15:02 | |
| *** zhongjun_ has quit IRC | 15:06 | |
| *** zhongjun_ has joined #openstack-smaug | 15:07 | |
| *** yuval_ has joined #openstack-smaug | 16:01 | |
| *** yuval_ is now known as yuval | 16:04 | |
| *** yuval has quit IRC | 17:15 | |
| *** yuval has joined #openstack-smaug | 17:15 | |
| *** yuval is now known as Guest60999 | 17:16 | |
| *** Guest60999 has quit IRC | 17:16 | |
| *** yuval_ has joined #openstack-smaug | 17:51 | |
| *** yuval_ has quit IRC | 17:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!