Monday, 2016-04-11

openstackgerritchenying proposed openstack/smaug: Add name property to protectable instances model  https://review.openstack.org/30208603:22
openstackgerritsmile-luobin proposed openstack/smaug: Implement restoration  https://review.openstack.org/30327703:24
openstackgerritsmile-luobin proposed openstack/smaug: Implement restoration  https://review.openstack.org/30327703:26
openstackgerritsmile-luobin proposed openstack/smaug: Implement Cinder Restoration  https://review.openstack.org/30327703:34
*** gampel has quit IRC06:00
*** gampel has joined #openstack-smaug06:15
*** CrayZee has quit IRC06:21
*** CrayZee has joined #openstack-smaug06:24
*** yuval has joined #openstack-smaug06:36
openstackgerritchenying proposed openstack/smaug: Add show protectables instance endpoint  https://review.openstack.org/30214406:43
openstackgerritchenying proposed openstack/smaug: Add a field parameters to resource plans  https://review.openstack.org/30273006:53
*** saggi has joined #openstack-smaug07:02
saggihi everyone07:03
xiangxinyonghello saggi07:18
xiangxinyonggood morning07:19
saggixiangxinyong: good morning!07:25
xiangxinyongsaggi:^-^07:26
xiangxinyongsaggi: please take a look at this two link.07:28
xiangxinyonghttps://etherpad.openstack.org/p/createplannew07:28
xiangxinyonghttps://etherpad.openstack.org/p/restorecheckpointnew07:28
xiangxinyongThese two links are about the create plan and restore from checkpoint interface.07:29
xiangxinyongdo you agree?07:29
*** yinwei_computer has joined #openstack-smaug07:30
yuvalyinwei_computer: ping07:30
yinwei_computerping yuval07:30
yuvalhey :)07:30
yuvalhow are you?07:30
yinwei_computerdoing good07:30
yinwei_computerthanks07:30
yinwei_computerhow about you?07:30
yinwei_computerI saw you committed a patch for grah test07:30
yinwei_computergraph07:31
yuvalvery well07:31
yuvalyes07:31
yinwei_computerhave saggi aligned his graph de/serialization with you?07:31
yuvalI'm almost done with the graph de/serializtion07:32
yinwei_computerwow07:32
yuval:)07:32
yinwei_computerawesome work07:32
yuvaldo you have a couple of minutes to speak about the protection flow?07:32
yuvalhttps://github.com/openstack/smaug/blob/master/doc/source/specs/protection-service/protection-service.rst07:33
yinwei_computersure07:33
yinwei_computeractually I also have another issue to check with you07:33
yinwei_computerbut we could see this one first07:33
yuvalwhat is it?07:33
yuvalok07:33
yinwei_computerso what's up with this design doc?07:34
yinwei_computeryuval?07:35
yuvalAfter reading this document, I'm not sure I understand what is SyncCheckpointStatusTask is and what is its role. From what I figured, every task returned by a resource (i.e Cinder plugin protect) returns a task/flow which is finished when the protect itself is complete. Considering this, why are the "status_getters" neccesary?07:35
yinwei_computersure, take cinder as eg07:36
yinwei_computerprotect() in cinder volume means cinder.backup, which is async07:36
yinwei_computerthat's what we do in the protection task flow.07:37
yinwei_computerafter it's executed, we're not sure if the backup has been succeeded and thus not sure whether the checkpoint is available now07:37
yinwei_computerhere, available means ready to be restored from07:37
yinwei_computerthat's why we need a task to synchronize each backup's status and thus update the whole checkpoint's status07:38
yuvalWhy isn't it the protection plugin job to poll the status and complete the task only when the actual protection is complete and available?07:38
yinwei_computeryou mean poll inside the task?07:39
yinwei_computerthis comes to the parallelism of the task flow07:39
yuvalI mean that we don't care whether the protection plugin is sync/async. We leave it to the protection plugin to decide07:39
yinwei_computeryes, that's why we have each protection plugin to implement get_status07:40
yuvalhow does it affect parallelism?07:40
yinwei_computersay we have 1 server attached with 1 volume07:42
yinwei_computerif we have volume to be polled as available and then to execute server protect, the latency is latency_volume_backup + latency_server_backup07:43
yinwei_computerin fact, the two could be triggered at the same time in protection07:43
yinwei_computerof course, if there's dependency, user could implement it in the sync mode07:43
yinwei_computerit depends on the implementation07:44
yuvalok, consider this:07:44
yinwei_computerif it's sync mode, the SyncCheckpointStatusTask would finish quickly with one poll round07:44
yuvalvm can perform its operation in parallel to the cinder protect. consider the VM protection plugin return a task called "VM finish protect". But also, creates another task, named "VM actual protect" and link it to the "VM finish protect". "VM actual protect" doesn't depend on cinder at all. "VM finish protect" depends on both "VM actual protect" and the cinder protect07:47
yuval         VM finish protect07:48
yuval       /                  \07:48
yuval  cinder protect        VM actual protect07:48
yuvaltherefore, when "VM finish protect" completes, we are certain that "cinder protect" and "VM actual protect" have finished (and may happen in parallel), but we don't need to check any status07:49
yuval"VM finish protect" can be a no-op, but that depends on the protection plugin07:50
yuvalwhat do you think?07:50
yinwei_computeryes, that's why we use heat in restore07:51
yinwei_computerwhere heat actually add a 'attachment' resource between server and volume07:51
yinwei_computerbut in protection, we think current resource graph is ok.07:52
yinwei_computeryou point will introduce complexity of server protection plugin, right?  it has to take care of parallelism, which task graph should does already07:52
yuvalthis can actually work for restore as well:07:53
yuval   VM main task (attach volume)07:53
yuval       /                \07:53
yuval cinder restore      VM restore07:54
yinwei_computerhmm, restore doesn't need do so07:54
yuvalbecause of heat?07:54
yinwei_computerheat will do this work07:54
yinwei_computeryes07:54
yuvalwell, VM main task can just return the heat resource, then07:54
yuvalwhy would that introduce a complexity for the server plugin?07:55
yinwei_computerin your case, protection plugin need check if its child task is sync/async.  If it's sync, it needs add another task in addition to its own protection task.  And, it needs organize the task flow by itself07:56
yinwei_computerthe generic task stack is organized according to resource graph, but for your case, the protection plugin should customize its own task stack: it should know only push attachement task into task stack and before this link the server task to attachement task.07:58
yuvalprotection plugin doesn't need to know if its child task is sync/async. protection plugin needs to return a task (which might have floating task which it depends upon), and that task will happen after all of its dependencies have been resolved07:58
chenying_ping gampel07:58
yinwei_computerthe generic task stack is organized according to resource graph, but for your case, the protection plugin should customize its own task stack: it should know only push attachement task into task stack and before this link the server task to attachement task.07:58
chenying_Hi yuval Is eran there?07:59
yuvalchenying_: he will be back in a few minutes07:59
yuvalchenying_: I'll tell him you looked for him08:00
chenying_Ok thanks.08:00
openstackgerritchenying proposed openstack/smaug: Add a field parameters to resource plans  https://review.openstack.org/30273008:01
yuvalyinwei_computer: the protection plugin will not customize the stack. It must return a task. When this task is complete, the resource protect/restore is complete. For parallelism, this task can depend on other custom task by the protection plugin.08:02
yinwei_computerhow can it retrieve the task it to depend on?08:02
yinwei_computerit has to operate task stack08:03
yuvalit can return a flow, instead of a single task08:04
yinwei_computerbtw, I don't prefer a synchronous task which sounds waste of thread resources.  Pls. check this patch:https://review.openstack.org/#/c/303355/108:04
yuvaland taskflow can handle when you add a flow to another flow08:04
yuvalCan you view Google docs?08:04
yinwei_computeractually, we'd better not have plugin to operate much on task flow08:04
yuvalhttps://docs.google.com/presentation/d/1vkgGdyYSrKTRbjUWWiz8IqDrd3sAJr25PRtXht93A8I08:04
yinwei_computersure, I know that task flow feature08:05
yinwei_computerbut flow doesn't solve your case08:06
yinwei_computerflow1: attachement->server; flow1->volume08:06
yinwei_computerit becomes serial now08:06
yuvalcan you please take a look at the google docs I sent?08:07
yinwei_computersorry, I couldn't get it from huawei network08:08
yinwei_computeryou could send text to my huawei mail box yinwei3@huawei.com08:08
yuvalnot even pdf?08:08
yinwei_computerThe website is denied.You have no privilege to access "Personal Network Storage" category.08:09
yinwei_computerpdf is ok08:09
yinwei_computerpdf as attachment in mail08:09
yuvalsent08:10
yinwei_computerbtw, what do you think about this bug?08:10
yinwei_computerhttps://bugs.launchpad.net/smaug/+bug/156679308:11
openstackLaunchpad bug 1566793 in Smaug "authenticate in protection service client" [Undecided,New]08:11
yuvalyep, we must handle this one08:12
yuvalregarding your example with flow1:08:13
yinwei_computeri saw your pdf,08:14
yinwei_computerI do appreciate to have attachment node between server and volume.  But I'm not sure whether we  need attachment every joinment08:14
yinwei_computerwhat does attachment between vm to image mean?08:15
yuvalit works the other way around. first [volume_flow: protect_volume] is created. then [server] flow is created, and linked with 1) volume_flow 2) actual server protect08:15
yuvalthe attachment node is an example. It can be "create HeadResource" for the record08:16
yinwei_computerI'm not sure if I get this08:16
yuvalthe most important thing is, that once the main task returns, the resource is fully backed up / restored08:16
yinwei_computerhmm08:17
yinwei_computermost openstack api is async08:17
yinwei_computerthe point here is sync mode is not efficient08:17
yinwei_computeryou have each thread polling and wait there08:17
yuvalI'm saying, a protection plugin should manage its sync/async nature08:17
yinwei_computersure08:17
yinwei_computerno problem08:17
yuvalit can be a greenthread08:18
yinwei_computerif it's sync, then get status always return static mode:available or error08:18
yuvalI understand08:18
yinwei_computerif it's async, then get status always poll all resources of this kind resource type08:18
yinwei_computergenerally, it's a trade off between sync/asyn and simplicity of protection plugin implementation08:20
yinwei_computerif sync is the benefit of your task flow organization, then protection plugin must know more about task flow context.08:20
yuvalnot sure I understand what it must know more about the taskflow context08:21
yinwei_computerAt the beginning of protection plugin design, we want to mask most of task flow details from protection plugin08:22
yinwei_computerif you check current cinder/nova/image protection plugin, they only implement functions08:22
yinwei_computerthey don't care dependency08:22
yinwei_computersince resource graph framework will take care of it08:22
yinwei_computerif we don't achieve consensus here, we could check it with gampel08:24
yinwei_computermy point here are: 1. it's not necessary to have protection plugin to implement a sync mode task; 2. we don't want to expose too much task flow details to protection plugin, that's why we optimized the base protection plugin and thus move task stack work to graph walker listener.08:25
yinwei_computerthe principle is to have the framework take more and have protection plugin to take less.08:26
yuvalI see. What I was worried about, is that we poll the statuses every time. Consider we have 1000 resources to protect - that's wasteful. By the way, not give each protection plugin a callback to call when it finished, and let it update the status in the SyncCheckpointTask ?08:28
yuval*why not give...\08:28
yinwei_computereach plugin will only have 1 thread to poll all on-going resources of its kind08:29
yuvalyes, but SyncCheckpointTask loops over all the status getters08:30
yinwei_computerand once the resource has reached static status:available, error, error deleting, their status will be persisted in bank08:30
yinwei_computerand for static status resources, SyncCheckpointTask won't poll in the next round08:31
yinwei_computerit's very simple memory optimization08:31
yuvalbrb08:33
openstackgerritsmile-luobin proposed openstack/smaug: Implement Cinder Restoration  https://review.openstack.org/30327708:39
openstackgerritchenying proposed openstack/smaug: Add checkpoint id to the response of the checkpoint POST action  https://review.openstack.org/30397108:53
openstackgerritxiangxinyong proposed openstack/smaug-dashboard: [Protection Providers] Submit index view function  https://review.openstack.org/30397308:56
openstackgerritxiangxinyong proposed openstack/smaug-dashboard: [Protection Providers] Submit index view function  https://review.openstack.org/30397309:04
yuvalyinwei_computer: hi09:16
openstackgerritwangliuan proposed openstack/smaug: Implement the  protection flow  https://review.openstack.org/28101109:18
openstackgerritxiangxinyong proposed openstack/smaug-dashboard: [Protection Providers] Submit detail view function  https://review.openstack.org/30398309:21
chenying_gampel:ping09:27
gampelI am in DF IRC weekly i will contact you latter09:32
chenying_OK Thanks.09:33
*** yinweiishere has joined #openstack-smaug09:37
openstackgerritchenying proposed openstack/smaug: Add checkpoint id to the response of the checkpoint POST action  https://review.openstack.org/30397109:55
openstackgerritwangliuan proposed openstack/smaug: Implement the workflow  https://review.openstack.org/28514911:13
xiangxinyongHi eran11:18
openstackgerritzengchen proposed openstack/smaug: Implement restart of Operation Engine service  https://review.openstack.org/30404811:21
xiangxinyonghttps://review.openstack.org/#/c/301543/11:27
openstackgerritzengchen proposed openstack/smaug: Implement restart of Operation Engine service  https://review.openstack.org/30404811:32
xiangxinyonghi saggi11:37
*** zhonghua-lee has quit IRC11:45
*** zhonghua-lee has joined #openstack-smaug11:46
chenzengping yinwei12:01
*** xiangxinyong has quit IRC12:04
*** xiangxinyong has joined #openstack-smaug12:06
saggixiangxinyong: hey12:30
saggixiangxinyong: How are you12:30
xiangxinyongfine.Thanks12:30
xiangxinyongsaggi:sorry to trouble you.12:30
saggixiangxinyong: np12:31
xiangxinyongI got a problem when i want to show a resouce tree in a plan.12:32
xiangxinyongcould you help me?12:32
xiangxinyonghttps://etherpad.openstack.org/p/tosaggi12:34
xiangxinyongplease take a look at the above link.12:34
* saggi is looking at the link12:35
xiangxinyongThis is the interface of showing a plan12:36
saggiok12:37
saggi....12:37
xiangxinyongi have no idea how to show a resource tree in the plan detail page.12:37
saggioh12:38
xiangxinyongbecause i could not find the dependency relateion in this json12:38
xiangxinyongbecause i could not find the dependency relation in this json12:38
xiangxinyongsaggi: What do you think about that?12:39
saggixiangxinyong: I understand the problem.12:39
saggiIf you just want to show them in relation to each other it's simple.12:40
saggiThe problem is parent relationships.12:40
xiangxinyongyeah12:40
* saggi is thinking about it12:40
*** xiangxinyong456 has joined #openstack-smaug12:41
saggiYou can start looking for them from the roots. But that's an expensive action.12:42
xiangxinyong456yeah, it seems like it is.the only way12:43
* saggi is thinking some more...12:43
xiangxinyong456firstly, i will list the all resources12:43
xiangxinyong456and then it will choose the resources in the plan12:44
saggiTechnically the only issue is with adding new resources. Any other flow doesn't really care about parent resoruces.12:44
saggixiangxinyong456: Listing all the resources is expensive.12:44
xiangxinyong456yeah, it is more expensive12:45
saggiWhat I would suggest is displaying them all the resources as a list in the same level all as root entities.12:47
saggiIf the user wants to modify or deselect they can.12:47
saggiThe can also expand those to check for auto-detected dependencies.12:47
saggiFor adding a new resource, you could open a new dialog with the root and let the user look for the resource they want to add, if it an entity that is already selected is returned it is marked as checked.12:47
saggixiangxinyong456: What do you think?12:48
*** xiangxi38 has joined #openstack-smaug12:49
saggiping yinwei_computer yinweiishere12:51
*** xiangxinyong456 has quit IRC12:51
xiangxi38saggi, you means we can list all the resources in the plan in the same level12:52
xiangxi38?12:52
saggiYes12:52
saggiBut for adding new resources we open a dialog with just the project.12:52
saggiSimilar to how you have a zip file and you add a new file to it12:53
saggixiangxi38:  how is that idea?12:53
saggiAnd why are you changing nicknames every minute :)12:53
xiangxi38sorry, i am using mobile12:54
xiangxi38saggi:  so we don't need a resource tree when we show a plan detail info12:56
saggiParitally, we still want to allow the user to expand the flat list to see what is autodetected.12:56
xiangxi38you means the users could expand the resource12:59
xiangxi38only dependent resources ?12:59
xiangxi38or users could expand the dependent resources?13:00
saggiIt could keep expending to investigate13:02
saggiBut it should't be able to add items this way13:03
saggiOnly through the dialog. So we always use the canonical view.13:03
xiangxi38saggi: i think we should show the whole resource tree when the users update the plan.13:05
xiangxi38and let the resources to select and disselect13:05
saggiBut it's an expensive action to look for a resource. And it might even have been deleted.13:06
xiangxi38yeah. understood13:06
saggiAlthough most of the time it will just be under project.13:06
saggiBut if it's not looking for it would be very hard.13:07
xiangxi38i means when we show the plan detail, we only show the users choosed resources13:07
saggiYes, and when adding new resources we show the whole tree13:08
xiangxi38and when we show the update plan dialog,13:08
xiangxi38we could show the whole tree13:09
saggiYes, but unexpended.13:09
saggiBut if we stumble to a selected resource we show it's selected.13:09
xiangxi38saggi: i think the users have 3 different operations.13:12
xiangxi38create a plan13:12
xiangxi38update a plan13:12
xiangxi38show a plan detail13:12
xiangxi38we may need 3 pages to show them.13:13
xiangxi38what do you think about it?13:14
saggiCreate and update are similar IMHO13:14
saggiYou create an empty plan13:14
saggithen start updating it13:15
xiangxi38yeah, do you think we need a edit button on every plan?13:16
xiangxi38in our fisrt version?13:18
xiangxi38or we only need the create plan and plan detail13:19
xiangxi38i think the users should not modify the plan in the plan detail page.13:20
saggiAfter you create it will just automatically put you and edit mode13:20
saggiBut it will be edit13:20
saggiThat way you reuse it13:20
saggiSo you create and decide the pland name and bank and the rest of the globals13:21
saggithen they ui will create and empty plan13:21
saggithen it will move to edit mode automatically13:21
saggiSo the user feels like it's a smooth experience13:21
xiangxi38saggi: understood.13:24
xiangxi38saggi: but specify the plan name, providers and resources.,they are together13:25
saggiYou create a plan with an empty resource list13:25
saggiand then edit. So after the user edits it will update the resource list13:26
saggitesting brown files13:27
xiangxi38saggi: oh, so when the plan created, where could the plan be edited13:31
saggiThe edit page :)13:31
saggiWhich you will automatically be redirected after creation13:32
xiangxi38oh.13:34
xiangxi38so we don't need to the resource tree in the plan detail13:34
xiangxi38do you agree?13:35
saggiyes, we don't need the full tree. Just to be able check autodetection.\13:37
xiangxi38ok13:38
xiangxi38and when we resit13:40
xiangxi38and when we restore from checkpoint,13:41
xiangxi38do you think we still need to show the resources and autodetect?13:42
saggiNo, for restore there is no autodetection13:44
saggifor restore you have all the information.13:44
saggiThe checkpoints contains the full tree13:45
xiangxi38I understand the checkpoint only contains the selected resources13:46
xiangxi38am i wrong?13:47
saggiNo the checkpoint contains all the resources that were backed up13:48
saggiThere is no autodiscovery for restore. There is nothing to discover.13:48
xiangxi38so when we restore from checkpoint,13:52
xiangxi38we only just show a resource table, and let the users to edit the resource parameters13:53
xiangxi38is it right?13:54
saggiYou have full tree in the checkpoint13:55
xiangxi38ok, the checkpoint has the all backup resources, and the checkpoint will record these backup resources's relationship?13:57
saggiyes14:04
saggixiangxi38: Do you have any more issues?14:06
xiangxi38no.14:06
saggigreat :)14:06
xiangxi38thanks saggi14:06
sagginp14:06
xiangxi38your answer is really help me a lot14:07
openstackgerritYuval Brik proposed openstack/smaug: Implement Graph pack and unpack  https://review.openstack.org/30416014:27
*** xiangxi38 has quit IRC14:33
*** xiangxinyong456 has joined #openstack-smaug14:34
*** xiangxi15 has joined #openstack-smaug14:42
*** xiangxinyong456 has quit IRC14:42
*** xiangxi15 has quit IRC14:53
*** xiangxinyong456 has joined #openstack-smaug14:54
chenying_Hi saggi14:57
chenying_saggi: ping14:57
saggichenying hi15:05
saggihow are you?15:05
*** yuval has quit IRC15:05
chenying_Hi saggi  In previous irc meeting we talked about CI, automatic tests on gerrit. It should be added to smaug.15:07
chenying_Do you have any reference implementation or links about the CI?15:08
*** xiangxinyong456 has quit IRC15:16
*** openstackgerrit has quit IRC15:18
*** openstackgerrit has joined #openstack-smaug15:18
*** xiangxinyong456 has joined #openstack-smaug15:19
*** xiangxinyong456 has quit IRC15:20
saggiNot really, gampel should have some15:21
chenying_Could you ask gampel to give some reference about the CI?15:36
openstackgerritMerged openstack/smaug: Fix Provider RPC client and server do not match  https://review.openstack.org/30267915:45
openstackgerritMerged openstack/smaug-dashboard: Init api tests for smaug-dashboard  https://review.openstack.org/30258715:51
openstackgerritchenying proposed openstack/smaug: The versioned object and db operation of operation_log  https://review.openstack.org/30421216:21
*** xiangxinyong has quit IRC19:25
*** x00350071 has joined #openstack-smaug19:26
*** openstackgerrit has quit IRC20:05
*** openstackgerrit has joined #openstack-smaug20:27
*** openstackgerrit has quit IRC20:41
*** openstackgerrit has joined #openstack-smaug20:55
*** saggi has quit IRC23:26
*** gampel has quit IRC23:26
*** saggi has joined #openstack-smaug23:40
*** gampel has joined #openstack-smaug23:41

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!