Wednesday, 2016-01-27

*** daemontool_ has quit IRC00:04
*** daemontool_ has joined #openstack-freezer00:06
*** daemontool_ has quit IRC00:13
*** dschroeder has quit IRC00:38
*** EinstCrazy has quit IRC01:01
*** c00281451 has quit IRC01:31
*** EinstCrazy has joined #openstack-freezer01:43
*** daemontool has joined #openstack-freezer02:31
*** SamYaple has quit IRC04:11
*** SamYaple has joined #openstack-freezer04:18
*** EinstCrazy has quit IRC04:25
*** EinstCrazy has joined #openstack-freezer05:21
*** EinstCrazy has quit IRC05:47
*** EinstCrazy has joined #openstack-freezer05:48
*** EinstCrazy has quit IRC07:12
*** EinstCrazy has joined #openstack-freezer07:22
*** EinstCrazy has quit IRC08:05
*** EinstCrazy has joined #openstack-freezer08:09
*** EinstCrazy has quit IRC09:55
*** zhangjn has joined #openstack-freezer10:37
*** emildi has joined #openstack-freezer11:01
SlashmeSamYaple: Hi, nice to see that we are not the only ones carring about backups and things done well :)11:02
SlashmeHere are my two cents 'cause I'm known to always give my opinion.11:02
SlashmeJust to start, a quick refresh on how the freezer infra works.11:02
SlashmeThe freezer-agent and freezer-scheduler are installed where you want to backup data. That can be inside a VM if you want to backup OpenStack workloads, on controllers to backup OpenStack infra, or even on your laptop to replace your dropbox if you like.11:02
SlashmeThe freezer-scheduler polls the freezer-api to get the list of backup jobs it is supposed to execute and their scheduling informations. The freezer-scheduler then fires the freezer-agent that executes the required backup. We have different types of backup (the argument is --mode), file is the default, but we also support mongo, mysql, sqlserver, nova trought nova api (not ideal) and cinder thought cinder api (idem). The backup is then compressed11:02
Slashme(your choice of compression algo), encrypted if necessary, and then sent to a storage media (we support Swift, Ssh ans local).11:02
SlashmeYou create backup jobs through the Horizon webUI or the python-freezerclient.11:02
SlashmeHere is for Freezer.11:02
SlashmeWe have the short-term goal to abstract the --mode with a plugin layer (ping szaher reldan ). Meaning the freezer-agent could do any type of backup trough loading plugins from a plugin folder. From what I understand, Ekko could re-use all the freezer infra, having freezer-agent installed on the compute node and using a specific Ekko mode.11:02
SlashmeAnyway and regardless of the kind of convergence Freezer and Ekko achieve, thanks for your openness and interest. It is nice to see people that are interested in the community side of OpenStack.11:03
*** pbourke has joined #openstack-freezer11:04
*** zhangjn has quit IRC11:06
*** zhangjn has joined #openstack-freezer11:09
zhangjnwhat's the choice Freezer or cEkko?11:21
*** zhangjn has quit IRC11:58
*** pbourke has quit IRC12:02
*** pbourke has joined #openstack-freezer12:03
*** p3r0t has joined #openstack-freezer12:13
p3r0thi folks12:13
p3r0tI have a question about storage where freezer is going to store the backups. Is it possible to use ceph instead of swift ?12:14
*** daemontool has quit IRC12:19
SlashmeIf you use the radosgw, then yes.12:26
SlashmeWhen you use the radosgw, ceph replaces swift and provide the swift api12:26
SlashmeIf you mean using the librados and pushing things directly in ceph then no. Or at least, not yet, it would be possible to implement I guess12:27
*** zhangjn has joined #openstack-freezer12:33
*** zhangjn has quit IRC12:34
*** zhangjn has joined #openstack-freezer12:35
*** daemontool has joined #openstack-freezer12:40
*** zhangjn has quit IRC12:44
daemontoolMorning12:46
Slashmemorning daemontool12:50
daemontoolhow is it going12:55
daemontool:)12:55
daemontoolI'm releasing freezer 1.2.1 to pypi12:56
daemontoolafter the bug merge https://review.openstack.org/#/c/271702/212:56
p3r0tSlashme, thanks13:53
daemontoolbe back in 30 min13:56
*** daemontool has quit IRC14:00
*** emildi has quit IRC14:08
*** EinstCrazy has joined #openstack-freezer14:16
*** daemontool has joined #openstack-freezer14:31
*** EmilDi has joined #openstack-freezer14:36
*** dims has joined #openstack-freezer14:44
*** samuelBartel has joined #openstack-freezer14:46
dimsdaemontool: checked with rpodolyaka over on Nova about the VM snapshot capabilities. his answer was " libvirt/xenapi/hyperv/vmware should all support live snapshoting" there's this note about some restrictions - https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L149914:49
*** EinstCrazy has quit IRC15:13
daemontooldims, thanks a lot appreciate your help :)15:42
*** dschroeder has joined #openstack-freezer15:56
*** samuelBartel has quit IRC17:02
*** dims has quit IRC17:58
*** EmilDi has quit IRC18:13
*** EinstCrazy has joined #openstack-freezer18:14
SamYapleSlashme: thanks for the run down18:14
SlashmeSamYaple: Np18:14
SamYapleSlashme: how is retention handled with freezer?18:15
*** EinstCrazy has quit IRC18:18
SamYapleOr anyone on the freezer team, how are you handling retention?18:20
SamYapleSearching through the freezer repos I find no mention of retention of any kind. Does this mean you must keep around all incrementals for the life of the backup?18:32
*** daemontool_ has joined #openstack-freezer18:45
*** daemontool has quit IRC18:47
*** dims has joined #openstack-freezer19:54
*** daemontool has joined #openstack-freezer20:06
*** daemontool_ has quit IRC20:09
daemontoolSamYaple,  did anyone provided you info regarding the retention?21:03
SamYapledaemontool: not yet21:05
daemontoolSamYaple,  you have two options you can use21:05
daemontoolremove_from_date and remove_older_than21:05
daemontoolif you do not use that, the backup will be kept for as long as the user wants21:05
SamYaplewhat about middle-of-chain removal?21:05
daemontooli.e. remove older than 30 days but keep 1 backup for 1 year?21:06
daemontoolthere's a bp for that but is not implemented21:06
daemontoolalso we never encountered an operational case from users provided feedback for that21:07
daemontoolbut if needed it can be done21:07
daemontoolprobably we'd need to add a time range21:07
SamYapleyes, but more like "take backup every hour but only keep a daily backup longterm"21:07
daemontoolfor deleteion21:07
daemontoolwe do not have that currently21:08
SamYaplehow is retention performed? In the case of object storage does this mean all the data is download, merge, and reuploaded?21:08
daemontoolnope21:08
daemontoolwe record timestamp in the object metadata on swift21:08
daemontoolso we only download the metadata21:08
daemontoolnot the data21:08
daemontoolaccording the metadata the object will be removed21:09
daemontoolat the next one I'll send you a bill =P21:09
daemontoolkidding...21:09
daemontoolswift manifest is probably what you need to look at21:09
SamYaplebut the actual retention act, (removing prior to certain date) how are you doing that?21:09
daemontoolwe have two way of checking the data21:10
daemontoolone is the timestamp21:10
daemontoolin the metadata of the object21:10
daemontoolthe other is the timestamp stored in the object name21:10
daemontoolso when you get the list of all the object from swift21:10
daemontoolyou know if the timestamp of the object21:11
daemontoolis whitin or out the timeframe21:11
daemontoolyou need21:11
daemontoolif is out then you delete the object to swift21:11
daemontooland the same happen21:11
daemontoolto the other21:11
daemontoolstorage media21:11
SamYaplethats not quite what im asking. If you take a backup day 1, 2, 3, 4.. and you delete day 1 and 2, are you merging those into day 3?21:11
daemontoollike ssh and local fs (i.e. mounted nfs(21:12
daemontoolnope21:12
daemontoolwhy you want to do that?21:12
daemontoolthe backup are compressed and encrypted21:12
SamYaplebecause if you delete day 1 and 2 of an incremental backup you can't access day 321:12
daemontoolah you mean21:12
daemontoolby 1 2 3 4 5 the incremental backup level?21:12
daemontoolyou always need level 021:13
SamYapleyes. have you not been talking about incremental backups?21:13
daemontoolyou need level 0 for inremental backups21:13
SamYapleso you do diffs off of level 0?21:13
daemontoolso I'm not sure if your example apply21:13
daemontoolor there's something I'm missing21:13
daemontoolyes21:13
daemontoolthere are 2 options21:13
daemontoolmax_level21:13
daemontooluse incremental21:13
daemontoolalways_level is more like a differential21:14
daemontoolso the new level is always the difference between level 0 and your currenet level21:14
daemontoolwhen restoring21:14
daemontoolonly 2 level needs to be restored21:14
SamYapleso you have level 0, 1, 2, 3 and you want to remove 1 and 2, do you download 1 2 and 3 and merge them?21:14
daemontoolso it is faster21:14
daemontoolnope21:14
daemontoolwhy you want to do that?21:14
daemontoolwhy you want to touch encrypted compressed backup stored in a media storage?21:14
SamYapleso they are all just diffs off of level 0?21:15
daemontoolyes21:15
SamYapleesh21:15
SamYaplethat won't work long term for block backups at all21:15
daemontooland why is that?21:15
daemontoolit works for rsync block based backups21:15
SamYapleif level 1 has 1gb of change, then level 2 will have 1gb + whatever else is changed21:15
SamYaplethats duplicating what you are backing up21:15
SamYaplenot to mention requiring level 0 forever21:16
daemontoolwhy level 2 has 1 gb + the incremental?21:16
daemontoollevel 2 have only the incremental21:16
daemontoollevel 0 you need it for restore21:16
daemontoolotherwise how will you restore the data if you need it?21:16
SamYaplelevel 2 has the same information you backed up in level 1, plus whatever has changed for the level 2 backup21:17
SamYaplebecause its a diff on level 021:17
daemontoolah you mean for the always_level option21:17
daemontoolit's only a level 121:17
daemontoolyou always have 2 levels21:18
daemontool0 and 1 if you set that21:18
daemontoolwhile with max_level21:18
daemontoolyou have 0 1 2 3 4 521:18
SamYapleso for incrementals, not diffs, how do you handle removing 1 2 3 out of 0 1 2 3 4 5?21:18
daemontoolif you that you can restore only 021:18
SamYapleso you _cant_ is what you are say?21:19
daemontoolbecause the differential data between 0 - 1 2 3 an 421:19
daemontoolis gone21:19
daemontoolso 4 is not usable21:19
SamYapleyea ok thats what i thought was happening21:19
SamYaplethats the issue21:19
daemontoolnot sure what's the issue there21:19
daemontoolexplain it please21:19
SamYapleso you have to keep all incrementals around for the backup around forever21:20
daemontoolnope21:20
SamYapleyou just said it though21:20
daemontoolin some point you want to restart level 021:20
daemontoollet's say you do21:20
daemontooldaily backup21:20
daemontooland every week you do a full backup21:20
SamYaplerestarting a level 0 is the problem21:20
daemontoolso you have 6 levels21:20
daemontool0..621:20
SamYaplethis is not something that is acceptable to the backup world21:20
daemontoolit's apoint in time state of your sistem21:21
daemontoolthat's how rsync works21:21
SamYaplei understand what you are saying, its just not a good way to do backups. it doesn't work for blockbased at all21:21
daemontooltar works21:21
daemontooletc etc21:21
daemontoolso21:21
daemontoolfor blockbased21:21
SamYapleyes. and it doesnt work for the scale of block-based backups21:21
daemontoolyou can use rolling block based21:21
daemontoolwhich is exactly how21:22
daemontoolrsync work21:22
daemontoolso now, why the rsync algorithm doesn't work for you?21:22
SamYapleyou can't force a new "level 0" backup because of the size of the backup. its potentially 100's of GBs21:22
SamYapledoing diffs means the size of the backup grows forever (until you do a new full)21:23
daemontoolit's the point in time of your system21:23
daemontoolof your data21:23
daemontoolif you do not want to keep them21:23
daemontoolyou can delete them21:23
SamYapleno without losing your latest backup21:23
SamYaplerequiring a new full after the first one should never happen21:23
SamYaplewith ekko we can have 0 1 2 3 4 5 6 7 8 9 and remove 2 3 6 7 without losing 0 1 4 5 8 921:24
daemontoolok21:24
daemontoolthat's libvirt that does that21:24
daemontoolnot ekko21:24
daemontoolbut you still need level 021:24
SamYaplethats not true at all21:25
SamYapleand we do not21:25
SamYaplewe can drop 0 1 2 3 4 5 6 7 8 and still ahve 921:25
daemontoolok21:25
daemontoolso you have all the data on 9?21:25
SamYaplelibvirt has nothing to do with that part21:25
SamYapleyup21:25
daemontoolok21:25
daemontoolyes qemu sorry21:25
daemontoolnot libvirt21:25
SamYapleno qemu has nothing to do with that either21:25
SamYaplethats what Ekko can do21:26
daemontoolok21:26
daemontoolso what's your point21:26
daemontool?21:26
daemontoolso ?21:26
daemontoolhow that21:26
daemontoolever has to do with the point I asked you to clarify21:26
daemontoolon the ml?21:26
daemontoolis that your point?21:26
SamYaplei was trying to figure out if freezr had retention for incremental backups and how it works21:26
daemontoolok21:27
SamYaplebut from what you are telling me you dont do retention which answers that question21:27
daemontoolwe do retention in the way I described21:27
SamYaplebut thats not actually retention21:27
daemontoolok21:27
SamYapleyou arent retaining anything21:27
daemontoolI see your point21:27
daemontoolI'm retaining the data21:28
daemontoolfor the time the user wants21:28
daemontooland remove the data the user do not want21:28
daemontoolin a way that if the user wants to restore the data in the point in time he wants21:28
daemontoolhe can21:28
daemontoolthat is our use case21:28
daemontoolI got you have a different idea21:29
daemontoolnot trying to convince you21:29
daemontooljust answering21:29
SamYapleI understand. What you are describing just doesn't work for block-based backups due to the size of them21:30
daemontoolI don't see that way21:30
daemontoolthe think I'm telling21:30
daemontoolhas been in production21:31
daemontoolfor time21:31
daemontoolat reasonable scale21:31
daemontoolnow21:31
daemontoolif you tell me21:31
daemontoolthere's a better way21:31
SamYapleit does not work at a scale. especially when you start talking about TB of data21:31
daemontoolthat's ok21:31
SamYaplelook at the problems the, arguably, two largest backup companies have with it (storagecraft and veeam)21:32
SamYaplethe way they have gotten around this is by merging incrementals21:32
SamYaplebut that doesn't work for object-storage and thats another limitation to scale21:32
daemontoolif the data are encrypted and compressed21:33
daemontoolthat is not efficient with any storage media21:33
daemontoolnot only with object storage21:33
SamYapleWell thats my point, ive solved that with Ekko21:33
SamYaplebut i was mainly curious to see if you already had a way t odo retention21:33
daemontoollet's talk about this more21:34
daemontoolI have a meeting now21:34
daemontoolsorry21:34
SamYapleok21:34
daemontoolthank you21:34
daemontoolthis was interesting :)21:34
SamYapleill bet i can bring what i use for ekko to freezer21:34
SamYapleeven for file level21:34
*** dims has quit IRC21:46
daemontoolSamYaple, that is fantastic22:06
daemontooland that open approach is brilliant22:06
daemontool++22:06
SamYaplethe only concern is I rely on changed block tracking on the hypervisor to tell me what to backup (rather than rsyncs algorythm) but that could be solved it other ways22:07
*** daemontool has quit IRC22:19
*** daemontool has joined #openstack-freezer22:30
daemontoolSamYaple, I'm here22:31
daemontoolyes I agree22:31
daemontoolI think we have to provide flexibility22:31
daemontoolif that's the best solution22:31
daemontoolto solve the vm backup in terms of efficiency22:31
daemontoolit make sense to provide that22:31
daemontoolwhat can be an alternative better approach for volumes or files22:32
daemontoolthat will be proposed22:32
daemontoolbut there are cases where for companies22:32
daemontoolstorage efficiency is not important22:32
daemontooltime efficiency is22:32
daemontoolfor others not22:32
SamYaple"storage efficiency is not important". I work at a backup company, storage efficiency is one of the biggest concerns22:33
daemontoolspace efficiency22:33
SamYaplebut luckily what i purpose is more effiecent space and time-wise22:33
daemontoolbetter then22:33
daemontoolin financial industry22:34
SamYapleso as Slashme suggested, I can probably reuse the freezer agent/scheduler if there is a better plugin functionality for it22:34
daemontooltime is22:34
daemontoolI didn't see that conversation22:34
daemontoolbut Slashme  has a excellent skills and knowledge22:34
daemontoolso most probably make sense his advise22:34
SamYapledaemontool: in the case of ekko, the 'downtime' of an instance is very, very little, far less than the time it takes to do the backup22:34
SamYaplea few seconds22:34
SamYaplethe backup happens behind the scene without affecting the instance22:35
SamYapleand its not truly downtime either, not like a snapshot which pauses the instance22:35
SamYaplemore like read-only rather than hard-down22:35
*** daemontool_ has joined #openstack-freezer22:37
daemontool_SamYaple, we need to write a bp22:37
daemontool_and get feedback22:37
daemontool_also from Nova22:38
daemontool_it's important otherwise we are going to have always issues in prod environments to place the component we want on the compute nodes22:38
daemontool_have an open discussion on that22:38
daemontool_do you agree/22:38
daemontool_?22:38
*** daemontool has quit IRC22:39
SamYapleI don't know what would be needed from nova22:39
SamYapleI don't actually speak to nova at all currently22:39
daemontool_we touch the hypervisor22:39
daemontool_without using the nova api22:39
SamYapleright22:39
daemontool_I saw that many times in the past and always was an issue22:39
SamYapleCan you give an example?22:39
daemontool_for what?22:40
SamYaple22:40:11 < daemontool_> I saw that many times in the past and always was an issue22:40
daemontool_like collectors of hypervisors metrics22:40
daemontool_for cloud big data analysis22:40
daemontool_rather than using the api22:40
daemontool_same apply for cinder22:41
SamYapleBut specifically what issues are you talking about?22:41
daemontool_the issue is that who manage the compute notes in production environment22:41
daemontool_does not want anything touching directly the hypervisor22:42
daemontool_this is the issue22:42
SamYapleNova should not be in charge of speaking to the hypervisor for these particular actions in my opinion22:42
daemontool_yes but we need to have their feedback22:42
daemontool_explain things22:42
SamYapleOh sure I agree22:42
daemontool_why we do that22:42
daemontool_why we take that decision22:42
daemontool_and so on22:42
SamYaplebut hinging this project on nova is a good way to get it killed ;)22:43
daemontool_listen their risks and concerns22:43
daemontool_address them and move forward22:43
daemontool_I don't think so22:43
daemontool_If our approach22:43
daemontool_is good for users22:43
daemontool_and we explain it well22:43
daemontool_clearly22:43
daemontool_openly22:43
daemontool_and we address the concerns22:44
daemontool_it will be accepted22:44
daemontool_if a solution is better22:44
SamYapleUnfortunately that has never been the case with nova. but nevertheless that is the plan of action yes22:44
daemontool_there's no need to hide (not saying you are saying that)22:44
daemontool_SamYaple, what I'm saying is let's think very well about it22:45
daemontool_the approach22:45
daemontool_and so on22:45
daemontool_let's do the plan together22:45
daemontool_deep thinking on it22:45
daemontool_provide a bp/spec22:45
daemontool_and approach it seriously to os community, TC and so on22:46
daemontool_it will be positive22:46
daemontool_and22:47
daemontool_probably I'd approach this22:47
daemontool_by proposing a solution for a specific problem22:47
daemontool_narrowing the scope22:47
SamYapleThere will definetely be a bp/spec with nova, but ekko won't be blocked by nova has been my point22:47
daemontool_how does it sounds?22:48
SamYaplewhat you are purposing is offloading alot of work into nova, and I don't see that being useful22:48
daemontool_SamYaple,  don't worry about the blocking things22:48
daemontool_ok22:48
SamYaplenova is already large and bloated22:48
daemontool_let's discuss about it22:48
daemontool_ok22:48
daemontool_I have to do a meeting22:48
daemontool_be right back soon22:48
SamYapleok22:50
*** dims has joined #openstack-freezer22:51
*** dims has quit IRC22:54
*** daemontool has joined #openstack-freezer22:58
*** daemontool_ has quit IRC23:01
*** dims_ has joined #openstack-freezer23:30
daemontoolSamYaple, I'm not proposing to offload to noav23:35
daemontoolnova23:35
SamYapledaemontool: i think we are actually on the same page about all of this, im just focusing on a alpha release23:36
daemontoollet's discuss this Friday23:36
daemontooltomorrow I can't23:36
daemontoolSamYaple,  ok great23:36

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!