| *** zhonghua is now known as zhonghua-lee | 06:26 | |
| *** openstackgerrit_ has joined #openstack-freezer | 06:51 | |
| openstackgerrit | Kenji Yasui proposed openstack/freezer: Delete python bytecode before every test run https://review.openstack.org/254050 | 07:44 |
|---|---|---|
| *** reldan has joined #openstack-freezer | 10:09 | |
| *** openstackgerrit_ has quit IRC | 10:23 | |
| *** daemontool has joined #openstack-freezer | 10:43 | |
| daemontool | Morning | 10:43 |
| daemontool | all: we should start planning a tenant based backup and restore mechanism | 10:43 |
| daemontool | helpful and also a hot topic that everybody needs for the BaaS | 10:44 |
| m3m0 | we've been thinking about that daemontool | 10:48 |
| m3m0 | but so far we only backup files, we need to create a new mode that interacts more with mysql | 10:49 |
| daemontool | why | 10:50 |
| daemontool | why? | 10:50 |
| daemontool | ops twive sorry | 10:50 |
| daemontool | s/twive/twice/ | 10:50 |
| m3m0 | in mysql we backup the whole directory | 10:50 |
| daemontool | we can execute the tenant based backup using the services api | 10:50 |
| daemontool | and use the same api for restore | 10:50 |
| m3m0 | then if that's the case let's do it with the api :) | 10:51 |
| daemontool | then the service in question will write correctly the data to the db | 10:51 |
| daemontool | I'm just thinking loud | 10:51 |
| daemontool | but I think our life ie way easier | 10:51 |
| m3m0 | can we restore in another instance? | 10:51 |
| daemontool | if we do through the services API | 10:51 |
| daemontool | something like http://paste.openstack.org/show/avJHQbg5ty92baqnknfH/ | 10:52 |
| daemontool | that steps | 10:52 |
| daemontool | I think that is critical for Back up as a service | 10:52 |
| daemontool | question is | 10:53 |
| daemontool | where they'd be executed | 10:53 |
| daemontool | form where | 10:53 |
| m3m0 | we have the same question for cinder and nova backups made from the ui | 10:54 |
| daemontool | frescof: reldan Slashme ^^ | 10:54 |
| m3m0 | who will execute those | 10:54 |
| daemontool | our approach is distributes | 10:54 |
| daemontool | distributed | 10:54 |
| daemontool | to scale | 10:54 |
| daemontool | so always VMs should do that | 10:54 |
| daemontool | now VMs can be dedicated or not, that's user choise | 10:54 |
| m3m0 | that's the approach we follow right now | 10:55 |
| m3m0 | ok ok | 10:55 |
| *** szaher_ has joined #openstack-freezer | 10:55 | |
| daemontool | but I think the freezer-scheduler and agent should do that | 10:55 |
| *** szaher has joined #openstack-freezer | 10:55 | |
| daemontool | after all is a sequence of orchestrated jobs | 10:55 |
| daemontool | i.e. keystone user data export, cinder volumes snapshots, vms snapshots, upload to the storage media, etc | 10:56 |
| daemontool | each step can be a job | 10:56 |
| daemontool | that is managed by the scheduler | 10:56 |
| m3m0 | then following the same logic an agent should do the tenant backup as well | 10:56 |
| daemontool | yes | 10:56 |
| daemontool | I think so | 10:56 |
| daemontool | after all the credentials and the tenant information are located in the node where the agent is running | 10:56 |
| daemontool | szaher: ^^ | 10:57 |
| daemontool | where's Vanni? | 10:57 |
| m3m0 | everywhere | 10:57 |
| daemontool | everywhere but here :) | 10:57 |
| m3m0 | kidding, is here :) | 10:57 |
| daemontool | ok if we can join irc | 10:57 |
| daemontool | we can have this discussion even now | 10:57 |
| daemontool | frescof: is there around? | 10:58 |
| m3m0 | in fact the tenant backup was proposed long time ago :) | 10:58 |
| daemontool | yep | 10:58 |
| m3m0 | before it was cool :) | 10:58 |
| daemontool | well, we couldn't make it, anyway | 10:58 |
| daemontool | before :) | 10:59 |
| daemontool | now many things are OK | 10:59 |
| daemontool | it would be good even doing something backup all_tenants | 10:59 |
| daemontool | in once | 10:59 |
| *** emildi has quit IRC | 11:00 | |
| m3m0 | and a single tenant as well | 11:00 |
| daemontool | reldan: what do you think? ^^ | 11:00 |
| daemontool | yes | 11:00 |
| daemontool | admin user in keystone can do all_tenants | 11:01 |
| szaher | yes daemontool :) Vanni is here around :) | 11:01 |
| reldan | daemontool: about backing up all tenant like you describe here http://paste.openstack.org/show/avJHQbg5ty92baqnknfH/? | 11:01 |
| daemontool | somethign liek that | 11:01 |
| daemontool | is just a first thought | 11:01 |
| daemontool | just to have something to discuss on | 11:01 |
| daemontool | we can use a different approach | 11:01 |
| daemontool | as usual | 11:01 |
| *** vannif has joined #openstack-freezer | 11:02 | |
| reldan | How about cindernative? | 11:02 |
| reldan | Because it is actually the only way of doing incremental cinder backups? | 11:03 |
| daemontool | reldan: that can be an option | 11:03 |
| daemontool | I'd say, let's first identify | 11:04 |
| daemontool | what to backup | 11:04 |
| daemontool | and where to get the data | 11:04 |
| daemontool | then we can decide how and where to store | 11:04 |
| daemontool | I think we should download the data and upload them to the storage media we want | 11:04 |
| daemontool | so we can fully support disaster recovery | 11:04 |
| daemontool | but that's just my opinion | 11:04 |
| reldan | Yes, but if you have a very big volume | 11:05 |
| reldan | like 100GB | 11:05 |
| daemontool | or 1T | 11:05 |
| reldan | to do a backup everyday and reupload an image | 11:05 |
| daemontool | it's huge | 11:05 |
| daemontool | yes | 11:05 |
| reldan | even if you have no changes at all | 11:05 |
| daemontool | ok let's think about it | 11:07 |
| daemontool | let's use all the native backup services api then | 11:07 |
| daemontool | at least for the beginning | 11:07 |
| daemontool | then we can improve | 11:07 |
| daemontool | but let's think it throughtly | 11:08 |
| daemontool | before | 11:08 |
| daemontool | like couple of days | 11:08 |
| daemontool | we need to have a hangout meeting I think | 11:08 |
| *** emildi has joined #openstack-freezer | 11:08 | |
| reldan | Okey. I feel that we can have application consistent backups and volume/directory backup | 11:08 |
| daemontool | also we need to start talking with Nova and Cinder to converge the backup efforts | 11:08 |
| daemontool | reldan: ++ | 11:09 |
| reldan | We should have some entities, I feel that one entity is application (MySql, MongoDB … ), second is snapshot mechanism (cinder snapshot, lvm, shadow, …) third is backup mechanism (tar, rsync, …) | 11:10 |
| reldan | And there are different combinations of any three of that. | 11:11 |
| reldan | Guys, please review: https://review.openstack.org/#/c/253117/ https://review.openstack.org/#/c/253106/ https://review.openstack.org/#/c/247840/ | 11:11 |
| daemontool | reldan: yes | 11:12 |
| reldan | So if you ask to make a full backup of tenant you should have some document with description - how do you want to backup all your resources | 11:12 |
| reldan | Do you need an application level and what do you prefer tar or rsync, what is downtime should be ... | 11:13 |
| daemontool | I think the tenant based backups are done using crash consistent approach | 11:15 |
| daemontool | at least at the beginning | 11:15 |
| daemontool | I have to go now | 11:15 |
| daemontool | I'll be back in 1 hour | 11:15 |
| daemontool | let's talk about it | 11:16 |
| daemontool | re: https://review.openstack.org/#/c/253106/ | 11:16 |
| reldan | Sure! | 11:16 |
| daemontool | so now that options does not have any effect at all? | 11:16 |
| daemontool | cause that was the only way to list the backups from swift | 11:16 |
| daemontool | now we have no way whatsoever to list backups objects | 11:16 |
| daemontool | unless we use the swift client | 11:16 |
| daemontool | and the approach is different for ssh and local media | 11:17 |
| daemontool | we need also to solve that asap | 11:17 |
| daemontool | I'll be back in 1 hour laterz | 11:17 |
| reldan | daemontool: I believe it shouldn’t be a part of freezer to list remote directories or show files in container | 11:17 |
| Slashme | +1 reldan | 11:20 |
| Slashme | The freezer agent should only be a backup-mashine | 11:20 |
| Slashme | Listing should go in the python-freezerclient | 11:20 |
| Slashme | daemontool: https://review.openstack.org/#/c/251710/ why did you +2 this ? What is the point of that change ? | 11:21 |
| *** szaher_ has quit IRC | 11:37 | |
| *** zhonghua-lee has quit IRC | 11:42 | |
| *** zhonghua-lee has joined #openstack-freezer | 11:42 | |
| *** reldan has quit IRC | 11:48 | |
| *** reldan has joined #openstack-freezer | 12:08 | |
| *** samuelBartel has joined #openstack-freezer | 13:09 | |
| *** zhonghua-lee has quit IRC | 13:14 | |
| *** zhonghua-lee has joined #openstack-freezer | 13:14 | |
| *** jokke__ is now known as jokke_ | 13:25 | |
| szaher | all please, we need to finalize this for me to start working on the agent as well https://etherpad.openstack.org/p/Freezer_arguments | 13:28 |
| openstackgerrit | Memo Garcia proposed openstack/freezer-web-ui: Remove empty tabs in freezer dashboard https://review.openstack.org/254189 | 13:29 |
| szaher | all: please, review this https://review.openstack.org/#/q/status:open,n,z | 13:29 |
| *** reldan has quit IRC | 13:37 | |
| *** samuelBartel has quit IRC | 13:37 | |
| m3m0 | could you send the correct url szaher? | 13:38 |
| szaher | sorry for that m3m0 :) here is the correct one https://review.openstack.org/#/c/252568/ | 13:39 |
| m3m0 | we talked about this last week, you need to specify a bp or a bug for each commit | 13:40 |
| szaher | we need to merge that before adding more features to the scheduler to avoid merge conflict | 13:40 |
| szaher | take another look, there is a blueprint for that commit | 13:40 |
| openstackgerrit | Memo Garcia proposed openstack/freezer-web-ui: Remove empty tabs in freezer dashboard https://review.openstack.org/254189 | 13:41 |
| *** samuelBartel has joined #openstack-freezer | 13:42 | |
| *** reldan has joined #openstack-freezer | 13:47 | |
| daemontool | reldan: Slashme m3m0 frescof vannif szaher ping | 14:00 |
| reldan | I’m here | 14:00 |
| szaher | me too | 14:00 |
| daemontool | ok | 14:00 |
| daemontool | samuelBartel: hi | 14:00 |
| daemontool | we need to talk about the tenant based backup and backup list | 14:00 |
| Slashme | daemontool: Yup ? | 14:00 |
| daemontool | we need at least to write a bp about it | 14:01 |
| daemontool | identifying what are we going to backup | 14:02 |
| daemontool | and how (even multiple options) | 14:02 |
| daemontool | and where | 14:03 |
| reldan | I actually don’t quite understand the use case. So we have a user and this user have an account. And he wants to be able to restore everything in the same way? Or what? | 14:03 |
| daemontool | so | 14:04 |
| daemontool | the use case is the backup as a service | 14:04 |
| daemontool | where a tenant wants to backup all the resources (i.e. data and settings) that belongs to him/her | 14:05 |
| samuelBartel | daemontool, hi | 14:05 |
| daemontool | that same resources needs to be restored | 14:05 |
| daemontool | on the same cloud platform or a different one | 14:05 |
| reldan | Okey, how about network and public IP? | 14:05 |
| reldan | should we be able to restore them as well? | 14:05 |
| daemontool | samuelBartel: did you get the chance to see that now we are an official openstack service? :) | 14:06 |
| daemontool | samuelBartel: thanks for everything you did so far, didn't get the chance to say thanks to you | 14:06 |
| daemontool | reldan: good question | 14:06 |
| daemontool | I think we should backup also the neutron data | 14:06 |
| daemontool | fwaas, lbaas, vpnaas | 14:06 |
| vannif | yes. the BaaS is a fundamental feature IMO | 14:07 |
| daemontool | it needs to be implemented also from the web ui and api of course | 14:08 |
| vannif | definitely | 14:08 |
| reldan | It’s very very big task | 14:08 |
| reldan | We should split it | 14:09 |
| vannif | that's the main interface for normal cloud users | 14:09 |
| reldan | Otherwise it is untrackable from my point of view | 14:09 |
| * vannif agrees with reldan | 14:09 | |
| szaher | I think it may be a good start to use modular mode agent | 14:12 |
| szaher | mod | 14:13 |
| daemontool | I'm wondering if splitting the agent can cause confusion and/or learning curve issue to the users.... | 14:14 |
| daemontool | all: we need to provide a technical proposal | 14:29 |
| daemontool | to the Nova Team | 14:29 |
| daemontool | about a possible approach/integration with freezer and nova instances backups / restore | 14:29 |
| daemontool | reldan: are you in? | 14:30 |
| daemontool | :) | 14:30 |
| samuelBartel | daemontool, great, I miss that information | 14:30 |
| daemontool | Slashme: m3m0 vannif frescof szaher samuelBartel ^^ | 14:30 |
| samuelBartel | you have really made a great job | 14:31 |
| reldan | I don’t know. We need to create a proof of concept I feel. Because they have backups, but it’s not what we want I suppose | 14:31 |
| samuelBartel | congrats all | 14:31 |
| reldan | So we should say - Hi guys, we need this and this and it can be done that way | 14:31 |
| daemontool | reldan: yes, also we can probaly provide a better solution | 14:32 |
| daemontool | to do what they do | 14:32 |
| daemontool | so they currently are snapshotting and uploading the image to glance | 14:32 |
| reldan | Yes, but they have access to the hypervisor and we have no | 14:32 |
| daemontool | my question is | 14:32 |
| daemontool | why the data needs to pass through nova | 14:33 |
| daemontool | sorry | 14:33 |
| daemontool | through glance | 14:33 |
| daemontool | to execute a backup and store the data somewhere? | 14:33 |
| daemontool | reldan: if we provide a good solution | 14:33 |
| daemontool | perhaps they can use the freezer-agent | 14:33 |
| daemontool | to execute the vms backup | 14:33 |
| reldan | What do you mean by nova backup? Do you mean ephimeral disk only? Or all attached volumes as well? | 14:34 |
| daemontool | let's think before the vm image file | 14:34 |
| samuelBartel | daemontool, I hope to have dedicated time to contribute again in the near future but I change team and I am more focused on NFV feature for the moment | 14:35 |
| daemontool | but the final vision would to backup vm image file + attached volumes + neutron networks + etc | 14:35 |
| daemontool | samuelBartel: ah... interesting, so you are working on Neutron now | 14:35 |
| daemontool | ? | 14:35 |
| vannif | I'd like to be quite general here: 1) what do we mean for backup of a VM. ephemeral disk, network state, mounted volumes. 2) different use cases: backup of a stopped vm, backup of a live VM without suspension (nova backup suspends the vm while the snapshotted image is created). | 14:36 |
| daemontool | vannif: I'm referring to 1 | 14:36 |
| vannif | moreover: data integrity. differenciate between the integrity of the image files (to be able to boot the vm again, for instance), and integrity of the data *inside* the VM, which, I think, requires some interaction with what is *inside* the VM (be it the freezer-scheduler or a qemu-agent) | 14:38 |
| daemontool | vannif: the idea would be to place the freezer-agent between the nova api and vm, then yes we need to talk to the hypervisor | 14:39 |
| samuelBartel | daemontool, not directly, the use case is more to work on vnf manager for nfv such as vCPE, vCDN and so on | 14:39 |
| daemontool | samuelBartel: ok | 14:39 |
| daemontool | so we probably need to implement an additional backup method on Nova | 14:40 |
| daemontool | where the use can use the native mode | 14:41 |
| daemontool | or freezer | 14:41 |
| daemontool | if freezer is used | 14:41 |
| daemontool | the agent will talk with the hypervisor | 14:41 |
| daemontool | and so on | 14:41 |
| daemontool | something like, I'd like to know what frescof think about this | 14:41 |
| daemontool | or also another option | 14:46 |
| daemontool | would be to execute the backups on glance | 14:46 |
| daemontool | we need to elaborate an approach | 14:47 |
| daemontool | and get in the backup/restore process of nova vms and cinder volumes | 14:48 |
| daemontool | our agent can definetely provide better features for backup and restore | 14:48 |
| daemontool | than the current ones | 14:48 |
| reldan | I feel that we are trying to solve very different tasks. One of them - creation backups of single machine from the inside. The second - OpenStack resources backup. | 14:50 |
| reldan | And it is really different tasks | 14:50 |
| daemontool | reldan: yes, we are already doing that | 14:50 |
| daemontool | both | 14:50 |
| reldan | I am not sure that it should be mixed | 14:51 |
| daemontool | where it is mixed? | 14:51 |
| daemontool | the code do you mean? | 14:51 |
| daemontool | the code can be splitted, that's not an issue | 14:51 |
| reldan | You see, we have gtar backup and rsync backup for a directory. It can be used without OS at all. And it has sense. | 14:52 |
| daemontool | yes | 14:52 |
| reldan | From other case we would like to do a backup of complex structure - like several VM, several Volumes, Network ... | 14:52 |
| daemontool | after all, everything is a file | 14:53 |
| daemontool | if you think about it | 14:53 |
| daemontool | for kvm | 14:53 |
| daemontool | the hypervisor contains everything | 14:53 |
| daemontool | all the necessary information | 14:53 |
| daemontool | my question | 14:53 |
| daemontool | how would be easier our life if we can execute actions directly on the hypervisor? | 14:54 |
| daemontool | or at least we would get quite significant flexibility | 14:54 |
| reldan | Yes and no. Let’s imagine that I have an online shop. I have 2 VM with mysql with master-slave replication and cinder volumes. 2 VM with nginx with frontend application on php, connected with mysql instances. And I would like to be able to make a backup, then create the same vm’s and volumes (let’s say for testing purposes) | 14:55 |
| reldan | So I need at some point to create a backup at the same time on all nodes | 14:56 |
| reldan | I need syncronization - and it is quite different from backuping a file on one disk | 14:56 |
| reldan | Plus I would like to restore network | 14:56 |
| reldan | And it is much more difficult | 14:57 |
| reldan | 1) I need a way do describe my environement (vm’s, volumes, networks, connection between each of them). 2) I need a way to recreate my environement 3) I need to restore data, attach instance back, fix IP for master/slave replication | 14:59 |
| reldan | And it’s quite different from gtar backup - now it contains a lot of application specific, toplology specific information | 15:00 |
| daemontool | the difference would be | 15:06 |
| daemontool | on making the snapshots at the same time | 15:07 |
| daemontool | and the possible interactions with the hypervisor | 15:07 |
| openstackgerrit | Eldar Nugaev proposed openstack/freezer: Removing obsolete parameters from freezer args (swift related). https://review.openstack.org/253106 | 15:15 |
| daemontool | vannif: do we have currently a way to show the backups from the API? | 15:15 |
| daemontool | the list of taken backups | 15:15 |
| openstackgerrit | Eldar Nugaev proposed openstack/freezer: Removing old integration tests https://review.openstack.org/253117 | 15:16 |
| daemontool | reldan: I'd split the problem | 15:20 |
| daemontool | for now, I'd focus on the best approach | 15:20 |
| daemontool | to backup the vms | 15:21 |
| vannif | no. the backup metadata is in the db but the freezer-scheduler does not have any browse/search tool in place to retrieve that information | 15:23 |
| reldan | daemontool: Ephemeral or volume based? | 15:23 |
| reldan | Probably we shouldn’t care about ephemeral | 15:24 |
| *** zhonghua-lee has quit IRC | 15:25 | |
| *** zhonghua-lee has joined #openstack-freezer | 15:26 | |
| daemontool | reldan: ephemeral may contains | 15:29 |
| daemontool | important informations | 15:29 |
| daemontool | like users | 15:29 |
| daemontool | settings | 15:30 |
| daemontool | config | 15:30 |
| reldan | But ephemeral disk can be erased even after rebooting | 15:30 |
| reldan | So you should keep only caches and nonsensetive information there | 15:31 |
| reldan | Data on ephemeral storage ceases to exist when the instance it is associated with is terminated. Rebooting the VM or restarting the host server, however, will not destroy ephemeral data. In the typical use case an instance's root filesystem is stored on ephemeral storage. This is often an unpleasant surprise for people unfamiliar with the cloud model of computing. | 15:31 |
| reldan | Oh, I see it is different in OS and AWS | 15:31 |
| daemontool | so the challenge is, how to backup all the rsources belonging to one tenant | 15:32 |
| daemontool | and if there's anything that can facilitate our job by doing actions on the comput node or glance or anything else | 15:42 |
| daemontool | all: had a quick conversation with jokke_ and the Nova PTL | 15:43 |
| daemontool | it turns out most probably the easies way to achieve this for nova vm is to leverage glance | 15:44 |
| daemontool | so like the approach we are using currently | 15:44 |
| daemontool | reldan: do you remember if the cinder compress the volumes when they are snapshotted and converted to an image? | 15:49 |
| reldan | Yes, but how we can have incremental backup in this case? | 15:49 |
| daemontool | not on glance | 15:50 |
| daemontool | only on the media storage after swift processing | 15:50 |
| daemontool | and only after the block based incremental | 15:50 |
| jokke_ | reldan: snapshot is incremental already | 15:50 |
| daemontool | is implemented on freezer | 15:50 |
| jokke_ | iirc | 15:51 |
| reldan | Snapshots are - yes, but we cannot sotre it on swift | 15:51 |
| reldan | store | 15:51 |
| jokke_ | reldan: so when you backup that glance image, which is result of snapshot that is automatically incremental backup | 15:51 |
| jokke_ | you don't need to do the incremental part yourself as the snapshotting took care of it already | 15:52 |
| daemontool | jokke_: that's for cinder or nova? | 15:52 |
| reldan | For cinder snapshot is just a delta and cannot be used without a volume | 15:53 |
| reldan | For nova volume based - same | 15:53 |
| jokke_ | reldan: initial snapshot can be used to create the initial volume and then multiple snapshots can be built on top of that ... same applies to the nova snapshots that are not volume based (well they have the base image always) | 15:55 |
| reldan | Sure, we can use snapshots and it will be incremental. But the current approach was - to creation a snapshot, creation volume, volume -> glance image, glance image -> download, upload to swift or local storage or ... | 15:58 |
| reldan | Sure we can use snapshots and backups in cinder and nova without freezer | 15:58 |
| *** dschroeder has joined #openstack-freezer | 15:59 | |
| *** daemontool has quit IRC | 16:06 | |
| openstackgerrit | Memo Garcia proposed openstack/freezer-web-ui: Remove empty tabs in freezer dashboard https://review.openstack.org/254189 | 16:28 |
| openstackgerrit | Memo Garcia proposed openstack/freezer-web-ui: Remove empty tabs in freezer dashboard https://review.openstack.org/254189 | 17:04 |
| *** samuelBartel has quit IRC | 17:31 | |
| *** reldan has quit IRC | 18:01 | |
| *** szaher has quit IRC | 18:35 | |
| *** vannif_ has joined #openstack-freezer | 19:16 | |
| *** vannif has quit IRC | 19:16 | |
| *** reldan has joined #openstack-freezer | 20:40 | |
| *** reldan has quit IRC | 21:24 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!