*** yangyapeng has quit IRC | 00:24 | |
*** ddieterly[away] has quit IRC | 00:26 | |
*** ddieterly has joined #openstack-freezer | 00:40 | |
*** yangyapeng has joined #openstack-freezer | 01:02 | |
*** EinstCrazy has joined #openstack-freezer | 01:29 | |
*** yangyape_ has joined #openstack-freezer | 01:41 | |
*** yangyapeng has quit IRC | 01:44 | |
*** dschroeder has quit IRC | 01:45 | |
*** ddieterly is now known as ddieterly[away] | 01:47 | |
*** yangyape_ has quit IRC | 01:49 | |
*** ddieterly[away] is now known as ddieterly | 02:21 | |
*** ddieterly is now known as ddieterly[away] | 02:26 | |
*** EinstCrazy has quit IRC | 02:35 | |
*** EinstCrazy has joined #openstack-freezer | 02:36 | |
*** ddieterly[away] has quit IRC | 03:26 | |
*** zhangjn has quit IRC | 05:45 | |
*** zhangjn has joined #openstack-freezer | 05:46 | |
*** EinstCra_ has joined #openstack-freezer | 06:59 | |
*** EinstCrazy has quit IRC | 07:02 | |
*** reldan has joined #openstack-freezer | 07:31 | |
*** daemontool has joined #openstack-freezer | 07:32 | |
daemontool | all, are we ok to tag current master for the freezer and freezer-api repo for Mitaka release? :) | 07:53 |
---|---|---|
daemontool | any outstanding bug needs to get in? | 07:53 |
daemontool | please approve the pending devstack fixes also | 07:53 |
*** reldan has quit IRC | 08:10 | |
*** daemontool has quit IRC | 08:41 | |
EinstCra_ | https://review.openstack.org/299277 https://review.openstack.org/298588 https://review.openstack.org/298628 I think these 3 bug need to get in | 08:50 |
*** EinstCra_ is now known as EinstCrazy | 08:51 | |
EinstCrazy | These 3 bugs is about cinder native backup and restore | 08:51 |
*** reldan has joined #openstack-freezer | 08:54 | |
slashme | Bugs | 09:16 |
slashme | We have a few | 09:16 |
slashme | Let's tackle that right now | 09:16 |
slashme | https://review.openstack.org/#/c/301800/ | 09:16 |
slashme | https://review.openstack.org/#/c/298628/ | 09:16 |
slashme | https://review.openstack.org/#/c/301808/ | 09:16 |
slashme | https://review.openstack.org/#/c/302160/ | 09:16 |
slashme | https://review.openstack.org/#/c/302151/ | 09:17 |
slashme | For 298628, it's not even merged into master | 09:17 |
slashme | I'll backport it as soon as it is approved | 09:17 |
*** yangyapeng has joined #openstack-freezer | 09:23 | |
*** reldan has quit IRC | 10:10 | |
*** reldan has joined #openstack-freezer | 10:11 | |
*** daemontool has joined #openstack-freezer | 10:22 | |
reldan | +Workflow please to Deklan https://review.openstack.org/#/c/301250/5 | 10:29 |
*** EinstCrazy has quit IRC | 10:30 | |
*** yangyapeng has quit IRC | 10:43 | |
*** reldan has quit IRC | 10:44 | |
*** reldan has joined #openstack-freezer | 11:03 | |
reldan | +Workflow please to Einst Crazy https://review.openstack.org/#/c/266813/ | 11:28 |
reldan | +Workflow plase to Daiki Kato https://review.openstack.org/#/c/297565/1 | 11:29 |
*** yangyapeng has joined #openstack-freezer | 11:40 | |
reldan | ValueError: git history requires a target version of pbr.version.SemanticVersion(2.0.1), but target version is pbr.version.SemanticVersion(2.0.0) | 11:47 |
reldan | 2016-04-07 11:27:07.810 | error in setup command: Error parsing /home/jenkins/workspace/gate-freezer-docs/setup.cfg: ValueError: git history requires a target version of pbr.version.SemanticVersion(2.0.1), but target version is pbr.version.SemanticVersion(2.0.0) | 11:47 |
reldan | daemontool: Do you know what is it? | 11:48 |
slashme | I know | 11:57 |
slashme | Release was tagged | 11:57 |
slashme | From now on, every backport to mitaka needs to increment the version | 11:58 |
openstackgerrit | Pierre Mathieu proposed openstack/freezer: Generate a full backup first time https://review.openstack.org/298628 | 11:59 |
yangyapeng | HI | 12:03 |
openstackgerrit | Saad Zaher proposed openstack/freezer: Add Freezer Tempest Tests https://review.openstack.org/301250 | 12:04 |
slashme | Just to be sure. | 12:04 |
slashme | - Kilo : 1.1.x | 12:04 |
slashme | - Liberty : 1.2.x | 12:04 |
slashme | - Mitaka : 2.0.x | 12:04 |
slashme | - Newton : 2.1.x ?? | 12:04 |
openstackgerrit | Pierre Mathieu proposed openstack/freezer: Generate a full backup first time https://review.openstack.org/298628 | 12:04 |
yangyapeng | when backup_cinder , generate the file in self.container , that can Overwrite the original backup. | 12:05 |
*** EinstCrazy has joined #openstack-freezer | 12:06 | |
yangyapeng | ping EinstCrazy | 12:07 |
EinstCrazy | yes? | 12:07 |
yangyapeng | https://review.openstack.org/298628 | 12:07 |
openstackgerrit | Einst Crazy proposed openstack/freezer: fix some rst syntax error of code block https://review.openstack.org/266813 | 12:11 |
*** reldan has quit IRC | 12:26 | |
*** reldan has joined #openstack-freezer | 12:35 | |
openstackgerrit | Pierre Mathieu proposed openstack/freezer: Add Freezer Tempest Tests https://review.openstack.org/301250 | 12:38 |
daemontool | meeting is in 1 hour right> | 13:07 |
daemontool | _ | 13:07 |
daemontool | ? | 13:07 |
daemontool | I'll be back shortly | 13:08 |
*** daemontool has quit IRC | 13:17 | |
openstackgerrit | Merged openstack/freezer: Generate a full backup first time https://review.openstack.org/298628 | 13:23 |
slashme | daemontool | 13:27 |
slashme | yes | 13:27 |
*** daemontool has joined #openstack-freezer | 13:28 | |
*** ddieterly has joined #openstack-freezer | 13:54 | |
*** ddieterly is now known as ddieterly[away] | 13:56 | |
slashme | Backports to liberty and mitaka https://review.openstack.org/302780 and https://review.openstack.org/302779 | 14:00 |
m3m0 | guys please move to #openstack-meeting-alt | 14:00 |
openstackgerrit | Einst Crazy proposed openstack/freezer: Use volume_id and timestamp to generate container name https://review.openstack.org/298588 | 14:11 |
openstackgerrit | Einst Crazy proposed openstack/freezer: Restore cinder native backup pass right params and call right api https://review.openstack.org/299277 | 14:21 |
*** ddieterly has joined #openstack-freezer | 14:42 | |
*** kelepirci_ has joined #openstack-freezer | 14:56 | |
*** kelepirci has quit IRC | 14:57 | |
daemontool | hang on I remember os.walk | 15:00 |
daemontool | supports | 15:00 |
daemontool | exclude | 15:00 |
daemontool | one sec | 15:00 |
*** kelepirci_ is now known as kelepirci | 15:01 | |
daemontool | mmmhh nope | 15:01 |
daemontool | :( | 15:01 |
ddieterly | are we still meeting? | 15:03 |
daemontool | well we are discussing about the solution | 15:03 |
daemontool | the meeting is over | 15:03 |
ddieterly | ok | 15:03 |
daemontool | anyway | 15:03 |
clsacramento | daemontool: do you agree we have to separate this exclusion from the engine? | 15:04 |
daemontool | clsacramento, the generation of the list yes | 15:04 |
daemontool | I do not agree | 15:04 |
daemontool | with scanning the file system twie | 15:04 |
daemontool | twice | 15:04 |
daemontool | in that case is better to use regex against each file | 15:05 |
daemontool | and scanning the fs once | 15:05 |
daemontool | is moving that outside the engine | 15:05 |
slashme | daemontool: Not possible with tar | 15:05 |
daemontool | s/is/if/ | 15:05 |
daemontool | they didn't do that becaus eif inefficient | 15:06 |
daemontool | s/if/is/ | 15:06 |
frescof | hmmmm why separating the exclusion from the engine ? | 15:08 |
clsacramento | daemontool: if you separate I dont see how to avoid walking the path twice | 15:08 |
slashme | frescof: Having a consistent exclude pattern between egines | 15:08 |
daemontool | clsacramento, ok, I'm not sure we have more benefit if we walk twice | 15:08 |
clsacramento | frescof: each engine implements exclusion differently, so we can't control what is really going to be excluded | 15:08 |
*** dschroeder has joined #openstack-freezer | 15:09 | |
daemontool | the idea of having a consistent exclude | 15:10 |
daemontool | is absolutely right | 15:10 |
daemontool | bloody binaries... until we do not get rid of them | 15:10 |
daemontool | we have this issues | 15:10 |
frescof | I agree bus as said this is not possible without the double walk | 15:11 |
daemontool | frescof, do you agree with the double walk? | 15:11 |
frescof | nope | 15:11 |
daemontool | it generated a lots of I/O... and time inefficient | 15:11 |
frescof | and what if the list of exclude files generated contains millions of files with a very long path ? | 15:12 |
frescof | it's not only an i/o problem, even memory | 15:12 |
slashme | frescof yes | 15:12 |
clsacramento | ok, the other solution that I thought of is to have the engine return the list of backed up files to be used by the checksum module | 15:12 |
ddieterly | you can use a bloom filter | 15:13 |
frescof | so, for me, no double scan | 15:13 |
frescof | better the inconsistency that that | 15:13 |
ddieterly | get a list of files to exclude, put them in a bloom filter, which will fit in memory, then for each file,you can later look in the bloom filter to see if it is there | 15:13 |
slashme | We can also ensure that all engines implement the exclude in the same way | 15:13 |
ddieterly | bloom filters are a very efficient way to do lookups on large datasets | 15:14 |
daemontool | ddieterly, I don't know about that | 15:14 |
daemontool | I'm going to read | 15:14 |
ddieterly | https://en.wikipedia.org/wiki/Bloom_filter | 15:15 |
daemontool | ok | 15:15 |
daemontool | ty | 15:15 |
ddieterly | they are used in very scalable situations | 15:15 |
frescof | hmmm and how that can be done with tar | 15:16 |
daemontool | good question... | 15:17 |
frescof | maybe there is something I'm missing here | 15:17 |
daemontool | I can do that in rsync do | 15:17 |
clsacramento | frescof: the bloom_filter ? | 15:17 |
frescof | yep | 15:17 |
*** yangyapeng has quit IRC | 15:17 | |
clsacramento | for tar we can pass a file containing the files to exclude | 15:18 |
frescof | ah lol | 15:18 |
frescof | so not only we scan the tree twice but we even save the list on disk | 15:18 |
slashme | frescof agreed, that's not really acceptable | 15:19 |
slashme | So how about we just ensure that our engines implement the exclude in the same way. Doing that, we don;t have to walk twice and still have a constistent behaviour | 15:20 |
frescof | with tests ? :) | 15:21 |
ddieterly | did anyone want to discuss https://review.openstack.org/#/c/301250/? | 15:22 |
ddieterly | OW, i'm going to go get a coffee ;-) | 15:22 |
clsacramento | I'm not sure we can force an engine to behave like we an on that | 15:23 |
clsacramento | for tar, it has its own of dealing with pattern and we cannot change it | 15:23 |
frescof | clsacramento, yep sure | 15:24 |
daemontool | I think we need to decide how do we want to match | 15:24 |
daemontool | like exact | 15:24 |
daemontool | brown == brown | 15:24 |
daemontool | match in any place | 15:24 |
daemontool | brown match brownsugar | 15:24 |
reldan | Hmm, could you please explain what is so difficult with excluding and checksum | 15:25 |
*** ddieterly is now known as ddieterly[away] | 15:25 | |
daemontool | reldan, I don't think is difficult, it's just that we need to have same way to exclude | 15:25 |
daemontool | amongst multiple engine | 15:25 |
daemontool | which make totally sense | 15:25 |
clsacramento | reldan: to be sure that what checksum assume is excluded is the same that actually is excluded if the exclusion is not standard among engines | 15:25 |
daemontool | so, brown == brown | 15:26 |
daemontool | or brown match brownsugar? | 15:26 |
reldan | Oh, I see now. Thank you | 15:26 |
reldan | 1) There are problem with incremental | 15:26 |
reldan | let’s say level 0 with exclude and level 1 without | 15:27 |
reldan | Or level 0 without and level 1 with | 15:27 |
frescof | reldan, hmm and what's the problem ? | 15:28 |
clsacramento | reldan: I think is not a problem, the checksum is run at the same time of backup with the same options | 15:28 |
clsacramento | and the checsum is run on the original files, not on the files in the backup container | 15:29 |
reldan | It is my qustion yes. My first backup is with —exlude=*.log And incremental without exclude. Should increment contains files excluded in level0 ? | 15:31 |
frescof | reldan, yes | 15:32 |
clsacramento | The it is implemented today, if dont pass again the exlude option on you incremental, it will not exclude | 15:33 |
*** EinstCrazy has quit IRC | 15:36 | |
reldan | And if I run my wirst backup without exclude and my incremental with —exclude. Should it mark all excluded files as deleted? | 15:36 |
*** ddieterly[away] is now known as ddieterly | 15:37 | |
reldan | And still don’t see the problem. During checksum - we just skip files by wildcard. And it is all. During restore we should have the same directory like before backup. So checksum should be the same | 15:40 |
reldan | What am I missing? | 15:40 |
reldan | Or it is discussion about arguments | 15:41 |
clsacramento | reldan: I think it is out of scope for the backup consistency checksum... in any case it has to checksum the user files regardless previous backups | 15:41 |
clsacramento | I'm trying to implement backup consistency. To do that, I am checksumming the files before backup, checksumming the files after restore and comparing the two checksums | 15:42 |
clsacramento | now if the backup is going to exclude files, the same files have to be excluded from the checksum | 15:42 |
reldan | Yes, right | 15:43 |
reldan | And I still don’t see any problem | 15:43 |
*** ddieterly is now known as ddieterly[away] | 15:44 | |
clsacramento | if have no way to ensure the files I exclude on the checksum are the same excluded on the backups because the engines don't implement file exclusion in the same way | 15:44 |
frescof | ddieterly[away], https://review.openstack.org/#/c/301250 seems ok to me | 15:45 |
reldan | If file is excluded it should not be in restore? Nope? | 15:45 |
reldan | We just don’t have such file | 15:45 |
clsacramento | but I don't have the original list of files backed up, just the result of the checksum | 15:46 |
*** ddieterly[away] is now known as ddieterly | 15:46 | |
reldan | Yes, so we just should compare checksum | 15:46 |
clsacramento | when I restore, I just have the restored files and the checksum of the original files | 15:47 |
clsacramento | yes | 15:47 |
reldan | yes | 15:47 |
ddieterly | frescof ok, thanks | 15:47 |
reldan | But still don’t understand what is the problem. During backup we calculate checksum without excluded files, during restore we restore everything except excluded | 15:47 |
reldan | checksums should be equal | 15:48 |
frescof | reldan, agree | 15:48 |
clsacramento | You mean, on backup I checksum all files? | 15:48 |
reldan | all files except excluded, yes | 15:48 |
clsacramento | ok, this is the problem, how do I retrieve the excluded files | 15:49 |
clsacramento | each engine will determine different files for the same pattern | 15:49 |
reldan | clsacramentol: we should have our regexp and if some engine supports different pattern - we can provide adaptor, that transform one format to another | 15:50 |
clsacramento | reldan: I think it is good idea and simpler of anything that we have came up so far | 15:51 |
clsacramento | We still need to agree on which pattern should be given on the --exclude option in freezer agent | 15:52 |
reldan | Otherwise it should be hell for our customer. | 15:52 |
reldan | I like tar format :) Currently it is only engine we have | 15:53 |
reldan | So I can write adapter | 15:53 |
reldan | str -> str | 15:53 |
slashme | +1 on reldan's solution :) | 15:53 |
clsacramento | thank you for thinking about that, we spent 40 minutes of the meeting for that and could not think of anything as good | 15:54 |
clsacramento | Me and slashme we prefer the glob pattern which is a little bit different from the tar one though | 15:55 |
reldan | Yes, I don’t insist that it shold be tar format. If we can transform glob to tar. I’m 100% fine with that solution | 15:55 |
slashme | Tar: .py | 15:56 |
slashme | Glob: *.py | 15:56 |
slashme | Regex: py$ | 15:56 |
reldan | tar -cf src.tar --exclude='*.o' src | 15:58 |
reldan | So I suppose for tar it is *.py as well | 15:58 |
reldan | https://www.gnu.org/software/tar/manual/html_chapter/tar_6.html | 15:59 |
clsacramento | Tar: if you pass --exclude=.py it will exclude any file that has aney .py anywhere | 15:59 |
clsacramento | in glob you need *.py* for that | 15:59 |
daemontool | so why not do that in rsync too (match the substring) and the problem is solved | 16:00 |
clsacramento | *.py works the same in both | 16:00 |
reldan | daemontool: Yes, I suppose we can keep it simple | 16:01 |
daemontool | so while we walk the fs | 16:01 |
daemontool | it's just an | 16:01 |
clsacramento | daemontool: you mean force rsync to function the same as tar ? | 16:01 |
daemontool | clsacramento, yes | 16:02 |
daemontool | it is important | 16:02 |
daemontool | but in my opinion | 16:02 |
daemontool | invest energies so write an adapter | 16:02 |
daemontool | it is a bit too much now | 16:02 |
clsacramento | daemontool: do we risk of having any other engine, that like tar we cant modify, and that behaves differently with the pattern? | 16:03 |
daemontool | I can't think of one now | 16:03 |
daemontool | rsync, tar, the data we get from glance, native backups | 16:04 |
clsacramento | daemontool: ok, than I'll make checksum behave like tar, and you make rsync behave like tar for now | 16:04 |
daemontool | those are the cases | 16:04 |
daemontool | clsacramento, I think that is good | 16:04 |
daemontool | what would be good to do | 16:04 |
reldan | Good! | 16:04 |
daemontool | is implement the listed-incremental feature | 16:04 |
daemontool | in tarfile module | 16:04 |
daemontool | in python | 16:04 |
daemontool | so we remove all the bloody binaries | 16:04 |
daemontool | and then we can implement the exclude we want | 16:04 |
daemontool | without binaries limitations | 16:05 |
daemontool | reldan, are you up to that? :D | 16:05 |
daemontool | lol | 16:05 |
clsacramento | daemontool: we just have to be careful to make sure that we walk exactly like tar with all patterns | 16:05 |
daemontool | you need to take a sabatical years off after that... | 16:05 |
daemontool | :) | 16:05 |
daemontool | yes | 16:05 |
daemontool | clsacramento, yes | 16:06 |
reldan | daemontool: It shouldn’t be too hard :) But probably it should be too hard to force it work fast :) | 16:06 |
daemontool | it's not easy | 16:06 |
daemontool | believe me | 16:06 |
reldan | But git does the same | 16:07 |
reldan | I can use the same approach from git | 16:07 |
daemontool | git uses its own format | 16:07 |
reldan | But more or less, I need to check dates owner size | 16:08 |
slashme | Pending Reviews: | 16:08 |
slashme | - Backports Cinder incremental : | 16:08 |
slashme | https://review.openstack.org/302780 | 16:08 |
slashme | https://review.openstack.org/302779 | 16:08 |
slashme | - Typos: | 16:08 |
slashme | https://review.openstack.org/#/c/297565 | 16:08 |
slashme | https://review.openstack.org/#/c/266813/ | 16:08 |
slashme | - Cinder Bugs: ?? Do these need backport to liberty/mitaka ?? | 16:08 |
slashme | https://review.openstack.org/#/c/298588/ | 16:08 |
slashme | https://review.openstack.org/#/c/299277/ | 16:08 |
reldan | If I have change - I just pack whole file and mark like it is new | 16:08 |
reldan | With deletion it is also not very complex | 16:09 |
clsacramento | daemontool: I wrote this conclusion on the etherpad, so we can keep in mind the translator if we ever need to | 16:10 |
reldan | I don’t say that it is easy, but doesn’t look like rocket science | 16:10 |
daemontool | clsacramento, thanks a lot :) | 16:10 |
daemontool | slashme, are those real bugs (Cinder bugs)? | 16:12 |
daemontool | reldan, if we use tarfile, we already have a lots of functions | 16:13 |
slashme | EinstCrazy / reldan ?? | 16:13 |
daemontool | and other thigns | 16:13 |
daemontool | we can reuse | 16:13 |
openstackgerrit | Merged openstack/freezer: Add Freezer Tempest Tests https://review.openstack.org/301250 | 16:13 |
daemontool | that's why I was proposing that | 16:13 |
daemontool | please do it, or I have to do it after rsync :) | 16:13 |
daemontool | clsacramento, if you want& have time | 16:14 |
daemontool | please review https://review.openstack.org/#/c/290461/ | 16:14 |
daemontool | it's not working the restore now | 16:14 |
daemontool | but the general code is there | 16:14 |
daemontool | the backup works | 16:14 |
reldan | slashme: I suppse https://review.openstack.org/#/c/298588/ can go to liberty/mitaka https://review.openstack.org/#/c/299277/ - Einst should know better | 16:14 |
slashme | I wanted his opinion | 16:15 |
daemontool | https://review.openstack.org/#/c/299277/ added a comment | 16:16 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/freezer: Updated from global requirements https://review.openstack.org/302932 | 16:17 |
*** yangyapeng has joined #openstack-freezer | 16:18 | |
clsacramento | daemontool: I'll take a look | 16:18 |
*** ddieterly is now known as ddieterly[away] | 16:23 | |
daemontool | clsacramento, ty | 16:23 |
*** yangyapeng has quit IRC | 16:25 | |
openstackgerrit | Merged openstack/freezer: Use volume_id and timestamp to generate container name https://review.openstack.org/298588 | 16:36 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/freezer: Updated from global requirements https://review.openstack.org/302932 | 16:39 |
*** daemontool has quit IRC | 16:44 | |
*** ddieterly[away] is now known as ddieterly | 16:46 | |
*** reldan has quit IRC | 17:03 | |
*** ddieterly is now known as ddieterly[away] | 17:08 | |
*** ddieterly[away] is now known as ddieterly | 17:13 | |
openstackgerrit | Deklan Dieterly proposed openstack/freezer: Add More Freezer Tempest Tests https://review.openstack.org/303007 | 17:35 |
*** ddieterly is now known as ddieterly[away] | 17:36 | |
*** yangyapeng has joined #openstack-freezer | 18:22 | |
*** yangyapeng has quit IRC | 18:27 | |
*** ddieterly[away] has quit IRC | 18:36 | |
*** ddieterly has joined #openstack-freezer | 18:42 | |
*** ddieterly is now known as ddieterly[away] | 18:44 | |
*** ddieterly[away] is now known as ddieterly | 18:46 | |
*** reldan has joined #openstack-freezer | 18:47 | |
*** ddieterly is now known as ddieterly[away] | 19:00 | |
*** reldan has quit IRC | 19:14 | |
*** ddieterly[away] is now known as ddieterly | 19:21 | |
*** ddieterly is now known as ddieterly[away] | 20:10 | |
*** ddieterly[away] is now known as ddieterly | 20:15 | |
openstackgerrit | Deklan Dieterly proposed openstack/freezer: Add More Freezer Tempest Tests https://review.openstack.org/303007 | 20:23 |
*** ddieterly is now known as ddieterly[away] | 20:49 | |
*** ddieterly[away] has quit IRC | 20:52 | |
*** ddieterly has joined #openstack-freezer | 22:01 | |
*** ddieterly is now known as ddieterly[away] | 22:18 | |
*** ddieterly[away] is now known as ddieterly | 22:19 | |
*** ddieterly is now known as ddieterly[away] | 22:30 | |
*** ddieterly[away] is now known as ddieterly | 22:30 | |
*** daemontool has joined #openstack-freezer | 22:48 | |
*** dschroeder has quit IRC | 23:25 | |
*** ddieterly is now known as ddieterly[away] | 23:33 | |
*** ddieterly[away] has quit IRC | 23:33 | |
*** ddieterly has joined #openstack-freezer | 23:36 | |
*** ddieterly is now known as ddieterly[away] | 23:48 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!