Tuesday, 2014-10-14

*** exploreshaifali has quit IRC00:18
*** echevemaster has joined #openstack-zaqar00:38
*** amitgandhinz has quit IRC00:42
openstackgerritJeffrey Zhang proposed a change to openstack/zaqar: Add enum34 requirement  https://review.openstack.org/12782801:28
*** cpallares has quit IRC01:31
*** cpallares has joined #openstack-zaqar01:34
*** jeffrey4l has quit IRC01:55
*** jeffrey4l has joined #openstack-zaqar02:08
*** X019 has quit IRC02:11
openstackgerritJeffrey Zhang proposed a change to openstack/zaqar: Sync the zaqar.conf.sample file  https://review.openstack.org/12815802:43
*** cpallares has quit IRC02:54
*** flwang has quit IRC02:54
*** flwang has joined #openstack-zaqar03:07
*** cpallares has joined #openstack-zaqar03:09
openstackgerritA change was merged to openstack/zaqar: Fix no links property in flavor list response  https://review.openstack.org/12362603:22
*** jeffrey4l has quit IRC03:44
*** jeffrey4l has joined #openstack-zaqar04:05
*** bradjones has quit IRC05:40
*** vipul has quit IRC05:40
*** bradjones has joined #openstack-zaqar05:46
*** vipul has joined #openstack-zaqar05:46
*** echevemaster has quit IRC05:57
*** jeffrey4l has quit IRC06:31
*** sgotliv has quit IRC06:35
*** jeffrey4l has joined #openstack-zaqar06:43
*** exploreshaifali has joined #openstack-zaqar06:48
*** exploreshaifali has quit IRC06:52
*** jeffrey4l has quit IRC06:59
*** flaper87|afk is now known as flaper8707:03
*** jeffrey4l has joined #openstack-zaqar07:08
*** sgotliv has joined #openstack-zaqar07:36
openstackgerritJeffrey Zhang proposed a change to openstack/zaqar: Fix the exception raised when using the nosetests run test  https://review.openstack.org/12819607:38
openstackgerritJeffrey Zhang proposed a change to openstack/zaqar: Fix the exception raised when running test by nose  https://review.openstack.org/12819607:41
*** exploreshaifali has joined #openstack-zaqar07:53
exploreshaifaliflaper87: all the configs ofmongodb comes from https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/options.py08:01
exploreshaifaliflaper87: since we need to seperate the configs, I don't understand what exactly should be done.08:16
exploreshaifalioptions.py as per I understood is defining configs for mongodb08:17
exploreshaifaliso that file should be intact08:17
exploreshaifaliit is?08:17
*** sgotliv has quit IRC08:50
*** sgotliv has joined #openstack-zaqar09:02
*** jeffrey4l has quit IRC09:20
*** jeffrey4l has joined #openstack-zaqar09:31
*** njirap has joined #openstack-zaqar10:26
*** sgotliv has quit IRC10:32
*** bradjones has quit IRC10:33
*** bradjones has joined #openstack-zaqar10:33
*** sgotliv has joined #openstack-zaqar10:41
openstackgerritA change was merged to openstack/python-zaqarclient: Updated from global requirements  https://review.openstack.org/12781410:56
*** malini has joined #openstack-zaqar11:51
openstackgerritFlavio Percoco proposed a change to openstack/python-zaqarclient: Setup developer docs for zaqarclient  https://review.openstack.org/12717111:57
openstackgerritFlavio Percoco proposed a change to openstack/python-zaqarclient: Extend some docstrings with useful information  https://review.openstack.org/12722711:57
openstackgerritFlavio Percoco proposed a change to openstack/python-zaqarclient: Add docs for `Client` instances  https://review.openstack.org/12718211:57
openstackgerritFlavio Percoco proposed a change to openstack/python-zaqarclient: Add reference docs for latest recommended client  https://review.openstack.org/12721511:57
*** malini has quit IRC11:57
*** julienvey has quit IRC12:10
*** jchai has joined #openstack-zaqar12:57
*** njirap has quit IRC13:07
flaper87jeffrey4l: zhiyan kgriffs|afk malini|afk sebasmagri cpallares exploreshaifali flwang guys and girls, just a heads up. Please, write your RFCs for Kilo in the next couple of weeks (before the summit) and help reviewing the existing ones.13:08
flaper87We've gotta get our plan for kilo sorted out13:08
exploreshaifaliflaper87: I don't know what a kilo mean?13:17
flaper87exploreshaifali: hehe, that's the name of the next OpenStack release13:17
flaper87the one you'll be working on13:17
*** mpanetta has joined #openstack-zaqar13:17
exploreshaifaliit is Juno?13:17
flaper87nope, Juno is going to be release this week13:18
exploreshaifaliflaper87: yes 16-oct13:18
flaper87exploreshaifali: exactly, the next one is called Kilo13:18
flaper87and that's the one you'll be hacking on13:18
exploreshaifaliok :)13:18
zhiyanSounds cool, will do13:19
zhiyanthanks for reminder ^13:19
*** njirap has joined #openstack-zaqar13:20
flaper87These are the RFCs we have so far: https://review.openstack.org/#/q/status:open+project:openstack/zaqar-specs,n,z13:20
*** yfujioka has quit IRC13:22
exploreshaifalizhiyan: please can you help me a bit what we are suppose to do?13:22
flaper87exploreshaifali: https://github.com/openstack/zaqar-specs13:23
flaper87you need to propose a patch similar to https://review.openstack.org/#/c/126531/13:24
flaper87That is, you need to copy the template.rs into specs/kilo/13:24
flaper87with the name of your spec file and edit it13:24
flaper87exploreshaifali: your spec should contain the info of what you're planning to do with the data planes13:24
flaper87data/control13:24
flaper87why that change is needed etc13:24
flaper87The template has all the questions that need to be answered13:25
flaper87:)13:25
flaper87it should be pretty straightforward13:25
exploreshaifaliflaper87: ok, Thanks!!13:25
* flaper87 goes and writes yet-another-spec13:25
openstackgerritA change was merged to openstack/zaqar-specs: Template for new storage driver's addition  https://review.openstack.org/12565813:26
*** sriram has joined #openstack-zaqar13:27
*** exploreshaifali has quit IRC13:30
openstackgerritFlavio Percoco proposed a change to openstack/zaqar-specs: Propose a drawbacks section  https://review.openstack.org/12827713:33
*** flaper87 has quit IRC13:35
*** malini has joined #openstack-zaqar13:48
*** flaper87|afk has joined #openstack-zaqar14:09
openstackgerritFlavio Percoco proposed a change to openstack/zaqar-specs: Make FIFO guarantee optional  https://review.openstack.org/12598614:12
*** flaper87|afk is now known as flaper8714:12
*** flaper87 has quit IRC14:14
*** flaper87 has joined #openstack-zaqar14:14
*** amitgandhinz has joined #openstack-zaqar14:15
*** anna_ has joined #openstack-zaqar15:04
*** jchai is now known as jchai_afk15:26
*** jchai_afk is now known as jchai15:27
*** sgotliv has quit IRC15:33
*** riveter has joined #openstack-zaqar16:09
*** kgriffs|afk is now known as kgriffs16:34
*** flaper87 is now known as flaper87|afk16:38
*** sriram has quit IRC16:46
*** sriram1 has joined #openstack-zaqar16:47
*** njirap has quit IRC17:07
*** exploreshaifali has joined #openstack-zaqar17:08
*** anna_ has quit IRC17:13
*** anna_ has joined #openstack-zaqar17:25
*** jchai is now known as jchai_afk17:41
*** sgotliv has joined #openstack-zaqar18:03
*** sebasmagri has quit IRC18:06
*** sebasmagri has joined #openstack-zaqar18:08
*** jchai_afk is now known as jchai18:25
*** earnThis has joined #openstack-zaqar19:28
exploreshaifalikgriffs, I am trying to understand https://etherpad.openstack.org/p/zaqar-scratch please can you help me in it19:33
kgriffssure19:34
kgriffslet me add an example showing what was decided19:34
*** bradjones has quit IRC19:36
cpallaresflaper87|afk: What are RFCs?19:38
kgriffsexploreshaifali: does that help?19:38
exploreshaifalikgriffs, very fist our objective is to separate data and control plane of storage layer19:38
exploreshaifaliand for that first task is to split the configs19:39
kgriffskk19:39
exploreshaifalikgriffs, these are the configs of datastore right?19:39
kgriffsso, that example I put right under "config options notes" is the schema flaper87 and I came up with19:40
kgriffswhereas before you had just "storage"19:40
kgriffsnow that is split into19:41
kgriffs"message_store"19:41
kgriffsand19:41
kgriffs"management_store"19:41
exploreshaifalimy question is how can we divide the configs of a datastore?19:41
exploreshaifalidatastore configs means the configurations of datastore with which our applications will going to interact19:42
exploreshaifaliand these config options will remain same always, their values may change19:43
exploreshaifalikgriffs, am I able to put my question clearly?19:43
kgriffsok, let me see if I understand the question19:44
kgriffstoday we have a single module that defines two classes19:44
exploreshaifaliyes19:45
kgriffsDataDriver19:45
kgriffsdn19:45
kgriffsand19:45
kgriffsControlDriver19:45
exploreshaifaliyes19:45
kgriffsbut, we use the same config section from the INI file to configure both19:45
kgriffswe just pick which one to load, depending on where pooling is enabled19:46
kgriffs(see also bootstrap.py)19:46
kgriffsactually19:46
kgriffsbootstrap loads a meta-driver from the pooling module19:46
kgriffsjust a sec19:47
exploreshaifalihttps://github.com/openstack/zaqar/blob/master/zaqar/queues/bootstrap.py19:48
kgriffsah, right19:48
kgriffsnow I remember19:48
kgriffsok, so it loads that meta-driver or "proxy" driver, which proxies requests to one of several true data drivers - basically acting as a router19:49
kgriffsbut to construct that, it calls the self.control property19:49
kgriffsand that is what loads up the control driver by setting control_mode to true19:49
kgriffsok, so instead of using the same config section to load both "storage" and "control" drivers, it should use unique sections, as shown on the etherpad19:50
exploreshaifaliin the etherpad we are spliting the configs, right19:50
kgriffsfurthermore, in bootstrap, it seems like the "storage" property should be renamed to "message_store" and likewise for "control" ==> "management_store" to be consistent with the new config19:51
kgriffsexploreshaifali: does that make sense?19:52
exploreshaifaliwe are spliting configs of what?19:53
kgriffsso, before we had a single config section19:53
kgriffs[drivers:storage:name-of-driver]19:53
kgriffsnow we have two sections19:53
kgriffs[drivers:message_store:name-of-message-store-driver]19:54
exploreshaifaliand each section will describe configs of their own datastores19:54
kgriffs[drivers:management_store:name-of-management-store-driver]19:54
kgriffsexploreshaifali: yeah19:54
exploreshaifalikgriffs, basically these configs describe what?19:55
exploreshaifalidetails to connect with datastore?19:55
kgriffswe could, for example, decide to keep the sqlalchemy management_store driver, but remove the message_store one. But I expect in most cases for each supported backend we will include 2 drivers.19:56
*** anna_ is now known as XX01919:56
kgriffsexploreshaifali: ah, yes19:56
*** XX019 is now known as X01919:56
kgriffshave a look at the example config19:56
kgriffsall of them have a connection string as a URI19:56
kgriffsand then they may have additional driver-specific options19:56
exploreshaifaliso with this we will modify options.py19:57
kgriffsfor example, SSL settings, which database to select, socket timeouts and sleeps, etc.19:57
kgriffsexploreshaifali: yes19:58
kgriffsbut...19:58
kgriffsI think it may be a good idea to actually put the drivers in separate dirs19:58
kgriffswe have queues/storage19:59
exploreshaifalicorrect19:59
kgriffsI was thinking, perhaps we should add19:59
kgriffsqueues/management_store19:59
kgriffsand rename the other to19:59
exploreshaifalithen in store we should have data  dir and one control dir19:59
kgriffsqueues/message_store19:59
exploreshaifaliyes that can be done or I was thinking something like20:00
kgriffsexploreshaifali: oh, that's another way to do it20:00
kgriffsqueues/storage/message20:00
kgriffsqueues/storage/managment20:00
exploreshaifaliexactly :)20:00
exploreshaifaliso I guess the second on would be more better as it is under storage dir?20:01
kgriffsso then management would have it's own driver and options20:01
exploreshaifaliand message will have its own driver and options20:01
kgriffsright20:02
*** malini has left #openstack-zaqar20:02
kgriffsso, first step is to make a patch20:02
riveterwouldn't that involve a lot of duplicated code?20:02
exploreshaifaliriveter, I was also thinking the same20:03
kgriffswhat would we duplicate?20:03
kgriffsoh, I see20:03
exploreshaifalibut before that I more thing need to be clear20:03
kgriffsthere will be overlap in many of the options20:03
exploreshaifaliwhat about rest of the files in storage dir?20:04
exploreshaifalilike base.py, errors.py20:04
kgriffsalthough, there will be some options that only apply to message vs management20:04
exploreshaifalithey will remain in storage dir only?20:04
riveterso would it make more sense to keep the current files & just make subclasses for the drivers?20:04
kgriffsmmm, not exactly20:05
kgriffstake the case of base.py20:05
kgriffsit defines DriverBase20:06
riveterya20:06
kgriffswhich should be inherited20:06
kgriffsbut20:06
*** jchai is now known as jchai_afk20:06
kgriffsit has other stuff in there that is specific to either message or management20:06
kgriffsthose things would be moved down on level in the directory tree20:06
kgriffsand then used in their respective contexts20:07
kgriffswrt options20:07
kgriffshmm20:07
exploreshaifaliyes20:07
* kgriffs thinking20:07
riveterbut there is a fair amount of common code too, no?20:07
kgriffsyeah20:07
exploreshaifalicommon code will remain in storage dir itself20:08
exploreshaifaliand specific code will move to their specific dirs20:08
kgriffswhat we will have to do is identify the common bits, and be able to reuse them at lower (more specific) levels/modules in the dir tree20:08
exploreshaifaliexactly20:08
riveterthis is a substantial patch we're talking about20:09
kgriffsyep20:09
kgriffsthe problem is the two drivers were not properly decoupled to begin with and now we have technical debt to clean up20:09
kgriffsbtw, for the sake of reusing code/options20:10
rivetercan it be broken down into smaller patches?20:10
kgriffsI think the dir will have to look more like20:10
kgriffsqueues/storage/mongodb/management20:10
kgriffsqueues/storage/mongodb/messages20:10
kgriffsso that management and messages can share all things mongo, which would naturally live directly under ueues/storage/mongodb20:11
kgriffsriveter: yes, I think it should be20:11
exploreshaifalikgriffs, yes that could be done20:12
kgriffsthe first patch would be to reorganize the directory structure but not split things up yet.20:12
kgriffs(as far as possible)20:12
kgriffsso, for starters, there would just be a queues/storage/mongodb/messages20:12
kgriffsit would still contain both classes in the same driver file, share options, etc.20:13
kgriffsidk, that may or may not provide a natural breakdown of the work.20:13
kgriffsbut the idea is, let's try to break it down into steps20:13
riveterwho's working on this?20:14
exploreshaifaliriveter, I am looking forward to work on this20:14
exploreshaifali:)20:14
kgriffsexploreshaifali: btw, I thought of something else we will have to figure out20:15
kgriffsbackwards-compat20:15
exploreshaifaliokay20:15
kgriffsi mean, really just from a config standpoint20:15
kgriffsit may be ok just to tell people they need to update their configs before deploying this new version20:15
kgriffsexploreshaifali: you might want to get flaper87's thoughts on it20:16
exploreshaifaliyeah20:16
exploreshaifalikgriffs, so if we talk about config spliting only20:16
exploreshaifalifor which I started discussion20:16
kgriffspersonally, I think it would be OK to ask people to update their configs since it should be pretty easy to do. we could even give them a migration script that helps. but we should ask around20:17
kgriffsexploreshaifali: ok, back to config splitting20:17
exploreshaifalihow can I identify which part is for management and which is for storage20:17
kgriffsmost config options will be identical (just copy them)20:18
kgriffslet me check20:18
exploreshaifalilike in python files, I can clearly identify this is for driver and this code is for data plane20:18
exploreshaifalibut here getting very less20:18
kgriffsok, so for mongo I think both use the same options except for "partitions"20:18
kgriffs"partitions" is only used by the data (AKA message_store) driver20:19
kgriffsis that the nature of your question?20:20
exploreshaifaliyes, a bit of20:20
exploreshaifaliso you mean that full options.py will be same20:20
kgriffsexcept for that one field20:20
exploreshaifaliexcept MONGODB_GROUP = 'drivers:storage:mongodb20:20
kgriffsyeah, that too20:20
kgriffsso under queues/storage/mongodb/options.py20:21
kgriffsyou would have some variable that declares the common options, which will be everything except 'partitions'20:21
exploreshaifalikgriffs, which field will vary other than MONGODB_GROUP = 'drivers:storage:mongodb'20:21
kgriffswell, what I would do is in that "base" options.py20:22
exploreshaifalikgriffs, what exactly you mean by partitions?20:22
kgriffsdon't declare a group of a _config_options20:22
kgriffsexploreshaifali: that is the name of one of the options20:22
kgriffslet me illustrate...20:23
kgriffshttps://etherpad.openstack.org/p/zaqar-scratch20:23
kgriffsat the top there...20:23
kgriffsI will clear the highlight so you can see me type20:23
kgriffsOK, see Line 60?20:23
exploreshaifaliyes20:24
kgriffsok, so I removed that20:24
kgriffsnow at the bottom I get rid of the group and helper function20:25
exploreshaifalikgriffs, hey we have decided to make queue/storage/mongodb/messages20:27
kgriffsoops, fixed20:28
exploreshaifalino need of import also20:28
*** sgotliv has quit IRC20:28
kgriffswhy not?20:28
exploreshaifalioh yes needed20:29
kgriffsI guess what I have there assumes COMMON_MONGODB_OPTIONS is hoisted into the package namespace20:30
kgriffsyou could also be more explicit20:30
kgriffsanyway, that's one way to do it20:31
exploreshaifaliyes :)20:31
exploreshaifalimany things are clear now20:31
exploreshaifalikgriffs, Thanks alot!!20:32
kgriffslooks like redis and sqlalchemy options are all the same (for now), but it might not be a bad idea to follow the same pattern so it will be ready if/when options diverge20:32
kgriffsexploreshaifali: sure thing! hope that helps. :)20:32
exploreshaifaliyes, we should follow same pattern for redis and sqlalchemy too20:33
kgriffscool, sounds like a plan!20:33
riveterkgriffs: do you have time to answer more questions?20:34
exploreshaifaliso first I should submit a patch describing dir structure?20:34
kgriffsriveter: sure20:34
riveteroh sorry20:34
kgriffsexploreshaifali: I would start with a patch that mostly just moves things around, yes.20:34
kgriffsexploreshaifali: btw, this may be a good time to get rid of the queues20:35
kgriffsdir20:35
kgriffsand move everything up one level20:35
kgriffslet's try to catch flaper87 and see what he thinks about that20:35
exploreshaifaliyes20:35
kgriffsriveter: what's up?20:35
riveterI'm looking for a good first patch20:36
exploreshaifalibut I think moving above queue may won't work because queue dir also contain other files and dirs like transport20:36
*** openstackgerrit has quit IRC20:36
riveterI posted some questions on #1351462, which I notice you first reported20:36
kgriffsexploreshaifali: ah, true. that should be in its own patch.20:37
kgriffsexploreshaifali: don't worry about that then for now20:37
exploreshaifaliall right :)20:37
exploreshaifaliThanks!20:37
kgriffsriveter: looking20:37
riveterit's the one about removing duplication from the pool metacontrollers20:38
kgriffsexploreshaifali: thanks for contributing!20:38
exploreshaifaliI am enjoying alot20:38
riveterI'm willing to take a crack at making __getattr__ work but I think the methods involved are different enough that it will be pretty ugly20:39
kgriffsexploreshaifali: rock on20:39
exploreshaifali:d20:39
exploreshaifali:D20:39
kgriffsriveter: yeah, so that may be why it ended up the way it is now20:39
riveterso, is it ok to look at other ways to remove some of the duplication?20:39
kgriffsi just feel like if we challenge ourselves we should be able to improve it at least a little20:40
kgriffsriveter: yeah, I think anything is fair game20:40
riveterand if so, do you have any preference for how?20:40
riveterI was thinking just to add a utility method that takes a return-on-fail value20:41
kgriffsriveter: no real preference, other than the less overhead the proxying causes, the better. A few microseconds can have a significant impact on throughput20:41
riveterhm, that's what I was wondering about20:41
riveterthe cached getattr is nice for that20:41
riveterso a decorator maybe?20:42
kgriffsperhaps20:43
kgriffsI'd say, start simple and then we can optimize if needed20:43
riveterok, fair enough20:43
riveterthanks, this is the friendliest irc channel I've seen in a long while20:43
kgriffsheh, glad to hear it!20:44
riveteroh one more thing - I assume the method signatures can't be changed?20:45
notmynamekgriffs: your mother was a hamster and your father smelt of elderberries!20:51
kgriffsLOOOOOL20:51
riveternotmyname: bonjour20:51
notmynameriveter: ;-)20:51
*** aj9 has joined #openstack-zaqar20:56
*** jchai_afk is now known as jchai20:56
kgriffsriveter: hmm, actually, changing the method signatures may be OK. My main concern would be breaking 3rd-party drivers if the contract changes, but I don't think anyone has created any yet.20:57
riveteroh interesting20:57
riveteryeah, cause some things would be easier if an arg could become a kwarg20:57
kgriffsoic20:57
riveterwell, not easier, but prettier20:57
*** aj9 has left #openstack-zaqar20:59
riveteranyway I have to go, but I work on this this weekend20:59
riveterthanks again for your help20:59
rivetercheers :)20:59
*** cpallares has quit IRC20:59
kgriffssure thing, thanks!21:02
*** sriram1 has quit IRC21:23
*** sgotliv has joined #openstack-zaqar21:34
*** jchai has quit IRC21:40
*** X019 has quit IRC21:46
*** sgotliv has quit IRC21:58
*** X019 has joined #openstack-zaqar21:58
*** mpanetta has quit IRC21:59
*** earnThis has quit IRC22:00
*** amitgandhinz has quit IRC22:15
*** sgotliv has joined #openstack-zaqar22:39
*** sgotliv has quit IRC22:43
*** openstackgerrit has joined #openstack-zaqar23:02
*** sgotliv has joined #openstack-zaqar23:02
exploreshaifalikgriffs, around?23:09
exploreshaifaliI want to know how transport and storage layers communicate with each other23:10
*** sgotliv has quit IRC23:32
*** yfujioka has joined #openstack-zaqar23:37

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!