Tuesday, 2015-09-01

*** lkarm has joined #senlin00:05
*** lkarm has quit IRC00:09
*** Qiming has quit IRC00:13
*** jruano has joined #senlin00:19
*** elynn_ has quit IRC00:26
*** elynn_ has joined #senlin00:26
*** lawrancejing has joined #senlin00:29
*** jdandrea has quit IRC01:24
*** Qiming has joined #senlin01:27
*** elynn__ has joined #senlin01:32
*** elynn_ has quit IRC01:35
*** mathspanda has joined #senlin01:54
openstackgerritQiming Teng proposed stackforge/senlin: Make profile schema versioned  https://review.openstack.org/21888801:55
*** Yanyanhu has joined #senlin01:55
*** Yanyan has joined #senlin01:56
*** jruano has quit IRC01:58
openstackgerritQiming Teng proposed stackforge/senlin: Sample profiles with type and version keys  https://review.openstack.org/21909201:59
*** Yanyanhu has quit IRC01:59
*** branw has joined #senlin02:07
Yanyanhi, Qiming, are you around02:08
Qimingy02:08
Yanyanjust a question want to confirm about the patch of versioned profile02:09
Yanyanso the profile type_name stored in DB will always be in format like type_name-verions02:10
Qimingyes02:10
Yanyanok, understand02:10
Qimingit will differentiate different versions of types easily02:10
Yanyansince I found in line 212, type keyword argument is always not given02:10
Yanyanok02:10
Yanyanhttps://review.openstack.org/#/c/218888/2/senlin/engine/service.py02:10
Qimingthe key of this revision is to remove 'type' from parameters, 'type' should be part of the spec02:12
Yanyanyes, this is much better than old design02:12
Qimingthen when you do profile creation, you will do 'senlin profile-create -s specfile name' rather than 'senlin profile-create -t os.heat.stack -s specfile name'02:13
Qimingthe '-t <type>' doesn't make good sense02:13
xuhaiweiI am still a little confused,  in profile_create.json the 'type' should not contain version information?02:14
Yanyanyep02:14
xuhaiweilike os.heat.stack-1.002:14
Qimingthat will introduce type string parsing problem02:15
Yanyanand not beautiful I think :)02:15
Qiminghow do we impose format specifications to the type string?02:15
xuhaiweibut in this file https://review.openstack.org/#/c/218888/2/senlin/engine/service.py line 251, it seems the version is desired in the type02:16
xuhaiwei type_name = db_profile.type.split('-')[0]02:17
QimingI did think about that alternative, however, I didn't convince myself it is better than separated version02:17
Qimingyes, you will see how much trouble we will be handling there02:18
Qiminga better way is to reparse the spec to get the type_name and version02:18
Qimingif you think that is necessary, I can propose another patch02:19
xuhaiweiI also think we should use the type style like 'os.heat.stack' instead of 'os.heat.stack-1.0', but  type_name = db_profile.type.split('-')[0] should be fixed I think02:20
Qimingokay, will do02:20
xuhaiweianother problem is in profile_create_resp.json, we are using os.heat.stack-1.002:21
Qimingyes02:22
xuhaiweithis should be fixed too?02:23
Qimingit is part of the response, read-only02:23
Qimingwhy fix it is necessary?02:23
xuhaiweioh02:23
Qimingwhen we are showing something to users, 'typename-version' is more condensed format, right?02:23
Qimingif it is confusing, we can delete it from DB02:24
xuhaiweiok, agree02:24
xuhaiweijust leave it02:24
openstackgerritQiming Teng proposed stackforge/senlin: Make profile schema versioned  https://review.openstack.org/21888802:25
xuhaiweiQiming, Yanyan, the fix in this patch  https://review.openstack.org/218796, I got this error when creating a node  ERROR: Unknown exception from SDK: Authentication cannot be scoped to multiple targets. Pick one of: project, domain or trust02:26
xuhaiweiboth sdk and senlin are the latest02:27
Qimingbut your DB is not up to date02:27
xuhaiweiI should recreate db?02:27
*** jruano has joined #senlin02:27
Qimingthe profile records in your db contains bad data02:28
Yanyanem, I think the context property store in DB is old one02:28
xuhaiweiI should recreate profile02:28
Qimingbefore this patch, 'project_name' and a lot of other data are stored into the 'context' field02:28
Yanyanrecreate profile should be able to solve the issue02:28
Qimingright02:28
Qimingthere could be other holes when talking about the context02:30
Qimingstill working on fixing them02:30
*** jruano has quit IRC02:33
*** jruano has joined #senlin02:39
*** lkarm has joined #senlin02:41
*** lkarm has quit IRC02:45
openstackgerritMerged stackforge/senlin: Make profile schema versioned  https://review.openstack.org/21888803:07
openstackgerritxu-haiwei proposed stackforge/senlin: Fix trust-based connection building in policy  https://review.openstack.org/21910603:13
*** jruano has quit IRC03:19
openstackgerritQiming Teng proposed stackforge/python-senlinclient: Revise support to profile-create/show  https://review.openstack.org/21911003:22
openstackgerritZhenguo Niu proposed stackforge/senlin-dashboard: Add profiles table  https://review.openstack.org/21911303:44
openstackgerritOpenStack Proposal Bot proposed stackforge/senlin-dashboard: Updated from global requirements  https://review.openstack.org/21911804:29
*** lawrancejing has quit IRC04:33
*** yonglihe has joined #senlin04:41
openstackgerritQiming Teng proposed stackforge/python-senlinclient: Revise support to profile-create/show  https://review.openstack.org/21911005:38
*** lawrancejing has joined #senlin06:00
openstackgerritQiming Teng proposed stackforge/senlin: Revise doc to reflect latest changes  https://review.openstack.org/21914706:04
Qimingadmin__, there?06:06
admin__Yes06:09
Qiminghi, admin__06:09
Qimingwondering if there are some placement policy prototypes to work on?06:09
admin__sure06:10
admin__VSphereDRS06:10
Qimingokay06:11
admin__?06:11
admin__just let me know youre requirments06:12
Qiminghttps://etherpad.openstack.org/p/senlin-liberty-workitems06:25
Qimingadmin__, I am trying to understand what we can land in the coming days06:25
admin__okay, we are working across AZ06:29
admin__will ping you when submit vsphereDRSplacmentPolicy for your rereview06:30
Qimingif the reworking of the heat patch has not started, I can take that over06:31
Qimingif you guys have started, I can shift to something else and jump onto this when there is a patch06:32
admin__please shift to something else06:37
admin__we are working on this and will catch up the progress06:38
xuhaiweiQiming, will you start the test of client?06:47
Qimingokay, xuhaiwei06:47
xuhaiweiI assigned utils to myself, but it's a little difficult to me I think06:48
Qimingright, there are many intricate mockings to be figured out06:49
Qiminganother issue related to function is node_update and cluster_update06:51
Qimingthey are not implemented yet06:51
openstackgerritQiming Teng proposed stackforge/senlin: Fix functional test for policy type listing  https://review.openstack.org/21915906:54
*** lkarm has joined #senlin06:57
Yanyanhi, Qiming, free now? I have a question about current design of some DB APIs06:57
Qimingokay06:58
YanyanI found we never use session.close() in senlin DB sqlalchemy api06:58
Yanyanwe always use session.commit()06:58
YanyanIs there any special reason for this implementation since I guess in some cases, maybe we should use session.close()06:59
Qimingcommit is kind of a close() ?06:59
Yanyanem, commit will save the change but the session is kept if the obj is not gc by python06:59
QimingNever saw any hints about the transaction problems about commit()07:00
Qimingokay, so you are not talking about the transaction problems07:00
Yanyanyes07:01
Qimingif the question is about whether session is being GC'ed, you can do a close07:01
YanyanI guess some sessions may not be closed correctly07:01
Yanyanumm, still not very sure about it07:01
*** lkarm has quit IRC07:01
Qimingdon't think so07:01
Yanyanbut will keep look at it07:02
Qimingso the reference chain is this: action->context->session07:02
Qimingwhen an action object is not used, python will gc the whole reference chain07:02
Yanyanyes07:02
Yanyanthis is what we expected07:03
Qimingunless the action object in memory is not released, e.g. still referenced07:03
Yanyanyes07:03
Qiminghowever, even in that case, the action is in a zombie status07:03
Qimingit is just a memory usage problem07:04
Qimingit won't cause any concurrency problem07:04
Yanyanbut I found use fresh session, module_query of some objects can get correct result07:04
Yanyanbut if use old session, it can't07:05
Qimingthere caches in memory07:05
Qimingwhy are you using old session if it is supposed to be abandoned?07:05
Yanyanthe old session is taken by context07:06
Yanyanso if we use context to do some db operations, e.g. get, the old session will take effect07:06
Qimingwhy we are using old context?07:07
Yanyanwe always specify context obj for DB operation07:08
Yanyanlet me find an example07:08
Yanyanhttp://git.openstack.org/cgit/stackforge/senlin/tree/senlin/engine/actions/base.py#n32407:08
Qimingyes07:09
Yanyane.g. the self.context of action will be used to query an action objet from DB07:09
Qimingwhere is it called?07:09
Yanyanin wait_for_dependents function http://git.openstack.org/cgit/stackforge/senlin/tree/senlin/engine/actions/cluster_action.py#n5907:11
Yanyanhere07:11
Qimingdon't see a problem there07:12
Yanyanyes, this part is ok. What I'm wondering is whether we should close session in some places like this: http://git.openstack.org/cgit/stackforge/senlin/tree/senlin/db/sqlalchemy/api.py#n123707:12
Qimingit is a call for the executing action to check its own status07:12
Yanyanor http://git.openstack.org/cgit/stackforge/senlin/tree/senlin/db/sqlalchemy/api.py#n130107:13
Qimingwe discussed this ...07:13
Yanyanah...07:14
Qimingcommit() is good enough to make sure data are persisted into db, with consistency ensured07:14
Qimingcalling close() will not change that fact07:14
Yanyanyes, that's true. The result is stored in DB correctly07:14
Qimingit won't solve concurrency problems07:14
Yanyanbut the get API can't fetch the latest result in DB...07:14
Yanyanunless we use a fresh session to perform get operation07:14
Qimingthat doesn't make sense07:15
Yanyanyes07:15
Qimingit is against computer science07:15
Yanyan...07:15
Qimingthis is science, not art07:15
Yanyanadded log in sqlalchemy API function, and found this issue...07:15
Yanyanthe same function, will different session, the return value is different07:16
Qimingthe same session wrote some data into db and did a commit07:16
Yanyanand the DB entry has been changed and contains correctly value07:16
Qimingthe same session cannot see the result07:16
Qimingthe new data can be seen only from a new session?07:16
Qimingthat is rediculous07:16
YanyanI'm also confused about this issue07:17
Yanyanmaybe I can show you the result?07:17
Qimingwhat I do expect from session.close() is that we can save some unused sessions, make them gc'ed as early as possible07:18
Qimingthere have been some problems previously, regarding the number of mysql session number07:18
Yanyanif the session was closed after DB writing, this problem didn't happen07:18
QimingI believe you are looking into a wrong solution07:19
Yanyanyes, understand this limitation07:19
Yanyanactually I haven't found a solution for this issue yet ... Just want to understand why it happen.07:20
Qimingalright07:21
YanyanI'm now doubting the session close could be the reason, but not sure about it07:21
Qiminglet's take a step back07:21
Qimingdon't hurry on a "quick" fix07:21
Yanyansure07:21
Qimingwe talked about the "badness" of contexts ...07:22
YanyanI'm try to read some doc about sesssion obj in python sqlalchmey lib07:22
Yanyanbut didn't find a very good document07:22
Yanyanif possible, I think we should use somthing like 'session' obj to access DB07:23
Qimingokay, I don't believe there is concurrency problems in the sqlalchemy lib, maybe that is true07:23
Yanyanwe actually don't need entire context, we just need session and some user info like project_id07:23
Qimingbut at least it is used by so many projects, I trust its maturity in this field07:23
Qimingyes07:24
Yanyanyes, I think the lib is ok07:24
Qimingthat is what I suggest ... take a step back07:24
Qimingwe are not seeing this kind of problems besides the action module07:24
QimingI want to know if we are incorrectly (unnecessarily) deserializing an Action object somewhere07:25
Yanyansince the deserializing of action obj will create a new session?07:26
Qimingwhen the constructor is invoked, a new context object, thus a new db session, is created.07:26
Yanyaninside a new context07:26
Qimingthat will create serious problems, not matter you "close" session of not07:26
Qimings/of/or/07:26
Yanyanbut since currently only list operation could rely on session to do some special operations07:27
Qimingthen we examing action code, whether an action, during its execution, is using some other contexts07:27
YanyanI guess it's ok for current design?07:27
Qimingfor example, oslo_context.get_current()07:27
Yanyanunderstand what you mean07:28
Yanyanactually just looked through the code of cluster_action and I think the usage of context is correct07:28
Yanyanwill further check node_action module07:28
Qimingeach action object should strictly work using its own context once initialized07:28
Yanyanyes07:28
Qimingan action may invoke profile code?07:29
Yanyanyes07:29
Qimingand we are creating new instances of RequestContext there?07:29
Qimingis there any missing pieces ?07:30
Qimingoslo_context.get_current() is dangerous07:30
Yanyanhmm, I think we have removed context usage in profile layer except those get_service_context07:30
Qimingif you want to ensure concurrency is solved, you will never create a new RequestContext in your current "thread"07:31
Qimingfaint07:31
Qimingget_sevice_context is instantiating a RequestContext instance?07:31
Yanyannope07:31
Yanyanoslo_context.get_current won't create RequestContext of senlin service I think07:32
Yanyanjust oslo's RequestContext07:32
Qiming...07:32
Qimingyou need some lectures on object-oriented design07:33
Qiming:)07:33
Yanyanah07:33
Yanyanhmm, will read the code about oslo_context.context07:34
Qimingyou are never creating a standalone oslo_context.RequestContext object07:34
Qimingall oslo_context.RequestContext objects you created are actually instances of the senlin's RequestContext subclass07:35
Yanyanoh07:36
Qimingin oslo_context code, they can treat these objects as oslo_context.RequestContext, but the nature of those objects won't change07:36
Qimingin parent class you treat an object as Shape07:36
Qimingin child classes you treat them as Rectangle or Triangle07:36
Qimingbut there is only one object07:37
QimingI may be wrong, if that is the case, I have to say "hello, Python, you idiot!"07:37
Yanyanhmm, ok07:38
Yanyanlet me go through the code again and make a test about it07:38
Qimingyes, what really matters is we need to be confident that the code structure is 100% correct07:44
Qimingwe cannot rely test cases to tell me that07:45
Yanyanyes, definitely07:45
Qimingthis is a great opportunity to revisit every single line of the action code07:45
YanyanYes, that's right :)07:46
Yanyanand also the DB session07:46
Yanyanat least can help me understand how it work07:47
Qimingregarding the context problem07:47
Qimingyou are right, we don't need a traditional "context" at the action layer07:47
Yanyanyes, maybe in some future patches, we should replace it with one or two new objects07:49
Qimingyep07:49
Qimingthere are several things we need at the action layer:07:50
Qiming db session, user, project, domain07:50
Yanyanyes07:51
Yanyanoh, understand why the object returned by get_current is a Senlin RequestContext, didn't notice this line http://git.openstack.org/cgit/openstack/oslo.context/tree/oslo_context/context.py#n6807:52
Qimingthere could be problems though07:53
Yanyanand the super() invoking in RequestContext.__init__ in Senlin service07:53
Qiminghttp://git.openstack.org/cgit/openstack/oslo.context/tree/oslo_context/context.py#n2807:54
Qimingthis context object reference is stored in threading.local(), which is a TLS (thread-local storage)07:54
Yanyanso basically, we only have one current context in all greenthreads?07:54
QimingI'm not 100% sure it would be eventlet-safe, as it was designed to be "thread-safe"07:55
Yanyanem, seems not eventlet safe07:55
Qimingthat is possible07:55
Yanyanthis is really dangerous...07:56
Qimingsometimes, it doesn't really matter, sometimes, it does07:57
Yanyanto rely on oslo_context.get_current to get correct context for each greenthread07:57
Qimingthis is something we HAVE TO figure out07:57
Qimingwhen we are sure we have been doing things right, we test if the code behaves as expected07:58
Qimingno the reverse direction07:58
Yanyanok, let me check the usage of context in the entire workflow to see whether we find some clues08:00
*** mathspanda has quit IRC08:08
*** lkarm has joined #senlin09:13
*** lkarm has quit IRC09:18
*** lawrancejing has quit IRC09:49
openstackgerritCindia-blue proposed stackforge/senlin: Add placement policy for vSphere users, which can help control affinity and unti-affinity freely of different nodes/node groups  https://review.openstack.org/21921210:07
Qimingadmin__, hi10:08
admin__hi10:10
Qimingthx for the patch!10:11
Qiminga first glance at it, two problems10:11
admin__please10:12
Qimingpost_op is not needed10:12
admin__let me know the pobblems10:12
admin__okay10:12
Qiminga policy only guides the engine to do the work, it doesn't do the dirty work, :)10:12
Qiminganother nit is about commit message10:13
Qimingthe convention is to have a subject line and one (or more) paragraphs of messages10:13
Qimingthe subject line should be no more than 60 characters10:14
Qimingsubject line is a text with no teminating period ('.')10:14
Qimingthe message paragraph is separated from the subject line with an empty line10:15
admin__okay10:15
Qimingeach line in the message paragraph is no longer than 80 chars10:15
Qimingjust some weird tradition10:15
Qimingthere is no "why" here , :)10:15
Qimingwe just ensure everyone's code looks like being written by another person10:16
Qiming:)10:16
admin__:) I will follow10:16
Qimingcool, will jump into the details later10:17
admin__okay10:17
Qiminga good starting point10:17
openstackgerritCindia-blue proposed stackforge/senlin: Add a placement-policy to enable DRS functions for vSphere users  https://review.openstack.org/21921210:34
openstackgerritCindia-blue proposed stackforge/senlin: Add a placement-policy to enable vSphere DRS functions  https://review.openstack.org/21921210:53
*** Yanyan has quit IRC10:55
openstackgerritQiming Teng proposed stackforge/python-senlinclient: Shell unit test (1)  https://review.openstack.org/21923511:05
*** Qiming has quit IRC11:11
*** jruano has joined #senlin11:26
openstackgerritMerged stackforge/senlin: Fix trust-based connection building in policy  https://review.openstack.org/21910611:42
openstackgerritMerged stackforge/senlin-dashboard: Add profiles table  https://review.openstack.org/21911312:00
*** Qiming has joined #senlin12:04
*** lkarm has joined #senlin12:07
openstackgerritQiming Teng proposed stackforge/python-senlinclient: Shell unit test (1)  https://review.openstack.org/21923512:37
*** yanyanhu has joined #senlin12:53
*** xuhaiwei_ has joined #senlin12:57
Qimingmeeting bot is not working?13:01
Qimingcanot see a thing there13:02
yanyanhuit works13:02
xuhaiwei_I saw it works13:02
yanyanhuI can see the topic change in the meeting channel13:02
*** LiuWei has quit IRC13:10
Qimingmeeting on #openstack-meeting13:29
Qimingfeel free to drop in13:29
*** Qiming has quit IRC13:30
*** Qiming has joined #senlin13:31
*** jdandrea has joined #senlin13:55
Qimingthanks all for attending the meeting13:55
*** jruano has quit IRC13:55
yanyanhu:)13:57
yanyanhu will leave, see U guys tomorrow13:58
*** yanyanhu has quit IRC13:58
Qimingsee you13:59
*** xuhaiwei_ has quit IRC14:01
*** jroyal has joined #senlin14:45
*** jroyal has quit IRC15:03
*** lkarm has quit IRC15:22
*** lkarm has joined #senlin15:22
*** Qiming has quit IRC15:23
*** zhenguo has quit IRC16:00
*** lkarm has quit IRC16:10
*** lkarm has joined #senlin16:11
*** lkarm has quit IRC16:11
*** lkarm has joined #senlin16:11
*** elynn__ has quit IRC16:29
*** jruano has joined #senlin16:39
*** lkarm_ has joined #senlin17:10
*** lkarm has quit IRC17:13
*** tiantian has joined #senlin18:09
*** huangtianhua has quit IRC18:12
*** jruano has quit IRC18:30
openstackgerritOpenStack Proposal Bot proposed stackforge/senlin-dashboard: Updated from global requirements  https://review.openstack.org/21911818:56
*** lkarm_ has quit IRC21:45
*** lkarm has joined #senlin21:46
*** lkarm has quit IRC21:51
*** Qiming has joined #senlin23:37
*** Qiming has quit IRC23:37
*** xuhaiwei has quit IRC23:38
*** Qiming has joined #senlin23:38

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!