Thursday, 2015-12-17

*** Qiming has quit IRC00:08
*** pratikmallya has joined #senlin00:42
*** Qiming has joined #senlin00:56
*** zhenguo has joined #senlin01:38
*** Yanyanhu has joined #senlin01:45
*** Yanyanhu has quit IRC02:03
*** elynn_ has joined #senlin02:05
*** elynn_ has quit IRC02:10
*** elynn_ has joined #senlin02:10
*** lixinhui has joined #senlin02:25
elynn_Seems gerrit UI has changed...02:25
Qimingyes, a new version02:25
elynn_Need time to familiar with it...02:26
elynn_Can't find the comment button now...02:28
Qimingnot logged in?02:29
elynn_oh, it's been moved from bottom to top, and the name changed to 'reply'...02:30
openstackgerritlvdongbing proposed openstack/senlin: Fix some nits in profile base  https://review.openstack.org/25876102:31
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem  https://review.openstack.org/25830803:09
*** Yanyanhu has joined #senlin03:21
Yanyanhuthe UI of review page changed for gerrit updating?03:22
Qimingseems so, girl03:23
Yanyanhucan't find those buttons I was familiar with :)03:23
openstackgerritMerged openstack/senlin: Updated from global requirements  https://review.openstack.org/25807403:25
openstackgerritMerged openstack/senlin: Remove senlin.conf.sample  https://review.openstack.org/25821503:31
openstackgerritMerged openstack/senlin: Fix some nits in profile base  https://review.openstack.org/25876103:32
Qimingbuttons are moved, screen is busy, a lot of pop-up windows now03:37
Yanyanhuyes. Honestly, I prefer the old UI. There are too much thing in a small field now.03:38
Qimingagree03:38
Yanyanhumake it difficult to find something. Maybe I will feel better after I'm more familiar with the new UI.03:38
Qiminghope so03:39
elynn_I think the better place for new one is that you can see this patch will affect what other patches.03:40
QimingOne change I'd like to see is the 'Incoming reviews' list should be customizable03:42
Qimingthere are too many patches I reviewed before and now are blocked03:43
Qimingeither there have been not new patch set, or they have been -1'd without response03:43
Yanyanhupatch with -1 but without response should be abandoned a while after the last update date?03:45
Yanyanhue.g. 2 month?03:45
QimingI still have this one in the incoming reviews: https://review.openstack.org/11037903:46
Qiminghttps://review.openstack.org/13530303:46
Qiminghttps://review.openstack.org/13285703:46
Qimingall over a year now03:46
Yanyanhusig...03:46
Yanyanhusigh03:46
YanyanhuI guess heat team should abandon these patches on behalf of proposer03:47
QimingI'm doing that03:47
Yanyanhuthey can restore it later when the work is resumed03:47
Qimingsometimes I have to play a bad guy03:47
Qimingyes03:47
Yanyanhumaybe a mail about this issue in mailinglist?03:48
Yanyanhuto raise some awareness about this problem03:48
*** lixinhui has quit IRC03:51
*** lixinhui has joined #senlin03:53
*** lixinhui has quit IRC04:15
*** pratikmallya has quit IRC04:16
*** elynn_ has quit IRC04:29
*** elynn_ has joined #senlin05:13
*** lixinhui has joined #senlin05:18
*** elynn_ has quit IRC06:17
*** elynn_ has joined #senlin06:17
openstackgerritjunxu proposed openstack/senlin: Fix policy_check for cluster binding scale-in and scale-out policies at sametime. 'policy_check' will check the cooldown of all scale policies for any scale action. This may have some problem, for example as follow.  https://review.openstack.org/25882306:30
*** pratikmallya has joined #senlin06:32
openstackgerritjunxu proposed openstack/senlin: Fix "test_vSphereDRSPolicy.py" file's mode to 644  https://review.openstack.org/25883106:57
openstackgerritYanyan Hu proposed openstack/senlin: Set cluster.next_index to correct value after node join/leave  https://review.openstack.org/25884507:24
Qiminghi07:35
Qiminghelp needed07:35
elynn_Hi07:35
Qimingwhen debugging db sync problems, found another bug07:35
Qimingdon't have bandwidth to fix it07:35
Qiminganybody can help?07:35
elynn_I can have a look.07:36
Qimingreproduce is simple07:36
elynn_What's the problem.07:36
Qimingcreate a node of heat stack07:36
Qimingdelete that stack from heat07:36
Qimingthen you cannot delete the node from senlin07:36
elynn_Qiming: I encounter this problem yesterday07:36
Qimingso please file a bug and fix it07:36
elynn_Then update openstacksdk can solve it.07:36
Yanyanhubut the heat stack was deleted correctly?07:37
Qimingheat stack doesn't exist at all07:37
QimingI wasn't able to delete heat stack07:37
Yanyanhuoh, I see07:37
Qimingit is complaining that some project cannot be deleted07:37
elynn_ignore_missing will be take care in the newest version I guess.07:37
QimingI manually deleted the stacks from heat database, it is a heat bug07:38
elynn_Seems the problem we see is not the same?07:38
elynn_You sad it's a heat problem?07:38
Qimingyes07:39
elynn_You create a stack, and then delete it will fail?07:39
Qimingyes, cannot delete07:39
elynn_in latest codes?07:40
elynn_Let me have a try...07:40
Qimingyes, latest code07:40
Qimingheat side trace: http://paste.openstack.org/show/482157/07:40
Qimingit is trying to delete some projects, could be the stack_user_project07:41
Qimingoh, stack_domain_project07:41
Qimingbut somehow, it is invoking keystoneclient to do this07:41
Qimingkeystoneclient is deprecated ... don't know if it worth a fixing or not07:42
Qiminganyway, that stack cannot be deleted07:42
elynn_it's an endpointnotfound error?...07:42
Qimingthen I manually delete that stack in heat db07:42
Qimingyes07:42
Qimingendpoint for identity service ...07:42
elynn_Should I update my keystone codes/07:43
elynn_?07:43
elynn_or keystoneclient codes...07:43
Qimingdon't know07:43
Yanyanhuhi, Qiming , this error didn't happen in my local env07:43
Qimingit is a long invocation chain, anything can be broken07:43
Yanyanhusenlin's code is latest, but heat is not I think07:44
Qimingpython-keystoneclient 2.0.007:44
elynn_There isn't any change in heat codes related with keystone07:44
elynn_Let me update my heat codes and have a try.07:44
Qimingthen maybe keystoneclient is buggy07:45
Qimingmy keystone is very new: keystone (8.0.0.0b4.dev56, /opt/stack/keystone)07:45
Qimingpip install stalls .... due to firewall07:46
elynn_my keystoneclient version is 1.8.107:46
Yanyanhusigh07:46
Yanyanhumine is 2.0.007:46
elynn_my keystone version is 9.0.0dev29607:47
Yanyanhukeystone version is 8.0.007:47
Qimingokay, my keystone is too old07:47
Yanyanhuso you're using latest heat with old keystone :)07:48
YanyanhuI'm using old version heat and keystone, so error didn't happen I guess07:49
elynn_Can't reproduce your problem.. My local env is fine follow your reproduce steps...07:49
Qimingelynn_, which problem?07:50
elynn_I can delete heat stacks created by senlin07:50
Qimingthe senlin side trace is here: http://paste.openstack.org/show/482156/07:50
elynn_on my local env.07:50
Qimingif you delete that stack from heat07:50
Qimingor you purge_delete it from heat07:51
Qimingthat stack won't exist in Heat anymore07:51
elynn_purge_delete is broken in heat for now....07:51
elynn_there's a patch to fix it.07:51
Qimingthen when senlin tries to do _check_action_complete07:51
Qimingthe first thing it calls is to do a stack_get()07:51
Qimingusing its recorded physical id07:52
elynn_https://review.openstack.org/#/c/255328/07:52
Qimingelynn_, not interested in heat07:52
elynn_Better don't use purge_delete for now.07:52
elynn_I see the same error log in senlin yesterday07:52
elynn_that stack not found error.07:52
Qiminglet's focus for the moment on things we can do better from senlin side, alright?07:53
elynn_then I uninstall openstacksdk and install the latest version of openstacksdk07:53
Qiming20 minutes passed07:54
QimingI haven't explained the problem we need to solve07:54
Qimingsigh ...07:54
Qimingrefer to this trace: http://paste.openstack.org/show/482157/07:54
Qimingoh, no, this one: http://paste.openstack.org/show/482156/07:55
Qimingwhen senlin tries to determine if the delete action is complete07:55
Qimingit will do a stack_get07:55
Qimingwith the previous physical id it remembered (i.e. stack id)07:56
Qimingthis won't cause any problem in heat normally07:56
elynn_I see your point...07:56
Qimingbecause in heat's interface, when you are asking for a stack with a stack id07:56
elynn_The problem is here https://github.com/openstack/senlin/blob/master/senlin/profiles/os/heat/stack.py#L11407:56
Qimingheat gives you the complte record, even if the stack is already deleted07:56
Qimingit is doing show as if you have specified 'show_deleted=True'07:57
Qimings/show/so/07:57
Qiminghowever, this is not reliable07:57
Qimingwe don't know if a stack has disappeared from heat database or not07:57
Qiminglet's suppose 'purge_delete' works today, you completely deleted that stack from heat db07:58
Qimingthe effect is the same07:58
Qimingwhen we do stack_get(), we get a ResourceNotFound Exception07:58
Qimingand when we do stack_get(), there is no way to specify ignore_missing07:59
Qimingwe have to capture and handle this exception07:59
Qimingfull trace is pasted here: http://paste.openstack.org/show/482159/08:00
elynn_I open a bug and I will fix it. https://bugs.launchpad.net/senlin/+bug/152711208:00
openstackLaunchpad bug 1527112 in senlin "Can't delete node if stack doesn't exists in heat db" [Undecided,New]08:00
Qimingthanks08:00
Yanyanhuor maybe we should use stack_find rather than stack_get to ignore notfound exception08:00
Yanyanhucurrently, stack_get will always raise exception is stack with given id does not exist08:01
elynn_That might be a good idea.08:01
Qimingexactly the problem I'm trying to explain08:01
Qimingthe same applies to nova server I think08:02
Yanyanhuyes08:02
Qimingwe got to make our side as robust as possible08:02
elynn_seems we just need to fix the heat profile, don't need to fix nova server profile. It won't happen when using nova.08:10
Qimingoaky08:10
*** Liuqing has joined #senlin08:10
elynn_https://github.com/openstack/senlin/blob/master/senlin/profiles/os/nova/server.py#L346-L35908:10
Qimingline 353 add ignore_missing?08:11
YanyanhuQiming, ignore_missing is true by default in nova driver08:12
Yanyanhusenlin nova driver08:12
Qimingokay, server_delete has that default to True08:12
Yanyanhuyea08:12
Qimingwell ...08:13
elynn_wait_for_server_delete will handle not found.08:13
Qimingthe heat stack impl is different08:13
Qimingthe exception was thrown in check_action_complete()08:13
Qimingplease help make them look similar08:14
elynn_OK, will add wait_for_stack_delete for heat profile.08:14
Yanyanhuhi, elynn_, I think you can try to add this interface into senlin heat driver08:15
elynn_Yanyanhu: I mean that ;)08:16
Yanyanhucurrently, there is no directly support for this interface in sdk08:16
Yanyanhu:)08:16
Yanyanhumaybe in future we can directly rely on sdk to implement it08:16
elynn_Is openstacksdk support wait_for_delete for stack?08:17
elynn_https://github.com/openstack/senlin/blob/master/senlin/drivers/openstack/nova_v2.py#L14208:18
*** lixinhui has quit IRC08:18
Yanyanhuelynn_, no I think08:21
elynn_Let me have a try~08:21
Yanyanhuif possible, maybe we should add it this support in sdk :)08:22
Yanyanhufor heat stack resource08:22
Qiminghttp://git.openstack.org/cgit/openstack/python-openstacksdk/tree/openstack/resource.py08:23
Qimingscroll to the bottom08:23
Qimingthere is wait_for_status and wait_for_delete08:23
Yanyanhuyep, but not sure it works08:23
Yanyanhuah, based on the implementation, it should08:24
Qimingkeystone v9 is so slow ....08:27
elynn_I think qiming is right, it works for me.08:27
Yanyanhucool!08:27
Yanyanhuit can simplify the code of heat stack profile significantly08:28
elynn_I will add unittest later.08:28
openstackgerritEthan Lynn proposed openstack/senlin: Add wait_for_stack_delete to heat driver  https://review.openstack.org/25886608:29
elynn_You can have a try first.08:30
Yanyanhusure, thanks :)08:34
*** pratikmallya has quit IRC08:38
openstackgerritEthan Lynn proposed openstack/senlin: Add wait_for_stack_delete to heat driver  https://review.openstack.org/25886608:46
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem  https://review.openstack.org/25830808:50
elynn_Don't know why that patch doesn't automatically link to launchpad.08:51
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem  https://review.openstack.org/25830808:51
Qimingcould be because that bug is not reviewed?08:53
Yanyanhuwhich one?08:53
elynn_https://review.openstack.org/25886608:54
Qimingbug #152711208:54
openstackbug 1527112 in senlin "Can't delete node if stack doesn't exists in heat db" [High,Triaged] https://launchpad.net/bugs/1527112 - Assigned to Ethan Lynn (ethanlynn)08:54
Yanyanhuthis usually didn't happen08:55
Yanyanhujust occasionally met it08:55
Qimingit happens to elynn several times08:55
*** lixinhui has joined #senlin08:56
Yanyanhuin most times, the link will be built successfully even when bug has not been reviewed I think08:56
*** xujun has joined #senlin08:56
Yanyanhumaybe elynn_ 's account issue LOL08:56
elynn_......08:56
Yanyanhujust kidding :)08:56
Qimingis it that we have to assign a bug?08:56
elynn_Only if the patch is merged, the link will appear to launchpad08:56
Yanyanhuoh, that's possible08:56
Qimingthere must be some tricks08:56
Qiming" Only if the patch is merged, the link will appear to launchpad"08:57
Qimingthat would be bad08:57
*** xujun has joined #senlin08:57
YanyanhuI guess assignee is needed to build the link>08:57
Yanyanhu...08:57
Qimingwe have to make sure that bugs are explictly assigned08:57
Qimingor else, people may jump to the same thing simultaneously08:57
Yanyanhuyes, agree08:58
Qimingit is weird that our local functional tests are all okay, but still fails at gate!!!09:07
Qimingso... tortured09:07
Yanyanhuyes, I suffered from this issue in last two weeks...09:08
Yanyanhuafter that, I tested concurrency issue by creating and deleting multiple big clusters concurrently09:09
Yanyanhurather than running functional test09:09
Yanyanhusince error stopped happening in my local env when running functional test...09:09
YanyanhuI create/delete 5 clusters concurrently with each of them has 200 nodes09:10
Yanyanhuusing test cloud_backend09:10
Qiming======09:10
QimingRan: 13 tests in 359.9469 sec.09:10
Qiming - Passed: 1309:10
Qiming - Skipped: 009:10
Qiming - Expected Fail: 009:10
Qiming - Unexpected Success: 009:10
Qiming - Failed: 009:10
Yanyanhuit just passed..09:11
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem  https://review.openstack.org/25830809:14
elynn_Hi Yanyanhu, how to run functional tests in local env?09:16
elynn_just tox -e functional?09:16
Yanyanhuyes, before that, you have to run senlin-engine and senlin-api09:17
elynn_Do I need to upload any image?09:17
Yanyanhuno, but you need to set the following configure option09:17
Yanyanhucloud_backend = openstack_test09:17
Yanyanhuby default, it is set to 'openstack'09:17
elynn_in senlin.conf?09:18
Yanyanhuyes09:18
elynn_What does it mean?09:18
elynn_What's this option for?09:18
Qimingelynn_, it is a fake layer09:25
Qiminga fake driver layer intercepting all senlin's interactions with other services09:26
Qimingthis is a pure functional test09:26
Qimingif we are using real drivers that allow senlin to talk real services, it would be integration test09:26
Yanyanhuhi, elynn_ , sorry just left for a while09:27
elynn_Wow, so we don't need other services when running functional tests?09:27
Qimingyes09:29
elynn_that's so cool!09:29
QimingYanyanhu is the hero09:30
elynn_InternalError: No Image found for cirros-0.3.2-x86_64-uec09:30
elynn_Seems I still need an older version cirros09:30
Qimingor you should change your profile09:30
Qiming0.3.4 is the default09:31
elynn_The gate still using cirros-0.3.2?09:31
Qimingnot sure, gate is using latest devstack I think09:33
*** pratikmallya has joined #senlin09:39
*** pratikmallya has quit IRC09:43
Yanyanhuhi, elynn_ , the image_get function is also faked in functional test09:45
Yanyanhuall needed fake driver are in senlin/test/functional/drivers directory09:46
Yanyanhuso the image name won't fail the test I think09:46
elynn_Yanyanhu: Then that's weird...09:46
elynn_http://paste.openstack.org/show/482169/09:46
Yanyanhulet me check it09:46
elynn_Oh, I know why09:47
elynn_I forgot to restart senlin-engine...09:47
*** openstackgerrit has quit IRC09:47
elynn_Bye openstackgerrit09:47
Yanyanhu... why it quit09:47
Yanyanhurestarting engine will load the fake driver09:48
*** openstackgerrit has joined #senlin09:48
Qiminglooks like a sport for him09:48
Yanyanhu...09:49
Yanyanhugood exercise :)09:49
openstackgerritLiuqing Jing proposed openstack/senlin-dashboard: Add policy detail page  https://review.openstack.org/25694209:51
*** Liuqing has quit IRC10:12
*** Liuqing has joined #senlin10:13
openstackgerritLiuqing Jing proposed openstack/senlin-dashboard: Use the default `url` tag instead  https://review.openstack.org/25892510:18
*** Qiming has quit IRC10:20
*** Liuqing has quit IRC10:25
*** Yanyanhu has quit IRC10:26
*** elynn_ has quit IRC10:31
*** Liuqing has joined #senlin10:50
*** Qiming has joined #senlin11:08
*** Liuqing has quit IRC11:14
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem  https://review.openstack.org/25830811:49
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem  https://review.openstack.org/25830812:35
*** pratikmallya has joined #senlin12:41
*** pratikmallya has quit IRC12:46
openstackgerritMerged openstack/senlin: Fix "test_vSphereDRSPolicy.py" file's mode to 644  https://review.openstack.org/25883112:56
*** gongysh has quit IRC13:02
*** gongysh has joined #senlin13:02
*** elynn_ has joined #senlin13:14
openstackgerritMerged openstack/senlin-dashboard: Add policy detail page  https://review.openstack.org/25694213:21
openstackgerritMerged openstack/senlin-dashboard: Use the default `url` tag instead  https://review.openstack.org/25892513:21
*** elynn_ has quit IRC14:20
openstackgerritQiming Teng proposed openstack/senlin: Fix action dependency problem  https://review.openstack.org/25830814:29
*** pratikmallya has joined #senlin15:12
*** pratikmallya has quit IRC15:50
*** xujun has quit IRC16:08
*** xujun has joined #senlin16:08
*** Qiming has quit IRC16:14
*** pratikmallya has joined #senlin16:26
*** pratikma_ has joined #senlin16:27
*** pratikmallya has quit IRC16:30
-openstackstatus- NOTICE: Zuul is moving in very slow motion since roughly 13:30 UTC; the Infra team is investigating.16:43
*** bdrich has joined #senlin17:45
*** bdrich has left #senlin17:58
*** bdrich has joined #senlin17:59
*** bdrich has quit IRC18:09
*** bdrich has joined #senlin18:09
*** bdrich has quit IRC18:34
jdandreaShould there be an extra os.pardir here per the comment on line 25? /blob/master/bin/senlin-api#L2918:50
jdandreaWhoops - https://github.com/openstack/senlin/blob/master/bin/senlin-api#L2918:51
*** pratikma_ is now known as pratikmallya20:35
*** bdrich has joined #senlin20:39
*** bdrich has quit IRC21:45
*** yuanying has joined #senlin23:12
*** Qiming has joined #senlin23:46
*** pratikmallya has quit IRC23:51

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!