| *** heyongli has quit IRC | 00:02 | |
| *** heyongli has joined #senlin | 00:04 | |
| *** shu-mutou-AFK is now known as shu-mutou | 00:05 | |
| *** Qiming has quit IRC | 00:41 | |
| *** Qiming has joined #senlin | 01:38 | |
| -openstackstatus- NOTICE: Gerrit is about to be restarted to help with page timeouts | 01:47 | |
| *** elynn has joined #senlin | 01:53 | |
| *** elynn_ has joined #senlin | 01:58 | |
| *** elynn has quit IRC | 01:58 | |
| elynn_ | Morning | 02:01 |
|---|---|---|
| openstackgerrit | Merged openstack/python-senlinclient: Updated from global requirements https://review.openstack.org/317509 | 02:04 |
| *** zzxwill has joined #senlin | 02:05 | |
| Qiming | hi | 02:14 |
| Qiming | question | 02:14 |
| Qiming | is image cirros-0.3.2-x86_64-uec available on gate? | 02:14 |
| Qiming | sorry cirros-0.3.4-x86_64-uec | 02:14 |
| elynn_ | devstack will install it through internet I think. | 02:15 |
| *** yuanying has joined #senlin | 02:17 | |
| Qiming | I'm asking because our functional test is using 0.3.2 | 02:23 |
| Qiming | my local env doesn't have 0.3.2 | 02:23 |
| elynn_ | Heat use 0.3.4 on gate for integrationtests | 02:25 |
| elynn_ | https://github.com/openstack/heat/blob/master/heat_integrationtests/prepare_test_env.sh#L38 | 02:25 |
| elynn_ | So 0.3.4 should available on gate. | 02:25 |
| Qiming | ok | 02:25 |
| Qiming | thx | 02:25 |
| elynn_ | Is it work just change 0.3.2 to 0.3.4 on your local env? | 02:25 |
| openstackgerrit | Qiming Teng proposed openstack/senlin: Bump cirros version to 0.3.4 https://review.openstack.org/317805 | 02:28 |
| Qiming | seems I have some problem setting cloud_backend | 02:58 |
| Qiming | switching to openstack_test solved the problem | 02:58 |
| Qiming | but anyway, bumping cirros to 0.3.4 doesn't hurt | 02:58 |
| elynn_ | Do you want to enable gate for real driver? | 02:59 |
| Qiming | not yet there | 03:03 |
| Qiming | when tempest is fully enabled, and we are working on functional tests using tempest, we can switch to use real drivers | 03:03 |
| Qiming | makes sense? | 03:03 |
| elynn_ | yes, some codes might need to be changed for real driver | 03:04 |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Add create_test_policy function to base test class https://review.openstack.org/317823 | 04:09 |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Add API test for cluster policies list/show https://review.openstack.org/317824 | 04:09 |
| *** elynn_ has quit IRC | 04:15 | |
| *** yanyanhu has joined #senlin | 04:28 | |
| *** elynn_ has joined #senlin | 05:35 | |
| *** elynn_ has quit IRC | 05:39 | |
| *** elynn_ has joined #senlin | 05:40 | |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Add API tests for action list/show https://review.openstack.org/317852 | 06:24 |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Add tempest tests for build-info API https://review.openstack.org/317856 | 06:37 |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Add tempest tests for build-info API https://review.openstack.org/317856 | 06:37 |
| openstackgerrit | xu-haiwei proposed openstack/senlin: A spec for container cluster service https://review.openstack.org/281102 | 06:41 |
| *** zzxwill has quit IRC | 07:05 | |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Add tempest tests for events list API https://review.openstack.org/317864 | 07:07 |
| openstackgerrit | Qiming Teng proposed openstack/senlin: Tune DB API for session sync https://review.openstack.org/317876 | 07:23 |
| Qiming | elynn_, there? | 07:57 |
| elynn_ | yes | 07:57 |
| Qiming | I'm trying to debug the db lock problem | 07:58 |
| Qiming | what I found | 07:58 |
| Qiming | (maybe irrelevant to the problem) | 07:58 |
| Qiming | is that the service table contains two records | 07:58 |
| Qiming | even if I'm runing just one engine | 07:58 |
| elynn_ | Each worker will have one record | 07:59 |
| Qiming | should be one worker | 08:00 |
| elynn_ | You set num_engine_workers=1 in your senlin.conf already? | 08:01 |
| Qiming | no | 08:01 |
| Qiming | seems the other dead engine always hang there in the db | 08:02 |
| elynn_ | ps -ef|grep senlin-engine | 08:03 |
| elynn_ | how many do you get? | 08:03 |
| elynn_ | then use command 'senlin-manage service list' to check service status | 08:03 |
| Qiming | one | 08:03 |
| elynn_ | I think the dead one's status is 'down' | 08:04 |
| Qiming | senlin-manage service list produces two records | 08:04 |
| Qiming | yep, the status is down | 08:04 |
| Qiming | one of them | 08:04 |
| elynn_ | That is correct, When we check whether a engine is alive or not in senlin-engine, it depends on the 'status' | 08:05 |
| Qiming | I see | 08:06 |
| elynn_ | if the engine didn't update it's record for 2*period_time, then will consider it as dead. | 08:06 |
| elynn_ | https://github.com/openstack/senlin/blob/master/senlin/engine/senlin_lock.py#L42 | 08:06 |
| Qiming | weird thing is | 08:06 |
| Qiming | single functional test most of the time passes | 08:07 |
| Qiming | multiple tests, however, will mostly fail | 08:07 |
| Qiming | leaving only one or two passes | 08:07 |
| elynn_ | You are using real drivers? | 08:08 |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Fix functional tests https://review.openstack.org/317894 | 08:20 |
| Qiming | no | 08:20 |
| elynn_ | That's strange, since gate is okay. | 08:22 |
| elynn_ | any errors in log? | 08:22 |
| Qiming | dbdeadlock | 08:22 |
| Qiming | there is no easy way to reproduce this | 08:22 |
| Qiming | have to change some code to sanitize the logs | 08:23 |
| Qiming | currently the logs are all about messages like " ... sleep for 3 seconds" | 08:23 |
| elynn_ | That sounds not good... | 08:24 |
| elynn_ | Enable multi engine can solve it or not? | 08:24 |
| Qiming | trying this: tox -e functional -- -r senlin.tests.functional.test_cluster_scaling -r senlin.tests.functional.test_cluster_basic | 08:24 |
| elynn_ | let me try. | 08:25 |
| Qiming | ... the first '-r' was ignored | 08:25 |
| elynn_ | hmm, it indeed ignore the first one... | 08:28 |
| elynn_ | seems ostestr didn't support two '-r' | 08:30 |
| elynn_ | try this tox -e functional -- -r '(senlin.tests.functional.test_cluster_scaling|senlin.tests.functional.test_cluster_basic)' | 08:31 |
| Qiming | ok, it works | 08:32 |
| elynn_ | messages like " ... sleep for 3 seconds" might because some locks in db can't be stole...Not sure why if I can't access the env. | 08:34 |
| elynn_ | Try to stop all senlin-engine and clean lock table in db, and run the tests again. | 08:34 |
| Qiming | sleep was there because some actions are waiting | 08:35 |
| Qiming | it is not a bug actually | 08:36 |
| Qiming | I'm looking into this: "DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') " | 08:36 |
| Qiming | maybe we need some oslo_db retry | 08:36 |
| elynn_ | Hmm, yes, maybe we can add deadlock retry decorator. | 08:37 |
| elynn_ | functional tests at my env works fine... | 08:38 |
| Qiming | something changed from oslo recently | 08:38 |
| Qiming | but in my env, no | 08:38 |
| Qiming | if fails occasionally | 08:38 |
| elynn_ | Wrap with retry and have a test? https://github.com/openstack/heat/blob/c9aff2fc7364ed78b31238b7c3bc035426b1788e/heat/db/sqlalchemy/api.py#L534-L535 | 08:40 |
| openstackgerrit | Ethan Lynn proposed openstack/senlin: Fix functional tests https://review.openstack.org/317894 | 08:51 |
| Qiming | okay, working on it | 08:54 |
| elynn_ | Hmm, indeed, deadlock happens occasionally in my env. | 08:55 |
| Qiming | okay, applied two wrappers | 09:04 |
| Qiming | testing | 09:04 |
| openstackgerrit | Merged openstack/senlin: Fix functional tests https://review.openstack.org/317894 | 09:42 |
| openstackgerrit | Merged openstack/senlin: Bump cirros version to 0.3.4 https://review.openstack.org/317805 | 09:50 |
| *** Qiming has quit IRC | 10:08 | |
| *** shu-mutou is now known as shu-mutou-AFK | 10:17 | |
| *** yanyanhu has quit IRC | 10:23 | |
| *** elynn_ has quit IRC | 11:17 | |
| *** Qiming has joined #senlin | 11:18 | |
| *** jdandrea_ has quit IRC | 12:22 | |
| *** jdandrea has joined #senlin | 12:22 | |
| openstackgerrit | Sean Dague proposed openstack/senlin: Revert "Allow parameter location to be specified in rst" https://review.openstack.org/318023 | 12:34 |
| openstackgerrit | Sean Dague proposed openstack/senlin: Migrate to os-api-ref upstream library https://review.openstack.org/318026 | 12:37 |
| openstackgerrit | Qiming Teng proposed openstack/senlin: Add DB retry decorator for DeadLock problem https://review.openstack.org/318115 | 14:01 |
| *** Liuqing has joined #senlin | 14:04 | |
| openstackgerrit | Sean Dague proposed openstack/senlin: Migrate to os-api-ref upstream library https://review.openstack.org/318026 | 14:10 |
| openstackgerrit | Merged openstack/senlin: Tune DB API for session sync https://review.openstack.org/317876 | 14:21 |
| *** Liuqing has quit IRC | 14:33 | |
| *** Drago has joined #senlin | 14:51 | |
| *** Drago has quit IRC | 14:51 | |
| *** Drago has joined #senlin | 14:51 | |
| *** zzxwill has joined #senlin | 15:05 | |
| openstackgerrit | Qiming Teng proposed openstack/senlin: Migrate to os-api-ref upstream library https://review.openstack.org/318026 | 15:12 |
| *** Qiming has quit IRC | 15:17 | |
| *** Qiming has joined #senlin | 15:17 | |
| openstackgerrit | Merged openstack/senlin: Updated from global requirements https://review.openstack.org/317512 | 16:16 |
| *** Qiming has quit IRC | 16:31 | |
| *** Drago has quit IRC | 16:40 | |
| *** Drago has joined #senlin | 17:40 | |
| *** Drago1 has joined #senlin | 20:07 | |
| *** Drago has quit IRC | 20:09 | |
| *** sridhar_ram has joined #senlin | 21:55 | |
| *** Drago1 has quit IRC | 23:29 | |
| *** Qiming has joined #senlin | 23:32 | |
| *** sridhar_ram1 has joined #senlin | 23:33 | |
| *** sridhar_ram has quit IRC | 23:33 | |
| openstackgerrit | Qiming Teng proposed openstack/senlin: Migrate to os-api-ref upstream library https://review.openstack.org/318026 | 23:41 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!