| openstackgerrit | XueFeng Liu proposed openstack/senlin: Fix node_lock can't be stole https://review.openstack.org/394407 | 00:27 |
|---|---|---|
| *** zhurong has joined #senlin | 01:30 | |
| *** yanyanhu has joined #senlin | 01:34 | |
| Qiming | yanyanhu, there? | 02:14 |
| yanyanhu | yes | 02:14 |
| Qiming | about this bug report: https://bugs.launchpad.net/senlin/+bug/1639122 | 02:14 |
| openstack | Launchpad bug 1639122 in senlin "senlin-cluster-deadlock" [Medium,Confirmed] - Assigned to XueFeng Liu (jonnary-liu) | 02:14 |
| *** elynn has joined #senlin | 02:15 | |
| Qiming | the description is a mess, cannot read it through | 02:15 |
| Qiming | it is wordy, and not touching the essence of the problem | 02:15 |
| Qiming | when I quickly go thru the report, it seems the so called "dead lock" bugs are all caused by engine restart | 02:15 |
| Qiming | that is something we were not designed for | 02:16 |
| yanyanhu | Qiming, yes, we didn't consider the case engine restarts before | 02:16 |
| Qiming | I have seen some patching coming in in attempt to solve the problem | 02:16 |
| Qiming | but those patches are solving the problem in the wrong way | 02:16 |
| Qiming | I mean, they are trying solve the DB problem case by case ... but there are too many cases to be enumerated, so ... it won't be a comprehensive solution | 02:17 |
| Qiming | instead, I think we need a discussion on how to cleanse the database when engine gets restarted | 02:18 |
| yanyanhu | I see. | 02:18 |
| Qiming | that is, how to reset each action that was owned by the dead engine, and release locks held by those actions | 02:19 |
| yanyanhu | I recalled ethan's work before partially touched this problem? about dead lock stealing | 02:20 |
| *** shu-mutou has joined #senlin | 02:20 | |
| *** elynn has quit IRC | 02:20 | |
| Qiming | yes, that patch has somehow solved the problem, but not completely | 02:20 |
| Qiming | we have a two-leveled lock design | 02:20 |
| *** elynn has joined #senlin | 02:20 | |
| Qiming | the lock-steal mechanism is a little bit lazy | 02:21 |
| yanyanhu | yes, it happens only in case engine starts | 02:21 |
| Qiming | it is stealing locks when an action wants to lock a node or a cluster | 02:21 |
| Qiming | while in real life, restarting engine would be rare | 02:22 |
| yanyanhu | ah, right, false statement | 02:22 |
| Qiming | (let's skip testing scenarios for the moment) | 02:22 |
| yanyanhu | yes. the key is we don need some events to trigger this checking and lock stealing | 02:23 |
| Qiming | elynn, thanks god you joined promptly | 02:23 |
| yanyanhu | s/don/do | 02:23 |
| Qiming | right | 02:23 |
| elynn | Qiming, what happened? | 02:23 |
| Qiming | so ... if we do house cleaning when an engine starts, I mean, let the new engine to cleanse all things once for all | 02:23 |
| elynn | lock-steal mechanism mess up? | 02:24 |
| Qiming | yes, elynn, it seems not a clean or comprehensive solution, there are still dangling locks somewhere, see : https://bugs.launchpad.net/senlin/+bug/1639122 | 02:24 |
| openstack | Launchpad bug 1639122 in senlin "senlin-cluster-deadlock" [Medium,Confirmed] - Assigned to XueFeng Liu (jonnary-liu) | 02:24 |
| Qiming | so I'm thinking maybe we can fix it this way | 02:24 |
| Qiming | since the lock-steal mechanism (thanks elynn again) is already there working | 02:25 |
| Qiming | and that mechanism is already stealing the lock in question | 02:25 |
| Qiming | why don't we trigger another thread to cleanse the database for the dead engine ? | 02:26 |
| Qiming | here: http://git.openstack.org/cgit/openstack/senlin/tree/senlin/engine/senlin_lock.py#n143 | 02:27 |
| Qiming | we checked and found that an engine is dead, and we steal the lock on the node | 02:28 |
| yanyanhu | so sounds like a periodically task | 02:29 |
| yanyanhu | maybe running beside each engine | 02:29 |
| Qiming | maybe we should trigger a worker thread to do a GC for the dead engine | 02:29 |
| Qiming | I thought about that before, but it is too heavy weight | 02:29 |
| Qiming | it is the same reason people hate polling | 02:29 |
| Qiming | with Python, not real multi-processing, we should avoid doing polling whenever possible, it is a waste of CPU cycles | 02:30 |
| elynn | What's wrong with that code? still trying to catching your discussion... | 02:30 |
| Qiming | http://git.openstack.org/cgit/openstack/senlin/tree/senlin/engine/senlin_lock.py#n143 | 02:30 |
| yanyanhu | if so, we need to consider the situation that this gc thread is dead | 02:31 |
| Qiming | here, after having found that an engine is dead, we steal the lock on line 151, right? elynn | 02:31 |
| elynn | yes | 02:31 |
| elynn | is it a problem? | 02:31 |
| Qiming | we are talking about an improvement to that | 02:31 |
| Qiming | we only stole the node lock today | 02:32 |
| Qiming | and we know the engine E has died | 02:32 |
| Qiming | why don't we trigger a "background" worker thread there, doing some further garbage collection job for the dead engine E? | 02:32 |
| elynn | How can we know that engine E has died? is there any notifications? | 02:33 |
| Qiming | no ... it was detected by your timeout logic | 02:33 |
| Qiming | function called on line 144 | 02:33 |
| elynn | Oh, you mean once when we run into that code and check that engine is dead, then we maybe could clean all dead actions | 02:33 |
| Qiming | yes, exactly | 02:34 |
| Qiming | all dead actions related to that engine | 02:34 |
| Qiming | and all node locks/cluster locks held by those dead actions | 02:34 |
| Qiming | it could be just a new sqlalchemy API | 02:34 |
| Qiming | if done right | 02:34 |
| yanyanhu | sounds good | 02:35 |
| elynn | Hmm, that could do. | 02:35 |
| Qiming | I'd suggest we don't block the return on line 153 | 02:35 |
| yanyanhu | maybe the first step is improve this cleaning logic, then we can consider to add gc thread to invoke it periodically | 02:35 |
| Qiming | instead we launch a background thread to do the job | 02:35 |
| Qiming | yanyanhu, that is another point I want to discuss/argue with you | 02:36 |
| Qiming | please be cautious when introducing periodical threads | 02:36 |
| elynn | I think we could add a thread after http://git.openstack.org/cgit/openstack/senlin/tree/senlin/engine/senlin_lock.py#n152 to clean all actions | 02:36 |
| Qiming | yep, elynn | 02:36 |
| Qiming | sounds feasible? | 02:36 |
| Qiming | yanyanhu, we only have a single process in the Python world | 02:37 |
| elynn | Maybe it's less impact for the current code. | 02:37 |
| yanyanhu | Qiming, yes, each engine is actually a single thread worker | 02:37 |
| XueFeng | Hi | 02:37 |
| elynn | That's a problem | 02:37 |
| yanyanhu | hi, XueFeng | 02:37 |
| Qiming | we should kill all pollers, all periodicals if possible | 02:37 |
| yanyanhu | Qiming, yes, for the architecture we are using now is actually event driven | 02:38 |
| Qiming | right, let's keep things event driven when possible | 02:38 |
| Qiming | until the day we introduce multiprocessing into the code | 02:39 |
| yanyanhu | Qiming, sure, that makes sense | 02:39 |
| Qiming | when one worker/thread is loop polling something, it is tax on all other workers ... unfair, unproductive, waste of resources ... | 02:39 |
| Qiming | elynn, if you (or others), can come up with a DB call that can reset all actions (AND the nodes/clusters locked by those actions) related to an engine known to be dead | 02:41 |
| Qiming | we will be more confident in writing complex/efficient DB queries | 02:41 |
| elynn | Actually I'm thinking making the lock table more general, for example when clean thread is up then add a lock there | 02:41 |
| Qiming | that is doable, but don't lock the world | 02:42 |
| yanyanhu | elynn, you mean something like fencing? :) | 02:42 |
| openstackgerrit | RUIJIE YUAN proposed openstack/senlin: engine work prepare for policy update v2 https://review.openstack.org/394711 | 02:43 |
| elynn | yanyanhu, no... just to make sure only one cleaning thread is running. | 02:43 |
| yanyanhu | I see | 02:43 |
| yanyanhu | yes, that is important | 02:43 |
| elynn | cleaning thread is the thread to reset actions lock. | 02:43 |
| Qiming | we will be more confident in writing complex/efficient DB queries, we can add (back) a 'senlin-manage purge-db' call which can clean up data in the 'action' and 'event' table which are too old based on a criteria | 02:43 |
| yanyanhu | to avoid potential race condition on lock table | 02:43 |
| Qiming | elynn, yep, absolutely | 02:43 |
| elynn | Qiming, I will start from a DB call maybe. | 02:44 |
| elynn | But the problem is how do we deal with that lock if it's not released. | 02:44 |
| Qiming | you can learn from Heat's DB logic on deleting stack tags ... if that helps | 02:44 |
| Qiming | action is locked by a dead engine? okay, it is no longer locked. It is ready to be executed again? not sure, need to check its state. | 02:46 |
| Qiming | node/cluster is locked by an action which was owned by a dead engine? it is no longer locked. Was it in state UPDATING? em ... now it should be put into ERROR maybe. | 02:47 |
| Qiming | we don't know what happened | 02:47 |
| Qiming | when you have questions how to set the node/cluster states, we can discuss. otherwise, use your own judgement would be fine. | 02:48 |
| elynn | Qiming, okay | 02:48 |
| elynn | Qiming, I could start from DB call first, right? | 02:49 |
| Qiming | sure, you decide where to start | 02:49 |
| Qiming | :) | 02:49 |
| Qiming | if still have doubts about the problems you are solving, check the latest comment posted by XueFeng here: https://review.openstack.org/#/c/394407/ | 02:50 |
| elynn | okay, will go through your comments on that patch. | 02:51 |
| Qiming | it was referred to as an 'inconsistency' problem | 02:51 |
| XueFeng | About how to avoid problem happened I have no good idea.Only summit patch 392218.So try to handle inconsistency. | 02:53 |
| Qiming | yes, read back our dicussion since 10:14 | 02:53 |
| XueFeng | Will read. | 02:54 |
| Qiming | XueFeng, hope this can solve the problem you encountered and help you understand our philosophy on the internal design, :) | 02:54 |
| XueFeng | ok,thanks:) | 02:55 |
| openstackgerrit | Merged openstack/python-senlinclient: Revise "enabled" related code in ClusterPolicyUpdate https://review.openstack.org/394314 | 02:59 |
| openstackgerrit | Merged openstack/python-senlinclient: cluster policy attach cannot work https://review.openstack.org/394280 | 02:59 |
| openstackgerrit | Merged openstack/python-senlinclient: policy binding update cannot work https://review.openstack.org/394254 | 03:00 |
| openstackgerrit | Shu Muto proposed openstack/senlin-dashboard: Angularize node tables https://review.openstack.org/387321 | 03:00 |
| openstackgerrit | Merged openstack/python-senlinclient: Policy binding attach cannot work https://review.openstack.org/394256 | 03:01 |
| openstackgerrit | RUIJIE YUAN proposed openstack/senlin: engine work prepare for policy update v2 https://review.openstack.org/394711 | 03:01 |
| openstackgerrit | Yanyan Hu proposed openstack/senlin: Minor fix on node-create API https://review.openstack.org/394723 | 03:27 |
| XueFeng | Hi elynn, | 03:39 |
| elynn | Hi XueFeng | 03:40 |
| XueFeng | elynn | 03:40 |
| XueFeng | "I think we could add a thread after http://git.openstack.org/cgit/openstack/senlin/tree/senlin/engine/senlin_lock.py#n152 to clean all actions " | 03:40 |
| XueFeng | I solve the problem that line 152 can't be run. | 03:41 |
| elynn | In which patch? | 03:41 |
| XueFeng | elynn | 03:41 |
| XueFeng | I think we could add a thread after http://git.openstack.org/cgit/openstack/senlin/tree/senlin/engine/senlin_lock.py#n152 to clean all actions | 03:41 |
| XueFeng | https://review.openstack.org/#/c/394407/ | 03:42 |
| XueFeng | 394407 | 03:42 |
| elynn | Actually I haven't quit understand why line 152 can't be run? | 03:42 |
| openstackgerrit | Merged openstack/senlin: Versioned request object for receiver list https://review.openstack.org/394340 | 03:43 |
| XueFeng | See line 143 | 03:43 |
| elynn | Can I call you XueFeng ? | 03:43 |
| XueFeng | OK | 03:44 |
| Qiming | XueFeng, in that patch | 03:44 |
| Qiming | Line 143 in the original code contains something you didn't change, Line 143 in the revised code is empty | 03:45 |
| Qiming | what are you referring to? | 03:45 |
| Qiming | I'd call that fix a piecemeal fix, it fixes a corner case of the problem | 03:45 |
| Qiming | it introduces two DB interactions on a critical path of lock acquire | 03:46 |
| Qiming | what we discussed this morning is about a more generic solution to solve lock/action/node/cluster problem | 03:46 |
| Qiming | the line 138 in your revised code contains some strange condition for setting forced to True | 03:48 |
| Qiming | it is at best some logic difficult to maintain | 03:48 |
| Qiming | for example, why don't we add the same logic before line 71? | 03:48 |
| Qiming | why we only set forced to True if action.owner is None? | 03:49 |
| Qiming | the thing is not about whether we trust your test results ... okay it is working | 03:49 |
| Qiming | the thing is that we want a more generic solution aiming at the root cause of a problem, which is "engine restarted, db appears inconsistent" | 03:50 |
| Qiming | that root cause will cause various problems, not just the stealing of node locks | 03:51 |
| Qiming | I believe there will be far more problems undercovered if we don't handle engine restart properly | 03:51 |
| xuhaiwei | hi sorry to interrupt, is senlin working well under the latest codes? I got this error when running senlin commands http://paste.openstack.org/show/588334/ | 03:56 |
| openstackgerrit | Merged openstack/senlin: Engine support for receiver_list2 https://review.openstack.org/394357 | 03:59 |
| XueFeng | "for example, why don't we add the same logic before line 71? " this original plan to do in another patch, in this patch' message mentioned "chandling will be in another patch. | 04:21 |
| XueFeng | Yes, need find root cause | 04:22 |
| XueFeng | And need design a more generic sloution | 04:23 |
| XueFeng | "for example, why don't we add the same logic before line 71? " this original plan to do in another patch, in this patch' message mentioned "cluster_lock handling will be in another patch." | 04:28 |
| XueFeng | haiwei, I met the same problem | 04:29 |
| xuhaiwei | XueFeng, it seems something has changed in keystoneauth1 | 04:30 |
| XueFeng | Maybe | 04:30 |
| XueFeng | And I update all modules | 04:31 |
| XueFeng | Also update senlinclient/openstack-sdk | 04:31 |
| xuhaiwei | anyway, I will file a bug report for this | 04:31 |
| XueFeng | Yes | 04:31 |
| XueFeng | Seem not compatibility | 04:32 |
| xuhaiwei | hi elynn, does Heat senlin resource supports nested template in OS::Senlin::Profile type resource? like this http://paste.openstack.org/show/588338/ | 04:39 |
| elynn | xuhaiwei, let me see... | 04:40 |
| xuhaiwei | thanks | 04:40 |
| elynn | Need to do a little change, try with {get_file: example.yaml} when you use heat template. | 04:41 |
| xuhaiwei | got it, thanks | 04:41 |
| Qiming | xuhaiwei, looks like a python-openstacksdk problem again | 05:55 |
| Qiming | xuhaiwei, seems you are invoking some service which doesn't support version negotiation? | 05:58 |
| Qiming | better check the 'uri' content here: File "/usr/local/lib/python2.7/dist-packages/openstack/session.py", line 168 | 05:58 |
| xuhaiwei | Qiming, the uri is the IP address of my host, seem no problem | 06:05 |
| xuhaiwei | in senlin-api logs, it logs http://host_ip, and to client side it returns http://host_ip:8778 with port number added | 06:07 |
| Qiming | what do you get if you do curl http://host_ip:8778 ? | 06:11 |
| xuhaiwei | {"versions": [{"status": "CURRENT", "updated": "2016-01-18T00:00:00Z", "links": [{"href": "/v1", "rel": "self"}, {"href": "http://developer.openstack.org/api-ref/clustering", "rel": "help"}], "min_version": "1.0", "max_version": "1.3", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.clustering-v1+json"}], "id": "1.0"}]} | 06:11 |
| Qiming | okay, that means senlin api is set up correctly and working properly | 06:12 |
| Qiming | openstacksdk wasn't able to parse this to get a versioned endpoint | 06:12 |
| XueFeng | openstacksdk version need update? | 06:14 |
| Qiming | it depends on which version are you using | 06:18 |
| Qiming | in my case, I'm using the master HEAD version, i.e. openstacksdk-0.9.10.dev1, so I don't have a problem of it | 06:21 |
| xuhaiwei | Qiming, XueFeng, I got the reason, it is caused by this patch https://review.openstack.org/#/c/386440/ | 06:21 |
| Qiming | line 77 didn't check if the response contains a JSON or not? | 06:22 |
| xuhaiwei | it is a little strange, if no exceptions returns the exception handing should not be ran | 06:25 |
| Qiming | it seems there still is an exception though | 06:27 |
| openstackgerrit | RUIJIE YUAN proposed openstack/senlin: api support policy update v2 https://review.openstack.org/394789 | 06:54 |
| openstackgerrit | RUIJIE YUAN proposed openstack/senlin: remove dead code in policy update https://review.openstack.org/394790 | 06:58 |
| openstackgerrit | Shu Muto proposed openstack/senlin-dashboard: Angularize node tables https://review.openstack.org/387321 | 07:00 |
| openstackgerrit | Qiming Teng proposed openstack/senlin: Ensure /v1 endpoint returns proper version info https://review.openstack.org/394804 | 07:36 |
| Qiming | HINT: if you are talking about "codes" instead of "code", make sure you are actually meaning it, :) http://english.stackexchange.com/questions/20455/is-it-wrong-to-use-the-word-codes-in-a-programming-context | 07:38 |
| *** yanyanhu has quit IRC | 07:44 | |
| openstackgerrit | RUIJIE YUAN proposed openstack/senlin: engine support policy delete v2 https://review.openstack.org/394806 | 07:48 |
| *** yanyanhu has joined #senlin | 07:51 | |
| openstackgerrit | Qiming Teng proposed openstack/senlin: Move container spec to approved dir https://review.openstack.org/394813 | 08:07 |
| Qiming | ruijie, online? | 08:11 |
| ruijie | yes, Qiming | 08:58 |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:02 | |
| Qiming | hi, ruijie | 09:04 |
| ruijie | hi, Qiming :) | 09:04 |
| Qiming | what's the progress of this blueprint: https://blueprints.launchpad.net/senlin/+spec/support-batching-policy | 09:05 |
| Qiming | can we mark it as completed? | 09:05 |
| Qiming | AFAICT, the spec is mainly about three things: policy itself, update operation and delete operation | 09:06 |
| ruijie | yes, Qiming. Currently, it support cluster_create/update. | 09:06 |
| ruijie | I am not sure if we need to support more cluster actions | 09:06 |
| Qiming | we don't have a decent solution for cluster_create | 09:06 |
| ruijie | sorry, cluster_delete :) | 09:07 |
| Qiming | before a cluster is created, no policy can be checked ... | 09:07 |
| Qiming | how about add/delete nodes? | 09:07 |
| ruijie | yes, but, like some other action cluster_add_nodes or del_nodes may work with it. | 09:07 |
| Qiming | scale out and scale in? | 09:07 |
| ruijie | yes Qiming, some actions need to create or delete nodes | 09:08 |
| Qiming | do we want to add support to more node creation/deletion operations or we just stop here? | 09:09 |
| ruijie | Infact, I hope to add support to more actions | 09:10 |
| ruijie | one thing is that, current logic will reduce efficiency in my opinion | 09:11 |
| Qiming | okay, I'm marking the bp as 'good progress' | 09:11 |
| Qiming | reason? | 09:11 |
| ruijie | since we need to sleep for period after each batch | 09:11 |
| Qiming | yes, that is what the policy says | 09:12 |
| Qiming | batches don't overlap with each other | 09:12 |
| Qiming | it is a tradeoff between robustness and efficiency | 09:13 |
| ruijie | In that case, do we need to consider the "TIME_OUT" exception | 09:13 |
| Qiming | if the interaction with sqlalchemy is ideal, e.g. every db operation will be immediately reflected into Python objects, we can do a lot optimizations | 09:13 |
| ruijie | if the working chain is too long | 09:14 |
| Qiming | we will never know the optimal working chain length, or batch size | 09:14 |
| Qiming | that is why we leave it a policy parameter for users to tune | 09:15 |
| ruijie | okay, that makes sense. | 09:15 |
| Qiming | in an ideal world, this should be implmented as a sliding window, but sqlalchemy interaction is not reliable ... we have encountered a lot of problems trying to get the latest status out of DB table | 09:16 |
| Qiming | python runtime and sqlalchemy package will cache things that have to be explicited invalidated ... | 09:16 |
| Qiming | tracking action status closely remains an open issue till today | 09:17 |
| ruijie | eh, then waiting until the whole batch completed will be better. | 09:18 |
| ruijie | okay, Qiming, I will add support to more actions like scale in/out, resize .etc after versioned resource support completed. | 09:18 |
| Qiming | many thanks! | 09:19 |
| ruijie | my pleasure :) | 09:20 |
| *** elynn has quit IRC | 09:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-senlinclient: Updated from global requirements https://review.openstack.org/394130 | 09:53 |
| openstackgerrit | Qiming Teng proposed openstack/senlin: A spec for generic event/notification support https://review.openstack.org/394874 | 09:58 |
| *** zhurong has quit IRC | 09:58 | |
| openstackgerrit | Qiming Teng proposed openstack/senlin: A spec for generic event/notification support https://review.openstack.org/394874 | 09:59 |
| openstackgerrit | Merged openstack/python-senlinclient: Updated from global requirements https://review.openstack.org/394130 | 10:04 |
| *** yanyanhu has quit IRC | 10:13 | |
| *** XueFeng has quit IRC | 10:21 | |
| openstackgerrit | Shan Guo proposed openstack/senlin: Replaces uuid.uuid4 with uuidutils.generate_uuid() https://review.openstack.org/394892 | 10:39 |
| openstackgerrit | xu-haiwei proposed openstack/senlin: Remove container nodes information from dependents property of cluster https://review.openstack.org/394896 | 10:51 |
| *** ruijie has quit IRC | 11:14 | |
| *** zhurong has joined #senlin | 12:05 | |
| openstackgerrit | Merged openstack/senlin-dashboard: Fix the wrong url to the policy detail page in cluster detail page https://review.openstack.org/390163 | 12:12 |
| openstackgerrit | Merged openstack/senlin-dashboard: Imported Translations from Zanata https://review.openstack.org/389288 | 12:12 |
| *** zhurong has quit IRC | 12:13 | |
| *** zhurong has joined #senlin | 12:21 | |
| *** zhurong has quit IRC | 12:38 | |
| *** yanyanhu has joined #senlin | 12:40 | |
| *** zhurong has joined #senlin | 12:43 | |
| *** Drago has joined #senlin | 12:46 | |
| openstackgerrit | lvdongbing proposed openstack/senlin: Versioned objects for profile request https://review.openstack.org/394941 | 12:47 |
| *** lvdongbing has joined #senlin | 12:49 | |
| *** Ruijie has joined #senlin | 12:50 | |
| yanyanhu | hi, guys, meeting will start in #openstack-meeting channel in minutes | 12:54 |
| yanyanhu | hi, meeting has started | 13:00 |
| *** lixinhui_ has joined #senlin | 13:01 | |
| *** elynn has joined #senlin | 13:02 | |
| openstackgerrit | Liuqing Jing proposed openstack/senlin-dashboard: Add spec file for profiles.service.js https://review.openstack.org/394949 | 13:04 |
| *** miaohb has joined #senlin | 13:09 | |
| *** miaohb has left #senlin | 13:09 | |
| openstackgerrit | Liuqing Jing proposed openstack/senlin-dashboard: Remove unnecessary variable assignment https://review.openstack.org/394952 | 13:16 |
| *** Liuqing has joined #senlin | 13:18 | |
| *** Liuqing has quit IRC | 13:19 | |
| *** Liuqing has joined #senlin | 13:20 | |
| *** zhurong has quit IRC | 13:37 | |
| *** openstackgerrit has quit IRC | 13:48 | |
| *** openstackgerrit has joined #senlin | 13:49 | |
| *** ChanServ sets mode: +v openstackgerrit | 13:49 | |
| *** elynn has quit IRC | 13:59 | |
| *** XueFeng has joined #senlin | 14:00 | |
| *** XueFeng has quit IRC | 14:00 | |
| *** yanyanhu has quit IRC | 14:01 | |
| *** Ruijie has quit IRC | 14:01 | |
| *** lvdongbing has quit IRC | 14:04 | |
| *** Liuqing has quit IRC | 14:10 | |
| *** Liuqing has joined #senlin | 14:11 | |
| *** Liuqing has quit IRC | 14:14 | |
| *** lixinhui_ has quit IRC | 14:25 | |
| *** Drago has quit IRC | 14:49 | |
| *** catintheroof has joined #senlin | 14:57 | |
| *** Drago has joined #senlin | 15:38 | |
| *** Drago has quit IRC | 15:42 | |
| *** Drago has joined #senlin | 15:48 | |
| *** Drago1 has joined #senlin | 17:02 | |
| *** Drago has quit IRC | 17:06 | |
| *** Drago1 has quit IRC | 17:06 | |
| *** Drago has joined #senlin | 17:13 | |
| *** Drago has quit IRC | 17:22 | |
| *** Drago has joined #senlin | 17:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/senlin: Updated from global requirements https://review.openstack.org/395198 | 21:31 |
| *** catintheroof has quit IRC | 22:22 | |
| *** catintheroof has joined #senlin | 22:22 | |
| *** catintheroof has quit IRC | 22:27 | |
| *** lixinhui_ has joined #senlin | 23:16 | |
| *** lixinhui_ has quit IRC | 23:17 | |
| *** catintheroof has joined #senlin | 23:29 | |
| *** catintheroof has quit IRC | 23:31 | |
| *** catintheroof has joined #senlin | 23:31 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!