| *** bobh has quit IRC | 00:04 | |
| *** cheneydc has joined #openstack-mistral | 01:13 | |
| rakhmerov | szaher: pls give more info in bug description on how to reproduce it. As far as debugging, I don't know exactly where it comes from. There's module rpc.py from which it most likely gets called so you need to see what exact method of rpc.py causes this and set a breakpoint there | 01:44 |
|---|---|---|
| rakhmerov | but at first glance it seems like some low level problem, maybe something is wrong with RabbitMQ | 01:45 |
| rakhmerov | there's also a known problem with eventlet when it's running under Apache | 01:47 |
| rakhmerov | if you're running API component under Apache it might be the case | 01:47 |
| *** rrecio has joined #openstack-mistral | 02:34 | |
| *** rrecio has quit IRC | 02:53 | |
| *** gyee has quit IRC | 03:07 | |
| *** cheneydc has quit IRC | 04:35 | |
| openstackgerrit | Merged openstack/mistral: Example Mistral docker container broke due to oslo.policy update https://review.openstack.org/330748 | 04:51 |
| openstackgerrit | hardik proposed openstack/python-mistralclient: Change action-get help to get action info by ID https://review.openstack.org/325921 | 07:15 |
| *** dmk0202 has joined #openstack-mistral | 07:32 | |
| *** dmk0202 has quit IRC | 07:33 | |
| openstackgerrit | Andras Kovi proposed openstack/mistral: Use client credentials to retrieve service list https://review.openstack.org/330469 | 08:04 |
| *** nmakhotkin has quit IRC | 08:10 | |
| *** igormarnat has quit IRC | 08:22 | |
| *** igormarnat has joined #openstack-mistral | 08:22 | |
| *** dmk0202 has joined #openstack-mistral | 08:27 | |
| *** mandre_ has joined #openstack-mistral | 08:37 | |
| *** mflobo1 has joined #openstack-mistral | 08:39 | |
| openstackgerrit | Marcos FermÃn Lobo proposed openstack/mistral: Allow to use both name and id to access action definitions https://review.openstack.org/325894 | 08:40 |
| *** harlowja_ has quit IRC | 08:41 | |
| *** pkoniszewski has quit IRC | 08:41 | |
| *** pkoniszewski has joined #openstack-mistral | 08:43 | |
| *** mandre has quit IRC | 08:43 | |
| *** mflobo has quit IRC | 08:43 | |
| *** mandre_ is now known as mandre | 08:43 | |
| *** ruhe has quit IRC | 08:43 | |
| *** ruhe has joined #openstack-mistral | 08:44 | |
| *** lkannan has quit IRC | 08:47 | |
| *** lkannan has joined #openstack-mistral | 08:49 | |
| openstackgerrit | hardik proposed openstack/python-mistralclient: Changed argument names as per other python clients https://review.openstack.org/330958 | 08:55 |
| *** openstackgerrit has quit IRC | 09:18 | |
| *** openstackgerrit has joined #openstack-mistral | 09:18 | |
| openstackgerrit | hardik proposed openstack/mistral: Fixed get_actions_list script to get designate actions https://review.openstack.org/330983 | 09:26 |
| openstackgerrit | hardik proposed openstack/python-mistralclient: Changed argument names as per other python clients https://review.openstack.org/330958 | 09:46 |
| *** openstackgerrit has quit IRC | 10:03 | |
| *** openstackgerrit has joined #openstack-mistral | 10:03 | |
| hparekh | rakhmerov, hi | 10:06 |
| hparekh | rakhmerov, Regarding https://review.openstack.org/#/c/330983/ | 10:07 |
| rakhmerov | hparekh: hi | 10:07 |
| rakhmerov | yep | 10:07 |
| hparekh | rakhmerov, I am thinking about to test such openstack actions and I have some approach for it. Like we can add some tempest tests for all openstack actions | 10:08 |
| rakhmerov | you just missed one whitespace in the script | 10:08 |
| rakhmerov | oooh, you're talking about my comment | 10:08 |
| rakhmerov | yes | 10:08 |
| rakhmerov | hparekh: how? | 10:08 |
| rakhmerov | we'd better do it sooner than later because it's already very difficult to maintain them | 10:09 |
| hparekh | rakhmerov, I will test in my environment and let you know. I guess tempest has some mechanism for this. | 10:09 |
| rakhmerov | I think we need to do it immediately after https://blueprints.launchpad.net/mistral/+spec/mistral-separate-openstack-actions | 10:09 |
| rakhmerov | hparekh: hm... interesting | 10:10 |
| rakhmerov | the number of action is huge and some APIs are volatile | 10:10 |
| rakhmerov | but ok, I guess there should be a way to automate it somehow | 10:10 |
| hparekh | rakhmerov, I will check about the options, we can discuss it in weekly meeting | 10:11 |
| rakhmerov | yes, let's do | 10:11 |
| openstackgerrit | hardik proposed openstack/mistral: Fixed get_actions_list script to get designate actions https://review.openstack.org/330983 | 10:12 |
| rakhmerov | hparekh: I included a topic into the agenda: https://wiki.openstack.org/wiki/Meetings/MistralAgenda | 10:16 |
| openstackgerrit | hardik proposed openstack/mistral: Fixed get_actions_list script to get glance actions https://review.openstack.org/331015 | 10:27 |
| openstackgerrit | hardik proposed openstack/mistral: Fixed get_actions_list script to get glance actions https://review.openstack.org/331015 | 10:29 |
| szaher | rakhmerov: Hi | 10:38 |
| rakhmerov | hi | 10:38 |
| szaher | rakhmerov: For the bug I filed for rpc problem I don't how to debug the api, How can I print the method | 10:38 |
| szaher | rakhmerov: rabbitmq is working fine and the cloud is working fine | 10:38 |
| rakhmerov | I'm not sure what method you're talking about | 10:39 |
| szaher | I created a user for mistral with admin privileges | 10:39 |
| rakhmerov | do you have a full stack trace? | 10:39 |
| rakhmerov | what is the place in Mistral that this error is caused by? | 10:39 |
| rakhmerov | my guess is that it should come from rpc.py module, but I may be wrong | 10:40 |
| szaher | it's in the API component once I run it gives me this error without doing anything | 10:40 |
| rakhmerov | what's you config file? Can you post it? | 10:41 |
| rakhmerov | specifically, I'd like to see host and port for API | 10:41 |
| rakhmerov | by default, it's localhost:8989 | 10:42 |
| rakhmerov | [api] | 10:42 |
| rakhmerov | # Mistral API server host and port | 10:42 |
| rakhmerov | host=0.0.0.0 | 10:42 |
| rakhmerov | port=8989 | 10:42 |
| szaher | rakhmerov: http://paste.openstack.org/show/516808/ | 10:43 |
| rakhmerov | may be the problem is here and Mistral API can't be bound to host and port you specified | 10:43 |
| szaher | this my configuration file. I would appreciate if you can take a look and let me know if I am doing something wrong | 10:44 |
| rakhmerov | looking.. | 10:44 |
| szaher | Thanks | 10:44 |
| rakhmerov | seems like you have settings by default | 10:44 |
| rakhmerov | so it's localhost:8989 | 10:45 |
| szaher | ouch no sorry. It's not the one I am using in my env | 10:45 |
| szaher | in my env the API is running on 172.16.150.20 | 10:45 |
| szaher | it's uncommented on the server | 10:46 |
| rakhmerov | ok | 10:46 |
| rakhmerov | one observation: I'm looking at my config file and rabbit props are in section [DEFAULT] | 10:47 |
| rakhmerov | not in [oslo_messaging_rabbit] | 10:47 |
| rakhmerov | so you can try that too | 10:47 |
| rakhmerov | but it's unlikely the problem actually.. | 10:48 |
| szaher | I knew the problem now | 10:48 |
| szaher | I changed the host to 127.0.0.1 and I'm not getting this problem | 10:49 |
| szaher | I cannot understand what's going on ! | 10:49 |
| rakhmerov | something with network interfaces I guess | 10:50 |
| rakhmerov | make sure this IP is actually known to some of the available network interfaces | 10:50 |
| szaher | venv/mistral/mistral# curl http://172.16.150.20:8989 | 10:51 |
| szaher | [{"status": "CURRENT", "link": {"href": "http://172.16.150.20:8989/v2", "target": "v2"}, "id": "v2.0"}]( | 10:51 |
| rakhmerov | btw, is 59624 the port you're trying to bind to? | 10:51 |
| szaher | no ! | 10:51 |
| rakhmerov | szaher: yes, it's a good response from API, meaning it works | 10:51 |
| rakhmerov | ok | 10:51 |
| rakhmerov | I'm asking because I see this in the bug description: Exception happened during processing of request from ('172.16.150.20', 59624) | 10:52 |
| szaher | rakhmerov: This response from the api while it's generating the problem and it's working on 172.16.150.20 | 10:52 |
| rakhmerov | ok | 10:52 |
| szaher | the API is working but generating this error message and I don't know from where it's coming ! | 10:52 |
| rakhmerov | it just generated this error output once when you started it? | 10:53 |
| rakhmerov | or on each request to API? | 10:53 |
| rakhmerov | how? | 10:53 |
| szaher | once I start it | 10:54 |
| rakhmerov | hm.. | 10:54 |
| szaher | and it's working fine and keeps generating this problem ! | 10:54 |
| rakhmerov | you're on master branch? | 10:54 |
| szaher | liberty and I think master as well was doing the same problem ! | 10:54 |
| rakhmerov | ok | 10:55 |
| rakhmerov | are you running API under Apache | 10:55 |
| rakhmerov | ? | 10:55 |
| szaher | No | 10:55 |
| rakhmerov | ok | 10:56 |
| szaher | now another weird problem I am having which is the authentication problem. I got the token from the logs and I initialized the glance client with the same token that mistral passed to it to get instance and I was able to work well with the glance API ! | 10:56 |
| rakhmerov | try to place your rabbit config properties under [DEFAULT] section | 10:56 |
| rakhmerov | and restart it | 10:56 |
| rakhmerov | now I guess the problem is with rabbit | 10:57 |
| rakhmerov | because if API started it means that wsgi server was successfully bound to host:port | 10:57 |
| szaher | Ok | 10:58 |
| rakhmerov | I'm 99% sure now that it's misconfiguration for rabbit | 10:58 |
| rakhmerov | here's what I have for rabbit in my config that's working: http://paste.openstack.org/show/516811/ | 10:59 |
| szaher | rakhmerov: moved rabbit to default and it's not working | 11:00 |
| rakhmerov | hm.. ok | 11:01 |
| rakhmerov | rabbit_hosts = 172.16.150.20:5672 | 11:01 |
| rakhmerov | pls replace with two options: rabbit_host = 172.16.150.20 and rabbit_port = 5672 | 11:02 |
| szaher | Ok | 11:02 |
| szaher | rakhmerov: still the same problem ! | 11:04 |
| szaher | http://paste.openstack.org/show/516813/ | 11:05 |
| rakhmerov | can you try to launch a workflow? | 11:05 |
| rakhmerov | I wonder if rabbitmq is actually connected to API | 11:05 |
| szaher | http://paste.openstack.org/show/516814/ | 11:06 |
| szaher | I can create workflows but I get authentication problem | 11:06 |
| rakhmerov | hm.. f..k | 11:07 |
| szaher | I don't know really how to get where the problem is ? | 11:08 |
| rakhmerov | which means that RabbitMQ is actually initialized propertly | 11:08 |
| szaher | I think so | 11:08 |
| rakhmerov | otherwise you wouldn't be able to even start a workflow | 11:08 |
| rakhmerov | because API talks to Engine via Rabbit | 11:08 |
| rakhmerov | any guesses what this number can mean? 59624 | 11:10 |
| rakhmerov | should be some port.. | 11:10 |
| szaher | give me a second plz | 11:10 |
| szaher | rakhmerov: everytime the port is different http://paste.openstack.org/show/516815/ | 11:11 |
| rakhmerov | I see | 11:12 |
| rakhmerov | python version? | 11:12 |
| rakhmerov | ooh, 2.7 | 11:13 |
| rakhmerov | I see | 11:13 |
| szaher | Python 2.7.9 | 11:13 |
| rakhmerov | how do you launch the components? | 11:13 |
| rakhmerov | what commands? | 11:13 |
| szaher | different ways | 11:13 |
| szaher | mistral-server --conf.. | 11:13 |
| szaher | then I got the error and I switched to python mistral/cmd/launch.py --server api --config-file /opt/stack/service/mistral/etc/mistral/mistral.conf | 11:14 |
| openstackgerrit | hardik proposed openstack/mistral: Fixed get_actions_list script to get glance actions https://review.openstack.org/331015 | 11:15 |
| rakhmerov | szaher: yeah, that's ok.. | 11:15 |
| szaher | so any ideas how can I solve these problems ? | 11:16 |
| rakhmerov | so, the general problem is that API is trying to write into a socket which is already closed | 11:18 |
| rakhmerov | as if you had a client that opened a TCP session and then didn't wait for all data from the server (Mistral API) and close the session | 11:19 |
| rakhmerov | in this case socket on the server side would be broken | 11:19 |
| rakhmerov | it may happen, for example, if some client sets too small HTTP timeout when making a request | 11:19 |
| rakhmerov | what client are you using to work with Mistral API? | 11:20 |
| rakhmerov | python-mistralclient? | 11:20 |
| szaher | python-mistralclient | 11:20 |
| rakhmerov | ok | 11:20 |
| rakhmerov | on the client-side, do you observe any serious delays when running commands? | 11:21 |
| rakhmerov | no errors on the client side? | 11:21 |
| szaher | no | 11:21 |
| szaher | rakhmerov: Look I don't mind the problem I am seeing in the API. I want to have at least the get image action working | 11:23 |
| szaher | http://paste.openstack.org/show/516821/ | 11:23 |
| rakhmerov | szaher: ok, that's weird. My recommendation for this broken pipe thing would be to 1) try on Py3 2) reinstall on other machine or operating system | 11:24 |
| rakhmerov | as far as auth problem, let me see... | 11:24 |
| rakhmerov | szaher: I'm just afraid that something might not work propertly if this error occurs | 11:25 |
| rakhmerov | for example, access to Keystone or something else | 11:25 |
| szaher | but it gets the token from keystone | 11:25 |
| rakhmerov | ok | 11:26 |
| rakhmerov | what were you able to find yesterday with nmakhotkin? | 11:27 |
| rakhmerov | on auth problem? | 11:27 |
| szaher | he asked me to use the same credentials on both client and server and I did and it's not working | 11:27 |
| szaher | he asked me to try with the same user to do glance image-list and it was working ! | 11:28 |
| rakhmerov | ok | 11:28 |
| szaher | rakhmerov: where do you initialize the client to run an action | 11:29 |
| rakhmerov | give me a sec | 11:29 |
| rakhmerov | https://github.com/openstack/mistral/blob/stable/liberty/mistral/actions/openstack/actions.py#L68 | 11:31 |
| rakhmerov | you can insert printing of all the stuff in there | 11:31 |
| rakhmerov | like glance_endpoint.url | 11:31 |
| rakhmerov | to make sure it's correct | 11:32 |
| szaher | Ok | 11:33 |
| rakhmerov | one assumption that I have: Mistral may be getting a private endpoint from Keystone | 11:33 |
| rakhmerov | which is not allowed to use from outside of OpenStack management network | 11:33 |
| rakhmerov | please see what the endpoint is | 11:33 |
| rakhmerov | and it'll be clear what the problem is | 11:33 |
| rakhmerov | szaher: any luck? | 11:38 |
| szaher | can't see any printing | 11:38 |
| rakhmerov | how are you doing it? | 11:39 |
| rakhmerov | just insert print("glance_endpoint.url: " + glance_endpoint.url) | 11:40 |
| szaher | rakhmerov: that is ok http://paste.openstack.org/show/516826/ / | 11:40 |
| szaher | LOG.info('>>>>>>>>>>>>>>>> Glance Token {0}'.format(ctx.auth_token)) | 11:41 |
| szaher | print ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoint >>> ", glance_endpoint.url | 11:41 |
| szaher | print ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoint >>> ", glance_endpoint | 11:41 |
| szaher | print ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> endpoint >>> ", ctx.auth_token | 11:41 |
| rakhmerov | yes | 11:41 |
| rakhmerov | what do you see? | 11:41 |
| szaher | nothing | 11:42 |
| szaher | http://paste.openstack.org/show/516827/ | 11:42 |
| rakhmerov | how come? | 11:42 |
| rakhmerov | maybe you modified the wrong file? | 11:43 |
| szaher | rakhmerov: what I see is http://paste.openstack.org/show/516827/ | 11:43 |
| rakhmerov | ooh, sorry | 11:44 |
| rakhmerov | don't use "print" | 11:44 |
| rakhmerov | use LOG.info() | 11:44 |
| szaher | Ok | 11:44 |
| rakhmerov | I guess console appender is disabled | 11:45 |
| szaher | rakhmerov: very strange !! can't see it in the logs !http://paste.openstack.org/show/516828/ | 11:47 |
| rakhmerov | hm.. | 11:49 |
| rakhmerov | what is +-+-+_+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-m ? | 11:49 |
| rakhmerov | is this something that you added? | 11:49 |
| szaher | Yes | 11:49 |
| rakhmerov | where did you insert those printings? | 11:49 |
| szaher | but not in the same file | 11:49 |
| rakhmerov | in the code | 11:49 |
| szaher | rpc.py | 11:49 |
| rakhmerov | yeah, I mean for printing endpoint stuff? | 11:50 |
| szaher | sorry this one was in the default executor | 11:50 |
| rakhmerov | ooh, yes! | 11:50 |
| rakhmerov | you have to restart executor if it's a separate process | 11:51 |
| szaher | the endpoint stuff in actions under actions/openstack/actions.py | 11:51 |
| rakhmerov | ok | 11:51 |
| szaher | I did restart everything everytime | 11:51 |
| rakhmerov | using the same config for executor? | 11:51 |
| szaher | Yes | 11:52 |
| rakhmerov | I don't know, you should see them then | 11:53 |
| rakhmerov | are you running executor separately? | 11:53 |
| szaher | I removed all .pyc file and install the package again and I can't see them ! | 11:53 |
| szaher | Yes | 11:53 |
| rakhmerov | it's weird, isn't it? | 11:54 |
| szaher | agree | 11:55 |
| rakhmerov | check again, may be you're fixing a wrong file (e.g. if you have multiple installations or something) | 11:57 |
| szaher | rakhmerov: I renamed the file and mistral is working fine !! | 11:57 |
| rakhmerov | or you can try to use a different config for executor with different log path | 11:58 |
| rakhmerov | which file? | 11:58 |
| szaher | mistral/actions/openstack/actions.py | 11:58 |
| rakhmerov | yeah, I mean you have exactly one installation? | 12:01 |
| szaher | Yes | 12:02 |
| rakhmerov | dunno what else to assume, honestly | 12:02 |
| rakhmerov | it can't work with that file renamed | 12:02 |
| rakhmerov | may be you're running something with mistral-server and something with python mistral/cmd/launch.py? | 12:03 |
| szaher | I don't know ! | 12:03 |
| szaher | I am developing in other openstack project and to be honest this the first time to get something like that ! | 12:03 |
| rakhmerov | in this case you would be running two different source code | 12:03 |
| szaher | I don't run mistral-server I only run with launch.py | 12:04 |
| rakhmerov | both engine and executor, right? | 12:04 |
| rakhmerov | ok, so | 12:04 |
| szaher | api, engine, executor | 12:05 |
| rakhmerov | then if you insert print ">>>> " into this file you should see those printings on executor console | 12:05 |
| rakhmerov | stdout | 12:05 |
| *** dmk0202 has quit IRC | 12:05 | |
| szaher | I am not sure | 12:05 |
| rakhmerov | not sure about what? | 12:06 |
| rakhmerov | I am sure | 12:06 |
| szaher | can you do it with your installation and tell me if it's working fine with you or not ? | 12:06 |
| rakhmerov | I don't have any OpenStack installation at hands at the moment | 12:06 |
| rakhmerov | and I need to run | 12:06 |
| rakhmerov | try to insert printings again with print """ | 12:07 |
| szaher | rakhmerov: Thanks for your help | 12:07 |
| rakhmerov | and look at the window where your executor is running | 12:07 |
| rakhmerov | you must see the output | 12:07 |
| rakhmerov | I'll wait a few more mins | 12:08 |
| rakhmerov | let me know what you got.. | 12:08 |
| rakhmerov | any news? | 12:11 |
| szaher | not yet | 12:11 |
| rakhmerov | szaher: if we don't solve it today I'll try to have someone look at it next week | 12:12 |
| rakhmerov | or I'll look at this myself | 12:12 |
| rakhmerov | there should be something simple here | 12:12 |
| rakhmerov | just need to properly debug it | 12:12 |
| szaher | Ok, Thanks | 12:15 |
| rakhmerov | szaher: ok, I'm leaving for now, may be online later | 12:15 |
| rakhmerov | please write here when you see those printings | 12:16 |
| szaher | Ok | 12:16 |
| *** dprince has joined #openstack-mistral | 12:19 | |
| szaher | rakhmerov: can you send me your keystone authtoken config ? | 13:29 |
| *** bobh has joined #openstack-mistral | 13:45 | |
| szaher | nmakhotkin: Hi | 13:59 |
| szaher | Are you here ? | 13:59 |
| *** dprince has quit IRC | 14:42 | |
| *** dprince has joined #openstack-mistral | 14:43 | |
| *** rrecio has joined #openstack-mistral | 14:51 | |
| *** rrecio_ has joined #openstack-mistral | 14:51 | |
| *** rrecio has quit IRC | 14:55 | |
| *** dprince has quit IRC | 15:08 | |
| *** dprince has joined #openstack-mistral | 15:09 | |
| *** dprince has quit IRC | 15:11 | |
| jpeeler | i have a mistral workflow question related to how on-error is handled. if a task is unsuccessful, is it by design that on-error is not executed or is there something wrong with the implemented action? | 15:19 |
| jpeeler | for a concrete example, see https://review.openstack.org/#/c/324894/ | 15:19 |
| jpeeler | i'm noticing that when the create_plan action fails, on-error isn't run. trying to determine if this is a bug or not | 15:20 |
| *** toddjohn has joined #openstack-mistral | 15:32 | |
| jpeeler | an even simpler example is located here https://github.com/rbrady/tripleo-common/commit/7dd747303ad1d78d59efc84177378cb0246d2fbf | 15:57 |
| *** tonytan4ever has joined #openstack-mistral | 15:59 | |
| jpeeler | the code here seems to indicate that on-error should execute: https://github.com/openstack/mistral/blob/master/mistral/actions/base.py#L53-L60 | 16:05 |
| *** dmk0202 has joined #openstack-mistral | 16:09 | |
| *** toddjohn has quit IRC | 16:16 | |
| *** toddjohn has joined #openstack-mistral | 16:23 | |
| *** dprince has joined #openstack-mistral | 16:28 | |
| *** toddjohn has quit IRC | 16:32 | |
| *** toddjohn has joined #openstack-mistral | 16:33 | |
| *** toddjohn has quit IRC | 16:37 | |
| *** gyee has joined #openstack-mistral | 16:42 | |
| *** harlowja has joined #openstack-mistral | 16:53 | |
| *** tonytan4ever has quit IRC | 17:07 | |
| *** gyee has quit IRC | 17:55 | |
| *** tonytan4ever has joined #openstack-mistral | 17:57 | |
| *** harlowja has quit IRC | 18:04 | |
| *** dprince has quit IRC | 18:06 | |
| *** dprince has joined #openstack-mistral | 18:06 | |
| jpeeler | for those following along, it turns out i hadn't updated the workflow in mistral >_< | 18:42 |
| jpeeler | no bug in mistral | 18:42 |
| *** toddjohn has joined #openstack-mistral | 19:12 | |
| *** toddjohn has quit IRC | 19:17 | |
| *** toddjohn has joined #openstack-mistral | 19:17 | |
| *** gyee has joined #openstack-mistral | 19:17 | |
| *** toddjohn has quit IRC | 19:22 | |
| *** harlowja has joined #openstack-mistral | 19:32 | |
| *** tonytan4ever has quit IRC | 19:39 | |
| *** dmk0202 has quit IRC | 21:33 | |
| *** dmk0202 has joined #openstack-mistral | 21:34 | |
| *** dprince has quit IRC | 21:36 | |
| *** dmk0202 has quit IRC | 21:50 | |
| *** bobh has quit IRC | 22:29 | |
| *** rrecio_ has quit IRC | 23:13 | |
| *** gyee has quit IRC | 23:46 | |
| *** rrecio has joined #openstack-mistral | 23:53 | |
| *** rrecio_ has joined #openstack-mistral | 23:55 | |
| *** rrecio has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!