| openstackgerrit | Merged openstack/storlets: Add missing main functions for unit test codes https://review.openstack.org/349570 | 00:05 |
|---|---|---|
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: Add unittests and cleanup codes https://review.openstack.org/346614 | 01:09 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: Add capability for crazy things https://review.openstack.org/347258 | 01:15 |
| *** takashi has joined #openstack-storlets | 04:09 | |
| takashi | kota_, eranrom: I'm sorry but I'm not available for today's weekly meeting. (I'm not sure it the schedule change has already decided) | 04:10 |
| kota_ | takashi: no worries | 04:11 |
| takashi | kota_: I think I need to dicuss with you about the file_manager interface. I think can take some time tommorow, if you are available. | 04:11 |
| kota_ | takashi: thanks | 04:12 |
| kota_ | will do | 04:12 |
| takashi | kota_: please check the comment I just added to that patch. | 04:12 |
| *** takashi has quit IRC | 04:14 | |
| eranrom | takashi: I have changed the schedule - thought its ok. Let me know if you wish to change it from Tue 0800 UTC. | 05:47 |
| eranrom | Otherwise, no problem about today. Let us know if you have some review order preference. | 05:48 |
| kota_ | oops | 07:16 |
| kota_ | sorry 8:00 UTC has passed | 07:16 |
| kota_ | I thought it's in one hour later. | 07:17 |
| eranrom | no problem. I though its in 40 mins :-) | 07:17 |
| eranrom | Shell we start? | 07:18 |
| kota_ | eranrom: sure | 07:18 |
| eranrom | ok, agenda is here: https://wiki.openstack.org/wiki/Meetings/Storlets | 07:19 |
| eranrom | let me know if you have anything else. | 07:19 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: WIP: Items to consider for FileManager https://review.openstack.org/349834 | 07:20 |
| kota_ | that one :-) | 07:20 |
| eranrom | ok. Let me have a look while we are talking. | 07:21 |
| eranrom | Are we ok with creating a mitaka branch on top of master? | 07:21 |
| eranrom | anything we want to land before this? | 07:21 |
| kota_ | I think no blocker to do that | 07:21 |
| eranrom | ok great. | 07:21 |
| eranrom | I read your correspondence with Takashi on the dead lock comment. I was not sure what conclusion you have came to and whether you still see a problem | 07:22 |
| kota_ | at swift <-> container conversation? | 07:23 |
| eranrom | It was about the writing being done in a saperate green thread | 07:24 |
| kota_ | ah, yes, that was comming from my question. | 07:24 |
| kota_ | so why it causes deadlock is my initial question but storlets seems not to ensure the data draining before ack to swift, it's answer I got. | 07:25 |
| eranrom | When you say storlets not ensure, you mean the middleware or the part running in the docker container? | 07:26 |
| kota_ | to switch the thread to wait the ack in swift while IO wait, the eventlet threading is needed there. | 07:26 |
| kota_ | i meant docker container. | 07:26 |
| kota_ | so storlet app may not to read() for all input stream, right? | 07:27 |
| eranrom | right. This will cause the writing thread to get stuck at some point, and since we are writing with a timeout (I think) it should be ok. Am I missing something? | 07:28 |
| kota_ | no? it fits my idea for now :-) | 07:28 |
| kota_ | but I and takashi_ has thought for more improvement for this | 07:29 |
| eranrom | ok. listening :-) | 07:29 |
| kota_ | if swift gateway got ack from container w/o draining, probably we should kill coroutines which writing to the fd to container. | 07:30 |
| kota_ | because it will not be read :/ | 07:30 |
| kota_ | and possibly, we should ensure all eventlet for writing fds to the container finished up when returning storlet response to handler. | 07:31 |
| kota_ | that is sort of our idea. | 07:32 |
| kota_ | I didn't make a patch and figure out how to do that yet | 07:32 |
| eranrom | Let me see if I get this right: | 07:33 |
| kota_ | sure | 07:33 |
| eranrom | We know that the storlet does not drain the input stream when we get a timeout in the thread writing to the storlet - right? | 07:34 |
| kota_ | probably | 07:35 |
| eranrom | and this is where you want to add the cleanup code? (e.g. "ensure all eventlet for writing fds to the container finished up") | 07:35 |
| eranrom | that is got timeout -> run cleanup | 07:36 |
| kota_ | maybe a part of my thought. | 07:36 |
| kota_ | my scenario is the case storlet app doesn't drain the input stream intentionally. | 07:37 |
| eranrom | ok. I guess it would be best to wait for a patch. At this point I have one comment (perhaps I have it because I do not fully understand your thought): | 07:39 |
| eranrom | You are writing: "we should ensure all eventlet for writing fds to the container finished up when returning storlet response to handler." | 07:40 |
| kota_ | yes | 07:40 |
| eranrom | my concern here is that we will start responding to the user only after all the writing has been done. I mean the idea is to have stream processing which may be very long | 07:40 |
| eranrom | and ideally we should be able to start streaming back the response, even before we consumed all the input... | 07:41 |
| eranrom | does it make sense? | 07:41 |
| kota_ | ah... ah | 07:41 |
| kota_ | streaming processing need to write the input stream after got response | 07:42 |
| kota_ | good perspective. | 07:42 |
| kota_ | probably, my original idea conflicts with that case | 07:43 |
| eranrom | That was my understanding. BTW it can also cause swift to timeout if the processing of the whole data takes time... | 07:44 |
| kota_ | ok, eranrom. that's obviously good input to consider the eventlet life-cycle. thanks. | 07:44 |
| eranrom | ok. so next topic? | 07:44 |
| kota_ | go ahread | 07:44 |
| eranrom | Taksahi has made a good point about the docker pull causing many gateway failures. | 07:45 |
| eranrom | The only solution I have in mind is keeping that image in the code... | 07:45 |
| eranrom | this is ~65MB when zipped. | 07:45 |
| kota_ | woops | 07:45 |
| eranrom | right... | 07:46 |
| eranrom | perhaps we can keep this in another repo, and instead of docker pull do git checkout or wget.... | 07:46 |
| eranrom | assuming git checkout / wget are better then docker pull.... | 07:47 |
| kota_ | or can we make a cache (or testing VM image) in openstack-infra? | 07:47 |
| kota_ | I'm not sure around the infra config but... something like dsvm... | 07:47 |
| eranrom | sounds right! | 07:48 |
| eranrom | So we basically need to ask infra to add a testing vm image - right? | 07:48 |
| kota_ | yes | 07:48 |
| eranrom | sounds like the right idea to me! | 07:48 |
| kota_ | if it could be, we can make a vm image including the docker image, right? | 07:48 |
| eranrom | that is the ubuntu 14.04 image yes. | 07:49 |
| eranrom | I will add your idea to the bug description so that we do not forget this. | 07:49 |
| kota_ | :-) | 07:50 |
| eranrom | ok, so next topic. | 07:51 |
| eranrom | review prioritisation. Its been a very busy week and you (NTT) had a really impressive progress! | 07:52 |
| kota_ | :-) | 07:52 |
| eranrom | I tried to review quickly so as not to cause too much of a delay | 07:53 |
| eranrom | any priorities? | 07:53 |
| kota_ | right now, I think they are not urgent... | 07:54 |
| eranrom | ok, so I will review as I see fit. Do let me know if something changes. | 07:54 |
| kota_ | recently I was working on a thing... | 07:54 |
| kota_ | wait | 07:54 |
| kota_ | https://review.openstack.org/#/c/347258/ | 07:55 |
| patchbot | kota_: patch 347258 - storlets - Add capability for crazy things | 07:55 |
| kota_ | here | 07:55 |
| kota_ | yeah, no docs sorry so that just drafting to make sure the capability | 07:55 |
| eranrom | oh yes. I saw it but did not get a chance to review yet. | 07:55 |
| kota_ | sort of multi input stream to storlet app container. | 07:55 |
| kota_ | but I know it's hard to do that in object-server side so it's a toy right now | 07:56 |
| kota_ | I need more consideration for the efficienccy and make sure if it's really right thing. | 07:56 |
| eranrom | ok, no problem. In fact we had this in the past (before we open sourced the project). This is the use case we had: | 07:56 |
| kota_ | any others from me are all I found/want to fix while writing the code :P | 07:57 |
| kota_ | oh, interesting on your use case | 07:57 |
| eranrom | media files. Media files are very large and their metadata is often being kept in additional .xml files. So you have one huge file and then several more .xml metadata files. | 07:58 |
| eranrom | The requirement (which came at the time from RAI - Radio television Italy) was that the storlet gets access to all of them | 07:58 |
| eranrom | so we implemented the ability to consume several streams | 07:58 |
| eranrom | the point was that the streams where not symmetric: one stream was from the swift url (this was the media file), and the other streams were specified in | 07:59 |
| eranrom | headers. Thus the storlet was invoked on the object server that held the media file, and there was not much data to move around... | 08:00 |
| kota_ | gotcha | 08:01 |
| eranrom | BTW we also had the ability for the storlet to write more then one object, but we can discuss this some other time :-) | 08:01 |
| kota_ | hew, that sounds great :-) | 08:02 |
| eranrom | ok, https://review.openstack.org/349834? | 08:02 |
| kota_ | oh, yea | 08:02 |
| kota_ | I didn't to ask to takashi in detail yet but it's too conflicted with my multi inputs :/ | 08:03 |
| kota_ | it's ok to conflict though | 08:03 |
| kota_ | ah | 08:03 |
| kota_ | wait, i mean | 08:03 |
| kota_ | patch 349566 | 08:04 |
| patchbot | kota_: https://review.openstack.org/#/c/349566/ - storlets - Validate file_manager option in DockerStorletRequest | 08:04 |
| kota_ | no | 08:04 |
| kota_ | that's a follow up | 08:04 |
| kota_ | patch 348482 | 08:04 |
| patchbot | kota_: https://review.openstack.org/#/c/348482/ - storlets - Create FileManager class (MERGED) | 08:04 |
| kota_ | yes, thank you patchbot | 08:04 |
| eranrom | oh, so I already merged something that conflicts with the multi-inputs | 08:05 |
| eranrom | :-) | 08:05 |
| kota_ | yeah | 08:05 |
| eranrom | sorry... | 08:05 |
| kota_ | so | 08:05 |
| kota_ | no no no | 08:05 |
| kota_ | it's OK because due to the conflict, i can find more improvement for that. | 08:06 |
| kota_ | cuircle back to patch 349824 | 08:06 |
| patchbot | kota_: https://review.openstack.org/#/c/349824/ - python-senlinclient - Fix typo (MERGED) | 08:06 |
| kota_ | no | 08:06 |
| kota_ | https://review.openstack.org/349834 | 08:06 |
| kota_ | sorry | 08:06 |
| kota_ | back to there | 08:07 |
| eranrom | ok :-) | 08:07 |
| eranrom | So you want me to hold that one till you fix the conflicts in multi-stream? | 08:07 |
| kota_ | I'd like to make the file management of get_storlet/get_dependency/put_log more general | 08:07 |
| eranrom | ok gotcha | 08:07 |
| eranrom | and this is what 3349834 is about? | 08:08 |
| kota_ | I don't have priority for them because I know they're conflict each other and I can resolve it. | 08:08 |
| kota_ | yes | 08:08 |
| kota_ | in the patch 349834 | 08:09 |
| patchbot | kota_: https://review.openstack.org/#/c/349834/ - storlets - WIP: Items to consider for FileManager | 08:09 |
| eranrom | ok, If I get you right 349834 is a pre-req to multi-streams | 08:09 |
| eranrom | is that right? | 08:09 |
| kota_ | yes | 08:09 |
| eranrom | ok, so I will look at 349834 first. | 08:09 |
| eranrom | maybe I will look at both, but comment on 349834 first :-) | 08:10 |
| kota_ | thanks | 08:10 |
| eranrom | sure. There is one last thing Doron has reminded me today: I think its high time we try and make the project official (specifically with all recent activity) | 08:11 |
| eranrom | I will send a mail to the list in the coming days... | 08:11 |
| kota_ | for openstack official? | 08:12 |
| eranrom | yes | 08:12 |
| kota_ | yey :-) | 08:12 |
| eranrom | :-) | 08:12 |
| eranrom | That is all I have. | 08:12 |
| kota_ | ok, thanks. glad to hear. | 08:13 |
| eranrom | alright, so talk to you later | 08:13 |
| eranrom | and thanks for joining | 08:14 |
| kota_ | \o/ | 08:14 |
| eranrom | A reminder to vote for the storlet related talks :-) | 11:15 |
| eranrom | https://www.openstack.org/summit/barcelona-2016/vote-for-speakers/ | 11:15 |
| eranrom | search for "storlet" and "crystal" | 11:15 |
| openstackgerrit | Merged openstack/storlets: Add unittests and cleanup codes https://review.openstack.org/346614 | 11:17 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Refactor dispatch_command https://review.openstack.org/345892 | 12:39 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Set os.environ parameters in subprocess https://review.openstack.org/349597 | 12:41 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Fix usage of account inside gateway/runtime module https://review.openstack.org/349607 | 13:01 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Refactor dispatch_command https://review.openstack.org/345892 | 13:43 |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Set os.environ parameters in subprocess https://review.openstack.org/349597 | 13:45 |
| openstackgerrit | Eran Rom proposed openstack/storlets: New storlet for testing pushdown https://review.openstack.org/350125 | 15:51 |
| openstackgerrit | Eran Rom proposed openstack/storlets: Storlets release 0.1.0 https://review.openstack.org/350157 | 16:43 |
| *** openstackgerrit_ has joined #openstack-storlets | 20:35 | |
| *** openstackgerrit_ has quit IRC | 20:36 | |
| *** notmyname has quit IRC | 20:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!