| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Add range support for python storlet https://review.openstack.org/365317 | 01:42 |
|---|---|---|
| openstackgerrit | Kota Tsuyuzaki proposed openstack/storlets: Prep works for Py3 gate https://review.openstack.org/364764 | 04:27 |
| openstackgerrit | Merged openstack/storlets: Creanup redundant README.md and README.rst https://review.openstack.org/347278 | 05:13 |
| *** takashi has joined #openstack-storlets | 05:27 | |
| openstackgerrit | Takashi Kajinami proposed openstack/storlets: Change docs for Python storlet support https://review.openstack.org/365213 | 05:30 |
| takashi | eranrom: thanks for your review about python storlet patches. I'll check you comment! | 06:05 |
| eranrom | takashi: sure, I am now looking at the ranges patch | 06:05 |
| eranrom | takashi: I went over all the patches in the blueprint. Awesome work! Thanks! | 06:11 |
| eranrom | Here is a summary of the major comments I have: | 06:11 |
| eranrom | in order of importance as I see it (feel free to disagree) | 06:12 |
| eranrom | 1. Consume the fds in the docker side according to their expected order, and adjust the datagram getters accordingly. We can discuss in what layer to do that. | 06:12 |
| eranrom | 2. Complete the necessary documentation in wiki and code. I will be happy to assist with the wiki doc. | 06:13 |
| eranrom | 3. Adjust the java code structure accordingly (push samples and tests down the tree) | 06:13 |
| eranrom | I can do #3 | 06:14 |
| eranrom | I think that the rest are local changes. | 06:15 |
| eranrom | I am ok with first landing and then addressing these. Your call. | 06:15 |
| eranrom | Having python is really exciting! | 06:15 |
| eranrom | I am only starting to 'digest' that :-) | 06:16 |
| takashi | eranrom: thanks for summarize things. I generally agree with that. I'll check your all remaks, and confirm whether we have other important points to be fixed before landing | 06:27 |
| takashi | from my side | 06:28 |
| eranrom | takashi: sure! | 06:28 |
| kota_ | looks like python supporting get good progress :-) | 07:30 |
| takashi | kota_: :-) | 07:31 |
| takashi | kota_: I'd also like to ask your comment about these patch sets. Especially, we need to confirm the interface design about python storlet (see patch 363303) | 07:37 |
| kota_ | patch bot seems absent | 07:38 |
| kota_ | looking | 07:39 |
| kota_ | what's your concern on patch 363303? | 07:39 |
| kota_ | takashi:^^ | 07:39 |
| takashi | kota_: when I create my initial implementation about python storlet, I tried to use iterator as an interface | 07:42 |
| kota_ | what interface means? Input/OutputStreams? | 07:44 |
| takashi | kota_: Let me show you more detailed example | 07:44 |
| kota_ | takashi: yes please | 07:45 |
| takashi | In my current patch, I defined the interface like Input: (in_md: a list of dict, in_fds: a list of FileObject, out_md_files: a list of MetadataFileObject which has dump method, out_files: a list of FileObject, params: a dict) Output: None | 07:45 |
| kota_ | https://review.openstack.org/#/c/363303/15/StorletSamples/python/simple/simple.py? | 07:45 |
| takashi | kota_: yes | 07:45 |
| takashi | In that file I implemented the interface above | 07:45 |
| takashi | but in my first idea | 07:45 |
| takashi | I tried to define the interface like Input: (in_md: a list of metadata, in_iters: a list of iterators to read objects, params: a dict) Output: (out_md: a list of processed metadata, out_iters: a list of iterators to read processed object | 07:47 |
| takashi | to totally hide FileObject interface from user applications | 07:47 |
| kota_ | 2 questions for quick look | 07:48 |
| takashi | kota_: ok | 07:48 |
| kota_ | why you set the different method name between out_md_files and out_files? | 07:49 |
| kota_ | one has "dump" and the other has "write" | 07:49 |
| kota_ | that seems not consistent | 07:49 |
| takashi | kota_: good point | 07:50 |
| kota_ | ah, plus, in_md[0] is just a dict? | 07:50 |
| takashi | kota_: a dict of swift metadata | 07:50 |
| takashi | let me explain about the reason why I have dump method in MetadataFile | 07:51 |
| kota_ | yup | 07:51 |
| takashi | In my old patches, I did't implemented it and used write method inside storlet apps | 07:51 |
| kota_ | ok | 07:52 |
| takashi | However, in that case, I need to serialize metadata (I mean, call json.dumps) before I write it to outputfile | 07:52 |
| takashi | IMO, json serializing is 'a protocol stuff' which should be hidden from user applications | 07:52 |
| takashi | and that is the reason why I created a 'dump' interface in MetadataFile, which hides that serializing from applications. | 07:54 |
| kota_ | let me think | 07:55 |
| kota_ | ... | 07:55 |
| takashi | Another possible solution is creating the interface into storlet applications to just get processed metadata. (that is related to my first concern) | 07:56 |
| kota_ | though i may say crazy idea for that, | 07:57 |
| kota_ | could we make... 1. make a template class for python storlet app and all storlet apps should inherit to wrap the i/o, 2. and then, in the wrap, could we take a tuple of return value from __call__ wiht like (out_md_files, out_files) | 07:58 |
| kota_ | i know it's far from Java api though. | 07:59 |
| kota_ | wait a sec, trying to write the code image | 08:00 |
| takashi | Do you mean FileObject, by out_md_files, out_files? | 08:00 |
| takashi | Oh. paste.bin is blocked by the proxy in my company... | 08:01 |
| kota_ | ok, will do in gist | 08:02 |
| takashi | kota_: thx. I'll write down details about my first concern while you are writing sample. | 08:04 |
| takashi | kota_: So, as I mentioned I tried to let storlet application returns a list of metadata and iterators in my first draft. | 08:15 |
| takashi | kota_: However, I noticed that interface may make it difficult to implement storlet application for muti-input/multi-output | 08:15 |
| takashi | kota_: In multi-input/multi-output case, we need to realize conncurrent reading/writing among multiple iterators | 08:16 |
| *** openstackgerrit has quit IRC | 08:18 | |
| takashi | My concern is that, is it reasonable to let users to implement complited iterators (ex. the synchronization mechanism among multiple output iterators)? | 08:19 |
| *** openstackgerrit has joined #openstack-storlets | 08:19 | |
| takashi | or is it better to expose the FileObject interface to users and let them deal with the concurrecy inside storlet apps? | 08:20 |
| kota_ | takashi: https://gist.github.com/bloodeagle40234/4ca4375131e35e788d45dbc2e29467a3 | 08:23 |
| kota_ | not yet consider how we could pass the data to out_files | 08:26 |
| kota_ | sorry, i have to leave here for a while, maybe back in 1 hour. | 08:27 |
| takashi | kota_: thanks. wiil check that! | 08:29 |
| takashi | kota_: np. I'll create some example implementations. I'm asking this to you because I'm afraid the interface design may cause some problems about multi-io support. | 08:30 |
| eranrom | kota_: I think that you suggestion is in fact closer to the java API that wraps the input streams with storlet classes. For example an input stream has a get md and set md methods that hide the serialization | 08:41 |
| eranrom | kota_: perhaps I am missing your point though | 08:41 |
| eranrom | regarding the gist, I like the idea. The actual implementation has one problem though: We cannot dump the out_md after the invocation, because we need to send it in the request headers before the body | 08:43 |
| eranrom | that is written during invoke | 08:43 |
| eranrom | that is as long as we do not have 'footers'. | 08:43 |
| eranrom | takashi: I guess that your concern has to do with whether to do the multi-threading inside or outside of the storlet - right? seems like that as long as you pass our own wrappers as done in the latest patch, you can always add threading to the read, without changing the API - right? | 08:53 |
| *** openstackgerrit has quit IRC | 09:04 | |
| *** openstackgerrit has joined #openstack-storlets | 09:05 | |
| takashi | eranrom: you are right. | 09:06 |
| takashi | maybe I need to arrange my concerns before discussion... | 09:13 |
| takashi | eranrom: by the way, I checked all of your comments now, thanks | 09:15 |
| takashi | eranrom: I found one thing 'unsafe' about cancel method in StorletDaemon patch, so will fix it. | 09:16 |
| eranrom | takashi: ok, sure. | 09:20 |
| takashi | eranrom: I'll pick up your other comments as a follow-up work, but IMO there are not so big blockers, excluding that (and remaining interface dicsussion?), from my side | 09:20 |
| eranrom | takashi: ok, cool. I am thinking that I will start working on the big-tent setup.py thing. I can do the java tests and samples refactoring as part of that | 09:22 |
| eranrom | at least the pushing down one level | 09:22 |
| takashi | eranrom: makes sense | 09:24 |
| takashi | I'm currently thinking two things now, after landing current python storlet. 1. Refactor and add unit test for current agent codes. 2. Refactor directory strucrture for python codes, to use setup.py as centric install scripts for all things | 09:25 |
| kota_ | back | 09:43 |
| openstackgerrit | Doron Chen proposed openstack/storlets: make the docker registry a service https://review.openstack.org/358186 | 09:49 |
| takashi | kota_: I created a sample storlet for each interface designs: https://github.com/anonymous/d0cc2fdec90363ead171c48abdd03510 | 09:49 |
| takashi | oh sorry. wring url | 09:49 |
| takashi | koa_: https://gist.github.com/anonymous/d0cc2fdec90363ead171c48abdd03510 | 09:49 |
| takashi | kota_: ^^ | 09:50 |
| takashi | I don't know why we need to block access to gist from the company network... | 09:50 |
| kota_ | looking | 09:51 |
| takashi | My concerin is that I'm afraid returning iterator requires us to implement very complicated things in storlet application | 09:51 |
| takashi | when using multi input/output | 09:52 |
| takashi | I didn't run the code, so please see just a sample | 09:53 |
| takashi | If we return iterators from application, we need to deal with concurrency inside iterators | 09:54 |
| kota_ | yup | 09:55 |
| kota_ | i'm with | 09:55 |
| kota_ | so... | 09:55 |
| takashi | and we need a kind of syncrhonization mechanism (in that sample, I use queue for that) between output iterators | 09:55 |
| kota_ | i think, it would be better clarify the i/o patterns | 09:59 |
| kota_ | maybe 3 patterns i'm guessing | 09:59 |
| kota_ | 1. storlet app can read all data from read_fd and it can prepare whole data to write | 10:00 |
| kota_ | this is too simple, any format can be used. | 10:00 |
| kota_ | 2. storelet app app can read all data from read_fd but it wants to write outgoing data streamingly | 10:01 |
| kota_ | this may be more complex but iterator seems a possible solution. | 10:01 |
| kota_ | 3. storlet app wants to read the data streamingly and it wants also to write outgoing data streamingly | 10:02 |
| kota_ | this is too complex rather than the others. | 10:02 |
| takashi | kota_: yes | 10:02 |
| kota_ | Probably, it needs something like 'hook point' during capture/sending data. | 10:03 |
| kota_ | ok | 10:03 |
| kota_ | now, we're thinking if we could cupslelarate the outgoing data | 10:04 |
| kota_ | I think, we could accpept a couple of formats for outgoing data | 10:05 |
| kota_ | not sure which format is suitable though, | 10:05 |
| kota_ | likely StringIO? Iter? or custom class which eables to implement the *hook point* easily. | 10:06 |
| takashi | kota_: Can you show an example about hook point? | 10:07 |
| kota_ | not soon, sorry. | 10:08 |
| kota_ | just a drafting idea | 10:08 |
| takashi | kota_: ok | 10:08 |
| takashi | kota_: Do you think we can land current interface design and do that cupselarating later? | 10:10 |
| kota_ | not sure | 10:11 |
| takashi | kota_: ok. thank you for your time! I'll think more about cupselarating. | 10:15 |
| takashi | maybe I need to list up some storlet examples, based on the i/o patterns, and think how we can deal with them. | 10:15 |
| kota_ | maybe my idea is getting concrete in my head, I have to dump and do PoC somewhere... | 10:15 |
| kota_ | perhaps, I could take a time to do that...later half of this week... | 10:16 |
| kota_ | sorry | 10:16 |
| takashi | kota_: np | 10:19 |
| openstackgerrit | Eran Rom proposed openstack/storlets: Towards python-centric repo https://review.openstack.org/365610 | 12:22 |
| eranrom | Hi, I created a new blueprint for the security feature. Today I have added to it a link to a spec in the wiki: https://wiki.openstack.org/wiki/Storlets/DataSecuritySpec | 12:25 |
| eranrom | comments are welcome | 12:25 |
| takashi | eranrom: that's the detail of the blueprint you added ,right? https://blueprints.launchpad.net/storlets/+spec/get-with-storlet | 12:52 |
| eranrom | yes | 12:52 |
| eranrom | Still no patches, just spec | 12:52 |
| takashi | eranrom: I will check its detail later, but at a glance it makes much sense to me | 12:53 |
| eranrom | takashi: Thanks! | 12:53 |
| eranrom | takashi: its not urgent | 12:53 |
| takashi | eranrom: At least I currently understand its usecase, and approve it. I don't know who is an appropriate approver for bp in OpenStack projects (PTL? or the other core reviewers than proposer?), but I can be. | 12:55 |
| eranrom | takashi: thanks. I think that we are the approvers of ourselves :-) | 12:56 |
| takashi | eranrom: :-) | 12:58 |
| flaper87 | See you guys have made progress, that's awesome! | 13:06 |
| flaper87 | keep it up | 13:06 |
| takashi | flaper87: thanks! | 13:24 |
| takashi | flaper87: Do you have some time? I'd like to ask one question about copy right stuff? | 13:25 |
| flaper87 | takashi: sure, what's up? | 13:26 |
| takashi | flaper87: I'd like to make soure what you mean by 'Change the copyright header to something like', written in big-tent etherpad | 13:26 |
| takashi | flaper87: Do you mean there are some files missing header for Apache 2.0 License? | 13:27 |
| flaper87 | takashi: I meant to say that all copyright headers should look like: https://github.com/openstack/storlets/blob/master/Engine/agent/storlet_daemon_factory/daemon_factory.py#L1-L14 | 13:28 |
| flaper87 | I noticed that some had other things | 13:28 |
| flaper87 | not sure if that was fixed already | 13:28 |
| takashi | flaper87: ok. I need to recheck | 13:29 |
| flaper87 | takashi: danke :) | 13:29 |
| flaper87 | it might be that there's nothing left to do | 13:29 |
| flaper87 | but I thought I'd mention it anyway | 13:30 |
| takashi | flaper87: One thing I want to confirm here is, do we need to change the copyright to OpenStack Foundation? | 13:30 |
| flaper87 | I think the current stand is you can either add the copyright to the openstack foundation or no copyright at all and just use the apache license header | 13:30 |
| * flaper87 normally adds the copyright to the foundation | 13:31 | |
| takashi | flaper87: I found some files which has a copyright of each company even in Nova: https://github.com/openstack/nova/blob/master/setup.py | 13:32 |
| takashi | flaper87: There are some files still has a copyright of IBM in storlets repository. I'm wondering if we need to change them: https://github.com/openstack/storlets/blob/master/Engine/SCommon/src/org/openstack/storlet/common/IStorlet.java#L2 | 13:34 |
| takashi | flaper87: If we can keep it, I'll change about the license header, and would like to keep current copy right, because they are totally written by IBM. | 13:35 |
| takashi | flaper87: Does it make sense to you? | 13:35 |
| takashi | Some files added after storlet get opensourced already have OpenStack copy right in their headers | 13:36 |
| takashi | maybe we still need to think how we deal with copy right when we modify them... :-( | 13:37 |
| takashi | flaper87: sorry. I need to be offline for a while. I'll check the log later, so please add any comments here, if you have some time. :-) | 13:42 |
| takashi | flaper87: thank you for helping us! | 13:42 |
| *** takashi has quit IRC | 13:42 | |
| eranrom | flaper87: Looking at the example you placed in the etherpad (https://github.com/openstack/zaqar/blob/master/zaqar/i18n.py#L3-L14) there is a Redhat copyright (# Copyright 2014 Red Hat, Inc.) | 14:17 |
| flaper87 | eranrom: no no, ignore that | 14:17 |
| eranrom | flaper87: ok | 14:17 |
| flaper87 | eranrom: I'd recommend to update the copyrights to the openstack foundation | 14:18 |
| flaper87 | or just skip it and use the license header only | 14:18 |
| eranrom | flaper87: ok, being a former IBMer I am not sure I can actually delete that. don't want to mess with them :-) | 14:19 |
| flaper87 | hahahaha, sounds good! | 14:19 |
| flaper87 | I mena, you can keep the old one and just add OpenStack Foundation at the top | 14:19 |
| flaper87 | mean* | 14:19 |
| flaper87 | I guess that's fine *shrugs* | 14:19 |
| eranrom | flaper87: sounds right to me | 14:19 |
| openstackgerrit | Daniele Pizzolli proposed openstack/storlets: account params can be changed in installation_vars.yml https://review.openstack.org/365677 | 14:27 |
| openstackgerrit | Daniele Pizzolli proposed openstack/storlets: account params can be changed in installation_vars.yml https://review.openstack.org/365677 | 14:49 |
| openstackgerrit | Daniele Pizzolli proposed openstack/storlets: account params can be changed in installation_vars.yml https://review.openstack.org/365677 | 14:51 |
| openstackgerrit | Merged openstack/storlets: account params can be changed in installation_vars.yml https://review.openstack.org/365677 | 17:46 |
| openstackgerrit | Merged openstack/storlets: Bump up ansible for linting https://review.openstack.org/365334 | 18:02 |
| *** openstackgerrit has quit IRC | 18:03 | |
| *** openstackgerrit has joined #openstack-storlets | 18:04 | |
| openstackgerrit | Merged openstack/storlets: Ignore only BADFD in _safe_close https://review.openstack.org/365076 | 18:06 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!