Tuesday, 2013-09-17

*** kgriffs is now known as kgriffs_afk00:08
*** oz_akan_ has joined #openstack-marconi00:09
*** malini is now known as malini_afk00:25
*** amitgandhi1 has joined #openstack-marconi00:36
*** amitgandhi has quit IRC00:40
*** malini_afk is now known as malini00:44
*** oz_akan_ has quit IRC00:48
*** oz_akan_ has joined #openstack-marconi00:49
*** oz_akan_ has quit IRC00:53
*** oz_akan_ has joined #openstack-marconi00:53
*** kgriffs_afk is now known as kgriffs00:59
openstackgerritMalini Kamalambal proposed a change to openstack/marconi: Fix Functional Tests  https://review.openstack.org/4685101:00
*** kgriffs is now known as kgriffs_afk01:13
*** vkmc has quit IRC01:20
*** jergerber has quit IRC01:23
*** jergerber has joined #openstack-marconi01:23
*** jergerber has quit IRC01:43
*** amitgandhi1 has quit IRC01:56
*** malini has left #openstack-marconi01:57
*** oz_akan_ has quit IRC01:57
*** kgriffs_afk is now known as kgriffs02:04
*** kgriffs is now known as kgriffs_afk02:14
*** ayoung is now known as ayoung-ZzZzZzZ02:57
*** ayoung-ZzZzZzZ has quit IRC03:00
*** kgriffs_afk is now known as kgriffs03:05
*** kgriffs is now known as kgriffs_afk03:14
*** kgriffs_afk is now known as kgriffs04:05
*** rektide has quit IRC04:09
*** kgriffs is now known as kgriffs_afk04:14
*** kgriffs_afk is now known as kgriffs05:05
*** kgriffs is now known as kgriffs_afk05:15
*** kgriffs_afk is now known as kgriffs06:06
*** russell_h has quit IRC06:09
*** al-maisan has quit IRC06:09
*** al-maisan has joined #openstack-marconi06:09
*** openstack has joined #openstack-marconi06:18
*** gleicon___ has joined #openstack-marconi06:20
*** gleicon__ has quit IRC06:23
*** gleicon___ is now known as gleicon__06:24
*** oz_akan_ has joined #openstack-marconi06:29
*** oz_akan_ has quit IRC06:45
*** gleicon__ has quit IRC06:53
*** gleicon__ has joined #openstack-marconi06:53
*** kgriffs_afk is now known as kgriffs07:06
*** kgriffs is now known as kgriffs_afk07:16
*** ykaplan has joined #openstack-marconi07:27
*** oz_akan_ has joined #openstack-marconi07:42
*** oz_akan_ has quit IRC07:46
*** gordonsim has joined #openstack-marconi07:50
*** kgriffs_afk is now known as kgriffs08:07
*** cthulhup has joined #openstack-marconi08:11
cthulhupgreetings all08:12
*** kgriffs is now known as kgriffs_afk08:16
*** ykaplan has quit IRC08:49
*** flaper87|afk is now known as flaper8709:03
*** kgriffs_afk is now known as kgriffs09:07
*** kgriffs is now known as kgriffs_afk09:17
openstackgerritA change was merged to openstack/marconi: chore: Update Falcon dep to allow version 0.1.7  https://review.openstack.org/4677509:33
*** kgriffs_afk is now known as kgriffs10:08
*** ykaplan has joined #openstack-marconi10:11
*** kgriffs is now known as kgriffs_afk10:17
*** oz_akan_ has joined #openstack-marconi10:54
*** oz_akan_ has quit IRC10:58
*** kgriffs_afk is now known as kgriffs11:08
*** kgriffs is now known as kgriffs_afk11:18
*** whenry has quit IRC11:25
*** tedross has joined #openstack-marconi11:35
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Update pbr dependency to latest  https://review.openstack.org/4626011:36
*** kgriffs_afk is now known as kgriffs11:38
*** kgriffs is now known as kgriffs_afk11:48
*** ykaplan has quit IRC12:05
*** cppcabrera has joined #openstack-marconi12:15
cppcabreraGood morning. :)12:19
*** ykaplan has joined #openstack-marconi12:20
flaper87cppcabrera: GOOOD MORNING12:21
flaper87:D12:21
cppcabreraflaper87: o/12:21
cppcabrera:)12:21
flaper87how are you doing?12:21
cppcabreraA little groggy, but otherwise pretty good. How about you?12:22
* flaper87 just learnt a new word: groggy12:24
flaper87doing well, at the office. We'll have our monthly PythonMilan meetup today12:24
cppcabrerahehe12:25
cppcabrerasweet! What kinds of topics will be covered at the meetup? Do they plan out talks and such before hand at this meet up?12:25
flaper87cppcabrera: yeah, today we'll talk about OpenStack12:27
flaper87guess who's talking12:27
* cppcabrera points at flaper8712:27
flaper87Actually, I'm the one organizing these meetups, we use Red Hat's Milan office12:27
cppcabrera:P12:27
flaper87:D12:27
cppcabrerahaha12:28
flaper87http://www.meetup.com/Python-Milano/events/133133302/12:28
flaper87cppcabrera: talks poll: http://www.meetup.com/Python-Milano/polls/867752/12:28
cppcabreraI can almost make out the meaning of most of the words on that page. Approximation by Spanish helps there. :)12:29
cppcabreraOpenstack sure got a lot of votes,12:30
flaper87:D12:31
flaper87YEAHHH!!!12:31
cppcabreraso you're going to be talking about all of openstack at the meetup? That's a pretty big topic, flaper87!12:34
cppcabreraFitting that into an hour sounds like a challenge.12:34
flaper87yeah, it is. I'll try to cover the main modules and then say something about openstack's community12:36
cppcabrera-infra needs a shout out. It's a pretty unique "feature" of the openstack community.12:37
* flaper87 is restructuring marconis source code12:38
flaper87what a painful process12:39
*** kgriffs_afk is now known as kgriffs12:39
flaper87cppcabrera: +112:40
cppcabrerarestructuring as per: marconi/{api, proxy, notifications} ?12:40
flaper87cppcabrera: yeah12:40
flaper87mmh, wait12:40
flaper87wasn't it queues ?12:40
cppcabreraIt was. :P12:40
cppcabreraI'd forgotten the name.12:40
flaper87ah ok12:40
flaper87phew12:40
cppcabreralol12:40
flaper87cppcabrera: re merging proxy patches12:41
cppcabreraI was just about to ask! :D12:41
flaper87I agree with you, I'd like to see some of the comments in the first patch addressed before merging it, though12:41
flaper87they seem to be very simple12:41
flaper87and very easy to forget too12:42
flaper87:D12:42
flaper87at least for me12:42
flaper87thoughts?12:42
flaper87by first patch, I mean "marconi proxy" the one I already reviewed12:42
cppcabreraSo that's - API version in routes, body_encoded, and... ensure_ascii12:42
flaper87cppcabrera: I think so, yes12:43
cppcabreraquick question on body_encoded - how is that meant to be used? AFAICT, I've had to do ```response.body = <thing>, response.body = response.body_encoded```, but it seems that it might be implicitly performed.12:44
flaper87you just have to set response.body with you're non encoded body12:46
flaper87falcon will do the rest12:46
flaper87right before sending the request12:46
flaper87and if you need your encoded body12:47
flaper87you can use response.body_encoded12:47
* flaper87 is using Enter as a separator12:47
flaper87:P12:47
*** kgriffs is now known as kgriffs_afk12:48
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Change Marconi's codebase structure  https://review.openstack.org/4693912:49
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Change Marconi's codebase structure  https://review.openstack.org/4693912:49
flaper87there it goes12:49
flaper87lets hope for the best12:49
flaper87:D12:49
flaper87that change shouldn't break marconi.proxy12:50
flaper87actually, it ******SHOULD****** all work12:50
flaper87but, you know...12:50
*** ykaplan has quit IRC12:51
cppcabrerabrb12:51
flaper87sure12:51
flaper87gordonsim: ping12:54
gordonsimflaper87: pong!12:56
flaper87gordonsim: yo!12:57
flaper87gordonsim: do you have some minutes to discuss Marconi's client API?12:57
flaper87tedross: ^12:57
gordonsimflaper87: sure12:57
flaper87awesome12:57
flaper87gordonsim: have you looked at https://wiki.openstack.org/wiki/Marconi/PythonClient12:58
flaper87?12:58
gordonsimflaper87: very briefly only12:59
flaper87gordonsim: https://etherpad.openstack.org/marconi-client-api12:59
flaper87lets mock something up there12:59
flaper87gordonsim: feel free to change things in that pad and write your comments there13:03
*** ykaplan has joined #openstack-marconi13:04
*** amitgandhi has joined #openstack-marconi13:09
*** amitgandhi has quit IRC13:10
*** amitgandhi has joined #openstack-marconi13:10
*** jergerber has joined #openstack-marconi13:13
*** oz_akan_ has joined #openstack-marconi13:13
cppcabreraflaper87: Ah, so it is mostly implicit. Cool. (body_encoded) :)13:13
*** oz_akan_ has quit IRC13:15
*** oz_akan_ has joined #openstack-marconi13:16
flaper87cppcabrera: gordonsim thanks for your inputs guys. I think that's enough info to start a POC of the API and see how it goes13:23
cppcabrera+1 - thanks gordonsim!13:24
flaper87gordonsim: are you signed up in review.openstack.org ?13:24
flaper87if you give me your username, I'll make sure to keep you in the loop13:24
flaper87and add you to the reviews13:24
cppcabreraI added the etherpad session link to the client bindings wiki page.13:26
flaper87cppcabrera: thanks, big +113:27
cppcabreranp~13:28
gordonsimflaper87: no, I should sign up now13:29
gordonsimflaper87: all: thanks for the chat!13:29
flaper87cppcabrera: https://review.openstack.org/#/c/46939/13:35
flaper87thoughts?13:35
cppcabreraChecking it out. :)13:35
flaper87awesome13:36
flaper87thanks13:36
cppcabreraFirst thought - Jenkins is happy about it, That's usually good news. :D13:36
flaper87I tried to make sure it doesn't conflict with your proxy patches13:36
flaper87it shouldn't13:36
flaper87please, double check :D13:36
cppcabreraIf it does, I'll live. I'm happy about the restructuring, overall.13:36
cppcabrera:D13:36
flaper87:D13:36
flaper87glad you said that13:37
flaper87:D13:37
openstackgerritA change was merged to openstack/python-marconiclient: Update .gitreview for incubation org move  https://review.openstack.org/4662913:39
flaper87guys, client patches need some care https://review.openstack.org/#/q/status:open+project:openstack/python-marconiclient,n,z :D13:39
flaper87kgriffs_afk: ^13:39
*** kgriffs_afk is now known as kgriffs13:39
cppcabreraflaper87: +2 for restructuring13:41
cppcabreraAfter fighting with oslo.config yesterday, all of the configuration details suddenly make sense to me.13:43
cppcabreraI see cfg.Stropt, and I **know** what's being talked about.13:43
amitgandhiamazing what a good nights sleep can do13:43
cppcabreraamitgandhi: Ha, I have been sleeping slightly better this week than last. :P13:44
* cppcabrera looks at client auth patch13:45
cppcabreraInteresting - authenticate returns a Client instance13:46
cppcabreraI wasn't expecting that.13:46
cppcabreraWorks for me.13:46
cppcabreraauthenticate :: version -> request -> client13:47
flaper87cppcabrera: wait, auth is in WIP13:47
flaper87it's actually a draft13:47
flaper87and you are the only one that can see it :D13:47
flaper87but yes, that's the overall idea13:48
flaper87I guess13:48
* cppcabrera can see the special WIP secrets13:48
*** kgriffs is now known as kgriffs_afk13:49
cppcabreraAs far as the auth patch, I'm happy with all of it so far, though I'm skeptical about utils.env and how that brings ceilometerclient in as a dep.13:49
cppcabreraI'm currently reviewing that patch of code for obvious py3k breaks.13:49
zyuanflaper87: is absolute import allowed?13:50
flaper87cppcabrera: I removed the ceilo thing, I just didn't see it before submitting the patch13:51
flaper87:D13:51
flaper87zyuan: yup13:51
flaper87relative's aren't though13:51
flaper87relatives13:51
zyuanoh... i used wrong word13:52
zyuani mean import .base13:52
zyuanimport ..unit.xxxx13:52
flaper87no, relative import are not allowed13:52
zyuanbecause tests is not a public module13:52
zyuanthen how can i import ..?13:52
zyuaneven C support local only import, why not python13:53
zyuanwhy not here13:53
flaper87why do you need to import things from tests ?13:53
zyuanbecause im doing forwarding tests13:53
zyuanproxy -> wsgi13:53
* cppcabrera catches up after intense WIP reviewing13:55
flaper87mmh, not sure I follow. I mean, I know what you're trying to test but I don't know how are you trying to test it13:55
*** ayoung has joined #openstack-marconi13:57
zyuani launched multiple marconi server as nodes in test, then test proxy with mock. the proxy->wsgi forwarding part is suppose to be testable by pulling wsgi test13:58
zyuanor i can do from tests.unit.transport import wsgi as wsgi_test13:59
zyuan:(13:59
flaper87TBH, I would try to keep them separate. Wsgi tests are for wsgi v1, if / when we'll add a new version for the API this can become an issue since proxy tests will be tight to the wsgi v1 tests (assuming I understood exactly what you're doing)14:01
zyuanflaper87: this is another issue. no matter how you separate, it still make sense to pull backends' test for proxy14:01
flaper87mmh, I don't fully agree with that, however, I think from tests.unit. etc is the rigth import path to use14:03
*** malini has joined #openstack-marconi14:06
*** kgriffs_afk is now known as kgriffs14:06
kgriffso/14:08
kgriffsgood morning14:09
kgriffsgood afternoon14:09
kgriffsflaper87: nice job breaking everyone's patches. ;)14:10
openstackgerritA change was merged to openstack/marconi: Change Marconi's codebase structure  https://review.openstack.org/4693914:14
flaper87kgriffs: :D14:15
*** kgriffs is now known as kgriffs_afk14:15
flaper87kgriffs_afk: sorry about that :(14:15
cppcabrerabrb demos~14:18
cppcabrerakgriffs_afk: morning. :)14:24
*** kgriffs_afk is now known as kgriffs14:27
*** kgriffs is now known as kgriffs_afk14:30
*** kgriffs-test has joined #openstack-marconi14:31
kgriffs-testtesting 1,2,3...14:31
*** kgriffs_afk is now known as kgriffs14:31
*** kgriffs-test has quit IRC14:31
* flaper87 just discovered kgriffs has unit-tests integrated14:32
flaper87at least they all pass14:32
flaper87:D14:32
*** ykaplan has quit IRC14:41
cppcabreralol14:43
zyuancppcabrera: found another issue14:45
*** ykaplan has joined #openstack-marconi14:45
zyuanbefore sending requests to marconi, you sometimes need to decode json14:46
cppcabreraoh?14:46
zyuanthere is no defending code for malformed json14:46
zyuanin proxy14:46
kgriffscppcabrera: if all you need to do is verify the JSON, then you can use the code from falcon - it is much faster than actually parsing into a dict14:46
kgriffssorry, not falcon14:47
kgriffsrse14:47
zyuankgriffs: different cases14:47
kgriffszyuan: what is the argument for not just passing through the raw body and letting the backend deal with it?14:47
zyuanif doing forwarding, there is no need to verify; just send the raw request as is14:47
zyuanif doing proxy processing, just need to catch an exception14:48
kgriffsoic14:48
zyuanso called "defending"14:48
zyuanre verifying api input14:48
zyuanerr, i don't know which is faster or whether it's really needed14:49
zyuanif we need to prevent ourself from crashing the server, there are 1 or 2 cases missing (i commented in test, hopefully)14:49
*** kgriffs is now known as kgriffs_afk14:50
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: WIP: add mongo driver for marconi proxy storage  https://review.openstack.org/4637514:54
cppcabreraThere, now I can go fix the minor marconi proxy issues.14:55
cppcabrerazyuan: hmmm, so re: JSON handling, what route/resource are you talking about?14:56
zyuanqueue metadata14:56
cppcabreragotcha14:57
zyuanbut wait...14:57
zyuananyway, i'm not very sure how metadata is handled in proxy14:57
zyuanbut some "bad" metadata tests failed14:57
zyuanlet me post the WIP patch on gerrit14:58
cppcabreraclient -> proxy -> (proxy stores queue name/metadata, generates location) -> marconi14:58
*** kgriffs_afk is now known as kgriffs14:58
zyuancppcabrera: yea, i know the rough steps, but i'm not sure which step caused the problem14:59
cppcabreraAhh, I see.14:59
zyuanah!!15:00
zyuandependency15:00
zyuanhope it works....15:00
*** kgriffs is now known as kgriffs_afk15:01
*** ametts has quit IRC15:01
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: test(proxy): forwarded endpoints tested  https://review.openstack.org/4696315:01
openstackgerritZhihao Yuan proposed a change to openstack/marconi: test(proxy): partition related endpoints tested  https://review.openstack.org/4636215:01
openstackgerritZhihao Yuan proposed a change to openstack/marconi: test(proxy): catalog tested  https://review.openstack.org/4676915:01
zyuanah ha, ah ha, ah ha ha ha....15:01
zyuancppcabrera: https://review.openstack.org/#/c/46963/15:02
cppcabreralots of reviews15:02
*** kgriffs_afk is now known as kgriffs15:02
zyuannot...15:02
zyuanrun dependencies15:02
zyuanwrong*15:02
zyuanT_T15:02
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: marconi proxy  https://review.openstack.org/4390915:03
cppcabreraI'm pretty sure I broke my own dependency chain with ^^15:06
cppcabreraWell, marconi-proxy patch 1 is ready to merge. :P15:06
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: marconi proxy (v1, health)  https://review.openstack.org/4435615:07
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: marconi-proxy forwarding  https://review.openstack.org/4436415:07
zyuanlet me run test15:07
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: feat: define interfaces for proxy storage  https://review.openstack.org/4632815:07
cppcabrerarebasing ^^15:07
flaper87cppcabrera: LGTM15:08
flaper87thanks a lot!15:09
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: WIP: add mongo driver for marconi proxy storage  https://review.openstack.org/4637515:12
cppcabreranp, flaper87. Thank you!15:13
cppcabreraSo let's see, what I'd like to have merged soon includes... (fetches list)15:13
cppcabreraThis: https://etherpad.openstack.org/marconi-proxy-issues15:13
cppcabreraall the way up to the proxy storage interface in the long chain of patches.15:14
cppcabreraincluding proxy storage interface, but not the mongo driver.15:14
cppcabreraAll this patch juggling has made me have to get pretty good @ 'git review'. :P15:15
cppcabrera```git review -d``` became a good friend of mine over the past week, heh.15:15
cppcabrerakgriffs, flaper87: ^ (re: merging proxy patches)15:15
flaper87cppcabrera: hehehe, you could also use rebase + edit mode :)15:20
flaper87cppcabrera: I'll focus on reviewing those patches in the next few days. Please, consider that I'll be with limited internet access tomorrow and the day after15:21
flaper87but I'll do my best15:21
cppcabrerathanks!15:22
cppcabrerathe mongo driver patch is fixing most of the design pain points.15:22
cppcabreraIt's where I realized most of what won't work in the long run, e.g., Redis in code, no interface to storage, round robin in storage...15:23
zyuannow functional test went back to non-working states on both of my freebsd and fedora machies...15:23
zyuanwith nosetests...15:24
openstackgerritZhihao Yuan proposed a change to openstack/marconi: test(proxy): partition related endpoints tested  https://review.openstack.org/4636215:25
openstackgerritZhihao Yuan proposed a change to openstack/marconi: test(proxy): catalog tested  https://review.openstack.org/4676915:28
flaper87http://www.sandywalsh.com/2013/09/notification-usage-in-openstack-report.html15:29
kgriffsinteresting note about swift15:31
cppcabreraI like the note on oslo adoption, and why that's currently pretty low.15:32
*** ykaplan has quit IRC15:32
cppcabreraI agree with the sentiment - if I can't pip install a particular oslo component, it makes me pretty uncomfortable using thye copy-paste approach.15:32
cppcabrerathanks for sharing, flaper8715:33
cppcabrerawe're likely to end up in the swift boat, since we expect a very high transactional volume.15:34
al-maisanflaper87: that looks interesting indeed15:34
cppcabrera"Which Notifications Should Services Emit?" is my favorite section as a developer of openstack services. It makes it clear what's expected.15:34
flaper87agreed, I can see Marconi emitting notifications, however, I'd prefer to wait 'til ceilo's support for statsd15:35
flaper87or something like that15:35
al-maisancppcabrera: yeah .. identifying what is "interesting" will be interesting ;)15:35
zyuancan someone here?15:37
zyuanhelp15:37
zyuanhttp://logs.openstack.org/69/46769/6/check/gate-marconi-python26/d18b147/console.html15:37
zyuanwhy i can't import bootstrap?15:37
zyuanit works on my machine, with tox15:37
zyuanwhat's wrong with jenkins?15:38
flaper87zyuan: from marconi.queues import bootstrap15:38
flaper87I guess you've some .pyc around15:38
zyuanmoved?15:38
zyuan...15:38
flaper87duuuude, you have to read git logs before rebasing :D15:39
zyuani'm rebsing upon cppcabrera's change -_15:39
cppcabreraDid I forget to change something... ? >.>15:40
zyuannothing...15:40
cppcabreraAh, k. I knew I had to handle the project restructure in the proxy-mongo driver since I use marconi.queues.storage.mongodb.utils.15:41
flaper87I just found 2 imports pointing to the old path in cmd15:42
flaper87submitting a patch now15:42
cppcabrerawhich makes me think it might not hurt to at some point move common storage tools into marconi.common.storage.*15:42
cppcabrera+1 flaper8715:42
openstackgerritFlavio Percoco proposed a change to openstack/marconi: Fix old import paths  https://review.openstack.org/4697315:42
cppcabreraAlso, am I the only one that frequently typos marconi as macaroni? My fingers just want to type out the food. :P15:42
openstackgerritZhihao Yuan proposed a change to openstack/marconi: test(proxy): catalog tested  https://review.openstack.org/4676915:42
flaper87kgriffs: could you set yourself as approver here? https://blueprints.launchpad.net/marconi/+spec/change-marconi-structure15:42
flaper87danke :)15:43
* flaper87 moves his fingers as a jedi15:43
zyuanafter reviewed the 1st proxy patch15:44
zyuanpls take a look at the twotest patches depending on it15:44
cppcabreraflaper87: Could you handle the errors in HACKING.rst, as well?15:44
cppcabrera+107 (import marconi.transport), +115 (import marconi.transport.wsgi as wsgi)15:44
flaper87zyuan: kk15:45
flaper87cppcabrera: will fix those as well15:45
cppcabrerawoot15:45
flaper87cppcabrera: thanks for noticing15:45
cppcabreraI figured I'd do a 'grin' session since we're on the topic now. :D15:45
zyuani stop pushing 3rd test patch before the first 2 being merged15:45
* flaper87 heading to a meeting15:45
flaper87brb15:45
kgriffsfalper87: done; i need to get in the habit of doing that. :p15:47
*** flaper87 is now known as flaper87|afk15:47
*** flaper87|afk is now known as flaper8715:55
*** ykaplan has joined #openstack-marconi15:57
zyuananother breaking change is 'nodes' -> 'hosts'15:58
*** ykaplan has quit IRC16:00
cppcabreragood point, zyuan16:02
cppcabreraI'm changing the entire world with the mongo patch. :P16:02
zyuan87            'nodes': data['hosts'],16:03
zyuan-_-|||16:03
*** amitgandhi has quit IRC16:03
maliniflaper87: do you know which openstack project uses testscenarios ? I checked tempest & nova so far.16:16
cppcabreramalini: It seems neutron might use testscenarios at this point: http://lists.openstack.org/pipermail/openstack-dev/2013-February/005457.html16:24
cppcabreraI found that while googling about.16:24
cppcabreraThere's also mention of another module, testresources16:24
malinithanks cppcabrera..I am doing some more reading on testscenarios.16:26
maliniLooks like it doesnt support Py3  yet..16:27
maliniits on a 'wishlist'16:27
cppcabrera+1 malini - I'd love to hear if you find out some really cool things about either.16:27
cppcabrerapython 3 support... :(16:27
*** flaper87 is now known as flaper87|afk16:49
cppcabreraI noticed that full screen emacs looked pretty cool, and that I had a good reason to use >2 windows for once, so this happened: https://twitter.com/cppcabrera/status/38001025273469337616:49
*** amitgandhi has joined #openstack-marconi16:54
zyuancppcabrera: there is nothing run with find_one and find.. limit(1) does not help.17:27
zyuanthe reason why benchmark showed that find_one is slow is not find_one itself is slow; the slow thing is the usage.17:28
zyuanjust use find_one instead of catching the exception by yourself17:28
kgriffsguys, what do you think about allowing a message to live up to 60 seconds after it expires (meaning, it could also be returned in listings)?17:35
kgriffsi would like to use mongo's TTL in my patch that is rendering the external gc unecessary17:35
kgriffsthat means I need to add a Date type field to the messages again17:36
cppcabreraI think that as long as we state that messages are garbage collected using a best effort algorithm, and we explicitly note that they may appear in listings, it';s cool.17:36
kgriffsand I was thinking to just stop filtering by 'e' so we don't have to deal with Date at all in our code17:36
kgriffs(other than setting on new messages)17:36
kgriffsflaper87: ^^^17:37
kgriffsmegan_w: ^^^17:37
zyuani +1 for live longer time than TTL17:37
cppcabrera+1 from me, btw17:37
cppcabrerafind vs find_one with an empty queue: http://paste.openstack.org/show/47173/17:38
kgriffslooks like both are doing find()17:39
kgriffs?17:39
cppcabreraoops17:39
cppcabreraI had the proper data in my terminal and I botched the paste. :P17:39
cppcabreraTrying again, with more data points, too.17:40
cppcabrerakgriffs: http://paste.openstack.org/show/47174/17:43
zyuancppcabrera: but after you catch the exception, they are the same17:43
cppcabreraThe exception is being caught above. ^^17:43
cppcabreraThat's included in the benchmark time.17:44
cppcabreraUnless I'm missing something?17:44
zyuanreturn?17:44
cppcabreraAhh, I'll try that.17:45
zyuannot needed now17:46
zyuanfind_one's source code17:46
zyuanif spec_or_id is not None and not isinstance(spec_or_id, dict):17:46
zyuan    spec_or_id = {"_id": spec_or_id}17:46
zyuanone more branch17:46
cppcabreraInteresting... changing 'pass' -> 'None' sped this up by a lot. >.>17:46
zyuanbut... serious, not worth17:46
cppcabreralol17:46
cppcabreraYeah, find_one() is good enough, IMO.17:46
cppcabreraEspecially if I plan to use the result.17:47
cppcabreraI wanted to dispel rumors that there's a massive difference between find and find_one that I'd heard while reading year old articles about mongo.17:47
cppcabrera+1 for the nitpick, btw, zyuan.17:47
cppcabreraa::b vs a :: b17:48
cppcabreraI like a :: b, too.17:48
cppcabrerahaha17:48
zyuanhow i install pbr?17:49
zyuanpip install -U pbr expands egg into marconi/17:50
zyuanbut i can't import?!17:50
zyuansame as d2to117:51
zyuanok, need to quite venv and install again...17:52
openstackgerritKurt Griffiths proposed a change to openstack/marconi: Update pbr dependency to latest  https://review.openstack.org/4626018:03
*** tedross has quit IRC18:06
*** ametts has joined #openstack-marconi18:09
kgriffscppcabrera: re those patches you want merged18:21
kgriffscppcabrera: ping18:21
cppcabrerakgriffs: pong18:21
cppcabrerathey're a messy bunch, and I've been working to clean up the poor design decisions in the mongo driver patch.18:21
kgriffshey, so I am thinking I will just ninjapprove these, with the caveat that we will do a full review/audit of the entire proxy in future patches18:22
cppcabrera+1 kgriffs - that was my hope and my promise18:22
zyuanwow, i love this:18:22
kgriffswe can do it this time since all the code is segregated into a new dir18:22
zyuan'{:.2%}'.format(0.132)18:22
*** tedross has joined #openstack-marconi18:22
zyuankgriffs: we did a lot18:23
cppcabrerakgriffs: agreed - if this touched main marconi logic, I would've worked around it all somehow.18:23
zyuanand we collect things need to improve in a etherpad18:23
cppcabreraor I should, marconi-queues logic.18:23
zyuanhttps://etherpad.openstack.org/marconi-proxy-issues18:23
zyuanwe need to resolve dependency first18:23
zyuanhopefully:18:24
cppcabrera*should -> should say18:24
zyuanproxy1 <- test1 <- test2 <- proxy 2 <- proxy 3 <- test3 <- proxy storage1 <- proxy mongo <- new test18:24
zyuancurrently the dependency is stucked18:25
kgriffscppcabrera: as I come across issues, let me just ping you18:26
kgriffshttps://review.openstack.org/#/c/44356/8/marconi/proxy/app.py18:26
kgriffsproxy should return it's own health response (don't send to backend)18:26
cppcabreranoted: etherpad'd18:26
zyuankgriffs: we can test it :)18:27
openstackgerritA change was merged to openstack/marconi: feat: marconi proxy  https://review.openstack.org/4390918:27
zyuankgriffs: https://review.openstack.org/#/c/46362/1418:28
kgriffscppcabrera: obviously we can't make this a habit. how will you avoid this situation in the future?18:28
cppcabreraTwo ways, kgriffs:18:28
cppcabrera1. I've learned about interface driven development this round, and how powerful it can be towards avoiding encoding hard requirements too early. I'll be using this more later.18:29
kgriffs+118:29
cppcabrera2. Going of (1), I'll prototype using that interface to see if that's enough.18:29
cppcabrera(2) is most of what I'm doing now - it's an amazing way to test my assumptions.18:29
kgriffsok, sounds good. prototyping can be really helpful when there are a lot of unknowns18:29
zyuankgriffs: why this can't be a "habit"?18:30
zyuanwhy we can't use marconi repo for prototyping?18:30
zyuanproto - test refinement, that looks agile18:30
cppcabreraBecause others will eventually come to depend on openstack/marconi to be as stable as possible.18:31
zyuaninterface -> impl, that's not18:31
zyuancppcabrera: we have release plan18:31
kgriffscppcabrera: https://review.openstack.org/#/c/44364/6/marconi/proxy/resources/forward.py18:31
openstackgerritA change was merged to openstack/marconi: feat: marconi proxy (v1, health)  https://review.openstack.org/4435618:31
kgriffsI was wondering if there's something we could add to Falcon to DRY having to do all these responders that only forward stuff18:32
zyuankgriffs: we added, i suppose18:32
cppcabrerakgriffs: flaper87|afk mentioned using a ForwardingMixin class.18:32
cppcabreraThat could DRY up those responders.18:32
zyuandon't forget https://review.openstack.org/#/c/46362/ and https://review.openstack.org/#/c/46769/18:33
zyuan(smile18:34
kgriffszyuan: yep, working my way down the list18:34
kgriffscppcabrera: so, the only thing there is that falcon now has to make http client requests itself18:40
cppcabrerayou mean regarding the Mixin class, kgriffs?18:43
kgriffsno18:43
kgriffsI think this is better solved in Falcon18:43
kgriffshttps://etherpad.openstack.org/marconi-proxy-issues18:43
kgriffsmake sense?18:43
cppcabreraah18:44
cppcabreraI see now.18:44
cppcabreraI was missing the context.18:44
kgriffsso, the single proxy func handles all forwards, and Falcon takes care of policing the http methods (get, post, put) per route18:45
cppcabreraHmm...18:46
zyuanproxy & pre-post processing18:46
kgriffsbrb18:48
cppcabrerak18:50
cppcabrerazyuan: +118:50
cppcabreraI see what you're saying.18:50
cppcabreraGiven both optional pre/post hooks + core proxy functionality.18:51
* cppcabrera goes to etherpad18:51
cppcabrerakgriffs, zyuan: I'm going to let the forwarding API thought sit on the back burner for a bit and return to the storage implementation.19:00
cppcabreraI'm fond of api.forward and api.route as names, and I'm on the wall about accepting pre/post closure arguments.19:00
*** cthulhup has quit IRC19:05
kgriffscppcabrera: how is the cached metadata to be used?19:11
kgriffshttps://review.openstack.org/#/c/44364/6/marconi/proxy/resources/metadata.py19:11
zyuanwell, at least i see where the bug (bad json not handled) is...19:14
*** gordonsim has quit IRC19:14
zyuankgriffs: i suggest to merge this first, so that i can change it in my forward test patch19:16
cppcabrerakgriffs: cached metadata is required for GET /v1/queues19:17
*** ayoung has quit IRC19:17
cppcabreraSpecifically, GET /v1/queues?detailed=true19:17
zyuanbut not /v1/queues/q/metadata?19:17
cppcabreraThat, too. :P19:18
cppcabreraI knew I was missing a route.19:18
cppcabreraWell, actually...19:18
cppcabreranot the latter case.19:18
cppcabreraOnly for listings.19:18
cppcabreraThe latter case can be forwarded.19:18
cppcabreraBut only the proxy knows about all the queues registered under a single project.19:18
cppcabreraTherefore, detailed listings require caching metadata.19:19
zyuannoted on etherpad19:19
kgriffsoh, so you are trying to avoid querying all backends and merging the listings?19:19
kgriffsthe proxy knows which backends are associated with which projects, so it could just query those backends, then merge the results19:22
cppcabrerayup, kgriffs.19:22
cppcabreraIt sounded very expensive compared to maintaining that data in the catalogue.19:22
*** tedross has quit IRC19:23
kgriffsrelatively, yes19:23
kgriffsi suppose metadata shouldn't take up much space in the catalog/cache19:23
cppcabreragenerally, it shouldn't. I feel the real cost is one of maintenance - keeping the marconi-queues listing consistent with the marconi-proxy implementation.19:24
*** malini is now known as malini_afk19:24
kgriffsyeah19:24
kgriffsI have a feeling people won't be listing queues very often19:25
kgriffsso it might be YAGNI19:25
kgriffslatency is less important there too19:25
zyuancontrol panel will19:25
zyuanoh, yea19:25
zyuanthen discard metadata in proxy storage?19:26
kgriffswell, i guess so, but nothing like the volume we expect to see though clients listing messages19:26
zyuani don;t know how front-end guys going to implement this19:26
zyuanbut if i can form an "attack" with refreshing control panel....19:27
zyuanthat's not good19:27
kgriffsmy preference would be to make n requests - i think the proxy can probably do them in parallel if we want using eventlet or gevent19:27
cppcabrera+1 for eventlet19:27
kgriffsjust ask for a queue listing for the project from each partition19:28
kgriffsthen merge sort19:28
cppcabreraHmmm...19:28
kgriffsmake an educated guess on how many to ask from each partition19:28
kgriffsso you get the page limit19:29
kgriffsif you get a few less, no big deal, as long as the "next" href is correct19:29
cppcabreraI like the idea for the sake of DRYness.19:30
kgriffsDRY data19:30
cppcabreraHowever, I'd like to revisit that in a later patch. How about after getting Hcaching through? Thoughts?19:31
kgriffsi suppose you could cache metadata for a short period of time, but whatever you do, you want to avoid having 2 sources of truth19:31
kgriffsno problem, I am just making sure we get these things on etherpad as I go along19:31
kgriffsspeaking of which… :D19:31
cppcabreraNoted. :)19:31
kgriffsbtw, some srlsy good hacking music: http://www.youtube.com/watch?v=LadRcupWDuI19:32
cppcabrera+1 kgriffs19:32
cppcabreraThat Tron Legacy OST is sweeeett19:33
kgriffsanything you want to point out on this one?19:33
kgriffshttps://review.openstack.org/#/c/4632819:33
kgriffscan we rename this to "errors"19:34
kgriffshttps://review.openstack.org/#/c/46328/3/marconi/proxy/storage/exceptions.py19:34
kgriffs?19:34
kgriffs(in a future patch)19:34
kgriffsI have it on my todo list to rename similar modules in queues19:34
cppcabreraa lot is changing in that interface. :x19:35
cppcabreraI've done some renaming, some shifting of arguments.19:35
kgriffskk19:35
cppcabreraIt's all included in the mongo implementation patch.19:36
kgriffsI added a note about "errors" to the etherpad19:36
cppcabrera+119:36
cppcabreraexceptions -> errors19:36
cppcabreraI did a git oops earlier today. I had implemented a simple RoundRobin class and accidentally nuked it before commiting it to any of my trees. :P19:36
cppcabreraReimplementing now.19:36
cppcabrera"Plan to throw one away", haha19:37
zyuangit stash is your friend19:37
zyuankgriffs: we don't need to merge the interface patch for now; that one is innocent...19:38
cppcabrerathanks for all your help with the proxy, zyuan, kgriffs, flaper87|afk.19:39
kgriffsno problem. That's be $250,00019:40
zyuan............19:40
cppcabrerawell, there goes my everything. you broke the bank, kgriffs. :P19:40
zyuanadd_proxy -> add_agent19:40
zyuanit reminds me MTA -- mail transfer agent, which does forwarding19:41
cppcabreraHmm, considering round-robin, it'd be cool if I could implement that as a functional closure that takes advantage of itertools.cycle... hmm...19:41
kgriffswe could just overload add_route, i guess19:42
zyuanthat one is COMPLEX19:42
kgriffsit would check whether it was given a class or a function19:43
zyuangive it another name pls. CS teacher tells me, one function do one thing19:43
kgriffsheh19:44
kgriffsthat is usually best, yes19:44
openstackgerritA change was merged to openstack/marconi: feat: marconi-proxy forwarding  https://review.openstack.org/4436419:46
openstackgerritA change was merged to openstack/marconi: feat: define interfaces for proxy storage  https://review.openstack.org/4632819:46
openstackgerritA change was merged to openstack/marconi: test(proxy): partition related endpoints tested  https://review.openstack.org/4636219:47
openstackgerritA change was merged to openstack/marconi: test(proxy): catalog tested  https://review.openstack.org/4676919:47
cppcabreralook at all the merging.19:50
cppcabrera$250,000 worth of merging19:50
zyuanpls allow me to make a YaoMing face...19:50
zyuanok i'm going to fix bugs in forwarding19:55
cppcabreralol @ YaoMing face19:56
cppcabreraLooking at the weighted_select code, that's totally a scanl + a reduce.19:57
zyuanwuu19:57
cppcabrerabecause of that acc, I can't think of a way to use a listcomp. :/19:57
*** tedross has joined #openstack-marconi20:10
*** ayoung has joined #openstack-marconi20:22
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: test(proxy): forwarded endpoints tested  https://review.openstack.org/4696320:38
zyuan^^ fixed 1 bug; awaiting some structural changes///20:38
zyuankgriffs: the term 'responder' has been used in Resource code, which refers to on_get, on_put, and even na_responder, etc.20:40
zyuanneed a new term to name "a resource with an all-in-one responder"20:40
kgriffsthat thing actually is a responder, just a standalone one - although, yes, it could be confusing because responders are currently defined as 1:1 (responder:verb)20:45
openstackgerritZhihao Yuan proposed a change to openstack/marconi: WIP: test(proxy): forwarded endpoints tested  https://review.openstack.org/4696320:46
zyuandominator20:46
zyuanha ha ha ha20:46
cppcabreralol dominator... :P20:46
zyuana20:47
zyuanhttp://psychopass.wikia.com/wiki/The_Dominator20:47
*** oz_akan_ has quit IRC20:56
*** oz_akan_ has joined #openstack-marconi20:56
*** oz_akan_ has quit IRC21:01
openstackgerritAlejandro Cabrera proposed a change to openstack/marconi: Add mongo driver for marconi proxy storage  https://review.openstack.org/4637521:16
cppcabreraHeading home, guys.21:18
kgriffsk21:19
zyuanbye21:19
cppcabreraI left a not-so-terrible patch for review above.21:19
kgriffswatch for a monster patch from yours truly21:19
cppcabreraMore removed than added. :D21:19
cppcabreraawesome, kgriffs. I'll check that out tomorrow.21:19
kgriffs(unrelated to proxy)21:19
cppcabrera(theme: monster patches)21:19
kgriffs…with big hairy feet21:19
cppcabreraI striked out the design flaws handled by patch #4637521:19
kgriffscool beans21:20
cppcabreraGood night~ :D21:20
*** cppcabrera has left #openstack-marconi21:20
kgriffsciao!21:20
kgriffso/21:20
zyuanneed a new create_http_method_map21:25
zyuanhehe21:25
zyuansee you kurt21:29
kgriffsttfn21:36
*** oz_akan_ has joined #openstack-marconi22:07
*** oz_akan_ has quit IRC22:12
*** amitgandhi has quit IRC22:27
*** tedross has quit IRC23:06
*** kgriffs is now known as kgriffs_afk23:11
*** tedross has joined #openstack-marconi23:35
*** tedross has quit IRC23:44

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!