| *** rongze has joined #openstack-marconi | 00:31 | |
| *** rongze has quit IRC | 00:37 | |
| *** nosnos has joined #openstack-marconi | 01:05 | |
| *** ayoung has joined #openstack-marconi | 01:37 | |
| *** sudorandom has left #openstack-marconi | 02:05 | |
| *** oz_akan_ has joined #openstack-marconi | 02:06 | |
| *** rongze has joined #openstack-marconi | 02:23 | |
| *** sudorandom has joined #openstack-marconi | 02:29 | |
| *** rongze_ has joined #openstack-marconi | 02:31 | |
| *** rongze has quit IRC | 02:33 | |
| *** oz_akan_ has quit IRC | 03:26 | |
| *** jergerber has joined #openstack-marconi | 03:51 | |
| *** chandankumar has joined #openstack-marconi | 03:56 | |
| *** jergerber has quit IRC | 03:57 | |
| *** rongze_ has quit IRC | 04:44 | |
| *** haomaiwang has joined #openstack-marconi | 05:09 | |
| *** haomaiwang has quit IRC | 05:14 | |
| *** haomaiwang has joined #openstack-marconi | 05:18 | |
| *** haomaiwang has quit IRC | 05:23 | |
| *** flwang has joined #openstack-marconi | 05:31 | |
| *** nosnos_ has joined #openstack-marconi | 05:34 | |
| *** rongze has joined #openstack-marconi | 05:35 | |
| *** nosnos_ has quit IRC | 05:35 | |
| *** nosnos_ has joined #openstack-marconi | 05:36 | |
| *** nosnos has quit IRC | 05:37 | |
| *** rongze has quit IRC | 05:41 | |
| *** haomaiwang has joined #openstack-marconi | 05:49 | |
| *** rongze has joined #openstack-marconi | 06:08 | |
| *** nosnos has joined #openstack-marconi | 06:59 | |
| *** nosnos_ has quit IRC | 06:59 | |
| *** flaper87|afk is now known as flaper87 | 08:12 | |
| flwang | flaper87: ping | 08:21 |
|---|---|---|
| flaper87 | flwang: pong | 08:21 |
| flwang | flaper87: some stupid question coming | 08:21 |
| flwang | flaper87: how to configure marconi to distinguish queues with tenant or user? | 08:22 |
| flaper87 | flwang: mmh, what do you mean? | 08:22 |
| flwang | flaper87: you know, the common queue url is http://127.0.0.1:8888/v1/queues/flwang | 08:24 |
| flwang | so is it possible that two different tenant/user have the same name queues? or they can't? | 08:24 |
| flwang | I saw it somewhere, but i can't remember it | 08:25 |
| flaper87 | flwang: ah yeah, they can. You just need to create the queue from a different project (tenant) | 08:28 |
| flaper87 | there's a header X-PROJECT-ID | 08:28 |
| flwang | flaper87: so is there any specific design consideration using header instead of a patch within url? you know, SQS using the user id with in URL | 08:30 |
| flwang | s/patch/path | 08:31 |
| flaper87 | flwang: We, as OpenStack, used to have the tenant in the URL but we're now moving away from that and putting it in the headers | 08:32 |
| flaper87 | plus, it was renamed from tenant to project | 08:33 |
| flaper87 | I think there's an email thread about that | 08:33 |
| flwang | you mean from url to headers? | 08:33 |
| *** Abhishek_ has joined #openstack-marconi | 08:34 | |
| flaper87 | flwang: yup | 08:35 |
| flwang | flaper87: ok, I will search it. | 08:36 |
| flwang | flaper87: #2, is the Email service in the scope of Marconi? | 08:36 |
| flaper87 | flwang: http://lists.openstack.org/pipermail/openstack-dev/2013-May/008965.html | 08:36 |
| flaper87 | there you go | 08:36 |
| flaper87 | flwang: that sounds like something that could be implemented as a notification plugin | 08:37 |
| flaper87 | we've talked about that :) | 08:37 |
| flwang | flaper87: I joined that brainsotrom meeting. so based on my understanding, Marconi notification sounds like the SNS of AWS, it's a Pub/Sub way, right? | 08:39 |
| flaper87 | flwang: Marconi is both. It's queues and notification for openstack | 08:39 |
| flwang | flaper87: i know | 08:39 |
| flaper87 | wherease AWS has 2 separate services | 08:39 |
| flaper87 | flwang: I meant to say that Marconi is SQS and SNS | 08:40 |
| flaper87 | so, you're right :D | 08:40 |
| flwang | so you mean Marocni is combine the SQS, SNS, SES all in one, right? | 08:40 |
| flaper87 | flwang: well, not SES | 08:41 |
| flwang | in other words, we have a pub/sub framework, so we just extend it to support SNS and SES, is it? | 08:41 |
| flaper87 | flwang: exactly | 08:41 |
| flwang | flaper87: so that's what i'm asking | 08:41 |
| flwang | can Marconi cover the SES functions? | 08:41 |
| flaper87 | It may be split in the future but as for now, it's all in one! | 08:41 |
| flwang | or is that our goal? | 08:42 |
| flaper87 | flwang: I think it can - I'm not an SES expert - but I'm not sure we want to officially support that | 08:42 |
| flaper87 | Marconi's goal is to provide queuing and notification - SQS and SNS - without adding unnecessary complexity | 08:43 |
| flwang | flaper87: got it | 08:43 |
| flwang | flaper87: 3rd question coming | 08:43 |
| *** ykaplan has joined #openstack-marconi | 08:43 | |
| flaper87 | flwang: shoot | 08:44 |
| flwang | flaper87: you know, there are only a few patches waiting for review, so why? less contributor? or we have created everything :D | 08:44 |
| flaper87 | flwang: The team is still small, we're working on getting more people. Recently Cindy joined us as part of the OPW intership program and ykaplan has been contributing as well. | 08:48 |
| flaper87 | flwang: there are many things to do in Marconi, we're far from feature complete and optimized | 08:48 |
| flwang | flaper87: i see. I'm eager to contribute more, but I think need more time to warm-up | 08:49 |
| flaper87 | flwang: please, feel free to ask anything you need | 08:53 |
| flaper87 | there are many areas you can contribute to :) | 08:53 |
| flaper87 | And there are also many bugs we just triaged last week | 08:53 |
| flwang | flaper87: yep, it would be nice if you can help figure out. and I really want to join the Marconi session in the next summit :) | 08:54 |
| flwang | flaper87: will go through the bug list and find some low-hanging-fruit again :D and meanwhile, pls let me know if there is any more complex bug I can fix | 08:57 |
| flaper87 | flwang: sure thing! Thanks flwang! | 08:57 |
| flwang | flaper87: thanks, more questions will be coming soon :D | 08:58 |
| *** yassine has joined #openstack-marconi | 09:06 | |
| *** Abhishek_ has quit IRC | 09:15 | |
| *** Abhishek_ has joined #openstack-marconi | 09:24 | |
| *** flwang has quit IRC | 09:41 | |
| *** ykaplan has quit IRC | 09:45 | |
| *** ykaplan has joined #openstack-marconi | 09:45 | |
| *** haomaiwang has quit IRC | 09:52 | |
| *** briancline has quit IRC | 09:52 | |
| *** haomaiwang has joined #openstack-marconi | 09:54 | |
| *** briancline has joined #openstack-marconi | 09:54 | |
| *** Abhishek_ has quit IRC | 09:58 | |
| *** rongze has quit IRC | 10:09 | |
| *** yassine has quit IRC | 10:38 | |
| *** rongze has joined #openstack-marconi | 10:40 | |
| *** rongze has quit IRC | 10:45 | |
| *** ykaplan has quit IRC | 10:53 | |
| *** ykaplan has joined #openstack-marconi | 11:56 | |
| *** yassine has joined #openstack-marconi | 12:05 | |
| *** Abhishek_ has joined #openstack-marconi | 12:24 | |
| *** Abhishek_ has quit IRC | 12:41 | |
| *** rongze has joined #openstack-marconi | 12:41 | |
| *** thomasem has joined #openstack-marconi | 12:43 | |
| *** rongze has quit IRC | 12:46 | |
| *** nosnos has quit IRC | 12:47 | |
| *** Abhishek has joined #openstack-marconi | 13:15 | |
| *** oz_akan_ has joined #openstack-marconi | 13:25 | |
| *** oz_akan_ has quit IRC | 13:26 | |
| *** oz_akan_ has joined #openstack-marconi | 13:26 | |
| *** jamieh has joined #openstack-marconi | 13:35 | |
| *** Abhishek has quit IRC | 13:44 | |
| *** cpallares has joined #openstack-marconi | 13:58 | |
| *** rongze has joined #openstack-marconi | 14:12 | |
| *** thomasm_ has joined #openstack-marconi | 14:15 | |
| *** thomasm_ has left #openstack-marconi | 14:17 | |
| *** thomasm has joined #openstack-marconi | 14:19 | |
| *** thomasm is now known as Guest8385 | 14:20 | |
| *** rustlebee is now known as russellb | 14:22 | |
| *** notmyname_ has joined #openstack-marconi | 14:22 | |
| *** cpallares has quit IRC | 14:23 | |
| *** thomasem has quit IRC | 14:23 | |
| *** notmyname has quit IRC | 14:23 | |
| *** notmyname_ is now known as notmyname | 14:23 | |
| *** Abhishek_ has joined #openstack-marconi | 14:24 | |
| *** jcru has joined #openstack-marconi | 14:24 | |
| *** mpanetta has joined #openstack-marconi | 14:29 | |
| *** openstackgerrit has quit IRC | 14:31 | |
| *** openstackgerrit has joined #openstack-marconi | 14:31 | |
| *** cpallares has joined #openstack-marconi | 14:33 | |
| openstackgerrit | Yeela Kaplan proposed a change to openstack/marconi: Implement tables and unit tests for sqlalchemy storage backend https://review.openstack.org/60223 | 14:35 |
| *** malini_afk is now known as malini | 14:43 | |
| *** Abhishek_ has quit IRC | 14:43 | |
| *** alcabrera has joined #openstack-marconi | 14:50 | |
| alcabrera | Good morning! | 14:51 |
| mpanetta | mornin | 14:56 |
| alcabrera | mpanetta: hey! How was your weekend? :) | 14:59 |
| mpanetta | Not too bad, went to see the new Hobbit movie | 15:00 |
| alcabrera | How was it? | 15:00 |
| mpanetta | Pretty good :) | 15:01 |
| mpanetta | meeting time... | 15:01 |
| *** whenry has joined #openstack-marconi | 15:08 | |
| *** mpanetta_ has joined #openstack-marconi | 15:11 | |
| *** mpanetta has quit IRC | 15:14 | |
| *** Alex_Gaynor has quit IRC | 15:14 | |
| *** Alex_Gaynor has joined #openstack-marconi | 15:14 | |
| *** Alex_Gaynor has quit IRC | 15:14 | |
| *** Alex_Gaynor has joined #openstack-marconi | 15:14 | |
| malini | flaper87: ping | 15:15 |
| *** Alex_Gaynor has quit IRC | 15:15 | |
| *** Alex_Gaynor has joined #openstack-marconi | 15:15 | |
| *** flwang has joined #openstack-marconi | 15:19 | |
| *** flwang has quit IRC | 15:22 | |
| *** kgriffs_afk is now known as kgriffs | 15:24 | |
| *** Alex_Gaynor_ has joined #openstack-marconi | 15:26 | |
| *** flwang has joined #openstack-marconi | 15:26 | |
| *** sudorandom_ has joined #openstack-marconi | 15:26 | |
| *** Alex_Gaynor has quit IRC | 15:27 | |
| *** sudorandom has quit IRC | 15:27 | |
| *** sudorandom_ is now known as sudorandom | 15:27 | |
| *** Alex_Gaynor_ is now known as Alex_Gaynor | 15:27 | |
| *** Guest8385 is now known as thomasem | 15:31 | |
| *** zyuan has joined #openstack-marconi | 15:35 | |
| *** zyuan has joined #openstack-marconi | 15:36 | |
| *** jergerber has joined #openstack-marconi | 15:37 | |
| *** jergerber has quit IRC | 15:37 | |
| *** jergerber has joined #openstack-marconi | 15:37 | |
| *** jergerber has quit IRC | 15:39 | |
| *** haomaiwang has quit IRC | 15:39 | |
| *** briancline has quit IRC | 15:39 | |
| *** haomaiwang has joined #openstack-marconi | 15:39 | |
| *** Abhishek has joined #openstack-marconi | 15:44 | |
| *** briancline has joined #openstack-marconi | 15:44 | |
| *** haomaiwa_ has joined #openstack-marconi | 15:46 | |
| flaper87 | HELLOOOOOOOOOOOOOOOO | 15:50 |
| flaper87 | malini: pong | 15:50 |
| flaper87 | alcabrera: mpanetta_ good morning | 15:50 |
| * flaper87 was in a call | 15:50 | |
| malini | flaper87: how are you? | 15:50 |
| * flaper87 went to see The Hobbit | 15:50 | |
| flaper87 | imax + 3D | 15:51 |
| malini | tht sounds cool :) | 15:51 |
| alcabrera | flaper87: yo! :) | 15:51 |
| *** torgomatic_ has joined #openstack-marconi | 15:52 | |
| *** cpallares1 has joined #openstack-marconi | 15:52 | |
| flaper87 | alcabrera: 'sup buddy! How was your weekend ? | 15:52 |
| flaper87 | malini: yeah! It was reaaally nice! | 15:52 |
| flaper87 | I was like wooowing the whole time | 15:52 |
| flaper87 | :D | 15:52 |
| malini | wow!! | 15:53 |
| malini | :D | 15:53 |
| *** Abhishek_ has joined #openstack-marconi | 15:53 | |
| malini | flaper87: I have a question for you on marconi-devstack | 15:53 |
| alcabrera | It was relaxing~ I got to write a bunch and spend plenty of time with family, flaper87! :) | 15:54 |
| malini | flaper87: Are you using the default marconi config with auth turned off ? | 15:54 |
| *** cpallares has quit IRC | 15:54 | |
| *** torgomatic has quit IRC | 15:54 | |
| *** torgomatic_ is now known as torgomatic | 15:55 | |
| *** Abhishek has quit IRC | 15:56 | |
| *** haomaiwang has quit IRC | 15:56 | |
| flaper87 | alcabrera: awesome, I'm glad to heard that! | 16:00 |
| flaper87 | malini: mmh, nope, devstack turns auth on | 16:00 |
| flaper87 | malini: any issues? | 16:01 |
| *** flaper87 is now known as flaper87|call | 16:02 | |
| malini | I dont think its validating the auth token, but cud be me | 16:03 |
| malini | I'll chk again | 16:03 |
| *** kgriffs has quit IRC | 16:04 | |
| *** flwang has quit IRC | 16:04 | |
| *** kgriffs has joined #openstack-marconi | 16:05 | |
| *** ChanServ sets mode: +o kgriffs | 16:05 | |
| *** haomaiwang has joined #openstack-marconi | 16:08 | |
| *** chandankumar has quit IRC | 16:17 | |
| *** haomaiwa_ has quit IRC | 16:18 | |
| *** dragondm has quit IRC | 16:18 | |
| *** briancline has quit IRC | 16:18 | |
| *** al-maisan has quit IRC | 16:18 | |
| *** apmelton1 has quit IRC | 16:18 | |
| *** russell_h has quit IRC | 16:18 | |
| *** lakspace has quit IRC | 16:18 | |
| *** mpanetta_ has quit IRC | 16:18 | |
| *** metral has quit IRC | 16:18 | |
| *** briancline has joined #openstack-marconi | 16:21 | |
| *** mpanetta has joined #openstack-marconi | 16:21 | |
| *** al-maisan has joined #openstack-marconi | 16:21 | |
| *** apmelton1 has joined #openstack-marconi | 16:21 | |
| *** metral has joined #openstack-marconi | 16:21 | |
| *** dragondm has joined #openstack-marconi | 16:21 | |
| *** russell_h has joined #openstack-marconi | 16:21 | |
| *** lakspace has joined #openstack-marconi | 16:21 | |
| *** rwsu_ has joined #openstack-marconi | 16:26 | |
| *** jergerber has joined #openstack-marconi | 16:26 | |
| *** rwsu has quit IRC | 16:27 | |
| *** rektide_ has quit IRC | 16:27 | |
| *** rektide has joined #openstack-marconi | 16:27 | |
| *** thomasem has quit IRC | 16:37 | |
| *** ametts has joined #openstack-marconi | 16:42 | |
| *** rektide has quit IRC | 16:45 | |
| *** rektide has joined #openstack-marconi | 16:46 | |
| *** flaper87|call is now known as flaper87 | 16:56 | |
| flaper87 | back | 16:56 |
| cpallares1 | o/ | 16:57 |
| *** cpallares1 is now known as cpallares | 16:57 | |
| *** thomasem has joined #openstack-marconi | 16:58 | |
| flaper87 | cpallares: HELLOOOOO! | 16:59 |
| flaper87 | cpallares: how are you doing? :) | 16:59 |
| cpallares | flaper87: I'm good :) how are you? | 16:59 |
| cpallares | how was your weekend? | 16:59 |
| flaper87 | cpallares: it was good. I went to see The Hobbit :) | 17:00 |
| flaper87 | what about yours? | 17:00 |
| cpallares | flaper87: Did you like it? I want to see it! | 17:00 |
| cpallares | It was good and relaxing :) | 17:00 |
| *** haomaiwang has quit IRC | 17:01 | |
| *** ykaplan has quit IRC | 17:02 | |
| flaper87 | cpallares: it was good. I did have couple of "WTF?" moments but I don't want to spoiler anything | 17:04 |
| cpallares | flaper87: oooh i need to see it now. I liked the first one and I really enjoyed the book when i read it years ago | 17:05 |
| mpanetta | flaper87: You went to see the Hobbit as well? | 17:07 |
| * flaper87 ties his hands | 17:07 | |
| flaper87 | mpanetta: yeaaah! | 17:07 |
| *** yassine has quit IRC | 17:11 | |
| kgriffs | malini: ping | 17:13 |
| kgriffs | flaper87: anything you want to add to the agenda? | 17:13 |
| kgriffs | https://wiki.openstack.org/wiki/Meetings/Marconi#Agenda | 17:13 |
| flaper87 | kgriffs: not this time around. | 17:14 |
| kgriffs | kk | 17:14 |
| kgriffs | flaper87: different topic | 17:15 |
| kgriffs | over is solum-land there's a patch under review that adds some functional tests | 17:15 |
| kgriffs | but, host/port is hard-coded | 17:15 |
| kgriffs | In marconi we have a conf file for that stuff | 17:16 |
| cpallares | speaking of the meeting someone should change the meeting time in the irc description :D | 17:16 |
| kgriffs | cpallares: lol | 17:16 |
| kgriffs | good catch | 17:16 |
| cpallares | kgriffs: o/ | 17:16 |
| cpallares | how are you? | 17:17 |
| kgriffs | flaper87: is there any reason you could think of for solum *not* following marconi's lead here? | 17:17 |
| kgriffs | cpallares: I'm groovy. How are you? | 17:17 |
| cpallares | i'm groovy myself, kgriffs :) | 17:18 |
| *** kgriffs changes topic to "OpenStack Queuing and Notification Service || Smile :D || Meetings every Tuesday @ 15:00 UTC || Wiki: ttps://wiki.openstack.org/wiki/Marconi || Paste: http://paste.openstack.org/ || Send messages and make some noise :D" | 17:18 | |
| flaper87 | kgriffs: I can definitely tell you why we're using a config file. We wanted to keep tests configurable in order to test different scenarios with the same installation of marconi. The idea there is to also have a way to have a base test and make it point to different scenarios | 17:19 |
| flaper87 | OTOH, I don't think Solum needs this | 17:19 |
| kgriffs | flaper87: kk, that makes sense | 17:20 |
| flaper87 | and the last scenario should have been deployment | 17:20 |
| flaper87 | but you understood what I meant :D | 17:21 |
| kgriffs | right. :D | 17:21 |
| kgriffs | malini: could you help review this patch? https://review.openstack.org/#/c/62234/ | 17:22 |
| kgriffs | flaper87: ^^^ Solum patch I was talking about | 17:24 |
| *** alcabrera is now known as alcabrera|afk | 17:26 | |
| malini | kgriffs: sure | 17:27 |
| kgriffs | malini: thanks! | 17:31 |
| *** flaper87 is now known as flaper87|afk | 17:41 | |
| *** jamieh has quit IRC | 18:15 | |
| *** neosergey has joined #openstack-marconi | 18:28 | |
| *** neosergey has quit IRC | 18:30 | |
| *** rongze has quit IRC | 18:41 | |
| *** primochka has joined #openstack-marconi | 18:56 | |
| *** primochka has quit IRC | 19:03 | |
| *** Abhishek_ has quit IRC | 19:36 | |
| *** balajiiyer has joined #openstack-marconi | 19:48 | |
| *** malini is now known as malini_afk | 20:00 | |
| *** malini_afk is now known as malini | 20:01 | |
| *** reed has joined #openstack-marconi | 20:23 | |
| *** malini is now known as malini_afk | 20:25 | |
| *** alcabrera|afk is now known as alcabrera | 20:47 | |
| *** openstackgerrit has quit IRC | 21:08 | |
| *** openstackgerrit has joined #openstack-marconi | 21:08 | |
| kgriffs | alcabrera: | 21:23 |
| kgriffs | ping | 21:23 |
| alcabrera | kgriffs: pong | 21:26 |
| alcabrera | hey, hey | 21:26 |
| kgriffs | alcabrera: yo | 21:26 |
| kgriffs | so, got a little research to share | 21:26 |
| kgriffs | and get feedback on | 21:26 |
| alcabrera | alright - I'm ready to listen. | 21:27 |
| kgriffs | alcabrera: https://gist.github.com/kgriffs/7994745 | 21:27 |
| kgriffs | I've been playing with NTPD | 21:28 |
| alcabrera | hmmm | 21:28 |
| kgriffs | so, I am about to test synchronization between different boxes | 21:30 |
| kgriffs | but first, I used those functions to check granularity of time.time() | 21:30 |
| kgriffs | turns out that with over 500,000 requests/sec you still get unique timestamps | 21:30 |
| mpanetta | damn | 21:30 |
| alcabrera | interesting | 21:31 |
| mpanetta | amazing heh | 21:31 |
| kgriffs | the clock granularity hovers around 1 microsecond | 21:31 |
| kgriffs | so, that's not going to be a problem for marconi if we want to use timestamps | 21:31 |
| mpanetta | wow I thought IEEE1588 was supposed to do that, I guess NTP can too :) | 21:31 |
| kgriffs | i am about to test clock drift between boxes | 21:32 |
| alcabrera | so what that tells me, is that given marconi on a single machine (or some oracle), we can scale up to 500K rps while relying solely on time.time() | 21:32 |
| alcabrera | let's see what time drift tells us | 21:32 |
| kgriffs | true, we could do it with an oracle | 21:34 |
| kgriffs | anybody remember the option for forcing ntpd to converge faster? | 21:35 |
| alcabrera | kgriffs: seems to be -x | 21:36 |
| alcabrera | I looked that up just now. | 21:36 |
| mpanetta | kgriffs: if NTP can't do what you want, check out http://ptpd.sourceforge.net/ | 21:39 |
| kgriffs | mpanetta: interesting, I'll check it out! | 21:52 |
| mpanetta | IEEE1588 is high res time stamping | 21:53 |
| alcabrera | mpanetta, kgriffs: I found this interesting comparison of NTP vs. PTP: http://www.fsmlabs.com/blog/choosing-between-ptp-and-ntp | 21:54 |
| alcabrera | the high-resolution aspect seems to be identical for ntp vs. ptp, but the communication style is a little different | 21:55 |
| mpanetta | Yeah | 21:55 |
| kgriffs | alcabrera: btw, I was just thinking, if we go the oracle route, we only need to be collision-resistant for a single queue | 21:55 |
| alcabrera | (e.g., sub-microsecond resolution) | 21:55 |
| mpanetta | well I knew about the second part, not the first. | 21:55 |
| kgriffs | so, you could have a pool of oracles and shard on queue name | 21:55 |
| kgriffs | then, you use a clock | 21:55 |
| kgriffs | and you don't have to worry about clock drift | 21:55 |
| alcabrera | hmmm | 21:56 |
| kgriffs | just, ensuring that no single queue can overrun a single oracle | 21:56 |
| kgriffs | but, it would be even cooler to not have an oracle | 21:56 |
| kgriffs | :D | 21:56 |
| alcabrera | heh | 21:56 |
| mpanetta | We should get our overlords to install atomic clocks at all the datacenters | 21:56 |
| alcabrera | it sounds like it would work as long as no single queue was servicing more than 1 worker at a time (since that has a nondeterministic order, anyway) and as long as no oracle was servicing more than 500K rps | 21:57 |
| alcabrera | actually, the entire notion of monotonically increasing markers just made me think of something I just watched | 21:58 |
| alcabrera | Lattices for distributed systems - e.g., how to guarantee consistency in a distributed context as long as the operations in question are monotonically increasing | 21:58 |
| kgriffs | alcabrera: darn; timekeeper costs $$$ | 21:59 |
| kgriffs | http://goo.gl/A5SDZb | 21:59 |
| alcabrera | I wasn't able to absorb most of it this run, but I wonder what if it's applicable to the marker generation problem for FIFO queues... | 21:59 |
| alcabrera | ? | 21:59 |
| alcabrera | let's see about this timekeeper | 21:59 |
| kgriffs | alcabrera: interesting. is that a CS paper? | 22:00 |
| alcabrera | kgriffs: It's both a CS paper and a RICON West 2013 talk | 22:01 |
| kgriffs | oic | 22:02 |
| alcabrera | kgriffs: http://www.youtube.com/watch?v=8dFO5Ir0xqY&feature=youtu.be | 22:02 |
| alcabrera | timekeeper boasts some impressive features, mostly time-smoothing and performance points | 22:02 |
| alcabrera | kgriffs: also, CRDTs (Conflict-Free Replicated Data Types) - one of the fundamental ideas behind Riak (DB): http://codemesh.io/#joel-jacobson, http://stackoverflow.com/questions/11254318/conflict-free-replicated-data-types-crdt-vs-paxos | 22:09 |
| * alcabrera is on a reading, watching, learning day | 22:09 | |
| *** balajiiyer has quit IRC | 22:10 | |
| kgriffs | ah yes | 22:10 |
| kgriffs | crazy stuff | 22:10 |
| *** jergerber has quit IRC | 22:19 | |
| *** oz_akan_ has quit IRC | 22:25 | |
| mpanetta | I love watching you 2 converse, thanks for the NTP vs PTP link alcabrera | 22:30 |
| alcabrera | mpanetta: no problem. :) | 22:33 |
| mpanetta | It puts to bed some of the misconceptions I had about NTP | 22:34 |
| alcabrera | I didn't even realize PTP existed 'til today. | 22:35 |
| mpanetta | Ah, yeah it has been around for a few years | 22:35 |
| mpanetta | Mostly for embedded and industrial spaces | 22:36 |
| alcabrera | Whenever I'm presented with a new technology or protocol, particularly as an alternative, I'm off to the Googles to compare, constrast for about 5 minutes. | 22:36 |
| * mpanetta has a lot of interest in those spaces | 22:36 | |
| alcabrera | *contrast | 22:36 |
| mpanetta | Cool :) | 22:36 |
| mpanetta | Good thing to do. | 22:36 |
| alcabrera | mpanetta: I have noted your interest in the embedded space. :D | 22:36 |
| alcabrera | Which reminds me... | 22:36 |
| *** thomasem has quit IRC | 22:36 | |
| alcabrera | Have you heard of the Parallela chip? | 22:37 |
| mpanetta | Yep :) Very interesting | 22:37 |
| mpanetta | Not sure how far they got tho | 22:37 |
| alcabrera | seems to be a hit | 22:37 |
| kgriffs | alcabrera: I think we need to fix this for icehouse | 22:37 |
| kgriffs | https://bugs.launchpad.net/marconi/+bug/1242926 | 22:37 |
| mpanetta | They had a ickstarter last year I think? Maybe earler this year, | 22:37 |
| alcabrera | my newsfeeds have been a-buzz with talk of it | 22:37 |
| kgriffs | it is a pretty critical ops thing | 22:37 |
| * mpanetta goes to search | 22:37 | |
| alcabrera | There's even plans to get a 1024-core chip out next year | 22:37 |
| alcabrera | kgriffs: lessee... | 22:37 |
| mpanetta | damn! | 22:37 |
| kgriffs | I upped the priority | 22:37 |
| kgriffs | ametts: ^^^ | 22:38 |
| kgriffs | we will need to get this in Rackspace's Q1 roadmap, if possible | 22:38 |
| kgriffs | if not, then perhaps some of our new contributors can help out | 22:38 |
| kgriffs | :D | 22:38 |
| mpanetta | Oh yeah, now I remember. I wish they allowed more access to the Zynq IO... Such a nice chip the Zynq. | 22:38 |
| alcabrera | mpanetta: http://www.parallella.org/ | 22:39 |
| mpanetta | alcabrera: already there :) | 22:39 |
| ametts | kgriffs: This is what you're talking about? https://bugs.launchpad.net/marconi/+bug/1242926 | 22:39 |
| alcabrera | kgriffs: So the keystone auth middleware is injected in such a way that bastion won't work...? | 22:39 |
| kgriffs | ametts: yes | 22:39 |
| mpanetta | kgriffs: isn't that what the bastion was for? | 22:39 |
| alcabrera | s/won't/can't | 22:39 |
| kgriffs | alcabrera: no, sorry it is a little confusing | 22:39 |
| kgriffs | the bastion can work but you have to do your own app.py | 22:40 |
| kgriffs | a couple things | 22:40 |
| kgriffs | first, health needs to be callable out of the box even when auth_strategy is enabled | 22:40 |
| alcabrera | gotcha | 22:40 |
| kgriffs | (without auth, but only if NOT coming from the LB, like eom.bastion does) | 22:41 |
| kgriffs | i mean, it should be a list of restricted things | 22:41 |
| kgriffs | and we default to including health in that list | 22:41 |
| alcabrera | would this be support for restricted routes in marconi? | 22:41 |
| kgriffs | alcabrera: the second part is less important, I guess, but we need a way to insert middleware between keystone and the app without having to write your own app.py | 22:41 |
| kgriffs | alcabrera: right | 22:41 |
| kgriffs | alcabrera: but, I don't think Rackspace could use it because it would assume use of auth_strategy | 22:42 |
| kgriffs | which currently precludes using post-auth middleware | 22:42 |
| *** jergerber has joined #openstack-marconi | 22:43 | |
| alcabrera | I see the need for this, and I agree with the priority,' | 22:49 |
| alcabrera | kgriffs: ^ | 22:49 |
| kgriffs | kk, thanks! | 22:49 |
| alcabrera | flaper87|afk, and cpallares' API patch work that labels certain endpoints as 'admin': True seems complementary to this effort | 22:50 |
| *** flaper87|afk is now known as flaper87 | 22:50 | |
| flaper87 | alcabrera: awesome! Thanks! | 22:50 |
| flaper87 | alcabrera: btw, ykaplan submitted a new version of her patch, could you take a lok there too? | 22:51 |
| flaper87 | kgriffs: ^^^^ | 22:51 |
| alcabrera | will do. I'll do it tomorrow morning, flaper87. :) | 22:51 |
| flaper87 | kgriffs: did you read in #falconframework the case where I think N route 1 controller could make sense | 22:51 |
| flaper87 | alcabrera: sounds good :) | 22:51 |
| flaper87 | thanks a lot! | 22:51 |
| alcabrera | flaper87, kgriffs: This particular concept (CRDTs: http://codemesh.io/slides/joel-jacobson.pdf) has eaten my brain for the rest of the night. | 22:52 |
| * flaper87 wonders what kgriffs is doing with ntpd | 22:53 | |
| * flaper87 is afraid he doesn't want to know | 22:53 | |
| flaper87 | :D | 22:53 |
| alcabrera | ;) | 22:53 |
| mpanetta | hah! | 22:54 |
| mpanetta | Secret squirly things | 22:54 |
| * flaper87 just bought this: http://www.amazon.it/gp/product/B001S2PPT0/ref=ox_ya_os_product | 22:55 | |
| alcabrera | flaper87: so that's what that Tweet was about! | 22:55 |
| alcabrera | Good times. :D | 22:55 |
| flaper87 | alcabrera: yeaaah!!! | 22:56 |
| mpanetta | Nice lense | 22:56 |
| *** mpanetta has quit IRC | 22:56 | |
| flaper87 | that lens stole my heart last July. Today it stole my money too. | 22:56 |
| kgriffs | flaper87: wrt the N:1 | 22:57 |
| kgriffs | yes, I saw it | 22:58 |
| kgriffs | are you proposing to have One Big Controller? | 22:58 |
| kgriffs | rather than resource-based? | 22:58 |
| flaper87 | kgriffs: and you didn't say anything??? :0 you just broke my heart | 22:58 |
| kgriffs | flaper87: heh... can't remember why - it may have been pulled away just then to a mtg or something | 22:58 |
| * kgriffs still loves flaper87 | 22:58 | |
| flaper87 | kgriffs: it's not actually 1 big controller, it's just a small controller that will call the API layer which will then actually process the request | 22:59 |
| * flaper87 feels better now | 22:59 | |
| kgriffs | hmm | 22:59 |
| flaper87 | I'd like transports to be stupid logic wise | 22:59 |
| flaper87 | and be fucking smart protocol wise | 22:59 |
| kgriffs | but all requests would be funneled through a single class? | 22:59 |
| flaper87 | yup | 23:00 |
| kgriffs | hmm | 23:00 |
| * kgriffs thinks about that | 23:00 | |
| kgriffs | flaper87: were you hoping to auto-generate the transports? | 23:06 |
| flaper87 | kgriffs: yup :D | 23:06 |
| kgriffs | so, off the top of my head, the WSGI transport has the following functions currently | 23:08 |
| kgriffs | 1. input validation | 23:08 |
| kgriffs | 2. protocol translation (http request to storage controller request, and then the response back out | 23:08 |
| kgriffs | 3. error translation - e.g., convert DoesNotExist to 404 Not Found | 23:09 |
| kgriffs | some input validation may be more efficiently done before passing stuff down the stack | 23:10 |
| kgriffs | error translation will be tricky to automate - you will need a way to configure the mapping, but also include metadata about the error, in case it needs to be returned in the diagnostic section of the response body | 23:10 |
| kgriffs | for the error translation, you will basically end up writing a DSL that may be more complex than just writing the Python | 23:11 |
| kgriffs | what was you plan to tackle that stuff? | 23:12 |
| flaper87 | kgriffs: I was thinking about moving most of that to the API layer | 23:16 |
| flaper87 | things like Exception translation will need to stay in the transport layer, though | 23:16 |
| kgriffs | flaper87: right | 23:16 |
| kgriffs | so, I think it makes sense to push down some logic into the API layer | 23:16 |
| kgriffs | but, I don't know that it is a great idea to auto-generate the transport | 23:16 |
| kgriffs | taking HTTP for example | 23:17 |
| flaper87 | oh wait, by autogenerate the transport I mean register endpoints dynamically | 23:17 |
| flaper87 | not the controller itself | 23:17 |
| kgriffs | just to be clear on terminology | 23:17 |
| kgriffs | we have routes, http resources, storage controllers | 23:18 |
| kgriffs | (currently) | 23:18 |
| flaper87 | yeah, that's in marconi, I was just talking about falcon routes and controllers | 23:18 |
| kgriffs | what I meant was the http resource classes should stay hand-coded so we can make sure exceptions are translated appropriately and optimize certain validations as needed | 23:18 |
| kgriffs | flaper87: falcon doesn't have controllers | 23:18 |
| kgriffs | those are resources (ala REST) | 23:19 |
| flaper87 | kgriffs: erm, sorry, I used the wrong term T_T | 23:19 |
| kgriffs | sure | 23:19 |
| kgriffs | django/rails I imagine use the term "controllers" | 23:19 |
| flaper87 | kgriffs: re hand-coded http resources, yes. We're on the same page there | 23:19 |
| kgriffs | ok | 23:19 |
| flaper87 | What I want to happen dynamicall is the routes registration | 23:19 |
| kgriffs | what does that buy us? | 23:19 |
| flaper87 | and I think that we can have just 1 controller that will fit them all. | 23:20 |
| kgriffs | flaper87: I'd prefer having separate resource classes rather than lumping everything into one big class | 23:20 |
| flaper87 | kgriffs: the fact that the API layer will tell the transport what are the available endpoints | 23:20 |
| kgriffs | smells like poor separation of conccerns | 23:20 |
| kgriffs | flaper87: but the transport has to know in advance what those are in order to implement the translation logic between HTTP land and storage/API land | 23:21 |
| flaper87 | kgriffs: no need, the transport needs to translate from HTTP to a common request object that the API layer knows how to use | 23:22 |
| flaper87 | transport -> api layer -> storage | 23:22 |
| kgriffs | but doesn't the http resource still need to some out of band knowledge, e.g., about error types that must be hand-coded? | 23:22 |
| flaper87 | kgriffs: yeah, that's the bit that needs to be resolved. | 23:23 |
| flaper87 | I was thinking that if we have a good separation of exceptions, we can easily map 1 exception to 1 http code | 23:23 |
| flaper87 | I'm not convinced, though | 23:24 |
| kgriffs | hmm | 23:24 |
| kgriffs | assuming we have a mapping engine that is smart enough to pull out error metadata and put it into the error body diagnostic info | 23:24 |
| kgriffs | I'm still not sold that this is saving us much work | 23:25 |
| kgriffs | dynamically generating the routes means having to dynamically generate the translation between protocol and API semantics | 23:25 |
| kgriffs | that's a tall order | 23:26 |
| kgriffs | I think that we may get there eventually, but the first step would be to push down as much logic as we can, down the stack | 23:26 |
| flaper87 | I don't want to generate the translation between the protocol and the API dynamically | 23:26 |
| kgriffs | flaper87: I think you will have to if you generate the routes | 23:26 |
| kgriffs | generating the routes implies generating the responders to those routes, or just assuming they exist | 23:27 |
| flaper87 | I'm assuming they exist | 23:27 |
| kgriffs | and if you are just assuming they exist, you might as well just define some "register()" hook on the transport that the bootstrap calls and it does it's own add_route things | 23:27 |
| flaper87 | that's why I thought about using 1 responder for all the routes | 23:27 |
| flaper87 | kgriffs: that's exactly what I'm saying | 23:28 |
| flaper87 | T_T | 23:28 |
| kgriffs | heh | 23:28 |
| flaper87 | damnit, I failed again at making my point! | 23:28 |
| flaper87 | (facepalm) | 23:28 |
| flaper87 | I think we just got mixed up with the terms | 23:28 |
| flaper87 | or I did | 23:28 |
| kgriffs | so, let me continue with my misunderstanding | 23:28 |
| kgriffs | and then you can correct me | 23:28 |
| kgriffs | so, I was envisioning that the API would read the schema and generate a bunch of add_route calls to a single resource class | 23:29 |
| kgriffs | i guess it would assume the appropriate responders already exist on the resource class | 23:29 |
| kgriffs | now, at this point, I am wondering how this saves us devs any work? | 23:30 |
| kgriffs | seems like just about as hard to write that add_route generator as it is to just write out a list of add_route calls | 23:30 |
| *** balajiiyer has joined #openstack-marconi | 23:30 | |
| kgriffs | (by hand) | 23:30 |
| kgriffs | that correspond to the responders that I already wrote by hand | 23:30 |
| kgriffs | I mean, we are talking about 15 minutes of dev time here for each new API to wire up the routes | 23:31 |
| * kgriffs is done with that train of thought | 23:31 | |
| flaper87 | well, my proposal was to use N routes and 1 responder! I don't think we need more than one responder. I admit I haven't thought much about the error translation. | 23:32 |
| kgriffs | having a single resource vs. multiple resources doesn't really matter so much; I think the crux of the issue is whether we can auto-generate the resources. | 23:33 |
| flaper87 | that said, the benefit it gives use is that the transport won't need to worry about the API version. it just needs to translate the incoming request to a MarconiRequest that the API layer will then process | 23:33 |
| kgriffs | (responder == on_get or on_post) | 23:33 |
| flaper87 | I don't think we should auto-generate them, that's too much magic that I'd prefer avoiding | 23:33 |
| kgriffs | we could also have a simply proxy that eats everything and spits it out the other end, fwiw | 23:33 |
| flaper87 | (HTTP -> MarconiRequest) -> API -> Storage | 23:34 |
| flaper87 | Transport = (HTTP -> MarconiRequest) | 23:34 |
| kgriffs | so, I like the idea of that flow there, pushing common logic down into API | 23:35 |
| kgriffs | I think that would be our first iteration | 23:35 |
| kgriffs | then, we can look at what is left over in the transports and see how much of that can be pushed elsewhere or automated | 23:35 |
| kgriffs | if we end up with zero hand-rolled code in the transport, yay! | 23:35 |
| *** oz_akan_ has joined #openstack-marconi | 23:36 | |
| kgriffs | but I don't think we can make that determination at this juncture | 23:36 |
| flaper87 | wAt this point, I've put lot of thoughts about this. I think we'll get there. | 23:36 |
| flaper87 | and I really want to get there | 23:36 |
| flaper87 | As said before, I would like transports to be stupid logic wise and smart protocol wise | 23:37 |
| kgriffs | ok, but we also need to not compromise on performance and user experience to get there | 23:37 |
| flaper87 | that will help us with avoiding backward incompatible changes between the transport and the api, it'll make transport forward compatible etc | 23:38 |
| kgriffs | sure, it sounds really nice, don't get me wrong! | 23:38 |
| flaper87 | kgriffs: I don't think it'll compromise UX. I agree about the performance concern, though | 23:38 |
| kgriffs | I just want to take it iteratively | 23:38 |
| kgriffs | take a step, experiment, get results, feed that into the next cycle | 23:38 |
| flaper87 | kgriffs: yeah, and please, doubt about whatever idea comes to my non-human mind. :) | 23:39 |
| kgriffs | LOl | 23:39 |
| flaper87 | and I agree, we should get there iteratively | 23:39 |
| kgriffs | ok | 23:39 |
| kgriffs | I think that a lot of things will be easier to reason about once we put the API layer in place and start pushing logic down into it | 23:39 |
| flaper87 | We, as in cpallares and cpallares and cpallares, are starting to work on the reousrces needed by the API layer. That is, schema, response objects, request objects etc | 23:40 |
| kgriffs | flaper87: just to confirm | 23:40 |
| *** oz_akan_ has quit IRC | 23:40 | |
| kgriffs | the idea is sort of create a domain-specific WSGI | 23:40 |
| kgriffs | where we normalize everything to standard request and response objects | 23:40 |
| flaper87 | exactly | 23:41 |
| kgriffs | kk | 23:41 |
| kgriffs | sounds pretty awesome | 23:41 |
| kgriffs | and something that may be nice to do as a standalone library - i mean, parts of it | 23:41 |
| *** oz_akan_ has joined #openstack-marconi | 23:41 | |
| kgriffs | maybe even a PEP. :p | 23:41 |
| flaper87 | Btw, I had another idea but this time is related to messages and storage. I'll share it tomorrow. I'll go and eat a book now | 23:42 |
| flaper87 | kgriffs: yeah, that'd be awesome | 23:42 |
| kgriffs | kk | 23:42 |
| flaper87 | also, if I stay here 1 more second, I'll end up buying another lens | 23:42 |
| kgriffs | flaper87: would it be weird to turn this idea on it's head and translate from X transport to WSGI? | 23:42 |
| kgriffs | I guess that would be slow | 23:42 |
| alcabrera | flaper87: +1 for eating books | 23:43 |
| kgriffs | flaper87: LOL | 23:43 |
| ametts | flaper87: Dude... don't you ever sleep? | 23:43 |
| flaper87 | kgriffs: that's kinda of what the websocket transport does, it's a bit slow! | 23:43 |
| flaper87 | ametts: nope :D | 23:43 |
| flaper87 | it's one of the benefits of not being human | 23:43 |
| kgriffs | flaper87: right, I remember that now - there is uwsgi as well | 23:43 |
| kgriffs | anyway, just want to make sure we aren't in the throes of NIH syndrome | 23:44 |
| flaper87 | lol @ NIH | 23:45 |
| flaper87 | kgriffs: btw, latest cache patch is up:https://review.openstack.org/#/c/32544/ | 23:50 |
| flaper87 | not sure why py27 is failing. "It works on my laptop" | 23:50 |
| kgriffs | lol, thanks - I noticed that and have it on my todo list to review | 23:51 |
| *** oz_akan_ has quit IRC | 23:51 | |
| flaper87 | I don't get it: http://paste.openstack.org/show/55095/ | 23:54 |
| flaper87 | :( | 23:54 |
| flaper87 | anyway, I'll figure that out tomorrow! | 23:57 |
| flaper87 | by guys, have a good one! | 23:57 |
| * flaper87 back to books | 23:57 | |
| *** flaper87 is now known as flaper87|afk | 23:58 | |
| alcabrera | I'm out for the night, as well. | 23:58 |
| alcabrera | Take care, guys. :) | 23:59 |
| *** alcabrera has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!