| honza | Trying to rebuild the undercloud and mirrors are down. Giving up for now, will try again in the morning. | 00:00 |
|---|---|---|
| honza | thanks for the assistance! | 00:00 |
| flwang | honza: good luck :) | 00:02 |
| *** yanyanhu has joined #openstack-zaqar | 01:40 | |
| *** lhx__ has joined #openstack-zaqar | 01:44 | |
| *** zhurong has joined #openstack-zaqar | 02:10 | |
| openstackgerrit | Merged openstack/zaqar-ui master: Imported Translations from Zanata https://review.openstack.org/457539 | 02:17 |
| openstackgerrit | Merged openstack/zaqar-ui master: Updated from global requirements https://review.openstack.org/457738 | 02:18 |
| *** lhx__ has quit IRC | 02:36 | |
| *** shu-mutou has joined #openstack-zaqar | 03:00 | |
| *** zhurong has quit IRC | 03:21 | |
| *** zhurong has joined #openstack-zaqar | 03:28 | |
| *** zhurong has quit IRC | 03:31 | |
| openstackgerrit | Merged openstack/zaqar-ui master: Refresh test environment https://review.openstack.org/456464 | 04:29 |
| *** zhurong has joined #openstack-zaqar | 04:31 | |
| *** nadya has joined #openstack-zaqar | 05:46 | |
| *** mpanetta has quit IRC | 05:47 | |
| *** nadya has quit IRC | 05:48 | |
| *** mpanetta has joined #openstack-zaqar | 05:48 | |
| *** zhurong has quit IRC | 06:12 | |
| *** rcernin has joined #openstack-zaqar | 06:28 | |
| *** pcaruana has joined #openstack-zaqar | 06:40 | |
| *** jtomasek has quit IRC | 06:45 | |
| *** tesseract has joined #openstack-zaqar | 06:56 | |
| *** zhurong has joined #openstack-zaqar | 07:12 | |
| *** jtomasek has joined #openstack-zaqar | 07:14 | |
| *** lhx__ has joined #openstack-zaqar | 07:26 | |
| *** yanyanhu has quit IRC | 07:38 | |
| *** lhx__ has quit IRC | 07:57 | |
| *** nadya has joined #openstack-zaqar | 08:05 | |
| *** yanyanhu has joined #openstack-zaqar | 08:06 | |
| *** nadya has quit IRC | 08:06 | |
| *** nadya has joined #openstack-zaqar | 08:07 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:40 | |
| *** yanyanhu has quit IRC | 09:54 | |
| *** nadya has quit IRC | 10:01 | |
| *** mpanetta has quit IRC | 10:15 | |
| *** mpanetta has joined #openstack-zaqar | 10:16 | |
| *** zhurong has quit IRC | 10:19 | |
| *** nadya has joined #openstack-zaqar | 10:45 | |
| *** mpanetta_ has joined #openstack-zaqar | 10:56 | |
| *** mpanetta has quit IRC | 10:58 | |
| *** zhurong has joined #openstack-zaqar | 11:17 | |
| *** openstackgerrit has quit IRC | 11:32 | |
| *** lhx__ has joined #openstack-zaqar | 11:42 | |
| *** lhx__ has quit IRC | 11:50 | |
| *** lhx__ has joined #openstack-zaqar | 11:50 | |
| *** lhx__ has quit IRC | 12:56 | |
| *** lhx__ has joined #openstack-zaqar | 12:57 | |
| *** lhx__ has quit IRC | 13:00 | |
| *** lhx__ has joined #openstack-zaqar | 13:00 | |
| *** lhx__ has quit IRC | 13:03 | |
| *** ig0r_ has joined #openstack-zaqar | 13:34 | |
| *** mpanetta_ has quit IRC | 14:52 | |
| *** mpanetta has joined #openstack-zaqar | 14:53 | |
| *** rcernin has quit IRC | 15:01 | |
| *** pcaruana has quit IRC | 15:14 | |
| *** jtomasek_ has joined #openstack-zaqar | 15:27 | |
| *** jtomasek has quit IRC | 15:27 | |
| *** nadya has quit IRC | 15:29 | |
| *** ig0r_ has quit IRC | 16:01 | |
| *** zhurong has quit IRC | 16:11 | |
| *** tesseract has quit IRC | 16:34 | |
| *** rakhmerov has quit IRC | 17:14 | |
| *** jtomasek_ has quit IRC | 17:17 | |
| *** rakhmerov has joined #openstack-zaqar | 17:19 | |
| *** jtomasek has joined #openstack-zaqar | 17:22 | |
| *** ig0r_ has joined #openstack-zaqar | 17:25 | |
| *** nadya has joined #openstack-zaqar | 17:55 | |
| *** nadya has quit IRC | 18:01 | |
| flwang | therve: ping re the swift pooling patch | 21:48 |
| *** rbrady has quit IRC | 21:49 | |
| flwang | flaper87: any luck you're around? | 21:56 |
| flwang | kgriffs: and you? | 21:56 |
| kgriffs | hi flwang, what's up? | 21:58 |
| flwang | kgriffs: i'm thinking adding quota for the message count (free messages) in a queue | 21:58 |
| flwang | but I just realized it will absolutely impact the performance of posting | 21:59 |
| flwang | do you have any comments for this? | 21:59 |
| flwang | suggestion/advice/whatever :) thanks | 21:59 |
| flwang | because allow unlimited messages is unacceptable for a public cloud | 22:00 |
| flwang | kgriffs: how does rackspace handle this now? | 22:00 |
| kgriffs | we've only done basic rate limiting so I can only speak generally | 22:03 |
| kgriffs | With performance, measuring is key, so I'd suggest starting with a naive proof of concept to measure the performance impact | 22:04 |
| kgriffs | That will give you a baseline so you know how much you need to optimize | 22:04 |
| kgriffs | beyond that, you'll want to pursue the usual performance engineering strategies, e.g.: | 22:05 |
| kgriffs | 1. Chunk your I/O requests if you can. For example, add a message, update the counter, and return the counter value in a single request to the data store. Stored procedures / Redis Lua can help with this. | 22:06 |
| kgriffs | 2. Avoid I/O if you can. Can you get away with not checking the quota on every request? | 22:06 |
| flwang | #2 is what i'm thinking, but i found it's hard | 22:07 |
| flwang | i'm thinking if i can use the pipeline to work around it | 22:07 |
| flwang | just like the notification pipeline | 22:07 |
| kgriffs | 3. Reduce the cost of I/O to handle the quota. Put the counter or whatever is a very fast data store close to the webheads. Doesn't have to be very durable (trade durability for speed) | 22:07 |
| flwang | after posted the messages, tagged something | 22:07 |
| kgriffs | yeah, unfortunately there aren't any super easy solutions to this because you have to keep track of some state shared between the web heads, while minimizing performance impact. | 22:09 |
| kgriffs | You might try handling the quota info locally to each web server by dividing a queue's quota by the number of web servers that could possible serve requests for that queue | 22:10 |
| kgriffs | but that can be tricky as well... you need a way to be able dynamically calculate that number based on the quota and number of servers | 22:11 |
| *** openstackgerrit has joined #openstack-zaqar | 22:11 | |
| openstackgerrit | Merged openstack/zaqar master: Fix failure when adding swift/redis as pool https://review.openstack.org/453439 | 22:11 |
| flwang | kgriffs: yep, i'm thinking save current message count in the queue infomation, because currently eachtime when we post messages, zaqar need to get the queue info. so with that way, we won't introduce new call at least | 22:12 |
| flwang | and meanwhile, create a new pipeline just like the notifier, to calculate messages after post, but unfortunately, the way is not accurate, because the free message number is changing all the time | 22:13 |
| flwang | given message TTL | 22:13 |
| openstackgerrit | Merged openstack/zaqar master: Remove enabled option for CORS https://review.openstack.org/451090 | 22:16 |
| flwang | kgriffs: btw, is rackspace using mongodb as the backend? | 22:18 |
| flwang | are you happy with that? | 22:18 |
| kgriffs | OK, sounds like you are on the right track. You might want to consider how hard of a quota you need. Can you get away with a less accurate quota? Would it still be good enough to meet your goal of putting in a quota in the first place? If you can guarantee a reasonable range, that may be sufficient, as long as you communicate to the customer the lower end of the range, if they go over it may or may not be a problem. | 22:20 |
| flwang | kgriffs: yep, that's what i'm thinking | 22:21 |
| kgriffs | re mongodb, that is what we launched on, yes. | 22:22 |
| flwang | because IMHO, a quota for messaging system is not to block the real usage of customer, but some bad hacking | 22:22 |
| kgriffs | in that case you definitely don't need to be super precise | 22:22 |
| flwang | so i don't think cloud provider will really care about another 10K messages if the quota is 100,000 | 22:23 |
| kgriffs | on the topic of data stores, it really comes down to durability, correctness, and latency | 22:23 |
| flwang | kgriffs: yep, it's not the same case like launching a server | 22:24 |
| kgriffs | Different workloads require different mixes of those three | 22:24 |
| kgriffs | if I want something super correct and durable and scalable I could use something based on Paxos or Raft (e.g., Riak, CockroachDB). | 22:25 |
| kgriffs | But latency suffers | 22:25 |
| flwang | kgriffs: yep, you're right | 22:25 |
| kgriffs | If I want to sacrifice some guarantee around deliverability or durability, I can scale and write things in parallel, thus reducing latency, improving availability | 22:26 |
| flwang | for our cloud, the rough plan is using swift as the first pool and after we get a stable mongo cluster, then add it as the 2nd pool | 22:26 |
| kgriffs | oh, interesting | 22:26 |
| flwang | do you think it's crazy? | 22:26 |
| flwang | because currently our mongo cluster is fully used by ceilometer | 22:27 |
| kgriffs | I spoke to the swift folks a few years back and they were really interested in Zaqar precisely because it would encourage people to stop using swift as a queue. :p | 22:27 |
| flwang | we would like to upgrade to gnocchi and then release the mongodb | 22:27 |
| flwang | for zaqar | 22:27 |
| kgriffs | gotcha | 22:27 |
| flwang | what do you mean 'stop using swift as a queue'? | 22:28 |
| kgriffs | btw, you can sort of cheat like Google did with spanner and use highly accurate clocks (syncing to local atomic clocks) to enforce a total order across the messages, but even then there is a little bit of variance. Their solution? Dynamically slow down requests so they are outside the collision window. | 22:29 |
| kgriffs | flwang I guess people were basically uploading tiny objects as "messages" and then removing them... I guess it caused problems for them or something. You may want to follow up with them. | 22:30 |
| flwang | kgriffs: i see. but for now, we're not tagging the swift driver as a FIFO backend | 22:31 |
| flwang | so should be fine | 22:31 |
| flwang | kgriffs: yep, i read the google pub/sub design/architecture many many times | 22:31 |
| flwang | kgriffs: TBH, i was wondering the backend used by AWS SQS and Google's Pub/Sub | 22:32 |
| flwang | if you got a chance to redesign zaqar, what do you want to change? | 22:33 |
| openstackgerrit | Feilong Wang proposed openstack/zaqar master: Support dead letter queue https://review.openstack.org/333657 | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!