| *** rama_y has quit IRC | 00:04 | |
| *** markvoelker has joined #openstack-performance | 01:13 | |
| *** zz_dimtruck is now known as dimtruck | 01:16 | |
| *** markvoelker has quit IRC | 01:18 | |
| *** arnoldje has joined #openstack-performance | 02:03 | |
| *** openstackgerrit has quit IRC | 02:47 | |
| *** openstackgerrit has joined #openstack-performance | 02:48 | |
| *** rama_y has joined #openstack-performance | 02:59 | |
| *** dimtruck is now known as zz_dimtruck | 03:28 | |
| *** furlongm has quit IRC | 03:41 | |
| *** furlongm has joined #openstack-performance | 03:42 | |
| *** arnoldje has quit IRC | 05:34 | |
| *** rama_y has quit IRC | 05:44 | |
| *** faizy has joined #openstack-performance | 06:32 | |
| *** openstackgerrit has quit IRC | 06:48 | |
| *** openstackgerrit has joined #openstack-performance | 06:48 | |
| *** tinbar has joined #openstack-performance | 07:17 | |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** openstackgerrit has joined #openstack-performance | 07:49 | |
| *** haukebruno has joined #openstack-performance | 08:35 | |
| *** dmk0202 has joined #openstack-performance | 08:37 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** openstackgerrit has joined #openstack-performance | 08:48 | |
| *** markvoelker has joined #openstack-performance | 09:53 | |
| *** ad_rien_ has joined #openstack-performance | 10:26 | |
| *** ad_rien_ has quit IRC | 10:29 | |
| *** ad_rien_ has joined #openstack-performance | 10:41 | |
| *** ad_rien_ has quit IRC | 10:46 | |
| *** ad_rien_ has joined #openstack-performance | 11:16 | |
| *** faizy has quit IRC | 11:53 | |
| *** markvoelker has quit IRC | 11:56 | |
| *** markvoelker has joined #openstack-performance | 12:12 | |
| *** zz_dimtruck is now known as dimtruck | 13:22 | |
| *** arnoldje has joined #openstack-performance | 13:29 | |
| *** dimtruck is now known as zz_dimtruck | 13:34 | |
| *** zz_dimtruck is now known as dimtruck | 13:36 | |
| *** arnoldje has quit IRC | 13:48 | |
| *** dimtruck is now known as zz_dimtruck | 14:03 | |
| *** zz_dimtruck is now known as dimtruck | 14:05 | |
| *** dimtruck is now known as zz_dimtruck | 14:16 | |
| *** arnoldje has joined #openstack-performance | 14:24 | |
| *** rama_y has joined #openstack-performance | 14:26 | |
| *** asingh has joined #openstack-performance | 14:27 | |
| *** ad_rien_ has quit IRC | 14:55 | |
| *** ad_rien_ has joined #openstack-performance | 14:56 | |
| *** ad_rien_ has quit IRC | 15:00 | |
| *** ad_rien_ has joined #openstack-performance | 15:03 | |
| *** zz_dimtruck is now known as dimtruck | 15:03 | |
| *** ad_rien_ has quit IRC | 15:08 | |
| *** faizy has joined #openstack-performance | 15:10 | |
| *** haukebruno has quit IRC | 15:18 | |
| *** rama_y has quit IRC | 15:19 | |
| *** tinbar has quit IRC | 15:23 | |
| *** faizy has quit IRC | 15:24 | |
| *** dmk0202 has quit IRC | 15:29 | |
| *** rama_y has joined #openstack-performance | 16:22 | |
| openstackgerrit | Alexandr Nevenchannyy proposed openstack/performance-docs: [WIP] OpenStack reliability test plan https://review.openstack.org/326598 | 16:27 |
|---|---|---|
| *** ad_rien_ has joined #openstack-performance | 16:28 | |
| openstackgerrit | Alexandr Nevenchannyy proposed openstack/performance-docs: [WIP] OpenStack reliability test plan https://review.openstack.org/326598 | 16:32 |
| *** Guest47242 is now known as med_ | 16:41 | |
| *** med_ has joined #openstack-performance | 16:41 | |
| *** ad_rien_ has quit IRC | 17:06 | |
| *** ad_rien_ has joined #openstack-performance | 17:08 | |
| *** ad_rien_ has quit IRC | 17:10 | |
| *** anevenchannyy has joined #openstack-performance | 17:23 | |
| *** ad_rien_ has joined #openstack-performance | 17:28 | |
| *** dimtruck is now known as zz_dimtruck | 17:30 | |
| *** ad_rien_ has quit IRC | 17:31 | |
| *** zz_dimtruck is now known as dimtruck | 17:33 | |
| *** greghaynes has joined #openstack-performance | 17:36 | |
| SpamapS | klindgren: Hi! I have a question about glance. If you don't mind me asking, how do you handle user image uploads? | 17:36 |
| SpamapS | harlowja: ^ any thoughts? | 17:36 |
| harlowja | who huh | 17:36 |
| harlowja | lol | 17:36 |
| harlowja | lol | 17:36 |
| harlowja | \o\ | 17:36 |
| harlowja | will look over | 17:37 |
| klindgren | SpamapS, we don't | 17:37 |
| harlowja | gotta get these oslo releases out | 17:37 |
| klindgren | Our internal cloud everything is based upon images that we provide | 17:37 |
| klindgren | so we have a image provisioning process that kicks a new image out every month with the latest patches/local fixes | 17:38 |
| klindgren | the public cloud team does something similar | 17:38 |
| SpamapS | klindgren: so no custom user images of any kind? | 17:38 |
| SpamapS | we're trying to figure out if anybody other tha RAX is running glance uploads at scale. | 17:38 |
| klindgren | opn the public cloud side people can take that image - customize it, snapshot it and use that to boot vm's off of | 17:38 |
| SpamapS | yeah, snapshotting a running box isn't image uploads though ;) | 17:39 |
| klindgren | um | 17:39 |
| klindgren | yea it is | 17:39 |
| SpamapS | It's fraught with problems. | 17:39 |
| SpamapS | Like how to move that to another cloud. | 17:39 |
| klindgren | we have a bash script to stream images from one glance server to another | 17:39 |
| SpamapS | And to other tenants | 17:39 |
| klindgren | in our public cloud | 17:39 |
| SpamapS | yes, but how do I build an image on your cloud, and run it on my private cloud? | 17:40 |
| klindgren | for us - you dont | 17:41 |
| klindgren | atleast I am not aware of our public team providing that functionality to end users | 17:42 |
| SpamapS | klindgren: do you guys provide access to the OpenStack API's at all, or just the godaddy ones? | 17:42 |
| klindgren | but we do have multiple regions, they do have a thing to allow you to move images/snapshots between regions | 17:42 |
| klindgren | public cloud - is just godaddy api/ui | 17:42 |
| klindgren | which they bascially proxy back to the underling openstack api's to do the real work | 17:43 |
| SpamapS | Yeah, so that explains it. Not really interested in enabling people to use things on their own clouds. :) | 17:43 |
| klindgren | I wouldn't say that - so much as our customers haven't actually asked for it? | 17:44 |
| SpamapS | klindgren: It's a chicken and egg problem. If you don't have OpenStack API's, you don't have customers that use them. :) | 17:46 |
| SpamapS | and your customers aren't going to buy a private cloud that uses them.. because they only have godaddy API's | 17:47 |
| * klindgren *shrugs* | 17:48 | |
| klindgren | The decisions on what apis and what features get exposed to customers is above my pay grade | 17:48 |
| klindgren | but to answer your question re: glance we still have lots of problems with glance. | 17:49 |
| klindgren | lack of quotas, sharing images between projects is highly painful | 17:49 |
| SpamapS | Yeah, I don't mean to call into question Godaddy's business practices here. We are building an OpenStack API based public cloud, and thus, are interested in how people scale glance's API. | 17:50 |
| klindgren | snapshots stream through the glance-api servers so I have to scale those rightly with my object storage servers that store the actual data | 17:50 |
| SpamapS | klindgren: yeah it's weird, I'm quickly coming to the conclusion that nobody is actually contributing to glance.. everybody is just working around it. | 17:50 |
| klindgren | unless I run a glance-api server per HV | 17:50 |
| klindgren | so their is this: | 17:51 |
| klindgren | https://etherpad.openstack.org/p/newton-glance-and-ops-midcycle-sync | 17:51 |
| klindgren | we also have the problem of how to get images from one openstack region to another | 17:51 |
| klindgren | since our openstack installs are shared nothing | 17:51 |
| klindgren | we also have an issue of how can you mark images as being from a trusted source | 17:52 |
| klindgren | like "this image comes directly from your hosting provider" | 17:52 |
| klindgren | also image lifecycle support on how can remove old images from glance without breaking functionality in nova | 17:53 |
| SpamapS | Syncing between regions isn't a problem if you allow uploads, because users just build and upload their image where they need it. | 17:54 |
| SpamapS | (Even if their build is to snapshot, and download) | 17:54 |
| klindgren | well we upload images for our company | 17:54 |
| klindgren | so its a problem for us :-) | 17:54 |
| SpamapS | Trusted image sources seems like a feature that should just be baked into glance. Like you should just be able to mark a tenant as "trusted" and everybody gets their images. | 17:54 |
| klindgren | also public cloud they have a bash script that streams a glance download | galnce upload | 17:55 |
| klindgren | between the two openstack installs to move images at customer requests | 17:55 |
| klindgren | Also quota | 17:55 |
| SpamapS | Yeah, quotas is punted to backends, which seems wrong. | 17:55 |
| SpamapS | klindgren: you use the swift backend? | 17:56 |
| klindgren | like if you enable quota that impacts the admin user as well as end users | 17:56 |
| klindgren | we use ceph (radosgw) | 17:56 |
| SpamapS | ++ :) | 17:56 |
| klindgren | we are actually using the S3 backend due to some hilarity in the way glance + swift on ceph works | 17:56 |
| klindgren | where we get timeouts trying to delete images | 17:56 |
| SpamapS | so radosgw doens't have quotas does it? | 17:56 |
| klindgren | it does but the error is not friendly | 17:57 |
| SpamapS | ah if it's S3, then you have all the images in one tenant right? | 17:57 |
| klindgren | its jsut like -welp your upload failed | 17:57 |
| klindgren | that sucks | 17:57 |
| harlowja | ya, don't get me started on the 'godaddy public cloud having its own API' SpamapS :-P | 17:57 |
| harlowja | i'm in the bucket of people that u are in, as afaik is klindgren , lol | 17:57 |
| SpamapS | harlowja: I don't judge. :) | 17:57 |
| harlowja | lol | 17:57 |
| SpamapS | It's just a choice. | 17:57 |
| harlowja | ya ya | 17:58 |
| klindgren | I am jsut happy I dont have to get openstack to work with all of Godaddys billing and single sign on stuff | 17:58 |
| klindgren | so in ceph I am pretty sure we jsut have a single "bucket" for all of glance to upload to | 17:59 |
| klindgren | and with radosgw you can have both S3 and swift see the same files | 17:59 |
| klindgren | since its jsut api abstractions | 17:59 |
| klindgren | SO our public cloud team decided to productize snapshots as "backups" | 18:00 |
| klindgren | so we have tons of snapshots happening all the time in our public cloud | 18:00 |
| klindgren | so glance is always seeing image uploads | 18:01 |
| SpamapS | klindgren: right, but they're from local sources that you control, so the scaling while not cheap, is straight forward. | 18:02 |
| klindgren | SpamapS, what is concerning you about external uploads to your glance infra? Seems to me any cloud of sufficent size the internal snapshot traffic would be significantly higher than the external upload traffic? | 18:09 |
| SpamapS | klindgren: It's the old 56k modem problem. | 18:13 |
| SpamapS | klindgren: sometimes slow connections are harder to handle than fast ones. | 18:13 |
| SpamapS | And users make horrible HTTP libraries. | 18:14 |
| SpamapS | Swift has had time to mature to accept that all and protect itself from resource starvation by the morons and bad guys. | 18:14 |
| klindgren | so your more concerned about the fact that people can hithte publicly exposed glance servers and starve them out via slow connections | 18:18 |
| *** arnoldje has quit IRC | 18:18 | |
| SpamapS | klindgren: I do kind of wonder if a decent WAF would mitigate most of it. | 18:18 |
| klindgren | or drive up resource consumption on glance | 18:19 |
| klindgren | since its a glance thread that need to handle this | 18:19 |
| SpamapS | klindgren: right, what I can see is one user with a poorly made upload script and a crappy connection using up all the glance processes/threads. | 18:19 |
| klindgren | iirc rax does this by having a set of servers specifically for user facing API's then anotehr set of infra for the other items like internal cloud kicked of snapshots and the like | 18:19 |
| SpamapS | Which is definitely something a WAF mitigates. | 18:19 |
| SpamapS | klindgren: RAX doesn't allow uploads on glance. | 18:20 |
| SpamapS | You upload to swift. | 18:20 |
| klindgren | so you upload to swift then tell glance where your image is | 18:20 |
| SpamapS | Then tell glance where to import the image. | 18:20 |
| klindgren | kk | 18:20 |
| SpamapS | And that code is not available in OpenStack. | 18:20 |
| SpamapS | because it's based on celery and "politics" | 18:20 |
| klindgren | eh I thought the tell glance where the iamge piece was in there. | 18:20 |
| SpamapS | klindgren: sorry, that sentence didn't parse. | 18:22 |
| klindgren | so you can do a glance image-create image name | 18:22 |
| klindgren | then a glance image-update and do --location to tell it where the image is stored in swift | 18:22 |
| klindgren | --location IMAGE_URL | 18:23 |
| klindgren | The URL where the data for this image resides. For example, if the image data is stored in swift, you could specify swift://account:key@example.com/container/obj. | 18:23 |
| SpamapS | klindgren: no that is glance v1 | 18:23 |
| SpamapS | which is deprecated | 18:23 |
| SpamapS | and wasn't intended as a user facing API | 18:23 |
| SpamapS | klindgren: I don't believe RAX allows that for users | 18:24 |
| SpamapS | klindgren: what you do in RAX is use glance v2 API and create an import task | 18:24 |
| klindgren | so update still has: urlplainxsd:string | 18:24 |
| klindgren | The URL to access the image file kept in external store. | 18:24 |
| klindgren | in the v2 api | 18:25 |
| SpamapS | klindgren: https://github.com/openstack-infra/shade/blob/master/shade/openstackcloud.py#L2539-L2758 | 18:25 |
| SpamapS | that's the shade code that does the uploads for clouds .. either with PUT through glance, or task through swift | 18:25 |
| SpamapS | klindgren: I know that API is still there, but dig through openstack-dev, you'll find some rage out there about the fact that it is basically ignored and doesn't work. | 18:26 |
| SpamapS | public clouds can't really do it that way | 18:26 |
| klindgren | kk - I haven't looked at it much - as it hasn't been an issue or requested feature for us | 18:27 |
| SpamapS | Yeah, it's unfortunately, a mess, and just further evidence that for whatever reason, glance and image upload isn't getting the attention it needs. | 18:28 |
| SpamapS | I'm just convinced that nobody is really running a public image upload service of sufficient size to care, except RAX, who has a custom solution. | 18:28 |
| SpamapS | But I'll reach out to vexx and ovh .. they're not THAT small | 18:29 |
| *** ad_rien_ has joined #openstack-performance | 18:30 | |
| *** ad_rien_ has quit IRC | 18:33 | |
| *** ad_rien_ has joined #openstack-performance | 18:34 | |
| *** ad_rien_ has quit IRC | 18:38 | |
| *** ad_rien_ has joined #openstack-performance | 18:43 | |
| *** ad_rien_ has quit IRC | 18:47 | |
| *** ad_rien_ has joined #openstack-performance | 18:48 | |
| *** ad_rien_ has quit IRC | 18:53 | |
| *** harlowja has quit IRC | 19:12 | |
| *** harlowja has joined #openstack-performance | 19:15 | |
| *** ad_rien_ has joined #openstack-performance | 19:31 | |
| *** ad_rien_ has quit IRC | 19:35 | |
| *** dimtruck is now known as zz_dimtruck | 19:54 | |
| *** zz_dimtruck is now known as dimtruck | 19:57 | |
| *** asingh has quit IRC | 20:39 | |
| *** dimtruck is now known as zz_dimtruck | 20:46 | |
| *** openstackgerrit has quit IRC | 20:48 | |
| *** openstackgerrit has joined #openstack-performance | 20:48 | |
| *** ad_rien_ has joined #openstack-performance | 20:51 | |
| *** ad_rien_ has quit IRC | 20:53 | |
| *** rama_y has quit IRC | 20:59 | |
| *** asingh has joined #openstack-performance | 21:00 | |
| *** arnoldje has joined #openstack-performance | 21:15 | |
| *** asingh has quit IRC | 21:53 | |
| *** asingh has joined #openstack-performance | 21:54 | |
| *** rama_y has joined #openstack-performance | 22:08 | |
| *** zz_dimtruck is now known as dimtruck | 22:35 | |
| *** asingh has quit IRC | 22:44 | |
| *** asingh has joined #openstack-performance | 22:45 | |
| *** markvoelker has quit IRC | 23:13 | |
| *** markvoelker has joined #openstack-performance | 23:44 | |
| *** markvoelker has quit IRC | 23:49 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!