*** catintheroof has joined #openstack-swift | 00:00 | |
*** catintheroof has quit IRC | 00:01 | |
*** catintheroof has joined #openstack-swift | 00:04 | |
*** catintheroof has quit IRC | 00:33 | |
*** catintheroof has joined #openstack-swift | 00:39 | |
*** tone_z has joined #openstack-swift | 00:50 | |
tone_z | tone_zrt | 01:02 |
---|---|---|
*** tone_z is now known as tone_zrt | 01:03 | |
*** cam has quit IRC | 01:05 | |
*** mingyu has joined #openstack-swift | 01:07 | |
*** catintheroof has quit IRC | 01:09 | |
*** mingyu has quit IRC | 01:13 | |
*** zhurong has joined #openstack-swift | 01:50 | |
*** aleph1 has quit IRC | 01:51 | |
*** JimCheung has quit IRC | 01:55 | |
*** JimCheung has joined #openstack-swift | 01:55 | |
*** bkopilov_ has quit IRC | 01:58 | |
*** JimCheung has quit IRC | 02:00 | |
*** jamielennox is now known as jamielennox|away | 02:03 | |
*** zhurong has quit IRC | 02:07 | |
*** JimCheung has joined #openstack-swift | 02:14 | |
*** jamielennox|away is now known as jamielennox | 02:17 | |
*** mingyu has joined #openstack-swift | 02:18 | |
*** JimCheung has quit IRC | 02:19 | |
*** flwang1 has quit IRC | 02:21 | |
*** flwang has joined #openstack-swift | 02:25 | |
*** arunman has joined #openstack-swift | 02:37 | |
*** psachin has joined #openstack-swift | 02:51 | |
*** Mype has joined #openstack-swift | 03:06 | |
*** mingyu has quit IRC | 03:06 | |
Mype | Hello, to add object to Swift do I need to generate a token each time ? Or is there another solution like an one time token setup or something like that ? Thanks in advance | 03:07 |
*** vint_bra has joined #openstack-swift | 03:09 | |
klrmn | Mype: there are several possible auth systems, but most of them can provide a token that will last some amount of time | 03:17 |
*** mingyu has joined #openstack-swift | 03:18 | |
Mype | So the best way to use OpenStack Swift is to generate a token for each request ? | 03:19 |
*** bkopilov_ has joined #openstack-swift | 03:20 | |
*** winggundamth has joined #openstack-swift | 03:22 | |
*** dja has quit IRC | 03:24 | |
*** dja has joined #openstack-swift | 03:27 | |
*** zhurong has joined #openstack-swift | 03:37 | |
*** zhurong has quit IRC | 03:40 | |
*** mingyu has quit IRC | 03:46 | |
*** mingyu has joined #openstack-swift | 03:48 | |
*** links has joined #openstack-swift | 03:49 | |
*** mingyu has quit IRC | 03:52 | |
*** mingyu has joined #openstack-swift | 03:59 | |
*** mingyu has quit IRC | 04:04 | |
*** vint_bra has quit IRC | 04:13 | |
*** clayg has quit IRC | 04:22 | |
*** chlong has joined #openstack-swift | 04:22 | |
*** gyee has quit IRC | 04:23 | |
*** mingyu has joined #openstack-swift | 04:23 | |
*** mingyu has quit IRC | 04:25 | |
*** mingyu has joined #openstack-swift | 04:27 | |
openstackgerrit | Mahati Chamarthy proposed openstack/swift master: Always check for unexpected requests in mocked_http_conn https://review.openstack.org/461483 | 04:29 |
mahatic | commit msg typo edit ^^ | 04:29 |
*** zhurong has joined #openstack-swift | 04:37 | |
*** mingyu has quit IRC | 04:47 | |
*** mingyu has joined #openstack-swift | 04:53 | |
*** chsc has joined #openstack-swift | 04:53 | |
*** chsc has joined #openstack-swift | 04:53 | |
*** klrmn has quit IRC | 04:54 | |
*** flwang has quit IRC | 04:55 | |
*** chsc has quit IRC | 04:58 | |
*** mingyu has quit IRC | 05:05 | |
*** winggundamth has quit IRC | 05:45 | |
*** newmember has joined #openstack-swift | 05:52 | |
*** ChubYann has quit IRC | 05:52 | |
*** mingyu has joined #openstack-swift | 06:05 | |
*** mingyu has quit IRC | 06:11 | |
*** tovin07_ has joined #openstack-swift | 06:12 | |
*** tovin07 has joined #openstack-swift | 06:13 | |
*** mingyu has joined #openstack-swift | 06:17 | |
*** jaosorior_away is now known as jaosorior | 06:19 | |
*** newmember has quit IRC | 06:26 | |
*** pcaruana has joined #openstack-swift | 06:27 | |
*** hseipp has joined #openstack-swift | 06:39 | |
*** hseipp has left #openstack-swift | 06:40 | |
*** bkopilov has joined #openstack-swift | 06:41 | |
*** mingyu has quit IRC | 06:44 | |
*** mingyu has joined #openstack-swift | 06:46 | |
*** mingyu has quit IRC | 06:47 | |
*** mingyu has joined #openstack-swift | 06:47 | |
*** lchabert2 has joined #openstack-swift | 06:49 | |
*** JimCheung has joined #openstack-swift | 06:56 | |
*** JimCheung has quit IRC | 07:00 | |
*** mingyu has quit IRC | 07:01 | |
*** lchabert2 has quit IRC | 07:02 | |
*** Mype has quit IRC | 07:07 | |
*** mingyu has joined #openstack-swift | 07:11 | |
*** zhurong has quit IRC | 07:24 | |
*** geaaru has joined #openstack-swift | 07:26 | |
*** psachin has quit IRC | 07:29 | |
*** zhurong has joined #openstack-swift | 07:43 | |
*** adriant has quit IRC | 07:43 | |
acoles | good morning | 07:47 |
*** silor has joined #openstack-swift | 08:01 | |
*** cbartz has joined #openstack-swift | 08:04 | |
*** mingyu has quit IRC | 08:17 | |
*** mingyu has joined #openstack-swift | 08:23 | |
*** joeljwright has joined #openstack-swift | 08:23 | |
*** ChanServ sets mode: +v joeljwright | 08:23 | |
*** mingyu has quit IRC | 08:25 | |
*** mingyu has joined #openstack-swift | 08:26 | |
*** mingyu has quit IRC | 08:34 | |
mahatic | acoles: o/ | 08:34 |
*** mingyu has joined #openstack-swift | 08:34 | |
*** mingyu has quit IRC | 08:45 | |
*** zhurong has quit IRC | 08:47 | |
openstackgerrit | Ngo Quoc Cuong proposed openstack/swift master: Trivial fix warnings in docstring https://review.openstack.org/462040 | 08:53 |
*** mingyu has joined #openstack-swift | 08:54 | |
*** mingyu has quit IRC | 08:57 | |
*** mingyu has joined #openstack-swift | 08:59 | |
*** zhurong has joined #openstack-swift | 09:01 | |
*** mingyu has quit IRC | 09:12 | |
*** mingyu has joined #openstack-swift | 09:19 | |
*** Jeffrey4l has quit IRC | 09:25 | |
*** Jeffrey4l has joined #openstack-swift | 09:27 | |
*** zhurong has quit IRC | 09:41 | |
*** flwang has joined #openstack-swift | 09:54 | |
*** tesseract has joined #openstack-swift | 09:56 | |
*** tovin07_ has quit IRC | 10:04 | |
*** mingyu has quit IRC | 10:17 | |
*** mingyu has joined #openstack-swift | 10:19 | |
*** zhurong has joined #openstack-swift | 10:19 | |
*** mvk has quit IRC | 10:22 | |
openstackgerrit | Ngo Quoc Cuong proposed openstack/swift master: Trivial fix warnings in docstring https://review.openstack.org/462040 | 10:23 |
*** zhurong has quit IRC | 10:30 | |
*** mingyu has quit IRC | 10:35 | |
openstackgerrit | Merged openstack/swift master: Use LogRecord.msg instead of LogRecord.message in tests https://review.openstack.org/461178 | 10:36 |
*** mingyu has joined #openstack-swift | 10:46 | |
openstackgerrit | Alistair Coles proposed openstack/swift master: Trivial fix for decrypter docstrings https://review.openstack.org/462075 | 10:48 |
*** mingyu has quit IRC | 10:55 | |
*** tone_zrt has quit IRC | 10:56 | |
*** mingyu has joined #openstack-swift | 11:02 | |
*** bkopilov has quit IRC | 11:03 | |
*** bkopilov_ has quit IRC | 11:04 | |
*** mingyu has quit IRC | 11:07 | |
*** mingyu has joined #openstack-swift | 11:09 | |
*** goutham has joined #openstack-swift | 11:14 | |
goutham | Hi all | 11:14 |
goutham | i need a small help | 11:15 |
goutham | i have setup devstack with swift... to save images with the help swift. | 11:15 |
goutham | glance images *** | 11:16 |
goutham | my devstack is up and glance images are also active but when i want to see in which container the images are stored im unable to fetch the details | 11:16 |
goutham | can anyone help?? | 11:16 |
acoles | goutham: can you elaborate on "im unable to fetch" - do you mena you don;t know what the container name is, or a request fails, or...? | 11:18 |
goutham | hi i mean the images are active and i cant find them in my local file system so i am hoping that the images are with swift | 11:19 |
goutham | how can i find them?? | 11:19 |
goutham | ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account ----> kept these things in my local.conf | 11:20 |
*** mvk has joined #openstack-swift | 11:20 | |
goutham | https://hastebin.com/ramehimepa.ini --- my glance_store section after succesful stack in glance-api.conf | 11:21 |
goutham | https://hastebin.com/amaqaboron.pl -- swift stat | 11:21 |
goutham | https://hastebin.com/itosexoxoh.ini ----> etc/glance/glance-swift-store.conf | 11:22 |
acoles | goutham: you'll probably do better asking in #openstack-glance, at least at first, to understand which *account* and container glance is using in swift to store images. IIRC glance can either putsimages in the user's swift account or in a glance service swift account, depending on config, but I can't tell you which way devstack is set up. | 11:23 |
goutham | ok i will ask them any person in particular who can help me out?? | 11:24 |
acoles | goutham: sorry I don't know anyone specifically on glance these days | 11:25 |
goutham | its okay thanks for the help acoles :) | 11:25 |
*** mingyu has quit IRC | 11:28 | |
*** mingyu has joined #openstack-swift | 11:29 | |
*** mingyu has quit IRC | 11:29 | |
*** mingyu has joined #openstack-swift | 11:48 | |
*** mingyu has quit IRC | 11:48 | |
*** pcaruana has quit IRC | 11:53 | |
*** pcaruana has joined #openstack-swift | 11:57 | |
*** links has quit IRC | 12:00 | |
goutham | acoles: it worked | 12:01 |
*** zhurong has joined #openstack-swift | 12:01 | |
acoles | goutham: great! so what was the answer? | 12:02 |
goutham | there was a mistake from my side in my /etc/glance/glance-swift-store.conf there is a parameter | 12:02 |
goutham | user = service:glance-swift | 12:02 |
goutham | so i sourced my openrc with source openrc glance-swift service | 12:03 |
goutham | then swift list gave a response | 12:03 |
goutham | glance | 12:03 |
goutham | swift list glance 08c2edb8-426e-4868-9661-739516eeae7f 08c2edb8-426e-4868-9661-739516eeae7f-00001 e25ac606-cf03-42f0-929b-b8e0f495f146 e25ac606-cf03-42f0-929b-b8e0f495f146-00001 f13d4fc4-4d6b-4591-917c-4e27d9f6a3f3 f13d4fc4-4d6b-4591-917c-4e27d9f6a3f3-00001 | 12:03 |
goutham | https://hastebin.com/apasijobop.go --> this gave the images list | 12:04 |
goutham | i have a doubt acoles why is that we have to items like <image-id> and <image-id-0001> what are these two things?? | 12:05 |
acoles | goutham: IDK. If I had to guess I'd say glance is uploading images as SLO and that 'id' is a manifest and 'id-0001' is a segment, in this case only one segment. | 12:10 |
*** vint_bra has joined #openstack-swift | 12:10 | |
acoles | goutham: try adding --lh to the swift list command and you'll get more details | 12:10 |
acoles | 'swift list glance --lh' | 12:10 |
goutham | yes it showed 32MB glance | 12:11 |
goutham | where can i find python-sdk for swift client any idea ?? | 12:11 |
acoles | goutham: https://github.com/openstack/python-swiftclient | 12:13 |
acoles | there's a link to docs on that page ^^ | 12:13 |
acoles | you can pip install python-swiftclient | 12:15 |
goutham | yes yes swift client is there i wanted python-sdk something like this https://docs.openstack.org/developer/python-novaclient/api.html\ | 12:17 |
goutham | https://docs.openstack.org/developer/python-novaclient/api.html *** | 12:17 |
*** klamath has joined #openstack-swift | 12:18 | |
*** klamath has quit IRC | 12:18 | |
*** klamath has joined #openstack-swift | 12:19 | |
acoles | goutham: https://docs.openstack.org/developer/python-swiftclient/client-api.html ? | 12:19 |
goutham | yes exactly what i wanted | 12:20 |
goutham | thanks acoles | 12:20 |
acoles | goutham: this layers more features on the basic client api https://docs.openstack.org/developer/python-swiftclient/service-api.html | 12:21 |
acoles | goutham: note thay are both part of same project as the CLI ^^ | 12:21 |
*** vint_bra has quit IRC | 12:22 | |
goutham | ok thanks acoles will get back to you in case of any help ... :) | 12:23 |
goutham | thanks for all the help | 12:23 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 12:24 |
*** catintheroof has joined #openstack-swift | 12:30 | |
*** f0o has joined #openstack-swift | 12:32 | |
f0o | Hi, I was wondering which TLS wrapper is recommended for running in front of swift-proxy. I usually use nginx but I'm open to alternatives. Any comments? | 12:33 |
tdasilva | f0o: I see a lot of recommendations for haproxy | 12:38 |
tdasilva | f0o: also, that's what tripleo (and I belive openstack-ansible) uses FWIW | 12:39 |
f0o | tdasilva: I think most use haproxy because it comes with loadbalancing... not sure. hard to find any benchmark around the various deployments. I've also seen some swearing on stud, but everything seems rather vague | 12:40 |
*** links has joined #openstack-swift | 12:41 | |
f0o | tdasilva: what do you run for production load? | 12:41 |
tdasilva | f0o: that's a valid point. I work for redhat, so we use tripleo | 12:42 |
rledisez | f0o: we use haproxy. with proper hardware, you can go up to 15Gb/s in SSL on one server | 12:43 |
f0o | oh nice | 12:44 |
f0o | we have 3 proxy nodes that are connected to 4x 1G LACP for public network, I was planing on running them in a DNS-RR setup rather than pushing them through a loadbalancer | 12:45 |
f0o | but if haproxy does a good job at terminating TLS I might just give it a go | 12:45 |
*** mingyu has joined #openstack-swift | 12:49 | |
*** mingyu has quit IRC | 12:54 | |
*** bkopilov_ has joined #openstack-swift | 13:00 | |
*** bkopilov has joined #openstack-swift | 13:00 | |
*** ouchkernel has quit IRC | 13:10 | |
*** mingyu has joined #openstack-swift | 13:14 | |
*** cbartz has quit IRC | 13:15 | |
*** geaaru has quit IRC | 13:21 | |
openstackgerrit | Joel Wright proposed openstack/swift master: Add Preamble and Postamble to SLO and SegmentedIterable https://review.openstack.org/365371 | 13:24 |
*** tongli has joined #openstack-swift | 13:26 | |
*** zhurong has quit IRC | 13:28 | |
*** chlong has quit IRC | 13:28 | |
openstackgerrit | Gábor Antal proposed openstack/swift master: Use more specific asserts in test/unit/obj tests https://review.openstack.org/342830 | 13:29 |
*** vint_bra has joined #openstack-swift | 13:38 | |
*** jaosorior is now known as jaosorior_away | 13:41 | |
*** psachin has joined #openstack-swift | 13:45 | |
*** _JZ_ has joined #openstack-swift | 13:47 | |
*** JimCheung has joined #openstack-swift | 13:56 | |
*** links has quit IRC | 13:58 | |
*** JimCheung has quit IRC | 14:01 | |
*** zhurong has joined #openstack-swift | 14:02 | |
*** NM has joined #openstack-swift | 14:02 | |
*** ouchkernel has joined #openstack-swift | 14:04 | |
*** klrmn has joined #openstack-swift | 14:10 | |
*** openstackgerrit has quit IRC | 14:18 | |
*** joeljwright has quit IRC | 14:26 | |
*** jordanP has joined #openstack-swift | 14:44 | |
*** zhurong has quit IRC | 14:52 | |
*** joeljwright has joined #openstack-swift | 14:58 | |
*** ChanServ sets mode: +v joeljwright | 14:58 | |
*** tovin07_ has joined #openstack-swift | 15:10 | |
*** chlong has joined #openstack-swift | 15:15 | |
*** tovin07_ has quit IRC | 15:15 | |
*** tovin07_ has joined #openstack-swift | 15:15 | |
*** klrmn has quit IRC | 15:26 | |
notmyname | good morning | 15:33 |
*** ouchkernel has quit IRC | 15:37 | |
notmyname | anyone familiar with http://crystal-sds.org ? | 15:37 |
notmyname | or http://iostack.eu ? | 15:38 |
*** geaaru has joined #openstack-swift | 15:40 | |
tovin07_ | hi notmyname | 15:45 |
tovin07_ | can you take a look at https://blueprints.launchpad.net/swift/+spec/osprofiler-support-in-swift | 15:45 |
tovin07_ | thank you :D | 15:45 |
tovin07_ | btw, i’m not familiar with crystal-sds/iostack :v | 15:45 |
acoles | notmyname: http://crystal-sds.org/?page_id=23 mentions Eran | 15:45 |
notmyname | acoles: yeah, I saw storlets on their block diagram | 15:46 |
notmyname | hmm.. but that's IBM eran. not post-ibm eran | 15:46 |
notmyname | tovin07_: ok, what about it | 15:46 |
tovin07_ | notmyname: it’s about integrating OSprofiler into Swift | 15:47 |
tovin07_ | an small example with glance https://tovin07.github.io/glance/glance-image-list-highlight.html | 15:48 |
tovin07_ | Can you approve it? I hope that I can make it land to Swift in this Pike | 15:52 |
notmyname | tovin07_: it seems that the basic integration is to add the osprofiler middleware into the (front of?) the wsgi pipeline? | 15:52 |
*** caccola has joined #openstack-swift | 15:52 | |
notmyname | tovin07_: there's not thing to approve. we don't use blueprints. http://lists.openstack.org/pipermail/openstack-dev/2016-May/094026.html | 15:53 |
tovin07_ | notmyname: not really, wsgi middleware is just a small piece. We can trace RPC, DB request too | 15:53 |
tovin07_ | oh, thanks for the link | 15:53 |
notmyname | tovin07_: do you have a patch worked up yet? | 15:54 |
tovin07_ | currently, I’m working on my local testbed | 15:54 |
tovin07_ | I will publish it to gerrit soon for review :D | 15:54 |
notmyname | tovin07_: from https://specs.openstack.org/openstack/oslo-specs/specs/mitaka/osprofiler-cross-service-project-profiling.html I got the impression that adding it to the pipeline would enable it. what other integration points are there? (swift doesn't call other openstack projects other than keystone) | 15:55 |
notmyname | tovin07_: also, what are the dependencies osprofiler will pull in? | 15:55 |
notmyname | eg from patch 254703 it seems that it depends on oslo.config. swift doesn't currently use oslo.config, so that might be a challenge | 15:57 |
patchbot | https://review.openstack.org/#/c/254703/ - nova - Integrate OSProfiler and Nova (MERGED) | 15:57 |
tovin07_ | what other integration points are there? -> swift do not need to be starting point of tracing at all | 15:58 |
tovin07_ | what are the dependencies osprofiler will pull in? -> https://git.openstack.org/cgit/openstack/osprofiler/tree/requirements.txt | 15:58 |
notmyname | I mean, the nova patch adds a bunch of decorators, but I dont' see that mentioned on https://specs.openstack.org/openstack/oslo-specs/specs/mitaka/osprofiler-cross-service-project-profiling.html at all | 15:58 |
notmyname | just trying to understand what the code impact will be | 15:59 |
*** tongli has quit IRC | 15:59 | |
notmyname | hmm... that's a lot of new dependencies | 15:59 |
*** chsc has joined #openstack-swift | 16:00 | |
*** chsc has joined #openstack-swift | 16:00 | |
tovin07_ | yes, that spec do not cover decorator things | 16:00 |
*** ganders has joined #openstack-swift | 16:00 | |
tovin07_ | this one has: https://docs.openstack.org/developer/osprofiler/api.html#five-ways-to-add-a-new-trace-point | 16:00 |
notmyname | tovin07_: ok, so the basic idea is to decorate the code at all the points that could be traced and then add the middleware so that it can respond to the "trace this request" header. also, any calls to other openstack services should be modified to pass along any trace info, if present | 16:05 |
notmyname | tovin07_: is that a decent summary? | 16:05 |
tovin07_ | yes (y) | 16:05 |
tovin07_ | the basic idea is that | 16:06 |
notmyname | tovin07_: ok. can you tell me more about where the trace info goes? I see a dependency on oslo.messaging, so is it putting stuff on a rabbit queue or something? | 16:06 |
tovin07_ | It canbe rabbitmq (if we use with messaging), redis, mongodb, loginsight, elasticseach, … or even file if we want. A list of supported “driver” (where traces are stored) here: https://git.openstack.org/cgit/openstack/osprofiler/tree/osprofiler/drivers | 16:08 |
notmyname | tovin07_: what's the current state of the patch you've been working on? how far have you gotten? | 16:09 |
tovin07_ | I started working on it early this week, sadly, it’s just a beginning —> there is no output yet | 16:13 |
*** caccola has quit IRC | 16:14 | |
notmyname | ok, that's fine. I was curious about how successful or not you'd been at integrating this into swift | 16:14 |
*** caccola has joined #openstack-swift | 16:14 | |
tovin07_ | ok, thank you! I will continue to work on it. | 16:16 |
notmyname | tovin07_: the extra dependencies you're adding are going to be a major hurdle to overcome in review. not only the oslo libraries, but also webob itself (something that swift explicitly replaced years ago). I really like the sample output from glance that you provided. it looks really cool, and it seems like osprofiler could be a really useful tool | 16:16 |
notmyname | also, many years ago we added an "xprofile" middleware that offers some degree of profiling https://docs.openstack.org/developer/swift/middleware.html#module-swift.common.middleware.xprofile | 16:16 |
notmyname | (honestly, xprofile isn't used or looked at much as far as I know. but it's important to mention in this conversation) | 16:17 |
*** chsc has quit IRC | 16:18 | |
tovin07_ | yep | 16:18 |
tovin07_ | people tend to use OSprofiler for benchmarking, find bottle-neck or trouble-shoot OpenStack services | 16:19 |
notmyname | tovin07_: but I want you to know before you get too far into this that (1) because of the existing xprofile and (2) because of the large list of new dependencies you'd be adding, getting osprofile integration merged into swift would be very difficult | 16:19 |
tovin07_ | (2) I know that too | 16:20 |
*** ganders has quit IRC | 16:20 | |
*** jordanP has quit IRC | 16:21 | |
*** ganders has joined #openstack-swift | 16:23 | |
*** pcaruana has quit IRC | 16:24 | |
*** clayg has joined #openstack-swift | 16:25 | |
*** JimCheung has joined #openstack-swift | 16:30 | |
notmyname | tovin07_: let me be explicit (just to make sure there's no confusion or incorrect expectations) | 16:32 |
f0o | I got my swift behind haproxy with TLS now and although I can create containers and upload files to it, I get `precondition failed` if I try to list containers or their contents. I can download the contents just fine by URL (assuming they're public). any ideas? | 16:33 |
*** arunman has quit IRC | 16:33 | |
*** JimCheung has quit IRC | 16:33 | |
*** arunman has joined #openstack-swift | 16:34 | |
*** arunman has quit IRC | 16:34 | |
notmyname | tovin07_: as is, because of the extra dependencies, there is zero chance that osprofile integration will land. however, if you were able to make it work with no new dependencies, chances would be *much* higher (even if it adds a decorator to a bunch of places throughout the codebase) | 16:35 |
*** NM has quit IRC | 16:35 | |
notmyname | f0o: in the 412 response, there should be an x-trans-id header. if so, grep the swift logs for that and look at the request that is actually getting to the proxy server. check to see if it looks well-formed. if there is no x-trans-id header, then the error is not coming from swift and is probably from haproxy itself | 16:37 |
notmyname | 412 is precondition failed, right? | 16:37 |
notmyname | yes | 16:37 |
f0o | notmyname: yeah it's a 412 but I cant find the x-trans-id | 16:38 |
*** tesseract has quit IRC | 16:38 | |
notmyname | f0o: then it's not from swift | 16:38 |
*** NM has joined #openstack-swift | 16:38 | |
notmyname | f0o: every response from swift will include an x-trans-id header in the response | 16:38 |
f0o | let me just tcpdump | 16:38 |
notmyname | so you can now focus your attention on the haproxy layer | 16:38 |
f0o | got it | 16:38 |
f0o | ok seems like something is off with replication, only one node replies with 200 - the rest is going 412 on the nodes | 16:40 |
f0o | fix one thing, another breaks haha | 16:41 |
*** xinli has joined #openstack-swift | 16:41 | |
notmyname | f0o: was this a cluster that you had verified to be working before setting up haproxy? | 16:41 |
notmyname | f0o: you might try `swift-recon --validate-servers` to do a basic check that the rings are right | 16:42 |
f0o | yeah it was working fine | 16:43 |
f0o | and I only proxified the public endpoint with haproxy | 16:43 |
notmyname | ok | 16:43 |
f0o | 3/3 hosts ok, 0 error[s] while checking hosts. | 16:43 |
notmyname | ok | 16:44 |
notmyname | well, ok assuming you expected only 3 results. might be a big problem if you have 300 servers in your cluster ;-) | 16:44 |
f0o | I also made sure that the TLS certificate is trusteed bt all parties | 16:44 |
f0o | nah just a small setup for now, 3 hosts, each in it's own DC stacked with 24 hdds | 16:45 |
notmyname | cool | 16:45 |
*** gyee has joined #openstack-swift | 16:45 | |
*** mvk has quit IRC | 16:47 | |
f0o | checked the logs and replication runs were successful just some seconds ago | 16:48 |
tovin07_ | notmyname: yes, thanks for clarifying, i’ll follow those points | 16:49 |
*** jordanP has joined #openstack-swift | 16:49 | |
mingyu | Hi all, I was benchmarking a swift cluster with Cosbench. I found that when cosbench wrote small objects into the cluster with high concurrency (100~150 workers), the performance fluctuated heavily. I tried to increase the number of workers of object servers and proxy servers, but didn't get any improvement. | 16:49 |
notmyname | mingyu: there's a lot of things that you could check. could be something a simple as being bottlenecked by memcache connections (try raising memcache_max_connections). could be you were limited by drive throughput. could be a cpu limitation on the proxy server | 16:51 |
notmyname | mingyu: I'd point you to pdardeau, but he doesn't seem to be around (actually, he was part of the OSIC layoffs at intel :-( ) | 16:52 |
notmyname | mingyu: but the basic steps with any benchmarking apply. measure everything, find the bottleneck, see what you can tune to avoid that bottleneck | 16:52 |
*** tovin07_ has left #openstack-swift | 16:52 | |
zaitcev | notmyname: maybe post this on a blog instead http://lists.openstack.org/pipermail/openstack/2017-April/019291.html | 16:54 |
notmyname | zaitcev: yeah, I realize I end up writing a blog post to answer stuff on the ML :-) | 16:54 |
notmyname | zaitcev: actually, I think adding it to the in-tree docs would be good | 16:54 |
mingyu | thank you @notmyname ! I'll try to raise memcache_max_connections. I have 90 hdds in this cluster and the cpu usage was below 80% when benchmarking. | 16:56 |
notmyname | mingyu: I only mention that setting because I saw some other benchmark requests that were improved by raising that value | 16:57 |
* notmyname is off to a morning of meetings... | 16:57 | |
f0o | notmyname: I feel stupid, horizon had a typo on the max limit... that caused the issue. tcpdump got me the full response after switching abck to non-tls and it stated `limit\=100001` in the request and `Maximum limit is 10000` in the response | 16:57 |
*** psachin has quit IRC | 17:05 | |
*** joeljwright has quit IRC | 17:07 | |
*** chsc has joined #openstack-swift | 17:07 | |
*** klrmn has joined #openstack-swift | 17:08 | |
*** ganders has quit IRC | 17:09 | |
timburke | :-/ one more reason i wish you guys would let me land https://review.openstack.org/#/c/391932/ -- i hate that 412 rarely means "precondition failed" in our codebase | 17:12 |
patchbot | patch 391932 - swift - Return 400 on bad requests (ABANDONED) | 17:12 |
*** SkyRocknRoll has joined #openstack-swift | 17:26 | |
*** zul has quit IRC | 17:26 | |
*** ganders has joined #openstack-swift | 17:28 | |
*** zul has joined #openstack-swift | 17:29 | |
*** ChubYann has joined #openstack-swift | 17:29 | |
*** SkyRocknRoll has quit IRC | 17:31 | |
*** NM has quit IRC | 17:40 | |
*** NM has joined #openstack-swift | 17:45 | |
*** mtreinish has quit IRC | 17:46 | |
*** NM has quit IRC | 17:47 | |
*** SkyRocknRoll has joined #openstack-swift | 17:47 | |
*** NM has joined #openstack-swift | 17:49 | |
*** mvk has joined #openstack-swift | 18:04 | |
*** mtreinish has joined #openstack-swift | 18:05 | |
*** jordanP has quit IRC | 18:17 | |
*** openstackgerrit has joined #openstack-swift | 18:31 | |
openstackgerrit | Merged openstack/swift master: Trivial fix warnings in docstring https://review.openstack.org/462040 | 18:31 |
-openstackstatus- NOTICE: Gerrit on review.openstack.org is being restarted to accomodate a memory leak in Gerrit. Service should return shortly. | 18:54 | |
*** jordanP has joined #openstack-swift | 18:56 | |
*** mingyu has quit IRC | 18:56 | |
*** mingyu has joined #openstack-swift | 18:58 | |
*** SkyRocknRoll has quit IRC | 19:25 | |
*** flwang has quit IRC | 19:31 | |
*** gyee has quit IRC | 19:55 | |
notmyname | oh. right. wednesday. we have a meeting (in an hour). I almost forgot | 19:57 |
*** vint_bra has quit IRC | 20:03 | |
*** ganders has quit IRC | 20:06 | |
*** silor has quit IRC | 20:09 | |
*** vint_bra has joined #openstack-swift | 20:12 | |
*** jordanP has quit IRC | 20:20 | |
*** catinthe_ has joined #openstack-swift | 20:21 | |
*** catintheroof has quit IRC | 20:24 | |
*** m_kazuhiro has joined #openstack-swift | 20:27 | |
*** joeljwright has joined #openstack-swift | 20:37 | |
*** ChanServ sets mode: +v joeljwright | 20:37 | |
*** caccola has quit IRC | 20:44 | |
*** pdardeau has joined #openstack-swift | 20:44 | |
*** catinthe_ has quit IRC | 20:45 | |
*** ChanServ sets mode: +v clayg | 20:55 | |
clayg | is that *today* !? | 20:55 |
notmyname | it's today! | 20:56 |
*** _JZ_ has quit IRC | 20:57 | |
kota_ | ggood morning | 20:58 |
joeljwright | kota_: morning! | 20:58 |
kota_ | hi joeljwright | 20:58 |
mattoliverau | Morning | 20:59 |
joeljwright | o/ | 20:59 |
notmyname | meeting time in #openstack-meeting | 21:00 |
openstackgerrit | Tim Burke proposed openstack/pyeclib master: Avoid segfault when raising exceptions https://review.openstack.org/385186 | 21:03 |
*** NM has quit IRC | 21:10 | |
openstackgerrit | Merged openstack/swift master: Fix (un)patch_policies https://review.openstack.org/440936 | 21:11 |
*** vint_bra has quit IRC | 21:12 | |
clayg | looks like the meeting's topic has moved on | 21:14 |
clayg | joeljwright: I'm confused by the TODO in patch 456205 (We need to pass the session here) - that plus the statement "All the client requests should be made through the Keystone session" leave me very confused ... | 21:15 |
patchbot | https://review.openstack.org/#/c/456205/ - python-swiftclient - WIP: Use keystone session when possible | 21:15 |
clayg | just what *is* this session object? | 21:15 |
*** vint_bra has joined #openstack-swift | 21:15 | |
*** NM has joined #openstack-swift | 21:15 | |
joeljwright | it's like a requests session that handles the auth | 21:15 |
joeljwright | clayg: ^^^ | 21:15 |
clayg | it's made out to be some wapper over requests that has the property of catching 401/403 and ... | 21:15 |
clayg | yeah, great that's what I thought :\ | 21:15 |
timburke | https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py | 21:15 |
*** chlong has quit IRC | 21:17 | |
clayg | timburke: knows how I like this | 21:19 |
*** mingyu has quit IRC | 21:20 | |
*** mingyu has joined #openstack-swift | 21:21 | |
clayg | seems like the one out of three case where where is-a would be better than has-a; none of this should be *required* tho? it's not like the Keystone API only works if you use python class? | 21:21 |
*** mingyu has quit IRC | 21:22 | |
joeljwright | clayg: not sure I can parse that… | 21:22 |
*** mingyu has joined #openstack-swift | 21:22 | |
*** _JZ_ has joined #openstack-swift | 21:24 | |
clayg | which part? is-a vs. has-a? I think it's odd that the keystone session object implements the request session api | 21:24 |
*** gyee has joined #openstack-swift | 21:25 | |
joeljwright | clayg: do you really not like it? | 21:25 |
*** Sukhdev has joined #openstack-swift | 21:25 | |
joeljwright | clayg: we use a requests session and handle auth/reauth/headers manually | 21:26 |
clayg | i just... I feel the coupling... if things are getting painful somewhere because we're not proxying all of our requests through this keystone session object - something is off the rails - because like what does a client library that's not even written in python do? Or what if I have service I want to use keystone to auth a- but my API is RPC instead of | 21:27 |
clayg | REST? | 21:27 |
joeljwright | yeah, I see your point | 21:28 |
joeljwright | I think not-python is a non-issue though | 21:28 |
clayg | joeljwright: and that's working for us right? What if someone remembers that ever since we switched over the requests we lost our 100-Continue support for uploads and goes to make it right only to find out they have to bypass the keystone session - or even the request session object to fix it | 21:28 |
joeljwright | you'd have to auth however made most sense there | 21:28 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 21:29 |
clayg | sweet - and we can't do that for why? | 21:29 |
joeljwright | clayg: I don't know what to say tbh… I think it could clean things up and make life easier in certain respects, but I share some of your concerns too… | 21:30 |
* clayg shrugs | 21:31 | |
*** NM has quit IRC | 21:31 | |
joeljwright | I'm also concerned that with keystone APIs being deprecated that at least some of this could be forced upon us | 21:32 |
openstackgerrit | Tim Burke proposed openstack/pyeclib master: Avoid segfault when raising exceptions https://review.openstack.org/385186 | 21:32 |
joeljwright | well, it's there as a point of discussion now :)] | 21:32 |
timburke | clayg: we keep lamenting the loss of 100-continue support for swiftclient... but i'm not sure we ever really had it. we have to work around httplib itself to get it in swift :-( | 21:34 |
clayg | i guess in another worldview we could just use keystone sessions cause that what everyone else does and then patch keystone sessions if we find it to be lacking for our usecase or have a bad interface somewhere... | 21:34 |
clayg | timburke: I think it used to import swift's buffered http | 21:35 |
notmyname | yeah, IIRC it did | 21:35 |
timburke | clayg: so...we had swift the *client* importing from swift the *server*? ugh | 21:36 |
notmyname | joeljwright: hmm... I didn't realize the keystone session changes had the impact clayg brings up. I agree that "it only works if every request goes through a keystone class" is wrong | 21:36 |
notmyname | timburke: remember that they used to be the same codebase ;-) | 21:36 |
timburke | notmyname: ...and so you'd have to install the *whole thing* to get a blessed client | 21:37 |
timburke | imagine it now: wanna put some data? go install libec! | 21:37 |
* mattoliverau goes to breakfast | 21:38 | |
notmyname | sure sure. that doesn't seem like a good idea now | 21:38 |
*** m_kazuhiro has quit IRC | 21:38 | |
notmyname | but I want to get back to keystone sessions | 21:39 |
clayg | notmyname: it only just *sounds* insane - in reality it's probably fine - you wouldn't mind it every request in swiftclient went though a swiftclient.Connection class - just think of keystone as being part of the swiftclient code base | 21:39 |
notmyname | joeljwright: so *everything* goes through keystone? | 21:39 |
notmyname | clayg: right! | 21:40 |
notmyname | but I can easily change the swiftclient code | 21:40 |
joeljwright | notmyname: well… not necessarily… not if you're not using keystone for auth | 21:40 |
joeljwright | we've got duck typing after all… we could just make a swiftclient.Connection class that looks like a requests session and a keystone session :) | 21:41 |
notmyname | hmmm | 21:41 |
notmyname | yeah, does keystone know what to do with a 498 response code? | 21:41 |
timburke | joeljwright: no, even if you're using v1 auth, we'll want a keystone session with *our* v1 plugin, and everything still goes through it | 21:41 |
notmyname | does keystone ever look at the response body? | 21:41 |
joeljwright | timburke: we can do it that way, but nothing is forcing us | 21:41 |
joeljwright | just sayin | 21:42 |
clayg | notmyname: I mean... true - decoupling enables agility - but *coupling* enables *consistency* - do you want to go fast or do you want every client everywhere to do it *right*? | 21:42 |
notmyname | clayg: yes, of course | 21:42 |
timburke | notmyname: does swiftclient know what to do with a 498? | 21:43 |
joeljwright | looks like this will require exactly as much discussion as I thought :) | 21:43 |
notmyname | timburke: I don't know, but I hope so since it's a response code we invented | 21:43 |
clarkb | clayg: the big issue ifnra ran into with swiftclient not using the sessions stuff was tokens were used even if they had expired | 21:43 |
clarkb | eventually some api request related to the upload would fail because it assumed the token would be valid for the lifetime of the logical upload | 21:44 |
notmyname | joeljwright: (1) new change proposed (2) swift devs yell about how terrible it will be (3) swift devs ignore it for a long time (4) swift devs see a use case and begrudgingly rewrite the previously proposed solution to integrate into swift (5) swift devs continue to yell a lot | 21:44 |
joeljwright | notmyname: :D | 21:45 |
timburke | notmyname: but hey, at least we've got a *process*! | 21:45 |
timburke | ISO 9000 here we come! | 21:45 |
clarkb | clayg: and what we found was NONE of the clients did any of that properly and it was an issue everywhere | 21:45 |
clarkb | so it got solved in one place and now things are happier. You are correct that python isn't required and you could write the same mechanism in $implementation | 21:46 |
*** flwang has joined #openstack-swift | 21:46 | |
notmyname | does the current "pass in a session" functionality we have get us any closer to a good place? joeljwright? | 21:47 |
joeljwright | the current 'pass in a session' just allows us to get tokens from a session | 21:48 |
joeljwright | we still handle the requests and reauth manually | 21:48 |
notmyname | and that's broken? | 21:49 |
joeljwright | threading the session through to replace the surrent requests session and letting it handling the reauthentication looks like a lot of work | 21:49 |
notmyname | for the case clarkb says? | 21:49 |
clayg | I have recent memory of api tokens expiring within a logical request in middleware - but that's not really the clients fault - i'm not sure I'm aware of a 401 response that wouldn't get reauth'd and retried? | 21:50 |
joeljwright | tokens would certainly expire, but then the client should reauth | 21:50 |
clayg | i did a quick search through open bugs and didn't see it | 21:50 |
notmyname | what's the case where swiftclient is broken today that is fixed when we pass everything through a keystone session class? | 21:50 |
clayg | it probably buys us something like... and I'm guessing here "Token support" ??? | 21:51 |
notmyname | clayg: what do we have today, if not "token support"? | 21:51 |
timburke | user gives us some creds with an unversioned endpoint and says "here, you figure it out" | 21:51 |
notmyname | timburke: that would be great to solve (ie handle transparently for the user) | 21:52 |
clayg | there's like ks_auth_plugin.Token and ks_auth_plugin.Password ??? | 21:52 |
joeljwright | the unversioned stuff is certainly a potential problem, and the explicitly versioned keystoneauth.session.vX are deprecated as far as I'm aware too | 21:52 |
clayg | maybe... like we don't know how re-auth if you only give us a Token? maybe the ks_auth_plugin.Token thing knows how to reauth from a token!? | 21:53 |
joeljwright | the push is to let keystoneauth figure it out, which was the abandoned patch that brought all this up again | 21:53 |
clayg | but that sounds silly - because then tokens don't really expire - so what's the point of a token | 21:53 |
clayg | joeljwright: but do you really need a whole bucket of python code just to use the auth api? It's one thing to say "oh sure you *could* write it in another language" and another to say "it's *literally* this simple; you could write it in Java and it'd still be less than 50 lines" | 21:55 |
notmyname | clayg: a while back I learned that what we refer to as "token" is actually the "token id" in keystone. keystone's "token" is a whole data structure packed with all kinds of stuff (the whole service catalog IIRC) | 21:55 |
clayg | notmyname: gtk! | 21:55 |
joeljwright | notmyname: there's certainly a lot going on in keystone v3 | 21:56 |
notmyname | clayg: but you have an excellent point that if you can reauth from an old token, then tokens don't actually ever expire | 21:56 |
clayg | anyway - it seems like "throw some creds at at unversioned endpoint and let keystone figure it out" should be orthogonal to "all http requests to the service are proxied through a keystone class" | 21:56 |
notmyname | clayg: it seems that way to me too | 21:57 |
clayg | notmyname: sorry - i don't really know *what* the ks_auth_plugin.Token does - it may not support reauth - i have no idea | 21:57 |
notmyname | clayg: I feel like I should stop typing, because I agree with everything you are typing | 21:57 |
clayg | i'll stop typing! | 21:58 |
notmyname | lol | 21:58 |
joeljwright | :) | 21:58 |
clarkb | ya its the session that supports reauth and I think actual creds are erquired for that | 21:58 |
notmyname | we'll all stop typing! | 21:58 |
joeljwright | have fun in Boston, I'm sure this will come up again :) | 21:59 |
timburke | i'm curious about how well keystone's session can be adapted to a signature-based auth mechanism a la boto and s3... | 22:01 |
joeljwright | I thought we all agreed to stop typing? ;) | 22:01 |
openstackgerrit | Tim Burke proposed openstack/pyeclib master: Switch from pep8 to flake8 for linting https://review.openstack.org/462304 | 22:08 |
timburke | i feel like we ought to stay on top of https://review.openstack.org/#/q/is:open+(project:openstack/pyeclib+OR+project:openstack/liberasurecode)+-label:Code-Review-1%252Cself+-label:Code-Review%252B2%252Cself better than we do... | 22:11 |
timburke | but then, i might be biased | 22:11 |
*** joeljwright has quit IRC | 22:13 | |
*** vint_bra has quit IRC | 22:13 | |
*** xinli has quit IRC | 22:19 | |
*** klrmn1 has joined #openstack-swift | 22:23 | |
*** Sukhdev has quit IRC | 22:23 | |
*** ediardo_ has joined #openstack-swift | 22:23 | |
*** blairo has joined #openstack-swift | 22:26 | |
*** chsc_ has joined #openstack-swift | 22:26 | |
*** _JZ_ has quit IRC | 22:27 | |
*** htruta` has joined #openstack-swift | 22:27 | |
*** tdasilva- has joined #openstack-swift | 22:28 | |
*** lcurtis has joined #openstack-swift | 22:28 | |
*** zaitcev_ has joined #openstack-swift | 22:28 | |
*** ChanServ sets mode: +v zaitcev_ | 22:28 | |
*** ediardo has quit IRC | 22:28 | |
*** htruta has quit IRC | 22:28 | |
*** chsc has quit IRC | 22:28 | |
*** blair has quit IRC | 22:28 | |
*** klrmn has quit IRC | 22:28 | |
*** tdasilva has quit IRC | 22:28 | |
*** zaitcev has quit IRC | 22:28 | |
*** d0ugal has quit IRC | 22:28 | |
*** ediardo_ is now known as ediardo | 22:29 | |
*** klamath has quit IRC | 22:34 | |
clayg | you can kota should just start merging stuff - or tell me or tdasilva- when you have a change that's ready to go and just needs a +A | 22:44 |
clayg | I feel like some of that stuff (esp. the older stuff e.g. patch 408259) is more like "this might be useful" kinda of stuff and I don't really have a good sense of maintainer-ship on that project to make the call? | 22:46 |
patchbot | https://review.openstack.org/#/c/408259/ - pyeclib - Install isa-l from source when testing liberasure-... | 22:46 |
clayg | where as stuff that could have a bug attached to it (like the segfault patch 385186) would be easier to say objectively "yeah this makes the project better for sure" | 22:47 |
patchbot | https://review.openstack.org/#/c/385186/ - pyeclib - Avoid segfault when raising exceptions | 22:47 |
clayg | personally - pyeclib isn't given me any grief - I am sorta freaked out about the zlib checksum stuff - but still don't have my head around it - I guess I'm hoping you'll eventually tell me what to +A and package so I don't have to be scared anymore | 22:48 |
clayg | ... i'm so scared ... | 22:48 |
patchbot | /me gives clayg a hug | 22:49 |
timburke | fwiw, that's not the only segfault lying in wait -- https://review.openstack.org/#/c/425471/ | 22:49 |
patchbot | patch 425471 - liberasurecode - Jerasure: Handle initialization errors correctly | 22:49 |
timburke | and without https://review.openstack.org/#/c/459028/ i haven't had much luck compiling on OS X lately | 22:50 |
patchbot | patch 459028 - liberasurecode - Un-inline get/set_metatdata_chksum | 22:50 |
openstackgerrit | Merged openstack/pyeclib master: Fix error message with invalid parity number https://review.openstack.org/430599 | 22:52 |
*** d0ugal has joined #openstack-swift | 22:53 | |
*** mingyu has quit IRC | 22:53 | |
*** lcurtis has quit IRC | 22:56 | |
timburke | and i really want to eventually get to https://review.openstack.org/#/c/431794/ being merged (which means i'll need to resolve the current conflict), but it's like five-deep on a chain to have that be an isolated change, so you don't have to track the cleanup that shook out from my getting there | 22:57 |
patchbot | patch 431794 - liberasurecode - Use preprocessor to build test suites | 22:57 |
timburke | and i'm not really looking forward to rebasing *all* of those | 22:57 |
clayg | timburke: but that's just an issue for jerasure? so who cares? | 23:09 |
timburke | anytime my c extension can bomb out the python interpreter, i kinda care | 23:11 |
clayg | idk, if a c extension segfaults with a plugin you don't use does it make it sound? | 23:15 |
clayg | cc tdasilva- maybe +A patch 425471? | 23:15 |
patchbot | https://review.openstack.org/#/c/425471/ - liberasurecode - Jerasure: Handle initialization errors correctly | 23:15 |
notmyname | is the pope catholic? does a bear poop in the woods? does an african swallow grip the coconut by the husk? | 23:16 |
notmyname | I'm sure there are now some non-english/non-american people in this channel who have no idea what's going on | 23:16 |
openstackgerrit | Merged openstack/swift master: Move EC-specific unit test to EC Test class https://review.openstack.org/457970 | 23:20 |
*** chsc_ has quit IRC | 23:39 | |
*** vint_bra has joined #openstack-swift | 23:46 | |
openstackgerrit | Tim Burke proposed openstack/pyeclib master: Add tox environment to test against liberasurecode master https://review.openstack.org/407829 | 23:50 |
*** mingyu has joined #openstack-swift | 23:54 | |
*** mingyu has quit IRC | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!