| timburke | i think it was this stretch: https://github.com/s3tools/s3cmd/blob/74eb55a724f30198e264def7c212571b5aebb90f/S3/FileLists.py#L523-L543 | 00:01 |
|---|---|---|
| timburke | it'd complain about an md5 mismatch instead of disabling the md5 check | 00:01 |
| *** rcernin_ has joined #openstack-swift | 00:02 | |
| *** rcernin has quit IRC | 00:03 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Move legacy-swift-dsvm-functional job in-tree https://review.openstack.org/589270 | 00:12 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add IPv6 DSVM tests https://review.openstack.org/591497 | 00:12 |
| *** rcernin has joined #openstack-swift | 00:29 | |
| *** rcernin has quit IRC | 00:29 | |
| *** rcernin has joined #openstack-swift | 00:30 | |
| *** rcernin_ has quit IRC | 00:32 | |
| *** hugokuo has quit IRC | 00:57 | |
| *** hugokuo has joined #openstack-swift | 01:04 | |
| *** openstackgerrit has quit IRC | 01:06 | |
| kota_ | good morning | 02:21 |
| *** psachin has joined #openstack-swift | 02:36 | |
| mattoliverau | kota_: morning | 02:36 |
| *** openstackgerrit has joined #openstack-swift | 03:19 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: Add support for multiple root encryption secrets https://review.openstack.org/577874 | 03:19 |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: Multi-key KMS keymaster https://review.openstack.org/591555 | 03:19 |
| mattoliverau | ^ timburke theres a kms version that seems to work on my barbican + swift dev env. It still needs tests, but that key master seems to have missed your multi key improvements | 03:20 |
| mattoliverau | now to go test the kmip multi key keymaster. | 03:23 |
| mattoliverau | ..hmm, maybe lunch first. | 03:23 |
| openstackgerrit | Merged openstack/slogging master: fix a typo in sample.proxy-server.conf https://review.openstack.org/570189 | 03:58 |
| *** links has joined #openstack-swift | 04:43 | |
| *** gyee has quit IRC | 05:05 | |
| *** pcaruana has joined #openstack-swift | 06:44 | |
| *** ccamacho has joined #openstack-swift | 07:01 | |
| *** rcernin has quit IRC | 07:02 | |
| *** quicker_io has joined #openstack-swift | 07:44 | |
| *** cbartz has joined #openstack-swift | 08:03 | |
| *** mikecmpbll has joined #openstack-swift | 08:03 | |
| *** openstackstatus has quit IRC | 08:12 | |
| *** mvkr has quit IRC | 08:20 | |
| *** mvkr has joined #openstack-swift | 08:52 | |
| *** mahatic has joined #openstack-swift | 08:58 | |
| *** ChanServ sets mode: +v mahatic | 08:58 | |
| *** hseipp has joined #openstack-swift | 09:11 | |
| *** gkadam has joined #openstack-swift | 09:18 | |
| *** mahatic has quit IRC | 09:19 | |
| *** pavelkv has quit IRC | 09:41 | |
| *** mahatic has joined #openstack-swift | 09:41 | |
| *** ChanServ sets mode: +v mahatic | 09:41 | |
| *** openstackstatus has joined #openstack-swift | 09:42 | |
| *** ChanServ sets mode: +v openstackstatus | 09:42 | |
| *** hoonetorg has quit IRC | 09:43 | |
| *** hoonetorg has joined #openstack-swift | 09:57 | |
| *** hseipp has quit IRC | 10:07 | |
| viks_ | hi, what all system(kernel) parameters(i.e. in /etc/sysctl.conf) should be tuned for proxy and storage nodes for better performance ? Is there any specific guide ? can anyone help me? | 10:16 |
| *** mvkr has quit IRC | 10:22 | |
| DHE | docs suggest some TIME_WAIT parameters because you'll be churning through those real fast... | 11:01 |
| DHE | otherwise I'd say you're better off tuning the services themselves for number of CPUs/threads to use | 11:02 |
| viks_ | DHE: i came across a script in github where some many kernel parameters are being tuned. But not sure it is required or not and if required on what basis should i set appropriate value. i.e. https://github.com/goldyfruit/openstack-os-tuning/blob/master/Swift/tuning.sh | 11:10 |
| viks_ | DHE: which docs are you referring abt `TIME_WAIT parameters`? | 11:11 |
| DHE | tcp_tw_* | 11:11 |
| DHE | https://docs.openstack.org/swift/ocata/deployment_guide.html#general-system-tuning | 11:12 |
| DHE | this github link is pretty thorough... | 11:12 |
| viks_ | DHE: i'm not sure all those params in that github link should be tuned? and what basis i should be setting appropriate value? any idea/ suggestions? | 11:16 |
| DHE | there's lots of parameters here. from how much dirty data can stay in memory before a disk flushout to iptables' nf_conntrack module settings | 11:18 |
| DHE | i'm not sure how much I agree with the notion it needs to be changed, unless you're mixing proxy and storage nodes and want to make sure the TCP stack is extra well fed | 11:19 |
| viks_ | in my setup i have separate proxy and storage nodes.. So is it okay to go with default values? | 11:23 |
| viks_ | i mean default values set by OS? | 11:26 |
| DHE | they're fine to use. maybe not optimal. how hard are you going to stress the system? | 11:34 |
| viks_ | ok.. i do not have any idea on how hard i should be stressing the system... just finding out if i should be tuning any kernel params to get a better performance out of it in specific to swift proxy and storage nodes.. | 11:45 |
| DHE | these settings really feel like they expect to run all components at 100% at some point. from tweaking swap usage to fixing the TCP stack memory usage to making read IOs much higher priority over write IOs | 12:15 |
| viks_ | ok | 12:19 |
| *** quicker_io has quit IRC | 12:59 | |
| *** psachin has quit IRC | 13:06 | |
| zaitcev | TIME_WAIT is the duration for the packets to exist in the network. The closed socket is kept around enough so a packet sent erroneously will still match against its state and not against an equivalent, but newly-open socket. | 13:37 |
| zaitcev | Do keep in mind though, if you have a dual-homed proxy, the same TIME_WAIT applies to inside and outside. So, you can only lower it if the proxy is protected by a load balancer. | 13:39 |
| DHE | I think I would disable connection tracking for connections swift internal connections and client connections could be good as well... should eliminate the need for conntrack tuning. | 13:45 |
| * DHE is still interested in Unix sockets for services to be able to run from... | 13:48 | |
| DHE | partially for this reason | 13:48 |
| DHE | or alternatively, run as a dedicated uwsgi application. I can make that work as well.. | 14:11 |
| *** ccamacho has quit IRC | 14:34 | |
| *** ccamacho has joined #openstack-swift | 14:56 | |
| timburke | good morning | 15:18 |
| *** ccamacho has quit IRC | 15:19 | |
| timburke | mattoliverau: thanks for the multi-key barbican patch! i knew there was something else i ought to do on that chain... | 15:19 |
| timburke | kota_: thanks for the review on p 575860! i'll look at splitting out the normal-SLO retro-fit from the rest of the patch, and try to explain better why i wanted that part in particular | 15:21 |
| patchbot | https://review.openstack.org/#/c/575860/ - swift - Include '-' in multipart ETags - 10h 2m 43s spent in CI | 15:21 |
| *** psachin has joined #openstack-swift | 15:26 | |
| * DHE keeps this script though... | 15:30 | |
| timburke | oh, DHE: you might be interested in https://review.openstack.org/#/c/324136/ -- i liked the idea, too, but i'm not sure when i'd be able to get around to reviving the patch | 15:32 |
| patchbot | patch 324136 - swift - Support binding to Unix domain sockets. (ABANDONED) - 0h 41m 44s spent in CI | 15:32 |
| clayg | tdasilva: you making any progress on the s3 stuff? | 15:33 |
| clayg | I finally feel like i'm getting somewhere with the multi-key stuff (I had to catch up on the kmips progress!) | 15:33 |
| timburke | clayg: there's all sort of people reviewing s3 stuff! it's great! kota_ wants me to break up one of the patches, zaitcev seems to like (or at least tolerate) what he sees... | 15:34 |
| clayg | that's GREAT | 15:34 |
| clayg | go team swift! | 15:34 |
| zaitcev | timburke: I'm not all that sure about the ACL thing. I deduced that the attacker sets the ACLs and then uses those ACLs in order to create buckets. Not sure if that's actually what you want to fix. | 15:36 |
| timburke | zaitcev: the root trouble is that the s3_acl option basically trows us into some alternate-reality where auth middlewares only really handle *authentication*, not so much *authorization* | 15:39 |
| tdasilva | timburke: one of my takeaways from looking at the acl patch yesterday was the need for more docs :/ | 15:39 |
| tdasilva | especially for users regarding the new behavior where container listing (json) might return more info | 15:40 |
| tdasilva | clayg, timburke: currently looking at p 577941 | 15:40 |
| patchbot | https://review.openstack.org/#/c/577941/ - swift - s3_acl: Require swift_owner authz to create buckets - 2h 3m 56s spent in CI | 15:40 |
| clayg | timburke: does p 586902 support N keymasters or exactly 2? | 15:40 |
| patchbot | https://review.openstack.org/#/c/586902/ - swift - Allow multiple keymasters - 3h 11m 57s spent in CI | 15:40 |
| timburke | zaitcev: as a result test:tester3 could create buckets in AUTH_test despite not being able to do so via the swift API | 15:40 |
| clayg | it looks like it just stashes the existing CRYPTO_KEY_CALLBACK if any? But maybe... that can chain? | 15:41 |
| *** links has quit IRC | 15:41 | |
| timburke | clayg: since all known keymasters subclass that one, they should all do it and it should chain. i suppose i hadn't actually tested with 3 or more though... | 15:42 |
| clayg | ok, cool | 15:43 |
| timburke | on the plus side, you should be able to stuff multiple trivial keymasters in a single pipeline to validate without standing up additional services | 15:43 |
| timburke | that patch really needs tests and docs; it almost certainly shouldn't merge as-is | 15:44 |
| timburke | as much as anything, i pushed it up because with all these multi-key-aware keymasters it seemed weird to not be able to mix and match a bit | 15:45 |
| DHE | timburke: https://blueprints.launchpad.net/swift/%2Bspec/wsgi-application-interface I also saw this which seem to have stagnated. It could be another option by running in uwsgi which would do the unix socket listening. And I'd probably want to run nginx in front of the proxy anyway | 15:46 |
| *** cbartz has quit IRC | 15:47 | |
| *** cbartz has joined #openstack-swift | 15:47 | |
| *** cbartz has quit IRC | 15:48 | |
| timburke | DHE: https://review.openstack.org/#/c/33169/ (which added a bunch of files under https://github.com/openstack/swift/tree/master/examples ) seems related? | 15:49 |
| patchbot | patch 33169 - swift - Add example Apache config files (MERGED) - 5h 5m 25s spent in CI | 15:49 |
| timburke | i must admit, though, i've never tried using them... | 15:50 |
| DHE | hmm... interesting.. | 15:50 |
| timburke | clayg: fwiw, i'd focus on https://review.openstack.org/#/c/590024/ ahead of the multiple-keymasters patch | 15:52 |
| patchbot | patch 590024 - swift - Factor out a proper BaseKeyMaster class - 1h 32m 21s spent in CI | 15:52 |
| clayg | timburke: ah, ok, thanks for the heads up | 15:54 |
| *** itlinux has joined #openstack-swift | 15:54 | |
| *** ccamacho has joined #openstack-swift | 15:56 | |
| *** gyee has joined #openstack-swift | 16:02 | |
| *** pcaruana has quit IRC | 16:02 | |
| *** d0ugal has quit IRC | 16:28 | |
| *** mikecmpbll has quit IRC | 16:31 | |
| DHE | timburke: that works... though I had to do some dirty stuff in nginx to do it... | 16:37 |
| DHE | apparently CONTENT_LENGTH is always sent to the proxy server, but is a blank field if the calling application didn't actually send such an HTTP header | 16:38 |
| timburke | hey, as long as it doesn't involve using gdb to patch it at run time, it's not *so* bad, amiright? | 16:38 |
| timburke | huh. curious | 16:38 |
| DHE | seems to be a quirk of nginx.. and I guess python treats the empty string as boolean true ? | 16:39 |
| *** ccamacho has quit IRC | 16:39 | |
| timburke | maybe related? https://bugs.launchpad.net/python-swiftclient/+bug/1586690 https://bugs.launchpad.net/python-swiftclient/+bug/1621581 | 16:43 |
| openstack | Launchpad bug 1586690 in python-swiftclient "Uploading empty(0 B) file fails" [Undecided,Incomplete] - Assigned to Uday Swami (swamius) | 16:43 |
| DHE | or maybe any access of the content-length field invokes string2int which dies at https://github.com/openstack/swift/blob/master/swift/common/swob.py#L213 | 16:43 |
| openstack | Launchpad bug 1621581 in python-swiftclient "swiftclient returns response headers without 'Content-Length' param, thus causing upload object to fail" [Undecided,Invalid] - Assigned to Arun Mani (arun-mani) | 16:43 |
| DHE | I'm just using curl to GET a container for an object listing and making sure the proxy does its thing. right now all my other services are down in my lab but I can see it trying to contact the account servers so that's very reassuring | 16:45 |
| timburke | fwiw, i'm pretty sure eventlet would bomb out on a blank (but still present) Content-Length: https://github.com/eventlet/eventlet/blob/v0.24.1/eventlet/wsgi.py#L434-L443 | 16:45 |
| DHE | right, but this is nginx acting as a middle man and that's where it's going wrong. | 16:45 |
| DHE | 400 bad request from eventlet be completely reasonable, but nginx is clearly misbehaving here | 16:46 |
| DHE | oh here we go, nginx has "if_not_empty" | 16:47 |
| DHE | that does the trick | 16:48 |
| DHE | that is cool... | 16:49 |
| *** psachin has quit IRC | 17:22 | |
| timburke | cd .. | 17:38 |
| *** gkadam has quit IRC | 17:53 | |
| DHE | while it looks like it will work, it doesn't seem to be using any kind of eventlet or similar IO system to allow one thread to handle multiple requests. I think this would negatively impact overall throughput. maybe it's not such a good idea for now. | 18:10 |
| timburke | man, there are a bunch of interesting S3 features that you only discover by poking around in random clients... https://github.com/aws/aws-sdk-go/blob/v1.15.11/service/s3/body_hash.go#L144-L145 | 18:29 |
| tdasilva | timburke: yeah, isn't it nice that the industry decided to default to a proprietary api? | 18:37 |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: s3_acl: Require swift_owner authz to create buckets https://review.openstack.org/577941 | 20:59 |
| kota_ | timburke: ack, and thx! unfortunately i'm at dayoff today for bridal anniversary lunch but i promise i'll be back to the patch tomorrow morning. | 21:16 |
| timburke | kota_: enjoy the day off! | 21:18 |
| *** mikecmpbll has joined #openstack-swift | 21:18 | |
| timburke | kota_: am i right to think that's *your* anniversary? happy anniversary! | 21:18 |
| tdasilva | timburke: can you share the reasoning for p 575838 ? | 21:20 |
| patchbot | https://review.openstack.org/#/c/575838/ - swift - Listing of versioned objects when versioning is no... - 1h 24m 27s spent in CI | 21:20 |
| timburke | so, some clients (cyberduck for example, iirc) request a versioned listing regardless of whether versioning is enabled on the bucket. even though we don't support enabling versioning (yet), we could at least reply with the objects that are present (with a null version) | 21:22 |
| tdasilva | timburke: i'm assuming that's because aws s3 has no concept of versioning being enabled/disabled? is that the case? | 21:24 |
| timburke | it *does* (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETversioningStatus.html which lists the three versioning states), it' just that the version-aware listing (see https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETVersion.html) works even when versioning is suspended or never-enabled | 21:28 |
| timburke | well that's a terrifying sort of a note. "A 200 OK response can contain valid or invalid XML. Make sure to design your application to parse the contents of the response and handle it appropriately." | 21:32 |
| timburke | why the hell would you send me a 200 when you're gonna send back a garbage response?? | 21:35 |
| *** mikecmpbll has quit IRC | 21:36 | |
| tdasilva | lol | 21:36 |
| *** mikecmpbll has joined #openstack-swift | 21:51 | |
| *** itlinux has quit IRC | 21:54 | |
| *** mikecmpbll has quit IRC | 22:27 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Stop mangling Authorization header for v4 signatures https://review.openstack.org/591862 | 22:38 |
| *** zaitcev has quit IRC | 22:45 | |
| *** kei-ichi has quit IRC | 22:51 | |
| *** kei-ichi has joined #openstack-swift | 22:55 | |
| kota_ | timburk: that's approximately 6 years anniversary to me. | 23:00 |
| timburke | congrats! | 23:01 |
| kota_ | thx | 23:01 |
| *** mvkr has joined #openstack-swift | 23:05 | |
| mattoliverau | morning | 23:15 |
| timburke | *sigh* it took me _way_ longer than it should have to realize that i messed up my pipeline ordering and put s3api after auth instead of before... | 23:42 |
| openstackgerrit | Merged openstack/swift master: Fix up test_static_web_pretend_to_be_giant_json https://review.openstack.org/591520 | 23:46 |
| mattoliverau | :( it happens. | 23:47 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!