| notmyname | DHE: I done quite a bit of research into SMR drives with swift. short answer is: "don't" | 00:04 |
|---|---|---|
| notmyname | DHE: this talk I co-presented at one of the summits will be very helpful to you, if you're considering SMR drives https://www.openstack.org/videos/tokio-2015/shingled-magnetic-recording-smr-drives-and-swift-object-storage | 00:04 |
| *** persia has left #openstack-swift | 00:10 | |
| DHE | notmyname: I'm running SMR in another scenario where archival is exactly what they're for. I've gotten a feel for how they behave (seagate archive 8t mainly) | 00:18 |
| DHE | queuing up video now... | 00:19 |
| notmyname | yeah, they key is the keeping the write rate under the critical threshold of the drive. if you go above it, the performance can take 3 orders of magnitude (ie 1000x) performance hit for a very long time | 00:21 |
| notmyname | and when you've got a system like swift where the background processes may end up using a lot of drive write capacity, it can be bad | 00:22 |
| notmyname | unfortunately, swift doesn't yet have the capability to strictly allocate reads and writes on a per drive basis against a set IOPS budget. (but it's something I'm really interested in) | 00:23 |
| DHE | yeah I've seen sustained throughput average ~35 megabytes/second but the second-by-second numbers vary between 10 and 100 megabytes/second | 00:23 |
| DHE | so you can do that with control groups | 00:24 |
| *** tovin07_ has joined #openstack-swift | 00:38 | |
| *** cshastri has joined #openstack-swift | 00:42 | |
| *** armaan has joined #openstack-swift | 00:48 | |
| *** germs has joined #openstack-swift | 01:14 | |
| *** germs has quit IRC | 01:14 | |
| *** germs has joined #openstack-swift | 01:14 | |
| *** germs has quit IRC | 01:19 | |
| *** armaan has quit IRC | 01:34 | |
| *** threestrands_ has joined #openstack-swift | 03:12 | |
| *** threestrands_ has quit IRC | 03:12 | |
| *** threestrands_ has joined #openstack-swift | 03:12 | |
| *** germs has joined #openstack-swift | 03:15 | |
| *** germs has quit IRC | 03:15 | |
| *** germs has joined #openstack-swift | 03:15 | |
| *** threestrands has quit IRC | 03:16 | |
| *** germs has quit IRC | 03:20 | |
| *** pdardeau has quit IRC | 03:51 | |
| *** psachin has joined #openstack-swift | 03:51 | |
| *** pdardeau has joined #openstack-swift | 03:56 | |
| *** links has joined #openstack-swift | 03:57 | |
| openstackgerrit | Tim Burke proposed openstack/swift feature/deep: Stop replicating rows from large DBs to sharded containers https://review.openstack.org/549678 | 05:26 |
| *** geaaru has quit IRC | 05:51 | |
| *** cshastri_ has joined #openstack-swift | 06:03 | |
| *** links has quit IRC | 06:04 | |
| *** links has joined #openstack-swift | 06:05 | |
| *** cshastri has quit IRC | 06:07 | |
| *** cshastri_ has quit IRC | 06:08 | |
| *** cshastri_ has joined #openstack-swift | 06:20 | |
| *** threestrands_ has quit IRC | 06:57 | |
| mattoliverau | Well this is interesting: https://amp.thehackernews.com/thn/2018/02/memcached-amplification-ddos.html | 07:19 |
| *** gkadam has joined #openstack-swift | 07:28 | |
| *** hseipp has joined #openstack-swift | 07:39 | |
| *** rcernin has quit IRC | 07:42 | |
| *** hoonetorg has quit IRC | 07:46 | |
| *** cbartz has joined #openstack-swift | 07:47 | |
| *** armaan has joined #openstack-swift | 07:48 | |
| *** hoonetorg has joined #openstack-swift | 08:03 | |
| *** pcaruana has joined #openstack-swift | 08:05 | |
| *** tesseract has joined #openstack-swift | 08:11 | |
| acoles | good morning | 08:44 |
| *** geaaru has joined #openstack-swift | 08:47 | |
| rledisez | good morning | 08:49 |
| rledisez | everybody is back home? | 08:49 |
| acoles | rledisez: I believe so. Some had flights cancelled and flew home yesterday. I was delayed but still got home on Saturday. | 08:50 |
| acoles | rledisez: we missed seeing you and alecuyer, did you get home from Paris easily? | 08:51 |
| rledisez | yes, we almost spent an other night in Paris, the pilot was not sure to land in Brest. But we made it :) | 08:54 |
| *** armaan has quit IRC | 08:58 | |
| *** armaan has joined #openstack-swift | 08:59 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Use deterministic shard names, use configured account prefix https://review.openstack.org/549141 | 09:03 |
| *** gkadam has quit IRC | 09:16 | |
| *** gkadam has joined #openstack-swift | 09:17 | |
| *** ejat has quit IRC | 09:18 | |
| *** zigo has quit IRC | 09:18 | |
| *** dosaboy has quit IRC | 09:20 | |
| *** ejat has joined #openstack-swift | 09:20 | |
| *** ejat has quit IRC | 09:20 | |
| *** ejat has joined #openstack-swift | 09:20 | |
| *** dosaboy has joined #openstack-swift | 09:21 | |
| *** zigo has joined #openstack-swift | 09:23 | |
| *** mvk has joined #openstack-swift | 09:59 | |
| *** tovin07_ has quit IRC | 10:05 | |
| *** cshastri_ has quit IRC | 10:08 | |
| openstackgerrit | Gilles Biannic proposed openstack/swift master: Add template in proxy to create custom and anonymous access-logs https://review.openstack.org/548948 | 10:11 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Remove ShardRange.create() method https://review.openstack.org/549733 | 10:37 |
| openstackgerrit | Merged openstack/swift feature/deep: Use deterministic shard names, use configured account prefix https://review.openstack.org/549141 | 11:47 |
| *** links has quit IRC | 13:23 | |
| openstackgerrit | wangqi proposed openstack/python-swiftclient master: Update links in README https://review.openstack.org/549758 | 13:23 |
| *** ianychoi has joined #openstack-swift | 13:41 | |
| *** psachin has quit IRC | 13:42 | |
| *** psachin has joined #openstack-swift | 13:43 | |
| *** jistr is now known as jistr-mtg | 13:49 | |
| *** gkadam has quit IRC | 13:52 | |
| *** SkyRocknRoll has joined #openstack-swift | 13:57 | |
| *** psachin has quit IRC | 13:59 | |
| *** germs has joined #openstack-swift | 14:08 | |
| *** germs has quit IRC | 14:08 | |
| *** germs has joined #openstack-swift | 14:08 | |
| *** links has joined #openstack-swift | 14:18 | |
| openstackgerrit | Merged openstack/swift feature/deep: Remove ShardRange.create() method https://review.openstack.org/549733 | 14:19 |
| *** cshastri_ has joined #openstack-swift | 14:28 | |
| *** germs has quit IRC | 14:35 | |
| *** links has quit IRC | 14:40 | |
| *** cshastri_ has quit IRC | 14:43 | |
| *** SkyRocknRoll has quit IRC | 14:46 | |
| *** mvk has quit IRC | 14:47 | |
| *** germs has joined #openstack-swift | 14:55 | |
| *** germs has quit IRC | 14:55 | |
| *** germs has joined #openstack-swift | 14:55 | |
| *** germs has quit IRC | 14:56 | |
| *** germs has joined #openstack-swift | 14:57 | |
| *** germs has quit IRC | 14:57 | |
| *** germs has joined #openstack-swift | 14:57 | |
| *** timss has quit IRC | 14:57 | |
| *** timss has joined #openstack-swift | 14:57 | |
| *** raginbaji has quit IRC | 15:00 | |
| *** mvk has joined #openstack-swift | 15:02 | |
| *** raginbajin has joined #openstack-swift | 15:03 | |
| *** armaan has quit IRC | 15:15 | |
| *** armaan has joined #openstack-swift | 15:16 | |
| *** jistr-mtg is now known as jistr | 15:16 | |
| *** openstackgerrit has quit IRC | 15:34 | |
| *** zaitcev_ has joined #openstack-swift | 15:34 | |
| *** ChanServ sets mode: +v zaitcev_ | 15:34 | |
| *** armaan has quit IRC | 15:36 | |
| *** armaan has joined #openstack-swift | 15:36 | |
| *** zaitcev has quit IRC | 15:38 | |
| *** _alastor_ has joined #openstack-swift | 16:05 | |
| *** cbartz has quit IRC | 16:20 | |
| notmyname | good morningh | 16:35 |
| tdasilva | morningth | 16:36 |
| acoles | morningyth :) | 16:37 |
| notmyname | it's the irish brogue still with me ;-) | 16:38 |
| *** openstackgerrit has joined #openstack-swift | 16:40 | |
| openstackgerrit | Merged openstack/python-swiftclient master: Update links in README https://review.openstack.org/549758 | 16:40 |
| *** germs has quit IRC | 16:45 | |
| *** armaan_ has joined #openstack-swift | 16:46 | |
| *** armaan has quit IRC | 16:48 | |
| timburke | good morning | 17:01 |
| *** gerardgine has joined #openstack-swift | 17:09 | |
| *** hseipp has quit IRC | 17:26 | |
| *** gerardgine has quit IRC | 17:29 | |
| *** armaan_ has quit IRC | 18:05 | |
| *** armaan has joined #openstack-swift | 18:05 | |
| *** links has joined #openstack-swift | 18:09 | |
| *** itlinux has joined #openstack-swift | 18:13 | |
| *** mvk has quit IRC | 18:17 | |
| *** silor has joined #openstack-swift | 18:18 | |
| *** tesseract has quit IRC | 18:22 | |
| *** mvk has joined #openstack-swift | 18:32 | |
| *** geaaru has quit IRC | 18:43 | |
| *** germs has joined #openstack-swift | 18:46 | |
| *** germs has quit IRC | 18:51 | |
| *** links has quit IRC | 18:51 | |
| timburke | https://review.openstack.org/#/c/548948/ reminds me just how little i really understand about our logging... | 18:58 |
| patchbot | patch 548948 - swift - Add template in proxy to create custom and anonymo... | 18:58 |
| notmyname | oh! I missed that patch. that's a great idea! | 19:10 |
| tdasilva | timburke: what's GPPR? | 19:15 |
| notmyname | GDPR is what he means | 19:16 |
| tdasilva | General Data Protection Regulation? | 19:16 |
| notmyname | yeah, the big EU thing that adds a bunch of requirements on what you can and can't store about people. it includes IP addresses in logs | 19:17 |
| tdasilva | interesting | 19:17 |
| notmyname | regardless of GDPR, it's interesting from an openstack perspective for ops running lots of openstack services and wanting to configure their logs to look similar | 19:22 |
| tdasilva | yeah, sounds cool, was just curious... | 19:22 |
| *** gerardgine has joined #openstack-swift | 19:27 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Track cleave progress per db file rather than per node https://review.openstack.org/549857 | 19:33 |
| acoles | timburke: ^^ should help with you patch 549678 | 19:34 |
| patchbot | https://review.openstack.org/#/c/549678/ - swift (feature/deep) - Stop replicating rows from large DBs to sharded co... | 19:34 |
| *** silor has quit IRC | 19:35 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep: Track cleave progress per db file rather than per node https://review.openstack.org/549857 | 19:38 |
| notmyname | acoles: from last week... https://en.wikipedia.org/wiki/Auto-antonym | 19:42 |
| notmyname | "hew" is a good one, if you get tired of cleave | 19:42 |
| acoles | notmyname: interesting, thanks! lots of 'big words'. | 19:43 |
| acoles | I like 'antogonym', but I'd like it to mean 'a word that antagonises' :) | 19:44 |
| acoles | some of the examples are tenuous imho e.g. 'Black Friday' | 19:46 |
| acoles | notmyname: 'rude' had come to mind after that conversation - rude joke vs rude health | 19:50 |
| notmyname | I'm not familiar with a double meaning there. "rude health" doesn't make sense to me | 19:51 |
| notmyname | better say that some examples should be sanctioned ;-) | 19:51 |
| notmyname | or, "I cannot recommend its use more highly" | 19:52 |
| acoles | notmyname: ah, another britishness perhaps ... https://en.oxforddictionaries.com/definition/rude | 19:52 |
| notmyname | ah, interesting | 19:53 |
| notmyname | /lunchtime | 19:53 |
| tdasilva | TIL: rude health | 19:57 |
| tdasilva | :) | 19:57 |
| tdasilva | timburke: i think p 540190 is ready to merge, right? | 19:57 |
| patchbot | https://review.openstack.org/#/c/540190/ - swift - py3: port common/wsgi.py | 19:57 |
| tdasilva | i saw your +1, guessing since it's your patch you didn't want to +2??? or were you planning to take another look? | 19:58 |
| timburke | tdasilva: bingo -- just a confirmation that your diff lgtm | 20:26 |
| tdasilva | timburke: ok, then let's merge it | 20:27 |
| timburke | +A all the things! | 20:28 |
| timburke | there's so much good stuff in https://review.openstack.org/#/q/project:openstack/swift+branch:master+is:open+label:Code-Review%252B2+-label:Code-Review-2+-label:Verified-1+-label:Workflow%252B1+-label:Workflow-1 | 20:30 |
| *** gerardgine has quit IRC | 20:32 | |
| *** d0ugal has quit IRC | 20:32 | |
| *** d0ugal has joined #openstack-swift | 20:34 | |
| *** gerardgine has joined #openstack-swift | 20:38 | |
| *** silor has joined #openstack-swift | 20:47 | |
| *** gerardgine has quit IRC | 20:48 | |
| *** gerardgine has joined #openstack-swift | 21:04 | |
| *** gyee has joined #openstack-swift | 21:22 | |
| notmyname | tdasilva: 19 patchsets and I see success on "swift-probetests-centos-7" https://review.openstack.org/#/c/549248/ | 21:26 |
| patchbot | patch 549248 - swift - Add swift probe tests to gate | 21:26 |
| notmyname | is it ready for review? | 21:26 |
| tdasilva | notmyname: yep, good to go | 21:26 |
| notmyname | sweet! | 21:26 |
| notmyname | FWIW, I'd love to have this as a gate job, too, and sooner rather than later | 21:31 |
| notmyname | (obviously after some time as a check job) | 21:31 |
| tdasilva | notmyname: yeah! totally agree, hopefully it will be stable, curious to see it running against other patches | 21:32 |
| *** d0ugal has quit IRC | 21:33 | |
| timburke | speaking of gate stability... | 21:37 |
| timburke | anyone else notice some RETRY_LIMITs on the legacy jobs? | 21:37 |
| timburke | i see things like `[Errno 13] Permission denied: '/usr/local/jenkins'` and `change_dir "/home/zuul/workspace" failed: No such file or directory (2)` in the logs... | 21:38 |
| timburke | also, there's http://logs.openstack.org/90/540190/6/gate/swift-tox-py27/db6e59c/job-output.txt.gz#_2018-03-05_20_52_30_898170 which i haven't been able to repro on master, even after a couple thousand runs... i'll try merging in the patch it was testing, but it wasn't really touching the replicator... | 21:39 |
| *** d0ugal has joined #openstack-swift | 21:41 | |
| tdasilva | timburke: yeah, also see that same error here: http://logs.openstack.org/48/549248/19/check/swift-tox-py27/faee8d3/job-output.txt.gz#_2018-03-04_15_40_44_546213 | 21:44 |
| *** threestrands_ has joined #openstack-swift | 21:45 | |
| *** threestrands_ has quit IRC | 21:45 | |
| *** threestrands_ has joined #openstack-swift | 21:45 | |
| timburke | i wonder if that logging is throwing things off a little? since it will (or can, at any rate) cause a trampoline... in which case we may get killed again before we have a chance to call terminate | 21:47 |
| timburke | that at least makes *some* sense when you see two `Lockup detected.. killing live coros.` log lines like in yours... mine seems weirder... | 21:48 |
| *** pcaruana has quit IRC | 21:57 | |
| *** itlinux has quit IRC | 22:03 | |
| *** silor has quit IRC | 22:12 | |
| *** rcernin has joined #openstack-swift | 22:34 | |
| *** itlinux has joined #openstack-swift | 22:50 | |
| *** itlinux has quit IRC | 22:51 | |
| timburke | FWIW, the RETRY_LIMIT errors were probably resolved by https://review.openstack.org/#/c/549871/ -- i'm gonna go recheck some jobs | 22:52 |
| patchbot | patch 549871 - openstack-infra/openstack-zuul-jobs - Create old jenkins directories (MERGED) | 22:52 |
| *** jappleii__ has joined #openstack-swift | 23:09 | |
| *** gerardgine has quit IRC | 23:11 | |
| *** jappleii__ has quit IRC | 23:12 | |
| *** threestrands_ has quit IRC | 23:12 | |
| *** armaan has quit IRC | 23:12 | |
| *** threestrands has joined #openstack-swift | 23:12 | |
| *** threestrands has quit IRC | 23:12 | |
| *** threestrands has joined #openstack-swift | 23:12 | |
| *** gerardgine has joined #openstack-swift | 23:13 | |
| clayg | tdasilva: timburke: do... did we decide what we're going to do with the probetest patch? is it really configured to be voting right now? | 23:15 |
| timburke | clayg: how about i push up a follow-up that would break them ;-) | 23:16 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Check whether new probe test job is voting https://review.openstack.org/549885 | 23:18 |
| *** kei_yama has joined #openstack-swift | 23:18 | |
| timburke | we'll find out whether it's voting in 15-20 mins | 23:19 |
| clayg | optimistic | 23:26 |
| timburke | mm... true -- i have to wait until *all* the jobs finish :-( | 23:31 |
| timburke | but test_replicate_lockup_detector strikes again! | 23:32 |
| timburke | i hate the lockup detector... | 23:33 |
| clayg | timburke: lp bug #1752324 is annoying/scary!? | 23:33 |
| openstack | Launchpad bug 1752324 in OpenStack Object Storage (swift) "Folded headers can cause 503 responses, or even no response at all" [Medium,Confirmed] https://launchpad.net/bugs/1752324 | 23:33 |
| timburke | but no one uses folded headers -- otherwise we would've heard about it earlier | 23:34 |
| notmyname | timburke: that bug tickles some memory somewhere | 23:36 |
| notmyname | I feel like we talked about this issue to some extent way back when. at least back in the webob days | 23:36 |
| notmyname | maybe it was more related to what happens when you include a single header key more than one time | 23:37 |
| notmyname | but I thought at least some subset of your bug report was working. like maybe just with a newline? | 23:37 |
| notmyname | isn't it an issue with eventlet, though, not swift? is swift even able to see what the bytes on wire were? we just get tuples or a dict | 23:38 |
| timburke | i'm pretty sure most of the work needs to be done in eventlet. that or cpython | 23:38 |
| torgomatic | FWIW, RFC 7230 basically says "stop it with the header folding" | 23:39 |
| clayg | good 'ol 7230 | 23:39 |
| torgomatic | so Swift's header parser isn't broken, it's *modern* | 23:39 |
| notmyname | whew... done. "Invalid. Go find a different place to file this bug" ;-) http://hitchhikers.wikia.com/wiki/Somebody_Else%27s_Problem_field | 23:39 |
| clayg | in the context of the link I imagined the field like a form field... like on a bug report... something akin to "won't fix" but just SEP | 23:42 |
| torgomatic | yep, especially since the HTTP header parsing happens in Python's httplib (I think), so supporting folded headers is even more annoying than it might otherwise be | 23:44 |
| notmyname | would be nice if we could catch that and make it a 4xx error instead of 5xx. it's the user doing the wrong thing, not swift | 23:49 |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Support -d <devs> and -p <partitions> in DB replicators. https://review.openstack.org/548628 | 23:50 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Check whether new probe test job is voting https://review.openstack.org/549885 | 23:52 |
| openstackgerrit | Merged openstack/swift master: Try to avoid leaving (killed) long-running rsyncs in the process table https://review.openstack.org/549494 | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!