*** baojg has quit IRC | 00:33 | |
*** vint_bra has joined #openstack-swift | 00:48 | |
*** tovin07_ has joined #openstack-swift | 01:00 | |
kota_ | good morning | 01:45 |
---|---|---|
kota_ | hello mattoliverau | 01:45 |
mattoliverau | kota_: morning o/ | 01:45 |
openstackgerrit | Matthew Oliver proposed openstack/swift master: Container sharding docs https://review.openstack.org/438322 | 02:02 |
openstackgerrit | Matthew Oliver proposed openstack/swift master: Add container sharding to Swift containers https://review.openstack.org/498285 | 02:02 |
kota_ | container sharding is getting progressed, nice! | 02:02 |
mattoliverau | kota_: yeah, slowly :P finally getting back into it after the craziness that has been the last few months | 02:05 |
*** hoonetorg has quit IRC | 02:43 | |
*** itlinux has quit IRC | 02:45 | |
*** kiennt has joined #openstack-swift | 02:54 | |
*** hoonetorg has joined #openstack-swift | 03:00 | |
*** silor has joined #openstack-swift | 03:07 | |
*** abhinavtechie has joined #openstack-swift | 03:07 | |
*** abhitechie has quit IRC | 03:10 | |
*** kei_yama has quit IRC | 03:26 | |
*** mrhillsman has left #openstack-swift | 03:31 | |
*** gkadam has joined #openstack-swift | 03:33 | |
mahatic | good morning | 03:37 |
mahatic | mathiasb: I'm late to the party, congrats on patch 364878 merge! | 03:40 |
patchbot | https://review.openstack.org/#/c/364878/ - swift - Retrieve encryption root secret from Barbican (MERGED) | 03:40 |
mattoliverau | mahatic: morning | 03:56 |
mahatic | mattoliverau: hello | 03:56 |
*** vint_bra has quit IRC | 03:57 | |
*** kei_yama has joined #openstack-swift | 03:59 | |
*** links has joined #openstack-swift | 04:00 | |
*** silor has quit IRC | 04:21 | |
*** kei_yama has quit IRC | 04:26 | |
*** silor has joined #openstack-swift | 04:47 | |
*** kiennt has quit IRC | 04:47 | |
*** kiennt has joined #openstack-swift | 04:48 | |
*** kei_yama has joined #openstack-swift | 04:51 | |
*** jaosorior has joined #openstack-swift | 05:18 | |
*** rcernin has joined #openstack-swift | 05:30 | |
*** silor has quit IRC | 05:34 | |
*** geaaru has joined #openstack-swift | 05:47 | |
*** abhitechie has joined #openstack-swift | 05:55 | |
*** abhinavtechie has quit IRC | 05:57 | |
*** abhinavtechie has joined #openstack-swift | 06:14 | |
*** abhitechie has quit IRC | 06:15 | |
openstackgerrit | Matthew Oliver proposed openstack/swift master: Add container sharding to Swift containers https://review.openstack.org/423906 | 06:17 |
openstackgerrit | Matthew Oliver proposed openstack/swift master: Container sharding docs https://review.openstack.org/438322 | 06:17 |
*** bkopilov has joined #openstack-swift | 06:29 | |
*** silor has joined #openstack-swift | 06:34 | |
*** pcaruana has joined #openstack-swift | 06:36 | |
*** gkadam_ has joined #openstack-swift | 06:45 | |
*** gkadam has quit IRC | 06:47 | |
*** hseipp has joined #openstack-swift | 06:56 | |
*** cschwede has joined #openstack-swift | 06:59 | |
*** ChanServ sets mode: +v cschwede | 06:59 | |
*** oshritf has joined #openstack-swift | 07:01 | |
openstackgerrit | lidong proposed openstack/python-swiftclient master: Update links in README https://review.openstack.org/498335 | 07:04 |
*** silor has quit IRC | 07:20 | |
*** d0ugal has joined #openstack-swift | 07:30 | |
*** tesseract has joined #openstack-swift | 07:31 | |
*** silor has joined #openstack-swift | 07:45 | |
acoles | good morning | 07:50 |
*** silor has quit IRC | 08:08 | |
*** d0ugal has quit IRC | 08:35 | |
*** jaosorior has quit IRC | 08:39 | |
*** kiennt_ has joined #openstack-swift | 08:46 | |
*** kiennt has quit IRC | 08:49 | |
*** jaosorior has joined #openstack-swift | 08:50 | |
*** abhinavtechie has quit IRC | 09:12 | |
mathiasb | thanks, mahatic! :) | 09:24 |
*** gabor_antal has quit IRC | 09:28 | |
*** cschwede has quit IRC | 09:40 | |
*** cschwede has joined #openstack-swift | 09:42 | |
*** ChanServ sets mode: +v cschwede | 09:42 | |
acoles | torgomatic: clayg thanks for getting https://review.openstack.org/#/c/460073/ into shape | 09:53 |
patchbot | patch 460073 - swift - Add probe test for ssync of unexpired metadata to ... (MERGED) | 09:53 |
*** mat128 has joined #openstack-swift | 09:57 | |
*** silor has joined #openstack-swift | 10:02 | |
*** kiennt has joined #openstack-swift | 10:11 | |
*** kei_yama has quit IRC | 10:12 | |
*** kiennt_ has quit IRC | 10:13 | |
*** silor has quit IRC | 10:19 | |
*** silor1 has joined #openstack-swift | 10:19 | |
*** silor1 is now known as silor | 10:21 | |
*** kiennt has quit IRC | 10:27 | |
*** tovin07_ has quit IRC | 10:28 | |
*** silor has quit IRC | 10:34 | |
openstackgerrit | Matthew Oliver proposed openstack/swift master: Small readability change to test_keystoneauth https://review.openstack.org/498202 | 10:35 |
*** CowboyPr1de has joined #openstack-swift | 10:37 | |
CowboyPr1de | Good morning all. I added a new region/storage node to swift cluster this weekend and am wondering if my replica count needs to be increased? https://pastebin.com/UYVZb317 | 10:38 |
*** d0ugal has joined #openstack-swift | 10:40 | |
openstackgerrit | Merged openstack/swift master: Small readability change to test_keystoneauth https://review.openstack.org/498202 | 10:51 |
*** jamielennox has quit IRC | 10:51 | |
*** jamielennox has joined #openstack-swift | 10:52 | |
*** baojg has joined #openstack-swift | 11:27 | |
*** gabor_antal has joined #openstack-swift | 11:31 | |
*** szaher has quit IRC | 11:32 | |
*** szaher has joined #openstack-swift | 11:36 | |
*** bkopilov has quit IRC | 11:39 | |
*** lifeless has quit IRC | 12:15 | |
*** lifeless has joined #openstack-swift | 12:17 | |
*** cschwede has quit IRC | 12:24 | |
*** cschwede has joined #openstack-swift | 12:26 | |
*** ChanServ sets mode: +v cschwede | 12:26 | |
*** catintheroof has joined #openstack-swift | 12:36 | |
*** mat128 has quit IRC | 13:03 | |
*** gkadam_ has quit IRC | 13:04 | |
*** lucasxu has joined #openstack-swift | 13:05 | |
*** SkyRocknRoll has joined #openstack-swift | 13:21 | |
*** bkopilov has joined #openstack-swift | 13:37 | |
*** mat128 has joined #openstack-swift | 13:56 | |
*** mat128 has quit IRC | 13:57 | |
*** abhitechie has joined #openstack-swift | 14:08 | |
*** baojg has quit IRC | 14:11 | |
*** baojg has joined #openstack-swift | 14:14 | |
*** links has quit IRC | 14:22 | |
*** geaaru has quit IRC | 14:52 | |
*** gyee has joined #openstack-swift | 14:56 | |
*** mat128 has joined #openstack-swift | 15:08 | |
*** mat128 has quit IRC | 15:11 | |
*** geaaru has joined #openstack-swift | 15:13 | |
*** oshritf has quit IRC | 15:30 | |
*** jistr is now known as jistr|off|mtg | 15:33 | |
*** jistr|off|mtg is now known as jistr | 15:33 | |
*** links has joined #openstack-swift | 15:33 | |
notmyname | good morning | 15:42 |
notmyname | CowboyPr1de: why are you thinking you need to add replicas? | 15:44 |
notmyname | CowboyPr1de: based on the pastebin, you've got 2 region and about 1/6 of the data has a replica in the 2nd region (1 of six servers, all with 4 drives of equal weight) | 15:45 |
notmyname | CowboyPr1de: there's nothing wrong with that output. but I don't know if it matches your goals or not | 15:46 |
*** pcaruana has quit IRC | 15:47 | |
*** mat128 has joined #openstack-swift | 16:01 | |
*** rcernin has quit IRC | 16:10 | |
*** vint_bra has joined #openstack-swift | 16:18 | |
*** hseipp has quit IRC | 16:22 | |
*** klrmn has quit IRC | 16:23 | |
*** links has quit IRC | 16:27 | |
timburke | good morning | 16:29 |
*** tesseract has quit IRC | 16:30 | |
*** abhinavtechie has joined #openstack-swift | 16:33 | |
*** dmsimard has joined #openstack-swift | 16:35 | |
dmsimard | timburke: Hi! Re: https://bugs.launchpad.net/openstack-ansible/+bug/1707220 | 16:35 |
openstack | Launchpad bug 1707220 in openstack-ansible "Swift Erasure Code fails with liberasurecode 1.4.0 on CentOS" [Low,In progress] - Assigned to Andy McCrae (andrew-mccrae) | 16:35 |
timburke | dmsimard: hi! yeah, i remember that guy! | 16:36 |
*** abhitechie has quit IRC | 16:36 | |
dmsimard | timburke: We'd be happy to update libec to 1.5.0 (from 1.4.0) in RDO to fix that issue for Ocata but the current upper-constraints pins it to 1.4.0: https://github.com/openstack/requirements/blob/0651044b20227dad1d8247be47614a2447c21793/upper-constraints.txt#L286 | 16:36 |
dmsimard | We'd rather not bump our version of the package ahead of upstream, could you submit a bump to u-c ? | 16:37 |
timburke | happily | 16:37 |
notmyname | timburke: any dependency restrictions that adds for pyeclib? | 16:38 |
dmsimard | timburke: Thanks ! Add me on the review ? dms [at] redhat | 16:38 |
timburke | notmyname: maybe? i was gonna propose both. we keep the interface stable enough that it shouldn't present any troubles for swift, and i don't know of any other openstack projects using either | 16:39 |
*** mat128 has quit IRC | 16:39 | |
notmyname | timburke: ya, makes sense. and now at the start of a cycle is a reasonable time to update it | 16:39 |
timburke | actually, now that i think about it more, i think requirements repo *only* has pyeclib :-) | 16:40 |
clayg | heyoh! | 16:41 |
notmyname | good morning clayg | 16:42 |
clayg | acoles: so i have this probe test for expired objects and revert (and w/ unexpired metadata w/e) | 16:42 |
timburke | dmsimard: looks like pyeclib got updated as part of https://github.com/openstack/requirements/commit/a76a545 (which is included in pike) -- i guess you just need it back on ocata? | 16:43 |
clayg | but it has *two* dependents (patch 498107 and patch 498107) | 16:44 |
patchbot | https://review.openstack.org/#/c/498107/ - swift - Make X-Backend-Replication consistent for HEAD | 16:44 |
clayg | umm... the other is p 458070 | 16:44 |
patchbot | https://review.openstack.org/#/c/458070/ - swift - Always require device dir for containers | 16:44 |
acoles | clayg: looking... | 16:45 |
clayg | I don't even know how to express that in gerrit? I bunch of Depends-On I guess... but how should I handle getting my working tree in order? two merge commits follow by the change? | 16:45 |
clayg | i guess I can just start with figuring out what's up with the dsvm environment and p 458070 | 16:46 |
patchbot | https://review.openstack.org/#/c/458070/ - swift - Always require device dir for containers | 16:46 |
dmsimard | timburke: right, the ocata upper-constraints still has 1.4.0 | 16:48 |
acoles | clayg: yeah I think rebase on one and Depends-On the other, but that is only useful if the 'other' is a weak dependency (i.e. you don't need the code changes for your patch to work, you just want to enforce a merge order) | 16:48 |
dmsimard | timburke: so it's a bump of upper-constraints from stable/ocata that we need | 16:48 |
*** rcernin has joined #openstack-swift | 16:48 | |
acoles | clayg: better may be to tell me which of the others I should review and try to get one merged ASAP | 16:49 |
notmyname | clayg: acoles: if you do need to push a patch to gerrit that has 2 parents, I can teach you how (note that only cores can propose patches like that) | 16:49 |
clayg | wooooooo | 16:50 |
acoles | notmyname: oh, that's interesting | 16:50 |
notmyname | hint: it's exactly the same as proposing merge commits to a feature branch. so you've both already done it before ;-) | 16:50 |
clayg | well i need to get both of the parents working - then yeah if they're not merged I'll want to purpose the "why" patch (with two parents) | 16:50 |
notmyname | IIRC `gerrit review -R ...` | 16:50 |
clayg | reivew *-R* - what sorcery!? | 16:51 |
timburke | https://review.openstack.org/#/c/498521/ if anyone else cares | 17:01 |
patchbot | patch 498521 - requirements (stable/ocata) - Update PyEClib | 17:01 |
*** mat128 has joined #openstack-swift | 17:03 | |
*** chlong has joined #openstack-swift | 17:07 | |
*** prometheanfire has joined #openstack-swift | 17:14 | |
prometheanfire | what's with the pyeclib bump in ocata? https://review.openstack.org/498521 | 17:14 |
patchbot | patch 498521 - requirements (stable/ocata) - Update PyEClib | 17:14 |
*** klrmn has joined #openstack-swift | 17:15 | |
timburke | prometheanfire: there are some segfaults that have caused issues for downstream (see also https://bugs.launchpad.net/openstack-ansible/+bug/1707220), but RDO (at least) feels uncomfortable advancing beyond what's listed in (ocata's) upper-constraints | 17:34 |
openstack | Launchpad bug 1707220 in openstack-ansible "Swift Erasure Code fails with liberasurecode 1.4.0 on CentOS" [Low,In progress] - Assigned to Andy McCrae (andrew-mccrae) | 17:34 |
prometheanfire | ya, I can understand limiting to UC | 17:37 |
prometheanfire | can you update the review with bug links and better comments as to why? | 17:37 |
*** cschwede has quit IRC | 17:41 | |
*** __david_ has joined #openstack-swift | 17:41 | |
*** baojg has quit IRC | 18:09 | |
*** baojg has joined #openstack-swift | 18:14 | |
*** itlinux has joined #openstack-swift | 18:23 | |
timburke | mattoliverau: if you want https://review.openstack.org/#/c/464217/ you may need to go review https://review.openstack.org/#/c/464216/ too :-) | 18:32 |
patchbot | patch 464217 - swift - domain_remap: stop mangling client-provided paths | 18:32 |
patchbot | patch 464216 - swift - domain_remap: be more careful about client-path ma... | 18:32 |
*** silor has joined #openstack-swift | 18:37 | |
*** SkyRocknRoll has quit IRC | 18:43 | |
*** silor has quit IRC | 18:44 | |
clayg | that moment when you realize the problem is "you're just not thinking 4th dimensionally!" | 18:53 |
*** catintheroof has quit IRC | 19:04 | |
__david_ | We've been working on adding a storage policy to an existing set of swift clusters in order to use multi-region functionality. The existing clusters are 3 copy with ACO rings only referencing local ACO nodes. In a staging setup with two clusters, I have a default policy in each cluster with ACO rings pointing to local nodes ( just like production) and then a new storage policy has been added with 4 copies. with equal number | 19:13 |
__david_ | As the account and container rings are not multi-region, you cannot access the objects on the remote cluster by default. I have been looking at mutli-region / storage policy docs and talks but I can't see any specific guidance about this. If I PUT objects to multi-region container in region 1 and then 'seed' the same containers (POST with storage policy metadata) in region 2, I can stat objects region 2 directly. Am I miss | 19:13 |
*** catintheroof has joined #openstack-swift | 19:17 | |
clayg | __david_: multi-region clusters are a single flat name space - metadata operations happen across the WAN to account & container database replicas distributed globally. | 19:19 |
clayg | __david_: if some of the remote replcias can't be reached for a metadata update those records are stored locally and trasfered async | 19:19 |
clayg | most metadata updates can happen out of band from the object data transfer | 19:20 |
clayg | all operations (data & metadata) are robust to the unavailability of remote replicas | 19:20 |
clayg | __david_: maybe https://docs.openstack.org/swift/latest/admin_guide.html#geographically-distributed-swift-considerations ??? | 19:24 |
timburke | clayg: on p 458070 -- why do we check_mount in account/container server, but for obj we do it down in diskfile? should that patch instead be messing with swift/common/db.py? | 19:39 |
patchbot | https://review.openstack.org/#/c/458070/ - swift - Always require device dir for containers | 19:39 |
clayg | the databases don't really have a diskfile-like layer abstraction - ask zaitcev about it sometime. | 19:40 |
clayg | tl;dr "no that patch shouldn't do that; yes some other useful patch in the future might move some things around around in that area" | 19:41 |
zaitcev | timburke: Peter Portante had a particular dislike for excessive mount checking and wanted to get rid of it or at least attenuate it. He saw too much of it happening in strace, I gather. | 19:44 |
timburke | i kinda feel like the broker would be an appropriate place for it? but what do i know... | 19:44 |
zaitcev | Yes, IIRC I moved it too, because obviously the mount checking is implemented differently for non-posix back-ends. | 19:45 |
* portante had a particular love for performance | 19:45 | |
timburke | zaitcev: so does this mean we should be getting it out of diskfile, too? hoist it up to obj/server.py? i still don't have the insight of "this *obviously* belongs *here*" | 19:47 |
__david_ | Thanks clayg. Adding a SSD storage policy or a 2-copy storage policy in a local cluster is straight-forward, but retro-filling a multi-region policy to two independent clusters is somewhat mind-bending . | 19:49 |
__david_ | If we choose to not turn our existing AC rings into multi-region rings, will that create problems? Region 2 from my example above has objects which are not reflected in the container listing in that region. De-coupling this fundamental tenant of Swift seems wrong, but I can't dete rmine another path without massive changes for our existing tenants/containers. | 19:49 |
clayg | timburke: it's obvious when you look at the diff - this was were ht was - so continuing to do it here when the goal is to keep the behavior the same for mount_check = true is "obviously correct" - if I moved in this patch that'd be doing two things - the second thing is ok, but it's a side show - not having check_dir behavior in database servers is stupid/annoying and causing problems with testing in | 19:51 |
clayg | environments that use dirs (e.g. vsaio, devstack, etc)_ | 19:51 |
clayg | i've almost got the devstack change working - i'll wip that patch for now | 19:52 |
clayg | FWIW zaitcev's patches to make the db brokers a better abstraction were great | 19:53 |
clayg | we just never managed to get the merged | 19:53 |
timburke | having a BaseDatabaseServer that subclasses BaseStorageServer feels weird to me. or rather, it feels weird that we only have this extra layer of abstraction so we can switch between check_mount and check_dir when we have a very similar switch in https://github.com/openstack/swift/blob/2.15.1/swift/obj/diskfile.py#L1193-L1198 | 19:54 |
*** wantstofail has joined #openstack-swift | 19:55 | |
timburke | wait, so the failure was *legit*? i was real close to just saying "recheck" ... maybe it's better i *don't* review it... | 19:56 |
*** dmsimard has left #openstack-swift | 19:58 | |
clayg | yeah devstack doesn't create the device dirs :'( | 20:00 |
clayg | the account/container server would create the device dirs - so when the object-server required they be there... THEY WERE! | 20:01 |
clayg | pretty cool | 20:01 |
clayg | i don't know how to help you "feel better" about having a class that implements a method that's used by two subclasses? | 20:01 |
* clayg shrugs | 20:01 | |
clayg | it seems better than having to sublcasses both implement the same method | 20:01 |
clayg | but if *that's* thing you wanna try to spin a diff for today - all I can hope for is follow up :\ | 20:02 |
*** mat128 has quit IRC | 20:03 | |
timburke | so what about places like https://github.com/openstack/swift/blob/2.15.1/swift/container/updater.py#L103 ? should *they* be doing check_dir too? | 20:06 |
clayg | yeah ideally! although afaik we mostly only use the "break device" stuff for the 507 behavior | 20:07 |
*** baojg_ has joined #openstack-swift | 20:07 | |
timburke | if we need that logic in account, container, and object servers, it seems like it belongs in BaseStorageServer. if we won't put it there because object server already has it covered on account of diskfile, it seems very strange to me that we aren't addressing the account/container servers in a similar way | 20:07 |
clayg | timburke: I'll agree it's not ideal - can't build a cathedral with mud & spackling! | 20:09 |
*** baojg has quit IRC | 20:09 | |
clayg | what we *really* need is a generic abstraction for all mounted filesystem based storage subsystems that both the sqlitedb and xfs .data file implementations can inherit from | 20:11 |
notmyname | https://en.wikipedia.org/wiki/Great_Mosque_of_Djenné | 20:11 |
notmyname | "mud cathedral" | 20:12 |
notmyname | clayg: yes! like an "under" diskfile ;-) | 20:12 |
clayg | or we could just say - this method call right here with the ismount - it needs to also with the isdir - if we stuff this code in like this it will 1) do that and b) we should be able to grok it enough to change it later. | 20:15 |
notmyname | would it help rledisez and his LOSF work? | 20:15 |
clayg | not really, no | 20:23 |
clayg | timburke: there's also that change that adds the magic file check for mounts ... something for containers cschwede was working on... | 20:26 |
clayg | I guess instead of a base class we could just let it be a pure function that takes self.root and self.mount_check ??? | 20:26 |
clayg | if nothing else it's weird that I moved self.mount_check = and not self.root = | 20:27 |
timburke | https://review.openstack.org/#/c/466255/ ? | 20:27 |
patchbot | patch 466255 - swift - Make mount_check option usable in containerized en... (MERGED) | 20:27 |
timburke | there's also https://review.openstack.org/#/c/311703/ ... | 20:29 |
patchbot | patch 311703 - swift - Respect check_mount setting in recons disk_usage | 20:29 |
torgomatic | __david_: are you trying to merge two completely separate Swift clusters together by adding the same storage policy with the same devices to both of them? | 20:34 |
__david_ | torgomatic - we want to add the ability to have containers that are replicate to another region. The existing accounts/containers that are local in each cluster we want to remain local. | 20:36 |
torgomatic | __david_: sounds like you should be looking at container sync, then | 20:36 |
clayg | or eran's patches for container sync anyway :\ | 20:36 |
mattoliverau | timburke: ok I'll take a look today :) had a thought about how to improve the sharer.. I'll flush it out, but of it needs to wait until sharer 2.0 then that's ok too :) | 20:46 |
*** lucasxu has quit IRC | 20:46 | |
mattoliverau | *sharder | 20:46 |
__david_ | Thanks torgomatic and clayg | 20:47 |
*** chlong has quit IRC | 20:53 | |
*** wantstofail has quit IRC | 21:03 | |
*** tonyb has quit IRC | 21:04 | |
*** catintheroof has quit IRC | 21:09 | |
*** tonyb has joined #openstack-swift | 21:11 | |
*** geaaru has quit IRC | 21:15 | |
clayg | timburke: i'm going to re-spin p 458070 with depends on p 498604 | 21:23 |
patchbot | https://review.openstack.org/#/c/458070/ - swift - Always require device dir for containers | 21:23 |
patchbot | https://review.openstack.org/#/c/498604/ - openstack-dev/devstack - Create correct directory layout for swift on purpose. | 21:23 |
clayg | timburke: any final thoughts on how to make the BaseDatabaseStorageServer thing suck less? | 21:23 |
*** chlong has joined #openstack-swift | 21:26 | |
*** chlong has quit IRC | 21:31 | |
*** rcernin has quit IRC | 21:32 | |
*** chlong has joined #openstack-swift | 21:43 | |
openstackgerrit | Clay Gerrard proposed openstack/swift master: Always require device dir for containers https://review.openstack.org/458070 | 22:10 |
*** itlinux has quit IRC | 22:28 | |
*** prometheanfire has left #openstack-swift | 22:29 | |
*** abhinavtechie has quit IRC | 22:54 | |
*** vint_bra has quit IRC | 23:14 | |
*** itlinux has joined #openstack-swift | 23:28 | |
*** itlinux has quit IRC | 23:28 | |
*** kei_yama has joined #openstack-swift | 23:30 | |
*** chlong has quit IRC | 23:34 | |
*** alenavolk has joined #openstack-swift | 23:47 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!