notmyname | great! | 00:00 |
---|---|---|
notmyname | mahatic: now I've got to ask the important question: what in the world are you doing awake right now?!?! ;-) | 00:00 |
notmyname | isn't it like 4am for you? | 00:00 |
notmyname | (or 5:30) | 00:00 |
mahatic | notmyname, lol. I really thought I'd get done with this by 2 am and now it's 5.30! | 00:01 |
*** dmsimard is now known as dmsimard_away | 00:01 | |
mahatic | git ate up more than an hour! | 00:02 |
mahatic | notmyname, about the patch: I don't have a test for scout_server_type yet. I plan to get this reviewed and then a diff patch for that test. | 00:03 |
mahatic | (because I think it will be easier to review, than all code at once. Correct me if I'm wrong) | 00:04 |
*** ho has joined #openstack-swift | 00:16 | |
ho | good morning! | 00:18 |
*** nellysmitt has joined #openstack-swift | 00:38 | |
*** dmorita has joined #openstack-swift | 00:40 | |
mattoliverau | ho: morning | 00:41 |
*** nellysmitt has quit IRC | 00:43 | |
klrmn | ok, probe tests care if the servers are stopped vs killed…who knew? | 00:45 |
*** jrichli has quit IRC | 01:16 | |
*** jamielennox is now known as jamielennox|away | 01:16 | |
*** jrichli has joined #openstack-swift | 01:16 | |
*** jrichli has quit IRC | 01:16 | |
*** jrichli has joined #openstack-swift | 01:18 | |
notmyname | I've been working a little today on using the openstack wiki swift page for project tracking stuff. like we talked about last week | 01:18 |
notmyname | for reference, I move the current page to https://wiki.openstack.org/wiki/OldSwift | 01:18 |
notmyname | and I've started on https://wiki.openstack.org/wiki/Swift | 01:18 |
notmyname | the "project links" section is where I want to spend most of my time | 01:19 |
notmyname | I want to make that more visible and more full-featured. some of that I'll have to lear how to do on the wiki | 01:19 |
notmyname | and the "related pages" section at the bottom is pretty cool, i think | 01:19 |
notmyname | next up, I'm going to work on the review dashboard and the priority reviews page to get those better | 01:20 |
notmyname | I'll be back for more tomorrow. | 01:21 |
notmyname | good night | 01:21 |
jrichli | good night! | 01:22 |
*** mahatic has quit IRC | 01:24 | |
*** jamielennox|away is now known as jamielennox | 01:31 | |
*** jamielennox is now known as jamielennox|away | 01:41 | |
ho | mattoliverau: I would like to have your re-review for #138342 (recon). Can I have it? | 02:06 |
*** jamielennox|away is now known as jamielennox | 02:09 | |
*** dmsimard_away is now known as dmsimard | 02:25 | |
mattoliverau | ho: sure, I'll take a look this arvo, sorry things just keep getting in my way of fun swift stuff today :( I guess that happens when you've been out of the office for a week. But think I'm all caught up now | 02:35 |
*** nellysmitt has joined #openstack-swift | 02:39 | |
*** nellysmitt has quit IRC | 02:44 | |
ho | mattoliverau: thanks! I'm sorry, I know you mast be busy... | 02:48 |
mattoliverau | ho: nah its fine, its my first day back is all, yours is at the top of my list :) | 02:58 |
ho | mattoliverau: great! | 03:02 |
*** doxavore has joined #openstack-swift | 03:16 | |
mattoliverau | dmorita: notmyname wanted to let you know yesterday that slogging is updated to work with current swift, and clayg wanted you to know that he's got a patch up for muliple reconciler (regarding your SP migration research) https://review.openstack.org/#/c/103779 | 03:53 |
*** reed has quit IRC | 03:56 | |
dmorita | mattoliverau: thanks for letting me know that. I noticed slogging update this morning and I would like to give notmyname a word of thank you. I will check clayg's patch soon deeply but it must be great patch :) | 04:03 |
dmorita | notmyname: I am very happy that slogging now supports json format and different timezone output. Thanks for merging my pull requests! In addition, I am very sorry for bothering you in hack-a-thon to modify unit tests. I should fix tests before sending pull requests. | 04:09 |
doxavore | any idea why tempauth would succeed for an /auth/v1.0 request and return an AUTH_tk, then a HEAD /d0/231526/AUTH_test returns a 400? | 04:13 |
doxavore | i've tried turning up debug logging, but the output is exactly the same | 04:14 |
doxavore | all of my nodes are completely fresh, only an empty "objects" dir in their mounted locations | 04:16 |
mattoliverau | doxavore: have you got memcache running? | 04:29 |
doxavore | yes, i did a flush_all on all of them and verified they were empty | 04:30 |
mattoliverau | doxavore: tempauth stores the token in memcache when you authenticate, usually memcache on the proxy. Is /d0/231526/AUTH_test the storage url returned then you authenticate? | 04:32 |
mattoliverau | *when | 04:32 |
doxavore | it's returning http://127.0.0.1:8080/v1/AUTH_services as the X-Storage-Url - the swift client 2.3.1 is calling the /d0 location when I issue a `swift stat` | 04:36 |
doxavore | even when I attempt to use a testing user that I've never used before, the stat is trying to load /d0/####/AUTH_account and getting a 400 - on the file system /srv/node/d0/objects is empty | 04:37 |
*** nellysmitt has joined #openstack-swift | 04:40 | |
mattoliverau | doxavore: well if you want to issue a HEAD to the proxy while authenticated, you are suppose to use http://127.0.0.1:8080/v1/AUTH_services as the url though the proxy to access your account. I guess the question is what are you trying to do? A normal user wont have access to HEAD a storage node directly, which is when the drive path may come into play. | 04:42 |
*** nellysmitt has quit IRC | 04:45 | |
doxavore | mattoliverau: so I have ST_AUTH, ST_USER, and ST_KEY all set in my env. When I run `swift stat` I get an "account not found error". here's what I'm trying to do and the log output associated with it: https://gist.github.com/doxavore/c93ca0792629c5082cd7 | 04:47 |
doxavore | that's where I'm getting the /d0 from | 04:50 |
doxavore | that is, i'm not trying to hit it directly, i'm just trying to use the cli client | 04:51 |
mattoliverau | doxavore: thats a call from the object server, so you can't use that.. if you use test:tester does it work? | 04:53 |
doxavore | no, i get the same errors | 04:55 |
doxavore | if i'm only using tempauth, at what point does it create the account DB? it almost looks like it's trying to access one that just isn't on the filesystem (because, nothing is on the file system...) | 04:56 |
mattoliverau | doxavore: in your proxy-server.conf, is it the default? (i.e unchanged) Cause the tester3 user, cause if that is the case the tester3 user isn't member of the .admin or .resller_admin groups, so they only have access if someone in the .admin group gives them access to a container via ACLs. | 05:02 |
mattoliverau | doxavore: https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L208-257 | 05:03 |
*** dmsimard is now known as dmsimard_away | 05:03 | |
mattoliverau | doxavore: actually https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L239-257 is more specific | 05:04 |
doxavore | mattoliverau: yes it's the default tempauth config. when i use the admin:admin/admin login which is still a .reseller_admin I get the same exact error. It's just not creating any accounts, with allow_account_management and account_autocreate both enabled | 05:04 |
mattoliverau | doxavore: you will have the account_autocreate = True in proxy-server.conf, so an account will be auto created when you start using it. | 05:06 |
doxavore | under app:proxy-server I have both "allow_account_management = true" and "account_autocreate = true" | 05:07 |
mattoliverau | doxavore: https://gist.github.com/doxavore/c93ca0792629c5082cd7 | 05:08 |
mattoliverau | once you export the admin details, what happens when you swift stat? | 05:09 |
doxavore | the same thing - account not found, syslog shows it giving a 400 on a different /d0 address | 05:10 |
doxavore | for the tester, tester3, and admin users in their respective accounts | 05:10 |
*** jrichli has quit IRC | 05:11 | |
mattoliverau | doxavore: is this on a SAIO? | 05:13 |
doxavore | it's running all the services, but installed 2.2.2 via .deb packages and our same puppet config that was working on 2.2.0 | 05:14 |
*** devlaps has quit IRC | 05:16 | |
doxavore | so, it's AIO but I'm not sure if "SAIO" is specifically the dev build. I don't mean to be vague :) | 05:16 |
mattoliverau | doxavore: lol, fair enough, what's the output of the curl command (comment in your gist)? is it authenticating? | 05:17 |
mattoliverau | doxavore: if you want to show the output then paste.openstack.org or gist it, don't dump in channel (which you've been doing, thanks) | 05:18 |
doxavore | mattoliverau: certainly. added to existing gist | 05:19 |
mattoliverau | doxavore: lol, sorry it's support to be curl not url (copy and paste error) | 05:19 |
mattoliverau | doxavore: cool so it authenticates, next run the following curl command.. see gist once I've actaully pasted it :P | 05:20 |
*** ppai has joined #openstack-swift | 05:22 | |
mattoliverau | doxavore: pasted, it should be one line and you should recieve a 204 | 05:23 |
doxavore | mattoliverau: updated, gives me a 400 without a body | 05:23 |
mattoliverau | NOTE: I used the authtoken you recieved | 05:23 |
mattoliverau | doxavore: hmm, I would have expected to get a 401 if we were just loosing the token (ie. a memcache issue). What does your logs say this time? Same thing? | 05:26 |
doxavore | mattoliverau: also just dropped in the log entry. So the proxy seems to be translating it from what we requested to HEAD /d0/231526/AUTH_test - which I imagine is trying to find the account DB somewhere on the file system to give container stats, and that doesn't work... | 05:26 |
doxavore | on the filesystem, there's a /srv/node/d0/objects directory but that's _it_ | 05:27 |
notmyname | dmorita: it was no bother for you to ask me about slogging. the unit tests were more for my comfort. and actually I think they're better overall now | 05:32 |
notmyname | dmorita: and yes. json output is much nicer :-) | 05:32 |
mattoliverau | doxavore: the /d0/<part>/<account> is normal.. this is a request to the account server which will be translated to disk. The question is why in your gist is it pointing to the object server? or is your "object" server your account server? | 05:34 |
mattoliverau | doxavore: because otherwise this may be why your receiving a 400. | 05:35 |
doxavore | mattoliverau: they're on the same node... | 05:35 |
doxavore | ahhhhhhh | 05:36 |
mattoliverau | doxavore: but it looks like it is trying to issue a command to the object server not the account server part | 05:36 |
doxavore | that's it. I was already sleep-deprived when I created these rings, and I threw everything to the same port | 05:36 |
mattoliverau | doxavore: lol, that would do it :) | 05:37 |
doxavore | fortunately, now I know what that looks like... :-/ | 05:37 |
doxavore | mattoliverau: thank you much for your help, that just wasn't clicking for me. | 05:38 |
mattoliverau | doxavore: anytime, I'm glad we got there in the end :), I should've just noticed the object-server in the logs you posted.. now I'll know what to look at next time also ;) | 05:39 |
*** pcaruana has quit IRC | 05:47 | |
*** SkyRocknRoll has joined #openstack-swift | 05:49 | |
*** doxavore has quit IRC | 05:59 | |
*** dmsimard_away has quit IRC | 06:09 | |
*** dmsimard_away has joined #openstack-swift | 06:09 | |
*** dmsimard_away is now known as dmsimard | 06:09 | |
notmyname | and that's what mahatic's patch will solve (or at least detect) with one simple cli command | 06:12 |
mattoliverau | nice, looking forward to it :) | 06:18 |
mattoliverau | notmyname: now shouldn't you be thinking about sleeping? | 06:18 |
notmyname | mattoliverau: I was just up talking about swift tshirt designs :-) | 06:18 |
mattoliverau | notmyname: oh, in that case, no sleep for you until its done :P | 06:19 |
notmyname | :-) | 06:19 |
*** nshaikh has joined #openstack-swift | 06:37 | |
*** nellysmitt has joined #openstack-swift | 06:41 | |
*** nellysmitt has quit IRC | 06:45 | |
*** silor has joined #openstack-swift | 06:49 | |
*** SkyRocknRoll has quit IRC | 07:04 | |
*** SkyRocknRoll has joined #openstack-swift | 07:21 | |
*** panbalag has quit IRC | 07:27 | |
*** panbalag has joined #openstack-swift | 07:46 | |
*** mkerrin has joined #openstack-swift | 07:54 | |
*** kajinamit has joined #openstack-swift | 08:02 | |
*** rledisez has joined #openstack-swift | 08:06 | |
*** fifieldt has joined #openstack-swift | 08:09 | |
*** chlong has quit IRC | 08:11 | |
*** SkyRocknRoll has quit IRC | 08:21 | |
*** nshaikh has quit IRC | 08:27 | |
*** geaaru has joined #openstack-swift | 08:30 | |
*** mmcardle has joined #openstack-swift | 08:36 | |
*** SkyRocknRoll has joined #openstack-swift | 08:39 | |
*** nellysmitt has joined #openstack-swift | 08:42 | |
*** jordanP has joined #openstack-swift | 08:45 | |
*** nellysmitt has quit IRC | 08:46 | |
*** nellysmitt has joined #openstack-swift | 08:47 | |
*** joeljwright has joined #openstack-swift | 08:49 | |
*** SkyRocknRoll has quit IRC | 08:50 | |
*** jistr has joined #openstack-swift | 08:51 | |
*** mkerrin has quit IRC | 08:53 | |
*** fifieldt has quit IRC | 08:56 | |
*** SkyRocknRoll has joined #openstack-swift | 09:02 | |
*** donagh_ has joined #openstack-swift | 09:03 | |
*** kajinamit has quit IRC | 09:08 | |
*** SkyRocknRoll has quit IRC | 09:15 | |
*** aix has quit IRC | 09:21 | |
*** SkyRocknRoll has joined #openstack-swift | 09:32 | |
*** acoles_away is now known as acoles | 09:40 | |
*** aix has joined #openstack-swift | 09:48 | |
*** nshaikh has joined #openstack-swift | 10:00 | |
*** aix has quit IRC | 10:05 | |
*** aix has joined #openstack-swift | 10:17 | |
*** geaaru has quit IRC | 10:20 | |
*** leopoldj has joined #openstack-swift | 10:32 | |
*** geaaru has joined #openstack-swift | 10:34 | |
*** dmorita has quit IRC | 10:38 | |
*** leopoldj has quit IRC | 10:38 | |
openstackgerrit | Alistair Coles proposed openstack/swift: Update guest VM OS recommendation in SAIO doc https://review.openstack.org/156539 | 10:49 |
*** foexle has joined #openstack-swift | 10:57 | |
*** SkyRocknRoll has quit IRC | 10:58 | |
openstackgerrit | Donagh McCabe proposed openstack/swift: Add multiple reseller prefixes and composite tokens https://review.openstack.org/137086 | 11:06 |
acoles | cschwede: hi, you here? | 11:15 |
*** SkyRocknRoll has joined #openstack-swift | 11:15 | |
*** dmsimard has quit IRC | 11:22 | |
*** SkyRocknRoll has quit IRC | 11:26 | |
*** dmsimard_away has joined #openstack-swift | 11:30 | |
*** dmsimard_away is now known as dmsimard | 11:30 | |
*** khivin has quit IRC | 11:33 | |
*** mwilliams_ct has joined #openstack-swift | 11:34 | |
*** mahatic has joined #openstack-swift | 11:42 | |
*** SkyRocknRoll has joined #openstack-swift | 11:44 | |
*** SkyRocknRoll has quit IRC | 11:55 | |
*** ho has quit IRC | 11:58 | |
*** SkyRocknRoll has joined #openstack-swift | 12:12 | |
*** SkyRocknRoll has quit IRC | 12:30 | |
*** arnaud_o has quit IRC | 12:44 | |
*** arnaud_o has joined #openstack-swift | 12:45 | |
*** SkyRocknRoll has joined #openstack-swift | 12:47 | |
*** ppai has quit IRC | 13:19 | |
*** HenryG has quit IRC | 13:49 | |
*** SkyRocknRoll has quit IRC | 13:51 | |
*** HenryG has joined #openstack-swift | 13:52 | |
*** tdasilva has joined #openstack-swift | 14:02 | |
*** mwilliams_ct has left #openstack-swift | 14:02 | |
*** Guest90435 has joined #openstack-swift | 14:02 | |
*** Guest90435 is now known as annegentle | 14:02 | |
*** jkugel has joined #openstack-swift | 14:06 | |
*** nshaikh has quit IRC | 14:06 | |
*** donagh_ has quit IRC | 14:33 | |
*** donagh_ has joined #openstack-swift | 14:33 | |
*** doxavore has joined #openstack-swift | 14:33 | |
openstackgerrit | Joel Wright proposed openstack/python-swiftclient: Fix crash when stat'ing objects with non-ascii names https://review.openstack.org/147846 | 14:48 |
*** badgas has joined #openstack-swift | 14:49 | |
openstackgerrit | Joel Wright proposed openstack/python-swiftclient: Fix crash when stat'ing objects with non-ascii names https://review.openstack.org/147846 | 14:52 |
*** SkyRocknRoll has joined #openstack-swift | 15:06 | |
*** geaaru has quit IRC | 15:23 | |
*** geaaru has joined #openstack-swift | 15:36 | |
*** daddyjoseph97 has joined #openstack-swift | 15:39 | |
*** SkyRocknRoll has quit IRC | 15:44 | |
*** doxavore has quit IRC | 15:45 | |
*** daddyjoseph97 has quit IRC | 15:45 | |
*** jrichli has joined #openstack-swift | 15:49 | |
*** khivin has joined #openstack-swift | 15:50 | |
*** dmsimard is now known as dmsimard_away | 15:54 | |
openstackgerrit | Donagh McCabe proposed openstack/swift: Support HTTP_X_SERVICE_IDENTITY_STATUS in keystoneauth https://review.openstack.org/156634 | 16:00 |
cschwede | acoles: yes, i’m here :) | 16:05 |
*** doxavore has joined #openstack-swift | 16:06 | |
*** reed has joined #openstack-swift | 16:11 | |
*** sgotliv has joined #openstack-swift | 16:14 | |
doxavore | when write_affinity is enabled, how does it select which devices it writes to? the write_affinity_node_count suggests it has nothing to do with the ring-specified handoff node, so how would the replicator find it to sync/remove? | 16:20 |
acoles | cschwede: hi. hope you had a good journey home. i had some more thoughts about undelete... | 16:21 |
cschwede | acoles: thanks, yes, trip was quite relaxed. and you? had some ideas about undelete on the way home? | 16:22 |
cschwede | i’m curious :) | 16:22 |
acoles | yes, good flight thanks. so this is based on the discussion in sfo around using a manifest like object to point to versions of an object | 16:23 |
acoles | couple of things: (a) it *might* be that the service token deature is useful, or can be adapted. I guess we would like to 'hide' the container with the object versions so the user only sees a single container listed (the one with the manifests). | 16:24 |
acoles | we could hide the version container in another reseller prefix namespace | 16:24 |
cschwede | yep, maybe even use a container in a different account | 16:24 |
cschwede | or a different prefix, yes | 16:24 |
acoles | which service tokens does. and maybe use service token auth to manage access to the version container/namespace | 16:25 |
cschwede | and manifests could be very similar to SLO manifests, or not? | 16:25 |
acoles | e.g. an undelete middleware (was that the idea?) would have a service token to be able to auth operation son the versions, which the user alone could not do | 16:26 |
cschwede | my first idea was a middleware, but would probably require a change in the object server too | 16:26 |
acoles | (would also mean that an admin/third party could create/delete versions via API if they ha daccess to a service token and the user token) | 16:27 |
serverascode | hi, hw question: is anyone using the supermicro 72 bay chassis? good idea bad idea? ( eg. http://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1R72L.cfm ) | 16:28 |
cschwede | acoles: well, at the end there is always someone who could delete stuff - but maybe only a server admin | 16:28 |
*** abhirc has quit IRC | 16:28 | |
cschwede | acoles: but if we could provide a solution where a user with a „normal“ account+token can store data without the possibility to delete it, that would be a big improvement | 16:29 |
acoles | cschwede: yes, the point being that with auth+service token an admin can perform manual deletes via API and not need a backdoor to cleanup. re-using service token might require some tweaks but there seemed to be some overlap in terms of restricting user operations and hiding object in another prefix | 16:30 |
cschwede | one question remains if we use manifests: if someone deletes the manifest (meaning the object should be deleted later on), the „real“ object needs an meta-data update too. to mark it for later deletion | 16:30 |
cschwede | acoles: i will have a closer look to the proposed service tokens and we could reuse them, maybe we can combine our efforts | 16:30 |
acoles | cschwede: yes, i think we said we might need fast-post to be able to update the real object x-delete-at ?? | 16:31 |
acoles | (well, post as copy would work too) | 16:31 |
cschwede | yes, that would simplifiy this a lot, because then we could avoid i/o-heavy rewrites | 16:31 |
acoles | cschwede: so, (b) second thought, re the manifests... | 16:31 |
acoles | cschwede: i think we said that a user delete would be transformed to a PUT manifest with a 'magic' content-type that allowed those 'deleted' objects t be filtered out from the container listing. | 16:33 |
acoles | s/t/to/ | 16:33 |
cschwede | if we keep that manifest, yes | 16:33 |
acoles | cschwede: i *think* that was so that we never lost track of the real object - a privileged container GET *would* include the 'deleted' manifests | 16:33 |
acoles | cschwede: ^^ is that correct ?? | 16:34 |
cschwede | yes, maybe using a query parameter like GET http://saio/v1/AUTH_test/container/?show_deleted or something similar | 16:34 |
acoles | ok, so the problem i see is that the manifest timestamp must always move forwards in time, otherwise the ocntainer server will ignore an update, so its hard to perform an undelete (i.e. PUT manifest with re-instated pointer) without the manifest timestamp being newer than the last delete, and therefore the listing timestamp being newer than the 'undeleted' real object | 16:36 |
acoles | cschwede: but i wondered if we can make use of two vector timestamps to help | 16:36 |
cschwede | from fast-post? | 16:37 |
acoles | it is perhaps similar to the container reconciler scenario, where an object needs to be PUT/DELETED/PUT without changing 'user-visible' time | 16:37 |
acoles | cschwede: no, the two-vector from reconciler/policy feature | 16:38 |
*** sgotliv has quit IRC | 16:38 | |
acoles | fast-post has a triple-timestamp ;) | 16:38 |
*** Nadeem has joined #openstack-swift | 16:38 | |
cschwede | ah yes, one more :) | 16:39 |
acoles | cschwede: i don't know for sure, but the thought was that when 'deleting' the manifest, we increment its timestamp offset part (but not the 'normal' timestamp) which is enough for container to create a new row | 16:39 |
notmyname | good morning, world | 16:39 |
* acoles also notes this has to work for container sync too | 16:39 | |
acoles | notmyname: morning, were you there for the undelete discussion? | 16:40 |
notmyname | serverascode: nice box. I'll ask around | 16:40 |
notmyname | acoles: in sfo or in scrollback just now? | 16:40 |
acoles | notmyname: sfo, but both | 16:40 |
serverascode | notmyname: cool thanks any info would be helpful :) | 16:40 |
notmyname | just checking scrollback now | 16:40 |
notmyname | serverascode: yes, in general, it should work fine. I've heard of other deployments that have similar (or greater!) density. you'll have to pay attention to capacity adjustments and CPU load with that many drives | 16:42 |
acoles | cschwede: ok, that was a bit of a brain dump from me. meant to be helpful though :) | 16:42 |
cschwede | acoles: thanks a lot, i will include your ideas into the next spec patchset :) | 16:42 |
acoles | cschwede: cool - so are you planning to update the spec? | 16:43 |
cschwede | acoles: yes | 16:43 |
acoles | excellent. well, i can always add comments there | 16:43 |
serverascode | notmyname: thanks! | 16:43 |
notmyname | serverascode: generally you only want boxes that big if you have a big cluster. so that each box isn't a huge percentage of the total capacity | 16:44 |
serverascode | yeah for sure | 16:45 |
*** nellysmitt has quit IRC | 16:47 | |
acoles | cschwede: the final comment, to which i don't have an answer, is how to allow a container to appear deleted, and yet retain all its manifests. | 16:49 |
acoles | and then be able to undelete the container | 16:49 |
acoles | (in the case when all the manifests are in 'deleted' state so the container is apparently empty) | 16:50 |
btorch | serverascode: what will be your head unit ? | 16:50 |
cschwede | acoles: the container with the „real“ objects could use the same container name but with a suffix, thus it can be undeleted and listed with the service token | 16:51 |
serverascode | btorch: head unit? proxy? | 16:51 |
btorch | serverascode: the node attached to the supermicro jbod unit | 16:51 |
btorch | serverascode: the server type/hardware that will have the supermicro attached | 16:52 |
serverascode | it's a server itself with 72 bays | 16:52 |
serverascode | unless I linked to the wrong page | 16:52 |
notmyname | btorch: looks like that box includes some CPU in the box | 16:52 |
btorch | serverascode: sorry I miss read it :) | 16:52 |
acoles | cschwede: yes. or if the real object container is in a different prefix it can just have the same name. | 16:53 |
btorch | serverascode: not sure how many storage nodes you will use but I would consider that first | 16:55 |
notmyname | btorch: what's up? haven't seen you around in a while! | 16:55 |
acoles | cschwede: in fact, i need to be reminded why we need to keep a 'deleted' manifest? was it to avoid dark data? seems like we could always find any 'orphaned' real objects by listing the real object container (assuming its name is a transform of the original) | 16:56 |
notmyname | acoles: cschwede: re undelete. looks like you've got it well in hand :-) | 16:56 |
btorch | serverascode: if it's several nodes spread around 5+ zones I would go with something like that but it's just a few nodes I would think twice, obviously you don't have to fill all the bays and start it slow and then backfill | 16:56 |
acoles | notmyname: yeah, right ;) | 16:56 |
btorch | notmyname: not much, I'm around just don't talk much in here | 16:56 |
notmyname | btorch: are you still doing swift things? last I heard you were doing rax private clouds or something | 16:57 |
btorch | yeah swift | 16:57 |
notmyname | cool | 16:57 |
serverascode | btorch: yeah will keep that in mind, they keep talking about 2PB usable | 16:57 |
cschwede | acoles: yes, that makes sense, we only need to take care of the name length constraint | 16:58 |
*** cca has joined #openstack-swift | 17:00 | |
*** mahatic has quit IRC | 17:04 | |
notmyname | acoles: cschwede: can you comment on https://bugs.launchpad.net/swift/+bug/1422237 before you leave today? | 17:05 |
openstack | notmyname: Error: malone bug 1422237 not found | 17:05 |
notmyname | openstack: thanks. you aren't allowed allowed to see that | 17:05 |
acoles | notmyname: ok | 17:05 |
*** daddyjoseph97 has joined #openstack-swift | 17:10 | |
notmyname | anyone have any experience with mechanical keyboards? specifically cherry mx switches (brown vs clear). I'm looking at http://www.wasdkeyboards.com/index.php/products/code-keyboard/code-87-key-mechanical-keyboard.html | 17:10 |
notmyname | but I've never used one | 17:11 |
btorch | serverascode: ah one thing another swift ops just pointed out, check with supermicro if they provide 10g instead of the 1g nics | 17:12 |
serverascode | btorch: yeah for sure good point | 17:13 |
*** dmsimard_away is now known as dmsimard | 17:19 | |
*** daddyjoseph97 has quit IRC | 17:22 | |
*** jistr has quit IRC | 17:24 | |
cschwede | notmyname: will comment on that later, in about 2 hours | 17:27 |
notmyname | cschwede: ok, thanks | 17:27 |
*** annegent_ has joined #openstack-swift | 17:29 | |
*** rledisez has quit IRC | 17:30 | |
openstackgerrit | Daniel Wakefield proposed openstack/python-swiftclient: Add flag to disable automatic checksum validation. https://review.openstack.org/156677 | 17:36 |
*** miqui has quit IRC | 17:37 | |
hurricanerix_ | Here is Hisashi Osanai's photo from dinner the first night of the hackathon for anybody that wants a copy (I did a little color correction) | 17:38 |
hurricanerix_ | http://blog.hurricanerix.me/images/_DSC4967.png | 17:38 |
hurricanerix_ | notmyname: I am taking time off till the 25th, so if anybody is really anxious about getting https://review.openstack.org/#/c/155985/ merged in, feel free to let them take it over. Otherwise I will work on it next week when I am back. | 17:39 |
notmyname | hurricanerix_: thanks | 17:40 |
hurricanerix_ | np | 17:40 |
notmyname | hurricanerix_: can you leave a comment in gerrit about that? | 17:40 |
hurricanerix_ | notmyname: oh sure. | 17:40 |
notmyname | thanks | 17:41 |
hurricanerix_ | done | 17:42 |
tdasilva | hurricanerix_: enjoy your time off :) | 17:44 |
hurricanerix_ | tdasilva: Thanks, I will. =) | 17:45 |
jrichli | hurricanerix_: thanks for sharing the photo! | 17:46 |
hurricanerix_ | jrichli: np | 17:46 |
notmyname | FYI, the _infra team has just proposed a change to the swift functional tests job to use "tox -e func" instead of ".functests" (ie nose). the main change is that tox sets up a venv with our requirements file and nose just uses whatever is on the system | 17:47 |
notmyname | If you have comments or concerns, please leave them on https://review.openstack.org/#/c/156676/ | 17:47 |
openstackgerrit | Alistair Coles proposed openstack/swift-specs: Updates to encryption spec https://review.openstack.org/154318 | 17:50 |
acoles | jrichli: torgomatic ^^ still work in progress but comments/corrections welcome | 17:51 |
jrichli | thanks, I will take a look | 17:51 |
openstackgerrit | Alistair Coles proposed openstack/swift-specs: Updates to encryption spec https://review.openstack.org/154318 | 17:52 |
*** gyee has joined #openstack-swift | 17:52 | |
*** doxavore has quit IRC | 17:53 | |
*** acorwin_ is now known as acorwin | 17:58 | |
*** abhirc has joined #openstack-swift | 18:05 | |
*** nellysmitt has joined #openstack-swift | 18:06 | |
*** doxavore has joined #openstack-swift | 18:11 | |
*** mmcardle has quit IRC | 18:11 | |
doxavore | notmyname: i use a CODE keyboard with cherry MX clear switches and it's beautiful. I'm not sure my office mates care for it as much :> | 18:13 |
notmyname | doxavore: I'm initially buying it for home. :-) | 18:13 |
notmyname | doxavore: but yeah, it looks pretty good. but I haven't actually used those switches before, so it's hard to say if I like it. I'd like to try it out before dropping 150 on it :-) | 18:14 |
*** annegent_ has quit IRC | 18:14 | |
notmyname | doxavore: I'm glad to hear you like it though | 18:14 |
notmyname | from the specs, it seems the clear switches have the same activation pressure as the keyboard I'm currently using at work (an apple chicklet wireless--don't laugh, I like it). so I think it should feel fine. but it's a lot different | 18:15 |
*** jbonjean has quit IRC | 18:16 | |
*** jbonjean has joined #openstack-swift | 18:17 | |
cschwede | notmyname: i used mechanical keyboards in the past, nowadays i prefer the keyboards from apple… you might look at http://www.daskeyboard.com/ as well if you’re looking into buying a mechanical keyboard | 18:19 |
notmyname | cschwede: ya, I've been comparing das keyboard (the das keyboard 4C) against http://www.wasdkeyboards.com. The CODE keyboard from WASD looks pretty interesting. if for nothing else, I like the backlighting :-) | 18:19 |
notmyname | http://www.wasdkeyboards.com/index.php/products/code-keyboard/code-87-key-mechanical-keyboard.html | 18:20 |
notmyname | oops. I already linked that :-) | 18:20 |
doxavore | I switch between it and the macbook's built-in keyboard pretty seamlessly without any grief. I just feel more accurate with that tactile click. I've had other WASD keyboards and can't recommend them enough. | 18:20 |
*** jbonjean has quit IRC | 18:21 | |
notmyname | I wish there were a place where I could go try out a few. maybe there is but I just don't even know what to search for | 18:21 |
*** sgotliv has joined #openstack-swift | 18:21 | |
cschwede | i remember typing on an IBM M keyboard - they were actually built like a tank. that was some tactile feedback! | 18:22 |
notmyname | ya | 18:22 |
notmyname | jeblair says he has a stack of model Ms at his house. | 18:23 |
notmyname | so let's see what the magic of twitter can do ;-) https://twitter.com/notmyname/status/567751032789139456 | 18:23 |
cschwede | notmyname: hmm, i would imagine that there is a market in SF for a store selling „extraordinary“ IT stuff? | 18:23 |
notmyname | no kidding | 18:23 |
notmyname | and staffed by people with waxed moustaches. | 18:24 |
notmyname | ;-) | 18:24 |
doxavore | is anyone here familiar with how devices are selected for the write_affinity_node_count? i'm having trouble tracking it down in any documentation and just want a slightly better understanding of how we should expect it to behave under failure conditions... | 18:26 |
notmyname | yes | 18:26 |
*** jbonjean has joined #openstack-swift | 18:27 | |
notmyname | doxavore: let's start with the way the default works (there's 3 ways affinity works) | 18:27 |
notmyname | the default is "shuffle" | 18:27 |
notmyname | so when swift makes a call to ring.get_nodes, it gets back the primary locations | 18:27 |
notmyname | (thinking of the GET path first) | 18:27 |
notmyname | so with shuffle, it calles random.shuffle() on the returned primary node list | 18:28 |
notmyname | so it's a simple load balancer (with no feedback) | 18:28 |
notmyname | the next affinity method is "timing" | 18:28 |
notmyname | with that mode, every time the proxy connects to a node, it tracks how long it took to connect (ie create the socket) | 18:29 |
notmyname | then it chooses the fastest one out of the primary node list | 18:29 |
notmyname | so that balances, but as a node gets busy it will get slower | 18:29 |
notmyname | and "farther away" nodes will not be selected | 18:29 |
notmyname | the last affinity mode is explicit. in that way the proxy is configured to know what its own region and zone is | 18:30 |
notmyname | and it sorts the returned primary nodes by finding one that better matches. eg same zone matches before same region | 18:30 |
notmyname | and you can explicitly configure a proxy to prefer a specific region or zone | 18:31 |
notmyname | so that's the read path. make saense? | 18:31 |
notmyname | sense even | 18:31 |
doxavore | yes | 18:32 |
notmyname | ok good :-) | 18:32 |
notmyname | so moving on to the write path | 18:32 |
notmyname | since a write has more than one node, it's similar but slightly different | 18:32 |
notmyname | eg shuffle affinity wouldn't do anything since it would still be sent to all primary nodes | 18:33 |
notmyname | so let's move to the write affinity | 18:33 |
notmyname | err, the explicit write affinity | 18:33 |
notmyname | you can set it to explicitly prefer a given region | 18:33 |
notmyname | so eg you'd configure proxyA in region1 to prefer region 1. and proxyB in region2 to prefer region 2 | 18:34 |
notmyname | what happens is that it will attempt to find nodes in that region to write to before it chooses nodes in a different region | 18:34 |
notmyname | importantly, THESE ARE NOT THE PRIMARY NODES | 18:34 |
*** cebruns_ is now known as cebruns | 18:34 | |
notmyname | because the primary nodes will be spread throughout the cluster | 18:35 |
notmyname | so one is probably the primary. but maybe not | 18:35 |
notmyname | and so the write_affinity_node_count is how many nodes it checks in the identified region before it puts something into a different region | 18:35 |
*** jordanP has quit IRC | 18:35 | |
notmyname | the default is 2*replica count (ie 6 for 3 replicas) | 18:35 |
klrmn | just in case anybody is wondering, today i am investigating why my tests think they have set up a swift node / gateway share but the cluster deploy page thinks it hasn't got them. i suppose there is an outside chance that this is related to changes made for the large cluster template scenario | 18:36 |
notmyname | so if you have proxyA, it will try to write to 6 drives in region 1 before it even attempts to find nodes in region2. regardless of what the primaries are. | 18:36 |
notmyname | so one of the region 1 nodes is probably one of the primaries | 18:36 |
notmyname | but the other two will be written to handoff nodes (handoffs == anything not a primary) | 18:37 |
klrmn | it helps if i do not send my messages to the wrong channel tho | 18:37 |
notmyname | and replication will eventually move it over to the right primary location | 18:37 |
notmyname | klrmn: :-) | 18:37 |
notmyname | doxavore: make sense? | 18:37 |
doxavore | yes, i follow that. | 18:37 |
*** mmcardle has joined #openstack-swift | 18:38 | |
doxavore | i'm familiar with the single handoff node in the ring, but how are those handoffS selected? | 18:38 |
notmyname | doxavore: the biggest thing to consider is that write affinity is only possible if you have bursty writes and/or you have a region-to-region network that's faster than your data ingest | 18:38 |
*** devlaps has joined #openstack-swift | 18:40 | |
notmyname | doxavore: handoffs are selected deterministically by walking through the ring. | 18:42 |
*** sgotliv has quit IRC | 18:42 | |
notmyname | hang on. I'll find the link | 18:42 |
doxavore | i guess it's the replicator that handles moving the files from the handoffs to the primary, and it makes sense when writing to the single handoff node listed in the ring, but when we're talking some number of handoffs not listed in the ring... | 18:42 |
notmyname | doxavore: https://github.com/openstack/swift/blob/master/swift/common/ring/ring.py#L310 | 18:43 |
notmyname | no,no. everything is listed in the right | 18:43 |
notmyname | *ring | 18:43 |
notmyname | it's the difference in get_nodes() and get_more_nodes() | 18:44 |
notmyname | doxavore: so with https://github.com/openstack/swift/blob/master/swift/common/ring/ring.py#L337 it walks through the ring looking for other regions. in big rings, it will walk through every 65536 partitions. for small rings it will look at every other partition | 18:45 |
*** jasondotstar has joined #openstack-swift | 18:45 | |
notmyname | doxavore: look at the 2 line comments in the for loops below that link | 18:45 |
doxavore | ok, so that's quite clear | 18:46 |
notmyname | doxavore: to summarize in english: walk through the ring and find stuff that's widely spaced (ish) until you run out of places to look | 18:47 |
*** geaaru has quit IRC | 18:49 | |
doxavore | and since we tie the proxy to a region, it will try to put all of those copies in the same region... is there a way to monitor how many of these files aren't in their primaries yet? something like the dispersion report, but ... for this. :) | 18:49 |
*** devlaps has quit IRC | 18:49 | |
*** acoles is now known as acoles_away | 18:50 | |
*** pjnaik1990_ has joined #openstack-swift | 18:51 | |
notmyname | doxavore: what you'd monitor is the replication cycle time | 18:51 |
pjnaik1990_ | Hello!!I am new to Openstack. I am interested to participate in OPW program to contribute to swift..I am looking for mentors who can help me and provide me with a projectidea. | 18:53 |
notmyname | pjnaik1990_: did you send me an email this morning? | 18:54 |
*** mmcardle has quit IRC | 18:54 | |
doxavore | notmyname: that was all very helpful. thanks for taking the time. i have to same i'm pretty impressed with how Swift is put together so far. | 18:54 |
notmyname | doxavore: cool. glad it helped. and thanks | 18:55 |
pjnaik1990_ | @notmyname:Hi!! yes | 18:55 |
notmyname | pjnaik1990_: I haven't had a chance to respond yet. I'll try to get to that later today | 18:56 |
notmyname | pjnaik1990_: but in general, yes. I'd be happy to give you some info. | 18:56 |
pjnaik1990_ | Thankyou!! | 18:56 |
notmyname | pjnaik1990_: but until i get the chance for email, you should find mahatic when she's online. she's currently doing and OPW for a project in swift and might be able to answer any questions you have | 18:56 |
notmyname | but for now, I've got to go to a meeting..... | 18:57 |
pjnaik1990_ | okay I will talk to her.Thanks | 18:57 |
*** mmcardle has joined #openstack-swift | 19:14 | |
*** annegent_ has joined #openstack-swift | 19:15 | |
*** abhirc has quit IRC | 19:17 | |
*** openstackgerrit has quit IRC | 19:20 | |
*** silor1 has joined #openstack-swift | 19:20 | |
*** openstackgerrit has joined #openstack-swift | 19:20 | |
*** silor has quit IRC | 19:20 | |
*** annegent_ has quit IRC | 19:20 | |
*** abhirc has joined #openstack-swift | 19:21 | |
openstackgerrit | Sean Dague proposed openstack/python-swiftclient: add functional tox target https://review.openstack.org/156719 | 19:25 |
*** silor1 has quit IRC | 19:29 | |
*** lcurtis has joined #openstack-swift | 19:30 | |
*** mahatic_ has joined #openstack-swift | 19:33 | |
mahatic_ | notmyname, hello! I logged back in to remind you if you could give any inputs for this -> https://review.openstack.org/#/c/153617/ | 19:35 |
mahatic_ | tata, off to sleep, (on time tonight :P) | 19:35 |
*** pjnaik1990_ has quit IRC | 19:35 | |
*** mahatic_ has quit IRC | 19:36 | |
*** pjnaik1990 has joined #openstack-swift | 19:39 | |
*** annegent_ has joined #openstack-swift | 19:40 | |
lcurtis | Hello all...wondering if there is any way of replacing sqlit db in swift with something else? | 19:42 |
lcurtis | sqlite | 19:42 |
*** mmcardle has quit IRC | 19:42 | |
lcurtis | i am concerned about performance overall if we hit 100million+ objects and more | 19:43 |
lcurtis | i have seen some solutions describe using ssd's | 19:43 |
lcurtis | to counteract | 19:43 |
*** zhill has joined #openstack-swift | 19:45 | |
tdasilva | lcurtis: hi, container performance is a current concern. Last week there were some discussion on this topic and there was a spec written about a proposed solution: https://review.openstack.org/#/c/139921/ | 19:47 |
lcurtis | thanks tdasilva | 19:48 |
lcurtis | is spreading objects over multiple containers most viable current solution? | 19:49 |
clayg | lol @ torgomatic's comment on https://review.openstack.org/#/c/156095/ | 19:54 |
clayg | torgomatic: times, they are a changin' | 19:55 |
torgomatic | hehe | 19:55 |
tdasilva | lcurtis: the recommendations I have seen proposed is to use SSDs and to try to limit the size of containers (by spreading objects over multiples containers) | 20:00 |
lcurtis | tdasilva thanks much | 20:01 |
clayg | tdasilva: did you ever dust off the put/close-connection patch? | 20:03 |
tdasilva | yes, I have it here, was finishing the obj. ver. middleware first and will post both | 20:04 |
tdasilva | clayg: just a reminder that unit tests are not passing yet | 20:05 |
clayg | tdasilva: nice | 20:06 |
*** nellysmitt has quit IRC | 20:09 | |
*** nellysmitt has joined #openstack-swift | 20:12 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 20:17 |
*** annegent_ has quit IRC | 20:18 | |
*** occupant has quit IRC | 20:18 | |
*** annegent_ has joined #openstack-swift | 20:41 | |
clayg | acoles_away: does anyknow else in channel have some experience with devstack lately? stack.sh is bailing out on me with an error from keystone creating the test swift users | 20:43 |
*** annegent_ has quit IRC | 20:53 | |
*** rdaly2 has joined #openstack-swift | 21:01 | |
*** annegent_ has joined #openstack-swift | 21:17 | |
*** aix has quit IRC | 21:20 | |
mattoliverau | Morning | 21:22 |
jrichli | morning | 21:24 |
openstackgerrit | John Dickinson proposed openstack/swift: update the getting started doc https://review.openstack.org/156388 | 21:24 |
*** nellysmitt has quit IRC | 21:25 | |
mattoliverau | jrichli: morning :) | 21:25 |
*** doxavore has quit IRC | 21:27 | |
*** annegent_ has quit IRC | 21:29 | |
notmyname | interesting mailing list post just now. "On Object placement" | 21:29 |
*** doxavore has joined #openstack-swift | 21:33 | |
mattoliverau | notmyname: hmm, yeah, I think he might have come into channel a few weeks ago to talk about it. Answered his questions on how the ring knew where to place and SP might be a solution in the way swift currently works. | 21:33 |
notmyname | wow. https://www.openstack.org/vote-vancouver/Presentation/faith-the-secret-ingredient-of-a-successful-system-integration | 21:44 |
torgomatic | notmyname: that's... yeah, "wow" is probably the best way to say it | 21:46 |
clayg | https://www.youtube.com/watch?v=lu3VTngm1F0 | 21:47 |
doxavore | this is old, but has the state of backup not changed? https://answers.launchpad.net/swift/+question/149943 I'm not sure a container-to-container sync would be appropriate if you don't ever want the receiving end to delete? (and for us, objects are immutable, so not too concerned with an update) | 21:47 |
openstackgerrit | Thiago da Silva proposed openstack/swift: versioned writes middleware https://review.openstack.org/134347 | 21:53 |
doxavore | i know that's likely an ugly topic, but I have to imagine the bigger Swift players aren't relying only on online backups of connected regions :> | 21:54 |
*** gyee has quit IRC | 21:55 | |
notmyname | doxavore: it depends on what you mean | 22:08 |
notmyname | doxavore: at swiftstack we've got many customers using multiple regions to keep offsite copies for DR | 22:08 |
doxavore | notmyname: to be more specific, what happens if someone issues a DELETE on an object in the main region? | 22:09 |
notmyname | deleted everywhere. it's more of DR (hot failover) rather than backups | 22:09 |
doxavore | say, they have a legal requirement that all the data in that account not be deleted (so long as some setting is set, etc) | 22:09 |
torgomatic | that is a very hard problem to solve | 22:10 |
torgomatic | both DELETE and PUT can destroy data, the latter by overwriting | 22:10 |
torgomatic | so one might think "block PUT if the object exists" | 22:10 |
torgomatic | but in a distributed, eventually-consistent system, you can't answer "does the object exist?" | 22:11 |
torgomatic | at best, you get a semipredicate: you can find it and decide "yes, it exists", or you can fail to find it and decide "well, maybe" | 22:11 |
torgomatic | for example, a copy might exist on some random node in the system that's not a primary node for that partition, or on any of the first 100 handoffs, because it hit a handoff and then a ring rebalance happened | 22:12 |
torgomatic | or a copy might be on a disk on a node that's down | 22:12 |
notmyname | doxavore: interesting enough, we're actually working with a customer right now who needs the "prevent overwrite or delete" feature. but they already have a sclable service with a good api that exists that tracks the stuff that is "locked". so it's kinda easy there because there's already an existing, consistent locking service | 22:13 |
torgomatic | so if you allow the PUT when you don't find the object, you run the risk of flattening the object anyway | 22:13 |
torgomatic | notmyname: yeah, it's easy when you have an external CP-type system to track this stuff in :) , but for those of us in AP-land, it's basically impossible | 22:14 |
notmyname | the -infra team needs us to land https://review.openstack.org/#/c/156719/1 asap. it's blocking some of their changes. one of which allows our tests to be run against our own requirements file (not the global requirements file) | 22:15 |
notmyname | torgomatic: +1000 | 22:15 |
*** rdaly2 has quit IRC | 22:17 | |
doxavore | notmyname: torgomatic: is there an easy way to block DELETE on an account, or would one look to just use middleware? in our case, the PUT of a schrodinger's file might be able to qualify as an acceptable risk :> | 22:18 |
torgomatic | doxavore: you'd have to write your own middleware to do it | 22:18 |
torgomatic | block DELETE, plus PUT of existing object | 22:19 |
torgomatic | so, given that you said "easy", probably not ;) | 22:19 |
doxavore | do all the replicators, anything else that could delete files, work through the object-server API to hit that middleware, or would we only be protecting against external threats, and not so much misconfiguration (which could ultimately still exist if middleware exists, but..) | 22:20 |
torgomatic | doxavore: you'd do it in the proxy, not object servers | 22:20 |
torgomatic | if your clients can hit object servers directly, you have no security against misbehavior at all | 22:21 |
eikke | lol @ 'faith' talk link | 22:26 |
torgomatic | eikke: I can't decide whether I should vote for it or not. It'd be like voting for a train wreck: amazing to watch, but better to avoid it in the first place. | 22:29 |
*** panbalag has quit IRC | 22:30 | |
eikke | torgomatic: I'd be in the "Let's not bring such things to our events"-camp | 22:31 |
*** badgas has quit IRC | 22:31 | |
*** gyee has joined #openstack-swift | 22:49 | |
*** jkugel has quit IRC | 22:56 | |
*** foexle has quit IRC | 22:58 | |
openstackgerrit | Clay Gerrard proposed openstack/python-swiftclient: add functional tox target https://review.openstack.org/156719 | 23:00 |
openstackgerrit | Thiago da Silva proposed openstack/swift: WIP: initial pass at refactoring the PUT method https://review.openstack.org/156825 | 23:05 |
tdasilva | clayg: ^^^ my first attempt at adding dependencies in gerrit, messy! | 23:07 |
*** chlong has joined #openstack-swift | 23:08 | |
*** joeljwright has quit IRC | 23:10 | |
*** doxavore has quit IRC | 23:25 | |
*** dmsimard is now known as dmsimard_away | 23:30 | |
*** Nadeem has quit IRC | 23:35 | |
*** jrichli has quit IRC | 23:38 | |
*** jrichli has joined #openstack-swift | 23:38 | |
*** madhuri_ has joined #openstack-swift | 23:43 | |
*** jrichli has quit IRC | 23:50 | |
tdasilva | notmyname: ever been to Zazie in SFO? | 23:51 |
notmyname | tdasilva: nope | 23:51 |
notmyname | what is it? | 23:52 |
tdasilva | restaurant, it was on the news today | 23:52 |
tdasilva | they pay full benefits to their employees | 23:52 |
tdasilva | and I think their website just went down :P | 23:53 |
notmyname | yup. seems that way :-) | 23:53 |
*** abhirc has quit IRC | 23:53 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!