| *** thurloat_isgone is now known as thurloat | 00:11 | |
| *** charz has quit IRC | 00:14 | |
| *** kota_ has quit IRC | 00:15 | |
| *** notmyname has quit IRC | 00:15 | |
| *** charz has joined #openstack-swift | 00:16 | |
| *** notmyname has joined #openstack-swift | 00:17 | |
| *** ChanServ sets mode: +v notmyname | 00:17 | |
| *** garthb__ has quit IRC | 00:18 | |
| *** dmorita has joined #openstack-swift | 00:19 | |
| *** goodygum has quit IRC | 00:19 | |
| openstackgerrit | Emett Speer proposed openstack/swift: [Labs] Update links to Cloud Admin Guide https://review.openstack.org/219933 | 00:21 |
|---|---|---|
| *** goodygum has joined #openstack-swift | 00:21 | |
| *** lyrrad has quit IRC | 00:27 | |
| *** m_kazuhiro has joined #openstack-swift | 00:28 | |
| *** eranrom has joined #openstack-swift | 00:29 | |
| *** AndreiaKumpera has quit IRC | 00:31 | |
| *** konrado has joined #openstack-swift | 00:35 | |
| *** bill_az_ has quit IRC | 00:36 | |
| *** zhill has quit IRC | 00:36 | |
| *** wshao has joined #openstack-swift | 00:37 | |
| *** eranrom has quit IRC | 00:37 | |
| *** wshao has quit IRC | 00:41 | |
| openstackgerrit | Emett Speer proposed openstack/swift: [Labs] Update links to Cloud Admin Guide https://review.openstack.org/219933 | 00:42 |
| *** wshao has joined #openstack-swift | 00:42 | |
| *** wshao has quit IRC | 00:47 | |
| *** nakagawamsa has joined #openstack-swift | 00:51 | |
| *** konrado_ has joined #openstack-swift | 00:57 | |
| *** konrado has quit IRC | 00:59 | |
| *** kei_yama has quit IRC | 01:01 | |
| *** kei_yama has joined #openstack-swift | 01:01 | |
| *** thurloat is now known as thurloat_isgone | 01:09 | |
| *** zul has quit IRC | 01:14 | |
| *** prometheanfire has joined #openstack-swift | 01:14 | |
| prometheanfire | notmyname: http://lab.jerasure.org/jerasure/jerasure/tags http://lab.jerasure.org/jerasure/gf-complete/tags | 01:15 |
| *** zul has joined #openstack-swift | 01:17 | |
| openstackgerrit | Merged openstack/swift: Add test that a tempurl POST cannot set a DLO manifest header https://review.openstack.org/217273 | 01:18 |
| openstackgerrit | Merged openstack/python-swiftclient: Stop Connection class modifying os_options parameter https://review.openstack.org/216240 | 01:18 |
| *** wbhuber has joined #openstack-swift | 01:22 | |
| *** konrado_ has quit IRC | 01:26 | |
| *** kota_ has joined #openstack-swift | 01:56 | |
| *** ChanServ sets mode: +v kota_ | 01:56 | |
| *** jkugel has joined #openstack-swift | 01:57 | |
| *** jkugel1 has joined #openstack-swift | 02:01 | |
| *** mplex has quit IRC | 02:02 | |
| *** jkugel has quit IRC | 02:04 | |
| *** mplex has joined #openstack-swift | 02:10 | |
| *** mplex has joined #openstack-swift | 02:10 | |
| *** elmo has quit IRC | 02:11 | |
| *** elmo has joined #openstack-swift | 02:19 | |
| openstackgerrit | Merged openstack/python-swiftclient: Drop flake8 ignores for already-passing tests https://review.openstack.org/214752 | 02:19 |
| openstackgerrit | Merged openstack/swift: Add unit test for container_update for unmounted device https://review.openstack.org/210610 | 02:22 |
| openstackgerrit | Merged openstack/swift: Test if container_sweep is executed on unmounted devices https://review.openstack.org/219639 | 02:24 |
| *** jkugel1 has quit IRC | 02:27 | |
| *** Kennan has quit IRC | 02:29 | |
| *** Kennan has joined #openstack-swift | 02:30 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift: Fix memcached exception out of range stacktrace https://review.openstack.org/219963 | 02:33 |
| *** haomaiwang has quit IRC | 02:36 | |
| *** pberis has joined #openstack-swift | 02:38 | |
| *** haomaiwang has joined #openstack-swift | 02:38 | |
| *** jrichli has joined #openstack-swift | 02:43 | |
| *** eranrom has joined #openstack-swift | 02:46 | |
| *** eranrom has quit IRC | 02:54 | |
| *** haomaiwang has quit IRC | 02:57 | |
| *** haomaiwa_ has joined #openstack-swift | 03:00 | |
| *** jkugel has joined #openstack-swift | 03:00 | |
| *** bkopilov has quit IRC | 03:05 | |
| *** kei_yama has quit IRC | 03:07 | |
| *** chsc has quit IRC | 03:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements https://review.openstack.org/89250 | 03:10 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 03:11 |
| *** proteusguy_ has quit IRC | 03:13 | |
| *** jkugel has quit IRC | 03:19 | |
| *** jkugel has joined #openstack-swift | 03:21 | |
| *** jkugel has quit IRC | 03:25 | |
| *** proteusguy_ has joined #openstack-swift | 03:30 | |
| *** jroll has quit IRC | 03:30 | |
| *** dimasot has joined #openstack-swift | 03:32 | |
| *** jroll has joined #openstack-swift | 03:33 | |
| openstackgerrit | Merged openstack/swift: go: replicator fix https://review.openstack.org/218984 | 03:38 |
| *** gyee has quit IRC | 03:40 | |
| *** sanchitmalhotra has joined #openstack-swift | 03:43 | |
| mattoliverau | kota_: I'm going through some bugs and left a comment in https://bugs.launchpad.net/swift/+bug/1460920 am I missing something or miss understand the bug? | 03:45 |
| openstack | Launchpad bug 1460920 in OpenStack Object Storage (swift) "Successful PUT object might be missing container update on EC" [High,In progress] - Assigned to Kota Tsuyuzaki (tsuyuzaki-kota) | 03:45 |
| *** wbhuber has quit IRC | 03:46 | |
| kota_ | mattoliverau: reading | 03:50 |
| *** sanchitmalhotra1 has joined #openstack-swift | 03:53 | |
| mattoliverau | kota_: tldr, the number of out_going headers with the container details to update are number of primary nodes in the object ring... not 3, so all requests to store_object _should_ have them. There should be more then only 3 with them.. That's what I thought anyway | 03:53 |
| mattoliverau | or not len(nodes) in container ring. | 03:53 |
| mattoliverau | tho am happy to be wrong :) | 03:54 |
| *** sanchitmalhotra has quit IRC | 03:55 | |
| kota_ | wallking thourgh to be fresh my memory.... | 03:55 |
| kota_ | so | 03:55 |
| kota_ | n_outgoing is the number of headers for object-server | 03:56 |
| mattoliverau | kota_: actually, maybe your right, on a third read :) | 03:56 |
| kota_ | and then | 03:56 |
| kota_ | for i, container in containers: | 03:57 |
| kota_ | 0<i<2 range. | 03:57 |
| mattoliverau | I see now | 03:57 |
| kota_ | i.e. 3 container update headers will be generated. | 03:57 |
| kota_ | ok. | 03:57 |
| kota_ | s/0<i<2/0<=i<=2/ | 03:58 |
| mattoliverau | so same with delete_at_nodes | 03:58 |
| kota_ | but | 03:58 |
| kota_ | i saw the change for the number of .durable file just now. | 03:58 |
| *** zhill has joined #openstack-swift | 03:58 | |
| *** pberis has quit IRC | 03:59 | |
| kota_ | the bug might not be happen? | 03:59 |
| mattoliverau | yeah but the successful durable files may or may not include the headers, that's the problem right. that is if the 3 that fail have headers, less likely tho | 03:59 |
| kota_ | because currently .durable file will be created nparity + 1 at least. | 03:59 |
| kota_ | yup. maybe I need to look at this again to consider current swift still have this issue. | 04:00 |
| kota_ | but i guess, it may still be. | 04:01 |
| *** ppai has joined #openstack-swift | 04:01 | |
| kota_ | not sure. | 04:01 |
| mattoliverau | kota_: so on a 10 + 4, the quruom will be 4+1? | 04:01 |
| kota_ | mattoliverau: quruom is 10 + 1 | 04:01 |
| *** links has joined #openstack-swift | 04:02 | |
| kota_ | mattoliverau: and .durable will be 4 + 1 | 04:02 |
| *** dimasot has quit IRC | 04:02 | |
| *** kei_yama has joined #openstack-swift | 04:02 | |
| *** ho has quit IRC | 04:02 | |
| kota_ | so .data will be made 10 + 1 as quorum... | 04:03 |
| *** wshao has joined #openstack-swift | 04:03 | |
| kota_ | ah, it is still have the issue. | 04:03 |
| kota_ | the quorum number 11 might not includes the container-update nodes (i.e. missing first 3 nodes) | 04:04 |
| kota_ | not related to .durable number. | 04:04 |
| mattoliverau | yeah | 04:05 |
| kota_ | sorry, I got phone call | 04:05 |
| mattoliverau | so maybe we need to make quroum copies of output_headers. | 04:05 |
| kota_ | that's a way. | 04:07 |
| kota_ | Or pushing the output_headers as footer if the output_headers are missing. | 04:08 |
| kota_ | into the rest of nodes. | 04:08 |
| kota_ | Or ... I might have another idea before...but not get back into my memory right now. | 04:10 |
| mattoliverau | lol, more footer stuff, soon we'll footer everything :P | 04:11 |
| *** baojg has joined #openstack-swift | 04:11 | |
| kota_ | mattoliverau: lol | 04:12 |
| mattoliverau | kota_: cool, sorry for confusing the issue.. it's funny how many times I've looked at the code and missed it :) | 04:12 |
| kota_ | mattoliverau: no worries. that was good chance to make me remember. | 04:13 |
| *** mahatic has joined #openstack-swift | 04:18 | |
| mattoliverau | morning mahatic :) | 04:21 |
| mahatic | mattoliverau: Good Morning! :) | 04:22 |
| mahatic | just going through the meeting log :) | 04:22 |
| jrichli | mahatic, mattoliverau: good morning! I am off to bed. | 04:22 |
| mahatic | jrichli: good night! | 04:22 |
| jrichli | thx. kota_ good morning! | 04:23 |
| notmyname | good morning mahatic | 04:23 |
| mahatic | good evening notmyname | 04:23 |
| *** jrichli has quit IRC | 04:24 | |
| *** baojg has quit IRC | 04:31 | |
| *** kota_ has quit IRC | 04:36 | |
| *** trifon has joined #openstack-swift | 04:36 | |
| *** wshao has quit IRC | 04:37 | |
| *** kota_ has joined #openstack-swift | 04:45 | |
| *** ChanServ sets mode: +v kota_ | 04:45 | |
| kota_ | jrichli: have a good night! | 04:46 |
| *** ppai has quit IRC | 04:49 | |
| *** trifon has quit IRC | 04:50 | |
| *** bkopilov has joined #openstack-swift | 04:57 | |
| mattoliverau | morning evening and night, this timezone stuff is confusing :P | 04:58 |
| kota_ | mattoliverau: exactly, we are working all over the world. | 04:59 |
| *** ppai has joined #openstack-swift | 05:00 | |
| *** zhill has quit IRC | 05:03 | |
| * mahatic is back in a bit | 05:06 | |
| *** mahatic has quit IRC | 05:06 | |
| *** kei_yama has quit IRC | 05:07 | |
| *** kei_yama has joined #openstack-swift | 05:11 | |
| openstackgerrit | Alexandra Settle proposed openstack/swift: Moving DLO functionality doc to middleware https://review.openstack.org/219991 | 05:12 |
| *** SkyRocknRoll has joined #openstack-swift | 05:16 | |
| *** hrou has quit IRC | 05:18 | |
| *** silor has joined #openstack-swift | 05:22 | |
| *** Kennan2 has joined #openstack-swift | 05:24 | |
| *** eranrom has joined #openstack-swift | 05:25 | |
| *** Kennan has quit IRC | 05:25 | |
| *** trifon has joined #openstack-swift | 05:25 | |
| *** eranrom has quit IRC | 05:27 | |
| *** eranrom has joined #openstack-swift | 05:28 | |
| *** sanchitmalhotra has joined #openstack-swift | 05:32 | |
| *** dimasot has joined #openstack-swift | 05:33 | |
| *** sanchitmalhotra1 has quit IRC | 05:34 | |
| *** SkyRocknRoll has quit IRC | 05:35 | |
| prometheanfire | notmyname: not a simple url for use, but at least tags | 05:45 |
| openstackgerrit | Matthew Oliver proposed openstack/swift: Moving DLO functionality doc to middleware https://review.openstack.org/219991 | 05:47 |
| *** SkyRocknRoll has joined #openstack-swift | 05:48 | |
| *** SkyRocknRoll has quit IRC | 05:48 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:48 | |
| *** SkyRocknRoll has quit IRC | 05:52 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 05:57 | |
| *** sanchitmalhotra has quit IRC | 05:59 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:04 | |
| *** mahatic has joined #openstack-swift | 06:08 | |
| *** ho has joined #openstack-swift | 06:11 | |
| *** sanchitmalhotra has joined #openstack-swift | 06:16 | |
| *** ho has quit IRC | 06:16 | |
| *** sanchitmalhotra1 has quit IRC | 06:18 | |
| *** SkyRocknRoll has quit IRC | 06:19 | |
| *** ho has joined #openstack-swift | 06:21 | |
| *** chmouel has quit IRC | 06:26 | |
| *** chmouel has joined #openstack-swift | 06:28 | |
| openstackgerrit | Romain LE DISEZ proposed openstack/swift: Allows to configure the rsync modules where the replicators will send data https://review.openstack.org/192186 | 06:34 |
| *** SkyRocknRoll has joined #openstack-swift | 06:35 | |
| *** nadeem has joined #openstack-swift | 06:36 | |
| *** nadeem has quit IRC | 06:44 | |
| *** nadeem has joined #openstack-swift | 06:50 | |
| *** nadeem has quit IRC | 06:56 | |
| *** ppai has quit IRC | 06:57 | |
| *** rledisez has joined #openstack-swift | 07:04 | |
| *** ppai has joined #openstack-swift | 07:10 | |
| *** nadeem has joined #openstack-swift | 07:22 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 07:30 | |
| *** jkugel has joined #openstack-swift | 07:30 | |
| *** sanchitmalhotra has quit IRC | 07:32 | |
| *** SkyRocknRoll has quit IRC | 07:34 | |
| *** geaaru has joined #openstack-swift | 07:39 | |
| *** nadeem has quit IRC | 07:40 | |
| openstackgerrit | Merged openstack/python-swiftclient: Cleanup and improve tests for download https://review.openstack.org/217246 | 07:43 |
| *** sanchitmalhotra has joined #openstack-swift | 07:55 | |
| *** sanchitmalhotra1 has quit IRC | 07:58 | |
| *** [1]eranrom has joined #openstack-swift | 08:01 | |
| *** kota_ has quit IRC | 08:02 | |
| *** links has quit IRC | 08:02 | |
| *** eranrom has quit IRC | 08:03 | |
| *** [1]eranrom is now known as eranrom | 08:03 | |
| *** acoles_ is now known as acoles | 08:06 | |
| acoles | good morning | 08:06 |
| mahatic | good morning! | 08:07 |
| mattoliverau | acoles: morning | 08:15 |
| *** xnox has quit IRC | 08:18 | |
| *** xnox has joined #openstack-swift | 08:18 | |
| cschwede | good morning! | 08:18 |
| acoles | mahatic: mattoliverau cschwede hi! | 08:19 |
| cschwede | looks like mattoliverau is doing some bug hunting today :) | 08:19 |
| *** sasi has joined #openstack-swift | 08:19 | |
| mattoliverau | Yeah, I'm a team player | 08:19 |
| *** sasi has left #openstack-swift | 08:19 | |
| *** sasik has joined #openstack-swift | 08:20 | |
| *** sasik has left #openstack-swift | 08:20 | |
| *** dmorita has quit IRC | 08:27 | |
| janonymous | morning | 08:30 |
| openstackgerrit | Merged openstack/python-swiftclient: Increase httplib._MAXHEADERS to 256. https://review.openstack.org/214144 | 08:37 |
| *** jistr has joined #openstack-swift | 08:37 | |
| *** nadeem has joined #openstack-swift | 08:42 | |
| *** nadeem has quit IRC | 08:43 | |
| *** nadeem has joined #openstack-swift | 08:46 | |
| pksingh | janonymous: morning | 08:49 |
| janonymous | pksingh: gm :) | 08:53 |
| *** d0ugal has left #openstack-swift | 09:02 | |
| clayg | Romain LE DISEZ <- anyone know a handle? | 09:04 |
| mahatic | clayg: you're awake now?! | 09:04 |
| clayg | it's an accident - i had meetings ALL FLIPPING DAY - so now I can't sleep because I didn't get anything done | 09:06 |
| mahatic | :) | 09:07 |
| acoles | clayg: ! | 09:14 |
| rledisez | hi clayg | 09:16 |
| rledisez | clayg: I think you're looking for me :) | 09:16 |
| *** pksingh has quit IRC | 09:17 | |
| rledisez | clayg: i just read your comment on rsync_module template. I droped the python string interpolation because ini parser were trying to interpolate it. patchset 3 was pushed in a hurry, i though of it after | 09:19 |
| *** pksingh has joined #openstack-swift | 09:19 | |
| clayg | rledisez: ohai! | 09:25 |
| clayg | rledisez: i almost didn't catch you - after reviewing your patch I was feeling kind of productive and was thinking about quittin' out | 09:25 |
| *** nadeem has quit IRC | 09:26 | |
| *** ho has quit IRC | 09:30 | |
| rledisez | clayg: did you have fail with probe test or it's just that you thought of it? | 09:30 |
| rledisez | clayg: but you're right, it's the same code duplicated 3 times, it must be a function | 09:31 |
| *** ho has joined #openstack-swift | 09:33 | |
| *** kota_ has joined #openstack-swift | 09:40 | |
| *** ChanServ sets mode: +v kota_ | 09:40 | |
| clayg | rledisez: with no config changes probetests worked fine - it wasn't until I set the config and ran probetests that I saw them get Skipped | 09:41 |
| *** marzif has joined #openstack-swift | 09:42 | |
| clayg | ah - but I have validate_rsync = true in [probe_test] section - so you know what - that's maybe not such a big deal ... well it depends how many people run with rsnc_validation on | 09:42 |
| clayg | we don't want more people than have to experience the wtf "i pulled down master and now all my probetests skip" experience | 09:43 |
| *** wbhuber has joined #openstack-swift | 09:43 | |
| clayg | acoles: mattoliverau: can you check /etc/swift/tests.conf on your saio's and see if you run validate_rysnc = true | 09:43 |
| clayg | ^ anone else | 09:43 |
| rledisez | i think that if some tests are broken by the patch, it need to be fixed. i'll take a look at it | 09:44 |
| rledisez | clayg: unrelated, but i pulled your branch on reuseport. i'll make some cleaning and testing. i also need to check some behaviors. if everything is alright i'll update the unit tests | 09:45 |
| rledisez | what worries me the most is that the HUP might be sent even if the start of the new instance fails. i'll test it | 09:46 |
| acoles | clayg: no. should I? | 09:47 |
| *** wbhuber has quit IRC | 09:47 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Don't send commit fro quorum *BAD* requests on EC https://review.openstack.org/220059 | 09:48 |
| mattoliverau | clayg: nope I'm running the default, which is commented out as False. | 09:48 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Don't send commits for quorum *BAD* requests on EC https://review.openstack.org/220059 | 09:49 |
| clayg | rledisez: well there you go, not even a big deal on the probetests | 09:49 |
| *** kota_ has quit IRC | 09:59 | |
| clayg | rledisez: also thank you for looking at my stab at the REUSEPORT stuff - feel free take any of that diff and just squash it into the next version of patch # 137659 | 10:02 |
| *** marzif has quit IRC | 10:08 | |
| *** mahatic has quit IRC | 10:08 | |
| *** sasik1 has joined #openstack-swift | 10:11 | |
| *** mahatic has joined #openstack-swift | 10:12 | |
| *** sc68cal has quit IRC | 10:13 | |
| *** sasik1 has left #openstack-swift | 10:13 | |
| *** SkyRocknRoll has joined #openstack-swift | 10:23 | |
| *** links has joined #openstack-swift | 10:31 | |
| charz | clayg: you should take a rest... | 10:35 |
| *** m_kazuhiro has quit IRC | 10:37 | |
| clayg | charz: i'm taking off on friday! | 10:40 |
| *** Kennan2 is now known as Kennan_on_vacati | 10:44 | |
| *** Kennan_on_vacati is now known as Kennan_Vacation | 10:45 | |
| *** aix has quit IRC | 10:46 | |
| *** marzif has joined #openstack-swift | 10:47 | |
| *** mahatic has quit IRC | 10:49 | |
| *** nadeem has joined #openstack-swift | 10:55 | |
| *** SkyRocknRoll has quit IRC | 11:01 | |
| *** SkyRocknRoll has joined #openstack-swift | 11:13 | |
| *** aix has joined #openstack-swift | 11:15 | |
| *** SkyRocknRoll has quit IRC | 11:22 | |
| *** ppai has quit IRC | 11:33 | |
| *** pbourke has quit IRC | 11:36 | |
| *** pbourke has joined #openstack-swift | 11:37 | |
| *** nadeem has quit IRC | 11:39 | |
| *** bkopilov has quit IRC | 11:51 | |
| *** breitz has quit IRC | 11:51 | |
| *** jkugel has quit IRC | 11:51 | |
| *** breitz has joined #openstack-swift | 11:51 | |
| *** nadeem has joined #openstack-swift | 12:05 | |
| *** thurloat_isgone is now known as thurloat | 12:10 | |
| *** km has quit IRC | 12:13 | |
| *** marzif has quit IRC | 12:14 | |
| *** marzif has joined #openstack-swift | 12:15 | |
| *** jkugel has joined #openstack-swift | 12:20 | |
| *** jkugel has quit IRC | 12:21 | |
| *** petertr7_away is now known as petertr7 | 12:22 | |
| *** ttrumm has joined #openstack-swift | 12:28 | |
| *** silor has quit IRC | 12:30 | |
| *** ho_ has joined #openstack-swift | 12:35 | |
| *** ho has quit IRC | 12:36 | |
| *** nadeem has quit IRC | 12:51 | |
| *** nadeem has joined #openstack-swift | 12:52 | |
| *** thurloat is now known as thurloat_isgone | 12:52 | |
| *** [1]eranrom has joined #openstack-swift | 12:57 | |
| *** eranrom has quit IRC | 12:57 | |
| *** [1]eranrom is now known as eranrom | 12:57 | |
| *** ho_ has quit IRC | 12:57 | |
| *** hrou has joined #openstack-swift | 13:01 | |
| *** jlhinson has joined #openstack-swift | 13:04 | |
| *** petertr7 is now known as petertr7_away | 13:09 | |
| *** kei_yama has quit IRC | 13:13 | |
| *** petertr7_away is now known as petertr7 | 13:15 | |
| *** wbhuber has joined #openstack-swift | 13:15 | |
| *** dustins has joined #openstack-swift | 13:15 | |
| *** itlinux has joined #openstack-swift | 13:16 | |
| *** jkugel has joined #openstack-swift | 13:19 | |
| *** ctrath has joined #openstack-swift | 13:19 | |
| *** AndreiaKumpera has joined #openstack-swift | 13:23 | |
| openstackgerrit | janonymous proposed openstack/swift: bin directory: :Replace python print operator with print function (pep H233, py33) https://review.openstack.org/206602 | 13:24 |
| openstackgerrit | janonymous proposed openstack/swift: bin directory: :Replace python print operator with print function (pep H233, py33) https://review.openstack.org/206602 | 13:29 |
| *** janonymous_ has joined #openstack-swift | 13:30 | |
| *** jlhinson has quit IRC | 13:40 | |
| *** links has quit IRC | 13:40 | |
| *** vinsh_ has joined #openstack-swift | 13:42 | |
| openstackgerrit | Merged openstack/swift: [Labs] Update links to Cloud Admin Guide https://review.openstack.org/219933 | 13:45 |
| *** vinsh has quit IRC | 13:45 | |
| *** nakagawamsa has quit IRC | 13:48 | |
| *** ttrumm has quit IRC | 13:50 | |
| *** jroll has quit IRC | 13:54 | |
| *** jroll has joined #openstack-swift | 13:54 | |
| *** vinsh has joined #openstack-swift | 13:55 | |
| *** vinsh_ has quit IRC | 13:57 | |
| *** lpabon has joined #openstack-swift | 13:58 | |
| *** geaaru has quit IRC | 13:59 | |
| *** petertr7 is now known as petertr7_away | 14:00 | |
| *** trifon has quit IRC | 14:02 | |
| *** sanchitmalhotra has quit IRC | 14:08 | |
| *** jrichli has joined #openstack-swift | 14:08 | |
| *** vinsh_ has joined #openstack-swift | 14:09 | |
| *** vinsh has quit IRC | 14:11 | |
| *** geaaru has joined #openstack-swift | 14:11 | |
| *** mahatic has joined #openstack-swift | 14:12 | |
| *** vinsh has joined #openstack-swift | 14:15 | |
| *** vinsh_ has quit IRC | 14:15 | |
| openstackgerrit | janonymous proposed openstack/swift: Python 3 using builtins instead of __builtin__, rename raw_input() to input() from six.moves https://review.openstack.org/200932 | 14:15 |
| *** jmccarthy__ has joined #openstack-swift | 14:19 | |
| *** bkopilov has joined #openstack-swift | 14:32 | |
| *** devlaps has quit IRC | 14:33 | |
| *** jlhinson has joined #openstack-swift | 14:37 | |
| *** lpabon_ has joined #openstack-swift | 14:40 | |
| *** lpabon has quit IRC | 14:40 | |
| *** garthb has joined #openstack-swift | 14:40 | |
| *** itlinux has quit IRC | 14:43 | |
| notmyname | good morning | 14:48 |
| notmyname | FWIW my saio has validate_rsync as true | 14:48 |
| mahatic | good morning | 14:50 |
| notmyname | mahatic! how are you? | 14:50 |
| mahatic | notmyname: I'm doing good :) what's up? | 14:51 |
| notmyname | I've got most of my day blocked off for some people coming into the office. so I'm just trying to spend a little time doing some work this AM ;-) | 14:51 |
| *** eranrom has quit IRC | 14:52 | |
| notmyname | mahatic: how's the encryption stuff going? | 14:52 |
| mahatic | I see. yeah, it's pretty early for you. | 14:52 |
| notmyname | not too bad. my kids just left for school | 14:52 |
| mahatic | it's going okay. I have had to go around quite a bit for some visa docs. it ate up so much time! :| tonight is dedicated to encryption :D (I need to go to the consulate in the morning) | 14:53 |
| mahatic | ah, you do wake up early | 14:53 |
| notmyname | is that for tokyo? | 14:53 |
| mahatic | yes | 14:54 |
| notmyname | cool. I hope all that goes well. I'm looking forward to meeting you in person! | 14:54 |
| mahatic | I hope so too. Likewise! | 14:54 |
| MooingLemur | Lately, I've been running into some issues with object-server which looks like it's blocking for 5-10 seconds at a time without accepting writes, and the TCP rwin drops to 0 during that time. I think the stall is happening from within the daemon, not due to disk IO. | 14:55 |
| *** janonymous_ has quit IRC | 14:55 | |
| notmyname | mahatic: you started working on swift almost exactly a year ago | 14:55 |
| MooingLemur | I'm now using servers_per_port, which seems not to have made it any better | 14:55 |
| notmyname | MooingLemur: interesting | 14:55 |
| MooingLemur | I don't know how to debug this, but I've done some tcpdumps from the proxies | 14:56 |
| *** petertr7_away is now known as petertr7 | 14:56 | |
| MooingLemur | and all of the object servers are dropping the window sizes at different times. The client will upload a file, it'll stall the first 10-15 seconds after the first 5 MB uploaded | 14:56 |
| MooingLemur | then go at 30 MB/s for a while, then drop to 0, then come back up, then drop to 0 | 14:56 |
| notmyname | what client are you using? | 14:57 |
| MooingLemur | curl in this case | 14:57 |
| mahatic | notmyname: yeah! I think my first patch was also merged around this time last year :) you got it all started! thank you! :) | 14:57 |
| notmyname | are you doing several connections at once? | 14:57 |
| MooingLemur | for testing, no, but the cluster is quite active | 14:57 |
| MooingLemur | this is production | 14:57 |
| notmyname | ok. | 14:58 |
| MooingLemur | so I'm wondering if there's contention in the daemon. One of the disks is still on 6000, and sometimes I even get timeouts hitting /healthcheck | 14:58 |
| MooingLemur | BTW, I'm using commit 462aad2a2449f81c8ccc81217db05f2c61514f11 | 14:59 |
| MooingLemur | I know that's close to 2.4, but not quite | 14:59 |
| MooingLemur | (I'm only using that on the object servers at this point, the proxies/account/container are all still using 2.2.0 | 15:00 |
| notmyname | yeah. only thing you're missing is the tempurl patch | 15:00 |
| *** vinsh has quit IRC | 15:00 | |
| *** vinsh_ has joined #openstack-swift | 15:00 | |
| *** minwoob has joined #openstack-swift | 15:00 | |
| *** vinsh has joined #openstack-swift | 15:01 | |
| MooingLemur | These object servers have between 32 and 72 disks each | 15:01 |
| notmyname | so one interesting thing is that /healthcheck is timing out. that doesn't talk to the storage nodes at all | 15:02 |
| *** jistr is now known as jistr|call | 15:02 | |
| notmyname | so you probably need to look at the logs on the cluster and look at the request timings | 15:02 |
| notmyname | see where the delay is. on the storage nodes or in the proxy | 15:02 |
| notmyname | that should narrow your search for the root cause | 15:02 |
| MooingLemur | I mean, I'm pretty sure it's the storage nodes, because nagios is checking /healthcheck on the object nodes directly | 15:03 |
| MooingLemur | and I'm seeing intermittent timeouts with that | 15:03 |
| notmyname | ah ok | 15:03 |
| notmyname | ok, then the next things is to figure out if it's happening for all object servers or just a subset. then figure out if it's for all disks or just a subset | 15:04 |
| notmyname | ie you might be able to drill down and find out if it's one bad disk that's causing the issues | 15:04 |
| notmyname | eg maybe the object server process is causing the whole system to lock up when it hits a bad disk | 15:04 |
| notmyname | (except thats what the servers_per_port is supposed to work around) | 15:05 |
| MooingLemur | I don't think it's a single bad disk, because the behavior happens no matter the request URL, and it's happening on all three of the object servers. But then again, I've been chasing this issue for a while and only the past couple days have I devoted my full attention to it. | 15:05 |
| MooingLemur | I thought servers_per_port would make it go away completely :) | 15:05 |
| MooingLemur | https://bpaste.net/show/1c1d5e2d16d6 <- object-server.conf | 15:06 |
| MooingLemur | does that look sane? | 15:06 |
| *** esker has joined #openstack-swift | 15:10 | |
| notmyname | MooingLemur: only a couple of comments, but not related to your issue | 15:10 |
| MooingLemur | I notice when object-server is restarted, netstat -plnt shows the same pid listening on all the ports. | 15:10 |
| MooingLemur | https://bpaste.net/show/b593f342ef65 | 15:10 |
| notmyname | MooingLemur: first, the disable_path in /tmp could give odd behavior if your system deletes stuff in /tmp on reboot | 15:11 |
| MooingLemur | yeah, I'm aware of that caveat :) | 15:11 |
| notmyname | MooingLemur: second, the servers_per_port of 4 is probably fine. when swifterdarrell did his testing, it seemed that a value of 3 was pretty good | 15:11 |
| notmyname | MooingLemur: ok, so 2 other things. servers_per_port is very new, so I'm really excited to see someone using it in prod! also, you're using ipv6! that's cool too. I don't know anyone else using that | 15:12 |
| notmyname | (I think it's great, so don't take my comment as saying it's bad) | 15:13 |
| MooingLemur | yeah, IIRC the move to eventlet broke ipv6 for a while too, haha | 15:14 |
| notmyname | MooingLemur: but everything is working fine with ipv6 now? | 15:14 |
| notmyname | that's cool | 15:14 |
| MooingLemur | yeah | 15:14 |
| MooingLemur | that's all good now | 15:14 |
| *** esker has quit IRC | 15:14 | |
| notmyname | ok, so back to the slowdowns | 15:14 |
| notmyname | when you get a slow request, have you pulled the logs via trans-id to see what's going on and to look at the logged timings? | 15:15 |
| MooingLemur | in the logs it's hard to see the slow request as slow other than it's a 3GB PUT at 10MB/s rather than 30MB/s. | 15:16 |
| MooingLemur | but what was really happening was 40MB/s, down to 0, back up to 40, etc. | 15:16 |
| notmyname | hmm | 15:17 |
| peluse | acoles, you around? | 15:17 |
| MooingLemur | what's your take on the listeners? It would seem that the same pid listening on all the object server ports would be a problem. If I change the ring, the ports will get listened by different pids | 15:18 |
| MooingLemur | at least for the ones that changed | 15:18 |
| MooingLemur | but once I restart the daemon, it shows them all being listened on by that single process | 15:18 |
| *** vinsh_ has joined #openstack-swift | 15:18 | |
| notmyname | ok, servers_per_port doesn't make the slowdown go away. that's not the goal. the goal is that a slowdown won't affect requests to a different drive. without that, one slow disk will cause the entire process to hang. now it just hangs the request to that one disk | 15:18 |
| *** jmccarthy__ has quit IRC | 15:19 | |
| notmyname | MooingLemur: is that PID the parent process? | 15:19 |
| *** nadeem has quit IRC | 15:19 | |
| MooingLemur | right, I'm aware of that, but it *looks* like it's still affecting the whole daemon, but I haven't proven it yet. | 15:20 |
| *** nadeem has joined #openstack-swift | 15:20 | |
| notmyname | yeah, but if for some reason it is the same process on all disks then that makes sense | 15:20 |
| MooingLemur | it is the parent of all the object-server procs | 15:20 |
| MooingLemur | except for itself of course | 15:21 |
| *** vinsh has quit IRC | 15:21 | |
| *** annegentle has joined #openstack-swift | 15:21 | |
| notmyname | swifterdarrell: does that sound right? for servers_per_port, the listed PID for a socket is the parent process and not the spawned child? | 15:22 |
| notmyname | I'm not sure | 15:22 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Re-organise ssync tests https://review.openstack.org/220198 | 15:22 |
| MooingLemur | They all seem to be listening on all the ports | 15:23 |
| MooingLemur | according to lsof -nPp <pid> | 15:23 |
| * MooingLemur will be back in ~1h | 15:24 | |
| minwoob | Quick question - how many fragments can there be per hashdir in a data file path? | 15:31 |
| notmyname | MooingLemur: you should validate that you have the ring completely pushed out, including to all the proxy servers. just to make sure that the proxy isn't still using the original port and sending everything there | 15:31 |
| minwoob | Exactly one? | 15:31 |
| *** chenhuayi has joined #openstack-swift | 15:31 | |
| *** rjaiswal has joined #openstack-swift | 15:34 | |
| *** kevinc_ has joined #openstack-swift | 15:38 | |
| *** jistr|call is now known as jistr | 15:44 | |
| *** breitz has quit IRC | 15:49 | |
| pbourke | hi, wondering would anyone have ideas for the cause of a HTTP 400 in the object-expirer logs? (http://paste.openstack.org/show/444410/) | 15:54 |
| pbourke | it seems to be when it trys to make a call to get_account_info() | 15:54 |
| pbourke | my object-expirer.conf: http://paste.openstack.org/show/444411/ | 15:54 |
| *** bkopilov has quit IRC | 15:58 | |
| *** marzif has quit IRC | 16:04 | |
| *** nadeem has quit IRC | 16:05 | |
| *** nadeem has joined #openstack-swift | 16:05 | |
| *** jistr has quit IRC | 16:06 | |
| *** bkopilov has joined #openstack-swift | 16:10 | |
| *** aix has quit IRC | 16:12 | |
| *** rledisez has quit IRC | 16:20 | |
| pbourke | ok ignore, conf file needed proxy in pipeline | 16:23 |
| MooingLemur | notmyname: I'm 100% sure. Proxies have been reloaded as well. tcpdump also confirms it's not using 6000 anymore. | 16:26 |
| *** ctrath has quit IRC | 16:27 | |
| *** petertr7 is now known as petertr7_away | 16:29 | |
| *** chsc has joined #openstack-swift | 16:29 | |
| *** nadeem has quit IRC | 16:34 | |
| *** annegentle has quit IRC | 16:36 | |
| *** annegentle has joined #openstack-swift | 16:37 | |
| *** esker has joined #openstack-swift | 16:38 | |
| *** lyrrad has joined #openstack-swift | 16:39 | |
| *** ctrath has joined #openstack-swift | 16:40 | |
| *** lpabon_ has quit IRC | 16:48 | |
| *** haomaiw__ has joined #openstack-swift | 17:02 | |
| *** haomaiwa_ has quit IRC | 17:03 | |
| *** chenhuayi has quit IRC | 17:15 | |
| acoles | peluse: ping | 17:16 |
| *** trifon has joined #openstack-swift | 17:18 | |
| *** kairo has quit IRC | 17:22 | |
| *** esker has quit IRC | 17:23 | |
| *** chsc has quit IRC | 17:25 | |
| *** nadeem has joined #openstack-swift | 17:26 | |
| *** nadeem has quit IRC | 17:26 | |
| *** chsc has joined #openstack-swift | 17:26 | |
| *** zul has quit IRC | 17:26 | |
| *** zul has joined #openstack-swift | 17:27 | |
| *** nadeem has joined #openstack-swift | 17:27 | |
| *** zul has quit IRC | 17:27 | |
| *** zul has joined #openstack-swift | 17:28 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Re-organise ssync tests https://review.openstack.org/220198 | 17:28 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Fix invalid frag_index header in ssync_sender when reverting EC tombstones https://review.openstack.org/217830 | 17:28 |
| notmyname | good morning, again | 17:28 |
| *** annegentle has quit IRC | 17:33 | |
| *** lpabon has joined #openstack-swift | 17:35 | |
| *** lpabon has quit IRC | 17:36 | |
| *** petertr7_away is now known as petertr7 | 17:38 | |
| *** annegentle has joined #openstack-swift | 17:42 | |
| * notmyname will be in meetings most of the day. ping me if you need something, and I'll check in later | 17:42 | |
| *** acoles is now known as acoles_ | 17:46 | |
| *** acoles_ is now known as acoles | 17:48 | |
| *** acoles is now known as acoles_ | 17:49 | |
| notmyname | MooingLemur: ok. | 17:53 |
| *** slo_ has joined #openstack-swift | 17:53 | |
| notmyname | MooingLemur: also, I checked my assumption. the PID listening on the port being the same is ok. that's the parent process | 17:53 |
| notmyname | MooingLemur: so what you're seeing from netstat is expected | 17:53 |
| wbhuber | peluse: For LP 1488610, with the new EC cluster up and running (thx to seamless ansible), I'm able to reproduce the same error specified in the bug you and ctennis saw. You're right that this is not recreatable on SAIO. Off I delve in... | 18:05 |
| openstack | Launchpad bug 1488610 in OpenStack Object Storage (swift) "EC connection refused exception when storage node is offline" [Undecided,Incomplete] https://launchpad.net/bugs/1488610 - Assigned to Bill Huber (wbhuber) | 18:05 |
| *** trifon has quit IRC | 18:06 | |
| *** esker has joined #openstack-swift | 18:17 | |
| *** esker has quit IRC | 18:19 | |
| *** garthb has quit IRC | 18:19 | |
| *** pgbridge has quit IRC | 18:20 | |
| *** esker has joined #openstack-swift | 18:21 | |
| *** geaaru has quit IRC | 18:24 | |
| *** gyee has joined #openstack-swift | 18:24 | |
| *** pgbridge has joined #openstack-swift | 18:28 | |
| *** marzif has joined #openstack-swift | 18:28 | |
| *** annegentle has quit IRC | 18:31 | |
| ctennis | wbhuber: I'll point out not only does it exist, but it's so verbose in the logs when it happens that my log file blew up to over a gigabyte in about an hour. | 18:32 |
| *** kevinc_ has quit IRC | 18:35 | |
| *** esker has quit IRC | 18:38 | |
| *** marzif has quit IRC | 18:38 | |
| *** eranrom has joined #openstack-swift | 18:39 | |
| *** esker has joined #openstack-swift | 18:39 | |
| *** esker has quit IRC | 18:39 | |
| *** esker has joined #openstack-swift | 18:40 | |
| *** trifon has joined #openstack-swift | 18:44 | |
| *** ctrath has quit IRC | 18:44 | |
| *** ctrath has joined #openstack-swift | 18:50 | |
| *** tongli has joined #openstack-swift | 18:51 | |
| *** tongli has quit IRC | 18:52 | |
| *** annegentle has joined #openstack-swift | 19:01 | |
| *** nadeem has quit IRC | 19:04 | |
| *** nadeem has joined #openstack-swift | 19:06 | |
| *** nadeem has quit IRC | 19:11 | |
| *** nadeem has joined #openstack-swift | 19:11 | |
| *** _hrou_ has joined #openstack-swift | 19:15 | |
| *** hrou has quit IRC | 19:19 | |
| wbhuber | ctennis: i hope you're not facing an inoperable cluster with logfiles eating up the root's disk memory. | 19:23 |
| ctennis | wbhuber: I'm not, but it certainly could happen gone unchecked | 19:23 |
| *** changbl has quit IRC | 19:30 | |
| *** thumpba has joined #openstack-swift | 19:32 | |
| *** mahatic has quit IRC | 19:38 | |
| *** _hrou_ has quit IRC | 19:38 | |
| *** nadeem has quit IRC | 19:43 | |
| *** nadeem has joined #openstack-swift | 19:43 | |
| *** trifon has quit IRC | 19:51 | |
| *** petertr7 is now known as petertr7_away | 20:01 | |
| *** petertr7_away is now known as petertr7 | 20:05 | |
| *** eranrom has quit IRC | 20:15 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Add delete markers to versioned_writes middleware https://review.openstack.org/214922 | 20:30 |
| *** dimasot has quit IRC | 20:33 | |
| *** kairo has joined #openstack-swift | 20:43 | |
| *** hugespoon has joined #openstack-swift | 20:44 | |
| *** pgbridge has quit IRC | 20:49 | |
| hugespoon | all, w/ a swift deploy using keystone, is it possible to set intra-project ACLs? i.e. user2 has no read access to a container in the account | 20:50 |
| *** nadeem has quit IRC | 20:50 | |
| hugespoon | the docs all seem to mention inter-project ACLs | 20:50 |
| *** nadeem has joined #openstack-swift | 20:50 | |
| *** rjaiswal has quit IRC | 20:50 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Centralize header parsing https://review.openstack.org/184336 | 20:51 |
| *** nadeem has quit IRC | 20:55 | |
| *** nadeem has joined #openstack-swift | 20:56 | |
| *** dustins has quit IRC | 21:05 | |
| *** greghaynes has quit IRC | 21:10 | |
| openstackgerrit | Merged openstack/swift: Replace a / b with a // b to use integer division where needed https://review.openstack.org/208214 | 21:26 |
| openstackgerrit | Merged openstack/swift: Merge branch 'master' into feature/crypto https://review.openstack.org/219775 | 21:30 |
| *** petertr7 is now known as petertr7_away | 21:31 | |
| *** jrichli has quit IRC | 21:34 | |
| *** zaitcev has joined #openstack-swift | 21:37 | |
| *** ChanServ sets mode: +v zaitcev | 21:37 | |
| *** annegentle has quit IRC | 21:38 | |
| *** marzif has joined #openstack-swift | 21:39 | |
| *** wbhuber has quit IRC | 21:45 | |
| *** jkugel has quit IRC | 21:54 | |
| *** haomaiwa_ has joined #openstack-swift | 21:54 | |
| *** slo_ has quit IRC | 21:55 | |
| *** haomaiw__ has quit IRC | 21:56 | |
| *** annegentle has joined #openstack-swift | 21:57 | |
| mattoliverau | Morning | 22:02 |
| *** annegentle has quit IRC | 22:03 | |
| *** nadeem has quit IRC | 22:08 | |
| peluse | morning | 22:14 |
| *** ctrath has quit IRC | 22:21 | |
| openstackgerrit | Michael Barton proposed openstack/swift: go: improve replication logging https://review.openstack.org/220333 | 22:28 |
| *** greghaynes has joined #openstack-swift | 22:32 | |
| notmyname | whew. back online | 22:36 |
| *** thumpba has quit IRC | 22:40 | |
| peluse | whew, me too :) all mac OS'd now and sorta fully functinoal... | 22:46 |
| mattoliverau | peluse: no more windows \o/ | 22:55 |
| mattoliverau | notmyname: welcome back | 22:55 |
| *** wbhuber has joined #openstack-swift | 22:56 | |
| peluse | mattoliverau, its about time I'd say!!! | 22:58 |
| *** jeblair has quit IRC | 22:59 | |
| *** jeblair has joined #openstack-swift | 23:00 | |
| *** jeblair has quit IRC | 23:01 | |
| *** jeblair has joined #openstack-swift | 23:02 | |
| *** gyee has quit IRC | 23:04 | |
| *** km has joined #openstack-swift | 23:06 | |
| notmyname | peluse: cool. did you choose LimeChat? | 23:06 |
| *** NM has joined #openstack-swift | 23:06 | |
| *** _hrou_ has joined #openstack-swift | 23:06 | |
| *** NM has quit IRC | 23:10 | |
| peluse | notmyname, so I have it installed on my mac but I stil have a windows PC sitting at the office that I remote into from my mac that has Xchat running all the time so I get history there and have access to stupid company tools that need windows... | 23:11 |
| notmyname | ah ok | 23:11 |
| peluse | notmyname, so cool that I can remote desktop to a windows maching from OSX! | 23:11 |
| *** minwoob has quit IRC | 23:11 | |
| peluse | notmyname, but I am using native mail and busycal now isntead of the only approved tools here - Office. They totally suck on OSX for us "visusally impaired" :) | 23:12 |
| mattoliverau | If you want your phone to wake you up at night (if you forget to silent your phone at night) then you should use quassel, set a server somewhere and then you can use the client everywhere :) | 23:12 |
| peluse | mattoliverau, yeah I think Curt does that but its so easy to have this remote machine always on.... | 23:12 |
| mattoliverau | peluse: yeah, and then notmyname finds out that your phone isn't on silent and messages ensue :P | 23:13 |
| peluse | now if I can just remember what I was in the middle of doing when I started this laptop migration... | 23:13 |
| mattoliverau | lol, let me guess... EC? | 23:13 |
| peluse | ahhh yes, that and I guess lining up a pile of bugs to fix as part of this collab huh? I sent mstill a note this morn. | 23:15 |
| peluse | notmyname, we should have a few new folks ramping up (we meaning us and rax) in the next month to start on bug fix then move into feature stuff | 23:15 |
| notmyname | peluse: great! | 23:16 |
| peluse | I think I'm heading to San Antonio at the end of the month to meet a few folks and help kick start the Swift side | 23:17 |
| notmyname | wear shorts | 23:17 |
| * peluse thinks he'll need some shirts :) | 23:17 | |
| peluse | oh shit, I need a new swift sticker for my new laptop too! | 23:18 |
| notmyname | speaking of shirts, I'm currently working on next swfit tshirt designs | 23:18 |
| peluse | sweet! | 23:18 |
| mattoliverau | peluse: I did a bit of a bug audit yesterday, if you check the collab spread sheet. but there are plenty more if your in the mood :P | 23:18 |
| notmyname | peluse: heh, ok. I'll try to bring some to tokyo | 23:18 |
| peluse | mattoliverau, yes, I saw that. awesome! | 23:18 |
| peluse | mattoliverau, BTW can you send me the link. Its on my other laptop running an OS I don't care to look at again :) | 23:18 |
| mattoliverau | lol, of course | 23:19 |
| mattoliverau | I failed a liitle at the audit, cause I ended up getting side tracked fixing one that had been in my name for a while instead of focusing solely on LP :P | 23:20 |
| *** chsc has quit IRC | 23:20 | |
| *** marzif has quit IRC | 23:21 | |
| *** jlhinson has quit IRC | 23:23 | |
| *** annegentle has joined #openstack-swift | 23:29 | |
| *** ho has joined #openstack-swift | 23:32 | |
| ho | good morning! | 23:33 |
| *** kei_yama has joined #openstack-swift | 23:33 | |
| mattoliverau | ho: morning | 23:44 |
| *** changbl has joined #openstack-swift | 23:48 | |
| *** petertr7_away is now known as petertr7 | 23:51 | |
| *** aix has joined #openstack-swift | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!