| *** yangyapeng has quit IRC | 00:19 | |
| *** zhurong has joined #openstack-freezer | 00:43 | |
| *** zhurong has quit IRC | 03:57 | |
| *** zhurong has joined #openstack-freezer | 04:16 | |
| *** grumpycatt has quit IRC | 05:41 | |
| *** jiaopengju has joined #openstack-freezer | 05:50 | |
| *** jiaopeng_ has joined #openstack-freezer | 05:52 | |
| *** jiaopen__ has joined #openstack-freezer | 05:52 | |
| *** jiaopeng_ has quit IRC | 05:52 | |
| *** jiaopen__ has quit IRC | 05:52 | |
| *** yangyapeng has joined #openstack-freezer | 07:06 | |
| *** daemontool has joined #openstack-freezer | 07:49 | |
| *** yangyapeng has quit IRC | 08:10 | |
| *** yangyapeng has joined #openstack-freezer | 08:10 | |
| *** yangyapeng has quit IRC | 09:10 | |
| *** vnogin has joined #openstack-freezer | 09:17 | |
| *** vnogin has quit IRC | 09:27 | |
| openstackgerrit | yfzhao proposed openstack/freezer master: Add Apache License Content in index.rst https://review.openstack.org/456526 | 09:31 |
|---|---|---|
| *** electrofelix has joined #openstack-freezer | 09:39 | |
| *** vnogin has joined #openstack-freezer | 09:45 | |
| *** vnogin has quit IRC | 09:50 | |
| openstackgerrit | Dmitry Stepanenko proposed openstack/freezer-api master: Implement migrations and config opts for oslo.db https://review.openstack.org/452115 | 09:50 |
| *** vnogin has joined #openstack-freezer | 09:50 | |
| *** zhurong has quit IRC | 10:17 | |
| *** daemontool_ has joined #openstack-freezer | 10:24 | |
| *** jiaopengju has quit IRC | 10:24 | |
| *** daemontool has quit IRC | 10:26 | |
| *** zhurong has joined #openstack-freezer | 10:45 | |
| openstackgerrit | howard lee proposed openstack/freezer master: Fix job stop info log print duplicately https://review.openstack.org/456563 | 11:13 |
| *** vnogin has quit IRC | 12:18 | |
| *** daemontool__ has joined #openstack-freezer | 12:43 | |
| *** vnogin has joined #openstack-freezer | 12:45 | |
| *** daemontool_ has quit IRC | 12:46 | |
| *** catintheroof has joined #openstack-freezer | 13:44 | |
| *** catintheroof has quit IRC | 13:45 | |
| *** catintheroof has joined #openstack-freezer | 13:45 | |
| *** catintheroof has quit IRC | 13:50 | |
| *** catintheroof has joined #openstack-freezer | 13:51 | |
| *** zhurong has quit IRC | 13:57 | |
| *** mnaser has joined #openstack-freezer | 13:59 | |
| mnaser | hey folks! we are in the process of deploying freezer and so far everything is great except for a tiny issue that im trying to troubleshoot | 13:59 |
| szaher | Hello Guys | 14:00 |
| mnaser | backups work ok (sent to swift) | 14:00 |
| szaher | Freezer Meeting ? | 14:00 |
| mnaser | i think thats in 2 hours szaher , its 2 pm gmt right now, it says it starts at 4 | 14:00 |
| mnaser | anyways, restores are having issues -- SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1750) -- i tried to do as much research as i can but i haven't gotten much | 14:01 |
| szaher | slashme m3m0 raliev daemontool__ vnogin dstepanenko freezer meeting in #openstack-meeting-alt | 14:01 |
| szaher | mnaser: Hi | 14:01 |
| szaher | No it's 2 GMT | 14:01 |
| szaher | where did you find it 4 gmt ? | 14:01 |
| mnaser | "IRC weekly meeting on #openstack-freezer recurring every Thursday start 4:00 PM - end 5:00 PM GMT " | 14:01 |
| mnaser | in the topic | 14:01 |
| szaher | mnaser: oh sorry for that but you might need to check the official wiki page | 14:03 |
| mnaser | szaher no worries, be useful to have someone update the topic but all good :) | 14:04 |
| szaher | mnaser: you can join the meeting in #openstack-meeting-alt | 14:04 |
| *** yangyapeng has joined #openstack-freezer | 14:08 | |
| *** daemontool__ has quit IRC | 14:41 | |
| *** catintheroof has quit IRC | 14:52 | |
| *** catintheroof has joined #openstack-freezer | 14:52 | |
| mnaser | ok so two things | 15:01 |
| mnaser | i filed this a few days ago | 15:01 |
| mnaser | https://bugs.launchpad.net/freezer/+bug/1681931 | 15:01 |
| openstack | Launchpad bug 1681931 in Freezer "Manage utility failing against cluster" [Undecided,New] | 15:01 |
| mnaser | i think https://github.com/openstack/freezer-api/blob/4.0.0/freezer_api/cmd/manage.py#L123 should be driver.get_options() | 15:02 |
| mnaser | because the get_options in here https://github.com/openstack/freezer-api/blob/4.0.0/freezer_api/storage/driver.py#L75-L89 does split the hosts | 15:02 |
| mnaser | so you either copy and paste things (or point towards it) | 15:03 |
| mnaser | i can send a quick patch for that if we're okay with pointing it to driver.get_options() | 15:03 |
| szaher | mnaser: This is for master branch ? | 15:04 |
| mnaser | yeah problem exists in master too | 15:04 |
| mnaser | oh | 15:04 |
| mnaser | sorry | 15:04 |
| mnaser | it doesnt | 15:04 |
| mnaser | actually, ugh, it does | 15:04 |
| mnaser | https://github.com/openstack/freezer-api/blob/master/freezer_api/cmd/manage.py#L130-L131 | 15:04 |
| *** yangyapeng has quit IRC | 15:05 | |
| mnaser | (brb reboot) | 15:05 |
| *** yangyapeng has joined #openstack-freezer | 15:06 | |
| *** vnogin has quit IRC | 15:11 | |
| mnaser | ok but the bigger problem right now is restores fail | 15:16 |
| mnaser | http://paste.openstack.org/show/606460/ | 15:16 |
| mnaser | this is against a swift cluster that runs ssl | 15:16 |
| mnaser | layer 0 gets downloaded but layer 1 fails with this | 15:17 |
| *** vnogin has joined #openstack-freezer | 15:18 | |
| szaher | mnaser: Can you get the full error message | 15:23 |
| szaher | ? | 15:23 |
| mnaser | szaher let me scrub it out of any private things | 15:26 |
| *** catintheroof has quit IRC | 15:27 | |
| *** yangyapeng has quit IRC | 15:27 | |
| *** catintheroof has joined #openstack-freezer | 15:27 | |
| *** catintheroof has quit IRC | 15:27 | |
| mnaser | szaher http://paste.openstack.org/show/606469/ | 15:28 |
| szaher | mnaser: which branch is this one ? How did you pass the cert to freezer ? | 15:39 |
| mnaser | szaher never pass cert anywhere | 15:39 |
| *** jiaopengju has joined #openstack-freezer | 15:42 | |
| mnaser | szaher the cert is actually a signed valid cert - https://object-storage-ca-ymq-1.vexxhost.net/ | 15:47 |
| *** Ananya26 has joined #openstack-freezer | 15:50 | |
| *** Ananya26 has quit IRC | 16:01 | |
| szaher | mnaser: I can't get what this has to do with elasticsearch ? | 16:18 |
| szaher | mnaser: for https://bugs.launchpad.net/freezer/+bug/1681931 it looks like it can be fixed with the split as you mentioned | 16:20 |
| openstack | Launchpad bug 1681931 in Freezer "Manage utility failing against cluster" [Undecided,New] | 16:20 |
| *** openstack has joined #openstack-freezer | 16:35 | |
| *** catintheroof has joined #openstack-freezer | 16:36 | |
| *** vnogin has quit IRC | 16:44 | |
| *** catintheroof has quit IRC | 16:44 | |
| *** vnogin has joined #openstack-freezer | 17:42 | |
| *** vnogin has quit IRC | 17:47 | |
| *** electrofelix has quit IRC | 18:05 | |
| *** vnogin has joined #openstack-freezer | 18:09 | |
| mnaser | woo | 18:19 |
| mnaser | i found the cryptic bug! | 18:19 |
| mnaser | swiftclient is not threadsafe | 18:19 |
| mnaser | doing ssl in multiprocess can make weird things happen | 18:19 |
| mnaser | so it looks like the swift client is shared and when running a restore + ssl things break! | 18:19 |
| vnogin | wow | 18:21 |
| vnogin | create bug please | 18:21 |
| mnaser | yeah i will do that | 18:22 |
| mnaser | that was the cause of this proble,m | 18:22 |
| mnaser | http://paste.openstack.org/show/606469/ | 18:22 |
| mnaser | os_clients.swift() would give you a single instance of swift | 18:22 |
| mnaser | so in multiprocess it crashes and burns | 18:22 |
| mnaser | i think i will move that part of the code into the storage driver so there is *per* storage instance swift instance | 18:22 |
| mnaser | that way its not shared across processes | 18:22 |
| mnaser | holy hell that was a painful one | 18:23 |
| szaher | mnaser: Great! | 19:08 |
| szaher | mnaser: Did you file a bug for swift guys ? what version of swift client is it ? | 19:08 |
| mnaser | szaher im not sure if this is swift or freezer, based on my research, the 'bug' bubbles up from python requests | 19:09 |
| mnaser | and the folks there pretty much said: ssl in multithread/process is unsafe, dont do it | 19:09 |
| mnaser | https://github.com/kennethreitz/requests/issues/1906 | 19:10 |
| mnaser | very long discussion hter | 19:10 |
| szaher | hmmm, Ok let me take a look | 19:10 |
| openstackgerrit | Mohammed Naser proposed openstack/freezer-api master: Allow bootstrapping against clusters and unify host options https://review.openstack.org/456735 | 19:13 |
| mnaser | that's for the freezer-api fix ^ and now ill file a bug for the ssl | 19:13 |
| openstackgerrit | Mohammed Naser proposed openstack/freezer-api master: Allow bootstrapping against clusters and unify host options https://review.openstack.org/456735 | 19:14 |
| mnaser | szaher https://bugs.launchpad.net/freezer/+bug/1682606 | 19:21 |
| openstack | Launchpad bug 1682606 in OpenStack Backup/Restore and DR (Freezer) "Swift restores fail from SSL endpoints" [Undecided,New] - Assigned to Mohammed Naser (mnaser) | 19:21 |
| mnaser | tracking issue here | 19:21 |
| mnaser | oh look at that | 19:32 |
| mnaser | https://bugs.launchpad.net/freezer/+bug/1563397 | 19:32 |
| openstack | Launchpad bug 1563397 in Freezer "SSLError during swift incremental restore" [Undecided,Fix released] | 19:32 |
| *** vnogin has quit IRC | 20:02 | |
| *** vnogin has joined #openstack-freezer | 20:03 | |
| openstackgerrit | Mohammed Naser proposed openstack/freezer master: Force Swift storage driver to always get a new client https://review.openstack.org/456758 | 20:55 |
| mnaser | there we go | 20:55 |
| mnaser | all done and fixed up | 20:55 |
| *** vnogin has quit IRC | 21:00 | |
| *** vnogin has joined #openstack-freezer | 21:01 | |
| *** vnogin has quit IRC | 21:38 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!