| *** kaisers has joined #openstack-manila | 00:11 | |
| *** baojg has joined #openstack-manila | 00:19 | |
| *** kaisers has quit IRC | 00:19 | |
| *** baojg has quit IRC | 00:25 | |
| *** deep-book-gk has joined #openstack-manila | 01:17 | |
| *** kaisers has joined #openstack-manila | 01:17 | |
| *** deep-book-gk has left #openstack-manila | 01:18 | |
| *** baojg has joined #openstack-manila | 01:20 | |
| *** kaisers has quit IRC | 01:26 | |
| *** baojg has quit IRC | 01:27 | |
| *** gouthamr has quit IRC | 02:07 | |
| *** jiaopengju has joined #openstack-manila | 02:20 | |
| *** kaisers has joined #openstack-manila | 02:23 | |
| *** baojg has joined #openstack-manila | 02:24 | |
| *** baojg has quit IRC | 02:29 | |
| *** tuanluong has joined #openstack-manila | 02:38 | |
| *** kaisers has quit IRC | 02:42 | |
| openstackgerrit | zhongshengping proposed openstack/puppet-manila master: Update openstackdocstheme>=1.16.0 https://review.openstack.org/489051 | 03:02 |
|---|---|---|
| openstackgerrit | zhongjun proposed openstack/manila master: Fix cannot deny ipv6 access rules https://review.openstack.org/488938 | 03:03 |
| openstackgerrit | zhongjun proposed openstack/manila master: Fix cannot deny ipv6 access rules https://review.openstack.org/488938 | 03:09 |
| *** baojg has joined #openstack-manila | 03:25 | |
| *** baojg has quit IRC | 03:32 | |
| *** kaisers has joined #openstack-manila | 03:39 | |
| *** kaisers has quit IRC | 03:49 | |
| *** kaisers has joined #openstack-manila | 04:14 | |
| *** eharney has quit IRC | 04:19 | |
| *** kaisers has quit IRC | 04:24 | |
| *** baojg has joined #openstack-manila | 04:28 | |
| *** rejy has joined #openstack-manila | 04:31 | |
| *** baojg has quit IRC | 04:32 | |
| *** kaisers has joined #openstack-manila | 04:54 | |
| *** kaisers has quit IRC | 05:05 | |
| *** arnewiebalck_ has joined #openstack-manila | 05:10 | |
| *** rejy has quit IRC | 05:15 | |
| *** rraja has joined #openstack-manila | 05:23 | |
| *** arnewiebalck_ has quit IRC | 05:28 | |
| *** rraja has quit IRC | 05:28 | |
| *** baojg has joined #openstack-manila | 05:29 | |
| *** baojg has quit IRC | 05:39 | |
| *** GK1wmSU has joined #openstack-manila | 05:40 | |
| *** GK1wmSU has left #openstack-manila | 05:42 | |
| *** _GK1wmSU has joined #openstack-manila | 05:54 | |
| *** _GK1wmSU has left #openstack-manila | 05:54 | |
| *** _GK1wmSU has joined #openstack-manila | 05:54 | |
| *** _GK1wmSU has left #openstack-manila | 05:57 | |
| *** kaisers has joined #openstack-manila | 06:01 | |
| *** pcaruana has joined #openstack-manila | 06:19 | |
| *** tuanluong has quit IRC | 06:19 | |
| *** kaisers has quit IRC | 06:33 | |
| *** tuanluong has joined #openstack-manila | 07:18 | |
| *** td has quit IRC | 07:28 | |
| *** kaisers has joined #openstack-manila | 07:30 | |
| *** baojg has joined #openstack-manila | 07:35 | |
| *** baojg has quit IRC | 07:40 | |
| *** kaisers has quit IRC | 07:41 | |
| *** td has joined #openstack-manila | 07:42 | |
| *** dsariel has joined #openstack-manila | 07:52 | |
| *** baojg has joined #openstack-manila | 08:37 | |
| *** kaisers has joined #openstack-manila | 08:38 | |
| *** baojg has quit IRC | 08:41 | |
| *** kaisers has quit IRC | 08:59 | |
| *** lpetrut has joined #openstack-manila | 09:15 | |
| *** tosky has joined #openstack-manila | 09:21 | |
| *** jiaopengju1 has joined #openstack-manila | 09:25 | |
| *** jiaopengju has quit IRC | 09:27 | |
| *** baojg has joined #openstack-manila | 09:38 | |
| *** baojg has quit IRC | 09:43 | |
| *** kaisers has joined #openstack-manila | 09:56 | |
| *** kaisers has quit IRC | 10:05 | |
| *** jiaopengju1 has quit IRC | 10:09 | |
| openstackgerrit | Merged openstack/manila-ui master: Updated from global requirements https://review.openstack.org/488238 | 10:36 |
| *** baojg has joined #openstack-manila | 10:39 | |
| *** baojg has quit IRC | 10:44 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/manila-ui master: Imported Translations from Zanata https://review.openstack.org/489174 | 10:47 |
| *** ganso has joined #openstack-manila | 10:51 | |
| *** kaisers has joined #openstack-manila | 11:02 | |
| *** tuanluong has quit IRC | 11:08 | |
| *** kaisers has quit IRC | 11:24 | |
| *** baojg has joined #openstack-manila | 11:41 | |
| *** baojg has quit IRC | 11:46 | |
| *** arnewiebalck_ has joined #openstack-manila | 12:01 | |
| openstackgerrit | Merged openstack/python-manilaclient master: Update reno for stable/pike https://review.openstack.org/488787 | 12:12 |
| *** kaisers has joined #openstack-manila | 12:20 | |
| *** catintheroof has joined #openstack-manila | 12:29 | |
| *** kaisers has quit IRC | 12:31 | |
| *** baojg has joined #openstack-manila | 12:41 | |
| openstackgerrit | Merged openstack/manila-ui master: Imported Translations from Zanata https://review.openstack.org/489174 | 12:46 |
| *** baojg has quit IRC | 12:48 | |
| *** rraja has joined #openstack-manila | 12:48 | |
| *** baojg has joined #openstack-manila | 12:51 | |
| *** baojg has quit IRC | 12:51 | |
| *** baojg has joined #openstack-manila | 12:54 | |
| *** cknight has joined #openstack-manila | 12:56 | |
| *** mtanino has joined #openstack-manila | 13:04 | |
| *** dustins has joined #openstack-manila | 13:05 | |
| *** arnewiebalck_ has quit IRC | 13:08 | |
| openstackgerrit | Merged openstack/manila master: Imported Translations from Zanata https://review.openstack.org/488993 | 13:14 |
| *** baojg has quit IRC | 13:15 | |
| *** baojg has joined #openstack-manila | 13:16 | |
| *** arnewiebalck_ has joined #openstack-manila | 13:28 | |
| *** kaisers has joined #openstack-manila | 13:28 | |
| *** arnewiebalck_ has quit IRC | 13:28 | |
| *** arnewiebalck_ has joined #openstack-manila | 13:30 | |
| *** gouthamr has joined #openstack-manila | 13:30 | |
| *** rraja has quit IRC | 13:43 | |
| *** gcb has joined #openstack-manila | 13:46 | |
| bswartz | tbarron: can you explain your comment above about leaking file descriptors? | 13:51 |
| *** eharney has joined #openstack-manila | 13:52 | |
| bswartz | tbarron: n/m I followed the link you posted | 13:53 |
| bswartz | I'm really confused | 13:53 |
| bswartz | tbarron: https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/591823 | 13:57 |
| openstack | Launchpad bug 591823 in lvm2 ""File descriptor \d+ (\S+) leaked on lvs invocation."" [Unknown,Fix released] | 13:57 |
| *** jiaopengju has joined #openstack-manila | 14:02 | |
| *** kaisers has quit IRC | 14:02 | |
| *** xyang1 has joined #openstack-manila | 14:05 | |
| *** jiaopengju has quit IRC | 14:07 | |
| *** arnewiebalck_ has quit IRC | 14:15 | |
| tbarron | bswartz: thanks | 14:20 |
| *** jiaopengju has joined #openstack-manila | 14:22 | |
| *** chenying__ has quit IRC | 14:33 | |
| *** openstackgerrit has quit IRC | 14:33 | |
| *** kaisers has joined #openstack-manila | 15:00 | |
| *** pcaruana has quit IRC | 15:04 | |
| *** markstur has joined #openstack-manila | 15:07 | |
| *** kaisers has quit IRC | 15:09 | |
| *** gcb has quit IRC | 15:23 | |
| *** jiaopengju has quit IRC | 15:24 | |
| *** dsariel has quit IRC | 15:36 | |
| *** jmlowe has quit IRC | 15:37 | |
| *** kaisers has joined #openstack-manila | 16:06 | |
| *** baojg has quit IRC | 16:07 | |
| *** kaisers has quit IRC | 16:25 | |
| *** kaisers has joined #openstack-manila | 16:46 | |
| bswartz | tbarron: based on the bug description it looks like something that needs to be worked around but it's not really our fault | 16:48 |
| bswartz | tbarron: we might want to use the LVM_SUPPRESS_FD_WARNINGS thing | 16:53 |
| bswartz | not sure how that works with the rootwrap | 16:53 |
| *** kaisers has quit IRC | 16:55 | |
| *** kaisers has joined #openstack-manila | 17:08 | |
| *** lpetrut has quit IRC | 17:15 | |
| bswartz | tbarron: actually upon deeper investigation I think the leaked file descriptor warning is a red herring | 17:15 |
| bswartz | the actual error was on the next line: "Can't remove merging snapshot logical volume lvm-shares/share-snapshot-686b8648-e34c-45fe-bc49-ad7cdfea763f." | 17:16 |
| bswartz | that's what caused error code 5 and what caused us to fail | 17:16 |
| bswartz | the leaked file descriptor warning amounts to log spam that we can ignore | 17:16 |
| tbarron | bswartz: ok, would be nice to suppress it then, but not urgent. I see the lvremove failure. | 17:18 |
| tbarron | bswartz: we're trying to lvremove while the snapshot is still merging? | 17:19 |
| bswartz | tbarron: the merge is an async operation | 17:24 |
| bswartz | we kick it off but don't wait for it to complete | 17:25 |
| bswartz | that may explain why the problem is intermittent | 17:25 |
| bswartz | sometimes it completes fast enough and sometimes it takes too long | 17:25 |
| bswartz | we need a way to block until it's done | 17:29 |
| *** tosky has quit IRC | 17:32 | |
| bswartz | tbarron: https://sourceware.org/git/?p=lvm2.git;a=blob;f=lib/metadata/lv_manip.c;h=1851783bf5cfaca7f53a2b457f04c552fa7d073b;hb=HEAD#l6281 | 17:32 |
| tbarron | bswartz: can we just do a retry loop on the lvremove? | 17:33 |
| bswartz | if we can find a way to tell the difference between this failure and other permanent failures | 17:34 |
| *** jmlowe has joined #openstack-manila | 17:34 | |
| bswartz | I'm trying to navigate the call graph inside lvconvert to see how it arrived at that error | 17:36 |
| bswartz | it seems very strange | 17:36 |
| bswartz | I'm wondering why we have an "-f" flag on lvremove now | 17:36 |
| bswartz | hmm maybe that just skips the interactive confirmation | 17:38 |
| *** jmlowe has quit IRC | 17:39 | |
| *** jmlowe has joined #openstack-manila | 17:39 | |
| bswartz | tbarron: I see that the existing code already has a retry loop | 17:41 |
| bswartz | around lvremove | 17:41 |
| tbarron | bswartz: ack, and I see three tries (all failing) in the logs | 17:48 |
| bswartz | yeah vish seems to have known about the flakiness of lvm calls | 17:48 |
| bswartz | I'm going to see if there's a way I can restructure the LVM driver code to make the merge operation happen synchronously | 17:49 |
| bswartz | when I first wrote this it did not occur to me that there were background timing issues | 17:49 |
| bswartz | so I'm going to try harder to eliminate them before reverting to a horrible polling mechanism | 17:49 |
| *** harlowja has joined #openstack-manila | 18:11 | |
| bswartz | tbarron: there's something odd here in the lvm revert code | 18:13 |
| * bswartz loads up the git blame | 18:13 | |
| bswartz | looks like the code got modified a few times and somehow we have multiple calls to unmount | 18:15 |
| bswartz | ugh | 18:16 |
| bswartz | okay I'm going to end up addressing multiple bugs with 1 fix I think | 18:16 |
| *** lpetrut has joined #openstack-manila | 18:17 | |
| bswartz | tbarron: I now suspect this whole problem is a result of a bad interaction between the revert to snapshot code and the mountable snapshots code in the LVM driver | 18:21 |
| ganso | bswartz: I remember testing this interaction | 18:23 |
| ganso | bswartz: but locally, I did not face such random issues | 18:23 |
| bswartz | ganso: I can see how it would have worked | 18:24 |
| bswartz | but not reliably | 18:24 |
| bswartz | I think the code bugs are my fault, but there are some readability issues added by tpsilva's patch that make it hard to understand why my code was wrong | 18:24 |
| ganso | bswartz: see https://review.openstack.org/#/c/431315/ https://review.openstack.org/#/c/433039/ https://review.openstack.org/#/c/444709/ | 18:25 |
| *** catintheroof has quit IRC | 18:59 | |
| *** catintheroof has joined #openstack-manila | 19:00 | |
| *** catintheroof has quit IRC | 19:00 | |
| *** catintheroof has joined #openstack-manila | 19:01 | |
| *** arnewiebalck_ has joined #openstack-manila | 19:03 | |
| bswartz | ganso: sorry I had a meeting | 19:33 |
| bswartz | ganso: I'm working on a fix | 19:34 |
| *** openstackgerrit has joined #openstack-manila | 19:41 | |
| openstackgerrit | Raissa Sarmento proposed openstack/manila master: Use tempest-plugin service client registration https://review.openstack.org/334596 | 19:41 |
| *** jmlowe has quit IRC | 20:16 | |
| *** cknight has quit IRC | 20:50 | |
| openstackgerrit | Ben Swartzlander proposed openstack/manila master: Change the way snapshots are exported in LVM https://review.openstack.org/488569 | 20:52 |
| bswartz | ^ this is not the fix | 20:53 |
| *** eharney has quit IRC | 20:59 | |
| *** lpetrut has quit IRC | 21:01 | |
| *** catintheroof has quit IRC | 21:17 | |
| *** dustins has quit IRC | 21:23 | |
| *** arnewiebalck_ has quit IRC | 21:32 | |
| *** gouthamr has quit IRC | 22:46 | |
| *** gouthamr has joined #openstack-manila | 23:15 | |
| *** catintheroof has joined #openstack-manila | 23:24 | |
| *** https_GK1wmSU has joined #openstack-manila | 23:39 | |
| *** https_GK1wmSU has left #openstack-manila | 23:40 | |
| *** catintheroof has quit IRC | 23:43 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/python-manilaclient master: Add rally job manifests https://review.openstack.org/489435 | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!