| *** kaisers2 has joined #openstack-manila | 00:09 | |
| *** kaisers1 has quit IRC | 00:10 | |
| *** http_GK1wmSU has joined #openstack-manila | 00:22 | |
| *** http_GK1wmSU has left #openstack-manila | 00:24 | |
| *** baojg has joined #openstack-manila | 00:28 | |
| openstackgerrit | Merged openstack/manila master: Fix multiple issues with revert to snapshot in LVM https://review.openstack.org/488569 | 00:32 |
|---|---|---|
| *** baojg has quit IRC | 00:33 | |
| *** jiaopengju has joined #openstack-manila | 00:34 | |
| openstackgerrit | Merged openstack/manila master: Fix cannot deny ipv6 access rules https://review.openstack.org/488938 | 00:43 |
| *** tinwood has quit IRC | 00:56 | |
| *** tinwood has joined #openstack-manila | 00:57 | |
| *** kaisers_ has joined #openstack-manila | 01:10 | |
| *** kaisers has quit IRC | 01:12 | |
| *** baojg has joined #openstack-manila | 01:29 | |
| *** baojg has quit IRC | 01:34 | |
| *** mtanino has quit IRC | 01:59 | |
| openstackgerrit | zhongshengping proposed openstack/puppet-manila master: Fix unit test for manila::client class https://review.openstack.org/490284 | 02:09 |
| *** gcb has quit IRC | 02:15 | |
| *** gcb has joined #openstack-manila | 02:16 | |
| *** baojg has joined #openstack-manila | 02:30 | |
| *** baojg has quit IRC | 02:38 | |
| *** ganso has quit IRC | 02:46 | |
| *** gouthamr has quit IRC | 03:03 | |
| *** tuanluong has joined #openstack-manila | 03:30 | |
| zhongjun | gouthamr: Hi, we talked about this bug (https://bugs.launchpad.net/python-manilaclient/+bug/1693701) last night. It is just want to show the 'created_at' and 'updated_at' columns in echo when we run access_list command, and it has been fix in the following patch. https://bugs.launchpad.net/manila/+bug/1682795 | 03:40 |
| openstack | Launchpad bug 1693701 in python-manilaclient "Allow user to see which access rule is recent created or recent updated" [Wishlist,Invalid] - Assigned to NidhiMittalHada (nidhimittal19) | 03:40 |
| openstack | Launchpad bug 1682795 in Manila "Allow user to see which access rule is recent created or recent updated." [Undecided,Fix released] - Assigned to yangweiwei (496176919-6) | 03:40 |
| zhongjun | bswartz: above | 03:41 |
| openstackgerrit | Merged openstack/python-manilaclient master: Stop skipping ipv6 delete access rule test https://review.openstack.org/490113 | 03:46 |
| *** vivsoni_ has quit IRC | 04:20 | |
| *** kaisers_ has quit IRC | 04:30 | |
| *** baojg has joined #openstack-manila | 04:35 | |
| *** baojg has quit IRC | 04:42 | |
| *** vivsoni has joined #openstack-manila | 04:49 | |
| *** kaisers has joined #openstack-manila | 05:00 | |
| *** kaisers has quit IRC | 05:05 | |
| *** kaisers has joined #openstack-manila | 05:05 | |
| *** kaisers has quit IRC | 05:09 | |
| openstackgerrit | junbo.li proposed openstack/manila master: Enable some off-by-default checks https://review.openstack.org/484707 | 05:14 |
| *** lpetrut has joined #openstack-manila | 05:14 | |
| *** arnewiebalck_ has joined #openstack-manila | 05:16 | |
| *** dsariel has joined #openstack-manila | 05:36 | |
| *** baojg has joined #openstack-manila | 05:38 | |
| *** baojg has quit IRC | 05:43 | |
| *** dims has quit IRC | 05:52 | |
| *** dims has joined #openstack-manila | 05:54 | |
| *** arnewiebalck_ has quit IRC | 06:02 | |
| *** dsariel has quit IRC | 06:06 | |
| *** kaisers has joined #openstack-manila | 06:10 | |
| *** lpetrut has quit IRC | 06:11 | |
| *** hoonetorg has quit IRC | 06:12 | |
| *** arnewiebalck_ has joined #openstack-manila | 06:15 | |
| *** kaisers has quit IRC | 06:15 | |
| *** arnewiebalck_ has quit IRC | 06:20 | |
| *** dsariel has joined #openstack-manila | 06:21 | |
| *** hoonetorg has joined #openstack-manila | 06:25 | |
| *** baojg has joined #openstack-manila | 06:39 | |
| *** baojg has quit IRC | 06:44 | |
| *** pcaruana has joined #openstack-manila | 06:48 | |
| *** gcb has quit IRC | 07:05 | |
| *** kaisers has joined #openstack-manila | 07:11 | |
| *** baojg has joined #openstack-manila | 07:40 | |
| *** baojg has quit IRC | 07:44 | |
| *** kaisers has quit IRC | 07:58 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/manila master: Imported Translations from Zanata https://review.openstack.org/490380 | 08:05 |
| *** lpetrut has joined #openstack-manila | 08:07 | |
| openstackgerrit | junbo.li proposed openstack/manila master: Enable some off-by-default checks https://review.openstack.org/484707 | 08:14 |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** baojg has joined #openstack-manila | 08:41 | |
| *** baojg has quit IRC | 08:48 | |
| *** kaisers has joined #openstack-manila | 08:55 | |
| *** dims has quit IRC | 09:04 | |
| *** arnewiebalck has joined #openstack-manila | 09:06 | |
| *** dims has joined #openstack-manila | 09:08 | |
| *** kaisers has quit IRC | 09:16 | |
| *** baojg has joined #openstack-manila | 09:43 | |
| *** baojg has quit IRC | 09:48 | |
| *** ganso has joined #openstack-manila | 09:56 | |
| *** dsariel has quit IRC | 10:00 | |
| *** kaisers has joined #openstack-manila | 10:13 | |
| *** dsariel has joined #openstack-manila | 10:13 | |
| *** jiaopengju has quit IRC | 10:14 | |
| *** dsariel has quit IRC | 10:19 | |
| *** huyang has joined #openstack-manila | 10:27 | |
| *** dsariel has joined #openstack-manila | 10:32 | |
| *** dsariel has quit IRC | 10:38 | |
| *** kaisers has quit IRC | 10:40 | |
| *** baojg has joined #openstack-manila | 10:44 | |
| *** baojg has quit IRC | 10:49 | |
| *** dsariel has joined #openstack-manila | 10:50 | |
| *** kaisers has joined #openstack-manila | 11:05 | |
| tommylikehu | hey bswartz , could you share the reason why you removed the lvchange command when reverting the share in lvm driver? https://review.openstack.org/#/c/488569/7/manila/share/drivers/lvm.py | 11:07 |
| *** openstackgerrit has joined #openstack-manila | 11:09 | |
| openstackgerrit | TommyLike proposed openstack/manila master: Fix: [LVM] Can't extend share when snapshots exist https://review.openstack.org/431315 | 11:09 |
| *** kaisers has quit IRC | 11:16 | |
| *** dsariel has quit IRC | 11:37 | |
| *** baojg has joined #openstack-manila | 11:45 | |
| *** baojg has quit IRC | 11:49 | |
| *** baojg has joined #openstack-manila | 11:49 | |
| *** dsariel has joined #openstack-manila | 11:52 | |
| *** eharney has joined #openstack-manila | 11:55 | |
| *** vivsoni has quit IRC | 11:57 | |
| *** zengyingzhe has joined #openstack-manila | 12:08 | |
| *** rraja has joined #openstack-manila | 12:09 | |
| *** kaisers has joined #openstack-manila | 12:13 | |
| *** tuanluong has quit IRC | 12:14 | |
| *** tinwood is now known as tinwood-afk | 12:30 | |
| *** catintheroof has joined #openstack-manila | 12:36 | |
| *** kaisers has quit IRC | 12:36 | |
| *** gcb has joined #openstack-manila | 12:41 | |
| openstackgerrit | Merged openstack/manila master: Enable some off-by-default checks https://review.openstack.org/484707 | 13:21 |
| *** kaisers has joined #openstack-manila | 13:34 | |
| *** tinwood-afk is now known as tinwood | 13:41 | |
| openstackgerrit | Merged openstack/manila master: Use tempest-plugin service client registration https://review.openstack.org/334596 | 13:42 |
| *** kaisers has quit IRC | 13:47 | |
| *** cknight has joined #openstack-manila | 13:54 | |
| bswartz | tommylikehu: it was removed because it wasn't needed | 14:00 |
| bswartz | tommylikehu: it was only included initially because I was doing revert wrong | 14:00 |
| *** baojg has quit IRC | 14:01 | |
| bswartz | the bug fix was to do the the revert right, and then the command wasn't needed anymore | 14:01 |
| *** baojg has joined #openstack-manila | 14:01 | |
| *** xyang1 has joined #openstack-manila | 14:04 | |
| tommylikehu | bswartz: hah, thanks! | 14:05 |
| *** dustins has joined #openstack-manila | 14:07 | |
| openstackgerrit | Ben Swartzlander proposed openstack/manila master: Fix NFSHelper 0-length netmask bug https://review.openstack.org/489639 | 14:07 |
| openstackgerrit | Merged openstack/manila master: Imported Translations from Zanata https://review.openstack.org/490380 | 14:25 |
| *** baojg has quit IRC | 14:32 | |
| *** baojg has joined #openstack-manila | 14:33 | |
| *** lpetrut has quit IRC | 14:39 | |
| *** kaisers has joined #openstack-manila | 14:44 | |
| *** MVenesio has joined #openstack-manila | 14:47 | |
| *** eharney has quit IRC | 14:53 | |
| *** eharney has joined #openstack-manila | 15:09 | |
| openstackgerrit | Merged openstack/python-manilaclient master: Add rally job manifests https://review.openstack.org/489435 | 15:09 |
| *** kaisers has quit IRC | 15:19 | |
| *** baojg has quit IRC | 16:00 | |
| *** kaisers has joined #openstack-manila | 16:16 | |
| *** jmlowe has quit IRC | 16:20 | |
| *** rraja has quit IRC | 16:26 | |
| *** kaisers has quit IRC | 16:27 | |
| *** JoseMello has joined #openstack-manila | 16:29 | |
| *** dsariel has quit IRC | 16:34 | |
| *** jmlowe has joined #openstack-manila | 16:42 | |
| *** pcaruana has quit IRC | 16:54 | |
| *** kaisers has joined #openstack-manila | 17:21 | |
| tbarron | bswartz: just want to check if it is expected that with synchronous lvm snapshot revert test cases will take 50-70s each | 17:28 |
| tbarron | bswartz: and if we understand the long pole phases in the revert | 17:29 |
| tbarron | bswartz: see for example http://logs.openstack.org/39/489639/7/check/gate-manila-tempest-minimal-dsvm-lvm-centos-7/62f0c3f/console.html#_2017-08-03_15_11_53_721810 | 17:29 |
| bswartz | tbarron: errr | 17:30 |
| bswartz | I would have guessed 10-15 seconds | 17:30 |
| tbarron | bswartz: note though that the slowest tests in the lvm jobs are host assisted migration, for which I raised: https://bugs.launchpad.net/manila/+bug/1708491 | 17:31 |
| openstack | Launchpad bug 1708491 in Manila "tempest api migration tests very long" [Undecided,New] | 17:31 |
| bswartz | I guess the test nodes are under heavy load due to tempest | 17:31 |
| bswartz | k | 17:31 |
| bswartz | I'm in favor of increasing timeouts in the short term | 17:31 |
| tbarron | when we get a "slow node" plus these tests running slow ... | 17:31 |
| bswartz | not because of the slow tests, but because of teh slow nodes | 17:31 |
| bswartz | I doubt much can be done about the slowness of lvm revert | 17:32 |
| bswartz | we could reduce parallelism which would speed up the individual tests but slow down the overall job | 17:32 |
| tbarron | bswartz: k, was just checking. Having it synchronous shows things better now. | 17:32 |
| bswartz | we could shuffle tests around to take better advantage of parallelism | 17:32 |
| tbarron | yeah, we could try to put a host assisted migration case in every thread :D | 17:33 |
| bswartz | or we can reduce the number of test cases (possibly reorganizing them to keep good coverage with fewer tests) | 17:33 |
| tbarron | but increasing timeouts may make sense if it's acceptable to infra | 17:33 |
| tbarron | we're certainly using more infra resources with all the rechecks than we would by running fewer jobs a bit longer | 17:34 |
| bswartz | what jobs other than the centos lvm job are having problems? | 17:34 |
| bswartz | I'd like to do a blanket increase, but if we can pick 1 or 2 that really need it we might get a better response | 17:35 |
| tbarron | xenial lvm has the same issues | 17:35 |
| tbarron | generic jobs are slow, if non-voting | 17:35 |
| bswartz | I see | 17:36 |
| tbarron | http://logs.openstack.org/39/489639/7/check/gate-manila-tempest-dsvm-mysql-generic-ubuntu-xenial-nv/05efe94/console.html | 17:36 |
| tbarron | ^^^ host assisted migration jobs run 10 minutes and some fail because of tempest timeouts | 17:36 |
| tbarron | we just ignore these failures and move on | 17:37 |
| bswartz | did ganso write those tests? | 17:37 |
| tbarron | dunno | 17:37 |
| tbarron | these aren't really rc candidate fixes unless we happen to see a solution right away | 17:38 |
| bswartz | I just haven't had time to dig into them | 17:38 |
| tbarron | but since we're scrubbing and doing lots of rechecks I wanted to make sure we note and don't lose the issues | 17:38 |
| tbarron | ack, me either | 17:39 |
| tbarron | I've been a lazy re-checker. | 17:39 |
| bswartz | timeouts are harder to debug than a failing test | 17:42 |
| *** cknight1 has joined #openstack-manila | 17:52 | |
| *** cknight has quit IRC | 17:52 | |
| *** gouthamr has joined #openstack-manila | 17:53 | |
| *** kaisers has quit IRC | 17:58 | |
| *** kaisers has joined #openstack-manila | 18:09 | |
| ganso | tbarron: are those failures consistent across several rechecks? on generic driver it sometimes fails on several steps for unknown reasons | 18:11 |
| tbarron | ganso: well those test cases seem to take a long time on generic even when the job passes, e.g. in that same review there's a run with no failed test cases but we see e.g. | 18:15 |
| tbarron | http://logs.openstack.org/39/489639/7/check/gate-manila-tempest-dsvm-mysql-generic-ubuntu-xenial-nv/05efe94/console.html | 18:15 |
| tbarron | that's 9 minutes on the test case instead of 10 | 18:16 |
| ganso | tbarron: there are no errors on m-dat.log so the migration did not even start | 18:18 |
| ganso | tbarron: in m-shr there are errors on cinder-client failing to attach | 18:18 |
| tbarron | ganso: interesting | 18:19 |
| tbarron | ganso: so there may be other issues with generic driver that the migration tests just happen to trigger | 18:20 |
| ganso | tbarron: yes. It has always been like this, those are long-standing issues with the generic driver, not new issues | 18:21 |
| tbarron | ganso: would you expect this with lvm job (no cinder involved): http://logs.openstack.org/39/489639/7/check/gate-manila-tempest-minimal-dsvm-lvm-ubuntu-xenial/3497af0/console.html#_2017-08-03_16_44_17_824670 ? | 18:21 |
| tbarron | ganso: btw, I'm not claiming any of these issues are new, I'm just trying to analyze why we have to issue rechecks so much. | 18:22 |
| tbarron | ganso: so I'm looking at which test cases are long-poles. | 18:23 |
| ganso | tbarron: for this test MigrationWithShareExtendingNFSTest.test_extend_on_migrated_share_1_True I think the duration is not that bad | 18:23 |
| tbarron | ganso so you expect it to take 85s even with lvm ... | 18:24 |
| tbarron | ganso: and this scenario test to take > 4 min? http://logs.openstack.org/39/489639/7/check/gate-manila-tempest-minimal-dsvm-lvm-ubuntu-xenial/3497af0/console.html#_2017-08-03_16_49_07_839495 | 18:25 |
| tbarron | ganso: I'm just checking, don't have an informed opinion myself. | 18:25 |
| ganso | tbarron: the numbers are slightly high indeed | 18:25 |
| ganso | tbarron: maybe this is a slow host, I don't recall the numbers last time I benchmarked this | 18:26 |
| tbarron | or 299s: http://logs.openstack.org/39/489639/7/check/gate-manila-tempest-minimal-dsvm-lvm-ubuntu-xenial/3497af0/console.html#_2017-08-03_17_01_39_610143 | 18:26 |
| tbarron | we may want to run those migration scenarios in a separate job | 18:27 |
| tbarron | I have a feeling there are a lot of valuable scenario tests we should write that | 18:27 |
| tbarron | are going to take too long if just added to current jobs. | 18:28 |
| *** kaisers has quit IRC | 18:35 | |
| ganso | tbarron: I remember a single test did not take more than 2 minutes in my lab | 18:36 |
| ganso | tbarron: using LVM | 18:36 |
| ganso | tbarron: I suggested moving them to a separate scenario job long time ago, but most people did not agree it was necessary back then and wanted to concentrate efforts somewhere else, like further optimizing the tests | 18:37 |
| ganso | tbarron: vponomaryov invested some effort on that and reduced the total time of those tests (not by much though) | 18:38 |
| tbarron | ganso: ack | 18:39 |
| *** kaisers has joined #openstack-manila | 18:39 | |
| tbarron | ganso: bswartz may be getting us some more time for these jobs anyways .... | 18:39 |
| *** kaisers has quit IRC | 18:46 | |
| *** MVenesio has quit IRC | 18:47 | |
| bswartz | ganso: hi | 18:59 |
| bswartz | we missed you at bug squash yesterday | 18:59 |
| bswartz | we were talking about the migration tests and trying to understand if there are any opportunities for optimization | 18:59 |
| ganso | bswartz: sorry I really couldn't be there :( | 19:00 |
| ganso | bswartz: regarding tests, I believe we could increase polling rate in the code and in tests... but that's pretty much about it | 19:01 |
| ganso | bswartz: if the problem is timeout, then we could separate the tests to a new host | 19:01 |
| bswartz | ganso: so the tests aren't doing a lot of I/O and there are long pauses in the waiter functions? | 19:07 |
| ganso | bswartz: not long pauses, and not a lot of I/O | 19:13 |
| ganso | bswartz: the only ones that do any I/O are the scenario tests | 19:14 |
| ganso | bswartz: there are a lot of waiters at each step | 19:14 |
| bswartz | okay I'll need to run some experiments | 19:14 |
| *** eharney has quit IRC | 19:22 | |
| *** Yogi1 has joined #openstack-manila | 19:38 | |
| *** kaisers has joined #openstack-manila | 19:43 | |
| *** harlowja has quit IRC | 19:57 | |
| bswartz | gouthamr: ping | 20:00 |
| gouthamr | bswartz: pong | 20:00 |
| *** eharney has joined #openstack-manila | 20:02 | |
| *** arnewiebalck_ has joined #openstack-manila | 20:13 | |
| *** kaisers has quit IRC | 20:15 | |
| *** lpetrut has joined #openstack-manila | 20:23 | |
| *** arnewiebalck_ has quit IRC | 20:35 | |
| *** Yogi1 has quit IRC | 21:05 | |
| *** kaisers has joined #openstack-manila | 21:07 | |
| *** harlowja has joined #openstack-manila | 21:15 | |
| *** jmlowe has quit IRC | 21:18 | |
| *** lpetrut has quit IRC | 21:37 | |
| *** kaisers has quit IRC | 21:51 | |
| *** dustins has quit IRC | 22:21 | |
| *** xyang1 has quit IRC | 22:25 | |
| *** JoseMello has quit IRC | 22:26 | |
| *** cknight1 has quit IRC | 23:02 | |
| *** ganso has quit IRC | 23:15 | |
| *** jmlowe has joined #openstack-manila | 23:27 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!