| *** tinwood has quit IRC | 00:00 | |
| *** tinwood has joined #openstack-manila | 00:01 | |
| openstackgerrit | Merged openstack/manila-ui: Updated from global requirements https://review.openstack.org/424884 | 00:05 |
|---|---|---|
| openstackgerrit | Merged openstack/manila: Add mountable snapshots support https://review.openstack.org/345526 | 00:07 |
| *** mtanino has joined #openstack-manila | 00:26 | |
| *** mtanino_ has quit IRC | 00:28 | |
| *** mtanino has quit IRC | 00:32 | |
| *** zengyingzhe has joined #openstack-manila | 00:48 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/manila: Updated from global requirements https://review.openstack.org/424616 | 00:48 |
| *** ganso has quit IRC | 00:54 | |
| tommylikehu | tbarron, bswartz: thanks for your help for the exportfs bugs, I have already tried with the nova vm, but the vm could propbably fail with IPv6 address even network interface is enabled with dhcp6 | 00:57 |
| tommylikehu | sometimes it could work | 00:57 |
| bswartz | tommylikehu: I'm trying another tactic tonight to get ipv6 working | 00:57 |
| tommylikehu | bswartz: did you mentioned this in the conversation between tbarron several hours ago? | 00:59 |
| bswartz | tommylikehu: I've been working on this all yesterday and today | 01:06 |
| bswartz | most of the things I've attempted have failed | 01:06 |
| bswartz | I've had some partial successes but typically each thing I fix breaks something else | 01:06 |
| tommylikehu | bswartz: As the mountable snapshot feature has been merged so the allow_access for snapshot will also be updated for IPv6? | 01:24 |
| *** mtanino has joined #openstack-manila | 01:25 | |
| bswartz | tommylikehu: if you have any merge conflicts with the patches that have merged you should resolve them | 01:47 |
| bswartz | I'm sorry your patch is the last big one still waiting but it's the one we've had the hardest time testing | 01:47 |
| bswartz | if the combination of the 2 features has a hole (like v6-support for snapshot access rules) then we'll have to decide what to do about that\ | 01:48 |
| bswartz | if it's not too hard to add it to the existing patch that would be ideal, but we can look at alternatives too | 01:48 |
| *** tommylikehu has left #openstack-manila | 01:49 | |
| *** tommylikehu has joined #openstack-manila | 01:50 | |
| tommylikehu | bswartz: I'd like to apply this snapshot changes (also new testcases for this) to IPv6, but I am worried we have to test the mountable snapshot for IPv6. obviously we are still in the trouble of testting share with IPv6 | 01:52 |
| bswartz | tommylikehu: don't worry about testing the combination of the 2 features for now | 01:57 |
| bswartz | that's why we have time after feature freeze | 01:57 |
| tommylikehu | bswartz : thanks, working on it | 01:57 |
| openstackgerrit | Mark Sturdevant proposed openstack/manila: Fix MapRFS test__execute to not impact others https://review.openstack.org/424932 | 02:00 |
| *** kaisers has quit IRC | 02:07 | |
| *** mtanino has quit IRC | 02:12 | |
| openstackgerrit | TommyLike proposed openstack/manila: Enable IPv6 in manila(allow access) https://review.openstack.org/312321 | 02:49 |
| bswartz | tbarron: you still around by any chance? | 02:58 |
| tbarron | bswartz: yeah, but got diverted on some downstream stuff, back now though | 02:58 |
| bswartz | tbarron: I've devolved to using cheap tricks to make ipv6 work | 02:58 |
| *** jcsp has quit IRC | 02:58 | |
| tbarron | cheap_tricks++ | 02:59 |
| bswartz | I'm now just creating an internal port directly on the ovs bridge | 02:59 |
| bswartz | and routing all the packets through that | 02:59 |
| *** jcsp has joined #openstack-manila | 02:59 | |
| bswartz | it's not even close to kosher | 02:59 |
| bswartz | but it does work | 02:59 |
| bswartz | someday I'll figure out why neutron can't work right | 03:00 |
| tbarron | "work" == you can launch a compute instance with its nic on that port, it acquires ipv6 address, you can export via ipv6 export location from hypervisor to that addr, and you can then mount inside the compute instance using that export location? | 03:01 |
| bswartz | tbarron: so far I've just established that I can ping and ssh to/from the guest | 03:02 |
| bswartz | I need to get a guest with an NFS client to actually test the manila part | 03:02 |
| tbarron | bswartz: ok, mine has been unidirectional up to this point, from guest to hypervisor | 03:02 |
| bswartz | tbarron: it's is so much uglier than what you describe | 03:02 |
| tbarron | :D | 03:02 |
| bswartz | I create a bogus neutron network and use that | 03:02 |
| bswartz | then I effectively install a backdoor to that network with the internal port | 03:03 |
| bswartz | all of the configuration is manual | 03:03 |
| bswartz | in fact I doubt this could be automated | 03:03 |
| tbarron | I don't care if it's static routes and tunnels for current purpose - purpose being basically an existence proof. | 03:03 |
| bswartz | but it achieves something I cannot achieve with neutron alone which is actual network connectivityu | 03:04 |
| tbarron | however the ipv6 net is stitched together is ok for current purposes | 03:04 |
| tbarron | BUT export from hypervisor to tenant via ipv6 has to work, and it doesn't currently. | 03:05 |
| bswartz | tbarron: I'm going to tear this down and rebuilt it using ubuntu | 03:05 |
| bswartz | then I'll test some mounts | 03:05 |
| *** tommylikehu has quit IRC | 03:05 | |
| bswartz | tbarron: can you share the steps you're using for the manila part? | 03:05 |
| tbarron | using ubuntu for your guest you mean | 03:05 |
| bswartz | how to create the share w/ ipv6 support and export it? | 03:05 |
| bswartz | yes using an ubuntu guest | 03:05 |
| bswartz | cirros is having more that a few issues | 03:06 |
| bswartz | s/that/than/ | 03:06 |
| bswartz | tbarron: also here's a pro-tip -- "Q_USE_SECGROUP=False" in your local.conf disables the neutron security groups apparatus | 03:06 |
| tbarron | bswartz just reconfigure lvm with ipv6 address from hypervisor instead of ipv4 address from hypervisor as lvm_export_ip | 03:08 |
| tbarron | create lvm share | 03:08 |
| tbarron | allow access to the ipv6 address acquired by your guest/nova-instance | 03:08 |
| tbarron | or any ipv6 address for that matter :( but that's what I did | 03:09 |
| bswartz | tbarron: did you do anything w/ share types? | 03:09 |
| tbarron | then run 'showmount -e' or 'exportfs ' | 03:09 |
| tbarron | it isn't exported | 03:09 |
| bswartz | okay I guess the share type won't matter because the scheduler only has 1 choice | 03:10 |
| tbarron | bswartz: I made a default share type with DHSS=False and only have (2) lvm backends, both with the ipv6 lvm_export_ip | 03:10 |
| *** tommylikehu has joined #openstack-manila | 03:10 | |
| tbarron | so as you say it doesn't matter | 03:10 |
| bswartz | okay so I need to modify my network config a bit more to have a predictable ipv6 address to export from | 03:11 |
| tbarron | you can just edit manila.conf and restart services after the ipv6 addr on the hypervisor is acquired | 03:11 |
| bswartz | tbarron: that's cheating | 03:12 |
| bswartz | :-p | 03:12 |
| tbarron | exactly | 03:12 |
| tbarron | I'm not bulding a plugin for devstack atm | 03:12 |
| tbarron | I'm trying (failing) to prove that the ipv6 exports/access works | 03:13 |
| tbarron | two issues: | 03:14 |
| tbarron | 1) ipv6 exportfs command fails in xenial, https://lists.debian.org/debian-kernel/2016/12/msg00189.html | 03:15 |
| tbarron | 2) our access helper for NFS doesn't know that it fails | 03:16 |
| tbarron | well, 3 issues | 03:16 |
| tbarron | 3) we don't put [] around the ipv6 addr when we give it to exportfs command in the NFS helper | 03:16 |
| tbarron | fixing #3 isn't sufficient to fix #1, sadly. | 03:16 |
| bswartz | !!! | 03:17 |
| openstack | bswartz: Error: "!!" is not a valid command. | 03:17 |
| tbarron | but it makes the exportfs command work on fedora/centos/rhel (but not xenial) | 03:17 |
| * bswartz slaps openstack around a bit with a large trout | 03:17 | |
| bswartz | okay this is news to me | 03:17 |
| tbarron | bswartz: none of #1-3 are openstack | 03:17 |
| tbarron | well, #2 and #3 are manila, therefore openstack | 03:17 |
| bswartz | so there's a bug upstream in exportfs, which is fixed, but the fix hasn't been pulled into xenial yet? | 03:18 |
| bswartz | gah | 03:19 |
| bswartz | the lvm_share_export_ip value is set in the devstack plugin isn't it? | 03:19 |
| tbarron | well I just found these out tonight - updated https://review.openstack.org/#/c/406776/ with some findings | 03:20 |
| bswartz | hmmm might be able to use HOST_IP to override this | 03:20 |
| bswartz | never tried an ipv6 address for HOST_IP -- should be fun | 03:20 |
| tbarron | w.r.t. bug upstream, fix not in xenial: reasonable inference but I don't know for sure. | 03:20 |
| bswartz | tbarron: well as long as it's fixed upstream, it should be reasonably trivial to download and compile our own exportfs | 03:21 |
| tbarron | bswartz: there's a IPV6 equivalent of host ip | 03:21 |
| bswartz | there is? | 03:21 |
| tbarron | bswartz: but I haven't used it, I just cheated | 03:21 |
| bswartz | what is the variable called? | 03:21 |
| * tbarron looks around | 03:22 | |
| bswartz | the problem is that the manila devstack plugin is hardcoded to use HOST_IP for the LVM driver's export IP | 03:22 |
| tbarron | that can be fixed/retrofitted post feature freeze ... | 03:22 |
| bswartz | it's HOST_IPV6 | 03:22 |
| bswartz | yes I know but this irks me | 03:22 |
| tbarron | like the exportfs nfs-kernel fix | 03:23 |
| bswartz | ZFS has a nice driver specific devstack config variable | 03:23 |
| bswartz | called MANILA_ZFSONLINUX_SHARE_EXPORT_IP | 03:23 |
| tbarron | and we could likely post-FF fix the manila bugs: | 03:24 |
| tbarron | (1) lack of check in NFS access helper that exportfs command actually succeeded (shell $? 0 not sufficient) | 03:25 |
| bswartz | tbarron: not disagreement but we have a super compressed RC cycle this time | 03:25 |
| bswartz | 1 week from FF to RC1 target | 03:25 |
| bswartz | 2 weeks from RC1 target to final RC | 03:25 |
| tbarron | (2) format ipv6 properly when handing to exportfs command | 03:25 |
| bswartz | 1 week from final RC to release | 03:25 |
| tbarron | yeah, I'm just thinking out loud. | 03:25 |
| bswartz | and that release happens while we're all in atlanta goofing off ^H^H^H^H^H^ I mean working | 03:25 |
| tbarron | I think some solid work has been done but don't know that we're really in a position to test it end-to-end in ocata | 03:26 |
| tbarron | but most of the infra could probably be merged without necessarily exposing it to end users as an ocata feature | 03:27 |
| tbarron | just food for thought | 03:27 |
| tbarron | tommylikehu: is there any reason that anyone would want to expose ipv6 access rules if we can't actually do mounts using them? | 03:28 |
| bswartz | tbarron: for most features I would say no, but ipv6 is not really a feature it's more like a giant bugfix | 03:29 |
| bswartz | ipv6 support shouldn't be optional | 03:29 |
| bswartz | so gradually fixing the bug makes sense | 03:30 |
| bswartz | I can imagine scenarios where vendors backport ipv6 support downstream | 03:30 |
| bswartz | and we'll be glad we have support in the core as part of ocata | 03:30 |
| bswartz | worse case, it doesn't work but having the code merged speeds up progress in pike | 03:30 |
| tbarron | yeah, that's why I favor trying to merge the good work but I am | 03:31 |
| tbarron | concerned with exposing the access list stuff yet | 03:31 |
| tbarron | maybe that's just a matter of the python manilaclient | 03:31 |
| tbarron | and not advertising any driver support until we have one working end-to-end | 03:32 |
| tommylikehu | tbarron, bswartz, I am likely to agree with that we keep working on this patches (maybe merge the framwork code) and only anounce we support this when at least one driver for dhss false and another for dhss true :( | 03:33 |
| bswartz | yeah I don't intend to announce that manila have ipv6 support after ocata ships | 03:33 |
| bswartz | I plan to characterize it as experimental ipv6 support | 03:33 |
| * tbarron rests easier | 03:36 | |
| tommylikehu | tbarron, bswartz So we would like merge the framework(not the client) in Ocata. if only the mount share bug exists? | 03:37 |
| bswartz | tommylikehu: I want to merge as much as we can, so long as (1) it doesn't break anything (2) we can prove it works, and (3) we can test that future changes don't break the new functionality | 03:38 |
| *** gcb has joined #openstack-manila | 03:38 | |
| tommylikehu | bswartz: ok, I think it's a bad idea for me to split this feature into two patches, so merge these two into one would be useful before we can continue review and test on it? | 03:40 |
| tbarron | tommylikehu: I actually liked having the plugin patch and the access list patch separate, I wish others had done that kind of thing | 03:41 |
| bswartz | I'd prefer not to split it unless part of the functionality was buggy or broken or had no tests | 03:41 |
| tbarron | oh, you are talking about splitting one of the existing patches | 03:41 |
| tommylikehu | bswartz: not split but merge the existed two patches | 03:42 |
| tbarron | tommylikehu: I had now problem cherry picking both to test | 03:42 |
| tbarron | no problem | 03:42 |
| tommylikehu | ok | 03:43 |
| tbarron | and I honestly think they are easier to code review separately | 03:43 |
| tbarron | we had way too many humongous patches | 03:43 |
| tbarron | it slowed me down reviewing | 03:43 |
| tommylikehu | tbarron: thanks | 03:43 |
| tbarron | bswartz: tommylikehu I have an early meeting tomorrow so I'm going to go to bed. Will look with fresh eyes tomorrow and will of course check the backlog here if you two discover anything interesting. | 03:54 |
| bswartz | tbarron: np | 03:55 |
| *** tommylikehu_ has joined #openstack-manila | 04:26 | |
| tommylikehu | tbarron: np | 04:27 |
| *** tommylikehu_ has quit IRC | 04:30 | |
| *** gcb has quit IRC | 04:43 | |
| *** gcb has joined #openstack-manila | 04:44 | |
| *** cknight has joined #openstack-manila | 04:44 | |
| *** ianychoi has joined #openstack-manila | 04:55 | |
| openstackgerrit | yankee proposed openstack/manila: Allow use of <is> with extra_spec_value in share type https://review.openstack.org/413338 | 05:23 |
| *** lpetrut has joined #openstack-manila | 05:33 | |
| *** cknight has quit IRC | 05:34 | |
| *** shausy has joined #openstack-manila | 05:34 | |
| *** kaisers has joined #openstack-manila | 06:06 | |
| *** ianychoi has quit IRC | 06:15 | |
| *** lpetrut has quit IRC | 06:45 | |
| *** makowals_ has quit IRC | 06:53 | |
| *** makowals has joined #openstack-manila | 07:06 | |
| *** jprovazn has joined #openstack-manila | 07:14 | |
| *** jprovazn has quit IRC | 07:26 | |
| *** jprovazn has joined #openstack-manila | 07:26 | |
| *** pcaruana has joined #openstack-manila | 07:28 | |
| openstackgerrit | Merged openstack/manila: [Tempest] Add functional tests for share groups feature https://review.openstack.org/355264 | 07:49 |
| openstackgerrit | TommyLike proposed openstack/manila: Enable IPv6 in manila(network plugins and drivers) https://review.openstack.org/406776 | 07:57 |
| *** kaisers_ has joined #openstack-manila | 08:07 | |
| *** kaisers has quit IRC | 08:11 | |
| *** ianychoi has joined #openstack-manila | 08:12 | |
| *** zengyingzhe has quit IRC | 08:20 | |
| openstackgerrit | TommyLike proposed openstack/manila: Enable IPv6 in manila(documentation) https://review.openstack.org/416870 | 08:27 |
| *** tommylikehu_ has joined #openstack-manila | 08:27 | |
| *** tommylikehu_ has quit IRC | 08:32 | |
| *** lpetrut has joined #openstack-manila | 08:42 | |
| *** lpetrut has quit IRC | 08:42 | |
| *** lpetrut has joined #openstack-manila | 08:44 | |
| tommylikehu | Hey tbarron, bswartz, rebuilded the path for exportfs command with IPvt6 friendly format and setting up new test environment now. | 08:44 |
| *** dsariel has joined #openstack-manila | 08:50 | |
| *** a-pugachev_ has joined #openstack-manila | 08:58 | |
| *** dsariel has quit IRC | 09:28 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/manila: Updated from global requirements https://review.openstack.org/424616 | 09:29 |
| *** ganso has joined #openstack-manila | 09:43 | |
| openstackgerrit | Rodrigo Barbieri proposed openstack/manila: Mountable snapshots scenario tests https://review.openstack.org/412001 | 09:49 |
| *** yumiriam has joined #openstack-manila | 09:59 | |
| *** alyson_ has joined #openstack-manila | 10:06 | |
| *** openstackgerrit has quit IRC | 10:17 | |
| *** ociuhandu has joined #openstack-manila | 10:22 | |
| *** dsariel has joined #openstack-manila | 10:37 | |
| *** cdelatte has quit IRC | 11:23 | |
| *** ociuhandu has quit IRC | 11:32 | |
| ganso | vponomaryov, bswartz, tbarron, markstur: we can merge https://review.openstack.org/#/c/412001/ and https://review.openstack.org/#/c/345625/ | 11:38 |
| vponomaryov | ganso: 2 comments in scenario tests | 11:56 |
| vponomaryov | ganso 3 | 11:57 |
| ganso | vponomaryov: thanks | 12:00 |
| vponomaryov | ganso: also, don't see scenario tests runing in your CI | 12:00 |
| ganso | vponomaryov: they don't run | 12:01 |
| vponomaryov | ganso: I was thinking you do run them | 12:01 |
| ganso | vponomaryov: no, not yet | 12:01 |
| vponomaryov | ganso: so, only LVM is running this test, right? | 12:01 |
| ganso | vponomaryov: correct | 12:01 |
| vponomaryov | ok | 12:01 |
| vponomaryov | ganso: also, don't you want to rebase your driver implementation - https://review.openstack.org/#/c/411474/ ? | 12:02 |
| vponomaryov | ganso: scenario test can wait | 12:03 |
| ganso | vponomaryov: alyson_ is already working on it | 12:03 |
| vponomaryov | ok | 12:03 |
| *** openstackgerrit has joined #openstack-manila | 12:09 | |
| openstackgerrit | Merged openstack/manila: Fix MapRFS test__execute to not impact others https://review.openstack.org/424932 | 12:09 |
| *** cdelatte has joined #openstack-manila | 12:10 | |
| *** porrua has joined #openstack-manila | 12:11 | |
| *** catintheroof has joined #openstack-manila | 12:28 | |
| *** tommylikehu_ has joined #openstack-manila | 12:29 | |
| *** tommylikehu_ has quit IRC | 12:29 | |
| *** tommylikehu_ has joined #openstack-manila | 12:30 | |
| openstackgerrit | Merged openstack/python-manilaclient: Add mountable snapshots support to manila client https://review.openstack.org/345625 | 12:30 |
| openstackgerrit | Rodrigo Barbieri proposed openstack/manila: Mountable snapshots scenario tests https://review.openstack.org/412001 | 12:40 |
| vponomaryov | ganso: https://review.openstack.org/#/c/412001/21..22/manila_tempest_tests/tests/scenario/test_share_basic_ops.py , line 563 | 12:46 |
| openstackgerrit | Miriam Yumi proposed openstack/manila-ui: Add mountable snapshots support to manila-ui https://review.openstack.org/415274 | 12:46 |
| vponomaryov | ganso: it is very-very unrecommended to do exactly so | 12:46 |
| -openstackstatus- NOTICE: Gerrit is going to be restarted due to slow performance | 12:46 | |
| vponomaryov | ganso: you catch ANY exception, hide it and make test pass | 12:46 |
| vponomaryov | ganso: you should catch only execution error, not timeout, not python exception | 12:47 |
| vponomaryov | ganso: and would be good to make "LOG.debug(e)" | 12:47 |
| ganso | vponomaryov: will look into it for next patch... btw, migration test is doing exactly the same | 12:47 |
| vponomaryov | ganso: it is very error-prone | 12:48 |
| -openstackstatus- NOTICE: Gerrit has been successfully restarted | 12:49 | |
| vponomaryov | ganso: and yes, "self.assertRaises(" is your friend here )) | 12:51 |
| vponomaryov | ganso: haha, looked who implemented it in migration test )) | 12:52 |
| ganso | vponomaryov: yes, I know | 12:53 |
| ganso | bswartz: ping | 12:54 |
| *** ociuhandu has joined #openstack-manila | 13:26 | |
| *** tongli has joined #openstack-manila | 13:27 | |
| *** shausy has quit IRC | 13:30 | |
| *** xyang1 has joined #openstack-manila | 13:39 | |
| *** timcl has joined #openstack-manila | 13:43 | |
| *** tommylikehu_ has quit IRC | 13:49 | |
| openstackgerrit | yankee proposed openstack/manila: Allow use of <is> with extra_spec_value in share type https://review.openstack.org/413338 | 14:12 |
| *** Yogi1 has joined #openstack-manila | 14:22 | |
| *** yankee_ has joined #openstack-manila | 14:23 | |
| *** dustins has joined #openstack-manila | 14:24 | |
| openstackgerrit | Rodrigo Barbieri proposed openstack/manila: Mountable snapshots scenario tests https://review.openstack.org/412001 | 14:29 |
| bswartz | ganso: had another late night last night -- just got into the office | 14:30 |
| *** gouthamr has joined #openstack-manila | 14:30 | |
| ganso | bswartz: Hi Ben, can we please have another python-manilaclient release? | 14:31 |
| ganso | bswartz: mountable snapshots manila-ui patch needs that | 14:31 |
| bswartz | ganso: in addition the one I'm doing tomorrow? | 14:31 |
| ganso | bswartz: I think so | 14:32 |
| ganso | bswartz: it will be too late to do it tomorrow | 14:32 |
| bswartz | ganso okay here it goes! | 14:32 |
| ganso | bswartz: because manila-ui patch has to merge | 14:32 |
| vponomaryov | ganso: https://review.openstack.org/#/c/412001/22..23/manila_tempest_tests/tests/scenario/test_share_basic_ops.py )) | 14:41 |
| ganso | vponomaryov: oh noes | 14:42 |
| ganso | vponomaryov: stupid IDE | 14:42 |
| *** tongli has quit IRC | 14:42 | |
| *** gcb has quit IRC | 14:42 | |
| vponomaryov | ganso: which? )0 | 14:42 |
| openstackgerrit | Rodrigo Barbieri proposed openstack/manila: Mountable snapshots scenario tests https://review.openstack.org/412001 | 14:42 |
| ganso | vponomaryov: pycharm | 14:43 |
| vponomaryov | ganso: don't tell it to cknight | 14:43 |
| bswartz | ganso: https://review.openstack.org/425212 | 14:43 |
| bswartz | pycharm has gotten a lot better | 14:43 |
| bswartz | it's way better than pydev | 14:43 |
| *** yankee_ has quit IRC | 14:45 | |
| ganso | bswartz: thanks | 14:45 |
| *** cknight has joined #openstack-manila | 14:50 | |
| ganso | bswartz: just a heads up, lvm job is a bit unstable | 14:51 |
| ganso | bswartz: depending on the node it is running, it ends up timing out | 14:51 |
| ganso | bswartz: it is running a lot of tests now | 14:51 |
| ganso | bswartz: mountable, revert, scenario | 14:51 |
| bswartz | ganso: is there something especially slow about mountable snapshots or revert tests? | 14:52 |
| ganso | bswartz: well, they have a lot of steps... create share, create a snapshot, revert or add rules | 14:52 |
| ganso | bswartz: then delete the rules, delete the snapshot, delete the share | 14:52 |
| bswartz | right but do any of those steps go outside the manila RPC layer/database/service code? | 14:53 |
| ganso | bswartz: the faster I've seen it run is 49 minutes | 14:53 |
| ganso | bswartz: but depending on the node it takes more than 1h07m | 14:53 |
| ganso | bswartz: which times out | 14:53 |
| bswartz | I can understand actual LVM operations taking some time because that's I/O | 14:53 |
| bswartz | yeah the timeout is 65 minutes IIRC | 14:54 |
| ganso | bswartz: I have not seen problems regarding RPC layer/database/service code | 14:54 |
| *** catinthe_ has joined #openstack-manila | 15:00 | |
| *** catintheroof has quit IRC | 15:02 | |
| *** eharney has joined #openstack-manila | 15:06 | |
| openstackgerrit | Rodrigo Barbieri proposed openstack/manila: Improve share migration scenario test validation https://review.openstack.org/425225 | 15:06 |
| ganso | vponomaryov: ^ | 15:06 |
| openstackgerrit | Tom Barron proposed openstack/python-manilaclient: Enable IPv6 in manila(client) https://review.openstack.org/328932 | 15:07 |
| bswartz | tbarron: I'm looking more closely at the exportfs issue now | 15:12 |
| bswartz | tbarron: which package actually contained the bug? nfs-utils? nfs-kernel-server? | 15:12 |
| tbarron | bswartz: here's what I've got: https://lists.debian.org/debian-kernel/2016/12/msg00189.html | 15:13 |
| tbarron | bswartz: which suggests that it is nfs-kernel-server | 15:13 |
| bswartz | yeah I read that whole message in detail this morning | 15:14 |
| bswartz | what made you think fedora has a fix for this while ubuntu does not? | 15:14 |
| tbarron | bswartz: I haven't done any more than reproduce manually and verify that I don't hit it when on fedora/centos | 15:14 |
| bswartz | ah, good old empirical evidence | 15:15 |
| tbarron | bswartz: just empirical at this point. If I run the exportfs commands on fedora/centos and follow with plain 'exportfs' or 'showmount -e' I see the ipv6 exports | 15:15 |
| tbarron | bswartz: I do have to put the square brackets around the address though | 15:15 |
| tbarron | bswartz: but it looks like tommylikehu is doing that now in his latest patch | 15:16 |
| openstackgerrit | Miriam Yumi proposed openstack/manila-ui: Add mountable snapshots support to manila-ui https://review.openstack.org/415274 | 15:19 |
| tbarron | ganso: now that I'm playing with lvm driver for the ipv6 testing/exploration I notice those lvm test failures in gate too. | 15:23 |
| tbarron | ganso: I haven't checked but we should see how long we're holding locks and there's non-trivial concurrency | 15:24 |
| ganso | tbarron: yes | 15:24 |
| tbarron | ganso: just brainstorming, or maybe there's non-manila serialization going on with lvm operations | 15:25 |
| tbarron | ganso: if the bottleneck is outside of manila I'm not really concerned since lvm is not a scalable driver for cloud anyways | 15:26 |
| *** mtanino has joined #openstack-manila | 15:30 | |
| ganso | tbarron: there are some failures that are not timeouts | 15:34 |
| ganso | tbarron: so I believe there are race conditions in there as well | 15:34 |
| tbarron | ganso: good material for the eliminate race conditions work ... | 15:35 |
| bswartz | argh | 15:38 |
| bswartz | death to race conditions | 15:38 |
| openstackgerrit | Alyson proposed openstack/manila: Add mountable snapshots support to HNAS driver https://review.openstack.org/411474 | 15:41 |
| *** catintheroof has joined #openstack-manila | 15:43 | |
| *** catinthe_ has quit IRC | 15:46 | |
| *** a-pugachev_ has quit IRC | 15:50 | |
| ganso | tbarron: http://logs.openstack.org/16/424616/6/check/gate-manila-tempest-minimal-dsvm-lvm-ubuntu-xenial/9aec2c9/logs/screen-m-shr.txt.gz?level=TRACE | 15:59 |
| ganso | tbarron: doesn't look like a race condition because the lvm resource is unique to that test class, which does not run tests in parallel | 15:59 |
| openstackgerrit | Alyson proposed openstack/manila: Add mountable snapshots support to HNAS driver https://review.openstack.org/411474 | 16:00 |
| ganso | tbarron: I am not familiar enough with LVM to understand why that one failed... it seems that the lvconvert and following lvchange operations succeeded but for some reason the lv was still there, so it failed to re-create the snapshot | 16:00 |
| tbarron | ganso: offhand looks like we may not be waiting for the driver to be initialized and then there are a cascade of failures? Is that log from the beginning of the tempest run? | 16:04 |
| tbarron | DriverNotInitialized: Share driver 'LVMShareDriver' not initialized. | 16:05 |
| ganso | tbarron: If I am not mistaken, that error DriverNotInitialized always shows up | 16:06 |
| ganso | tbarron: even on successful runs | 16:06 |
| tbarron | ganso: yeah, once at the beginning I guess | 16:07 |
| tbarron | ganso: and time passes after that before the next ERROR | 16:08 |
| tbarron | ganso: weird that we see nothing in between before "Recovering from a failed execute. Try number 1" | 16:09 |
| tbarron | ganso: we don't see a processutils log for the operation that is being retried. | 16:10 |
| tbarron | ganso: and the retried operation is a snapshot creation and we never see the share resource being created | 16:10 |
| *** adrianofr_ has joined #openstack-manila | 16:23 | |
| openstackgerrit | Merged openstack/manila: Fix devstack plugin to not depend on private network https://review.openstack.org/424685 | 16:25 |
| *** lpetrut has quit IRC | 16:26 | |
| ganso | tbarron: exactly | 16:29 |
| ganso | tbarron: I've seen this kind of error before in LVM runs | 16:29 |
| ganso | tbarron: although, very rare | 16:29 |
| ganso | tbarron: now they are a lot more frequent | 16:30 |
| tbarron | ganso: well, let's keep an eye out and see if it's really this same pattern over and over; it's a weird one. We may have to instrument the code better. | 16:30 |
| tbarron | ganso: a whole lot of big changes have landed in the last few days :D | 16:31 |
| tbarron | ganso: all that migration stuff e.g. | 16:32 |
| * tbarron janks ganso's chain | 16:32 | |
| tbarron | ganso: actually more likely snapshot related stuff given that log | 16:32 |
| markstur | oh snap! | 16:37 |
| markstur | this should've been called the osnap release | 16:37 |
| *** dsariel has quit IRC | 16:44 | |
| ganso | tbarron: yes | 16:47 |
| ganso | markstur: lol | 16:47 |
| tbarron | markstur: :-) | 16:48 |
| ganso | markstur: in Boston you can say "oh look now there are several new ways you can interact with your manila snapshots" | 16:48 |
| * markstur does that snap, snap, snap gesture | 16:49 | |
| ganso | markstur: then your customer will say "isn't this redundant with the create_share_from_snapshot we previously had?" | 16:49 |
| markstur | ganso: then the customer will say " and don't get me started on CGs..." | 16:50 |
| ganso | markstur: lol | 16:50 |
| ganso | gouthamr: ^ | 16:50 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack/manila: [DNM] test 1 https://review.openstack.org/425282 | 16:51 |
| * gouthamr wakes up | 16:51 | |
| gouthamr | what are CGs lol | 16:51 |
| tbarron | they are so yesterday .., | 16:52 |
| markstur | ganso, gouthamr: Maybe in Boston you can do the "Ins and outs of EXPERIMANTAL features with manila" | 16:52 |
| gouthamr | tbarron: never happened :) | 16:52 |
| tbarron | dustin is working on a generic share group test plan | 16:52 |
| ganso | gouthamr, markstur: yea, never heard of them... AFAIK manila never had CGs <runs away> | 16:52 |
| tbarron | and we're trying to blink the men in black's flashlight in our product manager's eyes | 16:53 |
| ganso | tbarron: lol | 16:53 |
| markstur | tbarron: ooooh. Do you have one of those? | 16:53 |
| markstur | tbarron: I thought you just had regular amnesia | 16:53 |
| markstur | like my benevolent brain cloud | 16:54 |
| tbarron | markstur: i can't quite remember but I think I have three of them for redundancy. They establish quorum and vote as to who will actually do the blinking deed. | 16:55 |
| ganso | tbarron: good thing they don't have race conditions, else it could be a double blinking | 16:56 |
| *** timcl has quit IRC | 16:56 | |
| tbarron | ganso: at least the blinks are idempotent | 16:57 |
| gouthamr | ganso: ours come pre-packaged with tooz | 16:57 |
| tbarron | gouthamr: just install tooz and then you don't ever have to worry about race conditions again | 16:57 |
| gouthamr | :D | 16:58 |
| *** timcl has joined #openstack-manila | 16:58 | |
| * gouthamr puts tooz decorators on every method he writes. customer switches back to raising IT tickets to get NFS share. customers can be stupid. | 16:59 | |
| *** a-pugachev has joined #openstack-manila | 17:00 | |
| openstackgerrit | Alyson proposed openstack/manila: Add mountable snapshots support to HNAS driver https://review.openstack.org/411474 | 17:20 |
| ganso | bswartz: now we gotta wait for 1.13 to appear here? https://launchpad.net/python-manilaclient | 17:21 |
| *** rushil has joined #openstack-manila | 17:24 | |
| bswartz | ganso: https://pypi.python.org/pypi/python-manilaclient/1.13.0 | 17:26 |
| ganso | bswartz: 404 | 17:26 |
| bswartz | ganso: when 404 changes to 200 then it's done | 17:26 |
| ganso | bswartz: oh ok | 17:26 |
| ganso | bswartz: thanks | 17:27 |
| *** timcl has quit IRC | 17:35 | |
| *** rushil has quit IRC | 17:45 | |
| bswartz | ganso: try now | 17:47 |
| ganso | bswartz: yea I saw it | 17:48 |
| ganso | bswartz: waiting for the manila-ui job to finish to recheck it | 17:48 |
| *** lpetrut has joined #openstack-manila | 17:49 | |
| *** timcl has joined #openstack-manila | 17:53 | |
| *** timcl has quit IRC | 17:55 | |
| ganso | markstur: replied to your comment | 18:00 |
| markstur | ganso: I was going to ask you about it here, but decided it was not an issue | 18:04 |
| markstur | or not likely | 18:04 |
| ganso | markstur: anything holding back a +W? | 18:11 |
| markstur | oh. Jenkins and meetings. I'll go look again. | 18:11 |
| ganso | markstur: nvm just got +W'ed | 18:12 |
| markstur | things move fast around here (sometimes) | 18:13 |
| gouthamr | when i'm back at work :) | 18:13 |
| bswartz | markstur: it's feature freeze week | 18:15 |
| markstur | It isn't always about your gouthamr | 18:16 |
| markstur | s/your/you/ | 18:16 |
| bswartz | if things don't move fast we're toast | 18:16 |
| *** timcl has joined #openstack-manila | 18:18 | |
| gouthamr | :P | 18:24 |
| *** rushil has joined #openstack-manila | 18:25 | |
| *** a-pugachev has quit IRC | 18:32 | |
| *** catinthe_ has joined #openstack-manila | 18:33 | |
| *** timcl has quit IRC | 18:35 | |
| *** catintheroof has quit IRC | 18:35 | |
| *** timcl has joined #openstack-manila | 18:36 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack/manila: Revert "[Devstack] Workaround osclient breakage" https://review.openstack.org/425333 | 18:36 |
| tbarron | vponomaryov: was going to propose that if I didn't see you do it ^^ :D | 18:37 |
| vponomaryov | tbarron: I expect it to fail now, just need to have such to poke from time to time | 18:38 |
| bswartz | vponomaryov: why do we have a timeout for security group creation? | 18:38 |
| vponomaryov | bswartz: I barely remember that it was working from some other kick other than first | 18:39 |
| *** rushil has quit IRC | 18:51 | |
| *** pcaruana has quit IRC | 18:53 | |
| ganso | vponomaryov, bswartz: shouldn't it have downloaded python-manilaclient 1.13 if it is available in pypi? http://logs.openstack.org/74/415274/5/check/gate-manila-ui-python27-ubuntu-xenial/7a93047/console.html | 18:53 |
| bswartz | ganso: it should download what's in uc.txt | 18:54 |
| bswartz | 1 sec | 18:54 |
| vponomaryov | ganso: https://github.com/openstack/requirements/blob/master/upper-constraints.txt | 18:54 |
| bswartz | ganso: https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L326 | 18:54 |
| ganso | bswartz: I thought that yesterday we had bumped the minimal version only, but if a newer was available it would use it | 18:55 |
| ganso | vponomaryov, bswartz: :\ | 18:55 |
| vponomaryov | ganso: we should wait until either bot automatically updates upper constraint or some of us | 18:55 |
| ganso | vponomaryov, bswartz: why do we have an upper constraint for master? | 18:55 |
| ganso | vponomaryov: if we want to freeze asap we need to do it ourselves I guess | 18:56 |
| vponomaryov | ganso: because people tend to make mistakes )) | 18:56 |
| bswartz | ganso: to prevent automatic breakage when clowns push a new version of something that doesn't work | 18:56 |
| vponomaryov | clowns ^_^ | 18:56 |
| ganso | bswartz: yea but python-manilaclient is owned by us | 18:56 |
| bswartz | uc.txt is the "highest known good" version of stuff | 18:56 |
| vponomaryov | ganso: haven't you seen red nose on one of us? )) | 18:57 |
| tbarron | vponomaryov: beat me to it | 18:57 |
| ganso | vponomaryov: not really | 18:57 |
| * tbarron clowns around a bit | 18:57 | |
| bswartz | ganso: we could modify our workflows to test against HEAD of master branch, but then we'd possibly release manila-ui versions that didn't work for real users | 18:58 |
| bswartz | the interlock of different projects and their dependent versions is a bit slow but I think it's intentional | 18:59 |
| bswartz | libraries are not supposed to move fast | 18:59 |
| bswartz | the real problem here is that client shouldn't be treated as libraries -- they're really not | 18:59 |
| openstackgerrit | Merged openstack/manila: Mountable snapshots scenario tests https://review.openstack.org/412001 | 19:01 |
| ganso | bswartz: well, I guess that for now, we need to bump the uc.txt then | 19:01 |
| *** FL1SK has quit IRC | 19:01 | |
| bswartz | ganso: https://review.openstack.org/#/c/425310/ | 19:01 |
| bswartz | the bot is way ahead of you | 19:02 |
| *** yumiriam has quit IRC | 19:02 | |
| *** rushil has joined #openstack-manila | 19:04 | |
| *** draynium has quit IRC | 19:05 | |
| *** ociuhandu has quit IRC | 19:07 | |
| ganso | bswartz: whoa thanks | 19:09 |
| vponomaryov | bswartz: our generic driver DHSS=False job is broken becaues of this bug -> https://bugs.launchpad.net/nova/+bug/1659391 | 19:12 |
| openstack | Launchpad bug 1659391 in OpenStack Compute (nova) "Server list API does not show existing servers" [Undecided,New] | 19:12 |
| vponomaryov | bswartz: it was broken about 26 hours ago | 19:12 |
| bswartz | that's weird | 19:13 |
| *** draynium has joined #openstack-manila | 19:13 | |
| bswartz | I haven't seen that breakage yet in my testing | 19:14 |
| vponomaryov | bswartz: because you should have used latest nova and generic driver with DHSS=False mode | 19:14 |
| bswartz | I am using latest nova | 19:14 |
| vponomaryov | to see devstackerror in manila plugin or just latest nova to see error manually listing servers | 19:14 |
| bswartz | I guess not listing vms as admin so often | 19:15 |
| bswartz | maybe it was 2 days ago | 19:15 |
| vponomaryov | bswartz: I can reproduced it on my lab | 19:15 |
| vponomaryov | bswartz: why ask? just look at commit history | 19:15 |
| vponomaryov | bswartz: here commit you need -> https://github.com/openstack/nova/commit/5555ecc0 | 19:16 |
| bswartz | I just can't remember how recently I tried that exact operation | 19:17 |
| bswartz | i've been fighting with neutron too much | 19:18 |
| bswartz | now nova is bothering me but for a different reason | 19:18 |
| *** esker has joined #openstack-manila | 19:21 | |
| bswartz | okay starting another test | 19:38 |
| *** jprovazn has quit IRC | 19:46 | |
| *** rushil has quit IRC | 19:48 | |
| *** rushil has joined #openstack-manila | 19:48 | |
| *** draynium has quit IRC | 19:59 | |
| *** draynium has joined #openstack-manila | 20:06 | |
| *** esker has quit IRC | 20:08 | |
| bswartz | well I give up | 20:17 |
| bswartz | why must there be so many problems with ipv6 and nova? | 20:18 |
| bswartz | (ノಠ益ಠ)ノ彡┻━┻ | 20:19 |
| *** esker has joined #openstack-manila | 20:20 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/manila: Updated from global requirements https://review.openstack.org/424616 | 20:22 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/manila-ui: Updated from global requirements https://review.openstack.org/425390 | 20:22 |
| tbarron | bswartz: dare I ask what ipv6 nova issues you are hitting? | 20:27 |
| *** catinthe_ has quit IRC | 20:27 | |
| bswartz | tbarron: when I attempt to run my dhcp6 client in my VM the whole network interface goes out to lunch | 20:27 |
| bswartz | I lose ipv4 access too | 20:27 |
| bswartz | also the metadata services hasn't work in days | 20:28 |
| bswartz | and the config-drive is a joke | 20:28 |
| tbarron | hmm, with centos I did the following cheat: 1) specify nic=$NETID on the cmd line for 'openstack server create ...' | 20:29 |
| tbarron | 2) it came up with ipv4 addr only | 20:29 |
| bswartz | so from what I can tell there's no way to set an IPv6 address on my VM's ethernet interface | 20:29 |
| bswartz | tbarron: I always do that | 20:30 |
| tbarron | 3) edit /etc/sysconfig/network-scripts/ifcfg-eth0 (/etc/network/interfaces for debian derivative ...) to tell it to enable IPv6 and do dhcp6 insteadd of slaac | 20:30 |
| tbarron | 4) reboot | 20:30 |
| bswartz | when I do ^ this it locks up hard | 20:30 |
| bswartz | disabling ipv6 in the guest allows everything to work fine | 20:30 |
| tbarron | It then acquired an ipv6 addr on eth0 from the cidr/subnet for that net | 20:31 |
| tbarron | bswartz: so I don't think that's a nova issue | 20:31 |
| bswartz | so you've witnessed dhcpv6 working? | 20:31 |
| tbarron | bswartz: yes | 20:31 |
| bswartz | that's comforting | 20:31 |
| bswartz | it's just a bswartz problem then | 20:31 |
| tbarron | but I get an unrouted address | 20:31 |
| tbarron | talking to some of our neutron people (I still haven't connected with folks who actually work on it) | 20:32 |
| tbarron | they say that "private" network in devstack may really be just an ipv4 artefact even though it has an ipv6 subnet too | 20:32 |
| tbarron | they suggest using a public network | 20:32 |
| tbarron | I said, but but but | 20:32 |
| tbarron | I want a tenant-owned network | 20:33 |
| bswartz | tbarron: what makes the network public? | 20:33 |
| bswartz | "public" is just a name | 20:33 |
| bswartz | as is "private" | 20:33 |
| tbarron | they said well, probably it's still a public net but uses IPv6 prefix delegation | 20:33 |
| tbarron | do 'openstack network show' on each, there's a flag I think | 20:33 |
| * tbarron looks | 20:34 | |
| bswartz | are you referring to the "provider:physical_network" field? | 20:35 |
| bswartz | because again that's just a string | 20:35 |
| tbarron | router:external ? | 20:35 |
| tbarron | just had this conversation a moment ago | 20:35 |
| bswartz | if the existence of a provider network is the key, then that's something I assumed 2 days ago and that led to the misery of monday night | 20:36 |
| tbarron | anyways, I just modified my "external" network to be shared | 20:36 |
| tbarron | and booted a vm on it | 20:36 |
| bswartz | interesting | 20:36 |
| tbarron | it acquires "public" ipv4 and ipv6 addrs | 20:37 |
| *** dschoenb has joined #openstack-manila | 20:37 | |
| *** dustins has quit IRC | 20:37 | |
| tbarron | presumably prefix delegation would somehow be used to make tenant owned versions of these | 20:37 |
| tbarron | and lots of firewalling to provide isolation | 20:37 |
| tbarron | dunno how this interacts with e.g. vxlan but put that aside for now | 20:38 |
| bswartz | tbarron: highly doubtful that prefix delegation is used | 20:38 |
| *** dschoenb is now known as dustins | 20:38 | |
| bswartz | however subnetpools are designed to do what you're describing | 20:38 |
| tbarron | bswartz: yeah, I saw those, could be | 20:38 |
| bswartz | in theory admins can make subnetpools that tenants can consume networks from, enabling tenants to create their own networks | 20:39 |
| tbarron | but anyways I was thinking for our POC purposes if we can export from hypervisor and mount from vm on so-called "public" network that may be sufficient | 20:39 |
| bswartz | vponomaryov: just did nova list --all and it worked fine | 20:40 |
| tbarron | after I made the "public" network shared I can, running as demo, create a compute instance attached to it and gain both ipv4 and ipv6 addrs | 20:40 |
| bswartz | tbarron: I will def try that | 20:41 |
| tbarron | bswartz: but i'm doing something dumb with my icmp & ssh security group rules | 20:41 |
| bswartz | for now I will give resort to more hackery | 20:41 |
| tbarron | I can't even ping/ssh this compute instance with ipv4 atm | 20:41 |
| bswartz | tbarron: did you try Q_USE_SECGROUP=False ? | 20:41 |
| tbarron | from the hypervisor | 20:41 |
| bswartz | tbarron: check your routes | 20:42 |
| tbarron | I route to br-ex and it turns me down with unreachables | 20:42 |
| tbarron | and no, I haven't tried with Q_USE_SECGROUP=False, didn't know about it | 20:42 |
| bswartz | tbarron: so the routing table says br-ex? | 20:43 |
| tbarron | br-ex makes sense as the next-hop from the hypervisor to compute instance | 20:43 |
| tbarron | routing table says IP that resides on br-ex | 20:43 |
| bswartz | if the route is correct and the host is still unreachable that means ARP (or ND) packets aren't flowing | 20:43 |
| tbarron | for both ipv4 and ipv6 | 20:43 |
| bswartz | tbarron what does "ip route get" say> | 20:43 |
| ganso | xyang1: Hi Xing, HNAS CI has voted in https://review.openstack.org/#/c/411474/ | 20:44 |
| tbarron | bswartz: 172.24.5.10 dev br-ex src 172.24.5.1 | 20:44 |
| tbarron | 172.24.5.10 is the ipv4 of the compute instance | 20:45 |
| bswartz | that doesn't sound terrible | 20:45 |
| bswartz | unless 172.24.5.0/24 is your external network for the whole hypervisor | 20:45 |
| tbarron | that sounds right, floating IPs are carved from 172.24.5.0/24 (well a pool) and they worked | 20:45 |
| xyang1 | ganso: ok | 20:45 |
| bswartz | the important thing is that the packets aren't being router to the gateway | 20:46 |
| bswartz | because routing to the gateway means the machine doesn't know where to route them | 20:46 |
| bswartz | s/router/routed/ | 20:46 |
| tbarron | it's not, my external interface on the hypervisor is a home router net at 192.168.0.0/24 | 20:46 |
| ganso | xyang1: thanks! :) | 20:47 |
| xyang1 | ganso: np:) | 20:47 |
| tbarron | bswartz: so I'm doing something dumb at this step but I think it's worth your trying to attach to the "public" network instead | 20:47 |
| *** porrua has quit IRC | 20:48 | |
| bswartz | gah! | 20:49 |
| bswartz | I think for some reason I don't have tommylikehu's patch on this system | 20:49 |
| bswartz | wait I do have it | 20:49 |
| bswartz | why am I getting "ERROR: Supported ip format examples: 10.0.0.2, 10.0.0.0/24" | 20:50 |
| tbarron | bswartz: you need three. client, plugins, and access-list | 20:50 |
| tbarron | python-client | 20:50 |
| bswartz | ah | 20:50 |
| bswartz | ty | 20:50 |
| tbarron | that's why I rebased it for him this morning :D | 20:50 |
| tbarron | for me and you really | 20:50 |
| bswartz | tbarron: "ERROR: Expected 4 octets in u'fd10::'" | 20:52 |
| bswartz | is that a server error or a client error? | 20:53 |
| bswartz | is there a second server patch I need? | 20:55 |
| tbarron | bswartz: you need both the plugin patch and the access list patch | 20:56 |
| tbarron | on the server side | 20:56 |
| bswartz | what order do you apply them? | 20:56 |
| bswartz | I'm getting a conflict between them | 20:56 |
| tbarron | i did plugin first, then access list. didn't have a conflict. | 20:56 |
| tbarron | you sure you are up to date? | 20:56 |
| bswartz | checkout or cherry-pick? | 20:56 |
| tbarron | there's a conflict with the generic driver patch but i didnt' use it. | 20:57 |
| tbarron | cherry-pick | 20:57 |
| bswartz | aha | 20:57 |
| tbarron | i cherry-picked again fresh this morning | 20:57 |
| bswartz | I was using checkout with an older version due to the merge conflict | 20:57 |
| bswartz | I never updated after the conflit was resolve | 20:57 |
| tbarron | i refreshed master, did checkout -b, then did the two cherry-picks | 20:58 |
| bswartz | yes I'm rebuilding my tree properly 1 sec | 20:58 |
| *** FL1SK has joined #openstack-manila | 21:01 | |
| bswartz | progress! | 21:01 |
| bswartz | "ERROR: The server has either erred or is incapable of performing the requested operation." | 21:01 |
| tbarron | in response to what, 'nova create ...' ? | 21:02 |
| bswartz | my database was fubar | 21:02 |
| tbarron | oh | 21:02 |
| bswartz | after pulling all that code down | 21:02 |
| tbarron | yeah, forgot to mention 'manila-manage db sync' | 21:02 |
| bswartz | omg it worked | 21:03 |
| tbarron | b/c of all our changes in the last few days | 21:03 |
| tbarron | now try to downgrade - oh that's a different tangent ... | 21:03 |
| bswartz | okay so that access rule appliee | 21:04 |
| bswartz | applied | 21:04 |
| bswartz | but I see no evidence of it | 21:04 |
| bswartz | is this the exportfs bug? | 21:04 |
| tbarron | we should see it in the dbase but (b/c of the bug) not in the exports | 21:05 |
| tbarron | we fail the exportfs silently, so there's a linux bug and a manila bug | 21:05 |
| tbarron | processutils returns 0 but stderr shows a problem | 21:05 |
| bswartz | so how can I test? | 21:05 |
| tbarron | well, that's where I am | 21:06 |
| bswartz | 2017-01-25 16:03:05.474 9153 DEBUG oslo_concurrency.processutils [req-5101dd4f-48db-442c-bd61-157e51966cc4 aabc9d7ac60c4d4ea6e90a2df2dd2ce0 7992f737ca4547c69117d3b60e4dc2d2 - - -] CMD "sudo manila-rootwrap /etc/manila/rootwrap.conf exportfs -o rw,no_subtree_check,no_root_squash [fd10::]/16:/opt/stack/data/manila/mnt/share-1d9e8df0-ff57-4e01-97e5-97b7ce41e434" returned: 0 in 0.328s execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/pr | 21:06 |
| tbarron | was thinking about building a fedora hypervised devstack | 21:06 |
| tbarron | generic driver breaks for multiple reasons in fedora but probably lvm will work (crosses fingers) | 21:06 |
| bswartz | that string looks ultra-funky | 21:07 |
| *** alyson_ has quit IRC | 21:07 | |
| bswartz | [fd10::]/16:/opt/stack/data/manila/mnt/share-1d9e8df0-ff57-4e01-97e5-97b7ce41e434 | 21:07 |
| bswartz | ^ in what universe was that correct? | 21:07 |
| tbarron | run it by hand on fedora | 21:07 |
| tbarron | was your export fd10:: | 21:07 |
| bswartz | fd10::/16 | 21:07 |
| bswartz | it has to be a network | 21:07 |
| tbarron | right | 21:07 |
| tbarron | well, it can be 128 bit network, i.e. host | 21:08 |
| bswartz | true | 21:08 |
| bswartz | but exporting host by host is painful | 21:08 |
| tbarron | the square brackets appear to be required according to man page and experiment | 21:08 |
| tbarron | but check that we're getting the syntax right | 21:09 |
| bswartz | no that's not right | 21:09 |
| bswartz | it can't be | 21:09 |
| tbarron | in the last iteration of the patch it was leaving them out | 21:09 |
| bswartz | square brackets can be used with ipv6 address literals, but never with subnets | 21:09 |
| tbarron | and the command also failed on fedora/centos but with a different message | 21:10 |
| bswartz | [fd10::] is valid | 21:10 |
| bswartz | neither [fd10::/16] nor [fd10::]/16 should be valid | 21:10 |
| tbarron | bswartz: look at the Examples section of http://man7.org/linux/man-pages/man8/exportfs.8.html | 21:12 |
| *** tommylikehu has quit IRC | 21:12 | |
| tbarron | fedora/centos appear to conform to that | 21:12 |
| *** tommylikehu has joined #openstack-manila | 21:12 | |
| tbarron | ubuntu is atm just broken so we don't really know what it does | 21:12 |
| bswartz | ugh | 21:12 |
| tbarron | example there: exportfs [fe80::]/64:/usr/tmp | 21:12 |
| bswartz | what is this world coming to? | 21:13 |
| tbarron | well, whoever is in charge determines what is true | 21:13 |
| bswartz | horrifying syntax is unix commands? | 21:13 |
| * tbarron couldn't help himself | 21:13 | |
| bswartz | I thought we were better than this | 21:13 |
| bswartz | s/is/in/ | 21:13 |
| bswartz | okay so the big question is what do I need to download and recompile to fix the bug on xenial? | 21:14 |
| markstur | ganso: What is that if true or false thing in here: https://review.openstack.org/#/c/415274/5/manila_ui/dashboards/project/shares/templates/shares/snapshots/_snapshot_detail_overview.html | 21:15 |
| markstur | i see it is used other places, but why? | 21:16 |
| *** rushil has quit IRC | 21:19 | |
| *** erlon has quit IRC | 21:20 | |
| tbarron | bswartz: looks like you could hack around this by writing into /etc/exports and running 'exportfs -a' | 21:21 |
| tbarron | bswartz: that's what generic driver does | 21:21 |
| tbarron | bswartz: I consider it very ugly, but as a temporary hack might be suitable | 21:21 |
| bswartz | tbarron: sadly that would be not testing out manila code | 21:21 |
| bswartz | s/out/our/ | 21:21 |
| bswartz | I'm willing to cut a lot of corners but not this one | 21:22 |
| tbarron | bswartz: that's what our generic driver manila code does, so we could do it in the access helper for NFS if we were willing to make it hacky too | 21:22 |
| bswartz | when we start testing stuff that the code doesn't implement then it's time to update the code | 21:22 |
| tbarron | bswartz: I really don't like it though | 21:22 |
| bswartz | yeah if it's a reasonable workaround to the exportfs bug | 21:22 |
| bswartz | in this case the cure is roughly as bad as the disease | 21:23 |
| *** erlon has joined #openstack-manila | 21:23 | |
| bswartz | if exportfs is broken in a ways we can't workaround then I'm going to lean towards not mering ipv6 until later | 21:23 |
| bswartz | not merging even | 21:23 |
| *** eharney has quit IRC | 21:30 | |
| tbarron | bswartz: the hack https://github.com/openstack/manila/blob/master/manila/share/drivers/helpers.py#L293 | 21:30 |
| tbarron | maybe you have to do that kind of thing in our service vm images?? | 21:32 |
| bswartz | tbarron: https://paste.fedoraproject.org/536556/85379955/ | 21:32 |
| openstackgerrit | Merged openstack/manila: Add mountable snapshots support to HNAS driver https://review.openstack.org/411474 | 21:32 |
| tbarron | bswartz: yeah, that would be cleaner | 21:33 |
| bswartz | tbarron: ^ this is how i'm pulling a fixed exportfs into my devstack VM | 21:33 |
| bswartz | and indeed it actually works with that horrifying syntax you showed me | 21:34 |
| bswartz | now to test the mount | 21:34 |
| bswartz | tbarron: the export location needs [] too | 21:38 |
| bswartz | still getting access denied even though exportfs says I should have access | 21:38 |
| tbarron | ok, exports are working for me now | 21:39 |
| *** ociuhandu has joined #openstack-manila | 21:39 | |
| *** rushil has joined #openstack-manila | 21:41 | |
| bswartz | okay the NFS on this machine is a little unstable now with all this hacking | 21:41 |
| bswartz | I'm mixing versions of things that I probably shouldn't be | 21:42 |
| tbarron | bswartz: shouldn't 'showmount -e' show these exports now? | 21:42 |
| bswartz | I'm sure there's a way to get everything upgraded to a mutually-compatible level | 21:42 |
| tbarron | bswartz: I see them with 'exportfs' command but not with 'showmount -e' | 21:42 |
| *** dustins is now known as dustins|out | 21:42 | |
| bswartz | tbarron: my rpc.mount service is barfing on the new arguments being inserted this the ultra-new exportfs command | 21:43 |
| *** cknight has quit IRC | 21:43 | |
| bswartz | tbarron: check your syslog and you'll see errors | 21:43 |
| bswartz | something else needs upgrading to make this actually work | 21:43 |
| bswartz | but I'm not sure what | 21:43 |
| tbarron | Jan 25 21:40:48 localhost rpc.mountd[1343]: /var/lib/nfs/etab:1: unknown keyword "no_pnfs" | 21:44 |
| bswartz | yeah same error | 21:44 |
| tbarron | gee, packaging linux could be a full time job for somebody | 21:44 |
| bswartz | I'm sure it's caused by mixing a new nfs-utils with an old something else | 21:44 |
| bswartz | lol | 21:44 |
| bswartz | if only there was a company that worked on this stuff.... | 21:45 |
| bswartz | anyways I'm going to take a break | 21:45 |
| bswartz | I will start digging back into this tonight | 21:45 |
| bswartz | currently I'm leaning toward letting ipv6 wait for pike | 21:46 |
| tbarron | bswartz: i'm gonna drink beer with jungleboyj tonight | 21:46 |
| bswartz | unless we can find a solution for testing this that isn't gross, the code will mostly rot in place | 21:46 |
| tbarron | bswartz: we should maybe look for a way to merge some of this infra in behind the scenes as it becomes possible though so the patches done don't get too stale | 21:47 |
| tbarron | yeah, that was two messages crossing on the wire | 21:47 |
| bswartz | tbarron: pike will be open for submissions next week | 21:47 |
| tbarron | there's been good work here, and valeriy just gave a good set of code review on this stuff | 21:47 |
| tbarron | so it would be nice to keep momentum | 21:47 |
| bswartz | yes | 21:48 |
| bswartz | I want to keep momentum | 21:48 |
| bswartz | at least through tomorrow | 21:48 |
| bswartz | if we can split out the parts that are testable, that would be good | 21:48 |
| tbarron | agree. I'm going to try to find out whether we are deploying OpenStack with ipv6 yet, and with what tenant network ownership model | 21:51 |
| tbarron | it's all very well to be developing features in manila because we think they are a great idea, but as I | 21:51 |
| tbarron | am finding out in this case, there may be missing context around us that we need to take account of. | 21:51 |
| tbarron | When we talk about ipv6 and openstack I think sometimes people are thinking mostly about service endpoints. | 21:52 |
| tbarron | which is quite a different issue than tenant networks | 21:52 |
| bswartz | tbarron: I'd assume the opposite | 21:53 |
| tbarron | bswartz: so you are thinking they are the same issue at root? | 21:54 |
| bswartz | ipv4 for services is pretty harmless but users of the cloud have good reasons to demand ipv6 just so they can access the world without NAT | 21:54 |
| tbarron | or the relative importance? | 21:54 |
| bswartz | manila is in an odd spot here providing a data service to tenants | 21:54 |
| bswartz | the way I see it, we need to fix ipv6, period | 21:54 |
| bswartz | and as we find bugs we need to push people to fix them (like this exportfs bug) | 21:55 |
| tbarron | bswartz: all I'm saying is that in devstack for example the interest seems to be mostly in deploying ipv6 endpoints, not in doing routing to ipv6 tentnat networks while providing tenant isolation and tenants spread with ipv6 on segmented nets | 21:56 |
| *** breitz has quit IRC | 22:00 | |
| *** breitz has joined #openstack-manila | 22:00 | |
| *** eharney has joined #openstack-manila | 22:09 | |
| *** lpetrut has quit IRC | 22:22 | |
| *** Yogi1 has quit IRC | 22:23 | |
| *** gouthamr has quit IRC | 22:24 | |
| *** kaisers_ has quit IRC | 22:51 | |
| *** catintheroof has joined #openstack-manila | 23:00 | |
| ganso | markstur: just saw your message, will fix tomorrow with Miriam | 23:06 |
| ganso | markstur: thanks for pointing that out | 23:07 |
| *** rushil has quit IRC | 23:07 | |
| markstur | ganso: OK. 9PM for you? | 23:07 |
| ganso | markstur: yes | 23:08 |
| ganso | markstur: in order for that patch to merge, https://review.openstack.org/#/c/425310/ needs to merge, and the wait queue for that patch is looooooooooooooooooooooong | 23:09 |
| *** draynium has quit IRC | 23:10 | |
| markstur | yikes. He was not exagerating with those ooooooooooooooooo | 23:13 |
| *** draynium has joined #openstack-manila | 23:17 | |
| *** rushil has joined #openstack-manila | 23:19 | |
| *** xyang1 has quit IRC | 23:20 | |
| *** kaisers has joined #openstack-manila | 23:21 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/manila: Updated from global requirements https://review.openstack.org/424616 | 23:23 |
| *** rushil has quit IRC | 23:33 | |
| *** rushil has joined #openstack-manila | 23:53 | |
| *** catintheroof has quit IRC | 23:58 | |
| *** catintheroof has joined #openstack-manila | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!