*** markstur has quit IRC | 00:10 | |
*** erlon has quit IRC | 00:30 | |
*** erlon has joined #openstack-manila | 00:43 | |
*** zul has quit IRC | 01:04 | |
*** erlon has quit IRC | 01:58 | |
*** dustins has joined #openstack-manila | 03:15 | |
*** dustins has quit IRC | 03:25 | |
*** pcaruana|elisa| has joined #openstack-manila | 07:40 | |
*** pcaruana|elisa| has quit IRC | 07:59 | |
*** pcaruana has joined #openstack-manila | 08:05 | |
*** e0ne has joined #openstack-manila | 09:43 | |
*** e0ne has quit IRC | 10:32 | |
*** e0ne has joined #openstack-manila | 10:36 | |
*** ganso has joined #openstack-manila | 10:49 | |
openstackgerrit | Rodrigo Barbieri proposed openstack/manila master: PLEASE DO NOT MERGE ME https://review.openstack.org/614737 | 11:03 |
---|---|---|
*** pcaruana has quit IRC | 11:05 | |
*** zul has joined #openstack-manila | 11:26 | |
*** luizbag has joined #openstack-manila | 11:28 | |
*** luizbag has quit IRC | 11:34 | |
*** pcaruana has joined #openstack-manila | 11:52 | |
*** erlon has joined #openstack-manila | 11:57 | |
*** jmlowe has quit IRC | 12:47 | |
*** zul has quit IRC | 12:52 | |
*** e0ne has quit IRC | 12:55 | |
*** zul has joined #openstack-manila | 12:56 | |
*** eharney has joined #openstack-manila | 13:01 | |
*** zul has quit IRC | 13:04 | |
*** zul has joined #openstack-manila | 13:06 | |
*** jmlowe has joined #openstack-manila | 13:07 | |
*** belmoreira has joined #openstack-manila | 13:17 | |
openstackgerrit | Rodrigo Barbieri proposed openstack/manila-specs master: Add spec for Manage-Unmanage of Share Servers https://review.openstack.org/607342 | 13:26 |
*** dustins has joined #openstack-manila | 13:34 | |
*** e0ne has joined #openstack-manila | 13:40 | |
*** gouthamr has left #openstack-manila | 13:54 | |
*** tpsilva has joined #openstack-manila | 13:55 | |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [CI][LVM] Run the LVM job on Bionic Beaver https://review.openstack.org/608761 | 14:07 |
*** e0ne_ has joined #openstack-manila | 14:09 | |
*** e0ne has quit IRC | 14:10 | |
*** gouthamr has joined #openstack-manila | 14:39 | |
*** markstur has joined #openstack-manila | 14:55 | |
tbarron | manila meeting has started on #openstack-meeting-alt | 15:03 |
*** jmlowe has quit IRC | 15:10 | |
*** e0ne_ has quit IRC | 15:34 | |
*** e0ne has joined #openstack-manila | 15:34 | |
gouthamr | we have documentation suggesting the use of linuxbridge network driver for the manila generic share driver | 16:00 |
gouthamr | but we don't test it, it's likely broken | 16:01 |
tbarron | ganso sorry to cut you off | 16:01 |
gouthamr | ganso: ^ about not being able to reproduce at the gate | 16:01 |
tbarron | gouthamr: it's not just our doc, the neutron doc w.r.t. openstack installation also instructs use of linuxbridge | 16:01 |
gouthamr | it's quite easy for small deployments to setup linux bridge, so maybe we should run one of our jobs with it | 16:02 |
gouthamr | and we might see the issue? | 16:02 |
ganso | tbarron: np | 16:02 |
tbarron | ganso: gouthamr and I have also been getting emails about this, probably from the same person | 16:02 |
ganso | I helped 2 people that were deploying the generic driver with linux bridge | 16:02 |
ganso | both of them initially had connection issues | 16:02 |
ganso | but one of them supposedly (I don't know how) sorted the connection issue out, and was able to connect manually to the service instance | 16:03 |
ganso | but the driver was not able to connect, for some reason | 16:03 |
ganso | we ran tcpdump and everything, the ssh handshake from the driver was seen, but the driver could not connect | 16:03 |
bswartz | was the ssh handshake successful? | 16:04 |
ganso | we checked security group rules, etc | 16:04 |
bswartz | One of the bad cases I've seen is when the service VM can't talk to the metadata server, and cloud-init fails to get the SSH key | 16:04 |
ganso | bswartz: I saw ssh handshake packages, I am not sure if it was successful. So it wasn't a connection issue in that case | 16:04 |
bswartz | In that case, SSH cannot succeed, and you have to use password login | 16:04 |
ganso | bswartz: ohhhhhh that could be the case | 16:04 |
bswartz | IIRC, the gate uses password login all the time because we were never able to track down the cloud-init problem | 16:05 |
gouthamr | bswartz: the reporter used password as well | 16:05 |
ganso | yes, but setting up the username and password in the driver still didn't allow it to suceed | 16:05 |
bswartz | ganso: are you sure you used the username and password that was pre-installed in the service image? | 16:05 |
bswartz | Which values did you use? | 16:05 |
ganso | the person could do it manually, the share server was in error state but he could ssh into it. The driver couldn't, not even with username and password instead of ssh keys | 16:06 |
bswartz | manila/manila? | 16:06 |
ganso | bswartz: yes | 16:06 |
bswartz | Okay maybe you've found yet another issue | 16:06 |
ganso | so yesterday, the last time I spoke to him, I asked him to run a devstack in another env and we would try to compare every configuration that could be relatable | 16:07 |
bswartz | When I hit series of problems last year I just banged my head into the wall until I found a less nettlesome issue to work on | 16:07 |
bswartz | The context I hit it in was when trying to use my alternative (lightweight) service image | 16:07 |
ganso | hmmmmmm | 16:09 |
ganso | anyway, thanks for the tip, I will try to check for the metadata server connectivity | 16:09 |
tbarron | ganso: we shouldn't need metadata server working for password auth though, right? | 16:10 |
ganso | tbarron: yes, we shouldn't | 16:10 |
tbarron | ganso: or is there something else from cloud-init that is required? | 16:10 |
ganso | tbarron: I am not sure why the driver still can't ssh without the ssh key | 16:11 |
ganso | tbarron: I don't know what else is required. Another weird thing is that, as far as I know, the metadata server and cloud-init are both needed for the instance to obtain its IP | 16:11 |
tbarron | is the driver using the same ssh client as when one runs the cmd manually from the vm? | 16:11 |
*** belmoreira has quit IRC | 16:11 | |
ganso | bswartz: ^ can confirm that? was the problem you had the instance had its IP? | 16:11 |
ganso | tbarron: the driver uses paramiko | 16:12 |
tbarron | s/from the vm/from the node where manila share runs to the vm/ | 16:14 |
tbarron | ganso: so I wonder if a simple python program with paramiko and passwd auth would have the same problem connecting to the vm. | 16:14 |
ganso | tbarron: yea I wonder that too, that could be a good test. Write a simple python code to connect via paramiko and see what is required or why it fails | 16:15 |
bswartz | It's been a while since I looked into this | 16:15 |
bswartz | But when I dug deep down I saw SSH failures | 16:16 |
bswartz | And the thing that makes it worse is that paramiko lies in its error messages | 16:16 |
bswartz | Some of the error messages that come out of paramiko are completely false | 16:16 |
* tbarron notes that octavia uses service vms and wonders if they ssh to them via paramiko | 16:16 | |
bswartz | You can verify this by stepping into paramiko with a debugger while the driver is trying to connect to the service VM | 16:17 |
ganso | bswartz: since I was just giving instructions via email I was relying on the other person to be able to debug | 16:18 |
bswartz | Yeah I'm just saying that it's incredibly hard to see what's going on because (1) it's all virtualized and seeing the internals requires special tooling and (2) the system is actively lying to you about why thing didn't work | 16:19 |
ganso | tbarron: one more thing I wanted to say was that, both of these folks that approached me with that issue attempted reinstalling everything with OVS instead of LB and still didn't get it to work | 16:19 |
bswartz | If there was an alternative to paramiko I would suggest using it | 16:20 |
bswartz | Otherwise the only solution is to contribute bugfixes to paramiko | 16:20 |
ganso | bswartz: it is already difficult to deal with liars, but when the machine is lying it is even worse xD | 16:21 |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver https://review.openstack.org/614802 | 16:24 |
ganso | gouthamr: so what were your breakthroughs? | 16:25 |
tbarron | ganso: I guess the mystery is why we don't have this issue with paramiko in gate. We were thinking the diff was something about ovs vs linuxbridge but I guess not. | 16:25 |
gouthamr | ganso: see ^ | 16:25 |
*** dustins has quit IRC | 16:25 | |
gouthamr | i switched the IP address to br-ex - what we discussed, and what the quagga documentation seemed to suggest | 16:25 |
bswartz | gouthamr: the commit message says bionic, but the code comments say xenial | 16:26 |
bswartz | which is it? | 16:26 |
gouthamr | :) bswartz: i'm testing centos and xenial with https://review.openstack.org/614802 | 16:26 |
ganso | gouthamr: nice! for some reason it didn't work when I did it. Maybe it had to be done in the proper order | 16:26 |
ganso | gouthamr: I created a debug patch this morning but I didn't see any zebras in it | 16:26 |
*** e0ne has quit IRC | 16:26 | |
gouthamr | ganso: perhaps, yes.. i did a fresh stack and freaked out :P | 16:27 |
ganso | the 10-second sleep did not work | 16:27 |
gouthamr | ? | 16:27 |
gouthamr | i saw the patch | 16:27 |
bswartz | gouthamr: why does the commit message say bionic then? | 16:27 |
gouthamr | bswartz: https://review.openstack.org/608761 is the true bionic change - i.e, switch the job to run on bionic | 16:27 |
ganso | gouthamr: I was looking for a route added by zebra, but there wasn't any. There was supposed to be one | 16:27 |
bswartz | So is the problem with xenial or bionic? | 16:28 |
gouthamr | ganso: hmm, strange thought there was one - looking | 16:28 |
gouthamr | bswartz: i haven't played with quagga on Xenial, ganso has.. gate's currently using centos - netapp-ci is using Xenial | 16:28 |
ganso | bswartz: the problem is with bionic | 16:28 |
gouthamr | bswartz: i don't want to break folks so decided to split out the quagga changes to test xenial and centos in isolation | 16:29 |
gouthamr | ganso: how much lag have you seen for the zebra route to show up? | 16:32 |
bswartz | gouthamr: I'm going to -1 your comment in that case | 16:41 |
gouthamr | bswartz: sure, am confused, but will address it :) | 16:42 |
ganso | gouthamr: I am not sure, shouldn't be more than 10 seconds | 16:45 |
*** belmoreira has joined #openstack-manila | 17:02 | |
gouthamr | regarding reproducing the connectivity bug, we can eliminate one variable by testing with manila.network.linux.interface.BridgeInterfaceDriver as the interface_driver on the gate | 17:02 |
gouthamr | we run two jobs with DHSS=true/generic | 17:03 |
gouthamr | in two separate repos - manila, manila-tempest-plugin | 17:03 |
gouthamr | one of these can be running with LinuxBridge instead of OVS as the mech driver | 17:03 |
*** belmoreira has quit IRC | 17:19 | |
*** belmoreira has joined #openstack-manila | 17:20 | |
*** e0ne has joined #openstack-manila | 17:31 | |
*** e0ne has quit IRC | 17:47 | |
*** luizbag has joined #openstack-manila | 17:54 | |
*** e0ne has joined #openstack-manila | 17:59 | |
*** e0ne has quit IRC | 18:02 | |
*** e0ne has joined #openstack-manila | 18:05 | |
*** e0ne has quit IRC | 18:07 | |
*** jmlowe has joined #openstack-manila | 18:22 | |
ganso | gouthamr: sounds good | 18:27 |
ganso | gouthamr: I am confused about https://review.openstack.org/#/c/608761/ and https://review.openstack.org/#/c/614802 | 18:41 |
ganso | gouthamr: the former is running on bionic and has quagga working | 18:42 |
ganso | gouthamr: why do we need the latter (it is running on centos)? | 18:42 |
* gouthamr cool, just got zuul's run on it | 18:42 | |
gouthamr | ganso: yep, i was going to remove the plugin changes from https://review.openstack.org/#/c/608761/ | 18:42 |
gouthamr | and make it dependent on https://review.openstack.org/#/c/614802 | 18:43 |
ganso | gouthamr: oh ok | 18:43 |
gouthamr | so that https://review.openstack.org/#/c/614802 can test centos | 18:43 |
ganso | gouthamr: great! | 18:43 |
gouthamr | looks like the ipv6 tests passed: http://logs.openstack.org/02/614802/1/check/manila-tempest-minimal-dsvm-lvm-centos-7/be8e085/job-output.txt.gz#_2018-11-01_17_53_05_140687 | 18:43 |
gouthamr | but the job timed out | 18:43 |
gouthamr | ganso: don't see the zebra route here: http://logs.openstack.org/02/614802/1/check/manila-tempest-minimal-dsvm-lvm-centos-7/be8e085/job-output.txt.gz#_2018-11-01_17_33_03_310558 | 18:44 |
gouthamr | ganso: so the logging isn't very useful at the moment, i'll just remove it | 18:45 |
ganso | gouthamr: yea me neither, IPv6 is passing so it is working | 18:45 |
gouthamr | ganso: yes, and those private networks are created in the test? | 18:45 |
ganso | gouthamr: yes | 18:46 |
ganso | gouthamr: we should still the 1 route for the private network created by the plugin. It might be that 10 seconds is not enough | 18:46 |
ganso | gouthamr: but I don't think it is worth it to send another patch and increase the sleep | 18:46 |
gouthamr | ganso: yes, i'm confused about that as well.. its working locally too | 18:47 |
gouthamr | yeah | 18:47 |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver https://review.openstack.org/614802 | 18:48 |
*** belmoreira has quit IRC | 18:48 | |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver https://review.openstack.org/614802 | 18:49 |
*** pcaruana has quit IRC | 19:05 | |
*** luizbag has quit IRC | 19:16 | |
*** luizbag_ has joined #openstack-manila | 19:21 | |
*** belmoreira has joined #openstack-manila | 19:32 | |
*** dustins has joined #openstack-manila | 20:05 | |
*** dustins has quit IRC | 20:06 | |
*** dustins has joined #openstack-manila | 20:06 | |
*** dustins has quit IRC | 20:07 | |
*** dustins has joined #openstack-manila | 20:08 | |
*** eharney has quit IRC | 20:08 | |
*** e0ne has joined #openstack-manila | 20:11 | |
*** belmoreira has quit IRC | 20:12 | |
ganso | xyang: ping | 20:18 |
xyang | hi | 20:19 |
*** luizbag_ has quit IRC | 20:20 | |
ganso | xyang: Hey Xing =) Could you please take a look at those 3 specs? https://review.openstack.org/#/c/607342/, https://review.openstack.org/#/c/391805 and https://review.openstack.org/#/c/609537 ? | 20:20 |
xyang | ganso: sure | 20:20 |
ganso | xyang: thank you! =D | 20:20 |
xyang | np | 20:21 |
*** e0ne has quit IRC | 20:28 | |
*** belmoreira has joined #openstack-manila | 20:35 | |
*** belmoreira has quit IRC | 20:35 | |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver https://review.openstack.org/614802 | 20:54 |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [CI][LVM] Run the LVM job on Bionic Beaver https://review.openstack.org/608761 | 20:54 |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver https://review.openstack.org/614802 | 20:55 |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [LVM][IPv6] Quagga changes to support Bionic Beaver https://review.openstack.org/614802 | 21:03 |
gouthamr | last time ^ | 21:03 |
*** erlon has quit IRC | 21:05 | |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [CI] Switch Xenial tempest jobs to Bionic Beaver https://review.openstack.org/604929 | 21:13 |
*** dustins has quit IRC | 21:31 | |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [DevRef] Add code review guideline https://review.openstack.org/609598 | 21:43 |
openstackgerrit | Goutham Pacha Ravi proposed openstack/manila master: [CI][LVM] Run the LVM job on Bionic Beaver https://review.openstack.org/608761 | 21:49 |
*** threestrands has joined #openstack-manila | 22:17 | |
*** tpsilva has quit IRC | 23:04 | |
*** ganso has quit IRC | 23:17 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!