| *** minwang2 has quit IRC | 00:13 | |
| *** alhu has joined #openstack-lbaas | 00:16 | |
| *** minwang2 has joined #openstack-lbaas | 00:28 | |
| *** harlowja has joined #openstack-lbaas | 00:50 | |
| *** minwang2 has quit IRC | 00:56 | |
| *** minwang2 has joined #openstack-lbaas | 01:03 | |
| *** yamamot__ has joined #openstack-lbaas | 01:07 | |
| *** yamamot__ has quit IRC | 01:11 | |
| *** yamamot__ has joined #openstack-lbaas | 01:22 | |
| *** minwang2 has quit IRC | 01:22 | |
| *** yamamot__ has quit IRC | 01:27 | |
| *** kfox1111 has quit IRC | 01:31 | |
| *** kfox1111 has joined #openstack-lbaas | 01:48 | |
| *** yamamot__ has joined #openstack-lbaas | 01:52 | |
| *** barra204 has quit IRC | 01:53 | |
| *** shakamunyi has quit IRC | 01:53 | |
| *** kfox1111 has quit IRC | 02:48 | |
| *** harlowja has quit IRC | 02:51 | |
| *** kfox1111 has joined #openstack-lbaas | 02:55 | |
| *** bana_k has joined #openstack-lbaas | 02:56 | |
| *** yamamot__ has quit IRC | 02:56 | |
| *** alhu has quit IRC | 03:04 | |
| *** alhu has joined #openstack-lbaas | 03:05 | |
| *** ktrmzn has joined #openstack-lbaas | 03:06 | |
| *** bana_k has quit IRC | 03:09 | |
| *** ChrisBenson has quit IRC | 03:24 | |
| *** ChrisBenson has joined #openstack-lbaas | 03:24 | |
| *** prabampm has joined #openstack-lbaas | 03:30 | |
| *** ChrisBenson has quit IRC | 03:33 | |
| *** ChrisBenson has joined #openstack-lbaas | 03:35 | |
| *** bana_k has joined #openstack-lbaas | 03:39 | |
| *** alhu has quit IRC | 03:43 | |
| *** yamamot__ has joined #openstack-lbaas | 03:56 | |
| *** yamamot__ has quit IRC | 04:03 | |
| openstackgerrit | Doug Wiegley proposed openstack/neutron-lbaas: Switch to using hacking checks from neutron-lib https://review.openstack.org/309855 | 04:25 |
|---|---|---|
| *** prabampm has quit IRC | 04:35 | |
| *** prabampm has joined #openstack-lbaas | 04:37 | |
| *** links has joined #openstack-lbaas | 04:43 | |
| *** armax has joined #openstack-lbaas | 04:46 | |
| *** yamamot__ has joined #openstack-lbaas | 04:59 | |
| *** armax has quit IRC | 04:59 | |
| *** yamamot__ has quit IRC | 05:04 | |
| *** yamamot__ has joined #openstack-lbaas | 06:01 | |
| *** yamamot__ has quit IRC | 06:06 | |
| *** rcernin has joined #openstack-lbaas | 06:19 | |
| *** numans has joined #openstack-lbaas | 06:22 | |
| *** yamamot__ has joined #openstack-lbaas | 07:02 | |
| *** sbalukoff has quit IRC | 07:05 | |
| *** sbalukoff has joined #openstack-lbaas | 07:06 | |
| *** reedip__ has joined #openstack-lbaas | 07:07 | |
| *** yamamot__ has quit IRC | 07:08 | |
| *** numans has quit IRC | 07:10 | |
| *** numans has joined #openstack-lbaas | 07:12 | |
| *** kevo has quit IRC | 07:14 | |
| openstackgerrit | Yang Yu proposed openstack/neutron-lbaas: Fix no such option defined for service_auth https://review.openstack.org/293352 | 07:26 |
| *** banszmar has quit IRC | 07:40 | |
| *** reedip__ has quit IRC | 07:43 | |
| *** eezhova has quit IRC | 07:56 | |
| *** eezhova has joined #openstack-lbaas | 07:57 | |
| *** banszmar has joined #openstack-lbaas | 07:58 | |
| *** yamamot__ has joined #openstack-lbaas | 08:04 | |
| *** yamamot__ has quit IRC | 08:10 | |
| *** bana_k has quit IRC | 08:34 | |
| *** yamamot__ has joined #openstack-lbaas | 09:06 | |
| *** yamamot__ has quit IRC | 09:11 | |
| *** prabampm has quit IRC | 09:20 | |
| *** bogdan has joined #openstack-lbaas | 09:47 | |
| bogdan | ERROR oslo_messaging.rpc.dispatcher IOError: [Errno 2] No such file or directory: '/etc/octavia/certs/ca_01.pem' | 09:47 |
| bogdan | what is this file? I cannot find any explanation over the internet | 09:47 |
| bogdan | anyone? | 09:54 |
| bogdan | anyone with a reference for octavia.config? there are so many unclear options? | 10:07 |
| *** yamamot__ has joined #openstack-lbaas | 10:08 | |
| *** yamamot__ has quit IRC | 10:13 | |
| *** jsheeren has joined #openstack-lbaas | 10:30 | |
| *** yamamoto has joined #openstack-lbaas | 11:09 | |
| *** permalac has joined #openstack-lbaas | 11:10 | |
| *** yamamoto has quit IRC | 11:15 | |
| *** bana_k has joined #openstack-lbaas | 11:21 | |
| *** yamamoto_ has joined #openstack-lbaas | 12:11 | |
| *** yamamoto_ has quit IRC | 12:16 | |
| *** yamamoto_ has joined #openstack-lbaas | 12:17 | |
| jsheeren | hi quick question about lbaasv2; is the UI for lbaasv2 available in mitaka? | 12:23 |
| bogdan | jsheeren yes, I have installed Mitaka Horizon and I have the Load Balancer for lbaasv2 in there | 12:28 |
| jsheeren | in the local_settings, is it the same enable_lb: true? | 12:28 |
| jsheeren | under the OPENSTACK_NEUTRON_NETWORK section? | 12:28 |
| bogdan | jsheeren AFAIK the enable_lb has to be true but UI will appear only after you configure your Neutron to run with lbaasv2 enabled | 12:29 |
| jsheeren | bogdan, thanks! the lbaasv2 agent is running fine .. hmme; i'll double check everything then | 12:30 |
| *** banszmar has quit IRC | 12:30 | |
| *** diogogmt has quit IRC | 12:36 | |
| *** diogogmt has joined #openstack-lbaas | 12:41 | |
| bogdan | any experts on Octavia? | 12:46 |
| jsheeren | bogdan, can i bother you for a sec more? | 12:46 |
| bogdan | yes | 12:46 |
| jsheeren | i'm seeing this in the log when i create a loadbalancer: AttributeError: 'HaproxyNSDriver' object has no attribute | 12:46 |
| jsheeren | 'loadbalancer' | 12:46 |
| jsheeren | followed this: http://docs.openstack.org/mitaka/networking-guide/adv-config-lbaas.html | 12:46 |
| jsheeren | lbaasv2 with an agent | 12:47 |
| bogdan | unfortunately I cant help with that one, I have never tried out with the agent | 12:48 |
| *** banszmar has joined #openstack-lbaas | 12:48 | |
| jsheeren | oh ok | 12:48 |
| jsheeren | bogdan, thanks anyway | 12:48 |
| bogdan | I started right away with the Octavia | 12:48 |
| *** permalac_ has joined #openstack-lbaas | 12:56 | |
| *** permalac has quit IRC | 12:59 | |
| *** amit213 has quit IRC | 13:00 | |
| *** yamamoto_ has quit IRC | 13:24 | |
| bogdan | any octavia experts? | 13:38 |
| *** alhu has joined #openstack-lbaas | 13:39 | |
| *** ducttape_ has joined #openstack-lbaas | 13:40 | |
| *** alhu1 has joined #openstack-lbaas | 13:42 | |
| *** alhu has quit IRC | 13:43 | |
| *** ducttape_ has quit IRC | 13:45 | |
| *** alhu1 has quit IRC | 13:47 | |
| *** woodster_ has joined #openstack-lbaas | 13:49 | |
| *** diogogmt has quit IRC | 14:07 | |
| Frito | bogdan: I think they may all be involved in the openstack summit this week | 14:16 |
| Frito | https://www.openstack.org/summit/austin-2016/ | 14:17 |
| *** links has quit IRC | 14:18 | |
| *** harlowja has joined #openstack-lbaas | 14:19 | |
| *** ajmiller has joined #openstack-lbaas | 14:28 | |
| *** bana_k has quit IRC | 14:30 | |
| *** harlowja has quit IRC | 14:31 | |
| bogdan | Frito, thanks, I didn't realize it is this week, will come back in a week if I am not done setting up the Octavia, I am slowly progressing ,... slowly, but progressing :) | 14:36 |
| Frito | haha, no worries. I feel the same way. I haven't gotten a successful end-to-end local test in devstack yet either. | 14:37 |
| bogdan | where are you stuck? | 14:40 |
| *** banszmar has quit IRC | 14:40 | |
| bogdan | I am now at the amphora image creation step :) | 14:40 |
| Frito | That's an excellent question... I /think/ I'm stuck at getting members added to the pool successfully. Or at least that's where I was last week. This week it seems my machine is taking forever to spin up an amphorae | 14:42 |
| Frito | Basically I'm trying to run "neutron lbaas-loadbalancer-create --name lb1 private-subnet" right now and get an instance. | 14:43 |
| Frito | I got a gist from ptoohill over the weekend but it's not cooperating with me so I'm in that trial / error mode | 14:44 |
| bogdan | I am right now at exactly the same point :) neutron creating balancer and I get my amphora vm created but I have not used an amphora image yet (just using the cirros one) and as expected the worker log is full of " Could not connect to instance. Retrying." | 14:51 |
| bogdan | so, now I have to create a proper image with haproxy on it etc. | 14:51 |
| bogdan | what are you using for amphora image? | 14:52 |
| Frito | I think you can remote into the amphorae if you're using devstack | 14:52 |
| Frito | if you do "nova list" can you see the amphorae? | 14:52 |
| *** bana_k has joined #openstack-lbaas | 14:52 | |
| bogdan | ideally yes, I should be able to remote to it but for whatever reason I am not able to access it, no ping nor ssh | 14:53 |
| *** harlowja has joined #openstack-lbaas | 14:53 | |
| bogdan | yes nova list --all-rpojects shows the amphora-* instance | 14:53 |
| Frito | cool. so you do have an IP there? | 14:53 |
| Frito | 10. I think | 14:53 |
| bogdan | yes I have an IP | 14:53 |
| bogdan | in the internal network I configured amphora to run | 14:54 |
| Frito | Try... | 14:54 |
| Frito | ssh -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@{your_ip} ?? | 14:54 |
| *** bana_k has quit IRC | 14:54 | |
| bogdan | aaah, this is something I forgot - to use the key, but I started with ping and it is not pinging while all my other cirros instances are pinging | 14:55 |
| bogdan | even in the same network | 14:55 |
| Frito | I'm not sure on the ping. I don't have one up right now to verify if they respond or not. | 14:55 |
| bogdan | but even without a key it should at least say I am not authorized or something | 14:56 |
| Frito | Or prompt for a password :-/ | 14:57 |
| bogdan | ssh: connect to host 192.168.1.141 port 22: No route to host | 14:57 |
| *** banszmar has joined #openstack-lbaas | 14:57 | |
| bogdan | in horizon I see the log of the instance and I see it has booted and got an IP, etc. | 14:58 |
| Frito | when you created the load balancer what command did you use? | 14:58 |
| bogdan | neutron lbaas-loadbalancer-create --name demo-lb demo-subnet | 14:58 |
| Frito | yea, that's odd (no route to host). | 14:59 |
| Frito | Honestly, it sounds like you've tried most of the things I would have tried up to this point. | 15:00 |
| Frito | and the create command looks fine to me too for what it's worth :-> | 15:00 |
| Frito | :-/* | 15:00 |
| Frito | once my ./stack finishes running I'll stand up an amphorae and see if I can ping it. | 15:01 |
| *** numans has quit IRC | 15:01 | |
| Frito | yea, when I "neutron lbaas-loadbalancer-list" I get my amphorae but with "PENDING_CREATE" | 15:05 |
| Frito | and it never updates | 15:05 |
| Frito | which is what I'm trying to figure out | 15:05 |
| bogdan | i found an issue - the subnet-id of the lb for whatever reason is assigned to my external subnet ...not the internal subnet... will dig deeper now :) | 15:08 |
| Frito | Ha, nice. Progress at least! | 15:09 |
| bogdan | in the neutron lb create command and in the octavia.conf I am using the internal subnet name, but I am still getting the external subnet assigned ...wil have to debug the python code I guess... I hate when I end up debugging the code... | 15:12 |
| *** diogogmt has joined #openstack-lbaas | 15:15 | |
| Frito | :-/ | 15:16 |
| Frito | Yea. I hear ya. Being frank, I've never had this much trouble getting started on a project before... | 15:16 |
| *** harlowja has quit IRC | 15:25 | |
| bogdan | me too | 15:30 |
| *** links has joined #openstack-lbaas | 15:43 | |
| *** neelashah has joined #openstack-lbaas | 15:45 | |
| *** neelashah has quit IRC | 16:00 | |
| *** fnaval has joined #openstack-lbaas | 16:18 | |
| *** sbalukoff has quit IRC | 16:29 | |
| *** jsheeren has quit IRC | 16:30 | |
| *** rcernin has quit IRC | 16:30 | |
| bogdan | Frito, I found the problem on not accessing the instance - the default security group was missing the ICMP... I thought I have enabled it but it seems it was for another default group :) | 16:31 |
| *** amotoki has joined #openstack-lbaas | 16:31 | |
| Frito | ahh, good catch :-) | 16:31 |
| bogdan | also the ssh with key succeeded :) now my next step is to obtain an amphora image, do you know if ready images exist or should I build one with the script? | 16:33 |
| bogdan | what is your progress? where are you stuck at? | 16:34 |
| *** yamamoto has joined #openstack-lbaas | 16:35 | |
| Frito | I'm still trying to make headway. As far as I've gotten is that there is a bug in the nova driver. Getting an error message about the network ID being in the wrong format or something. | 16:35 |
| Frito | Trying to step through that stuff now | 16:35 |
| bogdan | at which step do you get the network ID being in the wrong format? | 16:37 |
| bogdan | I've seen this error when omitted the last symbol from the network ID | 16:38 |
| bogdan | I guess you double checked you ID | 16:38 |
| Frito | It's on the lb create | 16:38 |
| bogdan | is this the ID that you pass in octavia.conf? | 16:38 |
| Frito | Fun part is I'm not really providing an idea | 16:38 |
| Frito | id* | 16:38 |
| Frito | neutron lbaas-loadbalancer-create --name lb1 private-subne | 16:39 |
| Frito | Just the name :-/ | 16:39 |
| bogdan | have you setup amp_network = in octavia.conf? | 16:39 |
| bogdan | it should point to the ID of the network where your amphora will live | 16:40 |
| bogdan | keep in mind - the ID of the network and not subnet - at least this is what worked for me | 16:41 |
| *** links has quit IRC | 16:41 | |
| bogdan | I think I was getting the error when trying to use name instead of ID for that parameter | 16:42 |
| Frito | Hrm. The only item I have for amp_network is commented out | 16:43 |
| ptoohill | amp_network is deprecated, amp_boot_network list is the new option, but the devstack plugin should have set that | 16:44 |
| ptoohill | amp_network will still work btw, its just not set by 'default' in the devstack plugin anymore | 16:45 |
| bogdan | was it documented that it is deprecated? | 16:48 |
| Frito | Yea, I do have the amp_boot_network_list item | 16:48 |
| Frito | w/ an id | 16:48 |
| Frito | well a array of a single id | 16:48 |
| bogdan | how about amp_secgroup_list = default ends up with security group of lb-3b42e800-2e3e-4699.... | 16:49 |
| bogdan | ptoohill should this be an ID list or namelist? | 16:50 |
| *** yamamoto_ has joined #openstack-lbaas | 16:51 | |
| Frito | ALso, should the id in amp_boot_network_list be listed in "neutron subnet-list" ?? | 16:51 |
| ptoohill | id list, theres example in the config | 16:51 |
| ptoohill | this is only on latest master, it just recently merged of course | 16:51 |
| Frito | nevermind. It's in "neutron net-list" | 16:51 |
| ptoohill | network id list, yes | 16:52 |
| *** amotoki has quit IRC | 16:53 | |
| *** fnaval_ has joined #openstack-lbaas | 16:54 | |
| bogdan | ok, let me check it out :) I am working with a couple days old file :) | 16:54 |
| *** yamamoto has quit IRC | 16:54 | |
| ptoohill | you can still set amp_network and comment out the other one if you like, but if youre deploying with devstack that should have been set. If not then i have a bug that i failed to see | 16:55 |
| *** amotoki has joined #openstack-lbaas | 16:55 | |
| Frito | Anyone have tips on being able to debug o-cw in pyCharm? I went to the screen and ctrl+c'ed the process then copied the script and --config-file args into a configuration in pycharm. I can get a breakpoint to actually hit and halt execution but the "variables" section never populates and if I hit "step over" or "step into" nothing happens. It's almost as if the process has just hung. | 16:55 |
| *** fnaval has quit IRC | 16:55 | |
| *** amotoki has quit IRC | 16:55 | |
| *** amotoki_ has joined #openstack-lbaas | 16:55 | |
| ptoohill | Frito: I would suggest pdb for a majority of this. pycharm can be super helpful and awesome, but it can also have a ton of issues | 16:56 |
| *** amotoki_ has quit IRC | 16:56 | |
| ptoohill | Frito: youre doing this in devstack yea? | 16:56 |
| *** amotoki_ has joined #openstack-lbaas | 16:57 | |
| ptoohill | You need to ensure your debug env/python is the same that devstack is using, this can be a pain because venvs | 16:57 |
| Frito | Yes sir. Devstack only. TrevorV helped my configure it. | 16:57 |
| Frito | Yea, I"m not using any venvs | 16:57 |
| ptoohill | devstack does venv | 16:57 |
| Frito | Ahh. I wasn't aware of tat. | 16:58 |
| Frito | that* | 16:58 |
| ptoohill | i would say save yourself pain of trying to figure out through pycharm, use pdb | 16:58 |
| ptoohill | import pdb; pdb.set_trace() *need to have pdb installed on system | 16:58 |
| Frito | Okaie-dokie. | 16:58 |
| Frito | Got ya. So make that edit in the file I'm trying to debug? | 16:58 |
| *** amotoki_ has quit IRC | 16:58 | |
| ptoohill | put that line anywhere and itll stop, then you have a couple commands to view the stack etc i.e 'l' or 'n' | 16:58 |
| ptoohill | yes | 16:59 |
| * Frito is somewhat a python newb so these types of tips / tricks I'm still unfamiliar with. | 16:59 | |
| ptoohill | put that import pdb line above where you want to start | 16:59 |
| Frito | Right on. Thx | 16:59 |
| ptoohill | Frito: This is a decent walkthrough of pdb | 17:00 |
| ptoohill | https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/ | 17:00 |
| ptoohill | Also, theres things like ipdb and a few others i think that gives more abilities that makes it easier, but pdb will get the info you want | 17:00 |
| ptoohill | i used to heavily rely on pycharm, now i just auto go-to pdb | 17:01 |
| ptoohill | only time i care to use pycharm debugger now is when im writting tests | 17:01 |
| ptoohill | one gotcha is you have to go clean up your debug points, but thats about it :) | 17:02 |
| Frito | I only have one right now so all good :-D | 17:02 |
| ptoohill | :P | 17:02 |
| *** yamamoto_ has quit IRC | 17:08 | |
| *** amotoki has joined #openstack-lbaas | 17:08 | |
| *** madhu_ak_ has joined #openstack-lbaas | 17:15 | |
| Frito | stupid question: How do I exit pdb after I'm past the point I want to debug? | 17:18 |
| *** kevo has joined #openstack-lbaas | 17:18 | |
| bogdan | I am using winpdb at win side and rpdb2 at python/linux side | 17:19 |
| bogdan | I guess you could run the graphical frontend on linux too | 17:19 |
| ptoohill | Frito: 'q' to quit, 'c' to continue to next break point or the rest of the program | 17:20 |
| Frito | Right on. I think I missed that when I skimed through it the first time. I skimmed b/c I have someone standing over my shoulder. | 17:21 |
| bogdan | After some thousands of load balancer creations I ended up with Nova not being able to serve instances due to the DiskFilter ...though I deleted all amphora instances... have you seen such an issue or shall I ask this in the nova channel? | 17:21 |
| openstackgerrit | Kobi Samoray proposed openstack/octavia: Implement custom header support for Octavia https://review.openstack.org/257901 | 17:21 |
| Frito | So I think I found the error | 17:23 |
| Frito | In /octavia/compute/drivers/nova_driver.py... Line 118 | 17:24 |
| Frito | network_ids = [{'[id-here]'}] | 17:24 |
| *** amotoki has quit IRC | 17:24 | |
| Frito | I think the [ ] inside the string is what's causing the immediate issue. | 17:24 |
| ptoohill | nics = [] is my line 118 | 17:27 |
| Frito | hrm | 17:28 |
| Frito | mine is | 17:28 |
| Frito | network_ids = network_ids or [] | 17:28 |
| Frito | so probably 2 lines up from where you are :-) | 17:28 |
| Frito | but yea, that line gets pushed into nics.extend(...) under the if network_ids | 17:29 |
| Frito | in the debugger I just updated that variable manually and it got past the exception I was getting | 17:30 |
| ptoohill | whats the exception youre seeing | 17:30 |
| openstackgerrit | Kobi Samoray proposed openstack/octavia: Implement custom header support for Octavia https://review.openstack.org/257901 | 17:30 |
| Frito | I don't have it handy at the moment but it was something to the tune of "Instance build failed. Network id not in the correct format" | 17:31 |
| ptoohill | i see, hmm you sure youre on latest? | 17:31 |
| Frito | Yep. | 17:31 |
| *** amotoki has joined #openstack-lbaas | 17:31 | |
| Frito | latest devstack and latest octavia | 17:31 |
| ptoohill | i was looking in github on master, your lines are different somehow then | 17:32 |
| Frito | weird | 17:32 |
| Frito | let me to a git tree and grab you the hash I'm on | 17:32 |
| Frito | 81fc37c | 17:32 |
| ptoohill | you should see my commit as the latest it looks like | 17:32 |
| Frito | from 4 days ago | 17:32 |
| ptoohill | 4230e00 | 17:32 |
| ptoohill | oh, then hrmm | 17:32 |
| madhu_ak_ | @Frito could you message what error you seeing at o-cw logs? | 17:33 |
| Frito | madhu_ak_: one sec, let me pull my debug stuff out | 17:33 |
| ptoohill | latest commit i see if from 11 days ago on master in github, hrmmmm | 17:33 |
| madhu_ak_ | sure | 17:33 |
| ptoohill | well for that file of course, but yea, ok i see the one you pasted, hrmmm | 17:34 |
| ptoohill | thats still my commit, just the gerrit commit hash i guess | 17:34 |
| ptoohill | can you show me what your amp_boot_net_list option contains | 17:35 |
| ptoohill | wondering if thats getting generated bad | 17:35 |
| Frito | madhu_ak_: 2016-04-25 12:34:53.881 10786 ERROR octavia.compute.drivers.nova_driver BadRequest: Bad networks format: network uuid is not in proper format ([30b3da1c-b456-4c16-90ec-a572789f64f9]) (HTTP 400) (Request-ID: req-bb4ffa15-f010-46fe-b55d-19ed81e785d8) | 17:35 |
| Frito | ptoohill: That's what I was wondering as well. | 17:36 |
| ptoohill | *disclaimer, this worked with a basic build via devstack, i didnt do any crazy testing on it, so i probably misssed something :( | 17:36 |
| madhu_ak_ | correct. Is the eror is same as this one: http://logs.openstack.org/53/306053/3/experimental/gate-tempest-dsvm-octavia/9162107/logs/screen-o-cw.txt.gz? | 17:36 |
| ptoohill | yay bugs! | 17:36 |
| madhu_ak_ | yesterday I was tryign to run our octavia scenario tests in our gate :) I found this early morning :) | 17:36 |
| openstackgerrit | Merged openstack/neutron-lbaas: Port more unit tests to Python 3 https://review.openstack.org/308204 | 17:36 |
| Frito | madhu_ak_: Yep, That looks suspiciously similar to me :-D | 17:37 |
| madhu_ak_ | currently I am stacking up devstack and will run octavia cenario tests. Lets see whether I can repoduce locally | 17:37 |
| ptoohill | yea, looks like the config isnt being generated correctly, i wonder if they chaged the osol req for list, the docs showed to submit with [] but... its not looking like it likes that. This worked at one point prior to merge, nothing was failing then | 17:37 |
| ptoohill | fun stuff | 17:37 |
| ptoohill | im sure its easily reproduceable | 17:38 |
| ptoohill | can you gist your config Frito | 17:38 |
| ptoohill | hopefully this is just a devstack plugin fix | 17:38 |
| Frito | Um. Sure? What config? | 17:39 |
| ptoohill | octavia | 17:39 |
| Frito | /etc/octavia/octavia.conf ? | 17:39 |
| ptoohill | for single it might need a ',' or something at the end, this might be funkiness with the option itself and how i did it in the plugin :( | 17:39 |
| ptoohill | yea | 17:39 |
| Frito | Yea, one sec | 17:40 |
| ptoohill | im looking at mine, im pretty sure its the same | 17:40 |
| Frito | Probably. I haven't changed anything in that config that I recall | 17:40 |
| ptoohill | blah, going to read through those docs again real quick and do some testing. This is my issue, i just love how it works prior to merge, during merge and then breaks a week later | 17:40 |
| ptoohill | yea, amp_boot_network_list = [04163e33-b44f-423d-8d2b-5010791a2590] it should look like that for you too | 17:41 |
| ptoohill | it probably needs a ',' or something | 17:41 |
| ptoohill | :/ | 17:41 |
| ptoohill | could try that real quick if you wanted :P | 17:41 |
| ptoohill | amp_boot_network_list = [04163e33-b44f-423d-8d2b-5010791a2590,] | 17:41 |
| ptoohill | heh | 17:41 |
| Frito | Yea, same thing | 17:41 |
| ptoohill | k, then im not sure why this decided to start failing... lmao.. | 17:42 |
| ptoohill | why :( | 17:42 |
| Frito | without the , and a different GUID but *shrugs* the same for all purposes | 17:42 |
| ptoohill | k, then im not sure whats wrong with it at this point, im stacking | 17:42 |
| Frito | amp_boot_network_list = [30b3da1c-b456-4c16-90ec-a572789f64f9] | 17:42 |
| Frito | Does the comma fix it? | 17:42 |
| ptoohill | oh, not sure | 17:42 |
| ptoohill | im not stacked yet | 17:43 |
| Frito | rgr. Are you not at summit? | 17:43 |
| ptoohill | this worked before and during merge, either something changed, or magic happened during the review process | 17:43 |
| ptoohill | not yet | 17:43 |
| *** amotoki has quit IRC | 17:44 | |
| ptoohill | If you wanted to play with that format in config to see if you get something to pick up there for the time being that'd be awesome | 17:44 |
| ptoohill | im going to read a bit while this is stacking | 17:44 |
| Frito | I tried putting a comma at the end of that [id,] and building another loadbalancer but it didn't work | 17:48 |
| madhu_ak_ | ptoohill, config is at: http://logs.openstack.org/53/306053/3/experimental/gate-tempest-dsvm-octavia/9162107/logs/etc/octavia/octavia.conf.txt.gz | 17:48 |
| *** madhu_ak_ is now known as madhu_ak | 17:49 | |
| ptoohill | k, based off docs that should have worked, so then it has to be the code itself. maybe the append, do you have amp_network populated still? | 17:50 |
| ptoohill | checking your config madhu_ak | 17:50 |
| Frito | I only restarted o-cw. Is there anything else I should of restarted? | 17:51 |
| Frito | Is anything cached? | 17:51 |
| ptoohill | thats it | 17:51 |
| ptoohill | shouldnt be | 17:51 |
| ptoohill | i had to have screwed something up, im just not sure why this didnt pop up before :/ | 17:51 |
| ptoohill | maybe it needs something like | 17:51 |
| ptoohill | amp_boot_net_list = [ | 17:51 |
| ptoohill | id... | 17:52 |
| ptoohill | ] | 17:52 |
| ptoohill | kinda like a multiopt, but.............. | 17:52 |
| ptoohill | HRMM | 17:52 |
| ptoohill | still stacking, ill play with it in a bit sorry about this guys, really not sure why this didnt fail before and during merge when all these same tests were run >< | 17:52 |
| Frito | All good. I'm just glad its happening on someone elses environment other than mando and myself. | 17:53 |
| ptoohill | :P | 17:53 |
| Frito | Every time we hit an issue it's just been an assumption that its something isolated to the way we set things up | 17:53 |
| ptoohill | yea, thanks for chiming in madhu_ak, wasnt aware this was failing elsewhere | 17:53 |
| ptoohill | Theres always something to deal with with devstack/openstack in general | 17:54 |
| ptoohill | sometimes its your fault, most of the time its because things are constantly moving. That's why a bunch of folks run in offline and only update every once in a while | 17:54 |
| Frito | That just seems to be a recipe for issues like this but worse | 17:57 |
| Frito | lol | 17:57 |
| madhu_ak | no problem ptoohill | 17:57 |
| ptoohill | i agree, but yea :/ | 17:57 |
| ptoohill | control test show same results, awesome, this was not the case before, i love things like this | 17:59 |
| ptoohill | debugging now | 17:59 |
| Frito | "I love things like this" makes me sense a healthy degree of sarcasm ;-) | 17:59 |
| ptoohill | :) | 18:00 |
| ptoohill | its exciting, i enjoy figuring out what i screwed up, best thing in the world! | 18:00 |
| ptoohill | hhmm | 18:04 |
| ptoohill | its like its reading it in as a string opt, wth | 18:06 |
| ptoohill | its defined as listopt, full on debug mode engage | 18:07 |
| ptoohill | or, it just doesnt need those brackets anymore, and validates what blogan said.. but.... wth why now why didnt it fail before AHH | 18:10 |
| ptoohill | so, to get by for now, remove the brackets :/ | 18:11 |
| ptoohill | well, hold on | 18:11 |
| ptoohill | ook, yea, now its complaining about image, so yea, remove brackets that should get you past this issue | 18:11 |
| ptoohill | ill submit a patch here shortly, not sure why/how this got past everything o.O | 18:11 |
| ptoohill | Frito: ^ | 18:12 |
| ptoohill | one more thing to verify/test then ill fix this | 18:12 |
| Frito | Okie dokie | 18:13 |
| *** diogogmt_ has joined #openstack-lbaas | 18:14 | |
| *** armax has joined #openstack-lbaas | 18:14 | |
| Frito | that patch is something that has to go through review and get merged and all that? | 18:14 |
| ptoohill | yes, so the tests that madhu_ak shown should have failed, all of those tests would have failed | 18:14 |
| ptoohill | i dont get it :/ | 18:14 |
| ptoohill | and docs showed the way i was doing it was right, but blogan brought up this point but was unsure.. so yea.. | 18:14 |
| ptoohill | FUN STUFF | 18:15 |
| ptoohill | or you mean my new patch? | 18:15 |
| ptoohill | if so, yes, but you can cherry pick it so you dont have to manually do that | 18:15 |
| ptoohill | *manually update code each time | 18:16 |
| ptoohill | until then | 18:16 |
| *** diogogmt has quit IRC | 18:16 | |
| *** diogogmt_ is now known as diogogmt | 18:16 | |
| ptoohill | not great, but itll work, again sorry, not sure how the heck this happened | 18:16 |
| Frito | No worreis. Like I said. I'm just glad it wasn't something that was localized to our envionments. | 18:16 |
| ptoohill | :) | 18:17 |
| ptoohill | alrighty, let's see if this fixes it :) | 18:18 |
| ptoohill | stack time | 18:18 |
| Frito | hehe | 18:19 |
| *** amotoki has joined #openstack-lbaas | 18:23 | |
| *** amotoki has quit IRC | 18:25 | |
| *** woodster_ has quit IRC | 18:28 | |
| ptoohill | madhu_ak: Frito Either one of you want to submit a bug for it? Ill do one shortly if not | 18:37 |
| ptoohill | making bug now | 18:38 |
| madhu_ak | https://bugs.launchpad.net/octavia/+bug/1574784 | 18:39 |
| openstack | Launchpad bug 1574784 in octavia "BadRequest: Bad networks format: network uuid is not in proper format when running octavia basic loadbalancer test" [Undecided,New] | 18:39 |
| madhu_ak | ptoohill, ^^ | 18:39 |
| Frito | sry,d idn't see. | 18:39 |
| ptoohill | ah, k, thank you | 18:41 |
| *** harlowja has joined #openstack-lbaas | 18:44 | |
| *** amotoki has joined #openstack-lbaas | 18:46 | |
| bogdan | ptoohill, I have amp_secgroup_list = 0ec65b33-f491-4631-bdd2-87d841d213a0 but my amphora still gets a new one generated lb-3582eb9... am I missing something? | 18:47 |
| bogdan | the ID above is the default security group in the service project | 18:47 |
| bogdan | i noticed that the lb-... generated ones are in service project too | 18:47 |
| ptoohill | hrmm | 18:47 |
| bogdan | which line should I debug to check what happens? | 18:48 |
| ptoohill | not quite sure | 18:48 |
| ptoohill | the compute_tasks is where those types of things get sent in to the amphora create | 18:49 |
| ptoohill | starting line 48 bogdan, you can see if the config value is what you expect. How are you deploying? | 18:49 |
| openstackgerrit | Phillip Toohill proposed openstack/octavia: Amphora fails to build https://review.openstack.org/310040 | 18:50 |
| bogdan | i am deploying on a standard openstack by cloning octavia and setting up | 18:50 |
| ptoohill | Frito: You can cherry pick that and restack assuming you dont have relone=True in localrc and it gets overwritten | 18:50 |
| bogdan | line 48 in which file? :) | 18:50 |
| bogdan | I guess I should debug if sec_grps and sec_grps.get('security_groups'): return sec_grps.get('security_groups')[0] | 18:52 |
| ptoohill | compute_tasks | 18:52 |
| bogdan | in allowed_address_pairs.py | 18:52 |
| bogdan | ok will come back once i debug it | 18:52 |
| Frito | ptoohill: Sure. I would say it would be in a couple but my machine is ssssssllllllllllloooooooooooooooowwwwwwwwwwwwwwwwwwwwwwwwwwwwwww | 18:52 |
| ptoohill | :) | 18:53 |
| Frito | That's just "git review -d 310040" right? | 18:53 |
| ptoohill | i always fully qualify with cherry-pick line, but that should work for you | 18:55 |
| Frito | Yea, I have no changes :-D | 18:55 |
| *** ducttape_ has joined #openstack-lbaas | 18:56 | |
| Frito | /and/ I'm not sure how to cherry pick you w/o having you set up as a remote | 18:56 |
| ptoohill | its in gerrit | 18:56 |
| *** bana_k has joined #openstack-lbaas | 18:56 | |
| ptoohill | https://review.openstack.org/#/c/310040/ | 18:57 |
| Frito | right | 18:57 |
| *** woodster_ has joined #openstack-lbaas | 18:57 | |
| Frito | do I do that from devstack or from /opt/stack/octavia? | 18:57 |
| ptoohill | Go there, then on left hand click the 'download' dropdown and grab the cherrypick link | 18:57 |
| Frito | devstack right? | 18:57 |
| ptoohill | octavia | 18:57 |
| Frito | ah, k, got ya | 18:57 |
| ptoohill | right hand side >< ** | 18:57 |
| Frito | Right | 18:59 |
| Frito | GOt it now | 18:59 |
| Frito | I had to git clean / git reset HEAD --hard | 18:59 |
| Frito | cleaning / restacking now | 19:00 |
| *** amotoki has quit IRC | 19:01 | |
| ptoohill | please feel free to submit a review once you test it out, +/-1 Frito. Have you done any reviews yet? | 19:01 |
| Frito | Not yet | 19:02 |
| Frito | :-) Will figure it out though | 19:02 |
| ptoohill | :) | 19:03 |
| bharathm | bogdan: lb-* secgroup is created by octavia for the VIP.. | 19:04 |
| bharathm | Nova amphora instance contains two security groups.. One is for management port (as defined in amp_secgroup_list in conf) and the other is for tenant port (VIP) that gets dynamically created during lb create | 19:06 |
| *** amotoki has joined #openstack-lbaas | 19:06 | |
| bogdan | ok, got it | 19:11 |
| *** sbalukoff has joined #openstack-lbaas | 19:13 | |
| *** armax has quit IRC | 19:15 | |
| bogdan | now when creating amphora a I get OverQuotaClient: Quota exceeded for resources: ['security_group'] | 19:20 |
| bogdan | how can I tell which quota exactly it is all about? | 19:21 |
| ptoohill | neutron quota-list bogdan, you should be able to increase that in the config or clean up any leftovers you dont need | 19:22 |
| bogdan | thanks ptoohill, it was the security groups per tenant 10 :) | 19:25 |
| Frito | ptoohill: Looks like that worked for me. Thanks. Trying to figure out how to add the +1 to the change | 19:29 |
| Frito | According to the Developers Guide there should be a "review" button on the UI or am I reading this wrong? | 19:30 |
| Frito | http://docs.openstack.org/infra/manual/developers.html#code-review | 19:31 |
| ptoohill | Yes, you should see a review button, are you logged in Frito ? | 19:32 |
| Frito | Yep @ logged in | 19:32 |
| ptoohill | oh, its 'reply' | 19:32 |
| ptoohill | at the top left-center-ish | 19:32 |
| Frito | Ahh, okay. Makes sense why I didn't see it then. I was looking for "Review" | 19:32 |
| Frito | Done | 19:33 |
| *** harlowja has quit IRC | 19:33 | |
| ptoohill | good deal :) | 19:33 |
| *** amotoki has quit IRC | 19:39 | |
| *** ducttape_ has quit IRC | 19:39 | |
| bogdan | it seems that both ports created for the amphora are getting assigned to the lb-* security group | 19:45 |
| bogdan | why is it not picking my amp_secgroup_list= setting? | 19:45 |
| *** ducttape_ has joined #openstack-lbaas | 19:45 | |
| *** harlowja has joined #openstack-lbaas | 19:46 | |
| *** yamamoto has joined #openstack-lbaas | 19:47 | |
| *** amotoki has joined #openstack-lbaas | 19:50 | |
| *** yamamoto has quit IRC | 20:07 | |
| *** yamamoto has joined #openstack-lbaas | 20:09 | |
| *** ggolin has quit IRC | 20:10 | |
| *** yamamoto has quit IRC | 20:12 | |
| ptoohill | sbalukoff: xgerman: johnsom: blogan: rm_work: https://review.openstack.org/#/c/310040/ somehow that decided to break, please take a peek when you can. See you guys in the morning :) | 20:12 |
| *** yamamoto has joined #openstack-lbaas | 20:14 | |
| *** yamamoto has quit IRC | 20:16 | |
| *** crc32 has joined #openstack-lbaas | 20:16 | |
| *** bana_k has quit IRC | 20:24 | |
| bogdan | any idea why both ports of an amphora get the lb-* security group and ignoring amp_secgroup_list setting? | 20:25 |
| *** yamamoto has joined #openstack-lbaas | 20:27 | |
| *** yamamoto has quit IRC | 20:28 | |
| *** ducttape_ has quit IRC | 20:29 | |
| *** amotoki has quit IRC | 20:31 | |
| *** harlowja has quit IRC | 20:31 | |
| *** sbalukoff has quit IRC | 20:33 | |
| *** yamamoto has joined #openstack-lbaas | 20:33 | |
| *** yamamoto has quit IRC | 20:34 | |
| *** amotoki has joined #openstack-lbaas | 20:38 | |
| *** ducttape_ has joined #openstack-lbaas | 20:40 | |
| bharathm | bogdan: I believe you are referring to both tenant ports (VIP & VRRP ports) of the amphora but not the management port? The "amp_secgroup_list" is the security group assigned for the managemnt port.. | 20:51 |
| bharathm | You can look up the management port as "neutron port-list | grep {mgmt ip address from nova list output}" | 20:52 |
| *** ducttape_ has quit IRC | 21:02 | |
| bogdan | bharathm, here is what I observe: right after lb creation if I look at sec group of the amphora nova instance I see the security group that I assigned via amp_secgroup_list, but after 1-2 seconds the instance gets its security group switched to the lb-*... sec group... | 21:08 |
| bogdan | why is that happening? | 21:09 |
| bharathm | Has the LB become ACTIVE? If not because of some error, the revert process would have cleaned that up | 21:11 |
| bharathm | bogdon: | 21:12 |
| bogdan | no, the LB stays Offline because it is in " Could not connect to instance. Retrying" due to the fact that I am not providing a valid amphora image (no haproxy , etc) | 21:15 |
| bogdan | but the question is why the security group of the amphora nova instance is getting changed right after instance creation | 21:15 |
| bogdan | will it get reverted back to amp_secgroup_list after the amphora is all up? | 21:16 |
| bogdan | I mean there is not reverting happening yet | 21:16 |
| bogdan | at least I do not see it in the logs | 21:17 |
| bharathm | bogdan: If the revert hasn't initiated yet, then something's wrong with the subnet you passed to lb create.. (I am guessing here).. Because once I accidentally did "neutron lbaas-loadbalaner-crete --name lb1 external-subnet" instead of using private-subnet that resulted in more or less the same outcome as you see | 21:21 |
| bogdan | I am using private subnet : neutron lbaas-loadbalancer-create --name lb26 sub | 21:24 |
| bogdan | bharathm, what lines to debug to check when the security group of the nova instance gets changed? | 21:24 |
| bharathm | Hmm.. can you try an image built for octavia (with haproxy etc packages) | 21:25 |
| bogdan | where can I get one from? | 21:25 |
| bharathm | bogdan: you can build one with the command in https://github.com/openstack/octavia/tree/master/diskimage-create | 21:26 |
| bharathm | Or if you have a running devstack with octavia, you can copy the already built image | 21:26 |
| bogdan | I might try but I am not sure if I will get it soon because I am on SLES, not Ubuntu | 21:26 |
| bogdan | I am not on devstack | 21:26 |
| bharathm | Oh.. Well for testing purposes you can try an ubuntu amphora image :-) | 21:28 |
| bogdan | btw, my octavia.conf has lb_network_name = sub | 21:28 |
| bharathm | that diskimage-create.sh will download and build the image from ubuntu mirror.. | 21:28 |
| bogdan | would it be a problem if I use the same subnet both for lb_network_name = and for lb create command? | 21:28 |
| bogdan | ok, but how can I build ubuntu image on SLES? | 21:29 |
| bogdan | ah, ok, will try it later but from my understanding the process of waiting ..WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying. is still on | 21:29 |
| bogdan | so, why would my security group get changed | 21:30 |
| bogdan | would it be a problem that I use the same subnet for amphora communication and for lb creation, could this be the cause? | 21:31 |
| bogdan | ok now some minutes later I got the REVERTED state and what happened is that the amphora instance got its IP Address and Security Group completely removed | 21:35 |
| bharathm | bogdan: I would say most likely using the same subnet is the cause.. As I mentioned before, I tried using the same network for create and management and I noticed the similar outcome wrt sec groups.. | 21:36 |
| bharathm | Ideally the management network and LB network should be different... Former is control plane and the later is for data.. I never extensively tried both on the same network but I am sure it would cause issues | 21:37 |
| *** yamamoto has joined #openstack-lbaas | 21:38 | |
| bogdan | ok, let me try with different networks | 21:41 |
| *** yamamoto has quit IRC | 21:46 | |
| openstackgerrit | min wang proposed openstack/octavia: Add failover scenario test for octavia https://review.openstack.org/309625 | 21:47 |
| bogdan | big thanks bharathm!!! using different networks for mgmt and for lb did work as expected - the instance got the amp_secgroup_list and in addition it got the lb-* group | 21:50 |
| *** cgross has quit IRC | 22:02 | |
| bharathm | thats great bogdan .. glad things worked out | 22:02 |
| *** cgross has joined #openstack-lbaas | 22:05 | |
| *** harlowja has joined #openstack-lbaas | 22:21 | |
| openstackgerrit | Franklin Naval proposed openstack/neutron-lbaas: WIP: Neutron-LBaaS: Member Scenario Test https://review.openstack.org/310069 | 22:21 |
| *** armax has joined #openstack-lbaas | 22:25 | |
| *** minwang2 has joined #openstack-lbaas | 22:31 | |
| *** yamamoto_ has joined #openstack-lbaas | 22:52 | |
| *** crc32 has quit IRC | 22:53 | |
| *** yamamoto_ has quit IRC | 22:54 | |
| *** harlowja has quit IRC | 23:08 | |
| *** amotoki has quit IRC | 23:12 | |
| *** minwang2 has quit IRC | 23:15 | |
| *** bogdan has quit IRC | 23:17 | |
| openstackgerrit | Franklin Naval proposed openstack/neutron-lbaas: Neutron-LBaaS: Member Scenario Test https://review.openstack.org/310069 | 23:21 |
| *** diogogmt has quit IRC | 23:21 | |
| *** fnaval_ has quit IRC | 23:21 | |
| *** neelashah has joined #openstack-lbaas | 23:23 | |
| *** neelashah has quit IRC | 23:30 | |
| *** fnaval has joined #openstack-lbaas | 23:31 | |
| openstackgerrit | Merged openstack/neutron-lbaas: Port more service unit tests to Python 3 https://review.openstack.org/308221 | 23:39 |
| *** chlong has joined #openstack-lbaas | 23:40 | |
| openstackgerrit | Merged openstack/neutron-lbaas: Port even more service tests to Python 3 https://review.openstack.org/308234 | 23:43 |
| openstackgerrit | Merged openstack/neutron-lbaas: Port test_db_loadbalancer to Python 3 https://review.openstack.org/308239 | 23:43 |
| openstackgerrit | Lingxian Kong proposed openstack/octavia: Add timestamp to octavia resources https://review.openstack.org/309253 | 23:47 |
| *** woodster_ has quit IRC | 23:48 | |
| *** ajmiller has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!