| openstackgerrit | Bar RH proposed openstack/python-octaviaclient master: Complement Octavia client with a set of features https://review.openstack.org/522666 | 00:27 |
|---|---|---|
| openstackgerrit | Bar RH proposed openstack/python-octaviaclient master: Complement Octavia client with a set of features https://review.openstack.org/522666 | 00:37 |
| openstackgerrit | Bar RH proposed openstack/python-octaviaclient master: Complement Octavia client with a set of features https://review.openstack.org/522666 | 00:38 |
| *** yuanying has quit IRC | 00:59 | |
| *** bar_ has quit IRC | 01:03 | |
| *** yuanying has joined #openstack-lbaas | 01:12 | |
| *** yamamoto has joined #openstack-lbaas | 01:29 | |
| sapd_ | When I use a normal user with user role, I can't get loadbalancer in octavia using this command "openstack loadbalancer list". With the error: "Forbidden (HTTP 403) (Request-ID: req-1681ffa9-8622-4712-b827-c8d6b2034a25)" | 02:30 |
| *** sam_dw has quit IRC | 02:31 | |
| johnsom | o/ | 02:31 |
| johnsom | So, by default Octavia has RBAC enabled and users need to be a member of the loadbalancer member role | 02:32 |
| johnsom | There is a doc that explains this in the octavia documentation under configuration and policy config | 02:32 |
| johnsom | There is also a sample policy.json file in the etc folder that disables the “new” OpenStack RBAC scheme and sets it back to admin and owner | 02:33 |
| johnsom | Also, fyi, the Octavia API reference describes those error codes and would have pointed you towards the policy config | 02:35 |
| johnsom | sapd_: https://docs.openstack.org/octavia/pike/configuration/policy.html | 02:38 |
| sapd_ | johnsom, I can't find policy.json in etc folder. Could you share it for me? | 02:47 |
| johnsom | sapd_: https://github.com/openstack/octavia/blob/master/etc/policy/admin_or_owner-policy.json | 02:48 |
| *** threestrands_ has joined #openstack-lbaas | 02:49 | |
| *** threestrands_ has quit IRC | 02:49 | |
| *** threestrands_ has joined #openstack-lbaas | 02:49 | |
| *** threestrands_ has quit IRC | 02:50 | |
| *** threestrands has quit IRC | 02:50 | |
| *** threestrands_ has joined #openstack-lbaas | 02:50 | |
| *** threestrands_ has quit IRC | 02:50 | |
| *** threestrands_ has joined #openstack-lbaas | 02:50 | |
| sapd_ | Thanks johnsom,. So Octavia still have not finished code with "new" Openstack RBAC scheme. | 02:53 |
| johnsom | No, Octavia is complete for RBAC | 02:54 |
| johnsom | You just need to give your users the role for the “new” as described on the docs page | 02:56 |
| *** atoth_ has quit IRC | 03:00 | |
| sapd_ | Thanks johnsom , Awesome :D | 03:06 |
| *** links has joined #openstack-lbaas | 04:26 | |
| *** yamamoto has quit IRC | 04:43 | |
| *** yamamoto has joined #openstack-lbaas | 04:49 | |
| *** longkb has joined #openstack-lbaas | 05:08 | |
| longkb | Hi everyone, | 05:08 |
| longkb | I am deploying devstack which include neutron-loadbancer with octavia. However, when I tried to create a loadbalancer with command "neutron lbaas-loadbalancer-create --name lb subnet_mgmt", the creation status always is "PENDING_CREATE" | 05:11 |
| longkb | Could anyone help me to figure out what is an error? Thank you very much | 05:11 |
| johnsom | What are you running devstack on? My bet is virtualbox | 05:13 |
| longkb | I am using vmware | 05:13 |
| longkb | Is there any difference between vmware and virtualbox? | 05:14 |
| johnsom | Ok good. Usually this is caused by virtualbox taking 10+ minutes to boot a vm | 05:15 |
| johnsom | In vmware, make sure your CPU config has virtualization enabled | 05:15 |
| johnsom | In workstation it is under settings, cpu, a checkbox for vt-x | 05:16 |
| longkb | Yes, I enabled the CPU virtualization. | 05:16 |
| longkb | I can create VM sucessfully | 05:16 |
| johnsom | Ok, so the typical issue is ruled out | 05:16 |
| johnsom | Yeah, if that isn’t on it can take 10+ minutes for the LB to go active | 05:17 |
| johnsom | So, next step would be to do a nova list to see it the amphora is booted up | 05:18 |
| longkb | Yes, I have checked. I amphora is still ACTIVE | 05:19 |
| johnsom | Next, check the octavia worker process log. Assuming you are running master branch devstack that is “journalctl -u devstack@o-cw | less” | 05:20 |
| longkb | I got an error | 05:21 |
| johnsom | I would also note, with pike or newer you no longer need neutron-lbaas | 05:21 |
| longkb | Could you explain more about "no longer need neutron-lbaas" :) | 05:22 |
| johnsom | Neutron-lbaas is heading towards deprecation. Octavia is replacing it | 05:22 |
| johnsom | Octavia had it’s own API endpoit | 05:23 |
| longkb | oh. I see. | 05:23 |
| johnsom | Can you pastebin your error so I can look at it for you? | 05:24 |
| longkb | This is my error | 05:25 |
| longkb | "WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectionError: HTTPSConnectionPool(host='192.1 68.0.12', port=9443): Max retries exceeded with url: /0.5/plug/vip/192.168.120.10 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb3b5aea3d0>: Failed to esta blish a new connection: [Errno 113] No route to host',))" | 05:25 |
| longkb | please help me to fix it! | 05:27 |
| johnsom | So, those warnings are normal while Octavia waits for nova/qemu to finish booting the vm. It could also mean there is a network configuration issue which would be very odd with devstack. | 05:28 |
| longkb | as I can see. My o-cw thread went to an infinitive loop | 05:30 |
| johnsom | Can you do a “nova list | grep amphora”, look at the lb-mgmt-net IP address. Make sure this does not conflict with the network your vmware is using. | 05:31 |
| johnsom | No infinite loop, it will keep trying connect to the VM until the timeout expires, then it will go to ERROR status | 05:32 |
| *** sticker has quit IRC | 05:33 | |
| longkb | my lb-mgmt-net address is 192.168.0.12 | 05:34 |
| longkb | And there is no existing network is confict with lb-mgmt-net | 05:34 |
| johnsom | Hmmm. Ok, get the nova uuid for the instance. Cd to /opt/stack/data/nova/instances/<uuid>/ | 05:36 |
| johnsom | There is a console.log file, look at the bottom of the file. It should be a login prompt if nova/qemu finished booting it | 05:37 |
| longkb | I got "Permission denided" | 05:38 |
| johnsom | Sudo | 05:39 |
| longkb | this console.log is empty | 05:39 |
| *** eN_Guruprasad_Rn has joined #openstack-lbaas | 05:40 | |
| johnsom | So, nova is broken. | 05:40 |
| *** eN_Guruprasad_Rn has joined #openstack-lbaas | 05:40 | |
| longkb | should I restart nova service? | 05:41 |
| johnsom | That log should be huge | 05:41 |
| johnsom | Try “openstack server show <uuid>” | 05:42 |
| johnsom | There is usually an error message section for the instance | 05:42 |
| *** yamamoto has quit IRC | 05:44 | |
| longkb | https://drive.google.com/file/d/1v7djaiNP_Y6ILq5zbuDJ35ofakBCd-48/view?usp=sharing | 05:45 |
| longkb | The instance stattus is still ACTIVE | 05:45 |
| longkb | You can see in the link | 05:46 |
| *** yamamoto has joined #openstack-lbaas | 05:48 | |
| johnsom | Can you run “top” and see if qemu is at 100% cpu usage? | 06:00 |
| longkb | it over 100% | 06:03 |
| longkb | It is 132% CPU | 06:03 |
| johnsom | Ok, I know the problem. | 06:04 |
| johnsom | The qemu is for a newer version than the OS you are running and you need to set a nova setting to make qemu boot a vm | 06:05 |
| johnsom | One minute, I will post the setting | 06:06 |
| longkb | Thank you very much | 06:06 |
| *** Alex_Staf has joined #openstack-lbaas | 06:06 | |
| johnsom | Assuming you are using ubuntu xenial for your devstack host: | 06:08 |
| johnsom | https://www.irccloud.com/pastebin/z7E7oUTq/ | 06:08 |
| johnsom | This is my "fixnova.sh" | 06:08 |
| johnsom | Once that runs, you can create a new load balancer and it will come up. | 06:09 |
| johnsom | The current one will need to go into ERROR status, then you can delete it. | 06:09 |
| johnsom | It probably already is in ERROR by now. | 06:09 |
| johnsom | The 100% CPU is the clue that qemu didn't actually boot the instance and it's just CPU spinning | 06:11 |
| longkb | In my devstack, there is no file source /opt/stack/devstack/inc/ini-config | 06:12 |
| johnsom | Hmm | 06:13 |
| johnsom | Well, you can manually edit the files | 06:13 |
| johnsom | it's file: /etc/nova/nova.conf, section [libvirt] key hw_machine_type value x86_64=pc-i440fx-xenial | 06:14 |
| johnsom | Same for the other file | 06:14 |
| johnsom | The source line just pulls in the the "iniset" command | 06:15 |
| longkb | Thanks you very much | 06:20 |
| longkb | I can create loadbalancer sucessfully with your instruction | 06:20 |
| johnsom | Excellent! I hope you enjoy Octavia | 06:21 |
| longkb | Thanks for your helping | 06:22 |
| longkb | I am implementing a bp that related to clustering serivce for VNF in Tacker | 06:22 |
| johnsom | Time for me to sign off for the night. Catch you tomorrow | 06:23 |
| longkb | Thank you very much! Johnsom ^_^ | 06:23 |
| longkb | good nice! | 06:23 |
| longkb | good night! | 06:23 |
| *** annp has joined #openstack-lbaas | 06:39 | |
| *** spectr has joined #openstack-lbaas | 06:44 | |
| *** rcernin has quit IRC | 06:47 | |
| *** rcernin has joined #openstack-lbaas | 06:49 | |
| *** spectr has quit IRC | 06:55 | |
| *** rcernin has quit IRC | 07:28 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 07:57 | |
| *** aojea has joined #openstack-lbaas | 08:01 | |
| *** yamamoto has quit IRC | 08:04 | |
| *** aojea has quit IRC | 08:06 | |
| *** pcaruana has joined #openstack-lbaas | 08:11 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: [DO NOT MERGE]: CI TEST https://review.openstack.org/523053 | 08:19 |
| *** slaweq has joined #openstack-lbaas | 08:24 | |
| *** kobis has joined #openstack-lbaas | 08:24 | |
| *** yamamoto has joined #openstack-lbaas | 08:33 | |
| *** yamamoto has quit IRC | 08:51 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: Support UDP load balance https://review.openstack.org/503606 | 09:15 |
| *** yamamoto has joined #openstack-lbaas | 09:31 | |
| *** spectr has joined #openstack-lbaas | 09:37 | |
| *** spectr has quit IRC | 09:37 | |
| *** bar_ has joined #openstack-lbaas | 09:38 | |
| *** yamamoto has quit IRC | 09:39 | |
| *** annp has quit IRC | 10:03 | |
| *** spectr has joined #openstack-lbaas | 10:22 | |
| *** spectr has quit IRC | 10:23 | |
| *** yamamoto has joined #openstack-lbaas | 10:27 | |
| *** yamamoto has quit IRC | 10:32 | |
| *** yamamoto has joined #openstack-lbaas | 10:53 | |
| *** yamamoto has quit IRC | 10:55 | |
| *** pcaruana has quit IRC | 10:55 | |
| *** cgoncalves has quit IRC | 10:56 | |
| *** cgoncalves has joined #openstack-lbaas | 10:57 | |
| *** cgoncalves has quit IRC | 10:57 | |
| *** cgoncalves has joined #openstack-lbaas | 10:58 | |
| *** pcaruana has joined #openstack-lbaas | 10:59 | |
| *** salmankhan has joined #openstack-lbaas | 10:59 | |
| *** yamamoto has joined #openstack-lbaas | 11:01 | |
| openstackgerrit | Bernard Cafarelli proposed openstack/octavia master: Allow amphora agent install from distribution packages https://review.openstack.org/522626 | 11:03 |
| *** sri_ has joined #openstack-lbaas | 11:06 | |
| *** yamamoto has quit IRC | 11:06 | |
| *** salmankhan has quit IRC | 11:10 | |
| *** yamamoto has joined #openstack-lbaas | 11:31 | |
| *** spectr has joined #openstack-lbaas | 11:35 | |
| *** spectr has quit IRC | 11:35 | |
| *** yamamoto has quit IRC | 11:36 | |
| *** threestrands_ has quit IRC | 11:42 | |
| *** atoth has joined #openstack-lbaas | 11:49 | |
| bcafarel | nmagnezi: don't be surprised if you build a CentOS/RDO amphora with ^, it won't do much lbaas as selinux is enabled with the '-p' flag | 11:50 |
| *** kong has quit IRC | 11:53 | |
| *** ying_zuo has quit IRC | 11:53 | |
| *** fyxim has quit IRC | 11:54 | |
| *** xgerman_ has quit IRC | 11:54 | |
| *** redondo-mk has quit IRC | 11:54 | |
| *** kong has joined #openstack-lbaas | 11:55 | |
| *** fyxim has joined #openstack-lbaas | 11:56 | |
| *** redondo-mk has joined #openstack-lbaas | 11:56 | |
| *** xgerman_ has joined #openstack-lbaas | 11:56 | |
| *** yamamoto has joined #openstack-lbaas | 12:01 | |
| *** yamamoto has quit IRC | 12:06 | |
| *** links has quit IRC | 12:29 | |
| *** links has joined #openstack-lbaas | 12:43 | |
| *** yamamoto has joined #openstack-lbaas | 12:47 | |
| *** yamamoto has quit IRC | 12:51 | |
| *** krypto has joined #openstack-lbaas | 13:19 | |
| *** krypto has quit IRC | 13:19 | |
| *** krypto has joined #openstack-lbaas | 13:19 | |
| *** eN_Guruprasad_Rn has quit IRC | 13:41 | |
| *** yamamoto has joined #openstack-lbaas | 13:45 | |
| *** yamamoto has quit IRC | 13:45 | |
| *** yamamoto has joined #openstack-lbaas | 13:49 | |
| *** ying_zuo has joined #openstack-lbaas | 13:51 | |
| *** yamamoto has quit IRC | 13:57 | |
| *** fnaval has joined #openstack-lbaas | 14:16 | |
| *** links has quit IRC | 14:32 | |
| *** krypto has quit IRC | 14:35 | |
| openstackgerrit | Bar RH proposed openstack/octavia master: Rename 'remove' to 'reset' in Quota API https://review.openstack.org/522402 | 14:53 |
| *** bar_ has quit IRC | 14:55 | |
| *** salmankhan has joined #openstack-lbaas | 14:57 | |
| *** yamamoto has joined #openstack-lbaas | 14:57 | |
| *** yamamoto has quit IRC | 15:04 | |
| *** gcheresh has quit IRC | 15:12 | |
| xgerman_ | o/ | 15:13 |
| *** Alex_Staf has quit IRC | 15:13 | |
| *** salmankhan has quit IRC | 15:34 | |
| *** salmankhan has joined #openstack-lbaas | 15:51 | |
| *** slaweq has quit IRC | 15:58 | |
| *** slaweq has joined #openstack-lbaas | 15:59 | |
| dayou | https://review.openstack.org/#/c/522962/ | 15:59 |
| *** slaweq has quit IRC | 15:59 | |
| *** slaweq has joined #openstack-lbaas | 16:00 | |
| dayou | It seems there are some mis alignment between our v2 api and the openstack sdks | 16:00 |
| *** slaweq has quit IRC | 16:00 | |
| dayou | Not sure there are more of this and my way to fix it is correct or not | 16:00 |
| *** slaweq has joined #openstack-lbaas | 16:00 | |
| *** kobis has quit IRC | 16:01 | |
| *** slaweq has quit IRC | 16:05 | |
| *** bbzhao has quit IRC | 16:09 | |
| *** bzhao has quit IRC | 16:10 | |
| *** bzhao has joined #openstack-lbaas | 16:14 | |
| *** bbzhao has joined #openstack-lbaas | 16:15 | |
| johnsom | dayou No, someone mistakenly added neutron-lbaas objects to openstacksdk. We just need to delete them | 16:16 |
| johnsom | dayou Oh, whoops. Monday morning. You are correct, I was thinking openstackclient not sdk. | 16:19 |
| johnsom | Ignore me until I get my coffee | 16:19 |
| *** longstaff has joined #openstack-lbaas | 16:35 | |
| *** eN_Guruprasad_Rn has joined #openstack-lbaas | 16:52 | |
| *** salmankhan has quit IRC | 16:59 | |
| *** AlexeyAbashkin has quit IRC | 16:59 | |
| *** slaweq has joined #openstack-lbaas | 17:08 | |
| *** pcaruana has quit IRC | 17:09 | |
| *** slaweq has quit IRC | 17:13 | |
| nmagnezi | johnsom, o/ | 17:21 |
| johnsom | nmagnezi o/ | 17:21 |
| nmagnezi | johnsom, tested your patch again against CentOS and it looked okay to me | 17:21 |
| nmagnezi | but not sure what makes it to fail CI | 17:22 |
| johnsom | Ok, I will take a look. It was passing last week.... | 17:22 |
| johnsom | Ha, yeah, that is odd, passes on centos, but fails on ubuntu | 17:25 |
| *** harlowja has joined #openstack-lbaas | 17:47 | |
| *** armax has joined #openstack-lbaas | 17:53 | |
| *** longstaf_ has joined #openstack-lbaas | 17:59 | |
| *** longstaff has quit IRC | 18:03 | |
| johnsom | nmagnezi If you have a few minutes: https://review.openstack.org/#/c/517455/ | 18:05 |
| johnsom | I know there is a user looking forward to that patch | 18:05 |
| *** sshank has joined #openstack-lbaas | 18:17 | |
| *** sshank has quit IRC | 18:24 | |
| *** eN_Guruprasad_Rn has quit IRC | 18:25 | |
| *** sshank has joined #openstack-lbaas | 18:31 | |
| *** sshank has quit IRC | 18:33 | |
| *** salmankhan has joined #openstack-lbaas | 18:33 | |
| *** sshank has joined #openstack-lbaas | 18:33 | |
| *** sshank has quit IRC | 18:33 | |
| *** sshank has joined #openstack-lbaas | 18:34 | |
| *** sshank has quit IRC | 18:34 | |
| *** sshank has joined #openstack-lbaas | 18:34 | |
| *** longstaff has joined #openstack-lbaas | 18:35 | |
| *** sshank has quit IRC | 18:37 | |
| *** longstaf_ has quit IRC | 18:38 | |
| *** sshank has joined #openstack-lbaas | 18:40 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 18:52 | |
| *** gcheresh has joined #openstack-lbaas | 18:52 | |
| *** AlexeyAbashkin has quit IRC | 18:56 | |
| *** longstaff has quit IRC | 19:01 | |
| *** longstaff has joined #openstack-lbaas | 19:02 | |
| *** pcaruana has joined #openstack-lbaas | 19:16 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 19:17 | |
| *** sanfern has joined #openstack-lbaas | 19:22 | |
| *** AlexeyAbashkin has quit IRC | 19:22 | |
| *** bar_ has joined #openstack-lbaas | 19:34 | |
| *** sshank has quit IRC | 19:35 | |
| openstackgerrit | Santhosh Fernandes proposed openstack/octavia master: [WIP] ACTIVE-ACTIVE rename vrrp_ to frontend_ https://review.openstack.org/521138 | 19:47 |
| *** sanfern has quit IRC | 19:47 | |
| *** sanfern has joined #openstack-lbaas | 19:47 | |
| *** sanfern has quit IRC | 19:48 | |
| *** sanfern has joined #openstack-lbaas | 19:48 | |
| *** sanfern has quit IRC | 19:49 | |
| *** sanfern has joined #openstack-lbaas | 19:49 | |
| *** sanfern has quit IRC | 19:49 | |
| *** sanfern has joined #openstack-lbaas | 19:50 | |
| *** sanfern has quit IRC | 19:50 | |
| *** salmankhan has quit IRC | 19:54 | |
| *** atoth has quit IRC | 19:56 | |
| rm_work | hey all :) back | 19:56 |
| rm_work | any status update? | 19:56 |
| johnsom | o/ | 19:56 |
| johnsom | Lots of patches up for review. I'm just getting back into things. Caught up on my 150 emails this morning... | 19:57 |
| rm_work | i JUST caught up on slack and IRC | 19:57 |
| rm_work | though i'm not sure i've fully processed everything I read... | 19:57 |
| johnsom | I'm going to heat up some lunch and work on some reviews. | 19:59 |
| rm_work | k | 19:59 |
| rm_work | i'm going to make sure my deploy didn't explode while i was away | 19:59 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 20:17 | |
| *** slaweq has joined #openstack-lbaas | 20:19 | |
| *** slaweq has quit IRC | 20:20 | |
| *** slaweq has joined #openstack-lbaas | 20:20 | |
| *** AlexeyAbashkin has quit IRC | 20:21 | |
| xgerman_ | o/ | 20:24 |
| *** threestrands_ has joined #openstack-lbaas | 20:32 | |
| *** threestrands_ has quit IRC | 20:32 | |
| *** threestrands_ has joined #openstack-lbaas | 20:32 | |
| *** threestrands_ has quit IRC | 20:33 | |
| *** threestrands_ has joined #openstack-lbaas | 20:34 | |
| *** threestrands has joined #openstack-lbaas | 20:34 | |
| *** threestrands has quit IRC | 20:34 | |
| *** threestrands has joined #openstack-lbaas | 20:34 | |
| *** sshank has joined #openstack-lbaas | 20:43 | |
| *** threestrands_ has quit IRC | 20:44 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Optimize update_health process https://review.openstack.org/504875 | 20:49 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 20:50 |
| rm_work | ^^ rebases | 20:50 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Merge amphora-agent elements https://review.openstack.org/518992 | 20:54 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Merge amphora-agent elements https://review.openstack.org/518992 | 20:55 |
| johnsom | rm_work I think something went wrong with that last change... | 20:57 |
| rm_work | hmmm | 20:57 |
| rm_work | this install-type stuff is interesting | 20:57 |
| rm_work | really we should just wipe out the rhel element i think, and fix it to use this | 20:57 |
| johnsom | The build-essentials got dropped | 20:57 |
| rm_work | yeah i did that | 20:57 |
| johnsom | Yes, that is my argument and -1 | 20:57 |
| rm_work | it doesn't need to be pkg-mapped | 20:57 |
| johnsom | Ah | 20:57 |
| rm_work | as bernard said, i was overriding the default pkg-mapping | 20:58 |
| rm_work | which i didn't know was a thing | 20:58 |
| johnsom | Got it | 20:58 |
| rm_work | but it is -- so trying to futz with build-essential is just a bad idea | 20:58 |
| rm_work | hmm though maybe I should have left it for the *specific* rhel7 distro | 20:58 |
| johnsom | Yeah. But I think the install type path is probably best for the pre-packaged bits | 20:58 |
| rm_work | yep that is awesome functionality i didn't know existed :P | 20:59 |
| rm_work | I might turn that patch over to them | 20:59 |
| rm_work | or just abandon and have them write the merge that way | 20:59 |
| rm_work | johnsom: i'm pretty close on the gentoo thing, just need to do the osutils bits | 21:02 |
| rm_work | i think it will boot a lot faster | 21:02 |
| rm_work | and we'll get newer HAProxy from it | 21:02 |
| rm_work | I think | 21:02 |
| xgerman_ | I think - famous last words | 21:03 |
| rm_work | but not sure if i'll have time to finish in the next few weeks, sadly | 21:03 |
| rm_work | lol yes | 21:03 |
| johnsom | Cool. | 21:03 |
| johnsom | FYI, Q2 is next week.... | 21:04 |
| xgerman_ | so soon… | 21:04 |
| johnsom | Yeah, no kidding | 21:04 |
| bar_ | hey, I suspect loadbalancer-failover is broken (loadbalancer is stuck in ERROR). Is there test/gate to prove me otherwise? | 21:11 |
| *** sshank has quit IRC | 21:12 | |
| johnsom | bar_ It's here: https://review.openstack.org/#/c/501559/ | 21:14 |
| bar_ | johnsom, :-( | 21:14 |
| johnsom | What do the logs have to say? | 21:14 |
| bar_ | I've just came across this while extending octavia client | 21:16 |
| *** sshank has joined #openstack-lbaas | 21:17 | |
| bar_ | failover calls in v2 cause the loadbalancers to ERROR | 21:17 |
| johnsom | It would be a very recent change if it is broken | 21:18 |
| johnsom | If you post the o-cw and o-hm logs I will take a look | 21:18 |
| bar_ | johnsom, cool. thanks | 21:19 |
| rm_work | johnsom: that failover test doesn't really work with the upstream driver yet :/ | 21:22 |
| rm_work | i haven't figured out how to make it work | 21:23 |
| rm_work | because i rely on knowing the state | 21:23 |
| rm_work | FLIP driver updates master/backup state in the DB | 21:23 |
| rm_work | the main driver doesn't | 21:23 |
| rm_work | but, is bar_ talking about "failovers" in general, or the Failover API calls? | 21:23 |
| rm_work | kp | 21:23 |
| rm_work | [=\[= | 21:23 |
| rm_work | [= | 21:23 |
| rm_work | p\= | 21:23 |
| rm_work | damnit | 21:23 |
| rm_work | something just fell on my keyboard and rolled around >_< | 21:24 |
| johnsom | lol | 21:25 |
| bar_ | rm_work, originally, I was asking about the failover protocol in general. I tried to trigger failover using v2 api call, and since the loadbalancer broke while the call itself was successful, I looked for tests | 21:25 |
| rm_work | hmmm | 21:27 |
| rm_work | in my experience, it seems to be working... | 21:27 |
| rm_work | but i still need to test against the very latest | 21:27 |
| bar_ | ok... it fails in cw due to ImmutableObject. | 21:30 |
| johnsom | bar_ I really need to see the logs | 21:30 |
| bar_ | I know | 21:30 |
| bar_ | just a sec | 21:31 |
| johnsom | ok | 21:31 |
| bar_ | http://paste.openstack.org/show/627510/ | 21:31 |
| bar_ | sorry for keeping you waiting | 21:31 |
| bar_ | johnsom, rm_work ^ | 21:32 |
| rm_work | yep | 21:33 |
| rm_work | that makes sense | 21:33 |
| rm_work | that's what is supposed to happen | 21:33 |
| rm_work | when your LB is pending-something, it won't do a failover | 21:33 |
| rm_work | were you in the middle of updating the LB? | 21:34 |
| bar_ | no | 21:34 |
| johnsom | Right, some other thread has "ownership" of the load balancer at the moment | 21:34 |
| johnsom | Hmm, brings up an interesting point though, maybe the revert to error should be after the TestLBStatusSetPendingInDB task so it doesn't get set to ERROR if some other thread has ownership. | 21:35 |
| johnsom | Since it was in PENDING it likely would have been set to another state from ERROR when it completed anyway, but. | 21:36 |
| bar_ | sorry, do you believe it to be PENDING due to another action that is not specified in the log? | 21:38 |
| johnsom | It should be there somewhere if you search for that LB ID | 21:39 |
| bar_ | I created that LB, waited on the lines for it to go active, and then charged on it with the failover call | 21:39 |
| bar_ | and it's very consistent... happened with all my LBs | 21:40 |
| *** pcaruana has quit IRC | 21:41 | |
| *** gcheresh has quit IRC | 21:43 | |
| xgerman_ | cores with Q2 looming: https://review.openstack.org/#/c/418530/ | 21:45 |
| johnsom | xgerman_ It looks like your failover API might not be working correctly. ^^^ | 21:45 |
| johnsom | Yeah, ok, it looks like we do have a conflict with failover API | 21:49 |
| johnsom | This is a problem: https://github.com/openstack/octavia/blob/master/octavia/api/v2/controllers/load_balancer.py#L551 | 21:49 |
| xgerman_ | my API? | 21:50 |
| xgerman_ | didn’t you work on it, too | 21:50 |
| johnsom | Because this came in: https://github.com/openstack/octavia/commit/8baf0ca1d7a9a64c2ae80b5d8731b36b0d16a901 | 21:50 |
| johnsom | Yeah, the failover API | 21:50 |
| johnsom | Ha, maybe | 21:50 |
| bar_ | lol | 21:51 |
| xgerman_ | so we just back out rm_work’s patch ;-) | 21:51 |
| johnsom | No, we can just delete that _test_lb_status line | 21:51 |
| xgerman_ | BUT the idea was you can’t fialover a pending/error LB | 21:51 |
| johnsom | Yeah, he moved that into the actual flow, so this line is duplicating it from the flow | 21:52 |
| xgerman_ | his is wrong then | 21:53 |
| xgerman_ | because the admin hits fialover and we mark the LB pending — tyhen we pout it on the queue | 21:53 |
| xgerman_ | so if we do it in the flow we igth get an update in the middle and then not faioover? | 21:53 |
| johnsom | Yeah, I follow you | 21:55 |
| nmagnezi | johnsom, re: https://review.openstack.org/#/c/517455/ , will catch this tomorrow morning my timezone | 21:56 |
| xgerman_ | now, if we do a health related failover we probably should do it in any case, too | 21:56 |
| nmagnezi | johnsom, but I just read tht story and I don't get how to reproducer Jason Niesz results an incorrect haproxy config | 21:57 |
| johnsom | Oye, we are re-setting this on the cw side too | 21:57 |
| nmagnezi | (the cmd he uses: openstack loadbalancer listener create --name jason-test3-listener --protocol HTTP --protocol-port 80 jason-test3) | 21:57 |
| rm_work | johnsom / xgerman_: IIRC I was the one arguing that failover SHOULDN'T bother checking and should "just work", but I think I was convinced to author the patch somehow | 22:02 |
| johnsom | nmagnezi Yeah, he locally changed the jinja2 template for the haproxy config which broke it. It will be hard to exactly reproduce his situation | 22:03 |
| nmagnezi | johnsom, yeah i figured that. just double-checked that i don't miss anything here (it's late..) | 22:04 |
| johnsom | rm_work It's just where we do it that is the issue here. There are two different entry points, one via the API handler queue and one via the health manager | 22:04 |
| nmagnezi | johnsom, at a first glance the good looks okay. I will give it a second look tomorrow morning | 22:04 |
| nmagnezi | good night :) | 22:04 |
| rm_work | so it needs to be done in the health manager itself, not in the flow? | 22:04 |
| rm_work | before triggering the flow | 22:04 |
| johnsom | The API path sets it to PENDING_UPDATE twice before the flow starts | 22:04 |
| johnsom | Yep | 22:04 |
| rm_work | fine by me | 22:05 |
| johnsom | I will whip up a patch | 22:05 |
| rm_work | i don't even use the upstream failover flow :P | 22:05 |
| bar_ | shouldn't we LOCK the flow somehow? | 22:05 |
| bar_ | checking the db from two different places, could result in race condition, no? | 22:06 |
| johnsom | bar_ This is the locking code | 22:07 |
| bar_ | ok, i missed that | 22:07 |
| *** rtjure has quit IRC | 22:08 | |
| johnsom | We lock the objects using the DB and the provisioning_status. In this case the object was already locked and we had an addition check/lock attempt in the flow that got rejected because it was already locked. | 22:08 |
| bar_ | and once it got into the diabolical ERROR status, is there a way the get rid of it? | 22:09 |
| rm_work | johnsom: erm, looking at the way this keystone auth config is split up into two sections... does this make it possible to have one auth endpoint/account for the frontend API (verifying requests) and a totally different endpoint/account for communicating with backends (nova/neutron)? | 22:10 |
| johnsom | Yes, you can delete it | 22:10 |
| *** aojea has joined #openstack-lbaas | 22:10 | |
| johnsom | rm_work Correct, that is how it is setup | 22:10 |
| rm_work | excellent | 22:10 |
| bar_ | johnsom, it did not work for me too good last time. I was left with zombies. (I'm redeploying right now) | 22:11 |
| *** sshank has quit IRC | 22:11 | |
| johnsom | bar_ network stuff left? | 22:12 |
| bar_ | loadbalancers | 22:12 |
| johnsom | That should not happen unless you are using neutron-lbaas | 22:12 |
| bar_ | nope. pure openstack client | 22:12 |
| johnsom | delete should clean up ERROR objects. I do have a patch up for review that fixes some network cleanup code that was not right and could block things depending on how the ERROR failure occured | 22:14 |
| bar_ | it could be ramification of the weird state the worker got into. | 22:14 |
| johnsom | bar_ reviews here are welcome: https://review.openstack.org/#/c/517455/ grin | 22:14 |
| bar_ | :-) I think you meant Nir. | 22:15 |
| bar_ | you need +2 | 22:16 |
| johnsom | No, non-core reviews are super valuable too | 22:16 |
| bar_ | ok, I'll try to get to it. | 22:16 |
| johnsom | It saves the cores time if others find stuff before they get around to reviewing... | 22:17 |
| bar_ | between, here's a brief one: https://review.openstack.org/#/c/522689/ | 22:17 |
| mnaser | fyi https://review.openstack.org/#/c/522114/ is tested and working in our env here (didnt want to run a local patch but time constraints :() | 22:17 |
| johnsom | Yeah, it's on my list. There was a bunch of stuff to look at after the holiday | 22:18 |
| bar_ | johnsom, legit, thanks. | 22:19 |
| *** sshank has joined #openstack-lbaas | 22:22 | |
| *** rcernin has joined #openstack-lbaas | 22:34 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix the failover API to not fail with immutable LB https://review.openstack.org/523242 | 22:48 |
| *** sshank has quit IRC | 22:49 | |
| *** salmankhan has joined #openstack-lbaas | 22:51 | |
| *** Alex_Staf has joined #openstack-lbaas | 22:56 | |
| *** sticker has joined #openstack-lbaas | 23:00 | |
| openstackgerrit | Merged openstack/neutron-lbaas master: Imported Translations from Zanata https://review.openstack.org/521435 | 23:02 |
| *** ipsecguy has quit IRC | 23:04 | |
| openstackgerrit | Merged openstack/octavia-dashboard master: Imported Translations from Zanata https://review.openstack.org/522448 | 23:07 |
| Alex_Staf | Hi guys, I have some question regarding the HA architecture and some other stuff, it would be great if u could assist me with that http://paste.openstack.org/show/627523/ | 23:08 |
| johnsom | Ha, just a few.... | 23:08 |
| xgerman_ | Alex_Staf can you out that on an etherpad so we cna comment | 23:12 |
| xgerman_ | put that on an etherpad | 23:12 |
| *** salmankhan has quit IRC | 23:12 | |
| johnsom | I was just going to use the "reply" option | 23:12 |
| Alex_Staf | johnsom, haha yep . I will 1 sec | 23:13 |
| Alex_Staf | https://etherpad.openstack.org/p/AlexQUestions_) | 23:14 |
| johnsom | I am not sure that link came through correctly. | 23:17 |
| johnsom | I am working on this etherpad: https://etherpad.openstack.org/p/AlexQUestions_ | 23:17 |
| Alex_Staf | https://etherpad.openstack.org/p/AlexQUestions_) | 23:21 |
| Alex_Staf | u lack a ")" in the end | 23:21 |
| Alex_Staf | Ok I see your | 23:21 |
| openstackgerrit | Merged openstack/octavia master: Rename 'remove' to 'reset' in Quota API https://review.openstack.org/522402 | 23:21 |
| johnsom | Ugh, ok, I will move my answers. | 23:22 |
| Alex_Staf | johnsom, xgerman_ thanks guys. Hope I will have bunch of documentation for those questions soon | 23:22 |
| *** slaweq has quit IRC | 23:25 | |
| openstackgerrit | Merged openstack/octavia master: Make the allowed_address_pairs driver better https://review.openstack.org/517455 | 23:25 |
| *** slaweq has joined #openstack-lbaas | 23:27 | |
| *** aojea has quit IRC | 23:31 | |
| *** slaweq has quit IRC | 23:33 | |
| *** sshank has joined #openstack-lbaas | 23:35 | |
| openstackgerrit | Merged openstack/octavia master: Handle race condition deleting security group rule https://review.openstack.org/522114 | 23:37 |
| dayou | @johnsom,https://github.com/openstack/octavia-dashboard/blob/master/octavia_dashboard/api/rest/lbaasv2.py#L24 | 23:42 |
| dayou | It seems octavia-dashboard still use the neutron lbaas v2 api from openstack sdk | 23:43 |
| dayou | Should we update the code to use python-octaviaclient instead? | 23:43 |
| johnsom | dayou openstack SDK only supports Octavia, it does not support neutron-lbaas. | 23:44 |
| johnsom | At least officially. | 23:44 |
| johnsom | It could be hacked to since the APIs are similar | 23:44 |
| johnsom | Alex_Staf I think I have answered everything | 23:45 |
| dayou | I just see from the comment https://github.com/openstack/octavia-dashboard/blob/master/octavia_dashboard/api/rest/lbaasv2.py#L14 | 23:45 |
| dayou | So I am wondering whether we should update the code of octavia-dashboard to make use of python-octaviaclient | 23:46 |
| johnsom | dayou My mistake for not fixing that when I set it up for openstack sdk / Octavia | 23:46 |
| dayou | Since python-openstacksdk seems to be buggy | 23:46 |
| dayou | What's the correct way to fix it? | 23:47 |
| johnsom | I would argue no since python-octaviaclient is mostly setup for OpenStack Client more than a python SDK. We really should just fix openstacksdk | 23:47 |
| dayou | But the https://github.com/openstack/octavia-dashboard/blob/master/octavia_dashboard/api/rest/barbican.py#L17 | 23:49 |
| dayou | barbican api is using barbican client | 23:49 |
| johnsom | Yep | 23:49 |
| dayou | So that's wh I am wondering | 23:49 |
| dayou | Also this really bugs me | 23:49 |
| dayou | Since I am working on some changes on dashboard | 23:50 |
| johnsom | It was just a call I made to use the openstacksdk. In theory the python-octaviaclient could also use the SDK | 23:50 |
| *** longstaff has quit IRC | 23:51 | |
| johnsom | for example, the neutron commands in python-openstackclient use the openstacksdk | 23:51 |
| dayou | /usr/local/lib/python2.7/dist-packages/openstack/load_balancer/v2/health_monitor.py | 23:51 |
| dayou | I have to manually edit this file to get healthmonitor resource return the correct field | 23:52 |
| johnsom | https://github.com/openstack/python-openstackclient/blob/master/openstackclient/network/v2/floating_ip.py#L19 | 23:52 |
| dayou | Other wise for healthmoniotr resource it's not returing the parent resource indicators, which make the listing of it hard | 23:52 |
| dayou | What's the quickst way to get this fixed? | 23:53 |
| dayou | either openstack sdk use octavia correctly or other way | 23:54 |
| johnsom | We probably just need to put up a patch for openstacksdk | 23:54 |
| *** ipsecguy has joined #openstack-lbaas | 23:56 | |
| dayou | Could you start one with the correct way to fix it? I can follow up on it if you don't have enough time on this. | 23:56 |
| johnsom | Sure, can you give me an example or a dashboard patch that isn't working? I'm not sure I fully understand the problem. | 23:57 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: Support UDP load balance https://review.openstack.org/503606 | 23:59 |
| dayou | I can put my dashboard patch on, so you can try it. It still took a few days work for me to get it to a point to be abled to reviewd. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!