| savvas_ | What would be the best way to monitor the process when load balancers are in PENDING_CREATE state? | 00:15 |
|---|---|---|
| johnsom | I usually tail the controller worker log | 00:15 |
| savvas_ | thx | 00:21 |
| savvas_ | that helps | 00:21 |
| savvas_ | been tailing on ERROR entries, this is warning, I would argue :p | 00:21 |
| savvas_ | Failed to establish a new connection: [Errno 113] No route to host | 00:21 |
| johnsom | Yeah, that is normal, it will keep retrying while the VM boots | 00:22 |
| savvas_ | ok so that's not even an indicator of the failure | 00:22 |
| johnsom | If it does timeout, it will go to ERROR. However, the default setting is like 25 minutes and not a good value for a production setup. It's set high for the slow gate tests | 00:23 |
| savvas_ | ye I've noticed that it times out after a while | 00:24 |
| savvas_ | think I've figured that one out too. My containers were previously created with different adapter settings. The amphora VMs respond to ICMP just fine directly from my management node, but the Octavia containers can not reach them | 00:27 |
| savvas_ | so I need to redo those as well, those settings weren't overwritten when I reran the octavia playbook, probably because it is part of the container setup | 00:28 |
| johnsom | Hmm, the security group should block icmp | 00:28 |
| savvas_ | hmm well I can ping every one of them | 00:29 |
| rm_work | ok, failed-over 300 amps and so far, so good | 00:33 |
| rm_work | (not all at once :P) | 00:33 |
| rm_work | i'm liking these percentages | 00:34 |
| rm_work | cool, done, no issues | 01:03 |
| rm_work | tomorrow I convert 7 "STANDALONE" topo LBs to "ACTIVE_STANDBY" :P | 01:03 |
| rm_work | zero downtime conversion, lol | 01:03 |
| rm_work | I wonder if it's worth documenting the process | 01:03 |
| *** hongbin has joined #openstack-lbaas | 01:23 | |
| *** threestrands has quit IRC | 01:33 | |
| *** JudeCross has quit IRC | 02:23 | |
| *** yamamoto has quit IRC | 02:32 | |
| *** yamamoto has joined #openstack-lbaas | 02:33 | |
| *** yamamoto has quit IRC | 02:37 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: DNM: Testing bionic nodes https://review.openstack.org/600539 | 02:43 |
| *** yamamoto has joined #openstack-lbaas | 03:05 | |
| *** yamamoto has quit IRC | 03:12 | |
| *** hongbin has quit IRC | 03:15 | |
| *** JudeCross has joined #openstack-lbaas | 03:44 | |
| *** yamamoto has joined #openstack-lbaas | 03:47 | |
| *** JudeCross has quit IRC | 03:49 | |
| *** pcaruana has joined #openstack-lbaas | 04:36 | |
| *** pcaruana has quit IRC | 04:43 | |
| *** yboaron_ has joined #openstack-lbaas | 04:47 | |
| *** JudeCross has joined #openstack-lbaas | 04:50 | |
| *** ccamposr has joined #openstack-lbaas | 05:16 | |
| *** ccamposr__ has joined #openstack-lbaas | 05:22 | |
| *** ccamposr has quit IRC | 05:25 | |
| *** evgenyf has quit IRC | 05:47 | |
| *** JudeCross has quit IRC | 06:04 | |
| *** sapd1 has joined #openstack-lbaas | 06:08 | |
| sapd1 | how can I export docs of octavia to pdf format? | 06:09 |
| *** yamamoto has quit IRC | 06:27 | |
| *** yamamoto has joined #openstack-lbaas | 06:27 | |
| *** Emine has quit IRC | 06:36 | |
| *** dims has quit IRC | 06:38 | |
| *** dims has joined #openstack-lbaas | 06:44 | |
| *** dims has quit IRC | 06:48 | |
| *** dims has joined #openstack-lbaas | 06:51 | |
| *** rcernin has quit IRC | 07:01 | |
| *** pcaruana has joined #openstack-lbaas | 07:01 | |
| *** oanson has joined #openstack-lbaas | 07:01 | |
| *** JudeCross has joined #openstack-lbaas | 07:01 | |
| *** velizarx has joined #openstack-lbaas | 07:55 | |
| *** celebdor has joined #openstack-lbaas | 08:05 | |
| *** celebdor has quit IRC | 08:06 | |
| *** celebdor has joined #openstack-lbaas | 08:06 | |
| *** yboaron_ has quit IRC | 08:14 | |
| *** yboaron_ has joined #openstack-lbaas | 08:14 | |
| *** abaindur has quit IRC | 08:26 | |
| *** yboaron_ has quit IRC | 08:42 | |
| *** yboaron_ has joined #openstack-lbaas | 08:46 | |
| *** JudeCross has quit IRC | 09:01 | |
| *** yboaron_ has quit IRC | 09:02 | |
| *** yboaron_ has joined #openstack-lbaas | 09:03 | |
| *** yamamoto has quit IRC | 09:19 | |
| *** yamamoto has joined #openstack-lbaas | 09:20 | |
| *** salmankhan has joined #openstack-lbaas | 09:21 | |
| *** yamamoto has quit IRC | 09:24 | |
| *** JudeCross has joined #openstack-lbaas | 09:29 | |
| *** JudeCross has quit IRC | 09:33 | |
| *** yboaron_ has quit IRC | 09:39 | |
| *** yboaron_ has joined #openstack-lbaas | 09:40 | |
| *** yamamoto has joined #openstack-lbaas | 09:42 | |
| *** JudeCross has joined #openstack-lbaas | 10:38 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** yamamoto has joined #openstack-lbaas | 10:40 | |
| *** JudeCross has quit IRC | 10:43 | |
| *** sapd1 has quit IRC | 10:49 | |
| *** rpittau has joined #openstack-lbaas | 10:50 | |
| *** JudeCross has joined #openstack-lbaas | 10:54 | |
| *** JudeCross has quit IRC | 10:59 | |
| openstackgerrit | Merged openstack/octavia master: Support REDIRECT_PREFIX action for L7Policy https://review.openstack.org/601086 | 11:27 |
| *** phuoc_ has quit IRC | 11:39 | |
| *** phuoc_ has joined #openstack-lbaas | 11:40 | |
| *** aojea_ has joined #openstack-lbaas | 11:55 | |
| *** aojea_ has quit IRC | 12:05 | |
| *** aojea_ has joined #openstack-lbaas | 12:06 | |
| *** yamamoto has quit IRC | 12:07 | |
| *** yamamoto has joined #openstack-lbaas | 12:08 | |
| *** aojea_ has quit IRC | 12:10 | |
| *** yamamoto has quit IRC | 12:12 | |
| *** savvas has joined #openstack-lbaas | 12:14 | |
| *** amuller has joined #openstack-lbaas | 12:19 | |
| *** savvas is now known as sabomia | 12:19 | |
| *** Emine has joined #openstack-lbaas | 12:23 | |
| ltomasbo | cgoncalves, xgerman_, johnsom, nmagnezi: it will be great if you could take a look at this one: https://review.openstack.org/#/c/604417 | 12:24 |
| *** aojea_ has joined #openstack-lbaas | 12:29 | |
| *** JudeCross has joined #openstack-lbaas | 12:32 | |
| *** JudeCross has quit IRC | 12:37 | |
| *** evgenyf has joined #openstack-lbaas | 12:37 | |
| *** sabomia has quit IRC | 12:46 | |
| *** yamamoto has joined #openstack-lbaas | 12:51 | |
| *** sabomia has joined #openstack-lbaas | 12:55 | |
| *** pcaruana has quit IRC | 12:57 | |
| *** aojea_ has quit IRC | 13:09 | |
| *** velizarx has quit IRC | 13:19 | |
| *** pcaruana has joined #openstack-lbaas | 13:27 | |
| *** sabomia has quit IRC | 13:28 | |
| *** CBR09 has joined #openstack-lbaas | 13:37 | |
| CBR09 | hello all, | 13:37 |
| CBR09 | I got issue when using octavia with active/standby tolology | 13:37 |
| CBR09 | the server which running octavia has rebooted | 13:38 |
| CBR09 | after that | 13:38 |
| CBR09 | my lb has fail to failover | 13:39 |
| *** yboaron_ has quit IRC | 13:39 | |
| CBR09 | and one amphora has gone, one amphora's role is standalone | 13:40 |
| CBR09 | it mean my lb topology has inconsistent! | 13:40 |
| CBR09 | How can I recover that? | 13:40 |
| CBR09 | I've tried with openstack loadbalancer failover but no luck | 13:41 |
| CBR09 | it still failover just one amphore and role is standalone | 13:41 |
| *** yboaron has joined #openstack-lbaas | 13:42 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 13:46 | |
| maciejjozefczyk | hey! | 13:46 |
| maciejjozefczyk | I have a little question, Octavia should be working and spawning amphoreas in separated, dedicated tenant, other than 'service' tenant? What is the good practise of deploying octavia? | 13:47 |
| cgoncalves | maciejjozefczyk, if you deploy with devstack, octavia will run under the 'admin' tenant by default. if you're deploying with a TripleO-based installer, the default is 'service' tenant. Amphorae will be on that tenant | 13:50 |
| cgoncalves | there is a dev/ops quick start guide: https://docs.openstack.org/octavia/latest/contributor/guides/dev-quick-start.html | 13:51 |
| maciejjozefczyk | cgoncalves: ok, thanks | 13:52 |
| maciejjozefczyk | So from operator point of view its better to deploy using some separated tenant, lets call it 'service-octavia'? | 13:53 |
| cgoncalves | maciejjozefczyk, it could be, yes. octavia creates resources (VMs, networks, ports, etc) so special quota limits may be required | 13:54 |
| maciejjozefczyk | cgoncalves: thank you | 13:55 |
| CBR09 | @johnsom: are you around here :( | 13:58 |
| *** amotoki_ is now known as amotoki | 14:04 | |
| *** sabomia has joined #openstack-lbaas | 14:10 | |
| *** fnaval has joined #openstack-lbaas | 14:12 | |
| *** celebdor has quit IRC | 14:14 | |
| *** JudeCross has joined #openstack-lbaas | 14:29 | |
| *** celebdor has joined #openstack-lbaas | 14:32 | |
| *** JudeCross has quit IRC | 14:34 | |
| *** celebdor has quit IRC | 14:39 | |
| *** celebdor has joined #openstack-lbaas | 14:39 | |
| xgerman_ | prob. another 30 minutes | 14:40 |
| *** velizarx has joined #openstack-lbaas | 14:47 | |
| jlaffaye_ | hello, I cant find reference to ca_private_key_passphrase in https://docs.openstack.org/octavia/rocky/configuration/configref.html and other versions as well | 14:48 |
| *** Emine has quit IRC | 14:55 | |
| *** celebdor has quit IRC | 15:04 | |
| *** yboaron has quit IRC | 15:06 | |
| *** yamamoto has quit IRC | 15:10 | |
| *** yamamoto has joined #openstack-lbaas | 15:10 | |
| jlaffaye_ | by the way I find difficult to reason about CA / client, like which one is used by the controller and the agent | 15:15 |
| *** CBR09 has quit IRC | 15:21 | |
| *** velizarx has quit IRC | 15:23 | |
| sabomia | Hey guys, I am having some difficulties with Octavia in my setup, was hoping someone notices what it is. I run the Octavia processes in a container (OpenStack Ansible) and basically my container can't bridge with the lbaas bridge. My amphora VMs on the other hand have no issues. This is my config: http://paste.openstack.org/show/731275/ | 15:54 |
| *** ccamposr__ has quit IRC | 15:56 | |
| johnsom | xgerman_ It looks like sabomia has a OSA setup similar to what you have setup, can you take a look? | 16:10 |
| *** JudeCross has joined #openstack-lbaas | 16:14 | |
| openstackgerrit | Hongbin Lu proposed openstack/neutron-lbaas master: [DNM] Test removal of lbaas policy rules https://review.openstack.org/607300 | 16:14 |
| xgerman_ | sabomia: | 16:18 |
| sabomia | hi xgerman_ :) | 16:19 |
| xgerman_ | so you should not have the bond0.704 up on your box if you put it into the br-lbaas | 16:19 |
| xgerman_ | this will blackhole | 16:19 |
| xgerman_ | so I would delete line 7-9 | 16:20 |
| xgerman_ | here is a picture what we run: https://github.com/rcbops/rpc-octavia/blob/master/rpc-octavia-network-diagram.png | 16:22 |
| *** JudeCross has quit IRC | 16:22 | |
| *** celebdor has joined #openstack-lbaas | 16:23 | |
| sabomia | ow ok | 16:31 |
| sabomia | would that explain why the interface gets taken out of the bridge? It seems this happens whenever an instance or container gets booted with access to physical network bond0.704 | 16:31 |
| sabomia | because I basically need bond0.704 as well as the container vifs to be part of the br-lbaas bridge for this to work | 16:32 |
| xgerman_ | yeah, you can have bond0.704 as an interface directly in your bridge | 16:39 |
| xgerman_ | it should not take it out but maybe it’s smart enough to know it’s no good… in my case it would just add both interfaces and blackhaul traffic | 16:40 |
| cgoncalves | I'm wondering how come test_update_health_zombie passes locally but not on Zuul http://logs.openstack.org/05/587505/22/check/openstack-tox-py27/18ad0e2/testr_results.html.gz | 16:49 |
| *** JudeCross has joined #openstack-lbaas | 16:54 | |
| *** JudeCross has quit IRC | 16:59 | |
| *** sabomia has quit IRC | 17:11 | |
| *** celebdor has quit IRC | 17:14 | |
| *** salmankhan has quit IRC | 17:16 | |
| *** JudeCross has joined #openstack-lbaas | 17:16 | |
| colin- | are there circumstances where octavia will create a subnet in the control plane if one does not exist? | 17:23 |
| johnsom | No, not with our stuff. Maybe the deployment tools do that OSA/Triple-o, etc. | 17:24 |
| colin- | thought not, thanks | 17:24 |
| *** evgenyf has quit IRC | 17:28 | |
| *** Emine has joined #openstack-lbaas | 17:43 | |
| *** openstackgerrit has quit IRC | 17:51 | |
| *** pcaruana has quit IRC | 18:02 | |
| *** aojea has joined #openstack-lbaas | 18:12 | |
| *** Emine has quit IRC | 18:15 | |
| *** Swami has joined #openstack-lbaas | 19:03 | |
| Swami | hi, We are having an Amphora that was created in Newton and we are trying to upgrade to Pike. In the process the Amphora VM was alive when we upgraded. After upgrade we tried deleting the VM with health-manager turned on, but the Amphora VM failed to come up and cleaned the Amphora entry from the DB and Loadbalancer went into ERROR state. We are seeing an "InvalidRequest" in the health manager log. Here is the log info. http://paste.openstack.org/show | 19:06 |
| Swami | /731290/ | 19:06 |
| Swami | Here is the link# http://paste.openstack.org/show/731290/ | 19:07 |
| Swami | What might be causing the 'InvalidRequest', will there be anything in the config change that might cause this issue. | 19:10 |
| Swami | It seems that it is failing when trying to update the listener. I don't have any listener configured at this point and it is just the loadbalancer with Amphora VM running when I tried this. Should I have a valid listener for health manager to recreate the Amphora VM. | 19:27 |
| johnsom | Swami Did you update the amphora image in glance or are you trying to use the old image? | 19:28 |
| Swami | I am trying to use the old image. | 19:29 |
| johnsom | That is likely the issue. The Pike release notes mention a new image is required. | 19:29 |
| Swami | Is this due to incompatibility with the agent code. | 19:30 |
| johnsom | Well, running amps would have continued to run, but on failover it expects the new image. At least that is my suspicion | 19:31 |
| Swami | But with the same image in glance I am able to create a new Lbaas and it works fine. | 19:31 |
| Swami | after the upgrade. | 19:31 |
| johnsom | Right. I'm just saying I suspect it's a failover issue. If you have the VM, you can log in and look at the amphora-agent log and see why it doesn't like the controller request. | 19:32 |
| Swami | johnsom: I don't have the VM right now, since I deleted it to test the failover. | 19:33 |
| johnsom | Right, the error was in the new one right? This is a standalone right? | 19:34 |
| Swami | The only thing that changed was the 'amphora_image_tag' We introduced the 'amp_image_tag' after the upgrade. Will that cause an issue. | 19:34 |
| Swami | johnsom: Yes this is a Standalone configuration. the error was after we upgraded, I tried to delete the old Amphora VM to check if it comes back again. | 19:35 |
| johnsom | No, the error you saw was after it had booted a replacement and was attempting to configure it. | 19:36 |
| Swami | But still the glance image had the same old 'Image id' along with the tag. | 19:36 |
| johnsom | Right, the issue is it's is failing over, onto a newton amp image, with a Pike controller. There must be some mis-match there that we didn't catch | 19:37 |
| johnsom | Well, we did catch one, as the release notes call out a new image should be uploaded to glance | 19:37 |
| Swami | These are five config items that I introduced in the config file. [health_manager] heartbeat_interval = 10 | 19:39 |
| Swami | heartbeat_timeout = 60 | 19:39 |
| Swami | health_check_interval = 3 and [controller_worker] amp_image_tag = 'amphora' and loadbalancer_topology = SINGLE and in the housekeeping [networking] port_detach_timeout = 900 | 19:39 |
| Swami | johnsom Ok I will try to upload a new image and will try to failover again and see if it works. | 19:40 |
| Swami | Will update you after I check it out. | 19:41 |
| Swami | Thanks for the information. | 19:41 |
| *** savvas_ is now known as sabomia_ | 20:02 | |
| *** sabomia_ is now known as sabomia` | 20:02 | |
| *** openstackgerrit has joined #openstack-lbaas | 20:31 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: DNM: Testing bionic nodes https://review.openstack.org/600539 | 20:31 |
| *** amuller has quit IRC | 20:44 | |
| *** abaindur has joined #openstack-lbaas | 20:47 | |
| *** abaindur has quit IRC | 20:47 | |
| *** abaindur has joined #openstack-lbaas | 20:48 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Add v2 two-node scenario test https://review.openstack.org/605163 | 21:06 |
| rm_work | jlaffaye_: yeah, sorry, it is a bit confusing -- i think we usually tell people to look at how our devstack plugin does the config for the certs -- but we ARE working on rewriting the install guide | 21:11 |
| *** Emine has joined #openstack-lbaas | 21:16 | |
| *** celebdor has joined #openstack-lbaas | 21:35 | |
| *** Emine has quit IRC | 21:53 | |
| rm_work | johnsom: putting up a patch for another flaky amphora scenario test (when run in a quick / busy environment) | 22:08 |
| johnsom | Ok | 22:11 |
| johnsom | I am working through the IPv6 issue with keepalived. Got it to stop segfaulting. Just not sure how yet. lol | 22:12 |
| *** fnaval has quit IRC | 22:12 | |
| rm_work | <_< | 22:26 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: Allow amp-list test to run in a busy environment https://review.openstack.org/607382 | 22:37 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Failover test https://review.openstack.org/501559 | 22:47 |
| *** aojea has quit IRC | 23:11 | |
| *** yamamoto has quit IRC | 23:21 | |
| *** yamamoto has joined #openstack-lbaas | 23:22 | |
| *** yamamoto has quit IRC | 23:24 | |
| *** yamamoto has joined #openstack-lbaas | 23:24 | |
| *** yamamoto has quit IRC | 23:29 | |
| *** yamamoto has joined #openstack-lbaas | 23:30 | |
| *** yamamoto has quit IRC | 23:34 | |
| *** Swami has quit IRC | 23:42 | |
| *** rcernin has joined #openstack-lbaas | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
