| *** luksky has quit IRC | 00:16 | |
| johnsom | mloza, the amphora are service instances, they are always owned by the Octavia service account. | 00:56 |
|---|---|---|
| johnsom | kosa777777 Some database work, deleting an instance in nova, and a failover call. But I have never done this, so there are likely more details to workout. | 00:58 |
| johnsom | Maybe cleaning up a neutron port too | 00:58 |
| *** henriqueof2 has quit IRC | 01:20 | |
| *** henriqueof2 has joined #openstack-lbaas | 01:21 | |
| *** henriqueof2 has quit IRC | 01:52 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 02:31 | |
| *** Dinesh_Bhor has quit IRC | 02:31 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 02:41 | |
| *** psachin has joined #openstack-lbaas | 02:52 | |
| *** yamamoto has joined #openstack-lbaas | 03:16 | |
| *** yamamoto has quit IRC | 04:18 | |
| *** ramishra has joined #openstack-lbaas | 04:26 | |
| *** yamamoto has joined #openstack-lbaas | 04:36 | |
| *** mkuf has quit IRC | 06:54 | |
| *** ccamposr has joined #openstack-lbaas | 07:00 | |
| *** mkuf has joined #openstack-lbaas | 07:06 | |
| *** gcheresh_ has joined #openstack-lbaas | 07:27 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-dashboard master: Fix auth url for Barbican client https://review.openstack.org/640686 | 07:38 |
| *** yamamoto has quit IRC | 07:40 | |
| *** yamamoto has joined #openstack-lbaas | 07:41 | |
| *** sapd1 has joined #openstack-lbaas | 07:57 | |
| *** luksky has joined #openstack-lbaas | 08:07 | |
| *** rpittau|sardegna is now known as rpittau | 08:13 | |
| *** pcaruana has joined #openstack-lbaas | 08:25 | |
| nmagnezi | kosa777777, any chance you have a floating ip attached to your vip? | 08:30 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test https://review.openstack.org/637073 | 09:12 |
| *** celebdor has joined #openstack-lbaas | 09:15 | |
| *** sapd1 has quit IRC | 09:41 | |
| *** yamamoto has quit IRC | 10:18 | |
| *** yamamoto has joined #openstack-lbaas | 10:36 | |
| *** yamamoto has quit IRC | 10:36 | |
| *** yamamoto has joined #openstack-lbaas | 10:37 | |
| *** yamamoto has quit IRC | 10:41 | |
| *** andrein has joined #openstack-lbaas | 10:59 | |
| *** luksky has quit IRC | 11:00 | |
| *** salmankhan has joined #openstack-lbaas | 11:04 | |
| andrein | Hello everyone. I'm trying to set up octavia on rocky, deployed via kolla-ansible. I'm getting the following error in octavia-worker when I create a load balancer: http://paste.openstack.org/show/747226/ | 11:06 |
| *** yamamoto has joined #openstack-lbaas | 11:14 | |
| andrein | Looks like my client cert was missing the key... need to follow the PKI docs again | 11:41 |
| *** luksky has joined #openstack-lbaas | 11:50 | |
| *** sapd1 has joined #openstack-lbaas | 12:45 | |
| *** andrein has quit IRC | 12:46 | |
| *** yamamoto has quit IRC | 12:48 | |
| *** andrein has joined #openstack-lbaas | 12:50 | |
| *** andrein has quit IRC | 12:52 | |
| *** andrein has joined #openstack-lbaas | 12:57 | |
| *** yamamoto has joined #openstack-lbaas | 13:16 | |
| *** yamamoto has quit IRC | 13:37 | |
| *** yamamoto has joined #openstack-lbaas | 13:39 | |
| *** pcaruana has quit IRC | 14:31 | |
| *** henriqueof2 has joined #openstack-lbaas | 14:56 | |
| mloza | what seems to be a problem if the lb is working and provisioning status is active but the operating status is offline? | 15:03 |
| *** sapd1 has quit IRC | 15:09 | |
| *** sapd1 has joined #openstack-lbaas | 15:10 | |
| *** fnaval has joined #openstack-lbaas | 15:19 | |
| *** pcaruana has joined #openstack-lbaas | 15:25 | |
| johnsom | mloza: operating status is the observed status, so your health check is failing or your health heartbeat packets from the amphora are not reaching the controller endpoint on 5555 | 15:27 |
| *** andrein has quit IRC | 15:32 | |
| *** luksky has quit IRC | 15:40 | |
| dulek | I'm getting "TemplateSyntaxError: expected token 'end of statement block', got '.'" in o-cw. What can be wrong? | 15:40 |
| dulek | I mean I understand it's template. :D | 15:40 |
| *** andrein has joined #openstack-lbaas | 15:42 | |
| johnsom | What action are you taking and with which driver? | 15:48 |
| *** gcheresh_ has quit IRC | 16:00 | |
| mloza | <johnsom> mloza, the amphora are service instances, they are always owned by the Octavia service account. -- Does this mean if I add octavia user in a project the service instance will get launch in their own space? | 16:01 |
| mloza | <johnsom> mloza: operating status is the observed status, so your health check is failing or your health heartbeat packets from the amphora are not reaching the controller endpoint on 5555 -- Thanks, will take a look. | 16:02 |
| johnsom | mloza: yes, you can setup a service project for Octavia. Most of us do. | 16:07 |
| johnsom | Just make sure you have good quotas and that things like the image and optional ssh keys are owned by that project. | 16:07 |
| *** henriqueof3 has joined #openstack-lbaas | 16:10 | |
| *** luksky has joined #openstack-lbaas | 16:11 | |
| *** henriqueof2 has quit IRC | 16:12 | |
| mloza | johnsom: I added octavia as admin in a tenant project but the service instance still gets launched in the admin project | 16:15 |
| *** henriqueof2 has joined #openstack-lbaas | 16:17 | |
| *** henriqueof3 has quit IRC | 16:20 | |
| *** henriqueof2 has quit IRC | 16:29 | |
| *** ramishra has quit IRC | 16:31 | |
| johnsom | mloza You need to set the account in the [service_auth] section of the octavia.conf | 16:33 |
| *** xgerman has joined #openstack-lbaas | 16:35 | |
| xgerman | o/ | 16:35 |
| mloza | johnsom: its the project_name that I have to set right? What if I want the service instance to be created under the project of the user who created it not just one project? | 16:38 |
| *** yamamoto has quit IRC | 16:41 | |
| *** rpittau is now known as rpittau|afk | 16:42 | |
| johnsom | mloza You cannot have the amphroa created under the user project. They are a managed service instance and should not be visible or accessible to the user. | 16:42 |
| dulek | johnsom: Whoops, sorry, missed your answer. That was Amphorae and seems to be during listener creation. | 16:47 |
| dulek | Might be listeners update as well. | 16:48 |
| dulek | "octavia-create-pool-flow" | 16:48 |
| johnsom | Yes, probably. So that likely means a jinja2 template got corrupted somehow. There is a template override in the octavia.conf : https://docs.openstack.org/octavia/latest/configuration/configref.html#haproxy_amphora.haproxy_template | 16:49 |
| johnsom | Check that it is not set and pointing to some local templates. Then I would check the standard templates in https://github.com/openstack/octavia/tree/master/octavia/common/jinja/haproxy/templates and see if those have been changed somehow. Then after that it's the VRRP or UDP/LVS templates potentially if you are using those features. | 16:50 |
| *** yamamoto has joined #openstack-lbaas | 16:52 | |
| dulek | johnsom: That's DevStack, so I strongly doubt any corruption. Data point - the setup also has networking-ovn provider enabled. | 16:56 |
| johnsom | Hmm, well, then it shouldn't be using any templates.... Hmmm, is there a stack trace? | 16:57 |
| *** yamamoto has quit IRC | 16:57 | |
| dulek | johnsom: No, no, sorry for confusing you, it's using both Octavia and networking-ovn and the error is while Octavia LB is getting created. | 17:03 |
| dulek | johnsom: I can try putting the logs somewhere, just a sec. | 17:04 |
| johnsom | Yeah, most interested in the worker log | 17:04 |
| dulek | johnsom: https://paste.fedoraproject.org/paste/SCYY7eUvPqMMawa0RVhoug | 17:05 |
| dulek | johnsom: I'm not overriding any template, in DevStack we just set the listeners timeouts. | 17:07 |
| *** psachin has quit IRC | 17:09 | |
| johnsom | dulek Oh, ok, I know what this is. You have an old version of jinja2 installed | 17:10 |
| dulek | johnsom: I was updating that system recently, it's possible I've some pip packages got overwritten… | 17:11 |
| dulek | johnsom: Although I would expect global requirements would handle that. | 17:12 |
| johnsom | dulek Yeah, it needs to be jinja 2.10 or newer | 17:12 |
| johnsom | https://github.com/openstack/octavia/blob/master/requirements.txt#L39 | 17:12 |
| cgoncalves | queens, right? we didn't get to fix in queens as it required bumping version | 17:13 |
| dulek | johnsom: Yep, looks like system packages overwritten this package. | 17:13 |
| dulek | cgoncalves: It's actually more-or-less master | 17:13 |
| dulek | But: | 17:14 |
| dulek | >>> jinja2.__version__ | 17:14 |
| dulek | '2.8.1' | 17:14 |
| cgoncalves | queens is more-or-less still master for me xD | 17:14 |
| cgoncalves | it will be for the next 5 years of support lol | 17:14 |
| johnsom | Sounds like you need help with that.... | 17:14 |
| johnsom | lol | 17:15 |
| dulek | Me? I'll handle it from here, no problem. ;) | 17:15 |
| cgoncalves | we got to fix that in RDO/OSP via .spec | 17:15 |
| dulek | Thanks! | 17:17 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-lib master: Sync data models and import new constants from Octavia https://review.openstack.org/640825 | 17:26 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Updates Octavia to support octavia-lib https://review.openstack.org/613709 | 17:27 |
| cgoncalves | johnsom, I hope you don't mind me messing with your changes | 17:27 |
| johnsom | cgoncalves Always nice to ask first. I was just about to do the octavia-lib patch | 17:27 |
| cgoncalves | ooops! | 17:27 |
| johnsom | But, since you beat me too it.... | 17:28 |
| openstackgerrit | Merged openstack/python-octaviaclient master: Add 'tls_container_ref' option into Pool for backend re-encrption https://review.openstack.org/619957 | 17:38 |
| openstackgerrit | Merged openstack/python-octaviaclient master: Add 2 new options to Pool for support backend certificates validation https://review.openstack.org/620211 | 17:41 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-lib master: Sync data models and import new constants from Octavia https://review.openstack.org/640825 | 17:42 |
| *** andrein has quit IRC | 17:47 | |
| openstackgerrit | Merged openstack/python-octaviaclient master: Add enable_tls option into Pool CLI https://review.openstack.org/624265 | 18:04 |
| johnsom | Welcome to feature freeze week folks! I have updated the priority review sheet. | 18:10 |
| johnsom | note: It is not feature freeze yet, but winter is coming.... | 18:11 |
| *** andrein has joined #openstack-lbaas | 18:14 | |
| *** ccamposr has quit IRC | 18:21 | |
| *** ccamposr has joined #openstack-lbaas | 18:21 | |
| *** ccamposr has quit IRC | 18:22 | |
| *** andrein has quit IRC | 18:29 | |
| *** HD|Laptop has joined #openstack-lbaas | 18:45 | |
| HD|Laptop | Hello all! | 18:45 |
| HD|Laptop | I am trying to use Octavia for magnum/k8s loadbalancing | 18:45 |
| HD|Laptop | after a long learning period I have managed to get everything connected and running - except that haproxy on the amphora (used the image from https://tarballs.openstack.org/octavia/test-images/ )seems to be messed ip | 18:46 |
| HD|Laptop | on the amphora I can connect to the backend just fine | 18:53 |
| HD|Laptop | but haproxy cannot | 18:53 |
| HD|Laptop | http://paste.debian.net/1071500/ <= here is all my config | 18:53 |
| *** pcaruana has quit IRC | 18:56 | |
| johnsom | HD|Laptop By default the netns does not have the loopback up, so you can't curl the VIP local. Just bring up the loopback interface and it will work. (loopback is not needed for normal operations) | 19:00 |
| HD|Laptop | johnsom: lo is up in the amphora netns, see line 35 | 19:02 |
| johnsom | 1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1 | 19:03 |
| johnsom | state DOWN | 19:03 |
| HD|Laptop | aah | 19:03 |
| *** andrein has joined #openstack-lbaas | 19:04 | |
| HD|Laptop | johnsom: ok, now at least curl'ing the frontend inside the amphora works | 19:05 |
| HD|Laptop | johnsom: but curl'ing either the frontend ip (192.168.100.12:8080) or the floating IP (10.161.14.56) still doesn't work | 19:09 |
| johnsom | From outside the amphora? | 19:10 |
| HD|Laptop | yep | 19:11 |
| HD|Laptop | from outside, on the host machine of the amphora | 19:11 |
| HD|Laptop | which has an interface on the 192.168.100.0/23 network | 19:11 |
| HD|Laptop | neutron security groups, port security etc. are all disabled | 19:12 |
| *** salmankhan has quit IRC | 19:12 | |
| HD|Laptop | interestingly, on the host: 10.161.14.56 => timeout, 192.168.100.12 => connection refused | 19:13 |
| mloza | how do i get rid of amphora instances in status BOOTING and ERROR that are in `openstack loadbalancer amphora list`. Probably, these were leftovers when i updated provisioning_status to ERROR in the database because the LB was in PENDING_UPDATE | 19:15 |
| HD|Laptop | mloza: I had ones stuck in PENDING_CREATE during deployment, ended up wiping them in the mysql db by hand | 19:20 |
| *** openstackgerrit has quit IRC | 19:23 | |
| eandersson | What happens if the health_manager update_db takes too long? | 19:32 |
| *** andrein has quit IRC | 19:34 | |
| johnsom | eandersson We drop that update packet. | 19:38 |
| eandersson | kk so it has no direct impact on the lb itself? | 19:38 |
| johnsom | And print some big error messages in the logs | 19:38 |
| eandersson | Yea - the error message might be too big :p | 19:38 |
| eandersson | "THIS IS NOT GOOD" | 19:39 |
| johnsom | No, assuming one of the HMs is processing ok in the next cycles. Basically it needs to get one in inside the overall time limit. It's six tries by default | 19:39 |
| eandersson | What is the highest delay we could hit before impact? | 19:43 |
| johnsom | Depends on your settings. The default heartbeat interval is 10 seconds, with a 60 second limit before a failover may start. So, the HM needs to process the heartbeat inside 10 seconds by default, and one out of six heartbeats must be successful. | 19:44 |
| *** blake has joined #openstack-lbaas | 19:46 | |
| xgerman | also keep in mind if your DB is “slow" we will pile up update requests in our queues which we resolve quickly if timestamp is old — but if you limit memory for the HM too tightly tat might trigger issues | 19:50 |
| HD|Laptop | johnsom: any idea what else could be wrong with my amphora? | 19:50 |
| xgerman | Haven’t seen that after we optimized our queries... | 19:50 |
| johnsom | HD|Laptop Well, if you can query the VIP from inside the amphora, I suspect it is not the amphora that is the issue. Here are some things to look at: | 19:52 |
| xgerman | you made sure that the ports i neutron are ok? openstack port list ? | 19:52 |
| johnsom | 1. Check the subnet you created the VIP on, does it have any host routes that may be conflicting? We honor the neutron host routes inside the amphora. | 19:52 |
| johnsom | 2. Check the ports, as xgerman mentioned. One will be down, one up. This is normal as the "down" port is an "allowed address pairs" neutron port and a "fake" port. | 19:53 |
| johnsom | 3. Check the security groups applied to those ports. They should be octavia managed SGs with just a few ports open, including your listener port. | 19:54 |
| johnsom | 4. Check the routes on your "client" instance, make sure there isn't a bad route on your test system that is blocking access. | 19:54 |
| johnsom | 5. You can install tcpdump inside the amphora and see if your request is making it to the amp. If it is, but you don't see a reply, you have a routing or L2 problem. | 19:55 |
| HD|Laptop | johnsom: as for 0. http://paste.debian.net/1071515/ | 19:56 |
| HD|Laptop | for 1. I cannot remember that I set up anything routing related except a neutron SNAT router from the 192.168.100.0/23 to the 10.161.14.0/24 network | 19:57 |
| cgoncalves | how's best (if possible at all) to set lower requirement constraints for an unreleased lib version? | 19:58 |
| johnsom | HD|Laptop That port list looks odd, there should be an octavia vrrp port listed | 19:58 |
| cgoncalves | https://review.openstack.org/#/c/613709/ requires https://review.openstack.org/#/c/640825/ | 19:58 |
| HD|Laptop | | dd37cd45-84c5-4c04-bdab-86e6eb260066 | | fa:16:3e:57:5f:3a | ip_address='192.168.100.4', subnet_id='483f4d33-2fff-4315-a30d-57114e0e5262' | ACTIVE | | 19:58 |
| cgoncalves | >=1.1.0 wouldn't pass lower-constraints job | 19:59 |
| HD|Laptop | this is the port entry for the vrrp_ip showing in "openstack loadbalancer amphora show": | vrrp_ip | 192.168.100.4 | | 19:59 |
| *** andrein has joined #openstack-lbaas | 19:59 | |
| johnsom | cgoncalves I don't think you can, but you can ask in the #openstack-requirements channel | 19:59 |
| HD|Laptop | johnsom: http://paste.debian.net/1071519/ here's amphora show in full | 20:00 |
| kosa777777 | nmagnezi no floating IPs, and yes mysql change was necessary, it worked but it would be great to have some API way of changing topology for existing loadbalancers, in case you change octavia config. | 20:00 |
| johnsom | HD|Laptop that is odd, it should have a name. Did you pass in an existing neutron port at LB create? | 20:00 |
| HD|Laptop | johnsom: the lb was created by magnum/heat from the kubernetes-cloud-controller. | 20:00 |
| cgoncalves | johnsom, right. I image the answer is merge octavia-lib patch and tag octavia-lib first | 20:01 |
| HD|Laptop | as for port security groups, I didn't enable them in neutron and iptables (host + amphora)/ebtables (host) doesn't show anything weird either | 20:01 |
| *** abaindur has joined #openstack-lbaas | 20:01 | |
| HD|Laptop | i'll go the tcpdump route, though. | 20:02 |
| HD|Laptop | AAAAH | 20:03 |
| HD|Laptop | I know whats happening! | 20:03 |
| johnsom | cgoncalves The depends on should at least help keep some order. | 20:03 |
| HD|Laptop | I forgot to set the subnet DHCP allocation range... | 20:03 |
| HD|Laptop | which means that .12 is colliding with an existing device on the network. | 20:03 |
| johnsom | There might also be an issue with the allowed-address-pairs function if port security is disabled. | 20:04 |
| HD|Laptop | johnsom: I was wondering why I didn't see a single packet inside the amphora, and then it hit me, what the hell was I seeing in the ARP cache of the compute host for that IP | 20:13 |
| HD|Laptop | unfortunately I had to tear down the entire k8s cluster as it used colliding IPs also... so it will be ~10min to tell if this is working now or not. | 20:13 |
| HD|Laptop | johnsom: yep, it works now! only thing missing now would be that octavia also creates a DNS record in designate for the loadbalancer floating IP | 20:32 |
| rm_work | that would probably be up to heat/k8s, because octavia itself isn't floating-ip aware | 20:34 |
| johnsom | HD|Laptop Since Octavia doesn't manage floating IPs, you will need to set that up with neutron. | 20:34 |
| johnsom | lol, yeah, what he said. | 20:34 |
| HD|Laptop | weird. when I create a server with "openstack server create xyz", it creates a dns entry xyz.my-domain.tld | 20:34 |
| HD|Laptop | but for the amphora, it says in the neutron-api "neutron_lib.exceptions.dns.DNSDomainNotFound: Domain xyz.my-domain.tld. not found in the external DNS service" | 20:35 |
| HD|Laptop | hmm. when I run "openstack port set <amphora_floatingip_port> --dns-name=mylb", it works. Can I somehow coerce octavia to set the name & dns-name? | 20:44 |
| johnsom | No, we had to explicitly disable the designate integration on our ports due to bugs. Neutron would fail to migrate ports during a failover with the dns integration enabled. | 20:45 |
| johnsom | On failover you will either get an error if the bugs are still present, or it will remove the DNS information. | 20:45 |
| johnsom | Basically, designate integration into Octavia is a ToDo item and not implemented yet. | 20:46 |
| HD|Laptop | ah okay, thanks for the info :) | 20:47 |
| HD|Laptop | hmm. I just created a second exposed deployment via kubectl ("kubectl expose deployment hello-node2 --type=LoadBalancer --port=8082") and it spins up a new lb/amphora | 20:47 |
| HD|Laptop | can I tell octavia to simply add a listener on the existing amphora/port 8082 instead? | 20:47 |
| johnsom | Yes, absolutely | 20:48 |
| johnsom | As long as they have the same VIP, you can add up to around 65,000 ports to it | 20:48 |
| HD|Laptop | so basically I have to ask this question again with the magnum team? ;) but good to know it is possible in theory | 20:50 |
| HD|Laptop | thanks for your time, I now have a decent working OpenStack with everything needed to run k8s! | 20:50 |
| johnsom | Excellent, good luck! | 20:50 |
| xgerman | +1 | 20:51 |
| *** openstackgerrit has joined #openstack-lbaas | 21:09 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add octavia-lib to the base job https://review.openstack.org/640868 | 21:09 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Updates Octavia to support octavia-lib https://review.openstack.org/613709 | 21:10 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-lib master: Sync data models and import new constants from Octavia https://review.openstack.org/640825 | 21:16 |
| *** abaindur has quit IRC | 21:25 | |
| *** blake has quit IRC | 21:32 | |
| *** yamamoto has joined #openstack-lbaas | 21:33 | |
| *** yamamoto has quit IRC | 21:38 | |
| *** salmankhan has joined #openstack-lbaas | 21:50 | |
| *** abaindur has joined #openstack-lbaas | 22:00 | |
| *** luksky has quit IRC | 23:02 | |
| *** abaindur has quit IRC | 23:04 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Support L7policy redirect http code https://review.openstack.org/625007 | 23:04 |
| *** abaindur has joined #openstack-lbaas | 23:06 | |
| *** openstackgerrit has quit IRC | 23:28 | |
| *** andrein has quit IRC | 23:35 | |
| *** salmankhan has quit IRC | 23:37 | |
| *** andrein has joined #openstack-lbaas | 23:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!