| *** fnaval has quit IRC | 00:01 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/queens: DNM: Testing stable/queens https://review.openstack.org/617398 | 00:07 |
|---|---|---|
| johnsom | The Octavia canary is squawking that stable/queens devstack is broken.... | 00:09 |
| johnsom | sigh | 00:09 |
| *** celebdor has joined #openstack-lbaas | 00:25 | |
| abaindur | johnsom: Getting a strange SSL error, hoping you could help understand what is wrong with the certs | 00:36 |
| abaindur | [('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')] | 00:36 |
| abaindur | Now i googled the error, and I verified I am not seeing any weird control characters in the PEM file, and it begins and ends with the proper -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- | 00:37 |
| abaindur | and I am also able to open it properly via "openssl x509" command, and view its contents | 00:37 |
| johnsom | Hmm, yeah, usually that means you are trying to use a DER format cert, or the file is corrupt in some way. | 00:38 |
| abaindur | i'm not the most familiar with SSL internals here, but i can paste the contents of the certs, 1 sec... | 00:39 |
| johnsom | However, that error implies it's the key file that is wrong, which would have a different header than BEGIN CERTIFICATE | 00:39 |
| abaindur | The error happens from here, https://github.com/openstack/octavia/blob/stable/queens/octavia/amphorae/drivers/haproxy/rest_api_driver.py#L330 | 00:39 |
| johnsom | Ah, ok | 00:40 |
| abaindur | after which it gets into urllib3 and openSSL code | 00:40 |
| johnsom | So that file should have a key concatenated with the certificate. | 00:40 |
| johnsom | http://logs.openstack.org/54/613454/3/check/openstack-tox-docs/ae6ce1e/html/admin/guides/certificates.html | 00:41 |
| johnsom | Step 14 in that doc | 00:41 |
| abaindur | Complete stacktrace: http://paste.openstack.org/show/734734/ | 00:42 |
| johnsom | The python requests library required that odd format at the time. | 00:42 |
| abaindur | here is our CA cert file: http://paste.openstack.org/show/734735/ | 00:42 |
| abaindur | using the same .pem cert there for ca_certificate and server_ca configs | 00:42 |
| abaindur | But it works when I used the certs generated by your sample script in repo | 00:43 |
| abaindur | these certs were generated elsewhere, by our own cert manager | 00:43 |
| abaindur | however in that case, I view the ca_01.pem file generated by your script, I do not see a private key in there. just the cert | 00:44 |
| abaindur | Same for the client_ca (we are using server_ca as the same as client_ca) | 00:44 |
| abaindur | johnsom: that's for the client_cert only? | 00:46 |
| johnsom | Correct, the [haproxy_amphora] client_cert file needs the key and cert concatenated | 00:47 |
| johnsom | The others don't | 00:47 |
| abaindur | I was thinking the error was coming from the server_ca, since thats passed ito the request error'ing out just above https://github.com/openstack/octavia/blob/stable/queens/octavia/amphorae/drivers/haproxy/rest_api_driver.py#L313 | 00:47 |
| *** sapd1_ has quit IRC | 01:48 | |
| *** sapd1 has joined #openstack-lbaas | 01:48 | |
| *** abaindur has quit IRC | 01:49 | |
| *** abaindur has joined #openstack-lbaas | 01:53 | |
| *** abaindur has quit IRC | 01:58 | |
| *** yamamoto has joined #openstack-lbaas | 03:30 | |
| *** abaindur has joined #openstack-lbaas | 03:54 | |
| *** ramishra has joined #openstack-lbaas | 04:02 | |
| *** yamamoto has quit IRC | 04:34 | |
| *** threestrands has joined #openstack-lbaas | 05:02 | |
| *** sapd1 has quit IRC | 05:12 | |
| *** sapd1_ has joined #openstack-lbaas | 05:12 | |
| *** dayou has quit IRC | 05:15 | |
| *** dayou has joined #openstack-lbaas | 05:41 | |
| *** yboaron_ has joined #openstack-lbaas | 06:37 | |
| *** ccamposr has joined #openstack-lbaas | 06:43 | |
| *** ccamposr__ has joined #openstack-lbaas | 06:47 | |
| *** ccamposr has quit IRC | 06:50 | |
| *** yboaron_ has quit IRC | 07:00 | |
| *** yboaron has joined #openstack-lbaas | 07:07 | |
| *** abaindur has quit IRC | 07:20 | |
| *** yamamoto has joined #openstack-lbaas | 07:42 | |
| *** threestrands has quit IRC | 08:04 | |
| *** yamamoto has quit IRC | 08:22 | |
| *** yamamoto has joined #openstack-lbaas | 08:24 | |
| openstackgerrit | Merged openstack/neutron-lbaas-dashboard stable/ocata: sni_container_refs needed if we want to use sni https://review.openstack.org/612225 | 08:24 |
| openstackgerrit | Merged openstack/neutron-lbaas-dashboard stable/pike: sni_container_refs needed if we want to use sni https://review.openstack.org/612224 | 08:25 |
| openstackgerrit | Merged openstack/neutron-lbaas-dashboard stable/rocky: sni_container_refs needed if we want to use sni https://review.openstack.org/612222 | 08:29 |
| *** yamamoto has quit IRC | 08:49 | |
| *** yamamoto has joined #openstack-lbaas | 08:50 | |
| *** yamamoto has quit IRC | 08:55 | |
| *** yamamoto has joined #openstack-lbaas | 08:55 | |
| *** ramishra has quit IRC | 09:28 | |
| *** ramishra has joined #openstack-lbaas | 09:34 | |
| *** sapd1_ has quit IRC | 09:48 | |
| *** sapd1__ has joined #openstack-lbaas | 09:48 | |
| *** yamamoto has quit IRC | 09:54 | |
| *** yamamoto has joined #openstack-lbaas | 09:55 | |
| *** yamamoto has quit IRC | 09:55 | |
| *** pcaruana has joined #openstack-lbaas | 09:56 | |
| *** yamamoto has joined #openstack-lbaas | 10:08 | |
| *** yamamoto has quit IRC | 10:17 | |
| *** ramishra_ has joined #openstack-lbaas | 10:18 | |
| *** yamamoto has joined #openstack-lbaas | 10:19 | |
| *** ramishra has quit IRC | 10:20 | |
| *** yamamoto has quit IRC | 10:28 | |
| openstackgerrit | guotao proposed openstack/octavia master: Dumplicate words was deleted in component-design.rst https://review.openstack.org/617560 | 10:36 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: Add client_ca_tls_container_ref to Octavia v2 listener API https://review.openstack.org/612267 | 12:19 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: Add an option to the Octavia V2 listener API for client cert https://review.openstack.org/612268 | 12:19 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: Add crl-file option for certification https://review.openstack.org/612269 | 12:19 |
| openstackgerrit | ZhaoBo proposed openstack/python-octaviaclient master: Add client_crl_container_ref for Listener API in CLI https://review.openstack.org/617619 | 12:21 |
| nmagnezi | ll | 12:23 |
| *** xgerman_ is now known as xgerman | 12:36 | |
| *** yamamoto has joined #openstack-lbaas | 12:42 | |
| *** yamamoto has quit IRC | 12:46 | |
| *** yamamoto has joined #openstack-lbaas | 12:50 | |
| *** aojea_ has joined #openstack-lbaas | 13:38 | |
| *** yamamoto has quit IRC | 13:39 | |
| *** yamamoto has joined #openstack-lbaas | 13:39 | |
| *** yamamoto has quit IRC | 13:41 | |
| *** yamamoto has joined #openstack-lbaas | 13:42 | |
| *** yamamoto has quit IRC | 13:53 | |
| *** yamamoto has joined #openstack-lbaas | 13:55 | |
| *** yamamoto has quit IRC | 13:59 | |
| *** yamamoto has joined #openstack-lbaas | 14:02 | |
| *** aojea_ has quit IRC | 14:03 | |
| *** aojea_ has joined #openstack-lbaas | 14:03 | |
| *** yamamoto has quit IRC | 14:04 | |
| *** aojea_ has quit IRC | 14:04 | |
| *** aojea_ has joined #openstack-lbaas | 14:05 | |
| *** yamamoto has joined #openstack-lbaas | 14:06 | |
| *** yamamoto has quit IRC | 14:11 | |
| *** yamamoto has joined #openstack-lbaas | 14:13 | |
| *** yamamoto has quit IRC | 14:18 | |
| *** yamamoto_ has joined #openstack-lbaas | 14:18 | |
| *** yamamoto_ has quit IRC | 14:18 | |
| *** yamamoto has joined #openstack-lbaas | 14:18 | |
| *** yamamoto has quit IRC | 14:23 | |
| *** aojea_ has quit IRC | 14:37 | |
| *** aojea_ has joined #openstack-lbaas | 14:38 | |
| *** yamamoto has joined #openstack-lbaas | 14:39 | |
| *** velizarx has joined #openstack-lbaas | 14:39 | |
| *** aojea_ has quit IRC | 14:42 | |
| *** aojea_ has joined #openstack-lbaas | 14:45 | |
| *** yamamoto_ has joined #openstack-lbaas | 14:56 | |
| *** yamamoto_ has quit IRC | 14:56 | |
| *** yamamoto has quit IRC | 14:56 | |
| *** yamamoto has joined #openstack-lbaas | 14:58 | |
| *** yamamoto has quit IRC | 14:59 | |
| *** velizarx has quit IRC | 15:02 | |
| *** yamamoto has joined #openstack-lbaas | 15:12 | |
| *** sapd1 has joined #openstack-lbaas | 15:20 | |
| *** yamamoto has quit IRC | 15:36 | |
| *** yamamoto has joined #openstack-lbaas | 15:39 | |
| *** salmankhan has joined #openstack-lbaas | 15:43 | |
| *** salmankhan has quit IRC | 15:57 | |
| *** salmankhan has joined #openstack-lbaas | 15:57 | |
| openstackgerrit | zhouxinyong proposed openstack/octavia master: Update the HTTP links to HTTPS in run.yaml. https://review.openstack.org/617718 | 15:59 |
| *** salmankhan has quit IRC | 16:01 | |
| *** yamamoto has quit IRC | 16:02 | |
| *** ccamposr__ has quit IRC | 16:14 | |
| *** velizarx has joined #openstack-lbaas | 16:17 | |
| *** yamamoto has joined #openstack-lbaas | 16:18 | |
| *** ramishra_ has quit IRC | 16:32 | |
| *** velizarx has quit IRC | 16:35 | |
| *** yamamoto has quit IRC | 16:38 | |
| *** irclogbot_1 has joined #openstack-lbaas | 16:41 | |
| *** irclogbot_1 has quit IRC | 16:43 | |
| *** ccamposr has joined #openstack-lbaas | 17:11 | |
| *** aojea_ has quit IRC | 17:33 | |
| *** aojea has joined #openstack-lbaas | 17:33 | |
| *** ianychoi has quit IRC | 18:00 | |
| *** ianychoi has joined #openstack-lbaas | 18:01 | |
| johnsom | In the keynotes again! We are on a roll | 18:09 |
| *** sapd1 has quit IRC | 18:11 | |
| jitek4 | johnsom: have a good summit ! | 18:20 |
| johnsom | I didn't go, but I am watching some of the videos. German and Carlos are there to represent! | 18:21 |
| xgerman | Yep. 1st talk (Octavia onboarding) went great - | 18:22 |
| xgerman | Yeah. Our logo was there several times -/ | 18:22 |
| johnsom | Excellent! Octavia was mentioned in at least two of the keynotes. | 18:22 |
| xgerman | Yeah. People love us :-) | 18:23 |
| openstackgerrit | boden proposed openstack/neutron-lbaas master: use neutron-lib for _model_query https://review.openstack.org/617782 | 18:40 |
| jitek4 | johnsom: video are already available ? | 18:43 |
| johnsom | jitek4 https://www.openstack.org/videos/ | 18:44 |
| johnsom | They are starting to get posted | 18:44 |
| *** aojea has quit IRC | 18:46 | |
| jitek4 | johnsom: thanks, I will take a look ! I wasn't expecting video to be uploaded so fast after sessions | 18:48 |
| *** irclogbot_1 has joined #openstack-lbaas | 19:11 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add amphora statistics to the admin API https://review.openstack.org/585031 | 19:20 |
| *** larsks has joined #openstack-lbaas | 19:43 | |
| larsks | Creating a load balancer requires a "public subnet". We have an external network with a public subnet, but while the network is visible to cloud users, the subnet is not. What's the correct way to set this up? | 19:46 |
| johnsom | larsks Octavia does not have a requirement to use public subnets. | 19:48 |
| larsks | johnsom: I'm looking at https://docs.openstack.org/octavia/pike/user/guides/basic-cookbook.html | 19:48 |
| larsks | Maybe "require" is the wrong word, but presumably if you want public inbound access to the lb you need one? | 19:48 |
| larsks | Or not? | 19:48 |
| johnsom | You can specify private subnets as well as public ones. With private subnets for the VIP, you can optionally use floating IPs. | 19:49 |
| larsks | Oh, reading further, I see that "public subnet" really just means "incoming subnet". | 19:49 |
| larsks | That makes more sense. | 19:49 |
| johnsom | However, you can also specify a network and not just a subnet | 19:49 |
| johnsom | This may help clarify: https://developer.openstack.org/api-ref/load-balancer/v2/index.html?expanded=create-a-load-balancer-detail#create-a-load-balancer | 19:50 |
| larsks | Thanks. I'll take a look and give both of those options a try. | 19:50 |
| johnsom | It talks to the various VIP port options | 19:50 |
| *** aojea has joined #openstack-lbaas | 19:51 | |
| *** irclogbot_1 has quit IRC | 20:09 | |
| *** irclogbot_1 has joined #openstack-lbaas | 20:13 | |
| larsks | johnsom: octavia successfully creates an amphora, but it doesn't seem to be available on the lb management network (even when trying to ping it directly from the relevant dhcp namespace). Is there a common cause for that? Regular nova servers come up just fine without networking issues (so neutron in general seems to work correctly). | 20:18 |
| johnsom | larsks Ping may be disabled, have you tried connecting to the amphora-agent port 9443 (It is a TLS port)? Is that open in the right security group (amp_secgroup_list)? | 20:22 |
| *** aojea has quit IRC | 20:23 | |
| *** yamamoto has joined #openstack-lbaas | 20:24 | |
| larsks | johnsom: okay, it is actually responding on port 9443. But the loadbalancer itself is stuck in 'PENDING CREATE' state, and worker.log is filled with 'Failed to establish a new connection: [Errno 113] No route to host',))' | 20:29 |
| johnsom | Ok, yeah, it will keep trying to connect and, depending on your timeout settings, it will either go ACTIVE or ERROR. | 20:30 |
| larsks | Right. But why is it failing to connect? The amphora seems to be up and running at the correct ip address. | 20:30 |
| johnsom | The default timeouts are super long, like 25 minutes or retries. Typically that is tuned down. | 20:30 |
| johnsom | Check you nova console log. Has the instance actually finished booting? Mis-configured hypervisors or virtualbox can take 10+ minutes to fully boot an instance. | 20:31 |
| johnsom | If that is not it, check that the worker process is for sure able to reach the instance on port 9443. | 20:32 |
| larsks | The instance shows state ACTIVE. The instance actually responds to a `ip netns exec qdhcp-e365e7cb-d4bd-4284-a7c7-f35d7c2e7c3a curl -i -k https://172.24.0.14:9443`, so it seems to be up and running. | 20:32 |
| larsks | (I get back a JSON 404 response) | 20:33 |
| johnsom | Nova ACTIVE is just that the process started, it does not mean anything is running in it. | 20:33 |
| larsks | Right, but the fact that I'm getting an HTTP response suggests that *something* is running. What's a better way to check? | 20:34 |
| johnsom | However that curl implies it is running, so I would look at the networking | 20:34 |
| larsks | Since the connection from the qdhcp namespace seems to work, where would I next look? I'm not sure how the octavia workers connect to the amphora. | 20:35 |
| johnsom | The Octavia worker process makes a TCP connection to the 9443 port on the amphoras. So, the worker, healthmanager, and housekeeping all need to have a route to the amphora/lb-mgmt-net | 20:36 |
| johnsom | How did you install? devstack, OSA, tripleo, puppet, or kolla? | 20:36 |
| larsks | Do they make that connection from inside an appropriate namespace? Or is the host on which they are running supposed to be able to route directly to that network? | 20:36 |
| larsks | This is a tripleo install. | 20:36 |
| johnsom | Hmmm, ok, I'm not super familiar with triple-o and our guy that does is at the summit this week. | 20:37 |
| johnsom | If I remember right, the controllers are running in a container under tripleo. That lb-mgmt-net should be plumbed into the container such that the processes should be able to reach it. | 20:38 |
| johnsom | Because different deployments handle the controllers differently, it's up to the deployer to plumb the network. In this case tripleo should have done that for you. | 20:39 |
| larsks | Okay, I'll take a look at that. FOr a lb that's gone into the "office" state, how do I ask octavia to try spinning it back up? | 20:40 |
| larsks | Err, "offline". | 20:40 |
| larsks | (Octavia finally timed out the one I'm looking at and shut down the nova instance) | 20:41 |
| johnsom | When it goes to ERROR, you can either delete it and re-create it, or use the "failover" APIs | 20:41 |
| larsks | I'll just delete and re-create. | 20:41 |
| larsks | Okay, well, there's the problem. While the controllers appear to have an interface on the correct network, it's unable to reach the amphora. Thanks, that gives me something to look at. | 20:43 |
| johnsom | +1 Good luck! | 20:43 |
| *** yamamoto has quit IRC | 21:00 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix devstack plugin for /var/log/dib-build exists https://review.openstack.org/617838 | 21:09 |
| *** aojea_ has joined #openstack-lbaas | 21:15 | |
| *** aojea_ has quit IRC | 21:48 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: DNM: two dumb downstream things to fix, IGNORE ME https://review.openstack.org/593986 | 21:52 |
| *** pcaruana has quit IRC | 21:56 | |
| *** yamamoto has joined #openstack-lbaas | 22:22 | |
| *** yamamoto has quit IRC | 22:27 | |
| *** yboaron has quit IRC | 22:33 | |
| *** aojea has joined #openstack-lbaas | 22:40 | |
| *** velizarx has joined #openstack-lbaas | 22:45 | |
| *** abaindur has joined #openstack-lbaas | 23:01 | |
| *** abaindur has quit IRC | 23:02 | |
| *** abaindur has joined #openstack-lbaas | 23:03 | |
| *** aojea has quit IRC | 23:12 | |
| *** yamamoto has joined #openstack-lbaas | 23:19 | |
| *** yamamoto has quit IRC | 23:24 | |
| *** velizarx has quit IRC | 23:25 | |
| *** yamamoto has joined #openstack-lbaas | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!