| *** spatel has quit IRC | 00:10 | |
| *** openstackgerrit has joined #openstack-lbaas | 00:33 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Make amphora use a single HAProxy instance https://review.opendev.org/668068 | 00:33 |
|---|---|---|
| *** hongbin has joined #openstack-lbaas | 00:57 | |
| *** ricolin has joined #openstack-lbaas | 01:23 | |
| *** mithilarun has quit IRC | 01:29 | |
| *** spatel has joined #openstack-lbaas | 01:30 | |
| *** irclogbot_2 has joined #openstack-lbaas | 02:06 | |
| *** irclogbot_2 has quit IRC | 02:13 | |
| *** KeithMnemonic has quit IRC | 02:13 | |
| *** irclogbot_3 has joined #openstack-lbaas | 02:16 | |
| *** altlogbot_0 has joined #openstack-lbaas | 02:18 | |
| *** irclogbot_3 has quit IRC | 02:19 | |
| *** rcernin has quit IRC | 02:29 | |
| *** altlogbot_0 has quit IRC | 02:34 | |
| *** rcernin has joined #openstack-lbaas | 02:46 | |
| *** irclogbot_2 has joined #openstack-lbaas | 03:20 | |
| *** psachin has joined #openstack-lbaas | 03:21 | |
| *** irclogbot_2 has quit IRC | 03:25 | |
| *** psachin has quit IRC | 03:38 | |
| *** rcernin has quit IRC | 03:39 | |
| *** hongbin has quit IRC | 03:44 | |
| *** spatel has quit IRC | 03:50 | |
| *** altlogbot_1 has joined #openstack-lbaas | 03:54 | |
| *** rcernin has joined #openstack-lbaas | 03:55 | |
| openstackgerrit | Noboru Iwamatsu proposed openstack/octavia master: Add failover logging to show the amphora details. https://review.opendev.org/667316 | 03:56 |
| *** altlogbot_1 has quit IRC | 03:59 | |
| *** gcheresh_ has joined #openstack-lbaas | 04:34 | |
| *** pcaruana has joined #openstack-lbaas | 04:55 | |
| *** ccamposr has joined #openstack-lbaas | 05:18 | |
| *** ccamposr__ has quit IRC | 05:20 | |
| *** yamamoto has quit IRC | 05:27 | |
| *** yamamoto has joined #openstack-lbaas | 05:33 | |
| *** gcheresh_ has quit IRC | 05:34 | |
| *** yamamoto has quit IRC | 05:40 | |
| *** gcheresh_ has joined #openstack-lbaas | 05:40 | |
| *** ajay33 has joined #openstack-lbaas | 05:40 | |
| *** mithilarun has joined #openstack-lbaas | 05:53 | |
| *** ccamposr__ has joined #openstack-lbaas | 05:57 | |
| *** gcheresh_ has quit IRC | 05:58 | |
| *** ccamposr has quit IRC | 06:00 | |
| *** mithilarun has quit IRC | 06:03 | |
| *** altlogbot_3 has joined #openstack-lbaas | 06:18 | |
| *** altlogbot_3 has quit IRC | 06:23 | |
| *** altlogbot_2 has joined #openstack-lbaas | 06:24 | |
| *** altlogbot_2 has quit IRC | 06:29 | |
| *** rpittau|afk is now known as rpittau | 06:33 | |
| *** yamamoto has joined #openstack-lbaas | 06:35 | |
| *** ricolin has quit IRC | 06:39 | |
| *** luksky11 has joined #openstack-lbaas | 06:39 | |
| *** yamamoto has quit IRC | 06:45 | |
| *** ricolin has joined #openstack-lbaas | 06:46 | |
| *** altlogbot_1 has joined #openstack-lbaas | 06:56 | |
| *** altlogbot_1 has quit IRC | 07:01 | |
| *** ianychoi_ is now known as ianychoi | 07:01 | |
| *** ivve has joined #openstack-lbaas | 07:04 | |
| *** gcheresh_ has joined #openstack-lbaas | 07:11 | |
| ivve | johnsom, sapd1: i solved the initial problem (from yesterday) however, when i poke the failover api it creates one new amphora and sets it in allocated, standalone. the lb's state enters pending_update and healthmanager keeps yelling that it can't reach the "other" amphora in the lb | 07:11 |
| ivve | so basically it looks like it created the backup and keeps whining about the master is not reachable. further poking failover won't work and i have tried manipulating updates on agent and fiddling with amphoras, but they are all immutable | 07:12 |
| *** irclogbot_0 has joined #openstack-lbaas | 07:16 | |
| *** rcernin has quit IRC | 07:17 | |
| ivve | (initial problem was dns, which is solved now) | 07:20 |
| *** irclogbot_0 has quit IRC | 07:21 | |
| *** irclogbot_0 has joined #openstack-lbaas | 07:24 | |
| *** irclogbot_0 has quit IRC | 07:27 | |
| *** altlogbot_0 has joined #openstack-lbaas | 07:48 | |
| *** altlogbot_0 has quit IRC | 07:49 | |
| *** irclogbot_2 has joined #openstack-lbaas | 07:52 | |
| *** irclogbot_2 has quit IRC | 07:55 | |
| *** irclogbot_2 has joined #openstack-lbaas | 08:08 | |
| *** tkajinam has quit IRC | 08:10 | |
| ivve | im getting some nova-api error regarding PortNotUsableDNS | 08:12 |
| *** irclogbot_2 has quit IRC | 08:13 | |
| *** yamamoto has joined #openstack-lbaas | 08:32 | |
| *** ccamposr has joined #openstack-lbaas | 08:51 | |
| *** ccamposr__ has quit IRC | 08:53 | |
| *** ianychoi has quit IRC | 08:54 | |
| *** yamamoto has quit IRC | 08:55 | |
| *** yamamoto has joined #openstack-lbaas | 09:02 | |
| *** yamamoto has quit IRC | 09:02 | |
| *** yamamoto has joined #openstack-lbaas | 09:03 | |
| *** yamamoto has quit IRC | 09:07 | |
| *** yamamoto has joined #openstack-lbaas | 09:08 | |
| *** ianychoi has joined #openstack-lbaas | 09:38 | |
| *** yamamoto has quit IRC | 09:44 | |
| *** irclogbot_0 has joined #openstack-lbaas | 09:48 | |
| *** irclogbot_0 has quit IRC | 09:51 | |
| *** psachin has joined #openstack-lbaas | 10:13 | |
| *** altlogbot_2 has joined #openstack-lbaas | 10:26 | |
| *** altlogbot_2 has quit IRC | 10:29 | |
| *** altlogbot_3 has joined #openstack-lbaas | 10:34 | |
| straldi | For the problem of yesterday I try different way to configure the keystone section and auth section in octavia.conf but I have always the same problem. I try to put the debug to true and in the api log I found 2019-07-11 12:34:41.931 10793 DEBUG wsme.api [req-19024fa2-20e4-4433-a75e-4a7ff1c2a373 - - - - -] Client-side error: Validation failure: Missing project ID in request where one is required. format_exception /usr/lib/python2.7/site-packa | 10:37 |
| straldi | but the --project is not mandatory in client openstak loadbalancer create. I add I have in my openrc file the OS_PROJECT_NAME set | 10:38 |
| *** altlogbot_3 has quit IRC | 10:39 | |
| *** altlogbot_3 has joined #openstack-lbaas | 10:40 | |
| *** irclogbot_1 has joined #openstack-lbaas | 10:40 | |
| *** altlogbot_3 has quit IRC | 10:45 | |
| *** irclogbot_1 has quit IRC | 10:45 | |
| *** altlogbot_1 has joined #openstack-lbaas | 10:56 | |
| *** altlogbot_1 has quit IRC | 11:01 | |
| *** tesseract has joined #openstack-lbaas | 11:21 | |
| *** yamamoto has joined #openstack-lbaas | 11:45 | |
| *** yamamoto has quit IRC | 11:53 | |
| *** altlogbot_2 has joined #openstack-lbaas | 11:56 | |
| *** altlogbot_2 has quit IRC | 12:01 | |
| sapd1 | ivve, oh, because the health manager has job from message queue, so it continue to failover loadbalancer. you can restart healthmanager. You have to check 2 amphora which are allocated for this loadbalancer, If you have only one, you can update status in your database from DELETED to ACTIVE then perform failover again. | 12:02 |
| sapd1 | this is tricky. | 12:02 |
| ivve | ok just status, no problem | 12:03 |
| ivve | sapd1: ill test that, thanks! | 12:04 |
| *** boden has joined #openstack-lbaas | 12:12 | |
| *** henriqueof has joined #openstack-lbaas | 12:23 | |
| *** henriqueof has quit IRC | 12:28 | |
| *** irclogbot_1 has joined #openstack-lbaas | 12:32 | |
| *** KeithMnemonic has joined #openstack-lbaas | 12:33 | |
| *** irclogbot_1 has quit IRC | 12:35 | |
| *** luksky11 has quit IRC | 12:41 | |
| *** altlogbot_1 has joined #openstack-lbaas | 12:50 | |
| *** altlogbot_1 has quit IRC | 12:53 | |
| *** yamamoto has joined #openstack-lbaas | 13:23 | |
| *** luksky11 has joined #openstack-lbaas | 13:29 | |
| *** goldyfruit has joined #openstack-lbaas | 13:33 | |
| *** irclogbot_0 has joined #openstack-lbaas | 13:40 | |
| *** irclogbot_0 has quit IRC | 13:45 | |
| *** yamamoto has quit IRC | 13:51 | |
| dulek | cgoncalves, johnsom: Hi! I assume that the Rocky version issue with HTTPS health monitors and not being able to set url_path is known? | 14:10 |
| dulek | cgoncalves, johnsom: We just stumbled upon it. My question is - for HTTPS can I just not set url_path and it'll default to "/"? | 14:12 |
| johnsom | dulek Rocky didn't support HTTPS health monitors | 14:12 |
| dulek | johnsom: So this is incorrect: https://github.com/openstack/octavia/blob/stable/rocky/octavia/common/constants.py#L35-L38 ? | 14:13 |
| johnsom | dulek Well, I should clarify, it supports the "HTTPS" type, but that is a TLS handshake, not a "GET" | 14:14 |
| dulek | johnsom: Oh, this actually does help me! :) | 14:14 |
| dulek | Though it's a bit hard now to code HM for both Rocky and Stein… | 14:15 |
| dulek | But in my case it'll work as I can live with "/" being the url_path. | 14:15 |
| dulek | johnsom: Thanks! | 14:15 |
| *** gcheresh_ has quit IRC | 14:17 | |
| johnsom | dulek Backend TLS came in Stein | 14:18 |
| johnsom | dulek In Stein, HTTPS becomes a GET, and we added TLS-HELLO | 14:19 |
| johnsom | straldi That sounds like the keystone_auth configuration is unable to get the user's project information. | 14:20 |
| *** yamamoto has joined #openstack-lbaas | 14:22 | |
| *** spatel has joined #openstack-lbaas | 14:23 | |
| *** spatel has quit IRC | 14:23 | |
| *** amuller has joined #openstack-lbaas | 14:27 | |
| dulek | johnsom: Can I discover that through versions? git blame doesn't help and there seems nothing on API reference about what version supports what? | 14:28 |
| straldi | johnsom yes seems like that, but I don't know why. All the other commands (openstack server, network, subnet, image,...) for component nova, glance, neutron work fine. So it is only octavia is not working properly. Now I install only octavia may be I have to configure neutron with lbaas pointing octavbia? | 14:29 |
| straldi | octavia | 14:29 |
| *** maciejjozefczyk has joined #openstack-lbaas | 14:29 | |
| johnsom | straldi We don't recommend using neutron-lbaas. It has been retired (code is removed from master). It also will not solve this issue. | 14:31 |
| johnsom | stralfi Try copying the keystone_auth configuration from your nova/neutron configuration files into the keystone_auth section in the octavia.conf. That isn't octavia code, it's a shared library we all use, so it should work the same. | 14:32 |
| straldi | yes I know and it is beacuse I decide to install only octavia on the top of my openstack infrastructure | 14:32 |
| johnsom | dulek I just noticed that myself. I don't even see the release note I thought was included in that change. I'm looking for zhao's patch now | 14:33 |
| johnsom | straldi Yes, standalone works much better. | 14:33 |
| maciejjozefczyk | johnsom, Hello. Going back to the topic here: https://storyboard.openstack.org/#!/story/2006196. What do you mean by 'planned persistent driver code interface'? | 14:34 |
| *** gcheresh_ has joined #openstack-lbaas | 14:35 | |
| dulek | johnsom: Yeah, we get some fuzzy behaviors as it happened we have 3 devs and each one tests with Queens, Rocky or Stein. :D | 14:35 |
| johnsom | maciejjozefczyk So the current provider driver design is for transient use. There is no mechanism to spawn a long running thread. The plan was to add a "long running" plugin option for the Driver Agent. Your same driver code would hook both the current provider driver, but also a longer running hook in driver agent. | 14:35 |
| johnsom | dulek Worst case, you should get 400 back if you pick a combination that doesn't work, but this should be called out. | 14:38 |
| dulek | johnsom: Hm, right, it was 400. | 14:39 |
| maciejjozefczyk | johnsom, Thanks. Do you have an open story for this? | 14:41 |
| johnsom | dulek The offending patch is here: https://review.opendev.org/#/c/475944/ | 14:41 |
| straldi | I tried just now to put exactly the conf I found in neutron.conf for keystone_auth (keeping the correct username and password) I restart all octavia services but I have the same problem. The strange thing is that in octavia log (DEBUG) say Client-side error: Validation failure: Missing project ID in request where one is required. | 14:42 |
| *** dalvarez has joined #openstack-lbaas | 14:42 | |
| straldi | So seems the project is mandatory, ahh I can create the loadbalancer using --project but I wold like to create the loadbalancer also without that aptional argument | 14:44 |
| johnsom | straldi No, the project ID by default comes from your keystone credentials. Only if that is not available or if the admin overrides is the --project used. Then, if that is also not specified, it will error that we could not find user credentials. | 14:45 |
| dulek | johnsom: Uhm… That patch is from 2017 and seems to be included even in Pike? | 14:46 |
| straldi | i source my keystone_demo | 14:47 |
| straldi | unset OS_SERVICE_TOKEN | 14:47 |
| straldi | export OS_USERNAME=demo | 14:47 |
| straldi | export OS_PASSWORD='8c735bb4554444b7' | 14:47 |
| straldi | export PS1='[\u@\h \W(keystone_demo)]\$ ' | 14:47 |
| straldi | export OS_AUTH_URL=http://192.168.60.171:5000/v3 | 14:47 |
| straldi | 14:47 | |
| straldi | export OS_PROJECT_NAME=demo | 14:47 |
| straldi | export OS_USER_DOMAIN_NAME=Default | 14:47 |
| straldi | export OS_PROJECT_DOMAIN_NAME=Default | 14:47 |
| straldi | export OS_IDENTITY_API_VERSION=3 | 14:47 |
| straldi | I have export OS_PROJECT_NAME=demo | 14:47 |
| johnsom | dulek Yeah, see that. It isn't the right one. I need to find Zhao's | 14:48 |
| johnsom | straldi Do you have "OS_AUTH_TYPE=password" configured? | 14:49 |
| straldi | yes for sure | 14:49 |
| johnsom | straldi Also, there are no other keystone errors in your API log? | 14:49 |
| johnsom | straldi Would you be ok sharing your octavia.conf in a paste.openstack.org? You can mark it private and send me the link directly if you would like. | 14:51 |
| straldi | here we are http://paste.openstack.org/show/754300/ | 14:57 |
| straldi | no need to be private. All ip is in my private subnet and all data is not secret | 14:58 |
| johnsom | dulek Blah, I think the changes to enable TLS on backends auto-magically changed that behavior. When the backend is TLS enabled, it auto-magically escalates the health monitor to a full HTTPS check. Still looking at Zhao's patches. | 14:58 |
| johnsom | straldi Ok cool, give me a few minutes to look it over. | 14:58 |
| *** gcheresh_ has quit IRC | 14:58 | |
| straldi | thanks very much | 14:58 |
| dulek | johnsom: You see there's one thing you cannot do in a portable way now. | 15:00 |
| *** yamamoto has quit IRC | 15:00 | |
| dulek | johnsom: Before this one I wasn't able to set url_path for HTTPS: https://review.opendev.org/#/c/604924/ | 15:00 |
| dulek | johnsom: And after this patch I can. | 15:00 |
| *** yamamoto has joined #openstack-lbaas | 15:01 | |
| *** yamamoto has quit IRC | 15:01 | |
| johnsom | dulek Yeah, I don't think either of the people involved in that work are online now | 15:01 |
| dulek | :) | 15:02 |
| dulek | Okay, fair enough. Thank you for help anyway! | 15:02 |
| *** Vorrtex has joined #openstack-lbaas | 15:02 | |
| johnsom | I can poke into it more (busy morning), but it might be worth opening a story if there is a change you would like to see. | 15:02 |
| dulek | I don't really see a useful change here. Rocky deployments with this issue are out there, I need to support them. | 15:05 |
| dulek | I can only figure out which API version = Rocky and work it around. | 15:05 |
| johnsom | dulek Rocky didn't have backend TLS, so all it can do is simple TLS checks | 15:05 |
| johnsom | URL path isn't valid on HTTPS on Rocky | 15:05 |
| johnsom | straldi Are you using keystone v3 API? Can you paste the output of "openstack endpoint list"? | 15:06 |
| straldi | yes I am | 15:07 |
| johnsom | straldi Also, can you check that you have an "octavia" account? You are using packstack right? They may be using the service account and not setting up an octavia account. | 15:07 |
| maciejjozefczyk | johnsom, Is there any open story this for 'long running' plugin option? Spec? Or just an idea? | 15:07 |
| johnsom | maciejjozefczyk I think there is a story or two. I just haven't had time to go dig for it yet. It might be mentioned in the provider driver specs. | 15:08 |
| maciejjozefczyk | johnsom, Ok, I'll try to find | 15:09 |
| straldi | johnsom here all your question: http://paste.openstack.org/show/754302/ | 15:09 |
| straldi | No I use packstack for nova, neutron, keystone, glance, cinder ... simply with this command packstack --install-hosts 192.168.60.171,192.168.60.94 but for octavia I install everything by hand | 15:11 |
| *** pcaruana has quit IRC | 15:11 | |
| straldi | following almost this docs: https://blog.zufardhiyaulhaq.com/manual-instalation-octavia-openstack-queens/ so create the user, the security group, the image amphora, flavor, ... | 15:14 |
| dalvarez | maciejjozefczyk: johnsom sorry i may have lost something, so instantiating a driver per API call is by design and expected right? | 15:16 |
| johnsom | straldi Since you are using keystone v3, let's try this. Remove the auth_uri line, then change the auth_url line to "auth_url=http://192.168.60.171:5000/v3" | 15:16 |
| straldi | ok I'm tryng | 15:17 |
| johnsom | dalvarez Yes, it was a choice we made during the design phase. | 15:17 |
| johnsom | dalvarez It helps protect against poorly written drivers that leak resources. They are transient by design. It also is supposed to allow dynamic updates of the drivers, but I think I saw a report that stevedore isn't doing that as was expected. | 15:18 |
| dalvarez | johnsom: got it, i guess there's good reasons for that but in ovn, it's causing us to instantiate new connection to OVSDB, getting a full dump of the db contents and process it, etc. | 15:18 |
| dalvarez | stevedore is apparently creating a new fresh instance every time but im not sure is that what's expected or a bug | 15:18 |
| johnsom | dalvarez Yeah, that sounds like a design issue on ovn. It sounds like it's leaking resources there. | 15:19 |
| johnsom | dalvarez Yes, that is what it is supposed to be doing. It's just the "pick up new code" part that might not be working. | 15:19 |
| dalvarez | johnsom: it's not really leaking anything, i mean, it's a new instance of the driver and it just connects to the db on startup | 15:20 |
| straldi | Now I have: | 15:20 |
| straldi | [keystone_authtoken] | 15:20 |
| straldi | www_authenticate_uri=http://192.168.60.171:5000/ | 15:20 |
| straldi | auth_url=http://192.168.60.171:5000/v3 | 15:20 |
| straldi | I remove auth_uri | 15:20 |
| straldi | but the same problem | 15:20 |
| dalvarez | johnsom: how is that leaking? | 15:20 |
| johnsom | dalvarez So we can look at what you need and figure out a way to either fix the ovn side or switch over to using the driver agent (yet to be implemented) | 15:20 |
| straldi | I put also in section [service_auth] the v3 | 15:20 |
| johnsom | dalvarez I thought the story said it was leaking connections to the ovsdb | 15:20 |
| johnsom | straldi Can you remove the "www_authenticate_uri" too? Not sure what that is needed for. | 15:21 |
| straldi | yes | 15:21 |
| dalvarez | johnsom: maciejjozefczyk what i read in the story is "From OVN perspective it is not optimal because it creates a connection to OVS dbs each time" | 15:21 |
| johnsom | dalvarez I'm in four conversations and listening to a meeting right now, so very time sliced here. | 15:22 |
| dalvarez | johnsom: ack :) let's do it when you have the time to read the story | 15:22 |
| straldi | same problem | 15:23 |
| dalvarez | johnsom: let's sync tomorrow when maciejjozefczyk or via the story as you suggested | 15:23 |
| johnsom | dalvarez Yeah, I would really like to capture the discussion in the story as then the folks offline now can also contribute. | 15:24 |
| dalvarez | ++ | 15:25 |
| johnsom | straldi Also, I saw a commented line "# auth_strategy = noauth" That should not be noauth, but commented out should be the correct default of "keystone". | 15:25 |
| johnsom | straldi Line 298 can be removed, event_stream_transport_url is not needed anymore. (this isn't the project ID problem though) | 15:27 |
| straldi | ok I'm trying | 15:27 |
| straldi | ok | 15:27 |
| johnsom | straldi You do have a project named "services" right? Usually it's "service" | 15:28 |
| johnsom | That would be the other change to keystone_auth I would check | 15:29 |
| straldi | yesss you find the misconfiguration. | 15:30 |
| straldi | auth_strategy = keystone solve | 15:30 |
| straldi | thanks a lot | 15:30 |
| straldi | I commented also event_stream_transport_url | 15:30 |
| straldi | thanks a lot | 15:30 |
| johnsom | straldi Ok, cool. Also, in the service_auth section, remove the URI and update the URL to the /v3 we did for keystone_auth | 15:30 |
| straldi | I did | 15:31 |
| johnsom | +1 | 15:31 |
| johnsom | Cool. Glad you are up and running | 15:31 |
| straldi | ok now I will play a little bit with octavia and I will let you know the feeling. Thanks a lot | 15:32 |
| ivve | johnsom: is there any workaround for PortNotUsableDNS_Remote atm ? | 15:32 |
| johnsom | ivve I don't know what that is. Can you provide context? Octavia doesn't use DN | 15:33 |
| johnsom | DNS | 15:33 |
| ivve | the error comes from nova-api when recreating amphorae | 15:34 |
| ivve | you pasted it a few years ago in a meeting, its the only hit i can find :) | 15:34 |
| ivve | Exception during message handling: PortNotUsableDNS: Port ad8ae71d-4517-419b-8937-50737253416a not usable for instance 47eec7dc-13c0-4bed-9736-f4803fcc52fd. Value amphora-2a9a4ca9-5c88-415b-b874-8bb67ec28d78 assigned to dns_name attribute does not match instance's hostname amphora-e8b4dee6-65e4-4489-8742-bb8e5835d9be | 15:34 |
| johnsom | Hmm, ok. We worked around that bug in neutron a long time ago. Is it back? | 15:35 |
| ivve | yea | 15:35 |
| ivve | well this version is | 15:35 |
| ivve | rocky | 15:35 |
| johnsom | Yeah, it was newton where we worked around it if I remember. Whenever designate integrated with neutron and caused this. | 15:36 |
| johnsom | mugsie Are you around? | 15:36 |
| mugsie | Kinda | 15:37 |
| ivve | still not tested latest stein in lab | 15:37 |
| johnsom | ivve Here is our workaround code: https://github.com/openstack/octavia/blob/master/octavia/network/drivers/neutron/allowed_address_pairs.py#L553 | 15:37 |
| ivve | (neutron) | 15:37 |
| ivve | checking | 15:37 |
| johnsom | mugsie So people are seeing PortNotUsableDNS errors again. I thought you all fixed that to not be a failure. | 15:38 |
| johnsom | Here is the full error if you don't have the scrollback: | 15:38 |
| johnsom | Exception during message handling: PortNotUsableDNS: Port ad8ae71d-4517-419b-8937-50737253416a not usable for instance 47eec7dc-13c0-4bed-9736-f4803fcc52fd. Value amphora-2a9a4ca9-5c88-415b-b874-8bb67ec28d78 assigned to dns_name attribute does not match instance's hostname amphora-e8b4dee6-65e4-4489-8742-bb8e5835d9be | 15:38 |
| mugsie | I thought it was fixed | 15:39 |
| mugsie | Must be a regression in neutron again | 15:40 |
| johnsom | I'm looking to see if our extension discovery code is also contributing. | 15:40 |
| johnsom | ivve Can you check a few things for me? If you do a "openstack extension list" do you see dns-integration in the list? | 15:41 |
| ivve | DNS Integration | dns-integration | Provides integration with DNS. | 15:42 |
| johnsom | ivve Also, can you look in your octavia worker logs for the following message: | 15:43 |
| johnsom | Jul 08 07:35:48 devstack octavia-worker[13140]: DEBUG octavia.network.drivers.neutron.base [None req-048a157e-7be7-476f-8ef7-9398f6bb5343 None None] Neutron extension dns-integration is not enabled {{(pid=13875) _check_extension_enabled /opt/stack/octavia/octavia/network/drivers/neutron/base.py:70}} | 15:43 |
| ivve | just gonna enable debug, one sec | 15:43 |
| *** maciejjozefczyk has quit IRC | 15:47 | |
| ivve | nopes | 15:47 |
| ivve | johnsom: doesn't seem to pop up | 15:48 |
| colin- | can i set a timeout for pending_create somehow? | 15:48 |
| colin- | there must be an upper bound i can enforce so that i don't have to change them in the db when they don't create | 15:48 |
| colin- | instead of 409 indefinitely | 15:48 |
| johnsom | ivve Hmmm, What triggered the failover that caused the ERROR? manual failover or health manager? if manual which type loadbalancer or amphora? | 15:48 |
| ivve | 2019-07-11 15:45:48.858 20 DEBUG octavia.network.drivers.neutron.base [-] Neutron extension dns-integration found enabled _check_extension_enabled /var/lib/kolla/venv/local/lib/python2.7/site-packages/octavia/network/drivers/neutron/base.py:66 | 15:49 |
| johnsom | colin- Yes! one minute, I will find it for you. The default is horribly long | 15:49 |
| ivve | johnsom: manual failover via api (cli: openstack loadbalancer failover <uuid> | 15:49 |
| ivve | ) | 15:50 |
| johnsom | colin- [haproxy_amphora] connection_max_retries = 120 and [haproxy_amphora] build_active_retries = 120 are typical settings to shorten that timeout | 15:51 |
| ivve | it managed to rebuild one of my loadbalancers | 15:54 |
| johnsom | ivve Sigh, somehow the workaround we put in place is no longer in our failover flow..... This task: FailoverPreparationForAmphora is missing. | 15:55 |
| ivve | but the other 3 are getting this failure | 15:55 |
| johnsom | ivve So this is a regression bug. | 15:55 |
| ivve | okay, so thats good atleast. found what is causing it | 15:55 |
| ivve | it worked for 1 out of 4 LBs | 15:55 |
| ivve | for some reason unknown to me | 15:55 |
| johnsom | ivve Can you open a story for us? https://storyboard.openstack.org/#!/dashboard/stories | 15:56 |
| *** pcaruana has joined #openstack-lbaas | 15:57 | |
| ivve | almost done | 16:03 |
| *** Vorrtex has quit IRC | 16:03 | |
| johnsom | Thank you | 16:03 |
| ivve | https://storyboard.openstack.org/#!/story/2006205 | 16:03 |
| ivve | should i add some other vital info? | 16:04 |
| *** mithilarun has joined #openstack-lbaas | 16:04 | |
| *** altlogbot_1 has joined #openstack-lbaas | 16:04 | |
| johnsom | ivee That is good enough. I have also added comments that will help whoever fixes this bug. | 16:06 |
| ivve | perfect | 16:06 |
| colin- | do those values elapsing without successful connection cause the LB state to go from PENDING_CREATE to something mutable? | 16:07 |
| johnsom | colin- Yes | 16:07 |
| colin- | weird, ok, mine are stuck in PENDING_CREATE. this was true in rocky? | 16:07 |
| *** altlogbot_1 has quit IRC | 16:07 | |
| colin- | (2-3 days old at this point, still PENDING_CREATE) | 16:08 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add warning log if auth_strategy is not keystone https://review.opendev.org/670342 | 16:09 |
| johnsom | colin- It has been that way since mitaka-ish. If it's been more than an hour, and you didn't change those settings from the default, it likely means someone did a non-graceful shutdown of the controller while it was working on creating that LB. | 16:10 |
| johnsom | colin- This is the work Ann is working on, the jobboard support, that will resolve that. | 16:11 |
| colin- | ok, the services have been up for several days and the retries/interval config i'm running should have reached its natural conclusion after two hours. will try to find something in the log data about why it wouldn't have transitioned out of PENDING_CREATE | 16:13 |
| colin- | get super uncomfortable changing this stuff in the db heh | 16:13 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add warning log if auth_strategy is not keystone https://review.opendev.org/670342 | 16:15 |
| *** luksky11 has quit IRC | 16:19 | |
| *** rpittau is now known as rpittau|adk | 16:20 | |
| *** rpittau|adk is now known as rpittau|afk | 16:20 | |
| *** ivve has quit IRC | 16:23 | |
| *** Vorrtex has joined #openstack-lbaas | 16:27 | |
| colin- | was a value on my side that was responsible johnsom, did bad math on retries x timeout | 16:35 |
| johnsom | colin- ok | 16:35 |
| colin- | had an arbitrarily high value on one | 16:35 |
| johnsom | Yeah, they can get large.... | 16:36 |
| *** tesseract has quit IRC | 16:36 | |
| *** altlogbot_3 has joined #openstack-lbaas | 16:44 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Add active-standby scenario jobs to check queue https://review.opendev.org/670354 | 16:49 |
| *** altlogbot_3 has quit IRC | 16:49 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Add active-standby scenario jobs to check queue https://review.opendev.org/670354 | 16:51 |
| *** psachin has quit IRC | 16:52 | |
| *** yamamoto has joined #openstack-lbaas | 17:04 | |
| *** yamamoto has quit IRC | 17:10 | |
| *** ataraday_ has quit IRC | 17:12 | |
| *** altlogbot_3 has joined #openstack-lbaas | 17:22 | |
| *** altlogbot_3 has quit IRC | 17:25 | |
| *** gcheresh_ has joined #openstack-lbaas | 17:28 | |
| *** altlogbot_2 has joined #openstack-lbaas | 17:30 | |
| *** altlogbot_2 has quit IRC | 17:35 | |
| *** altlogbot_0 has joined #openstack-lbaas | 17:36 | |
| *** gcheresh_ has quit IRC | 17:40 | |
| *** altlogbot_0 has quit IRC | 17:41 | |
| *** altlogbot_0 has joined #openstack-lbaas | 17:43 | |
| *** altlogbot_0 has quit IRC | 17:45 | |
| *** ajay33 has quit IRC | 18:10 | |
| *** luksky11 has joined #openstack-lbaas | 18:44 | |
| *** mithilarun has quit IRC | 18:52 | |
| *** irclogbot_2 has joined #openstack-lbaas | 19:28 | |
| *** irclogbot_2 has quit IRC | 19:33 | |
| *** altlogbot_3 has joined #openstack-lbaas | 19:39 | |
| *** altlogbot_3 has quit IRC | 19:41 | |
| *** mithilarun has joined #openstack-lbaas | 19:42 | |
| *** irclogbot_0 has joined #openstack-lbaas | 20:12 | |
| *** gcheresh_ has joined #openstack-lbaas | 20:13 | |
| *** irclogbot_0 has quit IRC | 20:16 | |
| *** pcaruana has quit IRC | 20:31 | |
| *** Vorrtex has quit IRC | 20:44 | |
| *** mithilarun has quit IRC | 20:56 | |
| *** gcheresh_ has quit IRC | 20:57 | |
| *** altlogbot_1 has joined #openstack-lbaas | 21:00 | |
| *** altlogbot_1 has quit IRC | 21:05 | |
| *** mithilarun has joined #openstack-lbaas | 21:22 | |
| *** irclogbot_1 has joined #openstack-lbaas | 21:23 | |
| *** irclogbot_1 has quit IRC | 21:26 | |
| *** irclogbot_2 has joined #openstack-lbaas | 21:29 | |
| colin- | if i have an amp that's stuck in a BOOTING state and delete the underlying nova resource before it's assigned to any pool or lb, will octavia housekeeping just purge the record and life carries on with no subsequent action (create/destroy)? | 21:31 |
| *** irclogbot_2 has quit IRC | 21:32 | |
| johnsom | It won't get purged, no. but all that will be there is the "BOOTING" record | 21:32 |
| colin- | so i would couple the nova delete with a database edit for each amp to DELETED in order to clean that up completely | 21:33 |
| johnsom | Yeah, I think so. | 21:33 |
| johnsom | Any idea how it got that way? Another non-graceful shutdown? | 21:33 |
| colin- | does the BOOTING state respect the same values we were discussing earlier? i think i'm still within the timeframe and am unsure why they aren't marked active yet | 21:34 |
| colin- | the worker isn't obsessing about them in the output or anything | 21:34 |
| johnsom | Yes, I think it's a similar timeout. It is the state while we wait on nova to start the amp | 21:34 |
| *** boden has quit IRC | 21:57 | |
| *** mithilarun has quit IRC | 22:00 | |
| *** mithilarun has joined #openstack-lbaas | 22:02 | |
| *** altlogbot_0 has joined #openstack-lbaas | 22:19 | |
| *** irclogbot_3 has joined #openstack-lbaas | 22:19 | |
| *** mithilarun has quit IRC | 22:21 | |
| *** altlogbot_0 has quit IRC | 22:23 | |
| *** irclogbot_3 has quit IRC | 22:24 | |
| *** mithilarun has joined #openstack-lbaas | 22:36 | |
| *** luksky11 has quit IRC | 22:37 | |
| *** mithilarun has quit IRC | 22:37 | |
| *** mithilarun has joined #openstack-lbaas | 22:49 | |
| *** tkajinam has joined #openstack-lbaas | 22:52 | |
| *** altlogbot_1 has joined #openstack-lbaas | 22:57 | |
| *** altlogbot_1 has quit IRC | 22:59 | |
| *** irclogbot_3 has joined #openstack-lbaas | 23:01 | |
| *** irclogbot_3 has quit IRC | 23:06 | |
| *** yamamoto has joined #openstack-lbaas | 23:07 | |
| *** irclogbot_1 has joined #openstack-lbaas | 23:09 | |
| *** yamamoto has quit IRC | 23:12 | |
| *** rcernin has joined #openstack-lbaas | 23:13 | |
| *** irclogbot_1 has quit IRC | 23:16 | |
| *** goldyfruit has quit IRC | 23:16 | |
| *** altlogbot_2 has joined #openstack-lbaas | 23:29 | |
| *** altlogbot_2 has quit IRC | 23:29 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Make amphora use a single HAProxy instance https://review.opendev.org/668068 | 23:48 |
| *** goldyfruit has joined #openstack-lbaas | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!