| *** mithilarun has quit IRC | 00:41 | |
| *** yamamoto has quit IRC | 00:49 | |
| *** ricolin has joined #openstack-lbaas | 00:55 | |
| *** goldyfruit has quit IRC | 01:59 | |
| rm_work | hmm yeah, not seeing how that revert could fail | 02:01 |
|---|---|---|
| rm_work | interesting | 02:01 |
| rm_work | though i'm looking at master | 02:01 |
| rm_work | need to check out Queens | 02:01 |
| rm_work | hmm yeah weird, same | 02:02 |
| rm_work | more log context would be good :D | 02:03 |
| rm_work | if the listener didn't have an ID somehow? uhhh, no don't think that's possible | 02:03 |
| rm_work | we log an error on either of the functions it runs failing | 02:04 |
| *** goldyfruit has joined #openstack-lbaas | 02:17 | |
| *** psachin has joined #openstack-lbaas | 03:06 | |
| *** yamamoto has joined #openstack-lbaas | 03:11 | |
| *** gthiemonge has quit IRC | 03:53 | |
| *** yamamoto has quit IRC | 03:53 | |
| *** gthiemonge has joined #openstack-lbaas | 03:54 | |
| *** yamamoto has joined #openstack-lbaas | 04:01 | |
| *** ramishra has joined #openstack-lbaas | 04:17 | |
| *** pcaruana has joined #openstack-lbaas | 04:45 | |
| *** gcheresh has joined #openstack-lbaas | 05:12 | |
| *** gcheresh has quit IRC | 05:19 | |
| *** rcernin has quit IRC | 05:59 | |
| *** vishalmanchanda has joined #openstack-lbaas | 06:05 | |
| *** ivve has joined #openstack-lbaas | 06:15 | |
| *** luksky has joined #openstack-lbaas | 06:43 | |
| *** ccamposr__ has joined #openstack-lbaas | 07:02 | |
| *** ccamposr has quit IRC | 07:05 | |
| *** tesseract has joined #openstack-lbaas | 07:19 | |
| *** ricolin_ has joined #openstack-lbaas | 07:40 | |
| *** ricolin has quit IRC | 07:43 | |
| *** rpittau|afk is now known as rpittau | 07:47 | |
| *** ccamposr has joined #openstack-lbaas | 07:59 | |
| *** ccamposr__ has quit IRC | 08:00 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add amphora failover API test https://review.opendev.org/633614 | 08:09 |
| *** ccamposr__ has joined #openstack-lbaas | 08:23 | |
| *** ccamposr has quit IRC | 08:25 | |
| *** gcheresh has joined #openstack-lbaas | 08:25 | |
| *** ccamposr has joined #openstack-lbaas | 08:30 | |
| *** ccamposr__ has quit IRC | 08:32 | |
| *** gcheresh has quit IRC | 08:41 | |
| *** gcheresh has joined #openstack-lbaas | 09:10 | |
| *** ricolin_ has quit IRC | 09:10 | |
| *** ricolin has joined #openstack-lbaas | 09:11 | |
| *** gcheresh has quit IRC | 09:16 | |
| *** ricolin has quit IRC | 09:57 | |
| *** gcheresh has joined #openstack-lbaas | 10:11 | |
| *** gcheresh has quit IRC | 10:19 | |
| *** yamamoto has quit IRC | 10:25 | |
| *** yamamoto has joined #openstack-lbaas | 10:57 | |
| *** yamamoto has quit IRC | 10:59 | |
| *** yamamoto has joined #openstack-lbaas | 11:00 | |
| *** gcheresh has joined #openstack-lbaas | 11:23 | |
| *** gcheresh has quit IRC | 11:58 | |
| *** sapd1_x has joined #openstack-lbaas | 12:02 | |
| *** ccamposr__ has joined #openstack-lbaas | 12:08 | |
| *** ccamposr has quit IRC | 12:11 | |
| *** goldyfruit has quit IRC | 12:22 | |
| openstackgerrit | Merged openstack/octavia master: Install DIB binary dependencies from bindep.txt https://review.opendev.org/632051 | 12:36 |
| openstackgerrit | Merged openstack/octavia stable/rocky: Fix allocate_and_associate DB deadlock https://review.opendev.org/665557 | 13:10 |
| *** goldyfruit has joined #openstack-lbaas | 13:15 | |
| *** vishalmanchanda has quit IRC | 13:28 | |
| *** yboaron_ has joined #openstack-lbaas | 13:31 | |
| *** yboaron_ has quit IRC | 13:35 | |
| emccormick | johnson FYI that debug message about shared attribute excluded by policy has no bearing on the SubnetNotFound result | 13:51 |
| emccormick | tried this in another region (running Rocky FWIW) and the log entries are the same | 13:52 |
| emccormick | 200 response + that debug message from Neutron. It's returning the subnet info to the request. I just can't find where the SubnetNotFound message gets triggered | 13:53 |
| emccormick | interestingly in the Queens region where I'm unable to add new loadbalancers, I have 3 others that are stuck PENDING_UPDATE. Maybe related to cgoncalves bug | 13:59 |
| *** Vorrtex has joined #openstack-lbaas | 14:03 | |
| emccormick | Is there any clean way to revert an LB back from PENDING_UPDATE to ACTIVE short of forcing it in the database? Status shows every element of it as online and active except the loadbalancer itself | 14:18 |
| cgoncalves | emccormick, SQL query: update loadbalancer set provisioning_status="ACTIVE" where id="YOUR_LB_ID"; | 14:30 |
| emccormick | that's what I was afraid of ;) | 14:30 |
| emccormick | Thanks | 14:30 |
| xgerman_ | you can add a feature request for the —force flag for failover or so :-) | 14:31 |
| *** yboaron_ has joined #openstack-lbaas | 14:34 | |
| cgoncalves | the --force option has been discussed on multiple occasions. the team is working supporting flow persistence. see https://storyboard.openstack.org/#!/story/2005072 | 14:35 |
| xgerman_ | Yep, saw that in the denver minutes… not sure if that’s the only way to get into *_pending though | 14:37 |
| emccormick | I think this thing has gotten itself in a bad state where amphorae do not exist and therefore are in error state | 14:38 |
| emccormick | out of sync with the servers that actually exist somehow. Fun | 14:38 |
| *** yboaron_ has quit IRC | 14:39 | |
| cgoncalves | emccormick, try marking provisioning_status of LB to ERROR and trigger loadbalancer failover | 14:39 |
| emccormick | they seem to slowly be getting there on their own. I now have 0 amphorae | 14:39 |
| xgerman_ | yeah, at one point the system will notice :-) | 14:40 |
| emccormick | it's not spawning new ones so far though... | 14:40 |
| *** gcheresh has joined #openstack-lbaas | 14:44 | |
| *** spatel has joined #openstack-lbaas | 14:48 | |
| spatel | Does lbaas mgmt network need to be routable or its just for internal heartbeat ? | 14:48 |
| *** fnaval has joined #openstack-lbaas | 14:56 | |
| *** sapd1_x has quit IRC | 14:58 | |
| emccormick | spatel It needs to allow access from your controllers to the amphorae | 15:00 |
| xgerman_ | non routable is fine | 15:00 |
| emccormick | So either routed or shared with some interface on your controllers | 15:00 |
| xgerman_ | if you manage out that all on one L2 subnet | 15:01 |
| spatel | In short my controller node need to access all compute nodes in L2 network | 15:01 |
| xgerman_ | out=put | 15:01 |
| spatel | I got your point, thank you.. | 15:01 |
| spatel | emccormick: xgerman_ | 15:01 |
| spatel | Does horizon GUI support uploading SSL certificate to Octivia or barbican ? | 15:02 |
| spatel | I haven't seen one.. so far not sure if i am missing anything here | 15:03 |
| cgoncalves | spatel, the Octavia dashboard does not, and should not IMO. it should be handled in a Barbican dashboard, which does not exist today | 15:04 |
| cgoncalves | I heard the Barbican team was planning to start one though | 15:04 |
| * redrobot pokes head in at the sound of Barbican | 15:04 | |
| spatel | cgoncalves: sounds good | 15:05 |
| cgoncalves | redrobot, can you confirm or deny the rumors? :) | 15:05 |
| emccormick | anything required to get new amphora to try to spin up for LBs in ERROR state? | 15:10 |
| emccormick | assuming there are no amphorae presently | 15:10 |
| emccormick | or just wait | 15:10 |
| emccormick | been about 5 minutes so far... | 15:10 |
| emccormick | apparently failover isn't a valid command in ERROR state | 15:11 |
| * johnsom holds the tape recorder closer to redrobot to get a quote | 15:12 | |
| johnsom | There used to be castellan_ui, but I think that died. | 15:12 |
| johnsom | emccormick You must have an up patched deployment, that bug about failover on ERROR obejcts was fixed some time ago | 15:12 |
| cgoncalves | emccormick, hmm I reckon there was a patch to fix that corner case when there are no amps associated to the LB | 15:13 |
| cgoncalves | this: https://review.opendev.org/#/c/620414/ | 15:13 |
| emccormick | was it backported to Queens? | 15:14 |
| johnsom | cgoncalves That is a different issue | 15:14 |
| johnsom | cgoncalves That patch should be abandoned. | 15:15 |
| johnsom | cgoncalves Though the issues are part of the failover refactor I have on my short list. | 15:15 |
| johnsom | emccormick It was: https://review.opendev.org/#/c/643740/ | 15:17 |
| emccormick | in the short term, If I go forcibly set them to active again in the DB, should the amphorae get created? | 15:18 |
| emccormick | I kinda have bitchy customers that won't be too happy if I can't fix it til I build a whole new image and push it out ;) | 15:19 |
| johnsom | emccormick It was released in 2.1.0 for queens. No image should be required to fix the failover issue, it's an API bug. | 15:20 |
| johnsom | You can try a force to active | 15:20 |
| johnsom | Then a failover | 15:20 |
| emccormick | fun. It completely ignored the failover request with no errors at all | 15:26 |
| emccormick | loadbalancer is still ACTIVE though *heh* | 15:26 |
| emccormick | good news is I can create new loadbalancers now that I fixed the first problem and I get amphorae | 15:28 |
| emccormick | FYI this was all caused initially by a second region getting added and none being specified in octavia.conf in the first region. It was contacting the right endpoints but for some reason not able to deal with the results that came back | 15:29 |
| *** goldyfruit has quit IRC | 15:41 | |
| *** yamamoto has quit IRC | 15:48 | |
| *** yamamoto has joined #openstack-lbaas | 15:49 | |
| *** yamamoto has quit IRC | 15:54 | |
| *** rpittau is now known as rpittau|afk | 15:57 | |
| emccormick | only thing that seems to trigger the amphora creation is deleting and recreating which is less than ideal | 16:34 |
| emccormick | not sure why it ignores failover requests | 16:34 |
| *** ataraday_ has joined #openstack-lbaas | 17:03 | |
| ataraday_ | johnsom, Hi! Can you please take a look on https://review.opendev.org/#/c/662791/ ? And if approach for pool flows is acceptable https://review.opendev.org/#/c/665381/ | 17:12 |
| johnsom | ataraday_ Ok. Might not get to it until next week. | 17:12 |
| ataraday_ | johnsom, OK, thanks! | 17:13 |
| *** gcheresh has quit IRC | 17:20 | |
| *** ccamposr has joined #openstack-lbaas | 17:22 | |
| *** ccamposr__ has quit IRC | 17:24 | |
| spatel | One more question if one of amphorae vm is dead in active-standby does Octivia rebuild it again? | 17:43 |
| *** tesseract has quit IRC | 17:50 | |
| johnsom | Yes | 18:01 |
| *** gcheresh has joined #openstack-lbaas | 18:02 | |
| *** ramishra has quit IRC | 18:08 | |
| mloza | hello, what would cause this error https://pastebin.com/raw/TVjm9AKj | 18:17 |
| johnsom | mloza 409 means conflict. In this case some other action is being executed against the member by one of the controllers. You need to retry until the object is no longer in use and the 409 clears. | 18:18 |
| johnsom | Oh, not member, load balancer. | 18:23 |
| *** goldyfruit has joined #openstack-lbaas | 18:51 | |
| *** mithilarun has joined #openstack-lbaas | 18:53 | |
| *** gcheresh has quit IRC | 18:56 | |
| *** psachin has quit IRC | 18:59 | |
| *** goldyfruit has quit IRC | 19:16 | |
| mloza | thanks | 19:20 |
| mloza | I keept getting this error 'The amphora 9c270c8a-c413-46bf-bb79-4ccf4f42d764 with IP 172.31.0.56 is missing from the DB, so it cannot be automatically deleted (the compute_id is unknown). An operator must manually delete it from the compute service.' | 19:20 |
| mloza | keep* | 19:20 |
| mloza | failover doesnt even bring it back | 19:21 |
| mloza | this is another issue | 19:21 |
| *** Vorrtex has quit IRC | 19:27 | |
| *** mithilarun has quit IRC | 19:29 | |
| *** fnaval has quit IRC | 19:29 | |
| *** mithilarun has joined #openstack-lbaas | 19:48 | |
| *** yamamoto has joined #openstack-lbaas | 19:49 | |
| *** yamamoto has quit IRC | 19:53 | |
| cgoncalves | mloza, that's a warning message, and the message hints at what to do to resolve it | 19:59 |
| *** goldyfruit has joined #openstack-lbaas | 19:59 | |
| cgoncalves | find the nova instance with IP 172.31.0.56 and delete it | 20:00 |
| mloza | thanks | 20:10 |
| *** mithilarun has quit IRC | 20:15 | |
| *** mithilarun has joined #openstack-lbaas | 20:15 | |
| *** mithilarun has quit IRC | 20:20 | |
| *** mithilarun has joined #openstack-lbaas | 20:24 | |
| *** fnaval has joined #openstack-lbaas | 20:44 | |
| *** pcaruana has quit IRC | 21:16 | |
| *** mithilarun has quit IRC | 21:25 | |
| *** mithilarun has joined #openstack-lbaas | 21:25 | |
| *** mithilarun has quit IRC | 21:30 | |
| *** openstackgerrit has quit IRC | 21:33 | |
| *** mithilarun has joined #openstack-lbaas | 21:36 | |
| *** squarebracket has joined #openstack-lbaas | 21:42 | |
| *** spatel has quit IRC | 21:49 | |
| *** ccamposr has quit IRC | 22:24 | |
| rm_work | cgoncalves: hmmm i BET we could actually improve that -- could look up that IP in neutron, check what port->compute it's hooked to, attempt an amp-agent /info call to see if it really belongs to us, and if it DOES, *then* we could delete it automatically? | 22:38 |
| rm_work | :D | 22:38 |
| *** mithilarun has quit IRC | 22:42 | |
| *** mithilarun has joined #openstack-lbaas | 22:42 | |
| *** spatel has joined #openstack-lbaas | 22:43 | |
| *** spatel has quit IRC | 22:47 | |
| *** mithilarun has quit IRC | 22:47 | |
| *** mithilarun has joined #openstack-lbaas | 23:01 | |
| *** luksky has quit IRC | 23:31 | |
| *** goldyfruit has quit IRC | 23:40 | |
| *** goldyfruit has joined #openstack-lbaas | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!