| *** hongbin has joined #openstack-lbaas | 00:58 | |
| *** ricolin has joined #openstack-lbaas | 01:03 | |
| *** yamamoto has quit IRC | 01:40 | |
| *** yamamoto has joined #openstack-lbaas | 01:40 | |
| *** altlogbot_2 has quit IRC | 03:16 | |
| *** altlogbot_0 has joined #openstack-lbaas | 03:17 | |
| *** hongbin has quit IRC | 03:27 | |
| *** yamamoto has quit IRC | 03:37 | |
| *** yamamoto has joined #openstack-lbaas | 03:45 | |
| *** yamamoto has quit IRC | 03:53 | |
| openstackgerrit | Merged openstack/octavia master: Switch TaskFlow engine to parallel https://review.opendev.org/676379 | 04:14 |
|---|---|---|
| *** gcheresh has joined #openstack-lbaas | 04:50 | |
| *** tkajinam has quit IRC | 04:53 | |
| *** tkajinam has joined #openstack-lbaas | 04:54 | |
| *** tkajinam has quit IRC | 05:00 | |
| *** tkajinam has joined #openstack-lbaas | 05:05 | |
| *** tkajinam has quit IRC | 05:31 | |
| *** gcheresh has quit IRC | 05:45 | |
| *** gcheresh has joined #openstack-lbaas | 05:57 | |
| *** tkajinam has joined #openstack-lbaas | 06:04 | |
| *** yamamoto has joined #openstack-lbaas | 06:16 | |
| *** hongbin has joined #openstack-lbaas | 06:28 | |
| *** hongbin has quit IRC | 06:32 | |
| *** yamamoto has quit IRC | 06:36 | |
| *** yamamoto has joined #openstack-lbaas | 06:37 | |
| *** yamamoto has quit IRC | 06:42 | |
| *** tkajinam has quit IRC | 06:49 | |
| *** yamamoto has joined #openstack-lbaas | 06:54 | |
| *** yamamoto has quit IRC | 06:54 | |
| *** tkajinam has joined #openstack-lbaas | 07:00 | |
| *** ricolin_ has joined #openstack-lbaas | 07:03 | |
| *** ricolin has quit IRC | 07:05 | |
| *** devfaz has quit IRC | 07:33 | |
| *** ricolin_ is now known as ricolin | 07:33 | |
| *** devfaz has joined #openstack-lbaas | 07:49 | |
| *** ivve has joined #openstack-lbaas | 07:53 | |
| *** yamamoto has joined #openstack-lbaas | 08:26 | |
| *** yamamoto has quit IRC | 08:28 | |
| *** yamamoto has joined #openstack-lbaas | 08:34 | |
| *** yamamoto has quit IRC | 08:38 | |
| *** tkajinam has quit IRC | 08:47 | |
| *** yamamoto has joined #openstack-lbaas | 08:47 | |
| *** ricolin has quit IRC | 09:07 | |
| *** sapd1_x has joined #openstack-lbaas | 09:45 | |
| *** yamamoto has quit IRC | 10:00 | |
| *** hongbin has joined #openstack-lbaas | 10:22 | |
| *** gcheresh has quit IRC | 10:36 | |
| *** gcheresh has joined #openstack-lbaas | 10:42 | |
| *** yamamoto has joined #openstack-lbaas | 10:53 | |
| openstackgerrit | Merged openstack/octavia-tempest-plugin master: Set connection_retry_interval back up to 1200 https://review.opendev.org/675745 | 10:54 |
| *** yamamoto has quit IRC | 10:59 | |
| *** yamamoto has joined #openstack-lbaas | 11:06 | |
| openstackgerrit | Merged openstack/octavia master: Set Libvirt hw_rng for devstack https://review.opendev.org/676013 | 11:07 |
| *** yamamoto has quit IRC | 11:10 | |
| *** hongbin has quit IRC | 11:16 | |
| *** yamamoto has joined #openstack-lbaas | 11:31 | |
| *** yamamoto_ has joined #openstack-lbaas | 11:41 | |
| *** yamamoto has quit IRC | 11:45 | |
| *** yamamoto_ has quit IRC | 12:20 | |
| *** luksky has joined #openstack-lbaas | 12:22 | |
| *** yamamoto has joined #openstack-lbaas | 12:28 | |
| *** ricolin has joined #openstack-lbaas | 12:37 | |
| *** hongbin has joined #openstack-lbaas | 12:43 | |
| *** yamamoto has quit IRC | 13:06 | |
| *** roukoswarf has joined #openstack-lbaas | 13:06 | |
| roukoswarf | johnsom: why does failover fail the healthy LB and not the error LB? | 13:07 |
| roukoswarf | keeps getting deadlocked with error/standalone | 13:07 |
| johnsom | roukoswarf I'm confused, it only fails over unhealthy load balancers. | 13:09 |
| roukoswarf | people non stop break LBs and yell at me about it. | 13:09 |
| johnsom | Are you manually triggering a failover via the API? | 13:09 |
| roukoswarf | yes, because the LB is immutable because theres a failed amphora, and the failed amphora is the master, and when i failover, it kills the healthy backup | 13:10 |
| roukoswarf | and now im stuck in master/error standalone/allocated | 13:10 |
| roukoswarf | failover causes a full outage and its deadlocked forever. | 13:11 |
| johnsom | Ah, you meant amphora. Well, it tries to guess at which is the backup, but since the active/standby HA is autonomous and can switch inside the amphorae at any time, the controller doesn't know which is which explicitly. The bigger question is why is your failover not completing? | 13:11 |
| roukoswarf | not sure, where can i check? | 13:12 |
| johnsom | What do you mean by deadlock? | 13:12 |
| roukoswarf | the logs are silent. | 13:12 |
| roukoswarf | as in, any number of failovers or intentionally killing a node to make it go unhealthy and failover doesnt bring anything back | 13:13 |
| johnsom | failover always logs. If it was a manually called failover via the API it is logged in the worker log. If the controller detected the failure and triggered a failover, it is in the health manager log | 13:13 |
| *** yamamoto has joined #openstack-lbaas | 13:17 | |
| roukoswarf | worker log just show connection issues to the master, which is dead (for some reason, other vms dont crash, just amphora), but nothing about the new one. | 13:17 |
| roukoswarf | oh, i found a few messages about the new one | 13:18 |
| roukoswarf | Trigger reload_allocations for port admin_state_up=True, [...] repeating every once and a while, then gives up it seems | 13:19 |
| roukoswarf | to that lb: HTTPSConnectionPool(host='172.31.1.203', port=9443): Max retries exceeded with url: /0.5/info | 13:23 |
| roukoswarf | but its alive, i can reach it, and the port is open. | 13:23 |
| roukoswarf | and after 30 minutes of downtime is in active state now... | 13:24 |
| roukoswarf | is there some kind of race condition? | 13:24 |
| roukoswarf | it just suddenly woke up, and both master and backup are healthy now | 13:25 |
| roukoswarf | i get stuff like: Nova instance with id: e15fba8f-e184-4ac8-bf84-b48d913ab947 not found. Assuming already deleted.: NotFound: Instance e15fba8f-e184-4ac8-bf84-b48d913ab947 could not be found. | 13:27 |
| roukoswarf | where it seems like octavia-workers are doing things all out of order? | 13:27 |
| *** yamamoto has quit IRC | 13:29 | |
| roukoswarf | if someone is applying a bunch of stuff at once in octavia with terraform and has no idea what they actually are running, is there cases where workers will fight eachother? | 13:30 |
| *** luksky has quit IRC | 13:33 | |
| johnsom | roukoswarf No, we have very strict locking and all of the controllers are safe to run in multiple instances. | 13:33 |
| roukoswarf | in what case would a worker try to delete an instance thats already been deleted? | 13:33 |
| roukoswarf | it failed to delete the master which was stuck in error and the instance was already deleted, again. | 13:34 |
| johnsom | Well, there are a couple of cases. 1. A nova failure. 2. A first failover failed due to a nova or neutron error and the controller retries the failover. | 13:35 |
| *** yamamoto has joined #openstack-lbaas | 13:35 | |
| roukoswarf | and for the failover, where it hung for a long period, it booted the vm, and then hit max_retries connecting to it before it was ready it seems. | 13:36 |
| roukoswarf | eventually, it suddently tried again and it got it running, after a lot of downtime | 13:36 |
| roukoswarf | nova has been stable, this case has seemingly happened to about 60% of our LBs over time, but no reported failures of instances. | 13:37 |
| johnsom | That sounds like a nova or neutron issue blocked the successful boot up of the amphora. It should be less than a minute to boot a VM (not just ACTIVE in nova, which doesn't mean it booted at all) | 13:37 |
| johnsom | In your health manager logs, are there messages that say "THIS IS NOT GOOD"? | 13:38 |
| roukoswarf | literally search for that string? | 13:38 |
| johnsom | What version of Octavia do you have installed? | 13:38 |
| *** yamamoto has quit IRC | 13:39 | |
| johnsom | Literally that string | 13:39 |
| roukoswarf | octavia==4.0.0 in the controllers | 13:40 |
| roukoswarf | octavia==4.1.0.dev53 in the amphorae | 13:41 |
| roukoswarf | not sure why we built master on the images.. | 13:42 |
| roukoswarf | what *should* i sync them to, could this be an issue? | 13:43 |
| johnsom | No, that should not matter | 13:43 |
| *** sapd1_x has quit IRC | 13:50 | |
| *** yamamoto has joined #openstack-lbaas | 13:53 | |
| *** sapd1_x has joined #openstack-lbaas | 13:56 | |
| *** sapd1_x has quit IRC | 13:57 | |
| *** sapd1_x has joined #openstack-lbaas | 14:21 | |
| *** Vorrtex has joined #openstack-lbaas | 14:23 | |
| *** yamamoto has quit IRC | 14:29 | |
| *** yamamoto has joined #openstack-lbaas | 14:30 | |
| *** spatel has joined #openstack-lbaas | 14:35 | |
| *** gcheresh has quit IRC | 15:29 | |
| *** ivve has quit IRC | 15:35 | |
| *** luksky has joined #openstack-lbaas | 15:55 | |
| *** sapd1_x has quit IRC | 15:58 | |
| *** yamamoto has quit IRC | 16:05 | |
| *** yamamoto has joined #openstack-lbaas | 16:05 | |
| *** yamamoto has quit IRC | 16:07 | |
| *** yamamoto has joined #openstack-lbaas | 16:11 | |
| *** yamamoto has quit IRC | 16:11 | |
| *** yamamoto has joined #openstack-lbaas | 16:11 | |
| *** yamamoto has quit IRC | 16:16 | |
| *** ricolin has quit IRC | 16:17 | |
| *** ramishra has joined #openstack-lbaas | 16:57 | |
| *** ramishra has quit IRC | 17:26 | |
| *** ivve has joined #openstack-lbaas | 18:10 | |
| *** yamamoto has joined #openstack-lbaas | 18:58 | |
| *** yamamoto has quit IRC | 19:03 | |
| *** vesper11 has quit IRC | 19:30 | |
| *** spatel has quit IRC | 20:27 | |
| *** spatel has joined #openstack-lbaas | 20:48 | |
| *** vesper11 has joined #openstack-lbaas | 20:52 | |
| *** spatel has quit IRC | 20:53 | |
| openstackgerrit | Colin Gibbons proposed openstack/octavia master: Standardizes terminology in Listener documentation https://review.opendev.org/676825 | 21:03 |
| *** Vorrtex has quit IRC | 21:15 | |
| *** lemko has joined #openstack-lbaas | 21:34 | |
| *** yamamoto has joined #openstack-lbaas | 21:55 | |
| *** luksky has quit IRC | 22:01 | |
| *** trident has quit IRC | 22:03 | |
| *** trident has joined #openstack-lbaas | 22:10 | |
| schaney | Hey Octavia Team, I'm looking to familiarize myself with the codebase. Do contributors generally coordinate with the core members for grabbing tasks? Or is it more free-for-all on open tasks? | 22:15 |
| johnsom | schaney Hi, welcome! | 22:16 |
| schaney | hey! | 22:16 |
| schaney | looking at https://storyboard.openstack.org/#!/story/2002907 but I don't want to step on anyone's toes | 22:16 |
| johnsom | We try to coordinate, generally by marking yourself as owner on the story in storyboard. | 22:16 |
| johnsom | Though I have to say, we aren't always good at setting that. | 22:17 |
| johnsom | So, it's always easy to ask here. | 22:17 |
| schaney | gotcha, will do =) | 22:17 |
| johnsom | That would be great, no one is working on that at the moment | 22:17 |
| schaney | awesome! thanks for the help :D | 22:17 |
| johnsom | Also, don't be shy, we are more than happy to answer questions and help out as we can | 22:17 |
| schaney | great, thank you, I will be taking you up on that | 22:18 |
| *** goldyfruit has quit IRC | 22:35 | |
| *** goldyfruit has joined #openstack-lbaas | 22:38 | |
| *** ivve has quit IRC | 22:46 | |
| *** yamamoto has quit IRC | 22:53 | |
| *** goldyfruit has quit IRC | 22:56 | |
| *** goldyfruit has joined #openstack-lbaas | 22:56 | |
| *** yamamoto has joined #openstack-lbaas | 22:57 | |
| *** roukoswarf has quit IRC | 22:59 | |
| *** tkajinam has joined #openstack-lbaas | 23:00 | |
| *** yamamoto has quit IRC | 23:02 | |
| *** spatel has joined #openstack-lbaas | 23:48 | |
| *** spatel has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!