| *** fnaval has quit IRC | 00:00 | |
| *** yamamoto has joined #openstack-lbaas | 00:06 | |
| *** sshank has joined #openstack-lbaas | 00:09 | |
| *** Swami has quit IRC | 01:00 | |
| *** sshank has quit IRC | 01:13 | |
| openstackgerrit | Hengqing Hu proposed openstack/python-octaviaclient master: Improve unit tests results with subTest https://review.openstack.org/531257 | 01:26 |
|---|---|---|
| *** jniesz has quit IRC | 01:47 | |
| *** harlowja has quit IRC | 02:31 | |
| *** threestrands_ has joined #openstack-lbaas | 02:49 | |
| *** threestrands_ has quit IRC | 02:49 | |
| *** threestrands_ has joined #openstack-lbaas | 02:49 | |
| *** threestrands has quit IRC | 02:51 | |
| *** sshank has joined #openstack-lbaas | 03:08 | |
| *** sanfern has joined #openstack-lbaas | 03:10 | |
| *** sshank has quit IRC | 03:18 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 03:44 | |
| *** jappleii__ has joined #openstack-lbaas | 03:44 | |
| *** jappleii__ has quit IRC | 03:45 | |
| *** jappleii__ has joined #openstack-lbaas | 03:45 | |
| *** jappleii__ has quit IRC | 03:46 | |
| *** threestrands_ has quit IRC | 03:46 | |
| *** jappleii__ has joined #openstack-lbaas | 03:47 | |
| *** jappleii__ has quit IRC | 03:48 | |
| *** AlexeyAbashkin has quit IRC | 03:48 | |
| *** jappleii__ has joined #openstack-lbaas | 03:48 | |
| *** rcernin has quit IRC | 03:49 | |
| *** rcernin has joined #openstack-lbaas | 03:49 | |
| *** gans has joined #openstack-lbaas | 04:02 | |
| *** gans has quit IRC | 04:12 | |
| *** gans has joined #openstack-lbaas | 04:15 | |
| *** gans has quit IRC | 04:20 | |
| *** armax has quit IRC | 04:26 | |
| *** armax has joined #openstack-lbaas | 04:27 | |
| *** links has joined #openstack-lbaas | 04:37 | |
| openstackgerrit | chenxiangui proposed openstack/neutron-lbaas-dashboard master: Modify links in README.rst https://review.openstack.org/535598 | 04:39 |
| openstackgerrit | Hengqing Hu proposed openstack/neutron-lbaas-dashboard master: Updating for new sphinx docs jobs https://review.openstack.org/535600 | 04:44 |
| openstackgerrit | Santhosh Fernandes proposed openstack/octavia master: L3 ACTIVE-ACTIVE Data model impact https://review.openstack.org/524722 | 05:03 |
| *** harlowja has joined #openstack-lbaas | 05:21 | |
| *** gcheresh has joined #openstack-lbaas | 05:27 | |
| *** gcheresh has quit IRC | 05:34 | |
| *** jappleii__ has quit IRC | 05:52 | |
| *** gcheresh has joined #openstack-lbaas | 06:02 | |
| *** armax has quit IRC | 06:03 | |
| *** gcheresh has quit IRC | 06:10 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron-lbaas-dashboard master: Imported Translations from Zanata https://review.openstack.org/535624 | 06:15 |
| *** logan- has quit IRC | 06:16 | |
| *** gcheresh has joined #openstack-lbaas | 06:25 | |
| openstackgerrit | Ganpat Agarwal proposed openstack/octavia master: ACTIVE-ACTIVE ExaBGP rest api driver https://review.openstack.org/527009 | 06:26 |
| *** harlowja has quit IRC | 06:29 | |
| *** annp has joined #openstack-lbaas | 06:33 | |
| *** sanfern has quit IRC | 06:38 | |
| *** sanfern has joined #openstack-lbaas | 06:39 | |
| *** logan- has joined #openstack-lbaas | 06:40 | |
| *** gcheresh has quit IRC | 06:43 | |
| *** pcaruana has joined #openstack-lbaas | 06:44 | |
| *** gans has joined #openstack-lbaas | 06:45 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia master: Make frontend interface attrs less vrrp specific https://review.openstack.org/521138 | 07:01 |
| *** b_bezak has joined #openstack-lbaas | 07:35 | |
| *** rcernin has quit IRC | 07:43 | |
| *** rm_work has quit IRC | 08:08 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 08:13 | |
| *** tesseract has joined #openstack-lbaas | 08:16 | |
| *** sanfern has quit IRC | 08:16 | |
| *** harlowja has joined #openstack-lbaas | 08:17 | |
| *** sanfern has joined #openstack-lbaas | 08:17 | |
| *** rm_work has joined #openstack-lbaas | 08:20 | |
| *** jmccrory has quit IRC | 08:26 | |
| *** sbalukoff_ has quit IRC | 08:27 | |
| *** sbalukoff_ has joined #openstack-lbaas | 08:27 | |
| *** annp has quit IRC | 08:28 | |
| *** annp has joined #openstack-lbaas | 08:29 | |
| *** jmccrory has joined #openstack-lbaas | 08:33 | |
| *** harlowja has quit IRC | 08:37 | |
| *** PagliaccisCloud has quit IRC | 08:38 | |
| *** PagliaccisCloud has joined #openstack-lbaas | 08:45 | |
| *** jmccrory has quit IRC | 08:50 | |
| *** gcheresh has joined #openstack-lbaas | 08:52 | |
| *** jmccrory has joined #openstack-lbaas | 08:52 | |
| *** rm_work has quit IRC | 08:56 | |
| *** gcheresh has quit IRC | 09:08 | |
| *** rm_work has joined #openstack-lbaas | 09:36 | |
| *** pcaruana has quit IRC | 09:50 | |
| *** pcaruana has joined #openstack-lbaas | 09:53 | |
| *** pcaruana has quit IRC | 09:53 | |
| *** pcaruana has joined #openstack-lbaas | 09:54 | |
| *** salmankhan has joined #openstack-lbaas | 10:21 | |
| *** gans has quit IRC | 10:39 | |
| *** gans has joined #openstack-lbaas | 10:39 | |
| *** salmankhan has quit IRC | 10:43 | |
| *** salmankhan has joined #openstack-lbaas | 10:47 | |
| *** salmankhan has quit IRC | 11:43 | |
| *** salmankhan has joined #openstack-lbaas | 11:50 | |
| *** gcheresh has joined #openstack-lbaas | 11:50 | |
| *** Alex_Staf has quit IRC | 11:52 | |
| *** Alex_Staf has joined #openstack-lbaas | 11:53 | |
| *** annp has quit IRC | 11:55 | |
| *** gcheresh has quit IRC | 12:04 | |
| *** salmankhan has quit IRC | 12:07 | |
| *** salmankhan has joined #openstack-lbaas | 12:07 | |
| *** hogepodge has quit IRC | 12:26 | |
| *** PagliaccisCloud has quit IRC | 12:27 | |
| *** strigazi has quit IRC | 12:27 | |
| *** irenab has quit IRC | 12:27 | |
| *** reedip has quit IRC | 12:28 | |
| *** hogepodge has joined #openstack-lbaas | 12:28 | |
| *** strigazi has joined #openstack-lbaas | 12:29 | |
| *** PagliaccisCloud has joined #openstack-lbaas | 12:30 | |
| *** irenab has joined #openstack-lbaas | 12:30 | |
| *** reedip has joined #openstack-lbaas | 12:42 | |
| *** gans has quit IRC | 12:53 | |
| *** gcheresh has joined #openstack-lbaas | 13:05 | |
| *** links has quit IRC | 13:28 | |
| *** gcheresh has quit IRC | 13:30 | |
| *** aojea_ has joined #openstack-lbaas | 13:50 | |
| *** fnaval has joined #openstack-lbaas | 13:51 | |
| *** fnaval has quit IRC | 13:51 | |
| *** fnaval has joined #openstack-lbaas | 13:52 | |
| *** aojea__ has joined #openstack-lbaas | 13:55 | |
| *** aojea_ has quit IRC | 13:58 | |
| *** aojea_ has joined #openstack-lbaas | 14:01 | |
| *** atoth has joined #openstack-lbaas | 14:02 | |
| *** aojea__ has quit IRC | 14:04 | |
| *** beagles_afk is now known as beagles | 14:04 | |
| *** aojea__ has joined #openstack-lbaas | 14:05 | |
| *** aojea_ has quit IRC | 14:08 | |
| *** salmankhan has quit IRC | 14:09 | |
| *** aojea_ has joined #openstack-lbaas | 14:11 | |
| *** aojea__ has quit IRC | 14:13 | |
| *** fnaval has quit IRC | 14:18 | |
| *** aojea__ has joined #openstack-lbaas | 14:20 | |
| *** aojea_ has quit IRC | 14:24 | |
| *** salmankhan has joined #openstack-lbaas | 14:28 | |
| *** aojea_ has joined #openstack-lbaas | 14:28 | |
| *** aojea__ has quit IRC | 14:29 | |
| *** aojea__ has joined #openstack-lbaas | 14:30 | |
| *** aojea_ has quit IRC | 14:34 | |
| *** dayou has quit IRC | 14:34 | |
| *** aojea_ has joined #openstack-lbaas | 14:36 | |
| *** aojea__ has quit IRC | 14:39 | |
| *** aojea_ has quit IRC | 14:44 | |
| *** dayou has joined #openstack-lbaas | 14:46 | |
| *** armax has joined #openstack-lbaas | 15:01 | |
| *** yamamoto has quit IRC | 15:05 | |
| *** yamamoto has joined #openstack-lbaas | 15:06 | |
| *** Alex_Staf has quit IRC | 15:11 | |
| *** yamamoto has quit IRC | 15:17 | |
| *** ndanl has quit IRC | 15:29 | |
| *** yamamoto has joined #openstack-lbaas | 15:48 | |
| *** fnaval has joined #openstack-lbaas | 15:54 | |
| *** ivve has quit IRC | 16:10 | |
| *** gcheresh has joined #openstack-lbaas | 16:15 | |
| *** b_bezak has quit IRC | 16:20 | |
| *** b_bezak has joined #openstack-lbaas | 16:21 | |
| *** gcheresh has quit IRC | 16:23 | |
| *** ivve has joined #openstack-lbaas | 16:25 | |
| *** b_bezak has quit IRC | 16:26 | |
| *** AlexeyAbashkin has quit IRC | 16:35 | |
| *** sanfern has quit IRC | 16:35 | |
| *** jniesz has joined #openstack-lbaas | 16:41 | |
| *** ivve has quit IRC | 16:56 | |
| *** ivve has joined #openstack-lbaas | 17:10 | |
| *** yamamoto has quit IRC | 17:31 | |
| *** ivve has quit IRC | 17:35 | |
| *** pcaruana has quit IRC | 17:36 | |
| *** yamamoto has joined #openstack-lbaas | 17:38 | |
| *** yamamoto has quit IRC | 17:38 | |
| *** ivve has joined #openstack-lbaas | 17:52 | |
| *** sanfern has joined #openstack-lbaas | 18:12 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 18:16 | |
| sanfern | hi johnsom , | 18:16 |
| johnsom | sanfern Hi | 18:17 |
| sanfern | we have multiple peer_ip in config file, now each distributor creation will create one record in peer tables | 18:18 |
| sanfern | jason was mentioning we need to make peer entry unique based on ip_address, local_as, | 18:19 |
| *** AlexeyAbashkin has quit IRC | 18:20 | |
| sanfern | we should not create peer table from get_create_distributor_flow, instead we need to make it one time entry into table using CLI | 18:20 |
| sanfern | peer table entry | 18:21 |
| johnsom | How do you know which peer to map a lb_amp to? | 18:21 |
| sanfern | all peers should map all amps. Peer list is static one time populated in the setup | 18:23 |
| sanfern | otherwise each Distribution creation will have duplicate entries of peers | 18:23 |
| johnsom | Ok, so here is my thought. Really, this is going to come in from flavors when those are ready. So, for now why not keep them in the config as a list of peers, populate the table on create (like it would with flavors) and make sure it handles the "already exists" case. | 18:27 |
| johnsom | As it would with flavors | 18:28 |
| johnsom | I thought the plan for the peers was to have one address on all of the ToR that simplifies this.... | 18:28 |
| *** rstarmer has quit IRC | 18:29 | |
| sanfern | I agree, then I will make ip+remote_as as primary constrain to avoid duplicate entries. | 18:32 |
| johnsom | +1 | 18:35 |
| sanfern | on ExaBGPAPI driver should I derive from DistributorAmpDriverMixin and only implement update exabgp config | 18:35 |
| johnsom | sanfern Yes, I think that makes sense | 18:36 |
| sanfern | ok | 18:37 |
| *** yamamoto has joined #openstack-lbaas | 18:39 | |
| sanfern | in ExaBGP agent will have start/stop/register_amp/unregister_amp/status API's | 18:39 |
| *** rstarmer has joined #openstack-lbaas | 18:39 | |
| johnsom | Right, but it will use the existing methods for those | 18:39 |
| sanfern | reg_amp will announce and unreg will withdraw routes. | 18:40 |
| johnsom | Start/stop may need to be be added to the generic distributor interface. I think I looked and foudn the existing are targeted to listener only | 18:40 |
| johnsom | Yep, in the agent code it will use the common method (all distributors do this), but the code will branch on the distributor type (L3) to execute your commands vs. mine. I assume you will have an L3 module. | 18:41 |
| *** atoth has quit IRC | 18:41 | |
| sanfern | Plugging VIP's to dummy0.cfg will be in common I need to remove from reg_amp and similarly unreg_amp | 18:42 |
| sanfern | yes my L3 module is exabgp rest api driver | 18:42 |
| johnsom | driver? I thought we were talking about the agent, not the driver side | 18:44 |
| sanfern | start/stop requires L3 module requires to exabgp | 18:44 |
| sanfern | yes Agent | 18:44 |
| johnsom | Yeah, start/stop interface should be generic for distributors, not exabgp. But in the code it will branch based on the type. | 18:45 |
| sanfern | ok | 18:45 |
| johnsom | I want to minimize the exabgp custom stuff if multiple drivers are going to need the same thing. | 18:46 |
| sanfern | +1 | 18:46 |
| sanfern | how to pass/return list of objects to the flow ? | 18:47 |
| johnsom | Same as we pass any other object | 18:48 |
| johnsom | So it's something you need to store in the flow before it runs? | 18:48 |
| johnsom | I.e. not generated inside the flow? | 18:48 |
| sanfern | yes | 18:48 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/controller/worker/controller_worker.py#L128 | 18:49 |
| *** yamamoto has quit IRC | 18:50 | |
| sanfern | ok | 18:52 |
| *** rstarmer has quit IRC | 19:21 | |
| *** harlowja has joined #openstack-lbaas | 19:31 | |
| *** rstarmer has joined #openstack-lbaas | 19:31 | |
| *** rstarmer has quit IRC | 19:32 | |
| *** sanfern has quit IRC | 19:39 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 19:45 | |
| *** rstarmer has joined #openstack-lbaas | 19:47 | |
| *** ivve has quit IRC | 19:49 | |
| *** AlexeyAbashkin has quit IRC | 19:49 | |
| *** ivve has joined #openstack-lbaas | 20:01 | |
| *** slaweq has joined #openstack-lbaas | 20:05 | |
| ivve | johnsom: got that lbaas v2 working finally :) | 20:23 |
| johnsom | ivve Glad to hear it | 20:23 |
| ivve | the patch submitted will fix it, i had some other stuff that was my fault | 20:24 |
| ivve | im not sure, but it might be heat related | 20:24 |
| ivve | it was the way of assigning flip to the vip | 20:25 |
| ivve | i was creating a port first but now i tried without and assigned a manual flip, then it was alright | 20:25 |
| ivve | i tired using the loadbalancer attribute vip_port_id to assign a flip to it via heat, but it fails | 20:26 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 20:27 | |
| johnsom | Hmm, interesting | 20:27 |
| ivve | indeed | 20:27 |
| ivve | im trying like this now https://hastebin.com/yacazaniro.css | 20:27 |
| ivve | but it fails | 20:28 |
| johnsom | The VIP port idea is the ID that would be used for the flip | 20:28 |
| ivve | yeah thats what i thought | 20:28 |
| ivve | lemme get the message | 20:28 |
| *** AlexeyAbashkin has quit IRC | 20:31 | |
| ivve | Create Failed | 20:31 |
| ivve | Resource Create Failed: Notfound: Resources.Vip To Float: External Network 40d90e43-F85b-4d4f-A1b6-194be8f12f1a Is Not Reachable From Subnet Df807edf-9f68-4623-9e73-8d9630da4049. Therefore, Cannot Associate Port 7c648f87-69b2-4ec1-B349-85916e033935 With A Floating Ip. Neutron Server Returns Request Ids: | 20:31 |
| ivve | this is interesting because manually it just assigns without issue | 20:32 |
| ivve | to that very network | 20:32 |
| johnsom | That is a very confusing error message.... | 20:33 |
| ivve | that portid is the vip_port_id | 20:34 |
| ivve | the network is a created network in the heat template | 20:34 |
| ivve | and the external network is reachable by the created router in the heat template | 20:34 |
| ivve | so the stack continues | 20:35 |
| ivve | but when its done, i manually press the assign button to get a flip to that network.. it works | 20:35 |
| johnsom | Yeah, sounds like heat may have a bug | 20:36 |
| ivve | ill ask in the heat channel but first make a few searches | 20:37 |
| ivve | btw, regarding octavia. the instances that are created in order for the entire service to be functional, where are they created? in a tenant? | 20:38 |
| openstackgerrit | Ihar Hrachyshka proposed openstack/octavia master: DNM testing whether lib/neutron switch breaks this repo https://review.openstack.org/535927 | 20:43 |
| *** salmankhan has quit IRC | 20:47 | |
| ivve | ooookay... so i updated it with the net hardcoded and that worked | 20:48 |
| ivve | lets try a fresh one | 20:49 |
| johnsom | ivve Yes, they are created in the tenant defined in the octavia.conf file | 20:49 |
| ivve | cool, so its possible to put say, in the admin tenant? or is there some kind of best practice? i will be using OSA | 20:50 |
| johnsom | ivve There is an OSA role for octavia. | 20:50 |
| johnsom | Devstack uses admin, but it is usually best for production to use a tenant. However that is more work as you need to set quotas and RBAC | 20:50 |
| openstackgerrit | Ihar Hrachyshka proposed openstack/neutron-lbaas master: DNM testing whether lib/neutron switch breaks this repo https://review.openstack.org/535941 | 20:51 |
| ivve | ah | 20:51 |
| ivve | now im guessing here, but could it be that the vip_port_id is created before the actual router connecting to the proper net for a flip? | 20:52 |
| ivve | depends_on: { get_resource: the router } | 20:53 |
| ivve | or what | 20:53 |
| johnsom | I don't know anything about how the heat stuff is setup | 20:53 |
| johnsom | I don't think you can create a port without a valid net, that just seems wrong. But that doesn't mean heat doesn't have some variable/storage issues | 20:54 |
| ivve | ill try depend on so that the vip waits until the router is completed | 20:54 |
| ivve | but one might think that it should be "smart" enough to figure it out | 20:55 |
| ivve | since order is not important | 20:55 |
| ivve | well at least not for me coding the yaml | 20:55 |
| *** aojea has joined #openstack-lbaas | 20:56 | |
| ivve | nah didn't help same error | 20:59 |
| ivve | oh well enough for today | 21:00 |
| ivve | johnsom: thanks for listening :) | 21:01 |
| johnsom | NP | 21:01 |
| ivve | https://bugs.launchpad.net/heat/+bug/1626619 | 21:02 |
| openstack | Launchpad bug 1626619 in OpenStack Heat newton "OS::Neutron::FloatingIP is missing hidden dependencies" [Undecided,Fix committed] - Assigned to Rabi Mishra (rabi) | 21:02 |
| ivve | but this is pike | 21:02 |
| *** aojea_ has joined #openstack-lbaas | 21:02 | |
| *** aojea has quit IRC | 21:05 | |
| *** aojea has joined #openstack-lbaas | 21:07 | |
| *** aojea_ has quit IRC | 21:10 | |
| *** rstarmer has quit IRC | 21:11 | |
| *** aojea_ has joined #openstack-lbaas | 21:12 | |
| *** aojea has quit IRC | 21:14 | |
| *** aojea__ has joined #openstack-lbaas | 21:17 | |
| *** aojea_ has quit IRC | 21:20 | |
| *** aojea_ has joined #openstack-lbaas | 21:22 | |
| *** aojea__ has quit IRC | 21:25 | |
| *** aojea__ has joined #openstack-lbaas | 21:27 | |
| *** rstarmer has joined #openstack-lbaas | 21:29 | |
| *** aojea_ has quit IRC | 21:30 | |
| *** aojea has joined #openstack-lbaas | 21:32 | |
| *** aojea__ has quit IRC | 21:35 | |
| *** aojea_ has joined #openstack-lbaas | 21:36 | |
| *** aojea has quit IRC | 21:39 | |
| *** aojea__ has joined #openstack-lbaas | 21:42 | |
| *** aojea_ has quit IRC | 21:45 | |
| *** aojea__ has quit IRC | 21:51 | |
| *** aojea has joined #openstack-lbaas | 21:52 | |
| *** aojea_ has joined #openstack-lbaas | 21:57 | |
| *** aojea has quit IRC | 22:01 | |
| *** tesseract has quit IRC | 22:01 | |
| *** aojea has joined #openstack-lbaas | 22:03 | |
| *** aojea_ has quit IRC | 22:06 | |
| *** aojea_ has joined #openstack-lbaas | 22:07 | |
| *** aojea has quit IRC | 22:10 | |
| *** aojea has joined #openstack-lbaas | 22:13 | |
| *** aojea_ has quit IRC | 22:15 | |
| *** rstarmer has quit IRC | 22:18 | |
| *** aojea_ has joined #openstack-lbaas | 22:18 | |
| *** aojea has quit IRC | 22:21 | |
| *** aojea has joined #openstack-lbaas | 22:23 | |
| *** aojea has quit IRC | 22:23 | |
| *** aojea_ has quit IRC | 22:26 | |
| *** salmankhan has joined #openstack-lbaas | 22:33 | |
| *** salmankhan has quit IRC | 22:38 | |
| rm_work | johnsom: this is ... interesting | 22:44 |
| rm_work | so out of about 50 pools | 22:44 |
| johnsom | Uh-Oh | 22:44 |
| rm_work | 20 or so are flapping between either OFFLINE/ONLINE or OFFLINE/ERROR | 22:44 |
| rm_work | basically every health check | 22:44 |
| rm_work | i'm trying to understand if the members are really broken or not | 22:44 |
| johnsom | op status? or prov? | 22:44 |
| rm_work | OP | 22:44 |
| rm_work | via HMs | 22:44 |
| rm_work | but regardless, i feel like something must be wonky | 22:45 |
| rm_work | i picked one to look at closer, which is flapping between ERROR and OFFLINE for the pool | 22:45 |
| johnsom | Yeah, would be interesting to see how the member status looks | 22:46 |
| rm_work | member status seems to be consistently ERROR | 22:47 |
| rm_work | 3 members in this pool, all ERROR | 22:47 |
| rm_work | but every other health run, it flaps between ERROR and OFFLINE for the pool | 22:47 |
| rm_work | member status hasn't changed | 22:47 |
| rm_work | just pool op_status | 22:47 |
| johnsom | Hmmmm, with all members in ERROR, I would expect pool to be in ERROR | 22:47 |
| rm_work | scanning through the code now to see if anything jumps out | 22:48 |
| rm_work | yes, me too | 22:48 |
| rm_work | so what is shoving it back to OFFLINE | 22:48 |
| johnsom | Yeah, to my memory, OFFLINE is only when there are no members defined or it's admin down | 22:48 |
| rm_work | yeah so confirmed, LB and MEMBER statuses never leave ERROR | 22:50 |
| rm_work | but pool flaps between ERROR and OFFLINE | 22:50 |
| rm_work | I bet we are doing something dumb | 22:50 |
| rm_work | scanning update_health() | 22:51 |
| rm_work | AH YEAH | 22:52 |
| rm_work | I bet this is bad | 22:52 |
| rm_work | one sec, code links incoming | 22:52 |
| rm_work | https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/update_db.py#L225-L228 | 22:53 |
| rm_work | so this seems the obvious candidate | 22:53 |
| rm_work | so... if a pool is in ERROR... is there a change it's not in that lsit | 22:53 |
| rm_work | *list | 22:53 |
| rm_work | *chance | 22:53 |
| rm_work | https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/update_db.py#L194 | 22:54 |
| rm_work | pools = listener['pools'] | 22:54 |
| rm_work | is that list filtered? could it be? | 22:54 |
| johnsom | It should be the actual list of backends from the haproxy stats. | 22:54 |
| rm_work | yeah so uhh | 22:54 |
| rm_work | if it's in error | 22:54 |
| rm_work | could HAProxy maybe not list it | 22:55 |
| johnsom | I'm wondering if we have a case where a pool on a listener is not bound to an lb??? | 22:55 |
| rm_work | it is | 22:55 |
| rm_work | one sec, i can pm you the DB entries for this stuff if you want | 22:55 |
| rm_work | but | 22:55 |
| rm_work | i don't think it's that | 22:55 |
| rm_work | looking at how listener there is filled... | 22:55 |
| johnsom | A status tree output would be handy | 22:55 |
| rm_work | http://paste.openstack.org/show/647546/ | 22:57 |
| rm_work | i mean, that's what i'm looking at | 22:57 |
| rm_work | and it flaps | 22:57 |
| rm_work | so the next run that'll be 9 ERRORs | 22:57 |
| rm_work | so yeah | 22:58 |
| rm_work | this seems like a pretty obvious possibility: | 22:58 |
| rm_work | our health message doesn't include the pool on the listener if the pool is in error | 22:59 |
| rm_work | (from haproxy) | 22:59 |
| rm_work | we then pull the listeners out of the health message | 22:59 |
| rm_work | and then pass them to process_pools | 22:59 |
| rm_work | https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/update_db.py#L219-L226 | 22:59 |
| rm_work | we hit that `if` on 225 | 23:00 |
| rm_work | and it's sunk | 23:00 |
| rm_work | i'm 99% sure that's it | 23:00 |
| rm_work | need to verify what our health message is looking like | 23:00 |
| rm_work | though ... then wouldn't it always hit that | 23:01 |
| johnsom | I doubt it. Doesn't make sense why it would flip flop. The only reason there wouldn't be a pool record is if the "backend" isn't defined in haproxy, which is how we do "admin down" or pending_create prov | 23:01 |
| rm_work | hmm | 23:01 |
| rm_work | i need to see the health message | 23:01 |
| johnsom | Bummer, no debug log for that (though it would be a lot of log traffic) | 23:04 |
| rm_work | yeah | 23:04 |
| rm_work | i can get what HAProxy is reporting, on the amp, right? | 23:04 |
| rm_work | have to remember how to do that | 23:04 |
| johnsom | yes | 23:04 |
| rm_work | no `nc`, no `socat` ... | 23:05 |
| johnsom | Yeah, you have to scp it over | 23:06 |
| rm_work | lol | 23:06 |
| rm_work | k got it | 23:09 |
| johnsom | echo "show stat" | socat unix-connect:4f30c4d2-1d05-4a75-885e-5ab790034235.sock stdio | 23:09 |
| rm_work | yeah i got it | 23:09 |
| johnsom | We filter that down in the code, but might as well get the whole blob | 23:10 |
| rm_work | PM'd | 23:10 |
| rm_work | ehh i guess it doesn't have IPs ... | 23:10 |
| rm_work | https://gist.github.com/rm-you/85790ff5b163cd977fb24fa14be2d4d9 | 23:10 |
| johnsom | Yeah, so the "BACKEND" record is there | 23:10 |
| johnsom | It does show "DOWN" | 23:11 |
| johnsom | So, it should be sending that pool record. Can you confirm with the status tree that there is only one pool on this LB? | 23:11 |
| rm_work | there's two | 23:12 |
| rm_work | the other pool isn't flapping | 23:12 |
| rm_work | ah hold up | 23:12 |
| rm_work | maybe you are only seeing the other pool lol | 23:12 |
| rm_work | need to compare IDs | 23:12 |
| rm_work | err | 23:12 |
| rm_work | well only one pool on this LISTENER | 23:12 |
| rm_work | two listeners | 23:12 |
| rm_work | so i guess this is right | 23:12 |
| johnsom | Ok, so two processes, two sockets | 23:13 |
| rm_work | yeah, i queried the socket for the listener the ERROR pool was on | 23:13 |
| johnsom | Maybe that is where the bug is | 23:13 |
| rm_work | OH | 23:15 |
| rm_work | ROFL | 23:15 |
| rm_work | yeah i think i see it | 23:15 |
| rm_work | multiple listeners | 23:15 |
| rm_work | the for loops aren't nested, right? | 23:15 |
| rm_work | so ... "listener" | 23:15 |
| rm_work | are processed once | 23:15 |
| rm_work | and then again? | 23:15 |
| rm_work | kinda | 23:15 |
| rm_work | ehh but you have tracking for that... | 23:16 |
| rm_work | ok i'm gonna shut up and finish tracing this instead of typing stream of consciousness :P | 23:16 |
| johnsom | Ok. | 23:18 |
| rm_work | YEP got it | 23:18 |
| rm_work | you define processed_pools above | 23:18 |
| johnsom | Worst case you tcpdump the health packets and decrypt by hand to see what's in it | 23:18 |
| rm_work | but instead of ADDING the pools that each listener processes to it | 23:18 |
| rm_work | it's overwritten | 23:18 |
| rm_work | so the second listener processes its pools and overwrites the list | 23:18 |
| rm_work | then it'll process the next pool as a "LB pool" | 23:18 |
| rm_work | https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/update_db.py#L148-L198 | 23:19 |
| rm_work | first and last line | 23:19 |
| johnsom | Yeah, but that should be correct I think. It's per listener, so each listener should overwrite it | 23:20 |
| rm_work | err no | 23:20 |
| rm_work | processed_pools is USED below | 23:20 |
| rm_work | outside the listener for-loop | 23:20 |
| johnsom | Right | 23:20 |
| johnsom | As it should | 23:20 |
| rm_work | right ... | 23:20 |
| rm_work | OUTSIDE the for-loop where you set it | 23:21 |
| rm_work | no? | 23:21 |
| rm_work | you say "# Don't re-process pools shared with listeners" | 23:21 |
| rm_work | which implies that list should be a constant append | 23:21 |
| johnsom | Yeah, I think I see. | 23:22 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/update_db.py#L198 | 23:22 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 23:22 | |
| johnsom | This should append | 23:22 |
| johnsom | but how do we have multple pools per listener? | 23:23 |
| johnsom | Can you PM me the openstack loadbalancer status? | 23:23 |
| rm_work | see this | 23:23 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix processing pool statuses for LBs with multiple listeners https://review.openstack.org/535992 | 23:24 |
| rm_work | that is what I believe should happen | 23:24 |
| rm_work | the processed_pools list isn't additive right now | 23:24 |
| johnsom | I don't think 209 needs to be there. pools on LB should be a unique constraint | 23:25 |
| rm_work | well yeah | 23:25 |
| rm_work | technically it doesn't matter right now | 23:25 |
| rm_work | BUT | 23:25 |
| rm_work | that list is supposed to be "every pool we process" | 23:25 |
| rm_work | i'd like it to be up-to-date in case someone comes along later and tries to use it | 23:25 |
| rm_work | it's supposed to be an ever-growing list of pools that we have updated already | 23:26 |
| rm_work | so every time we process one, we need to make sure it gets into that list | 23:26 |
| rm_work | right? | 23:26 |
| *** AlexeyAbashkin has quit IRC | 23:26 | |
| rm_work | technically we don't use it after 209, but that's no reason to be lame about keeping it accurate, lol | 23:27 |
| *** slaweq has quit IRC | 23:27 | |
| rm_work | we DO need to have the change on 207 though because otherwise it'll overwrite every loop lol | 23:27 |
| johnsom | Just extra mem and CPU time | 23:27 |
| rm_work | barely | 23:28 |
| rm_work | i mean, the other option is to make it a passthrough | 23:28 |
| rm_work | i can do that | 23:28 |
| johnsom | Yeah, that is bad looking at it. It should just throw those away on 207 | 23:28 |
| rm_work | one sec | 23:28 |
| rm_work | yes | 23:28 |
| rm_work | probably | 23:28 |
| rm_work | i'll make a pass-2 | 23:28 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix processing pool statuses for LBs with multiple listeners https://review.openstack.org/535992 | 23:30 |
| rm_work | better? | 23:30 |
| rm_work | need to fix tests probably (hopefully?) | 23:30 |
| rm_work | oh one more thing, crap | 23:32 |
| rm_work | how do we deal with "pools" that's passed in | 23:33 |
| rm_work | that's coming from the listener stuff | 23:33 |
| rm_work | so it literally can't be used here... | 23:34 |
| rm_work | i think this is the fix for that: | 23:34 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix processing pool statuses for LBs with multiple listeners https://review.openstack.org/535992 | 23:34 |
| rm_work | johnsom: ^^ | 23:35 |
| johnsom | rm_work Ummm, if you pass in [] where is it getting the pools list from the amp? | 23:35 |
| rm_work | that's the pool list from the LISTENER | 23:36 |
| rm_work | and the only way that's used is for status setting | 23:36 |
| rm_work | which... if the pool isn't on a Listener, SHOULD always be OFFLINE | 23:36 |
| johnsom | No, it could be used in a L7 policy | 23:36 |
| rm_work | ok, but then it'd be on the listener still, no? | 23:37 |
| rm_work | processed as part of whatever listener it's used on? | 23:37 |
| rm_work | because the status for the pools here come from HAProxy | 23:37 |
| rm_work | and we only get statuses for backends when they're on listeners... | 23:37 |
| rm_work | so it's impossible for the code that "gets the status" to run, if the pool wasn't on a listener | 23:37 |
| johnsom | Yeah, I'm trying to remember if that is true. Will haproxy not let you have an un-bound "backends" section | 23:38 |
| johnsom | Or do we | 23:39 |
| rm_work | hmmm | 23:39 |
| rm_work | if we do, we still were pulling it from the wrong place before | 23:39 |
| johnsom | yeah, I follow that | 23:39 |
| rm_work | because the thing that WAS being passed in there ... was from the listener loop | 23:39 |
| *** yamamoto has joined #openstack-lbaas | 23:40 | |
| rm_work | also, shouldn't we be marking it as "processed" so long as this function was run on it? even if we just set the status to OFFLINE | 23:40 |
| rm_work | eh maybe not | 23:40 |
| rm_work | but yeah, even if we need to pass pools, what was there before was *not it* | 23:40 |
| rm_work | maybe health['pools'] ? | 23:40 |
| rm_work | how do we construct this message again... | 23:41 |
| -openstackstatus- NOTICE: Zuul will be offline over the next 20 minutes to perform maintenance; active changes will be reenqueued once work completes, but new patch sets or approvals during that timeframe may need to be rechecked or reapplied as appropriate | 23:42 | |
| rm_work | health_daemon.py | 23:42 |
| rm_work | and no, there's only "listeners" in the status message | 23:43 |
| rm_work | i think passing [] is correct | 23:43 |
| johnsom | Ok | 23:44 |
| rm_work | i'm going to apply this patch locally to my test environment | 23:44 |
| rm_work | and see if i can repro / fix | 23:45 |
| rm_work | yep repro is super easy | 23:52 |
| *** fnaval has quit IRC | 23:52 | |
| rm_work | made LB, made two listeners, one pool each, same two members on each pool, one HM looking for a bad status | 23:53 |
| rm_work | one pool is fine, one flaps between ERROR and OFFLINE | 23:53 |
| rm_work | other stays in ONLINE | 23:53 |
| rm_work | now to apply the patch... | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!