| johnsom | Well, I wasn't just sitting watching zuul... grin | 00:07 |
|---|---|---|
| johnsom | Though slightly watching the paint peal while running tempest tests. | 00:08 |
| *** altlogbot_2 has quit IRC | 00:10 | |
| johnsom | Blah, end of day. s/peal/peel/g | 00:10 |
| *** altlogbot_3 has joined #openstack-lbaas | 00:11 | |
| *** rcernin has quit IRC | 00:21 | |
| *** rcernin has joined #openstack-lbaas | 00:21 | |
| *** goldyfruit has quit IRC | 01:06 | |
| *** rcernin has quit IRC | 01:22 | |
| *** rcernin has joined #openstack-lbaas | 01:22 | |
| *** happyhemant has quit IRC | 01:28 | |
| *** goldyfruit has joined #openstack-lbaas | 01:31 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239 | 02:13 |
| *** gthiemon1e has quit IRC | 02:24 | |
| *** gthiemonge has joined #openstack-lbaas | 02:25 | |
| *** ricolin has joined #openstack-lbaas | 02:44 | |
| *** goldyfruit has quit IRC | 02:54 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239 | 02:55 |
| *** ramishra has joined #openstack-lbaas | 03:38 | |
| *** psachin has joined #openstack-lbaas | 03:39 | |
| *** AlexStaf has joined #openstack-lbaas | 04:06 | |
| *** ivve has quit IRC | 04:26 | |
| *** rm_work has quit IRC | 05:31 | |
| *** rm_work has joined #openstack-lbaas | 05:34 | |
| *** ivve has joined #openstack-lbaas | 05:39 | |
| *** ivve has quit IRC | 05:54 | |
| *** gcheresh_ has joined #openstack-lbaas | 06:36 | |
| *** pcaruana has joined #openstack-lbaas | 07:17 | |
| *** tesseract has joined #openstack-lbaas | 07:18 | |
| *** rpittau|afk is now known as rpittau | 07:19 | |
| *** luksky has joined #openstack-lbaas | 07:29 | |
| *** lemko has joined #openstack-lbaas | 07:38 | |
| *** nmagnezi has joined #openstack-lbaas | 07:40 | |
| *** luksky has quit IRC | 07:42 | |
| *** luksky has joined #openstack-lbaas | 07:57 | |
| *** yboaron_ has joined #openstack-lbaas | 08:00 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [Jobboard] Importable flow functions https://review.opendev.org/659538 | 08:09 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [Jobboard] Importable flow functions https://review.opendev.org/659538 | 08:35 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Jobboard based controller https://review.opendev.org/647406 | 08:36 |
| *** ataraday_ has joined #openstack-lbaas | 08:39 | |
| *** ccamposr has joined #openstack-lbaas | 08:47 | |
| *** luksky has quit IRC | 08:48 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Jobboard based controller https://review.opendev.org/647406 | 08:49 |
| *** sapd1_x has joined #openstack-lbaas | 08:50 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Transition member flows to use dicts https://review.opendev.org/657842 | 09:09 |
| *** rcernin has quit IRC | 09:10 | |
| *** luksky has joined #openstack-lbaas | 09:22 | |
| kklimonda | hmm, in stein horizon is trying to access flavorprofiles as part of the LB creation and I'm getting 403 - that seems reasonable based on the policy and code, but I don't quite get how to make it work, is that a bug? | 09:28 |
| *** ricolin has quit IRC | 09:40 | |
| rm_work | kklimonda: is horizon newer than your octavia install? | 10:01 |
| rm_work | if octavia doesn't have the flavorprofiles API yet, it might break | 10:01 |
| rm_work | it really should do version discovery though... but maybe that wasn't handled properly | 10:02 |
| rm_work | that or possibly the flavorprofiles stuff is behind a policy that your users don't have... i didn't actually review the policy side of that so i'm not sure. you would be better off asking johnsom when he's up in a few hours | 10:03 |
| rm_work | s/better/best/ | 10:03 |
| kklimonda | hmmm, both octavia and horizon are from stein (although not final releases, but RCs for now) - I can see that horizon does query to flavorprofiles and the query is denied by policy for the user. I'll come back later when johnsom is up. Thanks. | 10:12 |
| kklimonda | actually scratch that, I see that even though kolla's config file says 4.0.0.0rc1, the image comes with final releases (4.0.0 and 15.0.0 respectively) | 10:13 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Jobboard based controller https://review.opendev.org/647406 | 10:25 |
| *** yboaron_ has quit IRC | 10:28 | |
| *** yboaron_ has joined #openstack-lbaas | 10:29 | |
| *** luksky has quit IRC | 10:47 | |
| *** luksky has joined #openstack-lbaas | 11:18 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239 | 11:37 |
| *** henriqueof has joined #openstack-lbaas | 11:43 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239 | 11:47 |
| *** boden has joined #openstack-lbaas | 11:58 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Transition member flows to use dicts https://review.opendev.org/657842 | 12:24 |
| *** psachin has quit IRC | 12:27 | |
| *** ricolin has joined #openstack-lbaas | 12:55 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239 | 13:02 |
| *** goldyfruit has joined #openstack-lbaas | 13:07 | |
| rm_work | lol our amps have postfix | 13:12 |
| *** luksky has quit IRC | 14:43 | |
| *** henriqueof has quit IRC | 14:45 | |
| johnsom | kklimonda So, dashboard should not be looking at flavor profiles at all. They are an admin only object. It should only be looking at flavors.... Let me look at the dashboard code. | 15:00 |
| *** Vorrtex has joined #openstack-lbaas | 15:01 | |
| johnsom | kklimonda Ok, bummer, the patch that merged for that is trying to look up the flavor profile to get the provider, which is not correct. We will have to fix that bug. | 15:03 |
| johnsom | It must have only been tested with the admin account. sigh. | 15:05 |
| johnsom | I have opened https://storyboard.openstack.org/#!/story/2005759 for this | 15:08 |
| colin- | what is causing my housekeeping to routinely spike up to ~50 CPU utilization and back down again? any guesses? | 15:23 |
| colin- | i'm not super familiar with the duties of this guy besides deleting stuff and purging records | 15:24 |
| cgoncalves | colin-, could you check if you are running with this patch: https://review.opendev.org/#/q/Iffc960c7c3a986328cfded1b4e408931ab0a7877 | 15:27 |
| *** gcheresh_ has quit IRC | 15:27 | |
| johnsom | FYI, I think I have a fix for the dashboard bug, starting testing now. | 15:30 |
| colin- | i do _not_ have that but have a deployment window coming up later where i will be implementing it | 15:31 |
| colin- | how satisfying thx cgoncalves | 15:31 |
| cgoncalves | colin-, you're welcome :) let us know if it helps | 15:33 |
| cgoncalves | also if it doesn't :) | 15:34 |
| *** ltomasbo has quit IRC | 15:36 | |
| *** ramishra has quit IRC | 15:51 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-dashboard master: Fix 403 issue when creating load balancers https://review.opendev.org/660768 | 15:51 |
| johnsom | kklimonda https://review.opendev.org/660768 | 15:51 |
| johnsom | Now that is service... lol | 15:51 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-dashboard stable/stein: Fix 403 issue when creating load balancers https://review.opendev.org/660769 | 15:52 |
| rm_work | #startmeeting Octavia | 16:00 |
| openstack | Meeting started Wed May 22 16:00:00 2019 UTC and is due to finish in 60 minutes. The chair is rm_work. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:00 | |
| openstack | The meeting name has been set to 'octavia' | 16:00 |
| rm_work | o/ | 16:00 |
| xgerman | O/ | 16:00 |
| ataraday_ | hi | 16:00 |
| rm_work | I am ... still working from yesterday, so bear with me | 16:00 |
| cgoncalves | o/ | 16:00 |
| johnsom | o/ | 16:00 |
| johnsom | Keep forgetting I need to raise my hand now.... lol | 16:00 |
| cgoncalves | good, already warmed up | 16:01 |
| rm_work | yes, you have been demoted (promoted?) to "regular participant" :D | 16:01 |
| rm_work | #topic Announcements | 16:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 16:01 | |
| johnsom | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html | 16:01 |
| johnsom | Some minor requirements changes are coming. | 16:02 |
| johnsom | I think we are mostly up to date on that, but thought I would highlight the thread | 16:02 |
| rm_work | I'm just working away... nothing really to mention specifically... any other announcements? | 16:03 |
| johnsom | I think the gate issues are now fixed with the requirements upper-constraints file updated this morning. | 16:03 |
| colin- | o/ | 16:03 |
| johnsom | Also, if you haven't done the user survey: | 16:04 |
| johnsom | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006393.html | 16:04 |
| johnsom | Please raise awareness that Octavia matters to you.... | 16:04 |
| nmagnezi | o/ | 16:04 |
| rm_work | ok... moving on then | 16:04 |
| rm_work | #topic Brief progress reports / bugs needing review | 16:05 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:05 | |
| *** henriqueof has joined #openstack-lbaas | 16:05 | |
| johnsom | I have pivoted from the unset work to help jump start the jobboard work. | 16:05 |
| johnsom | I have posted a patch that creates an "amphorav2" provider driver/controller: | 16:05 |
| johnsom | #link https://review.opendev.org/659689 | 16:06 |
| johnsom | I am currently working on the "demo" patch for switching those flows over to using the provider driver models. This will also remove the DB objects from the flows for the jobboard work. | 16:06 |
| johnsom | However, I was a genius and picked "listener" as the demo, which is probably the most complicated of them all. A still very WIP patch: | 16:07 |
| johnsom | #link https://review.opendev.org/660236 | 16:07 |
| johnsom | I hope I can wrap that up today. I think there is a follow on patch for octavia-lib to add the project ID to the objects. I'm pretty sure the vmware NSX driver needs that as well. | 16:08 |
| cgoncalves | it will be a breeze reviewing these jobboard patches | 16:08 |
| ataraday_ | I mostly rebase my changes on johnsom "amphorav2" provider driver/controlle and look at refator example - start modify patches that I already have on this topic. | 16:08 |
| rm_work | I am working on one of the items I signed up for this cycle, multi-vip! The strategy is to add "additional_vips" as a list of subnet_id (+ optionally ip_address) that will be added to the VIP port. This would allow ipv6+ipv4 on the same LB, as an example. Maybe take a look and give feedback now if you don't like the way it's set up on the user-facing side: https://review.opendev.org/#/c/660239/ | 16:08 |
| rm_work | #link https://review.opendev.org/#/c/660239 | 16:08 |
| johnsom | ataraday_ Feedback is welcome. Let me know if what I'm doing makes sense, etc. | 16:08 |
| ataraday_ | johnsom, all seems pretty good, thanks a lot for this huge piece of work! | 16:09 |
| johnsom | ataraday_ Also shame me if I end up doing something you have already posted. grin I'm kind of just running with this. | 16:09 |
| rm_work | I still have a bit of work on the backend / plugging side of things | 16:09 |
| cgoncalves | I pushed to Gerrit a patch I started in November that intends to implement VIP ACL API. I just rebased it before pushing, still much WIP. listener POST works, PUT doesn't | 16:09 |
| cgoncalves | #link https://review.opendev.org/#/q/topic:vip-acl | 16:09 |
| *** tesseract has quit IRC | 16:10 | |
| johnsom | I also want to highlight a critical patch for Octavia dashboard: | 16:10 |
| ataraday_ | also this small change ready for review https://review.opendev.org/#/c/659538/ - but it is based on johnsom's change anyway | 16:10 |
| johnsom | #link https://review.opendev.org/660768 | 16:10 |
| johnsom | backport to stein here: | 16:10 |
| johnsom | #link https://review.opendev.org/660769 | 16:10 |
| rm_work | so much stuff in progress \o/ | 16:10 |
| johnsom | We missed that it was trying to access flavor profiles which only admins can do. | 16:11 |
| rm_work | yeah, i had a feeling that might have happened | 16:12 |
| johnsom | Since lxkong isn't likely here I will also highlight his autoscaling demo using heat and octavia: https://youtu.be/dXsGnbr7DfM | 16:13 |
| johnsom | He posted it to the openstack discuss mailing list | 16:13 |
| xgerman | Woot!! | 16:14 |
| rm_work | I feel like we're getting into open discussion-ish | 16:14 |
| johnsom | opps, auto healing, not auto scaling | 16:14 |
| johnsom | Sorry, just thought it was an update from another team member. | 16:14 |
| rm_work | yeah that's true | 16:14 |
| *** tesseract has joined #openstack-lbaas | 16:14 | |
| *** tesseract has quit IRC | 16:14 | |
| rm_work | i am just trying to move things along because my vision is starting to get sparkly and i'm looking forward to sleep :D | 16:15 |
| cgoncalves | nice! I'll watch it after the meeting for sure | 16:15 |
| rm_work | any other progress reports? | 16:15 |
| johnsom | lol, ok, I'm done | 16:15 |
| rm_work | #topic Open Discussion | 16:16 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:16 | |
| rm_work | anything else folks want to discuss today? we haven't really had any specific agenda items in a while | 16:16 |
| rm_work | not sure if there is anything pressing | 16:16 |
| ataraday_ | about jobboard redis and zookeeper.. | 16:17 |
| johnsom | I don't have any other topics today. | 16:17 |
| ataraday_ | I put commemt on the 9th patch set on this change https://review.opendev.org/#/c/647406/9 | 16:18 |
| ataraday_ | if anyone interested could take a look | 16:18 |
| *** Vorrtex has quit IRC | 16:18 | |
| *** rpittau is now known as rpittau|afk | 16:18 | |
| johnsom | Ah, yes. I think that should be ok to move the loop into the flow using the "retry" flow logic | 16:19 |
| rm_work | hopefully yeah | 16:19 |
| johnsom | #link https://docs.openstack.org/taskflow/latest/user/atoms.html#retry | 16:20 |
| ataraday_ | OK, then I try to do this | 16:21 |
| rm_work | A little bit of a meta topic -- I think we should nominate someone to update our meeting wiki -- I think johnsom was doing it before when he ran the meetings, but I basically forgot it existed until now. Any volunteers for meeting scribe? | 16:21 |
| johnsom | Probably just need to move the loop out to a retry "times" element. | 16:21 |
| rm_work | #link https://wiki.openstack.org/wiki/Octavia/Weekly_Meeting_Agenda | 16:21 |
| * johnsom takes a step back. Needs a break.... | 16:22 | |
| rm_work | the minutes are automatically created, so this is actually like ... a pre-scribe (because the idea would be to put the next meeting up ahead of time so people can add topics) | 16:22 |
| rm_work | though also, they need to be added to here maybe? https://wiki.openstack.org/wiki/Octavia/Meeting_Minutes | 16:22 |
| rm_work | which is just a matter of copy/pasting a link once a week after the meeting | 16:23 |
| johnsom | Yeah, the two steps I had been doing is: create agenda, then after post the links | 16:23 |
| rm_work | any takers? ;) | 16:24 |
| rm_work | cgoncalves: have I delegated enough to you yet? | 16:24 |
| eandersson | o/ | 16:24 |
| * cgoncalves looks away | 16:24 | |
| rm_work | eandersson: a volunteer? :D | 16:24 |
| eandersson | lol | 16:24 |
| cgoncalves | oh, there! eandersson :) | 16:24 |
| rm_work | aha! he's volunteered! | 16:24 |
| johnsom | I don't know if anyone cares about the links or if they are fine just using the eavesdrop page | 16:24 |
| eandersson | unfortunately just meant to highlight that I showed up lol | 16:24 |
| johnsom | eandersson Perfect, thanks for volunteering! | 16:25 |
| eandersson | =]] | 16:25 |
| *** sapd1_x has quit IRC | 16:25 | |
| cgoncalves | rm_work, I can be eandersson's backup | 16:25 |
| rm_work | alright, that's all I had, eandersson will be our agenda pre-scribe moving forward ^_^ | 16:25 |
| cgoncalves | I'll be on PTO next Wednesday though | 16:26 |
| rm_work | honestly, you could probably do like a month ahead at once | 16:26 |
| rm_work | people can pick the right date hopefully for adding their topics | 16:26 |
| rm_work | it's just copy/pasting and changing a few digits | 16:26 |
| rm_work | and maybe we let the minutes page die, and refer people to the eavesdrop index | 16:27 |
| cgoncalves | sounds reasonable to me | 16:27 |
| *** lemko has quit IRC | 16:28 | |
| rm_work | ok, any other topics for today? we might be able to get out of here early ;) | 16:28 |
| rm_work | looks like maybe we're done? | 16:29 |
| rm_work | alright, thanks for the meeting folks, see you next week o/ | 16:30 |
| rm_work | #endmeeting | 16:30 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Train PTG etherpad: https://etherpad.openstack.org/p/octavia-train-ptg" | 16:30 | |
| openstack | Meeting ended Wed May 22 16:30:25 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:30 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-05-22-16.00.html | 16:30 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-05-22-16.00.txt | 16:30 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-05-22-16.00.log.html | 16:30 |
| * rm_work passes out | 16:30 | |
| ataraday_ | bye | 16:30 |
| *** Vorrtex has joined #openstack-lbaas | 16:35 | |
| *** ataraday_ has quit IRC | 16:35 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Allow multiple VIPs per LB https://review.opendev.org/660239 | 16:38 |
| *** ccamposr__ has joined #openstack-lbaas | 16:52 | |
| *** Swami has joined #openstack-lbaas | 16:53 | |
| *** ccamposr has quit IRC | 16:53 | |
| *** gthiemonge has quit IRC | 17:00 | |
| *** ricolin has quit IRC | 17:13 | |
| kklimonda | johnsom: wow, that was fast ;) | 17:19 |
| johnsom | kklimonda That was a critical bug. Thanks for reporting it to us. Glad I could help. | 17:20 |
| *** luksky has joined #openstack-lbaas | 17:23 | |
| openstackgerrit | Merged openstack/neutron-lbaas-dashboard stable/stein: Imported Translations from Zanata https://review.opendev.org/656620 | 17:28 |
| *** Swami has quit IRC | 17:35 | |
| *** yboaron_ has quit IRC | 18:00 | |
| *** yboaron_ has joined #openstack-lbaas | 18:01 | |
| *** ccamposr__ has quit IRC | 18:22 | |
| *** ganso has joined #openstack-lbaas | 18:38 | |
| ganso | hello folks. I just deployed ocatavia using https://docs.openstack.org/devstack/latest/guides/devstack-with-lbaas-v2.html but enabling horizon. After deployment is complete, I don't see the load balancers tab in Horizon, am I missing a configuration step? Could someone please assist? | 18:39 |
| johnsom | ganso You un-commented the octavia-dashboard section in the plugin? | 18:47 |
| ganso | johnsom: oh I forgot about # enable_plugin octavia-dashboard https://git.openstack.org/openstack/octavia-dashboard.git | 18:48 |
| ganso | johnsom: thanks! | 18:48 |
| johnsom | ganso Also please note the bug that was fixed this morning: https://review.opendev.org/#/c/660768/ | 18:52 |
| ganso | johnsom: thanks! I am deploying octavia to verify a slightly similar problem | 18:54 |
| ganso | johnsom: I suppose that bug above affects only when load balancers are *created*, right? | 18:55 |
| ganso | johnsom: the problem I am facing is that users with just role member of their project cannot load the load balancers tab, nor the overview page or create tenant networks | 18:58 |
| *** ramishra has joined #openstack-lbaas | 19:02 | |
| johnsom | ganso: it would impact some details pages as well. | 19:18 |
| ganso | johnsom: hmmm ok, seems it would be best to wait for it to merge first | 19:19 |
| ganso | johnsom: thanks! | 19:19 |
| johnsom | ganso I can't speak to an issue creating tenant networks as that is a neutron feature and not Octavia. | 19:20 |
| johnsom | ganso However, a role issue for load balancers might be a mis-understanding of the advanced RBAC we have in Octavia. See this page for more information: https://docs.openstack.org/octavia/latest/configuration/policy.html | 19:21 |
| johnsom | ganso They may have the advanced RBAC enabled, but have not created the right role / group configurations. They may also have this disabled and be running the "admin_or_owner" policy file. | 19:22 |
| *** sfilatov has joined #openstack-lbaas | 19:23 | |
| ganso | johnsom: thanks for the link. This matches closely what I am experiencing. A regular user without any octavia roles fails to load the load balancers tab, overview and create tenant networks. Once the role "load-balancer_member" is added to the same user, everything is fixed | 19:23 |
| ganso | johnsom: but the way I see it, the user experience is not being very good when the user doesn't have that role, so I am trying to reproduce this bad user experience problem and work on improving it if it as bad as my customer says it is | 19:24 |
| johnsom | ganso If you want to disable the advanced RBAC you can install the admin_or_owner-policy.json located here: https://github.com/openstack/octavia/tree/master/etc/policy | 19:24 |
| johnsom | ganso Cool, thank you for that. We are a small team and appreciate any help we can get. | 19:25 |
| ganso | johnsom: that could be a solution, but I would need to double check if my customer has other users using the advanced roles (I assume yes) | 19:25 |
| *** ramishra has quit IRC | 19:27 | |
| sfilatov | Hi! I got an issue with octavia: LB stops working(fip assigned on VIP is not accessible) after l3-neutron-agent full sync is triggered(for example after RMQ outage). It stops working because iptables rule in -A neutron-l3-agent-PREROUTING -d <fip> -j DNAT --to-destination <vip> is removed during full sync. Has anyone faced this issue or a similar one? | 19:28 |
| johnsom | sfilatov Not that I know of. That sounds like a pretty big bug in neutron if iptables rules are lost. | 19:29 |
| johnsom | sfilatov Have you inquired in #openstack-neutron? | 19:30 |
| sfilatov | not yet. I'll try openstack-neutron as well. Thx! | 19:31 |
| ganso | johnsom: btw I was already stacking while we were talking, and devstack failed with this error while configuring octavia-dashboard: http://paste.openstack.org/show/751954/ | 19:31 |
| ganso | johnsom: it looks like octavia-dashboard doesn't work with Python3. Is that correct? | 19:32 |
| johnsom | Ugh, yeah, it's a problem with python3. Horizon has the same issue. It's actually the nodejs manage command that is broken. | 19:32 |
| ganso | johnsom: ok, thanks! Will make sure to not enable python3 on next attempt | 19:33 |
| johnsom | ganso Or hack the file to have #!/usr/bin/env python3 instead of python | 19:34 |
| johnsom | I hit that with the horizon manage.py this morning when I was fixing that other bug. | 19:34 |
| ganso | johnsom: hmmm, will try that =) thanks! | 19:35 |
| johnsom | I might be able to fix our version actaully. I should look at that devstack script | 19:35 |
| johnsom | ganso It's this line: https://github.com/openstack/horizon/blob/master/manage.py#L1 | 19:35 |
| johnsom | Ah, that is being run a different way in the devstack plugin. Yeah, this should be a quick fix as well | 19:38 |
| openstackgerrit | Merged openstack/octavia-dashboard master: Fix 403 issue when creating load balancers https://review.opendev.org/660768 | 19:38 |
| johnsom | It's this line: https://github.com/openstack/octavia-dashboard/blob/master/devstack/plugin.sh#L11 | 19:39 |
| ganso | johnsom: thanks! | 19:39 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-dashboard master: Fix devstack plugin python3 support https://review.opendev.org/660813 | 19:47 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-dashboard stable/stein: Fix devstack plugin python3 support https://review.opendev.org/660814 | 19:48 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-dashboard stable/rocky: Fix devstack plugin python3 support https://review.opendev.org/660816 | 19:49 |
| johnsom | ganso Thanks for letting us know about that. Those should fix that issue with the devstack plugin | 19:49 |
| ganso | johnsom: thank you for addressing the issues =) | 19:55 |
| johnsom | Yeah, cool, those work just fine. | 20:07 |
| johnsom | 2019-05-22 20:02:10.479 lib/horizon:configure_horizon:81 /usr/bin/python3.6 manage.py compilemessages | 20:07 |
| johnsom | Ha, helps if I copy the right line.... | 20:08 |
| johnsom | 2019-05-22 20:03:20.101 | /opt/stack/octavia-dashboard/devstack/plugin.sh:octavia_dashboard_configure:11 /usr/bin/python3.6 ../manage.py compilemessages | 20:08 |
| *** pcaruana has quit IRC | 20:20 | |
| colin- | cgoncalves: that did help, thanks again | 20:20 |
| *** Vorrtex has quit IRC | 20:24 | |
| cgoncalves | colin-, cool! happy to hear that | 20:46 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Convert listener flows to use provider models https://review.opendev.org/660236 | 20:58 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Create Amphora V2 provider driver https://review.opendev.org/659689 | 21:00 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Convert listener flows to use provider models https://review.opendev.org/660236 | 21:00 |
| *** sfilatov has quit IRC | 21:05 | |
| *** boden has quit IRC | 21:19 | |
| *** henriqueof has quit IRC | 21:36 | |
| *** ut2k3 has joined #openstack-lbaas | 22:15 | |
| ut2k3 | Hi johnsom maybe you remember me and my problem with the empty amphora table. Maybe you can help me out with a follow up problem. The LBs are up, but not reachable via their FloatingIPs (which was working before) could it be that OS / Octavia tries to point them to the wrong internalIP/Port? | 22:16 |
| ut2k3 | Just for my Understanding, do the amphora need to have the same IP as the vip_address? | 22:17 |
| johnsom | ut2k3 We where just talking about a neutron bug earlier: https://bugs.launchpad.net/neutron/+bug/1826066 | 22:17 |
| openstack | Launchpad bug 1826066 in neutron "Iptables rules for unbound ports removed during agent sync" [Undecided,Incomplete] | 22:17 |
| johnsom | ut2k3 I suspect that is your problem | 22:17 |
| johnsom | ut2k3 Another user was saying that a rabbit outage caused this "agent full sync" and wiped out his floating IPs in neutron's iptables. | 22:18 |
| johnsom | Quoting the other user: | 22:19 |
| johnsom | LB stops working(fip assigned on VIP is not accessible) after l3-neutron-agent full sync is triggered(for example after RMQ outage). It stops working because iptables rule in -A neutron-l3-agent-PREROUTING -d <fip> -j DNAT --to-destination <vip> is removed during full sync. | 22:19 |
| ut2k3 | In my case the amphora table was empty and you helped me recreating them. So in theory detaching/attaching of floating IP should help or not? | 22:20 |
| johnsom | ut2k3 Can you check if those iptables rules are present for neutron? | 22:20 |
| johnsom | ut2k3 Probably. I don't think it is related to the amphora table being empty, we repaired those properly, so they still have the same VIP address. This is why I suspect it is the neutron bug. | 22:20 |
| ut2k3 | Ok, so just to have the understanding the amphora don't need to have the vip_address? Because the vip_address != the private IP address of the currently running amphora | 22:23 |
| johnsom | Correct, each amphora has two "ports", on is real, one is a neutron "Allowed Address Pairs" port that is fake. The AAP port had the VIP, the base port has a different IP. | 22:24 |
| johnsom | A server list will only show the base port | 22:24 |
| johnsom | But, if you do an openstack port show on that base port ID, you will see at the top the AAP configuration with the VIP on it. | 22:25 |
| ut2k3 | Ok let me check therefore `iptables -L neutron-l3-agent-PREROUTING` | 22:26 |
| johnsom | This is neutron's complicated way of having two IP addresses on a single port. | 22:27 |
| *** goldyfruit_ has joined #openstack-lbaas | 22:28 | |
| *** goldyfruit has quit IRC | 22:30 | |
| ut2k3 | The bug ticket you've shared is something related to neutron-vpn-agent, I am not using that. | 22:32 |
| ut2k3 | Other floating IPs are working. | 22:32 |
| johnsom | Right, the same happens the the L3 agent you are using for floating IPs | 22:33 |
| *** goldyfruit_ has quit IRC | 22:33 | |
| *** pnull has joined #openstack-lbaas | 22:35 | |
| ut2k3 | Ok, do you know how to solve or to check that in detail? Thats not on the host right, its on the router or where should be that iptables-rule exist? | 22:36 |
| johnsom | They would be in the router namespace where your l3-agents are installed | 22:37 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Convert listener flows to use provider models https://review.opendev.org/660236 | 22:39 |
| ut2k3 | When I do a `p netns exec qrouter-6253b80b-de18-4081-8ddf-45db94cef43a iptables -L neutron-l3-agent-PREROUTING -t nat` on a network node | 22:42 |
| *** goldyfruit has joined #openstack-lbaas | 22:42 | |
| ut2k3 | It shows a for me correct line with: `DNAT all -- anywhere x.x.x.x to:10.123.0.8` where x.x.x.x is my floating IP | 22:42 |
| ut2k3 | > | acae625f-01ff-4bfc-9b74-df0df3f59be6 | cluster-production-k8s-de-kar--4fo2nxngz7a6-api_lb-7erqpobqhuh3-loadbalancer-try3py43mbhm | 0e790846485a4db6b0f9ab6ec958c1ed | 10.123.0.9 | ACTIVE | amphora | | 22:42 |
| johnsom | hmmm, ok, that should be it | 22:42 |
| ut2k3 | That's the corresponding line from `openstack loadbalancer list` | 22:43 |
| johnsom | oh, that is not the right DNAT line then | 22:43 |
| johnsom | Well, I'm trying to remember how floats work with AAP ports. | 22:43 |
| johnsom | is .8 the base port? | 22:43 |
| ut2k3 | 10.123.0.8 == vip_address | 22:45 |
| ut2k3 | Ah sorry pasted you the wrong line | 22:45 |
| ut2k3 | | 9e950a88-af30-4eeb-8fb6-d8c20388db17 | ac8042c20554011e9a047fa163ef65c4 | 0e790846485a4db6b0f9ab6ec958c1ed | 10.123.0.8 | ACTIVE | amphora | | 22:45 |
| ut2k3 | | 9e950a88-af30-4eeb-8fb6-d8c20388db17 | ac8042c20554011e9a047fa163ef65c4 | 0e790846485a4db6b0f9ab6ec958c1ed | 10.123.0.8 | ACTIVE | amphora | | 22:45 |
| ut2k3 | http://paste.openstack.org/show/751959/ | 22:45 |
| johnsom | Let me build an LB with a FIP so I can compare | 22:45 |
| ut2k3 | Ok thanks | 22:47 |
| johnsom | So it should look like this: | 22:53 |
| johnsom | https://www.irccloud.com/pastebin/98K1bHZF/ | 22:53 |
| johnsom | The DNAT should point to the VIP IP | 22:54 |
| johnsom | https://www.irccloud.com/pastebin/AVCdlDlv/ | 22:55 |
| ut2k3 | http://paste.openstack.org/show/751960/ | 22:59 |
| johnsom | That looks ok | 22:59 |
| ut2k3 | From a host, I am able to ping the Amphora Instance IP itself, but not the vip_address or request something via the port that should be open. | 22:59 |
| johnsom | Yeah, you should not be able to ping the VIP | 23:00 |
| johnsom | But it's not responding to queries to the VIP on the port that it is load balancing? | 23:00 |
| ut2k3 | Interesting is that the status of the port "down" is, could that be the reason? | 23:02 |
| johnsom | No, the VIP port is a fake neutron port, it is always down | 23:02 |
| johnsom | The base port must be up however | 23:02 |
| ut2k3 | OK | 23:03 |
| ut2k3 | So thats normal: | 10389c9a-3250-41a3-a398-b3027c0ad1c8 | octavia-lb-9e950a88-af30-4eeb-8fb6-d8c20388db17 | fa:16:3e:a3:0e:1e | ip_address='10.123.0.8', subnet_id='843f80f1-cf5c-4b16-a04b-b234bb6b4e88' | DOWN | | 23:03 |
| johnsom | | 9c0ae11d-6124-4136-bc04-24177b176126 | octavia-lb-e64e658e-a0ac-46f6-8634-440d5f716212 | fa:16:3e:0d:53:14 | ip_address='10.0.0.4', subnet_id='254c0e46-e4ab-405e-abf3-ca62856828a4' | DOWN || | 23:03 |
| johnsom | yep | 23:03 |
| johnsom | So what do you get if you do a status tree call on the load balancer? | 23:03 |
| ut2k3 | Whats the command for that? | 23:04 |
| johnsom | openstack loadbalancer status show <lb ID> | 23:05 |
| ut2k3 | http://paste.openstack.org/show/751962/ | 23:06 |
| johnsom | So the LB is up and the members are healthy assuming your health manager process is running | 23:08 |
| ut2k3 | Yep, requesting 10.123.0.23 in that case on Port 6443 works fine | 23:09 |
| ut2k3 | (From an instance) | 23:10 |
| johnsom | Hmmm, I'm running out of ideas as it looks like octavia is healthy. | 23:11 |
| johnsom | You did a load balancer failover to repair these right? Not just an amphora failover? | 23:11 |
| ut2k3 | Yep I did a `openstack loadbalancer failover...` | 23:12 |
| johnsom | Did you try detaching the FIP and re-attaching it? | 23:12 |
| ut2k3 | Yes, in our case even the vip_address is not accessible from the private net | 23:13 |
| ut2k3 | So I think the problem is a bit deeper, not only the floating ip | 23:13 |
| johnsom | Well, I would still like to try the FIP detach as this still could be a neutron issue. Especially if you are using DVR | 23:14 |
| ut2k3 | Ok I gonna detach for a test the FIP of the LB with vip_address: 10.123.0.9 | 23:15 |
| ut2k3 | So its gone, let me try to do a `curl -v https://10.123.0.9:6443` | 23:16 |
| ut2k3 | Nope still not available: `curl: (7) Failed to connect to 10.123.0.9 port 6443: No route to host` | 23:16 |
| johnsom | No route to host implies you are not connected to the 10.123.0.9 subnet. | 23:17 |
| johnsom | Let's re-attach the FIP and see if it works | 23:17 |
| ut2k3 | http://paste.openstack.org/show/751964/ | 23:19 |
| ut2k3 | > (10.123.0.9) at <incomplete> on eth0 | 23:19 |
| ut2k3 | Gonna attach it back | 23:19 |
| *** rcernin has joined #openstack-lbaas | 23:20 | |
| johnsom | Just to double check, is fa:16:3e:f6:5c:49 the mac of the base port for the .23 port? | 23:21 |
| *** goldyfruit has quit IRC | 23:21 | |
| johnsom | Ah, nevermind, I got confused because the pastes have been from different load balancers | 23:22 |
| ut2k3 | Sorry for jumping here. | 23:23 |
| ut2k3 | fa:16:3e:f6:5c:49 is the MAC address of an Kubernetes instance | 23:23 |
| johnsom | How if the FIP after the attach? | 23:23 |
| ut2k3 | The 10.123.0.9 == vi_address | 23:23 |
| ut2k3 | FIP after attaching it back is still also not available | 23:23 |
| johnsom | Bummer. Well, I guess the next thing I would try is another loadbalancer failover | 23:24 |
| johnsom | Force it to rebuild the ports and security groups | 23:24 |
| ut2k3 | The LB is quite fresh, I did a failover 2-3 times already. | 23:25 |
| ut2k3 | Since it was not reachable | 23:25 |
| johnsom | wow, ok, then this is super odd as the status shows it as up and healthy. | 23:25 |
| johnsom | I guess you can check the security groups just to make sure the port is open on the VIP base port. | 23:26 |
| johnsom | After that I would jump inside the amphora with ssh, go into the netns and do a tcpdump to see if the packets are getting to the instance. | 23:26 |
| johnsom | Note, but default, from inside the amp, you cannot directly connect to the members without bringing up the lo interface. We don't bring it up as we don't need it. | 23:27 |
| johnsom | There should be a security group called lb-4d3df5bc-240b-4dc5-a03a-5c6e089f9add where the uuid is the load balancer ID. | 23:29 |
| johnsom | It should have your listener port open, plus 1025 which is used for sync on the base port. | 23:30 |
| ut2k3 | In my case the `octavia-lb-acae625f-01ff-4bfc-9b74-df0df3f59be6` == vip_address => don't have a security group | 23:31 |
| ut2k3 | But the belonging VRRP have a security group. | 23:32 |
| johnsom | Right, it is only on the base port, the VIP port is a fake neutron port | 23:32 |
| ut2k3 | The `octavia-lb-vrrp` contains also the vip_address as an allowed Address-Pair | 23:33 |
| johnsom | Hmmm, actually, my vip port does have an SG on it | 23:33 |
| johnsom | Yeah, on mine, both ports have the same SG on it | 23:34 |
| johnsom | Maybe that is the issue, maybe the VIP port lost it's SG somehow | 23:34 |
| johnsom | I would try applying that SG from the octavia-lb-vrrp port to the VIP (AAP) port. Just make sure you have the right ports and don't cross the LB SGs | 23:35 |
| johnsom | Doesn't seem to make a difference for me. If I delete that SG I can still connect. As I remembered, only the base port SG matters to neutron. the AAP port doesn't | 23:37 |
| ut2k3 | Yeah and since `octavia-lb...` is in another namespace I can't attach the SG anyway to it. | 23:39 |
| ut2k3 | http://paste.openstack.org/show/751967/ | 23:40 |
| ut2k3 | And thats the vrrp http://paste.openstack.org/show/751968/ | 23:41 |
| johnsom | Those look fine to me | 23:46 |
| ut2k3 | So without checking documentation from my understanding the VRRP IP is configured as secondary IP to the eth interface within the amphora right? | 23:49 |
| ut2k3 | Or to be more precise: The thing I don't understand is: how can I be able to access the VRRP IP but NOT the VIP. IMHO, they should be configured on the same network interface inside the amphora instance. So. How can it be possible that I don't even get an ARP response for the VIP while I can access/ping the VRRP address? | 23:50 |
| *** goldyfruit has joined #openstack-lbaas | 23:53 | |
| johnsom | They are both on the same interface inside the "amphora-haproxy" network namespace inside the amphora. | 23:55 |
| johnsom | The VIP address is the secondary IP on the interface | 23:55 |
| johnsom | I need to sign off for the day, it's already a few hours past when I planned to stop. I can continue to help you tomorrow or maybe some of the other cores will be on and be of assistance. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!