| *** yamamoto has joined #openstack-lbaas | 00:04 | |
| *** mithilarun has quit IRC | 00:41 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Make amphora use a single HAProxy instance https://review.opendev.org/668068 | 00:51 |
|---|---|---|
| *** ricolin_ has joined #openstack-lbaas | 00:51 | |
| *** ricolin_ is now known as ricolin | 00:52 | |
| *** ramishra has quit IRC | 00:56 | |
| *** yamamoto has quit IRC | 00:59 | |
| *** yamamoto has joined #openstack-lbaas | 00:59 | |
| *** yamamoto has quit IRC | 01:30 | |
| *** yamamoto has joined #openstack-lbaas | 01:40 | |
| *** yamamoto has quit IRC | 01:51 | |
| *** yamamoto has joined #openstack-lbaas | 02:15 | |
| *** yamamoto has quit IRC | 02:36 | |
| *** dasp has quit IRC | 02:46 | |
| *** dasp has joined #openstack-lbaas | 03:03 | |
| *** yamamoto has joined #openstack-lbaas | 03:15 | |
| *** yamamoto has quit IRC | 03:25 | |
| *** psachin has joined #openstack-lbaas | 03:35 | |
| *** ramishra has joined #openstack-lbaas | 03:42 | |
| *** numans has joined #openstack-lbaas | 03:52 | |
| *** ricolin_ has joined #openstack-lbaas | 03:54 | |
| *** ricolin has quit IRC | 03:57 | |
| *** ajay33 has joined #openstack-lbaas | 04:12 | |
| *** tkajinam has quit IRC | 05:01 | |
| *** gcheresh_ has joined #openstack-lbaas | 05:05 | |
| *** vishalmanchanda has joined #openstack-lbaas | 05:05 | |
| *** pcaruana has joined #openstack-lbaas | 05:06 | |
| *** ricolin_ is now known as ricolin | 05:37 | |
| openstackgerrit | Merged openstack/octavia master: only rollback DB when we have a connection to the DB https://review.opendev.org/668032 | 05:52 |
| *** tkajinam has joined #openstack-lbaas | 06:00 | |
| *** ramishra has quit IRC | 06:08 | |
| *** ramishra has joined #openstack-lbaas | 06:16 | |
| *** luksky has joined #openstack-lbaas | 06:27 | |
| *** lemko has joined #openstack-lbaas | 06:35 | |
| *** luksky has quit IRC | 06:48 | |
| *** ricolin_ has joined #openstack-lbaas | 06:51 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia master: Prevent UDP LBs to use different IP protocol versions in amphora driver https://review.opendev.org/668617 | 06:52 |
| *** ricolin has quit IRC | 06:53 | |
| *** luksky has joined #openstack-lbaas | 07:06 | |
| *** rpittau|afk is now known as rpittau | 07:06 | |
| *** ccamposr has quit IRC | 07:18 | |
| *** tesseract has joined #openstack-lbaas | 07:21 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add tests for mixed IP networks UDP members https://review.opendev.org/668619 | 07:27 |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add UDP test scenario https://review.opendev.org/656515 | 07:27 |
| *** luksky11 has joined #openstack-lbaas | 07:43 | |
| *** luksky has quit IRC | 07:45 | |
| *** luksky11 has quit IRC | 07:55 | |
| *** trident has quit IRC | 08:08 | |
| *** luksky11 has joined #openstack-lbaas | 08:08 | |
| *** trident has joined #openstack-lbaas | 08:09 | |
| *** luksky11 has quit IRC | 08:11 | |
| *** luksky11 has joined #openstack-lbaas | 08:27 | |
| *** luksky11 has quit IRC | 08:33 | |
| *** ivve has joined #openstack-lbaas | 08:34 | |
| *** tkajinam has quit IRC | 08:39 | |
| *** ricolin__ has joined #openstack-lbaas | 08:44 | |
| *** ricolin_ has quit IRC | 08:47 | |
| *** ricolin__ is now known as ricolin | 08:55 | |
| *** tesseract-RH has joined #openstack-lbaas | 08:57 | |
| *** tesseract has quit IRC | 08:58 | |
| *** luksky11 has joined #openstack-lbaas | 09:12 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Transition l7policy flows to dicts https://review.opendev.org/665977 | 09:25 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Transition l7rule flows to dicts https://review.opendev.org/668173 | 09:25 |
| *** tesseract-RH has quit IRC | 09:29 | |
| *** tesseract has joined #openstack-lbaas | 09:29 | |
| *** tesseract has quit IRC | 09:33 | |
| *** tesseract has joined #openstack-lbaas | 09:33 | |
| *** tesseract has quit IRC | 09:38 | |
| *** tesseract has joined #openstack-lbaas | 09:39 | |
| *** aojea has joined #openstack-lbaas | 10:03 | |
| *** aojea has quit IRC | 10:03 | |
| *** psachin has quit IRC | 11:06 | |
| *** sapd1_x has joined #openstack-lbaas | 11:14 | |
| *** sapd1_x has quit IRC | 11:30 | |
| *** boden has joined #openstack-lbaas | 12:04 | |
| *** gcheresh_ has quit IRC | 12:15 | |
| *** boden has quit IRC | 12:34 | |
| *** psachin has joined #openstack-lbaas | 12:44 | |
| *** happyhemant has joined #openstack-lbaas | 12:46 | |
| *** gcheresh_ has joined #openstack-lbaas | 12:54 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Transition amphora flows to dicts https://review.opendev.org/668898 | 13:01 |
| squarebracket | hurray my patch was merged | 13:10 |
| *** ajay33 has quit IRC | 13:14 | |
| *** boden has joined #openstack-lbaas | 13:17 | |
| openstackgerrit | Elod Illes proposed openstack/octavia stable/ocata: Add bindep.txt and ignore sha1 warning https://review.opendev.org/668901 | 13:21 |
| *** ccamposr has joined #openstack-lbaas | 13:24 | |
| *** ramishra has quit IRC | 13:26 | |
| *** ramishra has joined #openstack-lbaas | 13:28 | |
| *** vishalmanchanda has quit IRC | 13:32 | |
| *** psachin has quit IRC | 13:47 | |
| *** spatel has joined #openstack-lbaas | 13:56 | |
| *** gcheresh_ has quit IRC | 14:14 | |
| *** ivve has quit IRC | 14:14 | |
| *** ricolin has quit IRC | 14:16 | |
| *** mithilarun has joined #openstack-lbaas | 14:20 | |
| *** gcheresh_ has joined #openstack-lbaas | 14:58 | |
| *** vishalmanchanda has joined #openstack-lbaas | 14:58 | |
| *** gcheresh_ has quit IRC | 15:13 | |
| *** henriqueof has joined #openstack-lbaas | 15:24 | |
| *** Vorrtex has joined #openstack-lbaas | 15:38 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Make amphora use a single HAProxy instance https://review.opendev.org/668068 | 15:43 |
| *** ataraday_ has joined #openstack-lbaas | 15:51 | |
| *** mithilarun has quit IRC | 15:51 | |
| *** ajay33 has joined #openstack-lbaas | 15:52 | |
| spatel | johnsom: yt | 15:52 |
| *** mithilarun has joined #openstack-lbaas | 15:52 | |
| johnsom | spatel Yes. We are just about to the weekly IRC meeting time | 15:52 |
| spatel | go ahead.. i will catch you after meeting | 15:53 |
| johnsom | spatel We have 7 minutes. | 15:53 |
| johnsom | grin | 15:53 |
| *** ataraday_ has quit IRC | 15:53 | |
| spatel | https://github.com/rcbops/rpc-octavia/blob/master/INSTALLATION.md | 15:53 |
| johnsom | The rackspace stuff? | 15:54 |
| johnsom | I don't think they even use that anymore | 15:54 |
| spatel | i am successfully spun up amphora and it wire up with neutron but not getting ip from DHCP | 15:54 |
| spatel | if you see in that doc they used specially bridge v-br-lbaas and v-br-vlan | 15:55 |
| johnsom | spatel Why aren't you using the OpenStack Ansible role instead of this? | 15:55 |
| spatel | openstack-ansible doesn't wire up neutron with lb-mgmt network | 15:56 |
| johnsom | yes it does | 15:56 |
| xgerman | +1 | 15:56 |
| spatel | You are saying i don't need to do anything what that rackspace document saying? | 15:56 |
| johnsom | Ah, the author of both the RPC-octavia and OpenStack Ansible role has arrived.... | 15:57 |
| *** mithilarun has quit IRC | 15:57 | |
| johnsom | spatel yes | 15:57 |
| spatel | I think i am confused here.. | 15:57 |
| spatel | how neutron DHCP namespace going to talk to lb-mgmt network? | 15:58 |
| -spatel- [root@ostack-infra-2-1 ~]# ip netns list | 15:58 | |
| -spatel- qdhcp-acf559ef-2a89-4956-80d3-ec7bfd03b225 (id: 2) | 15:58 | |
| -spatel- qdhcp-1f81a77d-02d2-4f64-b767-22222fc5368c (id: 1) | 15:58 | |
| spatel | This is my lb-mgmt network ip address on dhcp ns | 15:59 |
| -spatel- [root@ostack-infra-2-1 ~]# ip netns exec qdhcp-1f81a77d-02d2-4f64-b767-22222fc5368c ip a | grep 172.27.12.2 | 15:59 | |
| -spatel- inet 172.27.12.2/21 brd 172.27.15.255 scope global ns-99b9b080-91 | 15:59 | |
| spatel | my amphora not able to talk to this IP that is why they are not able to get IP from dhcp | 15:59 |
| johnsom | spatel This is the OSA task that sets up the lb-mgmt-net: https://github.com/openstack/openstack-ansible-os_octavia/blob/master/tasks/octavia_mgmt_network.yml | 16:00 |
| spatel | http://paste.openstack.org/show/753835/ | 16:00 |
| rm_work | #startmeeting Octavia | 16:00 |
| openstack | Meeting started Wed Jul 3 16:00:55 2019 UTC and is due to finish in 60 minutes. The chair is rm_work. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:00 | |
| openstack | The meeting name has been set to 'octavia' | 16:00 |
| rm_work | Heyas :D | 16:01 |
| johnsom | o/ | 16:01 |
| nmagnezi | o/ | 16:01 |
| gthiemonge | o/ | 16:01 |
| *** mithilarun has joined #openstack-lbaas | 16:01 | |
| cgoncalves | o/ | 16:01 |
| rm_work | #topic Announcements | 16:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 16:01 | |
| xgerman | O/ | 16:02 |
| rm_work | CFP is ... closed? almost closed? | 16:02 |
| rm_work | We aren't submitting anything that I'm aware of for Shanghai, other than the project update | 16:02 |
| cgoncalves | closed yesterday EOD (PST time?) | 16:02 |
| rm_work | and ... onboarding? might be part of the regularly scheduled stuff... I need to wait to hear from the organizer folks | 16:02 |
| cgoncalves | I submitted one with dulek | 16:02 |
| rm_work | ah ok | 16:02 |
| johnsom | The on boarding is being handled differently this time. Not sure what it will look like. | 16:03 |
| cgoncalves | ah, sorry. CFP deadline was extended until July 8th | 16:03 |
| cgoncalves | "Accepting submissions until Jul 8th 8:59 am (Europe/Berlin)" | 16:03 |
| dulek | rm_work: Oh, so project updates emails were already sent to PTL's? | 16:03 |
| * dulek excuses for interrupting but as cgoncalves pinged him… | 16:04 | |
| rm_work | Not yet, waiting on that | 16:04 |
| dulek | rm_work: Okay, good, we're waiting for that too. :) | 16:04 |
| johnsom | Kendall said it would still be a week or so | 16:04 |
| rm_work | I'm glad you know, johnsom :D | 16:04 |
| rm_work | as the conference liason... | 16:05 |
| dulek | I'll just tag dmellado here then. ^ - project updates emails are going to be there in a week or so. | 16:05 |
| * rm_work shoots a quick email to Kendall to clarify that johnsom is the conference liason | 16:05 | |
| johnsom | Ha, sigh. Well, considering I am not planning to attend.... | 16:05 |
| rm_work | that's fine, you did it for Boston and only I went to that one :D | 16:05 |
| rm_work | ah, and german too | 16:06 |
| johnsom | True | 16:06 |
| xgerman | yep, and we had people helping at the lab | 16:07 |
| rm_work | anywho, any other announcements? | 16:07 |
| johnsom | The only other thing I can think of is the upper-constraints changes | 16:08 |
| johnsom | Tony had an e-mail chain about changes to how upper-constraints should be used and are distributred. | 16:08 |
| johnsom | I blasted out patches for all of the repos and branches to update our repos as I was starting to see random/wrong patches come in. | 16:09 |
| johnsom | #link https://review.opendev.org/#/q/topic:constraints-updates | 16:09 |
| cgoncalves | YAUCC (yet another u-c change) | 16:09 |
| johnsom | lol, yes | 16:09 |
| johnsom | So please help by reviewing so we can get this nailed down. | 16:10 |
| johnsom | In the process I did find some issues in our repos that I have corrected in these patches. | 16:10 |
| rm_work | yeah i've been really singularly focused so i haven't been doing reviews the past couple of days, need to do that | 16:10 |
| johnsom | This should help us not end up with broken packaging | 16:10 |
| johnsom | These are all fairly short patches, so should go quick | 16:11 |
| *** ataraday_ has joined #openstack-lbaas | 16:11 | |
| johnsom | I think that is all I have for announcements | 16:11 |
| rm_work | #topic Brief progress reports / bugs needing review | 16:13 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:13 | |
| rm_work | I've got ... a few patches that I wish people would look at -- but no matter how hard I rub my bedside lamp, Will Smith won't pop out of it, so I guess those will remain un-reviewed :D | 16:14 |
| *** rpittau is now known as rpittau|afk | 16:14 | |
| ataraday_ | As always please review: https://review.opendev.org/#/c/659538/ and https://review.opendev.org/#/c/662791/ | 16:15 |
| ataraday_ | and I've got a bunch of transition reviews https://review.opendev.org/#/q/status:open+project:openstack/octavia+branch:master+topic:jobboard_dicts | 16:16 |
| johnsom | Other than the UC patches and some other bug fixes, I have been focused on the single-haproxy process work with rm_work. It's a critical bug/issue so my current top priority. | 16:16 |
| rm_work | yep, basically have put everything else aside for that recently | 16:17 |
| rm_work | would be sweet if people took a look at multivip tho :) | 16:17 |
| cgoncalves | I've been focused on figuring out what's wrong with the centos job. hard to say I've made any progress as it still doesn't work :/ | 16:18 |
| johnsom | cgoncalves Bummer, I thought you had figured it out and made magic happen | 16:18 |
| cgoncalves | this is a priority for me as we could be merging stuff that fails on centos. I reckon the UDP work in Rocky that was not working on centos few days before release | 16:19 |
| cgoncalves | every day I find something fishy and work around it. I'm now on "why ubuntu CI job says it's on a nested virt env??" | 16:20 |
| johnsom | Ok, that is easy, it's a kernel module that loaded. Not the nested virt you are thinking of | 16:20 |
| cgoncalves | also why systemd says "Detected virtualization other." on DIB-built + TCG while same but for ubuntu says "Detected virtualization qemy." | 16:20 |
| cgoncalves | just confirmed 5 minutes ago both OS amps have the kvm_amd kernel module loaded | 16:21 |
| johnsom | Yeah, but that isn't used | 16:21 |
| cgoncalves | anyway, I don't want to hijack the meeting with this | 16:21 |
| rm_work | this is in the gate? because SOME hosts have nested virt support and some don't... right? | 16:22 |
| rm_work | or did we disable it globally now | 16:22 |
| rm_work | anywho, yeah... review patches :D | 16:22 |
| johnsom | We have had is disabled globally for some time now, probably over a year | 16:22 |
| rm_work | #link https://review.opendev.org/667484 | 16:22 |
| rm_work | #link https://review.opendev.org/660239 | 16:22 |
| rm_work | ah you did finally review multivip, sweet | 16:22 |
| rm_work | just need to find time to go back and look at it <_ | 16:23 |
| rm_work | < | 16:23 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix cryptsetup --pbkdf-memory failures https://review.opendev.org/668215 | 16:24 |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/stein: Fix cryptsetup --pbkdf-memory failures https://review.opendev.org/668216 | 16:25 |
| rm_work | ok so | 16:26 |
| rm_work | #topic Open Discussion | 16:26 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:26 | |
| rm_work | Anything? noticed we have more than the normal amount of folks here? | 16:27 |
| rm_work | ah, I do know that we're seeing some sort of issue with members in pools in senlin (internally at verizon media) but I can't speak to it and the guy who can I don't think is responding presently | 16:28 |
| rm_work | so all I can say is nebulously "we'll be looking at the senlin<->octavia integration stuff more closely soon" | 16:29 |
| rm_work | or possibly already are | 16:29 |
| johnsom | Cool | 16:30 |
| rm_work | no one else? guess we can close this up? | 16:31 |
| rm_work | thanks for showing up everyone \o/ | 16:31 |
| rm_work | back to work! or sleep! or whatever you were about to do! | 16:31 |
| rm_work | #endmeeting | 16:31 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Train PTG etherpad: https://etherpad.openstack.org/p/octavia-train-ptg" | 16:31 | |
| openstack | Meeting ended Wed Jul 3 16:31:49 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:31 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-07-03-16.00.html | 16:31 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-07-03-16.00.txt | 16:31 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-07-03-16.00.log.html | 16:31 |
| cgoncalves | slacking | 16:31 |
| xgerman | spatel: when neutron creates the subnet it should inject a dhcp server | 16:32 |
| xgerman | so surprised it did not… I haven’t looked at it in a while but octavia-ansible has tests for that code... | 16:33 |
| spatel | I can see neutron created lb-mgmt-net network and subnet range, also i can see dns service running | 16:34 |
| spatel | but i don't know how neutron dhcp wire up that to my lb-mgmt-net VLAN ? | 16:34 |
| spatel | since i have created RPC v-br-lbaas and v-br-vlan bridge, my br-lbaas stopped pinging :( | 16:36 |
| spatel | look like it totally blocked that interface for me.. | 16:36 |
| johnsom | Yeah, it's probably bad if you mixed OSA and the RPC-octavia stuff. They are two different deployment models. | 16:37 |
| xgerman | yep, especially since roc-octavia is deprecated (was for a time when roc-o was very disticnt form OSA) | 16:38 |
| spatel | johnsom: right now trying to fix this blocking interface issue.. don't know how to fix it.. i did reboot also but my br-lbaas not pingable from other host | 16:38 |
| johnsom | The DHCP stuff is all handled by neutron. The lb-mgmt-net subnet should be "dhcp enabled" in the neutron API. if that is set like it should be, then the neutron agents should have dhcp on the subnet. | 16:39 |
| xgerman | sounds like traffic is somewhere blackholing for him | 16:39 |
| johnsom | It could be that one of the bridges is down causing the DHCP to not make it from the neutron agent to one of the computes | 16:39 |
| xgerman | make sure that the interfaces on the bridges make sense | 16:39 |
| johnsom | And that they are all up, and that the addresses are either on the bridge interface or not as required. | 16:40 |
| xgerman | also what johnsom said — and then there is always the issue if it’s RAX that network ports are not working | 16:40 |
| spatel | I did reboot also controller node and verify v-br-vlan and v-br-lbaas not loaded anywhere but still not able to ping :( | 16:41 |
| johnsom | Have you checked your neutron agents, are they all up and healthy? | 16:42 |
| spatel | Yes neutron is up and running agent is up | 16:42 |
| spatel | i can spin up other vms | 16:42 |
| *** henriqueof has quit IRC | 16:43 | |
| spatel | xgerman: & johnsom look at this.. https://bugs.launchpad.net/openstack-ansible/+bug/1835157 | 16:43 |
| openstack | Launchpad bug 1835157 in openstack-ansible "Octavia strange br-lbaas bridge network issue" [Undecided,Invalid] | 16:43 |
| spatel | i have removed v-br* RPC stuff | 16:43 |
| spatel | mostly system reboot should fix any bridge issue.. | 16:44 |
| johnsom | I don't think that is the case. If the bridges are not configured in a persistent way, they will disappear after a reboot | 16:48 |
| xgerman | Both rpc-octavia and OSA configure them in a persistent way IHMO | 16:51 |
| johnsom | I am pretty sure OSA does. | 16:52 |
| spatel | I didn't configure them presistent way because i was tesing.. all my bridge working fine.. br-host, br-mgmt, br-vxlan etc.. only br-lbaas isn't working | 16:53 |
| spatel | i have removed every single RPC stuff also.. | 16:53 |
| spatel | very very odd issue.. | 16:54 |
| spatel | i am running 400 servers and never seen this kind of issue before.. | 16:54 |
| spatel | nothing in logs also saying why its block, also no activity on tcpdump | 16:54 |
| *** ccamposr has quit IRC | 16:54 | |
| johnsom | So the bridge is up and the STP isn't in blocking? | 16:57 |
| spatel | yes bridge is up and STP is disabled | 16:58 |
| spatel | i can seee in dmesg its saying | 16:59 |
| -spatel- [ 1217.768381] br-lbaas: port 2(d9cb19fb_eth14) entered blocking state | 16:59 | |
| -spatel- [ 1217.768385] br-lbaas: port 2(d9cb19fb_eth14) entered forwarding state | 16:59 | |
| spatel | when i restart network | 16:59 |
| *** ccamposr has joined #openstack-lbaas | 16:59 | |
| *** Vorrtex has quit IRC | 17:00 | |
| *** Vorrtex has joined #openstack-lbaas | 17:01 | |
| spatel | let me poke around.. | 17:01 |
| spatel | more fun.. i did system restart network | 17:02 |
| spatel | now i can ping 1 compute node out of 3 This make no sense.. | 17:03 |
| -spatel- [root@ostack-infra-2-1 network-scripts]# brctl showmacs br-lbaas | 17:04 | |
| -spatel- port nomac addris local?ageing timer | 17:04 | |
| -spatel- 138:ea:a7:33:b9:a8yes 0.00 | 17:04 | |
| -spatel- 138:ea:a7:33:b9:a8yes 0.00 | 17:04 | |
| -spatel- 1e4:11:5b:98:5d:65no 5.89 | 17:04 | |
| -spatel- 2fe:c0:e3:1e:18:f1yes 0.00 | 17:04 | |
| -spatel- 2fe:c0:e3:1e:18:f1yes 0.00 | 17:04 | |
| *** openstackgerrit has quit IRC | 17:04 | |
| spatel | 1e4:11:5b:98:5d:65this mac is pinging.. not other | 17:04 |
| *** Vorrtex has quit IRC | 17:09 | |
| *** ccamposr has quit IRC | 17:32 | |
| *** ramishra has quit IRC | 17:37 | |
| *** luksky11 has quit IRC | 17:46 | |
| *** vishalmanchanda has quit IRC | 18:02 | |
| *** lucashxu has joined #openstack-lbaas | 18:17 | |
| *** gcheresh_ has joined #openstack-lbaas | 18:19 | |
| lucashxu | hi there, I am trying to ssh into a load balancer and see the haproxy log. But the ssh connection can be established. I cannot ping the lb either. I have checked the lb_network_ip, and it is on lb-mgmt-net, having it own subnet. | 18:19 |
| lucashxu | Any idea on what I should do so that I can ssh into the lb created with using a private subnet? Thanks! | 18:20 |
| johnsom | Did you configure the nova keypair for Octavia to use? | 18:20 |
| johnsom | https://docs.openstack.org/octavia/latest/configuration/configref.html#controller_worker.amp_ssh_key_name | 18:21 |
| lucashxu | johnsom, i have changed the octavia.conf, so the keypair points to a public key that I am using locally | 18:21 |
| johnsom | Also, for Train (master) you can setup log offloading: https://docs.openstack.org/octavia/latest/admin/log-offloading.html | 18:21 |
| lucashxu | johnsom: great, thanks for the links! | 18:22 |
| xgerman_ | yeah, awesome write up... | 18:22 |
| johnsom | Thanks! | 18:23 |
| spatel | xgerman: i think something is wrong in OSA based octavia networking... | 18:24 |
| xgerman_ | Possible. I haven’t touched it in over a year | 18:24 |
| spatel | it feel OSA created and wire up br-vlan with lb-mgmt-net and that may be creating loop in br-lbaas which is totally block.. | 18:25 |
| johnsom | They run gate tests on that role, so I would be surprised if it is broken. | 18:25 |
| spatel | i have created br-lbass on other lab box and put them in same VLAN and they are pinning but only these 3 controller not are bricks :( | 18:25 |
| spatel | johnsom: i am 100% sure something is not right in ansible playbooks.. | 18:26 |
| johnsom | Have you asked in the openstack-ansible channel about this? | 18:26 |
| spatel | i haven't and i doubt people will help because hardly anyone deployed octavia | 18:26 |
| spatel | i am going to ask anyway | 18:27 |
| johnsom | lucashxu If you can't connect and the keypair was in place before the load balancer was created, check that you are in a network namespace that can get to the lb-mgmt-net. | 18:27 |
| johnsom | spatel There are multiple people with Octavia deployed via OSA in the openstack-ansible channel. Including the PTL | 18:28 |
| lucashxu | johnsom: yeah, that's something I am trying to resolve. I cannot find a place that I can get to the lb-mgmt-net | 18:28 |
| xgerman_ | +1 | 18:28 |
| spatel | I drop mesg in channel and lets see | 18:29 |
| spatel | lucashxu: is this a bug so i stop wasting my time :( | 18:29 |
| spatel | last 5 days i am banging my head on my desk.. | 18:29 |
| xgerman_ | lucashxu: make also sure the security groups allow port 22 | 18:31 |
| xgerman_ | if keys are configured Octavia will add it automatically but always safe to double check | 18:32 |
| lucashxu | xgerman_: yeah, I have checked the secgroup, port 22 is opened | 18:32 |
| lucashxu | so the lb-mgmt-net should at least be accessible from the controller, correct? Thanks guys for helping :) | 18:33 |
| spatel | lucashxu: are you using OSA ? | 18:38 |
| lucashxu | spatel: nope, i am using the tripleo | 18:39 |
| spatel | lucky you :) | 18:39 |
| spatel | OSA is just painful to get it work :( very few people available for help | 18:40 |
| *** lemko has quit IRC | 18:40 | |
| lucashxu | spatel: :) yeah, tripleO works for me pretty well so far. | 18:41 |
| *** luksky11 has joined #openstack-lbaas | 18:42 | |
| spatel | tripleO is complicated but i found Redhat did very good job with documentation | 18:43 |
| spatel | my 2 cloud running on OSA so i am trying to stick with it but i think i need to find other way soon | 18:44 |
| *** tesseract has quit IRC | 18:44 | |
| lucashxu | good luck :) | 18:45 |
| johnsom | lucashxu As a test, you could get on a neutron agent node, find the network namespace for the lb-mgmt-net network (sudo ip netns), it should be named qdhcp-42537612-5ae8-451e-b1e3-4a2d35c65160 where 42537612-5ae8-451e-b1e3-4a2d35c65160 is the lb-mgmt-net network ID in neutron (openstack subnet list). | 18:50 |
| xgerman_ | johnsom: ’s people did a bunch of Octavia installs with OSA back in the day :-) | 18:50 |
| *** KeithMnemonic has joined #openstack-lbaas | 18:51 | |
| johnsom | lucasxu Then do sudo ip netns exec qdhcp-42537612-5ae8-451e-b1e3-4a2d35c65160 ssh ubuntu@<lb-mgmt-net IP on the nova instance> | 18:51 |
| spatel | xgerman_: i love OSA only problem is getting help out if something broken :( | 18:51 |
| spatel | at present i am all blocked :) | 18:52 |
| johnsom | Here we can help with Octavia, but when it comes to the deployment tools, we can't really track them all. There are at least five supporting Octavia now. | 18:52 |
| lucashxu | johnsom: great, i will give it a try. Really appreciate it | 18:52 |
| *** ajay33 has quit IRC | 19:04 | |
| *** gcheresh_ has quit IRC | 19:10 | |
| *** mithilarun has quit IRC | 19:11 | |
| squarebracket | i'm kind of confused by all the certificates octavia wants... i need to generate two CAs? and then all also some certs from one of the CAs? is that correct? | 19:26 |
| squarebracket | and for a barebones POC, can I get away with just providing a packstack-generated self-signed cert? | 19:26 |
| squarebracket | oh, is it just saying i need a CA on both controller + amphora, such that both controller + amphora can validate the SSL cert it's getting? | 19:28 |
| johnsom | squarebracket There is a certificate guide here: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html | 19:29 |
| squarebracket | johnsom: that is exactly what i'm reading :) | 19:29 |
| johnsom | Octavia uses two-way TLS authentication, so there are two CAs typically. one issues certs for the amphora, one for the controllers | 19:29 |
| squarebracket | johnsom: i had assumed that the controllers would spawn an amphora, then the amphora would send a CSR to the controller which would sign a request, and then send back the valid cert. but i guess here, amphora is generating its own certs? | 19:32 |
| johnsom | squarebracket No, all of the csr/cert creation is done on the controllers. They get installed in the amphora on boot via config-drive | 19:33 |
| johnsom | There is no CA inside the amps | 19:33 |
| squarebracket | ok, that's what i had assumed. so then why the need for two CAs? doesn't only the controller need a CA cert (since it's the only one signing certs)? | 19:34 |
| johnsom | Well, as the document mentions, if you used the same CA, amps with their cert/private key could sign controller certs and pretend to be a controller | 19:35 |
| johnsom | The controllers need the CA cert to verify the amp certs, the amps need the controller CA cert to verify the controller certificates. | 19:36 |
| *** mithilarun has joined #openstack-lbaas | 19:36 | |
| johnsom | It is a bit complicated. Technically, if security isn't your top priority, you can use just one CA | 19:37 |
| squarebracket | oh, hah, my eyes skipped over the NOTE section >_> | 19:37 |
| squarebracket | right, i got it now. the controller signs all certs, but two different kinds of certs -- one for the amps, and one for itself for the "other end" of the two-way tls, which is itself. | 19:38 |
| johnsom | Yep | 19:39 |
| squarebracket | and just so i understand fully, the client CA isn't used to generate certs/keys for the controllers right? since the instructions also include CSR/cert generation. it's just needed since the amp needs the CA to verify the connection? | 19:51 |
| *** lemko has joined #openstack-lbaas | 19:53 | |
| johnsom | So, this is another tricky thing. The "server" is the amp as that is what we connect to for establishing the TLS handshake. The "client" is the controller. The Client CA public cert is installed in the amphora so they can validate the controller certificates. | 20:05 |
| *** goldyfruit has quit IRC | 20:24 | |
| *** goldyfruit has joined #openstack-lbaas | 20:24 | |
| *** gcheresh_ has joined #openstack-lbaas | 20:26 | |
| *** lucashxu has quit IRC | 20:28 | |
| *** gcheresh_ has quit IRC | 20:38 | |
| spatel | johnsom: no kidding.. | 20:39 |
| -spatel- 2019 Jun 28 10:34:30 swt-tor1-010101-1-3-hd %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port port-channel13 on VLAN0027. | 20:39 | |
| -spatel- 2019 Jun 28 10:44:16 swt-tor1-010101-1-3-hd %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port port-channel23 on VLAN0027. | 20:39 | |
| -spatel- 2019 Jun 28 10:48:13 swt-tor1-010101-1-3-hd %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port port-channel33 on VLAN0027. | 20:39 | |
| -spatel- 2019 Jun 30 22:23:29 swt-tor1-010101-1-3-hd %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port port-channel34 on VLAN0027. | 20:39 | |
| -spatel- 2019 Jul 2 22:25:34 swt-tor1-010101-1-3-hd %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port port-channel24 on VLAN0027. | 20:39 | |
| -spatel- 2019 Jul 2 22:26:36 swt-tor1-010101-1-3-hd %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port port-channel14 on VLAN0027. | 20:39 | |
| johnsom | spatel Yep, looks like you have a loop. I wonder if you are bridging off multiple controllers for the lb-mgmt-net and don't have stp enabled on the bridges | 20:41 |
| spatel | This is horrible, it may bring down my whole datacenter.. | 20:41 |
| johnsom | I.e. popping the neutron lb-mgmt-net off onto a VLAN multiple places that are interconnected | 20:41 |
| johnsom | ha, well, you have bigger problems if that is the case | 20:42 |
| spatel | good i have all STP protection enabled on switches so good it blocked ports | 20:42 |
| spatel | I belive that RPC document created loop | 20:42 |
| johnsom | Yeah, I would certainly expect that. | 20:43 |
| *** trident has quit IRC | 20:43 | |
| spatel | because it was linking br-lbass ---> br-vlan--->lb-mgmt-net | 20:43 |
| spatel | let me start over with fresh config and see if i get some positive result | 20:44 |
| *** trident has joined #openstack-lbaas | 20:45 | |
| *** goldyfruit has quit IRC | 20:47 | |
| *** goldyfruit has joined #openstack-lbaas | 21:02 | |
| xgerman_ | Yeah, that’s wrong. We use a vlan for the provider net. Then we use br-lbaas to get the tagged vlan untagged. Alternatively, you can connect a tagged port to the container as well (though that’s not automated) | 21:13 |
| *** mithilarun has quit IRC | 21:14 | |
| colin- | scary stuff! | 21:15 |
| colin- | nice that they switched to blocking fast though | 21:16 |
| *** pcaruana has quit IRC | 21:16 | |
| spatel | xgerman_: let me first get out of this mess.. don't know how to remove loop from those switch | 21:22 |
| *** openstackgerrit has joined #openstack-lbaas | 21:31 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Fix IPv6 tests if ipv6-private-subnet is stateless https://review.opendev.org/668996 | 21:31 |
| *** mithilarun has joined #openstack-lbaas | 21:33 | |
| *** boden has quit IRC | 21:34 | |
| spatel | xgerman_: how do i tell octavia to stop re-creating amphora ? | 21:35 |
| spatel | I am seeing in logs it creating and destroying amphora vms | 21:35 |
| xgerman_ | yeah, that’s an indication the managment net is not working | 21:36 |
| xgerman_ | you cna try to delete the LB from the CLI and if that does not work delete it from the DB | 21:36 |
| johnsom | Don't delete it from the DB | 21:37 |
| johnsom | If you want to stop it temporarily, shutdown your health manager and housekeeping processes. (gracefully as always) | 21:38 |
| spatel | i did delete LB from GUI but i can see its re-creating vms | 21:38 |
| johnsom | Do you have the spares pool enabled? | 21:38 |
| spatel | what is that ? | 21:38 |
| johnsom | https://docs.openstack.org/octavia/latest/configuration/configref.html#house_keeping.spare_amphora_pool_size | 21:38 |
| johnsom | If that is greater than zero, the controllers are trying to make sure there are spare amphora booted up. | 21:39 |
| spatel | i should set that 0 right? | 21:40 |
| spatel | let me first shutdown housekeeping process | 21:41 |
| johnsom | Yes, it should be zero, then restart the housekeeping process | 21:41 |
| spatel | and later i will make my way out | 21:41 |
| squarebracket | if i get an UnknownConnectionError with 'No connection adapters were found for...' from octavia.compute.drivers.nova_driver does that mean the nova config isn't correct? | 21:42 |
| spatel | damn it 5:49PM gotta go i will rsync with you guys again | 21:50 |
| spatel | have a great weekend and 4th july.. | 21:50 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Fix IPv6 tests if ipv6-private-subnet is stateless https://review.opendev.org/668996 | 21:50 |
| *** mithilarun has quit IRC | 21:52 | |
| *** mithilarun has joined #openstack-lbaas | 21:54 | |
| *** lemko has quit IRC | 22:00 | |
| squarebracket | n/m, it was various endpoint misconfigs | 22:02 |
| *** rcernin has quit IRC | 22:04 | |
| *** spatel has quit IRC | 22:05 | |
| *** ccamposr has joined #openstack-lbaas | 22:09 | |
| squarebracket | huh, now it thinks the security group doesn't exist, but it does.... | 22:27 |
| *** yamamoto has joined #openstack-lbaas | 22:45 | |
| *** mithilarun has quit IRC | 22:45 | |
| *** luksky11 has quit IRC | 22:48 | |
| *** tkajinam has joined #openstack-lbaas | 22:54 | |
| *** spatel has joined #openstack-lbaas | 23:06 | |
| *** rcernin has joined #openstack-lbaas | 23:09 | |
| openstackgerrit | Noboru Iwamatsu proposed openstack/octavia master: Add failover logging to show the amphora details. https://review.opendev.org/667316 | 23:29 |
| *** yamamoto has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!