| johnsom | Ok, I have fixed a bunch of stuff. It was my mistake that it wasn't passing traffic, I made a bad suggestion on the OPS change. I'm very sorry for that. I have fixed the functionality, but we can come back later and re-add the OPS session persistence option. It's a rare setting anyway now that I re-read it's purpose | 00:18 |
|---|---|---|
| johnsom | Failover of a pure UDP LB seems to be working for me. I will test a multiple-listener and a mixed one next. | 00:19 |
| johnsom | Then I will build a centos image and see if it was just my issue that was causing it to not work | 00:19 |
| bzhao__ | cgoncalves: Hi , have you test with the new session persistence options? | 00:22 |
| johnsom | bzhao__ Hi | 00:23 |
| johnsom | bzhao__ I made a mistake and I am sorry. The OPS change I requested was incorrect. I have fixed it to be functioning in my patch, but we should later put the options back for session persistence setting. | 00:24 |
| bzhao__ | johnsom: Hi, Micheal. Sorry for not online yesterday, just as the irc identify, my other account limited by china network. Can not login. ;-( | 00:24 |
| bzhao__ | johnsom: Never mind. ;-) | 00:24 |
| johnsom | bzhao__ Bummer you had IRC problems | 00:24 |
| bzhao__ | johnsom: I will add them later. ;-) | 00:25 |
| johnsom | bzhao__ I have a few more things to test, then I will post my patch and review the last needed patches. | 00:25 |
| johnsom | bzhao__ I think we are in good shape to get it into Rocky | 00:25 |
| johnsom | It is close, but it is working for me, just some cleanup stuff | 00:26 |
| bzhao__ | johnsom: OK, but I have a question, hmm, I didn't hit the error when failover error, when I reboot the vm/ delete it..Could you please show some details? ;-) | 00:26 |
| bzhao__ | johnsom: You help so much for me. Many thanks. | 00:27 |
| johnsom | bzhao__ See this: https://review.openstack.org/#/c/587690/17/octavia/amphorae/backends/agent/api_server/templates/keepalived.systemd.j2 | 00:27 |
| johnsom | bzhao__ If we didn't have an haproxy listener there was no systemd "need" to start the netns service | 00:28 |
| bzhao__ | johnsom: Yeah, I saw it, I tested it with the latest code yesterday. There is no error again.As I add pdb during the flow update the amphora lisnters tasks. ;-). The flow works fine, but after the failover flow finished, there is no RS configured successful in the amp ns, even though the config file is correct, I think it lacks a restart step. I mean the test above are in the previous order.. | 00:33 |
| bzhao__ | johnsom: Thanks for explain. ;-) | 00:34 |
| johnsom | bzhao__ Yeah, I looked at the flow order thing. It's because listener update is restarting keepalived and it has a requirement on the netns being present, but we really start those processes later in the flow. It needs a refactor, but I think the proposed flow order is ok at the moment. | 00:35 |
| bzhao__ | johnsom: I will focus on centos thing, after finish let OPS back. | 00:35 |
| johnsom | Hmmm, getting this with a mixed (2 UDP, 1 HAProxy) LB failover: ERROR octavia.controller.worker.tasks.amphora_driver_tasks [None req-073b09b4-7989-4fdf-a151-be7a8e81d44e None None] Failed to update listeners on amphora 7d414c67-2923-47f1-af48-1676fb47c6d5. Skipping this amphora as it is failing to update due to: _action() takes at least 5 arguments (5 given) | 00:35 |
| bzhao__ | johnsom: Yeah, it seems OK for now. | 00:35 |
| johnsom | bzhao__ Yeah, I am going to try centos tonight too. | 00:36 |
| johnsom | Need to figure out this error first | 00:36 |
| bzhao__ | johnsom: Ha, fight all day. ;-). Fight Rocky. I will test it you mentioned, as there is already a test env. Also the traceback look odd.."_action() takes at least 5 arguments (5 given) " ;-) | 00:39 |
| johnsom | Yeah, I think the add of protocol to that _action method is not complete | 00:39 |
| *** longkb has joined #openstack-lbaas | 00:42 | |
| bzhao__ | Oh, my bad.. The update_amphora_listeners may lack the protocol parameter during reload haproxy listener, I think. | 00:44 |
| johnsom | Yeah, I think I would have implemented this differently so we aren't sending the protocol | 00:45 |
| johnsom | We could have just searched for the configuration file in the amp to determine what type it was since we know the paths and know the filename formats. | 00:47 |
| johnsom | If the file exists in the haproxy path, it is TCP, if it exist in UDP path it is UDP, if none, 404 | 00:48 |
| bzhao__ | johnsom: loop the listeners on agent side with the same API url, check it is a UDP/haproxy lisnter on agent side? ;-) | 00:48 |
| bzhao__ | johnsom: Yeah, that's it. | 00:48 |
| johnsom | It's really just two os.path_exists calls, one for TCP, one for UDP path | 00:49 |
| johnsom | That way the API stays the same for those actions | 00:50 |
| johnsom | bzhao__ Yeah, ok, I have it. I will update in my patch | 00:51 |
| bzhao__ | Agree, but I think we need to leave only 1 client API for udp. "upload_udp_config", as the haproxy one url ends with "haproxy". ;-) | 00:51 |
| bzhao__ | johnsom: May I add the OPS option in the followup patch? ;-) | 00:52 |
| johnsom | bzhao__ Yes, go ahead and work on OPS. I think I am only going to do the actions, I will leave get/put UDP config, that is fine for now. The actions changes make be nervous as they are slightly complex. | 00:53 |
| *** hongbin has joined #openstack-lbaas | 00:53 | |
| bzhao__ | johnsom: Thanks. A refactor is comming. ;-) | 00:55 |
| *** hongbin has quit IRC | 01:13 | |
| *** hongbin has joined #openstack-lbaas | 01:13 | |
| *** bbbbzhao_ has joined #openstack-lbaas | 01:38 | |
| *** openstackgerrit has joined #openstack-lbaas | 02:08 | |
| openstackgerrit | Tatsuma Matsuki proposed openstack/octavia master: Separate the thread pool for health and stats update https://review.openstack.org/581585 | 02:08 |
| *** ramishra has joined #openstack-lbaas | 02:36 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Followup patch for UDP support https://review.openstack.org/587690 | 03:10 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: [UDP] Fix failed member always in DRAIN status https://review.openstack.org/588511 | 03:11 |
| bzhao__ | johnsom: Thanks, Michael. I think there must be very late. After I finish the OPS back and testing in local. I will check the centos staff and wait for your back tommorrow. Take a good rest. | 03:18 |
| johnsom | bzhao__ Only 8pm, so ok. I an testing my patch for failover and then will test CentOS as well. Then call it a night. | 03:19 |
| bzhao__ | johnsom: Ok, ha. | 03:20 |
| *** hongbin has quit IRC | 03:51 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Followup patch for UDP support https://review.openstack.org/587690 | 03:55 |
| johnsom | Ok, fixed the release note. That patch does fix the failover error I saw | 03:55 |
| johnsom | Yeah, looks like I fixed centos as well. I can get UDP through a centos 7 vm | 04:15 |
| johnsom | lol anaconda log. Man I spent some time with that in my redhat years | 04:17 |
| johnsom | Yeah, reboot comes up fine with UDP only. I rebooted it, then deleted the pair amp, still have UDP connections | 04:21 |
| johnsom | Yeah, HM failover works too. Ship it! | 04:23 |
| johnsom | grin | 04:23 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: [UDP] Fix failed member always in DRAIN status https://review.openstack.org/588511 | 04:24 |
| bbbbzhao_ | johnsom: Woo,nice. The OPS is nearly back. But not test in local. I'm leaving for lunch | 04:25 |
| johnsom | Ok. Yeah, I think we are good. Just need to get cores to review and merge | 04:25 |
| bbbbzhao_ | johnsom: Thank you :) | 04:26 |
| *** yamamoto has joined #openstack-lbaas | 04:29 | |
| xgerman_ | Ok. Will look in a bit and review | 04:34 |
| johnsom | Morning | 04:35 |
| johnsom | Well, it looks like Jacky was already on the first two | 04:36 |
| johnsom | Cores, can you please review https://review.openstack.org/#/c/587690 and https://review.openstack.org/#/c/588511/ and merge if you are comfortable with them. | 04:37 |
| *** yamamoto has quit IRC | 04:46 | |
| *** yamamoto has joined #openstack-lbaas | 04:50 | |
| bbbbzhao_ | So quick...I have not finish my lunch.. | 04:51 |
| bbbbzhao_ | :). Thanks very much | 04:52 |
| *** kobis1 has joined #openstack-lbaas | 05:06 | |
| *** kobis1 has quit IRC | 05:20 | |
| xgerman_ | looking… | 05:29 |
| *** yamamoto has quit IRC | 05:49 | |
| bzhao__ | Maybe I should not post a new for followup?..move it to the end of patch list | 05:52 |
| openstackgerrit | Merged openstack/octavia master: UDP for [2] https://review.openstack.org/529651 | 05:57 |
| johnsom | bzhao__: yeah, add to the end please so we can merge some patches | 05:58 |
| openstackgerrit | Merged openstack/octavia master: UDP for [3][5][6] https://review.openstack.org/539391 | 05:58 |
| bzhao__ | johnsom: Yeah. Thanks. I think I need to add after the [UDP] Fix failed member always in DRAIN status . | 06:00 |
| johnsom | Yes, that would be fine | 06:01 |
| *** kobis1 has joined #openstack-lbaas | 06:18 | |
| *** pcaruana has joined #openstack-lbaas | 06:27 | |
| *** luksky has joined #openstack-lbaas | 06:59 | |
| cgoncalves | bzhao__, hi. I have not yet because I still couldn't make to have the UDP socket open | 07:24 |
| *** nmagnezi_ has joined #openstack-lbaas | 07:25 | |
| bzhao__ | cgoncalves: Thanks for feedback,;-). I think I can get some time for centos tonight. Also johnsom had fixed the Upper UDP for centos. According to his test, it may work on centos. | 07:27 |
| cgoncalves | bzhao__, yes, the upper-case UDP and mapping netcat package to red hat family. I will stack again with latest patch sets now | 07:27 |
| bzhao__ | cgoncalves: Thanks . But I think the other tests still need to be done on centos. Ha, let's play it on centos first. ;-) | 07:29 |
| *** celebdor has joined #openstack-lbaas | 07:30 | |
| bzhao__ | cgoncalves: I will join after the refactor.. | 07:30 |
| cgoncalves | cool, thanks! | 07:32 |
| *** rcernin has quit IRC | 07:35 | |
| *** velizarx has joined #openstack-lbaas | 07:38 | |
| *** abaindur has quit IRC | 07:47 | |
| openstackgerrit | huangshan proposed openstack/octavia master: Add a periodic task for checking pending_* lbs in housekeeping https://review.openstack.org/589747 | 07:55 |
| *** velizarx has quit IRC | 07:56 | |
| *** velizarx has joined #openstack-lbaas | 08:01 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: [UDP] Bring back new session_persistence type "OPS" https://review.openstack.org/589748 | 08:01 |
| *** nmagnezi_ has quit IRC | 08:05 | |
| *** nmagnezi_ has joined #openstack-lbaas | 08:07 | |
| *** ktibi has joined #openstack-lbaas | 08:24 | |
| *** nmagnezi_ has quit IRC | 08:37 | |
| openstackgerrit | zhouchangxun proposed openstack/octavia master: Change the driver to a singleton https://review.openstack.org/589100 | 08:53 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: [UDP] Update amphora agent api ref https://review.openstack.org/588893 | 08:58 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: [UDP] Support HTTP GET and TCP check in udp healthmonitor https://review.openstack.org/589180 | 08:58 |
| *** salmankhan has joined #openstack-lbaas | 09:17 | |
| bzhao__ | My test env is nearly dead as 2 centos amp lauched..;-) | 09:21 |
| *** ktibi has quit IRC | 09:23 | |
| *** ktibi has joined #openstack-lbaas | 09:27 | |
| *** ktibi has quit IRC | 09:30 | |
| *** ktibi has joined #openstack-lbaas | 09:31 | |
| *** nmagnezi_ has joined #openstack-lbaas | 09:34 | |
| cgoncalves | it works!! | 09:35 |
| bzhao__ | yeah. Same . ;-) | 09:35 |
| cgoncalves | http://paste.openstack.org/show/727615/ | 09:35 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/6vRF618g/ | 09:37 |
| bbbbzhao_ | LOL | 09:37 |
| cgoncalves | great, also works for you | 09:39 |
| bzhao__ | ha. yeah. | 09:41 |
| bzhao__ | It is necessary to test more. I'm testing failover now. | 09:42 |
| *** luksky has quit IRC | 09:44 | |
| bzhao__ | Connection is OK if I delete the master amp. Great. | 09:45 |
| cgoncalves | testing failover too (standalone amp topology) | 09:47 |
| *** kobis1 has quit IRC | 09:48 | |
| bzhao__ | ;-) . Test in ACTIVE STANDBY topology. | 09:48 |
| cgoncalves | and just like that, from one day to the other, everything seem to work :) | 09:49 |
| cgoncalves | failover worked fine for me | 09:49 |
| bzhao__ | Ha, johnsom 's magic. ;-) | 09:49 |
| cgoncalves | yours too!! | 09:49 |
| bzhao__ | Cool, my test env is slow. still waiting for connection the rebuid amp. | 09:50 |
| bzhao__ | Oh yeah. Works!!! haha | 09:56 |
| openstackgerrit | huangshan proposed openstack/octavia master: Add a periodic task for checking pending_* lbs in housekeeping https://review.openstack.org/589747 | 10:01 |
| cgoncalves | awesome! | 10:03 |
| cgoncalves | pity that we are past feature freeze, otherwise we could have also made to support udp in heat | 10:04 |
| cgoncalves | I have a draft patch, only missing unit tests | 10:04 |
| *** luksky has joined #openstack-lbaas | 10:16 | |
| bzhao__ | Thanks for your great work. Apologize for the late work, that's my bad for the so big patch and not good for review. Then may make this feature delay. ;-( | 10:17 |
| openstackgerrit | huangshan proposed openstack/octavia master: Add a periodic task for checking pending_* lbs in housekeeping https://review.openstack.org/589747 | 10:20 |
| bzhao__ | https://review.openstack.org/#/c/587690/19/elements/amphora-agent/pkg-map johnsom Woo | 10:23 |
| *** kobis1 has joined #openstack-lbaas | 10:28 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: Remove user_group option https://review.openstack.org/589408 | 10:32 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: [UDP] Support HTTP GET and TCP check in udp healthmonitor https://review.openstack.org/589180 | 10:40 |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: DNM: Leave VIP NIC plugging for keepalived https://review.openstack.org/589292 | 10:44 |
| openstackgerrit | Merged openstack/octavia master: Followup patch for UDP support https://review.openstack.org/587690 | 11:21 |
| openstackgerrit | Merged openstack/octavia master: [UDP] Fix failed member always in DRAIN status https://review.openstack.org/588511 | 11:21 |
| bzhao__ | UDP-CONNECT Healthmonitor works also, ;-) | 11:37 |
| bzhao__ | Mixed case passed in failover on centos. | 11:50 |
| *** nmagnezi_ has quit IRC | 11:59 | |
| cgoncalves | \o/ | 12:17 |
| *** longkb has quit IRC | 12:20 | |
| *** amuller has joined #openstack-lbaas | 12:24 | |
| cgoncalves | confirmed, manual LB failover on act-standby topology with UDP listener works | 12:30 |
| *** nmagnezi_ has joined #openstack-lbaas | 12:33 | |
| bbbbzhao_ | yeah, excited. ;) Leaving for home | 12:43 |
| *** velizarx has quit IRC | 12:52 | |
| *** velizarx has joined #openstack-lbaas | 12:56 | |
| *** celebdor1 has joined #openstack-lbaas | 14:05 | |
| *** celebdor has quit IRC | 14:07 | |
| *** hongbin has joined #openstack-lbaas | 14:18 | |
| *** HW-Peter has joined #openstack-lbaas | 14:21 | |
| *** fnaval has joined #openstack-lbaas | 14:27 | |
| *** kobis1 has quit IRC | 14:56 | |
| *** nmagnezi_ has quit IRC | 15:12 | |
| xgerman_ | johnsom: this is weird http://logs.openstack.org/59/589259/4/check/openstack-ansible-functional-ubuntu-xenial/8f0c972/job-output.txt.gz | 15:16 |
| xgerman_ | http://logs.openstack.org/59/589259/4/check/openstack-ansible-functional-ubuntu-xenial/8f0c972/logs/openstack/octavia1/octavia/octavia-worker.log.txt.gz#_2018-08-08_14_24_42_745 | 15:17 |
| xgerman_ | there must be some incompatibility between worker and lastest diskimage | 15:18 |
| johnsom | Or you caught it mid-merge of the UDP chain..... | 15:18 |
| johnsom | That was one thing I fixed in the followup was reverting some incompatible changes to the amp api | 15:19 |
| *** velizarx has quit IRC | 15:20 | |
| johnsom | Yeah, looks like you had a patch 2 amp with a followup patch octavia where I reverted that change | 15:21 |
| xgerman_ | makes sense… so wait until tomorrow when we have. a new amp :-) | 15:21 |
| xgerman_ | or can we trigger the amp job manually? | 15:22 |
| johnsom | Good question, it would be nice if we could | 15:22 |
| *** luksky has quit IRC | 15:24 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: [UDP] Support HTTP GET and TCP check in udp healthmonitor https://review.openstack.org/589180 | 15:25 |
| *** celebdor1 has quit IRC | 15:25 | |
| *** celebdor1 has joined #openstack-lbaas | 15:31 | |
| *** pcaruana has quit IRC | 15:34 | |
| *** aojea has joined #openstack-lbaas | 15:51 | |
| *** jlaffaye_ is now known as jlaffaye | 16:23 | |
| *** celebdor1 has quit IRC | 16:25 | |
| *** ktibi has quit IRC | 16:27 | |
| *** nmagnezi_ has joined #openstack-lbaas | 16:39 | |
| *** ramishra has quit IRC | 17:27 | |
| *** salmankhan has quit IRC | 18:13 | |
| *** abaindur has joined #openstack-lbaas | 18:25 | |
| *** amuller has quit IRC | 18:36 | |
| *** sapd has quit IRC | 18:49 | |
| johnsom | #startmeeting Octavia | 20:00 |
| openstack | Meeting started Wed Aug 8 20:00:04 2018 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 20:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 20:00 |
| openstack | The meeting name has been set to 'octavia' | 20:00 |
| johnsom | Hi folks! | 20:00 |
| xgerman_ | o/ | 20:00 |
| nmagnezi_ | o/ | 20:00 |
| cgoncalves | hey-hey! | 20:00 |
| johnsom | Happy Rocky RC1 day | 20:00 |
| colin- | o/ | 20:00 |
| johnsom | First up, just a note, I am going to be on vacation Thursday through Monday and will have limited computer access | 20:01 |
| johnsom | We have relatives visiting and are heading to the coast. | 20:01 |
| johnsom | As I mentioned, today is RC1 day where we cut the stable/rocky branch. | 20:01 |
| johnsom | I think we are in good shape. | 20:02 |
| cgoncalves | well deserved, have fun! | 20:02 |
| johnsom | Thank you to everyone that worked some long hours to get UDP in shape | 20:02 |
| nmagnezi_ | Indeed | 20:02 |
| nmagnezi_ | Have fun! | 20:02 |
| johnsom | The core UDP patches have merged, so it made it into Rocky | 20:02 |
| johnsom | Unless there is a problem with it, I would like to see this one make it too: | 20:03 |
| johnsom | #link https://review.openstack.org/587255 | 20:03 |
| johnsom | So cores, if you haven't already reviewed, it would be appreciated. | 20:03 |
| nmagnezi_ | Yup, I'll review it right after the meeting | 20:03 |
| johnsom | Then after that I will cut RC1 | 20:03 |
| cgoncalves | can we also tag stable/queens? :D | 20:04 |
| johnsom | From here on if there is a bug fix you think we need in Rocky, please contact me with the link and add it to the priority list: | 20:04 |
| xgerman_ | https://review.openstack.org/#/c/587505/ | 20:04 |
| johnsom | #link https://etherpad.openstack.org/p/octavia-priority-reviews | 20:04 |
| xgerman_ | https://review.openstack.org/#/c/585864/ | 20:04 |
| xgerman_ | those two | 20:04 |
| johnsom | This will determine if we do an RC2 next week | 20:04 |
| johnsom | Also, the Denver Stein PTG etherpad is up: | 20:05 |
| johnsom | #link https://etherpad.openstack.org/p/octavia-stein-ptg | 20:05 |
| johnsom | Please add any topics you think we need to discuss to the topics list and if you are able to join us. | 20:06 |
| johnsom | Any other announcements today? | 20:06 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:07 |
| xgerman_ | I will be gone Monday for two weeks | 20:07 |
| cgoncalves | yes. you get 6 more months of PTLing :) | 20:07 |
| johnsom | Nice, enjoy the time off | 20:07 |
| cgoncalves | congrats | 20:07 |
| xgerman_ | 4 more years! | 20:07 |
| johnsom | Ha, well, yes. I announced that last week. | 20:08 |
| johnsom | Thanks? | 20:08 |
| johnsom | grin | 20:08 |
| nmagnezi_ | ha | 20:08 |
| nmagnezi_ | Mm I filed this | 20:08 |
| nmagnezi_ | #link https://storyboard.openstack.org/#!/story/2003309 | 20:08 |
| nmagnezi_ | Will be happy to some feedback | 20:08 |
| nmagnezi_ | Tried to fix it, but wasn't able to make systemd to respawn the amphora-agent | 20:09 |
| johnsom | Over the last week I have been busy helping out with the UDP protocol support and tracking down a barbican client bug that impacts us. Summary there is if the cloud is using admin or internal endpoints barbican client will fail to get the secret. | 20:09 |
| johnsom | nmagnezi Ok thanks. I will take a look. I had a question about your keepalived systemd story/patch as well. It's setup different than all of the other service definitions I have found, so wanted to understand more there | 20:10 |
| johnsom | This is the barbican client bug if you are interested: | 20:11 |
| johnsom | #link https://storyboard.openstack.org/#!/story/2003197 | 20:11 |
| johnsom | Any other progress updates? | 20:12 |
| johnsom | Ok | 20:13 |
| johnsom | #topic Some progress on the Storyboard issues | 20:13 |
| *** luksky has joined #openstack-lbaas | 20:13 | |
| johnsom | #link https://etherpad.openstack.org/p/storyboard-issues | 20:13 |
| cgoncalves | changes to CI jobs and housekeeping. thanks to johnsom for helping me with centos-based controller | 20:13 |
| johnsom | nmagnezi_ cgoncalves The storyboard team has fixed a few things. They had an intern that worked on some stuff | 20:14 |
| johnsom | Thought I would mention that there has been progress there. | 20:14 |
| nmagnezi_ | Thanks a lot for the followup! | 20:15 |
| johnsom | One nice thing is the link to our project is no longer a number (though that still works) | 20:15 |
| johnsom | At some point we should update our links, but since the number still works it's not a priority | 20:15 |
| cgoncalves | #link https://storyboard.openstack.org/#!/project/openstack/octavia | 20:15 |
| johnsom | Yep | 20:16 |
| nmagnezi_ | Nice | 20:16 |
| johnsom | Oh, darn, I forgot to remove the "UDP doesn't work on CentOS" release note. It worked for me, are you guys good with it? If so I will post a patch real quick to fix that . | 20:17 |
| cgoncalves | still no priority/severity fields :/ | 20:17 |
| cgoncalves | or which project version is affeted | 20:17 |
| cgoncalves | johnsom, you added a release note? I didn't see that! :D | 20:18 |
| nmagnezi_ | johnsom, would be nice to have it that fix if that's not too much trouble | 20:18 |
| cgoncalves | I tested today with latest patch chain. I reported my findings here on the channel this morning (CET). TL;DR: it works! | 20:18 |
| johnsom | Sure, no problem. nmagnezi_ if you can review the fix right after the meeting it would be great. | 20:18 |
| nmagnezi_ | Sure can | 20:18 |
| johnsom | Yeah, ok I will ping you guys when I have a patch up. I saw a typo in the api-ref too I can fix | 20:19 |
| cgoncalves | also tested failover on standalone and act/stby UDP-listener LB -- works! | 20:19 |
| johnsom | #topic Open Discussion | 20:19 |
| johnsom | Other topics for today? | 20:19 |
| johnsom | colin- Welcome BTW! Not sure if you have joined a meeting before. | 20:19 |
| johnsom | No other topics today? | 20:21 |
| colin- | thanks, first time :) | 20:22 |
| johnsom | This is the open discussion part of the meeting, so if you have any questions/comments for the team it's a good time to get attention. grin | 20:22 |
| xgerman_ | or tell us which bug you like to work on :-) | 20:22 |
| colin- | thanks for the support thus far, hope to return the favor that's all | 20:23 |
| johnsom | Nice, appreciated. | 20:23 |
| johnsom | Ok, well, if we don't have any more topics we can wrap up early this week. | 20:23 |
| nmagnezi_ | we have stuff to review | 20:24 |
| nmagnezi_ | :) | 20:24 |
| johnsom | Thanks again for all of your work on Rocky. We closed out 66 "priority" patches! | 20:24 |
| johnsom | #endmeeting | 20:24 |
| openstack | Meeting ended Wed Aug 8 20:24:44 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:24 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-08-08-20.00.html | 20:24 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-08-08-20.00.txt | 20:24 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-08-08-20.00.log.html | 20:24 |
| *** SumitNaiksatam has joined #openstack-lbaas | 20:25 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix UDP release note for CentOS https://review.openstack.org/590031 | 20:25 |
| johnsom | nmagnezi_ xgerman_ ^^^ release notes fix | 20:25 |
| nmagnezi_ | johnsom, reviewed. Looking at the delete amps patch now | 20:26 |
| johnsom | Thank you! | 20:26 |
| johnsom | Funny, that integer type for the protocol must have been there a long time.... I just noticed it today while looking at the api-ref for the UDP updates | 20:29 |
| nmagnezi_ | johnsom, should we be backwards compatible here? :P | 20:30 |
| nmagnezi_ | johnsom, your time to grin :) | 20:30 |
| johnsom | lol | 20:31 |
| johnsom | Well, given actually using an integer would not get you far.... grin | 20:31 |
| johnsom | You just want to open the "Octavia number registery" don't you | 20:32 |
| nmagnezi_ | johnsom, just wanted to bump the minor API version so we'll be at 2.2 :D | 20:33 |
| johnsom | nmagnezi_ We are at 2.2 now | 20:33 |
| nmagnezi_ | Well I should be kicked out of here for this | 20:33 |
| nmagnezi_ | (Or I can just say it way too late for me) | 20:33 |
| johnsom | UDP was 2.1 | 20:34 |
| johnsom | opps, 2.2 | 20:34 |
| xgerman_ | +1 | 20:34 |
| xgerman_ | yep, I was wondering if we should make our tests/infrastructure better so we just need to run some const array and add versions | 20:34 |
| johnsom | I kind of did that to make it easier to add versions, but maybe there is more improvement that could be added | 20:35 |
| johnsom | cgoncalves BTW, I want to get dual down amphora failover into stable/queens and then will cut. | 20:36 |
| johnsom | The backport has a problem I need to figure out | 20:36 |
| xgerman_ | yeah, need to mull a bit more and put that in | 20:36 |
| johnsom | We should set a goal of next week for a stable/queens release | 20:37 |
| nmagnezi_ | johnsom, +2 W+1 https://review.openstack.org/#/c/587255 | 20:38 |
| johnsom | Thank you! | 20:39 |
| cgoncalves | johnsom, ok. I just feel that we've been constantly delaying the cut ;) | 20:40 |
| johnsom | Yeah, trying to get important fixes in it | 20:40 |
| xgerman_ | +1 | 20:40 |
| cgoncalves | we can release as often as we want, no? | 20:40 |
| johnsom | Yes, but it kind of sucks to do a stable release when you know there is a nasty bug in the review stage | 20:41 |
| johnsom | People packaging probably don't want to make octavia packages every week... | 20:42 |
| cgoncalves | on our side, we have automation tools ;) | 20:42 |
| cgoncalves | RDO master \o/ | 20:42 |
| johnsom | Some days I wish I could automate reviews.... | 20:43 |
| xgerman_ | just get some bots | 20:43 |
| xgerman_ | core-bot1 and 2 | 20:43 |
| johnsom | Ha, yeah | 20:43 |
| johnsom | Though in honesty, I appreciate the second set of eyes on stuff | 20:43 |
| xgerman_ | hence core-bote-2 | 20:44 |
| cgoncalves | that's what core-bot2 would be for -- second set of "eyes" | 20:44 |
| cgoncalves | xgerman_, *hi5* | 20:44 |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: Remove user_group option https://review.openstack.org/589408 | 20:45 |
| abaindur | hi johnsom: in what version of openstacl client does the lbaas cli exist? | 20:53 |
| johnsom | abaindur For Octavia you install the python-octaviaclient plugin. If you are using neutron-lbaas you have to use the "neutron" command, it is not supported via OpenStack client (because it is deprecated). | 20:54 |
| abaindur | we have installed the python-openstackclient from the upperconstraints version for queens, but it cant find loadbalancer clis | 20:54 |
| abaindur | oh, octaviaclient | 20:54 |
| abaindur | No... we are not using neutron lbaas | 20:54 |
| johnsom | Yeah, most of the services have OpenStack client plugins now | 20:54 |
| abaindur | this is Octavia directly | 20:54 |
| johnsom | Yep, perfect, install python-octaviaclient along with OpenStack client and the "openstack loadbalancer" commands will become available | 20:55 |
| johnsom | log out/back in to get command completion | 20:55 |
| xgerman_ | nmagnezi_: johnsom we probably need to merge the lbaas proxy-gate stuff as well | 21:03 |
| xgerman_ | https://review.openstack.org/#/c/539350/ | 21:04 |
| johnsom | Links? | 21:04 |
| xgerman_ | see above | 21:04 |
| xgerman_ | still hoping we can merge it before lbaasv2 gets removed ;-) | 21:06 |
| *** pcaruana has joined #openstack-lbaas | 21:07 | |
| *** salmankhan has joined #openstack-lbaas | 21:08 | |
| *** colby_home has joined #openstack-lbaas | 21:09 | |
| colby_home | Hey Guys. I have a question about the certificate setup for the amphora. I looked at the script to generate the certs (not ideal for production). So I created a server_ca,server_key,server_cert,client_ca,client_key,client_cert. Which ones get assigned to which config option. Reading through docs doesn't give a lot of info | 21:11 |
| *** salmankhan has quit IRC | 21:13 | |
| johnsom | colby_home Agreed, we still need a detailed install guide. This is how it is configured for the test gates: https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L295-L305 | 21:13 |
| johnsom | based on that script, which, yes, not good for production use | 21:13 |
| johnsom | Does that answer your question? | 21:14 |
| colby_home | I did a separate server CA and Client CA (recommended for production). How do those get configured? | 21:14 |
| johnsom | Well, I don't think they should be different CA's. | 21:18 |
| johnsom | Let me walk through the configuration items and explain them to see if that helps. | 21:19 |
| colby_home | This was in that script to generate: | 21:19 |
| colby_home | echo "Note: For production use the ca issuing the client certificate and the ca issuing the server" | 21:19 |
| colby_home | echo "certificate need to be different so a hacker can't just use the server certificate from a" | 21:19 |
| colby_home | echo "compromised amphora to control all the others." | 21:19 |
| colby_home | echo "\nTo use the certificates copy them to the directory specified in the octavia.conf" | 21:19 |
| johnsom | Ah, hang on, yeah, I think I follow. I'm a bit rusty on this config section | 21:20 |
| xgerman_ | yeah, we use the same CA — | 21:22 |
| johnsom | The "server CA" cert should be certificates ca_certificate and ca_private_key | 21:22 |
| johnsom | This is the CA that issues certificates to the amphora and the CA that verifies the certificate presented by the amphora agent. | 21:22 |
| xgerman_ | if you need two CAs look at this sequence of commands: https://github.com/rcbops/rpc-octavia/blob/master/playbooks/rpc-octavia-generate-certs.yml | 21:22 |
| johnsom | The "client CA" cert would be haproxy_amphora server_ca | 21:22 |
| johnsom | haproxy_amphora client_cert is the certificate the controllers present to the amphora-agent | 21:23 |
| johnsom | so would be the client cert | 21:23 |
| johnsom | xgerman_ Are you using a split CA in OSA, I don't think so, I think you are using a single CA like the gates. | 21:24 |
| xgerman_ | nope, split CA | 21:24 |
| johnsom | The comment in the file is accurate and valid about splitting them | 21:24 |
| johnsom | Ok | 21:25 |
| johnsom | colby_home Did I confuse you worse, or help? | 21:26 |
| colby_home | I think that helps :) | 21:27 |
| colby_home | so haproxy_amphora client_cert should be the cert signed by the client CA? | 21:28 |
| johnsom | yes | 21:29 |
| colby_home | gotcha ok I think I got it | 21:30 |
| johnsom | Ah, I see one part we are missing to the picture. | 21:32 |
| johnsom | The client CA cert is controller-worker client_ca | 21:33 |
| abaindur | [certificates] | 21:38 |
| abaindur | ca_certificate = ${OCTAVIA_CERTS_DIR}/ca_01.pem | 21:38 |
| abaindur | ca_private_key = ${OCTAVIA_CERTS_DIR}/private/cakey.pem | 21:38 |
| abaindur | ca_private_key_passphrase = <enter_your_password_here> | 21:38 |
| abaindur | [haproxy_amphora] | 21:38 |
| abaindur | client_cert = ${OCTAVIA_CERTS_DIR}/client.pem | 21:38 |
| abaindur | server_ca = ${OCTAVIA_CERTS_DIR}/ca_01.pem | 21:38 |
| abaindur | I just did that ^^ | 21:38 |
| abaindur | where OCTAVIA_CERTS_DIR was the directory the built in create_certificates.sh put the files in | 21:39 |
| johnsom | colby_home You also need to set the client ca in [controller-worker] | 21:39 |
| johnsom | Otherwise, yes, that looks right | 21:39 |
| colby_home | ah ok that makes more sense | 21:39 |
| johnsom | Yeah, I was puzzled too | 21:39 |
| abaindur | what does client_ca need to be set to? | 21:40 |
| johnsom | Your ca_01.pem from the client CA that generated client.pem | 21:40 |
| johnsom | or 02 or whatever it is called | 21:41 |
| abaindur | So the same as ca_certificate from [certificates] section? | 21:41 |
| abaindur | and server_ca in [haproxy_amphora] | 21:41 |
| johnsom | no, it should be a client CA where [certificates] is the server CA | 21:41 |
| abaindur | I dont see the devstack plugin setting the client_ca option in [controller_worker] section | 21:43 |
| johnsom | https://github.com/openstack/octavia/blob/master/etc/octavia.conf#L237 | 21:45 |
| johnsom | or here https://docs.openstack.org/octavia/latest/configuration/configref.html#controller_worker.client_ca | 21:45 |
| abaindur | right im using the same CA | 21:47 |
| abaindur | so its the same .pem file as ive set in those other 2 places | 21:47 |
| abaindur | i guess colby_home is using separate CAs | 21:47 |
| johnsom | Right, ah, confusing, ha. Yeah, if you have two CAs those are going to be different | 21:47 |
| colby_home | Im deploying in a production env (we dont really have a dev one for openstack) so I wanted to do the certs in the best way possible | 21:48 |
| johnsom | split CA is the best option | 21:48 |
| abaindur | i'm not even worrying about all that and production yet. just trying to get it working in a basic setup :) | 21:49 |
| abaindur | what is the amphora_agent section for? deprecated? | 21:49 |
| abaindur | [amphora_agent] | 21:49 |
| abaindur | # agent_server_ca = /etc/octavia/certs/client_ca.pem | 21:49 |
| abaindur | # agent_server_cert = /etc/octavia/certs/server.pem | 21:49 |
| johnsom | They are not deprecated, but they are used inside the amphroa agent config file. They are ghost records here. I originally didn't add them to this file, but people kept adding them back... | 21:51 |
| johnsom | So, ignore those, the controller will set them on amphora deployment | 21:51 |
| colby_home | Just to be clear, the api server does not need the certs right? Only the worker/healthmonitor/housekeeper | 21:52 |
| *** SumitNaiksatam has left #openstack-lbaas | 21:52 | |
| johnsom | Correct | 21:53 |
| colby_home | thanks | 21:53 |
| johnsom | API does not talk directly to the amphora | 21:53 |
| *** aojea has quit IRC | 21:55 | |
| *** pcaruana has quit IRC | 22:02 | |
| *** aojea has joined #openstack-lbaas | 22:08 | |
| *** aojea has quit IRC | 22:09 | |
| *** aojea has joined #openstack-lbaas | 22:11 | |
| cgoncalves | https://githubengineering.com/glb-director-open-source-load-balancer/ | 22:19 |
| *** rcernin has joined #openstack-lbaas | 22:19 | |
| johnsom | Yay, yet another load balnacer package... | 22:19 |
| johnsom | Ah, L4 this time | 22:20 |
| johnsom | lol, oh, really ECMP... Got it. Same tech in the L3 act/act distributor | 22:21 |
| *** aojea has quit IRC | 22:30 | |
| openstackgerrit | Merged openstack/octavia master: Delete amphora regardless of status https://review.openstack.org/587255 | 22:35 |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: systemd should recover Keepalived https://review.openstack.org/588993 | 22:50 |
| *** hongbin has quit IRC | 22:54 | |
| *** zioproto_ has joined #openstack-lbaas | 22:55 | |
| openstackgerrit | Merged openstack/octavia master: Fix UDP release note for CentOS https://review.openstack.org/590031 | 22:55 |
| *** fnaval has quit IRC | 22:57 | |
| *** amotoki_ has joined #openstack-lbaas | 23:01 | |
| *** irenab has quit IRC | 23:02 | |
| *** oanson has quit IRC | 23:02 | |
| *** dulek has quit IRC | 23:02 | |
| *** amotoki has quit IRC | 23:02 | |
| *** dosaboy has quit IRC | 23:02 | |
| *** zioproto has quit IRC | 23:02 | |
| *** zioproto_ is now known as zioproto | 23:02 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: Leave VIP NIC plugging for keepalived https://review.openstack.org/589292 | 23:05 |
| *** abaindur has quit IRC | 23:09 | |
| *** nmagnezi_ has quit IRC | 23:10 | |
| rm_work | o/ sorry missed the meeting | 23:23 |
| *** luksky has quit IRC | 23:26 | |
| johnsom | All good. RC1 release is up for review | 23:30 |
| rm_work | cool | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!