| eandersson | patch looks good atm have two nodes | 00:33 |
|---|---|---|
| eandersson | one patch, one not | 00:33 |
| eandersson | patched one has 100 connections, unpatched almost 6k | 00:34 |
| eandersson | no actual load applied on either | 00:34 |
| colin- | yeah, glad to see something apparent that we can track | 00:35 |
| colin- | and a positive correlation from the change :) | 00:35 |
| *** openstackgerrit has quit IRC | 00:52 | |
| *** sapd1 has joined #openstack-lbaas | 01:26 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 01:32 | |
| *** phuochoang has quit IRC | 01:37 | |
| *** ramishra has joined #openstack-lbaas | 01:52 | |
| *** fnaval has quit IRC | 02:13 | |
| *** fnaval_ has joined #openstack-lbaas | 02:13 | |
| *** hongbin has joined #openstack-lbaas | 02:21 | |
| *** abaindur has quit IRC | 02:22 | |
| colin- | opened a story under the project to track what we've been discussing, first time so lmk if i missed anything | 02:23 |
| johnsom | colin- Looks pretty good. I usually tag for backport-candidate. | 02:25 |
| johnsom | If of course there is a chance this could be backported | 02:25 |
| colin- | and do you typically try to associate the gerrit changes with the stories in comments? | 02:26 |
| johnsom | Yes, it's Story: 2004993 and Task: 29464 | 02:26 |
| johnsom | Like on this one: https://review.openstack.org/#/c/617385/ | 02:27 |
| colin- | cool, thx | 02:29 |
| *** Dinesh_Bhor has quit IRC | 02:44 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 02:48 | |
| johnsom | bzhao__ Are you online? | 02:55 |
| bzhao__ | johnsom: Yeah. | 02:55 |
| johnsom | bzhao__ Hi! | 02:55 |
| bzhao__ | johnsom: Hi, ;-) | 02:56 |
| johnsom | bzhao__ I have started a patch addressing my comments on that patch. | 02:56 |
| johnsom | bzhao__ I hope that is ok and you had not started. | 02:57 |
| bzhao__ | johnsom: Change back to add new columns into tables, but not a new table ? | 02:57 |
| bzhao__ | johnsom: That's OK, I'm not start yet. ;-) | 02:57 |
| johnsom | bzhao__ I started at the top of my comments and have been fixing your patch. | 02:57 |
| bzhao__ | johnsom: Just return from long holiday.. | 02:57 |
| johnsom | bzhao__ Nice! | 02:58 |
| johnsom | bzhao__ I am starting to review and work on your TLS chain starting this week. | 02:58 |
| bzhao__ | johnsom: Thanks, I will cooperate with your suggest. ;-) | 02:59 |
| bzhao__ | johnsom: As I following the style based on the first patch, so that may be a multiple fixes. | 03:00 |
| johnsom | bzhao__ Yeah, I understand that. These updates will need to be done across the chain. Sorry we didn't have reviews early. | 03:00 |
| johnsom | bzhao__ This is partly why I figured I would just work on the fixes as well. | 03:01 |
| bzhao__ | johnsom: Thanks, I know there are so many features need to be landed this cycle. You all have done a great work. | 03:02 |
| johnsom | bzhao__ So I'm not going to finish this update tonight. I will read your comments and address them in my change. I will finish the update tomorrow and post an amended patch. | 03:03 |
| bzhao__ | johnsom: OK, thanks for help. I will hold on and continue the tempest part maybe. | 03:04 |
| johnsom | bzhao__ If you are bored and looking for something to do, we could use tempest tests. I did comment on your https server patch. I also added a base TLS scenario, etc. for an example: https://review.openstack.org/#/c/628075/ | 03:05 |
| bzhao__ | johnsom: Yeah, thanks for this work. I will refactor the exist tempest patch. ;P | 03:07 |
| johnsom | bzhao__ Ok, thanks. I am going to have dinner now. I think we can get all of this in for Stein. That is my plan. Glad you are back, that will help. | 03:08 |
| bzhao__ | johnsom: Ha, sorry for my absence, there are so many troubles in my 2 months, so I'm really happy for back to work with you all. ;-) | 03:11 |
| bzhao__ | johnsom: Happy dinner. ;-) | 03:13 |
| eandersson | So patch is looking great | 03:33 |
| eandersson | 12k on node 1 (unpatched) 158 on patched | 03:34 |
| eandersson | I'll do a push tomorrow to fix clean up unit-tests and clean up based on feedback | 03:35 |
| eandersson | If you guys have time review it and provide feedback | 03:35 |
| rm_work | Yeah I'll definitely look | 03:42 |
| eandersson | Thanks rm_work | 03:43 |
| eandersson | Trying to clean up the tests I broke like 57 other things lol | 04:32 |
| *** fnaval_ has quit IRC | 04:38 | |
| *** hongbin has quit IRC | 04:55 | |
| *** openstackgerrit has joined #openstack-lbaas | 05:07 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: Fix oslo messaging connection leakage https://review.openstack.org/636428 | 05:07 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: Fix oslo messaging connection leakage https://review.openstack.org/636428 | 05:08 |
| eandersson | rm_work, johnsom https://review.openstack.org/#/c/636428/3/octavia/controller/queue/event_queue.py | 05:12 |
| eandersson | I don't know how to handle this | 05:12 |
| eandersson | btw at 17k vs 158 connections :p | 05:15 |
| eandersson | so pretty sure the patch is effective | 05:15 |
| openstackgerrit | Merged openstack/octavia master: Refactor the pluggin of the VIP https://review.openstack.org/604479 | 05:28 |
| *** psachin has joined #openstack-lbaas | 05:50 | |
| *** Dinesh_Bhor has quit IRC | 06:30 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 06:39 | |
| *** yboaron_ has joined #openstack-lbaas | 06:42 | |
| *** ccamposr has joined #openstack-lbaas | 06:43 | |
| *** obondarev has joined #openstack-lbaas | 07:05 | |
| *** obondarev has left #openstack-lbaas | 07:05 | |
| *** AlexStaf has joined #openstack-lbaas | 07:28 | |
| openstackgerrit | Reedip proposed openstack/octavia-tempest-plugin master: Check Monitor in Member Scenario Tests https://review.openstack.org/634891 | 07:34 |
| *** gcheresh has joined #openstack-lbaas | 07:40 | |
| *** rpittau has joined #openstack-lbaas | 07:57 | |
| *** ipo has joined #openstack-lbaas | 08:00 | |
| *** numans is now known as numans_afk | 08:01 | |
| *** ipo has quit IRC | 08:06 | |
| *** yboaron_ has quit IRC | 08:15 | |
| *** numans_afk is now known as numans | 08:17 | |
| *** velizarx has joined #openstack-lbaas | 08:46 | |
| *** yboaron_ has joined #openstack-lbaas | 09:08 | |
| *** velizarx has quit IRC | 09:10 | |
| *** yamamoto has quit IRC | 09:10 | |
| *** yamamoto has joined #openstack-lbaas | 09:12 | |
| *** yboaron_ has quit IRC | 09:14 | |
| *** yboaron_ has joined #openstack-lbaas | 09:14 | |
| *** velizarx has joined #openstack-lbaas | 09:24 | |
| *** yboaron_ has quit IRC | 09:26 | |
| *** yboaron_ has joined #openstack-lbaas | 09:26 | |
| *** obondarev has joined #openstack-lbaas | 09:49 | |
| *** mkuf_ is now known as mkuf | 10:01 | |
| *** obondarev has quit IRC | 10:04 | |
| *** obondarev has joined #openstack-lbaas | 10:05 | |
| *** yamamoto has quit IRC | 10:30 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Check Monitor in Member Scenario Tests https://review.openstack.org/634891 | 10:41 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Check Monitor in Member Scenario Tests https://review.openstack.org/634891 | 10:42 |
| openstackgerrit | Yang JianFeng proposed openstack/octavia master: Add listener and pool protocol validation https://review.openstack.org/594040 | 10:49 |
| *** yamamoto has joined #openstack-lbaas | 11:05 | |
| *** velizarx has quit IRC | 11:13 | |
| *** yamamoto has quit IRC | 11:14 | |
| *** Dinesh_Bhor has quit IRC | 11:21 | |
| openstackgerrit | Nir Magnezi proposed openstack/neutron-lbaas stable/rocky: nlbaas2octavia: do not change SG owned by user https://review.openstack.org/636596 | 11:22 |
| *** yamamoto has joined #openstack-lbaas | 11:28 | |
| *** sapd1 has quit IRC | 11:45 | |
| rm_work | erk, https://review.openstack.org/604479 is going to wreak total havoc on my L3 driver T_T lol | 11:52 |
| *** rpittau has quit IRC | 11:52 | |
| *** velizarx has joined #openstack-lbaas | 12:08 | |
| *** salmankhan has joined #openstack-lbaas | 12:11 | |
| *** salmankhan1 has joined #openstack-lbaas | 12:22 | |
| *** salmankhan has quit IRC | 12:24 | |
| *** salmankhan1 is now known as salmankhan | 12:24 | |
| *** yamamoto has quit IRC | 12:28 | |
| *** rpittau has joined #openstack-lbaas | 12:29 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: Amphora logging https://review.openstack.org/624835 | 12:34 |
| *** yamamoto has joined #openstack-lbaas | 12:36 | |
| *** trown|outtypewww is now known as trown | 13:06 | |
| *** sapd1 has joined #openstack-lbaas | 13:08 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 13:34 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia-tempest-plugin master: Save amphora logs in gate https://review.openstack.org/626406 | 13:36 |
| *** velizarx has quit IRC | 13:39 | |
| *** obondarev has quit IRC | 13:42 | |
| openstackgerrit | Merged openstack/octavia-tempest-plugin master: Use cls.skipException and raise it https://review.openstack.org/630049 | 13:43 |
| *** Dinesh_Bhor has quit IRC | 13:44 | |
| *** obondarev has joined #openstack-lbaas | 13:47 | |
| *** velizarx has joined #openstack-lbaas | 13:52 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 14:22 | |
| *** yamamoto has quit IRC | 14:44 | |
| *** yamamoto has joined #openstack-lbaas | 14:49 | |
| *** yamamoto has quit IRC | 14:49 | |
| *** yamamoto has joined #openstack-lbaas | 14:50 | |
| *** yamamoto has quit IRC | 14:50 | |
| *** sapd1 has quit IRC | 14:52 | |
| *** yamamoto has joined #openstack-lbaas | 15:00 | |
| *** yamamoto has quit IRC | 15:00 | |
| *** velizarx has quit IRC | 15:00 | |
| *** fnaval has joined #openstack-lbaas | 15:07 | |
| *** sapd1 has joined #openstack-lbaas | 15:08 | |
| *** Dinesh_Bhor has quit IRC | 15:16 | |
| *** yboaron_ has quit IRC | 15:19 | |
| *** obondarev has quit IRC | 15:21 | |
| *** dayou has quit IRC | 15:29 | |
| *** dayou has joined #openstack-lbaas | 15:52 | |
| *** gcheresh has quit IRC | 15:53 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: WIP: Add initscript element to RHEL8 and Fedora28 https://review.openstack.org/636661 | 16:00 |
| *** ianychoi has quit IRC | 16:08 | |
| colin- | so far so good with our test of the PR branch in the oslo change, after ~16h of operation the number of files on the upstream branch open for amqp connections is 70,412 while on the PR branch it is 158 | 16:18 |
| colin- | no traces yet that i can find | 16:18 |
| johnsom | +1 | 16:18 |
| jiteka | johnsom: is any chance that it could be backported to Rocky ? | 16:23 |
| johnsom | The rabbit patch? Yes, I think so. At least in it's current form it should be fine to go back to queens. | 16:23 |
| jiteka | johnsom: that's a good news then | 16:24 |
| *** ramishra has quit IRC | 16:29 | |
| *** yamamoto has joined #openstack-lbaas | 16:30 | |
| *** ccamposr has quit IRC | 16:31 | |
| cgoncalves | rocky and queens :) | 16:36 |
| *** yamamoto has quit IRC | 16:38 | |
| *** obondarev has joined #openstack-lbaas | 16:52 | |
| *** obondarev has quit IRC | 16:53 | |
| *** obondarev has joined #openstack-lbaas | 16:55 | |
| colin- | guessing the answer is no but is anyone else running this change? | 16:55 |
| colin- | just curious if our observations are consistent | 16:56 |
| johnsom | Sorry, I am not at the moment | 16:56 |
| colin- | all good, has not been very long heh :) | 16:56 |
| *** obondarev has quit IRC | 17:00 | |
| cgoncalves | colin-, so it will reduce both cpu and memory usage footprint? | 17:00 |
| colin- | i think the memory utilization we were referring to was virtual memory on the containers running the api process, as it grew relative to the number of open files for that container | 17:01 |
| colin- | (up to 29G at some points) | 17:01 |
| johnsom | Yeah, it should do both. It looks like oslo messaging has heartbeating enabled by default, so it would have been cycling all of those open connections. | 17:02 |
| colin- | but have seen a dramatic reduction in CPU load on a busy host that was also running octavia-api and struggling with the burden of the extra files | 17:02 |
| colin- | now when i sort octavia procs in top they are organized how i would expect with healthmanager the busier of the four (usually) | 17:02 |
| cgoncalves | right | 17:03 |
| *** AlexStaf has quit IRC | 17:03 | |
| cgoncalves | cool, thank you! | 17:03 |
| johnsom | Agreed, HM is going to be the busiest | 17:03 |
| *** obondarev has joined #openstack-lbaas | 17:16 | |
| *** sapd1 has quit IRC | 17:16 | |
| *** psachin has quit IRC | 17:18 | |
| *** rpittau has quit IRC | 17:24 | |
| colin- | not sure if the runC thing got discussed here but have heard some of you mention running lxc containers so wanted to share this in case it affects anybody: https://github.com/lxc/lxc/commit/6400238d08cdf1ca20d49bafb85f4e224348bf9d | 17:27 |
| johnsom | Yeah, aware of it. But so far all we have is an experimental test gate that uses lxd. It's far from production ready IMO | 17:27 |
| johnsom | Docker is the worst with that issue from my understanding... | 17:28 |
| colin- | it's a much, much simpler exploit on docker imo | 17:28 |
| johnsom | So if people are running with k8s, time to get patching. | 17:28 |
| colin- | yeah, first fixed version is available here https://github.com/docker/docker-ce/releases/tag/v18.09.2 | 17:29 |
| *** sapd1 has joined #openstack-lbaas | 17:29 | |
| colin- | do you use lxc for anything outside of octavia? | 17:29 |
| johnsom | Yes, our whole control plane is in lxc containers | 17:29 |
| colin- | ah neat | 17:29 |
| johnsom | We do not use lxc for the amphora | 17:30 |
| *** sapd1 has quit IRC | 17:37 | |
| colin- | curious how large a fleet anyone using containerized control planes is running, we have discussed the advantages/costs of it internally a bit | 17:53 |
| cgoncalves | services in OSP are containerized. not a number but anyways :) | 18:02 |
| *** sapd1 has joined #openstack-lbaas | 18:27 | |
| jiteka | johnsom: is it with openstack-ansible ? I remember that it deploy all control plane using lxc | 18:27 |
| johnsom | Yes, our deployments are based on OpenStack Ansible | 18:28 |
| *** yamamoto has joined #openstack-lbaas | 18:31 | |
| *** trown is now known as trown|lunch | 18:34 | |
| *** salmankhan has quit IRC | 18:40 | |
| *** velizarx has joined #openstack-lbaas | 18:41 | |
| *** AlexStaf has joined #openstack-lbaas | 19:01 | |
| *** obondarev has quit IRC | 19:10 | |
| *** obondarev has joined #openstack-lbaas | 19:15 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: Fix oslo messaging connection leakage https://review.openstack.org/636428 | 19:32 |
| *** obondarev has quit IRC | 19:38 | |
| *** irclogbot_1 has quit IRC | 19:47 | |
| colin- | "Running multiple instances of the individual Octavia controller components on separate physical hosts is recommended in order to provide scalability and availability of the controller software." | 19:48 |
| colin- | can anyone tell me if the adjective physical was deliberate or just incidental in that sentence? | 19:49 |
| colin- | if deliberate, why call out physical? just curious | 19:49 |
| *** trown|lunch is now known as trown | 19:52 | |
| johnsom | incidental | 19:54 |
| johnsom | Basically, you don't want all of you controller instances on the same physical host, as if it goes down, your control plane goes down. So we spread our controller containers over three physical hosts | 19:56 |
| nmagnezi | johnsom, o/ | 19:57 |
| johnsom | nmagnezi Hi! | 19:57 |
| nmagnezi | In the open discussion part lets talk about https://review.openstack.org/#/c/553381/ | 19:57 |
| nmagnezi | Jacky did a great job here, I would like to help to get this in | 19:58 |
| johnsom | Ok, I can add it to the agenda | 19:58 |
| nmagnezi | The reason I did not post a new ps is because some cores vote in favor and some against | 19:58 |
| nmagnezi | Thank you :) | 19:58 |
| johnsom | #startmeeting Octavia | 20:00 |
| openstack | Meeting started Wed Feb 13 20:00:10 2019 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 20:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 20:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 20:00 | |
| openstack | The meeting name has been set to 'octavia' | 20:00 |
| johnsom | Hi folks | 20:00 |
| colin- | \o | 20:00 |
| nmagnezi | o/ | 20:00 |
| eandersson | o/ | 20:00 |
| johnsom | pinging rm_work grin | 20:00 |
| cgoncalves | ~o~ | 20:00 |
| *** irclogbot_1 has joined #openstack-lbaas | 20:00 | |
| johnsom | #topic Announcements | 20:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 20:01 | |
| johnsom | TC nominations are open | 20:01 |
| johnsom | If you are interested in running for the TC, the details are here: | 20:01 |
| johnsom | https://governance.openstack.org/election/ | 20:01 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Add Python 3.7 support https://review.openstack.org/635236 | 20:01 |
| johnsom | The week of Feb 25th is the last library release for Stein. | 20:02 |
| johnsom | I would really, really, really like to get the octavia-lib changes in. | 20:02 |
| cgoncalves | #link https://review.openstack.org/#/q/project:openstack/octavia-lib+status:open | 20:02 |
| johnsom | Faster on the copy/paste.... lol | 20:03 |
| nmagnezi | lol | 20:03 |
| nmagnezi | was about to post the same | 20:03 |
| cgoncalves | slackers :P | 20:03 |
| johnsom | Also note, the week of March 4th is feature freeze | 20:03 |
| johnsom | And final clients. | 20:03 |
| johnsom | Any other announcements this week? | 20:04 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:04 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 20:04 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: Fix oslo messaging connection leakage https://review.openstack.org/636428 | 20:05 |
| johnsom | Ok, I have updated the octavia-lib patches (except for the constant migration patches) and flavors is all wrapped up. | 20:05 |
| colin- | nice | 20:05 |
| johnsom | Right now my focus is 100% on reviews and helping patches get merged. | 20:05 |
| johnsom | I worked on the VIP refactor patch german had proposed, that is good if not merged now. | 20:05 |
| cgoncalves | merged | 20:06 |
| xgerman | thanks! | 20:06 |
| johnsom | I am currently working with zhao on the TLS patch chain. The first of which should be posted today for review. | 20:06 |
| johnsom | I am planning to really push to get the TLS features in for Stein as I think they have great value. (TLS client auth and backend re-encryption) | 20:07 |
| johnsom | Any other updates? I see Erik was crafty at pushing his patch during the progress report... Good stuff there. | 20:08 |
| colin- | ah b-channel, that would be great | 20:08 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: Fix oslo messaging connection leakage https://review.openstack.org/636428 | 20:08 |
| colin- | look at that, he can't help himself :p | 20:08 |
| eandersson | :D | 20:08 |
| johnsom | b-channel? | 20:08 |
| eandersson | I am gonna try to keep this patch as consistent as possible with how nova / neutron etc does it (so ideally keep things like asserts for now at least) | 20:09 |
| eandersson | I have a question about the EventStream | 20:09 |
| colin- | backend re-encryption is terminating the client https request and re-initiating it from the amp to the backend with a new negotiation right? | 20:09 |
| eandersson | Should I just leave that code path as-is for now? | 20:09 |
| johnsom | colin- Yes, frontend VIP is the same, but when we proxy to the member, this path is also over TLS. | 20:10 |
| johnsom | eandersson In my opinion, the event streamer stuff needs to be totally ripped out. This is in plan with the v1/neutron-lbaas retirement this year. | 20:11 |
| cgoncalves | +1 | 20:12 |
| johnsom | I don't think it's used much and is... I think in the future we need a better design for general eventing, but that code is not a good base. | 20:12 |
| eandersson | Makes sense and I agree | 20:13 |
| cgoncalves | in the spirit of getting rid of some warnings in my new shiny IDE, I ended up adding python 3.7 support (+ minor refactors) and fixing a couple of functional tests under python 3.6 | 20:13 |
| johnsom | So, maybe a simple __del__ hook for now, re-address if someone cares/uses it. | 20:13 |
| cgoncalves | also not sure I reported last week on a new tempest test + job for amphora spare pool | 20:13 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: Fix oslo messaging connection leakage https://review.openstack.org/636428 | 20:14 |
| johnsom | Yeah, interesting stuff there. I need to look at that. | 20:14 |
| eandersson | I would prefer to keep it out of the scope for this patch to make it easier to backport etc | 20:14 |
| cgoncalves | octavia-grenade in stable/rocky is still faulty, consistently (please do not recheck). I tried to run grenade locally but ran into several issues. I will be continuing that | 20:14 |
| eandersson | And we can just follow up for master | 20:15 |
| johnsom | Ok | 20:15 |
| eandersson | Since this is primarily jsut impacting the api service from my testing | 20:15 |
| johnsom | Oh, and I got an itch to try out LXD again over the weekend and was successful (after way too much effort) to launch lxd/lxc amps. | 20:16 |
| eandersson | Nice | 20:16 |
| johnsom | There is a patch up with a passing gate. It shaves ~30 minutes off a tempest scenario run. | 20:16 |
| colin- | strong!@ | 20:16 |
| johnsom | That said, it has none of our kernel tuning, disables most of the lxc security, nova throws a ton of errors, and UDP probably doesn't work. | 20:17 |
| colin- | still intending to circle back to that first opportunity johnsom, container amps are an objective | 20:17 |
| colin- | hehe | 20:17 |
| colin- | fair | 20:17 |
| johnsom | So, highly not recommended for production workloads.... | 20:18 |
| johnsom | Maybe in open discussion I will ask if we even want to merge that or not. | 20:18 |
| johnsom | #topic Talk about cascade delete via the dashboard | 20:18 |
| *** openstack changes topic to "Talk about cascade delete via the dashboard (Meeting topic: Octavia)" | 20:18 | |
| johnsom | #link https://review.openstack.org/#/c/553381/ | 20:18 |
| nmagnezi | Yup, thank you for bringing this up | 20:18 |
| johnsom | nmagnezi You have the floor | 20:18 |
| nmagnezi | Basically Jacky did a great job with this | 20:19 |
| nmagnezi | It's just that the discussion on the patch prevented it from getting in | 20:19 |
| nmagnezi | Now, I know Jacky probably didn't have the cycles to follow up | 20:19 |
| nmagnezi | And I can help with that | 20:19 |
| nmagnezi | But we need agree if we want to change the default to cascade | 20:20 |
| nmagnezi | And if we want to add a warning | 20:20 |
| johnsom | Wish rm_work was here, he is one of the -1's | 20:20 |
| cgoncalves | the warning would be in the delete confirmation window itself, no?! | 20:20 |
| nmagnezi | The reason I'm saying this is because some cores (my included) voted -1 and some +2 | 20:20 |
| * nmagnezi looks at xgerman | 20:20 | |
| xgerman | the motivation for cascade in our API was for the dashboard | 20:20 |
| nmagnezi | cgoncalves, I think so, yeah | 20:21 |
| xgerman | I even had to code an lbaasv2 neutron extension just for the cascade flag | 20:21 |
| xgerman | so yes, it should go in! | 20:21 |
| nmagnezi | I think that keeping things simple by simply added a warning text is enough, but would like to hear others | 20:21 |
| cgoncalves | all it would take is string change. we just need consensus | 20:21 |
| johnsom | Agreed, that was part of the intent. I have not had time to try this out however. | 20:21 |
| cgoncalves | I'd say let's be explicit about delete cascade in the delete message and be done with it | 20:22 |
| nmagnezi | johnsom, I posted my test result to the patch but I can wait for you to test before I touch this | 20:22 |
| johnsom | For those that have loaded it up, does it pop up a confirmation now or does it just do the cascade delete? | 20:22 |
| xgerman | yep, warn, confirm - done | 20:22 |
| nmagnezi | Unless Jacky is here? | 20:22 |
| johnsom | He likely won't be online for a few hours | 20:22 |
| nmagnezi | johnsom, I uploaded this https://pasteboard.co/HR3IkXF.png | 20:23 |
| johnsom | Ok, cool. Yeah, my vote would be to update the text so it is explicit that it will delete the whole LB and call it good. | 20:23 |
| nmagnezi | I think the same, just wanted to double check since this is changing the default to --cascade | 20:24 |
| cgoncalves | good, we are all in agreement. let's just ping dayou and ask if he could do it, otherwise nmagnezi or any of us can do | 20:24 |
| johnsom | Yeah, make sure it's called out in the release notes | 20:24 |
| nmagnezi | Yup | 20:24 |
| nmagnezi | johnsom, fair point | 20:25 |
| nmagnezi | dayou, let me know if you want to continue with this, otherwise I can help out | 20:25 |
| johnsom | Yeah, web GUI is the keep-it-simple path IMO, so the simpler it is for users, the better. | 20:25 |
| nmagnezi | +1 | 20:25 |
| johnsom | Also FYI, he is going to help me out by adding the flavor drop down on LB create. | 20:25 |
| cgoncalves | yay! | 20:26 |
| johnsom | Though I need the SDK folks to finish merging: https://review.openstack.org/#/q/project:openstack/openstacksdk+owner:%22Michael+Johnson+%253Cjohnsomor%2540gmail.com%253E%22 | 20:26 |
| johnsom | As he will need the list flavors method | 20:27 |
| johnsom | Ok, nmagnezi Can you comment on the patch with the decision, wording update, and release note request? | 20:28 |
| rm_work | ah | 20:28 |
| nmagnezi | johnsom, doing that as we speak :) | 20:28 |
| johnsom | Thank you | 20:28 |
| rm_work | catching up | 20:28 |
| johnsom | rm_work Sorry, we made the decision... | 20:28 |
| johnsom | grin | 20:29 |
| nmagnezi | rm_work, we decided to refactor the UI plugin to adobe flash | 20:29 |
| johnsom | With adobe air components. Because they are cool | 20:29 |
| johnsom | Funny enough I just found an Adobe Air sticker cleaning out a closet this weekend | 20:30 |
| johnsom | #topic Talk about log offloading | 20:31 |
| *** openstack changes topic to "Talk about log offloading (Meeting topic: Octavia)" | 20:31 | |
| johnsom | #link https://review.openstack.org/624835 | 20:31 |
| johnsom | This is one I wanted to circle back on. Sorry if we already hashed through this, but since we have some operators on I wanted to ask about logging infrastructure. | 20:31 |
| cgoncalves | ah, cool. I wanted to ping people about this :) | 20:32 |
| johnsom | Do you folks have a centralized logging solution in your deployment? | 20:32 |
| johnsom | If so, is it per project views, just for admin use, etc.? | 20:32 |
| nmagnezi | cgoncalves and I mentioned this in an internal discussion today, I don't know the exact details of this just yet (need to compare to how configure this for other OpenStack projects) | 20:35 |
| nmagnezi | But it was mentioned that for amps we will have a separate user (or view? Not 100% sure I remember the term) | 20:36 |
| cgoncalves | nmagnezi, not sure what you mean with that last message | 20:36 |
| johnsom | Ok, so here is my concern with the current patch. It puts both admin and tenant logs in one stream. I feel like if we merge this and then make it more flexible, we may have painted ourselves in a backward compatibility issue. I would like to spend a bit more time on it before we merge. | 20:37 |
| colin- | any impact to resource footprint of amps? | 20:37 |
| xgerman | well, you can configure it whatever way you want | 20:37 |
| xgerman | colin-: well, you ship logs off so maybe some cpu cycles and network - | 20:38 |
| colin- | should be fairly minimal, even on a busy one i'd imagine | 20:38 |
| xgerman | +1 | 20:38 |
| nmagnezi | cgoncalves, meaning that the logs collected from amps should communicate with rsyslog using some user that we specifically configure in Octavia | 20:38 |
| johnsom | It actually improves performance over writing them local. | 20:39 |
| xgerman | you cna make a custom logging template to ship off more or less | 20:39 |
| xgerman | I think question is what should we deafult to | 20:39 |
| xgerman | There are two scenarios: | 20:40 |
| xgerman | 1) We ship everyhting to some central logging and operator carves it up the way he sees fit | 20:40 |
| johnsom | I would personally like to see two endpoints configurable, one for admin logs, one for the tenant traffic flow logs. They could be configured to the same place, but could be enabled/disabled individually | 20:40 |
| xgerman | 2) users run their own log servers (maybe a vm in tenant network) and we ship haproxy logs there | 20:40 |
| xgerman | combination of 1+2 | 20:40 |
| johnsom | Or adding in my option | 20:41 |
| xgerman | that wold combination of 1+2 | 20:41 |
| johnsom | It would be a different #1, but still leaving an option for #2 later. | 20:42 |
| johnsom | Basically I might want to send the tenant flow logs to a customer facing kibana/logstash, and the admin to a different one. | 20:42 |
| xgerman | yeah, as I said we can change the log template default | 20:42 |
| xgerman | but for(2) we migth need to add some new field to the API so usrs can cinfigure log servers | 20:43 |
| colin- | yeah how trivial/non-trivial is that? | 20:43 |
| colin- | i ahve trouble approximating it | 20:43 |
| johnsom | Right. I don't want to go down that path in Stein. | 20:43 |
| cgoncalves | in that case we would need to extend the API to pass in the user's log server | 20:43 |
| xgerman | so for the template you cna overwrite like with the haproxy one | 20:43 |
| johnsom | The issue with the tenant hosted syslog is then the routing, etc. So that is going to need some strong docs, etc. | 20:43 |
| johnsom | I'm thinking doing the central locations right would be a good first step. It would require an addition config option to split the log streams, etc. | 20:44 |
| cgoncalves | yeah :/ | 20:44 |
| cgoncalves | +1 | 20:45 |
| xgerman | Sure we can add that and change the template — but most people (I know) only run one logstash | 20:46 |
| cgoncalves | would it make sense to log to the local node syslog instead of forwarding logs from all amps in multiple nodes to a central one? | 20:46 |
| johnsom | Mostly I was hoping for some operator feedback on if that is useful or not | 20:46 |
| colin- | being able to differentiate tenant v non-tenant logs would be useful for us | 20:46 |
| cgoncalves | I' thinking like if there's something wrong between amp and log server. we wouldn't get any logs | 20:46 |
| johnsom | Sure, we can leave the admin logs still writing out local. | 20:47 |
| xgerman | that’s what the log template is for | 20:47 |
| johnsom | I think it's more important to not log the tenant traffic flows local. | 20:47 |
| cgoncalves | johnsom, agreed | 20:48 |
| colin- | definitely but having remote non-tenant logs would be useful too i think | 20:48 |
| johnsom | Is log template just configuring the haproxy log format or like a chunk of rsyslog config? | 20:48 |
| xgerman | rsyslog config | 20:48 |
| xgerman | haproxy-log format isn’t done | 20:48 |
| xgerman | https://review.openstack.org/#/c/624835/16/octavia/amphorae/backends/logging/templates/10-rsyslog.conf.template | 20:49 |
| xgerman | the system will take an operator one like with the haproxy-template | 20:49 |
| cgoncalves | I'm tempted to say writing out local for admin logs would be a good default. we can more safely assume that option is available across most/all deployments | 20:50 |
| johnsom | Yeah, should be a configuration item IMO | 20:50 |
| xgerman | that’s what if does right now | 20:50 |
| xgerman | configure like in custom template or configure like exlicit option | 20:51 |
| johnsom | Hmm, ok. I need to think about this a bit more. I also want to leave some time for open discussion. | 20:51 |
| johnsom | #topic Open Discussion | 20:51 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 20:51 | |
| cgoncalves | oh, wait. maybe I'm confused. when I say local I mean to compute node's syslog | 20:51 |
| johnsom | Any other topics for folks today? | 20:52 |
| cgoncalves | ok, we can continue discussion at another time and offline | 20:52 |
| xgerman | +1 | 20:52 |
| cgoncalves | FYI, I'm confirmed to Denver. Summit + PTG | 20:52 |
| xgerman | Congrats!! | 20:52 |
| johnsom | cgoncalves I don't think there is any way the amp could send it's logs to the compute host's syslog. That is breaking some of the isolation barriers. It would still be inside the amp, like it is today I think. | 20:53 |
| johnsom | Sadly, I am not. Not sure when I can lock that in. | 20:53 |
| cgoncalves | would that be such an achievement over there? :D | 20:53 |
| johnsom | We are confirmed to have an Octavia room for the PTG. | 20:54 |
| cgoncalves | well, you guys are "locals" ;) | 20:54 |
| rm_work | eiddccidnvgeddinjvjljetdnnrtbujivkekejfneinf | 20:54 |
| rm_work | grrrrrr | 20:54 |
| johnsom | lol | 20:54 |
| cgoncalves | road-trip! | 20:54 |
| colin- | it's like the irc equivalent of sliding under the closing door at the last moment :) | 20:54 |
| colin- | welcome | 20:55 |
| johnsom | Well, it's a straight flight for me. Just need to see what my employer is going to do | 20:55 |
| rm_work | i should be there as well | 20:55 |
| cgoncalves | if budget is short, road-trip! | 20:55 |
| rm_work | waiting on confirmation from work, but | 20:55 |
| rm_work | i think will be there regardless | 20:56 |
| xgerman | sweet - tell them you need to hand out some AOL CDs | 20:56 |
| johnsom | So one more quick topic. The LXD gate stuff. It works, but is not exactly production ready. My question is if we even want to merge that as a non-voting gate. We would likely need a "Here is why you shouldn't do this" doc. | 20:56 |
| johnsom | It would probably be cheaper for me to fly that drive. | 20:57 |
| colin- | The LXD gate stuff = conatiner amps? or something else | 20:58 |
| colin- | or a CI construct | 20:58 |
| rm_work | i could drive down and then we drive over :P | 20:58 |
| johnsom | Yeah, ~$250 bucks to drive | 20:58 |
| johnsom | It is amps running in LXC containers instead of service VMs | 20:59 |
| rm_work | do you fit in a miata? :D | 20:59 |
| colin- | personally in favor of anything that moves that ball forward, just my $0.02 | 20:59 |
| johnsom | lol, not for 1,254 miles each way I don't | 20:59 |
| johnsom | Ok,, about out of time. Thanks folks! Have a good week. | 20:59 |
| rm_work | i drove up from TX to WA with my 6'4" friend and our luggage :P | 21:00 |
| johnsom | #endmeeting | 21:00 |
| xgerman | o/ | 21:00 |
| *** openstack changes topic to "Discussions for Octavia | Stein priority review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 21:00 | |
| openstack | Meeting ended Wed Feb 13 21:00:12 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-02-13-20.00.html | 21:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-02-13-20.00.txt | 21:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-02-13-20.00.log.html | 21:00 |
| johnsom | We could stop in Boise and pick up Doug | 21:01 |
| rm_work | no we could not :P | 21:01 |
| rm_work | unless we switched to your car i guess lol | 21:01 |
| rm_work | or he wants to ride on the roof | 21:01 |
| johnsom | lol | 21:01 |
| cgoncalves | shotgun! you read it here first (as if I would be tagging along, lol...) | 21:02 |
| johnsom | Though we could probably build a raft for the miata, No | 21:03 |
| johnsom | Starting to sound more and more like a top gear episode | 21:03 |
| xgerman | just switch cars in Boise | 21:04 |
| johnsom | Everybody aboard the OpenStack bus | 21:05 |
| johnsom | Code while you see the country! | 21:06 |
| johnsom | We just need xgerman to borrow one of those fancy google buses | 21:08 |
| xgerman | ? no access to them | 21:08 |
| johnsom | Well, "borrow" | 21:09 |
| johnsom | Like hopefully they wouldn't miss it for a few weeks | 21:09 |
| eandersson | I am happy with the patch now ^ https://review.openstack.org/#/c/636428 | 21:14 |
| eandersson | We should probably move octavia.common.rpc to the octavia-lib tbh (at least that is how neutron does it) | 21:15 |
| eandersson | but we could do that as a follow up patch | 21:15 |
| eandersson | I really wanted to keep the rpc impl consistent with all other projects | 21:17 |
| johnsom | Yeah, I don't think so. It's a amphora agent specific thing at this point. | 21:20 |
| rm_work | Yeah damn I forgot to submit my review last night 😑 | 21:43 |
| rm_work | But that means this one will be quick | 21:43 |
| rm_work | Looking now | 21:43 |
| rm_work | reviewed -- eandersson / johnsom | 22:03 |
| openstackgerrit | German Eichberger proposed openstack/octavia-tempest-plugin master: Save amphora logs in gate https://review.openstack.org/626406 | 22:05 |
| *** trown is now known as trown|outtypewww | 22:06 | |
| *** openstackgerrit has quit IRC | 22:22 | |
| *** eandersson has quit IRC | 22:33 | |
| *** sapd1 has quit IRC | 22:40 | |
| *** sapd1 has joined #openstack-lbaas | 22:55 | |
| *** eandersson has joined #openstack-lbaas | 22:59 | |
| eandersson | rm_work, thanks | 23:02 |
| eandersson | I would prefer to keep following the same pattern used by nova, neutron, designate etc | 23:03 |
| eandersson | e.g. https://github.com/openstack/nova/blob/master/nova/rpc.py | 23:04 |
| eandersson | Basically trying to be as consistent as possible | 23:04 |
| johnsom | Well, nova especially, but neutron too tend to have a lot of "old way" cruft piled up too. | 23:05 |
| eandersson | Yea | 23:06 |
| eandersson | Honestly ever other service has the same "old way" | 23:06 |
| johnsom | Yeah, I haven't looked at the messaging stuff. I just know things like policy and API features (sort, etc.) are all old/wrong in some of those | 23:07 |
| eandersson | I basically rather start at the "old way" | 23:07 |
| eandersson | and we can then move forward from there | 23:07 |
| eandersson | and ideally work with the oslo.messaging team to carve a more modern path forward | 23:08 |
| eandersson | (backed by docs) | 23:08 |
| *** velizarx has quit IRC | 23:08 | |
| eandersson | I personally would never use globals or asserts :p | 23:09 |
| johnsom | Yeah, docs. Frankly oslo messaging just doesn't work they way I am used to queues working. Definitely room for improvment | 23:09 |
| eandersson | but I do appriciate that I can go to any project and directly understand the implementation | 23:09 |
| eandersson | Yep | 23:09 |
| johnsom | Now is time to take a stand! We don't need to bring cruft into Octavia just because... grin | 23:09 |
| eandersson | I see my patch as the very short term fix | 23:09 |
| eandersson | and long term we should move it to octavia-lib and probably modernize it | 23:09 |
| eandersson | haha well 7 million rmq connections is something that needs be fixed a year ago lool | 23:10 |
| johnsom | Ha, go look at the oslo messaging implementation and tell me you can directly understand the implementation.... lol | 23:10 |
| eandersson | Sorry, was referring to the general workflow lol not how it internally works | 23:10 |
| johnsom | I don't think it should be in octavia-lib reall | 23:10 |
| johnsom | really. That is pretty focused on the provider drivers, which we aren't using queues for. | 23:10 |
| eandersson | Although I do have a lot of experience with the internals of it as I replaced amqp with my own library at one point | 23:11 |
| eandersson | Sure - that was just an idea =] | 23:11 |
| johnsom | Ah! A perfect volunteer! grin | 23:11 |
| eandersson | I am busy enough maintaing my own AMQP library lol https://github.com/eandersson/amqpstorm | 23:11 |
| eandersson | but who knows - I was never meant to do any Octavia code at all lol | 23:15 |
| eandersson | busy enough with Designate / Senlin / Magnum, so wouldn't suprise me if I end up doing more =] | 23:15 |
| *** sapd1 has quit IRC | 23:16 | |
| *** abaindur has joined #openstack-lbaas | 23:16 | |
| eandersson | I removed | 23:22 |
| eandersson | > lass ProducerHandler(abstract_handler.BaseHandler) | 23:22 |
| eandersson | and the tox tests passed without an issue | 23:22 |
| eandersson | johnsom, want me to remove the above ^? | 23:26 |
| eandersson | nvm it's used in the setup.cfg > queue_producer = octavia.api.handlers.queue.producer:ProducerHandler | 23:27 |
| johnsom | I think if you do the v1 tempest tests will fail | 23:27 |
| johnsom | We are going to go on quite the diet when the neutron-lbaas retirement occurs.... | 23:28 |
| *** openstackgerrit has joined #openstack-lbaas | 23:28 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/octavia master: Fix oslo messaging connection leakage https://review.openstack.org/636428 | 23:28 |
| eandersson | At 230k rmq connections now in lab on old code :D | 23:31 |
| johnsom | Adding that to the charge-backs? | 23:32 |
| eandersson | You know it haha | 23:32 |
| johnsom | 1 LB = 60 rabbit connections at $1 each... | 23:32 |
| xgerman | nobody would. buy LB that way | 23:33 |
| xgerman | but I like it: The lB is free its just the AMQ we need to charge for | 23:33 |
| johnsom | Have you used AWS recently? They nickel and dime for everything. | 23:34 |
| xgerman | I have some accounts but don’t use them | 23:34 |
| xgerman | will probably change in the future | 23:35 |
| xgerman | aka I had this vision to backup my home stuff into glacier but lack of time... | 23:35 |
| johnsom | If you have a lot of stuff, check out backblaze | 23:37 |
| xgerman | I don’t… | 23:38 |
| xgerman | still mad at myself for not signing up for crashplan back in the day | 23:38 |
| colin- | don't be | 23:47 |
| colin- | they went outta business >:( | 23:47 |
| colin- | assuming we are talking about the sweet, unlimited storage personal one they offered for years. used that a lot | 23:48 |
| johnsom | That is what backblaze offers for $5 a month | 23:49 |
| johnsom | Ha, ok, since yesterday, $6 | 23:49 |
| colin- | trying to remember my beef with that product | 23:53 |
| colin- | not-whole-disk selection maybe? | 23:53 |
| *** fnaval has quit IRC | 23:56 | |
| xgerman | dang, looks like I am always too late for backup… but so far so good (without off site backup) | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
