| *** yamamoto has quit IRC | 01:03 | |
| *** yamamoto has joined #openstack-lbaas | 01:03 | |
| *** dosaboy has quit IRC | 01:19 | |
| *** zioproto has quit IRC | 01:19 | |
| *** irenab has quit IRC | 01:22 | |
| *** hongbin has joined #openstack-lbaas | 01:48 | |
| *** pck has quit IRC | 01:49 | |
| *** andreykurilin has quit IRC | 01:49 | |
| *** zigo has quit IRC | 01:49 | |
| *** rm_work has quit IRC | 01:49 | |
| *** pck has joined #openstack-lbaas | 01:49 | |
| *** andreykurilin has joined #openstack-lbaas | 01:49 | |
| *** rm_work has joined #openstack-lbaas | 01:55 | |
| *** sapd1 has joined #openstack-lbaas | 02:34 | |
| xgerman_ | ^^ happy-dance | 02:34 |
|---|---|---|
| *** openstack has joined #openstack-lbaas | 02:52 | |
| dayou | ^^ chicken-winner | 02:52 |
| *** ChanServ sets mode: +o openstack | 02:52 | |
| *** openstackstatus has joined #openstack-lbaas | 03:00 | |
| *** ChanServ sets mode: +v openstackstatus | 03:00 | |
| rm_work | hmm, johnsom it seems the housekeeping is a bit fragile still :/ | 03:03 |
| rm_work | recently i've seen the DB connection hiccup and then never fix itself, and so it just stops spares-pooling | 03:04 |
| johnsom | Hmmm | 03:04 |
| rm_work | "Database connection was found disconnected; reconnecting" | 03:05 |
| johnsom | Some missing exception handling? | 03:05 |
| rm_work | and then from there on out only the cert-rotate kept going | 03:05 |
| rm_work | maybe | 03:05 |
| johnsom | I know it retried for a while | 03:05 |
| rm_work | gonna look if i have time, trying to get a new version pushed sometime soon tho (our prod is painfully out of date) | 03:05 |
| rm_work | it's been like 3 months since I managed a deploy :3 | 03:06 |
| rm_work | hoping to get it out this week so i can be running Rocky by the time the PTG happens :) | 03:07 |
| *** yboaron_ has joined #openstack-lbaas | 03:17 | |
| *** ramishra has joined #openstack-lbaas | 03:37 | |
| *** hongbin has quit IRC | 03:51 | |
| *** lxkong has joined #openstack-lbaas | 04:04 | |
| *** abaindur has quit IRC | 04:08 | |
| *** abaindur has joined #openstack-lbaas | 04:09 | |
| *** irenab has joined #openstack-lbaas | 04:41 | |
| *** yboaron_ has quit IRC | 04:44 | |
| *** abaindur has quit IRC | 05:32 | |
| *** abaindur has joined #openstack-lbaas | 05:33 | |
| *** abaindur has quit IRC | 05:38 | |
| *** reedipb has joined #openstack-lbaas | 05:44 | |
| numans | reedipb, Hi. So we have wrongly implemented the batch member update in ovn driver then right ? | 05:50 |
| reedipb | numans : some changes are required , its not 'wrong' :) | 05:51 |
| numans | reedipb, ok. | 05:51 |
| reedipb | I am working on it numans, maybe I will post an update in another hour or so :) | 05:51 |
| reedipb | btw Good Morning :D | 05:51 |
| numans | reedipb, great. thanks. | 05:51 |
| numans | reedipb, good morning.. will be noon soon :) | 05:52 |
| reedipb | till it aint noon , its a great morning :P | 05:52 |
| *** abaindur has joined #openstack-lbaas | 06:09 | |
| *** pcaruana has joined #openstack-lbaas | 06:50 | |
| *** rcernin has quit IRC | 07:00 | |
| *** yboaron has joined #openstack-lbaas | 07:03 | |
| *** velizarx has joined #openstack-lbaas | 07:06 | |
| *** dolly has quit IRC | 07:16 | |
| *** takamatsu has quit IRC | 07:16 | |
| *** abaindur has quit IRC | 07:33 | |
| *** ataraday has joined #openstack-lbaas | 07:37 | |
| *** ataraday has quit IRC | 07:43 | |
| *** threestrands has quit IRC | 07:50 | |
| *** ktibi has joined #openstack-lbaas | 07:56 | |
| *** zigo has joined #openstack-lbaas | 07:57 | |
| *** celebdor has joined #openstack-lbaas | 07:57 | |
| *** Emine has joined #openstack-lbaas | 07:58 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia-dashboard master: Imported Translations from Zanata https://review.openstack.org/597418 | 07:58 |
| reedipb | johnsom: around? | 08:13 |
| reedipb | johnsom : I would like to ask you something about the batch_update function... Let me know when you are around :) | 08:19 |
| *** reedipb has quit IRC | 08:26 | |
| *** yboaron_ has joined #openstack-lbaas | 10:34 | |
| *** yboaron has quit IRC | 10:36 | |
| *** yamamoto_ has joined #openstack-lbaas | 10:41 | |
| *** yamamoto has quit IRC | 10:45 | |
| *** celebdor has quit IRC | 10:47 | |
| *** velizarx has quit IRC | 11:03 | |
| *** velizarx has joined #openstack-lbaas | 11:18 | |
| *** reedipb has joined #openstack-lbaas | 11:19 | |
| *** yboaron_ has quit IRC | 11:57 | |
| *** yboaron_ has joined #openstack-lbaas | 11:58 | |
| *** amuller has joined #openstack-lbaas | 12:28 | |
| *** yamamoto has joined #openstack-lbaas | 12:33 | |
| *** yamamoto_ has quit IRC | 12:36 | |
| *** celebdor has joined #openstack-lbaas | 12:39 | |
| *** celebdor has quit IRC | 12:39 | |
| *** celebdor has joined #openstack-lbaas | 12:40 | |
| *** velizarx has quit IRC | 12:58 | |
| openstackgerrit | Evgeny Fedoruk proposed openstack/octavia master: Specification for extending provider driver library https://review.openstack.org/597505 | 13:02 |
| *** pcaruana has quit IRC | 13:04 | |
| *** velizarx has joined #openstack-lbaas | 13:13 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Fix passphrase None errors https://review.openstack.org/597510 | 13:17 |
| *** yboaron_ has quit IRC | 13:20 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Fix passphrase None errors https://review.openstack.org/597510 | 13:22 |
| *** velizarx has quit IRC | 13:29 | |
| *** velizarx has joined #openstack-lbaas | 13:31 | |
| *** pcaruana has joined #openstack-lbaas | 14:50 | |
| cgoncalves | dayou, hi. is the LB delete cascade available via the dashboard? | 14:52 |
| *** colin- has joined #openstack-lbaas | 14:57 | |
| *** fnaval has joined #openstack-lbaas | 15:03 | |
| *** Emine has quit IRC | 15:27 | |
| johnsom | reedipb What is the question? | 15:32 |
| reedipb | johnsom: I have thought of the solution from OVN, but want to still confirm the design for other proivider drivers | 15:33 |
| reedipb | johnsom: currently with batch updates, we need to delete all Existing members which are not passed in the Request. | 15:33 |
| reedipb | johnsom: with Amphora, its easy. DB is directly accessed with the session | 15:34 |
| reedipb | johnsom: for other Provider Drivers who do not access Octavia DB for List/Show functions, they may not be able to find out the Existing Pool Members | 15:35 |
| reedipb | ( unless they adopt something internally ) | 15:35 |
| johnsom | Right, there are many parts of the API that assume the appliance or driver agent is tracking objects in some way. I know one was planning to tag all of the child objects with the ID, etc. | 15:36 |
| reedipb | ok ok ... sounds good .. | 15:37 |
| johnsom | There is also a spec coming that will add a "backup" API to the Octavia API that would dump a full LB config. | 15:38 |
| cgoncalves | https://review.openstack.org/#/c/597505/ | 15:51 |
| cgoncalves | ^ this spec maybe? | 15:52 |
| *** celebdor1 has joined #openstack-lbaas | 15:54 | |
| *** celebdor has quit IRC | 15:57 | |
| johnsom | Yeah, but I guess not exactly what I was expecting | 15:58 |
| *** ktibi has quit IRC | 16:01 | |
| *** tobias-urdin has joined #openstack-lbaas | 16:14 | |
| *** ramishra has quit IRC | 16:15 | |
| *** celebdor1 has quit IRC | 16:17 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: ML2/OVN + DVR enabled job https://review.openstack.org/597578 | 16:21 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: DNM: ML2/OVN + DVR enabled job https://review.openstack.org/597578 | 16:23 |
| *** colin- has left #openstack-lbaas | 17:00 | |
| *** colin- has joined #openstack-lbaas | 17:00 | |
| *** abaindur has joined #openstack-lbaas | 17:28 | |
| *** abaindur has quit IRC | 17:34 | |
| *** Emine has joined #openstack-lbaas | 17:44 | |
| *** velizarx has quit IRC | 18:20 | |
| *** abaindur has joined #openstack-lbaas | 18:21 | |
| *** amuller has quit IRC | 18:40 | |
| *** Emine has quit IRC | 19:08 | |
| *** pcaruana has quit IRC | 19:09 | |
| *** celebdor1 has joined #openstack-lbaas | 19:16 | |
| johnsom | #startmeeting Octavia | 20:00 |
| openstack | Meeting started Wed Aug 29 20:00:09 2018 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 20:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 20:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 20:00 | |
| openstack | The meeting name has been set to 'octavia' | 20:00 |
| cgoncalves | o/ | 20:00 |
| nmagnezi | o/ | 20:00 |
| johnsom | Hi folks | 20:00 |
| johnsom | #topic Announcements | 20:00 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 20:00 | |
| johnsom | Same reminder about the PTG etherpad: | 20:00 |
| johnsom | #link https://etherpad.openstack.org/p/octavia-stein-ptg | 20:01 |
| johnsom | The PTG is coming up fast, so I will try to put a rough schedule together soon | 20:01 |
| johnsom | Also a note, the TC election nominations start tomorrow. | 20:01 |
| johnsom | #link https://governance.openstack.org/election/ | 20:01 |
| johnsom | In case you are interested in running for the TC | 20:02 |
| johnsom | The official Rocky release patch is up for review. I have already checked the SHAs and our stuff looks good. | 20:02 |
| nmagnezi | Nice | 20:03 |
| johnsom | Any other announcements today? | 20:03 |
| johnsom | #link https://review.openstack.org/597529 | 20:04 |
| johnsom | Finally found the Rocky release link... | 20:04 |
| cgoncalves | Octavia Queens 2.0.2 tagging is blocked until stable team reviews a commit that caught that attention | 20:04 |
| johnsom | Yeah, I saw that. | 20:04 |
| cgoncalves | #link https://review.openstack.org/#/c/593954/ | 20:05 |
| johnsom | FYI, I have already bumped the OpenStack Ansible SHA to use that 2.0.2 version | 20:05 |
| johnsom | I think it will be ok since it includes a default. | 20:05 |
| johnsom | That is all I have so, moving on down the agenda | 20:06 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:06 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 20:06 | |
| johnsom | I have wrapped up most of my internal work so should have a bit more time to work on things upstream. | 20:06 |
| johnsom | I recently wrote a tool to stress test the health manager process. It can also be used to populate a DB with load balancers. | 20:07 |
| johnsom | I will probably post that in my github space sometime in the near future. | 20:07 |
| cgoncalves | nice, thanks! | 20:08 |
| cgoncalves | do share the link later :) | 20:08 |
| johnsom | Testing has shown that we have a bit of a performance regressing in the HM and I have a patch for that in the works. | 20:08 |
| nmagnezi | johnsom, if only we had such tools for nlbaas :D | 20:08 |
| johnsom | I also noticed the UDP code is not reporting listener health correctly and the work around code is in my way with the HM fix, so I will also be posting a fix for that soonish | 20:08 |
| johnsom | nmagnezi HA, well, you can run my existing API stress tool against neutron-lbaas.... But the results aren't pretty | 20:09 |
| nmagnezi | yeah.. we know.. :) | 20:09 |
| johnsom | It does pass well on Octavia however | 20:10 |
| johnsom | That is mostly what I have been up to over the week. Any other updates? | 20:10 |
| johnsom | Oh, I forgot one item | 20:10 |
| johnsom | #link https://review.openstack.org/#/c/594786/ | 20:10 |
| johnsom | I have posted an alternate option for how we can handle API versioning in the tempest plugin. | 20:11 |
| johnsom | rm_work Also posted an idea for it. | 20:11 |
| johnsom | We should try to review those and merge one soonish. | 20:11 |
| johnsom | Without this, the tempest plugin fails when run against queens Octavia | 20:12 |
| nmagnezi | Yes, we saw that in-house | 20:12 |
| nmagnezi | Thanks so much for submitting that one | 20:12 |
| johnsom | It tries to test features that were added in Rocky on the Queens cloud | 20:12 |
| nmagnezi | Yeah stuff like listener timeout and such | 20:12 |
| johnsom | Yep | 20:13 |
| johnsom | This patch tests that patch with stable/queens: | 20:13 |
| johnsom | #link https://review.openstack.org/#/c/595257/ | 20:13 |
| johnsom | Reviews would be appreciated. | 20:13 |
| nmagnezi | Ack | 20:14 |
| cgoncalves | ah, forgot to re-review. you made jobs voting, thanks | 20:14 |
| johnsom | Any other updates? | 20:14 |
| nmagnezi | Speaking of reviews, I have a question about other patch you posted | 20:14 |
| cgoncalves | lol, I already re-reviewed :) | 20:14 |
| nmagnezi | #link https://review.openstack.org/#/c/585031/ | 20:15 |
| johnsom | Sure, what is the question? | 20:15 |
| nmagnezi | I also posted this in gerrit, what would happen if we have a new controller (with this patch) with an older amp | 20:15 |
| nmagnezi | And we try to ask for stats | 20:15 |
| johnsom | You should get an answer as this doesn't connect to the AMP, it just pulls data we already have in the database in a different way. | 20:16 |
| nmagnezi | Oh, alright. didn't get to test it just yet sadly, just a thought that I had | 20:17 |
| johnsom | Yeah, as of now, there is never a connection from the API process directly to an Amphora. Only the other three processes | 20:18 |
| johnsom | The background on this patch and the two in tempest-plugin is I needed a gate test that tests out VRRP failover for an internal request. | 20:19 |
| johnsom | The best way I could come up with to figure out which amp is passing traffic was to look at the per-amphora stats. | 20:19 |
| johnsom | The gate test works, but I still need to add some tempest API tests to the middle patch in that chain, so I have it WIP right now | 20:20 |
| johnsom | Ok, if we don't have any more updates, I will move on. | 20:21 |
| johnsom | #topic Upgrade-checker community goal | 20:21 |
| *** openstack changes topic to "Upgrade-checker community goal (Meeting topic: Octavia)" | 20:21 | |
| johnsom | One of the two community goals for Stein is to have an "upgrade-checker" script like nova has. | 20:22 |
| johnsom | Frankly I think this is low hanging fruit if someone is looking for a small project. | 20:22 |
| johnsom | There are details here: | 20:22 |
| johnsom | #link http://lists.openstack.org/pipermail/openstack-dev/2018-August/133888.html | 20:22 |
| johnsom | Just wanted to raise awareness in case someone is looking for a little project. | 20:23 |
| johnsom | I don't think we have much that would need to go in there at the moment. | 20:23 |
| johnsom | #topic Open Discussion | 20:24 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 20:24 | |
| johnsom | Any other topics today? | 20:25 |
| cgoncalves | we've received couple of reports these past days that octavia can bring down all loadbalancers due to a DB outage | 20:26 |
| cgoncalves | #link https://storyboard.openstack.org/#!/story/2003575 | 20:26 |
| cgoncalves | I plan to work on this as soon as I can | 20:26 |
| cgoncalves | a traceback can be read here: https://bugzilla.redhat.com/show_bug.cgi?id=1603138#c4 | 20:27 |
| openstack | bugzilla.redhat.com bug 1603138 in openstack-octavia "Controller replacement with Octavia- After controller replacement amphora is in ERROR state" [Urgent,Closed: duplicate] - Assigned to cgoncalves | 20:27 |
| johnsom | Yeah, bummer. I have added a comment to that story with an idea of how to approach it | 20:27 |
| cgoncalves | examples of operations that can trigger this issue are: node reboot, db service restart, cloud upgrade (causing DB downtime) | 20:28 |
| cgoncalves | so, whatever causes DB downtime... | 20:28 |
| johnsom | Well, ideally you are running a DB cluster so minor things like a DB restart should not impact Octavia | 20:28 |
| nmagnezi | a tmp network disruption can also block the connection to DB | 20:29 |
| cgoncalves | yeah, I need to check why it still caused that on 3-controller HA deployments | 20:29 |
| johnsom | There is built in DB retries in the oslo DB layer, so short network blips should not trigger it either. Plus, with the default settings it would have to be unreachable for 60 seconds or more across all of the HMs | 20:30 |
| johnsom | Maybe there is a bug in galera (if that is what you are using for clustering) or oslo DB. We should do some testing. | 20:31 |
| johnsom | Either way, I think we should improve how the HMs handle the situation | 20:32 |
| cgoncalves | agreed. we should catch db_exc.DBConnectionError exceptions in health_check() | 20:32 |
| xgerman_ | sorry, had some prod issue to attend... | 20:32 |
| cgoncalves | xgerman_, let me guess. octavia brought down all LBs due to DB outage? xd | 20:33 |
| xgerman_ | no, unrelated to Octavia | 20:33 |
| nmagnezi | He had to look around for zombies | 20:33 |
| cgoncalves | ha! :) | 20:33 |
| *** luksky11 has joined #openstack-lbaas | 20:34 | |
| johnsom | cgoncalves Feel free to ping me if you want to bounce ideas around.... | 20:34 |
| cgoncalves | johnsom, I most certainly will, thanks :) | 20:35 |
| johnsom | Other topics for today? | 20:35 |
| xgerman_ | PTG? Did you plug the etherpad? | 20:36 |
| xgerman_ | ah saw it | 20:37 |
| johnsom | Yeah, I posted the normal reminder at the start of the meeting | 20:38 |
| xgerman_ | +1 | 20:38 |
| johnsom | xgerman_ lol, oh, we will be talking about Octavia..... (topic 18 in the etherpad) | 20:38 |
| cgoncalves | I like item "18. Octavia". let's discuss Octavia at the PTG xD | 20:38 |
| johnsom | Ok, if we don't have any more topics, I will close out the meeting. | 20:39 |
| *** abaindur_ has joined #openstack-lbaas | 20:40 | |
| johnsom | Ok, thanks folks! | 20:40 |
| johnsom | #endmeeting | 20:40 |
| *** openstack changes topic to "Discussion of OpenStack Load Balancing (Octavia) | https://etherpad.openstack.org/p/octavia-priority-reviews" | 20:40 | |
| openstack | Meeting ended Wed Aug 29 20:40:34 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:40 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-08-29-20.00.html | 20:40 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-08-29-20.00.txt | 20:40 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-08-29-20.00.log.html | 20:40 |
| *** abaindur has quit IRC | 20:41 | |
| *** yamamoto has quit IRC | 21:08 | |
| rm_work | I think I like yours better johnsom, now that we have the version stuff to work with :P | 21:12 |
| rm_work | (that didn't exist when I proposed mine) | 21:13 |
| johnsom | Cool. I think it's a good option too | 21:13 |
| *** celebdor1 has quit IRC | 21:16 | |
| johnsom | Survey Monkey is going IPO... wow | 21:19 |
| *** rcernin has joined #openstack-lbaas | 21:49 | |
| *** yamamoto has joined #openstack-lbaas | 21:53 | |
| *** fnaval has quit IRC | 21:57 | |
| *** abaindur_ has quit IRC | 22:02 | |
| *** abaindur has joined #openstack-lbaas | 22:03 | |
| *** yamamoto has quit IRC | 22:03 | |
| *** luksky11 has quit IRC | 22:03 | |
| *** abaindur has quit IRC | 22:07 | |
| *** threestrands has joined #openstack-lbaas | 22:19 | |
| *** threestrands has quit IRC | 22:19 | |
| *** threestrands has joined #openstack-lbaas | 22:22 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!