| *** yamamoto has quit IRC | 00:00 | |
| *** abaindur has quit IRC | 00:02 | |
| *** abaindur has joined #openstack-lbaas | 00:04 | |
| *** henriqueof has quit IRC | 00:15 | |
| *** abaindur has quit IRC | 00:49 | |
| *** abaindur has joined #openstack-lbaas | 00:55 | |
| *** Swami has quit IRC | 01:08 | |
| *** yamamoto has joined #openstack-lbaas | 01:09 | |
| *** yamamoto has quit IRC | 01:13 | |
| *** takamatsu_ has quit IRC | 01:39 | |
| *** takamatsu_ has joined #openstack-lbaas | 01:45 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 02:30 | |
| *** rcernin has quit IRC | 02:32 | |
| *** hongbin has joined #openstack-lbaas | 02:38 | |
| *** hongbin has quit IRC | 02:43 | |
| *** takamatsu_ has quit IRC | 02:53 | |
| *** yamamoto has joined #openstack-lbaas | 02:57 | |
| *** psachin has joined #openstack-lbaas | 03:01 | |
| *** abaindur has quit IRC | 03:01 | |
| *** yamamoto has quit IRC | 03:02 | |
| *** yamamoto has joined #openstack-lbaas | 03:15 | |
| *** yamamoto has quit IRC | 03:25 | |
| *** yamamoto has joined #openstack-lbaas | 03:25 | |
| *** ramishra has joined #openstack-lbaas | 03:57 | |
| *** abaindur has joined #openstack-lbaas | 05:34 | |
| *** ramishra has quit IRC | 05:43 | |
| *** ramishra has joined #openstack-lbaas | 05:45 | |
| *** ramishra has quit IRC | 05:55 | |
| *** ramishra has joined #openstack-lbaas | 06:03 | |
| *** ivve has joined #openstack-lbaas | 06:08 | |
| openstackgerrit | omkar telee proposed openstack/neutron-lbaas master: Feature correction: L7Policy/Rule for A10Networks https://review.openstack.org/639571 | 06:20 |
|---|---|---|
| *** Dinesh_Bhor has quit IRC | 06:58 | |
| *** ramishra has quit IRC | 06:58 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 07:01 | |
| *** ramishra has joined #openstack-lbaas | 07:01 | |
| *** Dinesh_Bhor has quit IRC | 07:12 | |
| *** ivve has quit IRC | 07:17 | |
| *** takamatsu_ has joined #openstack-lbaas | 07:21 | |
| *** yamamoto has quit IRC | 07:25 | |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/rocky: Fix grenade job to clone Octavia from base branch https://review.openstack.org/639349 | 07:28 |
| *** ivve has joined #openstack-lbaas | 07:34 | |
| *** gcheresh has joined #openstack-lbaas | 07:43 | |
| *** ccamposr has joined #openstack-lbaas | 07:45 | |
| *** yamamoto has joined #openstack-lbaas | 07:59 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 08:10 | |
| *** yboaron_ has joined #openstack-lbaas | 08:13 | |
| *** pcaruana has joined #openstack-lbaas | 08:13 | |
| *** Dinesh_Bhor has quit IRC | 08:14 | |
| *** yboaron_ has quit IRC | 08:18 | |
| *** yamamoto has quit IRC | 08:23 | |
| *** pcaruana has quit IRC | 08:28 | |
| *** takamatsu_ has quit IRC | 08:31 | |
| *** pcaruana has joined #openstack-lbaas | 08:42 | |
| *** abaindur has quit IRC | 08:46 | |
| *** pcaruana has quit IRC | 08:51 | |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/rocky: Enable debug for Octavia services in grenade job https://review.openstack.org/639599 | 08:54 |
| *** pcaruana has joined #openstack-lbaas | 08:58 | |
| *** pcaruana|afk| has joined #openstack-lbaas | 09:01 | |
| *** pcaruana has quit IRC | 09:03 | |
| *** ivve has quit IRC | 09:11 | |
| *** takamatsu has joined #openstack-lbaas | 09:15 | |
| *** ivve has joined #openstack-lbaas | 09:26 | |
| *** sapd1 has quit IRC | 09:33 | |
| *** takamatsu has quit IRC | 09:44 | |
| *** yamamoto has joined #openstack-lbaas | 10:09 | |
| *** yamamoto has quit IRC | 10:14 | |
| *** yamamoto has joined #openstack-lbaas | 10:16 | |
| Adri2000 | johnsom: redeploying horizon from scatch (made easy by openstack-ansible as horizon is running in a dedicated lxc container I could just drop/recreate) fixed my issue :) thanks again | 10:18 |
| *** salmankhan has joined #openstack-lbaas | 10:32 | |
| *** Dinesh_Bhor has joined #openstack-lbaas | 10:57 | |
| *** yamamoto has quit IRC | 10:58 | |
| *** yamamoto has joined #openstack-lbaas | 11:00 | |
| *** Dinesh_Bhor has quit IRC | 11:00 | |
| *** gcheresh_ has joined #openstack-lbaas | 11:10 | |
| *** gcheresh has quit IRC | 11:10 | |
| *** sapd1 has joined #openstack-lbaas | 11:11 | |
| *** gcheresh_ has quit IRC | 11:22 | |
| *** takamatsu has joined #openstack-lbaas | 11:24 | |
| *** ivve has quit IRC | 12:00 | |
| *** ramishra has quit IRC | 12:04 | |
| *** ramishra has joined #openstack-lbaas | 12:26 | |
| *** gcheresh_ has joined #openstack-lbaas | 12:54 | |
| *** ivve has joined #openstack-lbaas | 12:57 | |
| *** yamamoto has quit IRC | 13:00 | |
| *** pcaruana|afk| has quit IRC | 13:09 | |
| *** celebdor has joined #openstack-lbaas | 13:18 | |
| *** yamamoto has joined #openstack-lbaas | 13:36 | |
| *** yamamoto has quit IRC | 13:41 | |
| *** yamamoto has joined #openstack-lbaas | 13:42 | |
| *** henriqueof has joined #openstack-lbaas | 13:45 | |
| *** yamamoto has quit IRC | 14:00 | |
| *** yamamoto has joined #openstack-lbaas | 14:03 | |
| *** yamamoto has quit IRC | 14:03 | |
| *** fnaval has quit IRC | 14:20 | |
| *** psachin has quit IRC | 14:27 | |
| *** yamamoto has joined #openstack-lbaas | 14:38 | |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/rocky: Fix grenade job to clone Octavia from base branch https://review.openstack.org/639349 | 14:38 |
| *** Adri2000 has quit IRC | 14:43 | |
| *** yamamoto has quit IRC | 14:43 | |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/rocky: Fix grenade job to clone Octavia from base branch https://review.openstack.org/639349 | 14:46 |
| *** fnaval has joined #openstack-lbaas | 14:47 | |
| *** pcaruana has joined #openstack-lbaas | 14:57 | |
| *** gcheresh_ has quit IRC | 15:02 | |
| *** yamamoto has joined #openstack-lbaas | 15:05 | |
| *** yamamoto has quit IRC | 15:09 | |
| *** cbrumm_ has quit IRC | 15:30 | |
| *** cbrumm_ has joined #openstack-lbaas | 15:32 | |
| *** dmellado has quit IRC | 15:42 | |
| *** dmellado has joined #openstack-lbaas | 15:43 | |
| *** sapd1 has quit IRC | 15:46 | |
| *** gcheresh_ has joined #openstack-lbaas | 15:47 | |
| openstackgerrit | Vlad Gusev proposed openstack/octavia stable/rocky: Fix grenade job to clone Octavia from base branch https://review.openstack.org/639349 | 15:58 |
| *** s10 has joined #openstack-lbaas | 15:59 | |
| *** ivve has quit IRC | 16:01 | |
| s10 | Should the octavia-grenade job in Octavia stable/rocky become non-voting? It's fails for 2.5 month because something in stable/queens. | 16:03 |
| johnsom | I think it is being worked on and it needs to get fixed. I am inclined to leave it for now. | 16:05 |
| *** takamatsu has quit IRC | 16:09 | |
| *** gcheresh_ has quit IRC | 16:09 | |
| *** s10 has quit IRC | 16:16 | |
| *** ramishra has quit IRC | 16:25 | |
| *** dmellado has quit IRC | 16:32 | |
| *** yamamoto has joined #openstack-lbaas | 16:32 | |
| *** dmellado has joined #openstack-lbaas | 16:34 | |
| openstackgerrit | Vlad Gusev proposed openstack/octavia master: WIP: Add support for the oslo_middleware.http_proxy_to_wsgi https://review.openstack.org/639736 | 16:34 |
| *** yamamoto has quit IRC | 16:37 | |
| cgoncalves | I still need to test it but likely the patch that broke stable/queens grenade job was https://review.openstack.org/#/c/624804/ | 16:43 |
| cgoncalves | http://logs.openstack.org/49/639349/5/check/octavia-grenade/461ebf7/logs/screen-o-cw.txt.gz?level=WARNING#_Feb_27_08_32_43_986674 | 16:44 |
| *** pcaruana has quit IRC | 16:57 | |
| *** rtjure has quit IRC | 17:00 | |
| *** ccamposr has quit IRC | 17:00 | |
| *** ccamposr has joined #openstack-lbaas | 17:01 | |
| *** celebdor has quit IRC | 17:23 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add 2 new fields into Pool API for support re-encryption https://review.openstack.org/614447 | 17:25 |
| johnsom | cgoncalves Someone in the horizon meeting today said you were working on this: https://storyboard.openstack.org/#!/story/2005101 | 17:26 |
| johnsom | I created a story for you | 17:26 |
| *** ccamposr has quit IRC | 17:27 | |
| *** trown is now known as trown|lunch | 17:28 | |
| *** dims has quit IRC | 17:35 | |
| *** ivve has joined #openstack-lbaas | 17:36 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Pool support sni cert for backend re-encryption https://review.openstack.org/614432 | 17:38 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add 2 new fields into Pool API for support re-encryption https://review.openstack.org/614447 | 17:39 |
| *** takamatsu has joined #openstack-lbaas | 17:41 | |
| *** yamamoto has joined #openstack-lbaas | 17:41 | |
| *** yamamoto has quit IRC | 17:46 | |
| *** dims has joined #openstack-lbaas | 17:48 | |
| *** takamatsu has quit IRC | 18:01 | |
| *** yamamoto has joined #openstack-lbaas | 18:07 | |
| *** yamamoto has quit IRC | 18:12 | |
| *** trown|lunch is now known as trown | 18:25 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add 2 new fields into Pool API for support re-encryption https://review.openstack.org/614447 | 18:34 |
| *** salmankhan has quit IRC | 18:40 | |
| *** ivve has quit IRC | 18:43 | |
| *** takamatsu has joined #openstack-lbaas | 18:52 | |
| *** yamamoto has joined #openstack-lbaas | 19:32 | |
| *** yamamoto has quit IRC | 19:37 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Fix parallel plug vip https://review.openstack.org/638992 | 19:42 |
| *** celebdor has joined #openstack-lbaas | 19:54 | |
| johnsom | #startmeeting Octavia | 20:00 |
| openstack | Meeting started Wed Feb 27 20:00:03 2019 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 20:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 20:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 20:00 | |
| openstack | The meeting name has been set to 'octavia' | 20:00 |
| johnsom | Hi folks | 20:00 |
| colin- | hi | 20:00 |
| nmagnezi | o/ | 20:00 |
| johnsom | #topic Announcements | 20:00 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 20:00 | |
| cgoncalves | hi | 20:01 |
| johnsom | The TC elections are on. You should have received an e-mail with your link to the ballot. | 20:01 |
| *** henriqueof has quit IRC | 20:01 | |
| johnsom | The octavia-lib feature freeze is now in effect. | 20:01 |
| johnsom | I have also released version 1.1.0 for Stein with our recent updates. | 20:01 |
| colin- | nice | 20:02 |
| johnsom | And the most important, NEXT WEEK IS FEATURE FREEZE FOR EVERYTHING ELSE | 20:02 |
| johnsom | As usual, we are working against the priority list: | 20:03 |
| johnsom | #link https://etherpad.openstack.org/p/octavia-priority-reviews | 20:03 |
| johnsom | Any other announcements today? | 20:03 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:04 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 20:04 | |
| johnsom | I have mostly been focused on the TLS patch chains. The TLS client authentication patches have now merged. They work well in my testing. | 20:04 |
| johnsom | I'm currently working on the backend re-encyrption chain. I hope I can finish that up today, give it a test, and we can get that merged too. | 20:05 |
| johnsom | If all goes well, I might try to help the volume backed storage patch and see if we can get it working for Stein. I created a test gate, but the patch fails... | 20:06 |
| johnsom | Any other updates? | 20:07 |
| cgoncalves | I have been working on multiple fronts | 20:07 |
| xgerman | o/ | 20:07 |
| cgoncalves | 1. RHEL 8 DIB and amphora support (tempest tests passing) | 20:07 |
| cgoncalves | #link https://review.openstack.org/#/c/623137/ | 20:07 |
| colin- | appreciate the oslo merge, rebuilt and running at that point in master now | 20:07 |
| cgoncalves | #link https://review.openstack.org/#/c/638581/ | 20:07 |
| cgoncalves | 2. Allow ERROR'd load balancers to be failed over | 20:08 |
| cgoncalves | #link #link https://review.openstack.org/#/c/638790/ | 20:08 |
| cgoncalves | 3. iptables-based active-standby tempest test | 20:08 |
| cgoncalves | #link https://review.openstack.org/#/c/637073/ | 20:08 |
| cgoncalves | 4. general bug fix backports | 20:08 |
| johnsom | Cool, thank you for working on the backports! | 20:09 |
| xgerman | +1 | 20:09 |
| cgoncalves | stable/rocky grenade job is sadly still broken. I apologize for not having invested much time on it yet | 20:09 |
| johnsom | That is next on the agenda, I wanted to check in on that issue. | 20:10 |
| johnsom | #topic Status of the Rocky grenade gate | 20:10 |
| *** openstack changes topic to "Status of the Rocky grenade gate (Meeting topic: Octavia)" | 20:10 | |
| johnsom | I just wanted to get an update on that. I saw your note earlier about a potential cause. | 20:11 |
| cgoncalves | right | 20:11 |
| cgoncalves | #link https://review.openstack.org/#/c/639395/ | 20:11 |
| johnsom | Are you actively working on that or is it an open item? | 20:11 |
| cgoncalves | ^ this backport allow us now to see what's going on wrong creating a member | 20:11 |
| cgoncalves | that is where the grenade job is failing on | 20:11 |
| cgoncalves | the error is: http://logs.openstack.org/49/639349/5/check/octavia-grenade/461ebf7/logs/screen-o-cw.txt.gz?level=WARNING#_Feb_27_08_32_43_986674 | 20:11 |
| cgoncalves | the rocky grenade job started failing between Dec 14-17 if I got that right | 20:12 |
| cgoncalves | so I'm thinking if https://review.openstack.org/#/c/624804/ is what introduced the regression | 20:13 |
| cgoncalves | the member create call fails still on queens, not rocky | 20:13 |
| xgerman | with all those regressions looks like we are lacking gates | 20:13 |
| cgoncalves | xgerman, speaking of that, your VIP refactor patch partially broke active-standby in master :P | 20:14 |
| xgerman | I put up a fix | 20:14 |
| johnsom | Yeah, not sure how the scenario tests pased but grenade is not. | 20:14 |
| cgoncalves | xgerman, I don't see it. we can chat about that after grenade | 20:14 |
| johnsom | xgerman It looks like in my rush I forgot to switch it off of amphorae.... | 20:15 |
| johnsom | lol | 20:15 |
| xgerman | yeah, two small changes and it came up on my devstack | 20:15 |
| cgoncalves | xgerman, ah, I see it now. you submitted a new PS to Michael's change | 20:16 |
| xgerman | yep | 20:16 |
| johnsom | Cool, I just rechecked my act/stdby patch which is setup to test taht | 20:16 |
| cgoncalves | #link https://review.openstack.org/#/c/638992/ | 20:16 |
| johnsom | #link https://review.openstack.org/#/c/584681 | 20:16 |
| johnsom | Ok, so cgoncalves you are actively working on the grenade issue? | 20:17 |
| cgoncalves | johnsom, I will starting actively tomorrow, yes | 20:17 |
| johnsom | Ok, cool. Thanks. Just wanted to make sure we didn't think each other was looking at it, when in reality none of us were.... | 20:18 |
| johnsom | #topic Open Discussion | 20:18 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 20:18 | |
| johnsom | I have one open discussion topic, but will open the floor up first to other discussions | 20:18 |
| cgoncalves | I'm sure you'll be looking at it too, at least reviewing ;) | 20:18 |
| johnsom | Other topics today? | 20:19 |
| johnsom | Ok, then I will go. | 20:19 |
| colin- | would like to soicit guidance | 20:19 |
| colin- | very briefly | 20:19 |
| johnsom | Sure, go ahead colin- | 20:19 |
| colin- | an increaing number of internal customers are asking about the performance capabilities of the VIPs we create with octavia, and we're going to endeavor to measure that really carefully in terms of average latency, connection concurrency, and throughput (as these all vary dramatically based on cloud hw) | 20:20 |
| johnsom | Yes, I did a similar exercise last year. | 20:21 |
| colin- | so, aside from economies of scale with multiple tcp/udp/http listeners, does anyone have advice on how to capture this information really effectively with octavia and its amphorae? | 20:21 |
| colin- | and i'm hoping to use this same approach to measure the benefits of various nova flavors and haproxy configruations later in stein | 20:21 |
| openstackgerrit | Vlad Gusev proposed openstack/octavia master: Add support for the oslo_middleware.http_proxy_to_wsgi https://review.openstack.org/639736 | 20:23 |
| johnsom | Yeah, so I setup a lab, had three hosts for traffic generation, three for content serving. one for the amp | 20:23 |
| johnsom | I used iperf3 for the TCP (L4) tests and tsung for the HTTP tests | 20:23 |
| openstackgerrit | Vlad Gusev proposed openstack/octavia master: Add support for the oslo_middleware.http_proxy_to_wsgi https://review.openstack.org/639736 | 20:23 |
| johnsom | I wrote a custom module for nginx (ugh, but it was easy) that returned static buffers. | 20:23 |
| colin- | did you add any monitoring/observability tools for visualizing? | 20:24 |
| openstackgerrit | Vlad Gusev proposed openstack/octavia master: Add support for the oslo_middleware http_proxy_to_wsgi https://review.openstack.org/639736 | 20:24 |
| colin- | or was shell output sufficient for your purposes | 20:24 |
| johnsom | I did one series where traffic crossed hosts, one with everything on one host (eliminates the neutron issues). | 20:24 |
| johnsom | tsung comes with reporting tools | 20:24 |
| colin- | oh ok | 20:25 |
| johnsom | I also did some crossing a neutron router vs. all L2 | 20:25 |
| johnsom | Then it's just a bunch of time tweaking all of the knobs | 20:25 |
| colin- | good feedback, thank you | 20:25 |
| johnsom | For the same-host tests, iperf3 with 20 parallel flows, 1vcpu, 1GB ram, 2GB disk did ~14gbps | 20:26 |
| johnsom | But of course your hardware, cloud config, butterflys flapping wings is Tahiti, all impacts what you get. | 20:27 |
| johnsom | caveat, caveat, caveat..... | 20:27 |
| colin- | yeah indeed. if anyone else has done this differently or tested different hardware NICs this way please lmk! that's all i had | 20:28 |
| johnsom | Yeah, get ready to add a ton of ****** | 20:28 |
| johnsom | for all the caveats | 20:28 |
| johnsom | I can share the nginx hack code too if you decide you want it. | 20:29 |
| johnsom | Ok, so we have this issue where if people kill -9 the controller processes we can leave objects in PENDING_* | 20:30 |
| xgerman | also are you running the vip on an overlay? Or dedicated vlan, etc. | 20:30 |
| johnsom | I have an idea for an interim solution until we do jobboard/resumption. | 20:31 |
| xgerman | johnsom: that type of thing was supposed to get fixed when we adopt job-board | 20:31 |
| johnsom | lol, yeah, that | 20:31 |
| xgerman | our task-(flow) engine should have a way to deal with that | 20:31 |
| xgerman | that’s why we went with an engine | 20:31 |
| johnsom | It does, in fact multiple ways, but that will take some development time to address IMO | 20:32 |
| johnsom | So, as a short term, interim fix I was thinking that we could have the processes create a UUID unique to it's instance, write that out to a file somewhere, then check it on startup and mark anything it "owned" as ERROR. | 20:33 |
| johnsom | Thoughts? Comments? | 20:33 |
| cgoncalves | if $time.now() > $last_updated_time+$timeout -> ERROR? | 20:33 |
| johnsom | The hardest part is where to write the file.... | 20:33 |
| johnsom | It would require a DB schema change, which we would want to get in before feature freeze (just to be nice for upgrades, etc.). So thought I would throw the idea out now. | 20:34 |
| johnsom | I think the per-process UUID would be more reliable than trying to do a timeout. | 20:36 |
| openstackgerrit | Vlad Gusev proposed openstack/octavia master: Add support for the oslo_middleware http_proxy_to_wsgi https://review.openstack.org/639736 | 20:36 |
| cgoncalves | hmmm | 20:37 |
| cgoncalves | what then flipping status to PENDING_UPDATE? maybe only valid to certain resources | 20:37 |
| johnsom | The only downside is we don't have a /var/lib/octavia on the controllers today, so it's an upgrade/packaging issue | 20:38 |
| cgoncalves | and not backportable | 20:38 |
| johnsom | Right, the "don't do that" still applies to older releases | 20:39 |
| johnsom | I didn't follow the PENDING_UPDATE comment | 20:39 |
| cgoncalves | nah, never mind. it prolly doesn't make any sense anyway xD (I was thinking along the same lines of allowing ERROR'd LBs to be failed over) | 20:40 |
| johnsom | It would have to flip them to ERROR because we don't know where in the flow they killed it | 20:40 |
| johnsom | Yeah, maybe a follow on could attempt to "fix" it, but that is again logic to identify where it died. Which is starting the work on jobboard/resumption. | 20:41 |
| cgoncalves | thinking of a backportable solution, wouldn't timeouts suffice? | 20:41 |
| johnsom | I don't like that approach for a few reasons. We seem to have widely varying performance in the field, so picking the right number would be hard, sort of making it an hour or something, which defeats the purpose of a timely cleanup | 20:42 |
| xgerman | mmh, people would likley be happy if we just flip PENDING to ERROR with the housekeeper after a while | 20:43 |
| johnsom | I mean we already have flows that timeout after 25 minutes due to some deployments, so it would have to be longer than that. | 20:43 |
| xgerman | some operators tend to trade resources for less work… so there’s that | 20:43 |
| johnsom | Yeah, the nice thing about the UUID too is it shames the operator for kill -9 | 20:44 |
| johnsom | We know exactly what happened | 20:44 |
| xgerman | or for having servers explode | 20:44 |
| xgerman | or poweswitch istakes | 20:44 |
| cgoncalves | also more and more clouds run services in containers, so docker restart would basically mean kill -9 | 20:44 |
| xgerman | yep | 20:44 |
| johnsom | Yep, k8s is horrible | 20:45 |
| cgoncalves | you don't need k8s to run services in containers ;) | 20:45 |
| colin- | stop, my eyes will roll out of my head | 20:45 |
| cgoncalves | I mean openstack services! | 20:45 |
| xgerman | yeah, we should rewrite octavia as a function-as-a-service | 20:45 |
| johnsom | I know, but running the openstack control plane in k8s means lots of random kills | 20:46 |
| colin- | indeed | 20:46 |
| xgerman | so how difficult is job board? did we ever look into the effort? | 20:46 |
| johnsom | Anyway, this is an option, yes, may not solve all of the ills. | 20:46 |
| johnsom | Yeah, we did, it's going to probably be a cycles worth of effort to go full job board. | 20:47 |
| johnsom | There might be a not-so-full job board that would meet our needs too, but that again is going to take some time. | 20:47 |
| xgerman | I would rather start on the “right” solution then do crudges | 20:48 |
| cgoncalves | I was unaware of jobboards until now. does it sync state across multiple controller nodes? | 20:48 |
| johnsom | not really, but accomplishes the same thing. | 20:48 |
| cgoncalves | asking because if octavia worker N on node X goes down, worker N+1 on node X+1 takes over | 20:49 |
| johnsom | So first it enables persistence of the flow data. I uses a set of "worker" processes. The main jobboard assigns and monitors the workers completion of each task | 20:49 |
| johnsom | Right, effectively that is what happens. | 20:50 |
| cgoncalves | without a syncing mechanism, how would octavia know which pending resources to ERROR? | 20:50 |
| xgerman | do we need a zookeeper for jobboard. Yuck! | 20:50 |
| johnsom | Much of the state is stored in the DB | 20:50 |
| cgoncalves | ok | 20:50 |
| colin- | jobboard = ?, for the uninitiated | 20:50 |
| johnsom | Yeah, so there was a locking requirement I remember from the analysis. I don't think zookeeper was the only option, but maybe | 20:50 |
| colin- | is this a work tracking tool? | 20:50 |
| colin- | ah, disregard | 20:51 |
| xgerman | https://docs.openstack.org/taskflow/ocata/jobs.html | 20:51 |
| johnsom | #link https://docs.openstack.org/taskflow/latest/user/jobs.html | 20:52 |
| johnsom | Anyway, I didn't want to go deep on the future solution. | 20:52 |
| johnsom | What I am hearing is we would prefer to leave this issue until we have resources to work on the full solution and that an interim solution is not valuable | 20:52 |
| xgerman | #vote? | 20:53 |
| cgoncalves | I still didn't get why timeouts wouldn't be a good interim (and backportable) solution | 20:53 |
| johnsom | What would you pick as a timeout? | 20:53 |
| cgoncalves | what ever is in the config file | 20:54 |
| johnsom | We know some clouds complete tasks in less than a minute, others it takes over 20 | 20:54 |
| cgoncalves | if load creation: build timeout + heartbeat timeout | 20:54 |
| cgoncalves | otherwise, just heartbeat timeout. no? | 20:54 |
| johnsom | So 26 minutes? | 20:54 |
| cgoncalves | better than forever and ever | 20:55 |
| cgoncalves | and not being able to delete/error | 20:55 |
| johnsom | I don't think we can backport this even if it has a timeout really | 20:55 |
| johnsom | The timeout would be a new feature to the housekeeping process | 20:56 |
| cgoncalves | no API or DB schema changes. no new config option | 20:56 |
| johnsom | The other thing that worries me about timeouts is folks setting it and not understanding the ramifications | 20:56 |
| colin- | yeah that's tricky, i too don't want to leave them (forever) in the state where they can't be deleted | 20:56 |
| cgoncalves | it would be a new periodic in housekeeping | 20:56 |
| *** henriqueof has joined #openstack-lbaas | 20:57 | |
| xgerman | yeah, I am hunted by untuned tieouts almost every day | 20:57 |
| colin- | xgerman: thanks for the link | 20:58 |
| johnsom | Yep. I think it breaks the risk of regression and self-contained rules | 20:58 |
| johnsom | And certainly the "New feature" rule | 20:58 |
| johnsom | Well, we are about out of time. Thanks folks. | 20:59 |
| cgoncalves | "Fix an issue where resources could eternally be left in a transient state" ;) | 20:59 |
| johnsom | If you all want to talk about job board more, let me know and I can put it on the agenda. | 20:59 |
| cgoncalves | I will certainly read more about it | 20:59 |
| johnsom | I just think it's a super dangerous thing in our model to change the state out from under other processes | 21:00 |
| johnsom | #endmeeting | 21:00 |
| *** openstack changes topic to "Discussions for Octavia | Stein priority review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 21:00 | |
| openstack | Meeting ended Wed Feb 27 21:00:25 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-02-27-20.00.html | 21:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-02-27-20.00.txt | 21:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-02-27-20.00.log.html | 21:00 |
| johnsom | Maybe I will propose the code and you all can vote on the patch. | 21:01 |
| *** ivve has joined #openstack-lbaas | 21:03 | |
| cgoncalves | nice | 21:04 |
| cgoncalves | I am not against jobboard at all. I was just considering of a quick and interim fix we could also backport | 21:05 |
| * cgoncalves disconnects | 21:05 | |
| johnsom | Yeah, jobboard is the end goal, but a lot of work. I was going for an interim solution that reliably improves the situation for Stein forward | 21:05 |
| *** abaindur has joined #openstack-lbaas | 21:12 | |
| openstackgerrit | Brian Haley proposed openstack/neutron-lbaas master: Update neutron quota_driver path https://review.openstack.org/639829 | 21:13 |
| *** ivve has quit IRC | 21:14 | |
| openstackgerrit | Vlad Gusev proposed openstack/octavia master: Add support for the oslo_middleware http_proxy_to_wsgi https://review.openstack.org/639736 | 21:18 |
| *** yamamoto has joined #openstack-lbaas | 21:20 | |
| *** yamamoto has quit IRC | 21:25 | |
| *** abaindur has quit IRC | 21:37 | |
| *** celebdor has quit IRC | 21:58 | |
| rm_work | Ah right, meeting :/ | 22:09 |
| rm_work | Had an internal meeting exactly overlap | 22:09 |
| *** trown is now known as trown|outtypewww | 22:15 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Fix parallel plug vip https://review.openstack.org/638992 | 22:56 |
| *** celebdor has joined #openstack-lbaas | 22:58 | |
| *** yamamoto has joined #openstack-lbaas | 23:00 | |
| *** rcernin has joined #openstack-lbaas | 23:06 | |
| *** sapd1 has joined #openstack-lbaas | 23:06 | |
| *** celebdor has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!