| *** annp has quit IRC | 00:50 | |
| *** annp has joined #openstack-lbaas | 00:51 | |
| *** yamamoto has joined #openstack-lbaas | 00:53 | |
| *** yamamoto has quit IRC | 00:59 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Creates provider driver base class and exceptions https://review.openstack.org/558013 | 00:59 |
|---|---|---|
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Create noop provider driver and data model https://review.openstack.org/558320 | 01:02 |
| *** annp has quit IRC | 01:12 | |
| *** annp has joined #openstack-lbaas | 01:13 | |
| *** sapd has joined #openstack-lbaas | 01:42 | |
| *** dayou has quit IRC | 01:44 | |
| *** harlowja has quit IRC | 01:50 | |
| rm_work | johnsom: so someday soon you can get to reviews? :P | 01:54 |
| rm_work | <3 | 01:54 |
| openstackgerrit | Jacky Hu proposed openstack/octavia-dashboard master: Use pool name as hint for selecting pool id https://review.openstack.org/553124 | 01:54 |
| *** yamamoto has joined #openstack-lbaas | 01:55 | |
| openstackgerrit | Jacky Hu proposed openstack/octavia-dashboard master: Align model with v2 api https://review.openstack.org/554198 | 01:56 |
| openstackgerrit | Jacky Hu proposed openstack/octavia-dashboard master: Being able to change insert headers of listener https://review.openstack.org/549999 | 01:56 |
| *** dayou has joined #openstack-lbaas | 01:58 | |
| *** yamamoto has quit IRC | 02:00 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 02:02 | |
| johnsom | rm_work: ha, it doesn’t seem it was that long ago I did a bunch of reviews... I will try to take a look again tomorrow. | 02:06 |
| *** AlexeyAbashkin has quit IRC | 02:06 | |
| johnsom | Hammering out the noop driver has me a bit fried tonight. | 02:06 |
| johnsom | However there are other cores coming online tonight, so... grin | 02:07 |
| *** harlowja has joined #openstack-lbaas | 02:31 | |
| *** longkb__ has joined #openstack-lbaas | 02:41 | |
| *** wolsen has joined #openstack-lbaas | 02:53 | |
| *** yamamoto has joined #openstack-lbaas | 02:56 | |
| *** yamamoto has quit IRC | 03:02 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 03:04 | |
| *** AlexeyAbashkin has quit IRC | 03:09 | |
| *** links has joined #openstack-lbaas | 03:32 | |
| *** yamamoto has joined #openstack-lbaas | 03:33 | |
| rm_work | johnsom: yeah true. ;) xgerman_ did review a few of mine but i am not sure i agree with any of his -1s lol | 03:50 |
| xgerman_ | Ha, you can argue and maybe other cores give you a +2 | 03:54 |
| *** harlowja has quit IRC | 03:55 | |
| *** annp has quit IRC | 03:57 | |
| *** annp has joined #openstack-lbaas | 03:58 | |
| openstackgerrit | wangqi proposed openstack/octavia master: Replace db function get_session by get_writer_session https://review.openstack.org/558348 | 04:03 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add usage admin resource https://review.openstack.org/557548 | 04:25 |
| *** yamamoto has quit IRC | 04:45 | |
| *** yamamoto has joined #openstack-lbaas | 04:46 | |
| *** ivve has quit IRC | 05:23 | |
| *** longkb_ has joined #openstack-lbaas | 05:38 | |
| *** longkb__ has quit IRC | 05:40 | |
| *** Alex_Staf has joined #openstack-lbaas | 05:54 | |
| *** longkb__ has joined #openstack-lbaas | 06:02 | |
| *** longkb__ has quit IRC | 06:06 | |
| *** longkb__ has joined #openstack-lbaas | 06:11 | |
| *** pcaruana has joined #openstack-lbaas | 06:53 | |
| *** voelzmo has joined #openstack-lbaas | 06:57 | |
| *** ivve has joined #openstack-lbaas | 06:57 | |
| *** tesseract has joined #openstack-lbaas | 07:00 | |
| *** velizarx has joined #openstack-lbaas | 07:02 | |
| *** imacdonn has quit IRC | 07:04 | |
| *** imacdonn has joined #openstack-lbaas | 07:04 | |
| *** links has quit IRC | 07:05 | |
| *** links has joined #openstack-lbaas | 07:05 | |
| *** ivve has quit IRC | 07:15 | |
| *** voelzmo has quit IRC | 07:16 | |
| *** velizarx has quit IRC | 07:17 | |
| *** ivve has joined #openstack-lbaas | 07:28 | |
| *** velizarx has joined #openstack-lbaas | 07:31 | |
| *** voelzmo has joined #openstack-lbaas | 07:38 | |
| *** ivve has quit IRC | 07:38 | |
| *** yamamoto_ has joined #openstack-lbaas | 07:42 | |
| *** yamamoto has quit IRC | 07:45 | |
| *** voelzmo has quit IRC | 07:50 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 07:53 | |
| *** links has quit IRC | 08:04 | |
| *** dmellado has quit IRC | 08:08 | |
| *** dmellado has joined #openstack-lbaas | 08:08 | |
| *** links has joined #openstack-lbaas | 08:14 | |
| *** voelzmo has joined #openstack-lbaas | 08:20 | |
| *** ivve has joined #openstack-lbaas | 08:23 | |
| *** voelzmo has quit IRC | 08:25 | |
| *** voelzmo has joined #openstack-lbaas | 08:31 | |
| *** gokhan_ has quit IRC | 09:00 | |
| *** annp has quit IRC | 09:04 | |
| *** annp has joined #openstack-lbaas | 09:05 | |
| *** gokhan_ has joined #openstack-lbaas | 09:05 | |
| *** salmankhan has joined #openstack-lbaas | 09:14 | |
| *** voelzmo has quit IRC | 09:16 | |
| *** voelzmo has joined #openstack-lbaas | 09:16 | |
| *** dmellado has quit IRC | 09:23 | |
| *** dmellado has joined #openstack-lbaas | 09:23 | |
| *** voelzmo has quit IRC | 09:32 | |
| *** voelzmo has joined #openstack-lbaas | 09:32 | |
| *** voelzmo has quit IRC | 09:33 | |
| *** voelzmo has joined #openstack-lbaas | 09:33 | |
| *** voelzmo has quit IRC | 09:33 | |
| *** voelzmo has joined #openstack-lbaas | 09:36 | |
| *** voelzmo has quit IRC | 09:37 | |
| *** voelzmo has joined #openstack-lbaas | 09:37 | |
| *** voelzmo has quit IRC | 09:37 | |
| *** salmankhan has quit IRC | 09:42 | |
| *** salmankhan has joined #openstack-lbaas | 09:52 | |
| *** voelzmo has joined #openstack-lbaas | 10:09 | |
| *** voelzmo has quit IRC | 10:14 | |
| *** voelzmo has joined #openstack-lbaas | 10:26 | |
| *** voelzmo has quit IRC | 10:55 | |
| *** voelzmo has joined #openstack-lbaas | 11:00 | |
| *** ianychoi_ has joined #openstack-lbaas | 11:11 | |
| *** ianychoi has quit IRC | 11:14 | |
| *** rcernin has quit IRC | 11:20 | |
| *** salmankhan1 has joined #openstack-lbaas | 11:27 | |
| *** salmankhan has quit IRC | 11:29 | |
| *** salmankhan1 is now known as salmankhan | 11:29 | |
| *** salmankhan has quit IRC | 11:31 | |
| *** velizarx has quit IRC | 11:31 | |
| *** salmankhan has joined #openstack-lbaas | 11:34 | |
| *** voelzmo has quit IRC | 11:36 | |
| *** salmankhan1 has joined #openstack-lbaas | 11:36 | |
| *** voelzmo has joined #openstack-lbaas | 11:37 | |
| *** voelzmo has quit IRC | 11:37 | |
| *** voelzmo has joined #openstack-lbaas | 11:37 | |
| *** salmankhan has quit IRC | 11:38 | |
| *** salmankhan1 is now known as salmankhan | 11:38 | |
| *** voelzmo has quit IRC | 11:38 | |
| *** voelzmo has joined #openstack-lbaas | 11:39 | |
| *** voelzmo has quit IRC | 11:39 | |
| *** yamamoto_ has quit IRC | 11:39 | |
| *** voelzmo has joined #openstack-lbaas | 11:39 | |
| *** voelzmo has quit IRC | 11:39 | |
| *** velizarx has joined #openstack-lbaas | 11:40 | |
| *** pratrik has joined #openstack-lbaas | 11:46 | |
| *** velizarx has quit IRC | 11:56 | |
| *** voelzmo has joined #openstack-lbaas | 12:12 | |
| *** voelzmo has quit IRC | 12:17 | |
| *** velizarx has joined #openstack-lbaas | 12:18 | |
| pratrik | Hi guys, I just created a small patch that makes it possible to specify multiple az's for Nova. The reasoning behind this is simply, if you have multiple az's in your openstack cluster, you may want to put them in separate az's (because of underlying storage for example). The patch is pretty straight forward and shouldn't really break anything. I've tried it out on our cluster and it works as intended. It seemed like a "bi | 12:18 |
| pratrik | was hoping that someone, if the patch makes sense, would review it and maybe apply it the "official way". | 12:19 |
| pratrik | http://paste.openstack.org/show/718281/ | 12:19 |
| *** longkb__ has quit IRC | 12:21 | |
| *** voelzmo has joined #openstack-lbaas | 12:25 | |
| *** voelzmo has quit IRC | 12:29 | |
| *** voelzmo has joined #openstack-lbaas | 12:29 | |
| *** salmankhan has quit IRC | 12:32 | |
| *** salmankhan has joined #openstack-lbaas | 12:34 | |
| *** yamamoto has joined #openstack-lbaas | 12:40 | |
| *** yamamoto has quit IRC | 12:45 | |
| *** yamamoto has joined #openstack-lbaas | 13:32 | |
| *** yamamoto has quit IRC | 13:36 | |
| *** fnaval has quit IRC | 13:37 | |
| *** fnaval has joined #openstack-lbaas | 13:37 | |
| *** fnaval has quit IRC | 13:42 | |
| *** fnaval has joined #openstack-lbaas | 13:50 | |
| *** yamamoto has joined #openstack-lbaas | 14:09 | |
| *** Alex_Staf has quit IRC | 14:13 | |
| *** voelzmo has quit IRC | 14:19 | |
| *** voelzmo has joined #openstack-lbaas | 14:19 | |
| *** voelzmo has quit IRC | 14:20 | |
| *** voelzmo has joined #openstack-lbaas | 14:20 | |
| *** voelzmo has quit IRC | 14:20 | |
| *** voelzmo has joined #openstack-lbaas | 14:21 | |
| *** voelzmo has quit IRC | 14:21 | |
| *** voelzmo has joined #openstack-lbaas | 14:21 | |
| *** voelzmo has quit IRC | 14:22 | |
| *** voelzmo has joined #openstack-lbaas | 14:22 | |
| *** voelzmo has quit IRC | 14:23 | |
| *** ivve has quit IRC | 14:37 | |
| *** links has quit IRC | 14:37 | |
| *** ptoohill has quit IRC | 14:47 | |
| *** ptoohill has joined #openstack-lbaas | 14:47 | |
| *** velizarx has quit IRC | 14:53 | |
| cgoncalves | johnsom: hi. am I missing something or https://github.com/openstack/octavia/blob/master/octavia/cmd/agent.py#L75-L76 are not being used anywhere? | 15:03 |
| johnsom | They should be, let me have a quick look | 15:04 |
| johnsom | I mean, I'm 99% sure they are being used as I have looked at that endpoint over TLS recently | 15:05 |
| johnsom | cgoncalves http://docs.gunicorn.org/en/latest/settings.html#certfile | 15:08 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/cmd/agent.py#L35 | 15:08 |
| johnsom | The "options" are passed through to the gunicorn base app | 15:09 |
| cgoncalves | johnsom: ah, gunicorn consumes those options. sorry... | 15:12 |
| pratrik | Does anyone have an opinion about my patch regarding availability zones ? Does it make sense ? | 15:16 |
| johnsom | pratrik Hi, I haven't had time to look at it yet. We have some nova AZ support already, and I'm sure rm_work would be interested in the patch. I will take a look later today. | 15:17 |
| pratrik | johnsom: ok no worries, I was just curious if I was missing something. But to me it makes sense to put the amphora's in different az's. | 15:18 |
| pratrik | Not really sure how / if you could solve that with affinity, but I think its cleaner to use az's. | 15:18 |
| johnsom | pratrik Yes. It is super sad nova does not do a good job with that and the AZ support. | 15:18 |
| johnsom | pratrik Do you have the same networks available across AZs? So that Active/Standby Active/Active will work when split across AZs? | 15:19 |
| pratrik | nova does what you tell it I suppose, and we just either 1 ) not tell nova at all (which will make nova choose), or 2 ) specify one az that both amphoras will be put in. this small patch gives you the possibility to put one amphora in each defined az. | 15:20 |
| pratrik | (and yes, same network must exist in both az's for it to work) | 15:20 |
| johnsom | Ok, cool. Yeah, we have been shying away from becoming the new nova scheduler, but that may just have to happen if AZs don't improve in nova | 15:21 |
| pratrik | But maybe that should be mentioned as a comment to the parameter (that the network part needs to be configured correctly) | 15:22 |
| pratrik | I dont think you're the "new nova scheduler" just because you let the user define which az's to use for his amphoras. | 15:22 |
| johnsom | I was just commenting on our previous conversations about the state of AZs | 15:23 |
| pratrik | You need to give some input to nova on how to place the vms, otherwise it will just use the default scheduler, which is fine I guess in most cases - but here you actually want to give the user a choice.. | 15:23 |
| pratrik | At least that makes sense to me.. | 15:23 |
| pratrik | johnsom: ok ok | 15:24 |
| pratrik | Anyway the patch is simple but does the job for us :) | 15:25 |
| johnsom | pratrik Why not post a patch to gerrit? We can't really use that code due to licensing, etc. | 15:25 |
| pratrik | johnsom: hehe well because laziness, but yes, maybe I should do it the official way :) | 15:26 |
| *** tesseract has quit IRC | 15:32 | |
| openstackgerrit | Merged openstack/python-octaviaclient master: Updated from global requirements https://review.openstack.org/555591 | 15:34 |
| *** atoth has quit IRC | 15:41 | |
| johnsom | FYI: The Vancouver summit early bird pricing ends April 4 at 11:59pm Pacific Time / April 5 6:59 UTC | 15:50 |
| *** atoth has joined #openstack-lbaas | 15:56 | |
| *** samccann has joined #openstack-lbaas | 15:57 | |
| johnsom | For those of you with PTG passes for the summit, you MUST register with your unique code before May 11, 2018 | 15:57 |
| xgerman_ | same for sepaker | 15:58 |
| *** links has joined #openstack-lbaas | 16:26 | |
| *** AlexeyAbashkin has quit IRC | 16:31 | |
| *** bcafarel has quit IRC | 16:50 | |
| openstackgerrit | Merged openstack/octavia master: Add deadlock retry wrapper for inital quota create https://review.openstack.org/557094 | 16:56 |
| openstackgerrit | Merged openstack/octavia master: Update API-REF for x-forwarded-port is string https://review.openstack.org/557781 | 16:56 |
| *** links has quit IRC | 17:13 | |
| *** annp has quit IRC | 17:17 | |
| *** annp has joined #openstack-lbaas | 17:18 | |
| *** salmankhan has quit IRC | 17:20 | |
| rm_work | pratrik: yes, I already have a patch up that handles multi-AZ (and az-antiaffinity) | 17:25 |
| rm_work | but unfortunately it will never merge as-is | 17:25 |
| rm_work | I'd been meaning to look at splitting that out | 17:26 |
| rm_work | I'll take a look at what you are doing but it is probably similar | 17:26 |
| rm_work | pratrik: it's this huge patch which is also for a custom network driver (because in our case, different AZs are in different L2s so we need to do network routing at L3) | 17:27 |
| rm_work | https://review.openstack.org/#/c/435612/ | 17:27 |
| rm_work | this part looks particularly familiar :P https://review.openstack.org/#/c/435612/117/octavia/controller/worker/task_utils.py | 17:29 |
| rm_work | and this you might find useful: https://review.openstack.org/#/c/435612/117/octavia/controller/worker/tasks/amphora_driver_tasks.py | 17:30 |
| rm_work | pratrik: ^^ | 17:30 |
| rm_work | makes sure that your amps are split between zones always | 17:30 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 17:37 | |
| *** AlexeyAbashkin has quit IRC | 17:42 | |
| rm_work | pratrik: and of course: https://review.openstack.org/#/c/435612/117/octavia/controller/worker/tasks/database_tasks.py | 17:44 |
| rm_work | actually I really need to talk to other folks, maybe in the next meeting... about whether or not people think code like this can merge (because it requires a custom nova scheduler at the moment) | 17:45 |
| rm_work | and also whether i want to be juggling yet more feature patches when I'm still waiting for reviews on the ones I have :( | 17:48 |
| rm_work | xgerman_: responded to all points on https://review.openstack.org/#/c/549263/4 and IMO there is no issue here | 17:56 |
| *** sshank has joined #openstack-lbaas | 17:56 | |
| rm_work | xgerman_: also https://review.openstack.org/#/c/555454/5 | 18:01 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 18:02 | |
| *** AlexeyAbashkin has quit IRC | 18:06 | |
| rm_work | johnsom: you here today? | 18:27 |
| johnsom | Yes, polishing up some unit tests, then going to weigh in our patches | 18:27 |
| rm_work | woo | 18:27 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Create noop provider driver and data model https://review.openstack.org/558320 | 18:37 |
| *** isssp has joined #openstack-lbaas | 18:38 | |
| *** mugsie_ has joined #openstack-lbaas | 18:41 | |
| *** pcaruana has quit IRC | 18:44 | |
| *** shan5464_ has joined #openstack-lbaas | 18:45 | |
| *** numans_ has joined #openstack-lbaas | 18:46 | |
| *** numans has quit IRC | 18:47 | |
| *** mordred has quit IRC | 18:47 | |
| *** shananigans has quit IRC | 18:47 | |
| *** ispp has quit IRC | 18:47 | |
| *** mugsie has quit IRC | 18:47 | |
| *** samccann has quit IRC | 18:48 | |
| *** mordred has joined #openstack-lbaas | 18:55 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 18:55 | |
| *** dmellado has quit IRC | 18:56 | |
| *** voelzmo has joined #openstack-lbaas | 18:58 | |
| *** dmellado has joined #openstack-lbaas | 18:58 | |
| *** AlexeyAbashkin has quit IRC | 19:00 | |
| *** sshank has quit IRC | 19:00 | |
| *** dmellado has quit IRC | 19:12 | |
| *** voelzmo has quit IRC | 19:17 | |
| *** voelzmo has joined #openstack-lbaas | 19:19 | |
| *** dmellado has joined #openstack-lbaas | 19:21 | |
| *** harlowja has joined #openstack-lbaas | 19:23 | |
| *** ivve has joined #openstack-lbaas | 19:28 | |
| *** yamamoto has quit IRC | 19:30 | |
| *** dmellado has quit IRC | 19:31 | |
| *** yamamoto has joined #openstack-lbaas | 19:36 | |
| *** ivve has quit IRC | 19:37 | |
| *** ianychoi__ has joined #openstack-lbaas | 19:38 | |
| *** pcaruana has joined #openstack-lbaas | 19:38 | |
| *** ianychoi_ has quit IRC | 19:41 | |
| *** yamamoto has quit IRC | 19:42 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Insert WaitForPortDetach in delete amphora flow https://review.openstack.org/558611 | 19:47 |
| *** pcaruana has quit IRC | 19:50 | |
| *** yamamoto has joined #openstack-lbaas | 19:52 | |
| *** dmellado has joined #openstack-lbaas | 19:55 | |
| *** yamamoto has quit IRC | 19:57 | |
| *** yamamoto has joined #openstack-lbaas | 19:59 | |
| *** yamamoto has quit IRC | 19:59 | |
| *** yamamoto has joined #openstack-lbaas | 20:01 | |
| xgerman_ | ^^ rm_work - this might be related to your troubles | 20:04 |
| *** velizarx has joined #openstack-lbaas | 20:05 | |
| *** yamamoto has quit IRC | 20:06 | |
| rm_work | hmmmmmmm | 20:10 |
| *** velizarx has quit IRC | 20:10 | |
| rm_work | i thought we did a wait at a deeper level too | 20:10 |
| rm_work | but maybe this could be the issue | 20:10 |
| rm_work | though my "try harder" patch really targets a different issue | 20:10 |
| rm_work | which is stuff that has gotten into a broken state due to a failure partway through a flow | 20:11 |
| xgerman_ | yes, you would be able to delete with a vip port being gone crazy | 20:11 |
| xgerman_ | I am not opposed to the try harder but let’s let it simmer for a bit… or at least unti johnsom weighted in | 20:12 |
| *** yamamoto has joined #openstack-lbaas | 20:16 | |
| *** yamamoto has quit IRC | 20:21 | |
| *** yamamoto has joined #openstack-lbaas | 20:23 | |
| *** yamamoto has quit IRC | 20:23 | |
| rm_work | xgerman_: yes i think this patch is correct, the wait should be there IMO | 20:30 |
| xgerman_ | +1 | 20:30 |
| *** sapd has quit IRC | 20:30 | |
| rm_work | we probably want to backport this also | 20:30 |
| *** sapd has joined #openstack-lbaas | 20:30 | |
| *** sapd has quit IRC | 20:30 | |
| *** sapd has joined #openstack-lbaas | 20:31 | |
| xgerman_ | +1000 | 20:34 |
| johnsom | xgerman_ But you know that your issue is not this, it's that nova never actually does the delete. This will just add more delay.... Plus, there is nothing later in that flow that cares if the port is there or not, so why wait there? | 20:42 |
| xgerman_ | because we do the same in the failover flow | 20:43 |
| johnsom | But failover flow cares, this delete flow doesn't. Failover needs to move that port it is waiting on, this delete doesn't | 20:43 |
| xgerman_ | It make sense to me to wait for all of them being detached cleanly | 20:43 |
| xgerman_ | but we need to have the ports deleted so we can delete the sec grp | 20:44 |
| johnsom | The last two steps in that flow are just database "mark things deleted". There is no security group delete after that.... | 20:45 |
| *** velizarx has joined #openstack-lbaas | 20:45 | |
| xgerman_ | our flows are a mess — why would we not use tge delete amphora flow in some unordered flow and instead loop over amps in DeleteAmphoraeOnLoadBalancer | 20:50 |
| xgerman_ | let me fix that right away | 20:51 |
| johnsom | I'm just saying, that is way too late in the flow. I think that wait needs to go after a detach | 20:52 |
| johnsom | Not that we have really seen this issue anyway | 20:52 |
| *** velizarx has quit IRC | 20:53 | |
| *** velizarx has joined #openstack-lbaas | 20:55 | |
| xgerman_ | well, then what ports is Adam trying to delete? | 20:55 |
| johnsom | Be super careful there too. The flows are not a mess and there is probably a good reason for this ordering. The flow you selected is just for admin/spares cleanup | 20:55 |
| xgerman_ | sure, but we should reuse flows where we can | 20:55 |
| xgerman_ | ok, will run an errand and then rework that | 20:57 |
| johnsom | xgerman_ Right, and we do | 20:57 |
| *** sshank has joined #openstack-lbaas | 21:00 | |
| *** sshank has quit IRC | 21:00 | |
| johnsom | I don't see any coorilation between Adam | 21:00 |
| johnsom | 's patch and needing to wait for a port to detach from an amp frankly. Ugh, coorilation -> correlation | 21:01 |
| *** velizarx has quit IRC | 21:07 | |
| *** yamamoto has joined #openstack-lbaas | 21:24 | |
| *** yamamoto has quit IRC | 21:29 | |
| *** rcernin has joined #openstack-lbaas | 21:53 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 22:00 | |
| *** AlexeyAbashkin has quit IRC | 22:05 | |
| xgerman_ | johnsom: well, he si deleting ports which are left over the only ports we shun into that sec grp are the vip and frontend ports | 22:16 |
| rm_work | hmm yeah i found the bit i was thinking of | 22:16 |
| rm_work | and it actually doesn't require the delay | 22:16 |
| johnsom | Yeah, but no detach | 22:16 |
| rm_work | it's unrelated | 22:17 |
| rm_work | because on a LB delete, the amp deletion actually happens well after the ports are cleaned up | 22:17 |
| rm_work | the ports detaching is unrelated to the compute action | 22:17 |
| johnsom | ^^^ right | 22:17 |
| rm_work | had to dig through but that explains the nagging feeling i had that we DID handle this somehow | 22:17 |
| xgerman_ | so what ports are you then deleting? | 22:18 |
| rm_work | when we do a failover and it breaks | 22:18 |
| rm_work | it leaves stuff around | 22:18 |
| rm_work | because it has started doing the amp delete but it doesn't finish completely, and rolls back | 22:19 |
| rm_work | but then the amp is gone and we've got a port sitting around | 22:19 |
| rm_work | and then the amp gets cleaned up (usually due to the OTHER bug I have a patch for, around deleted amp expiry) | 22:19 |
| xgerman_ | ok, so this is a side effect of the fail over? In my case I often see the vip port still being in the sec grp | 22:19 |
| rm_work | yeah that would be a totally different fail i think | 22:20 |
| rm_work | ah i think also it happens during a LB delete when the LB never went ACTIVE | 22:20 |
| rm_work | so like it got partway through provisioning, usually due to the compute node never coming up right | 22:20 |
| xgerman_ | yeah, the LB delete looks very different from the amphora delete OHMO | 22:20 |
| rm_work | and so it doesn't have all the entries hooked up correctly | 22:20 |
| rm_work | yeah, well, it can be | 22:21 |
| rm_work | since in an amp delete we are maintaining network state | 22:21 |
| rm_work | and just removing a compute | 22:21 |
| rm_work | and in LB delete we are cleaning up network, then compute | 22:21 |
| xgerman_ | yes and no -we delete the vm and then we move to delete the vrrp ports | 22:23 |
| xgerman_ | so IF nova is slow we would hit some issue | 22:23 |
| xgerman_ | ? | 22:23 |
| johnsom | Let's just agree, no more upstream patches based of that bad lab where nova is broken and lies.... | 22:24 |
| xgerman_ | I have trouble finding amps in pending delete but that’s me | 22:24 |
| xgerman_ | also nova being slow is a real possibility we should account for | 22:25 |
| xgerman_ | and we should delete the amps in an unordered flow instead with all those loops | 22:25 |
| johnsom | Right, because nova lies. Do a nova show on one of those marked "DELETED" in our DB. At the top, the "special" X attributes. They are all in "deleting" even though status shows otherwise | 22:26 |
| *** yamamoto has joined #openstack-lbaas | 22:26 | |
| johnsom | I'm not so sure myself. | 22:26 |
| johnsom | I think the flows look pretty good. | 22:26 |
| xgerman_ | yeah, if we do one or two amps — if we do 50 like with Active-Active then this would greatly benefit from being parallel | 22:28 |
| johnsom | I think we do need to move some serial actions up into the flow. The failover with two down is my highest interest. We just need to be super careful here. | 22:28 |
| xgerman_ | +1 | 22:29 |
| *** yamamoto has quit IRC | 22:30 | |
| *** voelzmo has quit IRC | 22:31 | |
| *** voelzmo has joined #openstack-lbaas | 22:41 | |
| *** voelzmo has quit IRC | 22:46 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Allow members to be set as "backup" https://review.openstack.org/552632 | 22:48 |
| *** voelzmo has joined #openstack-lbaas | 22:48 | |
| rm_work | I found something related yesterday actually | 22:51 |
| rm_work | i totally forgot i even added this: | 22:51 |
| rm_work | https://review.openstack.org/#/c/435612/117/octavia/controller/worker/tasks/compute_tasks.py@157 | 22:52 |
| rm_work | https://www.youtube.com/watch?v=Eyp4Sdq6i0s | 22:53 |
| johnsom | Couldin't you do an "in"? | 22:53 |
| *** voelzmo has quit IRC | 22:53 | |
| rm_work | i just copy-pasta'd from below | 22:53 |
| rm_work | we could rewrite that | 22:53 |
| rm_work | ah but no | 22:54 |
| rm_work | because i changed the logic | 22:54 |
| rm_work | on the re-delete, i change the log slightly, and don't raise on a failure | 22:54 |
| rm_work | but, that was my attempt at zombie-hunting | 22:55 |
| rm_work | (one of them) | 22:55 |
| rm_work | because... ffff nova | 22:55 |
| johnsom | Yeah, the nasty in that lab, you can keep deleting it, but it won't go away. Thus how the hm was at 100% cpu and eating a ton of memory, it only had 10 threads configured and a bunch of zombies... | 22:55 |
| johnsom | I guess that is adding additional round trips to nova even when things are fine.... | 22:56 |
| rm_work | yeah but on an async delete | 22:59 |
| rm_work | so meh | 22:59 |
| rm_work | and it's only once | 22:59 |
| rm_work | (on an LB delete) | 22:59 |
| *** fnaval has quit IRC | 23:03 | |
| *** fnaval has joined #openstack-lbaas | 23:10 | |
| *** yamamoto has joined #openstack-lbaas | 23:26 | |
| xgerman_ | that LB delete flow makes less and less sense. We delete the vrrp port twice? | 23:28 |
| *** fnaval has quit IRC | 23:29 | |
| xgerman_ | also in the amp case we just delete the amp without detaching anyhting | 23:29 |
| xgerman_ | also we skip the pending delete on the ap in the LB delete flow | 23:32 |
| xgerman_ | ap=amp | 23:32 |
| *** yamamoto has quit IRC | 23:32 | |
| xgerman_ | and the server show doesn’t work on DELETED servers — though they still show up on list with —all | 23:34 |
| *** threestrands has joined #openstack-lbaas | 23:36 | |
| *** threestrands has quit IRC | 23:37 | |
| *** threestrands has joined #openstack-lbaas | 23:38 | |
| *** threestrands has quit IRC | 23:38 | |
| *** threestrands has joined #openstack-lbaas | 23:38 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Correctly validate member subnet_id in batches https://review.openstack.org/558653 | 23:39 |
| rm_work | johnsom: ^^ fixed | 23:39 |
| *** threestrands has quit IRC | 23:39 | |
| *** threestrands has joined #openstack-lbaas | 23:39 | |
| *** threestrands has quit IRC | 23:39 | |
| *** threestrands has joined #openstack-lbaas | 23:39 | |
| rm_work | oops | 23:40 |
| rm_work | pep8 >_< | 23:40 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Correctly validate member subnet_id in batches https://review.openstack.org/558653 | 23:40 |
| *** threestrands has quit IRC | 23:40 | |
| *** threestrands has joined #openstack-lbaas | 23:41 | |
| *** threestrands has quit IRC | 23:41 | |
| *** threestrands has joined #openstack-lbaas | 23:41 | |
| johnsom | rm_work I think the status code is wrong | 23:42 |
| rm_work | ? | 23:42 |
| rm_work | 400? | 23:42 |
| xgerman_ | yep | 23:42 |
| rm_work | that's what the other ones do | 23:42 |
| rm_work | <_< | 23:42 |
| xgerman_ | 400 wrong parameter sounds right for me, too | 23:42 |
| rm_work | yes | 23:43 |
| johnsom | Hmm, do we translate that somewhere? Because I would expect NotFound exception to be 404 | 23:43 |
| xgerman_ | yes, that needs to be fixed | 23:43 |
| rm_work | errr | 23:43 |
| rm_work | the subnet not being found? | 23:43 |
| rm_work | the subnet not existing is an invalid param | 23:43 |
| rm_work | which is a 400 | 23:43 |
| xgerman_ | making sure it returns a 400 | 23:43 |
| rm_work | it's not a 404 on "can't find a member create endpoint" lol | 23:44 |
| rm_work | i stand by 400 | 23:44 |
| johnsom | I mean, 400 seems like what the user should get, but the exception raised is NotFound which I would expect to be 404 | 23:44 |
| rm_work | errr | 23:44 |
| rm_work | but it's only used in THIS context | 23:44 |
| rm_work | so it needs to be a 400 | 23:44 |
| rm_work | or by definition we'd have to pick a different one | 23:44 |
| rm_work | that is a 400 <_< | 23:44 |
| johnsom | Yeah, ok, I think I get it. We put in 400 because 404 is endpoint missing. Got it | 23:44 |
| johnsom | Just odd | 23:44 |
| rm_work | yes | 23:45 |
| xgerman_ | ok | 23:45 |
| johnsom | right, ok we translate it | 23:45 |
| rm_work | johnsom: should i include a fix for that lifecycle revert task too? | 23:48 |
| johnsom | rm_work That would be great! | 23:49 |
| rm_work | i mean in that CR or a different one | 23:49 |
| johnsom | Oh, maybe another one since I already +2 this one.... depends on if others have issues with that first one. | 23:50 |
| johnsom | They are two different issues, but ok if they are called out in the commit message. I think their backport situation would be the same.... | 23:51 |
| rm_work | kk. hmmm i'm not sure how this got a tuple... | 23:53 |
| rm_work | ooooohhh k | 23:54 |
| rm_work | not sure on the test for this one :/ | 23:56 |
| rm_work | i'll post the fix so you can see it | 23:56 |
| *** voelzmo has joined #openstack-lbaas | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!