| *** catintheroof has joined #openstack-lbaas | 00:23 | |
| *** aojea has quit IRC | 00:49 | |
| *** ssmith has quit IRC | 01:09 | |
| openstackgerrit | huangshan proposed openstack/octavia master: Fix DB update reverts for listener provisioning_status https://review.openstack.org/493401 | 01:42 |
|---|---|---|
| *** catintheroof has quit IRC | 01:47 | |
| *** gongysh has joined #openstack-lbaas | 02:07 | |
| *** gongysh has quit IRC | 02:47 | |
| *** catintheroof has joined #openstack-lbaas | 02:50 | |
| *** gongysh has joined #openstack-lbaas | 02:53 | |
| *** catintheroof has quit IRC | 03:03 | |
| *** links has joined #openstack-lbaas | 03:47 | |
| *** gcheresh has joined #openstack-lbaas | 04:40 | |
| *** gcheresh has quit IRC | 05:02 | |
| *** reedip has quit IRC | 05:23 | |
| *** reedip has joined #openstack-lbaas | 05:37 | |
| *** pcaruana has joined #openstack-lbaas | 05:45 | |
| *** gcheresh has joined #openstack-lbaas | 06:36 | |
| *** slaweq has joined #openstack-lbaas | 06:40 | |
| *** rcernin has joined #openstack-lbaas | 06:57 | |
| isantosp | Hi, how could I integrate in my puppet environment octavia-pike? | 07:07 |
| *** rcernin has quit IRC | 07:16 | |
| *** gcheresh has quit IRC | 07:17 | |
| *** rcernin has joined #openstack-lbaas | 07:19 | |
| *** gcheresh has joined #openstack-lbaas | 07:22 | |
| *** Alex_Staf has joined #openstack-lbaas | 07:37 | |
| isantosp | I mean, the delorean repo | 08:04 |
| *** yamamoto has joined #openstack-lbaas | 08:33 | |
| *** slaweq has quit IRC | 08:33 | |
| *** aojea has joined #openstack-lbaas | 08:37 | |
| *** gcheresh has quit IRC | 08:37 | |
| *** aojea has quit IRC | 08:42 | |
| openstackgerrit | Nir Magnezi proposed openstack/neutron-lbaas master: Add status in VMware driver https://review.openstack.org/323645 | 08:44 |
| *** yamamoto has quit IRC | 08:49 | |
| *** gcheresh has joined #openstack-lbaas | 08:52 | |
| *** gcheresh has quit IRC | 08:53 | |
| *** aojea has joined #openstack-lbaas | 08:57 | |
| *** aojea has quit IRC | 09:01 | |
| *** belharar has joined #openstack-lbaas | 09:20 | |
| *** slaweq has joined #openstack-lbaas | 10:19 | |
| *** atoth has quit IRC | 10:33 | |
| *** slaweq has quit IRC | 10:34 | |
| *** slaweq has joined #openstack-lbaas | 10:36 | |
| *** belharar has quit IRC | 11:18 | |
| *** belharar has joined #openstack-lbaas | 11:18 | |
| *** rm_mobile has joined #openstack-lbaas | 11:29 | |
| *** atoth has joined #openstack-lbaas | 11:29 | |
| rm_mobile | I should be "physically present" at the meeting today, but whether I'll be fully conscious is doubtful | 11:30 |
| *** slaweq has quit IRC | 11:43 | |
| *** slaweq has joined #openstack-lbaas | 11:43 | |
| *** slaweq has quit IRC | 11:44 | |
| *** slaweq has joined #openstack-lbaas | 11:45 | |
| *** slaweq has quit IRC | 12:05 | |
| *** slaweq has joined #openstack-lbaas | 12:05 | |
| *** slaweq has quit IRC | 12:10 | |
| *** slaweq has joined #openstack-lbaas | 12:18 | |
| *** catintheroof has joined #openstack-lbaas | 12:49 | |
| *** atoth has quit IRC | 12:50 | |
| *** gongysh has quit IRC | 13:00 | |
| *** gongysh has joined #openstack-lbaas | 13:00 | |
| *** gongysh has quit IRC | 13:00 | |
| *** belharar_ has joined #openstack-lbaas | 13:02 | |
| *** belharar has quit IRC | 13:02 | |
| *** atoth has joined #openstack-lbaas | 13:07 | |
| *** belharar_ has quit IRC | 13:19 | |
| *** belharar_ has joined #openstack-lbaas | 13:19 | |
| *** chlong_ has joined #openstack-lbaas | 13:31 | |
| *** aojea has joined #openstack-lbaas | 13:32 | |
| *** ssmith has joined #openstack-lbaas | 13:41 | |
| *** cpusmith has joined #openstack-lbaas | 13:42 | |
| *** ssmith has quit IRC | 13:46 | |
| *** cpusmith_ has joined #openstack-lbaas | 13:59 | |
| *** chlong_ has quit IRC | 13:59 | |
| *** cpusmith has quit IRC | 14:03 | |
| *** atoth has quit IRC | 14:12 | |
| *** cpusmith_ has quit IRC | 14:14 | |
| *** cpusmith_ has joined #openstack-lbaas | 14:19 | |
| *** cpusmith has joined #openstack-lbaas | 14:20 | |
| *** cpusmith_ has quit IRC | 14:23 | |
| *** atoth has joined #openstack-lbaas | 14:25 | |
| *** slaweq has quit IRC | 14:30 | |
| *** links has quit IRC | 14:32 | |
| *** fnaval has joined #openstack-lbaas | 14:32 | |
| *** belharar_ has quit IRC | 14:39 | |
| *** Alex_Staf has quit IRC | 14:45 | |
| *** belharar has joined #openstack-lbaas | 15:05 | |
| *** rcernin has quit IRC | 15:07 | |
| isantosp | Is there any keystone configuration to be added in neutron.conf to work with octavia? (neutron-newton) (octavia-pike) | 15:22 |
| johnsom | isantosp No, I don't think so. The only thing I can think of is RBAC policy work if you want to run octavia under a unique user as opposed to an admin user. | 15:24 |
| isantosp | @johnsom Im seeing this error: ERROR: neutronclient.shell Driver error: Unable to establish connection to http://127.0.0.1:5000/v2.0/tokens: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x8bb8750>: Failed to establish a new connection: [Errno 111] | 15:27 |
| isantosp | ECONNREFUSED',)) | 15:27 |
| isantosp | any idea about what can it be? | 15:28 |
| johnsom | Yeah, I know what that is. It's not octavia specific, but a problem. Just a second let me did up some pointers for your | 15:28 |
| mnaser | isantosp are you running aio? | 15:29 |
| mnaser | looks like neutron cannot authenticate to keystone | 15:29 |
| johnsom | isantosp First thing, do an "openstack endpoint list" and find the keystone entry and it's URL. | 15:30 |
| johnsom | My devstack is http://172.20.55.13/identity as an example. Note, the format no longer has a port number as keystone has changed the way it advertises it's endpoint for the uwsgi community goal. | 15:31 |
| johnsom | Then, in neutron.conf, find the [keystone_authtoken] and the "auth_url" setting and make it point to the endpoint you found above (or at least look like it). | 15:32 |
| johnsom | You should do the same for all of the "auth_url" settings in all of the neutron configuration files. | 15:32 |
| johnsom | neutron-lbaas.conf if you have it as well | 15:33 |
| *** links has joined #openstack-lbaas | 15:33 | |
| isantosp | @johnsom aham, ok, I'll try, thank you :) | 15:39 |
| isantosp | mnaser no, we are not running aio | 15:39 |
| mnaser | isantosp ^ what johnsom said but also a little hack | 15:43 |
| mnaser | restart neutron with debug=True | 15:43 |
| mnaser | grep logs for 127.0.0.1 | 15:43 |
| mnaser | you'll find some keystone setting pointing to the wrong place | 15:43 |
| isantosp | aham nice, I was debuging it only in a loadbalancer creation :) thanks | 15:48 |
| *** links has quit IRC | 15:49 | |
| *** pcaruana has quit IRC | 15:57 | |
| *** aojea has quit IRC | 16:13 | |
| *** aojea has joined #openstack-lbaas | 16:25 | |
| *** rcernin has joined #openstack-lbaas | 16:49 | |
| *** JudeC has joined #openstack-lbaas | 16:59 | |
| johnsom | Octavia meeting starting soon on #openstack-meeting | 16:59 |
| *** leyal has quit IRC | 17:00 | |
| *** strigazi_OFF is now known as strigazi | 17:00 | |
| *** leyal has joined #openstack-lbaas | 17:00 | |
| *** sshank has joined #openstack-lbaas | 17:06 | |
| *** aojea has quit IRC | 17:19 | |
| nmagnezi | johnsom, that was a short one | 17:22 |
| nmagnezi | (sorry to be late) | 17:22 |
| johnsom | Yeah, I kind of expected it to be. RC weeks are kind of a break for most folks | 17:23 |
| johnsom | The logs are here if you would like: https://wiki.openstack.org/wiki/Octavia/Meeting_Minutes | 17:23 |
| nmagnezi | johnsom, not for me.. just a bad hour for me to make it to the meeting.. but I'm trying :) | 17:24 |
| johnsom | Yeah, maybe we can re-evaluate in Queens | 17:24 |
| nmagnezi | +1 | 17:25 |
| johnsom | There is talk of letting the teams use the project IRC channels for the meetings, so that may give us more flexibility | 17:25 |
| nmagnezi | that could actually be a good idea. will also help to everyone here to follow up on topic in case they missed the actual meeting (assuming not everyone open the meeting minutes) | 17:28 |
| *** aojea has joined #openstack-lbaas | 17:29 | |
| *** strigazi is now known as strigazi_OFF | 17:55 | |
| *** tomtomtom has joined #openstack-lbaas | 17:59 | |
| tomtomtom | rm_work, johnsom, the ip addresses are acting as expected now thank you. I am now running into this issue (http://imgur.com/a/xcPyT) do you know if this is coming from the web server or the load balancer when I hit it in the web browser? | 18:00 |
| johnsom | Hmm, 504 | 18:01 |
| johnsom | Is it a large transfer or just a normal size GET? | 18:02 |
| tomtomtom | large transfer | 18:03 |
| rm_work | tomtomtom: can you SSH into the amphora, activate the netns, and make sure you can curl the members from there? | 18:03 |
| rm_work | ah | 18:03 |
| rm_work | you might need to set timeouts | 18:03 |
| tomtomtom | yeah, I set the health monitor timeout to 300 | 18:03 |
| johnsom | Yeah, ok. It's likely you are hitting one of the default timeouts | 18:03 |
| tomtomtom | johnsom I can curl the page from the namespace | 18:03 |
| rm_work | err | 18:04 |
| rm_work | not the healthmonitor timeout | 18:04 |
| rm_work | the listener timeout | 18:04 |
| johnsom | Yeah, that doesn't matter here | 18:04 |
| johnsom | If you want to double check, you can look at the haproxy log in the amphora, but I'm pretty sure it's the timeouts. | 18:04 |
| rm_work | wait yeah where is that... | 18:05 |
| rm_work | i thought somewhere we specified the max connection time | 18:05 |
| rm_work | but i can't find it | 18:06 |
| johnsom | So, we have not exposed this via the API yet. It's a wish list item. But you can adjust them in your jinja template. | 18:06 |
| tomtomtom | is there a reference for the jinja template? | 18:07 |
| tomtomtom | like where it is? | 18:07 |
| johnsom | These are the timeout options: | 18:07 |
| johnsom | https://www.irccloud.com/pastebin/ojW3SE7z/ | 18:07 |
| johnsom | Docs for those are here: http://cbonte.github.io/haproxy-dconv/1.6/configuration.html | 18:08 |
| tomtomtom | ok thanks | 18:08 |
| johnsom | jinja template is here: https://github.com/openstack/octavia/blob/master/octavia/common/jinja/haproxy/templates/base.j2#L32 | 18:08 |
| *** pcaruana has joined #openstack-lbaas | 18:09 | |
| *** JudeC has quit IRC | 18:15 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add flag to disable SSHD on the amphora image https://review.openstack.org/492332 | 18:18 |
| rm_work | johnsom: could https://review.openstack.org/#/c/493252/9/octavia/controller/healthmanager/update_db.py use a context-manager? | 18:28 |
| *** belharar has quit IRC | 18:30 | |
| johnsom | For the db session? maybe? I'm always squeamish with context managers and exception handling cleanup, etc. | 18:30 |
| rm_work | hmm | 18:30 |
| rm_work | johnsom: can I merge https://review.openstack.org/#/c/493328/1 safely? | 18:32 |
| rm_work | or no | 18:32 |
| rm_work | like, what am I allowed to merge | 18:32 |
| johnsom | Technically just the two patches we are considering for RC2 | 18:33 |
| rm_work | k | 18:33 |
| johnsom | That one is really more of a feature with the version string at startup, etc. | 18:33 |
| rm_work | 252 and 646 | 18:33 |
| johnsom | Yes | 18:33 |
| johnsom | Then, once we are good with RC2, we can decide when to cut Pike and open things up for Queens. I think we can do it pretty soon actually as I don't think we are going to get translations for octaiva, just the dashboard | 18:34 |
| rm_work | k there's 646 | 18:34 |
| *** sshank has quit IRC | 18:37 | |
| rm_work | ok and 252 | 18:39 |
| rm_work | verified it all looked like what i have | 18:39 |
| johnsom | Groovy, when those merge I will put in the backport for stable/pike, we can merge those and I will put up the RC2 request | 18:40 |
| rm_work | k | 18:41 |
| johnsom | Speaking of translations, an update came in last night: https://review.openstack.org/#/c/494147 | 18:50 |
| rm_work | woo, korean | 18:51 |
| johnsom | Makes me miss the really good Korean restaurant we had in town... | 18:52 |
| johnsom | Now it's a dumb chain sandwich shop. | 18:52 |
| rm_work | i always wonder about translations now, since one of my old projects once merged translations in Russian that turned out to be advertisements for some spam site | 18:52 |
| rm_work | but no one realized because we didn't speak russian <_< | 18:52 |
| rm_work | until some russian speaking users came to our IRC and were like "... wut" | 18:52 |
| johnsom | Hahaha, I think the i18n folks have a pretty good system | 18:53 |
| rm_work | there's some review I assume? that actually relies on another user being able to read it? | 18:53 |
| rm_work | because the person who did that to us actually took the time to really properly format everything | 18:54 |
| rm_work | it all fit like it was supposed to <_< | 18:54 |
| johnsom | They use zanata: https://translate.openstack.org/?dswid=3085 | 18:54 |
| johnsom | I mean, it is still on us as cores to validate, but yeah, google translate only goes so far... | 18:54 |
| rm_work | i guess at least now with google translate we can see if it's OBVIOUSLY spam | 18:55 |
| *** rm_mobile has quit IRC | 18:59 | |
| *** aojea has quit IRC | 19:15 | |
| openstackgerrit | Merged openstack/octavia master: Fix a bad revert method and add hacking check https://review.openstack.org/493646 | 19:40 |
| openstackgerrit | Merged openstack/octavia master: Fix health monitor DB locking. https://review.openstack.org/493252 | 19:51 |
| *** rcernin has quit IRC | 19:55 | |
| *** sshank has joined #openstack-lbaas | 20:00 | |
| *** aojea has joined #openstack-lbaas | 20:05 | |
| *** aojea has quit IRC | 20:10 | |
| *** sshank has quit IRC | 20:11 | |
| *** sshank has joined #openstack-lbaas | 20:14 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 20:17 |
| *** sshank has quit IRC | 20:18 | |
| *** sshank has joined #openstack-lbaas | 20:18 | |
| rm_work | johnsom: we don't actually... check anything about the health messages, other than verify the HMAC, do we? we just take whatever comes in and pass it to both the health_update and stats_update functions | 20:28 |
| rm_work | it seems | 20:28 |
| rm_work | there's no intelligent routing of health messages | 20:28 |
| johnsom | Umm, not sure. I haven't dug into much of that | 20:28 |
| johnsom | intelligent routing, no | 20:29 |
| johnsom | It is intended to be dumb | 20:29 |
| rm_work | johnsom: for reference, i'm adding another type of message, essentially an "emergency ping" to trigger the active-standby VIP migration, since we don't have BGP | 20:29 |
| johnsom | health messages should randomly pick a HM to send to from the list | 20:29 |
| rm_work | so the master keepalived can run an alert script and it'll broadcast an emergency message and the HMs can pick it up and trigger a flow for the VIP move | 20:30 |
| rm_work | i mean on the Healthmanager side | 20:30 |
| rm_work | when it receives a UDP message | 20:30 |
| rm_work | there's no intelligent handling | 20:30 |
| rm_work | it's just like "ok this message has health and stats, run both types of update functions on it" | 20:30 |
| johnsom | Ah, ok, so creating a new message type or such? | 20:30 |
| rm_work | yes | 20:30 |
| johnsom | Yeah, that is cool. We, wait for it, might want to have loadable drivers for the message handling.... | 20:31 |
| rm_work | lololol | 20:31 |
| johnsom | We have talked about using a new message type for the "I need my config back" too | 20:31 |
| rm_work | so yeah it looks like this system was set up to handle drivers | 20:31 |
| rm_work | by someone who didn't know how it works (carlos) | 20:31 |
| rm_work | and so it's like | 20:31 |
| johnsom | Like just replace the whole darn thing or actually load more than one? | 20:31 |
| rm_work | kinda architected in the right way, but everything is hardcoded anyway | 20:32 |
| rm_work | it's more like | 20:32 |
| rm_work | instead of stevedore loading anything | 20:32 |
| rm_work | it's just | 20:32 |
| *** aojea has joined #openstack-lbaas | 20:32 | |
| rm_work | the main "driver" is included and loaded into the class property | 20:32 |
| rm_work | so I could probably disentangle it and get it working as a driver system | 20:33 |
| rm_work | umm, one sec i'll link | 20:33 |
| johnsom | Cool, like a named dispatch or something? | 20:33 |
| johnsom | https://docs.openstack.org/stevedore/latest/reference/index.html#namedispatchextensionmanager | 20:33 |
| rm_work | i mean | 20:34 |
| rm_work | the comment... lol | 20:34 |
| rm_work | https://github.com/openstack/octavia/blob/master/octavia/cmd/health_manager.py#L41 | 20:34 |
| rm_work | (that was put in by xgerman_ later I think, not at the original writing time) | 20:34 |
| xgerman_ | me? | 20:35 |
| rm_work | hmm though not sure | 20:35 |
| rm_work | xgerman_: well, maybe it was generically some German guy | 20:35 |
| rm_work | who wanted to be anonymous | 20:35 |
| johnsom | Yeah, I think that came in with the code... | 20:35 |
| xgerman_ | nah, that was me | 20:35 |
| xgerman_ | the idea was to have multiple drivers handling stuff | 20:36 |
| rm_work | well | 20:36 |
| rm_work | "came in with the code" | 20:36 |
| rm_work | that patch took almost a year to merge | 20:36 |
| rm_work | and had like 6 authors | 20:36 |
| rm_work | when I talk about "original code" i mean draft1 by carlos | 20:36 |
| rm_work | a ton of the code that merged in that patch was "tacked on" | 20:36 |
| johnsom | Ah, gotcha | 20:36 |
| rm_work | before it even made it to the repo | 20:36 |
| johnsom | Well, doesn't matter, let's knock out a wall and add a drive through | 20:36 |
| xgerman_ | ? | 20:37 |
| rm_work | lolol | 20:37 |
| johnsom | xgerman_ https://github.com/openstack/octavia/blob/master/octavia/cmd/health_manager.py#L41 | 20:37 |
| rm_work | i don't fully understand that metaphor but what i am doing now is more like | 20:37 |
| rm_work | construct a covered patio outside the house with PVC and plastic sheeting, and call it an extra room | 20:38 |
| xgerman_ | I think I caught up — we want to have a new message type to force failover/etc | 20:39 |
| rm_work | <N> new message types, I think | 20:39 |
| xgerman_ | instead of just disabling health messages we want to send said message | 20:39 |
| xgerman_ | to get it done faster? | 20:39 |
| rm_work | right now I am just trying to make sure I don't BREAK the existing messages | 20:40 |
| rm_work | by sending one that doesn't include like... [listeners] | 20:40 |
| rm_work | xgerman_: in my case it's because keepalived needs to failover the vip from one amp to the other in the pair | 20:40 |
| johnsom | I was just thinking we want to hand out tasty health messages to which ever driver wants them.... | 20:40 |
| xgerman_ | yes, that was my plan - have a list of drivers and they can do whatever they want | 20:41 |
| *** pcaruana has quit IRC | 20:41 | |
| rm_work | xgerman_: and it can't use BGP, it needs to call into neutron | 20:41 |
| rm_work | anyway yeah, I am about to ... | 20:41 |
| rm_work | umm | 20:41 |
| rm_work | make a patch | 20:41 |
| rm_work | one sec | 20:41 |
| xgerman_ | yes, BUT we already use the absence of a message to trigger failover | 20:41 |
| xgerman_ | just throwing that out there | 20:41 |
| johnsom | He is talking about act/stdby failover which doesn't involve the controller today | 20:42 |
| johnsom | He needs to tell something outside (neutron?) that hey I just made this amp the master. | 20:42 |
| johnsom | Which, personally, seems too slow to me, but, hey, it's a driver.... | 20:42 |
| xgerman_ | yes, understood, but it should also happen if the amphora goes dead | 20:43 |
| xgerman_ | I am just saying he might not need a new message but new behavior | 20:43 |
| johnsom | In theory, it would happen before the controller knows the amp went dead. | 20:43 |
| xgerman_ | yes, but in his case | 20:43 |
| xgerman_ | … | 20:43 |
| xgerman_ | my thing is the current messaging format is rich enough to convey that haproxy is down so all he needs is new behavior on the controller | 20:45 |
| xgerman_ | e.g. you cna set LB to unhealthy | 20:45 |
| rm_work | xgerman_: if we wait until the HM knows the amp is dead | 20:46 |
| rm_work | there's no difference between ACTIVE_STANDBY and SINGLE + spares | 20:46 |
| rm_work | the benefit of the pair is that keepalived is monitoring things FAST | 20:46 |
| xgerman_ | no, the current format let;s you say your LB is dead | 20:46 |
| rm_work | how | 20:46 |
| rm_work | absence of messages -> missed_messages * message_time == long | 20:47 |
| johnsom | Well, you could push an artificial heartbeat with the "down" concept. It's just bit odd. Really it should be the new master that sends rm_work's new message to say, "hey, I'm master now" because the dead amp isn't going to send another heartbeat | 20:47 |
| rm_work | it's about 30s until the HMs detect an amp is down, from when it goes down | 20:47 |
| rm_work | yes, what johnsom said | 20:48 |
| xgerman_ | oh, ok, but then we should put in each message master=true in case his “special” message gets lost - after all it’s UDP | 20:49 |
| *** vegarl has joined #openstack-lbaas | 20:49 | |
| johnsom | That is a good idea actually | 20:49 |
| rm_work | yeah | 20:49 |
| rm_work | that's not bad | 20:49 |
| rm_work | i am adding a "broadcast" method for this though, in addition to the normal round-robin that health daemon does | 20:50 |
| rm_work | so the idea is this "emergency" message is sent everywhere | 20:50 |
| rm_work | and they immediately all race to lock, which is fine | 20:50 |
| rm_work | but better that than send it once to one place and it is missed] | 20:50 |
| johnsom | The only downside I see if you have to check the old state from the DB. | 20:50 |
| rm_work | hmmm | 20:50 |
| xgerman_ | yep, that’s true | 20:50 |
| rm_work | my plan was, try to do a health busy-lock | 20:51 |
| rm_work | and then do the FLIP failover, and then unlock | 20:51 |
| johnsom | Yeah, the blast-o-gram option would work, just tie up some threads trying to get a lock. | 20:51 |
| rm_work | I think that's fine | 20:51 |
| rm_work | but | 20:52 |
| rm_work | feel free to tell me it's not and we can brainstorm or something | 20:52 |
| johnsom | As long as whatever you are calling handles extra set calls | 20:52 |
| rm_work | in my case it's neutron's flip associate | 20:52 |
| rm_work | which ... | 20:52 |
| rm_work | you can call with whatever destination | 20:52 |
| rm_work | even if it's the same one | 20:52 |
| johnsom | In case hm1 gets the lock, makes the change, releases lock, hm2 was still waiting for lock | 20:52 |
| rm_work | if it's the same it should noop | 20:52 |
| rm_work | but that is a good point | 20:52 |
| xgerman_ | yeah, not really sure if broadcast buys us that much but… | 20:53 |
| *** atoth has quit IRC | 20:54 | |
| xgerman_ | since we sent a message pretty often… | 20:54 |
| rm_work | well, keepalived will only run this script once | 20:54 |
| johnsom | Yeah, don't use an actual network broadcast... You would be cycling unicast | 20:54 |
| rm_work | to do an "emergency" message | 20:54 |
| rm_work | after that, I think just setting master=true in the normal message is fine | 20:54 |
| rm_work | rofl no | 20:54 |
| rm_work | one sec | 20:54 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: amphora emergency udp message extension https://review.openstack.org/494323 | 20:57 |
| rm_work | super crazy early draft ^^ | 20:57 |
| rm_work | just showing the framework i'm setting up now | 20:57 |
| rm_work | haven't gotten very far | 20:58 |
| xgerman_ | it irks me philosophically that we now send commands from amphora to control plane - I would us rather report status (e.g. “I am master”) to decouple the reaction from the message | 20:59 |
| johnsom | rm_work FYI, I would heavily test this. I have experienced some "interesting" behaviors with keepalived, so I would make sure it's triggering this when/how you expect. | 20:59 |
| johnsom | Yeah, we need to make sure other amps can't failover others | 21:00 |
| rm_work | right | 21:00 |
| rm_work | hmm | 21:00 |
| rm_work | yeah somehow i was thinking these messages were verifiable | 21:01 |
| rm_work | but i guess they aren't? | 21:01 |
| rm_work | oh | 21:01 |
| rm_work | i can verify source-ip? | 21:01 |
| rm_work | maybe | 21:01 |
| rm_work | we get it back, just throw it away | 21:01 |
| johnsom | I think we salt the hmac with the amp ID which is "private" so, that should be good, but we should double check they actually did that..... | 21:01 |
| rm_work | lol yes | 21:01 |
| rm_work | i mean, the only action here is | 21:02 |
| rm_work | "failover needs to happen for this amp_id" | 21:02 |
| rm_work | and we look up which amp it is and where the failover needs to go | 21:02 |
| rm_work | i'm not going to be like | 21:02 |
| *** gcheresh has joined #openstack-lbaas | 21:02 | |
| xgerman_ | which is the same as saying “I am master: | 21:02 |
| rm_work | right | 21:02 |
| rm_work | yes that is correct | 21:02 |
| rm_work | but it needs to be sent out-of-band | 21:02 |
| xgerman_ | yes | 21:02 |
| rm_work | not "wait for the next health message" | 21:02 |
| johnsom | Yeah, I lean heavily towards "I'm master" vs. "go do something" | 21:02 |
| rm_work | right | 21:02 |
| rm_work | i mean that's what it is | 21:03 |
| *** aojea has quit IRC | 21:03 | |
| rm_work | type: failover | 21:03 |
| rm_work | just means "hey i'm master now" | 21:03 |
| rm_work | but i guess i could have it just compile the same message | 21:03 |
| rm_work | one sec | 21:03 |
| johnsom | Yeah, I really don't like failover as it's overloaded already with the 12 layers of failover we have | 21:03 |
| xgerman_ | +1 | 21:03 |
| rm_work | how do I tell if the amphora thinks it is a master or backup | 21:04 |
| rm_work | do we send that in some config? | 21:04 |
| xgerman_ | I like the “I am master message” even if keepalived can change VIP so we cna update DB/status | 21:04 |
| xgerman_ | it’s in config | 21:05 |
| rm_work | looking for it | 21:05 |
| rm_work | not seeing it | 21:06 |
| johnsom | Well, technically it doesn't matter which is which. However, when we build, the "MASTER" has a slightly different priority config just to make the non-preempt work right. | 21:06 |
| johnsom | At any given time, currently, we don't know which is which. | 21:06 |
| rm_work | yeah ok so | 21:06 |
| rm_work | how would an amp know it needs to set the "master" flag to True or False | 21:06 |
| rm_work | except specifically from one of these events | 21:07 |
| johnsom | I thought you said you were going to use the script hook | 21:07 |
| rm_work | yes, but if it can't tell that, it can't be something we also include in the next heartbeat messages | 21:07 |
| xgerman_ | I think we should make that a permanent status | 21:07 |
| johnsom | Ah, yeah, well, it doesn't.... | 21:07 |
| xgerman_ | keepalived knows it outputs it in the logs | 21:07 |
| johnsom | I guess you could write it out to the FS | 21:07 |
| rm_work | was trying to appease xgerman_ because it wasn't a bad idea | 21:08 |
| rm_work | but i think maybe it's not possible? | 21:08 |
| rm_work | keepalived config has it I guess | 21:08 |
| johnsom | Yeah, but not reliably | 21:08 |
| rm_work | yeah but then that's a state we have to track | 21:08 |
| rm_work | ugh | 21:08 |
| rm_work | no | 21:08 |
| rm_work | i think i'm back to "just send it this once | 21:08 |
| rm_work | " | 21:08 |
| johnsom | You can't trust the log for the state | 21:08 |
| rm_work | yeah | 21:08 |
| rm_work | can't really trust anything for the state | 21:08 |
| rm_work | honestly | 21:08 |
| johnsom | You can send a signal to the process and have it dump a state file, but you don't want to do that every heartbeat | 21:09 |
| rm_work | so basically.... back to just doing a one-off message | 21:09 |
| rm_work | but i think i will copy the message format and just add the field | 21:09 |
| xgerman_ | +1 | 21:10 |
| rm_work | I am a little worried about SEQ tho | 21:10 |
| rm_work | because it's always going to be 1? | 21:10 |
| rm_work | because this is not a daemon, it's a start-once script | 21:10 |
| rm_work | so there's normal health messages with a valid SEQ | 21:10 |
| rm_work | and then all the sudden one comes in with "1" | 21:10 |
| rm_work | which would just look like an out-of-sequence message | 21:10 |
| johnsom | I asked for that, but I don't know if they implemented it. It's to stop replay issues. | 21:10 |
| rm_work | yeah which... i guess makes us vulnerable to those again | 21:11 |
| xgerman_ | not if we keep book | 21:11 |
| johnsom | Assuming they implemented it | 21:11 |
| rm_work | but yeah I'm not sure where it's read | 21:11 |
| *** gcheresh has quit IRC | 21:11 | |
| rm_work | xgerman_: the normal health sender daemon keeps the seq | 21:11 |
| rm_work | this isn't that, it's a one-off run by keepalived | 21:12 |
| rm_work | so it will always have new state | 21:12 |
| rm_work | I guess I could implement it as like... | 21:12 |
| xgerman_ | if you know that AMP A is master and some message comes telling you that - yoi cna discard it | 21:12 |
| rm_work | the health daemon process emits the emergency message on a signal -- and keepalived script just sends that signal to the process | 21:12 |
| johnsom | So, I have to step back and ask, why can't normal VRRP moving the private VIP address just work like it does upstream? | 21:13 |
| rm_work | xgerman_: yes but if i format the message exactly the same as the other health messages, except with the "master" field to signal it needs to take over, then the receiving end may just discard it for being out-of-sequence | 21:14 |
| rm_work | johnsom: no BGP? | 21:14 |
| johnsom | I mean, it's old tech that should just work..... | 21:14 |
| rm_work | BGP? | 21:14 |
| rm_work | sure | 21:14 |
| johnsom | We don't have BGP upstream | 21:14 |
| rm_work | err | 21:14 |
| rm_work | look at what the failover script does | 21:14 |
| rm_work | it is doing BGP | 21:14 |
| johnsom | Your failover script? | 21:14 |
| rm_work | no | 21:14 |
| rm_work | i mean | 21:14 |
| rm_work | upstream octavia-keepalived.conf | 21:15 |
| rm_work | it just does a garp | 21:15 |
| johnsom | We currently have zero BGP in octavia today | 21:15 |
| rm_work | which relies on BGP working | 21:15 |
| johnsom | No, garp is not related to BGP at all | 21:15 |
| rm_work | err | 21:15 |
| rm_work | ok what is the garp related to | 21:16 |
| rm_work | whatever it is | 21:16 |
| rm_work | that is the thing we don't support | 21:16 |
| rm_work | we do static routing | 21:16 |
| rm_work | ARPing is not how our network does routing configuration | 21:17 |
| johnsom | ARP has nothing to do with routing either. | 21:17 |
| rm_work | ok well | 21:18 |
| rm_work | what happens is | 21:18 |
| rm_work | something sends an ARP | 21:18 |
| johnsom | It's a map between an IP and a MAC for layer 2 | 21:18 |
| rm_work | and nothing happens | 21:18 |
| rm_work | because the switches don't listen for it | 21:18 |
| rm_work | they have static routes | 21:18 |
| johnsom | I have static routes, devstack has static routes | 21:18 |
| rm_work | I can send 1000 garps | 21:18 |
| rm_work | and it will not affect where packets go | 21:18 |
| rm_work | ah maybe it's "FLIPs are L3" | 21:19 |
| johnsom | What you must have is a static ARP table | 21:20 |
| rm_work | isn't that "static routes"? | 21:20 |
| johnsom | if garps and the ip migration doesn't work | 21:20 |
| johnsom | no | 21:20 |
| johnsom | ARPs play a role in networks with no router at all | 21:20 |
| *** catintheroof has quit IRC | 21:21 | |
| xgerman_ | ARP is MAC->IP; Routing is how to get to that IP | 21:22 |
| *** aojea has joined #openstack-lbaas | 21:22 | |
| openstackgerrit | Merged openstack/octavia master: Add flag to disable SSHD on the amphora image https://review.openstack.org/492332 | 21:22 |
| johnsom | When two hosts are on the same ethernet segment, configured with the same IP subnet, and host A with IP 10.1.1.5 wants to send a packet to host B with IP 10.1.1.10, host A arps a "who has IP 10.1.1.5?" and host b responds with "MAC x:x:x:x:x..." | 21:23 |
| rm_work | ah | 21:23 |
| rm_work | so yeah | 21:23 |
| rm_work | FLIPs are L3 | 21:23 |
| rm_work | not L2 | 21:23 |
| johnsom | A GARP just tells all of the hosts on the ethernet segment, hey, update your arp table, IP 10.1.1.5 is now at MAC x:X:x:xX: | 21:23 |
| rm_work | so can't redirect a FLIP with garp | 21:23 |
| rm_work | right? | 21:24 |
| johnsom | FLIP is a NAT right, IP 192.1.1.1 gets mapped to 10.1.1.5. So, your FLIP/NAT device has an interface on 10.1.1.0/8 subnet right? That interface should use ARP and honor GARP to change the MAC that owns the IP | 21:25 |
| *** cpusmith has quit IRC | 21:25 | |
| johnsom | If it doesn't, it means your FLIP device is doing something really strange and building some sort of static ARP table, which is a bad practice (part of why DVR has issues). | 21:26 |
| rm_work | sec | 21:26 |
| rm_work | i have a better idea | 21:27 |
| *** JudeC has joined #openstack-lbaas | 21:27 | |
| rm_work | JudeC: please explain the way in which our networking is dump | 21:27 |
| rm_work | *dumb | 21:27 |
| JudeC | Clarify our. you mean GoDaddy? | 21:28 |
| rm_work | why can't we change where a FLIP points via a gARP | 21:28 |
| rm_work | yes | 21:28 |
| johnsom | I mean one other way to fix this is write some code that sits on the VIP network and listens for the garps and does your FLIP magic.... | 21:28 |
| xgerman_ | it has to support ARP otherwise you wouldn;t reach anyhting | 21:29 |
| rm_work | one sec finding the irc logs for JudeC to catch up | 21:30 |
| johnsom | Well, in normal networks, but if the sender already "knows" the MAC it could just stamp it in | 21:30 |
| johnsom | Bad practice, but would work | 21:30 |
| rm_work | http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/latest.log.html#t2017-08-16T21:13:43 | 21:31 |
| JudeC | Well we are using static routes to route traffic from our leaf switches down to the VM. | 21:31 |
| johnsom | Yeah, we aren't talking about the L3 routing layer, we are talking L2 | 21:31 |
| JudeC | I believe we are layer 3 all the way down to the VM in our network, we dont route to an access switch. | 21:34 |
| JudeC | Its been a while but doesn't the GARP/ARP need to have an AS in the way to update its MAC table? | 21:36 |
| johnsom | No | 21:36 |
| johnsom | ARP/GARP is layer 2 | 21:37 |
| johnsom | No routing involved at all | 21:37 |
| rm_work | so what receives the ARP? | 21:38 |
| rm_work | a switch? | 21:38 |
| rm_work | and our switches have fully static routes, right? | 21:39 |
| rm_work | -> ARPs do nothing | 21:39 |
| rm_work | from how I understand it | 21:39 |
| johnsom | (172.21.21.3) at dc:9f:db:80:c3:31 | 21:39 |
| johnsom | That is an arp table entry | 21:39 |
| rm_work | ok but | 21:39 |
| rm_work | where does that live | 21:39 |
| JudeC | yeah, our switches don't use a layer 2 switch at all. | 21:39 |
| johnsom | On the hosts | 21:39 |
| rm_work | it doesn't live on the wire | 21:40 |
| rm_work | what hosts | 21:40 |
| rm_work | every host in the network? | 21:40 |
| johnsom | So, go into one of your amps, do an "arp -a" what do you have? | 21:40 |
| rm_work | a big list of IPs | 21:40 |
| rm_work | but like | 21:40 |
| rm_work | that's on this host | 21:40 |
| rm_work | my laptop doesn't get the ARP from my openstack VM | 21:40 |
| rm_work | so it has to go to ... somewhere, right? the switch? | 21:41 |
| rm_work | somewhere that has an ARP table | 21:41 |
| johnsom | Again, two hosts, a cross over cable (ok, dating myself) between them. | 21:42 |
| JudeC | I am checking on something really quick as to not appear like a dumb dumb one sec :P | 21:42 |
| rm_work | get that johnsom | 21:42 |
| johnsom | host A will show Host B's IP and MAC in it's ARP table, Host B will show Host A's IP and MAC | 21:42 |
| rm_work | but that's not how the internet works | 21:42 |
| rm_work | so the switches hold the ARP tables | 21:43 |
| rm_work | right? | 21:43 |
| johnsom | No | 21:43 |
| rm_work | what does? | 21:43 |
| johnsom | Anything that needs to send an IP packet. The kernel of the instance in most cases | 21:43 |
| rm_work | errr | 21:44 |
| rm_work | ok but let's say the two machines are: | 21:44 |
| rm_work | my laptop at home | 21:44 |
| johnsom | A switch just has a list of MAC to physical port mappings, no need for an ARP table | 21:44 |
| rm_work | an opstack VM | 21:44 |
| johnsom | Same subnet? | 21:44 |
| rm_work | no | 21:44 |
| johnsom | They will not see each other in the arp table | 21:44 |
| rm_work | ok | 21:44 |
| rm_work | so | 21:44 |
| rm_work | the same is true for us | 21:44 |
| *** yamamoto has joined #openstack-lbaas | 21:44 | |
| johnsom | You would have to have a router | 21:45 |
| johnsom | no | 21:45 |
| rm_work | most VMs are not in the same subnet as other VMs | 21:45 |
| *** yamamoto has quit IRC | 21:45 | |
| johnsom | Our amps are, that is how we can move the same IP back and forth | 21:45 |
| rm_work | right but ours aren't and can't | 21:45 |
| rm_work | that's the whole point | 21:45 |
| rm_work | we can't move an IP back and forth | 21:45 |
| rm_work | because they just can't be connected to the same subnets | 21:45 |
| rm_work | so we use FLIPs | 21:45 |
| rm_work | which are L3 | 21:45 |
| johnsom | It's a NAT | 21:46 |
| *** aojea has quit IRC | 21:46 | |
| johnsom | So, yeah, your issue is different subnets. When you plug the neutron port for the VIP on the amp, you pick a random subnet? | 21:47 |
| rm_work | yes | 21:47 |
| johnsom | oye | 21:48 |
| rm_work | nova picks it | 21:48 |
| rm_work | it's up to whatever subnet is available on the host it schedules the VM to | 21:48 |
| rm_work | we don't get to pick subnets, only networks | 21:48 |
| rm_work | thus FLIPs | 21:48 |
| JudeC | Yeah so basically we use static routes that say 192.168.1.14/32 routes to 10.0.0.124 where 10.0.0.124 is the fixed IP address of your host. | 21:49 |
| JudeC | 192.* would be your float (network) | 21:50 |
| *** sshank has quit IRC | 21:50 | |
| johnsom | Basically every instance gets a subnet of /32 | 21:50 |
| JudeC | as far as the static route is concerned yes. | 21:50 |
| *** yamamoto has joined #openstack-lbaas | 21:51 | |
| JudeC | so it literally looks like in the switch config: | 21:51 |
| JudeC | ip route 192.168.1.14/32 10.0.0.124 | 21:51 |
| JudeC | for every single VM | 21:51 |
| JudeC | and its all managed via neutron which makes a call out to an external API that actually writes the routes to the L3 switches. | 21:52 |
| JudeC | its a mess | 21:52 |
| *** apuimedo has quit IRC | 21:53 | |
| johnsom | Yeah, so customers can't use broadcast or multicast with their instances... Well, this isn't here nor there. | 21:53 |
| *** apuimedo has joined #openstack-lbaas | 21:54 | |
| JudeC | whoa whoa whoa | 21:54 |
| JudeC | I just had an idea | 21:54 |
| johnsom | Basically our act/stdby won't work with that kind of networking. It's designed for typical subnets for the amp VIP networks. What you are trying to do is more like the act/act | 21:54 |
| JudeC | The static route API that neutron uses can write routes that have a larger network, as long as that network in contiguous I think a GARP should work... | 21:55 |
| rm_work | i don't think we can guarantee it's contiguous | 21:56 |
| JudeC | wait nvm its needs to be the nexthop too.... askfjlskd | 21:56 |
| rm_work | but yes, our active-standby isn't | 21:56 |
| rm_work | the same | 21:56 |
| *** aojea has joined #openstack-lbaas | 21:56 | |
| rm_work | we don't use one IP with AAP | 21:56 |
| rm_work | we use a FLIP | 21:56 |
| rm_work | and we need to tell neutron to move it | 21:56 |
| rm_work | which, again, is why i am doing this whole thing | 21:56 |
| *** yamamoto has quit IRC | 21:58 | |
| johnsom | Yeah, this is just going to be slow. | 21:58 |
| rm_work | ~7s | 22:00 |
| *** sshank has joined #openstack-lbaas | 22:00 | |
| johnsom | Yeah, you are going to have to use the notify script, send it up in a heartbeat, have something that tells neutron to update your NAT. The other problem for you is keepalived runs inside the netns, so isolated from amp-agent/lb-mgmt-net | 22:01 |
| johnsom | It handles tenant traffic, so it's isolated off | 22:02 |
| rm_work | errr | 22:02 |
| rm_work | crap | 22:02 |
| rm_work | ok so | 22:02 |
| rm_work | yeah, signal i think | 22:02 |
| rm_work | that actually solves some problems | 22:02 |
| xgerman_ | I guess ACTIVE-ACTIVE won’t be great in your env neither | 22:05 |
| xgerman_ | OVS based does ARP and the other one needs routers/BGP | 22:05 |
| rm_work | johnsom: oh well actually lol it doesn't matter | 22:10 |
| rm_work | remember we don't have tenant networks | 22:10 |
| rm_work | all our traffic is technically on the same network | 22:10 |
| rm_work | IE, mgmt-on-vip-net | 22:10 |
| rm_work | XD | 22:10 |
| johnsom | I think you are getting into the realm of local driver here.... | 22:13 |
| johnsom | custom amp image | 22:14 |
| johnsom | Maybe | 22:15 |
| rm_work | i mean yeah we have some patches to the amp image | 22:15 |
| rm_work | getting the netmask right on the VIP is one | 22:15 |
| rm_work | i WAS originally adding this directly to my FLIP driver patch | 22:16 |
| rm_work | but i thought maybe we'd want some part of the framework upstream... | 22:16 |
| rm_work | for sending custom messages | 22:16 |
| rm_work | so should i just abandon this and fold it back into the FLIP thing? | 22:18 |
| rm_work | but yeah i think i'm going to implement this as a signal | 22:18 |
| xgerman_ | I still see value in us reporting who is master/backup — after all we have a column in the DB | 22:33 |
| xgerman_ | but if the fidelity sucks… | 22:33 |
| *** fnaval has quit IRC | 22:35 | |
| *** aojea has quit IRC | 22:39 | |
| *** aojea has joined #openstack-lbaas | 22:40 | |
| johnsom | Yeah, it gets a bit strange too, so would be good to have a solid use case. I.e. we get a "MASTER" from amp 1 we have to go find amp 2 and change it's status too. | 22:45 |
| johnsom | I think we *could* do it with the notify scripts, but it would need testing and then the database stuffs on the controller side | 22:45 |
| johnsom | I guess it wouldn't be too bad, we could just use a negative where clause in the DB. | 22:46 |
| *** fnaval has joined #openstack-lbaas | 22:46 | |
| *** fnaval has quit IRC | 22:47 | |
| *** fnaval has joined #openstack-lbaas | 22:47 | |
| xgerman_ | yeah, not urgent but for the back of our heads | 22:50 |
| *** ssmith has joined #openstack-lbaas | 22:52 | |
| *** ssmith has quit IRC | 23:03 | |
| *** ssmith has joined #openstack-lbaas | 23:04 | |
| *** sshank has quit IRC | 23:10 | |
| rm_work | i guess for now i'm implement this as part of FLIP instead of standalone :/ | 23:16 |
| rm_work | that's fine | 23:16 |
| rm_work | if I don't have to account for more general cases, it's a lot simpler/faster | 23:17 |
| *** ssmith has quit IRC | 23:19 | |
| rm_work | johnsom: yeah looks like we do nothing with SEQ | 23:19 |
| *** ssmith has joined #openstack-lbaas | 23:19 | |
| rm_work | ¯\_(ツ)_/¯ | 23:20 |
| rm_work | ^^ this emoji describes my general mental state recently with regards to ... a lot of this | 23:20 |
| rm_work | johnsom: you can remove your -2 on https://review.openstack.org/#/c/488885/ now right? | 23:29 |
| johnsom | No | 23:29 |
| johnsom | We are still in feature freeze | 23:29 |
| rm_work | ? thought we were merging stuff | 23:29 |
| johnsom | Just the two for RC2 | 23:30 |
| rm_work | my SSHD thing merged, so I figured we were good and just went and +A'd a couple of the other ones that were ready | 23:30 |
| rm_work | I guess I should stop those | 23:30 |
| johnsom | Well, let them go, we will just hope RC2 is it and those don't entangle | 23:31 |
| rm_work | whatev already removed my +A but gate is in-progress so we'll see what happens | 23:31 |
| johnsom | Yeah, they will likely merge | 23:32 |
| *** aojea has quit IRC | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!