| johnsom | rm_work xgerman_ Can you guys poke the RC 2 backports? https://review.openstack.org/494332 and https://review.openstack.org/494331 | 00:04 |
|---|---|---|
| johnsom | That way I can cut RC2 tomorrow morning | 00:04 |
| rm_work | +2'd | 00:05 |
| johnsom | Thanks | 00:05 |
| *** ssmith has quit IRC | 00:08 | |
| *** wasmum has quit IRC | 00:21 | |
| *** aojea has joined #openstack-lbaas | 00:40 | |
| *** JudeC has quit IRC | 00:43 | |
| *** aojea has quit IRC | 00:45 | |
| *** aojea has joined #openstack-lbaas | 01:41 | |
| *** aojea has quit IRC | 01:47 | |
| *** rtjure has quit IRC | 03:34 | |
| *** rtjure has joined #openstack-lbaas | 03:39 | |
| *** aojea has joined #openstack-lbaas | 03:43 | |
| *** aojea has quit IRC | 03:47 | |
| *** links has joined #openstack-lbaas | 04:01 | |
| *** tomtomtom has quit IRC | 04:42 | |
| *** openstack has quit IRC | 04:42 | |
| *** openstack has joined #openstack-lbaas | 04:46 | |
| *** kbyrne has joined #openstack-lbaas | 04:47 | |
| *** yuanying has joined #openstack-lbaas | 04:47 | |
| *** thomasem has joined #openstack-lbaas | 04:47 | |
| *** kevinbenton has joined #openstack-lbaas | 04:47 | |
| *** isantosp_ has joined #openstack-lbaas | 04:48 | |
| *** dayou1 has joined #openstack-lbaas | 04:48 | |
| *** rtjure has joined #openstack-lbaas | 04:48 | |
| *** bcafarel has joined #openstack-lbaas | 04:50 | |
| *** fnaval has joined #openstack-lbaas | 04:50 | |
| *** ptoohill has joined #openstack-lbaas | 04:50 | |
| *** PagliaccisCloud has joined #openstack-lbaas | 04:50 | |
| *** jidar has joined #openstack-lbaas | 04:51 | |
| *** m-greene has joined #openstack-lbaas | 04:51 | |
| *** m-greene_ has joined #openstack-lbaas | 04:52 | |
| *** Guest75213 has joined #openstack-lbaas | 04:52 | |
| *** raginbajin has joined #openstack-lbaas | 04:55 | |
| *** mnaser has joined #openstack-lbaas | 04:57 | |
| *** gcheresh has joined #openstack-lbaas | 05:47 | |
| *** rcernin has joined #openstack-lbaas | 05:57 | |
| *** ianychoi has quit IRC | 06:23 | |
| *** ianychoi has joined #openstack-lbaas | 06:26 | |
| *** pcaruana has joined #openstack-lbaas | 06:42 | |
| *** aojea has joined #openstack-lbaas | 06:45 | |
| *** tesseract has joined #openstack-lbaas | 06:47 | |
| *** aojea has quit IRC | 06:49 | |
| *** ajo has joined #openstack-lbaas | 07:19 | |
| *** openstackgerrit has joined #openstack-lbaas | 07:20 | |
| openstackgerrit | Ji Chengke proposed openstack/octavia master: Change 14.04 to 16.04 in devstack setup guide https://review.openstack.org/494407 | 07:20 |
| *** Alex_Staf has joined #openstack-lbaas | 07:30 | |
| *** rtjure has quit IRC | 07:35 | |
| *** rtjure has joined #openstack-lbaas | 07:39 | |
| *** JudeC has joined #openstack-lbaas | 07:45 | |
| *** Alex_Staf has quit IRC | 07:55 | |
| *** tesseract has quit IRC | 07:58 | |
| *** openstackgerrit has quit IRC | 08:17 | |
| *** Alex_Staf has joined #openstack-lbaas | 08:18 | |
| *** aojea has joined #openstack-lbaas | 08:19 | |
| *** ajo has quit IRC | 08:27 | |
| *** eezhova has joined #openstack-lbaas | 08:29 | |
| *** logan- has quit IRC | 08:37 | |
| *** aojea has quit IRC | 08:37 | |
| *** JudeC has quit IRC | 08:46 | |
| *** kong has quit IRC | 10:10 | |
| *** kong has joined #openstack-lbaas | 10:13 | |
| *** belharar has joined #openstack-lbaas | 10:18 | |
| *** fnaval has quit IRC | 10:47 | |
| *** aojea has joined #openstack-lbaas | 11:38 | |
| *** atoth has joined #openstack-lbaas | 11:40 | |
| *** aojea has quit IRC | 11:50 | |
| *** aojea has joined #openstack-lbaas | 12:17 | |
| *** saphi has joined #openstack-lbaas | 12:25 | |
| saphi | Hi everyone, I'm installing neutron-lbaas-dashboard in stable/pike branch but After restart apache2 I can't access to horizon with an error on apache error log: NotImplementedError: The DisableLoadBalancer BatchAction class must have both action_past and action_present methods | 12:26 |
| *** saphi has quit IRC | 12:35 | |
| amotoki | followup comment: it must be a bug of neutron-lbaas-dashboard | 12:37 |
| amotoki | horizon pike dropped action_past and action_present *attributes* (which are deprecated for a long time) | 12:37 |
| amotoki | the change happened early Pike | 12:38 |
| *** saphi has joined #openstack-lbaas | 12:42 | |
| *** saphi has quit IRC | 12:46 | |
| *** catintheroof has joined #openstack-lbaas | 12:49 | |
| *** catintheroof has quit IRC | 12:50 | |
| *** catintheroof has joined #openstack-lbaas | 12:50 | |
| *** aojea has quit IRC | 12:51 | |
| *** aojea has joined #openstack-lbaas | 12:52 | |
| *** belharar has quit IRC | 12:56 | |
| *** saphi has joined #openstack-lbaas | 13:05 | |
| *** aojea has quit IRC | 13:11 | |
| *** aojea has joined #openstack-lbaas | 13:15 | |
| *** aojea has quit IRC | 13:23 | |
| *** aojea has joined #openstack-lbaas | 13:30 | |
| johnsom | Check that you are following the installation instructions exactly and not loading both files. | 13:33 |
| johnsom | Dashboard was working fine a week or two ago | 13:34 |
| *** belharar has joined #openstack-lbaas | 13:37 | |
| *** ssmith has joined #openstack-lbaas | 13:39 | |
| *** cpusmith has joined #openstack-lbaas | 13:40 | |
| *** saphi has quit IRC | 13:40 | |
| *** aojea has quit IRC | 13:41 | |
| *** aojea_ has joined #openstack-lbaas | 13:43 | |
| *** ssmith has quit IRC | 13:44 | |
| *** cpusmith has quit IRC | 13:44 | |
| *** aojea__ has joined #openstack-lbaas | 13:45 | |
| *** aojea_ has quit IRC | 13:48 | |
| *** b3nt_pin is now known as beagles | 13:49 | |
| *** aojea__ has quit IRC | 13:50 | |
| *** aojea has joined #openstack-lbaas | 13:50 | |
| *** aojea has quit IRC | 13:56 | |
| *** aojea has joined #openstack-lbaas | 13:59 | |
| *** belharar has quit IRC | 14:00 | |
| *** belharar has joined #openstack-lbaas | 14:01 | |
| *** armax has joined #openstack-lbaas | 14:02 | |
| *** slaweq has joined #openstack-lbaas | 14:05 | |
| *** aojea has quit IRC | 14:12 | |
| *** aojea has joined #openstack-lbaas | 14:25 | |
| *** belharar has quit IRC | 14:26 | |
| *** fnaval has joined #openstack-lbaas | 14:30 | |
| amotoki | johnsom: I am a bit surprised the dashboard worked well as horizon dropped action_present/past in Pike-2 https://github.com/openstack/horizon/commit/ad642346e0fefa285b0a471bbd5fa9d2779477b0 | 14:32 |
| amotoki | if nobody works on it, I can send a fix tomorrow | 14:32 |
| *** fnaval has quit IRC | 14:39 | |
| *** eezhova has quit IRC | 14:43 | |
| *** fnaval has joined #openstack-lbaas | 14:52 | |
| *** gcheresh has quit IRC | 14:57 | |
| *** KeithMnemonic has joined #openstack-lbaas | 14:58 | |
| *** KeithMnemonic1 has joined #openstack-lbaas | 15:00 | |
| *** KeithMnemonic has quit IRC | 15:00 | |
| johnsom | amotoki that would be appreciated | 15:02 |
| *** KeithMnemonic has joined #openstack-lbaas | 15:03 | |
| *** KeithMnemonic has quit IRC | 15:03 | |
| *** KeithMnemonic1 has quit IRC | 15:03 | |
| *** KeithMnemonic has joined #openstack-lbaas | 15:03 | |
| *** dougwig has joined #openstack-lbaas | 15:06 | |
| *** eezhova has joined #openstack-lbaas | 15:18 | |
| *** logan- has joined #openstack-lbaas | 15:31 | |
| *** rcernin has quit IRC | 15:32 | |
| *** dmellado has joined #openstack-lbaas | 15:37 | |
| *** logan- has quit IRC | 15:40 | |
| *** logan- has joined #openstack-lbaas | 15:40 | |
| *** ssmith has joined #openstack-lbaas | 15:42 | |
| *** belharar has joined #openstack-lbaas | 16:03 | |
| *** rcernin has joined #openstack-lbaas | 16:08 | |
| *** ssmith has quit IRC | 16:15 | |
| *** eezhova has quit IRC | 16:18 | |
| *** aojea has quit IRC | 16:18 | |
| *** saphi has joined #openstack-lbaas | 16:19 | |
| *** eezhova has joined #openstack-lbaas | 16:31 | |
| *** aojea has joined #openstack-lbaas | 16:36 | |
| *** eezhova has quit IRC | 16:38 | |
| *** belharar has quit IRC | 16:53 | |
| *** JudeC has joined #openstack-lbaas | 16:58 | |
| *** sshank has joined #openstack-lbaas | 17:01 | |
| *** saphi has quit IRC | 17:13 | |
| *** aojea has quit IRC | 17:13 | |
| *** eezhova has joined #openstack-lbaas | 17:16 | |
| *** aojea has joined #openstack-lbaas | 17:34 | |
| *** aojea has quit IRC | 17:36 | |
| *** aojea has joined #openstack-lbaas | 17:50 | |
| *** aojea has quit IRC | 17:54 | |
| *** SumitNaiksatam has joined #openstack-lbaas | 17:59 | |
| *** vegarl has joined #openstack-lbaas | 18:03 | |
| *** JudeC has quit IRC | 18:27 | |
| *** sshank has quit IRC | 18:38 | |
| *** catintheroof has quit IRC | 18:54 | |
| *** KeithMnemonic has quit IRC | 19:21 | |
| *** sshank has joined #openstack-lbaas | 19:42 | |
| *** gcheresh has joined #openstack-lbaas | 19:56 | |
| *** armax has quit IRC | 20:13 | |
| *** rcernin has quit IRC | 20:16 | |
| *** JudeC has joined #openstack-lbaas | 20:20 | |
| *** sshank has quit IRC | 20:22 | |
| *** sshank has joined #openstack-lbaas | 20:27 | |
| *** apuimedo has quit IRC | 20:43 | |
| *** apuimedo has joined #openstack-lbaas | 20:47 | |
| *** sshank has quit IRC | 20:49 | |
| *** sshank has joined #openstack-lbaas | 21:04 | |
| *** gcheresh has quit IRC | 21:11 | |
| *** dougwig has quit IRC | 21:16 | |
| *** armax has joined #openstack-lbaas | 21:17 | |
| *** eezhova has quit IRC | 21:26 | |
| *** eezhova has joined #openstack-lbaas | 21:29 | |
| *** eezhova has quit IRC | 21:32 | |
| *** rtjure has quit IRC | 21:47 | |
| rm_work | johnsom: how is the keepalived service started? | 22:04 |
| *** aojea has joined #openstack-lbaas | 22:06 | |
| *** pck has joined #openstack-lbaas | 22:08 | |
| *** pck has quit IRC | 22:08 | |
| *** pck has joined #openstack-lbaas | 22:08 | |
| *** pck is now known as pckizer | 22:09 | |
| rm_work | johnsom: hey, uhh... I don't know if we're restarting the keepalived process correctly when it gets sent a new config | 22:20 |
| rm_work | it seems we *always* just do: upload_vrrp_config, manage_service_vrrp(start) | 22:20 |
| rm_work | never reload | 22:20 |
| rm_work | so like, in this case i am testing by failing the *backup*, but what happens is: backup goes down, healthmanager recycles it, sends a new vrrp config to the old one, runs "start" | 22:21 |
| rm_work | it should be reload, right? | 22:21 |
| rm_work | the `reload_vrrp_service` function in `vrrp_rest_driver.py` is never used T_T | 22:24 |
| rm_work | only `start_vrrp_service` | 22:24 |
| rm_work | there is no driver task for it, only start/stop | 22:24 |
| rm_work | ok so yeah this is broken | 22:26 |
| johnsom | Looking | 22:26 |
| rm_work | but the question is, does it MATTER | 22:26 |
| rm_work | look for get_vrrp_subflow | 22:26 |
| rm_work | in our failover flow, that's the only one we'd ever call | 22:26 |
| rm_work | and it always does upload_config->start | 22:26 |
| rm_work | non-conditional | 22:26 |
| xgerman_ | yeah, why would you need to restart vrrp? | 22:26 |
| xgerman_ | it’s not like we change the VIP or anyhting | 22:27 |
| rm_work | hmmm | 22:27 |
| rm_work | well, why do we bother re-uploading the config even then? | 22:27 |
| xgerman_ | if a machine is down we just replace it | 22:27 |
| rm_work | ok so ... lulz... | 22:28 |
| rm_work | what IP does it use? looks like.. uhh | 22:28 |
| xgerman_ | yep, that bug might make it work ;- | 22:28 |
| rm_work | vrrp_ip right? | 22:28 |
| johnsom | Yeah, failover it's not started yet, it will always be a start | 22:28 |
| rm_work | sooooo in *my* setup, that IP *does* change | 22:28 |
| xgerman_ | yep, vrrp-ip | 22:28 |
| rm_work | when it does a failover, the new amp will have a different vrrp_ip | 22:29 |
| xgerman_ | well, that is a new challenge | 22:29 |
| johnsom | Where are we ever uploading a VRRP config to an already running VRRP? | 22:29 |
| rm_work | that's not the case upstream? (sometimes i lose track of what exactly my differences are) | 22:29 |
| xgerman_ | we made it so that we could bring up stuff with the same port (IP) | 22:29 |
| rm_work | i didn't think that was a change on my side | 22:29 |
| rm_work | oh maybe it is | 22:29 |
| xgerman_ | you should still have the port | 22:30 |
| rm_work | yeah ok, so it's just broken for me | 22:30 |
| rm_work | well, we can't salvage that port | 22:30 |
| xgerman_ | in you world the machine might be on a subnet the port isn’t on? | 22:30 |
| johnsom | Where are we ever uploading a VRRP config to an already running VRRP? | 22:30 |
| rm_work | because there's a 95% chance or so that it's unreachable from whatever host the new VM is spun up on | 22:30 |
| rm_work | johnsom: i'm just looking at the amp agent log... | 22:30 |
| rm_work | let me pastebin | 22:30 |
| xgerman_ | rm_work I don’t say that lightly but your network is screwing you | 22:31 |
| rm_work | http://paste.openstack.org/show/mzwf9iUXssb4U7R915ud/ | 22:31 |
| rm_work | xgerman_: yeah it's really dumb, thus the whole reason for the L3 VIP driver | 22:32 |
| rm_work | anyway, that paste shows what happens on the amp agent for the MASTER as I repeatedly failover the BACKUP | 22:32 |
| xgerman_ | without HA VIPs you are basically bever HA | 22:32 |
| rm_work | line 10-21 is the first failover, 22-32 is the second | 22:33 |
| rm_work | let me look at the flows | 22:34 |
| xgerman_ | yeah, we use teh same flows for create with VRRP | 22:38 |
| johnsom | rm_work What do you mean first and second failover? The logs would be wiped with each failover | 22:39 |
| rm_work | ? | 22:39 |
| rm_work | this is for the one that DIDN'T fail | 22:39 |
| rm_work | i'm failing one and watching logs on the other | 22:39 |
| johnsom | Ah, ok, yeah, that makes sense. It just iterates over all the amps on the LB, so basically it's pushing down the same config and a "start" which is already started. | 22:40 |
| rm_work | yeah | 22:40 |
| xgerman_ | yep | 22:40 |
| xgerman_ | takes the LB | 22:40 |
| rm_work | but in my case the vrrp_ips change | 22:40 |
| xgerman_ | instead of specific AMP | 22:40 |
| rm_work | so the one that didn't fail, needs a new config? I THINK? | 22:40 |
| johnsom | But you are saying the config would change? I thought the unicast peer was the VIP address and not the base address | 22:40 |
| rm_work | ah hmm | 22:40 |
| rm_work | let me see if there's any mention of the peer address | 22:40 |
| rm_work | i thought that had to be there? for them to communicate? | 22:41 |
| johnsom | Right, but that is the only thing that *could* change in that config | 22:41 |
| rm_work | yes, unicast_peer | 22:41 |
| rm_work | it updates the config with the new peer | 22:41 |
| rm_work | but the service isn't restarted/reloaded | 22:41 |
| rm_work | so the running version has the old peer | 22:41 |
| xgerman_ | in our case we re-use port so doesn’t matter | 22:42 |
| rm_work | sooo... can we do it right though? :/ | 22:42 |
| johnsom | Well, maybe.... | 22:42 |
| rm_work | johnsom: yeah so my question would be | 22:42 |
| rm_work | if the *new peer* has the correct config and is newly started | 22:43 |
| rm_work | does it initiate the pair by connecting | 22:43 |
| rm_work | and the bad config on the older node doesn't matter? | 22:43 |
| rm_work | i feel like it probably should matter if the configs are correct.. but | 22:43 |
| johnsom | I think so, but ... I'm pretty sure that is just for the initial connections and they share the peer list from then on, but, not 100% on that | 22:43 |
| rm_work | i guess this really is just a my problem | 22:44 |
| johnsom | Let me take a look at what actually happens here. | 22:44 |
| xgerman_ | so my fear is if we restart that keeplaived forgets who is master since we probably have the priorities wrong but… | 22:44 |
| johnsom | Give me a couple minutes | 22:44 |
| rm_work | hmm k | 22:44 |
| rm_work | yeah xgerman_ my understanding of how any of this vrrp stuff works is really shaky | 22:45 |
| johnsom | Priorities are static per our config, that will not change. Once a BACKUP always a backup. | 22:45 |
| xgerman_ | but if we do failober BACKUP becomes master | 22:45 |
| rm_work | johnsom: so when the master comes back up... it'll try to take back over? | 22:45 |
| xgerman_ | and we still write down MASTER config | 22:45 |
| xgerman_ | to the new node | 22:45 |
| xgerman_ | rm_work there is some code to prevent flopping | 22:46 |
| johnsom | xgerman virtually, but it's config is always a BACKUP with the lower priority | 22:46 |
| johnsom | Right | 22:46 |
| xgerman_ | not sure if a restart of both disturbs the balance | 22:47 |
| rm_work | hmm so german's point is that if we restart the config on the backup, will it revert to being a backup? | 22:47 |
| rm_work | and then we have two backups? or what | 22:47 |
| johnsom | Yeah, it *could* re-evaluate positions, but I think it should just load the config and maintain it's state. Though this could be why it's a start. | 22:48 |
| xgerman_ | indeed | 22:48 |
| johnsom | It has been a few years since I was deep into this stuff | 22:48 |
| johnsom | Like mitaka | 22:48 |
| xgerman_ | poking with a stick has been my MO | 22:49 |
| rm_work | i might just modify the start script | 22:52 |
| rm_work | such that when it gets "start", it can see if it's running, and do a reload if it's already started | 22:52 |
| rm_work | rather than try to muck with the logic in the flows themselves which are <_< | 22:52 |
| *** kbyrne has quit IRC | 22:52 | |
| rm_work | yeah that is easy | 22:54 |
| *** kbyrne has joined #openstack-lbaas | 22:55 | |
| johnsom | Yeah, so HUP -- This causes keepalived to close down all interfaces, reload its configuration, and start up with the new configuration. | 22:58 |
| johnsom | So, yeah, reload will definitely cause a re-negotiation and flip the VIP over to the other host | 22:58 |
| johnsom | rm_work Question, in your environment, why are you even messing with keepalived/VRRP? | 22:59 |
| johnsom | Oh, just the "down" notification? | 22:59 |
| rm_work | keepalived needs to run a script | 22:59 |
| rm_work | yes | 22:59 |
| rm_work | i'm removing the garp config | 22:59 |
| rm_work | and adding a notify_master | 22:59 |
| johnsom | I don't think you can actually disable the garp in keepalived. That is kind of it's only reason in life | 23:00 |
| rm_work | err | 23:00 |
| rm_work | it's some config lines | 23:00 |
| rm_work | if i remove them, it shouldn't do it? i assume | 23:00 |
| rm_work | garp_master_refresh | 23:00 |
| rm_work | garp_master_refresh_repeat | 23:00 |
| johnsom | No, that just changes the defaults | 23:00 |
| rm_work | no? | 23:00 |
| rm_work | lol k | 23:00 |
| rm_work | well | 23:00 |
| johnsom | Let me look, but I don't think you can disable it totally. | 23:01 |
| rm_work | it doesn't REALLY matter if they're garping | 23:01 |
| johnsom | But, shouldn't matter for you right? | 23:01 |
| rm_work | nothing looks at it | 23:01 |
| rm_work | it'd just be ignored | 23:01 |
| rm_work | maybe I can set the time up to something ridiculous | 23:01 |
| rm_work | so it at least doesn't spam stuff over the network as often | 23:01 |
| rm_work | "every 20 years, send a garp please" | 23:02 |
| rm_work | effectively off | 23:02 |
| johnsom | vrrp_garp_master_repeat 0 maybe | 23:02 |
| rm_work | yeah | 23:02 |
| rm_work | was just thinking | 23:02 |
| rm_work | anyway i'm basically failing at getting it to actually run the notify scripts anyway <_< | 23:04 |
| rm_work | it doesn't seem to want to do it | 23:04 |
| *** fnaval has quit IRC | 23:06 | |
| rm_work | i have all of these set: | 23:06 |
| rm_work | notify_master "/bin/haproxy-vrrp-alert" | 23:06 |
| rm_work | notify_backup "/bin/haproxy-vrrp-alert" | 23:06 |
| rm_work | notify_fault "/bin/haproxy-vrrp-alert" | 23:06 |
| rm_work | none of them run | 23:06 |
| rm_work | bleh | 23:06 |
| *** aojea has quit IRC | 23:07 | |
| johnsom | perms right? | 23:07 |
| rm_work | hmm | 23:07 |
| rm_work | what user does it run as | 23:07 |
| johnsom | root | 23:07 |
| rm_work | seems root | 23:08 |
| rm_work | yes | 23:08 |
| rm_work | <_< | 23:08 |
| johnsom | But are the scripts exec | 23:08 |
| rm_work | I mean i can run the script manually | 23:08 |
| rm_work | so yes | 23:08 |
| rm_work | trying again | 23:08 |
| rm_work | also, 0 for repeat doesn't work | 23:08 |
| johnsom | Yeah, I pretty much guessed turning off garp was a no-go | 23:09 |
| rm_work | i just set the refresh time to 20 years | 23:09 |
| rm_work | I think that is workable :P | 23:10 |
| rm_work | OOOOOR not | 23:10 |
| rm_work | that makes it do it every second | 23:10 |
| rm_work | ok it seems to be fine with montly | 23:11 |
| rm_work | *monthly | 23:11 |
| rm_work | yearly is also too long | 23:11 |
| rm_work | i wonder what MAX is | 23:12 |
| johnsom | You are putting the scripts inside the vrrp_instance block right? | 23:14 |
| rm_work | yes | 23:14 |
| rm_work | http://paste.openstack.org/show/618728/ | 23:14 |
| rm_work | eventually it'll be only notify_master but | 23:15 |
| rm_work | i just want it to freaking trigger, lol | 23:15 |
| johnsom | try dropping the quotes | 23:16 |
| rm_work | hmmm | 23:16 |
| xgerman_ | didn’t you guys implement etcd for health? | 23:16 |
| johnsom | Hahahaha | 23:17 |
| xgerman_ | just use a lock there and then switch :-) | 23:17 |
| johnsom | That was just josh drive-by | 23:17 |
| xgerman_ | he probably would feel vindicated right now ;-) | 23:17 |
| johnsom | kinda like the patches to fix health manager lifecycle | 23:17 |
| xgerman_ | lol | 23:17 |
| xgerman_ | he feels like a manager | 23:17 |
| xgerman_ | like the guy who write the UDP code | 23:18 |
| johnsom | So, the other though, if dropping the quotes doesn't work, is to try the group approach | 23:18 |
| johnsom | https://www.irccloud.com/pastebin/E7Dq1IBq/ | 23:18 |
| johnsom | Just for giggles | 23:18 |
| *** fnaval has joined #openstack-lbaas | 23:19 | |
| johnsom | Also, what is syslog saying, it usually spews on MASTER elect, I would expect something in there about the notify script | 23:19 |
| rm_work | hmmm | 23:19 |
| rm_work | yeah it didn't | 23:19 |
| rm_work | weird | 23:19 |
| rm_work | i'm looking at journalctrl | 23:19 |
| johnsom | rm_work BTW, I think -C for keepalived is what you are looking for. | 23:21 |
| rm_work | -C ? | 23:21 |
| johnsom | https://www.irccloud.com/pastebin/RUJQwcDH/ | 23:21 |
| rm_work | ah | 23:21 |
| rm_work | ok | 23:21 |
| rm_work | though we use the default start script so i'm not sure how to change what it does | 23:22 |
| xgerman_ | maybe you should use a different tool pacemaker, carp? | 23:22 |
| johnsom | So, with your notify in the config and keepalive running, kill -USR1 and look at /tmp/keepalived.data | 23:22 |
| rm_work | cmd = ("/usr/sbin/service octavia-keepalived {action}".format( | 23:22 |
| xgerman_ | you might like that better than keeplaived: https://github.com/jedisct1/UCarp | 23:23 |
| johnsom | Yeah, well, you are hacking this far, it's just a flag in the systemd service script | 23:23 |
| johnsom | xgerman You are starting to sound like Stephen | 23:23 |
| xgerman_ | it seems we are using a hammer for a screw; if we don’t need GARP why not use a tool which doesn’t offer it | 23:24 |
| johnsom | CARP does GARP too | 23:24 |
| rm_work | because this one is already installed and configured :P | 23:24 |
| rm_work | and i'd have to use an even bigger hammer to dislodge it | 23:25 |
| johnsom | and since we are already using keepalived it's probably less work | 23:25 |
| rm_work | ^^ this | 23:25 |
| rm_work | if i had to do it from scratch i'd use something simpler maybe | 23:25 |
| johnsom | Not to mention, ucarp seems to have even less docs that keepalived: http://www.ucarp.org/ | 23:25 |
| xgerman_ | well, they advertise: “no need for any dedicated extra network link between redundant hosts." | 23:27 |
| johnsom | We don't either | 23:27 |
| johnsom | Even though I kind of wanted to do that as part of the design | 23:28 |
| rm_work | interesting, so yeah no-quotes causes the script to load | 23:29 |
| rm_work | but uhh | 23:29 |
| rm_work | it also seems to call it the first time it comes up? | 23:29 |
| johnsom | no notify posibility with ucarp either | 23:29 |
| johnsom | Yeah, that makes sense | 23:29 |
| rm_work | <_< | 23:30 |
| johnsom | With quotes it is expecting some params | 23:30 |
| rm_work | that's .... un-ideal | 23:30 |
| johnsom | Well, one of them has to become the master right? | 23:30 |
| rm_work | i mean, when the master comes up the first time, it does the master notify, which in my case would be a FLIP reassign | 23:30 |
| xgerman_ | the example on the UCARP page explicitly shows scripts whcih run when becoming master and when loosing that status | 23:31 |
| rm_work | i need it to happen only when it becomes master due to a fail | 23:31 |
| rm_work | not just when it comes up and is like "woo i'm master" | 23:31 |
| johnsom | Oh, the up/down scripts | 23:31 |
| xgerman_ | yep | 23:31 |
| johnsom | to like do the interface up/down | 23:31 |
| xgerman_ | well, I guess you could put in anyhting | 23:31 |
| xgerman_ | anyhow, gotta get my kids… AFK for a long while | 23:32 |
| johnsom | rm_work Not sure how to work around that or why it matters. It seems like this is an easy way to trigger your flip assignment in the first place | 23:34 |
| rm_work | well | 23:34 |
| rm_work | i mean i guess so :P | 23:34 |
| rm_work | i suppose it's a noop if there is no change | 23:34 |
| rm_work | so whatever | 23:34 |
| rm_work | it should be fine | 23:34 |
| rm_work | *airquotes* | 23:34 |
| johnsom | I'm also thinking more, that, maybe you should just code up another topology and make this a supported config moving flips. It's just a standard neutron flip call right? | 23:35 |
| rm_work | well | 23:35 |
| rm_work | I have a driver for this remember | 23:35 |
| rm_work | FLIP is a driver | 23:35 |
| rm_work | and this goes along with that | 23:35 |
| *** openstackgerrit has joined #openstack-lbaas | 23:39 | |
| openstackgerrit | Michael Johnson proposed openstack/neutron-lbaas master: Set stable/pike to pull stable/pike neutron https://review.openstack.org/494745 | 23:39 |
| johnsom | I need to do a bit of house keeping in release prep. | 23:39 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 23:39 |
| rm_work | https://review.openstack.org/#/c/435612/36/octavia/amphorae/backends/agent/api_server/keepalived.py | 23:40 |
| rm_work | johnsom: ^^ that is what i went wirth | 23:40 |
| johnsom | Is this something you plan to try to merge? | 23:41 |
| rm_work | not in this state | 23:48 |
| rm_work | once I finally get it to "working working" state, I'm going to clean it up a lot | 23:48 |
| rm_work | and I'll have to generalize a thing or two | 23:49 |
| rm_work | but then maybe | 23:49 |
| rm_work | it SHOULD run in devstack without problem | 23:49 |
| johnsom | Ok, we just need to remember that that reload will flip the VIP | 23:49 |
| rm_work | the idea is that I think it's useful to people | 23:49 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!