| *** abaindur has joined #openstack-lbaas | 00:24 | |
| *** abaindur has quit IRC | 00:27 | |
| *** abaindur has joined #openstack-lbaas | 00:27 | |
| *** hongbin_ has joined #openstack-lbaas | 01:00 | |
| *** ltomasbo has quit IRC | 01:00 | |
| *** tobias-urdin has quit IRC | 01:00 | |
| *** phuoc_ has joined #openstack-lbaas | 01:17 | |
| *** phuoc has quit IRC | 01:17 | |
| *** abaindur has quit IRC | 01:34 | |
| sapd1_ | rm_work: build success now :D | 02:04 |
|---|---|---|
| *** jiteka has quit IRC | 02:11 | |
| *** eandersson has quit IRC | 02:11 | |
| *** jiteka has joined #openstack-lbaas | 02:11 | |
| *** jiteka has quit IRC | 02:15 | |
| rm_work | cool :) | 02:20 |
| *** eandersson has joined #openstack-lbaas | 02:24 | |
| *** eandersson has quit IRC | 02:31 | |
| *** hongbin_ has quit IRC | 03:02 | |
| *** abaindur has joined #openstack-lbaas | 03:32 | |
| *** abaindur has quit IRC | 03:35 | |
| *** abaindur has joined #openstack-lbaas | 03:35 | |
| *** sapd1_ has quit IRC | 03:45 | |
| *** sapd1 has joined #openstack-lbaas | 03:45 | |
| *** ramishra has joined #openstack-lbaas | 04:15 | |
| *** dayou_ has joined #openstack-lbaas | 04:19 | |
| *** eandersson has joined #openstack-lbaas | 04:34 | |
| *** yamamoto has quit IRC | 04:35 | |
| *** yamamoto has joined #openstack-lbaas | 04:35 | |
| *** yamamoto has quit IRC | 04:36 | |
| *** yboaron_ has joined #openstack-lbaas | 04:45 | |
| *** yamamoto has joined #openstack-lbaas | 05:10 | |
| *** abaindur has quit IRC | 05:31 | |
| *** rcernin has quit IRC | 07:02 | |
| *** celebdor has joined #openstack-lbaas | 07:41 | |
| *** dayou_ has quit IRC | 07:42 | |
| *** yboaron_ has quit IRC | 08:07 | |
| *** yboaron_ has joined #openstack-lbaas | 08:08 | |
| *** ccamposr has joined #openstack-lbaas | 08:27 | |
| openstackgerrit | Reedip proposed openstack/octavia-tempest-plugin master: Add configuration support for skipping tests https://review.openstack.org/599393 | 08:41 |
| *** jiteka_ has joined #openstack-lbaas | 08:45 | |
| *** yboaron_ has quit IRC | 08:46 | |
| *** yboaron_ has joined #openstack-lbaas | 08:46 | |
| *** jiteka_ has quit IRC | 08:50 | |
| *** ccamposr_ has joined #openstack-lbaas | 08:52 | |
| *** ccamposr has quit IRC | 08:54 | |
| *** ccamposr__ has joined #openstack-lbaas | 08:57 | |
| *** ccamposr_ has quit IRC | 08:59 | |
| *** salmankhan has joined #openstack-lbaas | 09:16 | |
| *** ccamposr__ has quit IRC | 09:19 | |
| *** ccamposr__ has joined #openstack-lbaas | 09:19 | |
| *** baffle has joined #openstack-lbaas | 09:28 | |
| *** tobias-urdin has joined #openstack-lbaas | 09:35 | |
| maciejjozefczyk | rm_work I can help to make it public | 09:39 |
| maciejjozefczyk | For OVH actual Octavia implementation is not that compatible, so using floating IP, keepalived but without ARP looks great. | 09:40 |
| baffle | I'm trying to get Octavia working in my installation; The haproxy amph now fails when trying to bring up eth1:0 inside the network namespace, due to it both adding the "default" route, and then trying to add 0.0.0.0/0 from host routes as well; ip ofcourse respons with "RTNETLINK answers: File exists Failed to bring up eth1:0." and then bringing up the amph fails. | 09:44 |
| baffle | Do I just bastardize the template and rebuild my image, or am I missing something else? | 09:45 |
| openstackgerrit | Yang JianFeng proposed openstack/octavia master: Add quota support to octavia's l7policy and l7rule https://review.openstack.org/590620 | 09:54 |
| openstackgerrit | Yang JianFeng proposed openstack/octavia master: Refactor 'check_quota_met' and 'decrement_quota' https://review.openstack.org/596665 | 09:58 |
| *** pcaruana has joined #openstack-lbaas | 10:04 | |
| sapd1 | johnsom: Could you review my patch? | 10:26 |
| rm_work | maciejjozefczyk: oh are you basically using the L3 driver? | 10:38 |
| rm_work | maciejjozefczyk: https://review.openstack.org/#/c/435612/ | 10:38 |
| rm_work | (uses FLIPs on L3, keepalived with no ARP, calls back to the HM over the health message to trigger a neutron flip move :P | 10:39 |
| rm_work | ) | 10:39 |
| rm_work | I guess it might work for you out-of-the-box? | 10:39 |
| rm_work | if so, I could actually clean it up and see if we could get it mergable | 10:39 |
| maciejjozefczyk | rm_work: wow | 10:41 |
| maciejjozefczyk | rm_work: thats the thing! yes would like to use only floating_ip without this allowed_ip_pairs logic | 10:41 |
| rm_work | maciejjozefczyk: well, it kinda uses it... | 10:41 |
| rm_work | try out that whole chain | 10:41 |
| rm_work | from multi-az (does anti-affinity over AZs!), and evacuate (lets you maintenance an AZ at a time for patching), then FLIP driver | 10:42 |
| rm_work | that is what I run here | 10:42 |
| rm_work | actually this is my patch list on top of master, in order: http://paste.openstack.org/show/730406/ | 10:43 |
| rm_work | probably you don't need the first one | 10:43 |
| rm_work | or the second one i suppose, people kept asking me about adoption here tho :P | 10:44 |
| rm_work | anyway, if you have questions let me know, would love to see if that driver is useful for anyone else | 10:44 |
| rm_work | I can walk you through it | 10:44 |
| rm_work | some parts are a little dirty right now since i haven't had the motivation yet to clean it up and add good testing T_T | 10:45 |
| rm_work | but i can vouch that it works, been using it for over a year and a half here | 10:45 |
| maciejjozefczyk | rm_work: its great work! We're just started prodding Octavia and we went through the issues this should solve for us | 10:47 |
| maciejjozefczyk | rm_work: I'll dig into it to understand how it works | 10:47 |
| rm_work | yeah, give it a shot, and poke at me if you need anything | 10:47 |
| maciejjozefczyk | rm_work: thanks a lot! | 10:47 |
| rm_work | basically -- i have keepalived use a notify script, so when it takes over as master, it sends a custom signal to the agent | 10:47 |
| rm_work | the agent sends a special broadcast type of health message over the normal health monitoring channels | 10:48 |
| rm_work | the HMs detect that, and one of them takes the "failover" task, which moves the FLIP and refreshes the other (dead) amp | 10:48 |
| rm_work | I have a tempest test for the failover it does here: https://review.openstack.org/#/c/501559/ | 10:48 |
| maciejjozefczyk | rm_work: that will perfectly work with our network solution | 10:49 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Failover test https://review.openstack.org/501559 | 10:49 |
| rm_work | dumb pep8 bug lol | 10:50 |
| rm_work | maciejjozefczyk: yeah i hope you can give me some feedback on how it works for you, and please dont hesitate to hit me up if you have problems/questions | 10:50 |
| rm_work | important notes: "ha_port_id" in the amp table is overloaded to be the FLIP id | 10:50 |
| rm_work | and there's a couple config bits you should probably set | 10:51 |
| rm_work | for multi-az, just list all your available AZs | 10:52 |
| rm_work | (I assume you have multiple?) | 10:52 |
| rm_work | have you patched nova to accept multiple --availability-zone ? | 10:52 |
| rm_work | if so there's a setting you should set as well, though that's less common | 10:52 |
| rm_work | sorry, really excited to find someone else stuck in this situation :P | 10:52 |
| maciejjozefczyk | rm_work: we're not using AZs unfortunately | 10:55 |
| maciejjozefczyk | :( | 10:55 |
| rm_work | k, then you can ignore that bit | 10:55 |
| rm_work | actually i guess probably you can just cherry-pick the FLIP driver patch without it ... maybe | 10:55 |
| rm_work | i don't remember if i had to do anything to make it specifically work on top of the AZ patch | 10:56 |
| maciejjozefczyk | ok | 10:56 |
| rm_work | ah, in [networking] in your config, you'll want: | 11:08 |
| rm_work | `allow_vip_subnet_id = False` and `allow_vip_port_id = False`, and you want to set `valid_vip_networks = <a list of your flip networks>` | 11:08 |
| rm_work | finally got around to taking a look at the kolla-ansible octavia config <_< https://review.openstack.org/604043 | 11:13 |
| *** pcaruana has quit IRC | 11:15 | |
| *** pcaruana has joined #openstack-lbaas | 11:20 | |
| *** pcaruana has quit IRC | 11:32 | |
| *** pcaruana has joined #openstack-lbaas | 11:39 | |
| maciejjozefczyk | rm_work: good, what about openstack client? the vip_* field is mandatory while creating lb, isn't it? | 11:50 |
| *** pcaruana has quit IRC | 11:50 | |
| maciejjozefczyk | aah, ok | 11:50 |
| *** salmankhan1 has joined #openstack-lbaas | 12:30 | |
| *** salmankhan has quit IRC | 12:33 | |
| *** yboaron_ has quit IRC | 12:34 | |
| *** salmankhan1 has quit IRC | 12:35 | |
| *** yboaron_ has joined #openstack-lbaas | 12:35 | |
| *** yamamoto has quit IRC | 12:42 | |
| *** yamamoto has joined #openstack-lbaas | 12:42 | |
| *** salmankhan has joined #openstack-lbaas | 12:51 | |
| baffle | maciejjozefczyk: Wouldn't it work with "--vip-network-id <flip network id>"? | 13:28 |
| *** yamamoto has quit IRC | 13:29 | |
| *** yboaron_ has quit IRC | 13:38 | |
| *** ccamposr__ has quit IRC | 14:03 | |
| johnsom | We should note that FLIP failover is much slower than the native Act/Stdby failover | 14:03 |
| *** yboaron_ has joined #openstack-lbaas | 14:20 | |
| xgerman_ | did we ever time it in comparison to SINGLE ? | 14:34 |
| johnsom | Yeah, would be interesting to compare | 14:34 |
| xgerman_ | because with the spare pools it shouldn’t be much slower (once detected) | 14:36 |
| xgerman_ | so all we might need to do is beef up detetcion speed | 14:37 |
| *** KeithMnemonic has joined #openstack-lbaas | 14:40 | |
| *** yamamoto has joined #openstack-lbaas | 14:48 | |
| *** Swami has joined #openstack-lbaas | 14:57 | |
| *** yboaron_ has quit IRC | 14:59 | |
| *** yamamoto has quit IRC | 15:03 | |
| *** luis5tb has joined #openstack-lbaas | 15:09 | |
| luis5tb | ping johnsom (I'm ltomasbo) | 15:11 |
| johnsom | Hello | 15:12 |
| luis5tb | johnsom, I have two things to comment with you | 15:12 |
| johnsom | Ok | 15:12 |
| luis5tb | johnsom, one is if you discuss about the patch I mentioned the other day in the weekly irc meeting (sorry I finally could not attend it) | 15:12 |
| luis5tb | johnsom, and second, is that I'm seeing some weird behaviour in octavia (db/repositories/create_pool_on_load_balancer) | 15:13 |
| luis5tb | johnsom, seems the transaction, sometimes, is not completed and the last bits are not executed, leading to inconsistencies, and breaking for instance ovn-octavia driver support | 15:13 |
| johnsom | Yes I had it on the agenda and we did discuss it, but the outcome was no one on the team had time to do the research we discussed at the PTG, so we postponed for another week to give more time for folks to research. | 15:14 |
| luis5tb | johnsom, you mean changing the SG on the VIP? | 15:14 |
| johnsom | Correct | 15:14 |
| luis5tb | johnsom, I actually research that option, and it is not working | 15:14 |
| luis5tb | johnsom, at least it does not work with ml2/ovs driver | 15:14 |
| luis5tb | johnsom, as security groups that apply are the ones on the amphora port, not the one on the VIP | 15:15 |
| johnsom | The request is people research how SGs work in neutron and a few ideas folks have on how to stack the SGs. | 15:15 |
| johnsom | FYI, you can always read the meeting transcripts here: https://wiki.openstack.org/wiki/Octavia/Meeting_Minutes#2018-09-19_Weekly_meeting: | 15:15 |
| luis5tb | johnsom, oohh, true! thanks! | 15:16 |
| johnsom | looking at "create_pool_on_load_balancer" now | 15:17 |
| luis5tb | johnsom, ok, given what I discuss with the neutron folks, if they enforce VIP SG, it will not be through allow address pair but a new feature being merged... | 15:17 |
| johnsom | Yeah, we discussed other options. Just many of us have not looked at the neutron SG code in a long time | 15:18 |
| johnsom | So, what part of create_pool_on_load_balancer isn't getting run? | 15:18 |
| luis5tb | johnsom, I'm not getting the listener updated | 15:19 |
| johnsom | And you are doing a direct POST to create your pool? or single-call-create? | 15:21 |
| luis5tb | well, I'm using kuryr to create the lbaas, but it is going throug the POST method at pool.py | 15:23 |
| luis5tb | johnsom, ^ | 15:24 |
| johnsom | ok, so not single-call-create path. Still looking/testing | 15:24 |
| luis5tb | johnsom, probably is happening for the amphora driver too, but in that case the fact that listener_id is not there is not breaking the support I guess... | 15:25 |
| *** celebdor has quit IRC | 15:25 | |
| johnsom | So I just created a pool and the default_pool_id got filled in as expected... If that was broken I would expect a huge number of our tests to fail as it is the common code path. | 15:26 |
| johnsom | Do you have the json being passed to the API? | 15:27 |
| luis5tb | johnsom, it is not happening all the time | 15:27 |
| luis5tb | I'm creating lb -> listener -> pool -> member | 15:28 |
| johnsom | It is a valid case that you create pools that are not bound to listeners. Pools can be created, correctly, that are only bound to the load balancer. These can then be used for L7 policies. | 15:28 |
| luis5tb | johnsom, yes, but in this case, I passed the listener_id, and it is a problem that it is dropping it | 15:28 |
| johnsom | Yeah, that is what I do as well, I have a test script I use to create LBs. | 15:29 |
| luis5tb | johnsom, https://github.com/openstack/octavia/blob/master/octavia/api/v2/controllers/pool.py#L244-L245 | 15:30 |
| luis5tb | johnsom, there, db_pool is not setting listener_id, even if it is passed there | 15:30 |
| luis5tb | (sometimes...) | 15:30 |
| luis5tb | johnsom, I added some logs, and I see that the transactions seems to complete... | 15:32 |
| luis5tb | still, missing the listener id... | 15:33 |
| johnsom | Yeah, I'm not immediately seeing where there could be an issue. Can you add a few log points for me? | 15:33 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/api/v2/controllers/pool.py#L185 log the contents of the pool object. | 15:34 |
| luis5tb | johnsom, I have a problem doing that... including LOG.debug in that function, makes it work... | 15:34 |
| luis5tb | johnsom, but, I did print the listener that is obtained on that funtion on line 234 | 15:35 |
| luis5tb | so, listener id is there for sure... | 15:35 |
| johnsom | Also here: https://github.com/openstack/octavia/blob/master/octavia/api/v2/controllers/pool.py#L233 log the pool_dict | 15:35 |
| luis5tb | johnsom, I printed listener ID right after that line 234 | 15:35 |
| luis5tb | and it has the listner id value! | 15:36 |
| johnsom | Ok, so on the run you had problems with, listener_id was there???? | 15:36 |
| luis5tb | johnsom, yes | 15:36 |
| luis5tb | johnsom, but it is not on db_pool after executing line 244 | 15:37 |
| johnsom | Well, wait, I thought you said the listener was not getting it's "default_pool_id" updated. That would not be in db_pool as that is a listener table update, not the returned pool object | 15:40 |
| luis5tb | johnsom, seems this is not getting the updated object sometimes: https://github.com/openstack/octavia/blob/master/octavia/db/repositories.py#L225 | 15:42 |
| luis5tb | johnsom, I meant that db_pool is not getting the listener_id set and thus not passing it to the driver | 15:43 |
| luis5tb | johnsom, well, maybe it is getting set, but not returned in the get at https://github.com/openstack/octavia/blob/master/octavia/db/repositories.py#L225 | 15:43 |
| johnsom | But the pool object returned there does not have a listener_id parameter | 15:44 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/db/models.py#L262 | 15:45 |
| luis5tb | johnsom, but I saw that provider_pool sometimes include the listener id, and other times it does not... | 15:46 |
| luis5tb | johnsom, and actually, it tried to set it at https://github.com/openstack/octavia/blob/master/octavia/api/drivers/utils.py#L242 | 15:46 |
| johnsom | Ok, so this if different than the "default_pool_id" in the listener DB table... | 15:47 |
| luis5tb | johnsom, and, if I do: db_pool.listener_d = listener_id, things work... | 15:48 |
| luis5tb | (on the pool.py before the db_pool_to_provider_pool | 15:49 |
| luis5tb | johnsom, I see listeners is a property on pool model... | 15:51 |
| johnsom | What DB are you using? | 15:51 |
| johnsom | Yes, listeners is on the returned pool object | 15:52 |
| luis5tb | mariadb | 15:53 |
| luis5tb | johnsom, ^^ | 15:53 |
| johnsom | Ok. | 15:53 |
| johnsom | Yeah, I would have to have a test case to really run this to ground. The only thing I can think is sqlalchemy isn't handling the sub-transaction reliably for some reason. You could try a "session.flush()" before this line: https://github.com/openstack/octavia/blob/master/octavia/db/repositories.py#L225 | 15:54 |
| luis5tb | johnsom, ok, I'll try | 15:55 |
| luis5tb | if not that, the listener property is not getting the value... | 15:55 |
| luis5tb | btw, the listener.update there | 15:56 |
| luis5tb | ahh, no, sorry, that is listner table... | 15:56 |
| luis5tb | going to try the flush | 15:56 |
| johnsom | Yeah, I just haven't seen this and would really just have to walk that code path watching the listener id as I go to narrow it down | 15:56 |
| luis5tb | ok... thanks very much for the help! | 15:58 |
| luis5tb | johnsom, ^^ | 15:58 |
| johnsom | Sure, good luck | 15:58 |
| luis5tb | johnsom, I was mostly asking in case it was a known issue (and to ask about the other patch...) | 15:58 |
| luis5tb | johnsom, I'll let you know if flush helps... | 15:58 |
| johnsom | No open story for that as far as I remember | 15:59 |
| *** luis5tb has quit IRC | 16:03 | |
| *** luis5tb has joined #openstack-lbaas | 16:06 | |
| luis5tb | johnsom, ok, flush is not helping either... | 16:06 |
| luis5tb | johnsom, seems that moving that pool.get inside the subtransaction may help... | 16:15 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/octavia master: Ensure pool.get obtains listeners information https://review.openstack.org/604152 | 16:23 |
| luis5tb | johnsom, ^^ this fixes it... | 16:24 |
| openstackgerrit | Merged openstack/neutron-lbaas master: Fix memory leak in the haproxy provider driver https://review.openstack.org/603460 | 16:26 |
| *** Swami has quit IRC | 16:36 | |
| *** luis5tb has quit IRC | 16:46 | |
| *** salmankhan has quit IRC | 17:01 | |
| *** ramishra has quit IRC | 17:02 | |
| *** yamamoto has joined #openstack-lbaas | 17:03 | |
| *** Swami has joined #openstack-lbaas | 17:05 | |
| *** yamamoto has quit IRC | 17:20 | |
| *** eandersson has quit IRC | 17:34 | |
| *** jiteka has joined #openstack-lbaas | 17:37 | |
| *** eandersson has joined #openstack-lbaas | 17:39 | |
| *** yamamoto has joined #openstack-lbaas | 17:50 | |
| *** yamamoto has quit IRC | 17:56 | |
| *** salmankhan has joined #openstack-lbaas | 18:42 | |
| *** abaindur has joined #openstack-lbaas | 18:46 | |
| *** salmankhan has quit IRC | 18:47 | |
| johnsom | FYI, I have uploaded my "stresshm" tool I used to stress test the health manager. It can also be used to populate the database with "ghost" load balancers. | 19:15 |
| johnsom | https://github.com/johnsom/stresshm | 19:15 |
| johnsom | Same disclaimers, it was slapped together.... | 19:15 |
| *** yamamoto has joined #openstack-lbaas | 19:58 | |
| *** yamamoto has quit IRC | 20:06 | |
| *** abaindur has quit IRC | 20:32 | |
| *** abaindur has joined #openstack-lbaas | 20:33 | |
| *** abaindur has quit IRC | 20:35 | |
| *** abaindur has joined #openstack-lbaas | 20:35 | |
| rm_work | that sounds yiddish | 21:20 |
| johnsom | Boy, bionic wants to run home to python2.7 fairly often.... sigh | 21:30 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Refactor the AAP driver to not depend on nova https://review.openstack.org/604226 | 21:33 |
| xgerman_ | Boom! | 21:34 |
| johnsom | Cool! | 21:35 |
| johnsom | xgerman_ Reviewed | 21:45 |
| xgerman_ | Thanks… will fix l8ter today. Kids school’s out soon-ish | 21:46 |
| *** yamamoto has joined #openstack-lbaas | 22:08 | |
| *** yamamoto has quit IRC | 22:24 | |
| openstackgerrit | Merged openstack/octavia master: Updates README-Vagrant.md to use OSC commands https://review.openstack.org/603055 | 22:27 |
| rm_work | johnsom: just deployed to prod | 22:47 |
| rm_work | the return times look A++ | 22:47 |
| johnsom | Excellent | 22:47 |
| *** rcernin has joined #openstack-lbaas | 22:53 | |
| rm_work | yep very nice | 22:55 |
| rm_work | unfortunately I can't +2 it any harder | 22:55 |
| rm_work | i could have our deploy-bot +1 it i guess | 22:55 |
| johnsom | Ha | 22:56 |
| johnsom | rm_work Are you running the HM fix too? https://review.openstack.org/#/c/600332/ | 22:56 |
| rm_work | ugh i will be sad if i forgot to pull that in | 22:57 |
| rm_work | let me check | 22:57 |
| rm_work | yes | 22:57 |
| rm_work | i have both | 22:57 |
| rm_work | was worried for a sec | 22:58 |
| johnsom | You could +2 that one too... grin | 22:58 |
| rm_work | johnsom: http://paste.openstack.org/show/730490/ | 23:00 |
| johnsom | I am futzing with DIB on bionic. I am able to reproduce my issue I saw on the bionic nodepool instances | 23:00 |
| johnsom | rm_work is that the "before" | 23:01 |
| johnsom | ? | 23:01 |
| rm_work | no | 23:01 |
| rm_work | that's now | 23:01 |
| rm_work | before it was creeping up to the 0.4s | 23:01 |
| johnsom | Wow, a whole magnitude slower than my workstation | 23:01 |
| rm_work | well | 23:01 |
| rm_work | the DB isn't on the same machine | 23:01 |
| rm_work | sooooo | 23:01 |
| rm_work | yes | 23:01 |
| rm_work | lol | 23:01 |
| rm_work | there's some minimal wire time involved :P | 23:01 |
| rm_work | each one is doing ~6/s | 23:02 |
| rm_work | 360 amps / 10s heartbeats / 6 HMs | 23:02 |
| rm_work | math works out lovely right now, heh | 23:03 |
| rm_work | are the gates still f'd? | 23:06 |
| rm_work | looks like it? | 23:06 |
| johnsom | Yeah | 23:06 |
| *** andreykurilin has quit IRC | 23:34 | |
| *** andreykurilin has joined #openstack-lbaas | 23:35 | |
| *** rcernin has quit IRC | 23:36 | |
| *** rcernin has joined #openstack-lbaas | 23:36 | |
| *** Swami has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!