| openstackgerrit | min wang proposed openstack/octavia: Implements: blueprint anti-affinity server group https://review.openstack.org/272344 | 00:05 |
|---|---|---|
| *** Purandar has joined #openstack-lbaas | 00:13 | |
| *** piet has joined #openstack-lbaas | 00:14 | |
| *** armax has quit IRC | 00:16 | |
| openstackgerrit | German Eichberger proposed openstack/octavia: [WIP] Cascading LB delete https://review.openstack.org/282587 | 00:21 |
| openstackgerrit | Stephen Balukoff proposed openstack/neutron-lbaas: L7 capability extension implementation for lbaas v2 https://review.openstack.org/148232 | 00:24 |
| sbalukoff | ^^^ Contains a minor change to policy ordering code (ie. so that Evgeny's desire to number things with 1, as described in all the CLI stuff and API stuff, actually works that way.) | 00:25 |
| sbalukoff | I don't have any more outstanding changes for neutron-lbaas planned unless I find another bug while CLI testing. | 00:25 |
| sbalukoff | I want to do a little more CLI testing to make sure I've nailed the obvious bugs that I can find before I upload over Evgeny's patch there. | 00:26 |
| johnsom | sbalukoff So I am good to do some check outs and be back in testing business? | 00:26 |
| sbalukoff | johnsom: Except for the L7 CLI updates, yes. | 00:26 |
| sbalukoff | johnsom: Want me to commit what I have now, as far as the L7 CLI fixes, so that you can test and look for more bugs in parallel? | 00:27 |
| johnsom | Yes, I'm not going to wait for CLI | 00:28 |
| sbalukoff | johnsom: Ok, give me a minute and I'll get the CLI fixes (such as they are right now) committed... | 00:29 |
| sbalukoff | <openstackgerrit> Stephen Balukoff proposed openstack/python-neutronclient: L7 capability implementation for lbaas v2 https://review.openstack.org/217276 | 00:34 |
| *** crc32 has quit IRC | 00:34 | |
| sbalukoff | ^^^ fix for the pool not being found problem in the CLI | 00:34 |
| rm_work | kk | 00:34 |
| sbalukoff | There may be other CLI bugs; Looking into them now. | 00:34 |
| rm_work | i'm out for a bit | 00:34 |
| rm_work | will see if L7 is merged when I get back ;P | 00:35 |
| openstackgerrit | Michael Johnson proposed openstack/octavia: Change HMAC compare to use constant_time_compare https://review.openstack.org/283330 | 00:41 |
| reedip | sbalukoff | 00:43 |
| reedip | sbalukoff:ping | 00:43 |
| sbalukoff | reedip: Pong. | 00:44 |
| sbalukoff | Haha! Ok-- so yeah, this L7 CLI still needs some work. Can't create rules with it right now. :P | 00:45 |
| reedip | sbalukoff: was checking the patch for L7 CLI ( evidently it is already being discussed here :) ) | 00:45 |
| sbalukoff | reedip: Yep. Still needs work on that one before it'll be ready. | 00:45 |
| reedip | sbalukoff: one query, is --action a Necessary Parameter for UpdateL7Policy ? | 00:45 |
| sbalukoff | reedip: Not necessarily. If the action isn't changing, it's not necessary. | 00:46 |
| *** Purandar has quit IRC | 00:46 | |
| reedip | sbalukoff : Ok, then that would be an AI for the next patch :) | 00:46 |
| sbalukoff | (Or rather, it shouldn't be.) | 00:46 |
| sbalukoff | AI/ | 00:46 |
| sbalukoff | ? | 00:46 |
| reedip | sbalukoff: Action Item :D | 00:46 |
| reedip | sbalukoff: will comment on the patch itself... | 00:46 |
| sbalukoff | reedip: Ok, so you're saying the CLI forces you to specify action? | 00:47 |
| sbalukoff | Ok. | 00:47 |
| reedip | sbalukoff: yep... the CLI uses _add_common_parser | 00:47 |
| sbalukoff | Oh, haha! | 00:47 |
| reedip | sbalukoff: and the function has required=True for --action | 00:47 |
| reedip | so you need to pass another attribute in update: is_create =False | 00:48 |
| reedip | so that you can differentiate it in the Update and Create process | 00:48 |
| sbalukoff | reedip: Will do. Please keep in mind that I'm fixing someone else's code here. XD | 00:48 |
| reedip | sbalukoff: Yep, been there , done that .... :) | 00:48 |
| reedip | will comment on the patch | 00:48 |
| sbalukoff | In fact, I only *started* look at this CLI yesterday. I've been concentrating almost entirely on getting Octavia and the neutron-lbaas L7 stuff right. | 00:49 |
| openstackgerrit | min wang proposed openstack/octavia: Implements: blueprint anti-affinity server group https://review.openstack.org/272344 | 00:49 |
| sbalukoff | Who knew people would want to test out the L7 functionality from end to end, eh? ;) | 00:49 |
| *** ducttape_ has joined #openstack-lbaas | 00:50 | |
| johnsom | Grin | 00:51 |
| *** manishg has joined #openstack-lbaas | 00:51 | |
| johnsom | It's all off WIP, so.... | 00:52 |
| sbalukoff | I'm actually pretty happy with the state of the neutron-lbaas L7 patch, as well as the shared-pools CLI update. | 00:53 |
| sbalukoff | But the l7 CLI update is fairly broken at the moment. | 00:53 |
| johnsom | Yeah, things look really close. | 00:53 |
| *** jwarendt has quit IRC | 00:53 | |
| sbalukoff | To be fair, though Evgeny hadn't gotten many comments on it since he first started work on it in August, nor did he have a functional back-end to use with it. | 00:54 |
| *** manishg has quit IRC | 00:54 | |
| sbalukoff | So, it's not that surprising to me that we're noticing assumptions he made months ago with how this is supposed to work that aren't being borne out by reality at the moment. | 00:54 |
| *** manishg has joined #openstack-lbaas | 00:55 | |
| sbalukoff | (Generally, I'm pretty happy with the code Evgeny has written.) | 00:55 |
| johnsom | Yeah, no worries. | 00:55 |
| johnsom | So, if I create a policy (which works now) should that push anything to the haproxy.cfg? | 00:55 |
| johnsom | Or is it only when the rule comes in? | 00:56 |
| sbalukoff | johnsom: No, you need at least one rule on that policy. | 00:56 |
| sbalukoff | So... well, give me a minute on that. XD | 00:56 |
| sbalukoff | Though, you *should* see a line on your o-cw console, as well as in the octavia database, which show the policy being created. | 00:57 |
| johnsom | Flow 'octavia-create-l7policy-flow' transitioned into state 'SUCCESS' | 00:58 |
| sbalukoff | Woot! | 01:00 |
| johnsom | So I guess this is what you are working on: Unable to find 'rule' in request body | 01:00 |
| *** ajmiller has quit IRC | 01:05 | |
| sbalukoff | johnsom: Yep. | 01:06 |
| reedip | sbalukoff: commented... | 01:07 |
| sbalukoff | reedip: Thanks! | 01:08 |
| *** minwang2 has quit IRC | 01:19 | |
| *** manishg is now known as wfh_ | 01:21 | |
| *** wfh_ is now known as manishg_wfh | 01:21 | |
| *** Aish has quit IRC | 01:25 | |
| *** Aish has joined #openstack-lbaas | 01:27 | |
| *** Aish has left #openstack-lbaas | 01:27 | |
| *** ducttape_ has quit IRC | 01:31 | |
| *** paco20151113 has joined #openstack-lbaas | 01:39 | |
| *** yamamoto has quit IRC | 01:44 | |
| *** kevo has quit IRC | 01:45 | |
| *** pc_m has quit IRC | 01:51 | |
| *** pc_m has joined #openstack-lbaas | 01:54 | |
| *** ducttape_ has joined #openstack-lbaas | 01:54 | |
| johnsom | sbalukoff How is the CLI going? | 01:55 |
| sbalukoff | About to push up a new patch set. | 01:56 |
| johnsom | Ok, cool. After that, can you look at this? https://github.com/openstack/octavia/blob/master/octavia/api/v1/controllers/pool.py#L90 | 01:56 |
| johnsom | I think we have a shared pools bug | 01:56 |
| sbalukoff | Ok, new patch set is up, note that I don't have Reedip's fix in that yet. | 01:56 |
| sbalukoff | But at least you should be able to add rules now. :P | 01:57 |
| johnsom | ok | 01:57 |
| johnsom | Cool, will try | 01:57 |
| johnsom | What I am seeing is we are deleting a pool while a member is still in create. | 01:57 |
| sbalukoff | Oh? | 01:57 |
| sbalukoff | Ok, so things should be immutable at that point and they aren't, effectively? | 01:58 |
| johnsom | I think the status check during the pool delete is broken because it is going pool.listeners for fing the listeres and lb to check | 01:58 |
| *** Bjoern_ has joined #openstack-lbaas | 01:58 | |
| johnsom | Yeah, that is what I see. I can't tell if its related to the model issues (this is a gate run on master) or if the check is now broken | 01:59 |
| sbalukoff | Hmmm.. Ok. | 01:59 |
| sbalukoff | pool.listeners should work, in any case. | 01:59 |
| johnsom | This is the log if you really want to burn your brain: http://logs.openstack.org/55/283255/2/check/gate-neutron-lbaasv2-dsvm-scenario/3e4a28e/logs/screen-o-cw.txt.gz | 01:59 |
| sbalukoff | It should have the list of listeners that reference the pool via default_pool_id or via an l7policy that is in use. | 01:59 |
| johnsom | Ok, if that code looks good to you, and "should still work" then it might just be a model issue | 01:59 |
| sbalukoff | I'll look at the log, eh. | 02:00 |
| sbalukoff | Can you give me a timestamp or something that I should search for in there? | 02:00 |
| *** pc_m has quit IRC | 02:02 | |
| johnsom | 2016-02-23 00:34:28.896 is the member create finishing | 02:02 |
| johnsom | 2016-02-23 00:34:26.762 is the delete pool starting | 02:02 |
| johnsom | It's hard to track as the log doesn't have a thread id in it | 02:03 |
| sbalukoff | Aah. | 02:03 |
| sbalukoff | Ok. | 02:03 |
| *** ducttape_ has quit IRC | 02:04 | |
| sbalukoff | Oh, I think I understand the logic here: | 02:05 |
| sbalukoff | The member doesn't know that the pool it's on is being deleted because in the middle of that process, the link from the pool to the listener goes away... | 02:05 |
| *** pc_m has joined #openstack-lbaas | 02:06 | |
| sbalukoff | So... we need to check the pool status for member (and other subordinate object) operations. | 02:06 |
| johnsom | Well, member create started first | 02:06 |
| sbalukoff | Wait--- was this pool actually assigned to a listener yet? | 02:07 |
| johnsom | 2016-02-23 00:34:10.670 | 02:07 |
| johnsom | Yes | 02:07 |
| sbalukoff | Man, we should really add a thread ID to these debug logs... :P | 02:08 |
| sbalukoff | Hmmm.... | 02:08 |
| sbalukoff | So the pool was associated to a listener... | 02:08 |
| sbalukoff | Through the default_pool? | 02:09 |
| sbalukoff | Or some other means? | 02:09 |
| johnsom | You are telling me! We need to figure out how to do that | 02:09 |
| sbalukoff | (some other means being through an l7 policy) | 02:09 |
| johnsom | It is an old test, so however it was originally done. | 02:09 |
| johnsom | API log is here: http://logs.openstack.org/55/283255/2/check/gate-neutron-lbaasv2-dsvm-scenario/3e4a28e/logs/screen-o-api.txt.gz | 02:10 |
| johnsom | It gives a clear sequence of requests | 02:10 |
| sbalukoff | Old test should be using old URI? (i.e. the one that includes the listener_id?) | 02:11 |
| johnsom | Yes, I expect so. | 02:12 |
| sbalukoff | By the way, how / where was this generated in the first place? | 02:14 |
| johnsom | https://review.openstack.org/#/c/283255/ | 02:14 |
| sbalukoff | OK. | 02:15 |
| *** allan_h has quit IRC | 02:15 | |
| johnsom | FYI, barbican has broken our API gates | 02:16 |
| sbalukoff | Dammit! | 02:16 |
| sbalukoff | Argh. | 02:16 |
| sbalukoff | Ok, so there's no way this should have happened if the pool was associated with a load balancer (which is by definition, must be: | 02:17 |
| sbalukoff | 127.0.0.1 - - [23/Feb/2016 00:34:08] "POST /v1/loadbalancers/361de8f2-d524-4532-bad8-fab4b71a1199/pools/60043f71-f70e-4b44-92d5-16cbb7943554/members HTTP/1.1" 202 259 | 02:17 |
| sbalukoff | 127.0.0.1 - - [23/Feb/2016 00:34:08] "GET /v1/loadbalancers/361de8f2-d524-4532-bad8-fab4b71a1199 HTTP/1.1" 200 349 | 02:17 |
| sbalukoff | that is to say: Creating a member should not have completed in a sub-second interval like that. | 02:18 |
| sbalukoff | So let me look closer into that first log... | 02:18 |
| johnsom | Well, that API log doesn't say the member was created in under a second. | 02:20 |
| johnsom | We would need the output of that get | 02:20 |
| johnsom | Actually, we probably have that in the test output | 02:20 |
| sbalukoff | Looking at the size of the response, the output of the get was that the load_balancer was in an ACTIVE state. | 02:21 |
| sbalukoff | You can see that in the previous commands that do that same get on the loadbalancer. | 02:21 |
| *** armax has joined #openstack-lbaas | 02:21 | |
| *** piet has quit IRC | 02:23 | |
| sbalukoff | I think it's a bug here: https://github.com/openstack/octavia/blob/master/octavia/api/v1/controllers/member.py#L76 | 02:24 |
| sbalukoff | members don't have a listener_id, per se. Or at least the API command won't necessarily. | 02:24 |
| sbalukoff | members definitely have a pool_id (API command will) | 02:24 |
| sbalukoff | So we need to be looking to see if the pool has a listener that way. | 02:24 |
| sbalukoff | Anyway, yes, I think this is a shared_pools bug. | 02:25 |
| sbalukoff | Probably affects health monitors, too. | 02:25 |
| *** yamamoto has joined #openstack-lbaas | 02:25 | |
| sbalukoff | Do you want to file a bug report on this and assign it to me? (Or, I can just see if I can fix it in the next 30 minutes before I have to leave for a social engagement.) | 02:26 |
| johnsom | My head is hurting as I can't find the load balancer ID in the test logs. Something is really screwy with the gate logs | 02:29 |
| johnsom | sbalukoff I think you have a better grasp on the issue, can you open it? | 02:29 |
| sbalukoff | Sure! | 02:29 |
| sbalukoff | If I don't have time to fix it before I have to go, I'll get online late tonight and fix it there. As far as where to base it: Do you mind if I put it at the end of the L7 chain? (I want to try to not disturb that chain at all, eh.) | 02:30 |
| johnsom | That's ok | 02:31 |
| sbalukoff | Ok, bug report created and assigned to me... | 02:41 |
| sbalukoff | Let me see how quickly I can bang this out. :) | 02:42 |
| johnsom | Thanks | 02:42 |
| johnsom | I see a minor jinja issue in the rules. I thought they weren't working at all, but path is very explicit, so foobar doesn't match starts_with, but /foobar does | 02:42 |
| *** TrevorV has joined #openstack-lbaas | 02:43 | |
| sbalukoff | johnsom: Er... do you mean that haproxy won't run with 'foobar' there? | 02:44 |
| openstackgerrit | Franklin Naval proposed openstack/octavia: Octavia: Basic LoadBalancer Scenario Test https://review.openstack.org/172199 | 02:44 |
| johnsom | It runs, the rule doesn't fire | 02:45 |
| sbalukoff | Oh! | 02:45 |
| sbalukoff | That's expected behavior. | 02:45 |
| johnsom | Yeah. It's just picky | 02:45 |
| fnaval | that commit should be much better - please take a look folks when you have a chance | 02:45 |
| sbalukoff | Any actual path that a browser uses is going to start with / | 02:45 |
| fnaval | shaved off 7K lines | 02:45 |
| *** piet has joined #openstack-lbaas | 02:45 | |
| sbalukoff | We can change the validation for the path to look for a leading slash if you want-- though technically the behavior here isn't in error, per se. | 02:46 |
| fnaval | bbl | 02:46 |
| johnsom | No, let's not change it | 02:46 |
| sbalukoff | Ok. | 02:46 |
| johnsom | That is haproxy behavior | 02:47 |
| johnsom | sbalukoff https://review.openstack.org/#/c/278223/9 | 02:47 |
| sbalukoff | I was going to say: If my hypothesis about that member stuff is incorrect, then my next best guess would be SQLAlchemy caching. | 02:47 |
| johnsom | You could convince me to put this in as a bug and take the -1 off. It's just if the block above is indented this will fail | 02:47 |
| sbalukoff | TrevorV and I ran into some of that in the code he's writing. :P | 02:48 |
| sbalukoff | johnsom: Ok, that indentation is intentional. | 02:49 |
| sbalukoff | haproxy doesn't care, and it makes things slightly more human-readable. | 02:49 |
| sbalukoff | I'll add a comment to that effect. :) | 02:49 |
| johnsom | Are you positive it doesn't care???? | 02:49 |
| johnsom | I thought there were sub-blocks in this config | 02:49 |
| sbalukoff | johnsom: Yep. Try messing with the indentation manually on the aphora and restarting haproxy. | 02:50 |
| TrevorV | sbalukoff what happened? | 02:50 |
| sbalukoff | haproxy doesn't give a hoot about indentation, from what I can tell. | 02:50 |
| sbalukoff | It only cares about order. | 02:50 |
| johnsom | Yeah, I have done that. I'll look for the frontend line with sub-blocks. | 02:50 |
| sbalukoff | TrevorV: The bug that johnsom was troubleshooting was a pool deletion while a member was still being created. This is probably because the listener / LB status is probably not being set correctly by the member create, because of the way we're figuring out which listeners need updating in the api code. | 02:51 |
| TrevorV | sbalukoff johnsom so "more" on that from last week | 02:52 |
| TrevorV | Basically blogan and I sat down and troubleshot that same section of code I was/am writing for single-create, and have discovered that either SQLAlchemy has a bug in its "subtransactions" for "with" contexts, or it just doesn't work the way we expect it to and need to refactor our code more appropriately. | 02:53 |
| TrevorV | All I know is the issues sbalukoff and I were trying to identify were "solved" after we force-updated certain objects in the DB. | 02:54 |
| *** ajmiller has joined #openstack-lbaas | 02:54 | |
| openstackgerrit | Franklin Naval proposed openstack/octavia: Octavia: Basic LoadBalancer Scenario Test https://review.openstack.org/172199 | 02:54 |
| TrevorV | Honestly johnsom I want to take a much closer look at it all, but right now I feel like everyone is waiting on me for the single-create and I don't want to divert too far, you know? | 02:54 |
| johnsom | TrevorV Yeah, we have more runway on bugs than we do on features, so getting single create in is probably the priority | 02:55 |
| sbalukoff | +1 | 02:56 |
| TrevorV | johnsom is there a hard-deadline for it that I should be more aware of? | 02:56 |
| TrevorV | I can definitely burn more hours than I have been if that's the case. | 02:56 |
| johnsom | Feature complete deadline for Mitaka is Monday | 02:56 |
| TrevorV | No shit? | 02:57 |
| johnsom | you could go for an extension, but that is probably up to Doug | 02:57 |
| TrevorV | Alright, I'll burn some "midnight oil" I believe the phrase is. | 02:57 |
| sbalukoff | TrevorV: Good luck! That's a great feature and I hope it lands, eh! | 02:57 |
| johnsom | Yeah, sorry, I thought we covered that in the meetings. M3 cut is the 29th | 02:57 |
| TrevorV | johnsom you honestly probably did cover it, with me not really paying attention enough to realize I'm a part of that deadline. | 02:58 |
| TrevorV | My bad. | 02:58 |
| TrevorV | Now I'm more concerned about not having Neutron LBaaS side worked on at all... | 02:58 |
| TrevorV | Eek | 02:58 |
| sbalukoff | Ok, I've gotta run for now, folks. Please feel free to e-mail me or assign bugs to me as you find them in the L7 or shared pools stuff. I'll be back on really late tonight to try to bang out a fix for the one you found there, johnsom. | 02:58 |
| johnsom | Ok, thanks | 02:59 |
| TrevorV | kk l8r | 02:59 |
| *** sbalukoff has quit IRC | 03:05 | |
| *** Bjoern_ has quit IRC | 03:05 | |
| fnaval | hmmm how do I get Tempest to be an enabled service in the gate job? | 03:08 |
| fnaval | in settings should I change this to enabled? OCTAVIA_TEMPEST=${OCTAVIA_TEMPEST:-"disabled"} | 03:09 |
| johnsom | Nope, you would set that variable | 03:09 |
| johnsom | sbalukoff Fun for when you get back: https://review.openstack.org/#/c/278223/9 | 03:12 |
| johnsom | Multiple rules don't seem to work or scal | 03:12 |
| johnsom | scale | 03:12 |
| fnaval | johnsom: hmm. ok. is there a get_hook.sh that I can set to get tempest? | 03:13 |
| fnaval | ok, i'll research more about this. | 03:13 |
| *** yuanying_ has joined #openstack-lbaas | 03:16 | |
| *** yuanying has quit IRC | 03:19 | |
| *** yuanying_ has quit IRC | 03:20 | |
| fnaval | madhu_ak: I sent an email about the tempest plugin / gate jobs to you - please take a look. if you can clarify, it'd be most appreicated! thanks in advance | 03:20 |
| *** Bjoern_ has joined #openstack-lbaas | 03:42 | |
| *** TrevorV has quit IRC | 03:43 | |
| *** minwang2 has joined #openstack-lbaas | 03:58 | |
| *** neelashah has joined #openstack-lbaas | 03:58 | |
| *** johnsom_ has joined #openstack-lbaas | 04:03 | |
| *** links has joined #openstack-lbaas | 04:03 | |
| *** yuanying has joined #openstack-lbaas | 04:08 | |
| *** johnsom_ has quit IRC | 04:12 | |
| *** johnsom_ has joined #openstack-lbaas | 04:13 | |
| *** ducttape_ has joined #openstack-lbaas | 04:18 | |
| johnsom_ | Sorry fnaval I had to run to dinner. You would set that in the gate hook. We don't have that setup yet for Octavia. We should though. | 04:22 |
| *** neelashah has quit IRC | 04:23 | |
| *** kevo has joined #openstack-lbaas | 04:33 | |
| *** numans has joined #openstack-lbaas | 04:42 | |
| *** armax has quit IRC | 04:47 | |
| *** ducttape_ has quit IRC | 05:01 | |
| *** minwang2 has quit IRC | 05:04 | |
| *** minwang2 has joined #openstack-lbaas | 05:09 | |
| *** allan_h has joined #openstack-lbaas | 05:17 | |
| *** sbalukoff has joined #openstack-lbaas | 05:22 | |
| openstackgerrit | Franklin Naval proposed openstack/octavia: Octavia: Basic LoadBalancer Scenario Test https://review.openstack.org/172199 | 05:32 |
| sbalukoff | johnsom: Are you still around? | 05:33 |
| *** Aish has joined #openstack-lbaas | 05:36 | |
| *** woodster_ has quit IRC | 05:36 | |
| *** Aish has quit IRC | 05:36 | |
| *** ajmiller has quit IRC | 05:38 | |
| *** allan_h has quit IRC | 05:56 | |
| openstackgerrit | Franklin Naval proposed openstack/octavia: Octavia: Basic LoadBalancer Scenario Test https://review.openstack.org/172199 | 06:03 |
| *** TrevorV has joined #openstack-lbaas | 06:05 | |
| *** piet has quit IRC | 06:07 | |
| fnaval | sbalukoff: this might be a legit bug: https://bugs.launchpad.net/octavia/+bug/1548628 | 06:17 |
| openstack | Launchpad bug 1548628 in octavia "Config Params Conflict: test_build_agent_config unit test failure" [Undecided,New] | 06:17 |
| *** minwang2 has quit IRC | 06:17 | |
| fnaval | I kinda found it last week but hacked around it with all the copied tempest code, now it's hitting again with the imported code. I don't think Tempest config should be hacked to get around it. | 06:17 |
| *** amotoki has joined #openstack-lbaas | 06:19 | |
| fnaval | anyway, it would block any scenario tempest tests being run, since it hits in the sonar gate with tox -ecover | 06:20 |
| *** fnaval has quit IRC | 06:20 | |
| *** mariusv has quit IRC | 06:25 | |
| *** minwang2 has joined #openstack-lbaas | 06:27 | |
| *** pc_m has quit IRC | 06:30 | |
| *** sbalukoff has quit IRC | 06:32 | |
| *** mariusv has joined #openstack-lbaas | 06:32 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia: Change HMAC compare to use constant_time_compare https://review.openstack.org/283330 | 06:32 |
| *** mariusv is now known as Guest98039 | 06:32 | |
| openstackgerrit | Zhou Zhihong proposed openstack/neutron-lbaas: Add check for listener's admin_state_up status https://review.openstack.org/266816 | 06:33 |
| openstackgerrit | Michael Johnson proposed openstack/octavia: Change HMAC compare to use constant_time_compare https://review.openstack.org/283330 | 06:33 |
| johnsom_ | sbalukoff Yes, sorry, alternate nic | 06:34 |
| *** fnaval has joined #openstack-lbaas | 06:36 | |
| *** minwang2 has quit IRC | 06:46 | |
| *** TrevorV has quit IRC | 06:47 | |
| *** Guest98039 has quit IRC | 06:53 | |
| *** marius- has joined #openstack-lbaas | 06:53 | |
| *** marius- has quit IRC | 06:54 | |
| *** minwang2 has joined #openstack-lbaas | 07:08 | |
| *** armax has joined #openstack-lbaas | 07:09 | |
| *** minwang2 has quit IRC | 07:18 | |
| *** bana_k has joined #openstack-lbaas | 07:27 | |
| johnsom_ | Ok, calling it for the evening. | 07:36 |
| *** johnsom_ has quit IRC | 07:36 | |
| *** evgenyf has joined #openstack-lbaas | 07:38 | |
| *** Bjoern_ has quit IRC | 07:42 | |
| *** minwang2 has joined #openstack-lbaas | 07:44 | |
| *** bana_k has quit IRC | 07:54 | |
| *** kevo has quit IRC | 07:56 | |
| *** pcaruana has joined #openstack-lbaas | 07:58 | |
| *** pcaruana is now known as pcaruana|afk| | 08:08 | |
| *** pcaruana|afk| is now known as pcaruana | 08:08 | |
| *** Oku_OS has quit IRC | 08:13 | |
| *** mariusv has joined #openstack-lbaas | 08:20 | |
| *** pc_m has joined #openstack-lbaas | 08:23 | |
| *** reedip is now known as outofmemory | 08:25 | |
| *** outofmemory is now known as reedip | 08:41 | |
| *** jschwarz has joined #openstack-lbaas | 08:45 | |
| *** minwang2 has quit IRC | 08:48 | |
| *** sbalukoff has joined #openstack-lbaas | 09:03 | |
| sbalukoff | AUGH! WHY IS OUR GATE BROKEN NOW??? | 09:22 |
| *** paco20151113 has quit IRC | 09:41 | |
| bharathm | sbalukoff: It seems barbican devstack lib errors out | 09:49 |
| sbalukoff | It makes me want to cry. | 09:49 |
| bharathm | Given the patches we have under review, this is bad.. Barbican team may have to fix this.. | 09:56 |
| bharathm | I created a bug https://bugs.launchpad.net/barbican/+bug/1548722 | 10:08 |
| openstack | Launchpad bug 1548722 in Barbican "Gates failing due to broken devstack lib of Barbican" [Undecided,New] | 10:08 |
| *** Oku_OS has joined #openstack-lbaas | 10:52 | |
| rm_work | urk really | 10:53 |
| rm_work | barbican y u do dis | 10:53 |
| rm_work | 99% sure this is what broke the gate: https://review.openstack.org/#/c/281779/ | 11:04 |
| rm_work | submitted a fix: https://review.openstack.org/283509 | 11:08 |
| rm_work | we'll see | 11:08 |
| rm_work | also I'm pretty tired so maybe I'm crazy but why have we still not rebased the L7 patch on the pools patch for python-neutronclient!? | 11:13 |
| rm_work | screw it, i'm just going to do it | 11:13 |
| rm_work | can be undone in the morning easily enough if that isn't right | 11:13 |
| rm_work | sbalukoff: ^^ | 11:15 |
| sbalukoff | rm_work: That's fine. | 11:15 |
| rm_work | LOL you're still here too | 11:16 |
| sbalukoff | I'm too flustered right now to code and I'm having trouble sleeping... so I'm writing an e-mail with my thoughts. | 11:16 |
| rm_work | sbalukoff: i wish i'd been around earlier, could have gotten this gatefix for barbican sorted out | 11:16 |
| sbalukoff | Will probably try to get some sleep here in the next half hour or so. | 11:16 |
| rm_work | really only took like 10 minutes to track down | 11:16 |
| rm_work | heh well, good luck with the sleeping | 11:17 |
| rm_work | i am no stranger to sleep issues | 11:17 |
| rm_work | ... as one might surmise given i'm working right now | 11:17 |
| rm_work | AAUAUUGH | 11:19 |
| rm_work | and rackspace's ubuntu mirror is apparently broken (maybe mid-update?) so I can't stack | 11:19 |
| sbalukoff | Heh. | 11:21 |
| sbalukoff | I'm sorry, man. | 11:21 |
| rm_work | lol no worries, i think it's a sign that i should TRY to sleep now | 11:22 |
| rm_work | hopefully tomorrow means this is all fixed and everything is good | 11:22 |
| rm_work | and your stuff will all merge and there will be unicorns and rainbows | 11:22 |
| sbalukoff | I'm flustered by the number of -1's johnsom added a few hours ago... and almost all over missing unit tests. | 11:24 |
| rm_work | ah | 11:24 |
| rm_work | i wouldn't worry about it | 11:24 |
| rm_work | i saw all of those, and the responses | 11:25 |
| sbalukoff | I've written him an e-mail about it; I'm hoping he'll agree to my compromise. | 11:25 |
| rm_work | and i noticed why i didn't see any issues -- i never tested with more than one rule, lol | 11:25 |
| rm_work | i was like "one rule created? sweet, working" | 11:25 |
| rm_work | though it should still be fine | 11:25 |
| rm_work | no REAL issues AFAICT | 11:25 |
| sbalukoff | rm_work: Multiple rules work. | 11:25 |
| rm_work | yeah | 11:25 |
| rm_work | just his scaling concern, but... eh | 11:26 |
| sbalukoff | I've tested it-- I think johnsom misunderstands how they're supposed to work. | 11:26 |
| rm_work | so 100% of what his comments are essentially boil down to what i'd be willing to just call "known bugs / limitations" | 11:26 |
| rm_work | and we can work on fixes later | 11:26 |
| sbalukoff | Anyway, I hope my comment explaining things was clear enough. :/ | 11:26 |
| rm_work | yeah | 11:26 |
| rm_work | i understood | 11:26 |
| rm_work | merge feature -> fix bugs | 11:26 |
| rm_work | I don't think it'll take much convincing to switch those to +2s | 11:27 |
| sbalukoff | Ok. | 11:27 |
| sbalukoff | I'm seriously shaken at this point though... | 11:27 |
| rm_work | i am hoping we can do the same with trevor's patches for single-create | 11:28 |
| rm_work | sbalukoff: i wouldn't worry about it <_< | 11:28 |
| rm_work | we'll get it sorted | 11:28 |
| rm_work | I think it'll merge as-is and we can go over some rough bits later | 11:28 |
| sbalukoff | If I have to write all those unit tests and then try to convince people to test all 6000+ lines of code in that chain... there's no way Octavia's L7 is getting in by Monday, let alone the neutron-lbaas L7. | 11:28 |
| rm_work | is M3 a final or a feature-freeze? | 11:29 |
| sbalukoff | feature freeze. | 11:29 |
| rm_work | I assume we have some time for bug-fixes | 11:29 |
| rm_work | yeah so whatever | 11:29 |
| rm_work | we merge and deal with it after M3 | 11:29 |
| sbalukoff | we have a few weeks afterward for bug fixes. | 11:29 |
| rm_work | the feature is "complete" as far as I am concerned | 11:29 |
| rm_work | sometimes you just have to move forward | 11:30 |
| rm_work | again, I don't know what kind of email you're writing, but i seriously don't expect a fight in the morning to merge (some of them already have +A, right?) | 11:31 |
| rm_work | stupid gate failures.... | 11:31 |
| sbalukoff | yeah, several do have +A's. | 11:31 |
| rm_work | yeah i think my gatefix for barbican worked | 11:32 |
| rm_work | seems my stack is setting up properly now | 11:32 |
| rm_work | ... that may have been partially my fault | 11:32 |
| rm_work | since i wrote a lot of their current devstack plugin code <_< | 11:32 |
| rm_work | but I am pretty sure there was no avoiding this | 11:33 |
| rm_work | would have been nice to see some warning before that patch landed though | 11:33 |
| rm_work | even a cursory grep across the git repos of major projects | 11:33 |
| rm_work | for usage of TENANT vars | 11:33 |
| rm_work | ... i wonder if it affects octavia or neutron | 11:34 |
| rm_work | looks like it might possibly <_< | 11:35 |
| sbalukoff | Ouch. | 11:36 |
| rm_work | octavia/devstack/settings:OCTAVIA_TENANT_ID=${OCTAVIA_TENANT_ID:-${ADMIN_TENANT_ID}} | 11:36 |
| sbalukoff | Well... growing pains. | 11:36 |
| rm_work | $ADMIN_TENANT_ID most likely was set by devstack and is blank now | 11:36 |
| sbalukoff | And this is the week for features to land which probably contain a few annoying bugs still.. | 11:36 |
| rm_work | will check it out | 11:36 |
| rm_work | hmm looks like maybe it is fine -- not sure where $ADMIN_TENANT_ID comes from | 11:38 |
| rm_work | devstack such magic smoke and mirrors | 11:38 |
| sbalukoff | Ok, I'm going to veg. in front of the TV for a bit then try to sleep. Will probably be back online around 11:00am PDT. | 11:38 |
| rm_work | lol 11am CDT for me | 11:39 |
| sbalukoff | Good luck getting that working, rm_work! | 11:39 |
| rm_work | later | 11:39 |
| *** yamamoto has quit IRC | 11:40 | |
| *** paco20151113 has joined #openstack-lbaas | 11:45 | |
| rm_work | man, i wish neutron's client supported --poll | 11:46 |
| rm_work | hmm interesting -- noticing that it is pulling the certs from barbican on EVERY update, even stuff like a new member in a pool | 11:53 |
| rm_work | that shouldn't happen | 11:53 |
| rm_work | filed a bug: https://bugs.launchpad.net/octavia/+bug/1548782 | 11:57 |
| openstack | Launchpad bug 1548782 in octavia "Incorrectly fetching Barbican certificate data on every action" [Undecided,New] | 11:57 |
| rm_work | may not have searched well before doing that... hopefully that is not a duplicate | 11:57 |
| rm_work | and with that, I'm out for the night >_< | 11:57 |
| *** kobis has joined #openstack-lbaas | 12:04 | |
| *** kobis has quit IRC | 12:04 | |
| *** prabampm has quit IRC | 12:04 | |
| *** prabampm1 has joined #openstack-lbaas | 12:04 | |
| *** kobis has joined #openstack-lbaas | 12:04 | |
| rm_work | for anyone who wants to do L7 testing, I'll just leave this here: https://raw.githubusercontent.com/rm-you/devstack_deploy/1c00e14ca8275de47ec67896e6a2d520a61cf890/stackme.sh | 12:05 |
| rm_work | ^^ includes every relevant patchset | 12:05 |
| rm_work | (barbican gatefix, octavia l7 changes, neutron-lbaas l7 changes, python-neutronclient changes) | 12:05 |
| *** links has quit IRC | 12:38 | |
| *** yamamoto has joined #openstack-lbaas | 12:39 | |
| *** numans has quit IRC | 12:55 | |
| *** yamamoto has quit IRC | 12:58 | |
| *** localloop127 has joined #openstack-lbaas | 12:59 | |
| *** ducttape_ has joined #openstack-lbaas | 13:03 | |
| *** _ducttape_ has joined #openstack-lbaas | 13:12 | |
| *** ducttape_ has quit IRC | 13:14 | |
| *** paco20151113 has quit IRC | 13:15 | |
| *** yamamoto has joined #openstack-lbaas | 13:20 | |
| *** links has joined #openstack-lbaas | 13:23 | |
| *** _ducttape_ has quit IRC | 13:23 | |
| *** numans has joined #openstack-lbaas | 13:36 | |
| *** armax has quit IRC | 14:00 | |
| *** pcaruana has quit IRC | 14:02 | |
| *** prabampm1 has quit IRC | 14:04 | |
| *** piet has joined #openstack-lbaas | 14:15 | |
| *** neelashah has joined #openstack-lbaas | 14:17 | |
| *** Bjoern_ has joined #openstack-lbaas | 14:20 | |
| *** Bjoern_ is now known as Bjoern_zZzZzZzZ | 14:20 | |
| *** manishg_wfh has quit IRC | 14:21 | |
| *** nmagnezi has joined #openstack-lbaas | 14:23 | |
| *** links has quit IRC | 14:30 | |
| *** armax has joined #openstack-lbaas | 14:30 | |
| *** links has joined #openstack-lbaas | 14:31 | |
| *** Bjoern_zZzZzZzZ is now known as Bjoern_ | 14:33 | |
| *** ducttape_ has joined #openstack-lbaas | 14:55 | |
| *** neelashah1 has joined #openstack-lbaas | 15:05 | |
| *** links has quit IRC | 15:07 | |
| *** neelashah has quit IRC | 15:07 | |
| *** TrevorV has joined #openstack-lbaas | 15:07 | |
| *** manishg_wfh has joined #openstack-lbaas | 15:11 | |
| *** rtheis has joined #openstack-lbaas | 15:11 | |
| *** ajmiller has joined #openstack-lbaas | 15:18 | |
| *** pcaruana has joined #openstack-lbaas | 15:18 | |
| *** evgenyf has quit IRC | 15:20 | |
| *** pcaruana has quit IRC | 15:20 | |
| *** pcaruana has joined #openstack-lbaas | 15:21 | |
| *** pcaruana|afk| has joined #openstack-lbaas | 15:21 | |
| *** rtheis has quit IRC | 15:23 | |
| *** rtheis has joined #openstack-lbaas | 15:24 | |
| *** rtheis has quit IRC | 15:28 | |
| *** pcaruana|afk| has quit IRC | 15:29 | |
| *** rtheis has joined #openstack-lbaas | 15:31 | |
| *** rtheis has quit IRC | 15:35 | |
| *** pcaruana has quit IRC | 15:37 | |
| *** rtheis has joined #openstack-lbaas | 15:41 | |
| *** yamamoto has quit IRC | 15:45 | |
| *** pcaruana has joined #openstack-lbaas | 15:49 | |
| *** piet has quit IRC | 15:53 | |
| *** piet has joined #openstack-lbaas | 15:56 | |
| *** manishg_wfh has quit IRC | 15:58 | |
| *** H3y has joined #openstack-lbaas | 16:01 | |
| *** H3y has quit IRC | 16:03 | |
| *** fnaval has quit IRC | 16:04 | |
| *** H3y has joined #openstack-lbaas | 16:04 | |
| *** minwang2 has joined #openstack-lbaas | 16:04 | |
| *** rtheis has quit IRC | 16:14 | |
| *** evgenyf has joined #openstack-lbaas | 16:16 | |
| *** piet has quit IRC | 16:19 | |
| *** kobis has quit IRC | 16:21 | |
| *** rtheis has joined #openstack-lbaas | 16:23 | |
| *** minwang2 has quit IRC | 16:23 | |
| *** fnaval has joined #openstack-lbaas | 16:29 | |
| *** Aish has joined #openstack-lbaas | 16:34 | |
| *** jschwarz has quit IRC | 16:41 | |
| *** bana_k has joined #openstack-lbaas | 16:45 | |
| openstackgerrit | Chaozhe Chen(ccz) proposed openstack/neutron-lbaas: Add `paramiko` in test-requirements https://review.openstack.org/283681 | 16:46 |
| *** yamamoto has joined #openstack-lbaas | 16:46 | |
| *** armax has quit IRC | 16:46 | |
| *** rtheis has quit IRC | 16:46 | |
| *** rtheis has joined #openstack-lbaas | 16:47 | |
| *** jwarendt has joined #openstack-lbaas | 16:50 | |
| *** rtheis has quit IRC | 16:51 | |
| *** nmagnezi has quit IRC | 16:52 | |
| *** nmagnezi has joined #openstack-lbaas | 16:53 | |
| *** yamamoto has quit IRC | 16:53 | |
| openstackgerrit | Franklin Naval proposed openstack/octavia: Octavia: Basic LoadBalancer Scenario Test https://review.openstack.org/172199 | 16:59 |
| *** minwang2 has joined #openstack-lbaas | 17:00 | |
| *** armax has joined #openstack-lbaas | 17:05 | |
| *** rtheis has joined #openstack-lbaas | 17:06 | |
| *** manishg_wfh has joined #openstack-lbaas | 17:09 | |
| *** Purandar has joined #openstack-lbaas | 17:09 | |
| *** rtheis has quit IRC | 17:12 | |
| *** rtheis has joined #openstack-lbaas | 17:13 | |
| *** allan_h has joined #openstack-lbaas | 17:15 | |
| *** bana_k has quit IRC | 17:28 | |
| TrevorV | Hey sbalukoff you online just yet? | 17:30 |
| *** localloop127 has quit IRC | 17:32 | |
| *** piet has joined #openstack-lbaas | 17:37 | |
| *** jschwarz has joined #openstack-lbaas | 17:43 | |
| *** Aish has quit IRC | 17:46 | |
| fnaval | blogan: https://bugs.launchpad.net/octavia/+bug/1548628 | 17:50 |
| openstack | Launchpad bug 1548628 in octavia "Config Params Conflict: test_build_agent_config unit test failure" [Undecided,New] | 17:50 |
| *** rtheis has quit IRC | 17:56 | |
| *** rtheis has joined #openstack-lbaas | 17:57 | |
| *** Aish has joined #openstack-lbaas | 17:59 | |
| *** kevo has joined #openstack-lbaas | 18:00 | |
| *** rtheis has quit IRC | 18:01 | |
| *** minwang2 has quit IRC | 18:04 | |
| *** ducttape_ has quit IRC | 18:28 | |
| *** localloop127 has joined #openstack-lbaas | 18:29 | |
| *** jschwarz has quit IRC | 18:29 | |
| *** localloop127 has quit IRC | 18:33 | |
| *** rtheis has joined #openstack-lbaas | 18:41 | |
| *** Purandar has quit IRC | 18:42 | |
| *** ducttape_ has joined #openstack-lbaas | 18:44 | |
| *** minwang2 has joined #openstack-lbaas | 18:45 | |
| openstackgerrit | German Eichberger proposed openstack/octavia: [WIP] Cascading LB delete https://review.openstack.org/282587 | 18:46 |
| xgerman | blogan: when you get a chance can you please look at https://review.openstack.org/#/c/282587/3/octavia/api/v1/controllers/load_balancer.py if I did the pecan thing right? | 18:47 |
| *** klindgren__ has joined #openstack-lbaas | 18:51 | |
| madhu_ak | fnaval : looking into your patch | 18:58 |
| *** minwang2 has quit IRC | 19:00 | |
| klindgren__ | Hello - I have question re: lbaasv2 is there a plan to change lbaasv2 to accept networkid instead of a specific subnet_id/name when creating a vip. That way we can have multiple subnets configured under a network and the lbaas stuff can automatically choose the subnet that has space. | 19:03 |
| *** klindgren__ is now known as klindgren | 19:04 | |
| *** manishg_wfh has quit IRC | 19:06 | |
| *** manishg_wfh has joined #openstack-lbaas | 19:15 | |
| *** numans has quit IRC | 19:21 | |
| fnaval | madhu_ak: i think i've figured it out - hold off on investigating further. i'll be pushing something ups oon | 19:22 |
| fnaval | soon | 19:22 |
| madhu_ak | oh. okay. for that patch, sounds like commands.py file has been removed from tempest codebase | 19:25 |
| madhu_ak | which I why, I see it is failing in sonar gate | 19:25 |
| *** Purandar has joined #openstack-lbaas | 19:27 | |
| madhu_ak | fnaval, I am worried aout this one: https://review.openstack.org/#/c/283755/ | 19:27 |
| madhu_ak | fnaval, not sure whats the commit message is heading to. | 19:27 |
| *** Purandar has quit IRC | 19:28 | |
| fnaval | madhu_ak: it's probably related to this: https://bugs.launchpad.net/octavia/+bug/1548628 | 19:28 |
| openstack | Launchpad bug 1548628 in octavia "Config Params Conflict: test_build_agent_config unit test failure" [High,New] | 19:28 |
| fnaval | oops wrong link | 19:28 |
| *** Purandar has joined #openstack-lbaas | 19:28 | |
| fnaval | this link: https://github.com/openstack/qa-specs/blob/master/specs/tempest/reintegrate-tempest-lib.rst | 19:29 |
| fnaval | it looks like they're going back into tempest | 19:29 |
| fnaval | no more tempest_lib; well at least deprecated | 19:29 |
| *** manishg_wfh has quit IRC | 19:29 | |
| madhu_ak | yeah, too much confusion right now | 19:29 |
| *** localloop127 has joined #openstack-lbaas | 19:29 | |
| madhu_ak | then we are safe in importing tempest files directly | 19:30 |
| *** diogogmt has joined #openstack-lbaas | 19:30 | |
| *** rtheis has quit IRC | 19:32 | |
| sbalukoff | Morning, folks. | 19:33 |
| *** localloop127 has quit IRC | 19:34 | |
| *** rtheis has joined #openstack-lbaas | 19:34 | |
| blogan | xgerman: looks fine, does pecan accept query parameters on delete and pass them to that method? | 19:35 |
| blogan | i ahven't tested it out | 19:35 |
| xgerman | neither have I | 19:35 |
| johnsom | sbalukoff Morning. All I can say is decaf man... grin | 19:35 |
| blogan | well you could just start the octavia-api without devstack and use the noop drivers, or even the simulated handler | 19:35 |
| blogan | and then you can easily set a debug point with pycharm or pdb | 19:36 |
| blogan | or just simple print statements :) | 19:36 |
| johnsom | I have moved all of the unit test issues into bugs. That is fine with me. I figured the -1's would either get a "oh yeah, I can fix that quick" or "can we move them to bugs". | 19:36 |
| xgerman | ok, cool — was just wondering if loadbalancer/delete_cascade/123 is the URL | 19:36 |
| sbalukoff | johnsom: Man, I wish I were still online last night when you did that... | 19:36 |
| *** rtheis has quit IRC | 19:37 | |
| rm_work | i told you not to worry about it :P | 19:37 |
| sbalukoff | Seriously, I was at a friend's house helping to plan a Disneyland vacation and made the mistake of checking my e-mail. One should never be that devastated when planning a Disneyland vacation... | 19:37 |
| johnsom | I was at one point, but I was on a different nick | 19:37 |
| blogan | xgerman: ohhh, i didn't see that part lol, sorry i thought i saw the delete method as the exposed and you were expecting to do /loadbalancer/123?cascade=True | 19:37 |
| blogan | i see now its not htat though | 19:37 |
| *** rtheis has joined #openstack-lbaas | 19:37 | |
| xgerman | yeah, passing parameters doesn’t workk in wsme | 19:38 |
| xgerman | so I need a new endpoint? | 19:38 |
| *** rtheis has quit IRC | 19:38 | |
| xgerman | https://www.irccloud.com/pastebin/9NijIsij/ | 19:38 |
| *** rtheis has joined #openstack-lbaas | 19:38 | |
| blogan | xgerman: i'm not sure if that method will expose that endpoint, you may need to create another controller, call it a DeleteCascade controller and then in the _lookup method route to that new cotnroller | 19:38 |
| blogan | xgerman: but I"m not totally sure | 19:38 |
| sbalukoff | rm_work: The gate job failed again for that "update data model conversion" patch. Do you know if your barbican fix fixes things for py34 as well? | 19:38 |
| xgerman | yeah, same here — thought I better ask | 19:38 |
| xgerman | or we just make delete to cascade | 19:39 |
| xgerman | since otherwise the LBaaS V2 will check and not send to delete | 19:39 |
| xgerman | the current lb delete already deletes all he lsietners | 19:39 |
| xgerman | when you call it with curl instead of LBaaS V2 which checks and throws an exception before trying | 19:40 |
| blogan | xgerman: i'm willing to bet that you'll need to build it like the helath monitor controller | 19:40 |
| xgerman | ok | 19:40 |
| johnsom | It looks like the py34 issue was with reaching a mirror at rax | 19:40 |
| xgerman | so you want an extra cascade method or make the current delete cascade and leave it to LBaaS V2 to figure out what to do | 19:40 |
| sbalukoff | Aah, ok. So it's the "keep rechecking until it works" game. Got it. | 19:41 |
| *** minwang2 has joined #openstack-lbaas | 19:42 | |
| johnsom | yeah, I think there were something like three issues that broke openstack, some ubuntu hash mis-match issue with some of the mirrors, and this pypi mirror issue | 19:42 |
| xgerman | brb | 19:42 |
| sbalukoff | Fun! | 19:42 |
| johnsom | All in the last 24 hours | 19:42 |
| sbalukoff | I suppose that's to be expected: This is the week when everyone is merging probably-slightly-still-buggy new features. | 19:43 |
| rm_work | sbalukoff: erk,it should, it was a script isssue | 19:43 |
| sbalukoff | rm_work: Ok. Thank you for getting on top of that fix so fast last night, eh. | 19:43 |
| rm_work | yeah same rax apt issue i saw last night | 19:43 |
| rm_work | >_< | 19:43 |
| rm_work | one more recheck >_> | 19:43 |
| sbalukoff | Eeyup. | 19:44 |
| johnsom | sbalukoff So, I still have a few to +A, working on those now. I also still need to read up on haproxy acls a bit to make sure I understand expected behavior. | 19:44 |
| blogan | xgerman: no i mean if what you ahev done works, then keep it, if it doesn't then you'll need to create a whole new class, pecan contrller, call it LoadBalancerCascadeDelete, and it'll just ahve one method in it for the cascade delete | 19:44 |
| johnsom | sbalukoff Though the fact that "face value" confuses me, I am really worried about the support load from L7... | 19:45 |
| blogan | though with the BaseController being a pecan RestController, you may have to use a regulr pecan controller | 19:45 |
| *** evgenyf has quit IRC | 19:45 | |
| johnsom | We are going to need to be pretty detailed with our docs and include lots of examples. | 19:45 |
| sbalukoff | johnsom: Ok. Please feel free to ping me if you have questions there. Once you get how they work, what I'm doing in my code should become pretty obvious. (And I've been working with them for years, as we've had a very similar feature in our old, trying-to-obsolete, proprietary load balancing solution on our pre-OpenStack cloud for years.) | 19:46 |
| *** manishg_wfh has joined #openstack-lbaas | 19:46 | |
| sbalukoff | johnsom: I think writing a document which gives some recipes for how to do the most common things will help reduce that support load a lot. | 19:46 |
| johnsom | Yeah, I have used them for years on Cisco and F5 gear too. Thus, the concerns | 19:46 |
| sbalukoff | Seriously-- you can do all kinds of interesting things with L7 policies / rules. But 99% of the time, there are only two options anyone does. | 19:47 |
| sbalukoff | These are: REDIRECT_TO_POOL based on PATH STARTS_WITH | 19:47 |
| sbalukoff | And REDIRECT_TO_POOL based on HOST REGEX | 19:47 |
| rm_work | yeah that is all i tested | 19:48 |
| rm_work | i never even tried adding multiple rules :/ | 19:48 |
| rm_work | i only know very basic L7 theory | 19:48 |
| sbalukoff | Well, you do occasionally see a combination of the two. | 19:48 |
| sbalukoff | At some point, the tenant will say to themselves "why don't I just create a second load balancer?" | 19:49 |
| sbalukoff | ...and that's usually how it goes. | 19:49 |
| *** Purandar has quit IRC | 19:52 | |
| *** Purandar has joined #openstack-lbaas | 19:52 | |
| sbalukoff | johnsom and xgerman: I responded to your comments on the L7 database structures patch. I would love to get your feedback on that (specifically, do my explanations make sense and do you have further questions)? | 19:54 |
| *** piet has quit IRC | 19:56 | |
| *** piet has joined #openstack-lbaas | 19:56 | |
| *** Bjoern_ has quit IRC | 20:05 | |
| *** sc68cal has joined #openstack-lbaas | 20:10 | |
| rm_work | i wish there was a way to failfast these tests | 20:10 |
| *** ducttape_ has quit IRC | 20:11 | |
| sc68cal | sbalukoff: when you did the alembic migration for https://review.openstack.org/#/c/218560/21 - was that generated from the schema or did you write it by hand? | 20:16 |
| sc68cal | starting from an empty alembic script | 20:16 |
| sc68cal | s/generated from the schema/autogenerated/ | 20:17 |
| sbalukoff | sc68cal: At this point I don't recall anymore. | 20:18 |
| sbalukoff | sc68cal: I think the original structure may have come from an autogenerated script, but everything inside is generated by hand. | 20:18 |
| sbalukoff | I'm not aware of any way to do auto-generated alembic migrations based on SQLalchemy model or something. | 20:18 |
| sbalukoff | I do recall that I tested the ETL in thre. | 20:19 |
| sbalukoff | there. | 20:19 |
| sc68cal | sbalukoff: ok - because I think I did an autogenerated alembic migration one time and it ended up spitting out a lot of stuff like trying to drop constraints and doing stuff that was mysql specfic - hence the breakage of postgres in https://bugs.launchpad.net/bugs/1548807 | 20:19 |
| openstack | Launchpad bug 1548807 in neutron "neutron lbaas DB support is broken on PostgreSQL" [Critical,Confirmed] - Assigned to Stephen Balukoff (sbalukoff) | 20:19 |
| sbalukoff | Though, of course, we don't really have good automated tests for ETL stuff. | 20:19 |
| xgerman | sc68cal nice to see you in the LBaaS room ;-) | 20:20 |
| sc68cal | xgerman: just stopping by to comiserate | 20:20 |
| xgerman | :-) | 20:20 |
| sbalukoff | sc68cal: Is there a way to do migrations based on one DB flavor? I thought the point of alembic was to not have to worry about the DB engine on the back-end. | 20:21 |
| *** crc32 has joined #openstack-lbaas | 20:21 | |
| sc68cal | I just noticed our postgresql job was failing in our experimental queue for fwaas so came by to sympathize | 20:21 |
| sc68cal | sbalukoff: I think the issue is that alembic *by itself* will autogenerate migrations that are tied tightly to the db the dev runs it on | 20:21 |
| sc68cal | sbalukoff: hence why I think people write them by creating empty alemibc scripts and doing it all by hand | 20:22 |
| sbalukoff | ...really? | 20:22 |
| sc68cal | or doing an autogenerate and deleting all the cruft | 20:22 |
| sbalukoff | Ok, in this case we really don't want to delete the cruft because that ETL is important if you have a running installation. | 20:22 |
| sc68cal | sbalukoff: ok - so I guess the question is - you baked in these lines - which I guess are mysql specific things that sqlalchemy does - https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/db/migration/alembic_migrations/versions/mitaka/expand/6aee0434f911_independent_pools.py#L45 | 20:24 |
| sc68cal | I guess sqlalchemy as part of the orm makes those constraints | 20:24 |
| sc68cal | for mysql - and then postgres it does something different | 20:24 |
| sbalukoff | Fun. | 20:24 |
| sbalukoff | Ok, if I can't trust op.drop_constraint... I'm not sure how to fix this. | 20:25 |
| sc68cal | so at least in the main neutron repo - we used sa.Column objects | 20:26 |
| sbalukoff | I could comment out those lines and put a note for anyone using MySQL that they're going to need to do that. But wouldn't that then break any migration tests that use MySQL? | 20:26 |
| sc68cal | we have some migrations that do a check on the engine type | 20:26 |
| sc68cal | if mysql: bleh | 20:26 |
| sc68cal | see - examples like this | 20:27 |
| sc68cal | https://github.com/openstack/neutron/blob/master/neutron/db/migration/alembic_migrations/versions/liberty/expand/52c5312f6baf_address_scopes.py | 20:27 |
| sbalukoff | sc68cal: You're saying we should not care about people potentially using this code in production with MySQL? | 20:27 |
| sc68cal | op.create_table where we pass in sa.Column objects | 20:27 |
| sc68cal | let me find one with a constraint | 20:27 |
| sc68cal | yeah here we go | 20:28 |
| sc68cal | https://github.com/openstack/neutron/blob/master/neutron/db/migration/alembic_migrations/versions/liberty/expand/31337ec0ffee_flavors.py#L40 | 20:28 |
| sbalukoff | sc68cal: Right. But in this case we're manipulating tables and columns that already exist in the DB. | 20:28 |
| sbalukoff | And we don't want to lose the data in those tables. | 20:28 |
| sbalukoff | How do I go about manipulating constraints on already-existing columns? | 20:29 |
| sc68cal | sbalukoff: there's got to be a constructor for a corresponding SA type that you can create | 20:29 |
| johnsom | Would with migration.remove_fks_from_table help here? | 20:29 |
| johnsom | https://review.openstack.org/#/c/278796/7/neutron_vpnaas/db/migration/alembic_migrations/versions/liberty/contract/56893333aa52_fix_identifier_map_fk.py | 20:29 |
| *** Purandar has quit IRC | 20:30 | |
| sbalukoff | johnsom: It might. | 20:30 |
| sbalukoff | What's frustrating about this is that I have no good way to test this except take a shot in the dark... | 20:31 |
| sbalukoff | Especially because nothing appeared broken in the neutron-lbaas tree. Apparently neutron has different testing for DB manipulations. | 20:31 |
| *** localloop127 has joined #openstack-lbaas | 20:31 | |
| sbalukoff | sc68cal: How would I go about generating an automatic migration? | 20:32 |
| sc68cal | sbalukoff: my tl;dr was don't | 20:33 |
| sc68cal | they suck and bake in engine specific things | 20:33 |
| sbalukoff | So, again: Screw anyone using this code in production. | 20:33 |
| sc68cal | dude - you're breaking the postgresql gate | 20:33 |
| sc68cal | https://bugs.launchpad.net/bugs/1548807 | 20:34 |
| openstack | Launchpad bug 1548807 in neutron "neutron lbaas DB support is broken on PostgreSQL" [Critical,Confirmed] - Assigned to Stephen Balukoff (sbalukoff) | 20:34 |
| sbalukoff | I'm being asked to make the choice of "break a gate" or "break anyone using this code in production" | 20:34 |
| sc68cal | so none of this "screw people in production" nonsense | 20:34 |
| sbalukoff | *sigh* | 20:34 |
| sbalukoff | Ok. | 20:34 |
| sc68cal | you broke it - so either fix it or revert that whole big patch please | 20:34 |
| sc68cal | reach out to Ann K or HenryG for db help if you need it | 20:35 |
| *** ducttape_ has joined #openstack-lbaas | 20:35 | |
| *** localloop127 has quit IRC | 20:36 | |
| *** Purandar has joined #openstack-lbaas | 20:36 | |
| *** Purandar has quit IRC | 20:37 | |
| sbalukoff | Would I fix it by manipulating that one migration file, or by suggesting a new migration file? In other words, does the gate in question give up at the first error? | 20:38 |
| sc68cal | yeah it gives up right at that line in your migration script - http://logs.openstack.org/95/283495/1/check/gate-manila-tempest-dsvm-neutron-postgres-lvm-multibackend/a27485f/logs/devstacklog.txt.gz#_2016-02-23_10_40_31_494 | 20:38 |
| sbalukoff | Ok, thanks. And we won't know if my fix works until it's merged? | 20:39 |
| sbalukoff | (Awesome!) | 20:39 |
| sbalukoff | Ok, I'll get to fixing it. | 20:39 |
| sc68cal | do you guys have an experimental job for postgresql? | 20:39 |
| sbalukoff | I don't think so. | 20:39 |
| johnsom | No | 20:40 |
| sc68cal | if not maybe xgerman can make a change in fwaas and do a check experimental | 20:40 |
| sbalukoff | Otherwise we probably would have noticed this. | 20:40 |
| sc68cal | that'll run the postgresql | 20:40 |
| sbalukoff | If someone can make that change while we work on this, that would be great-- I don't want to hold up work on Neutron. | 20:41 |
| sc68cal | it's odd - why does the fwaas postrgresql job install neutron lbaas | 20:41 |
| sbalukoff | er... while I work on fixing that migration. | 20:41 |
| xgerman | mmh | 20:41 |
| xgerman | I usually stay out of testing... | 20:41 |
| sc68cal | well - alternatively, if we want to figure out why fwaas is installing lbaas in our tests, and decouple - that's fine with me too if you guys don't want to support postgres | 20:42 |
| xgerman | I don;t think we want to to support postgres in FwaaS neither — but amax wants it | 20:42 |
| xgerman | armax | 20:42 |
| sbalukoff | I'll fix it; I'm just hoping to have a better way to know the problem is actually fixed than commit - merge - wait | 20:42 |
| armax | xgerman: what do I want exactly? | 20:43 |
| xgerman | https://review.openstack.org/#/c/279339/ - postgres support | 20:43 |
| xgerman | my vote was to remove it ;-) | 20:43 |
| armax | oh | 20:46 |
| armax | I suggested removing it all of it didn’t i? | 20:46 |
| xgerman | armax: "We could also consider making pg support conditional as it's done in the link referenced inline." | 20:46 |
| xgerman | so we moved in that direction... | 20:47 |
| armax | ok | 20:47 |
| armax | it’s likely you’re not using any of that code yet | 20:47 |
| armax | but that’s fiine | 20:47 |
| armax | I guess | 20:47 |
| xgerman | yeah, it’s a bigger Neutron problem since if one component decides to support postgres with all the entanglements the others need, too | 20:48 |
| xgerman | so probably some decision needed... | 20:49 |
| sbalukoff | Wait... what I I just put the MySQL stuff in that migration in a try: block. | 20:52 |
| sc68cal | I think the issue is that it's an anti-pattern to write migrations that are directly manipulating things that are engine-specific artifacts that SQLAlchemy creates | 20:54 |
| sc68cal | (if that constraint was created by sqlalchemy) | 20:54 |
| johnsom | Yes | 20:56 |
| sc68cal | https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/db/loadbalancer/models.py#L253 ? | 20:58 |
| TrevorV | rm_work you said the L7 stuff would merge today :P | 20:58 |
| johnsom | It might | 20:59 |
| TrevorV | I'd love to see that, because my review is behind a bit | 20:59 |
| TrevorV | o_0 | 20:59 |
| sc68cal | ah - it's not fwaas depending on it- it's neutron-pg-full | 21:02 |
| sc68cal | *gate-tempest-dsvm-neutron-pg-full | 21:02 |
| TrevorV | Right now I have a conflict that wasn't a conflict. But it wouldn't let me do a commit --amend cuz it was "during a cherry pick" | 21:04 |
| TrevorV | so I don't really know how to rebase my review o_0 | 21:04 |
| rm_work | finish the cherry-pick | 21:04 |
| rm_work | then "git reset HEAD~1" | 21:04 |
| rm_work | then commit --amend | 21:04 |
| dougwig | oh, a pg argument. | 21:05 |
| rm_work | or possibly redo your commit (depending on if the existing commit still exists in your chain) | 21:05 |
| rm_work | TrevorV: can you finish the cherry-pick and show me your git log? | 21:05 |
| *** manishg_wfh has quit IRC | 21:05 | |
| TrevorV | rm_work sorry, I just negated it a second ago to run fresh tox just in case I missed something | 21:05 |
| TrevorV | I'll do it once more here in a second | 21:06 |
| TrevorV | Gist you "all the things" without doing a git-review or whatnot | 21:06 |
| rm_work | yeah | 21:06 |
| rm_work | the whole log from when you try to start a cherry-pick, to when you finish it, and a git-status, and a git-log | 21:06 |
| rm_work | is ideal | 21:06 |
| TrevorV | okay, well I run git-status when the cherry-pick throws a fit | 21:07 |
| TrevorV | Before cherry pick I'll give the top 2 of git-log, and then after I'll do the same | 21:07 |
| rm_work | ok we'll be back in the gate momentarily sbalukoff | 21:08 |
| sbalukoff | Ok, I think I have fix that works here (that uses the remove_fks_from_table -- thanks for that johnsom; I had no idea it existed). Running tests locally before I commit it. | 21:09 |
| sc68cal | dougwig: armax: I mean if we just delete the gate-tempest-dsvm-neutron-pg-full job then that works for me | 21:09 |
| armax | ok | 21:09 |
| sc68cal | that'll make it clear that we don't like pgsql and don't want to support it. | 21:10 |
| dougwig | seems like if we cared, we'd gate on it. | 21:10 |
| sbalukoff | Does op.create_foreign_key work in PostgreSQL migrations? (I'm guessing not?) | 21:11 |
| dougwig | hmm | 21:11 |
| dougwig | https://www.irccloud.com/pastebin/gZpmqpgo/ | 21:11 |
| dougwig | maybe it's time the postgres version wasn't nova-net, hmm? | 21:11 |
| sc68cal | sbalukoff: it should | 21:12 |
| johnsom | If I had the time I would be very curious about this issue since postgres historically has been a superset of features and tracked the standards closer than mysql | 21:13 |
| dougwig | johnsom: pg is much stricter. causes issue all the time. | 21:13 |
| sc68cal | well the main issue is the migration that was written deletes a constraint that SQLAlchemy creates under the covers, instead of using an SQLAlchemy object to delete it | 21:14 |
| sc68cal | so that it'll delete the constraint in both DBs correctly | 21:14 |
| *** Purandar has joined #openstack-lbaas | 21:14 | |
| sbalukoff | sc68cal: Ok. | 21:16 |
| *** neelashah1 has quit IRC | 21:21 | |
| openstackgerrit | Stephen Balukoff proposed openstack/neutron-lbaas: Fix shared-pools migration in PostgreSQL gate https://review.openstack.org/283802 | 21:21 |
| sbalukoff | sc68cal: Is there some way you can test that on your gate? Can you create a dummy commit which lists that change-id as a dependency or something? | 21:22 |
| sc68cal | sbalukoff: try doing "check experimental" on your patch? | 21:22 |
| sbalukoff | Ok. | 21:24 |
| sbalukoff | And now we wait to see whether PostgreSQL is happy with that. :P | 21:24 |
| rm_work | johnsom: were you not yet good on https://review.openstack.org/#/c/280478/ ? | 21:25 |
| dougwig | just talked to armax. we're going to try adding neutron-pg to the check queue (nv) and see if the stability issues it used to have have been resolved. | 21:26 |
| dougwig | anyone have a problem with that? | 21:26 |
| johnsom | rm_work I need to give that one another pass | 21:26 |
| sbalukoff | As long as it's non-voting, knock yourself out, eh. :) | 21:26 |
| johnsom | I was just finishing a re-test of https://review.openstack.org/#/c/278223/ to see if I can pull the -1 | 21:28 |
| rm_work | johnsom: technically there's not a whole lot to test individually | 21:28 |
| rm_work | i kinda just treated the whole stack as a lump | 21:29 |
| rm_work | honestly given the way testing works i might have just been ok with it not being split up as much <_< | 21:29 |
| rm_work | i still had to review all the lines anyway | 21:29 |
| johnsom | Well, I have the whole stack installed, but I am running targeted tests | 21:29 |
| *** amotoki has quit IRC | 21:30 | |
| sbalukoff | rm_work: When writing this, keeping things split up did help me find where breakages were sooner, and put the fix tests for those iin the appropriate place. Also, smaller patches probably seem less intimidating to people (who would have thought this would get to 6100+ lines?) Otherwise... I will wholeheartedly say that maintaining a patch chain this big is a serious pain in the ass. | 21:31 |
| rm_work | yeah, was it worth it? :P | 21:32 |
| sbalukoff | "no comment" | 21:32 |
| johnsom | Yes, smaller patches make it easier to cover a chunk and not get fried | 21:32 |
| sbalukoff | Ask me again in a couple weeks? | 21:32 |
| *** localloop127 has joined #openstack-lbaas | 21:32 | |
| johnsom | Ok, functionality is confirmed. | 21:34 |
| johnsom | I would like to see a test run to see how many ACLs haproxy can handle on a line. Just so we have an idea of the limit. | 21:34 |
| sbalukoff | johnsom: You can create rules that are exact duplicates of each other. | 21:34 |
| sbalukoff | And ANDing them together should still work. | 21:34 |
| sbalukoff | (That is to say, we don't disallow that behavior because haproxy doesn't.) | 21:35 |
| johnsom | I'm just curious how long of a line can it take before breaking. Someone with some time and scripting skills could probably figure it out pretty quick | 21:35 |
| sbalukoff | johnsom: I'll poke at that right now. | 21:36 |
| *** localloop127 has quit IRC | 21:36 | |
| johnsom | Ok, not time sensitive, I already +A'd that one, but would be handy to know | 21:37 |
| rm_work | can bedis just tell us? :P | 21:38 |
| sbalukoff | Ok, I kinda want to restack here anyway-- things are a bit crufy on this box. | 21:38 |
| rm_work | sbalukoff: bit.do/devstack :P | 21:39 |
| openstackgerrit | Merged openstack/octavia: Update data model conversion to be more complete https://review.openstack.org/280166 | 21:39 |
| rm_work | woot | 21:39 |
| rm_work | one down | 21:40 |
| rm_work | 10 to go | 21:40 |
| rm_work | johnsom: if you +A the model update, we get a two-for-one :P | 21:40 |
| johnsom | I'm working on it. Had to stop to open a unit test bug.... | 21:40 |
| sbalukoff | rm_work: Thanks for that, eh! | 21:41 |
| johnsom | Hey, I +A'd it | 21:41 |
| rm_work | :P | 21:41 |
| openstackgerrit | Doug Wiegley proposed openstack/neutron-lbaas: Expand gate hooks to allow for more than just octavia testing https://review.openstack.org/282900 | 21:42 |
| *** rtheis has quit IRC | 21:46 | |
| *** rtheis has joined #openstack-lbaas | 21:49 | |
| *** nmagnezi_ has joined #openstack-lbaas | 21:49 | |
| *** neelashah has joined #openstack-lbaas | 21:49 | |
| *** nmagnezi has quit IRC | 21:50 | |
| *** nmagnezi_ is now known as nmagnezi | 21:50 | |
| openstackgerrit | Franklin Naval proposed openstack/octavia: Octavia: Basic LoadBalancer Scenario Test https://review.openstack.org/172199 | 21:50 |
| nmagnezi | hi guys do any of know which script/alembic/etc adds the lbaas tables to the neutron db? | 21:54 |
| sbalukoff | nmagnezi: Are you asking because of the PostgreSQL breakage? | 21:56 |
| nmagnezi | sbalukoff, nop. i'm configuring stuff manually (testing Octavia on centos) and need to do some deployment steps without devstack | 21:57 |
| sbalukoff | Oh, ok! | 21:57 |
| nmagnezi | :) | 21:57 |
| sbalukoff | So, all of that should be in neutron-lbaas project under: neutron_lbaas/db/migrations | 21:57 |
| sbalukoff | It's all alembic migrations, so if you can use that to apply them you'll have an easier time. | 21:58 |
| nmagnezi | sbalukoff, aye. taking a look now :) | 22:00 |
| nmagnezi | sbalukoff, this is invoked with neutron-db-manage? | 22:01 |
| sbalukoff | Yes, I think so. | 22:02 |
| sbalukoff | The relevant line from a devstack install is: $NEUTRON_BIN_DIR/neutron-db-manage --service lbaas --config-file $NEUTRON_CONF --config-file /$Q_PLUGIN_CONF_FILE upgrade head | 22:02 |
| sbalukoff | And it should be roughly equivalent in production. | 22:02 |
| nmagnezi | sbalukoff, thanks a lot! | 22:03 |
| openstackgerrit | Franklin Naval proposed openstack/octavia: Octavia: Basic LoadBalancer Scenario Test https://review.openstack.org/172199 | 22:03 |
| nmagnezi | sbalukoff, it worked :) | 22:05 |
| sbalukoff | Great! | 22:06 |
| johnsom | sbalukoff https://review.openstack.org/#/c/265690/1..22/octavia/api/v1/types/l7policy.py line 30, why isn't that UuidType? | 22:08 |
| sbalukoff | I think because it can occasionally be None. Though you're right that that might just be a minor error. | 22:09 |
| johnsom | Ok, I will put the bug in that way. It it doesn't work it should probably be cut in size | 22:09 |
| sbalukoff | Yep. | 22:09 |
| johnsom | As rm_work mentioned | 22:09 |
| rm_work | yeah pool_id can probably be UUIDType | 22:12 |
| johnsom | Your bug numbers are going to look pretty good for Mitaka | 22:12 |
| sbalukoff | Heh! | 22:12 |
| rm_work | it's project_id that needs to be String(36) | 22:12 |
| sbalukoff | Well, if some others want to grab some of them, I won't object, eh. | 22:12 |
| rm_work | but regardless, returns don't need the max_length | 22:12 |
| *** manishg_wfh has joined #openstack-lbaas | 22:12 | |
| rm_work | just POST/PUT | 22:12 |
| sbalukoff | rm_work: True. | 22:12 |
| johnsom | righyt | 22:12 |
| rm_work | ugh got another gate failure | 22:23 |
| rm_work | on 282288 | 22:23 |
| sbalukoff | Gonna be slow going getting these in. :/ | 22:24 |
| sbalukoff | Almost looks like the amphora spontaneously went offline with that one. :/ | 22:26 |
| sbalukoff | Or at least, the CW task which tracked an update check got stuck. | 22:27 |
| sbalukoff | In any case, it's weird. | 22:28 |
| sbalukoff | (er... listener update change) | 22:28 |
| *** sc68cal has quit IRC | 22:30 | |
| rm_work | probably a nova issue? | 22:30 |
| rm_work | or the timeout bug that we haven't merged the fix for yet, right? | 22:31 |
| rm_work | the one johnsom was working on | 22:31 |
| sbalukoff | It could be either. | 22:33 |
| *** localloop127 has joined #openstack-lbaas | 22:33 | |
| sbalukoff | Well... | 22:33 |
| sbalukoff | probably not the timeout bug. | 22:33 |
| sbalukoff | I dunno. | 22:33 |
| johnsom | Why are we getting "Wait for load balancer ran for 600 seconds" errors on the listener test? I thought that was noop.... | 22:34 |
| rm_work | errrrrrrrrr | 22:34 |
| rm_work | that is | 22:34 |
| rm_work | a very good point | 22:34 |
| rm_work | wtf | 22:34 |
| johnsom | http://logs.openstack.org/78/280478/7/gate/gate-neutron-lbaasv2-dsvm-listener/0dbc907/console.html | 22:34 |
| rm_work | yeah i'm digging through it at the moment | 22:35 |
| johnsom | rm_work thanks! | 22:35 |
| johnsom | sbalukoff This doesn't look good: http://logs.openstack.org/78/280478/7/gate/gate-neutron-lbaasv2-dsvm-listener/0dbc907/logs/screen-o-cw.txt.gz#_2016-02-23_22_16_22_567 | 22:36 |
| rm_work | it looks like it got stuck in PENDING_UPDATE | 22:36 |
| rm_work | but no clue why | 22:36 |
| rm_work | uhhh | 22:36 |
| rm_work | yeah that's bad | 22:36 |
| rm_work | why didn't we see that the first like | 22:36 |
| rm_work | 20 times the tests ran on this patch | 22:36 |
| *** localloop127 has quit IRC | 22:38 | |
| johnsom | Well, umm, that exact failing method is changed in this patch: https://review.openstack.org/#/c/265430 | 22:41 |
| sbalukoff | Looking now. | 22:43 |
| sbalukoff | Looks like pool creation didn't finish there? | 22:44 |
| sbalukoff | Or wait... | 22:44 |
| sbalukoff | Hmmmm.... | 22:44 |
| openstackgerrit | Franklin Naval proposed openstack/octavia: Octavia: Basic LoadBalancer Scenario Test https://review.openstack.org/172199 | 22:47 |
| sbalukoff | Do we know which test is being run at that moment? | 22:47 |
| johnsom | neutron_lbaas.tests.tempest.v2.api.test_listeners_non_admin.ListenersTestJSON.test_update_listener | 22:47 |
| johnsom | Also remember, this is a test with noop drivers | 22:48 |
| sbalukoff | but the data model should be complete here-- it attempted to do an update of the 'default_pool_id' to 'None', it seems. | 22:49 |
| sbalukoff | Whyyyyy did that happen... hmmm... | 22:49 |
| *** nmagnezi has quit IRC | 22:49 | |
| sbalukoff | (wondering whether this is something that got fixed along the way and got un-fixed by a rebase.) | 22:49 |
| sbalukoff | yeah, it's fixed in a later patch. | 22:50 |
| sbalukoff | Ok... I'll roll that fix into this one. | 22:50 |
| sbalukoff | *sigh* | 22:50 |
| rm_work | T_T | 22:51 |
| sbalukoff | Still confusing that this was never caught in a previous automated test. | 22:51 |
| sbalukoff | (tempest test, I mean...) | 22:52 |
| rm_work | yeah | 22:52 |
| rm_work | i don't understand how it failed now and never before | 22:52 |
| rm_work | it's the same test | 22:52 |
| rm_work | and this doesn't look like something that would be intermittent. ... | 22:52 |
| sbalukoff | Yep. | 22:52 |
| sbalukoff | (running tests locally) | 22:55 |
| rm_work | brb making some breakfast really quick | 22:56 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Fix model update flows https://review.openstack.org/280478 | 22:56 |
| TrevorV | Oh the absurdity of that statement rm_work | 22:56 |
| sbalukoff | I guess we'll watch that one. | 22:56 |
| sbalukoff | Working on bubbling that up through the chain now... | 22:57 |
| TrevorV | Awesome sbalukoff | 22:57 |
| dougwig | to anyone that cares about the old namespace driver: https://review.openstack.org/#/c/283841/ | 22:57 |
| *** diogogmt has quit IRC | 22:58 | |
| *** allan_h has quit IRC | 22:58 | |
| *** armax has quit IRC | 22:58 | |
| *** pc_m has quit IRC | 22:58 | |
| *** mariusv has quit IRC | 22:58 | |
| *** yuanying has quit IRC | 22:58 | |
| *** _laco has quit IRC | 22:58 | |
| *** rtheis has quit IRC | 22:58 | |
| *** pcaruana has quit IRC | 22:58 | |
| *** reedip has quit IRC | 22:58 | |
| *** markvan has quit IRC | 22:58 | |
| *** Dave has quit IRC | 22:58 | |
| *** jorgem[away] has quit IRC | 22:58 | |
| *** clduser has quit IRC | 22:58 | |
| *** intr1nsic has quit IRC | 22:58 | |
| *** jwarendt has quit IRC | 22:58 | |
| *** sbalukoff has quit IRC | 22:58 | |
| *** logan- has quit IRC | 22:58 | |
| *** davidlenwell has quit IRC | 22:58 | |
| *** bradjones has quit IRC | 22:58 | |
| *** jidar has quit IRC | 22:58 | |
| *** manishg_wfh has quit IRC | 22:58 | |
| *** crc32 has quit IRC | 22:58 | |
| *** H3y has quit IRC | 22:58 | |
| *** TrevorV has quit IRC | 22:58 | |
| *** Kiall has quit IRC | 22:58 | |
| *** chlong_ has quit IRC | 22:58 | |
| *** zigo has quit IRC | 22:58 | |
| *** eezhova has quit IRC | 22:58 | |
| *** ebagdasa has quit IRC | 22:58 | |
| *** ducttape_ has quit IRC | 22:58 | |
| *** fnaval has quit IRC | 22:58 | |
| *** dnovosel has quit IRC | 22:58 | |
| *** itsuugo has quit IRC | 22:58 | |
| *** elarson has quit IRC | 22:58 | |
| *** xgerman has quit IRC | 22:58 | |
| *** rm_work has quit IRC | 22:58 | |
| *** dougwig has quit IRC | 22:58 | |
| *** barclaac has quit IRC | 22:58 | |
| *** amitry has quit IRC | 22:58 | |
| *** mdavidson has quit IRC | 22:58 | |
| *** kbyrne has quit IRC | 22:58 | |
| *** ramishra has quit IRC | 22:58 | |
| *** neelashah has quit IRC | 22:58 | |
| *** Purandar has quit IRC | 22:58 | |
| *** Aish has quit IRC | 22:58 | |
| *** ajmiller has quit IRC | 22:58 | |
| *** kevinbenton has quit IRC | 22:58 | |
| *** hockeynut has quit IRC | 22:58 | |
| *** HenryG has quit IRC | 22:58 | |
| *** ptoohill has quit IRC | 22:58 | |
| *** Oku_OS has quit IRC | 22:58 | |
| *** blogan has quit IRC | 22:58 | |
| *** bedis has quit IRC | 22:58 | |
| *** pai15 has quit IRC | 22:58 | |
| *** crc32|znc has quit IRC | 22:58 | |
| *** lmiccini has quit IRC | 22:58 | |
| *** kfox1111 has quit IRC | 22:58 | |
| *** ajo has quit IRC | 22:58 | |
| *** redrobot has quit IRC | 22:58 | |
| *** thomasem has quit IRC | 22:58 | |
| *** pck has quit IRC | 22:58 | |
| *** ctracey has quit IRC | 22:58 | |
| *** mhayden has quit IRC | 22:58 | |
| *** mhayden has joined #openstack-lbaas | 23:01 | |
| *** ctracey has joined #openstack-lbaas | 23:01 | |
| *** pck has joined #openstack-lbaas | 23:01 | |
| *** thomasem has joined #openstack-lbaas | 23:01 | |
| *** redrobot has joined #openstack-lbaas | 23:01 | |
| *** ajo has joined #openstack-lbaas | 23:01 | |
| *** intr1nsic has joined #openstack-lbaas | 23:01 | |
| *** clduser has joined #openstack-lbaas | 23:01 | |
| *** jorgem[away] has joined #openstack-lbaas | 23:01 | |
| *** Dave has joined #openstack-lbaas | 23:01 | |
| *** markvan has joined #openstack-lbaas | 23:01 | |
| *** reedip has joined #openstack-lbaas | 23:01 | |
| *** pcaruana has joined #openstack-lbaas | 23:01 | |
| *** rtheis has joined #openstack-lbaas | 23:01 | |
| *** jidar has joined #openstack-lbaas | 23:01 | |
| *** bradjones has joined #openstack-lbaas | 23:01 | |
| *** davidlenwell has joined #openstack-lbaas | 23:01 | |
| *** logan- has joined #openstack-lbaas | 23:01 | |
| *** sbalukoff has joined #openstack-lbaas | 23:01 | |
| *** jwarendt has joined #openstack-lbaas | 23:01 | |
| *** ramishra has joined #openstack-lbaas | 23:01 | |
| *** kbyrne has joined #openstack-lbaas | 23:01 | |
| *** mdavidson has joined #openstack-lbaas | 23:01 | |
| *** amitry has joined #openstack-lbaas | 23:01 | |
| *** dougwig has joined #openstack-lbaas | 23:01 | |
| *** rm_work has joined #openstack-lbaas | 23:01 | |
| *** xgerman has joined #openstack-lbaas | 23:01 | |
| *** barclaac has joined #openstack-lbaas | 23:01 | |
| *** manishg_wfh has joined #openstack-lbaas | 23:01 | |
| *** crc32 has joined #openstack-lbaas | 23:01 | |
| *** H3y has joined #openstack-lbaas | 23:01 | |
| *** TrevorV has joined #openstack-lbaas | 23:01 | |
| *** Kiall has joined #openstack-lbaas | 23:01 | |
| *** chlong_ has joined #openstack-lbaas | 23:01 | |
| *** zigo has joined #openstack-lbaas | 23:01 | |
| *** eezhova has joined #openstack-lbaas | 23:01 | |
| *** ebagdasa has joined #openstack-lbaas | 23:01 | |
| *** diogogmt has joined #openstack-lbaas | 23:01 | |
| *** allan_h has joined #openstack-lbaas | 23:01 | |
| *** armax has joined #openstack-lbaas | 23:01 | |
| *** pc_m has joined #openstack-lbaas | 23:01 | |
| *** mariusv has joined #openstack-lbaas | 23:01 | |
| *** yuanying has joined #openstack-lbaas | 23:01 | |
| *** _laco has joined #openstack-lbaas | 23:01 | |
| *** manishg_wfh has quit IRC | 23:01 | |
| TrevorV | You're such a kidder dougwig | 23:01 |
| TrevorV | You know no one wants that around anymore :P | 23:02 |
| *** rtheis has quit IRC | 23:02 | |
| *** Oku_OS has joined #openstack-lbaas | 23:02 | |
| *** blogan has joined #openstack-lbaas | 23:02 | |
| *** bedis has joined #openstack-lbaas | 23:02 | |
| *** crc32|znc has joined #openstack-lbaas | 23:02 | |
| *** pai15 has joined #openstack-lbaas | 23:02 | |
| *** lmiccini has joined #openstack-lbaas | 23:02 | |
| *** kfox1111 has joined #openstack-lbaas | 23:02 | |
| *** kevinbenton has joined #openstack-lbaas | 23:03 | |
| *** hockeynut has joined #openstack-lbaas | 23:03 | |
| *** HenryG has joined #openstack-lbaas | 23:03 | |
| *** ptoohill has joined #openstack-lbaas | 23:03 | |
| dougwig | and this one: https://review.openstack.org/#/c/282900/ | 23:03 |
| *** neelashah has joined #openstack-lbaas | 23:03 | |
| *** Purandar has joined #openstack-lbaas | 23:03 | |
| *** Aish has joined #openstack-lbaas | 23:03 | |
| *** ajmiller has joined #openstack-lbaas | 23:03 | |
| *** ducttape_ has joined #openstack-lbaas | 23:03 | |
| *** fnaval has joined #openstack-lbaas | 23:03 | |
| *** itsuugo has joined #openstack-lbaas | 23:03 | |
| *** dnovosel has joined #openstack-lbaas | 23:03 | |
| *** elarson has joined #openstack-lbaas | 23:03 | |
| *** manishg_wfh has joined #openstack-lbaas | 23:03 | |
| *** dnovosel_ has joined #openstack-lbaas | 23:03 | |
| *** armax has quit IRC | 23:04 | |
| *** ajmiller_ has joined #openstack-lbaas | 23:05 | |
| sbalukoff | Are we back from the netsplit? | 23:06 |
| *** dnovosel has quit IRC | 23:07 | |
| *** ajmiller_ has quit IRC | 23:07 | |
| *** yamamoto has joined #openstack-lbaas | 23:07 | |
| openstackgerrit | Trevor Vardeman proposed openstack/octavia: WIP - Get Me A Load Balancer Controller https://review.openstack.org/257013 | 23:08 |
| *** piet has quit IRC | 23:11 | |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Add L7 database structures https://review.openstack.org/265430 | 23:11 |
| *** piet has joined #openstack-lbaas | 23:11 | |
| *** yamamoto has quit IRC | 23:12 | |
| *** manishg_wfh has quit IRC | 23:18 | |
| *** manishg_wfh has joined #openstack-lbaas | 23:18 | |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Update repos for L7 policy / methods https://review.openstack.org/265529 | 23:19 |
| fnaval | madhu_ak: this should be ready for review once all the gates pass. please take a look when you get a chance: https://review.openstack.org/#/c/172199/ thanks in advance! | 23:19 |
| madhu_ak | sure fnaval | 23:19 |
| *** ducttape_ has quit IRC | 23:20 | |
| openstackgerrit | Trevor Vardeman proposed openstack/octavia: WIP - Get Me A Load Balancer Controller https://review.openstack.org/257013 | 23:21 |
| *** TrevorV has quit IRC | 23:21 | |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Update repos for L7 rules / validations https://review.openstack.org/276643 | 23:24 |
| *** piet has quit IRC | 23:25 | |
| sbalukoff | AUGH! I just noticed my chain missed the peer_port issue. | 23:29 |
| sbalukoff | *sigh* | 23:29 |
| sbalukoff | Ok, going back to the start to rebase on that one... again... | 23:29 |
| *** diogogmt has quit IRC | 23:36 | |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Assign peer_port on listener creation https://review.openstack.org/282288 | 23:36 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Add L7 database structures https://review.openstack.org/265430 | 23:37 |
| *** neelashah has quit IRC | 23:39 | |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Update repos for L7 policy / methods https://review.openstack.org/265529 | 23:40 |
| johnsom | sbalukoff I think we are good except for the comments on the database patch. When you are done getting those back in the gates, if you have a few minutes, convince me we are doing the right thing with the data model. | 23:40 |
| sbalukoff | Ok, will ping you when this chain is done being rebased. | 23:41 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Update repos for L7 rules / validations https://review.openstack.org/276643 | 23:42 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Add L7 api - policies https://review.openstack.org/265690 | 23:44 |
| rm_work | back | 23:50 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Add L7 api - rules https://review.openstack.org/277718 | 23:51 |
| openstackgerrit | German Eichberger proposed openstack/octavia: [WIP] Cascading LB delete https://review.openstack.org/282587 | 23:51 |
| rm_work | sbalukoff: rebasing those should just be a checkout, like 8 cherry-picks, and a review, right? | 23:52 |
| sbalukoff | rm_work: Er... yes, I think. | 23:53 |
| rm_work | last time i had a big chain i just scripted it | 23:54 |
| sbalukoff | Please do take a look at the data_model Listener update method that I changed-- it turns out what I thought was a fix probably wasn't, so I have a *new* fix in there. | 23:54 |
| sbalukoff | Which still begs the question as to why this never happened before, which I don't know. | 23:54 |
| johnsom | I did notice my HMAC patch which is chained on your stuff failed with the same issue last night. | 23:55 |
| sbalukoff | I'm being paranoid and running tox locally before I submit stuff for review. | 23:55 |
| sbalukoff | Juuuust in case. | 23:55 |
| *** ducttape_ has joined #openstack-lbaas | 23:55 | |
| sbalukoff | johnsom: That is really odd! | 23:55 |
| sbalukoff | I mean, this whole system is set up to avoid this kind of problem. :/ | 23:56 |
| johnsom | Yeah, in theory at least | 23:56 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Add L7 controller worker flows and tasks https://review.openstack.org/277768 | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!