| *** Bjoern has joined #openstack-lbaas | 00:04 | |
| *** yuanying has quit IRC | 00:09 | |
| *** bdrich_ has joined #openstack-lbaas | 00:14 | |
| *** yuanying has joined #openstack-lbaas | 00:16 | |
| *** yuanying has quit IRC | 00:19 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia: Add SubjectAltName and KeyUsage for localcertgen https://review.openstack.org/275990 | 00:20 |
|---|---|---|
| *** ducttape_ has joined #openstack-lbaas | 00:21 | |
| *** yuanying has joined #openstack-lbaas | 00:22 | |
| *** Bjoern is now known as Bjoern_zZzZzZzZ | 00:24 | |
| *** longstaff has quit IRC | 00:24 | |
| *** Bjoern_zZzZzZzZ is now known as Bjoern | 00:29 | |
| *** bana_k has quit IRC | 00:36 | |
| *** chlong has joined #openstack-lbaas | 00:40 | |
| *** Bjoern has quit IRC | 00:41 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia: Add SubjectAltName and KeyUsage for localcertgen https://review.openstack.org/275990 | 00:44 |
| *** Aish has quit IRC | 01:05 | |
| rm_work | sbalukoff: happen to remember which bug this was? | 01:08 |
| johnsom | I will find it for you | 01:09 |
| rm_work | thanks johnsom | 01:09 |
| rm_work | I guess I could do that >_> | 01:09 |
| johnsom | https://bugs.launchpad.net/octavia/+bug/1494420 | 01:09 |
| openstack | Launchpad bug 1494420 in octavia "REST Driver TLS - Certificate has no `subjectAltName" [Low,Confirmed] | 01:09 |
| *** yuanying has quit IRC | 01:11 | |
| *** yuanying has joined #openstack-lbaas | 01:12 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia: Add SubjectAltName and KeyUsage for localcertgen https://review.openstack.org/275990 | 01:14 |
| openstackgerrit | min wang proposed openstack/octavia: Implements: blueprint anti-affinity server group https://review.openstack.org/272344 | 01:14 |
| *** minwang2 has quit IRC | 01:15 | |
| *** yuanying has quit IRC | 01:21 | |
| *** piet has joined #openstack-lbaas | 01:22 | |
| *** yuanying has joined #openstack-lbaas | 01:23 | |
| *** paco20151113 has joined #openstack-lbaas | 01:23 | |
| *** piet has quit IRC | 01:35 | |
| *** piet has joined #openstack-lbaas | 01:35 | |
| *** piet has quit IRC | 01:35 | |
| *** piet has joined #openstack-lbaas | 01:36 | |
| *** bharathm has quit IRC | 01:36 | |
| *** jaff_cheng has joined #openstack-lbaas | 01:48 | |
| *** yamamoto_ has joined #openstack-lbaas | 01:48 | |
| *** _cjones_ has quit IRC | 01:49 | |
| *** piet has quit IRC | 01:52 | |
| *** mixos has joined #openstack-lbaas | 01:55 | |
| *** mgarza_ has joined #openstack-lbaas | 01:55 | |
| *** yamamoto_ has quit IRC | 01:59 | |
| *** yamamoto_ has joined #openstack-lbaas | 02:00 | |
| johnsom | sbalukoff Ok, shared pools review complete. Sorry it took so long to get through. | 02:01 |
| *** ducttape_ has quit IRC | 02:03 | |
| *** johnsom has quit IRC | 02:06 | |
| *** yamamoto_ has quit IRC | 02:10 | |
| *** yamamoto_ has joined #openstack-lbaas | 02:10 | |
| sbalukoff | Thanks, johnsom! | 02:11 |
| sbalukoff | And no worries-- it's a huge patch that touches everything. | 02:11 |
| *** yamamoto_ has quit IRC | 02:13 | |
| *** mixos has quit IRC | 02:15 | |
| *** jwarendt has quit IRC | 02:16 | |
| sbalukoff | Lots of little stuff... I'll try to have these addressed sometime tonight or tomorrow, eh. | 02:20 |
| *** alejandrito has joined #openstack-lbaas | 02:20 | |
| *** piet has joined #openstack-lbaas | 02:21 | |
| *** ducttape_ has joined #openstack-lbaas | 02:26 | |
| *** minwang2 has joined #openstack-lbaas | 02:26 | |
| *** minwang2 has quit IRC | 02:29 | |
| *** Dave has quit IRC | 02:30 | |
| *** madhu_ak has quit IRC | 02:32 | |
| *** piet has quit IRC | 02:35 | |
| *** longstaff has joined #openstack-lbaas | 02:36 | |
| *** longstaff has quit IRC | 02:52 | |
| *** bana_k has joined #openstack-lbaas | 02:56 | |
| *** bana_k has quit IRC | 02:57 | |
| *** hang has joined #openstack-lbaas | 03:00 | |
| *** bdrich_ has quit IRC | 03:01 | |
| *** johnsom has joined #openstack-lbaas | 03:02 | |
| *** jaff_cheng has quit IRC | 03:03 | |
| sbalukoff | johnsom: Are you still around? | 03:09 |
| *** yamamoto_ has joined #openstack-lbaas | 03:09 | |
| *** yuanying has quit IRC | 03:15 | |
| *** allan_h has quit IRC | 03:16 | |
| *** alejandrito has quit IRC | 03:17 | |
| *** bdrich_ has joined #openstack-lbaas | 03:18 | |
| *** yuanying has joined #openstack-lbaas | 03:19 | |
| *** yuanying has quit IRC | 03:24 | |
| *** woodster_ has quit IRC | 03:26 | |
| *** yuanying has joined #openstack-lbaas | 03:35 | |
| *** piet has joined #openstack-lbaas | 03:41 | |
| *** yuanying_ has joined #openstack-lbaas | 03:41 | |
| *** yuanying has quit IRC | 03:44 | |
| *** woodster_ has joined #openstack-lbaas | 03:44 | |
| *** links has joined #openstack-lbaas | 03:49 | |
| blogan | made this bug https://bugs.launchpad.net/octavia/+bug/1541675 | 03:50 |
| openstack | Launchpad bug 1541675 in octavia "intermittent session persistence scenario test failure" [Undecided,New] | 03:50 |
| *** bdrich_ has quit IRC | 03:50 | |
| blogan | so whenever that scenario job fails bc of the session persistence, please do a recheck bug 1541675 | 03:50 |
| *** amotoki has joined #openstack-lbaas | 03:52 | |
| *** hang has quit IRC | 03:54 | |
| openstackgerrit | Phillip Toohill proposed openstack/octavia: Allow user-data on amphora creation https://review.openstack.org/275558 | 03:57 |
| *** ducttape_ has quit IRC | 03:57 | |
| *** ducttape_ has joined #openstack-lbaas | 04:04 | |
| *** minwang2 has joined #openstack-lbaas | 04:05 | |
| *** piet has quit IRC | 04:06 | |
| *** mgarza_ has quit IRC | 04:06 | |
| *** yuanying_ has quit IRC | 04:06 | |
| *** yuanying has joined #openstack-lbaas | 04:07 | |
| *** yuanying has quit IRC | 04:08 | |
| *** yuanying has joined #openstack-lbaas | 04:08 | |
| *** ctracey has quit IRC | 04:12 | |
| *** mhayden has quit IRC | 04:12 | |
| *** davidlenwell_ has joined #openstack-lbaas | 04:13 | |
| *** davidlenwell has quit IRC | 04:13 | |
| *** chlong has quit IRC | 04:13 | |
| *** mhayden has joined #openstack-lbaas | 04:15 | |
| *** ducttape_ has quit IRC | 04:16 | |
| *** ctracey has joined #openstack-lbaas | 04:18 | |
| *** pai15- has quit IRC | 04:19 | |
| *** amit213 has quit IRC | 04:19 | |
| *** bradjones has quit IRC | 04:19 | |
| *** cgross has quit IRC | 04:19 | |
| openstackgerrit | Phillip Toohill proposed openstack/octavia: Allow user-data on amphora creation https://review.openstack.org/275558 | 04:19 |
| *** pai15 has joined #openstack-lbaas | 04:20 | |
| *** cgross has joined #openstack-lbaas | 04:20 | |
| *** bradjones has joined #openstack-lbaas | 04:21 | |
| *** bradjones has quit IRC | 04:21 | |
| *** bradjones has joined #openstack-lbaas | 04:21 | |
| *** chlong has joined #openstack-lbaas | 04:28 | |
| *** minwang2 has quit IRC | 04:39 | |
| openstackgerrit | German Eichberger proposed openstack/octavia: Replace depcrecated Nova networks with Nova interfaces https://review.openstack.org/273733 | 04:39 |
| *** Aish has joined #openstack-lbaas | 04:41 | |
| *** Aish has quit IRC | 04:42 | |
| *** minwang2 has joined #openstack-lbaas | 04:43 | |
| *** Aish has joined #openstack-lbaas | 04:43 | |
| *** Aish has quit IRC | 04:43 | |
| *** fnaval has quit IRC | 05:01 | |
| *** davidlenwell_ has quit IRC | 05:02 | |
| *** davidlenwell has joined #openstack-lbaas | 05:05 | |
| *** minwang2 has quit IRC | 05:06 | |
| *** fnaval has joined #openstack-lbaas | 05:18 | |
| *** piet has joined #openstack-lbaas | 05:19 | |
| *** minwang2 has joined #openstack-lbaas | 05:23 | |
| rm_work | blogan: so is there a session persistence scenario test that is perma-failing? I noticed I don't think i've seen the scenario tests pass in a long time <_< | 05:27 |
| sbalukoff | Hey folks... | 05:58 |
| rm_work | ah hey one passed | 05:58 |
| rm_work | hey sbalukoff | 05:58 |
| sbalukoff | I'm about at my wits end with this damned pecan app... I want to replace calls to SimulatedControllerHandler to a new one I wrote called SimulatedBadControllerHandler so that I can simulate a handler error (as johnsom is requesting) | 05:59 |
| sbalukoff | I can't figure out how the SimulatedControllerHandler is getting called in the first place. | 06:00 |
| *** crc32 has joined #openstack-lbaas | 06:00 | |
| sbalukoff | And I'm pretty sure mock hates me. | 06:00 |
| rm_work | mock hates everyone :P | 06:00 |
| sbalukoff | (This is Octavia, obviously.) | 06:00 |
| *** piet has quit IRC | 06:00 | |
| sbalukoff | So among other things: How the hell is the api deciding to look for the SimulatedControllerHandler in the first place? | 06:01 |
| rm_work | isn't it part of config? | 06:02 |
| rm_work | octavia.api.handlers = | 06:02 |
| rm_work | simulated_handler = octavia.api.v1.handlers.controller_simulator.handler:SimulatedControllerHandler | 06:02 |
| rm_work | queue_producer = octavia.api.v1.handlers.queue.producer:ProducerHandler | 06:02 |
| *** numans has joined #openstack-lbaas | 06:02 | |
| rm_work | the default config is "simulated_handler" | 06:03 |
| sbalukoff | Where did you find that? | 06:03 |
| rm_work | setup.cfg | 06:03 |
| rm_work | is where all the stevedore driver stuff is defined | 06:03 |
| sbalukoff | ... dammit! | 06:03 |
| rm_work | :P | 06:03 |
| rm_work | glad to be of assistance | 06:03 |
| rm_work | so yeah, add your line there, and do a config override in your test | 06:04 |
| sbalukoff | There's not a simple way to replace any calls to the SimulatedControllerHandler using mock? | 06:04 |
| rm_work | err | 06:04 |
| rm_work | I mean | 06:04 |
| rm_work | the normal way mocks work? | 06:04 |
| sbalukoff | Well, using patch, I mean. | 06:04 |
| rm_work | patch should work | 06:05 |
| sbalukoff | Ok, for the life of me I can't figure out how to do that. | 06:05 |
| sbalukoff | I have tried... may permutations. | 06:05 |
| sbalukoff | many | 06:05 |
| sbalukoff | I think stevedore might be fucking with me... it might just be easier to define a new controller handler. :P | 06:06 |
| sbalukoff | So, I'll give that a try. | 06:06 |
| rm_work | @mock.patch("octavia.api.v1.handlers.controller_simulator.handler.SimulatedControllerHandler.whatever") ? I think, or something like that | 06:06 |
| rm_work | man that's long | 06:07 |
| sbalukoff | With a "new=......SimulatedBadControllerHandler.whatever" option as well? | 06:07 |
| rm_work | err, that passes the mock into the test as an arg | 06:07 |
| sbalukoff | I've got to replace class methods? I can't replace the whole class? | 06:07 |
| rm_work | err | 06:07 |
| rm_work | I guess maybe you could | 06:08 |
| sbalukoff | Yeah. | 06:08 |
| rm_work | why would you need to? | 06:08 |
| rm_work | maybe i don't get what you're testing | 06:08 |
| rm_work | wouldn't you be testing specific bad methods or something? | 06:08 |
| sbalukoff | Well.... I'll give that a shot, but if that doesn't work, I guess I'll just define a new controller handler which returns errors (for testing controller handler errors) | 06:08 |
| rm_work | you can patch as many methods as you want | 06:08 |
| rm_work | just remember the order of the args is bottom-to-top | 06:09 |
| rm_work | ... i think | 06:09 |
| sbalukoff | Wait, the order of the arges is bottom-to-top? | 06:09 |
| rm_work | i think so | 06:09 |
| rm_work | i remember it's odd | 06:09 |
| sbalukoff | Ok... | 06:09 |
| rm_work | ah so you can mock the whole class | 06:10 |
| rm_work | @mock.patch("octavia.api.v1.handlers.controller_simulator.handler.SimulatedControllerHandler") | 06:10 |
| sbalukoff | I'm trying to test a code path that isn't being unit tested right now: It's what the API does when the handler returns an error. | 06:10 |
| rm_work | def test(self, mockedClass) | 06:10 |
| sbalukoff | Mind you-- the original code didn't test any of this shit either. | 06:10 |
| rm_work | should make mockedClass the mock for *the class* | 06:10 |
| rm_work | (redundant explanation) | 06:10 |
| sbalukoff | johnsom is asking me to solve some technical debt at the same time as the massive shared_pools patch. | 06:10 |
| rm_work | lol | 06:10 |
| rm_work | sounds about right | 06:11 |
| sbalukoff | Yeah. I'm a little upset. | 06:11 |
| sbalukoff | Especially because I'm pretty sure nobody tested that before because it's fucking hard. | 06:11 |
| rm_work | I rewrote most of python-barbicanclient because I needed consumer registration to work for containers <_< | 06:11 |
| rm_work | these things happen | 06:12 |
| rm_work | yeah there's certain stuff that is just ... weird to test with mocks | 06:12 |
| rm_work | wait until you try mocking db cursors that are being used as context managers | 06:13 |
| rm_work | a context manager returning an iterator | 06:13 |
| rm_work | funtimes | 06:13 |
| sbalukoff | Blah. | 06:13 |
| rm_work | does @mock.patch("octavia.api.v1.handlers.controller_simulator.handler.SimulatedControllerHandler") work? | 06:14 |
| rm_work | AFAICT it should | 06:14 |
| rm_work | ah yeah: | 06:15 |
| rm_work | Note When you nest patch decorators the mocks are passed in to the decorated function in the same order they applied (the normal python order that decorators are applied). This means from the bottom up | 06:15 |
| sbalukoff | Well... the problem is that a I'm not calling those method directly. I want the pecan app that gets built from the api definition to call those methods. | 06:15 |
| rm_work | which tests are you messing with | 06:16 |
| rm_work | i'll pull it uo | 06:16 |
| rm_work | *up | 06:16 |
| sbalukoff | I mean, it's seriously bizarre to me that octavia/tests/functional/api/v1/base.py ... the setUp method there.. even works. | 06:17 |
| sbalukoff | I'm messing with stuff in octavia/tests/functional/api/v1/test_health_monitor.py | 06:17 |
| rm_work | ah lol | 06:17 |
| rm_work | that patcher | 06:17 |
| sbalukoff | yeah. | 06:17 |
| sbalukoff | Trying to figure out WTF it's actually doing. | 06:17 |
| rm_work | I'm surprised that reference isn't saved as self.patcher or something | 06:18 |
| rm_work | if it were you could access it | 06:18 |
| sbalukoff | It doesn't need to be: The mock is closed after the app is built. | 06:18 |
| openstackgerrit | Phillip Toohill proposed openstack/octavia: Allow user-data on amphora creation https://review.openstack.org/275558 | 06:18 |
| rm_work | err, on *cleanup*, not immediately | 06:19 |
| rm_work | I thought | 06:19 |
| rm_work | it's a post-test action | 06:19 |
| rm_work | though the way this is done seems really odd | 06:19 |
| sbalukoff | I tried changing it to self.patcher variable and then messing with it... didn't seem to work. | 06:19 |
| rm_work | why not decorate and keep it cleanly scoped | 06:19 |
| rm_work | let me see... | 06:20 |
| rm_work | oh you need to store the result of | 06:20 |
| rm_work | patcher.start() | 06:20 |
| *** armax has quit IRC | 06:20 | |
| rm_work | that's the actual MagicMock | 06:21 |
| sbalukoff | Ok. | 06:21 |
| rm_work | do like... | 06:21 |
| rm_work | self.controller_mock = patcher.start() | 06:21 |
| rm_work | on line 47 here | 06:21 |
| sbalukoff | Ok, but I'm not sure what to do with it after that. | 06:21 |
| rm_work | err | 06:21 |
| rm_work | then from whatever test method, just... | 06:21 |
| rm_work | step up the mock | 06:22 |
| rm_work | *set up | 06:22 |
| rm_work | the way you need it | 06:22 |
| sbalukoff | Could you give me an example. | 06:22 |
| sbalukoff | I really don't understand what you mean when you say 'setup the mock' | 06:22 |
| rm_work | it's a mock, do whatever you want | 06:22 |
| sbalukoff | I've been trying to do that all day and am finding the documentation on this extremely obtuse. | 06:22 |
| rm_work | so like | 06:22 |
| rm_work | if you want a specific function on the class to raise something | 06:22 |
| rm_work | do like | 06:23 |
| rm_work | self.controller_mock.somefunction.side_effect = Exception() | 06:23 |
| rm_work | it's a magicmock, so, whatever attribute you hit on the mock, turns into a new mock object | 06:24 |
| rm_work | that you can do mock stuff with | 06:24 |
| sbalukoff | self.controller_mock = patcher.start() ? | 06:24 |
| rm_work | yeah, line 47 | 06:24 |
| rm_work | then in your test method, you should be able to hit self.controller_mock to do stuff to it | 06:25 |
| rm_work | and then you want to reset it at the end of the test prolly | 06:25 |
| sbalukoff | Ok... I think I understand.. | 06:25 |
| rm_work | self.controller_mock.reset() | 06:25 |
| *** crc32 has quit IRC | 06:25 | |
| rm_work | I think | 06:25 |
| sbalukoff | Ok, I'll give it a shot. | 06:25 |
| sbalukoff | Thanks for your help on this. | 06:25 |
| rm_work | nope reset does nothing lol | 06:25 |
| rm_work | but anywho | 06:26 |
| rm_work | ah | 06:26 |
| rm_work | self.controller_mock.reset_mock() | 06:26 |
| sbalukoff | Ok! | 06:27 |
| rm_work | ehhh nm don't do that either, doesn't QUITE do what I wanted | 06:27 |
| rm_work | bleh mocks are lulzy | 06:27 |
| rm_work | umm just like... | 06:27 |
| rm_work | if you set a side_effect | 06:27 |
| rm_work | unset it | 06:27 |
| rm_work | as soon as you're done :P | 06:27 |
| sbalukoff | Ok.... I'll try to figure out how to do that. | 06:28 |
| rm_work | self.controller_mock.myfunc.side_effect = None | 06:28 |
| rm_work | gleh really this mock just shouldn't be set like this | 06:28 |
| rm_work | but whatever | 06:28 |
| rm_work | wait is setUp called once for EACH test? | 06:28 |
| rm_work | let me check | 06:29 |
| rm_work | yeah nevermind that whole thing | 06:29 |
| rm_work | do whatever you want | 06:29 |
| rm_work | it resets anyway | 06:29 |
| rm_work | it is late and I am overthinking | 06:29 |
| rm_work | sbalukoff: ^^ it's easier than I thought, just do whatever you want and it'll work itself out :P | 06:30 |
| sbalukoff | Ok... I'm trying the side_effect thing... | 06:30 |
| rm_work | yeah you can give it whatever exception you want | 06:31 |
| rm_work | OR if you give it an iterable type, it'll return the items one by one | 06:31 |
| rm_work | so it changes on every call! woo | 06:31 |
| rm_work | that's how you mock iterables | 06:31 |
| rm_work | mymock.__iter__.side_effect = [1,2,3] | 06:31 |
| rm_work | http://cdn.theatlantic.com/assets/media/img/mt/2014/09/The_More_You_Know/lead_large.png | 06:32 |
| sbalukoff | self.controller_mock.health_monitor.create.side_effect = exceptions.APIException() | 06:33 |
| sbalukoff | Doesn't appear to be working. | 06:33 |
| sbalukoff | Though, I'm still not sure if I'm referencing the method call correctly. | 06:33 |
| rm_work | ahh so | 06:34 |
| rm_work | if it's something that's CALLED | 06:34 |
| rm_work | you may need () | 06:34 |
| rm_work | self.controller_mock.health_monitor.create().side_effect = exceptions.APIException() | 06:34 |
| rm_work | or possibly | 06:34 |
| rm_work | self.controller_mock.health_monitor().create.side_effect = exceptions.APIException() | 06:34 |
| rm_work | depending on what those are | 06:34 |
| rm_work | mocks are fun! | 06:34 |
| rm_work | are you using an IDE? | 06:35 |
| rm_work | debugging helps *a lot* | 06:35 |
| sbalukoff | I'm not presently. | 06:35 |
| sbalukoff | Never got around to setting up pycharm. :P | 06:35 |
| rm_work | T_T | 06:35 |
| rm_work | pycharm rocks | 06:35 |
| rm_work | best thing since sliced pie | 06:36 |
| sbalukoff | ... | 06:36 |
| sbalukoff | Or it may be that the original code doesn't return an error if the handler flakes. | 06:36 |
| rm_work | lol | 06:36 |
| sbalukoff | So, I may have been chasing my tail the whole time. | 06:36 |
| rm_work | that could also be it, who knows :P | 06:36 |
| *** woodster_ has quit IRC | 06:36 | |
| rm_work | what test are you in again? | 06:36 |
| sbalukoff | I'm trying to test the post method of octavia/api/v1/controllers/health_monitor.py | 06:39 |
| sbalukoff | Specifically, toward the end-- I want to pretend a handler error occurs. | 06:39 |
| rm_work | hmm | 06:39 |
| rm_work | can you gist whatever test code you have now? | 06:40 |
| fnaval | do I need to create a network and port and subnet before I create a load balancer in octavia? similar to neutron-lbaas? | 06:40 |
| sbalukoff | Sure... hold on... | 06:40 |
| *** bana_k has joined #openstack-lbaas | 06:40 | |
| rm_work | fnaval: if you don't, octavia will create one for you | 06:40 |
| fnaval | that is, if I want to spin up 2 servers to interact with it | 06:40 |
| rm_work | fnaval: well, a port | 06:40 |
| rm_work | you need a network and subnet | 06:40 |
| fnaval | ok cool thanks rm_work - i was hoping I could rip that code out; but I'll keep it. | 06:41 |
| sbalukoff | https://gist.github.com/anonymous/82679495dbbbeab284cf | 06:41 |
| sbalukoff | That's as far as I've gotten-- The problem is that call succeeds (I get a 202 accepted code) | 06:42 |
| *** bana_k has quit IRC | 06:45 | |
| rm_work | yeah you want health_monitor.create().side_effect | 06:45 |
| rm_work | I think | 06:45 |
| rm_work | oh you have that | 06:45 |
| rm_work | hmm | 06:46 |
| rm_work | debugging my way through | 06:47 |
| sbalukoff | I really appreciate the help! | 06:48 |
| rm_work | ok so | 06:49 |
| rm_work | self.controller_mock().health_monitor.create.side_effect = exceptions.APIException() | 06:49 |
| rm_work | I THINK | 06:49 |
| rm_work | but | 06:50 |
| rm_work | even though that is correct | 06:50 |
| rm_work | your code still 202s | 06:50 |
| rm_work | lol | 06:50 |
| rm_work | digging into that | 06:50 |
| sbalukoff | Let me try exactly what you've written. | 06:51 |
| *** minwang2 has quit IRC | 06:51 | |
| sbalukoff | Yeah... didn't work. :P | 06:51 |
| sbalukoff | I should really get pycharm going, I suppose. :P | 06:52 |
| rm_work | almost there | 06:52 |
| *** eranra has quit IRC | 06:52 | |
| rm_work | ok so it totally gets the exception | 06:53 |
| rm_work | goes to the except block | 06:53 |
| rm_work | ... does a listener update and moves on | 06:53 |
| rm_work | octavia/api/v1/controllers/health_monitor.py line 102 | 06:53 |
| rm_work | the mock works! :P | 06:54 |
| sbalukoff | So, it's actually the original code? | 06:54 |
| rm_work | yes | 06:54 |
| sbalukoff | Er... sorry... the... yeah. | 06:55 |
| rm_work | but that IS the way you need the mock | 06:55 |
| rm_work | self.controller_mock().health_monitor.create.side_effect = exceptions.APIException() | 06:55 |
| rm_work | mocks are FUN! :P | 06:55 |
| sbalukoff | :P | 06:55 |
| rm_work | you can tell from debugging and watching the calls that show uo | 06:55 |
| rm_work | *up | 06:55 |
| sbalukoff | Thanks, man. | 06:55 |
| rm_work | np | 06:55 |
| rm_work | yeah you can see that it does nothing here with the exception | 06:56 |
| rm_work | well, i'm not on your patchset actually | 06:56 |
| sbalukoff | ... | 06:56 |
| rm_work | with excutils.save_and_reraise_exception(reraise=False): | 06:56 |
| sbalukoff | It's because reraise=False. | 06:56 |
| rm_work | ah, it sets the status to error, yeah | 06:56 |
| sbalukoff | In that call. | 06:56 |
| rm_work | but yes | 06:56 |
| rm_work | is that desired functionality? | 06:57 |
| sbalukoff | Wow. | 06:57 |
| rm_work | i think it might be | 06:57 |
| sbalukoff | No, it's not. | 06:57 |
| rm_work | don't 500 | 06:57 |
| rm_work | just set status to ERROR | 06:57 |
| sbalukoff | Don't 500? | 06:57 |
| rm_work | no? | 06:57 |
| sbalukoff | Ok. | 06:57 |
| rm_work | it's an internal processing failure | 06:57 |
| rm_work | not something the user can fix, right? | 06:57 |
| sbalukoff | Aah... right. | 06:57 |
| rm_work | so you're not looking for a 500 | 06:58 |
| rm_work | you're looking for the listener to be in ERROR | 06:58 |
| sbalukoff | That's right. | 06:59 |
| sbalukoff | Thank you VERY much with this. | 06:59 |
| sbalukoff | I think I've got it from here. | 06:59 |
| rm_work | yeah np | 07:00 |
| rm_work | i credit pycharm | 07:00 |
| *** bana_k has joined #openstack-lbaas | 07:01 | |
| sbalukoff | I really need to get pycharm working, apparently! | 07:02 |
| openstackgerrit | Reedip proposed openstack/neutron-lbaas: Make subnet_id for lbaas v2 member create optional https://review.openstack.org/267935 | 07:04 |
| fnaval | sbalukoff: btw, that PR that i'm working on is going to explode to a few more thousand lines for all of the required libs for setting up the network, subnet and port for the octavia load balancer test. | 07:05 |
| fnaval | sbalukoff: would that still be ok? | 07:06 |
| sbalukoff | fnaval: Ouch, I'm really sorry... Um... maybe make that a dependent patch set? | 07:06 |
| sbalukoff | But if it can't be helped, it can't be helped, eh. | 07:06 |
| sbalukoff | :/ | 07:06 |
| sbalukoff | Are these libs being copied from the tempest project? | 07:06 |
| sbalukoff | Is it possible to just reference the other project and not copy the code itself? | 07:07 |
| fnaval | sbalukoff: some are but i'm trying to use as much from tempest_lib as I can | 07:08 |
| sbalukoff | Is the problem that tempest_lib is just not complete enough? | 07:08 |
| fnaval | but there are some classes that need to be created that inherits stuff from tempest_lib | 07:08 |
| fnaval | sbalukoff: correct - tempest_lib just provides a very base skeleton | 07:09 |
| sbalukoff | I'd really like to avoid the situation we have with neutron-lbaas-- where copying huge chunks of code from the tempest project has led to a situation where our gate is being held together with duct-tape and baling wire... | 07:09 |
| sbalukoff | Do you understand what I mean by that? | 07:09 |
| fnaval | sbalukoff: yes, I saw a lot of that in neutron-lbaas - i'm sifting through the code and only bringing the code that's only needed | 07:10 |
| sbalukoff | (ie. nobody ever updates the neutron-lbaas code that's been copied wholesale from tempest... so as the tempest that the gate uses moves on, our tempest tests break all over the place.) | 07:10 |
| fnaval | sbalukoff: when you siad to just reference the other project, did you mean reference neutron-lbaas? | 07:11 |
| fnaval | would octavia always run with neutron-lbaas? or should it be stand alone? | 07:11 |
| sbalukoff | No, I meant to reference tempest. | 07:12 |
| fnaval | I think that folks wanted to move off of tempest and just use tempest_lib | 07:12 |
| sbalukoff | Which might accomplish what I'm after. | 07:12 |
| sbalukoff | But if it's the case that, as you say, tempest_lib isn't complete enough... when we might have to import code from the tempest project? | 07:13 |
| fnaval | well, I'll probably submit a patch set tomorrow and everyone can review. I'll try to refactor as much as I can to make it less spaghetti-ish. | 07:13 |
| sbalukoff | I honestly don't know enough about it to say for sure. | 07:13 |
| fnaval | I haven't gotten it working yet - but I feel like I'm close. | 07:13 |
| sbalukoff | Other than I hate it when we have crufty code we didn't write lying around because we copied it wholesale from another project. :P | 07:13 |
| fnaval | yea I see it in neutron-lbaas tempest directory | 07:14 |
| fnaval | lots of unused classes; e.g., to Swift, ObjectStorage, etc. | 07:14 |
| sbalukoff | fnaval: I appreciate your hard work. I really do! I look forward to being able to run tempest tests against Octavia directly-- it's going to become increasingly necessary as our possible topologies get more complicated. (We really ought to have it now with the active-standby code... but alas we don't.) | 07:14 |
| fnaval | thanks, yeah i'm looking forward to getting this all working | 07:15 |
| fnaval | once this base stuff gets in, creating tests would be easy | 07:16 |
| blogan | holy crap this is a long scrollback | 07:18 |
| sbalukoff | blogan: it's mostly filled with my frustrations around mock. | 07:19 |
| *** kobis has joined #openstack-lbaas | 07:19 | |
| sbalukoff | But it turns out that rm_work is awesome and figured it out for me. | 07:19 |
| blogan | sbalukoff: that never ends | 07:19 |
| blogan | but it has gotten easier | 07:19 |
| blogan | sbalukoff: by complimenting rm_work you have now initiated the end of humanity | 07:19 |
| sbalukoff | Well, once you know what to do, it all becomes easier! | 07:19 |
| sbalukoff | But the docs around mock are pretty obtuse to someone who's not done a whole lot with it before. | 07:20 |
| sbalukoff | blogan: It was a risk I had to take-- he has earned it. | 07:20 |
| blogan | yeah, you kind of have to learn how to read the mock docs too and then they start making sense | 07:20 |
| sbalukoff | I'm sure I'll find some way to insult him tomorrow, which should hopefully stop The End before too many continents are destroyed. | 07:20 |
| blogan | like they don't go too in depth on non decorator mocking, like diong a patch.start() | 07:21 |
| sbalukoff | Yeah. | 07:21 |
| blogan | i mean they do on decorator, not the non decorator way | 07:21 |
| sbalukoff | I knew what you meant. | 07:21 |
| rm_work | the decorators are generally the way to go, though | 07:26 |
| rm_work | I'm still not sure exactly why you had to do the patcher.start thing there | 07:26 |
| rm_work | but it ... seems to work | 07:27 |
| sbalukoff | Neither am I... | 07:27 |
| sbalukoff | But yes, it works... | 07:27 |
| *** eranra has joined #openstack-lbaas | 07:28 | |
| rm_work | I can ALMOST recall when blogan was working on that to begin with | 07:29 |
| rm_work | I am pretty sure it was him | 07:29 |
| *** nmagnezi has joined #openstack-lbaas | 07:30 | |
| sbalukoff | Haha | 07:31 |
| rm_work | I really wonder if I could ... fix it now | 07:33 |
| rm_work | to not need that | 07:33 |
| sbalukoff | Um.... ? | 07:33 |
| * rm_work tries | 07:34 | |
| sbalukoff | Good luck! | 07:34 |
| rm_work | OH GOD EVERYTHING IS BREAKING I'M SO SORRY I REGRET EVERYTHING | 07:35 |
| sbalukoff | HAHA | 07:35 |
| sbalukoff | It's like sticking your finger in the light socket... | 07:35 |
| *** Alex_Stef has joined #openstack-lbaas | 07:43 | |
| rm_work | yeah so it looks like a lot of that is actually unnecessary | 07:46 |
| rm_work | the patcher.stop() doesn't seem to affect anything | 07:46 |
| rm_work | ah i see why maybe? | 07:51 |
| rm_work | eh | 07:51 |
| rm_work | it explains all this here: https://docs.python.org/3/library/unittest.mock.html#patch-methods-start-and-stop | 07:52 |
| rm_work | but i still don't quite get why stop() matters | 07:52 |
| rm_work | tests are run->end | 07:52 |
| rm_work | GC will get anything that was leftover | 07:52 |
| rm_work | and if you patch in the right namespace it's pretty safe | 07:52 |
| rm_work | ah but he's trying to patch globally <_< | 07:53 |
| rm_work | alright, this has been an episode of "watch rm_work figure out what blogan figured out over a year ago | 07:54 |
| rm_work | " | 07:54 |
| rm_work | and with that, i'm going to pass out. night all. | 07:55 |
| rm_work | good luck sbalukoff | 07:55 |
| sbalukoff | Thanks, rm_work! | 07:55 |
| blogan | reason i do the patcher is bc the decorators become a mess sometimes, and just being able to do it in a setup and cleanup makes it easier and less messy, but decorators are easier | 07:58 |
| sbalukoff | Less boilerplate code. | 08:01 |
| *** jschwarz has joined #openstack-lbaas | 08:37 | |
| *** _cjones_ has joined #openstack-lbaas | 08:41 | |
| *** _cjones_ has quit IRC | 08:42 | |
| *** _cjones_ has joined #openstack-lbaas | 08:43 | |
| *** _cjones_ has quit IRC | 09:00 | |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** openstackgerrit has joined #openstack-lbaas | 09:18 | |
| *** numans has quit IRC | 09:23 | |
| *** Dave has joined #openstack-lbaas | 09:33 | |
| openstackgerrit | Evgeny Fedoruk proposed openstack/neutron-lbaas: L7 capability extension implementation for lbaas v2 https://review.openstack.org/148232 | 09:40 |
| openstackgerrit | Brandon Logan proposed openstack/octavia: Adding network driver for a vip on a single amp https://review.openstack.org/197858 | 09:50 |
| *** mugsie has quit IRC | 10:39 | |
| *** mugsie has joined #openstack-lbaas | 10:40 | |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Shared pools support https://review.openstack.org/256369 | 10:53 |
| *** yamamoto_ has quit IRC | 11:29 | |
| *** rtheis has joined #openstack-lbaas | 12:12 | |
| *** numans has joined #openstack-lbaas | 12:24 | |
| *** cgross has quit IRC | 12:48 | |
| *** numan_ has joined #openstack-lbaas | 12:52 | |
| *** bank_ has joined #openstack-lbaas | 12:52 | |
| *** mhayden has quit IRC | 12:53 | |
| *** numans has quit IRC | 12:53 | |
| *** bana_k has quit IRC | 12:53 | |
| *** mhayden has joined #openstack-lbaas | 12:57 | |
| *** cgross has joined #openstack-lbaas | 13:04 | |
| *** ducttape_ has joined #openstack-lbaas | 13:11 | |
| *** localloop127 has joined #openstack-lbaas | 13:18 | |
| *** ducttape_ has quit IRC | 13:27 | |
| *** yamamoto has joined #openstack-lbaas | 13:31 | |
| *** liamji has joined #openstack-lbaas | 13:36 | |
| *** yamamoto has quit IRC | 13:43 | |
| *** longstaff has joined #openstack-lbaas | 13:44 | |
| openstackgerrit | venkatamahesh proposed openstack/octavia: Fix misspellings https://review.openstack.org/276240 | 13:52 |
| openstackgerrit | venkatamahesh proposed openstack/octavia: Use uppercase 'S' in word "OpenStack" https://review.openstack.org/276245 | 13:55 |
| *** links has quit IRC | 13:56 | |
| openstackgerrit | Evgeny Fedoruk proposed openstack/neutron-lbaas: L7 capability extension implementation for lbaas v2 https://review.openstack.org/148232 | 14:00 |
| *** barra204 has quit IRC | 14:04 | |
| *** shakamunyi has joined #openstack-lbaas | 14:06 | |
| *** paco20151113 has quit IRC | 14:06 | |
| *** longstaff has quit IRC | 14:10 | |
| *** neelashah has joined #openstack-lbaas | 14:22 | |
| *** nmagnezi_ has joined #openstack-lbaas | 14:25 | |
| *** nmagnezi has quit IRC | 14:28 | |
| openstackgerrit | venkatamahesh proposed openstack/octavia: Fix misspellings https://review.openstack.org/276240 | 14:33 |
| *** bdrich_ has joined #openstack-lbaas | 14:35 | |
| *** Bjoern has joined #openstack-lbaas | 14:42 | |
| *** Alex_Stef has quit IRC | 14:45 | |
| *** piet has joined #openstack-lbaas | 14:48 | |
| *** TrevorV has joined #openstack-lbaas | 14:49 | |
| *** ducttape_ has joined #openstack-lbaas | 14:53 | |
| *** longstaff has joined #openstack-lbaas | 14:58 | |
| *** doug-fish has joined #openstack-lbaas | 15:03 | |
| *** bdrich_ has quit IRC | 15:06 | |
| *** nmagnezi_ has quit IRC | 15:09 | |
| *** nmagnezi_ has joined #openstack-lbaas | 15:20 | |
| *** rtheis has quit IRC | 15:25 | |
| *** rtheis has joined #openstack-lbaas | 15:28 | |
| *** mgarza has joined #openstack-lbaas | 15:29 | |
| openstackgerrit | Trevor Vardeman proposed openstack/octavia: Amphora create will add a 'fake' heartbeat on creation https://review.openstack.org/271512 | 15:29 |
| *** woodster_ has joined #openstack-lbaas | 15:29 | |
| *** bdrich_ has joined #openstack-lbaas | 15:30 | |
| *** Alex_Stef has joined #openstack-lbaas | 15:43 | |
| *** fnaval has quit IRC | 15:45 | |
| *** Aish has joined #openstack-lbaas | 15:49 | |
| *** jwarendt has joined #openstack-lbaas | 15:50 | |
| *** piet has quit IRC | 15:55 | |
| TrevorV | johnsom when you jump online later, ping me. I'm not quite sure what's going wrong here. | 15:57 |
| *** bdrich_ has quit IRC | 15:58 | |
| *** piet has joined #openstack-lbaas | 16:01 | |
| *** fnaval has joined #openstack-lbaas | 16:02 | |
| *** liamji has quit IRC | 16:11 | |
| *** liamji has joined #openstack-lbaas | 16:12 | |
| *** jwarendt has quit IRC | 16:14 | |
| *** jwarendt has joined #openstack-lbaas | 16:14 | |
| *** piet has quit IRC | 16:19 | |
| *** jschwarz has quit IRC | 16:22 | |
| *** nmagnezi_ has quit IRC | 16:22 | |
| *** yamamoto has joined #openstack-lbaas | 16:23 | |
| *** mixos has joined #openstack-lbaas | 16:23 | |
| *** allan_h has joined #openstack-lbaas | 16:23 | |
| *** _ducttape_ has joined #openstack-lbaas | 16:26 | |
| *** yamamoto has quit IRC | 16:27 | |
| *** ducttape_ has quit IRC | 16:29 | |
| *** kobis has quit IRC | 16:33 | |
| *** piet has joined #openstack-lbaas | 16:33 | |
| johnsom | TrevorV ping | 16:34 |
| johnsom | Reading the scroll back. It looks like sbalukoff had the same fun with Mock as I did. It does get easier though as you do more of it | 16:34 |
| TrevorV | Ha ha ha, mock is the bane of my sanity | 16:36 |
| johnsom | TrevorV I suspect that by putting the timestamp in amphora_health at that point is starting the clock too early. It looks like the failover timer expired on those before they were up | 16:36 |
| *** liamji has quit IRC | 16:36 | |
| TrevorV | Yeah, I agree, except when I ran a create on "master" it updated super early... just about the same time I spoofed the heartbeat... | 16:36 |
| johnsom | I see you had the same though in the latest patch | 16:36 |
| TrevorV | And it NEVER updated the heartbeat entry even though it should have been reporting a heartbeat. | 16:36 |
| TrevorV | I admit, I've had a hard time trying to find where the heartbeat is updating in the DB from the amphora, so I can't really trouble shoot just yet | 16:37 |
| johnsom | Looking for the code that updates the DB? | 16:38 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/update_health_mixin.py#L83 | 16:39 |
| blogan | johnsom, TrevorV: would it be more appropriate if the LB didn't go active until the heartbeat started coming in? | 16:44 |
| TrevorV | blogan maybe. johnsom thoughts? | 16:45 |
| blogan | problem with that is the lb won't go active unless the o-hm is running | 16:46 |
| blogan | so probably nto a good idea at this point | 16:46 |
| johnsom | It seems like we want to catch and failover amps that aren't sending heartbeat. | 16:46 |
| TrevorV | So have a task that has a wait/timeout for an amphora_health entry, right? | 16:46 |
| johnsom | Yeah, that is an issue too | 16:46 |
| johnsom | Yes | 16:46 |
| *** armax has joined #openstack-lbaas | 16:46 | |
| TrevorV | blogan are you WFH today? | 16:46 |
| blogan | i'd just hate to have to have some wait time that can be nondeterministic | 16:46 |
| *** jwarendt has quit IRC | 16:47 | |
| blogan | TrevorV: before lunch yes | 16:47 |
| TrevorV | blogan you on internal IRC? | 16:47 |
| blogan | yeah | 16:47 |
| *** _ducttape_ has quit IRC | 16:53 | |
| *** ducttape_ has joined #openstack-lbaas | 16:54 | |
| *** crc32 has joined #openstack-lbaas | 16:57 | |
| rm_work | TrevorV: you should probably rebase onto the EventStreamer patch actually >_< | 16:57 |
| rm_work | TrevorV: that changes a ton of the update_health stuff T_T | 16:57 |
| rm_work | (fixes it from being really weird) | 16:57 |
| rm_work | just realized that's where you were touching the code | 16:57 |
| TrevorV | Well, I didn't touch the event streamer, but yeah. | 16:58 |
| TrevorV | I just insert an entry in the DB right now | 16:58 |
| rm_work | right but it redoes the health_mixin stuff | 16:59 |
| rm_work | it's going to cause some merge conflicts | 16:59 |
| rm_work | if you're touching anything there | 16:59 |
| TrevorV | I still don't use that is my point. | 16:59 |
| TrevorV | I have a db task created | 16:59 |
| TrevorV | That's all | 16:59 |
| rm_work | ah k | 16:59 |
| TrevorV | Yeah | 17:00 |
| johnsom | Yeah, no conflict | 17:00 |
| johnsom | TrevorV I think it is just a timing thing. If this change doesn't work, we can either forward date your initial timestamp with some config offset or increase the health check window. I would lean towards the offset myself, if we need to go there | 17:03 |
| johnsom | Sadly zuul is a little busy again today | 17:03 |
| TrevorV | Well so that's what I'm saying johnsom I was just testing both master and my change, and my change NEVER picks up a new heartbeat. | 17:04 |
| rm_work | yeah, might want a configurable offset for the first one | 17:04 |
| rm_work | so if your environment is sucky you can up it | 17:04 |
| johnsom | Is the controller_ip_port_list config setting set in you test env? | 17:05 |
| johnsom | If that isn't set at amp build time, they will never call home | 17:05 |
| TrevorV | Oh, sorry, I was unclear | 17:05 |
| TrevorV | I get heartbeats via master when I run o-hm, o-cw, and o-api off that code base, but I never get a heartbeat on MY code change... | 17:06 |
| johnsom | It's still too early for me, so no worries | 17:06 |
| TrevorV | I get my fake one, but the amp never reports a new one as far as I can tell | 17:06 |
| blogan | i think we'll need the config offset anyway bc this timing will be quite variable across clouds | 17:07 |
| rm_work | yes | 17:08 |
| rm_work | but that IS weird TrevorV | 17:08 |
| johnsom | TrevorV Ok, I have no idea why that would be | 17:08 |
| TrevorV | Right, that's exactly right. | 17:09 |
| TrevorV | I'm still tinkering though, so I'll let you know if I find something silly I've done | 17:09 |
| rm_work | anyone have the bug number handy for the session_persistence failure on scenario tests? before i go digging for it | 17:10 |
| blogan | https://bugs.launchpad.net/octavia/+bug/1541675 | 17:10 |
| openstack | Launchpad bug 1541675 in octavia "intermittent session persistence scenario test failure" [Undecided,New] | 17:10 |
| rm_work | lol so you did have it blogan | 17:11 |
| blogan | i just scrolled up to last night | 17:11 |
| rm_work | waaat | 17:11 |
| rm_work | you have WAY more scrollback than I do >_> | 17:11 |
| blogan | i scroll up and travel into the past | 17:11 |
| blogan | i haven't disocnnected since last night | 17:11 |
| rm_work | i tried and my scrollback ends mid-mock-discussion | 17:11 |
| rm_work | me either | 17:11 |
| rm_work | stuff scrolls off | 17:11 |
| blogan | ah | 17:11 |
| johnsom | rm_work Any chance you could give this a try and +A? https://review.openstack.org/#/c/253724/ | 17:11 |
| blogan | weechat brah | 17:11 |
| johnsom | It's a conflict beast I would like to get merged | 17:11 |
| johnsom | Or blogan | 17:12 |
| johnsom | grin | 17:12 |
| blogan | johnsom: no you didn't ask me first | 17:12 |
| rm_work | oh man | 17:12 |
| johnsom | blogan I'm trying to respect your reduced capacity | 17:12 |
| rm_work | just upped scrollback from 300 to 1500 lines :P | 17:12 |
| johnsom | (I did leave that open, yes) | 17:12 |
| blogan | johnsom: you mean my reduced intellect! | 17:12 |
| rm_work | johnsom: i'll look | 17:13 |
| rm_work | oh man this thing | 17:13 |
| johnsom | Thanks | 17:13 |
| blogan | would love to see a scenario test for the failover, but need to actually get the tempst framework set up in octavia and get all that merged first | 17:14 |
| rm_work | i've looked at this a couple of times but I don't know anything about active/standby, lol | 17:14 |
| johnsom | blogan +1 and for act/standby | 17:14 |
| rm_work | i'll try | 17:14 |
| blogan | rm_work: its simple, there's an active and a standby, when active goes down, standby becomes active :) | 17:14 |
| rm_work | lol | 17:14 |
| rm_work | yeah | 17:14 |
| rm_work | but that WORKS now, doesn't it? | 17:14 |
| johnsom | rm_work Here is how I tested: | 17:14 |
| johnsom | Config for Act/Standby, restart o-cw and o-hm, setup an LB, log into either master or backup, stop the amphora agent, watch the wonders of this patch build a new master or standby, profit | 17:16 |
| rm_work | johnsom: spinning up a new devstack with your patch now | 17:17 |
| rm_work | i'll come back to this in 40 minutes or so >_> | 17:17 |
| johnsom | Thank you | 17:17 |
| rm_work | (this takes forever to start) | 17:17 |
| johnsom | Yeah, devstack is a bit evil. So is the darn DIB that downloads a new ubuntu cloud image every darn day so it's got the latest patches | 17:18 |
| TrevorV | johnsom | 17:18 |
| johnsom | Yo | 17:18 |
| TrevorV | could the problem be that the heartbeat in the health manager is "datetime.datetime.utcnow()" while mine is "datetime.datetime.now()"??? | 17:19 |
| johnsom | Ah, yes. | 17:19 |
| TrevorV | ... WOOT | 17:19 |
| * johnsom slaps head | 17:19 | |
| rm_work | possibly | 17:19 |
| TrevorV | Yeah... well, now gotta test it ha ha ha | 17:19 |
| openstackgerrit | Trevor Vardeman proposed openstack/octavia: Amphora create will add a 'fake' heartbeat on creation https://review.openstack.org/271512 | 17:22 |
| blogan | lol | 17:24 |
| blogan | that is totally it | 17:24 |
| blogan | 6 hours off | 17:24 |
| blogan | lets just increase the stale amphora time to 7 hours | 17:24 |
| rm_work | johnsom: so while i'm looking at this... | 17:25 |
| rm_work | https://review.openstack.org/#/c/218735/ | 17:25 |
| johnsom | Ah, yes. | 17:26 |
| johnsom | I have had it on the back burner as I thought the L7 and active/active spec were priority | 17:26 |
| johnsom | I will take a pass at this today | 17:26 |
| *** _cjones_ has joined #openstack-lbaas | 17:26 | |
| *** prabampm has quit IRC | 17:29 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia: Test fix for ssl context wrapper spam https://review.openstack.org/276368 | 17:31 |
| rm_work | I mean, they might be, but that has been up for SO long... | 17:31 |
| rm_work | though if it interferes with the L7 stuff, I dunno | 17:31 |
| rm_work | looks like maybe somehow it magically doesn't | 17:32 |
| johnsom | Yeah, I will review it anyway, it has been a long time and I feel bad about it | 17:35 |
| *** fnaval has quit IRC | 17:35 | |
| johnsom | I have a keen interest too, as it does touch the code Carlos and I worked on before | 17:36 |
| *** fnaval has joined #openstack-lbaas | 17:36 | |
| openstackgerrit | Merged openstack/octavia: Fix misspellings https://review.openstack.org/276240 | 17:37 |
| openstackgerrit | German Eichberger proposed openstack/octavia: Adds a parameter to specify endpoint type https://review.openstack.org/271476 | 17:38 |
| *** Alex_Stef has quit IRC | 17:39 | |
| rm_work | and https://review.openstack.org/#/c/273825/ is a super quick one I'd hope | 17:39 |
| rm_work | oh hmm that might interfere with my changes... ah well | 17:39 |
| rm_work | if it merges it's easy to fix | 17:39 |
| rm_work | maybe i'll just fix mine now | 17:40 |
| *** SumitNaiksatam has joined #openstack-lbaas | 17:44 | |
| *** jwarendt has joined #openstack-lbaas | 17:44 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia: Add SubjectAltName and KeyUsage for localcertgen https://review.openstack.org/275990 | 17:45 |
| openstackgerrit | Phillip Toohill proposed openstack/octavia: Allow user-data on amphora creation https://review.openstack.org/275558 | 17:46 |
| *** eranra has quit IRC | 17:54 | |
| *** amit213 has joined #openstack-lbaas | 17:55 | |
| *** madhu_ak has joined #openstack-lbaas | 17:55 | |
| *** amit213 has quit IRC | 17:55 | |
| *** amit213 has joined #openstack-lbaas | 17:56 | |
| *** Aish has quit IRC | 17:56 | |
| *** mixos has quit IRC | 18:01 | |
| rm_work | how does one "configure for active-standby"? what option is it in the conf | 18:01 |
| rm_work | don't actually know which driver does this | 18:02 |
| rm_work | it's not explicit to me from looking at the available ones | 18:02 |
| johnsom | loadbalancer_topology = ACTIVE_STANDBY | 18:02 |
| rm_work | ah k | 18:02 |
| openstackgerrit | Merged openstack/octavia: Replace depcrecated Nova networks with Nova interfaces https://review.openstack.org/273733 | 18:05 |
| rm_work | "replace" is now not entirely accurate :P | 18:06 |
| johnsom | replace? | 18:07 |
| rm_work | for the commit message of the patch that just merged | 18:07 |
| johnsom | rm_work +A'd event streamer | 18:07 |
| rm_work | cool | 18:07 |
| rm_work | now the neutron side needs to go in and we're good maybe? | 18:07 |
| rm_work | i wonder if that already merged | 18:07 |
| johnsom | I thought I heard it did, but looking it didn't | 18:08 |
| johnsom | https://review.openstack.org/#/c/241474/ | 18:08 |
| ptoohill | This should have a +2 from me, ill fix that | 18:09 |
| ptoohill | but | 18:09 |
| ptoohill | there was some updates I was planning on making, which could always be done later | 18:09 |
| ptoohill | blogan ^ thoughts? | 18:09 |
| ptoohill | To clarify my thoughts as hes probably at lunch: The n-lbaas side can only be used by octavia driver, this should be opened up so any driver can make use of it. Though, at this point I dont think its really a blocker and could be added later. That's up to others to decide I suppose | 18:14 |
| rm_work | hmm | 18:14 |
| rm_work | yeah that is a good point | 18:14 |
| rm_work | i know we were in octavia-mode when making it | 18:14 |
| ptoohill | yea | 18:15 |
| rm_work | but, it's also ok as-is i think because it's just part of the octavia driver right? | 18:15 |
| rm_work | drivers can do whatever they want | 18:15 |
| ptoohill | Yea they can | 18:15 |
| rm_work | ideally later it could be genericized | 18:15 |
| ptoohill | but | 18:15 |
| ptoohill | yea | 18:15 |
| ptoohill | Which I'm ok with, others may not be | 18:15 |
| rm_work | honestly I just want our octavia stuff to work *now* | 18:15 |
| rm_work | is that selfish? >_> | 18:15 |
| ptoohill | exactly | 18:15 |
| ptoohill | Not im my eyes :P | 18:15 |
| rm_work | if other driver authors want it to work, they can put in some effort too :P | 18:16 |
| rm_work | dougwig: ^^ | 18:16 |
| rm_work | thoughts? as a vendor | 18:16 |
| ptoohill | :) | 18:16 |
| openstackgerrit | min wang proposed openstack/octavia: Implements: blueprint anti-affinity server group https://review.openstack.org/272344 | 18:16 |
| ptoohill | well i mean theyre going to have to implement something on their backend to handle it | 18:16 |
| ptoohill | so even at that, it'd be a little bit out before someone was even ready to use it honestly | 18:16 |
| *** Aish has joined #openstack-lbaas | 18:17 | |
| ptoohill | I'm okay with either choice, just want to get things moving | 18:17 |
| rm_work | +1 moving | 18:19 |
| rm_work | blogan: https://review.openstack.org/#/c/273825/ | 18:20 |
| dougwig | rm_work: ok, what am i missing? | 18:28 |
| ptoohill | dougwig: You plan on/want to use the event streamer? | 18:29 |
| rm_work | dougwig: the question was whether you would be annoyed as a vendor that https://review.openstack.org/#/c/241474/ wasn't easily usable by other vendors | 18:29 |
| rm_work | for status updates | 18:29 |
| rm_work | it's coded for octavia only right now | 18:29 |
| ptoohill | The actual question is: To make this generic now OR get this in and get a gernic fix in at a later point | 18:30 |
| *** piet has quit IRC | 18:31 | |
| ptoohill | yea | 18:31 |
| ptoohill | gernic | 18:31 |
| *** piet has joined #openstack-lbaas | 18:31 | |
| rm_work | :P | 18:32 |
| rm_work | alright johnsom it's building an amp-pair | 18:33 |
| rm_work | Soon™ | 18:33 |
| ptoohill | Never*? | 18:33 |
| ptoohill | :P | 18:33 |
| rm_work | T_T | 18:33 |
| johnsom | Go, go, octavia gadget | 18:33 |
| ptoohill | I miss that toon :) | 18:35 |
| rm_work | active! | 18:37 |
| rm_work | ok sshing | 18:37 |
| rm_work | will take ~4min | 18:37 |
| rm_work | hmm weird getting connection refused | 18:38 |
| rm_work | am i using the wrong interface? | 18:38 |
| dougwig | rm_work: nope, i can massage it if i want it more general. if i can't, then i'm a fly-by driver submitter, so i have no rights. | 18:38 |
| rm_work | 10.* or 192.* | 18:38 |
| rm_work | dougwig: that was my thought | 18:39 |
| rm_work | 10.* might be working, i thought we used 192.* tho | 18:39 |
| ptoohill | 192 for mgmt net? | 18:40 |
| ptoohill | context :P Ill stay out of it | 18:40 |
| rm_work | yes | 18:40 |
| rm_work | lb-mgmt-net=192.168.0.4; private=10.0.0.4 | 18:40 |
| rm_work | thought we ssh'd on mgmt-net | 18:40 |
| ptoohill | I thought so too | 18:40 |
| rm_work | still waiting on private-net but at least it didn't "connection-refused" me instantly | 18:40 |
| rm_work | ok now it timed out <_< | 18:41 |
| ptoohill | well yea, youre trying to ssh into amp right? | 18:41 |
| *** bank_ has quit IRC | 18:41 | |
| rm_work | ah | 18:41 |
| ptoohill | you use the mgmt net, need to specificy the cert which is in /etc/octavia/.ssh/octavia_ssh_cert | 18:41 |
| rm_work | i see | 18:41 |
| ptoohill | so | 18:41 |
| rm_work | no i did that | 18:41 |
| rm_work | but i wasn't root | 18:41 |
| rm_work | and it didn't give me a perm error, just passed ... nothing? I guess | 18:42 |
| ptoohill | ssh -i /etc/octavia/.ssh/octavia_ssh_key ubutnu@192.168.0.4 | 18:42 |
| rm_work | and ssh didn't say it was denied for publickey auth | 18:42 |
| rm_work | yes that is what i did | 18:42 |
| ptoohill | hrmm | 18:42 |
| rm_work | but as stack user | 18:42 |
| rm_work | works as root user | 18:42 |
| rm_work | <_< | 18:42 |
| rm_work | whatever | 18:42 |
| ptoohill | thought our images only allowed that key for ubuntu | 18:42 |
| ptoohill | hmm | 18:42 |
| rm_work | no i mean | 18:42 |
| ptoohill | that doesnt sound right | 18:42 |
| rm_work | on the actual devstack box | 18:42 |
| ptoohill | oh | 18:42 |
| rm_work | i guess stack user doesn't have read perms for the private key | 18:43 |
| ptoohill | ah, gotcha | 18:43 |
| rm_work | but i was expecting a permissions error if it didn't | 18:43 |
| ptoohill | its the new TLS spec | 18:43 |
| ptoohill | guess what the prob is | 18:43 |
| *** amotoki has quit IRC | 18:43 | |
| ptoohill | >< | 18:43 |
| rm_work | hmmmmmmmm | 18:45 |
| rm_work | ah maybe it wasn't that | 18:45 |
| rm_work | yeah i literally just didn't wait long enough for openssh to come up i guess >_< | 18:45 |
| rm_work | apparently the LB being active and the agent responding happens before openssh works | 18:45 |
| ptoohill | Yea, i'm not quite sure whats exactly going on, but we do have a few things starting up and it takes a bit for connectivity | 18:47 |
| ptoohill | another example | 18:47 |
| ptoohill | if agent fails you can never get in | 18:47 |
| ptoohill | most of the time | 18:47 |
| ptoohill | o.o | 18:47 |
| rm_work | <_< | 18:47 |
| rm_work | uhh so | 18:47 |
| rm_work | 2016-02-04 18:44:51.390 32382 INFO octavia.controller.healthmanager.health_manager [-] Stale amphora's id is: 6e897162-317d-480d-965a-d31db0d571f2 | 18:47 |
| rm_work | 2016-02-04 18:44:51.397 32382 INFO octavia.controller.healthmanager.health_manager [-] Failed over 1 amphora | 18:47 |
| rm_work | it says it failed it over | 18:47 |
| rm_work | ... nova list isn't showing it being deleted or a new one coming up | 18:48 |
| ptoohill | :/ | 18:48 |
| ptoohill | That's not good | 18:48 |
| rm_work | no activity in o-cw or o-hm after that | 18:48 |
| ptoohill | can you user see the admin stuff? | 18:48 |
| ptoohill | ok, then thats probably not it | 18:48 |
| ptoohill | You could potentially be making the calls as a user and youre not seeing the admin amphoras | 18:49 |
| rm_work | sshing back into the broken one | 18:49 |
| ptoohill | or vms | 18:49 |
| rm_work | nah, nova-list is how i saw where to ssh into | 18:49 |
| ptoohill | kk | 18:49 |
| rm_work | shutting down the agent/haproxy on the other one now | 18:50 |
| openstackgerrit | Merged openstack/octavia: Implementing EventStreamer https://review.openstack.org/218735 | 18:50 |
| openstackgerrit | Merged openstack/neutron-lbaas: Make sample web server close connections https://review.openstack.org/272725 | 18:50 |
| ptoohill | So, it sounded like we had dougwig's blessing on the evenstreamer stuff? | 18:51 |
| rm_work | yes | 18:51 |
| ptoohill | good deal | 18:51 |
| *** bana_k has joined #openstack-lbaas | 18:54 | |
| rm_work | johnsom: this isn't looking great | 18:54 |
| rm_work | I will try another fresh box maybe | 18:54 |
| rm_work | waiting for it to fail-over the other one now | 18:55 |
| rm_work | what is the default failover time? :/ | 18:56 |
| rm_work | it's taking ... a while | 18:56 |
| rm_work | i'm going to actually DELETE the nova vm and see what happens | 18:57 |
| johnsom | Delete doesn't work | 18:57 |
| johnsom | Failover fails with nova delete | 18:57 |
| *** allan_h has quit IRC | 18:57 | |
| johnsom | We talked about that at the mid-cycle | 18:57 |
| *** allan_h has joined #openstack-lbaas | 18:57 | |
| rm_work | ah ok | 18:57 |
| johnsom | It takes a little bit, the default is probably a bit high. | 18:58 |
| rm_work | well it's not working here | 18:58 |
| rm_work | it failed over the first one i shut down the agent on | 18:58 |
| rm_work | ... except it never deleted and recreated the vm | 18:58 |
| rm_work | and it's not even failing over the second one | 18:58 |
| rm_work | i am going to start over on a fresh devstack instance | 18:58 |
| rm_work | will take another hour <_< | 18:58 |
| johnsom | Hmm, check the amphora_health table | 18:58 |
| rm_work | last update for both is over 5 minutes ago | 18:59 |
| rm_work | both marked as busy | 18:59 |
| johnsom | Ok, looking at scroll back, so it said failing over in o-hm log? Then you should have seen a bunch of controller worker stuff in the o-hm log as it built up the new one | 19:00 |
| rm_work | must have failed somewhere during the failover flow | 19:00 |
| rm_work | yeah, NOTHING in o-cw | 19:00 |
| rm_work | last message in o-cw: | 19:00 |
| rm_work | 2016-02-04 18:37:18.553 32350 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ACTIVE in DB for load balancer id: 226cc7a0-e767-49ea-b7e6-f9df71aff860 | 19:00 |
| johnsom | Yeah, it all goes in o-hm for failover | 19:00 |
| rm_work | ah | 19:01 |
| rm_work | last update in o-hm: | 19:01 |
| rm_work | 2016-02-04 18:44:51.390 32382 INFO octavia.controller.healthmanager.health_manager [-] Stale amphora's id is: 6e897162-317d-480d-965a-d31db0d571f2 | 19:01 |
| rm_work | 2016-02-04 18:44:51.397 32382 INFO octavia.controller.healthmanager.health_manager [-] Failed over 1 amphora | 19:01 |
| rm_work | which is that same one | 19:01 |
| rm_work | and it just stopped | 19:01 |
| rm_work | still no mention of the second one that I broke | 19:01 |
| johnsom | Yeah, there should be spew after that | 19:01 |
| rm_work | nothing :/ | 19:01 |
| johnsom | Maybe try again with debug. Sigh. | 19:01 |
| rm_work | turned on debug | 19:03 |
| rm_work | restarted o-hm | 19:03 |
| rm_work | updated both records to busy=0 | 19:04 |
| rm_work | failed over 2 amphora | 19:04 |
| rm_work | and now it is just repeating: | 19:04 |
| rm_work | 2016-02-04 19:03:35.289 9564 DEBUG octavia.controller.healthmanager.health_manager [-] Starting amphora health check health_check /opt/stack/octavia/octavia/controller/healthmanager/health_manager.py:44 | 19:04 |
| rm_work | like i said, going to start over | 19:07 |
| johnsom | Ugh. | 19:07 |
| johnsom | I think I need to build one up again too. | 19:07 |
| TrevorV | rm_work johnsom I don't quite follow what's happened... Is failover broken in general? | 19:17 |
| openstackgerrit | Merged openstack/octavia: Use signing digest defined in config to sign CSR https://review.openstack.org/273825 | 19:18 |
| johnsom | Not sure, could just be my patch for failover active/standby | 19:18 |
| TrevorV | Ooh is that what he's testing right now johnsom ? | 19:18 |
| johnsom | Yep | 19:18 |
| TrevorV | I was trying to figure out if he was helping me vet my review :P | 19:18 |
| blogan | ok im back | 19:21 |
| TrevorV | johnsom I'm not sure how to identify the problem here... I can see it failover my amphora, but its like after adding a fake heartbeat, it never gets a different heartbeat from the amp | 19:21 |
| johnsom | You could ssh into the amp, look in the /var/log/upstart/amphora-agent log and see if there is a hint there | 19:23 |
| TrevorV | johnsom something else that's amusing... the failing over amphora goes into "BOOTING" state, but never comes out... | 19:24 |
| johnsom | Oye | 19:24 |
| TrevorV | Oye? | 19:24 |
| sbalukoff | Holy moly that's a huge backlog. | 19:25 |
| johnsom | A polite saying for what I'm thinking. | 19:25 |
| sbalukoff | Also: Howdy, folks! | 19:25 |
| blogan | oye = fook | 19:26 |
| TrevorV | johnsom https://en.wikipedia.org/wiki/Oy_vey like that? | 19:26 |
| johnsom | Yep, like that | 19:26 |
| TrevorV | Got it... I normally use "Oye!" as an exclamation similar to "HEY" | 19:26 |
| johnsom | Hi sbalukoff Looks like you went after the comments last night. Also shared in the mock fun we all have | 19:27 |
| sbalukoff | Haha! Indeed. | 19:27 |
| sbalukoff | I think my latest patchset addresses most of the things you brought up. | 19:27 |
| sbalukoff | Thought we probably want to talk a bit more about the reordering of some of those tasks in the flows, I think. | 19:27 |
| johnsom | I have a couple of things I need to get started, then we should chat about the flow re-order stuff. | 19:27 |
| johnsom | Yep. Otherwise, I think we are darn close to +2 from me | 19:28 |
| sbalukoff | In short: I'm pretty sure the order I have is correct (at least, leads to the desired outcome on the amphora), but you're right that it probably makes the task that deletes the object from the data model unnecessary. | 19:28 |
| sbalukoff | Nice! | 19:28 |
| sbalukoff | I'm really to talk whenever you are, eh! | 19:28 |
| sbalukoff | ready | 19:28 |
| sbalukoff | Sorry, still a bit tired-- wanted to get this patchset updated and it took until about 3:00am last night. | 19:29 |
| johnsom | Yeah, you nailed most of it from the look at things this morning | 19:29 |
| rm_work | still building devstack again... | 19:29 |
| sbalukoff | Also, 300 more lines of code for adding unit/functional tests (due to you tasking me with fixing existing techincal debt. ;) ) | 19:29 |
| johnsom | rm_work I am building up a fresh vm too | 19:30 |
| *** piet has quit IRC | 19:30 | |
| johnsom | sbalukoff If it makes you feel better, I had a fleeting moment of sadness about that. You did make changes in that code though | 19:30 |
| sbalukoff | Haha! No I get it. | 19:31 |
| sbalukoff | But... I won't remember this. *dundunduuuuun!* | 19:31 |
| sbalukoff | ;) | 19:31 |
| sbalukoff | er... forget. | 19:31 |
| sbalukoff | Well... maybe remember is right. | 19:31 |
| sbalukoff | I can hardly remember anything with any kind of detail if it was over 2 weeks ago. | 19:31 |
| sbalukoff | Dang, I must be more tired than I thought. | 19:31 |
| rm_work | T_T | 19:32 |
| rm_work | adding some tests still seems very minor to me compared to "you need to rewrite the whole client" and "also make the tests not just for what you're touching, but for the whole client because there's nothing" | 19:33 |
| rm_work | so it's hard for me to have much sympathy :P | 19:33 |
| *** piet has joined #openstack-lbaas | 19:33 | |
| sbalukoff | rm_work: I hear you. But I'm still going to complain about it. ;) | 19:33 |
| sbalukoff | Because... you know me. | 19:33 |
| rm_work | heh | 19:33 |
| sbalukoff | No-- in johnsom's position I would probably do the same thing. | 19:34 |
| sbalukoff | case in point, my stubbornness regarding fnaval's tempest patch. ;) | 19:34 |
| sbalukoff | (Which is really good, by the way-- I just want to see it also include a small example test that actually works. But apparently I've been asking for more than I realized.) | 19:34 |
| sbalukoff | (Because when fnaval's patch lands... we can start harrassing people to include tempest tests for significant new functionality they add or alter as well.) | 19:35 |
| fnaval | yep working on it; i'm working on adding the neutron dependency (i.e., networks, subnets, ports) | 19:36 |
| TrevorV | johnsom what were you saying I should look for to see if heartbeats are happening appropriately? | 19:37 |
| *** allan_h has quit IRC | 19:37 | |
| *** allan_h has joined #openstack-lbaas | 19:38 | |
| johnsom | TrevorV That log is for the sending side, maybe it has a clue as to why they aren't coming in | 19:40 |
| TrevorV | I can't even access the instance is my point... | 19:41 |
| TrevorV | Nova says "ACTIVE", but I can't route to the network. | 19:41 |
| TrevorV | Do we ssh as root to the amphora? | 19:41 |
| TrevorV | with /etc/octavia/.ssh/amphora_(thing) right? | 19:42 |
| sbalukoff | it's octavia_thing, but yes. | 19:42 |
| sbalukoff | Er.. | 19:42 |
| sbalukoff | No, ubuntu@ | 19:42 |
| TrevorV | ubuntu @ | 19:42 |
| TrevorV | okay | 19:42 |
| johnsom | TrevorV yeah, ssh -i /amphroakeypath ubuntu@mgmt-net-ip | 19:42 |
| TrevorV | Yeah, the amp stood up when trying to do failover doesn't have its network set up right I'm thinking... | 19:43 |
| TrevorV | Maybe I broke it. | 19:43 |
| TrevorV | Entirely possible I guess... | 19:43 |
| TrevorV | I can't ssh into it at least | 19:43 |
| TrevorV | ssh -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@192.168.0.5 | 19:44 |
| TrevorV | That's what I ran | 19:44 |
| TrevorV | I'm gonna make a fresh devstack server... brb | 19:44 |
| sbalukoff | It's a re-stacking party! Woo-hoo! | 19:45 |
| ptoohill | You get a restack and you get a restack and YOU get a restack! | 19:46 |
| *** allan_h has quit IRC | 19:48 | |
| rm_work | TrevorV: it takes a bit even after ACTIVE for SSH to work :( | 19:51 |
| TrevorV | I waited 5 minutes.... | 19:51 |
| rm_work | yeah | 19:51 |
| rm_work | it took ... about that for me | 19:51 |
| TrevorV | My last attempt I waited longer... | 19:51 |
| TrevorV | Idk, we'll see. | 19:51 |
| TrevorV | Startin fresh, just in case | 19:52 |
| sbalukoff | johnsom: I'm going to work on rebasing my whole L7 chain here in a bit. Please ping me if / when you have time to talk about your comments this afternoon, eh! | 19:53 |
| rm_work | ok just waiting for ACTIVE LB now | 19:54 |
| johnsom | Yep, sounds good | 19:55 |
| johnsom | So, amp should be up in ~30 seconds if and only if grep 'vmx\|svm' /proc/cpuinfo matches on the devstack host | 19:56 |
| *** allan_h has joined #openstack-lbaas | 19:57 | |
| *** _ducttape_ has joined #openstack-lbaas | 19:59 | |
| TrevorV | amp was sending heartbeats (on master branch) before 30 seconds | 19:59 |
| TrevorV | For me at least | 19:59 |
| TrevorV | On my branch, it was my fake heartbeat... and that's it. | 19:59 |
| TrevorV | Does anyone here know how to see a list of things they've reviewed in gerrit? | 20:02 |
| *** ducttape_ has quit IRC | 20:03 | |
| rm_work | https://review.openstack.org/#/dashboard/self | 20:06 |
| rm_work | waiting for failover ... | 20:08 |
| *** madhu_ak has quit IRC | 20:08 | |
| rm_work | ok... | 20:08 |
| rm_work | 2016-02-04 20:08:29.349 25520 INFO octavia.controller.healthmanager.health_manager [-] Stale amphora's id is: a0dda925-7243-4e42-90e0-31d03a8d259f | 20:08 |
| rm_work | 2016-02-04 20:08:29.365 25520 INFO octavia.controller.healthmanager.health_manager [-] Failed over 1 amphora | 20:08 |
| rm_work | nothing else besides the normal spam of it running health_check | 20:09 |
| rm_work | and VMs aren't being recycled | 20:09 |
| rm_work | johnsom: :( | 20:09 |
| rm_work | i wonder if it will even failover SINGLE topologyy | 20:10 |
| rm_work | maybe it's related to something not your patch? | 20:10 |
| johnsom | Maybe. I'm still working on getting my fresh VM up | 20:11 |
| rm_work | doing two more | 20:11 |
| rm_work | one with your patch and one on master | 20:11 |
| rm_work | leaving them in SINGLE | 20:11 |
| johnsom | I messed with the failover flow overall, so I would still be guilty if SINGLE with my patch fails too | 20:14 |
| johnsom | I am stacking now | 20:16 |
| *** _ducttape_ has quit IRC | 20:25 | |
| *** ducttape_ has joined #openstack-lbaas | 20:26 | |
| *** Bjoern has quit IRC | 20:32 | |
| *** madhu_ak has joined #openstack-lbaas | 20:34 | |
| rm_work | waiting on my latest two | 20:35 |
| TrevorV | johnsom I think rm_work just helped me identify *my* problem | 20:35 |
| TrevorV | Idk about yours :P | 20:35 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Add L7 database structures https://review.openstack.org/265430 | 20:36 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Shared pools support https://review.openstack.org/256369 | 20:36 |
| openstackgerrit | Trevor Vardeman proposed openstack/octavia: Amphora create will add a 'fake' heartbeat on creation https://review.openstack.org/271512 | 20:36 |
| *** Bjoern has joined #openstack-lbaas | 20:37 | |
| *** Bjoern has quit IRC | 20:38 | |
| xgerman | dougwig: I am wondering if you can give some love to https://review.openstack.org/#/c/272720/ | 20:45 |
| *** piet has quit IRC | 20:47 | |
| *** Aish has quit IRC | 20:47 | |
| *** piet has joined #openstack-lbaas | 20:49 | |
| *** Aish has joined #openstack-lbaas | 20:49 | |
| *** piet has quit IRC | 20:49 | |
| *** Aish has quit IRC | 20:51 | |
| TrevorV | johnsom still isn't fixed... I can't even... I don't even. I'm broken. I don't get it. | 20:55 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Update repos for L7 objects / methods https://review.openstack.org/265529 | 20:55 |
| *** Aish has joined #openstack-lbaas | 20:56 | |
| TrevorV | johnsom in "update_health_mixin" line 83 | 20:58 |
| TrevorV | well... more line 84. | 20:59 |
| TrevorV | last_update=(datetime.datetime.utcnow()) | 20:59 |
| TrevorV | Is that surrounded by parenthesis so you can wrap it to a new line? | 20:59 |
| johnsom | Yeah, likely | 20:59 |
| TrevorV | Wait, will a heartbeat come in from an amphora if it has no listeners? | 21:00 |
| rm_work | it should | 21:00 |
| rm_work | I never bothered creating listeners in my testing | 21:01 |
| rm_work | and i definitely got them to "mark as stale" by disabling the agent (though failover didn't work? so who knows) | 21:01 |
| johnsom | It shouldn't update the db if the number of listeners in the db doesn't match those reporting. | 21:01 |
| TrevorV | Yeah, I'm collecting that idea... | 21:02 |
| TrevorV | So I changed my "database_task" to use the repository "replace" method, but it still failed. | 21:02 |
| TrevorV | Ugh, here comes ANOTHER clean stack... | 21:03 |
| johnsom | Well, this is probably a problem... | 21:08 |
| johnsom | ImportError: No module named neutron_queue | 21:08 |
| johnsom | Fresh devstack, o-hm log | 21:08 |
| TrevorV | johnsom yeah, saw that too, but that's only come up since the event-streamer patch merged | 21:08 |
| johnsom | Yep | 21:08 |
| rm_work | yeah so i have a feeling i know what caused that (changes in eventstreamer) | 21:08 |
| rm_work | but | 21:08 |
| rm_work | no clue how it would have passed gate | 21:09 |
| TrevorV | However, the failover attempt still logged after that output | 21:09 |
| TrevorV | While I'm waiting... again... for fresh devstack, i'm going to do some reviewing. | 21:09 |
| TrevorV | Back to bothering you after that johnsom :P | 21:10 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Shared pools support https://review.openstack.org/256369 | 21:11 |
| johnsom | Ok, I checked out the failover patch, going to test now | 21:12 |
| *** armax has quit IRC | 21:12 | |
| rm_work | going to dig into why eventstreamer is causing this issue | 21:13 |
| johnsom | setup.cfg issue? | 21:14 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Add L7 database structures https://review.openstack.org/265430 | 21:16 |
| rm_work | johnsom: possibly | 21:17 |
| *** neelashah has quit IRC | 21:17 | |
| rm_work | yep | 21:18 |
| openstackgerrit | Adam Harwell proposed openstack/octavia: Fix event_queue entry in setup.cfg from EventStreamer break https://review.openstack.org/276451 | 21:19 |
| rm_work | sad | 21:19 |
| johnsom | Ok, moment of truth here. I have an LB up with my patch, standalone, health is coming in. About to failover | 21:19 |
| rm_work | ^^ | 21:19 |
| rm_work | single doesn't seem to be failing over either | 21:21 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Update repos for L7 objects / methods https://review.openstack.org/265529 | 21:21 |
| johnsom | looks like the default timeout is 60 seconds | 21:21 |
| johnsom | Ok, got the "failed over | 21:22 |
| johnsom | went busy | 21:22 |
| johnsom | And not much else is happening. | 21:22 |
| TrevorV | Yeah | 21:23 |
| TrevorV | Failover... fails. | 21:23 |
| TrevorV | It should start spewing "trying to connect" over and over until it says something else... | 21:23 |
| rm_work | yeah | 21:23 |
| rm_work | nothing | 21:23 |
| johnsom | Yeah. I blame bit-rot... grin | 21:24 |
| rm_work | waiting to find out if it works WITHOUT your patch | 21:24 |
| johnsom | Ok, digging in | 21:24 |
| rm_work | had to reset my devstack | 21:24 |
| rm_work | with the fix above ^^ | 21:24 |
| TrevorV | Thanks for the help guys, I just can't seem to find what's wrong... | 21:24 |
| rm_work | https://review.openstack.org/276451 | 21:24 |
| rm_work | TrevorV: i can look at yours in a bit | 21:24 |
| rm_work | once i finish up this last test | 21:24 |
| TrevorV | All good rm_work | 21:24 |
| rm_work | see if something jumps out at me | 21:24 |
| TrevorV | I'ma +1 your review for now | 21:25 |
| sbalukoff | Man, I'm beating the shit out of zuul's change queue... | 21:25 |
| TrevorV | You too? sbalukoff | 21:25 |
| TrevorV | sbalukoff I'm re-reviewing your shared pools patch | 21:25 |
| sbalukoff | TrevorV: You gotta admit, it was asking for it. | 21:25 |
| rm_work | yeah zuul is | 21:25 |
| rm_work | slow today | 21:25 |
| sbalukoff | TrevorV: Thanks! | 21:25 |
| rm_work | lots of shit T_T | 21:26 |
| rm_work | been waiting for the changes i did THIS MORNING to finish running | 21:26 |
| TrevorV | No problem... I noticed you didn't address a comment I had before, sbalukoff :( | 21:26 |
| sbalukoff | Oh, what comment? | 21:26 |
| sbalukoff | (There have been a lot... especially from you. ;) ) | 21:26 |
| TrevorV | Wait... nevermind.. its just another location my comment should have addressed. | 21:26 |
| TrevorV | The "_test_lb_and_listener_status" | 21:26 |
| openstackgerrit | Adam Harwell proposed openstack/octavia: Test fix for ssl context wrapper spam https://review.openstack.org/276368 | 21:26 |
| TrevorV | You have another location where that name applies | 21:26 |
| TrevorV | Or SHOULD be applied | 21:27 |
| TrevorV | You'll see the comment when I post it :P | 21:27 |
| sbalukoff | Uh-oh. | 21:27 |
| TrevorV | I put "nit:" and won't -1 from that | 21:27 |
| sbalukoff | I would! | 21:27 |
| sbalukoff | ;) | 21:27 |
| sbalukoff | But then, I'm an asshole. | 21:27 |
| TrevorV | Fine, then I will MUAH HA HA HA HA | 21:27 |
| rm_work | this totally works BTW: https://review.openstack.org/#/c/275990/ | 21:28 |
| rm_work | and this fixes the last bit: https://review.openstack.org/#/c/276368/2 | 21:28 |
| rm_work | and NO MORE SPAM FROM SSL SHIT | 21:28 |
| rm_work | :P:P:P | 21:28 |
| TrevorV | You know, it occurs to me... You've written a bunch of "get this from DB" methods... any reason you're not just importing repositories and doing an init step to set them up right? | 21:28 |
| TrevorV | sbalukoff ? | 21:29 |
| sbalukoff | TrevorV: I was trying to remain closer to the original code. | 21:29 |
| TrevorV | I misread, you didn't rewrite the query section, you just added a helper method. | 21:30 |
| TrevorV | You're fine, sorry | 21:30 |
| sbalukoff | :) | 21:30 |
| TrevorV | Your review may not be the place for it, but we really need to have constants for our controllers. | 21:31 |
| TrevorV | if controller == 'listeners' | 21:31 |
| TrevorV | Should be something like | 21:31 |
| TrevorV | if controller == constants.listener_controller | 21:31 |
| TrevorV | or something like that | 21:31 |
| rm_work | ideally no strings, yeah | 21:31 |
| TrevorV | I'll put up a patch set sometime this weekend for it if I remember. | 21:32 |
| TrevorV | In my own time I mean | 21:32 |
| sbalukoff | TrevorV: You're talking about the _lookup stuff in the API code? Yeah-- I thought it was odd, but went with the established pattern. :P | 21:33 |
| rm_work | i forget why that's a thing | 21:35 |
| TrevorV | sbalukoff its in the controllers | 21:35 |
| sbalukoff | rm_work: I +1'd your two patches above. | 21:35 |
| rm_work | if only you were core <_< | 21:35 |
| sbalukoff | Haha | 21:35 |
| sbalukoff | TrevorV: If you could post that comment to the shared pools patch with the function you'd like renamed, I'll get the done real quick before going to lunch, eh. | 21:36 |
| TrevorV | I +1'd a couple of reviews of yours too adam! I'm looking at the cert one | 21:36 |
| TrevorV | sbalukoff no hurry, I'm still going through the rest of the review again | 21:37 |
| TrevorV | Found another thing I'm nit-picking | 21:37 |
| sbalukoff | Heh. Ok. | 21:37 |
| sbalukoff | I think I know where that function name is that you don't like, though... so I may fix that pre-emptively. ;) | 21:38 |
| TrevorV | Screws up my comments, but hey :P | 21:38 |
| TrevorV | sbalukoff found another one with the name | 21:40 |
| TrevorV | o_0 | 21:40 |
| TrevorV | I'll post the ones I have so far, and then just keep going, sound good sbalukoff ? | 21:40 |
| *** _ducttape_ has joined #openstack-lbaas | 21:40 | |
| *** ducttape_ has quit IRC | 21:42 | |
| blogan | dear god | 21:43 |
| blogan | i can't step away from this channel | 21:43 |
| sbalukoff | TrevorV: Sounds good! | 21:43 |
| sbalukoff | blogan: It's my fault. | 21:43 |
| sbalukoff | Sorry! | 21:43 |
| *** madhu_ak_ has joined #openstack-lbaas | 21:43 | |
| TrevorV | :P | 21:44 |
| blogan | my name wasn't mentioned so im just going to assume i dont needt o read it | 21:44 |
| TrevorV | heh sbalukoff you didn't change the one I was commenting on at all :P ha ha ha | 21:44 |
| TrevorV | I accidentally commented on 23 so copied them over to 24 | 21:44 |
| rm_work | blogan: https://review.openstack.org/276368 | 21:45 |
| rm_work | err sorry | 21:45 |
| sbalukoff | TrevorV: Aah, ok. | 21:45 |
| rm_work | https://review.openstack.org/#/c/276451/ | 21:45 |
| rm_work | this one | 21:45 |
| blogan | rm_work: bertrand has a good point | 21:45 |
| blogan | i dont know wtf this is for | 21:45 |
| rm_work | what | 21:46 |
| blogan | the review | 21:46 |
| rm_work | which | 21:46 |
| blogan | ohh the first | 21:46 |
| *** madhu_ak has quit IRC | 21:46 | |
| rm_work | ah | 21:46 |
| rm_work | yeah uhh i might make a bug for it | 21:46 |
| rm_work | it's more spam | 21:46 |
| blogan | ah damn, i missed that in the setup.cfg didnt i | 21:46 |
| rm_work | guess so | 21:48 |
| rm_work | i did too | 21:48 |
| rm_work | i was testing with the ACTUAL driver | 21:48 |
| johnsom | We all did | 21:48 |
| *** neelashah has joined #openstack-lbaas | 21:48 | |
| rm_work | which was correct >_< | 21:48 |
| rm_work | but the noop one was broken and is default lol | 21:48 |
| rm_work | ok I am seeing SINGLE failover *is not broken* without johnsom's patch | 21:52 |
| rm_work | or german's | 21:52 |
| rm_work | whoever | 21:53 |
| rm_work | anyway, it's something in that patch | 21:53 |
| *** armax has joined #openstack-lbaas | 21:53 | |
| rm_work | it works on Master | 21:53 |
| TrevorV | what? | 21:53 |
| johnsom | Yeah, I close to the answer | 21:53 |
| rm_work | i was afraid it would be broken on master too | 21:53 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Shared pools support https://review.openstack.org/256369 | 21:53 |
| TrevorV | Oh you both are looking at ACTIVE/PASSIVE patch right? | 21:53 |
| rm_work | yes | 21:53 |
| johnsom | Though I am going to fix the code that eats the exception | 21:53 |
| rm_work | k | 21:53 |
| TrevorV | Oh... thought I was important... *cries* | 21:53 |
| rm_work | that would be nice | 21:53 |
| rm_work | ^^ to both johnsom and TrevorV :P | 21:54 |
| TrevorV | I don't have code that eats exceptions... | 21:54 |
| rm_work | lol yes | 21:54 |
| rm_work | same response to his thing about exceptions and your think about being important :P | 21:54 |
| rm_work | meant to be responding to him but it worked as a response to you too :P | 21:55 |
| TrevorV | Not really... "it would be nice for me to think of you as important"? | 21:55 |
| TrevorV | idk, whateva. I'm just reviewing... heh | 21:57 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Add L7 database structures https://review.openstack.org/265430 | 21:59 |
| openstackgerrit | Adam Harwell proposed openstack/octavia: Fix for ssl context wrapper spam https://review.openstack.org/276368 | 22:07 |
| rm_work | there was already a bug | 22:07 |
| rm_work | :P | 22:08 |
| johnsom | yep | 22:09 |
| openstackgerrit | min wang proposed openstack/octavia: Implements: blueprint anti-affinity server group https://review.openstack.org/272344 | 22:10 |
| johnsom | rm_work Found two issues. 1. healthmanager is eating exceptions. 2. missing rebinds for the failed amp data | 22:10 |
| johnsom | Testing now | 22:10 |
| openstackgerrit | Stephen Balukoff proposed openstack/octavia: Update repos for L7 objects / methods https://review.openstack.org/265529 | 22:10 |
| rm_work | kk | 22:10 |
| *** nmagnezi_ has joined #openstack-lbaas | 22:11 | |
| rm_work | sbalukoff is going for "most patchsets in a day" achievement | 22:12 |
| sbalukoff | Just trying to keep my dependency chain up to date. :) | 22:12 |
| sbalukoff | Also, yes I am. | 22:12 |
| sbalukoff | ;) | 22:12 |
| rm_work | yeah I remember those long dependency chains :( | 22:12 |
| sbalukoff | There are actually three more in the chain I haven't been updating just yet because I've realized I need to insert a new patch in the middle there. :P | 22:13 |
| sbalukoff | Also, L7Rule validation checks... there's going to be a lot of them. | 22:13 |
| sbalukoff | Yay for allowing mostly-arbitrary input! | 22:13 |
| sbalukoff | But I'mma go get the lunch for now. BBIAB. | 22:15 |
| openstackgerrit | Aishwarya Thangappa proposed openstack/octavia: Adding "region and endpoint_type" parameters to barbican_acl.py https://review.openstack.org/272861 | 22:15 |
| johnsom | Totally bit rot. I rebased onto patches for the disable health monitoring on delete, which was missing the rebinds. | 22:18 |
| xgerman | rm_work Brandon says the CI is better so I will it test it ;-) | 22:19 |
| rm_work | johnsom: :( | 22:20 |
| rm_work | check-queue in zuul so backed up | 22:21 |
| rm_work | wish it'd at least be under 1.5h to START testing | 22:21 |
| xgerman | our internal zulu is 11 h — so be thankful, could be worse | 22:22 |
| *** davidlenwell has quit IRC | 22:22 | |
| rm_work | T_T | 22:22 |
| johnsom | Yeah, it's a nightmare | 22:22 |
| rm_work | one chance per day to run tests? :/ | 22:23 |
| *** davidlenwell has joined #openstack-lbaas | 22:23 | |
| xgerman | yep, done by Irish people. Submit your patch, go to the pub, | 22:24 |
| johnsom | I guess someone "appropriated" some hardware that was in the node pool by mistake | 22:24 |
| nmagnezi_ | xgerman, that's a decent work flow :) | 22:28 |
| xgerman | :-) | 22:29 |
| *** localloop127 has quit IRC | 22:29 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia: Updates the failover flow for active/standby https://review.openstack.org/253724 | 22:31 |
| johnsom | rm_work ^^^ | 22:31 |
| rm_work | kk | 22:31 |
| rm_work | restacking lol | 22:31 |
| rm_work | see you in 40 minutes | 22:32 |
| johnsom | Let me know what you think of this: | 22:32 |
| johnsom | https://review.openstack.org/#/c/253724/16/octavia/controller/worker/controller_worker.py | 22:32 |
| johnsom | Not sure if that is the right way to do that. | 22:33 |
| rm_work | there's a .... log and reraise? | 22:34 |
| rm_work | right? | 22:34 |
| rm_work | or is that only n-lbaas | 22:34 |
| rm_work | because this way you lose the original exception | 22:34 |
| rm_work | and that's a kind of nasty catch-all | 22:34 |
| johnsom | Yeah, the problem is right above this is a thread pool that eats the exceptions | 22:35 |
| johnsom | controller/healthmanager/healthmanager.py | 22:35 |
| rm_work | hmm | 22:35 |
| rm_work | so with excutils.save_and_reraise_exception(reraise=True) | 22:35 |
| rm_work | wouldn't work because the exception would get eaten anyway? | 22:36 |
| johnsom | Well, I know the exception wasn't coming out of that method before. | 22:37 |
| rm_work | save_and_reraise does logging, right? | 22:37 |
| rm_work | i assume that's the "save" part | 22:37 |
| johnsom | I am barely familiar with that | 22:37 |
| rm_work | me too >_< | 22:37 |
| johnsom | Reading now: http://docs.openstack.org/developer/oslo.utils/api/excutils.html | 22:37 |
| rm_work | oh | 22:38 |
| rm_work | ok no, save is not related to logging | 22:38 |
| rm_work | but you can log inside there | 22:38 |
| rm_work | so I think that's the better approach | 22:38 |
| rm_work | and I guess the default for reraise is True, so no need to be explicit | 22:39 |
| johnsom | We don't really want a failed thread to stop the whole health manager | 22:42 |
| openstackgerrit | min wang proposed openstack/octavia: Implements: blueprint anti-affinity server group https://review.openstack.org/272344 | 22:53 |
| *** rtheis has quit IRC | 22:54 | |
| *** armax has quit IRC | 22:57 | |
| *** allan_h has quit IRC | 23:05 | |
| *** allan_h has joined #openstack-lbaas | 23:05 | |
| *** jwarendt has quit IRC | 23:06 | |
| *** bana_k has quit IRC | 23:06 | |
| *** nmagnezi_ has quit IRC | 23:08 | |
| *** bana_k has joined #openstack-lbaas | 23:10 | |
| *** neelashah has quit IRC | 23:14 | |
| openstackgerrit | Trevor Vardeman proposed openstack/octavia: Amphora create will add a 'fake' heartbeat on creation https://review.openstack.org/271512 | 23:18 |
| *** SumitNaiksatam has quit IRC | 23:22 | |
| *** SumitNaiksatam has joined #openstack-lbaas | 23:23 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia: Updates the failover flow for active/standby https://review.openstack.org/253724 | 23:26 |
| johnsom | Ok, switched that exception handling and fixed the tox issues I thought I had run the tests on | 23:26 |
| johnsom | That one is good to go | 23:27 |
| *** allan_h has quit IRC | 23:27 | |
| *** mgarza has quit IRC | 23:27 | |
| *** crc32 has quit IRC | 23:29 | |
| *** chlong has quit IRC | 23:30 | |
| rm_work | johnsom: ok so maybe we continue to not raise, but ... at least LOG something? | 23:31 |
| rm_work | ah | 23:31 |
| rm_work | k was scrolled up | 23:31 |
| johnsom | Ok, with that fixed, I will take a quick look at TrevorV's patch | 23:42 |
| TrevorV | johnsom alright man, we got it figured out | 23:42 |
| TrevorV | :D | 23:42 |
| johnsom | Or not. | 23:42 |
| TrevorV | heh | 23:42 |
| TrevorV | Adam identified the logic error I had, and we added a config value to "append" to the first heartbeat | 23:43 |
| TrevorV | Made the default 300 | 23:43 |
| TrevorV | seconds | 23:43 |
| TrevorV | So basically it says it heartbeats 5 minutes after it goes "ACTIVE" in nova | 23:43 |
| TrevorV | We watched that number get replaced by an "earlier" one when the REAL heartbeat came in | 23:43 |
| johnsom | Very generous | 23:43 |
| TrevorV | That number can change, just threw it out there because rm_work suggested it takes 5 minutes to make in devstack ha ha | 23:44 |
| johnsom | Yeah, that is fine | 23:44 |
| TrevorV | Either way, the real heartbeat replaces it with the appropriate heartbeat time, so it works out in the end. | 23:44 |
| johnsom | Yeah, nice | 23:44 |
| TrevorV | So when zuul gives its go-ahead, we're solid on the reviewing | 23:45 |
| *** TrevorV has quit IRC | 23:48 | |
| rm_work | yeah we want to be a bit generous because it's bad-news-bears if we cut it too short and it causes problems in gate | 23:48 |
| openstackgerrit | Michael Johnson proposed openstack/octavia: Fixes Octavia flows that are not reverting https://review.openstack.org/259550 | 23:49 |
| rm_work | in a real deployment you'd want like .... 30 seconds or something | 23:49 |
| johnsom | Yep | 23:49 |
| rm_work | but also you don't want to hammer an already stressed environment | 23:49 |
| rm_work | so you don't want to cut it too close anywhere | 23:49 |
| rm_work | that's a cascade failure waiting to happen | 23:49 |
| rm_work | brb | 23:52 |
| *** _ducttape_ has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
