rm_work | johnsom: the idea is that now, other bits take care of this (because we KNOW it's a deleted amp) | 00:02 |
---|---|---|
rm_work | i thought i already had code in there for this | 00:02 |
rm_work | but maybe not? | 00:04 |
rm_work | but you should be able to just explicitly check if it's deleted | 00:04 |
rm_work | no more beating around the bush | 00:04 |
johnsom | rm_work I'm not following you. This whole issue is where nova fails to delete the service vm after we delete an amp. That VM would still be sending heartbeats even though we consider it deleted. | 00:04 |
rm_work | right | 00:04 |
rm_work | but the amp record will still exist | 00:04 |
rm_work | so we can look it up | 00:04 |
rm_work | and be like "oh, it's deleted" | 00:04 |
johnsom | Well, it won't once the cleanup job comes around | 00:04 |
rm_work | right but... | 00:05 |
rm_work | we should know pretty quick | 00:05 |
rm_work | and the cleanup should be like | 00:05 |
rm_work | hours to days | 00:05 |
rm_work | but i get your point... | 00:06 |
johnsom | Right, so you have german's lab where nova goes out to lunch in "deleting", they sit there for weeks stuck in this state sending heartbeats. We want to fail fast those and not try to fail them over | 00:07 |
johnsom | So I found a completely un-used task, nuked that, after that there are two places using the get lbs on amp (one is sketch), so I think it's safe to add a filter on deleted to that db repo method to exclude those | 00:08 |
rm_work | k | 00:10 |
rm_work | i look forward to the next patchset :P | 00:15 |
*** SumitNaiksatam has quit IRC | 00:15 | |
rm_work | oh or did you already | 00:16 |
johnsom | Just about to post | 00:16 |
rm_work | k | 00:17 |
johnsom | running tests | 00:17 |
johnsom | So, grab a coffee or something. grin | 00:17 |
*** threestrands has quit IRC | 00:25 | |
*** threestrands has joined #openstack-lbaas | 00:28 | |
openstackgerrit | Michael Johnson proposed openstack/octavia master: Improve the error logging for zombie amphora https://review.openstack.org/561369 | 00:31 |
*** longkb1 has joined #openstack-lbaas | 00:32 | |
johnsom | rm_work There you go | 00:32 |
rm_work | lolol | 00:33 |
rm_work | nice file delete | 00:33 |
rm_work | hah | 00:33 |
rm_work | also like the TODO | 00:33 |
rm_work | that subquery also seemed kinda dumb, wasn't sure the point exactly | 00:33 |
rm_work | this one https://review.openstack.org/#/c/561369/7/octavia/db/repositories.py | 00:33 |
rm_work | so the subquery gets... | 00:33 |
rm_work | err it'd be like | 00:34 |
rm_work | SELECT load_balancer_id FROM amphora where id = $AMPID; | 00:34 |
rm_work | right? | 00:34 |
rm_work | which ... why does that even need to be a subquery | 00:34 |
rm_work | that's just a join | 00:34 |
rm_work | this whole thing makes no sense to me | 00:35 |
rm_work | if you just deleted it and said | 00:35 |
rm_work | "ok, we need to get the LB record for this amp" | 00:35 |
rm_work | it would not look remotely like that | 00:35 |
johnsom | Right it gets the list of lb id's (if there was more than one, then gets the whole LB objects for each | 00:35 |
rm_work | ok but like | 00:35 |
rm_work | i know we said "multi-lb per amp blah blah" | 00:35 |
rm_work | but | 00:35 |
rm_work | that's physically impossible with this database schema | 00:36 |
johnsom | (not defending, just clarifying) | 00:36 |
rm_work | so unless we changed the DB schema... | 00:36 |
rm_work | what are we doing | 00:36 |
rm_work | can we just agree to stop dancing around that multi-lb-amp thing | 00:36 |
rm_work | and just acknowledge it's not a thing | 00:37 |
rm_work | and start fixing it | 00:37 |
rm_work | instead of continuing to add more backflips-and-hoops logic | 00:37 |
rm_work | johnsom: ^^ | 00:39 |
rm_work | it simplifies a lot of our code | 00:39 |
rm_work | which we continue to dodge around because "we might do this maybe kinda in the future but probably not" | 00:39 |
johnsom | I kind of agree, I was just looking for the patch I thought we had from someone wanting to do this multi-lb on amp thing | 00:40 |
rm_work | i mean ... make your own driver in that case | 00:40 |
* rm_work shrugs | 00:40 | |
rm_work | all of this is pluggable | 00:40 |
rm_work | and this subquery BS is almost unreadable | 00:40 |
johnsom | It was this one: https://review.openstack.org/#/c/508060/ | 00:41 |
rm_work | and the logic it spawns is batshit crazy | 00:41 |
johnsom | It is true | 00:41 |
johnsom | Personally I think we are heading towards smaller not fatter amps | 00:41 |
rm_work | yes | 00:42 |
rm_work | reading the story now | 00:42 |
rm_work | like... i get it? but ... eugh | 00:42 |
johnsom | Yeah, I personally would support just calling this driver to be single lb per amp | 00:43 |
rm_work | anyway, IMO, we should write the query to match the CURRENT SCHEMA, and then if we do this schema change, it can fix the query | 00:43 |
rm_work | well, i want to say that too, but I see now that this is a db schema issue that would apply to any amp driver | 00:43 |
rm_work | :( | 00:43 |
rm_work | so to be generic we would need to do this | 00:43 |
rm_work | but AT THAT POINT, when the schema changes, we could change this query to be different | 00:44 |
johnsom | That patch is so old it should be abandoned frankly | 00:44 |
rm_work | and that would be a huge schema change, like, non-backwards-compat IMO? | 00:44 |
rm_work | so i don't even know how we manage that | 00:44 |
rm_work | dropping a column?! | 00:44 |
rm_work | <_< | 00:44 |
johnsom | So, are you going to ask me to re-write this method or are we applying the Spackle? | 00:46 |
rm_work | reviewed with a -2 and some comments | 00:47 |
johnsom | I meant my patch | 00:47 |
rm_work | I think the spackle should be "eggshell" color | 00:47 |
rm_work | yeah ummm | 00:47 |
rm_work | i don't want to just force this whole change into that patch | 00:47 |
rm_work | but | 00:47 |
rm_work | if you are rewriting that query ... | 00:47 |
rm_work | maybe now is the time to at least fix THAT one | 00:47 |
johnsom | starts purple then turns white as it dries | 00:47 |
rm_work | because that query is like | 00:48 |
rm_work | http://www.funnybeing.com/wp-content/uploads/2017/01/WTF-Is-This-Shit-600x378.jpg | 00:48 |
johnsom | It's really not too bad.... | 00:48 |
rm_work | you think so? lol | 00:48 |
rm_work | it's textbook "subquery that is so simple it should be a join" compounded by "SQLAlchemy is lulz to define certain things" | 00:49 |
johnsom | The join and single query will be long too | 00:49 |
rm_work | let me take a crack at it | 00:49 |
rm_work | i'll pastebin what i think is right | 00:49 |
johnsom | Yeah, I mean, the subquery thing is efficient and allows the nice modeling of the list of LBs. It's just not really needed as it won't be a list | 00:50 |
rm_work | does this look right? http://paste.openstack.org/show/722841/ | 00:55 |
johnsom | That might work. It's missing the not deleted for the amp though | 00:57 |
rm_work | http://paste.openstack.org/show/722842/ | 00:57 |
rm_work | updated to provide commentary | 00:58 |
rm_work | oh i didn't see that on yours | 00:58 |
rm_work | ah whoops | 00:58 |
johnsom | Why not just switch it from returning all() and a list to returning one LB object? | 00:59 |
rm_work | I WANT TO | 01:00 |
rm_work | but | 01:00 |
rm_work | then I'd have to go fix everywhere that uses the function too | 01:00 |
rm_work | because it expects a list | 01:00 |
rm_work | one sec, screw it | 01:00 |
johnsom | There are only two places | 01:02 |
rm_work | johnsom: http://paste.openstack.org/show/722843/ | 01:02 |
rm_work | yeah | 01:02 |
rm_work | realized that | 01:02 |
johnsom | Yep, that should do it. | 01:03 |
johnsom | Are you hacking my patch or should I? | 01:03 |
rm_work | ah, if you want, I have it all done tho | 01:06 |
rm_work | well, mostly | 01:06 |
johnsom | I'll do it real quick | 01:10 |
johnsom | Keep your review clean.. lol | 01:10 |
rm_work | lol | 01:10 |
rm_work | i mean i finished | 01:10 |
rm_work | so | 01:10 |
*** JudeC__ has quit IRC | 01:10 | |
johnsom | oh | 01:11 |
johnsom | ok, then just do it | 01:11 |
rm_work | tests running | 01:14 |
rm_work | there were a LOT of test changes lol | 01:14 |
*** SumitNaiksatam has joined #openstack-lbaas | 01:14 | |
rm_work | like | 01:14 |
rm_work | 60 | 01:14 |
*** hongbin has joined #openstack-lbaas | 01:16 | |
rm_work | k | 01:19 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: Improve the error logging for zombie amphora https://review.openstack.org/561369 | 01:19 |
rm_work | tests take a long time now <_< | 01:19 |
rm_work | make sure that looks good to you | 01:19 |
johnsom | cannot merge... hmnmm | 01:20 |
rm_work | wut | 01:20 |
rm_work | one sec let me pull and rebase | 01:20 |
rm_work | maybe now? | 01:20 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: Improve the error logging for zombie amphora https://review.openstack.org/561369 | 01:20 |
rm_work | not sure what that was about | 01:21 |
rm_work | obviously some change... | 01:21 |
rm_work | parent changed, so i must have just been behind | 01:21 |
rm_work | AHHHH | 01:22 |
rm_work | it wasn't based on the failover-failed one we merged | 01:22 |
rm_work | which is where i said i added some skip-logic already | 01:22 |
rm_work | note https://review.openstack.org/#/c/561369/7..9/octavia/controller/worker/controller_worker.py | 01:22 |
rm_work | which is not in THIS patch (that change already landed) | 01:23 |
johnsom | Yeah, that looks pretty good to me | 01:28 |
rm_work | look forward to seeing the scenario results | 01:29 |
rm_work | i love the new scenario HM traffic test | 01:29 |
rm_work | it really makes sure the HM is working properly | 01:29 |
rm_work | and by HM, I mean ... both | 01:29 |
rm_work | the new HealthMonitor traffic scenario really verifies that the HealthManager process is working properly ;) | 01:29 |
*** threestrands has quit IRC | 01:35 | |
*** threestrands has joined #openstack-lbaas | 01:35 | |
*** threestrands has quit IRC | 01:36 | |
*** threestrands has joined #openstack-lbaas | 01:37 | |
*** threestrands has quit IRC | 01:37 | |
*** threestrands has joined #openstack-lbaas | 01:37 | |
*** kiennt26 has joined #openstack-lbaas | 01:37 | |
*** threestrands has quit IRC | 01:38 | |
openstackgerrit | Adam Harwell proposed openstack/octavia master: Experimental multi-az support https://review.openstack.org/558962 | 01:38 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: AZ Evacuation resource https://review.openstack.org/559873 | 01:38 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 01:38 |
*** threestrands has joined #openstack-lbaas | 01:38 | |
*** threestrands has quit IRC | 01:38 | |
*** threestrands has joined #openstack-lbaas | 01:38 | |
*** threestrands has quit IRC | 01:39 | |
*** threestrands has joined #openstack-lbaas | 01:40 | |
*** threestrands has quit IRC | 01:40 | |
*** threestrands has joined #openstack-lbaas | 01:40 | |
*** threestrands has quit IRC | 01:41 | |
*** blake has quit IRC | 01:56 | |
rm_work | johnsom: augh, amp failover may be broken | 01:56 |
rm_work | errr | 01:56 |
rm_work | i mean sorry | 01:56 |
rm_work | *not* failover explicitly | 01:56 |
rm_work | and i said the wrong thing anyway | 01:56 |
rm_work | the "loadbalancer failover" API call might be broken | 01:57 |
rm_work | if you run it on a LB that has already had a failover of any type (ie, run it more than one time) | 01:57 |
rm_work | the fix is trivial | 01:57 |
rm_work | I want to update the tempest testing first though, to have it like... do a few tries in a row | 01:57 |
johnsom | Yeah, we always need to test multi failover | 01:58 |
johnsom | Api or not | 01:59 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 02:00 |
bzhao__ | johnsom: Thank you very much for https://storyboard.openstack.org/#!/story/2002165 and https://storyboard.openstack.org/#!/story/2002160 | 02:03 |
bzhao__ | But I still want to make sure that you listed task, for the client certificate authentication part, you suggest that 2 fields would be extended for linstener API, such as "client_ca_tls_container_ref" for ca certificate, and a "option" for client ca certification(for this, maybe something can named it as "client_ca_cert_type", or some other things like this? :)) | 02:03 |
bzhao__ | And the 3rd task "add header insertion" part, I found You suggest that support the ca certification on real service not LB itself. What I introduced is LB(haproxy) to check the client ca certificate to decide whether LB forward the arrived request. Then, I check the our api-ref, the "header insertion" field is insert something in HTTP header then forward the requests to backend server, so I'm confu | 02:03 |
bzhao__ | o I'm confused that whether we are following, haha. But I still think the 3rd task is very valuable for server-client authentication. Here, I just want to make sure that we are the same. | 02:04 |
*** threestrands has joined #openstack-lbaas | 02:11 | |
*** threestrands has quit IRC | 02:11 | |
*** threestrands has joined #openstack-lbaas | 02:11 | |
johnsom | bzhao__: I am cooking dinner now, but might be able to chat in an hour or two. I guess I was not clear on those as I had a different intent. I will comment on the story too | 02:35 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 02:44 |
bzhao__ | johnsom: Thank you. :) haha | 02:46 |
*** kiennt26 has quit IRC | 02:53 | |
*** threestrands has quit IRC | 03:13 | |
*** kiennt26 has joined #openstack-lbaas | 03:32 | |
johnsom | bzhao__ Ok, can chat now. | 03:50 |
bzhao__ | johnsom: :) | 03:51 |
johnsom | So, for the first one, I think the API needs to be extended with two parts: | 03:51 |
johnsom | 1. The reference to the ca certificate used to validate the client certificate. | 03:51 |
johnsom | 2. An option for "No verification" or None, "Optional verification" meaning it will as for a cert, but still pass traffic if no cert is given or it fails validation, and finally "Required" meaning no connection if it isn't a valid cert. | 03:52 |
johnsom | Optional is handy when you want to present one web site to non-cert holder, but a different if they present a valid cert. | 03:53 |
johnsom | Come to think about it, there might be another task to add L7 support for valid cert connections. | 03:54 |
johnsom | bzhao__ Does that help clarify my thinking on that Task? | 03:54 |
bzhao__ | johnsom: Thanks, I'm thinking. . | 03:55 |
*** blake has joined #openstack-lbaas | 03:57 | |
johnsom | This is the equivalent haproxy config: http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.1-verify | 03:57 |
bzhao__ | johnsom: I agree. Thanks for this link. It's very helpful to understand. :) | 03:59 |
bzhao__ | johnsom: Then , I follow u | 04:00 |
*** yamamoto has joined #openstack-lbaas | 04:00 | |
johnsom | bzhao__ Great! I have used this in the past to allow new users to sign up and get a cert, but users with certs get logged directly into the web application. | 04:01 |
bzhao__ | johnsom: sorry for the pool haproxy configuration knowledge. ha | 04:01 |
johnsom | bzhao__ No worries! Ok, next is the "insert headers" task | 04:01 |
bzhao__ | johnsom: So cool practice | 04:02 |
johnsom | This is about giving the backend server information about the client certificate authentication at the load balancer. | 04:02 |
*** blake has quit IRC | 04:02 | |
johnsom | When we validate a client certificate, we want options to insert an HTTP header that tells the backend server we validated the certificate and this user is "trusted". | 04:02 |
bzhao__ | johnsom: Yeah, so we must to make sure the backend server support HTTPS | 04:03 |
johnsom | The second header would extract the "common name" or CN or subject of the certificate and pass it to the backend server in the HTTP header | 04:03 |
bzhao__ | johnsom: Correct, but AFAIK, the HTTPS offloading will be processed in LB. Please feel free to correct. HA | 04:04 |
johnsom | Both of these do not require HTTPS on the backend | 04:04 |
bzhao__ | johnsom: OK | 04:04 |
johnsom | bzhao__ Correct, the client cert is validated and the TLS terminated at the load balancer. | 04:04 |
johnsom | But, because we are doing that, we need to give the backend web server information about that user. | 04:05 |
johnsom | Optionally we would insert an HTTP header in the connection to the backend server, like we do today for "X-Forwarded-For", but one that tells the web server we successfully validated their client certificate and one that provides the "common name" that was contained in the certificate we validated. | 04:06 |
johnsom | For example, the common name can be the user's e-mail address | 04:06 |
bzhao__ | johnsom: let me re-taste you said. Oh, | 04:06 |
bzhao__ | johnsom: Thanks, thanks very much, michael. It make sence to me . It is true that is valuable to implement. | 04:07 |
johnsom | bzhao__ Great! yes, I have used this in the past. The first header lets me know they had a validated certificate, the second let's me know the user we validated. | 04:08 |
johnsom | These are powerful security tools | 04:09 |
*** blake has joined #openstack-lbaas | 04:09 | |
bzhao__ | johnsom: Woow, you thought is very advanced, ha . I greatly admire. :) | 04:09 |
bzhao__ | johnsom: Yeah, our customers want this function. | 04:10 |
bzhao__ | johnsom: Thank you for direction. | 04:10 |
johnsom | bzhao__ Ok, sounds good! Let me know if you have additional questions or concerns. I think both RFE are good and should be done. | 04:11 |
rm_work | hmmm what do i need to do to get bash autocomplete to work for the client? >_> | 04:11 |
johnsom | bzhao__ Also, we will get UDP in for Rocky. It's on my list to review again as soon as I can. Looking forward to this | 04:11 |
rm_work | like openstack-cli in general, not just our plugin | 04:11 |
johnsom | rm_work Log out and back in again. It loads, but only in the next bash session. | 04:12 |
bzhao__ | johnsom: OK, the second RFE, is for health check for HTTP head host for dns | 04:12 |
johnsom | If that doesn't work, you are missing the autocomplete package | 04:12 |
bzhao__ | johnsom: Thanks for UDP part. | 04:12 |
rm_work | johnsom: L7 for cert validation IS actually a use-case I need internally <_< | 04:12 |
rm_work | so maybe could work on that | 04:12 |
johnsom | bzhao__ yes, I think the host header is good. | 04:13 |
johnsom | rm_work We are talking about: https://storyboard.openstack.org/#!/story/2002165 | 04:13 |
rm_work | johnsom: i'm talking about on my mac, for the autocomplete | 04:13 |
rm_work | i think i'm missing a package | 04:13 |
*** blake has quit IRC | 04:14 | |
rm_work | andyeah, i was just going through scrollback | 04:14 |
rm_work | so referring to the first thing | 04:14 |
bzhao__ | johnsom: For health check, you suggested that make it more common, I just want to told u that I will figure out that common part for Header. HA, not any more questions now, so the fields could be not only the 2 fields I mentioned in the RFE. | 04:14 |
rm_work | re: [20:54:09] <johnsom>Come to think about it, there might be another task to add L7 support for valid cert connections. | 04:14 |
johnsom | rm_work bash-completion I think | 04:15 |
bzhao__ | johnsom: maybe something like "text" field for common usage | 04:15 |
bzhao__ | johnsom: sorry, I will leave for lunch, happy to see you comment when my back. :) . Thank you Michael. | 04:16 |
rm_work | hmmm bash-completion already installed via brew :/ sad | 04:16 |
rm_work | i wonder if i need to get the openstack cli bash hooks in some special place | 04:16 |
johnsom | bzhao__ I'm not sure I am following. My comment was supporting it for HEAD and GET is the same work, so we should do it at the same time and not limit to HEAD. | 04:16 |
johnsom | bzhao__ I think there is two parameters, one is HTTP version, one is for the host string. | 04:17 |
johnsom | rm_work yeah, it drops a file somewhere. I think it actually comes from cliff package | 04:17 |
rm_work | maybe it just isn't in the right place for OSX/Brew bash-completion | 04:18 |
johnsom | rm_work it is /etc/bash_completion.d for me and the file is osc.bash_completion | 04:18 |
rm_work | yours works in OSX? | 04:18 |
johnsom | No, ubuntu | 04:18 |
rm_work | ah | 04:18 |
rm_work | yeah heh | 04:18 |
rm_work | can look for osc.bash_completion tho | 04:18 |
rm_work | LOL | 04:19 |
rm_work | locate: warning: database ‘/usr/local/var/locate/locatedb’ is more than 8 days old (actual age is 313.2 days) | 04:19 |
rm_work | little bit old | 04:19 |
johnsom | lol, yeah, a "bit" over 8 days | 04:20 |
johnsom | bzhao__ Are we good on the host header for health monitor? Do we need to discuss more? | 04:24 |
rm_work | hmmm seems i don't have osc.bash_completion anywhere :( | 04:25 |
johnsom | Hmm, oddness | 04:25 |
johnsom | Want me to paste bin it for you? | 04:25 |
rm_work | ohh i think i need to generate it | 04:26 |
rm_work | https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/complete.html | 04:26 |
johnsom | Oh, that is a large file | 04:26 |
johnsom | Hmm, it is auto-magic on ubuntu | 04:26 |
*** hongbin has quit IRC | 04:26 | |
rm_work | heh and their requirements are bad | 04:27 |
rm_work | need to manually install oslo_log to use the command | 04:27 |
rm_work | or it explodes | 04:27 |
*** links has joined #openstack-lbaas | 04:28 | |
rm_work | got it! woo | 04:29 |
*** annp has joined #openstack-lbaas | 04:29 | |
johnsom | Yeah, the completion is necessary | 04:30 |
rm_work | yes | 04:31 |
rm_work | i had just made a ton of aliases | 04:31 |
johnsom | Ok, I need to call it a night for IRC. It's 9:30 PM here. I will catch you all in the morning. | 04:35 |
* rm_work waves | 04:35 | |
*** blake has joined #openstack-lbaas | 04:45 | |
*** blake has quit IRC | 04:50 | |
*** SumitNaiksatam_ has joined #openstack-lbaas | 04:52 | |
*** SumitNaiksatam has quit IRC | 04:53 | |
*** SumitNaiksatam_ is now known as SumitNaiksatam | 04:53 | |
bzhao__ | johnsom: Sorry, I'm back for lunch. I saw you comments, I think I didn't get u at that time, but for now, I know what you want. Sure, we can support GET and HEAD. The extend fields are clear. Thanks. For "Are we good on the host header for health monitor? Do we need to discuss more?", I think we get a agreement for this. :) Thank you and good night. | 05:01 |
openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 05:01 |
*** JudeC_ has joined #openstack-lbaas | 05:10 | |
*** JudeC_ has quit IRC | 06:05 | |
*** kobis has joined #openstack-lbaas | 06:23 | |
*** kobis has quit IRC | 06:23 | |
*** kobis has joined #openstack-lbaas | 06:24 | |
*** kobis has quit IRC | 06:24 | |
*** pcaruana has joined #openstack-lbaas | 06:35 | |
*** kobis has joined #openstack-lbaas | 06:51 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 07:04 | |
*** nmanos has joined #openstack-lbaas | 07:07 | |
*** kobis has quit IRC | 07:17 | |
*** kobis has joined #openstack-lbaas | 07:24 | |
*** JudeC_ has joined #openstack-lbaas | 07:32 | |
*** Alexey_Abashkin has joined #openstack-lbaas | 07:34 | |
*** AlexeyAbashkin has quit IRC | 07:35 | |
*** Alexey_Abashkin is now known as AlexeyAbashkin | 07:35 | |
*** AlexeyAbashkin has quit IRC | 07:46 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 07:56 | |
*** JudeC_ has quit IRC | 08:01 | |
*** AlexeyAbashkin has quit IRC | 08:23 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 08:26 | |
*** AlexeyAbashkin has quit IRC | 08:33 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 08:38 | |
openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 08:40 |
*** AlexeyAbashkin has quit IRC | 08:45 | |
*** kiennt26 has quit IRC | 09:07 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 09:31 | |
*** kobis has quit IRC | 09:42 | |
*** links has quit IRC | 09:48 | |
*** AlexeyAbashkin has quit IRC | 09:50 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 10:02 | |
*** links has joined #openstack-lbaas | 10:05 | |
*** AlexeyAbashkin has quit IRC | 10:13 | |
*** Alexey_Abashkin has joined #openstack-lbaas | 10:13 | |
*** Alexey_Abashkin is now known as AlexeyAbashkin | 10:16 | |
*** longkb1 has quit IRC | 10:16 | |
*** longkb has joined #openstack-lbaas | 10:17 | |
*** kobis has joined #openstack-lbaas | 10:18 | |
*** AlexeyAbashkin has quit IRC | 10:23 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 10:25 | |
*** Alexey_Abashkin has joined #openstack-lbaas | 10:38 | |
*** AlexeyAbashkin has quit IRC | 10:39 | |
*** Alexey_Abashkin is now known as AlexeyAbashkin | 10:39 | |
*** links has quit IRC | 10:39 | |
*** links has joined #openstack-lbaas | 10:56 | |
*** AlexeyAbashkin has quit IRC | 11:01 | |
*** yamamoto has quit IRC | 11:14 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 11:51 | |
*** amuller has joined #openstack-lbaas | 11:58 | |
*** yamamoto has joined #openstack-lbaas | 12:10 | |
*** yamamoto has quit IRC | 12:21 | |
*** longkb has quit IRC | 12:27 | |
*** fnaval has joined #openstack-lbaas | 12:31 | |
*** longkb has joined #openstack-lbaas | 12:39 | |
openstackgerrit | Nir Magnezi proposed openstack/octavia master: DNM: testing v2 scenario gate https://review.openstack.org/573218 | 12:42 |
openstackgerrit | Nir Magnezi proposed openstack/octavia master: DNM: testing v2 scenario gate https://review.openstack.org/573218 | 12:44 |
openstackgerrit | Nir Magnezi proposed openstack/octavia master: DNM: Testing gate addition https://review.openstack.org/573219 | 12:47 |
*** yamamoto has joined #openstack-lbaas | 12:49 | |
*** rcernin has quit IRC | 12:56 | |
*** b_bezak has joined #openstack-lbaas | 12:57 | |
*** rpietrzak has joined #openstack-lbaas | 12:59 | |
*** longkb has quit IRC | 13:00 | |
openstackgerrit | Rafal Pietrzak proposed openstack/octavia master: Signed-off-by: Rafal Pietrzak <rafal.pietrzak@pl.ibm.com> Author: Rafal Pietrzak <rafal.pietrzak@pl.ibm.com> https://review.openstack.org/573227 | 13:06 |
*** AlexeyAbashkin has quit IRC | 13:08 | |
*** Alexey_Abashkin has joined #openstack-lbaas | 13:08 | |
*** Alexey_Abashkin is now known as AlexeyAbashkin | 13:10 | |
openstackgerrit | Rafal Pietrzak proposed openstack/octavia master: Adding support for the octavia listener X-Forwarded-Proto header insertion. Author: Rafal Pietrzak <rafal.pietrzak@pl.ibm.com> https://review.openstack.org/573227 | 13:12 |
openstackgerrit | Rafal Pietrzak proposed openstack/octavia master: Adding support for the octavia listener X-Forwarded-Proto header insertion. https://review.openstack.org/573227 | 13:13 |
*** atoth has joined #openstack-lbaas | 13:19 | |
*** openstackgerrit has quit IRC | 13:34 | |
*** AlexeyAbashkin has quit IRC | 13:43 | |
*** b_bezak has quit IRC | 13:49 | |
*** b_bezak has joined #openstack-lbaas | 13:50 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 13:50 | |
*** b_bezak has quit IRC | 13:54 | |
*** Alexey_Abashkin has joined #openstack-lbaas | 14:02 | |
*** AlexeyAbashkin has quit IRC | 14:02 | |
*** Alexey_Abashkin is now known as AlexeyAbashkin | 14:02 | |
*** links has quit IRC | 14:23 | |
*** Alexey_Abashkin has joined #openstack-lbaas | 14:38 | |
*** AlexeyAbashkin has quit IRC | 14:41 | |
*** Alexey_Abashkin is now known as AlexeyAbashkin | 14:41 | |
xgerman_ | rm_work: johnsom PSA: http://lists.openstack.org/pipermail/openstack-dev/2018-June/131252.html | 14:46 |
xgerman_ | no impact for us but FYI | 14:46 |
xgerman_ | (and might hose lbaas_v2) | 14:47 |
*** rpietrzak has quit IRC | 14:57 | |
johnsom | I don't see that we use that field in either repo | 15:02 |
xgerman_ | drivers? | 15:12 |
johnsom | At least not in our repo | 15:12 |
xgerman_ | probably worth to let the driver authors know so they can adjust (not that I think they will) | 15:13 |
*** Alexey_Abashkin has joined #openstack-lbaas | 15:20 | |
*** AlexeyAbashkin has quit IRC | 15:22 | |
*** Alexey_Abashkin is now known as AlexeyAbashkin | 15:22 | |
*** AlexeyAbashkin has quit IRC | 15:22 | |
*** AlexeyAbashkin has joined #openstack-lbaas | 15:23 | |
-openstackstatus- NOTICE: Zuul update for Ansible 2.5 in progress. Scheduler crashed as unexpected side effect of pip upgrade. Will be back and running shortly. | 15:30 | |
*** kobis has quit IRC | 15:31 | |
*** Alexey_Abashkin has joined #openstack-lbaas | 15:37 | |
*** AlexeyAbashkin has quit IRC | 15:37 | |
*** Alexey_Abashkin is now known as AlexeyAbashkin | 15:37 | |
*** AlexeyAbashkin has quit IRC | 15:53 | |
*** kobis has joined #openstack-lbaas | 15:56 | |
*** JudeC_ has joined #openstack-lbaas | 16:01 | |
-openstackstatus- NOTICE: The zuul upgrade to ansible 2.5 is complete and zuul is running again. Changes uploaded or approved between 15:25 and 15:45 will need to be rechecked. Please report any problems in #openstack-infra | 16:10 | |
*** yamamoto has quit IRC | 16:14 | |
*** kobis1 has joined #openstack-lbaas | 16:17 | |
*** kobis has quit IRC | 16:18 | |
*** kobis1 has quit IRC | 16:19 | |
*** kobis has joined #openstack-lbaas | 16:20 | |
*** yamamoto has joined #openstack-lbaas | 16:21 | |
*** SumitNaiksatam has quit IRC | 16:25 | |
*** kobis has quit IRC | 16:25 | |
*** yamamoto has quit IRC | 16:26 | |
*** JudeC_ has quit IRC | 16:29 | |
*** kobis has joined #openstack-lbaas | 16:42 | |
*** kobis has quit IRC | 16:44 | |
*** openstackgerrit has joined #openstack-lbaas | 16:46 | |
*** SumitNaiksatam has joined #openstack-lbaas | 16:52 | |
*** JudeC_ has joined #openstack-lbaas | 17:12 | |
*** yamamoto has joined #openstack-lbaas | 17:23 | |
*** sshank has joined #openstack-lbaas | 17:30 | |
*** atoth has quit IRC | 17:30 | |
*** yamamoto has quit IRC | 17:31 | |
*** amuller has quit IRC | 17:31 | |
*** openstackgerrit has quit IRC | 17:34 | |
cgoncalves | where/how can I see tracebacks in a devstack env? journalctl isn't showing any and I'm sure octavia is crashing | 17:35 |
johnsom | They should all go in the journal, "journalctl -u devstack@o-cw | less" | 17:36 |
xgerman_ | I see them all in journalctl — previously they were oin /opt/stack/logs | 17:36 |
cgoncalves | right, that's what I'm doing | 17:37 |
cgoncalves | journalctl -a -f -u devstack@o-api to be precise but doesn't make any difference | 17:37 |
johnsom | Otherwise, there have been a few times that a try/except block "eats" the exception and traceback | 17:37 |
johnsom | Look around the code you are expecting to fail and see if there might be a try block eating it | 17:38 |
cgoncalves | no try block | 17:39 |
*** atoth has joined #openstack-lbaas | 17:42 | |
cgoncalves | I'll keep looking. I must be missing something trivial | 17:44 |
*** atoth has quit IRC | 17:46 | |
rm_work | cgoncalves: debug = True ? | 17:48 |
rm_work | does it hide tracebacks otherwise? | 17:49 |
rm_work | actually I feel like it shouldn't... but | 17:49 |
johnsom | Yeah, they should come out at a level higher than debug | 17:49 |
johnsom | I think they are all at ERROR level | 17:50 |
*** atoth has joined #openstack-lbaas | 17:50 | |
cgoncalves | meh, found it, stupid of me. the caller is indeed wrapping catching all exception and not printing the exception msg | 17:52 |
johnsom | My fun this morning is making our DELETED vs delete story consistent... | 17:55 |
johnsom | This kind of sums it up: | 17:55 |
johnsom | https://www.irccloud.com/pastebin/VUkQtH3L/ | 17:55 |
cgoncalves | rm_work, the barbican acl patch actually doesn't work. when I ran it the other day I did as admin user :/ | 17:55 |
rm_work | cgoncalves: hah, ok yeah, i was honestly super surprised | 17:57 |
rm_work | johnsom: T_T | 17:58 |
xgerman_ | cgoncalves: you probably get a lol out of that. I need to write an Octavia installer for OSP12… | 18:01 |
rm_work | uuuuuuhh | 18:12 |
rm_work | I just had a case where the user didn't set a HM check method | 18:12 |
rm_work | and what got into the haproxy config was: | 18:12 |
rm_work | option httpchk None / | 18:13 |
rm_work | or maybe it was because he updated the method | 18:13 |
rm_work | err | 18:14 |
rm_work | updated another field | 18:14 |
*** rpietrzak has joined #openstack-lbaas | 18:16 | |
*** sshank has quit IRC | 18:20 | |
*** kobis has joined #openstack-lbaas | 18:22 | |
*** SumitNaiksatam has quit IRC | 18:23 | |
*** SumitNaiksatam has joined #openstack-lbaas | 18:29 | |
rm_work | johnsom: yeah we allow nulling-out the http_method field on a HM on update | 18:29 |
rm_work | originally we default it to GET (correctly) | 18:29 |
rm_work | but if you actually pass 'http_method': None | 18:29 |
rm_work | during an update | 18:29 |
rm_work | it'll null it | 18:29 |
rm_work | and then the template happily puts "None" into the haproxy config | 18:30 |
rm_work | which breaks healthchecks | 18:30 |
johnsom | Neat | 18:30 |
rm_work | IMO this should not be possible | 18:30 |
rm_work | or the default should reassert? | 18:30 |
rm_work | could just drop it into the template... | 18:31 |
rm_work | but I think we should reassert the default somehow | 18:31 |
rm_work | just struggling to decide where/how | 18:31 |
rm_work | it's basically impossible to test updates in our functional testing | 18:32 |
rm_work | OH WAIT NO IT ISN'T NOW | 18:32 |
rm_work | DB gets updated after the driver call! | 18:32 |
rm_work | O_o | 18:32 |
rm_work | o_O | 18:32 |
rm_work | O_o | 18:32 |
rm_work | johnsom: so we changed our "UPDATE" model | 18:33 |
rm_work | we persist those in the DB without relying on the controller-worker to do the update | 18:33 |
rm_work | so our functional tests just became 100% more useful | 18:33 |
cgoncalves | xgerman_, well you could try backporting the octavia bits from tripleo queens (tripleo-heat-templates, tripleo-common, puppet-octavia) | 18:34 |
xgerman_ | yeah, I probably will just write some shell scripts… | 18:36 |
rm_work | xgerman_: T_T | 18:37 |
*** kobis has quit IRC | 18:37 | |
rpietrzak | Hi, I just want thanks you Michael for reviewing my very, very short proposal regarding the X-Forwarded-Proto header;) I will follow you guidance, and ofcourse this test should be based on the listner type, as you have written. Thanks :) | 18:38 |
rm_work | johnsom: what do you think should happen if a user passes in "None" as the http_method on an update explicitly? | 18:40 |
rm_work | should we just ignore it | 18:40 |
rm_work | or should we "reset to default"? | 18:40 |
rm_work | I kinda want to have a "NOT NULL" on the DB for that field too <_< | 18:42 |
rm_work | eh probably easier to backport without tho | 18:43 |
johnsom | rpietrzak No problem. Let me know if we can help in any way. Thanks for the contribution! | 18:43 |
johnsom | rm_work Just a sec, let me look at this. | 18:44 |
johnsom | rm_work So this isn't that it wasn't specified (Unset) it's that they actually passed in None? | 18:45 |
rm_work | yes | 18:45 |
rm_work | so we update the field in the DB to "None" | 18:45 |
rm_work | or NULL rather | 18:45 |
rm_work | and the models -> haproxy template don't have defaults | 18:46 |
rm_work | so: "option httpchk None /" | 18:46 |
rm_work | maybe a fix in the haproxy template to default :/ | 18:46 |
rm_work | that's where I am right now | 18:46 |
johnsom | rm_work Umm, how???? https://github.com/openstack/octavia/blob/master/octavia/api/v2/types/health_monitor.py#L122 | 18:46 |
rm_work | Enums must not check against None? | 18:47 |
rm_work | http://paste.openstack.org/show/722920/ | 18:47 |
rm_work | add this to octavia/tests/functional/api/v2/test_health_monitor.py | 18:47 |
rm_work | WSMEEEEEE | 18:48 |
johnsom | Yeah looking at that now: https://github.com/openstack/wsme/blob/master/wsme/types.py#L282 | 18:49 |
*** SumitNaiksatam has quit IRC | 18:50 | |
*** SumitNaiksatam has joined #openstack-lbaas | 18:50 | |
rm_work | I just went into my DB and did a mass update | 18:51 |
rm_work | UPDATE health_monitor SET http_method = 'GET' WHERE http_method IS NULL; | 18:51 |
rm_work | to try to stave off this problem | 18:51 |
rm_work | because of the way the UI here sends updates, like 90% of them were this way <_< | 18:51 |
johnsom | rm_work Something doesn't add up. That wsme check is pretty straight forward | 18:54 |
rm_work | i'd think so yes | 18:54 |
rm_work | walking through my test now... | 18:55 |
johnsom | https://github.com/openstack/wsme/blob/master/wsme/types.py#L305 | 18:55 |
rm_work | nope, it's gotta be WSME | 18:56 |
rm_work | literally looking at what gets passed into the `put` function in the controller | 18:56 |
rm_work | it's got `http_method = None` | 18:56 |
rm_work | not Unset | 18:56 |
*** kobis has joined #openstack-lbaas | 18:58 | |
*** beisner-sick is now known as beisner | 19:00 | |
rm_work | johnsom: http://paste.openstack.org/show/722921/ | 19:01 |
rm_work | at the very bottom of octavia/tests/unit/api/v2/types/test_health_monitors.py | 19:01 |
rm_work | clearly shows WSME allowing None | 19:01 |
rm_work | ah it should be ValueError, whatever | 19:02 |
rm_work | the point is it doesn't even raise anything | 19:02 |
rm_work | errrr, it doesn't even run the validate | 19:03 |
rm_work | O_o | 19:03 |
johnsom | That is "not good" as the wsme comment code even calls out the None, so either it's a bug or we aren't doing something right | 19:03 |
*** rm_mobile has joined #openstack-lbaas | 19:08 | |
rm_mobile | Ah nm | 19:08 |
rm_mobile | Was just too early. It does validate the children... | 19:08 |
rm_mobile | But short circuits for both Unset *and* None | 19:08 |
rm_mobile | Look at types.py 361 | 19:09 |
*** rm_mobile has quit IRC | 19:09 | |
*** rm_mobile has joined #openstack-lbaas | 19:09 | |
rm_work | i think because theoretically you might want to allow a purposeful unsetting of an Enum | 19:16 |
rm_work | of course, IMO you would just add None to the Enum <_< | 19:16 |
*** mugsie has quit IRC | 19:19 | |
*** mugsie has joined #openstack-lbaas | 19:21 | |
*** mugsie has quit IRC | 19:21 | |
*** mugsie has joined #openstack-lbaas | 19:21 | |
*** mugsie has quit IRC | 19:21 | |
*** mugsie has joined #openstack-lbaas | 19:21 | |
*** mugsie has quit IRC | 19:21 | |
*** mugsie has joined #openstack-lbaas | 19:21 | |
*** atoth has quit IRC | 19:22 | |
*** mugsie has quit IRC | 19:22 | |
*** Guest68045 has joined #openstack-lbaas | 19:24 | |
*** Guest68045 has quit IRC | 19:24 | |
*** mugsie_ has joined #openstack-lbaas | 19:24 | |
*** mugsie_ has quit IRC | 19:27 | |
*** mugsie_ has joined #openstack-lbaas | 19:27 | |
*** mugsie_ is now known as _mugsie | 19:27 | |
*** _mugsie is now known as mugsie_ | 19:28 | |
*** kobis has quit IRC | 19:51 | |
*** rpietrzak has quit IRC | 20:09 | |
johnsom | Crumb | 20:16 |
rm_work | right? | 20:20 |
rm_work | and some other words | 20:20 |
rm_work | that are non-baked-goods related | 20:21 |
johnsom | Well, that was double. One for wsme, one for a bug I just found in the tempest code I wrote | 20:21 |
rm_work | ah | 20:21 |
johnsom | I wish we could fix wsme, but I bet that changing that would break *someone* | 20:22 |
*** openstackgerrit has joined #openstack-lbaas | 20:31 | |
openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Fix a potential IP address conflict https://review.openstack.org/573431 | 20:31 |
johnsom | Since the tests run in parallel I hit an issue where two LBs happened to pick the same random fixed IP at the same time. | 20:32 |
johnsom | rm_work FYI, this https://review.openstack.org/#/c/561369 passes the tempest tests local and live. | 20:35 |
rm_work | LOL is that seriously a thing | 20:36 |
rm_work | johnsom: so i am in favor of updating the renderer to default the value it passes to the template | 20:36 |
rm_work | let the DB get nulled I guess | 20:36 |
rm_work | assume the drivers can handle defaulting | 20:37 |
rm_work | ? | 20:37 |
*** pcaruana has quit IRC | 20:39 | |
johnsom | So, if the type of the HM is TCP it should be None right? | 20:40 |
johnsom | Frankly, we should remove the defaults from the WSME types and validate/default in the controller code too. It's lame to be setting url_path for TCP health monitors | 20:48 |
johnsom | rm_work Should I spin a patch for HM? | 20:49 |
rm_work | johnsom: err, wait, let me see what it looks like | 20:50 |
rm_work | yeah ok so you're right re: db | 20:51 |
rm_work | so yeah i think the answer is the renderer | 20:52 |
rm_work | johnsom: if you've got it open and/or in-progress, sure | 20:52 |
johnsom | Yep | 20:52 |
openstackgerrit | Rafal Pietrzak proposed openstack/octavia master: Adding support for the octavia listener X-Forwarded-Proto header insertion. https://review.openstack.org/573227 | 21:02 |
rm_work | ugh seriously how hard would it be to do TLS Reencryption >_> | 21:29 |
johnsom | not that bad | 21:29 |
rm_work | yeah seems to just need a ca-file | 21:32 |
rm_work | and ... that's it | 21:32 |
rm_work | ffff | 21:32 |
rm_work | i might just do this | 21:32 |
rm_work | it's so dumb that we still don't support something so trivial and yet so important | 21:32 |
johnsom | I have open questions about CRLs | 21:32 |
rm_work | CRLs are shitty | 21:32 |
johnsom | Bonus would be supporting a client cert too | 21:32 |
rm_work | use short-lived certs | 21:32 |
johnsom | CRLs are not great, but we can't really do OCSP... | 21:33 |
rm_work | Anchor had the right approach IMO <_< | 21:33 |
rm_work | i know you weren't a fan, but.... | 21:33 |
johnsom | errr, I'm not a believer in short lived certs | 21:33 |
johnsom | But this is philosophy beer talk | 21:34 |
rm_work | yes | 21:34 |
johnsom | Ugh: constants.HEALTH_MONITOR_DEFAULT_EXPECTED_CODES | 21:34 |
rm_work | yeah | 21:34 |
johnsom | Why didn't we write a novel | 21:34 |
rm_work | lol | 21:34 |
rm_work | we still can ;) | 21:34 |
johnsom | Just trying to figure out how to fit it on a line without being too dumb | 21:34 |
rm_work | paste me the block | 21:35 |
johnsom | https://www.irccloud.com/pastebin/ZK1pRkfR/ | 21:37 |
johnsom | Gets me: | 21:37 |
johnsom | https://www.irccloud.com/pastebin/nutRetGG/ | 21:38 |
johnsom | https://www.irccloud.com/pastebin/Dq2G83XR/ | 21:39 |
johnsom | That is what I'm going with... | 21:39 |
rm_work | rofl | 21:42 |
rm_work | why not just http://paste.openstack.org/show/722926/ | 21:44 |
johnsom | Doesn't that whine that you have a hanging (? | 21:44 |
rm_work | no | 21:45 |
rm_work | we do that all over the place | 21:45 |
johnsom | Ah, yeah, I must be thinking of something else | 21:47 |
*** SumitNaiksatam has quit IRC | 22:23 | |
*** rcernin has joined #openstack-lbaas | 22:24 | |
johnsom | Ugh this enum thing..... | 22:52 |
johnsom | http_method is not a valid option for health monitors of type None | 22:52 |
johnsom | That exception is my new code, but the fact that it took None for the "Mandatory" type field is .... | 22:52 |
rm_work | <_< | 23:11 |
rm_work | so, literally was just gonna do this | 23:18 |
rm_work | like | 23:18 |
rm_work | two line change | 23:18 |
rm_work | johnsom: http://paste.openstack.org/show/722930/ | 23:18 |
johnsom | Yeah, it's bigger than that.... Plus I'm fixing the dumb defaults | 23:18 |
rm_work | i mean... at the renderer | 23:18 |
rm_work | obviously tests need updating, but | 23:18 |
rm_work | <_< | 23:18 |
rm_work | leave it up to the driver | 23:19 |
johnsom | https://www.irccloud.com/pastebin/X3qprx5z/ | 23:19 |
rm_work | it doesn't matter if we always pass something to http_method, because the template won't render the line at all if it's not of type HTTP | 23:19 |
rm_work | yeah you're getting into the business of a whole validation layer on the front side of the API | 23:20 |
johnsom | But we should be validating the API input to match or API spec. Right now that doesn't line up. | 23:20 |
rm_work | I was thinking might lighter touch | 23:20 |
rm_work | *much lighter | 23:20 |
rm_work | yeah, I suppose so ;) | 23:20 |
*** fnaval has quit IRC | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!