| *** goldyfruit_ has joined #openstack-lbaas | 02:35 | |
| *** goldyfruit_ has quit IRC | 02:36 | |
| *** goldyfruit_ has joined #openstack-lbaas | 02:36 | |
| *** rcernin_ has joined #openstack-lbaas | 03:13 | |
| *** rcernin has quit IRC | 03:13 | |
| *** hongbin has joined #openstack-lbaas | 03:34 | |
| *** rcernin_ has quit IRC | 03:56 | |
| *** rcernin has joined #openstack-lbaas | 04:00 | |
| *** goldyfruit_ has quit IRC | 04:24 | |
| *** goldyfruit_ has joined #openstack-lbaas | 04:24 | |
| *** hongbin has quit IRC | 04:24 | |
| *** goldyfruit_ has quit IRC | 04:48 | |
| *** tkajinam has quit IRC | 04:52 | |
| *** goldyfruit_ has joined #openstack-lbaas | 05:13 | |
| *** tkajinam has joined #openstack-lbaas | 05:23 | |
| *** tkajinam has quit IRC | 05:58 | |
| *** pcaruana has joined #openstack-lbaas | 06:00 | |
| *** pcaruana has quit IRC | 06:14 | |
| *** gcheresh has joined #openstack-lbaas | 06:38 | |
| *** tkajinam has joined #openstack-lbaas | 06:41 | |
| *** tkajinam_ has joined #openstack-lbaas | 06:52 | |
| *** rcernin has quit IRC | 06:54 | |
| *** tkajinam has quit IRC | 06:55 | |
| *** henriqueof has quit IRC | 07:02 | |
| *** henriqueof1 has joined #openstack-lbaas | 07:02 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 07:50 | |
| *** maciejjozefczyk_ has joined #openstack-lbaas | 07:50 | |
| *** maciejjozefczyk_ has quit IRC | 07:50 | |
| maciejjozefczyk | Happy New Year! | 08:06 |
|---|---|---|
| gthiemonge | HNY! ;-) | 08:12 |
| *** tesseract has joined #openstack-lbaas | 08:17 | |
| rm_work | o/ | 08:18 |
| *** pcaruana has joined #openstack-lbaas | 08:29 | |
| *** AlexStaf has joined #openstack-lbaas | 08:30 | |
| *** lemko has joined #openstack-lbaas | 08:31 | |
| *** rpittau|afk is now known as rpittau | 08:36 | |
| *** tridde has quit IRC | 08:53 | |
| *** trident has joined #openstack-lbaas | 08:55 | |
| *** tkajinam_ has quit IRC | 08:58 | |
| *** dmellado has quit IRC | 09:07 | |
| *** dmellado has joined #openstack-lbaas | 09:08 | |
| *** sapd1_x has joined #openstack-lbaas | 10:02 | |
| *** sapd1_x has quit IRC | 10:49 | |
| *** rpittau is now known as rpittau|bbl | 11:16 | |
| *** lemko has quit IRC | 11:20 | |
| *** servagem has joined #openstack-lbaas | 13:06 | |
| *** goldyfruit_ has quit IRC | 13:20 | |
| *** rpittau|bbl is now known as rpittau | 13:26 | |
| *** pcaruana has quit IRC | 14:20 | |
| *** TrevorV has joined #openstack-lbaas | 14:32 | |
| *** pcaruana has joined #openstack-lbaas | 14:32 | |
| *** ramishra has quit IRC | 15:16 | |
| *** ramishra has joined #openstack-lbaas | 15:17 | |
| *** gcheresh has quit IRC | 15:39 | |
| *** AlexStaf has quit IRC | 16:06 | |
| *** KeithMnemonic has quit IRC | 16:30 | |
| *** KeithMnemonic has joined #openstack-lbaas | 16:31 | |
| *** pcaruana has quit IRC | 17:05 | |
| xgerman | stumbled across this primer: https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236 — pretty good and we might want to link to it | 17:07 |
| xgerman | also Happy New Year!! | 17:07 |
| *** rpittau is now known as rpittau|afk | 17:09 | |
| johnsom | Happy New Year! | 17:18 |
| *** pcaruana has joined #openstack-lbaas | 17:28 | |
| rm_work | We should probably cut a release for the client | 17:46 |
| johnsom | Yeah, some nice features there. Should we bump to 2.x since it no longer supports py2? | 17:59 |
| rm_work | Ah that's not a bad call | 18:02 |
| rm_work | I like it | 18:02 |
| johnsom | Yeah, I think it is a good idea. | 18:04 |
| johnsom | rm_work Are you handling that, or should I work on it? | 18:15 |
| rm_work | I'll do it in a moment I guess | 18:17 |
| rm_work | I always make you do that stuff lol | 18:17 |
| rm_work | Actually don't we have a release manager? :D | 18:17 |
| * rm_work glances at cgoncalves | 18:17 | |
| *** adeberg has joined #openstack-lbaas | 18:19 | |
| johnsom | He's on vacation, lol | 18:21 |
| rm_work | heh | 18:24 |
| johnsom | I have a minute and can do it. I'm just working on doing some reviews this morning. | 18:24 |
| rm_work | I'm trying to figure out how to properly convert this stuff from running on Apache mod_wsgi to uwsgi | 18:25 |
| rm_work | Because mod_wsgi is a shitshow | 18:25 |
| rm_work | Specifically around python versions | 18:26 |
| johnsom | Yes it is | 18:27 |
| rm_work | uwsgi has about 8000 config options, and the documentation just kinda throws them all in a big list lol | 18:27 |
| johnsom | yep | 18:27 |
| johnsom | The Apache side is pretty straight forward, just proxying through to a uwsgi unix socket: | 18:28 |
| johnsom | ProxyPass "/load-balancer" "unix:/var/run/uwsgi/octavia-wsgi.socket|uwsgi://uwsgi-uds-octavia-wsgi/" retry=0 | 18:29 |
| rm_work | Ah no, I'm dropping Apache altogether | 18:29 |
| rm_work | JUST uwsgi | 18:30 |
| rm_work | We already run another layer above it | 18:30 |
| johnsom | Ah. Well, the devstack uwsgi config is https://zuul.opendev.org/t/openstack/build/6305f48fc0b34c908b459b8b3585a82e/log/controller/logs/etc/octavia/octavia-uwsgi.ini.gz | 18:30 |
| johnsom | Yeah, removing layers is always a good idea if you can | 18:31 |
| johnsom | I don't remember the tweaks i have done to that in the past | 18:32 |
| rm_work | Oh we run octavia-wsgi not octavia-api | 18:33 |
| rm_work | Interesting | 18:33 |
| rm_work | Didn't realize lol | 18:33 |
| rm_work | Never followed that goal | 18:33 |
| johnsom | Yeah, the API is a very light weight server I highly don't recommend in production. | 18:33 |
| johnsom | We use the uwsgi | 18:33 |
| rm_work | Right but we run octavia-api as the wsgi file lol | 18:34 |
| rm_work | Which apparently has been working somehow... | 18:34 |
| johnsom | Yeah, I think there is a wrapper in there | 18:34 |
| johnsom | Ah, no, I remember now, the setup tools creates the bin file that has the wrapper | 18:35 |
| johnsom | We should update this: https://docs.openstack.org/octavia/latest/admin/apache-httpd.html | 18:35 |
| *** pvradu has joined #openstack-lbaas | 18:36 | |
| johnsom | This is the tricky bit: https://github.com/openstack/octavia/blob/master/setup.cfg#L42 | 18:36 |
| johnsom | Seems like it's been forever since we did that stuff, but.... | 18:36 |
| rm_work | Yeah... | 18:40 |
| *** pvradu has quit IRC | 18:40 | |
| *** trident has quit IRC | 18:48 | |
| *** trident has joined #openstack-lbaas | 18:49 | |
| adeberg | Hey all :) anyone could help me understand how the "Operating status" supposed to behave ? Any doc/portion of code I could read ? I keep having my LB as "Offline" while they seem to work fine (members+pool+LB are offline but configured HM displays online). Don't know where to start looking | 18:50 |
| johnsom | adeberg It's in the documentation here: https://docs.openstack.org/api-ref/load-balancer/v2/index.html#status-codes | 18:51 |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Support hacking 2.0.0 https://review.opendev.org/699302 | 18:52 |
| adeberg | thx, I'll give it another look, but the admin_state_up is true for all my members (unless administratively disabled means otherwise) | 18:53 |
| johnsom | adeberg I am happy to answer questions too. | 18:54 |
| johnsom | admin_state_up is the enable/disable setting (terminology we had to inherit from neutron). admin_state_up of False would lead to operating status DOWN once the provisioning status is done being "PENDING_UPDATE". | 18:55 |
| johnsom | haleyb We ARE the cool kids... lol | 18:55 |
| haleyb | :) | 18:55 |
| adeberg | then, what does mean "administratively disabled" ? | 18:55 |
| johnsom | adeberg "administratively disabled is the description in the documentation for the "OFFLINE" status in our API | 18:56 |
| adeberg | johnsom, what I don't get here, is that, from the amphora (within the haproxy netns) I'm able to curl my backend/member but the member still shows OFFLINE. What's the process responsible for updating that state ? | 18:59 |
| johnsom | adeberg Sorry, I should have looked at the docs again. I meant OFFLINE when I said DOWN in my last comment. | 18:59 |
| johnsom | adeberg Hmm, that is interesting. It likely means your health manager controllers are not receiving the health updates from the amphora. Check you lb-mgmt-net and your controller IP/port list setting: https://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list | 19:02 |
| adeberg | johnsom, will do | 19:02 |
| johnsom | adeberg If you enable debug on the health manager, you should see "packet received" log messages from the amphora. | 19:03 |
| adeberg | johnsom, ok I don't see any only "Submitting periodic callback" lines so that might be my issue | 19:05 |
| johnsom | adeberg Yep, that is the problem | 19:05 |
| adeberg | johnsom, to give a bit of context I was trying to upgrade a kolla-ansible lab from rocky to stein and forgot to double check everything was working *before* | 19:06 |
| johnsom | Ah, yeah, for some reason kolla-ansible doesn't setup the lb-mgmt-net for you like the other deploy tools do | 19:07 |
| adeberg | johnsom, thx a lot. Will keep fixing my deployment and let you know :) | 19:08 |
| *** gcheresh has joined #openstack-lbaas | 19:14 | |
| rm_work | though in that case wouldn't the entire controlplane not work? ie, no configuration of LBs either? | 19:24 |
| rm_work | with no mgmt net the whole thing should just fail to function | 19:24 |
| johnsom | Yeah, but it could be a security group or one-way route or something... | 19:42 |
| openstackgerrit | Merged openstack/octavia master: Fix diskimage-create.sh for Debian https://review.opendev.org/700012 | 19:45 |
| *** gcheresh has quit IRC | 19:52 | |
| johnsom | proposed openstack/releases master: Release python-octaviaclient 2.0.0 https://review.opendev.org/701449 | 20:02 |
| *** AlexStaf has joined #openstack-lbaas | 20:28 | |
| *** pcaruana has quit IRC | 20:41 | |
| *** AlexStaf has quit IRC | 20:43 | |
| *** maciejjozefczyk has quit IRC | 20:54 | |
| *** servagem has quit IRC | 20:54 | |
| johnsom | Ok, my afternoon fun. Get creative to clean up abandoned ports nova leaves when someone deletes an amp via OSC and then does a failover... I know I can check the security group to get the base port list, we do this on LB delete. Not sure about member ports yet. I need to see if nova leaves those around sometimes as well. | 21:08 |
| *** pvradu has joined #openstack-lbaas | 21:13 | |
| *** gcheresh has joined #openstack-lbaas | 21:23 | |
| rm_work | i keep meaning to write something that'll do a full graph audit | 21:27 |
| rm_work | and just show me any objects in our project that aren't supposed to be there :D | 21:28 |
| johnsom | Yeah. I'm thinking we should start using the port description field to store the LB ID. Since tags are an extension. | 21:28 |
| johnsom | My current problem is, find them when I have no amphora record. Plus we don't store the member subnet port IDs anywhere. So, tricky to tie back to the right LB. | 21:29 |
| rm_work | yeah | 21:31 |
| rm_work | eugh how the heck do i pass a config file path to the octavia-api when it's being run as a wsgi app >_> | 21:32 |
| rm_work | uwsgi doesn't seem to want to let me pass config params through, maybe that's just not something wsgi expects? | 21:32 |
| johnsom | Nova seems to clean up the lb-mgmt-net port and the member subnet ports on "server delete", but not the vrrp (base) port. But, as I have seen, I don't really trust it to do any of them | 21:32 |
| *** rcernin has joined #openstack-lbaas | 21:37 | |
| *** rcernin has quit IRC | 21:37 | |
| *** rcernin has joined #openstack-lbaas | 21:38 | |
| johnsom | Hmm, all of the logs are dumping in compressed garbage for me.... | 21:39 |
| johnsom | i.e. https://zuul.opendev.org/t/openstack/build/e8f193e3cf9b435680c22ec2bf65bf25/log/controller/logs/screen-o-cw.txt.gz | 21:39 |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Remove all usage of six library https://review.opendev.org/701290 | 21:39 |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Remove all usage of six library https://review.opendev.org/701290 | 21:41 |
| johnsom | rm_work well somehow the API is picking up our config file: https://zuul.opendev.org/t/openstack/build/f3da5dfca22d4e72933a44bddcea3989/log/controller/logs/screen-o-api.txt.gz#3 | 21:43 |
| rm_work | yeah, i think because it's in the standard loc? | 21:43 |
| rm_work | oh | 21:43 |
| rm_work | THAT is the uwsgi config | 21:43 |
| rm_work | i mean the octavia config | 21:43 |
| *** gcheresh has quit IRC | 21:44 | |
| rm_work | https://zuul.opendev.org/t/openstack/build/f3da5dfca22d4e72933a44bddcea3989/log/controller/logs/screen-o-api.txt.gz#44 | 21:44 |
| openstackgerrit | Merged openstack/octavia master: Fix tests to correctly call reset_mock() https://review.opendev.org/699287 | 21:44 |
| rm_work | whelp | 21:46 |
| rm_work | i would have -1'd that | 21:46 |
| rm_work | that's what i get for not reviewing actively enough I guess | 21:46 |
| *** pvradu has quit IRC | 21:47 | |
| haleyb | rm_work: ^^ that change for reset_mock ? | 21:47 |
| rm_work | yes | 21:48 |
| rm_work | I commented just now, but not useful lol | 21:48 |
| rm_work | when i saw the comment name here i knew exactly what happened T_T | 21:48 |
| rm_work | those lines are just dead code, someone copy/pasted around badly | 21:49 |
| *** TrevorV has quit IRC | 21:49 | |
| rm_work | i thought i got all of them already but apparently not | 21:49 |
| haleyb | rm_work: i can see the bottom one can be removed, but the top looks correct as there is an assert after it | 21:51 |
| rm_work | ask yourself this: the test passed when it was a completely bogus noop line. is a reset necessary for the test to pass? :D | 21:52 |
| adeberg | @johnsom, turns out it was some routing issue, after switching back to the lb-mgmt-net ip for the controller_ip_port_list and making the health-manager bind on it everything's fine | 21:53 |
| haleyb | rm_work: yeah, you're right, typically these tests are assert_called_once_with() which would have failed, it just got lucky it's additive | 21:54 |
| rm_work | so it's *still* a dead line | 21:55 |
| johnsom | adeberg Ok. So, probably the amphora didn't have a proper route back on the lb-mgmt-net. check your neutron subnet configuration for the subnet the amphora sit on the lb-mgmt-net. Either the gateway is pointing to a router that doesn't know the way back, is missing, or there are host routes hijacking it. | 21:55 |
| johnsom | stating the obvious, but maybe giving some ideas to check | 21:57 |
| adeberg | johnsom, at this point i'm not sure it's worth the trouble as all I wanted was to assert my upgrade went well. We're using provider networks and I know the setup is correct in my other envs. i'll see tomorrow depending on my mood xD | 21:57 |
| johnsom | Ok, sounds good. | 21:58 |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Remove test calls to reset_mock() https://review.opendev.org/701465 | 22:01 |
| rm_work | lol thanks, i was just going to be sad, and not bother making a patch lol | 22:03 |
| haleyb | don't want anyone to be ;( | 22:04 |
| haleyb | that actually looks mad... | 22:04 |
| johnsom | Ok, I have decided this port cleanup issue is big enough to be a separate story/patch. https://storyboard.openstack.org/#!/story/2007077 | 22:45 |
| johnsom | Currently failover still succeeds, and the ports will be cleaned up on LB delete. It's just the edge case of someone deleting the amp that leads to port leakage. | 22:46 |
| johnsom | If LB delete detects this issue, it walks every port in the project hunting for the port to cleanup. Not something I want to add to failover... | 22:47 |
| rm_work | lol | 22:48 |
| rm_work | could that be something that housekeeping is responsible for? | 22:48 |
| rm_work | because it can also happen "later" | 22:48 |
| johnsom | Yeah, that would require some locking work in housekeeping, but I included that as something to explore in the story | 22:48 |
| johnsom | I.e. don't kill ports that are actually being *added* to the LB, etc. | 22:49 |
| rm_work | whelp, with all the wsgi nonsense for every project during a cycle goal, somehow "specifying a config file that isn't in a default location" never came up, heh | 22:49 |
| johnsom | I think having the ID in the port description will make this all much cleaner. I just don't want to layer that on this patch. I will circle back and do a followup | 22:50 |
| rm_work | yeah | 22:50 |
| johnsom | I vaguely think it did, but was brushed off. | 22:50 |
| rm_work | < | 22:50 |
| rm_work | <_< | 22:51 |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Remove all deprecated driver code that moved to octavia-lib https://review.opendev.org/701473 | 22:51 |
| johnsom | rm_work if you pass our config parameter to uwsgi in the systemd service config it doesn't pick up the right config? | 23:07 |
| johnsom | for example: /usr/local/bin/uwsgi --ini /etc/octavia/octavia-uwsgi.ini --config-file /etc/octavia/octavia-other-config.conf | 23:08 |
| *** tkajinam has joined #openstack-lbaas | 23:08 | |
| *** servagem has joined #openstack-lbaas | 23:13 | |
| *** tesseract has quit IRC | 23:13 | |
| johnsom | rm_work Ah, you might need to pass it as --pyargv | 23:14 |
| johnsom | which I guess can go in the uwsgi.ini as well | 23:15 |
| *** goldyfruit_ has joined #openstack-lbaas | 23:17 | |
| rm_work | oh is that how? | 23:32 |
| rm_work | i will test --pyargv | 23:32 |
| *** ccamposr has quit IRC | 23:36 | |
| *** ccamposr has joined #openstack-lbaas | 23:37 | |
| *** pvradu has joined #openstack-lbaas | 23:44 | |
| rm_work | I don't think that works | 23:44 |
| rm_work | i tried using it to change the config file to some garbage and it still started up correctly | 23:44 |
| johnsom | The format you pass into that is a bit odd, I think it needs the --config-file not just config-file | 23:47 |
| johnsom | Ah, argh, so the setuptools script doesn't pass argv down and our service code doesn't try sys.argv: https://github.com/openstack/octavia/blob/master/octavia/common/service.py#L21 | 23:52 |
| johnsom | Ah, it's actually PBR that creates that script | 23:56 |
| johnsom | It seems like if we don't get argv passed in, maybe we should pull the system argv in | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!