| *** TMM has quit IRC | 00:36 | |
| *** TMM has joined #openstack-lbaas | 00:36 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Add skip_if_not_implemented to the service client https://review.opendev.org/714003 | 00:38 |
|---|---|---|
| *** spatel has joined #openstack-lbaas | 00:38 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Add skip_if_not_implemented to the service client https://review.opendev.org/714003 | 00:44 |
| *** nicolasbock has quit IRC | 01:07 | |
| *** spatel has quit IRC | 01:10 | |
| *** sapd1 has joined #openstack-lbaas | 01:46 | |
| *** sapd1 has quit IRC | 01:48 | |
| *** sapd1 has joined #openstack-lbaas | 01:48 | |
| *** rcernin has quit IRC | 02:03 | |
| *** rcernin has joined #openstack-lbaas | 02:03 | |
| dawzon | Is there a way to force tox to use a local version of octavia-lib? I need my modified local version to pass the tests but it always uses 1.5.0 | 02:22 |
| johnsom | dawzon: the easiest way is to install the new one over the tox installed version | 02:48 |
| johnsom | Cd .tox/pep8/bin | 02:49 |
| johnsom | Source activate | 02:49 |
| johnsom | Your prompt will change, you are now in the pep8 virtualenv. | 02:49 |
| johnsom | You can the pip uninstall and install the new one | 02:50 |
| johnsom | deactivate will exit the virtualenv | 02:50 |
| johnsom | There is one for each tox environment | 02:50 |
| johnsom | There are other ways, but this will work for you | 02:51 |
| dawzon | Thanks! So I'm guessing that in order for my patch to pass zuul I'll have to get the octavia-lib patch merged first | 02:52 |
| johnsom | Yes | 02:53 |
| johnsom | You can try using the depends-on tag in the commit message, but that doesn’t always work | 02:54 |
| *** yamamoto has joined #openstack-lbaas | 03:01 | |
| *** yamamoto has quit IRC | 03:13 | |
| *** psachin has joined #openstack-lbaas | 03:26 | |
| *** yamamoto has joined #openstack-lbaas | 04:05 | |
| *** yamamoto has quit IRC | 04:12 | |
| *** vishalmanchanda has joined #openstack-lbaas | 05:21 | |
| *** rcernin is now known as rcernin|brb | 05:30 | |
| *** pckizer has quit IRC | 05:44 | |
| *** yamamoto has joined #openstack-lbaas | 06:13 | |
| *** yamamoto has quit IRC | 06:36 | |
| *** gcheresh has joined #openstack-lbaas | 07:31 | |
| *** psachin has quit IRC | 07:41 | |
| *** sapd1_x has joined #openstack-lbaas | 07:42 | |
| *** rpittau|afk is now known as rpittau | 07:57 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 08:00 | |
| openstackgerrit | Merged openstack/octavia master: Fix load balancer update with provider filtered params https://review.opendev.org/695589 | 08:06 |
| *** gcheresh has quit IRC | 08:07 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Fix load balancer update with provider filtered params https://review.opendev.org/714043 | 08:09 |
| *** sapd1_x has quit IRC | 08:17 | |
| *** yamamoto has joined #openstack-lbaas | 08:54 | |
| *** yamamoto has quit IRC | 08:58 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Fix load balancer update with provider filtered params https://review.opendev.org/714043 | 09:20 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/stein: Fix load balancer update with provider filtered params https://review.opendev.org/714058 | 09:29 |
| *** ramishra has quit IRC | 09:38 | |
| *** ramishra has joined #openstack-lbaas | 09:46 | |
| *** ramishra has quit IRC | 09:50 | |
| *** ramishra has joined #openstack-lbaas | 09:50 | |
| *** yamamoto has joined #openstack-lbaas | 09:57 | |
| *** yamamoto has quit IRC | 10:11 | |
| *** yamamoto has joined #openstack-lbaas | 10:21 | |
| sapd1 | I'm creating L7 policy for a listener. There is no error in Octavia-api and octavia-worker. But the configuration of haproxy inside amphora wasnt changed. | 10:25 |
| *** nicolasbock has joined #openstack-lbaas | 11:03 | |
| kklimonda | hmm, should pool members have operating_status set to ERROR when healthmonitor check fails due to unexpected return code? | 11:07 |
| kklimonda | Is there a way for users to see the cause of error somehow? | 11:07 |
| cgoncalves | kklimonda, yes, pool members should be in ERROR in such case you described. see https://docs.openstack.org/api-ref/load-balancer/v2/#status-codes | 11:27 |
| cgoncalves | kklimonda, no way to for users to see the cause of error via octavia api but they could reproduce the check themselves directly against the member | 11:28 |
| kklimonda | thanks, that's less than ideal but such is life :) | 11:28 |
| kklimonda | (I know, patches welcome :D) | 11:29 |
| cgoncalves | sapd1, I have not tested L7 policies in months. may be a bug maybe introduced in the single-process listeners work, just a possibility of regression | 11:29 |
| cgoncalves | sapd1, we do have traffic scenario tests for L7 policies so disregard what I said above | 11:32 |
| cgoncalves | https://github.com/openstack/octavia-tempest-plugin/blob/master/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py | 11:32 |
| *** rpittau is now known as rpittau|bbl | 11:37 | |
| *** yamamoto has quit IRC | 11:51 | |
| sapd1 | cgoncalves, That was my fault. Sorry. | 11:52 |
| cgoncalves | sapd1, no problem. good to have your input that it works ;) | 11:58 |
| *** ramishra has quit IRC | 12:06 | |
| *** ramishra has joined #openstack-lbaas | 12:28 | |
| *** gcheresh has joined #openstack-lbaas | 12:35 | |
| *** gcheresh has quit IRC | 12:59 | |
| *** yamamoto has joined #openstack-lbaas | 13:10 | |
| *** gcheresh has joined #openstack-lbaas | 13:10 | |
| *** yamamoto has quit IRC | 13:16 | |
| *** yamamoto has joined #openstack-lbaas | 13:24 | |
| *** yamamoto has quit IRC | 13:29 | |
| *** TrevorV has joined #openstack-lbaas | 13:37 | |
| -openstackstatus- NOTICE: Gerrit (review.opendev.org) will be down for maintenance starting at 14:00 (in less than 20 mins), probably until 15:00 UTC | 13:43 | |
| *** yamamoto has joined #openstack-lbaas | 13:43 | |
| *** rpittau|bbl is now known as rpittau | 13:52 | |
| *** yamamoto has quit IRC | 13:55 | |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is offline for maintenance until 15:00 UTC http://lists.opendev.org/pipermail/service-announce/2020-March/000001.html | 14:07 | |
| *** ChanServ changes topic to "The Gerrit service on review.opendev.org is offline for maintenance until 15:00 UTC http://lists.opendev.org/pipermail/service-announce/2020-March/000001.html" | 14:07 | |
| *** yamamoto has joined #openstack-lbaas | 14:17 | |
| *** yamamoto has quit IRC | 14:18 | |
| *** yamamoto has joined #openstack-lbaas | 14:20 | |
| *** yamamoto has quit IRC | 14:21 | |
| *** TrevorV has quit IRC | 14:42 | |
| *** TrevorV has joined #openstack-lbaas | 14:45 | |
| *** ChanServ changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 15:04 | |
| -openstackstatus- NOTICE: Gerrit maintenance is concluded at this time and requested renames have been performed | 15:04 | |
| openstackgerrit | Brian Haley proposed openstack/octavia-tempest-plugin master: Use same flake8 extensions as Octavia https://review.opendev.org/714142 | 15:16 |
| *** gcheresh has quit IRC | 15:44 | |
| *** vishalmanchanda has quit IRC | 16:40 | |
| *** openstack has joined #openstack-lbaas | 16:58 | |
| *** ChanServ sets mode: +o openstack | 16:58 | |
| *** maciejjozefczyk has quit IRC | 17:59 | |
| rm_work | kklimonda: if you use log offloading, and expose the haproxy logs to the users, then they can look and see :) | 18:03 |
| cgoncalves | ah, good point. I forgot that option | 18:05 |
| rm_work | speaking of which, today I try to fix the UDP HMs | 18:07 |
| rm_work | T_T | 18:07 |
| johnsom | We have docs for that! <grin> https://docs.openstack.org/octavia/latest/admin/log-offloading.html#tenant-flow-logs | 18:12 |
| * johnsom feels like he is stuck in a skit for Portlandia. Put a bird on it! | 18:12 | |
| rm_work | portlandia is so eerily accurate i had to stop watching it because i have visited portland a lot and it's just ... like watching real life happen | 18:14 |
| johnsom | It is very true. Especially the early seasons | 18:14 |
| johnsom | https://youtu.be/G-ppr9LDaQE | 18:15 |
| rm_work | My favorite was their NPR thing | 18:21 |
| rm_work | https://www.youtube.com/watch?v=YsufMtUOXtI | 18:22 |
| johnsom | lol, yeah, that is a good one too | 18:22 |
| rm_work | the blue apron reference was just ... perfect | 18:23 |
| rm_work | do you know know to make a catchall for a log handler? | 18:27 |
| rm_work | tried: qualname = sqlalchemy.* | 18:27 |
| rm_work | not working | 18:27 |
| rm_work | ah because root_logger overrides it | 18:30 |
| rm_work | err, logger_root | 18:30 |
| johnsom | It depends on the logger used and the OS overrides | 18:32 |
| johnsom | If you look at our amphora template: https://github.com/openstack/octavia/blob/master/octavia/common/jinja/logging/templates/10-rsyslog.conf.template | 18:32 |
| johnsom | I use *.* stop to block logging to the local disk (if that setting is enabled) | 18:33 |
| johnsom | Anther way is to define a input or output with no filters. | 18:33 |
| johnsom | https://github.com/openstack/octavia/blob/master/devstack/etc/rsyslog/10-octavia-log-offloading.conf | 18:34 |
| johnsom | Our devstack config ^^ defines two inputs and filters on facility and severity. | 18:34 |
| rm_work | oh i mean python | 18:34 |
| johnsom | Oh, ha | 18:34 |
| johnsom | Yeah, less familiar with that | 18:34 |
| rm_work | we use these split log configs for every service | 18:35 |
| rm_work | and it's overriding `debug=true` in the main config | 18:35 |
| rm_work | AH got it, fff, forgot to add my new logger to the logger list | 18:37 |
| johnsom | Eh gads. I'm like six levels deep of _methods here. | 18:45 |
| johnsom | Me things this stuff needs to be pulled out of test_base.py | 18:46 |
| johnsom | Also because I might be on a path of adding a few more. sigh. | 18:47 |
| *** KeithMnemonic has joined #openstack-lbaas | 19:06 | |
| KeithMnemonic | johnsom Hi Michael how are you doing. I hope you are holding up well. Do you have a minute or two for a few quick questions on db migration for the octavia db. specifically here https://github.com/openstack/octavia/tree/stable/pike/octavia/db/migration/alembic_migrations/versions this is not a question about pike specifically but more general | 19:09 |
| rm_work | daily plug for https://review.opendev.org/#/q/topic:az-tweaks+status:open | 19:21 |
| rm_work | johnsom: could you also respond on https://review.opendev.org/#/c/711298/ ? | 19:22 |
| johnsom | KeithMnemonic Hi, caught me at lunch (back now). I am well and hope you are as well. What can I answer about migrations? | 19:44 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Support HTTP and TCP checks in UDP healthmonitor https://review.opendev.org/589180 | 19:45 |
| rm_work | ^^ rebase | 19:45 |
| rm_work | should be good to go though, deployed in my cloud and seems to work :D | 19:45 |
| KeithMnemonic | i sort of figured it out, i have a customer with two files in the versions dir and have no idea where they came from. they are not upstream and not in any of the suse or old hos repos | 19:46 |
| KeithMnemonic | these are causing his db failure | 19:46 |
| johnsom | Ah, well then, "not our problem" lol | 19:46 |
| KeithMnemonic | they are basically "noop"s as they do nothing other than a version change | 19:46 |
| johnsom | Yeah, well, they would need to make sure the chain is still intact. | 19:46 |
| KeithMnemonic | but not sure if they got these from some vendor that hooks into octavia or what | 19:47 |
| johnsom | That would be a horrible idea and a sin itself, lol | 19:47 |
| KeithMnemonic | does this version ever ring a bell to you "82ca3350c2a3" | 19:47 |
| johnsom | Nope, we never had it upstream | 19:48 |
| johnsom | https://www.irccloud.com/pastebin/bkgr9mTt/ | 19:49 |
| johnsom | That is all I have | 19:49 |
| johnsom | Wonder if they mixed neutron-lbaas migrations into Octavia? Bad bad bad idea of course | 19:50 |
| johnsom | Nope, not in the last neutron-lbaas repo I have laying around either | 19:51 |
| johnsom | No hits in codesearch.openstack.org either, so likely a customer science experiment | 19:51 |
| johnsom | rm_work So, my experience with OSC has been that --long means everything, no matter how ugly it gets. | 19:53 |
| johnsom | rm_work That is why I raised the flag. | 19:54 |
| rm_work | hmm | 19:54 |
| rm_work | ok | 19:54 |
| johnsom | We can of course poll others for their thoughts. | 19:54 |
| rm_work | i kinda dislike that but i see the point | 19:54 |
| johnsom | You could add "--more" lol..... | 19:54 |
| johnsom | "--bit-more" "--some" "--my-fav" | 19:55 |
| johnsom | Let me look at "openstack network list --long" and compare | 19:55 |
| johnsom | Hmm, yeah, ok, so maybe I am remembering wrong. --long on network list is only three lines of column titles(normal is two). certainly not all of the columns, though guessing if that is just because no one updated the OSC plugin or not. | 19:58 |
| johnsom | It just seems odd to have an intermediary option with no "give me all of them" that would be nice for filtering via OSC. | 19:59 |
| johnsom | We can't do "--long" and "--all" because --all has other meaning | 19:59 |
| johnsom | https://docs.openstack.org/python-openstackclient/train/contributor/command-options.html#additional-fields | 20:01 |
| KeithMnemonic | johnsonm i think we figured it out, if you run octavia-db-manage revision it seems to create them | 20:02 |
| KeithMnemonic | and messes up the history,... | 20:02 |
| johnsom | KeithMnemonic "revision" *is* the command to create a *new* revision..... lol | 20:03 |
| johnsom | revision Create a new revision file. | 20:03 |
| KeithMnemonic | you needed a laugh today i bet | 20:04 |
| johnsom | lol, yes | 20:04 |
| openstackgerrit | Merged openstack/octavia-tempest-plugin master: Use same flake8 extensions as Octavia https://review.opendev.org/714142 | 20:06 |
| johnsom | upgrade, head, and check_migration are probably the only commands a non-developer type would run | 20:07 |
| johnsom | Maybe current and history | 20:07 |
| johnsom | Ah, I guess "head" is the old command. "current" replaced it | 20:08 |
| rm_work | hmmmmmmmmm | 21:19 |
| KeithMnemonic | johnsom we fixed it, thanks and take care | 21:19 |
| johnsom | o/ | 21:19 |
| *** TrevorV has quit IRC | 21:27 | |
| rm_work | johnsom: hmm, DB goes to R/O -- lots of LBs go to ERROR. thoughts? | 21:36 |
| rm_work | I saw some failover notifications, which semi-makes-sense? except, i thought we fixed that issue; and also, if the DB is R/O it wouldn't even get as far as marking the LB in PENDING | 21:36 |
| rm_work | right? | 21:36 |
| johnsom | How can they be marked ERROR if the DB is read only? | 21:37 |
| rm_work | right | 21:37 |
| rm_work | well i mean, it came back | 21:37 |
| rm_work | i'm worried that we disable healthchecks for some amount of time on a DB outage, but not if it's still readable? | 21:38 |
| johnsom | Yeah, so, the safety we put in only detects a DB missing, not read only. So, if it exists, but is read only the safety "wait for a few minutes" will not fire | 21:38 |
| rm_work | yeah :/ | 21:38 |
| rm_work | so then the second it comes back up | 21:38 |
| rm_work | everything tried to go failover, i assume | 21:38 |
| johnsom | Right, HMs will get busy | 21:38 |
| rm_work | is it possible it tried to failover both backup and master at the same time? | 21:39 |
| johnsom | No | 21:39 |
| rm_work | which would obviously fail currently -- not sure if we have guards in place for that | 21:39 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/health_manager.py#L115 | 21:40 |
| johnsom | Yeah, first one in locks the LB, so only one amp per lb at a time | 21:40 |
| johnsom | But, remember, the first one might fail, go error, and the second one come back around.... | 21:40 |
| openstackgerrit | Merged openstack/octavia master: Add oslo middleware healthcheck to Octavia API https://review.opendev.org/711127 | 21:43 |
| openstackgerrit | Merged openstack/octavia stable/train: Remove the barbican "Grant access" from cookbook https://review.opendev.org/713535 | 21:43 |
| openstackgerrit | Merged openstack/octavia stable/stein: Remove the barbican "Grant access" from cookbook https://review.opendev.org/713536 | 21:43 |
| openstackgerrit | Merged openstack/octavia stable/rocky: Remove the barbican "Grant access" from cookbook https://review.opendev.org/713537 | 21:43 |
| rm_work | hmmm looks like we're seeing port detach timeouts which might account for it | 21:44 |
| rm_work | a lot of LBs failed over due to the DB R/O thing | 21:44 |
| rm_work | and then ... about half of them had one amp do that | 21:44 |
| johnsom | Yeah, if the DB is RO, nova is probably hosed too | 21:45 |
| johnsom | The new failover flow will help with that for sure. | 21:45 |
| rm_work | wtf i'm getting plugging errors NOW tho | 21:56 |
| rm_work | the DB RO thing was like, a day ago | 21:56 |
| openstackgerrit | Merged openstack/octavia master: Add a periodic image build job https://review.opendev.org/706393 | 21:58 |
| *** KeithMnemonic has quit IRC | 22:02 | |
| *** nicolasbock has quit IRC | 22:24 | |
| *** hongbin has joined #openstack-lbaas | 22:33 | |
| *** lxkong has quit IRC | 23:23 | |
| *** lxkong has joined #openstack-lbaas | 23:24 | |
| *** TMM has quit IRC | 23:25 | |
| *** TMM has joined #openstack-lbaas | 23:25 | |
| *** irclogbot_3 has quit IRC | 23:27 | |
| *** irclogbot_1 has joined #openstack-lbaas | 23:28 | |
| *** guilhermesp_ has joined #openstack-lbaas | 23:36 | |
| *** guilhermesp has quit IRC | 23:37 | |
| *** guilhermesp_ is now known as guilhermesp | 23:37 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Fix padding logic for UDP health daemon https://review.opendev.org/714240 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!