| openstackgerrit | Tatsuma Matsuki proposed openstack/octavia master: Separate the thread pool for health and stats update https://review.openstack.org/581585 | 00:13 |
|---|---|---|
| *** harlowja has quit IRC | 00:23 | |
| abaindur | johnsom: we are having some random trouble with the amp ssh_key name now. it was working earlier, now it cant find the key in a multi tenant setting | 00:48 |
| abaindur | we created the key under the same tenant as the tenant specified in octavia's keystone_authtoken and service_auth sections | 00:48 |
| abaindur | openstack keypair list shows the same key name as we have set for amp_ssh_key_name | 00:49 |
| johnsom | The keypair in nova should be created under your Octavia service account (service_auth in the octavia.conf) and then that ID loaded in the config file | 00:49 |
| abaindur | yep, its service tenant, and thats same tenant i created the ssh key in | 00:50 |
| johnsom | You aren't trying to boot the amphora instance under a tenant account are you? | 00:50 |
| abaindur | there is no ID for the keypair, just name | 00:50 |
| abaindur | openstack keypair list | 00:50 |
| abaindur | +-------+-------------------------------------------------+ | 00:50 |
| abaindur | | Name | Fingerprint | | 00:50 |
| abaindur | +-------+-------------------------------------------------+ | 00:50 |
| abaindur | | arjun | 00:50 |
| abaindur | ok, now i am creating the LB itself via a different tenant | 00:51 |
| johnsom | LB is fine, it's the amphora compute instance, the project id shown in nova is the service_auth project ID from your octavia.conf right? | 00:51 |
| johnsom | The keypair and nova instance must both be owned by the octavia service account as listed in [service_auth]. | 00:53 |
| abaindur | yep. the keypair is under service | 00:55 |
| abaindur | and thats the tenant/project under service_auth | 00:55 |
| abaindur | ComputeBuildException: Failed to build compute instance due to: Invalid key_name provided. (HTTP 400) | 00:56 |
| johnsom | Then I have not heard of an issue with the ssh key. What is the worker log giving you? | 00:56 |
| abaindur | That error ^^ | 00:56 |
| abaindur | I see similar error coming from nova-api logs | 00:56 |
| abaindur | I added my own log, vberified it is the correct key name | 00:59 |
| abaindur | 2018-08-21 17:58:55.282 5567 DEBUG octavia.controller.worker.tasks.compute_tasks [-] Compute create execute for amphora with id a2b047f7-7548-4d20-b590-58bf4f67c4e4 execute /opt/pf9/pf9-octavia/lib/python2.7/site-packages/octavia/controller/worker/tasks/compute_tasks.py:62 | 00:59 |
| abaindur | 2018-08-21 17:58:55.284 5567 INFO octavia.controller.worker.tasks.compute_tasks [-] ARJUN: using key_name = amp_ssh_key | 00:59 |
| abaindur | in octavia.conf, I have set: | 00:59 |
| abaindur | mp_ssh_key_name = amp_ssh_key | 00:59 |
| abaindur | amp_ssh_key_name = amp_ssh_key | 01:00 |
| johnsom | Your keypair list above had the key name as "arjun" it should be amp_ssh_key_name = arjun | 01:00 |
| abaindur | i changed em around | 01:00 |
| abaindur | i deleted everything and changed th names to retry | 01:00 |
| abaindur | I noticed prior i had an ssh key called "arjun" in my other tenant as well. i thought maybe the duplicate name was causing some issue finding correct one. so i deleted it, and re-named the one in service tenant | 01:01 |
| johnsom | You openrc info you used for the key pair create matches exactly the [service_auth] section? | 01:01 |
| abaindur | yes... i can only view the keypair under service tenant | 01:02 |
| johnsom | Username and project ID should be the only things that are important there | 01:02 |
| johnsom | I guess you have to debug nova then as I don't have any answers. We pass that field straight through to the nova server create API as you saw in the nova log. You could always remove the key from octavia conf, let it boot, check in nova the project ID of the amp and make sure it matches what you expect. | 01:04 |
| johnsom | I wish the keypair had a project_id column to make it easy to double check, but nova.... | 01:06 |
| abaindur | hmm maybe something nova related | 01:09 |
| abaindur | this is a completely new setup than one i was testing earlier | 01:09 |
| abaindur | Octavia looks like its doing right job. Things work when i rmove amp_ssh_key_name and boot it without key | 01:10 |
| abaindur | i'll have to check w/ someone else familiar with nova | 01:10 |
| johnsom | Yeah, that error was straight from nova, so something is fishy in nova land. The only thing I have seen like this is when the keypair is under the wrong service account, but we have already covered that... | 01:11 |
| *** dmellado has quit IRC | 01:22 | |
| *** rcernin_ has joined #openstack-lbaas | 01:23 | |
| *** rcernin has quit IRC | 01:25 | |
| *** abaindur has quit IRC | 01:29 | |
| *** rcernin has joined #openstack-lbaas | 02:00 | |
| *** rcernin_ has quit IRC | 02:03 | |
| *** hongbin has joined #openstack-lbaas | 02:22 | |
| *** dmellado has joined #openstack-lbaas | 02:38 | |
| sapd1 | Do we need limit listener and l7policy on a loadbalancer? | 02:52 |
| johnsom | sapd1 Listener is limited already by the number of ports available. I guess you could kill an amp by overloading it with listeners, but the user would be shooting themselves in the foot. | 02:58 |
| johnsom | l7policy is just rules added to the config, I think that would be driver dependent. | 02:59 |
| sapd1 | johnsom: ok. I understand | 03:00 |
| johnsom | Yeah, I guess I'm not really sure if there is a good use case for quota on those or not. | 03:00 |
| sapd1 | johnsom: Because some cloud providers such as AWS has limitations for that. Example: https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-limits.html | 03:01 |
| johnsom | lol, but we are better than ELB... grin | 03:02 |
| johnsom | Geez, only 20 lbs? I have seen users with 250+ | 03:02 |
| sapd1 | If user create more l7 policies maybe affect to LB performance. | 03:02 |
| sapd1 | johnsom: Maybe this is a default quota, So you could contact support to increase quota :D I think | 03:03 |
| johnsom | True, but they are only impacting their own performance. I feel that quota is really used to protect the cloud resources more than the user degrading their own use. What do you think? | 03:03 |
| sapd1 | yep. But We should care about SLA when provide service :D | 03:04 |
| johnsom | I agree about SLA, but I guess I would set SLA in a way that does not involve their application performance, as application quality varies. | 03:06 |
| johnsom | If you feel those quotas are needed for your deployment, please feel free to create a patch adding them. The rest of us can always set them -1 by default. I think we just didn't have a good reason to add them | 03:07 |
| sapd1 | johnsom: So Let me try. :D | 03:10 |
| johnsom | +1 | 03:12 |
| johnsom | sapd1 Just make sure the defaults are better than AWS.... lol grin | 03:12 |
| sapd1 | johnsom: How to set default quota for octavia? | 03:12 |
| sapd1 | johnsom: Ok. I agree | 03:12 |
| johnsom | https://github.com/openstack/octavia/blob/master/etc/octavia.conf#L414 | 03:13 |
| sapd1 | thankyou. better than AWS =)) | 03:13 |
| *** hongbin has quit IRC | 03:33 | |
| sapd1 | johnsom: After quota has been met, If I create a loadbalancer on horizon I will be logout because return code is 403 :/ | 04:02 |
| sapd1 | root@ovs-capacity-4:~# openstack loadbalancer create --vip-network-id d4f88e6c-3e45-4dbb-95cb-bf93faddf8a0 --name test-quota | 04:02 |
| sapd1 | Quota has been met. (HTTP 403) (Request-ID: req-9b619efe-ef9a-4e64-9272-95d42d8560e6) | 04:02 |
| johnsom | Hmm, sounds like a horizon/dashboard bug. Please open a story | 04:02 |
| *** abaindur has joined #openstack-lbaas | 04:04 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Fix tests to honor Octavia API versioning https://review.openstack.org/594786 | 04:04 |
| johnsom | rm_work ^^^ That is an API version approach | 04:07 |
| sapd1 | johnsom: https://storyboard.openstack.org/#!/story/2003522 | 04:09 |
| johnsom | sapd1 I will switch it to the horizon dashboard project as it's not an Octavia bug, it's like octavia-dashboard | 04:10 |
| sapd1 | johnsom: Thanks | 04:11 |
| johnsom | Ok, with that patch I am done for the day. Catch you all tomorrow | 04:12 |
| *** abaindur has quit IRC | 04:53 | |
| *** abaindur has joined #openstack-lbaas | 04:54 | |
| *** yboaron_ has joined #openstack-lbaas | 04:55 | |
| *** cgoncalves has joined #openstack-lbaas | 06:32 | |
| *** pcaruana has joined #openstack-lbaas | 06:41 | |
| *** dmellado has quit IRC | 07:04 | |
| *** dmellado has joined #openstack-lbaas | 07:06 | |
| *** rcernin has quit IRC | 07:07 | |
| *** ispp has joined #openstack-lbaas | 07:08 | |
| *** dmellado has quit IRC | 07:12 | |
| *** yboaron_ has quit IRC | 07:12 | |
| *** ispp has quit IRC | 07:12 | |
| *** dmellado has joined #openstack-lbaas | 07:19 | |
| *** luksky11 has joined #openstack-lbaas | 07:29 | |
| *** velizarx has joined #openstack-lbaas | 07:38 | |
| *** celebdor has joined #openstack-lbaas | 07:52 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia-dashboard master: Imported Translations from Zanata https://review.openstack.org/594902 | 07:53 |
| *** abaindur has quit IRC | 08:19 | |
| *** yboaron_ has joined #openstack-lbaas | 08:19 | |
| *** velizarx has quit IRC | 08:21 | |
| *** velizarx has joined #openstack-lbaas | 08:22 | |
| *** pcaruana has quit IRC | 08:28 | |
| *** pcaruana has joined #openstack-lbaas | 08:30 | |
| *** celebdor has quit IRC | 08:35 | |
| *** dolly has quit IRC | 11:25 | |
| *** fnaval has joined #openstack-lbaas | 13:25 | |
| *** fyx has joined #openstack-lbaas | 13:43 | |
| *** dmellado has quit IRC | 14:01 | |
| *** dolly has joined #openstack-lbaas | 14:24 | |
| dolly | ey! I'm back! Got the stable/queens 2.0.2 release running on my osp 13 setup. Everything is working as expected *except* recreating the amp when I manually delete it. It seems like the same issue persists in 2.0.2 as in earlier versions, here's the log of whats happening http://paste.openstack.org/show/728603/ | 14:30 |
| dolly | It actually continously tries to recreate the amp, but gets the same error all the time (it does so while we speak) | 14:30 |
| cgoncalves | it's the AddrFormatError issue | 14:32 |
| cgoncalves | https://storyboard.openstack.org/#!/story/2003052 | 14:32 |
| cgoncalves | oh, right. it was you that also ran into it :) | 14:33 |
| dolly | yes exactly that issue =) | 14:34 |
| dolly | I thought that it had been fixad in 2.0.2 ? | 14:34 |
| dolly | Do we have any idea about why its happening ? | 14:34 |
| dolly | I'm more than happy to assist | 14:35 |
| cgoncalves | can you check if the instance has a lb-mgmt-net IP assigned? | 14:36 |
| cgoncalves | basically asking the same johnsom asked me on the story :) | 14:36 |
| dolly | Well, it creates the instance and immidetely destroys it | 14:37 |
| dolly | so, not really sure how to check that | 14:37 |
| dolly | but there is a setting to disable cleanup of amps right ? I think I saw that somewhere in the settings | 14:37 |
| cgoncalves | either mark the amp busy in the database or stop the health manager service | 14:38 |
| johnsom | dolly: stop the health manager process and set disable-revert in the worker config | 14:38 |
| dolly | johnsom: after setting disable-revert, do I need to restart anything = | 14:39 |
| cgoncalves | docker restart octavia_* :P | 14:42 |
| dolly | got it, was thinking if it took the setting dynamically | 14:42 |
| dolly | But if I stop the healt-manager before destroying one of the amps, would that mean that there is nothing to detect that the amp is gone and thus nothing starting the process of creating a new amp ? :) | 14:43 |
| dolly | I'll just stop the healthmanager after it has recreated the amp, so it diesn't keep on creating millions of vms | 14:45 |
| cgoncalves | ok | 14:46 |
| *** ktibi has joined #openstack-lbaas | 14:48 | |
| dolly | ok | 14:51 |
| dolly | got the amp now | 14:52 |
| dolly | not destroyed | 14:52 |
| dolly | but it doesn't seem to be connected to anything | 14:52 |
| dolly | so from that perspective it makes sense that octavia cant get the ipaddr | 14:59 |
| dolly | (because there is none) | 15:00 |
| cgoncalves | can you check if an IP address was assigned? | 15:00 |
| dolly | hm, where do I see that ? | 15:01 |
| dolly | I mean no addresses shows up when doing openstack server show <amp-id> | 15:05 |
| cgoncalves | oh, that should be a start yeah | 15:08 |
| dolly | Its booted and everything | 15:09 |
| dolly | but doesnt got a network | 15:09 |
| cgoncalves | openstack port list and grep by possible port for amp? | 15:09 |
| *** pcaruana has quit IRC | 15:10 | |
| dolly | well how would I know which port that would be used by the amp ? | 15:14 |
| cgoncalves | top of my head I'd say to check neutron logs and look for port creation requests (if any) around same time when the amp was created | 15:16 |
| cgoncalves | it could be that neutron is erroing and octavia is not catching it, I don't know | 15:16 |
| dolly | yeah ive been looking in the neutron logs by the id of the amp | 15:17 |
| dolly | but the only thing I see there are GET's | 15:17 |
| dolly | there doesn't seem to be anything related to create | 15:17 |
| cgoncalves | what about nova? | 15:17 |
| dolly | let me check the logs for the amp that is actually created | 15:17 |
| dolly | nova is fine | 15:17 |
| dolly | logs says that instance is created | 15:17 |
| cgoncalves | it is nova who requests port creation, no? | 15:18 |
| dolly | well, I'm a bit "new in the game" so I wouldn't be able to answer that. But I think its a bit weird that I only see GET's in the log | 15:18 |
| *** evgenyf has joined #openstack-lbaas | 15:19 | |
| *** velizarx has quit IRC | 15:22 | |
| *** openstackgerrit has quit IRC | 15:31 | |
| *** evgenyf has quit IRC | 15:37 | |
| *** yboaron_ has quit IRC | 15:40 | |
| *** luksky11 has quit IRC | 15:41 | |
| *** dmellado has joined #openstack-lbaas | 15:45 | |
| dolly | http://paste.openstack.org/show/728613/ | 15:47 |
| dolly | Please correct me if I'm wrong, | 15:47 |
| dolly | but when the health-manager recreates the amp, there is no information about network nor sec-group in the request. | 15:47 |
| dolly | Don't we need to pass this to nova to place it correctly ? Or am I missing something obvious ? | 15:48 |
| *** pcaruana has joined #openstack-lbaas | 15:49 | |
| cgoncalves | hmmm interesting :) | 15:50 |
| cgoncalves | johnsom, ^ | 15:50 |
| dolly | Well that at least would make sense from my perspective. Parameter is missing to nova, nova creates the amp without network, octavia asks the neutron api for the network where the amp is attached, gets None as response since its not attached anywhere. | 15:59 |
| cgoncalves | I think so too. I'm looking at get_failover_flow, still not much acquainted with TaskFlow | 16:03 |
| johnsom | dolly still catching up on the back scroll, but yes, nova creates the lb-mgmt-net port at boot. | 16:07 |
| dolly | johnsom: no worries | 16:08 |
| johnsom | Nova won't let you boot an instance without a network. (at least it didn't). | 16:08 |
| dolly | Hm, pretty sure you can now. | 16:08 |
| johnsom | You will see in your HM log a TaskFlow task called "CertComputeCreate" which calls nova with the network ID to attach, here https://github.com/openstack/octavia/blob/master/octavia/controller/worker/tasks/compute_tasks.py#L93 | 16:11 |
| johnsom | So the only way that would not have a network (and really shouldn't even boot) is if you are missing the "amp_boot_network_list" octavia.conf setting for your HM processes | 16:12 |
| dolly | well that parameter is commented out in the config | 16:14 |
| dolly | is it enough with the default ? | 16:14 |
| dolly | I mean is that parameter default set | 16:15 |
| johnsom | There is no default for that field | 16:15 |
| dolly | Hm, so how come that its working to create the amps initially ? | 16:16 |
| johnsom | It must be defined in your worker config file | 16:16 |
| dolly | OH ! etc/octavia/conf.d/octavia-worker/worker-post-deploy.conf:amp_boot_network_list = a369be6a-efac-4575-8b5b-fc155907a231 <- | 16:17 |
| dolly | freaking puppet / director making these magic things :P | 16:18 |
| dolly | So its defined for the worker but not for the health-manager | 16:18 |
| dolly | or any of the other services | 16:18 |
| johnsom | worker, health, and housekeeping should all have the exact same config as they all use the worker library | 16:19 |
| johnsom | dolly https://docs.openstack.org/octavia/latest/_images/octavia-component-overview.svg | 16:19 |
| johnsom | From the intro docs. | 16:19 |
| cgoncalves | playbooks/roles/octavia-controller-config/templates/worker-post-deploy.conf.j2:amp_boot_network_list = {{ lb_mgmt_net_id }} | 16:21 |
| cgoncalves | if the HM requires amp_boot_network_list to be set, it might not be. looking | 16:22 |
| johnsom | cgoncalves spares pool won't work if it isn't set for housekeeping too | 16:22 |
| dolly | it is not set in the "main" octavia.conf that is shared in /etc/octavia/ between all containers (if you havent set it manually) | 16:23 |
| cgoncalves | johnsom, amp_boot_network_list is under [controller_worker] so... yeah... misleading | 16:23 |
| johnsom | cgoncalves Is it? Look at the diagram I linked? | 16:24 |
| cgoncalves | *facepalm* | 16:25 |
| johnsom | I didn't screw everything up with the controller worker.... grin | 16:25 |
| cgoncalves | ok, that is somewhat on me as I touched that part of tripleo to fix a deployment issue | 16:26 |
| cgoncalves | dolly, could you kindly set amp_boot_network_list also for health manager and check if it solves? | 16:26 |
| dolly | cgoncalves: pretty sure it does, I just set all the settings you define in /etc/octavia/conf.d/octavia-worker/worker-post-deploy.conf in /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf and restarted my containers | 16:31 |
| dolly | I could see that the network now is attached to the amp | 16:31 |
| dolly | but you also need to define the sec group | 16:31 |
| dolly | not really sure why you would want to have different configs for all the containers | 16:32 |
| dolly | spontainiously it seems to complicate things, just use one config in all containers, no ? | 16:32 |
| dolly | recreating the lb now, will try to delete one amp to see that it recreates is as expected | 16:33 |
| johnsom | Agreed, our current configuration setup is not setup to splitting the config file for each service. Really the only one that *might* be less is the API process. | 16:33 |
| *** velizarx has joined #openstack-lbaas | 16:34 | |
| *** openstackgerrit has joined #openstack-lbaas | 16:38 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Fix tests to honor Octavia API versioning https://review.openstack.org/594786 | 16:38 |
| dolly | BOOM! | 16:38 |
| dolly | And *finally* I got it working the way I wanted to =) | 16:38 |
| cgoncalves | awesome!! | 16:39 |
| dolly | And I manage to learn a thing or two on the way as well! | 16:39 |
| dolly | Deleted the master amp, switchover worked, recreation of the new amp worked, | 16:39 |
| cgoncalves | dolly, thanks so much for reporting and help troubleshooting | 16:39 |
| dolly | When the amp came back though, there was a shortage on the vip (no available backends) for a couple of seconds, but that maybe is expected ? | 16:40 |
| cgoncalves | I will follow-up with fixes on tripleo | 16:40 |
| johnsom | dolly Congratulations! | 16:40 |
| dolly | I'm watching the VIP via curl | 16:40 |
| cgoncalves | probably the newly created amphora took over the vrrp ip...? | 16:41 |
| johnsom | dolly, if you are running in "standalone" topology, then failover will incur an outage, usually less than a minute. | 16:41 |
| dolly | johnsom: running ACTIVE_STANDBY | 16:41 |
| dolly | its no biggie, I'm just curious | 16:42 |
| cgoncalves | heh, which is even more awesome to hear it is working fine | 16:42 |
| johnsom | If you are in active/standby it should not have an outage. You may see a slight delay in a request response, but should not see any connection errors or timeouts | 16:42 |
| cgoncalves | not only in master but queens! | 16:42 |
| dolly | johnsom: ok, so here's what I see. $ > watch 0,1 curl --silent VIP. This gives me a text on which vm answers (I have three). So when deleteing the amp that is acting as MASTER the connection/curl "hangs" for a couple of seconds, then start showing me my VM's again. When the healthmanager had recreated the amp, my curl was showing me "no available backends" for a couple of seconds... And then everything came back to "normal" | 16:45 |
| dolly | Does that makes sence ? I would expect that the amp that got recreated just silently "joined" and no interuption at all would occur. | 16:46 |
| johnsom | Yeah, ok, so that is something wrong / bug. It means the newly created amp pre-empted the existing amphora. Let's file a story against that as there is code that should prevent that. | 16:46 |
| johnsom | Please include in the story if the amphora image is centos or ubuntu | 16:47 |
| johnsom | I'm guessing it's centos.... (hoping?) | 16:47 |
| dolly | Yes its centos | 16:47 |
| johnsom | Yeah, it could be a version issue with the keepalived in centos. | 16:47 |
| cgoncalves | from our periodic amp builds (tarballs.o.o) | 16:47 |
| dolly | Would you need logs as well ? | 16:48 |
| dolly | (from what) | 16:48 |
| johnsom | Ok, yeah, let's open a story (bug) on it. | 16:48 |
| johnsom | the syslog from the amp that did not get recreated, specifically lines from the keepalived process would be helpful | 16:48 |
| johnsom | But not critical | 16:49 |
| dolly | ok | 16:49 |
| *** velizarx has quit IRC | 16:50 | |
| johnsom | FYI, these two lines are supposed to prevent that: https://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/keepalived/jinja/templates/keepalived_base.template#L27-L28 | 16:51 |
| johnsom | But, maybe got broken somehow | 16:51 |
| cgoncalves | that sounds familiar | 16:52 |
| cgoncalves | nmagnezi, ^ nopreempt | 16:52 |
| johnsom | cgoncalves You might like this: https://review.openstack.org/595257 | 16:54 |
| johnsom | Opps, ok, not the syntax error part | 16:54 |
| johnsom | lol | 16:54 |
| cgoncalves | \o/ !! | 16:55 |
| johnsom | Fixed the tempest plugin to honor the API version | 16:55 |
| johnsom | Runs against queens now | 16:55 |
| dolly | cgoncalves: hm, we make use of this parameter in our heat-template, OctaviaAmphoraSshKeyFile: | 16:56 |
| cgoncalves | speaking of CI, can this get some review attention https://review.openstack.org/#/c/588883/ | 16:57 |
| dolly | But since I use another image now.. | 16:57 |
| dolly | I guess that one isnt inserted or how dos it work ? | 16:57 |
| cgoncalves | so that we can proceed with https://review.openstack.org/#/c/587414, https://review.openstack.org/#/c/587442/ and https://review.openstack.org/594078 | 16:57 |
| cgoncalves | dolly, should work. you just need to login with a different user name | 16:58 |
| cgoncalves | 'centos' instead of 'cloud-user' | 16:58 |
| dolly | oh | 16:58 |
| johnsom | cgoncalves This stuff confused me before, going to be around for a few minutes if I have questions? | 16:58 |
| cgoncalves | johnsom, sure | 16:59 |
| dolly | cgoncalves: awesome that worked | 16:59 |
| cgoncalves | great, because I did that part. what a relieve xD | 16:59 |
| dolly | :D | 16:59 |
| dolly | cgoncalves: so where do I find keepalived logs ? | 17:04 |
| johnsom | nmagnezi If you are around: https://review.openstack.org/#/c/588883 | 17:04 |
| nmagnezi | johnsom, yeah I noticed nopreempt is not respected, was pulled of to a customer issue so didn't get the chance to dig more | 17:04 |
| johnsom | Ok, yeah, I *think* this is working ok on ubuntu version of keepalive, but don't quote me on it. | 17:05 |
| nmagnezi | interesting | 17:06 |
| johnsom | cgoncalves yeah, let's work through that change | 17:06 |
| nmagnezi | I know for a fact that it does not on centos. Maybe indeed the keepalived version makes the difference | 17:06 |
| dolly | nmagnezi: ok, well then I don't need to dig more then if you know that its broken on centos ? | 17:06 |
| dolly | (i just noticed) | 17:07 |
| johnsom | I did some failover demos/tests recently (see the new gate patch in tempest-plugin) and I don't think I had that problem, but worth testing again when looking for it | 17:07 |
| cgoncalves | dolly, I don't know by heart. have you checked /var/log/messages? also try something like "journalctl -f -u keepalived" | 17:07 |
| dolly | I cant really find any logs anyway in the centos | 17:07 |
| dolly | yep, doesnt say much | 17:07 |
| cgoncalves | is the priority of master and backup the same? | 17:08 |
| johnsom | Ah, sorry, yeah, I use ubuntu mostly, so the logs might live somewhere else on a centos amp | 17:08 |
| dolly | Oh wait | 17:08 |
| dolly | there is actually logs | 17:08 |
| nmagnezi | johnsom, voted | 17:09 |
| johnsom | Thank you | 17:09 |
| nmagnezi | On the centos amp you'll see what you need for keepalived on /var/log/messages | 17:09 |
| nmagnezi | dolly ^^ | 17:10 |
| dolly | nmagnezi: got it | 17:10 |
| dolly | nmagnezi: johnsom: got the logs from keepalived, you want me to create a bug-report of some sort or ? (regarding MASTER/BACKUP transition) | 17:15 |
| johnsom | dolly Yes please: https://storyboard.openstack.org/#!/project/openstack/octavia | 17:16 |
| dolly | johnsom: ok | 17:17 |
| dolly | I just add a new "story" with the issue ? | 17:17 |
| dolly | (sorry never used this before) | 17:17 |
| johnsom | Yes, all openstack projects are moving to use storyboard instead of launchpad. | 17:18 |
| dolly | Ok ok | 17:18 |
| johnsom | Just assign the project as "openstack/octavia" | 17:18 |
| dolly | ok ok | 17:18 |
| cgoncalves | does anyone happen to have a act-standby topology that could vrrp priority in both? | 17:20 |
| cgoncalves | nevermind, 100 / 90 | 17:20 |
| *** ktibi has quit IRC | 17:24 | |
| *** bcafarel has quit IRC | 17:25 | |
| *** dulek has quit IRC | 17:25 | |
| *** dims has quit IRC | 17:25 | |
| *** Eugene__ has quit IRC | 17:25 | |
| dolly | Hm, cant you attach logs to this storyboard thing | 17:30 |
| johnsom | Yeah, you can paste them in a comment (It's an open bug with the storyboard team). | 17:32 |
| johnsom | It takes markup too, so you can preview and label it code | 17:32 |
| dolly | Just seemed weird to paste logs as a comment, I thought I could upload an attachment | 17:33 |
| dolly | non the less, created the story | 17:33 |
| dolly | Received advert with lower priority 90, ours 100, forcing new election <- that seems like an interesting line. | 17:34 |
| dolly | https://storyboard.openstack.org/#!/story/2003528 | 17:34 |
| colin- | i'm having a hard time determining the state of the UDP project from the storyboard page- is that ongoing and slated for rocky or a later release? (or is that known yet?) | 17:34 |
| colin- | https://storyboard.openstack.org/#!/story/1657091 | 17:34 |
| johnsom | colin- The release notes are helpful for this: https://docs.openstack.org/releasenotes/octavia/ | 17:36 |
| johnsom | UDP is a Rocky feature | 17:36 |
| colin- | got it, ty | 17:36 |
| *** dims_ has joined #openstack-lbaas | 17:36 | |
| johnsom | There is still some work to do on it for all of the features, but the basic UDP support is in Rocky | 17:36 |
| colin- | do you know off the top of your head the minimum version of neutron it would require? | 17:36 |
| colin- | if that's in these notes too i'll eat my keyboard | 17:37 |
| johnsom | No it's not in there as there is no dependency on neutron for the UDP support. So, it should work as far back as at least liberty???? | 17:37 |
| dolly | johnsom cgoncalves nmagnezi Again, thanks for all the help, fantastic work!. I need to run, but I'll come back tomorrow. | 17:37 |
| colin- | oh that's great to hear, thanks | 17:37 |
| rm_work | johnsom: we don't actually touch tune.maxrewrite or tune.bufsize, do we... thought we had stuff for that but our base config is lacking, so i guess we use defaults? | 17:48 |
| johnsom | rm_work So many settings, you mean in haproxy config? | 17:49 |
| rm_work | yeah | 17:49 |
| johnsom | No, we aren't changing that today | 17:49 |
| johnsom | I do have some tuning I need to add though... I can send them to you if you are going to spin a patch | 17:49 |
| rm_work | interested, but was just trying to answer a user's question | 17:50 |
| cgoncalves | dolly, thank you! I'll try to keep you posted on the misconfiguration of octavia. feel more than free to nag me on that | 17:51 |
| johnsom | rm_work | 17:53 |
| johnsom | https://www.irccloud.com/pastebin/6B64F31R/ | 17:53 |
| rm_work | cool thanks | 17:53 |
| johnsom | I want to add those to our default (oh, except no log which is already there) | 17:53 |
| rm_work | k yeah | 17:53 |
| *** dmellado has quit IRC | 18:22 | |
| *** bcafarel has joined #openstack-lbaas | 18:27 | |
| *** evgenyf has joined #openstack-lbaas | 18:31 | |
| evgenyf | / | 18:35 |
| openstackgerrit | Merged openstack/octavia master: Temporarily remove octavia-v2-dsvm-scenario-ubuntu.bionic https://review.openstack.org/588883 | 18:38 |
| openstackgerrit | Merged openstack/octavia-dashboard master: Imported Translations from Zanata https://review.openstack.org/594902 | 18:41 |
| rm_work | oh hey evgenyf! :) | 18:44 |
| rm_work | long time no see | 18:44 |
| evgenyf | rm_work: Hi ! Yes, indeed, is there a meeting today? | 18:51 |
| rm_work | I think so | 18:51 |
| rm_work | in about an hour? | 18:52 |
| evgenyf | good, I started a provider driver development for Octavia and have questions ) | 18:52 |
| johnsom | o/ | 18:52 |
| johnsom | evgenyf Fire away | 18:53 |
| evgenyf | Hi Johnsom ! | 18:53 |
| evgenyf | Several things actually, I thought of composing an email to Octavia fellows and list my points | 18:54 |
| johnsom | Up to you or we can discuss here | 18:55 |
| evgenyf | But we can chat about it here if you have time. It's all about data models transferring from controller data models to provider data models | 18:55 |
| johnsom | Yeah, I have an hour before the weekly IRC meeting | 18:56 |
| evgenyf | Good ! the most important is the missing backrefs in provider data models, I saw the utils code which removes back refs | 18:57 |
| johnsom | Yes, the objects are not database bound and are all independent objects. | 18:57 |
| johnsom | This was part of the spec as the vendors were not happy passing around so much information for each call | 18:58 |
| evgenyf | Yes, I remember this discussion | 18:59 |
| evgenyf | As I mentioned when ,our (Radware) solution depends on fully populated LB graph | 19:01 |
| johnsom | Hmmm, ok. Let's discuss options. | 19:02 |
| evgenyf | I wonder how can a vendor get the full LB on each call from a controller | 19:03 |
| johnsom | We do not have a way to do that today. Both drivers implemented to date don't need that. | 19:03 |
| johnsom | Just so I understand, you are basically deleting the existing config and recreating for each CRUD call? | 19:04 |
| evgenyf | Who is the second vendor? the first is octavia itself right? | 19:04 |
| johnsom | Yes, one other vendor has a driver working, just not released yet | 19:04 |
| evgenyf | Not really, we send the full graph and the back-end is doing diffs with current setup | 19:05 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Gate on CentOS 7 and check on Ubuntu Bionic https://review.openstack.org/587414 | 19:06 |
| johnsom | How do you feel about making an Octavia API call to get that graph representation? | 19:07 |
| johnsom | I have a "vision" for a "backup your load balancer configuration" feature. This might work nicely for your need too. | 19:08 |
| evgenyf | call Octavia API call from driver code you mean? | 19:08 |
| johnsom | Yes | 19:08 |
| evgenyf | Sounds good, for that, each CRUD call from controller for any entity should have an lb_id accessible from the provider data model | 19:09 |
| johnsom | No, it looks like the current data model does not include the lb_id in all requests. You don't have any references on your side? | 19:11 |
| johnsom | They all have a reference to their parent object though | 19:12 |
| johnsom | We can probably make that query parameter work though through the API query parameters | 19:13 |
| evgenyf | Yes, given a provider DM object, Octavia API can find the LB id, you mean | 19:15 |
| johnsom | Yes | 19:15 |
| evgenyf | So how can we do it? Is it going to be a spec? | 19:16 |
| johnsom | Yes, it will need to be a new spec and new feature. | 19:16 |
| openstackgerrit | Nir Magnezi proposed openstack/neutron-lbaas master: DNM: Test changes to locale not triggering tempest https://review.openstack.org/595322 | 19:17 |
| evgenyf | ok, in general, I have other suggestions in terms of controller/provider data models transitions, it may simplify the code in controllers and utils code. Actuall, the only difference between full DM and provider DM is a removal of all data "not necessary for provider" | 19:18 |
| evgenyf | Anyhow, I will propose an RST | 19:20 |
| johnsom | Well, there are translations as well. The Octavia data model has some fields that are not super clear or named well, so some of those were translated to "driver" friendly names. | 19:22 |
| johnsom | A proposal would be great. | 19:22 |
| evgenyf | Yes, enabled-admin_state_up, etc | 19:24 |
| evgenyf | Great! thanks | 19:24 |
| johnsom | Cores - There is a translation looking for a +2/+W https://review.openstack.org/#/c/594903/ | 19:27 |
| nmagnezi | I'm on it | 19:30 |
| johnsom | Thanks! We can have the RC3 discussion during the meeting | 19:30 |
| nmagnezi | Sure :) | 19:31 |
| *** evgenyf has quit IRC | 19:46 | |
| *** pcaruana has quit IRC | 19:47 | |
| cgoncalves | cores: https://review.openstack.org/#/c/595336/ | 19:58 |
| nmagnezi | johnsom, o/ | 20:00 |
| johnsom | #startmeeting Octavia | 20:00 |
| openstack | Meeting started Wed Aug 22 20:00:54 2018 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 20:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 20:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 20:00 | |
| openstack | The meeting name has been set to 'octavia' | 20:00 |
| rm_work | o/ | 20:01 |
| nmagnezi | O/ | 20:01 |
| colin- | o/ | 20:01 |
| johnsom | Hi folks! | 20:01 |
| cgoncalves | o/ | 20:01 |
| johnsom | #topic Announcements | 20:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 20:01 | |
| johnsom | I continue to update the PTG topics etherpad | 20:01 |
| johnsom | #link https://etherpad.openstack.org/p/octavia-stein-ptg | 20:01 |
| johnsom | Please add your topics and attendance to the etherpad | 20:02 |
| johnsom | Rocky RC2 Octavia projects has been released | 20:02 |
| johnsom | We will talk below about if we want an RC3. | 20:02 |
| johnsom | Also, I have posted a patch for stable/queens 2.0.2, but it is waiting for release team merge | 20:02 |
| johnsom | Any other announcements today? | 20:03 |
| johnsom | Oh, I should note, I will be on vacation on Friday this week. | 20:04 |
| johnsom | So limited availability this coming weekend | 20:04 |
| nmagnezi | Enjoy :-) | 20:04 |
| johnsom | Going fishing.... | 20:05 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:05 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 20:05 | |
| cgoncalves | so much vacations. tsss, tsss :P | 20:05 |
| johnsom | lol, cgoncalves look who is talking.... | 20:05 |
| colin- | nice, enjoy | 20:05 |
| johnsom | I have been focused on making sure we get stuff merged for the Rocky release | 20:05 |
| johnsom | The OpenStack demons threw a bandit change, a KVM breakage, and a stable/rocky nova breakage at us.... | 20:06 |
| johnsom | So, a bit of a dance to get our stuff in. | 20:06 |
| johnsom | I also worked on our octavia-tempest-plugin to make it aware of the API version it is testing. | 20:07 |
| johnsom | I have a patch up for that, but it looks like I have a py3 bug to fix. | 20:07 |
| cgoncalves | #link https://review.openstack.org/#/c/594786/ | 20:07 |
| johnsom | ha, faster than I was | 20:07 |
| johnsom | Yeah, that one | 20:07 |
| cgoncalves | also octavia-tempest-plugin jobs for stable/queens | 20:08 |
| johnsom | Comments welcome, unless you are just mocking me for my poor py3 form.... | 20:08 |
| cgoncalves | #link https://review.openstack.org/#/c/595257/ | 20:08 |
| cgoncalves | cool stuff! | 20:08 |
| johnsom | Otherwise, still busy with internal stuffs. | 20:08 |
| johnsom | Any other updates from folks? | 20:08 |
| johnsom | ok.... | 20:10 |
| cgoncalves | resuming work on CI jobs moving to v2, etc now that cut stable/rocky | 20:10 |
| johnsom | Yes, thanks for that work cleaning up some of the zuul configs. | 20:10 |
| cgoncalves | it also came to my attention that we may have issues when deploying octavia on a ipv6 service network | 20:11 |
| johnsom | For lb-mgmt-net or VIP? | 20:11 |
| nmagnezi | The octavia api service | 20:11 |
| nmagnezi | Fails to bind to an IPv6 address | 20:11 |
| johnsom | Oh, so you aren't deploying as a wsgi app???? | 20:12 |
| cgoncalves | #link https://bugzilla.redhat.com/show_bug.cgi?id=1618689 | 20:12 |
| openstack | bugzilla.redhat.com bug 1618689 in openstack-octavia "[OSP13][Octavia] octavia-api fails to start with IPv6" [Urgent,New] - Assigned to nmagnezi | 20:12 |
| cgoncalves | #link https://review.openstack.org/#/c/594078/ | 20:12 |
| nmagnezi | cgoncalves, you have a live setup to double check? | 20:12 |
| cgoncalves | I do not | 20:12 |
| johnsom | Yeah, so you are running with the python "simple-server" | 20:13 |
| johnsom | hmmm, cgoncalves We had originally decided to integrate the IPv6 tests into the existing tests. That is how it is setup today. | 20:14 |
| johnsom | But I guess this is control plane IPv6 in this test? | 20:14 |
| cgoncalves | yes, control plane | 20:14 |
| cgoncalves | better yet, all ipv6 | 20:14 |
| johnsom | Yeah. I found a bug with IPv6 VIP running in active/standby recently too | 20:15 |
| johnsom | #link https://storyboard.openstack.org/#!/story/2003451 | 20:15 |
| johnsom | It works fine in standalone, but not active/standby | 20:15 |
| johnsom | I will be looking at that in the next week or two | 20:15 |
| cgoncalves | "DAD went out to buy a pack of cigarettes" | 20:16 |
| johnsom | FYI, you might want to consider moving away from using simple-server to a better wsgi app server. | 20:16 |
| johnsom | lol, yeah, that error | 20:17 |
| cgoncalves | ok, thanks | 20:17 |
| johnsom | We use Apache2 with uwsgi in the gates | 20:17 |
| openstackgerrit | boden proposed openstack/neutron-lbaas master: use setup_extension in unit tests https://review.openstack.org/595345 | 20:18 |
| johnsom | lol | 20:18 |
| johnsom | Ok, any other updates? | 20:18 |
| colin- | nothing to share :) | 20:18 |
| johnsom | #topic Do we want to cut an RC3? | 20:18 |
| *** openstack changes topic to "Do we want to cut an RC3? (Meeting topic: Octavia)" | 20:18 | |
| johnsom | I see roughly three patches as candidates for an RC3 today: | 20:19 |
| johnsom | #link https://review.openstack.org/#/c/594545/ | 20:19 |
| johnsom | Remove user_group option (deprecated cleanup) | 20:19 |
| johnsom | and two translation updates: | 20:19 |
| johnsom | #link https://review.openstack.org/594903 | 20:19 |
| johnsom | #link https://review.openstack.org/594059 | 20:20 |
| johnsom | which has merged already | 20:20 |
| johnsom | Any reason to not add those to the Rocky release with an RC3? | 20:20 |
| cgoncalves | I don't see a reason why not to | 20:20 |
| nmagnezi | I'll vote: yes | 20:21 |
| johnsom | Ha | 20:21 |
| colin- | seems reasonable | 20:21 |
| nmagnezi | We better remove options we don't want to support as long as we can | 20:21 |
| nmagnezi | :) | 20:21 |
| johnsom | Ok, just wanted to check with folks. I will post a patch after the meeting | 20:21 |
| johnsom | #topic Python 3 first goal | 20:22 |
| *** openstack changes topic to "Python 3 first goal (Meeting topic: Octavia)" | 20:22 | |
| johnsom | Doug is leading the Python 3 first goal for Stein. He has asked for volunteer projects that want to move sooner rather than later. | 20:22 |
| johnsom | This is switching the bulk of the jobs to run under py3 by default (we currently run both, but things like docs, pep8 are p27 by default) | 20:23 |
| johnsom | I have been holding off volunteering simply because my time to help/focus on that is limited. | 20:23 |
| johnsom | I thought I would ask if anyone has strong enough feeling on this topic that they want to volunteer and lead the Octavia transition. | 20:23 |
| cgoncalves | Stein has a long dev cycle. we could try to make it | 20:24 |
| cgoncalves | we have py35 jobs today and have been on the green side AFAIK | 20:24 |
| johnsom | Ha, yeah, it will happen by the end of stein, it's just if we want to be "early" (adopters) | 20:24 |
| johnsom | Yes, we run the actual tests on both today | 20:24 |
| nmagnezi | This kinda also relate to the fact that we will need to adapt to RHEL8 which should drop support of Python 2.x. | 20:25 |
| nmagnezi | #link https://www.phoronix.com/scan.php?page=news_item&px=RHEL-8-No-Python-2 | 20:25 |
| johnsom | It is mostly the 'other' jobs in our case, like docs/api-reg/pep8 | 20:25 |
| nmagnezi | But I personally can't make promises for early adoption since we internally are still very busy with queens and rocky | 20:25 |
| johnsom | Ok, just asking if anyone wanted to jump on this. If not, we will pick it up in a bit. | 20:26 |
| nmagnezi | Can it wait for a week or two so we'll know better? | 20:26 |
| * johnsom Watches everyone take a step back.... | 20:26 | |
| johnsom | Yep, no rush really. | 20:27 |
| *** velizarx has joined #openstack-lbaas | 20:27 | |
| johnsom | Ok, fair enough. | 20:27 |
| johnsom | #topic Open Discussion | 20:27 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 20:27 | |
| nmagnezi | We only started the planning for OSP15 (Stein) now. | 20:27 |
| johnsom | Other topics for today? | 20:27 |
| cgoncalves | why not have a patch that switches open for couple of weeks where we every now and then rebase to get a sense on how it is doing? | 20:27 |
| johnsom | Most of the jobs are infra owned in project-config, so not something we can just do without signing up to be "early" | 20:28 |
| cgoncalves | uhm? we can override parameters in our jobs | 20:29 |
| johnsom | I don't think so, I believe it is different parent jobs. cgoncalves does this mean you have interest and want to volunteer? | 20:30 |
| * johnsom grins | 20:30 | |
| cgoncalves | time permitting, yes | 20:30 |
| johnsom | There is a few e-mail chains on the dev mailing list from Doug if you want more details. | 20:31 |
| cgoncalves | devstack_localrc: USE_PYTHON3: true | 20:31 |
| nmagnezi | cgoncalves, may the force be with you | 20:31 |
| nmagnezi | johnsom, you're going to lead that effort for neutron-lbaas? :D | 20:32 |
| johnsom | Also, if someone has time, I could use a second set of eyes on why the py27 n-lbaas jobs are failing on stable/rocky. I temporarily disabled them to get our gates going. All I see is nova-compute throwing a critical error and exiting. Asking for help in nova channel didn't go far. | 20:32 |
| johnsom | cgoncalves We already do that setting for our gates.... That is the py3 gates we already run | 20:33 |
| cgoncalves | johnsom, I know. so we move that to the base octavia job and for the today's py27 jobs we set it to False | 20:33 |
| johnsom | Hahaha | 20:34 |
| cgoncalves | dunno xD | 20:34 |
| johnsom | If OpenStack was so easy | 20:34 |
| cgoncalves | no? I'll try to find Doug's email | 20:34 |
| johnsom | #link http://lists.openstack.org/pipermail/openstack-dev/2018-August/133232.html | 20:37 |
| cgoncalves | https://media.giphy.com/media/3ohs7KViF6rA4aan5u/giphy.gif | 20:37 |
| johnsom | Right.... | 20:37 |
| johnsom | Any other topics today? | 20:38 |
| johnsom | Ok, then.... Going once | 20:39 |
| johnsom | lunch time for me | 20:39 |
| colin- | thx for hosting | 20:40 |
| johnsom | Ok, thanks folks! Have a great week. | 20:40 |
| nmagnezi | o/ | 20:40 |
| johnsom | #endmeeting | 20:40 |
| *** openstack changes topic to "Discussion of OpenStack Load Balancing (Octavia) | https://etherpad.openstack.org/p/octavia-priority-reviews" | 20:40 | |
| openstack | Meeting ended Wed Aug 22 20:40:30 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:40 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-08-22-20.00.html | 20:40 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-08-22-20.00.txt | 20:40 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-08-22-20.00.log.html | 20:40 |
| *** harlowja has joined #openstack-lbaas | 20:42 | |
| *** velizarx has quit IRC | 21:00 | |
| *** velizarx has joined #openstack-lbaas | 21:02 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Fix tests to honor Octavia API versioning https://review.openstack.org/594786 | 21:07 |
| *** velizarx has quit IRC | 21:12 | |
| rm_work | johnsom: umm, you know of a way to debug the message hmac? | 21:35 |
| rm_work | getting a ton of calculated hmac mismatches | 21:35 |
| johnsom | rm_work So you have a mis-match amp and controller. We put a release note in about that. | 21:36 |
| rm_work | hmmmmm | 21:36 |
| rm_work | i haven't updated in a while tho, lol | 21:36 |
| rm_work | weird, but i'll check | 21:36 |
| johnsom | The fix for the hmac.compare_digest on python3 requires you to upgrade your health managers before updating the amphora image. The health manager is compatible with older amphora images, but older controllers will reject the health heartbeats from images with this fix. | 21:36 |
| rm_work | yeah i'm not on py3 either heh | 21:37 |
| johnsom | Yeah, but that change changed the format for all versions | 21:37 |
| rm_work | ah | 21:37 |
| rm_work | hmmmmm | 21:37 |
| rm_work | i've definitely upgraded my HMs tho | 21:37 |
| rm_work | at least in sync with the image | 21:38 |
| rm_work | hmmm | 21:38 |
| rm_work | will dig more | 21:38 |
| rm_work | some of the amps work... | 21:38 |
| rm_work | just not all | 21:38 |
| rm_work | looking to see if there's old/new ones | 21:38 |
| rm_work | yeah i think these are older images | 21:39 |
| rm_work | but that's... odd | 21:39 |
| rm_work | since that wouldn't have been one of the issues | 21:39 |
| *** abaindur has joined #openstack-lbaas | 21:39 | |
| johnsom | The HM should fall back to the old format. Wonder if it's just logging and still accepting | 21:39 |
| rm_work | it SAYS "dropping packet" | 21:40 |
| rm_work | :/ | 21:40 |
| johnsom | Ah, yeah, looking at the patch, it is still logging. Sigh | 21:41 |
| johnsom | https://review.openstack.org/#/c/571333/9/octavia/amphorae/backends/health_daemon/status_message.py | 21:41 |
| rm_work | seems it's all OLDER messages tho, lol | 21:41 |
| rm_work | err | 21:41 |
| rm_work | OLDER amps | 21:41 |
| rm_work | that are getting that failure | 21:41 |
| rm_work | oh | 21:41 |
| rm_work | right | 21:41 |
| rm_work | k | 21:41 |
| johnsom | Yeah, it tries the new format, then goes back to the old. But that first try logs | 21:42 |
| rm_work | T_T | 21:42 |
| rm_work | yeah ok | 21:42 |
| rm_work | i see now | 21:42 |
| rm_work | it still submits it | 21:43 |
| rm_work | ok so i confirmed what i wanted to confirm, which is that the HM is getting member up/down events flapping for a LB | 21:46 |
| *** velizarx has joined #openstack-lbaas | 21:46 | |
| rm_work | now trying to figure out if i can confirm that it's haproxy actually flapping, or just a HM issue | 21:46 |
| johnsom | Super highly likely the tenant app is flappy | 21:47 |
| rm_work | yeah need to confirm | 21:48 |
| rm_work | what am i looking for in the `show stat` | 21:48 |
| rm_work | which column would it be | 21:48 |
| johnsom | chkfail [...S]: number of failed checks. (Only counts checks failed when | 21:49 |
| johnsom | the server is up.) | 21:49 |
| rm_work | k | 21:49 |
| johnsom | chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts | 21:50 |
| johnsom | transitions to the whole backend being down, rather than the sum of the | 21:50 |
| johnsom | counters for each server. | 21:50 |
| johnsom | lastchg [..BS]: number of seconds since the last UP<->DOWN transition | 21:50 |
| johnsom | might also be fun | 21:50 |
| johnsom | check_status would give details too | 21:51 |
| rm_work | check_status seems to always be L4OK | 21:56 |
| rm_work | hmmmmmmmmm | 21:56 |
| rm_work | ok so lastchg is really high <_< | 21:56 |
| rm_work | for all of this | 21:56 |
| rm_work | so .... | 21:56 |
| rm_work | must be something with the status messages being messed up? | 21:57 |
| rm_work | hmm | 21:57 |
| *** velizarx has quit IRC | 22:11 | |
| *** abaindur has quit IRC | 22:12 | |
| *** abaindur has joined #openstack-lbaas | 22:13 | |
| rm_work | AH figured it out | 22:25 |
| rm_work | they'd noticed which IP traffic was coming from, and ACL'd it open, and blocked everything else | 22:25 |
| rm_work | so the backup amp was blocked | 22:25 |
| rm_work | so the health messages were interleaving | 22:26 |
| johnsom | There you go | 22:27 |
| *** abaindur has quit IRC | 22:28 | |
| *** rcernin has joined #openstack-lbaas | 22:34 | |
| *** fnaval has quit IRC | 22:39 | |
| *** colby_ has quit IRC | 23:18 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: DNM: three dumb downstream things to fix, IGNORE https://review.openstack.org/593986 | 23:28 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
