| *** hongbin has quit IRC | 00:35 | |
| *** spatel has joined #openstack-lbaas | 00:49 | |
| *** hongbin has joined #openstack-lbaas | 00:57 | |
| *** redrobot has quit IRC | 01:08 | |
| *** spatel has quit IRC | 01:10 | |
| *** hongbin has quit IRC | 02:23 | |
| *** hongbin has joined #openstack-lbaas | 02:31 | |
| openstackgerrit | Zihao Wang proposed openstack/octavia master: Bump hacking min version to 3.0.1 https://review.opendev.org/752845 | 02:50 |
|---|---|---|
| openstackgerrit | XiaoYu Zhu proposed openstack/octavia master: Alternative Distributor for L3 Active-Active, N+1 Amphora Setup https://review.opendev.org/746688 | 02:56 |
| *** ramishra has joined #openstack-lbaas | 02:59 | |
| *** sapd1 has joined #openstack-lbaas | 03:20 | |
| *** psachin has joined #openstack-lbaas | 03:41 | |
| *** ramishra has quit IRC | 03:59 | |
| *** zzzeek has quit IRC | 04:15 | |
| *** zzzeek has joined #openstack-lbaas | 04:17 | |
| *** hongbin has quit IRC | 04:20 | |
| *** ramishra has joined #openstack-lbaas | 04:21 | |
| *** hongbin has joined #openstack-lbaas | 04:24 | |
| *** hongbin has quit IRC | 04:29 | |
| *** rcernin has quit IRC | 04:31 | |
| *** hongbin has joined #openstack-lbaas | 04:38 | |
| *** rcernin has joined #openstack-lbaas | 04:40 | |
| *** vishalmanchanda has joined #openstack-lbaas | 05:15 | |
| *** hongbin has quit IRC | 05:29 | |
| *** hongbin has joined #openstack-lbaas | 05:31 | |
| *** hongbin has joined #openstack-lbaas | 05:36 | |
| *** rcernin has quit IRC | 05:39 | |
| *** hongbin has quit IRC | 05:40 | |
| *** sapd1 has quit IRC | 05:46 | |
| *** gthiemon1e is now known as gthiemonge | 06:00 | |
| *** rcernin has joined #openstack-lbaas | 06:01 | |
| *** gcheresh has joined #openstack-lbaas | 06:02 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Set Grub timeout to 0 for fast boot times https://review.opendev.org/752853 | 06:03 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Set Grub timeout to 0 for fast boot times https://review.opendev.org/752854 | 06:04 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/stein: Set Grub timeout to 0 for fast boot times https://review.opendev.org/752855 | 06:05 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Update devstack plugin https://review.opendev.org/752856 | 06:07 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Update devstack plugin https://review.opendev.org/752858 | 06:07 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/stein: Update devstack plugin https://review.opendev.org/752859 | 06:08 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Add missing log line for finishing amp operations https://review.opendev.org/752860 | 06:09 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Add missing log line for finishing amp operations https://review.opendev.org/752861 | 06:11 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/stein: Add missing log line for finishing amp operations https://review.opendev.org/752862 | 06:12 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Fix user permission for WSGI configuration Task: 35692 Story: 2006172 https://review.opendev.org/752863 | 06:13 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Fix user permission for WSGI configuration Task: 35692 Story: 2006172 https://review.opendev.org/752864 | 06:14 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Add some details on enable_anti_affinity option https://review.opendev.org/752865 | 06:15 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Add some details on enable_anti_affinity option https://review.opendev.org/752866 | 06:16 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/stein: Add some details on enable_anti_affinity option https://review.opendev.org/752867 | 06:16 |
| *** rcernin has quit IRC | 06:20 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Fix operational status for disabled UDP listeners https://review.opendev.org/752869 | 06:23 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Fix invalid DOWN status when updating a UDP pool https://review.opendev.org/752870 | 06:23 |
| *** mchlumsky has quit IRC | 06:28 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Fix operational status for disabled UDP listeners https://review.opendev.org/752871 | 06:29 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/train: Fix invalid DOWN status when updating a UDP pool https://review.opendev.org/752872 | 06:29 |
| *** mchlumsky has joined #openstack-lbaas | 06:35 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/stein: Fix operational status for disabled UDP listeners https://review.opendev.org/752875 | 06:46 |
| *** ccamposr has joined #openstack-lbaas | 06:47 | |
| *** ccamposr__ has joined #openstack-lbaas | 07:15 | |
| *** ccamposr has quit IRC | 07:17 | |
| *** sapd1 has joined #openstack-lbaas | 07:22 | |
| *** BlackFX has joined #openstack-lbaas | 07:34 | |
| BlackFX | Hi guys. I seem to have an issue with Octavia, all my pool members are operating status = offline, but I have captured the traffic between the LB and the member and I can see the member returning 200 | 07:35 |
| *** rcernin has joined #openstack-lbaas | 07:39 | |
| *** wuchunyang has joined #openstack-lbaas | 07:45 | |
| *** rcernin has quit IRC | 07:52 | |
| BlackFX | Interestingly I cant see the traffic or an interface in the amphora instance. | 08:08 |
| *** gcheresh has quit IRC | 08:11 | |
| BlackFX | [ 4.060126] virtio_net virtio1 eth1: renamed from ens4 | 08:19 |
| BlackFX | I get that in dmesg, but there is no ens4/eth1 | 08:19 |
| *** gcheresh has joined #openstack-lbaas | 08:44 | |
| *** gcheresh has quit IRC | 09:00 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add not-voting aarch64 scenario job https://review.opendev.org/747629 | 09:08 |
| *** gcheresh has joined #openstack-lbaas | 09:10 | |
| *** psachin has quit IRC | 09:11 | |
| *** ataraday_ has joined #openstack-lbaas | 09:19 | |
| *** lemko has quit IRC | 09:35 | |
| *** lemko has joined #openstack-lbaas | 09:35 | |
| *** ramishra has quit IRC | 10:05 | |
| *** wuchunyang has quit IRC | 10:06 | |
| *** ramishra has joined #openstack-lbaas | 10:45 | |
| *** ramishra_ has joined #openstack-lbaas | 11:10 | |
| *** ramishra has quit IRC | 11:12 | |
| *** sapd1 has quit IRC | 11:17 | |
| *** terdei has joined #openstack-lbaas | 11:49 | |
| *** servagem has joined #openstack-lbaas | 12:00 | |
| *** vishalmanchanda has quit IRC | 12:11 | |
| *** njohnston has joined #openstack-lbaas | 12:26 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Fix copy output in two-node jobs https://review.opendev.org/752936 | 12:54 |
| ataraday_ | cgoncalves, Hi! Could you take a look at https://review.opendev.org/#/c/738609/ ? | 12:57 |
| cgoncalves | ataraday_, LGTM | 13:01 |
| ataraday_ | cgoncalves, Thanks! | 13:10 |
| *** numans_ has joined #openstack-lbaas | 13:27 | |
| *** numans_ is now known as numans | 13:28 | |
| *** ataraday_ has quit IRC | 13:28 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Add HTTP/2 example to the load balancing cookbook https://review.opendev.org/752952 | 13:30 |
| *** TrevorV has joined #openstack-lbaas | 13:40 | |
| *** ramishra has joined #openstack-lbaas | 13:44 | |
| *** ramishra_ has quit IRC | 13:48 | |
| *** gcheresh has quit IRC | 13:51 | |
| *** gcheresh has joined #openstack-lbaas | 14:28 | |
| *** hongbin has joined #openstack-lbaas | 14:31 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Add ALPN support for TLS-enabled pools https://review.opendev.org/752095 | 14:42 |
| *** armax has joined #openstack-lbaas | 15:10 | |
| *** redrobot has joined #openstack-lbaas | 15:24 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add non-voting aarch64 scenario job https://review.opendev.org/747629 | 15:32 |
| *** hongbin has quit IRC | 15:38 | |
| johnsom | Thanks for all of the backport updates! | 15:40 |
| johnsom | I think I have one I need to do as well. | 15:40 |
| *** hongbin has joined #openstack-lbaas | 15:44 | |
| cgoncalves | it may be the one (UDP-related) I purposely skipped. was not caffeinated that early in the morning to resolve the merge conflict :P | 15:44 |
| johnsom | I think it is the VRRP port missing patch | 15:45 |
| cgoncalves | ah, also skipped that one purposely. "grin" | 15:46 |
| *** vishalmanchanda has joined #openstack-lbaas | 16:10 | |
| *** sapd1 has joined #openstack-lbaas | 16:10 | |
| *** weiguo has joined #openstack-lbaas | 16:28 | |
| johnsom | BlackFX The tenant traffic is inside a network namespace, use "sudo ip netns exec amphora-haproxy ip a" to see the other interfaces. | 16:31 |
| weiguo | Hi all. Our tenant reported that they were able to get a full list of octavia loadbalancer list (from all tenants) with an unscoped token. We were able to reproduce it with both rocky and train releases we deployed. This raises a serious security concern and we wonder if this is expected and we might have missed some configuration? | 16:35 |
| weiguo | And the call we used is like this: curl -X GET https:${URL}:9876/v2.0/loadbalancers/ -H "Accept: application/json" -H "User-Agent: openstacksdk/0.31.1 keystoneauth1/3.14.0 python-requests/2.18.4 CPython/2.7.12" -H "X-Auth-Token: ${my_unscoped_token}" | 16:37 |
| johnsom | That would only happen if they have one of the admin roles, such as the global "Admin" for the cloud, the load-balancer_admin role, or the load-balancer_global_observer role | 16:37 |
| weiguo | not in our test cases, where we only grant normal user / _member_ role. | 16:38 |
| johnsom | This is the code that checks that: https://github.com/openstack/octavia/blob/master/octavia/api/v2/controllers/base.py#L224 | 16:43 |
| weiguo | Thanks, Michael. Let me put some debug flag in that file and verify if this is a bug or not. | 16:46 |
| johnsom | Yeah, I am going to try to reproduce this locally. The general case doesn't reproduce it, but I need to get an unscoped token | 16:47 |
| *** ccamposr__ has quit IRC | 16:56 | |
| weiguo | You can get an unscope token this way; source my_open.rc; export OS_PROJECT_ID=; export OS_PROJECT_NAME=; my_unscoped_token=$(openstack token issue -f value -c id) | 16:57 |
| johnsom | I did it this way: openstack --os-username=<username> --os-user-domain-name=<domain> --os-password=<password> token issue | 16:58 |
| *** ccamposr has joined #openstack-lbaas | 16:58 | |
| weiguo | Yup, that would do as well | 16:59 |
| johnsom | with a clean environment. I get a 403 Forbidden back. | 16:59 |
| weiguo | ok, let me do a bit testing on our end and report back | 17:00 |
| johnsom | Yeah, even doing it the other way, where there is more context in the environment, I get the users load balancer list and not all of them. | 17:05 |
| *** hongbin has quit IRC | 18:05 | |
| *** hongbin has joined #openstack-lbaas | 18:11 | |
| *** AlexStaf has quit IRC | 18:40 | |
| *** hongbin has quit IRC | 19:05 | |
| *** vishalmanchanda has quit IRC | 19:11 | |
| openstackgerrit | Merged openstack/octavia stable/ussuri: Update amphora v2 for the failover refactor https://review.opendev.org/750632 | 19:21 |
| johnsom | Well, that will make my backport work easier.... | 19:53 |
| *** hongbin has joined #openstack-lbaas | 20:16 | |
| weiguo | Hi Michael, a small question about your clean environment, did you use legacy policy vs the new one? We are using the legacy policy model. Wonder if that might have made a difference. | 20:36 |
| johnsom | I am not using a policy override file | 20:36 |
| johnsom | Yeah, even with the owner-admin policy file I only get my own list back | 20:43 |
| weiguo | do you mind sharing your owner-admin policy file? | 20:44 |
| johnsom | https://opendev.org/openstack/octavia/src/branch/master/etc/policy | 20:45 |
| weiguo | thx | 20:45 |
| johnsom | Note, that version is in yaml, since you are on train you might want to grab the older json version | 20:47 |
| weiguo | We did and verified that we are using the similar one. | 20:52 |
| johnsom | weiguo Also make sure you don't have auth_strategy = noauth in you octavia.conf. That will elevate everyone. | 20:58 |
| weiguo | Mike, we have that set to keystone. BTW, does your test user have a default project set up? According to keystone doc, to get an unscoped token "Your identity must not have a “default project” associated with it that you also have role assignments, and thus authorization, upon." | 21:02 |
| weiguo | In theory, if you get an unscoped token, a cURL call to the API endpoint without any project ID info in the URL should not be able tell what project you belong to. You mentioned that you got a small list back for your own project, which makes me suspect that the token you got is not actual a 'clean' unscoped token | 21:04 |
| *** sapd1 has quit IRC | 21:05 | |
| johnsom | Hmm, that is interesting. I followed your steps to get this last token. I wonder how I would see if that user has a default or not. | 21:08 |
| weiguo | openstack user show xxxx should be able to tell | 21:08 |
| johnsom | I am using this user: | 21:09 |
| johnsom | https://www.irccloud.com/pastebin/yyooDX5m/ | 21:09 |
| johnsom | weiguo Ok, there was a stray tenant ID in there. I can reproduce it with the override file, but cannot with the advanced RBAC enabled (default). | 21:13 |
| weiguo | Cool, at least that matches what we found so far | 21:14 |
| weiguo | we had hoped that we didn't have to create extra roles which requires extra work for our upstream user integration system. Hence we have sticked to our 'legacy' override model. | 21:16 |
| johnsom | Well, you can give everyone a role pretty easy. It's not like each user has to have them added | 21:17 |
| *** sapd1 has joined #openstack-lbaas | 21:18 | |
| *** gcheresh has quit IRC | 21:18 | |
| johnsom | Yeah, so that sample is bogus I think. | 21:30 |
| johnsom | No, ok | 21:34 |
| weiguo | ok, we use basically that. Guess that is the problem? We just tried without the override file and the requested was denied with 403. | 21:34 |
| weiguo | So with that override file, upscoped token will be allowed to list all for sure, yes? | 21:35 |
| johnsom | That appears to be what is happening. I'm just trying to figure out why | 21:36 |
| weiguo | Cool. We will keep trying our end and let you know if we see anything funny. Thanks very much for helping! | 21:37 |
| johnsom | rm_work Are you around? | 21:41 |
| rm_work | I am, sup? | 21:41 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/api/v2/controllers/base.py#L224 | 21:41 |
| johnsom | So currious case. With an unscoped token, it has no project_id, that is allowing it through with the policy override file admin-owner | 21:42 |
| *** TrevorV has quit IRC | 21:55 | |
| rm_work | eugh, amphora agent still splits a bunch of logging to syslog doesn't it | 22:05 |
| rm_work | rather than its unit | 22:05 |
| rm_work | johnsom: also, what am i looking at in that function you linked? | 22:05 |
| johnsom | I was wondering if you remembered much about that. It's doing the wrong thing with unscoped tokens | 22:06 |
| johnsom | I think I have figured it out however. | 22:06 |
| rm_work | it's like a haze | 22:08 |
| rm_work | i think I just refactored? | 22:08 |
| rm_work | wtf, my UDP fix didn't work | 22:09 |
| rm_work | i couldn't replicate the issue in my testing so thought it had | 22:09 |
| rm_work | but just had another instance in prod with the new image where it has the bad check_script present | 22:09 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fixes API list handling of unscoped tokens https://review.opendev.org/753169 | 22:24 |
| johnsom | weiguo That patch will fix your issue | 22:25 |
| weiguo | Awesome, we will verify and let you know. | 22:26 |
| weiguo | johnsom works like a charm. We saw from the debug output eariler in that function earlier and wonder if the "none" project_id had anything to do with the issue. Thanks for the super speedy fix! | 22:33 |
| johnsom | Yeah, confluence of issues there with the policy definition and the unscoped token. I'm adding a test now. | 22:34 |
| rm_work | johnsom: where should the LOG messages from the amphora agent end up in centos? | 22:41 |
| rm_work | not seeing what I'm looking for in journalctl -u amphor-agent, or /var/log/messages, or /var/log/amphora-agent.log | 22:41 |
| johnsom | should and are is two different answers | 22:41 |
| rm_work | which all have different data lol | 22:41 |
| johnsom | If I remember right, gunicorn goes into amphora-agent and the oslo.log goes into the syslog/messages. | 22:42 |
| johnsom | I don't think I fixed that, but I might have. | 22:42 |
| openstackgerrit | Merged openstack/octavia stable/ussuri: Remove haproxy_check_script for UDP-only LBs https://review.opendev.org/751907 | 22:44 |
| rm_work | ^^ whelp that patch doesn't work | 22:46 |
| rm_work | apparently the first few tests we ran were false positives | 22:46 |
| rm_work | it looks like that `vrrp_check_script_update` function doesn't ever run after the initial create | 22:47 |
| rm_work | it LOOKS like it should | 22:47 |
| rm_work | but I think in reality it doesn't | 22:47 |
| rm_work | just haven't figured out why yet | 22:47 |
| rm_work | johnsom: http://paste.openstack.org/show/798179/ | 22:52 |
| rm_work | lvs_udp_check.sh is written 6 seconds after the haproxt one (so, on listener create, not LB create) | 22:52 |
| rm_work | but, the other script is NOT removed? | 22:52 |
| johnsom | Well, if there is an TCP listener, the script would still exist, it just wouldn't have exit 1 in it | 22:53 |
| rm_work | that file is written during `upload_udp_listener_config` | 22:53 |
| rm_work | (for the record, one listener EVER, only UDP) | 22:54 |
| johnsom | Yeah, I'm just saying, you probably don't what to blind nuke it | 22:54 |
| rm_work | it should be nuked given the logic we merged already, that patch that just backported | 22:55 |
| rm_work | but that path apparently isn't ever called post-create | 22:55 |
| rm_work | i think that function needs to be run again in the `upload_udp_listener_config` function | 22:56 |
| rm_work | like.... | 22:57 |
| *** rcernin has joined #openstack-lbaas | 23:02 | |
| *** zzzeek has quit IRC | 23:10 | |
| *** rcernin has quit IRC | 23:11 | |
| *** rcernin has joined #openstack-lbaas | 23:11 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fixes API list handling of unscoped tokens https://review.opendev.org/753169 | 23:12 |
| *** zzzeek has joined #openstack-lbaas | 23:13 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Followup Remove haproxy_check_script for UDP-only https://review.opendev.org/753189 | 23:16 |
| rm_work | sorry, that took longer than expected because I had to fix a test... but, like that ^^ johnsom | 23:16 |
| johnsom | Yeah, that was what I was doing. 4 line change, 310 lines of tests | 23:17 |
| *** zzzeek has quit IRC | 23:17 | |
| *** zzzeek has joined #openstack-lbaas | 23:20 | |
| rm_work | johnsom: tested that and it actually runs now on a listener update | 23:21 |
| rm_work | and fixes VRRP | 23:21 |
| rm_work | the amp-side stuff is a bit of a maze | 23:21 |
| rm_work | really thought that was supposed to run more often... | 23:21 |
| johnsom | Yeah, there could really be some restructuring there. | 23:23 |
| johnsom | The old init systems can come out too. I don't think upstart is going to come back | 23:24 |
| johnsom | rm_work While you are here, how about a quick look at https://review.opendev.org/753169 and https://review.opendev.org/750518 | 23:27 |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/ussuri: Fix amphora failover when VRRP port is missing https://review.opendev.org/753190 | 23:30 |
| rm_work | on the first one, no idea, but i trust that it works because the person you wrote it for confirmed it works :D | 23:43 |
| johnsom | Well, there are tests too | 23:44 |
| rm_work | yeah... | 23:44 |
| rm_work | well, doesn't look like you put in any backdoors or disabled TLS, so +2 | 23:44 |
| rm_work | gonna let tempest sort out the rest | 23:45 |
| johnsom | The other is a release not fix, so trivial | 23:47 |
| rm_work | yep already +A | 23:47 |
| johnsom | However it needs to get in the release | 23:47 |
| *** spatel has joined #openstack-lbaas | 23:49 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/train: Fix amphora failover when VRRP port is missing https://review.opendev.org/753193 | 23:52 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!