| *** yamamoto has joined #openstack-lbaas | 00:49 | |
| *** yamamoto has quit IRC | 00:54 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add client_ca_tls_container_ref to listener API https://review.openstack.org/612267 | 01:51 |
|---|---|---|
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add an option to the Octavia V2 listener API for client cert https://review.openstack.org/612268 | 01:52 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add crl-file option for certification https://review.openstack.org/612269 | 01:52 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add new ssl header into Listener for client certificate https://review.openstack.org/612270 | 01:53 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: L7rule support client certificate cases https://review.openstack.org/612271 | 01:53 |
| *** yamamoto has joined #openstack-lbaas | 03:08 | |
| *** yamamoto has quit IRC | 03:13 | |
| *** yamamoto has joined #openstack-lbaas | 04:29 | |
| *** yamamoto has quit IRC | 04:34 | |
| *** yamamoto has joined #openstack-lbaas | 04:45 | |
| *** yamamoto has quit IRC | 05:49 | |
| *** gcheresh has joined #openstack-lbaas | 07:18 | |
| *** yamamoto has joined #openstack-lbaas | 07:50 | |
| *** yamamoto has quit IRC | 07:55 | |
| *** celebdor has joined #openstack-lbaas | 08:21 | |
| *** celebdor has quit IRC | 08:27 | |
| *** yamamoto has joined #openstack-lbaas | 09:10 | |
| *** yamamoto has quit IRC | 09:14 | |
| *** yamamoto has joined #openstack-lbaas | 09:49 | |
| *** yamamoto has quit IRC | 09:54 | |
| *** yamamoto has joined #openstack-lbaas | 11:02 | |
| *** yamamoto has quit IRC | 11:06 | |
| *** yamamoto has joined #openstack-lbaas | 11:41 | |
| *** yamamoto has quit IRC | 11:45 | |
| *** yamamoto has joined #openstack-lbaas | 11:53 | |
| *** yamamoto has quit IRC | 12:04 | |
| *** yamamoto has joined #openstack-lbaas | 12:07 | |
| *** yamamoto has quit IRC | 12:17 | |
| *** yamamoto has joined #openstack-lbaas | 12:49 | |
| *** yamamoto has quit IRC | 12:53 | |
| *** yamamoto has joined #openstack-lbaas | 13:00 | |
| openstackgerrit | Nir Magnezi proposed openstack/octavia master: Encrypt certs and keys https://review.openstack.org/627064 | 13:41 |
| openstackgerrit | Carlos Goncalves proposed openstack/python-octaviaclient master: Adds loadbalancer amphora configure to OSC https://review.openstack.org/633626 | 15:25 |
| *** gcheresh has quit IRC | 15:48 | |
| openstackgerrit | Merged openstack/python-octaviaclient master: Adds loadbalancer amphora configure to OSC https://review.openstack.org/633626 | 15:51 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add active/standby scenario test https://review.openstack.org/637073 | 16:18 |
| cgoncalves | johnsom, pretty sure https://review.openstack.org/#/c/613685/ broke act-stby when [taskflow]/engine=parallel | 16:20 |
| cgoncalves | engine=serial works fine. tested locally | 16:20 |
| cgoncalves | uhm, maybe it wasn't 613685 as rocky and queens jobs also failed | 16:21 |
| cgoncalves | argh! I meant to say https://review.openstack.org/#/c/604479/ | 16:22 |
| johnsom | I would be surprised if the parallel engine was the issue. | 16:23 |
| johnsom | The VIP patch is more likely | 16:25 |
| cgoncalves | I could only reproduce the issue at the gate with engine=parallel. serial runs fine | 16:27 |
| johnsom | cgoncalves: are you working on that, or should I have a look? | 16:27 |
| johnsom | Which job? Can you give me a link? | 16:27 |
| cgoncalves | I'm not familiar with the flows there. I can try but if you have some time that would be great | 16:28 |
| johnsom | Maybe we are running out of resources with that setting in the gate | 16:28 |
| cgoncalves | https://review.openstack.org/#/c/584681/ | 16:28 |
| cgoncalves | https://review.openstack.org/#/c/637073/ | 16:28 |
| cgoncalves | yours ran successfully until some point in time | 16:28 |
| johnsom | It may need to only be enabled in the multinode jobs. If I can ever figure out what is wrong with neutron there | 16:30 |
| cgoncalves | I don't follow. how come parallel consume more resources? | 16:30 |
| *** yamamoto has quit IRC | 16:30 | |
| cgoncalves | also, how come your act-stby used to pass and now it does not? | 16:31 |
| *** yamamoto has joined #openstack-lbaas | 16:32 | |
| *** yamamoto has quit IRC | 16:32 | |
| cgoncalves | this is not urgent or anything, so don't spend your Sunday morning with this :) | 16:32 |
| *** yamamoto has joined #openstack-lbaas | 16:33 | |
| johnsom | Yeah, today is going to be about backend re-encryption patches. | 16:35 |
| johnsom | With parallel enabled it boots the amps at the same time, so more cpu, more transient ram, more disk io | 16:35 |
| johnsom | I will take a quick look once I get some coffee and get going. | 16:36 |
| *** yamamoto has quit IRC | 16:37 | |
| johnsom | Active standby is important since everyone but OSP uses it. Grin | 16:37 |
| johnsom | I was able to finish the tls client auth stuff yesterday, so starting on the backend reencrypt today. | 16:39 |
| cgoncalves | la-la-la! | 16:40 |
| cgoncalves | I get exact same failure locally as CI. 8GB RAM and nested virtualization enabled | 16:41 |
| cgoncalves | this to say I'm not convinced it is resources | 16:42 |
| cgoncalves | reverted code to refactor vip~1. runs fine with parallel enabled | 16:48 |
| cgoncalves | ah, right, I had forgotten. rocky and queens jobs failed (Permission denied: '/etc/octavia/.ssh/octavia_ssh_key') but due to different reasons than master | 16:49 |
| cgoncalves | so, yeah, 99% sure it was the refactor vip | 16:49 |
| johnsom | And you questioned the half awake PTL.... | 16:57 |
| johnsom | As you should have... lol | 16:58 |
| nmagnezi | johnsom, good morning :D | 17:14 |
| johnsom | It's a morning.... | 17:14 |
| nmagnezi | Well I hope it will improve... :) | 17:15 |
| johnsom | Me too. Hope your day is going well. | 17:16 |
| nmagnezi | So far so good | 17:19 |
| *** yamamoto has joined #openstack-lbaas | 17:20 | |
| *** yamamoto has quit IRC | 17:25 | |
| johnsom | cgoncalves Ok, so this is the problem with the VIP patch: https://github.com/openstack/octavia/blob/master/octavia/network/drivers/neutron/allowed_address_pairs.py#L598 | 17:26 |
| *** pcaruana has quit IRC | 17:26 | |
| johnsom | It's another one of these where the method works across both amps when it should only act on one. | 17:26 |
| johnsom | So either that task is at the wrong point in the flow, or needs to be changed to only act on one amphora. | 17:27 |
| cgoncalves | johnsom, I don't think it is there. the error is that vip port was not found (L596), so before the for loop | 17:27 |
| johnsom | cgoncalves It is there: http://logs.openstack.org/81/584681/18/check/octavia-v2-act-stdby-dsvm-scenario/73b18a5/controller/logs/screen-o-cw.txt.gz?#_Feb_16_11_53_56_657697 | 17:29 |
| cgoncalves | you're right | 17:30 |
| johnsom | Looking at the flow, I think that method probably just needs to have the ability to work on one amp. I would need to do a grep to confirm all of it uses, but that would be my approach to fixing this. | 17:32 |
| johnsom | The only step after is an amp specific call, then it merges back with the other build and reloads the LB details. | 17:32 |
| *** celebdor has joined #openstack-lbaas | 17:37 | |
| johnsom | Eh, ok, so this is going to be tricky, as the output of that is used in other tasks and flows. Namely AmphoraVRRPUpdate | 17:38 |
| johnsom | Yeah, ok. I would probably do a single amp get details in the parallel flow, then after the merge do a combined call for the later items in the flow to use. | 17:42 |
| johnsom | cgoncalves You or me? | 17:43 |
| cgoncalves | johnsom, you | 17:43 |
| johnsom | Ok, could you put a story in? | 17:44 |
| cgoncalves | and a big thank you! | 17:44 |
| cgoncalves | sure | 17:44 |
| cgoncalves | https://storyboard.openstack.org/#!/story/2005080 | 17:48 |
| cgoncalves | storyboard is freaking slow | 17:48 |
| johnsom | Thanks | 17:48 |
| johnsom | Yes it is | 17:48 |
| *** celebdor has quit IRC | 17:52 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix parallel plug vip https://review.openstack.org/638992 | 18:24 |
| johnsom | cgoncalves ^^^^ That should fix this, however I have not tested it or updated the unit/functionals yet. | 18:25 |
| openstackgerrit | Michael Johnson proposed openstack/octavia-tempest-plugin master: Add an active/standby scenario test https://review.openstack.org/584681 | 18:26 |
| johnsom | That should kick the tires and let me know if it is worth finishing. (or someone else if they have time) | 18:26 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix parallel plug vip https://review.openstack.org/638992 | 18:28 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix the loss of access to barbican secrets https://review.openstack.org/637646 | 18:57 |
| *** yamamoto has joined #openstack-lbaas | 19:08 | |
| *** yamamoto has quit IRC | 19:13 | |
| cgoncalves | johnsom, thank you! I'll test it today | 19:24 |
| johnsom | It's broken | 19:24 |
| johnsom | Looking at it now | 19:24 |
| cgoncalves | lol | 19:24 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix parallel plug vip https://review.openstack.org/638992 | 19:52 |
| cgoncalves | johnsom, http://paste.openstack.org/show/745950/ | 20:50 |
| johnsom | Yeah, same in the gate | 20:50 |
| johnsom | Sigh. I will have to load this up in a devstack and poke at it. | 20:50 |
| johnsom | Not sure I'm going to take that on today thoguh | 20:51 |
| cgoncalves | well, it is Sunday. you prolly shouldn't be near a keyboard anyway :) | 20:51 |
| johnsom | True, but need to get these features straightened out. | 20:52 |
| johnsom | I see zuul isn't exactly happy today either. DNS errors, post job permission denied.... | 20:53 |
| cgoncalves | true | 20:53 |
| *** yamamoto has joined #openstack-lbaas | 20:56 | |
| *** yamamoto has quit IRC | 21:01 | |
| johnsom | cgoncalves I moved the priority of the RHEL 8 down. Let me know if we should pull it from the Stein list | 21:35 |
| johnsom | i.e. if that dependent patch isn't going to land | 21:36 |
| cgoncalves | johnsom, ok. yeah, it will depend on that patch being merged | 21:36 |
| cgoncalves | hmm, tempest runs under a different user in CI and thus does not have permissions to read the private key in /etc/octavia/.ssh. is there a hook option in Zuul one can use to run arbitrary commands post-stack/before tempest? | 21:41 |
| cgoncalves | pre-run is pre-stack. run is stack + tempest run, no? | 21:42 |
| johnsom | pre-run is a zuul2 legacy job thing right? | 21:42 |
| cgoncalves | is it? I had imagined not | 21:42 |
| johnsom | Oh, ok, yeah, never mind. | 21:43 |
| johnsom | Ummm, yes, that should work | 21:44 |
| cgoncalves | a play in pre-run? | 21:46 |
| johnsom | yes | 21:46 |
| cgoncalves | so pre-run is after stack? | 21:47 |
| johnsom | before | 21:47 |
| cgoncalves | right. so doesn't help | 21:47 |
| johnsom | Ah, I see your problem. So, hmm, devstack plugin creates that. Tempest isn't running under stack, which makes some sense. | 21:48 |
| cgoncalves | right | 21:48 |
| johnsom | You could have your tempest base load a new SSH key into nova that your test knows | 21:49 |
| cgoncalves | well, one option could be create /etc/octavia/.ssh and set 0755 | 21:49 |
| cgoncalves | ... in pre-run I mean | 21:49 |
| johnsom | Since tempest has an "admin" context it could just overwrite the one devstack loads. | 21:49 |
| cgoncalves | true | 21:50 |
| johnsom | That is what I would do | 21:50 |
| cgoncalves | uhm, actually no. it would work for the purpose of running tempest, yes, but it would mess up the environment | 21:52 |
| cgoncalves | like if you run tempest not in a CI environment | 21:52 |
| johnsom | This is true | 21:53 |
| cgoncalves | only idea I have left it the one I had mentioned: create and 0755 /etc/octavia/.ssh in pre-run | 21:54 |
| johnsom | That isn't awesome either... | 21:54 |
| cgoncalves | it would be a less pleasant workaround applicable only in upstream CI | 21:55 |
| cgoncalves | if anyone has other ideas, please share :) | 21:55 |
| *** rcernin has joined #openstack-lbaas | 22:00 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test https://review.openstack.org/637073 | 22:00 |
| johnsom | That path probably doesn't exist yet | 22:01 |
| johnsom | I think you need directory and the a second set file permissions. Since, ansible... | 22:03 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test https://review.openstack.org/637073 | 22:03 |
| cgoncalves | good point | 22:04 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test https://review.openstack.org/637073 | 22:04 |
| cgoncalves | ^ fixed | 22:05 |
| johnsom | Are you sure that is going to set the perms. I thought it didn't | 22:06 |
| cgoncalves | why do you think it will not? | 22:08 |
| cgoncalves | https://docs.ansible.com/ansible/latest/modules/file_module.html | 22:08 |
| johnsom | If directory, all intermediate subdirectories will be created if they do not exist. Since Ansible 1.7 they will be created with the supplied permissions. | 22:08 |
| johnsom | Oh, right, they will be.... not will not be | 22:09 |
| johnsom | Ugh, context switches | 22:09 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test https://review.openstack.org/637073 | 22:40 |
| *** yamamoto has joined #openstack-lbaas | 22:45 | |
| *** yamamoto has quit IRC | 22:49 | |
| cgoncalves | hmm, devstack plugin sets the private key file with 0600. I missed that | 23:13 |
| cgoncalves | next: OCTAVIA_USE_PREGENERATED_SSH_KEY | 23:13 |
| cgoncalves | but of course! ansible openssh_keypair module would be handy (new in 2.8). CI has Ansible 2.5 | 23:16 |
| *** yamamoto has joined #openstack-lbaas | 23:17 | |
| *** yamamoto has quit IRC | 23:22 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test https://review.openstack.org/637073 | 23:27 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia-tempest-plugin master: Add iptables-based active/standby scenario test https://review.openstack.org/637073 | 23:29 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add crl-file option for certification https://review.openstack.org/612269 | 23:30 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Add new ssl header into Listener for client certificate https://review.openstack.org/612270 | 23:30 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: L7rule support client certificate cases https://review.openstack.org/612271 | 23:31 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!