| rm_work | johnsom: around> | 00:06 |
|---|---|---|
| rm_work | ? | 00:06 |
| *** ivve has quit IRC | 00:16 | |
| rm_work | johnsom: https://review.opendev.org/#/c/696947/ | 00:35 |
| rm_work | ataraday_: ok, all +2'd from me, up to https://review.opendev.org/#/c/668898/ which I do have a concern on, with the passing of "dict_config" bool | 00:39 |
| rm_work | johnsom: ^^ your court now | 00:39 |
| ataraday_ | rm_work, https://review.opendev.org/#/c/668898/ working on resolving comments there - remove "dict_config" and add more constants | 00:41 |
| rm_work | :) | 00:41 |
| rm_work | sorry about ALL the constants stuff lol | 00:41 |
| rm_work | i actually was ignoring "id" i thought | 00:42 |
| rm_work | because ... yeah | 00:42 |
| rm_work | so many | 00:42 |
| johnsom | rm_work o/ | 00:45 |
| johnsom | Still not 100% sure we couldn't use moves. We have used that for exceptions before.... | 00:46 |
| *** sorrison has joined #openstack-lbaas | 01:02 | |
| openstackgerrit | Merged openstack/python-octaviaclient master: Fix API breakage due to new exceptions module https://review.opendev.org/696947 | 01:08 |
| rm_work | johnsom: what he said makes sense | 01:19 |
| rm_work | johnsom: if you can figure it out, i'm down to replace that | 01:20 |
| johnsom | Yeah, decided I had better things to do today | 01:20 |
| rm_work | like merging the first 5 or so of ataraday_'s patches? :D | 01:20 |
| johnsom | Well, reviewing a bunch of stuff. Yeah, I plan to look at those next | 01:22 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Transition amphora flows to dicts https://review.opendev.org/668898 | 01:24 |
| rm_work | ataraday_: thanks so much for doing all those constants! I really didn't expect you to do the other constants that weren't even part of your change T_T | 01:30 |
| rm_work | but really awesome that you did :D | 01:30 |
| ataraday_ | rm_work, when you start adding new constants it is hard to stop :D | 01:31 |
| ataraday_ | althouh I think they all are used in changes I made, so this is all related | 01:33 |
| *** sorrison has quit IRC | 01:38 | |
| *** sorrison has joined #openstack-lbaas | 01:38 | |
| *** sorrison has quit IRC | 01:40 | |
| rm_work | cool, well, first 5 of your changes are +2 from me | 01:50 |
| rm_work | ) | 01:50 |
| rm_work | :) | 01:50 |
| rm_work | let's get this chain shrunk a bit | 01:50 |
| rm_work | none of this is going to break anything until we're entirely done and people switch to the v2 driver, so whatever, we can do final testing+cleanup after all of the chain merges | 01:51 |
| rm_work | or on the very last patch | 01:51 |
| johnsom | It still touches shared code though, so we should be looking at the patches carefully | 01:52 |
| *** sorrison has joined #openstack-lbaas | 01:55 | |
| *** sorrison has quit IRC | 01:58 | |
| rm_work | Yeah, I mean I am watching what's happening, but most of the meaty changes are in v2/ files | 02:01 |
| rm_work | and all the scenarios are still passing, so | 02:01 |
| *** sorrison has joined #openstack-lbaas | 02:02 | |
| johnsom | I am going to have to finish looking at that monster listener patch tomorrow. | 02:03 |
| rm_work | it's not THAT big | 02:04 |
| johnsom | It was my mess in the first place, so clearly it needs *extra* attention. lol | 02:04 |
| rm_work | unless you're talking about a different one than i'm thinking | 02:04 |
| rm_work | +849? meh | 02:04 |
| rm_work | https://review.opendev.org/#/c/660239/ | 02:05 |
| rm_work | +2665, -1072 | 02:05 |
| rm_work | :D | 02:05 |
| rm_work | It's not exactly a competition though, admittedly | 02:06 |
| rm_work | ... and if it was, it'd be more like ... golf rules | 02:06 |
| *** sorrison has quit IRC | 02:15 | |
| *** sorrison has joined #openstack-lbaas | 02:16 | |
| *** gthiemonge has quit IRC | 02:35 | |
| *** gthiemonge has joined #openstack-lbaas | 02:35 | |
| sorrison | rm_work: Had a look into that failure in my tempest job to do with missing yaml | 02:42 |
| sorrison | It also fails in your job too with the same error, it's just non-voting. | 02:43 |
| sorrison | octavia-v2-act-stdby-dsvm-scenario-two-node fails but the py2 one passes | 02:44 |
| *** zaneb has left #openstack-lbaas | 02:59 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Convert Lb flows to use provider dicts https://review.opendev.org/671725 | 03:09 |
| openstackgerrit | Merged openstack/octavia master: Use retry for AmphoraComputeConnectivityWait https://review.opendev.org/662791 | 03:25 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Convert health_monitor flows to use provider models https://review.opendev.org/679779 | 03:28 |
| *** sorrison has quit IRC | 04:04 | |
| *** sorrison has joined #openstack-lbaas | 04:10 | |
| *** sorrison has quit IRC | 04:15 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Transition l7policy flows to dicts https://review.opendev.org/665977 | 04:19 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Transition l7rule flows to dicts https://review.opendev.org/668173 | 04:38 |
| *** sorrison has joined #openstack-lbaas | 04:45 | |
| *** sorrison has quit IRC | 04:52 | |
| *** sorrison has joined #openstack-lbaas | 04:57 | |
| *** tkajinam has quit IRC | 05:02 | |
| openstackgerrit | Sam Morrison proposed openstack/octavia-tempest-plugin master: Add new scenario test to create LB in specific AZ https://review.opendev.org/695349 | 05:18 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Jobboard based controller https://review.opendev.org/647406 | 05:19 |
| *** tkajinam has joined #openstack-lbaas | 05:33 | |
| *** sapd1_x has joined #openstack-lbaas | 06:23 | |
| *** gcheresh has joined #openstack-lbaas | 06:26 | |
| *** ataraday_ has quit IRC | 06:39 | |
| *** sorrison has quit IRC | 07:14 | |
| *** tkajinam_ has joined #openstack-lbaas | 07:18 | |
| *** tkajinam_ has quit IRC | 07:19 | |
| *** tkajinam_ has joined #openstack-lbaas | 07:20 | |
| *** tkajinam has quit IRC | 07:21 | |
| *** sapd1_x has quit IRC | 07:23 | |
| *** sorrison has joined #openstack-lbaas | 07:25 | |
| *** CobHead has quit IRC | 07:50 | |
| *** trident has quit IRC | 07:51 | |
| *** trident has joined #openstack-lbaas | 07:51 | |
| *** sorrison has quit IRC | 07:53 | |
| *** sorrison has joined #openstack-lbaas | 07:54 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 07:56 | |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional job https://review.opendev.org/696967 | 08:16 |
| *** tesseract has joined #openstack-lbaas | 08:16 | |
| *** sorrison has quit IRC | 08:26 | |
| *** sorrison has joined #openstack-lbaas | 08:27 | |
| *** sorrison has quit IRC | 08:42 | |
| *** rpittau|afk is now known as rpittau | 08:43 | |
| *** sorrison has joined #openstack-lbaas | 08:43 | |
| *** sorrison has quit IRC | 08:48 | |
| *** ivve has joined #openstack-lbaas | 08:51 | |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add job for executing functional tests on CentOS 7 https://review.opendev.org/697032 | 09:04 |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Use VM as client in traffic operation scenario tests https://review.opendev.org/687562 | 09:05 |
| *** gcheresh has quit IRC | 09:08 | |
| *** gcheresh has joined #openstack-lbaas | 09:08 | |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional job https://review.opendev.org/696967 | 09:14 |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional job https://review.opendev.org/696967 | 09:15 |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs https://review.opendev.org/696967 | 09:15 |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs https://review.opendev.org/696967 | 09:18 |
| *** tkajinam_ has quit IRC | 09:20 | |
| *** sorrison has joined #openstack-lbaas | 09:23 | |
| *** sorrison has quit IRC | 09:28 | |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs https://review.opendev.org/696967 | 09:40 |
| *** dasp has quit IRC | 09:42 | |
| *** dasp has joined #openstack-lbaas | 09:43 | |
| *** abaindur has quit IRC | 09:47 | |
| *** sorrison has joined #openstack-lbaas | 09:58 | |
| *** sorrison has quit IRC | 10:04 | |
| *** rcernin has quit IRC | 10:06 | |
| *** pcaruana has joined #openstack-lbaas | 10:08 | |
| *** dmellado has quit IRC | 10:33 | |
| *** dmellado has joined #openstack-lbaas | 10:35 | |
| *** sorrison has joined #openstack-lbaas | 10:40 | |
| *** sorrison has quit IRC | 10:48 | |
| *** sorrison has joined #openstack-lbaas | 10:49 | |
| *** sorrison has quit IRC | 10:53 | |
| *** sorrison has joined #openstack-lbaas | 10:54 | |
| *** sorrison has quit IRC | 10:59 | |
| *** sorrison has joined #openstack-lbaas | 11:00 | |
| *** sorrison has quit IRC | 11:05 | |
| *** sorrison has joined #openstack-lbaas | 11:09 | |
| *** sorrison has quit IRC | 11:13 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/octavia-tempest-plugin master: Discover used LB algorithm used for test https://review.opendev.org/672264 | 11:16 |
| *** rpittau is now known as rpittau|bbl | 11:18 | |
| openstackgerrit | Maciej Józefczyk proposed openstack/octavia-tempest-plugin master: Discover LB algorithm used for test https://review.opendev.org/672264 | 11:20 |
| *** sorrison has joined #openstack-lbaas | 11:26 | |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs https://review.opendev.org/696967 | 11:27 |
| *** sorrison has quit IRC | 11:34 | |
| *** vishalmanchanda has joined #openstack-lbaas | 11:34 | |
| *** sorrison has joined #openstack-lbaas | 11:38 | |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs https://review.opendev.org/696967 | 11:39 |
| maciejjozefczyk | Hey! cgoncalves ltomasbo johnsom :) I'm starting moving ovn provider driver code to separate repository. What do you think about naming it as ' octavia-ovn-provider-driver'? Or ' octavia-ovn-provider'? | 11:40 |
| *** trident has quit IRC | 11:40 | |
| *** sorrison has quit IRC | 11:42 | |
| *** trident has joined #openstack-lbaas | 11:43 | |
| openstackgerrit | Federico Ressi proposed openstack/python-octaviaclient master: Add DevStack CI functional jobs https://review.opendev.org/696967 | 11:43 |
| *** sorrison has joined #openstack-lbaas | 11:44 | |
| *** sorrison has quit IRC | 11:49 | |
| *** sorrison has joined #openstack-lbaas | 11:52 | |
| *** gcheresh has quit IRC | 11:52 | |
| cgoncalves | maciejjozefczyk, I think either are good options. | 11:54 |
| *** sorrison has quit IRC | 11:56 | |
| maciejjozefczyk | cgoncalves thanks! | 11:59 |
| *** sorrison has joined #openstack-lbaas | 12:00 | |
| *** sorrison has quit IRC | 12:04 | |
| *** gcheresh has joined #openstack-lbaas | 12:04 | |
| *** sorrison has joined #openstack-lbaas | 12:13 | |
| *** sorrison has quit IRC | 12:18 | |
| *** sorrison has joined #openstack-lbaas | 12:24 | |
| *** sorrison has quit IRC | 12:28 | |
| *** servagem has joined #openstack-lbaas | 12:35 | |
| *** sorrison has joined #openstack-lbaas | 12:37 | |
| *** sorrison has quit IRC | 12:42 | |
| *** sorrison has joined #openstack-lbaas | 12:54 | |
| *** sorrison has quit IRC | 12:58 | |
| maciejjozefczyk | cgoncalves, https://review.opendev.org/#/c/697076/1 :) | 13:08 |
| *** dasp has quit IRC | 13:12 | |
| *** dasp has joined #openstack-lbaas | 13:12 | |
| *** sorrison has joined #openstack-lbaas | 13:13 | |
| dulek | Hi! We see Kuryr gates are red, timeouting waiting for the LB to get created. | 13:17 |
| dulek | I'd blame one of 3 last commits: https://opendev.org/openstack/octavia/commits/branch/master | 13:17 |
| dulek | We do run with Python 3 for a while now, so shouldn't be that. | 13:18 |
| dulek | A link to a failing run: https://zuul.opendev.org/t/openstack/build/524e34982ce14415864bcf716926ff0b | 13:20 |
| dulek | I don't see anything interesting in logs though. | 13:20 |
| *** sorrison has quit IRC | 13:23 | |
| openstackgerrit | Michał Dulko proposed openstack/octavia master: DNM: Revert "Use 2048-bits keys for devstack certificates" https://review.opendev.org/697084 | 13:25 |
| openstackgerrit | Michał Dulko proposed openstack/octavia master: DNM: Revert "Use retry for AmphoraComputeConnectivityWait" https://review.opendev.org/697086 | 13:27 |
| dulek | I'll try to figure out which one caused it using the above. ^ | 13:27 |
| *** sorrison has joined #openstack-lbaas | 13:35 | |
| *** rpittau|bbl is now known as rpittau | 13:46 | |
| *** sorrison has quit IRC | 13:46 | |
| *** sorrison has joined #openstack-lbaas | 13:53 | |
| *** sorrison has quit IRC | 13:58 | |
| rm_work | o/ | 14:01 |
| *** sorrison has joined #openstack-lbaas | 14:08 | |
| *** sorrison has quit IRC | 14:12 | |
| *** strobert1 has quit IRC | 14:14 | |
| *** strobert1 has joined #openstack-lbaas | 14:17 | |
| *** sorrison has joined #openstack-lbaas | 14:23 | |
| *** trident has quit IRC | 14:24 | |
| *** trident has joined #openstack-lbaas | 14:25 | |
| *** sorrison has quit IRC | 14:30 | |
| *** pcaruana has quit IRC | 14:33 | |
| *** TrevorV has joined #openstack-lbaas | 14:35 | |
| *** sorrison has joined #openstack-lbaas | 14:44 | |
| *** sorrison has quit IRC | 14:48 | |
| dulek | Okay, so it seems none of the above caused it. Any other clues? | 14:58 |
| *** pcaruana has joined #openstack-lbaas | 15:04 | |
| *** sorrison has joined #openstack-lbaas | 15:04 | |
| cgoncalves | why do you think it is octavia and not nova or neutron, for example? | 15:06 |
| cgoncalves | ah, you pasted a link to the job. looking | 15:06 |
| *** sorrison has quit IRC | 15:09 | |
| dulek | cgoncalves: You can stop, I'll try my theory first. | 15:11 |
| dulek | To not waste your time. | 15:11 |
| dulek | As it might be related to us actually. | 15:11 |
| dulek | You've dropped Python 2, but I see we're using CentOS 7 amphora. | 15:12 |
| dulek | And as the build happened this morning: https://tarballs.openstack.org/octavia/test-images/ | 15:12 |
| cgoncalves | dulek, ok. I don't see anything relevant in octavia logs or nova | 15:12 |
| dulek | It's possible CentOS 7 version from link above isn't working at all anymore. | 15:12 |
| dulek | So I'm going to try with Ubuntu amp. | 15:13 |
| cgoncalves | nit: kuryr-kubernetes-tempest-containerized is non-voting. remove it from your gate queue. | 15:13 |
| *** sorrison has joined #openstack-lbaas | 15:14 | |
| dulek | cgoncalves: Thanks for spotting this! | 15:16 |
| *** sorrison has quit IRC | 15:20 | |
| dulek | cgoncalves: octavia/cmd/agent.py is what runs on the amp as the amp server? | 15:23 |
| *** sorrison has joined #openstack-lbaas | 15:23 | |
| cgoncalves | dulek, yes | 15:25 |
| dulek | Alright, seems like my hypothesis is correct. | 15:25 |
| cgoncalves | FWIW, http://zuul.openstack.org/builds?job_name=octavia-v2-dsvm-py2-scenario-centos-7 | 15:26 |
| cgoncalves | dulek, which is? | 15:26 |
| dulek | cgoncalves: The agent is no longer able to start on CentOS 7 Amp due to lack of Python 3. | 15:28 |
| *** sorrison has quit IRC | 15:28 | |
| dulek | cgoncalves: It's run against Python 2, so for example gunicorn is not installed there: https://opendev.org/openstack/octavia/src/branch/master/requirements.txt#L58 | 15:28 |
| cgoncalves | oh, shoot. yeah, we killed py2 yesterday | 15:33 |
| dulek | https://review.opendev.org/#/c/697114/ - let's see how this goes. | 15:34 |
| dulek | cgoncalves: Do you know where I can find jobs producing those: https://tarballs.openstack.org/octavia/test-images/ ? | 15:34 |
| dulek | Is it possible Ubuntu image there still uses Python 2 and won't work too? | 15:35 |
| cgoncalves | although you're using image from RDO which is expected to install the amphora-agent RPM | 15:35 |
| dulek | Those are from RDO? | 15:35 |
| cgoncalves | Ubuntu is expected to use py3 | 15:35 |
| cgoncalves | bleh, I need coffee. no, they are not. I had the impression you had linked us before to the RDO images | 15:35 |
| cgoncalves | but it was to https://tarballs.openstack.org/octavia/test-images/ | 15:36 |
| *** tesseract has quit IRC | 15:37 | |
| cgoncalves | haleyb, hey. were you planning to update the centos 7 job to use py3? | 15:39 |
| cgoncalves | unless we switch it to py3, we should delete the periodic job publish-openstack-octavia-amphora-image-centos7 | 15:40 |
| *** sorrison has joined #openstack-lbaas | 15:40 | |
| haleyb | cgoncalves: yes, was working on that today, should add a story | 15:46 |
| *** tesseract has joined #openstack-lbaas | 15:48 | |
| *** TrevorV has quit IRC | 15:50 | |
| *** sorrison has quit IRC | 15:50 | |
| cgoncalves | cool! | 15:55 |
| rm_work | ah so we'll have a py3 cent7? :D | 15:56 |
| cgoncalves | rm_work, u no laike? :) | 15:57 |
| *** sorrison has joined #openstack-lbaas | 15:57 | |
| cgoncalves | centos7-py3 controller + centos8-py3 amp. hmmm :) | 15:57 |
| *** sorrison has quit IRC | 16:02 | |
| rm_work | it'll be cool :D | 16:02 |
| *** sorrison has joined #openstack-lbaas | 16:09 | |
| *** TrevorV has joined #openstack-lbaas | 16:12 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: WIP: Add support to CentOS 8 amphora images https://review.opendev.org/697128 | 16:13 |
| *** sorrison has quit IRC | 16:14 | |
| *** ivve has quit IRC | 16:15 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: WIP: Add support to CentOS 8 amphora images https://review.opendev.org/697128 | 16:18 |
| *** jamesdenton has quit IRC | 16:25 | |
| cgoncalves | active-standby centos 8 based load balancer up and running ;) | 16:25 |
| *** sorrison has joined #openstack-lbaas | 16:29 | |
| *** sorrison has quit IRC | 16:37 | |
| *** ataraday_ has joined #openstack-lbaas | 16:42 | |
| rm_work | sweet | 16:42 |
| rm_work | cgoncalves: so you gonna review the jobboard stuff today? | 16:43 |
| rm_work | i've been through like the first 6 patches and commented and ann fixed stuff yesterday, so i'm +2 now on a bunch | 16:43 |
| rm_work | we could really get this stuff moving XD | 16:43 |
| *** sorrison has joined #openstack-lbaas | 16:44 | |
| cgoncalves | rm_work, I started reviewing the AZ admin + client patches today | 16:47 |
| *** sorrison has quit IRC | 16:48 | |
| johnsom | I plan to continue no the jobboard reviews today | 16:52 |
| rm_work | cgoncalves: ok | 16:52 |
| rm_work | cgoncalves: i took your words to heart and started trying to get the jobboard stuff moving finally | 16:52 |
| rm_work | so, up to you | 16:53 |
| rm_work | but a big part of that chain is +2, and i'm going through more today | 16:53 |
| rm_work | granted, it's all just logical code review, until 100% of it is in, none of it is really going to work, I think | 16:53 |
| rm_work | so I haven't bothered testing it in reality | 16:53 |
| rm_work | so long as all our v1 stuff continues to work and pass tests, i don't see a problem merging these pieces, up to the very last patch where we'll want stuff to actually be provably working | 16:54 |
| *** sorrison has joined #openstack-lbaas | 16:54 | |
| cgoncalves | rm_work, let me know what works best, merging AZ or jobboard first | 16:57 |
| rm_work | well, i just had to go back and -1 a bunch of jobboard stuff actually, though not for anything major... just an extra pop() that i'm not understanding | 17:00 |
| rm_work | but yes, I do really need AZ soon (yesterday) | 17:00 |
| rm_work | so if you can get that in, great | 17:00 |
| johnsom | I think there is a pressing timeline of need for AZ, so maybe splitting up the reviews across cores is a good approach. I have already done the AZ reviews and started the jobboard work. | 17:00 |
| ataraday_ | rm_work, https://review.opendev.org/#/c/657842/31/octavia/api/drivers/amphora_driver/v2/driver.py - we need to pop ID as we have in original_member (old object dict) and it will be conflicting with the on in update dict | 17:01 |
| johnsom | rm_work I think we can cut the octavia-lib AZ needs right? | 17:01 |
| ataraday_ | I put comment on healthmonitor change about that | 17:01 |
| *** sorrison has quit IRC | 17:02 | |
| haleyb | johnsom: i'll have to find a place for the ovn driver, guess it can just live on it's own under the neutron umbrella | 17:02 |
| *** tesseract has quit IRC | 17:03 | |
| johnsom | haleyb Yeah, I wanted to make it clear that the Octavia team hasn't agreed to adopt that third party driver. You can still give Octavia core access in gerrit though, if you want. | 17:03 |
| haleyb | johnsom: yes, it's an odd bird, and I know the company that will do most of the early work ;-) i'll figure something out regarding the name/location | 17:06 |
| *** sorrison has joined #openstack-lbaas | 17:10 | |
| *** rpittau is now known as rpittau|afk | 17:12 | |
| *** sorrison has quit IRC | 17:15 | |
| rm_work | ataraday_: ah ok, so actually that really is intended | 17:16 |
| rm_work | johnsom: what? you mean cut a release? | 17:17 |
| rm_work | ataraday_: but is that true for the rest? because all of them do it -- pool / hm / l7policy / etc | 17:17 |
| johnsom | rm_work Don't you need octavia-lib released with the AZ changes? | 17:17 |
| rm_work | i don't think that ID should conflict? | 17:17 |
| rm_work | johnsom: so, we're waiting on one more lib merge | 17:17 |
| rm_work | johnsom: needs the first API change to merge before it can (it's a dep) | 17:18 |
| rm_work | johnsom: THEN we could cut it, yes | 17:18 |
| rm_work | that's part of why i'm trying to get the first couple patches moving | 17:18 |
| rm_work | because i know there's a few stages involved | 17:18 |
| cgoncalves | rm_work, AZ admin API is in the gate now | 17:18 |
| rm_work | oh ok cool :D just noticed | 17:18 |
| johnsom | Ok, I was just looking at the octavia-lib list which seemed ready | 17:19 |
| rm_work | tempest tests also? | 17:19 |
| rm_work | johnsom: very shortly | 17:19 |
| rm_work | johnsom: once https://review.opendev.org/#/c/694707/ lands | 17:19 |
| rm_work | cgoncalves: https://review.opendev.org/#/c/695291/ | 17:20 |
| rm_work | that goes with the change you +A'd | 17:20 |
| cgoncalves | yeah, skimming it now | 17:20 |
| ataraday_ | rm_work, I got errors with update command about conflicting passed ID, I can recheck if I don't miss somethings as it was couple of months ago :) | 17:20 |
| rm_work | ataraday_: hmmmm ok, yeah so that's kinda what i was getting at ... maybe i am missing something, and it really does have to be gone | 17:21 |
| *** sorrison has joined #openstack-lbaas | 17:21 | |
| rm_work | ataraday_: is it easy to test? | 17:21 |
| rm_work | ataraday_: i noticed there was at least one patch (listeners?) that DIDN'T seem to do it | 17:21 |
| ataraday_ | yeah, I will recheck, I've got devstack reinstalling so in an hour or something... | 17:23 |
| rm_work | kk | 17:24 |
| rm_work | i mean... also maybe it just doesn't matter <_< | 17:24 |
| rm_work | if it's not used | 17:24 |
| rm_work | I thought it might be used | 17:24 |
| rm_work | but i'm not sure | 17:24 |
| rm_work | johnsom: so yeah, once that patch merges, then the lib patch will go to the gate, and then once THAT merges, we can cut a lib release | 17:28 |
| *** sorrison has quit IRC | 17:28 | |
| *** sorrison has joined #openstack-lbaas | 17:30 | |
| *** sorrison has quit IRC | 17:36 | |
| *** sorrison has joined #openstack-lbaas | 17:37 | |
| *** gcheresh has quit IRC | 17:41 | |
| *** sorrison has quit IRC | 17:41 | |
| *** sorrison has joined #openstack-lbaas | 17:52 | |
| *** sorrison has quit IRC | 17:56 | |
| ataraday_ | rm_work, I checked update loadbalancer without poping ID and got error http://paste.openstack.org/show/787057/ not conflicting, but it expectes ID not loadbalancer_id, I think it will be the same for rest of resources - will check listener now. | 17:58 |
| rm_work | ahhhh hold on | 17:59 |
| rm_work | YES DUH sorry | 17:59 |
| rm_work | I get it | 17:59 |
| rm_work | though ... i still have one comment | 17:59 |
| rm_work | can you guess what it is? :D | 18:00 |
| ataraday_ | how it is working now? | 18:01 |
| rm_work | no | 18:01 |
| rm_work | "constant?" | 18:01 |
| rm_work | ^_^ | 18:01 |
| rm_work | lol | 18:01 |
| ataraday_ | okay >_< | 18:02 |
| rm_work | hahaha no worries, we can do a pass at the end | 18:03 |
| rm_work | seriously it's fine | 18:03 |
| rm_work | i am gonna do like a global find, on all the common ones | 18:03 |
| rm_work | and fix it all | 18:03 |
| rm_work | ataraday_: switching to +2s | 18:05 |
| rm_work | don't touch | 18:05 |
| rm_work | lol | 18:05 |
| rm_work | you | 18:07 |
| rm_work | *you'll need to rebase in a sec i think because of the merging AZ patch? | 18:07 |
| rm_work | though I can also do that for you (since I said I would) :D | 18:07 |
| ataraday_ | I can rebase no worries, if johnsom will have some comments I may fix constants as well :) | 18:08 |
| johnsom | ID = 'id' lol, I have resisted this urge for a while. | 18:09 |
| *** sorrison has joined #openstack-lbaas | 18:11 | |
| rm_work | hahah yeah | 18:14 |
| rm_work | i specifically avoided commenting on those | 18:14 |
| rm_work | but ataraday_ did it all anyway XD | 18:14 |
| *** sorrison has quit IRC | 18:16 | |
| johnsom | It probably does save a small amount of RAM, lol | 18:16 |
| johnsom | Huge shout-out for all of the constants work BTW! | 18:17 |
| *** maciejjozefczyk has quit IRC | 18:18 | |
| *** vishalmanchanda has quit IRC | 18:21 | |
| *** vishalmanchanda has joined #openstack-lbaas | 18:21 | |
| *** armax has quit IRC | 18:22 | |
| johnsom | ataraday_ Thoughts on this failure in the listener patch? https://zuul.opendev.org/t/openstack/build/01fe86a687a74a38b8aec472e7a8cfa1/log/controller/logs/screen-o-cw.txt.gz?severity=4 | 18:25 |
| ataraday_ | johnsom, this is how retry works, the expection is thrown and then task is rertied | 18:26 |
| johnsom | Hmm, I thought we had a better way. These ERROR logs are going to freak operators out. Our current WARNING is bad enough with people thinking it is failing. | 18:27 |
| *** sorrison has joined #openstack-lbaas | 18:28 | |
| ataraday_ | johnsom, if we caught the expection the task won't retry | 18:29 |
| johnsom | Yeah, I follow. Just need to think about this a bit. | 18:29 |
| *** sorrison has quit IRC | 18:32 | |
| ataraday_ | Sure, I got only one idea - drop redis support - use only zookeper - no reties will be needed :D But this is serious reduction | 18:33 |
| ataraday_ | and using zookeper I think is not an option as default for upstream gates I guess | 18:35 |
| johnsom | Well, I think there are two paths in addition, maybe more. 1. Figure out how to mask/block that traceback for these retry exceptions. 2. Switch to using the taskflow notify capability and retry in the code again. | 18:35 |
| *** elico has joined #openstack-lbaas | 18:36 | |
| johnsom | I am rechecking as one of the listener tests failed. It could be just a slow RAX host, but I would like to see it pass. | 18:36 |
| *** abaindur has joined #openstack-lbaas | 18:37 | |
| *** sorrison has joined #openstack-lbaas | 18:40 | |
| ataraday_ | yeah, this is the one I wrote you about - see it failing from time to time | 18:41 |
| johnsom | Ok | 18:41 |
| *** abaindur has quit IRC | 18:42 | |
| *** sorrison has quit IRC | 18:44 | |
| rm_work | yeah it seems to fail in a number of the runs | 18:46 |
| rm_work | various ones do :( | 18:46 |
| rm_work | we'll have to figure it out, but IMO we should figure it out in the final patch in the chain | 18:47 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: WIP: Add support to CentOS 8 amphora images https://review.opendev.org/697128 | 18:47 |
| rm_work | my vote is that we merge the first 8 or so now and then really dive in on the last patch :) | 18:47 |
| *** ccamposr has joined #openstack-lbaas | 18:47 | |
| *** sorrison has joined #openstack-lbaas | 18:49 | |
| *** ccamposr__ has quit IRC | 18:50 | |
| rm_work | hmm in a little bit i'm gonna be grabbing lunch and a siesta, but then i'll be back around, so if those patches get rebased, I will +2 again today :) | 18:51 |
| johnsom | Ok, so for the ERROR logging, I think I will either open stories or create an etherpad to track "known issues". | 18:51 |
| *** abaindur has joined #openstack-lbaas | 18:51 | |
| rm_work | yeah maybe we can catch, mangle the exception, and reraise? | 18:52 |
| johnsom | Do we have a preference? | 18:52 |
| *** abaindur has quit IRC | 18:52 | |
| rm_work | i prefer etherpad | 18:52 |
| rm_work | less overhead | 18:52 |
| *** abaindur has joined #openstack-lbaas | 18:52 | |
| johnsom | Well, it's either logging changes or we switch back to in-task retry with the "update_progress" called to keep the job board happy. Those are the two top-of-head thoughts | 18:53 |
| johnsom | https://docs.openstack.org/taskflow/latest/user/atoms.html#taskflow.task.Task.update_progress | 18:53 |
| rm_work | eh | 18:53 |
| johnsom | I know we talked through this once, but I don't remember the whole decision tree we went down. | 18:53 |
| rm_work | i think we could just catch, update the stacktrace to be something more palatable, and reraise? | 18:53 |
| johnsom | https://etherpad.openstack.org/p/octavia-worker-v2-issue-tracker | 18:54 |
| *** sorrison has quit IRC | 18:54 | |
| ataraday_ | I think I posted comment with description of all think in the story | 18:56 |
| johnsom | Ah, ok | 18:56 |
| ataraday_ | Hm, no - it was just comment on https://review.opendev.org/#/c/647406/9 on the 9th patch set and then we discuss this on meeting | 18:59 |
| *** gmann is now known as gmann_afk | 19:00 | |
| *** jamesdenton has joined #openstack-lbaas | 19:00 | |
| *** gcheresh has joined #openstack-lbaas | 19:01 | |
| *** armax has joined #openstack-lbaas | 19:04 | |
| johnsom | I tried to capture that in the etherpad. Not sure etherpad is going to work well with all the details we need to capture, we can always switch later. | 19:10 |
| openstackgerrit | Merged openstack/octavia master: Availability Zone admin API https://review.opendev.org/693765 | 19:14 |
| *** gcheresh has quit IRC | 19:19 | |
| *** gcheresh has joined #openstack-lbaas | 19:23 | |
| *** ivve has joined #openstack-lbaas | 19:28 | |
| *** ramishra has quit IRC | 19:45 | |
| *** gcheresh has quit IRC | 20:21 | |
| *** gmann_afk is now known as gmann | 20:25 | |
| *** vesper11 has quit IRC | 20:49 | |
| *** vesper11 has joined #openstack-lbaas | 20:51 | |
| *** ccamposr__ has joined #openstack-lbaas | 20:58 | |
| *** sorrison has joined #openstack-lbaas | 20:59 | |
| *** ccamposr has quit IRC | 21:01 | |
| *** sorrison has quit IRC | 21:04 | |
| *** abaindur has quit IRC | 21:08 | |
| *** TrevorV has quit IRC | 21:10 | |
| *** gcheresh has joined #openstack-lbaas | 21:20 | |
| *** sorrison has joined #openstack-lbaas | 21:27 | |
| *** servagem has quit IRC | 21:33 | |
| *** gcheresh has quit IRC | 21:39 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Convert listener flows to use provider models https://review.opendev.org/660236 | 21:56 |
| johnsom | rebased after the AZ API patch | 21:56 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Convert pool flows to use dicts https://review.opendev.org/665381 | 22:00 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Transition member flows to use dicts https://review.opendev.org/657842 | 22:01 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Transition amphora flows to dicts https://review.opendev.org/668898 | 22:01 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Convert Lb flows to use provider dicts https://review.opendev.org/671725 | 22:01 |
| johnsom | rebasing a stack now that the az api patch merged. | 22:01 |
| johnsom | Hmm, wondering if we don't have a problem in the jobboard chain, I see the pool patch also failed the v2 job with the same listener CRUD timeout. However this one was on inap.... | 22:05 |
| *** pcaruana has quit IRC | 22:06 | |
| sorrison | johnson: I've also been looking into gate failures for octavia-v2-act-stdby-dsvm-scenario-two-node due to python-yaml not being installed. octavia-v2-act-stdby-dsvm-py2-scenario-two-node works | 22:06 |
| johnsom | Well, the rebase will fire off another run to test it | 22:06 |
| ataraday_ | johnsom, thanks for rebase! | 22:07 |
| johnsom | sorrison Thank you for looking at the two-node jobs! They need help. | 22:07 |
| johnsom | sorrison However, I am looking at the worker v2 jobs: octavia-v2-dsvm-scenario-amphora-v2 This is a special test for the jobboard work. | 22:07 |
| johnsom | Only slightly confusing lol | 22:07 |
| johnsom | ataraday_ I'm going to go through the pool patch now | 22:08 |
| sorrison | johnson: yeah I'm still trying to figure out how the DIB all fits together. There is something that is run in the py2 version that isn't run in the py3 one | 22:08 |
| johnsom | sorrison I am the local (to Octavia) DIB expert, so let me know if I can help with something | 22:09 |
| sorrison | oooo | 22:09 |
| sorrison | I'll ping you when I get in deeper | 22:10 |
| johnsom | +1 | 22:10 |
| johnsom | Happy to setup a screen sharing session too if it would help you. Just let me know | 22:10 |
| sorrison | thanks. I'll have another stab at it first | 22:11 |
| johnsom | Last I left the two-node jobs, there was some intermittent network connectivity issue between the nodes. Especially with IPv6. | 22:15 |
| * haleyb has not completed his two-node job triage | 22:17 | |
| *** armax has quit IRC | 22:31 | |
| *** armax has joined #openstack-lbaas | 22:32 | |
| lxkong | sorrison, rm_work, i know you are working on AZ support for LB, but Is it possible to create amphorae for active/passive LB in different AZs? | 22:33 |
| lxkong | i mean, in the current feature design | 22:34 |
| sorrison | lxkong: Not in the initial work but I think that could be added | 22:35 |
| johnsom | lxkong No, currently active/passive only does nova anti-affinity. There is a patch that does a cross AZ thing, but it requires a special network setup for the cloud and other special patches. | 22:35 |
| johnsom | lxkong Do you have shared L2 networks across AZs? | 22:35 |
| *** elico has quit IRC | 22:36 | |
| lxkong | johnsom: We have a private cloud customer asking for this feature. They have three AZs deployed in a single region, and they configured very fast L2 network between AZs. | 22:37 |
| johnsom | lxkong Ok, so you do have L2 across AZs such that the VIP could move across AZs. Good to know. That makes the problem space much better. I still wish nova would allow us to do anti-affinity with AZs using server groups, but they don't have that feature. | 22:38 |
| lxkong | johnsom: ok, so is it easy to implement in octavia side? | 22:39 |
| johnsom | lxkong Ha, no, not easy..... Basically it's adding nova scheduling to Octavia so it can "decide" the AZs. So, not easy. | 22:39 |
| cgoncalves | johnsom, is there a launchpad for nova? | 22:39 |
| johnsom | lxkong I think rm_work has a patch that mostly gets you there, it would just need to be fixed to not require the special networking setup. | 22:40 |
| lxkong | johnsom: cool, i willl find that patch and take a look | 22:40 |
| sorrison | cgoncalves: https://bugs.launchpad.net/nova/ | 22:41 |
| johnsom | lxkong rm_work will probably be back on in about three hours if you want to chat with him about it | 22:41 |
| lxkong | johnsom: that would be great :-) | 22:41 |
| cgoncalves | sorrison, ah, lol, sorry. I meant to ask if there's a bug # open in nova requesting AZ anti-affinity | 22:41 |
| johnsom | cgoncalves I don't remember if there is or not | 22:41 |
| sorrison | na I haven't heard that request in the nova circles | 22:42 |
| johnsom | Yeah, the api-ref for server groups only lists "by host", no mention of AZ support: https://docs.openstack.org/api-ref/compute/#server-groups-os-server-groups | 22:44 |
| sorrison | johnsom: a bit stuck with this dib issue if you have a sec | 22:49 |
| sorrison | I'm wondering if its an ordering issue | 22:49 |
| sorrison | python3 broken: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8cd/695291/12/check/octavia-v2-act-stdby-dsvm-scenario-two-node/8cdbc7d/controller/logs/dib-build/amphora-x64-haproxy.qcow2_log.txt.gz | 22:49 |
| johnsom | sorrison Sure, I can make time. | 22:49 |
| sorrison | python2 working: https://b649c19a68edc56feea8-49130948639ed40b079dd8450de896f5.ssl.cf2.rackcdn.com/695291/12/check/octavia-v2-act-stdby-dsvm-py2-scenario-two-node/c4d7087/controller/logs/dib-build/amphora-x64-haproxy.qcow2_log.txt.gz | 22:49 |
| sorrison | the svc-map needs python-yaml but it doesn't install it in the py3 job | 22:50 |
| johnsom | sorrison Hmm, well, we have PyYAML in our requirements.txt file: https://github.com/openstack/octavia/blob/master/diskimage-create/requirements.txt | 22:51 |
| *** elico has joined #openstack-lbaas | 22:51 | |
| *** elico has quit IRC | 22:52 | |
| *** elico has joined #openstack-lbaas | 22:52 | |
| openstackgerrit | Brian Haley proposed openstack/octavia-tempest-plugin master: Create python3 Centos-7 scenario job https://review.opendev.org/697198 | 22:53 |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Add back the Centos-7 scenario job https://review.opendev.org/697199 | 22:53 |
| *** tkajinam has joined #openstack-lbaas | 22:56 | |
| *** rcernin has joined #openstack-lbaas | 22:57 | |
| johnsom | sorrison Ok, yes, this is a known issue with devstack. This task in the devstack ansible installs it: https://zuul.opendev.org/t/openstack/build/8cdbc7d3ff584597a09a8753486406d7/log/job-output.txt#2646 | 22:58 |
| johnsom | sorrison But it only installs it in python2, not py3 | 22:58 |
| johnsom | The real issue is probably that our gates are not installing that diskimage-create/requirements.txt list of packages | 22:59 |
| johnsom | sorrison https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L57 | 23:02 |
| johnsom | So in there, no where do we install the requirements.txt. That is likely the bug, we should use the devstack alias for python (to get the correct version) and do a pip install -r of our requirements. | 23:03 |
| sorrison | ok, I noticed that DIB_LOCAL_ELEMENTS: openstack-ci-mirrors wasn't set for the 2 node jobs either from what I can tell | 23:03 |
| johnsom | Hmm, though come to think of it, DIB should be doing that already. Let me look at that error again. | 23:04 |
| sorrison | The logs are from https://review.opendev.org/#/c/695291/12 if you want the non-raw logs | 23:04 |
| johnsom | Yeah, ok, the plot gets thicker: https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/svc-map/package-installs.yaml | 23:07 |
| johnsom | This is likely a mistake | 23:07 |
| johnsom | Well, maybe not if it installs svc-map inside the image. | 23:08 |
| johnsom | Yeah, ok, this is a DIB bug | 23:08 |
| johnsom | so, extra-data.d runs outside the chroot, which means the package-install will not install the required yaml library for the element. It's missing an dependency and is invoking python incorrectly. | 23:10 |
| cgoncalves | johnsom is indeed the local DIB expert ;) | 23:11 |
| sorrison | glad someone here is! | 23:18 |
| sorrison | Still can't figure out what makes this issue a python3 one when it works in python2 | 23:20 |
| *** elico has quit IRC | 23:23 | |
| *** armax has quit IRC | 23:23 | |
| johnsom | I think the problem is the bang path in that script: #!/usr/bin/env python | 23:25 |
| johnsom | I think it's running py2 though it should be py3 as that is where the requirements are installed. | 23:25 |
| sorrison | yeah ok, I see that commit changes it to #!/usr/local/bin/dib-python | 23:26 |
| sorrison | Do we need to set DIB_PYTHON_VERSION=3 somewhere? | 23:28 |
| sorrison | no scratch that, looks like that is set correctly | 23:29 |
| *** abaindur has joined #openstack-lbaas | 23:40 | |
| johnsom | Yeah, dib-python is only present inside the chroot, so doesn't help here. I'm bouncing ideas off of Ian in the DIB channel. | 23:43 |
| *** abaindur has quit IRC | 23:45 | |
| *** ivve has quit IRC | 23:47 | |
| *** abaindur has joined #openstack-lbaas | 23:52 | |
| *** abaindur has quit IRC | 23:53 | |
| johnsom | sorrison I will take a pass at a DIB fix and will let you know. | 23:53 |
| *** abaindur has joined #openstack-lbaas | 23:53 | |
| sorrison | thanks! | 23:53 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!