| *** xgerman has quit IRC | 00:02 | |
| *** tkajinam has quit IRC | 00:16 | |
| *** tkajinam has joined #openstack-lbaas | 00:16 | |
| *** luksky has quit IRC | 00:29 | |
| *** zigo has quit IRC | 00:54 | |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is being restarted quickly to make heap memory and jgit config adjustments, downtime should be less than 5 minutes | 01:09 | |
| *** zzzeek has quit IRC | 01:52 | |
| *** zzzeek has joined #openstack-lbaas | 01:53 | |
| *** amotoki has quit IRC | 02:01 | |
| *** amotoki has joined #openstack-lbaas | 02:02 | |
| *** zzzeek has quit IRC | 02:05 | |
| *** zzzeek has joined #openstack-lbaas | 02:06 | |
| *** jamesdenton has quit IRC | 03:17 | |
| *** jamesdenton has joined #openstack-lbaas | 03:18 | |
| *** psachin has joined #openstack-lbaas | 03:21 | |
| *** armax has quit IRC | 03:38 | |
| *** zzzeek has quit IRC | 04:19 | |
| *** zzzeek has joined #openstack-lbaas | 04:21 | |
| *** zzzeek has quit IRC | 04:26 | |
| *** zzzeek has joined #openstack-lbaas | 04:27 | |
| *** vishalmanchanda has joined #openstack-lbaas | 04:55 | |
| *** ccamposr has joined #openstack-lbaas | 05:14 | |
| *** zzzeek has quit IRC | 05:26 | |
| *** zzzeek has joined #openstack-lbaas | 05:29 | |
| *** ramishra_ has joined #openstack-lbaas | 05:32 | |
| *** ramishra has quit IRC | 05:35 | |
| *** zzzeek has quit IRC | 06:15 | |
| *** gcheresh has joined #openstack-lbaas | 06:16 | |
| *** zzzeek has joined #openstack-lbaas | 06:17 | |
| *** zzzeek has quit IRC | 06:49 | |
| *** zzzeek has joined #openstack-lbaas | 06:51 | |
| *** damien_r has joined #openstack-lbaas | 07:04 | |
| *** damien_r has quit IRC | 07:05 | |
| *** psachin has quit IRC | 07:17 | |
| *** zzzeek has quit IRC | 07:21 | |
| *** zzzeek has joined #openstack-lbaas | 07:23 | |
| *** ramishra_ is now known as ramishra | 08:08 | |
| *** rpittau|afk is now known as rpittau | 08:14 | |
| *** gcheresh has quit IRC | 08:21 | |
| *** ccamposr__ has joined #openstack-lbaas | 08:34 | |
| *** ccamposr has quit IRC | 08:38 | |
| *** ccamposr has joined #openstack-lbaas | 08:45 | |
| *** ccamposr__ has quit IRC | 08:47 | |
| *** zzzeek has quit IRC | 08:50 | |
| *** zzzeek has joined #openstack-lbaas | 08:50 | |
| *** rcernin has quit IRC | 08:56 | |
| *** rcernin has joined #openstack-lbaas | 08:56 | |
| *** gcheresh has joined #openstack-lbaas | 09:01 | |
| *** luksky has joined #openstack-lbaas | 09:10 | |
| *** zzzeek has quit IRC | 09:16 | |
| *** zzzeek has joined #openstack-lbaas | 09:17 | |
| *** rcernin has quit IRC | 09:23 | |
| *** rcernin has joined #openstack-lbaas | 09:32 | |
| *** zzzeek has quit IRC | 09:40 | |
| *** zzzeek has joined #openstack-lbaas | 09:41 | |
| *** zzzeek has quit IRC | 09:46 | |
| *** rcernin has quit IRC | 09:48 | |
| *** zzzeek has joined #openstack-lbaas | 09:49 | |
| *** rcernin has joined #openstack-lbaas | 09:49 | |
| *** gcheresh has quit IRC | 09:51 | |
| openstackgerrit | XiaoYu Zhu proposed openstack/octavia master: Alternative Distributor for L3 Active-Active, N+1 Amphora Setup https://review.opendev.org/c/openstack/octavia/+/746688 | 09:55 |
|---|---|---|
| *** ramishra has quit IRC | 09:59 | |
| *** zzzeek has quit IRC | 10:01 | |
| *** gcheresh has joined #openstack-lbaas | 10:03 | |
| *** zzzeek has joined #openstack-lbaas | 10:03 | |
| *** ramishra has joined #openstack-lbaas | 10:04 | |
| *** gcheresh has quit IRC | 10:13 | |
| *** rcernin has quit IRC | 10:31 | |
| openstackgerrit | zhangxingjun proposed openstack/octavia stable/stein: repair operating status is offline https://review.opendev.org/c/openstack/octavia/+/766198 | 10:48 |
| *** rcernin has joined #openstack-lbaas | 10:53 | |
| *** zzzeek has quit IRC | 11:17 | |
| *** zzzeek has joined #openstack-lbaas | 11:18 | |
| *** gcheresh has joined #openstack-lbaas | 11:22 | |
| *** yangjianfeng has joined #openstack-lbaas | 12:03 | |
| *** rcernin has quit IRC | 12:04 | |
| *** yangjianfeng has quit IRC | 13:21 | |
| *** devfaz has quit IRC | 14:02 | |
| *** TrevorV has joined #openstack-lbaas | 14:17 | |
| *** noonedeadpunk has quit IRC | 14:30 | |
| *** noonedeadpunk_ has joined #openstack-lbaas | 14:33 | |
| *** sapd1 has joined #openstack-lbaas | 15:13 | |
| *** TrevorV has quit IRC | 15:18 | |
| *** lxkong has quit IRC | 15:39 | |
| *** devfaz has joined #openstack-lbaas | 15:50 | |
| *** sapd1 has quit IRC | 15:56 | |
| *** armax has joined #openstack-lbaas | 16:00 | |
| rm_work | #startmeeting Octavia | 16:01 |
| openstack | Meeting started Wed Dec 9 16:01:05 2020 UTC and is due to finish in 60 minutes. The chair is rm_work. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:01 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:01 | |
| openstack | The meeting name has been set to 'octavia' | 16:01 |
| johnsom | Opps, yes, thanks | 16:01 |
| gthiemonge | Hi | 16:01 |
| rm_work | o/ | 16:01 |
| mchlumsky | Hello | 16:01 |
| johnsom | #topic Announcements | 16:01 |
| mchlumsky | Hi guys! I just want to introduce myself. My name is Martin Chlumsky and I work at Ubisoft. We have several hundred loadbalancers in Octavia (Stein) in several Openstack deployments. I've been tasked to improve our Octavia deployments (upgrade to Ussuri, better monitoring, etc...) so you'll see me in here asking questions and I'd like to contribute | 16:01 |
| mchlumsky | where I can (time permitting). :) | 16:01 |
| johnsom | After today I am on vacation until January 4th | 16:02 |
| rm_work | #chair johnsom | 16:02 |
| openstack | Current chairs: johnsom rm_work | 16:02 |
| johnsom | mchlumsky Welcome! | 16:02 |
| rm_work | #topic Announcements | 16:02 |
| *** openstack changes topic to " Announcements (Meeting topic: Octavia)" | 16:02 | |
| johnsom | Opps, lol | 16:02 |
| rm_work | now you're good :D | 16:03 |
| johnsom | I will repeat myself: After today I am on vacation until January 4th | 16:03 |
| gthiemonge | johnsom: enjoy! | 16:03 |
| johnsom | mchlumsky This channel is a great resource to ask questions about Octavia. | 16:03 |
| rm_work | Similar for me after next week -- I think i have one more meeting in me | 16:03 |
| mchlumsky | johnsom I noticed already giot a few answers. thanks! :) | 16:04 |
| johnsom | Nice, enjoy your break | 16:04 |
| johnsom | Any other announcements this week? | 16:04 |
| johnsom | #topic Brief progress reports / bugs needing review | 16:05 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:05 | |
| johnsom | I have mostly been working on reviews and poking at the bandit issues on stable/stein. Otherwise downstream work | 16:06 |
| gthiemonge | Can we start using the priority review etherpad? I know that some people have added patches to the awaiting prioritization list... | 16:07 |
| rm_work | yeah I really need to get back in the groove of reviews | 16:07 |
| johnsom | Yeah, good question. Milestone 1 has now passed | 16:07 |
| rm_work | my guess is the first opportunity i'll have to do so will really be in January | 16:08 |
| *** sapd1 has joined #openstack-lbaas | 16:09 | |
| johnsom | gthiemonge I would say feel free to freshen up the priority review etherpad and get it started for Wallaby | 16:09 |
| gthiemonge | ok, I'll update the list tomorrow | 16:09 |
| * johnsom likes not being PTL this cycle and still delegating. grin | 16:10 | |
| johnsom | Any other updates this week? | 16:11 |
| johnsom | Thank you all for helping review the stable branch patches. I think we have made good progress this week | 16:11 |
| johnsom | #topic Stable/Stein gate jobs are broken due to bandit py27->py3x transition | 16:11 |
| *** openstack changes topic to "Stable/Stein gate jobs are broken due to bandit py27->py3x transition (Meeting topic: Octavia)" | 16:11 | |
| johnsom | As you may have seen on the openstack-discuss e-mail list, bandit released a version that no longer supports python2.7. | 16:12 |
| johnsom | This broke a number of projects including the stable/stein branch test jobs. | 16:12 |
| johnsom | rm_work and I have poked at the issue and I will probably spend a bit of time on that today again. | 16:13 |
| johnsom | This is blocking some of our tempest patches from passing. | 16:13 |
| johnsom | We will pin bandit on the stein branch to a version that supports python2.7 | 16:14 |
| johnsom | #topic Open Discussion | 16:14 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:14 | |
| johnsom | Any other topics this week? | 16:14 |
| mchlumsky | I have a story I would like to work on (https://storyboard.openstack.org/#!/story/2008060) as we will likely need this feature. I think it should be labelled as a RFE (but I'm not too sure of the complexity of the change yet so maybe it's a spec?). I might require some guidance but I'll reach out unless you have some vital pieve of information to | 16:14 |
| mchlumsky | give me right now. ;) | 16:14 |
| johnsom | Ah, yeah. So, basically you will want to enable changing that setting and then trigger a load balancer failover process to enact the change. | 16:15 |
| johnsom | I would advise against trying to hot-change CPU or RAM via nova as it unfortunately triggers an instance reboot, which can cause the security content (TLS keys) to be lost, which will trigger a failover as well. | 16:17 |
| mchlumsky | Would this be done via 2 separate API operations? or can this be done behind the scenes via a PUT to the LB? | 16:17 |
| johnsom | Does that align to your thoughts? | 16:17 |
| johnsom | You can do it all behind one API call | 16:17 |
| mchlumsky | I won't touch anything on the nova side. I was totally thinking of doing a failover to get this done. I'm just not sure what will happen when you change the topology from active-standby to standalone for example | 16:19 |
| rm_work | you'd need to delete the backup basically, and send a new config to the remaining amp, generated with STANDALONE as the type | 16:20 |
| johnsom | Well, with the changes I made during Victoria to the failover flow, it should handle the case ok. It will just think that nova lost one of the amphora instances. | 16:20 |
| rm_work | but yeah, the failover flow itself should basically just "do that" | 16:20 |
| rm_work | new failover flow is great! :thumbsup: | 16:20 |
| johnsom | I think I would do a full LB failover. I wouldn't mess with converting an existing instance. Converting would mean a lot of changes inside the amp (keepalived, etc.) that just isn't worth it | 16:21 |
| rm_work | ah yeah that's probably cleanest | 16:21 |
| mchlumsky | ok, great. | 16:21 |
| rm_work | actually yeah, would it literally just be changing the topo to standalone, and triggering the failover? XD | 16:22 |
| johnsom | Yeah, pretty much. The tricky part is handling the rollback correctly maybe. | 16:22 |
| rm_work | i suppose that might just work | 16:22 |
| mchlumsky | thanks for making this all so easy for me :p | 16:23 |
| johnsom | So, I think this is pretty straight forward and we had a good discussion on it. I'm not sure it needs a full spec to hammer out the details. I would just tag this story as RFE and propose the change | 16:23 |
| mchlumsky | awesome. | 16:24 |
| johnsom | Any other topics this week? | 16:24 |
| gthiemonge | yes! here | 16:25 |
| gthiemonge | one of my colleagues is working on a tempest test for TLS | 16:25 |
| gthiemonge | https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/763169 | 16:25 |
| gthiemonge | it looks like it's failing on stein | 16:25 |
| gthiemonge | I'm wondering if it's worth it to spend some time to fix stein, or we could just skip the test for some old releases | 16:26 |
| johnsom | Well, if the feature was present in stein, we should not skip the test. | 16:26 |
| rm_work | how long until we can EOL stein again? :D | 16:27 |
| rm_work | we should really be doing one EOL per cycle... | 16:27 |
| rm_work | what's up next? | 16:27 |
| johnsom | For one thing, I would strongly advise to not change the existing tests, but add additional. | 16:27 |
| gthiemonge | ok, we can check the test in stein... perhaps one backport is missing there | 16:27 |
| johnsom | It appears the existing tests were incorrectly modified | 16:28 |
| gthiemonge | I haven't reviewed the patch yet... but yeah the issue might be in the test itself | 16:29 |
| johnsom | I can review later, but that is my first comment. Don't change existing tests unless absolutely necessary. | 16:29 |
| gthiemonge | ok thanks | 16:30 |
| johnsom | Especially with the TLS tests, it's easy and quick to just create another listener for your new test case. | 16:30 |
| johnsom | The current TLS tests are very detailed and still valid IMO | 16:31 |
| johnsom | Any other topics to raise? | 16:32 |
| johnsom | Ok, thanks everyone. Have a great week and holiday! | 16:33 |
| gthiemonge | johnsom: thank you | 16:34 |
| johnsom | #endmeeting | 16:34 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 16:34 | |
| openstack | Meeting ended Wed Dec 9 16:34:15 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:34 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-12-09-16.01.html | 16:34 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-12-09-16.01.txt | 16:34 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-12-09-16.01.log.html | 16:34 |
| mchlumsky | thank you! :) | 16:34 |
| rm_work | o/ | 16:35 |
| *** dayou has quit IRC | 17:10 | |
| *** rpittau is now known as rpittau|afk | 17:12 | |
| *** TrevorV has joined #openstack-lbaas | 17:19 | |
| *** xgerman has joined #openstack-lbaas | 17:25 | |
| xgerman | johnsom: rm_work cgoncalves Wonder if we are having some virtual Octavia Holiday Party/Happy Hour... | 17:27 |
| *** ccamposr has quit IRC | 17:32 | |
| *** ccamposr has joined #openstack-lbaas | 17:32 | |
| *** luksky has quit IRC | 17:35 | |
| *** noonedeadpunk_ is now known as noonedeadpunk | 18:01 | |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is currently responding slowly or timing out due to resource starvation, investigation is underway | 18:15 | |
| *** ccamposr__ has joined #openstack-lbaas | 18:49 | |
| *** ccamposr has quit IRC | 18:51 | |
| *** lxkong has joined #openstack-lbaas | 19:40 | |
| *** ccamposr has joined #openstack-lbaas | 19:57 | |
| *** ccamposr__ has quit IRC | 19:58 | |
| *** ccamposr has quit IRC | 20:00 | |
| *** ccamposr has joined #openstack-lbaas | 20:04 | |
| *** ccamposr has quit IRC | 20:08 | |
| *** dayou has joined #openstack-lbaas | 20:11 | |
| *** ccamposr has joined #openstack-lbaas | 20:14 | |
| *** ccamposr has quit IRC | 20:25 | |
| *** ccamposr has joined #openstack-lbaas | 20:25 | |
| *** ccamposr has quit IRC | 20:27 | |
| *** ccamposr has joined #openstack-lbaas | 20:35 | |
| *** ccamposr__ has joined #openstack-lbaas | 20:39 | |
| *** ccamposr has quit IRC | 20:39 | |
| *** vishalmanchanda has quit IRC | 20:54 | |
| *** ccamposr__ has quit IRC | 20:58 | |
| *** ccamposr has joined #openstack-lbaas | 21:00 | |
| *** ccamposr has quit IRC | 21:00 | |
| *** ccamposr has joined #openstack-lbaas | 21:00 | |
| *** sapd1 has quit IRC | 21:04 | |
| *** ccamposr has quit IRC | 21:26 | |
| *** ccamposr has joined #openstack-lbaas | 21:27 | |
| *** ccamposr has quit IRC | 21:45 | |
| *** ccamposr has joined #openstack-lbaas | 21:50 | |
| *** rcernin has joined #openstack-lbaas | 22:02 | |
| *** rcernin has quit IRC | 22:04 | |
| *** rcernin has joined #openstack-lbaas | 22:05 | |
| *** gcheresh has quit IRC | 22:11 | |
| *** ccamposr has quit IRC | 22:13 | |
| *** ccamposr has joined #openstack-lbaas | 23:21 | |
| *** TrevorV has quit IRC | 23:22 | |
| *** lemko3 has joined #openstack-lbaas | 23:28 | |
| *** lemko has quit IRC | 23:31 | |
| *** lemko3 is now known as lemko | 23:31 | |
| lxkong | The reason for our UDP listener is not working is `/var/lib/octavia/vrrp/check_scripts/lvs_udp_check.sh: 4: /var/lib/octavia/vrrp/check_scripts/lvs_udp_check.sh: shopt: not found`, so that the VIP is not bond to VRRP port. | 23:32 |
| lxkong | johnsom, rm_work, was that a know issue before? We are using stable/ussuri | 23:33 |
| lxkong | i looked at my devstack environment, the script is the same | 23:33 |
| lxkong | but it's working (master octavia on my devstack) | 23:33 |
| johnsom | That script hasn't changed since it originally merged | 23:35 |
| johnsom | Three years ago | 23:35 |
| rm_work | hmmm | 23:35 |
| rm_work | shopt??? isn't that part of bash | 23:35 |
| rm_work | what OS is your image based on lxkong ? | 23:36 |
| johnsom | It seems like an image build issue maybe? like bash isn't installed, but busybox or something? | 23:36 |
| rm_work | yeah | 23:36 |
| johnsom | lol, yeah that ^^^^ | 23:36 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/templates/keepalived_lvs_check_script.sh.j2 | 23:36 |
| rm_work | used to deal with that more in embedded, like back in my #maemo or Sharp Zaurus days | 23:36 |
| rm_work | I miss Maemo | 23:37 |
| lxkong | we are using ubuntu... | 23:37 |
| rm_work | hmmm | 23:37 |
| rm_work | can you get into the amp and just... try running `shopt`? | 23:38 |
| lxkong | In the syslog, we can see | 23:38 |
| lxkong | https://www.irccloud.com/pastebin/jA3vD3TI/ | 23:38 |
| lxkong | then I ran `bash -x /var/lib/octavia/vrrp/check_script.sh` | 23:38 |
| lxkong | and met with that error | 23:39 |
| rm_work | yeah ok | 23:39 |
| rm_work | when you run shopt | 23:39 |
| rm_work | what happens? | 23:39 |
| lxkong | shopt is working though... | 23:39 |
| rm_work | ... hmm | 23:39 |
| lxkong | https://www.irccloud.com/pastebin/iFYkRO9L/ | 23:40 |
| lxkong | we just started testing UDP recently and found it's not working | 23:40 |
| lxkong | The image was built using `diskimage-create.sh` from octavia repo. | 23:41 |
| lxkong | hmm, wait, in devstack, it's also failing | 23:44 |
| lxkong | https://www.irccloud.com/pastebin/GgV89Nzz/ | 23:44 |
| lxkong | so that may not be the reason of VIP not bond | 23:44 |
| lxkong | in the amphora of devstack, i can see the VIP address on the VRRP port. | 23:44 |
| lxkong | but in our cloud, it's not | 23:45 |
| johnsom | in the check_script.sh, what happens if you change sh to bash? | 23:45 |
| lxkong | https://www.irccloud.com/pastebin/b47xTC9A/ | 23:46 |
| johnsom | Yeah, that is interesting, it appears to still be running the check as the "Check keepalived pid file" message is there | 23:46 |
| johnsom | Yeah, so that is a bug that has been there for 5 years. lol Something must have changed in the image. | 23:47 |
| lxkong | johnsom: i also changed to use bash in our amphora, i can see the VIP now | 23:48 |
| lxkong | let me check if the UDP is working or not | 23:48 |
| lxkong | johnsom: yeah, it's working ! | 23:48 |
| lxkong | johnsom, rm_work, thanks, maybe i need to make a hotfix for us first. | 23:49 |
| johnsom | Yeah, so something must have changed in ubuntu. Those are 3 and 5 years old. | 23:49 |
| johnsom | lxkong are you going to post a patch for octavia? | 23:49 |
| rm_work | hmmmm | 23:50 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/templates/keepalived_check_script.conf.j2 | 23:50 |
| rm_work | weird | 23:50 |
| lxkong | johnsom: yep, i can do that | 23:53 |
| johnsom | Thank you | 23:53 |
| lxkong | but i don't understand, why it's working in my devstack | 23:53 |
| lxkong | with the same error msg. | 23:53 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!