| openstackgerrit | Merged openstack/octavia stable/train: Fix memory consumption issues with default connection_limit https://review.opendev.org/c/openstack/octavia/+/747662 | 00:04 |
|---|---|---|
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/train: Fix operational status for disabled UDP listeners https://review.opendev.org/c/openstack/octavia/+/752871 | 00:06 |
| johnsom | Rebasing things so we can get the stable/train patches moving. | 00:07 |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/ussuri: Followup Remove haproxy_check_script for UDP-only https://review.opendev.org/c/openstack/octavia/+/758034 | 00:10 |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/train: Fix load balancers with failed amphora failover https://review.opendev.org/c/openstack/octavia/+/763733 | 00:14 |
| *** vesper11 has joined #openstack-lbaas | 01:09 | |
| *** rcernin has quit IRC | 01:37 | |
| *** rcernin has joined #openstack-lbaas | 01:39 | |
| *** spatel has joined #openstack-lbaas | 01:48 | |
| *** sapd1_x has joined #openstack-lbaas | 02:30 | |
| *** rcernin has quit IRC | 02:36 | |
| *** sapd1_x has quit IRC | 02:42 | |
| *** rcernin has joined #openstack-lbaas | 03:06 | |
| *** xgerman has quit IRC | 04:02 | |
| openstackgerrit | Merged openstack/octavia stable/victoria: Fix load balancers with failed amphora failover https://review.opendev.org/c/openstack/octavia/+/763731 | 04:04 |
| *** sapd1_x has joined #openstack-lbaas | 04:23 | |
| *** vishalmanchanda has joined #openstack-lbaas | 04:38 | |
| *** gcheresh has joined #openstack-lbaas | 04:47 | |
| *** sapd1_x has quit IRC | 05:01 | |
| *** gcheresh has quit IRC | 05:23 | |
| *** gcheresh has joined #openstack-lbaas | 05:29 | |
| *** rcernin has quit IRC | 05:43 | |
| *** rcernin has joined #openstack-lbaas | 05:55 | |
| *** gcheresh has quit IRC | 06:08 | |
| openstackgerrit | Merged openstack/octavia stable/ussuri: Add some details on enable_anti_affinity option https://review.opendev.org/c/openstack/octavia/+/752865 | 06:20 |
| openstackgerrit | Merged openstack/octavia stable/ussuri: [Amphorav2] Healthmonitor operation minor fixes https://review.opendev.org/c/openstack/octavia/+/753969 | 06:22 |
| *** spatel has quit IRC | 06:37 | |
| *** gcheresh has joined #openstack-lbaas | 06:38 | |
| *** zzzeek has quit IRC | 07:13 | |
| *** zzzeek has joined #openstack-lbaas | 07:14 | |
| *** rcernin has quit IRC | 07:38 | |
| *** ccamposr__ has joined #openstack-lbaas | 07:42 | |
| *** rcernin has joined #openstack-lbaas | 07:44 | |
| *** ccamposr has quit IRC | 07:45 | |
| *** rcernin has quit IRC | 07:48 | |
| *** rcernin has joined #openstack-lbaas | 08:01 | |
| *** damien_r has joined #openstack-lbaas | 08:12 | |
| *** rcernin has quit IRC | 08:16 | |
| *** damien_r has quit IRC | 08:17 | |
| *** rpittau|afk is now known as rpittau | 08:28 | |
| *** rcernin has joined #openstack-lbaas | 08:32 | |
| *** rcernin has quit IRC | 08:46 | |
| *** tkajinam has quit IRC | 08:49 | |
| *** sapd1_x has joined #openstack-lbaas | 09:00 | |
| *** luksky has joined #openstack-lbaas | 09:15 | |
| *** rcernin has joined #openstack-lbaas | 09:42 | |
| *** rcernin has quit IRC | 09:42 | |
| *** sapd1_x has quit IRC | 10:00 | |
| *** happyhemant has joined #openstack-lbaas | 10:05 | |
| noonedeadpunk | hi! Does anybody have any idea why this might be happening? https://zuul.opendev.org/t/openstack/build/493fb4d9f67a44e19d7547ffac3513f0/log/logs/host/octavia-api.service.journal-00-28-19.log.txt#981 | 10:13 |
| noonedeadpunk | I see it only on Debian buster (which has py3.7 on board) | 10:13 |
| noonedeadpunk | I'm not even sure where it's raised as not trace :( | 10:14 |
| *** sapd1_x has joined #openstack-lbaas | 10:15 | |
| gthiemonge | noonedeadpunk: could it be a client that timed out before receiving the reply? the api would write into a closed socket | 10:21 |
| noonedeadpunk | well client is tempest in this case I guess. as eventually because of it's failure I went to look through service logs https://fcdf6ccfb4f2f01e908c-450f6f29734ae7f18d179491368c8872.ssl.cf5.rackcdn.com/755576/5/gate/openstack-ansible-deploy-aio_metal-debian-buster/493fb4d/logs/openstack/aio1-utility/stestr_results.html | 10:23 |
| noonedeadpunk | hm, from other side it might be our LB issue... | 10:24 |
| noonedeadpunk | yeah it was marked as down https://zuul.opendev.org/t/openstack/build/493fb4d9f67a44e19d7547ffac3513f0/log/logs/host/haproxy.service.journal-00-28-19.log.txt#1618 | 10:25 |
| noonedeadpunk | but eventually everything was marked that way - interesting | 10:25 |
| noonedeadpunk | thanks for the point | 10:25 |
| noonedeadpunk | I'm wondering if octavia that run on host can influence work of the haproxy that is also launched on the same host.... | 10:27 |
| noonedeadpunk | (or keepalived) | 10:27 |
| gthiemonge | no, it should not | 10:28 |
| noonedeadpunk | yeah, indeed as it manages things inside netns at least | 10:29 |
| gthiemonge | there are many health check timeouts in haproxy around the same time (~00:25) | 10:30 |
| gthiemonge | for many services | 10:30 |
| noonedeadpunk | yeah and interesting what could cause them... smth like oom maybe.... | 10:35 |
| noonedeadpunk | yeah, `Dec 1 20:30:09 aio1 kernel: [ 4265.267970] uwsgi invoked oom-killer` - sorry for bothering :( | 10:37 |
| noonedeadpunk | was just weird that fails only for debian and at the exactly same tests... | 10:37 |
| *** sapd1_x has quit IRC | 10:43 | |
| *** ramishra_ has joined #openstack-lbaas | 10:54 | |
| *** ramishra has quit IRC | 10:57 | |
| *** sapd1_x has joined #openstack-lbaas | 11:15 | |
| openstackgerrit | Arkady Shtempler proposed openstack/octavia-tempest-plugin master: New negative test case - test_delete_udp_scenario https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/755281 | 11:34 |
| *** sapd1_x has quit IRC | 11:40 | |
| *** vesper11 has quit IRC | 12:53 | |
| *** haleyb has joined #openstack-lbaas | 13:09 | |
| *** TrevorV has joined #openstack-lbaas | 13:16 | |
| *** sapd1_x has joined #openstack-lbaas | 13:24 | |
| openstackgerrit | Merged openstack/octavia stable/train: Fix load balancers with failed amphora failover https://review.opendev.org/c/openstack/octavia/+/763733 | 13:33 |
| *** haleyb has quit IRC | 13:46 | |
| *** spatel has joined #openstack-lbaas | 13:52 | |
| *** wuchunyang has joined #openstack-lbaas | 13:53 | |
| *** spatel has quit IRC | 13:53 | |
| *** haleyb has joined #openstack-lbaas | 14:13 | |
| *** damien_r has joined #openstack-lbaas | 15:20 | |
| *** devfaz has quit IRC | 15:38 | |
| openstackgerrit | Merged openstack/octavia master: Fix nf_conntrack_buckets sysctl in Amphora https://review.opendev.org/c/openstack/octavia/+/748749 | 15:39 |
| *** devfaz has joined #openstack-lbaas | 15:40 | |
| openstackgerrit | wu.chunyang proposed openstack/octavia master: Add notifications specification documens https://review.opendev.org/c/openstack/octavia/+/727915 | 15:41 |
| *** sapd1_x has quit IRC | 15:44 | |
| *** wuchunyang has quit IRC | 15:53 | |
| *** ataraday_ has joined #openstack-lbaas | 15:59 | |
| johnsom | #startmeeting Octavia | 16:00 |
| openstack | Meeting started Wed Dec 2 16:00:20 2020 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:00 | |
| openstack | The meeting name has been set to 'octavia' | 16:00 |
| ataraday_ | hi | 16:00 |
| johnsom | Hi everyone | 16:00 |
| haleyb | hi | 16:00 |
| cgoncalves | hi | 16:00 |
| gthiemonge | Hi | 16:00 |
| johnsom | I hope you all enjoyed the break from this meeting. grin | 16:01 |
| johnsom | #topic Announcements | 16:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 16:01 | |
| johnsom | I don't have any announcements this week. Does anyone else? | 16:01 |
| johnsom | #topic Brief progress reports / bugs needing review | 16:02 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:02 | |
| johnsom | I have been working on stable branch reviews and poking the tempest patch backlog | 16:02 |
| johnsom | Any other updates this week? | 16:03 |
| ataraday_ | I'm a bit overwhelmed with internal tasks. Please review this fix https://review.opendev.org/c/openstack/octavia/+/760380 | 16:03 |
| gthiemonge | I've been working on octavia-dashboard, I added configuration for TLS options in pools and listeners | 16:03 |
| johnsom | atarady_ Same here. | 16:04 |
| johnsom | I will take a look at that after the meeting | 16:04 |
| ataraday_ | I've got a question about tempest default provider - it set as 'octavia', should it be change to v2 path as well? | 16:05 |
| ataraday_ | For testing I changed it to amphora https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/763319 | 16:05 |
| johnsom | Yes. We had "octavia" in there to test backward compatibility to when it was called "octavia". That deprecation cycle should be passed now I think, cgoncalves may know better. | 16:06 |
| johnsom | Maybe we should add an upgrade check for that. Or alias "octavia" over to the new driver as well. | 16:06 |
| johnsom | Aliasing it over as well might just be simpler. | 16:08 |
| *** luksky has quit IRC | 16:08 | |
| ataraday_ | OK, may be add it as part of alias change then | 16:08 |
| *** damien_r has quit IRC | 16:08 | |
| ataraday_ | Thank you! | 16:09 |
| johnsom | That would be my advice to not break people upgrading that haven't switched the config | 16:09 |
| johnsom | Cool, any other comments or updates? | 16:09 |
| johnsom | #topic Stable branches | 16:09 |
| *** openstack changes topic to "Stable branches (Meeting topic: Octavia)" | 16:09 | |
| johnsom | Please help us get the stable branches released. There are many backport patches open for review. | 16:10 |
| johnsom | I have been trying to work on reviews there, please help out. | 16:10 |
| johnsom | We *really* need to do some stable branch releases. | 16:10 |
| johnsom | Any questions/comments on the stable branches? | 16:10 |
| gthiemonge | ack, I'll review them | 16:10 |
| johnsom | Things seem to be merging there, it is just down to getting patches merged. | 16:11 |
| *** redrobot has quit IRC | 16:11 | |
| johnsom | merged/reviewed. lol | 16:11 |
| johnsom | #topic Open Discussion | 16:11 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:11 | |
| johnsom | Any other topics this week? | 16:11 |
| johnsom | Ok, thanks everyone. Have a good week. | 16:13 |
| johnsom | #endmeeting | 16:13 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews" | 16:13 | |
| openstack | Meeting ended Wed Dec 2 16:13:19 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:13 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-12-02-16.00.html | 16:13 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-12-02-16.00.txt | 16:13 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-12-02-16.00.log.html | 16:13 |
| gthiemonge | thanks johnsom | 16:13 |
| *** devfaz has quit IRC | 16:23 | |
| *** devfaz has joined #openstack-lbaas | 16:47 | |
| *** ataraday_ has quit IRC | 16:52 | |
| openstackgerrit | Brian Haley proposed openstack/octavia stable/ussuri: Handle listener not found during loadbalancer status update https://review.opendev.org/c/openstack/octavia/+/765087 | 16:54 |
| *** Guest18921 has joined #openstack-lbaas | 16:58 | |
| *** Guest18921 is now known as redrobot | 17:01 | |
| *** rpittau is now known as rpittau|afk | 17:04 | |
| *** luksky has joined #openstack-lbaas | 17:18 | |
| *** xgerman has joined #openstack-lbaas | 17:33 | |
| *** vishalmanchanda has quit IRC | 18:58 | |
| *** ramishra_ has quit IRC | 19:00 | |
| *** gcheresh has quit IRC | 19:14 | |
| *** ccamposr__ has quit IRC | 19:20 | |
| *** ccamposr__ has joined #openstack-lbaas | 19:21 | |
| *** ccamposr has joined #openstack-lbaas | 19:25 | |
| *** ccamposr__ has quit IRC | 19:28 | |
| *** mchlumsky has quit IRC | 20:01 | |
| *** mchlumsky has joined #openstack-lbaas | 20:03 | |
| *** mchlumsky has quit IRC | 20:05 | |
| *** mchlumsky has joined #openstack-lbaas | 20:06 | |
| *** vesper11 has joined #openstack-lbaas | 20:15 | |
| *** vesper has joined #openstack-lbaas | 20:19 | |
| *** vesper11 has quit IRC | 20:19 | |
| *** vesper has quit IRC | 21:06 | |
| *** gregwork has joined #openstack-lbaas | 21:09 | |
| *** vesper11 has joined #openstack-lbaas | 21:30 | |
| *** TrevorV has quit IRC | 21:33 | |
| *** gcheresh has joined #openstack-lbaas | 21:53 | |
| haleyb | johnsom: have a minute for an API question? I seem to be finding the corner cases trying to get the OVN provider passing tests :( | 22:03 |
| johnsom | haleyb Sure, go for it | 22:05 |
| * haleyb reels in johnsom as he took the bait :) | 22:05 | |
| haleyb | I see a difference in the member_* tests | 22:06 |
| haleyb | https://opendev.org/openstack/octavia-tempest-plugin/src/branch/master/octavia_tempest_plugin/tests/api/v2/test_member.py#L2173 | 22:06 |
| haleyb | the create and update (and delete) tests all create a member, but the update one checks for NO_MONITOR | 22:06 |
| haleyb | but only in the case of the noop driver | 22:07 |
| haleyb | search above that and see the code is not the same at L913 | 22:08 |
| haleyb | the issue is the OVN provider has set CONF.loadbalancer_feature_enabled.health_monitor_enabled=False which causes it to fail that update test because it assumes OFFLINE | 22:09 |
| haleyb | the code was updated in your refactor, it used to look at that config option | 22:09 |
| johnsom | Yeah, that if conditional doesn't need to be there. This test suite doesn't create a health monitor on the pool, so it should always be "NO_MONITOR". | 22:09 |
| johnsom | This junk: CONF.loadbalancer_feature_enabled.health_monitor_enabled just needs to be deleted. | 22:10 |
| haleyb | i was just wondering if all those tests should just be the same with the simple | 22:10 |
| haleyb | self.assertEqual(const.NO_MONITOR, member[const.OPERATING_STATUS]) | 22:10 |
| haleyb | johnsom: well, that option is valid, right? the health_monitor_enabled one | 22:11 |
| johnsom | No, that "enabled" junk needs to be deleted. It was part of the test cleanup work I was doing. It's all garbage. | 22:12 |
| haleyb | https://opendev.org/openstack/octavia-tempest-plugin/src/branch/master/octavia_tempest_plugin/config.py#L231 | 22:12 |
| haleyb | that ^^ ? | 22:12 |
| johnsom | Ok, so I see why update is different here: | 22:12 |
| johnsom | https://opendev.org/openstack/octavia-tempest-plugin/src/branch/master/octavia_tempest_plugin/tests/api/v2/test_member.py#L2130 | 22:12 |
| johnsom | haleyb Yeah, that is all of those junk options that were added for the OVN provider. They all need to go away. Well, all of them with the work "provider" in the description. | 22:13 |
| johnsom | So, back to the update test. | 22:13 |
| haleyb | so it's admin state down, so should transision to OFFLINE | 22:14 |
| * haleyb didn't noticed that difference | 22:14 | |
| johnsom | It starts in admin state down. This why no-op will behave differently than a live driver. Yeah. Admin down should == OFFLINE per the API. | 22:15 |
| johnsom | Does that help you out on the NO_MONITOR/OFFLINE/ONLINE scenario? | 22:16 |
| haleyb | johnsom: so it's probably a bug in the OVN driver not looking at that and setting OFFLINE correctly | 22:16 |
| johnsom | Ok, yeah, could be. It may not be honoring admin down for the health monitor. | 22:17 |
| haleyb | johnsom: yes, many thanks, even the examples i found showed NO_MONITOR | 22:17 |
| haleyb | well, it doesn't support the health monitor, but i think the default on member create is NO_MONITOR, and each driver needs to change if necessary, like OFFLINE in this case | 22:18 |
| *** gcheresh has quit IRC | 22:18 | |
| johnsom | As for the "enabled"s... This was the stuff I was working to pull out of the test code. There were still a few places that were using some of those. I had planned to fix those and then push a capstone patch to remove them from the config. But I got sidetracked as you know. | 22:19 |
| johnsom | haleyb Here is what the initial state will be after a create call via the API: https://github.com/openstack/octavia/blob/master/octavia/db/prepare.py#L192 | 22:20 |
| haleyb | johnsom: ack, the OVN driver actually sets a lot of them to False | 22:20 |
| johnsom | Yeah, which does strange conditionals in the tests and becomes bad news. | 22:21 |
| haleyb | johnsom: so in the OVN case there is no health monitor, so that's why it's NO_MONITOR (as expected), looks like another patch to the driver to set OFFLINE | 22:23 |
| johnsom | Yeah, probably. | 22:25 |
| haleyb | well, i confirmed it ignores admin_state_up in create :( only looks in update | 22:26 |
| johnsom | Job security | 22:27 |
| haleyb | https://dilbert.com/strip/1995-11-13 | 22:28 |
| johnsom | Nice | 22:28 |
| *** rcernin has joined #openstack-lbaas | 22:34 | |
| *** tkajinam has joined #openstack-lbaas | 23:00 | |
| *** ccamposr__ has joined #openstack-lbaas | 23:18 | |
| *** ccamposr has quit IRC | 23:20 | |
| *** happyhemant has quit IRC | 23:54 | |
| *** luksky has quit IRC | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!