Wednesday, 2019-11-27

*** ivve has quit IRC00:04
*** strobert1 has quit IRC00:32
ataraday_johnsom, Hi! I've got a question how to bump requirements for taskflow properly on change https://review.opendev.org/#/c/647406/59/requirements.txt Should I send change for openstack/requirements first? What is the procedure nowadays?00:43
johnsomataraday_ Hi, looking00:44
johnsomataraday_ So global-requirements is unlimited and upper-constraints is 3.8.0, so your bump to 3.7.1 in our requirements file is all you need.00:45
johnsomOh, maybe a bump in lower-requirements in our repo00:46
ataraday_oh, yeah, probably that is the thing00:47
johnsomataraday_ Yeah, you just need to bump the lower-constraints.txt in our repo too00:47
ataraday_thanks! sorry for bothering with that :)00:48
johnsomNo worries00:48
openstackgerritAnn Taraday proposed openstack/octavia master: Jobboard based controller  https://review.opendev.org/64740600:49
*** ataraday_ has quit IRC00:56
*** strobert1 has joined #openstack-lbaas01:09
*** ricolin has joined #openstack-lbaas01:52
openstackgerritMerged openstack/octavia stable/train: Fix a potential race condition with certs-ramfs  https://review.opendev.org/69482302:03
openstackgerritMerged openstack/octavia master: Fix filtering with API tags  https://review.opendev.org/69511202:05
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Introduce a simple HTTPS server for TLS test.  https://review.opendev.org/62889102:05
johnsomWell, things are merging again, so I guess we can claim victory in our multi-day, 6+ outside issues breaking our gates, battle. lol02:06
eanderssonSo tired of ci breaking -_-02:09
johnsomYes, yes we are too02:10
johnsomeandersson FYI, this is probably the same thing that is hitting designate grenade: https://bugs.launchpad.net/grenade/+bug/185409302:10
openstackLaunchpad bug 1854093 in grenade "grenade fails due to PyYAML conflict" [Undecided,New]02:10
eanderssonYea sounds about right02:10
johnsomOnce you peal back a few of the other issues02:10
eanderssonI feel like it's something new everytime I hit recheck02:11
johnsomYeah, we are at least 6 issues deep. Though the requirements mistake that hit us probably didn't impact you as it was gunicorn02:11
eanderssonAt least I finally found the memory leak in the openstacksdk02:11
eandersson(unrelated to CI breaking)02:12
johnsomOh nice! I wonder if that will help our horizon plugin. It uses the SDK for the Octavia calls02:12
eanderssonoh lol good call02:12
eanderssonWe have had a memory leak forever in horizon02:12
eanderssonI didn't even make that connection02:12
eanderssonhttp://eavesdrop.openstack.org/irclogs/%23openstack-sdks/%23openstack-sdks.2019-11-22.log.html02:14
eanderssonI was many hours deep when I thought I found it and felt stupid that I posted a line and then said nvm02:14
johnsomStory of my life02:14
eanderssonbut turns out that it was just that line causing it and lucky that modred picked up on it02:14
eanderssonAlmost thanksgiving vacation so time to recharge.. will retry the gate after that lol02:16
johnsomYep, have a good one!02:17
eanderssonYou too02:17
johnsomI am out for the evening myself.02:17
*** goldyfruit_ has joined #openstack-lbaas03:02
*** abaindur has quit IRC03:15
*** goldyfruit_ has quit IRC04:01
*** armax has quit IRC04:07
openstackgerritAjay Kumar proposed openstack/octavia master: Add install guide for Ubuntu  https://review.opendev.org/67284204:52
*** gcheresh has joined #openstack-lbaas04:55
*** gcheresh has quit IRC05:02
*** spatel has joined #openstack-lbaas05:43
*** spatel has quit IRC05:47
*** threestrands has quit IRC05:51
*** psachin has joined #openstack-lbaas06:10
*** numans has joined #openstack-lbaas06:31
*** pcaruana has joined #openstack-lbaas06:34
*** ccamposr__ has joined #openstack-lbaas06:51
*** ccamposr has quit IRC06:54
*** maciejjozefczyk has joined #openstack-lbaas07:57
*** tesseract has joined #openstack-lbaas08:04
*** gcheresh has joined #openstack-lbaas08:06
*** tkajinam has quit IRC08:21
*** Dinesh_Bhor has joined #openstack-lbaas08:55
*** rpittau|afk is now known as rpittau08:57
*** ivve has joined #openstack-lbaas08:58
*** baffle has quit IRC09:27
*** openstackstatus has quit IRC09:30
*** rcernin has quit IRC10:31
*** baffle has joined #openstack-lbaas10:37
*** rcernin has joined #openstack-lbaas10:46
*** tkajinam has joined #openstack-lbaas10:50
openstackgerritGregory Thiemonge proposed openstack/octavia master: Use 2048-bits keys for devstack certificates  https://review.opendev.org/69603710:51
*** sapd1 has joined #openstack-lbaas10:59
*** rcernin has quit IRC11:31
*** sapd1 has quit IRC11:39
*** tkajinam has quit IRC12:11
*** tkajinam has joined #openstack-lbaas12:15
*** tkajinam has quit IRC12:16
*** CobHead has quit IRC12:32
*** ccamposr has joined #openstack-lbaas12:37
*** ccamposr__ has quit IRC12:39
*** ccamposr__ has joined #openstack-lbaas13:01
*** ccamposr has quit IRC13:04
*** psachin has quit IRC13:53
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: DNM checking scenarios on all branches  https://review.opendev.org/69630914:00
*** ccamposr__ has quit IRC14:01
*** ccamposr has joined #openstack-lbaas14:05
openstackgerritMerged openstack/octavia stable/queens: Fix controller worker graceful shutdown  https://review.opendev.org/69537914:08
openstackgerritMerged openstack/octavia stable/stein: Fix controller worker graceful shutdown  https://review.opendev.org/69537614:08
openstackgerritMerged openstack/octavia stable/train: Fix controller worker graceful shutdown  https://review.opendev.org/69537414:08
openstackgerritMerged openstack/octavia stable/rocky: Fix controller worker graceful shutdown  https://review.opendev.org/69537714:09
openstackgerritMerged openstack/octavia stable/stein: Fix filtering for provider capabilities list API  https://review.opendev.org/69482814:54
openstackgerritMerged openstack/octavia stable/stein: Fix listeners with SNI certificates  https://review.opendev.org/69482214:54
openstackgerritMerged openstack/octavia stable/train: Fix filtering for provider capabilities list API  https://review.opendev.org/69482714:54
openstackgerritMerged openstack/octavia stable/stein: Fix a potential race condition with certs-ramfs  https://review.opendev.org/69482414:54
*** goldyfruit_ has joined #openstack-lbaas14:55
openstackgerritMichael Johnson proposed openstack/octavia stable/train: Fix filtering with API tags  https://review.opendev.org/69632415:22
openstackgerritMichael Johnson proposed openstack/octavia stable/stein: Fix filtering with API tags  https://review.opendev.org/69632515:22
*** ricolin_ has joined #openstack-lbaas15:26
openstackgerritMerged openstack/octavia stable/queens: loadbalancer vip-network-id IP availability check  https://review.opendev.org/68588315:26
*** armax has joined #openstack-lbaas15:26
*** ricolin has quit IRC15:28
*** ricolin_ is now known as ricolin15:29
*** ricolin has quit IRC15:34
*** mloza has joined #openstack-lbaas15:42
*** ricolin has joined #openstack-lbaas15:43
*** gcheresh has quit IRC15:56
*** ataraday_ has joined #openstack-lbaas15:56
rm_work#startmeeting Octavia16:00
openstackMeeting started Wed Nov 27 16:00:29 2019 UTC and is due to finish in 60 minutes.  The chair is rm_work. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Octavia)"16:00
openstackThe meeting name has been set to 'octavia'16:00
rm_worko/16:00
ataraday_hi16:00
haleybhi16:01
johnsomo/16:01
rm_work#topic Announcements16:01
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:01
rm_workGates work again! woooooo16:01
cgoncalveshi16:02
gthiemongehi16:02
johnsomWith lots of bubble gum and bailing wire16:02
rm_work>_>16:02
rm_workAny other announcements?16:02
johnsomI will be un-available the next two days16:03
johnsomTravelling for the US holiday.16:03
rm_workah yeah, we're right before a major US holiday, so probably a lot of folks will disappear until next week16:03
rm_workthough we have plenty non-US folks on this team, so it won't be a complete ghost-town here :D16:04
haleyb<poof>16:04
rm_workok16:05
rm_work#topic Brief progress reports / bugs needing review16:05
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:05
rm_workI'm still working on AZ stuff -- this is absolutely imperative for me to get working+merged in basically the next 2 weeks ... so, I'm fully focused on that.16:07
ataraday_As usual want to highlight https://review.opendev.org/#/c/662791/16:07
ataraday_#link https://review.opendev.org/#/c/662791/16:07
rm_workOnce I get that wrapped up, my next focus will be ataraday_'s patches, I absolutely promise16:07
rm_workI hope we might even get a number of those merged before end of year finally? :D16:07
ataraday_rm_work, great, thanks!16:08
*** ivve has quit IRC16:08
ataraday_I hope too ^)16:08
rm_workand this would just be a neat thing to get in soon (and is a danger for lots of rebasing as it touches everywhere): https://review.opendev.org/#/c/691135/16:09
johnsomI have been playing wack-a-mole with gate issues (I think we were up to six).16:09
rm_workand so, this is the next AZ patch (and i think it is ready to merge):16:10
rm_work#link https://review.opendev.org/#/c/693765/16:10
rm_workrechecking now that the gate is better16:10
johnsomIn between still working on the failover refactor. I'm up to the amphora VIP base port work. I have started the re-factor of the network driver to be more straight forward.16:10
haleybjohnsom: what do you think the plan should be with the grenade job?  you said you filed a bug yesterday?  i can track this mess16:11
johnsomI.e. I created a "create port" and "delete" port method instead of the "create port and do twelve other things across the compute driver as well" methods we have. This will improve our ability to retry, etc.16:11
johnsomhaleyb The open but is here: https://bugs.launchpad.net/grenade/+bug/185409316:11
openstackLaunchpad bug 1854093 in grenade "grenade fails due to PyYAML conflict" [Undecided,New]16:11
rm_workcool that would be good haleyb16:12
openstackgerritBrian Haley proposed openstack/octavia master: Stop testing python 2  https://review.opendev.org/68737016:12
rm_worknice to have someone explicitly track this stuff to get it moving forward16:12
haleybi also have that stuff ^^16:12
rm_workyeah16:12
johnsomThere was some discussion yesterday in the qa channel, but no one has signed up to fix it. Basically devstack-gate.sh is installing the python(3)-yaml package instead of using pip, which breaks various other projects, which break us16:12
rm_worklooking forward to that. we definitely need to just do it in stages, simply turn off the tests in stage1, then get more deep into removing py2 after that16:13
johnsom+116:13
haleybrm_work: yes, i started doing six removal, it's quite invasive16:13
johnsomYeah, I think it would be best to have the six work in a different patch from just dropping the py27 jobs.16:15
haleybyes, i removed it all from the patch ^^ too, it just removes the jobs16:16
johnsom+116:16
rm_workhaleyb: reviewed, fix the zuul syntax error really quick if you would16:18
rm_workok so, anything else for patches?16:18
rm_workmaybe we can move on16:18
rm_work#topic Open Discussion16:18
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:18
johnsomI want to give a shout out to eandersson for finding a memory leak in openstacksdk16:19
rm_workyeah, i think that patch is super close now16:19
johnsom#link https://review.opendev.org/#/c/695795/16:19
rm_worknoice16:19
johnsomThis may have cause the octavia-dashboard to leak memory as we use openstacksdk and this was found in the connection setup code.16:19
johnsomSo, an FYI that if you are seeing a memory leak in horizon and running octavia-dashboard, this is likely the fix.16:20
rm_workyeah i bet this hit us16:21
rm_workoctavia dashboard suddenly stopped working last week16:21
rm_workand i was like ... wut16:21
rm_workbut hadn't looked into it yet16:21
johnsomAny other topics today?16:26
rm_workguess not?16:27
rm_workwe can call it16:27
rm_workthanks for making it out again folks!16:27
rm_work#endmeeting16:27
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews"16:27
openstackMeeting ended Wed Nov 27 16:27:49 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:27
openstackMinutes:        http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-11-27-16.00.html16:27
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-11-27-16.00.txt16:27
openstackLog:            http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-11-27-16.00.log.html16:27
*** mloza has quit IRC16:30
openstackgerritAdam Harwell proposed openstack/octavia master: Stop testing python 2  https://review.opendev.org/68737016:32
*** mloza has joined #openstack-lbaas16:36
openstackgerritBrian Haley proposed openstack/octavia master: Stop testing python 2  https://review.opendev.org/68737016:38
openstackgerritBrian Haley proposed openstack/octavia master: Stop testing python 2  https://review.opendev.org/68737016:48
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Add a mixed HTTP and HTTPS scenario test  https://review.opendev.org/69635816:51
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: [WIP] Test the Availability Zone [Profile] admin API  https://review.opendev.org/69529116:57
rm_workgates are slammed it seems17:05
johnsomYeah, we probably unblocked a bunch of people, lol17:12
johnsomgthiemonge So, https://zuul.opendev.org/t/openstack/build/b224f14e5afa4a4b96100695396e3eb4 the queens run here, this is the current open issue with the queens branch that I have a proposed fix for here: https://review.opendev.org/68353717:34
gthiemongejohnsom: yep, when I see "provisioning_status updated to an invalid state of ERROR", I recheck :D17:35
johnsomWell, that will not likely pass until we get that other fix it. It's the 500 and the error in o-cw that call out that issue17:36
rm_workwell, your patch keeps failing for one reason or another T_T17:37
johnsomLooking at the rocky job now17:37
rm_workalready gonna need another recheck17:37
gthiemongejohnsom: https://zuul.opendev.org/t/openstack/build/544573e47a184736b1b5d2651c77f0ac and https://zuul.opendev.org/t/openstack/build/6d4a8f4b84b242ddaf8f2a4cd20dc6e217:38
johnsomrm_work Yeah, joy, nova: Server 9a04e796-1e7d-41a8-8ba2-481a8b227495 failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: BUILD. Current task state: spawning.17:38
rm_work:/17:38
openstackgerritAdam Harwell proposed openstack/octavia stable/queens: Flush the eth1 default route inside the netns  https://review.opendev.org/68353717:39
rm_workjust rebased it to rekick17:39
johnsomThanks17:39
johnsomThanks for the rechecks this morning. I started in and saw you were right before me... lol17:39
rm_worklol yeah17:43
gthiemongejohnsom: tomorrow I will waste some CI cycles to find out which one of the 2 commits introduced the issue (or if it is a random issue that happens from time to time)17:45
johnsomYeah, it looks like it was the VM patch, which is really hard to read with those changes. Still looking at it17:45
gthiemongejohnsom: if the option is not enabled (and it's not enabled in CI jobs), the commit only refactors the http*/udp request functions17:48
johnsomYeah, I see that. The curl stuff is an issue, but not running in this rocky run17:49
*** mloza has quit IRC17:50
gthiemongeanyway thanks for your time johnsom!17:52
openstackgerritAnn Taraday proposed openstack/octavia master: Convert listener flows to use provider models  https://review.opendev.org/66023617:52
openstackgerritAnn Taraday proposed openstack/octavia master: Convert pool flows to use dicts  https://review.opendev.org/66538117:52
openstackgerritAnn Taraday proposed openstack/octavia master: Transition member flows to use dicts  https://review.opendev.org/65784217:52
openstackgerritAnn Taraday proposed openstack/octavia master: Transition amphora flows to dicts  https://review.opendev.org/66889817:52
openstackgerritAnn Taraday proposed openstack/octavia master: Convert Lb flows to use provider dicts  https://review.opendev.org/67172517:52
openstackgerritAnn Taraday proposed openstack/octavia master: Convert health_monitor flows to use provider models  https://review.opendev.org/67977917:52
openstackgerritAnn Taraday proposed openstack/octavia master: Transition l7policy flows to dicts  https://review.opendev.org/66597717:52
openstackgerritAnn Taraday proposed openstack/octavia master: Transition l7rule flows to dicts  https://review.opendev.org/66817317:52
openstackgerritAnn Taraday proposed openstack/octavia master: Jobboard based controller  https://review.opendev.org/64740617:52
*** tesseract has quit IRC17:53
*** bcafarel has quit IRC17:57
*** mloza has joined #openstack-lbaas18:00
*** servagem has quit IRC18:10
*** bcafarel has joined #openstack-lbaas18:11
*** servagem has joined #openstack-lbaas18:12
*** rpittau is now known as rpittau|afk18:14
*** ricolin has quit IRC18:16
johnsomgthiemonge My guess is that neutron didn't have the IPv6 path setup and that IPv6 member was still operational_status DOWN as the first request was redirected to the IPv4 member. You could trace back my looking in the HM log to see when that IPv6 member became operational and then dig through the neutron log. With the extra overhead of this VM patch and the fact it was a RAX host, I'm not surprised some actions18:17
johnsomare taking longer than they normally would.18:17
rm_workalso, do the clients not support setting a real proxy>18:21
rm_work?18:21
johnsomLike an HTTP or SOCKs proxy? They should18:21
johnsomThough I ague this patch is not using tempest as it was intended to be used, as an end-user perspective test suite.18:22
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: [WIP] Test the Availability Zone [Profile] admin API  https://review.opendev.org/69529118:26
openstackgerritAdam Harwell proposed openstack/octavia master: Fix update API when barbican secret is missing  https://review.opendev.org/69198718:32
gthiemongejohnsom: could we first send some requests to check that both members are replying correctly (don't check reply counters), then send 20 requests and check that those 20 requests are evenly dispatched?18:34
johnsomYeah, or just check the operating status first. I think we do that in some other tests. Though it would still be good to RCA why it isn't coming up right away.18:36
rm_workuhh don't we already have a "wait for some replies to start working" step?18:40
rm_worki swear we did18:40
johnsomYeah, we do, but it waits for any response18:40
rm_workah so maybe just one member18:41
johnsomThis test however is not waiting for the operating status of both members to go ONLINE, it only looks at the LB going ACTIVE18:41
rm_workthough that could be waiting a long time if the LB is actually broken somehow and has only one member :D18:41
rm_workyeah maybe we should do that18:41
johnsomYeah, that is what the test saw, the first member, IPv4, got one extra hit.'18:41
openstackgerritMerged openstack/octavia stable/stein: Fix filtering with API tags  https://review.opendev.org/69632519:30
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Add a mixed HTTP and HTTPS scenario test  https://review.opendev.org/69635819:51
*** gcheresh has joined #openstack-lbaas19:55
johnsomhttps://www.irccloud.com/pastebin/2lnoW2wj/20:05
*** haleyb is now known as haleyb|away20:06
openstackgerritMerged openstack/octavia stable/train: Fix filtering with API tags  https://review.opendev.org/69632420:09
johnsomThat is in haproxy 2.120:11
openstackgerritMerged openstack/octavia master: Remove unnecessary interface configurations  https://review.opendev.org/58077820:32
openstackgerritMerged openstack/octavia-tempest-plugin master: Add TLS SNI scenario tests  https://review.opendev.org/69077820:32
*** maciejjozefczyk has quit IRC20:33
*** bcafarel has quit IRC20:55
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: DNM checking scenarios on all branches  https://review.opendev.org/69640721:07
*** ccamposr__ has joined #openstack-lbaas21:35
*** ccamposr has quit IRC21:38
*** gcheresh has quit IRC21:47
*** rcernin has joined #openstack-lbaas21:56
*** tobias-urdin has left #openstack-lbaas22:05
*** pcaruana has quit IRC22:06
openstackgerritSam Morrison proposed openstack/octavia-tempest-plugin master: Add new scenario test to create LB in specific AZ  https://review.opendev.org/69534922:22
sorrisonrm_work: you around?22:24
openstackgerritMerged openstack/octavia-tempest-plugin master: Handle NotImplemented flavor exception  https://review.opendev.org/67613523:23
*** elico has joined #openstack-lbaas23:38
*** goldyfruit_ has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!