Wednesday, 2020-04-15

« Tuesday, 2020-04-14 Index Thursday, 2020-04-16 »
*** threestrands has joined #openstack-lbaas00:00
*** sapd1 has joined #openstack-lbaas01:30
*** sapd1 has quit IRC01:36
*** sapd1 has joined #openstack-lbaas01:49
*** ramishra has quit IRC03:12
*** psachin has joined #openstack-lbaas03:33
*** ramishra has joined #openstack-lbaas03:35
*** KeithMnemonic has quit IRC04:30
dawzonWould it make more sense for a cipher blacklist to be a colon-separated string or a ListOpt?  Since it's not going to be consumed by OpenSSL it doesn't necessarily have to be formatted the same04:41
*** vishalmanchanda has joined #openstack-lbaas05:40
*** sapd1 has quit IRC05:41
*** tobberydberg has joined #openstack-lbaas05:45
*** threestrands has quit IRC05:54
*** sapd1 has joined #openstack-lbaas05:55
*** ccamposr has joined #openstack-lbaas06:40
*** ccamposr__ has quit IRC06:42
*** gcheresh has joined #openstack-lbaas06:53
*** ataraday_ has joined #openstack-lbaas06:55
*** maciejjozefczyk has joined #openstack-lbaas07:06
*** ccamposr__ has joined #openstack-lbaas07:26
*** rpittau|afk is now known as rpittau07:28
*** ccamposr has quit IRC07:29
*** born2bake has joined #openstack-lbaas07:50
*** tkajinam has quit IRC08:21
*** sapd1 has quit IRC09:04
*** sapd1 has joined #openstack-lbaas09:16
*** sapd1 has quit IRC09:21
rm_workdawzon: i think the idea was that it'd be easiest to allow users to just copy/paste the most common format for the lists without having to reformat them to be separated manually to fit some custom format we define09:23
rm_workso it's not so much that they're going to be CONSUMED by openssl, as much as they are *provided by* openssl09:23
*** gcheresh has quit IRC09:33
openstackgerritMerged openstack/octavia-tempest-plugin master: Update hacking for Python3  https://review.opendev.org/71565309:36
*** rpittau is now known as rpittau|bbl10:23
*** sapd1 has joined #openstack-lbaas10:33
*** sapd1 has quit IRC10:37
*** sapd1 has joined #openstack-lbaas10:38
*** sapd1 has quit IRC10:45
*** sapd1 has joined #openstack-lbaas10:50
openstackgerritAdam Harwell proposed openstack/octavia master: Refactor the failover flows  https://review.opendev.org/70531710:50
openstackgerritAdam Harwell proposed openstack/octavia master: Use routed network filter if it exists  https://review.opendev.org/70615310:54
*** sapd1 has quit IRC10:55
*** sapd1 has joined #openstack-lbaas10:55
*** sapd1 has quit IRC11:01
*** sapd1 has joined #openstack-lbaas11:15
*** sapd1 has quit IRC11:37
*** kevinz has joined #openstack-lbaas11:40
kevinzHi Octavia, I have a problem after setting o-hm0 in the network nodes. The DNS(/etc/resolv.conf) will always be set to domain openstacklocal11:42
kevinzsearch openstacklocal11:42
kevinznameserver 192.168.100.311:42
kevinznameserver 192.168.100.211:42
kevinzso that the network host are always lost dns capability. I wonder do we have a method to avoid the dns over write in the network host?11:43
*** servagem has joined #openstack-lbaas11:46
cgoncalveskevinz, hi. your network has DHCP servers enabled with nameservers and you set up o-hm0 to be configured via DHCP. you can either 1) disable nameservers in that subnet (openstack subnet set --no-dns-nameservers), 2) make your DHCP client not to request for DNS servers (example: https://github.com/openstack/octavia/blob/master/etc/dhcp/dhclient.conf) or 3) configure o-hm0 statically11:54
*** sapd1 has joined #openstack-lbaas11:54
kevinzcgoncalves: Thanks for quick response. I will try the method! Should be really good11:56
cgoncalveskevinz, no problem! good luck11:57
kevinzcgoncalves:  :-D11:59
*** gcheresh has joined #openstack-lbaas12:27
*** rpittau|bbl is now known as rpittau12:49
*** KeithMnemonic has joined #openstack-lbaas12:54
*** TrevorV has joined #openstack-lbaas13:37
*** rcernin has quit IRC13:37
*** gcheresh has quit IRC13:49
*** ataraday_ has quit IRC14:01
openstackgerritBrian Haley proposed openstack/octavia master: Don't inherit enforcing bash errexit in devstack plugin  https://review.opendev.org/72004114:13
*** gcheresh has joined #openstack-lbaas14:38
*** tkajinam has joined #openstack-lbaas14:43
*** sapd1 has quit IRC14:52
*** sapd1 has joined #openstack-lbaas14:53
*** ianychoi has quit IRC15:13
*** gcheresh has quit IRC15:20
*** ataraday_ has joined #openstack-lbaas15:33
mnaserhmm15:36
mnaserhave y'all seen an issue in train where the amphora responds with a 500 when the certificate rotation happens?15:36
mnaserit's been destroying an environment that i'm working with, where floods of 500s come in all at once as it rotates one-by-one and fails to do that one-by-one and they all slowly but surely hit error state15:37
johnsommnaser Yeah, I think rm_work just found/fixed that. It's a python3 amphora issue.15:38
rm_workmnaser: yep patch is up15:38
johnsommnaser https://review.opendev.org/#/c/719922/15:38
mnaserjohnsom: i didnt see any changes in here -- https://github.com/openstack/octavia/blob/b0c2cd7b4c835c391cfedf12cf9f9ff8a0aabd17/octavia/amphorae/backends/agent/api_server/certificate_update.py15:39
rm_workwe can't merge anything tho because of gate failures15:39
mnaserok so not landed yet15:39
rm_worki just had most of my fleet explode this week15:39
rm_workbecause of that <_<15:39
mnaserrm_work: tough one because you dont have a vm leftover15:40
rm_workah, they don't go away15:40
rm_workit just puts the amp in ERROR15:41
rm_workand breaks our communication with it15:41
rm_workbut it doesn't get recycled because healthchecks still pass15:41
rm_workwas actually remarkably simple to debug :D15:41
mnaserah shucks, i haven't dove that much into it15:41
mnaserthe hard/annoying part is this will require rotating all amphoras..15:42
rm_workyep15:42
rm_worki've rotated out my whole fleet ... several times15:42
rm_workah, though be VERY careful15:42
rm_workthe current failover mechanism will have a hard time15:42
johnsomataraday_ Hi. I think we have a problem with the jobboard patch. It seems that the conductors are getting started even if the amphorav2 driver isn't being used. This conductor startup causes taskflow to go out and try to access redis (both the python module and the DB).15:43
rm_workbecause it won't be able to connect to the other amp when it tries to fail the first one15:43
rm_workand you'll get WEIRD and BAD results15:43
rm_workyou will want to apply the patch for the failover rewrite15:43
rm_workhttps://review.opendev.org/#/c/705317/15:43
rm_workthen failover can handle this case15:43
rm_workotherwise you will be in for not fun times15:43
mnaserrm_work: so need to land both that + the py3 fix to be able to get out of this easily?15:44
mnaserseems like a big patch to cherry-pick in a prod environment :X15:44
rm_workI did it! :D15:45
rm_workof course it also broke my UDP LBs15:45
rm_workbut i fixed that :)15:45
johnsomYeah, there are two bugs we are working on in it too15:45
mnaserit sounds like this opens up a rabbithole15:46
rm_workyeah it's not a GOOD situation15:46
rm_worki also haven't slept much in the past few days <_<15:46
mnaserrm_work: what if we ssh'd into teh amphoras and manually put in a generated key15:47
mnaserso that they started responding again?15:47
rm_workmight work15:47
rm_worki don't actually know how to generate the correct pem15:47
rm_workbut if you can figure that out it would fix it :)15:47
rm_workthough doing that with ... so many... <_<15:47
*** ianychoi has joined #openstack-lbaas15:47
ataraday_johnsom, hi15:47
mnaseryeah but you can automate the ssh in/out of that15:48
rm_workso what you CAN do, is apply the patch to one worker, shut down the other ones temporarily, and try a failover or two to see if they work15:48
rm_workthen apply it more broadly if everything seems ok15:48
rm_workthat is what i did15:48
*** sapd1 has quit IRC15:49
ataraday_johnsom, you got this on your devstack or on some change?15:49
*** sapd1 has joined #openstack-lbaas15:49
mnaserrm_work: can you force a cert rotate in octavia?15:50
mnaseroh nevermind it wouldn't even work because you still need to generate the cert yourself15:50
rm_workyes you can15:50
rm_workbut15:50
rm_workyeah it would just fail again :D15:50
mnaserwell i mean you can ssh into the amphora, update with manual cert, add that 2 line patch for flask, force rotation and confrim it doesnt break?15:51
rm_workyes15:51
rm_worki did that15:51
rm_workwhen i wrote the patch15:51
mnaserrm_work: could you also be really and "borrow" the working key from another amphora...?15:51
mnasers/really/really bad/15:51
rm_worki don't think so because those certs are keyed to the amp's ID15:51
rm_workwe validate that on communication15:51
mnaserah ok15:52
johnsomataraday_ Kong and I have both seen it on our devstacks. I think one issue is the devstack plugin.sh is using "=" instead of "==", but I think we also need to consider not always starting the conductors.15:52
mnaserrm_work: how can i force a rotate btw?15:53
ataraday_johnsom, omg, how gate do not get this15:53
*** tkajinam has quit IRC15:53
johnsomataraday_ Yeah, I have not had time to figure that one out....15:54
rm_workmnaser: in the DB, set the cert_busy to 0 and the cert_expiry time to ... like, now15:54
rm_workor NOW() :D15:54
rm_workon the amphora table record15:55
johnsomataraday_ FYI, a paste from Kong: http://dpaste.com/36KYZ9B15:55
rm_worksorry, it's not an API call :/15:55
ataraday_johnsom, I can push a quick patch that will fix plugin.sh and run conductor conditionally15:55
ataraday_if nothing proposed...15:55
johnsomataraday_ Thank you. Yeah, we have been working other recent bugs, so there is no real proposed patch.15:56
mnaserrm_work: ok cool thanks15:57
rm_work#startmeeting Octavia16:01
openstackMeeting started Wed Apr 15 16:01:17 2020 UTC and is due to finish in 60 minutes.  The chair is rm_work. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
*** openstack changes topic to " (Meeting topic: Octavia)"16:01
openstackThe meeting name has been set to 'octavia'16:01
rm_work#chair johnsom16:01
openstackCurrent chairs: johnsom rm_work16:01
rm_work#chair cgoncalves16:01
openstackCurrent chairs: cgoncalves johnsom rm_work16:01
cgoncalveshi16:01
johnsomo/16:01
gthiemongeHi16:01
haleybhi16:01
ataraday_hi16:01
rm_workhey all!16:02
rm_work#topic Announcements16:03
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"16:03
rm_workUssuri RC1 is next week!16:03
* rm_work actually read the agenda page this time16:03
johnsomYeah, we need to get bug fixes in!16:03
rm_workso, we need to work extra hard to review a number of outstanding patches that are very important to land16:04
rm_workanything else? we'll discuss those in more detail later16:04
*** gcheresh has joined #openstack-lbaas16:06
johnsomWe should probably mention that a sqlalchemy release is causing functional test failures.16:06
johnsomJust as an FYI. 1.3.16 introduced the issue.16:06
johnsomI think that is all I have.16:07
rm_work#topic Brief progress reports / bugs needing review16:07
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"16:07
johnsomI have been focused on bug fixing, reviews, and releases this week.16:08
rm_workI have a couple of bugs that I'd like to land fixes for -- ranging from super urgent to trivial16:09
*** maciejjozefczyk has quit IRC16:09
rm_worksuper urgent:16:09
rm_work#link https://review.opendev.org/71992216:09
rm_worktotally trivial:16:09
rm_work#link https://review.opendev.org/71992116:09
rm_workalready merging: https://review.opendev.org/71888116:09
rm_work(once gate is fixed)16:09
rm_workand .... technically a feature but I vote to allow it to merge anyway for rc1: https://review.opendev.org/58918016:10
cgoncalvesrm_work, on that last change, I commented but did not vote. probably I should have voted to get your attention, sorry16:11
rm_workah k16:11
gthiemongeI have some octavia-dashboard backports that are waiting for reviews: https://review.opendev.org/#/q/I3668f3dc5d3eb6a288994386294cc018035540c8+is:open16:11
openstackgerritAnn Taraday proposed openstack/octavia master: Run taskflow jobboard conductor conditionally  https://review.opendev.org/72023716:13
rm_workcommented coreycb16:13
rm_workerr16:13
rm_workcgoncalves:16:13
rm_workcgoncalves: look at the newest code, the retry/delay all moved out to be at a higher level so it's common16:15
rm_worki'll fix the date thing16:15
cgoncalvesrm_work, yep, I suspect that was the fix. thanks16:16
johnsomI would say we should have a priority list, but I think the etherpad updates broke our tracking etherpad16:17
*** psachin has quit IRC16:17
rm_work:(16:17
cgoncalvesetherpad is just slow16:18
openstackgerritAdam Harwell proposed openstack/octavia master: Support HTTP and TCP checks in UDP healthmonitor  https://review.opendev.org/58918016:18
johnsomThere is discussion that the update broken it in the opendev channel16:18
johnsombroke, oye16:18
johnsomIt doesn't load at all for me16:19
cgoncalvesoh, ok. I could open it a few minutes ago16:19
cgoncalvesyeah, timed out now16:19
cgoncalvesrm_work, we are in feature freeze. if you want the UDP health monitor patch in, it probably needs a FFE16:20
cgoncalves(Feature Freeze Exception)16:20
rm_workand who grants that?16:22
rm_workPTL?16:22
* rm_work looks at himself16:22
rm_workwe lost our last day of merging to gate issues and 75% of our cores being on holiday :D16:23
cgoncalvesappears to reading https://docs.openstack.org/project-team-guide/release-management.html16:23
rm_workcool16:23
rm_worki hereby grant a FFE16:23
*** gcheresh has quit IRC16:24
rm_worknow that we've settled that... anything else before we move to the next topic?16:24
cgoncalvesit would be wise in this specific case to coordinate with other cores16:24
rm_workI mean, the other cores are necessarily involved as they have to +2/+A :D so16:24
rm_workthat's kinda implied16:24
rm_workI grant the exception, if you agree you vote16:25
cgoncalvesok16:25
rm_workgranting the FFE does nothing if you don't merge it ^_^16:25
rm_workcool, so next:16:26
rm_work#topic Open Discussion16:26
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"16:26
rm_workanything?16:27
cgoncalvesyes16:27
johnsomThere is an agenda item still16:27
rm_workah there it is16:28
cgoncalvesI added this to the agenda: How to handle https://review.opendev.org/#/c/711275/ (see Carlos' comments in Gerrit)16:28
cgoncalvesso this ^ patch depends on an octavia-lib patch that is available in 1.5.0+16:28
cgoncalvestrain has octavia-lib==1.4.0 in upper-constraints.txt16:29
cgoncalvesso I am not sure how we could backport the octavia-lib patch to stable/train, release a new version of octavia-lib and have it available in train for octavia to consume16:30
cgoncalveshttps://github.com/openstack/requirements/blob/stable/train/upper-constraints.txt#L14116:30
johnsomYeah, umm, You can't backport a new positional parameter to a method in octaiva-lib. That could break drivers16:31
cgoncalvesvery good point16:31
rm_workyeah we may just be stuck16:31
cgoncalves+116:32
cgoncalvesin any case, how would we backport bug fixes to octavia-lib stable branches and have them released if u-c.txt has it set to ==1.4.0?16:33
*** rpittau is now known as rpittau|afk16:33
johnsomYeah, it would require a UC bump16:34
*** maciejjozefczyk has joined #openstack-lbaas16:35
cgoncalvesproblem could we might have to bump more than once :/16:36
cgoncalvesif there's no other way, so be it16:36
*** ataraday_ has quit IRC16:37
rm_workyeah i'm not totally sure i understand what the issue is, i'd have to try doing it maybe16:38
johnsomWe have some strange relationships with octavia-lib versions I sure would like to figure out a better way to handle....16:39
*** maciejjozefczyk has quit IRC16:41
cgoncalvestrain neutron-lib: https://review.opendev.org/#/c/679986/16:43
cgoncalvesothers also bumping versions in stable branches u-c.txt: https://review.opendev.org/#/q/project:openstack/requirements+NOT+branch:master16:46
johnsomYeah, you pretty much have to be able to do so for bug fixes16:47
johnsomSo, another topic we might want to discuss: Add skip to the tests for sqlalchemy.16:47
cgoncalvesyeah, ok. I just wanted to make sure it is possible and how to handle it16:47
johnsomI would propose we do so as we know the issue is limited to the tests (sqlite) to unblock the gates16:48
*** generalfuzz is now known as headphoneJames16:49
cgoncalvessounds reasonable16:49
rm_workok16:49
johnsomOk, I will propose one16:49
xgermanWonder if we should do a virtual video happy hour?16:51
rm_worki'm down at some point16:53
rm_worktho my schedule is lulz :P16:53
cgoncalves+1. xgerman brings the beer16:53
rm_workwe all bring the beer :D16:54
xgerman:beer16:54
rm_workfor ourselves. because we are all in quarantine.16:54
xgermanI have all kind of slack emoji beers16:54
xgermanWe can use poll to figure out the times…16:56
johnsomSure, sounds good.16:56
johnsomIt also sounds like the PTG will be virtual, so maybe we will have good participation there as well.16:56
*** gcheresh has joined #openstack-lbaas16:56
xgermanUh, oh, no trip to Vancouber :-(16:57
rm_work:(16:57
rm_workyeah i was really looking forward to that16:57
cgoncalvessuch a cool city16:57
rm_workok well thanks everyone, meeting time about up16:58
johnsom#link https://www.openstack.org/events/opendev-ptg-2020/16:58
johnsomo/16:59
johnsom#endmeeting17:00
rm_work#endmeeting17:00
*** openstack changes topic to "Discussions for OpenStack Octavia | Priority bug review list: https://etherpad.openstack.org/p/octavia-priority-reviews"17:00
openstackMeeting ended Wed Apr 15 17:00:40 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-04-15-16.01.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-04-15-16.01.txt17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/octavia/2020/octavia.2020-04-15-16.01.log.html17:00
openstackgerritMichael Johnson proposed openstack/octavia master: Disable two tests due to sqlalchemy/sqlite bug  https://review.opendev.org/72024417:08
johnsomrm_work Test skips ^^^17:08
*** gcheresh has quit IRC17:12
rm_workkk17:19
*** sapd1 has quit IRC17:30
cgoncalvesunsolicited newsletter from HPE. me: I want to unsubscribe from all communications, website: no problem! just please provide your first and last name17:36
johnsomlol, just the name.17:36
cgoncalvesI'm now called "A B"17:36
johnsomThey used to have a really formal system for handling that, but I bet it went with the other half of the company.17:37
rm_worklol yeah I am "A B" in a lot of places also :D17:48
rm_workand often use a@b.com17:48
xgermanI a, test@example.com...17:53
xgermanI am17:53
rm_workok so, once that patch for test skips merges, we can recheck the conditional conductors patch, and then we can recheck the udp template fix patch, and THEN we can start looking at the other ones17:56
rm_workactually rechecks should be fine for anything after the test-skips merge?17:57
rm_workthe conductor thing wasn't actually failing the gates IIRC?17:57
cgoncalvesanother boring day in octavialand17:57
johnsomYeah, it looks like the etherpad is back, it would be nice to update the list (I just did a few I knew) and have something to work against.17:58
* johnsom has to disappear for another meeting17:58
*** hongbin has joined #openstack-lbaas19:01
*** osmanlicilegi has quit IRC19:45
*** vishalmanchanda has quit IRC19:54
openstackgerritMerged openstack/octavia master: Disable two tests due to sqlalchemy/sqlite bug  https://review.opendev.org/72024420:04
*** gcheresh has joined #openstack-lbaas20:07
openstackgerritAdam Harwell proposed openstack/octavia master: Healthmanager opts aren't CLI-related  https://review.opendev.org/71992120:10
rm_workk lots of rechecks going20:11
*** servagem has quit IRC20:34
*** gcheresh has quit IRC20:47
*** gcheresh has joined #openstack-lbaas20:59
*** gcheresh has quit IRC21:52
*** TrevorV has quit IRC22:18
*** born2bake has quit IRC22:41
*** rcernin has joined #openstack-lbaas22:45
*** hongbin has quit IRC22:53
*** tkajinam has joined #openstack-lbaas22:59
*** hongbin has joined #openstack-lbaas23:01
« Tuesday, 2020-04-14 Index Thursday, 2020-04-16 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!