Thursday, 2020-06-04

« Wednesday, 2020-06-03 Index Friday, 2020-06-05 »
*** armax has quit IRC00:31
*** spatel has quit IRC00:43
*** TMM has quit IRC00:52
*** TMM has joined #openstack-lbaas00:52
JayLiusorry I01:16
JayLiusorry I missed the ptg session, what's new about active/active? Plz let me know01:16
JayLiuWe discussed with neutron people and they think it matters to add ecmp for octavia. Additionally, ovn can support ecmp too.01:17
JayLiuSo...I think it maybe feasible to use L3 ecmp as distributor?01:18
*** rcernin has quit IRC01:27
JayLiuAbout session persistence, I am sure ovs can use ip hash to complement it, but I don't know whether ovn can do this.01:30
*** rcernin has joined #openstack-lbaas01:30
*** also_stingrayza has joined #openstack-lbaas01:49
*** stingrayza has quit IRC01:52
*** armax has joined #openstack-lbaas01:54
*** rcernin has quit IRC02:37
*** armax has quit IRC02:48
*** sapd1 has joined #openstack-lbaas03:11
*** rcernin has joined #openstack-lbaas03:31
*** rcernin has joined #openstack-lbaas03:37
rm_workjohnsom: wait did we really? damnit. I probably even agreed before didn't i04:15
rm_workMaybe I just need convincing again04:15
*** sapd1 has quit IRC04:22
*** sapd1 has joined #openstack-lbaas04:37
*** wuchunyang has joined #openstack-lbaas04:39
*** gcheresh_ has joined #openstack-lbaas04:47
*** gcheresh_ has quit IRC04:53
*** gcheresh_ has joined #openstack-lbaas04:55
*** gcheresh_ has quit IRC05:02
*** wuchunyang has quit IRC05:07
johnsomrm_work All I am going to say is: yep05:18
rm_workdo you think my point is not valid?05:19
rm_workwe've got this thing that's just no longer useful -- so sure, don't update it -- but maybe we need to actually remove it05:19
rm_workthat can be in a followup, but we need to agree to do it05:19
johnsomOh, I agree it needs to go away. It's just deciding on timing. It's only there for backward compat with *old* amphora05:20
johnsomsplit config should go away....05:20
rm_workk05:20
johnsomWhen is the question I don't know really05:21
johnsomI haven't looked at that in a while.05:21
johnsomIt's just super odd to add new feature support to a deprecated amp image05:21
johnsomBut it sucks that it will also just ignore that config if the amp image is too old.05:22
johnsomhttps://wiki.openstack.org/wiki/Octavia/Non-arbitrary_Decisions#One_haproxy_process_per_listener.2C_or_one_haproxy_process_per_loadbalancer.3F_.28Rescinded.2905:23
johnsomJust to reflect on the decisions that led us here. lol05:24
johnsomOk, it has been many hours all week for me, I need some sleep. Catch you tomorrow.05:24
johnsomJayLiu Hi, sorry we missed you at the PTG. Yes, I think it is feasible. Your specifications are at the top of my review list. I hope to get you some comments this week.05:25
openstackgerritAdam Harwell proposed openstack/octavia master: Add minimum TLS version option in octavia.conf  https://review.opendev.org/72212105:27
*** oklhost has quit IRC05:35
*** oklhost has joined #openstack-lbaas05:39
*** vishalmanchanda has joined #openstack-lbaas05:51
openstackgerritMerged openstack/octavia master: Add TLS version configuration for listeners  https://review.opendev.org/72136206:27
*** gcheresh_ has joined #openstack-lbaas06:32
JayLiu<johnsom> Thx !!!! Very good news for me! I will continue to work on this matter.06:33
*** wuchunyang has joined #openstack-lbaas06:37
openstackgerritMerged openstack/octavia master: Add TLS version configuration for pools  https://review.opendev.org/72177307:04
*** rcernin has quit IRC07:15
cgoncalvesJayLiu, hi! OVN recently implemented consistent-hashing capabilities: https://github.com/ovn-org/ovn/commit/5af304e7478adcf5ac50ed41e96a55bebebff3e807:16
*** maciejjozefczyk_ has joined #openstack-lbaas07:17
*** rcernin has joined #openstack-lbaas07:20
*** maciejjozefczyk has quit IRC07:20
*** rcernin has quit IRC07:21
*** rcernin has joined #openstack-lbaas07:21
openstackgerritAdam Harwell proposed openstack/octavia master: Add minimum TLS version option in octavia.conf  https://review.opendev.org/72212107:22
JayLiu<cgoncalves> It's really useful! :D thx07:26
*** sapd1 has quit IRC07:29
*** ccamposr__ has joined #openstack-lbaas07:35
*** ccamposr has quit IRC07:37
*** rcernin has quit IRC07:41
*** rpittau|afk is now known as rpittau07:50
openstackgerritCarlos Goncalves proposed openstack/octavia master: Add a periodic fast forward upgrade job  https://review.opendev.org/73298607:50
*** ataraday_ has joined #openstack-lbaas08:07
*** ataraday_ has quit IRC08:26
*** also_stingrayza is now known as stingrayza08:32
*** wuchunyang has quit IRC08:39
*** tkajinam has quit IRC08:48
*** kevinz has quit IRC09:57
*** yamamoto has quit IRC09:58
*** rpittau is now known as rpittau|bbl10:02
openstackgerritCarlos Goncalves proposed openstack/octavia master: WIP: zero downtime upgrade  https://review.opendev.org/73326210:05
*** yamamoto has joined #openstack-lbaas10:31
openstackgerritCarlos Goncalves proposed openstack/octavia master: WIP: zero downtime upgrade  https://review.opendev.org/73326210:31
*** stingrayza has quit IRC10:51
*** wuchunyang has joined #openstack-lbaas10:53
openstackgerritCarlos Goncalves proposed openstack/octavia master: WIP: zero downtime upgrade  https://review.opendev.org/73326211:00
*** wuchunyang has quit IRC11:11
*** ccamposr__ has quit IRC11:14
*** yamamoto has quit IRC11:18
*** ccamposr has joined #openstack-lbaas11:20
*** rpittau|bbl is now known as rpittau11:35
*** stingrayza has joined #openstack-lbaas11:48
*** yamamoto has joined #openstack-lbaas11:49
*** yamamoto has quit IRC11:55
*** ianychoi_ has joined #openstack-lbaas12:32
*** ccamposr__ has joined #openstack-lbaas12:32
*** sapd1_x has joined #openstack-lbaas12:34
*** njohnston_ has joined #openstack-lbaas12:35
*** TMM has quit IRC12:35
*** TMM has joined #openstack-lbaas12:35
*** dmellado_ has joined #openstack-lbaas12:35
*** zzzeek has quit IRC12:36
*** tobberydberg has quit IRC12:36
*** dmellado has quit IRC12:36
*** ccamposr has quit IRC12:36
*** njohnston has quit IRC12:36
*** zigo has quit IRC12:36
*** ianychoi has quit IRC12:36
*** numans has quit IRC12:36
*** sapd1_y has quit IRC12:36
*** dmellado_ is now known as dmellado12:37
*** zzzeek has joined #openstack-lbaas12:38
*** tobberydberg has joined #openstack-lbaas12:38
*** zigo_ has joined #openstack-lbaas12:45
*** wuchunyang has joined #openstack-lbaas12:46
openstackgerritCarlos Goncalves proposed openstack/octavia master: WIP: zero downtime upgrade  https://review.opendev.org/73326212:48
*** dosaboy has quit IRC12:50
*** dosaboy has joined #openstack-lbaas12:51
*** spatel has joined #openstack-lbaas12:57
*** spatel has quit IRC13:00
*** tkajinam has joined #openstack-lbaas13:02
*** rpittau is now known as rpittau|brb13:24
*** KeithMnemonic has joined #openstack-lbaas13:46
KeithMnemonicdoes anyone know if there is a way to override the default loadbalancer_topology. For example if my default is ACTIVE_STANDBY (set in the .conf file) and I just want to create one test LB as SINGLE13:47
*** rpittau|brb is now known as rpittau14:01
*** TrevorV has joined #openstack-lbaas14:02
johnsomKeithMnemonic You can do that by created a load balancer flavor that uses SINGLE. Assuming your version has flavors support.14:11
KeithMnemoniclol,  you know it does not ;-)14:23
johnsomrm_work FYI: https://storyboard.openstack.org/#!/story/200758114:47
*** gcheresh_ has quit IRC14:47
rm_workk14:48
*** yamamoto has joined #openstack-lbaas14:55
*** armax has joined #openstack-lbaas15:28
openstackgerritAndreas Jaeger proposed openstack/octavia-dashboard master: Cleanup py27 support  https://review.opendev.org/73364515:34
*** gcheresh_ has joined #openstack-lbaas15:57
*** rcernin has joined #openstack-lbaas16:03
*** tkajinam has quit IRC16:05
*** rcernin has quit IRC16:08
*** numans has joined #openstack-lbaas16:28
rm_workwasn't that the one that is a mismatch between octavia and octavia-lib?16:31
*** rpittau is now known as rpittau|afk16:31
johnsomWhich what?16:32
rm_workthat story16:32
johnsomNo16:33
johnsomI don't think so.16:33
rm_workerr no, looks the same as the one that we had a fix for tho16:33
rm_workdid we never manage to backport that?16:33
rm_workbecause it had a lib change?16:34
johnsomThis one? https://review.opendev.org/#/c/688548/16:35
johnsomrm_work From the traceback the error they are reporting is after it's pulled off the rabbit queue, so past the amphora driver/octavia-lib part16:40
rm_workyeah... jmm16:41
rm_work*hmm16:41
johnsomI'm not sure how the elif new_members would evaluate and it would try to get a pool16:41
johnsomLooking here: https://github.com/openstack/octavia/blob/stable/train/octavia/controller/worker/v1/controller_worker.py#L47216:42
rm_workBatch updating members: old='[]', new='['68f8c438-1023-4fea-9cb9-968c56eed027']', updated='[]'16:42
rm_workso16:42
rm_workthat looks like the issue from the patch...16:42
rm_workwhich did not make it back even to train16:42
johnsomArgh, chrome just crashed again16:44
johnsomSo, I still don't see how it's related. There is a new_member ID, the first thing the CW method does is pull it from the DB. It's like the member record in the DB has no pool?16:44
johnsomNo, actually, it's that the member lookup in the DB returned nothing16:45
rm_worknew_members[0] is a None16:45
rm_workper that traceback16:45
rm_worktiming error?16:45
johnsomYeah, so the DB lookup didn't find the memebr16:45
rm_workDB is using replication i bet16:45
rm_workand they're reading from a slave and not the one writing16:46
rm_workoctavia does not perform well with multi-master16:47
johnsomOh, this might be an API commit vs CW pull issue.....16:47
johnsomWhy we put these in: https://github.com/openstack/octavia/blob/master/octavia/controller/worker/v1/controller_worker.py#L13016:47
rm_workbut it didn't get NoResultFound16:49
rm_workit actually returned an object?16:49
*** dayou has quit IRC16:49
rm_work2020-04-21 14:21:50.096 19 ERROR oslo_messaging.rpc.server pool = new_members[0].pool16:49
rm_work2020-04-21 14:21:50.096 19 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'pool'16:49
rm_workit would have been an indexerror16:49
rm_workoh no, you're messing with me16:49
rm_worknew_members is the struct from the call16:50
rm_worknot from the DB16:50
johnsomno?16:50
rm_workerr wait16:50
rm_workfff which version am i looking at16:50
johnsomBut you are right, I don't know why it didn't raise NoResultFoudn16:51
rm_workyeah ok i don't understand how this is possible16:51
rm_workDB can't return a result row that doesn't have a pool, for a member16:51
rm_workun-possible16:51
johnsomYeah, it's a relation too I think16:52
rm_workyes16:52
johnsomBut the error is new_members[0] == None16:52
johnsomThe pool part doesn't matter here16:52
rm_workright16:52
johnsomIt's a question of how the _member_repo.get for the ID returned [] instead of a member.16:53
johnsomOk, I think I know how/why16:55
rm_workelucidate?16:59
johnsomhttps://github.com/openstack/octavia/blob/stable/train/octavia/db/repositories.py#L13616:59
rm_workROFL WAT16:59
rm_workFML16:59
rm_workthat's not...16:59
rm_workwhat we want I think?17:00
johnsomIt's the difference between first() and one()17:00
johnsomBut now I am really puzzled, so trying to understand more17:00
rm_workbut I feel like we want "if not model: raise NotFoundError"17:00
rm_workbut we'd have to go through and make sure EVERYTHING is set up to properly handle that17:01
johnsomWell, that is basically what one() does17:01
rm_workbut also, again, the real issue here is that the member doesn't exist, which goes back to what i said about multi--master clusters17:01
rm_workyeah17:02
johnsomRight, I *thought* we had switched/handled this at one point. This is why I am so confused right now.17:02
johnsomPlus all of those tenacity wrappers are *useless* which again puzzles me17:02
rm_workyeah17:02
*** dayou has joined #openstack-lbaas17:03
johnsomWell, multi-master or just how our API tier commit happens. We switched such that the request is handed to the driver before we commit the DB transaction, such that if rabbit doesn't accept it, we don't commit the DB and rolls back clean.17:03
rm_workahh17:03
johnsomBut that is why the tenacity wrappers are there17:03
rm_workblegh17:03
rm_workthen yeah we need to fix that to raise properly17:04
johnsomWell, I'm still WHY/HOW did this happen.17:04
rm_workit's been like this FOREVER...17:05
johnsomYeah, so why isn't like every create call broken right now17:05
johnsomOh: https://github.com/openstack/octavia/blob/master/octavia/controller/worker/v1/controller_worker.py#L42517:06
rm_workrofl17:06
rm_workok we need to clean this up17:06
johnsomYeah, ok, that is why. I had to do that as I couldn't change the repo get call to one() as it would break many things that expect None.17:07
rm_workwell, we SHOULD clean up stuff that expects None17:07
johnsomYeah, ok, it's just because batch has a "create" in it and didn't get fixed with the other creates.17:07
rm_workwell if we fix it in there it's backportabl17:07
johnsomYeah, we should be able to fix this in a backport-able way.17:08
johnsomI can take this one17:09
rm_workok, i assume it's quick17:10
johnsomYeah, not too bad I don't think17:10
johnsomUmmmm17:19
johnsomThe v2 code is very different, like I'm not sure how it works17:20
rm_workO_o17:24
johnsomAh, right, v2 moved to not having this problem17:24
johnsomWe started behaving more like a provider driver in v2, so it already has the objects and doesn't need to wait for the DB17:25
openstackgerritMichael Johnson proposed openstack/octavia master: Fix batch member create for v1 amphora driver  https://review.opendev.org/73367017:50
johnsomrm_work ^^^ That should fix it17:50
rm_workyeah looks right17:52
johnsomI guess our test gates are slow enough tempest didn't catch this17:55
*** haleyb has quit IRC18:42
*** dayou has quit IRC18:50
openstackgerritCarlos Goncalves proposed openstack/octavia stable/train: Pin diskimage-builder on Python 2 git installs  https://review.opendev.org/73368119:04
*** dayou has joined #openstack-lbaas19:05
*** vishalmanchanda has quit IRC19:11
openstackgerritCarlos Goncalves proposed openstack/octavia stable/stein: Pin diskimage-builder on git installs  https://review.opendev.org/73368419:14
johnsomHmmm, maybe we should just not use git there...19:18
cgoncalvesgit + py3 should still work19:22
*** gcheresh_ has quit IRC19:23
*** born2bake has joined #openstack-lbaas19:28
born2bakeHi johnsom, I am wondering if you can help me again :) http://paste.openstack.org/show/794384/ Loadbalancer cannot create VM so its spawning forever and then is deleted and loadbalancer is in error state. However, I can create amphora instance manually without any issues.19:35
johnsomborn2bake If you see the instance as ACTIVE in nova, it is likely the lb-mgmt-net is not setup correctly. The controllers (except API) need to be able to reach the amphora lb-mgmt-net interfaces.19:41
born2bakeInstance created by octavia is just spawning and never active...its spawning and then in delete state19:43
johnsomFrom the controller container try to ssh to the lb-mgmt-net address (we don't need ssh, but it is a test). I would guess you can't reach it19:43
johnsomborn2bake Oh! looking closer at that log "Waiting for compute id f3e3ef93-56a9-43ef-a887-b512b088ce1d to go active timeout." That is a problem with nova19:45
johnsomNova failed to even start the VM19:46
johnsomCheck the nova logs or see if you can see the "error" field in the "openstack server show"19:46
johnsomSomeone recently had that (can't remember if it was you or not), but the nova flavor was created in the wrong service account, so nova was failing on a flavor error. Maybe you have the same?19:47
born2bakehttps://seashells.io/v/SnGHQ7ad19:48
born2bakein the logs I can see instance spawned successfully and then its deleted. (nova-compute log)19:49
johnsomYeah, nova logs are a mystery to me sometimes. I can tell you that "go active timeout" means nova never set the instance to ACTIVE inside the timeout.19:51
johnsomIt looks like your timeout was 5 minutes. Which for Nova to go active should be a few seconds unless you have a really strange scheduler19:53
johnsomI would try starting a load balancer, then do "openstack server show <amphora UUID>" a few times and watch the status and error fields in nova. It should go to ACTIVE in a few seconds after it starts the VM process19:54
born2bakeits very weird cause I can create instance within 20 secs and its running fine if I do that just via openstack server create19:59
born2bakeokay, will try to investigate and let you know if I find it20:01
johnsomYeah, it's likely a config issue or some resource was created under the wrong project like a flavor. Maybe if you have a custom sized image the flavor might have a disk size mis-match, etc.20:02
johnsomOk, I will be around for a while so feel free to ping me back20:02
*** rcernin has joined #openstack-lbaas20:05
*** maciejjozefczyk_ has quit IRC20:09
*** rcernin has quit IRC20:10
openstackgerritMichael Johnson proposed openstack/octavia master: Add minimum TLS version option in octavia.conf  https://review.opendev.org/72212120:11
*** maciejjozefczyk has joined #openstack-lbaas20:13
born2bakejohnsom I know this is ridiculous .... flavor was too small for amphora lol21:52
born2bakechanged it to bigger ram/cpu/disk and its all good now21:52
johnsomAwesome!21:53
*** rcernin has joined #openstack-lbaas22:06
*** rcernin has quit IRC22:11
*** born2bake has quit IRC22:16
*** dougwig has quit IRC22:35
*** dawzon has quit IRC22:35
*** kklimonda has quit IRC22:36
*** nicolasbock has quit IRC22:36
*** luketollefson has quit IRC22:36
*** rpittau|afk has quit IRC22:36
*** mnaser has quit IRC22:36
*** emccormick has quit IRC22:36
*** NobodyCam has quit IRC22:36
*** gregwork has quit IRC22:36
*** beisner has quit IRC22:37
*** lxkong has quit IRC22:37
*** andrein has quit IRC22:37
*** xgerman__ has joined #openstack-lbaas22:37
*** emccormick has joined #openstack-lbaas22:37
*** NobodyCam has joined #openstack-lbaas22:37
*** xgerman has quit IRC22:37
*** xgerman__ is now known as xgerman22:37
*** jrosser has quit IRC22:37
*** fyx has quit IRC22:37
*** johnsom has quit IRC22:37
*** stevenglasford has quit IRC22:37
*** headphoneJames has quit IRC22:37
*** gmann has quit IRC22:37
*** rm_work has quit IRC22:38
*** dougwig has joined #openstack-lbaas22:38
*** gregwork has joined #openstack-lbaas22:38
*** gmann has joined #openstack-lbaas22:38
*** lxkong has joined #openstack-lbaas22:39
*** luketollefson has joined #openstack-lbaas22:40
*** dawzon has joined #openstack-lbaas22:40
*** andrein has joined #openstack-lbaas22:40
*** jrosser has joined #openstack-lbaas22:41
*** mnaser has joined #openstack-lbaas22:41
*** stevenglasford has joined #openstack-lbaas22:41
*** beisner has joined #openstack-lbaas22:42
*** nicolasbock has joined #openstack-lbaas22:42
*** johnsom has joined #openstack-lbaas22:43
*** rcernin has joined #openstack-lbaas22:48
*** rm_work has joined #openstack-lbaas22:51
*** tkajinam has joined #openstack-lbaas22:56
*** TrevorV has quit IRC22:57
*** kklimonda has joined #openstack-lbaas23:04
*** rpittau|afk has joined #openstack-lbaas23:04
*** fyx has joined #openstack-lbaas23:12
*** rcernin has quit IRC23:49
« Wednesday, 2020-06-03 Index Friday, 2020-06-05 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!