Thursday, 2020-09-24

openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Set nodepool swap size back to 8GB  https://review.opendev.org/75383800:02
*** sapd1 has joined #openstack-lbaas00:48
openstackgerritMerged openstack/octavia master: Fix image build when installing agent from package  https://review.opendev.org/75186301:48
lxkongjohnsom, could you please have a look at https://review.opendev.org/753594 for the  issue https://storyboard.openstack.org/#!/task/39681?02:04
lxkongnot sure if we still have window to merge that in Victoria02:04
*** rcernin has quit IRC02:38
*** zzzeek has quit IRC02:40
*** zzzeek has joined #openstack-lbaas02:41
*** rcernin has joined #openstack-lbaas02:41
*** sapd1 has quit IRC02:47
*** sapd1 has joined #openstack-lbaas02:47
*** zzzeek has quit IRC02:50
*** zzzeek has joined #openstack-lbaas02:52
*** xgerman has quit IRC03:04
openstackgerritMaAoyu proposed openstack/python-octaviaclient master: Remove install unnecessary packages  https://review.opendev.org/75392503:07
CirilloAfter some modifications i can create a Load Balancer now guys03:12
Cirillobut it stays at Pending_create state03:12
Cirilloand after some time it change to Error, giving the error "octavia.amphorae.driver_exceptions.exceptions.TimeOutException: contacting the amphora timed out"03:13
CirilloHave any idea what can be?03:13
*** psachin has joined #openstack-lbaas03:27
*** ramishra has joined #openstack-lbaas03:54
rm_workjohnsom: https://review.opendev.org/#/c/753594/ !!! I wonder if this is what was happening to me? lol04:04
rm_worki had said that the latest thing that broke was a cert rotation04:04
rm_worki am suddenly 90% sure this is the problem04:04
rm_workbringing already deleted amps back from DELETED to ERROR04:05
rm_worksorrison: ^^ also you04:06
sorrisonahhh, yes we must be hitting that. I can test this patch out04:08
Cirillorm_work: Do you think that is the same problem?04:11
rm_workCirillo: your problem is most likely an issue with the network connectivity on the management net04:11
rm_worksorrison: yeah it might be a bit until we can see if the problem "goes away" since it isn't exactly something i knew how to repro04:12
rm_worksorrison: i guess we could try to set it up to repro itself....04:12
rm_workand then watch carefully04:12
Cirillorm_work: Very good, i will check it right now04:13
CirilloThank you04:13
rm_workCirillo: likely when the Amphora is created on the management network, the controller node can't reach it. If you set the retry times decently high, you can test it out manually while it's failing to connect04:14
rm_workor there's an option somewhere to prevent the revert flows from running, so it will leave the broken amphora... i forget what that is, sec04:14
Cirillohmm04:14
rm_workah04:15
rm_work`[task_flow]` section04:15
rm_workdisable_revert = False04:15
rm_workonly use for debugging04:15
rm_workerr, rather you'd set it to True04:15
CirilloDo you know how to check if it are using Amphora V2?04:16
Cirilloi will set tu debug True04:16
Cirilloand use revert too04:16
*** psachin has quit IRC04:45
rm_workprobably v104:52
rm_work[api_settings] default_provider_driver04:52
rm_workif you didn't set that, or set up custom providers in the API, then it'll be v104:52
CirilloHmmm nice Rm_work05:19
CirilloBut How Can set it to use Amphora V2?05:20
cgoncalvesCirillo, https://docs.openstack.org/octavia/latest/install/install-amphorav2.html05:28
cgoncalvesCirillo, amphorav2 should not be used in production environments. it is an experimental feature.05:28
Cirilloohh i do not know, thank you cgoncalves05:29
*** Cirillo has quit IRC05:31
*** gcheresh has joined #openstack-lbaas05:32
openstackgerritMerged openstack/octavia stable/ussuri: Fixes API list handling of unscoped tokens  https://review.opendev.org/75338206:06
openstackgerritMerged openstack/octavia stable/stein: Fixes API list handling of unscoped tokens  https://review.opendev.org/75338406:06
*** maciejjozefczyk has joined #openstack-lbaas06:07
*** maciejjozefczyk_ has joined #openstack-lbaas06:15
*** maciejjozefczyk has quit IRC06:17
openstackgerritMerged openstack/octavia stable/ussuri: Set Grub timeout to 0 for fast boot times  https://review.opendev.org/75285306:27
openstackgerritMerged openstack/octavia master: Add a requirements.txt check job  https://review.opendev.org/75192506:27
openstackgerritMerged openstack/octavia master: Update protocol combination api-ref for PROXYV2  https://review.opendev.org/75046806:27
*** vishalmanchanda has joined #openstack-lbaas06:30
*** ataraday_ has quit IRC07:03
*** ataraday_ has joined #openstack-lbaas07:20
*** zzzeek has quit IRC07:33
*** zzzeek has joined #openstack-lbaas07:36
*** ccamposr has joined #openstack-lbaas07:37
*** ccamposr__ has quit IRC07:39
*** zzzeek has quit IRC08:01
*** zzzeek has joined #openstack-lbaas08:01
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add non-voting aarch64 scenario job  https://review.opendev.org/74762908:04
cgoncalvesFYI, HAProxy fixed a regression in v2.2 that we worked around on our side08:11
cgoncalveshttps://github.com/FireBurn/haproxy/commit/2591393dad44d874f53ff3cc17cc1e88f1b91b2208:11
cgoncalveshttps://review.opendev.org/#/c/744515/08:11
*** yamamoto has quit IRC08:23
*** yamamoto has joined #openstack-lbaas08:25
*** yamamoto has quit IRC08:25
rm_workk, doesn't affect me because we don't do TLS support XD08:28
rm_worktho we may at some point in the future08:28
cgoncalvesdo you do plain text auths? :P08:30
openstackgerritAnn Taraday proposed openstack/octavia stable/ussuri: [Amphorav2] Healthmonitor operation minor fixes  https://review.opendev.org/75396908:30
rm_workerr08:31
rm_workthat wouldn't affect the agent since that's an haproxy bug :D08:31
rm_workwe don't do any haproxy auths08:32
rm_workhaproxy modes we suggest are PROXY and HTTPS08:32
*** rcernin has quit IRC08:35
*** yamamoto has joined #openstack-lbaas08:39
*** rcernin has joined #openstack-lbaas08:41
*** rcernin has quit IRC08:57
*** yamamoto has quit IRC08:58
*** maciejjozefczyk has joined #openstack-lbaas09:06
*** gryf has joined #openstack-lbaas09:06
*** maciejjozefczyk_ has quit IRC09:08
gryfhi. is it intended, that having "enable_plugin octavia https://opendev.org/openstack/octavia" in my local.conf on devstack, result is inability to build amphora image?09:08
openstackgerritMerged openstack/octavia master: Ignore DELETED amphorae when performing certificate rotation  https://review.opendev.org/75359409:09
cgoncalvesgryf, hi. no, it should work. enabling the plugin is the right think to do and amphora image build is enabled by default. do you have logs?09:09
gryfcgoncalves, yes, I do. give me a sec.09:12
*** spatel has joined #openstack-lbaas09:14
gryfcgoncalves, http://paste.openstack.org/show/798319/ and my local.conf: http://paste.openstack.org/show/798318/09:17
*** spatel has quit IRC09:18
*** yamamoto has joined #openstack-lbaas09:31
cgoncalvesgryf, can you please share the rest of the stack log?09:31
gryfcgoncalves, hm. for some reason, paste.openstack.org is truncating the logsā€¦09:36
*** yamamoto has quit IRC09:36
gryfcgoncalves, here is the tail of the log: http://paste.openstack.org/show/798321/09:39
cgoncalvesgryf, thanks. so, diskimage-builder is not installed on your system. DIB was being installed as an octavia requirement, but patch https://review.opendev.org/#/c/741960/ removed that09:42
gryfright.09:42
cgoncalvesgryf, as a quick workaround add the following to your local.conf file09:43
cgoncalvesLIBS_FROM_GIT+=diskimage-builder09:43
gryfwell, I did it manually, yet still there was issues with building the image.09:44
cgoncalvesIMO a proper fix would be to pip install requirements from octavia/diskimage-create/requirements.txt in the octavia devstack plugin right before the start of the image build09:45
cgoncalvesgryf, same error or different one?09:45
gryfI guess so. it should be transparent for the devstack, isn't it :)?09:45
cgoncalvesI don't see diskimage-builder in LIBS_FROM_GIT in the local.conf you pasted09:46
gryfdifferent.09:46
cgoncalvesok. please paste the log09:47
gryfyes, since I don't need it explicitly. If Octavia need it, the right place for the requirement should be within Octavia, right?09:47
cgoncalvesgryf, yes. that is why I said "the proper fix would be [...] in the octavia devstack plugin" ;)09:48
gryf;)09:49
*** lxkong has quit IRC09:52
*** zzzeek has quit IRC09:57
*** zzzeek has joined #openstack-lbaas10:00
openstackgerritLingxian Kong proposed openstack/octavia stable/ussuri: Ignore DELETED amphorae when performing certificate rotation  https://review.opendev.org/75405610:03
openstackgerritLingxian Kong proposed openstack/octavia stable/train: Ignore DELETED amphorae when performing certificate rotation  https://review.opendev.org/75405710:03
openstackgerritCarlos Goncalves proposed openstack/octavia master: Ensure install of diskimage-create.sh requirements  https://review.opendev.org/75405810:07
cgoncalvesgryf, see ^. untested, feedback welcome :)10:07
gryfcgoncalves, will check, thanks!10:09
*** yamamoto has joined #openstack-lbaas10:24
*** yamamoto has quit IRC10:26
openstackgerritCarlos Goncalves proposed openstack/octavia stable/stein: Ignore DELETED amphorae when performing certificate rotation  https://review.opendev.org/75406210:33
*** rcernin has joined #openstack-lbaas10:34
*** salmankhan has joined #openstack-lbaas10:42
*** yamamoto has joined #openstack-lbaas10:42
*** maciejjozefczyk_ has joined #openstack-lbaas10:45
*** maciejjozefczyk has quit IRC10:48
*** yamamoto has quit IRC10:51
*** rcernin has quit IRC10:56
*** sapd1 has quit IRC11:03
gryfcgoncalves, I've tested your patch locally, didn't crash, good job ;)11:08
cgoncalvesgryf, nice! thanks for reporting this issue and testing the patch!11:08
gryfcgoncalves, yw :)11:15
*** yamamoto has joined #openstack-lbaas11:29
*** gcheresh has quit IRC11:37
*** yamamoto has quit IRC11:40
rm_workcgoncalves: do you recall offhand, if we create the HA Port in the user's project or in Octavia's service project?11:46
cgoncalvesrm_work, service project. HA port is a provider implementation detail11:49
*** gcheresh has joined #openstack-lbaas12:05
*** yamamoto has joined #openstack-lbaas12:06
*** zzzeek has quit IRC12:09
*** zzzeek has joined #openstack-lbaas12:12
*** salmankhan has quit IRC12:28
*** zzzeek has quit IRC12:40
*** maciejjozefczyk has joined #openstack-lbaas12:41
*** zzzeek has joined #openstack-lbaas12:42
*** maciejjozefczyk_ has quit IRC12:44
*** salmankhan has joined #openstack-lbaas12:44
openstackgerritMerged openstack/octavia stable/ussuri: Fix memory consumption issues with default connection_limit  https://review.opendev.org/74765112:44
*** yamamoto has quit IRC12:48
*** yamamoto has joined #openstack-lbaas12:54
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add SCTP protocol scenario tests  https://review.opendev.org/73864312:57
*** yamamoto has quit IRC13:03
*** sapd1 has joined #openstack-lbaas13:15
rm_workcgoncalves: ok that is what I THOUGHT, but it looks like the HA port is ending up in the user's project O_o13:24
rm_workand we didn't patch anything around that, so maybe it changed13:25
rm_workgonna have to go look13:25
rm_workjohnsom: ^^13:25
johnsomThe HA port is the VIP, which can be in the tenant if they brought their own port.13:26
johnsomIn the DB VIP table, it will be octavia_owned == 013:26
rm_workright but it seems to be doing it even for non-BYOP13:27
rm_workhmm13:27
openstackgerritCarlos Goncalves proposed openstack/octavia stable/stein: Ignore DELETED amphorae when performing certificate rotation  https://review.opendev.org/75406213:29
openstackgerritCarlos Goncalves proposed openstack/octavia stable/stein: Fix the amphora no-op driver  https://review.opendev.org/75409413:29
johnsomhttps://github.com/openstack/octavia/blob/master/octavia/api/v2/controllers/load_balancer.py#L273 goes to https://github.com/openstack/octavia/blob/9d112ecec13afa42f6c498110a5530b7570cbd47/octavia/network/drivers/neutron/allowed_address_pairs.py#L45013:30
johnsomhttps://github.com/openstack/octavia/blob/9d112ecec13afa42f6c498110a5530b7570cbd47/octavia/network/drivers/neutron/allowed_address_pairs.py#L51213:32
cgoncalvesgood point the "bring your own port", I totally forgot that option13:35
*** rcernin has joined #openstack-lbaas13:35
*** yamamoto has joined #openstack-lbaas13:36
*** yamamoto has quit IRC13:45
*** TrevorV has joined #openstack-lbaas13:46
rm_workhmmm13:50
rm_workthen wtf is going on in our env13:50
*** rcernin has quit IRC13:51
*** ataraday_ has quit IRC14:07
openstackgerritCarlos Goncalves proposed openstack/octavia stable/ussuri: Fix AttributeError on TLS-enabled pool provisioning  https://review.opendev.org/75410714:11
openstackgerritCarlos Goncalves proposed openstack/octavia stable/ussuri: Fix backend certificate file paths  https://review.opendev.org/75410814:11
*** maciejjozefczyk_ has joined #openstack-lbaas14:25
johnsomThe vip has to be tenant owned to allow floating ips14:26
*** maciejjozefczyk has quit IRC14:27
*** gcheresh has quit IRC14:56
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add SCTP protocol scenario tests  https://review.opendev.org/73864315:20
*** ramishra has quit IRC15:26
*** gcheresh has joined #openstack-lbaas15:30
*** yamamoto has joined #openstack-lbaas15:43
*** yamamoto has quit IRC15:50
*** ramishra has joined #openstack-lbaas16:12
rm_workjohnsom: ok, so then *all* vips ARE in the tenant project?16:13
rm_workregardless of whether they "brought their own port" or not16:14
johnsomYes16:14
rm_workok. i'm not crazy, thanks16:14
rm_workwriting some auditing stuff16:14
rm_workI found a TON of old vrrp ports16:14
rm_workthere was a leak somewhere... at some point16:14
rm_workhmm almost all of them created around the same time, so guessing it was something with maybe a bug in failover code at some point?16:16
rm_workthough... hmm16:16
rm_workyeah almost all in june16:16
*** spatel has joined #openstack-lbaas16:16
rm_work... some from august tho16:16
johnsomWell, I have seen this issue too, where sometimes nova deletes them with the vm, sometimes not. We have code in to try to delete them anyway to catch the times they don't go away if I remember right16:17
rm_workhmm16:18
rm_workeven one from september16:18
johnsomSorry I wasn't clear enough earlier. The quality of my answers degrades with the hour in the morning of the response. lol16:18
rm_workyeah, this is cementing my theory that we should provide some auditing code, if only for testing purposes for ourselves16:18
rm_workI've got something in progress16:19
rm_workjust like a ... contrib script16:19
rm_workcalculating what objects should exist given the LBs/amps present, and then checking against that list to see what the deltas are, missing/extra objects16:19
johnsomThe new LB failover will clean them up now as well, so that helps, but yeah, an audit script would be nice too.16:20
rm_workno extra servers (good!) ... some extra SGs, lots of extra ports16:20
rm_workand some missing too <_< but those all line up with ERROR LBs :P16:20
*** gcheresh has quit IRC16:21
rm_workhmmm the VRRP ports SHOULD be bound/UP I think16:28
johnsomYes, they must be16:31
*** gcheresh has joined #openstack-lbaas16:31
rm_worka start: http://paste.openstack.org/show/798348/16:41
*** servagem has quit IRC16:44
johnsomThe tricky part is appropriately handling objects that are legitimately transitioning.16:46
rm_workwell that only finds vrrp ports that are unbound, and don't have amps that exist16:46
rm_workor SGs that exist for LBs that don't16:47
rm_workit'd ignore anything in a PENDING state, I think16:47
rm_worksince it'd still qualify as "expected"16:47
johnsomRight16:47
*** gcheresh has quit IRC16:49
rm_workis there a way to see what's using a SG?16:49
rm_workor is that a whole ball of FUBAR16:49
johnsomWalk the ports16:49
rm_worklol16:49
rm_workspectacular16:49
johnsomAt least that used to be the way it was16:49
*** spatel has quit IRC16:50
rm_worksounds right16:50
johnsomYou might want a "dry run" option in there that doesn't go straight to delete.16:52
*** servagem has joined #openstack-lbaas16:55
rm_worklol yeah16:57
rm_workthis is prototype16:57
rm_worki'm debugging as my dry-run :D16:57
*** salmankhan has quit IRC17:06
*** ccamposr__ has joined #openstack-lbaas17:07
*** ccamposr has quit IRC17:09
*** Cirillo has joined #openstack-lbaas17:48
openstackgerritMerged openstack/octavia master: Fix building Ubuntu 20.04 focal amphorae  https://review.opendev.org/72254218:11
*** ccamposr__ has quit IRC18:15
*** ccamposr has joined #openstack-lbaas18:16
*** maciejjozefczyk_ has quit IRC18:51
*** Cirillo has quit IRC19:10
*** openstackgerrit has quit IRC19:14
*** lxkong has joined #openstack-lbaas19:32
*** TrevorV has quit IRC19:35
*** yamamoto has joined #openstack-lbaas19:48
*** yamamoto has quit IRC19:52
*** lemko has quit IRC19:57
*** lemko has joined #openstack-lbaas19:57
*** zzzeek has quit IRC20:12
*** zzzeek has joined #openstack-lbaas20:14
*** zzzeek has quit IRC20:18
*** zzzeek has joined #openstack-lbaas20:20
*** zzzeek has quit IRC20:25
*** zzzeek has joined #openstack-lbaas20:26
*** zzzeek has quit IRC20:30
*** zzzeek has joined #openstack-lbaas20:33
*** zzzeek has quit IRC20:43
*** zzzeek has joined #openstack-lbaas20:46
*** zzzeek has quit IRC20:51
*** ccamposr has quit IRC20:51
*** ccamposr has joined #openstack-lbaas20:52
*** zzzeek has joined #openstack-lbaas20:53
*** ccamposr has quit IRC20:54
*** ccamposr has joined #openstack-lbaas20:54
*** gcheresh has joined #openstack-lbaas21:05
*** gcheresh has quit IRC21:09
*** gthiemonge has quit IRC21:11
*** gthiemonge has joined #openstack-lbaas21:11
*** ccamposr has quit IRC21:11
*** ccamposr has joined #openstack-lbaas21:11
*** ccamposr__ has joined #openstack-lbaas21:23
*** ccamposr has quit IRC21:24
*** ccamposr has joined #openstack-lbaas21:46
*** ccamposr__ has quit IRC21:48
*** yamamoto has joined #openstack-lbaas21:49
*** yamamoto has quit IRC21:53
*** ccamposr has quit IRC21:55
*** ccamposr__ has joined #openstack-lbaas21:55
*** ccamposr__ has quit IRC21:59
*** ccamposr has joined #openstack-lbaas22:05
*** ccamposr has quit IRC22:06
*** ccamposr has joined #openstack-lbaas22:06
*** yamamoto has joined #openstack-lbaas22:13
*** ccamposr has quit IRC22:18
*** zzzeek has quit IRC22:24
*** zzzeek has joined #openstack-lbaas22:25
*** openstackgerrit has joined #openstack-lbaas22:38
openstackgerritMerged openstack/octavia master: Ensure install of diskimage-create.sh requirements  https://review.opendev.org/75405822:38
johnsomOye, finally22:38
*** vishalmanchanda has quit IRC22:39
*** zzzeek has quit IRC22:44
*** zzzeek has joined #openstack-lbaas22:46
*** rcernin has joined #openstack-lbaas22:58
*** ccamposr has joined #openstack-lbaas23:10
*** rcernin has quit IRC23:14
*** rcernin has joined #openstack-lbaas23:14
sorrisonrm_work: so applied that patch last night, this morning we've had no amps in ERROR and we usually do, so far so good23:22
rm_workwoot23:26
rm_workyou were seeing it DAILY? lol23:26
rm_workit's going out to prod as we speak for me23:27
rm_workso i'll know by end of next week prolly23:27
rm_workwith at least minimal confidence23:27
johnsomWhich patch?23:30
rm_workthe master version of https://review.opendev.org/75406223:36
johnsomAh23:36
*** zzzeek has quit IRC23:43
*** zzzeek has joined #openstack-lbaas23:45

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!