Friday, 2020-01-24

« Thursday, 2020-01-23 Index Saturday, 2020-01-25 »
*** luksky has quit IRC00:12
*** trident has quit IRC00:13
*** trident has joined #openstack-lbaas00:14
*** mithilarun has joined #openstack-lbaas00:16
*** goldyfruit_ has quit IRC00:34
*** goldyfruit_ has joined #openstack-lbaas00:34
*** mithilarun has quit IRC01:01
*** mithilarun has joined #openstack-lbaas01:02
openstackgerritMerged openstack/octavia master: Use LUKS1 for certfs-ramfs for all distributions  https://review.opendev.org/69888501:14
*** yamamoto has joined #openstack-lbaas01:16
*** goldyfruit_ has quit IRC01:21
*** goldyfruit has joined #openstack-lbaas01:21
*** mithilarun has quit IRC01:41
*** yamamoto has quit IRC02:19
*** yamamoto has joined #openstack-lbaas02:25
*** goldyfruit has quit IRC02:25
*** vishalmanchanda has joined #openstack-lbaas02:25
*** armax has joined #openstack-lbaas02:40
*** yamamoto has quit IRC03:38
*** psachin has joined #openstack-lbaas03:38
*** goldyfruit has joined #openstack-lbaas03:38
*** goldyfruit has quit IRC03:43
*** goldyfruit has joined #openstack-lbaas03:43
*** goldyfruit has quit IRC04:05
*** goldyfruit has joined #openstack-lbaas04:11
*** ramishra has joined #openstack-lbaas04:19
*** goldyfruit has quit IRC04:27
*** goldyfruit has joined #openstack-lbaas04:27
*** goldyfruit has quit IRC04:33
*** goldyfruit has joined #openstack-lbaas04:33
*** yamamoto has joined #openstack-lbaas04:36
*** armax has quit IRC04:37
*** goldyfruit has quit IRC04:50
*** goldyfruit has joined #openstack-lbaas04:50
*** goldyfruit has quit IRC04:59
openstackgerritCarlos Goncalves proposed openstack/octavia stable/train: Use LUKS1 for certfs-ramfs for all distributions  https://review.opendev.org/70410706:31
*** yamamoto has quit IRC06:40
*** yamamoto has joined #openstack-lbaas06:43
*** yamamoto has quit IRC06:58
*** yamamoto has joined #openstack-lbaas07:08
*** yamamoto has quit IRC07:15
*** yamamoto has joined #openstack-lbaas07:19
*** yamamoto has quit IRC07:29
*** yamamoto has joined #openstack-lbaas07:40
*** yamamoto has quit IRC07:41
*** yamamoto has joined #openstack-lbaas07:45
openstackgerritAnn Taraday proposed openstack/octavia master: Add option to set default ssl ciphers in haproxy  https://review.opendev.org/68533707:50
*** yamamoto has quit IRC07:58
*** luksky has joined #openstack-lbaas08:02
*** yamamoto has joined #openstack-lbaas08:03
*** yamamoto has quit IRC08:09
*** yamamoto has joined #openstack-lbaas08:09
*** maciejjozefczyk has joined #openstack-lbaas08:10
*** yamamoto has quit IRC08:12
*** yamamoto has joined #openstack-lbaas08:16
*** tkajinam has quit IRC08:20
*** tesseract has joined #openstack-lbaas08:34
*** rpittau|afk is now known as ironic08:51
*** ironic is now known as rpittau08:51
*** yamamoto has quit IRC08:55
*** yamamoto has joined #openstack-lbaas08:57
*** yamamoto has quit IRC08:58
*** pcaruana has joined #openstack-lbaas09:14
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add tests for mixed IP networks UDP members  https://review.opendev.org/66861909:28
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add UDP test scenario  https://review.opendev.org/65651509:28
*** yamamoto has joined #openstack-lbaas09:41
*** yamamoto has quit IRC09:45
openstackgerritMerged openstack/octavia master: Transition amphora flows to dicts  https://review.opendev.org/66889809:51
openstackgerritMerged openstack/octavia master: Convert Lb flows to use provider dicts  https://review.opendev.org/67172509:54
openstackgerritMerged openstack/octavia master: Add logging filter for AmpConnectionRetry exception  https://review.opendev.org/70055309:54
*** yamamoto has joined #openstack-lbaas09:55
*** yamamoto has quit IRC10:03
*** yamamoto has joined #openstack-lbaas10:12
openstackgerritMerged openstack/octavia stable/train: Add listener and pool protocol validation  https://review.opendev.org/70185010:36
openstackgerritMerged openstack/octavia stable/stein: Add listener and pool protocol validation  https://review.opendev.org/70204310:38
*** yamamoto has quit IRC10:57
*** yamamoto has joined #openstack-lbaas10:59
*** yamamoto has quit IRC11:01
*** yamamoto has joined #openstack-lbaas11:02
*** yamamoto has quit IRC11:19
*** luksky has quit IRC11:23
*** yamamoto has joined #openstack-lbaas11:24
*** maciejjozefczyk_ has joined #openstack-lbaas11:28
*** rpittau is now known as rpittau|bbl11:28
*** maciejjozefczyk has quit IRC11:30
*** yamamoto has quit IRC11:42
*** yamamoto has joined #openstack-lbaas11:45
*** xgerman has quit IRC11:45
*** xgerman has joined #openstack-lbaas11:45
*** ccamposr__ has quit IRC11:49
*** yamamoto has quit IRC11:56
*** luksky has joined #openstack-lbaas12:01
*** maciejjozefczyk_ has quit IRC12:23
*** yamamoto has joined #openstack-lbaas12:36
*** maciejjozefczyk_ has joined #openstack-lbaas12:37
*** yamamoto has quit IRC12:43
openstackgerritAnn Taraday proposed openstack/octavia master: Convert health_monitor flows to use provider models  https://review.opendev.org/67977913:02
openstackgerritAnn Taraday proposed openstack/octavia master: Transition l7policy flows to dicts  https://review.opendev.org/66597713:02
openstackgerritAnn Taraday proposed openstack/octavia master: Transition l7rule flows to dicts  https://review.opendev.org/66817313:02
openstackgerritAnn Taraday proposed openstack/octavia master: Jobboard based controller  https://review.opendev.org/64740613:02
openstackgerritAnn Taraday proposed openstack/octavia master: Jobboard based controller  https://review.opendev.org/64740613:08
*** rpittau|bbl is now known as rpittau13:40
lukskyjohnsom - it didn't go away :/13:40
lukskyjohnsom - and one more strange thing, after upgrading every piece of octavia to stable/queens I get situation where LB is in OFFLINE state, healthmonitor is in ACTIVE, and member is in OFFLINE, but... LB is working:13:47
lukskyhttp://paste.openstack.org/show/788753/13:47
cgoncalvesluksky, hi. that should be because your member is in OFFLINE operating status and since that is the only member associated to the load balancer, the LB also reports OFFLINE13:51
lukskyoctavia version: 2.1.2, and amphora-agent 2.1.313:52
cgoncalvesluksky, I'd check if the member is indeed up and the health monitor configured properly (e.g. the expected code from the member response is 200)13:52
lukskyok, but all is working13:52
lukskychecked logs from member:13:52
luksky195.167.157.117 - - [24/Jan/2020:13:52:45 +0000] "GET / HTTP/1.0" 200 612 "-" "-"13:52
luksky195.167.157.130 - - [24/Jan/2020:13:52:46 +0000] "GET / HTTP/1.0" 200 612 "-" "-"13:52
lukskyand the site is also shows up correctly :/13:53
lukskyI have one more LB which didn't failover to new amphoras, and there member is ONLINE13:55
cgoncalveshmm, in that case :/13:56
cgoncalvessorry, I don't follow your last comment13:56
lukskyI have two LBs, one of it I created after upgrade, and second was running during upgrade.13:58
lukskythe 'old' one is showing member status correctly13:59
lukskyevery new LB - shows member as OFFLINE, LB is in status OFFLINE - *AND* new LBs are working :/14:00
luksky(old one also)14:01
lukskyI have also situation, where member has 'operating_status' "No monitor" - when I deleted it and add once more - operating_status changed to 'offline' - but LB *IS* working :/14:03
cgoncalvesok, I understand now, thanks.14:04
cgoncalvesI will stack Queens locally and see if I can reproduce14:04
cgoncalveswhat I'm understanding is that the problem is specific to both queens controller and amphora images14:05
cgoncalvesqueens controller + pike amphora works, right?14:05
lukskychecking, please wait14:06
lukskyversion of controler:14:06
lukskypip list | grep -i octavia14:06
lukskyoctavia (2.1.2)14:06
lukskyit is more complicated that I thought, sorry :/14:07
lukskyon one (the 'old' one) LB I have following two amphoras version (the one with 3.0.0.0b2.dev62) is from time where I have tested different version of images):14:08
lukskyubuntu@amphora-253e6a3d-776a-4b9f-9c30-dd7dd67a8c4b:~$ amphora-agent --version14:08
luksky%prog 3.0.0.0b2.dev6214:08
lukskyubuntu@amphora-70f56c40-f543-4826-a36e-b93c26f948df:~$ amphora-agent --version14:09
luksky%prog 2.1.3.dev814:09
lukskyI will failover this 3.0.0 now14:09
lukskythe 3.0.0 was master14:10
cgoncalves3.y.z is Rocky release. backward compatibility is only with older versions in the amphora. say, Queens amphora + Rocky controller is expected to work.14:11
lukskyyes, understood14:11
lukskydon't want mess here, so here is paste14:15
lukskyfrom OFFLINE LB:14:15
lukskyhttp://paste.openstack.org/show/788755/14:15
cgoncalvesamphorae running queens, good14:17
lukskyyes, the one LB which is online, (which had 3.0.0 amphora), is still online, but I added one more member, and this NEW member is ... OFFLINE - but is working ok14:18
lukskyhttps://pasteboard.co/IRsuQgL.png14:20
lukskymember with IP 106 is serving the content now14:21
cgoncalveshmm. would you know why is admin state up = No for the 106 member?14:22
cgoncalvesin your previous paste (http://paste.openstack.org/show/788753/), the member has admin_state_up=True14:23
lukskysory, not this screenshot14:23
lukskyyes, I put this admin = off14:23
lukskyyes, I put this admin state to No14:23
lukskyhttps://pasteboard.co/IRswrlp.png14:24
cgoncalvesyou toggled now the admin state in both members14:25
cgoncalvesgive the health monitor a few seconds to check for health14:25
lukskyI did - now I see 10614:26
lukskywhich is offline14:26
lukskyhttp://195.167.157.155/ - this main ip of LB14:26
lukskyI'm pretty sure, that if I delete 107 member and add him once again it will be OFFLINE, and will be working :)14:29
*** goldyfruit has joined #openstack-lbaas14:34
cgoncalvesluksky, could you please share the haproxy config file in the amphora?14:34
lukskyyes14:35
cgoncalvesit should be in amphora@/var/lib/amphora/14:35
lukskyfrom both ?14:36
lukskyhttp://paste.openstack.org/show/788756/14:36
cgoncalvesluksky, whichever is the active amphora should be enough14:36
lukskyhttp://paste.openstack.org/show/788757/ - this MASTER, above is BACKUP14:37
cgoncalvesthanks. the configs look the same, good14:38
cgoncalvesbtw I'm still spinning up a queens cloud here14:39
lukskyI deleted 107 from LB and added once more :D14:39
lukskyhttps://pasteboard.co/IRsCTUO.png14:40
lukskymessed up with amphoras sorry :( - here are configs for master and backup for LB with member 107 and 106:14:50
lukskyhttp://paste.openstack.org/show/788758/14:50
*** psachin has quit IRC14:52
*** goldyfruit_ has joined #openstack-lbaas14:54
*** goldyfruit has quit IRC14:55
*** goldyfruit_ has quit IRC14:55
*** goldyfruit_ has joined #openstack-lbaas14:55
*** coreycb has quit IRC14:57
*** coreycb has joined #openstack-lbaas14:57
openstackgerritCarlos Goncalves proposed openstack/octavia master: Add listener allowed_cidrs to the feature matrix  https://review.opendev.org/70385115:02
lukskyI think, this may be related to database changes/schema ?15:06
lukskyhere is octavia-db-manage history: http://paste.openstack.org/show/788759/15:06
*** goldyfruit_ has quit IRC15:07
*** goldyfruit_ has joined #openstack-lbaas15:07
cgoncalveswhy do you think that? you seem to have latest db schema for queens15:08
cgoncalvesluksky, what does haproxy show in the journal?15:09
lukskyhttp://paste.openstack.org/show/788760/15:10
*** yamamoto has joined #openstack-lbaas15:10
lukskyfrom second amphora also ?15:11
cgoncalvesno need, only active15:12
cgoncalvesServer c30a65f6-4419-40ac-905a-684fac6b02b4/a6bad16b-5734-4fcd-b6d4-8d58e532f170 is UP, reason: Layer7 check passed, code: 200, info: "HTTP status check returned code <3C>200<3E>"15:13
cgoncalvesit was reported DOWN before15:13
lukskyhere is master: http://paste.openstack.org/show/788761/15:14
lukskyyes, 2 days ago... :)15:15
*** yamamoto has quit IRC15:15
*** goldyfruit_ has quit IRC15:16
*** goldyfruit_ has joined #openstack-lbaas15:17
johnsomluksky: This probably means you lb-mgmt-net is not working correctly. The amphora health messages may not be getting to the health manager process15:19
lukskyit is L2, separated vlan15:20
johnsomIf you enable debug in the config file and restart your health manager you should see debug log messages that say it received a message from an amphora IP15:21
lukskyall logging I made from octavia controller to amphoras15:21
lukskyok15:21
johnsomIf you don’t see that, the health messages are not getting back to the controller.15:22
lukskyand configuration of network (neither on switches nor in neutron) wasn't touched15:22
luksky2020-01-24 16:22:50.632 30093 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from ('10.99.99.23', 42060) dorecv /usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:18915:23
lukskyjohnsom - this kind of messages ?15:23
johnsomHmm, ok, yes those15:23
johnsomOh! You said you had an hmac error right?15:24
johnsomThat is dropping them15:24
lukskyyes... still ...15:24
lukskydidn't find way to get rid of it15:24
luksky2020-01-24 16:24:20.729 30093 WARNING octavia.amphorae.backends.health_daemon.status_message [-] calculated hmac(hex=True): 66336161373264306366306165623638636532356631613436313962613466303161636534353132663861393865393337376265393239346539383630663565 not equal to msg hmac: 8c2c938c2d8d52cc8d924d9580aa8b530b81ca4d0c8d4dcd2ccc6a017bf214c1ab485727f956055a9b652e04d64661018aeecf9698430310bc3da57579adbe6a dropping packet15:24
johnsomYeah, ok, we need to figure out why that is happening.  This is a security check. The controller isn’t trusting the health message so discarding them15:26
*** dulek has quit IRC15:28
*** armax has joined #openstack-lbaas15:30
johnsomluksky Can you check that this configuration setting is the same on all of your controllers? https://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.heartbeat_key15:31
lukskyok, I turned off two controllers, one remain15:31
johnsomIt needs to be the same in the worker, health manager, and housekeeping octavia.conf files.15:31
lukskyI have only one config file15:32
lukskyoctavia.conf15:32
lukskyand there are subsections15:32
johnsomYeah, ok, so that key is set in the [health_manager] section of the octavia.conf right?15:33
lukskygrep heartbeat_key /etc/octavia/octavia.conf15:33
lukskyheartbeat_key = xxxxx15:33
lukskyyes, checked on amphora - it is the same15:34
johnsomOk, then if you ssh into one of the amphora, does that match the key in /etc/octavia/amphora-agent.conf?15:34
lukskyyes15:35
johnsomHmmm, ok, then give me a second to check the versions you are running again.15:35
luksky2.1.2 on controller, 2.1.3.dev8 on amphora15:36
johnsomWhen was the last time you restarted the health manager porcess?15:37
lukskyoo, many times15:39
luksky   Active: active (running) since Fri 2020-01-24 16:22:34 CET; 17min ago15:39
*** dulek has joined #openstack-lbaas15:44
johnsomHmmm, I am really not sure why you would still be seeing those hmac errors with those  versions and if the keys all match.15:45
johnsomCan you humor me and boot a fresh load balancer, in your current configuration and see if it is also logging these hmac errors?15:46
lukskyYou mean to rebuild whole VM ?15:47
*** maciejjozefczyk_ has quit IRC15:48
johnsomJust create another load balancer from scratch, configure it the same as the other.15:48
lukskyaa OK15:48
lukskyI did it, twice15:48
lukskysame situation15:49
lukskybut, there may be some diffrences, will do it know once more time15:49
*** maciejjozefczyk has joined #openstack-lbaas15:49
*** vishalmanchanda has quit IRC15:51
*** vishalmanchanda has joined #openstack-lbaas15:51
johnsomUsually when I see this, one controller config file has a different key in the configuration file. So, when the amphora boots from controller B but the health message receives on controller A, the keys don't match and you get this hmac error. But we have kind of gone through that debug process. Now that you are down to one controller and all of the processes use the same configuration file, it should just work.15:52
*** gregwork has quit IRC15:55
*** TrevorV has joined #openstack-lbaas15:55
lukskyand now this new LB is working correctly (DON'T UNDURSTAND :/)15:56
lukskythis the new one15:56
lukskyhttps://pasteboard.co/IRt7VLL.png15:56
johnsomOk, so somewhere along the line we had a mis-match of something.15:57
lukskywill delete one of this member from this new shiny working LB and add it again15:57
lukskyit is online = works as expected15:58
lukskywill failover all other amphoras, and check now15:59
johnsomExcellent!15:59
lukskyin other (not working) LBs15:59
lukskythank You !15:59
lukskyI have three amphora in BOOTING state - (for a long time) - how can I delete them?16:01
luksky| 08b7d7f2-88f3-467a-9ee8-f77c3756d688 | None                                 | BOOTING   | None   | 10.99.99.26   | None            |16:02
luksky| 319af4dd-c70e-402c-8685-83a363cef313 | None                                 | BOOTING   | None   | 10.99.99.38   | None            |16:02
luksky| 9c9d7381-4ce2-4376-9bd5-8ee365d87152 | None                                 | BOOTING   | None   | 10.99.99.24   | None            |16:02
johnsomNova issues?16:02
lukskyyes...16:02
lukskythey are yp16:02
lukskyI mean - i can try to log in, but ssh key doesn't work16:03
lukskysomthing goes wrong with them16:03
johnsomWell, normally these will stop retrying and timeout. However, I'm guessing some of the controller stuff you have been doing killed the controller that owned those.16:03
lukskyok, can I delete this compute instances  (simply by openstack server delete) ?16:04
johnsomIt is important that you don't kill -9 the controllers, only gracefully shut them down. (We are working on this bug/issue right now)16:04
johnsomYes, openstack server delete should be ok here.16:04
lukskyok16:04
lukskythey will disappear from octavia database, or I need do some mysql clean up ?16:05
johnsomIf the controller is still retrying those, it will automatically move them to error and move on.16:05
johnsomIf not, you might need to delete them from the db (kill -9 scenario)16:06
lukskyok, thx16:06
lukskyfor i in `openstack loadbalancer amphora list | grep MASTER | awk -F"|" '{print $2}'`;do openstack loadbalancer amphora failover $i;done16:23
lukskyall LBs are online now :)16:23
*** yamamoto has joined #openstack-lbaas16:30
*** maciejjozefczyk has quit IRC16:31
*** luksky has quit IRC17:00
openstackgerritMerged openstack/octavia master: Fix the interface filenames for Red Hat amphora images  https://review.opendev.org/70051317:08
*** maciejjozefczyk has joined #openstack-lbaas17:09
johnsomAh, I am glad someone got time to research that and make sure it was right. I didn't have a devstack to load it up on to investigate this.17:10
openstackgerritMerged openstack/octavia stable/train: Fix multi-listener LB client auth/re-encryption  https://review.opendev.org/69860117:16
*** goldyfruit_ has quit IRC17:18
openstackgerritMerged openstack/octavia stable/rocky: Add listener and pool protocol validation  https://review.opendev.org/70204417:21
*** maciejjozefczyk has quit IRC17:24
*** tesseract has quit IRC17:27
*** rpittau is now known as rpittau|afk17:35
johnsomFYI, the next ubuntu LTS, 20.04 is starting to ship packages. It has haproxy 2.0.12 in it at the moment.17:41
johnsomhttps://packages.ubuntu.com/focal/haproxy17:41
*** servagem has quit IRC17:54
*** yamamoto has quit IRC17:56
*** yamamoto has joined #openstack-lbaas17:57
*** mithilarun has joined #openstack-lbaas18:16
*** yamamoto has quit IRC18:35
*** psachin has joined #openstack-lbaas18:47
openstackgerritMerged openstack/octavia stable/train: Revert "Fix gate issue with DIB"  https://review.opendev.org/69616519:07
*** yamamoto has joined #openstack-lbaas19:13
*** psachin has quit IRC19:23
*** yamamoto has quit IRC19:24
*** luksky has joined #openstack-lbaas20:24
*** goldyfruit_ has joined #openstack-lbaas20:26
*** TrevorV has quit IRC20:55
*** mithilarun has quit IRC20:59
*** mithilarun has joined #openstack-lbaas20:59
*** maciejjozefczyk has joined #openstack-lbaas21:04
*** mithilarun has quit IRC21:04
*** born2bake has joined #openstack-lbaas21:23
*** maciejjozefczyk has quit IRC21:23
openstackgerritAdam Harwell proposed openstack/octavia master: Update the lb_id on an amp earlier if we know it  https://review.opendev.org/69808221:26
openstackgerritAdam Harwell proposed openstack/octavia master: Conf option to use VIP ip as source ip for backend  https://review.opendev.org/70253521:26
openstackgerritAdam Harwell proposed openstack/octavia master: Allow AZ to override valid_vip_networks config  https://review.opendev.org/69952121:26
rm_worki should probably poke people about https://review.opendev.org/#/c/604043/ again because it will get them in a situation similar to what we saw above with the heartbeat keys and HM issues (though for different reasons)21:30
johnsomYep21:32
rm_workhmm rebasing tho and it looks like they may have adapted my code for the HM piece at least21:34
rm_workalthough i did some dumb things in that patch21:39
rm_workfixed21:42
johnsomNow if it would create the lb-mgmt-net.....21:44
rm_workwish i had any idea how to do that, lol21:44
johnsomrm_work Question for the failover flow.  Right now, we deploy the LB/listener(s), then deploy VRRP, then call ListenerStart.21:52
johnsomThe only thing ListenerStart does that matters is setup keepalived to monitor the deployed haproxy processes.21:52
johnsomNow with single proc, there really is only going to be one ID to monitor.21:53
rm_workso you could collapse that to just ... be part of the vrrp deploy?21:53
rm_workguessing that's what you're getting at?21:53
johnsomI am trying to decide if the haproxy monitor script needs to stay in ListenerStart (one more round trip) or if I should just merge it into the VRRP deploy code.21:53
rm_workright21:53
rm_worki would say you could probably combine them21:53
rm_worka lot of that structure was a little....21:54
rm_workobtuse21:54
rm_workerr, though I wonder about UDP, is that the same?21:54
johnsomWell, it was all over complicated because of that *#$% multi-proc decision21:54
rm_workyeah :(21:54
johnsomNo, that has it's own checks21:55
johnsomThis is purely HAproxy21:55
rm_workah so we deploy keepalived but it doesn't need to handle checks?21:55
rm_worki thought it still did the running of the check scripts21:55
rm_workthat's what it looked like to me when i was poking at it this week21:55
johnsomIt has it's own stuff21:56
rm_workerr, ok...21:56
johnsomIt's a whole different keepalived process21:56
rm_workah, yeah so still keepalived, but21:56
rm_workdifferent deployment mechanism for it?21:56
johnsomYEs21:56
rm_workk21:56
rm_workthen yeah, combine away21:56
johnsomThis literally just writes out a script with a list of haproxy PID files to go check21:57
johnsomJust trying to think if we would ever have a Haproxy process create/start/stop independent of setting up the VIP in an Active/Standby topology21:58
johnsomNah, with single proc I think we are fine21:59
*** mithilarun has joined #openstack-lbaas22:08
*** mithilarun has quit IRC22:10
*** mithilarun has joined #openstack-lbaas22:10
born2bakewhat can be an issue if health-monitors, housekeeping and api have this errors: https://pastebin.com/9cbQs9pG22:31
born2bakeand lb cant get operating status online22:31
johnsomThat one is pretty straight forward: "Lost connection to MySQL server during query"22:34
*** rcernin has quit IRC22:34
johnsomThere is something wrong with your mysql server or the network path from the controllers to that mysql server.22:34
openstackgerritBrian Haley proposed openstack/octavia-tempest-plugin master: Change to enable memory_tracker  https://review.opendev.org/70420222:34
born2bakei have 3 controllers, thing is that neutron is installed only on 2 of them, octavia on 3 of them22:36
johnsomMost Octavia deployments have three controller instances22:37
johnsomIf not more22:37
born2bakethat's weird cause other services are working fine and mariadb is fine as well22:38
johnsomI guess that helps narrow it down, something wrong with the host/instance with the Octavia controllers.22:40
johnsomAll of those errors are oslo_db related, there is no Octavia code there.  Other than the DB connection string, we don't set any oslo_db settings, it is all defaults.22:41
johnsomNow, I don't know if kolla is doing something it shouldn't to oslo_db or adding oslo_db settings to our configuration files.22:42
rm_workjohnsom: ah i need to glance at my multi-vip patch (which i really need to rebase, but not looking forward to that) and see if your combination affects me there22:43
johnsomYou can check that no one added to our config, look at the [database] section in the octavia.conf. It should only contain a connection string setting.22:43
born2bakeworker and api logs are cool without errors22:46
born2bakeand when i am trying to open lb ip in browser got 503 Service Unavailable msg22:49
johnsomOpening the Octavia LB VIP IP? That means there are no member servers healthy in the pool.22:55
openstackgerritAdam Harwell proposed openstack/octavia master: Support HTTP and TCP checks in UDP healthmonitor  https://review.opendev.org/58918022:57
*** born2bake has quit IRC23:01
openstackgerritBrian Haley proposed openstack/octavia-tempest-plugin master: Create new octavia-dsvm-base-py3 job name  https://review.opendev.org/70420423:03
openstackgerritBrian Haley proposed openstack/octavia-tempest-plugin master: Change to enable memory_tracker  https://review.opendev.org/70420223:10
johnsomhaleyb Yeah, if nobody has it enabled by default, maybe we don't want it on by default either. Maybe it's more of a "turn it on if you have oom problems" type of thing23:18
haleybjohnsom: tobiko did i think, not many others.  like i mentioned, first step in getting job working again, guess i'll need a dependent patch to see23:20
johnsomI still see peakmem enabled in the devstack repo too.  But, yeah, I would maybe just delete that line all together23:22
openstackgerritBrian Haley proposed openstack/octavia master: DNM: Dependent patch for two-node job debugging  https://review.opendev.org/70420523:22
johnsomA bunch of that was copied over from the main jobs since zuul has that issue with replacing instead of merging partent/child at certain levels.23:22
haleybjohnsom: https://www.youtube.com/watch?v=qpGx4foRdPw23:23
haleybbeer-o-clock here23:23
johnsomo/23:24
« Thursday, 2020-01-23 Index Saturday, 2020-01-25 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!