Thursday, 2020-04-09

openstackgerritMerged openstack/octavia master: Jobboard based controller  https://review.opendev.org/64740600:10
openstackgerritNoah Mickus proposed openstack/python-octaviaclient master: Add the ability to specify the cipher list for a listener  https://review.opendev.org/71781100:24
rm_workjohnsom: hmm, looking at the way the DB-down detection was done, and actually this is a little fragile still00:25
johnsomHow so?00:26
rm_workit tries to do a DB read and if it fails, it sleeps for DELAY seconds and tries again00:26
rm_workthe IDEA is that there's <DELAY> seconds after the DB comes back up for stuff to check in00:27
rm_workbut ... the DB could come back up in the last second of that sleep period, and then it could try the check again and have it work00:27
johnsomRight, it captures the exception than sleeps the full check interval (60 seconds)00:27
rm_workright00:27
rm_workit needs to sleep the full DELAY *after* the DB comes up from being down00:27
rm_worknot just... each time the DB is detected down00:27
johnsomYou sure the loop isn't setup that way?00:28
rm_worki'll look again, but... fairly sure00:28
johnsomIn other news, the AZ dashboard patch is giving me trouble00:28
rm_workah wait, maybe it is00:28
rm_workno you're right, phew00:28
rm_workmissed that the exception code has the wait() in there00:28
openstackgerritNoah Mickus proposed openstack/python-octaviaclient master: Add the ability to specify the cipher list for a listener  https://review.opendev.org/71781100:29
rm_worktrying to figure out how to detect Read-Only mode on the DB and do the same Wait for it to be not read-only...00:32
johnsomYeah, that is... hard unless you want to hose up your DB with small write tests00:33
rm_workyeah :/00:34
johnsomThere is probably a mysql query, but that would break the ORM illusion00:34
rm_workwas looking to see if there was something in mysql I can READ00:34
rm_workthat will just tell me it's RO without having to try a write00:34
rm_workbut i guess that's not portable00:34
rm_worki could work on moving the health table to redis :D00:35
johnsomI'm 90% sure there is, but again, that would defeat the purpose of oslo.db/sqlalchemy00:35
rm_worknow that we have something else requiring it ;)00:35
johnsomNo argument from me00:35
rm_workactually looking at this code, i don't really understand how it would do a failover on a RO DB...  i guess it's just, the instant the DB comes back up...00:37
rm_workbecause get_stale_amphora() does a with_for_update() select, which SHOULD require read-write?00:37
rm_workright?00:37
rm_workwhich ... MIGHT kick us into that DBConnectionError exception block...00:38
rm_workso in that case, maybe the wait_for_connection() just needs to verify it can write, and we'd be fine?00:38
rm_workwill need to spin up devstack to test00:38
rm_work(and figure out how to make mysql R/O)00:39
openstackgerritNoah Mickus proposed openstack/python-octaviaclient master: Add the ability to specify the cipher list for a pool  https://review.opendev.org/71783400:39
nmickusjohnsom The ciphers cli patches should be updated per your comments00:40
johnsomUgh, I am very rusty at even trying to debug horizon.00:41
johnsomnmickus Thanks!00:41
rm_workhmmm i wonder if i start a transaction if that will fail on a R/O DB00:50
johnsomUgh, I wonder if this is just the horizon funkyness with python2/python3...00:57
johnsomAh, ok. The underscore strikes again01:01
*** yamamoto has joined #openstack-lbaas01:02
johnsomdashboard has none, openstacksdk has one01:02
johnsomsorrison If you are around, the dashboard patch needs work01:02
johnsomI am flipping back to looking at the client patches01:02
sorrisonjohnsom: yip, I'm around. Just that 1 minor thing?01:05
johnsomI think so.... However it blocks me from really testing.01:05
johnsomHope you can take a quick look and confirm that is the only issue there while I pivot to review a few other patches.01:06
johnsomI'm already an hour into "I should be making dinner" time, so trying to maximize my review minutes01:06
openstackgerritSam Morrison proposed openstack/octavia-dashboard master: Availability zone support  https://review.opendev.org/71456301:08
johnsomsorrison So I take it what I captured made sense?01:09
sorrisonyeah just a minor bug, not sure how I missed that as I had it working but then realised my openstack sdk version wasn't what I submitted upstream01:11
johnsomIs everything ok in the sdk that did go upstream?01:12
* johnsom hopes so as I reviewed it and the release window is closed....01:12
sorrisonyeah the sdk is all good01:16
johnsomOh good01:17
johnsomSurvey says!01:23
johnsomsorrison That works01:23
rm_workjohnsom: so I can do a SELECT FOR UPDATE with a junk WHERE clause and it'll give an ERROR if the DB is R/O01:23
rm_workso like...01:23
rm_workSELECT id FROM amphora WHERE id=0 FOR UPDATE;01:24
rm_worknever going to match a real amp, but still throws the error due to the FOR UPDATE clause01:24
rm_work`ERROR 1223 (HY000): Can't execute the query because you have a conflicting read lock`01:24
johnsomrm_work Umm, do me a favor, shut things down, restart mysql, do whatever it was that set it RO, and try again. That is not the error I would expect. That implies there is another hung transaction still hanging around01:25
rm_worksure, but AFAIU the way it works is that you do a FLUSH WITH READ LOCK01:26
rm_workhttps://support.scalearc.com/kb/articles/how-to-set-a-mysql-database-to-read-only-mode01:26
rm_workthe global "read_only" means nothing apparently, i was able to UNLOCK the table manually and do writes even, with read_only still true01:27
johnsomNope, flush with read lock just unlocks any table that has a for update lock01:27
johnsomThe global is along the right path of what I would expect though01:28
rm_workerr, i don't think so01:28
rm_workone sec01:28
johnsomYeah, that error is definitely not what you get when mysql is in RO01:29
johnsomLike with that you can't do a select right?01:30
rm_workhttp://paste.openstack.org/show/791833/01:30
rm_workno, i can select01:30
rm_workbut not write01:30
rm_workand yes, the command seems nonsensically worded01:31
rm_workfor what it actually does01:31
rm_workpff, DBAs, am i right? lol01:31
johnsomI am more concerned about the error you got01:31
rm_workseems right01:32
rm_workI think "read lock" means "lock for reading only"01:32
rm_work? apparently01:32
rm_worki mean it's pretty obvious with that paste01:32
johnsomYeah, ok, no that is a different state.... Undo that01:33
rm_workyeah easy enough -- per that guide, `UNLOCK TABLES;` does it01:33
johnsomModify your mysql systemd service, add --read-only=ON as a parameter, startup and try01:33
rm_workhmm, DBA here just confirmed what i pasted is what we do, lol01:33
rm_workbut will try yours01:34
johnsom"what we do"? So this isn't like a percona or mysql read only, but something "custom"?01:34
rm_worki mean, it's "how you put a mysql db into readonly mode"01:34
johnsomNO01:34
johnsomI just gave you how to do that.01:35
rm_workaccording to "the internet" and apparently our runbooks lol01:35
johnsomright...01:35
rm_workI don't think we restart them when we do that?01:35
rm_workyours requires restarting them01:35
johnsomI'm just saying, if the "my database just hosed and went RO or I just connected to the RO slave" is the case you want to fix in Octavia, those commands aren't going to get you there01:35
rm_worklol01:38
rm_workset --read-only=ON and it still lets me write01:39
rm_workall it does is set that read_only global var, apparently01:40
rm_workwhich does nothing01:40
rm_worki can still write to tables01:40
johnsomThat is not right01:40
rm_workhttp://paste.openstack.org/show/791834/01:40
johnsomCheck the account you are using, is it SUPER or CONNECTION_ADMIN?01:41
rm_workprolly, i'm root01:41
johnsomWell, there you go, accounts with SUPER bypass the RO mode01:41
rm_workO_o01:41
rm_workthat's a shitty RO mode01:42
johnsomHave you met mysql?01:42
rm_workno wonder everyone uses this other thing instead01:42
johnsomTry with the octavia account01:42
johnsomor devstack or whatever01:42
rm_workliterally in the official mysql docs: https://dev.mysql.com/doc/refman/5.7/en/replication-solutions-backups-read-only.html01:42
johnsomYes, but you are missing my point that the lock line is doing something else.01:43
rm_work? it's doing a lock on all tables such that things can read but not write01:43
rm_workseems to be working exactly as it's supposed to according to docs01:43
johnsomThe error you get with just a RO and not LOCK is different01:43
rm_workhttps://dev.mysql.com/doc/refman/8.0/en/flush.html#flush-tables-with-read-lock01:44
rm_workpossibly, but as far as I can tell, official advice is to do the LOCK method for R/O01:45
johnsomDude, I have read all of that and refreshed my memory. If you don't believe me or care, implement it that way but don't come to me the next time you run into this problem01:45
rm_workI mean, what you're telling me conflicts with what the mysql docs are saying01:45
johnsomNo it does nto01:46
johnsomnot01:46
rm_workhttps://dev.mysql.com/doc/refman/5.7/en/replication-solutions-backups-read-only.html01:46
rm_work"Put the master M1 in a read-only state by executing these statements on it:"01:46
johnsomYes, statements. First one doing lock junk for snapshot stuff, etc. The second transitioning into actual RO. DB failures and RO slaves do not do the LOCK.01:47
johnsomIf the slaves did the LOCK line, then the masters would start throwing that error you got too01:47
johnsomI am off to dinner. good luck01:49
rm_worklooking at an actual slave db to compare01:51
*** yamamoto has quit IRC02:04
rm_workdo we actually support anything besides MySQL (and sqlite for testing)?02:08
rm_workopenstack stopped supporting postgres officially?02:08
*** yamamoto has joined #openstack-lbaas03:09
*** logan- has quit IRC04:45
*** logan_ has joined #openstack-lbaas04:47
*** logan_ is now known as logan-04:48
*** tamerlanabu has joined #openstack-lbaas05:09
tamerlanabuHi!05:09
tamerlanabuI have a question regarding amphorae and health manager. How does amphorae determine where health manager is located and where it should be sending heartbeats to?05:10
tamerlanabuRight now it tries to send UDP packages to localhost:5555 which is not correct.05:10
*** tamerlanabu has quit IRC05:14
*** tamerlanabu has joined #openstack-lbaas05:15
*** armax has quit IRC05:28
*** yamamoto has quit IRC05:59
*** yamamoto has joined #openstack-lbaas06:15
*** vishalmanchanda has joined #openstack-lbaas06:59
*** gcheresh has joined #openstack-lbaas07:01
*** maciejjozefczyk has joined #openstack-lbaas07:10
*** rpittau|afk is now known as rpittau07:18
*** psachin has joined #openstack-lbaas07:37
*** threestrands has quit IRC08:00
*** gcheresh has quit IRC08:14
cgoncalvestamerlanabu, hi. amphorae send heartbeats to every health manager listed in "controller_ip_port_list" in a round-robin fashion. in your case, it seems that your "controller_ip_port_list" option is not set correctly.08:24
cgoncalvesit should be a list of IP:port of your health manager ports in the loadbalancer management network (lb-mgmt-net)08:25
cgoncalveshttps://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list08:25
tamerlanabucgoncalves, thanks, figured it out.08:31
*** tamerlanabu has quit IRC08:31
*** gcheresh has joined #openstack-lbaas08:38
*** tobias-urdin has quit IRC09:02
*** tkajinam has quit IRC09:03
*** born2bake has joined #openstack-lbaas09:28
*** tobias-urdin has joined #openstack-lbaas09:30
*** ataraday_ has joined #openstack-lbaas09:50
*** gcheresh has quit IRC09:56
*** rpittau is now known as rpittau|bbl10:13
*** rpittau|bbl is now known as rpittau11:57
*** gcheresh has joined #openstack-lbaas12:08
*** psachin has quit IRC12:26
*** jamesden_ has joined #openstack-lbaas12:55
*** yamamoto has quit IRC12:57
*** jamesdenton has quit IRC12:57
*** yamamoto has joined #openstack-lbaas12:58
openstackgerritMerged openstack/octavia master: Add availability-zone to loadbalancer object docs  https://review.opendev.org/71635813:04
*** rcernin has quit IRC13:54
*** yamamoto has quit IRC14:00
*** TrevorV has joined #openstack-lbaas14:01
*** yamamoto has joined #openstack-lbaas14:02
*** yamamoto has quit IRC14:02
*** yamamoto has joined #openstack-lbaas14:02
*** gcheresh has quit IRC14:04
*** yamamoto has quit IRC14:06
*** yamamoto has joined #openstack-lbaas14:06
*** tkajinam has joined #openstack-lbaas14:13
*** gcheresh has joined #openstack-lbaas14:14
*** yamamoto has quit IRC14:32
*** ataraday_ has quit IRC14:33
*** vishalmanchanda has quit IRC15:09
*** armax has joined #openstack-lbaas15:10
*** yamamoto has joined #openstack-lbaas15:11
*** yamamoto has quit IRC15:23
openstackgerritMerged openstack/octavia-dashboard master: Availability zone support  https://review.opendev.org/71456315:36
*** maciejjozefczyk has quit IRC15:49
*** tkajinam has quit IRC15:49
*** rpittau is now known as rpittau|afk15:59
johnsomnmickus I'm going to see if I can figure out why the clients are not sending the tls_ciphers field. I have done something similar myself, I think it is a variable mis-match or something similar16:00
nmickusjohnsom alright, it seemed to be working fine for me for last night but I might not have had it set up right16:01
johnsomYou did have it working?  hmmm16:02
johnsomWhen I run it, nothing gets sent and when I use the --debug flag the json body doesn't have the field16:02
nmickusMight not have had it set up right though16:03
nmickusis it both the pools and listeners?16:03
johnsomI haven't tried the pools yet, I started with listeners16:05
nmickusalright let me know if you find anything16:06
nmickusi would assume it would have a similar problem16:06
johnsomYeah, they are basically the same patch16:08
johnsomI am running this:16:09
johnsomopenstack --debug loadbalancer listener set --tls-ciphers AES 2dee12bd-adc6-4a1a-9756-3cb9bb7d619416:09
johnsomas an example. I expect it to come back with an error, but the debug request json going to the API is:16:09
johnsomREQ: curl -g -i -X PUT http://10.21.21.78/load-balancer/v2.0/lbaas/listeners/2dee12bd-adc6-4a1a-9756-3cb9bb7d6194 -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/3.18.0 python-requests/2.23.0 CPython/3.6.9" -H "X-Auth-Token: {SHA256}e0b1719d22d68457deea1add6f6ca987fb61cac26d96e6369682699898528b49" -d '{"listener": {}}'16:09
johnsomI think it is this: https://github.com/openstack/python-octaviaclient/blob/master/octaviaclient/osc/v2/utils.py#L18616:12
*** Trevor_V has joined #openstack-lbaas16:16
nmickusLooks like it could be it16:16
nmickusHow do I make sure my changes are used in devstack for testing?16:17
*** TrevorV has quit IRC16:19
johnsomFor the client?16:20
nmickusyeah16:20
johnsomjust "sudo pip uninstall -y python-octaviaclient; sudo pip install ."16:20
johnsomWhen you are in the client code directory16:20
johnsomYeah, fixing the mapping table solved the problem16:26
nmickusAwesome I'll update that and test it16:28
nmickusjohnsom I'm using that command and its returning an error saying `Missing value auth-url required for auth plugin password` is that would i should be seeing?17:01
johnsomThat means you have no Openstack credential, you haven't logged in.17:02
nmickusrunning it with debug I can't seem to find the json values in the output17:02
johnsomcd /opt/stack/devstack; source openrc admin admin17:02
*** ccamposr__ has quit IRC17:02
johnsomYeah, it is failing before it even hits our code because you were not logged in17:03
nmickusalright it's giving me an error running your command `clean_up SetListener: public endpoint for load-balancer service in RegionOne region not found`17:08
nmickusand when i run it without arguments "tls_ciphers" shows up in the json17:08
nmickusjohnsom does that sound correct to you?17:12
johnsomHmm, that sounds like Octavia isn't installed if it can't find the service.17:13
johnsomWithout arguments I bet that is from a show/lookup response.17:15
johnsomIf it got a response with that in the body it must be installed, so that is odd.17:15
*** ccamposr has joined #openstack-lbaas17:16
nmickusits in my local.conf17:18
johnsomyeah, I don't get why that endpoint not found was there. I guess just ignore that.17:18
nmickushmm well if it sounds right I'll git review17:19
johnsomYeah, go for it. I can also pull it down and test it.17:19
openstackgerritNoah Mickus proposed openstack/python-octaviaclient master: Add the ability to specify the cipher list for a listener  https://review.opendev.org/71781117:20
nmickusAlright if that works for you as well, I will update the pools17:22
johnsomOk, give me 10 mintues17:22
johnsomnmickus Yes, that version seems to be working17:28
nmickusawesome i will get on fixing the pools patch17:31
johnsomThank you!17:31
openstackgerritNoah Mickus proposed openstack/python-octaviaclient master: Add the ability to specify the cipher list for a pool  https://review.opendev.org/71783417:59
openstackgerritNoah Mickus proposed openstack/python-octaviaclient master: Add the ability to specify the cipher list for a listener  https://review.opendev.org/71781118:07
rm_workFYI again: if you use `pip install -e .` it will *just use* your local version of the code rather than copying it in, so it will ALWAYS be up to date with every change you make without having to do another install every time18:15
rm_workI always recommend that18:15
*** ccamposr__ has joined #openstack-lbaas18:17
johnsomOk, for those playing the home edition of Octavia, I'm testing the TLS ciphers with:18:19
johnsomopenstack loadbalancer listener set --tls-ciphers TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256 2dee12bd-adc6-4a1a-9756-3cb9bb7d619418:20
johnsomThen a connection:18:20
johnsomhttps://www.irccloud.com/pastebin/tK9wIKvf/18:20
johnsomWhich is the result I want to see18:20
*** ccamposr__ has quit IRC18:21
*** ccamposr has quit IRC18:23
johnsomhttps://www.irccloud.com/pastebin/elcBITDR/18:25
johnsomGroovy18:27
*** gcheresh has quit IRC18:43
cgoncalvesnice!18:49
cgoncalvesthe "home edition of Octavia" killed me xD18:49
*** gcheresh has joined #openstack-lbaas18:51
rm_workLOL cgoncalves19:19
rm_workwe approved at EXACTLY the same time19:19
rm_workjohnsom: BTW did you change your mind on https://review.opendev.org/#/c/711298/ because IIRC you looked and found other commands weren't adding everything either?19:22
johnsomrm_work What did we decide on that???? Maybe others have an opinion?19:22
rm_work*I* personally don't think --long needs to add EVERYTHING19:23
rm_workwe can always tweak what it includes (to include more) later19:23
rm_worki consider it like .... primary tier by default, then secondary tier info with --long, and tertiary tier info with show19:24
johnsomcgoncalves xgerman Have an opinion?19:24
rm_worklike, common info for everyone; common info for people working an issue; debugging info19:24
johnsomYeah, show can't list though. It's like --long --wide --all --kitchen-sink19:24
rm_workyeah but i mean, for tertiary info, yeah, just use ashow19:25
rm_workdon't rely on list19:25
johnsomI just don't want to have to change it later as it's a think people code to, etc.19:25
xgermanI like kiutchen sink19:25
rm_workpeople shouldn't be coding in a way that doesn't allow new fields to exist19:25
rm_workthat's ALREADY how our api works19:25
rm_workwe already agree adding new fields is always OK, that's the contract for us right?19:25
johnsomhttps://review.opendev.org/#/c/711298/19:26
rm_workremoving would break things but adding....19:26
rm_worksorry yes, forgot to link19:26
johnsomYeah, we rarely add to list though.19:26
rm_workoh, no, i did link :D19:26
rm_workyeah, but I don't see it as breaking19:26
johnsomLike that role column. It seems like every week I regret we added "role" to the amphora cli....19:27
xgermanI think —wise would probably be ok.19:28
xgermanWide19:29
xgermanKube does something similar19:29
johnsomThe question is should --long be some more columns or all columns or ???19:29
johnsomrm_work Can we leave out role?19:30
rm_workerrr19:30
johnsomI guess it is in the base one isn't it.19:30
johnsomsigh19:30
openstackgerritMerged openstack/python-octaviaclient master: Fix doc8 check  https://review.opendev.org/71353119:31
openstackgerritMerged openstack/python-octaviaclient master: Add the ability to specify the cipher list for a pool  https://review.opendev.org/71783419:32
openstackgerritMerged openstack/python-octaviaclient master: Add the ability to specify the cipher list for a listener  https://review.opendev.org/71781119:32
xgermanCall it operator-view and omit role?19:33
rm_worklol19:34
johnsomlol19:34
rm_workthat is a whole different question19:34
rm_work:D19:34
johnsomI want to hack the client so that role always shows "Unkown" unless you add the --I-know-this-isn't-a-status-field parameter19:35
openstackgerritMerged openstack/python-octaviaclient master: Add --long to amphora-list to show more columns  https://review.opendev.org/71129820:05
*** gcheresh has quit IRC21:04
*** Trevor_V has quit IRC21:05
*** ramishra has quit IRC22:03
*** born2bake has quit IRC22:14
*** tkajinam has joined #openstack-lbaas22:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!