| *** ysandeep|out is now known as ysandeep | 05:09 | |
| *** ysandeep is now known as ysandeep|lunch | 08:31 | |
| *** ysandeep|lunch is now known as ysandeep | 09:15 | |
| opendevreview | Gregory Thiemonge proposed openstack/octavia master: Fix PING health-monitor with recent haproxy releases https://review.opendev.org/c/openstack/octavia/+/836149 | 11:49 |
|---|---|---|
| opendevreview | Gregory Thiemonge proposed openstack/octavia master: Fix PING health-monitor with recent Centos releases https://review.opendev.org/c/openstack/octavia/+/836170 | 11:49 |
| *** ysandeep is now known as ysandeep|afk | 11:51 | |
| *** ysandeep|afk is now known as ysandeep | 13:28 | |
| *** ysandeep is now known as ysandeep|out | 14:53 | |
| Ammad | hello | 19:14 |
| Ammad | Anyone available from octavia team ? | 19:14 |
| johnsom | Hi | 19:15 |
| Ammad | Need some help with octavia 9.0.1 | 19:15 |
| Ammad | Hi Johnsom | 19:15 |
| johnsom | What can we help with? | 19:16 |
| Ammad | I have upgraded my octavia to 9.0.0 to 9.0.1. I am unable to create pool in loadbalancer | 19:16 |
| Ammad | 2022-04-02 00:10:12.163 7620 DEBUG wsme.api [req-74fd3a57-75b8-48c1-b636-387ef534c2ea - 96301fdbcc1c46fb8e00bb5f07f7d16d - default default] Client-side error: is not a valid option for format_exception /usr/lib/python3/dist-packages/wsme/api.py:222 | 19:16 |
| Ammad | this is the error I am getting while creating simple HTTP pool | 19:17 |
| johnsom | Hmm, that is pretty odd, there wasn't much change in 9.0.1. There isn't more information or a traceback in the API log?? | 19:18 |
| Ammad | I see only three commits from 9.0.0 to 9.0.1 | 19:18 |
| Ammad | https://opendev.org/openstack/octavia/commit/4039d35ce221b58490d4aa5b3497e29dbf7664ac | 19:18 |
| Ammad | This one looks major | 19:18 |
| Ammad | No more trace back api logs just say this log while creating | 19:19 |
| johnsom | Yes, but that is all worker side. Your error is happening before it gets there. I'm cluing off the "client-side-error" in the log entry | 19:19 |
| johnsom | Ok, can you run your pool command with the --debug flag? Like this "openstack --debug loadbalancer pool create ..." | 19:20 |
| johnsom | Then paste bin it to paste.openstack.org | 19:20 |
| Ammad | Let me | 19:21 |
| johnsom | Yeah, I see three patches as well, none of which made any changes to the API service, so I think it's something else at play here. | 19:28 |
| Ammad | https://paste.openstack.org/show/813632/ | 19:29 |
| Ammad | here it is | 19:29 |
| johnsom | Thank you, reading | 19:29 |
| johnsom | Did you upgrade using kolla? | 19:30 |
| johnsom | Also, which provider driver did you use for the load balancer? | 19:33 |
| Ammad | No the installation is manual on ubuntu 20.04 vm | 19:33 |
| Ammad | using amphora | 19:33 |
| johnsom | Ok, really interesting. Give me a minute to look at some code. | 19:34 |
| Ammad | not using job board | 19:34 |
| johnsom | Oh good | 19:34 |
| johnsom | Ok, this is a database error. Did you do the database migration during the upgrade? | 19:37 |
| Ammad | No | 19:38 |
| johnsom | Ok, let's look at doing that as a starting point here. | 19:38 |
| Ammad | octavia-db-manage --config-file /etc/octavia/octavia.conf upgrade head | 19:39 |
| johnsom | yep | 19:39 |
| Ammad | Do I use this upgrade command ? | 19:39 |
| johnsom | I get: | 19:40 |
| johnsom | https://www.irccloud.com/pastebin/wLkfikLA/ | 19:41 |
| johnsom | As my database is up to date | 19:41 |
| johnsom | I'm guessing you are going to see some migrations run | 19:41 |
| Ammad | root@loadbalancer-khi04:~# su -s /bin/bash octavia -c "octavia-db-manage --config-file /etc/octavia/octavia.conf upgrade head" 2022-04-02 00:40:32.029 7865 INFO alembic.runtime.migration [-] Context impl MySQLImpl. 2022-04-02 00:40:32.029 7865 INFO alembic.runtime.migration [-] Will assume non-transactional DDL. | 19:41 |
| Ammad | Didn't see any migration run | 19:42 |
| johnsom | Hmmm, ok | 19:42 |
| Ammad | seen the same output as yours | 19:42 |
| johnsom | How do you feel about making a logging code change to the code, restarting the API service? | 19:43 |
| johnsom | Is see the (not good code) that is raising this, but it's not logging the real issue. The author was kind enough to leave a TODO to fix it. lol | 19:44 |
| Ammad | I am not good with coding. But with your help I can edit code and restart the service | 19:45 |
| johnsom | Ok, no problem, this is a super small change. Give me one minute and I will provide instructions | 19:45 |
| Ammad | Sure | 19:45 |
| johnsom | This is the line raising the error: https://github.com/openstack/octavia/blob/stable/xena/octavia/api/v2/controllers/pool.py#L153 | 19:46 |
| Ammad | Ohh | 19:47 |
| johnsom | We are going to edit the octavia/api/v2/controllers/pool.py file. You will find it under /usr/lib/python3(8 or 9). | 19:49 |
| johnsom | At line 153, we will add a log line: | 19:50 |
| johnsom | https://www.irccloud.com/pastebin/x0W0NTwr/ | 19:50 |
| johnsom | Once, you do that, restart the octavia API process, retry your command, and pastebin me the error log message from the API log | 19:51 |
| Ammad | Ok let me do it | 19:51 |
| Ammad | https://paste.openstack.org/show/813634/ | 19:54 |
| Ammad | Here are the api debug logs | 19:54 |
| Ammad | Looks like some DB issue | 19:55 |
| Ammad | I am using percona xtradb cluster 8.0 | 19:56 |
| johnsom | Yeah, I have not seen this outside of a case where the DB migration wasn't run. Let me think about this for a second | 19:56 |
| johnsom | Oh, the API isn't pointing to a secondary instance that is out of sync is it? | 19:56 |
| johnsom | I mean, protocol "HTTP" should have been in there from the start, so... very odd | 19:58 |
| Ammad | There is only one VM of all octavia services not multiple. | 19:59 |
| johnsom | Yeah, I was talking about percona instances | 19:59 |
| Ammad | No, its sync with group. All other services like keystone nova neutron cinder heat magnum works fine. | 20:00 |
| johnsom | Can you connect to the "octavia" database and do a "select * from protocol;" | 20:00 |
| Ammad | let me | 20:00 |
| johnsom | You should see HTTP in that list | 20:00 |
| johnsom | https://www.irccloud.com/pastebin/KFqIO8ZG/ | 20:00 |
| johnsom | Mine is a newer version, so there may be a few extra in my list. | 20:01 |
| Ammad | https://paste.openstack.org/show/813635/ | 20:01 |
| Ammad | HTTP is there | 20:02 |
| Ammad | SET GLOBAL FOREIGN_KEY_CHECKS=0; | 20:19 |
| johnsom | That is very dangerous. I would not recommend setting that | 20:19 |
| Ammad | Ohh | 20:19 |
| Ammad | What else can we do to fix this ? | 20:22 |
| Ammad | will reverting back to 9.0.0 helps ? | 20:23 |
| johnsom | No, this is not related to the 9.0.1 change, there was no change to the protocols table in the database or the API tier. The real question here is why is the DB saying the foreign key isn't there, where you can see it is there. | 20:26 |
| Ammad | Look like its bug in percona xtadb | 20:30 |
| Ammad | Look like its bug in percona xtadbxtradb 8.0 | 20:30 |
| johnsom | Yeah, I can't see why that insert should fail | 20:31 |
| Ammad | What SQL version you recommend to use ? | 20:32 |
| Ammad | mariadb ? mysql ? | 20:32 |
| johnsom | We test on mariadb | 20:33 |
| Ammad | Let me migrate the DB to mariadb 10.6 | 20:34 |
| Ammad | Let see if this fixes the issue | 20:34 |
| johnsom | Yeah, looking at your snippet, the INSERT SQL and the foreign key all look correct and this hasn't changed in years in the code. I really don't know why it would say it's invalid. The only thing I can think is the database had a locale change somewhere along the line and the tables weren't updated. Like an admin changed to UTF8 in the database tools but forgot to update the tables, etc... | 21:14 |
| johnsom | Maybe a show create table on both the pool and the protocols tables and make sure the engine settings are the same, etc. | 21:15 |
| johnsom | I am going to open a bug though, that error message you got is horrible and that TODO needs to be fixed. | 21:16 |
| johnsom | https://storyboard.openstack.org/#!/story/2009957 | 21:24 |
| Ammad | I have installed mariadb 10.6 and tried to restore the dump but it gives below error | 21:27 |
| Ammad | Enter password: ERROR 1273 (HY000) at line 34: Unknown collation: 'utf8mb4_0900_ai_ci' | 21:27 |
| johnsom | Yeah, that is the engine setting I was wondering about. if some admin changed that on some tables. | 21:27 |
| johnsom | Ammad All of mine have: ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 | | 21:28 |
| johnsom | I get that output from "show create table protocol;" and "show create table pool;" | 21:29 |
| Ammad | I have then followed the link https://tecadmin.net/resolved-unknown-collation-utf8mb4_0900_ai_ci/ and changed it via sed to utf8 | 21:30 |
| Ammad | and restored the dump | 21:30 |
| Ammad | Now pool creation is working fine. | 21:30 |
| johnsom | Yeah, so it's very likely something/someone changed the DB to make the charset/collation inconsistent in the DB. I don't we set that in our DB migrations, it just uses the default for the database at the time the migration is run. | 21:32 |
| Ammad | I think this utf8mb4 is default for percona xtradb 8.0 | 21:34 |
| johnsom | I wonder if your database was upgraded sometime after the initial migration run of octavia | 21:34 |
| Ammad | Yes, I have upgraded from octavia 8.0 to 9.0 | 21:35 |
| johnsom | I meant the percona version | 21:35 |
| johnsom | From Octavia perspective, we don't really care as long as it's a UTF8 compatible and all of the tables have the same setting. | 21:35 |
| Ammad | Yes, I have upgraded the percona 8.0.20 to 8.0.26 once. | 21:36 |
| Ammad | Yes i think utf8mb4 is also backward compatible. | 21:37 |
| johnsom | That might have led us here. There might have been a charset/collation migration step missed. | 21:37 |
| Ammad | Ohh. Will this be ok that I should restore the dump back to percona from mariadb ? | 21:38 |
| johnsom | If you still have the old percona DB, it would be interesting to check those settings on those two tables. You could probably migrate the tables to one of the collations and continue to use the percona setup | 21:39 |
| Ammad | its now from utf8mb4 to utf8mb3. | 21:39 |
| johnsom | As long as all of the octavia tables are consistent with the collation, I don't think it will be a problem. | 21:40 |
| Ammad | I have upgraded percona months back. I have to see if the backup of old db is available. | 21:40 |
| Ammad | Let me migrate the dump back to percona and see if it works | 21:41 |
| johnsom | Ok, and check those two tables after you restore. | 21:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!