| *** jpena|off is now known as jpena | 07:01 | |
| *** whoami-rajat__ is now known as whoami-rajat | 11:33 | |
| dansmith | zzzeek: if you get a chance, I wonder if you could have a look at the ML thread, starting hereish: http://lists.openstack.org/pipermail/openstack-discuss/2022-June/029096.html | 13:42 |
|---|---|---|
| dansmith | dbcounter is my sqla plugin thing, which works fine i the gate, and fine locally for most people, | 13:42 |
| dansmith | but a small number end up with that inability to load the plugin, maybe only on glance | 13:43 |
| dansmith | it's installed properly and other projects are able to load it both before and after glance fails with it | 13:43 |
| dansmith | and it's not constant, obviously, because it works a lot of the time | 13:43 |
| zzzeek | so it looks like the url is coming in with the wrong data somehow | 13:46 |
| zzzeek | have they viewed the conf file on an env that reprodcues the issue? | 13:46 |
| zzzeek | only glance right? | 13:47 |
| dansmith | zzzeek: the conf file is right (see one of my later replies) | 13:48 |
| dansmith | seems to be only glance so far | 13:48 |
| zzzeek | dansmith: ok but you haven't reprodcued it locally right | 13:48 |
| dansmith | zzzeek: here's an example config https://zuul.opendev.org/t/openstack/build/ff5a7a6fc87840579b102b244f28200f/log/controller/logs/etc/glance/glance-api_conf.txt#21 | 13:48 |
| dansmith | zzzeek: I can't yeah, some people have had it, obviously.. never happens in gate that we know of | 13:49 |
| zzzeek | i am saying, when the error happens, right there, whats in the config, is something writing into that file or something or are we completely sure the URL is correct in the conf | 13:49 |
| zzzeek | dansmith: any chance your URL appender can maybe check the existing URL to see if that plugin token is already present, or otherwise log what it's seeing? | 13:49 |
| dansmith | when melwitt is around I'll see if she still has a system stuck in this state | 13:50 |
| dansmith | since we only write the config in devstack early, and we know the configs are getting created properly in other projects (and it's the same function that makes the url) I had assumed that if it was right in one run it was right in her config | 13:51 |
| dansmith | but I guess maybe we haven't checked that specifically, now that I think about it | 13:52 |
| dansmith | and for log the actual thing, you mean log our CONF.connection separate from sqla processing it to see if we're loading it right and sqla is mangling it (or not)? | 13:52 |
| zzzeek | dansmith: well there's a chain of ownership here w/ that URL and somewhere it's beraking, so , log everything you can until you find where it breaks? | 13:53 |
| zzzeek | dansmith: like, I dont know a way that a correct URL can get into create_engine() and then somehow get corrupted like that | 13:53 |
| zzzeek | dansmith: i'd want to see, most likely, what's being passed to oslo.db create_engine | 13:54 |
| dansmith | right and I expect we're not doing anything with it other than passing it in, but maybe something is I'll go look | 13:54 |
| dansmith | I thought maybe since sqla is parsing the url (and we're not in most cases, AFAIK) that it was more likely something in sqla | 13:54 |
| zzzeek | even if the bug is in SQLAlchemy, glance is doing *somehting* different | 13:54 |
| dansmith | but I guess I need to go check for glance doing something to its url | 13:55 |
| zzzeek | sure, bug can be in SQLA but I need to see the function inputs | 13:55 |
| dansmith | sure | 13:55 |
| zzzeek | b.c. only one app has the issue | 13:55 |
| dansmith | if it always happened, it'd be one thing.. seems like a weird thing | 13:55 |
| zzzeek | like, if glance has *other* plugin= tokens. maybe it's triyng to concatenate them | 13:56 |
| zzzeek | or it does something w the query string internally | 13:56 |
| dansmith | zzzeek: presumably this is where we pass the config to oslo.do to get the connection right? https://github.com/openstack/glance/blob/bfcab95ff2d7e13afe05fed8835159949490afd8/glance/db/sqlalchemy/api.py#L85-L85 | 13:58 |
| dansmith | so we'll log right before that, but grepping seems to turn up no other even examining of the connection url, fwiw | 13:59 |
| zzzeek | dansmith: the best logging would be here: https://github.com/openstack/oslo.db/blob/3f889d5b6e4def3259e46a5ffe23c1350b4c10bf/oslo_db/sqlalchemy/engines.py#L187 | 14:00 |
| zzzeek | dansmith: oslo.db has some logic that can "extend the URL paramters" that uses some function written by someone named zzzeek to append new paramters | 14:01 |
| dansmith | zzzeek: and is url there stringifyable? | 14:01 |
| zzzeek | URL is alwys stringable sure | 14:01 |
| dansmith | ack | 14:01 |
| zzzeek | the URL object is a named tuple now | 14:01 |
| zzzeek | liek a bug in url.update_query_string() seems more likely , which would still be on me to fix | 14:02 |
| dansmith | ack, well, hopefully it'll be clearer when she's up and can throw some debug in there | 14:07 |
| dansmith | thanks! | 14:07 |
| *** whoami-rajat__ is now known as whoami-rajat | 14:43 | |
| *** jpena is now known as jpena|off | 16:15 | |
| melwitt | dansmith, zzzeek: I unfortunately no longer have the env :( because I believed I couldn't repro the behavior outside of the devstack run. if it happens again I will keep it | 17:59 |
| zzzeek | melwitt: ok! | 17:59 |
| dansmith | ack | 18:05 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!