| *** jpena|off is now known as jpena | 06:58 | |
| opendevreview | Merged openstack/project-team-guide master: Fix the dependency hierarchy in repo retirement process https://review.opendev.org/c/openstack/project-team-guide/+/837787 | 12:55 |
|---|---|---|
| *** jpena is now known as jpena|off | 14:14 | |
| dansmith | zzzeek: around for a db question? | 16:04 |
| opendevreview | Merged openstack/governance master: Update Dmitriy Rabotyagov email https://review.opendev.org/c/openstack/governance/+/839470 | 16:10 |
| opendevreview | Merged openstack/governance master: Remove oslo's TaCT SIG liaisons https://review.opendev.org/c/openstack/governance/+/839317 | 16:10 |
| zzzeek | dansmith: hey | 16:28 |
| dansmith | hey, so.. I'm trying to gather some mysql stats.. specifically what I want is "number of queries" per database (ideally separated between select and insert, but not critical) | 16:28 |
| dansmith | seems like performance_schema statement history is the way to do that, | 16:29 |
| dansmith | but it ends up being really heavy for the number of queries we make during a full CI run | 16:29 |
| dansmith | I don't need to capture all the queries, just the counts ideally, but it like it's either that or nothing | 16:29 |
| dansmith | global "all queries across all databases" is not useful | 16:29 |
| dansmith | zzzeek: so, do you know of any other way? | 16:29 |
| zzzeek | i havent worked with performance_schema, whats "heavy" about it, it slows performance or just too much config / disk space / etc ? | 16:30 |
| dansmith | it inflates our memory footprint by a lot. when I picked an arbitrary upper limit of keeping 1m rows, it OOMs | 16:30 |
| zzzeek | other ways are, log MySQL queries and do a lot of grepping, or on the app side put an event handler to log queries in the way you want | 16:31 |
| dansmith | 300k isn't enough, so .. | 16:31 |
| dansmith | I want to do it out-of-body so not requiring changes to projects | 16:31 |
| dansmith | ack on the query log, not terrible I guess, but it's all just so heavy for me to just get a damn integer count :P | 16:32 |
| zzzeek | well the collectd plugin I hvae doesnt do queries right now, just connections, in theory it could do that | 16:32 |
| dansmith | how? | 16:32 |
| zzzeek | oh id have to write new code for it, that deos the above event handlers | 16:32 |
| dansmith | oh this is not a plugin for collectd, but a plugin for the clients that reports to collectd? | 16:33 |
| zzzeek | OK dependeing on how motivated you are, there is a way to get event handlers injected into running apps via configuration alone, you still have to ship the code that's doing the event handling through | 16:33 |
| zzzeek | the collectd thing is a plugin I wrote that is a plugin on both sides, collectd and the app side | 16:33 |
| zzzeek | but it doesnt log queries right now | 16:33 |
| dansmith | okay | 16:33 |
| dansmith | I don't want queries, just query counts, but yeah sounds like your plugin might be a reasonable route | 16:34 |
| dansmith | re: the injection of the event handler, is that mysql config or oslo.db config? | 16:34 |
| zzzeek | the plugin thing goes into the database URL in the query string | 16:34 |
| zzzeek | that's how the collectd plugin sets up also | 16:34 |
| dansmith | ah, neat | 16:35 |
| zzzeek | is this like a customer featre thing we want to roll out at some point or .... | 16:35 |
| zzzeek | b.c. if you need something today, or like, this month, i would just point you how to write your own plugin | 16:35 |
| dansmith | no, this is for upstream CI monitoring for when we add a feature that causes us to do a billion new db requests | 16:35 |
| dansmith | zzzeek: if there's something I can look at as a guide or example, I might go for that | 16:36 |
| zzzeek | well the collectd plugin has tripleo integration, though I think nobody has ever used it | 16:36 |
| zzzeek | and the project itself is at https://github.com/sqlalchemy/sqlalchemy-collectd | 16:36 |
| zzzeek | now, if you wanted to write your own quicky SQLAlchemy plugin that can come in via config, the interface you implement is this one: https://docs.sqlalchemy.org/en/14/core/connections.html?highlight=createengineplugin#sqlalchemy.engine.CreateEnginePlugin | 16:37 |
| dansmith | another really hacky thing I thought about was to set the limit on the statement log table in performance_schema to be something low, and then try to count-and-purge in a transaction every few seconds while a CI job is running | 16:37 |
| dansmith | presumably it's just a database so I can do that | 16:37 |
| zzzeek | in fact the example shown there is almost exactly waht you want already | 16:37 |
| dansmith | hmm, yeah, that's interesting | 16:38 |
| zzzeek | so you can add mysql://foo:Bar@host?plugin=dans_logging_plugin | 16:38 |
| dansmith | yeah, nifty | 16:38 |
| zzzeek | anyway the colletd plugin above uses the same API to get its listeners in, and it transmits them to the collectd daemon which has a plugin on that end also | 16:39 |
| zzzeek | i think it's using udp i forget | 16:39 |
| dansmith | ack | 16:40 |
| dansmith | okay well, this seems like a reasonable thing to chase for a bit | 16:40 |
| zzzeek | so anyhway, yes | 16:40 |
| zzzeek | i had hoped we wanted collectd stuff a long time ago back when joe T was doing a lot of it | 16:40 |
| dansmith | thanks a lot.. not really what I was expecting, but definitely better than grepping logs just to count I think | 16:40 |
| zzzeek | but then he left and nobody wanted it :) | 16:41 |
| dansmith | the classic story | 16:41 |
| zzzeek | but yeah, if we want to revive the collectd plugin, beats me having to learn kubernetes | 16:41 |
| zzzeek | :) | 16:41 |
| dansmith | the benefit of collectd I guess is that it handles the persistence for us, | 16:42 |
| dansmith | although locally I have tons of problems with collectd going off the rails and leaking memory until it OOMs, but that's probably one of the billion plugins I have loaded | 16:42 |
| zzzeek | well also it is made for counting things | 16:42 |
| zzzeek | likely a plugin. i think the core is pretty solid | 16:43 |
| dansmith | yeah | 16:43 |
| zzzeek | but certainly, my own plugin is a very plausible cause of OOM :) since it's hardly been used | 16:43 |
| zzzeek | like, it's not suppose to | 16:43 |
| dansmith | I mean locally as in my own infra monitoring disks and nics and such | 16:44 |
| dansmith | zzzeek: hot damn, that's cool | 17:04 |
| zzzeek | yup I rule | 17:04 |
| zzzeek | :) | 17:04 |
| dansmith | I am unable to disagree at the present time :) | 17:05 |
| zzzeek | ha | 17:05 |
| zzzeek | which, the collectd thing? | 17:05 |
| zzzeek | or just the quickie plugin recipe | 17:05 |
| dansmith | you claim ruling status without certainty? :) .. but I mean the plugin-in-url thing in general, bonus points for the quick example being most of the hard work :) | 17:07 |
| zzzeek | yeah. i made that plugin API so i could do the collectd plugin for openstack | 17:10 |
| dansmith | ah, well, it's pretty nifty | 17:11 |
| zzzeek | OK well i have to take my car into the dealer for repairs but if you work on something there keep me in the loop | 17:12 |
| opendevreview | Gage Hugo proposed openstack/governance master: Retire openstack-helm-docs https://review.opendev.org/c/openstack/governance/+/839100 | 17:16 |
| dansmith | zzzeek: ack, thanks | 17:23 |
| opendevreview | Ghanshyam proposed openstack/project-team-guide master: Explicitly mention the each steps of repo retirement https://review.opendev.org/c/openstack/project-team-guide/+/839807 | 18:02 |
| dansmith | zzzeek: back around by chance? | 21:04 |
| dansmith | maybe nevermind | 21:30 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!