| *** jpena|off is now known as jpena | 07:07 | |
| opendevreview | Slawek Kaplonski proposed openstack/governance master: Define Technical preview framework https://review.opendev.org/c/openstack/governance/+/839880 | 08:56 |
|---|---|---|
| opendevreview | Slawek Kaplonski proposed openstack/governance master: Remove release names starting from 2023.1 release https://review.opendev.org/c/openstack/governance/+/839897 | 10:16 |
| *** iurygregory__ is now known as iurygregory | 11:14 | |
| *** jpena is now known as jpena|off | 16:30 | |
| dansmith | zzzeek: is there anything different about how neutron's db stuff works (that you know of) that would affect this plugin based approach? | 16:49 |
| dansmith | it seems to work great for every service other than neutron, which both seemed to choke with weird timeouts and hangs on my first attempt and later attempts seem to not always be registering on new engines (or something maybe) which causes me to not collect data past a certain point | 16:49 |
| opendevreview | Merged openstack/governance master: Remove TC Liaisons framework https://review.opendev.org/c/openstack/governance/+/837891 | 17:13 |
| opendevreview | Merged openstack/governance master: Fix keystone PTL model documentation https://review.opendev.org/c/openstack/governance/+/838940 | 17:18 |
| opendevreview | Merged openstack/governance master: Retire openstack-helm-docs https://review.opendev.org/c/openstack/governance/+/839100 | 17:22 |
| opendevreview | Merged openstack/governance master: Resolution to drop the lower constraints maintenance https://review.opendev.org/c/openstack/governance/+/838004 | 17:22 |
| *** spotz_ is now known as spotz | 19:53 | |
| zzzeek | dansmith: no, nothing neutron does would affect this particular approach | 20:29 |
| dansmith | zzzeek: I think I'm on the right track, | 20:30 |
| dansmith | it seems like they have some different threading behavior than some of the others | 20:30 |
| zzzeek | dansmith: ok. still should not affect the plugin being registered or the event handler | 20:30 |
| dansmith | zzzeek: no, neither of those, | 20:30 |
| zzzeek | if your acutal event handler is doing something not very threadsafe, that could be a problem | 20:30 |
| dansmith | yeah, so I think they maybe create their engine in the main thread and then fork, but others do the opposite maybe? | 20:31 |
| dansmith | either way, I think I'm good now | 20:31 |
| dansmith | first attempt was to, for each before_cursor_execute, I would connect to a different mysql db to increment a counter (knowing the naivety) | 20:32 |
| dansmith | that was pretty slow, so I made the counter db ENGINE=memory | 20:32 |
| dansmith | which was faster but still slow | 20:32 |
| dansmith | now I'm batching them with a little worker thread to do fewer writes less often which seems to work pretty good | 20:33 |
| dansmith | trying to avoid having to build and run a separate collection daemon, or go full collectd, etc if I can help it without much fuss | 20:33 |
| zzzeek | engine=memory...you mean sqlite ? | 20:33 |
| zzzeek | or the mysql memory engine, I dont know if i know much about that | 20:34 |
| dansmith | no, the mysql memory engine | 20:34 |
| dansmith | we don't have sqlite on the workers | 20:34 |
| zzzeek | well sqlite is bundeld into python | 20:34 |
| zzzeek | but yes | 20:34 |
| dansmith | oh? well, we don't have the tool at least :) | 20:34 |
| zzzeek | mysql memory engine...is that server side? i dont know what that does | 20:34 |
| dansmith | yep, server side, and lets me increment the counters from the subnodes, which won't work for sqlite | 20:35 |
| dansmith | (easily) | 20:35 |
| zzzeek | are they tables that persist to disk or they remain just in memory ? | 20:35 |
| dansmith | https://dev.mysql.com/doc/refman/8.0/en/memory-storage-engine.html | 20:35 |
| zzzeek | i see | 20:35 |
| zzzeek | got it | 20:35 |
| dansmith | just in memory, they go away at restart, which is perfect for this | 20:35 |
| dansmith | 15 integers plus db names.. so really tiny | 20:35 |
| zzzeek | so....you are just logging into the mysql console to query these tables while things are running? | 20:36 |
| dansmith | no, I have a tool that grabs the contents of the db after, plus a bunch of other things | 20:36 |
| dansmith | https://zuul.opendev.org/t/openstack/build/d6f6ec9364a149899f7bb8d32fe4c6bd/log/controller/logs/devstacklog.txt#19544 | 20:37 |
| dansmith | (note keystone's huge number, btw) | 20:37 |
| zzzeek | oh at the bottom | 20:39 |
| zzzeek | is keystone using memcached? it's not supposed to have all those SELECTs | 20:39 |
| dansmith | idk, but even with caching that number seems crazy high to me :) | 20:39 |
| dansmith | that's just at the end of devstack, before tempest | 20:39 |
| dansmith | https://zuul.opendev.org/t/openstack/build/d6f6ec9364a149899f7bb8d32fe4c6bd/log/controller/logs/etc/keystone/keystone_conf.txt#16 | 20:39 |
| dansmith | looks enabled to me | 20:40 |
| zzzeek | also, neutron has a lot of SELECT,, but in my experience they are much more complex SELECTs that are a lot more expensive | 20:40 |
| dansmith | yes, a tempest run locally ended up with well over 300k selects by neutron | 20:40 |
| zzzeek | but yeah keystone is bad thre should be a way to fix that | 20:40 |
| dansmith | where other services were way under that (except for keystone, which was about 150k) | 20:40 |
| dansmith | agreed | 20:41 |
| zzzeek | like i think keystone is likyle a lot more fixable than neutron b.c. keystone is doing the same simple lookups over and over, that should be a caching issue | 20:42 |
| dansmith | other runs, depending on what services are enabled end up with 80k keystone selects at the end of a devstack (pre-tests) run, | 20:43 |
| dansmith | so it seems like there's some n^2 behavior going on somewhere | 20:43 |
| dansmith | so yeah I assume it should be fixable | 20:43 |
| dansmith | with so few users in a devstack I can't imagine how they're getting to 80k | 20:43 |
| dansmith | https://zuul.opendev.org/t/openstack/build/9f869c884d064daea13c7782e882b5a2/log/controller/logs/devstacklog.txt#22584 | 20:44 |
| dansmith | 80k ^ | 20:44 |
| dansmith | (that one has a faulty neutron count, so ignore that line) | 20:44 |
| zzzeek | ha wow | 20:47 |
| opendevreview | Ghanshyam proposed openstack/governance master: Remove release names starting from 2023.1 release https://review.opendev.org/c/openstack/governance/+/839897 | 23:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!