| *** fatema_ has joined #storyboard | 00:34 | |
| *** aspiers has quit IRC | 00:41 | |
| fatema_ | is there a doc for other tests like this one for pep8 https://docs.openstack.org/hacking/latest/user/hacking.html ^^ | 00:43 |
|---|---|---|
| *** openstack has joined #storyboard | 01:18 | |
| *** ChanServ sets mode: +o openstack | 01:18 | |
| *** fatema_ has quit IRC | 01:33 | |
| *** fatema_ has joined #storyboard | 02:41 | |
| *** fatema_ has quit IRC | 03:05 | |
| *** udesale has joined #storyboard | 03:46 | |
| *** noama has joined #storyboard | 04:41 | |
| *** aspiers has joined #storyboard | 05:22 | |
| *** florianf has joined #storyboard | 06:10 | |
| *** jtomasek has joined #storyboard | 06:43 | |
| *** lifeless has quit IRC | 06:49 | |
| *** jpich has joined #storyboard | 07:55 | |
| *** dtantsur|afk is now known as dtantsur | 09:03 | |
| *** udesale has quit IRC | 11:27 | |
| *** tosky has joined #storyboard | 11:28 | |
| *** noama has quit IRC | 11:37 | |
| *** jamesmcarthur has joined #storyboard | 14:40 | |
| *** udesale has joined #storyboard | 14:45 | |
| *** dtantsur is now known as dtantsur|afl | 15:59 | |
| *** dtantsur|afl is now known as dtantsur|afk | 15:59 | |
| fungi | ianw's observations earlier in #openstack-infra about the random 5xx internal server errors from storyboard.openstack.org are probably also of interest in here: ---http://eavesdrop.openstack.org/irclogs/%23openstack-infra/latest.log.html#t2018-06-25T03:04:00 | 16:12 |
| fungi | suggests that perhaps communication issues with rabbitmq can lead to blocking subsequent operations | 16:12 |
| *** fatema_ has joined #storyboard | 16:20 | |
| *** jpich has quit IRC | 16:29 | |
| openstackgerrit | Fatema Khalid Sherif proposed openstack-infra/storyboard-webclient master: Expose ordering criteria in search UI https://review.openstack.org/577870 | 16:46 |
| *** udesale has quit IRC | 16:56 | |
| *** jamesmcarthur has quit IRC | 17:06 | |
| *** jamesmcarthur has joined #storyboard | 17:35 | |
| *** jamesmcarthur has quit IRC | 17:44 | |
| *** fatema_ has quit IRC | 17:53 | |
| *** jamesmcarthur has joined #storyboard | 18:03 | |
| *** jamesmcarthur has quit IRC | 18:14 | |
| *** jamesmcarthur has joined #storyboard | 18:15 | |
| *** jamesmcarthur has quit IRC | 18:15 | |
| *** jamesmcarthur has joined #storyboard | 18:15 | |
| *** diablo_rojo has joined #storyboard | 18:17 | |
| *** jamesmcarthur has quit IRC | 18:18 | |
| *** jamesmcarthur has joined #storyboard | 18:22 | |
| *** jamesmcarthur has quit IRC | 18:31 | |
| *** jamesmcarthur has joined #storyboard | 18:49 | |
| *** jamesmcarthur has quit IRC | 18:57 | |
| *** jamesmcarthur has joined #storyboard | 18:59 | |
| *** jamesmcarthur has quit IRC | 19:04 | |
| *** jamesmcarthur has joined #storyboard | 19:04 | |
| openstackgerrit | Merged openstack-infra/storyboard master: use required enums to validate 'type' args https://review.openstack.org/545170 | 19:14 |
| openstackgerrit | Merged openstack-infra/storyboard master: mark worklist filter_criteria as a required field https://review.openstack.org/545405 | 19:15 |
| openstackgerrit | Merged openstack-infra/storyboard master: mark FilterCriterion title as a mandatory field https://review.openstack.org/545406 | 19:16 |
| dhellmann | fungi , diablo_rojo : if I wanted to work on that storyboard board API performance issue, could I get a dump of our storyboard database to use as data? | 19:27 |
| dhellmann | I'm running against a sqlite database right now and it has very little data and responds quite quickly :-) | 19:27 |
| fungi | dhellmann: definitely. i think i need to redact api keys from the dump but can work on that in a sec | 19:28 |
| dhellmann | fungi : there's no rush; I'm looking at the code to figure out where to start | 19:29 |
| dhellmann | I'm guessing I will also need to set up mysql, which for some reason I always find to be a non-trivial exercise | 19:29 |
| fungi | you and me both :/ | 19:30 |
| dhellmann | I may try using sqlite. The performance is bound to be worse, right? | 19:33 |
| fungi | dhellmann: current theory is that 1. the queries it uses are more expensive than they need to be (possibly specific to tag queries?) and 2. we're missing indices on one or more columns where we'd benefit from them | 19:34 |
| dhellmann | tag queries do seem to be expensive | 19:34 |
| dhellmann | other boards are much snappier | 19:34 |
| dhellmann | so I will start there | 19:34 |
| fungi | looks like if strip or redact contents of the accesstokens table that takes care of any sensitive credentials. the bigger challenge is filtering out stories where stories.private is true and then any tasks where tasks.story_id matches the corresponding stories.id for those | 19:40 |
| dhellmann | you could just set all of the story and task descriptions to some single string | 19:41 |
| dhellmann | I don't need real data, just bulk | 19:41 |
| dhellmann | I guess you'd need to do that for all comments not just description, but still | 19:42 |
| fungi | oh, right, the comments table filtering any where comments.story_id matches the stories.id too | 19:43 |
| fungi | we could instead dump the content of the db for storyboard-dev, assuming it exhibits the same performance degredation | 19:44 |
| dhellmann | it does | 19:44 |
| dhellmann | and that sounds like an easier option | 19:44 |
| fungi | then i'd probably just omit the accesstokens rows and be done with it | 19:44 |
| dhellmann | wfm | 19:44 |
| * fungi cheers for simple workarounds | 19:45 | |
| fungi | we don't (can't, really) import private lp bugs, so unless someone has created a private story on sb-dev and put actually sensitive information into it (which i doubt) then there should be nothing sensitive | 19:46 |
| dhellmann | someone may have tested it, but I hope not with real info | 19:46 |
| fungi | my select query says sb-dev has 13 private stories, all of whose titles at least look innocuous | 19:48 |
| fungi | i wonder if we need to periodically expire content from the accesstokens table. i see _lots_ of tokens from when the server was first deployed | 19:49 |
| fungi | 4312 rows in production | 19:49 |
| fungi | there's an expires_at column, so should be straightforward | 19:50 |
| fungi | mysql> delete from accesstokens where expires_at < now(); | 19:53 |
| fungi | Query OK, 429 rows affected (0.07 sec) | 19:53 |
| fungi | that leaves 5 accesstokens in the sb-dev db to redact, which is trivial to handle manually | 19:53 |
| dhellmann | selecting on story status looks pretty expensive, too | 20:01 |
| dhellmann | it builds a case statement in sql | 20:01 |
| fungi | dhellmann: wget --no-check-certificate https://storyboard-dev.openstack.org/storyboard-dev_redacted_2018-06-25.sql.gz | 20:01 |
| fungi | lmk when you have it and i'll clean it out of the webroot | 20:02 |
| dhellmann | fungi : got it, thanks | 20:03 |
| fungi | cleaned up | 20:03 |
| *** florianf has quit IRC | 20:03 | |
| dhellmann | well, sqlite doesn't like that format so I guess I need to try mysql | 20:07 |
| fungi | yeah, you should just be able to gunzip it and then `source storyboard-dev_redacted_2018-06-25.sql;` in an interactive mysqlclient command-line session | 20:12 |
| dhellmann | well, I was able to edit the file to remove all of the table commands and leave the inserts and sqlite is still really fast | 20:17 |
| dhellmann | so yeah, I guess I need to set up mysql | 20:18 |
| fungi | also, this is in a 4gb ubuntu trusty (14.04 lts) vm with a remote mysql database in a trove instance | 20:22 |
| dhellmann | oh, yeah | 20:22 |
| dhellmann | I have a 12G box with local ssd | 20:22 |
| fungi | so yeah performance differences may not be as exaggerated in your local deployment | 20:22 |
| fungi | i don't see any signs there's been memory pressure on the vm though | 20:23 |
| dhellmann | some of these filtering queries look pretty complicated | 20:32 |
| dhellmann | like I said, asking for story status turns into a big case statement on the sum of task status calculated a few times | 20:32 |
| diablo_rojo | oof | 20:32 |
| dhellmann | it seems like that would be more efficient if we store it separately, even if that means updating it requires updating 2 rows | 20:33 |
| dhellmann | but we'll see | 20:33 |
| dhellmann | my local mysql should be slower since it's not on ssd | 20:36 |
| * dhellmann misses the sound of the disk head moving back and forth | 20:36 | |
| *** jamesmcarthur has quit IRC | 20:38 | |
| diablo_rojo | Ha ha a comforting sound? | 20:40 |
| dhellmann | it made me feel like I was *doing* something :-) | 20:43 |
| dhellmann | ok, data imported and I can reproduce the performance issue | 20:44 |
| dhellmann | with the query running mysqld has 100% of the CPU | 20:46 |
| diablo_rojo | Dang. | 20:48 |
| fungi | nice | 20:48 |
| fungi | i wonder if resizing our trove instance to get more cpu (or moving the db to a local mysqld on the server) would help | 20:49 |
| fungi | granted, this is almost certainly a case where either the database schema or the queries (or both) could be made orders of magnitude more efficient | 20:50 |
| fungi | mordred: ^ you may also be interested in some of whatever findings dhellmann arrives at | 20:51 |
| mordred | yes- I am interested - and I could imagine us missing an index or two | 20:53 |
| dhellmann | there's definitely no index on the story_storytag table | 20:54 |
| dhellmann | I'm still waiting for the query to return so I can compare the times before and after | 20:54 |
| dhellmann | 7m19.189s | 21:02 |
| *** EmilienM has joined #storyboard | 21:03 | |
| *** EmilienM_PTO has quit IRC | 21:04 | |
| *** EmilienM has quit IRC | 21:04 | |
| *** EmilienM has joined #storyboard | 21:04 | |
| fungi | that's a goodly quantity of time | 21:09 |
| fungi | SotK: looking into the get project by name decoding, i see that _route() isn't using the escaped something in the by name fallback, it's pulling from the unquoted args list instead. could that be the problem? | 21:11 |
| fungi | er, from the un-unquoted args list | 21:12 |
| fungi | and isn't using the something resulting from unquote(args[0]) | 21:14 |
| dhellmann | ok, adding the index drops that to 2.7 seconds | 21:18 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: add an index for the story_storytags table https://review.openstack.org/577931 | 21:21 |
| dhellmann | explain shows it looking at 5 rows instead of 13249 | 21:24 |
| dhellmann | so that seems like an improvement | 21:24 |
| dhellmann | showing the results in the UI is still a bit sluggist | 21:28 |
| dhellmann | sluggish* | 21:28 |
| dhellmann | I wonder if it would be better to do it in 2 steps, load the basic data including the column titles, the show a whirlygig while the columns are populated | 21:28 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Add to & Reorganize StoryBoard GUI Docs https://review.openstack.org/577934 | 21:36 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Add to & Reorganize StoryBoard GUI Docs https://review.openstack.org/577934 | 21:38 |
| SotK | fungi: that's probably the problem, I wonder why on earth it works in the test suite and in my local instance | 21:39 |
| fungi | dhellmann: wow, indices _do_ make a difference! ;) | 21:50 |
| dhellmann | yeah, that was surprising | 21:56 |
| dhellmann | I expected to need to do some work on the python code, too, but that doesn't look so bad | 21:57 |
| fungi | well, we'll presumably need an alembic migration | 21:57 |
| dhellmann | a few loops where we could use generator expressions, but otherwise pretty straightforward | 21:57 |
| dhellmann | yeah, see https://review.openstack.org/577931 | 21:57 |
| fungi | oh, i missed that earlier | 21:57 |
| * dhellmann drops offline for the evening | 21:59 | |
| diablo_rojo | Thanks dhellmann! Have a good evening | 22:04 |
| fungi | approved, i'll keep an eye on things and make sure it shows up once merged | 22:20 |
| openstackgerrit | Merged openstack-infra/storyboard master: add an index for the story_storytags table https://review.openstack.org/577931 | 22:54 |
| fungi | and there it is | 22:54 |
| fungi | should hopefully wind up in production over the next 15 to 30 minutes | 22:55 |
| fungi | Jun 25 23:17:59 storyboard puppet-user[6632]: (/Stage[main]/Storyboard::Application/Exec[migrate-storyboard-db]) Triggered 'refresh' from 1 events | 23:18 |
| fungi | should be in production as of a minute ago | 23:19 |
| fungi | https://storyboard.openstack.org/#!/board/53 loaded for me in a couple seconds | 23:20 |
| *** jamesmcarthur has joined #storyboard | 23:33 | |
| *** tosky has quit IRC | 23:34 | |
| fungi | dhellmann: all looking good so far--thanks again!!! | 23:40 |
| *** jamesmcarthur has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!