Thursday, 2020-07-16

« Wednesday, 2020-07-15 Index Friday, 2020-07-17 »
*** jamesmcarthur has joined #storyboard00:05
*** jamesmcarthur has quit IRC00:10
*** jamesmcarthur has joined #storyboard00:12
*** jamesmcarthur has quit IRC00:16
*** jamesmcarthur has joined #storyboard00:18
*** jamesmcarthur has quit IRC00:21
*** jamesmcarthur has joined #storyboard00:21
*** jamesmcarthur has quit IRC02:11
*** jamesmcarthur has joined #storyboard02:11
*** jamesmcarthur has quit IRC02:16
*** jamesmcarthur has joined #storyboard03:29
*** jamesmcarthur has quit IRC03:32
*** jamesmcarthur has joined #storyboard03:32
*** jamesmcarthur has quit IRC04:08
*** jamesmcarthur has joined #storyboard04:08
*** jamesmcarthur has quit IRC04:09
*** jamesmcarthur_ has joined #storyboard04:09
*** jamesmcarthur_ has quit IRC04:11
*** udesale has joined #storyboard04:57
*** udesale has quit IRC07:26
*** udesale has joined #storyboard07:31
*** tosky has joined #storyboard08:02
*** tosky has quit IRC08:16
*** tosky has joined #storyboard08:33
*** ianychoi has joined #storyboard14:05
*** SotK has quit IRC14:55
*** irclogbot_1 has quit IRC14:55
*** SotK has joined #storyboard14:57
*** irclogbot_1 has joined #storyboard14:57
*** tsmith2 has quit IRC15:00
*** tsmith2 has joined #storyboard15:06
*** udesale_ has joined #storyboard15:32
*** udesale has quit IRC15:34
* SotK wonders if folks are around for (and want) a meeting later?16:14
fungii am around16:22
*** udesale_ has quit IRC16:25
*** diablo_rojo has joined #storyboard17:50
fungidiablo_rojo?17:58
fungii guess you rejoined after SotK asked17:58
diablo_rojoLol sorry18:03
diablo_rojoYeah I can meet.18:03
diablo_rojoI dont have anything new.18:03
SotKlets have a quick-ish meeting then, its been a few weeks18:06
SotKand I actually have looked into performance a bit18:06
SotK#startmeeting storyboard18:06
openstackMeeting started Thu Jul 16 18:06:47 2020 UTC and is due to finish in 60 minutes.  The chair is SotK. Information about MeetBot at http://wiki.debian.org/MeetBot.18:06
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.18:06
*** openstack changes topic to " (Meeting topic: storyboard)"18:06
openstackThe meeting name has been set to 'storyboard'18:06
SotK#link https://wiki.openstack.org/wiki/Meetings/StoryBoard Agenda-ish18:07
SotKI don't think there's any announcements or migration updates, so lets go straight to in-progress18:07
SotK#topic In Progress Work18:07
*** openstack changes topic to "In Progress Work (Meeting topic: storyboard)"18:07
fungiyes! eager to hear what performance insights you've gleaned18:08
diablo_rojoAs am I :)18:12
SotKI didn't actually get round to generating a dataset comparable to the production db yet (my test data currently has 10000 stories, compared with the 28138 reported by storyboard.o.o)18:12
fungiit's within the same order of magnitude18:13
SotKindeed18:13
SotKas memory serves the worst query we have (according to the old log) was the general story browsing one18:13
fungiin aggregate, yes18:14
SotKwhich is approximately this: http://paste.openstack.org/show/796021/18:14
fungimultiple joins in a nested query18:15
SotKon my test data that takes ~100ms to execute18:16
diablo_rojoYeah that matches my recollections as well18:16
SotKpart of the problem is the ORDER BY, since story_summary is the massive subquery rather than an actual table, there's no useful index there18:17
fungioh, right18:18
SotKmoving that into the filtered_stories subquery maintains the ordering in the result, and gets the time down to ~80ms18:18
diablo_rojoStep in the right direction.18:18
fungia 20% improvement is nothing to sneeze at, but may also not be a noticeable speedup on human timescales18:19
SotKindeed, so I then moved the filtering that currently lives in filtered_stories into the story_summary subquery, removing the big join18:21
fungithat seems like it should have far greater impact18:21
SotKyeah, that got it down to ~14ms18:21
fungiyowza18:21
fungii'm always dubious of subqueries anyway, and that's a great illustration of why18:22
diablo_rojoHoly moly.18:22
SotKso the hard part is going to be working out how best to get SQLAlchemy to emit that query using the ORM18:25
fungihow/where in the code is it currently built?18:26
SotKthe `story_summary` subquery comes from https://opendev.org/opendev/storyboard/src/branch/master/storyboard/db/models.py#L465-L49018:26
SotKthen the rest is https://opendev.org/opendev/storyboard/src/branch/master/storyboard/db/api/stories.py#L73-L13418:27
SotKthe `_story_build_query` function called in that second link constructs the `filtered_stories` subquery18:28
fungiso basically need to dereference what _story_build_query() is doing i guess18:31
fungimordred: ^  if you're around, can you suggest an orm-friendly alternative to how that's been designed?18:31
SotKI think really we just need to stop using StorySummary entirely18:32
fungioh, right, we talked about that18:32
SotKor at least come up with a way to filter it directly and efficiently rather than joining to a filtered set of IDs18:32
fungibasically just replace it with a custom search, and then we can optimize the hell out of search instead18:33
SotKyeah18:33
SotKthe main thing it provides is ability to aggregate the task statuses in the ORM directly18:34
fungiwhich we already identified need to be done differently too18:34
fungibecause showing "active" stories for a project with no active tasks in those stories is counter-intuitive and probably not what the user wants anyway18:34
SotKaha, I hadn't even considered that aspect18:35
SotKyeah, we'll be able to do that much more easily in a rewritten way18:35
diablo_rojoSeems like a good way to go to me.18:38
diablo_rojoHeh yeah that makes a lot of sense18:38
SotKI'll continue work in that direction then18:39
SotKhopefully I should have a patch up for that before next week's meeting18:39
SotKmy first attempt didn't work even slightly as intended, but did a good job to testing out my laptop's fans18:39
diablo_rojoLol a nice little space heater?18:40
fungior white noise generator18:41
fungiSotK: thanks for working on that!18:41
diablo_rojoYes thank you so much for all your hard work on that.18:42
fungii started looking back over what i might have done wrong with the swift acls in rackspace for storyboard-dev's attachments, but so far no real insights18:42
diablo_rojoPeople will be ecstatic when we get that merged.18:42
SotKyeah it should improve quality of life quite a lot hopefully18:42
fungii think it's plumbed correctly in the sb config because i'm getting a permission failure rather than something else like auth failure or connection error18:42
fungii think my next step is going to be trying to use client tools to write to the container18:43
SotKit'd be nice to test it on storyboard-dev when I do get a patch up, to get an idea of the kind of improvement we get on a still bigger db18:43
SotKfungi: that would be my recommendation, yeah18:43
fungiand see if i can debug by repeating the same things sb is trying to do18:43
fungion query optimization, i wonder how we can identify what the slowdown is for rendering automatic boards/worklists18:45
fungiwhile not as frequently called as the story summary pages, those are the most noticeably slow views18:45
SotKagreed, I think that's what I'll investigate next18:46
diablo_rojoAside from search yeah18:46
fungibut i'm not sure what queries are at fault, or if it's an only moderately slow query getting called an explosive number of times18:46
diablo_rojoI could see that18:46
SotKI suspect StorySummary is somewhat to blame there too18:47
fungioh, could that be getting called to get data for the cards?18:48
fungibut also, more generally, as we optimize the more commonly called queries we free up resources so other queries don't have to compete for cpu/io18:49
diablo_rojoThat would be nice if it was two birds with one stone so to speak18:50
SotKyeah, we use it in the filters because we want to support filtering by story status18:50
SotKthe code there is pretty questionable in general I think, there should be plenty of room for improvement18:51
SotK(there is nothing worse than reading your own old code :D)18:51
diablo_rojoLOL yep.18:52
diablo_rojoMade sense at the time though I imagine18:52
SotKhaha, it always does18:52
fungii always think of it as hatemail my past self sent me18:53
SotK#topic Open Discussion18:56
*** openstack changes topic to "Open Discussion (Meeting topic: storyboard)"18:56
SotKanything else?18:56
funginot on my end18:56
diablo_rojoNothing from me either.18:56
diablo_rojo..aside from apologizing for not being all that helpful lately.18:57
SotKno need to apologise :)18:58
SotKanyway, time for me to make dinner, thank you for the meeting!18:58
fungithanks SotK!18:59
SotK#endmeeting18:59
*** openstack changes topic to "StoryBoard - A task tracker for OpenStack development needs :: http://storyboard.openstack.org/ :: https://wiki.openstack.org/wiki/StoryBoard"18:59
diablo_rojoHave a good rest of your day!18:59
openstackMeeting ended Thu Jul 16 18:59:11 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)18:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/storyboard/2020/storyboard.2020-07-16-18.06.html18:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/storyboard/2020/storyboard.2020-07-16-18.06.txt18:59
openstackLog:            http://eavesdrop.openstack.org/meetings/storyboard/2020/storyboard.2020-07-16-18.06.log.html18:59
diablo_rojoThank you SotK!18:59
*** diablo_rojo has quit IRC19:31
*** tosky has quit IRC23:04
*** SotK has quit IRC23:16
*** irclogbot_1 has quit IRC23:16
*** SotK has joined #storyboard23:22
*** irclogbot_1 has joined #storyboard23:22
« Wednesday, 2020-07-15 Index Friday, 2020-07-17 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!