| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 00:19 |
|---|---|---|
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 01:36 |
| *** adam_g has quit IRC | 04:44 | |
| *** adam_g has joined #zuul | 04:45 | |
| tobiash_ | jeblair: no worries, I didn't have in mind of swapping the asserts. For consistency I'll just stick like the rest of the code. | 04:59 |
| *** dmellado has quit IRC | 05:14 | |
| *** pabelanger has quit IRC | 05:14 | |
| *** dmellado has joined #zuul | 05:16 | |
| *** pabelanger has joined #zuul | 05:20 | |
| *** jasondotstar has quit IRC | 05:31 | |
| *** jasondotstar has joined #zuul | 05:35 | |
| *** isaacb has joined #zuul | 06:25 | |
| *** eventingmonkey has quit IRC | 06:30 | |
| *** eventingmonkey has joined #zuul | 06:32 | |
| tobiash_ | jeblair: what are the longtem plans with gearman/zookeeper? Is it planned to get rid of gearman completely? | 07:06 |
| tobiash_ | This maybe could ease use cases like automatic/transparent restart of jobs when an executor shuts down | 07:07 |
| tobiash_ | Maybe useful for automatic zuul-executor scaling later on | 07:08 |
| tobiash_ | or persistent job queues for maintenance breaks | 07:09 |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** hashar has joined #zuul | 08:04 | |
| lennyb | hi, is it OK that zuul quires changes from the gerrit from the projects that are not listed in layout.yaml? | 09:04 |
| *** openstackgerrit has joined #zuul | 10:30 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool feature/zuulv3: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/445325 | 10:30 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool feature/zuulv3: Fix test_leaked_node_not_deleted for v3 https://review.openstack.org/449375 | 10:46 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool feature/zuulv3: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/445325 | 10:46 |
| *** hashar is now known as hasharAway | 11:00 | |
| *** hasharAway is now known as hashar | 12:00 | |
| Shrews | jhesketh: sorry that merge has been a PITA. thx for doing that | 12:44 |
| Shrews | i wonder if we should just delete the nodepool_id code rather than fixing the test_leaked_node_not_deleted test | 12:44 |
| Shrews | tobiash_: spec mentions that gearman will remain for v3. https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#ansible | 12:48 |
| Shrews | at least on the zuul side of things. nodepool doesn't need it now | 12:48 |
| jhesketh | Shrews: all good.. if we can get it merged before more patches land in master that'd be handy :-) | 13:02 |
| jhesketh | (or v3 for that matter) | 13:02 |
| jhesketh | Shrews: I also have the follow up to those tests: https://review.openstack.org/#/c/449375/7 | 13:02 |
| Shrews | jhesketh: yup. already +2'd the first. the 2nd is the one i'm wondering whether we should skip and just delete the nodepool_id code | 13:03 |
| Shrews | i might toss that up today while you sleep | 13:04 |
| jhesketh | Shrews: right.. that probably needs some discussion. mordred had thought it may still be of use | 13:04 |
| Shrews | k | 13:05 |
| *** isaacb_ has joined #zuul | 13:55 | |
| *** isaacb has quit IRC | 13:58 | |
| jeblair | jhesketh, Shrews: i think the results of the most recent discussion is that we should not port nodepool_id to v3 | 14:12 |
| pabelanger | morning! | 14:14 |
| jhesketh | jeblair, Shrews: sure.. I can do that in a follow up commit, I think it makes more sense than in the merge (and simplifies it) | 14:26 |
| tobiash_ | Shrews: thx | 14:28 |
| jeblair | tobiash_, Shrews: we've talked about replacing more of gearman with zk in zuul, but much later, after v3 is finished. i call it zuul v4. :) there's a note about it here: http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-zookeeper-workers.html#future-work | 14:29 |
| tobiash_ | jeblair: ah, just wanted to ask if that was a decision or posponed to v4 ;) | 14:31 |
| jeblair | tobiash_: i don't think we've quite decided to do it yet, but it's a possibility for us to consider later. i think it's promising. | 14:32 |
| tobiash_ | jeblair: fully distributed zuul, pretty challenging, but a killer feature :) | 14:36 |
| *** isaacb_ has quit IRC | 14:41 | |
| *** tobiash_ is now known as tobiash | 14:42 | |
| mordred | ++ | 14:44 |
| *** isaacb_ has joined #zuul | 14:47 | |
| *** hashar has quit IRC | 15:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Add ability to select flavor by name or id https://review.openstack.org/449784 | 15:04 |
| mordred | jeblair: ok - I think that ^^ has it now | 15:04 |
| mordred | jeblair: maybe we want to add in a voluptuous thing to encode "you have to at least have one of flavor-name or min-ram" | 15:05 |
| *** isaacb__ has joined #zuul | 15:11 | |
| *** isaacb_ has quit IRC | 15:15 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Fetch list of AZs from nova if it's not configured https://review.openstack.org/450345 | 15:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Cleanup from config syntax change https://review.openstack.org/451868 | 15:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Validate flavor specification in config https://review.openstack.org/451875 | 15:43 |
| jeblair | mordred: lgtm, and i think that ^ should do that | 15:43 |
| *** isaacb__ has quit IRC | 15:56 | |
| Shrews | those lgtm, too, and i'm ready to +3 them, but i'd rather see jhesketh's merge get done first | 15:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add canonical hostname to source object https://review.openstack.org/451110 | 15:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add source to projects https://review.openstack.org/451596 | 15:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add a project index to Tenant https://review.openstack.org/451597 | 15:57 |
| jeblair | Shrews: good point, i'll review that now | 15:57 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/445325 | 16:03 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove support for nodepool_id https://review.openstack.org/451889 | 16:09 |
| SpamapS | jeblair: idea for cool feature for gertty... a key binding that does "diff against patchset of the last review I performed" | 17:03 |
| jeblair | SpamapS: ++ | 17:11 |
| SpamapS | btw, presentty and cowsay were saved, and will be in the next Debian stable release. :) | 17:12 |
| jeblair | SpamapS: whew! | 17:13 |
| jeblair | SpamapS: http://paste.openstack.org/show/604899/ | 17:15 |
| SpamapS | haha | 17:18 |
| SpamapS | It's all thanks to the cowsay maintainers. ;) | 17:18 |
| clarkb | wait does presentty dep on cowsay in debain so was going to be removed if cowsay was removed? | 17:20 |
| clarkb | also is cowsay being removed because everyone using ansible hates it now? | 17:20 |
| clarkb | I guess not actually remvoed but was that the argument? | 17:21 |
| SpamapS | clarkb: yes, no, perhaps it should have been. ;) | 17:22 |
| SpamapS | clarkb: cowsay was going to be removed because some of the cow ascii art had no license. | 17:22 |
| SpamapS | clarkb: but they removed the ones that had unclear licensing. | 17:23 |
| clarkb | ah that makes sense | 17:23 |
| clarkb | also cowsay is perfectly useable if you remove all the art from the project itself | 17:23 |
| clarkb | (you can provide your own) | 17:23 |
| SpamapS | Well most of the art did have licensing. | 17:24 |
| SpamapS | Just a few didn't. | 17:24 |
| SpamapS | And a few were poorly documented, so I believe it has been resolved now, and meow, bong, and beavis, have been removed. | 17:24 |
| SpamapS | oh no, hm.. those were moved to cowsay-off which are offensive cows | 17:25 |
| SpamapS | or something. who cares. :) | 17:26 |
| jeblair | i would attend a cowsay-off contest where the best cowsayers compete. like at a cownty fair. | 17:31 |
| SpamapS | would we invite all of the moovers and shaykers? | 17:35 |
| jeblair | definitely all the Prime talent | 17:36 |
| SpamapS | I suspect you're just ribbing me. One cud take that personally. | 17:42 |
| Shrews | cows go moo | 17:45 |
| Shrews | am i playing right? | 17:45 |
| SpamapS | Shrews: you're milking it a little.... | 17:46 |
| jlk | This channel is just bovine, but can we steer things back into a more serious direction? | 17:46 |
| SpamapS | jlk: good idea. I think we should make zuul executors more like cattle.. less like pets.. | 17:48 |
| jeblair | oh good! a chance to trot out my cattle/pets thing! cattle sell for something like $1500 a head, so when they get sick, you better believe the rancher has a vet take a look at them. | 17:52 |
| SpamapS | jeblair: you can get a good look at a T-bone by SSH'ing into the bull, but I'd rather take the sysadmin's word for it. | 17:53 |
| *** unrahul_ has joined #zuul | 17:55 | |
| *** TheJulia_ has joined #zuul | 17:56 | |
| *** colettecello has joined #zuul | 17:57 | |
| *** TheJulia has quit IRC | 17:59 | |
| *** phschwartz has quit IRC | 17:59 | |
| *** clarkb has quit IRC | 17:59 | |
| *** gothicmindfood has quit IRC | 17:59 | |
| *** jlk has quit IRC | 17:59 | |
| *** unrahul has quit IRC | 17:59 | |
| *** mordred has quit IRC | 17:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Pass source to project instantiations https://review.openstack.org/451596 | 17:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add a project index to Tenant https://review.openstack.org/451597 | 17:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use new tenant project index for zuul role refs https://review.openstack.org/451928 | 17:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove unused Tenant.getRepo method https://review.openstack.org/451929 | 17:59 |
| *** jlk has joined #zuul | 18:00 | |
| *** jlk has quit IRC | 18:00 | |
| *** jlk has joined #zuul | 18:00 | |
| *** phschwartz has joined #zuul | 18:00 | |
| *** clarkb has joined #zuul | 18:01 | |
| *** mordred has joined #zuul | 18:02 | |
| *** unrahul_ is now known as unrahul | 18:03 | |
| *** colettecello has quit IRC | 18:10 | |
| *** gothicmindfood has joined #zuul | 18:10 | |
| Shrews | jeblair: did you ever write up your zuul test environment tips somewhere? | 18:46 |
| *** TheJulia_ is now known as TheJulia | 18:56 | |
| jlk | getting so tired of reading tox logs | 18:57 |
| Shrews | jlk: use ttrun and read stdout instead | 19:00 |
| jlk | well that's basically what I'm doing | 19:00 |
| jlk | I'm reading stdout, of the tox run | 19:00 |
| jeblair | Shrews: er, no. :( | 19:11 |
| Shrews | i found ZUUL_TEST_ROOT. good enough for now, and was the thing i noted to remember a while back | 19:13 |
| Shrews | but didn't note it anywhere :/ | 19:14 |
| SpamapS | jlk: I use ttrun and pipe through ccze -A ... | 19:44 |
| SpamapS | sounds silly, but breaking up the grey monotony with a few red/yellow/green words helps | 19:44 |
| jeblair | (might ccze be a good ttrun option?) | 19:45 |
| SpamapS | perhaps. It's meant as a generic log highlither. So it makes words like "error" red, and "OK" green. | 19:46 |
| SpamapS | highlither's are very flexible tho | 19:46 |
| *** yolanda has quit IRC | 19:47 | |
| fungi | okay, i've finally managed to work my way through the encrypted secrets stack (twice so i knew what i was looking at the second time) and it all lgtm | 19:52 |
| fungi | also reread the ml thread so i could be sure it matched what was discussed | 19:53 |
| * fungi will need an extra beer tonight | 19:53 | |
| SpamapS | I just found my way through it too | 19:55 |
| SpamapS | Seems solid | 19:55 |
| jeblair | fungi: thank you! (also, are you planning on reviewing it a 3rd time with beer goggles?) | 19:55 |
| jeblair | SpamapS: you too :) | 19:55 |
| * SpamapS stacks yaml thing on top of it | 19:55 | |
| SpamapS | jesusaur: did you still want a second set of eyes on test fails in your merge conflict patches? | 19:57 |
| fungi | jeblair: not a terrible idea ;) | 19:57 |
| jesusaur | SpamapS: yeah, specifically around the failures in post queues, such as in test_client_enqueue_ref | 19:58 |
| jesusaur | SpamapS: I'm currently looking into why the check queue isn't reporting start (and possibly not running jobs) in test_delayed_merge_conflict | 19:58 |
| SpamapS | oooo TIL I learned a new vim trick | 19:59 |
| SpamapS | :e **/**<partialfilename>[tab] | 19:59 |
| SpamapS | it uses fuzzy matches too | 19:59 |
| SpamapS | finds the file somewhere under cwd | 19:59 |
| SpamapS | jesusaur: Ok I'll attack the post queues fails. | 20:00 |
| jeblair | jesusaur: oh i hadn't noticed the new patchset, sorry. i'll get in line after SpamapS. :) | 20:01 |
| jlk | py27: commands succeeded | 20:02 |
| jlk | congratulations :) | 20:02 |
| jlk | ^^^ that has begun to make me unreasonably happy. | 20:02 |
| jeblair | haven't seen that in a while, myself | 20:04 |
| SpamapS | hah | 20:04 |
| pabelanger | so much scroll back | 20:04 |
| SpamapS | well this yaml change seems to get it to you about 1 minute faster.. so there's that. | 20:05 |
| jesusaur | SpamapS: awesome, it takes me like 20 minutes to run the full set of tests locally | 20:08 |
| * SpamapS just confirmed Ansible does prefer libyaml/cparser/etc | 20:08 | |
| SpamapS | jesusaur: oh, we can help with that. | 20:08 |
| jesusaur | I guess my laptop isn't quite powerful enough to run tox and firefox at the same time | 20:08 |
| SpamapS | assuming you're on Ubuntu/Debian.. | 20:09 |
| SpamapS | none /var/lib/zookeeper tmpfs defaults 0 0 | 20:09 |
| SpamapS | adding that to your fstab helps a bit | 20:09 |
| jesusaur | oooh | 20:09 |
| * jesusaur does the thing | 20:09 | |
| SpamapS | and then pointing ZUUL_TEST_ROOT to a tmpfs will get you a lot too | 20:09 |
| clarkb | just don't run a real zookeeper there | 20:10 |
| SpamapS | clarkb: if you're doing it right, it's fine. ;) | 20:10 |
| SpamapS | assuming you don't reboot all three+ at once. | 20:10 |
| SpamapS | but yeah, best to persist to disk for real zk :) | 20:10 |
| SpamapS | hrm | 20:13 |
| SpamapS | I still get some Alarmclock fails | 20:13 |
| SpamapS | setting OS_TEST_TIMEOUT=120 fixes it | 20:15 |
| SpamapS | couple tests are just slow for some reason | 20:15 |
| SpamapS | jesusaur: ever thought of porting Zuul to use this: https://github.com/libgit2/pygit2 ? | 20:16 |
| SpamapS | The bulk of the CPU load during these tests is system load.. and I think it's from all the forking of git | 20:17 |
| clarkb | whats the git lib that Zara was looking at recently? that one looked pretty good | 20:18 |
| clarkb | dulwich | 20:18 |
| SpamapS | This one looks pretty solid. | 20:18 |
| clarkb | iirc it had weird licensing due to the C lib? | 20:18 |
| SpamapS | seems like an area where an optimized C implementation would win. | 20:18 |
| SpamapS | ew yeah | 20:19 |
| SpamapS | it's GPLv2 with linking exception to open source projects | 20:19 |
| SpamapS | so, like MySQL | 20:19 |
| SpamapS | Prolly not Apache 2.0 compatible | 20:19 |
| jeblair | we need to be able to create merges, primarily the way cgit does it (which is *very good*), but also being able to approximate what gerrit does (which cgit also does). that's the primary driver that got us to gitpython. i have not re-evaluated the others for that recently. | 20:19 |
| clarkb | dulwich is more like jgit in design I think, so possibly not great in that respect | 20:20 |
| clarkb | it give you proper python primitives for doing things | 20:20 |
| SpamapS | it's complicated enough that I think one needs more reasons to switch than "the test suite is slow" | 20:21 |
| SpamapS | Perhaps when somebody runs it at scale we can re-evaluate ;) | 20:21 |
| jeblair | implementing our own octomerge algorithm is not high on my list of things to do (and i like writing code!) | 20:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Pass source to project instantiations https://review.openstack.org/451596 | 20:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add a project index to Tenant https://review.openstack.org/451597 | 20:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove unused Tenant.getRepo method https://review.openstack.org/451929 | 20:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use new tenant project index for zuul role refs https://review.openstack.org/451928 | 20:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add source to project and remove unused tenant attrs https://review.openstack.org/451969 | 20:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fully qualify project configuration names https://review.openstack.org/451970 | 20:25 |
| SpamapS | Ran 210 (+16) tests in 820.741s (+46.911s) | 20:39 |
| SpamapS | FAILED (id=39, failures=2 (-2), skips=30) | 20:39 |
| SpamapS | Still hit the timeout even with OS_TEST_TIMEOUT=120 .. trying with 9999 now | 20:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fully qualify project configuration names https://review.openstack.org/451970 | 20:52 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/nodepool master: [WIP]: Add opensuse 42.2 DIB testing https://review.openstack.org/450045 | 21:17 |
| *** dkranz has joined #zuul | 21:23 | |
| openstackgerrit | Dirk Mueller proposed openstack-infra/nodepool master: [WIP]: Add opensuse 42.2 DIB testing https://review.openstack.org/450045 | 21:25 |
| *** dkranz has quit IRC | 21:28 | |
| jhesketh | Morning | 21:54 |
| *** harlowja has quit IRC | 22:02 | |
| mordred | it's a jhesketh | 22:07 |
| jeblair | jesusaur, SpamapS: i think i spotted the issue with test_client_enqueue_ref; left comments on 446275 | 22:08 |
| jesusaur | jeblair: cool, thanks | 22:21 |
| *** jlk has quit IRC | 22:43 | |
| *** jlk has joined #zuul | 22:44 | |
| *** jlk has quit IRC | 22:44 | |
| *** jlk has joined #zuul | 22:44 | |
| SpamapS | oh yay, I had gotten side-tracked by tests that timeout | 22:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fully qualify project configuration names https://review.openstack.org/451970 | 22:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add source to project and remove unused tenant attrs https://review.openstack.org/451969 | 22:51 |
| jesusaur | jeblair: your comment says that the current code schedules a merge for Refs, but one of the issues I was seeing was tracebacks in the merger when a merge was scheduled for a Ref because zuul_ref couldn't be constructed (+ not defined for types str and NoneType, because a Ref doesn't have a branch attribute) | 22:52 |
| jesusaur | jeblair: are you sure that we currently schedule merge jobs for post pipeline jobs? | 22:52 |
| jeblair | jesusaur: er, yeah, my comment is wrong. sorry. (that would only happen for a ref which changes the config and the updatesConfig method of Refs always returns false). so let's strike that paragraph. | 22:55 |
| jesusaur | ok | 22:55 |
| jesusaur | I'm going to test with setting merge_state to COMPLETE if we try to merge a Ref and see how that fares | 22:59 |
| jesusaur | jeblair: also, the other possible path to getLayout is from handling either ReconfigureEvent or TenantReconfigureEvent, do we need to worry about scheduling a merge if we are handling those events? | 23:03 |
| jesusaur | w00t, ok, down to 2 failures | 23:15 |
| jesusaur | i might actually get this working this week \o/ | 23:15 |
| SpamapS | nice | 23:15 |
| * mordred hands jesusaur a somewhat unused pancake | 23:16 | |
| jesusaur | mordred: lol | 23:16 |
| SpamapS | I'm still getting hard alarm clock timeouts, even on feature/zuulv3 tip on my laptop... | 23:16 |
| SpamapS | starting to think there's something else b0rken | 23:16 |
| jesusaur | SpamapS: yeah, I've taken to rerunning with --failing until the number of failing tests doesn't change; i really hope i'm not introducing a race somewhere | 23:17 |
| SpamapS | well thing is.. I'm running with OS_TEST_TIMEOUT=9999 ... so I'm not sure where the alarm clock is even coming from at this point. | 23:18 |
| jesusaur | oh, huh | 23:18 |
| jeblair | jesusaur: what's the path from ReconfigureEvent to getLayout? | 23:22 |
| *** harlowja has joined #zuul | 23:24 | |
| jeblair | jesusaur: the paths i'm seeing all go through processOneItem, so should be okay i think. unless i'm missing something. | 23:33 |
| jesusaur | jeblair: reEnqueueItem calls getLayout, which is called in _reenqueueTenant | 23:34 |
| jeblair | jesusaur: ah there it is :) | 23:34 |
| SpamapS | PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND | 23:37 |
| SpamapS | 16862 clint 20 0 3977472 1.336g 8136 D 106.3 8.7 8:36.40 python | 23:38 |
| SpamapS | 16856 clint 20 0 3400540 1.253g 8232 R 51.5 8.1 8:45.11 python | 23:38 |
| SpamapS | oops | 23:38 |
| SpamapS | meant to pastebin that.. | 23:38 |
| SpamapS | big RAM :-P | 23:38 |
| jeblair | jesusaur: it shouldn't need to schedule one (the normal algorithm will do that), but it does need to be state machine aware, so it should probably at least still keep the check for COMPLETE and otherwise return None. | 23:41 |
| *** openstack has joined #zuul | 23:46 | |
| SpamapS | So, in addition to going a bit faster.. I believe the libyaml patch also saves memory usage. | 23:53 |
| * SpamapS just trying to figure out how to run things reliably on a 16GB laptop | 23:53 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks https://review.openstack.org/446275 | 23:53 |
| SpamapS | jesusaur: so I am going to EOD soon, been watching you and jeblair and have held back diving in again. You're down to 2 fails you say? | 23:54 |
| jesusaur | SpamapS: actually just one now: my new test case around delayed merge conflicts | 23:55 |
| jesusaur | I'm going to EOD soon as well, but I think I'll be able to figure out my issue with the number of reports tomorrow, and then it'll be ready for review | 23:57 |
| SpamapS | sweet | 23:57 |
| SpamapS | I'll focus on some other stuff | 23:57 |
| jesusaur | cool | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!