Wednesday, 2018-01-31

*** kmalloc has quit IRC00:29
tristanCcorvus: would you mind replacing the V3 development section with the relevant storyboard link in the chan topic?01:05
tristanCcorvus: and is ansible-2.4 support in the scope of the 3.0 release?01:06
SpamapSI thought we had already removed the cap01:09
tristanCThe patches are still in flight, i added this topic: https://review.openstack.org/#/q/topic:zuul-ansible-2.401:14
mordredtristanC, SpamapS: I think we're waiting on figuring out how to roll out the transition for openstack01:31
*** rlandy has quit IRC01:32
*** haint_ has joined #zuul01:51
*** haint93 has quit IRC01:55
*** harlowja has quit IRC02:15
*** JasonCL has quit IRC03:19
*** mgagne has quit IRC03:22
*** jamielennox has quit IRC03:23
*** mgagne has joined #zuul03:24
*** mgagne is now known as Guest8724003:24
*** jamielennox has joined #zuul03:29
*** JasonCL has joined #zuul03:38
*** JasonCL has quit IRC03:39
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement a static driver for Nodepool  https://review.openstack.org/53555303:59
*** bhavik1 has joined #zuul04:03
*** bhavik1 has quit IRC04:14
Wei_LiutristanC: I implement a task to send the signal to scheduler pid to make it reload the tenant config with command "kill -HUP `cat /var/lib/zuul/run/zuul-scheduler.pid`", but it failed due to "No such process". Bubblewrap is used in my executor, so does it mean I can not kill a process out of bubblewrap?04:18
tristanCWei_Liu: indeed, bubblewrap doesn't give access to the host processes.04:21
tristanCWei_Liu: since you may want to scale executor on dedicated instance, perhaps you can setup ssh access and use "ssh scheduler kill -HUP"...04:22
SpamapSWei_Liu: I suggest ssh'ing back into the scheduler. They aren't intended to be on the same box usually.04:22
*** elyezer has quit IRC04:34
*** elyezer has joined #zuul04:37
*** elyezer has quit IRC04:44
*** elyezer has joined #zuul04:49
*** harlowja has joined #zuul04:51
*** elyezer has quit IRC04:55
*** elyezer has joined #zuul04:57
Wei_LiutristanC, SpampaS, Thanks05:13
*** threestrands has quit IRC05:51
*** xinliang has quit IRC05:57
*** threestrands has joined #zuul06:01
*** threestrands has quit IRC06:01
*** threestrands has joined #zuul06:01
*** xinliang has joined #zuul06:09
*** xinliang has quit IRC06:09
*** xinliang has joined #zuul06:09
*** xinliang has quit IRC06:21
*** xinliang has joined #zuul06:21
Wei_LiutristanC, SpamapS: wow, it worked, Thanks a lot. One more thing, can I use the same way to update the site-variables.yaml?06:30
tristanCWei_Liu: what do you mean by site-variables.yaml?06:31
SpamapSThat's a config item06:31
SpamapSvariables that can be set on every job06:32
SpamapSuseful for overriding defaults in zuul-jobs06:32
tristanChum, if it's in git repository, then maybe you could git push?06:32
Wei_LiutristanC, yes06:33
Wei_LiuSpamapS: sometimes, I want to publish some global variables to my zuul jobs, I will use the site-variables.yaml,06:35
SpamapSYes that's what it is for.06:37
SpamapSWei_Liu: although, if you just want global variables for a single tenant, you can set them on the base job.06:37
SpamapSwhich might be simpler and easier for users to discover06:38
tobiashSpamapS: did you already repropose the memory governor to master?06:46
tobiashI cannot find it06:46
tobiashSpamapS: do you mind if I cherry pick this to master?06:50
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Add memory awareness to system load governor  https://review.openstack.org/53942606:54
tobiashSpamapS: cherry picked and restored my last vote on that ^^06:54
*** elyezer has quit IRC06:57
*** harlowja has quit IRC07:01
*** elyezer has joined #zuul07:01
SpamapStobiash: I did not, pabelanger was taking it over07:02
SpamapStobiash: glad you're picking it up. :)07:02
tobiashSpamapS: ok :)07:02
* SpamapS just hasn't had the time :-P07:02
tobiashno problem, just wondered where that landed07:02
*** elyezer has quit IRC07:08
*** elyezer has joined #zuul07:12
*** jpena|off is now known as jpena08:43
*** elyezer has quit IRC08:57
*** elyezer has joined #zuul08:58
*** elyezer has quit IRC10:02
*** elyezer has joined #zuul10:14
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix statsd documentation about events  https://review.openstack.org/53948210:18
*** sshnaidm|afk is now known as sshnaidm10:56
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: [WIP] Allow operator to set held node's TTL  https://review.openstack.org/53949311:00
*** threestrands has quit IRC12:08
andreaftobiash, mordred: will zuul only merge dict variables from jobs? If I have a list variable and I what to enable children jobs to extend the list, do I have to make the list a dict with some fake values?12:40
andreaftobiash, mordred: the specific case is the list of extension of files to be renamed to _txt in stage-output - the list is made of unique elements so a dict would fix but having fake values would be a bit odd12:42
tobiashandreaf: I think that's in line with default behavior in ansible which by default overwrites top level variables12:43
tobiashotherwise one would need to implement tree merging which could have unwanted side effects for others expecting vars to overwrite each other12:43
tobiashbut I agree that your use case seems not to be fully supported yet12:44
tobiashandreaf: but you also could do this slightly different12:45
andreaftobiash: thanks - how12:45
andreaf?12:45
tobiashlike having a myfoofiles_<jobname> variable which is a list12:46
tobiashand add several of these in your inheritance structure12:46
tobiashyou could merge them in ansible then12:46
tobiashbut I guess that's more complicated12:47
andreaftobiash: heh nice idea - I guess jobname would still risk to cause overlaps though12:48
tobiashjob names are unique12:49
*** jpena is now known as jpena|lunch12:49
andreaftobiash: heh ok you mean using the full job name - yeah that works12:50
andreaftobiash: also I could have a single dictionary where keys are job names and values are lists and have ansible merge the lists12:51
tobiashandreaf: that would also work12:51
tobiashthat's probably better12:51
*** _ari_|brno is now known as _ari_12:56
*** rlandy has joined #zuul13:20
Shrewspabelanger: clarkb: corvus: So last night I had a thought about this fix for a wedged provider (per https://review.openstack.org/539248). Instead of declining the request based on whether we determine all of our nodes are unsued (which is not easily determined), why don't we choose to delete the oldest unused node if we are about to pause request handling? Then we are guaranteed to unpause "soon", and we don't13:27
Shrewsunnecessarily decline requests (which could happen for a long time until a request comes in that uses one of the already available nodes).13:27
ShrewsThis seems to be a MUCH better solution since it is pro-active rather than reactive.13:28
ShrewsIf there is no unused node to delete, then resources will free up through normal means anyway.13:29
ShrewsSo far, I've not been able to see any downside to this.13:29
tobiashShrews: I think that sounds good13:32
Shrewstobiash: cool. turns out deciding if 1 node is unused is MUCH more easily determined than deciding if ALL nodes are unused.13:34
*** cmurphy has joined #zuul13:34
tobiashand cheaper13:34
cmurphywhat is the difference between a playbook using `hosts: all` and `hosts: primary`? I notice that run.yaml playbooks seem to use the first and post.yaml seem to use the second but I don't know why13:36
cmurphyspecifically i'm trying to help figure out why this patch to run a new job on an opensuse-423 nodeset isn't running any plays and wondering if that's why https://review.openstack.org/#/c/52006313:42
Shrewscmurphy: that job seems to be using nodeset opensuse-423, which does not define a 'primary' node, which i think comes from here: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/nodesets.yaml#n2013:51
*** jpena|lunch is now known as jpena13:51
Shrewscmurphy: 'hosts: all' is sort of a catch-all13:51
cmurphyShrews: oh that's interesting13:52
cmurphyShrews: is there any particular reason we should or shouldn't define a primary for that nodeset?13:53
Shrewscmurphy: *shrug*13:53
Shrewsyou can see other nodesets define a primary, but they all have additional nodes in the set too13:54
Shrewswell, not all i guess13:54
cmurphyif the playbook has hosts: all and the nodeset has multiple nodes would the job get run on all of them?13:55
Shrewsyes13:55
cmurphyokay so we probably don't want to change to hosts: all13:55
Shrewsin this case, looks like you can use 'all' since there is only a single node13:56
Shrewsshould be the same result as 'hosts: opensuse-423'13:56
Shrewslooks like the use 'primary' was just a copy-pasta error maybe13:57
cmurphywell the playbook was written for a job using nodeset legacy-ubuntu-xenial which has primary defined13:58
Shrewslegacy-opensuse-423 had a 'primary' node: https://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/nodesets.yaml#n13413:58
Shrewsyeah, looks like all the legacy ones do13:59
cmurphyokay sounds like changing to hosts: all is going to be fine13:59
cmurphythanks Shrews13:59
*** yolanda has quit IRC14:01
*** yolanda has joined #zuul14:03
*** elyezer has quit IRC14:06
*** elyezer has joined #zuul14:07
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL  https://review.openstack.org/53949314:11
mordredandreaf, tobiash: haven't read the whole scrollback, but yes, currently only dicts are merged, lists are replaced14:16
andreafmordred: cool thanks14:17
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix lazy initialization of GithubUser with apps  https://review.openstack.org/53954514:17
*** dkranz has joined #zuul14:17
mordredShrews: yes, I like your idea14:17
Shrewsmordred: bueno. oh, i owe you reviews today too. will try to get to that14:19
Shrewsi also owe myself breakfast. biab14:21
*** sshnaidm is now known as sshnaidm|mtg14:27
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL  https://review.openstack.org/53949314:34
fungicmurphy: Shrews: the "primary" node was used to accommodate old-style multinode jobs converted from v2 configuration, but native v3 jobs are no longer constrained to a primary/subnode model14:51
*** JasonCL has joined #zuul14:51
fungiand i'm pretty sur single-node jobs were converted as if they were just a special case "one-node multinode" design14:52
fungiso a primary with no subnodes14:52
fungimainly because the nodepool metadata used to differ between primary and subnodes14:53
*** elyezer has quit IRC14:53
fungiand nodepool v2 treated stand-alone nodes as a primary with no other subnodes14:54
cmurphywhy then did this single node job get converted to hosts: all instead of hosts: primary if it doesn't really matter? http://git.openstack.org/cgit/openstack/keystone/tree/playbooks/legacy/keystone-dsvm-functional/run.yaml14:55
cmurphyi guess maybe there was no way of knowing it was a single node job when it was converted14:55
*** elyezer has joined #zuul14:56
Shrewscmurphy: yeah, the script we used to convert them wasn't perfect15:00
* Shrews still gets chills thinking about that script.... *shudder*15:01
pabelangeralso using hosts: all means we don't need to add specific nodesets to jobs. A little easier to switch between fedora / ubuntu / etc15:03
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL  https://review.openstack.org/53949315:10
*** elyezer has quit IRC15:11
*** elyezer has joined #zuul15:24
corvusandreaf: well, a lot of the devstack stuff works by having a 'fake' value of "True", so it may not be that odd.15:26
*** Guest87240 is now known as mgagne15:27
*** mgagne has joined #zuul15:27
andreafcorvus: heh you're not jeblair anymore - that's why I didn't see you online :)15:28
mordredShrews: never think about that script again15:31
andreafcorvus: well if I have a dict with true / false I kind of set the expectation that false actually means false and something other than False / True triggers an error15:31
andreafcorvus: but actually it may even be easier to implement that than the dict of lists15:32
corvusandreaf: yeah, the devstack conf modules also honor false -- they act as if the value wasn't there.  that lets child jobs override parents and remove things15:32
*** sshnaidm|mtg has quit IRC15:32
*** ChanServ changes topic to "Discussion of the project gating system Zuul | Docs: http://docs.openstack.org/infra/zuul/ | Source: https://git.openstack.org/cgit/openstack-infra/zuul/ | Roadmap: https://storyboard.openstack.org/#!/board/53 | Channel logs: http://eavesdrop.openstack.org/irclogs/%23zuul/"15:33
andreafcorvus: the thing I'm struggling with atm is providing some kind of backward compatibility - if the var is a dict to A, if it's a list to B - how do I do that check in ansible?15:33
corvustristanC: good idea, topic changed, thanks :)15:34
*** sshnaidm|mtg has joined #zuul15:34
corvusandreaf: i bet you could use a jinja expression for that.  something like "when: varname is mapping"  maybe?15:36
ShrewsFYI, I have no water at home (water main break). I may be in and out as I move to different places to remain comfortable today.15:44
mordredShrews: wait - you had running water to begin with?15:47
clarkbShrews: yes I think deleting oldest ready node if there is a ready node would work15:47
clarkbthen we only delete one node instead of potentially 50 for example15:47
andreafcorvus: when extensions_to_txt | type_debug == 'dict' works fine15:51
andreafcorvus: I'm not sure type_debug was meant for that though15:51
*** sshnaidm|mtg has quit IRC15:52
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL  https://review.openstack.org/53949315:52
clarkblooks like mirror02.dfw.rax.openstack.org has the dfw mirror cname and cacti makes it looks reasonably happy?15:56
clarkber ww15:56
*** sshnaidm|mtg has joined #zuul16:08
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL  https://review.openstack.org/53949316:21
tobiashShrews: do you think it's possible to add a 'fast track' for requests which can be fulfilled immediately by ready nodes regardless if the provider is paused or not?16:44
tobiashthat would be also an optimization if we're at quota, provider is paused but there are ready nodes of some types16:45
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] zuul autohold: allow operator to specify nodes TTL  https://review.openstack.org/53959616:48
Shrewstobiash: the question should be "is that a good idea" since it could allow requests to jump our priority queue16:49
ShrewsI don't have the answer to that16:49
clarkbI think if we have excess ready nodes able to jump the queue that is fine (at least thinking about it really quickly that allows us to free resources for older  requests)16:50
tobiashthat's also what I thought16:52
tobiashas long as we don't create a starvation problem16:52
*** sshnaidm|mtg is now known as sshnaidm17:01
tobiashpabelanger: did you see SpamapS's comment on https://review.openstack.org/#/c/537953/ ?17:01
pabelangertobiash: yah, I can push up a quick fix.17:02
tobiashkk17:02
*** harlowja has joined #zuul17:02
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Enabled ssh retries for ansible  https://review.openstack.org/53795317:05
pabelangerokay, updated17:05
tobiash:)17:06
tobiashmordred, ianw: I think https://storyboard.openstack.org/#!/story/2001487 can be resolved now right?17:11
*** bhavik1 has joined #zuul17:12
mordredtobiash: yah17:16
mordredtobiash: done17:16
*** leifmadsen_ is now known as leifmadsen17:21
*** myoung is now known as myoung|rabbit17:23
*** bhavik1 has quit IRC17:34
jlkSo, GitHub just shipped a feature that officially recognizes the Co-Authored-By lines in a commit, and gives author credit in the UI to those listed.17:40
jlke.g. https://github.com/openstack-infra/zuul/commit/4fc12549072d02b97fa46ab36b4618f320ce507f17:40
jlkhttps://github.com/blog/2496-commit-together-with-co-authors17:42
tobiashjlk: cool17:51
tobiashjlk: do you know if there's an api to query just the code review status of a pr?17:51
tobiashI didn't find one while trying to evaluate needed reviews for the gate in zuul17:52
jlktoabctl: one sec.17:52
jlktoabctl: so, you kind of have to sort that out for yourself.  GET /repos/:owner/:repo/pulls/:number/reviews   You can get all the reviews for a given PR, and then you have to determine A) what was the last review that mattered per person, and does that person have merge rights.17:54
jlkthere isn't an exposed API of "this PR has all the necessary reviews to be merged", that's more about branch protection settings, which include other factors.17:54
mordredjlk: neat!17:56
tobiashjlk: hrm, we probably need this in the future17:59
tobiashWhen working with codeowners this is going to be complicated17:59
jlkyeah, it's a tough nut to crack18:01
tobiashMaybe I should open a support ticket with a feature request18:01
jlkI think what we want to know is "are all requested reviews met"18:01
tobiashYes18:01
jlkthat's part of what you get with code owners.18:01
clarkbcorvus added a similar feature to gerrit18:01
jlkYou can get https://developer.github.com/v3/pulls/review_requests/#list-review-requests18:01
jlkwhich is a list of all the requested reviews, and map them up to what you've figured out from the existing reviews18:02
jlkIn theory, figuring this all out would be easier with GraphQL, at least in the number of queries necessary. But that's still not on for Apps18:03
tobiashAh, ok, that could make a workaround possible18:03
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Look for 0 in .stestr directory instead of failing  https://review.openstack.org/53962418:11
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Skip checksums on stat calls in fetch-subunit-output  https://review.openstack.org/53962518:11
clarkbmordred: ^ changes like that make me wonder if ansible should have a checksum module for checksumming and stat should just stat18:12
*** jpena is now known as jpena|away18:13
mordredclarkb: yes. I very much do not like that stat checksums by default18:13
mordredclarkb: to be fair - what I REALLY want most of the time is an 'exists' module that just returns a single bool18:14
mordredclarkb: rather than needing to look at stat_results.stat.exists18:14
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] zuul autohold: allow operator to specify nodes TTL  https://review.openstack.org/53959618:15
jlkmordred: I think a lot of people want that too18:17
jlk"does this exist yes/no"18:17
mordredjlk: I'm writing it right now18:17
mordredjlk: I have reached my annoyance threshold18:17
jlkdeveloping in anger18:18
mordredit's how I work :)18:18
*** myoung|rabbit is now known as myoung18:22
*** dkranz has quit IRC18:44
mordredjlk: https://github.com/ansible/ansible/pull/3557218:45
mordredjlk: I tagged you in it :)18:45
jlkreading18:46
corvusShrews, mhu: should we drop the 'nodepool hold' command?  (i think i thought we had already).  in v3, it can really only be used to hold a ready node, not a used node.  is that still useful?18:50
corvusprompted by https://review.openstack.org/53949318:51
*** harlowja has quit IRC18:52
corvusShrews: also https://review.openstack.org/535553 and parent are ready for your +318:52
corvusmordred: ^ you may be interested18:52
*** sshnaidm is now known as sshnaidm|off18:55
mordredcorvus: ooh, that's exciting18:56
jlkmordred: I've thrown some words back at you on that PR.19:00
mordredwoot19:06
mordredjlk: done19:20
jlkmordred: have you thought about making it a fact module, to skip the "register" part, and have it set a fact name based on the path? (that might be TOO magic)19:21
jlkI don't know how long it's been doing this, but I like that GitHub UI shows when a force push happens in a PR timeline.19:23
*** jpena|away is now known as jpena|off19:24
corvusi've culled through many of the open nodepool changes; we're down to 52 open changes now, a bit less than half are pre-v3 changes.  it'd be great of other zuul-core folks can pitch in on abandoning or WIPing old changes.  here's the messages i've been leaving: http://paste.openstack.org/show/658218/19:26
*** dkranz has joined #zuul19:27
corvusmordred: about half of the nodepool ones left are from you, so you may be particularly suited to triaging those :)19:27
*** dkranz has quit IRC19:29
*** harlowja has joined #zuul19:37
Shrewscorvus: huh, i thought we had removed the hold command19:39
Shrewsit's pretty worthless right now19:40
corvusShrews: apparently a shared delusion!19:42
Shrewsthose are the best19:43
Shrewscorvus: i left a comment on the parent change19:47
mordredcorvus: I think most of my nodepool patches in that stack are bong - but I'll go through them19:53
mordredjlk: yah - the force push visualization is very helpful19:54
mordredjlk: for the other thing - I think *I* would like that, but I have a hunch it's too magic to get right19:54
jlknod19:54
mordredjlk: what I REALLY want, lf course, is 'when: exists(path)'19:54
tobiashlike a remote lookup plugin?19:55
tobiashis that possible?19:55
openstackgerritMerged openstack-infra/zuul master: Enabled ssh retries for ansible  https://review.openstack.org/53795319:55
mordredtobiash: nope19:55
mordredtobiash: I'm just wishing for magical ponies19:56
mordredjlk: zomg. we could do fact setting and go full-on old school unix on it - have it set a "last_exists" fact every time - kind of like $?19:56
jlkerm....19:56
mordredexists: path=foo .... when: last_exists ...19:56
mordred**terrible** idea - just saying19:56
*** elyezer has quit IRC20:14
*** dkranz has joined #zuul20:14
openstackgerritAndrea Frittoli proposed openstack-infra/zuul-jobs master: Change the list of extensions to a dict  https://review.openstack.org/53968320:57
openstackgerritAndrea Frittoli proposed openstack-infra/zuul-jobs master: Change the list of extensions to a dict  https://review.openstack.org/53968320:58
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Merger: retry network operations  https://review.openstack.org/53935621:20
corvusShrews: the func tests passed the recheck so i approved 53555321:22
Shrewscorvus: great thx21:23
*** dkranz has quit IRC21:58
*** myoung is now known as myoung|off22:09
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Support the fragment form of Gerrit URLs  https://review.openstack.org/53970522:13
corvusmordred, Shrews: ^22:14
corvusclarkb: can you review https://review.openstack.org/538353 when you get a chance?22:16
clarkbjlk: ya looking22:22
clarkber22:22
clarkbcorvus: ^22:22
* jlk waves22:22
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add available RAM to statsd  https://review.openstack.org/53970722:22
corvusjhesketh: with ^ are you okay to approve https://review.openstack.org/539426 ?22:23
corvusjlk: https://review.openstack.org/539545 seems to me like an easy github review22:26
* jlk reads22:27
corvustobiash: https://review.openstack.org/537428 approved with comments22:48
openstackgerritMerged openstack-infra/zuul master: Fix statsd documentation about events  https://review.openstack.org/53948222:53
corvusSpamapS: what's the latest on re2?22:57
openstackgerritMerged openstack-infra/zuul master: Fix lazy initialization of GithubUser with apps  https://review.openstack.org/53954523:00
SpamapScorvus: I have not gotten a reply back on my latest PR push. I'll ping the reviewers.23:03
corvusSpamapS: ack23:04
SpamapShttps://github.com/facebook/pyre2/pull/1023:04
SpamapSjust commented23:04
corvusSpamapS: should 536389 Depends-On that?23:05
corvusSpamapS: (i mean, zuul won't do anything at this point since it's not a defined project, but that's not the only reason to include that)23:06
corvusi guess it really depends on that plus a release23:06
SpamapScorvus: yes actually.23:06
SpamapSWe could ask if they want to join our cross-repo testing party. :)23:07
corvusmaybe someday -- though i also think we can get to the point where you could depends-on that PR and actually have it work, without otherwise adding their repo to the system.23:08
corvus(basically, dynamically evaluate required-projects.  which is also one of my ideas for reducing memory usage, so that may come sooner than you would otherwise think)23:08
SpamapSTrue, I think the way the driver works now it does a search for the PR's status, so while it wouldn't trigger gate entry for the zuul patch on commit of the pyre2 commit, a Depends-On would prevent it from merging to zuul until the pyre2 PR merges, yes?23:09
corvusSpamapS: i think so, though i'm not 100% sure that works for projects that aren't in the tenant.  (though there's no reason it shouldn't/couldn't)23:11
SpamapSfurther experimentation is necessary23:11
corvus++23:11
*** rlandy is now known as rlandy|bbl23:35

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!