corvus | mwhahaha: if you add the "debug: true" setting to the project-pipeline config (i recommend you do this in the change where you're also changing the config, it'll only apply to that one so it won't be global) when zuul reports back, it will include a bunch of info about which variants did/did not match | 00:49 |
---|---|---|
mwhahaha | Ok I'll have to try that. It wasn't honoring my change to branches in the results | 00:50 |
corvus | yeah, add that to that change and it should tell you why | 00:53 |
corvus | https://zuul-ci.org/docs/zuul/user/config.html#attr-project.%3Cpipeline%3E.debug | 00:54 |
*** ianychoi_ has quit IRC | 02:27 | |
*** ianychoi has joined #zuul | 02:30 | |
openstackgerrit | Fergus Yu proposed openstack-infra/zuul master: Fix indent error for layout.yaml-sample. https://review.openstack.org/576684 | 02:52 |
*** mnaser has quit IRC | 03:42 | |
*** mnaser has joined #zuul | 03:45 | |
*** mwhahaha has quit IRC | 03:48 | |
*** sdoran has quit IRC | 03:48 | |
*** robcresswell has quit IRC | 03:49 | |
*** LinuxJedi has quit IRC | 03:49 | |
*** jtanner has quit IRC | 03:49 | |
*** zxiiro has quit IRC | 03:49 | |
*** gregdek has quit IRC | 03:49 | |
*** mattclay has quit IRC | 03:49 | |
*** spsurya has quit IRC | 03:49 | |
tristanC | corvus: Shrews: could we do a nodepool-3.0.2 or 3.1.0 release soon? The last 3.0.1 is missing the paused handler and the connection-port (for static nodes) fix | 03:49 |
*** jbryce has quit IRC | 03:49 | |
*** maxamillion has quit IRC | 03:49 | |
*** gundalow has quit IRC | 03:50 | |
*** portdirect has quit IRC | 03:50 | |
*** abelur has quit IRC | 03:50 | |
*** TheJulia has quit IRC | 03:50 | |
*** hogepodge has quit IRC | 03:50 | |
*** mnaser has quit IRC | 03:51 | |
*** fdegir has quit IRC | 03:51 | |
*** patrickeast has quit IRC | 03:51 | |
*** Guest19794 has quit IRC | 03:51 | |
*** mrhillsman has quit IRC | 03:51 | |
*** kmalloc has quit IRC | 03:51 | |
openstackgerrit | Merged openstack-infra/zuul master: Add a CVE-2018-12557 release note https://review.openstack.org/576545 | 04:06 |
*** openstack has joined #zuul | 04:32 | |
*** ChanServ sets mode: +o openstack | 04:32 | |
*** gundalow has joined #zuul | 05:23 | |
*** abelur has joined #zuul | 05:25 | |
*** TheJulia has joined #zuul | 05:25 | |
*** zxiiro has joined #zuul | 05:25 | |
*** jtanner has joined #zuul | 05:28 | |
*** mwhahaha has joined #zuul | 05:28 | |
*** sdoran has joined #zuul | 05:28 | |
*** patrickeast has joined #zuul | 05:30 | |
*** LinuxJedi has joined #zuul | 05:30 | |
*** maxamillion has joined #zuul | 05:31 | |
*** mattclay has joined #zuul | 05:34 | |
*** fdegir has joined #zuul | 05:35 | |
*** kmalloc has joined #zuul | 05:40 | |
*** mnaser has joined #zuul | 05:42 | |
*** swest has quit IRC | 05:47 | |
*** spsurya has joined #zuul | 05:47 | |
*** mnaser has quit IRC | 05:51 | |
*** mnaser has joined #zuul | 05:51 | |
*** robcresswell has joined #zuul | 05:53 | |
*** abelur has quit IRC | 05:55 | |
*** mnaser has quit IRC | 05:56 | |
*** mwhahaha has quit IRC | 05:56 | |
*** jtanner has quit IRC | 05:56 | |
*** kmalloc has quit IRC | 05:56 | |
*** sdoran has quit IRC | 05:57 | |
*** gundalow has quit IRC | 05:57 | |
*** spsurya has quit IRC | 05:57 | |
*** zxiiro has quit IRC | 05:57 | |
*** maxamillion has quit IRC | 05:57 | |
*** robcresswell has quit IRC | 05:58 | |
*** LinuxJedi has quit IRC | 05:58 | |
*** TheJulia has quit IRC | 05:58 | |
*** fdegir has quit IRC | 05:58 | |
*** patrickeast has quit IRC | 05:58 | |
*** mattclay has quit IRC | 05:58 | |
*** mnaser has joined #zuul | 06:10 | |
*** nchakrab has joined #zuul | 06:13 | |
*** patrickeast has joined #zuul | 06:20 | |
*** mrhillsman has joined #zuul | 06:20 | |
*** TheJulia has joined #zuul | 06:20 | |
*** jbryce has joined #zuul | 06:21 | |
*** zxiiro has joined #zuul | 06:21 | |
*** Rohaan has joined #zuul | 06:21 | |
*** portdirect has joined #zuul | 06:21 | |
*** portdirect has quit IRC | 06:21 | |
*** portdirect has joined #zuul | 06:21 | |
*** maxamillion has joined #zuul | 06:23 | |
*** mattclay has joined #zuul | 06:23 | |
*** kmalloc has joined #zuul | 06:23 | |
*** fdegir has joined #zuul | 06:23 | |
*** gundalow has joined #zuul | 06:24 | |
*** abelur has joined #zuul | 06:24 | |
*** jtanner has joined #zuul | 06:24 | |
*** mwhahaha has joined #zuul | 06:25 | |
*** LinuxJedi has joined #zuul | 06:26 | |
*** spsurya has joined #zuul | 06:26 | |
*** robcresswell has joined #zuul | 06:26 | |
*** sdoran has joined #zuul | 06:26 | |
*** gtema has joined #zuul | 06:44 | |
*** gregdek has joined #zuul | 06:55 | |
*** Guest19794 has joined #zuul | 06:59 | |
*** hogepodge has joined #zuul | 07:00 | |
*** Rohaan___ has joined #zuul | 07:12 | |
*** Rohaan has quit IRC | 07:12 | |
*** hashar has joined #zuul | 07:20 | |
*** swest has joined #zuul | 07:43 | |
*** Rohaan___ has quit IRC | 07:48 | |
*** Rohaan___ has joined #zuul | 07:49 | |
*** Rohaan___ has quit IRC | 08:05 | |
*** pcaruana has joined #zuul | 08:11 | |
*** Rohaan___ has joined #zuul | 08:24 | |
*** swest has quit IRC | 08:24 | |
*** wei_liu_ has joined #zuul | 08:46 | |
*** spsurya has quit IRC | 08:46 | |
*** wei_liu_ has quit IRC | 08:48 | |
*** sshnaidm|afk is now known as sshnaidm | 08:57 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: github: prevent KeyError in getProjectBranches when rate limitted https://review.openstack.org/576799 | 09:13 |
corvus | tristanC: can the nodepool release wait till next week? | 09:26 |
openstackgerrit | Ian Wienand proposed openstack-infra/zuul-base-jobs master: Add validate-host and log-inventory https://review.openstack.org/576802 | 09:28 |
tristanC | corvus: sure | 09:28 |
*** electrofelix has joined #zuul | 09:34 | |
openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/zuul-jobs master: Log output and exception when traceroute fails https://review.openstack.org/576351 | 09:45 |
*** jesusaur has quit IRC | 09:49 | |
*** jesusaur has joined #zuul | 09:53 | |
*** nchakrab_ has joined #zuul | 09:58 | |
*** nchakrab has quit IRC | 10:01 | |
*** nchakrab_ has quit IRC | 10:02 | |
*** nchakrab has joined #zuul | 10:19 | |
*** jpena is now known as jpena|lunch | 11:09 | |
*** gtema has quit IRC | 11:37 | |
*** nchakrab has quit IRC | 11:43 | |
*** spsurya has joined #zuul | 11:43 | |
Shrews | we really should do a np restart to get that paused handler fix in | 11:46 |
Shrews | and i forgot a release note for that fix. will make one | 11:48 |
openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add release note for paused handler fix https://review.openstack.org/576827 | 11:54 |
*** nchakrab has joined #zuul | 12:09 | |
*** jpena|lunch is now known as jpena | 12:15 | |
*** nchakrab_ has joined #zuul | 12:17 | |
*** nchakrab has quit IRC | 12:21 | |
*** rlandy has joined #zuul | 12:22 | |
tobiash | Shrews: +3 | 12:33 |
openstackgerrit | Merged openstack-infra/nodepool master: Add release note for paused handler fix https://review.openstack.org/576827 | 12:43 |
openstackgerrit | Merged openstack-infra/zuul master: Fix indent error for layout.yaml-sample. https://review.openstack.org/576684 | 12:45 |
*** myoung|off is now known as myoung | 12:48 | |
*** gtema has joined #zuul | 12:52 | |
*** nchakrab has joined #zuul | 13:17 | |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Make streaming test run longer than connection timeout https://review.openstack.org/572522 | 13:19 |
*** nchakrab_ has quit IRC | 13:20 | |
Shrews | fyi: I will be leaving in 30m for another dr appt | 13:21 |
mordred | Shrews: good luck with that | 13:23 |
Shrews | thx | 13:28 |
*** nchakrab_ has joined #zuul | 13:31 | |
*** nchakrab has quit IRC | 13:34 | |
*** maxamillion has quit IRC | 13:38 | |
*** maxamillion has joined #zuul | 13:38 | |
*** hashar is now known as hasharAway | 13:39 | |
*** Rohaan___ is now known as Rohaan | 13:45 | |
*** nchakrab has joined #zuul | 13:48 | |
*** nchakrab_ has quit IRC | 13:51 | |
tristanC | Shrews: good luck, hope it goes well! | 14:04 |
*** nchakrab_ has joined #zuul | 14:16 | |
*** nchakrab has quit IRC | 14:19 | |
*** Rohaan has quit IRC | 14:58 | |
*** nchakrab_ has quit IRC | 15:04 | |
*** acozine1 has joined #zuul | 15:09 | |
*** acozine1 has quit IRC | 15:10 | |
*** acozine1 has joined #zuul | 15:16 | |
*** rlandy is now known as rlandy|mtg | 15:48 | |
*** hasharAway is now known as hashar | 16:01 | |
openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] web: add tenant-scoped autohold, enqueue https://review.openstack.org/576907 | 16:26 |
*** rlandy|mtg is now known as rlandy | 16:46 | |
*** sshnaidm is now known as sshnaidm|off | 16:52 | |
tobiash | corvus, mordred: do you find these messages useful (the starting finished queue processor)? http://paste.openstack.org/show/723945/ | 17:05 |
tobiash | corvus, mordred: I'm asking because they are log that for every pipeline in every tenant | 17:06 |
tobiash | corvus, mordred: this spams 150 lines every few seconds in our deployment | 17:06 |
*** jpena is now known as jpena|off | 17:07 | |
tobiash | corvus, mordred: I'm not asking about the 'what do the pipelines' but just about the start and end of each pipeline processing step | 17:07 |
mordred | tobiash: I don't personally - but I don't commune with the scheduler logs as deeply as corvus does | 17:09 |
tobiash | the start/finish messages actually make it hard for me to spot the actual work within our now 78 pipelines distributed over 13 tenants | 17:14 |
openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] web: add tenant-scoped autohold, enqueue https://review.openstack.org/576907 | 17:18 |
*** spsurya has quit IRC | 17:23 | |
*** gtema has quit IRC | 17:23 | |
*** gtema has joined #zuul | 17:25 | |
*** gtema has quit IRC | 17:33 | |
pabelanger | so, I've noticed the fetch-zuul-cloner role doesn't take into account the git URL in the CLI, for example | 17:45 |
pabelanger | the zuul-cloner role sets repo_src_dir: "/home/zuul/src/review.rdoproject.org" then the following fails: | 17:46 |
pabelanger | /usr/zuul-env/bin/zuul-cloner -m clonemap.yaml --cache-dir /opt/git git://git.openstack.org openstack/tripleo-quickstart | 17:46 |
pabelanger | because it looks in the wrong directory | 17:46 |
*** SpamapS has quit IRC | 17:46 | |
pabelanger | to fix it , I think we need to set repo_src_dir and update the zuul-cloner copy script to parse out the correct URL and append it to reps_src_dir | 17:47 |
pabelanger | otherwise, have to do a bunch of symlinks for mulitple zuul connections | 17:47 |
*** electrofelix has quit IRC | 17:59 | |
*** bhavik1 has joined #zuul | 18:10 | |
*** hashar is now known as hasharAway | 18:10 | |
*** myoung is now known as myoung|lunch | 18:11 | |
mordred | pabelanger: well, it's supposed to be a short-lived tool with no features | 18:26 |
mordred | pabelanger: if I were you, I'd fork it for rdo so you can not be stuck while getting folks off of it | 18:27 |
mordred | (just because that way you can just hard-code rdoproject.org in the fork, right?) | 18:27 |
*** gtema has joined #zuul | 18:28 | |
*** bhavik1 has quit IRC | 18:28 | |
*** yolanda_ has joined #zuul | 18:35 | |
goern | hey all, I have extended one job (the linter) but my part requires coala (which is not installed by default) what are the keywords to search for? | 18:35 |
*** ianychoi has quit IRC | 18:36 | |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Allow fetch-zuul-cloner to support multiple connections https://review.openstack.org/576933 | 18:37 |
pabelanger | mordred: sure, we say it is short-lived, but it lives in zuul-jobs. So, I'd expect people to consume it. I don't have an issue forking it, but makes migrating from JJB harder. Maybe it should just be in openstack-zuul-jobs | 18:38 |
pabelanger | also, what I am thinking for an update ^ | 18:38 |
pabelanger | but will break openstack-infra today, because of REPO_SRC_DIR default, we'd need to stage the change | 18:38 |
pabelanger | also, totally untested | 18:38 |
*** yolanda__ has joined #zuul | 18:39 | |
*** yolanda has quit IRC | 18:39 | |
*** yolanda_ has quit IRC | 18:40 | |
mordred | goern: heya! so ... 'it depends' (I know, everyone's favorite answer) | 18:44 |
mordred | goern: is this the linter job in zuul-jobs you're talking about extending? | 18:44 |
fungi | goern: what linter job did you extend? if it runs under tox, then adding your additional linter dependency to your test-requirements.txt would probably work, or to a deps list in the specific testenv in your tox.ini | 18:44 |
mordred | goern: (and do you have a link I could look at?) | 18:44 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Allow fetch-zuul-cloner to support multiple connections https://review.openstack.org/576933 | 18:45 |
mordred | pabelanger: I'm not opposed to fixing it in zuul-jobs- just mostly was thinking that, as you mention, doing that is going to require _care_ - and I didn't want you to get too blocked on improving a thing we'd eventually like to delete | 18:47 |
mordred | pabelanger: but maybe I'm wrong about the deleting thing and maybe it should stay around in zuul-jobs for longer to facilitate migrations for people? | 18:48 |
mordred | pabelanger: (thinking out loud) | 18:48 |
pabelanger | yah, I think we should maybe discuss the jenkins migration story a bit more. While I've been able to make some good progress with zuul-migrate and rdoproject.org, there is still a lot it doesn't cover. But I'd say 80% of the jobs are functional by running it. Which, is an easy win to move to zuul | 18:50 |
pabelanger | but also agree, we can just fork it in rdo to keep moving forward | 18:50 |
mordred | pabelanger: ++ | 18:51 |
mordred | pabelanger: I mostly want to make sure I'm not making your life harder while we discuss it | 18:52 |
mordred | oh - and yeah - zuul-migrate is ... imperfect at best | 18:52 |
pabelanger | worked well for single node jobs | 18:53 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Allow fetch-zuul-cloner to support multiple connections https://review.openstack.org/576933 | 18:53 |
mordred | pabelanger: yah. the extra complicated stuff is where it goes south | 18:55 |
pabelanger | yup, I think we if ever revisit. Maybe strip out everything but stuff for single node jobs. More complicated jobs (eg: multinode) just need to be native zuulv3 jobs | 18:57 |
mordred | yah. there's just no way to do them automatically | 18:59 |
goern | mordred, fungi ja... it is ... https://github.com/goern/thoth-zuul-test-jobs/blob/feature/adding-linter-coala/roles/linters/tasks/lint_coala.yaml and we are using Pipenv all over the place.. rather then tox or requirements.txt | 19:01 |
goern | mordred, fungi so the questions seems to be: how to use Pipenv with jobs? | 19:02 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Allow fetch-zuul-cloner to support multiple connections https://review.openstack.org/576933 | 19:06 |
*** myoung|lunch is now known as myoung | 19:06 | |
*** SpamapS has joined #zuul | 19:06 | |
*** SpamapS has quit IRC | 19:07 | |
*** SpamapS has joined #zuul | 19:07 | |
*** yolanda_ has joined #zuul | 19:08 | |
*** yolanda_ has quit IRC | 19:09 | |
*** yolanda_ has joined #zuul | 19:10 | |
fungi | goern: it's mostly a question of methodology i guess. we settled on tox as an entrypoint for executing those sorts of jobs because that way there's a single, simple command our developers can also run to repeat the same tests locally. this allows us to provide a more consistent experience in the ci system as in their local dev environments | 19:10 |
goern | fungi, sounds about right | 19:11 |
fungi | goern: so the way i'd approach it is first ask what you've documented that your developers should run locally to test their changes, and orchestrate the same set of steps in your job definitions | 19:11 |
*** yolanda__ has quit IRC | 19:12 | |
fungi | if that's some assortment of shell commands including pipenv and pip install and running coala from the env or whatever, then encode the same steps in your job definitions | 19:12 |
fungi | we found that if jobs perform substantially different steps than the devs run locally, they get confused, frustrated, even angry when they can't reproduce failures the ci system is reporting back to them | 19:13 |
goern | fungi, ja... we are coming from travis... so its `pipenv install; pipenv run coala` | 19:13 |
fungi | in that case, that's probably what i'd recommend putting in the job. how does pipenv install know to install coala? | 19:14 |
fungi | i'm admittedly not that familiar with pipenv... presumably a requirements file or something? | 19:14 |
goern | fungi, it has a look at a file called Pipfile in the / of the repo | 19:14 |
fungi | ahh, okay, that sounds familiar. so if your pipfile includes coala then in theory that should just do what you want right? | 19:15 |
goern | fungi, ja, but I need to pip install --user pipenv first, right? $HOME/.local/bin is in the path of that jobs... so it would find subsequent pipenv calls | 19:15 |
fungi | goern: sure. if you expect your developers to install and use pipenv then ideally your job would install it the same way they do | 19:16 |
mordred | yah - in our tox jobs, we actually have a base job that installs tox - so I think potentially a similar base job that installs pipenv that can be used by people using pipenv is not a terrible idea | 19:16 |
goern | fungi, ack. is there a base/general job that handles tox? | 19:16 |
fungi | there is. it's in the zuul-jobs standard library | 19:17 |
mordred | (also, that's the sort of base job that would make sense in zuul-jobs) | 19:17 |
mordred | https://github.com/openstack-infra/zuul-jobs/blob/master/zuul.yaml#L15 | 19:17 |
mordred | goern: ^^ | 19:17 |
*** yolanda_ has quit IRC | 19:17 | |
mordred | goern: fwiw, there's another tool we use that we have plumbed in that may or may not be useful to you called bindep | 19:18 |
goern | mordred, I'll basically copy that job and make it install pipenv | 19:18 |
mordred | goern: ++ | 19:18 |
*** zaro__ has joined #zuul | 19:18 | |
mordred | goern: bindep is how we have people specify distro package depends they need - and the base jobs there will install what's needed - you don't have to do anything with it if you don't want to - but it's a use case that comes up so I figured I'd mention it | 19:19 |
fungi | also known as http://git.zuul-ci.org/cgit/zuul-jobs/tree/zuul.yaml#n15 | 19:19 |
mordred | fungi: good point | 19:19 |
*** yolanda has joined #zuul | 19:19 | |
fungi | or https | 19:20 |
goern | mordred, fungi bindep sounds like binary dependency... install moar RPM or DEB?! pipenv is a python module | 19:20 |
mordred | https://git.zuul-ci.org/cgit/zuul/tree/bindep.txt <-- is an example in case that helps | 19:20 |
mordred | goern: oh- totally | 19:20 |
mordred | goern: we use ot more for things like "lxml requires libxml-dev to be installed for pip to be able to install it" | 19:20 |
goern | mordred, gotthat | 19:21 |
fungi | goern: right. tox uses pip to install things but you could make a pipenv job which did a `pip install --user pipenv;pipenv install` and as mordred says that would probably be a great addition to the zuul-jobs standard library upstream | 19:21 |
fungi | it would probably also do `pipenv run {something}` which was then parameterized so it could be overloaded to run whatever a specific inheriting job wanted to run with pipenv | 19:22 |
goern | fungi, ack, where do I add the job in https://github.com/goern/thoth-zuul-test-jobs so that I can work on it and migrate/sendPr later on | 19:22 |
goern | zuul.d/_included-jobs.yaml seems to be wrong?! | 19:23 |
goern | just add a zuul.d/base-pipenv.yaml? | 19:23 |
mordred | goern: yah - all the files in the directory get read | 19:24 |
goern | mordred, on, any proposed name to match your schemata? | 19:25 |
mordred | they do get read in alphnumeric order - which is I imagine why that file was written with a _ in it | 19:25 |
mordred | goern: we typically just make a file called "jobs.yaml" | 19:25 |
*** zaro__ has quit IRC | 19:25 | |
mordred | goern: but honestly whatever works for you - when we lift it into zuul-jobs later, we can just copy the yaml section in | 19:25 |
goern | true | 19:25 |
*** zaro__ has joined #zuul | 19:26 | |
pabelanger | okay cool, so https://review.openstack.org/576933/ does work but will break jobs. as show by jobs failing, I'll fork for now until we can come up with plan | 19:26 |
mordred | pabelanger: sweet | 19:26 |
pabelanger | mordred: indeed, depends-on cross project is awesome! | 19:26 |
mnaser | mordred: another crazy frontend idea that i hope one day i get around is to enable logins to the dashboard | 19:28 |
mnaser | i.e. login via gerrit or github | 19:28 |
mnaser | and seeing only your jobs in the dashboard | 19:29 |
* mnaser has wanted something like that for a while | 19:29 | |
mordred | mnaser: there's some discussion somewhere about what authentication looks like for zuul api in general - I don't think that use case has been mentioned | 19:33 |
mnaser | i think it's one to keep in mind | 19:33 |
mnaser | could introduce all sorts of cool ideas like | 19:34 |
mnaser | retriggering jobs as a user from the ui rather than a comment | 19:34 |
fungi | right, i think we've wanted an authenticated rpc for stuff like that far back into zuul v2 time, just never got time to spec it out and grapple with the security concerns it brings | 19:36 |
fungi | certainly retriggering jobs without having to leave hacky comments in a code review system has been one of the primary use cases | 19:36 |
*** gtema has quit IRC | 19:36 | |
openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] web: add tenant-scoped autohold, enqueue https://review.openstack.org/576907 | 20:19 |
*** pcaruana has quit IRC | 20:41 | |
openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] web: add tenant-scoped autohold, enqueue https://review.openstack.org/576907 | 20:49 |
*** acozine1 has quit IRC | 20:53 | |
*** myoung is now known as myoung|off | 21:07 | |
*** yolanda has quit IRC | 21:35 | |
*** yolanda has joined #zuul | 21:40 | |
corvus | tobiash, mordred: as annoying as those messages are, the start/stop messages actually are very useful when debugging issues (i used them the other day). i'd like to keep them in the debug log, but we can make sure they aren't in the regular log. ideally the normal (info-level) log is what one should use to see what's happening in the system anyway. | 22:07 |
*** hasharAway has quit IRC | 22:25 | |
openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] web: add tenant-scoped autohold, enqueue https://review.openstack.org/576907 | 22:25 |
*** rlandy has quit IRC | 23:26 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!