Friday, 2019-10-25

*** rfolco|rover has joined #oooq00:16
*** Goneri has joined #oooq01:10
*** Goneri has quit IRC01:20
*** rfolco|rover has quit IRC01:51
*** ykarel|away has joined #oooq03:57
*** whoami-rajat__ has joined #oooq04:05
*** ykarel|away is now known as ykarel04:06
*** whoami-rajat_ has joined #oooq04:10
*** whoami-rajat__ has quit IRC04:13
*** ykarel is now known as ykarel|afk04:37
*** udesale has joined #oooq04:44
*** ykarel|afk is now known as ykarel05:00
*** marios has joined #oooq05:12
*** marios is now known as marios|ruck05:12
*** hamzy has quit IRC05:19
*** hamzy has joined #oooq05:21
*** surpatil has joined #oooq05:25
ykarelqueens promoting \o/05:45
marios|ruckykarel: nice thanks i just came from email i saw the DNM change :D05:46
ykarel:)05:47
marios|ruckstein too05:47
ykarelyup i saw05:48
ykareli reran rocky phase105:48
ykarelbut still failed, need some patches in packstack05:49
ykarelhttps://review.opendev.org/#/c/690867/05:50
*** jfrancoa has joined #oooq06:01
marios|ruckchandankumar: +Ad the train jobs https://review.opendev.org/#/c/68700406:18
*** tesseract has joined #oooq06:19
*** udesale has quit IRC06:24
*** holser has joined #oooq06:24
*** udesale has joined #oooq06:25
*** udesale has quit IRC06:27
*** jtomasek has joined #oooq06:36
*** holser has quit IRC06:37
*** holser has joined #oooq06:38
*** apetrich has joined #oooq06:50
*** udesale has joined #oooq06:52
*** apetrich has quit IRC06:53
*** apetrich has joined #oooq06:54
*** sshnaidm is now known as sshnaidm|off06:58
*** ccamacho has joined #oooq07:01
*** tosky has joined #oooq07:18
*** saneax has joined #oooq07:20
*** d0ugal has quit IRC07:26
*** d0ugal has joined #oooq07:33
*** kopecmartin|off is now known as kopecmartin07:45
*** akahat has joined #oooq07:45
*** udesale has quit IRC07:49
*** udesale has joined #oooq07:49
*** zbr has joined #oooq07:53
*** recheck has joined #oooq07:53
*** amoralej|off is now known as amoralej08:01
*** bogdando has joined #oooq08:02
*** yolanda has quit IRC08:14
*** yolanda has joined #oooq08:14
*** chem|pto has quit IRC08:31
*** derekh has joined #oooq08:43
*** pierreprinetti has joined #oooq08:49
*** skramaja has quit IRC08:56
*** ykarel is now known as ykarel|lunch09:03
*** whoami-rajat_ has quit IRC09:21
*** ykarel|lunch is now known as ykarel09:30
*** ykarel is now known as ykarel|meeting09:36
*** dtantsur|afk has quit IRC10:16
zbranyone aware?10:23
zbrawake10:23
*** jbadiapa has quit IRC10:25
*** epoojad1 has joined #oooq10:26
*** jbadiapa has joined #oooq10:26
*** akahat has quit IRC10:33
*** ykarel|meeting is now known as ykarel10:54
*** ccamacho has quit IRC10:55
pandabuone anche quelle11:23
pandaoops11:23
*** ccamacho has joined #oooq11:27
zbris anyone aware of tox-direct and its yolo mode? need to test it, sounds like a cool way to test platform support.11:35
*** ykarel is now known as ykarel|afk11:35
zbranyone aware of a tripleo-ci-centos-7-scenario003-standalone failure? got one at https://review.opendev.org/#/c/691271/1 but strange that https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario003-standalone does not include it.12:12
zbrthat's almost perfect, job failed, but is not visible in stats :D12:13
*** epoojad1 has quit IRC12:16
*** ykarel|afk is now known as ykarel12:17
*** rlandy has joined #oooq12:20
rlandychandankumar: hi - you around? I'm thinking not12:21
rlandyzbr: hi ... want to sync re: rpm passing between jobs?12:22
zbrrlandy: bit later I am pressed now with somethign.12:23
zbralso found the issue above, apparently zuul history is borked. posted on infra.12:23
marios|ruckykarel: o/ do you agree comment #4 https://bugs.launchpad.net/tripleo/+bug/184951612:29
openstackLaunchpad bug 1849516 in tripleo "rdo periodic centos-7-ovb-3ctlr_1comp-featureset001-train fails tempest cant download cirros image " [Critical,Fix released]12:29
ykarelmarios|ruck, looking12:29
ykarelmarios|ruck, nope12:30
ykarelthe fix u linked are for packstack12:30
marios|ruckykarel: ah ok what else do we need12:30
ykarelit's needed fix in tripleo as well12:31
ykarelu remmeber my message from yesterday re. os_tempest12:31
ykarel?12:31
marios|ruckykarel: ack ok going now12:31
ykarel<ykarel> earlier we used to use cached image, but now with os_tempest image is being downloaded12:31
ykarel<ykarel> so ^^ should be fixed12:31
ykarel<ykarel> tempest guys would be more aware around the reasoning part12:31
ykarel* ratailor_ (~ratailor@2402:8100:3022:d306:6aed:c029:83d7:fc22) has joined12:31
ykarel* ratailor has quit (Ping timeout: 250 seconds)12:31
ykarel<marios|ruck> ykarel: ack thanks12:31
ykarelmarios|ruck, ^^12:31
marios|ruckykarel: ack yeah i remember but the tempest guys are out still/or travelling maybe today12:32
marios|ruckykarel: and trying to update what i can12:32
marios|ruckykarel: ok i read the reviews too fast going to post the equivalents for us then thanks12:32
ykarelok, so good to update bug and remove fix released12:32
marios|ruckykarel: yes12:32
*** surpatil has quit IRC12:35
ykarelmarios|ruck, fyi i just triggered rocky phase1 https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-rocky-current-tripleo/242/12:35
ykarelshould green now as packstack rocky patch merged12:35
marios|ruckykarel: ack thanks12:40
*** rfolco|rover has joined #oooq12:43
marios|ruckkopecmartin: you around? can you please check comment #6 if you have some time cc chandankumar arxcruz https://bugs.launchpad.net/tripleo/+bug/184951612:44
openstackLaunchpad bug 1849516 in tripleo "rdo periodic centos-7-ovb-3ctlr_1comp-featureset001-train fails tempest cant download cirros image " [Critical,In progress]12:44
kopecmartinmarios|ruck: if i get it correctly cirros servers are the problem, right? can't you store the image somewhere on tripleo servers/mirrors and use that url in the jobs?12:50
toskylike on openstack infra12:51
toskymarios|ruck, kopecmartin: have you tried to change http with https? I've seen a few changes like that12:51
ykarelmarios|ruck, fyi triggered queens as well https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-queens-current-tripleo/300/12:52
ykarelthis should also go green12:52
toskyoh, right, you already commented about using https12:52
kopecmartintosky: we probably can change that in python-tempestconf, but it'll take some time , i was suggesting a faster solution12:52
kopecmartinmeanwhile le'ts do the change in python-tempestconf12:53
ykareltriggering stein too12:53
toskykopecmartin: I'd say there is no easy way around; caching on openstack infra for example works for internal jobs, but iirc not for external consumers12:53
*** soniya29 has joined #oooq12:55
ykarelkopecmartin, tosky, in packstack we are doing, use image from local if exist else download via https12:56
ykarelthough we have seen issue with https also12:56
ykarelbut less than http12:56
ykarelalso in validate-tempest tripleo image is used from local cache12:56
ykarelbut with os_tempest it's downloaded12:56
marios|rucktosky: ack yeah its on the bug comments via ykarel12:58
marios|rucktosky: (I mean https vs http)12:59
marios|ruckkopecmartin: yes it was cached at some point chandankumar and or arxcruz changed it ?12:59
marios|ruckkopecmartin: or perhaps12:59
marios|ruckkopecmartin: with validate tempest we were caching12:59
marios|ruckkopecmartin: and os tempest not12:59
marios|ruckor something... :D12:59
kopecmartinmarios|ruck: it looks like it, that's what ykarel wrote couple of minutes ago too13:00
marios|ruckkopecmartin: ack sorry wasn't looking at irc (its all on the bug too)13:01
rlandymarios|ruck: hi - I am seeing tempest failures on rhos-15 ( not downloading image) - seen anything similar lately: https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/tripleo-ci-rhel-8-standalone-rhos-15/f127875/logs/tempest.html.gz13:05
marios|ruckrlandy: downloading cirros image?13:05
marios|ruckrlandy: if so that's what we were just discussing here https://bugs.launchpad.net/tripleo/+bug/184951613:06
openstackLaunchpad bug 1849516 in tripleo "rdo periodic centos-7-ovb-3ctlr_1comp-featureset001-train fails tempest cant download cirros image " [Critical,In progress]13:06
marios|ruckrlandy: looking at logs13:06
marios|ruckrlandy: no that looks differnt ? AssertionError: False is not true : Public network connectivity check failed13:07
rlandymarios|ruck: thanks - will look into it more from the downstream side13:07
marios|ruckrlandy: not seeing something like that (apologies i missed the 'not' bit about the i thought you said 'downloading image' ) ... i need a screen break13:08
marios|ruckrlandy: all current things are there https://etherpad.openstack.org/p/ruckroversprint1713:08
rlandymarios|ruck: this spring is nearly over - hang in there :)13:08
rlandysprint13:08
marios|ruckrlandy: yeah13:08
*** ykarel is now known as ykarel|afk13:10
*** amoralej is now known as amoralej|lunch13:19
*** soniya29 has quit IRC13:32
*** tesseract has quit IRC13:36
*** jfrancoa has quit IRC13:36
*** Goneri has joined #oooq13:47
*** ykarel|afk is now known as ykarel13:57
*** amoralej|lunch is now known as amoralej13:57
*** rascasoft has quit IRC13:57
rlandybogdando: hi - looking at https://bugs.launchpad.net/tripleo/+bug/181917514:03
openstackLaunchpad bug 1819175 in tripleo "containers-multinode unable to pull images" [Critical,Triaged]14:03
rlandybogdando: we are experimenting with vexxhost and seeing connection failures there with logs like: http://logs.rdoproject.org/66/16566/17/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost-nhw/dc249b3/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz14:04
rlandypossible that the traces are related?14:04
bogdandorlandy: hi. I can see "No lock information provided for layer" so please retry with the recent patch https://review.opendev.org/#/c/690111/14:06
rlandybogdando: thanks - will try that14:06
rlandybogdando: so it looks that the job with the lock trace ran yesterday with openstack-tripleo-common-11.4.0-0.20191024200038.7186d0d.el7.noarch which should include https://review.opendev.org/#/c/690111/14:24
rlandyhow sure are we that the patch fixes the lock issue?14:24
rlandyhttp://logs.rdoproject.org/66/16566/19/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost-nhw/7e19e9b/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz14:24
bogdandorlandy: please ask Alex, I think with locking enabled we should not see that msg logged14:25
*** TrevorV has joined #oooq14:27
*** jaosorior has joined #oooq14:38
*** ccamacho has quit IRC14:39
marios|ruckrfolco|rover: all i can say is after the week we've had that's pretty good going https://i.imgur.com/YIGoVIr.png14:57
*** tosky_ has joined #oooq15:04
*** tosky has quit IRC15:06
marios|ruckykarel: ack ...  rfolco|rover is poking there now (folco re https://bugs.launchpad.net/tripleo/+bug/1849101 ) have handed over i will be out in a while for weekend15:10
openstackLaunchpad bug 1849101 in tripleo "rdo centos7 ovb 3 ctrl 1 comp featureset 001 timeout overcloud deploy" [High,Triaged]15:10
marios|ruckykarel: oops was meant for tripleo15:10
mjturekhas anyone seen the error seen here https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1685/logs/logs/build.log before?15:13
mjturekat 2019-10-25 15:06:3115:14
mjturekseems to happen randomly, never the same container, and doesn't mark then container as failed15:14
ykarel133 container, i think there were 135, good to check similar non ppc one15:17
ykarelyes 135 there https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/ab27d44/logs/build.log.txt.gz15:18
ykarelany idea why 133 on ppc? if no idea, good to see diff and find those 2 to begin with15:18
*** kopecmartin is now known as kopecmartin|off15:19
ykarelmjturek, ^^15:27
mjturekykarel: hmmm will find diff15:29
ykarelmjturek, containers-expected-to-build.log.txt contains expected list15:30
rfolco|rovermjturek, would be good if you add buildah logs15:30
rfolco|roverhttps://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_19c/690389/2/check/tripleo-build-containers-centos-7-buildah/19c92c9/logs/buildah-builds/15:30
ykarelthat would be good start to compare15:30
ykarelmjturek, and also check ur passing one, how many containers it built15:30
mjturekpassing one?15:31
rfolco|roveralso there is a push exception in your log15:31
ykarelmjturek, u said it fails randomly, so ia ssume you have some success jobs too15:31
mjturekykarel: seems to happen on a random container each time, but never seen one where it doesn't happen (at least since we last restored builds)15:31
ykarelmjturek, and yes if buildah_builds are missing as rfolco|rover said, good to collect them15:32
ykarelthere would have more detail15:32
ykarelmjturek, so no success job yet?15:32
mjturekall the containers claim to build15:33
mjturekbut return 115:33
mjturekso no success in that we haven't had a run with a return of 015:33
mjturekany idea where we'd find the buildah logs?15:34
*** tosky_ is now known as tosky15:36
*** marios|ruck is now known as marios|out15:36
ykarelmjturek, in our ci we do mv /tmp/kolla-* {{ workspace }}/logs/buildah-builds/ || true15:36
ykarelmjturek, u find which two containers were missing?15:37
mjturekawesome!15:37
ykarelas compared to ours15:37
mjtureknot yet15:37
mjturekbaha ^ remember the logging patch you were working on? Could you upload that today?15:37
bahaSure, I'll make it a priority15:38
mjturekthanks dude15:38
rfolco|rovermjturek, diff sorted-x86.txt sorted-ppc.txt15:39
rfolco|rover109,110d10815:39
rfolco|rover< trunk.registry.rdoproject.org/tripleomaster/centos-binary-prometheus-haproxy-exporter15:39
rfolco|rover< trunk.registry.rdoproject.org/tripleomaster/centos-binary-prometheus-memcached-exporter15:39
rfolco|roverprometheus and I know why15:40
rfolco|rover:)15:40
mjturek\o/15:40
rfolco|roverit happened to x86 as well15:40
ykarelnice15:40
mjturekwhat was the issue?15:40
rfolco|roverexample15:41
rfolco|roverhttp://logs.rdoproject.org/89/690389/1/openstack-check/tripleo-build-containers-rhel-8-master/c21b06f/logs/buildah-builds/kolla-qdwkkqts/docker/prometheus/prometheus-haproxy-exporter/prometheus-haproxy-exporter-build.log.txt.gz15:41
rfolco|roverintermittent15:41
rfolco|roverthe server was down temporarily15:41
rfolco|rovercould not download the source15:41
rfolco|rovernot saying its the case 100% sure, but sounds like15:42
rfolco|rovermjturek, add buildah-builds logs to your job asap :)15:42
*** jaosorior has quit IRC15:51
*** holser has quit IRC15:58
*** derekh has quit IRC16:07
zbrrfolco|rover: ok to retrigger jobs now?16:17
*** aakarsh has joined #oooq16:27
*** ykarel is now known as ykarel|away16:30
*** marios|out has quit IRC16:42
*** udesale has quit IRC16:48
rfolco|roverzbr, which jobs?16:52
zbri seen unrelated failures, one in tripleo-ci-centos-7-undercloud-containers.16:56
rfolco|roverzbr, link pls17:08
rfolco|roverzbr, if you think its transient, go ahead and recheck, I am not aware of any bugs on this job17:09
zbrrfolco|rover: is https://review.opendev.org/#/c/691271/ but it will not help you much because i updated the patch before getting the reply from zool, you need to check the history.17:09
zbrwe will get results in 1h,jobs already running.17:10
rfolco|roverzbr, tripleo-ci-centos-7-undercloud-containers SUCCESS in 1h 48m 04s17:10
rfolco|roverin ps117:10
rfolco|roverzbr, scen3 failed17:10
zbrthat was first run, in second it failed (msg is missing)17:11
zbrnow is the 3rd run, we will see.17:11
rfolco|roverzbr, undecloud containers is mostly green, lets wait for next run17:14
*** bogdando has quit IRC17:22
*** zbr has quit IRC17:45
*** amoralej is now known as amoralej|off18:09
*** ykarel|away has quit IRC18:38
*** zbr has joined #oooq19:01
*** holser has joined #oooq19:11
*** TrevorV has quit IRC19:50
*** holser has quit IRC20:04
zbrwhoever reads this first, wf this https://review.opendev.org/#/c/691271/20:05
zbrmore 10h to get zuul +1 it, what can i say.20:06
*** holser has joined #oooq20:15
chandankumarrlandy: rfolco|rover \o/20:23
* chandankumar is killing time at dubai airport20:24
rlandywhat are we celebrating?20:24
rlandychandankumar: hello20:24
chandankumarnot celebrating, I am about to reach home in another 10 hours20:24
chandankumarrlandy: Hey20:24
rlandysafe travels20:25
chandankumarrlandy: thanks!20:25
rlandychandankumar: the hack to pass repos to the child job works20:25
rlandyyou can see the repo being added and used20:26
chandankumarrlandy: I am looking at that job itself20:26
rlandybut we will have to add that repo to repo-setup like arx tried to do so it is added in all places repo-setup runs20:26
rlandyI updated the job to try repo-setup20:26
rlandyit failed but I got called back into vexxhost so haven't debugged20:27
rlandypatchset 3 worked20:27
chandankumarrlandy: no problem,20:27
chandankumarrlandy: https://review.opendev.org/#/c/691044/ here we are using artifacts and modifying release files?20:27
rlandychandankumar: ack20:27
rlandyit's hacky20:27
rlandyrepo-setup is the way to go20:28
rlandybut you do see the repo passed20:28
rlandyso we can prove in concept it works20:28
rlandyalso the rpm is updated20:28
rlandyhence the failure20:28
rlandyartifact is the way to go according to tristan20:28
rlandyI agree with that piece20:28
chandankumaryes20:28
rlandyanyways, proof of concept20:29
chandankumarrlandy: is it possible to include that hack as a part of zuul distro job itslef?20:29
chandankumar*itself20:29
rlandychandankumar: ack - we can call the role itself20:30
rlandyone of the questions on the card20:30
chandankumarso that we can include those stuff as a var in the zuul jobs so that we donot not need to carry hack20:30
rlandybut if you see arx's comment, there is a form of it in the role20:30
* rlandy gets20:30
chandankumarrlandy: ok20:31
chandankumarrlandy: I was discussing with jpena and https://github.com/rdo-infra/rdo-jobs/tree/master/playbooks/tripleo-ceph-integration we can convert this to distro zuul jobs and re use it20:32
rlandychandankumar: that is a much better way to go20:32
rlandyhttp://logs.rdoproject.org/44/23444/4/check/distgit-rpm-tripleo-master-testing-centos-7-standalone-featureset016/5fc4fc8/logs/undercloud/etc/yum.repos.d/20:32
*** EmilienM has quit IRC20:32
rlandy^^ there you see the zuul-build.repo20:32
rlandyhttp://logs.rdoproject.org/44/23444/4/check/distgit-rpm-tripleo-master-testing-centos-7-standalone-featureset016/5fc4fc8/logs/undercloud/var/log/yum.log.txt.gz20:33
rlandy^^ search there for ceph-ansible, you will see we have the updated rpm - hence the failure20:34
rlandylooking at arx's comment ...20:34
chandankumarrlandy: awesome20:34
chandankumarrlandy: Oct 24 20:26:36 Installed: ceph-ansible-4.0.0-0.rc1.443.g468f694.el8.noarch20:34
rlandyhttps://review.rdoproject.org/r/#/c/23444/4..5/playbooks/tripleo-ceph-integration/post.yaml20:35
rlandy^^ see comment on patchset 420:35
rlandysupposedly it's added20:35
rlandybut we never saw it work20:35
rlandyso probably needs some adjustment20:35
chandankumarrlandy: https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/tripleo-ceph-integration/run.yaml#L1520:36
chandankumarbut I need to check this part20:36
chandankumarwe are setting buildset_artifacts_url and when createrepo will be called20:36
rlandyright  - but there is no trace of the repo on the child job20:36
chandankumarit needs to include the baseurl baseurl=http://logs.rdoproject.org/44/23444/4/check/tripleo-ceph-integration-master/ae8b9f3/buildset20:36
rlandyhttps://review.rdoproject.org/r/#/c/23386/7/playbooks/run-rpm-build.yaml20:37
rlandy^^ it's there20:37
rlandyin the review20:37
rlandybut no trace of the repo being added20:37
rlandyso we need to debug that20:38
rlandy in theory arx's patch is correct20:38
rlandyand not that different from mine that works20:38
rlandyanyways, probably a minor adjustment to get it to work20:38
chandankumaryes, I will take a look, might be we have get something till monday20:39
rlandychandankumar: np - but we should settle this before building jobs20:39
rlandyso we know what jobs to build :)20:39
*** holser has quit IRC20:39
chandankumarok, sure!20:39
rlandypatchset 5  was my latest attempt - to use repo-setup20:41
rlandybut clearly needs some fix there20:42
rlandywhich will not need the change to release file20:42
chandankumarrlandy: I am checking whether this patch is applied or not https://github.com/ceph/ceph-ansible/pull/4657 ?20:43
*** aakarsh has quit IRC20:44
chandankumargit archive --format=tar.gz --prefix=ceph-ansible-4.0.0/ HEAD > ceph-ansible-4.0.0-5db9df4.tar.gz also needs to be fixed, otherwise master pr will be not pulled20:45
rlandythe tag needs to be fixed in prep containers20:46
chandankumarrlandy: you mean during tripleo container image prepare?20:48
rlandyat least we have passing vexxhost now - one less thing to worry about20:48
rlandychandankumar: ack20:48
*** dtrainor has quit IRC20:48
* rlandy gets logs20:48
rlandylog server is slow ....20:50
*** dtrainor has joined #oooq20:51
rlandyhttp://logs.rdoproject.org/44/23444/4/check/distgit-rpm-tripleo-master-testing-centos-7-standalone-featureset016/5fc4fc8/logs/undercloud/home/zuul/standalone_container_prep.log.txt.gz20:51
rlandy^^ see ceph tag20:51
rlandyceph_tag: v4.0.4-stable-4.0-nautilus-centos-7-x86_6420:52
chandankumaryes20:53
rlandychandankumar: that's fixable20:54
rlandywe've done that for downstream20:54
chandankumarrlandy: https://github.com/ceph/ceph-ansible/pull/4657 based on this patch, it needs to be modified here https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/tripleo-ceph-integration/run.yaml#L720:54
chandankumarit needs some additional vars20:55
chandankumarI am checking it20:55
rlandythat var needs to go toci though20:59
chandankumarrlandy: you mean make rpm var or container prep var?20:59
rlandysorry - container prep var20:59
chandankumarrlandy: yes,21:00
chandankumarrlandy: See ya, happy weekend :-)21:03
rlandychandankumar: k - travel safely - see you on the other side21:03
*** rlandy has quit IRC21:25
*** EmilienM has joined #oooq22:11
*** jtomasek has quit IRC22:58
*** jtomasek has joined #oooq22:59
*** jtomasek has quit IRC22:59
*** rfolco|rover has quit IRC23:04
*** recheck has quit IRC23:10
*** recheck has joined #oooq23:12
*** tosky has quit IRC23:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!