*** vvaldez has joined #oooq | 00:48 | |
hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 00:49 |
---|---|---|
*** tcw has quit IRC | 02:33 | |
*** tcw has joined #oooq | 02:37 | |
hubbot | FAILING CHECK JOBS on master: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 02:50 |
*** skramaja has joined #oooq | 02:59 | |
*** vvaldez has quit IRC | 03:07 | |
*** ykarel|away has joined #oooq | 03:35 | |
*** udesale has joined #oooq | 03:45 | |
*** ykarel|away is now known as ykarel | 04:02 | |
*** ratailor has joined #oooq | 04:28 | |
*** jaganathan has joined #oooq | 04:28 | |
*** pgadiya has joined #oooq | 04:46 | |
*** pgadiya has quit IRC | 04:46 | |
hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, master: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445 | 04:50 |
*** ratailor has quit IRC | 04:54 | |
*** ratailor_ has joined #oooq | 04:54 | |
*** quiquell|off is now known as quiquell|rover | 05:37 | |
*** bogdando has joined #oooq | 05:48 | |
*** brault has quit IRC | 06:15 | |
*** brault has joined #oooq | 06:23 | |
quiquell|rover | ykarel: Do you know what do we use opstools for ? | 06:29 |
*** sanjayu_ has joined #oooq | 06:31 | |
*** ykarel_ has joined #oooq | 06:34 | |
*** ykarel has quit IRC | 06:36 | |
quiquell|rover | sshnaidm: Do you know what do we use opstools for ? | 06:38 |
ykarel_ | quiquell|rover, no | 06:38 |
*** ykarel_ is now known as ykarel | 06:38 | |
*** holser_ has joined #oooq | 06:48 | |
hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, master: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445 | 06:50 |
*** quiquell|rover is now known as quique|rover|afk | 06:57 | |
*** Goneri has joined #oooq | 07:05 | |
*** ccamacho has joined #oooq | 07:06 | |
*** tesseract has joined #oooq | 07:11 | |
*** tosky has joined #oooq | 07:18 | |
*** amoralej|off is now known as amoralej | 07:39 | |
*** quique|rover|afk is now known as quiquell|rover | 07:42 | |
*** zoli|gone is now known as zoli | 07:53 | |
*** kopecmartin has joined #oooq | 07:55 | |
*** d0ugal_ has quit IRC | 08:03 | |
*** d0ugal has joined #oooq | 08:04 | |
sshnaidm | quiquell|rover, we install collectd from there, maybe something else | 08:05 |
*** sshnaidm is now known as sshnaidm|off | 08:05 | |
*** ykarel is now known as ykarel|lunch | 08:23 | |
*** skramaja has quit IRC | 08:33 | |
*** dtantsur|afk is now known as dtantsur | 08:39 | |
*** anande has joined #oooq | 08:42 | |
hubbot | FAILING CHECK JOBS on master: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 08:50 |
quiquell|rover | chandankumar, tosky: Do you know about this issue ? http://logs.openstack.org/32/576632/2/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/e50b49b/logs/undercloud/home/zuul/tempest/tempest.html.gz | 08:50 |
quiquell|rover | It's a rediecting loop | 08:50 |
*** ratailor__ has joined #oooq | 08:53 | |
*** ratailor_ has quit IRC | 08:57 | |
*** zoli is now known as zoli|lunch | 09:03 | |
*** ykarel|lunch is now known as ykarel | 09:07 | |
*** ykarel has quit IRC | 09:14 | |
*** anande has quit IRC | 09:24 | |
*** ykarel has joined #oooq | 09:49 | |
tosky | quiquell|rover: uhm, no; can you pinpoint when it started? Maybe it's easy to find the commit that changed the behavior | 09:54 |
quiquell|rover | tosky: Sure | 09:54 |
*** d0ugal has quit IRC | 09:55 | |
*** d0ugal has joined #oooq | 09:57 | |
quiquell|rover | tosky: I think it was transitory, now I see it passing at zuul's gate | 09:57 |
tosky | oh, oki | 09:57 |
tosky | there were no recent changes in the telemetry_tempest_plugin | 09:57 |
quiquell|rover | tosky: The redirection loop could be some reouting problem | 09:57 |
quiquell|rover | tosky: That got fixed at infra or similar | 09:57 |
tosky | right | 09:57 |
quiquell|rover | tosky: thanks btw | 09:58 |
tosky | np (when there is no effort is easy :) | 10:02 |
*** jaganathan has quit IRC | 10:41 | |
*** zoli|lunch is now known as zoli | 10:49 | |
hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 10:50 |
*** jaganathan has joined #oooq | 10:51 | |
*** honza has quit IRC | 10:55 | |
*** jaganathan has quit IRC | 10:58 | |
*** honza has joined #oooq | 11:00 | |
*** honza is now known as Guest71763 | 11:00 | |
*** quiquell|rover has quit IRC | 11:17 | |
*** jaganathan has joined #oooq | 11:30 | |
*** jaganathan has quit IRC | 11:30 | |
*** jaganathan has joined #oooq | 11:31 | |
*** udesale_ has joined #oooq | 11:32 | |
*** udesale_ has quit IRC | 11:33 | |
*** udesale_ has joined #oooq | 11:34 | |
*** udesale has quit IRC | 11:34 | |
*** quiquell has joined #oooq | 11:42 | |
*** udesale_ has quit IRC | 11:42 | |
*** quiquell is now known as quiquell|rover | 11:42 | |
*** jaganathan has quit IRC | 11:44 | |
*** jaganathan has joined #oooq | 11:45 | |
*** Goneri has quit IRC | 11:50 | |
*** panda|off is now known as panda | 11:56 | |
*** trown|outtypewww is now known as trown | 12:00 | |
panda | quiquell|rover: are failures in undercloud install known ? http://logs.openstack.org/79/576879/17/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/6acdf7c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-06-22_10_37_36 | 12:05 |
quiquell|rover | panda: Yep, we have identified the issue also | 12:06 |
panda | quiquell|rover: ok, thanks | 12:06 |
quiquell|rover | https://bugs.launchpad.net/tripleo/+bug/1778201 | 12:06 |
openstack | Launchpad bug 1778201 in tripleo "os-refresh-config undercloud install Error: Evaluation Error: Error while evaluating a Function Call, pick(): must receive at least one non empty" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 12:06 |
*** amoralej is now known as amoralej|lunch | 12:07 | |
panda | trown: line 91 of the etherpad, the playbook is trying to resolve the hosts: all, peers, children, hosts | 12:20 |
panda | zuuul inventory may not work as it is. | 12:21 |
*** moguimar has quit IRC | 12:21 | |
trown | panda: not sure I understand... I have used that inventory successfully | 12:22 |
trown | panda: trying to figure out your missing overcloud group thing... in my libvirt env that is there | 12:22 |
*** moguimar has joined #oooq | 12:23 | |
panda | trown: mmh, what version of ansible are you using ? | 12:23 |
trown | ansible-2.4.0.0-5.el7.noarch | 12:24 |
panda | more or less the same ... whay does it not understand the inventory then .. | 12:25 |
trown | panda: what error are you hitting specifically... maybe some yaml formatting thing? | 12:25 |
panda | fatal: [all:]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname all:: Name or service not known\r\n", "unreachable": true} | 12:25 |
panda | fatal: [peers:]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname peers:: Name or service not known\r\n", "unreachable": true} | 12:25 |
trown | though I have been copy pasting from etherpad... | 12:25 |
panda | fatal: [children:]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname children:: Name or service not known\r\n", "unreachable": true} | 12:26 |
panda | fatal: [hosts:]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname hosts:: Name or service not known\r\n", "unreachable": true} | 12:26 |
panda | during facts gathering | 12:26 |
*** jaganathan has quit IRC | 12:27 | |
trown | panda: hmm... mine is definitely only trying to gather facts from primary and secondary | 12:28 |
trown | panda: could you paste your inventory? | 12:28 |
*** rlandy has joined #oooq | 12:28 | |
panda | trown: http://paste.openstack.org/show/724108/ | 12:29 |
quiquell|rover | rlandy: Hello | 12:30 |
rlandy | quiquell|rover: hello | 12:31 |
quiquell|rover | rlandy: Did you say something about OVB create ? | 12:31 |
rlandy | quiquell|rover: I have been working on an doff on getting the last pieces of the rhos-13 gate going ... | 12:31 |
*** jaganathan has joined #oooq | 12:32 | |
quiquell|rover | rlandy: I cleaned up central ci there were resources problems | 12:32 |
rlandy | I noticed last night that the stack create was in continuous create and never completed | 12:32 |
trown | panda: weird... if I diff your file and mine every line is different... but I cant tell what is different... | 12:32 |
rlandy | I worked in the stage tenant | 12:32 |
rlandy | so I wanted to know if you saw that in the regular tenant | 12:32 |
rlandy | I see you have some passing jobs today so maybe it's fine now | 12:33 |
rlandy | I'll recheck | 12:33 |
panda | trown: debugging ansible, my version is missing the yaml plugin | 12:33 |
quiquell|rover | rlandy: Ok | 12:33 |
panda | trown: WARNING]: * Failed to parse /tmp/tmp.71Du0ufqEt/.quickstart/zuul-hosts.yaml with ini plugin: /tmp/tmp.71Du0ufqEt/.quickstart/zuul-hosts.yaml:5: Expected key=value host variable assignment, got: null | 12:33 |
quiquell|rover | rlandy: Let me know if it works now for you | 12:33 |
rlandy | I was worried the upgrade may have impacted it | 12:33 |
rlandy | but it look slike your stuff is rolling along | 12:34 |
panda | trown: so the inventoryis probably ok, I'll try to understand what's happening with my system | 12:34 |
trown | panda: oh, that would do it... parsing that file as .ini would not work :P | 12:34 |
trown | not sure how yaml plugin is installed... mine "just worked" | 12:34 |
quiquell|rover | rlandy: The periodic-master-rdo_trunk-ovb-minimal-pacemaker-multiple-nics is super weird | 12:34 |
rlandy | quiquell|rover: lol - that is one I did not set up :) | 12:35 |
rlandy | you still having trouble there> | 12:35 |
quiquell|rover | rlandy: I mean the issue, the job is alright | 12:35 |
quiquell|rover | rlandy: I have two runs now, one is taking the image correctly the next not | 12:35 |
quiquell|rover | rlandy: I am starting to thing that other jobs running in the same slave are affecting it | 12:35 |
rlandy | quiquell|rover: check which save each job is running on | 12:35 |
rlandy | there are two slaves | 12:36 |
rlandy | possible diff | 12:36 |
quiquell|rover | rlandy: Two consecutives in the same slave | 12:36 |
trown | panda: btw overcloud group is being added to the inventory very early in my reproducer env... before/during undercloud-setup ... trying to figure out why that did not happen in CI | 12:36 |
rlandy | quiquell|rover: we need to make sure the workspace is deleted properly between the runs | 12:36 |
rlandy | and that there is no memory from one job to the other | 12:36 |
*** kopecmartin has quit IRC | 12:39 | |
*** jaganathan has quit IRC | 12:41 | |
*** jaganathan has joined #oooq | 12:41 | |
panda | trown: ok, got some other hiccups with yaml and neede to recopy and paste, but works now | 12:41 |
trown | panda: in your test job something is making the subnode fail to get added to the inventory at all http://logs.openstack.org/79/576879/16/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/53638a7/logs/quickstart_files/hosts.txt.gz ... grep "Add subnode to ansible inventory" http://logs.openstack.org/79/576879/16/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/53638a7/logs/quickstart_install.log ... there | 12:46 |
trown | is no "changed" entry for it | 12:46 |
trown | panda: ah found it https://github.com/openstack/tripleo-quickstart/blob/master/roles/tripleo-inventory/tasks/inventory.yml#L45 | 12:47 |
panda | trown: running toci script, I'll check for that error | 12:47 |
panda | trown: also, the upstream patch fails because there is not overcloud group, added it on the inventory in nodeset, but there's an error in undercloud install and I can't make progress there | 12:48 |
trown | panda: ya I think the lack of overcloud group is not solved by your patch... that is what I was looking at | 12:49 |
trown | panda: because the zuul inventory is not used for our ansible runs from quickstart | 12:49 |
hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 12:50 |
trown | panda: issue is that the subnode does not get added to the inventory via our quickstart inventory role ... something is not working with populating the /etc/nodepool/sub_nodes_private file in your legacy playbook | 12:50 |
panda | mmmhh | 12:51 |
panda | trown: the playbook was copied directly from the upstream legacy/pre.yaml playbook | 12:51 |
trown | panda: commented in the review... I think it might be the groups['subnodes'] in that task | 12:53 |
*** ratailor__ has quit IRC | 12:53 | |
trown | panda: ya we have no subnodes group in our custom nodeset | 12:54 |
*** ratailor__ has joined #oooq | 12:54 | |
trown | panda: so that task that sets /etc/nodepool stuff doesnt run | 12:54 |
panda | trown: ok, so I should add subnodes: hosts: -secondary ? | 12:54 |
panda | trown: or both | 12:55 |
trown | panda: looking for the the legacy nodeset to see | 12:55 |
trown | panda: just secondary https://github.com/openstack-infra/openstack-zuul-jobs/blob/master/zuul.d/nodesets.yaml#L184-L186 | 12:56 |
panda | ok, replacing also zuul-cloner with the cp commands directly | 12:57 |
rlandy | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-13-ci-rhos-ovb-minimal-pacemaker-public-bond/ is half the trouble | 12:59 |
rlandy | quiquell|rover: ^^ disabling | 12:59 |
rlandy | taking resources | 12:59 |
quiquell|rover | rlandy: Ahh ok | 13:00 |
quiquell|rover | rlandy: Thanks | 13:00 |
*** skramaja has joined #oooq | 13:12 | |
panda | trown: undercloud install failed in libvirt reproducer as expected ... | 13:13 |
panda | trown: the rest worked, with some adjustments to the system (addded at the top of the etherpad) | 13:15 |
trown | panda: ya my undercloud install failed too | 13:21 |
*** myoung|off is now known as myoung|ruck | 13:22 | |
*** amoralej|lunch is now known as amoralej | 13:22 | |
trown | panda: how did you enable the ansible yaml plugin | 13:23 |
myoung|ruck | o/ good morning | 13:24 |
panda | trown: in /etc/ansible/ansible.cfg, section [inventory] uncomment the plugins | 13:24 |
trown | panda: hmm in my ansible config it has yaml in the default inventory plugins actually... but good to add that to the etherpad as possible issue and resolution | 13:25 |
panda | trown: fresh centos install | 13:25 |
panda | trown: fresh ansible install too, started more or less from scratch | 13:25 |
myoung|ruck | quiquell|rover: not sure (yet) about context re: opstools, but it's https://github.com/centos-opstools a set of tools to configure monitoring and such. THere are some tripleo specific integrations as well to set up log collection and forwarding, sensu / monitoring, healthchecks, and more recently afaik skydive project was added as well. I haven't played with it recently but I always thought http://opstools-ansible.readthedocs.io/en/ | 13:27 |
myoung|ruck | latest/tripleo_integration.html looked like a good target for future CI sprints vs. how we do log collection now (e.g. set up a node for the deployment and fwd logs to a micro elk stack) | 13:27 |
*** vvaldez has joined #oooq | 13:36 | |
myoung|ruck | rasca: ^^ maybe we could prototype this on your BMU job, would solve nicely the problem you were seeing yesterday w.r.t. how we collect logs (instantly searchable vs. "have to know what you're looking for, or download everything slowly" | 13:36 |
quiquell|rover | We need some +2 to unblock CI https://review.rdoproject.org/r/#/c/14432/ | 13:38 |
quiquell|rover | It's pinning puppet stdlib to good version | 13:39 |
myoung|ruck | quiquell|rover: note 17 mins ago https://github.com/puppetlabs/puppetlabs-stdlib/pull/890 they might revert as well | 13:39 |
*** ykarel has quit IRC | 13:40 | |
*** ykarel has joined #oooq | 13:40 | |
quiquell|rover | myoung|ruck: Cool, we can unpin after revert is merged | 13:41 |
quiquell|rover | myoung|ruck: Now CI is stuck there | 13:42 |
*** vvaldez has quit IRC | 13:43 | |
*** tcw has quit IRC | 13:45 | |
*** vvaldez has joined #oooq | 13:51 | |
*** bogdando has quit IRC | 13:51 | |
*** agopi has joined #oooq | 13:54 | |
myoung|ruck | quiquell|rover: o/ back and avail if you are | 13:54 |
*** tcw has joined #oooq | 13:55 | |
quiquell|rover | ok calling your bj | 13:56 |
*** vinaykns has joined #oooq | 13:56 | |
myoung|ruck | %gatestatus | 13:57 |
hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 13:57 |
quiquell|rover | hubbot: Is lying | 13:57 |
hubbot | quiquell|rover: Error: "Is" is not a valid command. | 13:57 |
*** moguimar has quit IRC | 14:03 | |
*** moguimar has joined #oooq | 14:04 | |
*** ratailor__ has quit IRC | 14:11 | |
*** ykarel is now known as ykarel|away | 14:14 | |
*** ykarel|away has quit IRC | 14:16 | |
*** udesale has joined #oooq | 14:21 | |
*** skramaja has quit IRC | 14:34 | |
*** jaganathan has quit IRC | 14:39 | |
hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 14:50 |
*** vvaldez has quit IRC | 14:52 | |
*** vvaldez has joined #oooq | 14:52 | |
*** moguimar has quit IRC | 14:53 | |
*** vvaldez has quit IRC | 14:56 | |
*** quiquell|rover is now known as quiquell|off | 15:01 | |
*** sanjayu_ has quit IRC | 15:06 | |
*** sanjay__u has joined #oooq | 15:11 | |
*** ykarel|away has joined #oooq | 15:35 | |
*** panda is now known as panda|off | 15:59 | |
*** tesseract has quit IRC | 16:02 | |
rlandy | rasca: helo - ha question for you | 16:14 |
rlandy | hello | 16:14 |
*** holser_ has quit IRC | 16:17 | |
*** ykarel|away has quit IRC | 16:25 | |
*** zoli is now known as zoli|gone | 16:29 | |
*** dtantsur is now known as dtantsur|afk | 16:37 | |
hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224 | 16:50 |
*** trown is now known as trown|lunch | 17:03 | |
*** amoralej is now known as amoralej|off | 17:33 | |
*** agopi has quit IRC | 17:48 | |
*** agopi has joined #oooq | 17:49 | |
*** trown|lunch is now known as trown | 18:04 | |
hubbot | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-scenario003-multinode-oooq-container, (1 more message) | 18:50 |
*** yolanda__ has joined #oooq | 19:12 | |
*** yolanda_ has quit IRC | 19:16 | |
*** yolanda__ has quit IRC | 19:18 | |
*** udesale has quit IRC | 19:35 | |
*** atoth has quit IRC | 19:42 | |
*** udesale has joined #oooq | 19:49 | |
*** udesale has quit IRC | 19:58 | |
hubbot | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-scenario003-multinode-oooq-container, (1 more message) | 20:50 |
*** rfolco_ has quit IRC | 21:00 | |
*** trown is now known as trown|outtypewww | 21:07 | |
*** holser_ has joined #oooq | 21:10 | |
*** yolanda has joined #oooq | 21:15 | |
*** holser_ has quit IRC | 21:37 | |
*** hamzy_ has joined #oooq | 22:06 | |
*** hamzy has quit IRC | 22:08 | |
hubbot | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-containers-multinode, tripleo-ci-centos-7-scenario003-multinode-oooq-container, (1 more message) | 22:50 |
*** myoung has joined #oooq | 22:53 | |
*** pliu_ has joined #oooq | 22:54 | |
*** rnoriega_ has joined #oooq | 22:54 | |
*** jjoyce has joined #oooq | 22:54 | |
*** _atoth has joined #oooq | 22:54 | |
*** rasca- has joined #oooq | 22:54 | |
*** lhinds|out has joined #oooq | 22:55 | |
*** lhinds has quit IRC | 22:56 | |
*** weshay_PTO has quit IRC | 22:57 | |
*** weshay has joined #oooq | 22:57 | |
*** pliu has quit IRC | 22:57 | |
*** rasca has quit IRC | 22:57 | |
*** faceman has quit IRC | 22:57 | |
*** fuzzball81 has quit IRC | 22:57 | |
*** rnoriega has quit IRC | 22:58 | |
*** jschluet has joined #oooq | 22:58 | |
*** jschlueter has quit IRC | 22:58 | |
*** myoung|ruck has quit IRC | 22:58 | |
*** myoung is now known as myoung|off | 23:03 | |
*** rlandy has quit IRC | 23:05 | |
*** vinaykns has quit IRC | 23:12 | |
*** vvaldez has joined #oooq | 23:27 | |
*** tosky has quit IRC | 23:41 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!