| *** jbadiapa has quit IRC | 00:07 | |
| *** rf0lc0 has quit IRC | 00:10 | |
| *** ysandeep has joined #oooq | 01:06 | |
| *** rlandy|bbl has quit IRC | 01:49 | |
| *** ykarel|away has joined #oooq | 03:02 | |
| *** rascasoft has quit IRC | 03:06 | |
| *** udesale has joined #oooq | 04:13 | |
| *** bhagyashris has joined #oooq | 04:20 | |
| *** ykarel|away has quit IRC | 04:24 | |
| *** ykarel|away has joined #oooq | 04:41 | |
| *** dsneddon has quit IRC | 05:19 | |
| *** surpatil has joined #oooq | 05:22 | |
| *** skramaja has joined #oooq | 05:27 | |
| *** dsneddon has joined #oooq | 05:29 | |
| *** dsneddon has quit IRC | 05:42 | |
| *** dsneddon has joined #oooq | 05:43 | |
| *** dsneddon has quit IRC | 05:49 | |
| *** soniya29 has joined #oooq | 06:02 | |
| *** ykarel|away is now known as ykarel | 06:03 | |
| *** jtomasek has joined #oooq | 06:09 | |
| *** raukadah is now known as chkumar|ruck | 06:10 | |
| *** marios|rover has joined #oooq | 06:39 | |
| *** epoojad1 has joined #oooq | 06:39 | |
| *** akahat has joined #oooq | 06:44 | |
| *** jfrancoa has joined #oooq | 06:52 | |
| chkumar|ruck | marios|rover: Good morning | 07:07 |
|---|---|---|
| chkumar|ruck | marios|rover: Regarding another scenario test failure which started from dec 10, may be related to this https://review.opendev.org/#/c/697106/ | 07:07 |
| marios|rover | chkumar|ruck: o/ morning that one you mean https://bugs.launchpad.net/tripleo/+bug/1855985 | 07:08 |
| openstack | Launchpad bug 1855985 in tripleo "tempest network test_list_agent test failed with mismatch error in train fs020" [Critical,Confirmed] | 07:08 |
| chkumar|ruck | marios|rover: there are two tests failure, one is filed | 07:09 |
| chkumar|ruck | another is related to different reason | 07:09 |
| *** tosky has joined #oooq | 07:09 | |
| marios|rover | chkumar|ruck: k, fs20 failed there https://review.rdoproject.org/r/#/c/24062/ | 07:09 |
| chkumar|ruck | marios|rover: I am re running https://review.rdoproject.org/r/#/c/24026/ here if again pops up going to file a bug | 07:11 |
| chkumar|ruck | marios|rover: where can I check qbr related logs? | 07:11 |
| marios|rover | chkumar|ruck: not sure | 07:13 |
| *** dsneddon has joined #oooq | 07:15 | |
| *** ykarel is now known as ykarel|lunch | 07:30 | |
| marios|rover | chkumar|ruck: fs1 rhel still a problem https://bugs.launchpad.net/tripleo/+bug/1853652/comments/19 | 07:39 |
| openstack | Launchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Triaged] | 07:39 |
| *** soniya29 has quit IRC | 07:54 | |
| *** bhagyashris has quit IRC | 07:54 | |
| *** sanjayu__ has joined #oooq | 07:57 | |
| *** tesseract has joined #oooq | 07:57 | |
| *** d0ugal has quit IRC | 07:57 | |
| *** d0ugal has joined #oooq | 07:59 | |
| *** bhagyashris has joined #oooq | 08:06 | |
| *** amoralej|off is now known as amoralej | 08:12 | |
| *** ccamacho has joined #oooq | 08:13 | |
| *** jbadiapa has joined #oooq | 08:20 | |
| *** bhagyashris has quit IRC | 08:30 | |
| *** ykarel|lunch is now known as ykarel | 08:31 | |
| jfrancoa | marios|rover: https://review.opendev.org/#/c/698231/ undercloud-upgrade job fixed | 08:36 |
| jfrancoa | marios|rover: I'll try to correct the nits in the three reviews and let's try to merge them | 08:37 |
| marios|rover | thanks jfrancoa | 08:37 |
| marios|rover | jfrancoa: yeah ccamacho ping me a little earlier thanks very much for diggin there so quickly | 08:37 |
| *** kopecmartin|off is now known as kopecmartin | 08:44 | |
| chkumar|ruck | marios|rover: I have not yet looked at master pipeline | 08:44 |
| marios|rover | chkumar|ruck: ack /me going to go afk in a bit will check them after | 08:44 |
| marios|rover | also \o/ chkumar|ruck | 08:44 |
| marios|rover | chkumar|ruck: are you ready for honolulu | 08:45 |
| marios|rover | (its where all the ruck|rovers go when they retire) | 08:45 |
| chkumar|ruck | marios|rover: sure | 08:45 |
| ccamacho | jfrancoa marios|rover also https://review.opendev.org/#/c/698223/2 is not needed to fix the gates | 08:47 |
| ccamacho | we just need the other 2 jfrancoa check it here https://review.opendev.org/#/c/698220/ | 08:47 |
| ccamacho | I dont think it hurts but there are a lot of duplications around that we might be able to purge | 08:50 |
| *** ccamacho is now known as ccamacho|pto | 08:50 | |
| marios|rover | ccamacho|pto: ack thanks | 08:50 |
| marios|rover | enjoy pto ccamacho|pto | 08:50 |
| ccamacho|pto | :) | 08:51 |
| ccamacho|pto | yeahp | 08:51 |
| ccamacho|pto | Ill be around in any case | 08:51 |
| *** jpena|off is now known as jpena | 08:53 | |
| * marios|rover errand biab | 08:55 | |
| *** marios|rover has quit IRC | 08:55 | |
| *** bhagyashris has joined #oooq | 09:01 | |
| jfrancoa | ccamacho|pto: I think we do need that variable | 09:02 |
| *** soniya29 has joined #oooq | 09:02 | |
| jfrancoa | ccamacho|pto: otherwise how would you tell the undercloud to deploy non-containerized? and also, keep in mind we still have a queens upgrades job which upgrades from non containerized to non containerized | 09:04 |
| ccamacho|pto | jfrancoa you say that in the featureset definition, right>? | 09:27 |
| ccamacho|pto | jfrancoa but yeah there are a lot of gaps depending on the fs | 09:28 |
| ccamacho|pto | but, we should have them all defined in one place | 09:28 |
| ccamacho|pto | to make it easier to debug | 09:28 |
| ccamacho|pto | but I vote for a follow up patch | 09:28 |
| *** udesale has quit IRC | 09:29 | |
| *** udesale has joined #oooq | 09:32 | |
| zbr | https://www.reddit.com/r/ansible/comments/e94yqn/experience_from_building_my_first_ansible/ | 09:37 |
| *** ccamacho|pto has quit IRC | 09:38 | |
| *** ccamacho has joined #oooq | 09:38 | |
| *** rascasoft has joined #oooq | 09:42 | |
| *** derekh has joined #oooq | 09:43 | |
| *** apetrich has joined #oooq | 09:50 | |
| *** ccamacho is now known as ccamacho|pto | 09:54 | |
| *** marios|rover has joined #oooq | 09:55 | |
| *** jbadiapa has quit IRC | 09:59 | |
| *** soniya29 has quit IRC | 10:00 | |
| *** dtantsur|afk is now known as dtantsur | 10:06 | |
| *** ysandeep has quit IRC | 10:10 | |
| marios|rover | chkumar|ruck: just fs20 centos missing afaics posting now | 10:14 |
| marios|rover | https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master | 10:14 |
| chkumar|ruck | marios|rover: wait | 10:14 |
| marios|rover | chkumar|ruck: sup | 10:14 |
| chkumar|ruck | marios|rover: https://review.rdoproject.org/r/#/c/24051/ | 10:15 |
| marios|rover | and fs1 | 10:15 |
| chkumar|ruck | added here | 10:15 |
| chkumar|ruck | RHEL8 fs01 & centos7 fs01 and fs020 and fs020 train | 10:15 |
| marios|rover | cool | 10:15 |
| marios|rover | yeah rhel8 still blocked afaics | 10:15 |
| *** soniya29 has joined #oooq | 10:15 | |
| chkumar|ruck | it got timeout there | 10:15 |
| marios|rover | * still happening https://bugs.launchpad.net/tripleo/+bug/1853652/comments/1 | 10:15 |
| openstack | Launchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Triaged] | 10:15 |
| marios|rover | chkumar|ruck: err i removed the 9 at the end should be comment 10 | 10:15 |
| marios|rover | 19! | 10:15 |
| marios|rover | :D | 10:15 |
| chkumar|ruck | marios|rover: https://review.opendev.org/#/c/698435/ | 10:16 |
| marios|rover | chkumar|ruck: different not tempest related | 10:16 |
| marios|rover | chkumar|ruck: its the timeout one | 10:16 |
| chkumar|ruck | marios|rover: if we remove fs020 from train then we have a promotion | 10:16 |
| chkumar|ruck | marios|rover: ok, let it run then | 10:17 |
| marios|rover | chkumar|ruck: fs20 has been a pita this sprint | 10:17 |
| chkumar|ruck | yes, blame ovn | 10:17 |
| chkumar|ruck | and other patches | 10:17 |
| marios|rover | chkumar|ruck: well train promoted monday so you're fine for prog call today anyway | 10:18 |
| marios|rover | chkumar|ruck: nice wednesday weekend currently running all green waiting on fs20 queens/rocky | 10:21 |
| marios|rover | chkumar|ruck: so if we are lucky with 12:15 < chkumar|ruck> marios|rover: https://review.rdoproject.org/r/#/c/24051/ | 10:22 |
| chkumar|ruck | marios|rover: yes | 10:22 |
| marios|rover | chkumar|ruck: we can hand over perfectly | 10:22 |
| chkumar|ruck | yes yes | 10:22 |
| marios|rover | before going to honolulu | 10:22 |
| marios|rover | chkumar|ruck: hmm the pipeline tweak merged | 10:28 |
| marios|rover | chkumar|ruck: https://review.rdoproject.org/r/#/c/24061/2/zuul.d/upstream.yaml so openstack-periodic-master has been running for 2 hours now http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/18d5ff4/ | 10:29 |
| marios|rover | chkumar|ruck: (so your https://review.rdoproject.org/r/#/c/24051/ is running at same time, doesn't matter just fyi) | 10:29 |
| marios|rover | chkumar|ruck: we need same for train just posted https://review.rdoproject.org/r/24073 | 10:39 |
| marios|rover | chkumar|ruck: like yesterday/day before train only ran 2x https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-train-promote-consistent-to-tripleo-ci-testing | 10:41 |
| marios|rover | chkumar|ruck: we are cheating ourselves out of 1 chance ;) | 10:41 |
| *** ykarel is now known as ykarel|afk | 10:51 | |
| chkumar|ruck | marios|rover: hehe | 10:52 |
| chkumar|ruck | marios|rover: http://logs.rdoproject.org/26/24026/3/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train/478617c/logs/stestr_results.html from last 2 runs, https://bugs.launchpad.net/tripleo/+bug/1855985 started passing | 10:52 |
| openstack | Launchpad bug 1855985 in tripleo "tempest network test_list_agent test failed with mismatch error in train fs020" [Critical,Confirmed] | 10:52 |
| chkumar|ruck | not sure what is going on there | 10:52 |
| chkumar|ruck | closing the bug currently | 10:52 |
| *** bhagyashris has quit IRC | 10:53 | |
| marios|rover | chkumar|ruck: ack cool. well fwiw it happened many times this sprint. all branches. fs20 some tempest issue, rerun passes | 10:54 |
| *** udesale has quit IRC | 11:04 | |
| *** bhagyashris has joined #oooq | 11:07 | |
| *** holser has joined #oooq | 11:08 | |
| marios|rover | chkumar|ruck: seen that yet? 2019-12-10 19:21:14 | Error: unable to exec into nova_compute: no container with name or ID nova_compute found: no such container | 11:13 |
| marios|rover | chkumar|ruck: rhel8 fs1 train | 11:14 |
| marios|rover | chkumar|ruck: http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train/6c35249/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 11:14 |
| chkumar|ruck | marios|rover: nope | 11:15 |
| marios|rover | chkumar|ruck: do you agree https://bugs.launchpad.net/tripleo/+bug/1853978/comments/19 | 11:16 |
| openstack | Launchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged] | 11:16 |
| marios|rover | chkumar|ruck: but honestly has that job EVER been green ?! https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train check that s***t out | 11:16 |
| chkumar|ruck | marios|rover: nope, that was failing because of different reasons | 11:17 |
| marios|rover | chkumar|ruck: do we have fs1 train running in test already? | 11:20 |
| marios|rover | chkumar|ruck: ah currenlty running in pipeline | 11:21 |
| chkumar|ruck | marios|rover: nope, but current train promotion pipeleine is running | 11:21 |
| marios|rover | yeah ack | 11:21 |
| marios|rover | but wdyt about comments/19? | 11:21 |
| chkumar|ruck | we can wait for the result then updates | 11:21 |
| chkumar|ruck | marios|rover: checking | 11:22 |
| *** akahat has quit IRC | 11:24 | |
| *** arxcruz is now known as arxcruz|pto | 11:32 | |
| *** udesale has joined #oooq | 11:35 | |
| *** ykarel|afk is now known as ykarel | 11:40 | |
| chkumar|ruck | marios|rover: one interesting thing, there is a capacity issue on rdo side only when train and master promotion share the resources a lot | 11:42 |
| chkumar|ruck | in the current going run I can see that | 11:42 |
| marios|rover | chkumar|ruck: hmmm yeah it makes sens. i was worried about that when i posted https://review.rdoproject.org/r/24073 | 11:43 |
| marios|rover | and yesterdays one too | 11:43 |
| marios|rover | chkumar|ruck: so they will overlap on some hours | 11:43 |
| marios|rover | chkumar|ruck: not sure what we can do about that i mean we have 3 pipelines that have to run in 24 hours, 2 of those like 3 times | 11:44 |
| marios|rover | so they will overlap | 11:44 |
| marios|rover | chkumar|ruck: please add a comment about that maybe? | 11:44 |
| marios|rover | chkumar|ruck: either LP or cix board | 11:44 |
| marios|rover | chkumar|ruck: damn you mean jobs failed on that already? | 11:45 |
| marios|rover | chkumar|ruck: i see master has fs20 fs1 centos fail is that what you mean ? | 11:46 |
| chkumar|ruck | marios|rover: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/9cf75cd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 11:48 |
| chkumar|ruck | marios|rover: commented added with this | 11:48 |
| marios|rover | sheise de la merde | 11:48 |
| chkumar|ruck | let me put all the failed jobs there | 11:48 |
| chkumar|ruck | marios|rover: may be decopupling the fs01 and fs020 will help? like first run fs01 once finishes then run fs020? | 11:50 |
| *** sshnaidm|afk is now known as sshnaidm | 11:52 | |
| marios|rover | chkumar|ruck: yeah but then we will have same problem again | 11:54 |
| marios|rover | chkumar|ruck: i mean we will lose another run per day | 11:54 |
| marios|rover | chkumar|ruck: if one run will take 8 hours | 11:54 |
| marios|rover | chkumar|ruck: fs20 takes like 4++ | 11:54 |
| chkumar|ruck | marios|rover: may be sshnaidm has some idea to split the tempest tests on fs020 | 11:56 |
| chkumar|ruck | marios|rover: brbr | 11:57 |
| marios|rover | chkumar|ruck: qe? | 11:57 |
| *** ykarel is now known as ykarel|afk | 11:57 | |
| *** rfolco has joined #oooq | 12:05 | |
| chkumar|ruck | *brb | 12:25 |
| *** epoojad1 is now known as epoojad1|afk | 12:25 | |
| *** jpena is now known as jpena|lunch | 12:29 | |
| *** epoojad1|afk has quit IRC | 12:30 | |
| *** dsneddon has quit IRC | 12:33 | |
| *** ykarel|afk is now known as ykarel| | 12:36 | |
| *** ykarel| is now known as ykarel | 12:36 | |
| chkumar|ruck | marios|rover: RHEL-8 fs01 train it is again hitting the same issue https://bugs.launchpad.net/tripleo/+bug/1853028 | 12:43 |
| openstack | Launchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged] | 12:43 |
| chkumar|ruck | I am not sure there is some issue with the image build itself | 12:43 |
| *** udesale has quit IRC | 12:50 | |
| marios|rover | chkumar|ruck: ack... folks from networking were poking at that already cix.. but problem is it isnt consistent it seems to be some time/race | 12:52 |
| marios|rover | chkumar|ruck: is it happening frequently on rhel8 you mean or not sure yet | 12:52 |
| chkumar|ruck | marios|rover: it is from current run of rhel8 fs01 train | 12:53 |
| chkumar|ruck | we will know more tomorrow | 12:53 |
| marios|rover | chkumar|ruck: you ok to give status on prog call? | 12:53 |
| marios|rover | in few mins starts | 12:53 |
| chkumar|ruck | marios|rover: yes | 12:53 |
| marios|rover | chkumar|ruck: cool thanks will join read-only then | 12:54 |
| chkumar|ruck | marios|rover: sure | 12:54 |
| *** dsneddon has joined #oooq | 12:55 | |
| *** ykarel is now known as ykarel|afk | 12:56 | |
| rfolco | panda, sshnaidm zbr need some awesome reviews on https://review.rdoproject.org/r/#/c/23875/ please, ps12 has a test snippet | 12:58 |
| rfolco | zbr, next step is to integrate dlrn_api ansible module (but I need to check with you where we are, I see some patches...) | 13:00 |
| *** dsneddon has quit IRC | 13:00 | |
| zbr | rfolco: did you read my msg since yesterday? | 13:00 |
| *** ykarel|afk has quit IRC | 13:00 | |
| rfolco | zbr, maybe not, sorry. | 13:00 |
| rfolco | oh | 13:01 |
| rfolco | this? | 13:01 |
| rfolco | <zbr> that is a response to a problem folco had yesterday, he was not able to make the dlrn_api module to work, this fixed the deployment. | 13:01 |
| rfolco | <zbr> mainly we can bundle roles, playbooks and modules and publish them to galaxy. | 13:01 |
| rfolco | <zbr> the only issue is that this works only with ansible 2.8, but i doubt we cannot upgrade to 2.8 everywhere if needed. | 13:01 |
| zbr | rfolco: https://www.reddit.com/r/ansible/comments/e94yqn/experience_from_building_my_first_ansible/ first link points to dlrn collection for ansible. https://www.reddit.com/r/ansible/comments/e94yqn/experience_from_building_my_first_ansible/ | 13:01 |
| chkumar|ruck | marios|rover: weshay for Train beta, we have go na? as we focus on upstream | 13:01 |
| zbr | mainly i packages it as a collection, making easy to install. | 13:01 |
| zbr | rfolco: it would be cool if you can attempt to consume it | 13:02 |
| rfolco | zbr, aaaah that answers my original question on how to install the module :) nice | 13:02 |
| rfolco | zbr, awesome, will take a look at this today | 13:03 |
| zbr | and it also installs the python module, in a bit questionable way but it should work for you. | 13:03 |
| weshay | chkumar|ruck, doesn't apply.. | 13:03 |
| chkumar|ruck | weshay: ok | 13:03 |
| weshay | chkumar|ruck, well... except for tempest | 13:03 |
| weshay | chkumar|ruck, is tempest ready for beta? | 13:03 |
| chkumar|ruck | weshay: yes it is always ready I think, we have not seen any blocker bugs | 13:03 |
| weshay | :) | 13:04 |
| weshay | chkumar|ruck, please remove the Dec-04 row | 13:04 |
| chkumar|ruck | weshay: done | 13:05 |
| weshay | panda, can you trigger a rdo promotion for queens please https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-queens-current-tripleo/ | 13:07 |
| weshay | soniya29, are you actively fixing the data in the skip list http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1 | 13:08 |
| zbr | rfolco: added few questions | 13:09 |
| soniya29 | weshay: yeah, some hidden_test cases are missing, trying to retrieve them. | 13:09 |
| rfolco | zbr, k thx | 13:09 |
| weshay | soniya29, k.. I would also like to see reviews submitted to gerrtit to remove the passing tests from the skip list | 13:10 |
| soniya29 | weshay, sure. I will let you know, once done | 13:11 |
| *** dsneddon has joined #oooq | 13:11 | |
| *** dsneddon has quit IRC | 13:16 | |
| panda | weshay: promotion running, restricted to curren-tripleo-rdo as we have another good candidate for current-tripleo today. | 13:17 |
| *** SurajPatil has joined #oooq | 13:18 | |
| marios|rover | thanks panda weshay yeah indeed wed/weekend was very green run | 13:18 |
| marios|rover | fs20 isn't in criteria even though i thought it was yesterday | 13:18 |
| marios|rover | weshay: we promoted queens yesterday too | 13:18 |
| *** rlandy has joined #oooq | 13:19 | |
| weshay | marios|rover, for queens? | 13:19 |
| marios|rover | yeah | 13:19 |
| marios|rover | and rocky will promote today too | 13:19 |
| marios|rover | https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-wednesday-weekend%20 | 13:19 |
| marios|rover | weshay: ^ | 13:20 |
| marios|rover | weshay: yeah panda ran the promotion yesterday for Q | 13:20 |
| *** surpatil has quit IRC | 13:21 | |
| weshay | marios|rover, ya.. but it wasn't running for rdo yet | 13:21 |
| weshay | marios|rover, /me updated queens criteria | 13:21 |
| marios|rover | weshay: you mean phase1? | 13:21 |
| marios|rover | weshay: 15:21 < weshay> marios|rover, ya.. but it wasn't running for rdo yet | 13:21 |
| marios|rover | weshay: or are you referring to fs20? | 13:21 |
| weshay | marios|rover, https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-queens-current-tripleo/ | 13:21 |
| weshay | and fs020 and fs001 were remarked out | 13:22 |
| marios|rover | weshay: ack | 13:22 |
| weshay | they both passed | 13:22 |
| marios|rover | so we were talkign with weshay about that earlier fs20 especially | 13:22 |
| marios|rover | lots of fails across all branches. often tempest. but then testproject rerun green | 13:22 |
| marios|rover | weshay: err 'we were talking with chkumar|ruck about that ' | 13:22 |
| marios|rover | weshay: but yes all green in https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-wednesday-weekend%20 | 13:23 |
| marios|rover | weshay: so Q and R can promote today's hash since it runs wednesday | 13:23 |
| weshay | marios|rover, I'm in favor of skipping inconsistent tests.. but that opinion is not totally agreed to across the team | 13:23 |
| weshay | chkumar|ruck, ^ | 13:23 |
| weshay | marios|rover, nice | 13:23 |
| marios|rover | weshay: testproject is good enough a tool imo | 13:23 |
| weshay | marios|rover, how are we on rhel8 fs001 | 13:23 |
| chkumar|ruck | weshay: we are already on that | 13:23 |
| marios|rover | http://38.145.34.55/centos7_rocky.log 2019-12-11 12:45:38,935 13396 INFO promoter Promoting the container images for dlrn hash 5beb8164ecf2f8448727bc3d9773b21a2ec22168 on rocky to current-tripleo | 13:24 |
| chkumar|ruck | weshay: https://review.opendev.org/#/c/698435/ | 13:24 |
| marios|rover | currently happening for R ^^ | 13:24 |
| chkumar|ruck | but we need to till curent promotion train run completes | 13:24 |
| marios|rover | and our panda automaton will invoke Queens for us weshay chkumar|ruck | 13:25 |
| marios|rover | pandatomaton | 13:25 |
| *** dsneddon has joined #oooq | 13:25 | |
| marios|rover | panda: i am poking at the manifest push btw | 13:25 |
| chkumar|ruck | panda: /nick pandatomaton | 13:25 |
| marios|rover | panda: we can sync after if you like? (but not at few mins before 7 like you usually do) | 13:25 |
| * marios|rover radio prog call right now | 13:25 | |
| *** amoralej is now known as amoralej|lunch | 13:26 | |
| weshay | chkumar|ruck, zbr .. chandan has to give status to the program call, let's chat 3rd party dep when I ping | 13:26 |
| weshay | chkumar|ruck, meh fuck this | 13:26 |
| chkumar|ruck | weshay: hmm | 13:26 |
| *** soniya29 has quit IRC | 13:27 | |
| weshay | chkumar|ruck, ok... pweeks is going to report for us | 13:28 |
| weshay | chkumar|ruck, let's chat w/ zbr in two min | 13:29 |
| weshay | zbr, you there? | 13:29 |
| rfolco | zbr, Two questions: 1) ansible2.9+ is required to install collections, we have 2.8. and 2) its under your namespace?? | 13:29 |
| weshay | zbr | 13:30 |
| weshay | zbr | 13:30 |
| weshay | zbr | 13:30 |
| weshay | zbr | 13:30 |
| *** soniya29 has joined #oooq | 13:30 | |
| weshay | zbr | 13:30 |
| chkumar|ruck | joined | 13:30 |
| weshay | zbr | 13:30 |
| weshay | zbr | 13:30 |
| *** dsneddon has quit IRC | 13:30 | |
| weshay | zbr | 13:30 |
| weshay | zbr | 13:30 |
| chkumar|ruck | ssbarnea: | 13:30 |
| weshay | zbr | 13:30 |
| chkumar|ruck | ssbarnea: | 13:30 |
| *** jpena|lunch is now known as jpena | 13:30 | |
| weshay | chkumar|ruck, is he there? | 13:30 |
| chkumar|ruck | he commented on a review few mins back | 13:30 |
| panda | marios|rover: ping me when you're ready | 13:31 |
| marios|rover | panda: i will take this under serious consideration | 13:31 |
| *** ykarel|afk has joined #oooq | 13:31 | |
| panda | marios|rover: I hate you too. | 13:31 |
| * marios|rover shrug | 13:32 | |
| marios|rover | coffee refill brb | 13:32 |
| zbr | i am here | 13:32 |
| *** surpatil has joined #oooq | 13:32 | |
| rlandy | zbr: nothing like being a wanted man | 13:33 |
| *** SurajPatil has quit IRC | 13:35 | |
| *** ykarel|afk is now known as ykarel | 13:46 | |
| *** dsneddon has joined #oooq | 13:54 | |
| *** soniya29 has quit IRC | 13:54 | |
| rfolco | rlandy, can you +w if https://review.rdoproject.org/r/#/c/23875/ looks good ? so I can fix any remaining issues after merging if any. There is a test snippet in comment for ps12 that I used local | 13:54 |
| rlandy | rfolco: yep - looking | 13:55 |
| rfolco | thanks rlandy | 13:55 |
| rlandy | rfolco: so I though we were going to have a separate pipe line for promotions | 13:56 |
| rlandy | thought | 13:57 |
| rlandy | is this just poc? | 13:57 |
| rfolco | for the job that promotes ? | 13:57 |
| rfolco | why? | 13:57 |
| rlandy | rfolco: thought that it would do all the component promotions | 13:57 |
| rlandy | rfolco: nvm for now, I'll w+ this so wecan get the reporting | 13:58 |
| rfolco | why it must be separate ? this way we can use zuul job dependency to define the criteria, and double check with dlrn | 13:58 |
| rlandy | I think we should look at further design this week | 13:58 |
| rfolco | rlandy, I see, combining components in one single promotion. Maybe. Not at this time. | 13:59 |
| rlandy | weshay: rfolco: we have a few decisions to make wrt component pipeline going forward. Can we meet for design discussion tomorrow? | 13:59 |
| rfolco | indeed | 13:59 |
| rlandy | rfolco: ack - w+ as poc for now | 13:59 |
| *** dsneddon has quit IRC | 13:59 | |
| weshay | rlandy, we can probably meet today if needed | 13:59 |
| rfolco | yes today please | 14:00 |
| rlandy | rfolco: you need the HTTP_PROTOCOL (added that yesterday) in https://review.rdoproject.org/r/#/c/23875/15/playbooks/tripleo-ci-base-promote-component-to-current-tripleo/run.yaml | 14:00 |
| rlandy | can do that in follow up patch | 14:00 |
| rlandy | rfolco: ok - you're w+'ed | 14:01 |
| rfolco | cool | 14:01 |
| weshay | mjturek, any updates? | 14:01 |
| rlandy | pls watch for fallout | 14:01 |
| rfolco | k | 14:02 |
| weshay | panda, ? | 14:04 |
| weshay | you joining? | 14:04 |
| *** dsneddon has joined #oooq | 14:05 | |
| mjturek | weshay: turned off firewalld and magically the curl worked | 14:05 |
| weshay | it's magically delicious | 14:06 |
| *** bhagyashris has quit IRC | 14:06 | |
| mjturek | so it's definitely a firewall rule | 14:06 |
| mjturek | indeed | 14:06 |
| mjturek | unfortunately we won't have a new held node for about 2 hours | 14:06 |
| mjturek | rfolco you up for a screen share around then? | 14:06 |
| rlandy | migi: I merged the periodic-tripleo-ci-rhel-8-standalone-rhos-17-compute-rhos-16 job | 14:07 |
| rlandy | it updates with rhos-17 components but it fails tempest | 14:07 |
| *** amoralej|lunch is now known as amoralej | 14:07 | |
| rlandy | which may be legit | 14:07 |
| rfolco | mjturek, yup | 14:07 |
| rlandy | the equivalent pure rhos-16 job passes | 14:07 |
| rfolco | mjturek, tmate or what | 14:07 |
| mjturek | rfolco I'm having trouble with tmate, so might have to just share over google hangouts | 14:08 |
| mjturek | rfolco unless you have cico access | 14:08 |
| rfolco | no I don't | 14:09 |
| mjturek | okay no worries | 14:09 |
| rfolco | lets use ci community gmeet | 14:09 |
| rfolco | one sec | 14:09 |
| mjturek | sounds good | 14:09 |
| *** dsneddon has quit IRC | 14:10 | |
| rfolco | mjturek, I m in | 14:12 |
| rfolco | https://meet.google.com/bqx-xwht-wky | 14:12 |
| mjturek | rfolco sorry, we won't have a node for another 2 hours | 14:12 |
| mjturek | do you have time around then? | 14:12 |
| rfolco | mjturek, I am confused. You don't have the node now ? only 2h from now? | 14:13 |
| mjturek | rfolco correct, the nodes are held for 12 hrs after a failed run | 14:13 |
| mjturek | the next run starts in two hours | 14:13 |
| rfolco | mjturek, ok. Just ping me. What I think that is happening is: zuul parent jobs are doing something w/ firewall that cico doesn't | 14:14 |
| mjturek | sounds plausible! | 14:14 |
| marios|rover | so chkumar|ruck just ping me from internal server saying his freenode went down, and its just as well cos he can't currently hear me say "great news chkumar|ruck apparently we get to keep ruck roving \o/ it's gonna be epic https://i.ytimg.com/vi/Zt1yB1V6sAA/maxresdefault.jpg " | 14:14 |
| *** holser has quit IRC | 14:19 | |
| *** holser__ has joined #oooq | 14:19 | |
| rfolco | mjturek, a typical standalone job in zuul/rdo: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/ae3f5e4/logs/undercloud/var/log/extra/network.txt.gz | 14:19 |
| mjturek | interesting! this looks pretty different | 14:20 |
| rfolco | mjturek, container build job does not save such info, I will take a look if we can add something | 14:20 |
| *** epoojad1 has joined #oooq | 14:24 | |
| *** sanjayu__ has quit IRC | 14:24 | |
| *** epoojad1 has quit IRC | 14:30 | |
| *** epoojad1 has joined #oooq | 14:32 | |
| marios|rover | https://tree.taiga.io/project/tripleo-ci-board/task/1315?kanban-status=1447274 | 14:36 |
| *** sanjayu__ has joined #oooq | 14:36 | |
| beagles | in u/s CI jobs is there any way that a directory and file that was created during the undercloud install step in /usr/share might be removed by the time the overcloud deploy step comes around? | 14:42 |
| *** dsneddon has joined #oooq | 14:43 | |
| *** dtantsur is now known as dtantsur|brb | 14:44 | |
| chkumar|ruck | marios|rover: ok i am back | 14:48 |
| marios|rover | chkumar|ruck: good news \o/ 16:24 -!- sanjayu__ [~saneax@125.19.27.210] has quit [Ping timeout: 268 seconds] | 14:50 |
| marios|rover | https://i.ytimg.com/vi/Zt1yB1V6sAA/maxresdefault.jpg | 14:50 |
| marios|rover | chkumar|ruck: that sorry wrong paste :D ^ | 14:50 |
| chkumar|ruck | marios|rover: you are rucking from today :P, /me is rovering | 14:50 |
| chkumar|ruck | s/today/tomorrow | 14:51 |
| chkumar|ruck | s/is/will | 14:51 |
| marios|rover | chkumar|ruck: ack this day just keeps getting better \o/ | 14:51 |
| chkumar|ruck | marios|rover: I cannot express my feeling, no words to say | 14:51 |
| *** ysandeep has joined #oooq | 14:59 | |
| rlandy | what happened here - why are chkumar|ruck and marios|rover still rucking/roving? | 15:04 |
| weshay | rlandy, so.. I'm working w/ rfolco about it.. and we'll make it official soon | 15:06 |
| weshay | rlandy, probably need to do it live in a standup | 15:07 |
| weshay | so hold tight | 15:07 |
| * rlandy should keep my mouth shut ... just feels unjust | 15:08 | |
| marios|rover | thanks rlandy :D but its only another week don't worry | 15:09 |
| *** TrevorV has joined #oooq | 15:11 | |
| *** Goneri has joined #oooq | 15:11 | |
| marios|rover | panda: weshay: we decided panda will manually run, but FYI there is no queens containers-build-push, is why we see no x86_64 tags in the rdo registry for queens namespace | 15:15 |
| marios|rover | weshay: wtf is okr please | 15:22 |
| marios|rover | organic knowledge request | 15:22 |
| weshay | marios|rover, objectives, key results | 15:22 |
| marios|rover | weshay: thanks :) | 15:23 |
| marios|rover | https://review.opendev.org/695878 awoooooo | 15:24 |
| *** sanjayu__ has quit IRC | 15:27 | |
| chkumar|ruck | marios|rover: periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train it passed | 15:27 |
| chkumar|ruck | I donot know how | 15:27 |
| chkumar|ruck | we even donot need skip list | 15:28 |
| chkumar|ruck | train will promote now | 15:28 |
| marios|rover | chkumar|ruck: nice | 15:28 |
| marios|rover | chkumar|ruck: good job we wanted to hand over a clean ship ;) | 15:28 |
| marios|rover | chkumar|ruck: we should do ruck|rover handoff and update the etherpad :D | 15:29 |
| marios|rover | chkumar|ruck: here you go https://etherpad.openstack.org/p/ruckroversprint19 | 15:29 |
| chkumar|ruck | zbr: weshay marios|rover zbr panda rfolco rlandy tomorrow there is a review.rdoproject.org will have downtime for a 1 hr | 15:30 |
| panda | chkumar|ruck: I saw it was monday ? | 15:30 |
| rlandy | ack | 15:30 |
| chkumar|ruck | context check internal tripleo ci email | 15:30 |
| marios|rover | chkumar|ruck: thanks | 15:30 |
| chkumar|ruck | sorry 2019-12-16 | 15:30 |
| marios|rover | chkumar|ruck: yes i vaguely remember thinking 'sweet that aint gonna be my problem any more' | 15:31 |
| marios|rover | karms | 15:31 |
| marios|rover | karma even | 15:31 |
| marios|rover | :D | 15:31 |
| chkumar|ruck | panda: you are correct I am seeing too much 12 these days | 15:31 |
| marios|rover | chkumar|ruck: cancelling honolulu booking | 15:31 |
| marios|rover | will see if they can push it back a week | 15:31 |
| chkumar|ruck | marios|rover: I have just done booking no cancelling now | 15:31 |
| panda | chkumar|ruck: did you read "murder on the orient express" lately ? | 15:32 |
| *** surpatil has quit IRC | 15:33 | |
| chkumar|ruck | panda: nope, watched this late night https://en.wikipedia.org/wiki/Laal_Kaptaan | 15:33 |
| marios|rover | chkumar|ruck: just ask them to put all the mojito in the freezer should be good for a week | 15:36 |
| marios|rover | surely | 15:36 |
| *** panda is now known as panda|bbl | 15:44 | |
| chkumar|ruck | ok last meeting done, time to sleep now | 15:46 |
| chkumar|ruck | marios|rover: see ya mate next day :-) | 15:47 |
| *** chkumar|ruck is now known as raukadah | 15:47 | |
| rfolco | rlandy, weshay component ci sync in 10 min, my cal says | 15:51 |
| rlandy | ack | 15:51 |
| marios|rover | raukadah: o/ | 15:59 |
| marios|rover | raukadah: you too | 15:59 |
| rfolco | rlandy, weshay I'm in | 16:01 |
| rfolco | https://meet.google.com/arr-rkdv-qws | 16:01 |
| weshay | k.. be there in a few | 16:01 |
| rlandy | one sec - just submitting patch | 16:02 |
| *** ysandeep has quit IRC | 16:23 | |
| *** jaosorior has joined #oooq | 16:25 | |
| mjturek | hey rfolco we've got that node now if you have some time | 16:28 |
| rfolco | hmm stuck in a mtg :( | 16:28 |
| rfolco | will ping you asap | 16:28 |
| mjturek | okay thanks! | 16:28 |
| * marios|rover out folks ttyl | 16:33 | |
| *** marios|rover is now known as marios|out | 16:33 | |
| *** marios|out has quit IRC | 16:53 | |
| *** dtantsur|brb is now known as dtantsur | 16:55 | |
| sshnaidm | weshay, rlandy we replaced bmc template on rdo-cloud now, if you see any problems with ovb - let me know please | 16:56 |
| rlandy | sshnaidm: ack - thanks | 17:10 |
| *** ykarel is now known as ykarel|away | 17:11 | |
| sshnaidm | it fixed jobs for OVB master branch https://review.opendev.org/#/c/683972/ | 17:13 |
| rlandy | weshay: https://review.opendev.org/#/c/691994/ | 17:14 |
| rlandy | https://code.engineering.redhat.com/gerrit/#/c/184172/ | 17:17 |
| rlandy | weshay: ^^ | 17:17 |
| *** panda|bbl is now known as panda | 17:20 | |
| raukadah | sshnaidm: is it going to affect current ovb periodic jobs also? | 17:21 |
| sshnaidm | raukadah, it shouldn't affect anything, just to keep eyes on | 17:21 |
| raukadah | sshnaidm: sure | 17:21 |
| raukadah | rlandy: sshnaidm need some help to fix master and train periodic pipeline, current resources are shared alot leading to no valid host found error multiple times | 17:23 |
| raukadah | can we run these twice instead of thrice? | 17:23 |
| raukadah | sshnaidm: rlandy more info here https://bugs.launchpad.net/tripleo/+bug/1854721 | 17:24 |
| openstack | Launchpad bug 1854721 in tripleo "FS 01 master periodic job failed during overcloud deploy with Exceeded maximum number of retries and 503 status code" [Critical,Won't fix] | 17:24 |
| rlandy | going to go dig car out of the snow - back in a bit | 17:25 |
| *** rlandy is now known as rlandy|brb | 17:26 | |
| *** rlandy|brb is now known as rlandy | 17:50 | |
| *** ykarel|away has quit IRC | 17:51 | |
| rlandy | raukadah; k - will look at it | 17:55 |
| sshnaidm | rlandy, raukadah there is one node that just didn't boot up: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/05b7001/logs/undercloud/var/log/extra/baremetal_list.txt.gz | 17:57 |
| sshnaidm | and I don't know how to debug it tbh | 17:57 |
| sshnaidm | like it didn't exist.. | 17:58 |
| rlandy | consistent or a once off? | 17:59 |
| *** derekh has quit IRC | 18:01 | |
| *** jpena is now known as jpena|off | 18:06 | |
| *** dsneddon has quit IRC | 18:07 | |
| *** dsneddon has joined #oooq | 18:17 | |
| rlandy | rfolco: weshay: if you're looking for me ... going to get flu shot - brb | 18:18 |
| *** rlandy is now known as rlandy|brb | 18:18 | |
| *** dsneddon has quit IRC | 18:22 | |
| *** dtantsur is now known as dtantsur|afk | 18:23 | |
| *** jfrancoa has quit IRC | 18:25 | |
| *** jaosorior has quit IRC | 18:27 | |
| *** tosky has quit IRC | 18:44 | |
| *** dsneddon has joined #oooq | 18:55 | |
| *** dsneddon has quit IRC | 19:00 | |
| *** rlandy|brb is now known as rlandy | 19:11 | |
| *** jtomasek has quit IRC | 19:17 | |
| *** amoralej is now known as amoralej|off | 19:23 | |
| *** dsneddon has joined #oooq | 19:29 | |
| *** dsneddon has quit IRC | 19:38 | |
| *** holser__ has quit IRC | 19:42 | |
| *** tesseract has quit IRC | 19:46 | |
| mjturek | weshay: rfolco has found the answer :), just need to implement a solution | 19:54 |
| *** dsneddon has joined #oooq | 19:57 | |
| weshay | ah cool | 19:58 |
| weshay | rfolco++ | 19:58 |
| rfolco | yeah if you need an iptables guy, don't call me | 19:58 |
| rfolco | :) | 19:58 |
| weshay | no worries.. you'll have all the fun of learning firewalld | 19:58 |
| weshay | rfolco, is it a firewall rule on the host? | 19:59 |
| weshay | or docker? | 19:59 |
| rfolco | weshay, its the chain | 20:01 |
| rfolco | weshay, openstack-INPUT is a nodepool chain created by prepare node... | 20:01 |
| weshay | ah ya | 20:01 |
| rfolco | weshay, so we just used an existing chain and it worked | 20:02 |
| rfolco | https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/pre.yaml#L81 | 20:02 |
| weshay | so the rule in ci.centos isn't applied | 20:02 |
| rfolco | yep | 20:02 |
| rfolco | applies but no effect | 20:02 |
| weshay | rfolco, so check if the chain exists.. if it does not.. call the task to add the rule again to a default chain | 20:03 |
| weshay | like input | 20:03 |
| rfolco | or DOCKER-ISOLATION | 20:04 |
| rfolco | yes | 20:04 |
| rfolco | will leave the improvement to mjturek | 20:04 |
| rfolco | as I am heads down in the promotion role | 20:04 |
| rfolco | rfolco@home(RADIO): Fire in the hole! | 20:05 |
| weshay | ha.. cool | 20:06 |
| rfolco | if you got the ref you were a gamer one day | 20:06 |
| rfolco | :) | 20:06 |
| rlandy | rfolco: weshay: could one of you vote on https://code.engineering.redhat.com/gerrit/#/c/187559/ for legitimacy | 20:17 |
| weshay | rlandy, done | 20:18 |
| rlandy | thank you | 20:18 |
| rlandy | woohoo - finally - os_tempest running on baremetal again | 20:25 |
| rlandy | raukadah: ^^ | 20:26 |
| rlandy | fixed allocation pools error | 20:26 |
| rlandy | ok - so no idea what this means ... Unable to freeze job graph: Decryption failed. | 20:31 |
| rlandy | waiting on infra to answer that | 20:31 |
| *** holser has joined #oooq | 20:48 | |
| weshay | rfolco, rlandy which rdo tenant are the component jobs running in? | 21:22 |
| weshay | https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds | 21:22 |
| weshay | I don't see anything from openstack-component-compute | 21:22 |
| weshay | meh.. maybe because it's just once a day | 21:23 |
| weshay | ok | 21:23 |
| rlandy | weshay: what are you looking for? | 21:23 |
| rlandy | upstream/downstream? | 21:24 |
| weshay | upstream ( rdo ) | 21:24 |
| rlandy | getting | 21:25 |
| rlandy | weshay: https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-compute | 21:25 |
| rlandy | and | 21:25 |
| rlandy | weshay: https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-keystone | 21:26 |
| weshay | that I see | 21:26 |
| weshay | aye | 21:26 |
| weshay | just nothing in the current json file | 21:26 |
| rlandy | the inheritance downstream failed | 21:26 |
| rlandy | what json file? | 21:26 |
| weshay | rlandy, I'll merge the view in cockpit and let it run for a few days | 21:26 |
| weshay | rlandy, https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds | 21:26 |
| rlandy | it only keep the last x-number builds | 21:27 |
| weshay | aye | 21:27 |
| rlandy | weshay: https://code.engineering.redhat.com/gerrit/#/c/187453/ didn;t go well | 21:27 |
| rlandy | waiting for response from infra | 21:27 |
| rlandy | I think I may have to create a role upstream | 21:27 |
| rlandy | get rid of the stuff in the playbook | 21:28 |
| rlandy | easier way about it | 21:28 |
| rlandy | just possible upstream impact | 21:28 |
| weshay | rlandy, better to convert things to roles | 21:28 |
| weshay | it's ok to risk the impact | 21:28 |
| rlandy | weshay: I am going o revert the change so I can test the role stuff | 21:29 |
| weshay | +1 | 21:29 |
| rlandy | https://code.engineering.redhat.com/gerrit/#/c/187573/ | 21:29 |
| rlandy | ok - let's try a different approach | 21:32 |
| weshay | rlandy, rfolco fyi.. got the cockpit started for compute and keystone https://review.rdoproject.org/r/24082 no need to review yet | 21:38 |
| rlandy | ok | 21:39 |
| *** tosky has joined #oooq | 22:15 | |
| *** rfolco has quit IRC | 22:25 | |
| *** sshnaidm is now known as sshnaidm|afk | 22:33 | |
| *** rfolco has joined #oooq | 22:45 | |
| *** TrevorV has quit IRC | 22:52 | |
| *** holser has quit IRC | 22:53 | |
| weshay | rfolco, rlandy http://dashboard-ci.tripleo.org/d/UDA4H3aZk/component-pipeline?orgId=1 | 23:07 |
| rlandy | weshay: very cool | 23:08 |
| *** ksambor_ has joined #oooq | 23:22 | |
| *** ksambor has quit IRC | 23:22 | |
| *** ksambor_ is now known as ksambor | 23:22 | |
| *** ksambor has quit IRC | 23:23 | |
| *** holser has joined #oooq | 23:26 | |
| *** rlandy is now known as rlandy|bbl | 23:27 | |
| *** brault has quit IRC | 23:30 | |
| *** brault has joined #oooq | 23:31 | |
| *** brault has quit IRC | 23:32 | |
| *** brault has joined #oooq | 23:33 | |
| *** ysandeep has joined #oooq | 23:37 | |
| *** tosky has quit IRC | 23:40 | |
| *** brault has quit IRC | 23:58 | |
| *** brault has joined #oooq | 23:59 | |
| *** ccamacho|pto has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
