| rlandy | weshay|ruck: https://review.rdoproject.org/r/#/c/25865/ | 00:01 |
|---|---|---|
| *** rlandy has quit IRC | 00:17 | |
| *** tosky has quit IRC | 00:18 | |
| *** Goneri has quit IRC | 01:15 | |
| *** jschlueter has quit IRC | 02:07 | |
| *** jschlueter has joined #oooq | 02:08 | |
| *** jmasud has quit IRC | 02:11 | |
| *** jmasud has joined #oooq | 03:38 | |
| *** jmasud has quit IRC | 03:56 | |
| *** apetrich has quit IRC | 04:00 | |
| *** ysandeep|away is now known as ysandeep | 04:23 | |
| *** skramaja has joined #oooq | 04:25 | |
| *** ykarel|away is now known as ykarel | 04:34 | |
| *** udesale has joined #oooq | 04:40 | |
| *** jmasud has joined #oooq | 05:14 | |
| *** jmasud has quit IRC | 05:47 | |
| chandankumar | ykarel, Hello | 05:51 |
| chandankumar | ykarel, regarding this failure https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/410fb3f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 05:51 |
| chandankumar | did we get the patch up ? | 05:51 |
| *** jmasud has joined #oooq | 05:56 | |
| ykarel | chandankumar, looking | 06:00 |
| ykarel | chandankumar, yes | 06:01 |
| ykarel | chandankumar, https://review.opendev.org/#/c/712571/ + Depends-On should clear it | 06:04 |
| ykarel | but need image rebuilds, | 06:04 |
| ykarel | not sure if those are already tested | 06:04 |
| ykarel | let's test those quickly and clear it | 06:05 |
| ykarel | i fire test patch | 06:05 |
| *** jmasud has quit IRC | 06:07 | |
| *** marios has joined #oooq | 06:25 | |
| *** dpawlik has quit IRC | 06:37 | |
| *** dpawlik has joined #oooq | 06:55 | |
| *** jmasud has joined #oooq | 07:00 | |
| *** dpawlik has quit IRC | 07:10 | |
| *** dpawlik has joined #oooq | 07:16 | |
| chandankumar | marios, https://review.rdoproject.org/r/25884 | 07:17 |
| chandankumar | please have a look | 07:17 |
| marios | chandankumar: o/ | 07:21 |
| marios | chandankumar: so things look...... ok? https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master | 07:21 |
| marios | i was expecting red | 07:21 |
| marios | and fire | 07:21 |
| chandankumar | marios, we tested the contents already in rdoinfo patch so less fire | 07:22 |
| marios | baremetal fs1 is red | 07:22 |
| chandankumar | marios, https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-containers-undercloud-minion-master/0067102/job-output.txt | 07:22 |
| marios | but still lots of green in http://dashboard-ci.tripleo.org/d/UDA4H3aZk/component-pipeline?orgId=1 | 07:22 |
| chandankumar | marios, ykarel has putted the patch to re-run it | 07:23 |
| chandankumar | marios, https://review.rdoproject.org/r/#/c/25332/ | 07:23 |
| marios | chandankumar: wf https://review.rdoproject.org/r/#/c/25884/ | 07:24 |
| chandankumar | doing that | 07:24 |
| chandankumar | thanks | 07:24 |
| marios | chandankumar: yeah just did | 07:24 |
| chandankumar | marios, https://review.opendev.org/712487 and https://review.opendev.org/712571 also needed for ovb | 07:25 |
| marios | chandankumar: ack will check | 07:27 |
| chandankumar | marios, did we get a bug for fs030 | 07:29 |
| marios | chandankumar: i am not aware of one | 07:30 |
| chandankumar | marios, the mariadb container pulling issue | 07:30 |
| marios | chandankumar: yeah local registry issue | 07:30 |
| chandankumar | going to hunt for some food, brb | 07:30 |
| *** jfrancoa has joined #oooq | 07:34 | |
| *** jfrancoa is now known as jfrancoa|babysit | 07:34 | |
| *** tesseract has joined #oooq | 08:12 | |
| *** bogdando has joined #oooq | 08:12 | |
| *** amoralej|off is now known as amoralej | 08:17 | |
| chandankumar | arxcruz|rover, fyi https://bugs.launchpad.net/tripleo/+bug/1867323 | 08:18 |
| openstack | Launchpad bug 1867323 in tripleo " standalone deploy failed at Error: error checking path "/run/libvirt": stat /run/libvirt: no such file or directory" [Critical,Confirmed] | 08:18 |
| *** tosky has joined #oooq | 08:46 | |
| *** jpena|off is now known as jpena | 08:53 | |
| *** bogdando_ has joined #oooq | 08:54 | |
| *** bogdando has quit IRC | 08:54 | |
| *** bogdando_ is now known as bogdando | 08:54 | |
| marios | panda: so why didn't it fail https://review.rdoproject.org/r/#/c/25782/11/ci-scripts/dlrnapi_promoter/dlrn_client.py | 08:58 |
| arxcruz|rover | chandankumar: thanks | 09:06 |
| panda | marios: because exceptions in the logging are not fatal | 09:06 |
| *** ykarel is now known as ykarel|lunch | 09:07 | |
| *** bogdando_ has joined #oooq | 09:16 | |
| marios | panda: grumble grumble | 09:16 |
| *** bogdando has quit IRC | 09:17 | |
| *** bogdando_ is now known as bogdando | 09:17 | |
| *** bogdando_ has joined #oooq | 09:21 | |
| *** bogdando has quit IRC | 09:21 | |
| *** bogdando_ is now known as bogdando | 09:21 | |
| *** bogdando_ has joined #oooq | 09:24 | |
| zbr | marios: please revote on https://review.opendev.org/#/c/705623/ (added test as requested....) | 09:25 |
| *** bogdando has quit IRC | 09:26 | |
| *** bogdando has joined #oooq | 09:27 | |
| *** bogdando_ has quit IRC | 09:29 | |
| marios | zbr: ack will check in a bit | 09:29 |
| *** bogdando_ has joined #oooq | 09:31 | |
| *** bogdando has quit IRC | 09:32 | |
| *** bogdando_ is now known as bogdando | 09:32 | |
| *** derekh has joined #oooq | 09:33 | |
| *** apetrich has joined #oooq | 09:50 | |
| marios | E Expected: [call("Check named hashes: named hashes for label 'current-tripleo' changed since last check. At promotion start: commit: %s, distro: %s, component: %s, timestamp: %s. Now: commit: %s, distro: %s, component: %s, timestamp: %s", 'd221f4b33cf2763875fc6394902f7923108a34da', '70bdcd40eb5cc62e4762a7db0086e09f6edf2e5c', None, '1528085427', 'e3d9fffbf82ec71deff60ba914f1db0e1625466a', | 09:54 |
| marios | 'Iba78e857267ac771d23919fbd1e3c9fcc5813c9', None, '1528085529')] | 09:54 |
| marios | panda: is it ok if i hate you a bit right now? ^ | 09:54 |
| marios | E Actual: [call("Check named hashes: named hashes for label 'current-tripleo' changed since last check. At promotion start: commit: d221f4b33cf2763875fc6394902f7923108a34da, distro: 70bdcd40eb5cc62e4762a7db0086e09f6edf2e5c, component: None, timestamp: 1528085427. Now: commit: e3d9fffbf82ec71deff60ba914f1db0e1625466a, distro: Iba78e857267ac771d23919fbd1e3c9fcc5813c9, component: None, timestamp: | 09:54 |
| marios | 1528085529")]. | 09:54 |
| marios | ;) | 09:54 |
| panda | marios: as long as you don't stop testing, you can loathe me. | 09:56 |
| *** ykarel|lunch is now known as ykarel | 09:56 | |
| marios | panda: but https://review.rdoproject.org/r/#/c/24774/70/ci-scripts/dlrnapi_promoter/test_dlrn_client_unit.py | 09:56 |
| marios | so why is that one not failing then? | 09:56 |
| marios | panda: asser_has_calls | 09:56 |
| panda | marios: oh shoot | 09:58 |
| panda | lol | 09:58 |
| marios | well | 09:58 |
| panda | that is actually very dangerous | 09:59 |
| marios | wtf is going on then | 09:59 |
| panda | I know what is happening | 09:59 |
| marios | i can no longer trust anything | 09:59 |
| panda | and I don't like it | 09:59 |
| marios | is this keyboard real/ | 09:59 |
| panda | but I know how to solve it in theory | 10:00 |
| panda | marios: so, it's a bit complicated, you need to understand how the magicmock object works | 10:00 |
| panda | marios: the magicmock is acually a sink object, it's very flexible | 10:00 |
| panda | marios: it has method that deal with missing attribute | 10:01 |
| panda | marios: it accepts everything you give it, or you call | 10:01 |
| panda | marios: so taht call works because the magicmock, even if it recognize taht the method is actually not existing, it behaves normally, for the sake fo the mock | 10:01 |
| panda | marios: there are ways to tell the mock object to be more strict | 10:02 |
| panda | marios: and fail if certain methods are called incorrectly | 10:02 |
| panda | marios: but I don't activate them here | 10:02 |
| panda | marios: wo typos re actually very very bad, because the mock object dosn't complain | 10:02 |
| panda | marios: indeed, if I change to the right call, the tests fai | 10:03 |
| panda | l | 10:03 |
| panda | dayum this is a big one | 10:04 |
| panda | thanks marios | 10:04 |
| marios | panda: ack gonna go back to fighting with assert_has_calls | 10:04 |
| marios | panda: still hate you tho | 10:04 |
| panda | marios: http://www.agmweb.ca/2015-05-09-a-better-mock/ | 10:05 |
| panda | marios: good, good, let the hat flow through you | 10:06 |
| chandankumar | akahat, https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d62/712251/1/gate/tripleo-ci-centos-7-undercloud-containers/d62172f/logs/undercloud/var/log/tempest/tempest_run.log and file a bug here https://bugs.launchpad.net/tripleo/+filebug | 10:12 |
| panda | marios: fixed in the next PS | 10:12 |
| panda | marios: converted all the patches to autospect=True, now retesting all unit tests. | 10:19 |
| marios | panda: k will rebase once i'm done with these ones | 10:19 |
| panda | 68 failed! | 10:19 |
| panda | .... | 10:19 |
| panda | maybe too much autospeccing | 10:19 |
| panda | yep wild autospaccing is bad, reverting to default. | 10:23 |
| panda | but this is actually very serious | 10:23 |
| akahat | arxcruz|rover, hi. Could you please triage this bug: https://bugs.launchpad.net/tripleo/+bug/1867332 | 10:24 |
| openstack | Launchpad bug 1867332 in tripleo "Mistrial tests getting skiped in undercloud deployment" [Undecided,New] | 10:24 |
| akahat | arxcruz|rover, i'm working on fixing it. | 10:24 |
| arxcruz|rover | akahat: ack | 10:30 |
| arxcruz|rover | chandankumar: do we discover mistral on tempestconf ? | 10:31 |
| arxcruz|rover | kopecmartin: ^ | 10:31 |
| chandankumar | arxcruz|rover, yes, it was a bug on tempestconf | 10:31 |
| arxcruz|rover | chandankumar: was it fixed? | 10:32 |
| chandankumar | arxcruz|rover, akahat is working on fix | 10:32 |
| akahat | arxcruz|rover, https://review.opendev.org/#/c/712903/ | 10:37 |
| *** chem has quit IRC | 10:50 | |
| arxcruz|rover | ok | 10:56 |
| marios | panda: are you still working on https://review.rdoproject.org/r/#/c/24774/ i see some fails now | 11:07 |
| marios | panda: gonna rebase onto that v71 anyway and take from there updated the tests | 11:07 |
| *** udesale_ has joined #oooq | 11:09 | |
| *** saneax has quit IRC | 11:12 | |
| *** udesale has quit IRC | 11:12 | |
| chandankumar | arxcruz|rover, kopecmartin https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ?view Agenda for today's meeting | 11:20 |
| panda | d'oh | 11:24 |
| *** saneax has joined #oooq | 11:25 | |
| panda | marios: download problems | 11:25 |
| panda | marios: zbr if they are stopping mirroring python 2.7 we are so screwed | 11:26 |
| marios | where 13:25 < panda> marios: download problems | 11:26 |
| *** sanjayu_ has joined #oooq | 11:28 | |
| dpawlik | sshnaidm|afk, Yesterday they merged me mirror for fedora 31, I will check also for ansible-role-tripleo-ci-zuul-runner | 11:28 |
| *** saneax has quit IRC | 11:30 | |
| marios | panda: lucky 13 ready review please when you next have time thanks https://review.rdoproject.org/r/#/c/25782/ | 11:31 |
| panda | marios: download problem may mean taht mirrors have stopped myrroring packages fro python27 | 11:33 |
| marios | panda: oh you mean that's wher the py tests failed on your patch? | 11:34 |
| panda | marios: that's where all the patches failed | 11:34 |
| marios | panda: when did tox-py27 go non voting btw just noticed it zbr ? | 11:34 |
| marios | panda: must have happened today was voting yesterday anyway | 11:35 |
| * marios food biab | 11:36 | |
| arxcruz|rover | chandankumar: what are we running in centos-8-undercloud-containers job regarding tempest? it's timeing out | 11:42 |
| arxcruz|rover | seems to be running full tempest | 11:42 |
| chandankumar | arxcruz|rover, not full tempest | 11:43 |
| chandankumar | arxcruz|rover, let me check | 11:43 |
| arxcruz|rover | chandankumar: tempest api and mistral | 11:45 |
| arxcruz|rover | but it's taking 2 and half hour | 11:45 |
| chandankumar | arxcruz|rover, only api , if mistral project patch then mistral tests | 11:46 |
| arxcruz|rover | chandankumar: right, but still... | 11:47 |
| arxcruz|rover | 2 and half hour for tempes api is too much | 11:47 |
| zbr | panda: i would not be surprised to see it happening, but in less than a week openstack can have its own pypi servers. | 11:48 |
| arxcruz|rover | chandankumar: well, nevermind, it was only two timeouts | 11:48 |
| zbr | I used artifactory in the past or the opensource pypi repo in the past, both reliable. | 11:49 |
| weshay|ruck | arxcruz|rover, 0/ | 11:57 |
| arxcruz|rover | weshay|ruck: o/ | 11:58 |
| weshay|ruck | arxcruz|rover, find anything interesting w/ those two tempest runs? | 12:00 |
| arxcruz|rover | weshay|ruck: nope, seems to be random failures | 12:03 |
| arxcruz|rover | network issues one | 12:03 |
| weshay|ruck | bhagyashris++ | 12:03 |
| arxcruz|rover | chandankumar: tosky kopecmartin https://review.opendev.org/712903 please review :) | 12:06 |
| arxcruz|rover | zuul is happy | 12:06 |
| *** soniya29 has joined #oooq | 12:07 | |
| arxcruz|rover | akahat: can you create a dnm patch with a depends-on on https://review.opendev.org/712903 to check if it will work ? | 12:09 |
| arxcruz|rover | or test it locally | 12:09 |
| *** rlandy has joined #oooq | 12:11 | |
| *** chem has joined #oooq | 12:14 | |
| *** udesale_ is now known as udesale | 12:16 | |
| panda | zbr: it's not all the python2.7 packages | 12:16 |
| panda | zbr: it's just pytest-html | 12:16 |
| panda | zbr: they are not mirroning 2.0.2 anymore | 12:16 |
| panda | 2.0.1 | 12:16 |
| panda | that's why everything is failing | 12:17 |
| panda | the workaround with 2.0.1 will not work anymore | 12:17 |
| zbr | not true, likely a broken mirror. | 12:17 |
| panda | ERROR: Could not find a version that satisfies the requirement pytest-html===2.0.1 (from -c https://releases.openstack.org/constraints/upper/master (line 810)) (from versions: 1.0, 1.1, 1.2, 1.3, 1.3.1, 1.3.2, 1.4, 1.5, 1.5.1, 1.6, 1.7, 1.8.0, 1.8.1, 1.9.0, 1.10.0, 1.10.1rc1, 1.10.1, 1.11.0, 1.11.1, 1.12.0, 1.13.0, 1.14.0, 1.14.1, 1.14.2, 1.15.0, 1.15.1, 1.15.2, 1.16.0, 1.16.1, 1.17.0, 1.18.0, 1.19.0, 1.20.0, | 12:18 |
| panda | 1.21.0, 1.21.1, 1.22.0, 1.22.1)\nERROR: No matching distribution found for pytest-html===2.0.1 (from -c https://releases.openstack.org/constraints/upper/master (line 810))\n" | 12:18 |
| *** jpena is now known as jpena|lunch | 12:30 | |
| weshay|ruck | ysandeep, https://hackmd.io/sn48EeOoRjWNzvtV9IudxQ?sync=&type= | 12:31 |
| ysandeep | weshay|ruck, thanks! | 12:33 |
| soniya29 | weshay | 12:33 |
| soniya29 | weshay|ruck, meeting? | 12:34 |
| *** ysandeep is now known as ysandeep|afk | 12:34 | |
| weshay|ruck | soniya29, sorry our 1-1 was yesterday :) | 12:36 |
| weshay|ruck | we're in the tempest planning mtg now | 12:36 |
| *** ykarel is now known as ykarel|afk | 12:44 | |
| chandankumar | weshay|ruck, https://review.opendev.org/#/q/topic:centos-8-jobs+(status:open+OR+status:merged) | 12:50 |
| zbr | panda: not sure who added pytest-html===2.0.1 there because is WRONG 2.x requires py36! | 12:52 |
| zbr | this file will never work on py27 | 12:52 |
| arxcruz|rover | zbr: It was me i think | 12:52 |
| arxcruz|rover | and you approve it | 12:52 |
| zbr | ovbiously that there is no proper validation of the file on CI, one that confirms that conditions put there make sense. | 12:53 |
| arxcruz|rover | i mean, i add it in the tox.ini | 12:53 |
| zbr | arxcruz|rover: not trying to blame someone ;), please use pattern like above and pick latest < 2.0 for python 2.7 | 12:53 |
| zbr | this should sort it | 12:53 |
| chandankumar | arxcruz|rover, weshay|ruck https://review.opendev.org/#/c/711599/ | 12:54 |
| panda | zbr: I remember adding <=2.0.1 | 12:54 |
| arxcruz|rover | in tox.ini is <= 2.0.1 | 12:54 |
| zbr | panda: that was correct, but someone added it to openstack constrains, which.... you know. | 12:54 |
| arxcruz|rover | https://github.com/openstack/tripleo-quickstart-extras/blob/master/tox.ini#L62 | 12:54 |
| zbr | not a bad idea to control versioning with constraints, but not with worng ones :D | 12:54 |
| *** matbu has quit IRC | 13:01 | |
| panda | zbr: you have a patch somewhere ? | 13:03 |
| zbr | nope | 13:04 |
| marios | chandankumar: rlandy: weshay|ruck: did we hit that one already anywhere else "/usr/bin/env: ‘python’: No such file or directory" (TASK [tripleo-inventory : Get IP of extra node in OVB stack) | 13:05 |
| weshay|ruck | panda, you're patch there.. was struck from the glory of merging right before we got to the finish line :( | 13:05 |
| marios | i didn't see it in notes https://hackmd.io/HrQd03c9SxOMtFPFrq50tg?both | 13:05 |
| weshay|ruck | marios, ya.. let's look that bit of the workflow is changing a lot on the tripleo side | 13:05 |
| weshay|ruck | via sbakers changes | 13:05 |
| rlandy | ? | 13:06 |
| marios | rlandy: weshay|ruck: k we talk in 10 mins anyway lets discuss then | 13:06 |
| weshay|ruck | marios, centos-8? | 13:06 |
| weshay|ruck | aye | 13:06 |
| marios | weshay|ruck: yeah fs39 | 13:06 |
| marios | * https://logserver.rdoproject.org/96/25796/3/check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/e976e0d/job-output.txt | 13:06 |
| weshay|ruck | marios, it's missing a flag in featureset probably | 13:07 |
| weshay|ruck | compare to fs001 | 13:07 |
| panda | weshay|ruck: all my patches are not getting any luck lately, but the upstream is in the gate again | 13:07 |
| weshay|ruck | k | 13:08 |
| weshay|ruck | marios, fyi.. the harndened overcloud image is failing.. we may have time now to look at that | 13:08 |
| zbr | panda: https://review.opendev.org/#/c/712942/ -- side effects of dropping py27 jobs too fast | 13:09 |
| zbr | i should ask infra about it, makes no sense for this purpose. | 13:09 |
| zbr | especially as we do have py27 conditions inside the constraints file. | 13:10 |
| zbr | panda: lets switch discussion to #openstack-requirements channel. | 13:12 |
| *** matbu has joined #oooq | 13:12 | |
| weshay|ruck | rlandy, can you help me take some notes in the upcoming mtg please? | 13:13 |
| marios | https://review.rdoproject.org/r/#/c/25884/ | 13:19 |
| chandankumar | rlandy, https://review.rdoproject.org/r/#/c/25332/ | 13:30 |
| *** skramaja has quit IRC | 13:30 | |
| arxcruz|rover | akahat: patch should be to trigger the tripleo job, not the mistral jobs, none of those jobs on your dnm uses tempestconf... | 13:32 |
| *** Goneri has joined #oooq | 13:33 | |
| arxcruz|rover | chandankumar: kopecmartin tosky I don't want to be pickly regarding the patch, so feel free to +w | 13:33 |
| tosky | arxcruz|rover: chandankumar already +w | 13:33 |
| arxcruz|rover | tosky: he -w | 13:33 |
| arxcruz|rover | because of my comment | 13:33 |
| tosky | I need to increase my refresh frequency | 13:33 |
| *** TrevorV has joined #oooq | 13:35 | |
| *** jpena|lunch is now known as jpena | 13:35 | |
| kopecmartin | arxcruz|rover: sorry, but what depends-on are we waiting for? | 13:36 |
| arxcruz|rover | kopecmartin: nevermind, i +w | 13:41 |
| arxcruz|rover | done | 13:41 |
| arxcruz|rover | boom! | 13:41 |
| *** ykarel|afk is now known as ykarel | 13:55 | |
| *** amoralej is now known as amoralej|lunch | 13:55 | |
| rlandy | marios: sorry - didn't get a chance to review your patches yesterday (due to hair-pulling over component pipeline). still in need of reviews? | 14:02 |
| marios | rlandy: np yeah still have some in progress scen10 and fs39... i think fs39 needs some fixing first tho | 14:02 |
| marios | rlandy: scen10 https://review.opendev.org/711507 https://review.rdoproject.org/r/25745 https://review.rdoproject.org/r/25746 https://review.rdoproject.org/r/25747 | 14:03 |
| rlandy | k- looking at those now | 14:03 |
| marios | rlandy: thanks | 14:03 |
| *** jmasud has quit IRC | 14:08 | |
| *** jmasud has joined #oooq | 14:09 | |
| rlandy | marios: https://review.opendev.org/#/c/711507/3 - some comments on placement. not major | 14:09 |
| rlandy | https://review.rdoproject.org/r/#/c/25745/ - looks goood | 14:10 |
| rlandy | good | 14:10 |
| rlandy | marios: ^^ ok to merge? | 14:10 |
| rlandy | whatever - going | 14:11 |
| marios | rlandy: yeah i think so its a non voting job and we want it in there even as non voting... needs further work (os-tempest) but it comes later | 14:11 |
| marios | rlandy: thanks for comments will check | 14:11 |
| rlandy | marios: last two look good - will merge when depends on does | 14:13 |
| marios | rlandy: thank you | 14:13 |
| *** jmasud has quit IRC | 14:13 | |
| *** pojadhav is now known as pojadhav|afk | 14:19 | |
| weshay|ruck | rlandy, network promoted today | 14:26 |
| weshay|ruck | :) | 14:26 |
| weshay|ruck | was failing | 14:26 |
| rlandy | weshay|ruck: nice - first pass on that component in a while | 14:28 |
| rlandy | panda_foreva/ panda_rulz/ ... lol | 14:29 |
| rlandy | component dirs | 14:30 |
| rlandy | panda: ^^ why are there no top-level equivalent dirs? | 14:30 |
| rlandy | oh nvm - I see them | 14:31 |
| rlandy | refresh | 14:31 |
| rlandy | weshay|ruck: which one first? centos8 baremetal or image build? | 14:31 |
| rlandy | maybe baremetal - could be possible | 14:32 |
| weshay|ruck | rlandy, baremetal please | 14:33 |
| weshay|ruck | it's been hard.. we can take that over once you have it stood up and failing in ironic | 14:33 |
| rlandy | weshay|ruck: ack - need to update the undercloud box first | 14:33 |
| rlandy | reprovision | 14:33 |
| *** chem has quit IRC | 14:44 | |
| *** chem has joined #oooq | 14:46 | |
| *** amoralej|lunch is now known as amoralej | 14:55 | |
| *** jmasud has joined #oooq | 15:06 | |
| rlandy | ugh - downstream zuul looks like a disaster area | 15:08 |
| * panda ready to skydive .... | 15:09 | |
| panda | skydiving in 5 .... 4 .... | 15:15 |
| panda | 3 and 1/2 | 15:15 |
| panda | 3 and 1/4 | 15:15 |
| panda | 2 and 5/4 | 15:17 |
| panda | skydive merge! | 15:19 |
| panda | marios: starting manual tests | 15:19 |
| akahat | arxcruz|rover, just saw your ping.. How to enable those jobs? | 15:22 |
| panda | marios: too soon, still failing with skydive | 15:24 |
| marios | panda: /o\ | 15:31 |
| panda | marios: trying again | 15:31 |
| *** ccamacho has quit IRC | 15:32 | |
| panda | marios: nope. still getting skydive .... | 15:33 |
| panda | marios: I'll check the repo | 15:33 |
| panda | why do I have opendev in chinese ? | 15:36 |
| *** bogdando has quit IRC | 15:40 | |
| panda | ok.. this is weird, I'm still getting skydive containers | 15:41 |
| ykarel | me to seeing it in Chinese :) | 15:41 |
| ykarel | in firefox | 15:41 |
| panda | yep | 15:41 |
| panda | that it's not my setting | 15:41 |
| panda | then* | 15:42 |
| ykarel | panda, where u getting skydive? | 15:42 |
| ykarel | in some of my testing i was getting it during container image prepare | 15:42 |
| ykarel | with correct env pass it was skipped | 15:42 |
| marios | weshay|ruck: rlandy: chandankumar: need that for fs39 when you get chance review please https://review.opendev.org/#/c/712962/ | 15:44 |
| rlandy | ack | 15:45 |
| panda | ykarel: the promoter does not use skiplist | 15:45 |
| panda | ykarel: we are getting the list unfiltered | 15:45 |
| panda | ykarel: I removed them from tripleo-common, but .... | 15:45 |
| panda | oh | 15:45 |
| ykarel | ohhk so it's different, need changes in tripleo-common | 15:45 |
| panda | I understand why | 15:45 |
| panda | ykarel: we have the change | 15:46 |
| ykarel | okk | 15:46 |
| panda | ykarel: the problem is that when we have a good candidate, the tripleo commit value is tied to that commit | 15:46 |
| ykarel | hmm i recall that place | 15:47 |
| panda | weshay|ruck: marios we would need to wait for a candidate hash that has the new tripleo-common package built and promoted with it .... | 15:47 |
| panda | weshay|ruck: I'm going to monkey patch the promoter at this point | 15:47 |
| panda | to see a full run in the promoter | 15:48 |
| *** ykarel is now known as ykarel|away | 15:48 | |
| panda | hacky hacky, patchy patchy, typy typy. | 15:49 |
| *** jbadiapa has joined #oooq | 16:05 | |
| weshay|ruck | panda, \0/ | 16:18 |
| weshay|ruck | panda, https://review.opendev.org/#/c/712517/ merged | 16:19 |
| weshay|ruck | ok re: ^ | 16:19 |
| weshay|ruck | cloudops and octavia promoted \0/ rlandy | 16:21 |
| weshay|ruck | sorry.. network and ocatavia | 16:22 |
| weshay|ruck | ugh.. nevermind | 16:22 |
| weshay|ruck | ignore me | 16:22 |
| rlandy | weshay|ruck: np - always good to celebrate | 16:24 |
| rlandy | env A undercloud reprovisioned for centso8 | 16:24 |
| marios | rlandy: what about that one we seen it elsewhere? 2020-03-13 15:43:01.262473 | primary | 2020-03-13 15:43:01 | + sudo dnf -y install epel-release https://logserver.rdoproject.org/96/25796/4/check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/aa00f0e/job-output.txt | 16:40 |
| marios | firstly why/what is installing epel-release and secondly wtf "sudo: dnf: command not found" centos8? | 16:40 |
| marios | rlandy: i am digging there just asking incase it is another one we encountered (don't see on the pad) | 16:40 |
| rlandy | marios: first time I've seen that | 16:51 |
| rlandy | will keep an eye | 16:51 |
| marios | rlandy: no worries, i will dig into it more on monday (its fs39, we didn't even merge yet just seeing it at testproject... links there fyi https://hackmd.io/HrQd03c9SxOMtFPFrq50tg?both#OVB-Jobs | 16:52 |
| marios | weshay|ruck: fyi fs39 uncovering a few new issues like that https://review.opendev.org/#/c/712962/ and now another one "sudo: dnf: command not found" will check that next week | 16:53 |
| marios | weshay|ruck: the errors coming from supp node/setup free ipa server tasks i guess they just aren't running anywhere else | 16:53 |
| *** jmasud has quit IRC | 16:53 | |
| panda | marios: weshay|ruck promotion failed. Authentication problems. mmmhh | 16:55 |
| weshay|ruck | panda, to docker.io? | 16:55 |
| weshay|ruck | panda, is the push w/ docker or buildah? | 16:55 |
| *** jmasud has joined #oooq | 16:56 | |
| panda | promoter failed: [localhost] (item=[u'7077c3a7b926d71502959cfce192abdf', u'aodh-api']) => {"ansible_loop_var": "item", "changed": false, "item": ["7077c3a7b926d71502959cfce192abdf", "aodh-api"], "msg": "Error pushing image docker.io/tripleo/centos-binary-aodh-api: denied: requested access to the resource is denied"} | 16:56 |
| panda | mmmhh | 16:56 |
| panda | release is missing | 16:56 |
| weshay|ruck | ah ya | 16:56 |
| panda | so no credential problem | 16:56 |
| weshay|ruck | :) | 16:56 |
| weshay|ruck | easy | 16:56 |
| panda | weshay|ruck: rerunning, but did you hear what happened with skydive ? | 17:01 |
| weshay|ruck | panda, what do you mean? | 17:01 |
| weshay|ruck | w/ skydive | 17:01 |
| weshay|ruck | hasn't been supported in a long time afaik | 17:02 |
| panda | weshay|ruck: ok, so the promoter doesn't download the containers list from tripleo-common master. | 17:02 |
| weshay|ruck | what what? | 17:02 |
| panda | weshay|ruck: it gets the comit hash of the package in the versions.csv of the fil | 17:02 |
| weshay|ruck | it used to pick the branch and hash I think | 17:02 |
| weshay|ruck | ya | 17:02 |
| weshay|ruck | this was more of an issue when the containers were changing a lot | 17:03 |
| panda | weshay|ruck: ok, so, a commit with the change will be available to the promoter only when tripleo-common is packaged with that patch, and that package reaches the promotion state | 17:03 |
| weshay|ruck | and we had to make sure we pulled the right commit hash | 17:03 |
| weshay|ruck | ah | 17:03 |
| panda | weshay|ruck: so right now I patched the promoter on the fly to exclude skydive | 17:04 |
| weshay|ruck | panda, so it should be in current soon.. if not already | 17:04 |
| weshay|ruck | ++ | 17:04 |
| * weshay|ruck checks the status | 17:04 | |
| panda | weshay|ruck: I'm backporting to the code the other changes instead | 17:04 |
| panda | weshay|ruck: like the fix for the release you just saw | 17:04 |
| marios | getting ready to go rlandy weshay|ruck will pickup from https://hackmd.io/7MBqFHurTA2e5H8kYRwgag?both#UNPINNING-NOTES--Component-pipeline on monday | 17:05 |
| weshay|ruck | panda, it's there https://trunk.rdoproject.org/centos8-master/component/tripleo/current/ | 17:05 |
| weshay|ruck | so we're close :) | 17:05 |
| rlandy | marios: thanks - see you monday | 17:05 |
| marios | panda: why don't you just apply the current patches and let it run over weekend | 17:05 |
| marios | isnt that what was discussed earlier | 17:05 |
| weshay|ruck | panda, and it's also in consistent https://trunk.rdoproject.org/centos8-master/component/tripleo/consistent/openstack-tripleo-common-12.1.1-0.20200313152816.4cc4647.el8.noarch.rpm | 17:05 |
| weshay|ruck | panda, so now I need to get it into promoted-components | 17:06 |
| weshay|ruck | panda, want me to wait for the tripleo comp jobs.. or you want it NOW | 17:06 |
| weshay|ruck | which is fine... | 17:06 |
| weshay|ruck | either way | 17:06 |
| *** tesseract has quit IRC | 17:07 | |
| weshay|ruck | promoting tripleo should be a no-op | 17:07 |
| weshay|ruck | but it's going to run very soon | 17:07 |
| weshay|ruck | panda, any preference? | 17:07 |
| weshay|ruck | hello | 17:08 |
| weshay|ruck | is it me your looking for | 17:08 |
| weshay|ruck | I can see it your eyes | 17:08 |
| weshay|ruck | I can see it in your smile | 17:08 |
| weshay|ruck | you're all I ever wanted | 17:08 |
| marios | i think that's my queue to go | 17:08 |
| weshay|ruck | marios, 0/ | 17:08 |
| marios | cue even? | 17:08 |
| weshay|ruck | you're all I'm looking for | 17:08 |
| weshay|ruck | marios, come on you love it | 17:09 |
| marios | problem is when i read this, i can hear you singing it in your voice in my head | 17:09 |
| marios | not pleasant | 17:09 |
| weshay|ruck | lolz | 17:09 |
| weshay|ruck | this manager works in mysterious ways | 17:09 |
| weshay|ruck | how do you manager your people wes.. | 17:09 |
| weshay|ruck | I rickroll them | 17:10 |
| weshay|ruck | genius I say | 17:10 |
| panda | marios: I want to see a full run | 17:10 |
| panda | marios: before I make it roll over the weekend | 17:10 |
| zbr | marios: weshay|ruck panda : easy https://review.rdoproject.org/r/#/c/25897/ | 17:10 |
| panda | marios: problem with variables will not fix themselves | 17:10 |
| marios | panda: yeah but sounds like monkey patching has been problematic so... | 17:10 |
| marios | panda: if were just working around an issue that will resolve itself once we get a new enough candidate | 17:11 |
| panda | marios: I'm only monkey patching to implement the skiplist for the containers | 17:11 |
| panda | marios: the reast I'm porting to the code | 17:11 |
| marios | panda: ok... i thought the issue you were facing (eg. with registry) was to do with your workaround for skydive perhaps not.. | 17:12 |
| panda | unfortunately there are problems in the tests, sometimes the test reuse the same code to test itself, which is not good, because a single error goes unnoticed | 17:12 |
| panda | weshay|ruck: I don't need the package now, it's moving forward with the monkey patch | 17:12 |
| panda | weshay|ruck: when it will be ready I'll remove it, or leave it, it doesn't matter | 17:13 |
| *** marios is now known as marios|out | 17:13 | |
| panda | marios|out: adios, marios | 17:13 |
| weshay|ruck | panda, k k | 17:14 |
| weshay|ruck | I suspect it will be in tripleo-ci-testing in a few hours.. I'll watch it today | 17:14 |
| weshay|ruck | panda, thanks brotha | 17:14 |
| weshay|ruck | rlandy, V | 17:21 |
| weshay|ruck | https://review.rdoproject.org/r/25898 | 17:21 |
| rlandy | weshay|ruck: ? | 17:21 |
| weshay|ruck | sorry.. need a review on https://review.rdoproject.org/r/25898 | 17:21 |
| *** dsneddon has quit IRC | 17:22 | |
| *** marios|out has quit IRC | 17:29 | |
| *** dpawlik has quit IRC | 17:30 | |
| *** TrevorV has quit IRC | 17:34 | |
| zbr | weshay|ruck: give me a hand with https://review.rdoproject.org/r/#/c/25897/ i do not want to end-up self-merging | 17:36 |
| *** sshnaidm|afk has quit IRC | 17:37 | |
| *** hamzy has joined #oooq | 17:45 | |
| rlandy | security should promote | 17:49 |
| rlandy | weshay|ruck: ^^ | 17:50 |
| rlandy | weshay|ruck: a bunch of changes to get dlrn/reparenting into downstream zuul :( | 17:50 |
| *** udesale_ has joined #oooq | 17:52 | |
| *** udesale has quit IRC | 17:54 | |
| *** derekh has quit IRC | 18:01 | |
| *** jpena is now known as jpena|off | 18:07 | |
| chandankumar | rlandy, anything needed from my side? | 18:11 |
| zbr | weshay|ruck: can you explain the reason for removal of tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-ovb-master on https://review.rdoproject.org/r/#/c/25898/3/zuul.d/ovb-jobs.yaml ? | 18:13 |
| chandankumar | see ya people, have a nice weekend ahead | 18:14 |
| *** chandankumar is now known as raukadah | 18:14 | |
| *** sanjayu_ has quit IRC | 18:15 | |
| panda | weshay|ruck: failed again on multiarch manifests, forgot to set to false, rerunning disabling manifest this time | 18:20 |
| panda | next optinmixzation for the promoter, is to restart from wher it left .... | 18:21 |
| panda | I already have the code, but it's not integrated nor tested ... | 18:21 |
| zbr | rlandy or panda: another one https://review.rdoproject.org/r/#/c/25899/ | 18:28 |
| *** udesale_ has quit IRC | 18:39 | |
| weshay|ruck | hrm.. tripleo-ci-testing ended up w/ a different hash than what we tagged. | 18:39 |
| weshay|ruck | interesting | 18:39 |
| rlandy | raukadah: working downstream for now- thanks | 18:39 |
| rlandy | weshay|ruck: zbr: ok to merge https://review.rdoproject.org/r/#/c/25899/? | 18:40 |
| * weshay|ruck looks | 18:40 | |
| raukadah | weshay|ruck, rlandy please have a look at this issue https://55d217e0e7dc5c9a38a1-bf339ca8b211499ccf5ec38f530d7d51.ssl.cf1.rackcdn.com/683126/31/check/tobiko-tripleo-multinode-containers/8eb4c4c/logs/undercloud/home/zuul/overcloud_deploy.log | 18:41 |
| zbr | yep | 18:41 |
| raukadah | me and marios are hitting the same it will be a legit issue | 18:41 |
| raukadah | weshay|ruck, rlandy also seen in fs030 also | 18:41 |
| rlandy | ok | 18:43 |
| rlandy | error pulling image "192.168.24.1:8787/tripleomaster/centos-binary-mariadb:3621159be13b41f8ead2e873b357f4a5-updated-20200313141128": unable to pull 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:3621159be13b41f8ead2e873b357f4a5-updated-20200313141128: unable to pull image: Error initializing source docker://192.168.24.1:8787/tripleomaster/centos-binary-mariadb:3621159be13b41f8ead2e873b357f4a5-updated-20200313141128: error | 18:44 |
| rlandy | pinging docker registry 192.168.24.1:8787: Get https://192.168.24.1:8787/v2/: http: server gave HTTP response to HTTPS client | 18:44 |
| *** jbadiapa has quit IRC | 18:44 | |
| weshay|ruck | rlandy, ping https://opendev.org/openstack/tripleo-ci/commit/b2e7a0138c45b5f98629350e9c9b5bbb3ac29361 | 18:49 |
| weshay|ruck | rlandy, where do you see your error? | 18:49 |
| weshay|ruck | oh | 18:49 |
| rlandy | weshay|ruck: looking | 18:51 |
| rlandy | weshay|ruck: where do I see my error? | 18:52 |
| weshay|ruck | rlandy, I have 8 min until another call.. I can show you later if you want..https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master | 18:52 |
| weshay|ruck | 2020-03-13 16:28:55.352658 | TASK [build-containers : Get contents of openstack repo baseurl for the version hash] | 18:52 |
| weshay|ruck | 2020-03-13 16:28:55.768873 | primary | 02d1c6c338c0c88157def8c27214ebf4 | 18:52 |
| weshay|ruck | 2020-03-13 16:29:06.152606 | primary | ok: Runtime: 0:00:00.010211 | 18:52 |
| weshay|ruck | from https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/46bf9b0/job-output.txt | 18:53 |
| weshay|ruck | https://trunk.rdoproject.org/centos8-master/tripleo-ci-testing/delorean.repo.md5 | 18:53 |
| rlandy | weshay|ruck: not followinhg | 18:53 |
| weshay|ruck | rlandy, question is this... | 18:53 |
| rlandy | ping me when you are done | 18:53 |
| weshay|ruck | 2020-03-13 16:28:54.004250 | TASK [build-containers : Check the contents of the openstack_repo_name repo] | 18:53 |
| weshay|ruck | 2020-03-13 16:28:54.924713 | primary | 15 | 18:53 |
| weshay|ruck | 2020-03-13 16:28:55.055427 | primary | ok: Runtime: 0:00:00.024034 | 18:53 |
| weshay|ruck | primary | 15 | 18:53 |
| * weshay|ruck wonders if that's the registered result of line 33 register: number_baseurl_lines | 18:54 | |
| weshay|ruck | https://opendev.org/openstack/tripleo-ci/blame/branch/master/roles/build-containers/tasks/main.yaml#L33 | 18:54 |
| weshay|ruck | rlandy, this may be a timing issue | 18:54 |
| weshay|ruck | perhaps the repo was not fully built | 18:55 |
| rlandy | well is this always off? | 18:55 |
| weshay|ruck | ah.. looking | 18:55 |
| rlandy | promoted_components could be changing while we are runinning promote-to-tripleo-ci-testing | 18:56 |
| weshay|ruck | ya.. last job was 15 too | 18:56 |
| weshay|ruck | rlandy, ya.. but it shouldn't be.. because the job is a dep | 18:56 |
| weshay|ruck | anyhoo.. the md5sum we pushed as containers was different that tripleo-ci-testing/md5sum | 18:57 |
| rlandy | weshay|ruck: ok - looking at this - but not totally in context | 18:57 |
| weshay|ruck | first time I've seen that happen | 18:57 |
| rlandy | ping me when you are done with meeting | 18:57 |
| weshay|ruck | k | 18:57 |
| rlandy | and we can check it | 18:57 |
| rlandy | "aggregate_hash": "9b22b1ffd6520b8c5737f58f87c3fb44", | 18:58 |
| rlandy | 9b22b1ffd6520b8c5737f58f87c3fb44 | 18:59 |
| rlandy | export FULL_HASH="9b22b1ffd6520b8c5737f58f87c3fb44" | 18:59 |
| rlandy | weshay|ruck: ^^ all consistent | 18:59 |
| rlandy | what am I missing? | 18:59 |
| weshay|ruck | rlandy, https://review.rdoproject.org/zuul/build/46bf9b0a3e374740a76fc474f3808f87/log/job-output.txt#3554 | 19:00 |
| weshay|ruck | it pushed w/ :02d1c6c338c0c88157def8c27214ebf4 | 19:01 |
| rlandy | containers are wrong | 19:01 |
| rlandy | now I know which job | 19:01 |
| panda | weshay|ruck: rlandy have you pushed all the components as the starting hash ? | 19:02 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/46bf9b0/logs/hash_info.sh | 19:02 |
| rlandy | hash_info is right | 19:02 |
| weshay|ruck | panda, containers yes | 19:02 |
| weshay|ruck | rlandy, ok.. no one is calling so let's chat | 19:02 |
| rlandy | ok | 19:02 |
| weshay|ruck | https://meet.google.com/esb-xwnc-nth?authuser=1 | 19:03 |
| weshay|ruck | panda, we've pushed containers .. see the current-tripleo md5.sum | 19:04 |
| *** amoralej is now known as amoralej|off | 19:06 | |
| weshay|ruck | rlandy, https://projects.engineering.redhat.com/browse/OSPPRODCH-102 | 19:20 |
| weshay|ruck | https://tree.taiga.io/project/morucci-software-factory/us/3405 | 19:21 |
| panda | weshay|ruck: full run completed | 19:37 |
| weshay|ruck | panda, ROCK AND ROLL | 19:37 |
| weshay|ruck | panda, well done | 19:37 |
| panda | 2020-03-13 19:18:26,947 13798 INFO promoter Summary: Promoted 1 hashes this round | 19:37 |
| panda | 2020-03-13 19:18:26,947 13798 INFO promoter ------- -------- Promoter terminated normally | 19:37 |
| panda | I'll start the service | 19:37 |
| panda | so it will run all wekend | 19:37 |
| weshay|ruck | panda, https://hub.docker.com/layers/tripleomaster/centos-binary-openstack-base/panda-man/images/sha256-bee605cfcff28f97f9658645c14b46874fb1ffd3bcd637e4405a40ccb35b7ad4?context=explore | 19:38 |
| panda | weshay|ruck: looks good ? | 19:38 |
| rlandy | weshay|ruck: https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/46bf9b0/etc/yum.repos.d/delorean.repo | 19:40 |
| panda | weshay|ruck: service in the promoter started., https://review.rdoproject.org/r/24774 updated with latest fixes and https://review.rdoproject.org/r/25782 rebased. | 19:43 |
| * panda out, have a nice weekend | 19:44 | |
| weshay|ruck | rlandy, http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8080/rdo/rhel8-master/current-tripleo/delorean.repo | 19:56 |
| weshay|ruck | rlandy, http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8080/rdo | 19:57 |
| weshay|ruck | rlandy, 2020-03-13 17:28:48.735483 | primary | included: /var/lib/zuul/builds/39f286d473b744bdbd45736f90cc8d59/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/configure-mirrors/tasks/mirror.yaml | 20:00 |
| weshay|ruck | rlandy, 2020-03-13 17:33:45.484392 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:21): source /etc/ci/mirror_info.sh | 20:00 |
| weshay|ruck | rlandy, 2020-03-13 17:33:16.960497 | RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master] | 20:01 |
| *** dsneddon has joined #oooq | 20:34 | |
| *** jfrancoa|babysit has quit IRC | 20:43 | |
| rlandy | weshay|ruck: no good ... | 20:50 |
| rlandy | 2020-03-13 20:48:32.069253 | TASK [multi-node-firewall : Ensure iptables] | 20:50 |
| rlandy | 2020-03-13 20:48:35.946912 | primary | ERROR | 20:50 |
| rlandy | 2020-03-13 20:48:35.947509 | primary | { | 20:50 |
| rlandy | 2020-03-13 20:48:35.947576 | primary | "msg": "Failed to download metadata for repo 'AppStream'", | 20:50 |
| weshay|ruck | hrm.. | 20:51 |
| rlandy | we will have to sed the delroean file | 20:51 |
| weshay|ruck | rlandy, pick it up monday.. | 20:52 |
| rlandy | weshay|ruck: ack .. will think on it | 20:52 |
| rlandy | https://review.rdoproject.org/zuul/stream/39f95704467949ee8fefaad163a2e5e9?logfile=console.log did ok though | 20:53 |
| rlandy | 2020-03-13 20:51:22.196418 | primary | + DISTRO_MIRROR_HOST=http://mirror.centos.org | 20:53 |
| rlandy | 2020-03-13 20:51:22.196472 | primary | + RDO_MIRROR_HOST=https://trunk.rdoproject.org | 20:53 |
| *** jmasud has quit IRC | 20:53 | |
| *** jmasud has joined #oooq | 20:55 | |
| rlandy | weshay|ruck: actually I take that back | 21:00 |
| rlandy | job reran | 21:00 |
| *** rlandy has quit IRC | 21:21 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!