| *** dsneddon_ has quit IRC | 00:04 | |
| *** tosky has quit IRC | 00:46 | |
| *** d0ugal has quit IRC | 01:06 | |
| *** d0ugal has joined #oooq | 01:08 | |
| *** dsneddon_ has joined #oooq | 02:00 | |
| *** dsneddon_ has quit IRC | 02:05 | |
| *** caowei99_ has joined #oooq | 03:06 | |
| *** ykarel has joined #oooq | 03:15 | |
| *** dsneddon_ has joined #oooq | 04:01 | |
| *** udesale has joined #oooq | 04:02 | |
| *** dsneddon_ has quit IRC | 04:05 | |
| *** bhagyashris has joined #oooq | 04:20 | |
| *** epoojad1 has joined #oooq | 04:23 | |
| *** bhagyashris has quit IRC | 04:45 | |
| *** epoojad1 has quit IRC | 04:48 | |
| *** skramaja has joined #oooq | 04:58 | |
| *** caowei99_ has quit IRC | 05:01 | |
| *** bhagyashris has joined #oooq | 05:03 | |
| *** dtrainor has joined #oooq | 05:10 | |
| *** epoojad1 has joined #oooq | 05:10 | |
| *** ratailor has joined #oooq | 05:23 | |
| *** bhagyashris has quit IRC | 05:27 | |
| *** bhagyashris has joined #oooq | 05:31 | |
| *** raukadah is now known as chandankumar | 05:44 | |
| *** kopecmartin has joined #oooq | 05:57 | |
| *** dsneddon_ has joined #oooq | 06:02 | |
| *** caowei99_ has joined #oooq | 06:02 | |
| *** marios has joined #oooq | 06:05 | |
| *** dsneddon_ has quit IRC | 06:06 | |
| *** jtomasek has joined #oooq | 06:26 | |
| *** yolanda has quit IRC | 06:45 | |
| *** bhagyashris has quit IRC | 06:45 | |
| *** akahat has joined #oooq | 06:45 | |
| *** ykarel is now known as ykarel|afk | 06:46 | |
| *** ksambor has joined #oooq | 06:51 | |
| *** bhagyashris has joined #oooq | 06:52 | |
| *** jfrancoa has joined #oooq | 06:54 | |
| *** ykarel|afk is now known as ykarel|lunch | 06:55 | |
| *** apetrich has joined #oooq | 07:34 | |
| *** jschlueter has quit IRC | 07:37 | |
| *** jbadiapa has joined #oooq | 07:38 | |
| *** jschlueter has joined #oooq | 07:38 | |
| *** dsneddon_ has joined #oooq | 08:03 | |
| *** dsneddon_ has quit IRC | 08:08 | |
| *** tesseract has joined #oooq | 08:15 | |
| *** tosky has joined #oooq | 08:20 | |
| *** bogdando has joined #oooq | 08:27 | |
| *** saneax has joined #oooq | 08:28 | |
| *** ratailor_ has joined #oooq | 08:40 | |
| *** dtantsur has joined #oooq | 08:41 | |
| *** ratailor has quit IRC | 08:42 | |
| *** jaosorior has joined #oooq | 08:48 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** ykarel|lunch is now known as ykarel | 09:06 | |
| *** yolanda has joined #oooq | 09:22 | |
| *** chem has joined #oooq | 09:38 | |
| *** ratailor__ has joined #oooq | 09:41 | |
| *** ratailor_ has quit IRC | 09:43 | |
| *** dsneddon_ has joined #oooq | 10:04 | |
| *** dsneddon_ has quit IRC | 10:08 | |
| *** bogdando has quit IRC | 10:09 | |
| *** bogdando has joined #oooq | 10:09 | |
| *** bhagyashris has quit IRC | 10:10 | |
| *** bhagyashris has joined #oooq | 10:12 | |
| bogdando | hi folks. Care to place +2 https://review.opendev.org/#/c/687288/? | 10:12 |
|---|---|---|
| bogdando | that should speedup all standalone and UC CI jobs somewhat... | 10:13 |
| bogdando | weshay|ruck: ^^ | 10:13 |
| bogdando | marios: ^^ | 10:14 |
| marios | bogdando: ack i was planning to do so on my next visit but was also hoping for some more (better ;)) reviews there first but sure its on my reviews list. | 10:16 |
| *** jschlueter has quit IRC | 10:17 | |
| *** jschlueter has joined #oooq | 10:19 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 10:20 | |
| sshnaidm|ruck | bogdando, a dummy question, why don't we use podman to pull/push images? | 10:21 |
| bogdando | sshnaidm|ruck: it cannot be an image serve registry | 10:22 |
| bogdando | it would only pull for its local cache | 10:22 |
| sshnaidm|ruck | bogdando, why don't we use quay then for image registry? | 10:23 |
| bogdando | there is a skopeo uploader alternative, but it is slow, not ready for use yet, and will be removed AFAIK... | 10:23 |
| bogdando | sshnaidm|ruck: python uploader uses buildah though | 10:23 |
| bogdando | when running with modify_image | 10:23 |
| bogdando | sshnaidm|ruck: that was not a thing historically by the time we developed that custom image-server server | 10:24 |
| bogdando | (meaning Quay) | 10:24 |
| sshnaidm|ruck | bogdando, I see | 10:24 |
| bogdando | and AFAIK it has not standalone deployment option | 10:24 |
| bogdando | but, there is a scenario that illustrate pull & re-push, which technically allows using another target | 10:25 |
| bogdando | like quay or local image-serve | 10:25 |
| bogdando | I composed scenarios to represent all possible use cases, and improvement is shown for all | 10:25 |
| sshnaidm|ruck | bogdando, there is also docker registry container available | 10:25 |
| bogdando | as it does not repeat processing of cross-used layers | 10:26 |
| bogdando | sshnaidm|ruck: right, but we cannot use that in product | 10:27 |
| bogdando | I was using it only to show an alternative target registry | 10:27 |
| *** ykarel_ has joined #oooq | 10:28 | |
| bogdando | even if we switch it for quauy in future releases, there is LTS'es and that image-serve will stay with us now for years | 10:28 |
| bogdando | I'm not really asking you folks to go deep into that, just see the numbers in benchmarks and think of real aspects in CI jobs | 10:29 |
| sshnaidm|ruck | bogdando, yeah, I think we can try | 10:30 |
| bogdando | thanks | 10:30 |
| *** ykarel has quit IRC | 10:30 | |
| *** sanjayu_ has joined #oooq | 10:51 | |
| *** saneax has quit IRC | 10:53 | |
| *** ykarel_ is now known as ykarel | 10:56 | |
| *** caowei99_ has quit IRC | 11:08 | |
| *** bogdando has quit IRC | 11:11 | |
| *** bogdando_ has joined #oooq | 11:12 | |
| *** bogdando_ is now known as bogdando | 11:14 | |
| *** udesale has quit IRC | 11:17 | |
| *** sanjayu__ has joined #oooq | 11:42 | |
| *** sanjayu_ has quit IRC | 11:44 | |
| *** rfolco_pto has joined #oooq | 11:50 | |
| marios | panda: o/ | 11:51 |
| panda | it's-a him, marios ! | 11:51 |
| marios | panda: are you around? can you please see that comment https://review.rdoproject.org/r/#/c/23037/9/ci-scripts/dlrnapi_promoter/tests/staging-setup/test_promoter_test.py | 11:51 |
| marios | panda: i am having some issues as that isn't 'normal' unit tests (mocking the mock kind of thing? i.e. not calling directly into the tested class but it uses a proxy class?) | 11:52 |
| marios | panda: i am just looking to test that one function in isolation but that isn't what any of the other tests in test_promoter_test.py are doing | 11:52 |
| marios | panda: any ideas or suggestions are welcome, either here or on the review whenyou have some time thanks | 11:54 |
| marios | panda: going to ask rlandy too once she is in as i know she wrote the test_promoter_test.py | 11:54 |
| panda | marios: test_promoter_test contains unit tests for the promoter test. You should not modify that, you can add a new test file using test_get_latest_hashes as inspiration | 11:54 |
| panda | marios: or e can rename test_get_latest_hashes to a more general test and add all them there | 11:55 |
| marios | panda: hmmm didn't spot that one looks promising/more what i want to do like 5 dlrnapi_promoter = pytest.importorskip('dlrnapi_promoter') | 11:56 |
| *** bhagyashris has quit IRC | 11:56 | |
| *** epoojad1 is now known as epoojad1|afk | 11:56 | |
| *** bhagyashris has joined #oooq | 11:57 | |
| *** rfolco_pto is now known as rfolco | 11:58 | |
| *** epoojad1|afk has quit IRC | 12:00 | |
| chandankumar | arxcruz: can we merge this https://review.rdoproject.org/r/#/c/23537/ ? | 12:02 |
| *** dsneddon_ has joined #oooq | 12:05 | |
| rfolco | marios, o/ | 12:07 |
| rfolco | marios, did anything change in the component standalone job ? | 12:07 |
| rfolco | or same plans | 12:07 |
| rfolco | panda, ^ | 12:08 |
| marios | rfolco: o/ we need to sync today | 12:08 |
| rfolco | yes we do | 12:09 |
| arxcruz | chandankumar: i don't see why not, i already put +1 there | 12:09 |
| marios | rfolco: i don't think panda is working with us anymore there. basically we don't want to even build containers anymore | 12:09 |
| rfolco | marios, upgrade ok | 12:09 |
| marios | rfolco: and instead use mofify_vars | 12:09 |
| marios | modify | 12:09 |
| *** dsneddon_ has quit IRC | 12:09 | |
| marios | rfolco: but i have mainly been catching up there and waiting for you as you said you were working there already | 12:10 |
| marios | rfolco: after scrum we can sync? | 12:10 |
| rfolco | marios, deal, the upgrade idea is new, I haven't started it | 12:10 |
| rfolco | marios, thx | 12:11 |
| marios | rfolco: ok, i got the impression it was your idea ;) (we discussed it on friday with weshay/chandan/mpryc a bit) | 12:11 |
| weshay|ruck | sec.. in interview | 12:11 |
| rfolco | marios, wes' idea | 12:11 |
| marios | rfolco: ack | 12:11 |
| marios | weshay|ruck: don't need something | 12:12 |
| *** ykarel is now known as ykarel|afk | 12:14 | |
| *** bhagyashris has quit IRC | 12:21 | |
| *** jpena is now known as jpena|lunch | 12:26 | |
| *** akahat has quit IRC | 12:30 | |
| sshnaidm|ruck | chandankumar, did you look at fs039 tempest failure? Do you know what is status of it? http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train/065dde0/logs/undercloud/home/zuul/tempest.log.txt.gz | 12:47 |
| *** ratailor__ has quit IRC | 12:51 | |
| *** dsneddon_ has joined #oooq | 12:53 | |
| *** rlandy has joined #oooq | 12:56 | |
| *** rlandy is now known as rlandy|rover | 12:56 | |
| rlandy|rover | sshnaidm|ruck: hi - is the ovn issue fixed now? | 12:59 |
| sshnaidm|ruck | rlandy|rover, yep | 12:59 |
| rlandy|rover | nice ok | 12:59 |
| rlandy|rover | so it was just fs020 that failed | 12:59 |
| rlandy|rover | on master | 12:59 |
| sshnaidm|ruck | rlandy|rover, yeah, rerunning it | 12:59 |
| sshnaidm|ruck | rlandy|rover, but we have a problem with train qcow2 images | 13:02 |
| rlandy|rover | sshnaidm|ruck: a problem on the server or with the job | 13:02 |
| rlandy|rover | it's happened before that the images were not there | 13:02 |
| chandankumar | sshnaidm|ruck: looking | 13:02 |
| rlandy|rover | did you try rerunning the job? | 13:02 |
| sshnaidm|ruck | rlandy|rover, that's what need to figure out, but no images on the server | 13:02 |
| sshnaidm|ruck | rlandy|rover, which job? | 13:02 |
| rlandy|rover | for centos images (not rhel images) fs002 | 13:03 |
| rlandy|rover | also stein phase 1 hasn't passed in ages | 13:03 |
| rlandy|rover | we keep rerunnig | 13:04 |
| sshnaidm|ruck | rlandy|rover, it passed | 13:04 |
| sshnaidm|ruck | rlandy|rover, but built image for different hash | 13:04 |
| rlandy|rover | sshnaidm|ruck: when you reran it? | 13:05 |
| sshnaidm|ruck | rlandy|rover, it uploaded to 4cb493fe4f79e736ae4fd7874b02d1d5ceed02c1_90d3ca61 while running for 267df13deb5eb15dcd31d42db8ae98132158cda8_6e9ed517 | 13:05 |
| sshnaidm|ruck | rlandy|rover, I didn't rerun anything | 13:05 |
| sshnaidm|ruck | rlandy|rover, http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-train-upload/e2607c6/job-output.txt | 13:06 |
| chandankumar | sshnaidm|ruck: rlandy|rover https://review.rdoproject.org/r/#/c/23537/ please it is good to go ahead | 13:07 |
| rlandy|rover | chandankumar: zbr has a -2 on that | 13:08 |
| chandankumar | rlandy|rover: It was got fixed | 13:08 |
| sshnaidm|ruck | chandankumar, do we have our jobs on this role or we're cores there? | 13:08 |
| rlandy|rover | sshnaidm|ruck: is this a once off or happened more than once? | 13:08 |
| sshnaidm|ruck | chandankumar, I don't think we're good to go before that | 13:08 |
| sshnaidm|ruck | rlandy|rover, idk, saw just last one, will look now.. | 13:09 |
| chandankumar | sshnaidm|ruck: there is no jobs added there, we can add it there | 13:09 |
| rlandy|rover | sshnaidm|ruck: if it just happened once, I would wait for the next run | 13:09 |
| chandankumar | sshnaidm|ruck: Do we need a standalone job or a job which will execise the rpm build? | 13:09 |
| sshnaidm|ruck | chandankumar, I'm strongly opposite using roles that 1) we don't gate them 2) we can't fix there quickly by ourselves only | 13:09 |
| chandankumar | using dlrn | 13:10 |
| sshnaidm|ruck | rlandy|rover, in previous job was correct, ran and built for 4cb493fe4f79e736ae4fd7874b02d1d5ceed02c1_90d3ca61 | 13:11 |
| sshnaidm|ruck | rlandy|rover, seems like in last job today it built images for previous hash..? ^^ | 13:11 |
| sshnaidm|ruck | weird | 13:12 |
| rlandy|rover | sshnaidm|ruck: could be a glitch | 13:12 |
| rlandy|rover | sshnaidm|ruck: how far out is the next build? | 13:12 |
| rlandy|rover | we can try rerun | 13:12 |
| rlandy|rover | there are no check jobs waiting on this | 13:12 |
| sshnaidm|ruck | rlandy|rover, train is running atm | 13:13 |
| rlandy|rover | sshnaidm|ruck: ok - so there is a job running now | 13:14 |
| rlandy|rover | that one uploaded the wrong hash? or the one before? | 13:14 |
| rlandy|rover | http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-train-upload/e2607c6/job-output.txt | 13:15 |
| rlandy|rover | nvm it's on that | 13:15 |
| rlandy|rover | 2019-11-18 13:14:46.285801 | primary | TASK [modify-image : Run virt-customize on the provided image] ***************** | 13:15 |
| rlandy|rover | 2019-11-18 13:14:46.285916 | primary | task path: /home/zuul/workspace/.quickstart/usr/local/share/ansible/roles/modify-image/tasks/libguestfs.yml:41 | 13:15 |
| sshnaidm|ruck | FULL_HASH=267df13deb5eb15dcd31d42db8ae98132158cda8_6e9ed517 | 13:16 |
| sshnaidm|ruck | rsync --verbose --archive --delay-updates --relative 4cb493fe4f79e736ae4fd7874b02d1d5ceed02c1_90d3ca61 uploader@images.rdoproject.org:/var/www/html/images/centos7/train/rdo_trunk | 13:16 |
| sshnaidm|ruck | "--relative" is for previous hash | 13:16 |
| sshnaidm|ruck | rlandy|rover, also "container_build_id": "4cb493fe4f79e736ae4fd7874b02d1d5ceed02c1_90d3ca61" | 13:17 |
| sshnaidm|ruck | rlandy|rover, damn.. http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-train-upload/e2607c6/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz | 13:18 |
| sshnaidm|ruck | rlandy|rover, what's going on there | 13:18 |
| rlandy|rover | http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8080/rdo/centos7-train/ | 13:19 |
| rlandy|rover | ^^ does that dir look different to you now? | 13:20 |
| rlandy|rover | [DIR]tripleo-ci-testing/2019-11-18 07:27 | 13:20 |
| *** dtantsur is now known as dtantsur|bbl | 13:21 | |
| rlandy|rover | http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8080/rdo/centos7-train/tripleo-ci-testing/delorean.repo | 13:21 |
| rlandy|rover | shoudl pick up that | 13:21 |
| sshnaidm|ruck | rlandy|rover, so maybe proxy delay? | 13:23 |
| *** epoojad1 has joined #oooq | 13:24 | |
| rlandy|rover | sshnaidm|ruck: weird this is right - http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-promote-consistent-to-tripleo-ci-testing/26c0950/job-output.txt | 13:24 |
| *** epoojad1 is now known as epoojad1|afk | 13:24 | |
| *** jpena|lunch is now known as jpena | 13:25 | |
| rlandy|rover | 019-11-18 10:27:54.948494 | primary | ++ shyaml get-value commits.0.commit_hash | 13:25 |
| rlandy|rover | 2019-11-18 10:27:54.990628 | primary | + COMMIT_HASH=7948344c88a6c73794bce813994028f69c4cd15b | 13:25 |
| rlandy|rover | 2019-11-18 10:27:54.990973 | primary | ++ shyaml get-value commits.0.distro_hash | 13:25 |
| rlandy|rover | 2019-11-18 10:27:55.031399 | primary | + DISTRO_HASH=6b2205004895998033175849344d2f5c3d5c8039 | 13:25 |
| rlandy|rover | 2019-11-18 10:27:55.031475 | primary | + FULL_HASH=7948344c88a6c73794bce813994028f69c4cd15b_6b220500 | 13:25 |
| rlandy|rover | 2019-11-18 10:27:55.031508 | primary | + cat | 13:25 |
| rlandy|rover | sshnaidm|ruck: ^^ that's the current fs002 run | 13:26 |
| sshnaidm|ruck | rlandy|rover, this looks correct | 13:26 |
| rlandy|rover | 2019-11-18 10:34:32.026923 | primary | TASK [container-prep : echo container_build_id] ******************************** | 13:29 |
| rlandy|rover | 2019-11-18 10:34:32.042615 | primary | Monday 18 November 2019 10:34:32 +0000 (0:00:00.071) 0:00:00.715 ******* | 13:29 |
| rlandy|rover | 2019-11-18 10:34:32.064282 | primary | ok: [undercloud] => { | 13:29 |
| rlandy|rover | 2019-11-18 10:34:32.064416 | primary | "container_build_id": "7948344c88a6c73794bce813994028f69c4cd15b_6b220500" | 13:29 |
| rlandy|rover | 2019-11-18 10:34:32.064445 | primary | } | 13:29 |
| rlandy|rover | sshnaidm|ruck: ^^ ok maybe it will correct itself in this run | 13:29 |
| sshnaidm|ruck | rlandy|rover, yep, rerunning rhel8-sc001 there, it failed | 13:30 |
| rlandy|rover | thanks | 13:34 |
| rlandy|rover | weshay|ruck: you can change your nick back | 13:34 |
| rlandy|rover | weshay|ruck: what's with the two cards you added in failing jobs? | 13:34 |
| rlandy|rover | sshnaidm|ruck: ^^ there are two bugs added there | 13:36 |
| rlandy|rover | not on our ruck rover entherpad | 13:36 |
| rlandy|rover | https://bugs.launchpad.net/tripleo/+bug/1852771 - says stein but the log linked is master | 13:36 |
| openstack | Launchpad bug 1852771 in tripleo "stein: tempest.api.compute.images.test_images.ImagesTestJSON fails w/ ailed to reach SAVING state (current state ACTIVE) within the required time (500 s)." [High,Triaged] | 13:36 |
| rlandy|rover | oh nvm - it is master | 13:37 |
| rlandy|rover | it is stein | 13:37 |
| sshnaidm|ruck | rlandy|rover, why do we do " copy overcloud-full.qcow2 to undercloud.qcow2]" still? | 13:39 |
| rlandy|rover | we need to get rid of fs002 altogether | 13:39 |
| rlandy|rover | and replace with image build/push | 13:39 |
| *** weshay|ruck is now known as weshay | 13:40 | |
| rlandy|rover | weshay: I'm closing https://bugs.launchpad.net/tripleo/+bug/1852771 - pls see comments - unless you object | 13:41 |
| openstack | Launchpad bug 1852771 in tripleo "stein: tempest.api.compute.images.test_images.ImagesTestJSON fails w/ ailed to reach SAVING state (current state ACTIVE) within the required time (500 s)." [High,Triaged] | 13:41 |
| weshay | has it run consistently? | 13:41 |
| rlandy|rover | weshay: passed the last four runs | 13:42 |
| rlandy|rover | 15, 16, 17, 18 | 13:42 |
| *** weshay has quit IRC | 13:46 | |
| *** epoojad1|afk has quit IRC | 13:48 | |
| *** Goneri has joined #oooq | 13:57 | |
| mjturek | we're getting this error during [build-containers: include_tasks] since about Friday. Anyone seen anything like this: "12:23:38 ERROR! Unexpected Exception, this is probably a bug: 'StrategyModule' object has no attribute '_cond_not_supported_warn'" | 13:59 |
| mjturek | https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/1757/consoleFull | 14:00 |
| mjturek | do I need to use a newer ansible? | 14:02 |
| *** rlandy|rover is now known as rlandy|rover|mtg | 14:20 | |
| mjturek | rfolco: any ideas? | 14:34 |
| *** chem has quit IRC | 14:35 | |
| rfolco | mjturek, let me see | 14:36 |
| *** chem has joined #oooq | 14:36 | |
| *** weshay has joined #oooq | 14:37 | |
| sshnaidm|ruck | mjturek, 2.5.8 is pretty old | 14:39 |
| mjturek | sshnaidm|ruck which do you guys use? | 14:40 |
| sshnaidm|ruck | mjturek, 2.8 | 14:40 |
| mjturek | sshnaidm|ruck do you pin to 2.8 or just grab whatever yum provides? | 14:40 |
| mjturek | pip* | 14:40 |
| sshnaidm|ruck | mjturek, we pin with pip, and it's also what yum provides now | 14:41 |
| mjturek | cool cool, think it could fix what we're seeing? | 14:41 |
| sshnaidm|ruck | mjturek, or discover another error :) | 14:42 |
| mjturek | lolol | 14:42 |
| mjturek | thanks sshnaidm|ruck, I'll submit a patdh now | 14:42 |
| rfolco | mjturek, backport to 2.6 exists | 14:43 |
| rfolco | https://github.com/ansible/ansible/issues/46275 | 14:43 |
| rfolco | mjturek, better to bump to 2.8 | 14:43 |
| mjturek | ahhh nice, thanks rfolco | 14:43 |
| mjturek | doing so now! | 14:43 |
| *** TrevorV has joined #oooq | 14:47 | |
| *** rlandy|rover|mtg is now known as rlandy|rover | 15:00 | |
| *** tesseract has quit IRC | 15:01 | |
| *** tesseract has joined #oooq | 15:01 | |
| rfolco | weshay, chandankumar pre planning has old bj only | 15:01 |
| *** dtantsur|bbl is now known as dtantsur | 15:01 | |
| rfolco | whats the gmeet for it | 15:01 |
| chandankumar | rfolco: we can create one? | 15:02 |
| chandankumar | rfolco: created | 15:02 |
| rfolco | or can use tripleo ci community call | 15:02 |
| rlandy|rover | sshnaidm|ruck: train images should be ok now | 15:02 |
| rfolco | ok | 15:02 |
| rfolco | chandankumar, gimme | 15:02 |
| chandankumar | rfolco: weshay https://meet.google.com/rnn-uvkn-rmk | 15:03 |
| sshnaidm|ruck | rlandy|rover, master 020 timed out.. | 15:04 |
| rlandy|rover | sshnaidm|ruck: we have a real problem there then | 15:04 |
| rlandy|rover | sshnaidm|ruck: do you have time to debug it or you want to look through the run times | 15:05 |
| sshnaidm|ruck | rlandy|rover, maybe to skip it for now | 15:05 |
| rlandy|rover | sshnaidm|ruck: idk - that's pretty fundamental | 15:05 |
| chandankumar | weshay: meeting time | 15:05 |
| sshnaidm|ruck | rlandy|rover, it's just timed out, nothing to debug.. | 15:05 |
| rlandy|rover | we could skip it for one promotion only | 15:05 |
| rlandy|rover | sshnaidm|ruck: ^^ it's running longer than it was | 15:06 |
| sshnaidm|ruck | rlandy|rover, periodic-tripleo-rhel-8-buildimage-overcloud-full-master just failed, so we have problem with current master run | 15:06 |
| rlandy|rover | sshnaidm|ruck: looking why it failed | 15:06 |
| sshnaidm|ruck | rlandy|rover, same as always | 15:06 |
| rlandy|rover | sshnaidm|ruck: just rerun | 15:07 |
| rlandy|rover | no new failure there | 15:07 |
| rlandy|rover | sshnaidm|ruck: ok - so let's skip fs020 for one promotion | 15:07 |
| rlandy|rover | then we need to get it back | 15:07 |
| rlandy|rover | just because we are five days out now | 15:07 |
| rlandy|rover | sshnaidm|ruck: ok? | 15:08 |
| sshnaidm|ruck | rlandy|rover, yeah, 6 days now | 15:08 |
| sshnaidm|ruck | rlandy|rover, yep | 15:08 |
| rlandy|rover | sshnaidm|ruck: fine - getting on promotion server to change criteria | 15:08 |
| rlandy|rover | I will change them back this afternoon | 15:08 |
| sshnaidm|ruck | rlandy|rover, on the server, right? | 15:08 |
| rlandy|rover | sshnaidm|ruck: yes - I'll do it | 15:08 |
| sshnaidm|ruck | rlandy|rover, ack, thanks | 15:08 |
| rlandy|rover | sshnaidm|ruck: also we need to rerun fs001 on rhel master after image build reruns | 15:09 |
| weshay | chandankumar, rfolco I need 3 min.. | 15:12 |
| chandankumar | weshay: ok | 15:13 |
| rlandy|rover | sshnaidm|ruck: ok damage is done - we should see master try to promote | 15:16 |
| *** dsneddon_ has quit IRC | 15:18 | |
| rlandy|rover | chandankumar: rfolco: weshay: can you schedule 5 mins at the next sprint close meeting so that we can tel the team about the new cix categories? | 15:21 |
| *** udesale has joined #oooq | 15:22 | |
| *** dsneddon_ has joined #oooq | 15:23 | |
| *** sanjayu__ has quit IRC | 15:28 | |
| *** dsneddon_ has quit IRC | 15:28 | |
| *** dsneddon_ has joined #oooq | 15:49 | |
| panda | where's marios ? | 15:50 |
| marios | panda: but if we put into the ini, then remove it from the role params? but what about statements like that https://review.rdoproject.org/r/#/c/23698/2/ci-scripts/container-push/roles/containers-promote/tasks/registry-push.yml@78 | 15:50 |
| rfolco | rlandy|rover, I will create the agenda, but marios will drive the retro for me | 15:50 |
| rfolco | chandankumar, fyi ^ | 15:51 |
| marios | panda: or do you mean have it in both the role, and in the ini? | 15:51 |
| marios | panda: (do you need something? 17:50 < panda> where's marios ? | 15:51 |
| panda | marios: the variable in the role is needed for the role. In the INI will only be a flag to activate it or not in the role | 15:51 |
| panda | marios: did you want to chat or it's too late ? | 15:52 |
| marios | panda: rather concentrate on that please for now cos is getting late, let me post it and we can talk on gerrit or tomorrow | 15:52 |
| marios | panda: or did you want to talk about something else? | 15:52 |
| panda | marios: fine, does my answer clarify ? | 15:52 |
| marios | panda: the py test thing ate my time today and i swore at/about you quite a lot today, fwiw :D | 15:52 |
| marios | panda: (you're welcome) | 15:53 |
| marios | panda: yeah so both 17:51 < marios> panda: or do you mean have it in both the role, and in the ini? | 15:53 |
| panda | marios: yes, but from now on it will take 30 seconds to verify that your functions work :) | 15:53 |
| panda | marios: yeah both | 15:53 |
| marios | panda: yeah its just a real royal pita trying to mock urlilb and readlines got stuck into a rabbit hole on that one (yeah yeah i know its worth it just frustrating not having somethign to show for it yet) | 15:54 |
| panda | marios: as I said, the dlrnapi_promoter calls "ansible-playbook container-push" in a subshell | 15:54 |
| marios | panda: but did you want me for something? 17:50 < panda> where's marios ? | 15:55 |
| panda | marios: so to not enable manifests, the dlrnapi_promoter needs to call "ansible-playbook -emanufest_push=false container-push" | 15:55 |
| panda | marios: and one way to tell dlrnapi_promoter to do this is to pass a flag in the .ini file | 15:55 |
| marios | panda: ack | 15:55 |
| *** dsneddon_ has quit IRC | 15:55 | |
| marios | panda: 17:55 < marios> panda: but did you want me for something? 17:50 < panda> where's marios ? | 15:55 |
| panda | marios: no, just wanted to see if you wanted to chat, and I did a m<tab> and didn't find your nick and I was worried something happened to you and omg what imma gonna do without you, oh gosh. (I typoed n<tab> BTW) | 15:56 |
| marios | panda: aww nice to know you were thinking of me today too :D (I was mainly swearing at you but it still counts i think?) | 15:57 |
| panda | marios: swearing is caring. Your father was right. | 15:59 |
| *** dsneddon_ has joined #oooq | 16:00 | |
| *** hamzy has quit IRC | 16:01 | |
| marios | panda: :D | 16:01 |
| *** udesale has quit IRC | 16:06 | |
| sshnaidm|ruck | rlandy|rover, I think build images issue fits well "Prodchain degrade" column, wdyt? | 16:07 |
| rfolco | marios, where is the ini file in https://review.rdoproject.org/r/#/c/23698/ | 16:08 |
| *** ykarel|afk is now known as ykarel | 16:08 | |
| marios | rfolco: updating now but i will add it to that one ./ci-scripts/dlrnapi_promoter/config/CentOS-7/staging.ini | 16:09 |
| marios | rfolco: that what you mean? | 16:09 |
| rfolco | marios, yes I just believe when I see it, thanks | 16:09 |
| rlandy|rover | sshnaidm|ruck: there is still a bug for image build? | 16:09 |
| marios | rfolco: :D why ? | 16:10 |
| marios | rfolco: i mean you have doubts? | 16:10 |
| sshnaidm|ruck | rlandy|rover, should be.. | 16:10 |
| rlandy|rover | sshnaidm|ruck: we closed the last one - we will have to open a new one | 16:10 |
| rfolco | marios, I am using your as base, so was reviewing yours first, just it. All good, still wip. | 16:11 |
| rfolco | marios, ping me when you submit the new one, so I can review w/ the ini idea | 16:11 |
| rfolco | ANEYE as panda says | 16:12 |
| panda | rfolco: I N I | 16:12 |
| marios | eye en eye | 16:12 |
| rfolco | :) | 16:12 |
| marios | panda: gonna get messy, either i make manifest_push a global, or we add it to all the methods. doing the 2nd way first we can discuss on review | 16:13 |
| panda | marios: yeah, unfortunately, we are pushing to change a very unmaintainable code, I hope this is the last change we need to do to the legacy before starting to fully use new tests. THen we can *slowly* improve the core logic, line by line, test by test, one slow line at a time. | 16:14 |
| panda | marios: I had a refactor active for that code 2 years ago, taht addressed scalability of configuration, but we didnt' have time to test it effectively | 16:17 |
| sshnaidm|ruck | rlandy|rover, opening.. | 16:19 |
| panda | marios: https://review.rdoproject.org/r/#/c/10209/ could be an interesting read if you have nothing else to do (LOL) | 16:19 |
| * marios eye starts twitching | 16:19 | |
| marios | panda: sure in a bit (tomorrow) | 16:20 |
| sshnaidm|ruck | rlandy|rover, launchpad is broken.. | 16:40 |
| sshnaidm|ruck | rlandy|rover, https://bugs.launchpad.net/tripleo/+bug/1853028 | 16:40 |
| openstack | Launchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged] | 16:40 |
| rlandy|rover | sshnaidm|ruck: launchpad seems to be working? | 16:43 |
| sshnaidm|ruck | rlandy|rover, yeah, finally.. | 16:43 |
| sshnaidm|ruck | tried 5 times to submit it | 16:43 |
| rlandy|rover | sshnaidm|ruck: k - thanks for opening that | 16:43 |
| *** skramaja has quit IRC | 16:43 | |
| rlandy|rover | we can leave some fun for the next ruck/rovers | 16:43 |
| weshay | rfolco, check ur email please | 16:44 |
| rfolco | weshay, checking | 16:45 |
| sshnaidm|ruck | rlandy|rover, yeah, let them enjoy it too | 16:45 |
| rfolco | weshay, I have answered | 16:46 |
| rfolco | capacity form | 16:46 |
| *** bogdando has quit IRC | 16:47 | |
| weshay | rfolco, k.. you may want to double check that stuart got it | 16:48 |
| *** marios is now known as marios|out | 16:50 | |
| *** hamzy has joined #oooq | 16:51 | |
| *** marios|out has quit IRC | 16:54 | |
| rfolco | weshay, what is a good update job that I can base on for the component testing, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates ? Can I create a base standalone-scenario000 to parent my component job off it ? | 17:21 |
| *** hamzy has quit IRC | 17:21 | |
| weshay | oh lord no | 17:22 |
| *** hamzy has joined #oooq | 17:22 | |
| weshay | rfolco, jump on a meet w/ me.. and let me explain how it works | 17:22 |
| rfolco | ok | 17:22 |
| weshay | rfolco, https://meet.google.com/nzh-dyzx-mmi | 17:23 |
| weshay | marios is gone | 17:23 |
| weshay | anyone else want to know how containers get updated? | 17:23 |
| weshay | rfolco, https://9722deaa313ebebb56dc-c08b881decb3106ff13d720dd4a26025.ssl.cf5.rackcdn.com/694307/3/check/tripleo-ci-centos-7-standalone/c3dcd4c/logs/delorean_logs/07/80/0780b668b0439d7874455fa92050c9b2b09e824f_dev/rpmbuild.log.txt.gz | 17:28 |
| weshay | INFO: Start(/home/zuul/DLRN/data/repos/07/80/0780b668b0439d7874455fa92050c9b2b09e824f_dev/puppet-tripleo-11.3.1-0.20191116195643.0780b66.el7.src.rpm) Config(dlrn-centos7-x86_64-1) | 17:29 |
| *** dsneddon_ has quit IRC | 17:32 | |
| *** dsneddon_ has joined #oooq | 17:34 | |
| sshnaidm|ruck | rlandy|rover, hmm.. I see promotion script for train didn't run in last 2 hours | 17:37 |
| rlandy|rover | sshnaidm|ruck: checking | 17:37 |
| rlandy|rover | sshnaidm|ruck: there was an issue on friday when this happened | 17:38 |
| rlandy|rover | there was an error of a test added twice | 17:38 |
| sshnaidm|ruck | rlandy|rover, what was it? | 17:38 |
| sshnaidm|ruck | rlandy|rover, test of what? | 17:38 |
| rlandy|rover | fs050 singlenode | 17:38 |
| rlandy|rover | you can usually see the error in the logs | 17:39 |
| rlandy|rover | the service logs that is | 17:39 |
| *** dsneddon_ has quit IRC | 17:39 | |
| rlandy|rover | I am on the server now - checking | 17:39 |
| sshnaidm|ruck | and why is there a lot of tail processes owned by 'ssbarnea' :) zbr|ooo ? | 17:39 |
| rlandy|rover | no idea | 17:40 |
| rlandy|rover | do you have the train.ini file open> | 17:40 |
| rlandy|rover | sshnaidm|ruck: it look like it's busy promoting elsewhere | 17:41 |
| sshnaidm|ruck | rlandy|rover, master is promoting | 17:41 |
| rlandy|rover | 2019-11-18 15:13:32,471 19847 ERROR promoter ERROR: The promoted hash was not found: https://images.rdoproject.org/centos7/train/rdo_trunk/267df13deb5eb15dcd31d42db8ae98132158cda8_6e9ed517 | 17:42 |
| rlandy|rover | ERROR: /home/centos/ci-config/ci-scripts/promote-images.sh failed with 3 exit code | 17:42 |
| sshnaidm|ruck | rlandy|rover, where is it from? | 17:43 |
| rlandy|rover | sshnaidm|ruck: so its probably been busy with master ... | 17:43 |
| rlandy|rover | 2019-11-18 14:19:35,889 19784 INFO promoter FINISHED promotion process | 17:43 |
| rlandy|rover | 2019-11-18 16:45:15,422 31247 INFO promoter STARTED promotion process for release: master | 17:43 |
| sshnaidm|ruck | rlandy|rover, yeah, it was the problem with images | 17:43 |
| sshnaidm|ruck | rlandy|rover, ok, let's wait a little | 17:44 |
| rlandy|rover | before that ... | 17:44 |
| sshnaidm|ruck | rlandy|rover, anyway I rerun periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train , it's only failed | 17:44 |
| sshnaidm|ruck | rlandy|rover, https://review.rdoproject.org/r/#/c/23626/ | 17:45 |
| rlandy|rover | sshnaidm|ruck: before that it was promoting train rhel - 8 | 17:45 |
| rlandy|rover | 2019-11-18 16:35:13,800 26120 INFO promoter SUCCESS \0/ promoting (u'rhel', u'8') tripleo-ci-testing as current-tripleo ({'timestamp': 1574068331, 'distro_hash': '6b2205004895998033175849344d2f5c3d5c8039', 'promote_name': 'tripleo-ci-testing', 'user': 'review_rdoproject_org', 'repo_url': 'http://trunk.rdoproject.org/rhel8-train/79/48/7948344c88a6c73794bce813994028f69c4cd15b_6b220500', 'full_hash': | 17:45 |
| weshay | rfolco, https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/standalone/defaults/main.yml#L20 | 17:46 |
| rlandy|rover | periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train - ok we need to look into that | 17:46 |
| weshay | rfolco, https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-7/ | 17:46 |
| rlandy|rover | just updating the nodes in downstream - will get to it | 17:46 |
| *** jpena is now known as jpena|off | 17:47 | |
| weshay | rfolco, QUICKSTART_DEFAULT_RELEASE_ARG | 17:50 |
| *** dtantsur is now known as dtantsur|afk | 18:03 | |
| sshnaidm|ruck | rlandy|rover, out for today | 18:03 |
| rlandy|rover | sshnaidm|ruck: k - see you tomorrow | 18:05 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 18:05 | |
| *** dsneddon_ has joined #oooq | 18:11 | |
| *** ykarel is now known as ykarel|away | 18:15 | |
| chandankumar | rfolco: rlandy|rover no problem :-) | 18:15 |
| *** ykarel|away has quit IRC | 18:20 | |
| rlandy|rover | lunch - brb | 18:30 |
| *** weshay has quit IRC | 18:37 | |
| *** gouthamr_ is now known as gouthamr | 18:51 | |
| *** tesseract has quit IRC | 18:52 | |
| rlandy|rover | "http://38.145.34.141/rcm-guest/images/redhat8/master/rdo_trunk/tripleo-ci-testing/overcloud-full.tar.md5" | 18:55 |
| rlandy|rover | we have an image problem | 18:56 |
| *** weshay has joined #oooq | 19:00 | |
| weshay | \0/ | 19:02 |
| weshay | hooray | 19:02 |
| weshay | getting better at this znc bouncer thang | 19:02 |
| *** tosky has quit IRC | 19:34 | |
| *** dsneddon_ has quit IRC | 19:35 | |
| *** dsneddon_ has joined #oooq | 19:40 | |
| *** dsneddon_ has quit IRC | 19:48 | |
| *** chem has quit IRC | 19:50 | |
| *** dsneddon_ has joined #oooq | 19:50 | |
| *** chem has joined #oooq | 19:53 | |
| *** dsneddon_ has quit IRC | 19:55 | |
| rlandy|rover | weshay: http://rhos-release.virt.bos.redhat.com:3030/rhosp shows rocky and queens are three days out ... afaict from the logs, http://38.145.34.55/centos7_rocky.log, and the build, all test passed so does that mean that has just not been a new build to promote? | 19:56 |
| weshay | rfolco, https://review.opendev.org/#/c/694836/1/config/release/tripleo-ci/CentOS-7/master-component.yml | 19:56 |
| weshay | rfolco, I thought there was going to be a release file per component per branch | 19:57 |
| weshay | rlandy|rover, looking | 19:57 |
| weshay | rlandy|rover, we should adjust how often those go red now imho | 19:57 |
| weshay | they only run every 3 days or so right? | 19:58 |
| rfolco | weshay, I think we can use same base one, customize per component job | 19:58 |
| weshay | rlandy|rover, so three days in the board, and the last run was green can be true at the same time | 19:58 |
| weshay | rfolco, k.. cool | 19:58 |
| rlandy|rover | weshay: k - thanks - just confirming | 19:59 |
| weshay | rfolco, and the base url will be overwritten as well? | 19:59 |
| weshay | rfolco, may want to include some code comments :) | 19:59 |
| rfolco | https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-7/master.yml#L41 | 19:59 |
| weshay | rfolco, that just returns a hash | 20:01 |
| rlandy|rover | "http://38.145.34.141/rcm-guest/images/redhat8/master/rdo_trunk/tripleo-ci-testing/overcloud-full.tar.md5" is passing now - yay | 20:02 |
| weshay | rlandy|rover, so the dep error was just our image being 8.0 eh? | 20:03 |
| rlandy|rover | weshay: indeterminate | 20:04 |
| rlandy|rover | now AlistairT reported the same issue | 20:04 |
| rlandy|rover | I may have closed that bug prematurely | 20:05 |
| *** dsneddon_ has joined #oooq | 20:06 | |
| *** dsneddon_ has quit IRC | 20:11 | |
| *** jfrancoa has quit IRC | 20:14 | |
| mjturek | rfolco: so for some reason our job is suddenly trying to build with kolla instead of buildah | 20:32 |
| mjturek | any idea what might have changed that caused this? | 20:32 |
| mjturek | we set herehttps://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/tripleo-upstream-builders.yml#L25 | 20:33 |
| mjturek | https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/tripleo-upstream-builders.yml#L25 | 20:33 |
| rfolco | there is no kolla vs buildah | 20:37 |
| rfolco | its docker vs buildah | 20:38 |
| rfolco | always kolla | 20:38 |
| mjturek | gotcha | 20:39 |
| mjturek | anyways found it | 20:39 |
| mjturek | https://github.com/openstack/tripleo-ci/commit/94154098d7b4ffedd63c52927a8053f2a0cd6426#diff-2621bf872fd0d129f03ad1b941e1bc80 | 20:39 |
| mjturek | switched from common.yaml to main.yaml | 20:39 |
| mjturek | https://review.rdoproject.org/r/#/c/23750/ should fix it | 20:48 |
| *** dsneddon_ has joined #oooq | 20:50 | |
| *** dsneddon_ has quit IRC | 20:58 | |
| *** holser has joined #oooq | 21:00 | |
| *** dsneddon_ has joined #oooq | 21:04 | |
| *** holser has quit IRC | 21:06 | |
| rlandy|rover | woohoo master promoted | 21:06 |
| *** dsneddon_ has quit IRC | 21:08 | |
| weshay | rlandy|rover++ | 21:09 |
| weshay | so crisp and so green green | 21:09 |
| rlandy|rover | and train is promoting | 21:10 |
| rlandy|rover | weshay: going to move days to red on rocky and queens to longer | 21:10 |
| rlandy|rover | suggestion? | 21:10 |
| rlandy|rover | 6 days | 21:10 |
| rlandy|rover | means missed two promotions? | 21:10 |
| weshay | rlandy|rover, I think we get on the code.eng repo | 21:10 |
| rlandy|rover | or days to yellow rather | 21:10 |
| weshay | and change red to 14 days | 21:10 |
| rlandy|rover | weshay: ack | 21:10 |
| rlandy|rover | and yellow - 6? | 21:11 |
| weshay | sec | 21:11 |
| weshay | maybe | 21:11 |
| rlandy|rover | no rush - still working with downstream rhel 8 nodes | 21:14 |
| *** holser has joined #oooq | 21:20 | |
| weshay | rlandy|rover, https://code.engineering.redhat.com/gerrit/185675 | 21:30 |
| weshay | rlandy|rover, so that defines the when, then we'd have to replace isTooHighRDOCI w/ isTooHighOLDBRANCH_RDOCI on the branches that only run every three days | 21:31 |
| rlandy|rover | weshay: ack - in the same review or a different review? | 21:31 |
| weshay | rlandy|rover, want to see if they buy it first | 21:31 |
| rlandy|rover | they? | 21:32 |
| weshay | rel-del | 21:32 |
| rlandy|rover | weshay: why are we even tracking centos as critical path of rhel is closer to what run in osp? | 21:32 |
| *** holser has quit IRC | 21:32 | |
| weshay | rlandy|rover, actually for queens and rocky.. centos7 = rhel7 and tracks it just fine | 21:33 |
| weshay | are you referring to train / master? | 21:33 |
| rlandy|rover | ok - I mean for newer releases | 21:33 |
| rlandy|rover | clearly rocky rhel 8 never existed | 21:33 |
| *** dsneddon_ has joined #oooq | 21:40 | |
| weshay | rlandy|rover, my experiment w/ scenarios I think worked pretty well.. got ovb to step 4 ceph deployment | 21:43 |
| weshay | so needs more work.. but I think it will work out in the end | 21:43 |
| rlandy|rover | weshay: that's a big step forward | 21:44 |
| rlandy|rover | to streamlining testing | 21:44 |
| rlandy|rover | and having a consistent tracking of features | 21:44 |
| weshay | hopefully | 21:45 |
| weshay | rlandy|rover, https://review.rdoproject.org/r/#/c/23718/ | 21:50 |
| *** TrevorV has quit IRC | 21:51 | |
| rlandy|rover | hmmm - why were we running 39 there anyways? | 21:52 |
| weshay | rlandy|rover, because tht can break tls easily | 21:52 |
| weshay | but that job sucks | 21:52 |
| rlandy|rover | weshay: it's not so unstable in master | 21:52 |
| rlandy|rover | why not both? | 21:52 |
| weshay | tru | 21:52 |
| weshay | hopefully we'll be ditching 039 when ade has a new singlenode tls job up | 21:53 |
| rlandy|rover | really it's all semantics | 21:53 |
| rlandy|rover | nobody watches those failures anyways | 21:54 |
| weshay | ya.. how long have we been waiting for vex.. a little over a year now I think | 21:54 |
| weshay | rhel8 went beta in dec.. 2018 | 21:54 |
| weshay | so almost 1year for centos-8, and a little over a year for vex | 21:55 |
| rlandy|rover | considering the world is only going to last another 220-odd years ... how much progress can we make at this pace? | 21:58 |
| *** rfolco has quit IRC | 22:04 | |
| *** tosky has joined #oooq | 22:26 | |
| *** jaosorior has quit IRC | 23:21 | |
| rlandy|rover | nice - we got rhos-16 back | 23:49 |
| *** dsneddon_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!