| *** tosky has quit IRC | 00:04 | |
| *** jmasud has joined #oooq | 00:50 | |
| *** jmasud has quit IRC | 00:55 | |
| *** jmasud has joined #oooq | 01:58 | |
| *** jmasud has quit IRC | 02:04 | |
| *** jmasud has joined #oooq | 02:08 | |
| *** jmasud has quit IRC | 02:10 | |
| *** rfolco has joined #oooq | 03:11 | |
| *** rfolco has quit IRC | 03:32 | |
| *** ykarel has joined #oooq | 04:46 | |
| *** rlandy has quit IRC | 04:46 | |
| *** jmasud has joined #oooq | 05:08 | |
| *** udesale has joined #oooq | 05:22 | |
| *** ratailor has joined #oooq | 05:49 | |
| *** marios has joined #oooq | 06:11 | |
| zbr|rover | morning | 07:31 |
|---|---|---|
| zbr|rover | yesteday weshay|ruck posted a link to a container build failure, at https://logserver.rdoproject.org/69/31369/2/check/periodic-tripleo-ci-build-containers-ubi-8-push-train/4922e3c/logs/build.log -- but tbh I am unable to see what failed. | 07:32 |
| zbr|rover | akahat|ruck: maybe you can give me a hint as i am blind | 07:33 |
| marios | zbr|rover: https://logserver.rdoproject.org/69/31369/2/check/periodic-tripleo-ci-build-containers-ubi-8-push-train/4922e3c/logs/containers-build-errors.log ? | 07:42 |
| marios | zbr|rover: good morning | 07:43 |
| chandankumar | marios: zbr|rover Good morning! | 07:51 |
| marios | \o | 07:51 |
| zbr|rover | marios: thanks. i always forget our way of logging errors.... | 07:54 |
| marios | someone else want to vote there and we can merge? https://review.opendev.org/c/openstack/tripleo-ci/+/766978 ( weshay|ruck voted but he is author too... ) | 07:57 |
| marios | zbr|rover: np ... we all know what it's like staring at those logs | 07:57 |
| marios | zbr|rover: sometimes you miss the error literally in the middle of the screen in bold | 07:57 |
| *** amoralej|off is now known as amoralej | 08:07 | |
| *** tosky has joined #oooq | 08:33 | |
| *** jmasud has quit IRC | 08:35 | |
| *** jpena|off is now known as jpena | 08:56 | |
| *** derekh has joined #oooq | 09:18 | |
| *** jmasud has joined #oooq | 09:27 | |
| *** jmasud has quit IRC | 09:39 | |
| *** tosky_ has joined #oooq | 09:47 | |
| *** tosky is now known as Guest24372 | 09:49 | |
| *** tosky_ is now known as tosky | 09:49 | |
| *** Guest24372 has quit IRC | 09:50 | |
| zbr|rover | marios: my impression is that depends-on on tripleo-common failed to use code in the change. | 09:59 |
| zbr|rover | I see error: line 234 === exceptions='\n'.join(exceptions) | 09:59 |
| zbr|rover | https://review.rdoproject.org/zuul/build/b1be8fa45cb9471da628220eb5874f77/log/logs/containers-build-errors.log | 09:59 |
| *** dtantsur|afk is now known as dtantsur | 09:59 | |
| zbr|rover | guess, what, that join is not at that line, not even by far. | 10:00 |
| marios | zbr|rover: you should be able to see the build test packages output maybe in the job_output.txt so check if it was applied | 10:01 |
| zbr|rover | the entire exception does not make any sense as it was supposed to use code from https://review.opendev.org/c/openstack/tripleo-common/+/767168/3/tripleo_common/image/builder/buildah.py | 10:01 |
| zbr|rover | another repository w/o functional testing | 10:02 |
| zbr|rover | ffs, why we do not have any coverage using tox. | 10:03 |
| *** jmasud has joined #oooq | 10:53 | |
| *** slaweq has joined #oooq | 11:02 | |
| slaweq | chandankumar: hi | 11:02 |
| slaweq | chandankumar: I need some help with run of the os_tempest role locally | 11:02 |
| slaweq | I was trying to do that with https://gitlab.cee.redhat.com/chkumar/tripleo-bootcamp-tempest/blob/master/tempest_workflow.md#lets-try-os_tempest | 11:03 |
| slaweq | chandankumar: and I have error like http://paste.openstack.org/show/801084/ | 11:03 |
| slaweq | chandankumar: can You take a look what I am missing there? | 11:03 |
| slaweq | thx in advance | 11:03 |
| slaweq | chandankumar: ok, I found the problem | 11:16 |
| akahat|ruck | slaweq, o/ | 11:16 |
| akahat|ruck | slaweq, you are missing collections | 11:17 |
| slaweq | akahat|ruck: yes | 11:17 |
| slaweq | I just installed it | 11:17 |
| akahat|ruck | slaweq, https://opendev.org/openstack/ansible-collections-openstack/src/branch/master/README.md | 11:17 |
| slaweq | akahat|ruck: shouldn't it be added in https://github.com/openstack/openstack-ansible-os_tempest/blob/master/requirements.yml ? | 11:18 |
| slaweq | to be installed automatically | 11:18 |
| slaweq | I can propose patch if needed | 11:18 |
| akahat|ruck | slaweq, zbr|rover might have good answer about it. | 11:20 |
| *** udesale_ has joined #oooq | 11:24 | |
| *** udesale has quit IRC | 11:26 | |
| bhagyashris | marios, hi, i am trying to fixing the requirements issue on patch https://review.rdoproject.org/r/#/c/28014/ but it's getting time_out. | 11:29 |
| bhagyashris | marios, so what i observed in previous logs "ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts." | 11:31 |
| bhagyashris | https://logserver.rdoproject.org/14/28014/32/check/tox-py36-ci-config/79bdaa7/tox/py36-1.log | 11:31 |
| bhagyashris | and the current logs: https://logserver.rdoproject.org/14/28014/52/check/tox-py36-ci-config/abe1bd5/tox/py36-1.log | 11:32 |
| bhagyashris | zbr|rover, ^^ | 11:32 |
| marios | bhagyashris: ack looking | 11:33 |
| marios | bhagyashris: btw akahat|ruck filed a bug for that py36 job it is pretty broken https://bugs.launchpad.net/tripleo/+bug/1908274 | 11:33 |
| openstack | Launchpad bug 1908274 in tripleo "tox-py36-ci-config job is broken " [Critical,Triaged] | 11:33 |
| zbr|rover | bhagyashris: be sure you pre-install pip 20.3+ before install, as there were some versions that could produce very slow installs including endless loops | 11:33 |
| marios | bhagyashris: i am trying to update the requirements in https://review.rdoproject.org/r/#/c/31358/ (patch isn't related to requirements it is just blocking me from merging it so trying to see how far we can get there) | 11:34 |
| bhagyashris | marios, s/requirements/test-requirements | 11:34 |
| marios | bhagyashris: but you are hitting different issues there | 11:35 |
| marios | bhagyashris: well both | 11:35 |
| marios | bhagyashris: i mean ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/requirements.txt | 11:35 |
| marios | bhagyashris: https://review.rdoproject.org/r/#/c/31358/3/test-requirements.txt | 11:35 |
| akahat|ruck | zbr|rover, is it not installing using ensure_pip role? | 11:36 |
| *** holser has joined #oooq | 11:37 | |
| bhagyashris | marios, yes i saw your patch and now it's failing at pyyaml and if you unpinned that then it will raise time out this is what my experience is | 11:37 |
| marios | bhagyashris: ack i see... ok i just updated it for pyyaml now | 11:38 |
| marios | bhagyashris: will check once it reports and dig further | 11:38 |
| bhagyashris | marios, ack | 11:38 |
| akahat|ruck | marios, zbr|rover what about ensure pip role.. it's not included here: https://github.com/rdo-infra/ci-config/blob/master/playbooks/pre-run-docker.yml | 11:39 |
| akahat|ruck | if we include it will it resolve the issue? | 11:39 |
| akahat|ruck | bhagyashris, ^^ | 11:39 |
| marios | akahat|ruck: you mean we don't have new enough pip? | 11:39 |
| akahat|ruck | marios, as per errors that what i think.. otherwise we won't get that warning. | 11:40 |
| akahat|ruck | marios, zbr|rover> bhagyashris: be sure you pre-install pip 20.3+ before install, as there were some versions that could produce very slow installs including endless loops | 11:40 |
| marios | akahat|ruck: ack | 11:40 |
| zbr|rover | marios: i did not look at build logs, but if is about pip install, that is 99% the cause. | 11:41 |
| zbr|rover | some combinations of bad requirements produced endless loops with previous versions of pip, using 20.3+ should tell you the real issue. | 11:42 |
| marios | zbr|rover: sounds like what bhagyashris is seeing ^^^ | 11:42 |
| marios | 13:32 < bhagyashris> and the current logs: | 11:42 |
| marios | https://logserver.rdoproject.org/14/28014/52/check/tox-py36-ci-config/abe1bd5/tox/py36-1.log | 11:42 |
| *** matbu has quit IRC | 11:42 | |
| marios | INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking | 11:42 |
| marios | zbr|rover: ^ | 11:43 |
| zbr|rover | hmm,.. this looks differnt. | 11:43 |
| chandankumar | soniya29: akahat|ruck kopecmartin do we have any topics to disccuss for tempest meeting? or we can cancel it? | 11:43 |
| zbr|rover | pypa had some CDN issues yesterday but they are fixed now, | 11:44 |
| *** matbu has joined #oooq | 11:44 | |
| zbr|rover | recheck should work in that case. | 11:44 |
| akahat|ruck | bhagyashris, could you try to add ensure_pip role here: https://github.com/rdo-infra/ci-config/blob/master/playbooks/pre-run-docker.yml hoping to get different result here. | 11:44 |
| zbr|rover | thanks for the link, yep that is true: if you fail to mention any version ranges it could take a very long time to install dependencies as it requires to download them and check metadata on each of them. | 11:58 |
| zbr|rover | i read the they disables XMLRPC yesterday, which probably only made the search worse. | 11:59 |
| *** matbu has quit IRC | 11:59 | |
| zbr|rover | but someone having a pinned version of dependencies (like lock file) or decent ranges should not endup having problems. | 11:59 |
| * zbr|rover temporary out, digging in garage for an ancient invoice for my dead synology, at 5 degrees. | 12:00 | |
| *** rfolco has joined #oooq | 12:01 | |
| chandankumar | soniya29: akahat|ruck kopecmartin cancelling it then, since no topic | 12:01 |
| akahat|ruck | chandankumar, ack | 12:02 |
| marios | easy one needs votes please https://review.rdoproject.org/r/31370 | 12:10 |
| *** ratailor has quit IRC | 12:13 | |
| *** ykarel_ has joined #oooq | 12:14 | |
| marios | food biab | 12:15 |
| *** ykarel has quit IRC | 12:17 | |
| bhagyashris | akahat|ruck, zbr|rover marios ensure_pip is not working https://review.rdoproject.org/r/#/c/28014/54/playbooks/pre-run-docker.yml :( | 12:24 |
| weshay|ruck | akahat|ruck, mtg | 12:31 |
| akahat|ruck | weshay|ruck, joining. | 12:31 |
| *** jpena is now known as jpena|lunch | 12:33 | |
| weshay|ruck | akahat|ruck, zbr|rover https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/767225 https://review.opendev.org/c/openstack/tripleo-common/+/767264/1/tripleo_common/image/builder/buildah.py | 12:43 |
| weshay|ruck | zbr akahat|ruck, https://bugs.launchpad.net/tripleo/+bug/1908328 | 12:44 |
| openstack | Launchpad bug 1908328 in tripleo "tripleo-tox-molecule centos/7/storage/x86_64/ceph-octopus/repodata/repomd.xml: [Errno 14] HTTP Error 404" [Critical,Triaged] - Assigned to Sorin Sbarnea (ssbarnea) | 12:44 |
| *** ykarel_ is now known as ykarel | 12:54 | |
| marios | easy one needs votes please https://review.rdoproject.org/r/31370 sshnaidm weshay|ruck when you have time thanks | 13:01 |
| weshay|ruck | dang.. I don't have wf on rdo-jobs | 13:03 |
| weshay|ruck | +2 | 13:03 |
| rfolco | weshay|ruck, do you know why master container build log is different from train ? The train one is much more informative with return codes for each container | 13:04 |
| rfolco | CMD "sudo buildah bud ... returned: 0 in xxxs | 13:04 |
| rfolco | master build log >> https://logserver.rdoproject.org/40/29840/6/check/periodic-tripleo-ci-build-containers-ubi-8-push/fd29a5c/logs/build.log | 13:05 |
| rfolco | train >> https://logserver.rdoproject.org/36/31336/2/check/periodic-tripleo-ci-build-containers-ubi-8-push-train/39f2ec2/logs/build.log | 13:05 |
| weshay|ruck | rfolco, fyi https://review.opendev.org/c/openstack/tripleo-common/+/767264/1/tripleo_common/image/builder/buildah.py and look for patches from sorin | 13:06 |
| rfolco | weshay|ruck, hmmm maybe I am missing master patch, will look thx | 13:07 |
| *** rlandy has joined #oooq | 13:12 | |
| *** amoralej is now known as amoralej|lunch | 13:13 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| weshay|ruck | pojadhav, you overwrote my change https://review.rdoproject.org/r/#/c/30492/11/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf_py3/test_gerrit_changes.py | 13:33 |
| weshay|ruck | pojadhav, always clean your local.. and pull from remote.. before working in gerrit | 13:33 |
| weshay|ruck | pojadhav, https://review.rdoproject.org/r/#/c/30492/9..11/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf_py3/test_gerrit_changes.py | 13:34 |
| rlandy | weshay|ruck: zbr|rover: well there's good news and bad news - stream jobs are green again ... but still no logs | 13:37 |
| weshay|ruck | rlandy, k.. zbr|rover is focused on two issues.. atm the logs will have to wait or have someone else look | 13:37 |
| rlandy | k - I'll look at it | 13:37 |
| weshay|ruck | rlandy, so this worked? https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/766527/15/playbooks/tasks/tempest.yml | 13:38 |
| rlandy | weshay|ruck: no - seeing if we can ditch that review now | 13:39 |
| rlandy | rerunning in testproject | 13:39 |
| weshay|ruck | rlandy, ah k.. reading yatin's comment | 13:39 |
| weshay|ruck | k | 13:39 |
| weshay|ruck | rlandy, you should check for an ansible version bump | 13:39 |
| rlandy | weshay|ruck: also I'd like to get https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/765342 in | 13:40 |
| rlandy | will clean up the failure downstream as well | 13:40 |
| rlandy | much better pass rate | 13:40 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-ci/+/762766 we can merge if the pass rate is good enough | 13:41 |
| weshay|ruck | rlandy, k.. pinged lance and ade about it | 13:42 |
| chandankumar | rlandy: hello | 13:43 |
| chandankumar | rlandy: https://logserver.rdoproject.org/80/31380/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/d36197d/job-output.txt | 13:43 |
| rlandy | chandankumar: hi | 13:43 |
| chandankumar | rlandy: on updating the rpms also not sure why it is pulling old one https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/767340/1/roles/ipa-multinode/tasks/ipaserver-subnode-install.yml | 13:44 |
| rlandy | chandankumar: you mean it's still pulling the old ones or that is why you put up the patch? | 13:45 |
| marios | weshay|ruck: chandankumar: fyi regarding 'the pacemaker packages mismatch' https://bugs.launchpad.net/tripleo/+bug/1907769/comments/8 we have it in the component ussuri job now | 13:45 |
| openstack | Launchpad bug 1907769 in tripleo "standalone-upgrade-ussuri can't start containers during deployment container-puppet step2" [Critical,Fix released] | 13:45 |
| chandankumar | rlandy: yes it is still pulling the old one | 13:45 |
| marios | chandankumar: weshay|ruck: but train doesn't look like it will promote any time soon https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable3 :/ | 13:45 |
| rlandy | chandankumar: fs039 does not run that role | 13:45 |
| weshay|ruck | marios, ya.. that will happen in periodic component pipeline for two reasons... no content-providers right.. and ussuri container builds are blocked | 13:46 |
| rlandy | ipaserver-subnode-install.yml is only for the multinode job | 13:46 |
| weshay|ruck | marios, nothing we can do about it at this thime | 13:46 |
| weshay|ruck | time | 13:46 |
| marios | weshay|ruck: *train* is what we need in this case i.e. it needs the right containers during deployment but ack | 13:46 |
| weshay|ruck | train is also blocked on container builds | 13:46 |
| marios | ack | 13:47 |
| chandankumar | rlandy: I am trying to fix this issue https://bugs.launchpad.net/tripleo/+bug/1907651 | 13:47 |
| openstack | Launchpad bug 1907651 in tripleo "ovb featureset039 check & periodic fails during freeipa setup - could not open java-1.8.0" [Critical,Triaged] | 13:47 |
| weshay|ruck | so ya.. those component lines are just blocked .. :( | 13:47 |
| rlandy | chandankumar: the change you made is fine - and should work with the multinode job. for fs039 we need to fix that elsewhere | 13:47 |
| marios | weshay|ruck: np we don't have them in criteria yet anyway i am just working there https://review.rdoproject.org/r/#/c/31358/ and telling you cos you moved that bug to fix-released so fyi | 13:47 |
| * rlandy gets | 13:47 | |
| pojadhav | weshay|ruck, oops sorry :( | 13:48 |
| chandankumar | rlandy: ok | 13:48 |
| rlandy | reading bug | 13:48 |
| chandankumar | let me do a codesearch | 13:48 |
| weshay|ruck | pojadhav, ya.. we want to isolate the tests to a dir.. and the data you use to a dir in tests | 13:48 |
| weshay|ruck | pojadhav, make sense? | 13:48 |
| pojadhav | weshay|ruck, yup | 13:48 |
| weshay|ruck | other than the test class name.. it looked fine otherwise.. wasn't sure if you were still iterating on more tests... or moving on to a diff .py | 13:49 |
| weshay|ruck | one test is ok.. my preference is to see at least 3 for each.. but can merge that one soon I think | 13:49 |
| rlandy | chandankumar: was just doing that :) | 13:51 |
| rlandy | chandankumar: role freeipa-setup | 13:51 |
| pojadhav | weshay|ruck, okay.. will rework on the patch.. after taking your change | 13:51 |
| *** jmasud has quit IRC | 13:51 | |
| rlandy | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/freeipa-setup/README.md | 13:52 |
| chandankumar | rlandy: thanks! looking | 13:53 |
| rlandy | chandankumar: here it is ... https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/9fef1c0e42b5ee755196e08705f7368c1f2b5e1b/roles/freeipa-setup/tasks/main.yml | 13:53 |
| rlandy | ^^ missing change | 13:53 |
| chandankumar | rlandy: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/767340/1/roles/ipa-multinode/tasks/ipaserver-subnode-install.yml | 13:54 |
| rlandy | chandankumar: ^^ yep - voted on that change | 13:55 |
| rlandy | oops - wrong link | 13:55 |
| chandankumar | sorry again wrong | 13:56 |
| chandankumar | let me fix that | 13:56 |
| rlandy | chandankumar: lol - are we all set? | 13:56 |
| chandankumar | rlandy: yes | 13:57 |
| rlandy | there are two files with the same change | 13:57 |
| rlandy | chandankumar: ^^ agreed? | 13:57 |
| chandankumar | yes | 13:57 |
| rlandy | roles/freeipa-setup and roles/ipa-multinode | 13:57 |
| rlandy | ok - cool | 13:57 |
| chandankumar | rlandy: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/767340 done thanks! | 14:00 |
| rlandy | chandankumar: +2 - that should do it for both | 14:01 |
| *** amoralej|lunch is now known as amoralej | 14:06 | |
| akahat|ruck | marios, let | 14:12 |
| akahat|ruck | let' | 14:12 |
| akahat|ruck | let's add -v option here: https://github.com/rdo-infra/ci-config/blob/master/tox.ini#L11 | 14:12 |
| akahat|ruck | some verbose output might help. | 14:12 |
| pojadhav | weshay|ruck, we can add negative scenarios(tests) as well for gerrit_changes script. if you are ok.. I will work on it. | 14:13 |
| weshay|ruck | pojadhav, great | 14:15 |
| weshay|ruck | pojadhav, ya.. target atleast 3 for each script... and we'll iterate | 14:15 |
| pojadhav | weshay|ruck, yup :) will do | 14:15 |
| marios | akahat|ruck: ack good idea | 14:18 |
| marios | akahat|ruck: i am switching to something else right now but will check later unless you do it first ;) | 14:18 |
| akahat|ruck | marios, okay.. i'll push the change. | 14:18 |
| * pojadhav brb | 14:29 | |
| *** TrevorV has joined #oooq | 14:34 | |
| *** jbadiapa has joined #oooq | 14:48 | |
| zbr|rover | weshay|ruck: ok to +W https://review.opendev.org/c/openstack/tripleo-ci/+/767255 (see notes) | 14:50 |
| zbr|rover | is used in only two repos and we can fix if needed. | 14:50 |
| weshay|ruck | k. thanks | 14:51 |
| weshay|ruck | rlandy, https://review.opendev.org/c/openstack/tripleo-ci/+/767255 | 14:51 |
| zbr|rover | weshay|ruck: in fact i am testing it now, to update the in-repo job to use podman. | 14:56 |
| *** ykarel has quit IRC | 14:58 | |
| *** ykarel has joined #oooq | 14:58 | |
| rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/221454 Use container-tools rhel8 when building containers - we have an override in the jobs internally - trying rhel8 downstream | 15:02 |
| weshay|ruck | rlandy, yes let's test.. but hold off on merge.. or else we may break everything | 15:03 |
| * zbr|rover wonders if he is the only one that finds runc-1.0.0-68.rc92 ... a funny versins. Did this had 92 release candidates? | 15:10 | |
| rlandy | cp: cannot stat '/home/zuul/workspace/logs/undercloud/var/log/postci.txt.gz': No such file or directory | 15:12 |
| rlandy | collect logs failure | 15:12 |
| zbr|rover | weshay|ruck: i have podman issues... running as user on c8stream, as root it works, bu as non root I get: Error: could not get runtime: open /proc/1891371/ns/user: no such file or directory | 15:15 |
| zbr|rover | google did not help much | 15:15 |
| rlandy | zbr|rover: we had a bunch of issues with access in stream - when latest seems to work better .. see the links chandankumar put in https://bugs.launchpad.net/tripleo/+bug/1907474 | 15:26 |
| openstack | Launchpad bug 1907474 in tripleo "CentOS8-stream: tempest fails to load tests to be skipped: "The error was: KeyError: 'getpwuid(): uid not found: 1000'"" [High,Triaged] - Assigned to chandan kumar (chkumar246) | 15:26 |
| rlandy | looking at the log collection issue on stream | 15:27 |
| rlandy | may be similar | 15:27 |
| chandankumar | rlandy: during our rr, collect-logs was behaving weiredly | 15:29 |
| rlandy | chandankumar: consistent failure on stream | 15:32 |
| rlandy | no log collection on nodes | 15:32 |
| rlandy | ssh timing out | 15:32 |
| chandankumar | rlandy: may be hold the node and run there? | 15:33 |
| rlandy | marios: weshay|ruck: zbr|rover: can anyone actually asked infra re: permission to control job ordering the gate queue or are we just assuming they will say no? one or two people would get this permission | 15:36 |
| marios | rlandy: i don't know if this is possible | 15:36 |
| marios | rlandy: i can try and reach out to some folks if that is your request | 15:37 |
| chandankumar | rlandy: review link and job name? | 15:37 |
| rlandy | marios: as PTL can we put you up as a sacrifice to ask? | 15:37 |
| rlandy | thanks | 15:37 |
| rlandy | feel like we are tiptoeing around without even trying our luck | 15:37 |
| marios | rlandy: ack | 15:37 |
| rlandy | chandankumar: https://logserver.rdoproject.org/openstack-dependencies-centos8stream/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-centos8stream-container-build-master/4ff7bc4/logs/quickstart_collect_logs.log | 15:37 |
| rlandy | see error at the end | 15:37 |
| marios | rlandy: i really don't know/doubt it will be possible mainly because i'm guessing this is an automated thing | 15:37 |
| chandankumar | rlandy: need to fire a testproject | 15:38 |
| rlandy | it's software | 15:38 |
| marios | rlandy: i.e. things lined up in teh queue i don't know if it is possible to | 15:38 |
| marios | rlandy: yeah well we should find out anyway would be a useful thing if possible | 15:38 |
| chandankumar | let me push a patch | 15:38 |
| rlandy | everything is theoretically possible | 15:38 |
| rlandy | we can do it rdo | 15:38 |
| marios | rlandy: enough time & money yes :D | 15:38 |
| rlandy | marios: I'm paying at this point - you in? | 15:38 |
| marios | yolo! | 15:38 |
| * marios throws mastercard | 15:38 | |
| rlandy | how much is all our time worth?? | 15:38 |
| rlandy | chandankumar: for log collection? | 15:39 |
| rlandy | - can try it out in the testproject | 15:39 |
| chandankumar | rlandy: yes | 15:39 |
| chandankumar | rlandy: https://review.rdoproject.org/r/#/c/31387/ | 15:43 |
| chandankumar | rlandy: need pub key | 15:43 |
| zbr|rover | weshay|ruck: even after switching to a change on train branch I got the same error with https://logserver.rdoproject.org/69/31369/4/check/periodic-tripleo-ci-build-containers-ubi-8-push-train/68595bf/logs/containers-build-errors.log | 15:45 |
| weshay|ruck | rlandy, can we close the centos-stream boog? | 15:46 |
| zbr|rover | this may explain why our test changes are useless, as they do not test the change being made. | 15:46 |
| zbr|rover | and that is how alex was able to introduce that error. | 15:46 |
| rlandy | weshay|ruck: ack | 15:46 |
| rlandy | I commented | 15:46 |
| zbr|rover | it would be "fun" to see if his revert may fail due to the same reason. | 15:47 |
| rlandy | chandankumar: sec ... | 15:47 |
| rlandy | not periodic | 15:47 |
| rlandy | https://review.rdoproject.org/r/#/c/31387/ | 15:47 |
| rlandy | no depends on here | 15:47 |
| rlandy | job runs with tripleo-ci | 15:47 |
| rlandy | I mean current tripleo | 15:48 |
| rlandy | not tripleo-ci-testing | 15:48 |
| chandankumar | ok | 15:48 |
| *** ysandeep is now known as ysandeep|away | 15:48 | |
| rlandy | so how would periodic help with pub key? | 15:48 |
| rlandy | and why all of a sudden? | 15:49 |
| chandankumar | rlandy: done | 15:49 |
| chandankumar | rlandy: in order to log into hold node, we need pub key | 15:49 |
| rlandy | https://review.rdoproject.org/r/#/c/31165/ | 15:50 |
| rlandy | ^^ testproject been using | 15:50 |
| * rlandy compares with non-stream job | 15:51 | |
| chandankumar | rlandy: ok | 15:53 |
| rlandy | chandankumar: but holding the node would be helpful | 15:53 |
| rlandy | something is cooking here and I don't know what | 15:53 |
| chandankumar | rlandy: comparing with non-stream job is a good idea | 15:54 |
| chandankumar | rlandy: I am 99% sure, some python package is causing it | 15:54 |
| chandankumar | rlandy: what is the code of your little awesome tool which compares rpms? | 15:55 |
| chandankumar | *where | 15:55 |
| *** jpena is now known as jpena|off | 15:57 | |
| *** jpena|off is now known as jpena | 15:59 | |
| rlandy | chandankumar: it compares particular rpms | 16:02 |
| rlandy | weshay|ruck's tool compares all | 16:02 |
| rlandy | webtool | 16:02 |
| weshay|ruck | rlandy, it may have been busted by the recent change to zipping | 16:02 |
| weshay|ruck | chandankumar, I can help if you need direction | 16:03 |
| rlandy | chandankumar: the tool output is in the logs - the ones we don't collect :) | 16:03 |
| rlandy | chicken-egg problem | 16:03 |
| chandankumar | oh no | 16:03 |
| rlandy | if we get on the node, we will see it | 16:03 |
| * rlandy checks what we output to the console | 16:03 | |
| rlandy | sec | 16:03 |
| rlandy | chandankumar: we we do spew out the raw packages: https://logserver.rdoproject.org/65/31165/9/check/periodic-tripleo-ci-centos-8-standalone-centos8stream-container-build-master/60bea5b/job-output.txt | 16:10 |
| rlandy | ie: what's installed with stream and without | 16:10 |
| rlandy | really weshay|ruck's tool would help here | 16:10 |
| rlandy | weshay|ruck: ^^ I've changed my mind | 16:10 |
| rlandy | we do need a control job here | 16:10 |
| weshay|ruck | ping as needed, free | 16:11 |
| rlandy | weshay|ruck: either way we're messed up on this ... because no logs = no list to compare | 16:15 |
| weshay|ruck | rlandy, what's the url? | 16:15 |
| rlandy | https://logserver.rdoproject.org/65/31165/9/check/periodic-tripleo-ci-centos-8-standalone-centos8stream-container-build-master/60bea5b/job-output.txt | 16:15 |
| rlandy | https://logserver.rdoproject.org/65/31165/9/check/periodic-tripleo-ci-centos-8-standalone-centos8stream-container-build-master/60bea5b/logs/ | 16:16 |
| rlandy | no logs to get the lists to compare | 16:16 |
| weshay|ruck | ya.. if collect-logs fails.. we have no rpms to diff | 16:17 |
| rlandy | yep | 16:19 |
| rlandy | chicken-egg party | 16:19 |
| rlandy | so we'll have to scrape the screen output | 16:20 |
| rlandy | ouch | 16:20 |
| rlandy | chandankumar: so we have access to the node yet? | 16:33 |
| weshay|ruck | rfolco, 0/ | 16:44 |
| weshay|ruck | rfolco, I see barbican worker failing w/ 125 in https://logserver.rdoproject.org/36/31336/4/check/periodic-tripleo-ci-build-containers-ubi-8-push-train/5f60c6c/logs/build.log but not https://logserver.rdoproject.org/36/31336/4/check/periodic-tripleo-ci-build-containers-ubi-8-push-train/5f60c6c/logs/report.html | 16:45 |
| chandankumar | rlandy: let me know if you get the results. | 16:49 |
| weshay|ruck | rfolco, ping | 16:49 |
| rfolco | o/ | 16:52 |
| rfolco | weshay|ruck, this log with return codes is only for train | 16:53 |
| *** marios is now known as marios|out | 16:53 | |
| rfolco | weshay|ruck, mas does not have it | 16:53 |
| rfolco | master | 16:53 |
| rfolco | weshay|ruck, there is no CMD ... returned: 0 in XXXs in master logs | 16:54 |
| rfolco | https://logserver.rdoproject.org/40/29840/6/check/periodic-tripleo-ci-build-containers-ubi-8-push/fd29a5c/logs/build.log | 16:54 |
| rfolco | weshay|ruck, any implementation that relies in build log output will break if anything changes | 16:56 |
| *** udesale_ has quit IRC | 16:56 | |
| rfolco | weshay|ruck, I am stepping back on that "feature", buildah images <image> is the source of true for checking if the image built or not | 16:58 |
| *** marios|out has quit IRC | 17:01 | |
| weshay|ruck | rfolco, let's chat for a sec | 17:01 |
| weshay|ruck | https://meet.google.com/ytw-bffs-zgp?authuser=1 | 17:02 |
| rfolco | weshay|ruck joining | 17:02 |
| weshay|ruck | sorry was getting breakfast | 17:02 |
| *** jfrancoa has quit IRC | 17:03 | |
| rlandy | there are a million of these diffs | 17:05 |
| rlandy | chandankumar: ^^ :( | 17:05 |
| rlandy | maybe we narrow it down? | 17:05 |
| rlandy | python only? | 17:06 |
| chandankumar | yes | 17:06 |
| chandankumar | python only | 17:06 |
| *** ykarel has quit IRC | 17:09 | |
| rlandy | chandankumar: http://pastebin.test.redhat.com/926440 | 17:14 |
| chandankumar | rlandy: need one more filter, may be ansible deps? | 17:31 |
| rlandy | yep | 17:32 |
| rlandy | chandankumar: hmm - not easy to narrow down just ansible deps | 17:47 |
| rlandy | and no access to the node any more | 17:48 |
| rlandy | we didn't hold it? | 17:48 |
| *** chem has quit IRC | 17:54 | |
| chandankumar | rlandy: we hold it | 17:56 |
| chandankumar | rlandy: yes, we hold it | 17:56 |
| *** chem has joined #oooq | 17:56 | |
| rlandy | I lost access | 17:56 |
| chandankumar | same here | 17:57 |
| *** jmasud has joined #oooq | 17:59 | |
| chandankumar | rlandy: I am leaving now, poked nicholas on internal channel | 18:01 |
| chandankumar | he might look into that | 18:01 |
| rlandy | chandankumar: sure | 18:01 |
| chandankumar | see ya | 18:01 |
| rlandy | thanks for your help | 18:01 |
| weshay|ruck | rlandy, chandankumar https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/766987 | 18:01 |
| rlandy | voted | 18:02 |
| *** jpena is now known as jpena|off | 18:02 | |
| rlandy | maybe the miracle that cleaned up the delegate issue will also clean up the logs issue | 18:03 |
| chandankumar | rlandy: weshay|ruck periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master https://review.rdoproject.org/zuul/build/07f8f1cf9fb044c48865c3e5b1c9a06d : SUCCESS in 3h 08m 04s | 18:03 |
| chandankumar | rlandy: weshay|ruck https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/767340 | 18:04 |
| chandankumar | one bug down | 18:05 |
| *** amoralej is now known as amoralej|off | 18:09 | |
| rlandy | chandankumar: nice | 18:25 |
| rlandy | weshay|ruck: can we get rid of the container-tools 2.0 lock in master release file? | 18:25 |
| *** derekh has quit IRC | 18:26 | |
| weshay|ruck | ah fak.. that's why master is working | 18:26 |
| weshay|ruck | rlandy, does victoria have the same? | 18:27 |
| weshay|ruck | assuming ussuri and train do not | 18:27 |
| weshay|ruck | rlandy, don't remove anything please | 18:27 |
| rlandy | weshay|ruck: ack | 18:29 |
| rlandy | they all have it | 18:29 |
| * rlandy checks downstream job progress | 18:29 | |
| weshay|ruck | rlandy, train? | 18:30 |
| rlandy | ugh - still says 2.0 | 18:31 |
| rlandy | no - I tested 16.2 and 17 | 18:31 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/54/221454/1/check/tripleo-build-containers-ubi-8-internal-rhel-8-rhos-16.2/41e69a2/etc/dnf/modules.d/container-tools.module | 18:31 |
| rlandy | buildah.x86_64 1.11.6-8.module+el8.3.0+8233+627fbb78 @rhosp-rhel-8.3-appstream | 18:32 |
| weshay|ruck | rlandy, train? | 18:32 |
| weshay|ruck | not 16.2 | 18:32 |
| rlandy | weshay|ruck: train also has the lock | 18:32 |
| weshay|ruck | hrm | 18:32 |
| rlandy | container-tools rhel8 | 18:34 |
| weshay|ruck | rlandy, rhel8 does or does not equal 2.0 | 18:35 |
| rlandy | weshay|ruck: by default rhel8 should be container-tools:rhel8 | 18:35 |
| rlandy | unless we lock it | 18:35 |
| weshay|ruck | understood | 18:35 |
| weshay|ruck | which we are | 18:35 |
| weshay|ruck | rlandy, can you in a hackmd summarize the locks | 18:35 |
| weshay|ruck | for c8-train - master and rhel8 16.2 - 17 | 18:36 |
| rlandy | it seems like alex thinks train should still be locked | 18:36 |
| rlandy | correct | 18:36 |
| rlandy | and it does look like this got rhel8 | 18:36 |
| rlandy | dnf module -y enable container-tools:rhel8; dnf module -y enable mariadb:10.3; dnf module -y enable virt:8.3; dnf module -y enable redis:5; fi | 18:36 |
| rlandy | yep | 18:37 |
| rlandy | works now with rhel8 | 18:37 |
| weshay|ruck | rlandy, summarize in a hackmd please | 18:37 |
| weshay|ruck | per branch.. up and down | 18:37 |
| rlandy | weshay|ruck: let me confirm the base image build upstream first | 18:37 |
| rlandy | Enabling module streams: | 18:39 |
| rlandy | container-tools rhel8 | 18:39 |
| rlandy | we need ti change those defaults | 18:40 |
| rlandy | weshay|ruck: sorry - hackmd link again? | 18:42 |
| * rlandy closed it | 18:42 | |
| weshay|ruck | rlandy, I don't have one | 18:43 |
| rlandy | hmmm ... thought we were doc'ing this somewhere | 18:44 |
| weshay|ruck | please create a hackmd summary per branch and the container tools settings and what their values are | 18:44 |
| weshay|ruck | ¯\_(ツ)_/¯ | 18:44 |
| rlandy | ack | 18:44 |
| rlandy | sorry - end of year brain scatter | 18:44 |
| rlandy | also everyon ehere freaking out over impending snow storm | 18:45 |
| *** dtantsur is now known as dtantsur|afk | 18:50 | |
| *** jmasud has quit IRC | 18:56 | |
| *** jmasud has joined #oooq | 19:11 | |
| *** jmasud has quit IRC | 19:30 | |
| *** jmasud has joined #oooq | 19:32 | |
| rlandy | collect logs thing is a container-tools issue | 19:42 |
| rlandy | https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/tasks/collect.yml#L93 | 19:42 |
| weshay|ruck | heh | 19:53 |
| weshay|ruck | rlandy, hey.. testproject jobs are not incorporating tripleo-common changes | 19:53 |
| weshay|ruck | is that a required projects thing? right | 19:54 |
| weshay|ruck | e.g. https://logserver.rdoproject.org/88/31388/1/check/periodic-tripleo-ci-build-containers-ubi-8-push-train/af66efd/zuul-info/inventory.yaml | 19:54 |
| rlandy | weshay|ruck: looking | 19:59 |
| rlandy | opendev.org/openstack/tripleo-common: | 20:00 |
| rlandy | repo is there | 20:00 |
| weshay|ruck | the repo is.. but the change is not | 20:00 |
| rlandy | change_url: https://review.opendev.org/c/openstack/tripleo-ci/+/755124 | 20:00 |
| rlandy | change_url: https://review.rdoproject.org/r/31388 | 20:00 |
| rlandy | there are only two recorded | 20:01 |
| weshay|ruck | rlandy, https://review.opendev.org/c/openstack/tripleo-common/+/767359 is missing | 20:01 |
| rlandy | checking job | 20:01 |
| * weshay|ruck wonders if it's the url itself | 20:01 | |
| rlandy | Depends-On: https://review.opendev.org/c/767359 | 20:02 |
| rlandy | Depends-On: https://review.opendev.org/c/openstack/tripleo-ci/+/755124/ | 20:02 |
| rlandy | ^^ those two are there | 20:02 |
| weshay|ruck | I don't see 767359 in https://logserver.rdoproject.org/36/31336/4/check/periodic-tripleo-ci-build-containers-ubi-8-push-train/5f60c6c/zuul-info/inventory.yaml | 20:02 |
| rlandy | Repo | Branch openstack/tripleo-common | stable/ussuri | 20:02 |
| rlandy | weshay|ruck: am I reading this right ... | 20:03 |
| rlandy | https://review.rdoproject.org/r/#/c/31388/ to test https://review.opendev.org/c/openstack/tripleo-common/+/767359 | 20:03 |
| weshay|ruck | ah.. for the love | 20:03 |
| rlandy | on what branch? | 20:03 |
| weshay|ruck | ya.. needs to be train | 20:03 |
| * weshay|ruck trys again.. sorry | 20:04 | |
| rlandy | weshay|ruck: np | 20:04 |
| rlandy | narrowing down collect logs issue | 20:04 |
| rlandy | weshay|ruck: ^^ same missing logs in container-tools job in downtream | 20:05 |
| rlandy | failure at https://review.opendev.org/c/openstack/tripleo-common/+/767359 | 20:05 |
| rlandy | ugh | 20:05 |
| rlandy | https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/tasks/collect.yml#L93 | 20:05 |
| weshay|ruck | I blame gerrit.. can't read this thing well | 20:05 |
| rlandy | the new gerrit is much harder to work with | 20:06 |
| weshay|ruck | rlandy, so is it timing out? | 20:06 |
| rlandy | or we are just old | 20:06 |
| rlandy | weshay|ruck: yeah - not connecting | 20:06 |
| rlandy | container_collection | 20:06 |
| rlandy | tracing that | 20:06 |
| weshay|ruck | that's a jinja template | 20:07 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-dependencies-container_tools | 20:07 |
| rlandy | ^^ the jobs that fail to build containers have logs | 20:07 |
| rlandy | the jobs that have containers, have no logs | 20:07 |
| weshay|ruck | is it running this? https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/files/collect-container-logs.sh | 20:07 |
| rlandy | <127.0.0.2> (0, b'BECOME-SUCCESS-zhdjdztjfcyghrmtvknqfpnitnaigyaj\n\n{"changed": false, "stat": {"exists": false}, "invocation": {"module_args": {"path": "/tmp/collect-container-logs.sh", "follow": false, "get_checksum": true, "checksum_algorithm": "sha1", "get_md5": false, "get_mime": true, "get_attributes": true}}}\n', b'OpenSSH_8.0p1, OpenSSL 1.1.1g FIPS 21 Apr 2020\r\ndebug1: Reading configuration data /home/zuul/workspace/.q | 20:08 |
| rlandy | uickstart | 20:08 |
| rlandy | yeah | 20:08 |
| rlandy | ^ never finishes | 20:09 |
| rlandy | def container tools | 20:09 |
| weshay|ruck | rlandy, we can run that by hand | 20:12 |
| weshay|ruck | rlandy, let's look at it together | 20:12 |
| rlandy | weshay|ruck: k | 20:12 |
| weshay|ruck | https://meet.google.com/ote-vqqu-ypc?authuser=1 | 20:13 |
| weshay|ruck | rlandy, | 20:19 |
| weshay|ruck | https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/files/collect-container-logs.sh | 20:19 |
| weshay|ruck | rlandy, https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/c2a155c/logs/containers-consolidated-builds.log | 20:32 |
| weshay|ruck | rlandy, https://trunk.rdoproject.org/api-centos8-train/api/civotes_agg_detail.html?ref_hash=e59fa31c15c5563057de9cfe85bf3826 | 20:33 |
| rlandy | weshay|ruck: buildah-1.11.6-8.module+el8.3.0+8233+627fbb78.x86_64 | 20:41 |
| *** jmasud has quit IRC | 20:43 | |
| rlandy | weshay|ruck: https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_podman/vars/redhat-8.2.yml#L21 | 20:52 |
| *** rfolco has quit IRC | 20:56 | |
| rlandy | https://hackmd.io/v2jCX9RwSeuP8EEFDHRa8g | 21:02 |
| *** jmasud has joined #oooq | 21:13 | |
| *** jmasud has quit IRC | 21:19 | |
| *** rlandy is now known as rlandy|bbl | 21:43 | |
| *** TrevorV has quit IRC | 21:43 | |
| *** slaweq has quit IRC | 22:42 | |
| *** derekh has joined #oooq | 22:58 | |
| *** jbadiapa has quit IRC | 23:14 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!