| *** rfolco has joined #oooq | 02:03 | |
| *** rfolco has quit IRC | 02:07 | |
| *** apetrich has quit IRC | 02:14 | |
| *** rfolco has joined #oooq | 02:20 | |
| *** rfolco has quit IRC | 02:24 | |
| *** Goneri has quit IRC | 02:48 | |
| *** pojadhav|Sick is now known as pojadhav|pto | 03:15 | |
| *** ratailor has joined #oooq | 03:17 | |
| *** ykarel has joined #oooq | 03:55 | |
| *** sshnaidm|afk has quit IRC | 04:08 | |
| *** matbu has quit IRC | 04:08 | |
| *** yolanda has quit IRC | 04:08 | |
| *** gouthamr_ has quit IRC | 04:08 | |
| *** ccamacho has quit IRC | 04:08 | |
| *** holser has quit IRC | 04:08 | |
| *** mjturek has quit IRC | 04:08 | |
| *** cgoncalves has quit IRC | 04:08 | |
| *** bhagyashris has quit IRC | 04:08 | |
| *** baha has quit IRC | 04:08 | |
| *** ysandeep|away has quit IRC | 04:08 | |
| *** tristanC has quit IRC | 04:08 | |
| *** ChanServ has quit IRC | 04:08 | |
| *** zbr|ruck has quit IRC | 04:08 | |
| *** zbr has quit IRC | 04:08 | |
| *** kopecmartin|pto has quit IRC | 04:08 | |
| *** akahat has quit IRC | 04:08 | |
| *** Tengu has quit IRC | 04:08 | |
| *** dpawlik2 has quit IRC | 04:08 | |
| *** fuzzball81 has quit IRC | 04:08 | |
| *** panda has quit IRC | 04:08 | |
| *** arxcruz has quit IRC | 04:08 | |
| *** EmilienM|PTO has quit IRC | 04:08 | |
| *** odyssey4me has quit IRC | 04:08 | |
| *** ratailor has quit IRC | 04:08 | |
| *** jmasud has quit IRC | 04:08 | |
| *** pojadhav|pto has quit IRC | 04:08 | |
| *** chandankumar has quit IRC | 04:08 | |
| *** migi has quit IRC | 04:08 | |
| *** irclogbot_1 has quit IRC | 04:08 | |
| *** amoralej|off has quit IRC | 04:08 | |
| *** weshay_ruck has quit IRC | 04:08 | |
| *** johnsom has quit IRC | 04:08 | |
| *** gouthamr has quit IRC | 04:08 | |
| *** dtantsur|afk has quit IRC | 04:08 | |
| *** zbr|ruck has joined #oooq | 04:14 | |
| *** zbr has joined #oooq | 04:14 | |
| *** baha has joined #oooq | 04:14 | |
| *** bhagyashris has joined #oooq | 04:14 | |
| *** cgoncalves has joined #oooq | 04:14 | |
| *** mjturek has joined #oooq | 04:14 | |
| *** ccamacho has joined #oooq | 04:14 | |
| *** gouthamr_ has joined #oooq | 04:14 | |
| *** yolanda has joined #oooq | 04:14 | |
| *** ratailor has joined #oooq | 04:14 | |
| *** odyssey4me has joined #oooq | 04:14 | |
| *** kopecmartin|pto has joined #oooq | 04:14 | |
| *** akahat has joined #oooq | 04:14 | |
| *** arxcruz has joined #oooq | 04:14 | |
| *** panda has joined #oooq | 04:14 | |
| *** fuzzball81 has joined #oooq | 04:14 | |
| *** EmilienM|PTO has joined #oooq | 04:14 | |
| *** dpawlik2 has joined #oooq | 04:14 | |
| *** Tengu has joined #oooq | 04:14 | |
| *** pojadhav|pto has joined #oooq | 04:14 | |
| *** jmasud has joined #oooq | 04:14 | |
| *** holser_ has joined #oooq | 04:14 | |
| *** johnsom has joined #oooq | 04:14 | |
| *** gouthamr has joined #oooq | 04:14 | |
| *** chandankumar has joined #oooq | 04:14 | |
| *** amoralej|off has joined #oooq | 04:14 | |
| *** tristanC has joined #oooq | 04:14 | |
| *** ysandeep|away has joined #oooq | 04:14 | |
| *** ChanServ has joined #oooq | 04:14 | |
| *** migi has joined #oooq | 04:14 | |
| *** irclogbot_1 has joined #oooq | 04:14 | |
| *** weshay_ruck has joined #oooq | 04:14 | |
| *** dtantsur|afk has joined #oooq | 04:14 | |
| *** tepper.freenode.net sets mode: +oo arxcruz ChanServ | 04:14 | |
| *** sshnaidm|afk has joined #oooq | 04:15 | |
| *** matbu has joined #oooq | 04:15 | |
| *** skramaja has joined #oooq | 04:17 | |
| *** Tengu has quit IRC | 04:25 | |
| *** Tengu has joined #oooq | 04:27 | |
| *** ysandeep|away is now known as ysandeep|rover | 04:32 | |
| *** soniya29 has joined #oooq | 04:37 | |
| *** ykarel has quit IRC | 04:40 | |
| *** ykarel has joined #oooq | 04:41 | |
| bhagyashris | ysandeep|rover, Good morning! :) fyi here is the patch https://review.opendev.org/#/c/740307/ to resolve the sc004 issue https://bugs.launchpad.net/tripleo/+bug/1886829 here is the test patch https://review.rdoproject.org/r/#/c/28452/ . Job is still failing with diff issue.Now looking into it. | 05:05 |
|---|---|---|
| openstack | Launchpad bug 1886829 in tripleo "Error: 'rabbitmqctl status | grep -F "{rabbit,"' returned 1 instead of one of [0] failing scenario004-standalone-train job" [Medium,Triaged] - Assigned to Michele Baldessari (michele) | 05:05 |
| bhagyashris | ysandeep|rover, you can check with above patch for component pipeline testing | 05:05 |
| ysandeep|rover | bhagyashris, thanks! | 05:06 |
| *** marios has joined #oooq | 05:09 | |
| *** udesale has joined #oooq | 05:09 | |
| ysandeep|rover | bhagyashris, Error: Failed to download metadata for repo \'centos-opstools\': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", | 05:21 |
| ysandeep|rover | https://logserver.rdoproject.org/52/28452/2/check/periodic-tripleo-ci-centos-8-scenario004-standalone-train/6f414fc/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 05:21 |
| bhagyashris | correct yeah now it's failing here ^ | 05:22 |
| ysandeep|rover | bhagyashris, [Failed to connect to mirror.regionone.rdo-cloud-tripleo.rdoproject.org port 80: Connection refused]" , but i can open - http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/centos/8/messaging/x86_64/rabbitmq-38/repodata/repomd.xml | 05:23 |
| ysandeep|rover | in my browser, | 05:23 |
| bhagyashris | me too can open | 05:23 |
| bhagyashris | but not sure where is problem | 05:24 |
| ysandeep|rover | bhagyashris, It might be a transient mirror issue, I will run my component job to confirm | 05:24 |
| bhagyashris | sure | 05:24 |
| bhagyashris | me to put recheck on my test patch | 05:24 |
| bhagyashris | waiting for result | 05:24 |
| ysandeep|rover | okay | 05:24 |
| *** soniya29 has quit IRC | 05:29 | |
| *** udesale_ has joined #oooq | 05:36 | |
| *** udesale has quit IRC | 05:38 | |
| *** udesale_ has quit IRC | 05:44 | |
| *** udesale has joined #oooq | 05:55 | |
| *** saneax has joined #oooq | 05:59 | |
| chandankumar | ysandeep|rover: do we have a bug open for this failure https://logserver.rdoproject.org/periodic-ci-config-molecule-python/opendev.org/openstack/tripleo-ci/master/periodic-molecule-container-push-delegated-centos-7/1d9b6bd/job-output.txt ? | 06:13 |
| ysandeep|rover | chandankumar, We haven't open yesterday - Checking previous rr notes | 06:14 |
| *** amoralej|off is now known as amoralej | 06:15 | |
| *** jfrancoa has joined #oooq | 06:16 | |
| ysandeep|rover | chandankumar, None which i could find, filing a new one for this | 06:19 |
| chandankumar | ysandeep|rover: sure | 06:20 |
| *** kopecmartin|pto is now known as kopecmartin | 06:26 | |
| bhagyashris | ysandeep|rover, got SUCCESS https://review.rdoproject.org/r/#/c/28452/ :) | 06:38 |
| ysandeep|rover | bhagyashris, great \o/ | 06:39 |
| ysandeep|rover | chandankumar, fyi.. https://bugs.launchpad.net/tripleo/+bug/1887120 | 06:39 |
| openstack | Launchpad bug 1887120 in tripleo "Periodic C7 container push molecule job - periodic-molecule-container-push-delegated-centos-7 is failing with ERROR: molecule_delegated: could not install deps " [High,Triaged] | 06:39 |
| bhagyashris | marios, thank you for review https://review.opendev.org/#/c/740307/ :) | 06:39 |
| *** soniya29 has joined #oooq | 06:40 | |
| marios | bhagyashris: o/ np | 06:41 |
| ysandeep|rover | marios, for these kind of job failure (periodic-molecule-container-push-delegated-centos-7)- yesterday in meeting we discussed RR will create a jira card apart from bz .. Should I create a card in storyless ? | 06:42 |
| ysandeep|rover | chandankumar, ^^ | 06:45 |
| marios | ysandeep|rover: yeah that's what was agreed | 06:46 |
| marios | ysandeep|rover: but you mean taiga right not jira (yet) | 06:46 |
| ysandeep|rover | marios, sry yes tiaga (seems I had too much jira in last 2 days )just want to confirm storyless tasks are the right place. | 06:47 |
| marios | ysandeep|rover: ack yes | 06:49 |
| ysandeep|rover | marios, thank you :) :) :) :) | 06:49 |
| marios | np ysandeep|rover | 06:51 |
| *** jfrancoa has quit IRC | 06:58 | |
| *** tosky has joined #oooq | 07:30 | |
| *** bhagyashris is now known as bhagyashris|lunc | 07:38 | |
| *** ratailor has quit IRC | 07:56 | |
| *** skramaja has quit IRC | 08:05 | |
| *** skramaja has joined #oooq | 08:05 | |
| *** ysandeep|rover is now known as ysandeep|lunch | 08:06 | |
| *** derekh has joined #oooq | 08:07 | |
| *** dtantsur|afk is now known as dtantsur | 08:09 | |
| *** ratailor has joined #oooq | 08:22 | |
| *** apetrich has joined #oooq | 08:27 | |
| *** cgoncalves has quit IRC | 08:27 | |
| *** gchamoul has joined #oooq | 08:28 | |
| *** cgoncalves has joined #oooq | 08:29 | |
| marios | need another core vote there please tripleo-ci https://review.opendev.org/#/c/739939/ (& waiting for zuul check to run again) | 08:32 |
| *** jbadiapa has joined #oooq | 08:34 | |
| *** Tengu has quit IRC | 08:46 | |
| chandankumar | arxcruz: kopecmartin soniya29 akahat please have a look at tempest agenda https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ?view#2020-07-10 and add or remove from it | 08:46 |
| *** bhagyashris|lunc is now known as bhagyashris | 08:58 | |
| *** ratailor has quit IRC | 09:01 | |
| *** ratailor_ has joined #oooq | 09:01 | |
| *** Tengu has joined #oooq | 09:08 | |
| *** ysandeep|lunch is now known as ysandeep | 09:13 | |
| *** ysandeep is now known as ysandeep|rover | 09:13 | |
| bhagyashris | ysandeep|rover, hey is this issue resolved https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-standalone-train/1832afc/job-output.txt it's related to "FAILED - RETRYING: Ping router ip address" | 09:37 |
| ysandeep|rover | bhagyashris, trying a test patch to confirm if its related to iptables rules | 09:40 |
| bhagyashris | ysandeep|rover, i saw there is one patch https://review.opendev.org/#/c/739963/ | 09:40 |
| bhagyashris | ysandeep|rover, i test for sc007 but it failed with same issue | 09:40 |
| ysandeep|rover | bhagyashris, yes trying with this as a depends-on | 09:40 |
| ysandeep|rover | you added https://review.opendev.org/#/c/739963/ as depends on? | 09:40 |
| ysandeep|rover | bhagyashris, ^^ | 09:41 |
| bhagyashris | ysandeep|rover, yes | 09:41 |
| bhagyashris | ysandeep|rover, fyi https://review.rdoproject.org/r/#/c/28452/ | 09:41 |
| ysandeep|rover | bhagyashris, that means its not related to iptables issue - https://logserver.rdoproject.org/52/28452/3/check/periodic-tripleo-ci-centos-8-scenario007-standalone-train/40aac7c/logs/undercloud/var/log/extra/network.txt.gz correct rules are present | 09:43 |
| ysandeep|rover | bhagyashris, looking for cause of failure | 09:43 |
| bhagyashris | ysandeep|rover, ack , i thought it was iptable issue | 09:46 |
| bhagyashris | https://logserver.rdoproject.org/52/28452/3/check/periodic-tripleo-ci-centos-8-scenario007-standalone-train/40aac7c/logs/undercloud/var/log/extra/errors.txt.txt.gz | 09:47 |
| ysandeep|rover | bhagyashris, yes initially i thought this might be becaues of missing icmp rule.. | 09:47 |
| bhagyashris | ERROR neutron.agent.l3.agent neutron_lib.exceptions.BridgeDoesNotExist: Bridge br-int does not exist. | 09:47 |
| soniya29 | chandankumar, added | 09:52 |
| marios | arxcruz: zbr|ruck: please when you next have time need another core vote there please tripleo-ci https://review.opendev.org/#/c/739939/ | 09:55 |
| arxcruz | marios: i'm not core, sorr | 10:00 |
| arxcruz | sorry* | 10:00 |
| marios | arxcruz: ack thanks | 10:00 |
| *** soniya29 has quit IRC | 10:03 | |
| *** Tengu has quit IRC | 10:05 | |
| *** Tengu has joined #oooq | 10:07 | |
| ysandeep|rover | bhagyashris, to me it seems br-int failed to be created because neutron_ovs_agent container failed to spawn properly | 10:15 |
| ysandeep|rover | https://logserver.rdoproject.org/52/28452/3/check/periodic-tripleo-ci-centos-8-scenario007-standalone-train/40aac7c/logs/undercloud/var/log/extra/failed_containers.log.txt.gz | 10:15 |
| ysandeep|rover | https://logserver.rdoproject.org/52/28452/3/check/periodic-tripleo-ci-centos-8-scenario007-standalone-train/40aac7c/logs/undercloud/var/log/extra/podman/containers/neutron_ovs_agent/stdout.log.txt.gz | 10:16 |
| ysandeep|rover | looks like container failed because of wrong version of python - seems caused by /usr/bin/python: No such file or directory | 10:16 |
| ysandeep|rover | bhagyashris, I am writing a bz for this | 10:17 |
| bhagyashris | ysandeep|rover, ok | 10:18 |
| *** dtantsur is now known as dtantsur|brb | 10:21 | |
| zbr | marios: zuul voted no | 10:23 |
| marios | zbr: thanks (it's unrelated to the change looks like issue with minion validation) | 10:25 |
| marios | zbr: there must be some bug for that the job is pretty bad shape https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-containers-undercloud-minion | 10:26 |
| marios | zbr|ruck: ysandeep|rover: do we have a bug for that? ^ | 10:26 |
| ysandeep|rover | marios, yes | 10:27 |
| ysandeep|rover | marios, https://bugs.launchpad.net/tripleo/+bug/1886914 | 10:27 |
| openstack | Launchpad bug 1886914 in tripleo "Undercloud minion failing validation ERROR: Heat Engine host count is 1 or less." [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 10:27 |
| marios | ysandeep|rover: thank you | 10:27 |
| ykarel | ysandeep|rover, i know that python error, it hit in past | 10:28 |
| * ykarel finds patch | 10:28 | |
| ysandeep|rover | ykarel, thanks i will try to fix it with reference patch | 10:29 |
| ykarel | ysandeep|rover, likely https://review.opendev.org/#/c/710980/ | 10:30 |
| zbr | emilien is in pto, so good luck with that. | 10:31 |
| ysandeep|rover | ykarel, thanks looking | 10:32 |
| ysandeep|rover | ykarel, yeah this is it train still have /usr/bin/python and need update- https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/neutron/neutron-ovs-agent-container-puppet.yaml#L61 | 10:35 |
| ykarel | yes that was not cherrypicked to train | 10:35 |
| ysandeep|rover | ykarel, i will cherrypick it to train | 10:35 |
| ykarel | ack | 10:35 |
| bhagyashris | ysandeep|rover, jfyi dont create new taiga card for sc007 failure because there is already taiga card for it https://tree.taiga.io/project/tripleo-ci-board/task/1850?kanban-status=1447275 | 10:38 |
| ysandeep|rover | bhagyashris, thank! yes i know we have that card, and as per yesterday discussion we just create taiga card for periodic CI-Config jobs only , rest all are bz only | 10:42 |
| bhagyashris | ysandeep|rover, ack | 10:42 |
| *** Tengu has quit IRC | 10:47 | |
| ysandeep|rover | bhagyashris, can i reuse your testpatch to test https://review.opendev.org/#/c/740440/ | 10:47 |
| bhagyashris | ysandeep|rover, just pushed | 10:47 |
| bhagyashris | https://review.rdoproject.org/r/#/c/28452/ | 10:47 |
| bhagyashris | ysandeep|rover, | 10:47 |
| *** Tengu has joined #oooq | 10:48 | |
| ysandeep|rover | bhagyashris++ you are too fast :) | 10:48 |
| bhagyashris | ysandeep|rover, yeah actually i was waiting and just checked your patches and and saw that ^ so pushed for test :) | 10:50 |
| bhagyashris | ysandeep|rover, thanks! :) | 10:50 |
| ysandeep|rover | bhagyashris, thanks to you :) | 10:50 |
| bhagyashris | ysandeep|rover, which task wes ask to take up? is this - https://tree.taiga.io/project/tripleo-ci-board/task/1815?kanban-status=1447275 ? | 11:01 |
| bhagyashris | ysandeep|rover, or is there any task that i can help ... as you are currently busy in RR stuff. | 11:02 |
| ysandeep|rover | bhagyashris, https://tree.taiga.io/project/tripleo-ci-board/task/1815 - https://review.opendev.org/#/c/737682/ is almost done - will be fixed once iptables patch merges. | 11:05 |
| bhagyashris | ysandeep|rover, ack | 11:06 |
| ysandeep|rover | bhagyashris, if you want feel free to take those task from userstory #1703 | 11:07 |
| ysandeep|rover | i didn't get much time to look at those | 11:08 |
| *** skramaja has quit IRC | 11:08 | |
| bhagyashris | ysandeep|rover, ack | 11:11 |
| *** ykarel has quit IRC | 11:12 | |
| ysandeep|rover | bhagyashris, thank you :) | 11:12 |
| *** ysandeep|rover is now known as ysandeep|break | 11:13 | |
| *** ykarel has joined #oooq | 11:13 | |
| chandankumar | zbr: marios https://review.rdoproject.org/r/#/c/28482/ it is fix periodic molecule job | 11:21 |
| marios | chandankumar: ack will do | 11:23 |
| *** derekh has quit IRC | 11:23 | |
| *** zbr|ruck has quit IRC | 11:23 | |
| *** derekh has joined #oooq | 11:27 | |
| *** zbr|ruck has joined #oooq | 11:27 | |
| bhagyashris | weshay_ruck, fyi sc004 train failure got fix with this change https://review.opendev.org/#/c/740307/1 | 11:28 |
| *** bhagyashris is now known as bhagyashris|brb | 11:30 | |
| *** ysandeep|break is now known as ysandeep|rover | 11:33 | |
| weshay_ruck | bhagyashris++ | 11:33 |
| marios | chandankumar: where are you testing https://review.rdoproject.org/r/#/c/28482/ i don't see the job there | 11:41 |
| marios | chandankumar: otherwise fine by me if it fixes the issue will revote | 11:42 |
| *** rlandy has joined #oooq | 11:47 | |
| rlandy | weshay_ruck++ on the buildah change | 11:50 |
| rlandy | deploy successful | 11:50 |
| rlandy | stack deleting | 11:50 |
| rlandy | tempest run | 11:50 |
| *** bhagyashris|brb is now known as bhagyashris | 11:50 | |
| rlandy | chandankumar: hey ... need to borrow your image build brain | 11:52 |
| chandankumar | rlandy: yes, tell me | 11:53 |
| rlandy | chandankumar: any idea why this would be needed all of a sudden? https://code.engineering.redhat.com/gerrit/#/c/205859/ | 11:53 |
| bhagyashris | ysandeep|rover, hey got SUCCESS https://review.rdoproject.org/r/#/c/28452/ :) | 11:54 |
| rlandy | weshay_ruck put in a patch for upstream as well but somehow that one is not neede | 11:54 |
| ysandeep|rover | bhagyashris, yes i saw that :) | 11:54 |
| rlandy | chandankumar: ^^ it seems that the difference is that there are repos on the rdo nodes | 11:54 |
| rlandy | and not on the ops overcloud nodes | 11:54 |
| weshay_ruck | rlandy, ah nice | 11:55 |
| chandankumar | rlandy: during run-time, does something is pulling images using buildah? | 11:56 |
| rlandy | weshay_ruck: here's an interesting catch though | 11:56 |
| chandankumar | or doing some operation related to buildah | 11:56 |
| rlandy | see the timeout on the ovb job on the upstream pacth | 11:56 |
| rlandy | weshay_ruck: some timeout in inventory node on downstream :) | 11:56 |
| rlandy | chandankumar: idk why we need buildah all of a sudden - my best guess is new container builds | 11:57 |
| * rlandy gets logs | 11:57 | |
| weshay_ruck | rlandy, chandankumar https://review.opendev.org/#/c/740378/ | 11:57 |
| chandankumar | https://opendev.org/openstack/tripleo-puppet-elements/src/branch/master/elements/overcloud-base/install.d/package-installs-overcloud-base buildah is also not there | 11:58 |
| rlandy | seems to be covered up in the upstream by having repos on the overcloud nodes | 11:58 |
| rlandy | chandankumar: which is debatable | 11:58 |
| rlandy | iirc, we shouldn't need that | 11:58 |
| rlandy | https://logserver.rdoproject.org/78/740378/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/444665c/job-output.txt | 12:00 |
| rlandy | vs | 12:00 |
| chandankumar | rlandy: the container build is happening on undercloud na, not on overcloud | 12:00 |
| rlandy | chandankumar: agree | 12:00 |
| chandankumar | marios: commented here https://review.rdoproject.org/r/#/c/28482/ | 12:00 |
| rlandy | the package is available on the undercloud | 12:00 |
| rlandy | but not required to be put on the overcloud | 12:01 |
| rlandy | https://review.opendev.org/#/c/740378/ | 12:01 |
| rlandy | ^^ adds that | 12:01 |
| chandankumar | rlandy: may be or due to tripleo-ansible patch? | 12:01 |
| rlandy | chandankumar: can you point to patch | 12:02 |
| chandankumar | rlandy: https://review.opendev.org/739415 not sure | 12:02 |
| rlandy | chandankumar: I wonder ... | 12:02 |
| rlandy | weshay_ruck: ^^ would that impact overcloud deploy | 12:03 |
| rlandy | possibly? | 12:03 |
| ykarel | mmm what needs buildah on overcloud nodes? | 12:03 |
| rlandy | ykarel: we picked up error only on OSP - RDO | 12:04 |
| ykarel | rlandy, at what step? | 12:04 |
| rlandy | it's possible that RDO covers this up - because there are repos on the overcloud nodes | 12:04 |
| * rlandy gets | 12:04 | |
| ykarel | repos are cleaned in overcloud image builds iirc | 12:04 |
| ykarel | and readded via quickstart | 12:05 |
| rlandy | see FATAL | ensure podman and deps are installed | overcloud-controller-1 | error={"changed": false, "failures": ["No package buildah available."], | 12:05 |
| rlandy | in https://sf.hosted.upshift.rdu2.redhat.com/logs/36/200436/30/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-17/45dd65b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 12:05 |
| ykarel | ^^ is the above patch | 12:05 |
| rlandy | ykarel: we picked up the same problem in baremetal deployments | 12:05 |
| ykarel | looking logs | 12:06 |
| rlandy | ykarel: in OSP they are not readded | 12:06 |
| rlandy | which may be correct | 12:06 |
| rlandy | they shouldn't be needed | 12:06 |
| rlandy | which weshay_ruck's patch we clear OVB deployments | 12:06 |
| ykarel | at this failure step repos should have been configured, me checks | 12:07 |
| *** saneax has quit IRC | 12:07 | |
| rlandy | zbr|ruck: ysandeep|rover: hey - ping me if you need backup | 12:07 |
| rlandy | around if needed | 12:07 |
| rlandy | ykarel: so - *should* we be relying on repos on the overcloud nodes | 12:08 |
| rlandy | if yes, we are missing something so add them to OSP builds | 12:09 |
| rlandy | which were funnily enough working until now | 12:09 |
| ykarel | rlandy, so two stories, i think tripleo-ansible should be fixed to not install buildah always | 12:12 |
| *** dtantsur|brb is now known as dtantsur | 12:12 | |
| ykarel | and wrt repos on overcloud nodes, that need to be synched for upstream and downstream jobs | 12:12 |
| rlandy | ykarel: ok - I wasn't sure that we actually should be putting repos on the overcloud nodes. | 12:14 |
| * rlandy looks for where we do this repo setting for the overcloud | 12:15 | |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-images/defaults/main.yml#L14? | 12:15 |
| rlandy | chandankumar: ^^ are we missing this in image builds in downstream? | 12:16 |
| rlandy | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-17.yml#n189 maybe | 12:17 |
| chandankumar | rlandy: checking | 12:19 |
| chandankumar | rlandy: both are same | 12:20 |
| *** soniya29 has joined #oooq | 12:21 | |
| chandankumar | rlandy: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-images/tasks/main.yml#L31 | 12:21 |
| chandankumar | see here | 12:21 |
| rlandy | chandankumar: ok - so then I don't know how in RDO quickstart we get repos on the overcloud nodes | 12:22 |
| *** jfrancoa has joined #oooq | 12:22 | |
| rlandy | repo-setup on all nodes | 12:22 |
| ykarel | rlandy, they are pushed when not building images | 12:22 |
| ykarel | yes by repo-setup on images instead of nodes | 12:23 |
| ykarel | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/ovb-setup.yml#L73 | 12:23 |
| rlandy | ib_repo_image_inject: "{{ not to_build|bool }}" | 12:24 |
| *** Tengu has quit IRC | 12:26 | |
| *** Tengu has joined #oooq | 12:27 | |
| rlandy | ykarel: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/ovb-setup.yml#L64 only injects images if to_build is false | 12:28 |
| rlandy | ib_repo_image_inject: "{{ not to_build|bool }}" | 12:28 |
| *** derekh has quit IRC | 12:29 | |
| rlandy | name: Enject repo in overcloud image | 12:30 |
| rlandy | Inject I assume :) | 12:30 |
| chandankumar | ykarel: rlandy all repos are injected while building the image itself | 12:30 |
| chandankumar | weshay_ruck: akahat meeting time | 12:31 |
| rlandy | chandankumar: so somehow missed https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-images/tasks/main.yml#L44 | 12:31 |
| ykarel | chandankumar, but those are cleaned up after images are built | 12:31 |
| weshay_ruck | zbr, https://review.rdoproject.org/r/#/c/28482/ tox pin | 12:31 |
| chandankumar | ykarel: it depends on this https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-images/tasks/main.yml#L24 | 12:31 |
| chandankumar | gating repo generated via build test packages i think | 12:32 |
| chandankumar | weshay_ruck: you are joining the tempest meeting? | 12:32 |
| * rlandy checks what is missing in osp build images | 12:32 | |
| chandankumar | weshay_ruck: https://meet.google.com/zvw-hnds-ncu?authuser=0 | 12:33 |
| ykarel | chandankumar, me not aware if the web-gating we using some where | 12:36 |
| ykarel | but yes /opt/gating-repo is created by build-test-packages | 12:36 |
| *** ratailor_ has quit IRC | 12:44 | |
| *** Tengu has quit IRC | 12:44 | |
| *** Tengu has joined #oooq | 12:46 | |
| marios | rlandy: weshay_ruck: need another core vote there please when you have some time https://review.opendev.org/#/c/739939/ | 12:48 |
| rlandy | looking | 12:48 |
| rlandy | marios: done | 12:49 |
| rlandy | marios: also w+'ed | 12:49 |
| marios | thanks rlandy | 12:49 |
| marios | rlandy: hoping the minion job is green now the fix for it merged let's see in gate | 12:49 |
| rlandy | there is always hope | 12:50 |
| marios | it *is* friday | 12:50 |
| ykarel | rlandy, wrt tripleo-ansible fix for buildah, should i send a patch or u will take care? | 12:51 |
| rlandy | ykarel: is still needed in general? https://review.opendev.org/#/c/739415/ | 12:53 |
| rlandy | I am going to try fix the missing repos from OSP first | 12:54 |
| ykarel | rlandy, no not needed generally, only needed for container builda and push | 12:54 |
| ykarel | so need to make it optional | 12:54 |
| rlandy | ykarel: so yeah - then I'll take care of it | 12:54 |
| rlandy | export DIB_YUM_REPO_CONF='/etc/yum.repos.d/rhos-release*.repo /etc/yum.repos.d/delorean* /etc/yum.repos.d/redhat.repo /etc/yum.repos.d/osptrunk-candidate-deps.repo /etc/yum.repos.d/rhosp-ansible-2.9.repo' | 12:54 |
| rlandy | hmmm ... it knows about these repos | 12:54 |
| ykarel | rlandy, okk | 12:55 |
| rlandy | why doesn't it add them?? | 12:55 |
| ykarel | it adds and cleans those | 12:55 |
| ykarel | once finished | 12:55 |
| rlandy | hmmm | 12:59 |
| rlandy | it's the difference of where we build/download image | 12:59 |
| rlandy | 2020-07-10 01:12:07.342223 | primary | will download https://images.rdoproject.org/centos8/master/rdo_trunk/tripleo-ci-testing/overcloud-full.tar with md5sum of a7418bacea789c8df4cdc40270e57a46 | 13:00 |
| rlandy | unpstream in periodic we download | 13:00 |
| rlandy | the issue is in the choice whether or not to add repos | 13:00 |
| rlandy | in the component pipeline we have to build images | 13:00 |
| rlandy | there is always a chnage | 13:00 |
| rlandy | and we then need to override the ib_repo_image_inject | 13:01 |
| rlandy | it's not dependent on whether the image build or not | 13:01 |
| rlandy | name: Always build images in periodic | 13:02 |
| rlandy | ^^ baremetal | 13:02 |
| rlandy | ib_repo_image_inject needs to be true whether or not we build | 13:05 |
| rlandy | ^^ that is the problem | 13:05 |
| * rlandy fixes | 13:05 | |
| rlandy | ykarel: ^^ got it - thanks for the pointers | 13:05 |
| zbr | weshay_ruck: pinning down reqs is the solution for the wrong problem. | 13:16 |
| zbr | the issue is that these jobs run with py27 instead of py36, that is the issue that we need to address | 13:16 |
| *** derekh has joined #oooq | 13:20 | |
| *** irclogbot_1 has quit IRC | 13:27 | |
| *** irclogbot_0 has joined #oooq | 13:28 | |
| *** TrevorV has joined #oooq | 13:30 | |
| chandankumar | ykarel: ysandeep|rover It contains https://review.opendev.org/#/c/740440/ cherry pick and other changes, I would suggest remove the cherry pick commit just add the reference with the commit | 13:32 |
| chandankumar | and all fixes | 13:33 |
| ykarel | i think cherry-pick can be there with addition message in commit | 13:33 |
| ysandeep|rover | chandankumar, yes good point .. cherry pick should be clean | 13:33 |
| ysandeep|rover | updating | 13:33 |
| ykarel | cherry-pick can't be clean always | 13:33 |
| ykarel | so we need to adjust it | 13:33 |
| *** udesale_ has joined #oooq | 13:36 | |
| *** udesale has quit IRC | 13:39 | |
| *** soniya29 has quit IRC | 13:43 | |
| ysandeep|rover | chandankumar, updated commit about additional change apart from cherrypicked changes | 13:43 |
| ysandeep|rover | https://review.opendev.org/#/c/740440/3 | 13:43 |
| *** soniya29 has joined #oooq | 13:44 | |
| *** rascasoft has joined #oooq | 13:55 | |
| *** bhagyashris is now known as bhagyashris|away | 14:10 | |
| *** dtantsur is now known as creepy_owlet | 14:36 | |
| *** ykarel is now known as ykarel|away | 14:41 | |
| ysandeep|rover | rlandy, downstream iptables rules need some more improvements - https://code.engineering.redhat.com/gerrit/#/c/205913/ please review when time allows | 14:42 |
| rlandy | sure | 14:43 |
| ysandeep|rover | thanks! :) | 14:43 |
| ysandeep|rover | we added tristanC there as well for review | 14:45 |
| rlandy | weshay_ruck: so remember the conversation about needing promotions and components getting too old ... | 14:48 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1886862 | 14:48 |
| openstack | Launchpad bug 1886862 in tripleo " Compute component pipeline tests are failing standalone deployment with "Failed container(s): ['nova_wait_for_compute_service']" - module 'os_traits' has no attribute 'HW_CPU_X86_AVX512VBMI'" [Critical,Triaged] | 14:48 |
| rlandy | ^^ that should clear in RDO when we next promote | 14:48 |
| *** ysandeep|rover is now known as ysandeep|away | 14:48 | |
| weshay_ruck | rlandy, ya.. /me looks | 14:48 |
| rlandy | but OSP has no clear promotion path due scenario010 | 14:48 |
| rlandy | weshay_ruck: ^^ | 14:48 |
| weshay_ruck | rlandy, ok.. so common just promoted in master | 14:49 |
| weshay_ruck | I see a few compute jobs passing there now | 14:49 |
| weshay_ruck | actually all tests passed but the tempest api timed out | 14:50 |
| weshay_ruck | rlandy, so it may be fixed | 14:50 |
| weshay_ruck | right.. re: scenario010 | 14:50 |
| rlandy | weshay_ruck: killing us ... just saying | 14:50 |
| weshay_ruck | rlandy, so we take scenario010 out of the promotion critieria for integration? | 14:51 |
| chandankumar | rlandy: what happened with sc10? | 14:51 |
| rlandy | weshay_ruck: well - it's a discussion | 14:53 |
| rlandy | we'll likely break that elsewhere | 14:53 |
| rlandy | chandankumar: we can't promote octavia in osp-17 component pipeline due to unresolved issue ... | 14:53 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1885602 | 14:54 |
| openstack | Launchpad bug 1885602 in tripleo "Octavia component: failing tempest, Details: {'faultcode': 'Client', 'faultstring': 'Policy does not allow this request to be performed.'," [Critical,Triaged] - Assigned to Brent Eagles (beagles) | 14:54 |
| rlandy | ^^ which is preventing the that test passing in tripleo-ci-testing | 14:54 |
| rlandy | and holding the integration line | 14:54 |
| rlandy | so we're in a catch-22 | 14:54 |
| weshay_ruck | rlandy, retriggered compute for an upstream promotion | 14:55 |
| rlandy | really it's a general case problem | 14:55 |
| rlandy | how long do we hold a component | 14:55 |
| weshay_ruck | rlandy, let's chat | 14:55 |
| rlandy | weshay_ruck: ack | 14:55 |
| weshay_ruck | https://meet.google.com/uvn-jgzk-xdy | 14:56 |
| chandankumar | rlandy: weshay_ruck https://review.opendev.org/739710 and https://review.rdoproject.org/r/28482 https://review.rdoproject.org/r/28480 and https://review.rdoproject.org/r/#/c/28451/ that, it, all good to go | 14:56 |
| chandankumar | rlandy: will take a look | 14:56 |
| *** ykarel|away has quit IRC | 15:14 | |
| *** amoralej is now known as amoralej|off | 15:17 | |
| chandankumar | see ya on monday! | 15:18 |
| *** chandankumar is now known as raukadah | 15:18 | |
| weshay_ruck | rlandy, http://mirror.centos.org/centos-8/8/cr/x86_64/os/Packages/ | 15:22 |
| weshay_ruck | raukadah, you have 2min? | 15:26 |
| weshay_ruck | sorry | 15:26 |
| raukadah | weshay_ruck: yes | 15:26 |
| weshay_ruck | raukadah, https://meet.google.com/uvn-jgzk-xdy | 15:26 |
| weshay_ruck | real quick | 15:26 |
| weshay_ruck | raukadah, http://mirror.centos.org/centos-8/8/cr/x86_64/os/Packages/ | 15:27 |
| weshay_ruck | http://mirror.centos.org/centos-8/8/cr/x86_64/os/Packages/ | 15:30 |
| weshay_ruck | rlandy, ^ | 15:30 |
| weshay_ruck | openstack-component-centos-8 | 15:35 |
| weshay_ruck | openstack-component-centos-7 | 15:35 |
| weshay_ruck | openstack-component-rhel-8 | 15:35 |
| weshay_ruck | rlandy, ^ | 15:35 |
| weshay_ruck | 3 times a week | 15:35 |
| weshay_ruck | rlandy, openstack-component-rhel-$repo-8 | 15:36 |
| rlandy | zbr|ruck: pls review https://review.rdoproject.org/r/#/c/28482/ before we merge something that may cause trouble | 15:58 |
| rlandy | weshay_ruck: need to make a change to master criteria on the promotion server in rdo ... https://review.rdoproject.org/r/#/c/28451/ | 16:00 |
| rlandy | tmate this? | 16:00 |
| * rlandy checks server | 16:00 | |
| zbr | rlandy: I added comment on https://bugs.launchpad.net/tripleo/+bug/1887120 | 16:01 |
| openstack | Launchpad bug 1887120 in tripleo "Periodic C7 container push molecule job - periodic-molecule-container-push-delegated-centos-7 is failing with ERROR: molecule_delegated: could not install deps " [High,Triaged] | 16:01 |
| weshay_ruck | ah ya.. but all the jobs will fail if that fails | 16:01 |
| zbr | you chance version, it will not fix the source of the error. | 16:02 |
| rlandy | weshay_ruck: I updated the master.ini criteria on rdo promoter server and restarted the service | 16:04 |
| weshay_ruck | k.. thanks | 16:05 |
| rlandy | weshay_ruck: pls check before you go on PTO | 16:05 |
| rlandy | zbr|ruck: ysandeep|away: ^^ fyi | 16:05 |
| rlandy | for the new container job | 16:05 |
| weshay_ruck | rlandy, I will.. I'm hoping for a master promotion.. to help w/ minion | 16:05 |
| weshay_ruck | zbr, so are you +1/-1 on https://review.rdoproject.org/r/#/c/28482/ | 16:06 |
| rlandy | weshay_ruck: hmmm .. maybe we should put back the 10 mins until master promotes | 16:07 |
| rlandy | then replace with 12 hr wait | 16:07 |
| zbr | weshay_ruck: added -1 with link to https://review.rdoproject.org/r/28487 which should do the trick. | 16:08 |
| zbr | we forgot to add basepython on the delegated one, likely this was the cause. | 16:08 |
| weshay_ruck | gate's gong to reset on my minion nv patch | 16:09 |
| rlandy | lunch ... brb | 16:11 |
| *** saneax has joined #oooq | 16:13 | |
| *** ccamacho has quit IRC | 16:17 | |
| *** creepy_owlet is now known as dtantsur|afk | 16:19 | |
| *** udesale_ has quit IRC | 16:19 | |
| *** marios is now known as marios|out | 16:19 | |
| *** saneax has quit IRC | 16:31 | |
| zbr | weshay_ruck: better for me to tell you no, than infra, read tripleo-ci-centos-8-containers-undercloud-minion | 16:32 |
| zbr | https://review.opendev.org/#/c/740324/1 | 16:32 |
| zbr | they will complain about that | 16:32 |
| weshay_ruck | ya.. | 16:32 |
| weshay_ruck | I know :) | 16:32 |
| zbr | when you make a job nv, you must remove it from gate. | 16:32 |
| weshay_ruck | ah.. yes | 16:32 |
| zbr | don't give them reasons. | 16:33 |
| weshay_ruck | +1 reminder | 16:33 |
| zbr | are you fixing it, or should I update it? | 16:33 |
| weshay_ruck | zbr, fixed | 16:35 |
| *** marios|out has quit IRC | 16:47 | |
| *** dmellado has joined #oooq | 16:50 | |
| weshay_ruck | rlandy, k.. compute component in master is going full green I think.. unless one job times out | 16:50 |
| weshay_ruck | oh wait | 16:50 |
| rlandy | weshay_ruck: cool | 16:50 |
| weshay_ruck | that's train | 16:50 |
| weshay_ruck | rlandy, it's green | 16:50 |
| rlandy | it's not easy to be green | 16:50 |
| weshay_ruck | ok.. kermit | 16:51 |
| *** dmellado has quit IRC | 17:04 | |
| *** dmellado has joined #oooq | 17:08 | |
| *** derekh has quit IRC | 17:09 | |
| *** rascasoft has quit IRC | 17:12 | |
| zbr | weshay_ruck: this, https://review.rdoproject.org/r/#/c/28487/ run passed, but did not trigger the job that was broken. | 17:21 |
| raukadah | zbr: testing here https://review.rdoproject.org/r/#/c/28483/ | 17:23 |
| *** dmellado has quit IRC | 17:26 | |
| *** jfrancoa has quit IRC | 17:33 | |
| *** Goneri has joined #oooq | 17:54 | |
| *** Goneri has quit IRC | 18:00 | |
| *** soniya29 has quit IRC | 18:09 | |
| rlandy | weshay_ruck: hmm ... ok we're going to have to fake out dlrn - https://logserver.rdoproject.org/25/25325/29/check/periodic-tripleo-ci-centos-8-standalone-baseos-centos8cr-master/b4ea5ec/job-output.txt | 18:19 |
| weshay_ruck | rlandy, /me looks | 18:58 |
| weshay_ruck | rlandy, oh.. on the reporting | 18:58 |
| weshay_ruck | I see | 18:59 |
| rlandy | weshay_ruck: not a big deal | 19:00 |
| rlandy | it's looking for a hash | 19:00 |
| rlandy | we just need a condition in get hash to deal with this repo | 19:00 |
| weshay_ruck | aye | 19:00 |
| weshay_ruck | rlandy, not sure if I totally grok this one.. when you have a sec https://review.opendev.org/#/c/736468/ | 19:04 |
| rlandy | looking | 19:04 |
| rlandy | weshay_ruck: lol - not so clear but I heard sandeep talk about it | 19:05 |
| rlandy | weshay_ruck: I assume that a single playbook carries the vars? | 19:06 |
| rlandy | I don;t really get it either | 19:06 |
| rlandy | weshay_ruck: design question ... to determine if we are working with a component outside dlrn ... | 19:55 |
| rlandy | set dlrn_server to be null? | 19:56 |
| rlandy | "" | 19:56 |
| rlandy | if we do that the jobs wont report | 19:56 |
| rlandy | do you want the jobs in dlrn | 19:56 |
| rlandy | if so, np, but we must pick another condition | 19:56 |
| rlandy | like component has baseos in it? | 19:56 |
| rlandy | if dlrn server is '', take md5 sum of repo to get hash | 19:57 |
| *** Goneri has joined #oooq | 20:01 | |
| *** TrevorV has quit IRC | 20:01 | |
| weshay_ruck | rlandy, ya.. I think we turn off reporting as a compoent | 20:28 |
| weshay_ruck | rlandy, and switch to report the build found in current-tripleo or tripleo-ci-testing | 20:28 |
| weshay_ruck | perhaps turn off the fact that it's a component? | 20:28 |
| weshay_ruck | this is just an idea | 20:28 |
| rlandy | idk ... advantages to using what we have | 20:30 |
| rlandy | if we can get get_hash to work with it | 20:30 |
| rlandy | trying out the md5sum idea | 20:30 |
| weshay_ruck | rlandy, quick question | 20:30 |
| rlandy | yep | 20:30 |
| weshay_ruck | rlandy, in ovb is the centos image downloaded from mirrors and uploaded to glance? | 20:31 |
| rlandy | weshay_ruck: not sure which image you are taking about | 20:31 |
| rlandy | in ovb we generally download image | 20:32 |
| rlandy | nothing to do with glance | 20:32 |
| weshay_ruck | rlandy, really? | 20:32 |
| weshay_ruck | can you join me in #vexxhost for a min | 20:32 |
| rlandy | ok | 20:32 |
| weshay_ruck | rhcos? | 20:32 |
| weshay_ruck | does that ring a bell? | 20:32 |
| rlandy | you're correct core os | 20:34 |
| rlandy | never used that | 20:34 |
| rlandy | you hear it kicked around at company meetings | 20:34 |
| rlandy | why are we all on the same tenant? | 20:35 |
| rlandy | weshay_ruck: afaict, not us | 20:35 |
| weshay_ruck | ya.. rhcos is openshift | 20:35 |
| rlandy | NOT GUILTY | 20:36 |
| weshay_ruck | I PLEAD | 20:37 |
| rlandy | Failed to attach VIF 089237a6-d4ef-4780-8d29-3f09a1020bd4 to bare metal node | 20:45 |
| * rlandy runs aways | 20:45 | |
| *** holser_ has quit IRC | 20:50 | |
| rlandy | weshay_ruck: what do you want to do about the promoter timing? | 20:50 |
| *** holser has joined #oooq | 20:51 | |
| rlandy | missing jobs ['periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master'] | 20:51 |
| * rlandy reruns | 20:51 | |
| *** Goneri has quit IRC | 20:51 | |
| rlandy | weshay_ruck: ha https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/cad44ac/logs/undercloud/var/log/tempest/stestr_results.html.gz | 20:56 |
| rlandy | Body: b'{"faultcode": "Client", "faultstring": "Policy does not allow this request to be performed.", "debuginfo": null}' | 20:56 |
| rlandy | now that scenario010 failure is blocking master | 20:57 |
| rlandy | weshay_ruck: so back to promotions ... | 21:48 |
| rlandy | change the time back or no? | 21:49 |
| *** Goneri has joined #oooq | 21:49 | |
| *** rlandy has quit IRC | 22:01 | |
| *** dmellado has joined #oooq | 22:20 | |
| *** tosky has quit IRC | 23:36 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!