Wednesday, 2022-02-09

« Tuesday, 2022-02-08 Index Thursday, 2022-02-10 »
*** dviroel|ruck|afk is now known as dviroel|ruck00:48
*** rlandy|ruck|bbl is now known as rlandy|ruck00:57
*** dviroel|ruck is now known as dviroel|ruck|out00:57
*** dviroel|ruck|out is now known as dviroel|out00:57
*** rlandy|ruck is now known as rlandy|out04:06
*** amoralej|off is now known as amoralej08:11
amoralejrlandy|out, ack, sorry for that i'll prepare a review to reproduce it before moving to it again08:14
*** jpena|off is now known as jpena08:31
*** sshnaidm|afk is now known as sshnaidm08:54
*** ysandeep|out is now known as ysandeep09:01
mariosneed reviews please https://review.opendev.org/c/openstack/tripleo-ci/+/828110 thank you 09:19
ysandeepmarios: looking09:25
mariosthanks ysandeep 09:30
ysandeeppojadhav, pm09:44
*** rlandy|out is now known as rlandy|ruck11:06
rlandy|ruckamoralej: thanks11:07
rlandy|ruckyeah dviroel|out and I looked at that enable opstools section11:07
rlandy|ruckI am not sure why it is not enabled11:08
*** dviroel|out is now known as dviroel|ruck11:10
mariosrlandy|ruck: this is the one i was referring to https://review.opendev.org/c/openstack/tripleo-quickstart/+/82812611:19
soniya29arxcruz, kopecmartin, raukadah, ysandeep, rlandy|ruck, please add/edit the today's agenda for tempest meeting:- https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ11:27
rlandy|ruckhttps://review.opendev.org/c/openstack/octavia-tempest-plugin/+/81636911:33
mariosrlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1959014/comments/1111:33
rlandy|ruckdviroel|ruck: akahat|rover: let's sync11:35
rlandy|ruckmeet.google.com/vzm-nrah-qqf11:35
rlandy|ruckamoralej: chatted with marios11:35
rlandy|ruckwe're going to update our tripleo-quickstart release files11:36
rlandy|ruckso we also enable opstools11:36
amoralejyes, currently it's duplicated11:36
amoralejbetween delorean-deps.repo and quickstart-centos-opstools11:36
amoralejactually, the quicstart one is disabled in providers11:37
amoralejso, the plan is to enable quickstart one and disable the one in delorean-deps.repo?11:37
mariosamoralej: yeah we think the issue surfaced when we switched c8 container&image builds touse the release files (previously using some other code in tripleo-repos)11:38
amoralejahh, got it11:39
mariosamoralej: so we need something similar /along the same lines like https://opendev.org/openstack/tripleo-quickstart/src/commit/2ce46f8205098327b0dbf2cec08d16bd52638296/config/release/tripleo-ci/CentOS-9/master.yml#L58-L66 11:39
mariosamoralej: so it doesn't get disabled 11:39
amoralejmarios, why not simply keeping it enabled by default?11:41
amoralejgetting it enabled shouldn't break anything11:42
mariosamoralej: not sure ... i guess nothing needs it during the deployment 11:42
mariosamoralej: and we only needed it for containers11:42
amoralejthat's right, but i'd help to keep things easier11:42
mariosamoralej: i don't know why it was decided to disable it on the normal run other than 'it isn't needed' ? maybe that is reason enough 11:42
marios:D11:42
amoralejmaybe :)11:42
amoralejso that fixes the enablement of that repo11:43
amoralejbut not the duplicated11:43
amoralejactually, efectively, we've being having the repo enabled in all jobs via delorean-deps.repo11:43
mariosamoralej: well the duplicated one should be disabled or set priority ... ah right ;) ^^ 11:43
rlandy|ruckarxcruz: can you take a look at: https://bugs.launchpad.net/tripleo/+bug/196035511:44
amoralejmarios, so, there is no point in adding logic to enable/disable in quickstart but keeping it always enabled in delorean-deps 11:47
mariosamoralej: was it alwyas in delorean-deps or some recent addition? yeah i thnk your suggestion is OK if they are the same content frmo deps vs the 8-stream one11:48
amoraleji was checking exactly that :)11:49
amoralejit has been in delorean-deps.repo since the beginning of centos8 at least11:49
amoralejhttps://review.rdoproject.org/r/c/rdo-infra/puppet-dlrn/+/2650811:49
mariosamoralej: ack 11:49
akahat|roverrlandy|ruck, https://review.rdoproject.org/r/c/testproject/+/3675611:50
mariostripleo ci o/ need reviews please https://review.opendev.org/c/openstack/tripleo-ci/+/828110 thank you 11:52
rlandy|ruckamoralej: just discussing with ruck/rovers11:52
rlandy|ruckwe are going o do what is in the centos-9 release files11:52
mariosrlandy|ruck: see 13:47 < amoralej> marios, so, there is no point in adding logic to enable/disable in quickstart but keeping it always enabled in delorean-deps 11:52
akahat|roverhttps://logserver.rdoproject.org/56/36756/28/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/d848579/logs/supplemental/home/centos/ipa_prep.sh.log.txt.gz 11:52
rlandy|ruckremove it from our release files?11:52
mariosrlandy|ruck: i think it makes sense.. so could do that instead and just make sure it isn't disabled 11:53
rlandy|ruckI think there is a diff whether there is a change to build or not11:53
mariosrlandy|ruck: yeah 'remove from our release files' will remove the duplicate11:53
akahat|roverfailed: mirror issues.11:53
mariosrlandy|ruck: so maybe do the easiest thing first to unblock the gate11:53
mariosrlandy|ruck: and add the conditional 11:53
mariosrlandy|ruck: hmm... not sure though we will have a duplicate... though it should be the same content11:53
marioswhich should not be a problem 11:53
amoralejmarios, but the gates are not longer blocked, right?11:54
mariosamoralej: ah right yes rlandy|ruck merged a revert correct11:54
amoralejyou scared me :)11:54
mariosbut i wonder... do we have delorean-deps in container build? in that case why did i need to explicitly enable it with https://opendev.org/openstack/tripleo-ci/src/commit/f052b61dbbd229e1776d3f34c2252ad636b7f78a/roles/tripleo-build-jobs-repos/tasks/load-vars.yaml#L16 11:55
mariosi.e. in that case it wasn't using the delorean-deps one but instead i had to explicitly enable the opstools repo11:56
amoraleji'd say, if we don't have delorean-deps in containers builds, provider jobs whould have been failing as there were not opstools repo enabled11:57
marioschecking sthing sec 11:57
amoralejalso, i wouldn't break those builds jobs with my change in delorean-deps ...11:57
arxcruzrlandy|ruck hmmm, never saw this error, i might need more time to debug12:00
mariosrlandy|ruck: amoralej: in 9 we don't have a duplicate opstools repo only for 8 12:00
mariosamoralej: it seems we are relying on the delorean-deps for 9?12:00
mariosamoralej: rlandy|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-9-push-master/3795657/etc/yum.repos.d/ 12:00
mariosamoralej: rlandy|ruck: no 'quickstart-centos-opstools' repo defined inhttps://opendev.org/openstack/tripleo-quickstart/src/commit/2ce46f8205098327b0dbf2cec08d16bd52638296/config/release/tripleo-ci/CentOS-9/master.yml#L198 12:01
amoralejmmm12:01
amoralejthat reminds me i need to move from buildlogs to mirror.centos.org12:02
mariosamoralej: so it is what you suggested i.e. only using the delorean-deps12:02
mariosrlandy|ruck: but then i wonder what that is meant to do https://opendev.org/openstack/tripleo-quickstart/src/commit/2ce46f8205098327b0dbf2cec08d16bd52638296/config/release/tripleo-ci/CentOS-9/master.yml#L198 dviroel|ruck we don't have quickstart-centos-opstools' repo defined 12:02
mariosinhttps://opendev.org/openstack/tripleo-quickstart/src/commit/2ce46f8205098327b0dbf2cec08d16bd52638296/config/release/tripleo-ci/CentOS-9/master.yml#L19812:02
soniya29arxcruz, kopecmartin, facing some network issues12:02
* marios stops spamming channel.12:02
mariosrlandy|ruck: dviroel|ruck: let me know if you want me to poke more at this otherwise i'll let you propose something and we can discuss on gerrit12:03
amoralejmarios, but note it's using the variable to enable/disable the repo in delorean-deps.repo12:04
amoralejhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-9/master.yml#L19812:04
amoralejcentos9-opstools that's the one in delorean-deps.repo12:04
mariosamoralej: ah i see it 12:04
rlandy|ruckreading back12:04
mariosamoralej: right makes more sense... but confirms what you suggest anyway we don't actually even need the extra definition in there just use deps12:05
amoralejso if you check in some other job which is not building containers12:05
amoralejthat repo should be disabled12:05
amoralejmarios, yep, i'd keep it simple by using delorean-deps.repo12:05
rlandy|ruckmarios: can you join review time again?12:05
rlandy|ruckon with dviroel|ruck 12:05
mariosrlandy|ruck: sure on my way 12:05
amoralejthen, if we want to be able to enable/disable, let's do similar to centos912:05
soniya29rlandy|ruck, ysandeep, raukadah, i am not able to join the meeting, facing network issues12:06
rlandy|ruckamoralej: ack - we just discussed ...12:11
rlandy|ruckstep 1: make centos-8 release files match c912:11
rlandy|ruckstep 2: investigate dropping the disable/enable line12:11
rlandy|ruckso there will no longer be a dup12:11
mariosto be clear amoralej we're doing what you suggested... 'make 8 like 9' means remove the extra opstools repo defined in the 8 files and just use the deps one12:12
amoralejok, nice12:12
amoralejactually, mmm let me see if there is any other duplication12:12
amoralejwe may have more12:13
amoralejcentos9 lgtm12:14
amoralejyeah, i think opstools is the only duplication12:15
rlandy|ruckakahat|rover: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/ec13046365624fc29a4a65a17cc97a63?logfile=console.log - the last run on c9 kvm job failed12:21
rlandy|ruckso the fix was not yet in master12:21
rlandy|ruckwe will have to track those hashes12:21
rlandy|ruckmaybe when octavia component promotes12:21
*** ysandeep is now known as ysandeep|break12:41
rlandy|ruckdviroel|ruck: akahat|rover: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39240 Temp remove fs002 from c8 master criteria 12:44
dviroel|ruckrlandy|ruck: marios https://review.opendev.org/c/openstack/tripleo-quickstart/+/828522 - I trying to make all c8 files work with the same logic; deleted the other opstools repo; let me know if makes sense12:47
rlandy|ruckdviroel|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart/+/828522 - left comment12:47
rlandy|ruckre: promoting-testing-hash files12:47
rlandy|ruckthe c9 ones do not have that clause12:47
rlandy|ruckidk if they should or should not12:47
rlandy|ruckmarios can weigh in there12:48
rlandy|ruckakahat|rover: thanks for putting in the fs039 rerun12:48
rlandy|ruckakahat|rover: anything else we missing ruck/rover -wise in your morning?12:48
akahat|roverrlandy|ruck, no.12:49
mariosrlandy|ruck: k will check the review dviroel|ruck 12:49
dviroel|ruckrlandy|ruck: yes, i'm thinking to make things follow the same logic in all files - not sure why we have in some files and other don't12:50
rlandy|ruckmystery of the week12:50
rlandy|ruckto opstools or not to opstools12:50
rlandy|ruckakahat|rover: ah - octavia component should run next12:51
rlandy|ruckwe will see those new fixes clear then  - I hope12:51
dviroel|ruckrlandy|ruck: yeah, maybe doesn't have because is harmless to have it always enabled12:51
rcastilloo/13:05
rlandy|ruckrcastillo: hey13:07
rlandy|ruckrcastillo: master c9 promoted 13:07
rlandy|ruckand the ipa job should be passing now13:07
*** amoralej is now known as amoralej|lunch13:07
rlandy|ruckcan you add the patch to get that voting/gating again?13:07
rlandy|ruckrcastillo: ^^13:07
rcastillorlandy|ruck: ack13:10
*** ysandeep|break is now known as ysandeep13:13
*** dasm|off is now known as dasm13:17
rcastillorlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/82852413:17
dasmo/13:17
rlandy|ruckrcastillo: thank you13:27
rlandy|ruckakahat|rover: hey ... can you help out with promoter question ...13:27
rlandy|ruckhttps://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/job/rdo_trunk-promote-wallaby-centos8-current-tripleo/88/13:28
rlandy|ruckpassed13:28
rlandy|ruckall13:28
akahat|roverrlandy|ruck, yes ..13:28
rlandy|ruckso we should have promoted wallaby c8 current-tripleo-rdo13:28
rlandy|ruckFeb 613:28
rlandy|ruckhttp://promoter.rdoproject.org/config/CentOS-8/13:29
rlandy|ruckakahat|rover: ah ... we need to remove the packstack jobs from criteria13:30
rlandy|ruckhmm .. no13:30
rlandy|ruckthose ran13:30
akahat|roverrlandy|ruck, weirdo-*-packstack jobs?13:30
rlandy|ruckakahat|rover: no - I was wrong13:31
rlandy|ruckthe jobs look right13:31
rlandy|ruckbut we didn't promote13:31
rlandy|ruckcurrent-tripleo-rdo/2022-01-27 22:42 13:31
rlandy|ruckwhy?13:31
rlandy|ruckshould have promoted feb 6th13:31
rlandy|ruckakahat|rover: see what I am looking at13:32
rlandy|ruckcan you trace that promoter log?13:32
akahat|roverrlandy|ruck, yes. looking in it.13:33
mariostripleo ci o/ need reviews please https://review.opendev.org/c/openstack/tripleo-ci/+/828110 thank you 13:34
rlandy|ruckakahat|rover: ussuri as well13:40
rlandy|ruckhttps://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/job/rdo_trunk-promote-ussuri-centos8-current-tripleo/13:40
rlandy|ruckbut we're not promoting13:40
rlandy|ruckamoralej|lunch: hey ... can you help me understand the dlrn hash reporting here ... 13:51
rlandy|ruckhttps://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/rdo-promote-get-hash-ussuri-centos8-current-tripleo/65/console13:51
rlandy|ruck+ export NEW_HASH=7d23a38590b5cf6750e96a41ca2f22ca13:52
rlandy|ruck+ NEW_HASH=7d23a38590b5cf6750e96a41ca2f22ca13:52
rlandy|ruck https://trunk.rdoproject.org/centos8-ussuri/current-tripleo/delorean.repo.md513:52
rlandy|ruckhttps://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal/93/console13:53
rlandy|ruckreports dlrn ...13:53
rlandy|ruck dlrnapi --url https://trunk.rdoproject.org/api-centos8-ussuri --username <> report-result --job-id tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal --agg-hash 665b36551e955e31e5ed4c82108b17bd 13:54
rlandy|rucktwo diff hashes13:54
ysandeeppojadhav, bhagyashris rlandy|ruck takashi is proposing patches in puppet related repos to move to c9.13:54
rlandy|rucknice13:54
ysandeeppojadhav, rlandy|ruck bhagyashris marios fyi.. https://review.opendev.org/q/topic:tripleo-c9 for puppet related repos13:55
mariosthanks ysandeep 13:56
*** amoralej|lunch is now known as amoralej13:59
akahat|roverrlandy|ruck, only promotion we got current-tripleo-rdo: http://promoter.rdoproject.org/promoter_logs/centos8_wallaby_2022-01-27T22:42.log-2022012814:06
akahat|roverfor wallaby ^^14:06
pojadhavysandeep, ack14:06
*** pojadhav is now known as pojadhav|brb14:06
akahat|roverand ussuri current-tripleo-rdo: http://promoter.rdoproject.org/promoter_logs/centos8_ussuri_2022-01-27T05:35.log-2022012814:06
* akahat|rover leaving for the day. 14:11
*** akahat|rover is now known as akahat|PTO14:11
arxcruzrlandy|ruck regarding https://bugs.launchpad.net/tripleo/+bug/1960355 i believe it was a race condition, i check the logs and I see no error, also, the latest fs035 run fails in a complete different issue 14:16
arxcruzdviroel|ruck ^14:16
arxcruzhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/8d6c815/logs/undercloud/var/log/tempest/stestr_results.html.gz14:16
arxcruzoh, nevermind, the tests were removed 14:17
dviroel|ruckarxcruz: yep, skipping14:17
dviroel|ruckskip or never promote again14:18
dviroel|ruckarxcruz: yesterday i was comparing the latest success one with the first failure14:19
arxcruzdviroel|ruck there's a server internal error14:19
dviroel|ruckarxcruz: last success was still using ctrlplane ip14:19
arxcruzprobably a bug in keystone/keystonemiddleware 14:19
arxcruzdviroel|ruck but it's being able to do a post and create the token 14:19
arxcruzbut it's being invalidated 14:20
arxcruzit seems to be an error in keystone service14:20
dviroel|ruckarxcruz: yeah, i think that i saw it invalidating the token14:20
arxcruzdviroel|ruck yup on the delete 14:21
rlandy|ruckfrenzy_friday: https://github.com/openstack/tripleo-quickstart/tree/master/config/release14:21
rlandy|ruckhttps://codesearch.opendev.org/?q=trunk.registry.rdoproject.org&i=nope&literal=nope&files=&excludeFiles=&repos=14:22
rlandy|ruckhttps://review.rdoproject.org/codesearch/?q=trunk.registry.rdoproject.org&i=nope&files=&repos=14:22
dviroel|ruckarxcruz: the request after 500 error14:22
dviroel|ruckarxcruz: "500 POST https://[2001:db8:fd00:1000::5]:13000/v3/OS-OAUTH1/request_token"14:22
arxcruzdviroel|ruck 2022-02-08 19:00:28.790 149 WARNING keystone.server.flask.application [req-8e2bc385-ce46-4eaf-a36c-ce8d9fc414a1 6f72adab6a0f4209935f9f2c53c64bef 1bd34489e9814280b3e27f11a32acc22 - default default] Invalid mapping id: 9dc88feb7d1e41cf8d4b060912c6785d: keystone.exception.ValidationError: Invalid mapping id: 9dc88feb7d1e41cf8d4b060912c6785d14:23
dviroel|ruckarxcruz: this error comes after request_token, right?14:23
rlandy|ruckhttps://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/multinode-vexxhost.yml#L5214:24
*** pojadhav|brb is now known as pojadhav14:24
rlandy|ruckcommand: "podman pull docker://trunk.registry.rdoproject.org/ceph/{{ item.split('/')[-1] }}"14:25
arxcruzdviroel|ruck that's the thing, i can't tell you, there's not much info there14:28
mariosrlandy|ruck: ok to merge that ? (reviewed but wasnt sure what the story was on that) https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39181/1#message-5cbc3efafcf8266b2a8923c51304a9d5a5812cd5 14:28
rlandy|ruckmeeting ... reading back14:29
rlandy|ruckmarios: yeah - idk what it will mess up tbh14:29
rlandy|ruckI could hose some stuff here14:30
mariosrlandy|ruck: k wfing14:30
arxcruzdviroel|ruck what I was able to find, in the same timestamp was this:14:31
arxcruzdviroel|ruck 2022-02-08 19:00:30.293 117 DEBUG keystone.auth.core [req-8b5a104c-9bb6-463a-9e65-b8a2e1e7ba45 - - - - -] MFA Rules not processed for user `6f72adab6a0f4209935f9f2c53c64bef`. Rule list: `[]` (Enabled: `True`). check_auth_methods_against_rules /usr/lib/python3.9/site-packages/keystone/auth/core.py:43814:31
arxcruzand checking the code, it throws an exception if not found14:31
rlandy|ruckmarios: basically, we only have the option to have one file per release tested in ci.centos14:31
arxcruzbut i couldn't find the error 500 14:31
rlandy|ruckthere was no way for master on c8 and c914:31
rlandy|ruckso that's basically an attempt at creating the same dir structure as tripleo-ci14:31
rlandy|ruckso we have the option14:31
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/82817114:32
rlandy|ruckmarios: ^^ would require this to take effect14:33
mariosrlandy|ruck: ack adding to my reviews but it is still wip 14:33
rlandy|rucklet me unwip that14:33
rlandy|ruckmarios; the one you merge is just an extra var atm14:33
rlandy|ruckso it should be ok14:34
dviroel|ruckarxcruz: ack, please comment in LP - thanks 14:35
mariosrlandy|ruck: ack 14:36
rlandy|ruckmarios: https://review.opendev.org/c/openstack/tripleo-quickstart/+/828171 - updated commit message there14:36
mariosrlandy|ruck: thanks14:38
*** pojadhav is now known as pojadhav|dinner15:00
rlandy|ruckdviroel|ruck: hi - need anything? ... otherwise will continue looking into current-tripleo-rdo promotions15:01
rlandy|ruckysandeep: ^^ need anything for downstream?15:01
rlandy|ruckdasm: let's touch base re: uefi boot ...15:02
rlandy|ruckit's working on 17?15:02
rlandy|ruckand we confirmed we are actually using that boot mode?15:02
rlandy|ruckwhen we use the uefi ipxe boot image?15:02
dasmrlandy|ruck: logs show it's using right boot type: uefi. 15:03
rlandy|ruckdasm++ nice15:03
dasmbut i haven't had a successful test run yet15:04
rlandy|ruckysandeep; ^^ thought about moving to that boot mode for 17?15:04
rlandy|ruckdasm: next step is to try the same with fs001 wallaby job15:04
rlandy|ruckc8 and c915:04
ysandeeprlandy|ruck, need anything for downstream? -> please see pcci chat channel on internal15:04
rlandy|ruckor master15:04
rlandy|rucklooking15:05
dasmok. i'll add that and i'll see if it's running15:05
rlandy|ruckdasm: awesome15:05
rlandy|ruckif that works,15:06
rlandy|ruckwe will need to make some similar trigger arrangement like scenario010-kvm15:06
dasmok15:06
rlandy|ruckwhere we trigger the job in downstream to run with upstream specified hash15:06
rlandy|ruckTengu: ^^ fyi15:07
rlandy|ruckuefi boot testing in progress by dasm 15:07
dasmrlandy|ruck: are you talking about this one? "periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-wallaby" ?15:10
rlandy|ruckdasm: w/o baremetal - that's a component job15:11
dasmk15:11
rlandy|ruckjust periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby15:11
rlandy|ruckor c915:11
dasmack15:11
Tengurlandy|ruck: uefi boot failing here with downstream.15:16
dviroel|ruckrlandy|ruck: no, nothing for now15:23
* dviroel|ruck was on lunch15:23
ysandeeprlandy|ruck, +1 to moving to uefi, But I am not in sync with progress dasm made.. If you want to meet and discuss - I am available/ share in tomorrow's scrum15:26
ysandeepTengu, did uefi boot finally worked for you on wallaby?15:27
Tenguysandeep: I didn't test wallaby, it was plain master.15:28
Tenguysandeep: but Kevin is working with wallaby on cs9 nowadays.15:28
TenguYou want to ping him15:28
ysandeepTengu, ack.. thanks!15:29
Tengunp15:29
dasmysandeep: sure. I'm still checking few things, but it seems to be working with small caveats.15:31
amoralejrlandy|ruck, what's the question?15:36
rlandy|ruckamoralej: it looks like the hash being reported in rdo-promote-get-hash-ussuri-centos8-current-tripleo is not the same hash reported on15:37
rlandy|ruckin https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal/93/15:37
amoralejmmm lemme check those jobs15:37
rlandy|ruckunless I am misreading this15:37
rlandy|ruck+ export NEW_HASH=7d23a38590b5cf6750e96a41ca2f22ca15:37
rlandy|ruckvs15:37
ysandeepdasm, great.. lets discuss more in tomorrow's scrum15:37
rlandy|ruckwhat is reported15:38
rlandy|ruckin dlrn 15:38
amoraleji need to page out stuff from my swap :)15:38
amoralej+that comes from old days...15:38
rlandy|ruckso we are not promoted15:38
dasmysandeep, sure!15:41
rlandy|ruckdviroel|ruck: we can ping with card on #sec-interop15:43
*** ysandeep is now known as ysandeep|out15:43
rlandy|ruckthe caesar there15:44
dviroel|ruckrlandy|ruck: ack15:46
rlandy|ruckrerunning failed octavia tests15:47
rlandy|ruckamoralej: does that make sense?16:00
rlandy|rucktrying to match if we are reporting all jobs on the same hash16:00
rlandy|ruck echo export AGG_HASH=7d23a38590b5cf6750e96a41ca2f22ca16:01
rlandy|ruckhttps://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/weirdo-ussuri-centos8-promote-packstack-scenario001/65/consoleFull16:01
amoralejrlandy|ruck, nop16:01
amoraleji'm trying to understand16:01
rlandy|ruckso the weirdo job reports on the right hash16:02
amoralejso, we didn't get any promotion after that?16:02
rlandy|ruckno ... 16:02
rlandy|ruckso I think all the weirdo jobs do the right thing  - and report on the right hash16:02
amoralejmmm16:02
rlandy|ruckI think the quickstart job does not16:03
amoralejso the get-hash jobs and the oooq one take the hashes independently16:03
amoralejgimme some minutes please16:03
rlandy|rucksure16:05
rlandy|ruckok - so I think this could happen if we promote current-tripleo16:08
rlandy|ruckwhile the job is running16:08
rlandy|ruck665b36551e955e31e5ed4c82108b17bd/2022-02-08 19:16 16:09
rlandy|ruck7d23a38590b5cf6750e96a41ca2f22ca/2022-02-06 04:23 16:09
rlandy|ruckquickstart job pulls DELOREAN_URL16:09
rlandy|ruckwhich can change16:10
amoralejyes16:10
amoraleji was about to say that16:10
amoralejthat's what happened16:10
amoralejrlandy|ruck, now i have the doubt which repo was actually testing the job16:11
amoralejin weirdo jobs we are passing the hash as a parameter to the job16:12
rlandy|ruckack16:12
rlandy|rucklooking at how that gets done16:12
amoralejso it tests and reposts in the same one even if there is a promotion in the middle16:12
amoraleji'd saly...16:13
amoralejsay16:13
amoralejin the oooq one i see it gets the agg hash at reporting time16:14
amoralejfrom delorean.repo.md5, but i'm not sure how it picks the delorean-repo at build time16:14
*** pojadhav|dinner is now known as pojadhav16:14
rlandy|ruckyeah - trying to figure that out now16:16
rlandy|rucklooking at those jobs16:16
* rlandy|ruck think the release file just resolves it16:17
rlandy|ruckchecking repos16:17
amoraleji'd say it used the good one16:17
amoralej7d2316:17
amoralejlook at ok: [172.19.3.90] => {"attempts": 1, "changed": false, "commit_hash": null, "distro_hash": null, "dlrn_url": "https://trunk.rdoproject.org/centos8-ussuri/current-tripleo/delorean.repo.md5", "error": "", "extended_hash": null, "full_hash": "7d23a38590b5cf6750e96a41ca2f22ca", "success": true}16:17
amoralejrepo-setup : tripleo-get-hash current-tripleo 16:18
amoralejTASK [repo-setup : Set fact dlrn_hash->7d23a38590b5cf6750e96a41ca2f22ca] *******16:18
rlandy|ruckok: [172.19.3.90] => {16:21
rlandy|ruck    "msg": [16:21
rlandy|ruck        "the tq release is: ussuri",16:21
rlandy|ruck        "the distro is: centos8",16:21
rlandy|ruck        "dlrn_hash is: 7d23a38590b5cf6750e96a41ca2f22ca"16:21
rlandy|ruck    ]16:21
rlandy|ruck}16:21
rlandy|ruckbut it reports on the newer hash16:21
amoralejbecause to report16:21
amoralejit pulls the .repo.md5 from current-tripleo16:21
rlandy|ruckcorrect16:22
rlandy|ruckit should take whatever was passed16:22
rlandy|ruckugh - so complicated16:22
amoralejtoo much16:23
amoralejcomplexity everywhere :)16:24
rlandy|ruckhttps://github.com/rdo-infra/ci-config/blob/master/jenkins-v2/jobs/scripts/dlrn-api-report.sh16:24
rlandy|ruckhttps://github.com/rdo-infra/ci-config/blob/master/jenkins-v2/jobs/scripts/dlrn-api-report.sh#L716:24
rlandy|ruck AGG_HASH=`curl -L ${DELOREAN_URL}.md5`16:24
rlandy|ruck^^ actually that16:24
rlandy|rucklooking for the var that is passed16:25
rlandy|ruck HASH_FILE="$WORKSPACE/delorean_master_centos9_current_tripleo_hash"16:27
rlandy|ruckCopied 1 artifact from "rdo-promote-get-hash-ussuri-centos8-current-tripleo" build number 6516:31
rlandy|ruck[EnvInject] - Injecting environment variables from a build step.16:31
rlandy|ruckwhere does that go?16:31
rlandy|ruckecho export INFO_URL="https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/${JOB_NAME}/${BUILD_NUMBER}/artifact/logs.html" >> ${WORKSPACE}/${BUILD_ID}/dlrnapi-varfile16:32
rlandy|ruck^^ weirdo jobs16:32
rlandy|ruckSkipping script  : . ${WORKSPACE}/${BUILD_ID}/dlrnapi-varfile16:33
rlandy|ruckrm -rf ${WORKSPACE}/${BUILD_ID}16:33
rlandy|ruckexport JOB_SUCCESS="false"16:33
rlandy|ruckexport DLRNAPI_USER="ciuser"16:33
rlandy|ruck# Report16:33
rlandy|ruck# DLRNAPI_PASSWD is provided by a credential binding16:33
rlandy|ruckcd weirdo16:33
rlandy|rucktox -e ansible-playbook -- -i ${WORKSPACE}/hosts playbooks/dlrn-api-report.yml16:33
rlandy|ruck^ that's how weirdo gets it right16:33
rlandy|ruckok - will have to edit our jobs somehow to do the same16:33
rlandy|ruck<brain hurts>16:34
ykarelrlandy|ruck, amoralej i recall i attempted to fix ^ long back with https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/1043416:37
ykarelit got abandoned as obsoleted, so likely similar can be used to fix it16:37
amoralejwow 2017 :)16:39
* dviroel|ruck brb16:39
amoralejpromote-newton :)16:39
ykarel:D16:40
rlandy|ruckykarel: thanks - looking at that16:40
rlandy|ruckand in all these years nobody bothered to revive it16:40
rlandy|ruckmight still be relevant wrt https://bugs.launchpad.net/tripleo/+bug/1859213, will check jobs.16:41
rlandy|rucklol16:41
rlandy|ruckdviroel|ruck: woohoo https://review.rdoproject.org/zuul/stream/a30904dea33f4281bfd2c366ee90b6c6?logfile=console.log periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master16:42
rlandy|rucksuccess: true 16:42
rlandy|ruckmaster c8 promotion - here we come16:42
ykarelyeah that's my comment, but don't recall what all happened that time16:44
mariosrlandy|ruck: when you have time please see https://review.opendev.org/c/openstack/tripleo-ci/+/828110 - I'll workflow it in the morning if it is still around then ;) or if there are comments 16:45
ykarelbut seems this should work, but would need some adjustment due to introduction of agg_hash16:45
mariosrlandy|ruck: so we can start considering the child patch for c8 teardown by friday would be good 16:45
rlandy|ruckmarios: ack - thanks16:46
mariosbhagyashris: thanks for updating the topic on takashi patches ;)16:47
*** ykarel is now known as ykarel|away16:48
rlandy|ruckhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4-centos7 hmmm16:52
rlandy|ruckrekicking 16:52
*** marios is now known as marios|out16:53
rlandy|rucklunch ... brb16:54
*** jpena is now known as jpena|off17:27
dviroel|ruckmaster c8 promoting :) 17:31
rlandy|ruckwill revert the fs002 patch17:32
dviroel|ruckrlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ansible/+/828417 - merged17:33
dviroel|ruckfs002 should be good now17:33
rlandy|ruckack - I'll revert after promote goes through17:33
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/82852417:59
rlandy|ruck^^ vote there pls17:59
rlandy|ruckdviroel|ruck: ^^17:59
rlandy|ruckhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3880818:00
dviroel|ruckrlandy|ruck: promoter failed?18:00
rlandy|ruckanother vote pls18:00
rlandy|ruckdidn't promote18:01
rlandy|rucklooking18:01
rlandy|ruck2022-02-09 17:44:25,878 2013128 ERROR    promoter Candidate hash 'aggregate: 6fdbde2576bd48e3e9624ac5a1dd61f8, commit: a9058bb68beff87f08a3462163d25dd45b019cc2, distro: df726a5c620cd1d0b5dff471247268382bc0db5b, extended: None, component: validation, timestamp: 1644383082': client registries_client FAILED promotion attempt to current-tripleo18:02
rlandy|ruck2022-02-09 17:44:25,878 2013128 ERROR    promoter Failed to promote overcloud images18:02
rlandy|ruckTraceback (most recent call last):18:02
rlandy|ruck-02-09 17:44:25,874 2013128 ERROR    promoter Command '['env', 'ANSIBLE_LOG_PATH=/home/promoter/web/promoter_logs/container-push/20220209-165207.log', 'ANSIBLE_DEBUG=False', 'ansible-playbook', '-v', '-e', '@/tmp/tmpv840s1_1.yaml', '/home/promoter/ci-config/ci-scripts/container-push/container-push.yml']' returned non-zero exit status 2.18:02
rlandy|ruckTraceback (most recent call last):18:02
dviroel|ruck [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}18:03
dviroel|rucklove that one ^18:03
rlandy|rucklol18:05
rlandy|ruckit keeps the mystery alive18:05
rlandy|rucktrain8 is promoting now18:05
rlandy|ruckshould try again18:05
rlandy|ruckthen we will see if it's a real error18:05
rlandy|ruckleaving fs002 in skip until promotion is done18:06
dviroel|ruckack18:06
rlandy|ruckwoohoo 17 promoted18:11
*** amoralej is now known as amoralej|off18:17
dasmbrb18:44
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/82811518:52
rlandy|ruckdviroel|ruck: ^^ one more - pls w+ if you agree18:52
rlandy|ruckugh master promotion failed agin18:54
rlandy|ruckagain18:54
rlandy|ruckdviroel|ruck: ^^ real issue now18:54
rlandy|rucklooking18:54
rlandy|ruckhttp://promoter.rdoproject.org/promoter_logs/container-push/20220209-182943.log18:54
rlandy|rucknow to find the one image that failed to promote18:56
dviroel|ruckrlandy|ruck: ack - reviewing code - will help on debug in a min18:59
rlandy|ruckdviroel|ruck: no worries - I got a little time before next meeting18:59
rlandy|ruckso looking18:59
rlandy|ruckwill pass back to you then19:00
* dasm is back19:20
rlandy|ruck6fdbde2576bd48e3e9624ac5a1dd61f8 - no tag on that in openstack-ceilometer-central19:20
rlandy|ruckhttps://hub.docker.com/r/tripleomaster/openstack-ceilometer-central/tags19:45
rlandy|ruckit is there now19:45
dviroel|ruckrlandy|ruck: great :)20:00
rlandy|ruckon promoter20:00
rlandy|rucktrying to find unsanitized logs20:01
rlandy|ruckmeeting time20:01
rlandy|rucklooks like it's pushing again20:02
dasmrlandy|ruck: unfortunately i called shots too quickly. all tests running with uefi are failing due to issues with introspection of provisioned bare metal machines. It probably means they're just not starting correctly. Right now I'm going through logs and searching for some reliable information.20:11
rlandy|ruckk20:12
rlandy|ruckdviroel|ruck: hey - out of meeting20:46
dviroel|ruckrlandy|ruck: o/20:47
rlandy|ruckdviroel|ruck: any magic insights20:47
rlandy|ruckstill staring at nothing useful atm20:48
dviroel|ruckrlandy|ruck: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-train-containers-build-push&project=openstack/tripleo-ci20:48
dviroel|ruckreal issue20:48
* dviroel|ruck was fixing some molecule jobs in tripleo-ansible too20:49
rlandy|ruck| ImportError: No module named collections_abc20:51
dviroel|ruckvirtualenv issue in py2.720:51
rlandy|ruckah ok20:51
rlandy|ruckwas staring at master promotion logs20:51
dviroel|ruckrlandy|ruck: my guess - trying to see which version is20:52
rlandy|ruckbut can't see the issue yet20:52
rlandy|ruckok - let's bug the c7 one20:52
rlandy|ruckyou can pass it on to arx tomorrow20:52
dviroel|ruckyeah, does not reproduce in my vm20:58
dviroel|ruckhttp://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/7/updates/x86_64/Packages/python-virtualenv-15.1.0-5.el7_9.noarch.rpm21:06
dviroel|rucksame version that i installed here21:06
rlandy|ruckugh - still  no idea why container push is failing21:13
rlandy|ruckno obvious error21:14
rlandy|ruck2022-02-09 20:29:21,395 p=2147676 u=promoter n=ansible | TASK [containers-promote : Tag and push images to registry] ********************21:14
dviroel|ruckdid you enabled the logs?21:15
rlandy|ruckwas looking if there was a fuller copy elsewhere but I don't see it21:24
rlandy|ruckif I enable, it will spew all the info they want to hide to public21:25
dviroel|ruckmake sense, we would need to remove this log from public access and enable the log then21:28
rlandy|ruckhttps://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/container-push.yml#L6821:30
rlandy|ruck # Setting no_log as this can potentially expose passwords21:30
rlandy|ruckdviroel|ruck: going to leave a note for bhagyashris to remove no_log on other promoter server21:30
rlandy|ruckpromote master21:31
rlandy|ruckand store logs somewhere we can see them21:31
rlandy|ruckhttps://github.com/rdo-infra/ci-config/blob/4ff0a90c9de3002c64707d051b4ed9d77dd41ff8/ci-scripts/container-push/roles/containers-promote/tasks/registry-push.yml#L5621:32
rlandy|ruck^^ failing piece21:32
rlandy|ruckbut idk why21:32
* dasm is leaving for now. Be back later.21:33
* dasm dasm|bbl21:33
dviroel|ruckrlandy|ruck: ack, seems to be a good idea - hard to say what is happening - master c8 is refusing to promote in all ways it can21:35
rlandy|ruckit's a revolution21:35
rlandy|ruckhttps://hub.docker.com/r/tripleomaster/nop/tags21:39
rlandy|ruckdviroel|ruck: interesting21:39
rlandy|ruckdon;t know where that came from21:39
rlandy|ruckhttps://github.com/rdo-infra/ci-config/search?q=nop21:41
dviroel|ruckoh, just to validate login21:44
rlandy|ruckhonestly idk21:46
rlandy|ruckneed to stare at something else now21:46
dviroel|ruckrlandy|ruck: going out for today - tty tomorrow22:32
*** dviroel|ruck is now known as dviroel|out22:33
rlandy|ruckdviroel|out: sure good night22:36
*** rlandy|ruck is now known as rlandy|out23:23
« Tuesday, 2022-02-08 Index Thursday, 2022-02-10 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!