| *** marios is now known as marios|ruck | 05:16 | |
| jpodivin | IDENTIFY | 05:25 |
|---|---|---|
| jpodivin | huh, wrong chan | 05:25 |
| marios|ruck | needs a worfkow at https://review.opendev.org/c/openstack/tripleo-ci/+/797769 Revert "Set T->U undercloud upgrade to non-voting" | 06:23 |
| marios|ruck | please thank you | 06:23 |
| *** amoralej|off is now known as amoralej | 06:33 | |
| ysandeep | marios|ruck, ack checking | 06:54 |
| *** jpena|off is now known as jpena | 06:57 | |
| ysandeep | done | 06:58 |
| marios|ruck | thank you ysandeep | 06:58 |
| ysandeep | all thanks to you for working with Alex on fixing that :D | 07:00 |
| anbanerj|rover | marios|ruck, Hey, good morning | 07:36 |
| marios|ruck | hello anbanerj|rover | 07:37 |
| *** ykarel|away is now known as ykarel | 07:46 | |
| arxcruz | chandankumar: hey man, remind me, what was the naming we decided on the skiplist? rdo and osp right ? | 09:29 |
| chandankumar | arxcruz: yes, rdo for upstream and osp for downstream | 09:30 |
| anbanerj|rover | hey marios|ruck, going afk for 20-30 mins | 09:30 |
| * anbanerj|rover afk | 09:30 | |
| marios|ruck | thanks anbanerj|rover | 09:33 |
| *** ykarel is now known as ykarel|lunch | 09:38 | |
| *** bhagyashris_ is now known as bhagyashris | 09:40 | |
| * anbanerj|rover back | 10:01 | |
| anbanerj|rover | marios|ruck, I am checking victoria and ussuri promotions | 10:12 |
| marios|ruck | anbanerj|rover: thanks | 10:17 |
| *** ykarel|lunch is now known as ykarel | 10:43 | |
| *** jpena is now known as jpena|lunch | 11:32 | |
| anbanerj|rover | marios|ruck, for ussuri https://trunk.rdoproject.org/api-centos8-ussuri/api/civotes_agg_detail.html?ref_hash=9b443df90ea8b0b2504919cb4d5f8e23 looks good - it failed the sc010 jobs (actavia timeout) and only periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-ussuri | 11:43 |
| *** amoralej is now known as amoralej|lunch | 11:47 | |
| rlandy | ysandeep: hi ... https://trello.com/c/U5zMaO51/1919-cixbz1954108osp170rhel8rhos-17couldnt-resolve-host-name-for-http-downloaddevelredhatcom-rcm-guest-puddles-openstack-rhos-release | 11:51 |
| marios|ruck | anbanerj|rover: k thanks - i would say post a testproject for it but it won't help us until we fix https://bugs.launchpad.net/tripleo/+bug/1933448 (e.g. https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34275 ) | 11:51 |
| rlandy | ^^ still seeing that in fs001 but only fs001 | 11:51 |
| rlandy | fs020 | 11:51 |
| marios|ruck | anbanerj|rover: otherwise we can't override the dlrn_hash used/reported | 11:51 |
| rlandy | and fs035 pass | 11:51 |
| anbanerj|rover | marios|ruck, ok, I'll keep the testP ready | 11:51 |
| marios|ruck | anbanerj|rover: actually you can check if tripleo-ci-testing has updated yet | 11:52 |
| marios|ruck | anbanerj|rover: i am looking at train going to try testproject there cos it didn't update yet i.e. https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 8d27e439b3c20b65f1ad51f1d9ab01c8 still the same as the one used in the failed job | 11:53 |
| marios|ruck | anbanerj|rover: you can check for ussuri ^^ | 11:53 |
| anbanerj|rover | marios|ruck, no it has already updated. There are more failing jobs in the latest hash. Lemme testP those instead | 11:54 |
| marios|ruck | anbanerj|rover: k | 11:56 |
| ysandeep | rlandy, checking | 11:57 |
| marios|ruck | anbanerj|rover: sshnaidm: ysandeep: can you please add this to your reviews https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34275 | 11:58 |
| marios|ruck | anbanerj|rover: thanks just saw you voted | 11:58 |
| anbanerj|rover | marios|ruck, yep done | 11:58 |
| rlandy | ysandeep: no worries - I am commenting on the card | 11:59 |
| rlandy | will bring up findings at CIX | 11:59 |
| ysandeep | rlandy, ack | 11:59 |
| rlandy | I wanted to know if it was just a fs001 thing but the big was logged against BM - so no | 11:59 |
| ysandeep | rlandy: I am kind of confused with card because that bug was for - Couldn't resolve host name for rhos-release-latest , the log you shared is failing on provisioning.. | 12:08 |
| ysandeep | marios|ruck, ack | 12:09 |
| rlandy | ysandeep: yeah - idk - but we still see that error | 12:11 |
| ysandeep | rlandy: sry, which error? | 12:11 |
| rlandy | ysandeep: the critical error is actually below: dnf.exceptions.RepoError: Unknown repo: 'delorean-*-deps' | 12:12 |
| rlandy | 2021-06-27T17:55:47-0400 CRITICAL Error: Unknown repo: 'delorean-*-deps' | 12:12 |
| rlandy | RuntimeError: Curl error (6): Couldn't resolve host name for http://download.devel.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/rhos-release-latest.noarch.rpm [Could not resolve host: download.devel.redhat.com] | 12:12 |
| ysandeep | rlandy, now i remember.. weshay|ruck got side tracked initially- actually that's unrelated error to issue... issue was with rhosp-release not with rhos-release | 12:14 |
| ysandeep | 2021-04-27 10:36:44 | 2021-04-27 10:36:44.549530 | 000c523b-c9aa-2e5d-582e-000000000227 | FATAL | Deploy release version package | overcloud-controller-2 | error={"changed": false, "failures": ["No package rhosp-release available."], "msg": "Failed to install some of the specified packages", "rc": 1, "results": [] | 12:14 |
| rlandy | ysandeep: I think we should close that card and log something new | 12:14 |
| ysandeep | fix is for rhosp-release https://review.opendev.org/c/openstack/tripleo-common/+/789563 | 12:14 |
| rlandy | yep - that is what the bug says | 12:15 |
| rlandy | joining CIX to discuss | 12:15 |
| ysandeep | rlandy: true, if fs035 passes, then original issue is already fixed | 12:15 |
| *** jpena|lunch is now known as jpena | 12:29 | |
| *** marios|ruck is now known as marios|ruck|call | 12:30 | |
| weshay|ruck | marios|ruck|call, probably need to add more branches https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L1878 | 12:54 |
| weshay|ruck | marios|ruck|call, https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L1888 | 12:54 |
| weshay|ruck | marios|ruck|call, let's see where else this is hitting | 12:55 |
| pojadhav | folks, pls review : https://review.opendev.org/q/topic:%22master-upgrades-jobs%22+(status:open%20OR%20status:merged) | 12:56 |
| bhagyashris | chandankumar, akahat pojadhav marios|ruck|call rlandy zbr soniya ysandeep scrum time ... | 13:00 |
| bhagyashris | weshay|ruck, ^ | 13:00 |
| marios|ruck|call | weshay|ruck: yes it will need branches because see links in https://bugs.launchpad.net/tripleo/+bug/1931516/comments/4 | 13:04 |
| marios|ruck|call | weshay|ruck: ussuri victoria wallaby @ ^^ | 13:05 |
| *** amoralej|lunch is now known as amoralej | 13:08 | |
| marios|ruck|call | weshay|ruck: fixing now | 13:09 |
| weshay|ruck | arxcruz, link please | 13:19 |
| weshay|ruck | arxcruz, don't see https://review.rdoproject.org/r/q/owner:arxcruz%2540redhat.com | 13:20 |
| akahat | chandankumar, have you pulled patch on the promoter server? | 13:20 |
| chandankumar | akahat: nope | 13:21 |
| chandankumar | akahat: you have forgot to add the release hack | 13:21 |
| chandankumar | akahat: can you add that one | 13:21 |
| weshay|ruck | arxcruz, ? | 13:21 |
| akahat | chandankumar, ack | 13:22 |
| ysandeep | weshay|ruck, rlandy when you have time - I would like to merge the revert till dual voting is fixed: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/249948 | 13:23 |
| weshay|ruck | arxcruz, | 13:34 |
| ysandeep | weshay|ruck, I see an email from you on Friday, Do you want to chat now? | 13:37 |
| weshay|ruck | ysandeep, sure | 13:38 |
| ysandeep | weshay|ruck, meet.google.com/hht-bwpg-nat | 13:38 |
| ysandeep | weshay|ruck, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34060/17/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf_py3/telegraf.d/jobs_blocking_promotion.conf | 13:44 |
| * anbanerj|rover lunch | 14:10 | |
| weshay|ruck | marios|ruck|call, you still on a call? | 14:11 |
| weshay|ruck | marios|ruck|call, so I think we need two bugs | 14:11 |
| weshay|ruck | looks at https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-common-ussuri/5604b0b/logs/undercloud/var/log/tempest/stestr_results.html.gz | 14:11 |
| weshay|ruck | marios|ruck|call, you are linking to periodic-tripleo-ci-centos-8-scenario002-standalone-victoria, which means the jobs value must be changed | 14:12 |
| marios|ruck|call | weshay|ruck: no :) | 14:15 |
| *** marios|ruck|call is now known as marios|ruck | 14:15 | |
| marios|ruck | weshay|ruck: my point is the luks one is the consistent one | 14:15 |
| *** ysandeep is now known as ysandeep|afk | 14:15 | |
| marios|ruck | weshay|ruck: the others are only seen in some jobs, like that crypsteup one 17:11 < weshay|ruck> looks at | 14:16 |
| marios|ruck | https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-common-ussuri/5604b0b/logs/undercloud/var/log/tempest/stestr_results.html.gz | 14:16 |
| weshay|ruck | looking | 14:17 |
| weshay|ruck | marios|ruck, ya.. so let's skip luks.. across branches and jobs | 14:17 |
| marios|ruck | weshay|ruck: k let me update to do jobs [] | 14:17 |
| weshay|ruck | the other temepst failures... in some of those jobs.. would need a new bug | 14:18 |
| weshay|ruck | marios|ruck, [] is correct | 14:18 |
| marios|ruck | weshay|ruck: yeah but only if we start seeing those consistently | 14:18 |
| marios|ruck | weshay|ruck: i mean for 'new bug' | 14:18 |
| weshay|ruck | right.. oh | 14:18 |
| weshay|ruck | and let me get health link | 14:18 |
| marios|ruck | weshay|ruck: but will make the jobs [] for now sec | 14:18 |
| weshay|ruck | marios|ruck, https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ#How-often-is-this-tempest-test-failing | 14:19 |
| weshay|ruck | for upstream at least | 14:19 |
| weshay|ruck | \s/current_test/test_ur_interested in | 14:19 |
| weshay|ruck | http://status.openstack.org/openstack-health/#/test/barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_and_boot | 14:20 |
| weshay|ruck | for example | 14:20 |
| weshay|ruck | no fails.. only 5 runs.. so | 14:20 |
| marios|ruck | weshay|ruck: heh i was just trying to copy paste in the url :) | 14:20 |
| weshay|ruck | wish we had this in rdo.. | 14:20 |
| weshay|ruck | so helps a little | 14:20 |
| marios|ruck | weshay|ruck: The last data in the subunit2sql database, from "2021-06-23T16:14:32.000Z", is >1 day old. There might be an issue with result collection. | 14:20 |
| marios|ruck | weshay|ruck: are we running that? | 14:20 |
| marios|ruck | weshay|ruck: yes i think we are this is the openstack health that sorin et al were workign on | 14:21 |
| weshay|ruck | that's upstream infra | 14:21 |
| weshay|ruck | we're not yet running a local version of this part | 14:22 |
| marios|ruck | weshay|ruck: but this is what they are planning to decommision? | 14:22 |
| marios|ruck | weshay|ruck: or was that logstash | 14:22 |
| weshay|ruck | ya.. so we may not even run those other two tests upstream | 14:23 |
| weshay|ruck | and could just be periodic integration/component | 14:23 |
| *** ysandeep|afk is now known as ysandeep | 14:44 | |
| marios|ruck | weshay|ruck: o/ | 14:54 |
| weshay|ruck | aye | 14:54 |
| marios|ruck | will reach out to them if they don't respond by tomorrow to my ping at https://trello.com/c/Wb8Jf4kP/2007-cixlp1933639tripleociproa-periodic-master-fs-1-fails-during-tests-tempestapiobjectstorage#comment-60d99271eb4c5e0a96eeb46a | 14:54 |
| weshay|ruck | k.. marios|ruck can we prepare a skip today and wf-1 | 14:55 |
| weshay|ruck | tomorrow we will be 8 days out | 14:55 |
| marios|ruck | weshay|ruck: we have one | 14:55 |
| weshay|ruck | oh.. perfect | 14:55 |
| marios|ruck | weshay|ruck: see https://bugs.launchpad.net/tripleo/+bug/1933639 | 14:55 |
| weshay|ruck | k.. /me looks | 14:55 |
| marios|ruck | weshay|ruck: well we don't want to merge it/there are more tests potentially | 14:55 |
| marios|ruck | weshay|ruck: see comment #5 from today for example | 14:56 |
| weshay|ruck | thanks | 14:57 |
| weshay|ruck | zbr, anbanerj|rover you folks have a sec? | 14:57 |
| weshay|ruck | actually give me 30 min | 14:58 |
| anbanerj|rover | weshay|ruck, yep sure | 14:58 |
| *** dviroel is now known as dviroel|lunch | 15:07 | |
| rlandy | ysandeep: merging https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/249948 | 15:11 |
| ysandeep | rlandy, thanks! | 15:11 |
| weshay|ruck | rlandy, did sunil make onto irc yet? | 15:25 |
| weshay|ruck | anbanerj|rover, ok.. want to sync up? | 15:25 |
| rlandy | weshay|ruck: doesn't look like it | 15:26 |
| anbanerj|rover | weshay|ruck, yep | 15:26 |
| weshay|ruck | anbanerj|rover, https://meet.google.com/keh-hvxq-har?authuser=1 | 15:27 |
| rlandy | weshay|ruck: will check in with him in a bit | 15:27 |
| weshay|ruck | zbr, can we merge https://review.opendev.org/c/opendev/elastic-recheck/+/729623/ ? | 15:33 |
| weshay|ruck | anbanerj|rover, https://zuul.openstack.org/builds?job_name=opendev-buildset-registry | 15:35 |
| weshay|ruck | anbanerj|rover, | 15:39 |
| weshay|ruck | nd versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to https://[2001:db8:fd00:1000::5]:13000: HTTPSConnectionPool(host='2001:db8:fd00:1000::5', port=13000): | 15:39 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria/6064e4e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 15:39 |
| rlandy | chandankumar: akahat: looking at https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32679 | 15:52 |
| rlandy | image promotion patch | 15:52 |
| rlandy | chandankumar: akahat: ready to review/try out? | 15:52 |
| *** ykarel is now known as ykarel|away | 15:52 | |
| rlandy | see test failures | 15:53 |
| *** amoralej is now known as amoralej|off | 15:53 | |
| akahat | rlandy, yes. we can try it out | 15:54 |
| rlandy | akahat: k - I don;t have a promotable hash atm | 15:55 |
| rlandy | but chandankumar has a test hash | 15:55 |
| rlandy | name | 15:55 |
| rlandy | otherwise can ping when we have a workable hash | 15:55 |
| akahat | rlandy, okay. will wait for the candidate hash. | 15:56 |
| *** ysandeep is now known as ysandeep|away | 15:56 | |
| rlandy | chandankumar: ^^ let us know if you agree or if you want to try promote to a fake named hash | 15:57 |
| rlandy | anbanerj|rover: hey - just FYI - looking at open ER patches | 15:59 |
| marios|ruck | anbanerj|rover: weshay|ruck: o/ me off in a couple mins | 15:59 |
| *** jpena is now known as jpena|off | 16:00 | |
| *** dviroel|lunch is now known as dviroel | 16:00 | |
| weshay|ruck | marios|ruck, k.. need me to follow anything? | 16:02 |
| marios|ruck | weshay|ruck: fyi that ran green https://review.rdoproject.org/r/c/testproject/+/34310 so train should promote if you wana check that | 16:05 |
| marios|ruck | weshay|ruck: info in the gchat | 16:05 |
| marios|ruck | weshay|ruck: dlrn_hash_tag: 8d27e439b3c20b65f1ad51f1d9ab01c8 | 16:05 |
| weshay|ruck | ah nice | 16:07 |
| weshay|ruck | thanks | 16:07 |
| weshay|ruck | marios|ruck, it's promoting now :) http://38.102.83.109/promoter_logs/centos8_train.log | 16:08 |
| marios|ruck | weshay|ruck: nice ... that's my queue | 16:08 |
| marios|ruck | :) | 16:08 |
| marios|ruck | have a good one o/ | 16:08 |
| *** marios|ruck is now known as marios|out | 16:09 | |
| chandankumar | rlandy: in the new image promotion patch, we have added more logging not much improvement there | 16:28 |
| rlandy | chandankumar:k - so Ill follow what you and akahat want to do here | 16:31 |
| rlandy | we can carry on with manual promotions - we are not blocked | 16:31 |
| rlandy | its just as you want me to test along with you | 16:31 |
| chandankumar | currently cherry-picked that patch | 16:32 |
| rlandy | bhagyashris: ysandeep|away: weshay|ruck: added new work item for sprint 47 on hackmd - may be more DF focused - can discuss tomorrow | 16:37 |
| rlandy | chandankumar: k- let's try another promotion tomorrow | 16:38 |
| weshay|ruck | rlandy, metalsmith and cephadm work should be done, but would be smart to review it | 16:39 |
| rlandy | be done (by DF?) | 16:39 |
| weshay|ruck | old items for us... new for qe unfortunately | 16:39 |
| weshay|ruck | rlandy, no as in the work is done.. it's in ci | 16:40 |
| rlandy | oh should be already done | 16:40 |
| rlandy | got it | 16:40 |
| weshay|ruck | rlandy, let's add a work item to review and socialize both items.. because it's not very socialized | 16:42 |
| rlandy | and make sure we have all the 17 settings right | 16:43 |
| weshay|ruck | rlandy, you've seen both things :) | 16:45 |
| weshay|ruck | but probably didn't connect the dots | 16:45 |
| weshay|ruck | and no working overcloud deployments in .... 4 months? maybe w/ 17 | 16:48 |
| weshay|ruck | makes a person kind of forget | 16:49 |
| rlandy | overcloud deployments are working now - so we can just kill the item | 16:49 |
| rlandy | no need to socialize it | 16:49 |
| weshay|ruck | rlandy, https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/1ebd79f/logs/undercloud/home/zuul/overcloud-baremetal-deployed.yaml.txt.gz | 16:51 |
| weshay|ruck | rlandy, they are working? | 16:51 |
| weshay|ruck | in 17? | 16:51 |
| chandankumar | rlandy: Ok, I tested the image server code https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32679 | 16:52 |
| chandankumar | commented on the patch what needs to be changed, | 16:52 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-17/aed46f1/logs/undercloud/home/zuul/overcloud-baremetal-deployed.yaml.txt.gz | 16:52 |
| rlandy | weshay|ruck: ^^ | 16:52 |
| chandankumar | now it is working fine | 16:52 |
| rlandy | there | 16:52 |
| chandankumar | rlandy: you can try now with a working hash | 16:53 |
| rlandy | chandankumar: k - don;t have a working hash as of now | 16:53 |
| rlandy | maybe tomorrow | 16:53 |
| chandankumar | I have tested it against wes-current-tripleo | 16:53 |
| rlandy | working is an interesting question ... not as of now ... https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-17/aed46f1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 16:53 |
| weshay|ruck | rlandy, I wouldn't say it's "fine" https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-17/aed46f1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 16:53 |
| weshay|ruck | lol.. ya the config is there | 16:54 |
| chandankumar | rlandy: please comment on this patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32679/30/ci-scripts/dlrnapi_promoter/qcow_client.py to improve logging of qcow promotion | 16:54 |
| chandankumar | when free,thanks! | 16:55 |
| rlandy | weshay|ruck: yeah - well that is what I commented in the meeting this morning | 16:55 |
| rlandy | fs020 ok | 16:55 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-17/3cf20e9/logs/undercloud/home/zuul/overcloud-baremetal-deployed.yaml.txt.gz | 16:55 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-17/3cf20e9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 16:56 |
| rlandy | so fs001 may need some work | 16:56 |
| rlandy | whatever - can discuss tomorrow | 16:56 |
| anbanerj|rover | weshay|ruck, https://review.rdoproject.org/r/c/testproject/+/34312 testP for victoria fs039 had hit "Failed to attach network adapter device" bug | 17:12 |
| anbanerj|rover | fs035 simply timed out without reason, reruning it | 17:12 |
| anbanerj|rover | For ussuri I could not run it against the old hash even with depends on https://review.rdoproject.org/r/c/testproject/+/34311 | 17:13 |
| rlandy | voted | 17:14 |
| eagles | what's the correct way to enable a tempest plugin (designate) to run only on scenario 3 for tripleo master? | 17:20 |
| weshay|ruck | rlandy, re: eagles question.. we have not yet consolidated the tempest config yet right? | 17:39 |
| rlandy | weshay|ruck: card is still in progress | 17:41 |
| weshay|ruck | eagles, atm.. scenario003's tempest config is here: https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L618 | 17:41 |
| weshay|ruck | eagles, is it running in another job you want to turn off? | 17:41 |
| eagles | no. I was trying to enable it for scenario 3 https://review.opendev.org/c/openstack/tripleo-ci/+/797335 | 17:42 |
| eagles | i created this patch to trigger s003 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/798093 and it didn't seem to run any designate tests. It occurred to me that the plugin might've been blacklisted - or I might simply have the syntax wrong for enabling it | 17:43 |
| weshay|ruck | eagles, that's centos-7 is that what you want? | 17:44 |
| weshay|ruck | doubt it | 17:44 |
| weshay|ruck | eagles, we're killing stein | 17:44 |
| eagles | doh | 17:46 |
| eagles | probably why lol thanks | 17:47 |
| weshay|ruck | eagles, see pm.. maybe ur taking care of the concern | 17:48 |
| weshay|ruck | rlandy, so.. metalsmith *is* working in 17 :) https://sf.hosted.upshift.rdu2.redhat.com/logs/72/190672/61/check/periodic-tripleo-ci-rhel-8-bm_envA-3ctlr_1comp-featureset035-rhos-17/aced8c8/logs/undercloud/home/zuul/overcloud-deploy.sh | 17:53 |
| weshay|ruck | nice job getting the overcloud deployed | 17:53 |
| weshay|ruck | 35 :) | 17:53 |
| rlandy | weshay|ruck: right - fs001 is another issue | 17:53 |
| rlandy | fs035 and fs020 | 17:53 |
| rlandy | are doing fine | 17:53 |
| weshay|ruck | aye.. nice nice | 17:53 |
| rlandy | now | 17:53 |
| eagles | thanks weshay|ruck, rlandy | 17:53 |
| weshay|ruck | ok.. so metal smith we're good | 17:53 |
| rlandy | weshay|ruck: the discussion started around ceph | 17:53 |
| rlandy | and where we are deploying that | 17:54 |
| weshay|ruck | rlandy, scenario001/004 should have cephadm | 17:54 |
| rlandy | since we are ceph5 | 17:54 |
| rlandy | and qe is ceph4 | 17:54 |
| rlandy | that's kind of how we got into this | 17:54 |
| rlandy | also 16.2 vs 17 testing | 17:54 |
| rlandy | and what's covered in what | 17:54 |
| weshay|ruck | rlandy, meh | 17:54 |
| weshay|ruck | rlandy, we could consider adding ceph to an ovb job.. but let's not worry about it now | 17:55 |
| rlandy | weshay|ruck: so really we are good from our side | 17:55 |
| weshay|ruck | :) | 17:55 |
| rlandy | weshay|ruck: ^^ ack | 17:55 |
| weshay|ruck | rlandy, ya.. this all went down like 6 months ago :) | 17:55 |
| rlandy | weshay|ruck: except that 17 never worked | 17:55 |
| rlandy | until nowish | 17:55 |
| rlandy | so upstream is fine | 17:55 |
| rlandy | but we juts got 17 into the mix | 17:56 |
| weshay|ruck | ya | 17:56 |
| rlandy | it's not really our discussion | 17:56 |
| rlandy | it's QE's | 17:56 |
| rlandy | except where we have to show why our tests are passing and QEs are failing on the same dlrn hash | 17:56 |
| rlandy | we're testing different things | 17:57 |
| rlandy | that's all | 17:57 |
| rlandy | weshay|ruck: either way, I removed the item from the sprint discussion | 17:58 |
| rlandy | the setting should be right now for 17 - it's juts fs001 that needs to be fixed/debugged | 17:58 |
| rlandy | chandankumar: ok -this upcoming 16.2 should be a promotable hash | 17:59 |
| weshay|ruck | k.. /me adding to that now | 17:59 |
| eagles | another question- is there a way to easily query the periodic jobs out that are timing out for https://bugs.launchpad.net/tripleo/+bug/1881087... tempest failures are sort of a different beast I think | 18:01 |
| rlandy | yep - getting link | 18:03 |
| rlandy | so if the jobs are not zuul timing out - only tempest timing out ... https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-train&job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train is probably best | 18:06 |
| rlandy | or better | 18:07 |
| rlandy | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-train&job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train&result=FAILURE | 18:07 |
| eagles | rlandy ah cool thanks | 18:08 |
| rlandy | granted there could be other failures in there | 18:08 |
| rlandy | eagles: ^^ approximation :) | 18:08 |
| eagles | rlandy: yeah, I'm also looking for just timed out deployments but I can see how that works | 18:08 |
| eagles | oh.. hmm doesn't actually say timed out if it was this kind of failure lol. hunting we will go | 18:16 |
| weshay|ruck | depends if it's a zuul timeout or tripleo timeout.. both have timers | 18:17 |
| eagles | right | 18:19 |
| rlandy | async task did not complete within the requested time - 5700s | 18:31 |
| rlandy | ^^ message to search for | 18:31 |
| eagles | rlandy: ack thaks! | 18:32 |
| rlandy | weshay|ruck: http://health.sbarnea.com/ - still the correct health location to check? | 18:47 |
| anbanerj|rover | rlandy, yes ^ is correct. But I found a lot of strings are not gettting a hit in the logstash | 18:53 |
| rlandy | anbanerj|rover: have time to meet up? | 19:03 |
| anbanerj|rover | rlandy, sure, but even I am trying to find out why there are no hits | 19:04 |
| anbanerj|rover | rlandy, https://meet.google.com/eoa-vzuw-pzp | 19:04 |
| weshay|ruck | anbanerj|rover, http://dashboard-ci.tripleo.org/d/eYMt45z7z/upstream-and-rdo-promotions-new?orgId=1 | 19:12 |
| weshay|ruck | anbanerj|rover, take a look there | 19:12 |
| weshay|ruck | rlandy, yes | 19:12 |
| rlandy | weshay|ruck: https://bugs.launchpad.net/tripleo/+bug/1881087 | 19:14 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train/4bb50ef/job-output.txt | 19:19 |
| weshay|ruck | anbanerj|rover, rlandy message:"async task did not" | 19:22 |
| rlandy | https://softwarefactory-project.io/analytics/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2w,to:now))&_a=(columns:!(_source),filters:!(),index:logstash,interval:auto,query:(language:kuery,query:'%20message:%22async%20task%20did%20not%22'),sort:!()) | 19:23 |
| weshay|ruck | https://review.rdoproject.org/analytics/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-3w,to:now))&_a=(columns:!(_source),filters:!(),index:logstash,interval:auto,query:(language:kuery,query:'message:%22async%20task%20did%20not%22'),sort:!()) | 19:24 |
| weshay|ruck | eagles, fyi https://review.rdoproject.org/analytics/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-3w,to:now))&_a=(columns:!(build_name),filters:!(),index:logstash,interval:auto,query:(language:kuery,query:'message:%22async%20task%20did%20not%22'),sort:!()) | 19:29 |
| weshay|ruck | so.. we could open a new bug.. probably and reference the old one | 19:29 |
| weshay|ruck | and we can start digging | 19:29 |
| weshay|ruck | https://review.rdoproject.org/analytics/goto/ed7eb9410f6b834192e562fa058df1ff | 19:30 |
| weshay|ruck | ^ | 19:30 |
| anbanerj|rover | rlandy, weshay|ruck https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5ba/787216/7/check/tripleo-ci-centos-8-scenario010-standalone/5ba92d8/job-output.txt | 19:33 |
| weshay|ruck | rlandy, anbanerj|rover http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?orgId=1&var-launchpad_tags=alert&var-releases=master&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-influxdb_filter=job_name%7C%3D~%7C%2F.*scenario010.*%2F&from=now-30d&to=now | 19:34 |
| weshay|ruck | I don't see that async error upstream | 19:37 |
| weshay|ruck | anbanerj|rover, rlandy message:"async task did not complete within the requested time" tags: console | 19:43 |
| rlandy | anbanerj|rover: I can't edit https://docs.google.com/spreadsheets/d/16rqgaSSoQrYNjsI4q0YInJxrOOJL3xyP3t3D6ZpluFY/edit?skip_itp2_check=true#gid=846893892 | 20:22 |
| rlandy | sent request to be able to edit | 20:22 |
| anbanerj|rover | rlandy, I think bhagyashris or zbr has edit rights | 20:28 |
| rlandy | anbanerj|rover: ok - will make a list elsewhere until I have edit rights | 20:55 |
| eagles | rlandy: weshay|ruck: there is still the problmem that designate_tempest_plugin is still in the tempest_excludelist.tx which seems to supersed the includelist. What's the proper way to remove the designate_tempest_plugin from the exclude list | 21:57 |
| * weshay|ruck looks.. but got log? | 21:59 | |
| rlandy | https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L454 | 22:05 |
| rlandy | are we talking about ^^ | 22:05 |
| weshay|ruck | eagles, where do you see it in excludes | 22:05 |
| rlandy | job log would help | 22:06 |
| rlandy | but I see it in skip | 22:06 |
| weshay|ruck | oh.. we can nuke it from the skip | 22:06 |
| weshay|ruck | rlandy, lp: 'https://bugs.launchpad.net/tripleo/+bug/invalid' | 22:06 |
| weshay|ruck | ya.. rlandy++ | 22:07 |
| weshay|ruck | that's probably what he meant | 22:07 |
| rlandy | I know what he means | 22:07 |
| weshay|ruck | :) | 22:07 |
| rlandy | exclude list show in the job te,pest log | 22:07 |
| rlandy | getting :) | 22:07 |
| rlandy | sec | 22:07 |
| weshay|ruck | eagles, which branch? | 22:08 |
| weshay|ruck | eagles, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/798388 | 22:09 |
| rlandy | now of course I can't find a good example | 22:10 |
| rlandy | but basically yeah, I assume if it's skipped, it would be in excludes | 22:10 |
| rlandy | weshay|ruck: I'll put in a review to nuke that? not sure why it was there to begin with | 22:12 |
| rlandy | let arxcruz review | 22:12 |
| rlandy | mark it w-1 for the moment | 22:12 |
| rlandy | let eagles test with it | 22:12 |
| rlandy | sec ... | 22:12 |
| weshay|ruck | it's very old | 22:12 |
| eagles | sorry stepped afk for a sec - 1s | 22:14 |
| eagles | rlandy: https://ddfc48525880952e4225-275879851208b5d95a5c6b3916687940.ssl.cf2.rackcdn.com/797335/2/check/tripleo-ci-centos-8-scenario003-standalone/24217fa/logs/undercloud/home/zuul/tempest/etc/tempest_excludelist.txt | 22:14 |
| eagles | the patch is https://review.opendev.org/c/openstack/tripleo-ci/+/797335/ | 22:15 |
| rlandy | eagles: sec - submitting patch you can try depends-on | 22:16 |
| weshay|ruck | kid duty 0/ | 22:17 |
| rlandy | eagles: pls try a depends-on: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/798390 | 22:20 |
| rlandy | marked it w-1 for the moment - as I want the tempest gurus to approve this | 22:21 |
| rlandy | but it should spring you free | 22:21 |
| eagles | rlandy: cool! thanks 10^6 | 22:21 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!