Friday, 2020-11-20

*** jmasud has joined #oooq00:08
*** tosky has quit IRC00:18
*** jmasud has quit IRC00:31
*** matbu has quit IRC00:58
*** jmasud has joined #oooq01:07
*** jmasud has quit IRC01:10
*** jmasud has joined #oooq01:15
*** jmasud has quit IRC01:41
*** jmasud has joined #oooq01:49
*** jmasud has quit IRC02:01
*** jmasud has joined #oooq02:05
*** jmasud has quit IRC02:31
*** rlandy|rover has quit IRC03:42
*** ratailor has joined #oooq05:38
*** marios has joined #oooq05:51
*** jmasud has joined #oooq06:23
*** ykarel_ has joined #oooq06:24
*** jmasud has quit IRC06:27
*** jmasud has joined #oooq06:29
*** jmasud has quit IRC06:33
*** jmasud has joined #oooq06:41
*** jmasud has quit IRC06:48
*** jmasud has joined #oooq06:51
*** marios is now known as marios|ruck06:51
*** jmasud has quit IRC07:00
*** matbu has joined #oooq07:07
*** ykarel_ is now known as ykarel07:35
*** amoralej|off is now known as amoralej07:39
*** ratailor is now known as ratailor|lunch07:49
*** saneax has joined #oooq08:36
*** jpena|off is now known as jpena08:37
*** derekh has joined #oooq09:15
*** tosky has joined #oooq09:57
*** dtantsur|afk is now known as dtantsur09:57
chandankumarykarel: jpena https://review.rdoproject.org/r/31143 please have a look10:50
*** saneax has quit IRC10:51
*** ratailor|lunch has quit IRC10:57
ykarel ack10:58
*** jbadiapa has joined #oooq11:29
marios|ruckfooooood11:30
marios|ruckbiab11:30
*** ksambor has quit IRC11:37
*** jfrancoa has quit IRC12:15
*** jpena is now known as jpena|lunch12:27
*** dmellado has quit IRC12:38
*** dmellado has joined #oooq12:46
*** rlandy has joined #oooq12:53
*** rlandy is now known as rlandy|rover12:53
rlandy|rovermarios|ruck: hey ...12:55
rlandy|roverlast day!!!12:55
marios|ruck\o/12:56
rlandy|rovermarios|ruck: thanks for the fix on stein12:57
marios|ruckrlandy|rover: let's see it think it's right12:57
chandankumarrlandy|rover: marios|ruck weshay|ruck https://review.rdoproject.org/r/#/c/31143/ and https://review.rdoproject.org/r/#/c/31144/12:57
rlandy|rovermarios|ruck: I messed up the rocky criteria ... either I left one job out of pipeline or added one too man to criteria - need to reconfirm with weshay|ruck12:58
rlandy|roverchandankumar: yep - looking at those now - thanks12:58
rlandy|roverran out of time yesterday12:58
marios|ruckrlandy|rover: poking at promotions now lookign at master might just post a testproject but not sure if there is a new scen10 bug yet12:59
rlandy|rovermarios|ruck: ack - looking at ^^ patches12:59
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org will be offline starting at 15:00 UTC (roughly two hours from now) for a weekend upgrade maintenance: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html13:01
marios|ruckah ^^^ rlandy|rover hmm so much for merging stuffs...13:02
marios|ruckoops missed that notice13:02
rlandy|roverugh - really???13:02
rlandy|rovermarios|ruck: let's just go off shift now, ok, right???13:02
marios|ruckbrb fetching coffee13:03
rlandy|roverchandankumar: https://review.rdoproject.org/r/#/c/31143/ looks good - one comment included13:05
rlandy|roverchandankumar: so looking at https://review.rdoproject.org/r/#/c/31144/3/zuul.d/integration-pipeline-rocky.yaml ...13:08
rlandy|rovermy comment stands13:08
rlandy|roverwe need a nodeset on that13:08
weshay|rucklatest upgrade issue13:09
weshay|ruckDepsolve Error occured:13:09
weshay|ruck2020-11-20 01:58:27.291280 | primary |  Problem: cannot install the best update candidate for package python3-tripleoclient-12.3.2-0.20201118002446.4a5027a.el8.noarch13:09
weshay|ruck2020-11-20 01:58:27.291296 | primary |   - nothing provides python3-openstackclient >= 5.2.0 needed by python3-tripleoclient-13.4.2-0.20201113043133.6e9fd9b.el8.noarch13:09
weshay|ruck2020-11-20 01:58:27.293100 | primary |13:09
chandankumarrlandy|rover: done13:10
rlandy|roverchandankumar: then I am +2 on both13:10
chandankumarrlandy|rover: cool13:10
rlandy|roverweshay|ruck: hey13:10
* chandankumar not sure it will bring some turbulance effect in ci\13:11
rlandy|roverweshay|ruck: I missed on job for the rocky line (or added one too many in the criteria) ...  should periodic-tripleo-ci-centos-7-undercloud-containers-rocky be in or out?13:11
rlandy|roverI ran it testproject yesterday to get the promotion last night13:11
weshay|ruckk.. I'll look in a minute13:13
weshay|ruckhave to write up this bug13:13
weshay|ruckis launchpad down?13:16
marios|ruckweshay|ruck: i've been using it today13:16
weshay|ruckIf we are in the middle of an update, Launchpad will be back in a couple of minutes. Otherwise, we are working to fix the unexpected problems. Check @launchpadstatus on Twitter for updates.13:16
weshay|ruckthere it goes13:17
marios|ruckhmm maybe related to that gerrit notice13:17
rlandy|rovertripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train13:17
marios|ruck* All of our existing integrations with launchpad and storyboard may not be fully functional. The changes to the database system impact aspects of how we lookup user details in order to update bugs.13:17
rlandy|rovertiming out13:18
marios|ruckrlandy|rover: of course it is13:19
marios|ruck:/13:19
rlandy|rovermarios|ruck: I see what you mean about the scenario01013:19
rlandy|roverfailed twice in master13:20
marios|ruckrlandy|rover: hmm not a lot though? https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train13:20
marios|ruckrlandy|rover: yeah k let me focus on that and see if ther eis a bug (scen10 i mean ) gimme few13:20
rlandy|rovermarios|ruck: k13:20
weshay|ruckwe NEED reposetup to to fail the job when this happens13:25
weshay|ruck<h1>Not Found</h1>13:25
weshay|ruck<p>The requested URL /centos8-ussuri/acac7a32b596342e0c8a8afd913ca6a9/delorean.repo was not found on this server.</p>13:25
weshay|ruck</body></html>13:25
weshay|ruckhttps://bugs.launchpad.net/tripleo/+bug/190503413:25
openstackLaunchpad bug 1905034 in tripleo "undercloud upgrade for ussuri fails on package dep python3-openstackclient >= 5.2.0" [Critical,Triaged]13:25
*** amoralej is now known as amoralej|lunch13:26
* rlandy|rover looks13:26
chandankumarweshay|ruck: https://review.rdoproject.org/r/#/c/31143/ +w it\13:27
weshay|ruckmoving that job to nv in 24 hours13:27
rlandy|roverhttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-undercloud-upgrade-ussuri13:30
rlandy|roverlooks like it depends on the project a lot13:30
rlandy|rovermarios|ruck: ^^ do you want to take this one and I'll continue with scenario010? or  you want me to look at ^^?13:31
* rlandy|rover open to either13:31
marios|ruckrlandy|rover: i am middle of filing 10 now13:31
marios|ruckrlandy|rover: confirmed issue13:31
marios|ruckrlandy|rover: gimme few then i will check that new bug13:31
chandankumarykarel: i think we donot this patch https://review.rdoproject.org/r/#/c/31153/ now? change label to vexxhost13:32
*** jpena|lunch is now known as jpena13:32
weshay|ruckmarios|ruck, rlandy|rover either of you write up the scenario010 octavia issue blocking master promtion.. just checking.. don't see it13:33
weshay|ruck2020-11-20 09:26:58Z [standalone.StandaloneServiceChain]: CREATE_FAILED  Error in 31 output role_data: Error in OctaviaWorker output role_data: The Referenced Attribute (OctaviaBase update_tasks) is incorrect.13:33
marios|ruckweshay|ruck: 15:31 < marios|ruck> rlandy|rover: i am middle of filing 10 now13:33
beaglesugh13:34
weshay|ruckk.. thanks13:34
rlandy|roverhttps://af8f541e885d71823a86-a17c91d89a0966538758fa02ae1f57d0.ssl.cf2.rackcdn.com/757821/8/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/9bce46b/logs/undercloud/etc/yum.repos.d/delorean.repo13:35
beaglesweshay|ruck:  looks like I broke it with https://review.opendev.org/#/c/750986/13:35
rlandy|roverhow does that happen?13:35
marios|ruckbeagles: i was going to come looking for commits in tht cos that is where the issue is for sure it fails during stack creation13:35
weshay|ruckrlandy|rover, both are non-voting13:35
weshay|rucktripleo-ci-centos-8-scenario010-standaloneFAILURE in 1h 21m 01s (non-voting)13:35
weshay|rucktripleo-ci-centos-8-scenario010-ovn-provider-standaloneFAILURE in 34m 09s (non-voting)13:35
weshay|ruckjust robbing mary to pay paul13:36
rlandy|roverlol13:36
marios|ruckweshay|ruck: rlandy|rover: also in check, but it is non voting so not blocking gate13:36
weshay|ruckand other weird dad'isms I don't understand13:36
rlandy|rovernon-voting will kill us13:36
marios|ruck        * https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_267/763266/1/check/tripleo-ci-centos-8-scenario010-standalone/267ba11/logs/undercloud/home/zuul/standalone_deploy.log13:36
rlandy|roverweshay|ruck: actually my question was around the missing delorean repo13:37
rlandy|rover<p>The requested URL /centos8-ussuri/acac7a32b596342e0c8a8afd913ca6a9/delorean.repo was not found on this server.</p>13:37
rlandy|rover</body></html>13:37
rlandy|roverhave we seen that one before?13:37
weshay|ruckrlandy|rover, I have not13:38
ykarelchandankumar, that can be held until you switch all other jobs13:38
weshay|ruckfirst time13:38
rlandy|roverweshay|ruck: mirror blip?13:38
weshay|ruckdon't know yet13:38
weshay|ruckwrote it up13:38
ykarelno it's not mirror13:38
weshay|ruckand retrying13:38
rlandy|roverbut will look into a chcek on repo setup13:38
ykarelcomponent repos don't follow that url pattern13:38
weshay|ruckykarel, it's not a component job13:38
weshay|ruckoh13:38
weshay|ruckwait13:38
ykareli mean component repo13:39
weshay|ruckaye13:39
weshay|rucksorry13:39
rlandy|roverthe way we set that repo up13:39
weshay|ruckit's just running repo-setup though13:39
rlandy|roverthat's the general dlrn repo13:39
weshay|rucknonthing fancy13:39
weshay|ruckhere13:39
ykarelseems it's mixing dlrn hash and dlrn path13:39
rlandy|rovernot singe comonent specific13:39
beaglesrlandy|rover: marios|ruck https://review.opendev.org/#/c/763561/13:40
marios|ruckbeagles: rlandy|rover: weshay|ruck: https://bugs.launchpad.net/tripleo/+bug/190503613:40
openstackLaunchpad bug 1905036 in tripleo "Scenario10 octavia jobs failing deploy stack creation with "Error Referenced Attribute (OctaviaBase update_tasks) is incorrect" [Critical,Triaged]13:40
weshay|ruckhttps://af8f541e885d71823a86-a17c91d89a0966538758fa02ae1f57d0.ssl.cf2.rackcdn.com/757821/8/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/9bce46b/logs/quickstart_files/playbook_executions.log13:41
weshay|ruckmarios|ruck, thanks.. milestone reminder13:41
rlandy|roverbeagles: thanks - following that13:41
beaglesIt hasn't been backported anywheres so maybe just revert it13:41
marios|ruckweshay|ruck: ack13:41
weshay|ruckrlandy|rover, ykarel looking at https://af8f541e885d71823a86-a17c91d89a0966538758fa02ae1f57d0.ssl.cf2.rackcdn.com/757821/8/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/9bce46b/logs/quickstart_files/playbook_executions.log13:41
rlandy|roverbeagles: add closes-bug on that?13:42
beaglesyujp13:42
beaglesone sec13:42
weshay|ruckthe one w/ --skip-tags tripleo-validations,teardown-all          /home/zuul/workspace/.quickstart/playbooks/multinode-overcloud-prep.yml13:42
rlandy|roverand?13:42
owalshhi, is tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates broken in the stein gate?13:43
owalshhttps://3ca3ec686f4ad84d08c5-f0dd46c0d8845d18269bbfd0711d9448.ssl.cf5.rackcdn.com/763170/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/f14c897/logs/undercloud/home/zuul/overcloud_update_prepare.log13:43
gchamoulweshay|ruck: Hi Wes, I am facing this issue "Unable to freeze job graph: Job tripleo-ci-centos-8-containers-multinode depends on tripleo-ci-centos-8-content-provider which was not run." on https://review.opendev.org/#/c/75649313:43
beaglesrlandy|rover:  updated commit to reflect bug13:43
rlandy|roverbeagles++ thanks13:43
marios|ruckowalsh: yeah sounds like https://bugs.launchpad.net/tripleo/+bug/1903581/comments/1813:43
openstackLaunchpad bug 1903581 in tripleo "tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io" [Critical,Fix released] - Assigned to Marios Andreou (marios-b)13:43
weshay|ruckgchamoul, looking13:44
gchamoulweshay|ruck: the patch is already rebased on top of rfolco's patch migrating to content provider ... and I don't know what's happening13:44
gchamoulthanks13:44
gchamoulweshay|ruck: I have two patches on top of the failed one which are green though ...13:45
marios|ruckowalsh: thanks, didn't realise it was also holding the stein gate... i will add a note on the bug but i thinkt he cherrypick at https://review.opendev.org/#/c/763546/ will help13:45
ykarelrlandy|rover, weshay|ruck, seems happening due to https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/multinode-undercloud-upgrade.yml#L11 and https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-8/ussuri.yml#L78-L8213:46
owalshmarios|ruck: ack, thanks!13:46
weshay|ruckgchamoul, I don't see it rebased on top of any unmerged ci patches13:46
weshay|ruckbut still looking13:46
chandankumarweshay|ruck: rlandy|rover marios|ruck Happy weekends, see ya on monday!13:47
*** chandankumar is now known as raukadah13:47
weshay|ruckChanServ, thanks13:47
weshay|rucklolz13:47
weshay|ruckhe always gets me13:47
marios|ruckhave a good one raukadah13:47
rlandy|roverraukadah: enjoy - see you monday13:48
* raukadah will make raukadah permanent someday13:48
gchamoulweshay|ruck: you did it at OCT 30th and it should contain at least the content provider migration patch from Rafael (merged at Oct 15th)13:49
weshay|ruckgchamoul, k.. I was just looking for anything unmerged.. I now think ur just saying the patch is on top of merged ci zuul layout changes13:51
gchamoulyes13:52
gchamoulsorry if it wasn't clear13:53
gchamoul:D13:53
rlandy|roverhttps://opendev.org/openstack/tripleo-quickstart-extras/commit/49f4aea85e8792070a2be6cbaf6a2668736cb422 been there for two years13:53
marios|ruckrlandy|rover: i re-added /1903581 to today's bugs based on the latest updates there ... otherwise it wasn't anywhere we closed that out. i don't think it makes sense to file a new bug for the stein cherrypick since it is still the same issue13:54
marios|ruckrlandy|rover: it would have been confusing for the hand-off13:54
marios|ruckrlandy|rover: i mean that it wasn't on the hackmd13:54
rlandy|rovermarios|ruck: it's fine - we'll explain it13:55
marios|ruckrlandy|rover: i added https://bugs.launchpad.net/tripleo/+bug/1903581/comments/18 and 19 there today13:55
openstackLaunchpad bug 1903581 in tripleo "tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io" [Critical,Fix released] - Assigned to Marios Andreou (marios-b)13:55
rlandy|roverI see13:55
weshay|ruckgchamoul, ya.. still looking.. no changes to the zuul layout in train ooo-validations in a month https://opendev.org/openstack/tripleo-validations/blame/branch/stable/train/zuul.d/layout.yaml13:55
rlandy|roverhttps://opendev.org/openstack/tripleo-quickstart/commit/11c2bb7cdfa401b9e3e2f65ec459e1b7a452ed76 been there for 4 months13:56
rlandy|roverit's failing now only??13:56
weshay|ruckrlandy|rover, repo-setup was not running in the upgrade portion of the playbook13:56
weshay|ruckfor some time13:57
gchamoulweshay|ruck: nope ... the strange thing is the two patches on top of this one are green. It should be red too!13:57
weshay|ruckgchamoul, aye.. /me hates this error13:57
rlandy|roverweshay|ruck: ah  ok so that makes sense now13:57
rlandy|roverso we have a three prong change13:57
rlandy|roverweshay|ruck: k - then we need to rethink this13:58
weshay|ruckhrm.. no changes here for 3 weeks https://opendev.org/openstack/tripleo-ci/commits/branch/master/zuul.d/multinode-jobs.yaml13:58
rlandy|roverfor upgrades13:58
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org will be offline starting at 15:00 UTC (roughly one hour from now) for a weekend upgrade maintenance: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html13:59
*** amoralej|lunch is now known as amoralej14:05
rlandy|roverhmmm - so how come this is ok?14:09
rlandy|roverhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6bd/757829/7/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/6bdf0bc/logs/undercloud/etc/yum.repos.d/delorean.repo14:09
*** dmellado has quit IRC14:09
*** dmellado has joined #oooq14:11
weshay|ruckgchamoul, not ideal.. but check out this patch.. https://review.opendev.org/#/c/763565/14:11
weshay|ruckgchamoul, calling out specific jobs in repo layouts is troublesome14:12
weshay|ruckon the to-do is to create more templates in tripleo-ci to narrow things14:12
weshay|ruckzbr, you avail today?14:13
zbryeah14:14
zbreven if i cant wait for the weekend to start14:14
weshay|ruckrlandy|rover, marios|ruck ok.. here's what I have for xfer to next ruck rover.. arxcruz  and rfolco https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=014:17
weshay|ruckrlandy|rover, marios|ruck thank you very much for your work this week :)14:18
rlandy|roverweshay|ruck: hold on - reviewing that list14:18
marios|ruckweshay|ruck: there is one missing from there but really my list is the https://hackmd.io/R0kCgz_7SHSix_cNgoC9pg?both14:18
marios|ruckweshay|ruck: not cos ther is something wrong with the tool14:19
marios|ruckweshay|ruck: but because we closed eh bug & card out already14:19
marios|ruckweshay|ruck: see https://bugs.launchpad.net/tripleo/+bug/1903581/comments/1814:19
openstackLaunchpad bug 1903581 in tripleo "tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io" [Critical,Fix released] - Assigned to Marios Andreou (marios-b)14:19
arxcruzmarios|ruck: can we sync on monday when folco is back ?14:19
arxcruzif you don't mind14:19
marios|ruckarxcruz: sure you can sync with rlandy|rover i am out monday14:19
arxcruzok14:19
marios|ruckarxcruz: but will be around tuesday if you need sthing14:19
rlandy|roverhttps://review.opendev.org/#/c/763475 -related to https://bugs.launchpad.net/tripleo/+bug/190493614:20
openstackrlandy|rover: Error: Could not gather data from Launchpad for bug #1904936 (https://launchpad.net/bugs/1904936). The error has been logged14:20
marios|ruckarxcruz: but https://hackmd.io/R0kCgz_7SHSix_cNgoC9pg?both "everything you need to know"14:20
rlandy|rovercan we close this out?14:20
weshay|ruckmarios|ruck, thanks.. moving that lp to in progress14:20
rlandy|roverugh14:20
rlandy|roverarxcruz: I will clean up the hackmd at my EOD14:20
arxcruzack14:20
rlandy|roverstill trying to access these bugs14:21
rlandy|roverweshay|ruck: ^^ cant' access LP atm14:21
weshay|ruckya.. lp sucks14:21
rlandy|roverI think we can close some of those out14:21
weshay|ruckbirthday for my daughter.. back in 514:21
rlandy|roverbit of course I can't reach then14:21
weshay|ruckgchamoul, you see my patch?14:21
rlandy|roverweshay|ruck: happy birthday daughter14:21
marios|ruckrlandy|rover: weshay|ruck: i added comments there https://bugs.launchpad.net/tripleo/+bug/1904936 today comment #3/4 ... i see it on fs35 but not consistent so don't think the revert from sbaker is relevant?14:22
openstackLaunchpad bug 1904936 in tripleo "openstack.exceptions.ResourceNotFound: No Introspection found for node, Could not find node" [Critical,Triaged] - Assigned to Steve Baker (steve-stevebaker)14:22
marios|ruckhappy birthday junior weshay14:22
gchamoulweshay|ruck: yes14:22
rlandy|roverhttps://bugs.launchpad.net/tripleo/+bug/1904936 ack ... marios|ruck agree there - looks inconsistent14:22
rlandy|roveridk if it's real14:22
marios|ruckrlandy|rover: right... could be real but inconsistent. or just transient14:22
rlandy|roverhttps://bugs.launchpad.net/tripleo/+bug/190035714:23
openstackLaunchpad bug 1900357 in tripleo "fs020 queens is failing tempest tests - test_floatingip.FloatingIPQosTest and test_server_multi_create_auto_allocate" [Critical,Triaged] - Assigned to Soniya Murlidhar Vyas (svyas)14:23
rlandy|roverthus makes me sad14:23
rlandy|rovernow we need to revert that skip14:24
rlandy|roversoniya29: you on this? https://bugs.launchpad.net/tripleo/+bug/1900357 - assigned to you now14:24
rlandy|roverto revert that skip14:24
rlandy|roverweshay|ruck: ok - the spreadsheet is legit - but depressing14:25
marios|ruckyou are about to close 62 tabs are you sure14:43
marios|rucki'm getting better...14:43
weshay|ruckmarios|ruck, before you leave.. please add any required notes/comments if you have them..14:44
marios|ruckweshay|ruck: ack not leaving14:44
marios|ruckjust clearing tabs so i can focus on a single thing now... :D14:44
weshay|ruckstaying fo eva14:44
marios|ruckyou can checkout any time you like...14:44
weshay|ruckEAGLES ROCK.. oooqFM14:44
rlandy|rovermarios|ruck++14:47
rlandy|roverperiodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-steintestprojectmastercheck25325,681 hr 45 mins 30 secs2020-11-20 12:15:41SUCCESS14:47
rlandy|roverSCRIPT_ROOT=/home/centos/ci-config/ DISTRO_NAME=centos DISTRO_VERSION=7 ansible-playbook -e manifest_push=False -e target_registries_push=True /home/centos/ci-config/ci-scripts/container-push/container-push.yml14:48
rlandy|roverstein promoting14:48
marios|ruckrlandy|rover: cool14:49
marios|ruckwell14:49
marios|ruckrlandy|rover: i was thinknig about that maybe we dno't want it to promote14:49
marios|ruckrlandy|rover: until the patch actually merges14:49
marios|ruckrlandy|rover: but in this case i think its pretty safe14:49
marios|ruckrlandy|rover: that cherrypick should merge fine14:49
rlandy|rovermarios|ruck: I'll stay with it until it merges14:50
marios|ruckrlandy|rover: thanks i see it has enough votes now so +A once zuul reports? https://review.opendev.org/#/c/763546/14:51
rlandy|rovermarios|ruck: ack14:51
rlandy|roveryou can w+ that14:51
*** Goneri has joined #oooq14:51
marios|ruckrlandy|rover: ack ok only the updates job is running there (and undercloud-upgrades but non voting)14:52
marios|ruckrlandy|rover: ok going to +A it then14:52
rlandy|rovermarios|ruck: ack14:52
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org is offline for a weekend upgrade maintenance, updates will be provided once it's available again: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html15:03
rlandy|roverhttps://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-ussuri/27aaad5/logs/undercloud/var/log/tempest/stestr_results.html.gz15:05
rlandy|roverugh here we go again15:06
rlandy|roverweshay|ruck: arxcruz: one more edit to the scenario002 skiplist entry ...15:06
rlandy|rovererror: Could not fetch gerrit15:06
rlandy|roverwill try again mater15:07
rlandy|roverlater15:07
rlandy|rovermarios|ruck: will address the failure in ussuri scenario00215:07
marios|ruckdamn i was about to post testproject but it won't be able to talk to gerrit for depends-on15:07
* rlandy|rover looks at fs050 failure15:07
rlandy|roveridk about fs05015:08
rlandy|roverwill testproject rerun15:08
marios|ruckweshay|ruck: added some notes there https://bugs.launchpad.net/tripleo/+bug/1905034/comments/5 but looks like ykarel patch will fix it15:15
openstackLaunchpad bug 1905034 in tripleo "repo-setup fail: undercloud upgrade for ussuri fails on package dep python3-openstackclient >= 5.2.0" [Critical,Triaged]15:15
rlandy|roverweshay|ruck: marios|ruck: do we have a final decision on upstream queens? email thread not conlucsive15:16
rlandy|roverconclusive15:16
marios|ruckrlandy|rover: we can't make a decision15:16
marios|ruckrlandy|rover: until it is made for us15:16
marios|ruckrlandy|rover: i.e. agreement re the imports on the dates discussed15:16
rlandy|roversounds like a lot of thing sin life :(15:16
marios|ruckrlandy|rover: we are just speculating about what we might do :D15:17
weshay|ruckrlandy|rover, seems to me.. we can remove everything other than container-multinode15:20
rlandy|roverhmmm ... waiting on PTL here15:21
weshay|ruckjust need the ptl to weigh in now that we have input from the various folks15:21
rlandy|roverand all of upstream is bye-bye now15:22
marios|ruckmy "people are messing with you" radar is going off right now15:29
marios|rucki call shenanigans rlandy|rover weshay|ruck ? ;)15:30
rlandy|roverokie dokie15:30
rlandy|rovergoing to run quick errand while zuul is out to lunch - brb15:31
weshay|ruckmarios|ruck, well.. with great power comes great responsibility15:31
rlandy|rovermarios|ruck: weshay|ruck: ^^15:31
marios|ruckweshay|ruck: it really depends on them agreeing on the no more imports15:31
*** rlandy|rover is now known as rlandy|rover|brb15:31
marios|ruckweshay|ruck: if that is true then fine15:31
weshay|ruckmarios|ruck, ya.. after reviewing the schedule for osp-13.. nothing is on deck for imports15:31
weshay|ruckmburns approved15:32
weshay|ruckthere may be an unscheduled import.. but there's a whole internal layer of ci that can deal w/ that15:32
marios|ruckweshay|ruck: k lets discuss next week and make final decisions. we need to socialise it upstream first obviously too and allow for feedback there15:32
weshay|ruckmarios|ruck, so you know upstream only has containers-multinode right?15:33
marios|ruckweshay|ruck: so whatever we do won't be 'right now' but if we started now perhaps over the next few weeks15:33
weshay|ruckso upstream is already done15:33
weshay|rucknow we're talking about periodic15:33
marios|ruckweshay|ruck: oh really? 17:33 < weshay|ruck> marios|ruck, so you know upstream only has containers-multinode right?15:33
marios|ruckweshay|ruck: well yeah for periodic it only makes sense if we are importing15:33
marios|ruckweshay|ruck: if we aren't then lets kill it and mark the branch gone15:34
marios|ruckweshay|ruck: well, at least unmaintained15:34
weshay|ruckwell and tripleo-ci-centos-7-undercloud-oooq15:34
weshay|ruckmarios|ruck, I'm ignoring the build jobs...15:34
weshay|ruckthose should stay15:34
weshay|ruckovercloud images, and containers15:34
marios|ruck17:31 < weshay|ruck> marios|ruck, well.. with great power comes great responsibility15:35
weshay|ruckmarios|ruck, http://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&var-influxdb_filter=branch%7C%3D%7Cstable%2Fqueens&from=now-30d&to=now&fullscreen&panelId=2215:35
marios|ruckweshay|ruck: where is my power :D15:35
weshay|ruckheh15:35
weshay|ruckmarios|ruck, all the sceario jobs for queens are nv15:35
weshay|ruckqueens periodic should be...15:36
weshay|ruck2020-10-21tripleo-ci-centos-7-content-provider15:36
weshay|ruck2020-10-21tripleo-ci-centos-7-containers-multinode15:36
weshay|ruck2020-10-21tripleo-buildimage-overcloud-hardened-full-centos-715:36
weshay|ruck2020-10-21tripleo-buildimage-overcloud-full-centos-715:36
weshay|ruck2020-10-21tripleo-buildimage-ironic-python-agent-centos-715:36
weshay|ruckwhere content-provider.. is container-build15:37
weshay|ruckmarios|ruck, and remove non-voting queens jobs15:38
weshay|ruckso... remove15:38
weshay|ruck2020-10-21tripleo-ci-centos-7-scenario010-multinode-oooq-container15:38
weshay|ruck2020-10-21tripleo-ci-centos-7-scenario007-multinode-oooq-container15:38
weshay|ruck2020-10-21tripleo-ci-centos-7-scenario003-multinode-oooq-container15:38
weshay|ruck2020-10-21tripleo-ci-centos-7-scenario002-multinode-oooq-container15:38
weshay|ruck2020-10-21tripleo-ci-centos-7-scenario001-multinode-oooq-container15:38
weshay|ruck2020-10-21tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades15:38
weshay|ruck2020-10-21openstack-tox-cover15:38
weshay|ruckmarios|ruck, Alex and I decided to move those to nv 1 or 2 cycles ago.. permenately15:39
weshay|ruckbut left them as criteria in periodic.. for imports15:39
weshay|rucknow we don't have imports15:39
weshay|ruckso nuke'em15:39
weshay|ruckduke nukem15:40
marios|ruckweshay|ruck: i agree with what you write above. my understanding per the mail thread is "final decisions still to be made" regarding the import15:40
marios|ruckweshay|ruck: so once we have that yeah the above plan makes sense.15:40
*** ykarel has quit IRC15:41
marios|ruckweshay|ruck: we might even go further15:41
weshay|ruckk...  I expect the TRAC to make a decision on this in early December.15:41
marios|ruckweshay|ruck: and consider marking the branch unmaintained. because eventually any jobs we leave there will break and no one will be fixing those15:41
marios|ruckor at least it won't be any ones priority15:41
weshay|ruckmarios|ruck, so let's revisit this in a couple weeks15:42
weshay|ruckmarios|ruck, no no15:42
weshay|rucklet's keep queens as maintained.. I think we HAVE to... but only w/ the one job15:42
weshay|ruckextended maint.. and all15:42
marios|ruckweshay|ruck: i don't think we *have* to because of the stable:follows-policy thing i mentioned as pointed out by apevec15:43
weshay|ruckI think that's not correct15:43
marios|ruckweshay|ruck: i.e. if we decide to make it unmaintained, there is a possibility the TC will raise an objection15:43
* weshay|ruck looks at opendev doc15:43
marios|ruckweshay|ruck: k well it would be good to find out15:43
marios|ruckweshay|ruck: i was looking at that today https://governance.openstack.org/tc/reference/tags/stable_follows-policy.html15:44
marios|ruckweshay|ruck: especially @ Rationale "some do not actively backport significant bugfixes, some don’t monitor proposed backports, or monitor the CI system on their stable branches…"15:45
marios|ruckweshay|ruck: you can see repo tags at https://opendev.org/openstack/governance/src/branch/master/reference/projects.yaml15:45
weshay|ruckmaybe it is up to us15:46
weshay|ruckSend an announcement to the openstack-discuss mailing list (in order to give some time for others to step up as maintainers if there are volunteers).15:46
weshay|ruckRemove any related zuul jobs that are defined in other repositories and not needed anymore.15:46
weshay|ruckPropose a patch against the given project/repository. (For example, see: https://review.opendev.org/#/c/677478/)15:46
weshay|ruckAfter the branch is tagged with $series-eol, request the infra team to delete the branch.15:46
weshay|ruckmarios|ruck, we could do a test run of that w/ rocky15:46
weshay|ruckdef.. no imports of rocky now15:47
marios|ruckweshay|ruck:  at https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases "The branch is under Extended Maintenance rules, but there are no maintainers." i mean if no one is maintaining it then we can just move to unmaintained?15:47
marios|rucki.e no community members availa to maintain it15:47
marios|ruckweshay|ruck: ack ok15:47
weshay|ruckso initiate destruction of tripleo rocky.. any time..15:48
weshay|ruckand in dec.. we'll find out about queens15:48
marios|ruckweshay|ruck: ok i will do that next week15:48
marios|ruckweshay|ruck: and see what happens15:48
marios|ruckweshay|ruck: hmm but why rocky first15:50
marios|ruckweshay|ruck: i mean queens is older15:50
weshay|ruckmarios|ruck, rocky first.. because there are no imports15:53
*** rlandy|rover|brb is now known as rlandy|rover15:53
marios|ruckweshay|ruck: ack15:53
weshay|ruckmarios|ruck, ping me next week and we can work on it together15:53
weshay|ruckwould be nice to lighten everyones load15:53
weshay|ruckand focus15:53
marios|ruckweshay|ruck: ack15:53
weshay|ruckrecording .. https://hackmd.io/m--0qqZiQRGfFKl8CPb40Q15:59
weshay|ruckfyi.. master is promotiong16:02
weshay|ruckpromoting even16:02
marios|rucknice so confirmed that is transient then https://bugs.launchpad.net/tripleo/+bug/190493616:06
openstackLaunchpad bug 1904936 in tripleo "openstack.exceptions.ResourceNotFound: No Introspection found for node, Could not find node" [Critical,Triaged] - Assigned to Steve Baker (steve-stevebaker)16:06
rlandy|rover2020-11-20 15:56:36 | (<ansible.plugins.callback.validation_json.CallbackModule object at16:06
rlandy|rover2020-11-20 15:56:36 | 0x7fc1335abef0>): [Errno 2] No such file or directory: '/var/log/validations/fa16:06
rlandy|rover2020-11-20 15:56:36 | 163ee7-056e-f249-02a3-000000000007_deploy_steps_playbook_2020-11-20T15:53:02.8816:06
rlandy|roverhttps://logserver.rdoproject.org/53/18953/35/check/periodic-tripleo-ci-centos-8-singlenode-featureset050-upgrades-ussuri/157992d/logs/undercloud/home/zuul/undercloud_install.log.txt.gz16:06
marios|ruckweshay|ruck: well at least inconsistent even if it is not transient and we will see it again...16:06
marios|rucki love those ones16:07
rlandy|rovernot sure if that is a gerrit hitch or real issue on ussuri fs05016:07
rlandy|rover unable to parse the registries configuration (/etc/containers/registries.conf):16:07
rlandy|roverah16:08
rlandy|roverhttps://logserver.rdoproject.org/53/18953/35/check/periodic-tripleo-ci-centos-8-singlenode-featureset050-upgrades-ussuri/157992d/logs/undercloud/etc/containers/registries.conf.txt.gz16:08
marios|ruckrlandy|rover: well i'd like to see what that change is doing but we can't 1895316:09
marios|ruckrlandy|rover: i know that matbu was wiring up the validations in some places16:09
marios|ruckrlandy|rover: so it might be a legitimate check issue16:09
marios|ruckrlandy|rover: are you seeing it in other patches too?16:09
rlandy|rovermarios|ruck: juts the ussuri promotion so g=far16:10
rlandy|roverfar16:10
rlandy|roverbut who know what's cooking in check right now16:11
marios|ruckrlandy|rover: ah i thought that was check queue sorry16:11
* marios|ruck staring at too many logs16:11
rlandy|rovermarios|ruck: no - it's a testproject rerun16:11
marios|rucki c16:11
rlandy|roverI think we edited the registries.conf recently16:11
rlandy|roverunqualified-search-registries = ['registry.redhat.io', 'registry.access.redhat.com', 'registry.fedoraproject.org', 'registry.centos.org', 'docker.io']16:15
rlandy|rovermarios|ruck: either way  - not in crietria16:16
rlandy|rovercriteria16:16
rlandy|rovernever passed16:16
rlandy|roverhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-singlenode-featureset050-upgrades-ussuri16:17
rlandy|roverno worries16:17
marios|ruckrlandy|rover: k great16:17
rlandy|rovermarios|ruck: so the only think blocking ussuri is the scenario002 tempest thing16:17
rlandy|roverwill I can fix when zuul joins us again16:18
marios|ruckrlandy|rover: victoria is really teh biggest problem in terms of urgency for promotions i think16:19
marios|ruckrlandy|rover: https://review.opendev.org/#/c/763185/ will clear that hopefully16:19
marios|ruckrlandy|rover: but i think victoria is our most red branch16:19
marios|ruckrlandy|rover: wrt the handoff i mean16:20
rlandy|rovermarios|ruck: ack - that is the only one which we never manage to promote this week16:21
rlandy|roverconsistently16:21
rlandy|roverweshay|ruck: will also be out monday afternoon for closing but will transfer with rfolco and arxcruz in the early morning16:22
arxcruzrlandy|rover: ack16:22
rlandy|rover6 days agotempesta day ago16:30
rlandy|rover^^ component needs work16:31
rlandy|roverlooking16:31
rlandy|roverhttps://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-tempest-master/ec2bc36/logs/undercloud/var/log/tempest/stestr_results.html.gz16:34
rlandy|roverhmm16:34
rlandy|rovertempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.148 via SSH timed out.16:35
rlandy|roverUser: cirros, Password: None16:35
rlandy|roverlook familiar?16:35
marios|ruckrlandy|rover: maybe the scen2 issue (https://bugs.launchpad.net/tripleo/+bug/1904223 ) but may be a different root caus would have to check16:39
openstackLaunchpad bug 1904223 in tripleo "centos-8-scenario002-standalone failing tempest - SSH timed out" [Critical,Triaged]16:39
marios|ruckrlandy|rover: well it fails for a different tempest test16:40
marios|ruckrlandy|rover: weshay|ruck: i am giving serious consideration to withdrawing for the weekend in the next 20 mins ish (or less ;) )16:41
marios|ruckrlandy|rover: weshay|ruck: what say you16:41
rlandy|rovermarios|ruck: not sur eon that  rerunning with testproject atm16:41
*** amoralej is now known as amoralej|off16:43
weshay|ruckmarios|ruck, 0/ take care brotha .. please just note anything in the googlesheet.. that you would xfer on the hack for the bugs16:44
rlandy|rovermarios|ruck: might as well - zuul is run out on us16:45
rlandy|roverhave a good weekend16:45
marios|ruckrlandy|rover: as always thanks for being an excellent ruck|rover partner16:45
rlandy|rovermarios|ruck: back at you - always a pleasure to ruck/rover with you16:47
marios|ruck\o/ i'd say let's do it again but... ;)16:54
rlandy|roverdon't, please16:54
*** marios|ruck is now known as marios|out16:55
*** tosky has quit IRC16:56
*** jmasud has joined #oooq17:06
*** marios|out has quit IRC17:18
rlandy|roverweshay|ruck: to return to the rocky question .. should periodic-tripleo-ci-centos-7-undercloud-containers-rocky be in the integration line or not17:29
rlandy|roverit's in criteria atm but not the line17:29
rlandy|roverwhich means one I got wrong17:29
weshay|rucksorry.. back17:38
weshay|ruckrlandy|rover, just leave rocky alone as we may take the opportunity next week to really kill it17:39
rlandy|roverweshay|ruck: well I've killed it by messing up either the line or the criteria17:41
rlandy|roverthose two at least should match17:41
*** jbadiapa has quit IRC17:41
* rlandy|rover just edits criteria then if we don't care17:41
weshay|ruckwe dont17:42
weshay|ruckrlandy|rover, but as you see.. it takes a committee to kill a branch17:42
rlandy|roverweshay|ruck: k - np - I'll at least fix one problem17:45
*** jpena is now known as jpena|off17:51
*** jmasud has quit IRC18:02
*** tosky has joined #oooq18:10
*** derekh has quit IRC18:12
*** jmasud has joined #oooq18:22
*** gouthamr_ has quit IRC18:30
rlandy|roverdone18:34
*** jmasud has quit IRC18:37
*** jmasud has joined #oooq18:38
*** jmasud has quit IRC18:39
*** gouthamr_ has joined #oooq18:46
weshay|ruckrlandy|rover, btw.. you said you saw some conflicting data in the cockpit.. can you point that out to me today please?18:50
rlandy|roverweshay|ruck: the promotions18:53
rlandy|roverfor components18:53
rlandy|roverI think regexps are picking up stuff that is not in the lines18:53
rlandy|roverlook for example at tempest18:53
* weshay|ruck looks18:54
weshay|ruckmaster?18:54
rlandy|roverperiodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master18:54
rlandy|roverwhy is this in the tempest component line list of tests?18:54
rlandy|roveryes18:54
rlandy|roveralso the dates don't match18:54
weshay|ruckthe dates?18:54
weshay|ruckoh I see the name issue18:55
weshay|ruckthanks18:55
rlandy|roverweshay|ruck: the dates just seem behind18:56
rlandy|roverlike c7 train promoted18:56
rlandy|roverbut it was a while - maybe this morning that I saw the cockpit reflect that18:56
rlandy|roverthe component dates are the same18:56
rlandy|roverI found I went back to dlrn to check18:57
rlandy|roverthe dates on the hashes18:57
rlandy|roverbut then again, I am used to that18:57
weshay|ruckok.. I think tempest is fixed18:57
rlandy|roverweshay|ruck: also - tempest should show a standalone test18:58
rlandy|roverit's in criteria18:58
rlandy|roverand runs18:58
rlandy|roverbt not in list on master18:58
weshay|ruckhttps://review.rdoproject.org/r/3116619:00
weshay|ruckk.. it is now19:00
rlandy|roverweshay|ruck: k - I don't grok those regexp too well but I assume you're rght :)19:03
rlandy|roverright19:03
weshay|ruckrlandy|rover, it's saved on the current dashboard19:03
weshay|ruckso you can see it there too19:03
weshay|ruckrlandy|rover, http://dashboard-ci.tripleo.org/d/M1DYSidGz/component-pipeline?orgId=1&fullscreen&panelId=43119:04
weshay|ruckyou can login .. passwd in creds doc.. and you can toy around w/ the settings live.. then make a change19:04
weshay|ruckbasically we want.. periodic-tripleo-ci-centos-8-.*-tempest-master19:05
weshay|ruckthe dates are trickier.. I'll look at that in a bit19:05
rlandy|rovercool19:06
rlandy|roverweshay|ruck: wrt date - they are probably right - it's juts the component change every two hours19:06
rlandy|roverand I think the cockpit update is behind19:07
weshay|ruckrlandy|rover, yes.. it's a little behind..19:07
weshay|ruckjust balancing the polling19:07
rlandy|roverweshay|ruck: off the top of my head, I think that's all I noticed. I may recall something else later19:08
weshay|ruckk19:10
weshay|ruckrlandy|rover, pooja should be cleaning up these scripts for us and writing tests19:10
weshay|ruckbut I'll check in w/ her next week19:10
rlandy|roverhttps://logserver.rdoproject.org/65/31165/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-tempest-master/e41a576/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz19:11
rlandy|roveroops19:11
weshay|ruckonce that is done.. I'm hoping to tune things up and also update grafana19:11
rlandy|roversorry - was just debugging last tempest run and pasted it19:11
rlandy|roverok - great19:11
rlandy|roverweshay|ruck: I am going to apply for monday as a day off - afternoon I have closing and I need to get bank check etc. in the morning but I will be around early to sync with new ruck/rovers19:12
weshay|ruckk.. thanks19:12
*** tosky has quit IRC19:13
*** dtantsur is now known as dtantsur|afk19:15
rlandy|roverweshay|ruck: while upstream is on vaca, is there anything downstream we need to do?19:24
rlandy|roverpromote 16.2?19:24
* weshay|ruck looks19:28
weshay|ruckrlandy|rover, ya.. know.. we should drop the hash into a file.. and create a link to it.. in https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-16.2/e8e73c0/logs/README.html19:32
weshay|ruckwould make it a little easier19:32
weshay|ruckso we ran on 3978e606e62c42fc84ec72942529c76419:32
weshay|ruckinfrared https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-16.2_dlrn-rhel-8.3-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/19:33
weshay|ruck14371303d10a4999814661ce950c109a19:33
rlandy|roverok19:34
weshay|ruckda fuk19:34
weshay|ruckhit the 16.2 trigger and 17 is running19:34
rlandy|roverwhy is 17 running?19:38
weshay|ruckfak if I know19:39
rlandy|roverugh idk19:42
rlandy|roverweshay|ruck: maybe we just sync with attila again next week19:42
rlandy|roverand not guess19:42
rlandy|roverweshay|ruck: the hash is in the file19:43
rlandy|roveram I missing something?19:43
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/triggerfiles/redhat8/rhos-16.2/tripleo-ci-testing/trigger_file.txt19:43
rlandy|rover^^ hash19:43
weshay|ruckruch row19:44
weshay|ruckruh row19:44
rlandy|roveroops doesn't match19:46
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/triggerfiles/redhat8/rhos-16.2/14371303d10a4999814661ce950c109a/19:46
rlandy|roverweshay|ruck: it's got the previous hash19:46
*** rlandy|rover has quit IRC21:14

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!