Thursday, 2020-11-19

« Wednesday, 2020-11-18 Index Friday, 2020-11-20 »
*** tosky has quit IRC00:14
*** jmasud has joined #oooq00:26
*** jmasud has quit IRC00:28
*** jmasud has joined #oooq00:37
*** jmasud has quit IRC00:44
*** jmasud has joined #oooq00:53
*** jmasud has quit IRC00:59
*** jmasud has joined #oooq00:59
*** dmellado has quit IRC01:03
*** dmellado has joined #oooq01:04
*** jmasud has quit IRC01:10
*** jmasud has joined #oooq01:16
*** jmasud has quit IRC01:25
*** Goneri has quit IRC01:29
*** matbu has quit IRC01:58
*** ysandeep|holiday is now known as ysandeep|off02:59
*** skramaja has joined #oooq03:10
*** apetrich has quit IRC03:19
*** udesale has joined #oooq03:45
*** raukadah is now known as chandankumar04:03
*** ykarel has joined #oooq04:19
*** jmasud has joined #oooq04:42
*** jmasud has quit IRC05:10
*** jmasud has joined #oooq05:11
*** marios has joined #oooq06:05
*** ratailor has joined #oooq06:08
*** saneax has joined #oooq06:10
*** marios has quit IRC06:15
*** marios has joined #oooq06:18
*** sanjayu_ has joined #oooq06:37
*** saneax has quit IRC06:40
*** udesale_ has joined #oooq07:08
*** udesale has quit IRC07:11
*** ksambor has joined #oooq07:18
chandankumarmarios: sshnaidm I am tracking rdocloud nodeset removal here https://etherpad.opendev.org/p/remove-rdocloud-jobs07:32
marioschandankumar: thanks07:35
*** marios is now known as marios|ruck07:37
*** jpena|off is now known as jpena07:42
*** apetrich has joined #oooq07:46
*** jmasud has quit IRC07:51
*** bogdando has joined #oooq07:52
*** bhagyashris|off is now known as bhagyashris07:55
*** matbu has joined #oooq08:01
*** amoralej|off is now known as amoralej08:08
*** udesale has joined #oooq08:13
*** udesale_ has quit IRC08:13
chandankumarykarel: jpena fyi https://etherpad.opendev.org/p/remove-rdocloud-jobs08:14
*** udesale_ has joined #oooq08:15
*** udesale has quit IRC08:18
*** udesale_ has quit IRC08:19
*** udesale has joined #oooq08:19
*** udesale_ has joined #oooq08:22
*** udesale has quit IRC08:23
*** udesale_ has quit IRC08:27
ykarelchandankumar, ack08:40
*** tosky has joined #oooq08:44
*** ykarel_ has joined #oooq09:04
*** ykarel has quit IRC09:07
*** matbu has quit IRC09:09
* chandankumar will back in an hour09:23
*** skramaja has quit IRC09:29
*** holser has joined #oooq09:33
*** dtantsur|afk is now known as dtantsur09:41
*** frenzy_friday has quit IRC09:43
*** frenzy_friday has joined #oooq09:44
*** matbu has joined #oooq09:52
*** ykarel_ is now known as ykarel09:57
*** ykarel_ has joined #oooq10:33
*** bogdando has quit IRC10:35
*** ykarel has quit IRC10:35
*** ratailor_ has joined #oooq11:18
*** ratailor has quit IRC11:21
*** ykarel__ has joined #oooq11:25
*** ykarel_ has quit IRC11:28
*** ykarel__ is now known as ykarel11:33
*** ratailor_ has quit IRC11:46
*** holser has quit IRC12:29
*** jpena is now known as jpena|lunch12:37
chandankumarykarel: please have a look when free https://review.rdoproject.org/r/#/c/31153/12:52
weshay|ruckchandankumar, thanks for rfe booog12:54
weshay|ruckquick flash poll12:55
weshay|ruckrfolco is out.. want me to drive the mtg? or should I stay out as promised?12:56
chandankumarweshay|ruck: I am taking care of that12:56
chandankumarweshay|ruck: you can join this one12:56
weshay|ruckok.. let me know if you need anything12:56
ykarelchandankumar, ack btw you seen jpena|lunch comment on other patch?12:57
ykarelon https://review.rdoproject.org/r/#/c/31143/12:57
marios|ruckweshay|ruck: there was a card created for that bug ... https://bugs.launchpad.net/tripleo/+bug/1885286/comments/9 do you have more info? any idea why it was just created?12:58
openstackLaunchpad bug 1885286 in tripleo "Increase in RETRY_LIMIT errors in zuul.openstack.org is preventing jobs from passing check/gate" [Critical,Fix released] - Assigned to Ronelle Landy (rlandy)12:58
chandankumarykarel: then I think we are good with this https://review.rdoproject.org/r/#/c/31143/ ?12:58
* weshay|ruck checks12:58
ykarelchandankumar, main idea is to sync all these patches, and decide how to move on in etherpad itself12:59
chandankumararxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar - scrum time https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=4285&view=detail&selectedIssue=TRIPLEOCI-24712:59
chandankumarykarel: I was keeping it short to review easily13:00
ykareli was more in favor of not using vexxhost at all in job names in vars13:00
ykarelhttps://etherpad.opendev.org/p/remove-rdocloud-jobs13:00
chandankumarykarel: then I think we can merge the nodeset patch13:00
weshay|ruckmarios|ruck, it *just* moved to fixed-released13:00
weshay|ruckthat's why13:00
*** rlandy has joined #oooq13:01
*** rlandy is now known as rlandy|rover13:01
chandankumarweshay|ruck: want to join the meeting?13:01
weshay|ruckif you want me to .. sure13:02
chandankumaryes please13:02
marios|ruckweshay|ruck: no i am the one that just moved it to fix released13:03
marios|ruckweshay|ruck: i did that after i saw the card13:03
marios|ruckweshay|ruck: ok fine whatever i was more wondering if ther ewas an issue and that's why you added the card or something13:03
marios|ruckweshay|ruck: k weird, we can just close it out then13:03
weshay|ruckmarios|ruck, yes I know13:04
weshay|ruckyou *just* moved it to fix-released... there probably was a card that was closed a long time ago and no one closed the bug13:04
weshay|ruckmarios|ruck, /me finding a lot of that.. where the cix is closed but the bug is not closed and should be13:05
*** amoralej is now known as amoralej|lunch13:05
weshay|ruckanyway.. this tool I'm writing is finding that.. will be easy to spot now13:05
marios|ruckweshay|ruck: the card appeared today13:06
marios|ruckweshay|ruck: before i closed it13:06
marios|ruckweshay|ruck: before i closed the bug13:06
weshay|ruckfyi.. ya. close the bug first13:07
*** jbadiapa has joined #oooq13:09
*** jfrancoa has quit IRC13:10
*** jfrancoa has joined #oooq13:10
chandankumarhttps://review.rdoproject.org/r/#/c/31133/ <- go script for quay13:13
rlandy|rovermarios|ruck: queens promoted :)13:13
chandankumarhttps://review.opendev.org/#/c/759823/13:14
arxcruzchandankumar: weshay|ruck https://review.rdoproject.org/r/#/c/31133/13:14
chandankumarhttps://review.opendev.org/#/c/763005/ might impact13:16
*** jpena|lunch is now known as jpena13:32
apetrichweshay|ruck, Forgot to ping you. I fixed some stuff we talked about and added the controls you asked to the heat dashboard https://review.rdoproject.org/r/#/c/30834/13:35
*** holser has joined #oooq13:35
*** chem has quit IRC13:46
*** Goneri has joined #oooq13:47
*** beagles has quit IRC13:50
*** chem has joined #oooq13:50
*** b3nt_pin has joined #oooq13:50
weshay|ruck[06:22:59] <weshay|ruck> jjoyce, any queens imports needed?13:55
weshay|ruck[06:25:07] * francoisp|afk is now known as francoisp13:55
weshay|ruck[06:27:49] <jjoyce> weshay|ruck: Not currently we are in the middle of a maintenance cycle for 13 currently.13:55
weshay|ruckmarios|ruck, rlandy|rover ^13:55
weshay|ruckrlandy|rover, marios|ruck we may want to consider... removing everything except build jobs and fs010 from queens as well13:56
fuzzball81weshay|ruck: We will likely need an import again at a later time.13:57
weshay|ruckarxcruz, reminder.. email opendev-discuss[tripleo]13:57
arxcruzweshay|ruck: oh, i sent to rh-dev13:57
arxcruzrhos-dev *13:58
weshay|ruckcan't have nice things13:58
arxcruz:(13:58
weshay|ruckjust cut / paste .. send :)13:59
weshay|ruckemail looks good13:59
arxcruzweshay|ruck: done, sorry, I pretty sure i understood you say rhos-dev13:59
weshay|ruckya.. I said openstack-discusss[tripleo]13:59
weshay|ruckno worries14:00
*** b3nt_pin has quit IRC14:00
*** b3nt_pin has joined #oooq14:00
weshay|ruckarxcruz, where's the log14:03
arxcruzweshay|ruck: /var/log/copy-quay.log on the toolbox14:03
weshay|ruckin the email?14:03
arxcruzweshay|ruck: I copied the log there, just 3 lines14:03
weshay|ruckarxcruz, my intent.. is that you posted the log url14:04
weshay|ruckso folks can check the sync when needed14:04
*** amoralej|lunch is now known as amoralej14:04
weshay|ruckarxcruz, paste me the url please here14:04
arxcruzweshay|ruck: hmmm i don't think toolbox have it14:05
*** ykarel_ has joined #oooq14:05
weshay|ruckarxcruz, ya.. make it happen please14:05
arxcruzweshay|ruck: ok14:05
*** ykarel has quit IRC14:07
weshay|ruckarxcruz, nginx is on the box..14:08
arxcruzweshay|ruck: yes, i'm looking into it right now14:08
weshay|ruckthanks14:08
*** TrevorV has joined #oooq14:16
weshay|ruckapetrich, ah k /me looks14:16
apetrichweshay|ruck, amazing!14:17
weshay|ruckapetrich, you can snapshot and post the dev work .. easier to review that way.. no worries.. can show you another time or in a few min if you want14:18
apetrichI saw about the snapshot on the readme but when I tried it didn't quite work14:19
apetrichI don't recall what I will do it next time and if I have any issue I will ask around14:19
weshay|ruckapetrich, ya.. there's a toggle on how much data it will snap...14:20
weshay|ruckusually it works if you don't do too much..14:20
weshay|ruckanyhoo.. bringing it up now to look14:20
weshay|ruckthanks14:20
rlandy|roverweshay|ruck: so we keep queens as is - to clarify14:29
rlandy|roverin case of needed import14:29
weshay|ruckrlandy|rover, well.. tbh not 100% sure.. it's near internal eol14:30
weshay|ruckperhaps we can lighten the load upstream14:30
weshay|ruckand just verify the voting job14:30
weshay|ruckthis is a thing to reaccess every 6months or so14:30
rlandy|roverweshay|ruck: it runs twice a week - not the biggest deal14:31
weshay|ruckk.. perhaps we'll get some conv.. going in the next prod-chain14:31
rlandy|rovermarios|ruck: periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queenstestprojectmastercheck31145,14 hrs 23 mins 55 secs2020-11-19 09:31:13SUCCESS14:32
rlandy|rover^^ so we expect another promotion today14:32
marios|ruckrlandy|rover: great \o/14:33
weshay|ruckrock14:33
marios|ruckrlandy|rover: it is a good hash too from 18th so more recent14:33
rlandy|roverack14:33
rlandy|roverk - trying our luck again with stein/rocky testproject14:37
*** yolanda has quit IRC14:40
rlandy|roverweshay|ruck: hmmm ... https://sf.hosted.upshift.rdu2.redhat.com/triggerfiles/redhat8/rhos-16.2/14:40
rlandy|roverwill there be a need for a current-tripleo version here?14:40
rlandy|roverie: will they ever need to trigger jenkins jobs off the change to current-tripleo?14:41
* rlandy|rover can't think of a case14:41
weshay|ruckrlandy|rover, they shouldn't trigger off current-tripleo.. but it may be nice for us..14:41
rlandy|roverweshay|ruck: k - then we will need an equivalent promotion script like we have in dockerfiles14:41
* rlandy|rover takes care of that later14:41
*** amoralej is now known as amoralej|brb14:43
weshay|ruckrlandy|rover, fyi.. /me working to get master tripleo-component -> master integration -> security-component promoted for sec folks14:46
rlandy|roverweshay|ruck: ^^ completely lost there14:46
rlandy|roverOH14:46
rlandy|roverI see14:46
weshay|ruck20 days ago14:47
weshay|rucksecurity14:47
weshay|ruck6 hours ago14:47
rlandy|roversec need tripleo component upgraded14:47
weshay|ruckthat libvirt fix needs to get through14:47
rlandy|roverweshay|ruck: thanks14:47
rlandy|rovermarios|ruck: openstack-periodic-integration-stable3-centos7 current run - looking hopeful14:49
*** fuzzball81 is now known as jjoyce14:53
weshay|ruckapetrich, https://snapshot.raintank.io/dashboard/snapshot/MOIrEG8NLlAa8aQvY5m3X807vDMBIJxX?orgId=214:54
weshay|ruckapetrich, let's look at that together14:54
weshay|ruckapetrich, does it make sense to break the gate graph by branch?14:55
weshay|ruckor is there not enough through put14:55
weshay|rucklike.. heat is not active enough to make a diff by branch14:55
weshay|ruckI think this is good14:57
weshay|ruckmerging14:57
apetrichweshay|ruck, sorry I was taking a breather before a meeting. thanks14:58
weshay|ruckapetrich, np14:59
marios|ruckrlandy|rover: ack15:01
*** amoralej|brb is now known as amoralej15:03
arxcruzweshay|ruck: http://38.102.83.131/copy-quay.txt15:13
arxcruzweshay|ruck: running right now15:13
weshay|ruckarxcruz, NICE15:14
arxcruzit's txt becuase nginx was trying to download .log15:14
arxcruzand i didn't want to dig too much into it15:14
weshay|ruckarxcruz, aye.. I had the same issue15:15
weshay|ruckso.. how did you expose the log w/o nginx or w/ nginx w/o download15:15
weshay|ruckarxcruz, not sure why nginx does that ..15:16
arxcruzweshay|ruck: had to mount a volume in /root/logs on toolbox with :z option15:16
weshay|ruckarxcruz, on http://dashboard-ci.tripleo.org/diff  click log15:16
weshay|ruckcouldn't fix that15:16
arxcruzweshay|ruck: /tmp and /var/log is selinux protected15:16
weshay|ruckhrm k15:17
arxcruzweshay|ruck: i also found this15:17
arxcruzpodman pull trunk.registry.rdoproject.org/tripleostein/centos-binary-swift-object-expirer:current-tripleo15:17
arxcruzon tripleostein15:17
arxcruzeven the log says this image was successfully built15:17
arxcruzi'm not able to pull it15:17
arxcruzTrying to pull trunk.registry.rdoproject.org/tripleostein/centos-binary-swift-object-expirer:current-tripleo...15:17
arxcruz  manifest unknown: manifest unknown15:17
arxcruzError: unable to pull trunk.registry.rdoproject.org/tripleostein/centos-binary-swift-object-expirer:current-tripleo: Error initializing source docker://trunk.registry.rdoproject.org/tripleostein/centos-binary-swift-object-expirer:current-tripleo: Error reading manifest current-tripleo in trunk.registry.rdoproject.org/tripleostein/centos-binary-swift-object-expirer: manifest unknown: manifest unknown15:17
*** tosky has quit IRC15:18
weshay|ruckarxcruz, hrm.. interesting.. never even heard of that container15:19
arxcruzweshay|ruck: it's on the list of built container15:21
weshay|ruckarxcruz, I don't see that container in docker.io15:21
weshay|ruckstein15:21
arxcruzweshay|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-stein-containers-build-push/93bd71b/logs/containers-successfully-built.log15:22
*** tosky has joined #oooq15:22
*** tosky has quit IRC15:26
*** ykarel_ has quit IRC15:29
rlandy|roverweshay|ruck: hey ... https://review.opendev.org/#/c/763233/2/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-stein.yml ... have a sec to discuss - while I am duplicating that change15:36
weshay|ruckrlandy|rover, in a mtg.. for 20 more min15:36
rlandy|roverweshay|ruck: no rush - pls ping when you have time15:37
*** tosky has joined #oooq15:50
*** b3nt_pin has quit IRC15:51
marios|ruck"you are about to close 52 tabs are you sure"16:07
marios|ruck:/16:07
rlandy|roverlol16:16
weshay|ruckrlandy|rover, k.. ready16:21
rlandy|roverone sec - triggering jobs16:21
rlandy|roverweshay|ruck: ok ... https://meet.google.com/bny-yssa-uvj16:23
*** beagles has joined #oooq16:33
*** jmasud has joined #oooq16:35
*** jpena is now known as jpena|off16:42
chandankumarweshay|ruck: rlandy|rover https://review.rdoproject.org/r/#/c/31143/16:47
rlandy|roverchandankumar++ thanks16:47
weshay|ruckchandankumar, thank you sir16:48
chandankumarweshay|ruck: rlandy|rover https://etherpad.opendev.org/p/remove-rdocloud-jobs16:48
rlandy|rovermarios|ruck: look legit to you? https://logserver.rdoproject.org/69/29969/17/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein/f88954d/logs/undercloud/home/zuul/overcloud_update_prepare.log.txt.gz16:49
marios|ruckrlandy|rover: looking16:49
rlandy|roverhmmm ... thought we fixed that16:50
marios|ruckrlandy|rover: hmm yeah thnis is related to the fix from jfrancoa16:50
rlandy|rovermarios|ruck: ugh - maybe not in yet16:50
marios|ruckrlandy|rover: https://review.opendev.org/#/c/762871/16:50
rlandy|rovermarios|ruck: yeah - it is in16:51
rlandy|rover2 days ago16:51
marios|ruckrlandy|rover: k gonna have to dig into that more tomorrow about to go in couple mins.16:52
marios|ruckrlandy|rover: add anything you find to hackmd and i'll pickup please?16:52
marios|ruckrlandy|rover: friday \o/16:52
rlandy|rovermarios|ruck: sure16:52
rlandy|rovermarios|ruck: counting the hours16:52
rlandy|roverchandankumar: https://review.opendev.org/#/c/762400/ - do I have all the changes here now?16:57
*** marios|ruck is now known as marios|out16:57
marios|outrlandy|rover: please check https://review.opendev.org/763005 and if it needs recheck :/16:58
marios|outthank you16:58
* marios|out out out16:58
rlandy|rovermarios|out: ack16:58
rlandy|roversee you tomorrow16:58
rlandy|roverlast day!!!!16:58
rlandy|roverone day more ....16:59
*** marios|out has quit IRC17:00
chandankumarrlandy|rover: done17:12
rlandy|roverthanks17:12
*** amoralej is now known as amoralej|off17:31
rlandy|roverweshay|ruck: 16.2 looking good on our side and created the trigger https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status18:00
rlandy|roverso we shoudl eb good to go when the jobs trigger from QE18:00
weshay|ruckrlandy|rover++18:00
*** sanjayu_ has quit IRC18:08
rlandy|roverweshay|ruck: https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-cloudops-master/dcbd89c/logs/undercloud/var/log/tempest/stestr_results.html.gz18:10
rlandy|rover^^ blocking cloudops component18:10
rlandy|rovergoing to remove job clause from skiplist18:11
rlandy|roverneeds to skip on all18:11
weshay|ruckrlandy|rover, it passed on the 18th18:11
rlandy|roverwell it's failing now18:12
weshay|ruckk18:12
rlandy|roverprobably because that hash promoted18:12
* rlandy|rover puts up patch18:12
weshay|ruckrlandy|rover, can I waive periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master on an introspection error?18:14
rlandy|roverweshay|ruck: if fs020 passed, yes18:14
weshay|ruckrlandy|rover, it's component18:14
weshay|rucktripleo component18:14
rlandy|roverOH - that tripleo component name thing again18:15
rlandy|roveryeah - probably ok18:15
rlandy|roverI'll rerun it18:15
rlandy|roverbut go ahead if it's blcoking sec18:15
rlandy|roverblocking18:15
weshay|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master%0918:16
weshay|ruckprobably something real. and running on vex18:16
*** jmasud has quit IRC18:20
weshay|ruck2020-11-19 16:40:08,357 p=243788 u=zuul n=ansible | 2020-11-19 16:40:08.355895 | fa163ea9-a69a-fa1d-c086-000000000016 |      FATAL | Nodes that failed introspection | localhost | error={18:21
weshay|ruck    "msg": " 16e4d0ae-37c5-4f5e-8217-e45bd5e37ac8"18:21
weshay|ruck}18:21
weshay|ruckso just one failed18:21
weshay|ruckrlandy|rover, meh.. trying again18:23
rlandy|roverarxcruz: weshay|ruck: https://review.opendev.org/763424 - sorry - one more time with skipping the luks test18:23
*** dtantsur is now known as dtantsur|afk18:23
weshay|ruckrlandy|rover, https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0&range=16:1618:26
rlandy|roverweshay|ruck: no access18:27
weshay|ruckgranted18:29
* weshay|ruck starts to close old bugs18:30
rlandy|roveroh fancy18:30
*** jmasud has joined #oooq18:32
rlandy|roverweshay|ruck: pls take a look at https://review.opendev.org/#/c/763233/ before I duplicate that across all c7 releases18:47
* rlandy|rover looks at network component now18:55
rlandy|roververy red18:55
*** jmasud has quit IRC19:05
*** chem has quit IRC20:12
*** chem has joined #oooq20:14
weshay|ruckrlandy|rover, you wanted to chat?20:28
rlandy|roverweshay|ruck: do we need to?20:43
rlandy|roverweshay|ruck: oh the component stuff20:44
rlandy|roverI went through all the components20:44
rlandy|roverreran some20:44
rlandy|roversome were failing on scneario00220:44
rlandy|rovernetwork bug is known20:45
rlandy|rovercompute bug is known20:45
rlandy|roverthink it's ok20:45
weshay|ruckk20:45
weshay|ruckthanks20:45
rlandy|roverweshay|ruck: https://review.opendev.org/#/c/763424/ merged - so scenario002 stuff should skip everywhere now20:46
rlandy|roverfreeing up a bunch of components20:46
weshay|ruckaye.. saw that20:46
weshay|ruckit's nice that skiplist is so quick20:46
rlandy|roverbaremetal failure is in rerun20:47
rlandy|roverneed to confirm tempest20:47
rlandy|roverI think that is the last component to look at20:47
weshay|ruckrlandy|rover, I think this is accurate now https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=020:47
* weshay|ruck cleaned up cards and bugs20:47
rlandy|roverweshay|ruck: aslo wanted a review on https://review.opendev.org/#/c/763233/ before I dup'ed it20:48
rlandy|roveron other releases20:48
weshay|ruckby moving bugs.. and cards.. not in the ss :)20:48
rlandy|roverss?20:48
weshay|ruckrlandy|rover, any test projects on it20:48
rlandy|roveroh spreadsheet20:48
weshay|ruckspreadsheet20:48
weshay|rucknot nazi's20:48
rlandy|roverlol20:48
rlandy|rovermy thought exactly20:48
weshay|ruckaye20:48
weshay|rucklooks good to me20:49
rlandy|roverweshay|ruck: wrt train c7 ... we are two tests away from promotion - both in rerun now - ovb fs002 and fs03520:51
rlandy|rover I think 002 will pass20:51
rlandy|roverfs035 may timeout20:51
weshay|ruckk20:51
rlandy|roverok to skip fs035 to get a promotion?20:51
rlandy|roverwe are weeks out on that20:51
weshay|ruckif another c7 ovb passes we'll just waive20:51
weshay|ruckya20:51
rlandy|roverweshay|ruck: k - will wait on fs002 and then ping you20:52
*** jbadiapa has quit IRC20:52
weshay|ruckrlandy|rover, I officially hate this review https://review.opendev.org/#/c/757821/21:01
rlandy|roveroh I thought it was mine21:01
weshay|ruckboth failing jobs21:01
rlandy|roverlooking21:01
weshay|ruckare in the 80%+ pass rate http://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&var-influxdb_filter=branch%7C%3D%7Cstable%2Fussuri21:01
rlandy|roveryou hate it because the failures are real or sporadic?21:02
weshay|ruckatm.. I think sporatic.. they were real21:03
rlandy|roverrecheck and pray routine21:03
*** TrevorV has quit IRC21:09
weshay|ruckrlandy|rover, ok.. introspect failed twice21:33
weshay|ruckvictoria works, master failed twice21:34
rlandy|roverweshay|ruck: which job are we talking about now?21:34
weshay|ruckso.. probably real21:34
* rlandy|rover lost track21:34
weshay|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master21:34
* weshay|ruck opens bug21:34
rlandy|rovertripleo21:34
rlandy|roverweshay|ruck: k - will look once at the bug21:35
rlandy|roverweshay|ruck: bug? ^^21:46
weshay|ruckrlandy|rover, https://bugs.launchpad.net/tripleo/+bug/190493621:47
openstackLaunchpad bug 1904936 in tripleo "openstack.exceptions.ResourceNotFound: No Introspection found for node, Could not find node" [Critical,Triaged]21:47
weshay|ruckrlandy|rover, I don't see the metalsmith yaml21:47
rlandy|roverk - will look at it21:47
rlandy|roverclosing https://bugs.launchpad.net/tripleo/+bug/1903996/ - we have two queens promotions21:48
openstackLaunchpad bug 1903996 in tripleo "Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE" [Critical,Triaged]21:48
weshay|ruck++21:48
rlandy|roverweshay|ruck: so I closed the bug https://bugs.launchpad.net/tripleo/+bug/1903996/ - I have not moved the card yet21:49
openstackLaunchpad bug 1903996 in tripleo "Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE" [Critical,Fix released]21:49
weshay|ruckI guess baremetal / metalsmith is not there because introspection failed21:49
rlandy|roverdo you want to see the output21:49
weshay|ruckrlandy|rover, we don't log nodes.json?21:52
weshay|ruckdont' see instackenv.json21:52
rlandy|roverweshay|ruck: sorry was busy with other review - looking now21:53
rlandy|roverweshay|ruck: ok so we have the list of imported nodes ... https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master/7ac06d9/logs/undercloud/home/zuul/overcloud_import_nodes.log.txt.gz21:56
weshay|ruckya21:57
* rlandy|rover looks for errors now21:57
rlandy|roveropenstack.exceptions.ResourceNotFound ... looking21:57
rlandy|roverttps://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master/7ac06d9/logs/undercloud/var/log/containers/ironic-inspector/ironic-inspector.log.txt.gz21:58
rlandy|rover2020-11-19 15:38:39.674 8 INFO ironic_inspector.conductor.manager [-] Coordination backend cannot be started, assuming no other instances are running. Error: No module named 'pymemcache'21:58
rlandy|roverweshay|ruck: ^^21:58
rlandy|rover2020-11-19 16:20:11.704 8 ERROR ironic_inspector.utils [req-f3c85062-7904-4bcf-a685-be0c6671d7df ce5cf8e655bf40dea455ee91426eda67 233cf286c3734182bd8f2aa9cfc3afd2 - default default] Could not find node e2e592cf-e6c5-474a-acf2-22a77b973088 in cache21:59
* weshay|ruck looks21:59
rlandy|roverthat's why the nodes are not in cache22:00
weshay|ruckrlandy|rover, well.. that's also here https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-master/2fcff2e/logs/undercloud/var/log/containers/ironic-inspector/ironic-inspector.log.txt.gz22:00
weshay|ruckand that job worked.. /me double checking22:00
weshay|ruckrlandy|rover, still could be it22:01
weshay|ruck default default] Could not find node 93699afc-3b8e-4d7c-8d85-55aaf274733d in cache22:01
rlandy|roverweshay|ruck: side point ... periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-train-upload - success22:14
rlandy|rovercan we edit config for train c7 and restart promoter?22:15
rlandy|roverfs035 will be missing22:15
rlandy|roverneed to waive that22:15
weshay|ruckk.. give me a few22:23
rlandy|roverweshay|ruck: np - edited the config for train c722:25
rlandy|roverwill restart promoter when queens is done22:25
rlandy|roverweshay|ruck: I can take care of it - but I know you like to be on top of what happens on the promoter so just letting you know22:26
weshay|ruckrlandy|rover, ok.. so you updated the config22:29
rlandy|roverweshay|ruck: yep22:29
rlandy|roverself service :)22:29
rlandy|roverweshay|ruck: but queens is busy promoting22:29
rlandy|roverso I left the restart22:29
weshay|ruckya..22:30
weshay|ruckk22:30
rlandy|roverweshay|ruck: can restart when queens is done22:30
rlandy|roverin the mean time I can rerun fs035 again22:30
rlandy|roveroff chance it passes22:31
rlandy|roverif not restart22:31
*** jmasud has joined #oooq22:41
rlandy|roverweshay|ruck: baremetal component will promote22:45
weshay|ruckrlandy|rover, on master?22:46
rlandy|roverperiodic-tripleo-ci-centos-8-standalone-baremetal-master reran and passed22:46
rlandy|roveryep22:46
weshay|ruckok.. interesting22:46
weshay|rucktripleo is going to promote too22:46
rlandy|roverhttp://dashboard-ci.tripleo.org/d/M1DYSidGz/component-pipeline?orgId=122:46
rlandy|rover2020-11-19 02:54:3322:46
rlandy|rovermaster22:46
rlandy|roverNone22:46
rlandy|roverperiodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master22:46
rlandy|rover2 hours, 45 minutes22:46
rlandy|roverSUCCESS22:46
rlandy|roverYES22:46
weshay|ruckI'll kick integration here in a bit for master22:46
rlandy|rover2020-11-19 01:02:26masterNone periodic-tripleo-ci-centos-8-scenario012-standalone-baremetal-master58 minutes, 8 secondsSUCCESSYES22:46
rlandy|rover2020-11-19 00:20:36masterNone periodic-tripleo-ci-centos-8-standalone-baremetal-master22:46
rlandy|roverok22:47
rlandy|roverdo you want to run promote components?22:47
*** jmasud has quit IRC22:47
rlandy|roverso you have the latest?22:47
weshay|rucktriggered openstack-promote-component22:47
rlandy|rovercool22:49
rlandy|rovertempest is the last component I need to look at22:50
rlandy|roverthe rest should be sorted22:50
rlandy|roverweshay|ruck: warehouse duty now - will restart promoter when I get back23:08
*** Goneri has quit IRC23:10
weshay|ruckrlandy|rover, c7-train is promoting now23:15
rlandy|rovercool23:15
*** jmasud has joined #oooq23:27
*** jmasud has quit IRC23:32
*** jmasud has joined #oooq23:33
*** jmasud has quit IRC23:35
*** jmasud has joined #oooq23:35
*** jmasud has quit IRC23:39
« Wednesday, 2020-11-18 Index Friday, 2020-11-20 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!