Friday, 2020-06-19

« Thursday, 2020-06-18 Index Saturday, 2020-06-20 »
*** rlandy|ruck2 has quit IRC00:01
*** dsneddon has quit IRC00:27
*** rfolco has quit IRC00:28
*** dsneddon has joined #oooq00:29
*** jmasud has quit IRC00:42
*** jmasud has joined #oooq00:55
*** Goneri has quit IRC01:07
*** jmasud has quit IRC01:12
*** jmasud has joined #oooq01:20
*** jmasud has quit IRC02:14
*** jmasud has joined #oooq03:02
*** jmasud has quit IRC03:23
*** jmasud has joined #oooq03:25
*** ysandeep|away is now known as ysandeep04:05
*** jmasud has quit IRC04:12
*** skramaja has joined #oooq04:35
*** apetrich has quit IRC04:42
*** ykarel|away is now known as ykarel04:51
*** jmasud has joined #oooq04:59
*** soniya29 is now known as soniya29|ruck05:08
*** udesale has joined #oooq05:34
*** pojadhav|away is now known as pojadhav05:38
*** soniya29|ruck is now known as soniya29|rover05:40
*** marios has joined #oooq06:00
*** jfrancoa has joined #oooq06:02
*** ratailor has joined #oooq06:08
*** ratailor has quit IRC06:22
*** tosky has joined #oooq06:30
*** ratailor has joined #oooq06:30
*** jbadiapa has quit IRC06:34
*** jmasud has quit IRC07:01
*** dtantsur|afk is now known as dtantsur07:17
*** amoralej|off is now known as amoralej07:19
*** bhagyashris|away is now known as bhagyashris07:39
*** jpena|off is now known as jpena07:57
*** jmasud has joined #oooq08:27
*** jbadiapa has joined #oooq08:32
jbadiapaHi giulio,08:32
jbadiapaI talked to randy regarding https://review.opendev.org/#/c/711423/08:33
jbadiapaHe told me that if we are sure that only ceph-nfs is the client for the mds to go down is ok with the changes...that's why I came back to you08:36
jbadiapais there any other client supported?08:36
*** soniya29|rover is now known as soniya29_lunch|r08:43
*** soniya29_lunch|r is now known as soniya29|rover08:44
*** ykarel is now known as ykarel|lunch08:45
zbrpanda: morning08:48
zbrlook at https://logserver.rdoproject.org/38/28038/20/check/molecule-tripleo-common-delegated-centos-7/e753f3c/job-output.txt08:50
zbrsearch for FAILED!, basically it creates the directory but next task fails to run08:51
zbrmy suspicion is that someone messed file permissions for zuul user, likely by running some commands as root08:51
zbrif test-python environment (or a subfolder) is created by root, for example.08:52
zbrneverming, i found the culprit, someone added a become: true to creation of venv....08:54
*** apetrich has joined #oooq09:13
*** ysandeep is now known as ysandeep|brb09:29
*** chem has quit IRC09:30
*** chem has joined #oooq09:32
*** ykarel|lunch is now known as ykarel09:48
*** dsneddon has quit IRC10:24
soniya29|roverchandankumar, ykarel, periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-stein job is failing , as per logs undercloud is failing10:33
soniya29|roverchandankumar, ykarel, logs:- https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-stein/338314c/job-output.txt10:35
*** jmasud has quit IRC10:38
*** pojadhav is now known as pojadhav|afj10:47
*** pojadhav|afj is now known as pojadhav|afk10:47
ykarelsoniya29|rover, as per logs, tempest is failing not undercloud https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-stein/338314c/logs/tempest.html.gz10:48
ykarelu seeing where ?10:48
soniya29|roverykarel, i looked here:-https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-stein/338314c/job-output.txt10:52
ykarelsoniya29|rover, in ^^ "cmd": "set -o pipefail &&subunit2junitxml /home/zuul/tempest/testrepository.subunit --output-to /home/zuul/tempest/tempest.xml 2>&1 >> tempest.log\n", failed10:53
ykarelmay be u got confused with undercloud                 : ok=18   changed=9    unreachable=0    failed=1    skipped=36   rescued=0    ignored=110:53
soniya29|roverykarel, yes. I took it as undercloud failure10:54
ykarelsoniya29|rover, ack, no it's not undercloud failure10:55
ykarelsee https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-stein/338314c/logs/10:56
ykarel_Tempest_test_failed.log10:56
*** marios has quit IRC11:08
chandankumarsoniya29|rover, ykarel fs21 job is meant to be ignored11:08
*** ratailor has quit IRC11:09
ykarelonly if it's tempest failure11:09
soniya29|roverchandankumar, ykarel, i didn't get?11:09
chandankumarykarel, does fs020 job is working fine11:10
ykarelchandankumar, i didn't checked11:10
chandankumarsoniya29|rover, fs021 and fs020 are same job only difference is fs21 runs skip tests as a tempest tests11:10
chandankumarsoniya29|rover, please check fs020 job also11:10
ykarelalso in addition fs021 exercise cr repo11:10
chandankumarykarel, cr repo one needs to be fixed also for ussuri and master11:11
soniya29|roverchandankumar, it is also failing in fs2011:11
ykarelchandankumar, yes11:11
chandankumarsoniya29|rover, job link11:11
soniya29|roverchandankumar, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-24hr&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein11:12
soniya29|roverchandankumar, ykarel, overcloud deployment failed11:17
chandankumararxcruz, kopecmartin agenda for today's tempest meeting https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ?view#2020-06-19 feel free to add or update11:18
*** derekh has joined #oooq11:36
*** pojadhav|afk is now known as pojadhav11:37
*** jpena is now known as jpena|lunch11:41
*** rlandy has joined #oooq11:56
*** rlandy is now known as rlandy|ruck11:57
rlandy|ruckchandankumar: hi11:58
chandankumarrlandy|ruck, Hello11:58
rlandy|ruckch: looks like we got dlrn fails in downstream. do we need to make a change there now? https://sf.hosted.upshift.rdu2.redhat.com/logs/84/200084/38/check/tripleo-ci-rhel-8-standalone-rhos-17/fbb97f7/logs/undercloud/home/zuul/dlrn.log11:59
rlandy|ruckchandankumar: ^^11:59
chandankumarrlandy|ruck, I am checking with jpena|lunch on that11:59
rlandy|ruckchandankumar: thanks - I left in your suggested setting in the release file12:00
*** marios has joined #oooq12:01
rlandy|rucksoniya29|rover: hello ...12:04
rlandy|ruckweshay|ruck: hey12:04
rlandy|rucksoniya29|rover: how are things today?12:04
rlandy|ruckhttps://hackmd.io/XcuH2OIVTMiuxyrqSF6ocw12:05
rlandy|ruckfollowing ^^12:05
*** ysandeep|brb is now known as ysandeep12:11
*** rfolco has joined #oooq12:12
rlandy|rucksoniya29|rover: have you had a meeting with weshay|ruck and former ruck/rovers?12:20
rlandy|ruckpojadhav: rfolco: ^^ did you all transfer info?12:21
rfolcotalked to sagi yesterday12:22
pojadhavrlandy|ruck, yup12:28
rlandy|ruckmarios: hey12:31
rlandy|ruckhttps://review.opendev.org/#/c/736816/12:31
rlandy|ruckneed your w+ on that12:31
rlandy|ruckpaying the usual rate12:31
weshay|ruckrlandy|ruck, w/ just soniya29|rover12:31
rlandy|ruckweshay|ruck: np - as long as you guys touched base12:32
mariosrlandy|ruck: o/ looking12:32
chandankumarweshay|ruck, tempest meeting time12:32
mariosrlandy|ruck: is it causing explosions for you? blocker for d/stream maybe or ovb or something?12:32
*** derekh has quit IRC12:33
rlandy|ruckmarios: upstream rather12:33
rlandy|ruckmarios: <EmilienM> rlandy|ruck: yes please approve it - I can't though12:34
mariosrlandy|ruck: ack done12:34
rlandy|ruckmarios: weshay|ruck: thanks both12:34
*** amoralej is now known as amoralej|lunch12:39
rfolcochandankumar, would you review this one pls ? https://review.opendev.org/#/c/733676/12:39
*** jpena|lunch is now known as jpena12:43
*** udesale_ has joined #oooq12:44
*** udesale has quit IRC12:46
*** derekh has joined #oooq12:51
zbrmarios: rlandy|ruck weshay|ruck : https://review.rdoproject.org/r/#/c/28038/ -- please - switch to py3 on promoter13:00
zbrbig in number of files, but not so much in the nature of changes13:00
zbrrfolco: ^13:01
rlandy|ruckzbr: yep - looks good - +213:02
chandankumararxcruz++13:03
arxcruzsoniya29|rover++ :) \13:03
soniya29|roverrlandy|ruck, hello13:07
soniya29|roverrlandy|ruck, had meeting with pojadhav and weshay|ruck today13:08
rlandy|rucksoniya29|rover: ok - as long as you are all set13:08
*** dtantsur is now known as dtantsur|brb13:11
chandankumarrlandy|ruck, weshay|ruck https://review.opendev.org/#/c/736999/13:12
marioszbr: ack adding to queue will check if its still around on next run but looks like it is on final approach13:19
rlandy|ruckweshay|ruck: soniya29|rover: I renamed the new bmc-template on rdocloud13:21
rlandy|ruckwe will have to see if that helps13:21
*** amoralej|lunch is now known as amoralej13:21
rlandy|ruckso stacks created from a couple mins ago should get the new bmc template13:21
zbrwho can give me access to tripleo-infra project on rdo?13:23
zbrpanda: ok, do it but I still need access to it, just in case hell breaks13:24
ykarelrlandy|ruck, new bmc template got tested in https://review.rdoproject.org/r/#/c/28004/ and looks good13:25
ykarelcan u also update in vexx host13:25
rlandy|ruckykarel: sshnaidm|off updated vexx13:26
rlandy|ruckykarel: rdocloud was set as public13:26
ykarelrlandy|ruck, okk good, may be he did earlier so it's already used in job13:26
rlandy|ruckso we needed admin help there13:26
ykarelokk got it13:26
ykareli seen the chat13:26
ykarelearlier13:26
*** ykarel is now known as ykarel|afk13:28
rlandy|ruckmy gosh we are running a lot of jobs13:28
rlandy|ruckno OVB started on rdocloud in a while13:28
*** pojadhav is now known as pojadhav|afk13:30
panda... I thought it would be quicker, but we are upgrading the promtoer server to CentOS7.813:33
pandaso it may not be accessible13:33
pandafor a while13:33
pandait's taking time to make a snapshot ...13:33
*** Goneri has joined #oooq13:36
chandankumarmarios, you are looking at train c8 user stories?13:36
marioschandankumar: not yet go ahead just update if you pick up something put your name on it?13:38
marioschandankumar: we can sync next week i will pickup something different13:38
chandankumarmarios, I will start picking it from monday13:39
marioschandankumar: sounds right13:39
*** TrevorV has joined #oooq13:42
weshay|ruckzbr, 1-113:51
ysandeepchandankumar, I am interested in train c8 user stories too .. Please keep in in loop too o/ I will probably bug you for context and planning13:53
chandankumarysandeep, sure13:58
ysandeepchandankumar, thank you :)13:58
rlandy|ruckysandeep: marios: weshay|ruck: looks like we have a downstream registry problem ... component pipeline turns red from glance14:02
rlandy|ruckonwards14:02
rlandy|rucklooking into it14:02
rlandy|ruckno outage reported14:02
rlandy|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-periodic-rhos-17 - that is pretty though14:03
ysandeeprlandy|ruck, :( not again14:03
* weshay|ruck looks14:04
rlandy|ruckpanda: ^^ we had a clean pass on integration line14:04
mariosrlandy|ruck: ack thanks for heads up14:04
rlandy|ruckwhy didn't we promote to current-tripleo14:04
rlandy|ruckpanda: ^^ still not wired up?14:05
rlandy|ruckcan we promote, pls14:05
weshay|ruckrlandy|ruck, so just the glance component is failing .. this is good14:05
weshay|ruckalso not seen in master14:05
weshay|ruckso will be interesting to see the root cause of the failure14:05
pandarlandy|ruck: upstream or downstream ?14:05
rlandy|ruckpanda: downstream14:05
weshay|ruckpanda, downstream14:05
rlandy|rucknetwork failed as well14:06
rlandy|ruckmaybe a hitch14:06
pandarlandy|ruck: ok, promoting now.14:08
*** tosky has quit IRC14:08
rlandy|ruckpanda: ok - so we're not automated yet14:08
pandarlandy|ruck: no, promoting 8f5b47d2198ea3eefdae76a9d7789f6614:08
pandarlandy|ruck: then I'll automate.14:08
weshay|ruckrequests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8082/v2/tripleoussuri/centos-binary-cinder-api/manifests/62340a910277dfc38f77529a3e80a1d614:09
weshay|ruckanyone see container pulls failing in periodic vexx?14:09
rlandy|ruckpanda: perfect - thanks14:10
rlandy|ruckweshay|ruck: looking14:10
rlandy|ruckvexx was doing well14:10
rlandy|ruckat least ovb was passing14:10
rlandy|ruckipa cleared up14:11
rlandy|ruckwhich is also nice14:11
weshay|ruckrlandy|ruck, soniya29|rover sshnaidm|off FYI.. this may be an easier way to visualize periodic promtion jobs http://dashboard-ci.tripleo.org/d/QrkFP-zGz/upstream-and-rdo-promotions?orgId=114:12
rlandy|ruckwhat's up with cloudops14:12
rlandy|rucknever promotes14:12
rlandy|ruckupstream or downstream14:12
weshay|ruckrlandy|ruck, check consistent against14:12
weshay|ruckthe promoted hash14:12
weshay|rucksame14:12
rlandy|ruckyeah -  know14:12
rlandy|ruckso inactive14:12
weshay|ruckso that's why14:12
rlandy|ruckjust musing why we have such an inactive component14:13
*** ysandeep is now known as ysandeep|away14:14
weshay|ruckrlandy|ruck, aye.. may be pinned.. but low on my hit list atm14:14
rlandy|ruckwe have other things to worry about14:14
weshay|ruckaye14:15
chandankumarweshay|ruck, rlandy|ruck for enabling cr repos https://review.opendev.org/#/c/736999/ and https://review.rdoproject.org/r/#/c/28168/14:15
chandankumarwill take care of that14:15
*** pojadhav|afk is now known as pojadhav14:15
weshay|ruckrlandy|ruck, any updates on ovb generally? I see 2 passes in vexx.. you/sagi fixed bmc I assume14:16
weshay|ruckI still see lots of stacks in rdo14:16
rlandy|ruckweshay|ruck: I updated the bmc-template on rdocloud per sagi's email14:16
rlandy|ruckhe updated the template on vexx14:16
rlandy|ruckthat only deals with introspection errors14:16
weshay|ruckk..14:17
* weshay|ruck pokes around14:17
rlandy|ruckweshay|ruck: so https://review.rdoproject.org/zuul/stream/b30a11358e984c5dbd513ae95fbe55cf?logfile=console.log14:17
rlandy|ruckfor example - started late enough to catch the change14:17
rlandy|ruckit is deploying now14:18
rlandy|ruckweshay|ruck: so the update went in 8:56 my time14:19
rlandy|ruckit 10:19 now14:19
rlandy|ruckso that should give you an idea of when the change would get picked up14:19
weshay|ruckah k14:20
* rlandy|ruck checks vexx14:20
rlandy|ruckweshay|ruck: also - we're still battling stuck stacks ...14:20
rlandy|ruck<rlandy|ruck> I think our current scripts would tackle a stuck server14:21
rlandy|ruck<kforde> rlandy|ruck ack. Might need some DB massaging14:21
rlandy|ruckweshay|ruck: ^^ there we are atm14:21
chandankumarweshay|ruck, rlandy|ruck https://review.opendev.org/#/q/topic:crrepo+(status:open+OR+status:merged)14:23
rlandy|ruckvexx has other issues ... 2020-06-19 14:09:35.689181 | TASK [ovb-manage : Attach instance to provision OVB network]14:27
rlandy|ruckweshay|ruck: vexx looks like diff failures in diff places - afaict14:28
*** ykarel|afk is now known as ykarel14:28
*** dtantsur|brb is now known as dtantsur14:28
rlandy|ruck2020-06-19 05:56:4214:36
rlandy|ruckperiodic-tripleo-ci-rhel-8-bm_envA-3ctlr_1comp-featureset001-baremetal-rhos-1714:36
rlandy|ruck4 hours, 53 minutes14:36
rlandy|ruckSUCCESS14:36
rlandy|ruckYES14:36
rlandy|ruckha14:36
rlandy|ruckysandeep|away: weshay|ruck: ^^ it's official baremetal is running in component pipeline14:38
weshay|rucksaw :)14:39
weshay|ruckrlandy++ ysandeep++14:39
rlandy|ruckweshay|ruck: you're out next week, right?14:41
rlandy|ruckovb on the other hand - no so much14:42
chandankumarweshay|ruck, rlandy|ruck I am leaving early today, see ya on monday14:45
chandankumarhappy weekend14:45
rlandy|ruckchandankumar: have a good weekend14:45
*** chandankumar is now known as raukadah14:45
rlandy|rucksee you monday14:46
*** ccamacho has quit IRC14:48
rlandy|ruck2020-06-19 10:32:12.566739 | primary | /home/zuul/workspace/dlrnapi_venv/bin/activate: line 31: $1: unbound variable14:50
rlandy|ruckno what?14:50
*** skramaja has quit IRC14:59
*** jtomasek has quit IRC15:08
weshay|ruckrlandy|ruck, where is that?15:20
rlandy|ruckweshay|ruck: possible hitch15:22
rlandy|ruckdidn;t see it again15:22
zbrweshay|ruck: talked with panda, promoter is back online after snapshot. we will do the work on tuesday because monday panda is recharging15:24
zbri am not confident to do this alone15:25
weshay|ruckk15:25
rlandy|ruckhttps://logserver.rdoproject.org/30/735930/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/4da5204/job-output.txt15:25
rlandy|ruckyeah - so possible dlnr issues15:25
rlandy|ruckweshay|ruck: so we are seeing OVB failures but not introspection15:26
weshay|ruckya.. in deployment15:26
* weshay|ruck out of 1-1's15:26
rlandy|ruck2 hr 9 min is the last introspection failure I see15:26
rlandy|ruckweshay|ruck: k - we have a REAL issue with promote jobs ...15:27
rlandy|ruckhttps://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-component-security-promote-consistent-to-component-ci-testing/3082b35/job-output.txt15:27
rlandy|ruckupstream and downstream now showing the same issue15:27
rlandy|ruck2020-06-19 15:05:00.530944 | primary | /home/zuul/workspace/dlrnapi_venv/bin/activate: line 31: $1: unbound variable15:27
weshay|ruckvex is failing on this again15:28
weshay|ruckpackages/metalsmith/_provisioner.py\", line 170, in _reserve_node\n    'Failed to reserve a node: %s' % exc)\nmetalsmith.exceptions.ReservationFailed: Failed to reserve a node: Allocation fd30a1f8-8766-42aa-985b-06133c659951 failed: Failed to process allocation fd30a1f8-8766-42aa-985b-06133c659951: none of the requested nodes are available and match the resource class baremetal.\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee15:28
weshay|ruckstdout/stderr for the exact error", "rc": 1}15:28
* rlandy|ruck looks into promote issue15:28
weshay|ruckrdo is failing on15:29
weshay|rucketalsmith.exceptions.DeploymentFailed: Failed to attach VIF 1df97598-93d0-433f-a6b5-cd376fc492da to bare metal node cd111159-1424-40f8-bdd8-9dedf96c7d55: Client Error for url: https://192.168.24.2:13385/v1/nodes/cd111159-1424-40f8-bdd8-9dedf96c7d55/vifs, Node cd111159-1424-40f8-bdd8-9dedf96c7d55 is locked by host undercloud.localdomain, please retry after the current operation is completed.\n", "module_stdout": "", "msg": "MODULE15:29
weshay|ruckFAILURE\nSee stdout/stderr for the exact error", "rc": 1}15:29
rfolcoweshay|ruck, 1x1?15:29
weshay|ruckrfolco, aye15:30
weshay|ruckrlandy|ruck, there is some background on the cix board15:30
weshay|ruckhttps://review.rdoproject.org/r/#/c/28131/15:30
rlandy|ruckweshay|ruck: on which error?15:30
weshay|ruckhttps://trello.com/c/uDzT39AG/1509-cixlp1879472tripleociproa-ovb-overcloud-deploy-fails-sporadically-with-not-enough-free-physical-ports-error15:31
*** Goneri has quit IRC15:32
*** matbu has quit IRC15:32
rlandy|ruckoh - looking in dlrn failure15:32
*** Goneri has joined #oooq15:32
*** matbu has joined #oooq15:32
rlandy|ruckzbr: possible dlrn failure from your change15:33
rlandy|ruck2020-06-19 15:05:00.530829 | primary | + unset VIRTUAL_ENV15:33
rlandy|ruck2020-06-19 15:05:00.530944 | primary | /home/zuul/workspace/dlrnapi_venv/bin/activate: line 31: $1: unbound variable15:33
rlandy|ruckzbr: pls  see https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-component-security-promote-consistent-to-component-ci-testing/3082b35/job-output.txt15:34
*** TrevorV has quit IRC15:35
zbrrlandy|ruck: unrelated, but easy to fix, just move the -u after the activate.15:35
zbractivate does not work with "set -u" on ancient versions of virtualenv15:36
zbrhttps://github.com/pypa/virtualenv/issues/134215:36
zbrsame applies to venv, if I am correct.15:36
* rlandy|ruck tries15:36
zbrrlandy|ruck: read https://github.com/pypa/virtualenv/issues/1029 -- OP and year ;)15:37
zbrguess who fixed it15:37
zbrtry to define VIRTUAL_ENV_DISABLE_PROMPT=115:37
*** TrevorV has joined #oooq15:37
rlandy|ruckzbr: why the issue now?15:37
rlandy|ruckthis code has been running for months15:38
rlandy|ruckthe -u is after the activate15:43
rlandy|ruck    source $WORKSPACE/dlrnapi_venv/bin/activate15:43
rlandy|ruck    pip install -U dlrnapi_client shyaml15:43
rlandy|ruckVIRTUAL_ENV_DISABLE_PROMPT=true is a workaround15:44
rlandy|ruckzbr: ^^ I think this is related to your change due to the timing15:45
rlandy|ruckI can try the workaround15:45
rlandy|ruckbut then people will yell because we used a workaround15:45
*** tosky has joined #oooq15:50
rlandy|ruckzbr: no show on https://review.rdoproject.org/r/2817015:53
rlandy|ruckstill failes15:53
rlandy|ruckfails15:53
*** amoralej is now known as amoralej|off15:56
*** marios is now known as marios|out15:59
rlandy|ruckweshay|ruck: zbr: soniya29|rover: https://review.rdoproject.org/r/#/c/28171/15:59
rlandy|ruckto fix unbound var error above16:00
rlandy|ruckzbr: I'm merging that revert when tests are done16:00
rlandy|ruckpanda: ^^ FYI16:02
*** ykarel is now known as ykarel|away16:03
rlandy|ruckhmmm .. tests still running ... brb16:09
*** rlandy|ruck is now known as rlandy|ruck|brb16:09
*** jmasud has joined #oooq16:11
*** dtantsur is now known as dtantsur|afk16:13
weshay|ruckrlandy|ruck|brb, FYI.. sagi and I discussed not loading vex as much as we are.. https://review.rdoproject.org/r/#/c/28146/16:13
weshay|ruckwe may need to back more jobs off16:14
weshay|ruckI think rdo is still hitting https://bugs.launchpad.net/tripleo/+bug/187947216:15
openstackLaunchpad bug 1879472 in tripleo "OVB overcloud deploy fails sporadically with "not enough free physical ports" error" [Critical,Triaged]16:15
*** udesale_ has quit IRC16:26
weshay|ruckrlandy|ruck|brb, fak https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-train-component-baremetal-promote-consistent-to-component-ci-testing/27f8308/job-output.txt16:36
*** rlandy|ruck|brb is now known as rlandy|ruck16:37
rlandy|ruckweshay|ruck: merging the revert16:38
rlandy|ruckhttps://review.rdoproject.org/r/#/c/28171/16:38
weshay|ruckk16:38
weshay|ruckcomponent pipeline upstream has issues16:38
rlandy|ruckweshay|ruck: k - see the other error16:39
rlandy|ruckweshay|ruck: looking into it16:39
weshay|ruckhttps://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-component-baremetal-promote-consistent-to-component-ci-testing/259ab9d/job-output.txt16:40
weshay|ruckrlandy|ruck, so.. I was accidently looking at train16:40
weshay|ruckwhich .. that's a OK failure16:40
weshay|ruck2020-06-19 16:26:01.012107 | primary | /home/zuul/workspace/dlrnapi_venv/bin/activate: line 31: $1: unbound variable16:40
weshay|ruckis the error in master16:40
rlandy|ruckweshay|ruck: ^^ that error we just merged the revert for16:40
rlandy|ruckweshay|ruck: the train thing is another problem16:41
weshay|ruckrlandy|ruck, train is fine.. centos-8 train is now componentized16:41
weshay|ruckwe can ignore that, that's sprint work16:41
rlandy|ruckweshay|ruck: periodic-tripleo-centos-8-train-component-baremetal-promote-consistent-to-component-ci-testing - ok error?16:41
rlandy|ruckok fantastic16:41
weshay|ruckrlandy|ruck, how does https://review.rdoproject.org/r/#/c/28171/ impact the 2020-06-19 16:26:01.012107 | primary | /home/zuul/workspace/dlrnapi_venv/bin/activate: line 31: $1: unbound variable16:42
weshay|ruckin consistent -> component-ci-testing16:42
rlandy|ruckweshay|ruck: see my ranting above16:42
rlandy|ruckweshay|ruck: testproject https://review.rdoproject.org/r/#/c/24960/16:43
*** Goneri has quit IRC16:43
rlandy|ruckI tried the workaround zbr suggested16:43
rlandy|ruckno dice16:43
rlandy|ruckwe revert16:43
weshay|ruckfine16:43
rlandy|ruckcan't break promotions16:43
weshay|ruckwhich file though?16:43
weshay|ruckin that patch?16:43
rlandy|ruckgetting16:44
rlandy|ruckhttps://review.rdoproject.org/r/#/c/28171/1/ci-scripts/tripleo-upstream/dlrnapi_venv.sh16:44
weshay|ruckk16:45
weshay|ruckthank you16:45
rlandy|ruckweshay|ruck: ok - the overcloud deploy issue on OVB?16:45
weshay|ruckrlandy|ruck, somebody promoted ironic over the compnent-pipeline to current tripleo16:45
rlandy|ruckwe got that sorted or I should investigate?16:45
rlandy|ruckshoot them16:45
weshay|ruckdo we have any idea how that happened?16:45
* rlandy|ruck will check16:45
rlandy|rucklet's look at dlrn16:45
weshay|ruckironic-lib in current-tripleo > than promoted-components16:47
*** dsneddon has joined #oooq16:47
rlandy|ruckpython-ironic-lib-4.3.0-0.20200605155925.df238ba.el8.src.rpm2020-06-05 16:00 101K16:49
rlandy|ruckvs16:49
rlandy|ruckpython-ironic-lib-4.3.0-0.20200605155925.df238ba.el8.src.rpm2020-06-05 16:00 101K16:49
rlandy|ruckweshay|ruck: ^^ am I crazy?16:49
rlandy|rucklooks the same16:49
weshay|ruckhrm16:49
* weshay|ruck looks16:49
rlandy|ruckhttps://trunk.rdoproject.org/centos8-master/component/baremetal/promoted-components/16:50
rlandy|ruckhttps://trunk.rdoproject.org/centos8-master/component/baremetal/current-tripleo/16:50
rlandy|rucksame16:50
weshay|ruckya.. but /me looks at repo16:50
weshay|ruckhrm.. wth.. I'm crazy16:51
rlandy|ruckweshay|ruck: well that would have been at least an explanation16:54
rlandy|ruckweshay|ruck: maybe mirrors - pls post repo link16:54
rlandy|ruckin fact afaik, you can't promote out of order16:56
rlandy|ruckI messed up once and tried to promote the wrong name - dlrn is smart enough to stop me16:56
*** ChanServ has quit IRC16:59
weshay|ruckrlandy|ruck, I'll focus on the master baremetal component promotion after you patch lands17:01
weshay|ruckbmc should be fixed so..17:01
*** derekh has quit IRC17:02
rlandy|ruckweshay|ruck: we'll have to rerun that line, no>17:02
weshay|ruckrlandy|ruck, just triggered it17:02
rlandy|ruckweshay|ruck: k - anything else I should work on here?17:03
rlandy|ruckgate failures /o\17:03
weshay|ruckrlandy|ruck, there's an ipa failure17:04
weshay|ruckrlandy|ruck, probably ipa setup.. saw it yesterday17:04
*** jpena is now known as jpena|off17:04
rlandy|ruck2020-06-19 15:48:35.834989 | primary | TASK [ipa-multinode : install tls dependencies] ********************************17:04
rlandy|ruck2020-06-19 15:48:35.835120 | primary | Friday 19 June 2020  15:48:35 +0000 (0:00:00.584)       0:13:38.422 ***********17:04
rlandy|ruck2020-06-19 15:48:38.518933 | primary | fatal: [undercloud]: FAILED! => {17:04
rlandy|ruck2020-06-19 15:48:38.519005 | primary |     "changed": false,17:04
rlandy|ruck2020-06-19 15:48:38.519033 | primary |     "results": []17:04
rlandy|ruck2020-06-19 15:48:38.519062 | primary | }17:04
rlandy|ruck2020-06-19 15:48:38.519103 | primary |17:04
rlandy|ruck2020-06-19 15:48:38.519147 | primary | MSG:17:04
rlandy|ruck2020-06-19 15:48:38.519181 | primary |17:04
rlandy|ruck2020-06-19 15:48:38.519206 | primary | Failed to download packages: Cannot download Packages/python3-qrcode-core-5.1-12.module_el8.2.0+370+b142e101.noarch.rpm: All mirrors were tried17:05
weshay|ruckstupid mirrors17:05
rlandy|ruckhttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-on-multinode-ipa17:05
rlandy|ruckpretty bad lately17:05
rlandy|ruckmaybe legit17:05
rlandy|ruck2020-06-19 12:21:52.167862 | primary | Checking DNS domain 70.69.158.in-addr.arpa., please wait ...17:05
rlandy|ruck2020-06-19 12:21:52.167878 | primary | DNS zone 70.69.158.in-addr.arpa. already exists in DNS and is handled by server(s): ['ns10.ovh.ca.', 'dns10.ovh.ca.']17:05
rlandy|ruckother failures are different17:06
weshay|ruckrlandy|ruck, I know ade is really needing that patch to land17:06
rlandy|ruckweshay|ruck: eek ... passed on check17:07
weshay|ruckaye17:07
weshay|ruckthe other is pip.. meh17:07
rlandy|ruckweshay|ruck: what's the right thing to do here (recheck and pray)?17:07
weshay|ruckgate is ok.. minus ipa17:07
weshay|ruckrlandy|ruck, /me looks at job history17:07
rlandy|ruckone of the cloud looks like it has a dns issue with that job17:08
weshay|ruckrlandy|ruck, we're going to need to run a elastic-recheck query on it17:08
weshay|ruckto really see17:08
rlandy|ruckack17:09
weshay|ruckrlandy|ruck, let's put up a change to include17:09
weshay|ruck2020-06-19 13:10:54.125489 | primary | The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information17:09
weshay|ruckin the elastic-recheck file17:09
weshay|ruckthat isn't captured atm17:09
weshay|ruckhttps://957cf5a247037a442a69-452dd4b4c84f55a29b71ee3516016f2d.ssl.cf2.rackcdn.com/729465/42/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/b0e585a/job-output.txt17:09
weshay|ruck2020-06-19T13:10:53Z DEBUG The ipa-server-install command failed, exception: DNSZoneAlreadyExists: DNS zone 124.72.198.in-addr.arpa. already exists in DNS and is handled by server(s): ['ns11.iweb-hosting.com.', 'ns12.iweb-hosting.com.']17:10
weshay|ruck^17:10
rlandy|ruckweshay|ruck: pls remind me where the rdoclould elastic recheck file is  - been a while17:10
weshay|ruckI'll get it..17:10
weshay|ruckrlandy|ruck, can you ping ade w/ ^17:10
rlandy|ruckweshay|ruck: so that second error only happens on one cloud17:10
weshay|ruckwhy is that happening?17:10
rlandy|ruckack17:10
* rlandy|ruck checks with ade17:10
weshay|ruckhttps://bugs.launchpad.net/tripleo/+bug/188428717:12
openstackLaunchpad bug 1884287 in tripleo "ipa-server install error: 2020-06-19T13:10:53Z DEBUG The ipa-server-install command failed, exception: DNSZoneAlreadyExists: DNS zone" [Critical,Triaged]17:12
*** marios|out has quit IRC17:14
rlandy|ruckweshay|ruck: updated that LP  = pinging ade17:17
*** ChanServ has joined #oooq17:37
*** tepper.freenode.net sets mode: +o ChanServ17:37
*** Goneri has joined #oooq17:37
*** dsneddon has quit IRC18:11
*** dsneddon has joined #oooq18:12
*** jmasud has quit IRC18:17
*** jmasud has joined #oooq18:21
*** jmasud has quit IRC18:23
*** jmasud has joined #oooq18:28
*** jmasud has quit IRC18:29
*** jmasud has joined #oooq18:33
*** jmasud has quit IRC18:50
*** jmasud has joined #oooq19:00
*** jmasud has quit IRC19:13
*** jmasud has joined #oooq19:21
*** jmasud has quit IRC19:33
*** yolanda has quit IRC19:41
*** jfrancoa has quit IRC19:52
*** sanjayu_ has quit IRC19:53
*** saneax has joined #oooq19:54
*** jfrancoa has joined #oooq20:01
*** TrevorV has quit IRC20:02
weshay|ruckrlandy|ruck, no luck w/ ovb on master or ussir20:04
weshay|ruckussri20:04
rlandy|ruckweshay|ruck: no luck with introspection downstream either20:05
rlandy|ruckweshay|ruck: so baremetal promoted20:05
rlandy|ruckor not yet20:05
weshay|ruckit did not.. we could waive it.. but I don't think it matters.. that ironic-lib patch is in..20:06
weshay|ruckstill a vif issue20:06
weshay|ruckwaiting on master logs20:06
weshay|ruckrlandy|ruck, we should think about turning ovb off in a bunch of repos20:06
weshay|ruckmaybe give the clouds more capacity20:07
rlandy|ruckvif issues is usually cloud related20:07
rlandy|ruckweshay|ruck: vexx only?20:07
weshay|ruckboth rdo and vex20:07
rlandy|ruckugh20:07
* rlandy|ruck looks for sshnaidm|off's patch to remove jobs from vexx 20:08
weshay|ruckperhaps we run ovb in periodic, component, and turn off 3rd party check20:08
rlandy|ruckweshay|ruck: maybe - we are getting no value20:08
rlandy|ruckweshay|ruck: disable all the tests moved from vexx here https://review.rdoproject.org/r/#/c/28146/2/zuul.d/ovb-jobs.yaml?20:09
rlandy|ruckwe are running too many versions/releases20:10
weshay|ruckrlandy|ruck, https://review.opendev.org/73707220:14
weshay|ruckrlandy|ruck, ya.. maybe limit to master?20:14
weshay|ruckfor now.. until we get more capacity20:14
rlandy|ruckweshay|ruck: it is only master20:15
rlandy|ruckwait - what are we talking about??20:15
weshay|ruckrlandy|ruck, oh... sorry ovb.. set to master only.. no branchful jobs20:16
weshay|ruckand only trigger on master20:16
rlandy|ruckweshay|ruck: yeah20:16
rlandy|rucktoo much going on20:16
rlandy|ruckleave the rest of the ovb jobs in periodic20:17
weshay|ruckrlandy|ruck, w/ regards to containers.. spec'ing out upstream steps https://tree.taiga.io/project/tripleo-ci-board/epic/181920:17
weshay|ruckrlandy|ruck, aye20:17
rlandy|ruckweshay|ruck: ^^ ack - looks fine - only trick is20:18
rlandy|ruckturning push off for one20:18
rlandy|ruckand getting push to work on new20:18
rlandy|ruckthat might be take a bit20:18
rlandy|ruckwhere we will either have dup containers20:18
rlandy|ruckor none20:18
weshay|ruckwe can turn the old one off20:18
rlandy|ruckunless we temp namespace it20:18
rlandy|ruckie: if push works well off the bat, fine20:19
weshay|ruckaye20:19
rlandy|ruckif it doesn't leave the old job there20:19
rlandy|ruckweshay|ruck: can we push to temp namespace to test?20:19
rlandy|ruckor I can try push downstream first20:19
rlandy|ruckless impact20:19
rlandy|ruckother than that, tasks look good20:20
weshay|rucksure.. we can poke at it donwstream.. you've done it :)20:23
rlandy|ruckweshay|ruck: I haven't enabled push20:24
rlandy|ruckbut I can do it asap - if that will help upstream20:24
weshay|ruckshould be a number of people looking for new tasks20:25
weshay|rucksandeep, pooja, bhagyashri rfolco etc20:25
weshay|ruckrfolco, added centos-8 train and new containers build user stories to the board20:26
weshay|ruckrfolco, please encourage and esure people pick up the new work20:27
rfolcoweshay|ruck, cool will do20:27
*** jfrancoa has quit IRC20:46
*** rfolco has quit IRC20:54
rlandy|ruckweshay|ruck: where are we with decreasing OVB job runs? should I work on that?20:55
weshay|ruckrlandy|ruck, no one is going to be too worried about .. I think it's a smart idea until we get it working 95% of the time20:56
rlandy|ruckweshay|ruck: ok - checking templates20:56
weshay|ruckrlandy|ruck, we could adjust the template maybe20:56
weshay|ruckya20:56
rlandy|ruckthat's where it's kicked from20:56
* rlandy|ruck gets20:56
rlandy|ruckhttps://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/project-templates.yaml20:57
rlandy|ruckweshay|ruck: want to gm to change that ^^?20:57
weshay|rucksure20:59
weshay|ruckhttps://meet.google.com/pca-svmt-kvn20:59
rlandy|ruckjoining20:59
weshay|ruckrlandy|ruck, https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001&job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset00121:03
rlandy|ruckhttps://review.rdoproject.org/r/28173 Reduce where OVB jobs are run - overloaded clouds21:13
*** saneax has quit IRC21:47
*** weshay|ruck is now known as weshay_pto22:00
*** jmasud has joined #oooq22:47
*** rlandy|ruck has quit IRC22:59
*** tosky has quit IRC23:45
« Thursday, 2020-06-18 Index Saturday, 2020-06-20 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!