Monday, 2020-08-24

*** tosky has quit IRC00:00
*** ratailor has joined #oooq03:32
*** ysandeep|away is now known as ysandeep04:14
*** ykarel|away has joined #oooq04:42
*** ykarel|away is now known as ykarel04:42
ysandeepHello Guys, good morning, do you know what happened with https://bugs.launchpad.net/tripleo/+bug/1879766 ?05:10
openstackLaunchpad bug 1879766 in tripleo "master ovb jobs failing on Destination directory /etc/pki/tls/private does not exist" [Critical,Triaged] - Assigned to chandan kumar (chkumar246)05:10
ysandeepWe are hitting something similiar in downstream - "missing /var/lib/pcsd" and /etc/pki/tls/private", https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-rhos-16.2/49f5774/job-output.txt05:10
ysandeepraukadah, ^^ incase you know05:11
raukadahysandeep: please add same kind of workaround https://review.rdoproject.org/r/#/c/28041/ downstream05:13
ysandeepraukadah, looking05:14
ykarelwe reverted this ^ and moved to upstream jobs05:14
ykarelhttps://review.rdoproject.org/r/#/c/28290/05:14
ykarelso good to check first if upstream patch is sufficient or not for downstream case05:16
raukadahhttps://review.opendev.org/#/c/738469/05:17
raukadahysandeep: ^^ will work05:17
raukadahykarel: thanks!05:17
ysandeepykarel raukadah , thank you, It looks like we are not inheriting from tripleo-ci job in downstream case, will try similiar workaround and see how it goes.05:21
ykarelack if it's not included then similar need to be added05:22
ysandeepack05:22
*** udesale has joined #oooq05:23
akahat|roverraukadah, ping05:28
akahat|roverraukadah, https://review.opendev.org/#/c/744882/ i added this job for master release and it works fine.05:28
akahat|roverraukadah, stable/train is failing.. adding another list item in the section for stable/train will skip the tests ?05:29
akahat|roveri added this patch *05:29
akahat|roverraukadah, https://review.opendev.org/#/c/747619/05:33
*** skramaja has joined #oooq05:37
*** raukadah is now known as chandankumar05:45
chandankumarakahat|rover: commented05:46
akahat|roverchandankumar, i didn't opened bug for it... do i need to?05:47
chandankumarakahat|rover: you can reuse the previous one05:48
chandankumarakahat|rover:  and just update the bug with more info there05:48
*** kopecmartin|pto is now known as kopecmartin05:48
akahat|roverchandankumar, okay05:48
*** jfrancoa has joined #oooq05:50
*** jfrancoa has quit IRC05:50
*** jfrancoa has joined #oooq06:04
*** dtantsur|afk is now known as dtantsur06:35
*** apetrich has joined #oooq06:45
*** sshnaidm|afk is now known as sshnaidm07:03
akahat|roverykarel, ysandeep chandankumar this job is continuously failing.. https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario009-multinode-oooq-container07:09
akahat|roverany what to do with it.. i mean deprecate, fix or just leave it as it is.07:10
ykarelit depends on what that scenario tests07:11
ykareland just leave it as it is shouldn't be an option :)07:11
arxcruz|ruckakahat|rover: morning07:11
ykarelohkk it's an openshift job, no sure it's relevant or not07:12
akahat|roverarxcruz|ruck, morning07:12
chandankumarykarel: it is no longer tested07:12
ykarelchandankumar, even in stable branches?07:12
chandankumarykarel: as per this https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario009-multinode-oooq-container&result=SUCCESS#07:13
chandankumarykarel: it currently runs on stable/train, stein and in periodic07:14
chandankumarakahat|rover: do a git blame, send the removal request in one of the branch07:14
ykarelbut as per ^^, it's been running but ffailing since couple of months07:15
arxcruz|ruckakahat|rover: good news is rocky and steil promote07:15
chandankumarand add as a reviewer who added the job07:15
arxcruz|ruckbad news is master is failing07:15
chandankumarakahat|rover: or even better open a bug and do a cix from there you will get the answer07:16
chandankumarto remove it or not07:16
akahat|roverarxcruz|ruck, that's not good. :|07:17
akahat|roverchandankumar, ykarel opening cix issue would be a better option.07:18
arxcruz|ruckakahat|rover: https://review.opendev.org/#/c/747619 needs a fix, let's do it then merge, there's an ovn issue as well07:19
akahat|roverarxcruz|ruck, looking07:19
arxcruz|ruckakahat|rover: did you open bug for this one? https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/d3f9ad8/job-output.txt07:19
arxcruz|ruckI can do it if you haven't07:19
*** tosky has joined #oooq07:20
akahat|roverarxcruz|ruck, okay. please go ahead..07:21
arxcruz|ruckhttps://bugs.launchpad.net/tripleo/+bug/189268207:27
openstackLaunchpad bug 1892682 in tripleo "Periodic Scenario 010 ovn provider job failing with ovn provider is not enabled" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)07:27
arxcruz|ruckykarel: hey man, do you have any idea what's this error https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/6792995/logs/undercloud/home/zuul/build-err.log.txt.gz ?07:36
ykarelarxcruz|ruck, i see07:37
ykarel2020-08-24 01:39:52.993 | error: Curl error (28): Timeout was reached for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=BaseOS&infra=genclo [Connection timed out after 30000 milliseconds] (http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=BaseOS&infra=genclo).07:37
ykarel2020-08-24 01:39:53.004 | Error: Error downloading packages:07:37
ykarel2020-08-24 01:39:53.005 |   Curl error (28): Timeout was reached for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=BaseOS&infra=genclo [Connection timed out after 30000 milliseconds]07:37
ykarelin https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/6792995/logs/undercloud/home/zuul/build.log.txt.gz07:37
ykarelcentos mirror shouldn't be used in these jobs,07:38
arxcruz|ruckykarel: thanks, didn't notice that07:38
ykarelinfra mirrors should be used07:38
arxcruz|ruckykarel: okay, let me see where this is set07:38
ykarelack07:39
arxcruz|ruckykarel: the repos are configured on diskimage_builder afaik07:41
arxcruz|ruckhmmm07:42
arxcruz|ruckhold on07:42
akahat|roverarxcruz|ruck, chandankumar https://review.opendev.org/#/c/747619/ ready to merge.07:54
*** akahat|rover is now known as akahat|lunch07:58
sshnaidmdo we have timeouts in all jobs now?08:00
sshnaidmarxcruz|ruck, ^08:00
arxcruz|rucksshnaidm: what you mean ?08:00
sshnaidmarxcruz|ruck, I see a lot of jobs finishing with timeout in upstream08:03
sshnaidmarxcruz|ruck, looks like a network problem, most of them fail to download from repos..08:03
*** yolanda has quit IRC08:03
arxcruz|rucksshnaidm: okay, i'm checking, was checking issues in the promoter job08:03
arxcruz|ruckgive me a few08:03
sshnaidmarxcruz|ruck, for example https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1c8/747129/14/check/tripleo-buildimage-overcloud-full-centos-8/1c8f306/build.log08:04
sshnaidmmaybe problem with mirrors..08:04
sshnaidmarxcruz|ruck, there is a problem with build images and build containers jobs, they update a host OS a lot.. do we have a new centos?08:21
sshnaidmykarel, ^08:21
arxcruz|ruck sshnaidm not that i am aware08:21
arxcruz|ruckykarel: Curl error (28): Timeout was reached for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=BaseOS&infra=genclo [Connection timed out after 30000 milliseconds] should not exist in that build-image08:22
arxcruz|ruckall the repos is pointing to rdoproject mirror08:22
arxcruz|ruckand the BaseOS repo as well08:22
*** ysandeep is now known as ysandeep|lunch08:23
ykarelsshnaidm, sorry what problem?08:25
ykareli don't see any recent update in centos repos08:26
ykarelsshnaidm, the above job looks green as per https://zuul.openstack.org/builds?job_name=tripleo-buildimage-overcloud-full-centos-808:34
sshnaidmykarel, yeah, I saw, but latest run doesn't look good: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1c8/747129/14/check/tripleo-buildimage-overcloud-full-centos-8/1c8f306/build.log08:35
sshnaidmit takes a lot of time to install packages, maybe problem with disk..08:35
ykarelsshnaidm, just once ^ or seeing in multiple runs?08:36
arxcruz|ruckchandankumar: akahat|lunch https://review.opendev.org/74764308:37
*** derekh has joined #oooq08:43
*** jbadiapa has joined #oooq08:44
ykarelsshnaidm, so i noticed those jobs are using centos mirrors, which should not be done, instead infra mirrors should be used08:48
ykarelsorry, in this job infra mirrors configured so looks good https://90362f3ffcbc7d10f6d2-5cc3bfcb3bb487674a81482b8b9096bc.ssl.cf2.rackcdn.com/747589/2/check/tripleo-buildimage-overcloud-full-centos-8/e01ebfd/yum.repos.d/CentOS-Base.repo08:50
ykareli can recheck, let me know if it's consistent across multiple jobs08:53
arxcruz|ruckcgoncalves: the skiplist is because these tests are failing, and it's a promotion blocker09:05
arxcruz|ruckcgoncalves: if you know how to fix the error, it is also valid :)09:06
cgoncalvesarxcruz|ruck, remove scenario010-ovn from promotion?09:06
cgoncalvesarxcruz|ruck, OVN provider driver is under the Neutron/OVN team09:06
arxcruz|ruckcgoncalves: i mean, add the tests on the skiplist09:06
cgoncalvesmy stance is: tests should not be disabled in job A just because they are failing in job B09:07
*** ysandeep|lunch is now known as ysandeep09:12
arxcruz|ruckcgoncalves: these are the only tests running on scenario010?09:14
cgoncalvesarxcruz|ruck, there should be more09:15
cgoncalvesarxcruz|ruck, I see a "jobs" parameter. I'm not certain what it does but if I had to guess it allows to specify tests that should be skipped in that jobs list09:16
arxcruz|ruckcgoncalves: not yet :/09:16
arxcruz|rucksshnaidm: do you know what this tags here means? https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L77209:23
sshnaidmarxcruz|ruck, I do09:23
arxcruz|rucki see the difference between periodic and non periodic is this tags09:23
arxcruz|rucksshnaidm: enlighten me then :D09:23
sshnaidmarxcruz|ruck, it means which tagged tasks/roles we run09:24
arxcruz|rucki see09:24
sshnaidmarxcruz|ruck, in periodic you don't need build09:24
arxcruz|rucksshnaidm: but need the other two right ?09:24
sshnaidmarxcruz|ruck, all the rest should be the same09:24
sshnaidmarxcruz|ruck, actually you can include build too, it just won't run09:25
sshnaidmarxcruz|ruck, actually the whole job definition must be same, only release file is different09:25
sshnaidmwhen you inherit it should inherit tags too09:26
zbrsshnaidm: chandankumar: e-r change when you can https://review.opendev.org/#/c/746890/ tx09:27
arxcruz|rucksshnaidm: zbr https://review.opendev.org/#/c/747649/09:38
arxcruz|ruckcgoncalves: can you help me here? where ovn is enabled on scenario010 ?10:01
arxcruz|rucki'm checking the job definition and I can't find it10:01
arxcruz|ruckthe only difference between the periodic and the check job, is that check have ovn in the list of enabled drivers10:02
arxcruz|ruckand periodic doesn't10:02
arxcruz|ruckbut where this is set, I don't know10:02
cgoncalvesarxcruz|ruck, OctaviaEnableDriverAgent: false disables the OVN provider driver10:02
cgoncalvesarxcruz|ruck, https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario010-standalone.yaml#L3610:03
arxcruz|ruckcgoncalves: I see, so, both jobs have this ci/environments/scenario010-standalone.yaml file10:03
arxcruz|ruckas you can see in https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L76710:04
arxcruz|ruckand https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L43210:05
cgoncalvesarxcruz|ruck, correct. those ovn jobs need to set OctaviaEnableDriverAgent: true10:05
arxcruz|ruckcgoncalves: where ?10:05
arxcruz|ruckcgoncalves: because, both are the same10:05
arxcruz|ruckbut periodic doesn't have ovn on the list of enabled, and the check one does10:06
cgoncalvesarxcruz|ruck, override in the job definition10:06
cgoncalvesarxcruz|ruck, if promotion blocker, file a CIX card?10:13
arxcruz|ruckcgoncalves: already did it10:13
cgoncalvesarxcruz|ruck, I didn't get an email on rhos-dev10:13
arxcruz|ruckcgoncalves: it's in new/triage10:14
akahat|luncharxcruz|ruck, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-scenario001-skipped-tempest-ussuri10:27
*** akahat|lunch is now known as akahat|rover10:27
arxcruz|ruckakahat|rover: you don't need to care about this job10:28
arxcruz|ruckit's supposed to fail10:28
akahat|roveroh.. like job failed successfully. :P10:29
arxcruz|ruckyes10:32
arxcruz|rucksshnaidm: have you seen this error before? https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/3de1342/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz10:32
arxcruz|ruckykarel: ^10:32
arxcruz|ruck2020-08-24 02:04:45 |         "[2020/08/24 02:04:18 AM] [WARNING] Error in 'ip route add 0.0.0.0/0 via 10.0.0.1 dev br-ex', restarting br-ex:",10:33
*** udesale has quit IRC10:33
ykarelno ^ is not real issue10:34
ykarelactual is FATAL | Gather podman infos | overcloud-controller-2 | error={"changed": false, "msg": "Unable to gather info for [10:34
ykareli have seen this error some days back10:34
ykareltoo10:34
sshnaidmarxcruz|ruck, yeah, it's not an error10:35
arxcruz|ruckykarel: sshnaidm ack, opening a bug10:36
ykarelarxcruz|ruck, ack http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2020-08-18.log.html#t2020-08-18T12:58:48 for context10:37
sshnaidmarxcruz|ruck, Error: error looking up container \"cceab9790805\": no container with name or ID cceab9790805 found: no such container10:37
chandankumarysandeep: does /tmp issue got resolved?10:37
ysandeepchandankumar, I was not able to replicate the issue in testproject, looks like its a random issue. But i have posted a patch for it https://code.engineering.redhat.com/gerrit/#/c/208473/10:38
arxcruz|ruckysandeep: thanks, i'll check with emilien10:39
ysandeeparxcruz|ruck, ?10:39
arxcruz|ruckysandeep: sorry, it meant to be ykarel10:39
arxcruz|ruck:D10:40
ysandeep:)10:40
*** udesale has joined #oooq10:41
akahat|roverarxcruz|ruck, some time back sshnaidm was talking about timeout issue.. i think it is valid.. but it is not happening every time.. I've seen few jobs which are failed because of this error.10:42
akahat|roverhttps://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-ussuri/44ad824/job-output.txt10:42
akahat|roverhttps://8442fc48a58b3d85474c-27618e467ab9b983fdfb7a89cc8b48ca.ssl.cf1.rackcdn.com/745327/1/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/e702a56/job-output.txt10:43
sshnaidmakahat|rover, omg, does it happen again?10:43
akahat|roverRUN END RESULT_TIMED_OUT:10:43
akahat|roversshnaidm, not seen today.. yet !!10:43
sshnaidmakahat|rover, again on tripleo-inventory : Ensure gather_facts has been run against localhost10:44
akahat|roversshnaidm, that task ment to skip.. https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-ussuri/59424a6/job-output.txt10:46
akahat|roverbut we don't know what happened. :P10:46
sshnaidmyeah, so between it and next task..10:47
akahat|roverarxcruz|ruck, we not worry about all skipped-tempest-ussuri jobs? Or only few jobs?10:53
arxcruz|ruckakahat|rover: exactly10:54
arxcruz|ruckno need to worrk if it's skipped-tempest10:54
akahat|roverarxcruz|ruck, okay.10:54
*** udesale has quit IRC10:55
akahat|roversshnaidm, i found one job which failed today.. https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri/42b5bf5/job-output.txt10:56
sshnaidmakahat|rover, I see.. what worth a bug, I think10:58
zbrarxcruz|ruck: please include link to bug on https://review.opendev.org/#/c/747649/111:08
zbri am happy to approve it once I see an upstream bug related to the pinning11:09
arxcruz|ruckzbr: https://github.com/pre-commit/pre-commit/issues/157811:10
*** pojadhav is now known as pojadhav|afk11:12
*** udesale has joined #oooq11:15
zbrthat is weird, i wonder if running "pip check" before pre-commit would indicate any issues11:16
akahat|roversshnaidm, https://643169a2915789dbd696-0cff35b3d101f5c13616cb96b9f1415f.ssl.cf5.rackcdn.com/746219/6/gate/tripleo-ci-centos-8-undercloud-containers/48539e2/job-output.txt11:17
zbrarxcruz|ruck: and I was right: pre-commit 2.7.1 has requirement virtualenv>=20.0.8, but you have virtualenv 16.7.10.11:17
akahat|roversshnaidm, this is ansible.. but RUN END RESULT_TIMED_OUT is something different.11:17
zbrbasically I was correct to -2 the patch, and the upstream bug is incorrect.11:17
sshnaidmakahat|rover, RUN END RESULT_TIMED_OUT is just a general job error, timeout can be caused by anything11:18
sshnaidmakahat|rover, need to see what was the latest task11:18
arxcruz|ruckzbr: okay, we can't use virtualenv>=20.0.8 right, I remember there were some issues11:18
akahat|roversshnaidm, we are seeing this issues since 21..11:19
akahat|roverchandankumar, have you seen this issues ^^11:20
zbrarxcruz|ruck: try it, there were many releases in between, also pinning down virtualenv withou a comment before it was another mistake11:21
zbrpinning down is ok, but only when previous like contains a link to the bug that required it11:22
zbrAlso good idea to add a "pip check" command before running pre-commit.11:23
arxcruz|ruckzbr: so you have there11:30
arxcruz|ruck 54     virtualenv<20  # known to break things11:30
arxcruz|ruckso...11:30
*** pojadhav|afk is now known as pojadhav11:36
*** ysandeep is now known as ysandeep|brb11:40
*** ysandeep|brb is now known as ysandeep11:54
*** rlandy has joined #oooq12:00
rlandyysandeep: hey - thanks for getting 17 rebuilt12:05
ysandeeprlandy, my pleasure :)12:06
rlandyysandeep: periodic-tripleo-rhel-8-buildimage-overcloud-full-rhos-17 failing12:06
ysandeeprlandy, after reverting back to older puppet version - container build worked12:06
rlandyare you looking at that or should I investigate?12:06
ysandeeprlandy, https://code.engineering.redhat.com/gerrit/#/c/208473/ should take care of this12:06
ysandeeprlandy, i have rerun the job, awaiting vm from nodepool12:07
rlandyysandeep: ok rerun periodic-tripleo-rhel-8-buildimage-overcloud-full-rhos-17 with current tripleo-ci-testing hash?12:07
rlandybecause after this run, I'm going to edit the criteria to get a promotion12:08
rlandyor we manually promote12:08
rlandyysandeep: I am sure we are all tired of so much red on 17 :)12:08
ysandeeprlandy, for periodic-tripleo-rhel-8-buildimage-overcloud-full-rhos-17 i have not played with hash, so yes it will pick default hash12:09
rlandyas long as it's force_periodic true, we're good12:10
rlandywe just need all the jobs reporting with the same hash12:10
ysandeeprlandy, https://code.engineering.redhat.com/gerrit/#/c/201382/23/.zuul.yaml12:10
ysandeepi have added force_periodic to each job12:11
rlandyysandeep: perfect - thanks12:11
rlandychandankumar: sorry - didn;t get a chance to test your change on openstack- downstream because of failing builds - will do that today12:12
rlandyysandeep: and now 16.2 pipeline is starting to run12:13
rlandyso we'll see all results12:13
rlandynice work!12:13
ysandeepteamwork++12:14
ysandeeprlandy, https://code.engineering.redhat.com/gerrit/#/c/201382/ (17 integration pipeline jobs)is currently running, only image build failed so far- i triggered that with https://code.engineering.redhat.com/gerrit/#/c/200295/ , So we just get all results in ~1.5 hour or so, hopefully you don't have to play much with criteria to promote.. things are going green12:18
ysandeeps/just/will/g12:18
rlandyysandeep: whatever fails, we12:19
rlandycan fix afterwards12:19
rlandy17 is soooooooo far behind :(12:20
rlandyI'll make a list of tests we need to fix12:20
ysandeepokay12:20
rlandyysandeep: the ipa test might fail but I have a test fix for that if it does happen12:20
ysandeeprlandy, okay :)12:22
rlandyweshay|ruck: ysandeep: we're going to have to do a manual promotion of 16.2 and 17 this afternoon ....12:28
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/82/201382/23/check/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/ad2d569/logs/containers-successfully-built.log12:28
rlandythe current state of the internal promoter expects the missing neutron openstack-neutron-mlnx-agent12:29
arxcruz|ruckakahat|rover: i'm in cix call12:29
akahat|roverarxcruz|ruck, joining12:30
rlandy- imagename: docker.io/tripleomaster/centos-binary-neutron-server:current-tripleo12:30
rlandyimagename: docker.io/tripleomaster/centos-binary-neutron-mlnx-agent:current-tripleo12:30
rlandybut we are not building it in 1712:31
weshay|ruckrlandy, k12:31
weshay|ruckarxcruz|ruck, akahat|rover otp.. please go to cix12:31
arxcruz|ruckweshay|ruck: i'm there already12:31
akahat|roverweshay|ruck, already in.12:31
*** rfolco has joined #oooq12:31
rlandychecking master12:32
weshay|ruckthanks12:33
rlandyhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/a1d7376/logs/containers-successfully-built.log12:33
rlandy^^ not there either so the RDO promoter must be dealing with that12:33
arxcruz|ruckweshay|ruck: who was supposed to run the cix meeting today ?12:39
*** udesale_ has joined #oooq12:51
*** udesale has quit IRC12:53
ysandeeprlandy, periodic-tripleo-rhel-8-buildimage-overcloud-full-rhos-17 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/4fee49c6b53c4bb598aa0d46672d81d7 : SUCCESS in 38m 37s12:55
ysandeep:)12:55
rlandyysandeep: yep - saw that - we're  ago for promotion12:57
rlandywill talk about it at scrum12:57
*** ratailor has quit IRC12:57
ysandeepyep12:57
rlandywe promote 16.2 and 17 today12:58
rfolcoscrum time https://meet.google.com/oiv-geho-mai?authuser=113:00
rfolcoarxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar13:01
rfolcoping scrum ^13:01
akahat|roverweshay|ruck, https://643169a2915789dbd696-0cff35b3d101f5c13616cb96b9f1415f.ssl.cf5.rackcdn.com/746219/6/gate/tripleo-ci-centos-8-undercloud-containers/48539e2/job-output.txt13:06
*** owalsh has quit IRC13:15
*** owalsh has joined #oooq13:16
arxcruz|ruckweshay|ruck: akahat|rover i'll be back in approx 2 hours13:23
weshay|ruckarxcruz|ruck, aye13:23
akahat|roverarxcruz|ruck, okay13:23
*** Goneri has joined #oooq13:25
*** TrevorV has joined #oooq13:29
*** ykarel is now known as ykarel|away13:30
rfolcoarxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar: if you have jira stories/tasks in kaban board (not in scrum board), please point me to the issue # so I can move to the sprint13:42
rlandyrfolco: ysandeep and I have three ...13:42
rfolcorlandy, gimme the numbers please13:43
rlandyhttps://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=3806&projectKey=TRIPLEOCI&view=detail&selectedIssue=TRIPLEOCI-3613:44
rlandyrfolco: ^^ if that refer to 16.2?13:44
rlandyhttps://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=3806&projectKey=TRIPLEOCI&view=detail&selectedIssue=TRIPLEOCI-513:44
rlandyrfolco: again - if that his 16.2 specific - I don't know13:44
rlandyand then the dependency one13:45
rlandyrfolco: https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=3806&projectKey=TRIPLEOCI&view=detail&selectedIssue=TRIPLEOCI-13013:46
rlandy^^ already added a subtask13:46
rfolcorlandy, no there is another epic for 16.13:51
rfolcohttps://projects.engineering.redhat.com/browse/TRIPLEOCI-6413:51
rlandyysandeep: wow  - only one real failure on 1713:52
rlandyweshay|ruck: ^^13:52
rlandypromote, promote this!!13:52
rlandylet13:52
rlandys clean the component mess13:53
weshay|ruckchandankumar, I'll join late for the interview planning.. need to attend DF call13:55
chandankumarweshay|ruck: ok13:55
*** mjturek has joined #oooq13:58
rfolcorlandy, epic >> story >> sub-task ... please use this hierarchy (do not create sub-tasks for an epic directly)14:00
chandankumararxcruz|ruck: sshnaidm rlandy interview prep meeting time14:01
rlandyrfolco: ok - I didn't create those tasks on the kanban borad - they already existed14:01
rlandyso we create more tasks on the epic14:01
rfolcohmm I am adding a user story so you can create tasks under it14:02
rlandyok - we'll need two14:02
chandankumarweshay|ruck: arxcruz|ruck we have added the interview program schedule on the doc itself, please have a look, weshay|ruck You are joining tomorrow na? Do you want to introduce the interview panle14:18
rfolcorlandy, TRIPLEO-139 now is replaced by TRIPLEO-144, already in scrum board14:18
rfolcorlandy, will delete 139 with your blessing14:18
rlandyrfolco: sure - whatever works14:19
rfolcorlandy, as for 16.2, create new stories here https://projects.engineering.redhat.com/browse/TRIPLEOCI-6414:19
rlandyok14:20
rlandyysandeep: ^^ going to add some stories here14:20
rlandyfor setting up the pipelin14:20
rlandypipeline14:20
rlandyand adding jobs14:20
ysandeeprlandy, thanks! i will add tomorrow14:21
rlandyrfolco: ok - I added TRIPLEOCI-145 on epic https://projects.engineering.redhat.com/browse/TRIPLEOCI-6414:23
rlandyhow does that get to the scrum board?14:23
weshay|ruckchandankumar, you guys done?14:24
rlandyweshay|ruck: ack14:24
* weshay|ruck reviews doc14:24
rfolcorlandy, backlog screen, select the epic and you drag the story into the sprint14:24
rfolcorlandy, here https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=4285&projectKey=TRIPLEOCI&view=planning.nodetail&selectedIssue=TRIPLEOCI-143&epics=visible&selectedEpic=TRIPLEOCI-6414:25
rfolcorlandy, just drag the story up to the sprint, let me know if it worked14:25
*** ykarel|away has quit IRC14:25
rlandyrfolco: ok - I think I did that14:26
rfolcorlandy, yes you did, thanks14:26
rlandyadding sub tasks14:26
weshay|ruckchandankumar, arxcruz|ruck rlandy sshnaidm questions are a little too easy for these candidates.. just FYI.. imho14:27
weshay|ruckconsider adding one difficult question each14:28
rlandyweshay|ruck: sure - let's ask them how to work a JIRA kanban board14:28
rlandyI can't get that right!14:28
weshay|rucklolz14:29
*** ysandeep is now known as ysandeep|dinner14:38
sshnaidmweshay|ruck, if they prove it's too easy for them, then it's fair enough :)14:39
weshay|ruckk14:43
rlandyrfolco: ysandeep|dinner: weshay|ruck: added more stories to the 16.2 epic: https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=4285&projectKey=TRIPLEOCI&view=planning.nodetail&selectedIssue=TRIPLEOCI-149&epics=visible&selectedEpic=TRIPLEOCI-6414:48
rlandyysandeep|dinner: I moved your story to add scenario jobs to the sprint - since you started on that14:48
rlandyweshay|ruck: rfolco: from the original ask, I am moving the OVB job task to the sprint as well14:49
rlandyto be put in the integration line for now14:49
rlandyok?14:49
rfolcorlandy, fine by me14:51
rlandythe alternative is real baremetal14:51
rlandybut I think OVB is easier in this case14:51
chandankumarsee ya tomorrow14:52
*** chandankumar is now known as raukadah14:52
weshay|rucksshnaidm, do we need to look at the upgrade jobs? https://d0e3fd572414115f797b-39524de8c5a1fb89d206195b6f692473.ssl.cf5.rackcdn.com/747129/14/check/tripleo-ci-centos-8-standalone-upgrade/6cb7b1a/logs/undercloud/var/log/tripleo-container-image-prepare.log14:53
weshay|ruckregular jobs look awesome https://1568fca279f072231be5-44bdffb92b22ec3589bdf2facd70188b.ssl.cf5.rackcdn.com/747129/14/check/tripleo-ci-centos-8-standalone/20f0df6/logs/undercloud/var/log/tripleo-container-image-prepare.log14:54
weshay|rucksshnaidm, I would bet the upgrade jobs are deploying from the parent registry and upgrading from docker.io14:55
weshay|ruckwhich may be just fine for now14:55
weshay|ruckdon't know if we want to incur the hit to build both releases in a parent14:56
sshnaidmweshay|ruck, no, I just did for scenarios in first iteration14:56
rlandyysandeep|dinner: if any scenario jobs are ready for the integration line, we need those first before the component lines14:56
weshay|ruck+114:56
sshnaidmweshay|ruck,  we can expand to other jobs and multinode in next iterations14:56
*** udesale__ has joined #oooq14:59
*** udesale_ has quit IRC15:03
*** jmasud_ has quit IRC15:03
*** matbu has quit IRC15:03
*** apetrich has quit IRC15:03
*** zbr has quit IRC15:04
*** zbr has joined #oooq15:05
*** jmasud has joined #oooq15:08
rlandyraukadah: rfolco: testing openstack- container name change downstream https://code.engineering.redhat.com/gerrit/#/c/195884/15:09
rfolcoweshay|ruck, this is what I meant by 'centos-binary' in so many places in ci-config: https://projects.engineering.redhat.com/browse/TRIPLEOCI-13715:11
*** apetrich has joined #oooq15:17
weshay|ruckrfolco, sounds like we just need a param across all of those15:17
weshay|ruckrfolco, /me notes... centos-binary will still be used in other releases15:17
weshay|ruckso .. sounds like config_engine + promoter config change + tests15:17
weshay|rucksound right?15:17
rfolcoweshay|ruck, yes in multiple roles containers-promote, promoter, dlrnapi_promoter, everywhere15:18
*** ysandeep|dinner is now known as ysandeep15:27
*** apetrich has quit IRC15:27
*** weshay|ruck has quit IRC15:31
*** weshay has joined #oooq15:32
ysandeeprlandy, >> ysandeep|dinner: if any scenario jobs are ready for the integration line, we need those first before the component lines15:33
ysandeephow about if i add them in single patch - https://code.engineering.redhat.com/gerrit/#/c/209530/315:33
ysandeepjobs in projects.yaml currently is just for testing*15:33
rlandyysandeep: that's fine if they all are ready to go at once15:33
rlandyif you find you need to hold back on the component ones15:34
rlandylet's not hold the integration ones15:34
ysandeeprlandy, okay15:34
rlandyalso - pls check all jobs only have rhos-16.2-trunk-patches15:34
ysandeeprlandy, Just want to confirm one point - override-checkout: can be inherited right?15:35
rlandyysandeep: ack15:35
ysandeeprlandy, sry that was a query. :)15:36
ysandeepwe can include override-checkout: rhos-16.2-trunk-patches in parent job , and child jobs will have it right?15:36
rlandyysandeep: yes - you are correct,  override-checkout is inherited15:36
ysandeeprlandy, thanks!15:37
rlandyawesome15:37
rlandyysandeep: our card is on the scrum board now15:37
rlandyyour card15:37
rlandyI'll work on the OVB one15:37
ysandeeprlandy, fyi.. sc1/02 is blocked because of missing panko_api container, we need merge of https://review.opendev.org/#/c/741228/15:37
rlandythere is also a multinode one and a real baremetal one15:37
rlandyyep15:38
rlandypls add that to the card15:38
ysandeeprlandy, will pick them up, one i sort these standalone one.. will probably start on baremetal job tomorrow parallely15:39
ysandeeponce*15:39
ysandeeprlandy, taking multinode and baremetal cards15:41
rlandyysandeep: ok - sure15:42
ysandeeprlandy, for now i will test on envA after 17 scheduled run15:42
*** skramaja has quit IRC15:56
arxcruz|ruckweshay: akahat|rover I'm back15:57
rlandyarxcruz|ruck: I think weshay got bumped from irc15:57
rlandyhist |ruck nick is gone15:58
rlandyhis15:58
weshayrlandy, arxcruz|ruck trying to fix my znc16:00
weshayarxcruz|ruck, remind me what we wanted to cover.. OH ovn16:04
weshayarxcruz|ruck, 1 sec16:04
*** weshay has quit IRC16:05
*** weshay has joined #oooq16:05
weshayarxcruz|ruck, k.. ready when you are16:06
arxcruz|ruckweshay: i born ready16:14
ysandeeprlandy, leaving for the day, first scenario job is ready for review - https://code.engineering.redhat.com/gerrit/#/c/209530/ , test runs are green. Sry you may have to wait for 17 check job to run compelety before merging it.16:15
arxcruz|ruckweshay: do you mave a meeting id ?16:15
rfolcorlandy, raukadah can I take over https://review.opendev.org/#/c/747259 again and make the prefix based on ci_branch ?16:16
ysandeepsee you guys tomorrow o/16:16
*** ysandeep is now known as ysandeep|away16:16
rlandyysandeep|away: bye - have a good night16:18
*** dtantsur is now known as dtantsur|afk16:19
weshayarxcruz|ruck, ah sorry16:20
weshaystarting16:20
weshayarxcruz|ruck, https://meet.google.com/ocr-htpe-sew?authuser=116:21
*** tosky has quit IRC16:49
weshayarxcruz|ruck, CentOS-8-x86_64-GenericCloud.qcow216:49
weshayrfolco, what's your dib test patch w/ fs00216:57
rfolcoweshay, https://review.rdoproject.org/r/#/c/28948/16:58
rfolcoweshay, downstream is this one https://code.engineering.redhat.com/gerrit/#/c/208524/16:59
rfolcoweshay, and buildimage jobs testproject: https://review.rdoproject.org/r/#/c/28113/17:00
*** jbadiapa has quit IRC17:00
*** derekh has quit IRC17:02
weshaythanks17:04
weshayrlandy, want help looking17:07
rlandyweshay: waht are we talking about now?17:08
*** udesale__ has quit IRC17:10
rlandyweshay: why was https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-train in criteria17:11
rlandyit never passed17:11
weshayI think it was meant to be a blocker.. when pooja opened it17:11
weshayrlandy, I also think 16.1 was blocked on ipa integration issues at the time17:12
rlandyok17:12
weshayrlandy, want help?17:34
rlandyweshay: yeah - let's work on promotions if you have time now17:36
weshayk.. I do17:37
weshayrlandy,  https://meet.google.com/mwg-vouu-azo?authuser=117:37
*** jtomasek has joined #oooq18:06
*** jtomasek has quit IRC18:27
*** jtomasek has joined #oooq18:37
*** jtomasek has quit IRC18:48
rlandyweshay: turning dlrn-promoter service off in internal19:01
weshayrlandy, +119:02
rfolcorlandy, can you please explain how this could be right ? https://review.opendev.org/#/c/747259/5/roles/build-containers/defaults/main.yaml19:08
rfolco {% if osp_branch is defined -%}19:08
rfolco  openstack19:08
rfolco  {%- else -%}19:08
rfolco  {{ ansible_distribution|lower }}-binary19:08
rfolco  {%- endif -%}19:08
rlandyrfolco: chandan did that so we could test on internal19:09
rlandywhat's wrong with it?19:09
rfolcothis is the upstream patch, has nothing about downstream19:09
weshayright.. so openstack- will now be both downstream and upstream19:10
rlandytripleo-ci is only upstream19:10
weshayrfolco, originally it was an internal convention only19:10
rlandyit's branchless19:10
weshayrlandy, rfolco ... we're speaking about when to name the containers openstack- vs. centos-binary19:11
weshayopenstack- is now coming to upstream as well rlandy19:11
weshayafaik19:11
rlandyweshay: true19:11
rlandybut it's a bigger problem to solve in upstream19:11
rfolcoyes, we cannot change that to osp_branch condition, it will break upstream19:12
weshayso instead of if osp_branch19:12
weshaywe want to deliniate by release19:12
weshayrfolco, if defined?19:12
weshaywould resolve as false19:12
rfolcoweshay, rlandy let me propose something that may apply for both downstream and upstream19:14
rlandyrfolco: testing that patch here: https://code.engineering.redhat.com/gerrit/#/c/195884/19:15
rlandywoohoo 17 has started to clear20:12
weshaynice20:14
weshaytripleo train promoted20:14
rlandygetting there20:15
rlandyweshay: what should I do about the centos 8 CR line?20:15
rlandymerge as is w/o the image build job?20:15
* rlandy thinks so20:15
rlandyfixing reviews to merge20:16
weshayrlandy, ha.. was just about to ping you20:20
weshayre: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/ci-scripts/dlrnapi_promoter/config/RedHat-8/component/rhos-16.2.yaml20:20
weshayrlandy, ya.. let's merge it20:20
weshayI want to get an improvement visible there in monitoring20:20
weshayrlandy, got the time since last build working20:21
weshayhttp://dashboard-ci.tripleo.org/d/XIedIk4Mk/component-pipeline-train?orgId=1&fullscreen&panelId=46520:21
rlandylooking20:21
weshayit's the date on consistent20:21
weshayper componet or for the integrated build in another view.. works w/ either20:21
weshaylet me know if that doesn't make sense.. I need this to be understandable by those not versed20:22
rlandywhat's the diff between time since last build?20:22
rlandyie: it never changed?20:22
rlandyso nothing to promote?20:22
rlandylooks fine to me20:22
weshayya.. if there isn't a new build.. there's nothing to promote20:23
rlandyyeah - good20:24
rlandyshocker - I have a linter failure20:24
rlandyhttps://logserver.rdoproject.org/40/29040/2/check/tox-linters/6d2e2bc/job-output.txt20:27
rlandyweshay: ^^ that a genuine error?20:27
*** jfrancoa has quit IRC20:28
rlandyhttps://logserver.rdoproject.org/40/29040/2/check/tox-linters/6d2e2bc/tox/linters-1.log?20:29
rlandy2020-08-24 20:22:33.933209 | rdo-centos-8 | An unexpected error has occurred: AssertionError: BUG: expected environment for python to be healthy() immediately after install, please open an issue describing your environment20:30
rlandy2020-08-24 20:22:33.933620 | rdo-centos-8 | Check the log at /home/zuul/.cache/pre-commit/pre-commit.log20:30
rlandy2020-08-24 20:22:33.964668 | rdo-centos-8 | ERROR: InvocationError for command /home/zuul/src/review.rdoproject.org/rdo-jobs/.tox/linters/bin/python -m pre_commit run -a (exited with code 1)20:30
* weshay looks20:37
weshayrlandy, hrm.. looks like that job should migrate to centos8 or py320:39
rlandyweshay: weird - this is the first review to catch that20:39
rlandylocalhost | Inventory Hostname: rdo-centos-820:40
rlandyit is running on centos 820:41
weshayrlandy, let's see what happens here https://review.rdoproject.org/r/2905820:41
rlandyk - thanks20:41
rlandylinters, what did I ever do to you????20:42
weshayrlandy, i think that worked20:45
rlandyweshay: awesome .. will rebase on that patch20:46
* weshay updated commit-msg20:46
weshayok.. off to play Teacher, gym teacher that is20:49
weshay0/20:49
weshayrlandy, ^20:49
rlandyweshay: thanks for all the help today20:50
weshaypsshh.. thank you!20:50
rlandyopenstack-dependencies-centos8cr21:10
rlandypipeline added21:10
*** TrevorV has quit IRC21:29
*** rlandy has quit IRC23:51

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!