Wednesday, 2020-11-04

*** rlandy has quit IRC00:11
*** chem has quit IRC00:52
*** chem has joined #oooq00:54
*** holser has quit IRC01:02
*** jmasud has quit IRC01:12
*** Goneri has quit IRC01:23
*** jmasud has joined #oooq01:43
*** jmasud has quit IRC02:15
*** yolanda has quit IRC02:57
*** skramaja has joined #oooq03:04
*** udesale has joined #oooq03:23
*** jmasud has joined #oooq04:03
*** ykarel|away has joined #oooq04:44
*** ykarel|away is now known as ykarel04:47
*** ykarel has quit IRC05:49
*** ratailor has joined #oooq05:52
*** ykarel has joined #oooq05:54
*** ysandeep|away is now known as ysandeep|ruck06:05
ysandeep|rucksshnaidm|afk, component jobs also starting failing while modifying container images - https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-baremetal-master/aeb0463/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz , I think we should mark https://bugs.launchpad.net/tripleo/+bug/1902190 as promotion blocker06:12
openstackLaunchpad bug 1902190 in tripleo "Master periodic jobs are failing with "msg": "Error running container image prepare: failed"" [Critical,New]06:12
*** marios has joined #oooq06:12
*** saneax has joined #oooq06:38
*** ykarel_ has joined #oooq06:59
*** ykarel has quit IRC07:02
*** ykarel_ has quit IRC07:18
*** jmasud has quit IRC07:25
chandankumarGood Afternoon #oooq07:33
*** jbadiapa has quit IRC07:33
chandankumarsshnaidm|afk: Hello, please have a look at this review https://review.opendev.org/#/c/759985/ it is hitting job freeze issue07:41
sshnaidm|afkchandankumar, welcome back07:57
sshnaidm|afkchandankumar, will look07:57
chandankumarsshnaidm|afk: thanks :-)07:57
*** amoralej|off is now known as amoralej08:02
*** sshnaidm|afk is now known as sshnaidm|rover08:08
*** jbadiapa has joined #oooq08:09
marioschandankumar: o/ see comment @ https://review.opendev.org/#/c/760094/308:11
marioschandankumar: is it OK? nothing runs there08:11
marioschandankumar: not even the content provider?08:12
chandankumarmarios: not ok, thanks for pointing out08:12
chandankumarmarios: need to put the depends on08:12
mariosack going to remove the -208:12
marioschandankumar: also o/ :) hello08:12
chandankumarmarios: thanks :-) \o/08:13
ysandeep|rucksshnaidm|rover, fyi.. architecture is different now.. when comparing current vs old good job logs08:15
ysandeep|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/72f9805/logs/subnode-1/var/log/extra/podman/containers/container-puppet-clustercheck/podman_info.log.txt.gz08:15
ysandeep|ruck~~~08:15
ysandeep|ruck"architecture": "x86_64",08:15
ysandeep|ruck~~~08:15
ysandeep|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/591edba/logs/subnode-1/var/log/extra/podman/containers/container-puppet-haproxy/podman_info.log.txt.gz08:15
ysandeep|ruck~~~08:15
ysandeep|ruck"architecture": "ppc64le",08:15
ysandeep|ruck~~~08:15
sshnaidm|roverysandeep|ruck, yeah, some containers were pushed as ppc yesterday08:15
chandankumararxcruz: hello, I am back, let me know when to discuss about rdo jobs ceph containers08:16
sshnaidm|roverysandeep|ruck, well, it's possible that all containers for master are of ppc now in docker.io08:21
ysandeep|ruckomg, let me cross check other containers to confirm that, i am not aware of context behind pushing ppc container08:23
ysandeep|rucksshnaidm|rover, not all containers.. in same job , keystone container have right arch - https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/591edba/logs/subnode-1/var/log/extra/podman/containers/keystone/podman_info.log.txt.gz08:25
sshnaidm|roverysandeep|ruck, this is built in job afaiu08:26
sshnaidm|rover$ skopeo inspect docker://docker.io/tripleomaster/openstack-keystone:current-tripleo | grep Architecture08:26
sshnaidm|rover    "Architecture": "ppc64le",08:26
ysandeep|ruckcorrect, looks like its updated in job - ImageName": "192.168.24.1:8787/tripleomaster/openstack-keystone:9b773a577e309e3bc25b8cd9085237df-updated-20201104012638",08:27
ysandeep|ruckso, i guess we need to repush containers with correct architecture.08:28
sshnaidm|roveryeah, seems like a glitch in promoter08:29
ysandeep|ruckbig question now - who have access to promoter server :)08:30
sshnaidm|roveryeah08:30
sshnaidm|roverexcept of weshay|ruck08:30
sshnaidm|roverdoes anyone know where is active promoter now? :)08:31
sshnaidm|rovermarios, arxcruz chandankumar ^08:31
chandankumarsshnaidm|rover: for c8?08:31
sshnaidm|roverchandankumar, yes08:31
chandankumarsshnaidm|rover: promoter.rdoproject.org08:32
chandankumarsshnaidm|rover: (38.145.34.5508:32
sshnaidm|roverchandankumar, is it still on rdo?08:32
chandankumarsshnaidm|rover: not sure,08:33
chandankumarbut it is running from very long time, it should be on rdo08:33
sshnaidm|roveryeah, it is08:33
sshnaidm|roverchandankumar, and who does have access? I don't..08:33
chandankumari see some open files in tmux there, may be weshay|ruck or rlandy was editing something08:34
chandankumarsshnaidm|rover: pass me the pub key08:34
sshnaidm|rovergreat08:34
sshnaidm|roverchandankumar, https://github.com/sshnaidm.keys08:34
chandankumarsshnaidm|rover: added08:35
chandankumarsshnaidm|rover: try now, then su - centos and then tmux a08:35
sshnaidm|roverchandankumar, nope08:35
sshnaidm|roveroh, root08:36
mariossshnaidm|rover: sec see pvt i have the ips but i dn't know if still valid08:38
mariossshnaidm|rover: they seem valid the http logs interface is showing on all 308:39
mariossshnaidm|rover: you can see from 'logs' what is running there08:39
sshnaidm|rovermarios, ack08:39
sshnaidm|rovermarios, so many promoters, so little time..08:39
mariosyeah ...08:40
sshnaidm|roverysandeep|ruck, ok, so these images are not in promoter anymore as it seems08:41
sshnaidm|roverysandeep|ruck, most likely we need either a new valid promotion for master or try to build and push images manually08:41
sshnaidm|roverysandeep|ruck, how are things with next promotion? any blockers except of "updates" job?08:42
*** tosky has joined #oooq08:42
ysandeep|rucksshnaidm|rover, let me cross check master pipeline08:42
sshnaidm|roverysandeep|ruck, hmm.. I see all is skipped, ubi8 job failed..08:43
sshnaidm|roverysandeep|ruck, oh, forgot we can take from rdo registry if it's still there08:44
sshnaidm|roverchecking..08:44
sshnaidm|roverysandeep|ruck, ok, in rdo registry they are ppc08:45
*** ykarel has joined #oooq08:48
ysandeep|rucksshnaidm|rover, yes, periodic jobs are failing which pull from rdo registry08:49
*** jpena|off is now known as jpena08:56
*** jfrancoa has joined #oooq09:00
sshnaidm|roverysandeep|ruck, the fact is that we build ppc containers for a few days from 01 Nov09:07
sshnaidm|roverchandankumar, do you know if we have ppc jobs? Or we push ppc containers with promoter?09:25
chandankumarsshnaidm|rover: nope, there might be some jobs running in ci.centos,org jenkins maintained by IBM09:30
chandankumarrfolco knows better09:30
ykarelsshnaidm|rover, chandankumar https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/ jobs build and push to trunk registry09:35
*** derekh has joined #oooq09:35
*** soniya29|afk is now known as soniya2909:37
*** holser has joined #oooq09:52
chandankumarsoniya29: arxcruz kopecmartin akahat added agenda for tempest meeting https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ?view#2020-11-04 please have a look09:57
ysandeep|rucksshnaidm|rover, may be some issue with our portal and it gave different arch - base ubi container10:05
sshnaidm|roverysandeep|ruck, what do you mean?10:07
ysandeep|rucksshnaidm|rover, architecture comes from base ubi container right? what if the base container we used to build other container was itself of wrong architecture10:08
sshnaidm|roverysandeep|ruck, I checked a few and they have amd6410:10
*** dtantsur|afk is now known as dtantsur10:11
sshnaidm|roverysandeep|ruck, I don't think there is a problem with base container: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/9f4e72b/logs/buildah-builds/container-builds/76557c3b-d888-4a58-80f7-f99dde27ecd8/base/base-build.log10:13
sshnaidm|roverthere is x8610:13
sshnaidm|roverysandeep|ruck, I think it's promoter, need to wait for weshay|ruck10:17
ysandeep|rucksshnaidm|rover, ack, I am just wondering how come rdo registry have those container as ppc arch, if its because of promoter.10:23
sshnaidm|roverysandeep|ruck, let's run in testproject and see..10:26
ysandeep|rucksshnaidm|rover, ack, fyi.. this run from today have correct arch(with new ubi image as well).. https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/4a4bb2a/logs/containers-successfully-built.log10:27
ysandeep|ruckskopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/openstack-mariadb:9b773a577e309e3bc25b8cd9085237df | grep -i arch10:29
ysandeep|ruck        "architecture": "x86_64",10:29
ysandeep|ruck    "Architecture": "amd64",10:29
sshnaidm|roverysandeep|ruck, yes10:30
sshnaidm|roverysandeep|ruck, as all other jobs I suppose10:31
ysandeep|rucklet see how this goes, if we don't hit crypto dependencies issues.. I guess we can rebuild the container with current-tripleo hash10:34
sshnaidm|roverysandeep|ruck, this a real mess, some of containers are ppc, some are amd64, from the same hash10:37
mariosbiab10:38
*** marios has quit IRC10:38
ysandeep|rucksshnaidm|rover, really omg10:38
ykarelsshnaidm|rover, as said ppc job can do that as it's pushing10:42
ykareland since that job not building all containers10:42
ykarelnot all images are overridden10:42
sshnaidm|roverykarel, from what I saw these jobs fail for a long time yet before building10:42
ykarelsshnaidm|rover, but u said ppc job pushing since 1st Nov?10:43
ykareli saw some push logs yesterday10:43
ykareldidn't checked from when it started though10:43
ykarelyesterday log https://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/2817/logs/logs/build.log10:44
ysandeep|ruckykarel, when this job tripleo-upstream-containers-build-master-ppc64le triggers?10:47
ykarelysandeep|ruck, every few hours10:47
ykarel15 0,12,18 * * *10:49
ykarelso three times a day10:49
ysandeep|ruckack10:49
ysandeep|ruckwe should have some suffix for ppc container to segregate them from other containers10:50
ykarelyes it used to be like that before switching to tcib based images10:50
ysandeep|ruckokay10:51
ysandeep|ruckchecking older logs of this job to confirm if we created container with existing current-tripleo hash10:52
sshnaidm|roverhttps://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/2817/logs/logs/build.log10:54
sshnaidm|roverysandeep|ruck, ykarel it's where it was pushed ^10:54
ykarelyes10:54
ysandeep|rucksshnaidm|rover, here - 68f97388ca61ef3b5c419b6480308f6a - https://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/2816/logs/logs/containers-successfully-built.log10:55
sshnaidm|roverwhere is this job defined at all10:55
ykarelci-config10:55
ykareljenkins10:55
ykarelci-config/jenkins/jobs/tripleo-upstream.yml10:56
ysandeep|ruckreally interesting finding at last :) haha10:57
ysandeep|ruckykarel++ sshnaidm++10:57
ykarelwould be better to turn push off in those jobs, until it's fixed10:59
sshnaidm|roverI'd disable it at all11:01
ykareljob can be kept running i think, just push off should be enough11:01
ykarelas other team is working on it11:01
ysandeep|rucksshnaidm|rover, also to fix check jobs in the meantime, Can i trigger container build with current-tripleo hash - https://review.rdoproject.org/r/#/c/28442/6/.zuul.yaml ? and then we will push those container from promoter server (pulling from rdo registry)11:02
ykarelppc containers are pushed to docker.io too?11:03
ysandeep|ruckykarel, yes :(11:03
sshnaidm|roverykarel, no, to rdo I think11:03
ykarel:(11:03
sshnaidm|roverthen it's pushed to docker.io by promoter11:03
sshnaidm|roverysandeep|ruck, I'm not sure, the last ubi can break it..11:04
ykarelokk i heard there was a recent promotion11:04
sshnaidm|roveryeah, yesterday11:04
ykarelsshnaidm|rover, you tested containers_base_image: ubi8:8.2  ?11:06
sshnaidm|roverykarel, nope11:06
ykarelack good to test that too11:06
sshnaidm|roverysandeep|ruck, can you take a look? ^ while I'm dealing with ppc11:07
ysandeep|rucksshnaidm|rover, i will do the testing11:07
sshnaidm|rovercool11:07
ysandeep|ruckykarel: do we need complete path - containers_base_image: registry.access.redhat.com/ubi8:8.2 or containers_base_image: ubi8:8.2  will also work?11:14
ykarelfull path not needed11:15
ysandeep|ruckykarel, ack posted https://review.rdoproject.org/r/#/c/24706/8/zuul.yaml11:16
ykarelack let's see11:16
*** udesale_ has joined #oooq11:19
*** udesale has quit IRC11:21
*** marios has joined #oooq11:36
*** dtantsur is now known as dtantsur|brb11:44
holserhey folks12:00
holserquick question12:01
holserWhere do we config yum.conf in quickstart?12:01
sshnaidm|roverholser, I don't think we do12:02
sshnaidm|roverwhy?12:02
holsersshnaidm|rover I have a question One of my job false failed due to http://pastebin.test.redhat.com/915465. So I can add retries but I think it's not a solution. The most proper way is set retries in yum config12:02
holsersshnaidm|rover https://c86609bbfe8918a7ad4c-d2d72e60aa4ca7af838e9dc1dfba5296.ssl.cf5.rackcdn.com/760980/4/check/tripleo-build-containers-ubi-8/0a1c096/logs/build.log12:03
holsersshnaidm|rover I can do http://pastebin.test.redhat.com/91547412:04
holserbut some other package may fail12:04
holserso the best way is to change yum.conf then possible yum_repository or yum module and then add retries if nothing works12:05
*** rh-jelabarre has quit IRC12:06
holsersshnaidm|rover need your help before I push http://pastebin.test.redhat.com/915474 to review12:07
sshnaidm|roverholser, I'm not sure it will help much12:07
sshnaidm|roverCurl error (7): Couldn't connect to server for http://mirror.regionone.limestone.opendev.org:8080/rdo/centos8/component/baremetal/1c/ed/1ced336eb2b630024f064848ead3cecafedc3565_028e9889/repodata/repomd.xml [Failed to connect to mirror.regionone.limestone.opendev.org port 8080: No route to host]12:07
sshnaidm|roverif you failed to download one, it's a problem with proxy/repo, most likely it'll fail next time too12:08
holserso, should I just recheck job?12:08
holser and move on?12:08
sshnaidm|roverholser, what is tripleo_image_serve ?12:09
holsersshnaidm|rover ¯\_(ツ)_/¯12:09
holsersshnaidm|rover patch https://review.opendev.org/#/c/760980/12:09
holsersshnaidm|rover failed zuul check tripleo-build-containers-ubi-812:10
holserhttps://zuul.opendev.org/t/openstack/build/fb2ad790ccce481cbc8f3c9f5717a69f12:11
sshnaidm|roverholser, yeah, it's because infra proxy failed12:11
* holser sighs12:12
sshnaidm|rovernot sure if rettries tripleo_image_serve may help here12:12
holsersshnaidm|rover just recheck?12:12
sshnaidm|roverholser, yep12:12
holsersshnaidm|rover++12:12
holserthanks a lot man12:12
sshnaidm|roverholser, we plan to add build retries, for different reasons, but it may help in such cases as well12:12
holsersshnaidm|rover cool12:13
ysandeep|rucksshnaidm|rover, ykarel it worked periodic-tripleo-ci-build-containers-ubi-8-push https://review.rdoproject.org/zuul/build/04d692ff5aaa4eed86cc1e79379310df : SUCCESS in 56m 37s12:15
ykarel\o/12:16
sshnaidm|roverysandeep|ruck, cool12:17
sshnaidm|roverysandeep|ruck, with a different ubi?12:17
sshnaidm|rovermaybe we can pin ubi image here: https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/templates/Dockerfile_redhat.j2#L112:17
sshnaidm|roverykarel, ysandeep|ruck hmm.. I see job has built for 9b773a577e309e3bc25b8cd9085237df ? But current-tripleo is 68f97388ca61ef3b5c419b6480308f6a : https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md512:19
ykarelpromote_hash was passed tripleo-ci-testing12:19
chandankumarsshnaidm|rover: above dockerfile is used downstream12:19
ykarelin test job12:19
sshnaidm|roverchandankumar, and what is used in upstream ubi jobs?12:20
ykarelsshnaidm|rover, don't think https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/templates/Dockerfile_redhat.j2#L1 is used in that job12:20
*** ratailor has quit IRC12:20
holserykarel can you help me with https://review.opendev.org/#/c/756896/8/tripleo_common/image/builder/buildah.py12:21
ykarelsshnaidm|rover, you can change https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/templates/build.sh.j2#L10612:21
sshnaidm|roverykarel, yeah, I see12:21
ysandeep|rucksshnaidm|rover, i will rerun the job with current-tripleo hash now, now that we now older ubi image works12:21
ykarelholser, looking12:21
sshnaidm|roverysandeep|ruck, oh, great12:21
ykarelholser, you seen https://review.opendev.org/#/c/752931/4 patch?12:22
holserykarel I moved logging.setup to take_actions...12:22
holserykarel yeah12:22
ykarelholser, but in patch i see it's used outside of take_action12:23
holserbut moving logging.setup to take_actions .... is not a solution12:23
ykarelhttps://review.opendev.org/#/c/756896/8/tripleo_common/image/builder/buildah.py@3712:23
sshnaidm|roverysandeep|ruck, ykarel so we are good to pin the "containers_base_image" to ubi:8.2?12:23
chandankumarsshnaidm|rover: https://opendev.org/openstack/tripleo-operator-ansible/src/branch/master/roles/tripleo_container_image_build/README.md#L1512:23
holserit's stevedore issue12:23
chandankumarsshnaidm|rover: pinning will work12:23
holserso Dug should fix it12:23
ysandeep|rucksshnaidm|rover, okay for me to pin12:24
ykarelsshnaidm|rover, yes atleast until centos8.3 is out12:24
ysandeep|ruckhttps://review.rdoproject.org/r/#/c/28442/7/.zuul.yaml to rebuild container with current-tripleo hash12:24
ykarelholser, so until stevedore is fixed, running in take_action is the only option i see12:25
holserykarel I shouldn't put logging.setup in every method I have self.log ... Normally instantiation should be done in __init__12:25
ykarelme don't know if that's bug or not and it will be fixed, but yes for sure it's regression12:25
ykarelholser, yes agree, with __init__ setup it would be better, but due to bug we can't12:27
*** jpena is now known as jpena|lunch12:32
*** rlandy has joined #oooq12:33
holserykarel can we try to try older stevedore12:36
ykarelholser, nope, as for libraries and clients we use upper-constraints pinned version12:38
ykarelif u want to try just for testing we can12:39
marioshttps://review.opendev.org/gitweb?p=openstack/tripleo-ci.git;a=shortlog;h=3bdbb46d49c4dfa376a2c1117643f37b1ed9a54012:40
mariosrlandy: ^^12:40
holserykarel yeah so set cliff<3.412:41
holserjust for testing12:41
holserthen we need to call Dug to help to debug and fix the actual issue12:42
holserwe shouldn't move logging to resolve race ...12:42
ykarelholser, in my bug earlier i added some reporducer steps12:43
ykarelthat can be used12:44
holserykarel thanks12:50
sshnaidm|roverchandankumar, https://review.opendev.org/#/c/761402/12:50
marioshttps://zuul.openstack.org/status12:51
mariosrlandy: chandankumar: akahat: fyi https://review.opendev.org/76118812:54
ysandeep|ruckarxcruz, Hi o/ , regarding https://review.opendev.org/#/c/760914/ - we need to skip neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest.test_qos for master as well. I commented on review. Can we add that in same review or should i propose a new review to skip same test in master.12:54
chandankumarrlandy: https://review.opendev.org/#/c/761402/12:54
arxcruzysandeep|ruck: sure12:55
arxcruzysandeep|ruck: i'll add it12:55
ysandeep|ruckarxcruz: thank you soo much , you++12:55
arxcruzysandeep|ruck: so that master fip-qos thing is okay right ?12:55
ysandeep|ruckarxcruz, yes12:55
sshnaidm|roverysandeep|ruck, product call?12:55
ysandeep|rucksshnaidm|rover, i will join12:56
sshnaidm|roverysandeep|ruck++12:56
mariosStderr: 'error building at STEP "RUN dnf -y install cyrus-sasl-lib cyrus-sasl-plain qpid-dispatch-router qpid-dispatch-tools && dnf clean all && rm -rf /var/cache/dnf": error while running runtime: exit status 1\n'12:56
marioschandankumar: ^^12:56
mariosrlandy: l^^12:56
ysandeep|rucksshnaidm|rover, could you check email from wes - program call reminder .. i added my answers on Question of the week. Does it looks okay to you?12:57
sshnaidm|roverysandeep|ruck, yeah, totally12:57
ysandeep|rucksshnaidm|rover, thanks12:57
*** amoralej is now known as amoralej|lunch12:58
*** dtantsur|brb is now known as dtantsur12:59
*** rh-jelabarre has joined #oooq13:01
chandankumararxcruz: joining in few mins13:02
soniya29chandankumar, weshay|ruck, pojadhav, bhagyashris meeting?13:02
*** frenzy_friday has joined #oooq13:05
holserykarel I created a bug in cliff - https://bugs.launchpad.net/python-cliff/+bug/190287213:05
openstackLaunchpad bug 1902872 in cliff "cliff-3.4.0 creates a race" [Undecided,Confirmed]13:05
holserthe issue is definitely there13:06
weshay|rucksoniya29, checking that I can get out of the 2 other mtgs13:06
holserykarel I think more projects that tripleo are affected13:06
ykarelholser, ack, you noticed in some other project than tripleo?13:07
holserykarel not yet13:07
*** rfolco has joined #oooq13:07
ykarelholser, ack13:08
frenzy_fridayHi, I am trying to deploy osp rel tripleo-ci/CentOS-8/master with quickstart on rhel 8. I got an error at task convert-image : convert image. In the logs I see + dnf install -y python3-tripleoclient13:12
frenzy_fridayWarning: failed loading '/etc/yum.repos.d/delorean.repo', skipping.13:12
frenzy_fridayLast metadata expiration check: 0:00:09 ago on Wed Nov  4 12:53:29 2020.13:12
frenzy_fridayError:13:12
frenzy_friday Problem: conflicting requests13:12
frenzy_friday  - nothing provides validations-common needed by python3-tripleoclient-14.0.1-0.20201102135917.bec4e6f.el8.noarch13:12
frenzy_friday  - nothing provides openstack-ironic-python-agent-builder needed by python3-tripleoclient-14.0.1-0.20201102135917.bec4e6f.el8.noarch13:12
frenzy_friday  - nothing provides openstack-tripleo-validations needed by python3-tripleoclient-14.0.1-0.20201102135917.bec4e6f.el8.noarch13:12
frenzy_friday  - nothing provides python3-ironic-inspector-client >= 1.5.0 needed by python3-tripleoclient-14.0.1-0.20201102135917.bec4e6f.el8.noarch13:12
frenzy_friday Am I missing some repo? I dont see any delorian repo with yum repolist as well13:13
*** skramaja has quit IRC13:14
rlandyfrenzy_friday: if you are deploying OSP - deploy on RHEL - what version of OSP? if you are deploying RDO - then use Centos master13:16
*** holser has quit IRC13:18
chandankumararxcruz: https://review.opendev.org/#/q/topic:bug/1900357+(status:open+OR+status:merged)13:19
weshay|ruckysandeep|ruck, please mention that 16.2 is red13:22
frenzy_fridayrlandy: I tried with train and master release "./quickstart.sh -R tripleo-ci/CentOS-8/master -T all -t all  127.0.0.2"13:22
ykarelfrenzy_friday, seems repos are not setup properly, /etc/yum.repos.d/delorean.repo have corrupted data, can u check what content's it has, and also repo setup log13:23
ysandeep|ruckweshay++13:24
weshay|ruckysandeep|ruck, thank you sir13:25
chandankumararxcruz: kopecmartin soniya29 akahat email forwarded13:25
akahatchandankumar, ack.13:26
weshay|ruckchandankumar, welcome back13:26
chandankumarweshay|ruck: thanks :-)13:26
ysandeep|rucksshnaidm|rover, job ran successfully - periodic-tripleo-ci-build-containers-ubi-8-push https://review.rdoproject.org/zuul/build/727953a32cd34cd0988f931704958920 : SUCCESS in 59m 22s13:36
ysandeep|ruckBefore:-13:37
ysandeep|ruck~~~13:37
ysandeep|ruck[root@localhost ~]# skopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/openstack-haproxy:68f97388ca61ef3b5c419b6480308f6a | grep -i arch13:37
ysandeep|ruck        "architecture": "ppc64le",13:37
ysandeep|ruck    "Architecture": "ppc64le",13:37
ysandeep|ruck~~~13:37
*** Goneri has joined #oooq13:37
ysandeep|ruckAfter:-13:37
ysandeep|ruck~~~13:37
ysandeep|ruck[root@localhost ~]# skopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/openstack-haproxy:68f97388ca61ef3b5c419b6480308f6a | grep -i arch13:37
ysandeep|ruck        "architecture": "x86_64",13:37
ysandeep|ruck    "Architecture": "amd64",13:37
ysandeep|ruck~~~13:37
ysandeep|rucksshnaidm|rover, but current-tripleo hash on registry still have "ppc64le"13:38
ysandeep|ruck~~~13:38
ysandeep|ruck[root@localhost ~]# skopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/openstack-haproxy:current-tripleo | grep -i arch13:38
ysandeep|ruck        "architecture": "ppc64le",13:38
ysandeep|ruck    "Architecture": "ppc64le",13:38
ysandeep|ruck~~~13:38
ysandeep|rucksshnaidm|rover, we can now pull 68f97388ca61ef3b5c419b6480308f6a hashes and push them13:40
*** jpena|lunch is now known as jpena13:40
ysandeep|ruckweshay|ruck, fyi.. you may want to check this one - https://bugs.launchpad.net/tripleo/+bug/190283113:40
openstackLaunchpad bug 1902831 in tripleo "tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing in check/gate/periodic with [ERROR]: Container(s) which finished with wrong return code: ['container-puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql', container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-rabbitmq']" [Critical,Triaged]13:40
ysandeep|rucksshnaidm|rover, or we can trigger main line again now as we have pinned ubi and we can promote new hash if job goes green13:43
ysandeep|ruckweshay|ruck, also is there is some with cix card automation, I added promotion-blocker flag on this bug in morning but it didn't create card yet13:45
ysandeep|ruckis there some* issue13:45
sshnaidm|roverysandeep|ruck, current-tripleo hash is set by promoter, not by job13:47
sshnaidm|roverysandeep|ruck, so it's expected13:47
sshnaidm|roverseems like need to do it manually now..13:48
ysandeep|ruckack13:48
chandankumarsshnaidm|rover: marios and rlandy fixes the job freeze issue13:49
sshnaidm|roverchandankumar, ack13:50
sshnaidm|roverweshay|ruck, I'm going to update containers manually from promoter13:50
*** saneax has quit IRC13:52
*** saneax has joined #oooq13:59
ysandeep|ruckweshay|ruck, sshnaidm|rover cix call timing is very late for me today(21:00 my local time), anyone of you have free slot to cover cix.14:01
sshnaidm|roverysandeep|ruck, yeah, I'll join14:01
weshay|rucksshnaidm|rover, ++14:01
ysandeep|rucksshnaidm|rover, thank you sir :)14:01
*** amoralej|lunch is now known as amoralej14:01
ysandeep|rucksshnaidm++14:01
weshay|rucksshnaidm|rover, ysandeep|ruck can you guys add a quick bug on ubi-8 please14:02
ysandeep|ruckweshay|ruck, we have it already14:02
ysandeep|ruckfetching14:03
weshay|ruckah ya14:03
weshay|rucksee it14:03
weshay|ruckysandeep|ruck, got it14:03
weshay|ruckthanks14:03
ysandeep|ruckweshay|ruck, fyi.. some issue with cix card automation.. cards are not creating by adding promotion-blocker tag14:04
*** saneax has quit IRC14:08
weshay|ruckysandeep|ruck, which bug14:09
ysandeep|ruckweshay|ruck, see this https://bugs.launchpad.net/tripleo/+bug/190283114:09
openstackLaunchpad bug 1902831 in tripleo "tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing in check/gate/periodic with [ERROR]: Container(s) which finished with wrong return code: ['container-puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql', container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-rabbitmq']" [Critical,Triaged]14:09
weshay|ruckysandeep|ruck, /me notes they need to be triaged bugs14:09
* weshay|ruck looks14:09
weshay|ruckysandeep|ruck, target milestone14:09
weshay|ruckysandeep|ruck, there is also a 5 hour delay on cix.. to give us time to fix14:10
ysandeep|ruckweshay|ruck, card didn't create because of missing milestone?14:10
weshay|ruckysandeep|ruck, aye14:11
ysandeep|ruckweshay|ruck, ahh i will wait then, its only 6 hour since we wrote the bug.. will add the milestone14:11
*** ysandeep|ruck is now known as ysandeep|away14:15
weshay|ruckysandeep|away, it's there now https://trello.com/c/4On0RMlU/1729-cixlp1902831tripleociproa-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-is-failing-in-check-gate-periodic-wit14:23
weshay|ruckrfolco, can you please read through https://bugs.launchpad.net/tripleo/+bug/1902831 when you have a sec14:24
openstackLaunchpad bug 1902831 in tripleo "tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing in check/gate/periodic with [ERROR]: Container(s) which finished with wrong return code: ['container-puppet-clustercheck', 'container-puppet-haproxy', 'container-puppet-mysql', container-puppet-memcached', 'container-puppet-keystone', 'container-puppet-rabbitmq']" [Critical,Triaged]14:24
*** holser has joined #oooq14:25
weshay|ruckrfolco, just need you to be aware of that.. nothing else at this time14:26
rfolcoweshay|ruck, ok, reading14:28
weshay|ruckrfolco, you are one of the few that have broad context here.. just want you in the loop14:28
rfolcoweshay|ruck, subscribed myself and mjturek to the bug14:31
rfolcoweshay|ruck, thx14:31
*** TrevorV has joined #oooq14:37
mjturekthanks rfolco checking it now14:57
rfolcomjturek, let me know if you want to chat about this14:57
sshnaidm|rovermjturek, I disabled the job for now15:37
sshnaidm|rovermjturek, will release it soon15:37
sshnaidm|roverdisabled push there: https://review.rdoproject.org/r/#/c/30761/15:37
*** ykarel has quit IRC15:41
*** jmasud has joined #oooq15:48
weshay|ruckrlandy, fyi.. updating all the 16.2 components at jpena's request15:50
weshay|ruckconsistent -> promoted components15:50
rlandyweshay|ruck: thanks15:52
rlandywe have to15:52
rlandyweshay|ruck: pls ping once that is done15:52
rlandywill rerun my test project on container and image builds15:53
weshay|ruckrlandy, it's done15:54
rlandyweshay|ruck, k - rerunning testproject ... which will move promoted-components to tripleo-ci-testing15:55
rlandyhttps://code.engineering.redhat.com/gerrit/#/c/207510/ kicked15:58
rlandyweshay|ruck: ^^ if this works, should I close those CIX cards?16:01
weshay|ruckrlandy, to clear it.. I'd need to see a deployment(s) pass w/o the rpm issues as well16:02
rlandyweshay|ruck: ack ok16:03
sshnaidm|roverrfolco, mjturek we switch to build containers with tcib and seems like in ppc jobs they build without ppc tag and overwrite x86 containers16:05
sshnaidm|roverneed to find a way to pass a ppc tag to them16:05
mjtureksshnaidm|rover: hm. very strange16:05
mjturekokay I'll be looking today16:05
sshnaidm|roverbefore that it was in kolla.conf afaiu16:05
mjturekI believe so, where does it need to be now?16:06
sshnaidm|roverbut we shouldn't use kolla anymore16:06
mjturekI mean, the tag16:06
sshnaidm|rovermjturek, need to pass to build image role, maybe rfolco can help ?16:06
akahatchandankumar, rfolco, hey.. i could see merge conflict here.. and it is not from branch. https://review.opendev.org/#/c/76009716:06
mjtureksorry for the trouble!16:06
akahatis is from depends-on link?16:06
rfolcomjturek, lets look together the code, we'll find out the best place to add tag16:07
mjturekwill do!16:07
rfolcomjturek, today you set this one correct? https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/templates/kolla-build.conf.j2#L516:10
rfolcomjturek, so what happens is that in tcib tag is just version_hash, while in the old kolla you had this tag.yaml...16:12
rfolcomjturek, https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/templates/build.sh.j2#L10516:12
rfolcohow tcib is using tag ^16:13
rfolcoand your tag logic for ppc was16:13
rfolcohttps://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/tag.yaml#L716:13
rfolcomjturek, actually your logic was this https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/main.yaml#L23-L2616:16
*** Goneri has quit IRC16:18
rfolcoand in kolla.conf you had this https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/templates/kolla-build.conf.j2#L916:18
mjturekahhh so need to set version_hash properly16:18
mjturekoh, no we need to add arch_tag to the end of version_hash16:18
rfolcomaybe just change this line and it works ?16:18
rfolcohttps://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/templates/build.sh.j2#L10516:18
mjturekI think you're right16:18
rfolcoyeah, it will be '' for x8616:18
mjturekexactly16:19
mjturekalright - want me to throw the patch up or you?16:19
rfolcomjturek, please go ahead16:20
*** Goneri has joined #oooq16:21
sshnaidm|roverweshay|ruck, rlandy please take a look: https://review.opendev.org/#/q/topic:update_containers+(status:open+OR+status:merged)16:25
weshay|rucksshnaidm|rover, moving to release files because promotion and check have two different release files16:28
rlandysshnaidm|rover: what happens if you have a gating repo that is used with a periodic job? like we often test project like that16:28
sshnaidm|roverweshay|ruck, what do you mean?16:28
rlandyupdate_containers: false16:28
rlandy^^ periodic release files16:28
sshnaidm|roverrlandy, we shouldn't test changes in promotion jobs16:28
rlandysshnaidm|rover: but we do all the time16:29
rlandyto see if the fix will clear the promotion16:29
rlandyas we do with component jobs16:29
sshnaidm|roverrlandy, but you should build containers for promotion job16:29
rlandyI agree with removing the setting from env files16:29
rlandybut not with adding it to release files16:30
sshnaidm|roverrlandy, we must not update containers in promotion jobs16:30
sshnaidm|roverrlandy, if you use it for checking patch, you can pass to is "update_containers: true"16:30
sshnaidm|roverrlandy, or featureset override, or "job.update_containers", whatever16:31
rlandysshnaidm|rover: ^^ that would required  .. wondering if we should add that to overrides16:31
rlandyack - my thought exactly16:31
sshnaidm|roverrlandy, overrides is not the best place..16:31
rlandysshnaidm|rover: I understand your sentiment16:31
rlandybut we need some way to do that periodic change testing16:31
rlandyso I am ok with featureset override16:32
sshnaidm|roverrlandy, you should use specific dlrn hash for that16:32
rlandyideally we should rebuild containers - correct - but we don't always16:32
sshnaidm|roverrlandy, featureset override will override the whole variable, better to use update_containers in job definition16:33
rlandy+2 on https://review.opendev.org/#/c/761285/16:33
rlandysshnaidm|rover: so in release files, we get:16:34
sshnaidm|roverrlandy, so maybe we can set "update_containers: "{{ job.update_containers | default(false) }}"16:34
rlandyupdate_containers: "{{ job.updatd_containers|default(false)|bool }}"16:34
rlandycorrect16:34
rlandyyeah - I can go with that16:35
rlandyweshay|ruck++  ... looking a lot better on 16.2 testproject jobs - container builds still outstanding16:39
weshay|ruckk cool16:40
sshnaidm|roverchandankumar, do you know if we different config for ubi8 for ussuri/train?16:46
sshnaidm|roverchandankumar, pinning ubi8 didn't work for branches..16:46
marios[tripleo] next meeting Tuesday Nov 10 @ 1400 UTC in #tripleo * http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018535.html16:48
sshnaidm|roverchandankumar, nevermind16:48
rlandyweshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/207510/ passed - rerunning rhos-release  job - which will run a standalone16:49
weshay|ruckNICE16:49
weshay|ruckso 16.2 is good16:50
*** udesale_ has quit IRC16:53
rlandyweshay|ruck: it's too close to call atm16:54
rlandywe will count all the zuul votes and let you know16:54
weshay|ruckrlandy, lolz16:54
rlandyweshay|ruck: see I'm learning how to be an american16:55
weshay|ruckyou are too smart, must be an immigrant16:55
mjturekrfolco sorry for the delay, wanted to see if I was missing anything https://review.opendev.org/#/c/761446/1/roles/build-containers/templates/build.sh.j216:55
mjturekI think this SHOULD work16:55
*** marios is now known as marios|out16:59
rfolcomjturek, +117:02
*** marios|out has quit IRC17:06
*** ksambor has quit IRC17:11
*** jfrancoa has quit IRC17:14
*** tosky has quit IRC17:15
*** amoralej is now known as amoralej|off17:30
*** jmasud has quit IRC17:31
*** jmasud has joined #oooq17:32
rlandyweshay|ruck: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/7f4f4c36eb1042ed917ed72832bd19f0?logfile=console.log - 16.2 standalone in tempest17:50
weshay|ruckrlandy, nice17:51
rlandyweshay|ruck: closing those two BZs17:52
weshay|ruck++17:53
rlandyweshay|ruck: anything else pressing to take on today?17:54
weshay|ruckrlandy, for the rest of the sprint.. I think we're just focused on closing out reviews on content-provider17:54
rlandyok17:54
*** frenzy_friday has quit IRC17:56
akahatweshay|ruck, rfolco ready to merge: https://review.opendev.org/#/c/760093/ https://review.opendev.org/#/c/760089/17:57
*** dtantsur is now known as dtantsur|afk17:57
akahathttps://review.opendev.org/#/c/760094/17:58
weshay|ruckakahat, https://review.opendev.org/#/c/759985 is not ready.. a dep for https://review.opendev.org/#/c/760093/17:59
weshay|ruckcan't merge w/o it17:59
akahat:(18:00
weshay|rucksorry.. that's just how it works18:00
*** derekh has quit IRC18:01
*** jpena is now known as jpena|off18:04
akahatweshay|ruck, why this job has https://review.opendev.org/#/c/759821/2..3//COMMIT_MSG depends on ?18:10
weshay|ruckakahat, the minion job was failing18:12
weshay|ruckneeded that change18:13
weshay|ruckakahat, there are 3 chained depends-on there18:13
weshay|ruckif you click through them18:14
akahatoh.. okay..18:14
*** jmasud has quit IRC18:35
*** holser has quit IRC19:32
*** jfrancoa has joined #oooq20:06
*** Goneri has quit IRC20:23
weshay|ruckrlandy, sshnaidm|rover we should chat tmorrow for a few min re: the ubi-8 changes with regards to the dep-pipeline and implications for internal container builfds20:35
rlandyweshay|ruck: ok20:35
*** tosky has joined #oooq20:35
*** Goneri has joined #oooq20:37
weshay|ruckrlandy, sshnaidm|rover https://meet.google.com/ewo-bvop-gaf?authuser=120:43
rlandyone sec20:44
*** Goneri has quit IRC20:46
*** chem has quit IRC20:48
*** chem has joined #oooq20:50
sshnaidm|roverrlandy, https://review.opendev.org/#/c/761463  https://review.opendev.org/76140220:51
sshnaidm|roverrlandy, https://review.opendev.org/#/c/761463/2/roles/container-build/defaults/main.yaml20:53
*** Goneri has joined #oooq21:26
*** jfrancoa has quit IRC21:39
*** Goneri has quit IRC21:51
*** holser has joined #oooq22:08
*** TrevorV has quit IRC22:29
*** rfolco has quit IRC22:30
*** jmasud has joined #oooq22:37
rlandyweshay|ruck: https://review.opendev.org/#/c/761463/ failing - want to wait until it completes?23:06
* rlandy needs to go23:06
rlandywill abandon restore23:06
rlandyor can recheck later23:07
*** jmasud has quit IRC23:07
*** rlandy is now known as rlandy|bbl23:07
*** jmasud has joined #oooq23:31
*** jmasud has quit IRC23:43

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!