Wednesday, 2022-02-02

« Tuesday, 2022-02-01 Index Thursday, 2022-02-03 »
*** dviroel is now known as dviroel|out00:59
rlandy|ruckysandeep|out: had to recheck 826521 - running last test - hopefully should hit gates soon01:23
*** rlandy|ruck is now known as rlandy|out01:23
*** ysandeep|out is now known as ysandeep03:33
ysandeeprlandy|out, thanks for keeping an eye on 82652103:33
*** marios is now known as marios|ruck06:05
marios|ruck\o/06:05
ysandeepo/ good morning marios|ruck 06:05
marios|rucko/06:05
*** amoralej|off is now known as amoralej07:22
*** ysandeep is now known as ysandeep|lunch09:39
amoralejmarios|ruck, i'm checking the new gcc issue10:00
amoralejbtw, marios|ruck you know if we are using AFS mirrors to create images?10:01
amoraleji think that new issue is similar to the ones we have been having with baseos and appstream coming from different mirrors10:01
marios|ruckamoralej: for c9 we are using the release files so we have the usual ${NODEPOOL_CENTOS_MIRROR}10:02
marios|ruckamoralej: for c8 we aren't so it is the issue with duplicate repos possibly 10:02
marios|ruck(aren't yet)10:02
amoraleji see10:02
amoralej2022-02-02 02:51:05.001 | + /home/zuul/dib_build.cR1VCPh5/hooks/extra-data.d/99-yum-repo-conf:main:25        :   sudo cp -L -f /etc/yum.repos.d/quickstart-centos-appstreams.repo /home/zuul/dib_build.cR1VCPh5/mnt/etc/yum.repos.d10:02
amoralej2022-02-02 02:51:05.035 | + /home/zuul/dib_build.cR1VCPh5/hooks/extra-data.d/99-yum-repo-conf:main:25        :   sudo cp -L -f /etc/yum.repos.d/quickstart-centos-base.repo /home/zuul/dib_build.cR1VCPh5/mnt/etc/yum.repos.d10:03
amoralejthose should have NODEPOOL_CENTOS_MIRROR?10:03
amoralejyes10:03
amoralejwe have duplicated mirrors10:03
amoraleji see it10:03
amoralej2022-02-02 02:51:13.289 | CentOS Stream 8 - AppStream                      17 MB/s |  19 MB     00:0110:03
amoralej2022-02-02 02:51:28.286 | quickstart-centos-appstreams                     55 MB/s |  19 MB     00:0010:03
amoralejand10:05
amoralejappstream: using metadata from Tue Feb  1 18:18:39 2022.10:05
amoralejquickstart-centos-appstreams: using metadata from Fri Jan 28 17:05:04 2022.10:05
amoralejso, i'd say that's the issue10:05
amoralejsaid that, centos should have a better solution to avoid this10:05
marios|ruckamoralej: thanks.. easiest path is if we get the release files used for c8 otherwise we'll have to hack/remove that duplicate temporarily 10:06
marios|ruckakahat: how is the testing going there (c8 release files ^^)10:07
amoraleji commented in the LP marios|ruck 10:14
marios|ruckthanks amoralej 10:14
akahatmarios|ruck, y'day i've got comments from dviroel|out, i fixed it.. waiting for results.10:30
*** ysandeep|lunch is now known as ysandeep10:53
akahatarxcruz, ysandeep o/ Soniya and Chandan is on PTO. is it okay if we cancel today's tempest call? 11:00
ysandeepakahat, sure11:00
arxcruzyes11:00
ysandeepakahat, I remember you created a playbook to deploy standalone, Do you still have that handy?11:00
akahatysandeep, yeah.. one min11:01
akahatysandeep, https://amolkahat.github.io/deploy-standalone-tripleo-using-tripleo-operator-ansible.html11:01
ysandeepakahat, thanks!11:02
ysandeeparxcruz: ^^ 11:02
*** rlandy|out is now known as rlandy|ruck11:03
marios|ruckthanks akahat 11:05
*** dviroel|out is now known as dviroel11:15
marios|ruckakahat: o/ hey can you join us if you have a few mins 11:19
marios|ruckhttps://meet.google.com/vzm-nrah-qqf11:19
marios|ruckakahat: ^^ 11:19
marios|ruckakahat: re the c8 builds and release files11:19
marios|ruckakahat: to see how far we are on that 11:20
ysandeeprlandy|ruck, https://review.rdoproject.org/r/c/rdo-jobs/+/3771211:22
rlandy|ruckpojadhav: hey - welcome back ... there are still some master c8 jobs in repos other than ci11:24
rlandy|ruckcan you look for those11:24
rlandy|ruckand propose patches to switch them to c9?11:24
rlandy|ruckmarios|ruck: https://review.rdoproject.org/r/c/rdo-jobs/+/3847511:27
akahatmarios|ruck, o/ sorry.. 11:34
akahati just saw your msg.. are you still in call?11:34
rlandy|ruckakahat: yep11:35
marios|ruck rlandy|ruck: https://review.rdoproject.org/zuul/status#38910 11:35
akahatmarios|ruck, https://zuul.opendev.org/t/openstack/status#82368811:37
marios|ruckhttps://review.rdoproject.org/zuul/buildset/2972c6887ecb4165b6d8f89c32af7372 rlandy|ruck 11:46
ysandeeprlandy|ruck: image build in pipelines were added here(merged yesterday): https://review.rdoproject.org/r/c/rdo-jobs/+/38634/6/zuul.d/integration-pipeline-wallaby-centos9.yaml 11:50
* akahat back in few hours11:58
rlandy|ruckrcastillo: multinode-ipa on c9 failure is still ongoing ... can you rework your patch to get c8 votin and gating and leave c9 non-voting12:00
marios|ruckrcastillo: (this patch https://review.opendev.org/c/openstack/tripleo-ci/+/826124 ) re rlandy|ruck ping ^^^ 12:00
rlandy|ruckat least we will get some coverage back12:00
rlandy|ruckarxcruz: master c9 containers in rdo registry are gone12:07
rlandy|ruckarxcruz: can you copy back the hash that is current-tripleo?12:07
pojadhavrlandy|ruck, thank you :)12:17
pojadhavrlandy|ruck, yeah will look into them12:17
*** pojadhav is now known as pojadhav|brb12:19
arxcruzrlandy|ruck in a sec 12:20
rlandy|ruckpojadhav|brb: thanks - also, downstream is slowly coming back - pls help ysandeep get those component lines back into shape12:20
rlandy|ruckbhagyashris: hey - what was the issue with the train c7 promotion?12:23
rlandy|ruckbhagyashris: looks like you have it working again12:23
bhagyashrisnot sure i just enabled the c7 train promotion on old promoter server and it promoted 12:24
rlandy|ruckbhagyashris: magic :)12:25
bhagyashrisrlandy|ruck, not sure what was the issue, because on the production server it was not showing the clear error, it was just stuck at some point and failed 12:25
bhagyashrisyeah :)12:25
rlandy|ruckbhagyashris: the line is running again today12:25
rlandy|rucklet's see if it promotes12:25
bhagyashrislets try the next promotion on production server12:26
bhagyashrisrlandy|ruck, ok12:26
rlandy|ruckbhagyashris: if you want, you can move it back now12:26
rlandy|ruckso we check current promotion12:26
rlandy|ruckthanks12:26
bhagyashrisrlandy|ruck, sure12:26
bhagyashrisrlandy|ruck, we can merge this patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38796 asap12:29
rlandy|ruckbhagyashris; thasnk - w+'ed it12:29
rlandy|ruckthanks12:29
rlandy|ruckbhagyashris: also asked pojadhav|brb to look into c8 master jobs that still exist in other repos ( not ours) before we tear down c8 ... maybe you can work together there12:30
rlandy|ruckso that we have a clear path on c8 removal12:31
bhagyashrisrlandy|ruck, sure 12:31
rlandy|rucklastly marios|ruck had one comment on your wallaby c9 patch12:31
bhagyashrislooking 12:31
rlandy|ruckbhagyashris: ^^ can you address so we can merge up your work12:31
rlandy|ruckthanks so much!!12:31
rlandy|ruckarxcruz: thank you12:31
rlandy|ruckdasm|off: when you are in, some OVB work for you12:32
ysandeeprcastillo, pojadhav|brb bhagyashris hey o/ thank you for showing interest in setting downstream line, Can we meet today to discuss the next steps and basic kt to bring you up to speed.12:38
*** pojadhav|brb is now known as pojadhav12:43
pojadhavrlandy|ruck, ack12:43
rlandy|ruckpojadhav: where are the wedding photos???12:43
rlandy|ruckwe want to see12:43
*** dasm|off is now known as dasm|rover12:44
dasm|rovero/12:44
dasm|roverrlandy|ruck: \o12:44
bhagyashrisysandeep, sure12:44
rlandy|ruckdasm|rover: happy early morning to you12:44
rlandy|ruck4:44am12:44
pojadhavrlandy|ruck, will upload on google drive and share link with you ;)12:44
rlandy|ruckpojadhav++12:44
rlandy|ruckdasm|rover: we have the program call in 15 mins  - marios|ruck will present this week - thought it was waaaay too early for you :)12:45
rlandy|ruckwill chat with you about some ovb experiments afterwards12:46
dasm|roverrlandy|ruck: it's early. but i'm trying to be online ~5am. it's 4:45am now :)12:46
rlandy|ruckdasm|rover: that's crazy12:46
dasm|roverto have some overlapping time with others12:46
dasm|roverrlandy|ruck: it depends ;) but we can talk about pros and cons after program call :P12:46
pojadhavysandeep, sure12:47
rlandy|ruckdasm|rover: ok - so akahat is showing some good results with the patch you worked on12:47
rlandy|ruckwhich is great12:47
rlandy|ruckwe need that in asap12:47
dasm|rovernice!12:47
ysandeeppojadhav, bhagyashris thanks, I am not sure if rcastillo is already in, I have send an invite for after ~45 mins - we can prepone/postpone based on your availability.12:47
* dviroel sends coffee to dasm|rover 12:48
pojadhavysandeep, ack12:48
* dasm|rover tries coffee from dviroel... nice12:48
rlandy|ruckmarios|ruck; so close on train - we are one test out periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train12:48
rlandy|ruckdviroel: make sure it's black - at 5am12:49
dasm|roverthe coffee is pitch black. like outside right now12:50
rlandy|ruckdasm|rover: akahat: marios|ruck: ysandeep: https://review.rdoproject.org/r/c/testproject/+/38921 - looking pretty good there12:52
rlandy|ruckone out on ussuri12:52
marios|ruckrlandy|ruck: :/ well i expect by tomorrow latest we'll get a promotion for T 12:52
rlandy|ruck requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://trunk.rdoproject.org/centos7-ussuri/tripleo-ci-testing/delorean.repo12:52
rlandy|ruckmarios|ruck: ack - probably one tempest failure12:53
rlandy|ruckakahat: ^^ error there?12:53
rlandy|ruckcentos-8 ussuri12:53
rlandy|ruckrest looks good12:54
* rlandy|ruck needs to do quick code review12:54
bhagyashrisrlandy|ruck, marios|ruck https://review.rdoproject.org/r/c/rdo-jobs/+/38829 Done12:55
*** amoralej is now known as amoralej|lunch12:56
marios|ruckbhagyashris: ack thanks will check again after call 13:01
rlandy|ruckmarios|ruck: we got a response on openstack-discuss re: opstools13:09
rlandy|ruck"The centos-release-opstools package has been updated to provide the13:10
rlandy|ruckcentos 8 stream repositories."13:10
marios|ruckrlandy|ruck: when you get a chance https://review.opendev.org/c/openstack/tripleo-quickstart/+/827476 13:13
marios|ruckrlandy|ruck: ack saw that and i saw that collectd is now in the repo so we might switch back soon ?13:13
rlandy|ruckvoted13:18
rlandy|ruckmarios|ruck: no rush - let's check with amoralej|lunch if he thinks we can switch safely now13:19
rlandy|ruckhttps://logserver.rdoproject.org/10/38910/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/5c81578/logs/undercloud/var/log/tempest/stestr_results.html.gz13:19
rlandy|ruckone train tempest failure13:19
rcastilloo/13:22
rlandy|ruckrcastillo: morning13:25
rcastillorlandy|ruck: morning13:26
rcastillorlandy|ruck: marios|ruck dasm|rover updated the standalone-on-multinode-ipa patch: https://review.opendev.org/c/openstack/tripleo-ci/+/82612413:27
dasm|roverack13:27
dasm|roverrcastillo: would it be better to revert this one? https://review.opendev.org/c/openstack/tripleo-ci/+/82415713:29
dasm|roverkinda revert revert revert? ;)13:29
rlandy|ruckrcastillo: hey - c9 needs to be removed from gate as well13:29
rlandy|ruckso non-voting and out of gate13:29
rlandy|ruckand let's get c8 coverage in13:29
rlandy|rucksorry about the back and forth on this13:29
rcastillodasm|rover: my patch has a few more fixes13:30
rcastillorlandy|ruck: coverage?13:30
dasm|roverrcastillo: ack13:30
rlandy|ruckvoting and gating13:30
rcastilloright13:31
rcastillorlandy|ruck: dasm|rover done13:32
dasm|roverrcastillo++13:32
dasm|roverrcastillo: just one question. why are you removing tls_ipa_files for ipa jobs? line 961?13:33
rlandy|ruckrcastillo: thanks - voted13:33
dasm|rovermultinode-ipa13:33
rlandy|ruckmarios|ruck++ 13:34
rlandy|ruckon shout out :)13:34
marios|ruckyeah amoralej|lunch++13:34
rlandy|ruckmarios|ruck: dasm|rover: wrt train fs035 ...13:38
rlandy|rucklooks like diff tempest tests failing 13:38
rlandy|rucksetUpClass (tempest.api.compute.images.test_list_image_filters13:38
rlandy|ruckListImageFiltersTestJSON)13:38
rlandy|ruckvs13:38
rlandy|rucktest_create_image_specify_multibyte_character_image_name[id-3b7c6fe4-dfe7-477c-9243-b06359db51e6]13:38
rlandy|rucktempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON13:38
marios|ruckrlandy|ruck: k, i didn't dig there was going to after the prog call 13:39
rlandy|ruckwe have a skiplist on similar13:39
rlandy|ruckgetting13:39
marios|ruckrlandy|ruck: we have a skiplist ... ah you mean the job istn' in it yet13:39
* dviroel coffee13:39
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/195901413:39
rlandy|ruckmarios|ruck: ^^ similar for victoria13:39
rlandy|ruck https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria13:39
rlandy|ruckmarios|ruck: after program call, if you agree, we can add train to the skiplist on ^^13:40
* dasm|rover checking train fs03513:40
rlandy|ruckdasm|rover: ^^ actually you can take that13:40
rlandy|ruckwe can give it one more run13:41
rlandy|ruckbut looks similar13:41
dasm|roverrlandy|ruck: i'm seeing it failing just once with on this tempest test. previous job runs were failing due to https://review.opendev.org/c/openstack/tripleo-quickstart/+/82714313:46
dasm|roverrlandy|ruck: i'm thinking about giving it another run. if it's gonna fail -- then we could add it.13:47
rlandy|ruckdasm|rover: sure - go for it13:48
rlandy|ruckpls use the specifed hash13:48
rlandy|ruckruck/rover tool will give it to you13:48
dasm|roverrunning it here: https://review.rdoproject.org/r/c/testproject/+/3893913:49
marios|ruckthanks dasm|rover 13:49
dasm|roverrlandy|ruck: oh. i'm running without the hash. lemme check that and i'll try again13:50
dasm|roverrestarted13:51
*** amoralej|lunch is now known as amoralej14:01
amoralejmarios|ruck, rlandy|ruck don't revert the opstools yet14:02
marios|ruckamoralej: ack wasn't planning on digging there until at least tomorrow 14:03
rlandy|ruckdasm|rover: hey  - will today be the first CIX call you attend?14:05
dasm|roverrlandy|ruck: i believe so14:05
rlandy|ruckdasm|rover: ok - so maybe let's let marios|ruck report this time14:06
rlandy|ruckand you can listen and report on your next shift?14:06
dasm|roversure.14:06
marios|rucko/14:06
rlandy|ruckjust so you get the flow of the call14:06
rlandy|ruckmarios|ruck: dasm|rover: ok  thank you14:07
marios|ruckysandeep: sshnaidm: please check simple one when you have 5 https://review.opendev.org/c/openstack/tripleo-quickstart/+/827476 14:07
ysandeepmarios|ruck, ack14:08
sshnaidmrcastillo frenzyfriday arxcruz jm1 and all who is interested in working on openstack ansible collections, we have an IRC channel for collection discussions, please feel free to join it:  #openstack-ansible-sig14:09
dviroeldasm|rover: https://review.rdoproject.org/r/c/rdo-jobs/+/38332 - why? i was using this change :P14:10
rcastillosshnaidm: ack14:10
dasm|roverdviroel: oh! i didn't notice you hijacked it :P14:10
dasm|roverdviroel: done :)14:10
dviroeldasm|rover: lol - np14:10
dasm|roverdviroel: there is a merge conflict so i just abandoned it, thinking it's already merged :P14:11
dviroeldasm|rover: don´t worry, i will rebase it 14:11
pojadhavrlandy|ruck, here are some wedding snaps :-) : https://drive.google.com/drive/u/1/folders/1FL0trGN55As9CfP8slxiQ2_vye-iWrdE14:18
rlandy|ruckarxcruz: hey ... just checking back on container copy14:19
rlandy|ruckit that possible14:19
rlandy|ruckotherwise we will need to promote14:19
rlandy|ruckpojadhav: wow - looks amazing, you'll have to tell us more at 'happy friday' meeting14:23
rlandy|ruckso we get the explanation :)14:23
pojadhavrlandy|ruck, sure ;)14:23
pojadhavrlandy|ruck, thank you !!!14:23
rlandy|ruckakahat: hey - before you are EoD, pls touch base with dasm|rover, marios|ruck and me about what is still needed to get https://review.opendev.org/c/openstack/tripleo-ci/+/823688/ to merge14:31
rlandy|ruckiiuc, 14:31
rlandy|ruckhttps://logserver.rdoproject.org/21/38921/1/check/periodic-tripleo-centos-8-ussuri-containers-build-push/160d587/job-output.txt14:31
dasm|roverakahat: your https://review.opendev.org/c/openstack/tripleo-ci/+/823688 looks pretty good14:31
ysandeeprcastillo: hey o/ I have send an invite for today for 17 line discussion, proposed time works for you?14:31
rlandy|ruckjust to fix ^^14:31
rcastilloysandeep: ack, time is perfect14:31
ysandeeprcastillo: great, thanks for confirming14:32
arxcruzrlandy|ruck copied already 14:32
arxcruzrlandy|ruck 82142cdf4dd511032b48543abbec9229 hash 14:32
ysandeeprlandy|ruck: when you have time, eyes on this one too https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/308443 14:33
rlandy|ruckarxcruz++ thank you14:33
akahatrlandy|ruck, dasm|rover all the jobs are passed. will get +1 from zuul soon.14:33
rlandy|ruckakahat: ack - just asking about one failure on testproject14:34
dasm|roverakahat: nice!14:34
rlandy|ruckhttps://review.rdoproject.org/r/c/testproject/+/3892114:34
rlandy|ruckperiodic-tripleo-centos-8-ussuri-containers-build-pushRETRY_LIMIT 7m 09s (non-voting14:34
marios|ruckrlandy|ruck: ack 14:35
marios|ruckthank you arxcruz 14:35
akahatrepos are setup wrong somewhere: https://trunk.rdoproject.org/centos7-ussuri/tripleo-ci-testing/delorean.repo14:36
rlandy|ruckhttps://logserver.rdoproject.org/21/38921/1/check/periodic-tripleo-centos-8-train-containers-build-push/2bd8a03/etc/yum.repos.d/14:38
rlandy|ruckakahat: ack .. also ^^ does not look right14:38
rlandy|ruckso image build is good14:38
rlandy|ruckidk about container build14:38
rlandy|ruckhttps://logserver.rdoproject.org/21/38921/1/check/periodic-tripleo-centos-8-train-containers-build-push/2bd8a03/etc/yum.repos.d/tripleo-centos-opstools.repo14:39
rlandy|ruckwrong opstools14:39
rlandy|ruckif using the release file14:39
rlandy|ruckdasm|rover: pls see if you agree14:39
dasm|roverchecking14:40
dasm|roverit all looks wrong wrt repos.14:40
akahatrlandy|ruck, it's because hardcoded value: https://opendev.org/openstack/tripleo-repos/src/branch/master/plugins/module_utils/tripleo_repos/main.py#L7314:40
dasm|roverakahat++ nice catch14:40
rlandy|ruckakahat: right but with your patch14:41
rlandy|ruckit should use the release file14:41
rlandy|rucknot those defined repos14:41
rlandy|ruckakahat: so I know it's defined there14:41
rlandy|ruckbut if you compare to image builds14:41
rlandy|ruckyou will see different file14:41
rlandy|ruckhttps://logserver.rdoproject.org/21/38921/1/check/periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-train/2c515cd/yum.repos.d/quickstart-centos-ceph-nautilus.repo14:42
rlandy|ruckakahat: ^^14:42
akahatrlandy|ruck, it might be because we have train on 7 and 8... 14:42
rlandy|ruckfrom your testproject14:42
rlandy|ruckperiodic-tripleo-centos-8-train-containers-build-push14:42
rlandy|ruck^^ that job is 814:42
rlandy|ruckand also on ussuri14:43
*** ysandeep is now known as ysandeep|dinner14:43
akahatrlandy|ruck, okay. got your point. 14:44
marios|ruckakahat: please check i think you missed one there https://review.opendev.org/c/openstack/tripleo-ci/+/823688/23#message-3d446dd60405e17cfcc1661fd6a5a3281923d9f2 14:51
marios|ruckakahat: otherwise i think OK 14:51
akahatmarios|ruck, yeah.. you might be right. but i've also made this change: https://review.opendev.org/c/openstack/tripleo-ci/+/823688/23/playbooks/tripleo-buildimages/pre.yaml#2314:57
akahatfor buildimages14:57
marios|ruckakahat: exactlyh 14:58
marios|ruckakahat: but this one is for containers 14:58
marios|ruckakahat: needs to run 2x so it picks up the mirrors 14:58
marios|ruckupstream ones look ok eg https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7eb/823688/21/check/tripleo-ci-centos-8-content-provider/7eb4a86/logs/undercloud/etc/yum.repos.d/quickstart-centos-base.repo baseurl=http://mirror.mtl01.iweb.opendev.org/centos/8-stream/BaseOS/x86_64/os/14:58
rlandy|ruckysandeep|dinner: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/308443 looks good14:59
rlandy|ruck+2'ed it15:00
rlandy|ruckyou can merge when ready15:00
akahatmarios|ruck, https://review.opendev.org/c/openstack/tripleo-ci/+/823688/20/playbooks/tripleo-buildimages/run-v3.yaml#5=15:00
akahati missed it.15:00
marios|ruckack dviroel++ thanks akahat 15:04
rlandy|ruckdasm|rover: hey - let's chat about ovb work15:04
dasm|roverk15:05
rlandy|ruckhttps://meet.google.com/rqb-ymqi-gog?pli=1&authuser=015:05
rlandy|ruckdviroel: hey - we need to promote c9 wallaby - two failing ovb jobs in eth current line - can you check pls15:07
dviroelrlandy|ruck: will take a look 15:13
dviroelrlandy|ruck: we didn't add ovb jobs to criteria yet, we merged yesterday these jobs15:15
dviroelrlandy|ruck: will look these failures15:16
dviroelone is the node_provision issue, will check if we missed neutron backports or is just a promotion needed15:19
dviroelslaweq: hi, do we need a backport of your fix https://review.opendev.org/c/openstack/neutron/+/826449? or is an issue that only hits master?15:22
dviroelslaweq: saw failures in wallaby too (https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-wallaby/b3660be/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)15:23
slaweqI don't think it's needed, patch which caused that issue is only in master15:23
*** ysandeep|dinner is now known as ysandeep15:24
slaweqIn wallaby it has to be different issue15:24
rlandy|ruckdviroel: thank you15:30
dviroelslaweq: ack, so, in that case, will search for more logs - ty15:31
dviroelslaweq: btw, when you have some time, can you pls take a look on this one? https://bugs.launchpad.net/tripleo/+bug/195958215:32
slaweqdviroel I really doubt I will have time for it soon. Tomorrow I'm last day at work before short holidays and I will come back on 16.0215:34
slaweqand for tomorrow I have few things which I need to finish already15:34
dviroelslaweq: is anyone that can help us with this one?15:35
slaweqYou can ask mlavalle, ralonsoh or someone else from networking dfg15:36
dviroelslaweq: thanks - enjoy your time off 15:37
rlandy|ruckslaweq: thanks for all your help - enjoy your vacation15:42
rlandy|ruckdviroel: ralonsoh should be able to help15:42
rlandy|ruckdviroel: try #rhos-dev on internal15:43
slaweqrlandy|ruck thx15:43
rlandy|ruckfor ralonsoh15:43
jpodivinrlandy|ruck: Hi. If you have time I've wanted to ask if there something we can do for the promotion criteria patch to get merged https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3686115:45
jpodivinit's nothing critical though. 15:46
rlandy|ruckjpodivin: rebased - w+'ed again15:46
rlandy|rucklet's see if that helps15:46
rlandy|rucklet me know if you don't get zuul +2 on that after runs15:46
jpodivinrlandy|ruck: thank you very much. I will. 15:47
rlandy|ruckdviroel: guess your name preference lost15:49
jpodivinrlandy|ruck: oh boy, eh, this is embarrassing, but better that than causing prod chain issues. Something seems to be going on with the periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-component-master-validation it might be better if you removed that w+ 15:50
jpodivinrlandy|ruck: sorry I realized I can do it myself. 15:53
dasm|roverhttps://civs1.civs.us/cgi-bin/results.pl?id=E_693bd8bbe63ca52f15:53
rlandy|ruckno worries15:53
dasm|rover*Z will be called 'https://civs1.civs.us/cgi-bin/results.pl?id=E_693bd8bbe63ca52f (Zen)' (Pending legal approval)*15:54
rlandy|ruckdviroel: should we get ovb out of criteria to promote wallaby c9?15:57
rlandy|ruckwe have no images promoted to current-tripleo15:57
marios|ruckrlandy|ruck: didn't arx copy work ok? 15:57
rlandy|ruckhence all the failures on ovb check15:57
rlandy|ruckmarios|ruck: that was container images on master15:58
marios|ruckah sorry drive by irc ;)15:58
rlandy|ruckthis is overcloud images on wallaby15:58
marios|ruckdidn't read the whole thing15:58
rlandy|ruckno worries15:58
rlandy|rucka lot flying around15:58
dviroelrlandy|ruck: we didn't add them yet (ovb ones)15:59
rlandy|ruck<dviroel> rlandy|ruck: we didn't add ovb jobs to criteria yet, we merged yesterday these jobs15:59
rlandy|ruckoh nvm15:59
rlandy|ruckdidn't read not added15:59
dviroelyeah :)15:59
rlandy|ruckcarry on15:59
marios|ruck:D15:59
marios|ruckkeep calm, and carry on15:59
rlandy|ruckhttps://logserver.rdoproject.org/21/38921/1/check/periodic-tripleo-centos-8-ussuri-containers-build-push/7db9364/etc/yum.repos.d/16:00
rlandy|ruckakahat: ^^ still out of the water here16:01
rlandy|ruckequests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://trunk.rdoproject.org/centos7-ussuri/tripleo-ci-testing/delorean.repo16:01
frenzyfridayrlandy|ruck, Hey 0/ 1:1 time :)16:04
rlandy|ruckfrenzyfriday: just finishing off cix call16:04
frenzyfridaycool, np :)16:05
rlandy|ruckwill be there in a few16:05
*** pojadhav is now known as pojadhav|afk16:09
*** dviroel is now known as dviroel|lunch16:09
marios|ruckrlandy|ruck: getting ready to go in few dasm|rover o/ 16:26
marios|ruckneed something before i do? 16:26
marios|ruckwill continue chase train tomorrow 16:26
marios|ruckand cix etc 16:26
dasm|rovermarios|ruck: anything in fire atm? is there anything i need to pay attention to?16:26
rlandy|ruckmarios|ruck: hey - just what you want to pass on16:26
marios|ruckdasm|rover: would be great to get progress on https://review.opendev.org/c/openstack/tripleo-ci/+/823688 16:27
rlandy|ruckmarios|ruck: dasm|rover: tomorrow we need to decide on promoting c9 master16:27
rlandy|ruckthe rest we will work on promoting todat16:27
rlandy|rucktoday16:27
marios|ruckwe really need that soon (there is testproject maybe you can carry on workingon that after akahat is out soon 16:27
marios|ruckdasm|rover: ^^ 16:27
rlandy|ruckand patch above16:27
dasm|roverack16:27
marios|ruckdasm|rover: one more day \o/ 16:27
marios|ruck;)16:27
dasm|roverLD16:27
dasm|rover:D16:27
*** ysandeep is now known as ysandeep|out16:28
* ysandeep|out out, see you tomorrow o/16:29
rlandy|ruckhttps://review.rdoproject.org/r/c/testproject/+/3892116:30
rlandy|ruckdasm|rover: akahat: ^^ rekicked testproject16:30
dasm|roverack16:32
rlandy|ruckysandeep|out: bhagyashris: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3866716:34
rlandy|ruckwhy are these jobs removed?16:34
rlandy|ruckthey run on internal16:35
rlandy|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-ussuri16:35
rlandy|ruckreverting that patch16:35
rlandy|ruckatm there are node issues 16:35
rlandy|ruckso they failed16:35
rlandy|ruckbut they need to be in criteria16:35
rlandy|ruckdasm: looks like there is still an issue with ussuri on https://review.opendev.org/c/openstack/tripleo-ci/+/823688 on containers build on testproject16:45
rlandy|ruckcan you look into that?16:45
dasm|roversure16:45
*** marios|ruck is now known as marios|out16:48
rlandy|ruckdasm|rover: https://review.opendev.org/c/openstack/tripleo-ci/+/823688/24/playbooks/tripleo-buildcontainers/pre.yaml#2316:50
rlandy|ruckshould that be less than 8?16:50
dasm|roverhmm.. looks good, i think16:52
rlandy|ruckdasm|rover: right?16:52
rlandy|ruckdasm|rover: I think that should be changed16:53
rlandy|ruckdasm|rover: https://meet.google.com/gqi-tiwr-tsp?pli=1&authuser=016:53
dasm|roverrlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/823688/24/playbooks/tripleo-buildcontainers/pre.yaml#2316:56
dasm|roverrlandy|ruck: i *think* this also requires change: https://review.opendev.org/c/openstack/tripleo-ci/+/823688/24/playbooks/tripleo-buildcontainers/run.yaml#2817:04
dasm|roverfrom "<=" to "<"17:04
dasm|roverotherwise, we're gonna rebuild c8 containers without vars17:04
rlandy|ruckyep17:05
rlandy|ruckshould follow the 9 pattern17:05
dasm|roverk17:06
rlandy|rucklunch - brb17:06
dasm|roverk17:06
*** dviroel|lunch is now known as dviroel17:12
dasm|roverrlandy|ruck: fyi, c8-train-ovb job passed: https://review.rdoproject.org/r/c/testproject/+/38939 so i rekicked openstack-periodic-integration-stable417:24
dasm|roverhttps://review.rdoproject.org/zuul/status/change/000000000000000000000000000000000000000017:24
rlandy|ruckwoohoo17:25
rlandy|ruckdasm|rover: can you rekick victoria pls17:26
dasm|roverintegration pipeline?17:26
dasm|roverprepping to do so in a moment: openstack-periodic-integration-stable217:27
dasm|roverstarted victoria as well17:28
rlandy|ruckty17:30
dasm|roverbrb17:58
rlandy|rucklooking back18:04
rlandy|ruckakahat: you still around?18:04
rlandy|ruckneed help with patch?18:04
akahatrlandy|ruck, yes. i'm here.18:05
rlandy|ruckakahat: late for you18:05
rlandy|ruckakahat: let me know if you want to hand off that patch - dasm|rover and I are here to help18:07
akahatrlandy|ruck, i'll be out in few mins. yeah please..18:08
akahatthank you.18:08
*** amoralej is now known as amoralej|off18:08
akahatTestproject is still same.18:08
rlandy|ruckok - so you have a new change or we should work on what is there now?18:08
rlandy|ruckok - let me look18:08
rlandy|ruckah18:14
rlandy|ruckI get it18:14
rlandy|ruckit's the nodeste18:14
* rlandy|ruck fixes18:14
dasm|roverback18:19
rlandy|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-8-ussuri-containers-build-push18:25
rlandy|ruckold job definition18:25
rlandy|ruckdasm|rover: rekicked testproject18:26
rlandy|ruckwith new jobs18:26
dasm|roverack18:26
dviroelrlandy|ruck: so, running again failed c9-ovb jobs, each one failed with a different issue, lets see if is consistent18:28
rlandy|ruckhttps://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=68d34e4fa3629c0fba323543939e22a818:33
rlandy|ruckhmm ... looking why this did not promote18:33
rlandy|ruck68d34e4fa3629c0fba323543939e22a8 same hash18:36
rlandy|ruckugh18:36
rlandy|ruckwe need to go in and tag that dir current-tripleo18:36
rlandy|ruckhttps://logserver.rdoproject.org/21/38921/2/check/periodic-tripleo-ci-build-containers-ubi-8-push-wallaby/595282d/job-output.txt18:48
rlandy|ruck^6 real failures18:48
rlandy|ruck2022-02-02 18:41:34.249366 | TASK [repo-setup : include_tasks]18:48
rlandy|ruck2022-02-02 18:41:34.440735 | primary |   "reason": "no action detected in task. This often indicates a misspelled module name, or incorrect module path.\n\nThe error appears to be in '/var/lib/zuul/builds/595282d1ebf843258e086356d14aeae9/untrusted/project_7/opendev.org/openstack/tripleo-quickstart/roles/repo-setup/tasks/yum-config-generic.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax 18:48
rlandy|ruckproblem.\n\nThe offending line appears to be:\n\n---\n- name: Configure 'generic' repos using yum-config module\n  ^ here\n"18:48
dasm|roveri see this:18:50
dasm|roverassert ('aodh-api' in 'REPOSITORY  TAG     IMAGE ID  CREATED  SIZE\n')18:50
dasm|roverhttps://logserver.rdoproject.org/21/38921/2/check/periodic-tripleo-ci-build-containers-ubi-8-push-wallaby/595282d/logs/report.html18:51
dasm|roverproc.stdout returns info containing headers18:51
rlandy|ruckdviroel: hey - can you help out here?18:57
rlandy|ruckhttps://logserver.rdoproject.org/21/38921/2/check/periodic-tripleo-ci-build-containers-ubi-8-push-wallaby/595282d/job-output.txt18:57
dviroelrlandy|ruck: i was just looking at it18:57
rlandy|ruckhttps://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/tasks/yum-config-generic.yml18:57
rlandy|ruckhttps://review.rdoproject.org/zuul/stream/da0b0dc1b48b48cda72139588492fcaf?logfile=console.log18:58
rlandy|ruckworked fine here18:58
rlandy|ruck    use_yum_config_module: false19:01
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/823688/25/roles/tripleo-build-jobs-repos/tasks/main.yaml#3519:01
rlandy|ruckwhy?19:01
dasm|roverdviroel mentioned to use that19:01
dviroelrlandy|ruck: dasm|rover: build images and build containers doesn't support collections19:02
rlandy|ruckso then we have a problem19:02
dviroelmarios fixes that with "tripleo-get-hash-set-fact-no-collections.yml"19:02
dviroelwe are still working on yum-config without collections19:02
dviroelrlandy|ruck: we just need to disable yum_config for now then19:02
dviroelrlandy|ruck: and we can enabled it again once yum-config without collections merge19:03
rlandy|ruckdviroel: only fails on wallaby19:04
dviroelis it breaking only on amols patch?19:04
rlandy|ruckyep19:04
dviroelrlandy|ruck: yes, we didn't merge for other branches yet19:04
rlandy|rucksee https://review.rdoproject.org/zuul/status19:04
rlandy|rucktestproject19:04
rlandy|ruckdviroel: k - so disable in the role itself?19:05
* rlandy|ruck gets19:05
rlandy|ruckhttps://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/tasks/setup_repos_new.yml#L2019:05
rlandy|ruck?19:05
dviroelrlandy|ruck: in the release file19:05
dviroelfor wallaby19:05
* rlandy|ruck looks19:05
dviroel"use_yum_config_module: true"19:06
dviroelwe can take care of enabling it again afterwards, don't worry19:07
dviroelis part of compose work19:07
dviroelrlandy|ruck: dasm|rover, fyi, no-collections mode is here https://review.opendev.org/c/openstack/tripleo-quickstart/+/82530719:08
rlandy|rucknow github is dead?19:08
dviroelOMG19:08
rlandy|ruckrun while you can!!!19:09
rlandy|ruckit's all falling19:09
rlandy|ruckdviroel: ok - back to old fashioned grep19:09
rlandy|ruckI assume this is only in wallaby19:09
rlandy|ruckputting in patch19:09
dviroelrlandy|ruck: only wallaby19:10
rlandy|ruckdviroel: will I break anything else??19:10
dviroelrlandy|ruck: shouldn't not, this will revert to the old repo-setup mode19:10
rlandy|ruckfamous last words :)19:11
rlandy|rucklet's try19:11
* dviroel with two 'not'19:11
dviroelhttps://www.githubstatus.com/incidents/fz1bdbw24y81?utm_ts=164382918319:14
dasm|roveryes, it's dead19:14
rlandy|ruckin sympathy with downstream19:15
dviroeldasm|rover: oh, in gates, https://review.opendev.org/c/openstack/puppet-nova/+/82688919:38
dasm|roverdviroel: what am i looking at?19:39
dviroeldasm|rover: it says, closes-bug https://bugs.launchpad.net/puppet-nova/+bug/195960119:39
dasm|roveroh, i completely forgot about that19:40
dasm|roverthanks19:40
dviroeldasm|rover: hum, there are other fixes proposed too https://review.opendev.org/q/topic:bug%252F195960119:40
dviroelwill tp afterwards19:41
dasm|rover"tp"?19:41
dviroelovb cs9-wallaby ovb jobs were failing in the same issue19:41
dviroeltestproject with some ovb jobs19:42
dasm|roverack19:42
dasm|roverlooks like it's working: https://review.rdoproject.org/r/c/testproject/+/3892119:46
dasm|rover:)19:46
dasm|roverbrb19:47
dviroelslaweq: thanks for https://bugs.launchpad.net/tripleo/+bug/1959582/comments/220:02
dviroelrlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/1959582/comments/220:02
rlandy|ruckok - getting there on testproject20:04
rlandy|ruckdasm|rover if testproject passes, pls put a depends on for amol's patch20:05
rlandy|ruckdviroel: thanks20:08
slaweqdviroel yw :)20:08
slaweqnow I really call it a day :)20:08
rlandy|ruckhave a great vacation20:10
dviroelbuild-containers start failing on c9-wallaby20:12
dviroelhttps://logserver.rdoproject.org/39/38639/4/check/periodic-tripleo-ci-build-containers-centos-9-push-wallaby/6565f4d/logs/container-builds/ef05d679-5254-40d1-9f01-8771c872be50/base/base-build.log20:12
dviroel:(20:12
dasm|roverback20:20
dasm|roverrlandy|ruck: https://review.rdoproject.org/r/c/testproject/+/38921 all passed!20:32
rlandy|ruckdasm|rover: ok - let's add depends on20:32
dasm|rovertest run for depends-on still in progress: https://zuul.opendev.org/t/openstack/status/change/823688,2520:32
rlandy|ruckdasm|rover: there is no depends-on on https://review.opendev.org/c/openstack/tripleo-ci/+/823688/20:33
dasm|roverrlandy|ruck: what i meant -- jobs for ^ are still running20:34
rlandy|ruckdasm|rover: pls add depends on https://review.opendev.org/c/openstack/tripleo-quickstart/+/827544 20:34
dasm|roveri wanted to wait few more before changing commit message, for a full test results20:34
dasm|roverk20:34
rlandy|ruckok20:34
rlandy|ruckdasm|rover: that's fine20:34
rlandy|ruckyou can wait20:34
dasm|roverdviroel: can you +W this https://review.opendev.org/c/openstack/tripleo-quickstart/+/827544 ?20:35
dviroeldasm|rover: yeah, i was waiting ci results20:36
dasm|roverit just finished -- all green20:36
dviroelfrom the pach itself, to see if nothing breaks (i'm 99% sure that nothing will break)20:37
dviroelyeah, it won't merge if something break anyways20:38
dasm|roverah20:38
dasm|roverthere is still that 1% :)20:38
dasm|roverdviroel: it failed due to > TASK [ovb-manage : Find out UUID of instance with metadata file]20:40
dasm|roverhttps://logserver.rdoproject.org/44/827544/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/36bbff5/job-output.txt20:40
dasm|roverheh20:40
dviroel?20:42
dasm|roverrlandy's patch which you +2'ed failed on wallaby job20:42
dviroelisn't 2022-02-02 14:39:57.258706 | primary | TASK [fetch-images : Check if we have a latest image] **************************20:42
dviroel2022-02-02 14:39:57.258808 | primary | Wednesday 02 February 2022  14:39:57 -0500 (0:00:00.044)       0:04:46.091 ****20:42
dviroel2022-02-02 14:39:58.720508 | primary | fatal: [undercloud]: FAILED! => {"changed": false, "cmd": ["test", "-f", "/home/zuul/images-cache/latest-overcloud-hardened-uefi-full.qcow2"], "delta": "0:00:00.004912", "end": "2022-02-02 14:39:58.515170", "msg": "non-zero return code", "rc": 1, "start": "2022-02-02 14:39:58.510258", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}20:42
dasm|roveror that20:43
rlandy|ruckthere are no current-tripleo wallaby images20:43
rlandy|ruckbecause we never ran image build jobs20:44
rlandy|ruckand the hash the current line ran on is the same20:44
rlandy|ruck so it never promoted again20:44
rlandy|ruckand the dir is not marked current-tripleo20:44
rlandy|ruckdviroel: we need to go to that images dir and create the symlink20:45
* rlandy|ruck gets20:45
dviroeldasm|rover: this one is looking good so far https://zuul.openstack.org/status#827544 (quickstart one)20:46
rlandy|rucksec - creating symlink20:47
dviroeldasm|rover: there are some errors in logs that don't actually throw jobs into failure - so I usually search for "failed: 1" in job-output.txt and scroll up20:48
dviroelusually works20:49
dasm|roveroh, right. i looked at red herring20:49
dasm|rovermy bad20:49
rlandy|ruckdone20:50
dviroelrlandy|ruck: nice, tks20:50
rlandy|ruckpls try test again20:50
* rlandy|ruck hack, hack, hack20:50
dasm|roverack20:51
dasm|roverdviroel: do we have something like "recheck rdo" to recheck *just* rdo 3rd party on this patch: https://review.opendev.org/c/openstack/tripleo-quickstart/+/82754421:01
dasm|roveror do i have to add testproject for that?21:01
rlandy|ruckdasm: ack ...21:02
rlandy|ruckgetting21:02
rlandy|ruckcheck-rdo21:02
dasm|roverthanks rlandy|ruck++21:03
dasm|rover:( promotion line for c8 train failed on: periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train21:04
dasm|roverchecking log21:04
dasm|roverhttps://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/e51f4b1c77a3a0d44f41dcdc23e4d6297dcaf4a0/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/236b05a/21:04
dasm|roverhttps://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/e51f4b1c77a3a0d44f41dcdc23e4d6297dcaf4a0/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/236b05a/logs/undercloud/home/zuul/undercloud_install.log.txt.gz21:05
dasm|rover> tripleoclient.exceptions.DeploymentError: Deployment failed21:05
rlandy|rucktrain already promoted21:06
rlandy|ruckyou're  ok21:06
dasm|roveroh21:06
dasm|roverphew21:06
rlandy|ruckdasm|rover: wallaby is 2 days out21:07
rlandy|ruckand victoria is one we need21:07
dasm|rovervictoria is still running21:07
rlandy|ruckwill chat with marios about master c9 in his morning21:07
rlandy|ruckyou;re fine21:07
rlandy|ruckno worries21:07
dasm|roveradded dpeends-on on akahat's patch21:10
dviroelok, will let my W+1 on 827544 before going afk21:15
dviroelis is working as expected21:15
dviroelwon't affect cs9-wallaby, since they use different release files21:15
* dviroel is in charge of dinner today21:16
dasm|rovernice!21:20
dviroelsee you all tomorrow o/21:21
dasm|rover\o dviroel 21:21
*** dviroel is now known as dviroel|out21:22
dasm|roverrlandy|ruck: jobs on akahat's patch look good. They're still running. I left a note to marios.22:01
rlandy|ruckdasm|rover: thank you22:01
dasm|rover\o22:01
*** dasm|rover is now known as dasm|off22:01
rlandy|ruckdasm|off: have a good night22:11
« Tuesday, 2022-02-01 Index Thursday, 2022-02-03 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!