Tuesday, 2021-11-02

« Monday, 2021-11-01 Index Wednesday, 2021-11-03 »
*** pojadhav|out is now known as pojadhav|ruck04:26
*** ykarel|away is now known as ykarel04:29
*** jpodivin_ is now known as jpodivin06:43
frenzy_fridayrechecking https://review.rdoproject.org/r/c/rdo-jobs/+/36504 since the gate jobs didnt start08:22
chandankumarykarel: \o please have a look https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/816123/1#message-3fd5f04890d60ead3e5138d11bbe2ededf427961 thanks!08:47
ykarelchandankumar, ack will update08:49
ykarelthough i don't like updating just for that08:49
ykarelas it will rerun jobs unnecessary08:49
chandankumarykarel: thanks, totally agree, I saw you updated the patch.09:01
*** pojadhav|ruck is now known as pojadhav|afk09:27
*** ykarel is now known as ykarel|lunch09:39
*** pojadhav|afk is now known as pojadhav|ruck09:40
sshnaidmfrenzy_friday, hi, can you please take a look at these errors? https://paste.opendev.org/show/810329/09:56
sshnaidmfrenzy_friday, for example here: https://logserver.rdoproject.org/67/36267/15/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/e73d2f2/logs/quickstart_collect_logs.log09:57
frenzy_fridaysshnaidm, hi, checking09:57
pojadhav|ruckchandankumar, updated https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/81577709:58
*** pojadhav|ruck is now known as pojadhav|lunch10:02
*** jpena|off is now known as jpena10:10
frenzy_fridaysshnaidm, can you send me the patch url pls?10:15
sshnaidmfrenzy_friday, this job is from here: https://review.rdoproject.org/r/c/testproject/+/36267 But I suspect it happens in others too10:16
frenzy_fridayhttps://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809979/11/roles/collect_logs/tasks/sova.yml#21 - this looks good to me, not sure which variable it did not find. Does is defined work with ternary?10:16
*** pojadhav|lunch is now known as pojadhav|ruck10:30
*** rlandy is now known as rlandy|ruck10:32
chandankumarmarios: patching podman failure10:33
rlandy|ruckpojadhav|ruck: hello - how are things on ruck/rover?10:36
rlandy|ruck17 line still down10:36
marioschandankumar: 2021-11-02 10:10:40.115974 | primary |   "msg": "Depsolve Error occured: \n Problem: package python3-tripleoclient-17.1.1-0.20211027132314.ce6a555.el9.noarch requires podman, but none of the providers can be installed\n  - conflicting requests\n  - package podman-1:3.4.2-0.2.el9.x86_64 is filtered out by modular filtering", 10:36
rlandy|ruckwill look into that more10:36
pojadhav|ruckrlandy|ruck, hello - we are good at promotions :)10:37
marioschandankumar: that one ?10:37
rlandy|ruckexcept 1710:37
sshnaidmfrenzy_friday, yeah, ternary there is problematic10:37
rlandy|ruck2 days agowallaby10:37
pojadhav|ruckrlandy|ruck, we are good at 17 components as well - https://code.engineering.redhat.com/gerrit/c/testproject/+/27623010:37
rlandy|ruckhttps://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/10:38
rlandy|ruckbetter as well10:38
sshnaidmfrenzy_friday, I think both vars in ternary must be defined10:38
pojadhav|ruckyep10:38
rlandy|ruckpojadhav|ruck: great - so it's just the 17 integration line10:38
chandankumarsshnaidm: https://bugs.launchpad.net/tripleo/+bug/194945610:38
chandankumarsorry10:38
chandankumarmarios: ^^ yes10:39
frenzy_fridaysshnaidm, will "{{ sova_config_file_output.stdout | default(pattern_config.json) }}" work?10:39
pojadhav|ruckyeah10:39
marioschandankumar: thanks10:39
frenzy_fridaylemme ask Arx. Hey arxcruz looks like https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809979/11/roles/collect_logs/tasks/sova.yml#21 is causing  https://paste.opendev.org/show/810329/10:40
frenzy_fridayDoes ternary work woth is defined?10:40
sshnaidmfrenzy_friday, probably, try it locally10:42
* sshnaidm doesn't like using ternary in ansible10:43
sshnaidmfrenzy_friday, or just use two set_fact with "when: "10:45
rlandy|ruckpojadhav|ruck: pls check what's holding up wallaby10:45
rlandy|ruckpromotions10:45
pojadhav|ruckrlandy|ruck, ok10:46
rlandy|ruckakahat: hello 10:47
*** ykarel|lunch is now known as ykarel10:54
akahatrlandy|ruck, hello o/11:00
rlandy|ruckakahat: sec - just working on downstream11:01
*** pojadhav is now known as pojadhav|ruck11:01
*** amoralej|off is now known as amoralej11:03
*** pojadhav|ruck is now known as pojadhav|afk11:04
arxcruzfrenzy_friday: weird it should work, i'll check after lunch 11:05
frenzy_fridaythanks arxcruz 11:07
arxcruzfrenzy_friday: do you have the link for this failure ?11:08
arxcruzi mean , the logs 11:08
frenzy_fridayaye,  https://review.rdoproject.org/r/c/testproject/+/3626711:09
chandankumarhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3653411:16
chandankumarhttps://review.rdoproject.org/r/c/rdo-jobs/+/3647611:16
chandankumarhttps://review.opendev.org/c/openstack/tripleo-repos/+/81599111:16
chandankumarhttps://review.opendev.org/c/openstack/tripleo-repos/+/816211/11:16
chandankumarhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81612311:16
chandankumarhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81610311:16
chandankumarhttps://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/81577711:16
chandankumarhttps://review.opendev.org/c/openstack/tripleo-ci/+/81631811:16
chandankumarhttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/28675811:16
chandankumarmarios: arxcruz frenzy_friday ^^ Please have a look, when free, thansk :-)11:17
rlandy|ruckmarios: fyi - https://hackmd.io/FGMoCiRfSNa8puA1BpTQ-Q - public review list page11:21
mariosrlandy|ruck: thanks 11:22
arxcruzfrenzy_friday: where's the logs? 11:24
arxcruzwhich job?  :)11:24
* frenzy_friday looks11:27
frenzy_fridayarxcruz, https://paste.opendev.org/show/810329/ here is the log sshnaidm pasted11:28
rlandy|ruckmarios: hey - can you review https://review.opendev.org/c/openstack/tripleo-repos/+/815991? thanks11:28
arxcruzfrenzy_friday: what about the full log ?11:30
frenzy_fridayarxcruz, not sure, sshnaidm do you have the log link for the ternary error?11:32
mariosrlandy|ruck: ack adding to reviews 11:36
rlandy|rucksshnaidm: hey - you add a patch to test stevebaker's modify-image work last week. do you still have that patch available?11:38
* frenzy_friday going afk for ~1hr11:38
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81610311:38
rlandy|rucksshnaidm: ^^ up for review again11:38
sshnaidmrlandy|ruck, yeah, but I think it didn't run there, otherwise would detect the error. Lemme try another one11:39
rlandy|ruckok - thanks11:39
sshnaidmfrenzy_friday, arxcruz I pasted a full log abover11:39
arxcruzsshnaidm: maybe is a bug on ansible? i'm checking and both tasks were skipped because condition was false, but it's kinda of impossible, because one task runs if the variable is not defined, and the other when the variable is defined, but both are not being executed...12:03
arxcruzhttps://paste.opendev.org/show/810333/12:04
arxcruzsshnaidm: ^12:04
sshnaidmarxcruz, look at the second execution of "Load sova patterns from URL" task: https://logserver.rdoproject.org/67/36267/15/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/e73d2f2/logs/quickstart_collect_logs.log12:13
sshnaidmarxcruz, the problem is with "sova_config_file_output" undefined12:15
sshnaidmjust remove ternary there and do 2 set_fact tasks12:15
arxcruzsshnaidm: ack 12:19
*** pojadhav|afk is now known as pojadhav|rucl12:21
*** pojadhav|rucl is now known as pojadhav|ruck12:21
rlandy|ruckarxcruz, sshnaidm, marios, soniya29, pojadhav, akahat, chandankumar, frenzy_friday: hello - reminder: community call at 1:30pm UTC. 12:22
rlandy|ruckpls add agenda items: https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both12:22
arxcruzrlandy|ruck: ok12:22
rlandy|ruckline 3012:22
rlandy|ruckwe have upgrades coming this week12:22
rlandy|ruckso we'll address them first12:23
*** jpena is now known as jpena|lunch12:27
rlandy|ruckakahat: sorry - can you chat now?12:28
akahatrlandy|ruck, yes.12:28
akahatrlandy|ruck, we are good to merge this: https://review.opendev.org/c/openstack/tripleo-ci/+/79995812:28
arxcruzsshnaidm_: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/81635012:28
arxcruzfrenzy_friday: ^12:28
akahatwe have all green results for sc10:https://review.opendev.org/q/topic:%22scenario010-branchful%22+(status:open%20OR%20status:merged)12:29
rlandy|ruckakahat: one sec12:30
rlandy|ruckakahat: https://meet.google.com/bjf-hksk-uzp?pli=1&authuser=012:31
*** ykarel is now known as ykarel|afk12:33
akahatchandankumar, marios please add your votes when you are free, thanks!  https://review.opendev.org/c/openstack/tripleo-ci/+/79995812:47
rlandy|ruckpojadhav|ruck: wallaby failure: periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-wallaby12:48
rlandy|ruckchandankumar: marios: ^^ pls see akahat's comments on ^^ https://review.opendev.org/c/openstack/tripleo-ci/+/79995812:48
rlandy|ruckto spring scenario010 free12:48
mariosakahat: rlandy|ruck: ack adding to reviews 12:50
chandankumarakahat: ack12:51
rlandy|ruckthank  you12:52
*** pojadhav is now known as pojadhav|ruck12:53
chandankumarakahat: https://review.opendev.org/c/openstack/tripleo-ci/+/799958/7#message-663fd07ec638e5d800ca073cd2196ca4cd848c4b12:54
mariosrlandy|ruck: chandankumar: any idea about this volume https://rhos-d.infra.prod.upshift.rdu2.redhat.com/dashboard/project/volumes/8631a58c-f5f0-4697-83ce-2608265de6d1/ created 1 feb 21 but unused currently. i want to delete but no idea what/why 13:05
* rlandy|ruck looks13:06
mariosrlandy|ruck: doesn't give us much info there... like who created etc. there are a few things i want to delete esp a couple of promoter servers that look to be redundant for couple months at least (from looking at their logs they aren't running) but they belong to bshewale and ysandeep so in 2 minds about waiting for them to come back first 13:07
mariosrlandy|ruck: maybe i just have to ask on a couple of scrum meetings or will send out an email to the team to ask 13:07
mariosand/or13:07
rlandy|ruckmarios: if they are internal, and promoter related, chandankumar and akahat will know which ones can go13:08
rlandy|ruckother volumes ack13:08
rlandy|ruckopen at a scrum meeting13:08
rlandy|rucknobody claims, them13:08
rlandy|ruckdelete away13:08
mariosk13:09
rlandy|ruckysandeep|pto will be back on monday13:09
mariosack 13:09
rlandy|ruckbhagyashri's probably promoter related13:09
rlandy|ruckakahat would know13:09
akahatchecking..13:09
chandankumarakahat: whuch tenant?13:09
mariosakahat: question is can i delete http://10.0.148.74/promoter_logs http://10.0.11.190/promoter_logs those two promoters13:09
mariosakahat: rhos-dev-stage tenant 13:10
mariosakahat: looks like they haven't run since 2021-08-12 & 2021-10-0813:10
akahat10.0.148.74 is used by Bhagyashri. But it is no longer in use.13:10
chandankumarakahat:  no one is using it 13:10
chandankumargood to delete 13:11
akahatand 10.0.11.190 we are using.13:11
mariosk i won't do it immediately but have to start asking anyway... getting the question out there. may wait for ysandeep to come back 13:11
mariosakahat: that the d/stream one? 13:11
akahatchandankumar, this is old promoter server ^^13:11
akahatyes downstream13:11
mariosakahat: k that is sandeep box... so may wait on that one till he's back, but looks like it didn't run for a while there 13:11
chandankumarakahat: I donot think we need old promoter server volume now13:12
chandankumaras everything is working fine13:12
akahatchandankumar, marios okay then we can remove it.13:12
akahat+!13:12
akahat+1 *13:13
rlandy|ruckmarios: there is one server I have on internal13:15
rlandy|ruckmarked with my name13:15
rlandy|rucknot on stage13:15
rlandy|ruckplease leave that one13:15
mariosrlandy|ruck: no worries... main issue right now is storage space we have quota for ram/cpu13:15
rlandy|ruckit's our emergency fall back on component promotions13:16
rlandy|rucksure13:16
mariosrlandy|ruck: so i need to find like 2/300 gb on the volumes to delete13:16
rlandy|ruckyep13:16
rlandy|rucktotally13:16
rlandy|ruckrcastillo: hello13:20
rlandy|ruckgood morning13:20
*** jpena|lunch is now known as jpena13:26
rlandy|ruckchandankumar: community call13:31
rcastillogood morning13:37
chandankumarmarios++ rlandy|ruck++ great discussion on upgrade jobs.14:02
rlandy|ruckall marios - I know very little about these things14:03
mariosi used to know but its all a bit hazy ;)14:03
*** ykarel|afk is now known as ykarel14:29
rlandy|ruckhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88a/814801/6/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/88a99be/logs/undercloud/var/log/tripleo-container-image-prepare.log14:30
rlandy|ruckarxcruz: ^^ are we still fighting the update/pgrdaes container hash?14:31
rlandy|ruckupgrades14:31
rlandy|ruckanother issue?14:31
rlandy|ruckwe were merging jobs like fire14:31
arxcruzrlandy|ruck: doesn't seems to be the same issue, but i'll check 14:31
arxcruzrlandy|ruck: this is on the upload container 14:32
frenzy_fridayrlandy|ruck, the list:  tripleo-ci-centos-8-content-provider, tripleo-ci-centos-8-undercloud-upgrade, tripleo-ci-centos-8-content-provider-train,  tripleo-ci-centos-8-content-provider-ussuri and openstack-tox-linters14:36
arxcruzhttps://i.imgur.com/FwclDDg.png14:41
arxcruzso yeah, it's starting to work now 14:42
rlandy|ruckarxcruz++ 14:42
rlandy|rucknice work on c914:42
rlandy|ruckarxcruz: sorry - just to get back to update/upgrade14:42
*** ykarel is now known as ykarel|away14:42
rlandy|ruckso this is not a mismatch of provider and install hashes?14:43
arxcruzrlandy|ruck: yes, so, i think this is not related14:43
rlandy|ruckhmmm14:43
rlandy|ruckwe hit that on check jobs as well14:43
rlandy|ruckpojadhav|ruck: ^^ pls see the errors here14:43
rlandy|ruckcan you investigate14:43
arxcruzrlandy|ruck: but i'll double check 14:43
rlandy|rucktripleo-ci-centos-8-scenario000-multinode-oooq-container-updatesfailure14:43
rlandy|rucktripleo-ci-centos-8-undercloud-upgrade14:43
rlandy|ruckcheck and gate14:43
rlandy|ruckwe were doing really well :)14:44
rlandy|ruckpojadhav|ruck: ok - 17 is back in action14:44
rlandy|ruckhttps://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/ is much better14:45
arxcruzrlandy|ruck: i'm not sure, i'm checking the prepare image role, because the error is there, doesn't seems to be related to my patch... 14:53
arxcruzat least i'm not seeing the relation 14:53
arxcruzi14:53
arxcruzi'll continue to debug, there are other runs that pass today 14:53
rlandy|ruckarxcruz: k - no problem - I'll look into it14:53
arxcruzso it is not all the time 14:53
rlandy|ruckjuts checking before I start14:53
rlandy|ruckI'll bug it if need be14:54
rlandy|ruckhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates14:54
rlandy|ruckhmmm14:54
rlandy|ruckbad trand14:54
rlandy|rucktrend14:54
rlandy|ruckbut all started now14:54
rlandy|ruckhome/zuul/undercloud_install.log 14:55
chandankumarrlandy|ruck: please keep an eye on this https://review.opendev.org/c/openstack/tripleo-common/+/816321 and recheck https://review.rdoproject.org/r/c/testproject/+/18953 once Alex comment or update that15:12
pojadhav|ruckrlandy|ruck, I checked wallaby failures, fS039 failed on node provision, rest fs01, fs02, full-tempest-api failed with tempest tests. will run testrun against it and will see issues are consistent one or not.15:12
chandankumarrlandy|ruck: thank you :-)15:12
chandankumarsee ya!15:12
*** amoralej is now known as amoralej|off15:13
rlandy|rucksshnaidm_: http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/nodepool/elements/virt-customize15:15
rlandy|ruckhttp://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/nodepool/virt_images/roles/zuul-user/tasks/main.yaml15:15
rlandy|ruckhttp://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/nodepool/virt_images/roles/zuul-user/tasks/main.yaml#n1515:16
pojadhav|ruckrlandy|ruck, wallaby failed jobs testrun - https://review.rdoproject.org/r/c/testproject/+/36274/615:16
rlandy|ruckpojadhav|ruck: will look - thanks15:25
sshnaidm_chandankumar, can we try ovb there again w/o container job? https://review.rdoproject.org/r/c/testproject/+/3626715:32
mariossshnaidm_: do you know why it fails? https://logserver.rdoproject.org/67/36267/17/check/tripleo-build-containers-stream9-development/f5110ad/logs/build.log 15:33
mariossshnaidm_: so if we don't build containers fs1/fs2 wont be able to run unless the hash is the same i.e. try to use the previous containers15:33
mariossshnaidm_: whatever it is is a new issue anyway they were building earlier/yesterday 15:36
sshnaidm_marios, msg="this version of runc doesn't work on cgroups v2"  https://logserver.rdoproject.org/67/36267/17/check/tripleo-build-containers-stream9-development/f5110ad/logs/container-builds/41f5fb46-cf93-430b-815d-342b93d91e4d/base/base-build.log15:39
sshnaidm_marios, probably wrong version of buildah/podman ?15:39
sshnaidm_I think chandankumar listed a bug about podman and appstream repo before..15:40
sshnaidm_maybe related15:40
mariossshnaidm_: it has depends-on https://review.opendev.org/c/openstack/tripleo-ci/+/816318 15:41
mariossshnaidm_: i also see that one though https://review.opendev.org/c/openstack/tripleo-common/+/816321 lets try adding it sec15:41
mariossshnaidm_: lets see if it helps15:41
sshnaidm_marios, ack15:44
*** sshnaidm_ is now known as sshnaidm15:49
* pojadhav|ruck leaving for the day.16:01
*** pojadhav|ruck is now known as pojadhav|out16:01
rlandy|rucklunch - brb16:20
jfrancoamarios: hey, if you're still around... I wanted to have a look at the logs from the standalone upgrade but last time it got triggered was in February https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone-upgrade 16:26
jfrancoamarios: what would be the best way to get it triggered again?16:26
jfrancoaa testproject?16:27
mariosjfrancoa: looking16:28
mariosjfrancoa: more recent runs there for victoria fyi https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-victoria 16:30
mariosjfrancoa: if you really wanted to run it for master you could try a testproject review 16:30
mariosjfrancoa: with just the job name16:30
jfrancoamarios: oh, ok..there is one per release..makes sense16:30
jfrancoamarios: thanks man! with that I have a starting point16:31
mariosjfrancoa: also ussuri one is available 16:31
mariosjfrancoa: ack np 16:31
mariosjfrancoa: fyi all the upgrades jobs stuff/definitions/templates and layout for check/gate lives there https://opendev.org/openstack/tripleo-ci/src/commit/0cad68cb2c8b970aeb0efd93104efc2669c48560/zuul.d/upgrades-jobs-templates.yaml#L144-L198 16:32
mariosthat points to the victoria pipeline for example ^^^ 16:32
mariosjfrancoa: (the link to standalone is you can see "standalone-upgrade-victoria" for the job name there 16:32
mariosif you want to checkout other jobs etc 16:32
jfrancoamarios: cool, that's pretty handy too16:33
jfrancoathanks marios16:33
*** marios is now known as marios|out16:47
*** jpena is now known as jpena|off17:32
rlandy|ruckyay - wallaby is promoting19:53
dpeacockFWIW folks, https://review.opendev.org/c/openstack/tripleo-quickstart/+/805384 was the ticket to solving my deployment woes; just tested it successfully and now have a wallaby deployment for the first time in a wee while20:00
rlandy|ruckdpeacock: added that patch to our team review list20:12
rlandy|ruckmarios had some comments there20:12
rlandy|ruckwe will didcuss20:12
rlandy|ruckdiscuss20:12
« Monday, 2021-11-01 Index Wednesday, 2021-11-03 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!