Monday, 2022-07-25

ysandeephello oooci and good morning o/ 06:05
akahat|ruckysandeep, o/ hello Good Morning.06:07
marios\o06:09
akahat|ruckarxcruz|rover, o/06:51
arxcruz|roverakahat|ruck hey06:51
arxcruz|rovergood morning06:51
akahat|ruckarxcruz|rover, good morning!!06:51
akahat|ruckarxcruz|rover, let's have sync.06:51
akahat|ruckpossible now? or after some time?06:52
arxcruz|roverakahat|ruck give me 10 minutes, i'm puting some jogs to re-run06:52
arxcruz|roversounds good for you?06:52
akahat|ruckarxcruz|rover, i've already put some master /train /wallaby job in re-run06:53
akahat|ruckhttps://review.rdoproject.org/zuul/status#4146906:53
arxcruz|roverakahat|ruck ouch xD06:58
arxcruz|roverakahat|ruck ok, let's sync 06:58
arxcruz|roverwait, let me grab another coffee 06:58
akahat|ruckarxcruz|rover, ok06:59
akahat|ruckarxcruz|rover, https://meet.google.com/ssw-rukz-acx?pli=106:59
jm1moin folks :)07:00
akahat|ruckarxcruz|rover, https://review.rdoproject.org/r/c/testproject/+/4146507:04
akahat|ruckarxcruz|rover, http://pastebin.test.redhat.com/106720407:25
arxcruz|roverakahat|ruck https://code.engineering.redhat.com/gerrit/c/testproject/+/42136307:29
akahat|ruckarxcruz|rover, thank you !! :)07:30
arxcruz|roverreviewbot 07:36
arxcruz|roverreviewbot please add https://review.opendev.org/c/openstack/tripleo-quickstart/+/85056807:37
arxcruz|roverreviewbot please add https://review.opendev.org/c/openstack/tripleo-quickstart/+/850568 to review list07:37
reviewbotI have added your review to the Review list07:37
*** ysandeep is now known as ysandeep|afk07:49
* akahat|ruck lunch07:55
arxcruz|roverakahat|ruck https://code.engineering.redhat.com/gerrit/c/testproject/+/421363 failed07:56
jm1akahat|ruck: 3 passes in a row, wanna close this one? https://bugzilla.redhat.com/show_bug.cgi?id=210980708:46
* bhagyashris lunch08:49
mariosneed review please https://review.opendev.org/c/openstack/tripleo-ci/+/841583 thank you08:49
reviewbotI have added your review to the Review list08:49
marioschandankumar: thanks for voting, sorry i noticed something just now and will update (moving thje job to minimal template because https://review.opendev.org/c/openstack/tripleo-ci/+/841583/26/zuul.d/multinode-jobs.yaml#16108:55
marios(its going to wipe your vote chandankumar sorry and thanks again)08:56
chandankumarack sir08:57
mariosthanks chandankumar 09:04
akahat|ruckjm1, we have green run. You can close that.09:36
jm1akahat|ruck: please go ahead and close it :)09:38
*** ysandeep|afk is now known as ysandeep09:43
rlandyarxcruz|rover: akahat|ruck: hey - how are things?10:35
rlandywant to sync?10:35
arxcruz|roverrlandy everything is in control, i'm fine to sync if you want 10:35
akahat|ruckrlandy, o/10:36
akahat|ruckrlandy, yes.10:36
rlandyarxcruz|rover: akahat|ruck - yeah ... it will be quick ... https://meet.google.com/hwe-umtv-gii?pli=1&authuser=010:36
*** jpodivin_ is now known as jpodivin11:01
arxcruz|roverrlandy 11:06
arxcruz|roverrlandy https://bugs.launchpad.net/tripleo/+bug/1982464 is a tracking but only 11:06
rlandyfrenzy_friday: dviroel_: cockpit downstream is not collecting data11:11
rlandyprobably from the weekend outage11:12
rlandyfrenzy_friday: dviroel_: can you check that? maybe needs a restart?11:12
rlandyakahat|ruck; jm1: https://bugs.launchpad.net/tripleo/+bug/1982552 - is unvalid the right state here?11:13
frenzy_fridayrlandy, checking11:13
akahat|ruckrlandy, yes. that was invalid.. but Rabi has comment to change version to 1.0.0 11:14
akahat|ruckmay be chandankumar, jm1 ysandeep can comment on this.11:14
*** jgilaber is now known as jgilaber|lunch11:16
mariosreview time o/ join us11:17
chandankumarakahat|ruck: https://github.com/ansible/ansible/issues/6895011:17
chandankumarakahat|ruck: Do you have the full error log?11:29
chandankumarof the above failure11:29
akahat|ruckchandankumar, https://bugzilla.redhat.com/show_bug.cgi?id=210980711:30
ysandeepmarios, https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/850506/1#message-6d5c31f3efeaeceece69705e7825d6e2086bcfe911:33
*** dviroel_ is now known as dviroel11:43
dviroelo/11:45
frenzy_fridaythe downstream cockpit containers have connection error to sf.hosted.upshift.rdu2.redhat.com restarting now11:47
jm1akahat|ruck, chandankumar, rlandy: https://bugs.launchpad.net/tripleo/+bug/1982552 is invalid because only downstream is affected11:50
jm1akahat|ruck, chandankumar, rlandy: this is the downtream bug https://bugzilla.redhat.com/show_bug.cgi?id=210980711:50
jm1akahat|ruck, chandankumar, rlandy: it occured once. the last three runs it passed. we have no idea why it failed. logs dont help here, see bug report comments11:51
frenzy_fridayrlandy, I have restarted the containers in incockpit. The data should show up in some time12:05
rlandyfrenzy_friday: thank you12:07
rlandyjm1: thanks for confirming12:07
*** ysandeep is now known as ysandeep|afk12:13
dviroelfrenzy_friday: hi, rebooting fixed the issue?12:24
frenzy_fridaydviroel, I see almost all the data now but http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/RMjmDrEWk/downstream-dashboard?orgId=1 -baremetal, rhos 17.0 trunk-patches, 16.0 are missing12:26
frenzy_fridayhttp://tripleo-cockpit.lab4.eng.bos.redhat.com/d/gHXbeNp7z/centos-9-and-centos-8-comparison-master?orgId=1 some data is missing here as well12:27
frenzy_fridayIn the logs I can still see influxdb doing the queries, so maybe it needs more time?12:28
dviroelfrenzy_friday: yeah, i think that trunk-patches wasn't showing nothing before, if you switch to display data from 30 days ago, still don't have anything12:30
frenzy_fridayyep12:31
* dviroel coffee12:32
marios15:35 #tripleo: < opendevreview> Marios Andreou proposed openstack/tripleo-quickstart master: Remove undercloud_enable_paunch from fs and tidy up fs 66  https://review.opendev.org/c/openstack/tripleo-quickstart/+/85088112:36
mariosoops 12:36
bhagyashrisrlandy, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/411458/19/zuul.d/standalone-jobs.yaml12:37
chandankumarrlandy++ frenzy_friday++ thanks, learnt to generate zuul secrets :-)12:37
frenzy_fridaychandankumar, Cool :D12:38
mariosjm1: o/ fyi https://review.opendev.org/c/openstack/tripleo-upgrade/+/850509/1#message-62f7c453d270192fd6b054c5c5db408ec30a9f25 (discussed in today's reviews call)12:40
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.12:40
* akahat|ruck brb12:41
chandankumarjm1: thank you for the links :-)12:41
jm1marios: ack, thanks!12:42
bhagyashrishttps://review.opendev.org/c/openstack/tripleo-ci/+/842973/5/zuul.d/standalone-jobs.yaml12:42
mariosjm1: just keeping that one fyi https://review.opendev.org/c/openstack/tripleo-quickstart/+/85050712:46
rlandyarxcruz|rover: akahat|ruck: what time is CIX on your calendar?13:24
arxcruz|roverrlandy in one hour 13:25
arxcruz|roveris it now ?13:25
rlandyarxcruz|rover: it did move - talking with apevec now13:26
akahat|ruckrlandy, same13:26
rlandyon rhos-ops13:26
rlandyclashes with our scrum meeting13:26
rlandyarxcruz|rover: akahat|ruck: guess you will have to miss second half of scrum to attend13:28
rlandywill chat with alan at prod chain about this13:28
arxcruz|roverok13:28
akahat|ruckrlandy, ack13:28
*** ysandeep|afk is now known as ysandeep13:30
jpodivinakahat|ruck: there appears to be a dependency issue in the tox-docs job for openstack/release. https://bugs.launchpad.net/tripleo/+bug/1982757 13:37
jpodivinmust have been a recent change13:37
akahat|ruckjohnsom, looking13:39
jpodivinhere it is https://review.opendev.org/c/openstack/requirements/+/850459/3/upper-constraints.txt#54113:40
akahat|ruckjpodivin, yeah.. definitely that patch caused this issue.13:42
jpodivinakahat|ruck: the version cap is from https://review.opendev.org/c/openstack/releases/+/811941 But there is no bug linked so I have no idea if it's still necessary 13:43
jpodivinI'm going to run some tests on a potential fix and post it as WIP if it works out. 13:44
* jpodivin it didn't13:45
akahat|ruckjpodivin, changing jsonschema version to 4.7.2 might resolve issue.13:45
jpodivinakahat|ruck: unfortunately the original bug is still present13:46
jpodivinso if we just remove the 4.0.0 cap it's going to cause other failures. 13:46
akahat|ruckjpodivin, I'm referring changing version 4.7.2 in openstack/releases/requirements.txt file13:46
jpodivinakahat|ruck: me too 13:47
akahat|ruckjpodivin, yeah.. let's try it.. if we don't face any issues then we are good with it.13:47
jpodivinI've just tried that.13:47
akahat|ruck:)13:47
jpodivin The 'validate' environment fails on ypeError: create.<locals>.Validator.__init__() got an unexpected keyword argument 'types'13:48
jpodivinso ... we'll have to patch the 'Validator' class. 13:48
dasm|offo/13:50
*** dasm|off is now known as dasm13:51
jpodivinor rather call 13:51
bhagyashrisakahat|ruck, arxcruz|rover rlandy rcastillo chandankumar 14:00
bhagyashrisscrum time14:00
bhagyashrisjm1, ^14:00
bhagyashrisysandeep, ^14:01
ysandeepI am there :D14:01
bhagyashrisdviroel, ^14:01
dviroeljoinning14:01
arxcruz|roverrlandy akahat|ruck wallaby c9 promoted, train should promoter as well, all jobs passed 14:33
arxcruz|roverwallaby c8 i'm running the two missing jjobs to promote 14:34
akahat|ruck\o/14:34
rlandyarxcruz|rover++14:34
rlandyarxcruz|rover: can you dequeue and enqueue w c8?14:34
rlandyfailed image build14:35
arxcruz|roverrlandy train or wallaby?14:35
arxcruz|rover oh, w as in wallaby 14:35
rlandyack14:36
arxcruz|roverrlandy i'm running the image-builder right now 14:36
mariosfrenzy_friday: that one https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/402889 14:36
mariosrlandy: ^^ 14:36
rlandyakahat|ruck: pls look at failures on tripleo component14:40
rlandy17 on rhel-8 still failing14:41
rlandyakahat|ruck: we may be missing a review there14:41
rlandypls compare with what is in train /16.214:41
akahat|ruckrlandy, yes.. filling bug14:42
rlandyakahat|ruck: it's the same issue14:42
rlandyI think14:42
akahat|ruckrlandy, no. we hit now nova issue.14:42
akahat|ruckrlandy, https://sf.hosted.upshift.rdu2.redhat.com/logs/2f/2f5112cd57be0e1f4aa7177473c47f86787ea11b/openstack-component-tripleo/periodic-tripleo-ci-rhel-8-scenario004-standalone-tripleo-rhos-17/1fc433c/logs/undercloud/var/log/containers/nova/nova-compute.log14:42
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/2f/2f5112cd57be0e1f4aa7177473c47f86787ea11b/openstack-component-tripleo/periodic-tripleo-ci-rhel-8-standalone-tripleo-rhos-17/98306af/logs/undercloud/var/log/tempest/stestr_results.html14:43
rlandy   raise libvirtError('virDomainDefineXML() failed')14:43
rlandy2022-07-25 13:32:02.505 7 ERROR nova.compute.manager [instance: 8fb4a195-beeb-4b94-950e-0575c1357aca] libvirt.libvirtError: unsupported configuration: Emulator '/usr/libexec/qemu-kvm' does not support machine type 'pc-q35-rhel9.0.0'14:43
rlandyok that14:43
chandankumarrlandy: dviroel: can we get https://review.opendev.org/c/openstack/tripleo-ci/+/850893 +w ?14:44
akahat|ruckrlandy, https://bugzilla.redhat.com/show_bug.cgi?id=211053514:46
akahat|ruckrlandy, one more job hit with different error, https://sf.hosted.upshift.rdu2.redhat.com/logs/2f/2f5112cd57be0e1f4aa7177473c47f86787ea11b/openstack-component-tripleo/periodic-tripleo-ci-rhel-8-bm_envE-3ctlr_1comp-featureset001-tripleo-rhos-17/4126c43/job-output.txt14:47
akahat|ruckrerunning it to confirm the issue.14:47
dviroelhttps://review.opendev.org/c/openstack/tripleo-ci/+/85068914:50
ysandeeprlandy, fyi.. baremetal jobs is getting stable for master line: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master&skip=0 (6/10 green - not bad)14:51
akahat|ruckrlandy, ysandeep https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-bm_envE-3ctlr_1comp-featureset001-tripleo-rhos-17&skip=0 14:54
akahat|ruckthis job has failure history since last month.14:55
akahat|ruckdo we need to consider it or skip?14:55
mariosjm1: you didn't like the suggestion? 14:59
marios:D14:59
mariosor is it cos i always say 'just send an email and say '... ;)14:59
* marios coffee brb14:59
ysandeepakahat|ruck, you can skip that for now its not in criteria15:01
jm1marios: all is fine! i will look for the last osp all hands / info shareing recordings and if i cannot find it there i will just ping maria. i might leave out that we discussed layoffs in our team though ;)15:01
akahat|ruckysandeep, ok. Thanks for info.15:01
ysandeepakahat|ruck, we have to move rhel-8 job to rhel-9, once 17.1 line is up15:01
ysandeepless baremetal env more lines :D15:02
akahat|ruck:D15:03
*** ysandeep is now known as ysandeep|dinner15:04
mariosjm1: :)15:05
*** dviroel is now known as dviroel|lunch15:08
* akahat|ruck back in hour15:09
chandankumarrlandy: dasm dviroel|lunch https://review.rdoproject.org/r/q/topic:ibm_cloud merge please15:10
bhagyashrisrlandy, frenzy_friday ysandeep|dinner marios 17.1 criteria file patch https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/420519 15:10
dasmchandankumar++15:13
arxcruz|roverrlandy train seems to be promoted15:17
arxcruz|rovertesting to current-tripleo: (subhash aggregate: 8b105f1c9d61037014914c5d4ad09e4c, commit: b90ee9e79c82658c4b7c9764d16b8c46abbe90e6, distro: a9b76eb5574f9d38cb0dc56619f5e1840621d135, extended: None, component: validation, timestamp: None) API returned different promoted hash: 'aggregate: 8b105f1c9d61037014914c5d4ad09e4c, commit: 4a9dbcbb4fc2851d503c30419783d033db285bad, distro: c9cbe2502bb180b40f28b15401b8e1a9e19a653a, extended: None,15:17
arxcruz|rover component: common, timestamp: None'15:17
arxcruz|rover2022-07-25 15:04:29,738 3266993 INFO     promoter Candidate hash 'aggregate: 8b105f1c9d61037014914c5d4ad09e4c, commit: b90ee9e79c82658c4b7c9764d16b8c46abbe90e6, distro: a9b76eb5574f9d38cb0dc56619f5e1840621d135, extended: None, component: validation, timestamp: 1658686002': SUCCESSFUL promotion to current-tripleo15:17
arxcruz|roveralthough i'm not seeing it in the dashboard 15:17
arxcruz|roverthe current-tripleo is pointing to this hash, that is from today 15:17
arxcruz|rovermaybe the script on dashboard didn't run yet 15:18
chandankumarrlandy: once this https://review.rdoproject.org/r/c/config/+/44148 merges, please recheck this patch https://review.rdoproject.org/r/c/testproject/+/44151 to trigger fs01 , feel free to update the patch 15:21
chandankumardasm: thanks :-)15:21
chandankumarsee ya \o15:23
rlandybhagyashris: hey - going to merge that15:28
rlandylet's see if we get a promo15:29
*** ysandeep|dinner is now known as ysandeep|out15:45
rlandybhagyashris: patch merged15:50
rlandyarxcruz|rover++ nice15:50
rlandychandankumar: w+'ed https://review.rdoproject.org/r/c/config/+/441415:56
rlandylunch - brb15:58
* akahat|ruck going offline in 20 mins.16:24
rlandyakahat|ruck: periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-tripleo-rhos-16.2 needs  arerun16:25
akahat|rucklooking16:26
*** dviroel|lunch is now known as dviroel16:26
dviroelrlandy: fyi, I updated rr schedule, me and frenzy_friday swapped our shifts16:28
dviroelrlandy: but i will cover her on frinday, since she is out16:28
rlandydviroel: sure16:29
akahat|ruckRunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/30091316:34
akahat|ruckrlandy, do you need anything from me before i go offline?16:37
arxcruz|roverrlandy akahat|ruck master fails only on fs035 what do you guys want to do?16:38
dasmbrb16:38
arxcruz|roverre-run or skip it ?16:38
akahat|ruckarxcruz|rover, sqlalchemy error still hitting this?16:41
arxcruz|roverakahat|ruck from the revert patch yes16:41
akahat|ruckoh..16:42
akahat|ruckarxcruz|rover, revert of reverted patch needs to create? :D16:42
arxcruz|roverakahat|ruck no, since the revert patch wasn't merged yet 16:43
akahat|ruckoh.. then it's okay.. I thought that patch got merged.16:44
dasmback16:45
* akahat|ruck offline.. see you tomorrow.16:47
rlandyarxcruz|rover: promote of it's not consistent16:52
rlandyakahat|ruck: I rekicked the 16.2 tripleo fs001 failure16:53
rlandyarxcruz|rover: looking at fs03516:53
arxcruz|roveri rekick fs035 now, i'll see if it will pass or not first 16:54
rlandyarxcruz|rover: looks like it's a deploy failure16:56
rlandynot a tempest thing16:56
rlandyFATAL | Check Keystone user assignment to roles status | undercloud | item=swift | error={"ansible_job_id": "480130905223.220827", "ansible_loop_var": "tripleo_keystone_resources_user_role_async_result_item", "attempts": 1, "changed": false, "extra_data": {"data": null, "details": "None", "response": "None"}, 16:57
arxcruz|roverrlandy ok... 16:57
rlandyarxcruz|rover: ^^ legit16:57
rlandyfailure16:57
arxcruz|roverok16:57
arxcruz|roverchecking 16:57
rlandyarxcruz|rover: probably bug/CIX this16:58
rlandysomething with keystone and ipv616:58
rlandyarxcruz|rover: ping security 16:59
rlandyarxcruz|rover: sisn;t you recently promote security component?17:00
arxcruz|roverbut it wasn't in master 17:00
rlandyok - checking latest commits then17:01
rlandyarxcruz|rover: deploy failures look like they started today17:09
rlandyarxcruz|rover: possible impact patch ... https://github.com/openstack/tripleo-heat-templates/commit/906991f099469f88c7cecf61877c52e12527ad5c - merged on 07/2117:11
arxcruz|roverhmmm17:12
arxcruz|roverok, i'll take a look 17:12
rlandyarxcruz|rover: you can try ping ade_lee_ on #tripleo17:16
rlandyhe's around17:16
rlandymight know off hand17:16
rlandyif not, you can bug it17:16
arxcruz|roverok17:20
* dviroel brb18:22
rlandyarxcruz|rover: late for you18:54
rlandycan sort through this in your mroning18:55
*** dviroel is now known as dviroel|afk20:17
arxcruz|roverrlandy wallaby c8 is promoting20:31
arxcruz|roverrlandy promoted, now only master missing 20:33
arxcruz|rovergonna sleep now 20:33
rlandyarxcruz|rover++ thanks21:04
* dasm => offline21:25
dasmo/21:25
*** dasm is now known as dasm|off21:25
*** undefined is now known as Guest602222:41
*** rlandy is now known as rlandy|out23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!