Wednesday, 2022-06-29

*** akahat|out is now known as akahat|ruck05:15
*** jm1|ruck is now known as jm1|rover07:06
jm1Hi ci 😄07:06
Tenguchandankumar: heya! fyi, Rabi found the last missing bit with podman for https://review.opendev.org/c/openstack/tripleo-ansible/+/84777407:07
Tenguchandankumar: ready to merge!07:07
Tenguzuul's all happy.07:07
*** amoralej|off is now known as amoralej07:34
*** jpena|off is now known as jpena07:39
chandankumarTengu: thanks, looks good08:18
chandankumarI need to open one more bug08:18
Tengu'k08:23
Tenguchandankumar: imho we can merge the patch - care to add the +W?08:25
chandankumarTengu: it depends on ubi9 patch08:26
Tenguoh. still not in? dang.08:26
Tengulemme rebase the patch on-to the ubi908:27
chandankumarTengu: yes, backup and restore job started failing 08:27
Tengu;_;08:27
Tenguchandankumar: need any help?08:27
Tenguhave to help my wife with the groceries shortly, but I'll be around in no time :)08:27
chandankumarTengu: ok next bug https://bugs.launchpad.net/tripleo/+bug/198019808:28
pojadhavakahat|ruck, jm1 : hey is this the known issue ??https://logserver.rdoproject.org/91/33791/25/check/periodic-tripleo-ci-centos-9-scenario012-standalone-compute-master/d0f4ef6/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz08:35
akahat|ruckpojadhav, no. This is new issue. 08:38
jm1akahat|ruck: will add it to known bugs08:39
akahat|ruckjm1, thanks!! Reporting lp for this.08:40
akahat|ruckjm1, pojadhav https://bugs.launchpad.net/tripleo/+bug/198020208:43
pojadhavakahat|ruck, ack 08:50
dpawlikhey, do you still have an issue with removing images on quay.rdoproject.org or it's better now ?08:50
Tenguchandankumar: checking.08:53
Tenguchandankumar: uho. puppet version vs facter version maybe?08:54
Tenguchandankumar: I think I'll do some changes in parallel in order to, if possible, get some more data out of the molecule job - for instance, installed packages and versions. That would really, really help debugging issues imho.08:57
chandankumarTengu: not sure about puppet versipn08:57
Tenguchandankumar: I can probably add this as some "debug output" in the test_deps role, since it's included by default in all molecule jobs. WDYT?08:57
chandankumarTengu: I think it is a good idea to capture those info08:58
chandankumarTengu: yes it will be included in all molecule job08:58
Tengu:) lemme check how to do this. Maybe writing in a dedicated file would be better?08:58
Tenguchandankumar: what would be nice? installed packages - what else?09:04
chandankumarTengu: install packages and repos configured09:04
Tengudnf repolist right?09:04
chandankumarTengu: yes09:04
Tengu'k. note, all has to be kept in ansible.log afaik, since it's in a container at this point. Unless we can bind-mount something in ALL molecule containers?09:05
chandankumarTengu: yes, it should appear in ansible.log09:07
chandankumardoing bind-mount for this is going to be much09:07
chandankumarTengu: please +w https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846166 it 09:09
Tenguchandankumar: sure09:16
Tenguand OK for the log.09:16
Tenguchandankumar: hmmm, I've based my patch onto yours - is it OK? guess we can correct the b'n'r later, as we're doing for the other issues?09:17
chandankumarTengu: It looks ok, yes09:21
Tenguwe'll see. Commented the LP with that link.09:22
Tenguno "related-bug" in my commit, since it's, well, not really related.09:22
chandankumarTengu: I think you need to hit https://review.opendev.org/c/openstack/tripleo-ansible/+/847774 rebase button09:24
chandankumarTengu: thank you for this one ansible.builtin.package_facts 09:30
chandankumarlearned new stuff09:31
*** rlandy|out is now known as rlandy09:33
rlandyakahat|ruck: jm1: how are things09:37
rlandylet's sync09:37
rlandyhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario001-standalone&skip=009:37
rlandyyou have a gate blocker09:37
Tenguchandankumar: ah, yeah, again. thanks fot the headsup!09:38
Tenguand, yeah, that package_facts is neat09:38
Tenguhello rlandy :)09:38
rlandyTengu: hello09:38
Tenguerk... master AND wallaby gate blocker?09:39
Tengureally?09:39
rlandyjm1: akahat|ruck; when you are around: https://meet.google.com/oro-zwcx-epz?pli=1&authuser=009:39
akahat|ruckrlandy, joining.09:40
chandankumarrlandy: can I start abandoning the patch from bottom to clear the gates?10:48
rlandychandankumar: I abandoned those that we expected to fail10:48
rlandyso everything in the gate now does not have an expected blocker10:49
rlandywe asked on opendev to move the patch to the top10:49
chandankumarrlandy: sc10 job is also affected i think10:49
rlandypinged Tengu about abandons10:49
chandankumarhttps://a7abc7e198a1b422a42c-194ac35e1d1618f04c867a7577cb6c34.ssl.cf2.rackcdn.com/842976/20/gate/tripleo-ci-centos-9-scenario010-standalone/732e388/job-output.txt10:49
rlandycorrect 10, 001, 00410:49
rlandyyep 10:49
rlandyany patch running master 001, 004 010 should be out the gate now10:50
chandankumarhttps://review.opendev.org/c/openstack/tripleo-ci/+/842976 abanonded this also10:57
soniyarlandy, arxcruz, please add/edit today's agenda for tempest review time meeting:- https://hackmd.io/xWJLuBHGSI2YGX7UlkwRbg11:09
mariosanyone joining review time? 11:16
*** dviroel|out is now known as dviroel11:20
rlandydropping - need to prep for calls11:42
rlandyakahat|ruck: pls rerun the failed wallaby c9 jobs ... 12:04
rlandyperiodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby12:04
rlandyperiodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby12:04
rlandyperiodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby12:04
rlandyjm1: same for 17 on rhel-912:05
rlandyreruns needed on12:05
rlandyperiodic-tripleo-ci-rhel-9-standalone-full-tempest-scenario-rhos-1712:05
rlandyperiodic-tripleo-ci-rhel-9-standalone-full-tempest-api-rhos-1712:05
*** amoralej is now known as amoralej|lunch12:06
dviroelsoniya: hi - here from jun 24th: https://31608e4f35b9c13baa68-7dedaad76deccd9ee39b1cd2f9dacaed.ssl.cf1.rackcdn.com/847241/1/gate/tripleo-ci-centos-9-standalone/d62bb9f/logs/undercloud/var/log/tempest/index.html12:11
dviroelnot running any test from tempest.scenario.test_minimum_basic.TestMinimumBasicScenario12:11
dviroelnow on Jun 27th: https://4f697cb28cb2fb22663a-3f036a988556af1e9af8d4c6bf51b036.ssl.cf2.rackcdn.com/847771/1/check/tripleo-ci-centos-9-standalone/3ea0a7c/logs/undercloud/var/log/tempest/index.html12:11
dviroelit is another test, a newer one, that was failing on fips job, with the floating IP error. So it might not be related to FIPS itself12:12
dviroelFIPS failures https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cd7/824479/33/check/tripleo-ci-centos-9-standalone/cd7328f/logs/undercloud/var/log/tempest/stestr_results.html12:12
dviroelif we plan to skip test_minimum_basic_scenario test, we may need to skip the entire class12:13
dviroel(which has 2 tests)12:13
dviroelLP bug https://bugs.launchpad.net/tripleo/+bug/192319412:14
*** beagles is now known as beagles|afk12:17
*** beagles|afk is now known as beagles12:17
*** beagles is now known as beagles_afk12:17
soniyadviroel, okay, so we need to skip this entire class, do we have bug reported about it?12:21
dviroelsoniya: not yet, rerunning fips change again, looks consistent in fips change only12:23
dviroelsoniya: i will open a bug if fails again12:23
soniyadviroel, let me know so that i can update the patch accordingly12:24
dviroelsoniya: ack - but something changed, and we have a new test running on this job12:25
dviroel:)12:25
soniyadviroel, ack12:26
rlandyakahat|ruck: I added a note to the question of the week answer in our section12:26
dviroelsoniya: another failure in THT https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ec/847759/4/check/tripleo-ci-centos-9-standalone/8ec02f0/logs/undercloud/var/log/tempest/stestr_results.html12:26
rlandyalso added both gate blockers12:26
akahat|ruckrlandy, yes. noted.12:27
dviroelsoniya: the fix was merged, after that skiplist revert was merged, at the end, skiplist was added back again. It is a flaky issue with no solution.12:33
dviroelsoniya: i will add a comment on the card and the same LP bug, there is no action for now IIUC12:34
dviroelsoniya: we just need to skip this test too12:34
jm1akahat|ruck++ thanks for giving the update in program call :)12:41
soniyadviroel, okay12:41
dviroelsoniya: https://bugs.launchpad.net/tripleo/+bug/1923194/comments/1412:44
soniyadviroel, ack12:46
*** pojadhav is now known as pojadhav|afk12:52
soniyarlandy, tempest review meeting?13:01
soniyaarxcruz, https://review.opendev.org/c/openstack/tempest/+/84785113:01
soniyaarxcruz, https://hackmd.io/xWJLuBHGSI2YGX7UlkwRbg13:02
jm1rlandy: regarding rhos 17 rhel 9, looks like bhagyashris gets us covered ;) https://code.engineering.redhat.com/gerrit/c/testproject/+/41240513:10
jm1rlandy: or do you want to rerun it again?13:10
arxcruzsoniya https://www.stackalytics.com/?release=all&module=tempest13:11
*** amoralej|lunch is now known as amoralej13:15
jm1rlandy: bhagyashris testproject successfully reran those two jobs with correct aggregate hash, checked that.13:17
* akahat|ruck afk.. back in few hours.13:23
rlandyjm1: checking13:24
rlandyjm1: you're right - bhagyashris has your back :)13:24
rlandyno worries13:24
rlandyjm1: and baremetal will promote13:24
rlandyrhel-913:25
jm1rlandy: yeah, baremetal is still running13:26
rlandyperiodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-17testprojectmastercheck211643,15013:27
rlandy2 hrs 56 mins 38 secs2022-06-28 22:06:34SUCCESS13:27
rlandyshould promote13:27
rlandyyou have a pass13:27
jm1rlandy, akahat|ruck: fyi after cix call i am eod13:51
jm1rlandy, akahat|ruck: rerunning c9 master tripleo and network component jobs again, at least one of the jobs failed because of intermittent failure13:58
rlandyok14:00
rlandyjm1: ack - ok14:00
*** dasm|afk is now known as dasm14:03
dasmo/14:03
jm1rlandy, akahat|ruck: recheck'ed c8 wallaby network component because of potential intermittent failure :/14:10
jm1rlandy: looks like you just reran c9 wallaby jobs. will add it to rr notes14:17
rlandyyep14:17
jm1rlandy: one thing i learnt this week is to always check zuul status for testprojects before running them myself ;)14:18
rlandygood :)14:18
dviroelsoniya: have you created the skiplist patch? If no, can I create?14:28
dviroelakahat|ruck: what is this ceph issue on check? known issue?14:29
dviroelTASK [tripleo.operator.tripleo_ceph_deploy : Run Ceph Deploy] failing14:30
dviroeloh, akahat|ruck is out14:30
soniyadviroel, no, give me few minutes14:30
dviroelsoniya: ack - thanks14:30
*** pojadhav|afk is now known as pojadhav14:35
jm1dviroel: yep, that error is known14:36
dviroeljm1: oh ok, seems that https://review.opendev.org/c/openstack/tripleo-ansible/+/848075 is the cause14:37
jm1dviroel: we have a fix and it has been merged some minutes ago14:37
jm1dviroel: yes, thats the fix14:37
dviroelah ok14:37
dviroelso my job failed because does not have it14:38
dviroel:)14:38
dviroelthanks, will recheck14:38
dviroeljust merged14:38
jm1rlandy: rerunning rhel8 osp16.2 octavia component because its out for 5 days14:53
jm1rlandy, akahat|ruck: eod now14:53
jm1rlandy, akahat|ruck: rhel8 osp16.2 octavia ( periodic-tripleo-ci-rhel-8-scenario010-standalone-octavia-rhos-16.2 ) fails on tempest tests very often but it has sporadic passes. updated rr notes14:57
* akahat|ruck reading 15:12
rlandyjm1[m]: thank you15:16
rlandyhave a good night15:16
rlandyakahat|ruck: watching node provision failures on wallaby c915:16
akahat|ruckrlandy, okay.. i'm looking in some new failures.15:17
rlandyakahat|ruck: fix for sc 01, 04 and 010 merged15:18
soniyadviroel, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/84814915:18
rlandytesting another fix for sc01215:18
akahat|ruckrlandy, yes. that is good now. 15:20
akahat|ruckthis is worring me: https://review.rdoproject.org/zuul/status#4146515:21
* dviroel lunch15:24
*** dviroel is now known as dviroel|lunch15:24
akahat|ruckrlandy, this mirror is not reachable: https://ab809b6e3c57fdbf1354-828d0c1d52a1a4d5db3905c705abe67e.ssl.cf5.rackcdn.com/847119/1/check/tripleo-ci-centos-9-content-provider/050552d/job-output.txt15:25
rlandychecking15:28
akahat|ruckok.. on new run it might have picked up new mirror.. 15:28
akahat|ruckjob is running now.15:28
rlandyakahat|ruck: just that provider or all of them?15:29
soniyadviroel|lunch, updated the patch15:30
akahat|ruckrlandy, just that provider.. and only one mirror..15:31
rlandyakahat|ruck: it should clear15:32
rlandybesides we don't control that15:32
Tengujm1[m]: fyi, when you hit the ansible-collection-containers-podman package, it's related to https://bugzilla.redhat.com/show_bug.cgi?id=210149515:33
*** marios is now known as marios|out15:48
rlandyakahat|ruck: going to take mom to airport - biab15:51
*** rlandy is now known as rlandy|biab15:51
akahat|ruckrlandy|biab, okay15:51
*** beagles_afk is now known as beagles16:11
*** rlandy|biab is now known as rlandy16:27
*** jpena is now known as jpena|off16:34
*** dviroel|lunch is now known as dviroel16:37
rlandyakahat|ruck: ugh  we got another gate blocker16:51
*** amoralej is now known as amoralej|off16:52
rlandycreating skiplist bug16:57
dviroelrlandy: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/848149 ?17:02
rlandydviroel; ?17:02
dviroelrlandy: same test that was failing in check for me17:03
rlandythis one is failing ipa job17:04
rlandyhttps://bugs.launchpad.net/tripleo/+bug/198025517:04
dviroelhttps://bugs.launchpad.net/tripleo/+bug/192319417:04
dviroelold issue17:04
rlandygoing  to create a skiplist entry17:04
rlandydviroel: overlap?17:04
rlandyblocking gate17:04
rlandystarted today17:05
dviroelrlandy: seems to be an old flaky issue17:05
dviroelrlandy: but looks like started to run this test a couple of days ago17:05
rlandyhttps://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/848149 is not merged17:06
dviroel^ skipping entire class, since it has these 2 jobs only17:06
dviroels/jobs/tets17:06
dviroeltests17:06
rlandydviroel: I think it should be more specific17:06
rlandyonly the second one is failing, right?17:06
dviroelbecause we already skip the first one17:07
rlandywhat was the history on this?17:07
rlandywe had them both skipped?17:07
rlandyand now we have the second one back17:07
rlandyand it's failing?17:08
rlandyso if we merge https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/848149/2/roles/validate-tempest/vars/tempest_skip.yml?17:08
dviroelrlandy: for some reason, some jobs are running the second test now, not sure why17:08
rlandywe'll be convered on both?17:08
dviroellook the standalone for example17:08
dviroelhere from jun 24th: https://31608e4f35b9c13baa68-7dedaad76deccd9ee39b1cd2f9dacaed.ssl.cf1.rackcdn.com/847241/1/gate/tripleo-ci-centos-9-standalone/d62bb9f/logs/undercloud/var/log/tempest/index.html17:08
rlandydviroel: guessing allowlist work17:08
dviroelnow on Jun 27th: https://4f697cb28cb2fb22663a-3f036a988556af1e9af8d4c6bf51b036.ssl.cf2.rackcdn.com/847771/1/check/tripleo-ci-centos-9-standalone/3ea0a7c/logs/undercloud/var/log/tempest/index.html17:08
dviroelrlandy: yes, probably 17:09
dviroel51 vs 52 tests - on standalone17:09
rlandyhttps://bugs.launchpad.net/tripleo/+bug/1923194 marked fixed released17:10
rlandyshould we not add the new bug?17:10
dviroelrlandy: we can - i added a comment on the old bug and related cix17:11
rlandydviroel: question is ...17:12
rlandydo we edit https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/848149/2/roles/validate-tempest/vars/tempest_skip.yml#106717:12
rlandyto just add the second entry?17:12
rlandyand the relevant bug?17:12
rlandyor do we really need both tests included?17:12
dviroeli was ok on skipping the class and maintain the same LP Bug, but if you want to add another LP Bug, we need another entry17:15
rlandydviroel: ^^??17:15
dviroelor add as a comment17:15
rlandyis this good enough?17:15
rlandyit's only referencing one test17:15
rlandyhttps://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/848149/2/roles/validate-tempest/vars/tempest_skip.yml#106717:16
rlandyie: then we should expand it17:16
dviroelrlandy: this change is skipping the entire class test (TestMinimumBasicScenario)17:16
dviroelright?17:16
rlandyif w keep original bug we need to reopen it17:16
rlandyso we can skip whole class and expand the bug ...17:16
rlandysec - let me edit17:17
rlandywill send to you for review17:17
dviroelthe idea on skipping the class is that, since it is a scenario class test, all test will need to do ssh on a floating ip, so all test will fail with the same issue17:17
rlandyright but we are adding a new skip17:18
rlandyie: when did we start running this test again?17:18
rlandywe should not be skipping new issues17:18
rlandylet's chat - easier17:18
rlandyhttps://meet.google.com/jyd-dgdz-vtq?pli=1&authuser=017:18
rlandydviroel: ^^17:19
rlandyhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-standalone&skip=017:30
rlandynot bad17:30
dviroelyeah, not so common on standalone, there is one from today https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ec/847759/4/check/tripleo-ci-centos-9-standalone/8ec02f0/logs/undercloud/var/log/tempest/stestr_results.html17:33
rlandydviroel: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/848167 Add hard_reboot_after_vol_snap_deletion to skiplist17:34
rlandypls review17:34
dviroelack - checking17:37
dviroeldone, lets wait zuul votes17:37
rlandythanks17:38
rlandywe can expand that if needed17:38
dviroelsure17:41
rlandydviroel: w+ing17:45
dviroelk17:45
rlandyto clear gate17:45
*** undefined_ is now known as Guest368417:55
*** Guest3684 is now known as rcastillo_17:55
*** rcastillo_ is now known as rcastillo17:57
*** beagles_ is now known as beagles18:24
* dviroel biab20:30
*** dviroel is now known as dviroel|biab20:30
rlandydviroel|biab: when you are around: https://review.rdoproject.org/r/c/config/+/4379321:16
rlandyrevie wpls21:16
rlandy2022-06-29 21:23:54.529115 | primary | | stack_status_reason   | Resource CREATE failed: OperationalError: resources.baremetal_env.resources.bmc.resources.bmc_other_ports.resources[2]: (pymysql.err.OperationalError) (1213, 'WSREP detected deadlock/conflict and aborted the transaction. Try restarting the transaction')21:25
rlandystill failing21:25
rlandytrying this ...21:27
*** dviroel|biab is now known as dviroel22:09
*** dasm is now known as dasm|off22:15
dasm|offo/22:15
dviroelrlandy: sorry, late 22:16
dviroelworked?22:16
rlandydviroel: nah22:25
rlandyasked rr to ping vexx in the morning22:25
rlandyneed to step out now22:25
*** rlandy is now known as rlandy|bbl22:25
dviroelack22:28
*** dviroel is now known as dviroel|out22:28
* dviroel|out dinner22:29

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!