Thursday, 2021-02-04

« Wednesday, 2021-02-03 Index Friday, 2021-02-05 »
rlandy|bblweshay|ruck: containers failed at ironic-conductor - getting there02:28
*** rlandy|bbl has quit IRC02:28
*** apetrich has quit IRC02:29
*** rlandy has joined #oooq03:31
*** rlandy has quit IRC03:35
*** udesale has joined #oooq03:55
*** pojadhav|afk is now known as pojadhav03:57
*** ysandeep|away is now known as ysandeep04:31
*** ysandeep is now known as ysandeep|ruck04:31
*** ykarel has joined #oooq04:49
*** saneax has joined #oooq05:09
*** ratailor has joined #oooq05:16
*** ykarel has quit IRC05:17
*** ykarel has joined #oooq05:18
ysandeep|ruckarxcruz|ruck, fyi.. I have created Ruck/Rover hackmd for this sprint https://hackmd.io/Ta6fdwi2Sme4map8WEiaQA05:18
*** ykarel_ has joined #oooq05:51
*** ykarel has quit IRC05:53
*** ykarel_ is now known as ykarel06:00
*** marios has joined #oooq06:04
*** jmasud has joined #oooq06:40
*** jpodivin has joined #oooq07:10
*** slaweq has joined #oooq07:15
* ysandeep|ruck afk for lunch07:41
*** jpena|off is now known as jpena07:51
*** amoralej|off is now known as amoralej08:07
arxcruz|ruckysandeep|ruck: hey, thanks, i was about to create08:13
arxcruz|rucklet me grab a coffee to wake up :)08:13
*** apetrich has joined #oooq08:16
*** tosky has joined #oooq08:46
*** skramaja has joined #oooq08:54
*** jmasud has quit IRC08:55
ysandeep|ruckarxcruz|ruck, some good news today about master blockers - periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-master - fix is in gate, ipa job got fixed itself(just need to figure out which patch fixed it)09:00
ysandeep|ruckarxcruz|ruck, just need to focus on periodic-tripleo-ci-centos-8-scenario010-standalone-master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master tempest failures - bug https://bugs.launchpad.net/tripleo/+bug/191404309:01
openstackLaunchpad bug 1914043 in tripleo "Octavia TrafficOperationsScenarioTest tests fail or timeout" [Critical,Triaged]09:01
arxcruz|ruckysandeep|ruck: ack09:03
arxcruz|ruckysandeep|ruck: i think i found the problem, give me a few minutes09:13
ysandeep|ruckarxcruz++ ack o/ jfyi.. about https://review.rdoproject.org/r/#/c/31817/ & https://review.opendev.org/c/openstack/tripleo-quickstart/+/77084009:14
*** ykarel is now known as ykarel|lunch09:15
arxcruz|ruckysandeep|ruck: oh, yeah...09:16
arxcruz|ruckysandeep|ruck: this is the fix09:16
ysandeep|ruckarxcruz|ruck, this ^^ merged last night, but today's run again failed09:17
arxcruz|ruckysandeep|ruck: the run you point me doesn't have the test_server_path09:17
arxcruz|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/af2ba17/logs/undercloud/home/zuul/tempest/etc/tempest.conf.txt.gz09:17
ysandeep|rucklet me grab today's run result09:18
ysandeep|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/bb9415c/logs/undercloud/home/zuul/tempest/etc/tempest.conf.txt.gz09:19
ysandeep|ruckit have test_server_path09:19
arxcruz|ruckysandeep|ruck: but the failure is timeout09:21
arxcruz|ruckso, it's now another issue09:21
*** derekh has joined #oooq09:22
ysandeep|ruckarxcruz|ruck, hmm could be, and periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master is failing during setupclass https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/12bec9b/logs/undercloud/var/log/tempest/stestr_results.html.gz09:24
arxcruz|ruckysandeep|ruck: this one doesn't have the fix09:25
arxcruz|ruckysandeep|ruck: it doesn't have the test_server_path09:25
ysandeep|ruckyeah https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/standalone-jobs.yaml#l437 still have tempest_tempestconf_profile_overrides09:30
ysandeep|rucki wonder if chandankumar was trying to fix periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master instead of periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-master with https://review.rdoproject.org/r/#/c/31817/1/zuul.d/standalone-jobs.yaml09:33
ysandeep|ruckbecause i don't see https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-master runs in pipeline09:34
ysandeep|ruckarxcruz|ruck, I am pushing similiar changes(https://review.rdoproject.org/r/#/c/31817) for periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master, will test with testproject and see how it goes09:37
arxcruz|ruckysandeep|ruck: ack10:07
*** ykarel|lunch is now known as ykarel10:16
*** udesale_ has joined #oooq10:32
*** udesale has quit IRC10:35
chandankumararxcruz|ruck: you taken care of sc10 ovn job?10:41
arxcruz|ruckchandankumar: i think ysandeep|ruck is working on that, if not i can do it10:42
arxcruz|ruckysandeep|ruck: ^10:42
ysandeep|ruckchandankumar, https://review.rdoproject.org/r/31832 testing here https://review.rdoproject.org/r/#/c/28458/10:43
*** jbadiapa has joined #oooq10:47
chandankumarneed to revert previous one also10:50
*** jmasud has joined #oooq10:54
*** dtantsur|afk is now known as dtantsur11:01
ysandeep|ruckLooks like we have some mirror/network issue11:14
ysandeep|ruck~~~11:14
ysandeep|ruckDEBUG: Errors during downloading metadata for repository 'delorean-current':11:14
ysandeep|ruckDEBUG:   - Curl error (7): Couldn't connect to server for http://mirror.regionone.limestone.opendev.org:8080/rdo/centos8-train/component/tripleo/9a/f0/9af0a189edcea47240e1e6ae92dcc57a87ac36cc_2f2cdb5c/repodata/repomd.xml [Failed to connect to mirror.regionone.limestone.opendev.org port 8080: No route to host]11:14
ysandeep|ruckDEBUG: Error: Failed to download metadata for repo 'delorean-current': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried11:14
ysandeep|ruck~~~11:14
ysandeep|ruckarxcruz|ruck, fyi.. https://bugs.launchpad.net/tripleo/+bug/1914585 I have pinged on #opendev for someone from infra to look11:32
openstackLaunchpad bug 1914585 in tripleo "Content provider jobs failed after failing to connect to mirrors, Failed to connect to mirror.regionone.limestone.opendev.org port 8080: No route to host" [High,Triaged]11:32
arxcruz|ruckysandeep|ruck: maybe is our proxy11:43
arxcruz|ruckysandeep|ruck: mostly failures seems to be from limestone11:46
arxcruz|ruckhttp://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-8-content-provider&var-influxdb_filter=passed%7C%3D%7CFalse11:46
ysandeep|rucktrue seems like issue with limestone11:52
*** sshnaidm|afk is now known as sshnaidm11:53
ysandeep|ruckarxcruz|ruck, chandankumar With patch - https://review.rdoproject.org/r/31832 testproject https://review.rdoproject.org/r/#/c/28458/ passed the earlier issue where it was failing on Setupclass11:59
ysandeep|ruckFailing on different tempest test now - https://logserver.rdoproject.org/58/28458/29/check/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/f10ba43/logs/undercloud/var/log/tempest/stestr_results.html.gz11:59
arxcruz|ruckysandeep|ruck: let's open a bug, and put it on skiplist12:02
arxcruz|ruckwell12:03
ysandeep|ruckarxcruz|ruck, I am on it o/ writing a bug for new failure12:04
arxcruz|ruckysandeep|ruck: i'm pretty sure this will repeat, and it's network issue, the door isn't oppened12:10
arxcruz|ruckhowever12:10
arxcruz|rucki don't have 100%12:11
ysandeep|ruckarxcruz|ruck, fyi.. i faced something similiar in my last ruck/rover with limestone mirror, infra had to reboot the mirror https://bugs.launchpad.net/tripleo/+bug/190114212:19
openstackLaunchpad bug 1901142 in tripleo "Check/gate jobs failing on task "configure-mirrors : Update yum/dnf cache" with Error: Failed to download metadata for repo 'AppStream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried" [Critical,Invalid] - Assigned to Sandeep Yadav (sandeepyadav93)12:19
arxcruz|ruckysandeep|ruck: which infra? ours or opendev?12:20
ysandeep|ruckarxcruz|ruck, opendev12:20
ysandeep|ruckchandankumar, sshnaidm could you please review https://review.rdoproject.org/r/#/c/31832/, we are hitting a new tempest test failure for which we are opening a new launchpad bug.12:21
sshnaidmysandeep|ruck, is settings now similar to what we have in upstream check job?12:27
ysandeep|rucksshnaidm, upstream job is failing too, I will post a similiar patch for upstream sc10-ovn job definition12:28
* ysandeep|ruck doing it right away 12:29
*** jpena is now known as jpena|lunch12:30
*** jmasud has quit IRC12:32
*** rlandy has joined #oooq12:34
ysandeep|rucksshnaidm, fyi.. https://review.opendev.org/c/openstack/tripleo-ci/+/77407812:35
weshay|rucksshnaidm, arxcruz|ruck ysandeep|ruck still want to sync or skip?12:35
weshay|rucksorry.. I'm late.. was hard to wake up today12:35
arxcruz|ruckweshay|ruck: ysandeep|ruck sshnaidm we synced yesterday, but if you have something we can chat12:36
weshay|ruckwe can skip12:36
arxcruz|ruckysandeep|ruck: did you create the bug? I can work on that if you want12:36
weshay|ruckysandeep|ruck, any questions about 16.2?12:36
weshay|ruckarxcruz|ruck, no.. that was just a review that Alex put up12:37
ysandeep|ruckweshay|ruck, nah, I saw your email.. i am good12:37
arxcruz|ruckweshay|ruck: ??12:37
weshay|ruckarxcruz|ruck, ya.. looks better https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-oooq12:38
weshay|ruckarxcruz|ruck, thanks12:38
ysandeep|ruckarxcruz|ruck, https://bugs.launchpad.net/tripleo/+bug/1914600 , fyi.. i noticed the failing test was not triggering earlier , i didn't see it ran on last green run https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/7e653b8/logs/undercloud/var/log/tempest/stestr_results.html.gz12:41
openstackLaunchpad bug 1914600 in tripleo "periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master is failing on tempest test with Exception: Server 192.168.24.122 on port 60092/8085 did not begin passing traffic within the timeout period." [Critical,Triaged]12:41
*** ratailor has quit IRC12:42
arxcruz|ruckysandeep|ruck: oh, that's interesting12:43
arxcruz|ruckthe TrafficOperationsScenarioTest isn't running on this12:43
arxcruz|ruckand mostly of the HealthMonitorScenarioTest are being skipped12:43
arxcruz|rucklet me check if that's a new test12:43
weshay|ruckI wonder if it makes sense to run all these tests for multinode-ipa12:44
weshay|ruckysandeep|ruck, last one tripped up on tempest https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master/638b627/logs/undercloud/var/log/tempest/stestr_results.html.gz12:44
arxcruz|ruckysandeep|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/7e653b8/logs/undercloud/home/zuul/tempest/etc/tempest_blacklist.txt.gz12:45
arxcruz|ruckthe test was in in the blacklist12:45
arxcruz|rucknow it's not anymore12:46
arxcruz|rucklet me see12:46
ysandeep|ruckweshay|ruck, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master it was running good for last 2 days, may be transient issue.. i will retrigger with test project12:46
arxcruz|ruckysandeep|ruck: https://opendev.org/openstack/openstack-tempest-skiplist/commit/ecb8c966af1fbc544fd69d3b7b185ea807713a9112:47
arxcruz|ruckso, it was removed recently12:47
arxcruz|ruckhowever, now still failing, not because of ssh timeout, but the port12:47
arxcruz|ruckysandeep|ruck: i'm re-adding it to skiplist12:47
arxcruz|ruckand pointing to the new bug you create12:47
ysandeep|ruckarxcruz|ruck, ack o/12:48
weshay|ruckbhagyashris, rlandy  I have a recruiting call for the first few of our planning mtg.. will be latest12:48
weshay|rucklate12:48
rlandyweshay|ruck: ack - ok12:49
ysandeep|ruckarxcruz|ruck, so far i have just added alert flag on that bug, please feel free to add other tags if necessary12:49
weshay|ruckysandeep|ruck, shall we close https://bugs.launchpad.net/tripleo/+bug/191404612:49
openstackLaunchpad bug 1914046 in tripleo "periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master fails on certificate validation" [Critical,Triaged]12:49
weshay|ruck?12:49
bhagyashrisweshay|ruck, ok ack12:49
ysandeep|ruckweshay|ruck, yes we can close, issue is fixed.12:50
weshay|ruck++12:50
ysandeep|ruckweshay|ruck, i have marked fix_released12:51
arxcruz|ruckysandeep|ruck: weshay|ruck chandankumar https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/77407912:52
arxcruz|ruckchandankumar: we should add the whole team as core of tempest-skiplist12:52
chandankumararxcruz|ruck: I think all of them have access if they are part of tripleo-ci group12:53
arxcruz|ruckysandeep|ruck: can you check if you can +2 the patch above?12:53
ysandeep|ruckarxcruz|ruck, i don't have +2 rights :)12:54
arxcruz|ruckchandankumar: ^12:54
chandankumarysandeep|ruck: ok, will add that12:54
chandankumararxcruz|ruck: done, ysandeep|ruck can you check now?12:58
ysandeep|ruckchandankumar, still same for me , I can only see +1 right there12:59
chandankumarysandeep|ruck: https://review.opendev.org/admin/groups/707b4c1ad2da0212c0fc33a7e12c4b9641761ac7,members13:00
arxcruz|ruckysandeep|ruck: maybe a refresh or re-login?13:02
bhagyashrisakahat, chandankumar rlandy pojadhav zbr ysandeep|ruck arxcruz|ruck13:02
bhagyashrisplanning meeting13:02
* ysandeep|ruck tried refresh, trying relogin now13:02
*** amoralej is now known as amoralej|lunch13:03
arxcruz|ruckchandankumar: weshay|ruck i'll +w https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/774079 because is blocking us :)13:05
chandankumararxcruz|ruck: ok13:05
ysandeep|ruckchandankumar, still same for me https://paste.opensuse.org/7956983813:05
ysandeep|ruckrelogin didn't help as well13:05
arxcruz|ruckzbr: meeting13:06
*** dtantsur is now known as dtantsur|brb13:14
*** rlandy is now known as rlandy|meeting13:15
*** jpena|lunch is now known as jpena13:33
chandankumarmarios: https://code.engineering.redhat.com/gerrit/#/c/226624/ to enable codesearch13:36
marioschandankumar: thank you13:36
chandankumarmarios: it will enable tripleo-environments13:40
matburlandy|meeting: sshnaidm: marios: weshay|ruck: if folks you can take a quick look on this patch and A+ if you are happy with it, it will be awesome: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/75538213:52
matbuthx13:52
mariosmatbu: ack in call but added to my reviews list for next run13:58
*** amoralej|lunch is now known as amoralej13:59
matbumarios: ack thx :)14:04
*** dtantsur|brb is now known as dtantsur14:16
sshnaidmchandankumar, this commit is only in devel branch https://github.com/ansible/ansible/commit/7f0eb7ad799e531a8fbe5cc4f46046a4b1aeb09314:53
sshnaidmnot sure it will be backported to some of 2.* releases :(14:54
chandankumarsshnaidm: depends on what version of ansible is shipped by stream14:54
sshnaidmchandankumar, it depends on ansible version that we use in ci jobs with tripleo-quickstart14:55
sshnaidmcurrently we still use 2.9 for example14:55
ysandeep|ruckarxcruz|ruck, good news periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master passed with our patches, I gave a rerun to periodic-tripleo-ci-centos-8-scenario010-standalone-master as well, lets see how it goes.14:56
arxcruz|ruckysandeep|ruck: http://www.quickmeme.com/img/7f/7ff8c2a1710d7844de557db96b526541ab82c39337b851882e9a177006813f58.jpg14:58
arxcruz|ruckops14:58
arxcruz|ruckwrong meme14:58
arxcruz|rucklol14:58
arxcruz|ruckysandeep|ruck: http://www.quickmeme.com/img/ce/ce68c60328bf7a4cf98700b75e40160cd64f8aba1d070a541a8b295849914522.jpg14:59
ysandeep|ruck:D14:59
weshay|ruckmatbu, do you wf?15:02
*** derekh has quit IRC15:05
*** rlandy|meeting is now known as rlandy15:07
weshay|ruckarxcruz|ruck, any idea what's happening here?15:11
weshay|ruckatal: [undercloud]: FAILED! => {15:11
weshay|ruck2021-02-03 18:21:40.874204 | primary |     "changed": true,15:11
weshay|ruck2021-02-03 18:21:40.874215 | primary |     "cmd": "set -e\nif [ -d /openstack/venvs/tempestconf-untagged/bin ]; then\n. /openstack/venvs/tempestconf-untagged/bin/activate\nfi\ndiscover-tempest-config --profile \"/home/zuul/tempest/etc/profile.yaml\"\n",15:11
weshay|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-rhos-17/9decc84/job-output.txt15:11
*** amoralej is now known as amoralej|dojo15:14
matbuweshay|ruck: sorry ?15:15
weshay|ruckmatbu, you have 3 +2's on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/75538215:17
weshay|ruckmatbu, you can wf.. you are core15:17
matbuweshay|ruck: ha yep :) well if CI folks don't want to wf it, i dont want to propagate nightmare :)15:31
weshay|ruckmatbu, k.. sagi rechecked fs001 it's in deploy now15:32
* weshay|ruck waits to see results and then I'll wf15:32
matbuweshay|ruck: yep i saw, thank you15:32
arxcruz|ruckweshay|ruck: it seems issue with the glance15:32
arxcruz|ruckweshay|ruck: is this standalone have glance available?15:33
weshay|ruckarxcruz|ruck, k.. so I think we still have a bug/card but the error looks diff..15:33
* weshay|ruck looks at bug15:33
*** skramaja has quit IRC15:33
weshay|ruckarxcruz|ruck, https://bugzilla.redhat.com/show_bug.cgi?id=192378915:35
openstackbugzilla.redhat.com bug 1923789 in openstack-cinder "RBD _get_pool_stats fails on Ceph Pacific" [Urgent,Assigned] - Assigned to eharney15:35
weshay|ruckwe'll wait on that15:35
*** derekh has joined #oooq15:36
ysandeep|ruckarxcruz|ruck, https://logserver.rdoproject.org/58/28458/30/check/periodic-tripleo-ci-centos-8-scenario010-standalone-master/bb7ae0d/logs/undercloud/var/log/tempest/stestr_results.html.gz failed :(15:40
*** udesale_ has quit IRC15:41
weshay|ruckysandeep|ruck, can you please add the ovb jobs to the osp-17 integration line.. and component jobs.. for bm, tripleo .. match 16.2 line15:48
*** sshnaidm is now known as sshnaidm|afk15:48
ysandeep|ruckweshay|ruck, sure will do, okay if i knock this tomorrow morning, getting late for me15:49
weshay|ruckysandeep|ruck, aye.. thanks15:50
ysandeep|ruckweshay|ruck, thanks, 16.2 today's run going good so far, I hope we can promote today :)15:51
ysandeep|ruckbefore we switch to 8.415:52
weshay|ruckysandeep|ruck++15:52
arxcruz|ruckysandeep|ruck: need a recheck15:53
ysandeep|ruckarxcruz|ruck, will do15:53
arxcruz|ruckysandeep|ruck: just did it15:53
ysandeep|ruckarxcruz|ruck, ack thanks!15:54
ysandeep|ruckweshay|ruck, fyi.. before i leave we faced issue intermittent issue with limestone mirror in the morning, things look better now, But i have pinged opendev folks and fungi is checking16:13
ysandeep|ruckhttp://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&var-influxdb_filter=cloud%7C%3D%7Climestone&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-8-content-provider&from=now-24h&to=now16:13
ysandeep|ruckhttps://bugs.launchpad.net/tripleo/+bug/1914585 bz for this ^^ , please buzz on opendev if issue reappers, they are checking in the meantime..16:14
openstackLaunchpad bug 1914585 in tripleo "Content provider jobs failed after failing to connect to mirrors, Failed to connect to mirror.regionone.limestone.opendev.org port 8080: No route to host" [High,Triaged]16:14
*** ysandeep|ruck is now known as ysandeep|away16:15
ysandeep|awayarxcruz|ruck, see you tomorrow o/16:15
arxcruz|ruckysandeep|away: namaste16:16
weshay|ruckysandeep|away, thanks!16:16
rlandyzbr: ever seen this on dnf package?16:20
rlandy2021-02-03 23:40:20.714 | >   File "/usr/lib/python3.6/site-packages/dnf/cli/commands/mark.py", line 92, in run16:20
rlandy2021-02-03 23:40:20.802 | >     rpmdb_version = self.sack._rpmdb_version()16:20
rlandy2021-02-03 23:40:20.802 | > AttributeError: 'MarkCommand' object has no attribute 'sack'16:20
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/18/225518/13/check/periodic-tripleo-rhel-8-4-buildimage-ironic-python-agent-rhos-16.2/b202ea0/logs/build.log.txt.gz16:21
zbrnope16:22
weshay|ruckmatbu, wf :)16:24
*** rascasoft has joined #oooq16:26
*** jpodivin has quit IRC16:29
*** ykarel has quit IRC16:33
*** ykarel has joined #oooq16:33
*** rascasoft has quit IRC16:48
matbuweshay|ruck: thanks16:53
*** jpena is now known as jpena|off17:01
*** marios is now known as marios|out17:15
*** openstack has joined #oooq17:23
*** ChanServ sets mode: +o openstack17:23
*** marios|out has quit IRC17:28
*** ykarel has quit IRC17:28
*** amoralej|dojo is now known as amoralej|off17:42
*** derekh has quit IRC18:00
rlandyysandeep|away: we have an issue with fs035 on 16.2 - will leave notes18:00
*** dtantsur is now known as dtantsur|afk18:12
rlandyweshay|ruck: ^^ pls don't promote 16.2 ... genuine failure going down on fs03518:24
rlandyinvestigating18:25
weshay|ruckrlandy, k.. ssl related?18:25
rlandy2021-02-04 12:56:53 | 2021-02-04 12:55:06.567785 | fa163eb4-684a-9799-ce09-0000000045ea |      FATAL | Check Keystone user assignment to roles status | undercloud | item=heat_stack_domain_admin | error={"ansible_job_id": "294953839651.318483", "ansible_loop_var": "tripleo_keystone_resources_user_role_async_result_item", "attempts": 30, "changed": false, "finished": 0, "started": 1, "tripleo_keystone_resources_user_role_async_resul18:25
rlandyt_item": {"ansible_job_id": "294953839651.318483", "ansible_loop_var": "tripleo_keystone_resources_data_user", "changed": true, "failed": false, "finished": 0, "results_file": "/root/.ansible_async/294953839651.318483", "started": 1, "tripleo_keystone_resources_data_user": {"heat_stack_domain_admin": {"domain": "heat_stack"}}}}18:25
rlandyweshay|ruck: ack - how did you know???!!18:25
rlandyseen upstream?18:25
weshay|rucklet's see.. haven't looked18:25
* rlandy gets logs18:25
weshay|ruckbut ssl is what that job should be testing18:26
weshay|ruckso just guessing18:26
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset035-internal-rhos-16.2/6a03a96/18:26
rlandyI testproject'ed that job yesterday18:26
rlandychecking there18:26
weshay|ruckhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable3%20&job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train%0918:26
weshay|rucknothing here really18:26
rlandy2021-02-03 19:35:10 | 2021-02-03 19:32:00.483037 | fa163e63-5545-f6cf-4468-0000000046e8 |      FATAL | Check Keystone user assignment to roles status | undercloud | item=heat_stack_domain_admin | error={"ansible_job_id": "39160589387.324840", "ansible_loop_var": "tripleo_keystone_resources_user_role_async_result_item", "attempts": 30, "changed": false, "finished": 0, "started": 1, "tripleo_keystone_resources_user_role_async_result18:27
rlandy_item": {"ansible_job_id": "39160589387.324840", "ansible_loop_var": "tripleo_keystone_resources_data_user", "changed": true, "failed": false, "finished": 0, "results_file": "/root/.ansible_async/39160589387.324840", "started": 1, "tripleo_keystone_resources_data_user": {"heat_stack_domain_admin": {"domain": "heat_stack"}}}}18:27
rlandysame in both18:27
rlandythe rest should pass18:27
*** amoralej|off is now known as amoralej18:27
weshay|ruckrlandy, you sure this isn't ironic18:30
rlandyfs001 just failed18:30
rlandychecking that one18:30
weshay|ruck nova.exception.InstanceDeployFailure: Failed to provision instance 0693d6fd-b1ef-4ea6-9ad0-8f4f8521ac54: Deploy failed for instance 0693d6fd-b1ef-4ea6-9ad0-8f4f8521ac54. Error: iSCSI connection was not seen by the file system after attempting to verify 3 times.18:30
weshay|ruckI wonder if that should be using metalsmith18:31
weshay|rucklet's look at 00118:31
rlandyin train?18:31
weshay|ruckoh right18:32
weshay|ruck ironic_inspector_client.v1.WaitTimeoutError: Timeout while waiting for introspection of nodes ['ab343d5d-57d2-41db-be17-41afe19e14da']18:34
weshay|ruckin 00118:34
weshay|ruck2021-02-04 10:38:57.152 ERROR /var/log/containers/neutron/dhcp-agent.log: 104161 ERROR neutron.agent.linux.ip_lib [req-e4db46f4-dfbe-4940-9de5-0d8c1d733bcd - - - - -] Device tap246156c1-9b cannot be used as it has no MAC address18:34
rlandyweshay|ruck: it's ok - will BZ it if need be18:34
*** jmasud_ has joined #oooq18:51
rlandyerror={"attempts": 30, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}18:51
*** jmasud has quit IRC18:51
rlandyFATAL | Check Keystone user status | undercloud | e18:51
rlandyboth are keystone user related18:52
rlandyperiodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-16.2openstack/tripleo-cimasteropenstack-periodic-integration-rhos-16.2master4 hrs 13 mins 39 secs2021-02-04 14:25:24SUCCESS18:52
rlandyand fs020 passes18:52
rlandylol - how is that possible?18:52
rlandyweshay|ruck: ^^ these test never cease to amaze me18:53
weshay|ruckrlandy, perhaps the 1ctrl vs. 3?18:53
rlandypacemaker?18:54
rlandyugh18:54
rlandynevermind18:54
weshay|ruckshould be the same as fs001, but it's worth revalidating that18:54
rlandyweshay|ruck: so question on container builds - if you have a sec18:54
rlandyactually probably easier to explain in person ...18:55
rlandyweshay|ruck: https://meet.google.com/hea-gzdg-fbz18:56
weshay|ruckrlandy, talking to sagi18:58
rlandyweshay|ruck: k - no rush18:58
*** jmasud_ has quit IRC19:08
weshay|ruckrlandy, k.. trying to join19:24
rlandysec - let me rejoin19:24
*** jmasud has joined #oooq19:33
*** jmasud has quit IRC19:54
*** radez has quit IRC21:38
*** amoralej has quit IRC21:39
*** jbadiapa has quit IRC21:57
*** slaweq has quit IRC23:03
*** rlandy has quit IRC23:15
*** jmasud has joined #oooq23:39
« Wednesday, 2021-02-03 Index Friday, 2021-02-05 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!