Wednesday, 2021-05-19

« Tuesday, 2021-05-18 Index Thursday, 2021-05-20 »
*** jmasud has quit IRC00:23
*** jmasud has joined #oooq00:25
*** jmasud has quit IRC01:11
*** saneax has joined #oooq02:57
*** jmasud has joined #oooq03:04
*** saneax has quit IRC03:17
*** skramaja has joined #oooq03:22
*** saneax has joined #oooq03:42
*** skramaja_ has joined #oooq04:17
*** skramaja has quit IRC04:17
*** ykarel has joined #oooq04:24
*** skramaja_ has quit IRC04:40
*** skramaja has joined #oooq04:45
*** saneax has quit IRC04:49
*** saneax has joined #oooq04:51
*** jfrancoa has joined #oooq05:00
*** ratailor has joined #oooq05:00
*** ysandeep|away is now known as ysandeep05:16
*** jpodivin has joined #oooq05:39
*** udesale has joined #oooq05:54
*** marios has joined #oooq05:59
*** slaweq has joined #oooq06:33
*** amoralej|off is now known as amoralej07:20
*** jfrancoa has quit IRC07:22
*** jfrancoa has joined #oooq07:23
*** jmasud has quit IRC07:32
*** tosky has joined #oooq07:38
*** ysandeep is now known as ysandeep|lunch07:40
*** skramaja has quit IRC07:44
*** skramaja has joined #oooq07:44
*** ChanServ has quit IRC07:54
*** ChanServ has joined #oooq07:54
*** services. sets mode: +o ChanServ07:54
*** derekh has joined #oooq07:55
*** jpena|off is now known as jpena07:56
*** jbadiapa has joined #oooq07:57
soniya29arxcruz, kopecmartin, ysandeep|lunch, please add/edit today's tempest agenda https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ08:56
kopecmartinsoniya29: i have a conflict , won't attend , if there are any questions on me i can answer them in the agenda during/after the meeting09:06
*** ysandeep|lunch is now known as ysandeep09:13
ysandeeppojadhav|rover, akahat|ruck Intermittently tempest tests in tripleo-ci-centos-8-containers-multinode-wallaby is failing for me with below error, Is it known?10:45
ysandeepparamiko.ssh_exception.BadHostKeyException: Host key for server '192.168.24.102' does not match: got 'AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZfUVUdvppIyu5pzZtZNK86vKeJ2rA0SiBDNuIRp4DJNv+V1FvWMUMCd9roJ+ZZKLrSaOzm9JR7brZYa6iqRYY=', expected 'AAAAC3NzaC1lZDI1NTE5AAAAIKsMDVTFcuwVHAtDCpLceTDjDwUEoUrpGPJclmzmYngi'10:45
ysandeep10:45
ysandeephttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3e/786619/7/gate/tripleo-ci-centos-8-containers-multinode-wallaby/f3e7f65/logs/undercloud/var/log/tempest/tempest_run.log10:45
ysandeepOther patches also hitting similar intermittently.. https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-containers-multinode-wallaby10:46
pojadhav|roverysandeep, it is not known10:46
ysandeeppojadhav|rover, two more example on different patches:-10:47
ysandeephttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e03/777106/30/check/tripleo-ci-centos-8-containers-multinode-wallaby/e03ba03/logs/undercloud/var/log/tempest/tempest_run.log10:47
ysandeephttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_43f/777108/29/check/tripleo-ci-centos-8-containers-multinode-wallaby/43fb5bb/logs/undercloud/var/log/tempest/tempest_run.log10:47
ysandeeppojadhav|rover, could you please report a bug and investigate.10:47
pojadhav|roverysandeep, if you see all logs having different tests10:48
pojadhav|rovernot the same one10:48
pojadhav|roverwhich is failing10:48
ysandeeppojadhav|rover, you want to chat meet.google.com/vmb-mxky-ube10:49
akahat|ruckysandeep, o/ can you please recheck it once? because job history is good.10:49
pojadhav|roverysandeep, ack10:49
ysandeepakahat|ruck, 3 patches failed on same error.. i think we should report a bug10:49
akahat|ruckysandeep, okay10:53
ysandeepakahat|ruck, thanks! pojadhav|rover and I spoke about this she is writing a bug for this.. akahat++ pojadhav++10:58
pojadhav|roverysandeep, akahat|ruck : https://bugs.launchpad.net/tripleo/+bug/192893311:00
openstackLaunchpad bug 1928933 in tripleo "wallaby : tripleo-ci-centos-8-containers-multinode-wallaby randomly failing tempest tests with paramiko.ssh_exception.BadHostKeyException" [Undecided,New]11:00
akahat|ruckysandeep, pojadhav|rover ack. :)11:00
ysandeeppojadhav|rover, thanks!11:01
akahat|ruckpojadhav|rover, https://bugs.launchpad.net/tripleo/+bug/192893611:18
openstackLaunchpad bug 1928936 in tripleo "Tempest: neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle" [High,New]11:18
pojadhav|roverakahat|ruck, ack11:20
soniya29kopecmartin, okay11:22
*** chem_ has joined #oooq11:26
*** chem has quit IRC11:28
*** jpena is now known as jpena|lunch11:31
*** dviroel|away is now known as dviroel11:45
weshay|ruckysandeep, not sure why your patch keeps failing11:57
weshay|ruckhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/786619/11:57
weshay|ruckhttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-containers-multinode-wallaby11:57
ysandeepweshay|ruck, https://bugs.launchpad.net/tripleo/+bug/192893311:57
openstackLaunchpad bug 1928933 in tripleo "wallaby : tripleo-ci-centos-8-containers-multinode-wallaby randomly failing tempest tests with paramiko.ssh_exception.BadHostKeyException" [Undecided,New]11:57
weshay|ruckit's not a general issue w/ the job11:57
ysandeepweshay|ruck, intermittent issue , you can find more patches failing on BadHostKeyException11:58
weshay|rucksoniya29, ^11:58
weshay|ruckthat trace is scenario manager11:59
soniya29weshay|ruck, let me have a look11:59
weshay|ruckany chance you can help us track down why it's failing intermittently?11:59
*** pojadhav|rover is now known as pojadhav|mtg11:59
soniya29weshay|ruck, it seens an ssh connection failure12:02
weshay|ruckpojadhav|mtg, skip this tempest test in all branches, on all jobs {0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details [359.963192s] ... FAILED12:07
weshay|ruck{0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details [359.963192s] ... FAILED12:07
weshay|rucksoniya29, mark it a promotion blocker, assign it to pcci tempest...12:08
weshay|rucksorry.. pojadhav|mtg ^12:08
pojadhav|mtgweshay|ruck, ack12:08
*** amoralej is now known as amoralej|lunch12:09
weshay|rucksoniya29, any chance we can have some try/except that makes that more reliable? or makes us more confident it's a connectivity issue?12:09
soniya29weshay|ruck, yeah, we can have try/except12:12
soniya29weshay|ruck, i will dig into this issue12:12
weshay|rucksoniya29++12:13
weshay|ruckif ssh claims it can't connect.. it would be nice to see verbose ssh logs.. or a ping..  etc12:13
*** ratailor has quit IRC12:22
*** jpena|lunch is now known as jpena12:22
*** ysandeep is now known as ysandeep|afk12:26
dviroelfolks, i'm going to do a medical exam now, will be back in around 1h .. 1h3012:28
*** dviroel is now known as dviroel|away12:28
*** chem_ has quit IRC12:29
weshay|ruckbhagyashris, akahat|ruck pojadhav|mtg meet.google.com/kyw-bcnr-roi12:29
*** chem_ has joined #oooq12:31
akahat|ruckbhagyashris, ping12:32
akahat|ruckbhagyashris, meeting12:32
*** ysandeep|afk is now known as ysandeep13:01
soniya29weshay|ruck, akahat|ruck, weshay|ruck tempest meeting?13:01
*** amoralej|lunch is now known as amoralej13:04
weshay|ruckysandeep, ok... ready nodes in sf13:31
weshay|ruck[whayutin@localhost tmp]$ cat /tmp/test | wc -l13:31
weshay|ruck4313:31
weshay|ruck:)13:31
ysandeepack o/13:31
weshay|ruckysandeep, ack to proceed to putting 16.2 check.. let's say 3 repos .. then let's attach osp-1313:32
weshay|ruckattack13:32
ysandeepaye o/13:33
*** pojadhav|mtg is now known as pojadhav|rover13:35
arxcruzysandeep: weshay|ruck https://bugzilla.redhat.com/show_bug.cgi?id=195753213:38
openstackbugzilla.redhat.com bug 1957532 in cloud-init "[AWS][cloud-init] From RHEL 82+ cloud-init no longer displays sshd keys fingerprints from instance launched from a backup image" [High,Assigned] - Assigned to eesposit13:38
arxcruzit might be related13:38
*** ysandeep is now known as ysandeep|mtg13:38
weshay|ruckI blame canonical13:39
pojadhav|roverweshay|ruck, ysandeep|mtg : https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/79214913:47
*** tosky has quit IRC13:56
*** tosky has joined #oooq13:59
*** saneax has quit IRC14:01
*** dviroel|away is now known as dviroel14:03
*** skramaja has quit IRC14:09
ysandeep|mtgweshay|ruck, soniya29: fyi.. internal html rendering fixed.. thanks to jpena14:13
soniya29ysandeep|mtg, that's great14:13
soniya29jpena, thank you14:13
ysandeep|mtgweshay|ruck, internal hitting retry_limit https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status14:25
ysandeep|mtgmight be related to nodepool increase14:25
*** ysandeep|mtg is now known as ysandeep14:25
weshay|ruckysandeep, yes14:30
weshay|ruckit might be14:30
* ysandeep pinging rhos-ops14:30
ysandeeppojadhav|rover, wasn't that failing with different error?14:34
pojadhav|roverysandeep, check rabi's comment - https://bugs.launchpad.net/tripleo/+bug/192891614:35
openstackLaunchpad bug 1928869 in tripleo "duplicate for #1928916 featureset001 - mysql fails to start - WSREP: failed to open gcomm backend connection: 131: No address to connect (FATAL)" [High,In progress]14:35
soniya29weshay|ruck, https://review.opendev.org/c/openstack/tempest/+/79217814:36
weshay|rucksoniya29++14:39
ysandeeppojadhav|rover, ack o/14:41
pojadhav|roverysandeep, :)14:41
*** pojadhav|rover is now known as pojadhav|afk14:58
weshay|ruckreview please https://review.opendev.org/c/openstack/tripleo-quickstart/+/79218614:58
ykarelduplicate15:00
ykarelhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/79023915:00
*** jmasud has joined #oooq15:00
weshay|ruckpojadhav|afk, put this review in that test project job https://review.opendev.org/c/openstack/tripleo-common/+/79213615:02
*** jpodivin has quit IRC15:02
weshay|ruckpojadhav|afk, nevermind.. I did it15:03
*** dviroel is now known as dviroel|luch15:18
*** dviroel|luch is now known as dviroel|lunch15:18
ysandeepweshay|ruck, do you have for few mins? need some help with enabling downstream check jobs.15:28
ysandeephttps://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/242568/1//COMMIT_MSG15:28
ysandeepdo you few mins?*15:28
*** ykarel has quit IRC15:29
mariostripleo-ci please add to your reviews thanks https://review.opendev.org/c/openstack/tripleo-repos/+/79212615:45
*** marios has quit IRC15:56
*** ysandeep is now known as ysandeep|away16:03
*** dviroel|lunch is now known as dviroel16:14
*** dviroel is now known as dviroel|away16:20
*** cgoncalves has quit IRC16:32
*** cgoncalves has joined #oooq16:33
*** udesale has quit IRC16:41
*** saneax has joined #oooq16:44
*** cgoncalves has quit IRC16:52
*** cgoncalves has joined #oooq16:53
*** saneax has quit IRC16:58
*** derekh has quit IRC17:02
*** rlandy has joined #oooq17:05
*** jpena is now known as jpena|off17:10
rlandyweshay|ruck: hello17:14
rlandyweshay|ruck: miss anything?17:14
weshay|ruckheh17:14
weshay|ruckrlandy, hi17:14
weshay|ruckwell.. confirmed the issue you and ysandeep|away found re: updating containers17:15
rlandynot updating17:15
weshay|ruckya17:15
rlandyhmmm... do we know why?17:15
weshay|ruckprobably should chat about that17:15
weshay|ruckrlandy, no not yet17:15
weshay|ruckrlandy, centos9 is a lot closer than we thought as well17:15
rlandyweshay|ruck: k - so where do we go first?17:16
rlandyweshay|ruck; what is available for centos 9?17:16
weshay|ruckcomponents I think17:16
weshay|ruckrlandy, nothing.. but it's getting close17:16
rlandyweshay|ruck: k - best place to put my time now??17:16
rlandygetting rhel puppet up and running?17:16
rlandygetting a centos 9 node?17:17
weshay|ruckrlandy, keeping notes here https://docs.google.com/document/d/1ngmliMp_uLS7RYORg4iaui9xgqMAiYWBecKXjTuMe-A/edit#heading=h.9e21hx8yze7p17:17
weshay|ruckrlandy, ya.. we need to get the components fixed17:17
rlandyweshay|ruck: meaning I should look into why components are not updating?17:17
weshay|ruckrlandy, I am as well17:18
rlandyweshay|ruck; I'll put today into that17:18
weshay|ruckthis log is useless atm tripleo-container-image-prepare.log.txt.gz17:18
rlandytomorrow will go back to rhel/centos 917:18
weshay|ruckdoesn't show updates even when there are updated containers17:18
rlandyI think we need to compare the actual process rather than the logs17:18
weshay|ruckrlandy, we're ahead of the game on el917:18
weshay|ruckcan wait until components are fixed imho17:18
weshay|ruckI'm in the code now17:18
rlandyweshay|ruck: we are never ahead of the game :))17:18
weshay|ruckgoing to add some debug17:18
rlandyk - will look at component17:19
rlandyping if you find something17:19
weshay|ruckrlandy, let me show you something though.. or go try that tool I emailed you17:19
weshay|ruckwill save u time17:19
rlandyI saw you email17:19
weshay|ruckk17:19
rlandywill check it out17:19
rlandy - actually wanted to look at the code path17:19
weshay|ruckI think I'm missing a requirements.txt17:20
rlandyrather than the outcome17:20
weshay|ruckrlandy, yes.. but you have to start w/ a component that you now has an udpate17:20
weshay|ruckand then pick the right container17:20
rlandywhich one is a candidate now?17:20
weshay|ruckdon't understand the question17:21
weshay|ruckI'm looking at17:21
weshay|ruckhttps://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-train/dca369f/logs/undercloud/var/log/extra/podman/containers/ovn_metadata_agent/podman_info.log.txt.gz17:21
weshay|ruckpython3-networking-ovn.noarch                  7.4.2-0.20210518193941.4c5cb11.el8        @network17:21
weshay|ruckpython3-networking-ovn-metadata-agent.noarch   7.4.2-0.20210518193941.4c5cb11.el8        @network17:21
weshay|rucknothing useful in https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-train/dca369f/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz17:21
weshay|ruckso.. we have nothing to go on.. afaict.. until we get more info17:22
rlandyweshay|ruck: are we sure upstream components are updating?17:22
weshay|ruckI just proved it w/ that log17:22
rlandyand I assume the patch we merged did nothing?17:22
weshay|ruckya.. didn't fix internal :(17:22
weshay|ruckrlandy, this stuff has changed a lot since we started this w/ Ian17:23
rlandypython3-networking-ovn.noarch                  7.4.2-0.20210518193941.4c5cb11.el8        @network17:23
rlandypython3-networking-ovn-metadata-agent.noarch   7.4.2-0.20210518193941.4c5cb11.el8        @network17:23
rlandyok good17:23
weshay|ruckbut the prepare.log shows NOTHING17:23
rlandyk - let's see where this goes17:23
rlandyafter today - will move on17:24
weshay|ruckrlandy, move on to what17:25
rlandygetting rhel 9 puppet jobs running17:26
rlandyor at least rhel 817:26
rlandyand centos downstream17:26
weshay|ruckrlandy, no.. hold on components w/ me until its fixed17:26
weshay|ruckthis is a major hole17:27
weshay|ruckplease17:27
rlandyk17:29
*** Goneri has joined #oooq17:31
*** Goneri has quit IRC17:35
weshay|ruckrlandy, do you have a test-project handy for any component job?17:39
weshay|ruckrlandy, I'd like to see the results of https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/79221817:39
rlandygetting17:40
rlandyhttps://code.engineering.redhat.com/gerrit/c/testproject/+/189440/101/.zuul.yaml17:40
rlandyweshay|ruck: ^^ feel free to edit/change that17:40
* weshay|ruck see's what's updatable atm17:40
weshay|ruckk17:40
rlandyweshay|ruck: should we expect delorean-current downstream?17:43
weshay|ruckno.. doesn't really make sense downstream17:44
rlandy2021-05-14 17:39:48,753 46880 ERROR tripleo_common.image.image_export [  ] [tripleorhos-16-2/openstack-swift-account] HTTP error: 401 Client Error: Unauthorized for url: https://docker-registry.upshift.redhat.com/v2/tripleorhos-16-2/openstack-swift-account/blobs/sha256:2ae2b76f9673ee54885d944834eab1f6a76da5eb8e55f0beac05d6e63ae11d8017:48
rlandyk - so taking that out of the updates17:48
rlandyalso we have URL access errors17:48
* rlandy tests17:48
rlandyweshay|ruck: ^^ you using that testproject?17:48
rlandynp if yes, I'll create another one17:48
weshay|ruckrlandy, ya.. but I see the 404's in working jobs too17:51
weshay|ruckyes.. I just kicked it17:51
rlandynp - will kick another17:51
*** amoralej is now known as amoralej|off18:02
*** jmasud has quit IRC18:20
*** dviroel|away is now known as dviroel18:26
*** jbadiapa has quit IRC18:30
frenzy_fridayelastic-recheck-query command is suddenly failing for this query https://opendev.org/openstack/tripleo-ci-health-queries/src/branch/master/output/elastic-recheck/1449136.yaml It passed every time till the last patch pushed. The same query is working dorectly on logstash dashboard as well18:49
weshay|ruckfrenzy_friday, if it finds 0 results does that cause an error?18:50
frenzy_fridayNo, I tried a random string , it returned 0 hits18:50
weshay|ruckhrm18:51
frenzy_fridayhttps://0050cb9fd8118437e3e0-3c2a18acb5109e625907972e3aa6a592.ssl.cf5.rackcdn.com/790065/7/check/openstack-tox-py38/4968a73/tox/test_results/1449136.yaml.log this is the log18:51
weshay|ruckso where is it getting <18:51
frenzy_fridayNo clue. This is where it is hitting the exception https://opendev.org/opendev/elastic-recheck/src/branch/master/elastic_recheck/cmd/query.py#L5818:52
*** jmasud has joined #oooq19:08
frenzy_fridayI think in the data that it is returning there is somewhere a list which ER is not expecting. Is the bug 1449136 relevant right now? Or can we revert https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/787569 and get the patch which copies test_results to the node https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/790065 merged?19:10
openstackbug 1449136 in tripleo "Pip fails to find distribution for package" [Critical,Incomplete] https://launchpad.net/bugs/144913619:10
*** jfrancoa has quit IRC19:24
*** jmasud has quit IRC20:02
*** jmasud has joined #oooq20:09
weshay|ruckrlandy, https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/e07b7d7/logs/container-builds/7fbe2db7-7279-4db2-935f-4c405865fb6c/base/ovn-base/20:15
« Tuesday, 2021-05-18 Index Thursday, 2021-05-20 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!