Monday, 2021-12-13

*** ysandeep|out is now known as ysandeep04:20
*** frenzy_friday is now known as anbanerj|ruck04:35
bhagyashris_folks kindly add into your review list https://review.rdoproject.org/r/c/rdo-jobs/+/36994 thanks :)06:21
*** bhagyashris_ is now known as bhagyashris06:21
*** ysandeep is now known as ysandeep|intv06:25
jfrancoaysandeep|intv: good morning! Can I ask you for a favor? I am still trying to find out why is this DISK_FULL thing happening, could you please put the node from the latest run of this patch https://review.rdoproject.org/r/c/testproject/+/36859 on hold and add my keys https://github.com/jfrancoa.keys ?07:13
jfrancoaysandeep|intv: thanks a lot!07:14
ysandeep|intvjfrancoa, i am in a interview, but can do tht in ~45 mins07:18
jfrancoaysandeep|intv: sure, when you'll have the time. thanks!07:18
*** abregman is now known as abregman|mtg08:39
*** ysandeep|intv is now known as ysandeep08:51
ysandeepjfrancoa: added on hold, requested infra to add your keys.08:52
jfrancoaysandeep: thanks a lot Sandeep!08:52
anbanerj|ruckhey marios, for card https://trello.com/c/aJQoSsxi/2260-cixlp1953742tripleociproa-weirdo-master-centos8-promote-packstack-scenario002-is-failing-the-synchronize-task-no-route-to-host should I remove the job from criteria ? (https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/37164)08:55
anbanerj|ruck^ not anything urgent, pls lemme know when you get some time :)08:55
*** ysandeep is now known as ysandeep|lunch08:58
mariosanbanerj|ruck: no i don't think so 09:02
mariosanbanerj|ruck: that is a long running card/issue and rlandy wanted to raise it is why we have that cix09:02
mariosanbanerj|ruck: cos it wasn't getting attention09:02
anbanerj|ruckmarios, oh, ok, got it. thanks09:02
*** ysandeep|lunch is now known as ysandeep09:21
*** dviroel|out is now known as dviroel|rover10:48
dviroel|rovermorning10:48
dviroel|roveranbanerj|ruck: hey o/11:01
dviroel|roveranbanerj|ruck: see that you are running some tp's11:01
anbanerj|ruckdviroel|rover, Hey 0/ morning!11:01
anbanerj|ruckdviroel|rover, yep, ussuri and victoria. I didnt check the downstream bugs yet. Gate is good today11:02
anbanerj|ruckUpdated a few cards in cix11:02
dviroel|roveranbanerj|ruck: ack, will go through the cards in a few too11:02
dviroel|roverhttps://review.opendev.org/c/openstack/tripleo-heat-templates/+/821264/ is in gates again11:03
anbanerj|ruckyes, i'll kich the ussuri tp after ^ merges11:05
*** rlandy is now known as rlandy|ruck11:10
rlandy|ruckanbanerj|ruck: dviroel|rover: hello11:13
rlandy|ruckanbanerj|ruck: dviroel|rover: still fighting ussuri11:13
rlandy|ruckysandeep: bhagyashris: hey - did you get the invite to the planning workshop11:14
anbanerj|ruckrlandy|ruck, hey. yes ussuri had 2 failed jobs - I pasted the reasons on tripleo-ci gchat space. Waiting for ^ to merge to kich the testproj11:14
rlandy|ruckanbanerj|ruck: k - we could probably start with fs0111:15
anbanerj|ruckrlandy|ruck, ack, updating11:17
dviroel|roverussuri is now on hash 0c303674a982c099b14bcb4198325ec711:17
dviroel|rovernot sure why didn't promote the previous hash11:17
rlandy|ruckchandankumar: ysandeep: bhagyashris: soniya29: can we move the planning meeting half an hour earlier?11:17
chandankumarrlandy|ruck: sure11:17
soniya29rlandy|ruck, sure11:20
*** jpena|off is now known as jpena11:27
rlandy|ruckanbanerj|ruck: dviroel|rover: let's sync .... https://meet.google.com/gzv-crgo-utx?pli=1&authuser=011:30
anbanerj|ruckrlandy|ruck, joining11:30
rlandy|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-stable4-centos7/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-train/fd3debc/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz11:34
rlandy|ruckcentos-binary-qdrouterd11:34
rlandy|ruckchandankumar: ysandeep: bhagyashris: soniya29: ... bhagyashris owns that meeting - I can't move it - so let's just meet 30 mins earlier11:47
chandankumarrlandy|ruck: ok11:56
*** dpawlik7 is now known as dpawlik11:56
rlandy|ruckakahat: hello12:00
rlandy|ruckakahat: let's sync re: scenario01012:01
akahatrlandy|ruck, hello o/12:06
akahatrlandy|ruck, sure. https://meet.google.com/xnv-vopt-coa12:07
rlandy|ruckakahat: joining 12:07
rlandy|ruckakahat: can you hear me>12:08
mariosbhagyashris: o/ when is the planning meeting is it today? I don't see it on calendar12:14
bhagyashrismarios, no we don't have planning meeting today rlandy|ruck was talking about UA/TC/PM sync12:15
mariosbhagyashris: ah k thanks :)12:20
chandankumarmarios: I saw you rechecked fs01 testproject, let me know if you hit this issue https://bugs.launchpad.net/tripleo/+bug/1954456 see on fs0212:28
chandankumar*seen12:28
marioschandankumar: yes... so i was staring at that for a while and cant work out why we don't get the right version 12:31
marioschandankumar: we have has the right version ansible-role-metalsmith-deployment-1.6.1-0.20211202211701.81d820f.el9.noarch.rpm12:32
marioschandankumar:  "current-tripleo has right version"       * https://trunk.rdoproject.org/centos9-master/current-tripleo/delorean.repo12:32
amoralejhi, i see some jobs in one of our gates has failed with timeout downloading from trunk.registry.rdoproject.org12:37
amoralejWrite Failure: HTTPSConnectionPool(host='trunk.registry.rdoproject.org', port=443): Read timed out.12:37
amoralejis this a known issue?12:37
chandankumarmarios: as per steve his looks suspicious "Warning: /dev/disk/by-uuid/cedbc2ac-f3e9-4607-a15e-5e5f3b7aa817 does not exist" , he will look into the logs12:42
marioschandankumar: thank you12:43
rlandy|ruckysandeep: bhagyashris: di dyou both get the invite for planning workshop today and tomorrow?13:04
rlandy|ruckysandeep: bhagyashris; I will be joining and dropping per other meetings13:04
ysandeeprlandy|ruck, nope i didn't get any invite13:04
rlandy|ruckysandeep: bhagyashris: sending13:05
ysandeeprlandy|ruck, thanks, fyi.. bhagyashris is out sick today. She just joined for planning sync13:05
rlandy|ruckysandeep: sent13:06
ysandeeprlandy|ruck, joined 13:06
rlandy|ruckysandeep: feel free to join and drop as time permits13:06
*** dviroel is now known as dviroel|rover13:08
rlandy|ruckakahat: let me know where you leave off with scenario010 at your EoD - will pick up then13:10
rlandy|ruckdviroel|rover: anbanerj|ruck; rdo zuul just got restarted13:18
rlandy|ruckdviroel|rover: anbanerj|ruck: requesting to restart downstream as well13:18
anbanerj|ruckack13:18
akahatrlandy|ruck, ack13:21
dviroel|roverrlandy|ruck: ack, tks13:23
rlandy|ruckdviroel|rover: anbanerj|ruck: pls remove that from criteria (comment out) and let's promote master13:44
anbanerj|ruckrlandy|ruck, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3716413:44
rlandy|ruckanbanerj|ruck: ^^ pls add the bug per dviroel|rover's comment13:50
rlandy|ruckthen we will ping ykarel - per meeting we are cleared to remove this13:50
dviroel|roveryeah, it is in commit message, but I think that might be better to add in the file itself13:51
rlandy|ruckdviroel|rover: ack13:51
rlandy|ruckso we know why it's commented out13:51
ykarelrlandy|ruck, amoralej is investigating it, promoting without it can cause issues(if issue is related to openstack package)13:52
ykarelso better to wait for his results13:52
anbanerj|ruckdviroel|rover, rlandy|ruck ack, added a ref to the bug in the file. thanks13:53
rlandy|ruckI hear - but broken since Nov 20 th without action13:53
rlandy|ruckysandeep: fyi - from CIX call - going to downgrade ovn for 16.2 an d1713:55
ysandeeprlandy|ruck, ack13:56
akahatysandeep, downstream zuul is working for you? I'm seeing "Something went wrong" on status page.13:59
rlandy|ruckakahat: it's being restarted14:01
anbanerj|ruckvictoria testP passed: https://review.rdoproject.org/r/c/testproject/+/37162 - should promote now14:01
rlandy|ruckpls see above14:01
rlandy|ruckanbanerj|ruck: great - thanks14:01
ysandeeprlandy|ruck, akahat dviroel|rover scrum time14:01
akahatokay.14:01
dviroel|roverack14:01
ysandeepchandankumar, scrum time14:02
marioshttps://bugs.launchpad.net/tripleo/+bug/195257314:03
marios"using 1.5.1 "        * python3-metalsmith-1.5.1-0.20210922085423.2a39acc.el9.noarch14:04
marios* https://logserver.rdoproject.org/67/36267/39/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/52147bd/logs/undercloud/var/log/extra/rpm-list.txt.gz14:04
ysandeephttps://bugs.launchpad.net/tripleo/+bug/195445614:04
rlandy|ruckhttps://trunk.rdoproject.org/centos9-master/report.html14:05
ysandeephttps://review.rdoproject.org/r/c/rdo-jobs/+/3699414:11
dviroel|roverhere https://review.rdoproject.org/r/c/testproject/+/3697614:19
dviroel|roverthe depends-on already merged - last run was Dec 10th14:19
amoralejykarel, rlandy|ruck it seems a problem with ordering, rabbitmq is not started and job keeps waiting the start of neutron-server 14:23
rlandy|ruckhttps://review.opendev.org/c/openinfra/python-tempestconf/+/82016914:27
ykarelamoralej, ahhk14:28
ykarelamoralej, may be caused by https://github.com/openstack/puppet-neutron/commit/feba1ff2ee1c6d33dc735a44be8121f78f64ee6e ?14:28
ykarellet me also check c9 testpatch if dates match14:29
amoralejprobably14:30
ykarelamoralej, yes dates matches with failure in https://review.rdoproject.org/r/c/rdo-jobs/+/36753 so likely that caused the issue14:31
rlandy|ruckamoralej: ok - if this is fixable, we will wait on your fix14:32
*** ysandeep is now known as ysandeep|dinner14:34
rlandy|ruckakahat: so your idea didn;t work out?14:34
akahatrlandy|ruck, yup. It didn't14:35
rlandy|ruckakahat: so ysandeep|dinner is out atm14:35
rlandy|ruckakahat: can you post your patch link14:35
* rlandy|ruck will look14:36
akahatrlandy|ruck, okay. we can have call after 2 hours.14:36
akahatthis is patch: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/29612314:36
akahatTestproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/20029514:36
rlandy|ruckakahat: ok - can see what the issue is here14:37
rlandy|ruckakahat; parenting doesn't override required projects - adds to them14:38
akahatoh..14:38
akahatfixing. 14:38
rlandy|ruckakahat: you'd need a complete separate parent14:39
rlandy|ruckso name: tripleo-ci-base-required-projects14:39
rlandy|ruck^^ needs to stay14:39
rlandy|ruckthe common one can be its parent14:39
rlandy|ruckand a second parent14:39
akahatfixing. 14:43
*** dviroel|rover is now known as dviroel|rover|lunch15:01
*** ysandeep|dinner is now known as ysandeep15:14
rlandy|ruckdviroel|rover|lunch: hey - https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone-on-multinode-ipa - trending red15:16
ysandeeprlandy|ruck, I was about to add workaround for ovn downgrade for 16.2 an 17, just want to check you haven't posted something already.15:17
ysandeepakahat, rlandy|ruck let me you if we still want to discuss about sc010 From above comments looks like rlandy|ruck already suggested something to solve the previous issue.15:20
rlandy|ruckysandeep: lol - not yet - haven't emerged from the rotating meetings yet15:20
rlandy|ruckysandeep: akahat: let's meet15:20
rlandy|ruckbecause we should settle on the solution15:20
ysandeeprlandy|ruck, ack, I will post workaround about ovn downgrade15:21
ysandeeprlandy|ruck, ack, I am available to chat now 15:21
rlandy|ruckysandeep: we'd need to do that on the container build15:21
ysandeepakahat, you around? 15:21
rlandy|ruckysandeep: akahat: https://meet.google.com/pvg-ifam-xby?pli=1&authuser=015:21
jpodivinchandankumar: I'm hitting a bug that's suspiciously similar to https://bugs.launchpad.net/tripleo/+bug/1934880 Same exact error although in a different job. The lp indicates that it resolved itself in time, is that true? 15:25
chandankumarjpodivin: yes, it is a known behaviour, the team is working on the proper fix15:26
jpodivinchandankumar++ thanks, I'm going to sing under it as an affected party and wait for it to go away. 15:28
akahatysandeep, https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/jobs15:44
rlandy|ruckysandeep: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/296123/32/zuul.d/required-projects-overrides.yaml#6115:46
mariosrdo zuul down? 15:52
akahatysandeep, https://review.opendev.org/c/openstack/tripleo-ci/+/82156015:55
rlandy|ruckmarios: ack15:56
rlandy|ruck<dpawlik|EoD> #status log restarting zuul after merging change https://github.com/ansible/zuul-config/pull/138/files15:56
rlandy|ruckfigure it's that15:56
akahatysandeep, https://code.engineering.redhat.com/gerrit/c/testproject/+/200295/216#message-db4ae6e39d871c110890b0461338ed2947157e9615:56
mariosah thanks rlandy|ruck 15:57
ade_leerlandy|ruck, hey -- afaranha was starting to look at the component pipeline train standalone ipa job16:09
ade_leerlandy|ruck, trying to understand why its failing -- iirc, this is one job we never got passing ..16:10
rlandy|ruckade_lee: yeah - it never worked16:10
rlandy|ruckat the time I think you suspected we were missing a backport16:10
ade_leerlandy|ruck, not sure -- but it looks like myswl isn't coming up ..16:11
afaranharlandy|ruck,  that's the issue, we found out 2 bz that are related to it https://paste.openstack.org/show/811638/16:11
afaranhait leads us to an issue on galera, but the bz is quite old, from 202016:12
*** dviroel|rover|lunch is now known as dviroel|rover16:12
rlandy|ruckoh ok - maybe16:12
rlandy|ruckwe didn't follow it up much16:12
dviroel|roverrlandy|ruck: ack, will check multinode-ipad16:16
dviroel|roveripa*16:16
ysandeeprlandy|ruck, Is it possible to check in details why jobs failed without involving infra each time- error like "This change depends on a change with an invalid configuration" is not verbose enough https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/29872116:18
rlandy|ruckysandeep: not really - but I know why those failed16:18
rlandy|ruckfixing that patch16:18
rlandy|ruckgive me a few16:19
ysandeepsure16:19
rlandy|ruckwe need to separate the two16:19
sshnaidmrlandy|ruck, content provider c9 passing: https://review.opendev.org/c/openstack/tripleo-ci/+/82124116:21
sshnaidmrlandy|ruck, will try standalone now16:21
rlandy|rucksshnaidm++ awesome16:21
dviroel|roverrlandy|ruck: i think that we just need to kick openstack-periodic-integration-stable4-centos7 pipeline again, containers-push didn't fail in testproject this time16:22
dviroel|roverrlandy|ruck: or testproject all missing jobs16:23
rlandy|ruckdviroel|rover: ok  - do you know how to rekick a line?16:25
dviroel|roverrlandy|ruck: know how to do, never did it. let me try and ping if needed16:26
rlandy|ruckdviroel|rover: sure - ping if you need help16:27
ade_leerlandy|ruck, so -- looking at the errors in that job -- maybe you guys have seen this before and have fixed this elsewhere but ..16:28
ade_leethe problem that is being reported is here -- 16:28
ade_leehttps://logserver.rdoproject.org/82/33582/20/check/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-security-train/56a1565/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log.txt.gz16:28
ade_leeCan't connect to MySQL server on 'overcloud.ctlplane.ooo.test16:29
ade_leenow mysql appears to be up 16:29
ade_leebut its config points to the standalone -- 16:29
ade_leehttps://logserver.rdoproject.org/82/33582/20/check/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-security-train/56a1565/logs/undercloud/var/lib/config-data/puppet-generated/mysql/etc/my.cnf.d/galera.cnf.gz16:30
ade_leeand https://logserver.rdoproject.org/82/33582/20/check/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-security-train/56a1565/logs/undercloud/etc/hosts.txt.gz16:30
ade_leeunless its going through haproxy ..16:30
ysandeeprlandy|ruck, I am out for the day, will test ovn workaround in my morning (want to test that before merging patches)16:30
rlandy|ruckysandeep: ack - thanks16:31
*** ysandeep is now known as ysandeep|out16:31
rlandy|ruckreading back16:32
rlandy|ruckade_lee: hmmm ... we'd have to compare the differences16:33
rlandy|ruckussuri is working16:33
rlandy|ruckso the deploy config is different in train16:34
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/188763316:35
rlandy|ruckade_lee: ^^ so apparently we have seen this16:35
rlandy|ruckharold comments there16:35
rlandy|ruckstatus: Triaged → Fix Released 16:36
rlandy|ruckwes marked that ^^16:36
rlandy|ruckstill broken16:36
ade_leerlandy|ruck, ack - thats what I'm seeing .. 16:36
ade_leenot sure what the fix was ?16:37
rlandy|ruckidk what causes that condition...16:37
rlandy|ruckI don't think there was a fix16:37
ade_leerlandy|ruck, the bug was opened against victoria16:38
rlandy|ruck"16:38
rlandy|ruckI am adding multinode-ipa job for stable/train [1] while testing, it fails having below trace back :"16:38
ade_leemaybe there was a fix but only from victoria on?  and still broken in train16:38
ade_lee?16:38
rlandy|ruckvictoria means the timeframe16:38
rlandy|rucknot the codebase16:38
ade_leeweshay, yo !16:39
rlandy|ruckade_lee: ^^ good luck with that - he hangs on slack these days16:39
ade_leeweshay, so what exactly did you mean when you said a fix was released in https://bugs.launchpad.net/tripleo/+bug/188763316:39
ade_leeha16:39
rlandy|ruckade_lee: probably have better luck pining harold and seeing where he left it16:40
rlandy|ruckand then to tripleo folks 16:41
ade_leerlandy|ruck, ack just pinged16:42
ade_leerlandy|ruck, is component pipeline the only place we run train stuff?16:42
*** marios is now known as marios|ou16:44
*** marios|ou is now known as marios|out16:44
*** amoralej is now known as amoralej|off16:44
rlandy|ruckade_lee: yep - no check/gate if it doesn't pass16:45
ade_leerlandy|ruck, where is link to ussuri job?16:51
rlandy|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri17:00
rlandy|ruckade_lee: ^^17:00
dviroel|roverrlandy|ruck: i just triggered upgrade-ussuri, but the patch isn't at current yet, will abandon-restore as soon the patch gets into current17:50
*** jpena is now known as jpena|off17:50
dviroel|roverir17:50
dviroel|roveror17:50
dviroel|roveri can just trigger with the depends-on17:51
dviroel|roverhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone-on-multinode-ipa - had green on check + green on testproject18:35
rlandy|ruckdviroel|rover: ack - just asked tristanC about that18:51
dviroel|roverrlandy|ruck: do you want to wait for current? i trigerred with depends-on, but I can abandon the patch18:53
rlandy|ruckdviroel|rover: it's fine18:53
rlandy|ruckworst  dlrn will kick tomorrow18:54
rlandy|ruckdviroel|rover: pls keep your depends on patch18:54
dviroel|roverok18:54
*** sshnaidm is now known as sshnaidm|afk19:06
rlandy|ruckdviroel|rover: rechecking master uefi image builds https://review.rdoproject.org/r/c/testproject/+/3625419:28
rlandy|ruckfs002 also failed19:28
dviroel|roverrlandy|ruck: gah, standalone-upgrade-ussuri failed20:09
dviroel|roverrlandy|ruck: https://logserver.rdoproject.org/61/37161/4/check/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/adc6793/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz20:09
rlandy|ruckdviroel|rover: sporadic or legit?20:09
dviroel|rover"Error: resource 'rabbitmq-bundle' is not running on any node"20:09
rlandy|ruckugh20:10
rlandy|ruckkick it again and pray20:10
dviroel|roverack20:10
rlandy|ruckdviroel|rover: sorry20:10
dviroel|roverfor what?20:11
rlandy|ruckannoying failures20:12
dviroel|rovercomponent job also failed https://logserver.rdoproject.org/61/37161/4/check/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-ussuri/ccc98dc/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz20:12
dviroel|roverok, rekicked20:14
rlandy|ruckdviroel|rover: I'll also kick one20:15
rlandy|ruckmay be one will20:15
rlandy|ruckwork20:15
*** dviroel|rover is now known as dviroel|rover|afk21:02
rlandy|ruckperiodic-tripleo-ci-centos-8-standalone-upgrade-ussuri https://review.rdoproject.org/zuul/build/b052d465ebee4e819faf3b7e6fc8287f : FAILURE in 1h 22m 35s (non-voting)22:11
rlandy|ruckdviroel|rover|afk: ^^22:11
rlandy|ruckno dice22:11
rlandy|ruckugh - again this time with the depends-on :)22:14
rlandy|ruckdviroel|rover|afk: trying another testproject with a revert https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/82152123:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!