Monday, 2021-08-23

« Friday, 2021-08-20 Index Tuesday, 2021-08-24 »
*** ysandeep|away is now known as ysandeep06:32
*** sshnaidm|afk is now known as sshnaidm06:34
pojadhavysandeep, 0/07:37
*** jpena|off is now known as jpena07:37
ysandeeppojadhav, hi07:37
pojadhavysandeep, do you know ceph related issue was going on.. is still there ? https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario010-standalone-network-rhos-17/58d5590/logs/undercloud/home/zuul/standalone_deploy.log07:37
ysandeeptripleo component need promotion to fix that, fix is stuck at current hash, A FTBFS is ongoing.. https://trello.com/c/1aKt6YKf/2064-cixftbfsosp-17007:40
pojadhavysandeep, ohh okay.. this is the same issue which ronelle discussed. got it.. thanks!07:41
ysandeepyup.. same issue07:41
arxcruz|offzbr: frenzy_friday i'm getting this error when i try to do a make up 07:56
arxcruz|offer-cron | elasticsearch.exceptions.SSLError: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate is not yet valid (_ssl.c:1131)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate is not yet valid (_ssl.c:1131))07:56
arxcruz|offshould i set some variable to ignore it ?07:57
frenzy_fridayarxcruz|off, didnt see this error till now. Checking07:58
frenzy_fridaywhich elasticsearch version is it?07:59
arxcruz|offfrenzy_friday: i got from your latest patch 07:59
arxcruz|offlet me check07:59
arxcruz|off elasticsearch==7.14.007:59
frenzy_fridayhm.. not sure which url it is failing for. me checks08:01
*** ykarel is now known as ykarel|lunch08:23
arxcruz|offfrenzy_friday: fixed, date issues :) 08:24
frenzy_fridayoh, okay. What did you change? I'll update the doc08:25
arxcruz|offfrenzy_friday: no, actually, i'm running docker on a vm because i don't have on my machine 08:26
arxcruz|offand i paused the vm when i turn my computer off08:27
arxcruz|offwhen i bring it up back, it did not update the date, it was august 1708:27
arxcruz|offso when tried to connect to es, it was failing08:27
frenzy_fridaygot it08:27
arxcruz|offsee, certificate is NOT YET valid 08:27
arxcruz|offlol 08:27
arxcruz|offfrenzy_friday: i was getting a lot of unknown bug 09:08
arxcruz|offbut then, after reach https://review.rdoproject.org/elasticsearch/logstash-2021.08.10/_stats 09:08
arxcruz|offnow the page is empty 09:08
frenzy_fridayarxcruz|off, shall we have a call?09:13
arxcruz|offsure 09:13
arxcruz|offgive me 5 min 09:13
frenzy_fridaysure09:13
arxcruz|offfrenzy_friday: ready 09:17
frenzy_fridayarxcruz|off, https://meet.google.com/ykw-xxva-akx09:17
*** ykarel|lunch is now known as ykarel09:36
*** ykarel is now known as ykarel|afk10:53
*** arxcruz|off is now known as arxcruz11:01
weshay|ruckpojadhav, 0/11:31
*** dviroel|out is now known as dviroel|ruck11:31
*** jpena is now known as jpena|lunch11:33
pojadhavweshay|ruck, hi11:36
weshay|ruckpojadhav, you coming ?11:36
pojadhavweshay|ruck, yup11:37
weshay|ruckchandankumar++ nice job w/ the FTBS11:37
chandankumarweshay|ruck: I have done nothing there11:37
chandankumarjust got the trello card from CIX11:38
*** rlandy is now known as rlandy|rover11:43
rlandy|roverweshay|ruck: hi - you wanted to  talk about https://trello.com/c/1aKt6YKf/2064-cixftbfsosp-170?11:45
chandankumardviroel|ruck: \o11:49
chandankumardviroel|ruck: please have a look at this https://review.opendev.org/c/openstack/tripleo-common/+/804797/1#message-c7c650b1231a53a24e1238d90a36b4cd19c273a7 when free, thanks!11:49
weshay|ruckya.. but talking to pooja11:50
dviroel|ruckchandankumar: hi, ok11:51
rlandy|roverdviroel|ruck: hi - ho ware things?11:58
dviroel|ruckrlandy|rover: hi, so far so good, it seems that only periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri was missing for ussuri promotion12:00
dviroel|ruckopenstack-periodic-integration-stable3 just started12:03
rlandy|rovercool12:05
rlandy|roverysandeep: hey - do you know what's up with all the standalone failures in 17 line or should I investigate that?12:25
ysandeeprlandy|rover: they started failing in currently run, I checked in my morning apart from 010 others were passing12:28
rlandy|roverysandeep: k - will check into it after meetings12:28
rlandy|rovermaybe cloud hitch12:28
*** jpena|lunch is now known as jpena12:29
rlandy|roverdviroel|ruck: feel free to comment at CIX - will jump in when needed12:31
dviroel|ruckrlandy|rover: ok, tks12:31
rlandy|roverchandankumar: hey - want to touch base on el9?12:52
chandankumarrlandy|rover: yes sure12:52
rlandy|roverchandankumar: https://meet.google.com/rku-bupo-rqz?pli=1&authuser=012:53
chandankumararxcruz, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, weshay, chandankumar, frenzy_friday, dviroel Scrum: https://meet.google.com/bqx-xwht-wky13:00
chandankumarsorry https://meet.google.com/xnf-tvdh-pmk?authuser=013:01
rlandy|roverzbr: arxcruz: ^^13:03
*** amoralej is now known as amoralej|lunch13:07
zbri am in another meeting, i will join later13:09
*** amoralej|lunch is now known as amoralej13:46
rlandy|roverznr: no worries13:48
rlandy|roverysandeep: pls ping when you are EoD - I will follow through promoting the tripleo component if needed 13:51
rlandy|roverwe ar replying on OVB in place of BM, right?13:51
ysandeeprlandy|rover, ack for eod, yes we are relying on OVB.. I am fixing BM for 17.. currently in manual testing mode.. hitting some issue.. but I am making progress13:52
weshay|ruckdviroel|ruck, need anything?13:53
arxcruzrlandy|rover: System is going down. Unprivileged users are not permitted to log in anymore. For technical details, see pam_nologin(8).13:53
arxcruzwhen i try to access the vm 13:53
rlandy|roverarxcruz: the candidate vm13:53
dviroel|ruckweshay|ruck: investigating that https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-ussuri 13:53
arxcruzrlandy|rover: the dandidate vm yes13:53
rlandy|roverysandeep may be working on it13:53
dviroel|ruckweshay|ruck: trying to search for more error than https://4d34507513d46a298d9b-c450cedbad3a93d818e1040974f11faf.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-standalone-upgrade-ussuri/3e39fdd/logs/quickstart_install.log13:54
rlandy|roverarxcruz: checking13:54
ysandeeprlandy|rover, arxcruz, I am working on candidate vm13:54
arxcruzysandeep: so you're the one to blame 13:54
weshay|ruckdviroel|ruck, probably best to debug the "periodic" version of those...13:55
arxcruzysandeep: https://www.youtube.com/watch?v=SrDSqODtEFM13:55
weshay|ruckthat way.. there is no additional code change to muck it up13:55
ysandeeparxcruz, :) I will ping you once I am done deploying standalone env13:56
weshay|ruckdviroel|ruck, so fails here: 2021-08-22 10:00:56.111025 | primary | TASK [os_tempest : Ping router ip address] *************************************13:56
dviroel|ruckyep13:56
weshay|ruck2021-08-22 08:59:54.415 ERROR /var/log/containers/neutron/server.log: 19 ERROR neutron.db.agentschedulers_db [req-d53e2492-8504-4b2f-922f-ce1edaccd4c1 - - - - -] Exception encountered during network rescheduling: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.3' ([Errno 113] EHOSTUNREACH)")13:57
weshay|ruckdviroel|ruck, https://9bbde2059085467a4330-af2016a5632320f910deb9dcbf495ac6.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-standalone-upgrade-ussuri/eb7bbdc/logs/undercloud/var/log/extra/errors.txt13:58
weshay|ruckhttps://9bbde2059085467a4330-af2016a5632320f910deb9dcbf495ac6.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-standalone-upgrade-ussuri/eb7bbdc/logs/undercloud/var/log/containers/mysql/mysqld.log13:59
weshay|ruckdviroel|ruck, looks like the upgrade never brought it back up13:59
weshay|ruckhttps://9bbde2059085467a4330-af2016a5632320f910deb9dcbf495ac6.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-standalone-upgrade-ussuri/eb7bbdc/logs/undercloud/var/log/containers/mysql/mysqld-upgrade.log13:59
weshay|rucklet's compare to a successful job13:59
weshay|ruckok.. nope.. the restart is not logged .. successful job https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b1e/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-standalone-upgrade-ussuri/b1e21a3/logs/undercloud/var/log/containers/mysql/mysqld-upgrade.log14:00
rlandy|roverdviroel|ruck: sorry ... need help with anything other than ^^?14:00
rlandy|rovercan look now14:00
dviroel|ruckrlandy|rover: no, not really, waiting ussuri jobs yet14:01
weshay|ruckdviroel|ruck, this is not an error: 14:04
weshay|ruck2021-08-22 09:40:39 |         "<13>Aug 22 09:40:12 puppet-user: Warning: Unknown variable: '::deployment_type'. (file: /etc/puppet/modules/tripleo/manifests/packages.pp, line: 39, column: 69)",14:04
weshay|ruckbut.. suspicious14:04
weshay|ruckhttps://9bbde2059085467a4330-af2016a5632320f910deb9dcbf495ac6.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-standalone-upgrade-ussuri/eb7bbdc/logs/undercloud/home/zuul/standalone_upgrade.log14:04
dviroel|ruckweshay|ruck: hum, same warning in the sucessfull job too https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b1e/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-standalone-upgrade-ussuri/b1e21a3/logs/undercloud/home/zuul/standalone_upgrade.log14:06
weshay|ruckdviroel|ruck, aye.. agree14:07
* weshay|ruck hunting https://review.opendev.org/q/tripleo+-age:5d+status:merged14:07
weshay|ruckhttps://review.opendev.org/q/tripleo+-age:5d+status:merged+branch:stable/ussuri14:08
weshay|ruck¯\_(ツ)_/¯14:09
weshay|ruck https://review.opendev.org/c/openstack/puppet-tripleo/+/804155/3/manifests/profile/pacemaker/database/mysql_bundle.pp14:09
weshay|ruckupgrades are tough..  spend a couple hours.. write up what you can.. and cix for help14:12
*** pojadhav- is now known as pojadhav14:14
rlandy|roverdviroel|ruck: weshay|ruck: note - vexxhost is having issues14:15
rlandy|roversee #rhos-ops14:15
rlandy|roverarxcruz: ^^ fyi14:16
rlandy|roverlogin problem14:16
ysandeeparxcruz, rlandy|rover instance is back, you can try to login now14:20
rlandy|roverarxcruz: ^^ ols try so we can see if you have access14:21
rlandy|roverpls14:21
weshay|ruckdviroel|ruck, let's get the bug open so that people see why their check jobs are getting blocked 14:29
dviroel|ruckweshay|ruck: ok14:30
dviroel|ruckweshay|ruck: the change that you pasted above, is it related or is a guess?14:31
weshay|ruckit's ussuri so low traffic14:31
weshay|ruckdviroel|ruck, I'm guessing at what could be the problem there..14:31
weshay|rucknot a lot of changes in train and ussuri though.. 14:31
dviroel|ruckweshay|ruck: ok, i'll open the bug with the os_tempest error: "Ping router ip address.."14:32
weshay|ruckdviroel|ruck, that and the mysql error14:33
dviroel|ruckweshay|ruck: yeah, maybe some patch that is in victoria and not in ussuri since: https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-victoria14:34
rlandy|roverdviroel|ruck: just fyi - testprojected the two wallaby failures14:35
arxcruzrlandy|rover: ysandeep i was able to log in 14:43
ysandeeparxcruz: cool!14:43
rlandy|roveryay!14:44
rlandy|roverprogress14:45
dviroel|ruckweshay|ruck: https://bugs.launchpad.net/tripleo/+bug/1940844 14:53
dviroel|ruckweshay|ruck: will continue to investigate to add more details to it14:53
weshay|ruckdviroel|ruck++ thank you14:54
dviroel|ruckweshay|ruck: didn't add promotion-blocker tag to this one14:57
weshay|ruckdviroel|ruck, that's fine for now.. but as you get closer to wanting to hand off.. add it14:58
*** amoralej is now known as amoralej|off15:08
zbrusing redhat sso seems to become somethign that needs training15:13
zbri tried to login to jira, end-up on sso.redhat.com which redirected my to https://sbarnea.com/ss/Screen-Shot-2021-08-23-16-14-25.42.png --- apparently google was not enough, now we also got salesforge in.15:14
*** dviroel|ruck is now known as dviroel|ruck|lunch15:17
*** jpena is now known as jpena|off15:36
sshnaidmrlandy|rover, hey15:49
sshnaidmrlandy|rover, can we talk about psi c9 issue?15:49
rlandy|roversshnaidm: ack 15:49
sshnaidmrlandy|rover, https://meet.google.com/vji-fzhb-bkp15:50
*** ysandeep is now known as ysandeep|dinner15:51
rlandy|roverchandankumar: ^^ hey was talking with sshnaidm about c9 - booking some time tomorrow so we can sync up16:05
rlandy|roverhe has standalone working with c8 containers16:05
rlandy|roverwe could out c9 containers in there16:06
*** ysandeep|dinner is now known as ysandeep16:10
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=%09openstack-promote-component16:17
rlandy|roverwoohoo - only one component not promoted16:17
rlandy|rovernetwork 17 - getting there16:17
rlandy|roverweshay|ruck: ^^16:17
rlandy|roverneed 17 promotion16:18
*** dviroel|ruck|lunch is now known as dviroel|ruck16:19
rlandy|roverysandeep: ^^ :)16:20
ysandeeprlandy|rover, tripleo component run report back: https://code.engineering.redhat.com/gerrit/c/testproject/+/211643 16:21
ysandeepall passed except bm failure(expected not in criteria)16:21
ysandeeprlandy|rover, tripleo will promote in next run promote-to-promoted run... then we will need integration line promotion for other component to pick the sc010 fix16:22
rlandy|roverysandeep: yep - saw  - thanks16:23
rlandy|roverit's in promoted-components16:23
rlandy|roverso can rekick 17 line16:23
rlandy|roverysandeep: ^^ rekicking 17 line16:26
ysandeepack o/16:47
* ysandeep wondering if you/wes can demo in community call.. how to retrigger the pipelines16:48
rlandy|roverack can do16:53
rlandy|roverzuul admin instructions16:53
rlandy|roverweshay|ruck: re: your patch on previous tripleo-ci-testing16:54
rlandy|roverpromoted-components :)16:54
weshay|ruckhrm.. that ran on components?16:56
rlandy|roverhttps://review.rdoproject.org/r/c/config/+/34934/3/playbooks/tripleo-ci-base-promote-consistent-to-tripleo-ci-testing/run.yaml16:57
rlandy|roverthat playbook moves consisent to tripleo-ci-testing16:57
rlandy|roverwe don't do that anymore16:57
rlandy|roverpromoted-components -> tripeo-ci-testing16:58
weshay|ruckoh ya.. ur right16:58
rlandy|roverso we need ...16:58
weshay|ruckI think I just made the change in the wrong place16:58
weshay|ruckand this is duplicated elsewhere16:58
rlandy|roverhttps://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-wallaby-promote-promoted-components-to-tripleo-ci-testing/8d1667f/job-output.txt16:58
rlandy|rovertaking example ^^16:58
rlandy|rover[trusted : review.rdoproject.org/config/playbooks/tripleo-ci-base-promote-hash/run.yaml@master]16:59
weshay|ruckreview.rdoproject.org/config/playbooks/tripleo-ci-base-promote-hash/run.yaml16:59
rlandy|roverhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-base-promote-hash/run.yaml17:00
rlandy|roverright thta17:00
rlandy|roverthat17:00
rlandy|roverpromote-primary-distro.yaml17:00
rlandy|roverhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/promote-primary-distro.yaml17:01
weshay|ruckwonder if it's a better use case after tripleo-repos get hash.. has been integrated here17:14
dviroel|ruckweshay|ruck: didn't make progress on that https://bugs.launchpad.net/tripleo/+bug/1940844 - tricky17:17
dviroel|ruckalex made some comment in the LP17:17
dviroel|ruckmaybe a network issue that we can't see in the logs17:18
weshay|ruckaye.. k.. if you add promotion-blocker it won't cix for 5 hours..17:18
weshay|ruckso .. 17:18
weshay|ruckI looked at the network settings too.. didn't see much diff.. damian will probably pick it up in his morning17:19
weshay|ruckdviroel|ruck, fyi https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario001-standalone 17:20
weshay|rucktempest.lib.exceptions.Conflict: Conflict with state of target resource17:20
weshay|ruckDetails: {'type': 'SecurityGroupInUse', 'message': 'Security Group deca90a1-f2ef-4c8b-8ae8-8f7f11a1dbc9 in use.', 'detail': ''}17:20
weshay|ruck}}}17:20
weshay|ruckjust 1 hit.. will be watching for more17:20
dviroel|ruckok17:20
dviroel|ruckso, will add the tag to it17:21
rlandy|roverci.centos rekicked17:37
weshay|ruckdviroel|ruck, rlandy|rover for the compute component: https://review.opendev.org/c/openstack/nova/+/80566317:39
dviroel|ruck++17:40
rlandy|rovernice17:41
dviroel|ruckrlandy|rover: openstack-periodic-integration-stable3 disaster might be related with vexxhost earlier issue, right?17:43
rlandy|roverdviroel|ruck: ack17:43
rlandy|roverno worries it will rerun17:43
dviroel|ruckok17:43
ysandeeprlandy|rover, weshay|ruck fyi.. I have analyzed https://bugs.launchpad.net/tripleo/+bug/1940729 which was discussed on df call, We only hit this on second rerun of overcloud deploy on existing environment.. We don't have a CI job that do overcloud deploy rerun(Update / Scale operation) in upstream/component ci..18:08
* ysandeep will check rabi's bug tomorrow18:08
weshay|ruckysandeep, rock.. and you have a handle on the ipv6 mis-config right?18:08
weshay|ruckre: another issue rabi was speaking to18:09
weshay|ruckah.. that's what ur talking about in your second comment18:09
weshay|rucknevermind18:09
weshay|ruckur ahead of me18:09
* ysandeep hoping once baremetal is up again we can possibly add scaleout on one of a baremetal env.. 18:11
weshay|ruckthat may or may not be worth the effort because it's covered by qe...18:24
weshay|ruckysandeep, if you have a sec.. let's chat18:24
weshay|ruckcan be tomorrow as well.. but you AIN'T GOT NOTHIN GOING ON... you are a FREE MAN18:25
ysandeepweshay|ruck, lets chat18:26
weshay|ruckysandeep, meet.google.com/hdn-puut-rcm18:27
rlandy|roverperiodic-tripleo-ci-centos-8-scenario001-standalone-wallaby18:35
rlandy|rover- ok - see that failed again18:35
dviroel|rucksame tempest tests18:39
dviroel|ruckthere are two wallaby failures for that:18:48
dviroel|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby/d97fad9/logs/undercloud/var/log/tempest/stestr_results.html.gz18:48
dviroel|ruckhttps://logserver.rdoproject.org/95/24995/93/check/periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby/23c3023/logs/undercloud/var/log/tempest/stestr_results.html.gz18:48
*** ysandeep is now known as ysandeep|away19:05
dviroel|ruck^ nova-api stop responding after GET http://192.168.24.3:8774/v2.1/servers/88242305-5529-4f74-a342-5a37f7d2500519:18
dviroel|ruckhttps://logserver.rdoproject.org/95/24995/93/check/periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby/23c3023/logs/undercloud/var/log/containers/nova/nova-api.log.txt.gz req-bef8458419:18
rlandy|roverdviroel|ruck: hey - you're looking into scenario001 failure?19:49
dviroel|ruckyes19:50
*** slaweq is now known as slaweq_19:50
rlandy|roverdviroel|ruck: k - need help?19:51
rlandy|rovertempest failure19:52
dviroel|ruckrlandy|rover: should we wait for one more failure? it seems that in master we have two tests failing, and one in wallaby19:53
dviroel|rucki was looking into wallaby, now looking at master19:54
rlandy|roverdviroel|ruck: wallaby has one in gate and one in testproject integration line19:55
rlandy|roverboth tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern19:55
rlandy|rover^^ right?19:55
rlandy|roveryou're seeing that failure?19:55
dviroel|ruckhere https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1dd/800341/23/check/tripleo-ci-centos-8-scenario001-standalone/1dda775/logs/undercloud/var/log/tempest/stestr_results.html we also have tempest.scenario.test_snapshot_pattern.TestSnapshotPattern19:56
dviroel|ruck^ and this one seems to be a glance issue19:56
dviroel|ruckjust trying to fing the root cause19:57
dviroel|ruckfind*19:57
rlandy|roverso we have different tempest failures19:58
rlandy|roverhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1dd/800341/23/check/tripleo-ci-centos-8-scenario001-standalone/1dda775/logs/undercloud/var/log/tempest/stestr_results.html19:59
rlandy|roversame19:59
rlandy|rovertempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received19:59
rlandy|roverDetails: 50319:59
dviroel|ruckhttps://17c6cb2f9fa2917c93e9-c8c2a5181a911daab043eb1d43163b4b.ssl.cf2.rackcdn.com/805559/1/gate/tripleo-ci-centos-8-scenario001-standalone/5ffb422/logs/undercloud/var/log/tempest/stestr_results.html tempest.scenario.test_snapshot_pattern.TestSnapshotPattern with a different error20:01
* dviroel|ruck has so many tabs open20:02
rlandy|roverdviroel|ruck: lol - welcome to ruck/rover life20:03
rlandy|roverwhen your laptop starts smoking from overuse, you've made it :)20:03
dviroel|ruckah, i just got a new one, still have lot of resources :)20:04
rlandy|roverlet's see when this hit integration20:05
rlandy|roverand check/gate20:05
dviroel|ruckglance failing here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1dd/800341/23/check/tripleo-ci-centos-8-scenario001-standalone/1dda775/logs/undercloud/var/log/containers/glance/api.log20:06
dviroel|ruck^test_snapshot_pattern.TestSnapshotPattern20:06
rlandy|roverhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby20:06
rlandy|roverlast success 08/2220:06
rlandy|roverw promoted yesterday20:07
rlandy|roverlet's check when glace updated20:07
rlandy|roverhttps://trunk.rdoproject.org/centos8-wallaby/component/glance/ hasn't updated in ages20:08
rlandy|rovermore likely tripleo20:09
rlandy|roverhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby20:11
rlandy|rover^^ see when that first started20:11
rlandy|roverthose could be vexx impacted20:13
dviroel|ruck2021-08-19 18:02 https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby/dc1177a/logs/undercloud/var/log/tempest/stestr_results.html.gz20:14
dviroel|rucksame errors: http://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-wallaby/dc1177a/logs/undercloud/var/log/containers/glance/api.log.txt.gz20:16
rlandy|rover2021-08-23 16:23:31.177031 | localhost | Provider: vexxhost-nodepool-tripleo20:17
rlandy|rover^^ gate failure20:17
rlandy|roverscratch that20:17
rlandy|rover2021-08-23 18:09:39.920713 | localhost | Provider: inap-mtl0120:18
rlandy|rover2021-08-23 18:09:39.920803 | localhost | Label: centos-8-stream20:18
rlandy|roverother than just vexx20:18
rlandy|rovercomparing passing and failing rpms20:19
rlandy|roverdviroel|ruck: hmmm https://review.opendev.org/c/openstack/tripleo-heat-templates/+/805280/ passes20:21
rlandy|roverwonder if we need that 20:21
rlandy|roverto get the rest to pass20:21
rlandy|roverhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario001-standalone&branch=master20:22
rlandy|rovermuch more consistent failure20:22
rlandy|roverhttps://review.opendev.org/c/openstack/tripleo-heat-templates/+/804896 merged august 2020:22
dviroel|ruckhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e46/805280/3/check/tripleo-ci-centos-8-scenario001-standalone/e469d21/logs/undercloud/var/log/containers/glance/api.log this one passes and also has that glance error20:24
dviroel|ruckthere is other passing jobs that don't have this error, weird20:24
rlandy|roverhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario001-standalone&branch=master - prob most consistent20:26
rlandy|roverhttps://review.opendev.org/c/openstack/tripleo-heat-templates/+/805541/20:26
rlandy|roverpasses20:26
dviroel|ruck^ has a depends on that is failing on scenario001 https://review.opendev.org/c/openstack/puppet-tripleo/+/80554020:29
rlandy|roveryeah20:29
rlandy|roverpuppet-tripleo20:29
weshay|ruckok.. scenario001 tempest.. anyone on that or shall I write it up20:29
dviroel|ruckrlandy|rover: but isn't failing on tempest :p20:30
rlandy|roverweshay|ruck: lol20:31
rlandy|roverdviroel|ruck: weshay|ruck: so here's what we know ...20:31
rlandy|roverwallaby and master20:31
weshay|ruckin the upstream gate it is.. in wallaby20:31
rlandy|rovermostly TestSnapshotPattern20:31
weshay|ruckperhaps two diff issues.. yes...20:32
weshay|rucksnapshot20:32
dviroel|ruckin wallaby i see tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern failing only20:32
dviroel|ruckin master has  tempest.scenario.test_snapshot_pattern.TestSnapshotPattern too20:32
weshay|ruckk.. so.. create one bug, two different entries in tempest skip file20:33
rlandy|roverdviroel|ruck: weshay|ruck: k - I'll write one bug20:33
rlandy|roverlet's start there20:33
weshay|ruckya20:33
rlandy|roverwe can edit20:33
rlandy|roverat leats we have a place to drop all this debug20:33
rlandy|roversec - coming up20:33
dviroel|ruck+120:33
weshay|ruck2 wallaby hits, 1 master here: http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?orgId=120:34
abregmanhey. do we have any docs on how to add new jobs to component pipline?20:41
weshay|ruckabregman, zuul or jenkins20:42
abregmank maybe a different question...where component pipeline jobs are running today? :)20:42
weshay|ruckzuul and jenkins20:43
weshay|ruckpick your poison20:43
rlandy|roverdviroel|ruck: weshay|ruck: https://bugs.launchpad.net/tripleo/+bug/194086620:43
rlandy|rover^^ let's capture debug there20:43
weshay|ruck++20:43
abregmanI honestly don't care. what is the "right" place?20:44
weshay|ruckboth20:44
rlandy|roverabregman; depends if you want to job to run in zuul or just be triggered with the component line and report there20:44
weshay|ruckdepends on what you want to execute.. if you want to run an upstream job.. use zuul20:44
weshay|ruckif you want to run something from p1/2/3 use jenkins20:44
dviroel|ruckrlandy|rover: will create the skiplist patch20:44
rlandy|roverdviroel|ruck++ thanks20:44
abregmanso probably Jenkins20:45
rlandy|roverabregman: if you already have a jenkins job that does what you want20:45
rlandy|roverthen let's just trigger it and report20:45
rlandy|roverif not, look into zuul jobs20:45
rlandy|roveryou can use attila's jobs as examples20:45
weshay|ruckabregman, you'll want to use https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/ as example.. 20:46
weshay|ruckya.. what rlandy|rover said20:46
abregmansounds good. I'll have a look. thank you both20:46
rlandy|roveryou will trigger off https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-upload-job-trigger-rhos-17/56796a4/20:46
weshay|rucknp20:46
rlandy|roverfor example20:46
rlandy|roverand then just report back to jenkins20:46
rlandy|roveryou need to match the hash under test20:46
rlandy|roverand pick up the right set of containers20:47
weshay|ruckI wouldn't start w/ 17 though.. they are still getting 17 working20:47
rlandy|roversimple as that :)20:47
weshay|ruckgo 4 the stable branch luke20:47
abregmanwill it be visible in tripleo-cockpit?20:47
rlandy|roveryes20:47
weshay|ruckabregman, yes20:47
abregmancool20:47
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-upload-job-trigger-rhos-17/56796a4/20:47
rlandy|roveroops20:47
rlandy|roverhttp://tripleo-cockpit.usersys.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=120:47
rlandy|roverif you scroll down20:47
rlandy|roveryou'll see the pipeline jenkins jobs there20:48
rlandy|roveronce you have the job name - we can make sure it's captured20:48
abregmanthat's great. Easier than I imagined20:48
weshay|ruckwe're here to please20:48
rlandy|roverabregman: attila is your new best friend here20:49
rlandy|roverwe'll provide containers and overcloud images you can use etc.20:50
rlandy|roverabregman: when your job is ready, we add it to criteria20:50
rlandy|roverand then it decides whether the component promotes or not20:51
rlandy|roverdone and done20:51
dviroel|ruckrlandy|rover: so, should i skip tripleo-ci-centos-8-scenario001-standalone + both periodics?20:52
rlandy|roverexcellent question20:52
abregmanrlandy|rover: where the criteria is defined? in tripleo-environments repo?20:52
* rlandy|rover gets20:53
weshay|ruckdviroel|ruck, I think so for now.. yes.. all varients of scenario00120:53
weshay|ruckfor those branches20:53
rlandy|roverhttp://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/ci-scripts/dlrnapi_promoter/config/RedHat-8/component20:53
abregmangreat. I definitely know much more now. thanks again20:55
rlandy|roverdviroel|ruck; thanks - voted21:02
dviroel|ruck\o/21:02
abregmanrlandy|rover: I see that Jenkins jobs are triggered every time there is a change in rhel8-osp16-2/component/network/component-ci-testing/commit.yaml21:09
abregmanrlandy|rover: is this the same also for Zuul?21:09
rlandy|roveryes21:09
rlandy|roverwe trigger those line once a day21:09
rlandy|roverwhich changes component-ci-testing21:09
rlandy|roverso from the zuul side, it's a time trigger21:10
abregmanrlandy|rover: so a periodic 24h trigger? but what/who updates the content of commit.yaml file?21:11
rlandy|roverthe first job in that line21:11
rlandy|roverso it works like this ...21:12
rlandy|rovertaking networking component as an example21:12
abregmanrlandy|rover: k so the first job is triggered every 24 hours and then all the other jobs are triggered based on monitoring changes in commit.yaml?21:12
rlandy|roverthe first job gets triggered once a day ( one a cycle)21:13
rlandy|roverthat job uses dlrn to move consistent to component-ci-testing21:13
rlandy|roverif that job is successful, the rest of the zuul jobs run21:13
rlandy|roverit's a zuul dependency21:14
rlandy|roverjenkins jobs pick their trigger as you saw21:14
rlandy|roverabregman: http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d/project-templates-components.yaml21:15
rlandy|roverhttp://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/zuul.d/pipelines.yaml#n27321:16
rlandy|roverabregman: ^^ line trigger21:16
rlandy|roverand the one above, the zuul dependencies21:16
abregmanaha..k. I need to write it all down somewhere21:16
rlandy|roverabregman: we'll need to write this all up for a bunch of people I guess21:17
abregmanrlandy|rover: k one last question for today because I need to do some processing as well, I see the jobs are not running networking tests (only some basic tempest tests). Will it be fine to add more component-specific tests in the future?21:17
rlandy|roverwe shoudl put together a hackmd or doc of sorts21:17
rlandy|roverwe have some of it here:21:18
rlandy|roverabregman:  to answer question above - YES please21:18
rlandy|roverthe more testing earlier, the better21:18
weshay|ruckabregman, YES please add a better deployment + better tests21:18
rlandy|roverlol21:18
weshay|ruckthink of the current jobs as HELLO-WORLD21:18
rlandy|roverwe are like stereo 21:19
weshay|ruckit's boiler plate.. 21:19
weshay|ruckbut we don't know what is the right job or tests .. hence the subject matter expert21:19
rlandy|roverhttps://docs.openstack.org/tripleo-docs/latest/ci/stages-overview.html#the-component-promotion-pipeline21:20
abregmank great. will do. but first I need to write some notes, probably draw little bit...just to understand better the workflow here21:20
rlandy|roverabregman: ^^ that's upstream doc but the concept is the same21:21
abregmangreat. I have enough docs now to process until the end of the year21:21
abregmanbut seriously, thanks a lot. this is all very helpful21:22
rlandy|roverend of the year is only a few weeks away :)21:22
rlandy|roverin fact a few days21:23
weshay|rucklolz21:23
weshay|ruckoh heeb humor21:23
abregmangood one :D21:23
-opendevstatus- NOTICE: The Gerrit service on review.opendev.org has been restarted for a patch version upgrade, resulting in a brief outage21:41
rlandy|roverweshay|ruck: hey - do you see arxcruz's open reviews?21:52
rlandy|roverfor fs001?21:52
arxcruzrlandy|rover: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/804396 and https://review.opendev.org/c/openstack/tripleo-quickstart/+/80439922:02
rlandy|roverah thanks looking22:03
arxcruzrlandy|rover: i'll update jira tomorrow morning 22:03
rlandy|roverarxcruz: we should be able to merge https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/804396 w/o issue right22:04
rlandy|roverarxcruz: which one needs to merge first?22:05
rlandy|roveroh depends22:05
rlandy|roverI see it22:05
rlandy|roverarxcruz: ok - so here's what I'd like to do ...22:06
rlandy|roverhttps://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/80439622:06
rlandy|rovermerge ^^22:06
rlandy|rover(should not impact w/o https://review.opendev.org/c/openstack/tripleo-quickstart/+/804399)22:07
rlandy|roverand merge https://review.opendev.org/c/openstack/tripleo-quickstart/+/804399 tomorrow when ruck/rover can watch it22:07
rlandy|roverok?22:07
rlandy|roverdviroel|ruck: weshay|ruck: ^^ fyi22:07
dviroel|ruckack22:15
* rlandy|rover back in a bit22:25
*** dviroel|ruck is now known as dviroel|out22:37
*** chem is now known as Guest520122:51
weshay|ruckrlandy|rover, ack22:59
« Friday, 2021-08-20 Index Tuesday, 2021-08-24 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!