Tuesday, 2022-05-10

*** rlandy|bbl is now known as rlandy|out01:22
*** soniya29 is now known as soniya29|ruck04:38
*** ysandeep|out is now known as ysandeep|rover04:42
ysandeep|roverdasm|off, ++ thanks for https://review.rdoproject.org/r/c/rdo-jobs/+/42673 04:53
dasm|offysandeep|rover: no worries. i hope it's gonna help a bit.05:00
dasm|offi noticed that it's past provisioning, but it still fails on something.05:00
dasm|offi didn't investigate a lot, tho.05:01
ysandeep|roverdasm|off, it fixed the issue we were hitting in check :D05:04
ysandeep|roverdasm|off, don't you sleep :D05:05
dasm|offysandeep|rover: i'm about to get to a bed :)05:11
ysandeep|roverdasm|off, good night o/ see you in few hours :D05:12
dasm|offysandeep|rover: have a quiet day!05:12
soniya29|ruckysandeep|rover, hello05:21
*** pojadhav|afk is now known as pojadhav05:23
ysandeep|roversoniya29|ruck: https://bugs.launchpad.net/tripleo/+bug/1972765 blocking master 05:24
ysandeep|roversoniya29|ruck: I will investigate and try to fix ^^ , please check gate and rdo promotions pipeline to see if any new issue.05:25
soniya29|ruckysandeep|rover, on it05:25
soniya29|ruckysandeep|rover, gate seems okay..no new issues05:26
ysandeep|roversoniya29|ruck, check looks okay? Can you see if we are also seeing  https://bugs.launchpad.net/tripleo/+bug/1972765 in check for master.05:27
soniya29|ruckysandeep|rover, sure, i will check it05:31
ysandeep|roverlooks like hound is down today - https://codesearch.opendev.org/ - is giving "Hound is not ready."05:35
ysandeep|roverchandankumar, hey o/ few days back we hit similiar issue to https://bugs.launchpad.net/tripleo/+bug/1972765 , I remember we only hit the issue when we pass vexx in --index-url (-i), older bug: https://bugs.launchpad.net/tripleo/+bug/1967943 , do you remember what was the resolution earlier - not mentioned in lp.05:59
chandankumarysandeep|rover: I think we asked dpawlik to clear the pypi mirror cache by restarting the service and it fixed the issue06:03
ysandeep|roverchandankumar, thanks! Let me check with #rhos-ops/ dpawlik 06:04
ysandeep|roversoniya29|ruck, rlandy|out left you a comment about https://bugzilla.redhat.com/show_bug.cgi?id=2083390 in rr hackmd.06:13
soniya29|ruckysandeep|rover, okay, i will check it out06:15
soniya29|ruckneed to CIX that bug06:16
soniya29|ruckysandeep|rover, periodic-tripleo-ci-build-containers-centos-9-push-master has failed for today's run06:18
ysandeep|roversoniya29|ruck, yes https://bugs.launchpad.net/tripleo/+bug/1972765 was for master line failure06:19
ysandeep|roversoniya29|ruck: should be resolved now, let me rekick the master line again.06:19
ysandeep|roverchandankumar++ thanks issue resolved after clearing the pypi mirror cache.06:20
chandankumarysandeep|rover: :-)06:20
soniya29|ruckysandeep|rover, tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001 is also yellow..retry_limit06:20
ysandeep|roversoniya29|ruck, same issue i covered that in same bug report06:21
soniya29|ruckysandeep|rover, ack06:21
ysandeep|roverovb was also failing on pip install https://logserver.rdoproject.org/97/841197/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/df87632/job-output.txt06:21
ysandeep|roversoniya29|ruck, lets sync to divide the work for today and conquer the ci world :D06:52
soniya29|ruckysandeep|rover, meet.google.com/dao-rsha-fdo06:52
ysandeep|roversoniya29|ruck: joined06:53
ysandeep|roversoniya29|ruck, https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/upstream.yaml#L16806:59
ysandeep|roversoniya29|ruck, https://review.rdoproject.org/zuul/buildset/6e5e1ac5ffaa42eaa40f61f4c1b895c307:03
ysandeep|roversoniya29|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-victoria/c6bbe44/job-output.txt07:04
ysandeep|rover ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/v2/collections/ansible/netcommon/': The read operation timed out07:04
soniya29|ruckysandeep|rover, https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-victoria/c54d9de/job-output.txt07:08
soniya29|ruckysandeep|rover, https://review.rdoproject.org/r/c/testproject/+/4267607:15
ysandeep|roversoniya29|ruck, fyi:-07:19
ysandeep|roverWallaby c9: https://review.rdoproject.org/r/c/testproject/+/2844607:19
ysandeep|roverWallaby c8: https://review.rdoproject.org/r/c/testproject/+/3195407:19
*** jpena|off is now known as jpena07:30
ysandeep|roversoniya29|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/fbef9c0/logs/undercloud/var/log/extra/package-list-installed.txt.gz07:39
ysandeep|roverpython3-mimeparse.noarch                     1.6.0-11.el8                               @delorean-train-testing  07:39
*** soniya29|ruck is now known as soniya29|ruck|lunch08:09
*** ykarel is now known as ykarel|away08:25
ysandeep|roversoniya29|ruck|lunch, this will fix bug in rr script for downstream case: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4268808:44
ysandeep|roverreviewbot: please add in review list: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4268808:44
reviewbotI have added your review to the Review list08:44
ysandeep|roverdasm|off, ^^ fyi..08:48
*** ysandeep|rover is now known as ysandeep|rover|lunch08:49
*** soniya29|ruck|lunch is now known as soniya29|ruck09:21
*** bhagyashris is now known as bhagyashris|out09:33
*** ysandeep|rover|lunch is now known as ysandeep|rover09:47
soniya29|ruckysandeep|rover, chasing victoria component line failures here:- https://review.rdoproject.org/r/c/testproject/+/4269209:58
soniya29|ruckmost of them are inconsistent09:58
ysandeep|roversoniya29|ruck, ack.. you got a typo: https://review.rdoproject.org/r/c/testproject/+/42692/1/.zuul.yaml#4 09:59
ysandeep|roverfailed after job name09:59
soniya29|ruckysandeep|rover, done :)10:00
ysandeep|rover++ thanks!10:00
soniya29|ruckysandeep|rover, do we need to monitor skip jobs as well?10:02
ysandeep|roverskip jobs?10:02
soniya29|rucki see couple of skip jobs failing in train c8 line10:02
soniya29|ruckysandeep|rover, for e.g.- periodic-tripleo-ci-centos-8-standalone-skipped-tempest-train10:03
ysandeep|roverI am not sure what are those jobs, let me check if I can find anything from git blame10:03
ysandeep|roversoniya29|ruck, ignore them for now if they are not in criteria10:05
ysandeep|roverchandankumar, marios what are these jobs - periodic-tripleo-ci-centos-8-standalone-skipped-tempest-train , looks like we just don't run tempest in them - run_tempest: false .10:07
ysandeep|roverI wonder if its worth to run duplicate jobs with tempest disabled10:07
chandankumarysandeep|rover: these jobs are added long time back, when we runned skipped tests instead of whitelist tempest tests10:08
chandankumarand those jobs are broken also10:08
chandankumarmay be we can remove these jobs or fix it10:08
ysandeep|roverchandankumar, ack, I will bring in next sprint planning discussion10:09
soniya29|ruckysandeep|rover, https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/47322b8/job-output.txt10:12
soniya29|ruckis this kind of failure known?10:13
soniya29|ruckcs8 fs01 tripleo train is consistently failing with this issue from 2nd of may10:13
ysandeep|roversoniya29|ruck, fs001 only failing in train component? is it passing in other component?10:14
soniya29|ruckysandeep|rover, is passing in network10:15
soniya29|ruckis failing in cloudops but it is not in criteria10:16
ysandeep|roversoniya29|ruck, same failure in cloudops?10:16
soniya29|rucks/cloudops/clients10:16
soniya29|ruckysandeep|rover, sorry, clients10:17
* ysandeep|rover checking10:17
soniya29|ruckysandeep|rover, is passing in baremetal10:17
ysandeep|roversoniya29|ruck, I just analyzed last 5 failure in tripleo component - failures are different earlier - tempest and deployment10:19
ysandeep|roversoniya29|ruck, the issue you pointed - I think that's inconsistent and will possibly clear with a rerun.10:19
ysandeep|roversoniya29|ruck, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train&skip=010:20
ysandeep|roversoniya29|ruck, last run also failed in tempest: https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/3de242f/job-output.txt10:20
ysandeep|roversoniya29|ruck, could you please rerun the job once more - now that vexxhost says - we should see better results on ovb.10:22
soniya29|ruckysandeep|rover, okay, i will rerun it 10:22
mariosysandeep|rover: o/ reading back sorry was getting food 10:23
mariosysandeep|rover: i don't know about those jobs but sounds like chandankumar does 10:24
marioswhere/when are we even running that 10:24
marios(don't answer thinking aloud, checking myself)10:24
soniya29|ruckysandeep|rover, https://review.rdoproject.org/r/c/testproject/+/4269310:25
mariosysandeep|rover: fyi https://github.com/rdo-infra/rdo-jobs/commit/1e9967c5c1ba7c3828e15503629798adc129b7d8 (chkumar's patch that added them)10:25
mariosysandeep|rover: chandankumar: so the idea behind it is to run all the skipped tests ... that is still a good idea i think but lets discuss in next scrum ysandeep|rover 10:26
ysandeep|rovermarios, thanks, they runs in openstack-periodic-weekend , I will discuss in ua/tc sync if they still worth to run and whether to fix/ delete them10:26
mariosk10:27
ysandeep|rovermarios++ thanks! we can discuss in planning/scrum instead as well10:27
mariosysandeep|rover: chandankumar: ack i mean we'd probably need to add c9 versions of those though 10:28
ysandeep|rover+1 but who monitors them - I don't remember monitoring them as rr - and looks like these were added long back... We need to make ruck/rovers aware about these jobs as well.10:31
mariosysandeep|rover: yeah it usually comes back to this... :/10:31
ysandeep|roversoniya29|ruck, https://code.engineering.redhat.com/gerrit/c/testproject/+/211643 tempest component 17/rhel-8 passed 10:33
*** rlandy|out is now known as rlandy10:33
ysandeep|roversoniya29|ruck, and we didn't hit that new mimeparse error, maybe new testtool fixed that as well10:33
soniya29|ruckysandeep|rover, we are saved :)10:34
ysandeep|roversoniya29|ruck, let me recheck one integration line job to confirm.. maybe issue is just in integration line and not in component line10:34
* ysandeep|rover updating tempest cix and bug10:35
soniya29|ruckysandeep|rover, ack10:36
rlandyysandeep|rover: soniya29|ruck: let's sync10:36
ysandeep|roverrlandy, ack10:37
soniya29|ruckrlandy, ack10:37
rlandysoniya29|ruck: chandankumar: ysandeep|rover: I see bhagyashris|out is not here10:37
rlandyI asked her to switch the TC/UA sync10:37
rlandyit's a clash for me10:37
rlandysoniya29|ruck: chandankumar: ysandeep|rover: can we meet an hour later?10:38
chandankumarrlandy: as per RHOS-PTO calender, she is out in second half10:38
chandankumarrlandy: yes works for me10:38
ysandeep|roverrlandy, she is on half day today - second half off10:38
ysandeep|roverrlandy: works for me as well10:38
ysandeep|roverrlandy, soniya29|ruck meet.google.com/fzj-hmum-aeb10:38
soniya29|ruckrlandy, works for me10:38
ysandeep|roverrlandy, https://bugs.launchpad.net/tripleo/+bug/1964940/comments/1510:46
ysandeep|roverrlandy, https://bugzilla.redhat.com/show_bug.cgi?id=2081631#c310:47
ysandeep|roverrlandy, https://review.opendev.org/c/openstack/ovsdbapp/+/84123810:47
ysandeep|roverrlandy, https://trello.com/c/L5svU2p0/2499-cixbz2082632rhos-17rhel-8volume-cinder-and-neutron-setup-class-tests-are-failing-on-tempest-component10:58
ysandeep|roverrlandy, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-security-rhos-17/3e66106/job-output.txt11:00
ysandeep|roverrlandy, soniya29|ruck https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4269411:16
rlandyysandeep|rover: thanks - merging11:17
soniya29|ruckysandeep|rover, thanks11:18
ysandeep|roverrlandy, soniya29|ruck https://review.rdoproject.org/r/c/rdo-jobs/+/42695 temp disabling rbac jobs in master line11:25
*** dviroel|afk is now known as dviroel11:28
chandankumarrlandy: registry sync meeting11:31
rlandychandankumar: will join in 5 - just responding to an email11:31
chandankumarrlandy: ok11:31
rlandyysandeep|rover: thanks  - merged https://review.rdoproject.org/r/c/rdo-jobs/+/4269511:32
* soniya29|ruck needs a break11:57
*** soniya29|ruck is now known as soniya29|ruck|break11:58
*** soniya29|ruck|break is now known as soniya29|ruck12:27
rlandychandankumar: marios: bhagyashri is out and I have a clashing meeting with the community call12:32
rlandychandankumar: marios: can you run that call12:33
rlandyafaik, we don't have any expected guests12:33
rlandyjm1: ^^ unless you have a topic to dicuss12:33
rlandydiscuss12:33
mariosrlandy: sure np12:33
jm1rlandy: no worry, we can move my presentation to next week. it is completely optional, so i am fine :)12:34
rlandyjm1: feel free to carry on w/o me12:35
rlandysorry - some manager thingy12:35
jm1rlandy: ack, let us see who is there in our community meeting12:35
rlandyyeah - you may get a guest12:35
dpawlikTengu: just FYI: after we update the qcow2 base image for centos 8 stream, we don't have the issue with sshd13:23
Tengudpawlik: yep, that's the w/a in CI already.13:27
Tengudpawlik: ysandeep|rover: pushed a patch to fetch the newest image.13:27
dasm|offysandeep|rover: i saw your patch. Thanks!13:31
*** dasm|off is now known as dasm13:31
dasmo/13:31
ysandeep|roverdasm: :)13:33
ysandeep|roverrlandy, when you get a chance could you please merge https://review.rdoproject.org/r/c/rdo-jobs/+/42535 and https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4268813:44
ysandeep|roversoniya29|ruck, rlandy wallaby promoted13:44
soniya29|ruckysandeep|rover, great :)13:44
ysandeep|roverrlandy, soniya29|ruck periodic-tripleo-ci-rhel-8-standalone-rhos-17 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/192c3b6e28834a0c99213541aeeb8a6a : SUCCESS in 1h 20m 48s --> Not seeing mimeparse module missing anymore.. may be testtools fixed this as well or something else, But I am not sure about rca here.13:47
rlandyysandeep|rover: will close out after the line kicks13:48
soniya29|ruckysandeep|rover, ack13:50
dviroeljm1: nice presentation, thanks o/13:53
rlandyysandeep|rover++ train fs001 - success true13:56
rlandyysandeep|rover: you want to revert rbac change? https://review.rdoproject.org/r/c/rdo-jobs/+/42535/1/zuul.d/integration-pipeline-main.yaml13:58
rlandyysandeep|rover: we should revert wallaby change??13:58
soniya29|ruckcentos-9-standalone-full-tempest-api-compute-master is green now13:58
ysandeep|roverrlandy, did I hit revert on wrong patch :D13:58
soniya29|ruckysandeep|rover, rlandy ^^13:59
rlandyysandeep|rover; ack ... pls send the other revert and I'll merge13:59
ysandeep|roverrlandy, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4253613:59
soniya29|ruckbut centos-8-standalone-full-tempest-api-compute-victoria is still failing for this issue - https://bugs.launchpad.net/tripleo/+bug/197216314:00
rlandychandankumar: soniya29|ruck: ysandeep|rover: meeting running over - will ping when done14:01
soniya29|ruckrlandy, sure14:01
rlandysorry14:01
rlandypojadhav: hey - just checking in - you have the retro tomorrow14:28
pojadhavrlandy, hello can I swap with other team member if possible..? I need to take PTO tomorrow as well for an important work. just get to know that. I need to travel for that. 14:32
rlandypojadhav: you can ask the next person but it is late to get someone to prep something now14:38
rlandypojadhav: so feel free to see if someone can switch with you14:38
soniya29|ruckperiodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2 is green now14:41
soniya29|ruckysandeep|rover, rlandy ^^14:41
rlandysoniya29|ruck++ yay14:41
rlandythanks14:41
pojadhavrlandy, ack if no one gets ready. I have to do then at any way.  May be the place I am traveling, will have less network speed. idk.. lets see. pinged people personally. will update you before my EOD.14:42
rlandypojadhav: k - let me know14:43
mariosrlandy: pojadhav: i can do it np i can prep something tomorrow morning 14:48
pojadhavmarios, thanks a ton :) actually I have urgent work for that I have to travel otherwise I will not skip this.14:50
pojadhavrlandy, I will swap with marios 14:50
mariosack np pojadhav 14:51
rlandymarios: pojadhav: thank you14:53
*** dviroel is now known as dviroel|lunch|afk14:53
soniya29|ruckrlandy, 16.2 seems okay with all components except compute15:00
rlandysoniya29|ruck: k   thanks - can you check compute when you come in tomorrow?15:05
rlandyand rerun if there is a real issue15:06
soniya29|ruckrlandy, i will rerun them before my eod15:10
rlandysoniya29|ruck++ thank you15:10
soniya29|ruckit looks like 17 rhel9 standalone compute has ssh timed_out issue which is transient15:11
soniya29|ruckrhel8 standalone full tempest api has hit node_failure15:12
soniya29|ruckother all are green15:12
soniya29|ruckrlandy, https://code.engineering.redhat.com/gerrit/c/testproject/+/40759915:16
soniya29|ruckysandeep|rover, ^^15:17
soniya29|ruckrlandy: ysandeep|rover: leaving for the day :)15:19
*** soniya29|ruck is now known as soniya29|out15:20
*** marios is now known as marios|out16:17
ysandeep|roverrlandy, shutdown sequence.. need anything?16:23
*** ysandeep|rover is now known as ysandeep|out16:23
rlandyysandeep|out: should be fine - thanks16:25
ysandeep|outack, thanks, see you tomorrow o/16:25
dasmrlandy: dviroel|lunch|afk hey, I'm reviving elastic recheck. Would you kindly check these and merge? It's gonna bring further changes easier: https://review.opendev.org/q/topic:revive-elastic-recheck16:30
rlandysure - after lunch16:32
dasmrlandy: no worries. it's not urgent16:33
rlandylunch - brb16:37
rlandydasm++ on moving check to uefi - much better results16:38
dasmnice16:41
*** jpena is now known as jpena|off17:17
* dasm is leaving for doc's appointment17:27
*** dasm is now known as dasm|bbl17:27
*** dviroel|lunch|afk is now known as dviroel\18:53
*** dviroel\ is now known as dviroel18:53
* dviroel back18:53
rlandydviroel: dasm|bbl; on review time if anyone wants to join19:04
rlandyfrenzy_friday: dpawlik: pls review https://review.opendev.org/q/topic:revive-elastic-recheck19:21
rlandydviroel and I took a shot but we don't know the code base19:21
* dasm|bbl dasm|around20:13
dasm|bbldviroel: can you +W https://review.opendev.org/c/opendev/elastic-recheck/+/841306 and https://review.opendev.org/c/opendev/elastic-recheck/+/824283 ?20:13
dviroeldasm|bbl: ack - seems good to merge20:27
dasm|bblthx20:27
*** dviroel is now known as dviroel|out21:22
*** rlandy is now known as rlandy|bbl22:17

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!