*** rlandy|bbl is now known as rlandy|out | 01:22 | |
*** soniya29 is now known as soniya29|ruck | 04:38 | |
*** ysandeep|out is now known as ysandeep|rover | 04:42 | |
ysandeep|rover | dasm|off, ++ thanks for https://review.rdoproject.org/r/c/rdo-jobs/+/42673 | 04:53 |
---|---|---|
dasm|off | ysandeep|rover: no worries. i hope it's gonna help a bit. | 05:00 |
dasm|off | i noticed that it's past provisioning, but it still fails on something. | 05:00 |
dasm|off | i didn't investigate a lot, tho. | 05:01 |
ysandeep|rover | dasm|off, it fixed the issue we were hitting in check :D | 05:04 |
ysandeep|rover | dasm|off, don't you sleep :D | 05:05 |
dasm|off | ysandeep|rover: i'm about to get to a bed :) | 05:11 |
ysandeep|rover | dasm|off, good night o/ see you in few hours :D | 05:12 |
dasm|off | ysandeep|rover: have a quiet day! | 05:12 |
soniya29|ruck | ysandeep|rover, hello | 05:21 |
*** pojadhav|afk is now known as pojadhav | 05:23 | |
ysandeep|rover | soniya29|ruck: https://bugs.launchpad.net/tripleo/+bug/1972765 blocking master | 05:24 |
ysandeep|rover | soniya29|ruck: I will investigate and try to fix ^^ , please check gate and rdo promotions pipeline to see if any new issue. | 05:25 |
soniya29|ruck | ysandeep|rover, on it | 05:25 |
soniya29|ruck | ysandeep|rover, gate seems okay..no new issues | 05:26 |
ysandeep|rover | soniya29|ruck, check looks okay? Can you see if we are also seeing https://bugs.launchpad.net/tripleo/+bug/1972765 in check for master. | 05:27 |
soniya29|ruck | ysandeep|rover, sure, i will check it | 05:31 |
ysandeep|rover | looks like hound is down today - https://codesearch.opendev.org/ - is giving "Hound is not ready." | 05:35 |
ysandeep|rover | chandankumar, hey o/ few days back we hit similiar issue to https://bugs.launchpad.net/tripleo/+bug/1972765 , I remember we only hit the issue when we pass vexx in --index-url (-i), older bug: https://bugs.launchpad.net/tripleo/+bug/1967943 , do you remember what was the resolution earlier - not mentioned in lp. | 05:59 |
chandankumar | ysandeep|rover: I think we asked dpawlik to clear the pypi mirror cache by restarting the service and it fixed the issue | 06:03 |
ysandeep|rover | chandankumar, thanks! Let me check with #rhos-ops/ dpawlik | 06:04 |
ysandeep|rover | soniya29|ruck, rlandy|out left you a comment about https://bugzilla.redhat.com/show_bug.cgi?id=2083390 in rr hackmd. | 06:13 |
soniya29|ruck | ysandeep|rover, okay, i will check it out | 06:15 |
soniya29|ruck | need to CIX that bug | 06:16 |
soniya29|ruck | ysandeep|rover, periodic-tripleo-ci-build-containers-centos-9-push-master has failed for today's run | 06:18 |
ysandeep|rover | soniya29|ruck, yes https://bugs.launchpad.net/tripleo/+bug/1972765 was for master line failure | 06:19 |
ysandeep|rover | soniya29|ruck: should be resolved now, let me rekick the master line again. | 06:19 |
ysandeep|rover | chandankumar++ thanks issue resolved after clearing the pypi mirror cache. | 06:20 |
chandankumar | ysandeep|rover: :-) | 06:20 |
soniya29|ruck | ysandeep|rover, tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001 is also yellow..retry_limit | 06:20 |
ysandeep|rover | soniya29|ruck, same issue i covered that in same bug report | 06:21 |
soniya29|ruck | ysandeep|rover, ack | 06:21 |
ysandeep|rover | ovb was also failing on pip install https://logserver.rdoproject.org/97/841197/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/df87632/job-output.txt | 06:21 |
ysandeep|rover | soniya29|ruck, lets sync to divide the work for today and conquer the ci world :D | 06:52 |
soniya29|ruck | ysandeep|rover, meet.google.com/dao-rsha-fdo | 06:52 |
ysandeep|rover | soniya29|ruck: joined | 06:53 |
ysandeep|rover | soniya29|ruck, https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/upstream.yaml#L168 | 06:59 |
ysandeep|rover | soniya29|ruck, https://review.rdoproject.org/zuul/buildset/6e5e1ac5ffaa42eaa40f61f4c1b895c3 | 07:03 |
ysandeep|rover | soniya29|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-victoria/c6bbe44/job-output.txt | 07:04 |
ysandeep|rover | ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/v2/collections/ansible/netcommon/': The read operation timed out | 07:04 |
soniya29|ruck | ysandeep|rover, https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-victoria/c54d9de/job-output.txt | 07:08 |
soniya29|ruck | ysandeep|rover, https://review.rdoproject.org/r/c/testproject/+/42676 | 07:15 |
ysandeep|rover | soniya29|ruck, fyi:- | 07:19 |
ysandeep|rover | Wallaby c9: https://review.rdoproject.org/r/c/testproject/+/28446 | 07:19 |
ysandeep|rover | Wallaby c8: https://review.rdoproject.org/r/c/testproject/+/31954 | 07:19 |
*** jpena|off is now known as jpena | 07:30 | |
ysandeep|rover | soniya29|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/fbef9c0/logs/undercloud/var/log/extra/package-list-installed.txt.gz | 07:39 |
ysandeep|rover | python3-mimeparse.noarch 1.6.0-11.el8 @delorean-train-testing | 07:39 |
*** soniya29|ruck is now known as soniya29|ruck|lunch | 08:09 | |
*** ykarel is now known as ykarel|away | 08:25 | |
ysandeep|rover | soniya29|ruck|lunch, this will fix bug in rr script for downstream case: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42688 | 08:44 |
ysandeep|rover | reviewbot: please add in review list: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42688 | 08:44 |
reviewbot | I have added your review to the Review list | 08:44 |
ysandeep|rover | dasm|off, ^^ fyi.. | 08:48 |
*** ysandeep|rover is now known as ysandeep|rover|lunch | 08:49 | |
*** soniya29|ruck|lunch is now known as soniya29|ruck | 09:21 | |
*** bhagyashris is now known as bhagyashris|out | 09:33 | |
*** ysandeep|rover|lunch is now known as ysandeep|rover | 09:47 | |
soniya29|ruck | ysandeep|rover, chasing victoria component line failures here:- https://review.rdoproject.org/r/c/testproject/+/42692 | 09:58 |
soniya29|ruck | most of them are inconsistent | 09:58 |
ysandeep|rover | soniya29|ruck, ack.. you got a typo: https://review.rdoproject.org/r/c/testproject/+/42692/1/.zuul.yaml#4 | 09:59 |
ysandeep|rover | failed after job name | 09:59 |
soniya29|ruck | ysandeep|rover, done :) | 10:00 |
ysandeep|rover | ++ thanks! | 10:00 |
soniya29|ruck | ysandeep|rover, do we need to monitor skip jobs as well? | 10:02 |
ysandeep|rover | skip jobs? | 10:02 |
soniya29|ruck | i see couple of skip jobs failing in train c8 line | 10:02 |
soniya29|ruck | ysandeep|rover, for e.g.- periodic-tripleo-ci-centos-8-standalone-skipped-tempest-train | 10:03 |
ysandeep|rover | I am not sure what are those jobs, let me check if I can find anything from git blame | 10:03 |
ysandeep|rover | soniya29|ruck, ignore them for now if they are not in criteria | 10:05 |
ysandeep|rover | chandankumar, marios what are these jobs - periodic-tripleo-ci-centos-8-standalone-skipped-tempest-train , looks like we just don't run tempest in them - run_tempest: false . | 10:07 |
ysandeep|rover | I wonder if its worth to run duplicate jobs with tempest disabled | 10:07 |
chandankumar | ysandeep|rover: these jobs are added long time back, when we runned skipped tests instead of whitelist tempest tests | 10:08 |
chandankumar | and those jobs are broken also | 10:08 |
chandankumar | may be we can remove these jobs or fix it | 10:08 |
ysandeep|rover | chandankumar, ack, I will bring in next sprint planning discussion | 10:09 |
soniya29|ruck | ysandeep|rover, https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/47322b8/job-output.txt | 10:12 |
soniya29|ruck | is this kind of failure known? | 10:13 |
soniya29|ruck | cs8 fs01 tripleo train is consistently failing with this issue from 2nd of may | 10:13 |
ysandeep|rover | soniya29|ruck, fs001 only failing in train component? is it passing in other component? | 10:14 |
soniya29|ruck | ysandeep|rover, is passing in network | 10:15 |
soniya29|ruck | is failing in cloudops but it is not in criteria | 10:16 |
ysandeep|rover | soniya29|ruck, same failure in cloudops? | 10:16 |
soniya29|ruck | s/cloudops/clients | 10:16 |
soniya29|ruck | ysandeep|rover, sorry, clients | 10:17 |
* ysandeep|rover checking | 10:17 | |
soniya29|ruck | ysandeep|rover, is passing in baremetal | 10:17 |
ysandeep|rover | soniya29|ruck, I just analyzed last 5 failure in tripleo component - failures are different earlier - tempest and deployment | 10:19 |
ysandeep|rover | soniya29|ruck, the issue you pointed - I think that's inconsistent and will possibly clear with a rerun. | 10:19 |
ysandeep|rover | soniya29|ruck, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train&skip=0 | 10:20 |
ysandeep|rover | soniya29|ruck, last run also failed in tempest: https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/3de242f/job-output.txt | 10:20 |
ysandeep|rover | soniya29|ruck, could you please rerun the job once more - now that vexxhost says - we should see better results on ovb. | 10:22 |
soniya29|ruck | ysandeep|rover, okay, i will rerun it | 10:22 |
marios | ysandeep|rover: o/ reading back sorry was getting food | 10:23 |
marios | ysandeep|rover: i don't know about those jobs but sounds like chandankumar does | 10:24 |
marios | where/when are we even running that | 10:24 |
marios | (don't answer thinking aloud, checking myself) | 10:24 |
soniya29|ruck | ysandeep|rover, https://review.rdoproject.org/r/c/testproject/+/42693 | 10:25 |
marios | ysandeep|rover: fyi https://github.com/rdo-infra/rdo-jobs/commit/1e9967c5c1ba7c3828e15503629798adc129b7d8 (chkumar's patch that added them) | 10:25 |
marios | ysandeep|rover: chandankumar: so the idea behind it is to run all the skipped tests ... that is still a good idea i think but lets discuss in next scrum ysandeep|rover | 10:26 |
ysandeep|rover | marios, thanks, they runs in openstack-periodic-weekend , I will discuss in ua/tc sync if they still worth to run and whether to fix/ delete them | 10:26 |
marios | k | 10:27 |
ysandeep|rover | marios++ thanks! we can discuss in planning/scrum instead as well | 10:27 |
marios | ysandeep|rover: chandankumar: ack i mean we'd probably need to add c9 versions of those though | 10:28 |
ysandeep|rover | +1 but who monitors them - I don't remember monitoring them as rr - and looks like these were added long back... We need to make ruck/rovers aware about these jobs as well. | 10:31 |
marios | ysandeep|rover: yeah it usually comes back to this... :/ | 10:31 |
ysandeep|rover | soniya29|ruck, https://code.engineering.redhat.com/gerrit/c/testproject/+/211643 tempest component 17/rhel-8 passed | 10:33 |
*** rlandy|out is now known as rlandy | 10:33 | |
ysandeep|rover | soniya29|ruck, and we didn't hit that new mimeparse error, maybe new testtool fixed that as well | 10:33 |
soniya29|ruck | ysandeep|rover, we are saved :) | 10:34 |
ysandeep|rover | soniya29|ruck, let me recheck one integration line job to confirm.. maybe issue is just in integration line and not in component line | 10:34 |
* ysandeep|rover updating tempest cix and bug | 10:35 | |
soniya29|ruck | ysandeep|rover, ack | 10:36 |
rlandy | ysandeep|rover: soniya29|ruck: let's sync | 10:36 |
ysandeep|rover | rlandy, ack | 10:37 |
soniya29|ruck | rlandy, ack | 10:37 |
rlandy | soniya29|ruck: chandankumar: ysandeep|rover: I see bhagyashris|out is not here | 10:37 |
rlandy | I asked her to switch the TC/UA sync | 10:37 |
rlandy | it's a clash for me | 10:37 |
rlandy | soniya29|ruck: chandankumar: ysandeep|rover: can we meet an hour later? | 10:38 |
chandankumar | rlandy: as per RHOS-PTO calender, she is out in second half | 10:38 |
chandankumar | rlandy: yes works for me | 10:38 |
ysandeep|rover | rlandy, she is on half day today - second half off | 10:38 |
ysandeep|rover | rlandy: works for me as well | 10:38 |
ysandeep|rover | rlandy, soniya29|ruck meet.google.com/fzj-hmum-aeb | 10:38 |
soniya29|ruck | rlandy, works for me | 10:38 |
ysandeep|rover | rlandy, https://bugs.launchpad.net/tripleo/+bug/1964940/comments/15 | 10:46 |
ysandeep|rover | rlandy, https://bugzilla.redhat.com/show_bug.cgi?id=2081631#c3 | 10:47 |
ysandeep|rover | rlandy, https://review.opendev.org/c/openstack/ovsdbapp/+/841238 | 10:47 |
ysandeep|rover | rlandy, https://trello.com/c/L5svU2p0/2499-cixbz2082632rhos-17rhel-8volume-cinder-and-neutron-setup-class-tests-are-failing-on-tempest-component | 10:58 |
ysandeep|rover | rlandy, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-security-rhos-17/3e66106/job-output.txt | 11:00 |
ysandeep|rover | rlandy, soniya29|ruck https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42694 | 11:16 |
rlandy | ysandeep|rover: thanks - merging | 11:17 |
soniya29|ruck | ysandeep|rover, thanks | 11:18 |
ysandeep|rover | rlandy, soniya29|ruck https://review.rdoproject.org/r/c/rdo-jobs/+/42695 temp disabling rbac jobs in master line | 11:25 |
*** dviroel|afk is now known as dviroel | 11:28 | |
chandankumar | rlandy: registry sync meeting | 11:31 |
rlandy | chandankumar: will join in 5 - just responding to an email | 11:31 |
chandankumar | rlandy: ok | 11:31 |
rlandy | ysandeep|rover: thanks - merged https://review.rdoproject.org/r/c/rdo-jobs/+/42695 | 11:32 |
* soniya29|ruck needs a break | 11:57 | |
*** soniya29|ruck is now known as soniya29|ruck|break | 11:58 | |
*** soniya29|ruck|break is now known as soniya29|ruck | 12:27 | |
rlandy | chandankumar: marios: bhagyashri is out and I have a clashing meeting with the community call | 12:32 |
rlandy | chandankumar: marios: can you run that call | 12:33 |
rlandy | afaik, we don't have any expected guests | 12:33 |
rlandy | jm1: ^^ unless you have a topic to dicuss | 12:33 |
rlandy | discuss | 12:33 |
marios | rlandy: sure np | 12:33 |
jm1 | rlandy: no worry, we can move my presentation to next week. it is completely optional, so i am fine :) | 12:34 |
rlandy | jm1: feel free to carry on w/o me | 12:35 |
rlandy | sorry - some manager thingy | 12:35 |
jm1 | rlandy: ack, let us see who is there in our community meeting | 12:35 |
rlandy | yeah - you may get a guest | 12:35 |
dpawlik | Tengu: just FYI: after we update the qcow2 base image for centos 8 stream, we don't have the issue with sshd | 13:23 |
Tengu | dpawlik: yep, that's the w/a in CI already. | 13:27 |
Tengu | dpawlik: ysandeep|rover: pushed a patch to fetch the newest image. | 13:27 |
dasm|off | ysandeep|rover: i saw your patch. Thanks! | 13:31 |
*** dasm|off is now known as dasm | 13:31 | |
dasm | o/ | 13:31 |
ysandeep|rover | dasm: :) | 13:33 |
ysandeep|rover | rlandy, when you get a chance could you please merge https://review.rdoproject.org/r/c/rdo-jobs/+/42535 and https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42688 | 13:44 |
ysandeep|rover | soniya29|ruck, rlandy wallaby promoted | 13:44 |
soniya29|ruck | ysandeep|rover, great :) | 13:44 |
ysandeep|rover | rlandy, soniya29|ruck periodic-tripleo-ci-rhel-8-standalone-rhos-17 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/192c3b6e28834a0c99213541aeeb8a6a : SUCCESS in 1h 20m 48s --> Not seeing mimeparse module missing anymore.. may be testtools fixed this as well or something else, But I am not sure about rca here. | 13:47 |
rlandy | ysandeep|rover: will close out after the line kicks | 13:48 |
soniya29|ruck | ysandeep|rover, ack | 13:50 |
dviroel | jm1: nice presentation, thanks o/ | 13:53 |
rlandy | ysandeep|rover++ train fs001 - success true | 13:56 |
rlandy | ysandeep|rover: you want to revert rbac change? https://review.rdoproject.org/r/c/rdo-jobs/+/42535/1/zuul.d/integration-pipeline-main.yaml | 13:58 |
rlandy | ysandeep|rover: we should revert wallaby change?? | 13:58 |
soniya29|ruck | centos-9-standalone-full-tempest-api-compute-master is green now | 13:58 |
ysandeep|rover | rlandy, did I hit revert on wrong patch :D | 13:58 |
soniya29|ruck | ysandeep|rover, rlandy ^^ | 13:59 |
rlandy | ysandeep|rover; ack ... pls send the other revert and I'll merge | 13:59 |
ysandeep|rover | rlandy, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42536 | 13:59 |
soniya29|ruck | but centos-8-standalone-full-tempest-api-compute-victoria is still failing for this issue - https://bugs.launchpad.net/tripleo/+bug/1972163 | 14:00 |
rlandy | chandankumar: soniya29|ruck: ysandeep|rover: meeting running over - will ping when done | 14:01 |
soniya29|ruck | rlandy, sure | 14:01 |
rlandy | sorry | 14:01 |
rlandy | pojadhav: hey - just checking in - you have the retro tomorrow | 14:28 |
pojadhav | rlandy, hello can I swap with other team member if possible..? I need to take PTO tomorrow as well for an important work. just get to know that. I need to travel for that. | 14:32 |
rlandy | pojadhav: you can ask the next person but it is late to get someone to prep something now | 14:38 |
rlandy | pojadhav: so feel free to see if someone can switch with you | 14:38 |
soniya29|ruck | periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2 is green now | 14:41 |
soniya29|ruck | ysandeep|rover, rlandy ^^ | 14:41 |
rlandy | soniya29|ruck++ yay | 14:41 |
rlandy | thanks | 14:41 |
pojadhav | rlandy, ack if no one gets ready. I have to do then at any way. May be the place I am traveling, will have less network speed. idk.. lets see. pinged people personally. will update you before my EOD. | 14:42 |
rlandy | pojadhav: k - let me know | 14:43 |
marios | rlandy: pojadhav: i can do it np i can prep something tomorrow morning | 14:48 |
pojadhav | marios, thanks a ton :) actually I have urgent work for that I have to travel otherwise I will not skip this. | 14:50 |
pojadhav | rlandy, I will swap with marios | 14:50 |
marios | ack np pojadhav | 14:51 |
rlandy | marios: pojadhav: thank you | 14:53 |
*** dviroel is now known as dviroel|lunch|afk | 14:53 | |
soniya29|ruck | rlandy, 16.2 seems okay with all components except compute | 15:00 |
rlandy | soniya29|ruck: k thanks - can you check compute when you come in tomorrow? | 15:05 |
rlandy | and rerun if there is a real issue | 15:06 |
soniya29|ruck | rlandy, i will rerun them before my eod | 15:10 |
rlandy | soniya29|ruck++ thank you | 15:10 |
soniya29|ruck | it looks like 17 rhel9 standalone compute has ssh timed_out issue which is transient | 15:11 |
soniya29|ruck | rhel8 standalone full tempest api has hit node_failure | 15:12 |
soniya29|ruck | other all are green | 15:12 |
soniya29|ruck | rlandy, https://code.engineering.redhat.com/gerrit/c/testproject/+/407599 | 15:16 |
soniya29|ruck | ysandeep|rover, ^^ | 15:17 |
soniya29|ruck | rlandy: ysandeep|rover: leaving for the day :) | 15:19 |
*** soniya29|ruck is now known as soniya29|out | 15:20 | |
*** marios is now known as marios|out | 16:17 | |
ysandeep|rover | rlandy, shutdown sequence.. need anything? | 16:23 |
*** ysandeep|rover is now known as ysandeep|out | 16:23 | |
rlandy | ysandeep|out: should be fine - thanks | 16:25 |
ysandeep|out | ack, thanks, see you tomorrow o/ | 16:25 |
dasm | rlandy: dviroel|lunch|afk hey, I'm reviving elastic recheck. Would you kindly check these and merge? It's gonna bring further changes easier: https://review.opendev.org/q/topic:revive-elastic-recheck | 16:30 |
rlandy | sure - after lunch | 16:32 |
dasm | rlandy: no worries. it's not urgent | 16:33 |
rlandy | lunch - brb | 16:37 |
rlandy | dasm++ on moving check to uefi - much better results | 16:38 |
dasm | nice | 16:41 |
*** jpena is now known as jpena|off | 17:17 | |
* dasm is leaving for doc's appointment | 17:27 | |
*** dasm is now known as dasm|bbl | 17:27 | |
*** dviroel|lunch|afk is now known as dviroel\ | 18:53 | |
*** dviroel\ is now known as dviroel | 18:53 | |
* dviroel back | 18:53 | |
rlandy | dviroel: dasm|bbl; on review time if anyone wants to join | 19:04 |
rlandy | frenzy_friday: dpawlik: pls review https://review.opendev.org/q/topic:revive-elastic-recheck | 19:21 |
rlandy | dviroel and I took a shot but we don't know the code base | 19:21 |
* dasm|bbl dasm|around | 20:13 | |
dasm|bbl | dviroel: can you +W https://review.opendev.org/c/opendev/elastic-recheck/+/841306 and https://review.opendev.org/c/opendev/elastic-recheck/+/824283 ? | 20:13 |
dviroel | dasm|bbl: ack - seems good to merge | 20:27 |
dasm|bbl | thx | 20:27 |
*** dviroel is now known as dviroel|out | 21:22 | |
*** rlandy is now known as rlandy|bbl | 22:17 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!