| *** raukadah is now known as chandankumar | 03:40 | |
| *** raukadah is now known as chandankumar | 04:22 | |
| *** ysandeep|out is now known as ysandeep | 04:26 | |
| ysandeep | akahat|rover, Hey o/ downstream line are all queued up in zuul - waiting for more than 20 hours, As per email on outage list - some minor update is going on - I will dequeue some line so that atleast new runs can run today. | 05:00 |
|---|---|---|
| ysandeep | rlandy, bhagyashris: Do you want to cancel today's scrum call for PTG. | 05:33 |
| marios | akahat|rover: o/ morning let me know if you need something (any reviews for bugs? anything else to look at ) | 05:52 |
| marios | chandankumar: thanks for setting up a meet ack | 05:53 |
| bhagyashris | ysandeep, yeah looking at PTG schedule https://etherpad.opendev.org/p/tripleo-zed-ptg-topics i think i will need to cancel the meeting till wednesday | 06:18 |
| ysandeep | +1, Atleast we should cancel till tomorrow, and decide for wednesday on Tuesday's evening after PTG | 06:19 |
| bhagyashris | ysandeep, ack | 06:19 |
| bhagyashris | marios, chandankumar ^ | 06:19 |
| marios | bhagyashris: +1 | 06:20 |
| bhagyashris | ack | 06:20 |
| bhagyashris | thanks | 06:20 |
| chandankumar | bhagyashris: ack | 06:20 |
| bhagyashris | ysandeep, chandankumar marios hey Monday's scrum is invited/schedule by ronnelle i mean she is the owner of monday's scrum so i can;t cancel that meeting i will ask ronnelle to cancel that one when she will come online | 06:25 |
| ysandeep | bhagyashris: ack, sounds good | 06:25 |
| bhagyashris | same with community meting it's owned by RHOS team meetings | 06:26 |
| marios | bhagyashris: k | 06:36 |
| ysandeep | bhagyashris, I have cancelled tripleo ci community mtg - i have access for RHOS team meetings calendar | 06:42 |
| bhagyashris | ysandeep, ack thanks | 06:42 |
| ysandeep | bhagyashris, can you confirm if its removed from your calendar view as well. | 06:42 |
| bhagyashris | ysandeep, yes removed | 06:43 |
| bhagyashris | and received mail as well | 06:43 |
| ysandeep | great, thanks! | 06:43 |
| bhagyashris | thanks ! | 06:43 |
| *** ysandeep is now known as ysandeep|afk | 06:49 | |
| akahat|rover | ysandeep|afk, thanks for the update. Downstream queue status looks good now. | 06:55 |
| akahat|rover | we have some problem with the upstream though.. i could see there are some pipelines waiting for more than 12 hours. | 06:56 |
| akahat|rover | s/upstream/rdo | 06:56 |
| akahat|rover | those also needs to dequeue? | 06:57 |
| akahat|rover | marios, chandankumar ysandeep|afk ^^ | 06:57 |
| akahat|rover | Pipelines: openstack-periodic-integration-stable1 | 06:57 |
| akahat|rover | openstack-periodic-integration-stable4 | 06:57 |
| akahat|rover | openstack-component-tempest | 06:58 |
| akahat|rover | openstack-promote-component | 06:58 |
| marios | akahat|rover: looking | 07:00 |
| marios | akahat|rover: yeah looks like openstack-periodic-integration-stable1 is stuck ... waiting on periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-wallabyqueued | 07:01 |
| marios | akahat|rover: same with openstack-periodic-integration-stable4 1 | 07:02 |
| marios | clearly there was some problem | 07:02 |
| marios | akahat|rover: so first go ask in rhos-ops too... see if there are any known issues with vexx etc... and yes i think you should dequeue... the problem however is | 07:02 |
| marios | akahat|rover: you don't want to requeue them all at the same time we will cause problems again | 07:02 |
| jpodivin | akahat|rover: I'm seeing lot of subsequent failures in the periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master and periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master-validation jobs, both during the overcloud deployment with "Evaluation Error: Error while evaluating a Resource Statement". Are we tracking this anywhere? | 07:03 |
| akahat|rover | marios, yeah.. i'll dequeue all the mentioned lines. and schedule only cs9 for now. | 07:04 |
| marios | akahat|rover: ack | 07:04 |
| akahat|rover | jpodivin, hey.. let me take a look. | 07:04 |
| jpodivin | akahat|rover: here you go https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master/40b5a1a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 07:05 |
| jpodivin | https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master-validation/d6c8202/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 07:05 |
| *** amoralej|off is now known as amoralej | 07:06 | |
| jpodivin | akahat|rover: it's possible the same problem is in other places, but these are the ones I know of. | 07:06 |
| akahat|rover | jparker, yes we are tracking this issue. | 07:08 |
| akahat|rover | https://bugs.launchpad.net/tripleo/+bug/1966625 | 07:09 |
| akahat|rover | jpodivin, we are waiting to get this merged: https://github.com/saz/puppet-ssh/pull/326 | 07:09 |
| jpodivin | akahat|rover: I see. Thank you very much. | 07:13 |
| *** jpena|off is now known as jpena | 07:36 | |
| marios | chandankumar: joining | 08:30 |
| marios | back in a bit | 09:09 |
| *** ysandeep|afk is now known as ysandeep | 09:11 | |
| *** arxcruz|out is now known as arxcruz | 09:34 | |
| *** jpena is now known as jpena|off | 09:35 | |
| akahat|rover | jpodivin, we are waiting to get this merged: https://github.com/saz/puppet-ssh/pull/326 | 09:38 |
| akahat|rover | jpodivin, sorry please ignore. | 09:38 |
| *** chem is now known as Guest1161 | 09:39 | |
| jpodivin | akahat|rover: I know ;) If there's anything I can do to help it along ping me. | 09:39 |
| *** jpena|off is now known as jpena | 09:44 | |
| rlandy | ysandeep: bhagyashris: ack - let's cancel scrum | 10:21 |
| bhagyashris | rlandy, yeah we had discussion about that | 10:21 |
| bhagyashris | rlandy, you are the owner of monday's scrum so could you plz cancel the meeting | 10:22 |
| bhagyashris | i dont have righs to cancel the monday's scrum | 10:22 |
| bhagyashris | rights | 10:22 |
| rlandy | done | 10:22 |
| bhagyashris | rlandy, thanks ! | 10:22 |
| rlandy | akahat|rover: hello ... | 10:24 |
| ysandeep | bhagyashris, psi is in bad shape, recheck won't help | 10:25 |
| rlandy | bhagyashris: pls see #rho-sops | 10:25 |
| rlandy | rhos-ops | 10:26 |
| rlandy | asking to stop all lines so we don;t have to keep clearing them | 10:26 |
| rlandy | akahat|rover: all master component jobs are failing | 10:26 |
| rlandy | akahat|rover: let's sync when you are back pls | 10:28 |
| rlandy | requeued train c7 line | 10:28 |
| rlandy | we need to keep train 8 clean now | 10:28 |
| rlandy | arxcruz; you around? | 10:28 |
| rlandy | arxcruz: can you pls check that the current-triple containers exist in rdo reg | 10:28 |
| rlandy | frenzy_friday: you back?? | 10:29 |
| rlandy | frenzy_friday: if so, we need to get a move on with moving the rdo registry - rcastillo will help you there | 10:29 |
| frenzy_friday | rlandy, yep | 10:29 |
| rlandy | frenzy_friday: ussuri will be gone | 10:29 |
| rlandy | frenzy_friday: you can work on moving victori | 10:29 |
| rlandy | first | 10:30 |
| rlandy | then all c9 | 10:30 |
| frenzy_friday | cool, updating my ussuri patch | 10:30 |
| rlandy | frenzy_friday: pls ping rcastillo when he is in to sync | 10:30 |
| rlandy | chandankumar: need to meet with you pls | 10:30 |
| rlandy | pls ping when you are around | 10:30 |
| akahat|rover | rlandy, o/ i'm here... let's sync | 10:32 |
| rlandy | akahat|rover: https://meet.google.com/wgh-ihgq-frx | 10:33 |
| akahat|rover | rcastillo, https://meet.google.com/btf-nodi-hek | 10:33 |
| akahat|rover | rlandy, joining | 10:33 |
| akahat|rover | rcastillo, please ignore. | 10:33 |
| akahat|rover | rlandy, https://hackmd.io/M1UXHF7iTbmJFkK4bQET9g | 10:34 |
| rlandy | arxcruz: it's the wallaby c9 containers | 10:35 |
| ysandeep | pojadhav, Have you mins, have a question about development cockpit | 10:37 |
| pojadhav | ysandeep, sure | 10:37 |
| rlandy | arxcruz: pls ping when you are in | 10:38 |
| rlandy | jm1: can you check that the toolbox tools are up and running | 10:39 |
| rlandy | looks like arxcruz's container copy script is not working | 10:39 |
| ysandeep | pojadhav, meet.google.com/hyv-fcdu-esq | 10:40 |
| arxcruz | rlandy pong | 10:40 |
| rlandy | arxcruz: pls join https://meet.google.com/wgh-ihgq-frx | 10:40 |
| rlandy | arxcruz: b491db10e878b4375b42c7fb01a4be43 | 10:42 |
| *** pojadhav- is now known as pojadhav | 10:58 | |
| chandankumar | rlandy: hello | 10:59 |
| chandankumar | rlandy: let me know when we can meet | 10:59 |
| rlandy | chandankumar: was chatting with ruck/rovers | 11:05 |
| rlandy | will meet with you after review time | 11:05 |
| chandankumar | rlandy: ok | 11:05 |
| *** pojadhav- is now known as pojadhav | 11:13 | |
| marios | i dropped from review time let me know if i should rejoin o/ | 11:16 |
| * marios went through those already | 11:17 | |
| rlandy | marios: I am on review time | 11:17 |
| marios | rlandy: k coming | 11:17 |
| rlandy | marios: let's just eync re: teardown | 11:17 |
| marios | akahat|rover: dasm|ruck|off: 13:53 #tripleo: < opendevreview> Marios Andreou proposed openstack/tripleo-ci master: Centos8 wallaby jobs teardown - deployment image build jobs | 11:20 |
| marios | https://review.opendev.org/c/openstack/tripleo-ci/+/834186 | 11:20 |
| rlandy | akahat|rover: we need a bug on https://77f3d3f1b9b55b1befcf-40416aec901d1e1b0fbe6fedfed92f1f.ssl.cf2.rackcdn.com/834186/5/check/tripleo-tox-molecule/975d36a/job-output.txt | 11:20 |
| marios | sorry wrong paste :/ | 11:20 |
| marios | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-tox-molecule&project=openstack/tripleo-ci | 11:20 |
| marios | akahat|rover: dasm|ruck|off: ^^^^ | 11:21 |
| rlandy | amoralej: ^^ fyi | 11:21 |
| akahat|rover | marios, rlandy okay. reporting bug. | 11:21 |
| marios | akahat|rover: thanks | 11:22 |
| marios | :/ new tripleo-ci gate blocker | 11:22 |
| marios | also tripleo-quickstart so not a general tripleo blocker but... | 11:23 |
| marios | still | 11:23 |
| marios | thanks | 11:23 |
| rlandy | amoralej: | 11:24 |
| rlandy | 2022-04-04 11:08:13.059898 | primary | [31m - cannot install both platform-python-3.6.8-46.el8.x86_64 and platform-python-3.6.8-45.el8.x86_64[0m | 11:24 |
| rlandy | 2022-04-04 11:08:13.059914 | primary | [31m - cannot install the best update candidate for package python36-devel-3.6.8-2.module_el8.5.0+751+b79b40be.x86_64[0m | 11:24 |
| rlandy | 2022-04-04 11:08:13.059930 | primary | [31m - cannot install the best update candidate for package platform-python-3.6.8-37.el8.x86_64[0m | 11:24 |
| akahat|rover | https://bugs.launchpad.net/tripleo/+bug/1967743 | 11:31 |
| *** pojadhav is now known as pojadhav|afk | 11:34 | |
| *** dviroel|out is now known as dviroel | 11:38 | |
| arxcruz | rlandy marios akahat|rover https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41439 | 11:38 |
| arxcruz | updating to run twice a day | 11:39 |
| marios | arxcruz: thanks will check in a sec | 11:40 |
| rlandy | arxcruz: thank you | 11:43 |
| rlandy | akahat|rover: hello - let's chat briefly about bug | 11:45 |
| jm1 | arxcruz, rlandy: so toolbox is working as expected? | 11:45 |
| rlandy | actually - I'll update | 11:46 |
| akahat|rover | rlandy, okay | 11:46 |
| arxcruz | jm1 yes, just it copied before the prune | 11:46 |
| amoralej | rlandy, looks like the usual unsync between baseos and appstream but that shouldn't happen when using AFS mirrors | 11:46 |
| rlandy | amoralej: so we're still not using AFS mirrors?? | 11:47 |
| rlandy | checking | 11:47 |
| amoralej | i hope we are ... | 11:47 |
| rlandy | this is tox | 11:48 |
| amoralej | that happens only in the molecule jobs? | 11:48 |
| rlandy | we don't do repo setup | 11:48 |
| rlandy | ack | 11:48 |
| rlandy | https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-tox-molecule&skip=0 | 11:48 |
| rlandy | one just went green again | 11:49 |
| amoralej | btw, you plan to move those jobs to centos9? rdo on centos0 is about to die | 11:49 |
| amoralej | yeah, will fix by itself probably when repos get on sync | 11:49 |
| rlandy | ok - I guess we should | 11:49 |
| amoralej | i meant, rdo on centos8 is about to die | 11:49 |
| rlandy | rcastillo: ^^ when you are in | 11:49 |
| rlandy | dasm|ruck|off: ^^ when you guys are in | 11:49 |
| amoralej | yoga is the last release of rdo on centos8 | 11:49 |
| rlandy | amoralej: ack - asking rcastillo and dasm|ruck|off to take care of moving those jobs to c9 | 11:50 |
| amoralej | ack | 11:50 |
| rlandy | marios: we got one green run - so will recheck you job later | 11:50 |
| amoralej | that's the only leftover from centos8 in tripleo-ci for master, right? | 11:50 |
| rlandy | may clear | 11:50 |
| rlandy | I think so - we will have to audir | 11:50 |
| rlandy | audit | 11:50 |
| marios | rlandy: k thanks no worries as discussed those are not burning ... glad we got standalone through so we start merging them by end of week thanks | 11:52 |
| jm1 | arxcruz, rlandy: fyi we have a jira card 894 for sync'ing our ci-infra roles with our toolbox instance. we have jira cards for other vexxhost instances as well. in case anyone is changing instances beware that playbooks might not be up to date atm. we should work on that... | 11:54 |
| rlandy | ysandeep: pojadhav: bhagyashris: pls hold on kicking any ovb jobs in downstream | 12:02 |
| rlandy | trying today to get components cleaned up | 12:02 |
| ysandeep | rlandy, ack.. will not run ovb, just baremetal job - that will only take 1 node from zuul | 12:03 |
| rlandy | ysandeep: we tried on friday | 12:03 |
| rlandy | and then cloud went down again | 12:03 |
| rlandy | we should have promoted tripleo and network then | 12:03 |
| rlandy | :( | 12:03 |
| rlandy | crying | 12:03 |
| ysandeep | rlandy, 17/rhel9 hitting legit failure: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/d1e451b/logs/container-builds/aabc549e-26f6-4ff9-ae65-329c3a5b6d36/base/ovn-bgp-agent/ovn-bgp-agent-build.log , akahat|rover dasm|ruck|off please investigate when time allows. | 12:03 |
| rlandy | ysandeep: I remember that failure | 12:04 |
| akahat|rover | ysandeep, ack | 12:04 |
| rlandy | we looked at that in upstream | 12:04 |
| rlandy | we spoke about it on friday | 12:04 |
| rlandy | getting | 12:04 |
| rlandy | ysandeep: it's not new | 12:04 |
| ysandeep | I thought tripleo component will clear that, but it didn't. | 12:05 |
| rlandy | ysandeep: it didn;t promote | 12:05 |
| akahat|rover | arxcruz, https://review.rdoproject.org/r/c/testproject/+/41434 | 12:05 |
| rlandy | ysandeep: died on friday | 12:05 |
| *** amoralej is now known as amoralej|lunch | 12:06 | |
| rlandy | ysandeep: that is why running component promotion now | 12:06 |
| rlandy | to see what is still a real issue | 12:06 |
| ysandeep | rlandy, ack, but the patch I had my eyes(tripleo-common) is already is already in. | 12:08 |
| ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/d1e451b/logs/system/installed_pkgs.txt | 12:08 |
| ysandeep | penstack-tripleo-common.noarch 15.4.1-0.20220328184445.0c754c6.el9osttrunk @delorean-component-tripleo | 12:08 |
| ysandeep | https://github.com/openstack/tripleo-common/commits/stable/wallaby | 12:08 |
| ysandeep | I was thinking https://github.com/openstack/tripleo-common/commit/0c754c6059f64b388e7497f4f30b52e526a232ed will fix the issue, but looks like we already have that in latest run above ^^ | 12:09 |
| rlandy | periodic-tripleo-rhel-9-rhos-17-component-tripleo-promote-to-promoted-components - success | 12:10 |
| rlandy | periodic-tripleo-rhel-8-rhos-17-component-tripleo-promote-to-promoted-components - failed | 12:10 |
| ysandeep | yes tripleo - 17 on rhel-9 was promoted - last_promotion=2022-04-01 14:34:42 | 12:10 |
| pojadhav- | rlandy, ack | 12:12 |
| rlandy | ok - it looks like these are the only failing components: | 12:14 |
| rlandy | periodic-tripleo-rhel-8-rhos-17-component-clients-promote-to-promoted-components | 12:14 |
| rlandy | periodic-tripleo-rhel-8-rhos-17-component-network-promote-to-promoted-components | 12:14 |
| rlandy | periodic-tripleo-rhel-8-rhos-17-component-tripleo-promote-to-promoted-components | 12:14 |
| rlandy | periodic-tripleo-rhel-9-rhos-17-component-clients-promote-to-promoted-components | 12:14 |
| rlandy | periodic-tripleo-rhel-8-rhos-16.2-component-network-promote-to-promoted-components | 12:14 |
| rlandy | akahat|rover: ysandeep: ^^ | 12:14 |
| rlandy | working on network 16.2 now | 12:14 |
| rlandy | then will rekick 16.e line | 12:14 |
| rlandy | https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/pipeline_component-network-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/181/ is the issue | 12:15 |
| bhagyashris | ack | 12:21 |
| rlandy | akahat|rover: pls investigate the failures on stable-1 | 12:23 |
| rlandy | rerun eth failed jobs in the mean time | 12:23 |
| rlandy | bug of real errors | 12:23 |
| akahat|rover | rlandy, yes. i'm already on i.t | 12:23 |
| rlandy | we need to promote that ASAP | 12:23 |
| rlandy | thank you | 12:23 |
| rcastillo | o/ | 12:25 |
| ysandeep | rlandy, akahat|rover ahh, looks like the patch I pointed above fixed one issue and now we are hitting another issue(deps issue now) | 12:29 |
| ysandeep | earlier error was:- | 12:29 |
| ysandeep | ~~~ | 12:29 |
| ysandeep | User ovn-bgp was not found in the supported list | 12:29 |
| ysandeep | ~~~ | 12:29 |
| ysandeep | Now error is:- | 12:29 |
| ysandeep | ~~~ | 12:29 |
| ysandeep | Error: Unable to find a match: ovn-bgp-agent | 12:29 |
| ysandeep | ~~~ | 12:29 |
| rlandy | ysandeep: rekicked clients line while waiting for network jenkins jobs to complete | 12:30 |
| rlandy | might kill that to start 16.2 | 12:30 |
| rlandy | we'll see | 12:30 |
| ysandeep | rlandy, ack.. let me know incase you need a hand | 12:31 |
| rlandy | only component missing promo in rhel 9 | 12:31 |
| * ysandeep reporting a bug for 17 on rhel-9, since I already know what the issue is. | 12:31 | |
| rlandy | then we will just have rhl-8 on 17 components | 12:31 |
| ysandeep | nice | 12:32 |
| ysandeep | rlandy, dasm|ruck|off akahat|rover https://bugzilla.redhat.com/show_bug.cgi?id=2071637 , cix sent | 12:38 |
| akahat|rover | ysandeep, ack | 12:39 |
| *** ysandeep is now known as ysandeep|break | 12:40 | |
| akahat|rover | rlandy, we will hit the jobs failures issue for c9, this hash c0c20385de3cb5e15090899f9bf98327 is missing from: trunk.registry.rdoproject.org. | 12:43 |
| akahat|rover | I've added containers build job, it will build the containers: https://review.rdoproject.org/r/c/testproject/+/41434/3/.zuul.yaml | 12:44 |
| akahat|rover | once that build jobs will get passed. | 12:44 |
| *** pojadhav- is now known as pojadhav | 12:45 | |
| rlandy | akahat|rover: post failures and some tempest one | 12:48 |
| rlandy | akahat|rover: you don;t need container builds | 12:49 |
| rlandy | arxcruz, copied them back | 12:49 |
| akahat|rover | rlandy, agreed but hash is promoted today.. and containers build jobs did not ran | 12:50 |
| rlandy | akahat|rover: we'r emissing the internal kvm jobs | 12:50 |
| akahat|rover | so no containers | 12:50 |
| rlandy | akahat|rover: https://meet.google.com/oxd-zaap-grg?pli=1&authuser=0 | 12:52 |
| *** jpena is now known as jpena|off | 12:54 | |
| akahat|rover | rlandy, https://review.rdoproject.org/zuul/status#41278 | 12:57 |
| *** ysandeep|break is now known as ysandeep | 13:00 | |
| rlandy | PTG time | 13:01 |
| *** jpena|off is now known as jpena | 13:03 | |
| *** beagles is now known as beagles_ptg | 13:06 | |
| rlandy | akahat|rover: re: https://review.rdoproject.org/r/c/testproject/+/41443/1/.zuul.yaml - pls use the rr tool | 13:08 |
| rlandy | fs039 and fs064 passed | 13:08 |
| rlandy | pls use the hash | 13:08 |
| *** amoralej|lunch is now known as amoralej | 13:08 | |
| rlandy | that rr toll reports to you | 13:08 |
| akahat|rover | rlandy, okay | 13:08 |
| rlandy | akahat|rover: otherwise the hash changes | 13:08 |
| rlandy | you should need 4 jobs plus one downstream | 13:09 |
| rlandy | akahat|rover: adding test project for kvm job | 13:24 |
| akahat|rover | rlandy, here: https://code.engineering.redhat.com/gerrit/c/testproject/+/301027 | 13:24 |
| rlandy | akahat|rover: looks good | 13:25 |
| rlandy | will kill mine | 13:25 |
| akahat|rover | rlandy, https://review.rdoproject.org/zuul/status#41437 | 13:41 |
| rlandy | akahat|rover:k - what is your concern about that? | 13:43 |
| rlandy | specifying a hash should help | 13:43 |
| akahat|rover | yeah.. that resolve the issue. | 13:44 |
| *** dasm|ruck|off is now known as dasm|ruck | 13:59 | |
| * dasm|ruck catching up | 13:59 | |
| rlandy | dasm|ruck: downstream is back | 14:03 |
| rlandy | promoted all 16.2 components excpet network | 14:04 |
| rlandy | which was missing jenkins job | 14:04 |
| rlandy | now in rerun | 14:04 |
| rlandy | will rekick 16.2 after network promotes | 14:04 |
| rlandy | rekcik clients | 14:04 |
| rlandy | rekicked | 14:04 |
| rlandy | as i failed 17 on rhel 8 and rhel 9 | 14:04 |
| rlandy | after that only two rhel 8 17 components lagging | 14:05 |
| rlandy | dasm|ruck: major task is to promote wallaby c9 | 14:05 |
| rlandy | stable 1 | 14:05 |
| rlandy | akahat|rover, has a testproject | 14:05 |
| rlandy | dasm|ruck: other than that - master needs some love if the next promotion is reasonable | 14:05 |
| rlandy | dasm|ruck: still follow bug that is blocking tripleo | 14:06 |
| rlandy | rhel 9 has a genuine error | 14:06 |
| rlandy | ysandeep logged it | 14:06 |
| dasm|ruck | rlandy: ack, thanks. | 14:17 |
| ade_lee | dviroel, rlandy dasm|ruck jpodivin the current mixed mode proposal will require us to be a little careful about making fips the default config for centos 9+ - we'll need to be able to turn fips off for mixed node deployments of 8/9 | 14:35 |
| dasm|ruck | ack | 14:35 |
| jpodivin | ade_lee: understood. It's going to be bit tricky. | 14:36 |
| dviroel | ade_lee++ | 14:37 |
| rlandy | ade_lee: ack - will keep that in mind | 14:43 |
| chandankumar | in mixed mode, the main twist is going to be pulling same openstack content on compute and other nodes, Right now dlrn builds the packages seperately for c8 and c9. they might have or might not have same openstack contents. | 14:45 |
| rlandy | there are assumptions everywhere in our code | 14:51 |
| rlandy | akahat|rover: pls sync with dasm|ruck before your EoD | 14:51 |
| rlandy | dasm|ruck: train and wallaby | 14:52 |
| rlandy | promotions are priority | 14:52 |
| dasm|ruck | centos 8 wallaby gives me: last_promotion=2022-03-31 10:20:29 | 15:02 |
| dasm|ruck | there are few jobs failed. checking if we're running them already | 15:03 |
| dasm|ruck | wait. should it be cs9 wallaby? we're tearing down cs8 wallaby https://review.opendev.org/c/openstack/tripleo-ci/+/834186 | 15:04 |
| dasm|ruck | cs9 wallaby rerun: https://review.rdoproject.org/r/c/testproject/+/41370 | 15:06 |
| dasm|ruck | cs8 victoria > last_promotion=2022-04-01 22:10:41 | 15:07 |
| dasm|ruck | cs7 train > last_promotion=2022-03-31 14:59:06 | 15:08 |
| ysandeep | pojadhav, mute | 15:09 |
| ysandeep | done, ignore | 15:10 |
| pojadhav | folks, accidently unmuted, my apologies | 15:25 |
| ysandeep | pojadhav: no worries :D | 15:34 |
| rlandy | dviroel: ok - see you got sharing going :) | 15:41 |
| dviroel | :) | 15:42 |
| jm1 | dviroel++ thank you, nice intro to compose pinning :) | 16:12 |
| rlandy | dviroel++ nicely done | 16:12 |
| ysandeep | dviroel++ one question, do we use older nodepool image in rdo? | 16:13 |
| dviroel | thanks all o/ | 16:14 |
| marios | chandankumar: rlandy: ysandeep: dviroel: ++ well done :) tripleo-ci represent | 16:14 |
| rcastillo | great sessions everyone! | 16:15 |
| rlandy | marios++ on all your participation | 16:15 |
| dviroel | ysandeep: no, not yet, we are in the middle of that yet, so it is not a perfect repo setup since system packages are up to date, but we should get there soon (missed that gap there) | 16:15 |
| rlandy | rcastillo: dasm|ruck: hey - pls see abve about request to move molecule jobs to c9 | 16:15 |
| ysandeep | dviroel: ack, thanks for info | 16:16 |
| *** marios is now known as marios|out | 16:17 | |
| ysandeep | frenzy_friday, where is our reviewbot? | 16:17 |
| rcastillo | rlandy: ack. Was looking at the molecule stuff in tripleo-ci and it needs some cleanup | 16:18 |
| rlandy | thanks | 16:18 |
| ysandeep | rlandy: please add in your review list: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39335 , Revert "Skip SSL verify for ruck_rover.py" | 16:18 |
| rlandy | ysandeep: 17 c9 components are all clean now | 16:18 |
| rlandy | all promoted | 16:18 |
| dasm|ruck | rcastillo: thanks | 16:18 |
| rlandy | periodic-tripleo-rhel-8-rhos-16.2-component-network-promote-to-promoted-components - also just promoted | 16:19 |
| *** amoralej is now known as amoralej|off | 16:19 | |
| rlandy | going to kick 16.2 now | 16:19 |
| ysandeep | rlandy, nice.. I spoke with jon, he is building the missing deps for container build job, I think we will be good by tomorrow. | 16:19 |
| frenzy_friday | ysandeep, the health server rebooted it seems, the bot died. I restarted this morning, checking what happened now | 16:25 |
| ysandeep | frenzy_friday, thanks, you back in Germany? | 16:25 |
| frenzy_friday | ysandeep, yep | 16:25 |
| ysandeep | frenzy_friday: great! | 16:26 |
| ysandeep | reviewbot: please add in review list: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39335 | 16:27 |
| frenzy_friday | bot pls add to review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39335: Revert "Skip SSL verify for ruck_rover.py" | 16:27 |
| frenzy_friday | hm.. issue with hackmd auth token | 16:28 |
| rlandy | frenzy_friday: how was the trip back? | 16:28 |
| rlandy | frenzy_friday: did you go alone? | 16:28 |
| frenzy_friday | rlandy, lengthy :D and I forgot all the german that I had learnt in these few mths | 16:28 |
| frenzy_friday | rlandy, nope, we both came to germany | 16:29 |
| rlandy | frenzy_friday: yay | 16:29 |
| rlandy | happy you both made it | 16:29 |
| frenzy_friday | bot pls add to review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39335: Revert "Skip SSL verify for ruck_rover.py" | 16:31 |
| reviewbot | I could not add the review to Review List | 16:31 |
| frenzy_friday | reviewbot, why! | 16:31 |
| *** jpena is now known as jpena|off | 16:31 | |
| rlandy | lunch - brb | 16:32 |
| frenzy_friday | ysandeep, looks like there is an issue connecting from the health server to hackmd after reboot. Checking | 16:36 |
| frenzy_friday | bot pls add to review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/39335: Revert "Skip SSL verify for ruck_rover.py" | 16:37 |
| reviewbot | I have added your review the to Review list | 16:37 |
| ysandeep | frenzy_friday: thanks! | 16:39 |
| ysandeep | frenzy_friday, looks like reviewbot is back to life | 16:40 |
| ysandeep | frenzy_friday, pojadhav fyi.. I am updating kernel on development cockpit server, I am hitting some performance issue when running python command from telegraf container, As per https://github.com/docker/for-linux/issues/738 - kernel update might help. | 16:43 |
| frenzy_friday | ysandeep, yeah, it is running on my local now, I'll check what happened to the server tomorrow | 16:44 |
| frenzy_friday | ack, for the kernel update | 16:45 |
| ysandeep | frenzy_friday: ack, thanks! Let me know if you need any help debugging network issue on health server. | 16:46 |
| * dviroel lunch | 17:01 | |
| *** dviroel is now known as dviroel|lunch | 17:01 | |
| dasm|ruck | rcastillo: are you around? how can i rerun c7 train jobs? this one failed on me: https://review.rdoproject.org/r/c/testproject/+/41367/2/.zuul.yaml | 17:17 |
| rcastillo | tox-molecule depends on ensure-docker, which isn't ready for cs9 yet | 17:17 |
| rcastillo | dasm|ruck: looking | 17:17 |
| dasm|ruck | i procured hash for c7 train, because rr script gave me nothing. | 17:18 |
| rcastillo | dasm|ruck: I believe there's a bug with the rr script and c7 hashes, yeah | 17:18 |
| dasm|ruck | rcastillo: any idea how to fix that? if not, i'll wait for Ronelle | 17:20 |
| rcastillo | let's see if we can figure this out | 17:22 |
| dasm|ruck | currently script returned me the same hash | 17:22 |
| dasm|ruck | *rr scritp | 17:22 |
| dasm|ruck | *script (ofc) | 17:22 |
| dasm|ruck | > tripleo-quickstart/roles/repo-setup/tasks/get-dlrn-hash.yml | 17:25 |
| rcastillo | dasm|ruck: try this hash 4c3dfec59da9dbd40c4f6e7aa9d44aa56e4f45a4_0ad01be2 | 17:26 |
| rcastillo | (from https://trunk.rdoproject.org/centos7-train/tripleo-ci-testing/delorean.repo ) | 17:26 |
| dasm|ruck | rcastillo: with "_0ad01be2" ? | 17:26 |
| dasm|ruck | ok, i see: https://trunk.rdoproject.org/centos7-train/4c/3d/4c3dfec59da9dbd40c4f6e7aa9d44aa56e4f45a4_0ad01be2/ | 17:27 |
| rcastillo | yeah, seems like we allow that hash format on c7 https://opendev.org/openstack/tripleo-quickstart/src/commit/4b4a265296fa5a1f9090d4e3d52e9c824dec057c/roles/repo-setup/defaults/main.yml#L74 | 17:27 |
| dasm|ruck | ack | 17:27 |
| dasm|ruck | lemme try that the | 17:27 |
| dasm|ruck | *then | 17:27 |
| dasm|ruck | restarting | 17:28 |
| dasm|ruck | hmm.. it still seems to be not working | 17:46 |
| rcastillo | dasm|ruck: darn, not sure if there's a right way to set the hashes for c7 but when I needed to do some thing like that I just ran the job without the hash and prayed tripleo-ci-testing didn't change under me | 17:50 |
| rcastillo | can't wait for the c7 teardown :) | 17:50 |
| dasm|ruck | xD | 17:51 |
| rlandy | dasm|ruck: need something? | 17:52 |
| dasm|ruck | rlandy: how to rekick c7 train? both my attempts failed | 17:52 |
| dasm|ruck | periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby | 17:52 |
| dasm|ruck | https://review.rdoproject.org/r/c/testproject/+/41367/2/.zuul.yaml | 17:52 |
| dasm|ruck | rlandy: ^ | 17:53 |
| rlandy | dasm|ruck: the rr should give you the right formula | 17:53 |
| rlandy | there is a testproject job already running for that | 17:53 |
| dasm|ruck | rlandy: i used it and it failed on me | 17:53 |
| dasm|ruck | ack | 17:53 |
| *** dviroel|lunch is now known as dviroel | 17:55 | |
| rlandy | dasm|ruck: hmm .. try w/o dlrn... | 17:55 |
| rlandy | like: https://review.rdoproject.org/r/c/testproject/+/35663/52/.zuul.yaml | 17:56 |
| rlandy | c7 is not setup for agg hash | 17:56 |
| rlandy | dasm|ruck: but there is already one running | 17:56 |
| rlandy | amd only one job left to pass | 17:56 |
| dasm|ruck | rlandy: ack | 17:56 |
| rlandy | you can try just kick the one missing job | 17:56 |
| rlandy | dasm|ruck: add to that the two that failed in train c8 testproject run | 17:57 |
| rlandy | while you are at it :) | 17:57 |
| rlandy | periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train | 17:57 |
| rlandy | periodic-tripleo-ci-centos-8-scenario003-standalone-train | 17:58 |
| rlandy | dasm|ruck: ^^ those two | 17:58 |
| rlandy | dasm|ruck: looking to promo train c7, c8 w c9 and 16.2 today | 17:58 |
| rlandy | dasm|ruck: w9 is just waiting for fs035 | 17:59 |
| rlandy | may skip it and promote | 18:00 |
| rlandy | job in rerun now | 18:00 |
| dasm|ruck | c9 fs35 is still running | 18:00 |
| rlandy | dasm|ruck: master also only missing one job: https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_agg_detail.html?ref_hash=3b08b3c2c5de2f5784df8cc16179a24b | 18:08 |
| rlandy | running again now in current master line | 18:08 |
| rlandy | we may get some promos today | 18:08 |
| rlandy | which would be good | 18:08 |
| dasm|ruck | i'm hoping for that | 18:08 |
| rlandy | oh and a bunch fo ovb jobs in master | 18:09 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/9c/9c27aacfcba453c8e739a2d36929375d9dd6a958/openstack-periodic-integration-rhos-16.2/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2/e29a475/logs/build.log | 18:48 |
| rlandy | weird error on container builds in 16.2 | 18:48 |
| rlandy | 2022-04-04 14:27:24 | 2022-04-04 14:27:24.198 82576 INFO tripleo_common.image.builder.buildah [-] Pushing docker-registry.upshift.redhat.com/tripleorhos-16-2/openstack-nova-compute-ironic:482934bf87bd64a1b41c2b247e826068 image with: sudo buildah push --tls-verify=False docker-registry.upshift.redhat.com/tripleorhos-16-2/openstack-nova-compute-ironic:482934bf87bd64a1b41c2b247e826068 docker://docker-registry.upshift.redhat.com/tripl | 18:49 |
| rlandy | eorhos-16-2/openstack-nova-compute-ironic:482934bf87bd64a1b41c2b247e826068[00m | 18:49 |
| * rlandy rekicks | 18:50 | |
| dasm|ruck | ack | 18:50 |
| rlandy | dasm|ruck: can you looking into failing wallaby c9 components | 18:54 |
| rlandy | baremetal, clients, common, etc. | 18:55 |
| dasm|ruck | yep | 18:55 |
| rlandy | train network etc. | 18:56 |
| rlandy | dasm|ruck: ie: the ones we care to import | 18:56 |
| rlandy | we know tripleo master is failing and bugged | 18:56 |
| dasm|ruck | > does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found. | 18:57 |
| dasm|ruck | it looks familiar | 18:57 |
| dasm|ruck | ovb baremetal | 18:57 |
| dasm|ruck | checking if that's consistent | 18:57 |
| dasm|ruck | last running build failed like that. 3 next ones were skipped | 18:58 |
| jm1 | rcastillo: you can now rebase your patches for ansible openstack collection to test them against master branch of the sdk :) if you cherry pick them against the stable branch, they will be tested against old sdk releases as well. and i left some comments at your patches ;) happy hacking ^^ | 19:00 |
| rcastillo | jm1 ack thx! | 19:01 |
| rlandy | woohoo wallaby c9 should promote | 19:10 |
| dasm|ruck | nice | 19:14 |
| * rlandy needs to run to pharmacy - will brb | 19:19 | |
| *** rlandy is now known as rlandy|brb | 19:20 | |
| *** rlandy|brb is now known as rlandy | 19:44 | |
| dasm|ruck | brb | 20:26 |
| * dviroel afk | 20:41 | |
| *** dviroel is now known as dviroel|afk | 20:41 | |
| * dasm|ruck is back | 20:50 | |
| rlandy | dasm|ruck: ok - train c7 should promote now | 20:56 |
| dasm|ruck | > Hash under test: https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=26ba4cc3e2b6d6b42394c6314a1d0084 | 22:17 |
| dasm|ruck | last_promotion=2022-04-04 19:56:23 | 22:17 |
| * dasm|ruck off | 22:26 | |
| *** dasm|ruck is now known as dasm| | 22:26 | |
| *** dasm| is now known as dasm|off | 22:26 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!