| *** bhagyashris is now known as bhagyashris|ruck | 05:02 | |
| bhagyashris|ruck | frenzy_friday, hey created new hackmd https://hackmd.io/0Lev7RRlRDCj9hiNUKN1zw and also updated the ruck rover status hackmd | 05:10 |
|---|---|---|
| marios | o/ | 05:11 |
| *** ysandeep|out is now known as ysandeep | 05:52 | |
| marios | bhagyashris|ruck: frenzy_friday: ping me if i can help with something | 06:02 |
| bhagyashris|ruck | ack | 06:02 |
| jm1 | o/ | 06:27 |
| *** frenzy_friday is now known as frenzyfriday|rover | 06:29 | |
| frenzyfriday|rover | bhagyashris|ruck, thanks. /me checks upstream | 06:30 |
| *** amoralej|off is now known as amoralej | 06:31 | |
| *** arxcruz|rover is now known as arxcruz | 07:18 | |
| marios | frenzyfriday|rover: bhagyashris|ruck: should we have a quick 5 mins sync in half hour? | 07:24 |
| frenzyfriday|rover | marios, sure | 07:25 |
| jm1 | chandankumar, ysandeep: chandankumar's testproject is confusing rr tool :D https://review.rdoproject.org/r/c/testproject/+/45593/ | 07:37 |
| jm1 | chandankumar, ysandeep: run "./ruck_rover.py --release train --distro centos-8" and it will report the hash of this testproject | 07:38 |
| chandankumar | jm1: sorry I will change the job name | 07:38 |
| jm1 | chandankumar, ysandeep: not sure whether we should somehow fix rr tool instead | 07:39 |
| *** jpena|off is now known as jpena | 07:39 | |
| ysandeep | confusing how? | 07:39 |
| jm1 | chandankumar, ysandeep: we often run specific periodic jobs in testprojects. but doing so will cause e.g. this https://trunk.rdoproject.org/api-centos8-train/api/civotes_agg_detail.html?ref_hash=878b7a2fd0e595098076a01a89cec57d | 07:41 |
| jm1 | chandankumar, ysandeep: this hash has only a single job, because it was a testproject | 07:41 |
| jm1 | chandankumar, ysandeep: but now rr thinks that c8 train is still running for the other c8 train jobs although it is not | 07:42 |
| jm1 | chandankumar, ysandeep: if you look at https://hackmd.io/8MEZ5R-USXGyRTDiamNW2Q?view you will see a looooot of pending jobs (output of rr tool will be similar) | 07:43 |
| ysandeep | I am not sure I am following it - How can testproject triggering a job can create a new hash? | 07:44 |
| ysandeep | we can new hash when we run promote job from promote-components to tripleo-ci-testing | 07:44 |
| ysandeep | we get* new hash | 07:44 |
| jm1 | ysandeep: no idea, but look at the links | 07:44 |
| bhagyashris|ruck | marios, sure. went for lunch | 07:45 |
| ysandeep | jm1, https://review.rdoproject.org/zuul/status#openstack-periodic-integration-stable4%20 | 07:45 |
| ysandeep | jm1, ^^ line triggered - that's the reason, nothing related to testproject | 07:45 |
| jm1 | ysandeep, chandankumar: omg ok the issue is somewhere else: the code that gets the failure reason is broken.. | 07:47 |
| jm1 | ysandeep, chandankumar: it returns the results from check jobs | 07:48 |
| ysandeep | jm1, where are you checking that.. could you please pass link/logs what are you looking? | 07:49 |
| jm1 | ysandeep: run "./ruck_rover.py --release train --distro centos-8" and it will return "https://logserver.rdoproject.org/93/45593/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/cb33743" | 07:50 |
| jm1 | ysandeep: this is pointing to chandankumar's testproject which brought be to the ref_hash link above. | 07:51 |
| ysandeep | that's expected | 07:51 |
| ysandeep | Once fs035 runs in scheduled run, if it passes - rr will take that job in consideration | 07:52 |
| ysandeep | the job which chandan ran is not check job, its a periodic job(even though is triggered why a testproject) | 07:53 |
| ysandeep | rr/promoter don't differentiate if job is run via testproject/scheduled run | 07:54 |
| jm1 | ysandeep: it is still a check job in zuul terms but i get the idea ^^ | 07:54 |
| ysandeep | As a best practice if we don't want our periodic job (run via testproject) to report to dlrn than we should add dlrn_report: false var. | 07:55 |
| jm1 | ysandeep: oh that is a good advice! can we please put that somewhere for people to find? :) | 07:56 |
| ysandeep | correcting var name: report_dlrn | 07:58 |
| ysandeep | Yes we should document it somewhere if its not documented already, we should use it when we are running periodic jobs with depends-on, otherwise we can get false promotions | 08:00 |
| jm1 | ysandeep: yeah i saw ronelle using that to get promotions ^^ | 08:01 |
| jm1 | ysandeep: it's not a bug, it's a feature XD | 08:02 |
| jm1 | ysandeep++ thanks for clarifying this hash mistery :D | 08:03 |
| ysandeep | nw, report_dlrn by defaults to true for periodic jobs, we only want to mark that false when we are testing with unmerged code(when we are testing if some unmerged code fixes our jobs) | 08:03 |
| *** ysandeep is now known as ysandeep|lunch | 08:26 | |
| bhagyashris|ruck | marios, hi, on your testproject patch container build push passed for rhos17.1 on rhel8 hope rhos17.1 on rhel8 will also pass | 08:39 |
| marios | ok cool thanks bhagyashris|ruck | 08:39 |
| marios | if it continues to be green then perhaps just some transient issue no need for BZ | 08:40 |
| bhagyashris|ruck | may be there were some outage on registery side | 08:40 |
| marios | we only need the BZ if we want to CIX this if it is transient then meh | 08:40 |
| *** ysandeep|lunch is now known as ysandeep | 09:08 | |
| bhagyashris|ruck | marios, rhos17.1 on rhel8 container build push is still failing with time out issue | 09:35 |
| frenzyfriday|rover | auth.redhat is back now | 09:43 |
| marios | bhagyashris|ruck: then probably worth filing that one. is it the same issue that rlandy|out raised a support ticket for i think? | 10:02 |
| marios | bhagyashris|ruck: ah unless 12:43 < frenzyfriday|rover> auth.redhat is back now | 10:02 |
| marios | perhaps that was the last one? can you rerun in testproject to check (I think you already did?) | 10:03 |
| frenzyfriday|rover | marios, bhagyashris|ruck auth was back at 10:55 according to the mail from Girish Patil | 10:04 |
| frenzyfriday|rover | ^ CEST | 10:04 |
| frenzyfriday|rover | https://redhat.service-now.com/help?id=rh_ticket&is_new_order=true&table=sc_request&sys_id=f465d7c997261950f5d07c7e6253af85 has no new updates | 10:06 |
| marios | folks please review when you have time https://review.rdoproject.org/r/c/rdo-jobs/+/45537 thank you | 10:18 |
| bhagyashris|ruck | marios, ok re-running | 10:18 |
| frenzyfriday|rover | hey, does anyone know why zuul jobs are not running on this patch https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/859297? | 10:19 |
| frenzyfriday|rover | marios, we should add --release zed option to the RR tool right? /me putting up a patch | 10:21 |
| marios | frenzyfriday|rover: sure sounds good | 10:28 |
| * marios brb | 10:28 | |
| *** rlandy|out is now known as rlandy | 10:30 | |
| rlandy | bhagyashris|ruck: frenzyfriday|rover: hi - did you sync with marios this morning? | 10:33 |
| rlandy | bhagyashris|ruck: frenzyfriday|rover: we have a legit fail on https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17&skip=0 | 10:33 |
| bhagyashris|ruck | rlandy, yup | 10:33 |
| rlandy | bhagyashris|ruck: frenzyfriday|rover: let's sync on downstream | 10:33 |
| rlandy | a few things there | 10:33 |
| bhagyashris|ruck | rlandy, ack | 10:33 |
| bhagyashris|ruck | give me 5 mins | 10:34 |
| rlandy | bhagyashris|ruck: frenzyfriday|rover: pls ping when ready | 10:35 |
| bhagyashris|ruck | rlandy, frenzyfriday|rover https://meet.google.com/jgy-xhcf-ozw | 10:35 |
| rlandy | bhagyashris|ruck: let's give frenzyfriday|rover a few minutes to respond | 10:35 |
| frenzyfriday|rover | 0/ | 10:35 |
| bhagyashris|ruck | rlandy, she joined | 10:35 |
| * frenzyfriday|rover joined :) | 10:35 | |
| rlandy | ok | 10:36 |
| soniya | rlandy, hey | 11:04 |
| soniya | plz see pvt | 11:04 |
| rlandy | review time anyone? | 11:17 |
| marios | rlandy: i just dropped | 11:17 |
| rlandy | I'm on now | 11:18 |
| *** ysandeep is now known as ysandeep|afk | 11:51 | |
| marios | rlandy: fyi https://github.com/rdo-infra/rdo-jobs/blob/5d942cc44cf47dd223c3dbf9b60a715e59ebb7be/zuul.d/component-pipeline-wallaby-centos9.yaml#L90 | 11:56 |
| marios | rlandy: i confused myself during the call - we do have an issue in that wallaby layout and 'how is it working' is a good question | 11:56 |
| marios | rlandy: https://github.com/rdo-infra/rdo-jobs/blob/5d942cc44cf47dd223c3dbf9b60a715e59ebb7be/zuul.d/component-pipeline-wallaby-centos9.yaml#L90 | 11:57 |
| jm1 | rlandy, pojadhav: next week's scrum mtg on thursday is conflicting with aoc ptg | 11:59 |
| rlandy | marios: hmmm - are those jobs in criteria? | 11:59 |
| marios | rlandy: no not in criteriar but triggers the scen10 jobs right? | 12:01 |
| rlandy | jm1: pojadhav: pls cancel scrum then on thursday | 12:01 |
| marios | rlandy: anyway just mentioned as we discussed in the call earlier, posted that just now https://review.rdoproject.org/r/c/rdo-jobs/+/45662 | 12:01 |
| rlandy | marios: ack - they would be internal | 12:01 |
| rlandy | marios: I'll check after 1-1 | 12:01 |
| *** ysandeep|afk is now known as ysandeep | 12:22 | |
| pojadhav | rlandy, jm1 : ack will cancel it | 12:35 |
| pojadhav | rlandy, jm1 : done | 12:36 |
| *** pojadhav is now known as pojadhav|afk | 12:36 | |
| rlandy | pojadhav|afk: thanks | 12:38 |
| rlandy | frenzyfriday|rover: hey - checking in on master hash and octavia change | 12:41 |
| rlandy | is it in current-tripleo? | 12:41 |
| rlandy | marios: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-9/component/wallaby.yaml - scenario010 jobs are not in criteria | 12:43 |
| chandankumar | dpawlik: hello https://review.rdoproject.org/r/c/testproject/+/45593 | 12:43 |
| chandankumar | dpawlik: we got the node hold on both IBM and vexxhost | 12:43 |
| chandankumar | sending Ip internally | 12:43 |
| rlandy | frenzyfriday|rover: checked https://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=blob;f=ci-scripts/dlrnapi_promoter/config/RedHat-9/component/rhos-17.1.yaml;h=cc21104e2410db0e0c299f8237964e9e95e68b8a;hb=HEAD | 12:45 |
| rlandy | looks complete I think | 12:45 |
| marios | rlandy: ack ok then that is why it didn't block but we should fix that anyway so we run the right job | 12:46 |
| marios | rlandy: thanks for checking | 12:47 |
| frenzyfriday|rover | rlandy, yeah, the component jobs are all added. There was one job which had a pending criteria, lemme check | 12:48 |
| frenzyfriday|rover | regarding the octavia thing - this is the id from github 344d6d76ce4481f30fe295114551243d61e7eea8. It should be in https://trunk.rdoproject.org/centos9-master/current-tripleo/versions.csv right? | 12:53 |
| marios | needs reviews please folks when you have time: https://review.rdoproject.org/r/c/rdo-jobs/+/45537 (https://review.rdoproject.org/r/c/testproject/+/45538) & https://review.rdoproject.org/r/c/rdo-jobs/+/45610 (https://review.rdoproject.org/r/c/testproject/+/45451) | 12:59 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 12:59 |
| marios | (tests in brackets) | 12:59 |
| marios | these are on the review list thanks (2 reviews, one for integration one for component zed jobs ) | 12:59 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 12:59 |
| marios | thank you | 12:59 |
| rlandy | marios: I'll put in a patch to fix wallaby | 13:05 |
| rlandy | frenzyfriday|rover: checking | 13:05 |
| rlandy | frenzyfriday|rover: you'd be looking for tripleo-ci-testing | 13:05 |
| frenzyfriday|rover | ohh | 13:06 |
| marios | rlandy: i posted it | 13:06 |
| rlandy | frenzyfriday|rover: so start here: https://trunk.rdoproject.org/centos9-master/component/octavia/ | 13:06 |
| marios | rlandy: https://review.rdoproject.org/r/c/rdo-jobs/+/45662 | 13:06 |
| rlandy | https://trunk.rdoproject.org/centos9-master/component/octavia/promoted-components/?C=M;O=D | 13:06 |
| rlandy | frenzyfriday|rover: ^^ | 13:06 |
| rlandy | openstack-octavia-amphora-agent-11.1.0-0.20221013164419.344d6d7.el9.noarch.rpm2022-10-13 16:50 14K | 13:06 |
| rlandy | ok - so in promoted components | 13:07 |
| rlandy | then | 13:07 |
| rlandy | https://trunk.rdoproject.org/centos9-master/component/octavia/tripleo-ci-testing/?C=M;O=D | 13:07 |
| rlandy | frenzyfriday|rover: ^^ not yet | 13:07 |
| frenzyfriday|rover | ack, thanks | 13:08 |
| rlandy | frenzyfriday|rover: so the next master run should pick it up | 13:08 |
| rlandy | marios: thanks - w+'ed | 13:09 |
| frenzyfriday|rover | does periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master have an internal job? | 13:09 |
| rlandy | https://trunk.rdoproject.org/centos9-master/component/octavia/tripleo-ci-testing/?C=M;O=D | 13:10 |
| rlandy | frenzyfriday|rover: not yet | 13:10 |
| marios | thanks rlandy | 13:11 |
| rlandy | marios: bhagyashris|ruck: dan responded to my request to check the registry push | 13:12 |
| bhagyashris|ruck | rlandy, ack i can't see the comment | 13:19 |
| bhagyashris|ruck | rlandy, virt customized task pass on fs001 on envB 17 on 9 hope job will pass | 13:20 |
| marios | rlandy: thx | 13:25 |
| marios | rlandy: on my latest test 9 passed 8 failed (container build push) | 13:25 |
| bhagyashris|ruck | rlandy, frenzyfriday|rover looks like evrything is going back ... i have added all the data in the hackmd you will just need to keep eye on currently running testproject patch | 13:27 |
| rlandy | bhagyashris|ruck: ok - will do | 13:28 |
| bhagyashris|ruck | i need to leave for the day because guest are coming and will need to cook some food. but may be will back after 3 to 4 hrs | 13:29 |
| *** amoralej is now known as amoralej|lunch | 13:34 | |
| frenzyfriday|rover | bhagyashris|ruck, ack, thanks | 13:46 |
| *** amoralej|lunch is now known as amoralej | 14:18 | |
| *** dasm|off is now known as dasm | 14:25 | |
| dasm | o/ | 14:25 |
| frenzyfriday|rover | is the downstream zuul slow or just for me? | 14:36 |
| rlandy | frenzyfriday|rover: marios: bhagyashris|ruck: dan responded that he had done some cleanup of the registry | 14:36 |
| frenzyfriday|rover | ack | 14:37 |
| marios | folks please add to your reviews queue https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/430798 thank you | 14:37 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 14:37 |
| frenzyfriday|rover | downstream still has other outages though: one of the following is timing out in almost all cases | 14:38 |
| frenzyfriday|rover | Retry_limit : https://osp-trunk.hosted.upshift.rdu2.redhat.com/ or https://bootstrap.pypa.io/pip/3.6/get-pip.py or https://docker-registry.upshift.redhat.com/v2/tripleorhos-17-1-rhel-8/openstack-heat-base/blobs/sha256:3574ac17976440865039b44e5bfd58ca7e504d41eb5f357e4623b234e54b9148 is not reachable | 14:38 |
| rlandy | frenzyfriday|rover: marios: bhagyashris|ruck: going to rekick the 17 on 8 line again to check containers push | 14:38 |
| frenzyfriday|rover | cool | 14:39 |
| marios | rlandy: k thanks | 14:40 |
| *** dviroel is now known as dviroel|lunch | 14:57 | |
| rlandy | marios: does https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/430798 fix your issue? | 15:03 |
| marios | rlandy: issue is fixed because promotions have made the version same in both compute and control, even though compute has -pending enabled. but i still think we should remove. links incoming pvt sec | 15:05 |
| marios | so i think we should remove it anyway ... no fire on it for now, but it has the potential to mess us again, with -pending bringing slightly newer version than the controller has (we only enabled that -pending for 8) | 15:05 |
| marios | rlandy: thank you for reviewieng | 15:05 |
| marios | ieeiie | 15:06 |
| rlandy | marios: ok - will +2 | 15:07 |
| rlandy | you can merge on monday if you want | 15:08 |
| ysandeep | Happy weekend folks, see you next week o/ | 15:08 |
| *** ysandeep is now known as ysandeep|out | 15:08 | |
| chandankumar | see ya people, happy weekend! | 15:11 |
| marios | thanks rlandy | 15:12 |
| marios | o/ ysandeep|out /me off in a sec too | 15:12 |
| marios | \o chandankumar | 15:12 |
| rlandy | see you all on wednesday | 15:13 |
| rlandy | periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17.1success | 15:22 |
| rlandy | periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17.1 (3rd attempt)success | 15:22 |
| marios | ok is that on mine? | 15:23 |
| marios | well at least it got through .. but seems still might be flaky regsitry | 15:23 |
| * marios off rlandy need sthing? | 15:24 | |
| rlandy | marios: ^^ looking better | 15:24 |
| rlandy | those both passed now | 15:24 |
| rlandy | marios: no - see you wednesday | 15:24 |
| marios | rlandy: o/ have a good break | 15:24 |
| *** marios is now known as marios|out | 15:25 | |
| *** amoralej is now known as amoralej|off | 15:37 | |
| *** dviroel|lunch is now known as dviroel | 16:07 | |
| * dasm brb | 16:11 | |
| frenzyfriday|rover | hey Ronelle, for master d09dab19623b9bbf7d04e8efba787113 I see everything has passed from criteria except kvm sc10 (both rdo and internal) But it is not in criteria. However I see thr promoter logs are complaining criteria is not met. Need some help. | 16:17 |
| frenzyfriday|rover | rlandy, ^ | 16:17 |
| rlandy | frenzyfriday|rover: checking | 16:20 |
| frenzyfriday|rover | fs002 was runing, but it already reported to dlrn I think https://review.rdoproject.org/zuul/stream/4fefe522c7f44e16be7cd4c41e31a953?logfile=console.log | 16:21 |
| rlandy | python3 roles/rrcockpit/files/telegraf_py3/ruck_rover.py --release master --distro centos-9 --aggregate_hash tripleo-ci-testing/d0/9d/d09dab19623b9bbf7d04e8efba787113 | 16:22 |
| rlandy | rr tool shows it should promote | 16:22 |
| rlandy | checking logs | 16:22 |
| rlandy | frenzyfriday|rover: patience :) | 16:24 |
| rlandy | it should | 16:24 |
| rlandy | fs002 just reported | 16:24 |
| frenzyfriday|rover | cool, I thought I missed some stuff with the internal/rdo thingy | 16:25 |
| rlandy | the last run was 2022-10-14 16:19:37,832 | 16:25 |
| rlandy | it reported at 16:11 | 16:26 |
| rlandy | which was before | 16:26 |
| rlandy | let's see the next run | 16:26 |
| rlandy | frenzyfriday|rover: ^^ if that does not promote, then yeah - we have a problem | 16:26 |
| frenzyfriday|rover | ack, thanks | 16:26 |
| rlandy | frenzyfriday|rover: it's promoting now ... http://promoter.rdoproject.org/promoter_logs/container-push/20221014-161929.log | 16:27 |
| rlandy | there you go | 16:27 |
| rlandy | "source_registry.namespace": "tripleomastercentos9" | 16:27 |
| frenzyfriday|rover | yaayy! | 16:27 |
| rlandy | frenzyfriday|rover: also next master hash just kicked | 16:28 |
| rlandy | frenzyfriday|rover; so https://trunk.rdoproject.org/centos9-master/component/octavia/tripleo-ci-testing/?C=M;O=D has the new image | 16:28 |
| rlandy | ie: we need to promo this hash next | 16:29 |
| frenzyfriday|rover | I see our commit id 344d6d76ce4481f30fe295114551243d61e7eea8 in commits.yaml | 16:29 |
| rlandy | and 17 on 9 just promoted | 16:30 |
| *** jpena is now known as jpena|off | 16:34 | |
| frenzyfriday|rover | rlandy, I am leaving for the day, upstream testproj are running, updated on hackmd | 16:57 |
| rlandy | frenzyfriday|rover: sure - have a good weekend | 16:58 |
| frenzyfriday|rover | you too :) | 16:58 |
| rlandy | will see what you want me to pick up | 16:58 |
| frenzyfriday|rover | rlandy, thanks. I have crossed out the tp which have finished | 17:03 |
| rlandy | lunch - brb | 17:21 |
| rlandy | dasm: hi | 18:06 |
| rlandy | dasm: does http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1 show data for you? | 18:06 |
| rlandy | dasm: I think we need to revert pojadhav's change | 18:07 |
| rlandy | rcastillo: ^^ do you see data there? | 18:07 |
| rlandy | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44768 | 18:08 |
| rlandy | dasm: ^^ | 18:08 |
| rlandy | pojadhav: had to revert https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44768 | 18:34 |
| rlandy | promotions adat disappeared | 18:34 |
| rlandy | data | 18:34 |
| * dasm is back (took longer than expected) | 19:30 | |
| dasm | rlandy: ack. | 19:31 |
| rlandy | dasm: nvm - I merged the revert | 19:31 |
| rlandy | needed the promo data back | 19:31 |
| dasm | i see. | 19:32 |
| dasm | interesting that promotions disappeared | 19:32 |
| dasm | esp after checking snapshot | 19:32 |
| rlandy | dasm: pls update with pojadhav next week | 20:01 |
| dasm | akc | 20:02 |
| dviroel | have a great weekend team | 21:25 |
| dviroel | o/ | 21:26 |
| *** dviroel is now known as dviroel|out | 21:27 | |
| rlandy | bye | 21:42 |
| dasm | o/ | 21:44 |
| *** dasm is now known as dasm|off | 23:58 | |
| dasm|off | o/ | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!