Friday, 2022-04-01

*** ysandeep|out is now known as ysandeep05:09
marioso/05:57
ysandeepgood morning marios o/06:01
marioshappy friday :)06:01
ysandeepHappy Friday everyone :D06:02
ysandeepmarios, do you have core on ansible-role-python_venv_build repo?06:02
ysandeeplooks like we need https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/836091 to clear a gate blocker https://bugs.launchpad.net/tripleo/+bug/196743006:03
mariosysandeep: yeah i was just loking at         * https://review.opendev.org/admin/groups/3474fc86368161e5288be01295041a089a1060b3,members06:04
marios:/06:04
mariosbrb06:04
akahatGood morning folks o/06:13
chandankumar\o06:16
chandankumarmarios: ysandeep pinged the cores on #openstack-ansible for above review06:16
soniya29akahat, good morning06:17
soniya29akahat, let me know when we can sync up?06:17
akahatsoniya29, we can sync up when marios will be back06:17
mariosakahat: o/06:19
akahatmarios, soniya29 https://meet.google.com/nmr-dhvx-gvv06:20
marioschandankumar: thanks i just sent to the list as well and added some people to cc that might help06:20
chandankumarmarios: got workflowed06:21
marioscool 06:21
mariosakahat: sure gimme a minute 06:21
mariosopening meet06:21
akahatmarios, okay.06:21
soniya29akahat, will need some time ..give me 5 mins06:21
ysandeepchandankumar, marios thanks!06:23
mariosakahat: rr status: https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ?view06:23
marios(channel topic)06:23
ysandeepbhagyashris, want to meet to sort out jira?06:25
bhagyashrissandeep yeah just added all the cards 06:26
bhagyashriswe can review then06:26
bhagyashrisgive me min06:26
akahatmarios, http://promoter.rdoproject.org/promoter_logs/centos9_master.log06:27
mariosakahat: https://review.rdoproject.org/r/c/testproject/+/4121906:28
mariosakahat:         * |                                  |            | | Missing Jobs    | periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone06:28
bhagyashrisysandeep, https://meet.google.com/ttd-ovwk-vxf06:29
mariosakahat: soniya29: is downstream back? 06:30
mariosakahat: waiting for https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/836091 (also affects downstream )06:32
*** akahat is now known as akahat|rover06:35
pojadhavmarios, yes downstream is back07:08
pojadhavI can access : https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds now07:09
*** jpena|off is now known as jpena07:11
mariospojadhav: yeah thanks it is but 16 line is hit by the gate blocker 07:12
pojadhavmarios, ack07:14
*** amoralej|off is now known as amoralej07:14
akahat|rover@all please review: https://review.rdoproject.org/r/c/config/+/40443, https://review.rdoproject.org/r/c/rdo-jobs/+/4076407:32
ysandeepakahat|rover, dasm|off 17 on rhel-9 container build is failing07:37
akahat|roverysandeep, ack.. looking in to it.07:38
ysandeepakahat|rover, can you join #openstack-pcci in internal07:38
ysandeepI can ping logs there07:38
akahat|roverysandeep, joined.07:39
ysandeepthanks!07:41
*** pojadhav- is now known as pojadhav08:13
ysandeepchandankumar, fyi.. https://review.opendev.org/c/openstack/tripleo-quickstart/+/836104 looks like rcastillo already posted something.08:15
* akahat|rover lunch08:15
marioschandankumar: ysandeep: please check when you have a sec need it for a cix https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/835974/3#message-039ee23993ea97f74deae8f604e2b7aff4b4812d 08:18
mariosakahat|rover: fyi ^^ 08:18
ysandeepmarios: +wed08:20
mariosthanks ysandeep 08:21
*** ysandeep is now known as ysandeep|lunch08:31
chandankumarpojadhav: marios: ysandeep|lunch when you get some time, please have a look at zuul job parenting doc: https://review.opendev.org/c/openstack/tripleo-docs/+/83578208:44
chandankumarI just need to add links there, all the content is added there08:44
marioschandankumar: ack adding to reviews08:54
*** ysandeep|lunch is now known as ysandeep09:15
ysandeepchandankumar, ack, will check in my next round of reviews09:15
mariosnyan cat on hackmd :D09:52
mariosonly moves when you type09:52
marioschandankumar: ysandeep: please review when you next have time  https://review.opendev.org/c/openstack/tripleo-ci/+/831543 c8/wallaby teardown standalones 09:53
ysandeepmarios, ack09:53
mariosthanks ysandeep 09:54
*** marios is now known as marios|food|biab09:54
pojadhavchandankumar, ack10:00
rlandychandankumar: akahat|rover: welcome back10:25
akahat|roverrlandy, o/10:26
*** marios|food|biab is now known as marios10:26
rlandyakahat|rover: looks like you guys finally got https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/836091 merged10:27
rlandycool10:27
akahat|roveryeah.. it got merged today10:28
rlandyakahat|rover: marios: dasm|off: thank you  ... going to reply to openstack-discuss to say this is fixed10:28
rlandyakahat|rover: did you sync yet with marios?10:29
mariosrlandy: ack chandankumar also helped by pinging folks on ansible chan here 10:29
mariosrlandy: yes me and akahat|rover had a call this morning10:29
akahat|roverrlandy, yes. we had sync10:29
mariosand ysandeep helped too on ansible-role-python_venv_build/+/836091 10:29
rlandyakahat|rover: ok - let's sync briefly about downstream and then we can talk when dasm|off is in10:30
akahat|roveranyone is able to reach https://code.engineering.redhat.com/gerrit/? 10:30
mariosakahat|rover: no seems down10:30
mariosbut was ok earlier 10:30
chandankumarrlandy: \o10:30
ysandeepakahat|rover, its slow - known issue 10:30
akahat|roveryeah.. it was working.. 10:30
rlandyakahat|rover: started working yesterday10:30
akahat|roverysandeep, okay. 10:30
rlandyakahat|rover: I have a testproject for the missing master job10:30
rlandyakahat|rover: periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-mastertestprojectmastercheck301027,910:31
rlandy1 hr 23 mins 18 secs2022-04-01 07:08:35SUCCESS10:31
mariosrlandy: great it should promote now then ^^^^ 10:31
ysandeeprlandy, akahat|rover fyi.. console is not coming on internal zuul.. I have pinged infra already - they have a fix but awaiting for gerrit to comes back.10:32
rlandyysandeep: ack - discussed that with nhicher yesterday10:32
rlandy<rlandy|rover> nhicher: can you tell if these jobs are actually running? the consoles are blank10:32
rlandy<nhicher> rlandy|rover: we have a change on gate to enable console, we were not able to merge it before https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/local/status10:32
rlandyysandeep: ^^10:32
akahat|roverI did run one testproject, it got success but didn't passed the right hash.. so no promotion!!10:33
ysandeeprlandy, akahat|rover fyi.. i have retriggered ovb job for tripleo component(currently running), hopefully after tripleo promotion - container build issue on 17/rhel-9 solves.10:34
rlandyakahat|rover: https://meet.google.com/rsi-dpvr-waa?pli=1&authuser=010:35
rlandywe need to talk about 16.210:35
rlandywe network there firts10:35
akahat|roverrlandy, joining in 5 mins10:35
rlandyand then we can look at which  hash ran on master10:35
rlandyysandeep: the component we need now is network10:36
rlandyfor 16.210:36
rlandythe rest can wait10:36
ysandeeprlandy: ack.. I am working on baremetal job for 17, need containers to run the job - that's why went looking why container build failing.10:37
rlandyakahat|rover: marios: kvm jobs ran on hash: dd343828a53c618e13e8e0187af3d2c110:37
akahat|roveroh.. we needed this hash for promotion: 7c3595fcdce0ec20189de8d5b99dec1610:38
rlandy"tripleotraincentos8" is promoting now10:39
mariosakahat|rover: rlandy: testproject https://review.rdoproject.org/r/c/testproject/+/41219 was for 7c3595fcdce0ec20189de8d5b99dec1610:39
mariosrlandy: heh again? it promoted already today train10:40
mariosakahat|rover: dasm|off: so after you get master wallaby 9 needs to be your main concern for upstream promotions10:41
akahat|rovermarios, ack10:42
rlandyakahat|rover: --ref refs/heads/master --newrev 04ce9923d0bfdc800b159a73f380e5843c6c7af311:04
rlandymarios: ack - akahat|rover is getting master promoted now11:07
rlandyysandeep: wanted to meet with you, dasm|off and rcastillo about the boot modes 11:10
rlandyysandeep: maybe tuesday if PTG is done11:10
akahat|roverc9 master is promoting: http://promoter.rdoproject.org/promoter_logs/centos9_master.log11:13
ysandeeprlandy: sounds good to me11:13
ysandeeprlandy: review time if you want to join.11:16
*** dviroel|afk is now known as dviroel11:22
*** ysandeep is now known as ysandeep|afk11:51
*** soniya29 is now known as soniya29|afk12:04
*** pojadhav- is now known as pojadhav12:18
*** amoralej is now known as amoralej|lunch12:27
rlandyakahat|rover; woohoo current-tripleo/2022-04-01 11:50  - maste - pls revert the change if you have not done so already12:45
rlandyugh - all the lines are kicking together12:48
chandankumarmarios: https://review.opendev.org/c/openstack/tripleo-ci/+/831543/17#message-12a6daffd174831f2fd95603f3281608ef7865ce please have a look, thanks!12:50
marioschandankumar: ack replied thanks we need both for now, we could remove the ones in layout (not on the job definition) as a future cleanup12:59
chandankumarmarios: thanks!13:07
akahat|roverchandankumar, marios victoria is https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable213:10
akahat|roverdo we need to backport this; https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/83609113:10
chandankumarakahat|rover: nope13:10
chandankumarakahat|rover: it consumes from master13:11
akahat|roverchandankumar, patch got merged. then why it is failing?13:12
mariosakahat|rover: ran earlier13:12
mariosakahat|rover: it ran e.g. 0500 this morning https://review.rdoproject.org/zuul/build/9dbec610ea3b42a49edf0f5fa89e17fc 13:12
mariosakahat|rover: Started at 2022-04-01 05:05:4813:12
mariosakahat|rover: next run should be green 13:13
akahat|rovermarios, okay. 13:13
*** amoralej|lunch is now known as amoralej13:15
*** ysandeep|afk is now known as ysandeep13:28
*** dasm|off is now known as dasm|ruck13:38
dasm|rucko/13:38
* dasm|ruck catching up on lines13:38
dasm|ruck1st good thing: bug fixed \o/13:39
rlandydasm|ruck: akahat|rover: let's sync at 2:30 UTC13:51
rlandyakahat|rover: can you catch dasm|ruck in the mean time13:52
rlandydasm|ruck: downstream is still unstable13:52
rlandynetwork line started kicking13:52
rlandyif that passes, will rekick 16.213:52
rlandydasm|ruck: master just promoted13:53
rlandytrain yesterday13:53
rlandyneed victoria and wallaby c9 today13:53
rlandygate is ok13:53
dasm|ruck2:30 pm utc sounds goot13:54
dasm|ruckgood13:54
akahat|roverdalvarez, https://review.rdoproject.org/r/c/testproject/+/4127813:56
akahat|roverdalvarez, please ignore13:56
akahat|roverdasm|ruck, ^^13:57
akahat|roverdasm|ruck, https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/83609113:58
akahat|roverdasm|ruck, https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-9/component/master.yaml14:05
akahat|roverdasm|ruck, https://code.engineering.redhat.com/gerrit/c/testproject/+/30102714:10
akahat|roverdasm|ruck, http://promoter.rdoproject.org/promoter_logs/centos9_wallaby.log14:11
dasm|ruckakahat|rover++14:13
rlandychandankumar: hello ... did you see the PTG etherpad for c9?14:41
rlandychandankumar: if you are around, we should dicuss14:41
rlandyalso ...14:41
rlandymarios: ysandeep: chandankumar: we have a slot at the April 26th all hands meeting14:42
rlandyto talk about EL(14:42
rlandyEL914:42
rlandyare you guys ok to present14:42
rlandyupstream and downstream14:42
rlandywe can cover both14:42
rlandyplus stream pinning etc14:42
rlandydviroel: rcastillo: ^^ fyi14:42
mariossure 14:43
rlandyyou guys might want to add a few slides14:43
rlandywe get 15 mins14:43
rlandyapril 2614:43
ysandeep+1 I am available on 26th, I can cover downstream14:43
mariosyeah we can discuss next week 14:43
dasm|rucknice: green sc10 kvm14:48
dviroelrlandy: ack, sounds good 14:48
chandankumarrlandy: sounds14:52
chandankumargood14:52
chandankumarrlandy: regarding ptg, I saw the etherpad, Where we want to put to cs9 parenting stuff?14:53
*** ysandeep is now known as ysandeep|out14:53
rlandychandankumar: ack - pls do14:54
chandankumarrlandy: quick phone?14:54
rlandychandankumar: you and marios are presenting that14:54
rlandypls work it out with him14:54
marioschandankumar: https://etherpad.opendev.org/p/tripleo-zed-centos-914:54
mariosthere no? ^ 14:54
chandankumarmarios: yes14:54
marioschandankumar: add some links line 3714:54
chandankumarmarios: sure14:55
marioschandankumar: we can discuss monday our morning before ptg 14:55
rlandychandankumar: marios: ack ... I just added a few points on that page14:55
mariosquick call to sync chandankumar 14:55
chandankumarmarios: sounds good :-)14:55
rlandyfeel free to add/edit14:55
rlandychandankumar: it's pretty cool to read your presentation from last PTG14:55
rlandyyou guys completed EVERYTHING there14:55
rlandyin 6 months14:55
rlandyamazing team work!!!14:56
rlandychandankumar: there are two sections - what did we learn and left to do...14:56
rlandyyou can add reparenting in either14:56
rlandyor both14:56
rlandydviroel; you all set with the pinning talk?14:57
dviroelrlandy: yes o/14:57
rlandymarios,: we are also up for CI load removal - but that is pretty light14:57
rlandydviroel++ 14:57
rlandydasm|ruck: 16.2 network line running now14:58
dasm|ruckack14:58
dasm|ruckit's less scary type of work if you know what you're doing ;)14:58
chandankumarrlandy: sure14:59
chandankumardviroel: jm1 rcastillo dasm|ruck need review on documentation https://review.opendev.org/c/openstack/tripleo-docs/+/835782 and fine print is here: https://effe4ed80a91c92fc386-ef4309a852fb3e3584cdcb1adbb4ea34.ssl.cf2.rackcdn.com/835782/6/check/openstack-tox-docs/da25d26/docs/ci/tripleo_ci_job_parenting.html15:00
chandankumarplease have a look, when free :-)15:00
dasm|ruckack15:00
dviroelchandankumar: will do15:00
rcastillochandankumar: ack15:00
dasm|ruckchandankumar: i see it's in review doc. thanks for that15:01
mariosrlandy: ack went over that a bit should be fine15:08
mariosrlandy: honestly its likely just going to be us and few folks from df ... lets see i don't expect a massive turnout but we'll see15:08
* dviroel lunch15:09
*** dviroel is now known as dviroel|lunch15:09
rlandymarios: what??  it could be the event of the century??? you never know :)15:10
marios;)15:10
* dasm|ruck brb15:29
mariosdasm|ruck: akahat|rover: need something ? am going in a few mins15:29
akahat|rovermarios, no. good for now.15:30
akahat|roverthank you!! :)15:30
marioso/ have a nice weekend tripleo-ci 15:35
jm1marios: thank you, you too ^^15:36
*** marios is now known as marios|out15:42
* dasm|ruck back15:50
*** dviroel|lunch is now known as dviroel16:00
rlandydasm|ruck: will sync with you after 1-116:01
rlandylike 30 mins16:01
dasm|ruckack16:01
rlandydasm|ruck: ready to sync16:18
rlandyugh - failed again ... periodic-tripleo-ci-rhel-8-containers-multinode-network-rhos-16.216:19
dasm|ruckrlandy: here16:19
dasm|ruckrlandy: https://meet.google.com/ysp-ydox-ncm16:19
*** amoralej is now known as amoralej|off16:38
*** rlandy is now known as rlandy|biab16:55
*** jpena is now known as jpena|off17:04
*** rlandy|biab is now known as rlandy17:45
jm1rlandy: Do we have any docs on the SLAs our team has to provide?19:04
rlandyjm1: 1 week on promotions19:04
rlandythat's basically it19:04
rlandygates are clean19:04
jm1rlandy: so it is more like one of your (and thus our) goals?19:05
rlandypretty much - it's the agreement we have with prod chain19:05
jm1rlandy: ok19:06
rlandydasm|ruck: container builds are failing on internal rhos-1719:12
dasm|ruckack19:13
rlandylooking into that19:13
dasm|rucklemme see what's happening19:13
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/20bb1b7/logs/build.log19:13
dasm|ruckack19:13
dasm|ruck> file already closed\nFailed to write to log, write 19:13
dasm|rucksounds like an intermittent error we were hitting for a quite some time19:13
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/20bb1b7/logs/container-builds/eb9471d0-16a8-4f56-842d-b000195c03b1/base/ovn-bgp-agent/ovn-bgp-agent-build.log19:13
rlandy+ [[ '' != \o\v\n\-\b\g\p ]]19:14
rlandy+ echo 'User ovn-bgp was not found in the supported list'19:14
rlandyUser ovn-bgp was not found in the supported list19:14
rlandy+ exit 119:14
rlandydasm|ruck: ^^ that's the issue19:14
dasm|ruckhmm.19:14
rlandychecking if it's the same for rhel-819:14
dasm|rucki see19:14
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/6cbe3bf/logs/container-builds/7a715a2c-09b5-4315-af58-7e4214f7ab80/base/ovn-bgp-agent/ovn-bgp-agent-build.log19:15
rlandydasm|ruck: ^^ same19:15
dasm|ruckmhm19:15
rlandythere is something you can write up19:15
rlandybugzilla19:15
rlandyhave you ever done that?19:15
dasm|rucknope. not with bugzilla19:15
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-1719:16
dasm|ruckhttps://opendev.org/openstack/tripleo-common/src/branch/master/container-images/kolla/base/uid_gid_manage.sh#L9019:18
dasm|ruckthat might be the plaec19:18
dasm|ruck*place19:18
dasm|ruckhttps://opendev.org/openstack/tripleo-common/src/branch/master/container-images/kolla/base/uid_gid_manage.sh#L6619:18
rlandydasm|ruck: https://meet.google.com/mcm-zfim-zuq?pli=1&authuser=019:18
dasm|ruckit's hese, upstream19:19
dasm|ruck*here19:30
rlandyhttps://bugzilla.redhat.com/show_bug.cgi?id=188351819:33
dasm|ruckrlandy: fyi, network 16.2: > SUCCESS in 2h 18m 37s19:50
dviroelrlandy: hi, missing a W+1 here https://review.opendev.org/c/openstack/tripleo-ci/+/83451519:53
*** dviroel is now known as dviroel|brb20:05
* jm1 out for today20:25
jm1team, have a nice weekend :)20:26
dasm|ruckjm1: o/20:26
rlandyyeah20:30
rlandydviroel|brb: done20:30
rlandydasm|ruck: wallaby c9 - missing fs001 and fs035 to promote hash20:32
rlandycan you testproject those20:32
rlandyrr tool will give you the right hash20:32
dasm|ruckyep20:32
dasm|ruckftr: https://review.rdoproject.org/r/c/testproject/+/4137020:34
rlandythank you20:36
rlandydasm|ruck: fs039 on victoria is success true21:06
dasm|ruck\o/ some success!21:06
dasm|ruckwhat's next? do we want to remove that from promotion line and rerun promotion? rlandy ?21:06
rlandylet me check status21:07
dasm|ruckack21:07
rlandydasm|ruck: hmmm ... rr tool says it should promote21:07
dasm|ruckcheckind21:08
dasm|ruck*checking (even)21:08
rlandyso then we need to give it 10 mins and see if it will promote21:08
dasm|ruckhmm... i didn't get a test result yet21:08
rlandyyou can see the result in the console21:09
dasm|ruckhttps://review.rdoproject.org/zuul/status/change/41367,121:09
rlandyreproted to dlrn21:09
dasm|rucktempest api says failure21:09
dasm|ruckmhm21:09
rlandymust have passed on a previous run21:09
dasm|ruckchecking21:09
rlandysee rr tool21:09
rlandydasm|ruck: https://trunk.rdoproject.org/api-centos8-victoria/api/civotes_agg_detail.html?ref_hash=5996b91e558f07113a06bfec6c8cfa9e21:10
dasm|ruckmh21:11
dasm|ruckmhm21:11
dasm|ruckwe should be good to go21:11
dasm|ruckthat's good :)21:11
rlandyyep21:11
rlandydasm|ruck: some failures in eh components - will need some work next week21:12
dasm|ruckack21:12
dasm|ruckwould be much better if we would have 16.221:12
dasm|ruckbut otherwise, can we start celebrating? :)21:12
rlandyperiodic-tripleo-ci-build-containers-ubi-8-push-train also failed21:13
rlandymay be transient21:13
rlandyrekicking train to check21:13
dasm|ruck:(21:13
dasm|ruckack21:13
rlandydasm|ruck: you know how to rekick lines?21:14
dasm|ruckbut train seems to be in decent shape21:14
dasm|ruckrlandy: with podman? yeah, i remember21:14
dasm|ruck> a day ago train current-tripleo centos-7 5 hours ago21:14
rlandysending you example command21:14
dasm|ruckk21:14
rlandyyep21:14
rlandyyeah21:15
rlandyjust hoping it's a transient error on container builds21:15
dasm|ruck+121:15
rlandyso we should at least check that job befire we lose the whole weekend21:15
dasm|ruckack21:16
*** dviroel|brb is now known as dviroel21:16
dasm|ruckfyi: rbac discussion during ptg: https://etherpad.opendev.org/p/rbac-zed-ptg21:20
rlandyk - great 21:29
rlandydasm|ruck: tempest component on wallaby needs some love next week pls21:29
dasm|ruckack21:29
rlandytempest component on a few lines21:29
rlandyI'm out22:43
rlandyhave a good weekend22:43
rlandydasm|ruck: ^^22:43
rlandyrcastillo: ^^22:43
dasm|ruckhave a good one too!22:46
* dasm|ruck leaves as well22:47
*** dasm|ruck is now known as dasm22:47
*** dasm is now known as dasm|ruck|off22:47
dasm|ruck|offttyl22:47

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!