Monday, 2022-03-21

*** rlandy is now known as rlandy|out00:03
*** pojadhav is now known as pojadhav|rover03:53
*** ysandeep|out is now known as ysandeep04:13
*** chkumar|rover is now known as chandankumar05:33
mariosarxcruz|ruck: o/ morning07:21
marioslet me know if you need something/reviews/digging/espresso etc07:22
chandankumarmarios: Good morning, I have merged RDO standalone parenting https://review.rdoproject.org/r/c/rdo-jobs/+/40339 and I think we can merge Amol standalone patch https://review.opendev.org/c/openstack/tripleo-ci/+/830987 once zuul gives +107:41
marioschandankumar: ack thanks... we added wf-1 on friday to prevent accidents over weekend (cos it already has 3 +2s ;))07:43
mariosi removed that a little while ago you probably saw 07:43
mariosso yeay 07:43
marioslets go 07:43
arxcruz|ruckmarios: good morning, ack, just started, i'll start checking stuff now 08:01
mariosarxcruz|ruck: o/ ack 08:02
mariosarxcruz|ruck: gates look pretty green 0 fails08:02
mariosbut was weekend so ... ;)08:02
arxcruz|ruckyeah08:03
arxcruz|ruckholly shit, do my eyes deceive me? only one failure on master?08:04
*** amoralej|off is now known as amoralej08:12
arxcruz|ruckjpodivin hey, do you have any news about https://bugs.launchpad.net/tripleo/+bug/1965426 ?08:15
jpodivinarxcruz|ruck: I do, although I'm not sure what to make of them.08:16
jpodivinThe jobs are coming up green, which is good. but ...08:17
jpodivinThe reason they are succeeding seems to be that the behavior triggering the failure "wrong branch being checked out" isn't happening anymore ...08:17
jpodivinbecause no branch of the project is being checked out. 08:18
jpodivinAnd I don't think we have merged anything that could fix this.08:18
jpodivinarxcruz|ruck: compare successful (https://f38d3934203b991a7ebe-d701f6a98461967df274f184d5a7d3cd.ssl.cf2.rackcdn.com/827628/2/gate/tripleo-ci-centos-8-content-provider/2dcbb84/job-output.txt) and failing (https://60ce53b5f45ed0ef7bd3-e873feb845d99f2e0685947947034235.ssl.cf1.rackcdn.com/827628/2/gate/tripleo-ci-centos-8-content-provider/19276d3/job-output.txt) run to see what I mean08:20
jpodivinThe one that succeeds never checks out the validations-libs component. At all. So it can't have a branch mismatch and neither the error08:21
arxcruz|ruckhmmmm08:24
arxcruz|ruckweird 08:24
jpodivinyeah, I don't like it.08:26
jpodivinEspecially because I had a general idea in my mind of what was wrong, and now the fix is useless :D08:27
*** jpena|off is now known as jpena08:35
*** ysandeep is now known as ysandeep|lunch09:22
mariosback in a bit 09:28
*** pojadhav|rover is now known as pojadhav|lunch09:36
*** ysandeep|lunch is now known as ysandeep10:16
*** pojadhav|lunch is now known as pojadhav|rover10:18
*** rlandy|out is now known as rlandy10:24
rlandyarxcruz|ruck: pojadhav|rover" hello10:24
rlandyhow are things?10:24
arxcruz|ruckrlandy https://meet.google.com/fod-xhqt-csw10:25
pojadhav|roverarxcruz|ruck, https://review.rdoproject.org/r/c/testproject/+/3358210:25
ykarelis the issue caused with https://review.rdoproject.org/r/c/config/+/39444 already known?10:33
ykarelThe conditional check 'push_registry in [\"quay.rdoproject.org\"]' failed10:33
ykarelarxcruz|ruck, pojadhav|rover ^ 10:34
arxcruz|ruckykarel i'm checking this now 10:34
ykarelseen in https://logserver.rdoproject.org/09/40609/5/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset001-wallaby/ebe42df/job-output.txt10:34
ykarelack10:34
chandankumarykarel: https://review.rdoproject.org/r/c/config/+/39444 got merged recently 10:34
ykarelchandankumar, yes i referred that only10:34
chandankumarah I missed that review10:36
rlandyjpodivin: hello - can you update the bug and card https://trello.com/c/LPQw83av/2410-cixlp1965426tripleociproa-tripleo-ci-centos-8-content-provider-failing-to-rpm-build-python-validations-libs-on-stable-branches10:37
rlandyahead of the CIX call10:37
rlandyarxcruz|ruck: https://bugs.launchpad.net/tripleo/+bug/196494010:38
rlandyfrenzyfriday: https://review.rdoproject.org/r/c/config/+/3944410:40
rlandypls see ykarel comment10:40
rlandyreverting this patch10:40
rlandybreaking 10:40
rlandypls revisit it10:40
frenzyfridayrlandy, checking10:40
rlandymarios: chandankumar: pls vote ob revert https://review.rdoproject.org/r/c/config/+/4038510:41
mariosohdear... probably need to default it to something? but anyway better to revert and fix 10:41
rlandyrevert in progress10:41
mariosrlandy: k voted 10:41
chandankumarrlandy: frenzyfriday ykarel fix https://review.rdoproject.org/r/c/config/+/4074610:41
rlandychandankumar: do you want to merge the fix rather?10:42
rlandyw'ed the revert10:42
chandankumarrlandy: yes, we can merge the fix10:42
frenzyfridaychandankumar, ack, this should fix it10:42
arxcruz|ruckykarel any update on https://bugs.launchpad.net/tripleo/+bug/1964940 ?10:42
rlandychandankumar: ok - we can merge the fix10:43
rlandywill -1 my revert10:43
rlandychandankumar: pls get votes on that asap10:43
rlandymarios: sorry - can you remove +110:44
rlandygoing with chandankumar's fix10:44
mariosyeah deafult should fix it 10:44
ykarelchandankumar, ack10:45
ykarelarxcruz|ruck, no new update apart from last comments10:46
chandankumarrlandy: marios hitted +w on https://review.opendev.org/c/openstack/tripleo-ci/+/830987 standalone reparenting10:57
chandankumarwith this, it completes the standalone parenting10:57
rlandyok10:57
rlandypojadhav|rover: testpoject on 17 has a failure - so you can see if its consistent10:59
rlandytripleo-downstream-trigger-nested-virt11:00
rlandy40743,111:00
rlandyunknown11:00
rlandy2 hr 6 min11:00
rlandypojadhav|rover: ^^ queued for 2.5 hours11:00
rlandyplease try that job again11:00
pojadhav|roverfs01 having stack quota exceeds issue..11:00
pojadhav|roverbut latest testrun failed with tempest tests which are multiple11:01
rlandypojadhav|rover: downstream looks stuck11:01
rlandythings are queued for hours11:01
rlandypls ping rhos-ops11:01
rlandysee 3 hrs for component promotions11:01
rlandyoh the 2 hr job just started11:01
rlandyclients was running for 7 hrs11:02
rlandypojadhav|rover: ^^11:02
pojadhav|roverrlandy, ack11:02
rlandypojadhav|rover: pls check if we need to clean resources there11:03
rlandyovb11:03
*** chem is now known as Guest272011:03
marioschandankumar: ack 11:14
marioschandankumar: well some moreon my patch i mean reparenting for scenario standalones11:14
marioschandankumar: but lets discuss in call now11:14
chandankumarmarios: sure11:14
rlandychandankumar: hello ... status of https://review.rdoproject.org/r/c/rdoinfo/+/40653?11:15
rlandyto update status of https://trello.com/c/EsgceuAN/2342-cixlp1960310tripleociproa-compute-tests-eg-testrebuildserverwithvolumeattached-failing-with-volume-id-failed-to-reach-available11:15
*** dviroel|out is now known as dviroel11:17
jpodivinrlandy: sure, but as I've said to arxcruz|ruck it's a bit complicated. 11:17
chandankumarrlandy: still working on getting the rpm built11:17
chandankumarupdated the comment on trello itself11:18
chandankumarrlandy: joining the review call11:21
rlandychandankumar: still busy with ruck rover11:21
rlandypls carry on w/o me11:21
rlandygo ahead and merge what you need11:22
arxcruz|ruckrlandy https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4074811:24
jpodivinrlandy: I've updated the trello and the bug. I'm going to provide more details soon, after I get my lunch :) 11:24
chandankumarrlandy: any update on downstream import switch?11:32
rlandychandankumar: no news yet11:33
chandankumarfor rhos-1711:33
rlandywill ask at prod chain call11:33
chandankumarok11:33
arxcruz|ruckrlandy https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4074911:33
chandankumarmarios: thanks!11:36
chandankumarrlandy: thanks!11:36
mariosthanks chandankumar :)11:37
*** ysandeep is now known as ysandeep|break11:39
arxcruz|ruckrlandy https://review.rdoproject.org/r/c/testproject/+/3791911:42
arxcruz|ruckrlandy https://review.rdoproject.org/r/c/testproject/+/3902511:43
* pojadhav|rover brb12:02
* dviroel coffe12:08
rlandypojadhav|rover; pls ping when you are back12:10
chandankumarrlandy: marios dviroel https://review.rdoproject.org/r/c/config/+/40382 review please12:18
chandankumarif it gets merged, I can test container build12:18
dviroelok12:18
marioschandankumar: k adding to list 12:19
rlandypojadhav|rover: wrt [rhos-dev] [CIX][osp17][rhel8][BZ:2066264][RHOS17] - OVB jobs jobs fs01 and fs035 are failing with tempest tests related to compute/image tests ... n test_image_web_download_import12:25
rlandy    image_uri=image_uri)12:25
rlandypls see https://opendev.org/openstack/openstack-tempest-skiplist/commit/073d4726171af21c49f1bb0e19325bd4582454c512:25
rlandypls check if it is the same error12:25
rlandyand add to skiplist if it is12:25
rlandyhttps://bugs.launchpad.net/tripleo/+bug/196554012:26
rlandydviroel logged that on friday12:26
rlandyarxcruz|ruck: ^^ may be the same bug in downstream12:27
dviroelyeah, and we promoted tempest component that day (finally)12:28
arxcruz|ruckrlandy ack 12:28
pojadhav|roverrlandy, ack12:47
chandankumarkopecmartin: hello12:50
chandankumarkopecmartin: while building rpm for tempest-master for wallaby hitting this issue https://review.rdoproject.org/r/c/rdoinfo/+/40653/4#message-4762919462cfcba2b6bcd31b023579c6ef794a77, seen it?12:51
rlandyarxcruz|ruck: "tripleoussuri" is promoting13:01
arxcruz|ruckrlandy https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40386 the revert patch 13:01
rlandyarxcruz|ruck: thanks  - will merge after promotion13:03
arxcruz|ruckack 13:03
*** ysandeep|break is now known as ysandeep13:05
*** amoralej is now known as amoralej|lunch13:06
kopecmartinchandankumar: hi, no .. i think i hit something similar very long time ago, therefore i had to modify the run-distgit playbook - https://review.rdoproject.org/r/c/rdo-jobs/+/27710/21/playbooks/run-distgit.yaml13:07
kopecmartinthat's ^^ a patch i used to use to test tempest bumps for T, U and V13:07
kopecmartini don't have anything like that for the newer releases13:08
kopecmartinthis always seemed to me, that it should be fixed by someone who maintains the DLRN playbook, even better if they could provide the jobs as well13:08
rlandypojadhav|rover: once you added the skiplits, pls rerun those tests to promote 1713:13
pojadhav|roverrlandy, added to skiplist https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/83446213:13
pojadhav|roverrerunning jobs now13:13
rlandypojadhav|rover: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/83446213:14
rlandyyou need the donwstream installee13:14
pojadhav|roverthere are some more tempest tests failing but those are sporadic. not consistent one.13:14
pojadhav|roverrlandy, ack checking13:14
arxcruz|ruckpojadhav|rover or remove the installers 13:15
rlandyarxcruz|ruck: rekick fs039 master testproject  - hit the mirror issue13:16
arxcruz|ruckrlandy ok 13:17
arxcruz|ruckty 13:17
pojadhav|roverrlandy, removed installers13:17
rlandypost failures13:18
rlandy2022-03-21 13:14:34.736149 | primary | ERROR: Could not install packages due to an OSError: HTTPConnectionPool(host='mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org', port=80): Max retries exceeded with url: /pypifiles/packages/a5/17/e8de7e01317d25e93e6584cb748509f2c3d401529922a8b5d36897200996/clg-3.1.0-py3-none-any.whl (Caused by NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 13:18
rlandy0x7feeb00af940>: Failed to establish a new connection: [Errno 113] No route to host',))13:18
arxcruz|ruckrlandy https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/83446413:21
arxcruz|ruckfs001 fail only on the web-download test, adding it to tskiplist 13:21
rlandypojadhav|rover: ^^ same test you are edit??13:22
rlandyediting13:22
pojadhav|roverrlandy, yes13:22
rlandypojadhav|rover: pls rebase on arxcruz|ruck change then13:23
rlandygoing to merge that13:23
pojadhav|roverrlandy, sure13:23
chandankumarkopecmartin: ok, but the rpm build job is failing at %check section where we run unit tests, https://logserver.rdoproject.org/53/40653/4/check/rdoinfo-DLRN-check-centos9/a567706/buildset/centos-wallaby-rdo/repos/component/tempest/5c/55/5c556626d16c2a4c6ee9c9da45d4954db6f83f27_dev/rpmbuild.log13:24
rlandyarxcruz|ruck: will rekick wallaby c9 once the post failure clears13:25
arxcruz|ruckok 13:25
chandankumarit has not reached at dist-git playbook13:25
pojadhav|roverrlandy, rebased my patch on arxcruz|ruck patch. https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/83446213:26
rlandyarxcruz|ruck: ^^ is that ok without any installers?13:27
rlandyif so I am +213:28
rlandyarxcruz|ruck: pls merge if you agree13:28
arxcruz|ruckpojadhav|rover need to remove from master as well 13:28
arxcruz|ruckor rhos-17 return wallaby ?13:29
pojadhav|roverrhos-17 is wallaby13:29
*** dasm|off is now known as dasm13:29
dasmo/13:29
rlandytrain c8 is promoting now13:29
rlandywallaby for rhos-17 is correct13:29
arxcruz|ruckthen it's fine 13:30
rlandyCIX time13:30
arxcruz|ruckis meet.google.com/whq-pcfz-xsr the right one? there's no one there 13:30
marioschandankumar: i am going to be ruck next (starting this friday) but i am out on public holiday on friday ... can you please cover me (payback ;)) ? otherwise I will try and swap with next ruck 13:30
chandankumarmarios: 25th Mar, done13:31
marioschandankumar: please thank you chandan++13:31
chandankumarhehe thanks to you marios , you always cover our back :-)13:32
marioschandankumar: quid pro quo it works both ways mate 13:32
mariosthanks chandankumar 13:32
rlandypojadhav|rover: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/834462 merged13:54
rlandycan you rerun 17 tests for promotion13:54
bhagyashrisakahat, arxcruz|ruck chandankumar dviroel dasm rcastillo pojadhav|rover soniya29 ysandeep rlandy 14:00
bhagyashrisscrum time14:00
dasmbhagyashris: thanks, will join after rbac meeting14:00
bhagyashrisdasm, ack14:01
soniya29bhagyashris, i will join after rbac meeting14:01
dviroeljoinning14:01
pojadhav|roverrlandy, yes reran : https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#31931414:03
rlandythank you14:03
*** amoralej|lunch is now known as amoralej14:07
arxcruz|ruckbrb14:08
rlandypojadhav|rover: should that job be dependent on neutron?14:09
pojadhav|roverrlandy, not sure14:11
rlandy[DIR]current-tripleo/2022-03-21 14:18 14:21
rlandyarxcruz|ruck: ^^ train promoted14:22
rlandyreverting patch14:22
rlandyhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4038714:23
rlandyarxcruz|ruck: pojadhav|rover: ^^14:23
pojadhav|roverrlandy, ack14:25
dasmmarios: what's harold irc handle?14:28
mariosdasm: hjensas in tripleo 14:28
dasmack, thanks14:28
ysandeepdasm: what's the issue you are hitting - could you please give some details?(I want to try my luck :D)14:29
chandankumardviroel: you might want to propose a patch add this centos-compose-repos.yml at https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base-upstream.yaml#L75 and https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base-upstream.yaml#L272 otherwise it will not run in integration line14:30
dasmysandeep: https://bugs.launchpad.net/tripleo/+bug/196478314:30
chandankumarcurrently we are using the jobs from base-upstream.yaml , in future we will be removing all jobs from base.yaml14:30
dasmysandeep: i tried to summarize it in the bug. if there's something missing, please let me know, i'll try to add more details14:33
arxcruz|ruckrlandy ack 14:33
dviroelchandankumar: will take a look after scrumm, tks14:33
ysandeepdasm, thanks! please feel free to reach out to steve/harald as planned- Incase I find something I will ping you/comment on bug.14:34
dasmysandeep: ack. pinged hjensas already. Will continue work on thate.14:34
dasm*that14:34
*** pojadhav|rover is now known as pojadhav|afk14:57
chandankumarrlandy: dviroel in your afternoon, please merge these https://review.rdoproject.org/r/c/config/+/40382 and https://review.rdoproject.org/r/c/config/+/40744, thanks!15:11
chandankumarsee ya!15:12
dviroelo/15:12
dasmo/15:13
rlandychandankumar: thanks  - see you tomorrow15:14
*** dviroel is now known as dviroel|lunch15:29
rlandymarios: ysandeep: https://code.engineering.redhat.com/gerrit/c/osp-trunk-playbooks/+/31777915:34
rlandyfyi pojadhav|afk: arxcruz|ruck: ^^15:34
arxcruz|ruckack15:35
ysandeeprlandy, marios: As we are scaling down c8/wallaby coverage, Because we not scaling down rhel8/17 coverage currently, I am a bit skeptical that we will start hitting EL8 specific issues in downstream only and we will have more work there.15:36
rlandyysandeep: since rhel 8.4 and centos 8 will start diverging15:50
rlandyand they have already15:50
rlandystream 8 is rhel 8.615:50
rlandyupstream will be a less accurate rep of downstream15:51
rlandywe can keep some jobs15:51
rlandybut we should be selective imho15:51
ysandeeprlandy, I agree with you that we need to be selective what what we keep. fyi, I have tagged you and marios on checkpoint doc in c8/wallaby coverage conversation.15:52
ysandeepso that you can read what is the proposal.15:53
rlandyysandeep: thank you for staying on top of that15:54
ysandeep:) my pleasure 15:55
mariosysandeep: ack as discussed we will be keeping at least one job we already know about (ffu undercloud upgrade 8)16:01
mariosysandeep: so we can add more into that line as needed16:01
ysandeepmarios++ thanks!16:02
*** ysandeep is now known as ysandeep|out16:05
rlandyysandeep|out: marios: will add proposal for what we want to keep to PTG16:06
rlandyarxcruz|ruck: got stable and c9 in rerun16:09
arxcruz|ruckrlandy i rerun kvm 16:09
rlandyand the two testprojects for master and16:09
rlandyand victoria16:09
rlandyfollowing vexxhost's upate16:09
rlandyupdate16:09
rlandylet's see how that goes16:09
arxcruz|ruckok 16:10
rlandyarxcruz|ruck: at least we are clearing some lines16:10
arxcruz|ruckI hope so :) 16:10
rlandyarxcruz|ruck: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status16:11
rlandybunch of kvm jobs running now16:11
rlandylunch - brb16:11
rlandypojadhav|afk: pls check the hackmd in your morning - leaving you notes there16:12
rlandyasking attila to rekick one 16.2 job16:12
rlandyalst one we need for promotion16:12
abregmanhey, I'm unable to see any jobs here http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/2tivP9BWz/component-pipeline?orgId=116:15
abregmaneven after changing the time range filter. any idea why?16:15
abregmanand another question related to that, if I would like to see all the jobs for a specific component, what would best source for that? (in a human readable format)16:16
abregmanarxcruz|ruck, marios, rlandy, ysandeep: ^16:23
rlandyhttp://tripleo-cockpit.lab4.eng.bos.redhat.com/d/v8gltz4Mz/rhos-17-full-component-pipeline?orgId=116:25
rlandyabregman: ^^ pls try that16:25
rlandyor16:26
rlandyhttp://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=116:26
rlandyscroll down16:26
abregmanrlandy: that's great. and is it the full list of jobs for each component?16:26
rlandyyou will see jobs per component16:26
rlandyyep16:26
abregmanrlandy; thank you :)16:26
rlandyupstream vs downstream16:26
mariosabregman: some docs i mentioned on the call there https://docs.openstack.org/tripleo-docs/latest/ci/stages-overview.html especially the component/integration stuff https://docs.openstack.org/tripleo-docs/latest/ci/stages-overview.html#the-tripleo-promotion-pipelines fwiw 16:28
mariosabregman: maybe you can re-use or at least point to it since it is referenced in the doc16:28
abregmanmarios: that's really good, thanks!16:29
mariosabregman: np16:30
*** pojadhav|afk is now known as pojadhav|rover16:30
pojadhav|roverrlandy, ack16:30
*** dviroel|lunch is now known as dviroel16:38
* pojadhav|rover leaving for the day16:40
*** pojadhav|rover is now known as pojadhav|out16:40
marioshave a good day tripleo-ci o/ /me hometime16:41
dviroelo/16:41
*** marios is now known as marios|out16:50
rlandydviroel: dasm: can you pls review  https://review.rdoproject.org/r/c/config/+/40382 and https://review.rdoproject.org/r/c/config/+/40744 for chandankumar before merge17:17
dviroelwill do17:18
rlandy "tripleomastercentos9" promoting17:31
rlandywoohoo17:31
*** jpena is now known as jpena|off17:32
dviroelwoohooo17:35
dasmbrb18:01
* dviroel going afk, will be back to check tp results19:52
*** dviroel is now known as dviroel|afk19:52
dasmdviroel|afk: o/19:55
dasmhey rcastillo. are you around? I'm checking https://bugs.launchpad.net/tripleo/+bug/1962783 where Steve mentioned that VMs weren't in uefi mode. Do you know how did he come up with that? I'd like to confirm if that could be the same issue with ipxe on my end20:38
*** dviroel|afk is now known as dviroel21:03
-opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily for a Gerrit patch upgrade and kernel update, but should return again shortly21:31
dviroeltks opendev status21:36
dviroelchandankumar: the baby doesn't let you sleep, right?21:39
*** dviroel is now known as dviroel|afk21:40
chandankumardviroel|afk: nana, baby is sleeping peacefully, I am travelling to pune for few days, 21:41
chandankumarcurrently at airport.21:41
dasmchandankumar: isn't for you like a middle of night?21:44
dasm(facepalm)21:44
dasmi just noticed your message chandankumar :)21:45
* dasm departs for now. Might get back later21:45
*** dasm is now known as dasm|afk21:45
*** rlandy is now known as rlandy|bbl21:47

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!