Thursday, 2021-09-23

*** bhagyashris is now known as bhagyashris|rover04:08
*** ysandeep|away is now known as ysandeep05:51
akahatchandankumar, ysandeep marios Please review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/80874806:01
ysandeepack added in review list for today.06:03
*** amoralej|off is now known as amoralej06:07
mariosakahat: ack -1 but mainly for the set_fact overriding what you have in defaults please check maybe i missed something but in general need more context please thanks06:17
akahatmarios, replied06:38
mariosakahat: ack 06:45
*** anbanerj|ruck is now known as frenzy_friday07:08
ysandeepchandankumar: when you have some free slot today, I need to discuss a tempest timeout issue in downstream.07:22
*** jpena|off is now known as jpena07:30
arxcruz|ruckbhagyashris|rover: hey you 07:58
bhagyashris|roverarxcruz|ruck, 0/08:00
erbotRequired files not ready after 301.0642304420471s for deps-cbs-validate 35744,1,f3774bfae6b3424ba65e03766e891b0508:18
jbadiapaarxcruz|ruck, bhagyashris|rover: I got this error "galaxy_info: \nrole_name: my_name  # if absent directory name hosting role is used instead \nnamespace: my_galaxy_namespace  # if absent, author is used instead" on a wallaby job https://review.opendev.org/c/openstack/tripleo-ansible/+/80914308:43
jbadiapathis seems to me the same error as https://bugs.launchpad.net/tripleo/+bug/194396508:43
arxcruz|ruckjbadiapa: https://review.opendev.org/c/openstack/tripleo-ansible/+/810574 wouldn't fix it ?08:53
arxcruz|ruckjust notice you propose it 08:53
bhagyashris|roverhttps://trello.com/c/fFuc4xDU/2103-cixlp1943965tripleociproa-new-molecule-version-wants-a-specific-namespace-rolename-format08:54
bhagyashris|roverfix merged:08:54
bhagyashris|roverhttps://review.opendev.org/c/openstack/tripleo-ansible/+/80964308:54
jbadiapaI saw that Kevin submitted the patch, I've just cherried pick his patch to wallaby, not sure about the  https://review.opendev.org/c/openstack/tripleo-ansible/+/810574/2/zuul.d/molecule.yaml#264 08:56
bhagyashris|roveryeah08:57
jbadiapabhagyashris|rover, I've just added the tripleo-wallaby label and updated the launchpad.09:04
bhagyashris|roverjbadiapa, ack thanks :)09:04
bhagyashris|roverfrenzy_friday, hey i have sent an invite for rr sync 09:07
frenzy_fridaybhagyashris|rover, can we postpone it a bit so that wes would also be there?09:09
bhagyashris|roverfrenzy_friday, we have planning meeting 09:09
bhagyashris|roverlets see09:10
frenzy_fridayah right. Np09:10
bhagyashris|roverfrenzy_friday, i have change the time lets see he will be available or not09:11
frenzy_fridaybhagyashris|rover, thanks!09:11
bhagyashris|roveradded rlandy as well09:12
*** holser is now known as holser_09:18
*** ysandeep is now known as ysandeep|lunch09:24
frenzy_fridayhey chandankumar, which role creates the overcloud-controller-1/ etc directories?09:29
arxcruz|ruckjbadiapa: i notice Tengu point out some things on your patch, I believe this will fix your problem :)09:32
* Tengu points nothing09:33
* Tengu knows nothing09:33
* Tengu is Jon Snow (without any sword :()09:33
jbadiapaXDDD09:33
*** holser_ is now known as holser09:36
Tengujbadiapa: on a more serious note: that disabling vote was needed on master, I do hope we won't need it on wallaby. if the role-addition job ends with a timed_out, that means we'll need to disable it (for now), and we'll need to backport what will be done on master as well to re-enable it09:42
Tengufinger crossed.09:42
Tenguapparently, the dependency resolution takes far too long for that job, and it just dies after 30 minutes (there's a hard stop in zuul.d/molecule.yaml)09:42
jbadiapaTengu, I was checking the status of the zuul job regarding the tripleo_ceph_*09:43
Tengu'k. lemme know the outcome - I'll try to follow that one as well, but.... my plate is already loaded.09:44
jbadiapadue to https://review.opendev.org/c/openstack/tripleo-ansible/+/810574/3/zuul.d/molecule.yaml#274, there was a conflict09:44
jbadiapasure, thanks 09:45
*** holser is now known as holser_09:45
chandankumarfrenzy_friday: \o, can you point me the task where it is excepted to created?09:57
chandankumarfrenzy_friday: https://review.opendev.org/c/openstack/tripleo-quickstart/+/810546 might have seen this also09:58
chandankumarysandeep|lunch: hello, what about in another 30 mins for tempest downstream timeout?09:58
frenzy_fridaychandankumar, thanks lemme chck ^ review09:59
chandankumarfrenzy_friday: weshay|ruck has opened one more bug https://bugs.launchpad.net/tripleo/+bug/194461709:59
frenzy_fridaychandankumar, yes, then I think https://bugs.launchpad.net/tripleo/+bug/1944416 should be fixed by https://review.opendev.org/c/openstack/tripleo-quickstart/+/810175 if I put a depends on https://review.opendev.org/c/openstack/tripleo-quickstart/+/81054610:02
chandankumarfrenzy_friday: not sure depends on will works, as it is on the same project, may we can do a rebase on top of that10:02
frenzy_fridaychandankumar, right!!10:03
*** holser_ is now known as holser10:13
*** ysandeep|lunch is now known as ysandeep10:24
ysandeepchandankumar: hey o/ >> what about in another 30 mins for tempest downstream timeout? - Works with me, let me know when you are free.10:25
chandankumarysandeep: let's meet in another 5 mins10:26
ysandeepack10:26
*** beagles is now known as eagles10:28
jbadiapaTengu, bad news, tripleo-ansible-centos-8-role-addition    timed_out.10:48
Tengujbadiapa: dang.....10:49
*** rlandy is now known as rlandy|rover10:49
Tengujbadiapa: soooo... yeah. you'll need to include the non-voting then. I did hope that wouldn't be necessary :(10:50
rlandy|roverarxcruz|ruck: bhagyashris|rover: hello - how are things today?10:50
Tengubut ok. I'm still trying to find a "smart" way without extending the 30 minutes limit.10:50
arxcruz|ruckrlandy|rover: so far so good, i'm debugging some issues with logs not being collected 10:51
bhagyashris|roverrlandy|rover, so far good10:52
rlandy|roverarxcruz|ruck: bhagyashris|rover: anything you need help with?10:54
rlandy|roverblocker on master promotion?10:55
chandankumarTengu: role-addition is not yet fixed?10:55
* rlandy|rover will ask at ruck/rover sync10:55
Tenguchandankumar: nope10:55
chandankumar:-(10:55
Tenguchandankumar: the pip-compile generated content isn't working10:55
Tenguand I got a strong "nope" from both Alex and Kevin10:56
chandankumaryes, I saw that10:56
chandankumarwe need to go with pinning packages then10:56
Tenguchandankumar: I'm trying to find what dependencies can be """fixed""". maybe I can make the stuff faster with only 2 «constraints»10:58
*** ysandeep is now known as ysandeep|afk11:19
Tenguchandankumar: fun thing is, running that same tox job on my laptop takes a couple of minutes:  ANSIBLE_SKIP_CONFLICT_CHECK=1 tox -e role-addition  118.96s user 17.25s system 50% cpu 4:27.58 total11:24
Tenguchandankumar: but apparently, it's stuck for unknown reason on the CI: https://zuul.opendev.org/t/openstack/stream/124878c0214144a391d47c20b9e0d0ae?logfile=console.log11:25
Tenguit's not moving beyond the installdeps.11:25
*** ysandeep|afk is now known as ysandeep11:29
bhagyashris|roverrlandy|rover, rr sync11:31
*** jpena is now known as jpena|lunch11:31
frenzy_fridayhttps://bugs.launchpad.net/tripleo/+bug/1944584 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/81047411:35
frenzy_fridayhttps://bugs.launchpad.net/tripleo/+bug/194441611:45
frenzy_fridayhttps://bugs.launchpad.net/tripleo/+bug/194461711:45
Tenguchandankumar: it's as if the pip-compile doesn't take the upper-constraint thing provided via the "--pip-args '-c ....'" parameter in the cli.11:47
Tenguchandankumar: and, apparently, wallaby will get that exact same issue.... so I'm a bit stuck right now.11:47
TenguI don't want to extend the timeout, there's no real reason imho for that. but we're missing data as to why it's actually stuck11:48
rlandy|roverperiodic-tripleo-ci-centos-7-containers-multinode-train11:49
chandankumarTengu: can you pass the review link?11:50
bhagyashris|roverarxcruz|ruck, rlandy|rover Done  https://review.rdoproject.org/r/c/testproject/+/32137 [DNM] Test c7 train failing jobs  11:50
arxcruz|ruckbhagyashris|rover: wow, you11:50
arxcruz|ruckbhagyashris|rover: you're fast!11:51
rlandy|roverysandeep: hey - have a few minutes to touch base about downstream11:52
ysandeeprlandy|rover: sure11:52
Tenguchandankumar: https://review.opendev.org/c/openstack/tripleo-ansible/+/81054711:52
Tenguchandankumar: feel free to take over - right now I'm a bit lost11:52
rlandy|roverysandeep: https://meet.google.com/gou-ndcv-okj?pli=1&authuser=011:53
arxcruz|ruckbhagyashris|rover: https://review.rdoproject.org/r/c/testproject/+/35747 testing f3911:53
bhagyashris|roverarxcruz|ruck, ack11:53
rlandy|roverhttps://code.engineering.redhat.com/gerrit/c/neutron/+/27593411:54
mariosneeds reviews please add to your queue https://review.opendev.org/c/openstack/tripleo-quickstart/+/810410 thank you12:01
*** amoralej is now known as amoralej|lunch12:07
ysandeeprlandy|rover, chandankumar when you have time, increasing timeout for bm jobs https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/276294 12:09
ysandeepfolks Planning mtg12:31
ysandeepignore ntp issue12:32
*** jpena|lunch is now known as jpena12:33
rlandy|roverbhagyashris|rover: https://logserver.rdoproject.org/37/32137/6/check/periodic-tripleo-ci-centos-7-containers-multinode-train/f0f4a77/job-output.txt12:36
rlandy|rover2021-09-23 12:00:33.463194 | primary | ERROR! Unable to retrieve file contents12:36
rlandy|rover2021-09-23 12:00:33.463345 | primary | Could not find or access '/home/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml' on the Ansible Controller.12:36
rlandy|rover^^ missing train12:37
bhagyashris|roverlooking12:43
rlandy|roverbhagyashris|rover: creating LP bug for that12:46
bhagyashris|roverack12:47
jbadiapaTengu, chandakumar: regarding this https://review.opendev.org/c/openstack/tripleo-ansible/+/810547 I could see several unreachabled on the ansible-tasks, but there was no log about them or at least I wasn't able to find them 12:51
rlandy|roverbhagyashris|rover: arxcruz|ruck: fyi ... https://bugs.launchpad.net/tripleo/+bug/194471912:51
Tengujbadiapa: yeah - as said, I'm a bit lost and stuck :/. Is there a way to get some more logs ?12:52
arxcruz|ruckrlandy|rover: train should search for promotion-testing-hash-train.yml not master, i'll check it 12:53
jbadiapaTengu, let me check12:56
rlandy|roverarxcruz|ruck: ^^ collect logs12:56
arxcruz|ruckrlandy|rover: ?12:59
*** amoralej|lunch is now known as amoralej12:59
rlandy|roverarxcruz|ruck: the error is before that ...12:59
rlandy|roverthat is collect logs getting master13:00
bhagyashris|roverrlandy|rover, chandankumar soniya29 planning meeting time13:00
bhagyashris|roverarxcruz|ruck, ^13:00
weshay|ruckmarios, here ya go :) https://www.youtube.com/watch?v=Ay1J2TxZDNM13:03
mariosweshay|ruck: thanks :)13:04
weshay|ruckarxcruz|ruck, bhagyashris|rover frenzy_friday fyi.. scenario-10 standalone passed deploy but failed in tempest. https://b8a2a7cef1d4bf7fa218-26901191b574c0e7e1d178b146c22a89.ssl.cf2.rackcdn.com/810474/2/check/tripleo-ci-centos-8-scenario010-standalone/95e8cb1/logs/undercloud/var/log/tempest/stestr_results.html13:12
* weshay|ruck writes it up13:12
weshay|ruckand skips13:12
frenzy_fridayweshay|ruck, should I remove it from criteria file?13:13
weshay|ruckfrenzy_friday, no.. we're trying to get it to voting13:14
frenzy_fridayoh ok, upstream. Got it13:15
weshay|ruckand we should.. rlandy|rover ^ perhaps this sprint.. for master13:15
*** holser is now known as holser_13:21
weshay|ruckfrenzy_friday, bhagyashris|rover arxcruz|ruck https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/81066713:21
arxcruz|ruckweshay|ruck: skipping these tests means nothing are being actually tested in scen10 13:22
arxcruz|ruckall octavia tests will be skipped13:23
arxcruz|ruckrlandy|rover: weshay|ruck ysandeep is this correct? https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/multinode-jobs.yaml#l2213:29
arxcruz|ruckfor train the release is set to 'master' 13:29
ysandeepwhat? i don't think so13:30
weshay|ruckarxcruz|ruck, fyi.. I'm not sure how often master is failing on https://bugs.launchpad.net/tripleo/+bug/1895248 yet.. only one job has executed since the breaking patch was fixed13:34
weshay|rucks/fixed/reverted13:34
pojadhavweshay|ruck, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/35742 pls review when you free.. thanks.!13:44
weshay|ruckpojadhav++13:45
chandankumarrlandy|rover: soniya29 arxcruz|ruck fyi https://review.opendev.org/q/topic:%2522workers%2522+(status:open+OR+status:merged)+owner:bdobreli%2540redhat.com13:54
arxcruz|ruckchandankumar: sorry, what's about?13:55
chandankumararxcruz|ruck: it is a big changes to tempest concurrency13:55
chandankumarit might break jobs in integration line so passing it to your attension13:56
arxcruz|ruckchandankumar: it will break for sure 13:56
rlandy|roverarxcruz|ruck: will look at train c7 after meeting13:56
rlandy|roverbuild test package failures13:57
arxcruz|ruckrlandy|rover: i found the problem 13:57
rlandy|roverarxcruz|ruck++13:57
arxcruz|ruckrlandy|rover: https://review.rdoproject.org/r/c/rdo-jobs/+/3575113:57
arxcruz|ruckrlandy|rover: basically, the job is set with release: master13:58
arxcruz|ruckinstead of release: train 13:58
rlandy|roverarxcruz|ruck: cool adding to bhagyashris|rover's tetsproject14:11
rlandy|roverbhagyashris|rover++ on closes-bug14:11
rlandy|roverhttps://review.rdoproject.org/r/c/testproject/+/3213714:12
rlandy|roverrekicked with depends-on14:12
chandankumarysandeep: soniya29 dviroel regarding fips work last week I added this https://review.rdoproject.org/r/c/openstack/tempest-distgit/+/35709 to disable tempest doc building an get the job working, we need to revert this patch and also update the sphinx version for tempest to enable doc building back, please include it in the tasks where fips works is tracked.14:13
soniya29chandankumar, sure14:14
ysandeepchandankumar, Ack, I will add this task once bhagyashris|rover add that story14:14
ysandeepbhagyashris|rover: If you want help with board, we can work tomorrow to fix the board.14:15
rlandy|roverbhagyashris|rover: fixed your comment on https://review.rdoproject.org/r/c/rdo-jobs/+/3575114:15
rlandy|rovercan you revote?14:15
bhagyashris|roverrlandy|rover, done14:20
bhagyashris|roverysandeep, sure will meet tomorrow 14:20
dviroelrlandy|rover, ysandeep chandankumar last fips updates on standalone failing jobs:14:24
dviroel- scenario002: https://review.rdoproject.org/r/c/testproject/+/35121/23#message-b3d9979633f4c0288f0ea71d63a76fce3a17aaf3 - didn't fail to collect all logs, we were able to identify that the issue occurs while attaching a volume to an instance, which uses a fips unsupported algorithm  - ade_lee is aware of this (barbican tempest test)14:24
dviroel- scenario004: we see swift tests failures there, which are beind debugged by cshwede on a local env, no more updates on that, I will ping him again for some updates14:24
rlandy|roverdviroel: thanks - pls add this info to a card on the new sprint board14:29
rlandy|roverwe need one for fips and one for rbac14:29
ysandeepdviroel: thanks for info!14:32
ysandeepdviroel: Its nearly eod for bhagyashris|rover and me, but we will create a story tomorrow on board for rbac/fips so that you can create those details there.14:34
dviroelysandeep: sure np, thanks14:36
ade_leerlandy|rover, dviroel thanks - I'm actually taking right now about the issues in the barbican tempest test14:38
dviroelgreat14:38
arxcruz|ruckrlandy|rover: fs39 passes 14:41
arxcruz|ruckhttps://review.rdoproject.org/r/c/testproject/+/3574714:41
rlandy|roverarxcruz|ruck: cool14:41
* rlandy|rover looking at c7 job14:41
arxcruz|ruckrlandy|rover: bhagyashris|rover update the patch and zuul marked as failed, i'm rechecking now 14:42
rlandy|roverarxcruz|ruck:     override-checkout: "stable/train"14:42
rlandy|rover    vars:14:42
rlandy|rover      branch_override: "stable/train"14:42
rlandy|rover      release: train14:42
arxcruz|ruckjrl?14:43
arxcruz|ruckrlandy|rover: ? 14:43
rlandy|roverarxcruz|ruck: https://meet.google.com/upz-mopy-cpo?pli=1&authuser=014:45
* frenzy_friday will be back in ~1hr14:48
rlandy|roverhttps://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-jobs-c8-train.yaml#L4614:49
ysandeeprlandy|rover: review plz.. https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/276334 temporary patch till we fix that job14:53
rlandy|roverdone14:56
* dviroel manila-meeting + lunch15:01
*** amoralej is now known as amoralej|off15:04
rlandy|roverarxcruz|ruck: merging https://review.rdoproject.org/r/c/rdo-jobs/+/35751 - thanks16:06
*** marios is now known as marios|out16:06
chandankumarsee ya people tomorrow :-)16:06
*** ysandeep is now known as ysandeep|dinner16:06
rlandy|roverbhagyashris|rover: arxcruz|ruck: pls ping when you are EoD - thanks16:06
*** rlandy|rover is now known as rlandy|ruck16:24
*** jpena is now known as jpena|off16:33
arxcruz|ruckrlandy|ruck: sorry, i'm at my end of the day already :) 16:51
rlandy|ruckarxcruz|ruck: k - anything I should pick up?16:51
rlandy|ruckyou working on anything?16:52
arxcruz|ruckrlandy|ruck: nope 16:52
rlandy|ruckk - watching main16:52
rlandy|ruckmaster16:52
arxcruz|ruckcentos 7 train is already executing tempest, even if it fails, the patch on rdo fix it so we are good on that at least 16:54
*** ysandeep|dinner is now known as ysandeep17:09
rlandy|ruckfrenzy_friday: pls add your PTO to the rhos calendar18:13
ysandeepakahat: left a suggestions on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/808748/8#message-e011e241a8abb0e10e02d5a998215c69d3c4f2ed , I think we need to give more context to reviewers. 18:31
erbotRequired files not ready after 302.2806086540222s for distgit-cbs-validate-centos8 35755,1,7d904690ea82401fb7bbf35c264b812718:56
ysandeeprlandy|ruck: interesting things happening in downstream bm jobs.. 19:23
ysandeeptempest ran well: https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/73/check/periodic-tripleo-ci-rhel-8-bm_envB-3ctlr_1comp-featureset001-baremetal-rhos-16.2/b500338/logs/undercloud/var/log/tempest/tempest_run.log 19:23
ysandeepbut still got timed out: https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/73/check/periodic-tripleo-ci-rhel-8-bm_envB-3ctlr_1comp-featureset001-baremetal-rhos-16.2/b500338/job-output.txt19:23
rlandy|ruckysandeep: yeah - chandan mentioned that19:24
* rlandy|ruck looks for review to increase timeout19:24
ysandeepwe already merged that.. increased to 6 hours19:25
ysandeepcannot increase more than 6 hours: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/276425 19:25
rlandy|ruckhttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/276294 merged19:25
rlandy|ruckyep setting from config19:25
rlandy|ruckwe can't go above that19:26
ysandeepThe job exceeds tenant max-job-  timeout 21600.19:26
ysandeepyup19:26
ysandeepBut I actually see better results now... earlier I was not seeing complete tempest run19:26
rlandy|ruckysandeep: so the job didn't timeout the run did19:26
rlandy|ruck2021-09-23 18:23:29.605210 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]19:27
rlandy|ruckysandeep: we could experiment and cut down on the tempest tests run there19:27
rlandy|ruckand see if we get by19:27
rlandy|ruckon the timeout19:27
rlandy|ruckand since bm sticks around19:27
rlandy|ruckwe could use a second job to do more comprehensive tempests tests19:28
rlandy|ruckwithout the deploy19:28
rlandy|ruckor something19:28
ysandeeptempest was pretty quick..  it only took ~15 mins.19:28
rlandy|ruckysandeep: need to take a call now19:29
* rlandy|ruck compares jobs in a bit19:29
ysandeepnw, I will figure something out19:29
ysandeepfyi.. did some analysis today: https://hackmd.io/m5HZvF8tQTmO5B74LEDiVg 19:30
rlandy|ruckwe can work on it tomorrow19:30
ysandeepyeah, not in criteria currently can work tomorrow 19:30
rlandy|ruckysandeep: may be easier if we debug together19:30
rlandy|ruckbook some time19:30
ysandeepack o/19:31
* ysandeep rechecks the job and disappears for the day, hoping with increased time out.. things might start passing intermittently.19:32
*** ysandeep is now known as ysandeep|out19:33
*** dviroel is now known as dviroel|out20:56
frenzy_fridayrlandy|ruck,  ack, done21:18
rlandy|ruckmaster is promoting21:32

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!