Monday, 2022-08-22

*** soniya29 is now known as soniya29|ruck03:39
*** soniya29 is now known as soniya29|ruck04:29
*** ysandeep|PTO is now known as ysandeep04:38
*** soniya29|ruck is now known as soniya29|ruck|afk05:06
* soniya29|ruck|afk will be back by an hour05:06
*** ysandeep is now known as ysandeep|afk05:18
akahatGood Morning!!05:21
akahato/05:21
*** soniya is now known as soniya29|ruck05:57
soniya29|ruckakahat, good morning05:57
jm1good morning folks :)06:49
chandankumarjm1: good morning & welcome back sir :-)07:08
chandankumarHow was the vacation ?07:09
jm1chandankumar: Great! Spent a week in north of germany near the sea, entertaining my nephew 😎07:16
jm1chandankumar: i hope ci wasn't burning too much last week?07:18
chandankumarjm1: it was fine, our awesome ruck/rover had taken care of that . :-)07:29
*** soniya29|ruck is now known as soniya29|ruck|lunch07:31
*** jpena|off is now known as jpena07:36
pojadhavfolks, needs +W https://review.opendev.org/c/openstack/tripleo-ci/+/851730 please07:59
pojadhavthank you chandankumar :)08:11
*** soniya29|ruck|lunch is now known as soniya29|ruck08:38
*** ysandeep|afk is now known as ysandeep08:55
*** soniya is now known as soniya|ruck09:08
* soniya|ruck facing power failures09:10
*** soniya|ruck is now known as soniya|ruck|afk09:24
* soniya|ruck|afk will be back in half hour09:24
chandankumarysandeep: hello09:51
ysandeepchandankumar, o/09:51
chandankumarysandeep: please have a look at this https://review.rdoproject.org/r/c/rdo-jobs/+/44540 and https://review.rdoproject.org/r/c/config/+/44455 when you get time.09:51
ysandeepchandankumar, ack09:51
*** soniya|ruck|afk is now known as soniya|ruck10:18
*** rlandy__ is now known as rlandy10:22
rlandyjm1: welcome back10:23
rlandyysandeep: hey - you around?10:23
rlandycan we chat for a few?10:23
ysandeeprlandy, yes.. need two mins to fetch my earphones10:24
rlandyfrenzyfriday: hi ... re: 17.a page on dashboard - did we merge that?10:24
rlandy17.110:24
rlandyfrenzyfriday: ^^ will need to touch base with you after that10:24
rlandysoniya|ruck: hi - pls ping if you need help10:24
rlandyysandeep: https://meet.google.com/ojx-iivx-gvw?pli=1&authuser=010:25
frenzyfridayrlandy, no, I think pojadhav was working on the dashboard patch10:33
soniya|ruckrlandy, sure10:35
pojadhavfrenzyfriday, rlandy : I haven't started working on it, just waiting for other cards related mini integration line to finish. If everything is done, will ready dashaboard page for 17.1 in 1-2 days.10:36
rlandyfrenzyfriday: pojadhav: can we chat now?10:48
frenzyfridaysure10:48
pojadhavrlandy, sure10:48
rlandyfrenzyfriday: pojadhav: https://meet.google.com/dtp-snib-kqd?pli=1&authuser=010:49
*** soniya is now known as soniya|ruck11:11
rlandysoniya|ruck: let;s sync when dviroel|out is in11:13
rlandyafter review time11:14
soniya|ruckrlandy, sure11:15
rlandyysandeep: can you look at comment on https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43307/?11:17
chandankumarrlandy: https://review.rdoproject.org/r/c/testproject/+/44151/23/.zuul.yaml11:21
chandankumarrlandy: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/defaults/main.yml#L5811:22
*** dviroel|out is now known as dviroel|rover11:28
dviroel|roverrlandy: soniya|ruck o/11:28
soniya|ruckdviroel|rover, hello11:28
dviroel|roversoniya|ruck: looking hackmd, 16-2 and 17 rhel 9 promoted :)11:36
soniya|ruckdviroel|rover, 16-2 promoted today itself :)11:37
chandankumarrlandy: https://review.rdoproject.org/r/c/config/+/4445511:38
rlandyjpodivin: hi - will you be at the cix call to discuss https://bugs.launchpad.net/tripleo/+bug/1987092?11:48
jpodivinrlandy: yep, I will. 11:49
jpodivinTakashi found some more info on it.11:51
ysandeepTengu: When you get a chance could you please cross check this looks okay now(Auditd): https://review.opendev.org/c/openstack/tripleo-quickstart/+/852594/4#message-a4f925618d4eabaec64298699c0abc93ac11a418 , If yes add your vote on patch and the depends-on patch.11:54
Tenguchecking11:55
* Tengu loves how the module pushes the rule name, and how things are properly ordered according to the index11:56
Tenguthis is already a good point11:56
* Tengu throwing flowers to himself11:56
Tenguysandeep: you got my +2 on both :11:57
Tengu:)11:57
Tenguthat's cool, and as said, we should soon be able to run on train and wallaby!11:58
ysandeepTengu++ great, thanks!11:58
ysandeepreviewbot, please add in review list: https://review.opendev.org/c/openstack/tripleo-quickstart/+/852594/ and https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85273412:00
reviewbotI could not add the review to Review List12:00
reviewbotI could not add the review to Review List12:00
ysandeepreviewbot: please add in review list  https://review.opendev.org/c/openstack/tripleo-quickstart/+/852594/12:00
reviewbotI could not add the review to Review List12:00
ysandeepfrenzyfriday, ^^ what's wrong with reviewbot 12:01
frenzyfridayysandeep, checking12:02
frenzyfridayadd to review list  https://review.opendev.org/c/openstack/tripleo-quickstart/+/852594/12:04
reviewbotI could not add the review to Review List12:04
frenzyfridayadd to review list  https://review.opendev.org/c/openstack/tripleo-quickstart/+/123456789012:04
reviewbotI could not add the review to Review List12:04
frenzyfridayhm.. /me checking12:04
ysandeeprlandy, https://trello.com/c/R6SuOv6E/2661-cixlp1983718tripleociproa-periodic-master-scen1-standalone-fails-timeout-manage-firewall-rules - This bug/cix will be closed once nftables will be defaults.12:04
dviroel|roversoniya|ruck: not sure if you have this useful link: https://softwarefactory-project.io/weeder/tenant/rdoproject.org/info12:05
dviroel|roverrlandy: ^12:05
ysandeepdviroel|rover, I am also seeing this url for the first time.12:06
ysandeepdviroel++ thanks for sharing12:06
soniya|ruckdviroel|rover, its great :)12:06
ysandeepTengu, https://logserver.rdoproject.org/54/31954/72/check/periodic-tripleo-ci-centos-9-scenario003-standalone-master/5ff2054/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz 12:13
ysandeep2022-08-22 07:35:04.401684 | fa163eab-abeb-72fc-ce1a-000000000999 |      FATAL | Generate nft ruleset in static file | standalone | error={"changed": false, "msg": "AnsibleUndefinedVariable: '_dport' is undefined"}12:13
Tenguuho?12:13
Tenguo____O12:14
Tenguhow..... I don't get how it can be undefined.12:15
Tenguoh. damned. scoping.12:16
Tengufsking jinja12:16
*** ysandeep is now known as ysandeep|afk12:22
* pojadhav stepping out for hr.. we will back till scrum.12:23
jpodivindviroel|rover: I've noticed that some redundancies have sneaked their way into our component pipeline. Nothing egregious, but it is confusing enough. https://review.rdoproject.org/r/c/rdo-jobs/+/4460512:33
jpodivinI'm planning to follow up with https://review.rdoproject.org/r/c/rdo-jobs/+/4460812:33
dviroel|roverjpodivin: ack, I will take a look12:35
akahatfolks please take a look when you are free, thanks. https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4459412:44
frenzyfridayysandeep|afk, looks like the bot is still adding the reviews but something has changed in hackmd's api response. Checking in some time12:45
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.12:45
rlandychandankumar: hey - could you pls send the link again for the ibm console ticket? ty13:07
chandankumarrlandy: https://issues.redhat.com/browse/RHOSZUUL-102913:08
chandankumarI have  added one more today https://issues.redhat.com/browse/RHOSZUUL-103313:08
chandankumarwe need both13:08
rlandyok - adding to prod chain13:09
chandankumarDo you want me to create issues for ovb jobs limit13:10
chandankumaralso?13:10
rlandychandankumar: ack if you can 13:12
rlandyI added the item13:12
rlandybut a tracking jira would help13:12
rlandyall: ^^ anything else for the prod chain?13:12
dviroel|roverchandankumar: hey, on friday/saturday i was trying to run a ovb job on IBM, with testproject, and it was failing on ovb stack 13:17
dviroel|roverchandankumar: but I didn't investigate the real issue13:17
dviroel|roverchandankumar: i was wondering if those cleanup scripts also run there13:17
chandankumardviroel|rover: job project link13:17
chandankumardviroel|rover: yes we enabled cleanup script ob ibm cloud13:17
chandankumarvia toolbox13:18
dviroel|roverchandankumar: https://review.rdoproject.org/r/c/testproject/+/3702913:18
dviroel|roverchandankumar: ack, nice :)13:18
rcastillo|rovero/13:22
chandankumarrlandy: here you go https://issues.redhat.com/browse/RHOSZUUL-103413:23
chandankumardviroel|rover: tp failed due to Message: No valid host was found. , Code: 500" |13:24
chandankumarknown issue13:24
chandankumarresource limit13:24
chandankumarduring stack creation13:25
dviroel|roverchandankumar: yeah, thats why i was asking about cleanups, i was considering some orphan vms, but maybe there was too many jobs running at that time13:28
chandankumaryup13:38
pojadhavscrum in 6 mins : arxcruz, rlandy, marios, ysandeep, bhagyashris, soniya|ruck , soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj,  dviroel|rover , rcastillo, dasm, jm113:55
pojadhavrlandy, jm1 chandankumar 14:01
pojadhavscrum time14:01
*** ysandeep|afk is now known as ysandeep14:01
pojadhavfrenzyfriday, ^^14:02
*** dasm|off is now known as dasm14:02
dasmo/14:02
rlandywill be there is a sec14:02
rlandydasm: hey - chandankumar added example test to your review14:07
dasmthanks rlandy. i saw something. i'm gonna check this one, thanks chandankumar++14:09
rlandydviroel|rover: let's chat about the 'disk full' issue with bm this afternoon - maybe we can get a biger nodepool node14:19
frenzyfridaychandankumar, dasm, rlandy : the mail is about the opensearch instance (https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global) and about creating a dashboard there. It is not related to ER - ER just consumes the api of opensearch14:25
dviroel|roverrlandy: ack14:26
dasmfrenzyfriday++14:27
ysandeep#folks didn't we agreed to move Monday's scrum half an hour earlier?14:40
ysandeepto avoid clash with cix14:41
jm1ysandeep: i was wondering the same14:44
* pojadhav dinner brb14:45
ysandeepjm1, I think pojadhav was out that day.. pojadhav ^^ we agreed to move - Monday's scrum half an hour earlier to avoid clash with cix14:46
rlandyysandeep: jm1: ack .. I remember that :)14:48
rlandypojadhav: ^^ 14:48
chandankumarrlandy: dviroel|rover ysandeep https://review.opendev.org/c/openstack/tripleo-ci/+/849856 is good to go14:54
ysandeepchandankumar, checking14:57
rlandylooking14:57
* dviroel|rover lunch14:58
rlandyvoted14:58
*** dviroel|rover is now known as dviroel|rover|lunch14:58
* soniya|ruck leaving for the day15:04
soniya|ruckrlandy, dviroel|rover|lunch, see you tomorrow15:04
rlandypojadhav: hey - can you add the template patch before your EoD? thank you15:05
*** soniya|ruck is now known as soniya|out15:05
chandankumarsee ya!15:06
ysandeeprlandy, dviroel|rover|lunch  I will start working on Confluence page in my morning. fyi.. I have asked the "Operator build workflow in cpass" query in podified slack channel also.. let's see..15:07
rlandyysandeep: k- thank you - otherwise I can ping jon in my afternoon15:08
rlandygot disconnected from slack15:08
ysandeepthanks, I am chatting with jon atm on slack15:11
*** ysandeep is now known as ysandeep|dinner15:13
pojadhavrlandy, ysandeep|dinner : thanks for discussion, I will change time for monday's scrum.15:23
pojadhavrlandy, this one ? https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/424287/1/zuul.d/component-pipeline-rhos-17.1-rhel8.yaml to add component jobs inside this components ?15:26
pojadhavrlandy, sorry got distracted due to dashboard work.. could you please once clarify about work 17.1 rhel8 one.15:35
* pojadhav brb16:00
frenzyfridayrlandy, do we need any update on this patch? https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/424293  16:06
frenzyfridayI have rechecked its testproj16:06
rlandychecking16:07
rlandyfrenzyfriday: ack registry_namespace16:09
rlandyhttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/424293/10/zuul.d/rhos17.1-rhel8-jobs.yaml16:09
rlandyline 6216:09
rlandyfrenzyfriday: also missing multinode jobs16:09
rlandyand ipa16:09
frenzyfridayack, updating the namespace16:09
pojadhavrlandy, I also need clarification about 17.1 work ? will do before my EOD16:11
*** dviroel|rover|lunch is now known as dviroel|rover16:12
rlandypojadhav: looking16:13
rlandypojadhav: you just need to add component-pipeline-rhos-17.1-rhel8 to https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-config.git;a=blob;f=zuul.d/tripleo.yaml;h=171c3b447efa3ed82d38b28316e8626cfba5ef8c;hb=HEAD16:15
rlandyfor integration16:15
pojadhavrlandy, yep doing it now16:15
rlandythe same16:16
rlandywhen it is merge16:16
rlandypojadhav: does that answer your question?16:16
pojadhavrlandy, yes16:16
pojadhavthank you16:16
dviroel|rovergah, i wasn't expecting periodic-tripleo-ci-centos-8-9-multinode-mixed-os on wallaby c9 :P16:27
dviroel|roverrunning, missing this one ^ only for wallaby c916:27
pojadhavrlandy, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/42561816:30
rlandypojadhav, thanks merging16:32
rlandylunch16:32
rlandybrb16:32
dviroel|roverah, found the issue16:34
dviroel|roverperiodic-tripleo-ci-centos-9-8-multinode-mixed-os  x periodic-tripleo-ci-centos-8-9-multinode-mixed-os16:35
dviroel|roverW+1 job rename on criteria16:37
*** jpena is now known as jpena|off16:38
pojadhavrlandy, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/42561916:41
*** ysandeep|dinner is now known as ysandeep|out16:57
dasmbrb17:05
dviroel|rovernew bug for FIPS: https://bugs.launchpad.net/tripleo/+bug/198732317:09
dviroel|rovernot a blocker17:09
frenzyfridayupdated https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/42429317:11
dasmback17:46
rlandypojadhav: waiting on the definition to be added here  before merging https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/425619/2/zuul.d/tripleo.yaml 17:49
pojadhavrlandy, added depends-on this https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/42429317:50
pojadhavon above patch17:50
rlandyrebasing17:50
rlandyack - thanks17:50
rlandydviroel|rover: you all set with the mixed rhel job?17:57
rlandythere was a name change17:57
rlandydviroel|rover: periodic-tripleo-ci-centos-9-8-multinode-mixed-os in c917:58
rlandynot 8-917:58
rlandychecking criteria17:58
rlandyperiodic-tripleo-ci-centos-9-8-multinode-mixed-os - correct in criteria18:00
rlandymissing 064 and 03918:00
dviroel|roverrlandy: yes, promoting now18:00
rlandycool18:01
dviroel|roverrlandy: merged criteria patch18:01
rlandydviroel|rover: I'll work on the disk full issue18:01
dviroel|roveri was chasing a hash from saturday18:01
* rlandy checks if can get bigger node18:01
dviroel|roverc9-wallaby promoted18:04
rlandynice!18:06
dviroel|roverrlandy: i have a log from a successful run, whith DISK_FULL error18:06
dviroel|roverfrom friday18:06
dviroel|roverone sec18:06
pojadhavrlandy, jfyi- WIP patch is up for osp17.1 dashboard. structure created with all panels updated. tomorrow will check to pull out promotion data into those panels. https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/42556318:21
rlandypojadhav++ thanks18:21
pojadhavrlandy, tommorrow I will start my day lil late due to some imp work.18:21
pojadhavgood night18:22
rlandypojadhav: pls get ananaya to review tomorrow morning18:22
rlandygood night18:22
pojadhavrlandy, sure18:22
*** rcastillo|rover is now known as rcastillo18:22
* pojadhav out18:23
*** pojadhav is now known as pojadhav|out18:23
dviroel|roveri think that we should consider moving stable4 to start at another time18:29
dviroel|roverit starts 2 hours after main pipeline, and they have so many jobs18:30
rlandydviroel|rover: if you can find a better time, you can propose a patch18:30
dviroel|roveryeah, i am looking at that18:30
dviroel|roverearlier in the day we have components only18:31
dviroel|roverno stable2 at 4AM anymore18:31
dviroel|roverbut if we plan to move stable4 to IBM, we don't need to worry about 18:34
rlandydviroel|rover: idk the timeline on that18:35
rlandyso move it if you feel it's better18:35
dviroel|roverack18:36
rlandypojadhav|out: left a comment on the dashboard review - thanks18:43
rlandydviroel|rover: for 17.1 multinode was just added18:44
rlandyipa passed multinode failed18:44
rlandygoing to rerun to see if there is a real issue18:44
dviroel|roverok, i am chasing this hash 49d693aeaed295536478ed83681ea9d418:46
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-containers-multinode-rhos-17.1&skip=018:47
rlandynever passed18:47
rlandycharming18:47
rlandyntp18:47
rlandymight be missing a setting18:48
dviroel|roverwallaby c8 promoted19:10
rlandygreat19:13
dviroel|rovertrain c8 promoting19:15
rlandyfixing multinode19:36
dviroel|rover17-1 promoted19:36
rlandydviroel|rover: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/425632 fyi19:47
dviroel|roverrlandy: ack19:50
rlandytest in rerun19:53
rlandydviroel|rover: want me to merge https://review.rdoproject.org/r/c/config/+/44617?20:26
dviroel|roverrlandy: yes, if you agree with it20:27
rlandylet's try it20:28
dasmdviroel|rover: have you seen NODE_FAILURES? It seems to be some kind of issue in recent hours20:52
dasms/hours/hour20:52
dviroel|roveron vexx?20:52
dasmyes20:53
dasmsome time ago i got 404 when being logged in to the vexx20:53
dviroel|roveryeah, we have an ticket open due to mirror/dns issues - but this is new20:54
dasmhmm... 20:55
dviroel|roverguilhermesp from vexx was working on a fix for their cloud20:55
dviroel|rovermaybe this is a side effect20:55
dasmmaybe. idk20:56
rlandydasm: <guilhermesp___> yeah we are implement our internal resolvers for now, this will improve caching as well20:56
dasmack20:56
rlandylet's see if it gets better or worse20:56
dasm++20:56
dviroel|roverhttps://review.rdoproject.org/zuul/builds?result=NODE_FAILURE&skip=020:57
dviroel|roverpromoter failed to promote train c8, lets see how it goes in next run20:59
rlandydviroel|rover: ok of we merge https://review.opendev.org/c/openstack/tripleo-ci/+/849856?21:14
rlandyfor chandankumar 21:14
* dviroel|rover looking21:15
dviroel|roverrlandy: voted, but didn't w+1, it is hard to know if will break something or note21:26
dviroel|roverwe might need to merge and check21:27
rlandylet's try21:29
rlandythings are in ok shape atm21:29
rlandychandankumar: ^^ pls check in your morning21:30
dviroel|roverrlandy: ok, going out for today, we are missing master and a few wallaby components, train should promote (if promoter don't fail again)21:48
dviroel|roveronly one fips bug today, which is not a blocker21:49
* dasm => offline21:51
dasmtake care team!21:51
*** dasm is now known as dasm|off21:51
*** dviroel|rover is now known as dviroel|out21:53
dviroel|outo/21:53
* rcastillo out for today22:26
rlandybye22:32
rlandysee you all tomorrow22:32
*** rlandy is now known as rlandy|out22:32

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!