Tuesday, 2022-07-26

*** ysandeep|out is now known as ysandeep01:55
*** ysandeep is now known as ysandeep|afk03:21
*** Guest6022 is now known as rcastillo04:22
*** ysandeep|afk is now known as ysandeep05:03
*** amoralej|off is now known as amoralej06:18
*** jgilaber|lunch is now known as jgilaber07:04
jm1o/07:20
*** jpena|off is now known as jpena07:34
*** jpena is now known as jpena|off07:40
*** jpena|off is now known as jpena07:42
*** ysandeep is now known as ysandeep|afk08:02
*** pojadhav is now known as pojadhav|lunch08:06
*** pojadhav|lunch is now known as pojadhav08:19
akahat|ruckarxcruz|rover, looks like mirrors are your enemy, they won't let you promote master today. :P https://review.rdoproject.org/zuul/status#4415009:03
*** tosky_ is now known as tosky09:10
mariostripleo ci please review when you have time https://review.opendev.org/c/openstack/tripleo-quickstart/+/850507 thank you09:32
*** rlandy|out is now known as rlandy10:34
rlandyarxcruz|rover: akahat|ruck: hello  - how are things?10:36
akahat|ruckrlandy, o/ 10:37
arxcruz|roverrlandy except for master, everything seems fine10:37
rlandyakahat|ruck, 10:37
akahat|ruckrlandy, downstream is good.. nothing new 10:37
rlandyhow's 16.210:37
arxcruz|roverfeatureset35 is not good 10:37
rlandyarxcruz|rover: tempest or deploy?10:37
arxcruz|roverrlandy i checked the latest failures, it's failing randomly in several places 10:37
arxcruz|roverrlandy fail on deploy, fail on tempest, fail on tempestconf 10:37
rlandyarxcruz|rover: akahat|ruck: let's sync quickly10:38
arxcruz|roverok10:38
akahat|ruckrlandy, ok10:38
rlandyhttps://meet.google.com/nth-hesg-fju10:38
ykarelakahat|ruck, arxcruz|rover is the issue with phase1 known one?10:51
ykarelERROR! Unexpected Exception, this is probably a bug: Non integer values in LooseVersion ('stable/1.0.0')10:51
ykarelcaused by https://review.opendev.org/c/openstack/tripleo-quickstart/+/84962010:51
akahat|ruckykarel, oh.. it hit again.10:52
akahat|ruckykarel, okay. i'll put fix, it is hitting again and again.10:53
ykarelakahat|ruck, ack Thanks10:53
* marios_ powercut10:58
*** marios_ is now known as marios11:00
rlandyjm1: ^^ can you look?11:00
rlandyat what ykarel reported above11:00
rlandyor rcastillo when you are in11:01
arxcruz|roverrlandy akahat|ruck https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4416411:03
arxcruz|roverskip fs03511:03
rlandyarxcruz|rover: thanks - pls revert when merged11:04
arxcruz|roverrlandy the issue ykarel reported is already a known issue, akahat|ruck is fixing it, no need jm1, we did talk with him before, it's not related to ansible-collections 11:04
rlandyarxcruz|rover: thanks - sorry jm111:04
*** dviroel__ is now known as dviroel11:05
rlandybhagyashris: hello https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/41145811:09
rlandyneed rebase pls11:09
rlandybhagyashris: also - chandankumar picked up on the ceph true diff  - I think we should change that to match upstream11:10
rlandybhagyashris: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/402889 - OVB one merged - pls put the last of the mulinode ones on the review list11:11
rlandythe then you can get done here :)11:11
rlandyfrenzy_friday: ^^ OVB reparenting has merged - you should be able to try that for 17.111:12
frenzy_fridayrlandy, yep, using the 17 parent on 17.111:12
rlandyfrenzy_friday: great11:12
rlandyfrenzy_friday: also - sis we merge the patch to add the 17.1 line to zuul?11:13
rlandyif so, we may need a restart to get that line to show11:13
rlandyhttps://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-config.git;a=blob;f=zuul.d/tripleo.yaml;h=2c6de1c65e5428d1454d4e68762e446220421db0;hb=HEAD11:14
rlandyno  17.1 here11:14
jm1ykarel, akahat|ruck, arxcruz|rover: could you please send me a link to a log file with this "Non integer values in LooseVersion" issue?11:20
jm1ykarel: is this affecting downstream only or did it show up in our tripleo centos jobs as well?11:20
ykareljm1, https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/job/tripleo-quickstart-promote-master-centos9-current-tripleo-delorean-minimal/lastFailedBuild/consoleText11:21
rlandyfrenzy_friday: ok- found job - w+'ed it11:21
ykareljm1, /me only noticed in centos9 jobs, not checked downstream ones11:21
jm1ykarel: ack. it happened downstream as well, so actually both are affected11:22
ykarelohkk11:22
jm1akahat|ruck, arxcruz|rover: do you have a patch prepared? we definitely should add "-vvv" to see the full traceback11:23
akahat|ruckjm1, where it will impact if we change it to 1.0.0?11:23
jm1akahat|ruck: well, it wont work11:23
akahat|ruckjm1, why?11:24
jm1akahat|ruck: 1.0.0 is way too old i suppose. instead we should add "-vvv" first to see what the real culprit is. It cannot be "stable/1.0.0" because that works most of the time11:25
rlandypojadhav: https://meet.google.com/tgb-zjnf-ccc?pli=1&authuser=011:28
bhagyashrisrlandy, ack11:28
akahat|ruckjm1, instead of 1.0.0  can name 1.5.0 or something else?11:31
* dviroel coffee11:32
akahat|ruckjm1, checking if it breaks here: https://review.opendev.org/c/openstack/tripleo-quickstart/+/85102011:32
Tengufolks, you may face an error in CI (aka gate blocker and all) with https://bugs.launchpad.net/tripleo/+bug/1982744 - RCI is once again broken openstack-selinux that doesn't install the modules correctly. The "rpm -V" in tripleo_bootstrap is catching the error as expected (which is good). Bad news is: I'm on PTO this week :).11:33
Tengu*RCA11:34
Tengu(brain's actually deeply disconnected ^^')11:34
jm1akahat|ruck, ykarel, arxcruz|rover: please vote https://review.opendev.org/c/openstack/tripleo-quickstart/+/85102211:34
ykarelack11:37
jm1reviewbot: add review https://review.opendev.org/c/openstack/tripleo-quickstart/+/85102211:37
jm1frenzy_friday: reviewbot somehow does not like me 😭11:37
frenzy_fridayjm1, it looks for syntax "add to review list <patch>11:38
jm1reviewbot: add to review list https://review.opendev.org/c/openstack/tripleo-quickstart/+/85102211:38
reviewbotI have added your review to the Review list11:38
frenzy_fridayor need reviews ... <patch>11:38
jm1reviewbot: !help11:38
jm1frenzy_friday: ^ that would be a nice addition11:38
frenzy_fridayI had a syntax for help - lemme check11:39
frenzy_fridayoh, it is when you ping 1-1 hi to reviewbot :D11:39
jm1frenzy_friday: then i get "[Notice] -reviewbot- I would rather not11:41
frenzy_fridayXD checking , I added some random msg in case someone pings anything it did not understand11:42
frenzy_fridayjm1, What did you type exactly?11:43
frenzy_fridayit understands only hello = ["hi", "hey", "hola", "halo", "hallochen", "namaste", "hello"]11:43
rlandyysandeep|afk: dviroel: pls ping when around - need to touch base before wednesday's meeting11:48
frenzy_fridaybhagyashris, pojadhav hey, I updated the parenting for the 17.1 ovb jobs - https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/421405 11:51
pojadhavfrenzy_friday, ack11:51
jm1ykarel, akahat|ruck, arxcruz|rover: another patch which fixes the symptoms https://review.opendev.org/c/openstack/tripleo-quickstart/+/85102412:01
jm1frenzy_friday: "/msg reviewbot help" is what i sent12:01
arxcruz|roverjm1 is 1.8.0 the right version?12:02
frenzy_fridayjm1, oh, I forgot to add help. I added only hi. I'll put a patch for that12:02
dviroelrlandy: ack, lets wait for ysandeep|afk 12:02
akahat|ruckjm1, you need to create 1.8.0 branch in repo 12:03
jm1arxcruz|rover, akahat|ruck: 1.8.0 is a tag, not a branch12:04
akahat|ruckjm1, yeah.. sorry.. tag i mean.12:04
jm1akahat|ruck, arxcruz|rover: https://opendev.org/openstack/ansible-collections-openstack/src/tag/1.8.012:05
arxcruz|roverjm1 ok12:05
jm1akahat|ruck, arxcruz|rover: you can also check rdoinfo, link is in commit message12:06
akahat|ruckjm1, ack12:07
chandankumarakahat|ruck: arxcruz|rover please take care of this bug https://bugzilla.redhat.com/show_bug.cgi?id=2109495 if fs01 rhos-16,2 is passing12:09
akahat|ruckchandankumar, ok12:10
rlandyakahat|ruck: rhos-17 rhel-9 container build is fixed - rerunning ceph qual now12:12
rlandywill rekick the line to promote later12:12
akahat|ruckrlandy, ack12:13
akahat|ruckrlandy, we can close BZ; https://bugzilla.redhat.com/show_bug.cgi?id=2109495 once we get green run of periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-1712:20
akahat|rucktestproject; https://code.engineering.redhat.com/gerrit/c/testproject/+/300913/11/.zuul.yaml12:20
akahat|ruckfor 16-2 and 17 rhel 8 we got green run12:20
rlandyakahat|ruck: sure - thanks - pls go ahead when you are ready12:21
*** ysandeep|afk is now known as ysandeep12:30
ysandeeprlandy, dviroel we can chat after podified mtgs12:30
rlandyysandeep: community call12:32
rlandycan talk after that12:32
ysandeepack for after community call12:32
*** amoralej is now known as amoralej|lunch13:09
ysandeeprlandy, dviroel do you want to meet now?13:14
rlandyysandeep: dviroel: we can 13:14
ysandeeprlandy, dviroel meet.google.com/mpa-mncb-mni13:14
dviroelsure13:14
rlandyhttps://meet.google.com/kxo-dmsc-ywf13:15
rlandydviroel: ysandeep: ^^13:15
ysandeepjoining13:15
dasm|offo/13:19
*** dasm|off is now known as dasm13:19
chandankumarrlandy: is there some issue with rdo zuul?13:20
chandankumarno new jobs are running in check13:21
rlandychandankumar: yes - see rhos-ops13:21
rlandyrestart13:21
bhagyashrisarxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm, jm1, marios, akahat|ruck arxcruz|rover 13:28
bhagyashrisTripleo CI community meeting in 2 mins13:28
bhagyashrishttps://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2022-07-26-Community-Call13:28
chandankumarrlandy: thanks, now it's back!13:28
bhagyashrishttps://meet.google.com/igc-nxwj-gws?authuser=0&hs=12213:28
mariosrlandy: akahat|ruck: arxcruz|rover: fyi first run of mixed-os job 'in the wild' @ https://zuul.openstack.org/status#850929 (tripleo-ci patch merged)13:31
mariosit will run on stable/wallaby across most repos (wherever we have multinode job)13:32
akahat|ruck\o/13:32
jm1ykarel: added more info about the LooseVersion issue here: https://bugzilla.redhat.com/show_bug.cgi?id=210980713:35
jm1ykarel: where is job tripleo-quickstart-gate-master-tripleo-ci-delorean-full-featureset052 defined?13:37
jm1arxcruz|rover, akahat|ruck: fyi did some deeper analysis https://bugzilla.redhat.com/show_bug.cgi?id=2109807#c513:38
ykareljm1, Thanks, that job is defined in rdo-infra/ci-config13:38
jm1ykarel: huh? where?13:39
rcastilloit's a jenkins job13:39
jm1ykarel: we only fixed zuul jobs, so maybe jenkins jobs are still broken13:40
jm1rcastillo: thanks!13:40
ykarelhttps://github.com/rdo-infra/ci-config/blob/master/jenkins-v2/jobs/tripleo-quickstart-main.yml13:40
rcastillohttps://github.com/rdo-infra/ci-config/blob/master/jenkins-v2/jobs/tripleo-quickstart-templates.yml13:40
*** amoralej|lunch is now known as amoralej13:41
jm1ykarel, rcastillo: thanks! in which environment does those jobs run? https://github.com/rdo-infra/ci-config/blob/master/jenkins-v2/jobs/tripleo-quickstart-builders.yml13:46
jm1ykarel, rcastillo: i am trying to figure out what installs openstack.cloud13:46
rcastillojm1: https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/tripleo-quickstart-gate-master-tripleo-ci-delorean-full-featureset052/13:47
rcastillolast success 1 mo 19 days ago13:47
rcastilloci.centos is going away next month I believe13:48
jm1rcastillo: need some advice here, where can i find the nodeset (probably called different in jenkins) and base jobs which define what is done before this script in https://github.com/rdo-infra/ci-config/blob/master/jenkins-v2/jobs/tripleo-quickstart-builders.yml is run?13:49
jm1rcastillo: so ci.centos is not relevant anymore?13:49
rcastillonot really, I'm working on migrating some of these jobs to zuul13:49
ykareljm1, it runs on centos ci infra and runs on nodes provided by duffy13:58
ykarelhttps://wiki.centos.org/QaWiki/CI/Duffy13:58
ykarelthose jobs running on 9-stream nodes13:58
dasmi'm sorry for derailing "Design Week" discussion. Thanks for all participation :)14:03
dasmysandeep: you're right. UA != TC :p14:03
jm1ykarel: what is run before this script here? something installs openstack.cloud before because ansible-galaxy reports that openstack.cloud is already installed https://github.com/rdo-infra/ci-config/blob/master/jenkins-v2/jobs/tripleo-quickstart-builders.yml14:04
ykareljm1, i think all should be in logs14:12
ykareland all runs on a container14:12
ykarellet me try to find the link14:12
ykareljm1, https://quay.io/repository/rdoinfra/cico-workspace-rdo?tab=info14:14
ykareland iirc container is c8 based14:14
ykareldocker file at https://github.com/amoralej/centosci/14:15
ykarelamoralej, ^ is correct or it moved to some other place14:15
amoralejmmm14:16
amoraleji'm not sure tbh14:16
amoraleji need some time14:16
rlandydasm: good discussion and your view points are always valuable14:17
amoralejykarel, jm1 but what's the question, afaik we are getting rid of those jobs14:19
rlandyakahat|ruck: scenario10 is the only missing 16.2 job14:22
rlandyfailed twice  - debugging14:22
rlandylooks like same bug we logged before14:22
rlandyone more run - then will skip promo14:23
akahat|ruckrlandy, loadbalancer is in invalid state of ERROR14:25
akahat|ruckrlandy, https://bugzilla.redhat.com/show_bug.cgi?id=210540814:29
rlandyakahat|ruck: arxcruz|rover: at tomorrow's program call - we need to start reporting on the 17.1 line14:30
rlandyfrenzy_friday++ bhagyashris++14:30
rlandyrunning mini line now14:30
arxcruz|roverrlandy so, tempest was updated, and this patch https://github.com/openstack/tempest/commit/be58876debad775d59c72a99daba86d61816bed1 get this test to be executed, which was being skipped before14:30
rlandybhagyashris: pls check if we get a prom14:30
arxcruz|roverthsi test is failing on us14:31
arxcruz|rovershall i open a bug, add on skiplist ?14:31
rlandyarxcruz|rover: pls do both14:31
arxcruz|roverok14:31
* akahat|ruck back in hour 14:35
jm1amoralej, ykarel: if we are getting rid of those jobs anyway, then it is probably not worth to investigate. our tripleo centos jobs are running fine and downstream is failing randomly (and very rarely).14:36
jm1amoralej: fyi this is the issue we are trying to investigate. but as written above it is happening very rarely https://bugzilla.redhat.com/show_bug.cgi?id=210980714:37
jm1ykarel: is tripleo-quickstart-gate-master-tripleo-ci-delorean-full-featureset052 failing consistently with this error? because we saw this error only once in a rhos job and all later jobs passed14:38
rlandyakahat|ruck: ack - we've seen this one before14:39
amoralejweird14:39
ykareljm1, afaik it's consistent14:42
jm1ykarel: hmm.. then due to time pressure we should probably go for https://review.opendev.org/c/openstack/tripleo-quickstart/+/85102414:44
jm1ykarel: although it merely fixes symptoms14:45
ykarelsounds good to me14:45
jm1ykarel: added it to our review list14:46
jm1bhagyashris, marios, ysandeep: do you also have two tripleo ci community meetings in your gcal?14:47
jm1rhos team meetings calendar has two entries?!?!14:48
jm1arxcruz|rover++ thanks for pinging iury and getting him into the call! :)14:50
ykarelThanks jm1 14:51
* dviroel lunch14:55
*** dviroel is now known as dviroel|lunch14:55
chandankumarsee ya!14:59
*** ysandeep is now known as ysandeep|out15:00
bhagyashrisjm1, yes i see two entries15:02
bhagyashrisrlandy, ack15:02
mariosyeah i do jm1 15:21
mariosthey are also duplicated in the rhos team meeting calendar not just mine15:21
mariosrlandy: please add to your review list https://review.rdoproject.org/r/c/config/+/44066/2#message-2962b1a485ba79c668913b5a1aaee035d15608a2 15:28
reviewbotI have added your review to the Review list15:28
mariosrlandy: if all good i will workflow in the morning 15:29
rlandymarios: will do - all ok with the opendev tests?15:30
rlandyarxcruz|rover: hey - need your help pls on a tempest conf issue15:31
arxcruz|roverrlandy yes?15:32
rlandynew ceph - posting in pvt15:32
arxcruz|roverok15:32
mariosrlandy: still watching it but holding the workflow as commented in that review 15:36
mariosrlandy: waiting for at least one green run 15:36
rlandymarios++ good start15:37
mariosrlandy: yeah if all goes well can start testing the rdo side tomorrow otherwise no biggie will fix whatever it is and carry on ;)15:37
rlandyarxcruz|rover++15:55
*** dviroel|lunch is now known as dviroel16:01
mariosjm1: ysandeep|out: please check when you next have time https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/851046 (from your review comments )16:12
rlandyakahat|ruck: going to have to skip promo 16.216:13
*** marios is now known as marios|out16:16
*** amoralej is now known as amoralej|off16:16
rlandyarxcruz|rover: akahat|ruck: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/421573 Temp skip scenario010 test in 16.2 criteria16:18
rlandywill revert16:18
akahat|ruckrlandy, voted.16:21
rlandyty16:25
*** jpena is now known as jpena|off16:33
rlandylunch - brb16:36
rlandyakahat|ruck: hey - still around?17:00
akahat|ruckrlandy, yes17:00
rlandycan you check the internal promoter17:00
rlandyit shou,d be promoting 17.1 now17:00
rlandyconfig added17:00
rlandyI think it needs to be added to the script17:00
rlandythanks'=17:01
akahat|ruckrlandy, do you have ip handy for internal promoter?17:01
rlandyhttps://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=blob;f=ci-scripts/dlrnapi_promoter/config_environments/rdo/RedHat-9/rhos-17.1.yaml;h=d1263b876fba1f5409eb70c016d13f880059abe5;hb=HEAD exists17:01
rlandysent17:01
dviroelrlandy: rcastillo dasm: team, i will miss the review time today, need to go offine in a few 18:05
dasmdviroel: ack18:06
* dviroel out18:19
rlandyok18:19
dasmstepping away for a bit18:45
arxcruz|roverrlandy https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/85107918:48
arxcruz|roveronce merged i'll rerun tempest to promote 18:48
rlandyarxcruz|rover: thanks - votde18:49
rlandyarxcruz|rover: late for you 18:49
rlandyanything you want me to do?18:49
arxcruz|roverrlandy all good, want to finish this 18:50
rlandyarxcruz|rover++ really great job this time on rr18:50
rlandyI did nothing18:50
arxcruz|roverrlandy chandam and dasm make my job this time easy18:50
arxcruz|roverhopefully next rr will also be easy 18:50
rlandydasm: joining review time?19:01
arxcruz|roverrlandy 19:09
arxcruz|roverrlandy https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4410319:09
arxcruz|rovermaster was promoted, the revert patch 19:09
rlandyarxcruz|rover: w+'ed19:10
dasmback19:15
dasmarxcruz|rover: it's good to know you're having nice time :)19:20
jm1ykarel, amoralej|off, akahat|ruck, arxcruz|rover, rcastillo: found the reason why jenkins jobs fail https://bugzilla.redhat.com/show_bug.cgi?id=2109807#c619:39
jm1ykarel, amoralej|off, rcastillo: is running quickstart.sh twice really what we want? is this even supported?19:40
rcastillojm1: it should be if we're passing in different playbooks iiuc19:51
rcastillowe can change the collect logs script to run the playbook directly if running quickstart.sh twice is causing issues19:51
rcastillobrb in 20 minutes19:52
jm1rcastillo: maybe we should put a warning like "version: must be a LooseVersion" at the beginning of https://opendev.org/openstack/tripleo-quickstart/src/branch/master/ansible-collection-requirements.yml 19:54
* jm1 out for today, have a nice evening :)19:55
*** rlandy is now known as rlandy|biab21:27
rcastillogoing for a while, bbl21:34
* dasm => offline22:17
*** dasm is now known as dasm|off22:17
*** rlandy|biab is now known as rlandy23:17
*** dviroel is now known as dviroel|out23:29

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!