Thursday, 2022-03-03

« Wednesday, 2022-03-02 Index Friday, 2022-03-04 »
*** dviroel|out is now known as dviroel01:30
dviroeltested: content-provider-wallaby + compose from 02-24 - same results01:35
dviroelnow testing: content-provider-wallaby + compose from 02-23: https://review.rdoproject.org/zuul/status#35174,6601:36
* dviroel mirrors are CentOS-Stream-9-20220224.001:43
*** rlandy|ruck|bbl is now known as rlandy|ruck01:50
rlandy|ruckstill failing01:53
rlandy|ruckleaving note for morning ruck/rovers01:54
dviroelyes, still failing with previous compose too01:55
rlandy|ruckdviroel: late for you01:57
rlandy|ruckdviroel: I also reverted the THT change01:57
rlandy|ruckno luck there either01:57
rlandy|ruckand wallaby only01:58
rlandy|rucknot master01:58
rlandy|ruckdviroel: so if it was the compose01:58
rlandy|ruckit would hit both01:58
rlandy|rucksomething hit wallaby only01:58
rlandy|rucknot sure01:59
rlandy|ruckbut out for now01:59
rlandy|ruckwill pick this up tomorrow01:59
*** rlandy|ruck is now known as rlandy|out01:59
dviroelo/02:00
rlandy|out- job:02:00
rlandy|out    name: tripleo-ci-centos-9-content-provider-wallaby02:00
rlandy|out    parent: tripleo-ci-centos-9-content-provider02:00
rlandy|out    override-checkout: stable/wallaby02:00
rlandy|out    vars:02:00
rlandy|out      branch_override: stable/wallaby02:00
rlandy|outnothing diff02:00
rlandy|outanyways out02:00
dviroelack - good night02:00
*** dviroel is now known as dviroel|out02:02
*** ysandeep|out is now known as ysandeep04:36
ysandeeppojadhav: good morning o/ You can test your other component patches now, issue which we faced yesterday is resolved now.04:37
pojadhavysandeep, good morning \o04:38
pojadhavysandeep, ack thank you :)04:38
ysandeepchandankumar: good morning, around?04:46
chandankumarysandeep: yes04:53
chandankumarysandeep: Good morning04:54
ysandeepchandankumar: Hey o/ good morning, I am stuck debugging container build failure on 17 on rhel-9, pinged you details in internal pcci channel, Please let me know if you have any pointers.04:55
chandankumarysandeep: checking04:57
chandankumarhttps://review.opendev.org/c/openstack/tripleo-ci/+/83163106:02
*** ysandeep is now known as ysandeep|lunch07:21
*** ysandeep|lunch is now known as ysandeep07:49
*** amoralej|off is now known as amoralej08:09
jm1sshnaidm ysandeep: for 9d now i dont get a reply for my ticket regarding a static ip and dns entry for tripleo-cockpit (downstream). pinged him 2d ago but again silence. what should i do? wait or escalate?08:23
ysandeepjm1: Did you dropped a comment asking for update already?08:23
ysandeep>> pinged him 2d ago but again silence - On ticket?08:24
jm1ysandeep: asked for an update 9d and 2d ago08:24
ysandeepjm1, Could you share with me the ticket id in a pm. 08:25
*** jpena|off is now known as jpena08:37
jm1arxcruz: We found the (or at least one) cause of the devstack reboot issue. Thanks again for showing me all this cool tempest magic yesterday ^^09:12
jm1arxcruz: fixing these bugs is going to take a while but its good to know where we have to look :)09:12
*** ysandeep is now known as ysandeep|afk10:25
jm1ysandeep|afk: added you to https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/831643 because i dont know if the old non-zuul reproducer is used somewhere downstream10:37
amoralejchandankumar, the centos8 promotion pipeline for master is totally removed?10:49
mariosyeah amoralej 10:50
mariosshould be ;)10:50
amoralejthanks marios 10:57
*** rlandy|out is now known as rlandy|ruck11:12
rlandy|ruckchandankumar++ for the tripleo-common fix11:15
*** dviroel|out is now known as dviroel11:15
*** ysandeep|afk is now known as ysandeep11:17
chandankumarmarios: https://review.opendev.org/c/openstack/tripleo-common/+/83158011:20
* bhagyashris|ruck tea brb11:28
* pojadhav afk for sometime...11:40
*** amoralej is now known as amoralej|lunch12:00
*** dasm|off is now known as dasm|rover12:08
dasm|rovero/12:08
rlandy|ruckdasm|rover: gm12:10
rlandy|ruckdasm|rover: chandankumar has a fix for the gate blocker12:10
rlandy|ruckopenstack/tripleo-common12:10
rlandy|ruck831580,112:10
rlandy|rucktop of the gate noe12:10
rlandy|rucknow12:10
rlandy|ruckwhen that merges, pls respond to the openstack-discuss post that the blocker is fixed12:10
dasm|roveryes, bhagyashris|ruck notified me. Good job chandankumar++ bhagyashris|ruck++12:11
dasm|roverack12:11
rlandy|ruckyou can close out the bug and the card12:11
rlandy|ruckdasm|rover: also trigger jobs are working again12:11
rlandy|ruckbut we still have the fs039 issue12:11
rlandy|rucklet's try get that sort before we switch ruck/rovers12:11
rlandy|ruckbhagyashris|ruck: ^^ anything else?12:12
dasm|roverack, i'm gonna check that one now12:12
rlandy|ruckrhos-17 promoted yesterday12:12
rlandy|ruckwe need to get a 16-2 promotion12:12
rlandy|ruckbhagyashris|ruck: pls resume build on the two train lines here: https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/12:13
rlandy|ruck16.2 just kicked12:13
dasm|roverack12:17
bhagyashris|ruckrlandy|ruck, regarding fs039 and fs064 nothing from my side 12:19
rlandy|ruckbhagyashris|ruck: there is a suggested patch that didn't work12:20
rlandy|ruckcan you pick up on that12:20
bhagyashris|rucki checked that patch 12:20
dasm|roverthe patch got +W 12:22
chandankumarpojadhav: ysandeep please have a look https://code.engineering.redhat.com/gerrit/q/topic:%22cs-parenting-multinode%22+(status:open%20OR%20status:merged) when free, thanks!12:23
ysandeepchandankumar, ack12:24
pojadhavchandankumar, sure12:27
*** pojadhav- is now known as pojadhav12:44
ysandeepchandankumar, left comment https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/314098/7#message-05f4bdc2431a0fe0b68759a1586a2c3140aef53d , I may have misunderstood 12:54
jm1ysandeep sshnaidm team: testing static ip etc on downstream cockpit after scrum mtg, it might be unavailable for some seconds12:59
sshnaidmjm1, maybe worth to remove redhat-internal-ddns-client.noarch from the host?13:00
ysandeepjm1: ack13:00
bhagyashris|ruckchandankumar, rlandy|ruck soniya29 13:00
bhagyashris|ruckscrum time13:00
sshnaidmjm1, it should have update internal dns, but now it's not needed13:01
frenzy_fridaypls add to review list, https://review.rdoproject.org/r/c/config/+/39289, https://review.rdoproject.org/r/c/config/+/39444/1/playbooks/tripleo-rdo-base/container-login.yaml, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3999413:02
reviewbotI have added your review the to Review list13:02
reviewbotI have added your review the to Review list13:02
reviewbotI have added your review the to Review list13:02
jm1sshnaidm: let me first get the static ip working, then remove the dyndns stuff13:03
frenzy_fridaypls add to review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4002013:03
reviewbotI have added your review the to Review list13:03
sshnaidmjm1, sure, just wonder if it doesn't interfere with it.13:04
jm1sshnaidm: atm the issue is with dhcp assigning the wrong ip, dont know the cause yet13:06
chandankumarhttps://hackmd.io/St8HYZCITxy1lBkpM4Spqg?view#Tripelo-CI-Parenting-Structure13:06
dviroelpls add to review list https://review.opendev.org/c/openstack/tripleo-ci/+/83101513:08
reviewbotI have added your review the to Review list13:08
frenzy_fridayfolks, please is optional ;) The bot looks for "add" "review" "list"13:09
*** amoralej|lunch is now known as amoralej13:12
pojadhavpls add to review list, https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/315552, WIP : ttps://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/31557413:20
reviewbotI have added your review the to Review list13:20
frenzy_fridaybot, add to review list https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/315574 - WIP13:21
reviewbotI have added your review the to Review list13:21
frenzy_fridaybot looks for http unfortunately :(13:21
bhagyashris|ruckhttps://hackmd.io/s9hZvlP-SKuR4fkE3kL8NA13:22
chandankumarsomebody at door13:27
chandankumarwill be back soon13:27
rlandy|ruckdasm|rover: Merged openstack/tripleo-common stable/wallaby: Reinstall centos-stream-release from centos repos  https://review.opendev.org/c/openstack/tripleo-common/+/831580 - 13:41
rlandy|ruckfor response to openstack-dicuss13:41
rlandy|ruckdiscuss13:41
dasm|roverack. sent an update already13:41
rlandy|ruckcool13:41
rlandy|ruckthank you13:41
ysandeeprlandy|ruck, fyi.. https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/312030/3/zuul.d/standalone-jobs.yaml#166 - idea to create a new parent for rhel-9 , we will do this soon. 13:44
ysandeeppojadhav, ^^ If you want to pick that up.13:45
pojadhavysandeep, ack13:45
ysandeeppojadhav, thanks! You will need to create a new parent for scenario jobs as well - We can talk tomorrow's morning about the high level plan.13:46
pojadhavysandeep, yes sure !13:46
bhagyashris|ruckfrenzy_friday, soniya29 https://bugs.launchpad.net/tripleo/+bug/196272713:47
rlandy|ruckrcastillo: your new bug is not on CIX13:50
jm1ysandeep sshnaidm team: downstream cockpit is down for a while until dhcpd is cleaned by IT13:51
rcastillorlandy|ruck: mb forgot to set milestone13:52
* pojadhav stepping out for 1 hour13:52
chandankumarrlandy|ruck: ysandeep can we merge this https://review.rdoproject.org/r/c/config/+/40039 and I can test the rdo multinode parenting13:54
rlandy|ruckdviroel: you still working on https://trello.com/c/F00VnyBm/2336-cixlp1956475tripleociproa-centos9-nova-migration-breaks-because-sftp-server-is-not-part-of-the-migration-wrapper?13:54
rlandy|ruckskipping it for ruck rovers13:54
chandankumarsorry this one13:54
dviroelrlandy|ruck: yes - running testproject atm - had 2 different failures in last runs 13:55
chandankumarrlandy|ruck: ysandeep https://review.rdoproject.org/r/c/config/+/4002313:55
rlandy|ruckdviroel: thank you13:55
amoralejrlandy|ruck, flexiblas is fixed and pushed to centos and afs mirrors, so we shouldn't need the workaround anymore13:55
rlandy|ruckrcastillo: we now see the node prov failure in master line14:01
rcastillorlandy|ruck: :/ what jobs?14:02
rlandy|ruckOVB14:02
rcastilloI'll make sure it's the same error and update the bug14:04
rlandy|ruckrcastillo++14:05
rlandy|ruckthanks - pls see master line running now14:05
rcastillorlandy|ruck: does seem like the same issue. Overcloud nodes not booting14:10
jm1ysandeep sshnaidm: downstream cockpit is now at tripleo-cockpit.lab4.eng.bos.redhat.com14:10
jm1not yet completely done but at least you can connect14:11
ysandeepjm1, ack14:11
rlandy|ruckrcastillo: ok - let's get that raised14:11
sshnaidmjm1, cool, need to fix the nginx.conf14:12
jm1sshnaidm: yes, i am working on that14:12
rcastillorlandy|ruck: updated lp14:13
rcastillo(https://bugs.launchpad.net/tripleo/+bug/1962783)14:14
rlandy|ruckrcastillo: is your env still available?14:17
rlandy|ruckcan you pass it to harold for debug14:17
rlandy|ruckif so, let's get on #tripleo14:17
rlandy|ruckrcastillo: review.rdoproject.org/paste/show/253/ stil relevant? if so, pls add to bug14:18
marios /query rlandy|ruck 14:28
marios;)14:28
rlandy|ruckresponding to query14:29
rlandy|ruckamoralej: thank you - bhagyashris|ruck is testing the revert14:34
rlandy|ruckysandeep: is there a DF call today? 14:37
rlandy|ruckwill need to skip for planning14:37
bhagyashris|ruckamoralej, rlandy|ruck frenzy_friday soniya29 dasm|rover revert : https://review.opendev.org/c/openstack/tripleo-quickstart/+/831763 testproject patch https://review.rdoproject.org/r/c/testproject/+/3993214:42
dasm|roverbhagyashris|ruck: thx14:42
amoralejnice14:42
ysandeepykarel++14:47
ade_lee_dviroel, seems like we had some good success on the c9 jobs with the latest openssl ..14:56
jm1ysandeep sshnaidm: downstream cockpit is up again, available at http://tripleo-cockpit.lab4.eng.bos.redhat.com/14:57
ysandeepjm1++ thanks!14:57
rlandy|rucknice15:00
*** rlandy|ruck is now known as rlandy|ruck|mtg15:01
ade_lee_dviroel, lets see if we can figure out the remainder ..15:01
sshnaidmjm1++15:01
rlandy|ruck|mtghttp://tripleo-cockpit.usersys.redhat.com/ - down for me15:04
rlandy|ruck|mtgoh I see15:04
jm1rlandy|ruck|mtg: downstream cockpit now has a static ip and dns entry at http://tripleo-cockpit.lab4.eng.bos.redhat.com/ :)15:05
dviroelade_lee_: ack - need to take a look on multinode-ipa yet15:10
ade_lee_dviroel, yeah, for that - we need to turn fips off on the ipa node somehow15:11
chandankumarrlandy|ruck|mtg: I have added the merge order https://hackmd.io/FGMoCiRfSNa8puA1BpTQ-Q?view#Mar-03-202215:17
chandankumarPlease merge those by EOD15:18
rlandy|ruck|mtgchandankumar; thanks15:18
rlandy|ruck|mtg    40023: Use tripleo-ci-base-multinode-standard as parent for multinode | https://review.rdoproject.org/r/c/config/+/4002315:18
rlandy|ruck|mtg    40024: tripleo-ci-base-multinode-periodic-centos-9 as part for multinode job | https://review.rdoproject.org/r/c/rdo-jobs/+/4002415:18
rlandy|ruck|mtgchandankumar: ^^ just those two?15:18
chandankumarrlandy|ruck|mtg: yes15:18
rlandy|ruck|mtgok15:18
chandankumardownstream once we will merge in india morning15:19
*** dviroel is now known as dviroel|lunch15:22
marioschandankumar: https://review.rdoproject.org/r/c/rdo-jobs/+/40024/1#message-b388cf29fc72a57667b5f893e39141a9d2fd3eb916:01
*** ysandeep is now known as ysandeep|out16:04
*** dviroel|lunch is now known as dviroel16:27
chandankumarmarios: it is defined here https://review.rdoproject.org/r/c/config/+/4002316:28
marioschandankumar: ah config repo gotcha16:31
rcastillostepping away for a bit16:54
* dviroel__ reboots cpe16:56
*** dviroel__ is now known as dviroel16:56
*** marios is now known as marios|out17:09
*** rlandy|ruck|mtg is now known as rlandy|ruck17:11
rlandy|rucklunch - brb17:11
*** amoralej is now known as amoralej|off17:36
*** jpena is now known as jpena|off18:01
rlandy|ruckrcastillo: want to join review time?19:03
rlandy|ruckgoing to discuss OVB 19:03
* dviroel missed review time again :( 19:28
rlandy|ruckdviroel: just more vexx fires19:33
dasm|roverrlandy|ruck https://bugs.launchpad.net/tripleo/+bug/196354519:39
rlandy|ruckdasm|rover: thank you19:42
rlandy|ruckade_lee_: ^^ so it looks like we have multiple fun bugs going on with fs039 test right now19:43
rlandy|ruckthe fix put in for c8 has not even gotten to that point in the test19:43
rlandy|ruckand now c9 is failing on a whole other issue19:43
ade_lee_rlandy|ruck, sigh19:44
rlandy|ruckade_lee_: we can tackle them one at a time19:44
rlandy|ruckalso vexx is a little hosed19:44
rlandy|ruckthe nodes were reconfigured to support uefi boot19:44
rlandy|ruckand now are in an interesting state19:44
rlandy|ruckdasm|rover: let's see if the current running train line does better19:48
rlandy|ruckthen we can tell if vexx is in better shape or not19:48
ade_lee_ack -- I need to get a few test runs going -- I'll ping back shortly19:48
rlandy|ruckno rush19:50
rlandy|ruckdviroel: don't worry - there is always tomorrow morning20:03
rlandy|ruckrcastillo: laptop still behaving?20:05
rlandy|ruckdviroel: can you review these two:20:08
rlandy|ruck    40023: Use tripleo-ci-base-multinode-standard as parent for multinode | https://review.rdoproject.org/r/c/config/+/4002320:08
rlandy|ruck    40024: tripleo-ci-base-multinode-periodic-centos-9 as part for multinode job | https://review.rdoproject.org/r/c/rdo-jobs/+/4002420:08
rlandy|ruckchandankumar requested we merge those two in our afternoon20:08
rlandy|ruckI will w+ after a second vote20:08
rlandy|ruckfixing syntax error20:09
dviroelrlandy|ruck: ack, will do20:09
rlandy|ruckoh nvm20:10
rlandy|ruckwill resolve after the config one merged20:10
rlandy|ruckdviroel: ^^ so both are reviewable20:10
dviroelrlandy|ruck: done, can't spot any issue on those20:23
rcastillorlandy|ruck: laptop's fine20:41
rlandy|ruckthanks21:14
ade_lee_rlandy|ruck, hey - so ..21:26
ade_lee_rlandy|ruck, ready to to talk about fs39?21:26
rlandy|ruckade_lee_: sorry - digging through a bunch of vexxhost failures'21:34
rlandy|ruckthe cloud was updated to support uefi boot21:35
rlandy|ruckbut hosed a bunch fo other stuff21:35
ade_lee_rlandy|ruck, ok21:37
*** dviroel is now known as dviroel|out21:37
ade_lee_rlandy|ruck, whenever you're ready , we can chat about fs39 -- or we can leave it till tomorrow ..21:38
ade_lee_rlandy|ruck, if you pass me links to bugs, I can take a look in the meantime21:38
rlandy|ruckade_lee_:  https://bugs.launchpad.net/tripleo/+bug/196354521:40
rlandy|ruckbutt keep in mind vexx is unstable21:40
ade_lee_rlandy|ruck, I think we're seeing that instability here -- everything is working just fine - ipa operations are successful, and then they're not21:45
ade_lee_no obvious reason21:46
ade_lee_rlandy|ruck, we should recheck and see if it reproduces21:46
*** dviroel|out is now known as dviroel22:21
*** dasm|rover is now known as dasm|off23:16
* dasm|off is leaving23:16
dasm|offgood night team!23:16
rlandy|ruckgood night23:17
*** rlandy|ruck is now known as rlandy|out23:36
« Wednesday, 2022-03-02 Index Friday, 2022-03-04 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!